2024-01-16T14:43:37.244 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T14:43:37.244 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201 -- /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/orig.config.yaml 2024-01-16T14:43:37.288 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T14:43:37.382 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7518201' kernel: flavor: default kdb: true sha1: 2668949e0971849893a3fb7f177ce9032fc65780 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-clone-stats sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4708 sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-clone-stats suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi141.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbPa7to2H1Fo2Lztaf7r0dVUX1CbnG8svzBIMu06hrmzdrZqtiSu/xizzWD+IclGIcxfOOW6b5Ye8WtSJF94M0= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOnR5DedqAgWvakFkagD0XmrbMCQmAI6uIumY44uRbyRlPArCx1CvZo2GQzXUoiCygrjtlWRD+y1HctngfOkIWc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-16_14:38:39 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T14:43:37.382 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa; will attempt to use it 2024-01-16T14:43:37.383 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks 2024-01-16T14:43:37.383 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T14:43:37.385 INFO:teuthology.task.internal:Checking packages... 2024-01-16T14:43:37.408 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '73a4532fc4d97adee0b985d4d42874ea351a1c86' 2024-01-16T14:43:37.409 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T14:43:37.409 INFO:teuthology.packaging:ref: None 2024-01-16T14:43:37.409 INFO:teuthology.packaging:tag: None 2024-01-16T14:43:37.409 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-16T14:43:37.409 INFO:teuthology.packaging:sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:43:37.409 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-clone-stats 2024-01-16T14:43:37.564 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g73a4532f 2024-01-16T14:43:37.565 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T14:43:37.572 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T14:43:37.572 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T14:43:37.581 INFO:teuthology.task.internal:Saving configuration 2024-01-16T14:43:37.596 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T14:43:37.603 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T14:43:37.629 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:47.460653', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbPa7to2H1Fo2Lztaf7r0dVUX1CbnG8svzBIMu06hrmzdrZqtiSu/xizzWD+IclGIcxfOOW6b5Ye8WtSJF94M0='} 2024-01-16T14:43:37.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:47.462068', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOnR5DedqAgWvakFkagD0XmrbMCQmAI6uIumY44uRbyRlPArCx1CvZo2GQzXUoiCygrjtlWRD+y1HctngfOkIWc='} 2024-01-16T14:43:37.652 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T14:43:37.658 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-16T14:43:37.659 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-16T14:43:37.659 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T14:43:37.720 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7867ae0160>, signals=[15]) 2024-01-16T14:43:37.721 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T14:43:37.727 INFO:teuthology.task.internal:Opening connections... 2024-01-16T14:43:37.727 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:43:37.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:37.821 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:43:37.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:37.911 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T14:43:37.917 DEBUG:teuthology.orchestra.run.smithi141:> uname -m 2024-01-16T14:43:37.934 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2024-01-16T14:43:37.934 DEBUG:teuthology.orchestra.run.smithi141:> cat /etc/os-release 2024-01-16T14:43:37.991 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="CentOS Stream" 2024-01-16T14:43:37.991 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="9" 2024-01-16T14:43:37.991 INFO:teuthology.orchestra.run.smithi141.stdout:ID="centos" 2024-01-16T14:43:37.991 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:43:37.991 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="9" 2024-01-16T14:43:37.991 INFO:teuthology.orchestra.run.smithi141.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:ANSI_COLOR="0;31" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:43:37.992 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:43:37.993 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2024-01-16T14:43:38.016 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-01-16T14:43:38.033 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-01-16T14:43:38.033 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-01-16T14:43:38.089 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-01-16T14:43:38.089 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-01-16T14:43:38.089 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-01-16T14:43:38.089 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:43:38.089 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-01-16T14:43:38.089 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:43:38.090 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:43:38.090 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-01-16T14:43:38.115 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T14:43:38.124 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T14:43:38.129 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T14:43:38.130 DEBUG:teuthology.orchestra.run.smithi141:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:43:38.132 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:43:38.148 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T14:43:38.153 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T14:43:38.153 DEBUG:teuthology.orchestra.run.smithi141:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:43:38.188 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:43:38.204 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T14:43:38.262 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T14:43:38.276 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'} 2024-01-16T14:43:38.276 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, timeout 300 2024-01-16T14:43:38.276 DEBUG:teuthology.orchestra.run.smithi141:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:43:38.277 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:43:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:43:38.292 DEBUG:teuthology.task.kernel:sha1 for mon.b is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:38.292 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:43:38.292 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-01-16T14:43:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:43:38.294 DEBUG:teuthology.task.kernel:sha1 for mon.a is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:38.294 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:43:38.294 DEBUG:teuthology.orchestra.run.smithi141:> uname -r 2024-01-16T14:43:38.348 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:43:38.348 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:38.348 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:43:38.349 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:38.353 INFO:teuthology.orchestra.run.smithi141.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:43:38.353 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:38.353 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:43:38.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:38.489 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum upgrade -y linux-firmware 2024-01-16T14:43:38.497 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum upgrade -y linux-firmware 2024-01-16T14:43:41.192 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.4 MB 00:01 2024-01-16T14:43:41.200 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS-9 - AppStream 5.9 MB/s | 6.4 MB 00:01 2024-01-16T14:43:43.024 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-01-16T14:43:43.244 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS-9 - BaseOS 2.2 MB/s | 2.0 MB 00:00 2024-01-16T14:43:44.816 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-01-16T14:43:45.389 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-01-16T14:44:00.154 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-01-16T14:44:06.120 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 5.9 MB 00:02 2024-01-16T14:44:07.563 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-01-16T14:44:08.121 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - AppStream 851 kB/s | 19 MB 00:22 2024-01-16T14:44:16.454 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - CRB 1.2 MB/s | 5.9 MB 00:04 2024-01-16T14:44:18.071 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-01-16T14:44:18.598 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 20 MB 00:10 2024-01-16T14:44:19.315 INFO:teuthology.orchestra.run.smithi141.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-01-16T14:44:23.503 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-01-16T14:44:24.139 INFO:teuthology.orchestra.run.smithi141.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-01-16T14:44:25.812 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-01-16T14:44:25.823 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 1 Package 2024-01-16T14:44:25.824 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:44:25.825 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 375 M 2024-01-16T14:44:25.825 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-01-16T14:44:26.465 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2024-01-16T14:44:26.475 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2024-01-16T14:44:26.475 INFO:teuthology.orchestra.run.smithi141.stdout: Package Arch Version Repository Size 2024-01-16T14:44:26.475 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2024-01-16T14:44:26.475 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrading: 2024-01-16T14:44:26.476 INFO:teuthology.orchestra.run.smithi141.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:44:26.476 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:44:26.476 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2024-01-16T14:44:26.476 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2024-01-16T14:44:26.476 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrade 1 Package 2024-01-16T14:44:26.476 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:44:26.476 INFO:teuthology.orchestra.run.smithi141.stdout:Total download size: 375 M 2024-01-16T14:44:26.477 INFO:teuthology.orchestra.run.smithi141.stdout:Downloading Packages: 2024-01-16T14:44:40.655 INFO:teuthology.orchestra.run.smithi141.stdout:linux-firmware-20240109-142.el9.noarch.rpm 26 MB/s | 375 MB 00:14 2024-01-16T14:44:40.657 INFO:teuthology.orchestra.run.smithi141.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:44:40.657 INFO:teuthology.orchestra.run.smithi141.stdout:Total 26 MB/s | 375 MB 00:14 2024-01-16T14:44:40.657 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction check 2024-01-16T14:44:40.746 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction check succeeded. 2024-01-16T14:44:40.746 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction test 2024-01-16T14:44:43.210 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction test succeeded. 2024-01-16T14:44:43.211 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction 2024-01-16T14:44:44.242 INFO:teuthology.orchestra.run.smithi188.stdout:linux-firmware-20240109-142.el9.noarch.rpm 20 MB/s | 375 MB 00:18 2024-01-16T14:44:44.243 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:44:44.244 INFO:teuthology.orchestra.run.smithi188.stdout:Total 20 MB/s | 375 MB 00:18 2024-01-16T14:44:44.244 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-01-16T14:44:44.368 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-01-16T14:44:44.369 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-01-16T14:44:46.846 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-01-16T14:44:46.847 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-01-16T14:44:49.526 INFO:teuthology.orchestra.run.smithi141.stdout: Preparing : 1/1 2024-01-16T14:44:49.761 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:49.844 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:51.419 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:51.419 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:44:51.419 INFO:teuthology.orchestra.run.smithi141.stdout:Upgraded: 2024-01-16T14:44:51.420 INFO:teuthology.orchestra.run.smithi141.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:44:51.420 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:44:51.420 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2024-01-16T14:44:51.717 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:44:51.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:51.866 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:44:51.867 DEBUG:teuthology.orchestra.run.smithi141:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:44:51.869 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi141.front.sepia.ceph.com to complete... 2024-01-16T14:44:53.110 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-01-16T14:44:53.324 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:53.415 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:54.554 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:54.554 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:54.951 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:54.952 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:44:54.952 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-01-16T14:44:54.952 INFO:teuthology.orchestra.run.smithi188.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:44:54.952 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:44:54.952 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-01-16T14:44:55.248 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:44:55.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:55.395 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:44:55.395 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:44:55.398 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi188.front.sepia.ceph.com to complete... 2024-01-16T14:45:06.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16 14:45:06 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:45:06.193 INFO:teuthology.orchestra.run.smithi141.stderr:FINISHED --2024-01-16 14:45:06-- 2024-01-16T14:45:06.193 INFO:teuthology.orchestra.run.smithi141.stderr:Total wall clock time: 14s 2024-01-16T14:45:06.193 INFO:teuthology.orchestra.run.smithi141.stderr:Downloaded: 1 files, 315M in 14s (22.5 MB/s) 2024-01-16T14:45:06.195 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:45:06.196 DEBUG:teuthology.orchestra.run.smithi141:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:45:08.090 INFO:teuthology.orchestra.run.smithi141.stdout:Verifying... ######################################## 2024-01-16T14:45:08.158 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing... ######################################## 2024-01-16T14:45:08.237 INFO:teuthology.orchestra.run.smithi141.stdout:Updating / installing... 2024-01-16T14:45:15.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16 14:45:15 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:45:15.603 INFO:teuthology.orchestra.run.smithi188.stderr:FINISHED --2024-01-16 14:45:15-- 2024-01-16T14:45:15.603 INFO:teuthology.orchestra.run.smithi188.stderr:Total wall clock time: 20s 2024-01-16T14:45:15.603 INFO:teuthology.orchestra.run.smithi188.stderr:Downloaded: 1 files, 315M in 20s (16.0 MB/s) 2024-01-16T14:45:15.606 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:45:15.606 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:45:15.624 INFO:teuthology.orchestra.run.smithi141.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:45:17.501 INFO:teuthology.orchestra.run.smithi188.stdout:Verifying... ######################################## 2024-01-16T14:45:17.649 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing... ######################################## 2024-01-16T14:45:17.729 INFO:teuthology.orchestra.run.smithi188.stdout:Updating / installing... 2024-01-16T14:45:25.108 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:45:31.603 INFO:teuthology.orchestra.run.smithi141.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:45:41.187 INFO:teuthology.orchestra.run.smithi188.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:45:53.809 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi141.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:45:53.809 DEBUG:teuthology.orchestra.run.smithi141:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:53.887 INFO:teuthology.orchestra.run.smithi141.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:53.887 INFO:teuthology.orchestra.run.smithi141.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:45:53.888 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:45:53.889 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:45:53.890 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:45:53.891 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:45:53.892 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:45:53.893 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:45:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:45:53.895 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:45:53.896 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:45:53.897 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:45:53.898 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:45:53.899 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:45:53.900 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:45:53.901 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:45:53.902 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:45:53.903 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:45:53.904 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:45:53.905 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:45:53.906 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:45:53.907 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:45:53.908 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:45:53.909 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:45:53.910 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:45:53.911 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:45:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:45:53.913 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:45:53.914 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:45:53.915 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:45:53.916 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:45:53.917 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:45:53.918 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:45:53.919 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:45:53.920 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:45:53.921 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:45:53.922 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:45:53.923 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:45:53.924 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:45:53.925 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:45:53.926 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:45:53.927 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:45:53.928 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:45:53.929 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:45:53.930 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:45:53.931 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:45:53.932 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:45:53.933 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:45:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:45:53.935 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:45:53.936 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:45:53.937 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:45:53.938 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:45:53.939 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:45:53.940 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:45:53.941 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:45:53.942 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:45:53.943 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:45:53.944 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:45:53.945 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:45:53.946 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:45:53.947 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:45:53.948 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:45:53.949 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:45:53.950 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:45:53.951 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:45:53.952 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:45:53.953 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:45:53.954 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:45:53.955 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:45:53.956 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:45:53.957 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:45:53.958 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:45:53.959 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:45:53.960 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:45:53.961 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:45:53.962 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:45:53.963 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:45:53.964 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:45:53.965 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:45:53.966 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:45:53.967 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:45:53.968 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:45:53.969 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:45:53.970 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:45:53.971 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:45:53.972 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:45:53.973 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:45:53.974 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:45:53.975 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:45:53.976 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:45:53.977 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:45:53.978 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:45:53.979 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:45:53.980 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:45:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:45:53.982 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:45:53.983 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:45:53.984 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:45:53.985 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:45:53.986 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:45:53.987 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:45:53.988 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:45:53.989 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:45:53.990 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:45:53.991 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:45:53.992 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:45:53.993 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:45:53.994 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:45:53.995 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:45:53.996 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:45:53.997 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:45:53.998 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:45:53.999 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:45:54.000 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:45:54.001 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:45:54.002 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:45:54.003 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:45:54.004 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:45:54.005 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:45:54.006 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:45:54.007 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:45:54.008 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:45:54.009 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:45:54.010 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:45:54.011 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:45:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:45:54.013 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:45:54.014 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:45:54.014 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:45:54.014 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:45:54.014 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:45:54.014 INFO:teuthology.orchestra.run.smithi141.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:45:54.014 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:45:54.014 DEBUG:teuthology.orchestra.run.smithi141:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:54.036 INFO:teuthology.orchestra.run.smithi141.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:45:54.036 INFO:teuthology.orchestra.run.smithi141.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:45:54.036 INFO:teuthology.orchestra.run.smithi141.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:54.036 INFO:teuthology.orchestra.run.smithi141.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:54.036 INFO:teuthology.orchestra.run.smithi141.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:54.036 INFO:teuthology.orchestra.run.smithi141.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:54.036 INFO:teuthology.orchestra.run.smithi141.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:fi 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:fi 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:45:54.037 INFO:teuthology.orchestra.run.smithi141.stdout:fi 2024-01-16T14:45:54.038 DEBUG:teuthology.orchestra.run.smithi141:> sudo rpm -qi grub2-tools 2024-01-16T14:45:54.112 INFO:teuthology.orchestra.run.smithi141.stdout:Name : grub2-tools 2024-01-16T14:45:54.112 INFO:teuthology.orchestra.run.smithi141.stdout:Epoch : 1 2024-01-16T14:45:54.112 INFO:teuthology.orchestra.run.smithi141.stdout:Version : 2.06 2024-01-16T14:45:54.112 INFO:teuthology.orchestra.run.smithi141.stdout:Release : 68.el9 2024-01-16T14:45:54.112 INFO:teuthology.orchestra.run.smithi141.stdout:Architecture: x86_64 2024-01-16T14:45:54.112 INFO:teuthology.orchestra.run.smithi141.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Group : Unspecified 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Size : 8264254 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:License : GPLv3+ 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Packager : builder@centos.org 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Vendor : CentOS 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Summary : Support tools for GRUB. 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:Description : 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:45:54.113 INFO:teuthology.orchestra.run.smithi141.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:45:54.114 INFO:teuthology.orchestra.run.smithi141.stdout:hardware devices. 2024-01-16T14:45:54.114 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:45:54.114 INFO:teuthology.orchestra.run.smithi141.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:45:54.115 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:45:54.115 INFO:teuthology.task.kernel:Updating grub on smithi141 to boot 6.7.0-g2668949e0971 2024-01-16T14:45:54.115 DEBUG:teuthology.orchestra.run.smithi141:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:45:54.701 INFO:teuthology.orchestra.run.smithi141.stderr:Generating grub configuration file ... 2024-01-16T14:45:56.074 INFO:teuthology.orchestra.run.smithi141.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:45:56.094 INFO:teuthology.orchestra.run.smithi141.stderr:done 2024-01-16T14:45:56.096 DEBUG:teuthology.orchestra.run.smithi141:> mktemp 2024-01-16T14:45:56.111 INFO:teuthology.orchestra.run.smithi141.stdout:/tmp/tmp.wWVBdvimpB 2024-01-16T14:45:56.111 DEBUG:teuthology.orchestra.run.smithi141:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wWVBdvimpB 2024-01-16T14:45:56.179 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 0666 /tmp/tmp.wWVBdvimpB 2024-01-16T14:45:56.321 DEBUG:teuthology.orchestra.remote:smithi141:/tmp/tmp.wWVBdvimpB is 6KB 2024-01-16T14:45:56.370 DEBUG:teuthology.orchestra.run.smithi141:> rm -fr /tmp/tmp.wWVBdvimpB 2024-01-16T14:45:56.383 DEBUG:teuthology.orchestra.run.smithi141:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:45:56.448 INFO:teuthology.orchestra.run.smithi141.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:45:56.448 INFO:teuthology.orchestra.run.smithi141.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:45:56.448 INFO:teuthology.orchestra.run.smithi141.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:45:56.448 INFO:teuthology.orchestra.run.smithi141.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:45:56.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:45:56.607 DEBUG:teuthology.orchestra.run.smithi141:> sudo shutdown -r now 2024-01-16T14:46:03.344 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:46:03.344 DEBUG:teuthology.orchestra.run.smithi188:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:46:03.420 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:46:03.421 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:46:03.422 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:46:03.423 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:46:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:46:03.425 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:46:03.426 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:46:03.427 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:46:03.428 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:46:03.429 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:46:03.430 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:46:03.431 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:46:03.432 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:46:03.433 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:46:03.434 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:46:03.435 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:46:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:46:03.437 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:46:03.438 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:46:03.439 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:46:03.440 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:46:03.441 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:46:03.442 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:46:03.443 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:46:03.444 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:46:03.445 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:46:03.446 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:46:03.447 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:46:03.448 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:46:03.449 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:46:03.450 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:46:03.451 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:46:03.452 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:46:03.453 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:46:03.454 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:46:03.455 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:46:03.456 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:46:03.457 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:46:03.458 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:46:03.459 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:46:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:46:03.461 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:46:03.462 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:46:03.463 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:46:03.464 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:46:03.465 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:46:03.466 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:46:03.467 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:46:03.468 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:46:03.469 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:46:03.470 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:46:03.471 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:46:03.472 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:46:03.473 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:46:03.474 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:46:03.475 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:46:03.476 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:46:03.477 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:46:03.478 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:46:03.479 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:46:03.480 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:46:03.481 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:46:03.482 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:46:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:46:03.484 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:46:03.485 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:46:03.486 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:46:03.487 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:46:03.488 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:46:03.489 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:46:03.491 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:46:03.492 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:46:03.493 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:46:03.494 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:46:03.495 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:46:03.496 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:46:03.497 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:46:03.498 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:46:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:46:03.500 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:46:03.501 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:46:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:46:03.503 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:46:03.504 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:46:03.505 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:46:03.506 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:46:03.507 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:46:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:46:03.510 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:46:03.511 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:46:03.513 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:46:03.516 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:46:03.517 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:46:03.518 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:46:03.519 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:46:03.520 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:46:03.521 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:46:03.522 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:46:03.523 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:46:03.524 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:46:03.525 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:46:03.526 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:46:03.527 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:46:03.528 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:46:03.529 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:46:03.530 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:46:03.531 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:46:03.532 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:46:03.533 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:46:03.534 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:46:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:46:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:46:03.537 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:46:03.538 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:46:03.539 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:46:03.540 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:46:03.541 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:46:03.542 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:46:03.543 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:46:03.544 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:46:03.545 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:46:03.546 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:46:03.547 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:46:03.548 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:46:03.549 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:46:03.550 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:46:03.551 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:46:03.552 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:46:03.553 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:46:03.554 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:46:03.554 DEBUG:teuthology.orchestra.run.smithi188:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:46:03.575 INFO:teuthology.orchestra.run.smithi188.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:fi 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:46:03.576 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:46:03.577 INFO:teuthology.orchestra.run.smithi188.stdout:fi 2024-01-16T14:46:03.577 INFO:teuthology.orchestra.run.smithi188.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:46:03.577 INFO:teuthology.orchestra.run.smithi188.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:46:03.577 INFO:teuthology.orchestra.run.smithi188.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:46:03.577 INFO:teuthology.orchestra.run.smithi188.stdout:fi 2024-01-16T14:46:03.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.06 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 68.el9 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Group : Unspecified 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 8264254 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:46:03.653 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : builder@centos.org 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:46:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:46:03.655 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:46:03.655 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 6.7.0-g2668949e0971 2024-01-16T14:46:03.655 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:46:04.211 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-01-16T14:46:05.690 INFO:teuthology.orchestra.run.smithi188.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:46:05.727 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-01-16T14:46:05.729 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-01-16T14:46:05.754 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.80NZ1iXqjt 2024-01-16T14:46:05.755 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.80NZ1iXqjt 2024-01-16T14:46:05.823 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.80NZ1iXqjt 2024-01-16T14:46:05.966 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.80NZ1iXqjt is 6KB 2024-01-16T14:46:06.014 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.80NZ1iXqjt 2024-01-16T14:46:06.028 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:46:06.098 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:46:06.098 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:46:06.098 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:46:06.098 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:46:06.100 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:46:06.259 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-01-16T14:46:26.630 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-16T14:46:26.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-16T14:46:26.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:36.270 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-16T14:46:36.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-01-16T14:46:36.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:45.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-16T14:46:54.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-16T14:46:54.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:54.784 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-01-16T14:46:57.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-16T14:47:03.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-01-16T14:47:03.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:09.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-16T14:47:09.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:27.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-16T14:47:42.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-16T14:47:42.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:44.478 DEBUG:teuthology.orchestra.run.smithi141:> true 2024-01-16T14:47:45.104 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-16T14:47:45.105 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.7.0-g2668949e0971"... 2024-01-16T14:47:45.105 DEBUG:teuthology.orchestra.run.smithi141:> uname -r 2024-01-16T14:47:45.148 INFO:teuthology.orchestra.run.smithi141.stdout:6.7.0-g2668949e0971 2024-01-16T14:47:45.148 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:47:45.148 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:47:45.148 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:47:46.149 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-16T14:47:46.150 DEBUG:teuthology.orchestra.run.smithi141:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:47:46.267 INFO:teuthology.orchestra.run.smithi141.stdout:ttyS1 2024-01-16T14:47:46.308 DEBUG:teuthology.parallel:result is None 2024-01-16T14:48:03.791 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T14:48:15.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-01-16T14:48:15.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:48:16.197 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-01-16T14:48:16.838 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-01-16T14:48:16.838 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.7.0-g2668949e0971"... 2024-01-16T14:48:16.838 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-01-16T14:48:16.891 INFO:teuthology.orchestra.run.smithi188.stdout:6.7.0-g2668949e0971 2024-01-16T14:48:16.891 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:48:16.891 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:48:16.891 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:48:17.892 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-16T14:48:17.892 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:48:18.029 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-01-16T14:48:18.091 DEBUG:teuthology.parallel:result is None 2024-01-16T14:48:18.091 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T14:48:18.099 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T14:48:18.099 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:48:18.108 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:48:18.163 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T14:48:18.170 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T14:48:18.177 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T14:48:18.177 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:48:18.184 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:48:18.248 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T14:48:18.254 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T14:48:18.254 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:48:18.262 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:48:18.388 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:18.395 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:18.430 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:48:18.444 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T14:48:18.450 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T14:48:18.451 DEBUG:teuthology.orchestra.run.smithi141:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:48:18.458 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:48:18.548 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T14:48:18.558 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T14:48:18.558 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:48:18.565 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:48:18.617 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:18.696 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:18.793 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:18.877 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:18.975 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:48:18.975 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:48:19.066 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:19.151 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:48:19.254 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:19.340 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:48:19.435 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:48:19.435 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:48:19.524 DEBUG:teuthology.orchestra.run.smithi141:> sudo service rsyslog restart 2024-01-16T14:48:19.532 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-01-16T14:48:19.659 INFO:teuthology.orchestra.run.smithi141.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:48:19.683 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:48:20.184 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T14:48:20.192 INFO:teuthology.task.internal:Starting timer... 2024-01-16T14:48:20.192 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T14:48:20.202 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T14:48:20.210 DEBUG:teuthology.orchestra.run.smithi141:> sudo service auditd rotate 2024-01-16T14:48:20.333 INFO:teuthology.orchestra.run.smithi141.stdout:Rotating logs: 2024-01-16T14:48:20.337 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-01-16T14:48:20.480 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-01-16T14:48:20.486 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T14:48:20.486 DEBUG:teuthology.orchestra.run.smithi141:> /usr/sbin/getenforce 2024-01-16T14:48:20.551 INFO:teuthology.orchestra.run.smithi141.stdout:Permissive 2024-01-16T14:48:20.552 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-01-16T14:48:20.608 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-01-16T14:48:20.609 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi141.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-01-16T14:48:20.609 DEBUG:teuthology.orchestra.run.smithi141:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:48:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:48:20.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:48:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:48:20.796 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T14:48:20.796 DEBUG:teuthology.orchestra.run.smithi141:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:48:20.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:48:20.997 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T14:48:21.007 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T14:48:21.022 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T14:48:21.023 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi141.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T14:53:57.419 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi141.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-01-16T14:53:57.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-16T14:53:57.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:53:57.566 DEBUG:teuthology.orchestra.run.smithi141:> true 2024-01-16T14:53:57.716 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-16T14:53:57.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-01-16T14:53:57.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:53:57.855 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-01-16T14:53:58.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-01-16T14:53:58.003 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T14:53:58.014 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T14:53:58.014 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:53:58.014 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:53:58.022 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:53:58.022 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:53:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:53:58.119 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:53:58.171 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:53:58.172 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:53:58.268 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: ntpd: command not found 2024-01-16T14:53:58.270 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-01-16T14:53:58.311 INFO:teuthology.orchestra.run.smithi141.stdout:506 Cannot talk to daemon 2024-01-16T14:53:58.315 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-01-16T14:53:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:53:58.368 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:53:58.416 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:53:58.422 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:53:58.542 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:53:58.547 INFO:teuthology.orchestra.run.smithi141.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:53:58.551 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:53:58.551 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-01-16T14:53:58.551 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.551 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.551 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.551 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.556 INFO:teuthology.orchestra.run.smithi141.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:53:58.556 INFO:teuthology.orchestra.run.smithi141.stdout:=============================================================================== 2024-01-16T14:53:58.557 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.557 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.557 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.557 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:53:58.558 INFO:teuthology.run_tasks:Running task install... 2024-01-16T14:53:58.567 DEBUG:teuthology.task.install:project ceph 2024-01-16T14:53:58.567 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}} 2024-01-16T14:53:58.567 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'} 2024-01-16T14:53:58.567 INFO:teuthology.task.install:Using flavor: default 2024-01-16T14:53:58.573 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T14:53:58.573 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-16T14:53:58.574 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:53:58.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:53:58.576 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:53:58.576 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:53:58.726 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:53:58.726 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:53:58.816 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:53:58.816 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:53:58.836 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:53:58.836 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:53:58.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:53:58.933 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:53:58.933 DEBUG:teuthology.orchestra.run.smithi141:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:53:58.990 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:53:58.990 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:53:58.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:53:59.073 DEBUG:teuthology.orchestra.run.smithi141:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:53:59.084 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:53:59.084 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:53:59.204 DEBUG:teuthology.orchestra.run.smithi141:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:53:59.223 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:53:59.294 INFO:teuthology.orchestra.run.smithi141.stdout:check_obsoletes = 1 2024-01-16T14:53:59.299 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum clean all 2024-01-16T14:53:59.350 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:53:59.439 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-01-16T14:53:59.444 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-01-16T14:53:59.655 INFO:teuthology.orchestra.run.smithi141.stdout:73 files removed 2024-01-16T14:53:59.697 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:53:59.791 INFO:teuthology.orchestra.run.smithi188.stdout:73 files removed 2024-01-16T14:53:59.830 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:54:00.555 INFO:teuthology.orchestra.run.smithi141.stdout:ceph packages for x86_64 212 kB/s | 78 kB 00:00 2024-01-16T14:54:00.877 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 144 kB/s | 78 kB 00:00 2024-01-16T14:54:01.039 INFO:teuthology.orchestra.run.smithi141.stdout:ceph noarch packages 24 kB/s | 9.5 kB 00:00 2024-01-16T14:54:01.402 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2024-01-16T14:54:01.523 INFO:teuthology.orchestra.run.smithi141.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-01-16T14:54:02.061 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-01-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2024-01-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2024-01-16T14:54:03.961 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-01-16T14:54:04.307 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-01-16T14:54:05.536 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-01-16T14:54:06.498 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-01-16T14:54:07.753 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-01-16T14:54:08.740 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-01-16T14:54:13.724 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 5.9 MB 00:01 2024-01-16T14:54:15.341 INFO:teuthology.orchestra.run.smithi141.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2024-01-16T14:54:16.620 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 5.9 MB 00:05 2024-01-16T14:54:18.579 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2024-01-16T14:54:21.078 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 20 MB 00:02 2024-01-16T14:54:24.758 INFO:teuthology.orchestra.run.smithi141.stdout:Extra Packages for Enterprise Linux 2.2 MB/s | 20 MB 00:09 2024-01-16T14:54:26.705 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-01-16T14:54:29.448 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:29.449 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:29.468 INFO:teuthology.orchestra.run.smithi188.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:54:29.469 INFO:teuthology.orchestra.run.smithi188.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:54:29.471 INFO:teuthology.orchestra.run.smithi188.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:54:29.472 INFO:teuthology.orchestra.run.smithi188.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:29.474 INFO:teuthology.orchestra.run.smithi188.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:29.478 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:54:29.479 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:54:29.560 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-01-16T14:54:29.579 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-01-16T14:54:29.579 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-01-16T14:54:29.579 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-01-16T14:54:29.579 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:54:29.580 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:54:29.581 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:54:29.582 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:54:29.583 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:54:29.584 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:54:29.585 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:54:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:54:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:54:29.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:54:29.589 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout:Install 135 Packages 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 16 Packages 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 204 M 2024-01-16T14:54:29.590 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-01-16T14:54:30.356 INFO:teuthology.orchestra.run.smithi188.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-01-16T14:54:30.358 INFO:teuthology.orchestra.run.smithi141.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-01-16T14:54:30.556 INFO:teuthology.orchestra.run.smithi188.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 4.2 MB/s | 861 kB 00:00 2024-01-16T14:54:30.623 INFO:teuthology.orchestra.run.smithi188.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 2.1 MB/s | 141 kB 00:00 2024-01-16T14:54:30.873 INFO:teuthology.orchestra.run.smithi188.stdout:(4/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 6.8 MB/s | 5.2 MB 00:00 2024-01-16T14:54:30.948 INFO:teuthology.orchestra.run.smithi188.stdout:(5/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 6.4 MB/s | 2.1 MB 00:00 2024-01-16T14:54:31.015 INFO:teuthology.orchestra.run.smithi188.stdout:(6/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 10 MB/s | 1.5 MB 00:00 2024-01-16T14:54:31.324 INFO:teuthology.orchestra.run.smithi188.stdout:(7/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 12 MB/s | 4.4 MB 00:00 2024-01-16T14:54:32.182 INFO:teuthology.orchestra.run.smithi188.stdout:(8/151): ceph-radosgw-18.0.0-7629.g73a4532f.el9 10 MB/s | 8.6 MB 00:00 2024-01-16T14:54:32.232 INFO:teuthology.orchestra.run.smithi188.stdout:(9/151): ceph-selinux-18.0.0-7629.g73a4532f.el9 511 kB/s | 25 kB 00:00 2024-01-16T14:54:32.466 INFO:teuthology.orchestra.run.smithi188.stdout:(10/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 12 MB/s | 18 MB 00:01 2024-01-16T14:54:32.542 INFO:teuthology.orchestra.run.smithi188.stdout:(11/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 2.9 MB/s | 217 kB 00:00 2024-01-16T14:54:32.608 INFO:teuthology.orchestra.run.smithi188.stdout:(12/151): libcephfs-devel-18.0.0-7629.g73a4532f 472 kB/s | 31 kB 00:00 2024-01-16T14:54:32.742 INFO:teuthology.orchestra.run.smithi188.stdout:(13/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 5.3 MB/s | 719 kB 00:00 2024-01-16T14:54:32.809 INFO:teuthology.orchestra.run.smithi188.stdout:(14/151): libcephsqlite-18.0.0-7629.g73a4532f.e 2.4 MB/s | 163 kB 00:00 2024-01-16T14:54:32.892 INFO:teuthology.orchestra.run.smithi188.stdout:(15/151): librados-devel-18.0.0-7629.g73a4532f. 1.5 MB/s | 127 kB 00:00 2024-01-16T14:54:32.985 INFO:teuthology.orchestra.run.smithi188.stdout:(16/151): libradosstriper1-18.0.0-7629.g73a4532 5.1 MB/s | 476 kB 00:00 2024-01-16T14:54:33.046 INFO:teuthology.orchestra.run.smithi141.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:33.048 INFO:teuthology.orchestra.run.smithi141.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:54:33.068 INFO:teuthology.orchestra.run.smithi141.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:54:33.070 INFO:teuthology.orchestra.run.smithi141.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:54:33.071 INFO:teuthology.orchestra.run.smithi141.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:54:33.072 INFO:teuthology.orchestra.run.smithi141.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:33.075 INFO:teuthology.orchestra.run.smithi141.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:54:33.079 INFO:teuthology.orchestra.run.smithi141.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:54:33.080 INFO:teuthology.orchestra.run.smithi141.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:54:33.160 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2024-01-16T14:54:33.178 INFO:teuthology.orchestra.run.smithi141.stdout:============================================================================================ 2024-01-16T14:54:33.178 INFO:teuthology.orchestra.run.smithi141.stdout: Package Arch Version Repository Size 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout:============================================================================================ 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout:Installing: 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:54:33.179 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:54:33.180 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrading: 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:54:33.181 INFO:teuthology.orchestra.run.smithi141.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout:Installing dependencies: 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:54:33.182 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:54:33.183 INFO:teuthology.orchestra.run.smithi141.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:54:33.184 INFO:teuthology.orchestra.run.smithi141.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:54:33.185 INFO:teuthology.orchestra.run.smithi141.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:54:33.186 INFO:teuthology.orchestra.run.smithi141.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:54:33.187 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:54:33.188 INFO:teuthology.orchestra.run.smithi141.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout:Installing weak dependencies: 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout:============================================================================================ 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout:Install 135 Packages 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrade 16 Packages 2024-01-16T14:54:33.189 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:54:33.190 INFO:teuthology.orchestra.run.smithi141.stdout:Total download size: 204 M 2024-01-16T14:54:33.190 INFO:teuthology.orchestra.run.smithi141.stdout:Downloading Packages: 2024-01-16T14:54:33.418 INFO:teuthology.orchestra.run.smithi188.stdout:(17/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 11 MB/s | 4.8 MB 00:00 2024-01-16T14:54:33.477 INFO:teuthology.orchestra.run.smithi188.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g73 789 kB/s | 45 kB 00:00 2024-01-16T14:54:33.535 INFO:teuthology.orchestra.run.smithi188.stdout:(19/151): python3-ceph-common-18.0.0-7629.g73a4 2.2 MB/s | 129 kB 00:00 2024-01-16T14:54:33.602 INFO:teuthology.orchestra.run.smithi188.stdout:(20/151): python3-cephfs-18.0.0-7629.g73a4532f. 2.4 MB/s | 161 kB 00:00 2024-01-16T14:54:33.686 INFO:teuthology.orchestra.run.smithi188.stdout:(21/151): python3-rados-18.0.0-7629.g73a4532f.e 3.8 MB/s | 321 kB 00:00 2024-01-16T14:54:33.770 INFO:teuthology.orchestra.run.smithi188.stdout:(22/151): python3-rbd-18.0.0-7629.g73a4532f.el9 3.5 MB/s | 298 kB 00:00 2024-01-16T14:54:33.845 INFO:teuthology.orchestra.run.smithi188.stdout:(23/151): python3-rgw-18.0.0-7629.g73a4532f.el9 1.3 MB/s | 99 kB 00:00 2024-01-16T14:54:33.912 INFO:teuthology.orchestra.run.smithi188.stdout:(24/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 1.3 MB/s | 85 kB 00:00 2024-01-16T14:54:34.262 INFO:teuthology.orchestra.run.smithi188.stdout:(25/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 8.6 MB/s | 3.0 MB 00:00 2024-01-16T14:54:34.309 INFO:teuthology.orchestra.run.smithi141.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-01-16T14:54:34.329 INFO:teuthology.orchestra.run.smithi188.stdout:(26/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 2.5 MB/s | 171 kB 00:00 2024-01-16T14:54:34.404 INFO:teuthology.orchestra.run.smithi188.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 378 kB/s | 28 kB 00:00 2024-01-16T14:54:34.496 INFO:teuthology.orchestra.run.smithi188.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 1.4 MB/s | 133 kB 00:00 2024-01-16T14:54:34.676 INFO:teuthology.orchestra.run.smithi141.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 2.3 MB/s | 861 kB 00:00 2024-01-16T14:54:34.752 INFO:teuthology.orchestra.run.smithi141.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 1.9 MB/s | 141 kB 00:00 2024-01-16T14:54:34.938 INFO:teuthology.orchestra.run.smithi188.stdout:(29/151): ceph-test-18.0.0-7629.g73a4532f.el9.x 15 MB/s | 41 MB 00:02 2024-01-16T14:54:35.031 INFO:teuthology.orchestra.run.smithi188.stdout:(30/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 3.3 MB/s | 1.8 MB 00:00 2024-01-16T14:54:35.093 INFO:teuthology.orchestra.run.smithi141.stdout:(4/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 6.1 MB/s | 2.1 MB 00:00 2024-01-16T14:54:35.097 INFO:teuthology.orchestra.run.smithi188.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g73 3.7 MB/s | 246 kB 00:00 2024-01-16T14:54:35.189 INFO:teuthology.orchestra.run.smithi188.stdout:(32/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 553 kB/s | 50 kB 00:00 2024-01-16T14:54:35.373 INFO:teuthology.orchestra.run.smithi188.stdout:(33/151): ceph-prometheus-alerts-18.0.0-7629.g7 83 kB/s | 15 kB 00:00 2024-01-16T14:54:35.452 INFO:teuthology.orchestra.run.smithi141.stdout:(5/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 4.1 MB/s | 1.5 MB 00:00 2024-01-16T14:54:35.465 INFO:teuthology.orchestra.run.smithi188.stdout:(34/151): ceph-volume-18.0.0-7629.g73a4532f.el9 2.8 MB/s | 262 kB 00:00 2024-01-16T14:54:35.640 INFO:teuthology.orchestra.run.smithi188.stdout:(35/151): ceph-mgr-diskprediction-local-18.0.0- 11 MB/s | 7.4 MB 00:00 2024-01-16T14:54:35.707 INFO:teuthology.orchestra.run.smithi188.stdout:(36/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 3.2 MB/s | 778 kB 00:00 2024-01-16T14:54:35.741 INFO:teuthology.orchestra.run.smithi188.stdout:(37/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 183 kB/s | 18 kB 00:00 2024-01-16T14:54:36.066 INFO:teuthology.orchestra.run.smithi188.stdout:(38/151): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-01-16T14:54:36.124 INFO:teuthology.orchestra.run.smithi188.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 920 kB 00:00 2024-01-16T14:54:36.150 INFO:teuthology.orchestra.run.smithi188.stdout:(40/151): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 106 kB 00:00 2024-01-16T14:54:36.152 INFO:teuthology.orchestra.run.smithi141.stdout:(6/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 6.3 MB/s | 4.4 MB 00:00 2024-01-16T14:54:36.175 INFO:teuthology.orchestra.run.smithi188.stdout:(41/151): elfutils-libelf-devel-0.190-2.el9.x86 420 kB/s | 20 kB 00:00 2024-01-16T14:54:36.209 INFO:teuthology.orchestra.run.smithi188.stdout:(42/151): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-01-16T14:54:36.234 INFO:teuthology.orchestra.run.smithi188.stdout:(43/151): flexiblas-3.0.4-8.el9.x86_64.rpm 572 kB/s | 32 kB 00:00 2024-01-16T14:54:36.393 INFO:teuthology.orchestra.run.smithi188.stdout:(44/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-01-16T14:54:36.418 INFO:teuthology.orchestra.run.smithi188.stdout:(45/151): flexiblas-openblas-openmp-3.0.4-8.el9 95 kB/s | 17 kB 00:00 2024-01-16T14:54:36.460 INFO:teuthology.orchestra.run.smithi188.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 248 kB/s | 10 kB 00:00 2024-01-16T14:54:36.485 INFO:teuthology.orchestra.run.smithi188.stdout:(47/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-01-16T14:54:36.519 INFO:teuthology.orchestra.run.smithi188.stdout:(48/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 778 kB/s | 44 kB 00:00 2024-01-16T14:54:36.569 INFO:teuthology.orchestra.run.smithi188.stdout:(49/151): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-01-16T14:54:36.594 INFO:teuthology.orchestra.run.smithi141.stdout:(7/151): ceph-common-18.0.0-7629.g73a4532f.el9. 7.0 MB/s | 19 MB 00:02 2024-01-16T14:54:36.602 INFO:teuthology.orchestra.run.smithi188.stdout:(50/151): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2024-01-16T14:54:36.628 INFO:teuthology.orchestra.run.smithi188.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 460 kB/s | 24 kB 00:00 2024-01-16T14:54:36.673 INFO:teuthology.orchestra.run.smithi188.stdout:(52/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 670 kB/s | 46 kB 00:00 2024-01-16T14:54:36.703 INFO:teuthology.orchestra.run.smithi188.stdout:(53/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-01-16T14:54:36.727 INFO:teuthology.orchestra.run.smithi141.stdout:(8/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 1.8 MB/s | 5.2 MB 00:02 2024-01-16T14:54:36.737 INFO:teuthology.orchestra.run.smithi188.stdout:(54/151): lua-5.4.4-4.el9.x86_64.rpm 3.0 MB/s | 187 kB 00:00 2024-01-16T14:54:36.754 INFO:teuthology.orchestra.run.smithi188.stdout:(55/151): openblas-0.3.21-2.el9.x86_64.rpm 713 kB/s | 34 kB 00:00 2024-01-16T14:54:36.828 INFO:teuthology.orchestra.run.smithi141.stdout:(9/151): ceph-selinux-18.0.0-7629.g73a4532f.el9 254 kB/s | 25 kB 00:00 2024-01-16T14:54:36.946 INFO:teuthology.orchestra.run.smithi188.stdout:(56/151): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-01-16T14:54:37.038 INFO:teuthology.orchestra.run.smithi188.stdout:(57/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-01-16T14:54:37.105 INFO:teuthology.orchestra.run.smithi188.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-01-16T14:54:37.155 INFO:teuthology.orchestra.run.smithi188.stdout:(59/151): postgresql-contrib-13.11-1.el9.x86_64 7.7 MB/s | 811 kB 00:00 2024-01-16T14:54:37.183 INFO:teuthology.orchestra.run.smithi188.stdout:(60/151): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2024-01-16T14:54:37.360 INFO:teuthology.orchestra.run.smithi141.stdout:(10/151): ceph-radosgw-18.0.0-7629.g73a4532f.el 11 MB/s | 8.6 MB 00:00 2024-01-16T14:54:37.397 INFO:teuthology.orchestra.run.smithi188.stdout:(61/151): postgresql-server-13.11-1.el9.x86_64. 24 MB/s | 5.7 MB 00:00 2024-01-16T14:54:37.428 INFO:teuthology.orchestra.run.smithi141.stdout:(11/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 3.2 MB/s | 217 kB 00:00 2024-01-16T14:54:37.487 INFO:teuthology.orchestra.run.smithi141.stdout:(12/151): libcephfs-devel-18.0.0-7629.g73a4532f 540 kB/s | 31 kB 00:00 2024-01-16T14:54:37.573 INFO:teuthology.orchestra.run.smithi188.stdout:(62/151): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-01-16T14:54:37.587 INFO:teuthology.orchestra.run.smithi141.stdout:(13/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 7.1 MB/s | 719 kB 00:00 2024-01-16T14:54:37.631 INFO:teuthology.orchestra.run.smithi188.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 1.0 MB/s | 228 kB 00:00 2024-01-16T14:54:37.646 INFO:teuthology.orchestra.run.smithi141.stdout:(14/151): libcephsqlite-18.0.0-7629.g73a4532f.e 2.8 MB/s | 163 kB 00:00 2024-01-16T14:54:37.656 INFO:teuthology.orchestra.run.smithi188.stdout:(64/151): python3-jsonpatch-1.21-16.el9.noarch. 301 kB/s | 24 kB 00:00 2024-01-16T14:54:37.684 INFO:teuthology.orchestra.run.smithi188.stdout:(65/151): python3-jsonpointer-2.0-4.el9.noarch. 352 kB/s | 18 kB 00:00 2024-01-16T14:54:37.704 INFO:teuthology.orchestra.run.smithi141.stdout:(15/151): librados-devel-18.0.0-7629.g73a4532f. 2.2 MB/s | 127 kB 00:00 2024-01-16T14:54:37.726 INFO:teuthology.orchestra.run.smithi188.stdout:(66/151): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-01-16T14:54:37.743 INFO:teuthology.orchestra.run.smithi188.stdout:(67/151): python3-markupsafe-1.1.1-12.el9.x86_6 544 kB/s | 32 kB 00:00 2024-01-16T14:54:37.779 INFO:teuthology.orchestra.run.smithi141.stdout:(16/151): libradosstriper1-18.0.0-7629.g73a4532 6.2 MB/s | 476 kB 00:00 2024-01-16T14:54:38.069 INFO:teuthology.orchestra.run.smithi188.stdout:(68/151): ceph-common-18.0.0-7629.g73a4532f.el9 2.4 MB/s | 19 MB 00:07 2024-01-16T14:54:38.188 INFO:teuthology.orchestra.run.smithi141.stdout:(17/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 8.7 MB/s | 18 MB 00:02 2024-01-16T14:54:38.219 INFO:teuthology.orchestra.run.smithi188.stdout:(69/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.7 MB/s | 4.8 MB 00:00 2024-01-16T14:54:38.252 INFO:teuthology.orchestra.run.smithi188.stdout:(70/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 428 kB/s | 217 kB 00:00 2024-01-16T14:54:38.277 INFO:teuthology.orchestra.run.smithi188.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 894 kB/s | 174 kB 00:00 2024-01-16T14:54:38.302 INFO:teuthology.orchestra.run.smithi188.stdout:(72/151): python3-prettytable-0.7.2-27.el9.noar 515 kB/s | 41 kB 00:00 2024-01-16T14:54:38.322 INFO:teuthology.orchestra.run.smithi141.stdout:(18/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 8.8 MB/s | 4.8 MB 00:00 2024-01-16T14:54:38.336 INFO:teuthology.orchestra.run.smithi188.stdout:(73/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-01-16T14:54:38.347 INFO:teuthology.orchestra.run.smithi141.stdout:(19/151): python3-ceph-argparse-18.0.0-7629.g73 286 kB/s | 45 kB 00:00 2024-01-16T14:54:38.370 INFO:teuthology.orchestra.run.smithi188.stdout:(74/151): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-01-16T14:54:38.380 INFO:teuthology.orchestra.run.smithi141.stdout:(20/151): python3-ceph-common-18.0.0-7629.g73a4 2.2 MB/s | 129 kB 00:00 2024-01-16T14:54:38.395 INFO:teuthology.orchestra.run.smithi188.stdout:(75/151): python3-pytz-2021.1-5.el9.noarch.rpm 516 kB/s | 47 kB 00:00 2024-01-16T14:54:38.420 INFO:teuthology.orchestra.run.smithi188.stdout:(76/151): python3-requests-oauthlib-1.3.0-12.el 528 kB/s | 43 kB 00:00 2024-01-16T14:54:38.439 INFO:teuthology.orchestra.run.smithi141.stdout:(21/151): python3-rados-18.0.0-7629.g73a4532f.e 5.4 MB/s | 321 kB 00:00 2024-01-16T14:54:38.462 INFO:teuthology.orchestra.run.smithi188.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 592 kB/s | 39 kB 00:00 2024-01-16T14:54:38.464 INFO:teuthology.orchestra.run.smithi141.stdout:(22/151): python3-cephfs-18.0.0-7629.g73a4532f. 1.4 MB/s | 161 kB 00:00 2024-01-16T14:54:38.515 INFO:teuthology.orchestra.run.smithi141.stdout:(23/151): python3-rbd-18.0.0-7629.g73a4532f.el9 3.9 MB/s | 298 kB 00:00 2024-01-16T14:54:38.565 INFO:teuthology.orchestra.run.smithi141.stdout:(24/151): python3-rgw-18.0.0-7629.g73a4532f.el9 997 kB/s | 99 kB 00:00 2024-01-16T14:54:38.590 INFO:teuthology.orchestra.run.smithi141.stdout:(25/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 1.1 MB/s | 85 kB 00:00 2024-01-16T14:54:38.659 INFO:teuthology.orchestra.run.smithi141.stdout:(26/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 2.4 MB/s | 171 kB 00:00 2024-01-16T14:54:38.670 INFO:teuthology.orchestra.run.smithi188.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-01-16T14:54:38.718 INFO:teuthology.orchestra.run.smithi141.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 490 kB/s | 28 kB 00:00 2024-01-16T14:54:38.762 INFO:teuthology.orchestra.run.smithi188.stdout:(79/151): uuid-1.6.2-55.el9.x86_64.rpm 197 kB/s | 56 kB 00:00 2024-01-16T14:54:38.768 INFO:teuthology.orchestra.run.smithi141.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 2.6 MB/s | 133 kB 00:00 2024-01-16T14:54:38.994 INFO:teuthology.orchestra.run.smithi141.stdout:(29/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 7.8 MB/s | 1.8 MB 00:00 2024-01-16T14:54:39.012 INFO:teuthology.orchestra.run.smithi188.stdout:(80/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-01-16T14:54:39.029 INFO:teuthology.orchestra.run.smithi188.stdout:(81/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 250 kB/s | 87 kB 00:00 2024-01-16T14:54:39.055 INFO:teuthology.orchestra.run.smithi188.stdout:(82/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 217 kB/s | 63 kB 00:00 2024-01-16T14:54:39.077 INFO:teuthology.orchestra.run.smithi141.stdout:(30/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 5.9 MB/s | 3.0 MB 00:00 2024-01-16T14:54:39.080 INFO:teuthology.orchestra.run.smithi188.stdout:(83/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 903 kB/s | 44 kB 00:00 2024-01-16T14:54:39.105 INFO:teuthology.orchestra.run.smithi188.stdout:(84/151): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-01-16T14:54:39.130 INFO:teuthology.orchestra.run.smithi188.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 2.5 MB/s | 186 kB 00:00 2024-01-16T14:54:39.169 INFO:teuthology.orchestra.run.smithi141.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g73 2.7 MB/s | 246 kB 00:00 2024-01-16T14:54:39.181 INFO:teuthology.orchestra.run.smithi188.stdout:(86/151): libgfortran-11.4.1-3.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-01-16T14:54:39.206 INFO:teuthology.orchestra.run.smithi188.stdout:(87/151): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 194 kB 00:00 2024-01-16T14:54:39.231 INFO:teuthology.orchestra.run.smithi188.stdout:(88/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.2 MB/s | 217 kB 00:00 2024-01-16T14:54:39.265 INFO:teuthology.orchestra.run.smithi188.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.9 MB/s | 241 kB 00:00 2024-01-16T14:54:39.286 INFO:teuthology.orchestra.run.smithi141.stdout:(32/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 432 kB/s | 50 kB 00:00 2024-01-16T14:54:39.290 INFO:teuthology.orchestra.run.smithi188.stdout:(90/151): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-01-16T14:54:39.340 INFO:teuthology.orchestra.run.smithi188.stdout:(91/151): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-01-16T14:54:39.365 INFO:teuthology.orchestra.run.smithi188.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-01-16T14:54:39.369 INFO:teuthology.orchestra.run.smithi141.stdout:(33/151): ceph-prometheus-alerts-18.0.0-7629.g7 184 kB/s | 15 kB 00:00 2024-01-16T14:54:39.382 INFO:teuthology.orchestra.run.smithi188.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 1.2 MB/s | 113 kB 00:00 2024-01-16T14:54:39.416 INFO:teuthology.orchestra.run.smithi188.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 3.8 MB/s | 187 kB 00:00 2024-01-16T14:54:39.528 INFO:teuthology.orchestra.run.smithi141.stdout:(34/151): ceph-mgr-diskprediction-local-18.0.0- 14 MB/s | 7.4 MB 00:00 2024-01-16T14:54:39.561 INFO:teuthology.orchestra.run.smithi141.stdout:(35/151): ceph-volume-18.0.0-7629.g73a4532f.el9 1.3 MB/s | 262 kB 00:00 2024-01-16T14:54:39.583 INFO:teuthology.orchestra.run.smithi188.stdout:(95/151): lua-devel-5.4.4-4.el9.x86_64.rpm 113 kB/s | 22 kB 00:00 2024-01-16T14:54:39.620 INFO:teuthology.orchestra.run.smithi141.stdout:(36/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 8.4 MB/s | 778 kB 00:00 2024-01-16T14:54:39.654 INFO:teuthology.orchestra.run.smithi141.stdout:(37/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 199 kB/s | 18 kB 00:00 2024-01-16T14:54:39.741 INFO:teuthology.orchestra.run.smithi188.stdout:(96/151): python3-devel-3.9.17-2.el9.x86_64.rpm 657 kB/s | 245 kB 00:00 2024-01-16T14:54:39.921 INFO:teuthology.orchestra.run.smithi141.stdout:(38/151): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-01-16T14:54:39.950 INFO:teuthology.orchestra.run.smithi188.stdout:(97/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 580 kB/s | 309 kB 00:00 2024-01-16T14:54:39.971 INFO:teuthology.orchestra.run.smithi141.stdout:(39/151): boost-program-options-1.75.0-8.el9.x8 2.2 MB/s | 106 kB 00:00 2024-01-16T14:54:40.017 INFO:teuthology.orchestra.run.smithi188.stdout:(98/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-01-16T14:54:40.021 INFO:teuthology.orchestra.run.smithi141.stdout:(40/151): bison-3.7.4-5.el9.x86_64.rpm 3.0 MB/s | 920 kB 00:00 2024-01-16T14:54:40.042 INFO:teuthology.orchestra.run.smithi188.stdout:(99/151): liboath-2.6.7-2.el9.x86_64.rpm 531 kB/s | 49 kB 00:00 2024-01-16T14:54:40.047 INFO:teuthology.orchestra.run.smithi141.stdout:(41/151): elfutils-libelf-devel-0.190-2.el9.x86 276 kB/s | 20 kB 00:00 2024-01-16T14:54:40.172 INFO:teuthology.orchestra.run.smithi141.stdout:(42/151): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 307 kB 00:00 2024-01-16T14:54:40.176 INFO:teuthology.orchestra.run.smithi188.stdout:(100/151): libunwind-1.6.2-1.el9.x86_64.rpm 428 kB/s | 67 kB 00:00 2024-01-16T14:54:40.197 INFO:teuthology.orchestra.run.smithi141.stdout:(43/151): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2024-01-16T14:54:40.226 INFO:teuthology.orchestra.run.smithi188.stdout:(101/151): luarocks-3.9.2-1.el9.noarch.rpm 826 kB/s | 151 kB 00:00 2024-01-16T14:54:40.339 INFO:teuthology.orchestra.run.smithi141.stdout:(44/151): flexiblas-openblas-openmp-3.0.4-8.el9 121 kB/s | 17 kB 00:00 2024-01-16T14:54:40.506 INFO:teuthology.orchestra.run.smithi141.stdout:(45/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.0 MB/s | 3.0 MB 00:00 2024-01-16T14:54:40.551 INFO:teuthology.orchestra.run.smithi188.stdout:(102/151): python3-asyncssh-2.13.2-3.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-01-16T14:54:40.556 INFO:teuthology.orchestra.run.smithi141.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 47 kB/s | 10 kB 00:00 2024-01-16T14:54:40.582 INFO:teuthology.orchestra.run.smithi141.stdout:(47/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-01-16T14:54:40.624 INFO:teuthology.orchestra.run.smithi141.stdout:(48/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 681 kB/s | 44 kB 00:00 2024-01-16T14:54:40.635 INFO:teuthology.orchestra.run.smithi188.stdout:(103/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 525 kB/s | 43 kB 00:00 2024-01-16T14:54:40.744 INFO:teuthology.orchestra.run.smithi188.stdout:(104/151): python3-cachetools-4.2.4-1.el9.noarc 299 kB/s | 32 kB 00:00 2024-01-16T14:54:40.774 INFO:teuthology.orchestra.run.smithi141.stdout:(49/151): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 661 kB 00:00 2024-01-16T14:54:40.808 INFO:teuthology.orchestra.run.smithi141.stdout:(50/151): libtool-2.4.6-45.el9.x86_64.rpm 3.1 MB/s | 578 kB 00:00 2024-01-16T14:54:40.811 INFO:teuthology.orchestra.run.smithi188.stdout:(105/151): python3-certifi-2021.10.8-2.el9.noar 227 kB/s | 15 kB 00:00 2024-01-16T14:54:40.833 INFO:teuthology.orchestra.run.smithi141.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 453 kB/s | 24 kB 00:00 2024-01-16T14:54:40.861 INFO:teuthology.orchestra.run.smithi188.stdout:(106/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-01-16T14:54:40.867 INFO:teuthology.orchestra.run.smithi141.stdout:(52/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 798 kB/s | 46 kB 00:00 2024-01-16T14:54:40.909 INFO:teuthology.orchestra.run.smithi141.stdout:(53/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-01-16T14:54:40.928 INFO:teuthology.orchestra.run.smithi188.stdout:(107/151): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-01-16T14:54:40.942 INFO:teuthology.orchestra.run.smithi141.stdout:(54/151): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-01-16T14:54:40.968 INFO:teuthology.orchestra.run.smithi141.stdout:(55/151): openblas-0.3.21-2.el9.x86_64.rpm 604 kB/s | 34 kB 00:00 2024-01-16T14:54:41.112 INFO:teuthology.orchestra.run.smithi188.stdout:(108/151): python3-google-auth-2.26.0-1.el9.noa 1.1 MB/s | 202 kB 00:00 2024-01-16T14:54:41.179 INFO:teuthology.orchestra.run.smithi188.stdout:(109/151): python3-cherrypy-18.6.1-2.el9.noarch 1.1 MB/s | 358 kB 00:00 2024-01-16T14:54:41.195 INFO:teuthology.orchestra.run.smithi188.stdout:(110/151): python3-jaraco-8.2.1-3.el9.noarch.rp 129 kB/s | 11 kB 00:00 2024-01-16T14:54:41.296 INFO:teuthology.orchestra.run.smithi188.stdout:(111/151): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-01-16T14:54:41.312 INFO:teuthology.orchestra.run.smithi188.stdout:(112/151): python3-jaraco-collections-3.0.0-8.e 199 kB/s | 23 kB 00:00 2024-01-16T14:54:41.388 INFO:teuthology.orchestra.run.smithi188.stdout:(113/151): python3-jaraco-functools-3.5.0-2.el9 213 kB/s | 19 kB 00:00 2024-01-16T14:54:41.405 INFO:teuthology.orchestra.run.smithi188.stdout:(114/151): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-01-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi188.stdout:(115/151): python3-jwt+crypto-2.4.0-1.el9.noarc 91 kB/s | 9.0 kB 00:00 2024-01-16T14:54:41.505 INFO:teuthology.orchestra.run.smithi188.stdout:(116/151): python3-jwt-2.4.0-1.el9.noarch.rpm 411 kB/s | 41 kB 00:00 2024-01-16T14:54:41.510 INFO:teuthology.orchestra.run.smithi141.stdout:(56/151): openblas-openmp-0.3.21-2.el9.x86_64.r 8.4 MB/s | 4.7 MB 00:00 2024-01-16T14:54:41.581 INFO:teuthology.orchestra.run.smithi188.stdout:(117/151): python3-logutils-0.3.5-21.el9.noarch 620 kB/s | 46 kB 00:00 2024-01-16T14:54:41.593 INFO:teuthology.orchestra.run.smithi141.stdout:(57/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-01-16T14:54:41.694 INFO:teuthology.orchestra.run.smithi141.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-01-16T14:54:41.736 INFO:teuthology.orchestra.run.smithi141.stdout:(59/151): postgresql-contrib-13.11-1.el9.x86_64 5.9 MB/s | 811 kB 00:00 2024-01-16T14:54:41.756 INFO:teuthology.orchestra.run.smithi188.stdout:(118/151): libarrow-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-01-16T14:54:41.761 INFO:teuthology.orchestra.run.smithi141.stdout:(60/151): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-01-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi188.stdout:(119/151): python3-more-itertools-8.12.0-2.el9. 395 kB/s | 79 kB 00:00 2024-01-16T14:54:41.856 INFO:teuthology.orchestra.run.smithi188.stdout:(120/151): python3-natsort-7.1.1-5.el9.noarch.r 581 kB/s | 58 kB 00:00 2024-01-16T14:54:41.940 INFO:teuthology.orchestra.run.smithi188.stdout:(121/151): python3-portend-3.1.0-2.el9.noarch.r 199 kB/s | 16 kB 00:00 2024-01-16T14:54:42.015 INFO:teuthology.orchestra.run.smithi188.stdout:(122/151): python3-kubernetes-26.1.0-2.el9.noar 1.9 MB/s | 1.0 MB 00:00 2024-01-16T14:54:42.049 INFO:teuthology.orchestra.run.smithi188.stdout:(123/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 834 kB/s | 90 kB 00:00 2024-01-16T14:54:42.107 INFO:teuthology.orchestra.run.smithi188.stdout:(124/151): python3-pecan-1.4.2-3.el9.noarch.rpm 837 kB/s | 272 kB 00:00 2024-01-16T14:54:42.132 INFO:teuthology.orchestra.run.smithi188.stdout:(125/151): python3-repoze-lru-0.7-16.el9.noarch 263 kB/s | 31 kB 00:00 2024-01-16T14:54:42.166 INFO:teuthology.orchestra.run.smithi188.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-01-16T14:54:42.191 INFO:teuthology.orchestra.run.smithi188.stdout:(127/151): python3-rsa-4.9-2.el9.noarch.rpm 713 kB/s | 59 kB 00:00 2024-01-16T14:54:42.225 INFO:teuthology.orchestra.run.smithi188.stdout:(128/151): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-01-16T14:54:42.251 INFO:teuthology.orchestra.run.smithi188.stdout:(129/151): python3-typing-extensions-4.4.0-2.el 609 kB/s | 51 kB 00:00 2024-01-16T14:54:42.277 INFO:teuthology.orchestra.run.smithi141.stdout:(61/151): postgresql-server-13.11-1.el9.x86_64. 11 MB/s | 5.7 MB 00:00 2024-01-16T14:54:42.309 INFO:teuthology.orchestra.run.smithi188.stdout:(130/151): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-01-16T14:54:42.376 INFO:teuthology.orchestra.run.smithi188.stdout:(131/151): python3-werkzeug-2.0.3-3.el9.1.noarc 3.4 MB/s | 427 kB 00:00 2024-01-16T14:54:42.401 INFO:teuthology.orchestra.run.smithi188.stdout:(132/151): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-01-16T14:54:42.403 INFO:teuthology.orchestra.run.smithi141.stdout:(62/151): python3-babel-2.9.1-2.el9.noarch.rpm 9.0 MB/s | 5.8 MB 00:00 2024-01-16T14:54:42.428 INFO:teuthology.orchestra.run.smithi141.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 1.7 MB/s | 228 kB 00:00 2024-01-16T14:54:42.443 INFO:teuthology.orchestra.run.smithi188.stdout:(133/151): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-01-16T14:54:42.468 INFO:teuthology.orchestra.run.smithi188.stdout:(134/151): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-01-16T14:54:42.487 INFO:teuthology.orchestra.run.smithi141.stdout:(64/151): python3-jsonpatch-1.21-16.el9.noarch. 298 kB/s | 24 kB 00:00 2024-01-16T14:54:42.512 INFO:teuthology.orchestra.run.smithi141.stdout:(65/151): python3-jsonpointer-2.0-4.el9.noarch. 219 kB/s | 18 kB 00:00 2024-01-16T14:54:42.545 INFO:teuthology.orchestra.run.smithi141.stdout:(66/151): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-01-16T14:54:42.571 INFO:teuthology.orchestra.run.smithi141.stdout:(67/151): python3-markupsafe-1.1.1-12.el9.x86_6 562 kB/s | 32 kB 00:00 2024-01-16T14:54:42.819 INFO:teuthology.orchestra.run.smithi188.stdout:(135/151): librados2-18.0.0-7629.g73a4532f.el9. 8.9 MB/s | 3.3 MB 00:00 2024-01-16T14:54:42.860 INFO:teuthology.orchestra.run.smithi188.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-01-16T14:54:42.911 INFO:teuthology.orchestra.run.smithi188.stdout:(137/151): thrift-0.15.0-2.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-01-16T14:54:42.994 INFO:teuthology.orchestra.run.smithi188.stdout:(138/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 5.8 MB/s | 3.0 MB 00:00 2024-01-16T14:54:43.070 INFO:teuthology.orchestra.run.smithi188.stdout:(139/151): NetworkManager-1.45.9-1.el9.x86_64.r 11 MB/s | 2.2 MB 00:00 2024-01-16T14:54:43.120 INFO:teuthology.orchestra.run.smithi188.stdout:(140/151): NetworkManager-libnm-1.45.9-1.el9.x8 9.0 MB/s | 1.8 MB 00:00 2024-01-16T14:54:43.145 INFO:teuthology.orchestra.run.smithi188.stdout:(141/151): NetworkManager-team-1.45.9-1.el9.x86 325 kB/s | 42 kB 00:00 2024-01-16T14:54:43.170 INFO:teuthology.orchestra.run.smithi188.stdout:(142/151): NetworkManager-tui-1.45.9-1.el9.x86_ 2.5 MB/s | 246 kB 00:00 2024-01-16T14:54:43.195 INFO:teuthology.orchestra.run.smithi188.stdout:(143/151): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2024-01-16T14:54:43.221 INFO:teuthology.orchestra.run.smithi188.stdout:(144/151): elfutils-debuginfod-client-0.190-2.e 490 kB/s | 35 kB 00:00 2024-01-16T14:54:43.238 INFO:teuthology.orchestra.run.smithi141.stdout:(68/151): ceph-test-18.0.0-7629.g73a4532f.el9.x 6.4 MB/s | 41 MB 00:06 2024-01-16T14:54:43.254 INFO:teuthology.orchestra.run.smithi188.stdout:(145/151): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 191 kB 00:00 2024-01-16T14:54:43.288 INFO:teuthology.orchestra.run.smithi188.stdout:(146/151): elfutils-libs-0.190-2.el9.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-01-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi188.stdout:(147/151): libacl-2.3.1-4.el9.x86_64.rpm 246 kB/s | 22 kB 00:00 2024-01-16T14:54:43.346 INFO:teuthology.orchestra.run.smithi141.stdout:(69/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.0 MB/s | 4.8 MB 00:00 2024-01-16T14:54:43.363 INFO:teuthology.orchestra.run.smithi188.stdout:(148/151): openssl-3.0.7-25.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-01-16T14:54:43.372 INFO:teuthology.orchestra.run.smithi141.stdout:(70/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 272 kB/s | 217 kB 00:00 2024-01-16T14:54:43.405 INFO:teuthology.orchestra.run.smithi188.stdout:(149/151): xfsdump-3.1.12-4.el9.x86_64.rpm 3.9 MB/s | 321 kB 00:00 2024-01-16T14:54:43.405 INFO:teuthology.orchestra.run.smithi141.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-01-16T14:54:43.439 INFO:teuthology.orchestra.run.smithi141.stdout:(72/151): python3-prettytable-0.7.2-27.el9.noar 640 kB/s | 41 kB 00:00 2024-01-16T14:54:43.464 INFO:teuthology.orchestra.run.smithi141.stdout:(73/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-01-16T14:54:43.488 INFO:teuthology.orchestra.run.smithi188.stdout:(150/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 13 MB/s | 2.1 MB 00:00 2024-01-16T14:54:43.497 INFO:teuthology.orchestra.run.smithi141.stdout:(74/151): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-01-16T14:54:43.523 INFO:teuthology.orchestra.run.smithi141.stdout:(75/151): python3-requests-oauthlib-1.3.0-12.el 769 kB/s | 43 kB 00:00 2024-01-16T14:54:43.530 INFO:teuthology.orchestra.run.smithi188.stdout:(151/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.5 MB/s | 1.1 MB 00:00 2024-01-16T14:54:43.546 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:54:43.547 INFO:teuthology.orchestra.run.smithi188.stdout:Total 15 MB/s | 204 MB 00:13 2024-01-16T14:54:43.557 INFO:teuthology.orchestra.run.smithi141.stdout:(76/151): python3-pytz-2021.1-5.el9.noarch.rpm 529 kB/s | 47 kB 00:00 2024-01-16T14:54:43.635 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-01-16T14:54:43.807 INFO:teuthology.orchestra.run.smithi141.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 138 kB/s | 39 kB 00:00 2024-01-16T14:54:43.832 INFO:teuthology.orchestra.run.smithi141.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-01-16T14:54:43.972 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-01-16T14:54:43.973 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-01-16T14:54:44.157 INFO:teuthology.orchestra.run.smithi141.stdout:(79/151): uuid-1.6.2-55.el9.x86_64.rpm 180 kB/s | 56 kB 00:00 2024-01-16T14:54:44.182 INFO:teuthology.orchestra.run.smithi141.stdout:(80/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 257 kB/s | 87 kB 00:00 2024-01-16T14:54:44.483 INFO:teuthology.orchestra.run.smithi141.stdout:(81/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-01-16T14:54:44.508 INFO:teuthology.orchestra.run.smithi141.stdout:(82/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 196 kB/s | 63 kB 00:00 2024-01-16T14:54:44.533 INFO:teuthology.orchestra.run.smithi141.stdout:(83/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 128 kB/s | 44 kB 00:00 2024-01-16T14:54:44.558 INFO:teuthology.orchestra.run.smithi141.stdout:(84/151): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-01-16T14:54:44.600 INFO:teuthology.orchestra.run.smithi141.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 2.8 MB/s | 186 kB 00:00 2024-01-16T14:54:44.651 INFO:teuthology.orchestra.run.smithi141.stdout:(86/151): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 802 kB 00:00 2024-01-16T14:54:44.676 INFO:teuthology.orchestra.run.smithi141.stdout:(87/151): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 194 kB 00:00 2024-01-16T14:54:44.709 INFO:teuthology.orchestra.run.smithi141.stdout:(88/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 217 kB 00:00 2024-01-16T14:54:44.743 INFO:teuthology.orchestra.run.smithi141.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 241 kB 00:00 2024-01-16T14:54:44.793 INFO:teuthology.orchestra.run.smithi141.stdout:(90/151): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.1 MB 00:00 2024-01-16T14:54:44.810 INFO:teuthology.orchestra.run.smithi141.stdout:(91/151): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-01-16T14:54:44.835 INFO:teuthology.orchestra.run.smithi141.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-01-16T14:54:44.869 INFO:teuthology.orchestra.run.smithi141.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 1.9 MB/s | 113 kB 00:00 2024-01-16T14:54:44.894 INFO:teuthology.orchestra.run.smithi141.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-01-16T14:54:45.111 INFO:teuthology.orchestra.run.smithi141.stdout:(95/151): python3-devel-3.9.17-2.el9.x86_64.rpm 905 kB/s | 245 kB 00:00 2024-01-16T14:54:45.662 INFO:teuthology.orchestra.run.smithi141.stdout:(96/151): lua-devel-5.4.4-4.el9.x86_64.rpm 29 kB/s | 22 kB 00:00 2024-01-16T14:54:45.937 INFO:teuthology.orchestra.run.smithi141.stdout:(97/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 297 kB/s | 309 kB 00:01 2024-01-16T14:54:46.054 INFO:teuthology.orchestra.run.smithi141.stdout:(98/151): liboath-2.6.7-2.el9.x86_64.rpm 419 kB/s | 49 kB 00:00 2024-01-16T14:54:46.137 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-01-16T14:54:46.138 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-01-16T14:54:46.162 INFO:teuthology.orchestra.run.smithi141.stdout:(99/151): libunwind-1.6.2-1.el9.x86_64.rpm 625 kB/s | 67 kB 00:00 2024-01-16T14:54:46.187 INFO:teuthology.orchestra.run.smithi141.stdout:(100/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 49 kB/s | 26 kB 00:00 2024-01-16T14:54:46.463 INFO:teuthology.orchestra.run.smithi141.stdout:(101/151): luarocks-3.9.2-1.el9.noarch.rpm 504 kB/s | 151 kB 00:00 2024-01-16T14:54:47.155 INFO:teuthology.orchestra.run.smithi141.stdout:(102/151): python3-asyncssh-2.13.2-3.el9.noarch 793 kB/s | 548 kB 00:00 2024-01-16T14:54:47.271 INFO:teuthology.orchestra.run.smithi141.stdout:(103/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 375 kB/s | 43 kB 00:00 2024-01-16T14:54:47.397 INFO:teuthology.orchestra.run.smithi141.stdout:(104/151): python3-cachetools-4.2.4-1.el9.noarc 259 kB/s | 32 kB 00:00 2024-01-16T14:54:47.480 INFO:teuthology.orchestra.run.smithi141.stdout:(105/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 649 kB/s | 838 kB 00:01 2024-01-16T14:54:47.505 INFO:teuthology.orchestra.run.smithi141.stdout:(106/151): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-01-16T14:54:47.781 INFO:teuthology.orchestra.run.smithi141.stdout:(107/151): python3-cheroot-8.6.0-4.el9.noarch.r 573 kB/s | 172 kB 00:00 2024-01-16T14:54:48.056 INFO:teuthology.orchestra.run.smithi141.stdout:(108/151): python3-cherrypy-18.6.1-2.el9.noarch 652 kB/s | 358 kB 00:00 2024-01-16T14:54:48.081 INFO:teuthology.orchestra.run.smithi141.stdout:(109/151): python3-google-auth-2.26.0-1.el9.noa 674 kB/s | 202 kB 00:00 2024-01-16T14:54:48.173 INFO:teuthology.orchestra.run.smithi141.stdout:(110/151): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-01-16T14:54:48.198 INFO:teuthology.orchestra.run.smithi141.stdout:(111/151): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-01-16T14:54:48.307 INFO:teuthology.orchestra.run.smithi141.stdout:(112/151): python3-jaraco-collections-3.0.0-8.e 174 kB/s | 23 kB 00:00 2024-01-16T14:54:48.332 INFO:teuthology.orchestra.run.smithi141.stdout:(113/151): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-01-16T14:54:48.433 INFO:teuthology.orchestra.run.smithi141.stdout:(114/151): python3-jaraco-text-3.2.0-6.el9.noar 157 kB/s | 20 kB 00:00 2024-01-16T14:54:48.458 INFO:teuthology.orchestra.run.smithi141.stdout:(115/151): python3-jwt+crypto-2.4.0-1.el9.noarc 72 kB/s | 9.0 kB 00:00 2024-01-16T14:54:48.550 INFO:teuthology.orchestra.run.smithi141.stdout:(116/151): python3-jwt-2.4.0-1.el9.noarch.rpm 352 kB/s | 41 kB 00:00 2024-01-16T14:54:48.667 INFO:teuthology.orchestra.run.smithi141.stdout:(117/151): python3-logutils-0.3.5-21.el9.noarch 398 kB/s | 46 kB 00:00 2024-01-16T14:54:48.784 INFO:teuthology.orchestra.run.smithi141.stdout:(118/151): python3-more-itertools-8.12.0-2.el9. 680 kB/s | 79 kB 00:00 2024-01-16T14:54:48.893 INFO:teuthology.orchestra.run.smithi141.stdout:(119/151): python3-natsort-7.1.1-5.el9.noarch.r 535 kB/s | 58 kB 00:00 2024-01-16T14:54:49.401 INFO:teuthology.orchestra.run.smithi141.stdout:(120/151): python3-pecan-1.4.2-3.el9.noarch.rpm 536 kB/s | 272 kB 00:00 2024-01-16T14:54:49.518 INFO:teuthology.orchestra.run.smithi141.stdout:(121/151): python3-portend-3.1.0-2.el9.noarch.r 142 kB/s | 16 kB 00:00 2024-01-16T14:54:49.735 INFO:teuthology.orchestra.run.smithi141.stdout:(122/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 416 kB/s | 90 kB 00:00 2024-01-16T14:54:49.877 INFO:teuthology.orchestra.run.smithi141.stdout:(123/151): python3-repoze-lru-0.7-16.el9.noarch 218 kB/s | 31 kB 00:00 2024-01-16T14:54:50.027 INFO:teuthology.orchestra.run.smithi141.stdout:(124/151): python3-kubernetes-26.1.0-2.el9.noar 667 kB/s | 1.0 MB 00:01 2024-01-16T14:54:50.144 INFO:teuthology.orchestra.run.smithi141.stdout:(125/151): python3-rsa-4.9-2.el9.noarch.rpm 509 kB/s | 59 kB 00:00 2024-01-16T14:54:50.228 INFO:teuthology.orchestra.run.smithi141.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 539 kB/s | 188 kB 00:00 2024-01-16T14:54:50.261 INFO:teuthology.orchestra.run.smithi141.stdout:(127/151): python3-tempora-5.0.0-2.el9.noarch.r 308 kB/s | 36 kB 00:00 2024-01-16T14:54:50.412 INFO:teuthology.orchestra.run.smithi141.stdout:(128/151): python3-typing-extensions-4.4.0-2.el 278 kB/s | 51 kB 00:00 2024-01-16T14:54:50.604 INFO:teuthology.orchestra.run.smithi141.stdout:(129/151): python3-websocket-client-1.2.3-2.el9 470 kB/s | 90 kB 00:00 2024-01-16T14:54:50.671 INFO:teuthology.orchestra.run.smithi141.stdout:(130/151): python3-webob-1.8.7-6.el9.noarch.rpm 562 kB/s | 230 kB 00:00 2024-01-16T14:54:50.804 INFO:teuthology.orchestra.run.smithi141.stdout:(131/151): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-01-16T14:54:51.088 INFO:teuthology.orchestra.run.smithi141.stdout:(132/151): re2-20211101-3.el9.x86_64.rpm 678 kB/s | 192 kB 00:00 2024-01-16T14:54:51.171 INFO:teuthology.orchestra.run.smithi141.stdout:(133/151): python3-werkzeug-2.0.3-3.el9.1.noarc 753 kB/s | 427 kB 00:00 2024-01-16T14:54:51.364 INFO:teuthology.orchestra.run.smithi141.stdout:(134/151): libarrow-9.0.0-10.el9.x86_64.rpm 724 kB/s | 4.4 MB 00:06 2024-01-16T14:54:51.705 INFO:teuthology.orchestra.run.smithi141.stdout:(135/151): librados2-18.0.0-7629.g73a4532f.el9. 6.2 MB/s | 3.3 MB 00:00 2024-01-16T14:54:51.747 INFO:teuthology.orchestra.run.smithi141.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-01-16T14:54:51.914 INFO:teuthology.orchestra.run.smithi141.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 14 MB/s | 2.2 MB 00:00 2024-01-16T14:54:51.989 INFO:teuthology.orchestra.run.smithi141.stdout:(138/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 4.8 MB/s | 3.0 MB 00:00 2024-01-16T14:54:52.065 INFO:teuthology.orchestra.run.smithi141.stdout:(139/151): NetworkManager-libnm-1.45.9-1.el9.x8 12 MB/s | 1.8 MB 00:00 2024-01-16T14:54:52.090 INFO:teuthology.orchestra.run.smithi141.stdout:(140/151): NetworkManager-team-1.45.9-1.el9.x86 492 kB/s | 42 kB 00:00 2024-01-16T14:54:52.140 INFO:teuthology.orchestra.run.smithi141.stdout:(141/151): NetworkManager-tui-1.45.9-1.el9.x86_ 3.3 MB/s | 246 kB 00:00 2024-01-16T14:54:52.166 INFO:teuthology.orchestra.run.smithi141.stdout:(142/151): acl-2.3.1-4.el9.x86_64.rpm 949 kB/s | 69 kB 00:00 2024-01-16T14:54:52.199 INFO:teuthology.orchestra.run.smithi141.stdout:(143/151): elfutils-debuginfod-client-0.190-2.e 615 kB/s | 35 kB 00:00 2024-01-16T14:54:52.233 INFO:teuthology.orchestra.run.smithi141.stdout:(144/151): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 191 kB 00:00 2024-01-16T14:54:52.267 INFO:teuthology.orchestra.run.smithi141.stdout:(145/151): elfutils-libs-0.190-2.el9.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-01-16T14:54:52.300 INFO:teuthology.orchestra.run.smithi141.stdout:(146/151): libacl-2.3.1-4.el9.x86_64.rpm 341 kB/s | 22 kB 00:00 2024-01-16T14:54:52.341 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-01-16T14:54:52.400 INFO:teuthology.orchestra.run.smithi141.stdout:(147/151): openssl-3.0.7-25.el9.x86_64.rpm 8.7 MB/s | 1.2 MB 00:00 2024-01-16T14:54:52.495 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:54:52.528 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:54:52.534 INFO:teuthology.orchestra.run.smithi141.stdout:(148/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 9.3 MB/s | 2.1 MB 00:00 2024-01-16T14:54:52.568 INFO:teuthology.orchestra.run.smithi141.stdout:(149/151): xfsdump-3.1.12-4.el9.x86_64.rpm 2.0 MB/s | 321 kB 00:00 2024-01-16T14:54:52.613 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:54:52.668 INFO:teuthology.orchestra.run.smithi141.stdout:(150/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.1 MB/s | 1.1 MB 00:00 2024-01-16T14:54:52.675 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:54:52.712 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:54:52.817 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:54:53.201 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:54:53.368 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:54:53.525 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:54:53.526 INFO:teuthology.orchestra.run.smithi141.stdout:(151/151): thrift-0.15.0-2.el9.x86_64.rpm 664 kB/s | 1.6 MB 00:02 2024-01-16T14:54:53.542 INFO:teuthology.orchestra.run.smithi141.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:54:53.543 INFO:teuthology.orchestra.run.smithi141.stdout:Total 10 MB/s | 204 MB 00:20 2024-01-16T14:54:53.553 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:54:53.634 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction check 2024-01-16T14:54:53.677 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:54:53.713 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:54:53.866 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:54:53.964 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:54:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction check succeeded. 2024-01-16T14:54:53.981 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction test 2024-01-16T14:54:54.240 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:54:54.293 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:54:54.318 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:54:54.465 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:54:54.502 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:54:54.586 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:54:54.922 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:54:55.106 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:54:55.179 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:54:55.370 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:55.426 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:55.833 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:54:55.865 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:54:55.950 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:54:56.011 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:54:56.043 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:54:56.082 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:54:56.127 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction test succeeded. 2024-01-16T14:54:56.128 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction 2024-01-16T14:54:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:54:56.238 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:54:56.278 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:54:56.322 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:54:56.356 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:54:56.462 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:54:56.501 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:54:56.551 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:54:56.583 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:54:56.683 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:54:56.779 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:54:56.950 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:54:57.013 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:54:57.683 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:54:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:54:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:54:57.868 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:54:57.907 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:54:57.966 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:54:58.155 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:54:58.203 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:54:58.332 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:54:58.536 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:54:58.626 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:54:58.726 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:54:58.767 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:54:58.807 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:54:58.875 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:54:58.930 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:54:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:54:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:54:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:54:59.411 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:54:59.448 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:54:59.484 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:54:59.521 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:54:59.667 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:54:59.717 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:54:59.783 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:54:59.856 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:55:00.116 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:55:00.180 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:55:00.353 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:55:00.417 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:55:00.441 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:55:00.480 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:55:00.526 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:55:00.557 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:55:00.605 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:55:00.674 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:55:00.709 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:55:00.762 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:55:00.832 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:55:01.324 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:55:01.468 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:55:01.511 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:55:01.553 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:55:01.619 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:55:01.656 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:55:01.801 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:55:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:55:01.972 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:55:02.043 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:55:02.540 INFO:teuthology.orchestra.run.smithi141.stdout: Preparing : 1/1 2024-01-16T14:55:02.676 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:55:02.711 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:55:02.793 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:55:02.846 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:55:02.876 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:55:02.976 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:55:03.048 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:55:03.122 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:55:03.160 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:55:03.193 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:55:03.405 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:55:03.425 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:55:03.449 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:55:03.457 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:55:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:55:03.638 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:55:03.675 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:55:03.831 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:55:03.874 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:55:04.025 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:55:04.147 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:55:04.330 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:55:04.451 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:55:04.454 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:55:04.515 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:55:04.540 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:55:04.733 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:55:04.766 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:55:04.832 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:55:05.176 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:55:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:55:05.401 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:55:05.612 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:55:05.675 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:55:05.965 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:55:06.078 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:55:06.078 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:55:06.118 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:55:06.121 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:55:06.152 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:55:06.221 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:55:06.284 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:55:06.325 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:55:06.382 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:55:06.436 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:55:06.518 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:55:06.540 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:55:06.552 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:55:06.586 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:55:06.630 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:55:06.672 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:55:06.672 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:55:06.708 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:55:06.763 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:55:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:55:06.825 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:55:06.900 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:55:06.931 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:55:07.058 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:55:07.161 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:55:07.252 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:55:07.280 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:55:07.344 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:55:07.371 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:55:07.399 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:55:07.411 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:55:08.103 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:55:08.208 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:55:08.272 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:55:08.329 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:55:08.376 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:55:08.446 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:55:08.646 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:55:08.702 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:55:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:55:08.889 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:55:08.953 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:55:09.041 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:55:09.082 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:55:09.121 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:55:09.186 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:55:09.244 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:55:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:55:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:55:09.547 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:55:09.594 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:55:09.732 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:55:09.780 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:55:09.828 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:55:09.862 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:55:09.890 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:55:09.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:55:10.015 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:55:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:55:10.153 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:55:10.233 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:55:10.507 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:55:10.563 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:55:10.568 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:55:10.607 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:55:10.738 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:55:10.765 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:55:10.765 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:55:10.765 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:55:10.766 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:10.796 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:55:10.809 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:55:11.141 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:55:11.197 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:55:11.262 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:55:11.316 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:55:11.401 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:55:11.444 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:55:11.496 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:55:11.551 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:55:11.618 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:55:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:55:11.798 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:55:11.837 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:55:11.882 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:55:11.946 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:55:12.000 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:55:12.145 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:55:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:55:12.330 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:55:12.403 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:55:13.367 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:55:13.428 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:55:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:55:13.498 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:55:13.745 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:55:13.778 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:55:13.877 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:55:14.658 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:55:14.804 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:55:16.324 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:55:16.401 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:55:16.505 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:55:16.549 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:55:16.913 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:55:16.948 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:55:17.059 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:55:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:55:17.181 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:55:17.664 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:55:17.725 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:55:17.826 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:55:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:55:20.334 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:55:20.374 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:55:21.013 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:55:21.045 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:55:21.206 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:55:21.206 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:55:21.206 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:55:21.206 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:21.263 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:55:23.536 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /root 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-01-16T14:55:23.537 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:23.754 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:23.800 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:23.800 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:23.800 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:55:23.800 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:25.533 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:25.610 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:25.610 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:25.610 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:55:25.610 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:25.610 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:25.610 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:25.850 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:25.917 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:25.917 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:25.917 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:55:25.917 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:25.918 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:25.918 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:26.417 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:26.483 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:26.483 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:26.483 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:55:26.483 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:26.483 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:26.484 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:26.603 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:55:27.221 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:27.257 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:27.301 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:55:27.396 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:55:27.428 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:27.743 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:27.779 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:28.905 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:28.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:29.071 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:29.247 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:55:29.273 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:29.364 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:29.364 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:29.364 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:55:29.364 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:29.364 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:29.364 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:29.424 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:29.461 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:29.548 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:55:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:55:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:30.534 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:30.535 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:30.535 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:55:30.535 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:30.535 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:30.535 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:30.585 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:30.679 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:30.679 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:30.679 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:55:30.679 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:30.752 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:30.856 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:30.856 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:30.856 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:55:30.856 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:30.856 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:30.856 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:31.244 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:31.341 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:31.342 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:31.342 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:55:31.342 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:31.342 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:31.342 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /sys 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /proc 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /mnt 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /var/tmp 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /home 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /root 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /tmp 2024-01-16T14:55:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:34.392 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:34.441 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:55:34.441 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:34.441 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:55:34.442 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:35.544 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:55:35.599 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:55:35.655 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:55:35.690 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:55:35.957 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:55:36.075 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:55:36.111 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:36.144 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:55:36.163 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:36.184 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:55:36.221 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:55:36.248 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:55:36.248 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:36.248 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:55:36.248 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:36.248 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:55:36.248 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:36.334 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:36.430 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:55:36.472 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:55:36.541 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:55:36.541 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:36.542 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:55:36.542 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:36.542 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:55:36.542 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:37.022 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:37.096 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:55:37.097 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:37.224 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:55:37.378 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:37.445 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:37.540 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:37.616 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:55:37.831 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:55:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:55:38.001 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:55:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:55:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:38.958 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:55:39.026 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:55:39.069 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:55:39.193 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:55:39.237 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:55:39.271 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:55:39.304 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:39.363 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:55:39.363 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:39.487 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:39.491 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:55:39.517 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:39.538 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:55:39.585 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:55:39.634 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:55:39.661 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:55:39.667 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:39.668 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:39.805 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:39.833 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:55:39.835 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:55:39.870 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:39.872 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:39.872 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:39.940 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:55:39.940 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:39.940 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:55:39.940 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:39.940 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:55:39.940 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:39.996 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:40.034 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:55:40.105 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:55:40.201 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:40.201 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:40.375 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:40.413 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:55:40.461 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:55:40.506 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:55:40.625 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:40.869 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:40.869 INFO:teuthology.orchestra.run.smithi188.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:55:40.869 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:55:41.064 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:41.135 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:55:41.135 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:41.135 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:55:41.135 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:41.135 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:55:41.135 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:41.263 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:55:41.264 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:41.264 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:55:41.264 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:41.332 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:41.408 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:55:41.408 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:41.408 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:55:41.408 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:41.408 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:55:41.409 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:41.775 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:41.850 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:55:41.850 INFO:teuthology.orchestra.run.smithi141.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:55:41.851 INFO:teuthology.orchestra.run.smithi141.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:55:41.851 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:41.851 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:55:41.851 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:45.721 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:55:45.721 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:55:45.721 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:55:45.721 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:55:45.721 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:55:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:55:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:55:45.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:55:45.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:55:45.736 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:55:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:55:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:55:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:55:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:55:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:55:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:55:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:55:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:55:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:55:46.005 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:55:46.059 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:55:46.115 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:55:46.154 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:55:46.419 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:55:46.546 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:55:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:46.659 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:55:46.699 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:55:46.748 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:55:46.910 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:46.980 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:55:47.019 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:55:47.937 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:55:47.937 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:47.937 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-01-16T14:55:47.937 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:47.937 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:47.937 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:47.937 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:55:47.938 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.939 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.940 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:55:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:55:47.942 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:55:47.943 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:55:47.944 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:55:47.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:47.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:55:47.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:55:47.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:55:47.949 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:55:47.950 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:55:47.950 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:55:47.950 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-01-16T14:55:47.982 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:48.025 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:48.106 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:55:48.169 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:55:48.569 DEBUG:teuthology.parallel:result is None 2024-01-16T14:55:49.514 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:55:49.581 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:55:49.628 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:55:49.741 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:55:49.798 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:55:49.834 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:55:49.868 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:49.911 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:55:49.911 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:50.035 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:55:50.085 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:55:50.132 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:55:50.173 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:55:50.206 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:50.207 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:50.327 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:55:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:55:50.404 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:50.405 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:55:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:55:50.885 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:55:50.925 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:55:50.970 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:55:51.088 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:51.344 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:55:51.344 INFO:teuthology.orchestra.run.smithi141.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:55:51.344 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:55.899 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:55:55.900 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:55:55.901 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:55:55.902 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:55:55.902 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:55:55.914 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:55:55.914 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:55:55.914 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:55:55.914 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:55:55.914 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:55:55.914 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:55:55.914 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:55:55.915 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:55:55.916 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:55:55.917 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:55:55.918 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:55:55.919 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:55:55.920 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:55:55.921 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:55:55.922 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:55:55.923 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:55:55.924 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:55:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout:Upgraded: 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:55:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout:Installed: 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:55:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:55:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:55:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:55:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:55:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:55:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:55:58.120 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2024-01-16T14:55:58.746 DEBUG:teuthology.parallel:result is None 2024-01-16T14:55:58.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:55:58.895 DEBUG:teuthology.orchestra.run.smithi141:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:55:58.948 INFO:teuthology.orchestra.run.smithi141.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:58.948 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:58.948 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:55:58.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:55:59.113 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:55:59.181 INFO:teuthology.orchestra.run.smithi188.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:59.181 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:55:59.181 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:55:59.183 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T14:55:59.183 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:55:59.183 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:55:59.259 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:55:59.259 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:55:59.363 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T14:55:59.364 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:55:59.364 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:55:59.438 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:55:59.547 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:55:59.548 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:55:59.644 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:55:59.740 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T14:55:59.741 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:55:59.741 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:55:59.840 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:55:59.978 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:55:59.978 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:56:00.075 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:56:00.213 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T14:56:00.213 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:00.213 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:56:00.289 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:56:00.401 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:00.402 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:56:00.500 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:56:00.592 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T14:56:00.681 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T14:56:00.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 777 /var/log/ceph 2024-01-16T14:56:00.686 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /var/log/ceph 2024-01-16T14:56:00.785 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T14:56:00.785 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:56:00.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:56:00.882 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T14:56:00.882 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:56:00.887 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:56:00.992 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-16T14:56:00.992 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-16T14:56:00.992 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-16T14:56:00.992 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:00.992 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:56:01.037 DEBUG:teuthology.orchestra.run.smithi141:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:01.239 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:01.312 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:01.312 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:56:01.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:01.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:01.736 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T14:56:01.736 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T14:56:01.736 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7518201', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-clone-stats', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4708, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-clone-stats', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi141.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbPa7to2H1Fo2Lztaf7r0dVUX1CbnG8svzBIMu06hrmzdrZqtiSu/xizzWD+IclGIcxfOOW6b5Ye8WtSJF94M0=', 'smithi188.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOnR5DedqAgWvakFkagD0XmrbMCQmAI6uIumY44uRbyRlPArCx1CvZo2GQzXUoiCygrjtlWRD+y1HctngfOkIWc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-16_14:38:39', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T14:56:01.736 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:56:01.741 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:56:01.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:56:01.821 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:56:01.913 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:01.913 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:56:01.951 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:56:01.951 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_1 2024-01-16T14:56:02.030 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:56:02.030 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:02.030 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-01-16T14:56:02.030 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:02.031 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:02.031 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-16 14:55:51.798439732 +0000 2024-01-16T14:56:02.031 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-16 14:53:24.621851755 +0000 2024-01-16T14:56:02.031 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-16 14:53:24.621851755 +0000 2024-01-16T14:56:02.031 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: 2024-01-16 14:53:24.621851755 +0000 2024-01-16T14:56:02.031 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:56:02.133 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-16T14:56:02.133 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-16T14:56:02.133 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000315095 s, 1.6 MB/s 2024-01-16T14:56:02.136 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:56:02.174 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_2 2024-01-16T14:56:02.251 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:56:02.251 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:02.251 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-01-16T14:56:02.251 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:02.251 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:02.251 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-16 14:55:51.799439711 +0000 2024-01-16T14:56:02.251 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-16 14:53:25.215839123 +0000 2024-01-16T14:56:02.252 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-16 14:53:25.215839123 +0000 2024-01-16T14:56:02.252 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: 2024-01-16 14:53:25.215839123 +0000 2024-01-16T14:56:02.252 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:56:02.372 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-16T14:56:02.372 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-16T14:56:02.372 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000319274 s, 1.6 MB/s 2024-01-16T14:56:02.375 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:56:02.417 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_3 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-16 14:55:51.799439711 +0000 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-16 14:53:25.787826960 +0000 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-16 14:53:25.787826960 +0000 2024-01-16T14:56:02.498 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: 2024-01-16 14:53:25.787826960 +0000 2024-01-16T14:56:02.499 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:56:02.602 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-16T14:56:02.602 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-16T14:56:02.603 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000295526 s, 1.7 MB/s 2024-01-16T14:56:02.605 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:56:02.645 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_4 2024-01-16T14:56:02.724 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:56:02.724 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:02.724 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-01-16T14:56:02.724 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:02.724 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:02.724 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-16 14:55:51.800439690 +0000 2024-01-16T14:56:02.724 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-16 14:53:26.348815030 +0000 2024-01-16T14:56:02.725 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-16 14:53:26.348815030 +0000 2024-01-16T14:56:02.725 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: 2024-01-16 14:53:26.348815030 +0000 2024-01-16T14:56:02.725 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:56:02.830 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-16T14:56:02.830 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-16T14:56:02.830 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000281088 s, 1.8 MB/s 2024-01-16T14:56:02.833 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:56:02.875 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:02.875 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:02.875 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:56:02.933 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:56:02.933 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-01-16T14:56:02.983 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-01-16 14:55:41.339035129 +0000 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-01-16 14:53:24.625595905 +0000 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-01-16 14:53:24.625595905 +0000 2024-01-16T14:56:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-01-16 14:53:24.625595905 +0000 2024-01-16T14:56:02.984 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:56:03.072 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-01-16T14:56:03.072 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-01-16T14:56:03.072 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000461885 s, 1.1 MB/s 2024-01-16T14:56:03.076 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:56:03.134 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-01-16 14:55:41.340035110 +0000 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-01-16 14:53:25.174585649 +0000 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-01-16 14:53:25.174585649 +0000 2024-01-16T14:56:03.182 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-01-16 14:53:25.174585649 +0000 2024-01-16T14:56:03.182 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:56:03.264 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-01-16T14:56:03.265 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-01-16T14:56:03.265 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000358999 s, 1.4 MB/s 2024-01-16T14:56:03.268 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:56:03.324 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-01-16T14:56:03.372 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:56:03.372 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:03.372 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-01-16T14:56:03.373 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:03.373 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:03.373 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-01-16 14:55:41.341035091 +0000 2024-01-16T14:56:03.373 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-01-16 14:53:25.729575281 +0000 2024-01-16T14:56:03.373 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-01-16 14:53:25.729575281 +0000 2024-01-16T14:56:03.373 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-01-16 14:53:25.729575281 +0000 2024-01-16T14:56:03.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:56:03.504 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-01-16T14:56:03.505 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-01-16T14:56:03.505 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000322832 s, 1.6 MB/s 2024-01-16T14:56:03.509 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:56:03.565 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-01-16 14:55:41.341035091 +0000 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-01-16 14:53:26.297564671 +0000 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-01-16 14:53:26.297564671 +0000 2024-01-16T14:56:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-01-16 14:53:26.297564671 +0000 2024-01-16T14:56:03.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:56:03.743 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-01-16T14:56:03.743 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-01-16T14:56:03.743 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000318382 s, 1.6 MB/s 2024-01-16T14:56:03.746 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:56:03.802 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:03.802 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi141.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:56:03.802 INFO:tasks.ceph:Generating config... 2024-01-16T14:56:03.803 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-16T14:56:03.803 INFO:tasks.ceph:[client] debug client = 20 2024-01-16T14:56:03.804 INFO:tasks.ceph:[client] debug ms = 1 2024-01-16T14:56:03.804 INFO:tasks.ceph:[client] pid file = 2024-01-16T14:56:03.804 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-16T14:56:03.804 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-16T14:56:03.804 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-16T14:56:03.805 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-16T14:56:03.805 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T14:56:03.805 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T14:56:03.805 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T14:56:03.805 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T14:56:03.805 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T14:56:03.805 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-16T14:56:03.805 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-16T14:56:03.806 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-16T14:56:03.806 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T14:56:03.806 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T14:56:03.955 INFO:teuthology.orchestra.run.smithi141.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T14:56:03.963 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T14:56:04.091 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:56:04.183 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.141'), ('mon.b', '172.21.15.188'), ('mon.c', '[v2:172.21.15.188:3301,v1:172.21.15.188:6790]')] 2024-01-16T14:56:04.184 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.141,172.21.15.188,[v2:172.21.15.188:3301,v1:172.21.15.188:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-16T14:56:04.185 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:04.185 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:56:04.237 DEBUG:teuthology.orchestra.run.smithi141:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.141 --add b 172.21.15.188 --addv c '[v2:172.21.15.188:3301,v1:172.21.15.188:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:04.377 INFO:teuthology.orchestra.run.smithi141.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:04.377 INFO:teuthology.orchestra.run.smithi141.stdout:monmaptool: generated fsid 540b621b-477a-463b-8a92-7e448bae48f5 2024-01-16T14:56:04.377 INFO:teuthology.orchestra.run.smithi141.stdout:setting min_mon_release = pacific 2024-01-16T14:56:04.377 INFO:teuthology.orchestra.run.smithi141.stdout:epoch 0 2024-01-16T14:56:04.377 INFO:teuthology.orchestra.run.smithi141.stdout:fsid 540b621b-477a-463b-8a92-7e448bae48f5 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:last_changed 2024-01-16T14:56:04.395772+0000 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:created 2024-01-16T14:56:04.395772+0000 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:min_mon_release 16 (pacific) 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:election_strategy: 1 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:0: [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] mon.a 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.b 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:2: [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] mon.c 2024-01-16T14:56:04.378 INFO:teuthology.orchestra.run.smithi141.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-16T14:56:04.380 DEBUG:teuthology.orchestra.run.smithi141:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:56:04.422 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 540b621b-477a-463b-8a92-7e448bae48f5... 2024-01-16T14:56:04.423 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:56:04.469 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:56:04.624 INFO:teuthology.orchestra.run.smithi141.stdout:[global] 2024-01-16T14:56:04.624 INFO:teuthology.orchestra.run.smithi141.stdout: chdir = "" 2024-01-16T14:56:04.624 INFO:teuthology.orchestra.run.smithi141.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:56:04.624 INFO:teuthology.orchestra.run.smithi141.stdout: auth supported = cephx 2024-01-16T14:56:04.624 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.624 INFO:teuthology.orchestra.run.smithi141.stdout: filestore xattr use omap = true 2024-01-16T14:56:04.624 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: mon clock drift allowed = 1.000 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: osd crush chooseleaf type = 0 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: auth debug = true 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: ms die on old message = true 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: ms die on bug = true 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: mon pg warn max object skew = 0 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:56:04.625 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: osd pool default size = 2 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd allow primary affinity = true 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd allow pg remap = true 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on no sortbitwise = false 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on too few osds = false 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: mon_allow_pool_size_one = true 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:56:04.626 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: osd default data pool replay window = 5 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: mon allow pool delete = true 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: mon cluster log file level = debug 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: debug asserts on shutdown = true 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: mon health detail to clog = false 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: mon host = "172.21.15.141,172.21.15.188,[v2:172.21.15.188:3301,v1:172.21.15.188:6790]" 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: fsid = 540b621b-477a-463b-8a92-7e448bae48f5 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout:[osd] 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: osd journal size = 100 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: osd scrub load threshold = 5.0 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: osd scrub max interval = 600 2024-01-16T14:56:04.627 INFO:teuthology.orchestra.run.smithi141.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd recover clone overlap = true 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd recovery max chunk = 1048576 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug shutdown = true 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug op order = true 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug verify stray on activate = true 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug trim objects = true 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd open classes on start = true 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug pg log writeout = true 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:56:04.628 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: osd map max advance = 10 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: journal zero on create = true 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: filestore apply finisher threads = 3 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: bdev debug aio = true 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug misdirected ops = true 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: bdev async discard = True 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: bdev enable discard = True 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: bluestore allocator = bitmap 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: bluestore block size = 96636764160 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: bluestore fsck on mount = True 2024-01-16T14:56:04.629 INFO:teuthology.orchestra.run.smithi141.stdout: debug bluefs = 1/20 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: debug bluestore = 1/20 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: debug osd = 20 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: debug rocksdb = 4/10 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd full ratio = 0.9 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: osd objectstore = bluestore 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: osd op complaint time = 180 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr] 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: debug mgr = 20 2024-01-16T14:56:04.630 INFO:teuthology.orchestra.run.smithi141.stdout: debug mon = 20 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: debug auth = 20 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: mgr/telemetry/nag = false 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: debug client = 20 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: debug finisher = 20 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout:[mon] 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: debug mon = 20 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: debug paxos = 20 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: debug auth = 20 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: mon data avail warn = 5 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: mon mgr mkfs grace = 240 2024-01-16T14:56:04.631 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd reporter subtree level = osd 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd prime pg temp = true 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: # 1m isn't quite enough 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:56:04.632 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: mon op complaint time = 120 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout:[client] 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: rgw cache enabled = true 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: rgw enable ops log = true 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: rgw enable usage log = true 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: client mount timeout = 600 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: debug client = 20 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: pid file = "" 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: rados mon op timeout = 900 2024-01-16T14:56:04.633 INFO:teuthology.orchestra.run.smithi141.stdout: rados osd op timeout = 900 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.a] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.b] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.c] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.a] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.c] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.b] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.d] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout:[mds] 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: debug mds = 20 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: debug mds balancer = 20 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: mds debug frag = True 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: mds debug scatterstat = True 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: mds op complaint time = 180 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: mds verify scatter = True 2024-01-16T14:56:04.634 INFO:teuthology.orchestra.run.smithi141.stdout: osd op complaint time = 180 2024-01-16T14:56:04.635 INFO:teuthology.orchestra.run.smithi141.stdout: rados mon op timeout = 900 2024-01-16T14:56:04.635 INFO:teuthology.orchestra.run.smithi141.stdout: rados osd op timeout = 900 2024-01-16T14:56:04.643 INFO:teuthology.orchestra.run.smithi188.stdout:[global] 2024-01-16T14:56:04.643 INFO:teuthology.orchestra.run.smithi188.stdout: chdir = "" 2024-01-16T14:56:04.643 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:56:04.643 INFO:teuthology.orchestra.run.smithi188.stdout: auth supported = cephx 2024-01-16T14:56:04.643 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: filestore xattr use omap = true 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: mon clock drift allowed = 1.000 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: osd crush chooseleaf type = 0 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: auth debug = true 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on old message = true 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on bug = true 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: mon pg warn max object skew = 0 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default size = 2 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow primary affinity = true 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow pg remap = true 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on no sortbitwise = false 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on too few osds = false 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: mon_allow_pool_size_one = true 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.645 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: osd default data pool replay window = 5 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: mon allow pool delete = true 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: mon cluster log file level = debug 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: debug asserts on shutdown = true 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: mon health detail to clog = false 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: mon host = "172.21.15.141,172.21.15.188,[v2:172.21.15.188:3301,v1:172.21.15.188:6790]" 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: fsid = 540b621b-477a-463b-8a92-7e448bae48f5 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout:[osd] 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: osd journal size = 100 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.646 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub load threshold = 5.0 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub max interval = 600 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd recover clone overlap = true 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd recovery max chunk = 1048576 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug shutdown = true 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug op order = true 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify stray on activate = true 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug trim objects = true 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd open classes on start = true 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug pg log writeout = true 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:56:04.647 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: osd map max advance = 10 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: journal zero on create = true 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: filestore apply finisher threads = 3 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: bdev debug aio = true 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug misdirected ops = true 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: bdev async discard = True 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: bdev enable discard = True 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore allocator = bitmap 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore block size = 96636764160 2024-01-16T14:56:04.648 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore fsck on mount = True 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluefs = 1/20 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluestore = 1/20 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug osd = 20 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug rocksdb = 4/10 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd full ratio = 0.9 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: osd objectstore = bluestore 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: osd op complaint time = 180 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr] 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug mgr = 20 2024-01-16T14:56:04.649 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: mgr/telemetry/nag = false 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: debug client = 20 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: debug finisher = 20 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout:[mon] 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: debug paxos = 20 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: mon data avail warn = 5 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: mon mgr mkfs grace = 240 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:56:04.650 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd reporter subtree level = osd 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd prime pg temp = true 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: # 1m isn't quite enough 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: mon op complaint time = 120 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout:[client] 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: rgw cache enabled = true 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable ops log = true 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable usage log = true 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: client mount timeout = 600 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: debug client = 20 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = "" 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: rados mon op timeout = 900 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: rados osd op timeout = 900 2024-01-16T14:56:04.652 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.a] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.b] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.c] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.a] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.c] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.b] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.d] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout:[mds] 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: debug mds = 20 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: debug mds balancer = 20 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: mds debug frag = True 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: mds debug scatterstat = True 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: mds op complaint time = 180 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: mds verify scatter = True 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: osd op complaint time = 180 2024-01-16T14:56:04.653 INFO:teuthology.orchestra.run.smithi188.stdout: rados mon op timeout = 900 2024-01-16T14:56:04.654 INFO:teuthology.orchestra.run.smithi188.stdout: rados osd op timeout = 900 2024-01-16T14:56:04.685 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T14:56:04.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T14:56:04.812 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T14:56:04.812 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:04.813 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T14:56:04.852 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:04.852 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T14:56:04.937 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:56:04.937 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:04.938 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:56:04.938 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:56:05.094 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:05.094 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:05.176 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:56:05.177 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:05.177 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:56:05.177 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:56:05.320 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:05.320 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:05.415 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T14:56:05.415 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T14:56:05.415 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:56:05.568 INFO:teuthology.orchestra.run.smithi141.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:56:05.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:56:05.764 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:56:05.775 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T14:56:05.775 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:56:05.946 INFO:teuthology.orchestra.run.smithi141.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:56:05.955 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-16T14:56:06.047 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:56:06.220 INFO:teuthology.orchestra.run.smithi141.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:56:06.229 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-16T14:56:06.315 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:56:06.478 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:56:06.487 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-16T14:56:06.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:56:06.743 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:56:06.752 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-16T14:56:06.843 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T14:56:06.843 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T14:56:06.970 INFO:teuthology.orchestra.run.smithi141.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T14:56:07.030 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-16T14:56:07.149 INFO:teuthology.orchestra.run.smithi141.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-16T14:56:07.210 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-16T14:56:07.339 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-16T14:56:07.395 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-16T14:56:07.518 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-16T14:56:07.571 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T14:56:07.571 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi141.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:56:07.571 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T14:56:07.669 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:07.669 INFO:tasks.ceph:role: osd.0 2024-01-16T14:56:07.669 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:56:07.669 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:07.767 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:07.768 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:08.286 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-16T14:56:08.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-16T14:56:08.294 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T14:56:08.483 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T14:56:08.591 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T14:56:08.680 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:08.680 INFO:tasks.ceph:role: osd.1 2024-01-16T14:56:08.680 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:56:08.680 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:08.777 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:09.264 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-16T14:56:09.270 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-16T14:56:09.270 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T14:56:09.397 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T14:56:09.485 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T14:56:09.574 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:09.574 INFO:tasks.ceph:role: osd.2 2024-01-16T14:56:09.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:56:09.575 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:56:09.670 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:09.670 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:09.670 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:09.671 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:09.671 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:09.671 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:09.671 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:09.671 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:09.671 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:09.671 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:10.187 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-16T14:56:10.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-16T14:56:10.193 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T14:56:10.365 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T14:56:10.467 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-16T14:56:10.557 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:10.581 INFO:tasks.ceph:role: osd.3 2024-01-16T14:56:10.582 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:56:10.582 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:56:10.683 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:10.684 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:11.124 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-16T14:56:11.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-16T14:56:11.132 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-16T14:56:11.289 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-16T14:56:11.386 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:11.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:11.553+0000 7f579ba0c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T14:56:11.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:11.554+0000 7f579ba0c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T14:56:11.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:11.554+0000 7f579ba0c500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:56:11.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:11.554+0000 7f579ba0c500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:56:11.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:11.554+0000 7f579ba0c500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:56:11.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:11.559+0000 7f579ba0c500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T14:56:14.714 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T14:56:14.809 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:14.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:14.962+0000 7f1d25021500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T14:56:14.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:14.963+0000 7f1d25021500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T14:56:14.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:14.963+0000 7f1d25021500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:56:14.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:14.963+0000 7f1d25021500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:56:14.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:14.963+0000 7f1d25021500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:56:14.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:14.965+0000 7f1d25021500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T14:56:18.116 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T14:56:18.211 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:18.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:18.360+0000 7fd6e50da500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T14:56:18.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:18.361+0000 7fd6e50da500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T14:56:18.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:18.361+0000 7fd6e50da500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:56:18.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:18.361+0000 7fd6e50da500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:56:18.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:18.361+0000 7fd6e50da500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:56:18.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:18.363+0000 7fd6e50da500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T14:56:21.277 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T14:56:21.373 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:21.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:21.524+0000 7fd2c9a05500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-16T14:56:21.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:21.525+0000 7fd2c9a05500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-16T14:56:21.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:21.525+0000 7fd2c9a05500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:56:21.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:21.525+0000 7fd2c9a05500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:56:21.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:21.525+0000 7fd2c9a05500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:56:21.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:21.527+0000 7fd2c9a05500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-16T14:56:24.686 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-16T14:56:24.781 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-16T14:56:24.884 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:24.884 INFO:tasks.ceph:role: osd.4 2024-01-16T14:56:24.884 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:56:24.885 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:56:24.984 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:24.984 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:24.984 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:24.984 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:24.985 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:24.985 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:24.985 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:24.985 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:24.985 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:24.985 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:25.488 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-01-16T14:56:25.495 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-01-16T14:56:25.496 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-16T14:56:25.679 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-16T14:56:25.786 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-16T14:56:25.881 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:25.881 INFO:tasks.ceph:role: osd.5 2024-01-16T14:56:25.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:56:25.881 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:56:25.977 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:25.978 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:26.454 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-01-16T14:56:26.461 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-01-16T14:56:26.461 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-16T14:56:26.599 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-16T14:56:26.691 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:26.784 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:26.784 INFO:tasks.ceph:role: osd.6 2024-01-16T14:56:26.784 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:56:26.784 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:26.881 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:27.402 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-01-16T14:56:27.409 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-01-16T14:56:27.409 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:27.566 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:27.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:27.758 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:56:27.758 INFO:tasks.ceph:role: osd.7 2024-01-16T14:56:27.758 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:56:27.758 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:56:27.852 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:56:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:56:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:56:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:56:27.853 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:56:27.853 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:56:27.853 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:56:27.853 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:56:27.853 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:56:27.853 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:56:28.304 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-01-16T14:56:28.310 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-01-16T14:56:28.311 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:28.453 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:28.548 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:28.691 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:28.709+0000 7fd0ff510500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-16T14:56:28.695 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:28.713+0000 7fd0ff510500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-16T14:56:28.695 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:28.713+0000 7fd0ff510500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:56:28.695 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:28.714+0000 7fd0ff510500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:56:28.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:28.714+0000 7fd0ff510500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:56:28.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:28.718+0000 7fd0ff510500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-16T14:56:31.738 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:31.746 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:56:31.898 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-16T14:56:31.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:32.121 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:32.139+0000 7feb29869500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-16T14:56:32.122 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:32.140+0000 7feb29869500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-16T14:56:32.122 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:32.140+0000 7feb29869500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:56:32.122 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:32.140+0000 7feb29869500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:56:32.122 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:32.140+0000 7feb29869500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:56:32.124 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:32.142+0000 7feb29869500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-16T14:56:35.297 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-16T14:56:35.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:35.526 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:35.544+0000 7fe373c15500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:35.545+0000 7fe373c15500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:35.545+0000 7fe373c15500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:35.546+0000 7fe373c15500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:35.546+0000 7fe373c15500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:56:35.529 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:35.547+0000 7fe373c15500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-16T14:56:38.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-16T14:56:38.572 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:38.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:38.725+0000 7f25e2b1e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-16T14:56:38.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:38.725+0000 7f25e2b1e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-16T14:56:38.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:38.726+0000 7f25e2b1e500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:56:38.709 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:38.726+0000 7f25e2b1e500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:56:38.709 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:38.726+0000 7f25e2b1e500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:56:38.710 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:38.728+0000 7f25e2b1e500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-16T14:56:41.622 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-16T14:56:41.717 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T14:56:41.717 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:41.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-16T14:56:41.811 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:41.811 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-16T14:56:41.906 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:41.906 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-16T14:56:41.999 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:41.999 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T14:56:42.136 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:42.137 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T14:56:42.233 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:42.233 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T14:56:42.324 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:42.325 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-16T14:56:42.418 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:42.418 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T14:56:42.519 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:42.519 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-16T14:56:42.615 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:42.615 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-16T14:56:42.708 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:42.708 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-16T14:56:42.803 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:42.804 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-16T14:56:42.893 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:42.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-16T14:56:43.024 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:43.025 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-16T14:56:43.113 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:43.113 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T14:56:43.170 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:43.170 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-16T14:56:43.221 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:43.222 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-16T14:56:43.277 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:43.277 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-16T14:56:43.326 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T14:56:43.326 DEBUG:teuthology.orchestra.run.smithi141:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:56:43.333 DEBUG:teuthology.orchestra.run.smithi188:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:56:43.419 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr.y] 2024-01-16T14:56:43.419 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA== 2024-01-16T14:56:43.419 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.a] 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ== 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.c] 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw== 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.0] 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA== 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.1] 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg== 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.2] 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g== 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.3] 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w== 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr.x] 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ== 2024-01-16T14:56:43.420 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.b] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.d] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.4] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.5] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.6] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.7] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[client.2] 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ== 2024-01-16T14:56:43.421 INFO:teuthology.orchestra.run.smithi141.stdout:[client.1] 2024-01-16T14:56:43.422 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ== 2024-01-16T14:56:43.422 INFO:teuthology.orchestra.run.smithi141.stdout:[client.3] 2024-01-16T14:56:43.422 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA== 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.y] 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA== 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.a] 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ== 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.c] 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw== 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.0] 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA== 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.1] 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg== 2024-01-16T14:56:43.458 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.2] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.3] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.x] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.b] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[mds.d] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.4] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.5] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.6] 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ== 2024-01-16T14:56:43.459 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.7] 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg== 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout:[client.0] 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout:[client.2] 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ== 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout:[client.1] 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ== 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout:[client.3] 2024-01-16T14:56:43.460 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA== 2024-01-16T14:56:43.462 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:43.471 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:43.611 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:43.619 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:43.760 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:43.765 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:43.909 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:43.914 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.049 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.056 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.196 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.203 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.341 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.348 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.488 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:44.493 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:56:44.630 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:44.634 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:44.775 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:44.781 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:56:44.924 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:44.928 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:45.068 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:45.076 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:45.217 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:45.222 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:45.358 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:45.365 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:56:45.506 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:45.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:45.657 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:45.665 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:45.806 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:45.811 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:45.951 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:45.956 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:56:46.090 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T14:56:46.090 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T14:56:46.181 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:56:46.336 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T14:56:46.420 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-16T14:56:46.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:56:46.669 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-16T14:56:46.744 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-16T14:56:46.838 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:56:46.994 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-16T14:56:47.079 DEBUG:teuthology.orchestra.run.smithi141:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:47.086 DEBUG:teuthology.orchestra.run.smithi188:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:56:47.138 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T14:56:47.138 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T14:56:47.139 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T14:56:47.143 INFO:tasks.ceph.mon.a:Started 2024-01-16T14:56:47.144 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-16T14:56:47.144 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-16T14:56:47.148 INFO:tasks.ceph.mon.b:Started 2024-01-16T14:56:47.149 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-16T14:56:47.149 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-16T14:56:47.153 INFO:tasks.ceph.mon.c:Started 2024-01-16T14:56:47.153 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T14:56:47.153 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-16T14:56:47.153 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-16T14:56:47.158 INFO:tasks.ceph.mgr.y:Started 2024-01-16T14:56:47.158 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T14:56:47.158 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T14:56:47.162 INFO:tasks.ceph.mgr.x:Started 2024-01-16T14:56:47.162 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T14:56:47.163 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T14:56:47.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.362+0000 7f53ef200640 1 Processor -- start 2024-01-16T14:56:47.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.362+0000 7f53ef200640 1 -- start start 2024-01-16T14:56:47.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ef200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ef200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ef200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 0x7f53f015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ef200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f013ff60 con 0x7f53f00cdc20 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ef200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f00ce440 con 0x7f53f013cf10 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ef200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00ce720 con 0x7f53f013fae0 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ed400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ede00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ed400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:55580/0 (socket says 172.21.15.141:55580) 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ede00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:41268/0 (socket says 172.21.15.141:41268) 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ed400640 1 -- 172.21.15.141:0/4193700512 learned_addr learned my addr 172.21.15.141:0/4193700512 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.364+0000 7f53ee800640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 0x7f53f015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ed400640 1 -- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 msgr2=0x7f53f00cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ed400640 1 -- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 msgr2=0x7f53f00cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ed400640 1 --2- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ed400640 1 --2- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:56:47.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ede00640 1 -- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 msgr2=0x7f53f013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-16T14:56:47.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ede00640 1 -- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 msgr2=0x7f53f013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:56:47.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ede00640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:56:47.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ede00640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:56:47.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ee800640 1 -- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 msgr2=0x7f53f015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-16T14:56:47.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ee800640 1 -- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 msgr2=0x7f53f015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:56:47.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ee800640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 0x7f53f015d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:56:47.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.365+0000 7f53ee800640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 0x7f53f015d9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:56:47.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.566+0000 7f53ed400640 1 --2- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.566+0000 7f53ede00640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.566+0000 7f53ee800640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 0x7f53f015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.567+0000 7f53ed400640 1 -- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 msgr2=0x7f53f013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:47.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.567+0000 7f53ed400640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.567+0000 7f53ed400640 1 -- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 msgr2=0x7f53f015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:47.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.567+0000 7f53ed400640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 0x7f53f015d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.567+0000 7f53ed400640 1 -- 172.21.15.141:0/4193700512 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00cea00 con 0x7f53f013fae0 2024-01-16T14:56:47.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.568+0000 7f53ed400640 1 --2- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f53f006e390 tx=0x7f53d802e280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7376cfd52cf097f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:47.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.569+0000 7f53eca00640 1 -- 172.21.15.141:0/4193700512 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8036640 con 0x7f53f013fae0 2024-01-16T14:56:47.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.569+0000 7f53eca00640 1 -- 172.21.15.141:0/4193700512 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d80367a0 con 0x7f53f013fae0 2024-01-16T14:56:47.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.569+0000 7f53eca00640 1 -- 172.21.15.141:0/4193700512 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8036b30 con 0x7f53f013fae0 2024-01-16T14:56:47.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.570+0000 7f53ef200640 1 -- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 msgr2=0x7f53f00cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:47.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.570+0000 7f53ef200640 1 --2- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f53f006e390 tx=0x7f53d802e280 comp rx=0 tx=0).stop 2024-01-16T14:56:47.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.570+0000 7f53ef200640 1 -- 172.21.15.141:0/4193700512 shutdown_connections 2024-01-16T14:56:47.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.570+0000 7f53ef200640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.141:3300/0 conn(0x7f53f00cdc20 0x7f53f015d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.570+0000 7f53ef200640 1 --2- 172.21.15.141:0/4193700512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f013fae0 0x7f53f00cd6e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.570+0000 7f53ef200640 1 --2- 172.21.15.141:0/4193700512 >> v2:172.21.15.188:3300/0 conn(0x7f53f013cf10 0x7f53f013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.570+0000 7f53ef200640 1 -- 172.21.15.141:0/4193700512 >> 172.21.15.141:0/4193700512 conn(0x7f53f006a7d0 msgr2=0x7f53f006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:47.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.571+0000 7f53ef200640 1 -- 172.21.15.141:0/4193700512 shutdown_connections 2024-01-16T14:56:47.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.571+0000 7f53ef200640 1 -- 172.21.15.141:0/4193700512 wait complete. 2024-01-16T14:56:47.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.573+0000 7f53ef200640 1 Processor -- start 2024-01-16T14:56:47.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.573+0000 7f53ef200640 1 -- start start 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ef200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f00cdc20 0x7f53f01576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ef200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53f013cf10 0x7f53f013be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ef200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53f013fae0 0x7f53f015c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ef200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f013d520 con 0x7f53f013fae0 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ef200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f015c8f0 con 0x7f53f013cf10 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ef200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f015cbc0 con 0x7f53f00cdc20 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ee800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53f013fae0 0x7f53f015c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ee800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53f013fae0 0x7f53f015c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41406/0 (socket says 172.21.15.141:41406) 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ee800640 1 -- 172.21.15.141:0/2775450491 learned_addr learned my addr 172.21.15.141:0/2775450491 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ede00640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f00cdc20 0x7f53f01576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.575+0000 7f53ed400640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53f013cf10 0x7f53f013be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:47.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53ee800640 1 -- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f00cdc20 msgr2=0x7f53f01576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:47.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53ee800640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f00cdc20 0x7f53f01576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53ee800640 1 -- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53f013cf10 msgr2=0x7f53f013be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:47.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53ee800640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53f013cf10 0x7f53f013be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53ee800640 1 -- 172.21.15.141:0/2775450491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d800e780 con 0x7f53f013fae0 2024-01-16T14:56:47.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53ee800640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53f013fae0 0x7f53f015c3b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f53e8004630 tx=0x7f53e8007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53e6000640 1 -- 172.21.15.141:0/2775450491 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e800ab30 con 0x7f53f013fae0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53e6000640 1 -- 172.21.15.141:0/2775450491 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e8009bf0 con 0x7f53f013fae0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53e6000640 1 -- 172.21.15.141:0/2775450491 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e800a030 con 0x7f53f013fae0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.576+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f015cea0 con 0x7f53f013fae0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.577+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f0070fe0 con 0x7f53f013fae0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.577+0000 7f53e6000640 1 -- 172.21.15.141:0/2775450491 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f53e800a190 con 0x7f53f013fae0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.577+0000 7f53e6000640 1 -- 172.21.15.141:0/2775450491 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f53e8015c30 con 0x7f53f013fae0 2024-01-16T14:56:47.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.578+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f015d3c0 con 0x7f53f013fae0 2024-01-16T14:56:47.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.582+0000 7f53e6000640 1 -- 172.21.15.141:0/2775450491 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f53f015d3c0 con 0x7f53f013fae0 2024-01-16T14:56:47.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.655+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f53f013d2f0 con 0x7f53f013fae0 2024-01-16T14:56:47.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.658+0000 7f53e6000640 1 -- 172.21.15.141:0/2775450491 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f53f013d2f0 con 0x7f53f013fae0 2024-01-16T14:56:47.640 INFO:teuthology.orchestra.run.smithi141.stderr:adjusted tunables profile to default 2024-01-16T14:56:47.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.660+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53f013fae0 msgr2=0x7f53f015c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:47.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.660+0000 7f53ef200640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53f013fae0 0x7f53f015c3b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f53e8004630 tx=0x7f53e8007700 comp rx=0 tx=0).stop 2024-01-16T14:56:47.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.660+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 shutdown_connections 2024-01-16T14:56:47.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.660+0000 7f53ef200640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53f013fae0 0x7f53f015c3b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.660+0000 7f53ef200640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53f013cf10 0x7f53f013be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.660+0000 7f53ef200640 1 --2- 172.21.15.141:0/2775450491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53f00cdc20 0x7f53f01576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:47.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.660+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 >> 172.21.15.141:0/2775450491 conn(0x7f53f006a7d0 msgr2=0x7f53f00aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:47.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.661+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 shutdown_connections 2024-01-16T14:56:47.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:47.661+0000 7f53ef200640 1 -- 172.21.15.141:0/2775450491 wait complete. 2024-01-16T14:56:47.667 INFO:tasks.ceph:check_enable_crimson: False 2024-01-16T14:56:47.667 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T14:56:47.667 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:47.667 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T14:56:47.751 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:47.751 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T14:56:47.879 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:47.879 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T14:56:48.009 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:56:48.009 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-16T14:56:48.071 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:48.088+0000 7f0fb5c30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:56:48.135 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:48.152+0000 7f0fb5c30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:56:48.142 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:48.142 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-16T14:56:48.170 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:48.188+0000 7f03e761b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:56:48.232 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:48.232 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-16T14:56:48.234 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:48.252+0000 7f03e761b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:56:48.361 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:48.361 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-16T14:56:48.487 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:56:48.487 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-16T14:56:48.619 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new a9dc7c32-2f84-4fe7-b960-b102964c1b38 0 2024-01-16T14:56:48.756 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:48.774+0000 7f0fb5c30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:56:48.819 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:48.836+0000 7f0fb5c30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:56:48.851 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.868+0000 7fcf37000640 1 Processor -- start 2024-01-16T14:56:48.851 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.868+0000 7fcf37000640 1 -- start start 2024-01-16T14:56:48.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.870+0000 7fcf37000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcf300cfde0 0x7fcf300b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.870+0000 7fcf37000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300d1c60 con 0x7fcf300cf9a0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.870+0000 7fcf37000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300b2730 con 0x7fcf3015d4b0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.870+0000 7fcf37000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf300d1ec0 con 0x7fcf300cfde0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.870+0000 7fcf35c00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fcf3015d4b0 0x7fcf300d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:56924/0 (socket says 172.21.15.188:56924) 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.870+0000 7fcf35c00640 1 -- 172.21.15.188:0/2472851878 learned_addr learned my addr 172.21.15.188:0/2472851878 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.870+0000 7fcf36600640 1 --2- 172.21.15.188:0/2472851878 >> v2:172.21.15.141:3300/0 conn(0x7fcf300cfde0 0x7fcf300b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293939209 0 0) 0x7fcf300b2730 con 0x7fcf3015d4b0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf10003520 con 0x7fcf3015d4b0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3988262922 0 0) 0x7fcf10003520 con 0x7fcf3015d4b0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 >> v2:172.21.15.141:3300/0 conn(0x7fcf300cfde0 msgr2=0x7fcf300b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 --2- 172.21.15.188:0/2472851878 >> v2:172.21.15.141:3300/0 conn(0x7fcf300cfde0 0x7fcf300b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 >> v1:172.21.15.141:6789/0 conn(0x7fcf300cf9a0 legacy=0x7fcf300b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300b29d0 con 0x7fcf3015d4b0 2024-01-16T14:56:48.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fcf20001670 con 0x7fcf3015d4b0 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 >> v1:172.21.15.188:6789/0 conn(0x7fcf3015d4b0 legacy=0x7fcf300d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf10003970 0x7fcf10003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf10005180 0x7fcf1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300d1ec0 con 0x7fcf10005180 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1000c020 con 0x7fcf100047c0 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1000c1e0 con 0x7fcf10003970 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf35200640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.871+0000 7fcf35c00640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf10003970 0x7fcf10003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf36600640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf10005180 0x7fcf1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf35200640 1 -- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf10003970 msgr2=0x7fcf10003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf35200640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf10003970 0x7fcf10003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf35200640 1 -- 172.21.15.188:0/2472851878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf10005180 msgr2=0x7fcf1000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:48.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf35200640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf10005180 0x7fcf1000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf35200640 1 -- 172.21.15.188:0/2472851878 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf10003520 con 0x7fcf100047c0 2024-01-16T14:56:48.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf36600640 1 -- 172.21.15.188:0/2472851878 reap_dead start 2024-01-16T14:56:48.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.872+0000 7fcf35c00640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf10003970 0x7fcf10003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:48.858 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.873+0000 7fcf35200640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf10004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcf18001750 tx=0x7fcf100057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=955a9694873fe92b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:48.858 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.873+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180052d0 con 0x7fcf100047c0 2024-01-16T14:56:48.858 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.873+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1802e050 con 0x7fcf100047c0 2024-01-16T14:56:48.858 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.873+0000 7fcf2fe00640 1 -- 172.21.15.188:0/2472851878 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18003830 con 0x7fcf100047c0 2024-01-16T14:56:48.858 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.874+0000 7fcf37000640 1 -- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 msgr2=0x7fcf10004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.858 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.874+0000 7fcf37000640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf10004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcf18001750 tx=0x7fcf100057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.874+0000 7fcf37000640 1 -- 172.21.15.188:0/2472851878 shutdown_connections 2024-01-16T14:56:48.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.874+0000 7fcf37000640 1 --2- 172.21.15.188:0/2472851878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf10004c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.874+0000 7fcf37000640 1 -- 172.21.15.188:0/2472851878 >> 172.21.15.188:0/2472851878 conn(0x7fcf300d3000 msgr2=0x7fcf300d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:48.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.874+0000 7fcf37000640 1 -- 172.21.15.188:0/2472851878 shutdown_connections 2024-01-16T14:56:48.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.874+0000 7fcf37000640 1 -- 172.21.15.188:0/2472851878 wait complete. 2024-01-16T14:56:48.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.876+0000 7fcf37000640 1 Processor -- start 2024-01-16T14:56:48.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.876+0000 7fcf37000640 1 -- start start 2024-01-16T14:56:48.861 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf37000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf301f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.861 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf37000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf3006b090 0x7fcf301f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.861 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf37000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf3006b510 0x7fcf3015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:48.861 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf37000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301f6600 con 0x7fcf3006b510 2024-01-16T14:56:48.861 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf37000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301f7550 con 0x7fcf100047c0 2024-01-16T14:56:48.861 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf37000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301f3b40 con 0x7fcf3006b090 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf3006b090 0x7fcf301f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf3006b090 0x7fcf301f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.188:41730/0 (socket says 172.21.15.188:41730) 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 -- 172.21.15.188:0/3136893310 learned_addr learned my addr 172.21.15.188:0/3136893310 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35c00640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf301f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf36600640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf3006b510 0x7fcf3015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 -- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 msgr2=0x7fcf301f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf301f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 -- 172.21.15.188:0/3136893310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf3006b510 msgr2=0x7fcf3015a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf3006b510 0x7fcf3015a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35200640 1 -- 172.21.15.188:0/3136893310 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf301f3e20 con 0x7fcf3006b090 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.877+0000 7fcf35c00640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf301f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf35200640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf3006b090 0x7fcf301f3080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf1802bc00 tx=0x7fcf18001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf2d600640 1 -- 172.21.15.188:0/3136893310 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180076c0 con 0x7fcf3006b090 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf2d600640 1 -- 172.21.15.188:0/3136893310 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18007a30 con 0x7fcf3006b090 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf2d600640 1 -- 172.21.15.188:0/3136893310 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18006be0 con 0x7fcf3006b090 2024-01-16T14:56:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf301f6810 con 0x7fcf3006b090 2024-01-16T14:56:48.863 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf300c98f0 con 0x7fcf3006b090 2024-01-16T14:56:48.863 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf2d600640 1 -- 172.21.15.188:0/3136893310 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcf180038f0 con 0x7fcf3006b090 2024-01-16T14:56:48.863 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.878+0000 7fcf2d600640 1 -- 172.21.15.188:0/3136893310 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcf1803d7d0 con 0x7fcf3006b090 2024-01-16T14:56:48.863 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.879+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf30067b60 con 0x7fcf3006b090 2024-01-16T14:56:48.866 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.884+0000 7fcf2d600640 1 -- 172.21.15.188:0/3136893310 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fcf18041020 con 0x7fcf3006b090 2024-01-16T14:56:48.875 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:48.892+0000 7f0fb5c30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:56:48.875 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:48.893+0000 7f03e761b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:56:48.936 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:48.954+0000 7f03e761b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:56:48.939 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:48.956+0000 7f0fb5c30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:56:48.945 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.963+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a9dc7c32-2f84-4fe7-b960-b102964c1b38", "id": 0} v 0) v1 -- 0x7fcf30067450 con 0x7fcf3006b090 2024-01-16T14:56:48.952 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.969+0000 7fcf2d600640 1 -- 172.21.15.188:0/3136893310 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a9dc7c32-2f84-4fe7-b960-b102964c1b38", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcf18001370 con 0x7fcf3006b090 2024-01-16T14:56:48.952 INFO:teuthology.orchestra.run.smithi188.stdout:0 2024-01-16T14:56:48.953 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.970+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf3006b090 msgr2=0x7fcf301f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:48.953 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.970+0000 7fcf37000640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf3006b090 0x7fcf301f3080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf1802bc00 tx=0x7fcf18001120 comp rx=0 tx=0).stop 2024-01-16T14:56:48.953 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.971+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 shutdown_connections 2024-01-16T14:56:48.953 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.971+0000 7fcf37000640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf3006b510 0x7fcf3015a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.953 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.971+0000 7fcf37000640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf3006b090 0x7fcf301f3080 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.954 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.971+0000 7fcf37000640 1 --2- 172.21.15.188:0/3136893310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf100047c0 0x7fcf301f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:48.954 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.971+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 >> 172.21.15.188:0/3136893310 conn(0x7fcf300d3000 msgr2=0x7fcf300b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:48.954 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.971+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 shutdown_connections 2024-01-16T14:56:48.954 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:48.971+0000 7fcf37000640 1 -- 172.21.15.188:0/3136893310 wait complete. 2024-01-16T14:56:48.970 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new ea3eb329-b238-48a3-a840-536e0547c04f 1 2024-01-16T14:56:48.986 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.005+0000 7f03e761b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:56:48.994 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:49.012+0000 7f0fb5c30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:56:49.053 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.071+0000 7f03e761b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:56:49.102 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.120+0000 7f03e761b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:56:49.112 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:49.129+0000 7f0fb5c30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:56:49.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.182+0000 7f302a600640 1 Processor -- start 2024-01-16T14:56:49.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.183+0000 7f302a600640 1 -- start start 2024-01-16T14:56:49.167 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.184+0000 7f302a600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f30240714a0 0x7f30240ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.184+0000 7f302a600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3024066430 con 0x7f3024160770 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.184+0000 7f302a600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302406a560 con 0x7f302406e5a0 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.184+0000 7f302a600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302406a820 con 0x7f30240714a0 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.185+0000 7f3023e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f30240714a0 0x7f30240ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.185+0000 7f3023e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f30240714a0 0x7f30240ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:42652/0 (socket says 172.21.15.188:42652) 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.185+0000 7f3029c00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f302406e5a0 0x7f3024160e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:56930/0 (socket says 172.21.15.188:56930) 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.185+0000 7f3023e00640 1 -- 172.21.15.188:0/1505278245 learned_addr learned my addr 172.21.15.188:0/1505278245 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.185+0000 7f3023e00640 1 -- 172.21.15.188:0/1505278245 >> v1:172.21.15.188:6789/0 conn(0x7f302406e5a0 legacy=0x7f3024160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.185+0000 7f3023e00640 1 -- 172.21.15.188:0/1505278245 >> v1:172.21.15.141:6789/0 conn(0x7f3024160770 legacy=0x7f3024070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.185+0000 7f3023e00640 1 -- 172.21.15.188:0/1505278245 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302406ab00 con 0x7f30240714a0 2024-01-16T14:56:49.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.187+0000 7f3023e00640 1 --2- 172.21.15.188:0/1505278245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30240714a0 0x7f30240ac8d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f300c009870 tx=0x7f300c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f2cca3fce728e50 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.187+0000 7f3023400640 1 -- 172.21.15.188:0/1505278245 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c02f840 con 0x7f30240714a0 2024-01-16T14:56:49.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.187+0000 7f3023400640 1 -- 172.21.15.188:0/1505278245 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c02f9a0 con 0x7f30240714a0 2024-01-16T14:56:49.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.187+0000 7f3023400640 1 -- 172.21.15.188:0/1505278245 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c02fd30 con 0x7f30240714a0 2024-01-16T14:56:49.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.188+0000 7f302a600640 1 -- 172.21.15.188:0/1505278245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30240714a0 msgr2=0x7f30240ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.188+0000 7f302a600640 1 --2- 172.21.15.188:0/1505278245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30240714a0 0x7f30240ac8d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f300c009870 tx=0x7f300c02ba20 comp rx=0 tx=0).stop 2024-01-16T14:56:49.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.188+0000 7f302a600640 1 -- 172.21.15.188:0/1505278245 shutdown_connections 2024-01-16T14:56:49.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.188+0000 7f302a600640 1 --2- 172.21.15.188:0/1505278245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30240714a0 0x7f30240ac8d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.188+0000 7f302a600640 1 -- 172.21.15.188:0/1505278245 >> 172.21.15.188:0/1505278245 conn(0x7f30240b15b0 msgr2=0x7f30240b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.188+0000 7f302a600640 1 -- 172.21.15.188:0/1505278245 shutdown_connections 2024-01-16T14:56:49.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.188+0000 7f302a600640 1 -- 172.21.15.188:0/1505278245 wait complete. 2024-01-16T14:56:49.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.190+0000 7f302a600640 1 Processor -- start 2024-01-16T14:56:49.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.190+0000 7f302a600640 1 -- start start 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f302a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f302406e5a0 0x7f3024158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f302a600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30240714a0 0x7f302414dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f302a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3024160770 0x7f302414e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f302a600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240cc7a0 con 0x7f30240714a0 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f302a600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240cbe90 con 0x7f302406e5a0 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f302a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30240cec30 con 0x7f3024160770 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f302406e5a0 0x7f3024158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f302406e5a0 0x7f3024158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:42668/0 (socket says 172.21.15.188:42668) 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3024160770 0x7f302414e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029200640 1 -- 172.21.15.188:0/1659504998 learned_addr learned my addr 172.21.15.188:0/1659504998 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3023e00640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30240714a0 0x7f302414dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029c00640 1 -- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f302406e5a0 msgr2=0x7f3024158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029c00640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f302406e5a0 0x7f3024158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029c00640 1 -- 172.21.15.188:0/1659504998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30240714a0 msgr2=0x7f302414dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029c00640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30240714a0 0x7f302414dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029c00640 1 -- 172.21.15.188:0/1659504998 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c0094d0 con 0x7f3024160770 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3029200640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f302406e5a0 0x7f3024158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.191+0000 7f3023e00640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30240714a0 0x7f302414dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f3029c00640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3024160770 0x7f302414e110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f301c00c6c0 tx=0x7f301c00cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f3020c00640 1 -- 172.21.15.188:0/1659504998 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c004d60 con 0x7f3024160770 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f3020c00640 1 -- 172.21.15.188:0/1659504998 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c00a640 con 0x7f3024160770 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f3020c00640 1 -- 172.21.15.188:0/1659504998 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00a9f0 con 0x7f3024160770 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30240cc150 con 0x7f3024160770 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30240ce870 con 0x7f3024160770 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f3020c00640 1 -- 172.21.15.188:0/1659504998 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f301c007070 con 0x7f3024160770 2024-01-16T14:56:49.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.192+0000 7f3020c00640 1 -- 172.21.15.188:0/1659504998 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f301c0127f0 con 0x7f3024160770 2024-01-16T14:56:49.177 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.193+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30240cc910 con 0x7f3024160770 2024-01-16T14:56:49.178 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.196+0000 7f3020c00640 1 -- 172.21.15.188:0/1659504998 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f301c021060 con 0x7f3024160770 2024-01-16T14:56:49.196 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:49.214+0000 7f0fb5c30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:49.215 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.233+0000 7f03e761b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:56:49.251 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:49.268+0000 7f0fb5c30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:56:49.257 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.274+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ea3eb329-b238-48a3-a840-536e0547c04f", "id": 1} v 0) v1 -- 0x7f3024067b60 con 0x7f3024160770 2024-01-16T14:56:49.264 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.282+0000 7f3020c00640 1 -- 172.21.15.188:0/1659504998 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea3eb329-b238-48a3-a840-536e0547c04f", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f301c018070 con 0x7f3024160770 2024-01-16T14:56:49.264 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3024160770 msgr2=0x7f302414e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3024160770 0x7f302414e110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f301c00c6c0 tx=0x7f301c00cb80 comp rx=0 tx=0).stop 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 shutdown_connections 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3024160770 0x7f302414e110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30240714a0 0x7f302414dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 --2- 172.21.15.188:0/1659504998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f302406e5a0 0x7f3024158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 >> 172.21.15.188:0/1659504998 conn(0x7f30240b15b0 msgr2=0x7f30240b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 shutdown_connections 2024-01-16T14:56:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.283+0000 7f302a600640 1 -- 172.21.15.188:0/1659504998 wait complete. 2024-01-16T14:56:49.284 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 37ae5e71-3489-4f67-b32f-cd0a206f83e1 2 2024-01-16T14:56:49.294 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.312+0000 7f03e761b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:49.315 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:49.332+0000 7f0fb5c30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:56:49.347 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.365+0000 7f03e761b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:56:49.375 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:49.392+0000 7f0fb5c30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:56:49.407 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.425+0000 7f03e761b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:56:49.461 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.479+0000 7f03e761b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:56:49.502 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.519+0000 7f1cfb200640 1 Processor -- start 2024-01-16T14:56:49.502 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.519+0000 7f1cfb200640 1 -- start start 2024-01-16T14:56:49.504 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cfb200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1cfc15d4b0 0x7f1cfc0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.504 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cfb200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1cfc0cf9a0 0x7f1cfc0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.504 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cfb200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc0d0320 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.504 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cfb200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cfc0b2730 con 0x7f1cfc0cf9a0 2024-01-16T14:56:49.504 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cfb200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cfc0d1ec0 con 0x7f1cfc15d4b0 2024-01-16T14:56:49.504 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cf9e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1cfc15d4b0 0x7f1cfc0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.504 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cf9e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1cfc15d4b0 0x7f1cfc0d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:42676/0 (socket says 172.21.15.188:42676) 2024-01-16T14:56:49.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cf9e00640 1 -- 172.21.15.188:0/1057763170 learned_addr learned my addr 172.21.15.188:0/1057763170 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:49.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.521+0000 7f1cf9400640 1 --2- 172.21.15.188:0/1057763170 >> v2:172.21.15.141:3300/0 conn(0x7f1cfc0cf9a0 0x7f1cfc0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.522+0000 7f1cf9e00640 1 -- 172.21.15.188:0/1057763170 >> v2:172.21.15.141:3300/0 conn(0x7f1cfc0cf9a0 msgr2=0x7f1cfc0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:49.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.522+0000 7f1cf9e00640 1 --2- 172.21.15.188:0/1057763170 >> v2:172.21.15.141:3300/0 conn(0x7f1cfc0cf9a0 0x7f1cfc0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.522+0000 7f1cf9e00640 1 -- 172.21.15.188:0/1057763170 >> v1:172.21.15.188:6789/0 conn(0x7f1cfc0cddf0 legacy=0x7f1cfc0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.522+0000 7f1cf9e00640 1 -- 172.21.15.188:0/1057763170 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc0b29d0 con 0x7f1cfc15d4b0 2024-01-16T14:56:49.505 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.522+0000 7f1cf9400640 1 --2- 172.21.15.188:0/1057763170 >> v2:172.21.15.141:3300/0 conn(0x7f1cfc0cf9a0 0x7f1cfc0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:49.506 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.523+0000 7f1cf9e00640 1 --2- 172.21.15.188:0/1057763170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 0x7f1cfc0d1550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc08b5b0 tx=0x7f1cec02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f20f73f3e0e0cf9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.506 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.523+0000 7f1cf8a00640 1 -- 172.21.15.188:0/1057763170 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec030860 con 0x7f1cfc15d4b0 2024-01-16T14:56:49.506 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.523+0000 7f1cf8a00640 1 -- 172.21.15.188:0/1057763170 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec0309c0 con 0x7f1cfc15d4b0 2024-01-16T14:56:49.506 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.523+0000 7f1cf8a00640 1 -- 172.21.15.188:0/1057763170 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec030d50 con 0x7f1cfc15d4b0 2024-01-16T14:56:49.506 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 -- 172.21.15.188:0/1057763170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 msgr2=0x7f1cfc0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.507 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 --2- 172.21.15.188:0/1057763170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 0x7f1cfc0d1550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc08b5b0 tx=0x7f1cec02b9a0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.507 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 -- 172.21.15.188:0/1057763170 shutdown_connections 2024-01-16T14:56:49.507 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 --2- 172.21.15.188:0/1057763170 >> v2:172.21.15.141:3300/0 conn(0x7f1cfc0cf9a0 0x7f1cfc0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.507 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 --2- 172.21.15.188:0/1057763170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 0x7f1cfc0d1550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.507 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 -- 172.21.15.188:0/1057763170 >> 172.21.15.188:0/1057763170 conn(0x7f1cfc0d3000 msgr2=0x7f1cfc0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.507 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 -- 172.21.15.188:0/1057763170 shutdown_connections 2024-01-16T14:56:49.507 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.524+0000 7f1cfb200640 1 -- 172.21.15.188:0/1057763170 wait complete. 2024-01-16T14:56:49.508 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.525+0000 7f1cfb200640 1 Processor -- start 2024-01-16T14:56:49.508 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.526+0000 7f1cfb200640 1 -- start start 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.526+0000 7f1cfb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1cfc0cddf0 0x7f1cfc15a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.526+0000 7f1cfb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cfc0cf9a0 0x7f1cfc0a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.526+0000 7f1cfb200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 0x7f1cfc1f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.526+0000 7f1cfb200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc15ac70 con 0x7f1cfc0cf9a0 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.526+0000 7f1cfb200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc0a0b40 con 0x7f1cfc15d4b0 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.526+0000 7f1cfb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc1f3870 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1cfc0cddf0 0x7f1cfc15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cfa800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 0x7f1cfc1f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1cfc0cddf0 0x7f1cfc15a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.188:41758/0 (socket says 172.21.15.188:41758) 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 -- 172.21.15.188:0/778847361 learned_addr learned my addr 172.21.15.188:0/778847361 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9400640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cfc0cf9a0 0x7f1cfc0a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.511 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 -- 172.21.15.188:0/778847361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 msgr2=0x7f1cfc1f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 0x7f1cfc1f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 -- 172.21.15.188:0/778847361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cfc0cf9a0 msgr2=0x7f1cfc0a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cfc0cf9a0 0x7f1cfc0a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 -- 172.21.15.188:0/778847361 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0094d0 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.527+0000 7f1cf9e00640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1cfc0cddf0 0x7f1cfc15a730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1cec000c00 tx=0x7f1cec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.528+0000 7f1cf2000640 1 -- 172.21.15.188:0/778847361 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec030830 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.528+0000 7f1cf2000640 1 -- 172.21.15.188:0/778847361 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec039d90 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.528+0000 7f1cf2000640 1 -- 172.21.15.188:0/778847361 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec043630 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.528+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cfc15da10 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.528+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cfc09f840 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.528+0000 7f1cf2000640 1 -- 172.21.15.188:0/778847361 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1cec043a90 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.528+0000 7f1cf2000640 1 -- 172.21.15.188:0/778847361 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f1cec02f7d0 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.529+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cfc067450 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.531+0000 7f1cf2000640 1 -- 172.21.15.188:0/778847361 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1cec048070 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.591 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.609+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd new", "uuid": "37ae5e71-3489-4f67-b32f-cd0a206f83e1", "id": 2} v 0) v1 -- 0x7f1cfc0b3c80 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.618+0000 7f1cf2000640 1 -- 172.21.15.188:0/778847361 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37ae5e71-3489-4f67-b32f-cd0a206f83e1", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1cec034070 con 0x7f1cfc0cddf0 2024-01-16T14:56:49.600 INFO:teuthology.orchestra.run.smithi188.stdout:2 2024-01-16T14:56:49.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1cfc0cddf0 msgr2=0x7f1cfc15a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1cfc0cddf0 0x7f1cfc15a730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1cec000c00 tx=0x7f1cec004730 comp rx=0 tx=0).stop 2024-01-16T14:56:49.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 shutdown_connections 2024-01-16T14:56:49.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc15d4b0 0x7f1cfc1f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cfc0cf9a0 0x7f1cfc0a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 --2- 172.21.15.188:0/778847361 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1cfc0cddf0 0x7f1cfc15a730 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 >> 172.21.15.188:0/778847361 conn(0x7f1cfc0d3000 msgr2=0x7f1cfc0b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 shutdown_connections 2024-01-16T14:56:49.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.619+0000 7f1cfb200640 1 -- 172.21.15.188:0/778847361 wait complete. 2024-01-16T14:56:49.617 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new e34bedbd-6cbd-4918-a1ce-ce2aa90b884b 3 2024-01-16T14:56:49.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.817+0000 7fc612000640 1 Processor -- start 2024-01-16T14:56:49.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.818+0000 7fc612000640 1 -- start start 2024-01-16T14:56:49.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc612000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc60c0ce8f0 0x7fc60c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc612000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c0d0de0 con 0x7fc60c0b2a00 2024-01-16T14:56:49.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc612000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c0d10c0 con 0x7fc60c160f80 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc612000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60c0d13a0 con 0x7fc60c0ce8f0 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc603e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc60c0ce8f0 0x7fc60c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc603e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc60c0ce8f0 0x7fc60c0ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:42694/0 (socket says 172.21.15.188:42694) 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc603e00640 1 -- 172.21.15.188:0/3443910907 learned_addr learned my addr 172.21.15.188:0/3443910907 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.819+0000 7fc60b400640 1 --1- 172.21.15.188:0/3443910907 >> v1:172.21.15.188:6789/0 conn(0x7fc60c160f80 0x7fc60c161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:56956/0 (socket says 172.21.15.188:56956) 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.820+0000 7fc603e00640 1 -- 172.21.15.188:0/3443910907 >> v1:172.21.15.188:6789/0 conn(0x7fc60c160f80 legacy=0x7fc60c161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.820+0000 7fc603e00640 1 -- 172.21.15.188:0/3443910907 >> v1:172.21.15.141:6789/0 conn(0x7fc60c0b2a00 legacy=0x7fc60c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:49.803 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.820+0000 7fc603e00640 1 -- 172.21.15.188:0/3443910907 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c0d1680 con 0x7fc60c0ce8f0 2024-01-16T14:56:49.804 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.821+0000 7fc603e00640 1 --2- 172.21.15.188:0/3443910907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0ce8f0 0x7fc60c0ced30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009870 tx=0x7fc5f402b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31fb1a857128b1a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.804 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.821+0000 7fc60aa00640 1 -- 172.21.15.188:0/3443910907 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4030850 con 0x7fc60c0ce8f0 2024-01-16T14:56:49.804 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.821+0000 7fc60aa00640 1 -- 172.21.15.188:0/3443910907 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f40309b0 con 0x7fc60c0ce8f0 2024-01-16T14:56:49.804 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.821+0000 7fc60aa00640 1 -- 172.21.15.188:0/3443910907 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4030d40 con 0x7fc60c0ce8f0 2024-01-16T14:56:49.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.822+0000 7fc612000640 1 -- 172.21.15.188:0/3443910907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0ce8f0 msgr2=0x7fc60c0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.822+0000 7fc612000640 1 --2- 172.21.15.188:0/3443910907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0ce8f0 0x7fc60c0ced30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009870 tx=0x7fc5f402b990 comp rx=0 tx=0).stop 2024-01-16T14:56:49.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.822+0000 7fc612000640 1 -- 172.21.15.188:0/3443910907 shutdown_connections 2024-01-16T14:56:49.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.822+0000 7fc612000640 1 --2- 172.21.15.188:0/3443910907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0ce8f0 0x7fc60c0ced30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.822+0000 7fc612000640 1 -- 172.21.15.188:0/3443910907 >> 172.21.15.188:0/3443910907 conn(0x7fc60c09c550 msgr2=0x7fc60c0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.822+0000 7fc612000640 1 -- 172.21.15.188:0/3443910907 shutdown_connections 2024-01-16T14:56:49.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.822+0000 7fc612000640 1 -- 172.21.15.188:0/3443910907 wait complete. 2024-01-16T14:56:49.806 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.824+0000 7fc612000640 1 Processor -- start 2024-01-16T14:56:49.806 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.824+0000 7fc612000640 1 -- start start 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc612000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0b2a00 0x7fc60c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc612000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc60c0ce8f0 0x7fc60c161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc612000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc60c160f80 0x7fc60c1618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc612000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c06b710 con 0x7fc60c0ce8f0 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc612000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c0c7750 con 0x7fc60c0b2a00 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc612000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c0c7a10 con 0x7fc60c160f80 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc60be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc60c160f80 0x7fc60c1618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.809 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc60b400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0b2a00 0x7fc60c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc60be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc60c160f80 0x7fc60c1618a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.188:41762/0 (socket says 172.21.15.188:41762) 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.825+0000 7fc60be00640 1 -- 172.21.15.188:0/344980977 learned_addr learned my addr 172.21.15.188:0/344980977 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc603e00640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc60c0ce8f0 0x7fc60c161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc60be00640 1 -- 172.21.15.188:0/344980977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0b2a00 msgr2=0x7fc60c15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc60be00640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0b2a00 0x7fc60c15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc60be00640 1 -- 172.21.15.188:0/344980977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc60c0ce8f0 msgr2=0x7fc60c161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc60be00640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc60c0ce8f0 0x7fc60c161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc60be00640 1 -- 172.21.15.188:0/344980977 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f40094d0 con 0x7fc60c160f80 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc60be00640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc60c160f80 0x7fc60c1618a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c780 tx=0x7fc60400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc603400640 1 -- 172.21.15.188:0/344980977 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604004b60 con 0x7fc60c160f80 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc603400640 1 -- 172.21.15.188:0/344980977 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60400ce90 con 0x7fc60c160f80 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc603400640 1 -- 172.21.15.188:0/344980977 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60400a850 con 0x7fc60c160f80 2024-01-16T14:56:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c0c7cf0 con 0x7fc60c160f80 2024-01-16T14:56:49.811 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.826+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c0c82b0 con 0x7fc60c160f80 2024-01-16T14:56:49.811 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.827+0000 7fc603400640 1 -- 172.21.15.188:0/344980977 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc60400a9b0 con 0x7fc60c160f80 2024-01-16T14:56:49.811 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.827+0000 7fc603400640 1 -- 172.21.15.188:0/344980977 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fc604005030 con 0x7fc60c160f80 2024-01-16T14:56:49.811 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.828+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c0c84a0 con 0x7fc60c160f80 2024-01-16T14:56:49.812 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.830+0000 7fc603400640 1 -- 172.21.15.188:0/344980977 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc604006e50 con 0x7fc60c160f80 2024-01-16T14:56:49.847 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:49.864+0000 7f0fb5c30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:56:49.891 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.908+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e34bedbd-6cbd-4918-a1ce-ce2aa90b884b", "id": 3} v 0) v1 -- 0x7fc60c0675d0 con 0x7fc60c160f80 2024-01-16T14:56:49.900 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.918+0000 7fc603400640 1 -- 172.21.15.188:0/344980977 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e34bedbd-6cbd-4918-a1ce-ce2aa90b884b", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc60400ad30 con 0x7fc60c160f80 2024-01-16T14:56:49.900 INFO:teuthology.orchestra.run.smithi188.stdout:3 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc60c160f80 msgr2=0x7fc60c1618a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc60c160f80 0x7fc60c1618a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c780 tx=0x7fc60400cc40 comp rx=0 tx=0).stop 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 shutdown_connections 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc60c160f80 0x7fc60c1618a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc60c0ce8f0 0x7fc60c161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 --2- 172.21.15.188:0/344980977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc60c0b2a00 0x7fc60c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 >> 172.21.15.188:0/344980977 conn(0x7fc60c09c550 msgr2=0x7fc60c0b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 shutdown_connections 2024-01-16T14:56:49.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:49.919+0000 7fc612000640 1 -- 172.21.15.188:0/344980977 wait complete. 2024-01-16T14:56:49.907 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:49.925+0000 7f03e761b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:56:49.918 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new b9728641-c62f-4941-bd36-4038df6d3246 4 2024-01-16T14:56:50.014 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.032+0000 7f0fb5c30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:56:50.066 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.084+0000 7f03e761b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:56:50.103 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.121+0000 7f55f8c00640 1 Processor -- start 2024-01-16T14:56:50.104 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.121+0000 7f55f8c00640 1 -- start start 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f8c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f55f406b580 0x7f55f406b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f8c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f55f415acf0 0x7f55f40cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f8c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f415b5e0 con 0x7f55f415e840 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f8c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f406bea0 con 0x7f55f406b580 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f8c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f415b300 con 0x7f55f415acf0 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f3e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f55f415acf0 0x7f55f40cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f3e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f55f415acf0 0x7f55f40cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:42702/0 (socket says 172.21.15.188:42702) 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f3e00640 1 -- 172.21.15.188:0/108178679 learned_addr learned my addr 172.21.15.188:0/108178679 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:50.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f3400640 1 --2- 172.21.15.188:0/108178679 >> v2:172.21.15.141:3300/0 conn(0x7f55f406b580 0x7f55f406b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f3e00640 1 -- 172.21.15.188:0/108178679 >> v2:172.21.15.141:3300/0 conn(0x7f55f406b580 msgr2=0x7f55f406b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f3e00640 1 --2- 172.21.15.188:0/108178679 >> v2:172.21.15.141:3300/0 conn(0x7f55f406b580 0x7f55f406b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.123+0000 7f55f3e00640 1 -- 172.21.15.188:0/108178679 >> v1:172.21.15.188:6789/0 conn(0x7f55f415e840 legacy=0x7f55f415a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.124+0000 7f55f3e00640 1 -- 172.21.15.188:0/108178679 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f415dd60 con 0x7f55f415acf0 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.124+0000 7f55f3400640 1 --2- 172.21.15.188:0/108178679 >> v2:172.21.15.141:3300/0 conn(0x7f55f406b580 0x7f55f406b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.125+0000 7f55f3e00640 1 --2- 172.21.15.188:0/108178679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415acf0 0x7f55f40cdc50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f55f406f240 tx=0x7f55ec02bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=632370cfcafa786a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.125+0000 7f55f2000640 1 -- 172.21.15.188:0/108178679 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec02cb20 con 0x7f55f415acf0 2024-01-16T14:56:50.107 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.125+0000 7f55f2000640 1 -- 172.21.15.188:0/108178679 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec02cc80 con 0x7f55f415acf0 2024-01-16T14:56:50.108 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.125+0000 7f55f2000640 1 -- 172.21.15.188:0/108178679 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec0343f0 con 0x7f55f415acf0 2024-01-16T14:56:50.108 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.125+0000 7f55f8c00640 1 -- 172.21.15.188:0/108178679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415acf0 msgr2=0x7f55f40cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.109 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.125+0000 7f55f8c00640 1 --2- 172.21.15.188:0/108178679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415acf0 0x7f55f40cdc50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f55f406f240 tx=0x7f55ec02bf80 comp rx=0 tx=0).stop 2024-01-16T14:56:50.109 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.126+0000 7f55f8c00640 1 -- 172.21.15.188:0/108178679 shutdown_connections 2024-01-16T14:56:50.109 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.126+0000 7f55f8c00640 1 --2- 172.21.15.188:0/108178679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415acf0 0x7f55f40cdc50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.109 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.126+0000 7f55f8c00640 1 --2- 172.21.15.188:0/108178679 >> v2:172.21.15.141:3300/0 conn(0x7f55f406b580 0x7f55f406b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.109 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.126+0000 7f55f8c00640 1 -- 172.21.15.188:0/108178679 >> 172.21.15.188:0/108178679 conn(0x7f55f40b1ba0 msgr2=0x7f55f40b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.109 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.126+0000 7f55f8c00640 1 -- 172.21.15.188:0/108178679 shutdown_connections 2024-01-16T14:56:50.109 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.126+0000 7f55f8c00640 1 -- 172.21.15.188:0/108178679 wait complete. 2024-01-16T14:56:50.110 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.127+0000 7f55f8c00640 1 Processor -- start 2024-01-16T14:56:50.110 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.127+0000 7f55f8c00640 1 -- start start 2024-01-16T14:56:50.112 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.128+0000 7f55f8c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55f406b580 0x7f55f4099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.112 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.128+0000 7f55f8c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55f415acf0 0x7f55f4158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.128+0000 7f55f8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415e840 0x7f55f415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.128+0000 7f55f8c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f40ada50 con 0x7f55f406b580 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.128+0000 7f55f8c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f4158f80 con 0x7f55f415e840 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.128+0000 7f55f8c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f40ace40 con 0x7f55f415acf0 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.128+0000 7f55f2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55f415acf0 0x7f55f4158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415e840 0x7f55f415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55f415acf0 0x7f55f4158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.188:41764/0 (socket says 172.21.15.188:41764) 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 -- 172.21.15.188:0/95294284 learned_addr learned my addr 172.21.15.188:0/95294284 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f3400640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55f406b580 0x7f55f4099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 -- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415e840 msgr2=0x7f55f415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415e840 0x7f55f415c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 -- 172.21.15.188:0/95294284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55f406b580 msgr2=0x7f55f4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55f406b580 0x7f55f4099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 -- 172.21.15.188:0/95294284 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec009b30 con 0x7f55f415acf0 2024-01-16T14:56:50.113 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f3e00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415e840 0x7f55f415c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.129+0000 7f55f2a00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55f415acf0 0x7f55f4158a40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00c400 tx=0x7f55dc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.130+0000 7f55eb400640 1 -- 172.21.15.188:0/95294284 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc0052d0 con 0x7f55f415acf0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.130+0000 7f55eb400640 1 -- 172.21.15.188:0/95294284 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc005430 con 0x7f55f415acf0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.130+0000 7f55eb400640 1 -- 172.21.15.188:0/95294284 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc003830 con 0x7f55f415acf0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.130+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f40ad120 con 0x7f55f415acf0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.130+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f40ad6e0 con 0x7f55f415acf0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.130+0000 7f55eb400640 1 -- 172.21.15.188:0/95294284 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f55dc003c90 con 0x7f55f415acf0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.130+0000 7f55eb400640 1 -- 172.21.15.188:0/95294284 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f55dc011aa0 con 0x7f55f415acf0 2024-01-16T14:56:50.114 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.131+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f40adbc0 con 0x7f55f415acf0 2024-01-16T14:56:50.115 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.133+0000 7f55eb400640 1 -- 172.21.15.188:0/95294284 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f55dc017070 con 0x7f55f415acf0 2024-01-16T14:56:50.161 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.179+0000 7f0fb5c30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:56:50.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.211+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b9728641-c62f-4941-bd36-4038df6d3246", "id": 4} v 0) v1 -- 0x7f55f4067b60 con 0x7f55f415acf0 2024-01-16T14:56:50.201 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.218+0000 7f55eb400640 1 -- 172.21.15.188:0/95294284 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b9728641-c62f-4941-bd36-4038df6d3246", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f55dc011590 con 0x7f55f415acf0 2024-01-16T14:56:50.201 INFO:teuthology.orchestra.run.smithi188.stdout:4 2024-01-16T14:56:50.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.219+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55f415acf0 msgr2=0x7f55f4158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.219+0000 7f55f8c00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55f415acf0 0x7f55f4158a40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00c400 tx=0x7f55dc00c8c0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.220+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 shutdown_connections 2024-01-16T14:56:50.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.220+0000 7f55f8c00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55f415e840 0x7f55f415c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.220+0000 7f55f8c00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55f415acf0 0x7f55f4158a40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.203 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.220+0000 7f55f8c00640 1 --2- 172.21.15.188:0/95294284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55f406b580 0x7f55f4099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.203 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.220+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 >> 172.21.15.188:0/95294284 conn(0x7f55f40b1ba0 msgr2=0x7f55f40b05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.203 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.220+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 shutdown_connections 2024-01-16T14:56:50.203 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.220+0000 7f55f8c00640 1 -- 172.21.15.188:0/95294284 wait complete. 2024-01-16T14:56:50.207 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.225+0000 7f03e761b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:56:50.217 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 51fe9078-c8a7-4bbb-bc65-7db69db669b4 5 2024-01-16T14:56:50.218 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.236+0000 7f0fb5c30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:56:50.259 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.277+0000 7f03e761b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:56:50.333 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.350+0000 7f0fb5c30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:50.368 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.386+0000 7f03e761b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:50.395 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.412+0000 7f0e30000640 1 Processor -- start 2024-01-16T14:56:50.395 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.413+0000 7f0e30000640 1 -- start start 2024-01-16T14:56:50.397 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.414+0000 7f0e30000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0e280ced30 0x7f0e280b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.397 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.414+0000 7f0e30000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e280cf1c0 con 0x7f0e28160f80 2024-01-16T14:56:50.397 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.414+0000 7f0e30000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e280d0de0 con 0x7f0e280ce8f0 2024-01-16T14:56:50.397 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.414+0000 7f0e30000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e280d10c0 con 0x7f0e280ced30 2024-01-16T14:56:50.397 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.414+0000 7f0e2cc00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f0e280ce8f0 0x7f0e280b2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:56966/0 (socket says 172.21.15.188:56966) 2024-01-16T14:56:50.397 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.414+0000 7f0e2cc00640 1 -- 172.21.15.188:0/3113047646 learned_addr learned my addr 172.21.15.188:0/3113047646 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-01-16T14:56:50.399 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e2e000640 1 --2- 172.21.15.188:0/3113047646 >> v2:172.21.15.141:3300/0 conn(0x7f0e280ced30 0x7f0e280b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.399 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822758078 0 0) 0x7f0e280d0de0 con 0x7f0e280ce8f0 2024-01-16T14:56:50.399 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e08003520 con 0x7f0e280ce8f0 2024-01-16T14:56:50.399 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 617250997 0 0) 0x7f0e08003520 con 0x7f0e280ce8f0 2024-01-16T14:56:50.399 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 >> v2:172.21.15.141:3300/0 conn(0x7f0e280ced30 msgr2=0x7f0e280b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.399 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 --2- 172.21.15.188:0/3113047646 >> v2:172.21.15.141:3300/0 conn(0x7f0e280ced30 0x7f0e280b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 >> v1:172.21.15.141:6789/0 conn(0x7f0e28160f80 legacy=0x7f0e28161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280d13a0 con 0x7f0e280ce8f0 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0e10001670 con 0x7f0e280ce8f0 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 >> v1:172.21.15.188:6789/0 conn(0x7f0e280ce8f0 legacy=0x7f0e280b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e080047c0 0x7f0e08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08005180 0x7f0e0800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e280d10c0 con 0x7f0e08005180 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0800c020 con 0x7f0e08003970 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0800c1e0 con 0x7f0e080047c0 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.415+0000 7f0e2d600640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2cc00640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e080047c0 0x7f0e08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.400 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2e000640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08005180 0x7f0e0800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2d600640 1 -- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e080047c0 msgr2=0x7f0e08004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2d600640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e080047c0 0x7f0e08004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2d600640 1 -- 172.21.15.188:0/3113047646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08005180 msgr2=0x7f0e0800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2d600640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08005180 0x7f0e0800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2d600640 1 -- 172.21.15.188:0/3113047646 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08003520 con 0x7f0e08003970 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2e000640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08005180 0x7f0e0800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.416+0000 7f0e2e000640 1 -- 172.21.15.188:0/3113047646 reap_dead start 2024-01-16T14:56:50.401 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.417+0000 7f0e2d600640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e08003d90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0e180015b0 tx=0x7f0e080057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=506e9400d5b4a464 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.402 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.417+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e180052d0 con 0x7f0e08003970 2024-01-16T14:56:50.402 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.417+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1802e050 con 0x7f0e08003970 2024-01-16T14:56:50.402 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.417+0000 7f0e27e00640 1 -- 172.21.15.188:0/3113047646 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18003830 con 0x7f0e08003970 2024-01-16T14:56:50.402 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.418+0000 7f0e30000640 1 -- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 msgr2=0x7f0e08003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.402 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.418+0000 7f0e30000640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e08003d90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0e180015b0 tx=0x7f0e080057f0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.418+0000 7f0e30000640 1 -- 172.21.15.188:0/3113047646 shutdown_connections 2024-01-16T14:56:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.418+0000 7f0e30000640 1 --2- 172.21.15.188:0/3113047646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e08003d90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.418+0000 7f0e30000640 1 -- 172.21.15.188:0/3113047646 >> 172.21.15.188:0/3113047646 conn(0x7f0e2809c550 msgr2=0x7f0e280d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.418+0000 7f0e30000640 1 -- 172.21.15.188:0/3113047646 shutdown_connections 2024-01-16T14:56:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.418+0000 7f0e30000640 1 -- 172.21.15.188:0/3113047646 wait complete. 2024-01-16T14:56:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.420+0000 7f0e30000640 1 Processor -- start 2024-01-16T14:56:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.420+0000 7f0e30000640 1 -- start start 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e30000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e280b6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e30000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e2814a2f0 0x7f0e2814c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e30000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e2814d590 0x7f0e2814cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e30000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2815d6c0 con 0x7f0e2814d590 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e30000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2815fe40 con 0x7f0e08003970 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e30000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e28160120 con 0x7f0e2814a2f0 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e2d600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e280b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e2cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e2814a2f0 0x7f0e2814c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e2d600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e280b6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:42726/0 (socket says 172.21.15.188:42726) 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e2d600640 1 -- 172.21.15.188:0/2117086516 learned_addr learned my addr 172.21.15.188:0/2117086516 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e2e000640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e2814d590 0x7f0e2814cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.421+0000 7f0e2d600640 1 -- 172.21.15.188:0/2117086516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e2814a2f0 msgr2=0x7f0e2814c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e2d600640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e2814a2f0 0x7f0e2814c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e2d600640 1 -- 172.21.15.188:0/2117086516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e2814d590 msgr2=0x7f0e2814cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e2d600640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e2814d590 0x7f0e2814cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e2d600640 1 -- 172.21.15.188:0/2117086516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28160400 con 0x7f0e08003970 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e2d600640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e280b6060 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0e1802ba60 tx=0x7f0e18001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e25600640 1 -- 172.21.15.188:0/2117086516 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e180076c0 con 0x7f0e08003970 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e25600640 1 -- 172.21.15.188:0/2117086516 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e18007a30 con 0x7f0e08003970 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e25600640 1 -- 172.21.15.188:0/2117086516 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18006be0 con 0x7f0e08003970 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2815d910 con 0x7f0e08003970 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.422+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2815de30 con 0x7f0e08003970 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.423+0000 7f0e25600640 1 -- 172.21.15.188:0/2117086516 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0e180038f0 con 0x7f0e08003970 2024-01-16T14:56:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.423+0000 7f0e25600640 1 -- 172.21.15.188:0/2117086516 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f0e1803dcc0 con 0x7f0e08003970 2024-01-16T14:56:50.407 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.423+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e28161cf0 con 0x7f0e08003970 2024-01-16T14:56:50.410 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.428+0000 7f0e25600640 1 -- 172.21.15.188:0/2117086516 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0e18041020 con 0x7f0e08003970 2024-01-16T14:56:50.453 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.470+0000 7f0fb5c30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:56:50.478 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.496+0000 7f03e761b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:56:50.488 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.505+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd new", "uuid": "51fe9078-c8a7-4bbb-bc65-7db69db669b4", "id": 5} v 0) v1 -- 0x7f0e28160590 con 0x7f0e08003970 2024-01-16T14:56:50.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.513+0000 7f0e25600640 1 -- 172.21.15.188:0/2117086516 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "51fe9078-c8a7-4bbb-bc65-7db69db669b4", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0e280cf1c0 con 0x7f0e08003970 2024-01-16T14:56:50.496 INFO:teuthology.orchestra.run.smithi188.stdout:5 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 msgr2=0x7f0e280b6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e280b6060 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0e1802ba60 tx=0x7f0e18001120 comp rx=0 tx=0).stop 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 shutdown_connections 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e2814d590 0x7f0e2814cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e2814a2f0 0x7f0e2814c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 --2- 172.21.15.188:0/2117086516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08003970 0x7f0e280b6060 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 >> 172.21.15.188:0/2117086516 conn(0x7f0e2809c550 msgr2=0x7f0e280b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 shutdown_connections 2024-01-16T14:56:50.497 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.514+0000 7f0e30000640 1 -- 172.21.15.188:0/2117086516 wait complete. 2024-01-16T14:56:50.512 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 4634a709-f2f1-4554-ad94-1867ba2e2010 6 2024-01-16T14:56:50.624 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.641+0000 7f0fb5c30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:50.635 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.653+0000 7f03e761b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:56:50.675 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.692+0000 7f0fb5c30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:50.684 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.702+0000 7f03e761b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:56:50.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.720+0000 7f1846600640 1 Processor -- start 2024-01-16T14:56:50.703 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.720+0000 7f1846600640 1 -- start start 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.721+0000 7f1846600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f184015d4b0 0x7f18400d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.721+0000 7f1846600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18400b41e0 0x7f18400cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1846600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18400d0320 con 0x7f18400cf9a0 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1846600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18400b2730 con 0x7f18400b41e0 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1846600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18400d1ec0 con 0x7f184015d4b0 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f184015d4b0 0x7f18400d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f184015d4b0 0x7f18400d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.188:41782/0 (socket says 172.21.15.188:41782) 2024-01-16T14:56:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845200640 1 -- 172.21.15.188:0/2231845798 learned_addr learned my addr 172.21.15.188:0/2231845798 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:50.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845c00640 1 --2- 172.21.15.188:0/2231845798 >> v2:172.21.15.141:3300/0 conn(0x7f18400b41e0 0x7f18400cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845200640 1 -- 172.21.15.188:0/2231845798 >> v2:172.21.15.141:3300/0 conn(0x7f18400b41e0 msgr2=0x7f18400cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845200640 1 --2- 172.21.15.188:0/2231845798 >> v2:172.21.15.141:3300/0 conn(0x7f18400b41e0 0x7f18400cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845200640 1 -- 172.21.15.188:0/2231845798 >> v1:172.21.15.141:6789/0 conn(0x7f18400cf9a0 legacy=0x7f18400cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:50.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.722+0000 7f1845200640 1 -- 172.21.15.188:0/2231845798 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18400b29d0 con 0x7f184015d4b0 2024-01-16T14:56:50.707 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.723+0000 7f1845200640 1 --2- 172.21.15.188:0/2231845798 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f184015d4b0 0x7f18400d1550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18300098a0 tx=0x7f183002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92bf51b2521f5dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.707 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f183f400640 1 -- 172.21.15.188:0/2231845798 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830031640 con 0x7f184015d4b0 2024-01-16T14:56:50.707 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f183f400640 1 -- 172.21.15.188:0/2231845798 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18300317a0 con 0x7f184015d4b0 2024-01-16T14:56:50.707 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f183f400640 1 -- 172.21.15.188:0/2231845798 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830031b30 con 0x7f184015d4b0 2024-01-16T14:56:50.707 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f1846600640 1 -- 172.21.15.188:0/2231845798 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f184015d4b0 msgr2=0x7f18400d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.707 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f1846600640 1 --2- 172.21.15.188:0/2231845798 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f184015d4b0 0x7f18400d1550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18300098a0 tx=0x7f183002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.707 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f1846600640 1 -- 172.21.15.188:0/2231845798 shutdown_connections 2024-01-16T14:56:50.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f1846600640 1 --2- 172.21.15.188:0/2231845798 >> v2:172.21.15.141:3300/0 conn(0x7f18400b41e0 0x7f18400cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f1846600640 1 --2- 172.21.15.188:0/2231845798 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f184015d4b0 0x7f18400d1550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f1846600640 1 -- 172.21.15.188:0/2231845798 >> 172.21.15.188:0/2231845798 conn(0x7f18400d3000 msgr2=0x7f18400d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.724+0000 7f1846600640 1 -- 172.21.15.188:0/2231845798 shutdown_connections 2024-01-16T14:56:50.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.725+0000 7f1846600640 1 -- 172.21.15.188:0/2231845798 wait complete. 2024-01-16T14:56:50.708 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.726+0000 7f1846600640 1 Processor -- start 2024-01-16T14:56:50.709 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.726+0000 7f1846600640 1 -- start start 2024-01-16T14:56:50.711 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.727+0000 7f1846600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18400b41e0 0x7f1840160090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.711 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.727+0000 7f1846600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18400cf9a0 0x7f1840158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.711 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.727+0000 7f1846600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f184015d4b0 0x7f18400ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.727+0000 7f1846600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18401605d0 con 0x7f18400cf9a0 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.727+0000 7f1846600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18401586b0 con 0x7f184015d4b0 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.727+0000 7f1846600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1840158900 con 0x7f18400b41e0 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.727+0000 7f1845200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18400b41e0 0x7f1840160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f184015d4b0 0x7f18400ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18400b41e0 0x7f1840160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.188:41794/0 (socket says 172.21.15.188:41794) 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 -- 172.21.15.188:0/3290195259 learned_addr learned my addr 172.21.15.188:0/3290195259 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f183fe00640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18400cf9a0 0x7f1840158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 -- 172.21.15.188:0/3290195259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f184015d4b0 msgr2=0x7f18400ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f184015d4b0 0x7f18400ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 -- 172.21.15.188:0/3290195259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18400cf9a0 msgr2=0x7f1840158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18400cf9a0 0x7f1840158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 -- 172.21.15.188:0/3290195259 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18300094d0 con 0x7f18400b41e0 2024-01-16T14:56:50.712 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f1845200640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18400b41e0 0x7f1840160090 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1830000c00 tx=0x7f18300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f183cc00640 1 -- 172.21.15.188:0/3290195259 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183003ac60 con 0x7f18400b41e0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f183cc00640 1 -- 172.21.15.188:0/3290195259 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183003adc0 con 0x7f18400b41e0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.728+0000 7f183cc00640 1 -- 172.21.15.188:0/3290195259 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830044630 con 0x7f18400b41e0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.729+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184015da10 con 0x7f18400b41e0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.729+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1840158ec0 con 0x7f18400b41e0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.729+0000 7f183cc00640 1 -- 172.21.15.188:0/3290195259 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f183002f050 con 0x7f18400b41e0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.729+0000 7f183cc00640 1 -- 172.21.15.188:0/3290195259 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f1830044a90 con 0x7f18400b41e0 2024-01-16T14:56:50.713 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.730+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184015d060 con 0x7f18400b41e0 2024-01-16T14:56:50.714 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.732+0000 7f183cc00640 1 -- 172.21.15.188:0/3290195259 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1830035020 con 0x7f18400b41e0 2024-01-16T14:56:50.791 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.809+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4634a709-f2f1-4554-ad94-1867ba2e2010", "id": 6} v 0) v1 -- 0x7f18400d2cd0 con 0x7f18400b41e0 2024-01-16T14:56:50.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.817+0000 7f183cc00640 1 -- 172.21.15.188:0/3290195259 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4634a709-f2f1-4554-ad94-1867ba2e2010", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f183002d040 con 0x7f18400b41e0 2024-01-16T14:56:50.800 INFO:teuthology.orchestra.run.smithi188.stdout:6 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18400b41e0 msgr2=0x7f1840160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18400b41e0 0x7f1840160090 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1830000c00 tx=0x7f18300046c0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 shutdown_connections 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f184015d4b0 0x7f18400ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18400cf9a0 0x7f1840158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 --2- 172.21.15.188:0/3290195259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18400b41e0 0x7f1840160090 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 >> 172.21.15.188:0/3290195259 conn(0x7f18400d3000 msgr2=0x7f18400b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 shutdown_connections 2024-01-16T14:56:50.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:50.819+0000 7f1846600640 1 -- 172.21.15.188:0/3290195259 wait complete. 2024-01-16T14:56:50.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 26a9f08d-325f-432f-8e0b-bd2e0f18ac56 7 2024-01-16T14:56:50.843 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:50.861+0000 7f03e761b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:56:50.845 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:50.862+0000 7f0fb5c30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:56:51.008 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.025+0000 7f9e2dc00640 1 Processor -- start 2024-01-16T14:56:51.008 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.025+0000 7f9e2dc00640 1 -- start start 2024-01-16T14:56:51.009 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.026+0000 7f9e2dc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9e28160f80 0x7f9e28161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.026+0000 7f9e2dc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e281618a0 con 0x7f9e280ced50 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.026+0000 7f9e2dc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280d0a00 con 0x7f9e280ce8f0 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.026+0000 7f9e2dc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280cf1e0 con 0x7f9e28160f80 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.027+0000 7f9e26a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9e28160f80 0x7f9e28161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.027+0000 7f9e26a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9e28160f80 0x7f9e28161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:42738/0 (socket says 172.21.15.188:42738) 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.027+0000 7f9e26a00640 1 -- 172.21.15.188:0/3408744135 learned_addr learned my addr 172.21.15.188:0/3408744135 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.027+0000 7f9e26a00640 1 -- 172.21.15.188:0/3408744135 >> v1:172.21.15.188:6789/0 conn(0x7f9e280ce8f0 legacy=0x7f9e280cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.027+0000 7f9e26a00640 1 -- 172.21.15.188:0/3408744135 >> v1:172.21.15.141:6789/0 conn(0x7f9e280ced50 legacy=0x7f9e280d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:51.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.027+0000 7f9e26a00640 1 -- 172.21.15.188:0/3408744135 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280d0de0 con 0x7f9e28160f80 2024-01-16T14:56:51.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.028+0000 7f9e26a00640 1 --2- 172.21.15.188:0/3408744135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e28160f80 0x7f9e28161360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009870 tx=0x7f9e1802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd0abe15446313e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.028+0000 7f9e26000640 1 -- 172.21.15.188:0/3408744135 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18030860 con 0x7f9e28160f80 2024-01-16T14:56:51.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e26000640 1 -- 172.21.15.188:0/3408744135 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e180309c0 con 0x7f9e28160f80 2024-01-16T14:56:51.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e26000640 1 -- 172.21.15.188:0/3408744135 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18030d50 con 0x7f9e28160f80 2024-01-16T14:56:51.012 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e2dc00640 1 -- 172.21.15.188:0/3408744135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e28160f80 msgr2=0x7f9e28161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.012 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e2dc00640 1 --2- 172.21.15.188:0/3408744135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e28160f80 0x7f9e28161360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009870 tx=0x7f9e1802b9a0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.012 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e2dc00640 1 -- 172.21.15.188:0/3408744135 shutdown_connections 2024-01-16T14:56:51.012 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e2dc00640 1 --2- 172.21.15.188:0/3408744135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e28160f80 0x7f9e28161360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.012 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e2dc00640 1 -- 172.21.15.188:0/3408744135 >> 172.21.15.188:0/3408744135 conn(0x7f9e2809c550 msgr2=0x7f9e280d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.013 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e2dc00640 1 -- 172.21.15.188:0/3408744135 shutdown_connections 2024-01-16T14:56:51.013 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.029+0000 7f9e2dc00640 1 -- 172.21.15.188:0/3408744135 wait complete. 2024-01-16T14:56:51.013 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.031+0000 7f9e2dc00640 1 Processor -- start 2024-01-16T14:56:51.013 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.031+0000 7f9e2dc00640 1 -- start start 2024-01-16T14:56:51.016 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e2dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e280ce8f0 0x7f9e280b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.016 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e2dc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e280ced50 0x7f9e28152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.016 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e2dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e28160f80 0x7f9e280b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e2dc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280b8f50 con 0x7f9e280ce8f0 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e2dc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e28153d20 con 0x7f9e280ced50 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e2dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e28153230 con 0x7f9e28160f80 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e27400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e28160f80 0x7f9e280b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e1fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e280ced50 0x7f9e28152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e27400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e28160f80 0x7f9e280b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.188:41804/0 (socket says 172.21.15.188:41804) 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e27400640 1 -- 172.21.15.188:0/2448751710 learned_addr learned my addr 172.21.15.188:0/2448751710 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.032+0000 7f9e26a00640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e280ce8f0 0x7f9e280b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e27400640 1 -- 172.21.15.188:0/2448751710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e280ced50 msgr2=0x7f9e28152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e27400640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e280ced50 0x7f9e28152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e27400640 1 -- 172.21.15.188:0/2448751710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e280ce8f0 msgr2=0x7f9e280b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e27400640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e280ce8f0 0x7f9e280b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e27400640 1 -- 172.21.15.188:0/2448751710 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e180094d0 con 0x7f9e28160f80 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e26a00640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e280ce8f0 0x7f9e280b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e27400640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e28160f80 0x7f9e280b0fc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c7b0 tx=0x7f9e2000cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e1ea00640 1 -- 172.21.15.188:0/2448751710 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e20004b60 con 0x7f9e28160f80 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e1ea00640 1 -- 172.21.15.188:0/2448751710 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e20004cc0 con 0x7f9e28160f80 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e1ea00640 1 -- 172.21.15.188:0/2448751710 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000aa90 con 0x7f9e28160f80 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.033+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e280b8620 con 0x7f9e28160f80 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.034+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e280b8be0 con 0x7f9e28160f80 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.034+0000 7f9e1ea00640 1 -- 172.21.15.188:0/2448751710 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9e20006b50 con 0x7f9e28160f80 2024-01-16T14:56:51.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.034+0000 7f9e1ea00640 1 -- 172.21.15.188:0/2448751710 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9e20004e30 con 0x7f9e28160f80 2024-01-16T14:56:51.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.035+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e280675d0 con 0x7f9e28160f80 2024-01-16T14:56:51.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.037+0000 7f9e1ea00640 1 -- 172.21.15.188:0/2448751710 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9e20005030 con 0x7f9e28160f80 2024-01-16T14:56:51.097 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.114+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd new", "uuid": "26a9f08d-325f-432f-8e0b-bd2e0f18ac56", "id": 7} v 0) v1 -- 0x7f9e280d3920 con 0x7f9e28160f80 2024-01-16T14:56:51.104 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.121+0000 7f9e1ea00640 1 -- 172.21.15.188:0/2448751710 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26a9f08d-325f-432f-8e0b-bd2e0f18ac56", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9e200051d0 con 0x7f9e28160f80 2024-01-16T14:56:51.104 INFO:teuthology.orchestra.run.smithi188.stdout:7 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e28160f80 msgr2=0x7f9e280b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e28160f80 0x7f9e280b0fc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c7b0 tx=0x7f9e2000cc70 comp rx=0 tx=0).stop 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 shutdown_connections 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e28160f80 0x7f9e280b0fc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e280ced50 0x7f9e28152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 --2- 172.21.15.188:0/2448751710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e280ce8f0 0x7f9e280b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 >> 172.21.15.188:0/2448751710 conn(0x7f9e2809c550 msgr2=0x7f9e280b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 shutdown_connections 2024-01-16T14:56:51.106 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:56:51.123+0000 7f9e2dc00640 1 -- 172.21.15.188:0/2448751710 wait complete. 2024-01-16T14:56:51.121 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T14:56:51.121 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T14:56:51.129 INFO:tasks.ceph.osd.0:Started 2024-01-16T14:56:51.129 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T14:56:51.129 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T14:56:51.136 INFO:tasks.ceph.osd.1:Started 2024-01-16T14:56:51.137 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T14:56:51.137 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T14:56:51.143 INFO:tasks.ceph.osd.2:Started 2024-01-16T14:56:51.144 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-16T14:56:51.144 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-16T14:56:51.150 INFO:tasks.ceph.osd.3:Started 2024-01-16T14:56:51.150 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-16T14:56:51.150 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-16T14:56:51.168 INFO:tasks.ceph.osd.4:Started 2024-01-16T14:56:51.169 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-16T14:56:51.169 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-16T14:56:51.175 INFO:tasks.ceph.osd.5:Started 2024-01-16T14:56:51.175 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-16T14:56:51.176 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-16T14:56:51.180 INFO:tasks.ceph.osd.6:Started 2024-01-16T14:56:51.180 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-16T14:56:51.181 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-16T14:56:51.185 INFO:tasks.ceph.osd.7:Started 2024-01-16T14:56:51.185 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.424+0000 7ff599000640 1 Processor -- start 2024-01-16T14:56:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.424+0000 7ff599000640 1 -- start start 2024-01-16T14:56:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.425+0000 7ff599000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff59406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.425+0000 7ff599000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940cfa70 con 0x7ff594099a60 2024-01-16T14:56:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.425+0000 7ff599000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940cff00 con 0x7ff59415e880 2024-01-16T14:56:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.425+0000 7ff599000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59406fc70 con 0x7ff594099ea0 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff58be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff594099a60 0x7ff59406e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36250/0 (socket says 172.21.15.141:36250) 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff58be00640 1 -- 172.21.15.141:0/2745899973 learned_addr learned my addr 172.21.15.141:0/2745899973 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff592800640 1 --2- 172.21.15.141:0/2745899973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff59406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff591400640 1 -- 172.21.15.141:0/2745899973 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180725192 0 0) 0x7ff5940cff00 con 0x7ff59415e880 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff591400640 1 -- 172.21.15.141:0/2745899973 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff574003520 con 0x7ff59415e880 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff591400640 1 -- 172.21.15.141:0/2745899973 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901498382 0 0) 0x7ff5940cfa70 con 0x7ff594099a60 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff591400640 1 -- 172.21.15.141:0/2745899973 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5940cff00 con 0x7ff594099a60 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff592800640 1 -- 172.21.15.141:0/2745899973 >> v1:172.21.15.188:6789/0 conn(0x7ff59415e880 legacy=0x7ff59406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff592800640 1 -- 172.21.15.141:0/2745899973 >> v1:172.21.15.141:6789/0 conn(0x7ff594099a60 legacy=0x7ff59406e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.426+0000 7ff592800640 1 -- 172.21.15.141:0/2745899973 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5940cf420 con 0x7ff594099ea0 2024-01-16T14:56:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.427+0000 7ff592800640 1 --2- 172.21.15.141:0/2745899973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff59406eed0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009dc0 tx=0x7ff58c02bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d495e9cc2a17924 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff591400640 1 -- 172.21.15.141:0/2745899973 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c02cd60 con 0x7ff594099ea0 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff591400640 1 -- 172.21.15.141:0/2745899973 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c034440 con 0x7ff594099ea0 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff591400640 1 -- 172.21.15.141:0/2745899973 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0347d0 con 0x7ff594099ea0 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff599000640 1 -- 172.21.15.141:0/2745899973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 msgr2=0x7ff59406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff599000640 1 --2- 172.21.15.141:0/2745899973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff59406eed0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009dc0 tx=0x7ff58c02bfb0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff599000640 1 -- 172.21.15.141:0/2745899973 shutdown_connections 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff599000640 1 --2- 172.21.15.141:0/2745899973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff59406eed0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff599000640 1 -- 172.21.15.141:0/2745899973 >> 172.21.15.141:0/2745899973 conn(0x7ff5940b2d00 msgr2=0x7ff5940b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.428+0000 7ff599000640 1 -- 172.21.15.141:0/2745899973 shutdown_connections 2024-01-16T14:56:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.429+0000 7ff599000640 1 -- 172.21.15.141:0/2745899973 wait complete. 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.430+0000 7ff599000640 1 Processor -- start 2024-01-16T14:56:51.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.430+0000 7ff599000640 1 -- start start 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff599000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff594099a60 0x7ff59409e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff599000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff5940c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff599000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff59415e880 0x7ff5940c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff599000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5940b4f30 con 0x7ff59415e880 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff599000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5940cad90 con 0x7ff594099a60 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff599000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59409ed30 con 0x7ff594099ea0 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff592800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff59415e880 0x7ff5940c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff592800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff59415e880 0x7ff5940c7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41464/0 (socket says 172.21.15.141:41464) 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff592800640 1 -- 172.21.15.141:0/487604083 learned_addr learned my addr 172.21.15.141:0/487604083 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff591e00640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff594099a60 0x7ff59409e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.431+0000 7ff58be00640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff5940c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:51.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff591e00640 1 -- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 msgr2=0x7ff5940c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff591e00640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff5940c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff591e00640 1 -- 172.21.15.141:0/487604083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff59415e880 msgr2=0x7ff5940c7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff591e00640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff59415e880 0x7ff5940c7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff591e00640 1 -- 172.21.15.141:0/487604083 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c009a70 con 0x7ff594099a60 2024-01-16T14:56:51.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff592800640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff59415e880 0x7ff5940c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:56:51.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff591e00640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff594099a60 0x7ff59409e7f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff584001eb0 tx=0x7ff58400c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff58a000640 1 -- 172.21.15.141:0/487604083 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5840052d0 con 0x7ff594099a60 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff58a000640 1 -- 172.21.15.141:0/487604083 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff584005430 con 0x7ff594099a60 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.432+0000 7ff58a000640 1 -- 172.21.15.141:0/487604083 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584003830 con 0x7ff594099a60 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.433+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5940c7550 con 0x7ff594099a60 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.433+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5940b4870 con 0x7ff594099a60 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.433+0000 7ff58a000640 1 -- 172.21.15.141:0/487604083 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff584003c90 con 0x7ff594099a60 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.433+0000 7ff58a000640 1 -- 172.21.15.141:0/487604083 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ff584012420 con 0x7ff594099a60 2024-01-16T14:56:51.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.434+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5940675d0 con 0x7ff594099a60 2024-01-16T14:56:51.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.437+0000 7ff58a000640 1 -- 172.21.15.141:0/487604083 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff584017070 con 0x7ff594099a60 2024-01-16T14:56:51.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.502+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff59406f990 con 0x7ff594099a60 2024-01-16T14:56:51.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.503+0000 7ff58a000640 1 -- 172.21.15.141:0/487604083 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7ff584011590 con 0x7ff594099a60 2024-01-16T14:56:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:56:51.485 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":10,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:56:51.117513+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:51.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff594099a60 msgr2=0x7ff59409e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:51.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff594099a60 0x7ff59409e7f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff584001eb0 tx=0x7ff58400c820 comp rx=0 tx=0).stop 2024-01-16T14:56:51.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 shutdown_connections 2024-01-16T14:56:51.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff59415e880 0x7ff5940c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff594099ea0 0x7ff5940c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 --2- 172.21.15.141:0/487604083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff594099a60 0x7ff59409e7f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:51.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 >> 172.21.15.141:0/487604083 conn(0x7ff5940b2d00 msgr2=0x7ff59406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:51.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 shutdown_connections 2024-01-16T14:56:51.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:51.504+0000 7ff599000640 1 -- 172.21.15.141:0/487604083 wait complete. 2024-01-16T14:56:51.501 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T14:56:51.502 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T14:56:51.812 INFO:tasks.ceph.osd.0.smithi141.stderr:2024-01-16T14:56:51.831+0000 7eff6bbde500 -1 Falling back to public interface 2024-01-16T14:56:51.843 INFO:tasks.ceph.osd.4.smithi188.stderr:2024-01-16T14:56:51.860+0000 7fa35741f500 -1 Falling back to public interface 2024-01-16T14:56:51.846 INFO:tasks.ceph.osd.1.smithi141.stderr:2024-01-16T14:56:51.865+0000 7f649d021500 -1 Falling back to public interface 2024-01-16T14:56:51.850 INFO:tasks.ceph.osd.2.smithi141.stderr:2024-01-16T14:56:51.869+0000 7fd7990fa500 -1 Falling back to public interface 2024-01-16T14:56:51.859 INFO:tasks.ceph.osd.3.smithi141.stderr:2024-01-16T14:56:51.878+0000 7f0284982500 -1 Falling back to public interface 2024-01-16T14:56:51.864 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-01-16T14:56:51.882+0000 7f6fe4412500 -1 Falling back to public interface 2024-01-16T14:56:51.868 INFO:tasks.ceph.osd.5.smithi188.stderr:2024-01-16T14:56:51.886+0000 7f4f5b225500 -1 Falling back to public interface 2024-01-16T14:56:51.887 INFO:tasks.ceph.osd.6.smithi188.stderr:2024-01-16T14:56:51.905+0000 7f30f9414500 -1 Falling back to public interface 2024-01-16T14:56:51.920 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:51.938+0000 7f0fb5c30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:56:51.963 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:51.981+0000 7f03e761b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:56:52.223 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:52.241+0000 7f0fb5c30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:56:52.267 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:52.285+0000 7f03e761b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:56:52.282 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:52.299+0000 7f0fb5c30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:56:52.308 DEBUG:teuthology.orchestra.run.smithi141:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:52.327 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:52.346+0000 7f03e761b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:56:52.396 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:52.414+0000 7f0fb5c30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:56:52.453 INFO:tasks.ceph.mgr.y.smithi141.stderr:2024-01-16T14:56:52.471+0000 7f03e761b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:56:52.468 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.487+0000 7f4093e00640 1 Processor -- start 2024-01-16T14:56:52.469 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.487+0000 7f4093e00640 1 -- start start 2024-01-16T14:56:52.470 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.488+0000 7f4093e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c1681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.488+0000 7f4093e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c0ce860 con 0x7f408c160c80 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.488+0000 7f4093e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c0ceb40 con 0x7f408c0d2030 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.488+0000 7f4093e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c0b3230 con 0x7f408c0d38f0 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.488+0000 7f4091400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f408c160c80 0x7f408c0cde60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36260/0 (socket says 172.21.15.141:36260) 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.488+0000 7f4091400640 1 -- 172.21.15.141:0/1025185380 learned_addr learned my addr 172.21.15.141:0/1025185380 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.488+0000 7f4091e00640 1 --2- 172.21.15.141:0/1025185380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c1681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.489+0000 7f408be00640 1 -- 172.21.15.141:0/1025185380 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486416272 0 0) 0x7f408c0ce860 con 0x7f408c160c80 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.489+0000 7f408be00640 1 -- 172.21.15.141:0/1025185380 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c003520 con 0x7f408c160c80 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.489+0000 7f408be00640 1 -- 172.21.15.141:0/1025185380 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682403942 0 0) 0x7f408c0ceb40 con 0x7f408c0d2030 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.489+0000 7f4091e00640 1 -- 172.21.15.141:0/1025185380 >> v1:172.21.15.188:6789/0 conn(0x7f408c0d2030 legacy=0x7f408c0d31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.489+0000 7f4091e00640 1 -- 172.21.15.141:0/1025185380 >> v1:172.21.15.141:6789/0 conn(0x7f408c160c80 legacy=0x7f408c0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.471 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.489+0000 7f4091e00640 1 -- 172.21.15.141:0/1025185380 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c0cee20 con 0x7f408c0d38f0 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.490+0000 7f4091e00640 1 --2- 172.21.15.141:0/1025185380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c1681b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4084009dc0 tx=0x7f408402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c6f43226784d0c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.490+0000 7f408be00640 1 -- 172.21.15.141:0/1025185380 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408402ccd0 con 0x7f408c0d38f0 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.490+0000 7f408be00640 1 -- 172.21.15.141:0/1025185380 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408402ce30 con 0x7f408c0d38f0 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.490+0000 7f408be00640 1 -- 172.21.15.141:0/1025185380 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084034660 con 0x7f408c0d38f0 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.491+0000 7f4093e00640 1 -- 172.21.15.141:0/1025185380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 msgr2=0x7f408c1681b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.491+0000 7f4093e00640 1 --2- 172.21.15.141:0/1025185380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c1681b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4084009dc0 tx=0x7f408402bf20 comp rx=0 tx=0).stop 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.491+0000 7f4093e00640 1 -- 172.21.15.141:0/1025185380 shutdown_connections 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.491+0000 7f4093e00640 1 --2- 172.21.15.141:0/1025185380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c1681b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.491+0000 7f4093e00640 1 -- 172.21.15.141:0/1025185380 >> 172.21.15.141:0/1025185380 conn(0x7f408c08aff0 msgr2=0x7f408c09c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.473 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.491+0000 7f4093e00640 1 -- 172.21.15.141:0/1025185380 shutdown_connections 2024-01-16T14:56:52.474 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.491+0000 7f4093e00640 1 -- 172.21.15.141:0/1025185380 wait complete. 2024-01-16T14:56:52.474 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.492+0000 7f4093e00640 1 Processor -- start 2024-01-16T14:56:52.474 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.493+0000 7f4093e00640 1 -- start start 2024-01-16T14:56:52.476 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.493+0000 7f4093e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f408c0d2030 0x7f408c0b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.476 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.493+0000 7f4093e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c14c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.493+0000 7f4093e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c160c80 0x7f408c0b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.493+0000 7f4093e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c0b8690 con 0x7f408c0d2030 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.493+0000 7f4093e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c0b8970 con 0x7f408c160c80 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.493+0000 7f4093e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c0b8c50 con 0x7f408c0d38f0 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f408c0d2030 0x7f408c0b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f408c0d2030 0x7f408c0b7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41474/0 (socket says 172.21.15.141:41474) 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 -- 172.21.15.141:0/1211373356 learned_addr learned my addr 172.21.15.141:0/1211373356 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4090a00640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c14c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091e00640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c160c80 0x7f408c0b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 -- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 msgr2=0x7f408c14c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c14c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.477 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 -- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c160c80 msgr2=0x7f408c0b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c160c80 0x7f408c0b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 -- 172.21.15.141:0/1211373356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084009a70 con 0x7f408c0d2030 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.494+0000 7f4091400640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f408c0d2030 0x7f408c0b7d80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c430 tx=0x7f407c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.495+0000 7f4089600640 1 -- 172.21.15.141:0/1211373356 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c005ce0 con 0x7f408c0d2030 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.495+0000 7f4089600640 1 -- 172.21.15.141:0/1211373356 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c005e40 con 0x7f408c0d2030 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.495+0000 7f4089600640 1 -- 172.21.15.141:0/1211373356 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c0072f0 con 0x7f408c0d2030 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.495+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c0b8f30 con 0x7f408c0d2030 2024-01-16T14:56:52.478 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.495+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c1f3490 con 0x7f408c0d2030 2024-01-16T14:56:52.479 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.496+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c1f36b0 con 0x7f408c0d2030 2024-01-16T14:56:52.479 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.496+0000 7f4089600640 1 -- 172.21.15.141:0/1211373356 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 2) v1 ==== 44721+0+0 (secure 0 0 0) 0x7f407c007450 con 0x7f408c0d2030 2024-01-16T14:56:52.479 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.496+0000 7f4089600640 1 -- 172.21.15.141:0/1211373356 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f407c048f70 con 0x7f408c0d2030 2024-01-16T14:56:52.480 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.498+0000 7f4089600640 1 -- 172.21.15.141:0/1211373356 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f408c1f36b0 con 0x7f408c0d2030 2024-01-16T14:56:52.541 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.559+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f408c1f36b0 con 0x7f408c0d2030 2024-01-16T14:56:52.542 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.560+0000 7f4089600640 1 -- 172.21.15.141:0/1211373356 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f408c1f36b0 con 0x7f408c0d2030 2024-01-16T14:56:52.542 INFO:teuthology.misc.health.smithi141.stdout: 2024-01-16T14:56:52.542 INFO:teuthology.misc.health.smithi141.stdout:{"epoch":10,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:56:51.117513+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:52.543 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f408c0d2030 msgr2=0x7f408c0b7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:52.543 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f408c0d2030 0x7f408c0b7d80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c430 tx=0x7f407c00c8f0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.543 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 shutdown_connections 2024-01-16T14:56:52.544 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c160c80 0x7f408c0b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.544 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f408c0d38f0 0x7f408c14c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.544 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 --2- 172.21.15.141:0/1211373356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f408c0d2030 0x7f408c0b7d80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:52.544 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 >> 172.21.15.141:0/1211373356 conn(0x7f408c08aff0 msgr2=0x7f408c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:52.544 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 shutdown_connections 2024-01-16T14:56:52.544 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:52.561+0000 7f4093e00640 1 -- 172.21.15.141:0/1211373356 wait complete. 2024-01-16T14:56:52.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-16T14:56:53.440 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:53.457+0000 7f0f56400640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-16T14:56:53.440 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:56:53.458+0000 7f0f56400640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-16T14:56:55.365 INFO:tasks.ceph.osd.4.smithi188.stderr:2024-01-16T14:56:55.381+0000 7fa35741f500 -1 osd.4 0 log_to_monitors true 2024-01-16T14:56:55.382 INFO:tasks.ceph.osd.1.smithi141.stderr:2024-01-16T14:56:55.400+0000 7f649d021500 -1 osd.1 0 log_to_monitors true 2024-01-16T14:56:55.404 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-01-16T14:56:55.421+0000 7f6fe4412500 -1 osd.7 0 log_to_monitors true 2024-01-16T14:56:55.418 INFO:tasks.ceph.osd.2.smithi141.stderr:2024-01-16T14:56:55.436+0000 7fd7990fa500 -1 osd.2 0 log_to_monitors true 2024-01-16T14:56:55.424 INFO:tasks.ceph.osd.5.smithi188.stderr:2024-01-16T14:56:55.442+0000 7f4f5b225500 -1 osd.5 0 log_to_monitors true 2024-01-16T14:56:55.440 INFO:tasks.ceph.osd.3.smithi141.stderr:2024-01-16T14:56:55.458+0000 7f0284982500 -1 osd.3 0 log_to_monitors true 2024-01-16T14:56:55.449 INFO:tasks.ceph.osd.6.smithi188.stderr:2024-01-16T14:56:55.467+0000 7f30f9414500 -1 osd.6 0 log_to_monitors true 2024-01-16T14:56:55.583 INFO:tasks.ceph.osd.0.smithi141.stderr:2024-01-16T14:56:55.601+0000 7eff6bbde500 -1 osd.0 0 log_to_monitors true 2024-01-16T14:56:58.131 INFO:tasks.ceph.osd.1.smithi141.stderr:2024-01-16T14:56:58.149+0000 7f6496e00640 -1 osd.1 0 waiting for initial osdmap 2024-01-16T14:56:58.134 INFO:tasks.ceph.osd.2.smithi141.stderr:2024-01-16T14:56:58.153+0000 7fd793600640 -1 osd.2 0 waiting for initial osdmap 2024-01-16T14:56:58.152 INFO:tasks.ceph.osd.1.smithi141.stderr:2024-01-16T14:56:58.170+0000 7f648d800640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:58.184 INFO:tasks.ceph.osd.2.smithi141.stderr:2024-01-16T14:56:58.203+0000 7fd78a000640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:58.561 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-01-16T14:56:58.579+0000 7f6fde400640 -1 osd.7 0 waiting for initial osdmap 2024-01-16T14:56:58.573 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-01-16T14:56:58.590+0000 7f6fd4e00640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:58.592 INFO:tasks.ceph.osd.5.smithi188.stderr:2024-01-16T14:56:58.610+0000 7f4f55200640 -1 osd.5 0 waiting for initial osdmap 2024-01-16T14:56:58.604 INFO:tasks.ceph.osd.5.smithi188.stderr:2024-01-16T14:56:58.622+0000 7f4f4bc00640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:58.623 INFO:tasks.ceph.osd.4.smithi188.stderr:2024-01-16T14:56:58.641+0000 7fa353200640 -1 osd.4 0 waiting for initial osdmap 2024-01-16T14:56:58.636 INFO:tasks.ceph.osd.4.smithi188.stderr:2024-01-16T14:56:58.653+0000 7fa347e00640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:58.948 INFO:tasks.ceph.osd.6.smithi188.stderr:2024-01-16T14:56:58.966+0000 7f30f3200640 -1 osd.6 0 waiting for initial osdmap 2024-01-16T14:56:58.961 INFO:tasks.ceph.osd.6.smithi188.stderr:2024-01-16T14:56:58.979+0000 7f30e9c00640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:58.999 INFO:tasks.ceph.osd.0.smithi141.stderr:2024-01-16T14:56:59.018+0000 7eff66000640 -1 osd.0 0 waiting for initial osdmap 2024-01-16T14:56:59.011 INFO:tasks.ceph.osd.0.smithi141.stderr:2024-01-16T14:56:59.029+0000 7eff5ca00640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:59.024 INFO:tasks.ceph.osd.3.smithi141.stderr:2024-01-16T14:56:59.043+0000 7f027ee00640 -1 osd.3 0 waiting for initial osdmap 2024-01-16T14:56:59.033 INFO:tasks.ceph.osd.3.smithi141.stderr:2024-01-16T14:56:59.051+0000 7f0275800640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:56:59.362 DEBUG:teuthology.orchestra.run.smithi141:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:59.546 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.564+0000 7fd184600640 1 Processor -- start 2024-01-16T14:56:59.546 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.564+0000 7fd184600640 1 -- start start 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd184600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd17c06e8c0 0x7fd17c0b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd184600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd17c06a450 0x7fd17c06a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd184600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c06ed70 con 0x7fd17c1578c0 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd184600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c06adc0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd184600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c0ad650 con 0x7fd17c06a450 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd181200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd17c06e8c0 0x7fd17c0b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd181200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd17c06e8c0 0x7fd17c0b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53480/0 (socket says 172.21.15.141:53480) 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd181200640 1 -- 172.21.15.141:0/3182348004 learned_addr learned my addr 172.21.15.141:0/3182348004 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd182600640 1 --2- 172.21.15.141:0/3182348004 >> v2:172.21.15.188:3300/0 conn(0x7fd17c06a450 0x7fd17c06a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.549 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd181200640 1 -- 172.21.15.141:0/3182348004 >> v2:172.21.15.188:3300/0 conn(0x7fd17c06a450 msgr2=0x7fd17c06a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:59.550 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd181200640 1 --2- 172.21.15.141:0/3182348004 >> v2:172.21.15.188:3300/0 conn(0x7fd17c06a450 0x7fd17c06a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.550 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd181200640 1 -- 172.21.15.141:0/3182348004 >> v1:172.21.15.188:6789/0 conn(0x7fd17c1578c0 legacy=0x7fd17c15d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:59.550 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd181200640 1 -- 172.21.15.141:0/3182348004 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c15e0c0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.550 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.566+0000 7fd182600640 1 --2- 172.21.15.141:0/3182348004 >> v2:172.21.15.188:3300/0 conn(0x7fd17c06a450 0x7fd17c06a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:56:59.551 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.567+0000 7fd181200640 1 --2- 172.21.15.141:0/3182348004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c0b57c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd17c070e80 tx=0x7fd16402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e8548426694355 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.551 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd17be00640 1 -- 172.21.15.141:0/3182348004 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164031870 con 0x7fd17c06e8c0 2024-01-16T14:56:59.551 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd17be00640 1 -- 172.21.15.141:0/3182348004 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1640319d0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.551 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd17be00640 1 -- 172.21.15.141:0/3182348004 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164031d60 con 0x7fd17c06e8c0 2024-01-16T14:56:59.551 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd184600640 1 -- 172.21.15.141:0/3182348004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 msgr2=0x7fd17c0b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.551 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd184600640 1 --2- 172.21.15.141:0/3182348004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c0b57c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd17c070e80 tx=0x7fd16402b910 comp rx=0 tx=0).stop 2024-01-16T14:56:59.552 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd184600640 1 -- 172.21.15.141:0/3182348004 shutdown_connections 2024-01-16T14:56:59.552 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd184600640 1 --2- 172.21.15.141:0/3182348004 >> v2:172.21.15.188:3300/0 conn(0x7fd17c06a450 0x7fd17c06a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.552 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd184600640 1 --2- 172.21.15.141:0/3182348004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c0b57c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.552 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.568+0000 7fd184600640 1 -- 172.21.15.141:0/3182348004 >> 172.21.15.141:0/3182348004 conn(0x7fd17c0b1d40 msgr2=0x7fd17c0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:59.552 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.569+0000 7fd184600640 1 -- 172.21.15.141:0/3182348004 shutdown_connections 2024-01-16T14:56:59.552 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.569+0000 7fd184600640 1 -- 172.21.15.141:0/3182348004 wait complete. 2024-01-16T14:56:59.553 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.570+0000 7fd184600640 1 Processor -- start 2024-01-16T14:56:59.553 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.570+0000 7fd184600640 1 -- start start 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.571+0000 7fd184600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd17c06a450 0x7fd17c153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.571+0000 7fd184600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c1f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.571+0000 7fd184600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd17c1578c0 0x7fd17c15a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.571+0000 7fd184600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c15e8c0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.571+0000 7fd184600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c15f440 con 0x7fd17c06a450 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.571+0000 7fd184600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c1546b0 con 0x7fd17c1578c0 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c1f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.555 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c1f3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53492/0 (socket says 172.21.15.141:53492) 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 -- 172.21.15.141:0/3409092383 learned_addr learned my addr 172.21.15.141:0/3409092383 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181c00640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd17c06a450 0x7fd17c153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 -- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd17c1578c0 msgr2=0x7fd17c15a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd17c1578c0 0x7fd17c15a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 -- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd17c06a450 msgr2=0x7fd17c153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd17c06a450 0x7fd17c153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 -- 172.21.15.141:0/3409092383 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640094d0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.572+0000 7fd181200640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c1f3410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd164000c00 tx=0x7fd1640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.573+0000 7fd179600640 1 -- 172.21.15.141:0/3409092383 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164031840 con 0x7fd17c06e8c0 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.573+0000 7fd179600640 1 -- 172.21.15.141:0/3409092383 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1640443f0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.573+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c154180 con 0x7fd17c06e8c0 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.573+0000 7fd179600640 1 -- 172.21.15.141:0/3409092383 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1640447a0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.556 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.573+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c15edf0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.557 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.574+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c0675d0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.557 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.575+0000 7fd179600640 1 -- 172.21.15.141:0/3409092383 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fd16404d450 con 0x7fd17c06e8c0 2024-01-16T14:56:59.558 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.576+0000 7fd179600640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd15406b230 0x7fd15406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.558 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.576+0000 7fd179600640 1 -- 172.21.15.141:0/3409092383 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fd16402d040 con 0x7fd17c06e8c0 2024-01-16T14:56:59.558 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.577+0000 7fd181c00640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd15406b230 0x7fd15406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.559 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.578+0000 7fd181c00640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd15406b230 0x7fd15406d6f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd16c004ea0 tx=0x7fd16c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.563 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.581+0000 7fd179600640 1 -- 172.21.15.141:0/3409092383 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1640771f0 con 0x7fd17c06e8c0 2024-01-16T14:56:59.746 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.762+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd17c15d640 con 0x7fd17c06e8c0 2024-01-16T14:56:59.747 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.764+0000 7fd179600640 1 -- 172.21.15.141:0/3409092383 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10956 (secure 0 0 0) 0x7fd16407a810 con 0x7fd17c06e8c0 2024-01-16T14:56:59.747 INFO:teuthology.misc.health.smithi141.stdout: 2024-01-16T14:56:59.747 INFO:teuthology.misc.health.smithi141.stdout:{"epoch":14,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:56:59.168868+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd15406b230 msgr2=0x7fd15406d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd15406b230 0x7fd15406d6f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd16c004ea0 tx=0x7fd16c009040 comp rx=0 tx=0).stop 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 msgr2=0x7fd17c1f3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c1f3410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd164000c00 tx=0x7fd1640046c0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 shutdown_connections 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd15406b230 0x7fd15406d6f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd17c1578c0 0x7fd17c15a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd17c06e8c0 0x7fd17c1f3410 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 --2- 172.21.15.141:0/3409092383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd17c06a450 0x7fd17c153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.766+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 >> 172.21.15.141:0/3409092383 conn(0x7fd17c0b1d40 msgr2=0x7fd17c0b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.767+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 shutdown_connections 2024-01-16T14:56:59.750 INFO:teuthology.misc.health.smithi141.stderr:2024-01-16T14:56:59.767+0000 7fd184600640 1 -- 172.21.15.141:0/3409092383 wait complete. 2024-01-16T14:56:59.762 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-16T14:56:59.762 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T14:56:59.763 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T14:56:59.763 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T14:56:59.769 INFO:tasks.ceph.mds.a:Started 2024-01-16T14:56:59.770 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T14:56:59.770 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T14:56:59.776 INFO:tasks.ceph.mds.c:Started 2024-01-16T14:56:59.776 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T14:56:59.777 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T14:56:59.784 INFO:tasks.ceph.mds.b:Started 2024-01-16T14:56:59.784 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T14:56:59.784 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T14:56:59.792 INFO:tasks.ceph.mds.d:Started 2024-01-16T14:56:59.792 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-16T14:56:59.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:56:59.913 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T14:56:59.919 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T14:56:59.940 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T14:56:59.943 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T14:56:59.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.987+0000 7f2fd8200640 1 Processor -- start 2024-01-16T14:56:59.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.987+0000 7f2fd8200640 1 -- start start 2024-01-16T14:56:59.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd8200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2fd015d610 0x7f2fd00ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd8200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2fd00ced60 0x7f2fd00d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd8200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00d2170 con 0x7f2fd00d1d20 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd8200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd00d1390 con 0x7f2fd00ced60 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd8200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd00b3400 con 0x7f2fd015d610 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd4e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2fd00ced60 0x7f2fd00d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd4e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2fd00ced60 0x7f2fd00d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53540/0 (socket says 172.21.15.141:53540) 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd4e00640 1 -- 172.21.15.141:0/986673942 learned_addr learned my addr 172.21.15.141:0/986673942 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.988+0000 7f2fd5800640 1 --2- 172.21.15.141:0/986673942 >> v2:172.21.15.188:3300/0 conn(0x7f2fd015d610 0x7f2fd00ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.989+0000 7f2fd4e00640 1 -- 172.21.15.141:0/986673942 >> v2:172.21.15.188:3300/0 conn(0x7f2fd015d610 msgr2=0x7f2fd00ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.989+0000 7f2fd4e00640 1 --2- 172.21.15.141:0/986673942 >> v2:172.21.15.188:3300/0 conn(0x7f2fd015d610 0x7f2fd00ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.989+0000 7f2fd4e00640 1 -- 172.21.15.141:0/986673942 >> v1:172.21.15.188:6789/0 conn(0x7f2fd00d1d20 legacy=0x7f2fd00cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.989+0000 7f2fd4e00640 1 -- 172.21.15.141:0/986673942 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd00b27f0 con 0x7f2fd00ced60 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.990+0000 7f2fd4e00640 1 --2- 172.21.15.141:0/986673942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00d17e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00b5dc0 tx=0x7f2fb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d391bf9ebd4ccf5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.990+0000 7f2fcfe00640 1 -- 172.21.15.141:0/986673942 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8030860 con 0x7f2fd00ced60 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.990+0000 7f2fcfe00640 1 -- 172.21.15.141:0/986673942 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb80309c0 con 0x7f2fd00ced60 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.990+0000 7f2fcfe00640 1 -- 172.21.15.141:0/986673942 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8030d50 con 0x7f2fd00ced60 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.990+0000 7f2fd8200640 1 -- 172.21.15.141:0/986673942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 msgr2=0x7f2fd00d17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.990+0000 7f2fd8200640 1 --2- 172.21.15.141:0/986673942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00d17e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00b5dc0 tx=0x7f2fb802b9a0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.991+0000 7f2fd8200640 1 -- 172.21.15.141:0/986673942 shutdown_connections 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.991+0000 7f2fd8200640 1 --2- 172.21.15.141:0/986673942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00d17e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.991+0000 7f2fd8200640 1 --2- 172.21.15.141:0/986673942 >> v2:172.21.15.188:3300/0 conn(0x7f2fd015d610 0x7f2fd00ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.991+0000 7f2fd8200640 1 -- 172.21.15.141:0/986673942 >> 172.21.15.141:0/986673942 conn(0x7f2fd009c540 msgr2=0x7f2fd00d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.991+0000 7f2fd8200640 1 -- 172.21.15.141:0/986673942 shutdown_connections 2024-01-16T14:56:59.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.991+0000 7f2fd8200640 1 -- 172.21.15.141:0/986673942 wait complete. 2024-01-16T14:56:59.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.992+0000 7f2fd8200640 1 Processor -- start 2024-01-16T14:56:59.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.992+0000 7f2fd8200640 1 -- start start 2024-01-16T14:56:59.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.993+0000 7f2fd8200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00b7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.993+0000 7f2fd8200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2fd00d1d20 0x7f2fd00ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.993+0000 7f2fd8200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fd015d610 0x7f2fd0151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.993+0000 7f2fd8200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00ca810 con 0x7f2fd00ced60 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.993+0000 7f2fd8200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00ccfe0 con 0x7f2fd015d610 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.993+0000 7f2fd8200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00cd260 con 0x7f2fd00d1d20 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.993+0000 7f2fd5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00b7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53552/0 (socket says 172.21.15.141:53552) 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 -- 172.21.15.141:0/671564523 learned_addr learned my addr 172.21.15.141:0/671564523 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd4e00640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2fd00d1d20 0x7f2fd00ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd6200640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fd015d610 0x7f2fd0151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 -- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2fd00d1d20 msgr2=0x7f2fd00ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2fd00d1d20 0x7f2fd00ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 -- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fd015d610 msgr2=0x7f2fd0151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fd015d610 0x7f2fd0151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 -- 172.21.15.141:0/671564523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80094d0 con 0x7f2fd00ced60 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd4e00640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2fd00d1d20 0x7f2fd00ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd5800640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00b7710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000d800 tx=0x7f2fc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fcd600640 1 -- 172.21.15.141:0/671564523 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc000b6c0 con 0x7f2fd00ced60 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fcd600640 1 -- 172.21.15.141:0/671564523 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc000b820 con 0x7f2fd00ced60 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fcd600640 1 -- 172.21.15.141:0/671564523 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc000bbb0 con 0x7f2fd00ced60 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.994+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd00cd540 con 0x7f2fd00ced60 2024-01-16T14:56:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.995+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd00b8150 con 0x7f2fd00ced60 2024-01-16T14:56:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.995+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd00b82e0 con 0x7f2fd00ced60 2024-01-16T14:56:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.997+0000 7f2fcd600640 1 -- 172.21.15.141:0/671564523 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f2fc000bd10 con 0x7f2fd00ced60 2024-01-16T14:56:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.998+0000 7f2fcd600640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2fa806b230 0x7f2fa806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:56:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.998+0000 7f2fcd600640 1 -- 172.21.15.141:0/671564523 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f2fc0089b90 con 0x7f2fd00ced60 2024-01-16T14:56:59.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.998+0000 7f2fd4e00640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2fa806b230 0x7f2fa806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:56:59.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:56:59.999+0000 7f2fd4e00640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2fa806b230 0x7f2fa806d6f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8036000 tx=0x7f2fb803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:56:59.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.003+0000 7f2fcd600640 1 -- 172.21.15.141:0/671564523 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fc0054fa0 con 0x7f2fd00ced60 2024-01-16T14:57:00.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.185+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fd00675d0 con 0x7f2fd00ced60 2024-01-16T14:57:00.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.186+0000 7f2fcd600640 1 -- 172.21.15.141:0/671564523 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11140 (secure 0 0 0) 0x7f2fc00585c0 con 0x7f2fd00ced60 2024-01-16T14:57:00.169 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:00.170 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":15,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:00.172919+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:00.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2fa806b230 msgr2=0x7f2fa806d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2fa806b230 0x7f2fa806d6f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8036000 tx=0x7f2fb803b040 comp rx=0 tx=0).stop 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 msgr2=0x7f2fd00b7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00b7710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000d800 tx=0x7f2fc000dcc0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 shutdown_connections 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2fa806b230 0x7f2fa806d6f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fd015d610 0x7f2fd0151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2fd00d1d20 0x7f2fd00ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 --2- 172.21.15.141:0/671564523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2fd00ced60 0x7f2fd00b7710 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 >> 172.21.15.141:0/671564523 conn(0x7f2fd009c540 msgr2=0x7f2fd00b0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 shutdown_connections 2024-01-16T14:57:00.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.189+0000 7f2fd8200640 1 -- 172.21.15.141:0/671564523 wait complete. 2024-01-16T14:57:00.188 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-01-16T14:57:00.188 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:00.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.396+0000 7fa8fae00640 1 Processor -- start 2024-01-16T14:57:00.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.396+0000 7fa8fae00640 1 -- start start 2024-01-16T14:57:00.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8fae00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa8f413df80 0x7fa8f415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8fae00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa8f413e360 0x7fa8f40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8fae00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f406c470 con 0x7fa8f415f9c0 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8fae00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f406c750 con 0x7fa8f413e360 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8fae00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f406ca30 con 0x7fa8f413df80 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8ebe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa8f413e360 0x7fa8f40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8ebe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa8f413e360 0x7fa8f40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53556/0 (socket says 172.21.15.141:53556) 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.398+0000 7fa8ebe00640 1 -- 172.21.15.141:0/30197586 learned_addr learned my addr 172.21.15.141:0/30197586 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.399+0000 7fa8f3e00640 1 --2- 172.21.15.141:0/30197586 >> v2:172.21.15.188:3300/0 conn(0x7fa8f413df80 0x7fa8f415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.399+0000 7fa8ebe00640 1 -- 172.21.15.141:0/30197586 >> v2:172.21.15.188:3300/0 conn(0x7fa8f413df80 msgr2=0x7fa8f415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.399+0000 7fa8ebe00640 1 --2- 172.21.15.141:0/30197586 >> v2:172.21.15.188:3300/0 conn(0x7fa8f413df80 0x7fa8f415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.399+0000 7fa8ebe00640 1 -- 172.21.15.141:0/30197586 >> v1:172.21.15.188:6789/0 conn(0x7fa8f415f9c0 legacy=0x7fa8f41408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:00.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.399+0000 7fa8ebe00640 1 -- 172.21.15.141:0/30197586 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f406cd10 con 0x7fa8f413e360 2024-01-16T14:57:00.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.399+0000 7fa8f3e00640 1 --2- 172.21.15.141:0/30197586 >> v2:172.21.15.188:3300/0 conn(0x7fa8f413df80 0x7fa8f415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.400+0000 7fa8ebe00640 1 --2- 172.21.15.141:0/30197586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413e360 0x7fa8f40cf290 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4072770 tx=0x7fa8dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1794637bc121dc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.400+0000 7fa8f3400640 1 -- 172.21.15.141:0/30197586 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc031870 con 0x7fa8f413e360 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.400+0000 7fa8f3400640 1 -- 172.21.15.141:0/30197586 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc0319d0 con 0x7fa8f413e360 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.400+0000 7fa8f3400640 1 -- 172.21.15.141:0/30197586 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc031d60 con 0x7fa8f413e360 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.400+0000 7fa8fae00640 1 -- 172.21.15.141:0/30197586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413e360 msgr2=0x7fa8f40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.400+0000 7fa8fae00640 1 --2- 172.21.15.141:0/30197586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413e360 0x7fa8f40cf290 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4072770 tx=0x7fa8dc02b910 comp rx=0 tx=0).stop 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.401+0000 7fa8fae00640 1 -- 172.21.15.141:0/30197586 shutdown_connections 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.401+0000 7fa8fae00640 1 --2- 172.21.15.141:0/30197586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413e360 0x7fa8f40cf290 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.401+0000 7fa8fae00640 1 --2- 172.21.15.141:0/30197586 >> v2:172.21.15.188:3300/0 conn(0x7fa8f413df80 0x7fa8f415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.401+0000 7fa8fae00640 1 -- 172.21.15.141:0/30197586 >> 172.21.15.141:0/30197586 conn(0x7fa8f406a800 msgr2=0x7fa8f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.401+0000 7fa8fae00640 1 -- 172.21.15.141:0/30197586 shutdown_connections 2024-01-16T14:57:00.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.401+0000 7fa8fae00640 1 -- 172.21.15.141:0/30197586 wait complete. 2024-01-16T14:57:00.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.402+0000 7fa8fae00640 1 Processor -- start 2024-01-16T14:57:00.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.402+0000 7fa8fae00640 1 -- start start 2024-01-16T14:57:00.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.403+0000 7fa8fae00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413df80 0x7fa8f41f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.403+0000 7fa8fae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8f413e360 0x7fa8f414fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.403+0000 7fa8fae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8f415f9c0 0x7fa8f415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.403+0000 7fa8fae00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4153b50 con 0x7fa8f413df80 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.403+0000 7fa8fae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f414d4f0 con 0x7fa8f415f9c0 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.403+0000 7fa8fae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f41f3200 con 0x7fa8f413e360 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413df80 0x7fa8f41f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413df80 0x7fa8f41f2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53570/0 (socket says 172.21.15.141:53570) 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 -- 172.21.15.141:0/1881567277 learned_addr learned my addr 172.21.15.141:0/1881567277 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8ebe00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8f413e360 0x7fa8f414fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f8e00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8f415f9c0 0x7fa8f415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 -- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8f413e360 msgr2=0x7fa8f414fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8f413e360 0x7fa8f414fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 -- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8f415f9c0 msgr2=0x7fa8f415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8f415f9c0 0x7fa8f415c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 -- 172.21.15.141:0/1881567277 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc0094d0 con 0x7fa8f413df80 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f3e00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413df80 0x7fa8f41f2cc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c7f0 tx=0x7fa8e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.404+0000 7fa8f0c00640 1 -- 172.21.15.141:0/1881567277 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4007820 con 0x7fa8f413df80 2024-01-16T14:57:00.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.405+0000 7fa8f0c00640 1 -- 172.21.15.141:0/1881567277 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e4007980 con 0x7fa8f413df80 2024-01-16T14:57:00.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.405+0000 7fa8f0c00640 1 -- 172.21.15.141:0/1881567277 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4007cf0 con 0x7fa8f413df80 2024-01-16T14:57:00.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.405+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f41532b0 con 0x7fa8f413df80 2024-01-16T14:57:00.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.405+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f4153870 con 0x7fa8f413df80 2024-01-16T14:57:00.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.406+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f414d760 con 0x7fa8f413df80 2024-01-16T14:57:00.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.407+0000 7fa8f0c00640 1 -- 172.21.15.141:0/1881567277 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fa8e40113f0 con 0x7fa8f413df80 2024-01-16T14:57:00.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.407+0000 7fa8f0c00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa8cc06b230 0x7fa8cc06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:00.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.407+0000 7fa8f0c00640 1 -- 172.21.15.141:0/1881567277 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fa8e4089890 con 0x7fa8f413df80 2024-01-16T14:57:00.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.408+0000 7fa8ebe00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa8cc06b230 0x7fa8cc06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:00.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.409+0000 7fa8ebe00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa8cc06b230 0x7fa8cc06d6f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc036000 tx=0x7fa8dc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:00.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.412+0000 7fa8f0c00640 1 -- 172.21.15.141:0/1881567277 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8e4054d40 con 0x7fa8f413df80 2024-01-16T14:57:00.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.641+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8f414d250 con 0x7fa8f413df80 2024-01-16T14:57:00.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.642+0000 7fa8f0c00640 1 -- 172.21.15.141:0/1881567277 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fa8f414d250 con 0x7fa8f413df80 2024-01-16T14:57:00.625 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:00.626 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-16T14:57:00.626 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 2 2024-01-16T14:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa8cc06b230 msgr2=0x7fa8cc06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa8cc06b230 0x7fa8cc06d6f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc036000 tx=0x7fa8dc03c040 comp rx=0 tx=0).stop 2024-01-16T14:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413df80 msgr2=0x7fa8f41f2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413df80 0x7fa8f41f2cc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c7f0 tx=0x7fa8e400ccb0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 shutdown_connections 2024-01-16T14:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa8cc06b230 0x7fa8cc06d6f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8f415f9c0 0x7fa8f415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8f413e360 0x7fa8f414fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 --2- 172.21.15.141:0/1881567277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8f413df80 0x7fa8f41f2cc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:00.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 >> 172.21.15.141:0/1881567277 conn(0x7fa8f406a800 msgr2=0x7fa8f40b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:00.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 shutdown_connections 2024-01-16T14:57:00.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:00.645+0000 7fa8fae00640 1 -- 172.21.15.141:0/1881567277 wait complete. 2024-01-16T14:57:01.919 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:01.927 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:02.646 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:02.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.869+0000 7fcf9ae00640 1 Processor -- start 2024-01-16T14:57:02.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.869+0000 7fcf9ae00640 1 -- start start 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.871+0000 7fcf9ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 0x7fcf94161a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.871+0000 7fcf9ae00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940cebe0 con 0x7fcf94070af0 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.871+0000 7fcf9ae00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940ceec0 con 0x7fcf9406ba30 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.871+0000 7fcf9ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940cf1a0 con 0x7fcf9415d600 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf98e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fcf94070af0 0x7fcf9406d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38954/0 (socket says 172.21.15.141:38954) 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf98e00640 1 -- 172.21.15.141:0/3582210573 learned_addr learned my addr 172.21.15.141:0/3582210573 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf93e00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 0x7fcf94161a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219168114 0 0) 0x7fcf940cebe0 con 0x7fcf94070af0 2024-01-16T14:57:02.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf74003520 con 0x7fcf94070af0 2024-01-16T14:57:02.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722773678 0 0) 0x7fcf940ceec0 con 0x7fcf9406ba30 2024-01-16T14:57:02.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf940cebe0 con 0x7fcf9406ba30 2024-01-16T14:57:02.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.872+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3747013531 0 0) 0x7fcf74003520 con 0x7fcf94070af0 2024-01-16T14:57:02.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 msgr2=0x7fcf94161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 0x7fcf94161a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 >> v1:172.21.15.188:6789/0 conn(0x7fcf9406ba30 legacy=0x7fcf940ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94161210 con 0x7fcf94070af0 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fcf8c001670 con 0x7fcf94070af0 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 >> v1:172.21.15.141:6789/0 conn(0x7fcf94070af0 legacy=0x7fcf9406d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf74003b30 0x7fcf74003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 0x7fcf740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf74004d00 0x7fcf74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7400ba30 con 0x7fcf74004d00 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7400bb70 con 0x7fcf74003b30 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7400bce0 con 0x7fcf9415d600 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf98e00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf74004d00 0x7fcf74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf93e00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf74003b30 0x7fcf74003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.873+0000 7fcf93400640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 0x7fcf740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.874+0000 7fcf98e00640 1 -- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 msgr2=0x7fcf740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.874+0000 7fcf98e00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 0x7fcf740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.874+0000 7fcf98e00640 1 -- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf74003b30 msgr2=0x7fcf74003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.874+0000 7fcf98e00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf74003b30 0x7fcf74003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.874+0000 7fcf98e00640 1 -- 172.21.15.141:0/3582210573 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74003520 con 0x7fcf74004d00 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.874+0000 7fcf93400640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf9415d600 0x7fcf740047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.875+0000 7fcf98e00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf74004d00 0x7fcf74005180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c007fd0 tx=0x7fcf740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c790bd33c0ab6894 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.875+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c005ad0 con 0x7fcf74004d00 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c005c50 con 0x7fcf74004d00 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf92a00640 1 -- 172.21.15.141:0/3582210573 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c02c850 con 0x7fcf74004d00 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf9ae00640 1 -- 172.21.15.141:0/3582210573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf74004d00 msgr2=0x7fcf74005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf9ae00640 1 --2- 172.21.15.141:0/3582210573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf74004d00 0x7fcf74005180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c007fd0 tx=0x7fcf740057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf98e00640 1 -- 172.21.15.141:0/3582210573 reap_dead start 2024-01-16T14:57:02.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf9ae00640 1 -- 172.21.15.141:0/3582210573 shutdown_connections 2024-01-16T14:57:02.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf9ae00640 1 -- 172.21.15.141:0/3582210573 >> 172.21.15.141:0/3582210573 conn(0x7fcf940cf700 msgr2=0x7fcf940cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:02.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf9ae00640 1 -- 172.21.15.141:0/3582210573 shutdown_connections 2024-01-16T14:57:02.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.876+0000 7fcf9ae00640 1 -- 172.21.15.141:0/3582210573 wait complete. 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.878+0000 7fcf9ae00640 1 Processor -- start 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.878+0000 7fcf9ae00640 1 -- start start 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf9ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf74004d00 0x7fcf9414e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf9ae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf9415f0b0 0x7fcf9415c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf9ae00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf941605e0 0x7fcf940ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf9ae00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415f540 con 0x7fcf941605e0 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf9ae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415f820 con 0x7fcf9415f0b0 2024-01-16T14:57:02.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf9ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415fb00 con 0x7fcf74004d00 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf98e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf941605e0 0x7fcf940ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf98e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf941605e0 0x7fcf940ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53596/0 (socket says 172.21.15.141:53596) 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.879+0000 7fcf98e00640 1 -- 172.21.15.141:0/1869300886 learned_addr learned my addr 172.21.15.141:0/1869300886 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf93e00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf74004d00 0x7fcf9414e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf93400640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf9415f0b0 0x7fcf9415c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf98e00640 1 -- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf74004d00 msgr2=0x7fcf9414e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf98e00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf74004d00 0x7fcf9414e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf98e00640 1 -- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf9415f0b0 msgr2=0x7fcf9415c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf98e00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf9415f0b0 0x7fcf9415c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf98e00640 1 -- 172.21.15.141:0/1869300886 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9415fde0 con 0x7fcf941605e0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf98e00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf941605e0 0x7fcf940ad920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c02c4c0 tx=0x7fcf8c001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.880+0000 7fcf8be00640 1 -- 172.21.15.141:0/1869300886 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c0049e0 con 0x7fcf941605e0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.881+0000 7fcf8be00640 1 -- 172.21.15.141:0/1869300886 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c004b60 con 0x7fcf941605e0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.881+0000 7fcf8be00640 1 -- 172.21.15.141:0/1869300886 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c001bd0 con 0x7fcf941605e0 2024-01-16T14:57:02.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.881+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf941600c0 con 0x7fcf941605e0 2024-01-16T14:57:02.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.881+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9415c570 con 0x7fcf941605e0 2024-01-16T14:57:02.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.882+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9414e8e0 con 0x7fcf941605e0 2024-01-16T14:57:02.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.883+0000 7fcf8be00640 1 -- 172.21.15.141:0/1869300886 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fcf8c032070 con 0x7fcf941605e0 2024-01-16T14:57:02.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.884+0000 7fcf8be00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf6c06b230 0x7fcf6c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:02.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.884+0000 7fcf8be00640 1 -- 172.21.15.141:0/1869300886 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcf8c03c070 con 0x7fcf941605e0 2024-01-16T14:57:02.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.885+0000 7fcf93e00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf6c06b230 0x7fcf6c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:02.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.886+0000 7fcf93e00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf6c06b230 0x7fcf6c06d6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf84007520 tx=0x7fcf8400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:02.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:02.890+0000 7fcf8be00640 1 -- 172.21.15.141:0/1869300886 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf8c075df0 con 0x7fcf941605e0 2024-01-16T14:57:03.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.127+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf940675d0 con 0x7fcf941605e0 2024-01-16T14:57:03.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.129+0000 7fcf8be00640 1 -- 172.21.15.141:0/1869300886 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7fcf8c039070 con 0x7fcf941605e0 2024-01-16T14:57:03.112 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:03.112 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-16T14:57:03.112 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 3 2024-01-16T14:57:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf6c06b230 msgr2=0x7fcf6c06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf6c06b230 0x7fcf6c06d6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf84007520 tx=0x7fcf8400b040 comp rx=0 tx=0).stop 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf941605e0 msgr2=0x7fcf940ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf941605e0 0x7fcf940ad920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c02c4c0 tx=0x7fcf8c001980 comp rx=0 tx=0).stop 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 shutdown_connections 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf6c06b230 0x7fcf6c06d6f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf941605e0 0x7fcf940ad920 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf9415f0b0 0x7fcf9415c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 --2- 172.21.15.141:0/1869300886 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf74004d00 0x7fcf9414e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.131+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 >> 172.21.15.141:0/1869300886 conn(0x7fcf940cf700 msgr2=0x7fcf94157ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.132+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 shutdown_connections 2024-01-16T14:57:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.132+0000 7fcf9ae00640 1 -- 172.21.15.141:0/1869300886 wait complete. 2024-01-16T14:57:03.131 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:03.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.341+0000 7efde6e00640 1 Processor -- start 2024-01-16T14:57:03.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.341+0000 7efde6e00640 1 -- start start 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.342+0000 7efde6e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde0144150 0x7efde006a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.342+0000 7efde6e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7efde006a920 0x7efde006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.343+0000 7efde6e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde006ad80 con 0x7efde006ef90 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.343+0000 7efde6e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00cc8b0 con 0x7efde006a920 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.343+0000 7efde6e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00ccb90 con 0x7efde0144150 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.343+0000 7efde6400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7efde006ef90 0x7efde00ce8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59528/0 (socket says 172.21.15.141:59528) 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.343+0000 7efde6400640 1 -- 172.21.15.141:0/554906740 learned_addr learned my addr 172.21.15.141:0/554906740 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.343+0000 7efde5a00640 1 --2- 172.21.15.141:0/554906740 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde0144150 0x7efde006a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.343+0000 7efde5000640 1 --2- 172.21.15.141:0/554906740 >> v2:172.21.15.188:3300/0 conn(0x7efde006a920 0x7efde006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.344+0000 7efde5a00640 1 -- 172.21.15.141:0/554906740 >> v2:172.21.15.188:3300/0 conn(0x7efde006a920 msgr2=0x7efde006ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:03.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.344+0000 7efde5a00640 1 --2- 172.21.15.141:0/554906740 >> v2:172.21.15.188:3300/0 conn(0x7efde006a920 0x7efde006ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.344+0000 7efde5a00640 1 -- 172.21.15.141:0/554906740 >> v1:172.21.15.141:6789/0 conn(0x7efde006ef90 legacy=0x7efde00ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.344+0000 7efde5a00640 1 -- 172.21.15.141:0/554906740 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00cce70 con 0x7efde0144150 2024-01-16T14:57:03.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.345+0000 7efde5a00640 1 --2- 172.21.15.141:0/554906740 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde0144150 0x7efde006a3e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efdd00097a0 tx=0x7efdd002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59e7ff5834bb7be7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.348+0000 7efddfe00640 1 -- 172.21.15.141:0/554906740 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0031830 con 0x7efde0144150 2024-01-16T14:57:03.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.348+0000 7efddfe00640 1 -- 172.21.15.141:0/554906740 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd0031990 con 0x7efde0144150 2024-01-16T14:57:03.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.348+0000 7efddfe00640 1 -- 172.21.15.141:0/554906740 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0031d20 con 0x7efde0144150 2024-01-16T14:57:03.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 -- 172.21.15.141:0/554906740 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde0144150 msgr2=0x7efde006a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 --2- 172.21.15.141:0/554906740 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde0144150 0x7efde006a3e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efdd00097a0 tx=0x7efdd002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 -- 172.21.15.141:0/554906740 shutdown_connections 2024-01-16T14:57:03.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 --2- 172.21.15.141:0/554906740 >> v2:172.21.15.188:3300/0 conn(0x7efde006a920 0x7efde006ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 --2- 172.21.15.141:0/554906740 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde0144150 0x7efde006a3e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 -- 172.21.15.141:0/554906740 >> 172.21.15.141:0/554906740 conn(0x7efde00b1d40 msgr2=0x7efde00b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:03.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 -- 172.21.15.141:0/554906740 shutdown_connections 2024-01-16T14:57:03.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.349+0000 7efde6e00640 1 -- 172.21.15.141:0/554906740 wait complete. 2024-01-16T14:57:03.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.351+0000 7efde6e00640 1 Processor -- start 2024-01-16T14:57:03.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.351+0000 7efde6e00640 1 -- start start 2024-01-16T14:57:03.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.352+0000 7efde6e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efde006a920 0x7efde01400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.352+0000 7efde6e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde006ef90 0x7efde0140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.352+0000 7efde6e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efde0144150 0x7efde0142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.352+0000 7efde6e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00c9740 con 0x7efde0144150 2024-01-16T14:57:03.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.352+0000 7efde6e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b4960 con 0x7efde006a920 2024-01-16T14:57:03.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.352+0000 7efde6e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00c99e0 con 0x7efde006ef90 2024-01-16T14:57:03.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.352+0000 7efde6400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efde0144150 0x7efde0142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efde0144150 0x7efde0142000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59032/0 (socket says 172.21.15.141:59032) 2024-01-16T14:57:03.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 -- 172.21.15.141:0/618193811 learned_addr learned my addr 172.21.15.141:0/618193811 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:03.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde5000640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde006ef90 0x7efde0140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde5a00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efde006a920 0x7efde01400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 -- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde006ef90 msgr2=0x7efde0140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde006ef90 0x7efde0140600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 -- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efde006a920 msgr2=0x7efde01400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efde006a920 0x7efde01400c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 -- 172.21.15.141:0/618193811 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0009400 con 0x7efde0144150 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.353+0000 7efde6400640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efde0144150 0x7efde0142000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdd800cbc0 tx=0x7efdd800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.354+0000 7efddd600640 1 -- 172.21.15.141:0/618193811 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd800ad40 con 0x7efde0144150 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.354+0000 7efddd600640 1 -- 172.21.15.141:0/618193811 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd8012400 con 0x7efde0144150 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.354+0000 7efddd600640 1 -- 172.21.15.141:0/618193811 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd800aea0 con 0x7efde0144150 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.354+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde00c9cc0 con 0x7efde0144150 2024-01-16T14:57:03.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.354+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde0142540 con 0x7efde0144150 2024-01-16T14:57:03.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.355+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde006f390 con 0x7efde0144150 2024-01-16T14:57:03.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.356+0000 7efddd600640 1 -- 172.21.15.141:0/618193811 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7efdd8007070 con 0x7efde0144150 2024-01-16T14:57:03.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.357+0000 7efddd600640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efdb806b230 0x7efdb806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.358+0000 7efddd600640 1 -- 172.21.15.141:0/618193811 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efdd8089770 con 0x7efde0144150 2024-01-16T14:57:03.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.358+0000 7efde5a00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efdb806b230 0x7efdb806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.359+0000 7efde5a00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efdb806b230 0x7efdb806d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efdd0036000 tx=0x7efdd003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.363+0000 7efddd600640 1 -- 172.21.15.141:0/618193811 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efde006f390 con 0x7efde0144150 2024-01-16T14:57:03.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.564+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efde00ca1e0 con 0x7efde0144150 2024-01-16T14:57:03.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.565+0000 7efddd600640 1 -- 172.21.15.141:0/618193811 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12898 (secure 0 0 0) 0x7efde00ca1e0 con 0x7efde0144150 2024-01-16T14:57:03.548 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:03.548 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":18,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:03.343685+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efdb806b230 msgr2=0x7efdb806d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efdb806b230 0x7efdb806d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efdd0036000 tx=0x7efdd003c040 comp rx=0 tx=0).stop 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efde0144150 msgr2=0x7efde0142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efde0144150 0x7efde0142000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdd800cbc0 tx=0x7efdd800a430 comp rx=0 tx=0).stop 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 shutdown_connections 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efdb806b230 0x7efdb806d6f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efde0144150 0x7efde0142000 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efde006ef90 0x7efde0140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 --2- 172.21.15.141:0/618193811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efde006a920 0x7efde01400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 >> 172.21.15.141:0/618193811 conn(0x7efde00b1d40 msgr2=0x7efde015df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 shutdown_connections 2024-01-16T14:57:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.568+0000 7efde6e00640 1 -- 172.21.15.141:0/618193811 wait complete. 2024-01-16T14:57:03.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:03.568 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:03.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.770+0000 7f9c98a00640 1 Processor -- start 2024-01-16T14:57:03.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.770+0000 7f9c98a00640 1 -- start start 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c98a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9c90160f80 0x7f9c90161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c98a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 0x7f9c900ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c98a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c901618a0 con 0x7f9c900cfbe0 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c98a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c900d0de0 con 0x7f9c90160f80 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c98a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900d10c0 con 0x7f9c900ce8f0 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c96a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9c900cfbe0 0x7f9c900d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59532/0 (socket says 172.21.15.141:59532) 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c96a00640 1 -- 172.21.15.141:0/1195342816 learned_addr learned my addr 172.21.15.141:0/1195342816 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.772+0000 7f9c96000640 1 --2- 172.21.15.141:0/1195342816 >> v2:172.21.15.188:3300/0 conn(0x7f9c90160f80 0x7f9c90161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c95600640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 0x7f9c900ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318378341 0 0) 0x7f9c901618a0 con 0x7f9c900cfbe0 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c70003520 con 0x7f9c900cfbe0 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1495649589 0 0) 0x7f9c70003520 con 0x7f9c900cfbe0 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 msgr2=0x7f9c900ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 0x7f9c900ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 >> v2:172.21.15.188:3300/0 conn(0x7f9c90160f80 msgr2=0x7f9c90161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 --2- 172.21.15.141:0/1195342816 >> v2:172.21.15.188:3300/0 conn(0x7f9c90160f80 0x7f9c90161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c900d13a0 con 0x7f9c900cfbe0 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9c880019f0 con 0x7f9c900cfbe0 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 >> v1:172.21.15.141:6789/0 conn(0x7f9c900cfbe0 legacy=0x7f9c900d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 0x7f9c70003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c70003ef0 0x7f9c700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70004d00 0x7f9c70005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900d0de0 con 0x7f9c70003ef0 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900d10c0 con 0x7f9c70004d00 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.773+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7000b980 con 0x7f9c900ce8f0 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.774+0000 7f9c95600640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c70003ef0 0x7f9c700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.774+0000 7f9c96000640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 0x7f9c70003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.774+0000 7f9c95600640 1 -- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 msgr2=0x7f9c70003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.774+0000 7f9c95600640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c900ce8f0 0x7f9c70003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.774+0000 7f9c95600640 1 -- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70004d00 msgr2=0x7f9c70005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.774+0000 7f9c95600640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70004d00 0x7f9c70005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.774+0000 7f9c95600640 1 -- 172.21.15.141:0/1195342816 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c70003520 con 0x7f9c70003ef0 2024-01-16T14:57:03.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.775+0000 7f9c95600640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c70003ef0 0x7f9c700047c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009ac0 tx=0x7f9c700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d803db11ea3af745 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.775+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7802c6b0 con 0x7f9c70003ef0 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.775+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7802c810 con 0x7f9c70003ef0 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.775+0000 7f9c94c00640 1 -- 172.21.15.141:0/1195342816 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7802cba0 con 0x7f9c70003ef0 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.776+0000 7f9c98a00640 1 -- 172.21.15.141:0/1195342816 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c70003ef0 msgr2=0x7f9c700047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.776+0000 7f9c98a00640 1 --2- 172.21.15.141:0/1195342816 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c70003ef0 0x7f9c700047c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009ac0 tx=0x7f9c700057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.776+0000 7f9c96a00640 1 -- 172.21.15.141:0/1195342816 reap_dead start 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.776+0000 7f9c98a00640 1 -- 172.21.15.141:0/1195342816 shutdown_connections 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.776+0000 7f9c98a00640 1 -- 172.21.15.141:0/1195342816 >> 172.21.15.141:0/1195342816 conn(0x7f9c9009c550 msgr2=0x7f9c900d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.776+0000 7f9c98a00640 1 -- 172.21.15.141:0/1195342816 shutdown_connections 2024-01-16T14:57:03.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.776+0000 7f9c98a00640 1 -- 172.21.15.141:0/1195342816 wait complete. 2024-01-16T14:57:03.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.779+0000 7f9c98a00640 1 Processor -- start 2024-01-16T14:57:03.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.779+0000 7f9c98a00640 1 -- start start 2024-01-16T14:57:03.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c98a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70003ef0 0x7f9c900b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c98a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c9015d6c0 0x7f9c900b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c98a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c901512f0 0x7f9c901517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c98a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9015e840 con 0x7f9c901512f0 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c98a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900cb680 con 0x7f9c70003ef0 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c98a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90151ce0 con 0x7f9c9015d6c0 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c96a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c901512f0 0x7f9c901517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c96a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c901512f0 0x7f9c901517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59050/0 (socket says 172.21.15.141:59050) 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c96a00640 1 -- 172.21.15.141:0/3249423085 learned_addr learned my addr 172.21.15.141:0/3249423085 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.781+0000 7f9c96000640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70003ef0 0x7f9c900b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c96a00640 1 -- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c9015d6c0 msgr2=0x7f9c900b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c95600640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c9015d6c0 0x7f9c900b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c96a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c9015d6c0 0x7f9c900b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c96a00640 1 -- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70003ef0 msgr2=0x7f9c900b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c96a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70003ef0 0x7f9c900b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c96a00640 1 -- 172.21.15.141:0/3249423085 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9015dbc0 con 0x7f9c901512f0 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c96000640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70003ef0 0x7f9c900b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:03.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c95600640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c9015d6c0 0x7f9c900b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c96a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c901512f0 0x7f9c901517a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9c9008b520 tx=0x7f9c8800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c8e000640 1 -- 172.21.15.141:0/3249423085 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8800a610 con 0x7f9c901512f0 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c8e000640 1 -- 172.21.15.141:0/3249423085 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8800a790 con 0x7f9c901512f0 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c8e000640 1 -- 172.21.15.141:0/3249423085 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8800ab40 con 0x7f9c901512f0 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9015dea0 con 0x7f9c901512f0 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.782+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9015e460 con 0x7f9c901512f0 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.784+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c900b39f0 con 0x7f9c901512f0 2024-01-16T14:57:03.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.785+0000 7f9c8e000640 1 -- 172.21.15.141:0/3249423085 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f9c8800aca0 con 0x7f9c901512f0 2024-01-16T14:57:03.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.786+0000 7f9c8e000640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9c6806b260 0x7f9c6806d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:03.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.786+0000 7f9c8e000640 1 -- 172.21.15.141:0/3249423085 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9c880890a0 con 0x7f9c901512f0 2024-01-16T14:57:03.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.787+0000 7f9c96000640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9c6806b260 0x7f9c6806d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:03.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.788+0000 7f9c96000640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9c6806b260 0x7f9c6806d720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9c900cd7e0 tx=0x7f9c80008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:03.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.790+0000 7f9c8e000640 1 -- 172.21.15.141:0/3249423085 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c900b39f0 con 0x7f9c901512f0 2024-01-16T14:57:03.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.980+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c9015e980 con 0x7f9c901512f0 2024-01-16T14:57:03.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.980+0000 7f9c8e000640 1 -- 172.21.15.141:0/3249423085 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c9015e980 con 0x7f9c901512f0 2024-01-16T14:57:03.963 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9c6806b260 msgr2=0x7f9c6806d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9c6806b260 0x7f9c6806d720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9c900cd7e0 tx=0x7f9c80008040 comp rx=0 tx=0).stop 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c901512f0 msgr2=0x7f9c901517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c901512f0 0x7f9c901517a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9c9008b520 tx=0x7f9c8800cf00 comp rx=0 tx=0).stop 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 shutdown_connections 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9c6806b260 0x7f9c6806d720 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c901512f0 0x7f9c901517a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c9015d6c0 0x7f9c900b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 --2- 172.21.15.141:0/3249423085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70003ef0 0x7f9c900b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:03.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 >> 172.21.15.141:0/3249423085 conn(0x7f9c9009c550 msgr2=0x7f9c900ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:03.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 shutdown_connections 2024-01-16T14:57:03.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:03.983+0000 7f9c98a00640 1 -- 172.21.15.141:0/3249423085 wait complete. 2024-01-16T14:57:03.984 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T14:57:04.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.204+0000 7fa9dd600640 1 Processor -- start 2024-01-16T14:57:04.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.204+0000 7fa9dd600640 1 -- start start 2024-01-16T14:57:04.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9dd600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa9d813df80 0x7fa9d815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9dd600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa9d813e360 0x7fa9d80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9dd600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d806c470 con 0x7fa9d815f9c0 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9dd600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d806c750 con 0x7fa9d813df80 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9dd600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d806ca30 con 0x7fa9d813e360 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa9d813df80 0x7fa9d815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa9d813df80 0x7fa9d815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59052/0 (socket says 172.21.15.141:59052) 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6a00640 1 -- 172.21.15.141:0/516504165 learned_addr learned my addr 172.21.15.141:0/516504165 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6a00640 1 -- 172.21.15.141:0/516504165 >> v2:172.21.15.188:3300/0 conn(0x7fa9d813e360 msgr2=0x7fa9d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6000640 1 --2- 172.21.15.141:0/516504165 >> v2:172.21.15.188:3300/0 conn(0x7fa9d813e360 0x7fa9d80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6a00640 1 --2- 172.21.15.141:0/516504165 >> v2:172.21.15.188:3300/0 conn(0x7fa9d813e360 0x7fa9d80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6a00640 1 -- 172.21.15.141:0/516504165 >> v1:172.21.15.188:6789/0 conn(0x7fa9d815f9c0 legacy=0x7fa9d81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6a00640 1 -- 172.21.15.141:0/516504165 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d806cd10 con 0x7fa9d813df80 2024-01-16T14:57:04.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.206+0000 7fa9d6000640 1 --2- 172.21.15.141:0/516504165 >> v2:172.21.15.188:3300/0 conn(0x7fa9d813e360 0x7fa9d80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:04.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.208+0000 7fa9d6a00640 1 --2- 172.21.15.141:0/516504165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813df80 0x7fa9d815bee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9d806ded0 tx=0x7fa9c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64021cfffd7510d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.208+0000 7fa9d5600640 1 -- 172.21.15.141:0/516504165 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8031870 con 0x7fa9d813df80 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.208+0000 7fa9d5600640 1 -- 172.21.15.141:0/516504165 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c80319d0 con 0x7fa9d813df80 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.208+0000 7fa9d5600640 1 -- 172.21.15.141:0/516504165 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8031d60 con 0x7fa9d813df80 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.208+0000 7fa9dd600640 1 -- 172.21.15.141:0/516504165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813df80 msgr2=0x7fa9d815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.208+0000 7fa9dd600640 1 --2- 172.21.15.141:0/516504165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813df80 0x7fa9d815bee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9d806ded0 tx=0x7fa9c802b910 comp rx=0 tx=0).stop 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.209+0000 7fa9dd600640 1 -- 172.21.15.141:0/516504165 shutdown_connections 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.209+0000 7fa9dd600640 1 --2- 172.21.15.141:0/516504165 >> v2:172.21.15.188:3300/0 conn(0x7fa9d813e360 0x7fa9d80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.209+0000 7fa9dd600640 1 --2- 172.21.15.141:0/516504165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813df80 0x7fa9d815bee0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.209+0000 7fa9dd600640 1 -- 172.21.15.141:0/516504165 >> 172.21.15.141:0/516504165 conn(0x7fa9d806a800 msgr2=0x7fa9d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.209+0000 7fa9dd600640 1 -- 172.21.15.141:0/516504165 shutdown_connections 2024-01-16T14:57:04.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.209+0000 7fa9dd600640 1 -- 172.21.15.141:0/516504165 wait complete. 2024-01-16T14:57:04.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.210+0000 7fa9dd600640 1 Processor -- start 2024-01-16T14:57:04.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.211+0000 7fa9dd600640 1 -- start start 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.211+0000 7fa9dd600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9d813df80 0x7fa9d81f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.211+0000 7fa9dd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813e360 0x7fa9d813d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9dd600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9d815f9c0 0x7fa9d814dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9dd600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d81f4970 con 0x7fa9d813e360 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9dd600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d81f30b0 con 0x7fa9d815f9c0 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9dd600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d81f3390 con 0x7fa9d813df80 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813e360 0x7fa9d813d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813e360 0x7fa9d813d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59066/0 (socket says 172.21.15.141:59066) 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 -- 172.21.15.141:0/2693580770 learned_addr learned my addr 172.21.15.141:0/2693580770 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 -- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9d813df80 msgr2=0x7fa9d81f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6a00640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9d813df80 0x7fa9d81f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9d813df80 0x7fa9d81f4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 -- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9d815f9c0 msgr2=0x7fa9d814dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d7400640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9d815f9c0 0x7fa9d814dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9d815f9c0 0x7fa9d814dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 -- 172.21.15.141:0/2693580770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c80094d0 con 0x7fa9d813e360 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.212+0000 7fa9d6000640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813e360 0x7fa9d813d240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000c7f0 tx=0x7fa9c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.213+0000 7fa9cea00640 1 -- 172.21.15.141:0/2693580770 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000cea0 con 0x7fa9d813e360 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.213+0000 7fa9cea00640 1 -- 172.21.15.141:0/2693580770 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c0004230 con 0x7fa9d813e360 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.213+0000 7fa9cea00640 1 -- 172.21.15.141:0/2693580770 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0007830 con 0x7fa9d813e360 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.213+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d81f3670 con 0x7fa9d813e360 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.213+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d81f3c30 con 0x7fa9d813e360 2024-01-16T14:57:04.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.214+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d80675d0 con 0x7fa9d813e360 2024-01-16T14:57:04.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.215+0000 7fa9cea00640 1 -- 172.21.15.141:0/2693580770 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fa9c0007990 con 0x7fa9d813e360 2024-01-16T14:57:04.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.216+0000 7fa9cea00640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa9b006b260 0x7fa9b006d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.216+0000 7fa9cea00640 1 -- 172.21.15.141:0/2693580770 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa9c0089660 con 0x7fa9d813e360 2024-01-16T14:57:04.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.216+0000 7fa9d6a00640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa9b006b260 0x7fa9b006d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.218+0000 7fa9d6a00640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa9b006b260 0x7fa9b006d720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8036000 tx=0x7fa9c803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:04.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.220+0000 7fa9cea00640 1 -- 172.21.15.141:0/2693580770 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9c00548f0 con 0x7fa9d813e360 2024-01-16T14:57:04.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.412+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa9d815aec0 con 0x7fa9d813e360 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.413+0000 7fa9cea00640 1 -- 172.21.15.141:0/2693580770 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa9c001e090 con 0x7fa9d813e360 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:57:04.396 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T14:57:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa9b006b260 msgr2=0x7fa9b006d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa9b006b260 0x7fa9b006d720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8036000 tx=0x7fa9c803c040 comp rx=0 tx=0).stop 2024-01-16T14:57:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813e360 msgr2=0x7fa9d813d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813e360 0x7fa9d813d240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000c7f0 tx=0x7fa9c000ccb0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 shutdown_connections 2024-01-16T14:57:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa9b006b260 0x7fa9b006d720 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9d815f9c0 0x7fa9d814dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9d813e360 0x7fa9d813d240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 --2- 172.21.15.141:0/2693580770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9d813df80 0x7fa9d81f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 >> 172.21.15.141:0/2693580770 conn(0x7fa9d806a800 msgr2=0x7fa9d80af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:04.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 shutdown_connections 2024-01-16T14:57:04.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.416+0000 7fa9dd600640 1 -- 172.21.15.141:0/2693580770 wait complete. 2024-01-16T14:57:04.418 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T14:57:04.418 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T14:57:04.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.642+0000 7f7f5fa00640 1 Processor -- start 2024-01-16T14:57:04.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.643+0000 7f7f5fa00640 1 -- start start 2024-01-16T14:57:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.644+0000 7f7f5fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f580d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.644+0000 7f7f5fa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f580d2180 con 0x7f7f580b3890 2024-01-16T14:57:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.644+0000 7f7f5fa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f580b3400 con 0x7f7f5815d610 2024-01-16T14:57:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.644+0000 7f7f5fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580d1390 con 0x7f7f580d17e0 2024-01-16T14:57:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f5da00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7f580b3890 0x7f7f580b3d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59548/0 (socket says 172.21.15.141:59548) 2024-01-16T14:57:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f5da00640 1 -- 172.21.15.141:0/1062189195 learned_addr learned my addr 172.21.15.141:0/1062189195 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f5d000640 1 --1- 172.21.15.141:0/1062189195 >> v1:172.21.15.188:6789/0 conn(0x7f7f5815d610 0x7f7f580ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:36346/0 (socket says 172.21.15.141:36346) 2024-01-16T14:57:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f4fc00640 1 --2- 172.21.15.141:0/1062189195 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f580d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f57e00640 1 -- 172.21.15.141:0/1062189195 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785704899 0 0) 0x7f7f580d2180 con 0x7f7f580b3890 2024-01-16T14:57:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f57e00640 1 -- 172.21.15.141:0/1062189195 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f38003520 con 0x7f7f580b3890 2024-01-16T14:57:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f57e00640 1 -- 172.21.15.141:0/1062189195 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446968731 0 0) 0x7f7f580b3400 con 0x7f7f5815d610 2024-01-16T14:57:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f4fc00640 1 -- 172.21.15.141:0/1062189195 >> v1:172.21.15.188:6789/0 conn(0x7f7f5815d610 legacy=0x7f7f580ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f4fc00640 1 -- 172.21.15.141:0/1062189195 >> v1:172.21.15.141:6789/0 conn(0x7f7f580b3890 legacy=0x7f7f580b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.645+0000 7f7f4fc00640 1 -- 172.21.15.141:0/1062189195 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f580b27f0 con 0x7f7f580d17e0 2024-01-16T14:57:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.647+0000 7f7f4fc00640 1 --2- 172.21.15.141:0/1062189195 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f580d1c40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f400097d0 tx=0x7f7f4002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb903fdd0879684 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.647+0000 7f7f57e00640 1 -- 172.21.15.141:0/1062189195 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40031890 con 0x7f7f580d17e0 2024-01-16T14:57:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.647+0000 7f7f57e00640 1 -- 172.21.15.141:0/1062189195 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f400319f0 con 0x7f7f580d17e0 2024-01-16T14:57:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.647+0000 7f7f57e00640 1 -- 172.21.15.141:0/1062189195 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40031d80 con 0x7f7f580d17e0 2024-01-16T14:57:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.648+0000 7f7f5fa00640 1 -- 172.21.15.141:0/1062189195 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 msgr2=0x7f7f580d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.648+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/1062189195 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f580d1c40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f400097d0 tx=0x7f7f4002b930 comp rx=0 tx=0).stop 2024-01-16T14:57:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.648+0000 7f7f5fa00640 1 -- 172.21.15.141:0/1062189195 shutdown_connections 2024-01-16T14:57:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.648+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/1062189195 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f580d1c40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.648+0000 7f7f5fa00640 1 -- 172.21.15.141:0/1062189195 >> 172.21.15.141:0/1062189195 conn(0x7f7f5809c540 msgr2=0x7f7f580d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.648+0000 7f7f5fa00640 1 -- 172.21.15.141:0/1062189195 shutdown_connections 2024-01-16T14:57:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.648+0000 7f7f5fa00640 1 -- 172.21.15.141:0/1062189195 wait complete. 2024-01-16T14:57:04.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.650+0000 7f7f5fa00640 1 Processor -- start 2024-01-16T14:57:04.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.650+0000 7f7f5fa00640 1 -- start start 2024-01-16T14:57:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5fa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f580b3890 0x7f7f580cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f58149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5fa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f5815d610 0x7f7f581f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5fa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5814a0f0 con 0x7f7f580b3890 2024-01-16T14:57:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5fa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580cbdc0 con 0x7f7f5815d610 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580cc0a0 con 0x7f7f580d17e0 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f580b3890 0x7f7f580cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f580b3890 0x7f7f580cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59082/0 (socket says 172.21.15.141:59082) 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 -- 172.21.15.141:0/208711336 learned_addr learned my addr 172.21.15.141:0/208711336 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f4fc00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f58149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5da00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f5815d610 0x7f7f581f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 -- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 msgr2=0x7f7f58149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f58149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 -- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f5815d610 msgr2=0x7f7f581f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f5815d610 0x7f7f581f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5d000640 1 -- 172.21.15.141:0/208711336 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40009400 con 0x7f7f580b3890 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f4fc00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f58149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.652+0000 7f7f5da00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f5815d610 0x7f7f581f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.653+0000 7f7f5d000640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f580b3890 0x7f7f580cce60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800c430 tx=0x7f7f4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:04.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.653+0000 7f7f55600640 1 -- 172.21.15.141:0/208711336 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f480074e0 con 0x7f7f580b3890 2024-01-16T14:57:04.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.653+0000 7f7f55600640 1 -- 172.21.15.141:0/208711336 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f48007660 con 0x7f7f580b3890 2024-01-16T14:57:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.653+0000 7f7f55600640 1 -- 172.21.15.141:0/208711336 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48004a20 con 0x7f7f580b3890 2024-01-16T14:57:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.653+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f580cc380 con 0x7f7f580b3890 2024-01-16T14:57:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.653+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f580cc940 con 0x7f7f580b3890 2024-01-16T14:57:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.654+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f580cd3a0 con 0x7f7f580b3890 2024-01-16T14:57:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.655+0000 7f7f55600640 1 -- 172.21.15.141:0/208711336 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f7f48004b80 con 0x7f7f580b3890 2024-01-16T14:57:04.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.656+0000 7f7f55600640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f3006b230 0x7f7f3006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:04.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.656+0000 7f7f55600640 1 -- 172.21.15.141:0/208711336 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f48089690 con 0x7f7f580b3890 2024-01-16T14:57:04.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.657+0000 7f7f4fc00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f3006b230 0x7f7f3006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:04.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.658+0000 7f7f4fc00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f3006b230 0x7f7f3006d6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f40036000 tx=0x7f7f4003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:04.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.661+0000 7f7f55600640 1 -- 172.21.15.141:0/208711336 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f48054890 con 0x7f7f580b3890 2024-01-16T14:57:04.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:04.849+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7f580ccb30 con 0x7f7f580b3890 2024-01-16T14:57:05.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.506+0000 7f7f55600640 1 -- 172.21.15.141:0/208711336 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7f48057eb0 con 0x7f7f580b3890 2024-01-16T14:57:05.490 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T14:57:05.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f3006b230 msgr2=0x7f7f3006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:05.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f3006b230 0x7f7f3006d6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f40036000 tx=0x7f7f4003c040 comp rx=0 tx=0).stop 2024-01-16T14:57:05.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f580b3890 msgr2=0x7f7f580cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:05.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f580b3890 0x7f7f580cce60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800c430 tx=0x7f7f4800c8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 shutdown_connections 2024-01-16T14:57:05.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f3006b230 0x7f7f3006d6f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f5815d610 0x7f7f581f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f580d17e0 0x7f7f58149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 --2- 172.21.15.141:0/208711336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f580b3890 0x7f7f580cce60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.511+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 >> 172.21.15.141:0/208711336 conn(0x7f7f5809c540 msgr2=0x7f7f5815c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:05.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.512+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 shutdown_connections 2024-01-16T14:57:05.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.512+0000 7f7f5fa00640 1 -- 172.21.15.141:0/208711336 wait complete. 2024-01-16T14:57:05.521 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T14:57:05.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.731+0000 7f28a6c00640 1 Processor -- start 2024-01-16T14:57:05.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.731+0000 7f28a6c00640 1 -- start start 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.732+0000 7f28a6c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f28a015d640 0x7f28a00d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f28a6c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a00b22a0 con 0x7f28a00d0140 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f28a6c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a00afec0 con 0x7f28a015da20 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f28a6c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a00b01a0 con 0x7f28a015d640 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f289fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f28a015d640 0x7f28a00d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f289fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f28a015d640 0x7f28a00d1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59096/0 (socket says 172.21.15.141:59096) 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f289fe00640 1 -- 172.21.15.141:0/3713142390 learned_addr learned my addr 172.21.15.141:0/3713142390 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f28a4c00640 1 --1- 172.21.15.141:0/3713142390 >> v1:172.21.15.141:6789/0 conn(0x7f28a00d0140 0x7f28a00b0f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59550/0 (socket says 172.21.15.141:59550) 2024-01-16T14:57:05.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f289fe00640 1 -- 172.21.15.141:0/3713142390 >> v1:172.21.15.188:6789/0 conn(0x7f28a015da20 legacy=0x7f28a00cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:05.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f289fe00640 1 -- 172.21.15.141:0/3713142390 >> v1:172.21.15.141:6789/0 conn(0x7f28a00d0140 legacy=0x7f28a00b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:05.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.733+0000 7f289fe00640 1 -- 172.21.15.141:0/3713142390 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a00b0480 con 0x7f28a015d640 2024-01-16T14:57:05.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.734+0000 7f289fe00640 1 --2- 172.21.15.141:0/3713142390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015d640 0x7f28a00d1b40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2890009870 tx=0x7f289002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2c681c681e7661 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:05.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.735+0000 7f289ea00640 1 -- 172.21.15.141:0/3713142390 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289002f850 con 0x7f28a015d640 2024-01-16T14:57:05.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.735+0000 7f289ea00640 1 -- 172.21.15.141:0/3713142390 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289002f9b0 con 0x7f28a015d640 2024-01-16T14:57:05.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.735+0000 7f289ea00640 1 -- 172.21.15.141:0/3713142390 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289002fd40 con 0x7f28a015d640 2024-01-16T14:57:05.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.735+0000 7f28a6c00640 1 -- 172.21.15.141:0/3713142390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015d640 msgr2=0x7f28a00d1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:05.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.735+0000 7f28a6c00640 1 --2- 172.21.15.141:0/3713142390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015d640 0x7f28a00d1b40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2890009870 tx=0x7f289002ba30 comp rx=0 tx=0).stop 2024-01-16T14:57:05.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.736+0000 7f28a6c00640 1 -- 172.21.15.141:0/3713142390 shutdown_connections 2024-01-16T14:57:05.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.736+0000 7f28a6c00640 1 --2- 172.21.15.141:0/3713142390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015d640 0x7f28a00d1b40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.736+0000 7f28a6c00640 1 -- 172.21.15.141:0/3713142390 >> 172.21.15.141:0/3713142390 conn(0x7f28a009c560 msgr2=0x7f28a00d3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:05.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.736+0000 7f28a6c00640 1 -- 172.21.15.141:0/3713142390 shutdown_connections 2024-01-16T14:57:05.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.736+0000 7f28a6c00640 1 -- 172.21.15.141:0/3713142390 wait complete. 2024-01-16T14:57:05.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.737+0000 7f28a6c00640 1 Processor -- start 2024-01-16T14:57:05.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.738+0000 7f28a6c00640 1 -- start start 2024-01-16T14:57:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a6c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a00d0140 0x7f28a00c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a6c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a015d640 0x7f28a0151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a6c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015da20 0x7f28a00ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a6c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a0151c80 con 0x7f28a015da20 2024-01-16T14:57:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a6c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00cd0a0 con 0x7f28a015d640 2024-01-16T14:57:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a6c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00cbac0 con 0x7f28a00d0140 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015da20 0x7f28a00ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015da20 0x7f28a00ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59112/0 (socket says 172.21.15.141:59112) 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 -- 172.21.15.141:0/2096333894 learned_addr learned my addr 172.21.15.141:0/2096333894 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f289fe00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a00d0140 0x7f28a00c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f289f400640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a015d640 0x7f28a0151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 -- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a00d0140 msgr2=0x7f28a00c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a00d0140 0x7f28a00c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 -- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a015d640 msgr2=0x7f28a0151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a015d640 0x7f28a0151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f28a4c00640 1 -- 172.21.15.141:0/2096333894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28900094d0 con 0x7f28a015da20 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.739+0000 7f289f400640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a015d640 0x7f28a0151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:05.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.740+0000 7f28a4c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015da20 0x7f28a00ccb60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f289800c7b0 tx=0x7f289800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:05.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.740+0000 7f2897e00640 1 -- 172.21.15.141:0/2096333894 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289800a640 con 0x7f28a015da20 2024-01-16T14:57:05.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.740+0000 7f2897e00640 1 -- 172.21.15.141:0/2096333894 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289800a7a0 con 0x7f28a015da20 2024-01-16T14:57:05.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.740+0000 7f2897e00640 1 -- 172.21.15.141:0/2096333894 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289800ab10 con 0x7f28a015da20 2024-01-16T14:57:05.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.740+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a00cbda0 con 0x7f28a015da20 2024-01-16T14:57:05.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.740+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a00cc360 con 0x7f28a015da20 2024-01-16T14:57:05.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.741+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a0067670 con 0x7f28a015da20 2024-01-16T14:57:05.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.742+0000 7f2897e00640 1 -- 172.21.15.141:0/2096333894 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f289800ac70 con 0x7f28a015da20 2024-01-16T14:57:05.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.743+0000 7f2897e00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f287806b2c0 0x7f287806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:05.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.743+0000 7f2897e00640 1 -- 172.21.15.141:0/2096333894 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2898089180 con 0x7f28a015da20 2024-01-16T14:57:05.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.744+0000 7f289fe00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f287806b2c0 0x7f287806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:05.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.745+0000 7f289fe00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f287806b2c0 0x7f287806d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2890036000 tx=0x7f2890008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:05.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.750+0000 7f2897e00640 1 -- 172.21.15.141:0/2096333894 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28980541e0 con 0x7f28a015da20 2024-01-16T14:57:05.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:05.942+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f28a00c6ac0 con 0x7f28a015da20 2024-01-16T14:57:06.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.517+0000 7f2897e00640 1 -- 172.21.15.141:0/2096333894 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2898057800 con 0x7f28a015da20 2024-01-16T14:57:06.500 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.520+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f287806b2c0 msgr2=0x7f287806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.520+0000 7f28a6c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f287806b2c0 0x7f287806d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2890036000 tx=0x7f2890008030 comp rx=0 tx=0).stop 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.520+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015da20 msgr2=0x7f28a00ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.520+0000 7f28a6c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015da20 0x7f28a00ccb60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f289800c7b0 tx=0x7f289800cc70 comp rx=0 tx=0).stop 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 shutdown_connections 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f287806b2c0 0x7f287806d780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28a015da20 0x7f28a00ccb60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a015d640 0x7f28a0151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 --2- 172.21.15.141:0/2096333894 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a00d0140 0x7f28a00c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 >> 172.21.15.141:0/2096333894 conn(0x7f28a009c560 msgr2=0x7f28a00b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 shutdown_connections 2024-01-16T14:57:06.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.521+0000 7f28a6c00640 1 -- 172.21.15.141:0/2096333894 wait complete. 2024-01-16T14:57:06.543 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T14:57:06.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.739+0000 7f0275000640 1 Processor -- start 2024-01-16T14:57:06.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.739+0000 7f0275000640 1 -- start start 2024-01-16T14:57:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.740+0000 7f0275000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f02700cf9a0 0x7f027015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.740+0000 7f0275000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f027015e1e0 0x7f02700702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.740+0000 7f0275000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02700b1ee0 con 0x7f0270143ae0 2024-01-16T14:57:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.740+0000 7f0275000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02700b2780 con 0x7f02700cf9a0 2024-01-16T14:57:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.740+0000 7f0275000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027015e990 con 0x7f027015e1e0 2024-01-16T14:57:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.741+0000 7f026e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f027015e1e0 0x7f02700702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.741+0000 7f026de00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f0270143ae0 0x7f02700cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:36350/0 (socket says 172.21.15.141:36350) 2024-01-16T14:57:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.741+0000 7f026de00640 1 -- 172.21.15.141:0/3561960339 learned_addr learned my addr 172.21.15.141:0/3561960339 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.741+0000 7f0265200640 1 --2- 172.21.15.141:0/3561960339 >> v2:172.21.15.188:3300/0 conn(0x7f02700cf9a0 0x7f027015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.742+0000 7f026d400640 1 -- 172.21.15.141:0/3561960339 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007774069 0 0) 0x7f02700b1ee0 con 0x7f0270143ae0 2024-01-16T14:57:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.742+0000 7f026d400640 1 -- 172.21.15.141:0/3561960339 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250003520 con 0x7f0270143ae0 2024-01-16T14:57:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.742+0000 7f026e800640 1 -- 172.21.15.141:0/3561960339 >> v2:172.21.15.188:3300/0 conn(0x7f02700cf9a0 msgr2=0x7f027015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.742+0000 7f026e800640 1 --2- 172.21.15.141:0/3561960339 >> v2:172.21.15.188:3300/0 conn(0x7f02700cf9a0 0x7f027015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.742+0000 7f026e800640 1 -- 172.21.15.141:0/3561960339 >> v1:172.21.15.188:6789/0 conn(0x7f0270143ae0 legacy=0x7f02700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.742+0000 7f026e800640 1 -- 172.21.15.141:0/3561960339 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02700b2130 con 0x7f027015e1e0 2024-01-16T14:57:06.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.743+0000 7f026e800640 1 --2- 172.21.15.141:0/3561960339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f027015e1e0 0x7f02700702c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f02680097a0 tx=0x7f026802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1519c3acc8946e4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:06.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.744+0000 7f026d400640 1 -- 172.21.15.141:0/3561960339 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026802d040 con 0x7f027015e1e0 2024-01-16T14:57:06.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.744+0000 7f026d400640 1 -- 172.21.15.141:0/3561960339 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0268031810 con 0x7f027015e1e0 2024-01-16T14:57:06.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.744+0000 7f026d400640 1 -- 172.21.15.141:0/3561960339 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268031ba0 con 0x7f027015e1e0 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 -- 172.21.15.141:0/3561960339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f027015e1e0 msgr2=0x7f02700702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 --2- 172.21.15.141:0/3561960339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f027015e1e0 0x7f02700702c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f02680097a0 tx=0x7f026802b8b0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 -- 172.21.15.141:0/3561960339 shutdown_connections 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 --2- 172.21.15.141:0/3561960339 >> v2:172.21.15.188:3300/0 conn(0x7f02700cf9a0 0x7f027015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 --2- 172.21.15.141:0/3561960339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f027015e1e0 0x7f02700702c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 -- 172.21.15.141:0/3561960339 >> 172.21.15.141:0/3561960339 conn(0x7f027006a800 msgr2=0x7f027006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 -- 172.21.15.141:0/3561960339 shutdown_connections 2024-01-16T14:57:06.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.745+0000 7f0275000640 1 -- 172.21.15.141:0/3561960339 wait complete. 2024-01-16T14:57:06.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.747+0000 7f0275000640 1 Processor -- start 2024-01-16T14:57:06.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.747+0000 7f0275000640 1 -- start start 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f0275000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02700cf9a0 0x7f02700aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f0275000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0270143ae0 0x7f0270159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f0275000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f027015e1e0 0x7f02700b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f0275000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270131750 con 0x7f027015e1e0 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f0275000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270131a30 con 0x7f02700cf9a0 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f0275000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0270131d10 con 0x7f0270143ae0 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f026e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f027015e1e0 0x7f02700b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f026e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f027015e1e0 0x7f02700b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59114/0 (socket says 172.21.15.141:59114) 2024-01-16T14:57:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f026e800640 1 -- 172.21.15.141:0/2281471272 learned_addr learned my addr 172.21.15.141:0/2281471272 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:06.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.748+0000 7f026de00640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02700cf9a0 0x7f02700aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f0265200640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0270143ae0 0x7f0270159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f026e800640 1 -- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0270143ae0 msgr2=0x7f0270159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f026e800640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0270143ae0 0x7f0270159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f026e800640 1 -- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02700cf9a0 msgr2=0x7f02700aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f026e800640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02700cf9a0 0x7f02700aeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f026e800640 1 -- 172.21.15.141:0/2281471272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268009400 con 0x7f027015e1e0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f026e800640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f027015e1e0 0x7f02700b1280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f02680097a0 tx=0x7f02680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f0266a00640 1 -- 172.21.15.141:0/2281471272 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026803ac60 con 0x7f027015e1e0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f0266a00640 1 -- 172.21.15.141:0/2281471272 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026803ade0 con 0x7f027015e1e0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f0266a00640 1 -- 172.21.15.141:0/2281471272 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02680443f0 con 0x7f027015e1e0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.749+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0270131ff0 con 0x7f027015e1e0 2024-01-16T14:57:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.750+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0270159dd0 con 0x7f027015e1e0 2024-01-16T14:57:06.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.751+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0270132510 con 0x7f027015e1e0 2024-01-16T14:57:06.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.751+0000 7f0266a00640 1 -- 172.21.15.141:0/2281471272 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f026804d450 con 0x7f027015e1e0 2024-01-16T14:57:06.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.753+0000 7f0266a00640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f024806b2c0 0x7f024806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:06.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.753+0000 7f0266a00640 1 -- 172.21.15.141:0/2281471272 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f02680abc30 con 0x7f027015e1e0 2024-01-16T14:57:06.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.754+0000 7f026de00640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f024806b2c0 0x7f024806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:06.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.755+0000 7f026de00640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f024806b2c0 0x7f024806d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0260001560 tx=0x7f0260009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:06.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.760+0000 7f0266a00640 1 -- 172.21.15.141:0/2281471272 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0270132510 con 0x7f027015e1e0 2024-01-16T14:57:06.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:06.985+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0270132510 con 0x7f027015e1e0 2024-01-16T14:57:07.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.520+0000 7f0266a00640 1 -- 172.21.15.141:0/2281471272 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0270132510 con 0x7f027015e1e0 2024-01-16T14:57:07.503 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 2 and data pool 3 2024-01-16T14:57:07.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.523+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f024806b2c0 msgr2=0x7f024806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.523+0000 7f0275000640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f024806b2c0 0x7f024806d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0260001560 tx=0x7f0260009040 comp rx=0 tx=0).stop 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f027015e1e0 msgr2=0x7f02700b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f027015e1e0 0x7f02700b1280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f02680097a0 tx=0x7f02680046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 shutdown_connections 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f024806b2c0 0x7f024806d780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f027015e1e0 0x7f02700b1280 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0270143ae0 0x7f0270159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 --2- 172.21.15.141:0/2281471272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02700cf9a0 0x7f02700aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 >> 172.21.15.141:0/2281471272 conn(0x7f027006a800 msgr2=0x7f027015c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 shutdown_connections 2024-01-16T14:57:07.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.524+0000 7f0275000640 1 -- 172.21.15.141:0/2281471272 wait complete. 2024-01-16T14:57:07.534 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:07.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.722+0000 7f327e600640 1 Processor -- start 2024-01-16T14:57:07.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.722+0000 7f327e600640 1 -- start start 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.723+0000 7f327e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 0x7f327815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.723+0000 7f327e600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f32780b3ae0 0x7f327806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.723+0000 7f327e600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3278070c40 con 0x7f327806d120 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.723+0000 7f327e600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3278071450 con 0x7f32780b3ae0 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.723+0000 7f327e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327806c470 con 0x7f32780ae5a0 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3277e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f327806d120 0x7f32780cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59566/0 (socket says 172.21.15.141:59566) 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3277e00640 1 -- 172.21.15.141:0/1694556937 learned_addr learned my addr 172.21.15.141:0/1694556937 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3277400640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 0x7f327815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f326e800640 1 --2- 172.21.15.141:0/1694556937 >> v2:172.21.15.188:3300/0 conn(0x7f32780b3ae0 0x7f327806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377325438 0 0) 0x7f3278070c40 con 0x7f327806d120 2024-01-16T14:57:07.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3258003520 con 0x7f327806d120 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3448484476 0 0) 0x7f3258003520 con 0x7f327806d120 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 msgr2=0x7f327815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 0x7f327815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 >> v2:172.21.15.188:3300/0 conn(0x7f32780b3ae0 msgr2=0x7f327806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 --2- 172.21.15.141:0/1694556937 >> v2:172.21.15.188:3300/0 conn(0x7f32780b3ae0 0x7f327806cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327806c750 con 0x7f327806d120 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.724+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f32700019f0 con 0x7f327806d120 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 >> v1:172.21.15.141:6789/0 conn(0x7f327806d120 legacy=0x7f32780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3276a00640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3258003970 0x7f3258003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3276a00640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 0x7f32580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3276a00640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3258004d00 0x7f3258005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278071450 con 0x7f3258003970 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327806c470 con 0x7f3258004d00 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325800b980 con 0x7f32780ae5a0 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3277400640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3258003970 0x7f3258003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f326e800640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 0x7f32580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3277400640 1 -- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 msgr2=0x7f32580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3277400640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32780ae5a0 0x7f32580047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3277400640 1 -- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3258004d00 msgr2=0x7f3258005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3277400640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3258004d00 0x7f3258005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.725+0000 7f3277400640 1 -- 172.21.15.141:0/1694556937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3258003520 con 0x7f3258003970 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.726+0000 7f3277400640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3258003970 0x7f3258003d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3268009ac0 tx=0x7f32580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18268c02b4fde7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.726+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326802c6b0 con 0x7f3258003970 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.726+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326802c810 con 0x7f3258003970 2024-01-16T14:57:07.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.726+0000 7f3276a00640 1 -- 172.21.15.141:0/1694556937 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326802cba0 con 0x7f3258003970 2024-01-16T14:57:07.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.727+0000 7f327e600640 1 -- 172.21.15.141:0/1694556937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3258003970 msgr2=0x7f3258003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.727+0000 7f327e600640 1 --2- 172.21.15.141:0/1694556937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3258003970 0x7f3258003d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3268009ac0 tx=0x7f32580057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.727+0000 7f3277e00640 1 -- 172.21.15.141:0/1694556937 reap_dead start 2024-01-16T14:57:07.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.727+0000 7f327e600640 1 -- 172.21.15.141:0/1694556937 shutdown_connections 2024-01-16T14:57:07.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.727+0000 7f327e600640 1 -- 172.21.15.141:0/1694556937 >> 172.21.15.141:0/1694556937 conn(0x7f327806a800 msgr2=0x7f327806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:07.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.727+0000 7f327e600640 1 -- 172.21.15.141:0/1694556937 shutdown_connections 2024-01-16T14:57:07.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.727+0000 7f327e600640 1 -- 172.21.15.141:0/1694556937 wait complete. 2024-01-16T14:57:07.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.729+0000 7f327e600640 1 Processor -- start 2024-01-16T14:57:07.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.729+0000 7f327e600640 1 -- start start 2024-01-16T14:57:07.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f327e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3258003970 0x7f32780cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f327e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3278159510 0x7f327814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f327e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3278159990 0x7f327814b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f327e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278156850 con 0x7f3278159510 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f327e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780cb8a0 con 0x7f3258003970 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f327e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3278156220 con 0x7f3278159990 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3278159510 0x7f327814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3278159510 0x7f327814b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59120/0 (socket says 172.21.15.141:59120) 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 -- 172.21.15.141:0/924169696 learned_addr learned my addr 172.21.15.141:0/924169696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 -- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3278159990 msgr2=0x7f327814b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3278159990 0x7f327814b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 -- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3258003970 msgr2=0x7f32780cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3258003970 0x7f32780cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.730+0000 7f326e800640 1 -- 172.21.15.141:0/924169696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780cbaf0 con 0x7f3278159510 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.731+0000 7f326e800640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3278159510 0x7f327814b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3260010910 tx=0x7f3260010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.731+0000 7f326fe00640 1 -- 172.21.15.141:0/924169696 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260009690 con 0x7f3278159510 2024-01-16T14:57:07.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.731+0000 7f326fe00640 1 -- 172.21.15.141:0/924169696 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326000ec20 con 0x7f3278159510 2024-01-16T14:57:07.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.731+0000 7f326fe00640 1 -- 172.21.15.141:0/924169696 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326001c400 con 0x7f3278159510 2024-01-16T14:57:07.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.731+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32780cbdd0 con 0x7f3278159510 2024-01-16T14:57:07.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.731+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32780cc790 con 0x7f3278159510 2024-01-16T14:57:07.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.732+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3278156ac0 con 0x7f3278159510 2024-01-16T14:57:07.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.733+0000 7f326fe00640 1 -- 172.21.15.141:0/924169696 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f326001c560 con 0x7f3278159510 2024-01-16T14:57:07.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.734+0000 7f326fe00640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f325006b2c0 0x7f325006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:07.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.735+0000 7f326fe00640 1 -- 172.21.15.141:0/924169696 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f326008a120 con 0x7f3278159510 2024-01-16T14:57:07.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.735+0000 7f3277400640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f325006b2c0 0x7f325006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:07.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.736+0000 7f3277400640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f325006b2c0 0x7f325006d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3268036000 tx=0x7f326802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:07.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.739+0000 7f326fe00640 1 -- 172.21.15.141:0/924169696 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3260055040 con 0x7f3278159510 2024-01-16T14:57:07.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.915+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3278161d60 con 0x7f3278159510 2024-01-16T14:57:07.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.918+0000 7f326fe00640 1 -- 172.21.15.141:0/924169696 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f3260058660 con 0x7f3278159510 2024-01-16T14:57:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:07.901 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":21,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:06.987935+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.920+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f325006b2c0 msgr2=0x7f325006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.920+0000 7f327e600640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f325006b2c0 0x7f325006d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3268036000 tx=0x7f326802e000 comp rx=0 tx=0).stop 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.920+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3278159510 msgr2=0x7f327814b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.920+0000 7f327e600640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3278159510 0x7f327814b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3260010910 tx=0x7f3260010dd0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.920+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 shutdown_connections 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.921+0000 7f327e600640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f325006b2c0 0x7f325006d780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.921+0000 7f327e600640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3278159990 0x7f327814b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.921+0000 7f327e600640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3278159510 0x7f327814b410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.921+0000 7f327e600640 1 --2- 172.21.15.141:0/924169696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3258003970 0x7f32780cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.921+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 >> 172.21.15.141:0/924169696 conn(0x7f327806a800 msgr2=0x7f327815e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.921+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 shutdown_connections 2024-01-16T14:57:07.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:07.921+0000 7f327e600640 1 -- 172.21.15.141:0/924169696 wait complete. 2024-01-16T14:57:07.921 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:08.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.137+0000 7efcad400640 1 Processor -- start 2024-01-16T14:57:08.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.137+0000 7efcad400640 1 -- start start 2024-01-16T14:57:08.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.138+0000 7efcad400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efca80cf9a0 0x7efca8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.138+0000 7efcad400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca80cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.138+0000 7efcad400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca8071690 con 0x7efca814b2c0 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.138+0000 7efcad400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca814b6a0 con 0x7efca80cf9a0 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.138+0000 7efcad400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca8071970 con 0x7efca8070ea0 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.138+0000 7efca5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efca80cf9a0 0x7efca8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efca80cf9a0 0x7efca8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59134/0 (socket says 172.21.15.141:59134) 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca5e00640 1 -- 172.21.15.141:0/3292329078 learned_addr learned my addr 172.21.15.141:0/3292329078 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca6800640 1 --1- 172.21.15.141:0/3292329078 >> v1:172.21.15.141:6789/0 conn(0x7efca814b2c0 0x7efca80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59580/0 (socket says 172.21.15.141:59580) 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca7200640 1 --2- 172.21.15.141:0/3292329078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca80cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca5e00640 1 -- 172.21.15.141:0/3292329078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 msgr2=0x7efca80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca5e00640 1 --2- 172.21.15.141:0/3292329078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca80cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca5e00640 1 -- 172.21.15.141:0/3292329078 >> v1:172.21.15.141:6789/0 conn(0x7efca814b2c0 legacy=0x7efca80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca5e00640 1 -- 172.21.15.141:0/3292329078 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8071c50 con 0x7efca80cf9a0 2024-01-16T14:57:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.139+0000 7efca7200640 1 --2- 172.21.15.141:0/3292329078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca80cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:08.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.140+0000 7efca5e00640 1 --2- 172.21.15.141:0/3292329078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca80cf9a0 0x7efca8070960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc90009870 tx=0x7efc9002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfaefc6833c1e5a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.140+0000 7efca5400640 1 -- 172.21.15.141:0/3292329078 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90030840 con 0x7efca80cf9a0 2024-01-16T14:57:08.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.140+0000 7efca5400640 1 -- 172.21.15.141:0/3292329078 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc900309a0 con 0x7efca80cf9a0 2024-01-16T14:57:08.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.140+0000 7efca5400640 1 -- 172.21.15.141:0/3292329078 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90030d30 con 0x7efca80cf9a0 2024-01-16T14:57:08.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 -- 172.21.15.141:0/3292329078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca80cf9a0 msgr2=0x7efca8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 --2- 172.21.15.141:0/3292329078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca80cf9a0 0x7efca8070960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc90009870 tx=0x7efc9002b980 comp rx=0 tx=0).stop 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 -- 172.21.15.141:0/3292329078 shutdown_connections 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 --2- 172.21.15.141:0/3292329078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca80cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 --2- 172.21.15.141:0/3292329078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca80cf9a0 0x7efca8070960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 -- 172.21.15.141:0/3292329078 >> 172.21.15.141:0/3292329078 conn(0x7efca806a800 msgr2=0x7efca806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 -- 172.21.15.141:0/3292329078 shutdown_connections 2024-01-16T14:57:08.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.141+0000 7efcad400640 1 -- 172.21.15.141:0/3292329078 wait complete. 2024-01-16T14:57:08.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.143+0000 7efcad400640 1 Processor -- start 2024-01-16T14:57:08.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.143+0000 7efcad400640 1 -- start start 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efcad400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca8161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efcad400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efca80cf9a0 0x7efca80ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efcad400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca814b2c0 0x7efca81425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efcad400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80cec60 con 0x7efca814b2c0 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efcad400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81624d0 con 0x7efca80cf9a0 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efcad400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca81627b0 con 0x7efca8070ea0 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca814b2c0 0x7efca81425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca814b2c0 0x7efca81425f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59150/0 (socket says 172.21.15.141:59150) 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 -- 172.21.15.141:0/1571626014 learned_addr learned my addr 172.21.15.141:0/1571626014 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 -- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 msgr2=0x7efca8161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca6800640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca8161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca8161eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 -- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efca80cf9a0 msgr2=0x7efca80ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efca80cf9a0 0x7efca80ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca7200640 1 -- 172.21.15.141:0/1571626014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc900094d0 con 0x7efca814b2c0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.144+0000 7efca6800640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca8161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.145+0000 7efca7200640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca814b2c0 0x7efca81425f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efca000cef0 tx=0x7efca000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.145+0000 7efc9ea00640 1 -- 172.21.15.141:0/1571626014 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca000d840 con 0x7efca814b2c0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.145+0000 7efc9ea00640 1 -- 172.21.15.141:0/1571626014 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca000d9c0 con 0x7efca814b2c0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.145+0000 7efc9ea00640 1 -- 172.21.15.141:0/1571626014 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca000dd70 con 0x7efca814b2c0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.145+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca8162a90 con 0x7efca814b2c0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.145+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca8163050 con 0x7efca814b2c0 2024-01-16T14:57:08.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.146+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca80675d0 con 0x7efca814b2c0 2024-01-16T14:57:08.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.147+0000 7efc9ea00640 1 -- 172.21.15.141:0/1571626014 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efca001e3b0 con 0x7efca814b2c0 2024-01-16T14:57:08.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.148+0000 7efc9ea00640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc8006b2c0 0x7efc8006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.148+0000 7efc9ea00640 1 -- 172.21.15.141:0/1571626014 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efca008a6d0 con 0x7efca814b2c0 2024-01-16T14:57:08.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.148+0000 7efca6800640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc8006b2c0 0x7efc8006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.149+0000 7efca6800640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc8006b2c0 0x7efc8006d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efc98000f60 tx=0x7efc98009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.152+0000 7efc9ea00640 1 -- 172.21.15.141:0/1571626014 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efca00555e0 con 0x7efca814b2c0 2024-01-16T14:57:08.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.328+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efca806ae80 con 0x7efca814b2c0 2024-01-16T14:57:08.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.331+0000 7efc9ea00640 1 -- 172.21.15.141:0/1571626014 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7efca0058c00 con 0x7efca814b2c0 2024-01-16T14:57:08.314 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:08.314 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":21,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:06.987935+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:08.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc8006b2c0 msgr2=0x7efc8006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc8006b2c0 0x7efc8006d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efc98000f60 tx=0x7efc98009040 comp rx=0 tx=0).stop 2024-01-16T14:57:08.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca814b2c0 msgr2=0x7efca81425f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca814b2c0 0x7efca81425f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efca000cef0 tx=0x7efca000a710 comp rx=0 tx=0).stop 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 shutdown_connections 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc8006b2c0 0x7efc8006d780 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efca814b2c0 0x7efca81425f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efca80cf9a0 0x7efca80ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 --2- 172.21.15.141:0/1571626014 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efca8070ea0 0x7efca8161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.333+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 >> 172.21.15.141:0/1571626014 conn(0x7efca806a800 msgr2=0x7efca806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.334+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 shutdown_connections 2024-01-16T14:57:08.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.334+0000 7efcad400640 1 -- 172.21.15.141:0/1571626014 wait complete. 2024-01-16T14:57:08.335 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T14:57:08.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.552+0000 7fdaa7200640 1 Processor -- start 2024-01-16T14:57:08.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.552+0000 7fdaa7200640 1 -- start start 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fdaa7200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdaa01547a0 0x7fdaa015b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fdaa7200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdaa00715a0 0x7fdaa006b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fdaa7200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa00b48c0 con 0x7fdaa006b930 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fdaa7200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa015e7b0 con 0x7fdaa01547a0 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fdaa7200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa006bd60 con 0x7fdaa00715a0 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdaa01547a0 0x7fdaa015b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdaa01547a0 0x7fdaa015b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59156/0 (socket says 172.21.15.141:59156) 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fdaa5200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdaa006b930 0x7fdaa006cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59584/0 (socket says 172.21.15.141:59584) 2024-01-16T14:57:08.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9fe00640 1 -- 172.21.15.141:0/1273617553 learned_addr learned my addr 172.21.15.141:0/1273617553 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:08.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda97200640 1 --2- 172.21.15.141:0/1273617553 >> v2:172.21.15.188:3300/0 conn(0x7fdaa00715a0 0x7fdaa006b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9fe00640 1 -- 172.21.15.141:0/1273617553 >> v2:172.21.15.188:3300/0 conn(0x7fdaa00715a0 msgr2=0x7fdaa006b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:08.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9fe00640 1 --2- 172.21.15.141:0/1273617553 >> v2:172.21.15.188:3300/0 conn(0x7fdaa00715a0 0x7fdaa006b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9fe00640 1 -- 172.21.15.141:0/1273617553 >> v1:172.21.15.141:6789/0 conn(0x7fdaa006b930 legacy=0x7fdaa006cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9fe00640 1 -- 172.21.15.141:0/1273617553 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa0071d40 con 0x7fdaa01547a0 2024-01-16T14:57:08.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.554+0000 7fda9f400640 1 -- 172.21.15.141:0/1273617553 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525622203 0 0) 0x7fdaa00b48c0 con 0x7fdaa006b930 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.555+0000 7fda9fe00640 1 --2- 172.21.15.141:0/1273617553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa01547a0 0x7fdaa015b290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdaa006c5a0 tx=0x7fda9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3592adb80b4c1970 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fda9f400640 1 -- 172.21.15.141:0/1273617553 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90031870 con 0x7fdaa01547a0 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fda9f400640 1 -- 172.21.15.141:0/1273617553 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda900319d0 con 0x7fdaa01547a0 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fda9f400640 1 -- 172.21.15.141:0/1273617553 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90031d60 con 0x7fdaa01547a0 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 -- 172.21.15.141:0/1273617553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa01547a0 msgr2=0x7fdaa015b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 --2- 172.21.15.141:0/1273617553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa01547a0 0x7fdaa015b290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdaa006c5a0 tx=0x7fda9002b910 comp rx=0 tx=0).stop 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 -- 172.21.15.141:0/1273617553 shutdown_connections 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 --2- 172.21.15.141:0/1273617553 >> v2:172.21.15.188:3300/0 conn(0x7fdaa00715a0 0x7fdaa006b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 --2- 172.21.15.141:0/1273617553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa01547a0 0x7fdaa015b290 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 -- 172.21.15.141:0/1273617553 >> 172.21.15.141:0/1273617553 conn(0x7fdaa0070a80 msgr2=0x7fdaa0070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:08.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 -- 172.21.15.141:0/1273617553 shutdown_connections 2024-01-16T14:57:08.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.556+0000 7fdaa7200640 1 -- 172.21.15.141:0/1273617553 wait complete. 2024-01-16T14:57:08.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.558+0000 7fdaa7200640 1 Processor -- start 2024-01-16T14:57:08.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.558+0000 7fdaa7200640 1 -- start start 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fdaa7200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa006b930 0x7fdaa00c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fdaa7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaa00715a0 0x7fdaa00cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fdaa7200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdaa01547a0 0x7fdaa014aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fdaa7200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa00cdaa0 con 0x7fdaa006b930 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fdaa7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa014bca0 con 0x7fdaa00715a0 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fdaa7200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa00c9ba0 con 0x7fdaa01547a0 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fda9fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa006b930 0x7fdaa00c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fda9fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa006b930 0x7fdaa00c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59166/0 (socket says 172.21.15.141:59166) 2024-01-16T14:57:08.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.559+0000 7fda9fe00640 1 -- 172.21.15.141:0/2622075047 learned_addr learned my addr 172.21.15.141:0/2622075047 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9fe00640 1 -- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdaa01547a0 msgr2=0x7fdaa014aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda97200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaa00715a0 0x7fdaa00cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9fe00640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdaa01547a0 0x7fdaa014aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9fe00640 1 -- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaa00715a0 msgr2=0x7fdaa00cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9fe00640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaa00715a0 0x7fdaa00cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9fe00640 1 -- 172.21.15.141:0/2622075047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda900094d0 con 0x7fdaa006b930 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9fe00640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa006b930 0x7fdaa00c9330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda90000c00 tx=0x7fda900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9cc00640 1 -- 172.21.15.141:0/2622075047 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90031860 con 0x7fdaa006b930 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9cc00640 1 -- 172.21.15.141:0/2622075047 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda900443f0 con 0x7fdaa006b930 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fda9cc00640 1 -- 172.21.15.141:0/2622075047 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda900447a0 con 0x7fdaa006b930 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa00cd450 con 0x7fdaa006b930 2024-01-16T14:57:08.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.560+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa00c9870 con 0x7fdaa006b930 2024-01-16T14:57:08.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.561+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa00675d0 con 0x7fdaa006b930 2024-01-16T14:57:08.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.562+0000 7fda9cc00640 1 -- 172.21.15.141:0/2622075047 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fda90044900 con 0x7fdaa006b930 2024-01-16T14:57:08.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.563+0000 7fda9cc00640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7806b2f0 0x7fda7806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:08.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.564+0000 7fda9cc00640 1 -- 172.21.15.141:0/2622075047 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda9002d040 con 0x7fdaa006b930 2024-01-16T14:57:08.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.564+0000 7fda97200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7806b2f0 0x7fda7806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:08.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.565+0000 7fda97200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7806b2f0 0x7fda7806d7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fda88006fd0 tx=0x7fda88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:08.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.568+0000 7fda9cc00640 1 -- 172.21.15.141:0/2622075047 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda90077320 con 0x7fdaa006b930 2024-01-16T14:57:08.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:08.796+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdaa00665f0 con 0x7fdaa006b930 2024-01-16T14:57:09.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.544+0000 7fda9cc00640 1 -- 172.21.15.141:0/2622075047 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fda9007a940 con 0x7fdaa006b930 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7806b2f0 msgr2=0x7fda7806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7806b2f0 0x7fda7806d7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fda88006fd0 tx=0x7fda88008040 comp rx=0 tx=0).stop 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa006b930 msgr2=0x7fdaa00c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa006b930 0x7fdaa00c9330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda90000c00 tx=0x7fda900046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 shutdown_connections 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7806b2f0 0x7fda7806d7b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdaa01547a0 0x7fdaa014aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaa00715a0 0x7fdaa00cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 --2- 172.21.15.141:0/2622075047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdaa006b930 0x7fdaa00c9330 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 >> 172.21.15.141:0/2622075047 conn(0x7fdaa0070a80 msgr2=0x7fdaa00b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 shutdown_connections 2024-01-16T14:57:09.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.548+0000 7fdaa7200640 1 -- 172.21.15.141:0/2622075047 wait complete. 2024-01-16T14:57:09.550 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T14:57:09.550 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T14:57:09.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.780+0000 7fface400640 1 Processor -- start 2024-01-16T14:57:09.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.780+0000 7fface400640 1 -- start start 2024-01-16T14:57:09.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7fface400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 0x7ffac815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7fface400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ffac813e360 0x7ffac80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7fface400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac806c470 con 0x7ffac815f9c0 2024-01-16T14:57:09.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7fface400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac806c750 con 0x7ffac813e360 2024-01-16T14:57:09.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7fface400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac806ca30 con 0x7ffac813df80 2024-01-16T14:57:09.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7ffac7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ffac815f9c0 0x7ffac81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59594/0 (socket says 172.21.15.141:59594) 2024-01-16T14:57:09.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7ffac7e00640 1 -- 172.21.15.141:0/1401081637 learned_addr learned my addr 172.21.15.141:0/1401081637 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7ffac6a00640 1 --2- 172.21.15.141:0/1401081637 >> v2:172.21.15.188:3300/0 conn(0x7ffac813e360 0x7ffac80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.782+0000 7ffac7400640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 0x7ffac815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326648445 0 0) 0x7ffac806c470 con 0x7ffac815f9c0 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8003520 con 0x7ffac815f9c0 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097770277 0 0) 0x7ffaa8003520 con 0x7ffac815f9c0 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 msgr2=0x7ffac815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 0x7ffac815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 >> v2:172.21.15.188:3300/0 conn(0x7ffac813e360 msgr2=0x7ffac80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 --2- 172.21.15.141:0/1401081637 >> v2:172.21.15.188:3300/0 conn(0x7ffac813e360 0x7ffac80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac806cd10 con 0x7ffac815f9c0 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ffac00019f0 con 0x7ffac815f9c0 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 >> v1:172.21.15.141:6789/0 conn(0x7ffac815f9c0 legacy=0x7ffac81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 0x7ffaa8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffaa80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8004d00 0x7ffaa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac806c750 con 0x7ffaa8003ef0 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac806ca30 con 0x7ffaa8004d00 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa800b980 con 0x7ffac813df80 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.783+0000 7ffac6a00640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffaa80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac7400640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 0x7ffaa8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac7e00640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8004d00 0x7ffaa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac6a00640 1 -- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 msgr2=0x7ffaa8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac6a00640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac813df80 0x7ffaa8003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac6a00640 1 -- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8004d00 msgr2=0x7ffaa8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac6a00640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8004d00 0x7ffaa8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac6a00640 1 -- 172.21.15.141:0/1401081637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa8003520 con 0x7ffaa8003ef0 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.784+0000 7ffac7e00640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8004d00 0x7ffaa8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.785+0000 7ffac6a00640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffaa80047c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009ac0 tx=0x7ffaa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e451395d0fd8e837 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.785+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab002c6d0 con 0x7ffaa8003ef0 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.785+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab002c830 con 0x7ffaa8003ef0 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.785+0000 7ffac6000640 1 -- 172.21.15.141:0/1401081637 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab002cbc0 con 0x7ffaa8003ef0 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.786+0000 7fface400640 1 -- 172.21.15.141:0/1401081637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 msgr2=0x7ffaa80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.786+0000 7fface400640 1 --2- 172.21.15.141:0/1401081637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffaa80047c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009ac0 tx=0x7ffaa80057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.786+0000 7ffac7e00640 1 -- 172.21.15.141:0/1401081637 reap_dead start 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.786+0000 7fface400640 1 -- 172.21.15.141:0/1401081637 shutdown_connections 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.786+0000 7fface400640 1 -- 172.21.15.141:0/1401081637 >> 172.21.15.141:0/1401081637 conn(0x7ffac806a800 msgr2=0x7ffac806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.786+0000 7fface400640 1 -- 172.21.15.141:0/1401081637 shutdown_connections 2024-01-16T14:57:09.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.786+0000 7fface400640 1 -- 172.21.15.141:0/1401081637 wait complete. 2024-01-16T14:57:09.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.788+0000 7fface400640 1 Processor -- start 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.788+0000 7fface400640 1 -- start start 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7fface400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffac81f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7fface400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac814fdc0 0x7ffac813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7fface400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac80714d0 0x7ffac8071950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7fface400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac81f3c80 con 0x7ffaa8003ef0 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7fface400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8071e90 con 0x7ffac80714d0 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7fface400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8150ab0 con 0x7ffac814fdc0 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffac81f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffac81f3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59188/0 (socket says 172.21.15.141:59188) 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 -- 172.21.15.141:0/2605536877 learned_addr learned my addr 172.21.15.141:0/2605536877 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7e00640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac80714d0 0x7ffac8071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 -- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac814fdc0 msgr2=0x7ffac813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac814fdc0 0x7ffac813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 -- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac80714d0 msgr2=0x7ffac8071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac80714d0 0x7ffac8071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.789+0000 7ffac7400640 1 -- 172.21.15.141:0/2605536877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac813dda0 con 0x7ffaa8003ef0 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.790+0000 7ffac7400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffac81f3740 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffab8010860 tx=0x7ffab8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.790+0000 7ffabf400640 1 -- 172.21.15.141:0/2605536877 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8009610 con 0x7ffaa8003ef0 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.790+0000 7ffabf400640 1 -- 172.21.15.141:0/2605536877 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab800eb60 con 0x7ffaa8003ef0 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.790+0000 7ffabf400640 1 -- 172.21.15.141:0/2605536877 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab801c400 con 0x7ffaa8003ef0 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.790+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac81f6430 con 0x7ffaa8003ef0 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.790+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac81f69f0 con 0x7ffaa8003ef0 2024-01-16T14:57:09.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.791+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac80675d0 con 0x7ffaa8003ef0 2024-01-16T14:57:09.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.792+0000 7ffabf400640 1 -- 172.21.15.141:0/2605536877 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ffab801c560 con 0x7ffaa8003ef0 2024-01-16T14:57:09.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.793+0000 7ffabf400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaa006b2f0 0x7ffaa006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:09.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.793+0000 7ffabf400640 1 -- 172.21.15.141:0/2605536877 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffab808a010 con 0x7ffaa8003ef0 2024-01-16T14:57:09.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.793+0000 7ffac6a00640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaa006b2f0 0x7ffaa006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:09.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.794+0000 7ffac6a00640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaa006b2f0 0x7ffaa006d7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffab0036000 tx=0x7ffab002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:09.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:09.797+0000 7ffabf400640 1 -- 172.21.15.141:0/2605536877 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffab8054f30 con 0x7ffaa8003ef0 2024-01-16T14:57:09.934 INFO:tasks.ceph.mds.a.smithi141.stderr:2024-01-16T14:57:09.951+0000 7f7e9c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:57:09.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.016+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ffac806df00 con 0x7ffaa8003ef0 2024-01-16T14:57:10.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.554+0000 7ffabf400640 1 -- 172.21.15.141:0/2605536877 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7ffab8058550 con 0x7ffaa8003ef0 2024-01-16T14:57:10.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaa006b2f0 msgr2=0x7ffaa006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaa006b2f0 0x7ffaa006d7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffab0036000 tx=0x7ffab002e000 comp rx=0 tx=0).stop 2024-01-16T14:57:10.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 msgr2=0x7ffac81f3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffac81f3740 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffab8010860 tx=0x7ffab8010d20 comp rx=0 tx=0).stop 2024-01-16T14:57:10.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 shutdown_connections 2024-01-16T14:57:10.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaa006b2f0 0x7ffaa006d7b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac80714d0 0x7ffac8071950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac814fdc0 0x7ffac813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 --2- 172.21.15.141:0/2605536877 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa8003ef0 0x7ffac81f3740 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.558+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 >> 172.21.15.141:0/2605536877 conn(0x7ffac806a800 msgr2=0x7ffac80cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:10.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.559+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 shutdown_connections 2024-01-16T14:57:10.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.559+0000 7fface400640 1 -- 172.21.15.141:0/2605536877 wait complete. 2024-01-16T14:57:10.561 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T14:57:10.561 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:10.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.780+0000 7fbc5b600640 1 Processor -- start 2024-01-16T14:57:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.780+0000 7fbc5b600640 1 -- start start 2024-01-16T14:57:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc5b600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5413df80 0x7fbc5415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc5b600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fbc540cf9a0 0x7fbc540cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc5b600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5406c470 con 0x7fbc5413e360 2024-01-16T14:57:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc5b600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5406c750 con 0x7fbc540cf9a0 2024-01-16T14:57:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc5b600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5406ca30 con 0x7fbc5413df80 2024-01-16T14:57:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc53e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbc5413e360 0x7fbc540cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59608/0 (socket says 172.21.15.141:59608) 2024-01-16T14:57:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc53e00640 1 -- 172.21.15.141:0/1094956266 learned_addr learned my addr 172.21.15.141:0/1094956266 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:10.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421741380 0 0) 0x7fbc5406c470 con 0x7fbc5413e360 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc58c00640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5413df80 0x7fbc5415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.782+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc34003520 con 0x7fbc5413e360 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc59600640 1 --2- 172.21.15.141:0/1094956266 >> v2:172.21.15.188:3300/0 conn(0x7fbc540cf9a0 0x7fbc540cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3288004260 0 0) 0x7fbc34003520 con 0x7fbc5413e360 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5413df80 msgr2=0x7fbc5415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5413df80 0x7fbc5415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 >> v2:172.21.15.188:3300/0 conn(0x7fbc540cf9a0 msgr2=0x7fbc540cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 --2- 172.21.15.141:0/1094956266 >> v2:172.21.15.188:3300/0 conn(0x7fbc540cf9a0 0x7fbc540cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5406cd10 con 0x7fbc5413e360 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fbc3c001670 con 0x7fbc5413e360 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 >> v1:172.21.15.141:6789/0 conn(0x7fbc5413e360 legacy=0x7fbc540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5413df80 0x7fbc34003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc34003fa0 0x7fbc340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34004d00 0x7fbc34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5406c750 con 0x7fbc34003fa0 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5406ca30 con 0x7fbc34004d00 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3400ba80 con 0x7fbc5413df80 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.783+0000 7fbc53e00640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc34003fa0 0x7fbc340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.784+0000 7fbc53e00640 1 -- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5413df80 msgr2=0x7fbc34003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.784+0000 7fbc53e00640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5413df80 0x7fbc34003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.784+0000 7fbc53e00640 1 -- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34004d00 msgr2=0x7fbc34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.784+0000 7fbc53e00640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34004d00 0x7fbc34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.784+0000 7fbc53e00640 1 -- 172.21.15.141:0/1094956266 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc34003520 con 0x7fbc34003fa0 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc53e00640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc34003fa0 0x7fbc340047c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c240 tx=0x7fbc340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b48573a678c840c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c02cb00 con 0x7fbc34003fa0 2024-01-16T14:57:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c02cc60 con 0x7fbc34003fa0 2024-01-16T14:57:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc53400640 1 -- 172.21.15.141:0/1094956266 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c005af0 con 0x7fbc34003fa0 2024-01-16T14:57:10.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc5b600640 1 -- 172.21.15.141:0/1094956266 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc34003fa0 msgr2=0x7fbc340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc5b600640 1 --2- 172.21.15.141:0/1094956266 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc34003fa0 0x7fbc340047c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c240 tx=0x7fbc340057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc59600640 1 -- 172.21.15.141:0/1094956266 reap_dead start 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc5b600640 1 -- 172.21.15.141:0/1094956266 shutdown_connections 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.785+0000 7fbc5b600640 1 -- 172.21.15.141:0/1094956266 >> 172.21.15.141:0/1094956266 conn(0x7fbc5406a800 msgr2=0x7fbc5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.786+0000 7fbc5b600640 1 -- 172.21.15.141:0/1094956266 shutdown_connections 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.786+0000 7fbc5b600640 1 -- 172.21.15.141:0/1094956266 wait complete. 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.787+0000 7fbc5b600640 1 Processor -- start 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.787+0000 7fbc5b600640 1 -- start start 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.788+0000 7fbc5b600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34003fa0 0x7fbc54154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.788+0000 7fbc5b600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5414e570 0x7fbc5415a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.788+0000 7fbc5b600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc5413c3a0 0x7fbc5413c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.788+0000 7fbc5b600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5413cd90 con 0x7fbc5413c3a0 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.788+0000 7fbc5b600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc54155b40 con 0x7fbc34003fa0 2024-01-16T14:57:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.788+0000 7fbc5b600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5414ea00 con 0x7fbc5414e570 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.788+0000 7fbc59600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc5413c3a0 0x7fbc5413c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc5413c3a0 0x7fbc5413c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59214/0 (socket says 172.21.15.141:59214) 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 -- 172.21.15.141:0/3541608262 learned_addr learned my addr 172.21.15.141:0/3541608262 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc53e00640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5414e570 0x7fbc5415a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc58c00640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34003fa0 0x7fbc54154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 -- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5414e570 msgr2=0x7fbc5415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5414e570 0x7fbc5415a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 -- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34003fa0 msgr2=0x7fbc54154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34003fa0 0x7fbc54154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 -- 172.21.15.141:0/3541608262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc540b2f70 con 0x7fbc5413c3a0 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc58c00640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34003fa0 0x7fbc54154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc59600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc5413c3a0 0x7fbc5413c850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc540cfea0 tx=0x7fbc4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc50c00640 1 -- 172.21.15.141:0/3541608262 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c00ed90 con 0x7fbc5413c3a0 2024-01-16T14:57:10.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc50c00640 1 -- 172.21.15.141:0/3541608262 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4c014420 con 0x7fbc5413c3a0 2024-01-16T14:57:10.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc540b3250 con 0x7fbc5413c3a0 2024-01-16T14:57:10.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.789+0000 7fbc50c00640 1 -- 172.21.15.141:0/3541608262 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c0147d0 con 0x7fbc5413c3a0 2024-01-16T14:57:10.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.790+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc54155080 con 0x7fbc5413c3a0 2024-01-16T14:57:10.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.791+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc540b2da0 con 0x7fbc5413c3a0 2024-01-16T14:57:10.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.791+0000 7fbc50c00640 1 -- 172.21.15.141:0/3541608262 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbc4c014930 con 0x7fbc5413c3a0 2024-01-16T14:57:10.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.792+0000 7fbc50c00640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc2c06b2f0 0x7fbc2c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:10.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.792+0000 7fbc50c00640 1 -- 172.21.15.141:0/3541608262 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc4c089fd0 con 0x7fbc5413c3a0 2024-01-16T14:57:10.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.793+0000 7fbc58c00640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc2c06b2f0 0x7fbc2c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:10.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.794+0000 7fbc58c00640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc2c06b2f0 0x7fbc2c06d7b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc44004770 tx=0x7fbc44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:10.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:10.796+0000 7fbc50c00640 1 -- 172.21.15.141:0/3541608262 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc4c054da0 con 0x7fbc5413c3a0 2024-01-16T14:57:10.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.006+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc5406ae40 con 0x7fbc5413c3a0 2024-01-16T14:57:10.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.007+0000 7fbc50c00640 1 -- 172.21.15.141:0/3541608262 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fbc4c0583c0 con 0x7fbc5413c3a0 2024-01-16T14:57:10.990 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:10.990 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:10.990 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc2c06b2f0 msgr2=0x7fbc2c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc2c06b2f0 0x7fbc2c06d7b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc44004770 tx=0x7fbc44008040 comp rx=0 tx=0).stop 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc5413c3a0 msgr2=0x7fbc5413c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc5413c3a0 0x7fbc5413c850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc540cfea0 tx=0x7fbc4c00e4b0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 shutdown_connections 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc2c06b2f0 0x7fbc2c06d7b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc5413c3a0 0x7fbc5413c850 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc5414e570 0x7fbc5415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 --2- 172.21.15.141:0/3541608262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc34003fa0 0x7fbc54154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 >> 172.21.15.141:0/3541608262 conn(0x7fbc5406a800 msgr2=0x7fbc5415daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 shutdown_connections 2024-01-16T14:57:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.010+0000 7fbc5b600640 1 -- 172.21.15.141:0/3541608262 wait complete. 2024-01-16T14:57:11.010 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:11.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.234+0000 7f6b88000640 1 Processor -- start 2024-01-16T14:57:11.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.235+0000 7f6b88000640 1 -- start start 2024-01-16T14:57:11.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.236+0000 7f6b88000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6b8006ebc0 0x7f6b8006f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.236+0000 7f6b88000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b800af150 con 0x7f6b80152250 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.236+0000 7f6b88000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b800b2d80 con 0x7f6b8015c360 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.236+0000 7f6b88000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8006f590 con 0x7f6b8006ebc0 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.236+0000 7f6b85600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6b80152250 0x7f6b8006b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59610/0 (socket says 172.21.15.141:59610) 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.236+0000 7f6b85600640 1 -- 172.21.15.141:0/2458858623 learned_addr learned my addr 172.21.15.141:0/2458858623 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.236+0000 7f6b86000640 1 --2- 172.21.15.141:0/2458858623 >> v2:172.21.15.141:3300/0 conn(0x7f6b8006ebc0 0x7f6b8006f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.237+0000 7f6b7fe00640 1 -- 172.21.15.141:0/2458858623 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082795163 0 0) 0x7f6b800af150 con 0x7f6b80152250 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.237+0000 7f6b7fe00640 1 -- 172.21.15.141:0/2458858623 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b60003520 con 0x7f6b80152250 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.237+0000 7f6b86000640 1 -- 172.21.15.141:0/2458858623 >> v1:172.21.15.188:6789/0 conn(0x7f6b8015c360 legacy=0x7f6b800b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.237+0000 7f6b86000640 1 -- 172.21.15.141:0/2458858623 >> v1:172.21.15.141:6789/0 conn(0x7f6b80152250 legacy=0x7f6b8006b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.237+0000 7f6b86000640 1 -- 172.21.15.141:0/2458858623 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800ce050 con 0x7f6b8006ebc0 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.238+0000 7f6b86000640 1 --2- 172.21.15.141:0/2458858623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b8006f050 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b780097d0 tx=0x7f6b7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ea43e16e77196f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.238+0000 7f6b7fe00640 1 -- 172.21.15.141:0/2458858623 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78031830 con 0x7f6b8006ebc0 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.238+0000 7f6b7fe00640 1 -- 172.21.15.141:0/2458858623 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78031990 con 0x7f6b8006ebc0 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.238+0000 7f6b7fe00640 1 -- 172.21.15.141:0/2458858623 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78031d20 con 0x7f6b8006ebc0 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.239+0000 7f6b88000640 1 -- 172.21.15.141:0/2458858623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 msgr2=0x7f6b8006f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.239+0000 7f6b88000640 1 --2- 172.21.15.141:0/2458858623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b8006f050 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b780097d0 tx=0x7f6b7802b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.239+0000 7f6b88000640 1 -- 172.21.15.141:0/2458858623 shutdown_connections 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.239+0000 7f6b88000640 1 --2- 172.21.15.141:0/2458858623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b8006f050 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.239+0000 7f6b88000640 1 -- 172.21.15.141:0/2458858623 >> 172.21.15.141:0/2458858623 conn(0x7f6b80070b10 msgr2=0x7f6b80070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.239+0000 7f6b88000640 1 -- 172.21.15.141:0/2458858623 shutdown_connections 2024-01-16T14:57:11.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.239+0000 7f6b88000640 1 -- 172.21.15.141:0/2458858623 wait complete. 2024-01-16T14:57:11.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.241+0000 7f6b88000640 1 Processor -- start 2024-01-16T14:57:11.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.241+0000 7f6b88000640 1 -- start start 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b88000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b801550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b88000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b80152250 0x7f6b800af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b88000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b8015c360 0x7f6b80144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b88000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8006cd00 con 0x7f6b8006ebc0 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b88000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8006c9b0 con 0x7f6b80152250 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b88000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b801534b0 con 0x7f6b8015c360 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b85600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b801550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b85600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b801550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59238/0 (socket says 172.21.15.141:59238) 2024-01-16T14:57:11.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.242+0000 7f6b85600640 1 -- 172.21.15.141:0/3811168185 learned_addr learned my addr 172.21.15.141:0/3811168185 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:11.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b85600640 1 -- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b8015c360 msgr2=0x7f6b80144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b85600640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b8015c360 0x7f6b80144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b85600640 1 -- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b80152250 msgr2=0x7f6b800af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b85600640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b80152250 0x7f6b800af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b85600640 1 -- 172.21.15.141:0/3811168185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b78009400 con 0x7f6b8006ebc0 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b85600640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b801550f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000c530 tx=0x7f6b7000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b7d600640 1 -- 172.21.15.141:0/3811168185 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70005ce0 con 0x7f6b8006ebc0 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b7d600640 1 -- 172.21.15.141:0/3811168185 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b70005e40 con 0x7f6b8006ebc0 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b7d600640 1 -- 172.21.15.141:0/3811168185 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700072b0 con 0x7f6b8006ebc0 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b80153790 con 0x7f6b8006ebc0 2024-01-16T14:57:11.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.243+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8006a4c0 con 0x7f6b8006ebc0 2024-01-16T14:57:11.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.244+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8006ce40 con 0x7f6b8006ebc0 2024-01-16T14:57:11.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.245+0000 7f6b7d600640 1 -- 172.21.15.141:0/3811168185 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6b70007410 con 0x7f6b8006ebc0 2024-01-16T14:57:11.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.246+0000 7f6b7d600640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6b5806b2c0 0x7f6b5806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.246+0000 7f6b7d600640 1 -- 172.21.15.141:0/3811168185 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b70089100 con 0x7f6b8006ebc0 2024-01-16T14:57:11.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.247+0000 7f6b84c00640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6b5806b2c0 0x7f6b5806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.248+0000 7f6b84c00640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6b5806b2c0 0x7f6b5806d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6b680012e0 tx=0x7f6b68007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.250+0000 7f6b7d600640 1 -- 172.21.15.141:0/3811168185 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b8006ce40 con 0x7f6b8006ebc0 2024-01-16T14:57:11.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.442+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b8006ce40 con 0x7f6b8006ebc0 2024-01-16T14:57:11.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.444+0000 7f6b7d600640 1 -- 172.21.15.141:0/3811168185 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f6b8006ce40 con 0x7f6b8006ebc0 2024-01-16T14:57:11.427 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:11.427 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:11.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6b5806b2c0 msgr2=0x7f6b5806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6b5806b2c0 0x7f6b5806d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6b680012e0 tx=0x7f6b68007d30 comp rx=0 tx=0).stop 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 msgr2=0x7f6b801550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b801550f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000c530 tx=0x7f6b7000c9f0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 shutdown_connections 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6b5806b2c0 0x7f6b5806d780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b8015c360 0x7f6b80144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b80152250 0x7f6b800af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.446+0000 7f6b88000640 1 --2- 172.21.15.141:0/3811168185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b8006ebc0 0x7f6b801550f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.447+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 >> 172.21.15.141:0/3811168185 conn(0x7f6b80070b10 msgr2=0x7f6b80070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.447+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 shutdown_connections 2024-01-16T14:57:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.447+0000 7f6b88000640 1 -- 172.21.15.141:0/3811168185 wait complete. 2024-01-16T14:57:11.449 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T14:57:11.449 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:57:11.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.673+0000 7f1392a00640 1 Processor -- start 2024-01-16T14:57:11.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.673+0000 7f1392a00640 1 -- start start 2024-01-16T14:57:11.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.674+0000 7f1392a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f138c143ae0 0x7f138c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.674+0000 7f1392a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c0cf7d0 0x7f138c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f1392a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c0cfc30 con 0x7f138c15e1e0 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f1392a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f138c0b1ee0 con 0x7f138c143ae0 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f1392a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0b2780 con 0x7f138c0cf7d0 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f138c143ae0 0x7f138c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f1390a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f138c15e1e0 0x7f138c15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59622/0 (socket says 172.21.15.141:59622) 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f1390a00640 1 -- 172.21.15.141:0/768678167 learned_addr learned my addr 172.21.15.141:0/768678167 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:11.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f138c143ae0 0x7f138c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59240/0 (socket says 172.21.15.141:59240) 2024-01-16T14:57:11.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138be00640 1 -- 172.21.15.141:0/768678167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c0cf7d0 msgr2=0x7f138c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:11.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138b400640 1 --2- 172.21.15.141:0/768678167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c0cf7d0 0x7f138c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138be00640 1 --2- 172.21.15.141:0/768678167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c0cf7d0 0x7f138c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138be00640 1 -- 172.21.15.141:0/768678167 >> v1:172.21.15.141:6789/0 conn(0x7f138c15e1e0 legacy=0x7f138c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138aa00640 1 -- 172.21.15.141:0/768678167 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377412718 0 0) 0x7f138c0cfc30 con 0x7f138c15e1e0 2024-01-16T14:57:11.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.675+0000 7f138be00640 1 -- 172.21.15.141:0/768678167 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c15e990 con 0x7f138c143ae0 2024-01-16T14:57:11.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.676+0000 7f138b400640 1 --2- 172.21.15.141:0/768678167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c0cf7d0 0x7f138c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:11.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.677+0000 7f138be00640 1 --2- 172.21.15.141:0/768678167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c143ae0 0x7f138c0cf290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f137c0097a0 tx=0x7f137c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12daf8e180d2bfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:11.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.677+0000 7f138aa00640 1 -- 172.21.15.141:0/768678167 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c031830 con 0x7f138c143ae0 2024-01-16T14:57:11.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.677+0000 7f138aa00640 1 -- 172.21.15.141:0/768678167 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c031990 con 0x7f138c143ae0 2024-01-16T14:57:11.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.677+0000 7f138aa00640 1 -- 172.21.15.141:0/768678167 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c031d20 con 0x7f138c143ae0 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.677+0000 7f1392a00640 1 -- 172.21.15.141:0/768678167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c143ae0 msgr2=0x7f138c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.677+0000 7f1392a00640 1 --2- 172.21.15.141:0/768678167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c143ae0 0x7f138c0cf290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f137c0097a0 tx=0x7f137c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.678+0000 7f1392a00640 1 -- 172.21.15.141:0/768678167 shutdown_connections 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.678+0000 7f1392a00640 1 --2- 172.21.15.141:0/768678167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c0cf7d0 0x7f138c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.678+0000 7f1392a00640 1 --2- 172.21.15.141:0/768678167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c143ae0 0x7f138c0cf290 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.678+0000 7f1392a00640 1 -- 172.21.15.141:0/768678167 >> 172.21.15.141:0/768678167 conn(0x7f138c06a800 msgr2=0x7f138c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.678+0000 7f1392a00640 1 -- 172.21.15.141:0/768678167 shutdown_connections 2024-01-16T14:57:11.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.678+0000 7f1392a00640 1 -- 172.21.15.141:0/768678167 wait complete. 2024-01-16T14:57:11.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.679+0000 7f1392a00640 1 Processor -- start 2024-01-16T14:57:11.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.680+0000 7f1392a00640 1 -- start start 2024-01-16T14:57:11.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.680+0000 7f1392a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f138c0cf7d0 0x7f138c138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1392a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c143ae0 0x7f138c13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1392a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c15e1e0 0x7f138c0cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1392a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15c190 con 0x7f138c15e1e0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1392a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c138950 con 0x7f138c0cf7d0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1392a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c137db0 con 0x7f138c143ae0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c15e1e0 0x7f138c0cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c15e1e0 0x7f138c0cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59254/0 (socket says 172.21.15.141:59254) 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 -- 172.21.15.141:0/991811142 learned_addr learned my addr 172.21.15.141:0/991811142 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 -- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c143ae0 msgr2=0x7f138c13c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f138b400640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c143ae0 0x7f138c13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f138be00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f138c0cf7d0 0x7f138c138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c143ae0 0x7f138c13c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 -- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f138c0cf7d0 msgr2=0x7f138c138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f138c0cf7d0 0x7f138c138de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.681+0000 7f1390a00640 1 -- 172.21.15.141:0/991811142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c009400 con 0x7f138c15e1e0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.682+0000 7f1390a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c15e1e0 0x7f138c0cc880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f138400c7b0 tx=0x7f138400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.682+0000 7f1383e00640 1 -- 172.21.15.141:0/991811142 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138400a640 con 0x7f138c15e1e0 2024-01-16T14:57:11.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.682+0000 7f1383e00640 1 -- 172.21.15.141:0/991811142 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138400a7c0 con 0x7f138c15e1e0 2024-01-16T14:57:11.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.682+0000 7f1383e00640 1 -- 172.21.15.141:0/991811142 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138400ab70 con 0x7f138c15e1e0 2024-01-16T14:57:11.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.682+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c13cce0 con 0x7f138c15e1e0 2024-01-16T14:57:11.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.682+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c138300 con 0x7f138c15e1e0 2024-01-16T14:57:11.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.683+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c0b2bf0 con 0x7f138c15e1e0 2024-01-16T14:57:11.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.684+0000 7f1383e00640 1 -- 172.21.15.141:0/991811142 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f138400acd0 con 0x7f138c15e1e0 2024-01-16T14:57:11.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.685+0000 7f1383e00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f136406b2f0 0x7f136406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:11.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.685+0000 7f1383e00640 1 -- 172.21.15.141:0/991811142 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1384089090 con 0x7f138c15e1e0 2024-01-16T14:57:11.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.685+0000 7f138be00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f136406b2f0 0x7f136406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:11.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.687+0000 7f138be00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f136406b2f0 0x7f136406d7b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f137c036000 tx=0x7f137c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:11.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.689+0000 7f1383e00640 1 -- 172.21.15.141:0/991811142 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f138c0b2bf0 con 0x7f138c15e1e0 2024-01-16T14:57:11.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.876+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f138c15e640 con 0x7f136406b2f0 2024-01-16T14:57:11.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.900+0000 7f1383e00640 1 -- 172.21.15.141:0/991811142 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264470 (secure 0 0 0) 0x7f137c045400 con 0x7f136406b2f0 2024-01-16T14:57:11.888 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:11.889 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.906+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f136406b2f0 msgr2=0x7f136406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.906+0000 7f1392a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f136406b2f0 0x7f136406d7b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f137c036000 tx=0x7f137c008030 comp rx=0 tx=0).stop 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.906+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c15e1e0 msgr2=0x7f138c0cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.906+0000 7f1392a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c15e1e0 0x7f138c0cc880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f138400c7b0 tx=0x7f138400cc70 comp rx=0 tx=0).stop 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 shutdown_connections 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f136406b2f0 0x7f136406d7b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f138c15e1e0 0x7f138c0cc880 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f138c143ae0 0x7f138c13c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 --2- 172.21.15.141:0/991811142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f138c0cf7d0 0x7f138c138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:11.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 >> 172.21.15.141:0/991811142 conn(0x7f138c06a800 msgr2=0x7f138c15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:11.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 shutdown_connections 2024-01-16T14:57:11.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:11.907+0000 7f1392a00640 1 -- 172.21.15.141:0/991811142 wait complete. 2024-01-16T14:57:11.910 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-16T14:57:10.436015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219492,"kb_used_data":2972,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755228,"statfs":{"total":773094113280,"available":772869353472,"internally_reserved":0,"allocated":3043328,"data_stored":1809080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.923739"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361647+0000","last_change":"2024-01-16T14:57:07.388330+0000","last_active":"2024-01-16T14:57:08.361647+0000","last_peered":"2024-01-16T14:57:08.361647+0000","last_clean":"2024-01-16T14:57:08.361647+0000","last_became_active":"2024-01-16T14:57:07.387937+0000","last_became_peered":"2024-01-16T14:57:07.387937+0000","last_unstale":"2024-01-16T14:57:08.361647+0000","last_undegraded":"2024-01-16T14:57:08.361647+0000","last_fullsized":"2024-01-16T14:57:08.361647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859055+0000","last_change":"2024-01-16T14:57:07.376852+0000","last_active":"2024-01-16T14:57:08.859055+0000","last_peered":"2024-01-16T14:57:08.859055+0000","last_clean":"2024-01-16T14:57:08.859055+0000","last_became_active":"2024-01-16T14:57:07.376562+0000","last_became_peered":"2024-01-16T14:57:07.376562+0000","last_unstale":"2024-01-16T14:57:08.859055+0000","last_undegraded":"2024-01-16T14:57:08.859055+0000","last_fullsized":"2024-01-16T14:57:08.859055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411451+0000","last_change":"2024-01-16T14:57:07.376399+0000","last_active":"2024-01-16T14:57:08.411451+0000","last_peered":"2024-01-16T14:57:08.411451+0000","last_clean":"2024-01-16T14:57:08.411451+0000","last_became_active":"2024-01-16T14:57:07.374502+0000","last_became_peered":"2024-01-16T14:57:07.374502+0000","last_unstale":"2024-01-16T14:57:08.411451+0000","last_undegraded":"2024-01-16T14:57:08.411451+0000","last_fullsized":"2024-01-16T14:57:08.411451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357034+0000","last_change":"2024-01-16T14:57:07.393925+0000","last_active":"2024-01-16T14:57:08.357034+0000","last_peered":"2024-01-16T14:57:08.357034+0000","last_clean":"2024-01-16T14:57:08.357034+0000","last_became_active":"2024-01-16T14:57:07.393608+0000","last_became_peered":"2024-01-16T14:57:07.393608+0000","last_unstale":"2024-01-16T14:57:08.357034+0000","last_undegraded":"2024-01-16T14:57:08.357034+0000","last_fullsized":"2024-01-16T14:57:08.357034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:12.497634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360231+0000","last_change":"2024-01-16T14:57:07.370383+0000","last_active":"2024-01-16T14:57:08.360231+0000","last_peered":"2024-01-16T14:57:08.360231+0000","last_clean":"2024-01-16T14:57:08.360231+0000","last_became_active":"2024-01-16T14:57:07.369925+0000","last_became_peered":"2024-01-16T14:57:07.369925+0000","last_unstale":"2024-01-16T14:57:08.360231+0000","last_undegraded":"2024-01-16T14:57:08.360231+0000","last_fullsized":"2024-01-16T14:57:08.360231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357036+0000","last_change":"2024-01-16T14:57:07.392767+0000","last_active":"2024-01-16T14:57:08.357036+0000","last_peered":"2024-01-16T14:57:08.357036+0000","last_clean":"2024-01-16T14:57:08.357036+0000","last_became_active":"2024-01-16T14:57:07.392457+0000","last_became_peered":"2024-01-16T14:57:07.392457+0000","last_unstale":"2024-01-16T14:57:08.357036+0000","last_undegraded":"2024-01-16T14:57:08.357036+0000","last_fullsized":"2024-01-16T14:57:08.357036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359822+0000","last_change":"2024-01-16T14:57:07.377276+0000","last_active":"2024-01-16T14:57:08.359822+0000","last_peered":"2024-01-16T14:57:08.359822+0000","last_clean":"2024-01-16T14:57:08.359822+0000","last_became_active":"2024-01-16T14:57:07.376937+0000","last_became_peered":"2024-01-16T14:57:07.376937+0000","last_unstale":"2024-01-16T14:57:08.359822+0000","last_undegraded":"2024-01-16T14:57:08.359822+0000","last_fullsized":"2024-01-16T14:57:08.359822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857828+0000","last_change":"2024-01-16T14:57:07.394001+0000","last_active":"2024-01-16T14:57:08.857828+0000","last_peered":"2024-01-16T14:57:08.857828+0000","last_clean":"2024-01-16T14:57:08.857828+0000","last_became_active":"2024-01-16T14:57:07.393670+0000","last_became_peered":"2024-01-16T14:57:07.393670+0000","last_unstale":"2024-01-16T14:57:08.857828+0000","last_undegraded":"2024-01-16T14:57:08.857828+0000","last_fullsized":"2024-01-16T14:57:08.857828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361786+0000","last_change":"2024-01-16T14:57:07.394940+0000","last_active":"2024-01-16T14:57:08.361786+0000","last_peered":"2024-01-16T14:57:08.361786+0000","last_clean":"2024-01-16T14:57:08.361786+0000","last_became_active":"2024-01-16T14:57:07.394335+0000","last_became_peered":"2024-01-16T14:57:07.394335+0000","last_unstale":"2024-01-16T14:57:08.361786+0000","last_undegraded":"2024-01-16T14:57:08.361786+0000","last_fullsized":"2024-01-16T14:57:08.361786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:23.443732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857711+0000","last_change":"2024-01-16T14:57:07.395598+0000","last_active":"2024-01-16T14:57:08.857711+0000","last_peered":"2024-01-16T14:57:08.857711+0000","last_clean":"2024-01-16T14:57:08.857711+0000","last_became_active":"2024-01-16T14:57:07.395367+0000","last_became_peered":"2024-01-16T14:57:07.395367+0000","last_unstale":"2024-01-16T14:57:08.857711+0000","last_undegraded":"2024-01-16T14:57:08.857711+0000","last_fullsized":"2024-01-16T14:57:08.857711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857014+0000","last_change":"2024-01-16T14:57:07.394374+0000","last_active":"2024-01-16T14:57:08.857014+0000","last_peered":"2024-01-16T14:57:08.857014+0000","last_clean":"2024-01-16T14:57:08.857014+0000","last_became_active":"2024-01-16T14:57:07.394125+0000","last_became_peered":"2024-01-16T14:57:07.394125+0000","last_unstale":"2024-01-16T14:57:08.857014+0000","last_undegraded":"2024-01-16T14:57:08.857014+0000","last_fullsized":"2024-01-16T14:57:08.857014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357258+0000","last_change":"2024-01-16T14:57:07.382452+0000","last_active":"2024-01-16T14:57:08.357258+0000","last_peered":"2024-01-16T14:57:08.357258+0000","last_clean":"2024-01-16T14:57:08.357258+0000","last_became_active":"2024-01-16T14:57:07.381985+0000","last_became_peered":"2024-01-16T14:57:07.381985+0000","last_unstale":"2024-01-16T14:57:08.357258+0000","last_undegraded":"2024-01-16T14:57:08.357258+0000","last_fullsized":"2024-01-16T14:57:08.357258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:31.589732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858575+0000","last_change":"2024-01-16T14:57:07.385013+0000","last_active":"2024-01-16T14:57:08.858575+0000","last_peered":"2024-01-16T14:57:08.858575+0000","last_clean":"2024-01-16T14:57:08.858575+0000","last_became_active":"2024-01-16T14:57:07.384705+0000","last_became_peered":"2024-01-16T14:57:07.384705+0000","last_unstale":"2024-01-16T14:57:08.858575+0000","last_undegraded":"2024-01-16T14:57:08.858575+0000","last_fullsized":"2024-01-16T14:57:08.858575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411099+0000","last_change":"2024-01-16T14:57:07.382942+0000","last_active":"2024-01-16T14:57:08.411099+0000","last_peered":"2024-01-16T14:57:08.411099+0000","last_clean":"2024-01-16T14:57:08.411099+0000","last_became_active":"2024-01-16T14:57:07.382609+0000","last_became_peered":"2024-01-16T14:57:07.382609+0000","last_unstale":"2024-01-16T14:57:08.411099+0000","last_undegraded":"2024-01-16T14:57:08.411099+0000","last_fullsized":"2024-01-16T14:57:08.411099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:07.098165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856153+0000","last_change":"2024-01-16T14:57:07.369499+0000","last_active":"2024-01-16T14:57:08.856153+0000","last_peered":"2024-01-16T14:57:08.856153+0000","last_clean":"2024-01-16T14:57:08.856153+0000","last_became_active":"2024-01-16T14:57:07.369089+0000","last_became_peered":"2024-01-16T14:57:07.369089+0000","last_unstale":"2024-01-16T14:57:08.856153+0000","last_undegraded":"2024-01-16T14:57:08.856153+0000","last_fullsized":"2024-01-16T14:57:08.856153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:45:27.672659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.355734+0000","last_change":"2024-01-16T14:57:07.392502+0000","last_active":"2024-01-16T14:57:08.355734+0000","last_peered":"2024-01-16T14:57:08.355734+0000","last_clean":"2024-01-16T14:57:08.355734+0000","last_became_active":"2024-01-16T14:57:07.392065+0000","last_became_peered":"2024-01-16T14:57:07.392065+0000","last_unstale":"2024-01-16T14:57:08.355734+0000","last_undegraded":"2024-01-16T14:57:08.355734+0000","last_fullsized":"2024-01-16T14:57:08.355734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358909+0000","last_change":"2024-01-16T14:57:07.394712+0000","last_active":"2024-01-16T14:57:08.358909+0000","last_peered":"2024-01-16T14:57:08.358909+0000","last_clean":"2024-01-16T14:57:08.358909+0000","last_became_active":"2024-01-16T14:57:07.393954+0000","last_became_peered":"2024-01-16T14:57:07.393954+0000","last_unstale":"2024-01-16T14:57:08.358909+0000","last_undegraded":"2024-01-16T14:57:08.358909+0000","last_fullsized":"2024-01-16T14:57:08.358909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859744+0000","last_change":"2024-01-16T14:57:07.384626+0000","last_active":"2024-01-16T14:57:08.859744+0000","last_peered":"2024-01-16T14:57:08.859744+0000","last_clean":"2024-01-16T14:57:08.859744+0000","last_became_active":"2024-01-16T14:57:07.383860+0000","last_became_peered":"2024-01-16T14:57:07.383860+0000","last_unstale":"2024-01-16T14:57:08.859744+0000","last_undegraded":"2024-01-16T14:57:08.859744+0000","last_fullsized":"2024-01-16T14:57:08.859744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361107+0000","last_change":"2024-01-16T14:57:07.375013+0000","last_active":"2024-01-16T14:57:08.361107+0000","last_peered":"2024-01-16T14:57:08.361107+0000","last_clean":"2024-01-16T14:57:08.361107+0000","last_became_active":"2024-01-16T14:57:07.373630+0000","last_became_peered":"2024-01-16T14:57:07.373630+0000","last_unstale":"2024-01-16T14:57:08.361107+0000","last_undegraded":"2024-01-16T14:57:08.361107+0000","last_fullsized":"2024-01-16T14:57:08.361107+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857112+0000","last_change":"2024-01-16T14:57:07.375790+0000","last_active":"2024-01-16T14:57:08.857112+0000","last_peered":"2024-01-16T14:57:08.857112+0000","last_clean":"2024-01-16T14:57:08.857112+0000","last_became_active":"2024-01-16T14:57:07.375533+0000","last_became_peered":"2024-01-16T14:57:07.375533+0000","last_unstale":"2024-01-16T14:57:08.857112+0000","last_undegraded":"2024-01-16T14:57:08.857112+0000","last_fullsized":"2024-01-16T14:57:08.857112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411312+0000","last_change":"2024-01-16T14:57:07.378765+0000","last_active":"2024-01-16T14:57:08.411312+0000","last_peered":"2024-01-16T14:57:08.411312+0000","last_clean":"2024-01-16T14:57:08.411312+0000","last_became_active":"2024-01-16T14:57:07.377672+0000","last_became_peered":"2024-01-16T14:57:07.377672+0000","last_unstale":"2024-01-16T14:57:08.411312+0000","last_undegraded":"2024-01-16T14:57:08.411312+0000","last_fullsized":"2024-01-16T14:57:08.411312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357439+0000","last_change":"2024-01-16T14:57:07.392203+0000","last_active":"2024-01-16T14:57:08.357439+0000","last_peered":"2024-01-16T14:57:08.357439+0000","last_clean":"2024-01-16T14:57:08.357439+0000","last_became_active":"2024-01-16T14:57:07.391913+0000","last_became_peered":"2024-01-16T14:57:07.391913+0000","last_unstale":"2024-01-16T14:57:08.357439+0000","last_undegraded":"2024-01-16T14:57:08.357439+0000","last_fullsized":"2024-01-16T14:57:08.357439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858112+0000","last_change":"2024-01-16T14:57:07.383055+0000","last_active":"2024-01-16T14:57:08.858112+0000","last_peered":"2024-01-16T14:57:08.858112+0000","last_clean":"2024-01-16T14:57:08.858112+0000","last_became_active":"2024-01-16T14:57:07.382181+0000","last_became_peered":"2024-01-16T14:57:07.382181+0000","last_unstale":"2024-01-16T14:57:08.858112+0000","last_undegraded":"2024-01-16T14:57:08.858112+0000","last_fullsized":"2024-01-16T14:57:08.858112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.356639+0000","last_change":"2024-01-16T14:57:07.391594+0000","last_active":"2024-01-16T14:57:08.356639+0000","last_peered":"2024-01-16T14:57:08.356639+0000","last_clean":"2024-01-16T14:57:08.356639+0000","last_became_active":"2024-01-16T14:57:07.391116+0000","last_became_peered":"2024-01-16T14:57:07.391116+0000","last_unstale":"2024-01-16T14:57:08.356639+0000","last_undegraded":"2024-01-16T14:57:08.356639+0000","last_fullsized":"2024-01-16T14:57:08.356639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859255+0000","last_change":"2024-01-16T14:57:07.383060+0000","last_active":"2024-01-16T14:57:08.859255+0000","last_peered":"2024-01-16T14:57:08.859255+0000","last_clean":"2024-01-16T14:57:08.859255+0000","last_became_active":"2024-01-16T14:57:07.382192+0000","last_became_peered":"2024-01-16T14:57:07.382192+0000","last_unstale":"2024-01-16T14:57:08.859255+0000","last_undegraded":"2024-01-16T14:57:08.859255+0000","last_fullsized":"2024-01-16T14:57:08.859255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:34.791403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857952+0000","last_change":"2024-01-16T14:57:07.384727+0000","last_active":"2024-01-16T14:57:08.857952+0000","last_peered":"2024-01-16T14:57:08.857952+0000","last_clean":"2024-01-16T14:57:08.857952+0000","last_became_active":"2024-01-16T14:57:07.383334+0000","last_became_peered":"2024-01-16T14:57:07.383334+0000","last_unstale":"2024-01-16T14:57:08.857952+0000","last_undegraded":"2024-01-16T14:57:08.857952+0000","last_fullsized":"2024-01-16T14:57:08.857952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856861+0000","last_change":"2024-01-16T14:57:07.376359+0000","last_active":"2024-01-16T14:57:08.856861+0000","last_peered":"2024-01-16T14:57:08.856861+0000","last_clean":"2024-01-16T14:57:08.856861+0000","last_became_active":"2024-01-16T14:57:07.376074+0000","last_became_peered":"2024-01-16T14:57:07.376074+0000","last_unstale":"2024-01-16T14:57:08.856861+0000","last_undegraded":"2024-01-16T14:57:08.856861+0000","last_fullsized":"2024-01-16T14:57:08.856861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858616+0000","last_change":"2024-01-16T14:57:07.387458+0000","last_active":"2024-01-16T14:57:08.858616+0000","last_peered":"2024-01-16T14:57:08.858616+0000","last_clean":"2024-01-16T14:57:08.858616+0000","last_became_active":"2024-01-16T14:57:07.387221+0000","last_became_peered":"2024-01-16T14:57:07.387221+0000","last_unstale":"2024-01-16T14:57:08.858616+0000","last_undegraded":"2024-01-16T14:57:08.858616+0000","last_fullsized":"2024-01-16T14:57:08.858616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411330+0000","last_change":"2024-01-16T14:57:07.377884+0000","last_active":"2024-01-16T14:57:08.411330+0000","last_peered":"2024-01-16T14:57:08.411330+0000","last_clean":"2024-01-16T14:57:08.411330+0000","last_became_active":"2024-01-16T14:57:07.377585+0000","last_became_peered":"2024-01-16T14:57:07.377585+0000","last_unstale":"2024-01-16T14:57:08.411330+0000","last_undegraded":"2024-01-16T14:57:08.411330+0000","last_fullsized":"2024-01-16T14:57:08.411330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358051+0000","last_change":"2024-01-16T14:57:07.389650+0000","last_active":"2024-01-16T14:57:08.358051+0000","last_peered":"2024-01-16T14:57:08.358051+0000","last_clean":"2024-01-16T14:57:08.358051+0000","last_became_active":"2024-01-16T14:57:07.389388+0000","last_became_peered":"2024-01-16T14:57:07.389388+0000","last_unstale":"2024-01-16T14:57:08.358051+0000","last_undegraded":"2024-01-16T14:57:08.358051+0000","last_fullsized":"2024-01-16T14:57:08.358051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858338+0000","last_change":"2024-01-16T14:57:07.393121+0000","last_active":"2024-01-16T14:57:08.858338+0000","last_peered":"2024-01-16T14:57:08.858338+0000","last_clean":"2024-01-16T14:57:08.858338+0000","last_became_active":"2024-01-16T14:57:07.392839+0000","last_became_peered":"2024-01-16T14:57:07.392839+0000","last_unstale":"2024-01-16T14:57:08.858338+0000","last_undegraded":"2024-01-16T14:57:08.858338+0000","last_fullsized":"2024-01-16T14:57:08.858338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856756+0000","last_change":"2024-01-16T14:57:07.386156+0000","last_active":"2024-01-16T14:57:08.856756+0000","last_peered":"2024-01-16T14:57:08.856756+0000","last_clean":"2024-01-16T14:57:08.856756+0000","last_became_active":"2024-01-16T14:57:07.385684+0000","last_became_peered":"2024-01-16T14:57:07.385684+0000","last_unstale":"2024-01-16T14:57:08.856756+0000","last_undegraded":"2024-01-16T14:57:08.856756+0000","last_fullsized":"2024-01-16T14:57:08.856756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357739+0000","last_change":"2024-01-16T14:57:07.372196+0000","last_active":"2024-01-16T14:57:08.357739+0000","last_peered":"2024-01-16T14:57:08.357739+0000","last_clean":"2024-01-16T14:57:08.357739+0000","last_became_active":"2024-01-16T14:57:07.369065+0000","last_became_peered":"2024-01-16T14:57:07.369065+0000","last_unstale":"2024-01-16T14:57:08.357739+0000","last_undegraded":"2024-01-16T14:57:08.357739+0000","last_fullsized":"2024-01-16T14:57:08.357739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858827+0000","last_change":"2024-01-16T14:57:06.556368+0000","last_active":"2024-01-16T14:57:08.858827+0000","last_peered":"2024-01-16T14:57:08.858827+0000","last_clean":"2024-01-16T14:57:08.858827+0000","last_became_active":"2024-01-16T14:57:06.555953+0000","last_became_peered":"2024-01-16T14:57:06.555953+0000","last_unstale":"2024-01-16T14:57:08.858827+0000","last_undegraded":"2024-01-16T14:57:08.858827+0000","last_fullsized":"2024-01-16T14:57:08.858827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:45.773231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411448+0000","last_change":"2024-01-16T14:57:07.383151+0000","last_active":"2024-01-16T14:57:08.411448+0000","last_peered":"2024-01-16T14:57:08.411448+0000","last_clean":"2024-01-16T14:57:08.411448+0000","last_became_active":"2024-01-16T14:57:07.381555+0000","last_became_peered":"2024-01-16T14:57:07.381555+0000","last_unstale":"2024-01-16T14:57:08.411448+0000","last_undegraded":"2024-01-16T14:57:08.411448+0000","last_fullsized":"2024-01-16T14:57:08.411448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856949+0000","last_change":"2024-01-16T14:57:06.555306+0000","last_active":"2024-01-16T14:57:08.856949+0000","last_peered":"2024-01-16T14:57:08.856949+0000","last_clean":"2024-01-16T14:57:08.856949+0000","last_became_active":"2024-01-16T14:57:06.555063+0000","last_became_peered":"2024-01-16T14:57:06.555063+0000","last_unstale":"2024-01-16T14:57:08.856949+0000","last_undegraded":"2024-01-16T14:57:08.856949+0000","last_fullsized":"2024-01-16T14:57:08.856949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:07.271545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857979+0000","last_change":"2024-01-16T14:57:06.555279+0000","last_active":"2024-01-16T14:57:08.857979+0000","last_peered":"2024-01-16T14:57:08.857979+0000","last_clean":"2024-01-16T14:57:08.857979+0000","last_became_active":"2024-01-16T14:57:06.554559+0000","last_became_peered":"2024-01-16T14:57:06.554559+0000","last_unstale":"2024-01-16T14:57:08.857979+0000","last_undegraded":"2024-01-16T14:57:08.857979+0000","last_fullsized":"2024-01-16T14:57:08.857979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857830+0000","last_change":"2024-01-16T14:57:07.382661+0000","last_active":"2024-01-16T14:57:08.857830+0000","last_peered":"2024-01-16T14:57:08.857830+0000","last_clean":"2024-01-16T14:57:08.857830+0000","last_became_active":"2024-01-16T14:57:07.382055+0000","last_became_peered":"2024-01-16T14:57:07.382055+0000","last_unstale":"2024-01-16T14:57:08.857830+0000","last_undegraded":"2024-01-16T14:57:08.857830+0000","last_fullsized":"2024-01-16T14:57:08.857830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357563+0000","last_change":"2024-01-16T14:57:06.558562+0000","last_active":"2024-01-16T14:57:08.357563+0000","last_peered":"2024-01-16T14:57:08.357563+0000","last_clean":"2024-01-16T14:57:08.357563+0000","last_became_active":"2024-01-16T14:57:06.558166+0000","last_became_peered":"2024-01-16T14:57:06.558166+0000","last_unstale":"2024-01-16T14:57:08.357563+0000","last_undegraded":"2024-01-16T14:57:08.357563+0000","last_fullsized":"2024-01-16T14:57:08.357563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:31.817106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855928+0000","last_change":"2024-01-16T14:57:07.377917+0000","last_active":"2024-01-16T14:57:08.855928+0000","last_peered":"2024-01-16T14:57:08.855928+0000","last_clean":"2024-01-16T14:57:08.855928+0000","last_became_active":"2024-01-16T14:57:07.377601+0000","last_became_peered":"2024-01-16T14:57:07.377601+0000","last_unstale":"2024-01-16T14:57:08.855928+0000","last_undegraded":"2024-01-16T14:57:08.855928+0000","last_fullsized":"2024-01-16T14:57:08.855928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:32.968418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857555+0000","last_change":"2024-01-16T14:57:06.563826+0000","last_active":"2024-01-16T14:57:08.857555+0000","last_peered":"2024-01-16T14:57:08.857555+0000","last_clean":"2024-01-16T14:57:08.857555+0000","last_became_active":"2024-01-16T14:57:06.563529+0000","last_became_peered":"2024-01-16T14:57:06.563529+0000","last_unstale":"2024-01-16T14:57:08.857555+0000","last_undegraded":"2024-01-16T14:57:08.857555+0000","last_fullsized":"2024-01-16T14:57:08.857555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358434+0000","last_change":"2024-01-16T14:57:07.391665+0000","last_active":"2024-01-16T14:57:08.358434+0000","last_peered":"2024-01-16T14:57:08.358434+0000","last_clean":"2024-01-16T14:57:08.358434+0000","last_became_active":"2024-01-16T14:57:07.391071+0000","last_became_peered":"2024-01-16T14:57:07.391071+0000","last_unstale":"2024-01-16T14:57:08.358434+0000","last_undegraded":"2024-01-16T14:57:08.358434+0000","last_fullsized":"2024-01-16T14:57:08.358434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358657+0000","last_change":"2024-01-16T14:57:06.560447+0000","last_active":"2024-01-16T14:57:08.358657+0000","last_peered":"2024-01-16T14:57:08.358657+0000","last_clean":"2024-01-16T14:57:08.358657+0000","last_became_active":"2024-01-16T14:57:06.560116+0000","last_became_peered":"2024-01-16T14:57:06.560116+0000","last_unstale":"2024-01-16T14:57:08.358657+0000","last_undegraded":"2024-01-16T14:57:08.358657+0000","last_fullsized":"2024-01-16T14:57:08.358657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:33.296388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411729+0000","last_change":"2024-01-16T14:57:07.396020+0000","last_active":"2024-01-16T14:57:08.411729+0000","last_peered":"2024-01-16T14:57:08.411729+0000","last_clean":"2024-01-16T14:57:08.411729+0000","last_became_active":"2024-01-16T14:57:07.395685+0000","last_became_peered":"2024-01-16T14:57:07.395685+0000","last_unstale":"2024-01-16T14:57:08.411729+0000","last_undegraded":"2024-01-16T14:57:08.411729+0000","last_fullsized":"2024-01-16T14:57:08.411729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857123+0000","last_change":"2024-01-16T14:57:06.553623+0000","last_active":"2024-01-16T14:57:08.857123+0000","last_peered":"2024-01-16T14:57:08.857123+0000","last_clean":"2024-01-16T14:57:08.857123+0000","last_became_active":"2024-01-16T14:57:06.553173+0000","last_became_peered":"2024-01-16T14:57:06.553173+0000","last_unstale":"2024-01-16T14:57:08.857123+0000","last_undegraded":"2024-01-16T14:57:08.857123+0000","last_fullsized":"2024-01-16T14:57:08.857123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357536+0000","last_change":"2024-01-16T14:57:07.382256+0000","last_active":"2024-01-16T14:57:08.357536+0000","last_peered":"2024-01-16T14:57:08.357536+0000","last_clean":"2024-01-16T14:57:08.357536+0000","last_became_active":"2024-01-16T14:57:07.381640+0000","last_became_peered":"2024-01-16T14:57:07.381640+0000","last_unstale":"2024-01-16T14:57:08.357536+0000","last_undegraded":"2024-01-16T14:57:08.357536+0000","last_fullsized":"2024-01-16T14:57:08.357536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:14.206644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857022+0000","last_change":"2024-01-16T14:57:06.556835+0000","last_active":"2024-01-16T14:57:08.857022+0000","last_peered":"2024-01-16T14:57:08.857022+0000","last_clean":"2024-01-16T14:57:08.857022+0000","last_became_active":"2024-01-16T14:57:06.556483+0000","last_became_peered":"2024-01-16T14:57:06.556483+0000","last_unstale":"2024-01-16T14:57:08.857022+0000","last_undegraded":"2024-01-16T14:57:08.857022+0000","last_fullsized":"2024-01-16T14:57:08.857022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:01.863989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360686+0000","last_change":"2024-01-16T14:57:07.380047+0000","last_active":"2024-01-16T14:57:08.360686+0000","last_peered":"2024-01-16T14:57:08.360686+0000","last_clean":"2024-01-16T14:57:08.360686+0000","last_became_active":"2024-01-16T14:57:07.379698+0000","last_became_peered":"2024-01-16T14:57:07.379698+0000","last_unstale":"2024-01-16T14:57:08.360686+0000","last_undegraded":"2024-01-16T14:57:08.360686+0000","last_fullsized":"2024-01-16T14:57:08.360686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411453+0000","last_change":"2024-01-16T14:57:06.555448+0000","last_active":"2024-01-16T14:57:08.411453+0000","last_peered":"2024-01-16T14:57:08.411453+0000","last_clean":"2024-01-16T14:57:08.411453+0000","last_became_active":"2024-01-16T14:57:06.555097+0000","last_became_peered":"2024-01-16T14:57:06.555097+0000","last_unstale":"2024-01-16T14:57:08.411453+0000","last_undegraded":"2024-01-16T14:57:08.411453+0000","last_fullsized":"2024-01-16T14:57:08.411453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:23.858452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411172+0000","last_change":"2024-01-16T14:57:07.380436+0000","last_active":"2024-01-16T14:57:08.411172+0000","last_peered":"2024-01-16T14:57:08.411172+0000","last_clean":"2024-01-16T14:57:08.411172+0000","last_became_active":"2024-01-16T14:57:07.380148+0000","last_became_peered":"2024-01-16T14:57:07.380148+0000","last_unstale":"2024-01-16T14:57:08.411172+0000","last_undegraded":"2024-01-16T14:57:08.411172+0000","last_fullsized":"2024-01-16T14:57:08.411172+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358031+0000","last_change":"2024-01-16T14:57:06.549803+0000","last_active":"2024-01-16T14:57:08.358031+0000","last_peered":"2024-01-16T14:57:08.358031+0000","last_clean":"2024-01-16T14:57:08.358031+0000","last_became_active":"2024-01-16T14:57:06.549362+0000","last_became_peered":"2024-01-16T14:57:06.549362+0000","last_unstale":"2024-01-16T14:57:08.358031+0000","last_undegraded":"2024-01-16T14:57:08.358031+0000","last_fullsized":"2024-01-16T14:57:08.358031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360491+0000","last_change":"2024-01-16T14:57:07.384510+0000","last_active":"2024-01-16T14:57:08.360491+0000","last_peered":"2024-01-16T14:57:08.360491+0000","last_clean":"2024-01-16T14:57:08.360491+0000","last_became_active":"2024-01-16T14:57:07.384214+0000","last_became_peered":"2024-01-16T14:57:07.384214+0000","last_unstale":"2024-01-16T14:57:08.360491+0000","last_undegraded":"2024-01-16T14:57:08.360491+0000","last_fullsized":"2024-01-16T14:57:08.360491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856634+0000","last_change":"2024-01-16T14:57:06.562353+0000","last_active":"2024-01-16T14:57:08.856634+0000","last_peered":"2024-01-16T14:57:08.856634+0000","last_clean":"2024-01-16T14:57:08.856634+0000","last_became_active":"2024-01-16T14:57:06.562043+0000","last_became_peered":"2024-01-16T14:57:06.562043+0000","last_unstale":"2024-01-16T14:57:08.856634+0000","last_undegraded":"2024-01-16T14:57:08.856634+0000","last_fullsized":"2024-01-16T14:57:08.856634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:25.879858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856809+0000","last_change":"2024-01-16T14:57:07.383191+0000","last_active":"2024-01-16T14:57:08.856809+0000","last_peered":"2024-01-16T14:57:08.856809+0000","last_clean":"2024-01-16T14:57:08.856809+0000","last_became_active":"2024-01-16T14:57:07.379683+0000","last_became_peered":"2024-01-16T14:57:07.379683+0000","last_unstale":"2024-01-16T14:57:08.856809+0000","last_undegraded":"2024-01-16T14:57:08.856809+0000","last_fullsized":"2024-01-16T14:57:08.856809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856935+0000","last_change":"2024-01-16T14:57:06.548684+0000","last_active":"2024-01-16T14:57:08.856935+0000","last_peered":"2024-01-16T14:57:08.856935+0000","last_clean":"2024-01-16T14:57:08.856935+0000","last_became_active":"2024-01-16T14:57:06.547835+0000","last_became_peered":"2024-01-16T14:57:06.547835+0000","last_unstale":"2024-01-16T14:57:08.856935+0000","last_undegraded":"2024-01-16T14:57:08.856935+0000","last_fullsized":"2024-01-16T14:57:08.856935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:46.715195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360277+0000","last_change":"2024-01-16T14:57:07.384606+0000","last_active":"2024-01-16T14:57:08.360277+0000","last_peered":"2024-01-16T14:57:08.360277+0000","last_clean":"2024-01-16T14:57:08.360277+0000","last_became_active":"2024-01-16T14:57:07.384300+0000","last_became_peered":"2024-01-16T14:57:07.384300+0000","last_unstale":"2024-01-16T14:57:08.360277+0000","last_undegraded":"2024-01-16T14:57:08.360277+0000","last_fullsized":"2024-01-16T14:57:08.360277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359317+0000","last_change":"2024-01-16T14:57:06.555188+0000","last_active":"2024-01-16T14:57:08.359317+0000","last_peered":"2024-01-16T14:57:08.359317+0000","last_clean":"2024-01-16T14:57:08.359317+0000","last_became_active":"2024-01-16T14:57:06.554904+0000","last_became_peered":"2024-01-16T14:57:06.554904+0000","last_unstale":"2024-01-16T14:57:08.359317+0000","last_undegraded":"2024-01-16T14:57:08.359317+0000","last_fullsized":"2024-01-16T14:57:08.359317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:59.674406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360712+0000","last_change":"2024-01-16T14:57:07.386645+0000","last_active":"2024-01-16T14:57:08.360712+0000","last_peered":"2024-01-16T14:57:08.360712+0000","last_clean":"2024-01-16T14:57:08.360712+0000","last_became_active":"2024-01-16T14:57:07.384338+0000","last_became_peered":"2024-01-16T14:57:07.384338+0000","last_unstale":"2024-01-16T14:57:08.360712+0000","last_undegraded":"2024-01-16T14:57:08.360712+0000","last_fullsized":"2024-01-16T14:57:08.360712+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859980+0000","last_change":"2024-01-16T14:57:06.549964+0000","last_active":"2024-01-16T14:57:08.859980+0000","last_peered":"2024-01-16T14:57:08.859980+0000","last_clean":"2024-01-16T14:57:08.859980+0000","last_became_active":"2024-01-16T14:57:06.549594+0000","last_became_peered":"2024-01-16T14:57:06.549594+0000","last_unstale":"2024-01-16T14:57:08.859980+0000","last_undegraded":"2024-01-16T14:57:08.859980+0000","last_fullsized":"2024-01-16T14:57:08.859980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361966+0000","last_change":"2024-01-16T14:57:07.385296+0000","last_active":"2024-01-16T14:57:08.361966+0000","last_peered":"2024-01-16T14:57:08.361966+0000","last_clean":"2024-01-16T14:57:08.361966+0000","last_became_active":"2024-01-16T14:57:07.385019+0000","last_became_peered":"2024-01-16T14:57:07.385019+0000","last_unstale":"2024-01-16T14:57:08.361966+0000","last_undegraded":"2024-01-16T14:57:08.361966+0000","last_fullsized":"2024-01-16T14:57:08.361966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:08.100477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857357+0000","last_change":"2024-01-16T14:57:06.558586+0000","last_active":"2024-01-16T14:57:08.857357+0000","last_peered":"2024-01-16T14:57:08.857357+0000","last_clean":"2024-01-16T14:57:08.857357+0000","last_became_active":"2024-01-16T14:57:06.558143+0000","last_became_peered":"2024-01-16T14:57:06.558143+0000","last_unstale":"2024-01-16T14:57:08.857357+0000","last_undegraded":"2024-01-16T14:57:08.857357+0000","last_fullsized":"2024-01-16T14:57:08.857357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357386+0000","last_change":"2024-01-16T14:57:07.393946+0000","last_active":"2024-01-16T14:57:08.357386+0000","last_peered":"2024-01-16T14:57:08.357386+0000","last_clean":"2024-01-16T14:57:08.357386+0000","last_became_active":"2024-01-16T14:57:07.393108+0000","last_became_peered":"2024-01-16T14:57:07.393108+0000","last_unstale":"2024-01-16T14:57:08.357386+0000","last_undegraded":"2024-01-16T14:57:08.357386+0000","last_fullsized":"2024-01-16T14:57:08.357386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858814+0000","last_change":"2024-01-16T14:57:06.555365+0000","last_active":"2024-01-16T14:57:08.858814+0000","last_peered":"2024-01-16T14:57:08.858814+0000","last_clean":"2024-01-16T14:57:08.858814+0000","last_became_active":"2024-01-16T14:57:06.551851+0000","last_became_peered":"2024-01-16T14:57:06.551851+0000","last_unstale":"2024-01-16T14:57:08.858814+0000","last_undegraded":"2024-01-16T14:57:08.858814+0000","last_fullsized":"2024-01-16T14:57:08.858814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360922+0000","last_change":"2024-01-16T14:57:07.397246+0000","last_active":"2024-01-16T14:57:08.360922+0000","last_peered":"2024-01-16T14:57:08.360922+0000","last_clean":"2024-01-16T14:57:08.360922+0000","last_became_active":"2024-01-16T14:57:07.394277+0000","last_became_peered":"2024-01-16T14:57:07.394277+0000","last_unstale":"2024-01-16T14:57:08.360922+0000","last_undegraded":"2024-01-16T14:57:08.360922+0000","last_fullsized":"2024-01-16T14:57:08.360922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:57:07.357749+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:57:07.357749+0000","last_peered":"2024-01-16T14:57:07.357749+0000","last_clean":"2024-01-16T14:57:07.357749+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:57:07.357749+0000","last_undegraded":"2024-01-16T14:57:07.357749+0000","last_fullsized":"2024-01-16T14:57:07.357749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357443+0000","last_change":"2024-01-16T14:57:06.547364+0000","last_active":"2024-01-16T14:57:08.357443+0000","last_peered":"2024-01-16T14:57:08.357443+0000","last_clean":"2024-01-16T14:57:08.357443+0000","last_became_active":"2024-01-16T14:57:06.546800+0000","last_became_peered":"2024-01-16T14:57:06.546800+0000","last_unstale":"2024-01-16T14:57:08.357443+0000","last_undegraded":"2024-01-16T14:57:08.357443+0000","last_fullsized":"2024-01-16T14:57:08.357443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:27.163769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858385+0000","last_change":"2024-01-16T14:57:07.395614+0000","last_active":"2024-01-16T14:57:08.858385+0000","last_peered":"2024-01-16T14:57:08.858385+0000","last_clean":"2024-01-16T14:57:08.858385+0000","last_became_active":"2024-01-16T14:57:07.395330+0000","last_became_peered":"2024-01-16T14:57:07.395330+0000","last_unstale":"2024-01-16T14:57:08.858385+0000","last_undegraded":"2024-01-16T14:57:08.858385+0000","last_fullsized":"2024-01-16T14:57:08.858385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359578+0000","last_change":"2024-01-16T14:57:06.548309+0000","last_active":"2024-01-16T14:57:08.359578+0000","last_peered":"2024-01-16T14:57:08.359578+0000","last_clean":"2024-01-16T14:57:08.359578+0000","last_became_active":"2024-01-16T14:57:06.547952+0000","last_became_peered":"2024-01-16T14:57:06.547952+0000","last_unstale":"2024-01-16T14:57:08.359578+0000","last_undegraded":"2024-01-16T14:57:08.359578+0000","last_fullsized":"2024-01-16T14:57:08.359578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.410994+0000","last_change":"2024-01-16T14:57:07.375198+0000","last_active":"2024-01-16T14:57:08.410994+0000","last_peered":"2024-01-16T14:57:08.410994+0000","last_clean":"2024-01-16T14:57:08.410994+0000","last_became_active":"2024-01-16T14:57:07.374043+0000","last_became_peered":"2024-01-16T14:57:07.374043+0000","last_unstale":"2024-01-16T14:57:08.410994+0000","last_undegraded":"2024-01-16T14:57:08.410994+0000","last_fullsized":"2024-01-16T14:57:08.410994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:11.783582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857406+0000","last_change":"2024-01-16T14:57:06.553851+0000","last_active":"2024-01-16T14:57:08.857406+0000","last_peered":"2024-01-16T14:57:08.857406+0000","last_clean":"2024-01-16T14:57:08.857406+0000","last_became_active":"2024-01-16T14:57:06.552712+0000","last_became_peered":"2024-01-16T14:57:06.552712+0000","last_unstale":"2024-01-16T14:57:08.857406+0000","last_undegraded":"2024-01-16T14:57:08.857406+0000","last_fullsized":"2024-01-16T14:57:08.857406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855802+0000","last_change":"2024-01-16T14:57:07.372041+0000","last_active":"2024-01-16T14:57:08.855802+0000","last_peered":"2024-01-16T14:57:08.855802+0000","last_clean":"2024-01-16T14:57:08.855802+0000","last_became_active":"2024-01-16T14:57:07.371750+0000","last_became_peered":"2024-01-16T14:57:07.371750+0000","last_unstale":"2024-01-16T14:57:08.855802+0000","last_undegraded":"2024-01-16T14:57:08.855802+0000","last_fullsized":"2024-01-16T14:57:08.855802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:53.804355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358274+0000","last_change":"2024-01-16T14:57:06.548768+0000","last_active":"2024-01-16T14:57:08.358274+0000","last_peered":"2024-01-16T14:57:08.358274+0000","last_clean":"2024-01-16T14:57:08.358274+0000","last_became_active":"2024-01-16T14:57:06.548446+0000","last_became_peered":"2024-01-16T14:57:06.548446+0000","last_unstale":"2024-01-16T14:57:08.358274+0000","last_undegraded":"2024-01-16T14:57:08.358274+0000","last_fullsized":"2024-01-16T14:57:08.358274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:06.341461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358850+0000","last_change":"2024-01-16T14:57:07.378261+0000","last_active":"2024-01-16T14:57:08.358850+0000","last_peered":"2024-01-16T14:57:08.358850+0000","last_clean":"2024-01-16T14:57:08.358850+0000","last_became_active":"2024-01-16T14:57:07.377940+0000","last_became_peered":"2024-01-16T14:57:07.377940+0000","last_unstale":"2024-01-16T14:57:08.358850+0000","last_undegraded":"2024-01-16T14:57:08.358850+0000","last_fullsized":"2024-01-16T14:57:08.358850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856741+0000","last_change":"2024-01-16T14:57:06.545628+0000","last_active":"2024-01-16T14:57:08.856741+0000","last_peered":"2024-01-16T14:57:08.856741+0000","last_clean":"2024-01-16T14:57:08.856741+0000","last_became_active":"2024-01-16T14:57:06.545341+0000","last_became_peered":"2024-01-16T14:57:06.545341+0000","last_unstale":"2024-01-16T14:57:08.856741+0000","last_undegraded":"2024-01-16T14:57:08.856741+0000","last_fullsized":"2024-01-16T14:57:08.856741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:54.829524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358936+0000","last_change":"2024-01-16T14:57:07.392468+0000","last_active":"2024-01-16T14:57:08.358936+0000","last_peered":"2024-01-16T14:57:08.358936+0000","last_clean":"2024-01-16T14:57:08.358936+0000","last_became_active":"2024-01-16T14:57:07.392091+0000","last_became_peered":"2024-01-16T14:57:07.392091+0000","last_unstale":"2024-01-16T14:57:08.358936+0000","last_undegraded":"2024-01-16T14:57:08.358936+0000","last_fullsized":"2024-01-16T14:57:08.358936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361159+0000","last_change":"2024-01-16T14:57:06.551276+0000","last_active":"2024-01-16T14:57:08.361159+0000","last_peered":"2024-01-16T14:57:08.361159+0000","last_clean":"2024-01-16T14:57:08.361159+0000","last_became_active":"2024-01-16T14:57:06.550951+0000","last_became_peered":"2024-01-16T14:57:06.550951+0000","last_unstale":"2024-01-16T14:57:08.361159+0000","last_undegraded":"2024-01-16T14:57:08.361159+0000","last_fullsized":"2024-01-16T14:57:08.361159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:15.986936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856799+0000","last_change":"2024-01-16T14:57:07.370960+0000","last_active":"2024-01-16T14:57:08.856799+0000","last_peered":"2024-01-16T14:57:08.856799+0000","last_clean":"2024-01-16T14:57:08.856799+0000","last_became_active":"2024-01-16T14:57:07.370635+0000","last_became_peered":"2024-01-16T14:57:07.370635+0000","last_unstale":"2024-01-16T14:57:08.856799+0000","last_undegraded":"2024-01-16T14:57:08.856799+0000","last_fullsized":"2024-01-16T14:57:08.856799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:36.350912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859279+0000","last_change":"2024-01-16T14:57:06.553886+0000","last_active":"2024-01-16T14:57:08.859279+0000","last_peered":"2024-01-16T14:57:08.859279+0000","last_clean":"2024-01-16T14:57:08.859279+0000","last_became_active":"2024-01-16T14:57:06.553590+0000","last_became_peered":"2024-01-16T14:57:06.553590+0000","last_unstale":"2024-01-16T14:57:08.859279+0000","last_undegraded":"2024-01-16T14:57:08.859279+0000","last_fullsized":"2024-01-16T14:57:08.859279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:16.749765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859350+0000","last_change":"2024-01-16T14:57:07.388410+0000","last_active":"2024-01-16T14:57:08.859350+0000","last_peered":"2024-01-16T14:57:08.859350+0000","last_clean":"2024-01-16T14:57:08.859350+0000","last_became_active":"2024-01-16T14:57:07.388132+0000","last_became_peered":"2024-01-16T14:57:07.388132+0000","last_unstale":"2024-01-16T14:57:08.859350+0000","last_undegraded":"2024-01-16T14:57:08.859350+0000","last_fullsized":"2024-01-16T14:57:08.859350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:24.013168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857725+0000","last_change":"2024-01-16T14:57:06.560743+0000","last_active":"2024-01-16T14:57:08.857725+0000","last_peered":"2024-01-16T14:57:08.857725+0000","last_clean":"2024-01-16T14:57:08.857725+0000","last_became_active":"2024-01-16T14:57:06.560557+0000","last_became_peered":"2024-01-16T14:57:06.560557+0000","last_unstale":"2024-01-16T14:57:08.857725+0000","last_undegraded":"2024-01-16T14:57:08.857725+0000","last_fullsized":"2024-01-16T14:57:08.857725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411687+0000","last_change":"2024-01-16T14:57:07.378876+0000","last_active":"2024-01-16T14:57:08.411687+0000","last_peered":"2024-01-16T14:57:08.411687+0000","last_clean":"2024-01-16T14:57:08.411687+0000","last_became_active":"2024-01-16T14:57:07.378103+0000","last_became_peered":"2024-01-16T14:57:07.378103+0000","last_unstale":"2024-01-16T14:57:08.411687+0000","last_undegraded":"2024-01-16T14:57:08.411687+0000","last_fullsized":"2024-01-16T14:57:08.411687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359610+0000","last_change":"2024-01-16T14:57:06.548228+0000","last_active":"2024-01-16T14:57:08.359610+0000","last_peered":"2024-01-16T14:57:08.359610+0000","last_clean":"2024-01-16T14:57:08.359610+0000","last_became_active":"2024-01-16T14:57:06.547577+0000","last_became_peered":"2024-01-16T14:57:06.547577+0000","last_unstale":"2024-01-16T14:57:08.359610+0000","last_undegraded":"2024-01-16T14:57:08.359610+0000","last_fullsized":"2024-01-16T14:57:08.359610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357044+0000","last_change":"2024-01-16T14:57:07.377694+0000","last_active":"2024-01-16T14:57:08.357044+0000","last_peered":"2024-01-16T14:57:08.357044+0000","last_clean":"2024-01-16T14:57:08.357044+0000","last_became_active":"2024-01-16T14:57:07.377016+0000","last_became_peered":"2024-01-16T14:57:07.377016+0000","last_unstale":"2024-01-16T14:57:08.357044+0000","last_undegraded":"2024-01-16T14:57:08.357044+0000","last_fullsized":"2024-01-16T14:57:08.357044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:08:30.500155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856442+0000","last_change":"2024-01-16T14:57:06.545409+0000","last_active":"2024-01-16T14:57:08.856442+0000","last_peered":"2024-01-16T14:57:08.856442+0000","last_clean":"2024-01-16T14:57:08.856442+0000","last_became_active":"2024-01-16T14:57:06.545133+0000","last_became_peered":"2024-01-16T14:57:06.545133+0000","last_unstale":"2024-01-16T14:57:08.856442+0000","last_undegraded":"2024-01-16T14:57:08.856442+0000","last_fullsized":"2024-01-16T14:57:08.856442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:18.053787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858230+0000","last_change":"2024-01-16T14:57:07.373879+0000","last_active":"2024-01-16T14:57:08.858230+0000","last_peered":"2024-01-16T14:57:08.858230+0000","last_clean":"2024-01-16T14:57:08.858230+0000","last_became_active":"2024-01-16T14:57:07.373537+0000","last_became_peered":"2024-01-16T14:57:07.373537+0000","last_unstale":"2024-01-16T14:57:08.858230+0000","last_undegraded":"2024-01-16T14:57:08.858230+0000","last_fullsized":"2024-01-16T14:57:08.858230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856963+0000","last_change":"2024-01-16T14:57:06.548121+0000","last_active":"2024-01-16T14:57:08.856963+0000","last_peered":"2024-01-16T14:57:08.856963+0000","last_clean":"2024-01-16T14:57:08.856963+0000","last_became_active":"2024-01-16T14:57:06.547868+0000","last_became_peered":"2024-01-16T14:57:06.547868+0000","last_unstale":"2024-01-16T14:57:08.856963+0000","last_undegraded":"2024-01-16T14:57:08.856963+0000","last_fullsized":"2024-01-16T14:57:08.856963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858908+0000","last_change":"2024-01-16T14:57:07.378203+0000","last_active":"2024-01-16T14:57:08.858908+0000","last_peered":"2024-01-16T14:57:08.858908+0000","last_clean":"2024-01-16T14:57:08.858908+0000","last_became_active":"2024-01-16T14:57:07.377843+0000","last_became_peered":"2024-01-16T14:57:07.377843+0000","last_unstale":"2024-01-16T14:57:08.858908+0000","last_undegraded":"2024-01-16T14:57:08.858908+0000","last_fullsized":"2024-01-16T14:57:08.858908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360527+0000","last_change":"2024-01-16T14:57:06.559586+0000","last_active":"2024-01-16T14:57:08.360527+0000","last_peered":"2024-01-16T14:57:08.360527+0000","last_clean":"2024-01-16T14:57:08.360527+0000","last_became_active":"2024-01-16T14:57:06.558281+0000","last_became_peered":"2024-01-16T14:57:06.558281+0000","last_unstale":"2024-01-16T14:57:08.360527+0000","last_undegraded":"2024-01-16T14:57:08.360527+0000","last_fullsized":"2024-01-16T14:57:08.360527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856775+0000","last_change":"2024-01-16T14:57:07.375900+0000","last_active":"2024-01-16T14:57:08.856775+0000","last_peered":"2024-01-16T14:57:08.856775+0000","last_clean":"2024-01-16T14:57:08.856775+0000","last_became_active":"2024-01-16T14:57:07.375625+0000","last_became_peered":"2024-01-16T14:57:07.375625+0000","last_unstale":"2024-01-16T14:57:08.856775+0000","last_undegraded":"2024-01-16T14:57:08.856775+0000","last_fullsized":"2024-01-16T14:57:08.856775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358102+0000","last_change":"2024-01-16T14:57:07.391481+0000","last_active":"2024-01-16T14:57:08.358102+0000","last_peered":"2024-01-16T14:57:08.358102+0000","last_clean":"2024-01-16T14:57:08.358102+0000","last_became_active":"2024-01-16T14:57:07.391111+0000","last_became_peered":"2024-01-16T14:57:07.391111+0000","last_unstale":"2024-01-16T14:57:08.358102+0000","last_undegraded":"2024-01-16T14:57:08.358102+0000","last_fullsized":"2024-01-16T14:57:08.358102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360198+0000","last_change":"2024-01-16T14:57:06.553585+0000","last_active":"2024-01-16T14:57:08.360198+0000","last_peered":"2024-01-16T14:57:08.360198+0000","last_clean":"2024-01-16T14:57:08.360198+0000","last_became_active":"2024-01-16T14:57:06.553259+0000","last_became_peered":"2024-01-16T14:57:06.553259+0000","last_unstale":"2024-01-16T14:57:08.360198+0000","last_undegraded":"2024-01-16T14:57:08.360198+0000","last_fullsized":"2024-01-16T14:57:08.360198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357259+0000","last_change":"2024-01-16T14:57:07.394393+0000","last_active":"2024-01-16T14:57:08.357259+0000","last_peered":"2024-01-16T14:57:08.357259+0000","last_clean":"2024-01-16T14:57:08.357259+0000","last_became_active":"2024-01-16T14:57:07.394135+0000","last_became_peered":"2024-01-16T14:57:07.394135+0000","last_unstale":"2024-01-16T14:57:08.357259+0000","last_undegraded":"2024-01-16T14:57:08.357259+0000","last_fullsized":"2024-01-16T14:57:08.357259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360937+0000","last_change":"2024-01-16T14:57:06.554017+0000","last_active":"2024-01-16T14:57:08.360937+0000","last_peered":"2024-01-16T14:57:08.360937+0000","last_clean":"2024-01-16T14:57:08.360937+0000","last_became_active":"2024-01-16T14:57:06.553376+0000","last_became_peered":"2024-01-16T14:57:06.553376+0000","last_unstale":"2024-01-16T14:57:08.360937+0000","last_undegraded":"2024-01-16T14:57:08.360937+0000","last_fullsized":"2024-01-16T14:57:08.360937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:35.199600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411544+0000","last_change":"2024-01-16T14:57:07.395607+0000","last_active":"2024-01-16T14:57:08.411544+0000","last_peered":"2024-01-16T14:57:08.411544+0000","last_clean":"2024-01-16T14:57:08.411544+0000","last_became_active":"2024-01-16T14:57:07.395288+0000","last_became_peered":"2024-01-16T14:57:07.395288+0000","last_unstale":"2024-01-16T14:57:08.411544+0000","last_undegraded":"2024-01-16T14:57:08.411544+0000","last_fullsized":"2024-01-16T14:57:08.411544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859322+0000","last_change":"2024-01-16T14:57:06.549558+0000","last_active":"2024-01-16T14:57:08.859322+0000","last_peered":"2024-01-16T14:57:08.859322+0000","last_clean":"2024-01-16T14:57:08.859322+0000","last_became_active":"2024-01-16T14:57:06.545986+0000","last_became_peered":"2024-01-16T14:57:06.545986+0000","last_unstale":"2024-01-16T14:57:08.859322+0000","last_undegraded":"2024-01-16T14:57:08.859322+0000","last_fullsized":"2024-01-16T14:57:08.859322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859106+0000","last_change":"2024-01-16T14:57:07.384928+0000","last_active":"2024-01-16T14:57:08.859106+0000","last_peered":"2024-01-16T14:57:08.859106+0000","last_clean":"2024-01-16T14:57:08.859106+0000","last_became_active":"2024-01-16T14:57:07.384613+0000","last_became_peered":"2024-01-16T14:57:07.384613+0000","last_unstale":"2024-01-16T14:57:08.859106+0000","last_undegraded":"2024-01-16T14:57:08.859106+0000","last_fullsized":"2024-01-16T14:57:08.859106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:19.171826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411722+0000","last_change":"2024-01-16T14:57:06.552465+0000","last_active":"2024-01-16T14:57:08.411722+0000","last_peered":"2024-01-16T14:57:08.411722+0000","last_clean":"2024-01-16T14:57:08.411722+0000","last_became_active":"2024-01-16T14:57:06.552103+0000","last_became_peered":"2024-01-16T14:57:06.552103+0000","last_unstale":"2024-01-16T14:57:08.411722+0000","last_undegraded":"2024-01-16T14:57:08.411722+0000","last_fullsized":"2024-01-16T14:57:08.411722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:35.728588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]}]},{"osd":6,"up_from":14,"seq":60129542147,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]}]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1539999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.919}]}]},{"osd":3,"up_from":14,"seq":60129542147,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.149}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2000000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.401}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:57:11.915 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:57:12.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.179+0000 7f0005000640 1 Processor -- start 2024-01-16T14:57:12.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.179+0000 7f0005000640 1 -- start start 2024-01-16T14:57:12.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7f0005000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f000015e3b0 0x7f0000164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7f0005000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00000b1ee0 con 0x7f00000cf9a0 2024-01-16T14:57:12.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7f0005000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00000b2780 con 0x7f0000143ae0 2024-01-16T14:57:12.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7f0005000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000015e990 con 0x7f000015e3b0 2024-01-16T14:57:12.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efff7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f00000cf9a0 0x7f000015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59628/0 (socket says 172.21.15.141:59628) 2024-01-16T14:57:12.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efff7e00640 1 -- 172.21.15.141:0/1976550217 learned_addr learned my addr 172.21.15.141:0/1976550217 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:12.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efffe800640 1 --2- 172.21.15.141:0/1976550217 >> v2:172.21.15.141:3300/0 conn(0x7f000015e3b0 0x7f0000164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efffd400640 1 -- 172.21.15.141:0/1976550217 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724114393 0 0) 0x7f00000b1ee0 con 0x7f00000cf9a0 2024-01-16T14:57:12.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efffd400640 1 -- 172.21.15.141:0/1976550217 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe0003520 con 0x7f00000cf9a0 2024-01-16T14:57:12.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efffe800640 1 -- 172.21.15.141:0/1976550217 >> v1:172.21.15.188:6789/0 conn(0x7f0000143ae0 legacy=0x7f00000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efffe800640 1 -- 172.21.15.141:0/1976550217 >> v1:172.21.15.141:6789/0 conn(0x7f00000cf9a0 legacy=0x7f000015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.181+0000 7efffe800640 1 -- 172.21.15.141:0/1976550217 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00000b2130 con 0x7f000015e3b0 2024-01-16T14:57:12.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7efffe800640 1 --2- 172.21.15.141:0/1976550217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000164de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efff8009d90 tx=0x7efff802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570d498476776db1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:12.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7efffd400640 1 -- 172.21.15.141:0/1976550217 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff802ccf0 con 0x7f000015e3b0 2024-01-16T14:57:12.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7efffd400640 1 -- 172.21.15.141:0/1976550217 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff802ce50 con 0x7f000015e3b0 2024-01-16T14:57:12.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7efffd400640 1 -- 172.21.15.141:0/1976550217 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff8034660 con 0x7f000015e3b0 2024-01-16T14:57:12.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7f0005000640 1 -- 172.21.15.141:0/1976550217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 msgr2=0x7f0000164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7f0005000640 1 --2- 172.21.15.141:0/1976550217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000164de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efff8009d90 tx=0x7efff802bf40 comp rx=0 tx=0).stop 2024-01-16T14:57:12.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7f0005000640 1 -- 172.21.15.141:0/1976550217 shutdown_connections 2024-01-16T14:57:12.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7f0005000640 1 --2- 172.21.15.141:0/1976550217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000164de0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.183+0000 7f0005000640 1 -- 172.21.15.141:0/1976550217 >> 172.21.15.141:0/1976550217 conn(0x7f000006a800 msgr2=0x7f000006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:12.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.184+0000 7f0005000640 1 -- 172.21.15.141:0/1976550217 shutdown_connections 2024-01-16T14:57:12.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.184+0000 7f0005000640 1 -- 172.21.15.141:0/1976550217 wait complete. 2024-01-16T14:57:12.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.185+0000 7f0005000640 1 Processor -- start 2024-01-16T14:57:12.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.185+0000 7f0005000640 1 -- start start 2024-01-16T14:57:12.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.186+0000 7f0005000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00000cf9a0 0x7f00001f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.186+0000 7f0005000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0000143ae0 0x7f00001419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.186+0000 7f0005000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.186+0000 7f0005000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00001f6250 con 0x7f000015e3b0 2024-01-16T14:57:12.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.186+0000 7f0005000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0000141380 con 0x7f0000143ae0 2024-01-16T14:57:12.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.186+0000 7f0005000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0000140770 con 0x7f00000cf9a0 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59276/0 (socket says 172.21.15.141:59276) 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 -- 172.21.15.141:0/2709502865 learned_addr learned my addr 172.21.15.141:0/2709502865 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efff7e00640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0000143ae0 0x7f00001419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 -- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00000cf9a0 msgr2=0x7f00001f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00000cf9a0 0x7f00001f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 -- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0000143ae0 msgr2=0x7f00001419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0000143ae0 0x7f00001419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 -- 172.21.15.141:0/2709502865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff8009a70 con 0x7f000015e3b0 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efffe800640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000141ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efff8037040 tx=0x7efff8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.187+0000 7efff6000640 1 -- 172.21.15.141:0/2709502865 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff803d600 con 0x7f000015e3b0 2024-01-16T14:57:12.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.188+0000 7efff6000640 1 -- 172.21.15.141:0/2709502865 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff803d760 con 0x7f000015e3b0 2024-01-16T14:57:12.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.188+0000 7efff6000640 1 -- 172.21.15.141:0/2709502865 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff803db10 con 0x7f000015e3b0 2024-01-16T14:57:12.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.188+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0000140a50 con 0x7f000015e3b0 2024-01-16T14:57:12.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.188+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0000141010 con 0x7f000015e3b0 2024-01-16T14:57:12.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.189+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00001f2e60 con 0x7f000015e3b0 2024-01-16T14:57:12.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.190+0000 7efff6000640 1 -- 172.21.15.141:0/2709502865 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efff803b040 con 0x7f000015e3b0 2024-01-16T14:57:12.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.190+0000 7efff6000640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7effd806b2f0 0x7effd806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.190+0000 7efff6000640 1 -- 172.21.15.141:0/2709502865 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efff80ac120 con 0x7f000015e3b0 2024-01-16T14:57:12.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.191+0000 7efffde00640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7effd806b2f0 0x7effd806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.192+0000 7efffde00640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7effd806b2f0 0x7effd806d7b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efff0000f60 tx=0x7efff00079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:12.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.195+0000 7efff6000640 1 -- 172.21.15.141:0/2709502865 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efff80770a0 con 0x7f000015e3b0 2024-01-16T14:57:12.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.384+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0000165320 con 0x7effd806b2f0 2024-01-16T14:57:12.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.405+0000 7efff6000640 1 -- 172.21.15.141:0/2709502865 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264470 (secure 0 0 0) 0x7efff0005080 con 0x7effd806b2f0 2024-01-16T14:57:12.391 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:12.392 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7effd806b2f0 msgr2=0x7effd806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7effd806b2f0 0x7effd806d7b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efff0000f60 tx=0x7efff00079b0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 msgr2=0x7f0000141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000141ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efff8037040 tx=0x7efff8004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 shutdown_connections 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7effd806b2f0 0x7effd806d7b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f000015e3b0 0x7f0000141ee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0000143ae0 0x7f00001419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 --2- 172.21.15.141:0/2709502865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00000cf9a0 0x7f00001f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 >> 172.21.15.141:0/2709502865 conn(0x7f000006a800 msgr2=0x7f0000142e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 shutdown_connections 2024-01-16T14:57:12.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.410+0000 7f0005000640 1 -- 172.21.15.141:0/2709502865 wait complete. 2024-01-16T14:57:12.412 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-16T14:57:10.436015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219492,"kb_used_data":2972,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755228,"statfs":{"total":773094113280,"available":772869353472,"internally_reserved":0,"allocated":3043328,"data_stored":1809080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.923739"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361647+0000","last_change":"2024-01-16T14:57:07.388330+0000","last_active":"2024-01-16T14:57:08.361647+0000","last_peered":"2024-01-16T14:57:08.361647+0000","last_clean":"2024-01-16T14:57:08.361647+0000","last_became_active":"2024-01-16T14:57:07.387937+0000","last_became_peered":"2024-01-16T14:57:07.387937+0000","last_unstale":"2024-01-16T14:57:08.361647+0000","last_undegraded":"2024-01-16T14:57:08.361647+0000","last_fullsized":"2024-01-16T14:57:08.361647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859055+0000","last_change":"2024-01-16T14:57:07.376852+0000","last_active":"2024-01-16T14:57:08.859055+0000","last_peered":"2024-01-16T14:57:08.859055+0000","last_clean":"2024-01-16T14:57:08.859055+0000","last_became_active":"2024-01-16T14:57:07.376562+0000","last_became_peered":"2024-01-16T14:57:07.376562+0000","last_unstale":"2024-01-16T14:57:08.859055+0000","last_undegraded":"2024-01-16T14:57:08.859055+0000","last_fullsized":"2024-01-16T14:57:08.859055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411451+0000","last_change":"2024-01-16T14:57:07.376399+0000","last_active":"2024-01-16T14:57:08.411451+0000","last_peered":"2024-01-16T14:57:08.411451+0000","last_clean":"2024-01-16T14:57:08.411451+0000","last_became_active":"2024-01-16T14:57:07.374502+0000","last_became_peered":"2024-01-16T14:57:07.374502+0000","last_unstale":"2024-01-16T14:57:08.411451+0000","last_undegraded":"2024-01-16T14:57:08.411451+0000","last_fullsized":"2024-01-16T14:57:08.411451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357034+0000","last_change":"2024-01-16T14:57:07.393925+0000","last_active":"2024-01-16T14:57:08.357034+0000","last_peered":"2024-01-16T14:57:08.357034+0000","last_clean":"2024-01-16T14:57:08.357034+0000","last_became_active":"2024-01-16T14:57:07.393608+0000","last_became_peered":"2024-01-16T14:57:07.393608+0000","last_unstale":"2024-01-16T14:57:08.357034+0000","last_undegraded":"2024-01-16T14:57:08.357034+0000","last_fullsized":"2024-01-16T14:57:08.357034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:12.497634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360231+0000","last_change":"2024-01-16T14:57:07.370383+0000","last_active":"2024-01-16T14:57:08.360231+0000","last_peered":"2024-01-16T14:57:08.360231+0000","last_clean":"2024-01-16T14:57:08.360231+0000","last_became_active":"2024-01-16T14:57:07.369925+0000","last_became_peered":"2024-01-16T14:57:07.369925+0000","last_unstale":"2024-01-16T14:57:08.360231+0000","last_undegraded":"2024-01-16T14:57:08.360231+0000","last_fullsized":"2024-01-16T14:57:08.360231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357036+0000","last_change":"2024-01-16T14:57:07.392767+0000","last_active":"2024-01-16T14:57:08.357036+0000","last_peered":"2024-01-16T14:57:08.357036+0000","last_clean":"2024-01-16T14:57:08.357036+0000","last_became_active":"2024-01-16T14:57:07.392457+0000","last_became_peered":"2024-01-16T14:57:07.392457+0000","last_unstale":"2024-01-16T14:57:08.357036+0000","last_undegraded":"2024-01-16T14:57:08.357036+0000","last_fullsized":"2024-01-16T14:57:08.357036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359822+0000","last_change":"2024-01-16T14:57:07.377276+0000","last_active":"2024-01-16T14:57:08.359822+0000","last_peered":"2024-01-16T14:57:08.359822+0000","last_clean":"2024-01-16T14:57:08.359822+0000","last_became_active":"2024-01-16T14:57:07.376937+0000","last_became_peered":"2024-01-16T14:57:07.376937+0000","last_unstale":"2024-01-16T14:57:08.359822+0000","last_undegraded":"2024-01-16T14:57:08.359822+0000","last_fullsized":"2024-01-16T14:57:08.359822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857828+0000","last_change":"2024-01-16T14:57:07.394001+0000","last_active":"2024-01-16T14:57:08.857828+0000","last_peered":"2024-01-16T14:57:08.857828+0000","last_clean":"2024-01-16T14:57:08.857828+0000","last_became_active":"2024-01-16T14:57:07.393670+0000","last_became_peered":"2024-01-16T14:57:07.393670+0000","last_unstale":"2024-01-16T14:57:08.857828+0000","last_undegraded":"2024-01-16T14:57:08.857828+0000","last_fullsized":"2024-01-16T14:57:08.857828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361786+0000","last_change":"2024-01-16T14:57:07.394940+0000","last_active":"2024-01-16T14:57:08.361786+0000","last_peered":"2024-01-16T14:57:08.361786+0000","last_clean":"2024-01-16T14:57:08.361786+0000","last_became_active":"2024-01-16T14:57:07.394335+0000","last_became_peered":"2024-01-16T14:57:07.394335+0000","last_unstale":"2024-01-16T14:57:08.361786+0000","last_undegraded":"2024-01-16T14:57:08.361786+0000","last_fullsized":"2024-01-16T14:57:08.361786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:23.443732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857711+0000","last_change":"2024-01-16T14:57:07.395598+0000","last_active":"2024-01-16T14:57:08.857711+0000","last_peered":"2024-01-16T14:57:08.857711+0000","last_clean":"2024-01-16T14:57:08.857711+0000","last_became_active":"2024-01-16T14:57:07.395367+0000","last_became_peered":"2024-01-16T14:57:07.395367+0000","last_unstale":"2024-01-16T14:57:08.857711+0000","last_undegraded":"2024-01-16T14:57:08.857711+0000","last_fullsized":"2024-01-16T14:57:08.857711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857014+0000","last_change":"2024-01-16T14:57:07.394374+0000","last_active":"2024-01-16T14:57:08.857014+0000","last_peered":"2024-01-16T14:57:08.857014+0000","last_clean":"2024-01-16T14:57:08.857014+0000","last_became_active":"2024-01-16T14:57:07.394125+0000","last_became_peered":"2024-01-16T14:57:07.394125+0000","last_unstale":"2024-01-16T14:57:08.857014+0000","last_undegraded":"2024-01-16T14:57:08.857014+0000","last_fullsized":"2024-01-16T14:57:08.857014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357258+0000","last_change":"2024-01-16T14:57:07.382452+0000","last_active":"2024-01-16T14:57:08.357258+0000","last_peered":"2024-01-16T14:57:08.357258+0000","last_clean":"2024-01-16T14:57:08.357258+0000","last_became_active":"2024-01-16T14:57:07.381985+0000","last_became_peered":"2024-01-16T14:57:07.381985+0000","last_unstale":"2024-01-16T14:57:08.357258+0000","last_undegraded":"2024-01-16T14:57:08.357258+0000","last_fullsized":"2024-01-16T14:57:08.357258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:31.589732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858575+0000","last_change":"2024-01-16T14:57:07.385013+0000","last_active":"2024-01-16T14:57:08.858575+0000","last_peered":"2024-01-16T14:57:08.858575+0000","last_clean":"2024-01-16T14:57:08.858575+0000","last_became_active":"2024-01-16T14:57:07.384705+0000","last_became_peered":"2024-01-16T14:57:07.384705+0000","last_unstale":"2024-01-16T14:57:08.858575+0000","last_undegraded":"2024-01-16T14:57:08.858575+0000","last_fullsized":"2024-01-16T14:57:08.858575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411099+0000","last_change":"2024-01-16T14:57:07.382942+0000","last_active":"2024-01-16T14:57:08.411099+0000","last_peered":"2024-01-16T14:57:08.411099+0000","last_clean":"2024-01-16T14:57:08.411099+0000","last_became_active":"2024-01-16T14:57:07.382609+0000","last_became_peered":"2024-01-16T14:57:07.382609+0000","last_unstale":"2024-01-16T14:57:08.411099+0000","last_undegraded":"2024-01-16T14:57:08.411099+0000","last_fullsized":"2024-01-16T14:57:08.411099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:07.098165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856153+0000","last_change":"2024-01-16T14:57:07.369499+0000","last_active":"2024-01-16T14:57:08.856153+0000","last_peered":"2024-01-16T14:57:08.856153+0000","last_clean":"2024-01-16T14:57:08.856153+0000","last_became_active":"2024-01-16T14:57:07.369089+0000","last_became_peered":"2024-01-16T14:57:07.369089+0000","last_unstale":"2024-01-16T14:57:08.856153+0000","last_undegraded":"2024-01-16T14:57:08.856153+0000","last_fullsized":"2024-01-16T14:57:08.856153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:45:27.672659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.355734+0000","last_change":"2024-01-16T14:57:07.392502+0000","last_active":"2024-01-16T14:57:08.355734+0000","last_peered":"2024-01-16T14:57:08.355734+0000","last_clean":"2024-01-16T14:57:08.355734+0000","last_became_active":"2024-01-16T14:57:07.392065+0000","last_became_peered":"2024-01-16T14:57:07.392065+0000","last_unstale":"2024-01-16T14:57:08.355734+0000","last_undegraded":"2024-01-16T14:57:08.355734+0000","last_fullsized":"2024-01-16T14:57:08.355734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358909+0000","last_change":"2024-01-16T14:57:07.394712+0000","last_active":"2024-01-16T14:57:08.358909+0000","last_peered":"2024-01-16T14:57:08.358909+0000","last_clean":"2024-01-16T14:57:08.358909+0000","last_became_active":"2024-01-16T14:57:07.393954+0000","last_became_peered":"2024-01-16T14:57:07.393954+0000","last_unstale":"2024-01-16T14:57:08.358909+0000","last_undegraded":"2024-01-16T14:57:08.358909+0000","last_fullsized":"2024-01-16T14:57:08.358909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859744+0000","last_change":"2024-01-16T14:57:07.384626+0000","last_active":"2024-01-16T14:57:08.859744+0000","last_peered":"2024-01-16T14:57:08.859744+0000","last_clean":"2024-01-16T14:57:08.859744+0000","last_became_active":"2024-01-16T14:57:07.383860+0000","last_became_peered":"2024-01-16T14:57:07.383860+0000","last_unstale":"2024-01-16T14:57:08.859744+0000","last_undegraded":"2024-01-16T14:57:08.859744+0000","last_fullsized":"2024-01-16T14:57:08.859744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361107+0000","last_change":"2024-01-16T14:57:07.375013+0000","last_active":"2024-01-16T14:57:08.361107+0000","last_peered":"2024-01-16T14:57:08.361107+0000","last_clean":"2024-01-16T14:57:08.361107+0000","last_became_active":"2024-01-16T14:57:07.373630+0000","last_became_peered":"2024-01-16T14:57:07.373630+0000","last_unstale":"2024-01-16T14:57:08.361107+0000","last_undegraded":"2024-01-16T14:57:08.361107+0000","last_fullsized":"2024-01-16T14:57:08.361107+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857112+0000","last_change":"2024-01-16T14:57:07.375790+0000","last_active":"2024-01-16T14:57:08.857112+0000","last_peered":"2024-01-16T14:57:08.857112+0000","last_clean":"2024-01-16T14:57:08.857112+0000","last_became_active":"2024-01-16T14:57:07.375533+0000","last_became_peered":"2024-01-16T14:57:07.375533+0000","last_unstale":"2024-01-16T14:57:08.857112+0000","last_undegraded":"2024-01-16T14:57:08.857112+0000","last_fullsized":"2024-01-16T14:57:08.857112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411312+0000","last_change":"2024-01-16T14:57:07.378765+0000","last_active":"2024-01-16T14:57:08.411312+0000","last_peered":"2024-01-16T14:57:08.411312+0000","last_clean":"2024-01-16T14:57:08.411312+0000","last_became_active":"2024-01-16T14:57:07.377672+0000","last_became_peered":"2024-01-16T14:57:07.377672+0000","last_unstale":"2024-01-16T14:57:08.411312+0000","last_undegraded":"2024-01-16T14:57:08.411312+0000","last_fullsized":"2024-01-16T14:57:08.411312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357439+0000","last_change":"2024-01-16T14:57:07.392203+0000","last_active":"2024-01-16T14:57:08.357439+0000","last_peered":"2024-01-16T14:57:08.357439+0000","last_clean":"2024-01-16T14:57:08.357439+0000","last_became_active":"2024-01-16T14:57:07.391913+0000","last_became_peered":"2024-01-16T14:57:07.391913+0000","last_unstale":"2024-01-16T14:57:08.357439+0000","last_undegraded":"2024-01-16T14:57:08.357439+0000","last_fullsized":"2024-01-16T14:57:08.357439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858112+0000","last_change":"2024-01-16T14:57:07.383055+0000","last_active":"2024-01-16T14:57:08.858112+0000","last_peered":"2024-01-16T14:57:08.858112+0000","last_clean":"2024-01-16T14:57:08.858112+0000","last_became_active":"2024-01-16T14:57:07.382181+0000","last_became_peered":"2024-01-16T14:57:07.382181+0000","last_unstale":"2024-01-16T14:57:08.858112+0000","last_undegraded":"2024-01-16T14:57:08.858112+0000","last_fullsized":"2024-01-16T14:57:08.858112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.356639+0000","last_change":"2024-01-16T14:57:07.391594+0000","last_active":"2024-01-16T14:57:08.356639+0000","last_peered":"2024-01-16T14:57:08.356639+0000","last_clean":"2024-01-16T14:57:08.356639+0000","last_became_active":"2024-01-16T14:57:07.391116+0000","last_became_peered":"2024-01-16T14:57:07.391116+0000","last_unstale":"2024-01-16T14:57:08.356639+0000","last_undegraded":"2024-01-16T14:57:08.356639+0000","last_fullsized":"2024-01-16T14:57:08.356639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859255+0000","last_change":"2024-01-16T14:57:07.383060+0000","last_active":"2024-01-16T14:57:08.859255+0000","last_peered":"2024-01-16T14:57:08.859255+0000","last_clean":"2024-01-16T14:57:08.859255+0000","last_became_active":"2024-01-16T14:57:07.382192+0000","last_became_peered":"2024-01-16T14:57:07.382192+0000","last_unstale":"2024-01-16T14:57:08.859255+0000","last_undegraded":"2024-01-16T14:57:08.859255+0000","last_fullsized":"2024-01-16T14:57:08.859255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:34.791403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857952+0000","last_change":"2024-01-16T14:57:07.384727+0000","last_active":"2024-01-16T14:57:08.857952+0000","last_peered":"2024-01-16T14:57:08.857952+0000","last_clean":"2024-01-16T14:57:08.857952+0000","last_became_active":"2024-01-16T14:57:07.383334+0000","last_became_peered":"2024-01-16T14:57:07.383334+0000","last_unstale":"2024-01-16T14:57:08.857952+0000","last_undegraded":"2024-01-16T14:57:08.857952+0000","last_fullsized":"2024-01-16T14:57:08.857952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856861+0000","last_change":"2024-01-16T14:57:07.376359+0000","last_active":"2024-01-16T14:57:08.856861+0000","last_peered":"2024-01-16T14:57:08.856861+0000","last_clean":"2024-01-16T14:57:08.856861+0000","last_became_active":"2024-01-16T14:57:07.376074+0000","last_became_peered":"2024-01-16T14:57:07.376074+0000","last_unstale":"2024-01-16T14:57:08.856861+0000","last_undegraded":"2024-01-16T14:57:08.856861+0000","last_fullsized":"2024-01-16T14:57:08.856861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858616+0000","last_change":"2024-01-16T14:57:07.387458+0000","last_active":"2024-01-16T14:57:08.858616+0000","last_peered":"2024-01-16T14:57:08.858616+0000","last_clean":"2024-01-16T14:57:08.858616+0000","last_became_active":"2024-01-16T14:57:07.387221+0000","last_became_peered":"2024-01-16T14:57:07.387221+0000","last_unstale":"2024-01-16T14:57:08.858616+0000","last_undegraded":"2024-01-16T14:57:08.858616+0000","last_fullsized":"2024-01-16T14:57:08.858616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411330+0000","last_change":"2024-01-16T14:57:07.377884+0000","last_active":"2024-01-16T14:57:08.411330+0000","last_peered":"2024-01-16T14:57:08.411330+0000","last_clean":"2024-01-16T14:57:08.411330+0000","last_became_active":"2024-01-16T14:57:07.377585+0000","last_became_peered":"2024-01-16T14:57:07.377585+0000","last_unstale":"2024-01-16T14:57:08.411330+0000","last_undegraded":"2024-01-16T14:57:08.411330+0000","last_fullsized":"2024-01-16T14:57:08.411330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358051+0000","last_change":"2024-01-16T14:57:07.389650+0000","last_active":"2024-01-16T14:57:08.358051+0000","last_peered":"2024-01-16T14:57:08.358051+0000","last_clean":"2024-01-16T14:57:08.358051+0000","last_became_active":"2024-01-16T14:57:07.389388+0000","last_became_peered":"2024-01-16T14:57:07.389388+0000","last_unstale":"2024-01-16T14:57:08.358051+0000","last_undegraded":"2024-01-16T14:57:08.358051+0000","last_fullsized":"2024-01-16T14:57:08.358051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858338+0000","last_change":"2024-01-16T14:57:07.393121+0000","last_active":"2024-01-16T14:57:08.858338+0000","last_peered":"2024-01-16T14:57:08.858338+0000","last_clean":"2024-01-16T14:57:08.858338+0000","last_became_active":"2024-01-16T14:57:07.392839+0000","last_became_peered":"2024-01-16T14:57:07.392839+0000","last_unstale":"2024-01-16T14:57:08.858338+0000","last_undegraded":"2024-01-16T14:57:08.858338+0000","last_fullsized":"2024-01-16T14:57:08.858338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856756+0000","last_change":"2024-01-16T14:57:07.386156+0000","last_active":"2024-01-16T14:57:08.856756+0000","last_peered":"2024-01-16T14:57:08.856756+0000","last_clean":"2024-01-16T14:57:08.856756+0000","last_became_active":"2024-01-16T14:57:07.385684+0000","last_became_peered":"2024-01-16T14:57:07.385684+0000","last_unstale":"2024-01-16T14:57:08.856756+0000","last_undegraded":"2024-01-16T14:57:08.856756+0000","last_fullsized":"2024-01-16T14:57:08.856756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357739+0000","last_change":"2024-01-16T14:57:07.372196+0000","last_active":"2024-01-16T14:57:08.357739+0000","last_peered":"2024-01-16T14:57:08.357739+0000","last_clean":"2024-01-16T14:57:08.357739+0000","last_became_active":"2024-01-16T14:57:07.369065+0000","last_became_peered":"2024-01-16T14:57:07.369065+0000","last_unstale":"2024-01-16T14:57:08.357739+0000","last_undegraded":"2024-01-16T14:57:08.357739+0000","last_fullsized":"2024-01-16T14:57:08.357739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858827+0000","last_change":"2024-01-16T14:57:06.556368+0000","last_active":"2024-01-16T14:57:08.858827+0000","last_peered":"2024-01-16T14:57:08.858827+0000","last_clean":"2024-01-16T14:57:08.858827+0000","last_became_active":"2024-01-16T14:57:06.555953+0000","last_became_peered":"2024-01-16T14:57:06.555953+0000","last_unstale":"2024-01-16T14:57:08.858827+0000","last_undegraded":"2024-01-16T14:57:08.858827+0000","last_fullsized":"2024-01-16T14:57:08.858827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:45.773231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411448+0000","last_change":"2024-01-16T14:57:07.383151+0000","last_active":"2024-01-16T14:57:08.411448+0000","last_peered":"2024-01-16T14:57:08.411448+0000","last_clean":"2024-01-16T14:57:08.411448+0000","last_became_active":"2024-01-16T14:57:07.381555+0000","last_became_peered":"2024-01-16T14:57:07.381555+0000","last_unstale":"2024-01-16T14:57:08.411448+0000","last_undegraded":"2024-01-16T14:57:08.411448+0000","last_fullsized":"2024-01-16T14:57:08.411448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856949+0000","last_change":"2024-01-16T14:57:06.555306+0000","last_active":"2024-01-16T14:57:08.856949+0000","last_peered":"2024-01-16T14:57:08.856949+0000","last_clean":"2024-01-16T14:57:08.856949+0000","last_became_active":"2024-01-16T14:57:06.555063+0000","last_became_peered":"2024-01-16T14:57:06.555063+0000","last_unstale":"2024-01-16T14:57:08.856949+0000","last_undegraded":"2024-01-16T14:57:08.856949+0000","last_fullsized":"2024-01-16T14:57:08.856949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:07.271545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857979+0000","last_change":"2024-01-16T14:57:06.555279+0000","last_active":"2024-01-16T14:57:08.857979+0000","last_peered":"2024-01-16T14:57:08.857979+0000","last_clean":"2024-01-16T14:57:08.857979+0000","last_became_active":"2024-01-16T14:57:06.554559+0000","last_became_peered":"2024-01-16T14:57:06.554559+0000","last_unstale":"2024-01-16T14:57:08.857979+0000","last_undegraded":"2024-01-16T14:57:08.857979+0000","last_fullsized":"2024-01-16T14:57:08.857979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857830+0000","last_change":"2024-01-16T14:57:07.382661+0000","last_active":"2024-01-16T14:57:08.857830+0000","last_peered":"2024-01-16T14:57:08.857830+0000","last_clean":"2024-01-16T14:57:08.857830+0000","last_became_active":"2024-01-16T14:57:07.382055+0000","last_became_peered":"2024-01-16T14:57:07.382055+0000","last_unstale":"2024-01-16T14:57:08.857830+0000","last_undegraded":"2024-01-16T14:57:08.857830+0000","last_fullsized":"2024-01-16T14:57:08.857830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357563+0000","last_change":"2024-01-16T14:57:06.558562+0000","last_active":"2024-01-16T14:57:08.357563+0000","last_peered":"2024-01-16T14:57:08.357563+0000","last_clean":"2024-01-16T14:57:08.357563+0000","last_became_active":"2024-01-16T14:57:06.558166+0000","last_became_peered":"2024-01-16T14:57:06.558166+0000","last_unstale":"2024-01-16T14:57:08.357563+0000","last_undegraded":"2024-01-16T14:57:08.357563+0000","last_fullsized":"2024-01-16T14:57:08.357563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:31.817106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855928+0000","last_change":"2024-01-16T14:57:07.377917+0000","last_active":"2024-01-16T14:57:08.855928+0000","last_peered":"2024-01-16T14:57:08.855928+0000","last_clean":"2024-01-16T14:57:08.855928+0000","last_became_active":"2024-01-16T14:57:07.377601+0000","last_became_peered":"2024-01-16T14:57:07.377601+0000","last_unstale":"2024-01-16T14:57:08.855928+0000","last_undegraded":"2024-01-16T14:57:08.855928+0000","last_fullsized":"2024-01-16T14:57:08.855928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:32.968418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857555+0000","last_change":"2024-01-16T14:57:06.563826+0000","last_active":"2024-01-16T14:57:08.857555+0000","last_peered":"2024-01-16T14:57:08.857555+0000","last_clean":"2024-01-16T14:57:08.857555+0000","last_became_active":"2024-01-16T14:57:06.563529+0000","last_became_peered":"2024-01-16T14:57:06.563529+0000","last_unstale":"2024-01-16T14:57:08.857555+0000","last_undegraded":"2024-01-16T14:57:08.857555+0000","last_fullsized":"2024-01-16T14:57:08.857555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358434+0000","last_change":"2024-01-16T14:57:07.391665+0000","last_active":"2024-01-16T14:57:08.358434+0000","last_peered":"2024-01-16T14:57:08.358434+0000","last_clean":"2024-01-16T14:57:08.358434+0000","last_became_active":"2024-01-16T14:57:07.391071+0000","last_became_peered":"2024-01-16T14:57:07.391071+0000","last_unstale":"2024-01-16T14:57:08.358434+0000","last_undegraded":"2024-01-16T14:57:08.358434+0000","last_fullsized":"2024-01-16T14:57:08.358434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358657+0000","last_change":"2024-01-16T14:57:06.560447+0000","last_active":"2024-01-16T14:57:08.358657+0000","last_peered":"2024-01-16T14:57:08.358657+0000","last_clean":"2024-01-16T14:57:08.358657+0000","last_became_active":"2024-01-16T14:57:06.560116+0000","last_became_peered":"2024-01-16T14:57:06.560116+0000","last_unstale":"2024-01-16T14:57:08.358657+0000","last_undegraded":"2024-01-16T14:57:08.358657+0000","last_fullsized":"2024-01-16T14:57:08.358657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:33.296388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411729+0000","last_change":"2024-01-16T14:57:07.396020+0000","last_active":"2024-01-16T14:57:08.411729+0000","last_peered":"2024-01-16T14:57:08.411729+0000","last_clean":"2024-01-16T14:57:08.411729+0000","last_became_active":"2024-01-16T14:57:07.395685+0000","last_became_peered":"2024-01-16T14:57:07.395685+0000","last_unstale":"2024-01-16T14:57:08.411729+0000","last_undegraded":"2024-01-16T14:57:08.411729+0000","last_fullsized":"2024-01-16T14:57:08.411729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857123+0000","last_change":"2024-01-16T14:57:06.553623+0000","last_active":"2024-01-16T14:57:08.857123+0000","last_peered":"2024-01-16T14:57:08.857123+0000","last_clean":"2024-01-16T14:57:08.857123+0000","last_became_active":"2024-01-16T14:57:06.553173+0000","last_became_peered":"2024-01-16T14:57:06.553173+0000","last_unstale":"2024-01-16T14:57:08.857123+0000","last_undegraded":"2024-01-16T14:57:08.857123+0000","last_fullsized":"2024-01-16T14:57:08.857123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357536+0000","last_change":"2024-01-16T14:57:07.382256+0000","last_active":"2024-01-16T14:57:08.357536+0000","last_peered":"2024-01-16T14:57:08.357536+0000","last_clean":"2024-01-16T14:57:08.357536+0000","last_became_active":"2024-01-16T14:57:07.381640+0000","last_became_peered":"2024-01-16T14:57:07.381640+0000","last_unstale":"2024-01-16T14:57:08.357536+0000","last_undegraded":"2024-01-16T14:57:08.357536+0000","last_fullsized":"2024-01-16T14:57:08.357536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:14.206644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857022+0000","last_change":"2024-01-16T14:57:06.556835+0000","last_active":"2024-01-16T14:57:08.857022+0000","last_peered":"2024-01-16T14:57:08.857022+0000","last_clean":"2024-01-16T14:57:08.857022+0000","last_became_active":"2024-01-16T14:57:06.556483+0000","last_became_peered":"2024-01-16T14:57:06.556483+0000","last_unstale":"2024-01-16T14:57:08.857022+0000","last_undegraded":"2024-01-16T14:57:08.857022+0000","last_fullsized":"2024-01-16T14:57:08.857022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:01.863989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360686+0000","last_change":"2024-01-16T14:57:07.380047+0000","last_active":"2024-01-16T14:57:08.360686+0000","last_peered":"2024-01-16T14:57:08.360686+0000","last_clean":"2024-01-16T14:57:08.360686+0000","last_became_active":"2024-01-16T14:57:07.379698+0000","last_became_peered":"2024-01-16T14:57:07.379698+0000","last_unstale":"2024-01-16T14:57:08.360686+0000","last_undegraded":"2024-01-16T14:57:08.360686+0000","last_fullsized":"2024-01-16T14:57:08.360686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411453+0000","last_change":"2024-01-16T14:57:06.555448+0000","last_active":"2024-01-16T14:57:08.411453+0000","last_peered":"2024-01-16T14:57:08.411453+0000","last_clean":"2024-01-16T14:57:08.411453+0000","last_became_active":"2024-01-16T14:57:06.555097+0000","last_became_peered":"2024-01-16T14:57:06.555097+0000","last_unstale":"2024-01-16T14:57:08.411453+0000","last_undegraded":"2024-01-16T14:57:08.411453+0000","last_fullsized":"2024-01-16T14:57:08.411453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:23.858452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411172+0000","last_change":"2024-01-16T14:57:07.380436+0000","last_active":"2024-01-16T14:57:08.411172+0000","last_peered":"2024-01-16T14:57:08.411172+0000","last_clean":"2024-01-16T14:57:08.411172+0000","last_became_active":"2024-01-16T14:57:07.380148+0000","last_became_peered":"2024-01-16T14:57:07.380148+0000","last_unstale":"2024-01-16T14:57:08.411172+0000","last_undegraded":"2024-01-16T14:57:08.411172+0000","last_fullsized":"2024-01-16T14:57:08.411172+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358031+0000","last_change":"2024-01-16T14:57:06.549803+0000","last_active":"2024-01-16T14:57:08.358031+0000","last_peered":"2024-01-16T14:57:08.358031+0000","last_clean":"2024-01-16T14:57:08.358031+0000","last_became_active":"2024-01-16T14:57:06.549362+0000","last_became_peered":"2024-01-16T14:57:06.549362+0000","last_unstale":"2024-01-16T14:57:08.358031+0000","last_undegraded":"2024-01-16T14:57:08.358031+0000","last_fullsized":"2024-01-16T14:57:08.358031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360491+0000","last_change":"2024-01-16T14:57:07.384510+0000","last_active":"2024-01-16T14:57:08.360491+0000","last_peered":"2024-01-16T14:57:08.360491+0000","last_clean":"2024-01-16T14:57:08.360491+0000","last_became_active":"2024-01-16T14:57:07.384214+0000","last_became_peered":"2024-01-16T14:57:07.384214+0000","last_unstale":"2024-01-16T14:57:08.360491+0000","last_undegraded":"2024-01-16T14:57:08.360491+0000","last_fullsized":"2024-01-16T14:57:08.360491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856634+0000","last_change":"2024-01-16T14:57:06.562353+0000","last_active":"2024-01-16T14:57:08.856634+0000","last_peered":"2024-01-16T14:57:08.856634+0000","last_clean":"2024-01-16T14:57:08.856634+0000","last_became_active":"2024-01-16T14:57:06.562043+0000","last_became_peered":"2024-01-16T14:57:06.562043+0000","last_unstale":"2024-01-16T14:57:08.856634+0000","last_undegraded":"2024-01-16T14:57:08.856634+0000","last_fullsized":"2024-01-16T14:57:08.856634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:25.879858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856809+0000","last_change":"2024-01-16T14:57:07.383191+0000","last_active":"2024-01-16T14:57:08.856809+0000","last_peered":"2024-01-16T14:57:08.856809+0000","last_clean":"2024-01-16T14:57:08.856809+0000","last_became_active":"2024-01-16T14:57:07.379683+0000","last_became_peered":"2024-01-16T14:57:07.379683+0000","last_unstale":"2024-01-16T14:57:08.856809+0000","last_undegraded":"2024-01-16T14:57:08.856809+0000","last_fullsized":"2024-01-16T14:57:08.856809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856935+0000","last_change":"2024-01-16T14:57:06.548684+0000","last_active":"2024-01-16T14:57:08.856935+0000","last_peered":"2024-01-16T14:57:08.856935+0000","last_clean":"2024-01-16T14:57:08.856935+0000","last_became_active":"2024-01-16T14:57:06.547835+0000","last_became_peered":"2024-01-16T14:57:06.547835+0000","last_unstale":"2024-01-16T14:57:08.856935+0000","last_undegraded":"2024-01-16T14:57:08.856935+0000","last_fullsized":"2024-01-16T14:57:08.856935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:46.715195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360277+0000","last_change":"2024-01-16T14:57:07.384606+0000","last_active":"2024-01-16T14:57:08.360277+0000","last_peered":"2024-01-16T14:57:08.360277+0000","last_clean":"2024-01-16T14:57:08.360277+0000","last_became_active":"2024-01-16T14:57:07.384300+0000","last_became_peered":"2024-01-16T14:57:07.384300+0000","last_unstale":"2024-01-16T14:57:08.360277+0000","last_undegraded":"2024-01-16T14:57:08.360277+0000","last_fullsized":"2024-01-16T14:57:08.360277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359317+0000","last_change":"2024-01-16T14:57:06.555188+0000","last_active":"2024-01-16T14:57:08.359317+0000","last_peered":"2024-01-16T14:57:08.359317+0000","last_clean":"2024-01-16T14:57:08.359317+0000","last_became_active":"2024-01-16T14:57:06.554904+0000","last_became_peered":"2024-01-16T14:57:06.554904+0000","last_unstale":"2024-01-16T14:57:08.359317+0000","last_undegraded":"2024-01-16T14:57:08.359317+0000","last_fullsized":"2024-01-16T14:57:08.359317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:59.674406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360712+0000","last_change":"2024-01-16T14:57:07.386645+0000","last_active":"2024-01-16T14:57:08.360712+0000","last_peered":"2024-01-16T14:57:08.360712+0000","last_clean":"2024-01-16T14:57:08.360712+0000","last_became_active":"2024-01-16T14:57:07.384338+0000","last_became_peered":"2024-01-16T14:57:07.384338+0000","last_unstale":"2024-01-16T14:57:08.360712+0000","last_undegraded":"2024-01-16T14:57:08.360712+0000","last_fullsized":"2024-01-16T14:57:08.360712+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859980+0000","last_change":"2024-01-16T14:57:06.549964+0000","last_active":"2024-01-16T14:57:08.859980+0000","last_peered":"2024-01-16T14:57:08.859980+0000","last_clean":"2024-01-16T14:57:08.859980+0000","last_became_active":"2024-01-16T14:57:06.549594+0000","last_became_peered":"2024-01-16T14:57:06.549594+0000","last_unstale":"2024-01-16T14:57:08.859980+0000","last_undegraded":"2024-01-16T14:57:08.859980+0000","last_fullsized":"2024-01-16T14:57:08.859980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361966+0000","last_change":"2024-01-16T14:57:07.385296+0000","last_active":"2024-01-16T14:57:08.361966+0000","last_peered":"2024-01-16T14:57:08.361966+0000","last_clean":"2024-01-16T14:57:08.361966+0000","last_became_active":"2024-01-16T14:57:07.385019+0000","last_became_peered":"2024-01-16T14:57:07.385019+0000","last_unstale":"2024-01-16T14:57:08.361966+0000","last_undegraded":"2024-01-16T14:57:08.361966+0000","last_fullsized":"2024-01-16T14:57:08.361966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:08.100477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857357+0000","last_change":"2024-01-16T14:57:06.558586+0000","last_active":"2024-01-16T14:57:08.857357+0000","last_peered":"2024-01-16T14:57:08.857357+0000","last_clean":"2024-01-16T14:57:08.857357+0000","last_became_active":"2024-01-16T14:57:06.558143+0000","last_became_peered":"2024-01-16T14:57:06.558143+0000","last_unstale":"2024-01-16T14:57:08.857357+0000","last_undegraded":"2024-01-16T14:57:08.857357+0000","last_fullsized":"2024-01-16T14:57:08.857357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357386+0000","last_change":"2024-01-16T14:57:07.393946+0000","last_active":"2024-01-16T14:57:08.357386+0000","last_peered":"2024-01-16T14:57:08.357386+0000","last_clean":"2024-01-16T14:57:08.357386+0000","last_became_active":"2024-01-16T14:57:07.393108+0000","last_became_peered":"2024-01-16T14:57:07.393108+0000","last_unstale":"2024-01-16T14:57:08.357386+0000","last_undegraded":"2024-01-16T14:57:08.357386+0000","last_fullsized":"2024-01-16T14:57:08.357386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858814+0000","last_change":"2024-01-16T14:57:06.555365+0000","last_active":"2024-01-16T14:57:08.858814+0000","last_peered":"2024-01-16T14:57:08.858814+0000","last_clean":"2024-01-16T14:57:08.858814+0000","last_became_active":"2024-01-16T14:57:06.551851+0000","last_became_peered":"2024-01-16T14:57:06.551851+0000","last_unstale":"2024-01-16T14:57:08.858814+0000","last_undegraded":"2024-01-16T14:57:08.858814+0000","last_fullsized":"2024-01-16T14:57:08.858814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360922+0000","last_change":"2024-01-16T14:57:07.397246+0000","last_active":"2024-01-16T14:57:08.360922+0000","last_peered":"2024-01-16T14:57:08.360922+0000","last_clean":"2024-01-16T14:57:08.360922+0000","last_became_active":"2024-01-16T14:57:07.394277+0000","last_became_peered":"2024-01-16T14:57:07.394277+0000","last_unstale":"2024-01-16T14:57:08.360922+0000","last_undegraded":"2024-01-16T14:57:08.360922+0000","last_fullsized":"2024-01-16T14:57:08.360922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:57:07.357749+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:57:07.357749+0000","last_peered":"2024-01-16T14:57:07.357749+0000","last_clean":"2024-01-16T14:57:07.357749+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:57:07.357749+0000","last_undegraded":"2024-01-16T14:57:07.357749+0000","last_fullsized":"2024-01-16T14:57:07.357749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357443+0000","last_change":"2024-01-16T14:57:06.547364+0000","last_active":"2024-01-16T14:57:08.357443+0000","last_peered":"2024-01-16T14:57:08.357443+0000","last_clean":"2024-01-16T14:57:08.357443+0000","last_became_active":"2024-01-16T14:57:06.546800+0000","last_became_peered":"2024-01-16T14:57:06.546800+0000","last_unstale":"2024-01-16T14:57:08.357443+0000","last_undegraded":"2024-01-16T14:57:08.357443+0000","last_fullsized":"2024-01-16T14:57:08.357443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:27.163769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858385+0000","last_change":"2024-01-16T14:57:07.395614+0000","last_active":"2024-01-16T14:57:08.858385+0000","last_peered":"2024-01-16T14:57:08.858385+0000","last_clean":"2024-01-16T14:57:08.858385+0000","last_became_active":"2024-01-16T14:57:07.395330+0000","last_became_peered":"2024-01-16T14:57:07.395330+0000","last_unstale":"2024-01-16T14:57:08.858385+0000","last_undegraded":"2024-01-16T14:57:08.858385+0000","last_fullsized":"2024-01-16T14:57:08.858385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359578+0000","last_change":"2024-01-16T14:57:06.548309+0000","last_active":"2024-01-16T14:57:08.359578+0000","last_peered":"2024-01-16T14:57:08.359578+0000","last_clean":"2024-01-16T14:57:08.359578+0000","last_became_active":"2024-01-16T14:57:06.547952+0000","last_became_peered":"2024-01-16T14:57:06.547952+0000","last_unstale":"2024-01-16T14:57:08.359578+0000","last_undegraded":"2024-01-16T14:57:08.359578+0000","last_fullsized":"2024-01-16T14:57:08.359578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.410994+0000","last_change":"2024-01-16T14:57:07.375198+0000","last_active":"2024-01-16T14:57:08.410994+0000","last_peered":"2024-01-16T14:57:08.410994+0000","last_clean":"2024-01-16T14:57:08.410994+0000","last_became_active":"2024-01-16T14:57:07.374043+0000","last_became_peered":"2024-01-16T14:57:07.374043+0000","last_unstale":"2024-01-16T14:57:08.410994+0000","last_undegraded":"2024-01-16T14:57:08.410994+0000","last_fullsized":"2024-01-16T14:57:08.410994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:11.783582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857406+0000","last_change":"2024-01-16T14:57:06.553851+0000","last_active":"2024-01-16T14:57:08.857406+0000","last_peered":"2024-01-16T14:57:08.857406+0000","last_clean":"2024-01-16T14:57:08.857406+0000","last_became_active":"2024-01-16T14:57:06.552712+0000","last_became_peered":"2024-01-16T14:57:06.552712+0000","last_unstale":"2024-01-16T14:57:08.857406+0000","last_undegraded":"2024-01-16T14:57:08.857406+0000","last_fullsized":"2024-01-16T14:57:08.857406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855802+0000","last_change":"2024-01-16T14:57:07.372041+0000","last_active":"2024-01-16T14:57:08.855802+0000","last_peered":"2024-01-16T14:57:08.855802+0000","last_clean":"2024-01-16T14:57:08.855802+0000","last_became_active":"2024-01-16T14:57:07.371750+0000","last_became_peered":"2024-01-16T14:57:07.371750+0000","last_unstale":"2024-01-16T14:57:08.855802+0000","last_undegraded":"2024-01-16T14:57:08.855802+0000","last_fullsized":"2024-01-16T14:57:08.855802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:53.804355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358274+0000","last_change":"2024-01-16T14:57:06.548768+0000","last_active":"2024-01-16T14:57:08.358274+0000","last_peered":"2024-01-16T14:57:08.358274+0000","last_clean":"2024-01-16T14:57:08.358274+0000","last_became_active":"2024-01-16T14:57:06.548446+0000","last_became_peered":"2024-01-16T14:57:06.548446+0000","last_unstale":"2024-01-16T14:57:08.358274+0000","last_undegraded":"2024-01-16T14:57:08.358274+0000","last_fullsized":"2024-01-16T14:57:08.358274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:06.341461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358850+0000","last_change":"2024-01-16T14:57:07.378261+0000","last_active":"2024-01-16T14:57:08.358850+0000","last_peered":"2024-01-16T14:57:08.358850+0000","last_clean":"2024-01-16T14:57:08.358850+0000","last_became_active":"2024-01-16T14:57:07.377940+0000","last_became_peered":"2024-01-16T14:57:07.377940+0000","last_unstale":"2024-01-16T14:57:08.358850+0000","last_undegraded":"2024-01-16T14:57:08.358850+0000","last_fullsized":"2024-01-16T14:57:08.358850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856741+0000","last_change":"2024-01-16T14:57:06.545628+0000","last_active":"2024-01-16T14:57:08.856741+0000","last_peered":"2024-01-16T14:57:08.856741+0000","last_clean":"2024-01-16T14:57:08.856741+0000","last_became_active":"2024-01-16T14:57:06.545341+0000","last_became_peered":"2024-01-16T14:57:06.545341+0000","last_unstale":"2024-01-16T14:57:08.856741+0000","last_undegraded":"2024-01-16T14:57:08.856741+0000","last_fullsized":"2024-01-16T14:57:08.856741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:54.829524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358936+0000","last_change":"2024-01-16T14:57:07.392468+0000","last_active":"2024-01-16T14:57:08.358936+0000","last_peered":"2024-01-16T14:57:08.358936+0000","last_clean":"2024-01-16T14:57:08.358936+0000","last_became_active":"2024-01-16T14:57:07.392091+0000","last_became_peered":"2024-01-16T14:57:07.392091+0000","last_unstale":"2024-01-16T14:57:08.358936+0000","last_undegraded":"2024-01-16T14:57:08.358936+0000","last_fullsized":"2024-01-16T14:57:08.358936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361159+0000","last_change":"2024-01-16T14:57:06.551276+0000","last_active":"2024-01-16T14:57:08.361159+0000","last_peered":"2024-01-16T14:57:08.361159+0000","last_clean":"2024-01-16T14:57:08.361159+0000","last_became_active":"2024-01-16T14:57:06.550951+0000","last_became_peered":"2024-01-16T14:57:06.550951+0000","last_unstale":"2024-01-16T14:57:08.361159+0000","last_undegraded":"2024-01-16T14:57:08.361159+0000","last_fullsized":"2024-01-16T14:57:08.361159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:15.986936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856799+0000","last_change":"2024-01-16T14:57:07.370960+0000","last_active":"2024-01-16T14:57:08.856799+0000","last_peered":"2024-01-16T14:57:08.856799+0000","last_clean":"2024-01-16T14:57:08.856799+0000","last_became_active":"2024-01-16T14:57:07.370635+0000","last_became_peered":"2024-01-16T14:57:07.370635+0000","last_unstale":"2024-01-16T14:57:08.856799+0000","last_undegraded":"2024-01-16T14:57:08.856799+0000","last_fullsized":"2024-01-16T14:57:08.856799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:36.350912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859279+0000","last_change":"2024-01-16T14:57:06.553886+0000","last_active":"2024-01-16T14:57:08.859279+0000","last_peered":"2024-01-16T14:57:08.859279+0000","last_clean":"2024-01-16T14:57:08.859279+0000","last_became_active":"2024-01-16T14:57:06.553590+0000","last_became_peered":"2024-01-16T14:57:06.553590+0000","last_unstale":"2024-01-16T14:57:08.859279+0000","last_undegraded":"2024-01-16T14:57:08.859279+0000","last_fullsized":"2024-01-16T14:57:08.859279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:16.749765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859350+0000","last_change":"2024-01-16T14:57:07.388410+0000","last_active":"2024-01-16T14:57:08.859350+0000","last_peered":"2024-01-16T14:57:08.859350+0000","last_clean":"2024-01-16T14:57:08.859350+0000","last_became_active":"2024-01-16T14:57:07.388132+0000","last_became_peered":"2024-01-16T14:57:07.388132+0000","last_unstale":"2024-01-16T14:57:08.859350+0000","last_undegraded":"2024-01-16T14:57:08.859350+0000","last_fullsized":"2024-01-16T14:57:08.859350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:24.013168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857725+0000","last_change":"2024-01-16T14:57:06.560743+0000","last_active":"2024-01-16T14:57:08.857725+0000","last_peered":"2024-01-16T14:57:08.857725+0000","last_clean":"2024-01-16T14:57:08.857725+0000","last_became_active":"2024-01-16T14:57:06.560557+0000","last_became_peered":"2024-01-16T14:57:06.560557+0000","last_unstale":"2024-01-16T14:57:08.857725+0000","last_undegraded":"2024-01-16T14:57:08.857725+0000","last_fullsized":"2024-01-16T14:57:08.857725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411687+0000","last_change":"2024-01-16T14:57:07.378876+0000","last_active":"2024-01-16T14:57:08.411687+0000","last_peered":"2024-01-16T14:57:08.411687+0000","last_clean":"2024-01-16T14:57:08.411687+0000","last_became_active":"2024-01-16T14:57:07.378103+0000","last_became_peered":"2024-01-16T14:57:07.378103+0000","last_unstale":"2024-01-16T14:57:08.411687+0000","last_undegraded":"2024-01-16T14:57:08.411687+0000","last_fullsized":"2024-01-16T14:57:08.411687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359610+0000","last_change":"2024-01-16T14:57:06.548228+0000","last_active":"2024-01-16T14:57:08.359610+0000","last_peered":"2024-01-16T14:57:08.359610+0000","last_clean":"2024-01-16T14:57:08.359610+0000","last_became_active":"2024-01-16T14:57:06.547577+0000","last_became_peered":"2024-01-16T14:57:06.547577+0000","last_unstale":"2024-01-16T14:57:08.359610+0000","last_undegraded":"2024-01-16T14:57:08.359610+0000","last_fullsized":"2024-01-16T14:57:08.359610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357044+0000","last_change":"2024-01-16T14:57:07.377694+0000","last_active":"2024-01-16T14:57:08.357044+0000","last_peered":"2024-01-16T14:57:08.357044+0000","last_clean":"2024-01-16T14:57:08.357044+0000","last_became_active":"2024-01-16T14:57:07.377016+0000","last_became_peered":"2024-01-16T14:57:07.377016+0000","last_unstale":"2024-01-16T14:57:08.357044+0000","last_undegraded":"2024-01-16T14:57:08.357044+0000","last_fullsized":"2024-01-16T14:57:08.357044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:08:30.500155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856442+0000","last_change":"2024-01-16T14:57:06.545409+0000","last_active":"2024-01-16T14:57:08.856442+0000","last_peered":"2024-01-16T14:57:08.856442+0000","last_clean":"2024-01-16T14:57:08.856442+0000","last_became_active":"2024-01-16T14:57:06.545133+0000","last_became_peered":"2024-01-16T14:57:06.545133+0000","last_unstale":"2024-01-16T14:57:08.856442+0000","last_undegraded":"2024-01-16T14:57:08.856442+0000","last_fullsized":"2024-01-16T14:57:08.856442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:18.053787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858230+0000","last_change":"2024-01-16T14:57:07.373879+0000","last_active":"2024-01-16T14:57:08.858230+0000","last_peered":"2024-01-16T14:57:08.858230+0000","last_clean":"2024-01-16T14:57:08.858230+0000","last_became_active":"2024-01-16T14:57:07.373537+0000","last_became_peered":"2024-01-16T14:57:07.373537+0000","last_unstale":"2024-01-16T14:57:08.858230+0000","last_undegraded":"2024-01-16T14:57:08.858230+0000","last_fullsized":"2024-01-16T14:57:08.858230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856963+0000","last_change":"2024-01-16T14:57:06.548121+0000","last_active":"2024-01-16T14:57:08.856963+0000","last_peered":"2024-01-16T14:57:08.856963+0000","last_clean":"2024-01-16T14:57:08.856963+0000","last_became_active":"2024-01-16T14:57:06.547868+0000","last_became_peered":"2024-01-16T14:57:06.547868+0000","last_unstale":"2024-01-16T14:57:08.856963+0000","last_undegraded":"2024-01-16T14:57:08.856963+0000","last_fullsized":"2024-01-16T14:57:08.856963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858908+0000","last_change":"2024-01-16T14:57:07.378203+0000","last_active":"2024-01-16T14:57:08.858908+0000","last_peered":"2024-01-16T14:57:08.858908+0000","last_clean":"2024-01-16T14:57:08.858908+0000","last_became_active":"2024-01-16T14:57:07.377843+0000","last_became_peered":"2024-01-16T14:57:07.377843+0000","last_unstale":"2024-01-16T14:57:08.858908+0000","last_undegraded":"2024-01-16T14:57:08.858908+0000","last_fullsized":"2024-01-16T14:57:08.858908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360527+0000","last_change":"2024-01-16T14:57:06.559586+0000","last_active":"2024-01-16T14:57:08.360527+0000","last_peered":"2024-01-16T14:57:08.360527+0000","last_clean":"2024-01-16T14:57:08.360527+0000","last_became_active":"2024-01-16T14:57:06.558281+0000","last_became_peered":"2024-01-16T14:57:06.558281+0000","last_unstale":"2024-01-16T14:57:08.360527+0000","last_undegraded":"2024-01-16T14:57:08.360527+0000","last_fullsized":"2024-01-16T14:57:08.360527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856775+0000","last_change":"2024-01-16T14:57:07.375900+0000","last_active":"2024-01-16T14:57:08.856775+0000","last_peered":"2024-01-16T14:57:08.856775+0000","last_clean":"2024-01-16T14:57:08.856775+0000","last_became_active":"2024-01-16T14:57:07.375625+0000","last_became_peered":"2024-01-16T14:57:07.375625+0000","last_unstale":"2024-01-16T14:57:08.856775+0000","last_undegraded":"2024-01-16T14:57:08.856775+0000","last_fullsized":"2024-01-16T14:57:08.856775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358102+0000","last_change":"2024-01-16T14:57:07.391481+0000","last_active":"2024-01-16T14:57:08.358102+0000","last_peered":"2024-01-16T14:57:08.358102+0000","last_clean":"2024-01-16T14:57:08.358102+0000","last_became_active":"2024-01-16T14:57:07.391111+0000","last_became_peered":"2024-01-16T14:57:07.391111+0000","last_unstale":"2024-01-16T14:57:08.358102+0000","last_undegraded":"2024-01-16T14:57:08.358102+0000","last_fullsized":"2024-01-16T14:57:08.358102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360198+0000","last_change":"2024-01-16T14:57:06.553585+0000","last_active":"2024-01-16T14:57:08.360198+0000","last_peered":"2024-01-16T14:57:08.360198+0000","last_clean":"2024-01-16T14:57:08.360198+0000","last_became_active":"2024-01-16T14:57:06.553259+0000","last_became_peered":"2024-01-16T14:57:06.553259+0000","last_unstale":"2024-01-16T14:57:08.360198+0000","last_undegraded":"2024-01-16T14:57:08.360198+0000","last_fullsized":"2024-01-16T14:57:08.360198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357259+0000","last_change":"2024-01-16T14:57:07.394393+0000","last_active":"2024-01-16T14:57:08.357259+0000","last_peered":"2024-01-16T14:57:08.357259+0000","last_clean":"2024-01-16T14:57:08.357259+0000","last_became_active":"2024-01-16T14:57:07.394135+0000","last_became_peered":"2024-01-16T14:57:07.394135+0000","last_unstale":"2024-01-16T14:57:08.357259+0000","last_undegraded":"2024-01-16T14:57:08.357259+0000","last_fullsized":"2024-01-16T14:57:08.357259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360937+0000","last_change":"2024-01-16T14:57:06.554017+0000","last_active":"2024-01-16T14:57:08.360937+0000","last_peered":"2024-01-16T14:57:08.360937+0000","last_clean":"2024-01-16T14:57:08.360937+0000","last_became_active":"2024-01-16T14:57:06.553376+0000","last_became_peered":"2024-01-16T14:57:06.553376+0000","last_unstale":"2024-01-16T14:57:08.360937+0000","last_undegraded":"2024-01-16T14:57:08.360937+0000","last_fullsized":"2024-01-16T14:57:08.360937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:35.199600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411544+0000","last_change":"2024-01-16T14:57:07.395607+0000","last_active":"2024-01-16T14:57:08.411544+0000","last_peered":"2024-01-16T14:57:08.411544+0000","last_clean":"2024-01-16T14:57:08.411544+0000","last_became_active":"2024-01-16T14:57:07.395288+0000","last_became_peered":"2024-01-16T14:57:07.395288+0000","last_unstale":"2024-01-16T14:57:08.411544+0000","last_undegraded":"2024-01-16T14:57:08.411544+0000","last_fullsized":"2024-01-16T14:57:08.411544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859322+0000","last_change":"2024-01-16T14:57:06.549558+0000","last_active":"2024-01-16T14:57:08.859322+0000","last_peered":"2024-01-16T14:57:08.859322+0000","last_clean":"2024-01-16T14:57:08.859322+0000","last_became_active":"2024-01-16T14:57:06.545986+0000","last_became_peered":"2024-01-16T14:57:06.545986+0000","last_unstale":"2024-01-16T14:57:08.859322+0000","last_undegraded":"2024-01-16T14:57:08.859322+0000","last_fullsized":"2024-01-16T14:57:08.859322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859106+0000","last_change":"2024-01-16T14:57:07.384928+0000","last_active":"2024-01-16T14:57:08.859106+0000","last_peered":"2024-01-16T14:57:08.859106+0000","last_clean":"2024-01-16T14:57:08.859106+0000","last_became_active":"2024-01-16T14:57:07.384613+0000","last_became_peered":"2024-01-16T14:57:07.384613+0000","last_unstale":"2024-01-16T14:57:08.859106+0000","last_undegraded":"2024-01-16T14:57:08.859106+0000","last_fullsized":"2024-01-16T14:57:08.859106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:19.171826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411722+0000","last_change":"2024-01-16T14:57:06.552465+0000","last_active":"2024-01-16T14:57:08.411722+0000","last_peered":"2024-01-16T14:57:08.411722+0000","last_clean":"2024-01-16T14:57:08.411722+0000","last_became_active":"2024-01-16T14:57:06.552103+0000","last_became_peered":"2024-01-16T14:57:06.552103+0000","last_unstale":"2024-01-16T14:57:08.411722+0000","last_undegraded":"2024-01-16T14:57:08.411722+0000","last_fullsized":"2024-01-16T14:57:08.411722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:35.728588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]}]},{"osd":6,"up_from":14,"seq":60129542147,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]}]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1539999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.919}]}]},{"osd":3,"up_from":14,"seq":60129542147,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.149}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2000000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.401}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:57:12.416 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T14:57:12.416 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T14:57:12.416 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T14:57:12.417 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T14:57:12.417 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T14:57:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.683+0000 7fb22a400640 1 Processor -- start 2024-01-16T14:57:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.683+0000 7fb22a400640 1 -- start start 2024-01-16T14:57:12.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.684+0000 7fb22a400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb22415da10 0x7fb224072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.684+0000 7fb22a400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb2240b2bd0 0x7fb2240b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.684+0000 7fb22a400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb224071190 con 0x7fb224150760 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.684+0000 7fb22a400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb224071470 con 0x7fb22415da10 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.684+0000 7fb22a400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb224071750 con 0x7fb2240b2bd0 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb222a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb22415da10 0x7fb224072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb222a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb22415da10 0x7fb224072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59282/0 (socket says 172.21.15.141:59282) 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb222a00640 1 -- 172.21.15.141:0/1128200057 learned_addr learned my addr 172.21.15.141:0/1128200057 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb223e00640 1 --2- 172.21.15.141:0/1128200057 >> v2:172.21.15.188:3300/0 conn(0x7fb2240b2bd0 0x7fb2240b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb222a00640 1 -- 172.21.15.141:0/1128200057 >> v2:172.21.15.188:3300/0 conn(0x7fb2240b2bd0 msgr2=0x7fb2240b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb222a00640 1 --2- 172.21.15.141:0/1128200057 >> v2:172.21.15.188:3300/0 conn(0x7fb2240b2bd0 0x7fb2240b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb222a00640 1 -- 172.21.15.141:0/1128200057 >> v1:172.21.15.188:6789/0 conn(0x7fb224150760 legacy=0x7fb22415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb222a00640 1 -- 172.21.15.141:0/1128200057 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224071a30 con 0x7fb22415da10 2024-01-16T14:57:12.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.685+0000 7fb223e00640 1 --2- 172.21.15.141:0/1128200057 >> v2:172.21.15.188:3300/0 conn(0x7fb2240b2bd0 0x7fb2240b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:12.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.686+0000 7fb222a00640 1 --2- 172.21.15.141:0/1128200057 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb22415da10 0x7fb224072230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2240cb740 tx=0x7fb20c02ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa4578e6c51f6eb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:12.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.686+0000 7fb222000640 1 -- 172.21.15.141:0/1128200057 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c030840 con 0x7fb22415da10 2024-01-16T14:57:12.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb222000640 1 -- 172.21.15.141:0/1128200057 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c0309a0 con 0x7fb22415da10 2024-01-16T14:57:12.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb222000640 1 -- 172.21.15.141:0/1128200057 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c030d30 con 0x7fb22415da10 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 -- 172.21.15.141:0/1128200057 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb22415da10 msgr2=0x7fb224072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 --2- 172.21.15.141:0/1128200057 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb22415da10 0x7fb224072230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2240cb740 tx=0x7fb20c02ca20 comp rx=0 tx=0).stop 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 -- 172.21.15.141:0/1128200057 shutdown_connections 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 --2- 172.21.15.141:0/1128200057 >> v2:172.21.15.188:3300/0 conn(0x7fb2240b2bd0 0x7fb2240b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 --2- 172.21.15.141:0/1128200057 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb22415da10 0x7fb224072230 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 -- 172.21.15.141:0/1128200057 >> 172.21.15.141:0/1128200057 conn(0x7fb22406a800 msgr2=0x7fb22406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 -- 172.21.15.141:0/1128200057 shutdown_connections 2024-01-16T14:57:12.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.687+0000 7fb22a400640 1 -- 172.21.15.141:0/1128200057 wait complete. 2024-01-16T14:57:12.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.689+0000 7fb22a400640 1 Processor -- start 2024-01-16T14:57:12.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.689+0000 7fb22a400640 1 -- start start 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb22a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2240b2bd0 0x7fb22415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb22a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb224150760 0x7fb224146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb22a400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb22415da10 0x7fb224158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb22a400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22414c880 con 0x7fb2240b2bd0 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb22a400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22414cb60 con 0x7fb22415da10 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb22a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22414ce40 con 0x7fb224150760 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb223400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2240b2bd0 0x7fb22415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb223400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2240b2bd0 0x7fb22415a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59286/0 (socket says 172.21.15.141:59286) 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.690+0000 7fb223400640 1 -- 172.21.15.141:0/126120015 learned_addr learned my addr 172.21.15.141:0/126120015 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb223400640 1 -- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb224150760 msgr2=0x7fb224146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb222a00640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb224150760 0x7fb224146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb223e00640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb22415da10 0x7fb224158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb223400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb224150760 0x7fb224146b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb223400640 1 -- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb22415da10 msgr2=0x7fb224158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb223400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb22415da10 0x7fb224158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb223400640 1 -- 172.21.15.141:0/126120015 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0094d0 con 0x7fb2240b2bd0 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb223400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2240b2bd0 0x7fb22415a330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb21400c430 tx=0x7fb21400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb21b400640 1 -- 172.21.15.141:0/126120015 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214011790 con 0x7fb2240b2bd0 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb21b400640 1 -- 172.21.15.141:0/126120015 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2140118f0 con 0x7fb2240b2bd0 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22414d120 con 0x7fb2240b2bd0 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb21b400640 1 -- 172.21.15.141:0/126120015 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214011ce0 con 0x7fb2240b2bd0 2024-01-16T14:57:12.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.691+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2240ceb80 con 0x7fb2240b2bd0 2024-01-16T14:57:12.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.692+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22414d310 con 0x7fb2240b2bd0 2024-01-16T14:57:12.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.693+0000 7fb21b400640 1 -- 172.21.15.141:0/126120015 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb21401c460 con 0x7fb2240b2bd0 2024-01-16T14:57:12.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.694+0000 7fb21b400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1fc06b2c0 0x7fb1fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:12.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.694+0000 7fb21b400640 1 -- 172.21.15.141:0/126120015 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb214089b40 con 0x7fb2240b2bd0 2024-01-16T14:57:12.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.695+0000 7fb222a00640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1fc06b2c0 0x7fb1fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:12.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.696+0000 7fb222a00640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1fc06b2c0 0x7fb1fc06d780 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb20c037000 tx=0x7fb20c03a040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:12.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.699+0000 7fb21b400640 1 -- 172.21.15.141:0/126120015 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2140549a0 con 0x7fb2240b2bd0 2024-01-16T14:57:12.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.934+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb22414c9c0 con 0x7fb2240b2bd0 2024-01-16T14:57:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.943+0000 7fb21b400640 1 -- 172.21.15.141:0/126120015 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171563 (secure 0 0 0) 0x7fb22414c9c0 con 0x7fb2240b2bd0 2024-01-16T14:57:12.929 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.947+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1fc06b2c0 msgr2=0x7fb1fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.947+0000 7fb22a400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1fc06b2c0 0x7fb1fc06d780 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb20c037000 tx=0x7fb20c03a040 comp rx=0 tx=0).stop 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.947+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2240b2bd0 msgr2=0x7fb22415a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.947+0000 7fb22a400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2240b2bd0 0x7fb22415a330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb21400c430 tx=0x7fb21400c8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 shutdown_connections 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1fc06b2c0 0x7fb1fc06d780 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb22415da10 0x7fb224158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb224150760 0x7fb224146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 --2- 172.21.15.141:0/126120015 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2240b2bd0 0x7fb22415a330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 >> 172.21.15.141:0/126120015 conn(0x7fb22406a800 msgr2=0x7fb224147bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 shutdown_connections 2024-01-16T14:57:12.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:12.948+0000 7fb22a400640 1 -- 172.21.15.141:0/126120015 wait complete. 2024-01-16T14:57:12.947 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":5,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":4065338852},{"type":"v1","addr":"172.21.15.188:6833","nonce":4065338852}]},"active_addr":"172.21.15.188:6833/4065338852","active_change":"2024-01-16T14:56:52.419166+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":3929481115}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":657250628}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":1406768844}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":994962660}]}]} 2024-01-16T14:57:12.951 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T14:57:12.951 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T14:57:12.951 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:13.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.174+0000 7f2d24000640 1 Processor -- start 2024-01-16T14:57:13.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.175+0000 7f2d24000640 1 -- start start 2024-01-16T14:57:13.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d24000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2d1c0ae5a0 0x7f2d1c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d24000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2d1c0cf290 0x7f2d1c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d24000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1c070c40 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d24000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c071450 con 0x7f2d1c0ae5a0 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d24000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c0cfc60 con 0x7f2d1c0cf290 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d21600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2d1c0ae5a0 0x7f2d1c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d20c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2d1c0b3ae0 0x7f2d1c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39978/0 (socket says 172.21.15.141:39978) 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.176+0000 7f2d20c00640 1 -- 172.21.15.141:0/3660548966 learned_addr learned my addr 172.21.15.141:0/3660548966 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.177+0000 7f2d22000640 1 --2- 172.21.15.141:0/3660548966 >> v2:172.21.15.188:3300/0 conn(0x7f2d1c0cf290 0x7f2d1c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.177+0000 7f2d21600640 1 -- 172.21.15.141:0/3660548966 >> v2:172.21.15.188:3300/0 conn(0x7f2d1c0cf290 msgr2=0x7f2d1c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.177+0000 7f2d21600640 1 --2- 172.21.15.141:0/3660548966 >> v2:172.21.15.188:3300/0 conn(0x7f2d1c0cf290 0x7f2d1c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.177+0000 7f2d21600640 1 -- 172.21.15.141:0/3660548966 >> v1:172.21.15.141:6789/0 conn(0x7f2d1c0b3ae0 legacy=0x7f2d1c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.177+0000 7f2d21600640 1 -- 172.21.15.141:0/3660548966 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c06c470 con 0x7f2d1c0ae5a0 2024-01-16T14:57:13.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.178+0000 7f2d21600640 1 --2- 172.21.15.141:0/3660548966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0ae5a0 0x7f2d1c15e430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0b3f70 tx=0x7f2d0c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738a0a8e894fb773 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.178+0000 7f2d1be00640 1 -- 172.21.15.141:0/3660548966 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c030610 con 0x7f2d1c0ae5a0 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.178+0000 7f2d1be00640 1 -- 172.21.15.141:0/3660548966 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0c030770 con 0x7f2d1c0ae5a0 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.178+0000 7f2d1be00640 1 -- 172.21.15.141:0/3660548966 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c030b00 con 0x7f2d1c0ae5a0 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 -- 172.21.15.141:0/3660548966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0ae5a0 msgr2=0x7f2d1c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 --2- 172.21.15.141:0/3660548966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0ae5a0 0x7f2d1c15e430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0b3f70 tx=0x7f2d0c02b960 comp rx=0 tx=0).stop 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 -- 172.21.15.141:0/3660548966 shutdown_connections 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 --2- 172.21.15.141:0/3660548966 >> v2:172.21.15.188:3300/0 conn(0x7f2d1c0cf290 0x7f2d1c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 --2- 172.21.15.141:0/3660548966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0ae5a0 0x7f2d1c15e430 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 -- 172.21.15.141:0/3660548966 >> 172.21.15.141:0/3660548966 conn(0x7f2d1c06a800 msgr2=0x7f2d1c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 -- 172.21.15.141:0/3660548966 shutdown_connections 2024-01-16T14:57:13.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.179+0000 7f2d24000640 1 -- 172.21.15.141:0/3660548966 wait complete. 2024-01-16T14:57:13.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.181+0000 7f2d24000640 1 Processor -- start 2024-01-16T14:57:13.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.181+0000 7f2d24000640 1 -- start start 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d24000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2d1c0ae5a0 0x7f2d1c148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d24000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0b3ae0 0x7f2d1c15c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d24000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c0cf290 0x7f2d1c15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d24000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c14e180 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d24000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c0aecc0 con 0x7f2d1c0cf290 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d24000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c14ec70 con 0x7f2d1c0ae5a0 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0b3ae0 0x7f2d1c15c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0b3ae0 0x7f2d1c15c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48262/0 (socket says 172.21.15.141:48262) 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 -- 172.21.15.141:0/3812729612 learned_addr learned my addr 172.21.15.141:0/3812729612 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 -- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2d1c0ae5a0 msgr2=0x7f2d1c148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2d1c0ae5a0 0x7f2d1c148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 -- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c0cf290 msgr2=0x7f2d1c15cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c0cf290 0x7f2d1c15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 -- 172.21.15.141:0/3812729612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c009400 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.182+0000 7f2d20c00640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0b3ae0 0x7f2d1c15c980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400c4d0 tx=0x7f2d0400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:13.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.183+0000 7f2d19600640 1 -- 172.21.15.141:0/3812729612 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d04005ce0 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.183+0000 7f2d19600640 1 -- 172.21.15.141:0/3812729612 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d04005e40 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.183+0000 7f2d19600640 1 -- 172.21.15.141:0/3812729612 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d04007270 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.183+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1c14e3d0 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.183+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1c14e990 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.184+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c0aea90 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.185+0000 7f2d19600640 1 -- 172.21.15.141:0/3812729612 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2d040073d0 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.186+0000 7f2d19600640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cf406b2c0 0x7f2cf406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.186+0000 7f2d19600640 1 -- 172.21.15.141:0/3812729612 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d04088fc0 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.186+0000 7f2d21600640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cf406b2c0 0x7f2cf406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.187+0000 7f2d21600640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cf406b2c0 0x7f2cf406d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c036000 tx=0x7f2d0c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:13.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.191+0000 7f2d19600640 1 -- 172.21.15.141:0/3812729612 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d040bbe90 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.368+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d1c0675d0 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.370+0000 7f2d19600640 1 -- 172.21.15.141:0/3812729612 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f2d04017020 con 0x7f2d1c0b3ae0 2024-01-16T14:57:13.353 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:13.353 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:13.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cf406b2c0 msgr2=0x7f2cf406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cf406b2c0 0x7f2cf406d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c036000 tx=0x7f2d0c03b040 comp rx=0 tx=0).stop 2024-01-16T14:57:13.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0b3ae0 msgr2=0x7f2d1c15c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0b3ae0 0x7f2d1c15c980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400c4d0 tx=0x7f2d0400c990 comp rx=0 tx=0).stop 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 shutdown_connections 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cf406b2c0 0x7f2cf406d780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c0cf290 0x7f2d1c15cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d1c0b3ae0 0x7f2d1c15c980 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 --2- 172.21.15.141:0/3812729612 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2d1c0ae5a0 0x7f2d1c148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.372+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 >> 172.21.15.141:0/3812729612 conn(0x7f2d1c06a800 msgr2=0x7f2d1c0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.373+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 shutdown_connections 2024-01-16T14:57:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.373+0000 7f2d24000640 1 -- 172.21.15.141:0/3812729612 wait complete. 2024-01-16T14:57:13.373 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T14:57:13.373 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.592+0000 7f25e4200640 1 Processor -- start 2024-01-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.592+0000 7f25e4200640 1 -- start start 2024-01-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.593+0000 7f25e4200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f25dc13df80 0x7f25dc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e4200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f25dc0cf9a0 0x7f25dc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e4200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc06c470 con 0x7f25dc13e360 2024-01-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e4200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc06c750 con 0x7f25dc0cf9a0 2024-01-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e4200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc06ca30 con 0x7f25dc13df80 2024-01-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e2200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f25dc0cf9a0 0x7f25dc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e2200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f25dc0cf9a0 0x7f25dc0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48264/0 (socket says 172.21.15.141:48264) 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e2200640 1 -- 172.21.15.141:0/899107446 learned_addr learned my addr 172.21.15.141:0/899107446 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e0e00640 1 --1- 172.21.15.141:0/899107446 >> v1:172.21.15.141:6789/0 conn(0x7f25dc13e360 0x7f25dc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39990/0 (socket says 172.21.15.141:39990) 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e1800640 1 --2- 172.21.15.141:0/899107446 >> v2:172.21.15.188:3300/0 conn(0x7f25dc13df80 0x7f25dc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e2200640 1 -- 172.21.15.141:0/899107446 >> v2:172.21.15.188:3300/0 conn(0x7f25dc13df80 msgr2=0x7f25dc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e2200640 1 --2- 172.21.15.141:0/899107446 >> v2:172.21.15.188:3300/0 conn(0x7f25dc13df80 0x7f25dc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e2200640 1 -- 172.21.15.141:0/899107446 >> v1:172.21.15.141:6789/0 conn(0x7f25dc13e360 legacy=0x7f25dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.594+0000 7f25e2200640 1 -- 172.21.15.141:0/899107446 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc06cd10 con 0x7f25dc0cf9a0 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.595+0000 7f25e2200640 1 --2- 172.21.15.141:0/899107446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc0cf9a0 0x7f25dc0cd840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25dc072770 tx=0x7f25d402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dad6d6bf320975f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25dbe00640 1 -- 172.21.15.141:0/899107446 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d402cb20 con 0x7f25dc0cf9a0 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25dbe00640 1 -- 172.21.15.141:0/899107446 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d402cc80 con 0x7f25dc0cf9a0 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25dbe00640 1 -- 172.21.15.141:0/899107446 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d40343f0 con 0x7f25dc0cf9a0 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25e4200640 1 -- 172.21.15.141:0/899107446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc0cf9a0 msgr2=0x7f25dc0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25e4200640 1 --2- 172.21.15.141:0/899107446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc0cf9a0 0x7f25dc0cd840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25dc072770 tx=0x7f25d402bf80 comp rx=0 tx=0).stop 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25e4200640 1 -- 172.21.15.141:0/899107446 shutdown_connections 2024-01-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25e4200640 1 --2- 172.21.15.141:0/899107446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc0cf9a0 0x7f25dc0cd840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25e4200640 1 --2- 172.21.15.141:0/899107446 >> v2:172.21.15.188:3300/0 conn(0x7f25dc13df80 0x7f25dc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.596+0000 7f25e4200640 1 -- 172.21.15.141:0/899107446 >> 172.21.15.141:0/899107446 conn(0x7f25dc06a800 msgr2=0x7f25dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.597+0000 7f25e4200640 1 -- 172.21.15.141:0/899107446 shutdown_connections 2024-01-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.597+0000 7f25e4200640 1 -- 172.21.15.141:0/899107446 wait complete. 2024-01-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.598+0000 7f25e4200640 1 Processor -- start 2024-01-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.598+0000 7f25e4200640 1 -- start start 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.599+0000 7f25e4200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25dc0cf9a0 0x7f25dc153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.599+0000 7f25e4200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc13df80 0x7f25dc14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.599+0000 7f25e4200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25dc13e360 0x7f25dc13d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.599+0000 7f25e4200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc151f20 con 0x7f25dc13df80 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.599+0000 7f25e4200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc151b80 con 0x7f25dc0cf9a0 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.599+0000 7f25e4200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc14cd90 con 0x7f25dc13e360 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc13df80 0x7f25dc14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc13df80 0x7f25dc14c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48276/0 (socket says 172.21.15.141:48276) 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 -- 172.21.15.141:0/1861288974 learned_addr learned my addr 172.21.15.141:0/1861288974 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e1800640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25dc0cf9a0 0x7f25dc153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e2200640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25dc13e360 0x7f25dc13d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 -- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25dc13e360 msgr2=0x7f25dc13d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25dc13e360 0x7f25dc13d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 -- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25dc0cf9a0 msgr2=0x7f25dc153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25dc0cf9a0 0x7f25dc153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 -- 172.21.15.141:0/1861288974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4009b30 con 0x7f25dc13df80 2024-01-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25e0e00640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc13df80 0x7f25dc14c540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f25c400c450 tx=0x7f25c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.600+0000 7f25d9600640 1 -- 172.21.15.141:0/1861288974 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4007480 con 0x7f25dc13df80 2024-01-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.601+0000 7f25d9600640 1 -- 172.21.15.141:0/1861288974 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c40075e0 con 0x7f25dc13df80 2024-01-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.601+0000 7f25d9600640 1 -- 172.21.15.141:0/1861288974 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4004a00 con 0x7f25dc13df80 2024-01-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.601+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc14d070 con 0x7f25dc13df80 2024-01-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.601+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc13dc70 con 0x7f25dc13df80 2024-01-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.602+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc152060 con 0x7f25dc13df80 2024-01-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.603+0000 7f25d9600640 1 -- 172.21.15.141:0/1861288974 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25c4004b60 con 0x7f25dc13df80 2024-01-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.604+0000 7f25d9600640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f25b406b2c0 0x7f25b406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.604+0000 7f25d9600640 1 -- 172.21.15.141:0/1861288974 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25c4088cd0 con 0x7f25dc13df80 2024-01-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.604+0000 7f25e1800640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f25b406b2c0 0x7f25b406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.605+0000 7f25e1800640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f25b406b2c0 0x7f25b406d780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25cc006fd0 tx=0x7f25cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.608+0000 7f25d9600640 1 -- 172.21.15.141:0/1861288974 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25dc152060 con 0x7f25dc13df80 2024-01-16T14:57:13.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.797+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25dc152060 con 0x7f25dc13df80 2024-01-16T14:57:13.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.801+0000 7f25d9600640 1 -- 172.21.15.141:0/1861288974 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f25dc152060 con 0x7f25dc13df80 2024-01-16T14:57:13.784 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:13.784 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:13.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f25b406b2c0 msgr2=0x7f25b406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f25b406b2c0 0x7f25b406d780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25cc006fd0 tx=0x7f25cc008040 comp rx=0 tx=0).stop 2024-01-16T14:57:13.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc13df80 msgr2=0x7f25dc14c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:13.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc13df80 0x7f25dc14c540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f25c400c450 tx=0x7f25c400c910 comp rx=0 tx=0).stop 2024-01-16T14:57:13.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 shutdown_connections 2024-01-16T14:57:13.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f25b406b2c0 0x7f25b406d780 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25dc13e360 0x7f25dc13d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25dc13df80 0x7f25dc14c540 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 --2- 172.21.15.141:0/1861288974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25dc0cf9a0 0x7f25dc153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:13.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 >> 172.21.15.141:0/1861288974 conn(0x7f25dc06a800 msgr2=0x7f25dc15ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:13.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 shutdown_connections 2024-01-16T14:57:13.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:13.803+0000 7f25e4200640 1 -- 172.21.15.141:0/1861288974 wait complete. 2024-01-16T14:57:13.805 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T14:57:13.806 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T14:57:13.806 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T14:57:13.806 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-16T14:57:13.807 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-16T14:57:13.807 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-16T14:57:13.807 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-16T14:57:13.808 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-16T14:57:14.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.113+0000 7f80dba00640 1 Processor -- start 2024-01-16T14:57:14.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.113+0000 7f80dba00640 1 -- start start 2024-01-16T14:57:14.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.115+0000 7f80dba00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80cc0e3120 0x7f80cc0e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.115+0000 7f80dba00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f80cc03c060 0x7f80cc058010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.115+0000 7f80dba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc053c70 0x7f80cc054100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.115+0000 7f80dba00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80cc058550 con 0x7f80cc0e3120 2024-01-16T14:57:14.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.115+0000 7f80dba00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80cc054640 con 0x7f80cc03c060 2024-01-16T14:57:14.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.115+0000 7f80dba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc0587d0 con 0x7f80cc053c70 2024-01-16T14:57:14.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.116+0000 7f80d9a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc053c70 0x7f80cc054100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.116+0000 7f80d9000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80cc0e3120 0x7f80cc0e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.116+0000 7f80d9000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80cc0e3120 0x7f80cc0e7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48280/0 (socket says 172.21.15.141:48280) 2024-01-16T14:57:14.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.116+0000 7f80d9000640 1 -- 172.21.15.141:0/1206873517 learned_addr learned my addr 172.21.15.141:0/1206873517 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.117+0000 7f80d9000640 1 -- 172.21.15.141:0/1206873517 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc053c70 msgr2=0x7f80cc054100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.117+0000 7f80d9000640 1 --2- 172.21.15.141:0/1206873517 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc053c70 0x7f80cc054100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.117+0000 7f80d9000640 1 -- 172.21.15.141:0/1206873517 >> v2:172.21.15.188:3300/0 conn(0x7f80cc03c060 msgr2=0x7f80cc058010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:57:14.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.117+0000 7f80d9000640 1 --2- 172.21.15.141:0/1206873517 >> v2:172.21.15.188:3300/0 conn(0x7f80cc03c060 0x7f80cc058010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.117+0000 7f80d9000640 1 -- 172.21.15.141:0/1206873517 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc058ab0 con 0x7f80cc0e3120 2024-01-16T14:57:14.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.119+0000 7f80d9000640 1 --2- 172.21.15.141:0/1206873517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 0x7f80cc0e7740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0536c0 tx=0x7f80bc02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c977cc108a10cd6a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.120+0000 7f80d3400640 1 -- 172.21.15.141:0/1206873517 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc030800 con 0x7f80cc0e3120 2024-01-16T14:57:14.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.120+0000 7f80d3400640 1 -- 172.21.15.141:0/1206873517 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc030960 con 0x7f80cc0e3120 2024-01-16T14:57:14.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.120+0000 7f80d3400640 1 -- 172.21.15.141:0/1206873517 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc030cf0 con 0x7f80cc0e3120 2024-01-16T14:57:14.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.120+0000 7f2d05800640 1 Processor -- start 2024-01-16T14:57:14.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.120+0000 7f585aa00640 1 Processor -- start 2024-01-16T14:57:14.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.121+0000 7f975e800640 1 Processor -- start 2024-01-16T14:57:14.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.121+0000 7f2d05800640 1 -- start start 2024-01-16T14:57:14.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.121+0000 7f80dba00640 1 -- 172.21.15.141:0/1206873517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 msgr2=0x7f80cc0e7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.121+0000 7f80dba00640 1 --2- 172.21.15.141:0/1206873517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 0x7f80cc0e7740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0536c0 tx=0x7f80bc02b940 comp rx=0 tx=0).stop 2024-01-16T14:57:14.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.121+0000 7f975e800640 1 -- start start 2024-01-16T14:57:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.122+0000 7f80dba00640 1 -- 172.21.15.141:0/1206873517 shutdown_connections 2024-01-16T14:57:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f80dba00640 1 --2- 172.21.15.141:0/1206873517 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc053c70 0x7f80cc054100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f80dba00640 1 --2- 172.21.15.141:0/1206873517 >> v2:172.21.15.188:3300/0 conn(0x7f80cc03c060 0x7f80cc058010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f80dba00640 1 --2- 172.21.15.141:0/1206873517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 0x7f80cc0e7740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f80dba00640 1 -- 172.21.15.141:0/1206873517 >> 172.21.15.141:0/1206873517 conn(0x7f80cc058ea0 msgr2=0x7f80cc059280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.122+0000 7f2d05800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 0x7f2cf80e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f2d05800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2cf803c070 0x7f2cf8058020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f2d05800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf8058560 con 0x7f2cf8053c80 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f2d05800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf8007b40 con 0x7f2cf803c070 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f2d05800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf80587e0 con 0x7f2cf80e3130 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.121+0000 7f585aa00640 1 -- start start 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f585aa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5854160a20 0x7f58540cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f585aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5854160e00 0x7f58540cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f585aa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58540d1e00 con 0x7f58540b3b20 2024-01-16T14:57:14.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f585aa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58540d3c40 con 0x7f5854160e00 2024-01-16T14:57:14.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f585aa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58540d3e50 con 0x7f5854160a20 2024-01-16T14:57:14.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.123+0000 7f2cffe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 0x7f2cf80e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f975e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f975814adf0 0x7f97580b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f80dba00640 1 -- 172.21.15.141:0/1206873517 shutdown_connections 2024-01-16T14:57:14.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f2cff400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2cf803c070 0x7f2cf8058020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f975e800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9758071a70 0x7f975806c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f975e800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9758070160 con 0x7f97580b3710 2024-01-16T14:57:14.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f975e800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97580703b0 con 0x7f9758071a70 2024-01-16T14:57:14.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f975e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9758070690 con 0x7f975814adf0 2024-01-16T14:57:14.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f2cff400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2cf803c070 0x7f2cf8058020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36322/0 (socket says 172.21.15.141:36322) 2024-01-16T14:57:14.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f80dba00640 1 -- 172.21.15.141:0/1206873517 wait complete. 2024-01-16T14:57:14.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f5859600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5854160a20 0x7f58540cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f5859600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5854160a20 0x7f58540cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:60928/0 (socket says 172.21.15.141:60928) 2024-01-16T14:57:14.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f5859600640 1 -- 172.21.15.141:0/236429526 learned_addr learned my addr 172.21.15.141:0/236429526 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.124+0000 7f2cff400640 1 -- 172.21.15.141:0/1190067996 learned_addr learned my addr 172.21.15.141:0/1190067996 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f5858c00640 1 --2- 172.21.15.141:0/236429526 >> v2:172.21.15.141:3300/0 conn(0x7f5854160e00 0x7f58540cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f9757400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f975814adf0 0x7f97580b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f9756a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f97580b3710 0x7f9758071360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40016/0 (socket says 172.21.15.141:40016) 2024-01-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f9756a00640 1 -- 172.21.15.141:0/2443602785 learned_addr learned my addr 172.21.15.141:0/2443602785 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:14.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f2cff400640 1 -- 172.21.15.141:0/1190067996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 msgr2=0x7f2cf80e7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f2cff400640 1 --2- 172.21.15.141:0/1190067996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 0x7f2cf80e7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f2cff400640 1 -- 172.21.15.141:0/1190067996 >> v1:172.21.15.188:6789/0 conn(0x7f2cf8053c80 legacy=0x7f2cf80540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f2cff400640 1 -- 172.21.15.141:0/1190067996 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8058ac0 con 0x7f2cf803c070 2024-01-16T14:57:14.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.125+0000 7f9757e00640 1 --2- 172.21.15.141:0/2443602785 >> v2:172.21.15.141:3300/0 conn(0x7f9758071a70 0x7f975806c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f9757400640 1 -- 172.21.15.141:0/2443602785 >> v2:172.21.15.141:3300/0 conn(0x7f9758071a70 msgr2=0x7f975806c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f5859600640 1 -- 172.21.15.141:0/236429526 >> v2:172.21.15.141:3300/0 conn(0x7f5854160e00 msgr2=0x7f58540cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f9757400640 1 --2- 172.21.15.141:0/2443602785 >> v2:172.21.15.141:3300/0 conn(0x7f9758071a70 0x7f975806c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f9757400640 1 -- 172.21.15.141:0/2443602785 >> v1:172.21.15.141:6789/0 conn(0x7f97580b3710 legacy=0x7f9758071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f9757400640 1 -- 172.21.15.141:0/2443602785 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9758070970 con 0x7f975814adf0 2024-01-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f5859600640 1 --2- 172.21.15.141:0/236429526 >> v2:172.21.15.141:3300/0 conn(0x7f5854160e00 0x7f58540cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f5859600640 1 -- 172.21.15.141:0/236429526 >> v1:172.21.15.141:6789/0 conn(0x7f58540b3b20 legacy=0x7f58540b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.126+0000 7f5859600640 1 -- 172.21.15.141:0/236429526 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58540d1a90 con 0x7f5854160a20 2024-01-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.127+0000 7f80dba00640 1 Processor -- start 2024-01-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.127+0000 7f80dba00640 1 -- start start 2024-01-16T14:57:14.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.127+0000 7f2cff400640 1 --2- 172.21.15.141:0/1190067996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf8058020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009870 tx=0x7f2ce802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=340d247945777050 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.127+0000 7f9757e00640 1 --2- 172.21.15.141:0/2443602785 >> v2:172.21.15.141:3300/0 conn(0x7f9758071a70 0x7f975806c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:14.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.128+0000 7f5859600640 1 --2- 172.21.15.141:0/236429526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5854160a20 0x7f58540cf850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f583c009870 tx=0x7f583c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeb454f947a0039c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.128+0000 7f2cfea00640 1 -- 172.21.15.141:0/1190067996 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8030840 con 0x7f2cf803c070 2024-01-16T14:57:14.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.128+0000 7f2cfea00640 1 -- 172.21.15.141:0/1190067996 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce80309a0 con 0x7f2cf803c070 2024-01-16T14:57:14.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.128+0000 7f9757400640 1 --2- 172.21.15.141:0/2443602785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f975814adf0 0x7f97580b31d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97400097a0 tx=0x7f974002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faf4645c1a042bcb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.128+0000 7f2cfea00640 1 -- 172.21.15.141:0/1190067996 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8030d30 con 0x7f2cf803c070 2024-01-16T14:57:14.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.128+0000 7fa275a00640 1 Processor -- start 2024-01-16T14:57:14.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.128+0000 7f5853e00640 1 -- 172.21.15.141:0/236429526 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c030840 con 0x7f5854160a20 2024-01-16T14:57:14.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f5853e00640 1 -- 172.21.15.141:0/236429526 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c0309a0 con 0x7f5854160a20 2024-01-16T14:57:14.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f5853e00640 1 -- 172.21.15.141:0/236429526 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c030d30 con 0x7f5854160a20 2024-01-16T14:57:14.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f9756000640 1 -- 172.21.15.141:0/2443602785 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740031830 con 0x7f975814adf0 2024-01-16T14:57:14.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f9756000640 1 -- 172.21.15.141:0/2443602785 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9740031990 con 0x7f975814adf0 2024-01-16T14:57:14.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f9756000640 1 -- 172.21.15.141:0/2443602785 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740031d60 con 0x7f975814adf0 2024-01-16T14:57:14.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f80dba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc03c060 0x7f80cc0e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f2d05800640 1 -- 172.21.15.141:0/1190067996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 msgr2=0x7f2cf8058020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f2d05800640 1 --2- 172.21.15.141:0/1190067996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf8058020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009870 tx=0x7f2ce802b980 comp rx=0 tx=0).stop 2024-01-16T14:57:14.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f80dba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80cc053c70 0x7f80cc1796a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f80dba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 0x7f80cc0e5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f80dba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc0e5b00 con 0x7f80cc0e3120 2024-01-16T14:57:14.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f80dba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc179be0 con 0x7f80cc053c70 2024-01-16T14:57:14.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f80dba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc0e3800 con 0x7f80cc03c060 2024-01-16T14:57:14.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f585aa00640 1 -- 172.21.15.141:0/236429526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5854160a20 msgr2=0x7f58540cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7f585aa00640 1 --2- 172.21.15.141:0/236429526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5854160a20 0x7f58540cf850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f583c009870 tx=0x7f583c02b980 comp rx=0 tx=0).stop 2024-01-16T14:57:14.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 -- 172.21.15.141:0/2443602785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f975814adf0 msgr2=0x7f97580b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 --2- 172.21.15.141:0/2443602785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f975814adf0 0x7f97580b31d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97400097a0 tx=0x7f974002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 -- 172.21.15.141:0/2443602785 shutdown_connections 2024-01-16T14:57:14.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f585aa00640 1 -- 172.21.15.141:0/236429526 shutdown_connections 2024-01-16T14:57:14.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f585aa00640 1 --2- 172.21.15.141:0/236429526 >> v2:172.21.15.141:3300/0 conn(0x7f5854160e00 0x7f58540cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f585aa00640 1 --2- 172.21.15.141:0/236429526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5854160a20 0x7f58540cf850 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f585aa00640 1 -- 172.21.15.141:0/236429526 >> 172.21.15.141:0/236429526 conn(0x7f58540d3120 msgr2=0x7f58540d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 --2- 172.21.15.141:0/2443602785 >> v2:172.21.15.141:3300/0 conn(0x7f9758071a70 0x7f975806c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 --2- 172.21.15.141:0/2443602785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f975814adf0 0x7f97580b31d0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97400097a0 tx=0x7f974002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 -- 172.21.15.141:0/2443602785 >> 172.21.15.141:0/2443602785 conn(0x7f975806a800 msgr2=0x7f975806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.129+0000 7fa275a00640 1 -- start start 2024-01-16T14:57:14.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 -- 172.21.15.141:0/2443602785 shutdown_connections 2024-01-16T14:57:14.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f975e800640 1 -- 172.21.15.141:0/2443602785 wait complete. 2024-01-16T14:57:14.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f585aa00640 1 -- 172.21.15.141:0/236429526 shutdown_connections 2024-01-16T14:57:14.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.130+0000 7f80d3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80cc053c70 0x7f80cc1796a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7f80d3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80cc053c70 0x7f80cc1796a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36336/0 (socket says 172.21.15.141:36336) 2024-01-16T14:57:14.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7f80d3e00640 1 -- 172.21.15.141:0/2373795557 learned_addr learned my addr 172.21.15.141:0/2373795557 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7f585aa00640 1 -- 172.21.15.141:0/236429526 wait complete. 2024-01-16T14:57:14.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa275a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa27014b2c0 0x7fa2700cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa275a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa270071070 0x7fa2700cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa275a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2700cfbf0 con 0x7fa2700cf7d0 2024-01-16T14:57:14.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa275a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27014b6a0 con 0x7fa270071070 2024-01-16T14:57:14.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa275a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa270071690 con 0x7fa27014b2c0 2024-01-16T14:57:14.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7f2d05800640 1 -- 172.21.15.141:0/1190067996 shutdown_connections 2024-01-16T14:57:14.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7f2d05800640 1 --2- 172.21.15.141:0/1190067996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf8058020 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7f2d05800640 1 --2- 172.21.15.141:0/1190067996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 0x7f2cf80e7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7f2d05800640 1 -- 172.21.15.141:0/1190067996 >> 172.21.15.141:0/1190067996 conn(0x7f2cf8058eb0 msgr2=0x7f2cf8059290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa26ea00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa27014b2c0 0x7fa2700cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa26ea00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa27014b2c0 0x7fa2700cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36338/0 (socket says 172.21.15.141:36338) 2024-01-16T14:57:14.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.131+0000 7fa26ea00640 1 -- 172.21.15.141:0/2930631015 learned_addr learned my addr 172.21.15.141:0/2930631015 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f80d9a00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 0x7f80cc0e5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f80d3e00640 1 -- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc03c060 msgr2=0x7f80cc0e4ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:57:14.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f80d3e00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80cc03c060 0x7f80cc0e4ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f80d3e00640 1 -- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 msgr2=0x7f80cc0e5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f2d05800640 1 -- 172.21.15.141:0/1190067996 shutdown_connections 2024-01-16T14:57:14.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7fa26f400640 1 --2- 172.21.15.141:0/2930631015 >> v2:172.21.15.141:3300/0 conn(0x7fa270071070 0x7fa2700cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7fa26ea00640 1 -- 172.21.15.141:0/2930631015 >> v2:172.21.15.141:3300/0 conn(0x7fa270071070 msgr2=0x7fa2700cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7fa26ea00640 1 --2- 172.21.15.141:0/2930631015 >> v2:172.21.15.141:3300/0 conn(0x7fa270071070 0x7fa2700cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7fa26ea00640 1 -- 172.21.15.141:0/2930631015 >> v1:172.21.15.188:6789/0 conn(0x7fa2700cf7d0 legacy=0x7fa270070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f80d3e00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 0x7f80cc0e5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f80d3e00640 1 -- 172.21.15.141:0/2373795557 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc0094d0 con 0x7f80cc053c70 2024-01-16T14:57:14.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.133+0000 7fa26ea00640 1 -- 172.21.15.141:0/2930631015 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa270071970 con 0x7fa27014b2c0 2024-01-16T14:57:14.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f80d9a00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80cc0e3120 0x7f80cc0e5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:14.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.133+0000 7f975e800640 1 Processor -- start 2024-01-16T14:57:14.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.132+0000 7f2d05800640 1 -- 172.21.15.141:0/1190067996 wait complete. 2024-01-16T14:57:14.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.133+0000 7f975e800640 1 -- start start 2024-01-16T14:57:14.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.133+0000 7f80d3e00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80cc053c70 0x7f80cc1796a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b450 tx=0x7f80c400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.133+0000 7f585aa00640 1 Processor -- start 2024-01-16T14:57:14.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.134+0000 7f585aa00640 1 -- start start 2024-01-16T14:57:14.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.134+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4011890 con 0x7f80cc053c70 2024-01-16T14:57:14.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.134+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c4011a10 con 0x7f80cc053c70 2024-01-16T14:57:14.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.134+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4011dc0 con 0x7f80cc053c70 2024-01-16T14:57:14.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.134+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc0e5540 con 0x7f80cc053c70 2024-01-16T14:57:14.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.134+0000 7fa26ea00640 1 --2- 172.21.15.141:0/2930631015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27014b2c0 0x7fa2700cf290 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa27006e650 tx=0x7fa26002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec267b217425808d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f2d05800640 1 Processor -- start 2024-01-16T14:57:14.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc0e3ac0 con 0x7f80cc053c70 2024-01-16T14:57:14.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f975e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9758071a70 0x7f9758145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f975e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97580b3710 0x7f975815d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f975e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f975815de80 0x7f9758147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f2d05800640 1 -- start start 2024-01-16T14:57:14.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f585aa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58540b3b20 0x7f585415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f585aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5854160a20 0x7f5854159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f585aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5854160e00 0x7f58540b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f585aa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585409cfa0 con 0x7f5854160e00 2024-01-16T14:57:14.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f585aa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58540b72e0 con 0x7f5854160a20 2024-01-16T14:57:14.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f585aa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585415abf0 con 0x7f58540b3b20 2024-01-16T14:57:14.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f975e800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97581f7d70 con 0x7f975815de80 2024-01-16T14:57:14.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f975e800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9758145a80 con 0x7f97580b3710 2024-01-16T14:57:14.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f975e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975815e3c0 con 0x7f9758071a70 2024-01-16T14:57:14.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.135+0000 7f9757400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9758071a70 0x7f9758145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9758071a70 0x7f9758145540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:60974/0 (socket says 172.21.15.141:60974) 2024-01-16T14:57:14.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757400640 1 -- 172.21.15.141:0/307033422 learned_addr learned my addr 172.21.15.141:0/307033422 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80a4000fc0 con 0x7f80cc053c70 2024-01-16T14:57:14.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7fa26e000640 1 -- 172.21.15.141:0/2930631015 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260030860 con 0x7fa27014b2c0 2024-01-16T14:57:14.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7fa26e000640 1 -- 172.21.15.141:0/2930631015 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2600309c0 con 0x7fa27014b2c0 2024-01-16T14:57:14.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7fa26e000640 1 -- 172.21.15.141:0/2930631015 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260030d50 con 0x7fa27014b2c0 2024-01-16T14:57:14.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f2d05800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf80e5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f361f200640 1 Processor -- start 2024-01-16T14:57:14.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f2d05800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cf8053c80 0x7f2cf803ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f2d05800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 0x7f2cf804c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f2d05800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf80e67c0 con 0x7f2cf8053c80 2024-01-16T14:57:14.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f2d05800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf80e4510 con 0x7f2cf803c070 2024-01-16T14:57:14.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f2d05800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf804f500 con 0x7f2cf80e3130 2024-01-16T14:57:14.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f1c7c800640 1 Processor -- start 2024-01-16T14:57:14.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f361f200640 1 -- start start 2024-01-16T14:57:14.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f1c7c800640 1 -- start start 2024-01-16T14:57:14.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f2cffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf80e5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757e00640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f975815de80 0x7f9758147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757400640 1 -- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97580b3710 msgr2=0x7f975815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757400640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97580b3710 0x7f975815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757400640 1 -- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f975815de80 msgr2=0x7f9758147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757400640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f975815de80 0x7f9758147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f9757400640 1 -- 172.21.15.141:0/307033422 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750009a70 con 0x7f9758071a70 2024-01-16T14:57:14.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f2cff400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cf8053c80 0x7f2cf803ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f2cffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf80e5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36364/0 (socket says 172.21.15.141:36364) 2024-01-16T14:57:14.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f2cffe00640 1 -- 172.21.15.141:0/4285355788 learned_addr learned my addr 172.21.15.141:0/4285355788 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f2d04e00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 0x7f2cf804c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f2cff400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cf8053c80 0x7f2cf803ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48346/0 (socket says 172.21.15.141:48346) 2024-01-16T14:57:14.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f9757e00640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f975815de80 0x7f9758147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:14.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f5859600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58540b3b20 0x7f585415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f5859600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58540b3b20 0x7f585415a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:60986/0 (socket says 172.21.15.141:60986) 2024-01-16T14:57:14.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.137+0000 7f5859600640 1 -- 172.21.15.141:0/3417021174 learned_addr learned my addr 172.21.15.141:0/3417021174 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.136+0000 7f585a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5854160e00 0x7f58540b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7f5858c00640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5854160a20 0x7f5854159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 -- 172.21.15.141:0/2930631015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27014b2c0 msgr2=0x7fa2700cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 --2- 172.21.15.141:0/2930631015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27014b2c0 0x7fa2700cf290 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa27006e650 tx=0x7fa26002b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 -- 172.21.15.141:0/2930631015 shutdown_connections 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 --2- 172.21.15.141:0/2930631015 >> v2:172.21.15.141:3300/0 conn(0x7fa270071070 0x7fa2700cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 --2- 172.21.15.141:0/2930631015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27014b2c0 0x7fa2700cf290 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 -- 172.21.15.141:0/2930631015 >> 172.21.15.141:0/2930631015 conn(0x7fa27006a800 msgr2=0x7fa27006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 -- 172.21.15.141:0/2930631015 shutdown_connections 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7fa275a00640 1 -- 172.21.15.141:0/2930631015 wait complete. 2024-01-16T14:57:14.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7f9757400640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9758071a70 0x7f9758145540 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f974002be60 tx=0x7f97400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7f2cffe00640 1 -- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 msgr2=0x7f2cf804c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7f2cffe00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cf80e3130 0x7f2cf804c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f2cffe00640 1 -- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cf8053c80 msgr2=0x7f2cf803ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f2cffe00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cf8053c80 0x7f2cf803ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7f585a000640 1 -- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58540b3b20 msgr2=0x7f585415a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.138+0000 7f585a000640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58540b3b20 0x7f585415a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f585a000640 1 -- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5854160a20 msgr2=0x7f5854159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f585a000640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5854160a20 0x7f5854159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f585a000640 1 -- 172.21.15.141:0/3417021174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0094d0 con 0x7f5854160e00 2024-01-16T14:57:14.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f2cffe00640 1 -- 172.21.15.141:0/4285355788 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80094d0 con 0x7f2cf803c070 2024-01-16T14:57:14.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f2cff400640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cf8053c80 0x7f2cf803ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:14.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f585a000640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5854160e00 0x7f58540b6720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f584c00c760 tx=0x7f584c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f361f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f362006c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f2cffe00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf80e5570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000d5f0 tx=0x7f2cf000d9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97400317d0 con 0x7f9758071a70 2024-01-16T14:57:14.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974003b9f0 con 0x7f9758071a70 2024-01-16T14:57:14.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974003bda0 con 0x7f9758071a70 2024-01-16T14:57:14.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f361f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f36200ce320 0x7f362006cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f361f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f36200b37d0 0x7f362015cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f1c7c800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1c74160f80 0x7f1c74161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f1c7c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740ce8f0 0x7f1c740ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f1c7c800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c741618a0 con 0x7f1c740cfbe0 2024-01-16T14:57:14.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f1c7c800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c740d0de0 con 0x7f1c74160f80 2024-01-16T14:57:14.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f1c7c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740d10c0 con 0x7f1c740ce8f0 2024-01-16T14:57:14.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c003b70 con 0x7f5854160e00 2024-01-16T14:57:14.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9740009400 con 0x7f9758071a70 2024-01-16T14:57:14.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97581f7720 con 0x7f9758071a70 2024-01-16T14:57:14.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f361f200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36200b2a80 con 0x7f36200ce320 2024-01-16T14:57:14.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f361f200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36200b2ea0 con 0x7f36200b37d0 2024-01-16T14:57:14.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f361f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36201596e0 con 0x7f3620159300 2024-01-16T14:57:14.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.139+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f80c401a400 con 0x7f80cc053c70 2024-01-16T14:57:14.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf00125f0 con 0x7f2cf803c070 2024-01-16T14:57:14.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f361de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f362006c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c003cd0 con 0x7f5854160e00 2024-01-16T14:57:14.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c00aa70 con 0x7f5854160e00 2024-01-16T14:57:14.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f80d0c00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ac06b390 0x7f80ac06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361e800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f36200b37d0 0x7f362015cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f362006c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:32780/0 (socket says 172.21.15.141:32780) 2024-01-16T14:57:14.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361de00640 1 -- 172.21.15.141:0/3458029385 learned_addr learned my addr 172.21.15.141:0/3458029385 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80c4089f30 con 0x7f80cc053c70 2024-01-16T14:57:14.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585415f8c0 con 0x7f5854160e00 2024-01-16T14:57:14.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0012750 con 0x7f2cf803c070 2024-01-16T14:57:14.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.140+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf80e47b0 con 0x7f2cf803c070 2024-01-16T14:57:14.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0012ae0 con 0x7f2cf803c070 2024-01-16T14:57:14.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf80e4d70 con 0x7f2cf803c070 2024-01-16T14:57:14.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7fa275a00640 1 Processor -- start 2024-01-16T14:57:14.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.142+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2cf80e3510 con 0x7f2cf803c070 2024-01-16T14:57:14.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.142+0000 7fa275a00640 1 -- start start 2024-01-16T14:57:14.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58540cb6a0 con 0x7f5854160e00 2024-01-16T14:57:14.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.143+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9758070f30 con 0x7f9758071a70 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.143+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5854067b60 con 0x7f5854160e00 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f1c79400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740ce8f0 0x7f1c740ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f1c79e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1c74160f80 0x7f1c74161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f1c79400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740ce8f0 0x7f1c740ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:32794/0 (socket says 172.21.15.141:32794) 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f1c79400640 1 -- 172.21.15.141:0/15489732 learned_addr learned my addr 172.21.15.141:0/15489732 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.142+0000 7f1c79400640 1 -- 172.21.15.141:0/15489732 >> v2:172.21.15.141:3300/0 conn(0x7f1c74160f80 msgr2=0x7f1c74161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.142+0000 7f1c79400640 1 --2- 172.21.15.141:0/15489732 >> v2:172.21.15.141:3300/0 conn(0x7f1c74160f80 0x7f1c74161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.142+0000 7f1c79400640 1 -- 172.21.15.141:0/15489732 >> v1:172.21.15.141:6789/0 conn(0x7f1c740cfbe0 legacy=0x7f1c740d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.142+0000 7f1c79400640 1 -- 172.21.15.141:0/15489732 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c740d13a0 con 0x7f1c740ce8f0 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f80d0c00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f80ac0735b0 0x7f80ac0759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 --> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80ac076080 con 0x7f80ac0735b0 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80c408c550 con 0x7f80cc053c70 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f80d9000640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ac06b390 0x7f80ac06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.142+0000 7f80d9a00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f80ac0735b0 0x7f80ac0759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.143+0000 7f80d9000640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ac06b390 0x7f80ac06d850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f80bc036000 tx=0x7f80bc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.143+0000 7f80d9a00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f80ac0735b0 0x7f80ac0759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.144+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2cf0012c40 con 0x7f2cf803c070 2024-01-16T14:57:14.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361de00640 1 -- 172.21.15.141:0/3458029385 >> v2:172.21.15.188:3300/0 conn(0x7f36200b37d0 msgr2=0x7f362015cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361de00640 1 --2- 172.21.15.141:0/3458029385 >> v2:172.21.15.188:3300/0 conn(0x7f36200b37d0 0x7f362015cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361de00640 1 -- 172.21.15.141:0/3458029385 >> v2:172.21.15.141:3300/0 conn(0x7f36200ce320 msgr2=0x7f362006cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361de00640 1 --2- 172.21.15.141:0/3458029385 >> v2:172.21.15.141:3300/0 conn(0x7f36200ce320 0x7f362006cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.141+0000 7f361de00640 1 -- 172.21.15.141:0/3458029385 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36200b1e00 con 0x7f3620159300 2024-01-16T14:57:14.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.143+0000 7f361de00640 1 --2- 172.21.15.141:0/3458029385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f362006c890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f36200b3e90 tx=0x7f361802d3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c01fcff5509fd161 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== osd.0 v2:172.21.15.141:6800/4153030184 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f80ac076080 con 0x7f80ac0735b0 2024-01-16T14:57:14.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f2cf7e00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd806b310 0x7f2cd806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.144+0000 7f361ca00640 1 -- 172.21.15.141:0/3458029385 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618034600 con 0x7f3620159300 2024-01-16T14:57:14.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cf008a9d0 con 0x7f2cf803c070 2024-01-16T14:57:14.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f2cf7e00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f2cd8073530 0x7f2cd8075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7fa275a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa270071070 0x7fa270141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 --> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2cd8076000 con 0x7f2cd8073530 2024-01-16T14:57:14.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cf008ad40 con 0x7f2cf803c070 2024-01-16T14:57:14.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f2d04e00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f2cd8073530 0x7f2cd8075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f361ca00640 1 -- 172.21.15.141:0/3458029385 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618030050 con 0x7f3620159300 2024-01-16T14:57:14.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.145+0000 7f361ca00640 1 -- 172.21.15.141:0/3458029385 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36180349a0 con 0x7f3620159300 2024-01-16T14:57:14.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f361f200640 1 -- 172.21.15.141:0/3458029385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 msgr2=0x7f362006c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f361f200640 1 --2- 172.21.15.141:0/3458029385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f362006c890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f36200b3e90 tx=0x7f361802d3a0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7fa275a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2700cf7d0 0x7fa27015d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f361f200640 1 -- 172.21.15.141:0/3458029385 shutdown_connections 2024-01-16T14:57:14.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f361f200640 1 --2- 172.21.15.141:0/3458029385 >> v2:172.21.15.188:3300/0 conn(0x7f36200b37d0 0x7f362015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f361f200640 1 --2- 172.21.15.141:0/3458029385 >> v2:172.21.15.141:3300/0 conn(0x7f36200ce320 0x7f362006cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f361f200640 1 --2- 172.21.15.141:0/3458029385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f362006c890 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f361f200640 1 -- 172.21.15.141:0/3458029385 >> 172.21.15.141:0/3458029385 conn(0x7f362006a800 msgr2=0x7f362006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f1c79400640 1 --2- 172.21.15.141:0/15489732 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740ce8f0 0x7f1c740ced50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009870 tx=0x7f1c5c0341b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4370c30cb26912e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f97400393b0 con 0x7f9758071a70 2024-01-16T14:57:14.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.146+0000 7fa275a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa27014b2c0 0x7fa270164280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7f361f200640 1 -- 172.21.15.141:0/3458029385 shutdown_connections 2024-01-16T14:57:14.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7f361f200640 1 -- 172.21.15.141:0/3458029385 wait complete. 2024-01-16T14:57:14.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7f1c78a00640 1 -- 172.21.15.141:0/15489732 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c03c440 con 0x7f1c740ce8f0 2024-01-16T14:57:14.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7fa26ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa270071070 0x7fa270141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7f2d04e00640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f2cd8073530 0x7f2cd8075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7f2cff400640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd806b310 0x7f2cd806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7f1c78a00640 1 -- 172.21.15.141:0/15489732 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c03c5a0 con 0x7f1c740ce8f0 2024-01-16T14:57:14.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7f1c78a00640 1 -- 172.21.15.141:0/15489732 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c03c970 con 0x7f1c740ce8f0 2024-01-16T14:57:14.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7fa275a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa270161e70 con 0x7fa270071070 2024-01-16T14:57:14.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7fa275a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701f71a0 con 0x7fa2700cf7d0 2024-01-16T14:57:14.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7fa275a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27015d690 con 0x7fa27014b2c0 2024-01-16T14:57:14.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7fa26f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa27014b2c0 0x7fa270164280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7fa26ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa270071070 0x7fa270141ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48358/0 (socket says 172.21.15.141:48358) 2024-01-16T14:57:14.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.147+0000 7fa26ea00640 1 -- 172.21.15.141:0/2794758782 learned_addr learned my addr 172.21.15.141:0/2794758782 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7fa6c3a00640 1 Processor -- start 2024-01-16T14:57:14.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7f1c7c800640 1 -- 172.21.15.141:0/15489732 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740ce8f0 msgr2=0x7f1c740ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7f1c7c800640 1 --2- 172.21.15.141:0/15489732 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740ce8f0 0x7f1c740ced50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009870 tx=0x7f1c5c0341b0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7fa26ea00640 1 -- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa27014b2c0 msgr2=0x7fa270164280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7fa26ea00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa27014b2c0 0x7fa270164280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7fa26ea00640 1 -- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2700cf7d0 msgr2=0x7fa27015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f584c00abd0 con 0x7f5854160e00 2024-01-16T14:57:14.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7fa26ea00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2700cf7d0 0x7fa27015d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.148+0000 7fa26ea00640 1 -- 172.21.15.141:0/2794758782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2600094d0 con 0x7fa270071070 2024-01-16T14:57:14.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f1c7c800640 1 -- 172.21.15.141:0/15489732 shutdown_connections 2024-01-16T14:57:14.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f1c7c800640 1 --2- 172.21.15.141:0/15489732 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740ce8f0 0x7f1c740ced50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f1c7c800640 1 --2- 172.21.15.141:0/15489732 >> v2:172.21.15.141:3300/0 conn(0x7f1c74160f80 0x7f1c74161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f974f400640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f973006b2c0 0x7f973006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97400ab880 con 0x7f9758071a70 2024-01-16T14:57:14.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f1c7c800640 1 -- 172.21.15.141:0/15489732 >> 172.21.15.141:0/15489732 conn(0x7f1c7409c550 msgr2=0x7f1c740d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f1c7c800640 1 -- 172.21.15.141:0/15489732 shutdown_connections 2024-01-16T14:57:14.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f974f400640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f97300734e0 0x7f9730075900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f2cff400640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd806b310 0x7f2cd806d7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8036000 tx=0x7f2ce8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 --> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9730075fb0 con 0x7f97300734e0 2024-01-16T14:57:14.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97400b02a0 con 0x7f9758071a70 2024-01-16T14:57:14.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7f9756a00640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f973006b2c0 0x7f973006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7fa26ea00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa270071070 0x7fa270141ff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa260009960 tx=0x7fa260004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7fa6c3a00640 1 -- start start 2024-01-16T14:57:14.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260039b50 con 0x7fa270071070 2024-01-16T14:57:14.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7f5851600640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f582c06b2c0 0x7f582c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f584c08a180 con 0x7f5854160e00 2024-01-16T14:57:14.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa260039cb0 con 0x7fa270071070 2024-01-16T14:57:14.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260039b50 con 0x7fa270071070 2024-01-16T14:57:14.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa270162080 con 0x7fa270071070 2024-01-16T14:57:14.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.150+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa270162640 con 0x7fa270071070 2024-01-16T14:57:14.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.151+0000 7f5851600640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f582c073460 0x7f582c075880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.151+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 --> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f582c075f30 con 0x7f582c073460 2024-01-16T14:57:14.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.151+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f584c08a4f0 con 0x7f5854160e00 2024-01-16T14:57:14.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.151+0000 7f361f200640 1 Processor -- start 2024-01-16T14:57:14.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.151+0000 7f361f200640 1 -- start start 2024-01-16T14:57:14.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.149+0000 7f1c7c800640 1 -- 172.21.15.141:0/15489732 wait complete. 2024-01-16T14:57:14.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c3a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa6bc14b2c0 0x7fa6bc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6bc0cf7d0 0x7fa6bc070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c3a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa6bc070ea0 0x7fa6bc0b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7f1c7c800640 1 Processor -- start 2024-01-16T14:57:14.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c3a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6bc14b6a0 con 0x7fa6bc070ea0 2024-01-16T14:57:14.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c3a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6bc0cfbf0 con 0x7fa6bc14b2c0 2024-01-16T14:57:14.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc071690 con 0x7fa6bc0cf7d0 2024-01-16T14:57:14.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== osd.1 v2:172.21.15.141:6808/3479163278 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2cd8076000 con 0x7f2cd8073530 2024-01-16T14:57:14.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7f9757e00640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f97300734e0 0x7f9730075900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c1000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa6bc14b2c0 0x7fa6bc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7f1c7c800640 1 -- start start 2024-01-16T14:57:14.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f5858c00640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f582c073460 0x7f582c075880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c1000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa6bc14b2c0 0x7fa6bc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36374/0 (socket says 172.21.15.141:36374) 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.152+0000 7fa6c1000640 1 -- 172.21.15.141:0/409217520 learned_addr learned my addr 172.21.15.141:0/409217520 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f5859600640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f582c06b2c0 0x7f582c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36200b37d0 0x7f3620152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36200ce320 0x7f3620157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f3620159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3620154120 con 0x7f36200ce320 2024-01-16T14:57:14.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362014c000 con 0x7f36200b37d0 2024-01-16T14:57:14.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3620153b60 con 0x7f3620159300 2024-01-16T14:57:14.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.154+0000 7f5858c00640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f582c073460 0x7f582c075880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f9757e00640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f97300734e0 0x7f9730075900 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa240000fc0 con 0x7fa270071070 2024-01-16T14:57:14.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7fa6c1000640 1 -- 172.21.15.141:0/409217520 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6bc0cf7d0 msgr2=0x7fa6bc070960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T14:57:14.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7fa6c1000640 1 --2- 172.21.15.141:0/409217520 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6bc0cf7d0 0x7fa6bc070960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7fa6c1000640 1 -- 172.21.15.141:0/409217520 >> v2:172.21.15.141:3300/0 conn(0x7fa6bc070ea0 msgr2=0x7fa6bc0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7fa6c1000640 1 --2- 172.21.15.141:0/409217520 >> v2:172.21.15.141:3300/0 conn(0x7fa6bc070ea0 0x7fa6bc0b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7fa6c1000640 1 -- 172.21.15.141:0/409217520 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc071970 con 0x7fa6bc14b2c0 2024-01-16T14:57:14.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36200ce320 0x7f3620157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36200ce320 0x7f3620157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48382/0 (socket says 172.21.15.141:48382) 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.153+0000 7f361d400640 1 -- 172.21.15.141:0/1123418378 learned_addr learned my addr 172.21.15.141:0/1123418378 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.154+0000 7f361e800640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f3620159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.154+0000 7f361d400640 1 -- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 msgr2=0x7f3620159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.154+0000 7f361d400640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3620159300 0x7f3620159870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.154+0000 7f361d400640 1 -- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36200b37d0 msgr2=0x7f3620152e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.155+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== osd.3 v2:172.21.15.141:6823/1855057890 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9730075fb0 con 0x7f97300734e0 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.154+0000 7f361d400640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36200b37d0 0x7f3620152e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.154+0000 7f361d400640 1 -- 172.21.15.141:0/1123418378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3618009830 con 0x7f36200ce320 2024-01-16T14:57:14.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.155+0000 7f9756a00640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f973006b2c0 0x7f973006d780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f975814a570 tx=0x7f9748007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.155+0000 7f361d400640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36200ce320 0x7f3620157ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f360800b450 tx=0x7f360800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.155+0000 7fa6c1000640 1 --2- 172.21.15.141:0/409217520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc14b2c0 0x7fa6bc0cf290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc06e650 tx=0x7fa6ac02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e82d281738951bf0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.155+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa260043410 con 0x7fa270071070 2024-01-16T14:57:14.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.156+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608011890 con 0x7f36200ce320 2024-01-16T14:57:14.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.156+0000 7f5859600640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f582c06b2c0 0x7f582c06d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f583c009870 tx=0x7f583c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.156+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== osd.2 v2:172.21.15.141:6815/1316482903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f582c075f30 con 0x7f582c073460 2024-01-16T14:57:14.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.156+0000 7fa6bb400640 1 -- 172.21.15.141:0/409217520 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac030610 con 0x7fa6bc14b2c0 2024-01-16T14:57:14.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa6bb400640 1 -- 172.21.15.141:0/409217520 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac030770 con 0x7fa6bc14b2c0 2024-01-16T14:57:14.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa266a00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa24806b320 0x7fa24806d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36080119f0 con 0x7f36200ce320 2024-01-16T14:57:14.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608011da0 con 0x7f36200ce320 2024-01-16T14:57:14.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3620153e40 con 0x7f36200ce320 2024-01-16T14:57:14.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa6bb400640 1 -- 172.21.15.141:0/409217520 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac030b00 con 0x7fa6bc14b2c0 2024-01-16T14:57:14.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7f1c7c800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c740ce8f0 0x7f1c740b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7f1c7c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740cfbe0 0x7f1c740b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7f1c7c800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c74160f80 0x7f1c74156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2600abdd0 con 0x7fa270071070 2024-01-16T14:57:14.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa266a00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fa2480734c0 0x7fa2480758e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 --> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa248075f90 con 0x7fa2480734c0 2024-01-16T14:57:14.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa260076ba0 con 0x7fa270071070 2024-01-16T14:57:14.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7fa267e00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa24806b320 0x7fa24806d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.157+0000 7f1c79e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c740ce8f0 0x7f1c740b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f1c79e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c740ce8f0 0x7f1c740b6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48394/0 (socket says 172.21.15.141:48394) 2024-01-16T14:57:14.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f1c79e00640 1 -- 172.21.15.141:0/917757370 learned_addr learned my addr 172.21.15.141:0/917757370 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362014ac90 con 0x7f36200ce320 2024-01-16T14:57:14.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740c7e70 con 0x7f1c740ce8f0 2024-01-16T14:57:14.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7fa6c3a00640 1 -- 172.21.15.141:0/409217520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc14b2c0 msgr2=0x7fa6bc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7fa6c3a00640 1 --2- 172.21.15.141:0/409217520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc14b2c0 0x7fa6bc0cf290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc06e650 tx=0x7fa6ac02b960 comp rx=0 tx=0).stop 2024-01-16T14:57:14.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f1c7a800640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c74160f80 0x7f1c74156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740c86a0 con 0x7f1c74160f80 2024-01-16T14:57:14.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f1c79400640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740cfbe0 0x7f1c740b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.158+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740b7280 con 0x7f1c740cfbe0 2024-01-16T14:57:14.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.159+0000 7fa6c3a00640 1 -- 172.21.15.141:0/409217520 shutdown_connections 2024-01-16T14:57:14.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.159+0000 7fa6c3a00640 1 --2- 172.21.15.141:0/409217520 >> v2:172.21.15.141:3300/0 conn(0x7fa6bc070ea0 0x7fa6bc0b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.159+0000 7fa6c3a00640 1 --2- 172.21.15.141:0/409217520 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6bc0cf7d0 0x7fa6bc070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.159+0000 7fa6c3a00640 1 --2- 172.21.15.141:0/409217520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc14b2c0 0x7fa6bc0cf290 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.159+0000 7fa6c3a00640 1 -- 172.21.15.141:0/409217520 >> 172.21.15.141:0/409217520 conn(0x7fa6bc06a800 msgr2=0x7fa6bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.159+0000 7fa26f400640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fa2480734c0 0x7fa2480758e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.160+0000 7fa267e00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa24806b320 0x7fa24806d7e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa258000f60 tx=0x7fa2580079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.160+0000 7fa6c3a00640 1 -- 172.21.15.141:0/409217520 shutdown_connections 2024-01-16T14:57:14.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.160+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f36200b26a0 con 0x7f36200ce320 2024-01-16T14:57:14.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.160+0000 7f1c79e00640 1 -- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740cfbe0 msgr2=0x7f1c740b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.160+0000 7f1c79e00640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c740cfbe0 0x7f1c740b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.161+0000 7fa6c3a00640 1 -- 172.21.15.141:0/409217520 wait complete. 2024-01-16T14:57:14.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.160+0000 7f1c79e00640 1 -- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c74160f80 msgr2=0x7f1c74156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.161+0000 7f1c79e00640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c74160f80 0x7f1c74156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.161+0000 7f1c79e00640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c0094d0 con 0x7f1c740ce8f0 2024-01-16T14:57:14.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.161+0000 7f1c7a800640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c74160f80 0x7f1c74156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:14.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.160+0000 7fa26f400640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fa2480734c0 0x7fa2480758e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.161+0000 7f1c79e00640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c740ce8f0 0x7f1c740b6790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400d5f0 tx=0x7f1c6400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.162+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c640125f0 con 0x7f1c740ce8f0 2024-01-16T14:57:14.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.162+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64012750 con 0x7f1c740ce8f0 2024-01-16T14:57:14.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.162+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64012b00 con 0x7f1c740ce8f0 2024-01-16T14:57:14.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.163+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== osd.4 v2:172.21.15.188:6800/2387947334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa248075f90 con 0x7fa2480734c0 2024-01-16T14:57:14.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.162+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c740b6cd0 con 0x7f1c740ce8f0 2024-01-16T14:57:14.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.163+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c740c8050 con 0x7f1c740ce8f0 2024-01-16T14:57:14.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.164+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f360801a400 con 0x7f36200ce320 2024-01-16T14:57:14.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.164+0000 7fa6c3a00640 1 Processor -- start 2024-01-16T14:57:14.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.164+0000 7fa6c3a00640 1 -- start start 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.166+0000 7fa6c3a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc070ea0 0x7fa6bc0adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.166+0000 7fa6c3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6bc0cf7d0 0x7fa6bc145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.166+0000 7fa6c3a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6bc14b2c0 0x7fa6bc0afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.166+0000 7fa6c3a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc1457b0 con 0x7fa6bc14b2c0 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.166+0000 7fa6c3a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc144b30 con 0x7fa6bc070ea0 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.166+0000 7fa6c3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc144e10 con 0x7fa6bc0cf7d0 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.167+0000 7fa6c1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc070ea0 0x7fa6bc0adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.167+0000 7fa6c1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc070ea0 0x7fa6bc0adc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36392/0 (socket says 172.21.15.141:36392) 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.167+0000 7fa6c1000640 1 -- 172.21.15.141:0/3154606116 learned_addr learned my addr 172.21.15.141:0/3154606116 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.167+0000 7fa6c1a00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6bc14b2c0 0x7fa6bc0afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.168+0000 7fa6c1000640 1 -- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6bc0cf7d0 msgr2=0x7fa6bc145bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.168+0000 7fa6c1000640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6bc0cf7d0 0x7fa6bc145bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.168+0000 7fa6c1000640 1 -- 172.21.15.141:0/3154606116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6bc14b2c0 msgr2=0x7fa6bc0afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.168+0000 7fa6c1000640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6bc14b2c0 0x7fa6bc0afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.168+0000 7fa6c1000640 1 -- 172.21.15.141:0/3154606116 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0094d0 con 0x7fa6bc070ea0 2024-01-16T14:57:14.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.168+0000 7fa6c1000640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc070ea0 0x7fa6bc0adc90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac000c00 tx=0x7fa6ac004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.164+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c74161360 con 0x7f1c740ce8f0 2024-01-16T14:57:14.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.170+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac039c90 con 0x7fa6bc070ea0 2024-01-16T14:57:14.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.170+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac039df0 con 0x7fa6bc070ea0 2024-01-16T14:57:14.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.170+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac043640 con 0x7fa6bc070ea0 2024-01-16T14:57:14.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.170+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc1450f0 con 0x7fa6bc070ea0 2024-01-16T14:57:14.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.170+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc0ae1d0 con 0x7fa6bc070ea0 2024-01-16T14:57:14.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.171+0000 7f3616000640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f35f806b390 0x7f35f806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.171+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3608089fe0 con 0x7f36200ce320 2024-01-16T14:57:14.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.172+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6bc0cbc40 con 0x7fa6bc070ea0 2024-01-16T14:57:14.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.172+0000 7f3616000640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7f35f80735b0 0x7f35f80759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.172+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 --> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35f8076080 con 0x7f35f80735b0 2024-01-16T14:57:14.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.172+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f360808c5a0 con 0x7f36200ce320 2024-01-16T14:57:14.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.172+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1c64012c60 con 0x7f1c740ce8f0 2024-01-16T14:57:14.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.173+0000 7f361e800640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7f35f80735b0 0x7f35f80759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.174+0000 7f361de00640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f35f806b390 0x7f35f806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.174+0000 7f1c72000640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c4c06b2f0 0x7f1c4c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.174+0000 7f361e800640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7f35f80735b0 0x7f35f80759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.174+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c64089f10 con 0x7f1c740ce8f0 2024-01-16T14:57:14.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.173+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa6ac0437a0 con 0x7fa6bc070ea0 2024-01-16T14:57:14.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7f1c72000640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7f1c4c073510 0x7f1c4c075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7fa6b8c00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa69406b310 0x7fa69406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c4c075fe0 con 0x7f1c4c073510 2024-01-16T14:57:14.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7f1c79400640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c4c06b2f0 0x7f1c4c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c6408c530 con 0x7f1c740ce8f0 2024-01-16T14:57:14.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6ac03e070 con 0x7fa6bc070ea0 2024-01-16T14:57:14.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7f361de00640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f35f806b390 0x7f35f806d850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3618036000 tx=0x7f3618009000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.175+0000 7fa6b8c00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7fa6940734b0 0x7fa6940758d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.176+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 --> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa694075f80 con 0x7fa6940734b0 2024-01-16T14:57:14.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.176+0000 7fa6bbe00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa69406b310 0x7fa69406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.176+0000 7f1c7a800640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7f1c4c073510 0x7f1c4c075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.176+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6ac077980 con 0x7fa6bc070ea0 2024-01-16T14:57:14.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.176+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 --> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80a4002cb0 con 0x7f80ac0735b0 2024-01-16T14:57:14.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.176+0000 7fa6c1a00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7fa6940734b0 0x7fa6940758d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7f1c79400640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c4c06b2f0 0x7f1c4c06d7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c03e000 tx=0x7f1c5c038000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7f80d0c00640 1 -- 172.21.15.141:0/2373795557 <== osd.0 v2:172.21.15.141:6800/4153030184 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f80a4002cb0 con 0x7f80ac0735b0 2024-01-16T14:57:14.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 --> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2cf8006610 con 0x7f2cd8073530 2024-01-16T14:57:14.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== osd.6 v2:172.21.15.188:6824/2194964726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f35f8076080 con 0x7f35f80735b0 2024-01-16T14:57:14.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f80ac0735b0 msgr2=0x7f80ac0759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7f80dba00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f80ac0735b0 0x7f80ac0759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7fa6bbe00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa69406b310 0x7fa69406d7d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc161280 tx=0x7fa6a4006bd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ac06b390 msgr2=0x7f80ac06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ac06b390 0x7f80ac06d850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f80bc036000 tx=0x7f80bc03b040 comp rx=0 tx=0).stop 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80cc053c70 msgr2=0x7f80cc1796a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 --2- 172.21.15.141:0/2373795557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80cc053c70 0x7f80cc1796a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b450 tx=0x7f80c400b910 comp rx=0 tx=0).stop 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80d9a00640 1 -- 172.21.15.141:0/2373795557 reap_dead start 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 shutdown_connections 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 >> 172.21.15.141:0/2373795557 conn(0x7f80cc058ea0 msgr2=0x7f80cc037e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 shutdown_connections 2024-01-16T14:57:14.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f80dba00640 1 -- 172.21.15.141:0/2373795557 wait complete. 2024-01-16T14:57:14.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7f2cf7e00640 1 -- 172.21.15.141:0/4285355788 <== osd.1 v2:172.21.15.141:6808/3479163278 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2d0004e840 con 0x7f2cd8073530 2024-01-16T14:57:14.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.177+0000 7f1c7a800640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7f1c4c073510 0x7f1c4c075930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.178+0000 7fa6c1a00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7fa6940734b0 0x7fa6940758d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.179+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f2cd8073530 msgr2=0x7f2cd8075950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.179+0000 7f2d05800640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f2cd8073530 0x7f2cd8075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.179+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd806b310 msgr2=0x7f2cd806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.179+0000 7f2d05800640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd806b310 0x7f2cd806d7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8036000 tx=0x7f2ce8008030 comp rx=0 tx=0).stop 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.179+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 msgr2=0x7f2cf80e5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.179+0000 7f2d05800640 1 --2- 172.21.15.141:0/4285355788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cf803c070 0x7f2cf80e5570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000d5f0 tx=0x7f2cf000d9f0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.179+0000 7f2d04e00640 1 -- 172.21.15.141:0/4285355788 reap_dead start 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.180+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 shutdown_connections 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.180+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 >> 172.21.15.141:0/4285355788 conn(0x7f2cf8058eb0 msgr2=0x7f2cf8056de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.180+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 shutdown_connections 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.180+0000 7f2d05800640 1 -- 172.21.15.141:0/4285355788 wait complete. 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.180+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== osd.5 v2:172.21.15.188:6812/917917352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1c4c075fe0 con 0x7f1c4c073510 2024-01-16T14:57:14.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.180+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== osd.7 v2:172.21.15.188:6808/2741327973 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa694075f80 con 0x7fa6940734b0 2024-01-16T14:57:14.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.181+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 --> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9758067450 con 0x7f97300734e0 2024-01-16T14:57:14.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.182+0000 7f974f400640 1 -- 172.21.15.141:0/307033422 <== osd.3 v2:172.21.15.141:6823/1855057890 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9758067450 con 0x7f97300734e0 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.183+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f97300734e0 msgr2=0x7f9730075900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.183+0000 7f975e800640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f97300734e0 0x7f9730075900 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.185+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f973006b2c0 msgr2=0x7f973006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.185+0000 7f975e800640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f973006b2c0 0x7f973006d780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f975814a570 tx=0x7f9748007a40 comp rx=0 tx=0).stop 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.185+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9758071a70 msgr2=0x7f9758145540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.185+0000 7f975e800640 1 --2- 172.21.15.141:0/307033422 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9758071a70 0x7f9758145540 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f974002be60 tx=0x7f97400046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.185+0000 7f9757e00640 1 -- 172.21.15.141:0/307033422 reap_dead start 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.185+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 shutdown_connections 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.185+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 >> 172.21.15.141:0/307033422 conn(0x7f975806a800 msgr2=0x7f9758162780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.190+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 shutdown_connections 2024-01-16T14:57:14.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.190+0000 7f975e800640 1 -- 172.21.15.141:0/307033422 wait complete. 2024-01-16T14:57:14.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.191+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 --> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f585415a440 con 0x7f582c073460 2024-01-16T14:57:14.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.195+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 --> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa240002cb0 con 0x7fa2480734c0 2024-01-16T14:57:14.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.196+0000 7f5851600640 1 -- 172.21.15.141:0/3417021174 <== osd.2 v2:172.21.15.141:6815/1316482903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f585415a440 con 0x7f582c073460 2024-01-16T14:57:14.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.196+0000 7fa266a00640 1 -- 172.21.15.141:0/2794758782 <== osd.4 v2:172.21.15.188:6800/2387947334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa240002cb0 con 0x7fa2480734c0 2024-01-16T14:57:14.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.196+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f582c073460 msgr2=0x7f582c075880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.196+0000 7f585aa00640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f582c073460 0x7f582c075880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fa2480734c0 msgr2=0x7fa2480758e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fa2480734c0 0x7fa2480758e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa24806b320 msgr2=0x7fa24806d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa24806b320 0x7fa24806d7e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa258000f60 tx=0x7fa2580079b0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa270071070 msgr2=0x7fa270141ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 --2- 172.21.15.141:0/2794758782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa270071070 0x7fa270141ff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa260009960 tx=0x7fa260004730 comp rx=0 tx=0).stop 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa26f400640 1 -- 172.21.15.141:0/2794758782 reap_dead start 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 shutdown_connections 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 >> 172.21.15.141:0/2794758782 conn(0x7fa27006a800 msgr2=0x7fa27006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 shutdown_connections 2024-01-16T14:57:14.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.197+0000 7fa275a00640 1 -- 172.21.15.141:0/2794758782 wait complete. 2024-01-16T14:57:14.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f582c06b2c0 msgr2=0x7f582c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f582c06b2c0 0x7f582c06d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f583c009870 tx=0x7f583c008030 comp rx=0 tx=0).stop 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5854160e00 msgr2=0x7f58540b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 --2- 172.21.15.141:0/3417021174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5854160e00 0x7f58540b6720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f584c00c760 tx=0x7f584c00cc20 comp rx=0 tx=0).stop 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585a000640 1 -- 172.21.15.141:0/3417021174 reap_dead start 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 shutdown_connections 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 >> 172.21.15.141:0/3417021174 conn(0x7f58540d3120 msgr2=0x7f58540cded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 shutdown_connections 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.198+0000 7f585aa00640 1 -- 172.21.15.141:0/3417021174 wait complete. 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.199+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 --> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f36200b23c0 con 0x7f35f80735b0 2024-01-16T14:57:14.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.201+0000 7f3616000640 1 -- 172.21.15.141:0/1123418378 <== osd.6 v2:172.21.15.188:6824/2194964726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f361000c040 con 0x7f35f80735b0 2024-01-16T14:57:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.202+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7f35f80735b0 msgr2=0x7f35f80759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.202+0000 7f361f200640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7f35f80735b0 0x7f35f80759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.202+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f35f806b390 msgr2=0x7f35f806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.202+0000 7f361f200640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f35f806b390 0x7f35f806d850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3618036000 tx=0x7f3618009000 comp rx=0 tx=0).stop 2024-01-16T14:57:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.202+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36200ce320 msgr2=0x7f3620157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.202+0000 7f361f200640 1 --2- 172.21.15.141:0/1123418378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36200ce320 0x7f3620157ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f360800b450 tx=0x7f360800b910 comp rx=0 tx=0).stop 2024-01-16T14:57:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.202+0000 7f361e800640 1 -- 172.21.15.141:0/1123418378 reap_dead start 2024-01-16T14:57:14.210 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:14.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.204+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 shutdown_connections 2024-01-16T14:57:14.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.204+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 >> 172.21.15.141:0/1123418378 conn(0x7f362006a800 msgr2=0x7f3620070a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.204+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 shutdown_connections 2024-01-16T14:57:14.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.204+0000 7f361f200640 1 -- 172.21.15.141:0/1123418378 wait complete. 2024-01-16T14:57:14.211 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T14:57:14.212 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:14.212 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T14:57:14.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.210+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 --> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6bc0cb960 con 0x7fa6940734b0 2024-01-16T14:57:14.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.210+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 --> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c740d1c40 con 0x7f1c4c073510 2024-01-16T14:57:14.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.211+0000 7fa6b8c00640 1 -- 172.21.15.141:0/3154606116 <== osd.7 v2:172.21.15.188:6808/2741327973 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa6b4009a70 con 0x7fa6940734b0 2024-01-16T14:57:14.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7f1c72000640 1 -- 172.21.15.141:0/917757370 <== osd.5 v2:172.21.15.188:6812/917917352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1c6c001f30 con 0x7f1c4c073510 2024-01-16T14:57:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7fa6940734b0 msgr2=0x7fa6940758d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7fa6940734b0 0x7fa6940758d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7f1c4c073510 msgr2=0x7f1c4c075930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7f1c7c800640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7f1c4c073510 0x7f1c4c075930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c4c06b2f0 msgr2=0x7f1c4c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa69406b310 msgr2=0x7fa69406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa69406b310 0x7fa69406d7d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc161280 tx=0x7fa6a4006bd0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc070ea0 msgr2=0x7fa6bc0adc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 --2- 172.21.15.141:0/3154606116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6bc070ea0 0x7fa6bc0adc90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac000c00 tx=0x7fa6ac004730 comp rx=0 tx=0).stop 2024-01-16T14:57:14.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7f1c7c800640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c4c06b2f0 0x7f1c4c06d7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c03e000 tx=0x7f1c5c038000 comp rx=0 tx=0).stop 2024-01-16T14:57:14.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c1a00640 1 -- 172.21.15.141:0/3154606116 reap_dead start 2024-01-16T14:57:14.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 shutdown_connections 2024-01-16T14:57:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 >> 172.21.15.141:0/3154606116 conn(0x7fa6bc06a800 msgr2=0x7fa6bc0aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c740ce8f0 msgr2=0x7f1c740b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.212+0000 7f1c7c800640 1 --2- 172.21.15.141:0/917757370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c740ce8f0 0x7f1c740b6790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400d5f0 tx=0x7f1c6400dab0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.213+0000 7f1c7a800640 1 -- 172.21.15.141:0/917757370 reap_dead start 2024-01-16T14:57:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.213+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 shutdown_connections 2024-01-16T14:57:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.213+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 >> 172.21.15.141:0/917757370 conn(0x7f1c7409c550 msgr2=0x7f1c740d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.213+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 shutdown_connections 2024-01-16T14:57:14.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.213+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 shutdown_connections 2024-01-16T14:57:14.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.213+0000 7fa6c3a00640 1 -- 172.21.15.141:0/3154606116 wait complete. 2024-01-16T14:57:14.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.213+0000 7f1c7c800640 1 -- 172.21.15.141:0/917757370 wait complete. 2024-01-16T14:57:14.218 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:14.218 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T14:57:14.219 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:14.219 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T14:57:14.220 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:14.220 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T14:57:14.221 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:14.221 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T14:57:14.222 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:14.222 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T14:57:14.223 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:14.223 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T14:57:14.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.568+0000 7f6923600640 1 Processor -- start 2024-01-16T14:57:14.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.569+0000 7f6923600640 1 -- start start 2024-01-16T14:57:14.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.571+0000 7f6923600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c15ce90 0x7f691c15d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.571+0000 7f6923600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f691c06c470 0x7f691c06c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f6923600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c15d830 con 0x7f691c153950 2024-01-16T14:57:14.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f6923600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c153d30 con 0x7f691c06c470 2024-01-16T14:57:14.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f6923600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0b2b70 con 0x7f691c15ce90 2024-01-16T14:57:14.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f6920c00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f691c153950 0x7f691c0cb0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:41520/0 (socket says 172.21.15.141:41520) 2024-01-16T14:57:14.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f6920c00640 1 -- 172.21.15.141:0/1444836692 learned_addr learned my addr 172.21.15.141:0/1444836692 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:14.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f6921600640 1 --2- 172.21.15.141:0/1444836692 >> v2:172.21.15.141:3300/0 conn(0x7f691c06c470 0x7f691c06c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.568+0000 7f7a2b600640 1 Processor -- start 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.568+0000 7f7a2b600640 1 -- start start 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.570+0000 7f7a2b600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7a240ae5b0 0x7f7a240cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.570+0000 7f7a2b600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a240cf770 con 0x7f7a240cf340 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.570+0000 7f7a2b600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a24070fc0 con 0x7f7a2414b2c0 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.570+0000 7f7a2b600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2414b6a0 con 0x7f7a240ae5b0 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.571+0000 7f7a29600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7a240cf340 0x7f7a24161fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40030/0 (socket says 172.21.15.141:40030) 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.571+0000 7f7a29600640 1 -- 172.21.15.141:0/2177357891 learned_addr learned my addr 172.21.15.141:0/2177357891 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:14.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2177357891 >> v2:172.21.15.188:3300/0 conn(0x7f7a240ae5b0 0x7f7a240cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41616142 0 0) 0x7f7a240cf770 con 0x7f7a240cf340 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04003520 con 0x7f7a240cf340 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1905802475 0 0) 0x7f7a04003520 con 0x7f7a240cf340 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 >> v2:172.21.15.188:3300/0 conn(0x7f7a240ae5b0 msgr2=0x7f7a240cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.572+0000 7f7a23400640 1 --2- 172.21.15.141:0/2177357891 >> v2:172.21.15.188:3300/0 conn(0x7f7a240ae5b0 0x7f7a240cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 >> v1:172.21.15.188:6789/0 conn(0x7f7a2414b2c0 legacy=0x7f7a24070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240712a0 con 0x7f7a240cf340 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7a1c0019f0 con 0x7f7a240cf340 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 >> v1:172.21.15.141:6789/0 conn(0x7f7a240cf340 legacy=0x7f7a24161fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a04003970 0x7f7a04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a040047c0 0x7f7a04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a04005180 0x7f7a0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2414b6a0 con 0x7f7a040047c0 2024-01-16T14:57:14.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0400c020 con 0x7f7a04003970 2024-01-16T14:57:14.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0400c1e0 con 0x7f7a04005180 2024-01-16T14:57:14.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f6921600640 1 -- 172.21.15.141:0/1444836692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c15ce90 msgr2=0x7f691c15d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f6921600640 1 --2- 172.21.15.141:0/1444836692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c15ce90 0x7f691c15d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f6921600640 1 -- 172.21.15.141:0/1444836692 >> v1:172.21.15.188:6789/0 conn(0x7f691c153950 legacy=0x7f691c0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a040047c0 0x7f7a04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.574+0000 7f7a28c00640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a04003970 0x7f7a04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.574+0000 7f7a23e00640 1 -- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a04005180 msgr2=0x7f7a0400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.574+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a04005180 0x7f7a0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.574+0000 7f7a23e00640 1 -- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a04003970 msgr2=0x7f7a04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.574+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a04003970 0x7f7a04003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.574+0000 7f7a23e00640 1 -- 172.21.15.141:0/2177357891 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240712a0 con 0x7f7a040047c0 2024-01-16T14:57:14.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.574+0000 7f7a28c00640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a04003970 0x7f7a04003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:14.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.573+0000 7f6921600640 1 -- 172.21.15.141:0/1444836692 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0b2e50 con 0x7f691c06c470 2024-01-16T14:57:14.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.575+0000 7f7a29600640 1 -- 172.21.15.141:0/2177357891 reap_dead start 2024-01-16T14:57:14.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.575+0000 7f6921600640 1 --2- 172.21.15.141:0/1444836692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c06c470 0x7f691c06c8e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6914009d90 tx=0x7f691402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe74e72be01a2ddd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.575+0000 7f691b400640 1 -- 172.21.15.141:0/1444836692 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691402cc60 con 0x7f691c06c470 2024-01-16T14:57:14.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.575+0000 7f38c2000640 1 Processor -- start 2024-01-16T14:57:14.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.575+0000 7f691b400640 1 -- 172.21.15.141:0/1444836692 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691402cdc0 con 0x7f691c06c470 2024-01-16T14:57:14.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.575+0000 7f691b400640 1 -- 172.21.15.141:0/1444836692 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914034660 con 0x7f691c06c470 2024-01-16T14:57:14.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.575+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a040047c0 0x7f7a04004c40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009aa0 tx=0x7f7a040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2276db28c8bc736 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.576+0000 7f38c2000640 1 -- start start 2024-01-16T14:57:14.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.576+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1402c6b0 con 0x7f7a040047c0 2024-01-16T14:57:14.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.576+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1402c810 con 0x7f7a040047c0 2024-01-16T14:57:14.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.576+0000 7f7a23400640 1 -- 172.21.15.141:0/2177357891 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1402cbc0 con 0x7f7a040047c0 2024-01-16T14:57:14.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.576+0000 7f6923600640 1 -- 172.21.15.141:0/1444836692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c06c470 msgr2=0x7f691c06c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.576+0000 7f6923600640 1 --2- 172.21.15.141:0/1444836692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c06c470 0x7f691c06c8e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6914009d90 tx=0x7f691402beb0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f7a2b600640 1 -- 172.21.15.141:0/2177357891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a040047c0 msgr2=0x7f7a04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f6923600640 1 -- 172.21.15.141:0/1444836692 shutdown_connections 2024-01-16T14:57:14.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f6923600640 1 --2- 172.21.15.141:0/1444836692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c06c470 0x7f691c06c8e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f6923600640 1 --2- 172.21.15.141:0/1444836692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c15ce90 0x7f691c15d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f6923600640 1 -- 172.21.15.141:0/1444836692 >> 172.21.15.141:0/1444836692 conn(0x7f691c0b0be0 msgr2=0x7f691c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a040047c0 0x7f7a04004c40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009aa0 tx=0x7f7a040057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f6923600640 1 -- 172.21.15.141:0/1444836692 shutdown_connections 2024-01-16T14:57:14.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.578+0000 7f7a2b600640 1 -- 172.21.15.141:0/2177357891 shutdown_connections 2024-01-16T14:57:14.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.578+0000 7f38c2000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f38bc160920 0x7f38bc0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.578+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2177357891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a040047c0 0x7f7a04004c40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.578+0000 7f7a2b600640 1 -- 172.21.15.141:0/2177357891 >> 172.21.15.141:0/2177357891 conn(0x7f7a2406a800 msgr2=0x7f7a2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.577+0000 7f6923600640 1 -- 172.21.15.141:0/1444836692 wait complete. 2024-01-16T14:57:14.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.578+0000 7f7a2b600640 1 -- 172.21.15.141:0/2177357891 shutdown_connections 2024-01-16T14:57:14.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.578+0000 7f7a2b600640 1 -- 172.21.15.141:0/2177357891 wait complete. 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.579+0000 7f38c2000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f38bc0cfea0 0x7f38bc161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.579+0000 7f38c2000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc0d0a10 con 0x7f38bc0cf9f0 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.579+0000 7f38c2000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc0d1f50 con 0x7f38bc0cfea0 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.579+0000 7f38c2000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc0cf080 con 0x7f38bc160920 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.579+0000 7f38bbe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f38bc0cf9f0 0x7f38bc161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40038/0 (socket says 172.21.15.141:40038) 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.579+0000 7f38bbe00640 1 -- 172.21.15.141:0/1460037436 learned_addr learned my addr 172.21.15.141:0/1460037436 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.579+0000 7f38c1600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f38bc0cfea0 0x7f38bc161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.580+0000 7f38c0c00640 1 --2- 172.21.15.141:0/1460037436 >> v2:172.21.15.188:3300/0 conn(0x7f38bc160920 0x7f38bc0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.580+0000 7f6923600640 1 Processor -- start 2024-01-16T14:57:14.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.580+0000 7fefee200640 1 Processor -- start 2024-01-16T14:57:14.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.580+0000 7fefee200640 1 -- start start 2024-01-16T14:57:14.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.580+0000 7f6923600640 1 -- start start 2024-01-16T14:57:14.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.581+0000 7f38c0c00640 1 -- 172.21.15.141:0/1460037436 >> v2:172.21.15.141:3300/0 conn(0x7f38bc0cfea0 msgr2=0x7f38bc161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.581+0000 7f38c0c00640 1 --2- 172.21.15.141:0/1460037436 >> v2:172.21.15.141:3300/0 conn(0x7f38bc0cfea0 0x7f38bc161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.581+0000 7f38c0c00640 1 -- 172.21.15.141:0/1460037436 >> v1:172.21.15.141:6789/0 conn(0x7f38bc0cf9f0 legacy=0x7f38bc161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.581+0000 7f38c0c00640 1 -- 172.21.15.141:0/1460037436 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc0d0310 con 0x7f38bc160920 2024-01-16T14:57:14.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.581+0000 7f7a2b600640 1 Processor -- start 2024-01-16T14:57:14.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fa0af600640 1 Processor -- start 2024-01-16T14:57:14.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7f7a2b600640 1 -- start start 2024-01-16T14:57:14.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7f38c1600640 1 --2- 172.21.15.141:0/1460037436 >> v2:172.21.15.141:3300/0 conn(0x7f38bc0cfea0 0x7f38bc161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:14.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fefee200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe80cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fefee200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fefe80d00b0 0x7fefe80ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fefee200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fefe80ce6f0 0x7fefe80d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fefee200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe80d0530 con 0x7fefe80d00b0 2024-01-16T14:57:14.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fefee200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe80b6750 con 0x7fefe80ce6f0 2024-01-16T14:57:14.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fefee200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe80cf1f0 con 0x7fefe815d590 2024-01-16T14:57:14.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7f7a2b600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a040047c0 0x7f7a241388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7f7a2b600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a240b2370 0x7f7a240aa120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7fefe6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fefe80d00b0 0x7fefe80ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7fefe6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fefe80d00b0 0x7fefe80ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48458/0 (socket says 172.21.15.141:48458) 2024-01-16T14:57:14.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7fefe6a00640 1 -- 172.21.15.141:0/1965482302 learned_addr learned my addr 172.21.15.141:0/1965482302 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7fefe7400640 1 --2- 172.21.15.141:0/1965482302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe80cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7f7a2b600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a240b27d0 0x7f7a240aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a2b600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2415d210 con 0x7f7a240b2370 2024-01-16T14:57:14.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.583+0000 7f38c0c00640 1 --2- 172.21.15.141:0/1460037436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc160920 0x7f38bc0b3f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38bc08c060 tx=0x7f38b402bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebc433a38d275bbc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a2b600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a240aaba0 con 0x7f7a240b27d0 2024-01-16T14:57:14.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a2b600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2415c310 con 0x7f7a040047c0 2024-01-16T14:57:14.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a28c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a040047c0 0x7f7a241388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a28c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a040047c0 0x7f7a241388a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:32872/0 (socket says 172.21.15.141:32872) 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a28c00640 1 -- 172.21.15.141:0/2682322285 learned_addr learned my addr 172.21.15.141:0/2682322285 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a240b2370 0x7f7a240aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f7a29600640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a240b27d0 0x7f7a240aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fefe6a00640 1 -- 172.21.15.141:0/1965482302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 msgr2=0x7fefe80cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fefe6a00640 1 --2- 172.21.15.141:0/1965482302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe80cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fefe6a00640 1 -- 172.21.15.141:0/1965482302 >> v2:172.21.15.188:3300/0 conn(0x7fefe80ce6f0 msgr2=0x7fefe80d0de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fefe6a00640 1 --2- 172.21.15.141:0/1965482302 >> v2:172.21.15.188:3300/0 conn(0x7fefe80ce6f0 0x7fefe80d0de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fefe6a00640 1 -- 172.21.15.141:0/1965482302 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80cef10 con 0x7fefe80d00b0 2024-01-16T14:57:14.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7f6923600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c06c470 0x7f691c148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f38bb400640 1 -- 172.21.15.141:0/1460037436 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b402ca30 con 0x7f38bc160920 2024-01-16T14:57:14.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6923600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f691c153950 0x7f691c15aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6923600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c15ce90 0x7f691c145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6923600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c15cc10 con 0x7f691c15ce90 2024-01-16T14:57:14.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6923600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c15af80 con 0x7f691c153950 2024-01-16T14:57:14.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6923600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c15c3d0 con 0x7f691c06c470 2024-01-16T14:57:14.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f38bb400640 1 -- 172.21.15.141:0/1460037436 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b402cb90 con 0x7f38bc160920 2024-01-16T14:57:14.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f7a28c00640 1 -- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a240b27d0 msgr2=0x7f7a240aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f7a28c00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a240b27d0 0x7f7a240aa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f7a28c00640 1 -- 172.21.15.141:0/2682322285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a240b2370 msgr2=0x7f7a240aa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f7a28c00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a240b2370 0x7f7a240aa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f7a28c00640 1 -- 172.21.15.141:0/2682322285 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2415c590 con 0x7f7a040047c0 2024-01-16T14:57:14.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a240b2370 0x7f7a240aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:14.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.582+0000 7fa0af600640 1 -- start start 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fa0af600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa0a815d590 0x7fa0a80cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fa0af600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa0a80ce8c0 0x7fa0a80b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fa0af600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80b6750 con 0x7fa0a80d00b0 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fa0af600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a80d0530 con 0x7fa0a815d590 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7fa0af600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a80cf1f0 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7fa0a7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa0a80d00b0 0x7fa0a80ce1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40040/0 (socket says 172.21.15.141:40040) 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7fa0a7e00640 1 -- 172.21.15.141:0/141791028 learned_addr learned my addr 172.21.15.141:0/141791028 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.584+0000 7f9596a00640 1 Processor -- start 2024-01-16T14:57:14.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f9596a00640 1 -- start start 2024-01-16T14:57:14.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7fa0acc00640 1 --2- 172.21.15.141:0/141791028 >> v2:172.21.15.141:3300/0 conn(0x7fa0a815d590 0x7fa0a80cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f38bb400640 1 -- 172.21.15.141:0/1460037436 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4034440 con 0x7f38bc160920 2024-01-16T14:57:14.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6920c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c06c470 0x7f691c148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6921600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c15ce90 0x7f691c145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6921600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c15ce90 0x7f691c145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48486/0 (socket says 172.21.15.141:48486) 2024-01-16T14:57:14.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.585+0000 7f6921600640 1 -- 172.21.15.141:0/3434667047 learned_addr learned my addr 172.21.15.141:0/3434667047 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7fefe6a00640 1 --2- 172.21.15.141:0/1965482302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80d00b0 0x7fefe80ce1b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe809cc70 tx=0x7fefd802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76ae6e5b91139d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f38c2000640 1 -- 172.21.15.141:0/1460037436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc160920 msgr2=0x7f38bc0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f38c2000640 1 --2- 172.21.15.141:0/1460037436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc160920 0x7f38bc0b3f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38bc08c060 tx=0x7f38b402bd00 comp rx=0 tx=0).stop 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f38c2000640 1 -- 172.21.15.141:0/1460037436 shutdown_connections 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f38c2000640 1 --2- 172.21.15.141:0/1460037436 >> v2:172.21.15.141:3300/0 conn(0x7f38bc0cfea0 0x7f38bc161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f38c2000640 1 --2- 172.21.15.141:0/1460037436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc160920 0x7f38bc0b3f30 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38bc08c060 tx=0x7f38b402bd00 comp rx=0 tx=0).stop 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f38c2000640 1 -- 172.21.15.141:0/1460037436 >> 172.21.15.141:0/1460037436 conn(0x7f38bc0d3090 msgr2=0x7f38bc0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7fa0a7400640 1 -- 172.21.15.141:0/141791028 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899093104 0 0) 0x7fa0a80b6750 con 0x7fa0a80d00b0 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7fa0ad600640 1 --2- 172.21.15.141:0/141791028 >> v2:172.21.15.188:3300/0 conn(0x7fa0a80ce8c0 0x7fa0a80b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7fa0a7400640 1 -- 172.21.15.141:0/141791028 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa088003520 con 0x7fa0a80d00b0 2024-01-16T14:57:14.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f9596a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f959006e3b0 0x7f9590153a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f9596a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9590159e20 con 0x7f95901598d0 2024-01-16T14:57:14.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f9596a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959015a100 con 0x7f95900cf3c0 2024-01-16T14:57:14.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f9596a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95900b31c0 con 0x7f959006e3b0 2024-01-16T14:57:14.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f7a28c00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a040047c0 0x7f7a241388a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00d7d0 tx=0x7f7a0c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f6921600640 1 -- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c06c470 msgr2=0x7f691c148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f6921600640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c06c470 0x7f691c148920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f691be00640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f691c153950 0x7f691c15aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.586+0000 7f6921600640 1 -- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f691c153950 msgr2=0x7f691c15aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f6921600640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f691c153950 0x7f691c15aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f6921600640 1 -- 172.21.15.141:0/3434667047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6914009a70 con 0x7f691c15ce90 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f691be00640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f691c153950 0x7f691c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:14.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f38c2000640 1 -- 172.21.15.141:0/1460037436 shutdown_connections 2024-01-16T14:57:14.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7fefe6000640 1 -- 172.21.15.141:0/1965482302 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8031890 con 0x7fefe80d00b0 2024-01-16T14:57:14.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7fefe6000640 1 -- 172.21.15.141:0/1965482302 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd80319f0 con 0x7fefe80d00b0 2024-01-16T14:57:14.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7fefe6000640 1 -- 172.21.15.141:0/1965482302 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8031d80 con 0x7fefe80d00b0 2024-01-16T14:57:14.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7fa0acc00640 1 -- 172.21.15.141:0/141791028 >> v2:172.21.15.188:3300/0 conn(0x7fa0a80ce8c0 msgr2=0x7fa0a80b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7fa0acc00640 1 --2- 172.21.15.141:0/141791028 >> v2:172.21.15.188:3300/0 conn(0x7fa0a80ce8c0 0x7fa0a80b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7fa0acc00640 1 -- 172.21.15.141:0/141791028 >> v1:172.21.15.141:6789/0 conn(0x7fa0a80d00b0 legacy=0x7fa0a80ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7fa0acc00640 1 -- 172.21.15.141:0/141791028 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80cef10 con 0x7fa0a815d590 2024-01-16T14:57:14.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f958fe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f959006e3b0 0x7f9590153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f38c2000640 1 -- 172.21.15.141:0/1460037436 wait complete. 2024-01-16T14:57:14.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.587+0000 7f7a20c00640 1 -- 172.21.15.141:0/2682322285 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c017070 con 0x7f7a040047c0 2024-01-16T14:57:14.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f7a20c00640 1 -- 172.21.15.141:0/2682322285 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c005070 con 0x7f7a040047c0 2024-01-16T14:57:14.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f7a20c00640 1 -- 172.21.15.141:0/2682322285 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c005440 con 0x7f7a040047c0 2024-01-16T14:57:14.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2415c870 con 0x7f7a040047c0 2024-01-16T14:57:14.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f958fe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f959006e3b0 0x7f9590153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36462/0 (socket says 172.21.15.141:36462) 2024-01-16T14:57:14.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f958fe00640 1 -- 172.21.15.141:0/250806423 learned_addr learned my addr 172.21.15.141:0/250806423 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2415ce30 con 0x7f7a040047c0 2024-01-16T14:57:14.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7fefee200640 1 -- 172.21.15.141:0/1965482302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80d00b0 msgr2=0x7fefe80ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7fefee200640 1 --2- 172.21.15.141:0/1965482302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80d00b0 0x7fefe80ce1b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe809cc70 tx=0x7fefd802b930 comp rx=0 tx=0).stop 2024-01-16T14:57:14.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7fefee200640 1 -- 172.21.15.141:0/1965482302 shutdown_connections 2024-01-16T14:57:14.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.588+0000 7f6921600640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c15ce90 0x7f691c145c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6914009eb0 tx=0x7f6914004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7fefee200640 1 --2- 172.21.15.141:0/1965482302 >> v2:172.21.15.188:3300/0 conn(0x7fefe80ce6f0 0x7fefe80d0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7fefee200640 1 --2- 172.21.15.141:0/1965482302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80d00b0 0x7fefe80ce1b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7fefee200640 1 --2- 172.21.15.141:0/1965482302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe80cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7fefee200640 1 -- 172.21.15.141:0/1965482302 >> 172.21.15.141:0/1965482302 conn(0x7fefe809c540 msgr2=0x7fefe80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f6918c00640 1 -- 172.21.15.141:0/3434667047 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691403e610 con 0x7f691c15ce90 2024-01-16T14:57:14.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f6918c00640 1 -- 172.21.15.141:0/3434667047 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691403e770 con 0x7f691c15ce90 2024-01-16T14:57:14.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f6918c00640 1 -- 172.21.15.141:0/3434667047 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691403eb00 con 0x7f691c15ce90 2024-01-16T14:57:14.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f6923600640 1 -- 172.21.15.141:0/3434667047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c15c650 con 0x7f691c15ce90 2024-01-16T14:57:14.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f958fe00640 1 -- 172.21.15.141:0/250806423 >> v1:172.21.15.188:6789/0 conn(0x7f95900cf3c0 legacy=0x7f9590070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.590+0000 7fefee200640 1 -- 172.21.15.141:0/1965482302 shutdown_connections 2024-01-16T14:57:14.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.590+0000 7f6923600640 1 -- 172.21.15.141:0/3434667047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c0cd600 con 0x7f691c15ce90 2024-01-16T14:57:14.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f958fe00640 1 -- 172.21.15.141:0/250806423 >> v1:172.21.15.141:6789/0 conn(0x7f95901598d0 legacy=0x7f95900b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:14.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f958fe00640 1 -- 172.21.15.141:0/250806423 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959015a3e0 con 0x7f959006e3b0 2024-01-16T14:57:14.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.590+0000 7fefee200640 1 -- 172.21.15.141:0/1965482302 wait complete. 2024-01-16T14:57:14.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.590+0000 7fa0acc00640 1 --2- 172.21.15.141:0/141791028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a815d590 0x7fa0a80cfb70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80cdaf0 tx=0x7fa0a002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46eace36a766ce98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.590+0000 7f3cf6c00640 1 Processor -- start 2024-01-16T14:57:14.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.589+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f4005250 con 0x7f7a040047c0 2024-01-16T14:57:14.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.591+0000 7f3cf6c00640 1 -- start start 2024-01-16T14:57:14.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.591+0000 7f38c2000640 1 Processor -- start 2024-01-16T14:57:14.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.591+0000 7fa0a7400640 1 -- 172.21.15.141:0/141791028 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002cac0 con 0x7fa0a815d590 2024-01-16T14:57:14.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.591+0000 7fa0a7400640 1 -- 172.21.15.141:0/141791028 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a002cc20 con 0x7fa0a815d590 2024-01-16T14:57:14.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.591+0000 7fa0a7400640 1 -- 172.21.15.141:0/141791028 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0034440 con 0x7fa0a815d590 2024-01-16T14:57:14.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.591+0000 7f38c2000640 1 -- start start 2024-01-16T14:57:14.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.591+0000 7f958fe00640 1 --2- 172.21.15.141:0/250806423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f959006e3b0 0x7f9590153a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9578009870 tx=0x7f957802ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=295a3cc7a13055e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.592+0000 7f6923600640 1 -- 172.21.15.141:0/3434667047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c0675d0 con 0x7f691c15ce90 2024-01-16T14:57:14.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.592+0000 7f958ea00640 1 -- 172.21.15.141:0/250806423 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957802f840 con 0x7f959006e3b0 2024-01-16T14:57:14.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.592+0000 7f958ea00640 1 -- 172.21.15.141:0/250806423 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957802f9a0 con 0x7f959006e3b0 2024-01-16T14:57:14.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.592+0000 7f958ea00640 1 -- 172.21.15.141:0/250806423 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957802fd30 con 0x7f959006e3b0 2024-01-16T14:57:14.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.592+0000 7fa0af600640 1 -- 172.21.15.141:0/141791028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a815d590 msgr2=0x7fa0a80cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.592+0000 7fa0af600640 1 --2- 172.21.15.141:0/141791028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a815d590 0x7fa0a80cfb70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80cdaf0 tx=0x7fa0a002bd90 comp rx=0 tx=0).stop 2024-01-16T14:57:14.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f7a20c00640 1 -- 172.21.15.141:0/2682322285 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7a0c007030 con 0x7f7a040047c0 2024-01-16T14:57:14.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f38c2000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38bc0cf9f0 0x7f38bc143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f38c2000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38bc0cfea0 0x7f38bc0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f38c2000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc0ca6a0 0x7f38bc15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f38c2000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0cb2d0 con 0x7f38bc0cf9f0 2024-01-16T14:57:14.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f38c2000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc143fa0 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f38c2000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc15ca30 con 0x7f38bc0cfea0 2024-01-16T14:57:14.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7fefee200640 1 Processor -- start 2024-01-16T14:57:14.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f9596a00640 1 -- 172.21.15.141:0/250806423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f959006e3b0 msgr2=0x7f9590153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f9596a00640 1 --2- 172.21.15.141:0/250806423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f959006e3b0 0x7f9590153a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9578009870 tx=0x7f957802ba20 comp rx=0 tx=0).stop 2024-01-16T14:57:14.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7fefee200640 1 -- start start 2024-01-16T14:57:14.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f38c1600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc0ca6a0 0x7f38bc15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7fa0af600640 1 -- 172.21.15.141:0/141791028 shutdown_connections 2024-01-16T14:57:14.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7fa0af600640 1 --2- 172.21.15.141:0/141791028 >> v2:172.21.15.188:3300/0 conn(0x7fa0a80ce8c0 0x7fa0a80b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7fa0af600640 1 --2- 172.21.15.141:0/141791028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a815d590 0x7fa0a80cfb70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7fa0af600640 1 -- 172.21.15.141:0/141791028 >> 172.21.15.141:0/141791028 conn(0x7fa0a809c540 msgr2=0x7fa0a80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.593+0000 7f6918c00640 1 -- 172.21.15.141:0/3434667047 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f691403b050 con 0x7f691c15ce90 2024-01-16T14:57:14.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f38c1600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc0ca6a0 0x7f38bc15cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36470/0 (socket says 172.21.15.141:36470) 2024-01-16T14:57:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f38c1600640 1 -- 172.21.15.141:0/1432620058 learned_addr learned my addr 172.21.15.141:0/1432620058 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f3cf6c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3cf00cf9a0 0x7f3cf0070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f3cf6c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3cf0070ea0 0x7f3cf00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f3cf6c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf0071690 con 0x7f3cf014b2c0 2024-01-16T14:57:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f3cf6c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf014b6a0 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f3cf6c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf0071970 con 0x7f3cf0070ea0 2024-01-16T14:57:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7f38bbe00640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38bc0cfea0 0x7f38bc0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.594+0000 7fa0af600640 1 -- 172.21.15.141:0/141791028 shutdown_connections 2024-01-16T14:57:14.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f9596a00640 1 -- 172.21.15.141:0/250806423 shutdown_connections 2024-01-16T14:57:14.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f3cf4c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3cf0070ea0 0x7f3cf00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f9596a00640 1 --2- 172.21.15.141:0/250806423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f959006e3b0 0x7f9590153a50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f9596a00640 1 -- 172.21.15.141:0/250806423 >> 172.21.15.141:0/250806423 conn(0x7f959006a800 msgr2=0x7f959006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f9596a00640 1 -- 172.21.15.141:0/250806423 shutdown_connections 2024-01-16T14:57:14.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f3cefe00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f3cf014b2c0 0x7f3cf00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:41546/0 (socket says 172.21.15.141:41546) 2024-01-16T14:57:14.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f3cf4c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3cf0070ea0 0x7f3cf00cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36476/0 (socket says 172.21.15.141:36476) 2024-01-16T14:57:14.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f3cefe00640 1 -- 172.21.15.141:0/3337378718 learned_addr learned my addr 172.21.15.141:0/3337378718 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:14.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f38c1600640 1 -- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38bc0cfea0 msgr2=0x7f38bc0ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f38c1600640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38bc0cfea0 0x7f38bc0ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f38c1600640 1 -- 172.21.15.141:0/1432620058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38bc0cf9f0 msgr2=0x7f38bc143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f38c1600640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38bc0cf9f0 0x7f38bc143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f38c1600640 1 -- 172.21.15.141:0/1432620058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4009a70 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7fefee200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80ce6f0 0x7fefe815e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f7a20c00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79fc06b2c0 0x7f79fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f7a20c00640 1 -- 172.21.15.141:0/2682322285 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a0c089520 con 0x7f7a040047c0 2024-01-16T14:57:14.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7fa0af600640 1 -- 172.21.15.141:0/141791028 wait complete. 2024-01-16T14:57:14.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7fefee200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80d00b0 0x7fefe8070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7fefee200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe81f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7f3ceea00640 1 -- 172.21.15.141:0/3337378718 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189233817 0 0) 0x7f3cf0071690 con 0x7f3cf014b2c0 2024-01-16T14:57:14.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7f38c1600640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc0ca6a0 0x7f38bc15cec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00e810 tx=0x7f38ac00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7f3ceea00640 1 -- 172.21.15.141:0/3337378718 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd0003520 con 0x7f3cf014b2c0 2024-01-16T14:57:14.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7f3cf4c00640 1 -- 172.21.15.141:0/3337378718 >> v2:172.21.15.141:3300/0 conn(0x7f3cf00cf9a0 msgr2=0x7f3cf0070960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:57:14.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79fc06b2c0 0x7f79fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7fefee200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe815daf0 con 0x7fefe80ce6f0 2024-01-16T14:57:14.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7f3ceea00640 1 -- 172.21.15.141:0/3337378718 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4071984090 0 0) 0x7f3cd0003520 con 0x7f3cf014b2c0 2024-01-16T14:57:14.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7fefee200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe815ea80 con 0x7fefe80d00b0 2024-01-16T14:57:14.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7fefee200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe81f7b50 con 0x7fefe815d590 2024-01-16T14:57:14.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.596+0000 7fefe7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80ce6f0 0x7fefe815e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f38b8c00640 1 -- 172.21.15.141:0/1432620058 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac004030 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc143700 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f38b8c00640 1 -- 172.21.15.141:0/1432620058 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac004190 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7fefe7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80ce6f0 0x7fefe815e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48496/0 (socket says 172.21.15.141:48496) 2024-01-16T14:57:14.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7fefe7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe81f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f6918c00640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68f406b390 0x7f68f406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f38b8c00640 1 -- 172.21.15.141:0/1432620058 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac004500 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7fefe7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe81f7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:32908/0 (socket says 172.21.15.141:32908) 2024-01-16T14:57:14.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7fefe7e00640 1 -- 172.21.15.141:0/1914220785 learned_addr learned my addr 172.21.15.141:0/1914220785 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f3cef400640 1 --2- 172.21.15.141:0/3337378718 >> v2:172.21.15.141:3300/0 conn(0x7f3cf00cf9a0 0x7f3cf0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f3cf4c00640 1 --2- 172.21.15.141:0/3337378718 >> v2:172.21.15.141:3300/0 conn(0x7f3cf00cf9a0 0x7f3cf0070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f3cf4c00640 1 -- 172.21.15.141:0/3337378718 >> v1:172.21.15.188:6789/0 conn(0x7f3cf014b2c0 legacy=0x7f3cf00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f3cf4c00640 1 -- 172.21.15.141:0/3337378718 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0071c50 con 0x7f3cf0070ea0 2024-01-16T14:57:14.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7fefe6a00640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80d00b0 0x7fefe8070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.597+0000 7f6918c00640 1 -- 172.21.15.141:0/3434667047 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69140abe60 con 0x7f691c15ce90 2024-01-16T14:57:14.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.595+0000 7f9596a00640 1 -- 172.21.15.141:0/250806423 wait complete. 2024-01-16T14:57:14.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7fefe7e00640 1 -- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80d00b0 msgr2=0x7fefe8070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7fefe7e00640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80d00b0 0x7fefe8070e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7fefe7e00640 1 -- 172.21.15.141:0/1914220785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80ce6f0 msgr2=0x7fefe815e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc143cc0 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7f9596a00640 1 Processor -- start 2024-01-16T14:57:14.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7f7a23e00640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79fc06b2c0 0x7f79fc06d780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a14036000 tx=0x7f7a14030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7f9596a00640 1 -- start start 2024-01-16T14:57:14.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.598+0000 7f6920c00640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68f406b390 0x7f68f406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.599+0000 7fefe7e00640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80ce6f0 0x7fefe815e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.599+0000 7fefe7e00640 1 -- 172.21.15.141:0/1914220785 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80094d0 con 0x7fefe815d590 2024-01-16T14:57:14.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.599+0000 7fefe6a00640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80d00b0 0x7fefe8070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:14.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.599+0000 7fefe7400640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80ce6f0 0x7fefe815e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:14.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.599+0000 7fefe7e00640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe81f7060 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fefe000b990 tx=0x7fefe000be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7fa0af600640 1 Processor -- start 2024-01-16T14:57:14.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc067450 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f3cf4c00640 1 --2- 172.21.15.141:0/3337378718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf0070ea0 0x7f3cf00cc160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0165330 tx=0x7f3ce802bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43ddf639b329e555 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f6920c00640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68f406b390 0x7f68f406d850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f690c000f60 tx=0x7f690c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7fa0af600640 1 -- start start 2024-01-16T14:57:14.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f3ceea00640 1 -- 172.21.15.141:0/3337378718 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0001670 con 0x7f3cf0070ea0 2024-01-16T14:57:14.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f3ceea00640 1 -- 172.21.15.141:0/3337378718 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce802c8c0 con 0x7f3cf0070ea0 2024-01-16T14:57:14.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f3ceea00640 1 -- 172.21.15.141:0/3337378718 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce802cc30 con 0x7f3cf0070ea0 2024-01-16T14:57:14.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f9596a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f959006e3b0 0x7f959013f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.600+0000 7f9596a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95900cf3c0 0x7f959013c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f38b8c00640 1 -- 172.21.15.141:0/1432620058 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f38ac00c6d0 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f9596a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95901598d0 0x7f95900b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f9596a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9590147e90 con 0x7f95900cf3c0 2024-01-16T14:57:14.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f9596a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9590144230 con 0x7f95901598d0 2024-01-16T14:57:14.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f9596a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9590144510 con 0x7f959006e3b0 2024-01-16T14:57:14.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f958fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f959006e3b0 0x7f959013f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f9594a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95901598d0 0x7f95900b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f3cf6c00640 1 -- 172.21.15.141:0/3337378718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf0070ea0 msgr2=0x7f3cf00cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/3337378718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf0070ea0 0x7f3cf00cc160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0165330 tx=0x7f3ce802bd90 comp rx=0 tx=0).stop 2024-01-16T14:57:14.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f9594a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95901598d0 0x7f95900b2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36492/0 (socket says 172.21.15.141:36492) 2024-01-16T14:57:14.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f3cf6c00640 1 -- 172.21.15.141:0/3337378718 shutdown_connections 2024-01-16T14:57:14.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/3337378718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf0070ea0 0x7f3cf00cc160 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/3337378718 >> v2:172.21.15.141:3300/0 conn(0x7f3cf00cf9a0 0x7f3cf0070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f3cf6c00640 1 -- 172.21.15.141:0/3337378718 >> 172.21.15.141:0/3337378718 conn(0x7f3cf006a800 msgr2=0x7f3cf006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f958f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95900cf3c0 0x7f959013c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.601+0000 7f9594a00640 1 -- 172.21.15.141:0/1177439842 learned_addr learned my addr 172.21.15.141:0/1177439842 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f38b8c00640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f389406b2c0 0x7f389406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f958f400640 1 -- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f959006e3b0 msgr2=0x7f959013f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f958f400640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f959006e3b0 0x7f959013f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f958f400640 1 -- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95901598d0 msgr2=0x7f95900b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f958f400640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95901598d0 0x7f95900b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f958f400640 1 -- 172.21.15.141:0/1177439842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780094d0 con 0x7f95900cf3c0 2024-01-16T14:57:14.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f38b8c00640 1 -- 172.21.15.141:0/1432620058 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38ac089700 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7fa0af600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a80ce8c0 0x7fa0a80cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f958fe00640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f959006e3b0 0x7f959013f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:14.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f3cf6c00640 1 -- 172.21.15.141:0/3337378718 shutdown_connections 2024-01-16T14:57:14.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f958f400640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95900cf3c0 0x7f959013c7e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f958000bd50 tx=0x7f958000be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7f3cf6c00640 1 -- 172.21.15.141:0/3337378718 wait complete. 2024-01-16T14:57:14.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7f6918c00640 1 -- 172.21.15.141:0/3434667047 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6914076d40 con 0x7f691c15ce90 2024-01-16T14:57:14.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7f38c0c00640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f389406b2c0 0x7f389406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.602+0000 7fa0af600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0a80d00b0 0x7fa0a815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0acc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a80ce8c0 0x7fa0a80cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7f9587e00640 1 -- 172.21.15.141:0/1177439842 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000ce90 con 0x7f95900cf3c0 2024-01-16T14:57:14.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7f9587e00640 1 -- 172.21.15.141:0/1177439842 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9580014440 con 0x7f95900cf3c0 2024-01-16T14:57:14.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7f9587e00640 1 -- 172.21.15.141:0/1177439842 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580014810 con 0x7f95900cf3c0 2024-01-16T14:57:14.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7f9596a00640 1 -- 172.21.15.141:0/1177439842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95901447f0 con 0x7f95900cf3c0 2024-01-16T14:57:14.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0acc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a80ce8c0 0x7fa0a80cc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48538/0 (socket says 172.21.15.141:48538) 2024-01-16T14:57:14.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0af600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0a815d590 0x7fa0a815c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0acc00640 1 -- 172.21.15.141:0/1036131654 learned_addr learned my addr 172.21.15.141:0/1036131654 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a806e440 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a806d830 con 0x7fa0a80d00b0 2024-01-16T14:57:14.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a806db10 con 0x7fa0a815d590 2024-01-16T14:57:14.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7f9596a00640 1 -- 172.21.15.141:0/1177439842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9590145f20 con 0x7f95900cf3c0 2024-01-16T14:57:14.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.603+0000 7fa0a7e00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0a80d00b0 0x7fa0a815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7fa0acc00640 1 -- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0a815d590 msgr2=0x7fa0a815c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7fa0acc00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0a815d590 0x7fa0a815c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7fa0acc00640 1 -- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0a80d00b0 msgr2=0x7fa0a815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7fa0acc00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0a80d00b0 0x7fa0a815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7fa0acc00640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a0009a70 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7fa0a7e00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0a80d00b0 0x7fa0a815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.604+0000 7fa0acc00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a80ce8c0 0x7fa0a80cc720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009e80 tx=0x7fa0a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.605+0000 7fa0a4c00640 1 -- 172.21.15.141:0/1036131654 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a003e610 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.605+0000 7fa0a4c00640 1 -- 172.21.15.141:0/1036131654 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a003e770 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.605+0000 7fa0a4c00640 1 -- 172.21.15.141:0/1036131654 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a003eae0 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.605+0000 7f3cf6c00640 1 Processor -- start 2024-01-16T14:57:14.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.605+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a806ddf0 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.605+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a80ccc60 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.605+0000 7f9587e00640 1 -- 172.21.15.141:0/1177439842 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f958001d3b0 con 0x7f95900cf3c0 2024-01-16T14:57:14.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.606+0000 7f9596a00640 1 -- 172.21.15.141:0/1177439842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9590147bf0 con 0x7f95900cf3c0 2024-01-16T14:57:14.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.606+0000 7f9587e00640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f956806b390 0x7f956806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.606+0000 7f9587e00640 1 -- 172.21.15.141:0/1177439842 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9580089be0 con 0x7f95900cf3c0 2024-01-16T14:57:14.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.607+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a80675d0 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.607+0000 7f958fe00640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f956806b390 0x7f956806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.607+0000 7f38c0c00640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f389406b2c0 0x7f389406d780 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f38b4036000 tx=0x7f38b402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.606+0000 7f3cf6c00640 1 -- start start 2024-01-16T14:57:14.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f958fe00640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f956806b390 0x7f956806d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9578036000 tx=0x7f9578008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.607+0000 7f3cf6c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3cf0070ea0 0x7f3cf0163010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.607+0000 7f3cf6c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3cf00cf9a0 0x7f3cf01628f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.607+0000 7f3cf6c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf014b2c0 0x7f3cf0164770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.607+0000 7f3cf6c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf0162e30 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cf6c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf01642e0 con 0x7f3cf014b2c0 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cf6c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf01636d0 con 0x7f3cf0070ea0 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cefe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3cf0070ea0 0x7f3cf0163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cf4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf014b2c0 0x7f3cf0164770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cf4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf014b2c0 0x7f3cf0164770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36506/0 (socket says 172.21.15.141:36506) 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cf4c00640 1 -- 172.21.15.141:0/2602749512 learned_addr learned my addr 172.21.15.141:0/2602749512 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cef400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3cf00cf9a0 0x7f3cf01628f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cef400640 1 -- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3cf0070ea0 msgr2=0x7f3cf0163010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cef400640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3cf0070ea0 0x7f3cf0163010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cef400640 1 -- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf014b2c0 msgr2=0x7f3cf0164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.608+0000 7f3cef400640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf014b2c0 0x7f3cf0164770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.609+0000 7f3cef400640 1 -- 172.21.15.141:0/2602749512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8009a70 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.609+0000 7f3cef400640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3cf00cf9a0 0x7f3cf01628f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c5e0 tx=0x7f3cd800caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.612+0000 7f3ce7e00640 1 -- 172.21.15.141:0/2602749512 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8004220 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.612+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf01639b0 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.613+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf0163f70 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.612+0000 7f9587e00640 1 -- 172.21.15.141:0/1177439842 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9590147bf0 con 0x7f95900cf3c0 2024-01-16T14:57:14.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.613+0000 7fa0a4c00640 1 -- 172.21.15.141:0/1036131654 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa0a003b050 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.614+0000 7fdbf3e00640 1 Processor -- start 2024-01-16T14:57:14.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.614+0000 7fa0a4c00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa08006b390 0x7fa08006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.614+0000 7fa0a4c00640 1 -- 172.21.15.141:0/1036131654 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0a00ac970 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.614+0000 7fdbf3e00640 1 -- start start 2024-01-16T14:57:14.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.615+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf0164470 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.615+0000 7fa0a7e00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa08006b390 0x7fa08006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.615+0000 7fefdf400640 1 -- 172.21.15.141:0/1914220785 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe000cba0 con 0x7fefe815d590 2024-01-16T14:57:14.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.615+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe81f7e30 con 0x7fefe815d590 2024-01-16T14:57:14.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.615+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe815e020 con 0x7fefe815d590 2024-01-16T14:57:14.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.615+0000 7fefdf400640 1 -- 172.21.15.141:0/1914220785 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe000cd00 con 0x7fefe815d590 2024-01-16T14:57:14.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.615+0000 7fefdf400640 1 -- 172.21.15.141:0/1914220785 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0014680 con 0x7fefe815d590 2024-01-16T14:57:14.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.616+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.616+0000 7fdbf3e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec0b3400 con 0x7fdbec0ced60 2024-01-16T14:57:14.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.616+0000 7fdbf3e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec0b27f0 con 0x7fdbec0d1ef0 2024-01-16T14:57:14.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.616+0000 7f7a20c00640 1 -- 172.21.15.141:0/2682322285 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a0c0542f0 con 0x7f7a040047c0 2024-01-16T14:57:14.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.616+0000 7fdbf3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec0d1390 con 0x7fdbec15d610 2024-01-16T14:57:14.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.616+0000 7fdbf1400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.617+0000 7fdbf0a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdbec0ced60 0x7fdbec0d17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40058/0 (socket says 172.21.15.141:40058) 2024-01-16T14:57:14.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.617+0000 7fdbf0a00640 1 -- 172.21.15.141:0/4068325500 learned_addr learned my addr 172.21.15.141:0/4068325500 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:14.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.618+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe81f7fc0 con 0x7fefe815d590 2024-01-16T14:57:14.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.618+0000 7fa0a7e00640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa08006b390 0x7fa08006d850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80a0a10 tx=0x7fa098009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.618+0000 7fdbf1400640 1 -- 172.21.15.141:0/4068325500 >> v1:172.21.15.188:6789/0 conn(0x7fdbec0d1ef0 legacy=0x7fdbec0b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.619+0000 7fdbf1400640 1 -- 172.21.15.141:0/4068325500 >> v1:172.21.15.141:6789/0 conn(0x7fdbec0ced60 legacy=0x7fdbec0d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.619+0000 7fdbf1400640 1 -- 172.21.15.141:0/4068325500 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec0b2ad0 con 0x7fdbec15d610 2024-01-16T14:57:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.619+0000 7fefdf400640 1 -- 172.21.15.141:0/1914220785 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fefe00147e0 con 0x7fefe815d590 2024-01-16T14:57:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.620+0000 7f3ce7e00640 1 -- 172.21.15.141:0/2602749512 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd800a390 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.620+0000 7f3ce7e00640 1 -- 172.21.15.141:0/2602749512 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8003610 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.620+0000 7f38b8c00640 1 -- 172.21.15.141:0/1432620058 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38ac054630 con 0x7f38bc0ca6a0 2024-01-16T14:57:14.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.621+0000 7fa0a4c00640 1 -- 172.21.15.141:0/1036131654 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0a0077850 con 0x7fa0a80ce8c0 2024-01-16T14:57:14.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.621+0000 7fefdf400640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefc006b310 0x7fefc006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.621+0000 7fefdf400640 1 -- 172.21.15.141:0/1914220785 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefe0089ae0 con 0x7fefe815d590 2024-01-16T14:57:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.620+0000 7fdbf1400640 1 --2- 172.21.15.141:0/4068325500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0ce820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc0097d0 tx=0x7fdbdc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab679d099168d10 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.622+0000 7fefe7400640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefc006b310 0x7fefc006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.623+0000 7f3ce7e00640 1 -- 172.21.15.141:0/2602749512 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3cd8003770 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.623+0000 7fefe7400640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefc006b310 0x7fefc006d7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fefd0006fd0 tx=0x7fefd0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.624+0000 7f3ce7e00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3cc806b2c0 0x7f3cc806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.624+0000 7f3ce7e00640 1 -- 172.21.15.141:0/2602749512 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cd8089ce0 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.624+0000 7f3cefe00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3cc806b2c0 0x7f3cc806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.625+0000 7f3cefe00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3cc806b2c0 0x7f3cc806d780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001eb0 tx=0x7f3ce00019b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.629+0000 7f3ce7e00640 1 -- 172.21.15.141:0/2602749512 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cd8054bc0 con 0x7f3cf00cf9a0 2024-01-16T14:57:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.632+0000 7fdbebe00640 1 -- 172.21.15.141:0/4068325500 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc031890 con 0x7fdbec15d610 2024-01-16T14:57:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.633+0000 7fdbebe00640 1 -- 172.21.15.141:0/4068325500 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc0319f0 con 0x7fdbec15d610 2024-01-16T14:57:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.633+0000 7fdbebe00640 1 -- 172.21.15.141:0/4068325500 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc031d80 con 0x7fdbec15d610 2024-01-16T14:57:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.633+0000 7fefdf400640 1 -- 172.21.15.141:0/1914220785 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefe0054a70 con 0x7fefe815d590 2024-01-16T14:57:14.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.636+0000 7fdbf3e00640 1 -- 172.21.15.141:0/4068325500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 msgr2=0x7fdbec0ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.636+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/4068325500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0ce820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc0097d0 tx=0x7fdbdc02b930 comp rx=0 tx=0).stop 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.638+0000 7fdbf3e00640 1 -- 172.21.15.141:0/4068325500 shutdown_connections 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.638+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/4068325500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0ce820 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.638+0000 7fdbf3e00640 1 -- 172.21.15.141:0/4068325500 >> 172.21.15.141:0/4068325500 conn(0x7fdbec09c540 msgr2=0x7fdbec0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.638+0000 7fdbf3e00640 1 -- 172.21.15.141:0/4068325500 shutdown_connections 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.638+0000 7fdbf3e00640 1 -- 172.21.15.141:0/4068325500 wait complete. 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.640+0000 7fdbf3e00640 1 Processor -- start 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.644+0000 7fdbf3e00640 1 -- start start 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.645+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbec0ced60 0x7fdbec150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.646+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbec0d1ef0 0x7fdbec0b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.646+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.646+0000 7fdbf3e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec14e410 con 0x7fdbec0ced60 2024-01-16T14:57:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.646+0000 7fdbf3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec14d4f0 con 0x7fdbec0d1ef0 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.646+0000 7fdbf3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec14d790 con 0x7fdbec15d610 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.647+0000 7fdbf1400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbec0ced60 0x7fdbec150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.647+0000 7fdbf0a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbec0d1ef0 0x7fdbec0b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.647+0000 7fdbf0a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbec0d1ef0 0x7fdbec0b80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36508/0 (socket says 172.21.15.141:36508) 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.647+0000 7fdbf0a00640 1 -- 172.21.15.141:0/2050747424 learned_addr learned my addr 172.21.15.141:0/2050747424 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.647+0000 7fdbf1e00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.648+0000 7fdbf1400640 1 -- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 msgr2=0x7fdbec0b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.648+0000 7fdbf1400640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.648+0000 7fdbf1400640 1 -- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbec0d1ef0 msgr2=0x7fdbec0b80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.648+0000 7fdbf1400640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbec0d1ef0 0x7fdbec0b80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.648+0000 7fdbf1400640 1 -- 172.21.15.141:0/2050747424 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc009400 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.652+0000 7fdbf0a00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbec0d1ef0 0x7fdbec0b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.652+0000 7fdbf1400640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbec0ced60 0x7fdbec150a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc033000 tx=0x7fdbdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.653+0000 7fdbe9600640 1 -- 172.21.15.141:0/2050747424 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc044600 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.653+0000 7fdbe9600640 1 -- 172.21.15.141:0/2050747424 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc044780 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.653+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbec14da70 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.653+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbec14e030 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.653+0000 7fdbe9600640 1 -- 172.21.15.141:0/2050747424 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc044b30 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.656+0000 7fdbe9600640 1 -- 172.21.15.141:0/2050747424 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdbdc0303e0 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.658+0000 7fdbe9600640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbc406b390 0x7fdbc406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.658+0000 7fdbe9600640 1 -- 172.21.15.141:0/2050747424 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbdc0ac9c0 con 0x7fdbec0ced60 2024-01-16T14:57:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.659+0000 7fdbf0a00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbc406b390 0x7fdbc406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:14.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.659+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc005250 con 0x7fdbec0ced60 2024-01-16T14:57:14.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.668+0000 7fdbf0a00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbc406b390 0x7fdbc406d850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b4f60 tx=0x7fdbd4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:14.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.669+0000 7fdbe9600640 1 -- 172.21.15.141:0/2050747424 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbdc035080 con 0x7fdbec0ced60 2024-01-16T14:57:14.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.993+0000 7f6923600640 1 -- 172.21.15.141:0/3434667047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f691c0b43c0 con 0x7f691c15ce90 2024-01-16T14:57:14.980 INFO:teuthology.orchestra.run.smithi141.stdout:60129542147 2024-01-16T14:57:14.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.994+0000 7f6918c00640 1 -- 172.21.15.141:0/3434667047 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6914039090 con 0x7f691c15ce90 2024-01-16T14:57:14.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.999+0000 7f6912000640 1 -- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68f406b390 msgr2=0x7f68f406d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:14.999+0000 7f6912000640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68f406b390 0x7f68f406d850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f690c000f60 tx=0x7f690c009040 comp rx=0 tx=0).stop 2024-01-16T14:57:14.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.000+0000 7f6912000640 1 -- 172.21.15.141:0/3434667047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c15ce90 msgr2=0x7f691c145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:14.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.000+0000 7f6912000640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c15ce90 0x7f691c145c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6914009eb0 tx=0x7f6914004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:14.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.000+0000 7f6912000640 1 -- 172.21.15.141:0/3434667047 shutdown_connections 2024-01-16T14:57:14.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.001+0000 7f6912000640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68f406b390 0x7f68f406d850 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.001+0000 7f6912000640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f691c15ce90 0x7f691c145c00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.001+0000 7f6912000640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f691c153950 0x7f691c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.001+0000 7f6912000640 1 --2- 172.21.15.141:0/3434667047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f691c06c470 0x7f691c148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:14.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.001+0000 7f6912000640 1 -- 172.21.15.141:0/3434667047 >> 172.21.15.141:0/3434667047 conn(0x7f691c0b0be0 msgr2=0x7f691c14c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:14.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.001+0000 7f6912000640 1 -- 172.21.15.141:0/3434667047 shutdown_connections 2024-01-16T14:57:14.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.002+0000 7f6912000640 1 -- 172.21.15.141:0/3434667047 wait complete. 2024-01-16T14:57:14.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.009+0000 7f9596a00640 1 -- 172.21.15.141:0/1177439842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9590147fd0 con 0x7f95900cf3c0 2024-01-16T14:57:14.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.012+0000 7f9587e00640 1 -- 172.21.15.141:0/1177439842 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9590147fd0 con 0x7f95900cf3c0 2024-01-16T14:57:14.997 INFO:teuthology.orchestra.run.smithi141.stdout:60129542147 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.018+0000 7f9585600640 1 -- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f956806b390 msgr2=0x7f956806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.018+0000 7f9585600640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f956806b390 0x7f956806d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9578036000 tx=0x7f9578008030 comp rx=0 tx=0).stop 2024-01-16T14:57:15.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.018+0000 7f9585600640 1 -- 172.21.15.141:0/1177439842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95900cf3c0 msgr2=0x7f959013c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.018+0000 7f9585600640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95900cf3c0 0x7f959013c7e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f958000bd50 tx=0x7f958000be40 comp rx=0 tx=0).stop 2024-01-16T14:57:15.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.019+0000 7f9585600640 1 -- 172.21.15.141:0/1177439842 shutdown_connections 2024-01-16T14:57:15.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.019+0000 7f9585600640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f956806b390 0x7f956806d850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.019+0000 7f9585600640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95901598d0 0x7f95900b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.019+0000 7f9585600640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95900cf3c0 0x7f959013c7e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.019+0000 7f9585600640 1 --2- 172.21.15.141:0/1177439842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f959006e3b0 0x7f959013f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.019+0000 7f9585600640 1 -- 172.21.15.141:0/1177439842 >> 172.21.15.141:0/1177439842 conn(0x7f959006a800 msgr2=0x7f9590152c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.020+0000 7f9585600640 1 -- 172.21.15.141:0/1177439842 shutdown_connections 2024-01-16T14:57:15.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.020+0000 7f9585600640 1 -- 172.21.15.141:0/1177439842 wait complete. 2024-01-16T14:57:15.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.1 2024-01-16T14:57:15.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.033+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f79f4005500 con 0x7f7a040047c0 2024-01-16T14:57:15.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.034+0000 7f7a20c00640 1 -- 172.21.15.141:0/2682322285 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a0c057910 con 0x7f7a040047c0 2024-01-16T14:57:15.017 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:15.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.036+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f38bc15cb70 con 0x7f38bc0ca6a0 2024-01-16T14:57:15.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.036+0000 7f38b8c00640 1 -- 172.21.15.141:0/1432620058 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f38ac057c50 con 0x7f38bc0ca6a0 2024-01-16T14:57:15.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.037+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3cf00675d0 con 0x7f3cf00cf9a0 2024-01-16T14:57:15.020 INFO:teuthology.orchestra.run.smithi141.stdout:60129542147 2024-01-16T14:57:15.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.037+0000 7f3ce7e00640 1 -- 172.21.15.141:0/2602749512 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3cd801e090 con 0x7f3cf00cf9a0 2024-01-16T14:57:15.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79fc06b2c0 msgr2=0x7f79fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79fc06b2c0 0x7f79fc06d780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a14036000 tx=0x7f7a14030000 comp rx=0 tx=0).stop 2024-01-16T14:57:15.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a040047c0 msgr2=0x7f7a241388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a040047c0 0x7f7a241388a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00d7d0 tx=0x7f7a0c00dc90 comp rx=0 tx=0).stop 2024-01-16T14:57:15.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 shutdown_connections 2024-01-16T14:57:15.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79fc06b2c0 0x7f79fc06d780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a240b27d0 0x7f7a240aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a240b2370 0x7f7a240aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 --2- 172.21.15.141:0/2682322285 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a040047c0 0x7f7a241388a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.038+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 >> 172.21.15.141:0/2682322285 conn(0x7f7a2406a800 msgr2=0x7f7a2414a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.022 INFO:teuthology.orchestra.run.smithi141.stdout:60129542147 2024-01-16T14:57:15.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.039+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 shutdown_connections 2024-01-16T14:57:15.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.039+0000 7f7a2b600640 1 -- 172.21.15.141:0/2682322285 wait complete. 2024-01-16T14:57:15.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.039+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fefe815e210 con 0x7fefe815d590 2024-01-16T14:57:15.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f389406b2c0 msgr2=0x7f389406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f389406b2c0 0x7f389406d780 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f38b4036000 tx=0x7f38b402e000 comp rx=0 tx=0).stop 2024-01-16T14:57:15.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc0ca6a0 msgr2=0x7f38bc15cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc0ca6a0 0x7f38bc15cec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00e810 tx=0x7f38ac00ecd0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7fefdf400640 1 -- 172.21.15.141:0/1914220785 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fefe0019020 con 0x7fefe815d590 2024-01-16T14:57:15.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 shutdown_connections 2024-01-16T14:57:15.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f389406b2c0 0x7f389406d780 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38bc0ca6a0 0x7f38bc15cec0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38bc0cfea0 0x7f38bc0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.025 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:15.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 --2- 172.21.15.141:0/1432620058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38bc0cf9f0 0x7f38bc143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 >> 172.21.15.141:0/1432620058 conn(0x7f38bc0d3090 msgr2=0x7f38bc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3cc806b2c0 msgr2=0x7f3cc806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3cc806b2c0 0x7f3cc806d780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001eb0 tx=0x7f3ce00019b0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3cf00cf9a0 msgr2=0x7f3cf01628f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3cf00cf9a0 0x7f3cf01628f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c5e0 tx=0x7f3cd800caa0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 shutdown_connections 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3cc806b2c0 0x7f3cc806d780 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3cf014b2c0 0x7f3cf0164770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3cf00cf9a0 0x7f3cf01628f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 --2- 172.21.15.141:0/2602749512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3cf0070ea0 0x7f3cf0163010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 >> 172.21.15.141:0/2602749512 conn(0x7f3cf006a800 msgr2=0x7f3cf0149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.041+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 shutdown_connections 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.042+0000 7f3cf6c00640 1 -- 172.21.15.141:0/2602749512 wait complete. 2024-01-16T14:57:15.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.042+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa0a806cb90 con 0x7fa0a80ce8c0 2024-01-16T14:57:15.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.042+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 shutdown_connections 2024-01-16T14:57:15.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.042+0000 7fa0a4c00640 1 -- 172.21.15.141:0/1036131654 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa0a0039020 con 0x7fa0a80ce8c0 2024-01-16T14:57:15.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.042+0000 7f38c2000640 1 -- 172.21.15.141:0/1432620058 wait complete. 2024-01-16T14:57:15.029 INFO:teuthology.orchestra.run.smithi141.stdout:60129542147 2024-01-16T14:57:15.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefc006b310 msgr2=0x7fefc006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefc006b310 0x7fefc006d7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fefd0006fd0 tx=0x7fefd0008040 comp rx=0 tx=0).stop 2024-01-16T14:57:15.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 msgr2=0x7fefe81f7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe81f7060 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fefe000b990 tx=0x7fefe000be50 comp rx=0 tx=0).stop 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 shutdown_connections 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefc006b310 0x7fefc006d7d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefe815d590 0x7fefe81f7060 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80d00b0 0x7fefe8070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 --2- 172.21.15.141:0/1914220785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80ce6f0 0x7fefe815e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.045+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 >> 172.21.15.141:0/1914220785 conn(0x7fefe809c540 msgr2=0x7fefe80b8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.046+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa08006b390 msgr2=0x7fa08006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.046+0000 7fa0af600640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa08006b390 0x7fa08006d850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80a0a10 tx=0x7fa098009040 comp rx=0 tx=0).stop 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.046+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a80ce8c0 msgr2=0x7fa0a80cc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.046+0000 7fa0af600640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a80ce8c0 0x7fa0a80cc720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009e80 tx=0x7fa0a0004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:15.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 shutdown_connections 2024-01-16T14:57:15.031 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.6 2024-01-16T14:57:15.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fefee200640 1 -- 172.21.15.141:0/1914220785 wait complete. 2024-01-16T14:57:15.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 shutdown_connections 2024-01-16T14:57:15.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa08006b390 0x7fa08006d850 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0a815d590 0x7fa0a815c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0a80d00b0 0x7fa0a815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 --2- 172.21.15.141:0/1036131654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0a80ce8c0 0x7fa0a80cc720 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 >> 172.21.15.141:0/1036131654 conn(0x7fa0a809c540 msgr2=0x7fa0a8160110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 shutdown_connections 2024-01-16T14:57:15.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.047+0000 7fa0af600640 1 -- 172.21.15.141:0/1036131654 wait complete. 2024-01-16T14:57:15.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.060+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdbbc005500 con 0x7fdbec0ced60 2024-01-16T14:57:15.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.061+0000 7fdbe9600640 1 -- 172.21.15.141:0/2050747424 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdbdc0779e0 con 0x7fdbec0ced60 2024-01-16T14:57:15.044 INFO:teuthology.orchestra.run.smithi141.stdout:60129542148 2024-01-16T14:57:15.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbc406b390 msgr2=0x7fdbc406d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbc406b390 0x7fdbc406d850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b4f60 tx=0x7fdbd4009040 comp rx=0 tx=0).stop 2024-01-16T14:57:15.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbec0ced60 msgr2=0x7fdbec150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:15.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbec0ced60 0x7fdbec150a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc033000 tx=0x7fdbdc0046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 shutdown_connections 2024-01-16T14:57:15.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbc406b390 0x7fdbc406d850 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbec15d610 0x7fdbec0b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbec0d1ef0 0x7fdbec0b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 --2- 172.21.15.141:0/2050747424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbec0ced60 0x7fdbec150a00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:15.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 >> 172.21.15.141:0/2050747424 conn(0x7fdbec09c540 msgr2=0x7fdbec06b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:15.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 shutdown_connections 2024-01-16T14:57:15.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:15.064+0000 7fdbf3e00640 1 -- 172.21.15.141:0/2050747424 wait complete. 2024-01-16T14:57:15.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.0 2024-01-16T14:57:15.051 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.5 2024-01-16T14:57:15.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.4 2024-01-16T14:57:15.056 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.3 2024-01-16T14:57:15.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.2 2024-01-16T14:57:15.066 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.7 2024-01-16T14:57:16.012 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T14:57:16.033 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T14:57:16.050 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T14:57:16.052 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T14:57:16.056 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T14:57:16.057 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T14:57:16.057 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T14:57:16.067 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T14:57:16.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.320+0000 7fe5f8a00640 1 Processor -- start 2024-01-16T14:57:16.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.320+0000 7fe5f8a00640 1 -- start start 2024-01-16T14:57:16.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.323+0000 7fe5f8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5e80eb020 0x7fe5e80552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.323+0000 7fe5f8a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8057df0 con 0x7fe5e80eb400 2024-01-16T14:57:16.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.323+0000 7fe5f8a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80ebf90 con 0x7fe5e8006160 2024-01-16T14:57:16.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.324+0000 7fe5f8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8057960 con 0x7fe5e80eb020 2024-01-16T14:57:16.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.324+0000 7fe5f5600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe5e80eb400 0x7fe5e80eb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40066/0 (socket says 172.21.15.141:40066) 2024-01-16T14:57:16.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.324+0000 7fe5f5600640 1 -- 172.21.15.141:0/3275174762 learned_addr learned my addr 172.21.15.141:0/3275174762 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.325+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157805196 0 0) 0x7fe5e80ebf90 con 0x7fe5e8006160 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.325+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d0003520 con 0x7fe5e8006160 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.325+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686501450 0 0) 0x7fe5e8057df0 con 0x7fe5e80eb400 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.325+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d00048a0 con 0x7fe5e80eb400 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 726821651 0 0) 0x7fe5d00048a0 con 0x7fe5e80eb400 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5e80eb020 msgr2=0x7fe5e80552c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5e80eb020 0x7fe5e80552c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 >> v1:172.21.15.188:6789/0 conn(0x7fe5e8006160 legacy=0x7fe5e80065f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80387a0 con 0x7fe5e80eb400 2024-01-16T14:57:16.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe5d8001670 con 0x7fe5e80eb400 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 >> v1:172.21.15.141:6789/0 conn(0x7fe5e80eb400 legacy=0x7fe5e80eb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5d0004d40 0x7fe5d0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0003760 0x7fe5d0003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0009970 0x7fe5d000bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8057960 con 0x7fe5d0004d40 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d000c350 con 0x7fe5d0003760 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.326+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d000c510 con 0x7fe5d0009970 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.327+0000 7fe5f5600640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0003760 0x7fe5d0003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.327+0000 7fe5f6a00640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0009970 0x7fe5d000bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.327+0000 7fe5f6000640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5d0004d40 0x7fe5d0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.328+0000 7fe5f6000640 1 -- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0009970 msgr2=0x7fe5d000bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.328+0000 7fe5f6000640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0009970 0x7fe5d000bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.328+0000 7fe5f6000640 1 -- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0003760 msgr2=0x7fe5d0003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.328+0000 7fe5f6000640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0003760 0x7fe5d0003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.328+0000 7fe5f6000640 1 -- 172.21.15.141:0/3275174762 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80387a0 con 0x7fe5d0004d40 2024-01-16T14:57:16.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.330+0000 7fe5f6000640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5d0004d40 0x7fe5d0005160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00695b0 tx=0x7fe5d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53942e4e7c88ea85 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.330+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00a67d0 con 0x7fe5d0004d40 2024-01-16T14:57:16.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.330+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f00a6970 con 0x7fe5d0004d40 2024-01-16T14:57:16.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.331+0000 7fe5f8a00640 1 -- 172.21.15.141:0/3275174762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5d0004d40 msgr2=0x7fe5d0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.331+0000 7fe5f8a00640 1 --2- 172.21.15.141:0/3275174762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5d0004d40 0x7fe5d0005160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00695b0 tx=0x7fe5d00057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.332+0000 7fe5f4c00640 1 -- 172.21.15.141:0/3275174762 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00a6da0 con 0x7fe5d0004d40 2024-01-16T14:57:16.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.332+0000 7fe5f6a00640 1 -- 172.21.15.141:0/3275174762 reap_dead start 2024-01-16T14:57:16.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.332+0000 7fe5f8a00640 1 -- 172.21.15.141:0/3275174762 shutdown_connections 2024-01-16T14:57:16.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.332+0000 7fe5f8a00640 1 -- 172.21.15.141:0/3275174762 >> 172.21.15.141:0/3275174762 conn(0x7fe5e80559a0 msgr2=0x7fe5e8055db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.333+0000 7fe5f8a00640 1 -- 172.21.15.141:0/3275174762 shutdown_connections 2024-01-16T14:57:16.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.333+0000 7fe5f8a00640 1 -- 172.21.15.141:0/3275174762 wait complete. 2024-01-16T14:57:16.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.337+0000 7fe5f8a00640 1 Processor -- start 2024-01-16T14:57:16.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.337+0000 7fe5f8a00640 1 -- start start 2024-01-16T14:57:16.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.339+0000 7fe5f8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0004d40 0x7fe5e804b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.339+0000 7fe5f8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5e80d45b0 0x7fe5e804e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.339+0000 7fe5f8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5e804ed30 0x7fe5e80d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.339+0000 7fe5f8a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e804f1e0 con 0x7fe5e804ed30 2024-01-16T14:57:16.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.339+0000 7fe5f8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80d4290 con 0x7fe5e80d45b0 2024-01-16T14:57:16.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.339+0000 7fe5f8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8052280 con 0x7fe5d0004d40 2024-01-16T14:57:16.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.340+0000 7fe5f5600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5e80d45b0 0x7fe5e804e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.340+0000 7fe5f5600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5e80d45b0 0x7fe5e804e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36524/0 (socket says 172.21.15.141:36524) 2024-01-16T14:57:16.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.341+0000 7fe5f5600640 1 -- 172.21.15.141:0/41307499 learned_addr learned my addr 172.21.15.141:0/41307499 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.341+0000 7fe5f6a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5e804ed30 0x7fe5e80d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.342+0000 7fe5f6000640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0004d40 0x7fe5e804b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.343+0000 7fe5f6a00640 1 -- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0004d40 msgr2=0x7fe5e804b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.343+0000 7fe5f6a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0004d40 0x7fe5e804b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.343+0000 7fe5f6a00640 1 -- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5e80d45b0 msgr2=0x7fe5e804e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.343+0000 7fe5f6a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5e80d45b0 0x7fe5e804e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.343+0000 7fe5f6a00640 1 -- 172.21.15.141:0/41307499 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80d3e00 con 0x7fe5e804ed30 2024-01-16T14:57:16.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.344+0000 7fe5f6a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5e804ed30 0x7fe5e80d1e80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80578d0 tx=0x7fe5e000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.344+0000 7fe5ee000640 1 -- 172.21.15.141:0/41307499 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00050d0 con 0x7fe5e804ed30 2024-01-16T14:57:16.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.345+0000 7fe5ee000640 1 -- 172.21.15.141:0/41307499 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e0005250 con 0x7fe5e804ed30 2024-01-16T14:57:16.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.345+0000 7fe5ee000640 1 -- 172.21.15.141:0/41307499 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00035f0 con 0x7fe5e804ed30 2024-01-16T14:57:16.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.345+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e80d4a30 con 0x7fe5e804ed30 2024-01-16T14:57:16.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.346+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e80d34d0 con 0x7fe5e804ed30 2024-01-16T14:57:16.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.350+0000 7fe5ee000640 1 -- 172.21.15.141:0/41307499 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe5e0003750 con 0x7fe5e804ed30 2024-01-16T14:57:16.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.350+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e80d3f90 con 0x7fe5e804ed30 2024-01-16T14:57:16.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.351+0000 7fe5ee000640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe5c806b2c0 0x7fe5c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.352+0000 7fe5f6000640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe5c806b2c0 0x7fe5c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.360+0000 7fe5ee000640 1 -- 172.21.15.141:0/41307499 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5e0058780 con 0x7fe5e804ed30 2024-01-16T14:57:16.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.361+0000 7fe5f6000640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe5c806b2c0 0x7fe5c806d780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0004740 tx=0x7fe5f006d030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.361+0000 7f1653400640 1 Processor -- start 2024-01-16T14:57:16.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.362+0000 7fe5ee000640 1 -- 172.21.15.141:0/41307499 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5e0054b50 con 0x7fe5e804ed30 2024-01-16T14:57:16.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.361+0000 7f1653400640 1 -- start start 2024-01-16T14:57:16.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.363+0000 7f1653400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f164403c070 0x7f16440e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.363+0000 7f1653400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f1644054110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.363+0000 7f1653400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1644054650 con 0x7f16440e3130 2024-01-16T14:57:16.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.363+0000 7f1653400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1644007b40 con 0x7f164403c070 2024-01-16T14:57:16.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.363+0000 7f1653400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440587e0 con 0x7f1644053c80 2024-01-16T14:57:16.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.365+0000 7f1652000640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f16440e3130 0x7f1644058020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:41576/0 (socket says 172.21.15.141:41576) 2024-01-16T14:57:16.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.365+0000 7f1652a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f1644054110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.365+0000 7f1651600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f164403c070 0x7f16440e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.365+0000 7f1652a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f1644054110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:32994/0 (socket says 172.21.15.141:32994) 2024-01-16T14:57:16.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.365+0000 7f1652a00640 1 -- 172.21.15.141:0/1155813018 learned_addr learned my addr 172.21.15.141:0/1155813018 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.366+0000 7f1650c00640 1 -- 172.21.15.141:0/1155813018 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575497594 0 0) 0x7f1644054650 con 0x7f16440e3130 2024-01-16T14:57:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.366+0000 7f1652a00640 1 -- 172.21.15.141:0/1155813018 >> v2:172.21.15.141:3300/0 conn(0x7f164403c070 msgr2=0x7f16440e7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.366+0000 7f1652a00640 1 --2- 172.21.15.141:0/1155813018 >> v2:172.21.15.141:3300/0 conn(0x7f164403c070 0x7f16440e7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.366+0000 7f1652a00640 1 -- 172.21.15.141:0/1155813018 >> v1:172.21.15.188:6789/0 conn(0x7f16440e3130 legacy=0x7f1644058020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.366+0000 7f1652a00640 1 -- 172.21.15.141:0/1155813018 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1644058ac0 con 0x7f1644053c80 2024-01-16T14:57:16.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.368+0000 7f1652a00640 1 --2- 172.21.15.141:0/1155813018 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f1644054110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f163c009870 tx=0x7f163c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dda27eb1b7d12949 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.369+0000 7f1650c00640 1 -- 172.21.15.141:0/1155813018 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c031830 con 0x7f1644053c80 2024-01-16T14:57:16.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.369+0000 7f1650c00640 1 -- 172.21.15.141:0/1155813018 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c031990 con 0x7f1644053c80 2024-01-16T14:57:16.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.369+0000 7f1650c00640 1 -- 172.21.15.141:0/1155813018 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c031d20 con 0x7f1644053c80 2024-01-16T14:57:16.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.370+0000 7f9fdde00640 1 Processor -- start 2024-01-16T14:57:16.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.370+0000 7f9fdde00640 1 -- start start 2024-01-16T14:57:16.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7f1653400640 1 -- 172.21.15.141:0/1155813018 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 msgr2=0x7f1644054110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7f1653400640 1 --2- 172.21.15.141:0/1155813018 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f1644054110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f163c009870 tx=0x7f163c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7f1653400640 1 -- 172.21.15.141:0/1155813018 shutdown_connections 2024-01-16T14:57:16.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7f1653400640 1 --2- 172.21.15.141:0/1155813018 >> v2:172.21.15.141:3300/0 conn(0x7f164403c070 0x7f16440e7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7f1653400640 1 --2- 172.21.15.141:0/1155813018 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f1644054110 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7f1653400640 1 -- 172.21.15.141:0/1155813018 >> 172.21.15.141:0/1155813018 conn(0x7f1644058eb0 msgr2=0x7f1644059290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7fb65ba00640 1 Processor -- start 2024-01-16T14:57:16.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.371+0000 7f1653400640 1 -- 172.21.15.141:0/1155813018 shutdown_connections 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.372+0000 7f9fdde00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9fd80b1620 0x7f9fd8070da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.372+0000 7f9fdde00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9fd81568f0 0x7f9fd8156d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.372+0000 7f9fdde00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd81572a0 con 0x7f9fd8071e20 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.372+0000 7f9fdde00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd8072680 con 0x7f9fd80b1620 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.372+0000 7f9fdde00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd8070300 con 0x7f9fd81568f0 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.373+0000 7f9fd6000640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f9fd8071e20 0x7f9fd815a290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:41592/0 (socket says 172.21.15.141:41592) 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.373+0000 7f9fd6000640 1 -- 172.21.15.141:0/249028552 learned_addr learned my addr 172.21.15.141:0/249028552 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.373+0000 7f9fd6a00640 1 --2- 172.21.15.141:0/249028552 >> v2:172.21.15.141:3300/0 conn(0x7f9fd80b1620 0x7f9fd8070da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.373+0000 7f9fd5600640 1 -- 172.21.15.141:0/249028552 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065415107 0 0) 0x7f9fd81572a0 con 0x7f9fd8071e20 2024-01-16T14:57:16.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.373+0000 7f9fd5600640 1 -- 172.21.15.141:0/249028552 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb8003520 con 0x7f9fd8071e20 2024-01-16T14:57:16.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7f9fd6a00640 1 -- 172.21.15.141:0/249028552 >> v2:172.21.15.188:3300/0 conn(0x7f9fd81568f0 msgr2=0x7f9fd8156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7f9fd6a00640 1 --2- 172.21.15.141:0/249028552 >> v2:172.21.15.188:3300/0 conn(0x7f9fd81568f0 0x7f9fd8156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7f9fd6a00640 1 -- 172.21.15.141:0/249028552 >> v1:172.21.15.188:6789/0 conn(0x7f9fd8071e20 legacy=0x7f9fd815a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7f9fd6a00640 1 -- 172.21.15.141:0/249028552 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd80723a0 con 0x7f9fd80b1620 2024-01-16T14:57:16.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.372+0000 7f1653400640 1 -- 172.21.15.141:0/1155813018 wait complete. 2024-01-16T14:57:16.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.372+0000 7fb65ba00640 1 -- start start 2024-01-16T14:57:16.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7fb65ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7fb65ba00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6541613e0 con 0x7fb6540b4460 2024-01-16T14:57:16.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7fb65ba00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb654161630 con 0x7fb6540d1cc0 2024-01-16T14:57:16.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.375+0000 7f1653400640 1 Processor -- start 2024-01-16T14:57:16.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.374+0000 7fb65ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb654161910 con 0x7fb65415fa90 2024-01-16T14:57:16.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.375+0000 7f1653400640 1 -- start start 2024-01-16T14:57:16.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7fb659000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7fb659000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540cde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:32996/0 (socket says 172.21.15.141:32996) 2024-01-16T14:57:16.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f1653400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f164403c070 0x7f1644009d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f1653400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f164404d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f1653400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16440e3130 0x7f164404f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f1653400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1644039a50 con 0x7f16440e3130 2024-01-16T14:57:16.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f1653400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1644039d30 con 0x7f164403c070 2024-01-16T14:57:16.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f1653400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164403a010 con 0x7f1644053c80 2024-01-16T14:57:16.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7fb653e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb6540b4460 0x7fb6540d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40076/0 (socket says 172.21.15.141:40076) 2024-01-16T14:57:16.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16440e3130 0x7f164404f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16440e3130 0x7f164404f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48586/0 (socket says 172.21.15.141:48586) 2024-01-16T14:57:16.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652a00640 1 -- 172.21.15.141:0/2233886696 learned_addr learned my addr 172.21.15.141:0/2233886696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652000640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f164403c070 0x7f1644009d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1651600640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f164404d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7fb653e00640 1 -- 172.21.15.141:0/3899215954 learned_addr learned my addr 172.21.15.141:0/3899215954 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:16.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f9fd6a00640 1 --2- 172.21.15.141:0/249028552 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd80b1620 0x7f9fd8070da0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8159850 tx=0x7f9fc802bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6478aed4ceda86 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7f9fd5600640 1 -- 172.21.15.141:0/249028552 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc802d6e0 con 0x7f9fd80b1620 2024-01-16T14:57:16.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f9fd5600640 1 -- 172.21.15.141:0/249028552 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc802d840 con 0x7f9fd80b1620 2024-01-16T14:57:16.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f9fd5600640 1 -- 172.21.15.141:0/249028552 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc802d6e0 con 0x7f9fd80b1620 2024-01-16T14:57:16.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652a00640 1 -- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 msgr2=0x7f164404d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652a00640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f164404d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652a00640 1 -- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f164403c070 msgr2=0x7f1644009d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.377+0000 7f1652a00640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f164403c070 0x7f1644009d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f1652a00640 1 -- 172.21.15.141:0/2233886696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c0094d0 con 0x7f16440e3130 2024-01-16T14:57:16.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f9fdde00640 1 -- 172.21.15.141:0/249028552 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd80b1620 msgr2=0x7f9fd8070da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f9fdde00640 1 --2- 172.21.15.141:0/249028552 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd80b1620 0x7f9fd8070da0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8159850 tx=0x7f9fc802bb30 comp rx=0 tx=0).stop 2024-01-16T14:57:16.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f1652a00640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16440e3130 0x7f164404f360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f163c009870 tx=0x7f163c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.376+0000 7fb659a00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fb6540d1cc0 0x7fb6540cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:41606/0 (socket says 172.21.15.141:41606) 2024-01-16T14:57:16.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7fb653400640 1 -- 172.21.15.141:0/3899215954 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968377546 0 0) 0x7fb6541613e0 con 0x7fb6540b4460 2024-01-16T14:57:16.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f9fdde00640 1 -- 172.21.15.141:0/249028552 shutdown_connections 2024-01-16T14:57:16.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f9fdde00640 1 --2- 172.21.15.141:0/249028552 >> v2:172.21.15.188:3300/0 conn(0x7f9fd81568f0 0x7f9fd8156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f9fdde00640 1 --2- 172.21.15.141:0/249028552 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd80b1620 0x7f9fd8070da0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.378+0000 7f9fdde00640 1 -- 172.21.15.141:0/249028552 >> 172.21.15.141:0/249028552 conn(0x7f9fd80b0b00 msgr2=0x7f9fd80b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.379+0000 7f9fdde00640 1 -- 172.21.15.141:0/249028552 shutdown_connections 2024-01-16T14:57:16.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.379+0000 7f164a000640 1 -- 172.21.15.141:0/2233886696 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c03a830 con 0x7f16440e3130 2024-01-16T14:57:16.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.379+0000 7fb653400640 1 -- 172.21.15.141:0/3899215954 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634003520 con 0x7fb6540b4460 2024-01-16T14:57:16.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.380+0000 7fb659000640 1 -- 172.21.15.141:0/3899215954 >> v1:172.21.15.188:6789/0 conn(0x7fb6540d1cc0 legacy=0x7fb6540cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.380+0000 7fb659000640 1 -- 172.21.15.141:0/3899215954 >> v1:172.21.15.141:6789/0 conn(0x7fb6540b4460 legacy=0x7fb6540d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.380+0000 7fb659000640 1 -- 172.21.15.141:0/3899215954 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb654161bf0 con 0x7fb65415fa90 2024-01-16T14:57:16.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.380+0000 7fb653400640 1 -- 172.21.15.141:0/3899215954 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121336971 0 0) 0x7fb654161630 con 0x7fb6540d1cc0 2024-01-16T14:57:16.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.379+0000 7f9fdde00640 1 -- 172.21.15.141:0/249028552 wait complete. 2024-01-16T14:57:16.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.381+0000 7f164a000640 1 -- 172.21.15.141:0/2233886696 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c03a9b0 con 0x7f16440e3130 2024-01-16T14:57:16.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.381+0000 7f164a000640 1 -- 172.21.15.141:0/2233886696 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c03ad60 con 0x7f16440e3130 2024-01-16T14:57:16.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.381+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164403a2f0 con 0x7f16440e3130 2024-01-16T14:57:16.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.381+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164404d930 con 0x7f16440e3130 2024-01-16T14:57:16.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.381+0000 7f9fdde00640 1 Processor -- start 2024-01-16T14:57:16.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.381+0000 7f9fdde00640 1 -- start start 2024-01-16T14:57:16.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.382+0000 7fb659000640 1 --2- 172.21.15.141:0/3899215954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540cde90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009dc0 tx=0x7fb64c02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3053d9c0e0ae7d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.382+0000 7ffadb400640 1 Processor -- start 2024-01-16T14:57:16.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.382+0000 7ffadb400640 1 -- start start 2024-01-16T14:57:16.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.383+0000 7f9fdde00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fd8071e20 0x7f9fd8143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.383+0000 7f9fdde00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fd80b1620 0x7f9fd81689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.383+0000 7f9fdde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd81568f0 0x7f9fd816af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7fb653400640 1 -- 172.21.15.141:0/3899215954 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c02cd50 con 0x7fb65415fa90 2024-01-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7f9fdde00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8167700 con 0x7f9fd81568f0 2024-01-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7fb653400640 1 -- 172.21.15.141:0/3899215954 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c034440 con 0x7fb65415fa90 2024-01-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7fb653400640 1 -- 172.21.15.141:0/3899215954 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c0347d0 con 0x7fb65415fa90 2024-01-16T14:57:16.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7f9fdde00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8168520 con 0x7f9fd80b1620 2024-01-16T14:57:16.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7f9fdde00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8143f30 con 0x7f9fd8071e20 2024-01-16T14:57:16.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164403a810 con 0x7f16440e3130 2024-01-16T14:57:16.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7f9fd6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fd80b1620 0x7f9fd81689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7f9fd6a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fd8071e20 0x7f9fd8143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7ffadb400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40b4510 0x7ffad40d3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7ffadb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ffad40d1700 0x7ffad40b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.384+0000 7ffadb400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40d3f00 con 0x7ffad40b48f0 2024-01-16T14:57:16.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7ffadb400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad40ce8d0 con 0x7ffad40d1700 2024-01-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7ffadb400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40cebb0 con 0x7ffad40b4510 2024-01-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7ffad8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40b4510 0x7ffad40d3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7ffad8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40b4510 0x7ffad40d3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33028/0 (socket says 172.21.15.141:33028) 2024-01-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7fb65ba00640 1 -- 172.21.15.141:0/3899215954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 msgr2=0x7fb6540cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7ffad8a00640 1 -- 172.21.15.141:0/4018123299 learned_addr learned my addr 172.21.15.141:0/4018123299 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7fb48d000640 1 Processor -- start 2024-01-16T14:57:16.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7fb48d000640 1 -- start start 2024-01-16T14:57:16.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7f9fd7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd81568f0 0x7f9fd816af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7f9fd6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fd80b1620 0x7f9fd81689b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36532/0 (socket says 172.21.15.141:36532) 2024-01-16T14:57:16.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7f9fd6000640 1 -- 172.21.15.141:0/3631927588 learned_addr learned my addr 172.21.15.141:0/3631927588 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7fb65ba00640 1 --2- 172.21.15.141:0/3899215954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540cde90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009dc0 tx=0x7fb64c02bfa0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7ffad8a00640 1 -- 172.21.15.141:0/4018123299 >> v2:172.21.15.141:3300/0 conn(0x7ffad40d1700 msgr2=0x7ffad40b60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7ffad9400640 1 --2- 172.21.15.141:0/4018123299 >> v2:172.21.15.141:3300/0 conn(0x7ffad40d1700 0x7ffad40b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7ffad8a00640 1 --2- 172.21.15.141:0/4018123299 >> v2:172.21.15.141:3300/0 conn(0x7ffad40d1700 0x7ffad40b60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7ffad8a00640 1 -- 172.21.15.141:0/4018123299 >> v1:172.21.15.188:6789/0 conn(0x7ffad40b48f0 legacy=0x7ffad40d0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7ffad8a00640 1 -- 172.21.15.141:0/4018123299 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad40cee90 con 0x7ffad40b4510 2024-01-16T14:57:16.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7f9fd6000640 1 -- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fd8071e20 msgr2=0x7f9fd8143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.385+0000 7f164a000640 1 -- 172.21.15.141:0/2233886696 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f163c04d450 con 0x7f16440e3130 2024-01-16T14:57:16.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7f9fd6000640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fd8071e20 0x7f9fd8143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.386+0000 7f9fd6000640 1 -- 172.21.15.141:0/3631927588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd81568f0 msgr2=0x7f9fd816af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7ffad9400640 1 --2- 172.21.15.141:0/4018123299 >> v2:172.21.15.141:3300/0 conn(0x7ffad40d1700 0x7ffad40b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:16.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7f5078a00640 1 Processor -- start 2024-01-16T14:57:16.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7f9fd6000640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd81568f0 0x7f9fd816af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7f9fd6000640 1 -- 172.21.15.141:0/3631927588 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc8009400 con 0x7f9fd80b1620 2024-01-16T14:57:16.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7f9fd7400640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd81568f0 0x7f9fd816af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:16.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb65ba00640 1 -- 172.21.15.141:0/3899215954 shutdown_connections 2024-01-16T14:57:16.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb65ba00640 1 --2- 172.21.15.141:0/3899215954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540cde90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb65ba00640 1 -- 172.21.15.141:0/3899215954 >> 172.21.15.141:0/3899215954 conn(0x7fb65409c540 msgr2=0x7fb6540d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb48d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4881469c0 0x7fb4880b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb48d000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb4880b3450 0x7fb488070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb48d000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4880ae7c0 con 0x7fb488145cf0 2024-01-16T14:57:16.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb48d000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4881619e0 con 0x7fb4880b3450 2024-01-16T14:57:16.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7fb48d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880b3900 con 0x7fb4881469c0 2024-01-16T14:57:16.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.388+0000 7fb65ba00640 1 -- 172.21.15.141:0/3899215954 shutdown_connections 2024-01-16T14:57:16.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7f164a000640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f162406b390 0x7f162406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7f164a000640 1 -- 172.21.15.141:0/2233886696 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f163c0abb20 con 0x7f16440e3130 2024-01-16T14:57:16.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.388+0000 7fb65ba00640 1 -- 172.21.15.141:0/3899215954 wait complete. 2024-01-16T14:57:16.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.387+0000 7f5078a00640 1 -- start start 2024-01-16T14:57:16.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.388+0000 7fb487400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb488145cf0 0x7fb4880ae0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40078/0 (socket says 172.21.15.141:40078) 2024-01-16T14:57:16.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.388+0000 7fb487400640 1 -- 172.21.15.141:0/251262100 learned_addr learned my addr 172.21.15.141:0/251262100 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:16.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.388+0000 7f1652000640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f162406b390 0x7f162406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f9fd6000640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fd80b1620 0x7f9fd81689b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000c450 tx=0x7f9fc000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7fb486a00640 1 --2- 172.21.15.141:0/251262100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4881469c0 0x7fb4880b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7ffad8a00640 1 --2- 172.21.15.141:0/4018123299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40b4510 0x7ffad40d3920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009870 tx=0x7ffac402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec44131ca178e039 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7fb487e00640 1 --2- 172.21.15.141:0/251262100 >> v2:172.21.15.188:3300/0 conn(0x7fb4880b3450 0x7fb488070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f5078a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c006b20 0x7f506c007f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f5078a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f506c0342c0 0x7f506c053960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f5078a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c055ce0 con 0x7f506c0dfa30 2024-01-16T14:57:16.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f5078a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506c008480 con 0x7f506c0342c0 2024-01-16T14:57:16.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f5078a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c055fc0 con 0x7f506c006b20 2024-01-16T14:57:16.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f9fcf400640 1 -- 172.21.15.141:0/3631927588 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc00074e0 con 0x7f9fd80b1620 2024-01-16T14:57:16.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd8167910 con 0x7f9fd80b1620 2024-01-16T14:57:16.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.389+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd8167ed0 con 0x7f9fd80b1620 2024-01-16T14:57:16.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7f5072800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c006b20 0x7f506c007f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7fb486a00640 1 -- 172.21.15.141:0/251262100 >> v2:172.21.15.188:3300/0 conn(0x7fb4880b3450 msgr2=0x7fb488070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7fb486a00640 1 --2- 172.21.15.141:0/251262100 >> v2:172.21.15.188:3300/0 conn(0x7fb4880b3450 0x7fb488070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7fb486a00640 1 -- 172.21.15.141:0/251262100 >> v1:172.21.15.141:6789/0 conn(0x7fb488145cf0 legacy=0x7fb4880ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7fb486a00640 1 -- 172.21.15.141:0/251262100 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488160d60 con 0x7fb4881469c0 2024-01-16T14:57:16.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7f5072800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c006b20 0x7f506c007f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33036/0 (socket says 172.21.15.141:33036) 2024-01-16T14:57:16.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7f5072800640 1 -- 172.21.15.141:0/910042101 learned_addr learned my addr 172.21.15.141:0/910042101 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7f9fcf400640 1 -- 172.21.15.141:0/3631927588 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc0007660 con 0x7f9fd80b1620 2024-01-16T14:57:16.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7f9fcf400640 1 -- 172.21.15.141:0/3631927588 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0004a20 con 0x7f9fd80b1620 2024-01-16T14:57:16.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.390+0000 7fb487e00640 1 --2- 172.21.15.141:0/251262100 >> v2:172.21.15.188:3300/0 conn(0x7fb4880b3450 0x7fb488070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:16.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7f5072800640 1 -- 172.21.15.141:0/910042101 >> v2:172.21.15.188:3300/0 conn(0x7f506c0342c0 msgr2=0x7f506c053960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7f5072800640 1 --2- 172.21.15.141:0/910042101 >> v2:172.21.15.188:3300/0 conn(0x7f506c0342c0 0x7f506c053960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7f5072800640 1 -- 172.21.15.141:0/910042101 >> v1:172.21.15.188:6789/0 conn(0x7f506c0dfa30 legacy=0x7f506c006f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7f5072800640 1 -- 172.21.15.141:0/910042101 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c036a50 con 0x7f506c006b20 2024-01-16T14:57:16.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7ffad3400640 1 -- 172.21.15.141:0/4018123299 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4030840 con 0x7ffad40b4510 2024-01-16T14:57:16.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7ffad3400640 1 -- 172.21.15.141:0/4018123299 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac40309a0 con 0x7ffad40b4510 2024-01-16T14:57:16.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7ffad3400640 1 -- 172.21.15.141:0/4018123299 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4030d90 con 0x7ffad40b4510 2024-01-16T14:57:16.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.392+0000 7fb65ba00640 1 Processor -- start 2024-01-16T14:57:16.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.391+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8005250 con 0x7f9fd80b1620 2024-01-16T14:57:16.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.392+0000 7fb65ba00640 1 -- start start 2024-01-16T14:57:16.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.392+0000 7fb486a00640 1 --2- 172.21.15.141:0/251262100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4881469c0 0x7fb4880b2f10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb4700097a0 tx=0x7fb47002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb5efe2d45898f0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.392+0000 7ffadb400640 1 -- 172.21.15.141:0/4018123299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40b4510 msgr2=0x7ffad40d3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.392+0000 7ffadb400640 1 --2- 172.21.15.141:0/4018123299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40b4510 0x7ffad40d3920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009870 tx=0x7ffac402b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7ffadb400640 1 -- 172.21.15.141:0/4018123299 shutdown_connections 2024-01-16T14:57:16.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7ffadb400640 1 --2- 172.21.15.141:0/4018123299 >> v2:172.21.15.141:3300/0 conn(0x7ffad40d1700 0x7ffad40b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7ffadb400640 1 --2- 172.21.15.141:0/4018123299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40b4510 0x7ffad40d3920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7ffadb400640 1 -- 172.21.15.141:0/4018123299 >> 172.21.15.141:0/4018123299 conn(0x7ffad409c540 msgr2=0x7ffad40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7fb486000640 1 -- 172.21.15.141:0/251262100 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470031780 con 0x7fb4881469c0 2024-01-16T14:57:16.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7fb486000640 1 -- 172.21.15.141:0/251262100 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4700318e0 con 0x7fb4881469c0 2024-01-16T14:57:16.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7fb486000640 1 -- 172.21.15.141:0/251262100 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470031c70 con 0x7fb4881469c0 2024-01-16T14:57:16.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7ffadb400640 1 -- 172.21.15.141:0/4018123299 shutdown_connections 2024-01-16T14:57:16.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb65ba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6540b4460 0x7fb6541f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb48d000640 1 -- 172.21.15.141:0/251262100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4881469c0 msgr2=0x7fb4880b2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb48d000640 1 --2- 172.21.15.141:0/251262100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4881469c0 0x7fb4880b2f10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb4700097a0 tx=0x7fb47002b820 comp rx=0 tx=0).stop 2024-01-16T14:57:16.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7f1652000640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f162406b390 0x7f162406d850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1634007fd0 tx=0x7f1634007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7f164a000640 1 -- 172.21.15.141:0/2233886696 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f164403a810 con 0x7f16440e3130 2024-01-16T14:57:16.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.393+0000 7ffadb400640 1 -- 172.21.15.141:0/4018123299 wait complete. 2024-01-16T14:57:16.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb48d000640 1 -- 172.21.15.141:0/251262100 shutdown_connections 2024-01-16T14:57:16.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb48d000640 1 --2- 172.21.15.141:0/251262100 >> v2:172.21.15.188:3300/0 conn(0x7fb4880b3450 0x7fb488070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb48d000640 1 --2- 172.21.15.141:0/251262100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4881469c0 0x7fb4880b2f10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb48d000640 1 -- 172.21.15.141:0/251262100 >> 172.21.15.141:0/251262100 conn(0x7fb4880b0b00 msgr2=0x7fb4880b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7f5072800640 1 --2- 172.21.15.141:0/910042101 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c006b20 0x7f506c007f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5064009870 tx=0x7f506402ff40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3cb938d255c935f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7f9fcf400640 1 -- 172.21.15.141:0/3631927588 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9fc0004b80 con 0x7f9fd80b1620 2024-01-16T14:57:16.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb65ba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6540d1cc0 0x7fb6540b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb65ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb65ba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65415c3a0 con 0x7fb6540b4460 2024-01-16T14:57:16.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb65ba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6540c9b70 con 0x7fb6540d1cc0 2024-01-16T14:57:16.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.394+0000 7fb65ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6540c80b0 con 0x7fb65415fa90 2024-01-16T14:57:16.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.395+0000 7f5071400640 1 -- 172.21.15.141:0/910042101 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064030cf0 con 0x7f506c006b20 2024-01-16T14:57:16.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.395+0000 7f5071400640 1 -- 172.21.15.141:0/910042101 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5064030e50 con 0x7f506c006b20 2024-01-16T14:57:16.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.395+0000 7f5071400640 1 -- 172.21.15.141:0/910042101 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064038660 con 0x7f506c006b20 2024-01-16T14:57:16.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.395+0000 7fb659000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6540b4460 0x7fb6541f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.395+0000 7fb659000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6540b4460 0x7fb6541f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48614/0 (socket says 172.21.15.141:48614) 2024-01-16T14:57:16.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.395+0000 7fb659000640 1 -- 172.21.15.141:0/301718678 learned_addr learned my addr 172.21.15.141:0/301718678 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f9fcf400640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb006b2c0 0x7f9fb006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 -- 172.21.15.141:0/910042101 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c006b20 msgr2=0x7f506c007f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 --2- 172.21.15.141:0/910042101 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c006b20 0x7f506c007f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5064009870 tx=0x7f506402ff40 comp rx=0 tx=0).stop 2024-01-16T14:57:16.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 -- 172.21.15.141:0/910042101 shutdown_connections 2024-01-16T14:57:16.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 --2- 172.21.15.141:0/910042101 >> v2:172.21.15.188:3300/0 conn(0x7f506c0342c0 0x7f506c053960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 --2- 172.21.15.141:0/910042101 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c006b20 0x7f506c007f40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 -- 172.21.15.141:0/910042101 >> 172.21.15.141:0/910042101 conn(0x7f506c01e690 msgr2=0x7f506c0553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 -- 172.21.15.141:0/910042101 shutdown_connections 2024-01-16T14:57:16.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f5078a00640 1 -- 172.21.15.141:0/910042101 wait complete. 2024-01-16T14:57:16.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7f9fcf400640 1 -- 172.21.15.141:0/3631927588 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fc00891c0 con 0x7f9fd80b1620 2024-01-16T14:57:16.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7fb659a00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7fb653e00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6540d1cc0 0x7fb6540b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.396+0000 7ffadb400640 1 Processor -- start 2024-01-16T14:57:16.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7fb659a00640 1 -- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6540d1cc0 msgr2=0x7fb6540b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7fb659a00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6540d1cc0 0x7fb6540b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7fb659a00640 1 -- 172.21.15.141:0/301718678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6540b4460 msgr2=0x7fb6541f2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7fb659a00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6540b4460 0x7fb6541f2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7fb659a00640 1 -- 172.21.15.141:0/301718678 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c009a70 con 0x7fb65415fa90 2024-01-16T14:57:16.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7f9fd6a00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb006b2c0 0x7f9fb006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7fb659000640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6540b4460 0x7fb6541f2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:16.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7ffadb400640 1 -- start start 2024-01-16T14:57:16.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.397+0000 7fb659a00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540b3b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6540ca290 tx=0x7fb63c00c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.398+0000 7fb48d000640 1 -- 172.21.15.141:0/251262100 shutdown_connections 2024-01-16T14:57:16.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.398+0000 7fb48d000640 1 -- 172.21.15.141:0/251262100 wait complete. 2024-01-16T14:57:16.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.398+0000 7fb650c00640 1 -- 172.21.15.141:0/301718678 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0074e0 con 0x7fb65415fa90 2024-01-16T14:57:16.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.398+0000 7fb650c00640 1 -- 172.21.15.141:0/301718678 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c007640 con 0x7fb65415fa90 2024-01-16T14:57:16.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.398+0000 7fb650c00640 1 -- 172.21.15.141:0/301718678 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c004a20 con 0x7fb65415fa90 2024-01-16T14:57:16.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.398+0000 7f5078a00640 1 Processor -- start 2024-01-16T14:57:16.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.398+0000 7f5078a00640 1 -- start start 2024-01-16T14:57:16.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.399+0000 7f9fd6a00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb006b2c0 0x7f9fb006d780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8036000 tx=0x7f9fc8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7ffadb400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad40b4510 0x7ffad40b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.399+0000 7f5078a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f506c006b20 0x7f506c044740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5078a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f506c0342c0 0x7f506c0211f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5078a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c0dfa30 0x7f506c0dafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5078a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c047e60 con 0x7f506c0342c0 2024-01-16T14:57:16.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5078a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0475c0 con 0x7f506c006b20 2024-01-16T14:57:16.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5078a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0478a0 con 0x7f506c0dfa30 2024-01-16T14:57:16.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5071e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f506c0342c0 0x7f506c0211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7ffadb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad40b48f0 0x7ffad414afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65415c5d0 con 0x7fb65415fa90 2024-01-16T14:57:16.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5071e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f506c0342c0 0x7f506c0211f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48628/0 (socket says 172.21.15.141:48628) 2024-01-16T14:57:16.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5071e00640 1 -- 172.21.15.141:0/1954475250 learned_addr learned my addr 172.21.15.141:0/1954475250 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7f5073200640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c0dfa30 0x7f506c0dafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7ffadb400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40d1700 0x7ffad415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7ffad8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad40b4510 0x7ffad40b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7ffad8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad40b4510 0x7ffad40b8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36584/0 (socket says 172.21.15.141:36584) 2024-01-16T14:57:16.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.400+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6540c9890 con 0x7fb65415fa90 2024-01-16T14:57:16.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7ffad8a00640 1 -- 172.21.15.141:0/4065188625 learned_addr learned my addr 172.21.15.141:0/4065188625 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7f5072800640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f506c006b20 0x7f506c044740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b6e80 con 0x7ffad40b48f0 2024-01-16T14:57:16.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b70d0 con 0x7ffad40b4510 2024-01-16T14:57:16.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b73b0 con 0x7ffad40d1700 2024-01-16T14:57:16.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7f5071e00640 1 -- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c0dfa30 msgr2=0x7f506c0dafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7f5071e00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c0dfa30 0x7f506c0dafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7ffad9400640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40d1700 0x7ffad415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7f5071e00640 1 -- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f506c006b20 msgr2=0x7f506c044740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7f5071e00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f506c006b20 0x7f506c044740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7f5071e00640 1 -- 172.21.15.141:0/1954475250 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640094d0 con 0x7f506c0342c0 2024-01-16T14:57:16.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.401+0000 7f5073200640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c0dfa30 0x7f506c0dafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:57:16.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.402+0000 7ffad8a00640 1 -- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40d1700 msgr2=0x7ffad415b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.402+0000 7ffad8a00640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40d1700 0x7ffad415b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.402+0000 7ffad8a00640 1 -- 172.21.15.141:0/4065188625 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad40b48f0 msgr2=0x7ffad414afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.402+0000 7ffad8a00640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad40b48f0 0x7ffad414afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.402+0000 7ffad8a00640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac40094d0 con 0x7ffad40b4510 2024-01-16T14:57:16.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.403+0000 7ffad8a00640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad40b4510 0x7ffad40b8170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffac402b9e0 tx=0x7ffac40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.402+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6540675d0 con 0x7fb65415fa90 2024-01-16T14:57:16.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.402+0000 7fb48d000640 1 Processor -- start 2024-01-16T14:57:16.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.403+0000 7fb48d000640 1 -- start start 2024-01-16T14:57:16.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.403+0000 7f5071e00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f506c0342c0 0x7f506c0211f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f505c00c410 tx=0x7f505c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.403+0000 7f506aa00640 1 -- 172.21.15.141:0/1954475250 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c0035f0 con 0x7f506c0342c0 2024-01-16T14:57:16.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.403+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c047b80 con 0x7f506c0342c0 2024-01-16T14:57:16.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.404+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c047240 con 0x7f506c0342c0 2024-01-16T14:57:16.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7f506aa00640 1 -- 172.21.15.141:0/1954475250 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c003770 con 0x7f506c0342c0 2024-01-16T14:57:16.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c005360 con 0x7f506c0342c0 2024-01-16T14:57:16.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7fb650c00640 1 -- 172.21.15.141:0/301718678 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb63c004b80 con 0x7fb65415fa90 2024-01-16T14:57:16.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.404+0000 7fb48d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4880b3450 0x7fb488134690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.404+0000 7fb48d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb488145cf0 0x7fb488145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.404+0000 7fb48d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4881469c0 0x7fb48815f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.404+0000 7fb48d000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb488144be0 con 0x7fb488145cf0 2024-01-16T14:57:16.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.404+0000 7fb48d000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48813ea20 con 0x7fb4881469c0 2024-01-16T14:57:16.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.404+0000 7fb48d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb488145540 con 0x7fb4880b3450 2024-01-16T14:57:16.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7fb487400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4880b3450 0x7fb488134690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7f506aa00640 1 -- 172.21.15.141:0/1954475250 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c003b60 con 0x7f506c0342c0 2024-01-16T14:57:16.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7fb487400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4880b3450 0x7fb488134690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33082/0 (socket says 172.21.15.141:33082) 2024-01-16T14:57:16.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7fb487400640 1 -- 172.21.15.141:0/3633157568 learned_addr learned my addr 172.21.15.141:0/3633157568 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.405+0000 7fb487e00640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4881469c0 0x7fb48815f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.406+0000 7fb486a00640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb488145cf0 0x7fb488145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.406+0000 7fb650c00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb62c06b390 0x7fb62c06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.407+0000 7fb650c00640 1 -- 172.21.15.141:0/301718678 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb63c089dd0 con 0x7fb65415fa90 2024-01-16T14:57:16.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.407+0000 7fb487e00640 1 -- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4880b3450 msgr2=0x7fb488134690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.407+0000 7fb487e00640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4880b3450 0x7fb488134690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.407+0000 7fb487e00640 1 -- 172.21.15.141:0/3633157568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb488145cf0 msgr2=0x7fb488145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.407+0000 7f506aa00640 1 -- 172.21.15.141:0/1954475250 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f505c003cc0 con 0x7f506c0342c0 2024-01-16T14:57:16.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.407+0000 7fb659000640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb62c06b390 0x7fb62c06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.408+0000 7fb487e00640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb488145cf0 0x7fb488145000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.408+0000 7fb487e00640 1 -- 172.21.15.141:0/3633157568 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478009400 con 0x7fb4881469c0 2024-01-16T14:57:16.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.408+0000 7f506aa00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f504c06b2c0 0x7f504c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.408+0000 7f506aa00640 1 -- 172.21.15.141:0/1954475250 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f505c089590 con 0x7f506c0342c0 2024-01-16T14:57:16.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.408+0000 7fb487e00640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4881469c0 0x7fb48815f290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb47800e7d0 tx=0x7fb47800ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7ffad0c00640 1 -- 172.21.15.141:0/4065188625 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4003a20 con 0x7ffad40b4510 2024-01-16T14:57:16.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7ffad0c00640 1 -- 172.21.15.141:0/4065188625 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac4003ba0 con 0x7ffad40b4510 2024-01-16T14:57:16.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7ffad0c00640 1 -- 172.21.15.141:0/4065188625 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4039450 con 0x7ffad40b4510 2024-01-16T14:57:16.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7fb659000640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb62c06b390 0x7fb62c06d850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb64c02bfe0 tx=0x7fb64c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad40d1e70 con 0x7ffad40b4510 2024-01-16T14:57:16.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7fb47f400640 1 -- 172.21.15.141:0/3633157568 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47800eee0 con 0x7fb4881469c0 2024-01-16T14:57:16.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad40b7970 con 0x7ffad40b4510 2024-01-16T14:57:16.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7f5072800640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f504c06b2c0 0x7f504c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7f9fcf400640 1 -- 172.21.15.141:0/3631927588 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fc0054150 con 0x7f9fd80b1620 2024-01-16T14:57:16.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7fb47f400640 1 -- 172.21.15.141:0/3633157568 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb478004020 con 0x7fb4881469c0 2024-01-16T14:57:16.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.409+0000 7fb47f400640 1 -- 172.21.15.141:0/3633157568 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4780043b0 con 0x7fb4881469c0 2024-01-16T14:57:16.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.410+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb470009400 con 0x7fb4881469c0 2024-01-16T14:57:16.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.410+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb488144240 con 0x7fb4881469c0 2024-01-16T14:57:16.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.411+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad40675d0 con 0x7ffad40b4510 2024-01-16T14:57:16.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.411+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb458005250 con 0x7fb4881469c0 2024-01-16T14:57:16.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.412+0000 7f5072800640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f504c06b2c0 0x7f504c06d780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f506403a000 tx=0x7f5064034000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.414+0000 7ffad0c00640 1 -- 172.21.15.141:0/4065188625 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ffac404d450 con 0x7ffad40b4510 2024-01-16T14:57:16.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.414+0000 7fb47f400640 1 -- 172.21.15.141:0/3633157568 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb47800c940 con 0x7fb4881469c0 2024-01-16T14:57:16.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.415+0000 7ffad0c00640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06b390 0x7ffaac06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.415+0000 7ffad0c00640 1 -- 172.21.15.141:0/4065188625 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffac40ac450 con 0x7ffad40b4510 2024-01-16T14:57:16.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.415+0000 7fb47f400640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb46006b390 0x7fb46006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.415+0000 7fb47f400640 1 -- 172.21.15.141:0/3633157568 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb47808a050 con 0x7fb4881469c0 2024-01-16T14:57:16.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.416+0000 7ffad3e00640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06b390 0x7ffaac06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.416+0000 7fb487400640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb46006b390 0x7fb46006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.417+0000 7f506aa00640 1 -- 172.21.15.141:0/1954475250 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f505c054470 con 0x7f506c0342c0 2024-01-16T14:57:16.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.417+0000 7ffad3e00640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06b390 0x7ffaac06d850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffabc000f60 tx=0x7ffabc0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.418+0000 7fb487400640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb46006b390 0x7fb46006d850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb480001f30 tx=0x7fb480001ac0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.421+0000 7fb650c00640 1 -- 172.21.15.141:0/301718678 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb63c054d90 con 0x7fb65415fa90 2024-01-16T14:57:16.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.428+0000 7ffad0c00640 1 -- 172.21.15.141:0/4065188625 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffac40772a0 con 0x7ffad40b4510 2024-01-16T14:57:16.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.438+0000 7f662c000640 1 Processor -- start 2024-01-16T14:57:16.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.438+0000 7f662c000640 1 -- start start 2024-01-16T14:57:16.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.440+0000 7f662c000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f66240d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.440+0000 7f662c000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240d2180 con 0x7f66240b3890 2024-01-16T14:57:16.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.440+0000 7f662c000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240b3400 con 0x7f662415d610 2024-01-16T14:57:16.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.440+0000 7f662c000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66240d1390 con 0x7f66240d17e0 2024-01-16T14:57:16.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.441+0000 7f6629600640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f662415d610 0x7f66240ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:41632/0 (socket says 172.21.15.141:41632) 2024-01-16T14:57:16.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.441+0000 7f6629600640 1 -- 172.21.15.141:0/3154650287 learned_addr learned my addr 172.21.15.141:0/3154650287 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:16.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.441+0000 7f6628c00640 1 --2- 172.21.15.141:0/3154650287 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f66240d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.443+0000 7f6628c00640 1 -- 172.21.15.141:0/3154650287 >> v1:172.21.15.188:6789/0 conn(0x7f662415d610 legacy=0x7f66240ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:16.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.443+0000 7f6628c00640 1 -- 172.21.15.141:0/3154650287 >> v1:172.21.15.141:6789/0 conn(0x7f66240b3890 legacy=0x7f66240b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.443+0000 7f6628c00640 1 -- 172.21.15.141:0/3154650287 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66240b27f0 con 0x7f66240d17e0 2024-01-16T14:57:16.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.445+0000 7f6628c00640 1 --2- 172.21.15.141:0/3154650287 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f66240d1c40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f660c0097d0 tx=0x7f660c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89c8435a4e5cdee3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.445+0000 7f6623e00640 1 -- 172.21.15.141:0/3154650287 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c02d040 con 0x7f66240d17e0 2024-01-16T14:57:16.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.445+0000 7fb47f400640 1 -- 172.21.15.141:0/3633157568 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb478054fe0 con 0x7fb4881469c0 2024-01-16T14:57:16.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.446+0000 7f6623e00640 1 -- 172.21.15.141:0/3154650287 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c031810 con 0x7f66240d17e0 2024-01-16T14:57:16.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.446+0000 7f6623e00640 1 -- 172.21.15.141:0/3154650287 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c031ba0 con 0x7f66240d17e0 2024-01-16T14:57:16.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.447+0000 7f662c000640 1 -- 172.21.15.141:0/3154650287 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 msgr2=0x7f66240d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.447+0000 7f662c000640 1 --2- 172.21.15.141:0/3154650287 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f66240d1c40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f660c0097d0 tx=0x7f660c02b8b0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.448+0000 7f662c000640 1 -- 172.21.15.141:0/3154650287 shutdown_connections 2024-01-16T14:57:16.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.448+0000 7f662c000640 1 --2- 172.21.15.141:0/3154650287 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f66240d1c40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.448+0000 7f662c000640 1 -- 172.21.15.141:0/3154650287 >> 172.21.15.141:0/3154650287 conn(0x7f662409c540 msgr2=0x7f66240d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.448+0000 7f662c000640 1 -- 172.21.15.141:0/3154650287 shutdown_connections 2024-01-16T14:57:16.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.448+0000 7f662c000640 1 -- 172.21.15.141:0/3154650287 wait complete. 2024-01-16T14:57:16.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.451+0000 7f662c000640 1 Processor -- start 2024-01-16T14:57:16.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.451+0000 7f662c000640 1 -- start start 2024-01-16T14:57:16.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.454+0000 7f662c000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f66240b3890 0x7f66241f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.454+0000 7f662c000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f662415be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.454+0000 7f662c000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f662415d610 0x7f66241f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.454+0000 7f6629600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f66240b3890 0x7f66241f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.454+0000 7f6629600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f66240b3890 0x7f66241f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48660/0 (socket says 172.21.15.141:48660) 2024-01-16T14:57:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.455+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66241f36c0 con 0x7f66240b3890 2024-01-16T14:57:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.455+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66241f73a0 con 0x7f662415d610 2024-01-16T14:57:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.455+0000 7f6629600640 1 -- 172.21.15.141:0/4220311789 learned_addr learned my addr 172.21.15.141:0/4220311789 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.455+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66240cbb70 con 0x7f66240d17e0 2024-01-16T14:57:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.455+0000 7f6628c00640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f662415be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.455+0000 7f662a000640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f662415d610 0x7f66241f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.456+0000 7f6629600640 1 -- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 msgr2=0x7f662415be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.456+0000 7f6629600640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f662415be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.456+0000 7f6629600640 1 -- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f662415d610 msgr2=0x7f66241f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.456+0000 7f6629600640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f662415d610 0x7f66241f6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.456+0000 7f6629600640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c009400 con 0x7f66240b3890 2024-01-16T14:57:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.456+0000 7f6629600640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f66240b3890 0x7f66241f3b50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f661400b430 tx=0x7f661400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.457+0000 7f6621600640 1 -- 172.21.15.141:0/4220311789 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614011830 con 0x7f66240b3890 2024-01-16T14:57:16.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.457+0000 7f6621600640 1 -- 172.21.15.141:0/4220311789 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66140119b0 con 0x7f66240b3890 2024-01-16T14:57:16.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.457+0000 7f6621600640 1 -- 172.21.15.141:0/4220311789 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614011d60 con 0x7f66240b3890 2024-01-16T14:57:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.457+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66241f70c0 con 0x7f66240b3890 2024-01-16T14:57:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.457+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66241f3350 con 0x7f66240b3890 2024-01-16T14:57:16.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.461+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f4005250 con 0x7f66240b3890 2024-01-16T14:57:16.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.464+0000 7f6621600640 1 -- 172.21.15.141:0/4220311789 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f661401a400 con 0x7f66240b3890 2024-01-16T14:57:16.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.465+0000 7f6621600640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65fc06b2c0 0x7f65fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:16.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.469+0000 7f6621600640 1 -- 172.21.15.141:0/4220311789 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6614089c90 con 0x7f66240b3890 2024-01-16T14:57:16.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.476+0000 7f6628c00640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65fc06b2c0 0x7f65fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:16.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.475+0000 7f6621600640 1 -- 172.21.15.141:0/4220311789 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6614058580 con 0x7f66240b3890 2024-01-16T14:57:16.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.477+0000 7f6628c00640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65fc06b2c0 0x7f65fc06d780 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f660c036000 tx=0x7f660c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:16.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.753+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe5e80d3f90 con 0x7fe5e804ed30 2024-01-16T14:57:16.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.754+0000 7fe5ee000640 1 -- 172.21.15.141:0/41307499 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe5e0054cf0 con 0x7fe5e804ed30 2024-01-16T14:57:16.735 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:16.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.757+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe5c806b2c0 msgr2=0x7fe5c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.757+0000 7fe5f8a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe5c806b2c0 0x7fe5c806d780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0004740 tx=0x7fe5f006d030 comp rx=0 tx=0).stop 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5e804ed30 msgr2=0x7fe5e80d1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5e804ed30 0x7fe5e80d1e80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80578d0 tx=0x7fe5e000e9e0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 shutdown_connections 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe5c806b2c0 0x7fe5c806d780 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5e804ed30 0x7fe5e80d1e80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5e80d45b0 0x7fe5e804e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 --2- 172.21.15.141:0/41307499 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5d0004d40 0x7fe5e804b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.758+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 >> 172.21.15.141:0/41307499 conn(0x7fe5e80559a0 msgr2=0x7fe5e8039e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.759+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 shutdown_connections 2024-01-16T14:57:16.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.759+0000 7fe5f8a00640 1 -- 172.21.15.141:0/41307499 wait complete. 2024-01-16T14:57:16.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.1 2024-01-16T14:57:16.766 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.788+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f164403a810 con 0x7f16440e3130 2024-01-16T14:57:16.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.789+0000 7f164a000640 1 -- 172.21.15.141:0/2233886696 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f164403a810 con 0x7f16440e3130 2024-01-16T14:57:16.771 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:16.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.792+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f162406b390 msgr2=0x7f162406d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.792+0000 7f1653400640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f162406b390 0x7f162406d850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1634007fd0 tx=0x7f1634007de0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16440e3130 msgr2=0x7f164404f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16440e3130 0x7f164404f360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f163c009870 tx=0x7f163c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 shutdown_connections 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f162406b390 0x7f162406d850 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16440e3130 0x7f164404f360 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1644053c80 0x7f164404d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 --2- 172.21.15.141:0/2233886696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f164403c070 0x7f1644009d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 >> 172.21.15.141:0/2233886696 conn(0x7f1644058eb0 msgr2=0x7f164403d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 shutdown_connections 2024-01-16T14:57:16.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.793+0000 7f1653400640 1 -- 172.21.15.141:0/2233886696 wait complete. 2024-01-16T14:57:16.799 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.6 2024-01-16T14:57:16.800 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.823+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ffad40d3c40 con 0x7ffad40b4510 2024-01-16T14:57:16.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.824+0000 7ffad0c00640 1 -- 172.21.15.141:0/4065188625 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffac407a8c0 con 0x7ffad40b4510 2024-01-16T14:57:16.806 INFO:teuthology.orchestra.run.smithi141.stdout:60129542150 2024-01-16T14:57:16.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.824+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9fa80057d0 con 0x7f9fd80b1620 2024-01-16T14:57:16.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.825+0000 7f9fcf400640 1 -- 172.21.15.141:0/3631927588 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9fc0018020 con 0x7f9fd80b1620 2024-01-16T14:57:16.807 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:16.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.826+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06b390 msgr2=0x7ffaac06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.826+0000 7ffadb400640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06b390 0x7ffaac06d850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffabc000f60 tx=0x7ffabc0079b0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.826+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad40b4510 msgr2=0x7ffad40b8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.826+0000 7ffadb400640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad40b4510 0x7ffad40b8170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffac402b9e0 tx=0x7ffac40046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 shutdown_connections 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06b390 0x7ffaac06d850 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad40d1700 0x7ffad415b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad40b48f0 0x7ffad414afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 --2- 172.21.15.141:0/4065188625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad40b4510 0x7ffad40b8170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 >> 172.21.15.141:0/4065188625 conn(0x7ffad409c540 msgr2=0x7ffad40cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 shutdown_connections 2024-01-16T14:57:16.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.827+0000 7ffadb400640 1 -- 172.21.15.141:0/4065188625 wait complete. 2024-01-16T14:57:16.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.828+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f65f4005500 con 0x7f66240b3890 2024-01-16T14:57:16.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb006b2c0 msgr2=0x7f9fb006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb006b2c0 0x7f9fb006d780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8036000 tx=0x7f9fc8008030 comp rx=0 tx=0).stop 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fd80b1620 msgr2=0x7f9fd81689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fd80b1620 0x7f9fd81689b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000c450 tx=0x7f9fc000c910 comp rx=0 tx=0).stop 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 shutdown_connections 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb006b2c0 0x7f9fb006d780 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fd81568f0 0x7f9fd816af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fd80b1620 0x7f9fd81689b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 --2- 172.21.15.141:0/3631927588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fd8071e20 0x7f9fd8143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 >> 172.21.15.141:0/3631927588 conn(0x7f9fd80b0b00 msgr2=0x7f9fd80ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f6621600640 1 -- 172.21.15.141:0/4220311789 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6614015090 con 0x7f66240b3890 2024-01-16T14:57:16.812 INFO:teuthology.orchestra.run.smithi141.stdout:60129542150 2024-01-16T14:57:16.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 shutdown_connections 2024-01-16T14:57:16.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.829+0000 7f9fdde00640 1 -- 172.21.15.141:0/3631927588 wait complete. 2024-01-16T14:57:16.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65fc06b2c0 msgr2=0x7f65fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65fc06b2c0 0x7f65fc06d780 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f660c036000 tx=0x7f660c03c040 comp rx=0 tx=0).stop 2024-01-16T14:57:16.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f66240b3890 msgr2=0x7f66241f3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f66240b3890 0x7f66241f3b50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f661400b430 tx=0x7f661400b8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 shutdown_connections 2024-01-16T14:57:16.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65fc06b2c0 0x7f65fc06d780 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f662415d610 0x7f66241f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f66240d17e0 0x7f662415be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 --2- 172.21.15.141:0/4220311789 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f66240b3890 0x7f66241f3b50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 >> 172.21.15.141:0/4220311789 conn(0x7f662409c540 msgr2=0x7f662406b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 shutdown_connections 2024-01-16T14:57:16.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.832+0000 7f662c000640 1 -- 172.21.15.141:0/4220311789 wait complete. 2024-01-16T14:57:16.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.833+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f506c048a50 con 0x7f506c0342c0 2024-01-16T14:57:16.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.834+0000 7f506aa00640 1 -- 172.21.15.141:0/1954475250 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f505c017020 con 0x7f506c0342c0 2024-01-16T14:57:16.815 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:16.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.837+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f504c06b2c0 msgr2=0x7f504c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.837+0000 7f5078a00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f504c06b2c0 0x7f504c06d780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f506403a000 tx=0x7f5064034000 comp rx=0 tx=0).stop 2024-01-16T14:57:16.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.837+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f506c0342c0 msgr2=0x7f506c0211f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.837+0000 7f5078a00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f506c0342c0 0x7f506c0211f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f505c00c410 tx=0x7f505c00c810 comp rx=0 tx=0).stop 2024-01-16T14:57:16.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 shutdown_connections 2024-01-16T14:57:16.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f504c06b2c0 0x7f504c06d780 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f506c0dfa30 0x7f506c0dafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f506c0342c0 0x7f506c0211f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 --2- 172.21.15.141:0/1954475250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f506c006b20 0x7f506c044740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 >> 172.21.15.141:0/1954475250 conn(0x7f506c01e690 msgr2=0x7f506c03a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 shutdown_connections 2024-01-16T14:57:16.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.838+0000 7f5078a00640 1 -- 172.21.15.141:0/1954475250 wait complete. 2024-01-16T14:57:16.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.4 2024-01-16T14:57:16.829 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.5 2024-01-16T14:57:16.834 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.7 2024-01-16T14:57:16.838 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.858+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb65409ca20 con 0x7fb65415fa90 2024-01-16T14:57:16.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.2 2024-01-16T14:57:16.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.859+0000 7fb650c00640 1 -- 172.21.15.141:0/301718678 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb63c0583b0 con 0x7fb65415fa90 2024-01-16T14:57:16.841 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.842 INFO:teuthology.orchestra.run.smithi141.stdout:60129542150 2024-01-16T14:57:16.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.861+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb4580057d0 con 0x7fb4881469c0 2024-01-16T14:57:16.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.862+0000 7fb47f400640 1 -- 172.21.15.141:0/3633157568 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb478017090 con 0x7fb4881469c0 2024-01-16T14:57:16.844 INFO:teuthology.orchestra.run.smithi141.stdout:60129542149 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb62c06b390 msgr2=0x7fb62c06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb62c06b390 0x7fb62c06d850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb64c02bfe0 tx=0x7fb64c030000 comp rx=0 tx=0).stop 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 msgr2=0x7fb6540b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540b3b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6540ca290 tx=0x7fb63c00c870 comp rx=0 tx=0).stop 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 shutdown_connections 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb62c06b390 0x7fb62c06d850 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb65415fa90 0x7fb6540b3b50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6540d1cc0 0x7fb6540b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 --2- 172.21.15.141:0/301718678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6540b4460 0x7fb6541f2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 >> 172.21.15.141:0/301718678 conn(0x7fb65409c540 msgr2=0x7fb6540b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 shutdown_connections 2024-01-16T14:57:16.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.863+0000 7fb65ba00640 1 -- 172.21.15.141:0/301718678 wait complete. 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb46006b390 msgr2=0x7fb46006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb46006b390 0x7fb46006d850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb480001f30 tx=0x7fb480001ac0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4881469c0 msgr2=0x7fb48815f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4881469c0 0x7fb48815f290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb47800e7d0 tx=0x7fb47800ec90 comp rx=0 tx=0).stop 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 shutdown_connections 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb46006b390 0x7fb46006d850 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4881469c0 0x7fb48815f290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb488145cf0 0x7fb488145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 --2- 172.21.15.141:0/3633157568 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4880b3450 0x7fb488134690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:16.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 >> 172.21.15.141:0/3633157568 conn(0x7fb4880b0b00 msgr2=0x7fb4880b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:16.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.864+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 shutdown_connections 2024-01-16T14:57:16.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:16.865+0000 7fb48d000640 1 -- 172.21.15.141:0/3633157568 wait complete. 2024-01-16T14:57:16.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.3 2024-01-16T14:57:16.863 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.0 2024-01-16T14:57:16.863 DEBUG:teuthology.parallel:result is None 2024-01-16T14:57:16.863 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T14:57:16.864 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:57:17.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.117+0000 7f7436400640 1 Processor -- start 2024-01-16T14:57:17.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.117+0000 7f7436400640 1 -- start start 2024-01-16T14:57:17.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f7436400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743013df80 0x7f743015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f7436400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f743013e360 0x7f74300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f7436400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f743015f9c0 0x7f74300cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f7436400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f743006c470 con 0x7f743015f9c0 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f7436400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f743006c750 con 0x7f743013e360 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f7436400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743006ca30 con 0x7f743013df80 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f742fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f743015f9c0 0x7f74300cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f742fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f743015f9c0 0x7f74300cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48662/0 (socket says 172.21.15.141:48662) 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f742fe00640 1 -- 172.21.15.141:0/2886699158 learned_addr learned my addr 172.21.15.141:0/2886699158 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.119+0000 7f742f400640 1 --2- 172.21.15.141:0/2886699158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743013df80 0x7f743015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.120+0000 7f7427e00640 1 --2- 172.21.15.141:0/2886699158 >> v2:172.21.15.188:3300/0 conn(0x7f743013e360 0x7f74300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.120+0000 7f742fe00640 1 -- 172.21.15.141:0/2886699158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743013df80 msgr2=0x7f743015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.120+0000 7f742fe00640 1 --2- 172.21.15.141:0/2886699158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743013df80 0x7f743015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.120+0000 7f742fe00640 1 -- 172.21.15.141:0/2886699158 >> v2:172.21.15.188:3300/0 conn(0x7f743013e360 msgr2=0x7f74300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:17.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.120+0000 7f742fe00640 1 --2- 172.21.15.141:0/2886699158 >> v2:172.21.15.188:3300/0 conn(0x7f743013e360 0x7f74300cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.120+0000 7f742fe00640 1 -- 172.21.15.141:0/2886699158 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743006cd10 con 0x7f743015f9c0 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.121+0000 7f742fe00640 1 --2- 172.21.15.141:0/2886699158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743015f9c0 0x7f74300cf7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f74300cfed0 tx=0x7f742802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fb30142237733d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.121+0000 7f742ea00640 1 -- 172.21.15.141:0/2886699158 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742802ca70 con 0x7f743015f9c0 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.121+0000 7f742ea00640 1 -- 172.21.15.141:0/2886699158 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742802cbd0 con 0x7f743015f9c0 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.121+0000 7f742ea00640 1 -- 172.21.15.141:0/2886699158 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280343f0 con 0x7f743015f9c0 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 -- 172.21.15.141:0/2886699158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743015f9c0 msgr2=0x7f74300cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 --2- 172.21.15.141:0/2886699158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743015f9c0 0x7f74300cf7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f74300cfed0 tx=0x7f742802bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 -- 172.21.15.141:0/2886699158 shutdown_connections 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 --2- 172.21.15.141:0/2886699158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743015f9c0 0x7f74300cf7d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 --2- 172.21.15.141:0/2886699158 >> v2:172.21.15.188:3300/0 conn(0x7f743013e360 0x7f74300cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 --2- 172.21.15.141:0/2886699158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743013df80 0x7f743015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 -- 172.21.15.141:0/2886699158 >> 172.21.15.141:0/2886699158 conn(0x7f743006a800 msgr2=0x7f743006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:17.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 -- 172.21.15.141:0/2886699158 shutdown_connections 2024-01-16T14:57:17.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.122+0000 7f7436400640 1 -- 172.21.15.141:0/2886699158 wait complete. 2024-01-16T14:57:17.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.123+0000 7f7436400640 1 Processor -- start 2024-01-16T14:57:17.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.124+0000 7f7436400640 1 -- start start 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.124+0000 7f7436400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743013df80 0x7f743013f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.124+0000 7f7436400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743013e360 0x7f7430071750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.124+0000 7f7436400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743015f9c0 0x7f7430071c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.124+0000 7f7436400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7430137400 con 0x7f743013e360 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.124+0000 7f7436400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74301367f0 con 0x7f743013df80 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.124+0000 7f7436400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7430136ad0 con 0x7f743015f9c0 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743013e360 0x7f7430071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743013e360 0x7f7430071750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48664/0 (socket says 172.21.15.141:48664) 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 -- 172.21.15.141:0/2349143458 learned_addr learned my addr 172.21.15.141:0/2349143458 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f742f400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743013df80 0x7f743013f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f742fe00640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743015f9c0 0x7f7430071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 -- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743015f9c0 msgr2=0x7f7430071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743015f9c0 0x7f7430071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 -- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743013df80 msgr2=0x7f743013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743013df80 0x7f743013f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 -- 172.21.15.141:0/2349143458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7428009b30 con 0x7f743013e360 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f742f400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743013df80 0x7f743013f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.125+0000 7f7427e00640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743013e360 0x7f7430071750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f741800d800 tx=0x7f741800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.126+0000 7f7427400640 1 -- 172.21.15.141:0/2349143458 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741800b6c0 con 0x7f743013e360 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.126+0000 7f7427400640 1 -- 172.21.15.141:0/2349143458 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741800b820 con 0x7f743013e360 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.126+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7430136db0 con 0x7f743013e360 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.126+0000 7f7427400640 1 -- 172.21.15.141:0/2349143458 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741800bbb0 con 0x7f743013e360 2024-01-16T14:57:17.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.126+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74300721d0 con 0x7f743013e360 2024-01-16T14:57:17.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.127+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74300675d0 con 0x7f743013e360 2024-01-16T14:57:17.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.128+0000 7f7427400640 1 -- 172.21.15.141:0/2349143458 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f741800bd10 con 0x7f743013e360 2024-01-16T14:57:17.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.129+0000 7f7427400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f740806b2c0 0x7f740806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.129+0000 7f7427400640 1 -- 172.21.15.141:0/2349143458 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f741808a0a0 con 0x7f743013e360 2024-01-16T14:57:17.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.129+0000 7f742f400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f740806b2c0 0x7f740806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.130+0000 7f742f400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f740806b2c0 0x7f740806d780 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7420006fd0 tx=0x7f7420008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:17.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.133+0000 7f7427400640 1 -- 172.21.15.141:0/2349143458 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7418054f10 con 0x7f743013e360 2024-01-16T14:57:17.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.305+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74301408d0 con 0x7f740806b2c0 2024-01-16T14:57:17.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.326+0000 7f7427400640 1 -- 172.21.15.141:0/2349143458 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264347 (secure 0 0 0) 0x7f7420009530 con 0x7f740806b2c0 2024-01-16T14:57:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:17.313 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:57:17.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.332+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f740806b2c0 msgr2=0x7f740806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.332+0000 7f7436400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f740806b2c0 0x7f740806d780 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7420006fd0 tx=0x7f7420008040 comp rx=0 tx=0).stop 2024-01-16T14:57:17.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743013e360 msgr2=0x7f7430071750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743013e360 0x7f7430071750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f741800d800 tx=0x7f741800dcc0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 shutdown_connections 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f740806b2c0 0x7f740806d780 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f743015f9c0 0x7f7430071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743013e360 0x7f7430071750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 --2- 172.21.15.141:0/2349143458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743013df80 0x7f743013f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 >> 172.21.15.141:0/2349143458 conn(0x7f743006a800 msgr2=0x7f743006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 shutdown_connections 2024-01-16T14:57:17.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.333+0000 7f7436400640 1 -- 172.21.15.141:0/2349143458 wait complete. 2024-01-16T14:57:17.331 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-16T14:57:16.438499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219508,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755212,"statfs":{"total":773094113280,"available":772869337088,"internally_reserved":0,"allocated":3055616,"data_stored":1815380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.090193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361647+0000","last_change":"2024-01-16T14:57:07.388330+0000","last_active":"2024-01-16T14:57:08.361647+0000","last_peered":"2024-01-16T14:57:08.361647+0000","last_clean":"2024-01-16T14:57:08.361647+0000","last_became_active":"2024-01-16T14:57:07.387937+0000","last_became_peered":"2024-01-16T14:57:07.387937+0000","last_unstale":"2024-01-16T14:57:08.361647+0000","last_undegraded":"2024-01-16T14:57:08.361647+0000","last_fullsized":"2024-01-16T14:57:08.361647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859055+0000","last_change":"2024-01-16T14:57:07.376852+0000","last_active":"2024-01-16T14:57:08.859055+0000","last_peered":"2024-01-16T14:57:08.859055+0000","last_clean":"2024-01-16T14:57:08.859055+0000","last_became_active":"2024-01-16T14:57:07.376562+0000","last_became_peered":"2024-01-16T14:57:07.376562+0000","last_unstale":"2024-01-16T14:57:08.859055+0000","last_undegraded":"2024-01-16T14:57:08.859055+0000","last_fullsized":"2024-01-16T14:57:08.859055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411451+0000","last_change":"2024-01-16T14:57:07.376399+0000","last_active":"2024-01-16T14:57:08.411451+0000","last_peered":"2024-01-16T14:57:08.411451+0000","last_clean":"2024-01-16T14:57:08.411451+0000","last_became_active":"2024-01-16T14:57:07.374502+0000","last_became_peered":"2024-01-16T14:57:07.374502+0000","last_unstale":"2024-01-16T14:57:08.411451+0000","last_undegraded":"2024-01-16T14:57:08.411451+0000","last_fullsized":"2024-01-16T14:57:08.411451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357034+0000","last_change":"2024-01-16T14:57:07.393925+0000","last_active":"2024-01-16T14:57:08.357034+0000","last_peered":"2024-01-16T14:57:08.357034+0000","last_clean":"2024-01-16T14:57:08.357034+0000","last_became_active":"2024-01-16T14:57:07.393608+0000","last_became_peered":"2024-01-16T14:57:07.393608+0000","last_unstale":"2024-01-16T14:57:08.357034+0000","last_undegraded":"2024-01-16T14:57:08.357034+0000","last_fullsized":"2024-01-16T14:57:08.357034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:12.497634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360231+0000","last_change":"2024-01-16T14:57:07.370383+0000","last_active":"2024-01-16T14:57:08.360231+0000","last_peered":"2024-01-16T14:57:08.360231+0000","last_clean":"2024-01-16T14:57:08.360231+0000","last_became_active":"2024-01-16T14:57:07.369925+0000","last_became_peered":"2024-01-16T14:57:07.369925+0000","last_unstale":"2024-01-16T14:57:08.360231+0000","last_undegraded":"2024-01-16T14:57:08.360231+0000","last_fullsized":"2024-01-16T14:57:08.360231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357036+0000","last_change":"2024-01-16T14:57:07.392767+0000","last_active":"2024-01-16T14:57:08.357036+0000","last_peered":"2024-01-16T14:57:08.357036+0000","last_clean":"2024-01-16T14:57:08.357036+0000","last_became_active":"2024-01-16T14:57:07.392457+0000","last_became_peered":"2024-01-16T14:57:07.392457+0000","last_unstale":"2024-01-16T14:57:08.357036+0000","last_undegraded":"2024-01-16T14:57:08.357036+0000","last_fullsized":"2024-01-16T14:57:08.357036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359822+0000","last_change":"2024-01-16T14:57:07.377276+0000","last_active":"2024-01-16T14:57:08.359822+0000","last_peered":"2024-01-16T14:57:08.359822+0000","last_clean":"2024-01-16T14:57:08.359822+0000","last_became_active":"2024-01-16T14:57:07.376937+0000","last_became_peered":"2024-01-16T14:57:07.376937+0000","last_unstale":"2024-01-16T14:57:08.359822+0000","last_undegraded":"2024-01-16T14:57:08.359822+0000","last_fullsized":"2024-01-16T14:57:08.359822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857828+0000","last_change":"2024-01-16T14:57:07.394001+0000","last_active":"2024-01-16T14:57:08.857828+0000","last_peered":"2024-01-16T14:57:08.857828+0000","last_clean":"2024-01-16T14:57:08.857828+0000","last_became_active":"2024-01-16T14:57:07.393670+0000","last_became_peered":"2024-01-16T14:57:07.393670+0000","last_unstale":"2024-01-16T14:57:08.857828+0000","last_undegraded":"2024-01-16T14:57:08.857828+0000","last_fullsized":"2024-01-16T14:57:08.857828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361786+0000","last_change":"2024-01-16T14:57:07.394940+0000","last_active":"2024-01-16T14:57:08.361786+0000","last_peered":"2024-01-16T14:57:08.361786+0000","last_clean":"2024-01-16T14:57:08.361786+0000","last_became_active":"2024-01-16T14:57:07.394335+0000","last_became_peered":"2024-01-16T14:57:07.394335+0000","last_unstale":"2024-01-16T14:57:08.361786+0000","last_undegraded":"2024-01-16T14:57:08.361786+0000","last_fullsized":"2024-01-16T14:57:08.361786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:23.443732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857711+0000","last_change":"2024-01-16T14:57:07.395598+0000","last_active":"2024-01-16T14:57:08.857711+0000","last_peered":"2024-01-16T14:57:08.857711+0000","last_clean":"2024-01-16T14:57:08.857711+0000","last_became_active":"2024-01-16T14:57:07.395367+0000","last_became_peered":"2024-01-16T14:57:07.395367+0000","last_unstale":"2024-01-16T14:57:08.857711+0000","last_undegraded":"2024-01-16T14:57:08.857711+0000","last_fullsized":"2024-01-16T14:57:08.857711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857014+0000","last_change":"2024-01-16T14:57:07.394374+0000","last_active":"2024-01-16T14:57:08.857014+0000","last_peered":"2024-01-16T14:57:08.857014+0000","last_clean":"2024-01-16T14:57:08.857014+0000","last_became_active":"2024-01-16T14:57:07.394125+0000","last_became_peered":"2024-01-16T14:57:07.394125+0000","last_unstale":"2024-01-16T14:57:08.857014+0000","last_undegraded":"2024-01-16T14:57:08.857014+0000","last_fullsized":"2024-01-16T14:57:08.857014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357258+0000","last_change":"2024-01-16T14:57:07.382452+0000","last_active":"2024-01-16T14:57:08.357258+0000","last_peered":"2024-01-16T14:57:08.357258+0000","last_clean":"2024-01-16T14:57:08.357258+0000","last_became_active":"2024-01-16T14:57:07.381985+0000","last_became_peered":"2024-01-16T14:57:07.381985+0000","last_unstale":"2024-01-16T14:57:08.357258+0000","last_undegraded":"2024-01-16T14:57:08.357258+0000","last_fullsized":"2024-01-16T14:57:08.357258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:31.589732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858575+0000","last_change":"2024-01-16T14:57:07.385013+0000","last_active":"2024-01-16T14:57:08.858575+0000","last_peered":"2024-01-16T14:57:08.858575+0000","last_clean":"2024-01-16T14:57:08.858575+0000","last_became_active":"2024-01-16T14:57:07.384705+0000","last_became_peered":"2024-01-16T14:57:07.384705+0000","last_unstale":"2024-01-16T14:57:08.858575+0000","last_undegraded":"2024-01-16T14:57:08.858575+0000","last_fullsized":"2024-01-16T14:57:08.858575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411099+0000","last_change":"2024-01-16T14:57:07.382942+0000","last_active":"2024-01-16T14:57:08.411099+0000","last_peered":"2024-01-16T14:57:08.411099+0000","last_clean":"2024-01-16T14:57:08.411099+0000","last_became_active":"2024-01-16T14:57:07.382609+0000","last_became_peered":"2024-01-16T14:57:07.382609+0000","last_unstale":"2024-01-16T14:57:08.411099+0000","last_undegraded":"2024-01-16T14:57:08.411099+0000","last_fullsized":"2024-01-16T14:57:08.411099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:07.098165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856153+0000","last_change":"2024-01-16T14:57:07.369499+0000","last_active":"2024-01-16T14:57:08.856153+0000","last_peered":"2024-01-16T14:57:08.856153+0000","last_clean":"2024-01-16T14:57:08.856153+0000","last_became_active":"2024-01-16T14:57:07.369089+0000","last_became_peered":"2024-01-16T14:57:07.369089+0000","last_unstale":"2024-01-16T14:57:08.856153+0000","last_undegraded":"2024-01-16T14:57:08.856153+0000","last_fullsized":"2024-01-16T14:57:08.856153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:45:27.672659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.355734+0000","last_change":"2024-01-16T14:57:07.392502+0000","last_active":"2024-01-16T14:57:08.355734+0000","last_peered":"2024-01-16T14:57:08.355734+0000","last_clean":"2024-01-16T14:57:08.355734+0000","last_became_active":"2024-01-16T14:57:07.392065+0000","last_became_peered":"2024-01-16T14:57:07.392065+0000","last_unstale":"2024-01-16T14:57:08.355734+0000","last_undegraded":"2024-01-16T14:57:08.355734+0000","last_fullsized":"2024-01-16T14:57:08.355734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358909+0000","last_change":"2024-01-16T14:57:07.394712+0000","last_active":"2024-01-16T14:57:08.358909+0000","last_peered":"2024-01-16T14:57:08.358909+0000","last_clean":"2024-01-16T14:57:08.358909+0000","last_became_active":"2024-01-16T14:57:07.393954+0000","last_became_peered":"2024-01-16T14:57:07.393954+0000","last_unstale":"2024-01-16T14:57:08.358909+0000","last_undegraded":"2024-01-16T14:57:08.358909+0000","last_fullsized":"2024-01-16T14:57:08.358909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859744+0000","last_change":"2024-01-16T14:57:07.384626+0000","last_active":"2024-01-16T14:57:08.859744+0000","last_peered":"2024-01-16T14:57:08.859744+0000","last_clean":"2024-01-16T14:57:08.859744+0000","last_became_active":"2024-01-16T14:57:07.383860+0000","last_became_peered":"2024-01-16T14:57:07.383860+0000","last_unstale":"2024-01-16T14:57:08.859744+0000","last_undegraded":"2024-01-16T14:57:08.859744+0000","last_fullsized":"2024-01-16T14:57:08.859744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361107+0000","last_change":"2024-01-16T14:57:07.375013+0000","last_active":"2024-01-16T14:57:08.361107+0000","last_peered":"2024-01-16T14:57:08.361107+0000","last_clean":"2024-01-16T14:57:08.361107+0000","last_became_active":"2024-01-16T14:57:07.373630+0000","last_became_peered":"2024-01-16T14:57:07.373630+0000","last_unstale":"2024-01-16T14:57:08.361107+0000","last_undegraded":"2024-01-16T14:57:08.361107+0000","last_fullsized":"2024-01-16T14:57:08.361107+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857112+0000","last_change":"2024-01-16T14:57:07.375790+0000","last_active":"2024-01-16T14:57:08.857112+0000","last_peered":"2024-01-16T14:57:08.857112+0000","last_clean":"2024-01-16T14:57:08.857112+0000","last_became_active":"2024-01-16T14:57:07.375533+0000","last_became_peered":"2024-01-16T14:57:07.375533+0000","last_unstale":"2024-01-16T14:57:08.857112+0000","last_undegraded":"2024-01-16T14:57:08.857112+0000","last_fullsized":"2024-01-16T14:57:08.857112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411312+0000","last_change":"2024-01-16T14:57:07.378765+0000","last_active":"2024-01-16T14:57:08.411312+0000","last_peered":"2024-01-16T14:57:08.411312+0000","last_clean":"2024-01-16T14:57:08.411312+0000","last_became_active":"2024-01-16T14:57:07.377672+0000","last_became_peered":"2024-01-16T14:57:07.377672+0000","last_unstale":"2024-01-16T14:57:08.411312+0000","last_undegraded":"2024-01-16T14:57:08.411312+0000","last_fullsized":"2024-01-16T14:57:08.411312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357439+0000","last_change":"2024-01-16T14:57:07.392203+0000","last_active":"2024-01-16T14:57:08.357439+0000","last_peered":"2024-01-16T14:57:08.357439+0000","last_clean":"2024-01-16T14:57:08.357439+0000","last_became_active":"2024-01-16T14:57:07.391913+0000","last_became_peered":"2024-01-16T14:57:07.391913+0000","last_unstale":"2024-01-16T14:57:08.357439+0000","last_undegraded":"2024-01-16T14:57:08.357439+0000","last_fullsized":"2024-01-16T14:57:08.357439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858112+0000","last_change":"2024-01-16T14:57:07.383055+0000","last_active":"2024-01-16T14:57:08.858112+0000","last_peered":"2024-01-16T14:57:08.858112+0000","last_clean":"2024-01-16T14:57:08.858112+0000","last_became_active":"2024-01-16T14:57:07.382181+0000","last_became_peered":"2024-01-16T14:57:07.382181+0000","last_unstale":"2024-01-16T14:57:08.858112+0000","last_undegraded":"2024-01-16T14:57:08.858112+0000","last_fullsized":"2024-01-16T14:57:08.858112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.356639+0000","last_change":"2024-01-16T14:57:07.391594+0000","last_active":"2024-01-16T14:57:08.356639+0000","last_peered":"2024-01-16T14:57:08.356639+0000","last_clean":"2024-01-16T14:57:08.356639+0000","last_became_active":"2024-01-16T14:57:07.391116+0000","last_became_peered":"2024-01-16T14:57:07.391116+0000","last_unstale":"2024-01-16T14:57:08.356639+0000","last_undegraded":"2024-01-16T14:57:08.356639+0000","last_fullsized":"2024-01-16T14:57:08.356639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859255+0000","last_change":"2024-01-16T14:57:07.383060+0000","last_active":"2024-01-16T14:57:08.859255+0000","last_peered":"2024-01-16T14:57:08.859255+0000","last_clean":"2024-01-16T14:57:08.859255+0000","last_became_active":"2024-01-16T14:57:07.382192+0000","last_became_peered":"2024-01-16T14:57:07.382192+0000","last_unstale":"2024-01-16T14:57:08.859255+0000","last_undegraded":"2024-01-16T14:57:08.859255+0000","last_fullsized":"2024-01-16T14:57:08.859255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:34.791403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857952+0000","last_change":"2024-01-16T14:57:07.384727+0000","last_active":"2024-01-16T14:57:08.857952+0000","last_peered":"2024-01-16T14:57:08.857952+0000","last_clean":"2024-01-16T14:57:08.857952+0000","last_became_active":"2024-01-16T14:57:07.383334+0000","last_became_peered":"2024-01-16T14:57:07.383334+0000","last_unstale":"2024-01-16T14:57:08.857952+0000","last_undegraded":"2024-01-16T14:57:08.857952+0000","last_fullsized":"2024-01-16T14:57:08.857952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856861+0000","last_change":"2024-01-16T14:57:07.376359+0000","last_active":"2024-01-16T14:57:08.856861+0000","last_peered":"2024-01-16T14:57:08.856861+0000","last_clean":"2024-01-16T14:57:08.856861+0000","last_became_active":"2024-01-16T14:57:07.376074+0000","last_became_peered":"2024-01-16T14:57:07.376074+0000","last_unstale":"2024-01-16T14:57:08.856861+0000","last_undegraded":"2024-01-16T14:57:08.856861+0000","last_fullsized":"2024-01-16T14:57:08.856861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858616+0000","last_change":"2024-01-16T14:57:07.387458+0000","last_active":"2024-01-16T14:57:08.858616+0000","last_peered":"2024-01-16T14:57:08.858616+0000","last_clean":"2024-01-16T14:57:08.858616+0000","last_became_active":"2024-01-16T14:57:07.387221+0000","last_became_peered":"2024-01-16T14:57:07.387221+0000","last_unstale":"2024-01-16T14:57:08.858616+0000","last_undegraded":"2024-01-16T14:57:08.858616+0000","last_fullsized":"2024-01-16T14:57:08.858616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411330+0000","last_change":"2024-01-16T14:57:07.377884+0000","last_active":"2024-01-16T14:57:08.411330+0000","last_peered":"2024-01-16T14:57:08.411330+0000","last_clean":"2024-01-16T14:57:08.411330+0000","last_became_active":"2024-01-16T14:57:07.377585+0000","last_became_peered":"2024-01-16T14:57:07.377585+0000","last_unstale":"2024-01-16T14:57:08.411330+0000","last_undegraded":"2024-01-16T14:57:08.411330+0000","last_fullsized":"2024-01-16T14:57:08.411330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358051+0000","last_change":"2024-01-16T14:57:07.389650+0000","last_active":"2024-01-16T14:57:08.358051+0000","last_peered":"2024-01-16T14:57:08.358051+0000","last_clean":"2024-01-16T14:57:08.358051+0000","last_became_active":"2024-01-16T14:57:07.389388+0000","last_became_peered":"2024-01-16T14:57:07.389388+0000","last_unstale":"2024-01-16T14:57:08.358051+0000","last_undegraded":"2024-01-16T14:57:08.358051+0000","last_fullsized":"2024-01-16T14:57:08.358051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858338+0000","last_change":"2024-01-16T14:57:07.393121+0000","last_active":"2024-01-16T14:57:08.858338+0000","last_peered":"2024-01-16T14:57:08.858338+0000","last_clean":"2024-01-16T14:57:08.858338+0000","last_became_active":"2024-01-16T14:57:07.392839+0000","last_became_peered":"2024-01-16T14:57:07.392839+0000","last_unstale":"2024-01-16T14:57:08.858338+0000","last_undegraded":"2024-01-16T14:57:08.858338+0000","last_fullsized":"2024-01-16T14:57:08.858338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856756+0000","last_change":"2024-01-16T14:57:07.386156+0000","last_active":"2024-01-16T14:57:08.856756+0000","last_peered":"2024-01-16T14:57:08.856756+0000","last_clean":"2024-01-16T14:57:08.856756+0000","last_became_active":"2024-01-16T14:57:07.385684+0000","last_became_peered":"2024-01-16T14:57:07.385684+0000","last_unstale":"2024-01-16T14:57:08.856756+0000","last_undegraded":"2024-01-16T14:57:08.856756+0000","last_fullsized":"2024-01-16T14:57:08.856756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357739+0000","last_change":"2024-01-16T14:57:07.372196+0000","last_active":"2024-01-16T14:57:08.357739+0000","last_peered":"2024-01-16T14:57:08.357739+0000","last_clean":"2024-01-16T14:57:08.357739+0000","last_became_active":"2024-01-16T14:57:07.369065+0000","last_became_peered":"2024-01-16T14:57:07.369065+0000","last_unstale":"2024-01-16T14:57:08.357739+0000","last_undegraded":"2024-01-16T14:57:08.357739+0000","last_fullsized":"2024-01-16T14:57:08.357739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858827+0000","last_change":"2024-01-16T14:57:06.556368+0000","last_active":"2024-01-16T14:57:08.858827+0000","last_peered":"2024-01-16T14:57:08.858827+0000","last_clean":"2024-01-16T14:57:08.858827+0000","last_became_active":"2024-01-16T14:57:06.555953+0000","last_became_peered":"2024-01-16T14:57:06.555953+0000","last_unstale":"2024-01-16T14:57:08.858827+0000","last_undegraded":"2024-01-16T14:57:08.858827+0000","last_fullsized":"2024-01-16T14:57:08.858827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:45.773231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411448+0000","last_change":"2024-01-16T14:57:07.383151+0000","last_active":"2024-01-16T14:57:08.411448+0000","last_peered":"2024-01-16T14:57:08.411448+0000","last_clean":"2024-01-16T14:57:08.411448+0000","last_became_active":"2024-01-16T14:57:07.381555+0000","last_became_peered":"2024-01-16T14:57:07.381555+0000","last_unstale":"2024-01-16T14:57:08.411448+0000","last_undegraded":"2024-01-16T14:57:08.411448+0000","last_fullsized":"2024-01-16T14:57:08.411448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856949+0000","last_change":"2024-01-16T14:57:06.555306+0000","last_active":"2024-01-16T14:57:08.856949+0000","last_peered":"2024-01-16T14:57:08.856949+0000","last_clean":"2024-01-16T14:57:08.856949+0000","last_became_active":"2024-01-16T14:57:06.555063+0000","last_became_peered":"2024-01-16T14:57:06.555063+0000","last_unstale":"2024-01-16T14:57:08.856949+0000","last_undegraded":"2024-01-16T14:57:08.856949+0000","last_fullsized":"2024-01-16T14:57:08.856949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:07.271545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857979+0000","last_change":"2024-01-16T14:57:06.555279+0000","last_active":"2024-01-16T14:57:08.857979+0000","last_peered":"2024-01-16T14:57:08.857979+0000","last_clean":"2024-01-16T14:57:08.857979+0000","last_became_active":"2024-01-16T14:57:06.554559+0000","last_became_peered":"2024-01-16T14:57:06.554559+0000","last_unstale":"2024-01-16T14:57:08.857979+0000","last_undegraded":"2024-01-16T14:57:08.857979+0000","last_fullsized":"2024-01-16T14:57:08.857979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857830+0000","last_change":"2024-01-16T14:57:07.382661+0000","last_active":"2024-01-16T14:57:08.857830+0000","last_peered":"2024-01-16T14:57:08.857830+0000","last_clean":"2024-01-16T14:57:08.857830+0000","last_became_active":"2024-01-16T14:57:07.382055+0000","last_became_peered":"2024-01-16T14:57:07.382055+0000","last_unstale":"2024-01-16T14:57:08.857830+0000","last_undegraded":"2024-01-16T14:57:08.857830+0000","last_fullsized":"2024-01-16T14:57:08.857830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357563+0000","last_change":"2024-01-16T14:57:06.558562+0000","last_active":"2024-01-16T14:57:08.357563+0000","last_peered":"2024-01-16T14:57:08.357563+0000","last_clean":"2024-01-16T14:57:08.357563+0000","last_became_active":"2024-01-16T14:57:06.558166+0000","last_became_peered":"2024-01-16T14:57:06.558166+0000","last_unstale":"2024-01-16T14:57:08.357563+0000","last_undegraded":"2024-01-16T14:57:08.357563+0000","last_fullsized":"2024-01-16T14:57:08.357563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:31.817106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855928+0000","last_change":"2024-01-16T14:57:07.377917+0000","last_active":"2024-01-16T14:57:08.855928+0000","last_peered":"2024-01-16T14:57:08.855928+0000","last_clean":"2024-01-16T14:57:08.855928+0000","last_became_active":"2024-01-16T14:57:07.377601+0000","last_became_peered":"2024-01-16T14:57:07.377601+0000","last_unstale":"2024-01-16T14:57:08.855928+0000","last_undegraded":"2024-01-16T14:57:08.855928+0000","last_fullsized":"2024-01-16T14:57:08.855928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:32.968418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857555+0000","last_change":"2024-01-16T14:57:06.563826+0000","last_active":"2024-01-16T14:57:08.857555+0000","last_peered":"2024-01-16T14:57:08.857555+0000","last_clean":"2024-01-16T14:57:08.857555+0000","last_became_active":"2024-01-16T14:57:06.563529+0000","last_became_peered":"2024-01-16T14:57:06.563529+0000","last_unstale":"2024-01-16T14:57:08.857555+0000","last_undegraded":"2024-01-16T14:57:08.857555+0000","last_fullsized":"2024-01-16T14:57:08.857555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358434+0000","last_change":"2024-01-16T14:57:07.391665+0000","last_active":"2024-01-16T14:57:08.358434+0000","last_peered":"2024-01-16T14:57:08.358434+0000","last_clean":"2024-01-16T14:57:08.358434+0000","last_became_active":"2024-01-16T14:57:07.391071+0000","last_became_peered":"2024-01-16T14:57:07.391071+0000","last_unstale":"2024-01-16T14:57:08.358434+0000","last_undegraded":"2024-01-16T14:57:08.358434+0000","last_fullsized":"2024-01-16T14:57:08.358434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358657+0000","last_change":"2024-01-16T14:57:06.560447+0000","last_active":"2024-01-16T14:57:08.358657+0000","last_peered":"2024-01-16T14:57:08.358657+0000","last_clean":"2024-01-16T14:57:08.358657+0000","last_became_active":"2024-01-16T14:57:06.560116+0000","last_became_peered":"2024-01-16T14:57:06.560116+0000","last_unstale":"2024-01-16T14:57:08.358657+0000","last_undegraded":"2024-01-16T14:57:08.358657+0000","last_fullsized":"2024-01-16T14:57:08.358657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:33.296388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411729+0000","last_change":"2024-01-16T14:57:07.396020+0000","last_active":"2024-01-16T14:57:08.411729+0000","last_peered":"2024-01-16T14:57:08.411729+0000","last_clean":"2024-01-16T14:57:08.411729+0000","last_became_active":"2024-01-16T14:57:07.395685+0000","last_became_peered":"2024-01-16T14:57:07.395685+0000","last_unstale":"2024-01-16T14:57:08.411729+0000","last_undegraded":"2024-01-16T14:57:08.411729+0000","last_fullsized":"2024-01-16T14:57:08.411729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857123+0000","last_change":"2024-01-16T14:57:06.553623+0000","last_active":"2024-01-16T14:57:08.857123+0000","last_peered":"2024-01-16T14:57:08.857123+0000","last_clean":"2024-01-16T14:57:08.857123+0000","last_became_active":"2024-01-16T14:57:06.553173+0000","last_became_peered":"2024-01-16T14:57:06.553173+0000","last_unstale":"2024-01-16T14:57:08.857123+0000","last_undegraded":"2024-01-16T14:57:08.857123+0000","last_fullsized":"2024-01-16T14:57:08.857123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357536+0000","last_change":"2024-01-16T14:57:07.382256+0000","last_active":"2024-01-16T14:57:08.357536+0000","last_peered":"2024-01-16T14:57:08.357536+0000","last_clean":"2024-01-16T14:57:08.357536+0000","last_became_active":"2024-01-16T14:57:07.381640+0000","last_became_peered":"2024-01-16T14:57:07.381640+0000","last_unstale":"2024-01-16T14:57:08.357536+0000","last_undegraded":"2024-01-16T14:57:08.357536+0000","last_fullsized":"2024-01-16T14:57:08.357536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:14.206644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857022+0000","last_change":"2024-01-16T14:57:06.556835+0000","last_active":"2024-01-16T14:57:08.857022+0000","last_peered":"2024-01-16T14:57:08.857022+0000","last_clean":"2024-01-16T14:57:08.857022+0000","last_became_active":"2024-01-16T14:57:06.556483+0000","last_became_peered":"2024-01-16T14:57:06.556483+0000","last_unstale":"2024-01-16T14:57:08.857022+0000","last_undegraded":"2024-01-16T14:57:08.857022+0000","last_fullsized":"2024-01-16T14:57:08.857022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:01.863989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360686+0000","last_change":"2024-01-16T14:57:07.380047+0000","last_active":"2024-01-16T14:57:08.360686+0000","last_peered":"2024-01-16T14:57:08.360686+0000","last_clean":"2024-01-16T14:57:08.360686+0000","last_became_active":"2024-01-16T14:57:07.379698+0000","last_became_peered":"2024-01-16T14:57:07.379698+0000","last_unstale":"2024-01-16T14:57:08.360686+0000","last_undegraded":"2024-01-16T14:57:08.360686+0000","last_fullsized":"2024-01-16T14:57:08.360686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411453+0000","last_change":"2024-01-16T14:57:06.555448+0000","last_active":"2024-01-16T14:57:08.411453+0000","last_peered":"2024-01-16T14:57:08.411453+0000","last_clean":"2024-01-16T14:57:08.411453+0000","last_became_active":"2024-01-16T14:57:06.555097+0000","last_became_peered":"2024-01-16T14:57:06.555097+0000","last_unstale":"2024-01-16T14:57:08.411453+0000","last_undegraded":"2024-01-16T14:57:08.411453+0000","last_fullsized":"2024-01-16T14:57:08.411453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:23.858452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411172+0000","last_change":"2024-01-16T14:57:07.380436+0000","last_active":"2024-01-16T14:57:08.411172+0000","last_peered":"2024-01-16T14:57:08.411172+0000","last_clean":"2024-01-16T14:57:08.411172+0000","last_became_active":"2024-01-16T14:57:07.380148+0000","last_became_peered":"2024-01-16T14:57:07.380148+0000","last_unstale":"2024-01-16T14:57:08.411172+0000","last_undegraded":"2024-01-16T14:57:08.411172+0000","last_fullsized":"2024-01-16T14:57:08.411172+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358031+0000","last_change":"2024-01-16T14:57:06.549803+0000","last_active":"2024-01-16T14:57:08.358031+0000","last_peered":"2024-01-16T14:57:08.358031+0000","last_clean":"2024-01-16T14:57:08.358031+0000","last_became_active":"2024-01-16T14:57:06.549362+0000","last_became_peered":"2024-01-16T14:57:06.549362+0000","last_unstale":"2024-01-16T14:57:08.358031+0000","last_undegraded":"2024-01-16T14:57:08.358031+0000","last_fullsized":"2024-01-16T14:57:08.358031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360491+0000","last_change":"2024-01-16T14:57:07.384510+0000","last_active":"2024-01-16T14:57:08.360491+0000","last_peered":"2024-01-16T14:57:08.360491+0000","last_clean":"2024-01-16T14:57:08.360491+0000","last_became_active":"2024-01-16T14:57:07.384214+0000","last_became_peered":"2024-01-16T14:57:07.384214+0000","last_unstale":"2024-01-16T14:57:08.360491+0000","last_undegraded":"2024-01-16T14:57:08.360491+0000","last_fullsized":"2024-01-16T14:57:08.360491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856634+0000","last_change":"2024-01-16T14:57:06.562353+0000","last_active":"2024-01-16T14:57:08.856634+0000","last_peered":"2024-01-16T14:57:08.856634+0000","last_clean":"2024-01-16T14:57:08.856634+0000","last_became_active":"2024-01-16T14:57:06.562043+0000","last_became_peered":"2024-01-16T14:57:06.562043+0000","last_unstale":"2024-01-16T14:57:08.856634+0000","last_undegraded":"2024-01-16T14:57:08.856634+0000","last_fullsized":"2024-01-16T14:57:08.856634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:25.879858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856809+0000","last_change":"2024-01-16T14:57:07.383191+0000","last_active":"2024-01-16T14:57:08.856809+0000","last_peered":"2024-01-16T14:57:08.856809+0000","last_clean":"2024-01-16T14:57:08.856809+0000","last_became_active":"2024-01-16T14:57:07.379683+0000","last_became_peered":"2024-01-16T14:57:07.379683+0000","last_unstale":"2024-01-16T14:57:08.856809+0000","last_undegraded":"2024-01-16T14:57:08.856809+0000","last_fullsized":"2024-01-16T14:57:08.856809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856935+0000","last_change":"2024-01-16T14:57:06.548684+0000","last_active":"2024-01-16T14:57:08.856935+0000","last_peered":"2024-01-16T14:57:08.856935+0000","last_clean":"2024-01-16T14:57:08.856935+0000","last_became_active":"2024-01-16T14:57:06.547835+0000","last_became_peered":"2024-01-16T14:57:06.547835+0000","last_unstale":"2024-01-16T14:57:08.856935+0000","last_undegraded":"2024-01-16T14:57:08.856935+0000","last_fullsized":"2024-01-16T14:57:08.856935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:46.715195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360277+0000","last_change":"2024-01-16T14:57:07.384606+0000","last_active":"2024-01-16T14:57:08.360277+0000","last_peered":"2024-01-16T14:57:08.360277+0000","last_clean":"2024-01-16T14:57:08.360277+0000","last_became_active":"2024-01-16T14:57:07.384300+0000","last_became_peered":"2024-01-16T14:57:07.384300+0000","last_unstale":"2024-01-16T14:57:08.360277+0000","last_undegraded":"2024-01-16T14:57:08.360277+0000","last_fullsized":"2024-01-16T14:57:08.360277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359317+0000","last_change":"2024-01-16T14:57:06.555188+0000","last_active":"2024-01-16T14:57:08.359317+0000","last_peered":"2024-01-16T14:57:08.359317+0000","last_clean":"2024-01-16T14:57:08.359317+0000","last_became_active":"2024-01-16T14:57:06.554904+0000","last_became_peered":"2024-01-16T14:57:06.554904+0000","last_unstale":"2024-01-16T14:57:08.359317+0000","last_undegraded":"2024-01-16T14:57:08.359317+0000","last_fullsized":"2024-01-16T14:57:08.359317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:59.674406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360712+0000","last_change":"2024-01-16T14:57:07.386645+0000","last_active":"2024-01-16T14:57:08.360712+0000","last_peered":"2024-01-16T14:57:08.360712+0000","last_clean":"2024-01-16T14:57:08.360712+0000","last_became_active":"2024-01-16T14:57:07.384338+0000","last_became_peered":"2024-01-16T14:57:07.384338+0000","last_unstale":"2024-01-16T14:57:08.360712+0000","last_undegraded":"2024-01-16T14:57:08.360712+0000","last_fullsized":"2024-01-16T14:57:08.360712+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859980+0000","last_change":"2024-01-16T14:57:06.549964+0000","last_active":"2024-01-16T14:57:08.859980+0000","last_peered":"2024-01-16T14:57:08.859980+0000","last_clean":"2024-01-16T14:57:08.859980+0000","last_became_active":"2024-01-16T14:57:06.549594+0000","last_became_peered":"2024-01-16T14:57:06.549594+0000","last_unstale":"2024-01-16T14:57:08.859980+0000","last_undegraded":"2024-01-16T14:57:08.859980+0000","last_fullsized":"2024-01-16T14:57:08.859980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361966+0000","last_change":"2024-01-16T14:57:07.385296+0000","last_active":"2024-01-16T14:57:08.361966+0000","last_peered":"2024-01-16T14:57:08.361966+0000","last_clean":"2024-01-16T14:57:08.361966+0000","last_became_active":"2024-01-16T14:57:07.385019+0000","last_became_peered":"2024-01-16T14:57:07.385019+0000","last_unstale":"2024-01-16T14:57:08.361966+0000","last_undegraded":"2024-01-16T14:57:08.361966+0000","last_fullsized":"2024-01-16T14:57:08.361966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:08.100477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857357+0000","last_change":"2024-01-16T14:57:06.558586+0000","last_active":"2024-01-16T14:57:08.857357+0000","last_peered":"2024-01-16T14:57:08.857357+0000","last_clean":"2024-01-16T14:57:08.857357+0000","last_became_active":"2024-01-16T14:57:06.558143+0000","last_became_peered":"2024-01-16T14:57:06.558143+0000","last_unstale":"2024-01-16T14:57:08.857357+0000","last_undegraded":"2024-01-16T14:57:08.857357+0000","last_fullsized":"2024-01-16T14:57:08.857357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357386+0000","last_change":"2024-01-16T14:57:07.393946+0000","last_active":"2024-01-16T14:57:08.357386+0000","last_peered":"2024-01-16T14:57:08.357386+0000","last_clean":"2024-01-16T14:57:08.357386+0000","last_became_active":"2024-01-16T14:57:07.393108+0000","last_became_peered":"2024-01-16T14:57:07.393108+0000","last_unstale":"2024-01-16T14:57:08.357386+0000","last_undegraded":"2024-01-16T14:57:08.357386+0000","last_fullsized":"2024-01-16T14:57:08.357386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858814+0000","last_change":"2024-01-16T14:57:06.555365+0000","last_active":"2024-01-16T14:57:08.858814+0000","last_peered":"2024-01-16T14:57:08.858814+0000","last_clean":"2024-01-16T14:57:08.858814+0000","last_became_active":"2024-01-16T14:57:06.551851+0000","last_became_peered":"2024-01-16T14:57:06.551851+0000","last_unstale":"2024-01-16T14:57:08.858814+0000","last_undegraded":"2024-01-16T14:57:08.858814+0000","last_fullsized":"2024-01-16T14:57:08.858814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360922+0000","last_change":"2024-01-16T14:57:07.397246+0000","last_active":"2024-01-16T14:57:08.360922+0000","last_peered":"2024-01-16T14:57:08.360922+0000","last_clean":"2024-01-16T14:57:08.360922+0000","last_became_active":"2024-01-16T14:57:07.394277+0000","last_became_peered":"2024-01-16T14:57:07.394277+0000","last_unstale":"2024-01-16T14:57:08.360922+0000","last_undegraded":"2024-01-16T14:57:08.360922+0000","last_fullsized":"2024-01-16T14:57:08.360922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:57:07.357749+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:57:07.357749+0000","last_peered":"2024-01-16T14:57:07.357749+0000","last_clean":"2024-01-16T14:57:07.357749+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:57:07.357749+0000","last_undegraded":"2024-01-16T14:57:07.357749+0000","last_fullsized":"2024-01-16T14:57:07.357749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357443+0000","last_change":"2024-01-16T14:57:06.547364+0000","last_active":"2024-01-16T14:57:08.357443+0000","last_peered":"2024-01-16T14:57:08.357443+0000","last_clean":"2024-01-16T14:57:08.357443+0000","last_became_active":"2024-01-16T14:57:06.546800+0000","last_became_peered":"2024-01-16T14:57:06.546800+0000","last_unstale":"2024-01-16T14:57:08.357443+0000","last_undegraded":"2024-01-16T14:57:08.357443+0000","last_fullsized":"2024-01-16T14:57:08.357443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:27.163769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858385+0000","last_change":"2024-01-16T14:57:07.395614+0000","last_active":"2024-01-16T14:57:08.858385+0000","last_peered":"2024-01-16T14:57:08.858385+0000","last_clean":"2024-01-16T14:57:08.858385+0000","last_became_active":"2024-01-16T14:57:07.395330+0000","last_became_peered":"2024-01-16T14:57:07.395330+0000","last_unstale":"2024-01-16T14:57:08.858385+0000","last_undegraded":"2024-01-16T14:57:08.858385+0000","last_fullsized":"2024-01-16T14:57:08.858385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359578+0000","last_change":"2024-01-16T14:57:06.548309+0000","last_active":"2024-01-16T14:57:08.359578+0000","last_peered":"2024-01-16T14:57:08.359578+0000","last_clean":"2024-01-16T14:57:08.359578+0000","last_became_active":"2024-01-16T14:57:06.547952+0000","last_became_peered":"2024-01-16T14:57:06.547952+0000","last_unstale":"2024-01-16T14:57:08.359578+0000","last_undegraded":"2024-01-16T14:57:08.359578+0000","last_fullsized":"2024-01-16T14:57:08.359578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.410994+0000","last_change":"2024-01-16T14:57:07.375198+0000","last_active":"2024-01-16T14:57:08.410994+0000","last_peered":"2024-01-16T14:57:08.410994+0000","last_clean":"2024-01-16T14:57:08.410994+0000","last_became_active":"2024-01-16T14:57:07.374043+0000","last_became_peered":"2024-01-16T14:57:07.374043+0000","last_unstale":"2024-01-16T14:57:08.410994+0000","last_undegraded":"2024-01-16T14:57:08.410994+0000","last_fullsized":"2024-01-16T14:57:08.410994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:11.783582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857406+0000","last_change":"2024-01-16T14:57:06.553851+0000","last_active":"2024-01-16T14:57:08.857406+0000","last_peered":"2024-01-16T14:57:08.857406+0000","last_clean":"2024-01-16T14:57:08.857406+0000","last_became_active":"2024-01-16T14:57:06.552712+0000","last_became_peered":"2024-01-16T14:57:06.552712+0000","last_unstale":"2024-01-16T14:57:08.857406+0000","last_undegraded":"2024-01-16T14:57:08.857406+0000","last_fullsized":"2024-01-16T14:57:08.857406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855802+0000","last_change":"2024-01-16T14:57:07.372041+0000","last_active":"2024-01-16T14:57:08.855802+0000","last_peered":"2024-01-16T14:57:08.855802+0000","last_clean":"2024-01-16T14:57:08.855802+0000","last_became_active":"2024-01-16T14:57:07.371750+0000","last_became_peered":"2024-01-16T14:57:07.371750+0000","last_unstale":"2024-01-16T14:57:08.855802+0000","last_undegraded":"2024-01-16T14:57:08.855802+0000","last_fullsized":"2024-01-16T14:57:08.855802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:53.804355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358274+0000","last_change":"2024-01-16T14:57:06.548768+0000","last_active":"2024-01-16T14:57:08.358274+0000","last_peered":"2024-01-16T14:57:08.358274+0000","last_clean":"2024-01-16T14:57:08.358274+0000","last_became_active":"2024-01-16T14:57:06.548446+0000","last_became_peered":"2024-01-16T14:57:06.548446+0000","last_unstale":"2024-01-16T14:57:08.358274+0000","last_undegraded":"2024-01-16T14:57:08.358274+0000","last_fullsized":"2024-01-16T14:57:08.358274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:06.341461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358850+0000","last_change":"2024-01-16T14:57:07.378261+0000","last_active":"2024-01-16T14:57:08.358850+0000","last_peered":"2024-01-16T14:57:08.358850+0000","last_clean":"2024-01-16T14:57:08.358850+0000","last_became_active":"2024-01-16T14:57:07.377940+0000","last_became_peered":"2024-01-16T14:57:07.377940+0000","last_unstale":"2024-01-16T14:57:08.358850+0000","last_undegraded":"2024-01-16T14:57:08.358850+0000","last_fullsized":"2024-01-16T14:57:08.358850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856741+0000","last_change":"2024-01-16T14:57:06.545628+0000","last_active":"2024-01-16T14:57:08.856741+0000","last_peered":"2024-01-16T14:57:08.856741+0000","last_clean":"2024-01-16T14:57:08.856741+0000","last_became_active":"2024-01-16T14:57:06.545341+0000","last_became_peered":"2024-01-16T14:57:06.545341+0000","last_unstale":"2024-01-16T14:57:08.856741+0000","last_undegraded":"2024-01-16T14:57:08.856741+0000","last_fullsized":"2024-01-16T14:57:08.856741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:54.829524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358936+0000","last_change":"2024-01-16T14:57:07.392468+0000","last_active":"2024-01-16T14:57:08.358936+0000","last_peered":"2024-01-16T14:57:08.358936+0000","last_clean":"2024-01-16T14:57:08.358936+0000","last_became_active":"2024-01-16T14:57:07.392091+0000","last_became_peered":"2024-01-16T14:57:07.392091+0000","last_unstale":"2024-01-16T14:57:08.358936+0000","last_undegraded":"2024-01-16T14:57:08.358936+0000","last_fullsized":"2024-01-16T14:57:08.358936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361159+0000","last_change":"2024-01-16T14:57:06.551276+0000","last_active":"2024-01-16T14:57:08.361159+0000","last_peered":"2024-01-16T14:57:08.361159+0000","last_clean":"2024-01-16T14:57:08.361159+0000","last_became_active":"2024-01-16T14:57:06.550951+0000","last_became_peered":"2024-01-16T14:57:06.550951+0000","last_unstale":"2024-01-16T14:57:08.361159+0000","last_undegraded":"2024-01-16T14:57:08.361159+0000","last_fullsized":"2024-01-16T14:57:08.361159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:15.986936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856799+0000","last_change":"2024-01-16T14:57:07.370960+0000","last_active":"2024-01-16T14:57:08.856799+0000","last_peered":"2024-01-16T14:57:08.856799+0000","last_clean":"2024-01-16T14:57:08.856799+0000","last_became_active":"2024-01-16T14:57:07.370635+0000","last_became_peered":"2024-01-16T14:57:07.370635+0000","last_unstale":"2024-01-16T14:57:08.856799+0000","last_undegraded":"2024-01-16T14:57:08.856799+0000","last_fullsized":"2024-01-16T14:57:08.856799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:36.350912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:09.484382+0000","last_change":"2024-01-16T14:57:06.553886+0000","last_active":"2024-01-16T14:57:09.484382+0000","last_peered":"2024-01-16T14:57:09.484382+0000","last_clean":"2024-01-16T14:57:09.484382+0000","last_became_active":"2024-01-16T14:57:06.553590+0000","last_became_peered":"2024-01-16T14:57:06.553590+0000","last_unstale":"2024-01-16T14:57:09.484382+0000","last_undegraded":"2024-01-16T14:57:09.484382+0000","last_fullsized":"2024-01-16T14:57:09.484382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:16.749765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859350+0000","last_change":"2024-01-16T14:57:07.388410+0000","last_active":"2024-01-16T14:57:08.859350+0000","last_peered":"2024-01-16T14:57:08.859350+0000","last_clean":"2024-01-16T14:57:08.859350+0000","last_became_active":"2024-01-16T14:57:07.388132+0000","last_became_peered":"2024-01-16T14:57:07.388132+0000","last_unstale":"2024-01-16T14:57:08.859350+0000","last_undegraded":"2024-01-16T14:57:08.859350+0000","last_fullsized":"2024-01-16T14:57:08.859350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:24.013168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857725+0000","last_change":"2024-01-16T14:57:06.560743+0000","last_active":"2024-01-16T14:57:08.857725+0000","last_peered":"2024-01-16T14:57:08.857725+0000","last_clean":"2024-01-16T14:57:08.857725+0000","last_became_active":"2024-01-16T14:57:06.560557+0000","last_became_peered":"2024-01-16T14:57:06.560557+0000","last_unstale":"2024-01-16T14:57:08.857725+0000","last_undegraded":"2024-01-16T14:57:08.857725+0000","last_fullsized":"2024-01-16T14:57:08.857725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411687+0000","last_change":"2024-01-16T14:57:07.378876+0000","last_active":"2024-01-16T14:57:08.411687+0000","last_peered":"2024-01-16T14:57:08.411687+0000","last_clean":"2024-01-16T14:57:08.411687+0000","last_became_active":"2024-01-16T14:57:07.378103+0000","last_became_peered":"2024-01-16T14:57:07.378103+0000","last_unstale":"2024-01-16T14:57:08.411687+0000","last_undegraded":"2024-01-16T14:57:08.411687+0000","last_fullsized":"2024-01-16T14:57:08.411687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359610+0000","last_change":"2024-01-16T14:57:06.548228+0000","last_active":"2024-01-16T14:57:08.359610+0000","last_peered":"2024-01-16T14:57:08.359610+0000","last_clean":"2024-01-16T14:57:08.359610+0000","last_became_active":"2024-01-16T14:57:06.547577+0000","last_became_peered":"2024-01-16T14:57:06.547577+0000","last_unstale":"2024-01-16T14:57:08.359610+0000","last_undegraded":"2024-01-16T14:57:08.359610+0000","last_fullsized":"2024-01-16T14:57:08.359610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357044+0000","last_change":"2024-01-16T14:57:07.377694+0000","last_active":"2024-01-16T14:57:08.357044+0000","last_peered":"2024-01-16T14:57:08.357044+0000","last_clean":"2024-01-16T14:57:08.357044+0000","last_became_active":"2024-01-16T14:57:07.377016+0000","last_became_peered":"2024-01-16T14:57:07.377016+0000","last_unstale":"2024-01-16T14:57:08.357044+0000","last_undegraded":"2024-01-16T14:57:08.357044+0000","last_fullsized":"2024-01-16T14:57:08.357044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:08:30.500155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856442+0000","last_change":"2024-01-16T14:57:06.545409+0000","last_active":"2024-01-16T14:57:08.856442+0000","last_peered":"2024-01-16T14:57:08.856442+0000","last_clean":"2024-01-16T14:57:08.856442+0000","last_became_active":"2024-01-16T14:57:06.545133+0000","last_became_peered":"2024-01-16T14:57:06.545133+0000","last_unstale":"2024-01-16T14:57:08.856442+0000","last_undegraded":"2024-01-16T14:57:08.856442+0000","last_fullsized":"2024-01-16T14:57:08.856442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:18.053787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858230+0000","last_change":"2024-01-16T14:57:07.373879+0000","last_active":"2024-01-16T14:57:08.858230+0000","last_peered":"2024-01-16T14:57:08.858230+0000","last_clean":"2024-01-16T14:57:08.858230+0000","last_became_active":"2024-01-16T14:57:07.373537+0000","last_became_peered":"2024-01-16T14:57:07.373537+0000","last_unstale":"2024-01-16T14:57:08.858230+0000","last_undegraded":"2024-01-16T14:57:08.858230+0000","last_fullsized":"2024-01-16T14:57:08.858230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856963+0000","last_change":"2024-01-16T14:57:06.548121+0000","last_active":"2024-01-16T14:57:08.856963+0000","last_peered":"2024-01-16T14:57:08.856963+0000","last_clean":"2024-01-16T14:57:08.856963+0000","last_became_active":"2024-01-16T14:57:06.547868+0000","last_became_peered":"2024-01-16T14:57:06.547868+0000","last_unstale":"2024-01-16T14:57:08.856963+0000","last_undegraded":"2024-01-16T14:57:08.856963+0000","last_fullsized":"2024-01-16T14:57:08.856963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858908+0000","last_change":"2024-01-16T14:57:07.378203+0000","last_active":"2024-01-16T14:57:08.858908+0000","last_peered":"2024-01-16T14:57:08.858908+0000","last_clean":"2024-01-16T14:57:08.858908+0000","last_became_active":"2024-01-16T14:57:07.377843+0000","last_became_peered":"2024-01-16T14:57:07.377843+0000","last_unstale":"2024-01-16T14:57:08.858908+0000","last_undegraded":"2024-01-16T14:57:08.858908+0000","last_fullsized":"2024-01-16T14:57:08.858908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360527+0000","last_change":"2024-01-16T14:57:06.559586+0000","last_active":"2024-01-16T14:57:08.360527+0000","last_peered":"2024-01-16T14:57:08.360527+0000","last_clean":"2024-01-16T14:57:08.360527+0000","last_became_active":"2024-01-16T14:57:06.558281+0000","last_became_peered":"2024-01-16T14:57:06.558281+0000","last_unstale":"2024-01-16T14:57:08.360527+0000","last_undegraded":"2024-01-16T14:57:08.360527+0000","last_fullsized":"2024-01-16T14:57:08.360527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856775+0000","last_change":"2024-01-16T14:57:07.375900+0000","last_active":"2024-01-16T14:57:08.856775+0000","last_peered":"2024-01-16T14:57:08.856775+0000","last_clean":"2024-01-16T14:57:08.856775+0000","last_became_active":"2024-01-16T14:57:07.375625+0000","last_became_peered":"2024-01-16T14:57:07.375625+0000","last_unstale":"2024-01-16T14:57:08.856775+0000","last_undegraded":"2024-01-16T14:57:08.856775+0000","last_fullsized":"2024-01-16T14:57:08.856775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358102+0000","last_change":"2024-01-16T14:57:07.391481+0000","last_active":"2024-01-16T14:57:08.358102+0000","last_peered":"2024-01-16T14:57:08.358102+0000","last_clean":"2024-01-16T14:57:08.358102+0000","last_became_active":"2024-01-16T14:57:07.391111+0000","last_became_peered":"2024-01-16T14:57:07.391111+0000","last_unstale":"2024-01-16T14:57:08.358102+0000","last_undegraded":"2024-01-16T14:57:08.358102+0000","last_fullsized":"2024-01-16T14:57:08.358102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360198+0000","last_change":"2024-01-16T14:57:06.553585+0000","last_active":"2024-01-16T14:57:08.360198+0000","last_peered":"2024-01-16T14:57:08.360198+0000","last_clean":"2024-01-16T14:57:08.360198+0000","last_became_active":"2024-01-16T14:57:06.553259+0000","last_became_peered":"2024-01-16T14:57:06.553259+0000","last_unstale":"2024-01-16T14:57:08.360198+0000","last_undegraded":"2024-01-16T14:57:08.360198+0000","last_fullsized":"2024-01-16T14:57:08.360198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357259+0000","last_change":"2024-01-16T14:57:07.394393+0000","last_active":"2024-01-16T14:57:08.357259+0000","last_peered":"2024-01-16T14:57:08.357259+0000","last_clean":"2024-01-16T14:57:08.357259+0000","last_became_active":"2024-01-16T14:57:07.394135+0000","last_became_peered":"2024-01-16T14:57:07.394135+0000","last_unstale":"2024-01-16T14:57:08.357259+0000","last_undegraded":"2024-01-16T14:57:08.357259+0000","last_fullsized":"2024-01-16T14:57:08.357259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360937+0000","last_change":"2024-01-16T14:57:06.554017+0000","last_active":"2024-01-16T14:57:08.360937+0000","last_peered":"2024-01-16T14:57:08.360937+0000","last_clean":"2024-01-16T14:57:08.360937+0000","last_became_active":"2024-01-16T14:57:06.553376+0000","last_became_peered":"2024-01-16T14:57:06.553376+0000","last_unstale":"2024-01-16T14:57:08.360937+0000","last_undegraded":"2024-01-16T14:57:08.360937+0000","last_fullsized":"2024-01-16T14:57:08.360937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:35.199600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411544+0000","last_change":"2024-01-16T14:57:07.395607+0000","last_active":"2024-01-16T14:57:08.411544+0000","last_peered":"2024-01-16T14:57:08.411544+0000","last_clean":"2024-01-16T14:57:08.411544+0000","last_became_active":"2024-01-16T14:57:07.395288+0000","last_became_peered":"2024-01-16T14:57:07.395288+0000","last_unstale":"2024-01-16T14:57:08.411544+0000","last_undegraded":"2024-01-16T14:57:08.411544+0000","last_fullsized":"2024-01-16T14:57:08.411544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859322+0000","last_change":"2024-01-16T14:57:06.549558+0000","last_active":"2024-01-16T14:57:08.859322+0000","last_peered":"2024-01-16T14:57:08.859322+0000","last_clean":"2024-01-16T14:57:08.859322+0000","last_became_active":"2024-01-16T14:57:06.545986+0000","last_became_peered":"2024-01-16T14:57:06.545986+0000","last_unstale":"2024-01-16T14:57:08.859322+0000","last_undegraded":"2024-01-16T14:57:08.859322+0000","last_fullsized":"2024-01-16T14:57:08.859322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859106+0000","last_change":"2024-01-16T14:57:07.384928+0000","last_active":"2024-01-16T14:57:08.859106+0000","last_peered":"2024-01-16T14:57:08.859106+0000","last_clean":"2024-01-16T14:57:08.859106+0000","last_became_active":"2024-01-16T14:57:07.384613+0000","last_became_peered":"2024-01-16T14:57:07.384613+0000","last_unstale":"2024-01-16T14:57:08.859106+0000","last_undegraded":"2024-01-16T14:57:08.859106+0000","last_fullsized":"2024-01-16T14:57:08.859106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:19.171826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411722+0000","last_change":"2024-01-16T14:57:06.552465+0000","last_active":"2024-01-16T14:57:08.411722+0000","last_peered":"2024-01-16T14:57:08.411722+0000","last_clean":"2024-01-16T14:57:08.411722+0000","last_became_active":"2024-01-16T14:57:06.552103+0000","last_became_peered":"2024-01-16T14:57:06.552103+0000","last_unstale":"2024-01-16T14:57:08.411722+0000","last_undegraded":"2024-01-16T14:57:08.411722+0000","last_fullsized":"2024-01-16T14:57:08.411722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:35.728588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.335}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1219999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3740000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:57:17.335 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:57:17.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.597+0000 7eff3ee00640 1 Processor -- start 2024-01-16T14:57:17.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.598+0000 7eff3ee00640 1 -- start start 2024-01-16T14:57:17.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.599+0000 7eff3ee00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7eff38143c40 0x7eff38071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.599+0000 7eff3ee00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff38070e50 con 0x7eff380ccc10 2024-01-16T14:57:17.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.599+0000 7eff3ee00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff38144020 con 0x7eff380717d0 2024-01-16T14:57:17.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.599+0000 7eff3ee00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff380723c0 con 0x7eff38143c40 2024-01-16T14:57:17.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.599+0000 7eff3da00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7eff38143c40 0x7eff38071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.600+0000 7eff3da00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7eff38143c40 0x7eff38071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48672/0 (socket says 172.21.15.141:48672) 2024-01-16T14:57:17.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.600+0000 7eff3da00640 1 -- 172.21.15.141:0/1289515041 learned_addr learned my addr 172.21.15.141:0/1289515041 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:17.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.600+0000 7eff3e400640 1 --1- 172.21.15.141:0/1289515041 >> v1:172.21.15.141:6789/0 conn(0x7eff380ccc10 0x7eff3806d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40086/0 (socket says 172.21.15.141:40086) 2024-01-16T14:57:17.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.600+0000 7eff3da00640 1 -- 172.21.15.141:0/1289515041 >> v1:172.21.15.188:6789/0 conn(0x7eff380717d0 legacy=0x7eff380cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:17.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.600+0000 7eff3da00640 1 -- 172.21.15.141:0/1289515041 >> v1:172.21.15.141:6789/0 conn(0x7eff380ccc10 legacy=0x7eff3806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.600+0000 7eff3da00640 1 -- 172.21.15.141:0/1289515041 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff38071e00 con 0x7eff38143c40 2024-01-16T14:57:17.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.600+0000 7eff37e00640 1 -- 172.21.15.141:0/1289515041 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960460383 0 0) 0x7eff38070e50 con 0x7eff380ccc10 2024-01-16T14:57:17.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.601+0000 7eff3da00640 1 --2- 172.21.15.141:0/1289515041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38143c40 0x7eff38071290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7eff28009870 tx=0x7eff2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c12d3573f99496e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:17.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.601+0000 7eff37e00640 1 -- 172.21.15.141:0/1289515041 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28031870 con 0x7eff38143c40 2024-01-16T14:57:17.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.601+0000 7eff37e00640 1 -- 172.21.15.141:0/1289515041 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff280319d0 con 0x7eff38143c40 2024-01-16T14:57:17.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.601+0000 7eff37e00640 1 -- 172.21.15.141:0/1289515041 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28031d60 con 0x7eff38143c40 2024-01-16T14:57:17.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.602+0000 7eff3ee00640 1 -- 172.21.15.141:0/1289515041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38143c40 msgr2=0x7eff38071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.602+0000 7eff3ee00640 1 --2- 172.21.15.141:0/1289515041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38143c40 0x7eff38071290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7eff28009870 tx=0x7eff2802b910 comp rx=0 tx=0).stop 2024-01-16T14:57:17.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.602+0000 7eff3ee00640 1 -- 172.21.15.141:0/1289515041 shutdown_connections 2024-01-16T14:57:17.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.602+0000 7eff3ee00640 1 --2- 172.21.15.141:0/1289515041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38143c40 0x7eff38071290 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.602+0000 7eff3ee00640 1 -- 172.21.15.141:0/1289515041 >> 172.21.15.141:0/1289515041 conn(0x7eff3806a800 msgr2=0x7eff3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:17.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.602+0000 7eff3ee00640 1 -- 172.21.15.141:0/1289515041 shutdown_connections 2024-01-16T14:57:17.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.602+0000 7eff3ee00640 1 -- 172.21.15.141:0/1289515041 wait complete. 2024-01-16T14:57:17.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.604+0000 7eff3ee00640 1 Processor -- start 2024-01-16T14:57:17.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.604+0000 7eff3ee00640 1 -- start start 2024-01-16T14:57:17.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3ee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff380717d0 0x7eff380afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3ee00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff380ccc10 0x7eff381f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3ee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38143c40 0x7eff381f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3ee00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380afff0 con 0x7eff380ccc10 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3ee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3813c750 con 0x7eff380717d0 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3ee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff381f3670 con 0x7eff38143c40 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff380ccc10 0x7eff381f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff380ccc10 0x7eff381f3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48680/0 (socket says 172.21.15.141:48680) 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 -- 172.21.15.141:0/3165591185 learned_addr learned my addr 172.21.15.141:0/3165591185 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 -- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38143c40 msgr2=0x7eff381f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38143c40 0x7eff381f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 -- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff380717d0 msgr2=0x7eff380afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3da00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff380717d0 0x7eff380afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff380717d0 0x7eff380afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.605+0000 7eff3d000640 1 -- 172.21.15.141:0/3165591185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280094d0 con 0x7eff380ccc10 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.606+0000 7eff3d000640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff380ccc10 0x7eff381f3130 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff2000b420 tx=0x7eff2000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.606+0000 7eff35600640 1 -- 172.21.15.141:0/3165591185 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20011830 con 0x7eff380ccc10 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.606+0000 7eff35600640 1 -- 172.21.15.141:0/3165591185 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff20011990 con 0x7eff380ccc10 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.606+0000 7eff35600640 1 -- 172.21.15.141:0/3165591185 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20011d40 con 0x7eff380ccc10 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.606+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff380b2030 con 0x7eff380ccc10 2024-01-16T14:57:17.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.606+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff380b22d0 con 0x7eff380ccc10 2024-01-16T14:57:17.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.607+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff38067450 con 0x7eff380ccc10 2024-01-16T14:57:17.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.608+0000 7eff35600640 1 -- 172.21.15.141:0/3165591185 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff2001b460 con 0x7eff380ccc10 2024-01-16T14:57:17.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.609+0000 7eff35600640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2c0 0x7eff1006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:17.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.609+0000 7eff35600640 1 -- 172.21.15.141:0/3165591185 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff20089740 con 0x7eff380ccc10 2024-01-16T14:57:17.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.609+0000 7eff3da00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2c0 0x7eff1006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:17.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.610+0000 7eff3da00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2c0 0x7eff1006d780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff28036000 tx=0x7eff2803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:17.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.613+0000 7eff35600640 1 -- 172.21.15.141:0/3165591185 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff200545a0 con 0x7eff380ccc10 2024-01-16T14:57:17.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.785+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7eff3806cc40 con 0x7eff1006b2c0 2024-01-16T14:57:17.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.805+0000 7eff35600640 1 -- 172.21.15.141:0/3165591185 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264347 (secure 0 0 0) 0x7eff28031cc0 con 0x7eff1006b2c0 2024-01-16T14:57:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:17.792 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2c0 msgr2=0x7eff1006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2c0 0x7eff1006d780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff28036000 tx=0x7eff2803c040 comp rx=0 tx=0).stop 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff380ccc10 msgr2=0x7eff381f3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff380ccc10 0x7eff381f3130 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff2000b420 tx=0x7eff2000b8e0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 shutdown_connections 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2c0 0x7eff1006d780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38143c40 0x7eff381f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff380ccc10 0x7eff381f3130 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 --2- 172.21.15.141:0/3165591185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff380717d0 0x7eff380afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 >> 172.21.15.141:0/3165591185 conn(0x7eff3806a800 msgr2=0x7eff380b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 shutdown_connections 2024-01-16T14:57:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:17.811+0000 7eff3ee00640 1 -- 172.21.15.141:0/3165591185 wait complete. 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-16T14:57:16.438499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219508,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755212,"statfs":{"total":773094113280,"available":772869337088,"internally_reserved":0,"allocated":3055616,"data_stored":1815380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.090193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361647+0000","last_change":"2024-01-16T14:57:07.388330+0000","last_active":"2024-01-16T14:57:08.361647+0000","last_peered":"2024-01-16T14:57:08.361647+0000","last_clean":"2024-01-16T14:57:08.361647+0000","last_became_active":"2024-01-16T14:57:07.387937+0000","last_became_peered":"2024-01-16T14:57:07.387937+0000","last_unstale":"2024-01-16T14:57:08.361647+0000","last_undegraded":"2024-01-16T14:57:08.361647+0000","last_fullsized":"2024-01-16T14:57:08.361647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859055+0000","last_change":"2024-01-16T14:57:07.376852+0000","last_active":"2024-01-16T14:57:08.859055+0000","last_peered":"2024-01-16T14:57:08.859055+0000","last_clean":"2024-01-16T14:57:08.859055+0000","last_became_active":"2024-01-16T14:57:07.376562+0000","last_became_peered":"2024-01-16T14:57:07.376562+0000","last_unstale":"2024-01-16T14:57:08.859055+0000","last_undegraded":"2024-01-16T14:57:08.859055+0000","last_fullsized":"2024-01-16T14:57:08.859055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411451+0000","last_change":"2024-01-16T14:57:07.376399+0000","last_active":"2024-01-16T14:57:08.411451+0000","last_peered":"2024-01-16T14:57:08.411451+0000","last_clean":"2024-01-16T14:57:08.411451+0000","last_became_active":"2024-01-16T14:57:07.374502+0000","last_became_peered":"2024-01-16T14:57:07.374502+0000","last_unstale":"2024-01-16T14:57:08.411451+0000","last_undegraded":"2024-01-16T14:57:08.411451+0000","last_fullsized":"2024-01-16T14:57:08.411451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357034+0000","last_change":"2024-01-16T14:57:07.393925+0000","last_active":"2024-01-16T14:57:08.357034+0000","last_peered":"2024-01-16T14:57:08.357034+0000","last_clean":"2024-01-16T14:57:08.357034+0000","last_became_active":"2024-01-16T14:57:07.393608+0000","last_became_peered":"2024-01-16T14:57:07.393608+0000","last_unstale":"2024-01-16T14:57:08.357034+0000","last_undegraded":"2024-01-16T14:57:08.357034+0000","last_fullsized":"2024-01-16T14:57:08.357034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:12.497634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360231+0000","last_change":"2024-01-16T14:57:07.370383+0000","last_active":"2024-01-16T14:57:08.360231+0000","last_peered":"2024-01-16T14:57:08.360231+0000","last_clean":"2024-01-16T14:57:08.360231+0000","last_became_active":"2024-01-16T14:57:07.369925+0000","last_became_peered":"2024-01-16T14:57:07.369925+0000","last_unstale":"2024-01-16T14:57:08.360231+0000","last_undegraded":"2024-01-16T14:57:08.360231+0000","last_fullsized":"2024-01-16T14:57:08.360231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357036+0000","last_change":"2024-01-16T14:57:07.392767+0000","last_active":"2024-01-16T14:57:08.357036+0000","last_peered":"2024-01-16T14:57:08.357036+0000","last_clean":"2024-01-16T14:57:08.357036+0000","last_became_active":"2024-01-16T14:57:07.392457+0000","last_became_peered":"2024-01-16T14:57:07.392457+0000","last_unstale":"2024-01-16T14:57:08.357036+0000","last_undegraded":"2024-01-16T14:57:08.357036+0000","last_fullsized":"2024-01-16T14:57:08.357036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359822+0000","last_change":"2024-01-16T14:57:07.377276+0000","last_active":"2024-01-16T14:57:08.359822+0000","last_peered":"2024-01-16T14:57:08.359822+0000","last_clean":"2024-01-16T14:57:08.359822+0000","last_became_active":"2024-01-16T14:57:07.376937+0000","last_became_peered":"2024-01-16T14:57:07.376937+0000","last_unstale":"2024-01-16T14:57:08.359822+0000","last_undegraded":"2024-01-16T14:57:08.359822+0000","last_fullsized":"2024-01-16T14:57:08.359822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:15.513272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857828+0000","last_change":"2024-01-16T14:57:07.394001+0000","last_active":"2024-01-16T14:57:08.857828+0000","last_peered":"2024-01-16T14:57:08.857828+0000","last_clean":"2024-01-16T14:57:08.857828+0000","last_became_active":"2024-01-16T14:57:07.393670+0000","last_became_peered":"2024-01-16T14:57:07.393670+0000","last_unstale":"2024-01-16T14:57:08.857828+0000","last_undegraded":"2024-01-16T14:57:08.857828+0000","last_fullsized":"2024-01-16T14:57:08.857828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361786+0000","last_change":"2024-01-16T14:57:07.394940+0000","last_active":"2024-01-16T14:57:08.361786+0000","last_peered":"2024-01-16T14:57:08.361786+0000","last_clean":"2024-01-16T14:57:08.361786+0000","last_became_active":"2024-01-16T14:57:07.394335+0000","last_became_peered":"2024-01-16T14:57:07.394335+0000","last_unstale":"2024-01-16T14:57:08.361786+0000","last_undegraded":"2024-01-16T14:57:08.361786+0000","last_fullsized":"2024-01-16T14:57:08.361786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:23.443732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857711+0000","last_change":"2024-01-16T14:57:07.395598+0000","last_active":"2024-01-16T14:57:08.857711+0000","last_peered":"2024-01-16T14:57:08.857711+0000","last_clean":"2024-01-16T14:57:08.857711+0000","last_became_active":"2024-01-16T14:57:07.395367+0000","last_became_peered":"2024-01-16T14:57:07.395367+0000","last_unstale":"2024-01-16T14:57:08.857711+0000","last_undegraded":"2024-01-16T14:57:08.857711+0000","last_fullsized":"2024-01-16T14:57:08.857711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857014+0000","last_change":"2024-01-16T14:57:07.394374+0000","last_active":"2024-01-16T14:57:08.857014+0000","last_peered":"2024-01-16T14:57:08.857014+0000","last_clean":"2024-01-16T14:57:08.857014+0000","last_became_active":"2024-01-16T14:57:07.394125+0000","last_became_peered":"2024-01-16T14:57:07.394125+0000","last_unstale":"2024-01-16T14:57:08.857014+0000","last_undegraded":"2024-01-16T14:57:08.857014+0000","last_fullsized":"2024-01-16T14:57:08.857014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357258+0000","last_change":"2024-01-16T14:57:07.382452+0000","last_active":"2024-01-16T14:57:08.357258+0000","last_peered":"2024-01-16T14:57:08.357258+0000","last_clean":"2024-01-16T14:57:08.357258+0000","last_became_active":"2024-01-16T14:57:07.381985+0000","last_became_peered":"2024-01-16T14:57:07.381985+0000","last_unstale":"2024-01-16T14:57:08.357258+0000","last_undegraded":"2024-01-16T14:57:08.357258+0000","last_fullsized":"2024-01-16T14:57:08.357258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:31.589732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858575+0000","last_change":"2024-01-16T14:57:07.385013+0000","last_active":"2024-01-16T14:57:08.858575+0000","last_peered":"2024-01-16T14:57:08.858575+0000","last_clean":"2024-01-16T14:57:08.858575+0000","last_became_active":"2024-01-16T14:57:07.384705+0000","last_became_peered":"2024-01-16T14:57:07.384705+0000","last_unstale":"2024-01-16T14:57:08.858575+0000","last_undegraded":"2024-01-16T14:57:08.858575+0000","last_fullsized":"2024-01-16T14:57:08.858575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411099+0000","last_change":"2024-01-16T14:57:07.382942+0000","last_active":"2024-01-16T14:57:08.411099+0000","last_peered":"2024-01-16T14:57:08.411099+0000","last_clean":"2024-01-16T14:57:08.411099+0000","last_became_active":"2024-01-16T14:57:07.382609+0000","last_became_peered":"2024-01-16T14:57:07.382609+0000","last_unstale":"2024-01-16T14:57:08.411099+0000","last_undegraded":"2024-01-16T14:57:08.411099+0000","last_fullsized":"2024-01-16T14:57:08.411099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:07.098165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856153+0000","last_change":"2024-01-16T14:57:07.369499+0000","last_active":"2024-01-16T14:57:08.856153+0000","last_peered":"2024-01-16T14:57:08.856153+0000","last_clean":"2024-01-16T14:57:08.856153+0000","last_became_active":"2024-01-16T14:57:07.369089+0000","last_became_peered":"2024-01-16T14:57:07.369089+0000","last_unstale":"2024-01-16T14:57:08.856153+0000","last_undegraded":"2024-01-16T14:57:08.856153+0000","last_fullsized":"2024-01-16T14:57:08.856153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:45:27.672659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.355734+0000","last_change":"2024-01-16T14:57:07.392502+0000","last_active":"2024-01-16T14:57:08.355734+0000","last_peered":"2024-01-16T14:57:08.355734+0000","last_clean":"2024-01-16T14:57:08.355734+0000","last_became_active":"2024-01-16T14:57:07.392065+0000","last_became_peered":"2024-01-16T14:57:07.392065+0000","last_unstale":"2024-01-16T14:57:08.355734+0000","last_undegraded":"2024-01-16T14:57:08.355734+0000","last_fullsized":"2024-01-16T14:57:08.355734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358909+0000","last_change":"2024-01-16T14:57:07.394712+0000","last_active":"2024-01-16T14:57:08.358909+0000","last_peered":"2024-01-16T14:57:08.358909+0000","last_clean":"2024-01-16T14:57:08.358909+0000","last_became_active":"2024-01-16T14:57:07.393954+0000","last_became_peered":"2024-01-16T14:57:07.393954+0000","last_unstale":"2024-01-16T14:57:08.358909+0000","last_undegraded":"2024-01-16T14:57:08.358909+0000","last_fullsized":"2024-01-16T14:57:08.358909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859744+0000","last_change":"2024-01-16T14:57:07.384626+0000","last_active":"2024-01-16T14:57:08.859744+0000","last_peered":"2024-01-16T14:57:08.859744+0000","last_clean":"2024-01-16T14:57:08.859744+0000","last_became_active":"2024-01-16T14:57:07.383860+0000","last_became_peered":"2024-01-16T14:57:07.383860+0000","last_unstale":"2024-01-16T14:57:08.859744+0000","last_undegraded":"2024-01-16T14:57:08.859744+0000","last_fullsized":"2024-01-16T14:57:08.859744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361107+0000","last_change":"2024-01-16T14:57:07.375013+0000","last_active":"2024-01-16T14:57:08.361107+0000","last_peered":"2024-01-16T14:57:08.361107+0000","last_clean":"2024-01-16T14:57:08.361107+0000","last_became_active":"2024-01-16T14:57:07.373630+0000","last_became_peered":"2024-01-16T14:57:07.373630+0000","last_unstale":"2024-01-16T14:57:08.361107+0000","last_undegraded":"2024-01-16T14:57:08.361107+0000","last_fullsized":"2024-01-16T14:57:08.361107+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:26.742686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857112+0000","last_change":"2024-01-16T14:57:07.375790+0000","last_active":"2024-01-16T14:57:08.857112+0000","last_peered":"2024-01-16T14:57:08.857112+0000","last_clean":"2024-01-16T14:57:08.857112+0000","last_became_active":"2024-01-16T14:57:07.375533+0000","last_became_peered":"2024-01-16T14:57:07.375533+0000","last_unstale":"2024-01-16T14:57:08.857112+0000","last_undegraded":"2024-01-16T14:57:08.857112+0000","last_fullsized":"2024-01-16T14:57:08.857112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411312+0000","last_change":"2024-01-16T14:57:07.378765+0000","last_active":"2024-01-16T14:57:08.411312+0000","last_peered":"2024-01-16T14:57:08.411312+0000","last_clean":"2024-01-16T14:57:08.411312+0000","last_became_active":"2024-01-16T14:57:07.377672+0000","last_became_peered":"2024-01-16T14:57:07.377672+0000","last_unstale":"2024-01-16T14:57:08.411312+0000","last_undegraded":"2024-01-16T14:57:08.411312+0000","last_fullsized":"2024-01-16T14:57:08.411312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357439+0000","last_change":"2024-01-16T14:57:07.392203+0000","last_active":"2024-01-16T14:57:08.357439+0000","last_peered":"2024-01-16T14:57:08.357439+0000","last_clean":"2024-01-16T14:57:08.357439+0000","last_became_active":"2024-01-16T14:57:07.391913+0000","last_became_peered":"2024-01-16T14:57:07.391913+0000","last_unstale":"2024-01-16T14:57:08.357439+0000","last_undegraded":"2024-01-16T14:57:08.357439+0000","last_fullsized":"2024-01-16T14:57:08.357439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858112+0000","last_change":"2024-01-16T14:57:07.383055+0000","last_active":"2024-01-16T14:57:08.858112+0000","last_peered":"2024-01-16T14:57:08.858112+0000","last_clean":"2024-01-16T14:57:08.858112+0000","last_became_active":"2024-01-16T14:57:07.382181+0000","last_became_peered":"2024-01-16T14:57:07.382181+0000","last_unstale":"2024-01-16T14:57:08.858112+0000","last_undegraded":"2024-01-16T14:57:08.858112+0000","last_fullsized":"2024-01-16T14:57:08.858112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.356639+0000","last_change":"2024-01-16T14:57:07.391594+0000","last_active":"2024-01-16T14:57:08.356639+0000","last_peered":"2024-01-16T14:57:08.356639+0000","last_clean":"2024-01-16T14:57:08.356639+0000","last_became_active":"2024-01-16T14:57:07.391116+0000","last_became_peered":"2024-01-16T14:57:07.391116+0000","last_unstale":"2024-01-16T14:57:08.356639+0000","last_undegraded":"2024-01-16T14:57:08.356639+0000","last_fullsized":"2024-01-16T14:57:08.356639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:17.138248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859255+0000","last_change":"2024-01-16T14:57:07.383060+0000","last_active":"2024-01-16T14:57:08.859255+0000","last_peered":"2024-01-16T14:57:08.859255+0000","last_clean":"2024-01-16T14:57:08.859255+0000","last_became_active":"2024-01-16T14:57:07.382192+0000","last_became_peered":"2024-01-16T14:57:07.382192+0000","last_unstale":"2024-01-16T14:57:08.859255+0000","last_undegraded":"2024-01-16T14:57:08.859255+0000","last_fullsized":"2024-01-16T14:57:08.859255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:34.791403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857952+0000","last_change":"2024-01-16T14:57:07.384727+0000","last_active":"2024-01-16T14:57:08.857952+0000","last_peered":"2024-01-16T14:57:08.857952+0000","last_clean":"2024-01-16T14:57:08.857952+0000","last_became_active":"2024-01-16T14:57:07.383334+0000","last_became_peered":"2024-01-16T14:57:07.383334+0000","last_unstale":"2024-01-16T14:57:08.857952+0000","last_undegraded":"2024-01-16T14:57:08.857952+0000","last_fullsized":"2024-01-16T14:57:08.857952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856861+0000","last_change":"2024-01-16T14:57:07.376359+0000","last_active":"2024-01-16T14:57:08.856861+0000","last_peered":"2024-01-16T14:57:08.856861+0000","last_clean":"2024-01-16T14:57:08.856861+0000","last_became_active":"2024-01-16T14:57:07.376074+0000","last_became_peered":"2024-01-16T14:57:07.376074+0000","last_unstale":"2024-01-16T14:57:08.856861+0000","last_undegraded":"2024-01-16T14:57:08.856861+0000","last_fullsized":"2024-01-16T14:57:08.856861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:00.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858616+0000","last_change":"2024-01-16T14:57:07.387458+0000","last_active":"2024-01-16T14:57:08.858616+0000","last_peered":"2024-01-16T14:57:08.858616+0000","last_clean":"2024-01-16T14:57:08.858616+0000","last_became_active":"2024-01-16T14:57:07.387221+0000","last_became_peered":"2024-01-16T14:57:07.387221+0000","last_unstale":"2024-01-16T14:57:08.858616+0000","last_undegraded":"2024-01-16T14:57:08.858616+0000","last_fullsized":"2024-01-16T14:57:08.858616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411330+0000","last_change":"2024-01-16T14:57:07.377884+0000","last_active":"2024-01-16T14:57:08.411330+0000","last_peered":"2024-01-16T14:57:08.411330+0000","last_clean":"2024-01-16T14:57:08.411330+0000","last_became_active":"2024-01-16T14:57:07.377585+0000","last_became_peered":"2024-01-16T14:57:07.377585+0000","last_unstale":"2024-01-16T14:57:08.411330+0000","last_undegraded":"2024-01-16T14:57:08.411330+0000","last_fullsized":"2024-01-16T14:57:08.411330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358051+0000","last_change":"2024-01-16T14:57:07.389650+0000","last_active":"2024-01-16T14:57:08.358051+0000","last_peered":"2024-01-16T14:57:08.358051+0000","last_clean":"2024-01-16T14:57:08.358051+0000","last_became_active":"2024-01-16T14:57:07.389388+0000","last_became_peered":"2024-01-16T14:57:07.389388+0000","last_unstale":"2024-01-16T14:57:08.358051+0000","last_undegraded":"2024-01-16T14:57:08.358051+0000","last_fullsized":"2024-01-16T14:57:08.358051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858338+0000","last_change":"2024-01-16T14:57:07.393121+0000","last_active":"2024-01-16T14:57:08.858338+0000","last_peered":"2024-01-16T14:57:08.858338+0000","last_clean":"2024-01-16T14:57:08.858338+0000","last_became_active":"2024-01-16T14:57:07.392839+0000","last_became_peered":"2024-01-16T14:57:07.392839+0000","last_unstale":"2024-01-16T14:57:08.858338+0000","last_undegraded":"2024-01-16T14:57:08.858338+0000","last_fullsized":"2024-01-16T14:57:08.858338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856756+0000","last_change":"2024-01-16T14:57:07.386156+0000","last_active":"2024-01-16T14:57:08.856756+0000","last_peered":"2024-01-16T14:57:08.856756+0000","last_clean":"2024-01-16T14:57:08.856756+0000","last_became_active":"2024-01-16T14:57:07.385684+0000","last_became_peered":"2024-01-16T14:57:07.385684+0000","last_unstale":"2024-01-16T14:57:08.856756+0000","last_undegraded":"2024-01-16T14:57:08.856756+0000","last_fullsized":"2024-01-16T14:57:08.856756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357739+0000","last_change":"2024-01-16T14:57:07.372196+0000","last_active":"2024-01-16T14:57:08.357739+0000","last_peered":"2024-01-16T14:57:08.357739+0000","last_clean":"2024-01-16T14:57:08.357739+0000","last_became_active":"2024-01-16T14:57:07.369065+0000","last_became_peered":"2024-01-16T14:57:07.369065+0000","last_unstale":"2024-01-16T14:57:08.357739+0000","last_undegraded":"2024-01-16T14:57:08.357739+0000","last_fullsized":"2024-01-16T14:57:08.357739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:23.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858827+0000","last_change":"2024-01-16T14:57:06.556368+0000","last_active":"2024-01-16T14:57:08.858827+0000","last_peered":"2024-01-16T14:57:08.858827+0000","last_clean":"2024-01-16T14:57:08.858827+0000","last_became_active":"2024-01-16T14:57:06.555953+0000","last_became_peered":"2024-01-16T14:57:06.555953+0000","last_unstale":"2024-01-16T14:57:08.858827+0000","last_undegraded":"2024-01-16T14:57:08.858827+0000","last_fullsized":"2024-01-16T14:57:08.858827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:45.773231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411448+0000","last_change":"2024-01-16T14:57:07.383151+0000","last_active":"2024-01-16T14:57:08.411448+0000","last_peered":"2024-01-16T14:57:08.411448+0000","last_clean":"2024-01-16T14:57:08.411448+0000","last_became_active":"2024-01-16T14:57:07.381555+0000","last_became_peered":"2024-01-16T14:57:07.381555+0000","last_unstale":"2024-01-16T14:57:08.411448+0000","last_undegraded":"2024-01-16T14:57:08.411448+0000","last_fullsized":"2024-01-16T14:57:08.411448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856949+0000","last_change":"2024-01-16T14:57:06.555306+0000","last_active":"2024-01-16T14:57:08.856949+0000","last_peered":"2024-01-16T14:57:08.856949+0000","last_clean":"2024-01-16T14:57:08.856949+0000","last_became_active":"2024-01-16T14:57:06.555063+0000","last_became_peered":"2024-01-16T14:57:06.555063+0000","last_unstale":"2024-01-16T14:57:08.856949+0000","last_undegraded":"2024-01-16T14:57:08.856949+0000","last_fullsized":"2024-01-16T14:57:08.856949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:07.271545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857979+0000","last_change":"2024-01-16T14:57:06.555279+0000","last_active":"2024-01-16T14:57:08.857979+0000","last_peered":"2024-01-16T14:57:08.857979+0000","last_clean":"2024-01-16T14:57:08.857979+0000","last_became_active":"2024-01-16T14:57:06.554559+0000","last_became_peered":"2024-01-16T14:57:06.554559+0000","last_unstale":"2024-01-16T14:57:08.857979+0000","last_undegraded":"2024-01-16T14:57:08.857979+0000","last_fullsized":"2024-01-16T14:57:08.857979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857830+0000","last_change":"2024-01-16T14:57:07.382661+0000","last_active":"2024-01-16T14:57:08.857830+0000","last_peered":"2024-01-16T14:57:08.857830+0000","last_clean":"2024-01-16T14:57:08.857830+0000","last_became_active":"2024-01-16T14:57:07.382055+0000","last_became_peered":"2024-01-16T14:57:07.382055+0000","last_unstale":"2024-01-16T14:57:08.857830+0000","last_undegraded":"2024-01-16T14:57:08.857830+0000","last_fullsized":"2024-01-16T14:57:08.857830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357563+0000","last_change":"2024-01-16T14:57:06.558562+0000","last_active":"2024-01-16T14:57:08.357563+0000","last_peered":"2024-01-16T14:57:08.357563+0000","last_clean":"2024-01-16T14:57:08.357563+0000","last_became_active":"2024-01-16T14:57:06.558166+0000","last_became_peered":"2024-01-16T14:57:06.558166+0000","last_unstale":"2024-01-16T14:57:08.357563+0000","last_undegraded":"2024-01-16T14:57:08.357563+0000","last_fullsized":"2024-01-16T14:57:08.357563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:31.817106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855928+0000","last_change":"2024-01-16T14:57:07.377917+0000","last_active":"2024-01-16T14:57:08.855928+0000","last_peered":"2024-01-16T14:57:08.855928+0000","last_clean":"2024-01-16T14:57:08.855928+0000","last_became_active":"2024-01-16T14:57:07.377601+0000","last_became_peered":"2024-01-16T14:57:07.377601+0000","last_unstale":"2024-01-16T14:57:08.855928+0000","last_undegraded":"2024-01-16T14:57:08.855928+0000","last_fullsized":"2024-01-16T14:57:08.855928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:32.968418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857555+0000","last_change":"2024-01-16T14:57:06.563826+0000","last_active":"2024-01-16T14:57:08.857555+0000","last_peered":"2024-01-16T14:57:08.857555+0000","last_clean":"2024-01-16T14:57:08.857555+0000","last_became_active":"2024-01-16T14:57:06.563529+0000","last_became_peered":"2024-01-16T14:57:06.563529+0000","last_unstale":"2024-01-16T14:57:08.857555+0000","last_undegraded":"2024-01-16T14:57:08.857555+0000","last_fullsized":"2024-01-16T14:57:08.857555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358434+0000","last_change":"2024-01-16T14:57:07.391665+0000","last_active":"2024-01-16T14:57:08.358434+0000","last_peered":"2024-01-16T14:57:08.358434+0000","last_clean":"2024-01-16T14:57:08.358434+0000","last_became_active":"2024-01-16T14:57:07.391071+0000","last_became_peered":"2024-01-16T14:57:07.391071+0000","last_unstale":"2024-01-16T14:57:08.358434+0000","last_undegraded":"2024-01-16T14:57:08.358434+0000","last_fullsized":"2024-01-16T14:57:08.358434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358657+0000","last_change":"2024-01-16T14:57:06.560447+0000","last_active":"2024-01-16T14:57:08.358657+0000","last_peered":"2024-01-16T14:57:08.358657+0000","last_clean":"2024-01-16T14:57:08.358657+0000","last_became_active":"2024-01-16T14:57:06.560116+0000","last_became_peered":"2024-01-16T14:57:06.560116+0000","last_unstale":"2024-01-16T14:57:08.358657+0000","last_undegraded":"2024-01-16T14:57:08.358657+0000","last_fullsized":"2024-01-16T14:57:08.358657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:33.296388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411729+0000","last_change":"2024-01-16T14:57:07.396020+0000","last_active":"2024-01-16T14:57:08.411729+0000","last_peered":"2024-01-16T14:57:08.411729+0000","last_clean":"2024-01-16T14:57:08.411729+0000","last_became_active":"2024-01-16T14:57:07.395685+0000","last_became_peered":"2024-01-16T14:57:07.395685+0000","last_unstale":"2024-01-16T14:57:08.411729+0000","last_undegraded":"2024-01-16T14:57:08.411729+0000","last_fullsized":"2024-01-16T14:57:08.411729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857123+0000","last_change":"2024-01-16T14:57:06.553623+0000","last_active":"2024-01-16T14:57:08.857123+0000","last_peered":"2024-01-16T14:57:08.857123+0000","last_clean":"2024-01-16T14:57:08.857123+0000","last_became_active":"2024-01-16T14:57:06.553173+0000","last_became_peered":"2024-01-16T14:57:06.553173+0000","last_unstale":"2024-01-16T14:57:08.857123+0000","last_undegraded":"2024-01-16T14:57:08.857123+0000","last_fullsized":"2024-01-16T14:57:08.857123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357536+0000","last_change":"2024-01-16T14:57:07.382256+0000","last_active":"2024-01-16T14:57:08.357536+0000","last_peered":"2024-01-16T14:57:08.357536+0000","last_clean":"2024-01-16T14:57:08.357536+0000","last_became_active":"2024-01-16T14:57:07.381640+0000","last_became_peered":"2024-01-16T14:57:07.381640+0000","last_unstale":"2024-01-16T14:57:08.357536+0000","last_undegraded":"2024-01-16T14:57:08.357536+0000","last_fullsized":"2024-01-16T14:57:08.357536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:14.206644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857022+0000","last_change":"2024-01-16T14:57:06.556835+0000","last_active":"2024-01-16T14:57:08.857022+0000","last_peered":"2024-01-16T14:57:08.857022+0000","last_clean":"2024-01-16T14:57:08.857022+0000","last_became_active":"2024-01-16T14:57:06.556483+0000","last_became_peered":"2024-01-16T14:57:06.556483+0000","last_unstale":"2024-01-16T14:57:08.857022+0000","last_undegraded":"2024-01-16T14:57:08.857022+0000","last_fullsized":"2024-01-16T14:57:08.857022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:01.863989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360686+0000","last_change":"2024-01-16T14:57:07.380047+0000","last_active":"2024-01-16T14:57:08.360686+0000","last_peered":"2024-01-16T14:57:08.360686+0000","last_clean":"2024-01-16T14:57:08.360686+0000","last_became_active":"2024-01-16T14:57:07.379698+0000","last_became_peered":"2024-01-16T14:57:07.379698+0000","last_unstale":"2024-01-16T14:57:08.360686+0000","last_undegraded":"2024-01-16T14:57:08.360686+0000","last_fullsized":"2024-01-16T14:57:08.360686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:59:28.768798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411453+0000","last_change":"2024-01-16T14:57:06.555448+0000","last_active":"2024-01-16T14:57:08.411453+0000","last_peered":"2024-01-16T14:57:08.411453+0000","last_clean":"2024-01-16T14:57:08.411453+0000","last_became_active":"2024-01-16T14:57:06.555097+0000","last_became_peered":"2024-01-16T14:57:06.555097+0000","last_unstale":"2024-01-16T14:57:08.411453+0000","last_undegraded":"2024-01-16T14:57:08.411453+0000","last_fullsized":"2024-01-16T14:57:08.411453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:23.858452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411172+0000","last_change":"2024-01-16T14:57:07.380436+0000","last_active":"2024-01-16T14:57:08.411172+0000","last_peered":"2024-01-16T14:57:08.411172+0000","last_clean":"2024-01-16T14:57:08.411172+0000","last_became_active":"2024-01-16T14:57:07.380148+0000","last_became_peered":"2024-01-16T14:57:07.380148+0000","last_unstale":"2024-01-16T14:57:08.411172+0000","last_undegraded":"2024-01-16T14:57:08.411172+0000","last_fullsized":"2024-01-16T14:57:08.411172+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:11.309943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358031+0000","last_change":"2024-01-16T14:57:06.549803+0000","last_active":"2024-01-16T14:57:08.358031+0000","last_peered":"2024-01-16T14:57:08.358031+0000","last_clean":"2024-01-16T14:57:08.358031+0000","last_became_active":"2024-01-16T14:57:06.549362+0000","last_became_peered":"2024-01-16T14:57:06.549362+0000","last_unstale":"2024-01-16T14:57:08.358031+0000","last_undegraded":"2024-01-16T14:57:08.358031+0000","last_fullsized":"2024-01-16T14:57:08.358031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360491+0000","last_change":"2024-01-16T14:57:07.384510+0000","last_active":"2024-01-16T14:57:08.360491+0000","last_peered":"2024-01-16T14:57:08.360491+0000","last_clean":"2024-01-16T14:57:08.360491+0000","last_became_active":"2024-01-16T14:57:07.384214+0000","last_became_peered":"2024-01-16T14:57:07.384214+0000","last_unstale":"2024-01-16T14:57:08.360491+0000","last_undegraded":"2024-01-16T14:57:08.360491+0000","last_fullsized":"2024-01-16T14:57:08.360491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:48.016705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856634+0000","last_change":"2024-01-16T14:57:06.562353+0000","last_active":"2024-01-16T14:57:08.856634+0000","last_peered":"2024-01-16T14:57:08.856634+0000","last_clean":"2024-01-16T14:57:08.856634+0000","last_became_active":"2024-01-16T14:57:06.562043+0000","last_became_peered":"2024-01-16T14:57:06.562043+0000","last_unstale":"2024-01-16T14:57:08.856634+0000","last_undegraded":"2024-01-16T14:57:08.856634+0000","last_fullsized":"2024-01-16T14:57:08.856634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:25.879858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856809+0000","last_change":"2024-01-16T14:57:07.383191+0000","last_active":"2024-01-16T14:57:08.856809+0000","last_peered":"2024-01-16T14:57:08.856809+0000","last_clean":"2024-01-16T14:57:08.856809+0000","last_became_active":"2024-01-16T14:57:07.379683+0000","last_became_peered":"2024-01-16T14:57:07.379683+0000","last_unstale":"2024-01-16T14:57:08.856809+0000","last_undegraded":"2024-01-16T14:57:08.856809+0000","last_fullsized":"2024-01-16T14:57:08.856809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:19.082440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856935+0000","last_change":"2024-01-16T14:57:06.548684+0000","last_active":"2024-01-16T14:57:08.856935+0000","last_peered":"2024-01-16T14:57:08.856935+0000","last_clean":"2024-01-16T14:57:08.856935+0000","last_became_active":"2024-01-16T14:57:06.547835+0000","last_became_peered":"2024-01-16T14:57:06.547835+0000","last_unstale":"2024-01-16T14:57:08.856935+0000","last_undegraded":"2024-01-16T14:57:08.856935+0000","last_fullsized":"2024-01-16T14:57:08.856935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:46.715195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360277+0000","last_change":"2024-01-16T14:57:07.384606+0000","last_active":"2024-01-16T14:57:08.360277+0000","last_peered":"2024-01-16T14:57:08.360277+0000","last_clean":"2024-01-16T14:57:08.360277+0000","last_became_active":"2024-01-16T14:57:07.384300+0000","last_became_peered":"2024-01-16T14:57:07.384300+0000","last_unstale":"2024-01-16T14:57:08.360277+0000","last_undegraded":"2024-01-16T14:57:08.360277+0000","last_fullsized":"2024-01-16T14:57:08.360277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359317+0000","last_change":"2024-01-16T14:57:06.555188+0000","last_active":"2024-01-16T14:57:08.359317+0000","last_peered":"2024-01-16T14:57:08.359317+0000","last_clean":"2024-01-16T14:57:08.359317+0000","last_became_active":"2024-01-16T14:57:06.554904+0000","last_became_peered":"2024-01-16T14:57:06.554904+0000","last_unstale":"2024-01-16T14:57:08.359317+0000","last_undegraded":"2024-01-16T14:57:08.359317+0000","last_fullsized":"2024-01-16T14:57:08.359317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:59.674406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360712+0000","last_change":"2024-01-16T14:57:07.386645+0000","last_active":"2024-01-16T14:57:08.360712+0000","last_peered":"2024-01-16T14:57:08.360712+0000","last_clean":"2024-01-16T14:57:08.360712+0000","last_became_active":"2024-01-16T14:57:07.384338+0000","last_became_peered":"2024-01-16T14:57:07.384338+0000","last_unstale":"2024-01-16T14:57:08.360712+0000","last_undegraded":"2024-01-16T14:57:08.360712+0000","last_fullsized":"2024-01-16T14:57:08.360712+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859980+0000","last_change":"2024-01-16T14:57:06.549964+0000","last_active":"2024-01-16T14:57:08.859980+0000","last_peered":"2024-01-16T14:57:08.859980+0000","last_clean":"2024-01-16T14:57:08.859980+0000","last_became_active":"2024-01-16T14:57:06.549594+0000","last_became_peered":"2024-01-16T14:57:06.549594+0000","last_unstale":"2024-01-16T14:57:08.859980+0000","last_undegraded":"2024-01-16T14:57:08.859980+0000","last_fullsized":"2024-01-16T14:57:08.859980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361966+0000","last_change":"2024-01-16T14:57:07.385296+0000","last_active":"2024-01-16T14:57:08.361966+0000","last_peered":"2024-01-16T14:57:08.361966+0000","last_clean":"2024-01-16T14:57:08.361966+0000","last_became_active":"2024-01-16T14:57:07.385019+0000","last_became_peered":"2024-01-16T14:57:07.385019+0000","last_unstale":"2024-01-16T14:57:08.361966+0000","last_undegraded":"2024-01-16T14:57:08.361966+0000","last_fullsized":"2024-01-16T14:57:08.361966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:08.100477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857357+0000","last_change":"2024-01-16T14:57:06.558586+0000","last_active":"2024-01-16T14:57:08.857357+0000","last_peered":"2024-01-16T14:57:08.857357+0000","last_clean":"2024-01-16T14:57:08.857357+0000","last_became_active":"2024-01-16T14:57:06.558143+0000","last_became_peered":"2024-01-16T14:57:06.558143+0000","last_unstale":"2024-01-16T14:57:08.857357+0000","last_undegraded":"2024-01-16T14:57:08.857357+0000","last_fullsized":"2024-01-16T14:57:08.857357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357386+0000","last_change":"2024-01-16T14:57:07.393946+0000","last_active":"2024-01-16T14:57:08.357386+0000","last_peered":"2024-01-16T14:57:08.357386+0000","last_clean":"2024-01-16T14:57:08.357386+0000","last_became_active":"2024-01-16T14:57:07.393108+0000","last_became_peered":"2024-01-16T14:57:07.393108+0000","last_unstale":"2024-01-16T14:57:08.357386+0000","last_undegraded":"2024-01-16T14:57:08.357386+0000","last_fullsized":"2024-01-16T14:57:08.357386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:27.319103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858814+0000","last_change":"2024-01-16T14:57:06.555365+0000","last_active":"2024-01-16T14:57:08.858814+0000","last_peered":"2024-01-16T14:57:08.858814+0000","last_clean":"2024-01-16T14:57:08.858814+0000","last_became_active":"2024-01-16T14:57:06.551851+0000","last_became_peered":"2024-01-16T14:57:06.551851+0000","last_unstale":"2024-01-16T14:57:08.858814+0000","last_undegraded":"2024-01-16T14:57:08.858814+0000","last_fullsized":"2024-01-16T14:57:08.858814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360922+0000","last_change":"2024-01-16T14:57:07.397246+0000","last_active":"2024-01-16T14:57:08.360922+0000","last_peered":"2024-01-16T14:57:08.360922+0000","last_clean":"2024-01-16T14:57:08.360922+0000","last_became_active":"2024-01-16T14:57:07.394277+0000","last_became_peered":"2024-01-16T14:57:07.394277+0000","last_unstale":"2024-01-16T14:57:08.360922+0000","last_undegraded":"2024-01-16T14:57:08.360922+0000","last_fullsized":"2024-01-16T14:57:08.360922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:29.126730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:57:07.357749+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:57:07.357749+0000","last_peered":"2024-01-16T14:57:07.357749+0000","last_clean":"2024-01-16T14:57:07.357749+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:57:07.357749+0000","last_undegraded":"2024-01-16T14:57:07.357749+0000","last_fullsized":"2024-01-16T14:57:07.357749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357443+0000","last_change":"2024-01-16T14:57:06.547364+0000","last_active":"2024-01-16T14:57:08.357443+0000","last_peered":"2024-01-16T14:57:08.357443+0000","last_clean":"2024-01-16T14:57:08.357443+0000","last_became_active":"2024-01-16T14:57:06.546800+0000","last_became_peered":"2024-01-16T14:57:06.546800+0000","last_unstale":"2024-01-16T14:57:08.357443+0000","last_undegraded":"2024-01-16T14:57:08.357443+0000","last_fullsized":"2024-01-16T14:57:08.357443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:27.163769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858385+0000","last_change":"2024-01-16T14:57:07.395614+0000","last_active":"2024-01-16T14:57:08.858385+0000","last_peered":"2024-01-16T14:57:08.858385+0000","last_clean":"2024-01-16T14:57:08.858385+0000","last_became_active":"2024-01-16T14:57:07.395330+0000","last_became_peered":"2024-01-16T14:57:07.395330+0000","last_unstale":"2024-01-16T14:57:08.858385+0000","last_undegraded":"2024-01-16T14:57:08.858385+0000","last_fullsized":"2024-01-16T14:57:08.858385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:39.853227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359578+0000","last_change":"2024-01-16T14:57:06.548309+0000","last_active":"2024-01-16T14:57:08.359578+0000","last_peered":"2024-01-16T14:57:08.359578+0000","last_clean":"2024-01-16T14:57:08.359578+0000","last_became_active":"2024-01-16T14:57:06.547952+0000","last_became_peered":"2024-01-16T14:57:06.547952+0000","last_unstale":"2024-01-16T14:57:08.359578+0000","last_undegraded":"2024-01-16T14:57:08.359578+0000","last_fullsized":"2024-01-16T14:57:08.359578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:13.055332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.410994+0000","last_change":"2024-01-16T14:57:07.375198+0000","last_active":"2024-01-16T14:57:08.410994+0000","last_peered":"2024-01-16T14:57:08.410994+0000","last_clean":"2024-01-16T14:57:08.410994+0000","last_became_active":"2024-01-16T14:57:07.374043+0000","last_became_peered":"2024-01-16T14:57:07.374043+0000","last_unstale":"2024-01-16T14:57:08.410994+0000","last_undegraded":"2024-01-16T14:57:08.410994+0000","last_fullsized":"2024-01-16T14:57:08.410994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:11.783582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857406+0000","last_change":"2024-01-16T14:57:06.553851+0000","last_active":"2024-01-16T14:57:08.857406+0000","last_peered":"2024-01-16T14:57:08.857406+0000","last_clean":"2024-01-16T14:57:08.857406+0000","last_became_active":"2024-01-16T14:57:06.552712+0000","last_became_peered":"2024-01-16T14:57:06.552712+0000","last_unstale":"2024-01-16T14:57:08.857406+0000","last_undegraded":"2024-01-16T14:57:08.857406+0000","last_fullsized":"2024-01-16T14:57:08.857406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.855802+0000","last_change":"2024-01-16T14:57:07.372041+0000","last_active":"2024-01-16T14:57:08.855802+0000","last_peered":"2024-01-16T14:57:08.855802+0000","last_clean":"2024-01-16T14:57:08.855802+0000","last_became_active":"2024-01-16T14:57:07.371750+0000","last_became_peered":"2024-01-16T14:57:07.371750+0000","last_unstale":"2024-01-16T14:57:08.855802+0000","last_undegraded":"2024-01-16T14:57:08.855802+0000","last_fullsized":"2024-01-16T14:57:08.855802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:53.804355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358274+0000","last_change":"2024-01-16T14:57:06.548768+0000","last_active":"2024-01-16T14:57:08.358274+0000","last_peered":"2024-01-16T14:57:08.358274+0000","last_clean":"2024-01-16T14:57:08.358274+0000","last_became_active":"2024-01-16T14:57:06.548446+0000","last_became_peered":"2024-01-16T14:57:06.548446+0000","last_unstale":"2024-01-16T14:57:08.358274+0000","last_undegraded":"2024-01-16T14:57:08.358274+0000","last_fullsized":"2024-01-16T14:57:08.358274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:17:06.341461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358850+0000","last_change":"2024-01-16T14:57:07.378261+0000","last_active":"2024-01-16T14:57:08.358850+0000","last_peered":"2024-01-16T14:57:08.358850+0000","last_clean":"2024-01-16T14:57:08.358850+0000","last_became_active":"2024-01-16T14:57:07.377940+0000","last_became_peered":"2024-01-16T14:57:07.377940+0000","last_unstale":"2024-01-16T14:57:08.358850+0000","last_undegraded":"2024-01-16T14:57:08.358850+0000","last_fullsized":"2024-01-16T14:57:08.358850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:34.447700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856741+0000","last_change":"2024-01-16T14:57:06.545628+0000","last_active":"2024-01-16T14:57:08.856741+0000","last_peered":"2024-01-16T14:57:08.856741+0000","last_clean":"2024-01-16T14:57:08.856741+0000","last_became_active":"2024-01-16T14:57:06.545341+0000","last_became_peered":"2024-01-16T14:57:06.545341+0000","last_unstale":"2024-01-16T14:57:08.856741+0000","last_undegraded":"2024-01-16T14:57:08.856741+0000","last_fullsized":"2024-01-16T14:57:08.856741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:54.829524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358936+0000","last_change":"2024-01-16T14:57:07.392468+0000","last_active":"2024-01-16T14:57:08.358936+0000","last_peered":"2024-01-16T14:57:08.358936+0000","last_clean":"2024-01-16T14:57:08.358936+0000","last_became_active":"2024-01-16T14:57:07.392091+0000","last_became_peered":"2024-01-16T14:57:07.392091+0000","last_unstale":"2024-01-16T14:57:08.358936+0000","last_undegraded":"2024-01-16T14:57:08.358936+0000","last_fullsized":"2024-01-16T14:57:08.358936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:13.918545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.361159+0000","last_change":"2024-01-16T14:57:06.551276+0000","last_active":"2024-01-16T14:57:08.361159+0000","last_peered":"2024-01-16T14:57:08.361159+0000","last_clean":"2024-01-16T14:57:08.361159+0000","last_became_active":"2024-01-16T14:57:06.550951+0000","last_became_peered":"2024-01-16T14:57:06.550951+0000","last_unstale":"2024-01-16T14:57:08.361159+0000","last_undegraded":"2024-01-16T14:57:08.361159+0000","last_fullsized":"2024-01-16T14:57:08.361159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:15.986936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856799+0000","last_change":"2024-01-16T14:57:07.370960+0000","last_active":"2024-01-16T14:57:08.856799+0000","last_peered":"2024-01-16T14:57:08.856799+0000","last_clean":"2024-01-16T14:57:08.856799+0000","last_became_active":"2024-01-16T14:57:07.370635+0000","last_became_peered":"2024-01-16T14:57:07.370635+0000","last_unstale":"2024-01-16T14:57:08.856799+0000","last_undegraded":"2024-01-16T14:57:08.856799+0000","last_fullsized":"2024-01-16T14:57:08.856799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:36.350912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:09.484382+0000","last_change":"2024-01-16T14:57:06.553886+0000","last_active":"2024-01-16T14:57:09.484382+0000","last_peered":"2024-01-16T14:57:09.484382+0000","last_clean":"2024-01-16T14:57:09.484382+0000","last_became_active":"2024-01-16T14:57:06.553590+0000","last_became_peered":"2024-01-16T14:57:06.553590+0000","last_unstale":"2024-01-16T14:57:09.484382+0000","last_undegraded":"2024-01-16T14:57:09.484382+0000","last_fullsized":"2024-01-16T14:57:09.484382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:16.749765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859350+0000","last_change":"2024-01-16T14:57:07.388410+0000","last_active":"2024-01-16T14:57:08.859350+0000","last_peered":"2024-01-16T14:57:08.859350+0000","last_clean":"2024-01-16T14:57:08.859350+0000","last_became_active":"2024-01-16T14:57:07.388132+0000","last_became_peered":"2024-01-16T14:57:07.388132+0000","last_unstale":"2024-01-16T14:57:08.859350+0000","last_undegraded":"2024-01-16T14:57:08.859350+0000","last_fullsized":"2024-01-16T14:57:08.859350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:24.013168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.857725+0000","last_change":"2024-01-16T14:57:06.560743+0000","last_active":"2024-01-16T14:57:08.857725+0000","last_peered":"2024-01-16T14:57:08.857725+0000","last_clean":"2024-01-16T14:57:08.857725+0000","last_became_active":"2024-01-16T14:57:06.560557+0000","last_became_peered":"2024-01-16T14:57:06.560557+0000","last_unstale":"2024-01-16T14:57:08.857725+0000","last_undegraded":"2024-01-16T14:57:08.857725+0000","last_fullsized":"2024-01-16T14:57:08.857725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:05.946854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411687+0000","last_change":"2024-01-16T14:57:07.378876+0000","last_active":"2024-01-16T14:57:08.411687+0000","last_peered":"2024-01-16T14:57:08.411687+0000","last_clean":"2024-01-16T14:57:08.411687+0000","last_became_active":"2024-01-16T14:57:07.378103+0000","last_became_peered":"2024-01-16T14:57:07.378103+0000","last_unstale":"2024-01-16T14:57:08.411687+0000","last_undegraded":"2024-01-16T14:57:08.411687+0000","last_fullsized":"2024-01-16T14:57:08.411687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:56.487710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.359610+0000","last_change":"2024-01-16T14:57:06.548228+0000","last_active":"2024-01-16T14:57:08.359610+0000","last_peered":"2024-01-16T14:57:08.359610+0000","last_clean":"2024-01-16T14:57:08.359610+0000","last_became_active":"2024-01-16T14:57:06.547577+0000","last_became_peered":"2024-01-16T14:57:06.547577+0000","last_unstale":"2024-01-16T14:57:08.359610+0000","last_undegraded":"2024-01-16T14:57:08.359610+0000","last_fullsized":"2024-01-16T14:57:08.359610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:30.438420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357044+0000","last_change":"2024-01-16T14:57:07.377694+0000","last_active":"2024-01-16T14:57:08.357044+0000","last_peered":"2024-01-16T14:57:08.357044+0000","last_clean":"2024-01-16T14:57:08.357044+0000","last_became_active":"2024-01-16T14:57:07.377016+0000","last_became_peered":"2024-01-16T14:57:07.377016+0000","last_unstale":"2024-01-16T14:57:08.357044+0000","last_undegraded":"2024-01-16T14:57:08.357044+0000","last_fullsized":"2024-01-16T14:57:08.357044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:08:30.500155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856442+0000","last_change":"2024-01-16T14:57:06.545409+0000","last_active":"2024-01-16T14:57:08.856442+0000","last_peered":"2024-01-16T14:57:08.856442+0000","last_clean":"2024-01-16T14:57:08.856442+0000","last_became_active":"2024-01-16T14:57:06.545133+0000","last_became_peered":"2024-01-16T14:57:06.545133+0000","last_unstale":"2024-01-16T14:57:08.856442+0000","last_undegraded":"2024-01-16T14:57:08.856442+0000","last_fullsized":"2024-01-16T14:57:08.856442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:18.053787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858230+0000","last_change":"2024-01-16T14:57:07.373879+0000","last_active":"2024-01-16T14:57:08.858230+0000","last_peered":"2024-01-16T14:57:08.858230+0000","last_clean":"2024-01-16T14:57:08.858230+0000","last_became_active":"2024-01-16T14:57:07.373537+0000","last_became_peered":"2024-01-16T14:57:07.373537+0000","last_unstale":"2024-01-16T14:57:08.858230+0000","last_undegraded":"2024-01-16T14:57:08.858230+0000","last_fullsized":"2024-01-16T14:57:08.858230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:29.534447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856963+0000","last_change":"2024-01-16T14:57:06.548121+0000","last_active":"2024-01-16T14:57:08.856963+0000","last_peered":"2024-01-16T14:57:08.856963+0000","last_clean":"2024-01-16T14:57:08.856963+0000","last_became_active":"2024-01-16T14:57:06.547868+0000","last_became_peered":"2024-01-16T14:57:06.547868+0000","last_unstale":"2024-01-16T14:57:08.856963+0000","last_undegraded":"2024-01-16T14:57:08.856963+0000","last_fullsized":"2024-01-16T14:57:08.856963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:59.882931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.858908+0000","last_change":"2024-01-16T14:57:07.378203+0000","last_active":"2024-01-16T14:57:08.858908+0000","last_peered":"2024-01-16T14:57:08.858908+0000","last_clean":"2024-01-16T14:57:08.858908+0000","last_became_active":"2024-01-16T14:57:07.377843+0000","last_became_peered":"2024-01-16T14:57:07.377843+0000","last_unstale":"2024-01-16T14:57:08.858908+0000","last_undegraded":"2024-01-16T14:57:08.858908+0000","last_fullsized":"2024-01-16T14:57:08.858908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:13.535659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360527+0000","last_change":"2024-01-16T14:57:06.559586+0000","last_active":"2024-01-16T14:57:08.360527+0000","last_peered":"2024-01-16T14:57:08.360527+0000","last_clean":"2024-01-16T14:57:08.360527+0000","last_became_active":"2024-01-16T14:57:06.558281+0000","last_became_peered":"2024-01-16T14:57:06.558281+0000","last_unstale":"2024-01-16T14:57:08.360527+0000","last_undegraded":"2024-01-16T14:57:08.360527+0000","last_fullsized":"2024-01-16T14:57:08.360527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:10.397222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.856775+0000","last_change":"2024-01-16T14:57:07.375900+0000","last_active":"2024-01-16T14:57:08.856775+0000","last_peered":"2024-01-16T14:57:08.856775+0000","last_clean":"2024-01-16T14:57:08.856775+0000","last_became_active":"2024-01-16T14:57:07.375625+0000","last_became_peered":"2024-01-16T14:57:07.375625+0000","last_unstale":"2024-01-16T14:57:08.856775+0000","last_undegraded":"2024-01-16T14:57:08.856775+0000","last_fullsized":"2024-01-16T14:57:08.856775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:10.170833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.358102+0000","last_change":"2024-01-16T14:57:07.391481+0000","last_active":"2024-01-16T14:57:08.358102+0000","last_peered":"2024-01-16T14:57:08.358102+0000","last_clean":"2024-01-16T14:57:08.358102+0000","last_became_active":"2024-01-16T14:57:07.391111+0000","last_became_peered":"2024-01-16T14:57:07.391111+0000","last_unstale":"2024-01-16T14:57:08.358102+0000","last_undegraded":"2024-01-16T14:57:08.358102+0000","last_fullsized":"2024-01-16T14:57:08.358102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:10.222246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360198+0000","last_change":"2024-01-16T14:57:06.553585+0000","last_active":"2024-01-16T14:57:08.360198+0000","last_peered":"2024-01-16T14:57:08.360198+0000","last_clean":"2024-01-16T14:57:08.360198+0000","last_became_active":"2024-01-16T14:57:06.553259+0000","last_became_peered":"2024-01-16T14:57:06.553259+0000","last_unstale":"2024-01-16T14:57:08.360198+0000","last_undegraded":"2024-01-16T14:57:08.360198+0000","last_fullsized":"2024-01-16T14:57:08.360198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:36.111764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.357259+0000","last_change":"2024-01-16T14:57:07.394393+0000","last_active":"2024-01-16T14:57:08.357259+0000","last_peered":"2024-01-16T14:57:08.357259+0000","last_clean":"2024-01-16T14:57:08.357259+0000","last_became_active":"2024-01-16T14:57:07.394135+0000","last_became_peered":"2024-01-16T14:57:07.394135+0000","last_unstale":"2024-01-16T14:57:08.357259+0000","last_undegraded":"2024-01-16T14:57:08.357259+0000","last_fullsized":"2024-01-16T14:57:08.357259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:58.823967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.360937+0000","last_change":"2024-01-16T14:57:06.554017+0000","last_active":"2024-01-16T14:57:08.360937+0000","last_peered":"2024-01-16T14:57:08.360937+0000","last_clean":"2024-01-16T14:57:08.360937+0000","last_became_active":"2024-01-16T14:57:06.553376+0000","last_became_peered":"2024-01-16T14:57:06.553376+0000","last_unstale":"2024-01-16T14:57:08.360937+0000","last_undegraded":"2024-01-16T14:57:08.360937+0000","last_fullsized":"2024-01-16T14:57:08.360937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:35.199600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411544+0000","last_change":"2024-01-16T14:57:07.395607+0000","last_active":"2024-01-16T14:57:08.411544+0000","last_peered":"2024-01-16T14:57:08.411544+0000","last_clean":"2024-01-16T14:57:08.411544+0000","last_became_active":"2024-01-16T14:57:07.395288+0000","last_became_peered":"2024-01-16T14:57:07.395288+0000","last_unstale":"2024-01-16T14:57:08.411544+0000","last_undegraded":"2024-01-16T14:57:08.411544+0000","last_fullsized":"2024-01-16T14:57:08.411544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.892629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859322+0000","last_change":"2024-01-16T14:57:06.549558+0000","last_active":"2024-01-16T14:57:08.859322+0000","last_peered":"2024-01-16T14:57:08.859322+0000","last_clean":"2024-01-16T14:57:08.859322+0000","last_became_active":"2024-01-16T14:57:06.545986+0000","last_became_peered":"2024-01-16T14:57:06.545986+0000","last_unstale":"2024-01-16T14:57:08.859322+0000","last_undegraded":"2024-01-16T14:57:08.859322+0000","last_fullsized":"2024-01-16T14:57:08.859322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:10.632270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.859106+0000","last_change":"2024-01-16T14:57:07.384928+0000","last_active":"2024-01-16T14:57:08.859106+0000","last_peered":"2024-01-16T14:57:08.859106+0000","last_clean":"2024-01-16T14:57:08.859106+0000","last_became_active":"2024-01-16T14:57:07.384613+0000","last_became_peered":"2024-01-16T14:57:07.384613+0000","last_unstale":"2024-01-16T14:57:08.859106+0000","last_undegraded":"2024-01-16T14:57:08.859106+0000","last_fullsized":"2024-01-16T14:57:08.859106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:06.508696+0000","last_clean_scrub_stamp":"2024-01-16T14:57:06.508696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:19.171826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:57:08.411722+0000","last_change":"2024-01-16T14:57:06.552465+0000","last_active":"2024-01-16T14:57:08.411722+0000","last_peered":"2024-01-16T14:57:08.411722+0000","last_clean":"2024-01-16T14:57:08.411722+0000","last_became_active":"2024-01-16T14:57:06.552103+0000","last_became_peered":"2024-01-16T14:57:06.552103+0000","last_unstale":"2024-01-16T14:57:08.411722+0000","last_undegraded":"2024-01-16T14:57:08.411722+0000","last_fullsized":"2024-01-16T14:57:08.411722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:05.357384+0000","last_clean_scrub_stamp":"2024-01-16T14:57:05.357384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:35.728588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.335}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1219999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3740000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":831488,"data_stored":669734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:57:17.813 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T14:57:17.813 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T14:57:17.813 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T14:57:17.814 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T14:57:18.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.072+0000 7fda27400640 1 Processor -- start 2024-01-16T14:57:18.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.072+0000 7fda27400640 1 -- start start 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.073+0000 7fda27400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fda2015a6b0 0x7fda2015aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.073+0000 7fda27400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20070e60 0x7fda20071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.073+0000 7fda27400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda20070410 con 0x7fda200cea00 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.073+0000 7fda27400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2015a530 con 0x7fda2015a6b0 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.073+0000 7fda27400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200adb00 con 0x7fda20070e60 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.073+0000 7fda17c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fda200cea00 0x7fda2015afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40094/0 (socket says 172.21.15.141:40094) 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.073+0000 7fda17c00640 1 -- 172.21.15.141:0/490005395 learned_addr learned my addr 172.21.15.141:0/490005395 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda24a00640 1 --2- 172.21.15.141:0/490005395 >> v2:172.21.15.188:3300/0 conn(0x7fda2015a6b0 0x7fda2015aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316739807 0 0) 0x7fda20070410 con 0x7fda200cea00 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda00005730 con 0x7fda200cea00 2024-01-16T14:57:18.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda25400640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20070e60 0x7fda20071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4289824088 0 0) 0x7fda00005730 con 0x7fda200cea00 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20070e60 msgr2=0x7fda20071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20070e60 0x7fda20071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 >> v2:172.21.15.188:3300/0 conn(0x7fda2015a6b0 msgr2=0x7fda2015aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 --2- 172.21.15.141:0/490005395 >> v2:172.21.15.188:3300/0 conn(0x7fda2015a6b0 0x7fda2015aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200adde0 con 0x7fda200cea00 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fda08001670 con 0x7fda200cea00 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 >> v1:172.21.15.141:6789/0 conn(0x7fda200cea00 legacy=0x7fda2015afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda00005b80 0x7fda00005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda000069d0 0x7fda00006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20070e60 0x7fda00007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2015a530 con 0x7fda000069d0 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200adb00 con 0x7fda00005b80 2024-01-16T14:57:18.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.074+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0000dc40 con 0x7fda20070e60 2024-01-16T14:57:18.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.075+0000 7fda17c00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda000069d0 0x7fda00006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.075+0000 7fda17c00640 1 -- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20070e60 msgr2=0x7fda00007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:18.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.075+0000 7fda17c00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20070e60 0x7fda00007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.075+0000 7fda17c00640 1 -- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda00005b80 msgr2=0x7fda00005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:18.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.075+0000 7fda24a00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda00005b80 0x7fda00005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.075+0000 7fda17c00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda00005b80 0x7fda00005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.075+0000 7fda17c00640 1 -- 172.21.15.141:0/490005395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda00005730 con 0x7fda000069d0 2024-01-16T14:57:18.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.076+0000 7fda17c00640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda000069d0 0x7fda00006e50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c210 tx=0x7fda00007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b04502224754df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.076+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08005ad0 con 0x7fda000069d0 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.076+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda08005c30 con 0x7fda000069d0 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.076+0000 7fda1fe00640 1 -- 172.21.15.141:0/490005395 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0802c7e0 con 0x7fda000069d0 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.077+0000 7fda27400640 1 -- 172.21.15.141:0/490005395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda000069d0 msgr2=0x7fda00006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.077+0000 7fda27400640 1 --2- 172.21.15.141:0/490005395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda000069d0 0x7fda00006e50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c210 tx=0x7fda00007a00 comp rx=0 tx=0).stop 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.077+0000 7fda25400640 1 -- 172.21.15.141:0/490005395 reap_dead start 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.077+0000 7fda27400640 1 -- 172.21.15.141:0/490005395 shutdown_connections 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.077+0000 7fda27400640 1 -- 172.21.15.141:0/490005395 >> 172.21.15.141:0/490005395 conn(0x7fda200b0c00 msgr2=0x7fda200b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.077+0000 7fda27400640 1 -- 172.21.15.141:0/490005395 shutdown_connections 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.077+0000 7fda27400640 1 -- 172.21.15.141:0/490005395 wait complete. 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.078+0000 7fda27400640 1 Processor -- start 2024-01-16T14:57:18.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.079+0000 7fda27400640 1 -- start start 2024-01-16T14:57:18.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.079+0000 7fda27400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda000069d0 0x7fda2009b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.079+0000 7fda27400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda20099a60 0x7fda20099ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.079+0000 7fda27400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20148260 0x7fda201486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.079+0000 7fda27400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2009b130 con 0x7fda20099a60 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.079+0000 7fda27400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20158120 con 0x7fda000069d0 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.079+0000 7fda27400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2009a520 con 0x7fda20148260 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda20099a60 0x7fda20099ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda20099a60 0x7fda20099ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48702/0 (socket says 172.21.15.141:48702) 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 -- 172.21.15.141:0/1788097090 learned_addr learned my addr 172.21.15.141:0/1788097090 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda24a00640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda000069d0 0x7fda2009b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 -- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20148260 msgr2=0x7fda201486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20148260 0x7fda201486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 -- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda000069d0 msgr2=0x7fda2009b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda000069d0 0x7fda2009b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 -- 172.21.15.141:0/1788097090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2009a800 con 0x7fda20099a60 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.080+0000 7fda17c00640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda20099a60 0x7fda20099ee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fda0802c380 tx=0x7fda08001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.081+0000 7fda1d600640 1 -- 172.21.15.141:0/1788097090 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08039070 con 0x7fda20099a60 2024-01-16T14:57:18.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.081+0000 7fda1d600640 1 -- 172.21.15.141:0/1788097090 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0802dd00 con 0x7fda20099a60 2024-01-16T14:57:18.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.081+0000 7fda1d600640 1 -- 172.21.15.141:0/1788097090 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0803c040 con 0x7fda20099a60 2024-01-16T14:57:18.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.081+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2009aae0 con 0x7fda20099a60 2024-01-16T14:57:18.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.081+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2009c420 con 0x7fda20099a60 2024-01-16T14:57:18.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.082+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2009b2a0 con 0x7fda20099a60 2024-01-16T14:57:18.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.083+0000 7fda1d600640 1 -- 172.21.15.141:0/1788097090 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fda08032070 con 0x7fda20099a60 2024-01-16T14:57:18.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.084+0000 7fda1d600640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd9f806b2f0 0x7fd9f806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.084+0000 7fda1d600640 1 -- 172.21.15.141:0/1788097090 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda080aada0 con 0x7fda20099a60 2024-01-16T14:57:18.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.084+0000 7fda24a00640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd9f806b2f0 0x7fd9f806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.085+0000 7fda24a00640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd9f806b2f0 0x7fd9f806d7b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fda10009a90 tx=0x7fda10008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.088+0000 7fda1d600640 1 -- 172.21.15.141:0/1788097090 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda08075ca0 con 0x7fda20099a60 2024-01-16T14:57:18.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.301+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fda20067640 con 0x7fda20099a60 2024-01-16T14:57:18.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.302+0000 7fda1d600640 1 -- 172.21.15.141:0/1788097090 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fda08046020 con 0x7fda20099a60 2024-01-16T14:57:18.284 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:18.284 INFO:teuthology.orchestra.run.smithi141.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T14:57:18.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd9f806b2f0 msgr2=0x7fd9f806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd9f806b2f0 0x7fd9f806d7b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fda10009a90 tx=0x7fda10008040 comp rx=0 tx=0).stop 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda20099a60 msgr2=0x7fda20099ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda20099a60 0x7fda20099ee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fda0802c380 tx=0x7fda08001a00 comp rx=0 tx=0).stop 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 shutdown_connections 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd9f806b2f0 0x7fd9f806d7b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda20148260 0x7fda201486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda20099a60 0x7fda20099ee0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 --2- 172.21.15.141:0/1788097090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda000069d0 0x7fda2009b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 >> 172.21.15.141:0/1788097090 conn(0x7fda200b0c00 msgr2=0x7fda200acb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 shutdown_connections 2024-01-16T14:57:18.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.305+0000 7fda27400640 1 -- 172.21.15.141:0/1788097090 wait complete. 2024-01-16T14:57:18.303 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T14:57:18.304 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:18.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.515+0000 7fd68be00640 1 Processor -- start 2024-01-16T14:57:18.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.515+0000 7fd68be00640 1 -- start start 2024-01-16T14:57:18.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.517+0000 7fd68be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd68c0d0100 0x7fd68c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.517+0000 7fd68be00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68c0d0a10 con 0x7fd68c0d0f40 2024-01-16T14:57:18.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.517+0000 7fd68be00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68c0cf080 con 0x7fd68c160920 2024-01-16T14:57:18.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.517+0000 7fd68be00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68c0c6be0 con 0x7fd68c0d0100 2024-01-16T14:57:18.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.517+0000 7fd681e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd68c0d0100 0x7fd68c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.517+0000 7fd68b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd68c0d0f40 0x7fd68c0d1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40108/0 (socket says 172.21.15.141:40108) 2024-01-16T14:57:18.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.517+0000 7fd68b400640 1 -- 172.21.15.141:0/3375380241 learned_addr learned my addr 172.21.15.141:0/3375380241 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:18.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.518+0000 7fd681e00640 1 -- 172.21.15.141:0/3375380241 >> v1:172.21.15.188:6789/0 conn(0x7fd68c160920 legacy=0x7fd68c0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:18.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.518+0000 7fd681e00640 1 -- 172.21.15.141:0/3375380241 >> v1:172.21.15.141:6789/0 conn(0x7fd68c0d0f40 legacy=0x7fd68c0d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.518+0000 7fd681e00640 1 -- 172.21.15.141:0/3375380241 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c0ce7e0 con 0x7fd68c0d0100 2024-01-16T14:57:18.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.519+0000 7fd681e00640 1 --2- 172.21.15.141:0/3375380241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0100 0x7fd68c0b3f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6740097a0 tx=0x7fd67402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c073ae3253974659 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.519+0000 7fd68a000640 1 -- 172.21.15.141:0/3375380241 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd674031830 con 0x7fd68c0d0100 2024-01-16T14:57:18.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.519+0000 7fd68a000640 1 -- 172.21.15.141:0/3375380241 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd674031990 con 0x7fd68c0d0100 2024-01-16T14:57:18.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.519+0000 7fd68a000640 1 -- 172.21.15.141:0/3375380241 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd674031d20 con 0x7fd68c0d0100 2024-01-16T14:57:18.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.519+0000 7fd68be00640 1 -- 172.21.15.141:0/3375380241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0100 msgr2=0x7fd68c0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.519+0000 7fd68be00640 1 --2- 172.21.15.141:0/3375380241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0100 0x7fd68c0b3f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6740097a0 tx=0x7fd67402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.520+0000 7fd68be00640 1 -- 172.21.15.141:0/3375380241 shutdown_connections 2024-01-16T14:57:18.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.520+0000 7fd68be00640 1 --2- 172.21.15.141:0/3375380241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0100 0x7fd68c0b3f30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.520+0000 7fd68be00640 1 -- 172.21.15.141:0/3375380241 >> 172.21.15.141:0/3375380241 conn(0x7fd68c0d3090 msgr2=0x7fd68c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:18.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.520+0000 7fd68be00640 1 -- 172.21.15.141:0/3375380241 shutdown_connections 2024-01-16T14:57:18.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.520+0000 7fd68be00640 1 -- 172.21.15.141:0/3375380241 wait complete. 2024-01-16T14:57:18.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.521+0000 7fd68be00640 1 Processor -- start 2024-01-16T14:57:18.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.521+0000 7fd68be00640 1 -- start start 2024-01-16T14:57:18.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd68be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd68c0d0100 0x7fd68c0b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd68be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0f40 0x7fd68c0b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd68be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd68c160920 0x7fd68c0b8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd68be00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0b6480 con 0x7fd68c0d0f40 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd68be00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0b6760 con 0x7fd68c160920 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd68be00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0b6a40 con 0x7fd68c0d0100 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd681e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0f40 0x7fd68c0b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.522+0000 7fd681e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0f40 0x7fd68c0b7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48714/0 (socket says 172.21.15.141:48714) 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd681e00640 1 -- 172.21.15.141:0/2667854694 learned_addr learned my addr 172.21.15.141:0/2667854694 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd68aa00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd68c0d0100 0x7fd68c0b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd68b400640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd68c160920 0x7fd68c0b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd681e00640 1 -- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd68c0d0100 msgr2=0x7fd68c0b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd681e00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd68c0d0100 0x7fd68c0b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd681e00640 1 -- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd68c160920 msgr2=0x7fd68c0b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd681e00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd68c160920 0x7fd68c0b8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd681e00640 1 -- 172.21.15.141:0/2667854694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd674009400 con 0x7fd68c0d0f40 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd68aa00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd68c0d0100 0x7fd68c0b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.523+0000 7fd681e00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0f40 0x7fd68c0b7f10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6740098c0 tx=0x7fd6740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.524+0000 7fd683400640 1 -- 172.21.15.141:0/2667854694 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67403ba70 con 0x7fd68c0d0f40 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.524+0000 7fd683400640 1 -- 172.21.15.141:0/2667854694 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67403bbf0 con 0x7fd68c0d0f40 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.524+0000 7fd683400640 1 -- 172.21.15.141:0/2667854694 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6740453f0 con 0x7fd68c0d0f40 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.524+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c0b6d20 con 0x7fd68c0d0f40 2024-01-16T14:57:18.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.524+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c0b6190 con 0x7fd68c0d0f40 2024-01-16T14:57:18.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.525+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c0b7240 con 0x7fd68c0d0f40 2024-01-16T14:57:18.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.526+0000 7fd683400640 1 -- 172.21.15.141:0/2667854694 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd67404e450 con 0x7fd68c0d0f40 2024-01-16T14:57:18.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.526+0000 7fd683400640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd66406b2c0 0x7fd66406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.527+0000 7fd683400640 1 -- 172.21.15.141:0/2667854694 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6740abeb0 con 0x7fd68c0d0f40 2024-01-16T14:57:18.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.527+0000 7fd68aa00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd66406b2c0 0x7fd66406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.528+0000 7fd68aa00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd66406b2c0 0x7fd66406d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd67c007eb0 tx=0x7fd67c0073c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.531+0000 7fd683400640 1 -- 172.21.15.141:0/2667854694 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd68c0b7240 con 0x7fd68c0d0f40 2024-01-16T14:57:18.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.720+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd68c0b7240 con 0x7fd68c0d0f40 2024-01-16T14:57:18.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.723+0000 7fd683400640 1 -- 172.21.15.141:0/2667854694 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7fd68c0b7240 con 0x7fd68c0d0f40 2024-01-16T14:57:18.705 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:18.705 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:18.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd66406b2c0 msgr2=0x7fd66406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd66406b2c0 0x7fd66406d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd67c007eb0 tx=0x7fd67c0073c0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0f40 msgr2=0x7fd68c0b7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0f40 0x7fd68c0b7f10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6740098c0 tx=0x7fd6740046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 shutdown_connections 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd66406b2c0 0x7fd66406d780 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd68c160920 0x7fd68c0b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd68c0d0f40 0x7fd68c0b7f10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 --2- 172.21.15.141:0/2667854694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd68c0d0100 0x7fd68c0b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 >> 172.21.15.141:0/2667854694 conn(0x7fd68c0d3090 msgr2=0x7fd68c0b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 shutdown_connections 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.726+0000 7fd68be00640 1 -- 172.21.15.141:0/2667854694 wait complete. 2024-01-16T14:57:18.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:18.726 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:18.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.948+0000 7f2a20000640 1 Processor -- start 2024-01-16T14:57:18.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.948+0000 7f2a20000640 1 -- start start 2024-01-16T14:57:18.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a20000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2a18099a60 0x7f2a18099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a20000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a1806e7c0 0x7f2a1806ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a20000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a180cfa70 con 0x7f2a1815e880 2024-01-16T14:57:18.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a20000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1809a3e0 con 0x7f2a18099a60 2024-01-16T14:57:18.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a20000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180cff00 con 0x7f2a1806e7c0 2024-01-16T14:57:18.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a1cc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2a18099a60 0x7f2a18099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a1d600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2a1815e880 0x7f2a1806ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40112/0 (socket says 172.21.15.141:40112) 2024-01-16T14:57:18.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.950+0000 7f2a1d600640 1 -- 172.21.15.141:0/1469743891 learned_addr learned my addr 172.21.15.141:0/1469743891 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.951+0000 7f2a17e00640 1 -- 172.21.15.141:0/1469743891 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225363356 0 0) 0x7f2a180cfa70 con 0x7f2a1815e880 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.951+0000 7f2a1e000640 1 --2- 172.21.15.141:0/1469743891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a1806e7c0 0x7f2a1806ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.951+0000 7f2a17e00640 1 -- 172.21.15.141:0/1469743891 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8003520 con 0x7f2a1815e880 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.951+0000 7f2a1cc00640 1 -- 172.21.15.141:0/1469743891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a1806e7c0 msgr2=0x7f2a1806ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.951+0000 7f2a1cc00640 1 --2- 172.21.15.141:0/1469743891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a1806e7c0 0x7f2a1806ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.951+0000 7f2a1cc00640 1 -- 172.21.15.141:0/1469743891 >> v1:172.21.15.141:6789/0 conn(0x7f2a1815e880 legacy=0x7f2a1806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.951+0000 7f2a1cc00640 1 -- 172.21.15.141:0/1469743891 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1806fc70 con 0x7f2a18099a60 2024-01-16T14:57:18.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.952+0000 7f2a1cc00640 1 --2- 172.21.15.141:0/1469743891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a18099a60 0x7f2a18099ea0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae4deef043f7a92 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.952+0000 7f2a17e00640 1 -- 172.21.15.141:0/1469743891 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00030610 con 0x7f2a18099a60 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.952+0000 7f2a17e00640 1 -- 172.21.15.141:0/1469743891 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a00030770 con 0x7f2a18099a60 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.952+0000 7f2a17e00640 1 -- 172.21.15.141:0/1469743891 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00030b00 con 0x7f2a18099a60 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 -- 172.21.15.141:0/1469743891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a18099a60 msgr2=0x7f2a18099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 --2- 172.21.15.141:0/1469743891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a18099a60 0x7f2a18099ea0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b960 comp rx=0 tx=0).stop 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 -- 172.21.15.141:0/1469743891 shutdown_connections 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 --2- 172.21.15.141:0/1469743891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a1806e7c0 0x7f2a1806ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 --2- 172.21.15.141:0/1469743891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a18099a60 0x7f2a18099ea0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 -- 172.21.15.141:0/1469743891 >> 172.21.15.141:0/1469743891 conn(0x7f2a180b2d00 msgr2=0x7f2a180b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 -- 172.21.15.141:0/1469743891 shutdown_connections 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.953+0000 7f2a20000640 1 -- 172.21.15.141:0/1469743891 wait complete. 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.955+0000 7f2a20000640 1 Processor -- start 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.955+0000 7f2a20000640 1 -- start start 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a20000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a1806e7c0 0x7f2a180b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a20000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a18099a60 0x7f2a1815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a20000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a1815e880 0x7f2a180c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a20000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1815c6c0 con 0x7f2a1806e7c0 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a20000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180ca250 con 0x7f2a1815e880 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a20000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180b5e20 con 0x7f2a18099a60 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a1806e7c0 0x7f2a180b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a1806e7c0 0x7f2a180b57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48738/0 (socket says 172.21.15.141:48738) 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 -- 172.21.15.141:0/1556415159 learned_addr learned my addr 172.21.15.141:0/1556415159 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1e000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a1815e880 0x7f2a180c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1cc00640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a18099a60 0x7f2a1815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 -- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a18099a60 msgr2=0x7f2a1815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a18099a60 0x7f2a1815c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 -- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a1815e880 msgr2=0x7f2a180c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a1815e880 0x7f2a180c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1d600640 1 -- 172.21.15.141:0/1556415159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a00009400 con 0x7f2a1806e7c0 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.956+0000 7f2a1e000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a1815e880 0x7f2a180c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:18.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.957+0000 7f2a1d600640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a1806e7c0 0x7f2a180b57d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800cbc0 tx=0x7f2a0800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.957+0000 7f2a15600640 1 -- 172.21.15.141:0/1556415159 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0800ad40 con 0x7f2a1806e7c0 2024-01-16T14:57:18.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.957+0000 7f2a15600640 1 -- 172.21.15.141:0/1556415159 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a08012400 con 0x7f2a1806e7c0 2024-01-16T14:57:18.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.957+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a180b6100 con 0x7f2a1806e7c0 2024-01-16T14:57:18.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.957+0000 7f2a15600640 1 -- 172.21.15.141:0/1556415159 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0800aea0 con 0x7f2a1806e7c0 2024-01-16T14:57:18.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.957+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a180b66c0 con 0x7f2a1806e7c0 2024-01-16T14:57:18.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.958+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a180c5ba0 con 0x7f2a1806e7c0 2024-01-16T14:57:18.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.960+0000 7f2a15600640 1 -- 172.21.15.141:0/1556415159 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2a08007070 con 0x7f2a1806e7c0 2024-01-16T14:57:18.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.960+0000 7f2a15600640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29f006b2c0 0x7f29f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:18.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.960+0000 7f2a15600640 1 -- 172.21.15.141:0/1556415159 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a08089c30 con 0x7f2a1806e7c0 2024-01-16T14:57:18.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.961+0000 7f2a1cc00640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29f006b2c0 0x7f29f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:18.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.962+0000 7f2a1cc00640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29f006b2c0 0x7f29f006d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2a00036000 tx=0x7f2a0003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:18.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:18.965+0000 7f2a15600640 1 -- 172.21.15.141:0/1556415159 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a08054a90 con 0x7f2a1806e7c0 2024-01-16T14:57:19.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.141+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2a180675d0 con 0x7f2a1806e7c0 2024-01-16T14:57:19.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.141+0000 7f2a15600640 1 -- 172.21.15.141:0/1556415159 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2a08017020 con 0x7f2a1806e7c0 2024-01-16T14:57:19.123 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.143+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29f006b2c0 msgr2=0x7f29f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.143+0000 7f2a20000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29f006b2c0 0x7f29f006d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2a00036000 tx=0x7f2a0003b040 comp rx=0 tx=0).stop 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.143+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a1806e7c0 msgr2=0x7f2a180b57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a1806e7c0 0x7f2a180b57d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800cbc0 tx=0x7f2a0800a430 comp rx=0 tx=0).stop 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 shutdown_connections 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29f006b2c0 0x7f29f006d780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a1815e880 0x7f2a180c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a18099a60 0x7f2a1815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 --2- 172.21.15.141:0/1556415159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a1806e7c0 0x7f2a180b57d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 >> 172.21.15.141:0/1556415159 conn(0x7f2a180b2d00 msgr2=0x7f2a180aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 shutdown_connections 2024-01-16T14:57:19.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.144+0000 7f2a20000640 1 -- 172.21.15.141:0/1556415159 wait complete. 2024-01-16T14:57:19.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:19.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.363+0000 7f5e51600640 1 Processor -- start 2024-01-16T14:57:19.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.363+0000 7f5e51600640 1 -- start start 2024-01-16T14:57:19.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e51600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c066300 0x7f5e4c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e51600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5e4c06b930 0x7f5e4c06dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e51600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4c0b3880 con 0x7f5e4c15a110 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e51600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4c06bdc0 con 0x7f5e4c06b930 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e51600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0b46e0 con 0x7f5e4c066300 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e4aa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5e4c15a110 0x7f5e4c06d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40124/0 (socket says 172.21.15.141:40124) 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e4aa00640 1 -- 172.21.15.141:0/1531150932 learned_addr learned my addr 172.21.15.141:0/1531150932 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.364+0000 7f5e4b400640 1 --2- 172.21.15.141:0/1531150932 >> v2:172.21.15.141:3300/0 conn(0x7f5e4c06b930 0x7f5e4c06dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.365+0000 7f5e41e00640 1 --2- 172.21.15.141:0/1531150932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c066300 0x7f5e4c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.365+0000 7f5e4a000640 1 -- 172.21.15.141:0/1531150932 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795190540 0 0) 0x7f5e4c0b3880 con 0x7f5e4c15a110 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.365+0000 7f5e4a000640 1 -- 172.21.15.141:0/1531150932 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e2c003520 con 0x7f5e4c15a110 2024-01-16T14:57:19.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.365+0000 7f5e4b400640 1 -- 172.21.15.141:0/1531150932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c066300 msgr2=0x7f5e4c06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.365+0000 7f5e4b400640 1 --2- 172.21.15.141:0/1531150932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c066300 0x7f5e4c06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.365+0000 7f5e4b400640 1 -- 172.21.15.141:0/1531150932 >> v1:172.21.15.141:6789/0 conn(0x7f5e4c15a110 legacy=0x7f5e4c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.365+0000 7f5e4b400640 1 -- 172.21.15.141:0/1531150932 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c15a4f0 con 0x7f5e4c06b930 2024-01-16T14:57:19.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.366+0000 7f5e4b400640 1 --2- 172.21.15.141:0/1531150932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c06b930 0x7f5e4c06dc10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009d90 tx=0x7f5e4402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7329ef6cbfa5bae2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.366+0000 7f5e4a000640 1 -- 172.21.15.141:0/1531150932 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4402cc50 con 0x7f5e4c06b930 2024-01-16T14:57:19.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.366+0000 7f5e4a000640 1 -- 172.21.15.141:0/1531150932 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4402cdb0 con 0x7f5e4c06b930 2024-01-16T14:57:19.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.366+0000 7f5e4a000640 1 -- 172.21.15.141:0/1531150932 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44034660 con 0x7f5e4c06b930 2024-01-16T14:57:19.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 -- 172.21.15.141:0/1531150932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c06b930 msgr2=0x7f5e4c06dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 --2- 172.21.15.141:0/1531150932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c06b930 0x7f5e4c06dc10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009d90 tx=0x7f5e4402bea0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 -- 172.21.15.141:0/1531150932 shutdown_connections 2024-01-16T14:57:19.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 --2- 172.21.15.141:0/1531150932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c06b930 0x7f5e4c06dc10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 --2- 172.21.15.141:0/1531150932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c066300 0x7f5e4c06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 -- 172.21.15.141:0/1531150932 >> 172.21.15.141:0/1531150932 conn(0x7f5e4c070b10 msgr2=0x7f5e4c070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 -- 172.21.15.141:0/1531150932 shutdown_connections 2024-01-16T14:57:19.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.367+0000 7f5e51600640 1 -- 172.21.15.141:0/1531150932 wait complete. 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.368+0000 7f5e51600640 1 Processor -- start 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.369+0000 7f5e51600640 1 -- start start 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.369+0000 7f5e51600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c066300 0x7f5e4c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.369+0000 7f5e51600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e4c06b930 0x7f5e4c0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.369+0000 7f5e51600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c15a110 0x7f5e4c141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.369+0000 7f5e51600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c142f10 con 0x7f5e4c066300 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.369+0000 7f5e51600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0a8a00 con 0x7f5e4c06b930 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.369+0000 7f5e51600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c142010 con 0x7f5e4c15a110 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c066300 0x7f5e4c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c066300 0x7f5e4c15a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48764/0 (socket says 172.21.15.141:48764) 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 -- 172.21.15.141:0/2177222347 learned_addr learned my addr 172.21.15.141:0/2177222347 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e41e00640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e4c06b930 0x7f5e4c0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 -- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c15a110 msgr2=0x7f5e4c141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c15a110 0x7f5e4c141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 -- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e4c06b930 msgr2=0x7f5e4c0a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e4c06b930 0x7f5e4c0a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 -- 172.21.15.141:0/2177222347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44009a70 con 0x7f5e4c066300 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e41e00640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e4c06b930 0x7f5e4c0a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e4aa00640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c066300 0x7f5e4c15a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c440 tx=0x7f5e3c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.370+0000 7f5e43400640 1 -- 172.21.15.141:0/2177222347 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c005ce0 con 0x7f5e4c066300 2024-01-16T14:57:19.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.371+0000 7f5e43400640 1 -- 172.21.15.141:0/2177222347 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c005e40 con 0x7f5e4c066300 2024-01-16T14:57:19.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.371+0000 7f5e43400640 1 -- 172.21.15.141:0/2177222347 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0072b0 con 0x7f5e4c066300 2024-01-16T14:57:19.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.371+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4c142290 con 0x7f5e4c066300 2024-01-16T14:57:19.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.371+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4c142850 con 0x7f5e4c066300 2024-01-16T14:57:19.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.372+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4c0675d0 con 0x7f5e4c066300 2024-01-16T14:57:19.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.373+0000 7f5e43400640 1 -- 172.21.15.141:0/2177222347 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5e3c007410 con 0x7f5e4c066300 2024-01-16T14:57:19.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.373+0000 7f5e43400640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e2406b2c0 0x7f5e2406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.373+0000 7f5e43400640 1 -- 172.21.15.141:0/2177222347 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e3c089100 con 0x7f5e4c066300 2024-01-16T14:57:19.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.374+0000 7f5e41e00640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e2406b2c0 0x7f5e2406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.375+0000 7f5e41e00640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e2406b2c0 0x7f5e2406d780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c153dc0 tx=0x7f5e34008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.377+0000 7f5e43400640 1 -- 172.21.15.141:0/2177222347 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e3c054030 con 0x7f5e4c066300 2024-01-16T14:57:19.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.556+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e4c071c10 con 0x7f5e4c066300 2024-01-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.557+0000 7f5e43400640 1 -- 172.21.15.141:0/2177222347 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e3c057650 con 0x7f5e4c066300 2024-01-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e2406b2c0 msgr2=0x7f5e2406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e2406b2c0 0x7f5e2406d780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c153dc0 tx=0x7f5e34008040 comp rx=0 tx=0).stop 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c066300 msgr2=0x7f5e4c15a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c066300 0x7f5e4c15a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c440 tx=0x7f5e3c00c900 comp rx=0 tx=0).stop 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 shutdown_connections 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e2406b2c0 0x7f5e2406d780 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e4c15a110 0x7f5e4c141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e4c06b930 0x7f5e4c0a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 --2- 172.21.15.141:0/2177222347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e4c066300 0x7f5e4c15a730 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 >> 172.21.15.141:0/2177222347 conn(0x7f5e4c070b10 msgr2=0x7f5e4c155ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 shutdown_connections 2024-01-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.560+0000 7f5e51600640 1 -- 172.21.15.141:0/2177222347 wait complete. 2024-01-16T14:57:19.561 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:19.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.776+0000 7f45c8c00640 1 Processor -- start 2024-01-16T14:57:19.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.776+0000 7f45c8c00640 1 -- start start 2024-01-16T14:57:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.778+0000 7f45c8c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45c413df80 0x7f45c415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.778+0000 7f45c8c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c406c470 con 0x7f45c40cf9a0 2024-01-16T14:57:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.778+0000 7f45c8c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c406c750 con 0x7f45c413e360 2024-01-16T14:57:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.778+0000 7f45c8c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c406ca30 con 0x7f45c413df80 2024-01-16T14:57:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.778+0000 7f45c1e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45c413df80 0x7f45c415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.778+0000 7f45c2800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f45c40cf9a0 0x7f45c40cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40136/0 (socket says 172.21.15.141:40136) 2024-01-16T14:57:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.778+0000 7f45c2800640 1 -- 172.21.15.141:0/3840968301 learned_addr learned my addr 172.21.15.141:0/3840968301 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.779+0000 7f45c1e00640 1 -- 172.21.15.141:0/3840968301 >> v1:172.21.15.188:6789/0 conn(0x7f45c413e360 legacy=0x7f45c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.779+0000 7f45c1e00640 1 -- 172.21.15.141:0/3840968301 >> v1:172.21.15.141:6789/0 conn(0x7f45c40cf9a0 legacy=0x7f45c40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.779+0000 7f45c1e00640 1 -- 172.21.15.141:0/3840968301 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c406cd10 con 0x7f45c413df80 2024-01-16T14:57:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.779+0000 7f45c0a00640 1 -- 172.21.15.141:0/3840968301 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394202020 0 0) 0x7f45c406c470 con 0x7f45c40cf9a0 2024-01-16T14:57:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.780+0000 7f45c1e00640 1 --2- 172.21.15.141:0/3840968301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c413df80 0x7f45c415bee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f45b40097a0 tx=0x7f45b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37aff041a0cdb3da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.780+0000 7f45c0a00640 1 -- 172.21.15.141:0/3840968301 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b4031830 con 0x7f45c413df80 2024-01-16T14:57:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.780+0000 7f45c0a00640 1 -- 172.21.15.141:0/3840968301 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b4031990 con 0x7f45c413df80 2024-01-16T14:57:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.780+0000 7f45c0a00640 1 -- 172.21.15.141:0/3840968301 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b4031d20 con 0x7f45c413df80 2024-01-16T14:57:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.781+0000 7f45c8c00640 1 -- 172.21.15.141:0/3840968301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c413df80 msgr2=0x7f45c415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.781+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3840968301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c413df80 0x7f45c415bee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f45b40097a0 tx=0x7f45b402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.781+0000 7f45c8c00640 1 -- 172.21.15.141:0/3840968301 shutdown_connections 2024-01-16T14:57:19.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.781+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3840968301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c413df80 0x7f45c415bee0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.781+0000 7f45c8c00640 1 -- 172.21.15.141:0/3840968301 >> 172.21.15.141:0/3840968301 conn(0x7f45c406a800 msgr2=0x7f45c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.781+0000 7f45c8c00640 1 -- 172.21.15.141:0/3840968301 shutdown_connections 2024-01-16T14:57:19.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.781+0000 7f45c8c00640 1 -- 172.21.15.141:0/3840968301 wait complete. 2024-01-16T14:57:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.783+0000 7f45c8c00640 1 Processor -- start 2024-01-16T14:57:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.783+0000 7f45c8c00640 1 -- start start 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c8c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c40cf9a0 0x7f45c414b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c8c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45c413df80 0x7f45c413a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45c413e360 0x7f45c40b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c8c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c41f1920 con 0x7f45c40cf9a0 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c8c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c41f1c00 con 0x7f45c413e360 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c8c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c414abe0 con 0x7f45c413df80 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c40cf9a0 0x7f45c414b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c40cf9a0 0x7f45c414b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48796/0 (socket says 172.21.15.141:48796) 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 -- 172.21.15.141:0/3378360692 learned_addr learned my addr 172.21.15.141:0/3378360692 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 -- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45c413df80 msgr2=0x7f45c413a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45c413df80 0x7f45c413a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 -- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45c413e360 msgr2=0x7f45c40b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45c413e360 0x7f45c40b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.784+0000 7f45c1e00640 1 -- 172.21.15.141:0/3378360692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b4009400 con 0x7f45c40cf9a0 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.785+0000 7f45c1e00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c40cf9a0 0x7f45c414b6c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f45b40098c0 tx=0x7f45b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.785+0000 7f45ba000640 1 -- 172.21.15.141:0/3378360692 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b403ba70 con 0x7f45c40cf9a0 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.785+0000 7f45ba000640 1 -- 172.21.15.141:0/3378360692 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b403bbd0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.785+0000 7f45ba000640 1 -- 172.21.15.141:0/3378360692 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b40453f0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.785+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c414aec0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.785+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c40b3b50 con 0x7f45c40cf9a0 2024-01-16T14:57:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.786+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c414b3e0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.787+0000 7f45ba000640 1 -- 172.21.15.141:0/3378360692 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f45b404e450 con 0x7f45c40cf9a0 2024-01-16T14:57:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.788+0000 7f45ba000640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459c06b2c0 0x7f459c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:19.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.788+0000 7f45ba000640 1 -- 172.21.15.141:0/3378360692 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45b40abed0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.789+0000 7f45c1400640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459c06b2c0 0x7f459c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:19.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.790+0000 7f45c1400640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459c06b2c0 0x7f459c06d780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45c414c3e0 tx=0x7f45ac007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:19.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.796+0000 7f45ba000640 1 -- 172.21.15.141:0/3378360692 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45c414b3e0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.971+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45c414b3e0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.972+0000 7f45ba000640 1 -- 172.21.15.141:0/3378360692 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f45c414b3e0 con 0x7f45c40cf9a0 2024-01-16T14:57:19.956 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.974+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459c06b2c0 msgr2=0x7f459c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.974+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459c06b2c0 0x7f459c06d780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45c414c3e0 tx=0x7f45ac007d30 comp rx=0 tx=0).stop 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.974+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c40cf9a0 msgr2=0x7f45c414b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.974+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c40cf9a0 0x7f45c414b6c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f45b40098c0 tx=0x7f45b40046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 shutdown_connections 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459c06b2c0 0x7f459c06d780 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45c413e360 0x7f45c40b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45c413df80 0x7f45c413a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 --2- 172.21.15.141:0/3378360692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45c40cf9a0 0x7f45c414b6c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 >> 172.21.15.141:0/3378360692 conn(0x7f45c406a800 msgr2=0x7f45c40aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:19.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 shutdown_connections 2024-01-16T14:57:19.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:19.975+0000 7f45c8c00640 1 -- 172.21.15.141:0/3378360692 wait complete. 2024-01-16T14:57:19.973 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:20.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.196+0000 7f7123e00640 1 Processor -- start 2024-01-16T14:57:20.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.197+0000 7f7123e00640 1 -- start start 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.198+0000 7f7123e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f712413cfa0 0x7f712415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.198+0000 7f7123e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f712413fcb0 0x7f71240b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.198+0000 7f7123e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240cdec0 con 0x7f71240cd530 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.198+0000 7f7123e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71240ce1a0 con 0x7f712413cfa0 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.198+0000 7f7123e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71240ce480 con 0x7f712413fcb0 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7122a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f712413cfa0 0x7f712415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7122a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f712413cfa0 0x7f712415afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48808/0 (socket says 172.21.15.141:48808) 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7122a00640 1 -- 172.21.15.141:0/4280338561 learned_addr learned my addr 172.21.15.141:0/4280338561 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7123400640 1 --2- 172.21.15.141:0/4280338561 >> v2:172.21.15.188:3300/0 conn(0x7f712413fcb0 0x7f71240b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7122a00640 1 -- 172.21.15.141:0/4280338561 >> v2:172.21.15.188:3300/0 conn(0x7f712413fcb0 msgr2=0x7f71240b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:20.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7122a00640 1 --2- 172.21.15.141:0/4280338561 >> v2:172.21.15.188:3300/0 conn(0x7f712413fcb0 0x7f71240b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7122a00640 1 -- 172.21.15.141:0/4280338561 >> v1:172.21.15.188:6789/0 conn(0x7f71240cd530 legacy=0x7f712413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:20.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.199+0000 7f7122a00640 1 -- 172.21.15.141:0/4280338561 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240ce760 con 0x7f712413cfa0 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.200+0000 7f7122a00640 1 --2- 172.21.15.141:0/4280338561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f712413cfa0 0x7f712415afb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f71240b3fd0 tx=0x7f711402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec1a4da7b58eba4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.200+0000 7f7121600640 1 -- 172.21.15.141:0/4280338561 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711402c9f0 con 0x7f712413cfa0 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7121600640 1 -- 172.21.15.141:0/4280338561 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711402cb50 con 0x7f712413cfa0 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7121600640 1 -- 172.21.15.141:0/4280338561 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140343f0 con 0x7f712413cfa0 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 -- 172.21.15.141:0/4280338561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f712413cfa0 msgr2=0x7f712415afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 --2- 172.21.15.141:0/4280338561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f712413cfa0 0x7f712415afb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f71240b3fd0 tx=0x7f711402be50 comp rx=0 tx=0).stop 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 -- 172.21.15.141:0/4280338561 shutdown_connections 2024-01-16T14:57:20.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 --2- 172.21.15.141:0/4280338561 >> v2:172.21.15.188:3300/0 conn(0x7f712413fcb0 0x7f71240b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 --2- 172.21.15.141:0/4280338561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f712413cfa0 0x7f712415afb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 -- 172.21.15.141:0/4280338561 >> 172.21.15.141:0/4280338561 conn(0x7f712406a800 msgr2=0x7f712406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 -- 172.21.15.141:0/4280338561 shutdown_connections 2024-01-16T14:57:20.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.201+0000 7f7123e00640 1 -- 172.21.15.141:0/4280338561 wait complete. 2024-01-16T14:57:20.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.203+0000 7f7123e00640 1 Processor -- start 2024-01-16T14:57:20.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.203+0000 7f7123e00640 1 -- start start 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7123e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71240cd530 0x7f712413b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7123e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f712413cfa0 0x7f712413eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7123e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f712413fcb0 0x7f7124153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7123e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712413f530 con 0x7f71240cd530 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7123e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7124166720 con 0x7f712413cfa0 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7123e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7124165e30 con 0x7f712413fcb0 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7122a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71240cd530 0x7f712413b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7122a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71240cd530 0x7f712413b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48824/0 (socket says 172.21.15.141:48824) 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.204+0000 7f7122a00640 1 -- 172.21.15.141:0/3051501296 learned_addr learned my addr 172.21.15.141:0/3051501296 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f7122a00640 1 -- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f712413fcb0 msgr2=0x7f7124153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f7122a00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f712413fcb0 0x7f7124153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f7122a00640 1 -- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f712413cfa0 msgr2=0x7f712413eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f7122a00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f712413cfa0 0x7f712413eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f7122a00640 1 -- 172.21.15.141:0/3051501296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7114009b30 con 0x7f71240cd530 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f7122a00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71240cd530 0x7f712413b560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7114036040 tx=0x7f7114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f711aa00640 1 -- 172.21.15.141:0/3051501296 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114009510 con 0x7f71240cd530 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f711aa00640 1 -- 172.21.15.141:0/3051501296 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711403e450 con 0x7f71240cd530 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71241660d0 con 0x7f71240cd530 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.205+0000 7f711aa00640 1 -- 172.21.15.141:0/3051501296 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711403e7e0 con 0x7f71240cd530 2024-01-16T14:57:20.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.206+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712413bc70 con 0x7f71240cd530 2024-01-16T14:57:20.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.207+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71240b2540 con 0x7f71240cd530 2024-01-16T14:57:20.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.208+0000 7f711aa00640 1 -- 172.21.15.141:0/3051501296 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f711403e940 con 0x7f71240cd530 2024-01-16T14:57:20.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.208+0000 7f711aa00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70fc06b2c0 0x7f70fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.209+0000 7f711aa00640 1 -- 172.21.15.141:0/3051501296 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71140aca50 con 0x7f71240cd530 2024-01-16T14:57:20.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.209+0000 7f7122000640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70fc06b2c0 0x7f70fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.210+0000 7f7122000640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70fc06b2c0 0x7f70fc06d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7124157ab0 tx=0x7f710c0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.213+0000 7f711aa00640 1 -- 172.21.15.141:0/3051501296 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7114077820 con 0x7f71240cd530 2024-01-16T14:57:20.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.443+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71240ae850 con 0x7f71240cd530 2024-01-16T14:57:20.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.444+0000 7f711aa00640 1 -- 172.21.15.141:0/3051501296 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f711407ae40 con 0x7f71240cd530 2024-01-16T14:57:20.426 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:20.426 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:20.426 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70fc06b2c0 msgr2=0x7f70fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70fc06b2c0 0x7f70fc06d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7124157ab0 tx=0x7f710c0079b0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71240cd530 msgr2=0x7f712413b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71240cd530 0x7f712413b560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7114036040 tx=0x7f7114004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 shutdown_connections 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70fc06b2c0 0x7f70fc06d780 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f712413fcb0 0x7f7124153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f712413cfa0 0x7f712413eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 --2- 172.21.15.141:0/3051501296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71240cd530 0x7f712413b560 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 >> 172.21.15.141:0/3051501296 conn(0x7f712406a800 msgr2=0x7f71240aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 shutdown_connections 2024-01-16T14:57:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.447+0000 7f7123e00640 1 -- 172.21.15.141:0/3051501296 wait complete. 2024-01-16T14:57:20.447 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:20.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.668+0000 7f1eb8a00640 1 Processor -- start 2024-01-16T14:57:20.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.668+0000 7f1eb8a00640 1 -- start start 2024-01-16T14:57:20.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.669+0000 7f1eb8a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1eb013df80 0x7f1eb015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.669+0000 7f1eb8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.669+0000 7f1eb8a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1eb015f9c0 0x7f1eb00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.669+0000 7f1eb8a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb006c470 con 0x7f1eb015f9c0 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.669+0000 7f1eb8a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb006c750 con 0x7f1eb013df80 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.669+0000 7f1eb8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb006ca30 con 0x7f1eb013e360 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1eb015f9c0 0x7f1eb00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1eb015f9c0 0x7f1eb00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48828/0 (socket says 172.21.15.141:48828) 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 -- 172.21.15.141:0/2632712547 learned_addr learned my addr 172.21.15.141:0/2632712547 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6000640 1 --2- 172.21.15.141:0/2632712547 >> v2:172.21.15.188:3300/0 conn(0x7f1eb013df80 0x7f1eb015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb5600640 1 --2- 172.21.15.141:0/2632712547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 -- 172.21.15.141:0/2632712547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 msgr2=0x7f1eb00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:20.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 --2- 172.21.15.141:0/2632712547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 -- 172.21.15.141:0/2632712547 >> v2:172.21.15.188:3300/0 conn(0x7f1eb013df80 msgr2=0x7f1eb015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:20.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 --2- 172.21.15.141:0/2632712547 >> v2:172.21.15.188:3300/0 conn(0x7f1eb013df80 0x7f1eb015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.670+0000 7f1eb6a00640 1 -- 172.21.15.141:0/2632712547 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb006cd10 con 0x7f1eb015f9c0 2024-01-16T14:57:20.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.672+0000 7f1eb6a00640 1 --2- 172.21.15.141:0/2632712547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb00cf7d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0072770 tx=0x7f1ea802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68dd7ceda721748 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.672+0000 7f1eb4c00640 1 -- 172.21.15.141:0/2632712547 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8031860 con 0x7f1eb015f9c0 2024-01-16T14:57:20.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.672+0000 7f1eb4c00640 1 -- 172.21.15.141:0/2632712547 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea80319c0 con 0x7f1eb015f9c0 2024-01-16T14:57:20.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.672+0000 7f1eb4c00640 1 -- 172.21.15.141:0/2632712547 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8031d50 con 0x7f1eb015f9c0 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 -- 172.21.15.141:0/2632712547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 msgr2=0x7f1eb00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/2632712547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb00cf7d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0072770 tx=0x7f1ea802b900 comp rx=0 tx=0).stop 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 -- 172.21.15.141:0/2632712547 shutdown_connections 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/2632712547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb00cf7d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/2632712547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/2632712547 >> v2:172.21.15.188:3300/0 conn(0x7f1eb013df80 0x7f1eb015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 -- 172.21.15.141:0/2632712547 >> 172.21.15.141:0/2632712547 conn(0x7f1eb006a800 msgr2=0x7f1eb006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 -- 172.21.15.141:0/2632712547 shutdown_connections 2024-01-16T14:57:20.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.673+0000 7f1eb8a00640 1 -- 172.21.15.141:0/2632712547 wait complete. 2024-01-16T14:57:20.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.675+0000 7f1eb8a00640 1 Processor -- start 2024-01-16T14:57:20.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.675+0000 7f1eb8a00640 1 -- start start 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb013df80 0x7f1eb0070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb015cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb01544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb8a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0070a90 con 0x7f1eb015f9c0 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb01f6f10 con 0x7f1eb013df80 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0070440 con 0x7f1eb013e360 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb01544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb01544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48842/0 (socket says 172.21.15.141:48842) 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb6a00640 1 -- 172.21.15.141:0/229360694 learned_addr learned my addr 172.21.15.141:0/229360694 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb6000640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb013df80 0x7f1eb0070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.676+0000 7f1eb5600640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb015cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb6a00640 1 -- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 msgr2=0x7f1eb015cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb6a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb015cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb6a00640 1 -- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb013df80 msgr2=0x7f1eb0070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb6a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb013df80 0x7f1eb0070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb6a00640 1 -- 172.21.15.141:0/229360694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80094d0 con 0x7f1eb015f9c0 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb6a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb01544e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80090b0 tx=0x7f1ea80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eae000640 1 -- 172.21.15.141:0/229360694 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea803ac60 con 0x7f1eb015f9c0 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eae000640 1 -- 172.21.15.141:0/229360694 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea803adc0 con 0x7f1eb015f9c0 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb0070720 con 0x7f1eb015f9c0 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eae000640 1 -- 172.21.15.141:0/229360694 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8044630 con 0x7f1eb015f9c0 2024-01-16T14:57:20.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.677+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb015d630 con 0x7f1eb015f9c0 2024-01-16T14:57:20.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.678+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb0070c00 con 0x7f1eb015f9c0 2024-01-16T14:57:20.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.679+0000 7f1eae000640 1 -- 172.21.15.141:0/229360694 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1ea804d450 con 0x7f1eb015f9c0 2024-01-16T14:57:20.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.681+0000 7f1eae000640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e8806b2c0 0x7f1e8806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:20.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.681+0000 7f1eae000640 1 -- 172.21.15.141:0/229360694 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ea80abff0 con 0x7f1eb015f9c0 2024-01-16T14:57:20.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.681+0000 7f1eb6000640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e8806b2c0 0x7f1e8806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:20.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.682+0000 7f1eb6000640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e8806b2c0 0x7f1e8806d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0006fd0 tx=0x7f1ea0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:20.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.685+0000 7f1eae000640 1 -- 172.21.15.141:0/229360694 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ea8076e50 con 0x7f1eb015f9c0 2024-01-16T14:57:20.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.872+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eb015d630 con 0x7f1eb015f9c0 2024-01-16T14:57:20.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.873+0000 7f1eae000640 1 -- 172.21.15.141:0/229360694 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f1eb015d630 con 0x7f1eb015f9c0 2024-01-16T14:57:20.855 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:20.856 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:20.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e8806b2c0 msgr2=0x7f1e8806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e8806b2c0 0x7f1e8806d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0006fd0 tx=0x7f1ea0008040 comp rx=0 tx=0).stop 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 msgr2=0x7f1eb01544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb01544e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80090b0 tx=0x7f1ea80046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 shutdown_connections 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e8806b2c0 0x7f1e8806d780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb015f9c0 0x7f1eb01544e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb013e360 0x7f1eb015cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 --2- 172.21.15.141:0/229360694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb013df80 0x7f1eb0070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.875+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 >> 172.21.15.141:0/229360694 conn(0x7f1eb006a800 msgr2=0x7f1eb006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.876+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 shutdown_connections 2024-01-16T14:57:20.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:20.876+0000 7f1eb8a00640 1 -- 172.21.15.141:0/229360694 wait complete. 2024-01-16T14:57:20.874 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:20.875 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:21.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.098+0000 7fa407e00640 1 Processor -- start 2024-01-16T14:57:21.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.098+0000 7fa407e00640 1 -- start start 2024-01-16T14:57:21.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408071e40 0x7fa4080ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa40815d770 0x7fa40815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40815e140 con 0x7fa408143810 2024-01-16T14:57:21.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa408143bf0 con 0x7fa40815d770 2024-01-16T14:57:21.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080722c0 con 0x7fa408071e40 2024-01-16T14:57:21.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa406a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa408143810 0x7fa40815e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40144/0 (socket says 172.21.15.141:40144) 2024-01-16T14:57:21.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa40815d770 0x7fa40815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa406a00640 1 -- 172.21.15.141:0/3687773211 learned_addr learned my addr 172.21.15.141:0/3687773211 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:21.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa405600640 1 -- 172.21.15.141:0/3687773211 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151579053 0 0) 0x7fa40815e140 con 0x7fa408143810 2024-01-16T14:57:21.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa406000640 1 --2- 172.21.15.141:0/3687773211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408071e40 0x7fa4080ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa405600640 1 -- 172.21.15.141:0/3687773211 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e8003520 con 0x7fa408143810 2024-01-16T14:57:21.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407400640 1 -- 172.21.15.141:0/3687773211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408071e40 msgr2=0x7fa4080ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:21.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407400640 1 --2- 172.21.15.141:0/3687773211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408071e40 0x7fa4080ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407400640 1 -- 172.21.15.141:0/3687773211 >> v1:172.21.15.141:6789/0 conn(0x7fa408143810 legacy=0x7fa40815e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa407400640 1 -- 172.21.15.141:0/3687773211 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40806c730 con 0x7fa40815d770 2024-01-16T14:57:21.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.100+0000 7fa406000640 1 --2- 172.21.15.141:0/3687773211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408071e40 0x7fa4080ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa407400640 1 --2- 172.21.15.141:0/3687773211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40815dc00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa400009d90 tx=0x7fa40002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87563d7240616f51 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa405600640 1 -- 172.21.15.141:0/3687773211 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40002cb30 con 0x7fa40815d770 2024-01-16T14:57:21.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa405600640 1 -- 172.21.15.141:0/3687773211 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40002cc90 con 0x7fa40815d770 2024-01-16T14:57:21.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa405600640 1 -- 172.21.15.141:0/3687773211 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400034440 con 0x7fa40815d770 2024-01-16T14:57:21.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa407e00640 1 -- 172.21.15.141:0/3687773211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 msgr2=0x7fa40815dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa407e00640 1 --2- 172.21.15.141:0/3687773211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40815dc00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa400009d90 tx=0x7fa40002bd80 comp rx=0 tx=0).stop 2024-01-16T14:57:21.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa407e00640 1 -- 172.21.15.141:0/3687773211 shutdown_connections 2024-01-16T14:57:21.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa407e00640 1 --2- 172.21.15.141:0/3687773211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40815dc00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa407e00640 1 --2- 172.21.15.141:0/3687773211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408071e40 0x7fa4080ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.102+0000 7fa407e00640 1 -- 172.21.15.141:0/3687773211 >> 172.21.15.141:0/3687773211 conn(0x7fa40806a7d0 msgr2=0x7fa40806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.103+0000 7fa407e00640 1 -- 172.21.15.141:0/3687773211 shutdown_connections 2024-01-16T14:57:21.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.103+0000 7fa407e00640 1 -- 172.21.15.141:0/3687773211 wait complete. 2024-01-16T14:57:21.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.104+0000 7fa407e00640 1 Processor -- start 2024-01-16T14:57:21.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.104+0000 7fa407e00640 1 -- start start 2024-01-16T14:57:21.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.105+0000 7fa407e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408071e40 0x7fa4081f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.105+0000 7fa407e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408143810 0x7fa40806d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.105+0000 7fa407e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.105+0000 7fa407e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080ce8e0 con 0x7fa40815d770 2024-01-16T14:57:21.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.105+0000 7fa407e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080cd840 con 0x7fa408071e40 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.105+0000 7fa407e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080cdb20 con 0x7fa408143810 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40806d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48868/0 (socket says 172.21.15.141:48868) 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 -- 172.21.15.141:0/3798161155 learned_addr learned my addr 172.21.15.141:0/3798161155 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa406a00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408071e40 0x7fa4081f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa406000640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408143810 0x7fa40806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 -- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408143810 msgr2=0x7fa40806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408143810 0x7fa40806d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 -- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408071e40 msgr2=0x7fa4081f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408071e40 0x7fa4081f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 -- 172.21.15.141:0/3798161155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400009a70 con 0x7fa40815d770 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407400640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40806d7d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa400009d90 tx=0x7fa400004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa3fea00640 1 -- 172.21.15.141:0/3798161155 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400009450 con 0x7fa40815d770 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa3fea00640 1 -- 172.21.15.141:0/3798161155 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400040450 con 0x7fa40815d770 2024-01-16T14:57:21.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4080cde00 con 0x7fa40815d770 2024-01-16T14:57:21.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.106+0000 7fa3fea00640 1 -- 172.21.15.141:0/3798161155 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400040800 con 0x7fa40815d770 2024-01-16T14:57:21.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.107+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4080ce3c0 con 0x7fa40815d770 2024-01-16T14:57:21.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.108+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4080ce5e0 con 0x7fa40815d770 2024-01-16T14:57:21.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.109+0000 7fa3fea00640 1 -- 172.21.15.141:0/3798161155 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa400040960 con 0x7fa40815d770 2024-01-16T14:57:21.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.110+0000 7fa3fea00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3e006b2c0 0x7fa3e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.110+0000 7fa3fea00640 1 -- 172.21.15.141:0/3798161155 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4000ac720 con 0x7fa40815d770 2024-01-16T14:57:21.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.111+0000 7fa406a00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3e006b2c0 0x7fa3e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.111+0000 7fa406a00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3e006b2c0 0x7fa3e006d780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80011e0 tx=0x7fa3f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.114+0000 7fa3fea00640 1 -- 172.21.15.141:0/3798161155 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4000775c0 con 0x7fa40815d770 2024-01-16T14:57:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.304+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa408067450 con 0x7fa40815d770 2024-01-16T14:57:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.305+0000 7fa3fea00640 1 -- 172.21.15.141:0/3798161155 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa40007abe0 con 0x7fa40815d770 2024-01-16T14:57:21.286 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3e006b2c0 msgr2=0x7fa3e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3e006b2c0 0x7fa3e006d780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80011e0 tx=0x7fa3f8009040 comp rx=0 tx=0).stop 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 msgr2=0x7fa40806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40806d7d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa400009d90 tx=0x7fa400004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 shutdown_connections 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3e006b2c0 0x7fa3e006d780 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa40815d770 0x7fa40806d7d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa408143810 0x7fa40806d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 --2- 172.21.15.141:0/3798161155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408071e40 0x7fa4081f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 >> 172.21.15.141:0/3798161155 conn(0x7fa40806a7d0 msgr2=0x7fa408071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 shutdown_connections 2024-01-16T14:57:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.307+0000 7fa407e00640 1 -- 172.21.15.141:0/3798161155 wait complete. 2024-01-16T14:57:21.306 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.536+0000 7fdc7e400640 1 Processor -- start 2024-01-16T14:57:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.536+0000 7fdc7e400640 1 -- start start 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc7e400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdc780cf9a0 0x7fdc7815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc7e400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdc7815e1e0 0x7fdc780702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc7e400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc780b1ee0 con 0x7fdc78143ae0 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc7e400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc780b2780 con 0x7fdc7815e1e0 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc7e400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7815e990 con 0x7fdc780cf9a0 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc77400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdc78143ae0 0x7fdc780cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40160/0 (socket says 172.21.15.141:40160) 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc77400640 1 -- 172.21.15.141:0/1838012230 learned_addr learned my addr 172.21.15.141:0/1838012230 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.538+0000 7fdc77e00640 1 --2- 172.21.15.141:0/1838012230 >> v2:172.21.15.141:3300/0 conn(0x7fdc7815e1e0 0x7fdc780702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.539+0000 7fdc6e800640 1 --2- 172.21.15.141:0/1838012230 >> v2:172.21.15.188:3300/0 conn(0x7fdc780cf9a0 0x7fdc7815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.539+0000 7fdc76a00640 1 -- 172.21.15.141:0/1838012230 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609441157 0 0) 0x7fdc780b1ee0 con 0x7fdc78143ae0 2024-01-16T14:57:21.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.539+0000 7fdc76a00640 1 -- 172.21.15.141:0/1838012230 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc58003520 con 0x7fdc78143ae0 2024-01-16T14:57:21.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.539+0000 7fdc77e00640 1 -- 172.21.15.141:0/1838012230 >> v2:172.21.15.188:3300/0 conn(0x7fdc780cf9a0 msgr2=0x7fdc7815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:21.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.539+0000 7fdc77e00640 1 --2- 172.21.15.141:0/1838012230 >> v2:172.21.15.188:3300/0 conn(0x7fdc780cf9a0 0x7fdc7815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.539+0000 7fdc77e00640 1 -- 172.21.15.141:0/1838012230 >> v1:172.21.15.141:6789/0 conn(0x7fdc78143ae0 legacy=0x7fdc780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.539+0000 7fdc77e00640 1 -- 172.21.15.141:0/1838012230 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc780b2130 con 0x7fdc7815e1e0 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.540+0000 7fdc77e00640 1 --2- 172.21.15.141:0/1838012230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7815e1e0 0x7fdc780702c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdc780cb830 tx=0x7fdc7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ec58077f5f997b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.540+0000 7fdc76a00640 1 -- 172.21.15.141:0/1838012230 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7002cc80 con 0x7fdc7815e1e0 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.540+0000 7fdc76a00640 1 -- 172.21.15.141:0/1838012230 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7002cde0 con 0x7fdc7815e1e0 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.540+0000 7fdc76a00640 1 -- 172.21.15.141:0/1838012230 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70034660 con 0x7fdc7815e1e0 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 -- 172.21.15.141:0/1838012230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7815e1e0 msgr2=0x7fdc780702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 --2- 172.21.15.141:0/1838012230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7815e1e0 0x7fdc780702c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdc780cb830 tx=0x7fdc7002bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 -- 172.21.15.141:0/1838012230 shutdown_connections 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 --2- 172.21.15.141:0/1838012230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7815e1e0 0x7fdc780702c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 --2- 172.21.15.141:0/1838012230 >> v2:172.21.15.188:3300/0 conn(0x7fdc780cf9a0 0x7fdc7815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 -- 172.21.15.141:0/1838012230 >> 172.21.15.141:0/1838012230 conn(0x7fdc7806a800 msgr2=0x7fdc7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 -- 172.21.15.141:0/1838012230 shutdown_connections 2024-01-16T14:57:21.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.541+0000 7fdc7e400640 1 -- 172.21.15.141:0/1838012230 wait complete. 2024-01-16T14:57:21.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.543+0000 7fdc7e400640 1 Processor -- start 2024-01-16T14:57:21.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.543+0000 7fdc7e400640 1 -- start start 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc7e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc780cf9a0 0x7fdc781416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc7e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc78143ae0 0x7fdc781599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc7e400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7815e1e0 0x7fdc780b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc7e400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc78159f20 con 0x7fdc78143ae0 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc7e400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc78140620 con 0x7fdc7815e1e0 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc7e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc78140900 con 0x7fdc780cf9a0 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc6e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc78143ae0 0x7fdc781599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc6e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc78143ae0 0x7fdc781599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48882/0 (socket says 172.21.15.141:48882) 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc6e800640 1 -- 172.21.15.141:0/3539352100 learned_addr learned my addr 172.21.15.141:0/3539352100 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc77400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc780cf9a0 0x7fdc781416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.544+0000 7fdc6e800640 1 -- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc780cf9a0 msgr2=0x7fdc781416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc77e00640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7815e1e0 0x7fdc780b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6e800640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc780cf9a0 0x7fdc781416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6e800640 1 -- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7815e1e0 msgr2=0x7fdc780b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6e800640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7815e1e0 0x7fdc780b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6e800640 1 -- 172.21.15.141:0/3539352100 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70009a70 con 0x7fdc78143ae0 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc77400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc780cf9a0 0x7fdc781416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc77e00640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7815e1e0 0x7fdc780b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6e800640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc78143ae0 0x7fdc781599e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000d5f0 tx=0x7fdc6000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6fe00640 1 -- 172.21.15.141:0/3539352100 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc600125f0 con 0x7fdc78143ae0 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6fe00640 1 -- 172.21.15.141:0/3539352100 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc60012750 con 0x7fdc78143ae0 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc6fe00640 1 -- 172.21.15.141:0/3539352100 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60012b00 con 0x7fdc78143ae0 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc78140be0 con 0x7fdc78143ae0 2024-01-16T14:57:21.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.545+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc781411a0 con 0x7fdc78143ae0 2024-01-16T14:57:21.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.547+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc78141330 con 0x7fdc78143ae0 2024-01-16T14:57:21.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.547+0000 7fdc6fe00640 1 -- 172.21.15.141:0/3539352100 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdc60012c60 con 0x7fdc78143ae0 2024-01-16T14:57:21.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.548+0000 7fdc6fe00640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5006b2c0 0x7fdc5006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.548+0000 7fdc6fe00640 1 -- 172.21.15.141:0/3539352100 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc6008a0b0 con 0x7fdc78143ae0 2024-01-16T14:57:21.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.549+0000 7fdc77400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5006b2c0 0x7fdc5006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.550+0000 7fdc77400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5006b2c0 0x7fdc5006d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdc68007fd0 tx=0x7fdc68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.552+0000 7fdc6fe00640 1 -- 172.21.15.141:0/3539352100 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc60054f10 con 0x7fdc78143ae0 2024-01-16T14:57:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.737+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdc780675d0 con 0x7fdc78143ae0 2024-01-16T14:57:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.738+0000 7fdc6fe00640 1 -- 172.21.15.141:0/3539352100 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdc60058530 con 0x7fdc78143ae0 2024-01-16T14:57:21.720 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:21.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5006b2c0 msgr2=0x7fdc5006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5006b2c0 0x7fdc5006d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdc68007fd0 tx=0x7fdc68009040 comp rx=0 tx=0).stop 2024-01-16T14:57:21.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc78143ae0 msgr2=0x7fdc781599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc78143ae0 0x7fdc781599e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000d5f0 tx=0x7fdc6000dab0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 shutdown_connections 2024-01-16T14:57:21.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5006b2c0 0x7fdc5006d780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7815e1e0 0x7fdc780b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc78143ae0 0x7fdc781599e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 --2- 172.21.15.141:0/3539352100 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc780cf9a0 0x7fdc781416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 >> 172.21.15.141:0/3539352100 conn(0x7fdc7806a800 msgr2=0x7fdc780b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 shutdown_connections 2024-01-16T14:57:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.741+0000 7fdc7e400640 1 -- 172.21.15.141:0/3539352100 wait complete. 2024-01-16T14:57:21.739 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:21.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.958+0000 7f8423200640 1 Processor -- start 2024-01-16T14:57:21.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.958+0000 7f8423200640 1 -- start start 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8423200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8424143810 0x7f8424071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8423200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84240ae620 con 0x7f842415d770 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8423200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842406c730 con 0x7f84240ae170 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8423200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8424143bf0 con 0x7f8424143810 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8421e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8424143810 0x7f8424071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8422800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f842415d770 0x7f842415dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40168/0 (socket says 172.21.15.141:40168) 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8421e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8424143810 0x7f8424071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48892/0 (socket says 172.21.15.141:48892) 2024-01-16T14:57:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.960+0000 7f8422800640 1 -- 172.21.15.141:0/2271544121 learned_addr learned my addr 172.21.15.141:0/2271544121 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:21.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.961+0000 7f8421e00640 1 -- 172.21.15.141:0/2271544121 >> v1:172.21.15.188:6789/0 conn(0x7f84240ae170 legacy=0x7f842415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.961+0000 7f8421e00640 1 -- 172.21.15.141:0/2271544121 >> v1:172.21.15.141:6789/0 conn(0x7f842415d770 legacy=0x7f842415dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.961+0000 7f8421e00640 1 -- 172.21.15.141:0/2271544121 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84240724d0 con 0x7f8424143810 2024-01-16T14:57:21.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.961+0000 7f8421400640 1 -- 172.21.15.141:0/2271544121 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434647008 0 0) 0x7f84240ae620 con 0x7f842415d770 2024-01-16T14:57:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.963+0000 7f8421e00640 1 --2- 172.21.15.141:0/2271544121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8424143810 0x7f8424071e40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84140097a0 tx=0x7f841402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac66e3887c48c03 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.963+0000 7f8421400640 1 -- 172.21.15.141:0/2271544121 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841402e070 con 0x7f8424143810 2024-01-16T14:57:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.963+0000 7f8421400640 1 -- 172.21.15.141:0/2271544121 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84140315a0 con 0x7f8424143810 2024-01-16T14:57:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.963+0000 7f8421400640 1 -- 172.21.15.141:0/2271544121 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414031930 con 0x7f8424143810 2024-01-16T14:57:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.963+0000 7f8423200640 1 -- 172.21.15.141:0/2271544121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8424143810 msgr2=0x7f8424071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.963+0000 7f8423200640 1 --2- 172.21.15.141:0/2271544121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8424143810 0x7f8424071e40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84140097a0 tx=0x7f841402b840 comp rx=0 tx=0).stop 2024-01-16T14:57:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.964+0000 7f8423200640 1 -- 172.21.15.141:0/2271544121 shutdown_connections 2024-01-16T14:57:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.964+0000 7f8423200640 1 --2- 172.21.15.141:0/2271544121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8424143810 0x7f8424071e40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.964+0000 7f8423200640 1 -- 172.21.15.141:0/2271544121 >> 172.21.15.141:0/2271544121 conn(0x7f842406a7d0 msgr2=0x7f842406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.964+0000 7f8423200640 1 -- 172.21.15.141:0/2271544121 shutdown_connections 2024-01-16T14:57:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.964+0000 7f8423200640 1 -- 172.21.15.141:0/2271544121 wait complete. 2024-01-16T14:57:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.965+0000 7f8423200640 1 Processor -- start 2024-01-16T14:57:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.965+0000 7f8423200640 1 -- start start 2024-01-16T14:57:21.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.966+0000 7f8423200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84240ae170 0x7f842415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.966+0000 7f8423200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8424143810 0x7f8424143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.966+0000 7f8423200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842415d770 0x7f8424146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.966+0000 7f8423200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84240afdf0 con 0x7f842415d770 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.966+0000 7f8423200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84240b00d0 con 0x7f8424143810 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.966+0000 7f8423200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84241447e0 con 0x7f84240ae170 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842415d770 0x7f8424146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842415d770 0x7f8424146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48904/0 (socket says 172.21.15.141:48904) 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 -- 172.21.15.141:0/224303674 learned_addr learned my addr 172.21.15.141:0/224303674 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 -- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84240ae170 msgr2=0x7f842415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8421e00640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84240ae170 0x7f842415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f841be00640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8424143810 0x7f8424143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84240ae170 0x7f842415ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 -- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8424143810 msgr2=0x7f8424143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8424143810 0x7f8424143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 -- 172.21.15.141:0/224303674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8414009400 con 0x7f842415d770 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8421e00640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84240ae170 0x7f842415ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f841be00640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8424143810 0x7f8424143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.967+0000 7f8422800640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842415d770 0x7f8424146420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f841c00c7b0 tx=0x7f841c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.968+0000 7f841a000640 1 -- 172.21.15.141:0/224303674 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c00a640 con 0x7f842415d770 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.968+0000 7f841a000640 1 -- 172.21.15.141:0/224303674 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c00a7c0 con 0x7f842415d770 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.968+0000 7f841a000640 1 -- 172.21.15.141:0/224303674 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c00ab70 con 0x7f842415d770 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.968+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8424144ac0 con 0x7f842415d770 2024-01-16T14:57:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.968+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842415d3f0 con 0x7f842415d770 2024-01-16T14:57:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.969+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8424144fe0 con 0x7f842415d770 2024-01-16T14:57:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.970+0000 7f841a000640 1 -- 172.21.15.141:0/224303674 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f841c00acd0 con 0x7f842415d770 2024-01-16T14:57:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.971+0000 7f841a000640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f83fc06b2f0 0x7f83fc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.971+0000 7f841a000640 1 -- 172.21.15.141:0/224303674 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f841c088f30 con 0x7f842415d770 2024-01-16T14:57:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.971+0000 7f8421e00640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f83fc06b2f0 0x7f83fc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:21.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.972+0000 7f8421e00640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f83fc06b2f0 0x7f83fc06d7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8414036000 tx=0x7f841403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:21.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:21.975+0000 7f841a000640 1 -- 172.21.15.141:0/224303674 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f841c053e30 con 0x7f842415d770 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.163+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8424067450 con 0x7f842415d770 2024-01-16T14:57:22.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.164+0000 7f841a000640 1 -- 172.21.15.141:0/224303674 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f841c018020 con 0x7f842415d770 2024-01-16T14:57:22.146 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.166+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f83fc06b2f0 msgr2=0x7f83fc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.166+0000 7f8423200640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f83fc06b2f0 0x7f83fc06d7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8414036000 tx=0x7f841403d040 comp rx=0 tx=0).stop 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842415d770 msgr2=0x7f8424146420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842415d770 0x7f8424146420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f841c00c7b0 tx=0x7f841c00cc70 comp rx=0 tx=0).stop 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 shutdown_connections 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f83fc06b2f0 0x7f83fc06d7b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842415d770 0x7f8424146420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8424143810 0x7f8424143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 --2- 172.21.15.141:0/224303674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84240ae170 0x7f842415ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 >> 172.21.15.141:0/224303674 conn(0x7f842406a7d0 msgr2=0x7f8424142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 shutdown_connections 2024-01-16T14:57:22.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.167+0000 7f8423200640 1 -- 172.21.15.141:0/224303674 wait complete. 2024-01-16T14:57:22.167 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:22.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.394+0000 7f5e07e00640 1 Processor -- start 2024-01-16T14:57:22.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.394+0000 7f5e07e00640 1 -- start start 2024-01-16T14:57:22.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.395+0000 7f5e07e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5e0013df80 0x7f5e0015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e07e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0006c470 con 0x7f5e0013e360 2024-01-16T14:57:22.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e07e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0006c750 con 0x7f5e000cf9a0 2024-01-16T14:57:22.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e07e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0006ca30 con 0x7f5e0013df80 2024-01-16T14:57:22.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e05400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5e0013df80 0x7f5e0015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e04a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5e0013e360 0x7f5e000cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40176/0 (socket says 172.21.15.141:40176) 2024-01-16T14:57:22.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e04a00640 1 -- 172.21.15.141:0/3795154573 learned_addr learned my addr 172.21.15.141:0/3795154573 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:22.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e05400640 1 -- 172.21.15.141:0/3795154573 >> v1:172.21.15.188:6789/0 conn(0x7f5e000cf9a0 legacy=0x7f5e000cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e05400640 1 -- 172.21.15.141:0/3795154573 >> v1:172.21.15.141:6789/0 conn(0x7f5e0013e360 legacy=0x7f5e000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5e05400640 1 -- 172.21.15.141:0/3795154573 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0006cd10 con 0x7f5e0013df80 2024-01-16T14:57:22.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.396+0000 7f5dffe00640 1 -- 172.21.15.141:0/3795154573 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176287217 0 0) 0x7f5e0006c470 con 0x7f5e0013e360 2024-01-16T14:57:22.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.398+0000 7f5e05400640 1 --2- 172.21.15.141:0/3795154573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013df80 0x7f5e0015bee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5df00097a0 tx=0x7f5df002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05b7ad96a51e386 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.398+0000 7f5dffe00640 1 -- 172.21.15.141:0/3795154573 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df0031830 con 0x7f5e0013df80 2024-01-16T14:57:22.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.398+0000 7f5dffe00640 1 -- 172.21.15.141:0/3795154573 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df0031990 con 0x7f5e0013df80 2024-01-16T14:57:22.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.398+0000 7f5dffe00640 1 -- 172.21.15.141:0/3795154573 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df0031d20 con 0x7f5e0013df80 2024-01-16T14:57:22.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.399+0000 7f5e07e00640 1 -- 172.21.15.141:0/3795154573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013df80 msgr2=0x7f5e0015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.399+0000 7f5e07e00640 1 --2- 172.21.15.141:0/3795154573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013df80 0x7f5e0015bee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5df00097a0 tx=0x7f5df002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.399+0000 7f5e07e00640 1 -- 172.21.15.141:0/3795154573 shutdown_connections 2024-01-16T14:57:22.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.399+0000 7f5e07e00640 1 --2- 172.21.15.141:0/3795154573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013df80 0x7f5e0015bee0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.399+0000 7f5e07e00640 1 -- 172.21.15.141:0/3795154573 >> 172.21.15.141:0/3795154573 conn(0x7f5e0006a800 msgr2=0x7f5e0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.399+0000 7f5e07e00640 1 -- 172.21.15.141:0/3795154573 shutdown_connections 2024-01-16T14:57:22.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.399+0000 7f5e07e00640 1 -- 172.21.15.141:0/3795154573 wait complete. 2024-01-16T14:57:22.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.401+0000 7f5e07e00640 1 Processor -- start 2024-01-16T14:57:22.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.401+0000 7f5e07e00640 1 -- start start 2024-01-16T14:57:22.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.402+0000 7f5e07e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e000cf9a0 0x7f5e0013c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.402+0000 7f5e07e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e0013df80 0x7f5e0013ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.402+0000 7f5e07e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013e360 0x7f5e00146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.402+0000 7f5e07e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00144e10 con 0x7f5e0013e360 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.402+0000 7f5e07e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00145970 con 0x7f5e000cf9a0 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.402+0000 7f5e07e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00145040 con 0x7f5e0013df80 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.402+0000 7f5e05e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013e360 0x7f5e00146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013e360 0x7f5e00146ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48918/0 (socket says 172.21.15.141:48918) 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 -- 172.21.15.141:0/1813238311 learned_addr learned my addr 172.21.15.141:0/1813238311 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e04a00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e0013df80 0x7f5e0013ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05400640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e000cf9a0 0x7f5e0013c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 -- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e0013df80 msgr2=0x7f5e0013ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e0013df80 0x7f5e0013ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 -- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e000cf9a0 msgr2=0x7f5e0013c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e000cf9a0 0x7f5e0013c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 -- 172.21.15.141:0/1813238311 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df0009400 con 0x7f5e0013e360 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5e05e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013e360 0x7f5e00146ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5df800b9e0 tx=0x7f5df800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.403+0000 7f5dfd600640 1 -- 172.21.15.141:0/1813238311 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8003b70 con 0x7f5e0013e360 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.404+0000 7f5dfd600640 1 -- 172.21.15.141:0/1813238311 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8003cd0 con 0x7f5e0013e360 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.404+0000 7f5dfd600640 1 -- 172.21.15.141:0/1813238311 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df800cb60 con 0x7f5e0013e360 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.404+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e00145320 con 0x7f5e0013e360 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.404+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e00139fd0 con 0x7f5e0013e360 2024-01-16T14:57:22.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.405+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e00145b20 con 0x7f5e0013e360 2024-01-16T14:57:22.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.406+0000 7f5dfd600640 1 -- 172.21.15.141:0/1813238311 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5df800ccc0 con 0x7f5e0013e360 2024-01-16T14:57:22.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.407+0000 7f5dfd600640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806b2c0 0x7f5dd806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.407+0000 7f5dfd600640 1 -- 172.21.15.141:0/1813238311 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5df8089970 con 0x7f5e0013e360 2024-01-16T14:57:22.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.407+0000 7f5e05400640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806b2c0 0x7f5dd806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.408+0000 7f5e05400640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806b2c0 0x7f5dd806d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5df0036000 tx=0x7f5df003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.415+0000 7f5dfd600640 1 -- 172.21.15.141:0/1813238311 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e00145b20 con 0x7f5e0013e360 2024-01-16T14:57:22.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.645+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e00145ae0 con 0x7f5e0013e360 2024-01-16T14:57:22.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.646+0000 7f5dfd600640 1 -- 172.21.15.141:0/1813238311 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f5df8054740 con 0x7f5e0013e360 2024-01-16T14:57:22.628 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:22.628 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:22.628 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806b2c0 msgr2=0x7f5dd806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806b2c0 0x7f5dd806d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5df0036000 tx=0x7f5df003c040 comp rx=0 tx=0).stop 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013e360 msgr2=0x7f5e00146ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013e360 0x7f5e00146ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5df800b9e0 tx=0x7f5df800bea0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 shutdown_connections 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806b2c0 0x7f5dd806d780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e0013e360 0x7f5e00146ea0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e0013df80 0x7f5e0013ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 --2- 172.21.15.141:0/1813238311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e000cf9a0 0x7f5e0013c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 >> 172.21.15.141:0/1813238311 conn(0x7f5e0006a800 msgr2=0x7f5e0015d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 shutdown_connections 2024-01-16T14:57:22.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.649+0000 7f5e07e00640 1 -- 172.21.15.141:0/1813238311 wait complete. 2024-01-16T14:57:22.649 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:22.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.870+0000 7f1337200640 1 Processor -- start 2024-01-16T14:57:22.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.870+0000 7f1337200640 1 -- start start 2024-01-16T14:57:22.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1337200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 0x7f1330146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1337200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1330071f00 0x7f1330166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1337200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13300b2e10 con 0x7f133006c7c0 2024-01-16T14:57:22.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1337200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133006cc20 con 0x7f1330071f00 2024-01-16T14:57:22.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1337200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133006cec0 con 0x7f1330071410 2024-01-16T14:57:22.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1335400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f133006c7c0 0x7f13300717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40182/0 (socket says 172.21.15.141:40182) 2024-01-16T14:57:22.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1335400640 1 -- 172.21.15.141:0/221959378 learned_addr learned my addr 172.21.15.141:0/221959378 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:22.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1335e00640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 0x7f1330146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1336800640 1 --2- 172.21.15.141:0/221959378 >> v2:172.21.15.188:3300/0 conn(0x7f1330071f00 0x7f1330166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818010996 0 0) 0x7f13300b2e10 con 0x7f133006c7c0 2024-01-16T14:57:22.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.872+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1310003520 con 0x7f133006c7c0 2024-01-16T14:57:22.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3829164737 0 0) 0x7f1310003520 con 0x7f133006c7c0 2024-01-16T14:57:22.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 msgr2=0x7f1330146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 0x7f1330146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 >> v2:172.21.15.188:3300/0 conn(0x7f1330071f00 msgr2=0x7f1330166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 --2- 172.21.15.141:0/221959378 >> v2:172.21.15.188:3300/0 conn(0x7f1330071f00 0x7f1330166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133006d1a0 con 0x7f133006c7c0 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1318001670 con 0x7f133006c7c0 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1336800640 1 --2- 172.21.15.141:0/221959378 >> v2:172.21.15.188:3300/0 conn(0x7f1330071f00 0x7f1330166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 >> v1:172.21.15.141:6789/0 conn(0x7f133006c7c0 legacy=0x7f13300717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 0x7f1310003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1310003fa0 0x7f13100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1310004d00 0x7f1310005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133006cc20 con 0x7f1310003fa0 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133006cec0 con 0x7f1310004d00 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131000ba80 con 0x7f1330071410 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1335400640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1310003fa0 0x7f13100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1335e00640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 0x7f1310003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1335400640 1 -- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 msgr2=0x7f1310003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1336800640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1310004d00 0x7f1310005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1335400640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1330071410 0x7f1310003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1335400640 1 -- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1310004d00 msgr2=0x7f1310005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1335400640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1310004d00 0x7f1310005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.873+0000 7f1335400640 1 -- 172.21.15.141:0/221959378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1310003520 con 0x7f1310003fa0 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.874+0000 7f1335400640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1310003fa0 0x7f13100047c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f131800c290 tx=0x7f13100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5549fa53027c3313 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318005ad0 con 0x7f1310003fa0 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1318005c30 con 0x7f1310003fa0 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1334a00640 1 -- 172.21.15.141:0/221959378 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131802c8d0 con 0x7f1310003fa0 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1337200640 1 -- 172.21.15.141:0/221959378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1310003fa0 msgr2=0x7f13100047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1337200640 1 --2- 172.21.15.141:0/221959378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1310003fa0 0x7f13100047c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f131800c290 tx=0x7f13100057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1336800640 1 -- 172.21.15.141:0/221959378 reap_dead start 2024-01-16T14:57:22.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1337200640 1 -- 172.21.15.141:0/221959378 shutdown_connections 2024-01-16T14:57:22.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1337200640 1 -- 172.21.15.141:0/221959378 >> 172.21.15.141:0/221959378 conn(0x7f133006a800 msgr2=0x7f133006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:22.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1337200640 1 -- 172.21.15.141:0/221959378 shutdown_connections 2024-01-16T14:57:22.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.875+0000 7f1337200640 1 -- 172.21.15.141:0/221959378 wait complete. 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.877+0000 7f1337200640 1 Processor -- start 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.877+0000 7f1337200640 1 -- start start 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1337200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1310003fa0 0x7f133015c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1337200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13301e4190 0x7f133015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1337200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133013f4b0 0x7f133013f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1337200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133013fea0 con 0x7f133013f4b0 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1337200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13301e28f0 con 0x7f13301e4190 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1337200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13301e4700 con 0x7f1310003fa0 2024-01-16T14:57:22.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1336800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133013f4b0 0x7f133013f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1336800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133013f4b0 0x7f133013f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48944/0 (socket says 172.21.15.141:48944) 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.878+0000 7f1336800640 1 -- 172.21.15.141:0/2806999577 learned_addr learned my addr 172.21.15.141:0/2806999577 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1335400640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13301e4190 0x7f133015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1335e00640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1310003fa0 0x7f133015c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1336800640 1 -- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1310003fa0 msgr2=0x7f133015c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1336800640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1310003fa0 0x7f133015c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1336800640 1 -- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13301e4190 msgr2=0x7f133015bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1336800640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13301e4190 0x7f133015bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1336800640 1 -- 172.21.15.141:0/2806999577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13301e49e0 con 0x7f133013f4b0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1336800640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133013f4b0 0x7f133013f960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f13300adf40 tx=0x7f1328004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f132e000640 1 -- 172.21.15.141:0/2806999577 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328007cb0 con 0x7f133013f4b0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f132e000640 1 -- 172.21.15.141:0/2806999577 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1328007e50 con 0x7f133013f4b0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f132e000640 1 -- 172.21.15.141:0/2806999577 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328014640 con 0x7f133013f4b0 2024-01-16T14:57:22.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.879+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13301e2580 con 0x7f133013f4b0 2024-01-16T14:57:22.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.880+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133015c140 con 0x7f133013f4b0 2024-01-16T14:57:22.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.881+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1330067450 con 0x7f133013f4b0 2024-01-16T14:57:22.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.882+0000 7f132e000640 1 -- 172.21.15.141:0/2806999577 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f13280147a0 con 0x7f133013f4b0 2024-01-16T14:57:22.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.883+0000 7f132e000640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f130806b2f0 0x7f130806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:22.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.883+0000 7f132e000640 1 -- 172.21.15.141:0/2806999577 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13280898a0 con 0x7f133013f4b0 2024-01-16T14:57:22.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.883+0000 7f1335e00640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f130806b2f0 0x7f130806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:22.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.884+0000 7f1335e00640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f130806b2f0 0x7f130806d7b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f133013f440 tx=0x7f1320008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:22.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:22.887+0000 7f132e000640 1 -- 172.21.15.141:0/2806999577 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1328058240 con 0x7f133013f4b0 2024-01-16T14:57:23.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.076+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13301e2a30 con 0x7f133013f4b0 2024-01-16T14:57:23.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.079+0000 7f132e000640 1 -- 172.21.15.141:0/2806999577 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f1328057dd0 con 0x7f133013f4b0 2024-01-16T14:57:23.061 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:23.061 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f130806b2f0 msgr2=0x7f130806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f130806b2f0 0x7f130806d7b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f133013f440 tx=0x7f1320008040 comp rx=0 tx=0).stop 2024-01-16T14:57:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133013f4b0 msgr2=0x7f133013f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133013f4b0 0x7f133013f960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f13300adf40 tx=0x7f1328004840 comp rx=0 tx=0).stop 2024-01-16T14:57:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 shutdown_connections 2024-01-16T14:57:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f130806b2f0 0x7f130806d7b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133013f4b0 0x7f133013f960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13301e4190 0x7f133015bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 --2- 172.21.15.141:0/2806999577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1310003fa0 0x7f133015c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 >> 172.21.15.141:0/2806999577 conn(0x7f133006a800 msgr2=0x7f13300b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 shutdown_connections 2024-01-16T14:57:23.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.082+0000 7f1337200640 1 -- 172.21.15.141:0/2806999577 wait complete. 2024-01-16T14:57:23.083 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:57:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.308+0000 7fce0f200640 1 Processor -- start 2024-01-16T14:57:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.308+0000 7fce0f200640 1 -- start start 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce0f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fce0813df80 0x7fce0815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce0f200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0806c470 con 0x7fce0813e360 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce0f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0806c750 con 0x7fce080cf9a0 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce0f200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0806ca30 con 0x7fce0813df80 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce07e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fce0813df80 0x7fce0815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce07400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fce0813e360 0x7fce080cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38624/0 (socket says 172.21.15.141:38624) 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce07e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fce0813df80 0x7fce0815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49852/0 (socket says 172.21.15.141:49852) 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce07400640 1 -- 172.21.15.141:0/1280002355 learned_addr learned my addr 172.21.15.141:0/1280002355 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce07e00640 1 -- 172.21.15.141:0/1280002355 >> v1:172.21.15.188:6789/0 conn(0x7fce080cf9a0 legacy=0x7fce080cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce07e00640 1 -- 172.21.15.141:0/1280002355 >> v1:172.21.15.141:6789/0 conn(0x7fce0813e360 legacy=0x7fce080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.310+0000 7fce07e00640 1 -- 172.21.15.141:0/1280002355 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0806cd10 con 0x7fce0813df80 2024-01-16T14:57:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.312+0000 7fce07e00640 1 --2- 172.21.15.141:0/1280002355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813df80 0x7fce0815bee0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80097a0 tx=0x7fcdf802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0b61b5444da08d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.312+0000 7fce06a00640 1 -- 172.21.15.141:0/1280002355 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8031830 con 0x7fce0813df80 2024-01-16T14:57:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.312+0000 7fce06a00640 1 -- 172.21.15.141:0/1280002355 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8031990 con 0x7fce0813df80 2024-01-16T14:57:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.312+0000 7fce06a00640 1 -- 172.21.15.141:0/1280002355 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8031d20 con 0x7fce0813df80 2024-01-16T14:57:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.312+0000 7fce0f200640 1 -- 172.21.15.141:0/1280002355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813df80 msgr2=0x7fce0815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.312+0000 7fce0f200640 1 --2- 172.21.15.141:0/1280002355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813df80 0x7fce0815bee0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80097a0 tx=0x7fcdf802b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.313+0000 7fce0f200640 1 -- 172.21.15.141:0/1280002355 shutdown_connections 2024-01-16T14:57:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.313+0000 7fce0f200640 1 --2- 172.21.15.141:0/1280002355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813df80 0x7fce0815bee0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.313+0000 7fce0f200640 1 -- 172.21.15.141:0/1280002355 >> 172.21.15.141:0/1280002355 conn(0x7fce0806a800 msgr2=0x7fce0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.313+0000 7fce0f200640 1 -- 172.21.15.141:0/1280002355 shutdown_connections 2024-01-16T14:57:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.313+0000 7fce0f200640 1 -- 172.21.15.141:0/1280002355 wait complete. 2024-01-16T14:57:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.314+0000 7fce0f200640 1 Processor -- start 2024-01-16T14:57:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.315+0000 7fce0f200640 1 -- start start 2024-01-16T14:57:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce080cf9a0 0x7fce081f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce0813df80 0x7fce0815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813e360 0x7fce080ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce081f3dc0 con 0x7fce0813e360 2024-01-16T14:57:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce08159910 con 0x7fce080cf9a0 2024-01-16T14:57:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce081f3800 con 0x7fce0813df80 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813e360 0x7fce080ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813e360 0x7fce080ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49866/0 (socket says 172.21.15.141:49866) 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 -- 172.21.15.141:0/749398053 learned_addr learned my addr 172.21.15.141:0/749398053 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce07e00640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce080cf9a0 0x7fce081f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce07400640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce0813df80 0x7fce0815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 -- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce0813df80 msgr2=0x7fce0815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce0813df80 0x7fce0815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 -- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce080cf9a0 msgr2=0x7fce081f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce080cf9a0 0x7fce081f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce0d200640 1 -- 172.21.15.141:0/749398053 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8009400 con 0x7fce0813e360 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.316+0000 7fce07400640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce0813df80 0x7fce0815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.317+0000 7fce0d200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813e360 0x7fce080ac8d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b9e0 tx=0x7fce0000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.317+0000 7fcdffe00640 1 -- 172.21.15.141:0/749398053 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00003b70 con 0x7fce0813e360 2024-01-16T14:57:23.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.317+0000 7fcdffe00640 1 -- 172.21.15.141:0/749398053 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00003cd0 con 0x7fce0813e360 2024-01-16T14:57:23.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.317+0000 7fcdffe00640 1 -- 172.21.15.141:0/749398053 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000cb60 con 0x7fce0813e360 2024-01-16T14:57:23.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.317+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce081f3ae0 con 0x7fce0813e360 2024-01-16T14:57:23.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.317+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce08159e70 con 0x7fce0813e360 2024-01-16T14:57:23.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.318+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0815a090 con 0x7fce0813e360 2024-01-16T14:57:23.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.319+0000 7fcdffe00640 1 -- 172.21.15.141:0/749398053 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fce0000ccc0 con 0x7fce0813e360 2024-01-16T14:57:23.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.320+0000 7fcdffe00640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcde006b2c0 0x7fcde006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:23.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.320+0000 7fcdffe00640 1 -- 172.21.15.141:0/749398053 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce000899d0 con 0x7fce0813e360 2024-01-16T14:57:23.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.321+0000 7fce07e00640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcde006b2c0 0x7fcde006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:23.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.322+0000 7fce07e00640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcde006b2c0 0x7fcde006d780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8036000 tx=0x7fcdf803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:23.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.324+0000 7fcdffe00640 1 -- 172.21.15.141:0/749398053 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce000547a0 con 0x7fce0813e360 2024-01-16T14:57:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.558+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce080675d0 con 0x7fce0813e360 2024-01-16T14:57:23.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.559+0000 7fcdffe00640 1 -- 172.21.15.141:0/749398053 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fce00057dc0 con 0x7fce0813e360 2024-01-16T14:57:23.541 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:23.541 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:57:23.541 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:57:23.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcde006b2c0 msgr2=0x7fcde006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcde006b2c0 0x7fcde006d780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8036000 tx=0x7fcdf803c040 comp rx=0 tx=0).stop 2024-01-16T14:57:23.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813e360 msgr2=0x7fce080ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:23.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813e360 0x7fce080ac8d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b9e0 tx=0x7fce0000bea0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 shutdown_connections 2024-01-16T14:57:23.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcde006b2c0 0x7fcde006d780 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce0813e360 0x7fce080ac8d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce0813df80 0x7fce0815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 --2- 172.21.15.141:0/749398053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce080cf9a0 0x7fce081f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.562+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 >> 172.21.15.141:0/749398053 conn(0x7fce0806a800 msgr2=0x7fce0815c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.563+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 shutdown_connections 2024-01-16T14:57:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:23.563+0000 7fce0f200640 1 -- 172.21.15.141:0/749398053 wait complete. 2024-01-16T14:57:23.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T14:57:06.988377+0000', 'modified': '2024-01-16T14:57:10.018720+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4249}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4249': {'gid': 4249, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6833/114446300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6832', 'nonce': 114446300}, {'type': 'v1', 'addr': '172.21.15.141:6833', 'nonce': 114446300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T14:57:23.966 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T14:57:23.966 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T14:57:24.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.200+0000 7f34f7800640 1 Processor -- start 2024-01-16T14:57:24.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.200+0000 7f34f7800640 1 -- start start 2024-01-16T14:57:24.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.201+0000 7f34f7800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f013df80 0x7f34f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.201+0000 7f34f7800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f34f013e360 0x7f34f00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.201+0000 7f34f7800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f006c470 con 0x7f34f015f9c0 2024-01-16T14:57:24.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.201+0000 7f34f7800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f006c750 con 0x7f34f013e360 2024-01-16T14:57:24.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.201+0000 7f34f7800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f006ca30 con 0x7f34f013df80 2024-01-16T14:57:24.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.202+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f013df80 0x7f34f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.202+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f013df80 0x7f34f015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:37080/0 (socket says 172.21.15.141:37080) 2024-01-16T14:57:24.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.202+0000 7f34f4e00640 1 -- 172.21.15.141:0/2419801553 learned_addr learned my addr 172.21.15.141:0/2419801553 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.202+0000 7f34efe00640 1 --2- 172.21.15.141:0/2419801553 >> v2:172.21.15.188:3300/0 conn(0x7f34f013e360 0x7f34f00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.203+0000 7f34ef400640 1 -- 172.21.15.141:0/2419801553 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466276168 0 0) 0x7f34f006c470 con 0x7f34f015f9c0 2024-01-16T14:57:24.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.203+0000 7f34ef400640 1 -- 172.21.15.141:0/2419801553 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d0003520 con 0x7f34f015f9c0 2024-01-16T14:57:24.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.203+0000 7f34f4e00640 1 -- 172.21.15.141:0/2419801553 >> v2:172.21.15.188:3300/0 conn(0x7f34f013e360 msgr2=0x7f34f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.203+0000 7f34f4e00640 1 --2- 172.21.15.141:0/2419801553 >> v2:172.21.15.188:3300/0 conn(0x7f34f013e360 0x7f34f00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.203+0000 7f34f4e00640 1 -- 172.21.15.141:0/2419801553 >> v1:172.21.15.188:6789/0 conn(0x7f34f015f9c0 legacy=0x7f34f01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.203+0000 7f34f4e00640 1 -- 172.21.15.141:0/2419801553 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f006cd10 con 0x7f34f013df80 2024-01-16T14:57:24.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.205+0000 7f34f4e00640 1 --2- 172.21.15.141:0/2419801553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f013df80 0x7f34f015bee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009870 tx=0x7f34e002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34a653c654f3842f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.205+0000 7f34ef400640 1 -- 172.21.15.141:0/2419801553 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0030840 con 0x7f34f013df80 2024-01-16T14:57:24.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34ef400640 1 -- 172.21.15.141:0/2419801553 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e00309a0 con 0x7f34f013df80 2024-01-16T14:57:24.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34ef400640 1 -- 172.21.15.141:0/2419801553 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0030d30 con 0x7f34f013df80 2024-01-16T14:57:24.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 -- 172.21.15.141:0/2419801553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f013df80 msgr2=0x7f34f015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 --2- 172.21.15.141:0/2419801553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f013df80 0x7f34f015bee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009870 tx=0x7f34e002b980 comp rx=0 tx=0).stop 2024-01-16T14:57:24.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 -- 172.21.15.141:0/2419801553 shutdown_connections 2024-01-16T14:57:24.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 --2- 172.21.15.141:0/2419801553 >> v2:172.21.15.188:3300/0 conn(0x7f34f013e360 0x7f34f00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 --2- 172.21.15.141:0/2419801553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f013df80 0x7f34f015bee0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 -- 172.21.15.141:0/2419801553 >> 172.21.15.141:0/2419801553 conn(0x7f34f006a800 msgr2=0x7f34f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 -- 172.21.15.141:0/2419801553 shutdown_connections 2024-01-16T14:57:24.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.206+0000 7f34f7800640 1 -- 172.21.15.141:0/2419801553 wait complete. 2024-01-16T14:57:24.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.208+0000 7f34f7800640 1 Processor -- start 2024-01-16T14:57:24.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.208+0000 7f34f7800640 1 -- start start 2024-01-16T14:57:24.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f7800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34f013df80 0x7f34f0156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f7800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f013e360 0x7f34f013d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f7800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f015f9c0 0x7f34f013c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f7800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f01556c0 con 0x7f34f013df80 2024-01-16T14:57:24.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f7800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f01559a0 con 0x7f34f013e360 2024-01-16T14:57:24.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f7800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0155c80 con 0x7f34f015f9c0 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34f013df80 0x7f34f0156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34f013df80 0x7f34f0156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49880/0 (socket says 172.21.15.141:49880) 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.209+0000 7f34f4e00640 1 -- 172.21.15.141:0/3942330749 learned_addr learned my addr 172.21.15.141:0/3942330749 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34efe00640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f013e360 0x7f34f013d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f5800640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f015f9c0 0x7f34f013c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f4e00640 1 -- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f015f9c0 msgr2=0x7f34f013c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f4e00640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f015f9c0 0x7f34f013c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f4e00640 1 -- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f013e360 msgr2=0x7f34f013d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f4e00640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f013e360 0x7f34f013d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f4e00640 1 -- 172.21.15.141:0/3942330749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e00094d0 con 0x7f34f013df80 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f4e00640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34f013df80 0x7f34f0156460 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f34e0000c00 tx=0x7f34e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34ecc00640 1 -- 172.21.15.141:0/3942330749 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0039a90 con 0x7f34f013df80 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34ecc00640 1 -- 172.21.15.141:0/3942330749 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0039bf0 con 0x7f34f013df80 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f0155f60 con 0x7f34f013df80 2024-01-16T14:57:24.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.210+0000 7f34ecc00640 1 -- 172.21.15.141:0/3942330749 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0043410 con 0x7f34f013df80 2024-01-16T14:57:24.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.211+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f00b1090 con 0x7f34f013df80 2024-01-16T14:57:24.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.213+0000 7f34ecc00640 1 -- 172.21.15.141:0/3942330749 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f34e002f390 con 0x7f34f013df80 2024-01-16T14:57:24.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.214+0000 7f34ecc00640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f34c806b2c0 0x7f34c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.214+0000 7f34ecc00640 1 -- 172.21.15.141:0/3942330749 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34e00abd80 con 0x7f34f013df80 2024-01-16T14:57:24.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.215+0000 7f34efe00640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f34c806b2c0 0x7f34c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.215+0000 7f34fa3ac740 1 Processor -- start 2024-01-16T14:57:24.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.215+0000 7f34fa3ac740 1 -- start start 2024-01-16T14:57:24.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69ac7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f34c806b2c0 0x7f34c806d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34f013c210 tx=0x7f34d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69b0a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e69a25880 con 0x560e69ac1ba0 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e69a34ae0 con 0x560e69ac6ea0 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e69a321d0 con 0x560e69ac65e0 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49892/0 (socket says 172.21.15.141:49892) 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 -- 172.21.15.141:0/2945789382 learned_addr learned my addr 172.21.15.141:0/2945789382 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34f4e00640 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69ac7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34f5800640 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69b0a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 -- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 msgr2=0x560e69b0a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69b0a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 -- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 msgr2=0x560e69ac7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69ac7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34efe00640 1 -- 172.21.15.141:0/2945789382 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e699f5860 con 0x560e69ac1ba0 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.216+0000 7f34f5800640 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69b0a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.217+0000 7f34efe00640 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac60a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f34d8011dc0 tx=0x7f34d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e49b9a1bb614c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.217+0000 7f34e5600640 1 -- 172.21.15.141:0/2945789382 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8011df0 con 0x560e69ac1ba0 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.217+0000 7f34e5600640 1 -- 172.21.15.141:0/2945789382 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8021440 con 0x560e69ac1ba0 2024-01-16T14:57:24.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.217+0000 7f34e5600640 1 -- 172.21.15.141:0/2945789382 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d80217b0 con 0x560e69ac1ba0 2024-01-16T14:57:24.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 -- 172.21.15.141:0/2945789382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 msgr2=0x560e69ac60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac60a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f34d8011dc0 tx=0x7f34d8014000 comp rx=0 tx=0).stop 2024-01-16T14:57:24.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 -- 172.21.15.141:0/2945789382 shutdown_connections 2024-01-16T14:57:24.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69b0a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac60a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/2945789382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69ac7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 -- 172.21.15.141:0/2945789382 >> 172.21.15.141:0/2945789382 conn(0x560e69a88c70 msgr2=0x560e69b0c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 -- 172.21.15.141:0/2945789382 shutdown_connections 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 -- 172.21.15.141:0/2945789382 wait complete. 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.218+0000 7f34fa3ac740 1 -- start start 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.219+0000 7f34fa3ac740 1 Processor -- start 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac1ba0 0x560e69b13bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac65e0 0x560e69b03a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac6ea0 0x560e69b03f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e69a2dac0 con 0x560e69ac6ea0 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e699ca2f0 con 0x560e69ac65e0 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e699e7c70 con 0x560e69ac1ba0 2024-01-16T14:57:24.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34fa3ac740 10 client.0 _ll_register_callbacks cb 0x560e699ec3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac6ea0 0x560e69b03f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac6ea0 0x560e69b03f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49894/0 (socket says 172.21.15.141:49894) 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 -- 172.21.15.141:0/1274761703 learned_addr learned my addr 172.21.15.141:0/1274761703 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34efe00640 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac65e0 0x560e69b03a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 -- 172.21.15.141:0/1274761703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac1ba0 msgr2=0x560e69b13bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac1ba0 0x560e69b13bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 -- 172.21.15.141:0/1274761703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac65e0 msgr2=0x560e69b03a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac65e0 0x560e69b03a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 -- 172.21.15.141:0/1274761703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d8011a40 con 0x560e69ac6ea0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34f5800640 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac6ea0 0x560e69b03f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f34e800f550 tx=0x7f34e800fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91b087e772c8ed7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.220+0000 7f34de000640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34de000640 1 -- 172.21.15.141:0/1274761703 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e8003b80 con 0x560e69ac6ea0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34de000640 1 -- 172.21.15.141:0/1274761703 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34e8003ce0 con 0x560e69ac6ea0 2024-01-16T14:57:24.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560e69a26aa0 con 0x560e69ac6ea0 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34de000640 1 -- 172.21.15.141:0/1274761703 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34e8003ea0 con 0x560e69ac6ea0 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 10 client.4650 fetch_fsmap learned FSMap version 8 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560e69a2c140 con 0x560e69ac6ea0 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34de000640 1 -- 172.21.15.141:0/1274761703 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560e69a26aa0 con 0x560e69ac6ea0 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34de000640 1 -- 172.21.15.141:0/1274761703 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f34e8012070 con 0x560e69ac6ea0 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 10 client.4650 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 10 client.4650 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 20 client.4650 populate_metadata read hostname 'smithi141' 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b25a00 0x560e69b298b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 4 client.4650 mds_command: new command op to 4249 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.221+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 --> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560e69a267c0 con 0x560e69b25a00 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.222+0000 7f34f4e00640 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b25a00 0x560e69b298b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.222+0000 7f34f4e00640 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b25a00 0x560e69b298b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=197d8af81c67b122 server_cookie=eaaea79cb6454b99 in_seq=0 out_seq=0 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.222+0000 7f34de000640 10 client.4650 ms_handle_connect on v2:172.21.15.141:6832/114446300 2024-01-16T14:57:24.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34de000640 1 -- 172.21.15.141:0/1274761703 <== mds.0 v2:172.21.15.141:6832/114446300 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560e69a267c0 con 0x560e69b25a00 2024-01-16T14:57:24.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34de000640 10 client.4650 handle_command_reply: tid=0 2024-01-16T14:57:24.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 client.4650 shutdown 2024-01-16T14:57:24.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac6ea0 msgr2=0x560e69b03f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac6ea0 0x560e69b03f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f34e800f550 tx=0x7f34e800fa10 comp rx=0 tx=0).stop 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 shutdown_connections 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac6ea0 0x560e69b03f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b25a00 0x560e69b298b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac65e0 0x560e69b03a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1274761703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac1ba0 0x560e69b13bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34de000640 0 client.4650 ms_handle_reset on v2:172.21.15.141:6832/114446300 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 >> 172.21.15.141:0/1274761703 conn(0x560e69a88c70 msgr2=0x560e69ac7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 shutdown_connections 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.223+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1274761703 wait complete. 2024-01-16T14:57:24.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.224+0000 7f34fa3ac740 20 client.4650 trim_cache size 0 max 16384 2024-01-16T14:57:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.240+0000 7f34fa3ac740 1 Processor -- start 2024-01-16T14:57:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.240+0000 7f34fa3ac740 1 -- start start 2024-01-16T14:57:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.241+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69b2d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.241+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69abd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.241+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69b75e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.241+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e69790f20 con 0x560e69ac1ba0 2024-01-16T14:57:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.241+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e69b25f30 con 0x560e69ac6ea0 2024-01-16T14:57:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.241+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e69ac76d0 con 0x560e69ac65e0 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69b2d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69b2d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49906/0 (socket says 172.21.15.141:49906) 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34efe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69abd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 -- 172.21.15.141:0/1586410051 learned_addr learned my addr 172.21.15.141:0/1586410051 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f5800640 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69b75e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 -- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 msgr2=0x560e69abd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69abd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 -- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 msgr2=0x560e69b75e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69b75e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 -- 172.21.15.141:0/1586410051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e699c9b80 con 0x560e69ac1ba0 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34efe00640 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69abd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34f4e00640 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69b2d8d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009870 tx=0x7f34e00aa040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7d99fdc21f528d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.242+0000 7f34e6000640 1 -- 172.21.15.141:0/1586410051 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0092070 con 0x560e69ac1ba0 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34e6000640 1 -- 172.21.15.141:0/1586410051 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e00921d0 con 0x560e69ac1ba0 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34e6000640 1 -- 172.21.15.141:0/1586410051 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0082bb0 con 0x560e69ac1ba0 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1586410051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 msgr2=0x560e69b2d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69b2d8d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009870 tx=0x7f34e00aa040 comp rx=0 tx=0).stop 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1586410051 shutdown_connections 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69b75e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69abd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/1586410051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69b2d8d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1586410051 >> 172.21.15.141:0/1586410051 conn(0x560e69a88c70 msgr2=0x560e69aee900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1586410051 shutdown_connections 2024-01-16T14:57:24.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.243+0000 7f34fa3ac740 1 -- 172.21.15.141:0/1586410051 wait complete. 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.244+0000 7f34fa3ac740 1 -- start start 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.244+0000 7f34fa3ac740 1 Processor -- start 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69ac24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34fa3ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69aed660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e69b72d00 con 0x560e69ac1ba0 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e69b15060 con 0x560e69ac6ea0 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34fa3ac740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e69b0fd50 con 0x560e69ac65e0 2024-01-16T14:57:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34fa3ac740 10 client.0 _ll_register_callbacks cb 0x7f34e8001650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34f4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49912/0 (socket says 172.21.15.141:49912) 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.245+0000 7f34f4e00640 1 -- 172.21.15.141:0/398706468 learned_addr learned my addr 172.21.15.141:0/398706468 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f5800640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69aed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34efe00640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69ac24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f4e00640 1 -- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 msgr2=0x560e69ac24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f4e00640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69ac24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f4e00640 1 -- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 msgr2=0x560e69aed660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f4e00640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69aed660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f4e00640 1 -- 172.21.15.141:0/398706468 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f00b1090 con 0x560e69ac1ba0 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f5800640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69aed660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34f4e00640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac1f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f34e0076db0 tx=0x7f34e8001ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c929edda9ae99ab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34de000640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34de000640 1 -- 172.21.15.141:0/398706468 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0092040 con 0x560e69ac1ba0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34de000640 1 -- 172.21.15.141:0/398706468 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34e0078d10 con 0x560e69ac1ba0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34d8021440 con 0x560e69ac1ba0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34de000640 1 -- 172.21.15.141:0/398706468 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34e007bbe0 con 0x560e69ac1ba0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34fa3ac740 10 client.4656 fetch_fsmap learned FSMap version 8 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.246+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560e699ef170 con 0x560e69ac1ba0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34de000640 1 -- 172.21.15.141:0/398706468 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8021440 con 0x560e69ac1ba0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34de000640 1 -- 172.21.15.141:0/398706468 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f34e0082720 con 0x560e69ac1ba0 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34fa3ac740 10 client.4656 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34fa3ac740 10 client.4656 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34fa3ac740 20 client.4656 populate_metadata read hostname 'smithi141' 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b72f30 0x560e69aedba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34fa3ac740 4 client.4656 mds_command: new command op to 4249 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 --> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560e6972e6c0 con 0x560e69b72f30 2024-01-16T14:57:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.247+0000 7f34efe00640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b72f30 0x560e69aedba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34efe00640 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b72f30 0x560e69aedba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5525987359e289d3 server_cookie=8cceb0a478259907 in_seq=0 out_seq=0 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34de000640 10 client.4656 ms_handle_connect on v2:172.21.15.141:6832/114446300 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34de000640 1 -- 172.21.15.141:0/398706468 <== mds.0 v2:172.21.15.141:6832/114446300 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x560e6972e6c0 con 0x560e69b72f30 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34de000640 10 client.4656 handle_command_reply: tid=0 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 client.4656 shutdown 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 msgr2=0x560e69ac1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac1f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f34e0076db0 tx=0x7f34e8001ea0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 shutdown_connections 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:6832/114446300,v1:172.21.15.141:6833/114446300] conn(0x560e69b72f30 0x560e69aedba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e69ac6ea0 0x560e69aed660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x560e69ac65e0 0x560e69ac24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 --2- 172.21.15.141:0/398706468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x560e69ac1ba0 0x560e69ac1f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 >> 172.21.15.141:0/398706468 conn(0x560e69a88c70 msgr2=0x560e69b13090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34de000640 0 client.4656 ms_handle_reset on v2:172.21.15.141:6832/114446300 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 shutdown_connections 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 1 -- 172.21.15.141:0/398706468 wait complete. 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.248+0000 7f34fa3ac740 20 client.4656 trim_cache size 0 max 16384 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4249, 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T14:57:24.235 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T14:57:24.236 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 16.724430131999998, 2024-01-16T14:57:24.236 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 8, 2024-01-16T14:57:24.236 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 22, 2024-01-16T14:57:24.236 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 22, 2024-01-16T14:57:24.236 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 24.311155839000001 2024-01-16T14:57:24.236 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f34c806b2c0 msgr2=0x7f34c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f34c806b2c0 0x7f34c806d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34f013c210 tx=0x7f34d8008040 comp rx=0 tx=0).stop 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34f013df80 msgr2=0x7f34f0156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34f013df80 0x7f34f0156460 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f34e0000c00 tx=0x7f34e0004730 comp rx=0 tx=0).stop 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 shutdown_connections 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f34c806b2c0 0x7f34c806d780 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34f015f9c0 0x7f34f013c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f013e360 0x7f34f013d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 --2- 172.21.15.141:0/3942330749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34f013df80 0x7f34f0156460 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 >> 172.21.15.141:0/3942330749 conn(0x7f34f006a800 msgr2=0x7f34f0154dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 shutdown_connections 2024-01-16T14:57:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.249+0000 7f34f7800640 1 -- 172.21.15.141:0/3942330749 wait complete. 2024-01-16T14:57:24.248 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-16T14:57:24.273 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-16T14:57:24.273 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-16T14:57:24.274 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-16T14:57:24.274 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:24.274 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:24.274 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-16T14:57:24.274 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:24.274 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:24.274 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-16T14:57:24.274 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:24.274 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:24.275 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-16T14:57:24.275 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:57:24.275 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:24.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:24.275 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T14:57:24.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:24.330 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link delete ceph-brx 2024-01-16T14:57:24.429 INFO:teuthology.orchestra.run.smithi141.stderr:Cannot find device "ceph-brx" 2024-01-16T14:57:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:57:24.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:24.434 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T14:57:24.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:24.492 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link delete ceph-brx 2024-01-16T14:57:24.588 INFO:teuthology.orchestra.run.smithi188.stderr:Cannot find device "ceph-brx" 2024-01-16T14:57:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:57:24.592 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-16T14:57:24.592 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:24.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.830+0000 7fcec2400640 1 Processor -- start 2024-01-16T14:57:24.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.831+0000 7fcec2400640 1 -- start start 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcec2400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcebc13df80 0x7fcebc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcec2400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13e360 0x7fcebc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcec2400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc06c470 con 0x7fcebc15f9c0 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcec2400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc06c750 con 0x7fcebc13df80 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcec2400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc06ca30 con 0x7fcebc13e360 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcebb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcebc13df80 0x7fcebc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcebb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcebc13df80 0x7fcebc15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49924/0 (socket says 172.21.15.141:49924) 2024-01-16T14:57:24.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcebb400640 1 -- 172.21.15.141:0/2066943017 learned_addr learned my addr 172.21.15.141:0/2066943017 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.832+0000 7fcebaa00640 1 --2- 172.21.15.141:0/2066943017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13e360 0x7fcebc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.833+0000 7fcebb400640 1 -- 172.21.15.141:0/2066943017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13e360 msgr2=0x7fcebc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.833+0000 7fcebb400640 1 --2- 172.21.15.141:0/2066943017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13e360 0x7fcebc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.833+0000 7fcebb400640 1 -- 172.21.15.141:0/2066943017 >> v1:172.21.15.188:6789/0 conn(0x7fcebc15f9c0 legacy=0x7fcebc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:24.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.833+0000 7fcebb400640 1 -- 172.21.15.141:0/2066943017 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc06cd10 con 0x7fcebc13df80 2024-01-16T14:57:24.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.834+0000 7fcebb400640 1 --2- 172.21.15.141:0/2066943017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13df80 0x7fcebc15bee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fceac009870 tx=0x7fceac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ea24bade7c5e36 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.834+0000 7fceba000640 1 -- 172.21.15.141:0/2066943017 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac031640 con 0x7fcebc13df80 2024-01-16T14:57:24.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.834+0000 7fceba000640 1 -- 172.21.15.141:0/2066943017 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac0317a0 con 0x7fcebc13df80 2024-01-16T14:57:24.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.834+0000 7fceba000640 1 -- 172.21.15.141:0/2066943017 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac031b30 con 0x7fcebc13df80 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 -- 172.21.15.141:0/2066943017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13df80 msgr2=0x7fcebc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 --2- 172.21.15.141:0/2066943017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13df80 0x7fcebc15bee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fceac009870 tx=0x7fceac02b8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 -- 172.21.15.141:0/2066943017 shutdown_connections 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 --2- 172.21.15.141:0/2066943017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13e360 0x7fcebc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 --2- 172.21.15.141:0/2066943017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13df80 0x7fcebc15bee0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 -- 172.21.15.141:0/2066943017 >> 172.21.15.141:0/2066943017 conn(0x7fcebc06a800 msgr2=0x7fcebc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 -- 172.21.15.141:0/2066943017 shutdown_connections 2024-01-16T14:57:24.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.835+0000 7fcec2400640 1 -- 172.21.15.141:0/2066943017 wait complete. 2024-01-16T14:57:24.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.837+0000 7fcec2400640 1 Processor -- start 2024-01-16T14:57:24.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.837+0000 7fcec2400640 1 -- start start 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcec2400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13df80 0x7fcebc1e5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcec2400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13e360 0x7fcebc150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcec2400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcebc15f9c0 0x7fcebc152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcec2400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc154170 con 0x7fcebc13e360 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcec2400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc150bd0 con 0x7fcebc15f9c0 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcec2400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1534f0 con 0x7fcebc13df80 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13e360 0x7fcebc150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13e360 0x7fcebc150690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49928/0 (socket says 172.21.15.141:49928) 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 -- 172.21.15.141:0/3737876830 learned_addr learned my addr 172.21.15.141:0/3737876830 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebb400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13df80 0x7fcebc1e5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebbe00640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcebc15f9c0 0x7fcebc152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 -- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13df80 msgr2=0x7fcebc1e5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13df80 0x7fcebc1e5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 -- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcebc15f9c0 msgr2=0x7fcebc152cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcebc15f9c0 0x7fcebc152cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebaa00640 1 -- 172.21.15.141:0/3737876830 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac0094d0 con 0x7fcebc13e360 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebb400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13df80 0x7fcebc1e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.838+0000 7fcebbe00640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcebc15f9c0 0x7fcebc152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.839+0000 7fcebaa00640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13e360 0x7fcebc150690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcea400d800 tx=0x7fcea400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.839+0000 7fceb3400640 1 -- 172.21.15.141:0/3737876830 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea400b6c0 con 0x7fcebc13e360 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.839+0000 7fceb3400640 1 -- 172.21.15.141:0/3737876830 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea400b820 con 0x7fcebc13e360 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.839+0000 7fceb3400640 1 -- 172.21.15.141:0/3737876830 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea400bbb0 con 0x7fcebc13e360 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.839+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc1537d0 con 0x7fcebc13e360 2024-01-16T14:57:24.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.839+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc153d90 con 0x7fcebc13e360 2024-01-16T14:57:24.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.840+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc0675d0 con 0x7fcebc13e360 2024-01-16T14:57:24.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.841+0000 7fceb3400640 1 -- 172.21.15.141:0/3737876830 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcea400bd10 con 0x7fcebc13e360 2024-01-16T14:57:24.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.842+0000 7fceb3400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce9406b2c0 0x7fce9406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:24.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.842+0000 7fceb3400640 1 -- 172.21.15.141:0/3737876830 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcea408a210 con 0x7fcebc13e360 2024-01-16T14:57:24.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.843+0000 7fcebb400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce9406b2c0 0x7fce9406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:24.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.844+0000 7fcebb400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce9406b2c0 0x7fce9406d780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fceac036000 tx=0x7fceac03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:24.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:24.846+0000 7fceb3400640 1 -- 172.21.15.141:0/3737876830 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcea4054fe0 con 0x7fcebc13e360 2024-01-16T14:57:25.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.073+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcebc15b9d0 con 0x7fcebc13e360 2024-01-16T14:57:25.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.074+0000 7fceb3400640 1 -- 172.21.15.141:0/3737876830 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcea4058600 con 0x7fcebc13e360 2024-01-16T14:57:25.059 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:25.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce9406b2c0 msgr2=0x7fce9406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce9406b2c0 0x7fce9406d780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fceac036000 tx=0x7fceac03c040 comp rx=0 tx=0).stop 2024-01-16T14:57:25.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13e360 msgr2=0x7fcebc150690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:25.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13e360 0x7fcebc150690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcea400d800 tx=0x7fcea400dcc0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 shutdown_connections 2024-01-16T14:57:25.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce9406b2c0 0x7fce9406d780 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcebc15f9c0 0x7fcebc152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcebc13e360 0x7fcebc150690 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 --2- 172.21.15.141:0/3737876830 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcebc13df80 0x7fcebc1e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:25.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 >> 172.21.15.141:0/3737876830 conn(0x7fcebc06a800 msgr2=0x7fcebc070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:25.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 shutdown_connections 2024-01-16T14:57:25.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:25.077+0000 7fcec2400640 1 -- 172.21.15.141:0/3737876830 wait complete. 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:25.077 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:57:25.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:25.078 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T14:57:25.126 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42618sec preferred_lft 42618sec 2024-01-16T14:57:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T14:57:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:25.130 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T14:57:25.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:25.130 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:25.130 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:25.130 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link add name ceph-brx type bridge 2024-01-16T14:57:25.131 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip addr flush dev ceph-brx 2024-01-16T14:57:25.131 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set ceph-brx up 2024-01-16T14:57:25.131 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T14:57:25.131 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:25.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:25.573 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:25.652 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T14:57:25.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:25.656 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T14:57:25.712 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T14:57:25.713 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:25.713 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:25.713 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:25.713 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:25.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:25.714 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:25.714 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:25.714 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:25.715 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:25.715 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:25.715 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:26.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:26.379 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T14:57:26.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:26.422 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list-id 2024-01-16T14:57:26.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:26.508 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:26.508 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:26.508 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:26.508 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-16T14:57:26.508 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:26.677 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-16T14:57:26.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:26.677 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:26.677 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:26.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-16T14:57:26.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:26.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-16T14:57:26.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-16T14:57:26.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-16T14:57:26.678 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:26.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:26.942 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:26.942 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:26.942 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T14:57:26.942 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T14:57:26.942 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:27.098 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T14:57:27.098 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:27.098 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:27.139 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:57:27.139 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:27.139 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:27.231 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T14:57:27.313 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T14:57:27.358 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:27.360 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:27.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:27.361 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T14:57:27.450 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:57:27.450 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T14:57:27.498 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T14:57:27.573 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T14:57:27.612 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T14:57:27.613 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:27.614 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:27.614 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:27.614 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:27.614 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T14:57:27.614 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:27.614 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:27.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:27.615 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T14:57:27.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T14:57:27.698+0000 7ff1abe0c480 -1 init, newargv = 0x564df04a47a0 newargc=15 2024-01-16T14:57:27.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[43484]: starting ceph client 2024-01-16T14:57:27.692 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[43484]: starting fuse 2024-01-16T14:57:27.700 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T14:57:27.701 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T14:57:27.701 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:27.702 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T14:57:27.703 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T14:57:27.895 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.43484.asok 2024-01-16T14:57:27.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:27.911 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43484.asok 2024-01-16T14:57:27.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:27.912 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43484.asok status 2024-01-16T14:57:28.138 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:57:28.138 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "43484", 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4671, 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T14:57:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 4671 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 26399570 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 26399570 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.4671 192.168.0.1:0/26399570", 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/26399570", 2024-01-16T14:57:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T14:57:28.141 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 8, 2024-01-16T14:57:28.141 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 22, 2024-01-16T14:57:28.141 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:28.141 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T14:57:28.141 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T14:57:28.141 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:57:28.152 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:28.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.367+0000 7f9b24c00640 1 Processor -- start 2024-01-16T14:57:28.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.368+0000 7f9b24c00640 1 -- start start 2024-01-16T14:57:28.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b24c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9b200ae5a0 0x7f9b2015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:28.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b24c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 0x7f9b2006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b24c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b20070c40 con 0x7f9b2006d120 2024-01-16T14:57:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b24c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b20071450 con 0x7f9b200ae5a0 2024-01-16T14:57:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b24c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2006c470 con 0x7f9b200b3ae0 2024-01-16T14:57:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b1de00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9b200ae5a0 0x7f9b2015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b1e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9b2006d120 0x7f9b200cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38640/0 (socket says 172.21.15.141:38640) 2024-01-16T14:57:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b1e800640 1 -- 172.21.15.141:0/1583952339 learned_addr learned my addr 172.21.15.141:0/1583952339 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:28.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.369+0000 7f9b15200640 1 --2- 172.21.15.141:0/1583952339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 0x7f9b2006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:28.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.370+0000 7f9b1d400640 1 -- 172.21.15.141:0/1583952339 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484000473 0 0) 0x7f9b20070c40 con 0x7f9b2006d120 2024-01-16T14:57:28.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.370+0000 7f9b1de00640 1 -- 172.21.15.141:0/1583952339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 msgr2=0x7f9b2006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:28.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.370+0000 7f9b1de00640 1 --2- 172.21.15.141:0/1583952339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 0x7f9b2006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.370+0000 7f9b1de00640 1 -- 172.21.15.141:0/1583952339 >> v1:172.21.15.141:6789/0 conn(0x7f9b2006d120 legacy=0x7f9b200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:28.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.370+0000 7f9b1de00640 1 -- 172.21.15.141:0/1583952339 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2006c750 con 0x7f9b200ae5a0 2024-01-16T14:57:28.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.371+0000 7f9b1de00640 1 --2- 172.21.15.141:0/1583952339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b2015e430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009d90 tx=0x7f9b1002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c1aa5df029bf80 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:28.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.371+0000 7f9b1d400640 1 -- 172.21.15.141:0/1583952339 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1002cc50 con 0x7f9b200ae5a0 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.371+0000 7f9b1d400640 1 -- 172.21.15.141:0/1583952339 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1002cdb0 con 0x7f9b200ae5a0 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.371+0000 7f9b1d400640 1 -- 172.21.15.141:0/1583952339 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b10034660 con 0x7f9b200ae5a0 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 -- 172.21.15.141:0/1583952339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 msgr2=0x7f9b2015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 --2- 172.21.15.141:0/1583952339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b2015e430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009d90 tx=0x7f9b1002bea0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 -- 172.21.15.141:0/1583952339 shutdown_connections 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 --2- 172.21.15.141:0/1583952339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 0x7f9b2006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 --2- 172.21.15.141:0/1583952339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b2015e430 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 -- 172.21.15.141:0/1583952339 >> 172.21.15.141:0/1583952339 conn(0x7f9b2006a800 msgr2=0x7f9b2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 -- 172.21.15.141:0/1583952339 shutdown_connections 2024-01-16T14:57:28.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.372+0000 7f9b24c00640 1 -- 172.21.15.141:0/1583952339 wait complete. 2024-01-16T14:57:28.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.374+0000 7f9b24c00640 1 Processor -- start 2024-01-16T14:57:28.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.374+0000 7f9b24c00640 1 -- start start 2024-01-16T14:57:28.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b24c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b2006d120 0x7f9b200b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:28.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b24c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b201f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:28.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b24c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 0x7f9b20149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:28.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b24c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2015c070 con 0x7f9b200ae5a0 2024-01-16T14:57:28.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b24c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2015c350 con 0x7f9b2006d120 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b24c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2015c630 con 0x7f9b200b3ae0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b201f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b201f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49946/0 (socket says 172.21.15.141:49946) 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 -- 172.21.15.141:0/300675531 learned_addr learned my addr 172.21.15.141:0/300675531 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 -- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 msgr2=0x7f9b20149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 0x7f9b20149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 -- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b2006d120 msgr2=0x7f9b200b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b2006d120 0x7f9b200b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 -- 172.21.15.141:0/300675531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b10009a70 con 0x7f9b200ae5a0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.375+0000 7f9b15200640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b201f2410 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800d5f0 tx=0x7f9b0800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.376+0000 7f9b16a00640 1 -- 172.21.15.141:0/300675531 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b080125f0 con 0x7f9b200ae5a0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.376+0000 7f9b16a00640 1 -- 172.21.15.141:0/300675531 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b08012750 con 0x7f9b200ae5a0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.376+0000 7f9b16a00640 1 -- 172.21.15.141:0/300675531 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08012b00 con 0x7f9b200ae5a0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.376+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2015c910 con 0x7f9b200ae5a0 2024-01-16T14:57:28.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.376+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2015bcf0 con 0x7f9b200ae5a0 2024-01-16T14:57:28.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.377+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b200675d0 con 0x7f9b200ae5a0 2024-01-16T14:57:28.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.378+0000 7f9b16a00640 1 -- 172.21.15.141:0/300675531 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9b08012c60 con 0x7f9b200ae5a0 2024-01-16T14:57:28.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.379+0000 7f9b16a00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9af806b2c0 0x7f9af806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:28.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.379+0000 7f9b16a00640 1 -- 172.21.15.141:0/300675531 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b0808a140 con 0x7f9b200ae5a0 2024-01-16T14:57:28.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.380+0000 7f9b1de00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9af806b2c0 0x7f9af806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:28.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.381+0000 7f9b1de00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9af806b2c0 0x7f9af806d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9b10036000 tx=0x7f9b1002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:28.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.384+0000 7f9b16a00640 1 -- 172.21.15.141:0/300675531 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b08054f10 con 0x7f9b200ae5a0 2024-01-16T14:57:28.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.613+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9b20071b30 con 0x7f9b200ae5a0 2024-01-16T14:57:28.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.614+0000 7f9b16a00640 1 -- 172.21.15.141:0/300675531 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9b08058530 con 0x7f9b200ae5a0 2024-01-16T14:57:28.596 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:28.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.616+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9af806b2c0 msgr2=0x7f9af806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:28.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.616+0000 7f9b24c00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9af806b2c0 0x7f9af806d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9b10036000 tx=0x7f9b1002e000 comp rx=0 tx=0).stop 2024-01-16T14:57:28.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.616+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 msgr2=0x7f9b201f2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:28.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.616+0000 7f9b24c00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b201f2410 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800d5f0 tx=0x7f9b0800dab0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 shutdown_connections 2024-01-16T14:57:28.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9af806b2c0 0x7f9af806d780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b200b3ae0 0x7f9b20149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b200ae5a0 0x7f9b201f2410 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 --2- 172.21.15.141:0/300675531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b2006d120 0x7f9b200b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 >> 172.21.15.141:0/300675531 conn(0x7f9b2006a800 msgr2=0x7f9b200b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 shutdown_connections 2024-01-16T14:57:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:28.617+0000 7f9b24c00640 1 -- 172.21.15.141:0/300675531 wait complete. 2024-01-16T14:57:28.615 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:28.615 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:28.615 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T14:57:28.615 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-16T14:57:28.615 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:28.615 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:28.615 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:28.616 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:28.616 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-16T14:57:28.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:28.616 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T14:57:28.664 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:28.664 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:28.664 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:28.664 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:28.665 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42615sec preferred_lft 42615sec 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T14:57:28.666 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:28.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:28.667 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:28.797 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T14:57:28.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:28.801 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T14:57:28.851 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T14:57:28.851 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:28.851 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:28.851 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:28.851 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:28.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:28.852 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:28.852 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:28.852 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:28.852 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:28.852 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:28.852 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:29.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:29.027 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T14:57:29.077 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T14:57:29.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:29.078 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list-id 2024-01-16T14:57:29.126 INFO:teuthology.orchestra.run.smithi141.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-16T14:57:29.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:29.128 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:29.128 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:29.128 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:29.128 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-16T14:57:29.128 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:29.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:29.262 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T14:57:29.349 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:29.349 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:29.349 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:29.349 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.349 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:29.350 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.350 INFO:teuthology.orchestra.run.smithi141.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:29.350 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 56:52:29:1c:43:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:29.350 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:29.350 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.350 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::5452:29ff:fe1c:43f8/64 scope link 2024-01-16T14:57:29.350 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:29.353 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T14:57:29.444 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:29.444 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 56:52:29:1c:43:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::5452:29ff:fe1c:43f8/64 scope link 2024-01-16T14:57:29.445 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:29.449 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-16T14:57:29.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-16T14:57:29.450 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:29.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:29.725 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:57:29.726 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:57:29.726 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.1 up 2024-01-16T14:57:29.726 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T14:57:29.726 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:57:29.897 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-16T14:57:29.897 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:29.897 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:29.944 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-16T14:57:29.945 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:29.945 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:30.038 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T14:57:30.134 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:30.189 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T14:57:30.189 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T14:57:30.190 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:30.191 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:30.192 INFO:teuthology.orchestra.run.smithi141.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:30.192 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:30.192 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:30.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:30.193 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T14:57:30.244 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T14:57:30.244 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-01-16T14:57:30.245 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-16T14:57:30.249 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T14:57:30.323 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T14:57:30.364 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:57:30.365 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:30.366 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:57:30.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:30.367 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T14:57:30.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi141.stderr:2024-01-16T14:57:30.452+0000 7f73f264a480 -1 init, newargv = 0x55ba0bcc75d0 newargc=15 2024-01-16T14:57:30.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi141.stderr:ceph-fuse[43978]: starting ceph client 2024-01-16T14:57:30.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi141.stderr:ceph-fuse[43978]: starting fuse 2024-01-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi141.stdout:42 2024-01-16T14:57:30.454 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 42] 2024-01-16T14:57:30.454 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:30.454 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T14:57:30.454 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T14:57:30.454 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T14:57:30.454 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T14:57:30.454 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T14:57:30.455 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T14:57:30.456 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:30.456 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:57:30.456 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.2")) 2024-01-16T14:57:30.456 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T14:57:30.650 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.2.43978.asok 2024-01-16T14:57:30.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:30.667 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.43978.asok 2024-01-16T14:57:30.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:30.667 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.43978.asok status 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "2", 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "43978", 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4686, 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 4686 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 1383281183 2024-01-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 1383281183 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.4686 192.168.0.2:0/1383281183", 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.2:0/1383281183", 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 8, 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 22, 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:30.900 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:57:30.913 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.142+0000 7f46dc000640 1 Processor -- start 2024-01-16T14:57:31.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.142+0000 7f46dc000640 1 -- start start 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46dc000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46d40cf9a0 0x7f46d4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46dc000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46dc000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d4071690 con 0x7f46d414b2c0 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46dc000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d414b6a0 con 0x7f46d40cf9a0 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46dc000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4071970 con 0x7f46d4070ea0 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46d8c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46d40cf9a0 0x7f46d4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46d8c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46d40cf9a0 0x7f46d4070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49952/0 (socket says 172.21.15.141:49952) 2024-01-16T14:57:31.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.144+0000 7f46d8c00640 1 -- 172.21.15.141:0/1192829696 learned_addr learned my addr 172.21.15.141:0/1192829696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:31.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.145+0000 7f46da000640 1 --2- 172.21.15.141:0/1192829696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.145+0000 7f46d8c00640 1 -- 172.21.15.141:0/1192829696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 msgr2=0x7f46d40cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:31.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.145+0000 7f46d8c00640 1 --2- 172.21.15.141:0/1192829696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d40cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.145+0000 7f46d8c00640 1 -- 172.21.15.141:0/1192829696 >> v1:172.21.15.141:6789/0 conn(0x7f46d414b2c0 legacy=0x7f46d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.145+0000 7f46d3e00640 1 -- 172.21.15.141:0/1192829696 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784507527 0 0) 0x7f46d4071690 con 0x7f46d414b2c0 2024-01-16T14:57:31.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.145+0000 7f46d8c00640 1 -- 172.21.15.141:0/1192829696 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d4071c50 con 0x7f46d40cf9a0 2024-01-16T14:57:31.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.147+0000 7f46d8c00640 1 --2- 172.21.15.141:0/1192829696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d4070960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009870 tx=0x7f46bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7448d90e6422e937 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.147+0000 7f46d3e00640 1 -- 172.21.15.141:0/1192829696 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc030840 con 0x7f46d40cf9a0 2024-01-16T14:57:31.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.147+0000 7f46d3e00640 1 -- 172.21.15.141:0/1192829696 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc0309a0 con 0x7f46d40cf9a0 2024-01-16T14:57:31.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.147+0000 7f46d3e00640 1 -- 172.21.15.141:0/1192829696 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc030d30 con 0x7f46d40cf9a0 2024-01-16T14:57:31.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.147+0000 7f46dc000640 1 -- 172.21.15.141:0/1192829696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 msgr2=0x7f46d4070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.147+0000 7f46dc000640 1 --2- 172.21.15.141:0/1192829696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d4070960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009870 tx=0x7f46bc02b980 comp rx=0 tx=0).stop 2024-01-16T14:57:31.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.148+0000 7f46dc000640 1 -- 172.21.15.141:0/1192829696 shutdown_connections 2024-01-16T14:57:31.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.148+0000 7f46dc000640 1 --2- 172.21.15.141:0/1192829696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d40cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.148+0000 7f46dc000640 1 --2- 172.21.15.141:0/1192829696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d4070960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.148+0000 7f46dc000640 1 -- 172.21.15.141:0/1192829696 >> 172.21.15.141:0/1192829696 conn(0x7f46d406a800 msgr2=0x7f46d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:31.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.148+0000 7f46dc000640 1 -- 172.21.15.141:0/1192829696 shutdown_connections 2024-01-16T14:57:31.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.148+0000 7f46dc000640 1 -- 172.21.15.141:0/1192829696 wait complete. 2024-01-16T14:57:31.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.149+0000 7f46dc000640 1 Processor -- start 2024-01-16T14:57:31.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.150+0000 7f46dc000640 1 -- start start 2024-01-16T14:57:31.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46dc000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d41f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46dc000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d41f68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46dc000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46d414b2c0 0x7f46d4141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46dc000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d41f6e00 con 0x7f46d40cf9a0 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46dc000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4142670 con 0x7f46d414b2c0 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46dc000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4143f50 con 0x7f46d4070ea0 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d41f68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d41f68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49966/0 (socket says 172.21.15.141:49966) 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 -- 172.21.15.141:0/2081847181 learned_addr learned my addr 172.21.15.141:0/2081847181 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d9600640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d41f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 -- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 msgr2=0x7f46d41f6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d41f6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 -- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46d414b2c0 msgr2=0x7f46d4141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46d414b2c0 0x7f46d4141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d8c00640 1 -- 172.21.15.141:0/2081847181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0094d0 con 0x7f46d40cf9a0 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.151+0000 7f46d9600640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d41f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.152+0000 7f46d8c00640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d41f68c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009870 tx=0x7f46bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.152+0000 7f46d1600640 1 -- 172.21.15.141:0/2081847181 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc039c90 con 0x7f46d40cf9a0 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.152+0000 7f46d1600640 1 -- 172.21.15.141:0/2081847181 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc039df0 con 0x7f46d40cf9a0 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.152+0000 7f46d1600640 1 -- 172.21.15.141:0/2081847181 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc043620 con 0x7f46d40cf9a0 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.152+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d4142390 con 0x7f46d40cf9a0 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.152+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d4143be0 con 0x7f46d40cf9a0 2024-01-16T14:57:31.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.153+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d40cc660 con 0x7f46d40cf9a0 2024-01-16T14:57:31.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.154+0000 7f46d1600640 1 -- 172.21.15.141:0/2081847181 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f46bc043780 con 0x7f46d40cf9a0 2024-01-16T14:57:31.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.155+0000 7f46d1600640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46ac06b2c0 0x7f46ac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:31.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.155+0000 7f46d1600640 1 -- 172.21.15.141:0/2081847181 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46bc03e070 con 0x7f46d40cf9a0 2024-01-16T14:57:31.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.156+0000 7f46d9600640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46ac06b2c0 0x7f46ac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:31.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.157+0000 7f46d9600640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46ac06b2c0 0x7f46ac06d780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46c40012e0 tx=0x7f46c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:31.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.159+0000 7f46d1600640 1 -- 172.21.15.141:0/2081847181 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46bc077b10 con 0x7f46d40cf9a0 2024-01-16T14:57:31.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.392+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f46d41440c0 con 0x7f46d40cf9a0 2024-01-16T14:57:31.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.393+0000 7f46d1600640 1 -- 172.21.15.141:0/2081847181 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f46bc0ace60 con 0x7f46d40cf9a0 2024-01-16T14:57:31.374 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.395+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46ac06b2c0 msgr2=0x7f46ac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.395+0000 7f46dc000640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46ac06b2c0 0x7f46ac06d780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46c40012e0 tx=0x7f46c4009040 comp rx=0 tx=0).stop 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.395+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 msgr2=0x7f46d41f68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.395+0000 7f46dc000640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d41f68c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009870 tx=0x7f46bc004730 comp rx=0 tx=0).stop 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 shutdown_connections 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46ac06b2c0 0x7f46ac06d780 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46d414b2c0 0x7f46d4141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46d40cf9a0 0x7f46d41f68c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 --2- 172.21.15.141:0/2081847181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46d4070ea0 0x7f46d41f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 >> 172.21.15.141:0/2081847181 conn(0x7f46d406a800 msgr2=0x7f46d40b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 shutdown_connections 2024-01-16T14:57:31.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:31.396+0000 7f46dc000640 1 -- 172.21.15.141:0/2081847181 wait complete. 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:31.396 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:57:31.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:31.397 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:31.454 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42614sec preferred_lft 42614sec 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T14:57:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:31.456 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T14:57:31.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:31.456 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:31.456 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:31.456 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link add name ceph-brx type bridge 2024-01-16T14:57:31.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip addr flush dev ceph-brx 2024-01-16T14:57:31.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set ceph-brx up 2024-01-16T14:57:31.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T14:57:31.457 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:31.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:31.895 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:31.968 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T14:57:31.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:31.972 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T14:57:32.023 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T14:57:32.023 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:32.023 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:32.023 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:32.023 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:32.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:32.023 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:32.023 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:32.023 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:32.023 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:32.023 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:32.024 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:32.059 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:32.067 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:57:32.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:32.745 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T14:57:32.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:32.798 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list-id 2024-01-16T14:57:32.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:32.847 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:32.848 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:32.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:32.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-16T14:57:32.848 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:32.982 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-16T14:57:32.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:32.982 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:32.982 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:32.983 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-16T14:57:32.983 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:32.983 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-16T14:57:32.983 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-16T14:57:32.983 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-16T14:57:32.983 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:33.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:33.258 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:33.258 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:33.258 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T14:57:33.258 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T14:57:33.258 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:33.419 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T14:57:33.419 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:33.419 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:33.466 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:57:33.467 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:33.467 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:33.559 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:57:33.647 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:57:33.699 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:33.700 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:33.701 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:33.701 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:57:33.701 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:33.701 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:33.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:33.702 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:57:33.750 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:57:33.750 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T14:57:33.754 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:57:33.823 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:57:33.868 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:57:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:57:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:57:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:33.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:33.871 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:57:33.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T14:57:33.963+0000 7fb06d63f480 -1 init, newargv = 0x562322ef3960 newargc=15 2024-01-16T14:57:33.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[41358]: starting ceph client 2024-01-16T14:57:33.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[41358]: starting fuse 2024-01-16T14:57:35.056 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:57:35.150 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:35.201 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:57:35.202 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:35.203 INFO:teuthology.orchestra.run.smithi188.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:35.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:35.204 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:57:35.253 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T14:57:35.253 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T14:57:35.253 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T14:57:35.254 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T14:57:35.255 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T14:57:35.433 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.41358.asok 2024-01-16T14:57:35.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:57:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:35.451 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41358.asok 2024-01-16T14:57:35.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:35.452 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41358.asok status 2024-01-16T14:57:35.686 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T14:57:35.686 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T14:57:35.686 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:35.686 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:35.686 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "41358", 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 4424, 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 4424 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T14:57:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 1348603003 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 1348603003 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.4424 192.168.0.1:0/1348603003", 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/1348603003", 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 8, 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 22, 2024-01-16T14:57:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:35.689 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T14:57:35.689 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T14:57:35.689 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T14:57:35.701 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:57:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.937+0000 7f810fe00640 1 Processor -- start 2024-01-16T14:57:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.938+0000 7f810fe00640 1 -- start start 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.939+0000 7f810fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8110160920 0x7f81100b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.939+0000 7f810fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100d0a10 con 0x7f81100cf9f0 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.939+0000 7f810fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8110161e30 con 0x7f81100cfe70 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.939+0000 7f810fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81100d1f50 con 0x7f8110160920 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.939+0000 7f810e000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f81100cf9f0 0x7f8110161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42786/0 (socket says 172.21.15.141:42786) 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.939+0000 7f810e000640 1 -- 172.21.15.141:0/222858419 learned_addr learned my addr 172.21.15.141:0/222858419 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810ea00640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8110160920 0x7f81100b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051792014 0 0) 0x7f81100d0a10 con 0x7f81100cf9f0 2024-01-16T14:57:35.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0003520 con 0x7f81100cf9f0 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439794458 0 0) 0x7f80f0003520 con 0x7f81100cf9f0 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8110160920 msgr2=0x7f81100b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8110160920 0x7f81100b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 >> v1:172.21.15.188:6789/0 conn(0x7f81100cfe70 legacy=0x7f8110161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100cf080 con 0x7f81100cf9f0 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f80f80019f0 con 0x7f81100cf9f0 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 >> v1:172.21.15.141:6789/0 conn(0x7f81100cf9f0 legacy=0x7f8110161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80f00039a0 0x7f80f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8110160920 0x7f80f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80f0004d00 0x7f80f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100d1f50 con 0x7f80f0004d00 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f000ba30 con 0x7f80f00039a0 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.940+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f000bbf0 con 0x7f8110160920 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.941+0000 7f810f400640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80f0004d00 0x7f80f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.941+0000 7f810ea00640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80f00039a0 0x7f80f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.941+0000 7f810f400640 1 -- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8110160920 msgr2=0x7f80f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.941+0000 7f810f400640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8110160920 0x7f80f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.941+0000 7f810f400640 1 -- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80f00039a0 msgr2=0x7f80f0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.941+0000 7f810f400640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80f00039a0 0x7f80f0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.941+0000 7f810f400640 1 -- 172.21.15.141:0/222858419 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0003520 con 0x7f80f0004d00 2024-01-16T14:57:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.942+0000 7f810f400640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80f0004d00 0x7f80f0005180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8108001720 tx=0x7f80f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe01a569c47910 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.942+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81080052d0 con 0x7f80f0004d00 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.942+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810802e050 con 0x7f80f0004d00 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.942+0000 7f810d600640 1 -- 172.21.15.141:0/222858419 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81080035f0 con 0x7f80f0004d00 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.943+0000 7f810fe00640 1 -- 172.21.15.141:0/222858419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80f0004d00 msgr2=0x7f80f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.943+0000 7f810fe00640 1 --2- 172.21.15.141:0/222858419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80f0004d00 0x7f80f0005180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8108001720 tx=0x7f80f00057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.943+0000 7f810f400640 1 -- 172.21.15.141:0/222858419 reap_dead start 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.943+0000 7f810fe00640 1 -- 172.21.15.141:0/222858419 shutdown_connections 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.943+0000 7f810fe00640 1 -- 172.21.15.141:0/222858419 >> 172.21.15.141:0/222858419 conn(0x7f81100d3090 msgr2=0x7f81100d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.943+0000 7f810fe00640 1 -- 172.21.15.141:0/222858419 shutdown_connections 2024-01-16T14:57:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.943+0000 7f810fe00640 1 -- 172.21.15.141:0/222858419 wait complete. 2024-01-16T14:57:35.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.945+0000 7f810fe00640 1 Processor -- start 2024-01-16T14:57:35.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.945+0000 7f810fe00640 1 -- start start 2024-01-16T14:57:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81101502d0 0x7f81100caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81101506b0 0x7f811015d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811015d850 0x7f81100b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811015de80 con 0x7f811015d850 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100ca160 con 0x7f81101506b0 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81100ca440 con 0x7f81101502d0 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811015d850 0x7f81100b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811015d850 0x7f81100b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55722/0 (socket says 172.21.15.141:55722) 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.946+0000 7f810f400640 1 -- 172.21.15.141:0/436225787 learned_addr learned my addr 172.21.15.141:0/436225787 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810ea00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81101502d0 0x7f81100caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810e000640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81101506b0 0x7f811015d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810f400640 1 -- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81101502d0 msgr2=0x7f81100caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810f400640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81101502d0 0x7f81100caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810f400640 1 -- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81101506b0 msgr2=0x7f811015d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810f400640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81101506b0 0x7f811015d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810f400640 1 -- 172.21.15.141:0/436225787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100ca720 con 0x7f811015d850 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810e000640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81101506b0 0x7f811015d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f810f400640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811015d850 0x7f81100b7380 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8108001840 tx=0x7f8108001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.947+0000 7f8106a00640 1 -- 172.21.15.141:0/436225787 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108041070 con 0x7f811015d850 2024-01-16T14:57:35.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.948+0000 7f8106a00640 1 -- 172.21.15.141:0/436225787 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8108007a30 con 0x7f811015d850 2024-01-16T14:57:35.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.948+0000 7f8106a00640 1 -- 172.21.15.141:0/436225787 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81080067d0 con 0x7f811015d850 2024-01-16T14:57:35.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.948+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81100caa00 con 0x7f811015d850 2024-01-16T14:57:35.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.948+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81100b78c0 con 0x7f811015d850 2024-01-16T14:57:35.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.949+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8110067450 con 0x7f811015d850 2024-01-16T14:57:35.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.950+0000 7f8106a00640 1 -- 172.21.15.141:0/436225787 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8108006930 con 0x7f811015d850 2024-01-16T14:57:35.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.951+0000 7f8106a00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80e806b2c0 0x7f80e806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:35.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.951+0000 7f8106a00640 1 -- 172.21.15.141:0/436225787 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81080aae60 con 0x7f811015d850 2024-01-16T14:57:35.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.951+0000 7f810ea00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80e806b2c0 0x7f80e806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:35.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.953+0000 7f810ea00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80e806b2c0 0x7f80e806d780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8100009a10 tx=0x7f8100008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:35.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:35.956+0000 7f8106a00640 1 -- 172.21.15.141:0/436225787 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8108075c30 con 0x7f811015d850 2024-01-16T14:57:36.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.184+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f811006c0b0 con 0x7f811015d850 2024-01-16T14:57:36.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.184+0000 7f8106a00640 1 -- 172.21.15.141:0/436225787 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8108031e80 con 0x7f811015d850 2024-01-16T14:57:36.166 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:57:36.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80e806b2c0 msgr2=0x7f80e806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80e806b2c0 0x7f80e806d780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8100009a10 tx=0x7f8100008040 comp rx=0 tx=0).stop 2024-01-16T14:57:36.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811015d850 msgr2=0x7f81100b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:36.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811015d850 0x7f81100b7380 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8108001840 tx=0x7f8108001120 comp rx=0 tx=0).stop 2024-01-16T14:57:36.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 shutdown_connections 2024-01-16T14:57:36.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80e806b2c0 0x7f80e806d780 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811015d850 0x7f81100b7380 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81101506b0 0x7f811015d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 --2- 172.21.15.141:0/436225787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81101502d0 0x7f81100caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:36.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 >> 172.21.15.141:0/436225787 conn(0x7f81100d3090 msgr2=0x7f811006c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:36.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.187+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 shutdown_connections 2024-01-16T14:57:36.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:36.188+0000 7f810fe00640 1 -- 172.21.15.141:0/436225787 wait complete. 2024-01-16T14:57:36.186 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:57:36.186 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:57:36.187 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T14:57:36.187 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-16T14:57:36.187 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:36.187 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:57:36.187 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:57:36.187 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:57:36.187 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-16T14:57:36.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.187 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T14:57:36.244 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:36.245 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42609sec preferred_lft 42609sec 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T14:57:36.246 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.247 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:57:36.332 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T14:57:36.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.337 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T14:57:36.388 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T14:57:36.388 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:57:36.388 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:57:36.389 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:57:36.389 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:57:36.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.390 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:36.390 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:36.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:57:36.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:57:36.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:57:36.390 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:36.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.570 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T14:57:36.618 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T14:57:36.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.618 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list-id 2024-01-16T14:57:36.667 INFO:teuthology.orchestra.run.smithi188.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-16T14:57:36.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.668 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:36.668 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:36.668 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:36.668 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-16T14:57:36.668 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:36.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.797 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 26:4a:0f:f2:e3:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::244a:fff:fef2:e35d/64 scope link 2024-01-16T14:57:36.887 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.892 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T14:57:36.985 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:57:36.985 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:57:36.985 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 26:4a:0f:f2:e3:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::244a:fff:fef2:e35d/64 scope link 2024-01-16T14:57:36.986 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:57:36.990 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-16T14:57:36.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:36.990 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:36.990 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:36.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-16T14:57:36.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T14:57:36.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-16T14:57:36.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-16T14:57:36.991 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-16T14:57:36.991 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:37.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:37.244 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:57:37.244 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:57:37.244 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.1 up 2024-01-16T14:57:37.244 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T14:57:37.244 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:57:37.409 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-16T14:57:37.409 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:37.409 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:37.454 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-16T14:57:37.454 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:57:37.454 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:37.546 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:57:37.637 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:37.692 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:57:37.692 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:57:37.692 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:57:37.692 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:57:37.692 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:57:37.692 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:57:37.692 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:37.693 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:37.694 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:37.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:37.695 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:57:37.748 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T14:57:37.748 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-01-16T14:57:37.748 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-16T14:57:37.755 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:57:37.831 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:57:37.872 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:57:37.872 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:57:37.872 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:57:37.872 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:57:37.872 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:57:37.872 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:57:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:37.874 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:57:37.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:37.875 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:57:37.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi188.stderr:2024-01-16T14:57:37.962+0000 7fd4f8036480 -1 init, newargv = 0x55ff1f0b3af0 newargc=15 2024-01-16T14:57:37.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi188.stderr:ceph-fuse[41878]: starting ceph client 2024-01-16T14:57:37.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi188.stderr:ceph-fuse[41878]: starting fuse 2024-01-16T14:57:37.961 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T14:57:37.961 INFO:teuthology.orchestra.run.smithi188.stdout:42 2024-01-16T14:57:37.961 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 42] 2024-01-16T14:57:37.961 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:57:37.961 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T14:57:37.961 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T14:57:37.961 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:57:37.962 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:57:37.963 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T14:57:37.963 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T14:57:37.963 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T14:57:37.963 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:57:37.963 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:57:37.963 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.3")) 2024-01-16T14:57:37.963 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T14:57:38.153 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.3.41878.asok 2024-01-16T14:57:38.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:57:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:57:38.170 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.41878.asok 2024-01-16T14:57:38.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.170 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.41878.asok status 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "3", 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "41878", 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T14:57:38.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 4525, 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 4525 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 2919187073 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T14:57:38.403 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 2919187073 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.4525 192.168.0.2:0/2919187073", 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.2:0/2919187073", 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 8, 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 22, 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T14:57:38.404 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T14:57:38.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.416 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T14:57:38.416 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:38.475 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T14:57:38.475 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:38.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.475 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:38.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.578 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T14:57:38.578 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:38.626 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T14:57:38.626 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:38.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.626 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-16T14:57:38.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.727 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T14:57:38.727 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:38.786 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T14:57:38.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:38.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.786 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:38.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.880 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T14:57:38.880 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:38.934 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T14:57:38.935 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:38.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:38.935 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-16T14:57:39.027 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-16T14:57:39.042 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:39.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.278+0000 7ff217e00640 1 Processor -- start 2024-01-16T14:57:39.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.278+0000 7ff217e00640 1 -- start start 2024-01-16T14:57:39.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.279+0000 7ff217e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff218143c40 0x7ff218071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.279+0000 7ff217e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff218070e50 con 0x7ff2180717d0 2024-01-16T14:57:39.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff217e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff218144020 con 0x7ff2180ccc10 2024-01-16T14:57:39.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff217e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2180723c0 con 0x7ff218143c40 2024-01-16T14:57:39.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff20de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff2180717d0 0x7ff2180cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42788/0 (socket says 172.21.15.141:42788) 2024-01-16T14:57:39.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff20de00640 1 -- 172.21.15.141:0/2617090000 learned_addr learned my addr 172.21.15.141:0/2617090000 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:39.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff216a00640 1 --2- 172.21.15.141:0/2617090000 >> v2:172.21.15.141:3300/0 conn(0x7ff218143c40 0x7ff218071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff216000640 1 -- 172.21.15.141:0/2617090000 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144914355 0 0) 0x7ff218070e50 con 0x7ff2180717d0 2024-01-16T14:57:39.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff216000640 1 -- 172.21.15.141:0/2617090000 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f8003520 con 0x7ff2180717d0 2024-01-16T14:57:39.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff216a00640 1 -- 172.21.15.141:0/2617090000 >> v1:172.21.15.188:6789/0 conn(0x7ff2180ccc10 legacy=0x7ff21806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:39.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.280+0000 7ff216a00640 1 -- 172.21.15.141:0/2617090000 >> v1:172.21.15.141:6789/0 conn(0x7ff2180717d0 legacy=0x7ff2180cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.281+0000 7ff216a00640 1 -- 172.21.15.141:0/2617090000 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218071e00 con 0x7ff218143c40 2024-01-16T14:57:39.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.282+0000 7ff216a00640 1 --2- 172.21.15.141:0/2617090000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff218143c40 0x7ff218071290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff2080097a0 tx=0x7ff20802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ac5840befd2be3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.282+0000 7ff216000640 1 -- 172.21.15.141:0/2617090000 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20802e070 con 0x7ff218143c40 2024-01-16T14:57:39.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.282+0000 7ff216000640 1 -- 172.21.15.141:0/2617090000 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2080315a0 con 0x7ff218143c40 2024-01-16T14:57:39.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.282+0000 7ff216000640 1 -- 172.21.15.141:0/2617090000 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208031930 con 0x7ff218143c40 2024-01-16T14:57:39.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.282+0000 7ff217e00640 1 -- 172.21.15.141:0/2617090000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff218143c40 msgr2=0x7ff218071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.282+0000 7ff217e00640 1 --2- 172.21.15.141:0/2617090000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff218143c40 0x7ff218071290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff2080097a0 tx=0x7ff20802b840 comp rx=0 tx=0).stop 2024-01-16T14:57:39.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.283+0000 7ff217e00640 1 -- 172.21.15.141:0/2617090000 shutdown_connections 2024-01-16T14:57:39.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.283+0000 7ff217e00640 1 --2- 172.21.15.141:0/2617090000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff218143c40 0x7ff218071290 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.283+0000 7ff217e00640 1 -- 172.21.15.141:0/2617090000 >> 172.21.15.141:0/2617090000 conn(0x7ff21806a800 msgr2=0x7ff21806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:39.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.283+0000 7ff217e00640 1 -- 172.21.15.141:0/2617090000 shutdown_connections 2024-01-16T14:57:39.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.283+0000 7ff217e00640 1 -- 172.21.15.141:0/2617090000 wait complete. 2024-01-16T14:57:39.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.285+0000 7ff217e00640 1 Processor -- start 2024-01-16T14:57:39.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.285+0000 7ff217e00640 1 -- start start 2024-01-16T14:57:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff217e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2180717d0 0x7ff21815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff217e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2180ccc10 0x7ff2180ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff217e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff218143c40 0x7ff2180ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff217e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff218132280 con 0x7ff2180ccc10 2024-01-16T14:57:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff217e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff218132560 con 0x7ff2180717d0 2024-01-16T14:57:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff217e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff218132840 con 0x7ff218143c40 2024-01-16T14:57:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff20de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2180ccc10 0x7ff2180ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff20de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2180ccc10 0x7ff2180ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55740/0 (socket says 172.21.15.141:55740) 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff20de00640 1 -- 172.21.15.141:0/3427675201 learned_addr learned my addr 172.21.15.141:0/3427675201 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff217400640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff218143c40 0x7ff2180ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff20de00640 1 -- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff218143c40 msgr2=0x7ff2180ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff216a00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2180717d0 0x7ff21815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.286+0000 7ff20de00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff218143c40 0x7ff2180ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff20de00640 1 -- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2180717d0 msgr2=0x7ff21815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff20de00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2180717d0 0x7ff21815a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff20de00640 1 -- 172.21.15.141:0/3427675201 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208009400 con 0x7ff2180ccc10 2024-01-16T14:57:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff20de00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2180ccc10 0x7ff2180ac8d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c410 tx=0x7ff20000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff20f400640 1 -- 172.21.15.141:0/3427675201 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200005ce0 con 0x7ff2180ccc10 2024-01-16T14:57:39.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff20f400640 1 -- 172.21.15.141:0/3427675201 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200005e40 con 0x7ff2180ccc10 2024-01-16T14:57:39.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff20f400640 1 -- 172.21.15.141:0/3427675201 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2000072f0 con 0x7ff2180ccc10 2024-01-16T14:57:39.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff218132b20 con 0x7ff2180ccc10 2024-01-16T14:57:39.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.287+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff218131510 con 0x7ff2180ccc10 2024-01-16T14:57:39.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.288+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff218133040 con 0x7ff2180ccc10 2024-01-16T14:57:39.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.289+0000 7ff20f400640 1 -- 172.21.15.141:0/3427675201 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff200007510 con 0x7ff2180ccc10 2024-01-16T14:57:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.290+0000 7ff20f400640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1f006b2c0 0x7ff1f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.290+0000 7ff20f400640 1 -- 172.21.15.141:0/3427675201 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff200089380 con 0x7ff2180ccc10 2024-01-16T14:57:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.291+0000 7ff216a00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1f006b2c0 0x7ff1f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.292+0000 7ff216a00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1f006b2c0 0x7ff1f006d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff208036000 tx=0x7ff20803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.295+0000 7ff20f400640 1 -- 172.21.15.141:0/3427675201 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff218133040 con 0x7ff2180ccc10 2024-01-16T14:57:39.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.491+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff218133040 con 0x7ff2180ccc10 2024-01-16T14:57:39.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.493+0000 7ff20f400640 1 -- 172.21.15.141:0/3427675201 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7ff218133040 con 0x7ff2180ccc10 2024-01-16T14:57:39.475 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:39.476 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1f006b2c0 msgr2=0x7ff1f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1f006b2c0 0x7ff1f006d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff208036000 tx=0x7ff20803d040 comp rx=0 tx=0).stop 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2180ccc10 msgr2=0x7ff2180ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2180ccc10 0x7ff2180ac8d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c410 tx=0x7ff20000c8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 shutdown_connections 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1f006b2c0 0x7ff1f006d780 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff218143c40 0x7ff2180ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2180ccc10 0x7ff2180ac8d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 --2- 172.21.15.141:0/3427675201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2180717d0 0x7ff21815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 >> 172.21.15.141:0/3427675201 conn(0x7ff21806a800 msgr2=0x7ff2180adb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:39.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 shutdown_connections 2024-01-16T14:57:39.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.496+0000 7ff217e00640 1 -- 172.21.15.141:0/3427675201 wait complete. 2024-01-16T14:57:39.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:39.498 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:39.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.722+0000 7f01dca00640 1 Processor -- start 2024-01-16T14:57:39.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.722+0000 7f01dca00640 1 -- start start 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01dca00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f01d4150760 0x7f01d4072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01dca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d415d300 0x7f01d415d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01dca00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d415dc80 con 0x7f01d40b2bd0 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01dca00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d40b3770 con 0x7f01d4150760 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01dca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d4071190 con 0x7f01d415d300 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01daa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f01d40b2bd0 0x7f01d40b3060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42798/0 (socket says 172.21.15.141:42798) 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01daa00640 1 -- 172.21.15.141:0/2229523650 learned_addr learned my addr 172.21.15.141:0/2229523650 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:39.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.724+0000 7f01da000640 1 --2- 172.21.15.141:0/2229523650 >> v2:172.21.15.188:3300/0 conn(0x7f01d4150760 0x7f01d4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d9600640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d415d300 0x7f01d415d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865177353 0 0) 0x7f01d415dc80 con 0x7f01d40b2bd0 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b4003520 con 0x7f01d40b2bd0 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2397675291 0 0) 0x7f01b4003520 con 0x7f01d40b2bd0 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d415d300 msgr2=0x7f01d415d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d415d300 0x7f01d415d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 >> v2:172.21.15.188:3300/0 conn(0x7f01d4150760 msgr2=0x7f01d4072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 --2- 172.21.15.141:0/2229523650 >> v2:172.21.15.188:3300/0 conn(0x7f01d4150760 0x7f01d4072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4071470 con 0x7f01d40b2bd0 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f01cc0019f0 con 0x7f01d40b2bd0 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 >> v1:172.21.15.141:6789/0 conn(0x7f01d40b2bd0 legacy=0x7f01d40b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01b40047c0 0x7f01b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d415d300 0x7f01b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40b3770 con 0x7f01b4003970 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d4071190 con 0x7f01b40047c0 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b400b980 con 0x7f01d415d300 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.725+0000 7f01da000640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.726+0000 7f01d9600640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01b40047c0 0x7f01b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.726+0000 7f01da000640 1 -- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d415d300 msgr2=0x7f01b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.726+0000 7f01da000640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d415d300 0x7f01b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.726+0000 7f01da000640 1 -- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01b40047c0 msgr2=0x7f01b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.726+0000 7f01da000640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01b40047c0 0x7f01b4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.726+0000 7f01da000640 1 -- 172.21.15.141:0/2229523650 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4003520 con 0x7f01b4003970 2024-01-16T14:57:39.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.726+0000 7f01d9600640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01b40047c0 0x7f01b4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:39.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.727+0000 7f01da000640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01b4003d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009ac0 tx=0x7f01b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8945e5de72daba0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.727+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c402c6d0 con 0x7f01b4003970 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.727+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c402c830 con 0x7f01b4003970 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.727+0000 7f01d8c00640 1 -- 172.21.15.141:0/2229523650 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c402cbc0 con 0x7f01b4003970 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.728+0000 7f01dca00640 1 -- 172.21.15.141:0/2229523650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 msgr2=0x7f01b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.728+0000 7f01dca00640 1 --2- 172.21.15.141:0/2229523650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01b4003d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009ac0 tx=0x7f01b40057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.728+0000 7f01daa00640 1 -- 172.21.15.141:0/2229523650 reap_dead start 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.728+0000 7f01dca00640 1 -- 172.21.15.141:0/2229523650 shutdown_connections 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.728+0000 7f01dca00640 1 -- 172.21.15.141:0/2229523650 >> 172.21.15.141:0/2229523650 conn(0x7f01d406a800 msgr2=0x7f01d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.728+0000 7f01dca00640 1 -- 172.21.15.141:0/2229523650 shutdown_connections 2024-01-16T14:57:39.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.728+0000 7f01dca00640 1 -- 172.21.15.141:0/2229523650 wait complete. 2024-01-16T14:57:39.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.730+0000 7f01dca00640 1 Processor -- start 2024-01-16T14:57:39.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.730+0000 7f01dca00640 1 -- start start 2024-01-16T14:57:39.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01dca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01d41e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01dca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d414f5c0 0x7f01d414fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01dca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d4158790 0x7f01d4158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01dca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d414ff80 con 0x7f01b4003970 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01dca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d41579d0 con 0x7f01d4158790 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01dca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d4157cb0 con 0x7f01d414f5c0 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01da000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01d41e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01da000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01d41e6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55756/0 (socket says 172.21.15.141:55756) 2024-01-16T14:57:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.731+0000 7f01da000640 1 -- 172.21.15.141:0/1262801879 learned_addr learned my addr 172.21.15.141:0/1262801879 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:39.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01daa00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d4158790 0x7f01d4158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01da000640 1 -- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d414f5c0 msgr2=0x7f01d414fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01d9600640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d414f5c0 0x7f01d414fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01da000640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d414f5c0 0x7f01d414fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01da000640 1 -- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d4158790 msgr2=0x7f01d4158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01da000640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d4158790 0x7f01d4158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01da000640 1 -- 172.21.15.141:0/1262801879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4157f90 con 0x7f01b4003970 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01daa00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d4158790 0x7f01d4158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01da000640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01d41e6fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01c402c2d0 tx=0x7f01c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01d2000640 1 -- 172.21.15.141:0/1262801879 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4003810 con 0x7f01b4003970 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01d2000640 1 -- 172.21.15.141:0/1262801879 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c4003970 con 0x7f01b4003970 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01d2000640 1 -- 172.21.15.141:0/1262801879 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4003d00 con 0x7f01b4003970 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d4158270 con 0x7f01b4003970 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.732+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d41594f0 con 0x7f01b4003970 2024-01-16T14:57:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.734+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d40675d0 con 0x7f01b4003970 2024-01-16T14:57:39.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.734+0000 7f01d2000640 1 -- 172.21.15.141:0/1262801879 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f01c4040450 con 0x7f01b4003970 2024-01-16T14:57:39.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.735+0000 7f01d2000640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f01ac06b2c0 0x7f01ac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:39.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.735+0000 7f01d2000640 1 -- 172.21.15.141:0/1262801879 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01c40ab980 con 0x7f01b4003970 2024-01-16T14:57:39.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.736+0000 7f01d9600640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f01ac06b2c0 0x7f01ac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:39.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.737+0000 7f01d9600640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f01ac06b2c0 0x7f01ac06d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f01d4068110 tx=0x7f01bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:39.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.739+0000 7f01d2000640 1 -- 172.21.15.141:0/1262801879 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01c4076880 con 0x7f01b4003970 2024-01-16T14:57:39.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.927+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01d40703c0 con 0x7f01b4003970 2024-01-16T14:57:39.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.928+0000 7f01d2000640 1 -- 172.21.15.141:0/1262801879 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f01c4044020 con 0x7f01b4003970 2024-01-16T14:57:39.909 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:39.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f01ac06b2c0 msgr2=0x7f01ac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f01ac06b2c0 0x7f01ac06d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f01d4068110 tx=0x7f01bc008040 comp rx=0 tx=0).stop 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 msgr2=0x7f01d41e6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01d41e6fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01c402c2d0 tx=0x7f01c4004730 comp rx=0 tx=0).stop 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 shutdown_connections 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f01ac06b2c0 0x7f01ac06d780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d4158790 0x7f01d4158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01d414f5c0 0x7f01d414fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 --2- 172.21.15.141:0/1262801879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01b4003970 0x7f01d41e6fe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.930+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 >> 172.21.15.141:0/1262801879 conn(0x7f01d406a800 msgr2=0x7f01d40b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.931+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 shutdown_connections 2024-01-16T14:57:39.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:39.931+0000 7f01dca00640 1 -- 172.21.15.141:0/1262801879 wait complete. 2024-01-16T14:57:39.930 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:40.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.144+0000 7fd945000640 1 Processor -- start 2024-01-16T14:57:40.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.144+0000 7fd945000640 1 -- start start 2024-01-16T14:57:40.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.145+0000 7fd945000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9400cf9a0 0x7fd94015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.145+0000 7fd945000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd940070c40 con 0x7fd9400b3ae0 2024-01-16T14:57:40.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.145+0000 7fd945000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94006c470 con 0x7fd9400ae5a0 2024-01-16T14:57:40.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.145+0000 7fd945000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd940071450 con 0x7fd9400cf9a0 2024-01-16T14:57:40.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.146+0000 7fd93d400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd9400b3ae0 0x7fd9400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42808/0 (socket says 172.21.15.141:42808) 2024-01-16T14:57:40.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.146+0000 7fd93d400640 1 -- 172.21.15.141:0/2372546414 learned_addr learned my addr 172.21.15.141:0/2372546414 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.146+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657631621 0 0) 0x7fd940070c40 con 0x7fd9400b3ae0 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.146+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd920003520 con 0x7fd9400b3ae0 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.146+0000 7fd93e800640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9400cf9a0 0x7fd94015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165207367 0 0) 0x7fd920003520 con 0x7fd9400b3ae0 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9400cf9a0 msgr2=0x7fd94015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9400cf9a0 0x7fd94015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 >> v1:172.21.15.188:6789/0 conn(0x7fd9400ae5a0 legacy=0x7fd94006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94006c750 con 0x7fd9400b3ae0 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd928001670 con 0x7fd9400b3ae0 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 >> v1:172.21.15.141:6789/0 conn(0x7fd9400b3ae0 legacy=0x7fd9400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9200039a0 0x7fd920003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9200047c0 0x7fd920004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd92000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd940071450 con 0x7fd920005180 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92000c030 con 0x7fd9200039a0 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92000c1f0 con 0x7fd9200047c0 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93e800640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9400cf9a0 0x7fd94015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93e800640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd92000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.147+0000 7fd93de00640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9200039a0 0x7fd920003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93d400640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9200047c0 0x7fd920004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93e800640 1 -- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9200047c0 msgr2=0x7fd920004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93e800640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9200047c0 0x7fd920004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93e800640 1 -- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9200039a0 msgr2=0x7fd920003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93e800640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9200039a0 0x7fd920003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93e800640 1 -- 172.21.15.141:0/2372546414 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920003520 con 0x7fd920005180 2024-01-16T14:57:40.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93de00640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9200039a0 0x7fd920003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:40.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.148+0000 7fd93d400640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9200047c0 0x7fd920004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:40.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.149+0000 7fd93e800640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd92000b980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd938009f00 tx=0x7fd9200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40922dd777ed683 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.149+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93802cc70 con 0x7fd920005180 2024-01-16T14:57:40.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.149+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93802cdd0 con 0x7fd920005180 2024-01-16T14:57:40.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.149+0000 7fd93ca00640 1 -- 172.21.15.141:0/2372546414 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd938003bb0 con 0x7fd920005180 2024-01-16T14:57:40.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.150+0000 7fd945000640 1 -- 172.21.15.141:0/2372546414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 msgr2=0x7fd92000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.150+0000 7fd945000640 1 --2- 172.21.15.141:0/2372546414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd92000b980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd938009f00 tx=0x7fd9200057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.150+0000 7fd93e800640 1 -- 172.21.15.141:0/2372546414 reap_dead start 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.150+0000 7fd945000640 1 -- 172.21.15.141:0/2372546414 shutdown_connections 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.150+0000 7fd945000640 1 -- 172.21.15.141:0/2372546414 >> 172.21.15.141:0/2372546414 conn(0x7fd94006a800 msgr2=0x7fd94006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.150+0000 7fd945000640 1 -- 172.21.15.141:0/2372546414 shutdown_connections 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.150+0000 7fd945000640 1 -- 172.21.15.141:0/2372546414 wait complete. 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.151+0000 7fd945000640 1 Processor -- start 2024-01-16T14:57:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.152+0000 7fd945000640 1 -- start start 2024-01-16T14:57:40.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.152+0000 7fd945000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd94014b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.152+0000 7fd945000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9401580e0 0x7fd9401565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd945000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd940156b30 0x7fd940160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd945000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd940157c50 con 0x7fd920005180 2024-01-16T14:57:40.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd945000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd940157040 con 0x7fd940156b30 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd945000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd940157320 con 0x7fd9401580e0 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd94014b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd94014b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55774/0 (socket says 172.21.15.141:55774) 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 -- 172.21.15.141:0/84844418 learned_addr learned my addr 172.21.15.141:0/84844418 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93d400640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9401580e0 0x7fd9401565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 -- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9401580e0 msgr2=0x7fd9401565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9401580e0 0x7fd9401565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93e800640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd940156b30 0x7fd940160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 -- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd940156b30 msgr2=0x7fd940160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd940156b30 0x7fd940160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 -- 172.21.15.141:0/84844418 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940157600 con 0x7fd920005180 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.153+0000 7fd93de00640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd94014b920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd93000e4d0 tx=0x7fd93000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.154+0000 7fd936000640 1 -- 172.21.15.141:0/84844418 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300053a0 con 0x7fd920005180 2024-01-16T14:57:40.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.154+0000 7fd936000640 1 -- 172.21.15.141:0/84844418 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9300035f0 con 0x7fd920005180 2024-01-16T14:57:40.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.154+0000 7fd936000640 1 -- 172.21.15.141:0/84844418 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930003980 con 0x7fd920005180 2024-01-16T14:57:40.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.154+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9401578e0 con 0x7fd920005180 2024-01-16T14:57:40.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.154+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9400ce410 con 0x7fd920005180 2024-01-16T14:57:40.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.155+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9400675d0 con 0x7fd920005180 2024-01-16T14:57:40.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.156+0000 7fd936000640 1 -- 172.21.15.141:0/84844418 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd930003ae0 con 0x7fd920005180 2024-01-16T14:57:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.157+0000 7fd936000640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd91806b2f0 0x7fd91806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.157+0000 7fd936000640 1 -- 172.21.15.141:0/84844418 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9300898e0 con 0x7fd920005180 2024-01-16T14:57:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.157+0000 7fd93d400640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd91806b2f0 0x7fd91806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.158+0000 7fd93d400640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd91806b2f0 0x7fd91806d7b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd928000f60 tx=0x7fd928009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.161+0000 7fd936000640 1 -- 172.21.15.141:0/84844418 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9300547e0 con 0x7fd920005180 2024-01-16T14:57:40.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.351+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd94015e480 con 0x7fd920005180 2024-01-16T14:57:40.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.352+0000 7fd936000640 1 -- 172.21.15.141:0/84844418 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd930057e00 con 0x7fd920005180 2024-01-16T14:57:40.333 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:40.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.354+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd91806b2f0 msgr2=0x7fd91806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.354+0000 7fd945000640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd91806b2f0 0x7fd91806d7b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd928000f60 tx=0x7fd928009040 comp rx=0 tx=0).stop 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.354+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 msgr2=0x7fd94014b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.354+0000 7fd945000640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd94014b920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd93000e4d0 tx=0x7fd93000e990 comp rx=0 tx=0).stop 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 shutdown_connections 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd91806b2f0 0x7fd91806d7b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd940156b30 0x7fd940160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd9401580e0 0x7fd9401565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 --2- 172.21.15.141:0/84844418 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd920005180 0x7fd94014b920 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 >> 172.21.15.141:0/84844418 conn(0x7fd94006a800 msgr2=0x7fd9400cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 shutdown_connections 2024-01-16T14:57:40.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.355+0000 7fd945000640 1 -- 172.21.15.141:0/84844418 wait complete. 2024-01-16T14:57:40.355 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.575+0000 7f5210a00640 1 Processor -- start 2024-01-16T14:57:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.576+0000 7f5210a00640 1 -- start start 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f5210a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080ae5a0 0x7f520815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f5210a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52080b3ae0 0x7f520806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f5210a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5208070c40 con 0x7f520806d120 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f5210a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5208071450 con 0x7f52080b3ae0 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f5210a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520806c470 con 0x7f52080ae5a0 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f520d600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52080b3ae0 0x7f520806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f520d600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52080b3ae0 0x7f520806cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55790/0 (socket says 172.21.15.141:55790) 2024-01-16T14:57:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.577+0000 7f520d600640 1 -- 172.21.15.141:0/459491716 learned_addr learned my addr 172.21.15.141:0/459491716 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:40.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.578+0000 7f520e000640 1 --2- 172.21.15.141:0/459491716 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080ae5a0 0x7f520815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.578+0000 7f520d600640 1 -- 172.21.15.141:0/459491716 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080ae5a0 msgr2=0x7f520815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.578+0000 7f520d600640 1 --2- 172.21.15.141:0/459491716 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080ae5a0 0x7f520815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.578+0000 7f520d600640 1 -- 172.21.15.141:0/459491716 >> v1:172.21.15.188:6789/0 conn(0x7f520806d120 legacy=0x7f52080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.578+0000 7f520d600640 1 -- 172.21.15.141:0/459491716 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520806c750 con 0x7f52080b3ae0 2024-01-16T14:57:40.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.579+0000 7f520d600640 1 --2- 172.21.15.141:0/459491716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52080b3ae0 0x7f520806cbe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009870 tx=0x7f51f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5b42e04f9dfc00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.579+0000 7f520cc00640 1 -- 172.21.15.141:0/459491716 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0031640 con 0x7f52080b3ae0 2024-01-16T14:57:40.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.579+0000 7f520cc00640 1 -- 172.21.15.141:0/459491716 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f00317a0 con 0x7f52080b3ae0 2024-01-16T14:57:40.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.579+0000 7f520cc00640 1 -- 172.21.15.141:0/459491716 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0031b30 con 0x7f52080b3ae0 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 -- 172.21.15.141:0/459491716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52080b3ae0 msgr2=0x7f520806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 --2- 172.21.15.141:0/459491716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52080b3ae0 0x7f520806cbe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009870 tx=0x7f51f002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 -- 172.21.15.141:0/459491716 shutdown_connections 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 --2- 172.21.15.141:0/459491716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52080b3ae0 0x7f520806cbe0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 --2- 172.21.15.141:0/459491716 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080ae5a0 0x7f520815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 -- 172.21.15.141:0/459491716 >> 172.21.15.141:0/459491716 conn(0x7f520806a800 msgr2=0x7f520806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 -- 172.21.15.141:0/459491716 shutdown_connections 2024-01-16T14:57:40.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.580+0000 7f5210a00640 1 -- 172.21.15.141:0/459491716 wait complete. 2024-01-16T14:57:40.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.582+0000 7f5210a00640 1 Processor -- start 2024-01-16T14:57:40.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.582+0000 7f5210a00640 1 -- start start 2024-01-16T14:57:40.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f5210a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f520806d120 0x7f52080cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f5210a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52080ae5a0 0x7f52080c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f5210a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080b3ae0 0x7f52080ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f5210a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52081602e0 con 0x7f520806d120 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f5210a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520815f6a0 con 0x7f52080ae5a0 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f5210a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520815f940 con 0x7f52080b3ae0 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f520e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f520806d120 0x7f52080cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f520e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f520806d120 0x7f52080cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55798/0 (socket says 172.21.15.141:55798) 2024-01-16T14:57:40.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.583+0000 7f520e000640 1 -- 172.21.15.141:0/2772958697 learned_addr learned my addr 172.21.15.141:0/2772958697 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:40.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f520e000640 1 -- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080b3ae0 msgr2=0x7f52080ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f520d600640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52080ae5a0 0x7f52080c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f520e000640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080b3ae0 0x7f52080ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f520e000640 1 -- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52080ae5a0 msgr2=0x7f52080c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f520e000640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52080ae5a0 0x7f52080c4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f520e000640 1 -- 172.21.15.141:0/2772958697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f00094d0 con 0x7f520806d120 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f520e000640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f520806d120 0x7f52080cc320 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f51f800d800 tx=0x7f51f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f5206000640 1 -- 172.21.15.141:0/2772958697 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800b6c0 con 0x7f520806d120 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f5206000640 1 -- 172.21.15.141:0/2772958697 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f800b820 con 0x7f520806d120 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f5206000640 1 -- 172.21.15.141:0/2772958697 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800bbb0 con 0x7f520806d120 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520815fc20 con 0x7f520806d120 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.584+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52080cc860 con 0x7f520806d120 2024-01-16T14:57:40.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.586+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52080675d0 con 0x7f520806d120 2024-01-16T14:57:40.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.586+0000 7f5206000640 1 -- 172.21.15.141:0/2772958697 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f51f800bd10 con 0x7f520806d120 2024-01-16T14:57:40.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.587+0000 7f5206000640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51e006b2c0 0x7f51e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.587+0000 7f5206000640 1 -- 172.21.15.141:0/2772958697 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51f808a160 con 0x7f520806d120 2024-01-16T14:57:40.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.588+0000 7f520d600640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51e006b2c0 0x7f51e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.589+0000 7f520d600640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51e006b2c0 0x7f51e006d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f51f0036000 tx=0x7f51f003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:40.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.592+0000 7f5206000640 1 -- 172.21.15.141:0/2772958697 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51f8054fc0 con 0x7f520806d120 2024-01-16T14:57:40.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.781+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f520806aec0 con 0x7f520806d120 2024-01-16T14:57:40.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.782+0000 7f5206000640 1 -- 172.21.15.141:0/2772958697 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f51f80585e0 con 0x7f520806d120 2024-01-16T14:57:40.764 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:40.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51e006b2c0 msgr2=0x7f51e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51e006b2c0 0x7f51e006d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f51f0036000 tx=0x7f51f003c040 comp rx=0 tx=0).stop 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f520806d120 msgr2=0x7f52080cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f520806d120 0x7f52080cc320 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f51f800d800 tx=0x7f51f800dcc0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 shutdown_connections 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51e006b2c0 0x7f51e006d780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52080b3ae0 0x7f52080ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52080ae5a0 0x7f52080c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 --2- 172.21.15.141:0/2772958697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f520806d120 0x7f52080cc320 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 >> 172.21.15.141:0/2772958697 conn(0x7f520806a800 msgr2=0x7f52080b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 shutdown_connections 2024-01-16T14:57:40.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:40.785+0000 7f5210a00640 1 -- 172.21.15.141:0/2772958697 wait complete. 2024-01-16T14:57:40.785 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:40.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.014+0000 7f8511000640 1 Processor -- start 2024-01-16T14:57:40.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.014+0000 7f8511000640 1 -- start start 2024-01-16T14:57:40.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.015+0000 7f8511000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c159590 0x7f850c070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.016+0000 7f8511000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f850c06d050 0x7f850c06d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:40.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.016+0000 7f8511000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c06d9a0 con 0x7f850c0b3c20 2024-01-16T14:57:40.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.016+0000 7f8511000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c071790 con 0x7f850c06d050 2024-01-16T14:57:40.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.016+0000 7f8511000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c071100 con 0x7f850c159590 2024-01-16T14:57:40.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.016+0000 7f850b400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c159590 0x7f850c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.016+0000 7f850be00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f850c06d050 0x7f850c06d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:40.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.016+0000 7f850b400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c159590 0x7f850c070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:57750/0 (socket says 172.21.15.141:57750) 2024-01-16T14:57:40.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.017+0000 7f850b400640 1 -- 172.21.15.141:0/1172465901 learned_addr learned my addr 172.21.15.141:0/1172465901 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:40.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.017+0000 7f850be00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f850c06d050 0x7f850c06d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:34998/0 (socket says 172.21.15.141:34998) 2024-01-16T14:57:40.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.017+0000 7f850b400640 1 -- 172.21.15.141:0/1172465901 >> v2:172.21.15.188:3300/0 conn(0x7f850c06d050 msgr2=0x7f850c06d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:40.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.017+0000 7f850b400640 1 --2- 172.21.15.141:0/1172465901 >> v2:172.21.15.188:3300/0 conn(0x7f850c06d050 0x7f850c06d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:40.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.017+0000 7f850b400640 1 -- 172.21.15.141:0/1172465901 >> v1:172.21.15.188:6789/0 conn(0x7f850c0b3c20 legacy=0x7f850c0ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:40.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.017+0000 7f850b400640 1 -- 172.21.15.141:0/1172465901 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c070860 con 0x7f850c159590 2024-01-16T14:57:41.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.018+0000 7f850b400640 1 --2- 172.21.15.141:0/1172465901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c159590 0x7f850c070160 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009870 tx=0x7f84fc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b5cb587b9b22391 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.019+0000 7f850aa00640 1 -- 172.21.15.141:0/1172465901 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc031640 con 0x7f850c159590 2024-01-16T14:57:41.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.019+0000 7f850aa00640 1 -- 172.21.15.141:0/1172465901 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc0317a0 con 0x7f850c159590 2024-01-16T14:57:41.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.019+0000 7f850aa00640 1 -- 172.21.15.141:0/1172465901 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc031b30 con 0x7f850c159590 2024-01-16T14:57:41.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 -- 172.21.15.141:0/1172465901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c159590 msgr2=0x7f850c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 --2- 172.21.15.141:0/1172465901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c159590 0x7f850c070160 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009870 tx=0x7f84fc02b8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 -- 172.21.15.141:0/1172465901 shutdown_connections 2024-01-16T14:57:41.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 --2- 172.21.15.141:0/1172465901 >> v2:172.21.15.188:3300/0 conn(0x7f850c06d050 0x7f850c06d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 --2- 172.21.15.141:0/1172465901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c159590 0x7f850c070160 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 -- 172.21.15.141:0/1172465901 >> 172.21.15.141:0/1172465901 conn(0x7f850c06a800 msgr2=0x7f850c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 -- 172.21.15.141:0/1172465901 shutdown_connections 2024-01-16T14:57:41.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.020+0000 7f8511000640 1 -- 172.21.15.141:0/1172465901 wait complete. 2024-01-16T14:57:41.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.022+0000 7f8511000640 1 Processor -- start 2024-01-16T14:57:41.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.022+0000 7f8511000640 1 -- start start 2024-01-16T14:57:41.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f8511000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c06d050 0x7f850c141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f8511000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f850c0b3c20 0x7f850c1f0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f8511000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f850c159590 0x7f850c140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f8511000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1415a0 con 0x7f850c159590 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f8511000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c140990 con 0x7f850c0b3c20 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f8511000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c140c70 con 0x7f850c06d050 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f850c159590 0x7f850c140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f850c159590 0x7f850c140090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55812/0 (socket says 172.21.15.141:55812) 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 -- 172.21.15.141:0/664260545 learned_addr learned my addr 172.21.15.141:0/664260545 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f8502800640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f850c0b3c20 0x7f850c1f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850b400640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c06d050 0x7f850c141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 -- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c06d050 msgr2=0x7f850c141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c06d050 0x7f850c141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 -- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f850c0b3c20 msgr2=0x7f850c1f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f850c0b3c20 0x7f850c1f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.023+0000 7f850be00640 1 -- 172.21.15.141:0/664260545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc0094d0 con 0x7f850c159590 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.024+0000 7f850be00640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f850c159590 0x7f850c140090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f850400e630 tx=0x7f850400eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.024+0000 7f8503e00640 1 -- 172.21.15.141:0/664260545 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850400c990 con 0x7f850c159590 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.024+0000 7f8503e00640 1 -- 172.21.15.141:0/664260545 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850400cb10 con 0x7f850c159590 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.024+0000 7f8503e00640 1 -- 172.21.15.141:0/664260545 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85040113f0 con 0x7f850c159590 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.024+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c140f50 con 0x7f850c159590 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.024+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c1f0ba0 con 0x7f850c159590 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.025+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c141750 con 0x7f850c159590 2024-01-16T14:57:41.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.026+0000 7f8503e00640 1 -- 172.21.15.141:0/664260545 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8504011550 con 0x7f850c159590 2024-01-16T14:57:41.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.027+0000 7f8503e00640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84e406b2c0 0x7f84e406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.027+0000 7f8503e00640 1 -- 172.21.15.141:0/664260545 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85040898c0 con 0x7f850c159590 2024-01-16T14:57:41.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.028+0000 7f850b400640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84e406b2c0 0x7f84e406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.029+0000 7f850b400640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84e406b2c0 0x7f84e406d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84fc036000 tx=0x7f84fc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.031+0000 7f8503e00640 1 -- 172.21.15.141:0/664260545 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f850c141750 con 0x7f850c159590 2024-01-16T14:57:41.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.219+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f850c067450 con 0x7f850c159590 2024-01-16T14:57:41.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.221+0000 7f8503e00640 1 -- 172.21.15.141:0/664260545 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f8504020090 con 0x7f850c159590 2024-01-16T14:57:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:41.204 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84e406b2c0 msgr2=0x7f84e406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84e406b2c0 0x7f84e406d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84fc036000 tx=0x7f84fc03c040 comp rx=0 tx=0).stop 2024-01-16T14:57:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f850c159590 msgr2=0x7f850c140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f850c159590 0x7f850c140090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f850400e630 tx=0x7f850400eaf0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 shutdown_connections 2024-01-16T14:57:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84e406b2c0 0x7f84e406d780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f850c159590 0x7f850c140090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f850c0b3c20 0x7f850c1f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 --2- 172.21.15.141:0/664260545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f850c06d050 0x7f850c141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 >> 172.21.15.141:0/664260545 conn(0x7f850c06a800 msgr2=0x7f850c0cb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 shutdown_connections 2024-01-16T14:57:41.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.224+0000 7f8511000640 1 -- 172.21.15.141:0/664260545 wait complete. 2024-01-16T14:57:41.225 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:41.226 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:41.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.458+0000 7f4cc2a00640 1 Processor -- start 2024-01-16T14:57:41.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.459+0000 7f4cc2a00640 1 -- start start 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cc2a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4cbc15b030 0x7f4cbc0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cc2a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4cbc0b31b0 0x7f4cbc0b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cc2a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc0b3610 con 0x7f4cbc0b5380 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cc2a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc0d0780 con 0x7f4cbc15b030 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cc2a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc0cf8f0 con 0x7f4cbc0b31b0 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cbbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4cbc15b030 0x7f4cbc0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cbbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4cbc15b030 0x7f4cbc0b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55816/0 (socket says 172.21.15.141:55816) 2024-01-16T14:57:41.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.460+0000 7f4cbbe00640 1 -- 172.21.15.141:0/1634490562 learned_addr learned my addr 172.21.15.141:0/1634490562 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:41.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.461+0000 7f4cbb400640 1 --2- 172.21.15.141:0/1634490562 >> v2:172.21.15.188:3300/0 conn(0x7f4cbc0b31b0 0x7f4cbc0b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.461+0000 7f4cbbe00640 1 -- 172.21.15.141:0/1634490562 >> v2:172.21.15.188:3300/0 conn(0x7f4cbc0b31b0 msgr2=0x7f4cbc0b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.461+0000 7f4cbbe00640 1 --2- 172.21.15.141:0/1634490562 >> v2:172.21.15.188:3300/0 conn(0x7f4cbc0b31b0 0x7f4cbc0b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.461+0000 7f4cbbe00640 1 -- 172.21.15.141:0/1634490562 >> v1:172.21.15.188:6789/0 conn(0x7f4cbc0b5380 legacy=0x7f4cbc15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.461+0000 7f4cbbe00640 1 -- 172.21.15.141:0/1634490562 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0b5810 con 0x7f4cbc15b030 2024-01-16T14:57:41.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.462+0000 7f4cbbe00640 1 --2- 172.21.15.141:0/1634490562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc0b2c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0b5c40 tx=0x7f4cac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7db05e4b97b58c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.462+0000 7f4cbaa00640 1 -- 172.21.15.141:0/1634490562 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac031870 con 0x7f4cbc15b030 2024-01-16T14:57:41.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.462+0000 7f4cbaa00640 1 -- 172.21.15.141:0/1634490562 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac0319d0 con 0x7f4cbc15b030 2024-01-16T14:57:41.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.462+0000 7f4cbaa00640 1 -- 172.21.15.141:0/1634490562 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac031d60 con 0x7f4cbc15b030 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 -- 172.21.15.141:0/1634490562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 msgr2=0x7f4cbc0b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/1634490562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc0b2c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0b5c40 tx=0x7f4cac02b910 comp rx=0 tx=0).stop 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 -- 172.21.15.141:0/1634490562 shutdown_connections 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/1634490562 >> v2:172.21.15.188:3300/0 conn(0x7f4cbc0b31b0 0x7f4cbc0b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/1634490562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc0b2c70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 -- 172.21.15.141:0/1634490562 >> 172.21.15.141:0/1634490562 conn(0x7f4cbc06a3e0 msgr2=0x7f4cbc06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 -- 172.21.15.141:0/1634490562 shutdown_connections 2024-01-16T14:57:41.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.463+0000 7f4cc2a00640 1 -- 172.21.15.141:0/1634490562 wait complete. 2024-01-16T14:57:41.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.465+0000 7f4cc2a00640 1 Processor -- start 2024-01-16T14:57:41.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.465+0000 7f4cc2a00640 1 -- start start 2024-01-16T14:57:41.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc2a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cbc0b31b0 0x7f4cbc151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4cbc0b5380 0x7f4cbc141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc2a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc14dfc0 con 0x7f4cbc15b030 2024-01-16T14:57:41.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc2a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1f4890 con 0x7f4cbc0b31b0 2024-01-16T14:57:41.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc2a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1f4b70 con 0x7f4cbc0b5380 2024-01-16T14:57:41.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc0a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc0a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc154330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55830/0 (socket says 172.21.15.141:55830) 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cc0a00640 1 -- 172.21.15.141:0/207018448 learned_addr learned my addr 172.21.15.141:0/207018448 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cbbe00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cbc0b31b0 0x7f4cbc151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.466+0000 7f4cbb400640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4cbc0b5380 0x7f4cbc141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc0a00640 1 -- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4cbc0b5380 msgr2=0x7f4cbc141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc0a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4cbc0b5380 0x7f4cbc141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc0a00640 1 -- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cbc0b31b0 msgr2=0x7f4cbc151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc0a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cbc0b31b0 0x7f4cbc151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc0a00640 1 -- 172.21.15.141:0/207018448 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0094d0 con 0x7f4cbc15b030 2024-01-16T14:57:41.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc0a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc154330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c390 tx=0x7f4cb400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cb3e00640 1 -- 172.21.15.141:0/207018448 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40035f0 con 0x7f4cbc15b030 2024-01-16T14:57:41.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cb3e00640 1 -- 172.21.15.141:0/207018448 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4003750 con 0x7f4cbc15b030 2024-01-16T14:57:41.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cb3e00640 1 -- 172.21.15.141:0/207018448 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4003b00 con 0x7f4cbc15b030 2024-01-16T14:57:41.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc1f4e50 con 0x7f4cbc15b030 2024-01-16T14:57:41.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.467+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc141f70 con 0x7f4cbc15b030 2024-01-16T14:57:41.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.468+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc0703e0 con 0x7f4cbc15b030 2024-01-16T14:57:41.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.469+0000 7f4cb3e00640 1 -- 172.21.15.141:0/207018448 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4cb4003c60 con 0x7f4cbc15b030 2024-01-16T14:57:41.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.470+0000 7f4cb3e00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c9406b2c0 0x7f4c9406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.470+0000 7f4cb3e00640 1 -- 172.21.15.141:0/207018448 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cb4088dc0 con 0x7f4cbc15b030 2024-01-16T14:57:41.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.471+0000 7f4cbbe00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c9406b2c0 0x7f4c9406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.472+0000 7f4cbbe00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c9406b2c0 0x7f4c9406d780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cac036000 tx=0x7f4cac03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.475+0000 7f4cb3e00640 1 -- 172.21.15.141:0/207018448 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cb4053cc0 con 0x7f4cbc15b030 2024-01-16T14:57:41.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.651+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cbc14e170 con 0x7f4cbc15b030 2024-01-16T14:57:41.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.652+0000 7f4cb3e00640 1 -- 172.21.15.141:0/207018448 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4cbc14e170 con 0x7f4cbc15b030 2024-01-16T14:57:41.634 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:41.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c9406b2c0 msgr2=0x7f4c9406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c9406b2c0 0x7f4c9406d780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cac036000 tx=0x7f4cac03c040 comp rx=0 tx=0).stop 2024-01-16T14:57:41.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 msgr2=0x7f4cbc154330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc154330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c390 tx=0x7f4cb400c850 comp rx=0 tx=0).stop 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 shutdown_connections 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c9406b2c0 0x7f4c9406d780 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4cbc15b030 0x7f4cbc154330 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4cbc0b5380 0x7f4cbc141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 --2- 172.21.15.141:0/207018448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cbc0b31b0 0x7f4cbc151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 >> 172.21.15.141:0/207018448 conn(0x7f4cbc06a3e0 msgr2=0x7f4cbc155f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 shutdown_connections 2024-01-16T14:57:41.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.655+0000 7f4cc2a00640 1 -- 172.21.15.141:0/207018448 wait complete. 2024-01-16T14:57:41.654 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:41.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.870+0000 7ff950200640 1 Processor -- start 2024-01-16T14:57:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.870+0000 7ff950200640 1 -- start start 2024-01-16T14:57:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.871+0000 7ff950200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff94813df80 0x7ff94815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.871+0000 7ff950200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff94813e360 0x7ff9480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff950200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94806c470 con 0x7ff94815f9c0 2024-01-16T14:57:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff950200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94806c750 con 0x7ff94813e360 2024-01-16T14:57:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff950200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94806ca30 con 0x7ff94813df80 2024-01-16T14:57:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff94e200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff94815f9c0 0x7ff9481408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42820/0 (socket says 172.21.15.141:42820) 2024-01-16T14:57:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff94e200640 1 -- 172.21.15.141:0/3498757171 learned_addr learned my addr 172.21.15.141:0/3498757171 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff94ce00640 1 --2- 172.21.15.141:0/3498757171 >> v2:172.21.15.188:3300/0 conn(0x7ff94813e360 0x7ff9480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff94d800640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff94813df80 0x7ff94815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564288735 0 0) 0x7ff94806c470 con 0x7ff94815f9c0 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.872+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff928003520 con 0x7ff94815f9c0 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343554820 0 0) 0x7ff928003520 con 0x7ff94815f9c0 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff94813df80 msgr2=0x7ff94815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff94813df80 0x7ff94815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 >> v2:172.21.15.188:3300/0 conn(0x7ff94813e360 msgr2=0x7ff9480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 --2- 172.21.15.141:0/3498757171 >> v2:172.21.15.188:3300/0 conn(0x7ff94813e360 0x7ff9480cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94806cd10 con 0x7ff94815f9c0 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ff9400019f0 con 0x7ff94815f9c0 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 >> v1:172.21.15.141:6789/0 conn(0x7ff94815f9c0 legacy=0x7ff9481408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff928003970 0x7ff928003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff94813df80 0x7ff9280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff928004d00 0x7ff928005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94806c750 con 0x7ff928004d00 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94806ca30 con 0x7ff928003970 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92800b980 con 0x7ff94813df80 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.873+0000 7ff94e200640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff928004d00 0x7ff928005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.874+0000 7ff94e200640 1 -- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff94813df80 msgr2=0x7ff9280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.874+0000 7ff94e200640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff94813df80 0x7ff9280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.874+0000 7ff94e200640 1 -- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff928003970 msgr2=0x7ff928003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.874+0000 7ff94d800640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff928003970 0x7ff928003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.874+0000 7ff94e200640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff928003970 0x7ff928003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.874+0000 7ff94e200640 1 -- 172.21.15.141:0/3498757171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff928003520 con 0x7ff928004d00 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.875+0000 7ff94e200640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff928004d00 0x7ff928005180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a790 tx=0x7ff9280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be514cdc1dd7dda6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.875+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940034630 con 0x7ff928004d00 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.875+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94002f070 con 0x7ff928004d00 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.875+0000 7ff947e00640 1 -- 172.21.15.141:0/3498757171 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9400349f0 con 0x7ff928004d00 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.875+0000 7ff950200640 1 -- 172.21.15.141:0/3498757171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff928004d00 msgr2=0x7ff928005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.875+0000 7ff950200640 1 --2- 172.21.15.141:0/3498757171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff928004d00 0x7ff928005180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a790 tx=0x7ff9280057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.875+0000 7ff94e200640 1 -- 172.21.15.141:0/3498757171 reap_dead start 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.876+0000 7ff950200640 1 -- 172.21.15.141:0/3498757171 shutdown_connections 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.876+0000 7ff950200640 1 -- 172.21.15.141:0/3498757171 >> 172.21.15.141:0/3498757171 conn(0x7ff94806a800 msgr2=0x7ff94806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.876+0000 7ff950200640 1 -- 172.21.15.141:0/3498757171 shutdown_connections 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.876+0000 7ff950200640 1 -- 172.21.15.141:0/3498757171 wait complete. 2024-01-16T14:57:41.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.877+0000 7ff950200640 1 Processor -- start 2024-01-16T14:57:41.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.877+0000 7ff950200640 1 -- start start 2024-01-16T14:57:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.878+0000 7ff950200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff928004d00 0x7ff94813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.878+0000 7ff950200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9480afab0 0x7ff9480aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.878+0000 7ff950200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff94814f350 0x7ff94814f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.878+0000 7ff950200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9480b0450 con 0x7ff9480afab0 2024-01-16T14:57:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.878+0000 7ff950200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94814cee0 con 0x7ff94814f350 2024-01-16T14:57:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.878+0000 7ff950200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94813dda0 con 0x7ff928004d00 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9480afab0 0x7ff9480aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9480afab0 0x7ff9480aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55850/0 (socket says 172.21.15.141:55850) 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 -- 172.21.15.141:0/352143125 learned_addr learned my addr 172.21.15.141:0/352143125 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94e200640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff94814f350 0x7ff94814f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 -- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff928004d00 msgr2=0x7ff94813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff928004d00 0x7ff94813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 -- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff94814f350 msgr2=0x7ff94814f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff94814f350 0x7ff94814f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 -- 172.21.15.141:0/352143125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94814d150 con 0x7ff9480afab0 2024-01-16T14:57:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.879+0000 7ff94ce00640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9480afab0 0x7ff9480aff10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e7f0 tx=0x7ff93000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.880+0000 7ff945600640 1 -- 172.21.15.141:0/352143125 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93000c940 con 0x7ff9480afab0 2024-01-16T14:57:41.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.880+0000 7ff945600640 1 -- 172.21.15.141:0/352143125 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93000caa0 con 0x7ff9480afab0 2024-01-16T14:57:41.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.880+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94814d430 con 0x7ff9480afab0 2024-01-16T14:57:41.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.880+0000 7ff945600640 1 -- 172.21.15.141:0/352143125 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93000ce50 con 0x7ff9480afab0 2024-01-16T14:57:41.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.880+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94814d9f0 con 0x7ff9480afab0 2024-01-16T14:57:41.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.881+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9480675d0 con 0x7ff9480afab0 2024-01-16T14:57:41.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.882+0000 7ff945600640 1 -- 172.21.15.141:0/352143125 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff930013470 con 0x7ff9480afab0 2024-01-16T14:57:41.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.883+0000 7ff945600640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff92006b2c0 0x7ff92006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:41.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.883+0000 7ff945600640 1 -- 172.21.15.141:0/352143125 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff930089d10 con 0x7ff9480afab0 2024-01-16T14:57:41.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.884+0000 7ff94d800640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff92006b2c0 0x7ff92006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:41.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.885+0000 7ff94d800640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff92006b2c0 0x7ff92006d780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff938009950 tx=0x7ff938008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:41.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:41.887+0000 7ff945600640 1 -- 172.21.15.141:0/352143125 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff930054af0 con 0x7ff9480afab0 2024-01-16T14:57:42.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.077+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff94806df00 con 0x7ff9480afab0 2024-01-16T14:57:42.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.078+0000 7ff945600640 1 -- 172.21.15.141:0/352143125 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff930021020 con 0x7ff9480afab0 2024-01-16T14:57:42.060 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.081+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff92006b2c0 msgr2=0x7ff92006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff92006b2c0 0x7ff92006d780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff938009950 tx=0x7ff938008040 comp rx=0 tx=0).stop 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9480afab0 msgr2=0x7ff9480aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9480afab0 0x7ff9480aff10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e7f0 tx=0x7ff93000ecb0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 shutdown_connections 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff92006b2c0 0x7ff92006d780 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff94814f350 0x7ff94814f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9480afab0 0x7ff9480aff10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 --2- 172.21.15.141:0/352143125 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff928004d00 0x7ff94813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 >> 172.21.15.141:0/352143125 conn(0x7ff94806a800 msgr2=0x7ff94806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 shutdown_connections 2024-01-16T14:57:42.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.082+0000 7ff950200640 1 -- 172.21.15.141:0/352143125 wait complete. 2024-01-16T14:57:42.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:42.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.305+0000 7f0491c00640 1 Processor -- start 2024-01-16T14:57:42.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.305+0000 7f0491c00640 1 -- start start 2024-01-16T14:57:42.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f0491c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f048c13df80 0x7f048c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f0491c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c13e360 0x7f048c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f0491c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f048c15f9c0 0x7f048c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f0491c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f048c06c470 con 0x7f048c13df80 2024-01-16T14:57:42.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f0491c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f048c06c750 con 0x7f048c15f9c0 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f0491c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048c06ca30 con 0x7f048c13e360 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f048aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f048c13df80 0x7f048c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f048aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f048c13df80 0x7f048c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55860/0 (socket says 172.21.15.141:55860) 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.307+0000 7f048aa00640 1 -- 172.21.15.141:0/481240947 learned_addr learned my addr 172.21.15.141:0/481240947 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f048b400640 1 --2- 172.21.15.141:0/481240947 >> v2:172.21.15.188:3300/0 conn(0x7f048c15f9c0 0x7f048c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f0483e00640 1 --2- 172.21.15.141:0/481240947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c13e360 0x7f048c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f048aa00640 1 -- 172.21.15.141:0/481240947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c13e360 msgr2=0x7f048c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f048aa00640 1 --2- 172.21.15.141:0/481240947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c13e360 0x7f048c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f048aa00640 1 -- 172.21.15.141:0/481240947 >> v2:172.21.15.188:3300/0 conn(0x7f048c15f9c0 msgr2=0x7f048c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f048aa00640 1 --2- 172.21.15.141:0/481240947 >> v2:172.21.15.188:3300/0 conn(0x7f048c15f9c0 0x7f048c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f048aa00640 1 -- 172.21.15.141:0/481240947 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c06cd10 con 0x7f048c13df80 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f048b400640 1 --2- 172.21.15.141:0/481240947 >> v2:172.21.15.188:3300/0 conn(0x7f048c15f9c0 0x7f048c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:42.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.308+0000 7f0483e00640 1 --2- 172.21.15.141:0/481240947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c13e360 0x7f048c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.309+0000 7f048aa00640 1 --2- 172.21.15.141:0/481240947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13df80 0x7f048c15bee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f048c06e6d0 tx=0x7f047c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95506136cafda39a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.309+0000 7f048a000640 1 -- 172.21.15.141:0/481240947 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c031880 con 0x7f048c13df80 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.309+0000 7f048a000640 1 -- 172.21.15.141:0/481240947 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c0319e0 con 0x7f048c13df80 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.309+0000 7f048a000640 1 -- 172.21.15.141:0/481240947 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c031d70 con 0x7f048c13df80 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 -- 172.21.15.141:0/481240947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13df80 msgr2=0x7f048c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 --2- 172.21.15.141:0/481240947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13df80 0x7f048c15bee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f048c06e6d0 tx=0x7f047c02b920 comp rx=0 tx=0).stop 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 -- 172.21.15.141:0/481240947 shutdown_connections 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 --2- 172.21.15.141:0/481240947 >> v2:172.21.15.188:3300/0 conn(0x7f048c15f9c0 0x7f048c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 --2- 172.21.15.141:0/481240947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c13e360 0x7f048c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 --2- 172.21.15.141:0/481240947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13df80 0x7f048c15bee0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 -- 172.21.15.141:0/481240947 >> 172.21.15.141:0/481240947 conn(0x7f048c06a800 msgr2=0x7f048c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 -- 172.21.15.141:0/481240947 shutdown_connections 2024-01-16T14:57:42.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.310+0000 7f0491c00640 1 -- 172.21.15.141:0/481240947 wait complete. 2024-01-16T14:57:42.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.312+0000 7f0491c00640 1 Processor -- start 2024-01-16T14:57:42.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.312+0000 7f0491c00640 1 -- start start 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0491c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f048c13df80 0x7f048c154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0491c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13e360 0x7f048c0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0491c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c15f9c0 0x7f048c157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0491c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c156e70 con 0x7f048c13e360 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0491c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c157150 con 0x7f048c13df80 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0491c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048c157430 con 0x7f048c15f9c0 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0483e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13e360 0x7f048c0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0483e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13e360 0x7f048c0adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55866/0 (socket says 172.21.15.141:55866) 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.313+0000 7f0483e00640 1 -- 172.21.15.141:0/2695789247 learned_addr learned my addr 172.21.15.141:0/2695789247 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f048aa00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f048c13df80 0x7f048c154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f048b400640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c15f9c0 0x7f048c157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0483e00640 1 -- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c15f9c0 msgr2=0x7f048c157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0483e00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c15f9c0 0x7f048c157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0483e00640 1 -- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f048c13df80 msgr2=0x7f048c154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0483e00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f048c13df80 0x7f048c154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0483e00640 1 -- 172.21.15.141:0/2695789247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c0094d0 con 0x7f048c13e360 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f048b400640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c15f9c0 0x7f048c157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0483e00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13e360 0x7f048c0adb20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f047400c7f0 tx=0x7f047400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0482a00640 1 -- 172.21.15.141:0/2695789247 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474007a30 con 0x7f048c13e360 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0482a00640 1 -- 172.21.15.141:0/2695789247 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0474007b90 con 0x7f048c13e360 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0482a00640 1 -- 172.21.15.141:0/2695789247 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047400ce40 con 0x7f048c13e360 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.314+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c157710 con 0x7f048c13e360 2024-01-16T14:57:42.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.315+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c0b3d60 con 0x7f048c13e360 2024-01-16T14:57:42.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.316+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048c1408b0 con 0x7f048c13e360 2024-01-16T14:57:42.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.317+0000 7f0482a00640 1 -- 172.21.15.141:0/2695789247 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f04740113f0 con 0x7f048c13e360 2024-01-16T14:57:42.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.317+0000 7f0482a00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f046406b2c0 0x7f046406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.318+0000 7f0482a00640 1 -- 172.21.15.141:0/2695789247 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0474089e10 con 0x7f048c13e360 2024-01-16T14:57:42.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.318+0000 7f048aa00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f046406b2c0 0x7f046406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.319+0000 7f048aa00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f046406b2c0 0x7f046406d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f047c036000 tx=0x7f047c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.322+0000 7f0482a00640 1 -- 172.21.15.141:0/2695789247 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f048c1408b0 con 0x7f048c13e360 2024-01-16T14:57:42.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.512+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f048c157c30 con 0x7f048c13e360 2024-01-16T14:57:42.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.513+0000 7f0482a00640 1 -- 172.21.15.141:0/2695789247 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f048c157c30 con 0x7f048c13e360 2024-01-16T14:57:42.495 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f046406b2c0 msgr2=0x7f046406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f046406b2c0 0x7f046406d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f047c036000 tx=0x7f047c03c040 comp rx=0 tx=0).stop 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13e360 msgr2=0x7f048c0adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13e360 0x7f048c0adb20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f047400c7f0 tx=0x7f047400cbf0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 shutdown_connections 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f046406b2c0 0x7f046406d780 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f048c15f9c0 0x7f048c157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f048c13e360 0x7f048c0adb20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 --2- 172.21.15.141:0/2695789247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f048c13df80 0x7f048c154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 >> 172.21.15.141:0/2695789247 conn(0x7f048c06a800 msgr2=0x7f048c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 shutdown_connections 2024-01-16T14:57:42.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.516+0000 7f0491c00640 1 -- 172.21.15.141:0/2695789247 wait complete. 2024-01-16T14:57:42.516 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:42.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.741+0000 7f02df200640 1 Processor -- start 2024-01-16T14:57:42.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.741+0000 7f02df200640 1 -- start start 2024-01-16T14:57:42.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.743+0000 7f02df200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f02d8143ae0 0x7f02d80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.743+0000 7f02df200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 0x7f02d815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.743+0000 7f02df200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80cfc30 con 0x7f02d815e1e0 2024-01-16T14:57:42.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.743+0000 7f02df200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d80b1ee0 con 0x7f02d8143ae0 2024-01-16T14:57:42.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.743+0000 7f02df200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80b2780 con 0x7f02d80cf7d0 2024-01-16T14:57:42.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.743+0000 7f02dd200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f02d815e1e0 0x7f02d815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42824/0 (socket says 172.21.15.141:42824) 2024-01-16T14:57:42.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.743+0000 7f02dd200640 1 -- 172.21.15.141:0/3682364582 learned_addr learned my addr 172.21.15.141:0/3682364582 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:42.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7e00640 1 --2- 172.21.15.141:0/3682364582 >> v2:172.21.15.188:3300/0 conn(0x7f02d8143ae0 0x7f02d80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02cfe00640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 0x7f02d815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888087653 0 0) 0x7f02d80cfc30 con 0x7f02d815e1e0 2024-01-16T14:57:42.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b8003520 con 0x7f02d815e1e0 2024-01-16T14:57:42.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160551809 0 0) 0x7f02b8003520 con 0x7f02d815e1e0 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 msgr2=0x7f02d815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 0x7f02d815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 >> v2:172.21.15.188:3300/0 conn(0x7f02d8143ae0 msgr2=0x7f02d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 --2- 172.21.15.141:0/3682364582 >> v2:172.21.15.188:3300/0 conn(0x7f02d8143ae0 0x7f02d80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d815e990 con 0x7f02d815e1e0 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f02d00019f0 con 0x7f02d815e1e0 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 >> v1:172.21.15.141:6789/0 conn(0x7f02d815e1e0 legacy=0x7f02d815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02b8003970 0x7f02b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 0x7f02b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80b1ee0 con 0x7f02b8004d00 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80b2780 con 0x7f02b8003970 2024-01-16T14:57:42.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.744+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b800b980 con 0x7f02d80cf7d0 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02dd200640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02d7e00640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02b8003970 0x7f02b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02cfe00640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 0x7f02b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02dd200640 1 -- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 msgr2=0x7f02b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02dd200640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 0x7f02b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02dd200640 1 -- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02b8003970 msgr2=0x7f02b8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02dd200640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02b8003970 0x7f02b8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02dd200640 1 -- 172.21.15.141:0/3682364582 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b8003520 con 0x7f02b8004d00 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.745+0000 7f02cfe00640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d80cf7d0 0x7f02b80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.746+0000 7f02dd200640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02b8005180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f02d000a810 tx=0x7f02b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b2a370ed48696d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.746+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0034630 con 0x7f02b8004d00 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.746+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d002f070 con 0x7f02b8004d00 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.746+0000 7f02d7400640 1 -- 172.21.15.141:0/3682364582 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00349f0 con 0x7f02b8004d00 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.747+0000 7f02df200640 1 -- 172.21.15.141:0/3682364582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 msgr2=0x7f02b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.747+0000 7f02df200640 1 --2- 172.21.15.141:0/3682364582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02b8005180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f02d000a810 tx=0x7f02b80057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.747+0000 7f02dd200640 1 -- 172.21.15.141:0/3682364582 reap_dead start 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.747+0000 7f02df200640 1 -- 172.21.15.141:0/3682364582 shutdown_connections 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.747+0000 7f02df200640 1 -- 172.21.15.141:0/3682364582 >> 172.21.15.141:0/3682364582 conn(0x7f02d806a800 msgr2=0x7f02d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.747+0000 7f02df200640 1 -- 172.21.15.141:0/3682364582 shutdown_connections 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.747+0000 7f02df200640 1 -- 172.21.15.141:0/3682364582 wait complete. 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.749+0000 7f02df200640 1 Processor -- start 2024-01-16T14:57:42.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.749+0000 7f02df200640 1 -- start start 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02df200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02d815cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02df200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d815d0f0 0x7f02d815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02df200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02d815aa20 0x7f02d815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02df200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d81313e0 con 0x7f02b8004d00 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02df200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8131610 con 0x7f02d815aa20 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02df200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d81318b0 con 0x7f02d815d0f0 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02d815cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02d815cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55876/0 (socket says 172.21.15.141:55876) 2024-01-16T14:57:42.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 -- 172.21.15.141:0/332586994 learned_addr learned my addr 172.21.15.141:0/332586994 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 -- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d815d0f0 msgr2=0x7f02d815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d815d0f0 0x7f02d815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 -- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02d815aa20 msgr2=0x7f02d815aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02d815aa20 0x7f02d815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.750+0000 7f02d7e00640 1 -- 172.21.15.141:0/332586994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8131b90 con 0x7f02b8004d00 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.751+0000 7f02d7e00640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02d815cbb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f02c8011600 tx=0x7f02c8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.751+0000 7f02d4c00640 1 -- 172.21.15.141:0/332586994 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8015870 con 0x7f02b8004d00 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.751+0000 7f02d4c00640 1 -- 172.21.15.141:0/332586994 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c80159d0 con 0x7f02b8004d00 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.751+0000 7f02d4c00640 1 -- 172.21.15.141:0/332586994 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8015d40 con 0x7f02b8004d00 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.751+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d8131e70 con 0x7f02b8004d00 2024-01-16T14:57:42.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.751+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d8159ab0 con 0x7f02b8004d00 2024-01-16T14:57:42.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.752+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d80675d0 con 0x7f02b8004d00 2024-01-16T14:57:42.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.753+0000 7f02d4c00640 1 -- 172.21.15.141:0/332586994 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f02c800f490 con 0x7f02b8004d00 2024-01-16T14:57:42.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.754+0000 7f02d4c00640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f02b006b2f0 0x7f02b006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:42.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.754+0000 7f02d4c00640 1 -- 172.21.15.141:0/332586994 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02c8089ff0 con 0x7f02b8004d00 2024-01-16T14:57:42.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.755+0000 7f02cfe00640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f02b006b2f0 0x7f02b006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:42.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.756+0000 7f02cfe00640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f02b006b2f0 0x7f02b006d7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f02d815a210 tx=0x7f02c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:42.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.759+0000 7f02d4c00640 1 -- 172.21.15.141:0/332586994 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02c80bce90 con 0x7f02b8004d00 2024-01-16T14:57:42.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.954+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02d8139d80 con 0x7f02b8004d00 2024-01-16T14:57:42.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.957+0000 7f02d4c00640 1 -- 172.21.15.141:0/332586994 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f02c8054e70 con 0x7f02b8004d00 2024-01-16T14:57:42.939 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:42.940 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f02b006b2f0 msgr2=0x7f02b006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f02b006b2f0 0x7f02b006d7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f02d815a210 tx=0x7f02c0008040 comp rx=0 tx=0).stop 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 msgr2=0x7f02d815cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02d815cbb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f02c8011600 tx=0x7f02c8011ac0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 shutdown_connections 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f02b006b2f0 0x7f02b006d7b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02d815aa20 0x7f02d815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f02d815d0f0 0x7f02d815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 --2- 172.21.15.141:0/332586994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f02b8004d00 0x7f02d815cbb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:42.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.960+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 >> 172.21.15.141:0/332586994 conn(0x7f02d806a800 msgr2=0x7f02d80afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:42.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.961+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 shutdown_connections 2024-01-16T14:57:42.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:42.961+0000 7f02df200640 1 -- 172.21.15.141:0/332586994 wait complete. 2024-01-16T14:57:42.962 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:42.962 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:43.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.193+0000 7fd50dc00640 1 Processor -- start 2024-01-16T14:57:43.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.193+0000 7fd50dc00640 1 -- start start 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.194+0000 7fd50dc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5081505b0 0x7fd5080b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.194+0000 7fd50dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd50815a690 0x7fd50806cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd50dc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50806d930 con 0x7fd50806d500 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd50dc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5080b1e30 con 0x7fd5081505b0 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd50dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080cdc10 con 0x7fd50815a690 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd507e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5081505b0 0x7fd5080b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd50d200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd50806d500 0x7fd50815d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37228/0 (socket says 172.21.15.141:37228) 2024-01-16T14:57:43.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd507e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5081505b0 0x7fd5080b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56008/0 (socket says 172.21.15.141:56008) 2024-01-16T14:57:43.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd50d200640 1 -- 172.21.15.141:0/1613443555 learned_addr learned my addr 172.21.15.141:0/1613443555 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:43.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd507400640 1 --2- 172.21.15.141:0/1613443555 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd50815a690 0x7fd50806cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd507e00640 1 -- 172.21.15.141:0/1613443555 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd50815a690 msgr2=0x7fd50806cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd507e00640 1 --2- 172.21.15.141:0/1613443555 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd50815a690 0x7fd50806cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd507e00640 1 -- 172.21.15.141:0/1613443555 >> v1:172.21.15.141:6789/0 conn(0x7fd50806d500 legacy=0x7fd50815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd507e00640 1 -- 172.21.15.141:0/1613443555 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080b20a0 con 0x7fd5081505b0 2024-01-16T14:57:43.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.195+0000 7fd506a00640 1 -- 172.21.15.141:0/1613443555 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528353175 0 0) 0x7fd50806d930 con 0x7fd50806d500 2024-01-16T14:57:43.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.196+0000 7fd507e00640 1 --2- 172.21.15.141:0/1613443555 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5081505b0 0x7fd5080b3b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800a940 tx=0x7fd4f802cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813f4ad23f9766d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.197+0000 7fd506a00640 1 -- 172.21.15.141:0/1613443555 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f802db30 con 0x7fd5081505b0 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.197+0000 7fd506a00640 1 -- 172.21.15.141:0/1613443555 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f802dc90 con 0x7fd5081505b0 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.197+0000 7fd506a00640 1 -- 172.21.15.141:0/1613443555 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8035440 con 0x7fd5081505b0 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.197+0000 7fd50dc00640 1 -- 172.21.15.141:0/1613443555 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5081505b0 msgr2=0x7fd5080b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.197+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1613443555 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5081505b0 0x7fd5080b3b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800a940 tx=0x7fd4f802cd80 comp rx=0 tx=0).stop 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.198+0000 7fd50dc00640 1 -- 172.21.15.141:0/1613443555 shutdown_connections 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.198+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1613443555 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd50815a690 0x7fd50806cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.198+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1613443555 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5081505b0 0x7fd5080b3b20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.198+0000 7fd50dc00640 1 -- 172.21.15.141:0/1613443555 >> 172.21.15.141:0/1613443555 conn(0x7fd50806a7d0 msgr2=0x7fd50806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.198+0000 7fd50dc00640 1 -- 172.21.15.141:0/1613443555 shutdown_connections 2024-01-16T14:57:43.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.198+0000 7fd50dc00640 1 -- 172.21.15.141:0/1613443555 wait complete. 2024-01-16T14:57:43.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.199+0000 7fd50dc00640 1 Processor -- start 2024-01-16T14:57:43.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.199+0000 7fd50dc00640 1 -- start start 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.200+0000 7fd50dc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50806d500 0x7fd50814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081505b0 0x7fd5081f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd50815a690 0x7fd5081e62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50dc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5081f6070 con 0x7fd50815a690 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50dc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5081e7050 con 0x7fd50806d500 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5081e6d70 con 0x7fd5081505b0 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd50815a690 0x7fd5081e62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd50815a690 0x7fd5081e62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56010/0 (socket says 172.21.15.141:56010) 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 -- 172.21.15.141:0/1376108177 learned_addr learned my addr 172.21.15.141:0/1376108177 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd507400640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081505b0 0x7fd5081f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd507e00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50806d500 0x7fd50814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 -- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081505b0 msgr2=0x7fd5081f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081505b0 0x7fd5081f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 -- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50806d500 msgr2=0x7fd50814e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50806d500 0x7fd50814e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd50d200640 1 -- 172.21.15.141:0/1376108177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f800a5a0 con 0x7fd50815a690 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd507400640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081505b0 0x7fd5081f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.201+0000 7fd507e00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50806d500 0x7fd50814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.202+0000 7fd50d200640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd50815a690 0x7fd5081e62f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd500002230 tx=0x7fd50000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.202+0000 7fd4ffe00640 1 -- 172.21.15.141:0/1376108177 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50000ac30 con 0x7fd50815a690 2024-01-16T14:57:43.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.202+0000 7fd4ffe00640 1 -- 172.21.15.141:0/1376108177 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50000ad90 con 0x7fd50815a690 2024-01-16T14:57:43.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.202+0000 7fd4ffe00640 1 -- 172.21.15.141:0/1376108177 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500012620 con 0x7fd50815a690 2024-01-16T14:57:43.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.202+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5081e3bd0 con 0x7fd50815a690 2024-01-16T14:57:43.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.202+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5081e4190 con 0x7fd50815a690 2024-01-16T14:57:43.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.203+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5081e3a30 con 0x7fd50815a690 2024-01-16T14:57:43.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.204+0000 7fd4ffe00640 1 -- 172.21.15.141:0/1376108177 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd500007070 con 0x7fd50815a690 2024-01-16T14:57:43.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.205+0000 7fd4ffe00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006b2c0 0x7fd4e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.205+0000 7fd4ffe00640 1 -- 172.21.15.141:0/1376108177 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd500089b30 con 0x7fd50815a690 2024-01-16T14:57:43.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.206+0000 7fd507e00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006b2c0 0x7fd4e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.207+0000 7fd507e00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006b2c0 0x7fd4e006d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8037000 tx=0x7fd4f8031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.210+0000 7fd4ffe00640 1 -- 172.21.15.141:0/1376108177 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5081e3a30 con 0x7fd50815a690 2024-01-16T14:57:43.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.401+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5081e3a30 con 0x7fd50815a690 2024-01-16T14:57:43.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.402+0000 7fd4ffe00640 1 -- 172.21.15.141:0/1376108177 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5081e3a30 con 0x7fd50815a690 2024-01-16T14:57:43.383 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:43.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.404+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006b2c0 msgr2=0x7fd4e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.404+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006b2c0 0x7fd4e006d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8037000 tx=0x7fd4f8031000 comp rx=0 tx=0).stop 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.404+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd50815a690 msgr2=0x7fd5081e62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.404+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd50815a690 0x7fd5081e62f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd500002230 tx=0x7fd50000cf50 comp rx=0 tx=0).stop 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 shutdown_connections 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006b2c0 0x7fd4e006d780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd50815a690 0x7fd5081e62f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081505b0 0x7fd5081f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 --2- 172.21.15.141:0/1376108177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50806d500 0x7fd50814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 >> 172.21.15.141:0/1376108177 conn(0x7fd50806a7d0 msgr2=0x7fd5080aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 shutdown_connections 2024-01-16T14:57:43.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.405+0000 7fd50dc00640 1 -- 172.21.15.141:0/1376108177 wait complete. 2024-01-16T14:57:43.405 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.639+0000 7fb106e00640 1 Processor -- start 2024-01-16T14:57:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.639+0000 7fb106e00640 1 -- start start 2024-01-16T14:57:43.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.640+0000 7fb106e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 0x7fb1000b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.640+0000 7fb106e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1000d1c60 con 0x7fb10015d4b0 2024-01-16T14:57:43.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.640+0000 7fb106e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1000b2730 con 0x7fb1000cf9a0 2024-01-16T14:57:43.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.640+0000 7fb106e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000d1ec0 con 0x7fb1000cfde0 2024-01-16T14:57:43.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.641+0000 7fb105a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb10015d4b0 0x7fb1000d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37238/0 (socket says 172.21.15.141:37238) 2024-01-16T14:57:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.641+0000 7fb105a00640 1 -- 172.21.15.141:0/132871168 learned_addr learned my addr 172.21.15.141:0/132871168 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.641+0000 7fb106400640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 0x7fb1000b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.641+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030528442 0 0) 0x7fb1000d1c60 con 0x7fb10015d4b0 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.641+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0003520 con 0x7fb10015d4b0 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98645318 0 0) 0x7fb0e0003520 con 0x7fb10015d4b0 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 msgr2=0x7fb1000b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 0x7fb1000b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 >> v1:172.21.15.188:6789/0 conn(0x7fb1000cf9a0 legacy=0x7fb1000b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1000b29d0 con 0x7fb10015d4b0 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb0f80019f0 con 0x7fb10015d4b0 2024-01-16T14:57:43.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 >> v1:172.21.15.141:6789/0 conn(0x7fb10015d4b0 legacy=0x7fb1000d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 0x7fb0e00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0003f70 0x7fb0e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0e0004d00 0x7fb0e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000d1ec0 con 0x7fb0e0004d00 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e000bac0 con 0x7fb0e0003f70 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e000bc80 con 0x7fb1000cfde0 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.642+0000 7fb106400640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0e0004d00 0x7fb0e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb105a00640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 0x7fb0e00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb105000640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0003f70 0x7fb0e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb106400640 1 -- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 msgr2=0x7fb0e00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb106400640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 0x7fb0e00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb106400640 1 -- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0003f70 msgr2=0x7fb0e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb106400640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0003f70 0x7fb0e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb106400640 1 -- 172.21.15.141:0/132871168 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0003520 con 0x7fb0e0004d00 2024-01-16T14:57:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.643+0000 7fb105a00640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000cfde0 0x7fb0e00039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.644+0000 7fb106400640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0e0004d00 0x7fb0e0005180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009ac0 tx=0x7fb0e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed498ff93c04fd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.644+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002c6b0 con 0x7fb0e0004d00 2024-01-16T14:57:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.644+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f002c810 con 0x7fb0e0004d00 2024-01-16T14:57:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.644+0000 7fb0ffe00640 1 -- 172.21.15.141:0/132871168 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002cba0 con 0x7fb0e0004d00 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.645+0000 7fb106e00640 1 -- 172.21.15.141:0/132871168 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0e0004d00 msgr2=0x7fb0e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.645+0000 7fb106e00640 1 --2- 172.21.15.141:0/132871168 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0e0004d00 0x7fb0e0005180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009ac0 tx=0x7fb0e00057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.645+0000 7fb106400640 1 -- 172.21.15.141:0/132871168 reap_dead start 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.645+0000 7fb106e00640 1 -- 172.21.15.141:0/132871168 shutdown_connections 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.645+0000 7fb106e00640 1 -- 172.21.15.141:0/132871168 >> 172.21.15.141:0/132871168 conn(0x7fb1000d3000 msgr2=0x7fb1000d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.645+0000 7fb106e00640 1 -- 172.21.15.141:0/132871168 shutdown_connections 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.645+0000 7fb106e00640 1 -- 172.21.15.141:0/132871168 wait complete. 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.647+0000 7fb106e00640 1 Processor -- start 2024-01-16T14:57:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.647+0000 7fb106e00640 1 -- start start 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0004d00 0x7fb10015a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000a0660 0x7fb1001545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1001f2f60 0x7fb1001f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000c77b0 con 0x7fb1001f2f60 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb100154af0 con 0x7fb0e0004d00 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000c6b30 con 0x7fb1000a0660 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1001f2f60 0x7fb1001f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1001f2f60 0x7fb1001f3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56034/0 (socket says 172.21.15.141:56034) 2024-01-16T14:57:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106400640 1 -- 172.21.15.141:0/2043255686 learned_addr learned my addr 172.21.15.141:0/2043255686 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb105a00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0004d00 0x7fb10015a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb105000640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000a0660 0x7fb1001545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106400640 1 -- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000a0660 msgr2=0x7fb1001545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.648+0000 7fb106400640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000a0660 0x7fb1001545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb106400640 1 -- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0004d00 msgr2=0x7fb10015a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb106400640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0004d00 0x7fb10015a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb106400640 1 -- 172.21.15.141:0/2043255686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1000c6e10 con 0x7fb1001f2f60 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb106400640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1001f2f60 0x7fb1001f3410 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0037040 tx=0x7fb0f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb0fd600640 1 -- 172.21.15.141:0/2043255686 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002c6b0 con 0x7fb1001f2f60 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb0fd600640 1 -- 172.21.15.141:0/2043255686 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f0003810 con 0x7fb1001f2f60 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb0fd600640 1 -- 172.21.15.141:0/2043255686 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0003bc0 con 0x7fb1001f2f60 2024-01-16T14:57:43.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1000c70f0 con 0x7fb1001f2f60 2024-01-16T14:57:43.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.649+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1000c9c30 con 0x7fb1001f2f60 2024-01-16T14:57:43.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.650+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb100067450 con 0x7fb1001f2f60 2024-01-16T14:57:43.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.651+0000 7fb0fd600640 1 -- 172.21.15.141:0/2043255686 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb0f0003d20 con 0x7fb1001f2f60 2024-01-16T14:57:43.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.652+0000 7fb0fd600640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0d806b2c0 0x7fb0d806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:43.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.652+0000 7fb0fd600640 1 -- 172.21.15.141:0/2043255686 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0f00ab310 con 0x7fb1001f2f60 2024-01-16T14:57:43.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.653+0000 7fb105a00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0d806b2c0 0x7fb0d806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:43.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.654+0000 7fb105a00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0d806b2c0 0x7fb0d806d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8002230 tx=0x7fb0f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:43.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.657+0000 7fb0fd600640 1 -- 172.21.15.141:0/2043255686 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0f0076210 con 0x7fb1001f2f60 2024-01-16T14:57:43.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.846+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1000ce910 con 0x7fb1001f2f60 2024-01-16T14:57:43.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.847+0000 7fb0fd600640 1 -- 172.21.15.141:0/2043255686 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0f00044e0 con 0x7fb1001f2f60 2024-01-16T14:57:43.829 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:43.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.849+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0d806b2c0 msgr2=0x7fb0d806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.849+0000 7fb106e00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0d806b2c0 0x7fb0d806d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8002230 tx=0x7fb0f8009040 comp rx=0 tx=0).stop 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.849+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1001f2f60 msgr2=0x7fb1001f3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.849+0000 7fb106e00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1001f2f60 0x7fb1001f3410 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0037040 tx=0x7fb0f0004730 comp rx=0 tx=0).stop 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 shutdown_connections 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0d806b2c0 0x7fb0d806d780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1001f2f60 0x7fb1001f3410 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1000a0660 0x7fb1001545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 --2- 172.21.15.141:0/2043255686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e0004d00 0x7fb10015a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 >> 172.21.15.141:0/2043255686 conn(0x7fb1000d3000 msgr2=0x7fb1000d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 shutdown_connections 2024-01-16T14:57:43.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:43.850+0000 7fb106e00640 1 -- 172.21.15.141:0/2043255686 wait complete. 2024-01-16T14:57:43.849 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:44.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.076+0000 7f55ebe00640 1 Processor -- start 2024-01-16T14:57:44.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.076+0000 7f55ebe00640 1 -- start start 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.077+0000 7f55ebe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f55e413df80 0x7f55e415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.077+0000 7f55ebe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f55e413e360 0x7f55e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.077+0000 7f55ebe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e406c470 con 0x7f55e415f9c0 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.077+0000 7f55ebe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e406c750 con 0x7f55e413df80 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.077+0000 7f55ebe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e406ca30 con 0x7f55e413e360 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e9400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f55e413df80 0x7f55e415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e9400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f55e413df80 0x7f55e415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56042/0 (socket says 172.21.15.141:56042) 2024-01-16T14:57:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e9400640 1 -- 172.21.15.141:0/741004361 learned_addr learned my addr 172.21.15.141:0/741004361 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:44.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e8a00640 1 --2- 172.21.15.141:0/741004361 >> v2:172.21.15.188:3300/0 conn(0x7f55e413e360 0x7f55e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e9400640 1 -- 172.21.15.141:0/741004361 >> v2:172.21.15.188:3300/0 conn(0x7f55e413e360 msgr2=0x7f55e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e9400640 1 --2- 172.21.15.141:0/741004361 >> v2:172.21.15.188:3300/0 conn(0x7f55e413e360 0x7f55e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e9400640 1 -- 172.21.15.141:0/741004361 >> v1:172.21.15.188:6789/0 conn(0x7f55e415f9c0 legacy=0x7f55e41408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.078+0000 7f55e9400640 1 -- 172.21.15.141:0/741004361 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e406cd10 con 0x7f55e413df80 2024-01-16T14:57:44.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.079+0000 7f55e9400640 1 --2- 172.21.15.141:0/741004361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413df80 0x7f55e415bee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f55e406ded0 tx=0x7f55d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d947cd5e8ceee09 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55e3e00640 1 -- 172.21.15.141:0/741004361 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4031870 con 0x7f55e413df80 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55e3e00640 1 -- 172.21.15.141:0/741004361 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d40319d0 con 0x7f55e413df80 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55e3e00640 1 -- 172.21.15.141:0/741004361 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4031d60 con 0x7f55e413df80 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 -- 172.21.15.141:0/741004361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413df80 msgr2=0x7f55e415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 --2- 172.21.15.141:0/741004361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413df80 0x7f55e415bee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f55e406ded0 tx=0x7f55d402b910 comp rx=0 tx=0).stop 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 -- 172.21.15.141:0/741004361 shutdown_connections 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 --2- 172.21.15.141:0/741004361 >> v2:172.21.15.188:3300/0 conn(0x7f55e413e360 0x7f55e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 --2- 172.21.15.141:0/741004361 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413df80 0x7f55e415bee0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 -- 172.21.15.141:0/741004361 >> 172.21.15.141:0/741004361 conn(0x7f55e406a800 msgr2=0x7f55e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 -- 172.21.15.141:0/741004361 shutdown_connections 2024-01-16T14:57:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.080+0000 7f55ebe00640 1 -- 172.21.15.141:0/741004361 wait complete. 2024-01-16T14:57:44.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.082+0000 7f55ebe00640 1 Processor -- start 2024-01-16T14:57:44.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.082+0000 7f55ebe00640 1 -- start start 2024-01-16T14:57:44.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.083+0000 7f55ebe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55e413df80 0x7f55e4152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.083+0000 7f55ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413e360 0x7f55e414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.083+0000 7f55ebe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55e415f9c0 0x7f55e4153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.083+0000 7f55ebe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e414baa0 con 0x7f55e413e360 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.083+0000 7f55ebe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e414ae90 con 0x7f55e415f9c0 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.083+0000 7f55ebe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e414b170 con 0x7f55e413df80 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413e360 0x7f55e414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413e360 0x7f55e414bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56050/0 (socket says 172.21.15.141:56050) 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 -- 172.21.15.141:0/923673678 learned_addr learned my addr 172.21.15.141:0/923673678 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e9400640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55e413df80 0x7f55e4152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e9e00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55e415f9c0 0x7f55e4153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 -- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55e413df80 msgr2=0x7f55e4152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55e413df80 0x7f55e4152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 -- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55e415f9c0 msgr2=0x7f55e4153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55e415f9c0 0x7f55e4153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.084+0000 7f55e8a00640 1 -- 172.21.15.141:0/923673678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40094d0 con 0x7f55e413e360 2024-01-16T14:57:44.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.085+0000 7f55e8a00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413e360 0x7f55e414bf30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00d7d0 tx=0x7f55cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.085+0000 7f55e1600640 1 -- 172.21.15.141:0/923673678 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc018070 con 0x7f55e413e360 2024-01-16T14:57:44.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.085+0000 7f55e1600640 1 -- 172.21.15.141:0/923673678 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc004020 con 0x7f55e413e360 2024-01-16T14:57:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.085+0000 7f55e1600640 1 -- 172.21.15.141:0/923673678 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0041b0 con 0x7f55e413e360 2024-01-16T14:57:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.085+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e414b450 con 0x7f55e413e360 2024-01-16T14:57:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.085+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e4152fb0 con 0x7f55e413e360 2024-01-16T14:57:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.086+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e415d770 con 0x7f55e413e360 2024-01-16T14:57:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.087+0000 7f55e1600640 1 -- 172.21.15.141:0/923673678 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f55cc004310 con 0x7f55e413e360 2024-01-16T14:57:44.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.088+0000 7f55e1600640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55bc06b2c0 0x7f55bc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.088+0000 7f55e1600640 1 -- 172.21.15.141:0/923673678 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55cc089a60 con 0x7f55e413e360 2024-01-16T14:57:44.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.089+0000 7f55e9400640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55bc06b2c0 0x7f55bc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.090+0000 7f55e9400640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55bc06b2c0 0x7f55bc06d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f55d4036000 tx=0x7f55d403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.092+0000 7f55e1600640 1 -- 172.21.15.141:0/923673678 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55cc0548d0 con 0x7f55e413e360 2024-01-16T14:57:44.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.286+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55e415af00 con 0x7f55e413e360 2024-01-16T14:57:44.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.287+0000 7f55e1600640 1 -- 172.21.15.141:0/923673678 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f55cc057ef0 con 0x7f55e413e360 2024-01-16T14:57:44.269 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55bc06b2c0 msgr2=0x7f55bc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55bc06b2c0 0x7f55bc06d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f55d4036000 tx=0x7f55d403c040 comp rx=0 tx=0).stop 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413e360 msgr2=0x7f55e414bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413e360 0x7f55e414bf30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00d7d0 tx=0x7f55cc00dc90 comp rx=0 tx=0).stop 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 shutdown_connections 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55bc06b2c0 0x7f55bc06d780 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55e415f9c0 0x7f55e4153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55e413e360 0x7f55e414bf30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 --2- 172.21.15.141:0/923673678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55e413df80 0x7f55e4152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 >> 172.21.15.141:0/923673678 conn(0x7f55e406a800 msgr2=0x7f55e415d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 shutdown_connections 2024-01-16T14:57:44.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.290+0000 7f55ebe00640 1 -- 172.21.15.141:0/923673678 wait complete. 2024-01-16T14:57:44.291 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:44.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.525+0000 7efc35e00640 1 Processor -- start 2024-01-16T14:57:44.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.525+0000 7efc35e00640 1 -- start start 2024-01-16T14:57:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.527+0000 7efc35e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 0x7efc300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.527+0000 7efc35e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc300cfc30 con 0x7efc3015e3b0 2024-01-16T14:57:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.527+0000 7efc35e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc300b1ee0 con 0x7efc300cf7d0 2024-01-16T14:57:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.527+0000 7efc35e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc300b2780 con 0x7efc30143ae0 2024-01-16T14:57:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.527+0000 7efc2f400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7efc3015e3b0 0x7efc300702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37252/0 (socket says 172.21.15.141:37252) 2024-01-16T14:57:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.527+0000 7efc2f400640 1 -- 172.21.15.141:0/2207349043 learned_addr learned my addr 172.21.15.141:0/2207349043 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2ea00640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 0x7efc300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362842328 0 0) 0x7efc300cfc30 con 0x7efc3015e3b0 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc10003520 con 0x7efc3015e3b0 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4210166201 0 0) 0x7efc10003520 con 0x7efc3015e3b0 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 msgr2=0x7efc300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 0x7efc300cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 >> v1:172.21.15.188:6789/0 conn(0x7efc300cf7d0 legacy=0x7efc3015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3015e990 con 0x7efc3015e3b0 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7efc28001670 con 0x7efc3015e3b0 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.528+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 >> v1:172.21.15.141:6789/0 conn(0x7efc3015e3b0 legacy=0x7efc300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2d600640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2d600640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc100047c0 0x7efc10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2d600640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 0x7efc10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc300b2780 con 0x7efc100039a0 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1000ba30 con 0x7efc100047c0 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1000bbf0 con 0x7efc30143ae0 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2ea00640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2e000640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc100047c0 0x7efc10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2ea00640 1 -- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 msgr2=0x7efc10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2f400640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 0x7efc10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2ea00640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc30143ae0 0x7efc10005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2ea00640 1 -- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc100047c0 msgr2=0x7efc10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2ea00640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc100047c0 0x7efc10004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.529+0000 7efc2ea00640 1 -- 172.21.15.141:0/2207349043 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10003520 con 0x7efc100039a0 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.531+0000 7efc2ea00640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc10003dc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc20009a90 tx=0x7efc100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faeef280cf2eb4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.531+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2002c680 con 0x7efc100039a0 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.531+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2002c7e0 con 0x7efc100039a0 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.531+0000 7efc2d600640 1 -- 172.21.15.141:0/2207349043 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2002cb70 con 0x7efc100039a0 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.531+0000 7efc35e00640 1 -- 172.21.15.141:0/2207349043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 msgr2=0x7efc10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.531+0000 7efc35e00640 1 --2- 172.21.15.141:0/2207349043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc10003dc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc20009a90 tx=0x7efc100057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.532+0000 7efc2f400640 1 -- 172.21.15.141:0/2207349043 reap_dead start 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.532+0000 7efc35e00640 1 -- 172.21.15.141:0/2207349043 shutdown_connections 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.532+0000 7efc35e00640 1 -- 172.21.15.141:0/2207349043 >> 172.21.15.141:0/2207349043 conn(0x7efc3006a800 msgr2=0x7efc3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.532+0000 7efc35e00640 1 -- 172.21.15.141:0/2207349043 shutdown_connections 2024-01-16T14:57:44.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.532+0000 7efc35e00640 1 -- 172.21.15.141:0/2207349043 wait complete. 2024-01-16T14:57:44.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.534+0000 7efc35e00640 1 Processor -- start 2024-01-16T14:57:44.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.534+0000 7efc35e00640 1 -- start start 2024-01-16T14:57:44.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc35e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc301f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc35e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc300ac8d0 0x7efc3015be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc35e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc300ca600 0x7efc300caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc35e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc300cc9c0 con 0x7efc100039a0 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc35e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc300caff0 con 0x7efc300ac8d0 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc35e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc300acd50 con 0x7efc300ca600 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc301f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc301f2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56070/0 (socket says 172.21.15.141:56070) 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 -- 172.21.15.141:0/1119385978 learned_addr learned my addr 172.21.15.141:0/1119385978 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 -- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc300ca600 msgr2=0x7efc300caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc300ca600 0x7efc300caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 -- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc300ac8d0 msgr2=0x7efc3015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc300ac8d0 0x7efc3015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.535+0000 7efc2ea00640 1 -- 172.21.15.141:0/1119385978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc300ad020 con 0x7efc100039a0 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.536+0000 7efc2ea00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc301f2d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efc20037040 tx=0x7efc20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.536+0000 7efc27400640 1 -- 172.21.15.141:0/1119385978 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc20003810 con 0x7efc100039a0 2024-01-16T14:57:44.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.536+0000 7efc27400640 1 -- 172.21.15.141:0/1119385978 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc20003970 con 0x7efc100039a0 2024-01-16T14:57:44.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.536+0000 7efc27400640 1 -- 172.21.15.141:0/1119385978 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc20003ce0 con 0x7efc100039a0 2024-01-16T14:57:44.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.536+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc300ad300 con 0x7efc100039a0 2024-01-16T14:57:44.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.536+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc300ad8c0 con 0x7efc100039a0 2024-01-16T14:57:44.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.537+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3006ad30 con 0x7efc100039a0 2024-01-16T14:57:44.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.538+0000 7efc27400640 1 -- 172.21.15.141:0/1119385978 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efc20034440 con 0x7efc100039a0 2024-01-16T14:57:44.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.539+0000 7efc27400640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc0806b2c0 0x7efc0806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.539+0000 7efc27400640 1 -- 172.21.15.141:0/1119385978 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc200ab9d0 con 0x7efc100039a0 2024-01-16T14:57:44.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.540+0000 7efc2e000640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc0806b2c0 0x7efc0806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.541+0000 7efc2e000640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc0806b2c0 0x7efc0806d780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efc18007fd0 tx=0x7efc18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.543+0000 7efc27400640 1 -- 172.21.15.141:0/1119385978 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc20076a10 con 0x7efc100039a0 2024-01-16T14:57:44.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.736+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc300675d0 con 0x7efc100039a0 2024-01-16T14:57:44.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.739+0000 7efc27400640 1 -- 172.21.15.141:0/1119385978 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7efc2004c090 con 0x7efc100039a0 2024-01-16T14:57:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:44.721 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc0806b2c0 msgr2=0x7efc0806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc0806b2c0 0x7efc0806d780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efc18007fd0 tx=0x7efc18009040 comp rx=0 tx=0).stop 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 msgr2=0x7efc301f2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc301f2d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efc20037040 tx=0x7efc20004730 comp rx=0 tx=0).stop 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 shutdown_connections 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efc0806b2c0 0x7efc0806d780 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efc300ca600 0x7efc300caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc300ac8d0 0x7efc3015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 --2- 172.21.15.141:0/1119385978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efc100039a0 0x7efc301f2d50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 >> 172.21.15.141:0/1119385978 conn(0x7efc3006a800 msgr2=0x7efc30142d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 shutdown_connections 2024-01-16T14:57:44.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.742+0000 7efc35e00640 1 -- 172.21.15.141:0/1119385978 wait complete. 2024-01-16T14:57:44.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:44.743 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:44.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.971+0000 7f7ea3a00640 1 Processor -- start 2024-01-16T14:57:44.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.971+0000 7f7ea3a00640 1 -- start start 2024-01-16T14:57:44.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.972+0000 7f7ea3a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e9c15da10 0x7f7e9c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.972+0000 7f7ea3a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7e9c0b2bd0 0x7f7e9c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.972+0000 7f7ea3a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c071190 con 0x7f7e9c150760 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.972+0000 7f7ea3a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c071470 con 0x7f7e9c15da10 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.972+0000 7f7ea3a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c071750 con 0x7f7e9c0b2bd0 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7e93c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e9c15da10 0x7f7e9c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7ea1000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7e9c150760 0x7f7e9c15d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37260/0 (socket says 172.21.15.141:37260) 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7ea1000640 1 -- 172.21.15.141:0/2147454704 learned_addr learned my addr 172.21.15.141:0/2147454704 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7ea1a00640 1 --2- 172.21.15.141:0/2147454704 >> v2:172.21.15.188:3300/0 conn(0x7f7e9c0b2bd0 0x7f7e9c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7e93c00640 1 -- 172.21.15.141:0/2147454704 >> v2:172.21.15.188:3300/0 conn(0x7f7e9c0b2bd0 msgr2=0x7f7e9c0b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7e93c00640 1 --2- 172.21.15.141:0/2147454704 >> v2:172.21.15.188:3300/0 conn(0x7f7e9c0b2bd0 0x7f7e9c0b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7e93c00640 1 -- 172.21.15.141:0/2147454704 >> v1:172.21.15.141:6789/0 conn(0x7f7e9c150760 legacy=0x7f7e9c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7e93c00640 1 -- 172.21.15.141:0/2147454704 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c071a30 con 0x7f7e9c15da10 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7e9be00640 1 -- 172.21.15.141:0/2147454704 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946722535 0 0) 0x7f7e9c071190 con 0x7f7e9c150760 2024-01-16T14:57:44.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.973+0000 7f7ea1a00640 1 --2- 172.21.15.141:0/2147454704 >> v2:172.21.15.188:3300/0 conn(0x7f7e9c0b2bd0 0x7f7e9c0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.974+0000 7f7e93c00640 1 --2- 172.21.15.141:0/2147454704 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c15da10 0x7f7e9c072230 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0cb740 tx=0x7f7e8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81707c927a454a2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.974+0000 7f7e9be00640 1 -- 172.21.15.141:0/2147454704 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84031830 con 0x7f7e9c15da10 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.974+0000 7f7e9be00640 1 -- 172.21.15.141:0/2147454704 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e84031990 con 0x7f7e9c15da10 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.974+0000 7f7e9be00640 1 -- 172.21.15.141:0/2147454704 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84031d20 con 0x7f7e9c15da10 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 -- 172.21.15.141:0/2147454704 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c15da10 msgr2=0x7f7e9c072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/2147454704 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c15da10 0x7f7e9c072230 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0cb740 tx=0x7f7e8402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 -- 172.21.15.141:0/2147454704 shutdown_connections 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/2147454704 >> v2:172.21.15.188:3300/0 conn(0x7f7e9c0b2bd0 0x7f7e9c0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/2147454704 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c15da10 0x7f7e9c072230 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 -- 172.21.15.141:0/2147454704 >> 172.21.15.141:0/2147454704 conn(0x7f7e9c06a800 msgr2=0x7f7e9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 -- 172.21.15.141:0/2147454704 shutdown_connections 2024-01-16T14:57:44.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.975+0000 7f7ea3a00640 1 -- 172.21.15.141:0/2147454704 wait complete. 2024-01-16T14:57:44.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.977+0000 7f7ea3a00640 1 Processor -- start 2024-01-16T14:57:44.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.977+0000 7f7ea3a00640 1 -- start start 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7ea3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9c0b2bd0 0x7f7e9c1474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7ea3a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c150760 0x7f7e9c1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7ea3a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e9c15da10 0x7f7e9c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7ea3a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c1597e0 con 0x7f7e9c150760 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7ea3a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0add00 con 0x7f7e9c15da10 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7ea3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c06d3b0 con 0x7f7e9c0b2bd0 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7e93c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c150760 0x7f7e9c1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7e93c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c150760 0x7f7e9c1592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56092/0 (socket says 172.21.15.141:56092) 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.978+0000 7f7e93c00640 1 -- 172.21.15.141:0/4207149155 learned_addr learned my addr 172.21.15.141:0/4207149155 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7ea1a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e9c15da10 0x7f7e9c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7ea1000640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9c0b2bd0 0x7f7e9c1474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7e93c00640 1 -- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9c0b2bd0 msgr2=0x7f7e9c1474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7e93c00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9c0b2bd0 0x7f7e9c1474b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7e93c00640 1 -- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e9c15da10 msgr2=0x7f7e9c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7e93c00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e9c15da10 0x7f7e9c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7e93c00640 1 -- 172.21.15.141:0/4207149155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84009400 con 0x7f7e9c150760 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7ea1a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e9c15da10 0x7f7e9c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.979+0000 7f7e93c00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c150760 0x7f7e9c1592a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009890 tx=0x7f7e840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.980+0000 7f7e99600640 1 -- 172.21.15.141:0/4207149155 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8403aa60 con 0x7f7e9c150760 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.980+0000 7f7e99600640 1 -- 172.21.15.141:0/4207149155 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8403abe0 con 0x7f7e9c150760 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.980+0000 7f7e99600640 1 -- 172.21.15.141:0/4207149155 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e840423f0 con 0x7f7e9c150760 2024-01-16T14:57:44.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.980+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c06d690 con 0x7f7e9c150760 2024-01-16T14:57:44.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.980+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c0cd1d0 con 0x7f7e9c150760 2024-01-16T14:57:44.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.981+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c0cd030 con 0x7f7e9c150760 2024-01-16T14:57:44.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.982+0000 7f7e99600640 1 -- 172.21.15.141:0/4207149155 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7e8404b3b0 con 0x7f7e9c150760 2024-01-16T14:57:44.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.983+0000 7f7e99600640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e7406b2c0 0x7f7e7406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:44.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.983+0000 7f7e99600640 1 -- 172.21.15.141:0/4207149155 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e840abcb0 con 0x7f7e9c150760 2024-01-16T14:57:44.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.983+0000 7f7ea1000640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e7406b2c0 0x7f7e7406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:44.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.984+0000 7f7ea1000640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e7406b2c0 0x7f7e7406d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c007fd0 tx=0x7f7e8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:44.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:44.987+0000 7f7e99600640 1 -- 172.21.15.141:0/4207149155 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e9c0cd030 con 0x7f7e9c150760 2024-01-16T14:57:45.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.179+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e9c0cd030 con 0x7f7e9c150760 2024-01-16T14:57:45.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.180+0000 7f7e99600640 1 -- 172.21.15.141:0/4207149155 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e9c0cd030 con 0x7f7e9c150760 2024-01-16T14:57:45.162 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e7406b2c0 msgr2=0x7f7e7406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e7406b2c0 0x7f7e7406d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c007fd0 tx=0x7f7e8c009040 comp rx=0 tx=0).stop 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c150760 msgr2=0x7f7e9c1592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c150760 0x7f7e9c1592a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009890 tx=0x7f7e840046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 shutdown_connections 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e7406b2c0 0x7f7e7406d780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e9c15da10 0x7f7e9c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e9c150760 0x7f7e9c1592a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 --2- 172.21.15.141:0/4207149155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9c0b2bd0 0x7f7e9c1474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 >> 172.21.15.141:0/4207149155 conn(0x7f7e9c06a800 msgr2=0x7f7e9c149a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 shutdown_connections 2024-01-16T14:57:45.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.183+0000 7f7ea3a00640 1 -- 172.21.15.141:0/4207149155 wait complete. 2024-01-16T14:57:45.183 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:45.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.415+0000 7f6a45600640 1 Processor -- start 2024-01-16T14:57:45.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.415+0000 7f6a45600640 1 -- start start 2024-01-16T14:57:45.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.416+0000 7f6a45600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6a400cf9a0 0x7f6a4015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.416+0000 7f6a45600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a4015e1e0 0x7f6a400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.416+0000 7f6a45600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400b1ee0 con 0x7f6a40143ae0 2024-01-16T14:57:45.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.416+0000 7f6a45600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400b2780 con 0x7f6a400cf9a0 2024-01-16T14:57:45.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.416+0000 7f6a45600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4015e990 con 0x7f6a4015e1e0 2024-01-16T14:57:45.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.416+0000 7f6a3ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6a40143ae0 0x7f6a400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37268/0 (socket says 172.21.15.141:37268) 2024-01-16T14:57:45.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.416+0000 7f6a3ea00640 1 -- 172.21.15.141:0/2737870032 learned_addr learned my addr 172.21.15.141:0/2737870032 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3f400640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a4015e1e0 0x7f6a400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2737870032 >> v2:172.21.15.188:3300/0 conn(0x7f6a400cf9a0 0x7f6a4015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803879707 0 0) 0x7f6a400b1ee0 con 0x7f6a40143ae0 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a20003520 con 0x7f6a40143ae0 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1382384661 0 0) 0x7f6a20003520 con 0x7f6a40143ae0 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a4015e1e0 msgr2=0x7f6a400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a4015e1e0 0x7f6a400702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 >> v2:172.21.15.188:3300/0 conn(0x7f6a400cf9a0 msgr2=0x7f6a4015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 --2- 172.21.15.141:0/2737870032 >> v2:172.21.15.188:3300/0 conn(0x7f6a400cf9a0 0x7f6a4015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400b2130 con 0x7f6a40143ae0 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6a30001670 con 0x7f6a40143ae0 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2737870032 >> v2:172.21.15.188:3300/0 conn(0x7f6a400cf9a0 0x7f6a4015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 >> v1:172.21.15.141:6789/0 conn(0x7f6a40143ae0 legacy=0x7f6a400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a200039a0 0x7f6a20003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.417+0000 7f6a3e000640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a20004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a3e000640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a4015e1e0 0x7f6a20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400b2780 con 0x7f6a200047c0 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4015e990 con 0x7f6a200039a0 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2000bae0 con 0x7f6a4015e1e0 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a20004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a3ea00640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a200039a0 0x7f6a20003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a35e00640 1 -- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a4015e1e0 msgr2=0x7f6a20005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a4015e1e0 0x7f6a20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a35e00640 1 -- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a200039a0 msgr2=0x7f6a20003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a200039a0 0x7f6a20003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.418+0000 7f6a35e00640 1 -- 172.21.15.141:0/2737870032 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20003520 con 0x7f6a200047c0 2024-01-16T14:57:45.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.419+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a20004c40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a28006fd0 tx=0x7f6a200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d9cb5d2408bb51 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.419+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28004230 con 0x7f6a200047c0 2024-01-16T14:57:45.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.419+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a28004390 con 0x7f6a200047c0 2024-01-16T14:57:45.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.419+0000 7f6a3e000640 1 -- 172.21.15.141:0/2737870032 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28003830 con 0x7f6a200047c0 2024-01-16T14:57:45.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.420+0000 7f6a45600640 1 -- 172.21.15.141:0/2737870032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 msgr2=0x7f6a20004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.420+0000 7f6a45600640 1 --2- 172.21.15.141:0/2737870032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a20004c40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a28006fd0 tx=0x7f6a200057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.420+0000 7f6a3f400640 1 -- 172.21.15.141:0/2737870032 reap_dead start 2024-01-16T14:57:45.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.420+0000 7f6a45600640 1 -- 172.21.15.141:0/2737870032 shutdown_connections 2024-01-16T14:57:45.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.420+0000 7f6a45600640 1 -- 172.21.15.141:0/2737870032 >> 172.21.15.141:0/2737870032 conn(0x7f6a4006a800 msgr2=0x7f6a4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.420+0000 7f6a45600640 1 -- 172.21.15.141:0/2737870032 shutdown_connections 2024-01-16T14:57:45.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.420+0000 7f6a45600640 1 -- 172.21.15.141:0/2737870032 wait complete. 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.422+0000 7f6a45600640 1 Processor -- start 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.422+0000 7f6a45600640 1 -- start start 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a45600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a400ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a45600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a400ab5a0 0x7f6a400aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a45600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a40166710 0x7f6a40166b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a45600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a401670d0 con 0x7f6a200047c0 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a45600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4015a240 con 0x7f6a40166710 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a45600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400add30 con 0x7f6a400ab5a0 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a3ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a400ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a3ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a400ad7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56116/0 (socket says 172.21.15.141:56116) 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.423+0000 7f6a3ea00640 1 -- 172.21.15.141:0/2413458087 learned_addr learned my addr 172.21.15.141:0/2413458087 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a400ab5a0 0x7f6a400aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a3f400640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a40166710 0x7f6a40166b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a3ea00640 1 -- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a400ab5a0 msgr2=0x7f6a400aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a3ea00640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a400ab5a0 0x7f6a400aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a3ea00640 1 -- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a40166710 msgr2=0x7f6a40166b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a3ea00640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a40166710 0x7f6a40166b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a3ea00640 1 -- 172.21.15.141:0/2413458087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400a85f0 con 0x7f6a200047c0 2024-01-16T14:57:45.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a400ab5a0 0x7f6a400aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:45.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a3ea00640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a400ad7f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000d810 tx=0x7f6a3000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a37400640 1 -- 172.21.15.141:0/2413458087 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a300074e0 con 0x7f6a200047c0 2024-01-16T14:57:45.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a37400640 1 -- 172.21.15.141:0/2413458087 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a30007640 con 0x7f6a200047c0 2024-01-16T14:57:45.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a400a88d0 con 0x7f6a200047c0 2024-01-16T14:57:45.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a37400640 1 -- 172.21.15.141:0/2413458087 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a300049e0 con 0x7f6a200047c0 2024-01-16T14:57:45.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.424+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a400a8e90 con 0x7f6a200047c0 2024-01-16T14:57:45.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.426+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a400675d0 con 0x7f6a200047c0 2024-01-16T14:57:45.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.426+0000 7f6a37400640 1 -- 172.21.15.141:0/2413458087 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6a30004b40 con 0x7f6a200047c0 2024-01-16T14:57:45.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.427+0000 7f6a37400640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a1806b2c0 0x7f6a1806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.427+0000 7f6a37400640 1 -- 172.21.15.141:0/2413458087 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a30089810 con 0x7f6a200047c0 2024-01-16T14:57:45.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.428+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a1806b2c0 0x7f6a1806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.429+0000 7f6a35e00640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a1806b2c0 0x7f6a1806d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a28035000 tx=0x7f6a2803a040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.431+0000 7f6a37400640 1 -- 172.21.15.141:0/2413458087 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a30054680 con 0x7f6a200047c0 2024-01-16T14:57:45.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.622+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a4006aed0 con 0x7f6a200047c0 2024-01-16T14:57:45.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.623+0000 7f6a37400640 1 -- 172.21.15.141:0/2413458087 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a300190a0 con 0x7f6a200047c0 2024-01-16T14:57:45.604 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a1806b2c0 msgr2=0x7f6a1806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a1806b2c0 0x7f6a1806d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a28035000 tx=0x7f6a2803a040 comp rx=0 tx=0).stop 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 msgr2=0x7f6a400ad7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a400ad7f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000d810 tx=0x7f6a3000dcd0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 shutdown_connections 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a1806b2c0 0x7f6a1806d780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a40166710 0x7f6a40166b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a400ab5a0 0x7f6a400aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 --2- 172.21.15.141:0/2413458087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a200047c0 0x7f6a400ad7f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 >> 172.21.15.141:0/2413458087 conn(0x7f6a4006a800 msgr2=0x7f6a4015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 shutdown_connections 2024-01-16T14:57:45.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.626+0000 7f6a45600640 1 -- 172.21.15.141:0/2413458087 wait complete. 2024-01-16T14:57:45.625 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:45.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.847+0000 7ff760a00640 1 Processor -- start 2024-01-16T14:57:45.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.847+0000 7ff760a00640 1 -- start start 2024-01-16T14:57:45.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.848+0000 7ff760a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff75813df80 0x7ff75815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.848+0000 7ff760a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75813e360 0x7ff7580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.848+0000 7ff760a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff75815f9c0 0x7ff7580cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.848+0000 7ff760a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75806c470 con 0x7ff75813df80 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.848+0000 7ff760a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75806c750 con 0x7ff75815f9c0 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.848+0000 7ff760a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75806ca30 con 0x7ff75813e360 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff75813df80 0x7ff75815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff75813df80 0x7ff75815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56132/0 (socket says 172.21.15.141:56132) 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 -- 172.21.15.141:0/1148525755 learned_addr learned my addr 172.21.15.141:0/1148525755 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 -- 172.21.15.141:0/1148525755 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75813e360 msgr2=0x7ff7580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 --2- 172.21.15.141:0/1148525755 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75813e360 0x7ff7580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 -- 172.21.15.141:0/1148525755 >> v2:172.21.15.188:3300/0 conn(0x7ff75815f9c0 msgr2=0x7ff7580cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 --2- 172.21.15.141:0/1148525755 >> v2:172.21.15.188:3300/0 conn(0x7ff75815f9c0 0x7ff7580cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.849+0000 7ff75e000640 1 -- 172.21.15.141:0/1148525755 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75806cd10 con 0x7ff75813df80 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.850+0000 7ff75e000640 1 --2- 172.21.15.141:0/1148525755 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff75815bee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff75806e6d0 tx=0x7ff74802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba39135a8969b3de server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.850+0000 7ff75cc00640 1 -- 172.21.15.141:0/1148525755 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74802ca70 con 0x7ff75813df80 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff75cc00640 1 -- 172.21.15.141:0/1148525755 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74802cbd0 con 0x7ff75813df80 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff75cc00640 1 -- 172.21.15.141:0/1148525755 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7480343f0 con 0x7ff75813df80 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 -- 172.21.15.141:0/1148525755 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 msgr2=0x7ff75815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 --2- 172.21.15.141:0/1148525755 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff75815bee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff75806e6d0 tx=0x7ff74802bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 -- 172.21.15.141:0/1148525755 shutdown_connections 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 --2- 172.21.15.141:0/1148525755 >> v2:172.21.15.188:3300/0 conn(0x7ff75815f9c0 0x7ff7580cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 --2- 172.21.15.141:0/1148525755 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75813e360 0x7ff7580cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 --2- 172.21.15.141:0/1148525755 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff75815bee0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 -- 172.21.15.141:0/1148525755 >> 172.21.15.141:0/1148525755 conn(0x7ff75806a800 msgr2=0x7ff75806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 -- 172.21.15.141:0/1148525755 shutdown_connections 2024-01-16T14:57:45.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.851+0000 7ff760a00640 1 -- 172.21.15.141:0/1148525755 wait complete. 2024-01-16T14:57:45.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.853+0000 7ff760a00640 1 Processor -- start 2024-01-16T14:57:45.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.853+0000 7ff760a00640 1 -- start start 2024-01-16T14:57:45.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff760a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff758131020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff760a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff75813e360 0x7ff75813f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff760a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75815f9c0 0x7ff75813fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff760a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff758158e90 con 0x7ff75813df80 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff760a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff758131f50 con 0x7ff75813e360 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff760a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff758131990 con 0x7ff75815f9c0 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff75e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff758131020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff75e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff758131020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56148/0 (socket says 172.21.15.141:56148) 2024-01-16T14:57:45.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.854+0000 7ff75e000640 1 -- 172.21.15.141:0/3387857937 learned_addr learned my addr 172.21.15.141:0/3387857937 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:45.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff75e000640 1 -- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75815f9c0 msgr2=0x7ff75813fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff75e000640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75815f9c0 0x7ff75813fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff75e000640 1 -- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff75813e360 msgr2=0x7ff75813f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:45.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff75e000640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff75813e360 0x7ff75813f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:45.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff75e000640 1 -- 172.21.15.141:0/3387857937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748009b30 con 0x7ff75813df80 2024-01-16T14:57:45.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff75e000640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff758131020 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7480093e0 tx=0x7ff748004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff756000640 1 -- 172.21.15.141:0/3387857937 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74803e450 con 0x7ff75813df80 2024-01-16T14:57:45.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff756000640 1 -- 172.21.15.141:0/3387857937 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74803e5b0 con 0x7ff75813df80 2024-01-16T14:57:45.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff756000640 1 -- 172.21.15.141:0/3387857937 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74803e960 con 0x7ff75813df80 2024-01-16T14:57:45.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff758131c70 con 0x7ff75813df80 2024-01-16T14:57:45.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.855+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7581593a0 con 0x7ff75813df80 2024-01-16T14:57:45.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.856+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75815bee0 con 0x7ff75813df80 2024-01-16T14:57:45.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.857+0000 7ff756000640 1 -- 172.21.15.141:0/3387857937 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff74803eac0 con 0x7ff75813df80 2024-01-16T14:57:45.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.858+0000 7ff756000640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff73006b2c0 0x7ff73006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:45.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.858+0000 7ff756000640 1 -- 172.21.15.141:0/3387857937 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7480acd30 con 0x7ff75813df80 2024-01-16T14:57:45.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.859+0000 7ff75d600640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff73006b2c0 0x7ff73006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:45.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.860+0000 7ff75d600640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff73006b2c0 0x7ff73006d780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff75813d890 tx=0x7ff740006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:45.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:45.862+0000 7ff756000640 1 -- 172.21.15.141:0/3387857937 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff748077c20 con 0x7ff75813df80 2024-01-16T14:57:46.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.050+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff7580675d0 con 0x7ff75813df80 2024-01-16T14:57:46.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.051+0000 7ff756000640 1 -- 172.21.15.141:0/3387857937 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff748043020 con 0x7ff75813df80 2024-01-16T14:57:46.033 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:46.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.053+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff73006b2c0 msgr2=0x7ff73006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.053+0000 7ff760a00640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff73006b2c0 0x7ff73006d780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff75813d890 tx=0x7ff740006c90 comp rx=0 tx=0).stop 2024-01-16T14:57:46.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 msgr2=0x7ff758131020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff758131020 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7480093e0 tx=0x7ff748004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 shutdown_connections 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff73006b2c0 0x7ff73006d780 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff75815f9c0 0x7ff75813fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff75813e360 0x7ff75813f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 --2- 172.21.15.141:0/3387857937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff75813df80 0x7ff758131020 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 >> 172.21.15.141:0/3387857937 conn(0x7ff75806a800 msgr2=0x7ff7580cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 shutdown_connections 2024-01-16T14:57:46.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.054+0000 7ff760a00640 1 -- 172.21.15.141:0/3387857937 wait complete. 2024-01-16T14:57:46.116 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-16T14:57:46.116 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:57:46.116 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:46.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.346+0000 7f0acfe00640 1 Processor -- start 2024-01-16T14:57:46.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.347+0000 7f0acfe00640 1 -- start start 2024-01-16T14:57:46.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acfe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0ad0160920 0x7f0ad00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0ad00cfea0 0x7f0ad0161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acfe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00d0a10 con 0x7f0ad00cf9f0 2024-01-16T14:57:46.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acfe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad00d1f50 con 0x7f0ad00cfea0 2024-01-16T14:57:46.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acfe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad00cf080 con 0x7f0ad0160920 2024-01-16T14:57:46.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acf400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0ad00cfea0 0x7f0ad0161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acf400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0ad00cfea0 0x7f0ad0161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56154/0 (socket says 172.21.15.141:56154) 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0ac5e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0ad00cf9f0 0x7f0ad0161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37276/0 (socket says 172.21.15.141:37276) 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.348+0000 7f0acf400640 1 -- 172.21.15.141:0/1644089247 learned_addr learned my addr 172.21.15.141:0/1644089247 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.349+0000 7f0acea00640 1 --2- 172.21.15.141:0/1644089247 >> v2:172.21.15.188:3300/0 conn(0x7f0ad0160920 0x7f0ad00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.349+0000 7f0acf400640 1 -- 172.21.15.141:0/1644089247 >> v2:172.21.15.188:3300/0 conn(0x7f0ad0160920 msgr2=0x7f0ad00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.349+0000 7f0acf400640 1 --2- 172.21.15.141:0/1644089247 >> v2:172.21.15.188:3300/0 conn(0x7f0ad0160920 0x7f0ad00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.349+0000 7f0acf400640 1 -- 172.21.15.141:0/1644089247 >> v1:172.21.15.141:6789/0 conn(0x7f0ad00cf9f0 legacy=0x7f0ad0161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.349+0000 7f0acf400640 1 -- 172.21.15.141:0/1644089247 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad00d0310 con 0x7f0ad00cfea0 2024-01-16T14:57:46.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.349+0000 7f0ace000640 1 -- 172.21.15.141:0/1644089247 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359500512 0 0) 0x7f0ad00d0a10 con 0x7f0ad00cf9f0 2024-01-16T14:57:46.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.350+0000 7f0acf400640 1 --2- 172.21.15.141:0/1644089247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad0161720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0ad008c060 tx=0x7f0ac802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e04828e9c1e61c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.350+0000 7f0ace000640 1 -- 172.21.15.141:0/1644089247 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac802cb20 con 0x7f0ad00cfea0 2024-01-16T14:57:46.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.350+0000 7f0ace000640 1 -- 172.21.15.141:0/1644089247 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac802cc80 con 0x7f0ad00cfea0 2024-01-16T14:57:46.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.350+0000 7f0ace000640 1 -- 172.21.15.141:0/1644089247 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac80343f0 con 0x7f0ad00cfea0 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 -- 172.21.15.141:0/1644089247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 msgr2=0x7f0ad0161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 --2- 172.21.15.141:0/1644089247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad0161720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0ad008c060 tx=0x7f0ac802bf80 comp rx=0 tx=0).stop 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 -- 172.21.15.141:0/1644089247 shutdown_connections 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 --2- 172.21.15.141:0/1644089247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad0161720 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 --2- 172.21.15.141:0/1644089247 >> v2:172.21.15.188:3300/0 conn(0x7f0ad0160920 0x7f0ad00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 -- 172.21.15.141:0/1644089247 >> 172.21.15.141:0/1644089247 conn(0x7f0ad00d3090 msgr2=0x7f0ad00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 -- 172.21.15.141:0/1644089247 shutdown_connections 2024-01-16T14:57:46.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.351+0000 7f0acfe00640 1 -- 172.21.15.141:0/1644089247 wait complete. 2024-01-16T14:57:46.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.353+0000 7f0acfe00640 1 Processor -- start 2024-01-16T14:57:46.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.353+0000 7f0acfe00640 1 -- start start 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0acfe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ad00cf9f0 0x7f0ad00caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0acfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad015a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0acfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ad0160920 0x7f0ad015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0acfe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad00cafd0 con 0x7f0ad00cfea0 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0acfe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad015ab70 con 0x7f0ad0160920 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0acfe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad00b6660 con 0x7f0ad00cf9f0 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0ac5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad015a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0ac5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad015a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56166/0 (socket says 172.21.15.141:56166) 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.354+0000 7f0ac5e00640 1 -- 172.21.15.141:0/2950546711 learned_addr learned my addr 172.21.15.141:0/2950546711 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:46.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0acea00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ad00cf9f0 0x7f0ad00caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0acf400640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ad0160920 0x7f0ad015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac5e00640 1 -- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ad00cf9f0 msgr2=0x7f0ad00caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac5e00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ad00cf9f0 0x7f0ad00caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac5e00640 1 -- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ad0160920 msgr2=0x7f0ad015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac5e00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ad0160920 0x7f0ad015df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac5e00640 1 -- 172.21.15.141:0/2950546711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac8009b30 con 0x7f0ad00cfea0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac5e00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad015a630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c460 tx=0x7f0ab800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac7400640 1 -- 172.21.15.141:0/2950546711 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab80074e0 con 0x7f0ad00cfea0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac7400640 1 -- 172.21.15.141:0/2950546711 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab8007660 con 0x7f0ad00cfea0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0ac7400640 1 -- 172.21.15.141:0/2950546711 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8004a20 con 0x7f0ad00cfea0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.355+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad00b6900 con 0x7f0ad00cfea0 2024-01-16T14:57:46.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.356+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad00b6ec0 con 0x7f0ad00cfea0 2024-01-16T14:57:46.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.356+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad0067450 con 0x7f0ad00cfea0 2024-01-16T14:57:46.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.357+0000 7f0ac7400640 1 -- 172.21.15.141:0/2950546711 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0ab8004b80 con 0x7f0ad00cfea0 2024-01-16T14:57:46.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.358+0000 7f0ac7400640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0aa806b2c0 0x7f0aa806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.358+0000 7f0ac7400640 1 -- 172.21.15.141:0/2950546711 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ab8088e20 con 0x7f0ad00cfea0 2024-01-16T14:57:46.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.359+0000 7f0acea00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0aa806b2c0 0x7f0aa806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.360+0000 7f0acea00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0aa806b2c0 0x7f0aa806d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0006fd0 tx=0x7f0ac0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.363+0000 7f0ac7400640 1 -- 172.21.15.141:0/2950546711 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ab8053c80 con 0x7f0ad00cfea0 2024-01-16T14:57:46.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.544+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ad008c7e0 con 0x7f0ad00cfea0 2024-01-16T14:57:46.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.546+0000 7f0ac7400640 1 -- 172.21.15.141:0/2950546711 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f0ab8018070 con 0x7f0ad00cfea0 2024-01-16T14:57:46.529 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:46.529 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:46.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.549+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0aa806b2c0 msgr2=0x7f0aa806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.549+0000 7f0acfe00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0aa806b2c0 0x7f0aa806d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0006fd0 tx=0x7f0ac0008040 comp rx=0 tx=0).stop 2024-01-16T14:57:46.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 msgr2=0x7f0ad015a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad015a630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c460 tx=0x7f0ab800c920 comp rx=0 tx=0).stop 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 shutdown_connections 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0aa806b2c0 0x7f0aa806d780 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ad0160920 0x7f0ad015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ad00cfea0 0x7f0ad015a630 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 --2- 172.21.15.141:0/2950546711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ad00cf9f0 0x7f0ad00caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 >> 172.21.15.141:0/2950546711 conn(0x7f0ad00d3090 msgr2=0x7f0ad006af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 shutdown_connections 2024-01-16T14:57:46.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.550+0000 7f0acfe00640 1 -- 172.21.15.141:0/2950546711 wait complete. 2024-01-16T14:57:46.551 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:46.551 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:46.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.769+0000 7fac2fe00640 1 Processor -- start 2024-01-16T14:57:46.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.769+0000 7fac2fe00640 1 -- start start 2024-01-16T14:57:46.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.770+0000 7fac2fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac3013cf10 0x7fac3013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.770+0000 7fac2fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fac300cddf0 0x7fac300b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.770+0000 7fac2fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300ce440 con 0x7fac3013fae0 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.770+0000 7fac2fe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3013ff60 con 0x7fac300cddf0 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.770+0000 7fac2fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac300ce720 con 0x7fac3013cf10 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fac300cddf0 0x7fac300b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fac300cddf0 0x7fac300b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56180/0 (socket says 172.21.15.141:56180) 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2f400640 1 -- 172.21.15.141:0/1977619937 learned_addr learned my addr 172.21.15.141:0/1977619937 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2ea00640 1 --2- 172.21.15.141:0/1977619937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac3013cf10 0x7fac3013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2f400640 1 -- 172.21.15.141:0/1977619937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac3013cf10 msgr2=0x7fac3013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2f400640 1 --2- 172.21.15.141:0/1977619937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac3013cf10 0x7fac3013f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2f400640 1 -- 172.21.15.141:0/1977619937 >> v1:172.21.15.188:6789/0 conn(0x7fac3013fae0 legacy=0x7fac300cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2f400640 1 -- 172.21.15.141:0/1977619937 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac300cea00 con 0x7fac300cddf0 2024-01-16T14:57:46.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.771+0000 7fac2ea00640 1 --2- 172.21.15.141:0/1977619937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac3013cf10 0x7fac3013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:46.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.772+0000 7fac2f400640 1 --2- 172.21.15.141:0/1977619937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac300cddf0 0x7fac300b2590 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fac2800bb10 tx=0x7fac2802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7b79d20908b2c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.772+0000 7fac2e000640 1 -- 172.21.15.141:0/1977619937 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2802ee70 con 0x7fac300cddf0 2024-01-16T14:57:46.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.772+0000 7fac2e000640 1 -- 172.21.15.141:0/1977619937 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac28036440 con 0x7fac300cddf0 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.772+0000 7fac2e000640 1 -- 172.21.15.141:0/1977619937 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac280367d0 con 0x7fac300cddf0 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 -- 172.21.15.141:0/1977619937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac300cddf0 msgr2=0x7fac300b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1977619937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac300cddf0 0x7fac300b2590 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fac2800bb10 tx=0x7fac2802e0c0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 -- 172.21.15.141:0/1977619937 shutdown_connections 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1977619937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac300cddf0 0x7fac300b2590 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1977619937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac3013cf10 0x7fac3013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 -- 172.21.15.141:0/1977619937 >> 172.21.15.141:0/1977619937 conn(0x7fac3006a7d0 msgr2=0x7fac3006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 -- 172.21.15.141:0/1977619937 shutdown_connections 2024-01-16T14:57:46.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.773+0000 7fac2fe00640 1 -- 172.21.15.141:0/1977619937 wait complete. 2024-01-16T14:57:46.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.775+0000 7fac2fe00640 1 Processor -- start 2024-01-16T14:57:46.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.775+0000 7fac2fe00640 1 -- start start 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac2fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac300cddf0 0x7fac30148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac2fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac3013cf10 0x7fac3013d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac2fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac3013fae0 0x7fac300711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac2fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac30149bb0 con 0x7fac3013cf10 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac2fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac30149280 con 0x7fac3013fae0 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac2fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac30149560 con 0x7fac300cddf0 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac3013cf10 0x7fac3013d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac3013cf10 0x7fac3013d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56194/0 (socket says 172.21.15.141:56194) 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 -- 172.21.15.141:0/1039027904 learned_addr learned my addr 172.21.15.141:0/1039027904 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 -- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac300cddf0 msgr2=0x7fac30148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac300cddf0 0x7fac30148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 -- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac3013fae0 msgr2=0x7fac300711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac3013fae0 0x7fac300711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.776+0000 7fac25e00640 1 -- 172.21.15.141:0/1039027904 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2800b770 con 0x7fac3013cf10 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.777+0000 7fac25e00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac3013cf10 0x7fac3013d2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fac1800b3f0 tx=0x7fac1800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.777+0000 7fac27e00640 1 -- 172.21.15.141:0/1039027904 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18011830 con 0x7fac3013cf10 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.777+0000 7fac27e00640 1 -- 172.21.15.141:0/1039027904 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac18011990 con 0x7fac3013cf10 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.777+0000 7fac27e00640 1 -- 172.21.15.141:0/1039027904 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18011d40 con 0x7fac3013cf10 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.777+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac30149840 con 0x7fac3013cf10 2024-01-16T14:57:46.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.777+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3014a040 con 0x7fac3013cf10 2024-01-16T14:57:46.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.778+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac30067450 con 0x7fac3013cf10 2024-01-16T14:57:46.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.779+0000 7fac27e00640 1 -- 172.21.15.141:0/1039027904 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac1801a400 con 0x7fac3013cf10 2024-01-16T14:57:46.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.780+0000 7fac27e00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac0806b2f0 0x7fac0806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:46.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.780+0000 7fac27e00640 1 -- 172.21.15.141:0/1039027904 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac18089a20 con 0x7fac3013cf10 2024-01-16T14:57:46.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.781+0000 7fac2ea00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac0806b2f0 0x7fac0806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:46.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.781+0000 7fac2ea00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac0806b2f0 0x7fac0806d7b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fac20008ea0 tx=0x7fac200083d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:46.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.784+0000 7fac27e00640 1 -- 172.21.15.141:0/1039027904 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac18054890 con 0x7fac3013cf10 2024-01-16T14:57:46.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.958+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac30149cf0 con 0x7fac3013cf10 2024-01-16T14:57:46.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.958+0000 7fac27e00640 1 -- 172.21.15.141:0/1039027904 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac180150a0 con 0x7fac3013cf10 2024-01-16T14:57:46.941 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac0806b2f0 msgr2=0x7fac0806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac0806b2f0 0x7fac0806d7b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fac20008ea0 tx=0x7fac200083d0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac3013cf10 msgr2=0x7fac3013d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac3013cf10 0x7fac3013d2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fac1800b3f0 tx=0x7fac1800b8b0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 shutdown_connections 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac0806b2f0 0x7fac0806d7b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac3013fae0 0x7fac300711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac3013cf10 0x7fac3013d2f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 --2- 172.21.15.141:0/1039027904 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac300cddf0 0x7fac30148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 >> 172.21.15.141:0/1039027904 conn(0x7fac3006a7d0 msgr2=0x7fac30153bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 shutdown_connections 2024-01-16T14:57:46.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:46.961+0000 7fac2fe00640 1 -- 172.21.15.141:0/1039027904 wait complete. 2024-01-16T14:57:46.961 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.179+0000 7ff0b7000640 1 Processor -- start 2024-01-16T14:57:47.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.179+0000 7ff0b7000640 1 -- start start 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0b7000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff0b013df80 0x7ff0b015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0b7000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 0x7ff0b00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0b7000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b006c470 con 0x7ff0b015f9c0 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0b7000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b006c750 con 0x7ff0b013df80 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0b7000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b006ca30 con 0x7ff0b013e360 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0afe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff0b013df80 0x7ff0b015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0afe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff0b013df80 0x7ff0b015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56204/0 (socket says 172.21.15.141:56204) 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0afe00640 1 -- 172.21.15.141:0/1840125545 learned_addr learned my addr 172.21.15.141:0/1840125545 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0afe00640 1 -- 172.21.15.141:0/1840125545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 msgr2=0x7ff0b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0afe00640 1 --2- 172.21.15.141:0/1840125545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 0x7ff0b00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0afe00640 1 -- 172.21.15.141:0/1840125545 >> v1:172.21.15.188:6789/0 conn(0x7ff0b015f9c0 legacy=0x7ff0b01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.181+0000 7ff0afe00640 1 -- 172.21.15.141:0/1840125545 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b006cd10 con 0x7ff0b013df80 2024-01-16T14:57:47.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.182+0000 7ff0afe00640 1 --2- 172.21.15.141:0/1840125545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b015bee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009870 tx=0x7ff0a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbaf40ce8ed5c89c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0aea00640 1 -- 172.21.15.141:0/1840125545 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0031640 con 0x7ff0b013df80 2024-01-16T14:57:47.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0aea00640 1 -- 172.21.15.141:0/1840125545 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a00317a0 con 0x7ff0b013df80 2024-01-16T14:57:47.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0aea00640 1 -- 172.21.15.141:0/1840125545 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0031b30 con 0x7ff0b013df80 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0b7000640 1 -- 172.21.15.141:0/1840125545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 msgr2=0x7ff0b015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0b7000640 1 --2- 172.21.15.141:0/1840125545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b015bee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009870 tx=0x7ff0a002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0b7000640 1 -- 172.21.15.141:0/1840125545 shutdown_connections 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0b7000640 1 --2- 172.21.15.141:0/1840125545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 0x7ff0b00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0b7000640 1 --2- 172.21.15.141:0/1840125545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b015bee0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.183+0000 7ff0b7000640 1 -- 172.21.15.141:0/1840125545 >> 172.21.15.141:0/1840125545 conn(0x7ff0b006a800 msgr2=0x7ff0b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.184+0000 7ff0b7000640 1 -- 172.21.15.141:0/1840125545 shutdown_connections 2024-01-16T14:57:47.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.184+0000 7ff0b7000640 1 -- 172.21.15.141:0/1840125545 wait complete. 2024-01-16T14:57:47.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.185+0000 7ff0b7000640 1 Processor -- start 2024-01-16T14:57:47.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.185+0000 7ff0b7000640 1 -- start start 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.186+0000 7ff0b7000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b0153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.186+0000 7ff0b7000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 0x7ff0b01574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.186+0000 7ff0b7000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0b015f9c0 0x7ff0b0157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.186+0000 7ff0b7000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0159690 con 0x7ff0b013df80 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.186+0000 7ff0b7000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0158070 con 0x7ff0b015f9c0 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.186+0000 7ff0b7000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0158350 con 0x7ff0b013e360 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b0153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b0153420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56212/0 (socket says 172.21.15.141:56212) 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 -- 172.21.15.141:0/3394207346 learned_addr learned my addr 172.21.15.141:0/3394207346 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 -- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 msgr2=0x7ff0b01574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 0x7ff0b01574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 -- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0b015f9c0 msgr2=0x7ff0b0157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0b5000640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0b015f9c0 0x7ff0b0157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0b015f9c0 0x7ff0b0157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 -- 172.21.15.141:0/3394207346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00094d0 con 0x7ff0b013df80 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.187+0000 7ff0afe00640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b0153420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00090b0 tx=0x7ff0a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.188+0000 7ff0a7e00640 1 -- 172.21.15.141:0/3394207346 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a003ac60 con 0x7ff0b013df80 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.188+0000 7ff0a7e00640 1 -- 172.21.15.141:0/3394207346 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a003adc0 con 0x7ff0b013df80 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.188+0000 7ff0a7e00640 1 -- 172.21.15.141:0/3394207346 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0044630 con 0x7ff0b013df80 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.188+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b01598e0 con 0x7ff0b013df80 2024-01-16T14:57:47.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.188+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b00a8c10 con 0x7ff0b013df80 2024-01-16T14:57:47.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.189+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b0159e00 con 0x7ff0b013df80 2024-01-16T14:57:47.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.190+0000 7ff0a7e00640 1 -- 172.21.15.141:0/3394207346 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff0a002f050 con 0x7ff0b013df80 2024-01-16T14:57:47.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.191+0000 7ff0a7e00640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff08806b2c0 0x7ff08806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.191+0000 7ff0a7e00640 1 -- 172.21.15.141:0/3394207346 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0a00abb10 con 0x7ff0b013df80 2024-01-16T14:57:47.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.192+0000 7ff0af400640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff08806b2c0 0x7ff08806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.193+0000 7ff0af400640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff08806b2c0 0x7ff08806d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0154b60 tx=0x7ff098006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.196+0000 7ff0a7e00640 1 -- 172.21.15.141:0/3394207346 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0a0076b10 con 0x7ff0b013df80 2024-01-16T14:57:47.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.392+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff0b00675d0 con 0x7ff0b013df80 2024-01-16T14:57:47.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.393+0000 7ff0a7e00640 1 -- 172.21.15.141:0/3394207346 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff0a007a130 con 0x7ff0b013df80 2024-01-16T14:57:47.376 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff08806b2c0 msgr2=0x7ff08806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff08806b2c0 0x7ff08806d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0154b60 tx=0x7ff098006c90 comp rx=0 tx=0).stop 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 msgr2=0x7ff0b0153420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b0153420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00090b0 tx=0x7ff0a00046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 shutdown_connections 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff08806b2c0 0x7ff08806d780 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0b015f9c0 0x7ff0b0157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0b013e360 0x7ff0b01574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 --2- 172.21.15.141:0/3394207346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0b013df80 0x7ff0b0153420 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 >> 172.21.15.141:0/3394207346 conn(0x7ff0b006a800 msgr2=0x7ff0b015e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 shutdown_connections 2024-01-16T14:57:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.396+0000 7ff0b7000640 1 -- 172.21.15.141:0/3394207346 wait complete. 2024-01-16T14:57:47.397 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:47.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.618+0000 7f306d600640 1 Processor -- start 2024-01-16T14:57:47.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.618+0000 7f306d600640 1 -- start start 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f306d600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3068143ae0 0x7f30680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f306d600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680cfc30 con 0x7f306815e3b0 2024-01-16T14:57:47.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f306d600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680b1ee0 con 0x7f30680cf7d0 2024-01-16T14:57:47.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f306d600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30680b2780 con 0x7f3068143ae0 2024-01-16T14:57:47.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f3067400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f306815e3b0 0x7f30680702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37284/0 (socket says 172.21.15.141:37284) 2024-01-16T14:57:47.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f3067400640 1 -- 172.21.15.141:0/1920281012 learned_addr learned my addr 172.21.15.141:0/1920281012 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:47.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f3066a00640 1 --2- 172.21.15.141:0/1920281012 >> v2:172.21.15.188:3300/0 conn(0x7f3068143ae0 0x7f30680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.620+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825744182 0 0) 0x7f30680cfc30 con 0x7f306815e3b0 2024-01-16T14:57:47.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3048003520 con 0x7f306815e3b0 2024-01-16T14:57:47.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556296466 0 0) 0x7f3048003520 con 0x7f306815e3b0 2024-01-16T14:57:47.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 >> v2:172.21.15.188:3300/0 conn(0x7f3068143ae0 msgr2=0x7f30680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 --2- 172.21.15.141:0/1920281012 >> v2:172.21.15.188:3300/0 conn(0x7f3068143ae0 0x7f30680cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 >> v1:172.21.15.188:6789/0 conn(0x7f30680cf7d0 legacy=0x7f306815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306815e990 con 0x7f306815e3b0 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3060001670 con 0x7f306815e3b0 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 >> v1:172.21.15.141:6789/0 conn(0x7f306815e3b0 legacy=0x7f30680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3048003970 0x7f3048003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30480047c0 0x7f3048004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f304800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680b2780 con 0x7f3048005180 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304800c020 con 0x7f30480047c0 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304800e3f0 con 0x7f3048003970 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.621+0000 7f3067400640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f304800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f3066a00640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3048003970 0x7f3048003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f305de00640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30480047c0 0x7f3048004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f3067400640 1 -- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3048003970 msgr2=0x7f3048003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f3067400640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3048003970 0x7f3048003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f3067400640 1 -- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30480047c0 msgr2=0x7f3048004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f3067400640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30480047c0 0x7f3048004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f3067400640 1 -- 172.21.15.141:0/1920281012 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048003520 con 0x7f3048005180 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f3066a00640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3048003970 0x7f3048003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.622+0000 7f305de00640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30480047c0 0x7f3048004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.623+0000 7f3067400640 1 -- 172.21.15.141:0/1920281012 reap_dead start 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.623+0000 7f3067400640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f304800b980 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f306000c190 tx=0x7f30480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d5a62231f5b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.623+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060005ad0 con 0x7f3048005180 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.623+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3060005c50 con 0x7f3048005180 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.623+0000 7f3066000640 1 -- 172.21.15.141:0/1920281012 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306002c8b0 con 0x7f3048005180 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.624+0000 7f306d600640 1 -- 172.21.15.141:0/1920281012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 msgr2=0x7f304800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.624+0000 7f306d600640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f304800b980 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f306000c190 tx=0x7f30480057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.624+0000 7f306d600640 1 -- 172.21.15.141:0/1920281012 shutdown_connections 2024-01-16T14:57:47.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.624+0000 7f306d600640 1 --2- 172.21.15.141:0/1920281012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f304800b980 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.624+0000 7f306d600640 1 -- 172.21.15.141:0/1920281012 >> 172.21.15.141:0/1920281012 conn(0x7f306806a800 msgr2=0x7f306806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.624+0000 7f306d600640 1 -- 172.21.15.141:0/1920281012 shutdown_connections 2024-01-16T14:57:47.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.624+0000 7f306d600640 1 -- 172.21.15.141:0/1920281012 wait complete. 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.626+0000 7f306d600640 1 Processor -- start 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.626+0000 7f306d600640 1 -- start start 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f306d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f3068136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f306d600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3068134410 0x7f3068159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f306d600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30681359e0 0x7f3068135e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f306d600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3068135550 con 0x7f3048005180 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f306d600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30681373f0 con 0x7f30681359e0 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f306d600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3068134940 con 0x7f3068134410 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f3068136eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f3068136eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56224/0 (socket says 172.21.15.141:56224) 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 -- 172.21.15.141:0/1096179814 learned_addr learned my addr 172.21.15.141:0/1096179814 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 -- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3068134410 msgr2=0x7f3068159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3068134410 0x7f3068159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 -- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30681359e0 msgr2=0x7f3068135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30681359e0 0x7f3068135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.627+0000 7f3066a00640 1 -- 172.21.15.141:0/1096179814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068134c20 con 0x7f3048005180 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.628+0000 7f3066a00640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f3068136eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f30580108e0 tx=0x7f3058010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.628+0000 7f305f400640 1 -- 172.21.15.141:0/1096179814 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305800eba0 con 0x7f3048005180 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.628+0000 7f305f400640 1 -- 172.21.15.141:0/1096179814 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305800ed00 con 0x7f3048005180 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.628+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3068134f00 con 0x7f3048005180 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.628+0000 7f305f400640 1 -- 172.21.15.141:0/1096179814 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305801c600 con 0x7f3048005180 2024-01-16T14:57:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.628+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3068159e40 con 0x7f3048005180 2024-01-16T14:57:47.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.629+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30680675d0 con 0x7f3048005180 2024-01-16T14:57:47.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.631+0000 7f305f400640 1 -- 172.21.15.141:0/1096179814 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f305801c760 con 0x7f3048005180 2024-01-16T14:57:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.632+0000 7f305f400640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f304006b2f0 0x7f304006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.632+0000 7f305f400640 1 -- 172.21.15.141:0/1096179814 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f305808a190 con 0x7f3048005180 2024-01-16T14:57:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.632+0000 7f305de00640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f304006b2f0 0x7f304006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:47.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.633+0000 7f305de00640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f304006b2f0 0x7f304006d7b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30500012e0 tx=0x7f3050009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.636+0000 7f305f400640 1 -- 172.21.15.141:0/1096179814 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30580550b0 con 0x7f3048005180 2024-01-16T14:57:47.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.825+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f30680cea50 con 0x7f3048005180 2024-01-16T14:57:47.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.826+0000 7f305f400640 1 -- 172.21.15.141:0/1096179814 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f30580586d0 con 0x7f3048005180 2024-01-16T14:57:47.808 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:47.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f304006b2f0 msgr2=0x7f304006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f304006b2f0 0x7f304006d7b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30500012e0 tx=0x7f3050009040 comp rx=0 tx=0).stop 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 msgr2=0x7f3068136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f3068136eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f30580108e0 tx=0x7f3058010da0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 shutdown_connections 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f304006b2f0 0x7f304006d7b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30681359e0 0x7f3068135e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3068134410 0x7f3068159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 --2- 172.21.15.141:0/1096179814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3048005180 0x7f3068136eb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 >> 172.21.15.141:0/1096179814 conn(0x7f306806a800 msgr2=0x7f306815bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 shutdown_connections 2024-01-16T14:57:47.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:47.829+0000 7f306d600640 1 -- 172.21.15.141:0/1096179814 wait complete. 2024-01-16T14:57:47.830 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T14:57:48.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.058+0000 7f5960200640 1 Processor -- start 2024-01-16T14:57:48.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.058+0000 7f5960200640 1 -- start start 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f5960200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59580b3ae0 0x7f595815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f5960200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f59580cf290 0x7f59580cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f5960200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5958070c40 con 0x7f59580ae5a0 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f5960200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5958071450 con 0x7f59580b3ae0 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f5960200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59580cfc60 con 0x7f59580cf290 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f595ce00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59580b3ae0 0x7f595815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f595ce00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59580b3ae0 0x7f595815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56228/0 (socket says 172.21.15.141:56228) 2024-01-16T14:57:48.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f595ce00640 1 -- 172.21.15.141:0/2126452047 learned_addr learned my addr 172.21.15.141:0/2126452047 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:48.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f595ce00640 1 -- 172.21.15.141:0/2126452047 >> v2:172.21.15.188:3300/0 conn(0x7f59580cf290 msgr2=0x7f59580cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:48.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.060+0000 7f595ce00640 1 --2- 172.21.15.141:0/2126452047 >> v2:172.21.15.188:3300/0 conn(0x7f59580cf290 0x7f59580cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.061+0000 7f595ce00640 1 -- 172.21.15.141:0/2126452047 >> v1:172.21.15.141:6789/0 conn(0x7f59580ae5a0 legacy=0x7f595806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.061+0000 7f595ce00640 1 -- 172.21.15.141:0/2126452047 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595806c470 con 0x7f59580b3ae0 2024-01-16T14:57:48.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.061+0000 7f5957e00640 1 -- 172.21.15.141:0/2126452047 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164285911 0 0) 0x7f5958070c40 con 0x7f59580ae5a0 2024-01-16T14:57:48.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.062+0000 7f595ce00640 1 --2- 172.21.15.141:0/2126452047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580b3ae0 0x7f595815e430 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f59580b3f70 tx=0x7f594002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c1bf4de2e88d58 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.062+0000 7f5957e00640 1 -- 172.21.15.141:0/2126452047 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594002f850 con 0x7f59580b3ae0 2024-01-16T14:57:48.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.062+0000 7f5957e00640 1 -- 172.21.15.141:0/2126452047 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594002f9b0 con 0x7f59580b3ae0 2024-01-16T14:57:48.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.062+0000 7f5957e00640 1 -- 172.21.15.141:0/2126452047 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594002fd40 con 0x7f59580b3ae0 2024-01-16T14:57:48.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.062+0000 7f5960200640 1 -- 172.21.15.141:0/2126452047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580b3ae0 msgr2=0x7f595815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.062+0000 7f5960200640 1 --2- 172.21.15.141:0/2126452047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580b3ae0 0x7f595815e430 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f59580b3f70 tx=0x7f594002ba30 comp rx=0 tx=0).stop 2024-01-16T14:57:48.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.063+0000 7f5960200640 1 -- 172.21.15.141:0/2126452047 shutdown_connections 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.063+0000 7f5960200640 1 --2- 172.21.15.141:0/2126452047 >> v2:172.21.15.188:3300/0 conn(0x7f59580cf290 0x7f59580cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.063+0000 7f5960200640 1 --2- 172.21.15.141:0/2126452047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580b3ae0 0x7f595815e430 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.063+0000 7f5960200640 1 -- 172.21.15.141:0/2126452047 >> 172.21.15.141:0/2126452047 conn(0x7f595806a800 msgr2=0x7f595806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.063+0000 7f5960200640 1 -- 172.21.15.141:0/2126452047 shutdown_connections 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.063+0000 7f5960200640 1 -- 172.21.15.141:0/2126452047 wait complete. 2024-01-16T14:57:48.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.064+0000 7f5960200640 1 Processor -- start 2024-01-16T14:57:48.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.064+0000 7f5960200640 1 -- start start 2024-01-16T14:57:48.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.065+0000 7f5960200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580ae5a0 0x7f59580ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.065+0000 7f5960200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59580b3ae0 0x7f5958148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.065+0000 7f5960200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59580cf290 0x7f59581f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.065+0000 7f5960200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59581f48c0 con 0x7f59580ae5a0 2024-01-16T14:57:48.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.065+0000 7f5960200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5958148ec0 con 0x7f59580cf290 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.065+0000 7f5960200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59581f3b30 con 0x7f59580b3ae0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580ae5a0 0x7f59580ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580ae5a0 0x7f59580ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56232/0 (socket says 172.21.15.141:56232) 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 -- 172.21.15.141:0/3951953088 learned_addr learned my addr 172.21.15.141:0/3951953088 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595ce00640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59580b3ae0 0x7f5958148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 -- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59580b3ae0 msgr2=0x7f5958148980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595e200640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59580cf290 0x7f59581f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59580b3ae0 0x7f5958148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 -- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59580cf290 msgr2=0x7f59581f3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59580cf290 0x7f59581f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 -- 172.21.15.141:0/3951953088 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400094d0 con 0x7f59580ae5a0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.066+0000 7f595d800640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580ae5a0 0x7f59580ce2c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f594800c890 tx=0x7f594800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.067+0000 7f5955600640 1 -- 172.21.15.141:0/3951953088 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948003da0 con 0x7f59580ae5a0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.067+0000 7f5955600640 1 -- 172.21.15.141:0/3951953088 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5948005070 con 0x7f59580ae5a0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.067+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59581f3e10 con 0x7f59580ae5a0 2024-01-16T14:57:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.067+0000 7f5955600640 1 -- 172.21.15.141:0/3951953088 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948005420 con 0x7f59580ae5a0 2024-01-16T14:57:48.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.067+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59581f43d0 con 0x7f59580ae5a0 2024-01-16T14:57:48.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.068+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59581f4a00 con 0x7f59580ae5a0 2024-01-16T14:57:48.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.069+0000 7f5955600640 1 -- 172.21.15.141:0/3951953088 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5948005580 con 0x7f59580ae5a0 2024-01-16T14:57:48.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.070+0000 7f5955600640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f593006b2c0 0x7f593006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.070+0000 7f5955600640 1 -- 172.21.15.141:0/3951953088 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59480894c0 con 0x7f59580ae5a0 2024-01-16T14:57:48.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.071+0000 7f595ce00640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f593006b2c0 0x7f593006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.072+0000 7f595ce00640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f593006b2c0 0x7f593006d780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5940036000 tx=0x7f5940039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.075+0000 7f5955600640 1 -- 172.21.15.141:0/3951953088 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59581f4a00 con 0x7f59580ae5a0 2024-01-16T14:57:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.316+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f59580675d0 con 0x7f59580ae5a0 2024-01-16T14:57:48.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.580+0000 7f5955600640 1 -- 172.21.15.141:0/3951953088 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v54) v1 ==== 155+0+0 (secure 0 0 0) 0x7f59480543c0 con 0x7f59580ae5a0 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f593006b2c0 msgr2=0x7f593006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f593006b2c0 0x7f593006d780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5940036000 tx=0x7f5940039040 comp rx=0 tx=0).stop 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580ae5a0 msgr2=0x7f59580ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580ae5a0 0x7f59580ce2c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f594800c890 tx=0x7f594800cd50 comp rx=0 tx=0).stop 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 shutdown_connections 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f593006b2c0 0x7f593006d780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59580cf290 0x7f59581f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59580b3ae0 0x7f5958148980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 --2- 172.21.15.141:0/3951953088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59580ae5a0 0x7f59580ce2c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.583+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 >> 172.21.15.141:0/3951953088 conn(0x7f595806a800 msgr2=0x7f59581618d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:48.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.584+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 shutdown_connections 2024-01-16T14:57:48.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.584+0000 7f5960200640 1 -- 172.21.15.141:0/3951953088 wait complete. 2024-01-16T14:57:48.586 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T14:57:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.819+0000 7f6045c00640 1 Processor -- start 2024-01-16T14:57:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.819+0000 7f6045c00640 1 -- start start 2024-01-16T14:57:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.820+0000 7f6045c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400b3ae0 0x7f604015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.820+0000 7f6045c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f60400cf290 0x7f60400cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f6045c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6040070c40 con 0x7f60400ae5a0 2024-01-16T14:57:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f6045c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6040071450 con 0x7f60400cf290 2024-01-16T14:57:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f6045c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60400cfc60 con 0x7f60400b3ae0 2024-01-16T14:57:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f60400ae5a0 0x7f604006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37302/0 (socket says 172.21.15.141:37302) 2024-01-16T14:57:48.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603ea00640 1 -- 172.21.15.141:0/2514104963 learned_addr learned my addr 172.21.15.141:0/2514104963 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:48.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603f400640 1 --2- 172.21.15.141:0/2514104963 >> v2:172.21.15.141:3300/0 conn(0x7f60400cf290 0x7f60400cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603e000640 1 -- 172.21.15.141:0/2514104963 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993324186 0 0) 0x7f6040070c40 con 0x7f60400ae5a0 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603e000640 1 -- 172.21.15.141:0/2514104963 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6020003520 con 0x7f60400ae5a0 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f6035e00640 1 --2- 172.21.15.141:0/2514104963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400b3ae0 0x7f604015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603f400640 1 -- 172.21.15.141:0/2514104963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400b3ae0 msgr2=0x7f604015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603f400640 1 --2- 172.21.15.141:0/2514104963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400b3ae0 0x7f604015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603f400640 1 -- 172.21.15.141:0/2514104963 >> v1:172.21.15.141:6789/0 conn(0x7f60400ae5a0 legacy=0x7f604006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f603f400640 1 -- 172.21.15.141:0/2514104963 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604006c470 con 0x7f60400cf290 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.821+0000 7f6035e00640 1 --2- 172.21.15.141:0/2514104963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400b3ae0 0x7f604015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:48.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f603f400640 1 --2- 172.21.15.141:0/2514104963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400cf290 0x7f60400cf720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6038009d90 tx=0x7f603802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b31eb8227f3a1d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f603e000640 1 -- 172.21.15.141:0/2514104963 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603802cce0 con 0x7f60400cf290 2024-01-16T14:57:48.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f603e000640 1 -- 172.21.15.141:0/2514104963 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603802ce40 con 0x7f60400cf290 2024-01-16T14:57:48.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f603e000640 1 -- 172.21.15.141:0/2514104963 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038034660 con 0x7f60400cf290 2024-01-16T14:57:48.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f6045c00640 1 -- 172.21.15.141:0/2514104963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400cf290 msgr2=0x7f60400cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f6045c00640 1 --2- 172.21.15.141:0/2514104963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400cf290 0x7f60400cf720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6038009d90 tx=0x7f603802bf30 comp rx=0 tx=0).stop 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f6045c00640 1 -- 172.21.15.141:0/2514104963 shutdown_connections 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f6045c00640 1 --2- 172.21.15.141:0/2514104963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400cf290 0x7f60400cf720 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f6045c00640 1 --2- 172.21.15.141:0/2514104963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400b3ae0 0x7f604015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.823+0000 7f6045c00640 1 -- 172.21.15.141:0/2514104963 >> 172.21.15.141:0/2514104963 conn(0x7f604006a800 msgr2=0x7f604006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.824+0000 7f6045c00640 1 -- 172.21.15.141:0/2514104963 shutdown_connections 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.824+0000 7f6045c00640 1 -- 172.21.15.141:0/2514104963 wait complete. 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.825+0000 7f6045c00640 1 Processor -- start 2024-01-16T14:57:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.825+0000 7f6045c00640 1 -- start start 2024-01-16T14:57:48.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.826+0000 7f6045c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60400ae5a0 0x7f60401606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.826+0000 7f6045c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400b3ae0 0x7f604015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.826+0000 7f6045c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400cf290 0x7f60400b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.826+0000 7f6045c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60401529b0 con 0x7f60400b3ae0 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.826+0000 7f6045c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604015d430 con 0x7f60400ae5a0 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.826+0000 7f6045c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6040159f10 con 0x7f60400cf290 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400b3ae0 0x7f604015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400b3ae0 0x7f604015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56242/0 (socket says 172.21.15.141:56242) 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 -- 172.21.15.141:0/2289464868 learned_addr learned my addr 172.21.15.141:0/2289464868 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f603f400640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400cf290 0x7f60400b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f603ea00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60400ae5a0 0x7f60401606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 -- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400cf290 msgr2=0x7f60400b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400cf290 0x7f60400b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 -- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60400ae5a0 msgr2=0x7f60401606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:48.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60400ae5a0 0x7f60401606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:48.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 -- 172.21.15.141:0/2289464868 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038009a70 con 0x7f60400b3ae0 2024-01-16T14:57:48.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.827+0000 7f6035e00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400b3ae0 0x7f604015cef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f602800c820 tx=0x7f602800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.828+0000 7f6037400640 1 -- 172.21.15.141:0/2289464868 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028007a30 con 0x7f60400b3ae0 2024-01-16T14:57:48.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.828+0000 7f6037400640 1 -- 172.21.15.141:0/2289464868 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028007b90 con 0x7f60400b3ae0 2024-01-16T14:57:48.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.828+0000 7f6037400640 1 -- 172.21.15.141:0/2289464868 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602800ce70 con 0x7f60400b3ae0 2024-01-16T14:57:48.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.828+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6040152080 con 0x7f60400b3ae0 2024-01-16T14:57:48.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.828+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6040152640 con 0x7f60400b3ae0 2024-01-16T14:57:48.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.829+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6040152b60 con 0x7f60400b3ae0 2024-01-16T14:57:48.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.830+0000 7f6037400640 1 -- 172.21.15.141:0/2289464868 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f60280113f0 con 0x7f60400b3ae0 2024-01-16T14:57:48.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.831+0000 7f6037400640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f601806b2f0 0x7f601806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:48.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.831+0000 7f6037400640 1 -- 172.21.15.141:0/2289464868 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6028089e10 con 0x7f60400b3ae0 2024-01-16T14:57:48.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.831+0000 7f603ea00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f601806b2f0 0x7f601806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:48.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.832+0000 7f603ea00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f601806b2f0 0x7f601806d7b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f60300011e0 tx=0x7f6030009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:48.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:48.835+0000 7f6037400640 1 -- 172.21.15.141:0/2289464868 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6040152b60 con 0x7f60400b3ae0 2024-01-16T14:57:49.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.025+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6040152b60 con 0x7f60400b3ae0 2024-01-16T14:57:49.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.032+0000 7f6037400640 1 -- 172.21.15.141:0/2289464868 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6040152b60 con 0x7f60400b3ae0 2024-01-16T14:57:49.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.035+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f601806b2f0 msgr2=0x7f601806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.035+0000 7f6045c00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f601806b2f0 0x7f601806d7b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f60300011e0 tx=0x7f6030009040 comp rx=0 tx=0).stop 2024-01-16T14:57:49.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.035+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400b3ae0 msgr2=0x7f604015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.035+0000 7f6045c00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400b3ae0 0x7f604015cef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f602800c820 tx=0x7f602800cc20 comp rx=0 tx=0).stop 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 shutdown_connections 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f601806b2f0 0x7f601806d7b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60400cf290 0x7f60400b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60400b3ae0 0x7f604015cef0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 --2- 172.21.15.141:0/2289464868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60400ae5a0 0x7f60401606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 >> 172.21.15.141:0/2289464868 conn(0x7f604006a800 msgr2=0x7f604014fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 shutdown_connections 2024-01-16T14:57:49.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.036+0000 7f6045c00640 1 -- 172.21.15.141:0/2289464868 wait complete. 2024-01-16T14:57:49.037 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T14:57:49.037 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:57:49.130 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:57:49.252 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T14:57:49.252 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:57:49.253 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:49.310 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:57:49.310 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:49.364 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:49.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.586+0000 7feb4dc00640 1 Processor -- start 2024-01-16T14:57:49.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.586+0000 7feb4dc00640 1 -- start start 2024-01-16T14:57:49.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.587+0000 7feb4dc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feb48160920 0x7feb480b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.587+0000 7feb4dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb48161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.587+0000 7feb4dc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb480d0a10 con 0x7feb480cf9f0 2024-01-16T14:57:49.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.587+0000 7feb4dc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb480d0330 con 0x7feb48160920 2024-01-16T14:57:49.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.587+0000 7feb4dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480d1f50 con 0x7feb480cfea0 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feb48160920 0x7feb480b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7feb480cf9f0 0x7feb48161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37312/0 (socket says 172.21.15.141:37312) 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47400640 1 -- 172.21.15.141:0/4207895384 learned_addr learned my addr 172.21.15.141:0/4207895384 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feb48160920 0x7feb480b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56256/0 (socket says 172.21.15.141:56256) 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb4d200640 1 --2- 172.21.15.141:0/4207895384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb48161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47e00640 1 -- 172.21.15.141:0/4207895384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 msgr2=0x7feb48161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47e00640 1 --2- 172.21.15.141:0/4207895384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb48161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb46a00640 1 -- 172.21.15.141:0/4207895384 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285434756 0 0) 0x7feb480d0a10 con 0x7feb480cf9f0 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47e00640 1 -- 172.21.15.141:0/4207895384 >> v1:172.21.15.141:6789/0 conn(0x7feb480cf9f0 legacy=0x7feb48161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb47e00640 1 -- 172.21.15.141:0/4207895384 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb480cf080 con 0x7feb48160920 2024-01-16T14:57:49.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.588+0000 7feb4d200640 1 --2- 172.21.15.141:0/4207895384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb48161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.589+0000 7feb47e00640 1 --2- 172.21.15.141:0/4207895384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb48160920 0x7feb480b3f30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d0954eccb2fb33 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb46a00640 1 -- 172.21.15.141:0/4207895384 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4002cc50 con 0x7feb48160920 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb46a00640 1 -- 172.21.15.141:0/4207895384 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4002cdb0 con 0x7feb48160920 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb46a00640 1 -- 172.21.15.141:0/4207895384 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40034660 con 0x7feb48160920 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 -- 172.21.15.141:0/4207895384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb48160920 msgr2=0x7feb480b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 --2- 172.21.15.141:0/4207895384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb48160920 0x7feb480b3f30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4002bea0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 -- 172.21.15.141:0/4207895384 shutdown_connections 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 --2- 172.21.15.141:0/4207895384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb48161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 --2- 172.21.15.141:0/4207895384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb48160920 0x7feb480b3f30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 -- 172.21.15.141:0/4207895384 >> 172.21.15.141:0/4207895384 conn(0x7feb480d3090 msgr2=0x7feb480d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:49.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 -- 172.21.15.141:0/4207895384 shutdown_connections 2024-01-16T14:57:49.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.590+0000 7feb4dc00640 1 -- 172.21.15.141:0/4207895384 wait complete. 2024-01-16T14:57:49.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.592+0000 7feb4dc00640 1 Processor -- start 2024-01-16T14:57:49.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.592+0000 7feb4dc00640 1 -- start start 2024-01-16T14:57:49.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.593+0000 7feb4dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb480cf9f0 0x7feb4814f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.593+0000 7feb4dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb481f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.593+0000 7feb4dc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb48160920 0x7feb4815f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.593+0000 7feb4dc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb480b72d0 con 0x7feb480cf9f0 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.593+0000 7feb4dc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4815f690 con 0x7feb48160920 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.593+0000 7feb4dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480b6ca0 con 0x7feb480cfea0 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb480cf9f0 0x7feb4814f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb480cf9f0 0x7feb4814f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56272/0 (socket says 172.21.15.141:56272) 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 -- 172.21.15.141:0/2937525226 learned_addr learned my addr 172.21.15.141:0/2937525226 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 -- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 msgr2=0x7feb481f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb481f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 -- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb48160920 msgr2=0x7feb4815f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb48160920 0x7feb4815f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 -- 172.21.15.141:0/2937525226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40009a70 con 0x7feb480cf9f0 2024-01-16T14:57:49.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.594+0000 7feb47e00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb480cf9f0 0x7feb4814f460 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.595+0000 7feb3fe00640 1 -- 172.21.15.141:0/2937525226 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40040650 con 0x7feb480cf9f0 2024-01-16T14:57:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.595+0000 7feb3fe00640 1 -- 172.21.15.141:0/2937525226 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb400407b0 con 0x7feb480cf9f0 2024-01-16T14:57:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.595+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4815f930 con 0x7feb480cf9f0 2024-01-16T14:57:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.595+0000 7feb3fe00640 1 -- 172.21.15.141:0/2937525226 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40040b60 con 0x7feb480cf9f0 2024-01-16T14:57:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.595+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb480b6f60 con 0x7feb480cf9f0 2024-01-16T14:57:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.596+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb48067450 con 0x7feb480cf9f0 2024-01-16T14:57:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.597+0000 7feb3fe00640 1 -- 172.21.15.141:0/2937525226 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feb40040cc0 con 0x7feb480cf9f0 2024-01-16T14:57:49.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.598+0000 7feb3fe00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb2006b2c0 0x7feb2006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:49.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.598+0000 7feb3fe00640 1 -- 172.21.15.141:0/2937525226 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb400ac9f0 con 0x7feb480cf9f0 2024-01-16T14:57:49.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.599+0000 7feb47400640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb2006b2c0 0x7feb2006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:49.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.600+0000 7feb47400640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb2006b2c0 0x7feb2006d780 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb30007fd0 tx=0x7feb30007960 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:49.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.603+0000 7feb3fe00640 1 -- 172.21.15.141:0/2937525226 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb40077850 con 0x7feb480cf9f0 2024-01-16T14:57:49.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.794+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb4808c7e0 con 0x7feb480cf9f0 2024-01-16T14:57:49.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.797+0000 7feb3fe00640 1 -- 172.21.15.141:0/2937525226 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7feb4007ae70 con 0x7feb480cf9f0 2024-01-16T14:57:49.780 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:49.780 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb2006b2c0 msgr2=0x7feb2006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb2006b2c0 0x7feb2006d780 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb30007fd0 tx=0x7feb30007960 comp rx=0 tx=0).stop 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb480cf9f0 msgr2=0x7feb4814f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb480cf9f0 0x7feb4814f460 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb40004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 shutdown_connections 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb2006b2c0 0x7feb2006d780 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb48160920 0x7feb4815f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb480cfea0 0x7feb481f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 --2- 172.21.15.141:0/2937525226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb480cf9f0 0x7feb4814f460 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:49.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 >> 172.21.15.141:0/2937525226 conn(0x7feb480d3090 msgr2=0x7feb4815b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:49.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 shutdown_connections 2024-01-16T14:57:49.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:49.800+0000 7feb4dc00640 1 -- 172.21.15.141:0/2937525226 wait complete. 2024-01-16T14:57:49.801 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:49.802 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:50.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.032+0000 7f441ec00640 1 Processor -- start 2024-01-16T14:57:50.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.033+0000 7f441ec00640 1 -- start start 2024-01-16T14:57:50.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.034+0000 7f441ec00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f44180cf9a0 0x7f441815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.034+0000 7f441ec00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44180b1ee0 con 0x7f441815e1e0 2024-01-16T14:57:50.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.034+0000 7f441ec00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44180b2780 con 0x7f4418143ae0 2024-01-16T14:57:50.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.034+0000 7f441ec00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441815e990 con 0x7f44180cf9a0 2024-01-16T14:57:50.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.034+0000 7f441cc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f441815e1e0 0x7f44180702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37320/0 (socket says 172.21.15.141:37320) 2024-01-16T14:57:50.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.034+0000 7f441cc00640 1 -- 172.21.15.141:0/1957559046 learned_addr learned my addr 172.21.15.141:0/1957559046 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:50.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.034+0000 7f440f200640 1 --2- 172.21.15.141:0/1957559046 >> v2:172.21.15.141:3300/0 conn(0x7f44180cf9a0 0x7f441815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.035+0000 7f440f200640 1 -- 172.21.15.141:0/1957559046 >> v1:172.21.15.188:6789/0 conn(0x7f4418143ae0 legacy=0x7f44180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.035+0000 7f440f200640 1 -- 172.21.15.141:0/1957559046 >> v1:172.21.15.141:6789/0 conn(0x7f441815e1e0 legacy=0x7f44180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.035+0000 7f440f200640 1 -- 172.21.15.141:0/1957559046 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180b2130 con 0x7f44180cf9a0 2024-01-16T14:57:50.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.035+0000 7f4417400640 1 -- 172.21.15.141:0/1957559046 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781723522 0 0) 0x7f44180b1ee0 con 0x7f441815e1e0 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.036+0000 7f440f200640 1 --2- 172.21.15.141:0/1957559046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44180cf9a0 0x7f441815dca0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f44000097a0 tx=0x7f440002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ed9556361b8652 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.036+0000 7f4417400640 1 -- 172.21.15.141:0/1957559046 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400030610 con 0x7f44180cf9a0 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.036+0000 7f4417400640 1 -- 172.21.15.141:0/1957559046 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4400030770 con 0x7f44180cf9a0 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.036+0000 7f4417400640 1 -- 172.21.15.141:0/1957559046 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400030b00 con 0x7f44180cf9a0 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.037+0000 7f441ec00640 1 -- 172.21.15.141:0/1957559046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44180cf9a0 msgr2=0x7f441815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.037+0000 7f441ec00640 1 --2- 172.21.15.141:0/1957559046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44180cf9a0 0x7f441815dca0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f44000097a0 tx=0x7f440002b960 comp rx=0 tx=0).stop 2024-01-16T14:57:50.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.037+0000 7f441ec00640 1 -- 172.21.15.141:0/1957559046 shutdown_connections 2024-01-16T14:57:50.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.037+0000 7f441ec00640 1 --2- 172.21.15.141:0/1957559046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44180cf9a0 0x7f441815dca0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.037+0000 7f441ec00640 1 -- 172.21.15.141:0/1957559046 >> 172.21.15.141:0/1957559046 conn(0x7f441806a800 msgr2=0x7f441806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.037+0000 7f441ec00640 1 -- 172.21.15.141:0/1957559046 shutdown_connections 2024-01-16T14:57:50.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.037+0000 7f441ec00640 1 -- 172.21.15.141:0/1957559046 wait complete. 2024-01-16T14:57:50.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.039+0000 7f441ec00640 1 Processor -- start 2024-01-16T14:57:50.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.039+0000 7f441ec00640 1 -- start start 2024-01-16T14:57:50.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f441ec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44180cf9a0 0x7f4418159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f441ec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4418143ae0 0x7f4418134980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f441ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f441815e1e0 0x7f4418132910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f441ec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4418132480 con 0x7f4418143ae0 2024-01-16T14:57:50.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f441ec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4418131870 con 0x7f44180cf9a0 2024-01-16T14:57:50.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f441ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4418131b50 con 0x7f441815e1e0 2024-01-16T14:57:50.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f440f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4418143ae0 0x7f4418134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f440f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4418143ae0 0x7f4418134980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56288/0 (socket says 172.21.15.141:56288) 2024-01-16T14:57:50.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.040+0000 7f440f200640 1 -- 172.21.15.141:0/2341095108 learned_addr learned my addr 172.21.15.141:0/2341095108 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:50.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f441cc00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f441815e1e0 0x7f4418132910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f4417e00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44180cf9a0 0x7f4418159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f440f200640 1 -- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f441815e1e0 msgr2=0x7f4418132910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f440f200640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f441815e1e0 0x7f4418132910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f440f200640 1 -- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44180cf9a0 msgr2=0x7f4418159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f440f200640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44180cf9a0 0x7f4418159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f440f200640 1 -- 172.21.15.141:0/2341095108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400009400 con 0x7f4418143ae0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f4417e00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44180cf9a0 0x7f4418159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f441cc00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f441815e1e0 0x7f4418132910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f440f200640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4418143ae0 0x7f4418134980 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4400000c00 tx=0x7f44000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f4414c00640 1 -- 172.21.15.141:0/2341095108 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400039c90 con 0x7f4418143ae0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f4414c00640 1 -- 172.21.15.141:0/2341095108 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440003f040 con 0x7f4418143ae0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4418131e30 con 0x7f4418143ae0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.041+0000 7f4414c00640 1 -- 172.21.15.141:0/2341095108 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400004030 con 0x7f4418143ae0 2024-01-16T14:57:50.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.042+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4418159ed0 con 0x7f4418143ae0 2024-01-16T14:57:50.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.043+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44180675d0 con 0x7f4418143ae0 2024-01-16T14:57:50.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.044+0000 7f4414c00640 1 -- 172.21.15.141:0/2341095108 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4400004190 con 0x7f4418143ae0 2024-01-16T14:57:50.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.045+0000 7f4414c00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43f006b2f0 0x7f43f006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.045+0000 7f4414c00640 1 -- 172.21.15.141:0/2341095108 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44000abcd0 con 0x7f4418143ae0 2024-01-16T14:57:50.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.046+0000 7f4417e00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43f006b2f0 0x7f43f006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.047+0000 7f4417e00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43f006b2f0 0x7f43f006d7b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4408000f60 tx=0x7f4408009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.050+0000 7f4414c00640 1 -- 172.21.15.141:0/2341095108 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4400076a10 con 0x7f4418143ae0 2024-01-16T14:57:50.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.238+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f441815d1b0 con 0x7f4418143ae0 2024-01-16T14:57:50.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.239+0000 7f4414c00640 1 -- 172.21.15.141:0/2341095108 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4400048020 con 0x7f4418143ae0 2024-01-16T14:57:50.222 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43f006b2f0 msgr2=0x7f43f006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43f006b2f0 0x7f43f006d7b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4408000f60 tx=0x7f4408009040 comp rx=0 tx=0).stop 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4418143ae0 msgr2=0x7f4418134980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4418143ae0 0x7f4418134980 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4400000c00 tx=0x7f44000046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 shutdown_connections 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43f006b2f0 0x7f43f006d7b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f441815e1e0 0x7f4418132910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4418143ae0 0x7f4418134980 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 --2- 172.21.15.141:0/2341095108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44180cf9a0 0x7f4418159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 >> 172.21.15.141:0/2341095108 conn(0x7f441806a800 msgr2=0x7f441806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 shutdown_connections 2024-01-16T14:57:50.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.242+0000 7f441ec00640 1 -- 172.21.15.141:0/2341095108 wait complete. 2024-01-16T14:57:50.242 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.470+0000 7f06c1c00640 1 Processor -- start 2024-01-16T14:57:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.470+0000 7f06c1c00640 1 -- start start 2024-01-16T14:57:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06c1c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f06bc144b70 0x7f06bc15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06c1c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f06bc0b1bf0 0x7f06bc15f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06c1c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06bc144f50 con 0x7f06bc15fa70 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06c1c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06bc15edc0 con 0x7f06bc144b70 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06c1c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06bc0b2360 con 0x7f06bc0b1bf0 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06baa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f06bc144b70 0x7f06bc15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06baa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f06bc144b70 0x7f06bc15b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56290/0 (socket says 172.21.15.141:56290) 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06baa00640 1 -- 172.21.15.141:0/3050036849 learned_addr learned my addr 172.21.15.141:0/3050036849 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.472+0000 7f06bb400640 1 --1- 172.21.15.141:0/3050036849 >> v1:172.21.15.141:6789/0 conn(0x7f06bc15fa70 0x7f06bc0cfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37328/0 (socket says 172.21.15.141:37328) 2024-01-16T14:57:50.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.473+0000 7f06ba000640 1 --2- 172.21.15.141:0/3050036849 >> v2:172.21.15.188:3300/0 conn(0x7f06bc0b1bf0 0x7f06bc15f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.473+0000 7f06baa00640 1 -- 172.21.15.141:0/3050036849 >> v2:172.21.15.188:3300/0 conn(0x7f06bc0b1bf0 msgr2=0x7f06bc15f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.473+0000 7f06baa00640 1 --2- 172.21.15.141:0/3050036849 >> v2:172.21.15.188:3300/0 conn(0x7f06bc0b1bf0 0x7f06bc15f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.473+0000 7f06baa00640 1 -- 172.21.15.141:0/3050036849 >> v1:172.21.15.141:6789/0 conn(0x7f06bc15fa70 legacy=0x7f06bc0cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.473+0000 7f06baa00640 1 -- 172.21.15.141:0/3050036849 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc0b2640 con 0x7f06bc144b70 2024-01-16T14:57:50.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.474+0000 7f06baa00640 1 --2- 172.21.15.141:0/3050036849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc15b7a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0afba0 tx=0x7f06a402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b3ec48c1ed180c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.474+0000 7f06b9600640 1 -- 172.21.15.141:0/3050036849 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4030860 con 0x7f06bc144b70 2024-01-16T14:57:50.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.474+0000 7f06b9600640 1 -- 172.21.15.141:0/3050036849 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a40309c0 con 0x7f06bc144b70 2024-01-16T14:57:50.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.474+0000 7f06b9600640 1 -- 172.21.15.141:0/3050036849 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4030d50 con 0x7f06bc144b70 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 -- 172.21.15.141:0/3050036849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 msgr2=0x7f06bc15b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 --2- 172.21.15.141:0/3050036849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc15b7a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0afba0 tx=0x7f06a402b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 -- 172.21.15.141:0/3050036849 shutdown_connections 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 --2- 172.21.15.141:0/3050036849 >> v2:172.21.15.188:3300/0 conn(0x7f06bc0b1bf0 0x7f06bc15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 --2- 172.21.15.141:0/3050036849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc15b7a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 -- 172.21.15.141:0/3050036849 >> 172.21.15.141:0/3050036849 conn(0x7f06bc06b4f0 msgr2=0x7f06bc06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 -- 172.21.15.141:0/3050036849 shutdown_connections 2024-01-16T14:57:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.475+0000 7f06c1c00640 1 -- 172.21.15.141:0/3050036849 wait complete. 2024-01-16T14:57:50.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.477+0000 7f06c1c00640 1 Processor -- start 2024-01-16T14:57:50.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.477+0000 7f06c1c00640 1 -- start start 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06c1c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06bc0b1bf0 0x7f06bc14eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06c1c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc14f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06c1c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc15fa70 0x7f06bc071220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06c1c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc13e2b0 con 0x7f06bc144b70 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06c1c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc14d0d0 con 0x7f06bc15fa70 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06c1c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc14f680 con 0x7f06bc0b1bf0 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06ba000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc14f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06ba000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc14f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56300/0 (socket says 172.21.15.141:56300) 2024-01-16T14:57:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.478+0000 7f06ba000640 1 -- 172.21.15.141:0/1685580589 learned_addr learned my addr 172.21.15.141:0/1685580589 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:50.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06ba000640 1 -- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06bc0b1bf0 msgr2=0x7f06bc14eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06ba000640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06bc0b1bf0 0x7f06bc14eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06ba000640 1 -- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc15fa70 msgr2=0x7f06bc071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06ba000640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc15fa70 0x7f06bc071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06ba000640 1 -- 172.21.15.141:0/1685580589 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a40094d0 con 0x7f06bc144b70 2024-01-16T14:57:50.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06ba000640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc14f020 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f06ac007ef0 tx=0x7f06ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06b2a00640 1 -- 172.21.15.141:0/1685580589 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac014620 con 0x7f06bc144b70 2024-01-16T14:57:50.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06b2a00640 1 -- 172.21.15.141:0/1685580589 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac014780 con 0x7f06bc144b70 2024-01-16T14:57:50.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06b2a00640 1 -- 172.21.15.141:0/1685580589 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac014b30 con 0x7f06bc144b70 2024-01-16T14:57:50.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.479+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06bc14f960 con 0x7f06bc144b70 2024-01-16T14:57:50.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.480+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc14cd60 con 0x7f06bc144b70 2024-01-16T14:57:50.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.481+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc0675d0 con 0x7f06bc144b70 2024-01-16T14:57:50.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.482+0000 7f06b2a00640 1 -- 172.21.15.141:0/1685580589 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f06ac014c90 con 0x7f06bc144b70 2024-01-16T14:57:50.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.482+0000 7f06b2a00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f069406b2c0 0x7f069406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.483+0000 7f06b2a00640 1 -- 172.21.15.141:0/1685580589 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06ac08a620 con 0x7f06bc144b70 2024-01-16T14:57:50.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.483+0000 7f06baa00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f069406b2c0 0x7f069406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.484+0000 7f06baa00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f069406b2c0 0x7f069406d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f06a4036000 tx=0x7f06a403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.487+0000 7f06b2a00640 1 -- 172.21.15.141:0/1685580589 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06ac055490 con 0x7f06bc144b70 2024-01-16T14:57:50.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.681+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06bc06b030 con 0x7f06bc144b70 2024-01-16T14:57:50.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.682+0000 7f06b2a00640 1 -- 172.21.15.141:0/1685580589 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f06ac058ab0 con 0x7f06bc144b70 2024-01-16T14:57:50.665 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f069406b2c0 msgr2=0x7f069406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f069406b2c0 0x7f069406d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f06a4036000 tx=0x7f06a403b040 comp rx=0 tx=0).stop 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 msgr2=0x7f06bc14f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc14f020 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f06ac007ef0 tx=0x7f06ac00e4b0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 shutdown_connections 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f069406b2c0 0x7f069406d780 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc15fa70 0x7f06bc071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06bc144b70 0x7f06bc14f020 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 --2- 172.21.15.141:0/1685580589 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06bc0b1bf0 0x7f06bc14eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 >> 172.21.15.141:0/1685580589 conn(0x7f06bc06b4f0 msgr2=0x7f06bc06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 shutdown_connections 2024-01-16T14:57:50.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.685+0000 7f06c1c00640 1 -- 172.21.15.141:0/1685580589 wait complete. 2024-01-16T14:57:50.688 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:50.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.912+0000 7f4fef600640 1 Processor -- start 2024-01-16T14:57:50.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.913+0000 7f4fef600640 1 -- start start 2024-01-16T14:57:50.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fef600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4fe815d590 0x7f4fe80cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fef600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4fe80d00b0 0x7f4fe80ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fef600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fef600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe80d0530 con 0x7f4fe80d00b0 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fef600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe80b6750 con 0x7f4fe815d590 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fef600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe80cf1f0 con 0x7f4fe80ce6f0 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fe7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4fe80d00b0 0x7f4fe80ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fe7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4fe80d00b0 0x7f4fe80ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56304/0 (socket says 172.21.15.141:56304) 2024-01-16T14:57:50.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.914+0000 7f4fe7e00640 1 -- 172.21.15.141:0/2210006614 learned_addr learned my addr 172.21.15.141:0/2210006614 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:50.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.915+0000 7f4fecc00640 1 --2- 172.21.15.141:0/2210006614 >> v2:172.21.15.188:3300/0 conn(0x7f4fe815d590 0x7f4fe80cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.915+0000 7f4fed600640 1 --2- 172.21.15.141:0/2210006614 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80d0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.915+0000 7f4fe7e00640 1 -- 172.21.15.141:0/2210006614 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 msgr2=0x7f4fe80d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.915+0000 7f4fe7e00640 1 --2- 172.21.15.141:0/2210006614 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80d0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.915+0000 7f4fe7e00640 1 -- 172.21.15.141:0/2210006614 >> v2:172.21.15.188:3300/0 conn(0x7f4fe815d590 msgr2=0x7f4fe80cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:50.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.915+0000 7f4fe7e00640 1 --2- 172.21.15.141:0/2210006614 >> v2:172.21.15.188:3300/0 conn(0x7f4fe815d590 0x7f4fe80cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.915+0000 7f4fe7e00640 1 -- 172.21.15.141:0/2210006614 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe80cef10 con 0x7f4fe80d00b0 2024-01-16T14:57:50.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.916+0000 7f4fe7e00640 1 --2- 172.21.15.141:0/2210006614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe80d00b0 0x7f4fe80ce1b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4fe806d4d0 tx=0x7f4fd002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483d3971b6190001 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fe7400640 1 -- 172.21.15.141:0/2210006614 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0031870 con 0x7f4fe80d00b0 2024-01-16T14:57:50.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fe7400640 1 -- 172.21.15.141:0/2210006614 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd00319d0 con 0x7f4fe80d00b0 2024-01-16T14:57:50.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fe7400640 1 -- 172.21.15.141:0/2210006614 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0031d60 con 0x7f4fe80d00b0 2024-01-16T14:57:50.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fef600640 1 -- 172.21.15.141:0/2210006614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe80d00b0 msgr2=0x7f4fe80ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fef600640 1 --2- 172.21.15.141:0/2210006614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe80d00b0 0x7f4fe80ce1b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4fe806d4d0 tx=0x7f4fd002b910 comp rx=0 tx=0).stop 2024-01-16T14:57:50.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fef600640 1 -- 172.21.15.141:0/2210006614 shutdown_connections 2024-01-16T14:57:50.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fef600640 1 --2- 172.21.15.141:0/2210006614 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80d0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fef600640 1 --2- 172.21.15.141:0/2210006614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe80d00b0 0x7f4fe80ce1b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fef600640 1 --2- 172.21.15.141:0/2210006614 >> v2:172.21.15.188:3300/0 conn(0x7f4fe815d590 0x7f4fe80cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.917+0000 7f4fef600640 1 -- 172.21.15.141:0/2210006614 >> 172.21.15.141:0/2210006614 conn(0x7f4fe809c540 msgr2=0x7f4fe80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:50.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.918+0000 7f4fef600640 1 -- 172.21.15.141:0/2210006614 shutdown_connections 2024-01-16T14:57:50.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.918+0000 7f4fef600640 1 -- 172.21.15.141:0/2210006614 wait complete. 2024-01-16T14:57:50.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.919+0000 7f4fef600640 1 Processor -- start 2024-01-16T14:57:50.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.919+0000 7f4fef600640 1 -- start start 2024-01-16T14:57:50.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.920+0000 7f4fef600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.920+0000 7f4fef600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe80d00b0 0x7f4fe809cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.920+0000 7f4fef600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe815d590 0x7f4fe80cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.920+0000 7f4fef600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe80c9380 con 0x7f4fe815d590 2024-01-16T14:57:50.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.920+0000 7f4fef600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe814c910 con 0x7f4fe80d00b0 2024-01-16T14:57:50.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.920+0000 7f4fef600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe80c95a0 con 0x7f4fe80ce6f0 2024-01-16T14:57:50.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe815d590 0x7f4fe80cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe815d590 0x7f4fe80cb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56316/0 (socket says 172.21.15.141:56316) 2024-01-16T14:57:50.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 -- 172.21.15.141:0/2342888553 learned_addr learned my addr 172.21.15.141:0/2342888553 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:50.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fecc00640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fe7e00640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe80d00b0 0x7f4fe809cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 -- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 msgr2=0x7f4fe80cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 -- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe80d00b0 msgr2=0x7f4fe809cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:50.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe80d00b0 0x7f4fe809cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:50.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.921+0000 7f4fed600640 1 -- 172.21.15.141:0/2342888553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd00094d0 con 0x7f4fe815d590 2024-01-16T14:57:50.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.922+0000 7f4fed600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe815d590 0x7f4fe80cb330 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000cc30 tx=0x7f4fe000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.922+0000 7f4fe4c00640 1 -- 172.21.15.141:0/2342888553 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe000ce40 con 0x7f4fe815d590 2024-01-16T14:57:50.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.922+0000 7f4fe4c00640 1 -- 172.21.15.141:0/2342888553 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe0003990 con 0x7f4fe815d590 2024-01-16T14:57:50.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.922+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe80c9880 con 0x7f4fe815d590 2024-01-16T14:57:50.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.922+0000 7f4fe4c00640 1 -- 172.21.15.141:0/2342888553 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0003d20 con 0x7f4fe815d590 2024-01-16T14:57:50.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.922+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe814c5a0 con 0x7f4fe815d590 2024-01-16T14:57:50.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.923+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe814ca80 con 0x7f4fe815d590 2024-01-16T14:57:50.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.924+0000 7f4fe4c00640 1 -- 172.21.15.141:0/2342888553 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4fe0003e80 con 0x7f4fe815d590 2024-01-16T14:57:50.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.925+0000 7f4fe4c00640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fc006b2c0 0x7f4fc006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:50.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.925+0000 7f4fe4c00640 1 -- 172.21.15.141:0/2342888553 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fe0089ff0 con 0x7f4fe815d590 2024-01-16T14:57:50.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.926+0000 7f4fecc00640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fc006b2c0 0x7f4fc006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.927+0000 7f4fecc00640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fc006b2c0 0x7f4fc006d780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800a8f0 tx=0x7f4fd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:50.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:50.929+0000 7f4fe4c00640 1 -- 172.21.15.141:0/2342888553 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fe0054e20 con 0x7f4fe815d590 2024-01-16T14:57:51.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.117+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4fe80cce20 con 0x7f4fe815d590 2024-01-16T14:57:51.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.118+0000 7f4fe4c00640 1 -- 172.21.15.141:0/2342888553 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4fe001d020 con 0x7f4fe815d590 2024-01-16T14:57:51.101 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fc006b2c0 msgr2=0x7f4fc006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fc006b2c0 0x7f4fc006d780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800a8f0 tx=0x7f4fd8008040 comp rx=0 tx=0).stop 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe815d590 msgr2=0x7f4fe80cb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe815d590 0x7f4fe80cb330 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000cc30 tx=0x7f4fe000a5c0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 shutdown_connections 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fc006b2c0 0x7f4fc006d780 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe815d590 0x7f4fe80cb330 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe80d00b0 0x7f4fe809cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 --2- 172.21.15.141:0/2342888553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe80ce6f0 0x7f4fe80cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 >> 172.21.15.141:0/2342888553 conn(0x7f4fe809c540 msgr2=0x7f4fe815c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.121+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 shutdown_connections 2024-01-16T14:57:51.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.122+0000 7f4fef600640 1 -- 172.21.15.141:0/2342888553 wait complete. 2024-01-16T14:57:51.123 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:51.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.346+0000 7f20ba400640 1 Processor -- start 2024-01-16T14:57:51.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.346+0000 7f20ba400640 1 -- start start 2024-01-16T14:57:51.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.348+0000 7f20ba400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f20b40ae5a0 0x7f20b415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.348+0000 7f20ba400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 0x7f20b406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.348+0000 7f20ba400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b4070c40 con 0x7f20b406d120 2024-01-16T14:57:51.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.348+0000 7f20ba400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b4071450 con 0x7f20b40ae5a0 2024-01-16T14:57:51.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.348+0000 7f20ba400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b406c470 con 0x7f20b40b3ae0 2024-01-16T14:57:51.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f20b406d120 0x7f20b40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37344/0 (socket says 172.21.15.141:37344) 2024-01-16T14:57:51.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b3e00640 1 -- 172.21.15.141:0/2432758427 learned_addr learned my addr 172.21.15.141:0/2432758427 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:51.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2a00640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 0x7f20b406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b3400640 1 --2- 172.21.15.141:0/2432758427 >> v2:172.21.15.188:3300/0 conn(0x7f20b40ae5a0 0x7f20b415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189086477 0 0) 0x7f20b4070c40 con 0x7f20b406d120 2024-01-16T14:57:51.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2094003520 con 0x7f20b406d120 2024-01-16T14:57:51.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1091336254 0 0) 0x7f2094003520 con 0x7f20b406d120 2024-01-16T14:57:51.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 msgr2=0x7f20b406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 0x7f20b406cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 >> v2:172.21.15.188:3300/0 conn(0x7f20b40ae5a0 msgr2=0x7f20b415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 --2- 172.21.15.141:0/2432758427 >> v2:172.21.15.188:3300/0 conn(0x7f20b40ae5a0 0x7f20b415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b406c750 con 0x7f20b406d120 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.349+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f20ac001670 con 0x7f20b406d120 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 >> v1:172.21.15.141:6789/0 conn(0x7f20b406d120 legacy=0x7f20b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2000640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 0x7f2094003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2000640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2094003ef0 0x7f20940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2000640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f2094005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b4071450 con 0x7f2094004d00 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b406c470 con 0x7f2094003ef0 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209400b980 con 0x7f20b40b3ae0 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b3e00640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f2094005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b2a00640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2094003ef0 0x7f20940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.350+0000 7f20b3400640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 0x7f2094003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.351+0000 7f20b3e00640 1 -- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 msgr2=0x7f2094003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.351+0000 7f20b3e00640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 0x7f2094003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.351+0000 7f20b3e00640 1 -- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2094003ef0 msgr2=0x7f20940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.351+0000 7f20b3e00640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2094003ef0 0x7f20940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.351+0000 7f20b3e00640 1 -- 172.21.15.141:0/2432758427 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2094003520 con 0x7f2094004d00 2024-01-16T14:57:51.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.351+0000 7f20b3400640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40b3ae0 0x7f2094003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:51.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.351+0000 7f20b2a00640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2094003ef0 0x7f20940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:57:51.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.352+0000 7f20b3e00640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f2094005180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00c240 tx=0x7f20940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6481bc8f7129adb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.352+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac005ad0 con 0x7f2094004d00 2024-01-16T14:57:51.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.352+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ac005c50 con 0x7f2094004d00 2024-01-16T14:57:51.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.352+0000 7f20b2000640 1 -- 172.21.15.141:0/2432758427 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac02c8c0 con 0x7f2094004d00 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.353+0000 7f20ba400640 1 -- 172.21.15.141:0/2432758427 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 msgr2=0x7f2094005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.353+0000 7f20ba400640 1 --2- 172.21.15.141:0/2432758427 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f2094005180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00c240 tx=0x7f20940057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.353+0000 7f20b3e00640 1 -- 172.21.15.141:0/2432758427 reap_dead start 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.353+0000 7f20ba400640 1 -- 172.21.15.141:0/2432758427 shutdown_connections 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.353+0000 7f20ba400640 1 -- 172.21.15.141:0/2432758427 >> 172.21.15.141:0/2432758427 conn(0x7f20b406a800 msgr2=0x7f20b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.353+0000 7f20ba400640 1 -- 172.21.15.141:0/2432758427 shutdown_connections 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.353+0000 7f20ba400640 1 -- 172.21.15.141:0/2432758427 wait complete. 2024-01-16T14:57:51.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.355+0000 7f20ba400640 1 Processor -- start 2024-01-16T14:57:51.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.355+0000 7f20ba400640 1 -- start start 2024-01-16T14:57:51.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.356+0000 7f20ba400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f20b40b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.356+0000 7f20ba400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40cd280 0x7f20b4157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.356+0000 7f20ba400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20b40cd880 0x7f20b40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.356+0000 7f20ba400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40ce240 con 0x7f2094004d00 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.356+0000 7f20ba400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b415c420 con 0x7f20b40cd880 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.356+0000 7f20ba400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b415c6e0 con 0x7f20b40cd280 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f20b40b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f20b40b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56324/0 (socket says 172.21.15.141:56324) 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 -- 172.21.15.141:0/3842280446 learned_addr learned my addr 172.21.15.141:0/3842280446 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 -- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40cd280 msgr2=0x7f20b4157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40cd280 0x7f20b4157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 -- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20b40cd880 msgr2=0x7f20b40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20b40cd880 0x7f20b40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 -- 172.21.15.141:0/3842280446 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b415c9c0 con 0x7f2094004d00 2024-01-16T14:57:51.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.357+0000 7f20b3400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f20b40b0470 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f20a40101c0 tx=0x7f20a4010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.358+0000 7f20ab400640 1 -- 172.21.15.141:0/3842280446 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a400e050 con 0x7f2094004d00 2024-01-16T14:57:51.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.358+0000 7f20ab400640 1 -- 172.21.15.141:0/3842280446 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a400f040 con 0x7f2094004d00 2024-01-16T14:57:51.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.358+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b415cca0 con 0x7f2094004d00 2024-01-16T14:57:51.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.358+0000 7f20ab400640 1 -- 172.21.15.141:0/3842280446 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4003bb0 con 0x7f2094004d00 2024-01-16T14:57:51.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.358+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b4157fb0 con 0x7f2094004d00 2024-01-16T14:57:51.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.359+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b40675d0 con 0x7f2094004d00 2024-01-16T14:57:51.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.360+0000 7f20ab400640 1 -- 172.21.15.141:0/3842280446 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f20a4003d10 con 0x7f2094004d00 2024-01-16T14:57:51.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.361+0000 7f20ab400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f208c06b2f0 0x7f208c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.361+0000 7f20ab400640 1 -- 172.21.15.141:0/3842280446 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20a408a970 con 0x7f2094004d00 2024-01-16T14:57:51.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.362+0000 7f20b2a00640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f208c06b2f0 0x7f208c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.363+0000 7f20b2a00640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f208c06b2f0 0x7f208c06d7b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f20b40ae280 tx=0x7f209c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.366+0000 7f20ab400640 1 -- 172.21.15.141:0/3842280446 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20a40557e0 con 0x7f2094004d00 2024-01-16T14:57:51.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.571+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f20b40cf970 con 0x7f2094004d00 2024-01-16T14:57:51.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.572+0000 7f20ab400640 1 -- 172.21.15.141:0/3842280446 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f20a4058e00 con 0x7f2094004d00 2024-01-16T14:57:51.555 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:57:51.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f208c06b2f0 msgr2=0x7f208c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f208c06b2f0 0x7f208c06d7b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f20b40ae280 tx=0x7f209c008040 comp rx=0 tx=0).stop 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 msgr2=0x7f20b40b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f20b40b0470 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f20a40101c0 tx=0x7f20a4010680 comp rx=0 tx=0).stop 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 shutdown_connections 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f208c06b2f0 0x7f208c06d7b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20b40cd880 0x7f20b40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20b40cd280 0x7f20b4157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 --2- 172.21.15.141:0/3842280446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2094004d00 0x7f20b40b0470 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 >> 172.21.15.141:0/3842280446 conn(0x7f20b406a800 msgr2=0x7f20b40cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 shutdown_connections 2024-01-16T14:57:51.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.575+0000 7f20ba400640 1 -- 172.21.15.141:0/3842280446 wait complete. 2024-01-16T14:57:51.576 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:57:51.576 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:51.624 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:51.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.902+0000 7f9d7f400640 1 Processor -- start 2024-01-16T14:57:51.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.902+0000 7f9d7f400640 1 -- start start 2024-01-16T14:57:51.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d7f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780cf9a0 0x7f9d78166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d7f400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7815c5f0 con 0x7f9d7813e360 2024-01-16T14:57:51.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d7f400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7806c470 con 0x7f9d7813df80 2024-01-16T14:57:51.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d7f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7806c750 con 0x7f9d780cf9a0 2024-01-16T14:57:51.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d6fc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9d7813e360 0x7f9d780cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37354/0 (socket says 172.21.15.141:37354) 2024-01-16T14:57:51.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d6fc00640 1 -- 172.21.15.141:0/2077402939 learned_addr learned my addr 172.21.15.141:0/2077402939 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:51.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d7d400640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780cf9a0 0x7f9d78166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791851638 0 0) 0x7f9d7815c5f0 con 0x7f9d7813e360 2024-01-16T14:57:51.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.904+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d58003520 con 0x7f9d7813e360 2024-01-16T14:57:51.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383492809 0 0) 0x7f9d58003520 con 0x7f9d7813e360 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780cf9a0 msgr2=0x7f9d78166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780cf9a0 0x7f9d78166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 >> v1:172.21.15.188:6789/0 conn(0x7f9d7813df80 legacy=0x7f9d7815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7806ca30 con 0x7f9d7813e360 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9d60001670 con 0x7f9d7813e360 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 >> v1:172.21.15.141:6789/0 conn(0x7f9d7813e360 legacy=0x7f9d780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780cf9a0 0x7f9d580039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d58004d00 0x7f9d58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7806c750 con 0x7f9d58003f70 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5800bac0 con 0x7f9d58004d00 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5800bc80 con 0x7f9d780cf9a0 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.905+0000 7f9d6fc00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.906+0000 7f9d6fc00640 1 -- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780cf9a0 msgr2=0x7f9d580039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.906+0000 7f9d6fc00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780cf9a0 0x7f9d580039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.906+0000 7f9d6fc00640 1 -- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d58004d00 msgr2=0x7f9d58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.906+0000 7f9d6fc00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d58004d00 0x7f9d58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.906+0000 7f9d6fc00640 1 -- 172.21.15.141:0/2077402939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d58003520 con 0x7f9d58003f70 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.907+0000 7f9d6fc00640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d580047c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9d780cc790 tx=0x7f9d580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42769b25e7dfd25 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.907+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60005ad0 con 0x7f9d58003f70 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.907+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d60005c30 con 0x7f9d58003f70 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.907+0000 7f9d77e00640 1 -- 172.21.15.141:0/2077402939 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d600072b0 con 0x7f9d58003f70 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.908+0000 7f9d7f400640 1 -- 172.21.15.141:0/2077402939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 msgr2=0x7f9d580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.908+0000 7f9d7f400640 1 --2- 172.21.15.141:0/2077402939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d580047c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9d780cc790 tx=0x7f9d580057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.908+0000 7f9d7d400640 1 -- 172.21.15.141:0/2077402939 reap_dead start 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.908+0000 7f9d7f400640 1 -- 172.21.15.141:0/2077402939 shutdown_connections 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.908+0000 7f9d7f400640 1 -- 172.21.15.141:0/2077402939 >> 172.21.15.141:0/2077402939 conn(0x7f9d7806a800 msgr2=0x7f9d7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.908+0000 7f9d7f400640 1 -- 172.21.15.141:0/2077402939 shutdown_connections 2024-01-16T14:57:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.908+0000 7f9d7f400640 1 -- 172.21.15.141:0/2077402939 wait complete. 2024-01-16T14:57:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.910+0000 7f9d7f400640 1 Processor -- start 2024-01-16T14:57:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.910+0000 7f9d7f400640 1 -- start start 2024-01-16T14:57:51.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.911+0000 7f9d7f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d780a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.911+0000 7f9d7f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780702c0 0x7f9d78070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.911+0000 7f9d7f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d7815a180 0x7f9d7815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.911+0000 7f9d7f400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780a92f0 con 0x7f9d58003f70 2024-01-16T14:57:51.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.911+0000 7f9d7f400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7815ab40 con 0x7f9d7815a180 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.911+0000 7f9d7f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78070d40 con 0x7f9d780702c0 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d780a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d780a8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56352/0 (socket says 172.21.15.141:56352) 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 -- 172.21.15.141:0/4171511644 learned_addr learned my addr 172.21.15.141:0/4171511644 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7d400640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d7815a180 0x7f9d7815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d6fc00640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780702c0 0x7f9d78070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 -- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780702c0 msgr2=0x7f9d78070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780702c0 0x7f9d78070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 -- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d7815a180 msgr2=0x7f9d7815a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d7815a180 0x7f9d7815a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.912+0000 7f9d7ca00640 1 -- 172.21.15.141:0/4171511644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78071020 con 0x7f9d58003f70 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.913+0000 7f9d7ca00640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d780a8db0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800c500 tx=0x7f9d6800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.913+0000 7f9d75600640 1 -- 172.21.15.141:0/4171511644 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680052d0 con 0x7f9d58003f70 2024-01-16T14:57:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.913+0000 7f9d75600640 1 -- 172.21.15.141:0/4171511644 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d68005430 con 0x7f9d58003f70 2024-01-16T14:57:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.913+0000 7f9d75600640 1 -- 172.21.15.141:0/4171511644 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68012870 con 0x7f9d58003f70 2024-01-16T14:57:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.913+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d78071300 con 0x7f9d58003f70 2024-01-16T14:57:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.913+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d780718c0 con 0x7f9d58003f70 2024-01-16T14:57:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.914+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d780675d0 con 0x7f9d58003f70 2024-01-16T14:57:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.915+0000 7f9d75600640 1 -- 172.21.15.141:0/4171511644 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9d680129d0 con 0x7f9d58003f70 2024-01-16T14:57:51.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.916+0000 7f9d75600640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d5006b2f0 0x7f9d5006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:51.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.916+0000 7f9d75600640 1 -- 172.21.15.141:0/4171511644 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d680896f0 con 0x7f9d58003f70 2024-01-16T14:57:51.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.916+0000 7f9d6fc00640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d5006b2f0 0x7f9d5006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:51.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.918+0000 7f9d6fc00640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d5006b2f0 0x7f9d5006d7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9d60034000 tx=0x7f9d60009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:51.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:51.920+0000 7f9d75600640 1 -- 172.21.15.141:0/4171511644 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d680545f0 con 0x7f9d58003f70 2024-01-16T14:57:52.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.114+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d7806df00 con 0x7f9d58003f70 2024-01-16T14:57:52.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.117+0000 7f9d75600640 1 -- 172.21.15.141:0/4171511644 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f9d68057c10 con 0x7f9d58003f70 2024-01-16T14:57:52.100 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:52.101 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":22,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:08.345341+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d5006b2f0 msgr2=0x7f9d5006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d5006b2f0 0x7f9d5006d7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9d60034000 tx=0x7f9d60009030 comp rx=0 tx=0).stop 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 msgr2=0x7f9d780a8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d780a8db0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800c500 tx=0x7f9d6800c9c0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 shutdown_connections 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d5006b2f0 0x7f9d5006d7b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9d7815a180 0x7f9d7815a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9d780702c0 0x7f9d78070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 --2- 172.21.15.141:0/4171511644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9d58003f70 0x7f9d780a8db0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 >> 172.21.15.141:0/4171511644 conn(0x7f9d7806a800 msgr2=0x7f9d78153950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 shutdown_connections 2024-01-16T14:57:52.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.120+0000 7f9d7f400640 1 -- 172.21.15.141:0/4171511644 wait complete. 2024-01-16T14:57:52.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:52.121 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:52.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.353+0000 7f8cbe400640 1 Processor -- start 2024-01-16T14:57:52.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.354+0000 7f8cbe400640 1 -- start start 2024-01-16T14:57:52.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.355+0000 7f8cbe400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8cb80d3910 0x7f8cb80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.355+0000 7f8cbe400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80d4160 con 0x7f8cb8160e10 2024-01-16T14:57:52.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.355+0000 7f8cbe400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80b2420 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.355+0000 7f8cbe400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80ceff0 con 0x7f8cb80d3910 2024-01-16T14:57:52.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.355+0000 7f8cae800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8cb80d3910 0x7f8cb80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.356+0000 7f8cae800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8cb80d3910 0x7f8cb80d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56364/0 (socket says 172.21.15.141:56364) 2024-01-16T14:57:52.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.356+0000 7f8cae800640 1 -- 172.21.15.141:0/2920604575 learned_addr learned my addr 172.21.15.141:0/2920604575 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:52.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.356+0000 7f8cae800640 1 -- 172.21.15.141:0/2920604575 >> v1:172.21.15.188:6789/0 conn(0x7f8cb80cfcf0 legacy=0x7f8cb8160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.356+0000 7f8cae800640 1 -- 172.21.15.141:0/2920604575 >> v1:172.21.15.141:6789/0 conn(0x7f8cb8160e10 legacy=0x7f8cb80d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.356+0000 7f8cae800640 1 -- 172.21.15.141:0/2920604575 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80b2140 con 0x7f8cb80d3910 2024-01-16T14:57:52.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.357+0000 7f8cae800640 1 --2- 172.21.15.141:0/2920604575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80d3910 0x7f8cb80d1100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009870 tx=0x7f8ca002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b328eeafbc050f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.359+0000 7f8cb6a00640 1 -- 172.21.15.141:0/2920604575 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0031870 con 0x7f8cb80d3910 2024-01-16T14:57:52.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.359+0000 7f8cb6a00640 1 -- 172.21.15.141:0/2920604575 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca00319d0 con 0x7f8cb80d3910 2024-01-16T14:57:52.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.359+0000 7f8cb6a00640 1 -- 172.21.15.141:0/2920604575 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0031d60 con 0x7f8cb80d3910 2024-01-16T14:57:52.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.360+0000 7f8cbe400640 1 -- 172.21.15.141:0/2920604575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80d3910 msgr2=0x7f8cb80d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.360+0000 7f8cbe400640 1 --2- 172.21.15.141:0/2920604575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80d3910 0x7f8cb80d1100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009870 tx=0x7f8ca002b910 comp rx=0 tx=0).stop 2024-01-16T14:57:52.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.360+0000 7f8cbe400640 1 -- 172.21.15.141:0/2920604575 shutdown_connections 2024-01-16T14:57:52.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.360+0000 7f8cbe400640 1 --2- 172.21.15.141:0/2920604575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80d3910 0x7f8cb80d1100 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.360+0000 7f8cbe400640 1 -- 172.21.15.141:0/2920604575 >> 172.21.15.141:0/2920604575 conn(0x7f8cb809c530 msgr2=0x7f8cb80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.360+0000 7f8cbe400640 1 -- 172.21.15.141:0/2920604575 shutdown_connections 2024-01-16T14:57:52.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.360+0000 7f8cbe400640 1 -- 172.21.15.141:0/2920604575 wait complete. 2024-01-16T14:57:52.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.362+0000 7f8cbe400640 1 Processor -- start 2024-01-16T14:57:52.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.362+0000 7f8cbe400640 1 -- start start 2024-01-16T14:57:52.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cbe400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80cfcf0 0x7f8cb815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cbe400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb80d3910 0x7f8cb8149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cbe400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8cb8160e10 0x7f8cb80b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cbe400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb814a450 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cbe400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb815c6d0 con 0x7f8cb80d3910 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cbe400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb815c9b0 con 0x7f8cb8160e10 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80cfcf0 0x7f8cb815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80cfcf0 0x7f8cb815bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56380/0 (socket says 172.21.15.141:56380) 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7400640 1 -- 172.21.15.141:0/828826561 learned_addr learned my addr 172.21.15.141:0/828826561 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cae800640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb80d3910 0x7f8cb8149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7e00640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8cb8160e10 0x7f8cb80b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7400640 1 -- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8cb8160e10 msgr2=0x7f8cb80b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8cb8160e10 0x7f8cb80b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7400640 1 -- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb80d3910 msgr2=0x7f8cb8149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.363+0000 7f8cb7400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb80d3910 0x7f8cb8149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cb7400640 1 -- 172.21.15.141:0/828826561 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca00094d0 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cb7e00640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8cb8160e10 0x7f8cb80b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cae800640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb80d3910 0x7f8cb8149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cb7400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80cfcf0 0x7f8cb815bfd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800c430 tx=0x7f8ca800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cafe00640 1 -- 172.21.15.141:0/828826561 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8011790 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cafe00640 1 -- 172.21.15.141:0/828826561 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca80118f0 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb815cc90 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cafe00640 1 -- 172.21.15.141:0/828826561 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8011ce0 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.364+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb815d250 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.366+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb80675d0 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.366+0000 7f8cafe00640 1 -- 172.21.15.141:0/828826561 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8ca801c460 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.367+0000 7f8cafe00640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c9006b2c0 0x7f8c9006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.367+0000 7f8cafe00640 1 -- 172.21.15.141:0/828826561 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ca8089c20 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.368+0000 7f8cae800640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c9006b2c0 0x7f8c9006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.369+0000 7f8cae800640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c9006b2c0 0x7f8c9006d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0036000 tx=0x7f8ca003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.371+0000 7f8cafe00640 1 -- 172.21.15.141:0/828826561 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ca8054a00 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.564+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8cb80b5690 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.564+0000 7f8cafe00640 1 -- 172.21.15.141:0/828826561 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ca8058020 con 0x7f8cb80cfcf0 2024-01-16T14:57:52.547 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c9006b2c0 msgr2=0x7f8c9006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c9006b2c0 0x7f8c9006d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0036000 tx=0x7f8ca003c040 comp rx=0 tx=0).stop 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80cfcf0 msgr2=0x7f8cb815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80cfcf0 0x7f8cb815bfd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800c430 tx=0x7f8ca800c8f0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 shutdown_connections 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c9006b2c0 0x7f8c9006d780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8cb8160e10 0x7f8cb80b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb80d3910 0x7f8cb8149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 --2- 172.21.15.141:0/828826561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cb80cfcf0 0x7f8cb815bfd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.567+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 >> 172.21.15.141:0/828826561 conn(0x7f8cb809c530 msgr2=0x7f8cb815f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.568+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 shutdown_connections 2024-01-16T14:57:52.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.568+0000 7f8cbe400640 1 -- 172.21.15.141:0/828826561 wait complete. 2024-01-16T14:57:52.570 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:52.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.793+0000 7fc055600640 1 Processor -- start 2024-01-16T14:57:52.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.793+0000 7fc055600640 1 -- start start 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc055600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc0500cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc055600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc0500cf7d0 0x7fc05015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc055600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0500cfc30 con 0x7fc05015e1e0 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc055600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0500b1ee0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc055600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500b2780 con 0x7fc050143ae0 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc047e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc0500cf7d0 0x7fc05015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc047e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc0500cf7d0 0x7fc05015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56396/0 (socket says 172.21.15.141:56396) 2024-01-16T14:57:52.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.795+0000 7fc047e00640 1 -- 172.21.15.141:0/3241361747 learned_addr learned my addr 172.21.15.141:0/3241361747 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:52.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.796+0000 7fc04ea00640 1 --2- 172.21.15.141:0/3241361747 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc0500cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.796+0000 7fc047e00640 1 -- 172.21.15.141:0/3241361747 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 msgr2=0x7fc0500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.796+0000 7fc047e00640 1 --2- 172.21.15.141:0/3241361747 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc0500cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.796+0000 7fc047e00640 1 -- 172.21.15.141:0/3241361747 >> v1:172.21.15.188:6789/0 conn(0x7fc05015e1e0 legacy=0x7fc05015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:52.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.796+0000 7fc047e00640 1 -- 172.21.15.141:0/3241361747 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05015e990 con 0x7fc0500cf7d0 2024-01-16T14:57:52.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.796+0000 7fc04ea00640 1 --2- 172.21.15.141:0/3241361747 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc0500cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:52.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.797+0000 7fc047e00640 1 --2- 172.21.15.141:0/3241361747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc05015dca0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc03800bb10 tx=0x7fc03802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04c688c8f99e6bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.797+0000 7fc04e000640 1 -- 172.21.15.141:0/3241361747 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03802ee70 con 0x7fc0500cf7d0 2024-01-16T14:57:52.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.797+0000 7fc04e000640 1 -- 172.21.15.141:0/3241361747 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc038036440 con 0x7fc0500cf7d0 2024-01-16T14:57:52.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.797+0000 7fc04e000640 1 -- 172.21.15.141:0/3241361747 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0380367d0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 -- 172.21.15.141:0/3241361747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 msgr2=0x7fc05015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 --2- 172.21.15.141:0/3241361747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc05015dca0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc03800bb10 tx=0x7fc03802e0c0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 -- 172.21.15.141:0/3241361747 shutdown_connections 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 --2- 172.21.15.141:0/3241361747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc05015dca0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 --2- 172.21.15.141:0/3241361747 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc0500cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 -- 172.21.15.141:0/3241361747 >> 172.21.15.141:0/3241361747 conn(0x7fc05006a800 msgr2=0x7fc05006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 -- 172.21.15.141:0/3241361747 shutdown_connections 2024-01-16T14:57:52.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.798+0000 7fc055600640 1 -- 172.21.15.141:0/3241361747 wait complete. 2024-01-16T14:57:52.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.801+0000 7fc055600640 1 Processor -- start 2024-01-16T14:57:52.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.801+0000 7fc055600640 1 -- start start 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc055600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc0500ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc055600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc050133710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc055600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc05015e1e0 0x7fc0500ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc055600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc050131780 con 0x7fc0500cf7d0 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc055600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc050134030 con 0x7fc05015e1e0 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc055600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc050134310 con 0x7fc050143ae0 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc04ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc0500ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc04ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc0500ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56402/0 (socket says 172.21.15.141:56402) 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.802+0000 7fc04ea00640 1 -- 172.21.15.141:0/1463252696 learned_addr learned my addr 172.21.15.141:0/1463252696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc047e00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc050133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc04ea00640 1 -- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 msgr2=0x7fc050133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc04ea00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc050133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc04ea00640 1 -- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc05015e1e0 msgr2=0x7fc0500ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc04ea00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc05015e1e0 0x7fc0500ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc04ea00640 1 -- 172.21.15.141:0/1463252696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03800b770 con 0x7fc0500cf7d0 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc047e00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc050133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:52.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc04ea00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc0500ca810 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc04000c7f0 tx=0x7fc04000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc046a00640 1 -- 172.21.15.141:0/1463252696 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040007a30 con 0x7fc0500cf7d0 2024-01-16T14:57:52.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc046a00640 1 -- 172.21.15.141:0/1463252696 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc040007b90 con 0x7fc0500cf7d0 2024-01-16T14:57:52.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc046a00640 1 -- 172.21.15.141:0/1463252696 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04000ce40 con 0x7fc0500cf7d0 2024-01-16T14:57:52.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0501319f0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.803+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0501345f0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.805+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc050131c30 con 0x7fc0500cf7d0 2024-01-16T14:57:52.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.805+0000 7fc046a00640 1 -- 172.21.15.141:0/1463252696 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc0400113f0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.807+0000 7fc046a00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc02806b2c0 0x7fc02806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:52.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.807+0000 7fc046a00640 1 -- 172.21.15.141:0/1463252696 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc040089ea0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.808+0000 7fc047e00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc02806b2c0 0x7fc02806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:52.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.809+0000 7fc047e00640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc02806b2c0 0x7fc02806d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc038038000 tx=0x7fc038008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:52.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:52.811+0000 7fc046a00640 1 -- 172.21.15.141:0/1463252696 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc040054be0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.001+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0500675d0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.002+0000 7fc046a00640 1 -- 172.21.15.141:0/1463252696 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc04001f0a0 con 0x7fc0500cf7d0 2024-01-16T14:57:52.985 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc02806b2c0 msgr2=0x7fc02806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc02806b2c0 0x7fc02806d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc038038000 tx=0x7fc038008040 comp rx=0 tx=0).stop 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 msgr2=0x7fc0500ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc0500ca810 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc04000c7f0 tx=0x7fc04000cbf0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 shutdown_connections 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc02806b2c0 0x7fc02806d780 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc05015e1e0 0x7fc0500ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc050143ae0 0x7fc050133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 --2- 172.21.15.141:0/1463252696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0500cf7d0 0x7fc0500ca810 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:52.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 >> 172.21.15.141:0/1463252696 conn(0x7fc05006a800 msgr2=0x7fc05015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:52.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.005+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 shutdown_connections 2024-01-16T14:57:52.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.006+0000 7fc055600640 1 -- 172.21.15.141:0/1463252696 wait complete. 2024-01-16T14:57:53.007 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:53.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.238+0000 7f8039c00640 1 Processor -- start 2024-01-16T14:57:53.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.238+0000 7f8039c00640 1 -- start start 2024-01-16T14:57:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.239+0000 7f8039c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f803413df80 0x7f803415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.239+0000 7f8039c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f803413e360 0x7f80340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8039c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f80340cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8039c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803406c470 con 0x7f803413df80 2024-01-16T14:57:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8039c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803406c750 con 0x7f803413e360 2024-01-16T14:57:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8039c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803406ca30 con 0x7f803415f9c0 2024-01-16T14:57:53.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8032a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f803413df80 0x7f803415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8032a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f803413df80 0x7f803415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42488/0 (socket says 172.21.15.141:42488) 2024-01-16T14:57:53.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8032a00640 1 -- 172.21.15.141:0/2358794715 learned_addr learned my addr 172.21.15.141:0/2358794715 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:53.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f8033400640 1 --2- 172.21.15.141:0/2358794715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f80340cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.240+0000 7f802be00640 1 --2- 172.21.15.141:0/2358794715 >> v2:172.21.15.188:3300/0 conn(0x7f803413e360 0x7f80340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.241+0000 7f8032a00640 1 -- 172.21.15.141:0/2358794715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 msgr2=0x7f80340cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.241+0000 7f8032a00640 1 --2- 172.21.15.141:0/2358794715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f80340cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.241+0000 7f8032a00640 1 -- 172.21.15.141:0/2358794715 >> v2:172.21.15.188:3300/0 conn(0x7f803413e360 msgr2=0x7f80340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.241+0000 7f8032a00640 1 --2- 172.21.15.141:0/2358794715 >> v2:172.21.15.188:3300/0 conn(0x7f803413e360 0x7f80340cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.241+0000 7f8032a00640 1 -- 172.21.15.141:0/2358794715 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803406cd10 con 0x7f803413df80 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.242+0000 7f8032a00640 1 --2- 172.21.15.141:0/2358794715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413df80 0x7f803415bee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f803406ded0 tx=0x7f802402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff68f14a07d2b729 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.242+0000 7f8032000640 1 -- 172.21.15.141:0/2358794715 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024031870 con 0x7f803413df80 2024-01-16T14:57:53.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.242+0000 7f8032000640 1 -- 172.21.15.141:0/2358794715 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80240319d0 con 0x7f803413df80 2024-01-16T14:57:53.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.242+0000 7f8032000640 1 -- 172.21.15.141:0/2358794715 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024031d60 con 0x7f803413df80 2024-01-16T14:57:53.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 -- 172.21.15.141:0/2358794715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413df80 msgr2=0x7f803415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 --2- 172.21.15.141:0/2358794715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413df80 0x7f803415bee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f803406ded0 tx=0x7f802402b910 comp rx=0 tx=0).stop 2024-01-16T14:57:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 -- 172.21.15.141:0/2358794715 shutdown_connections 2024-01-16T14:57:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 --2- 172.21.15.141:0/2358794715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f80340cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 --2- 172.21.15.141:0/2358794715 >> v2:172.21.15.188:3300/0 conn(0x7f803413e360 0x7f80340cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 --2- 172.21.15.141:0/2358794715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413df80 0x7f803415bee0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 -- 172.21.15.141:0/2358794715 >> 172.21.15.141:0/2358794715 conn(0x7f803406a800 msgr2=0x7f803406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 -- 172.21.15.141:0/2358794715 shutdown_connections 2024-01-16T14:57:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.243+0000 7f8039c00640 1 -- 172.21.15.141:0/2358794715 wait complete. 2024-01-16T14:57:53.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.245+0000 7f8039c00640 1 Processor -- start 2024-01-16T14:57:53.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.245+0000 7f8039c00640 1 -- start start 2024-01-16T14:57:53.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8039c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f803413df80 0x7f80341f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8039c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413e360 0x7f803413f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8039c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f803413d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8039c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80340ae360 con 0x7f803413e360 2024-01-16T14:57:53.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8039c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80340a84c0 con 0x7f803413df80 2024-01-16T14:57:53.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8039c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80340a8760 con 0x7f803415f9c0 2024-01-16T14:57:53.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f802be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413e360 0x7f803413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f802be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413e360 0x7f803413f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42504/0 (socket says 172.21.15.141:42504) 2024-01-16T14:57:53.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f802be00640 1 -- 172.21.15.141:0/2331919518 learned_addr learned my addr 172.21.15.141:0/2331919518 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8032a00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f803413df80 0x7f80341f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.246+0000 7f8033400640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f803413d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802be00640 1 -- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 msgr2=0x7f803413d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802be00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f803413d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802be00640 1 -- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f803413df80 msgr2=0x7f80341f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802be00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f803413df80 0x7f80341f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802be00640 1 -- 172.21.15.141:0/2331919518 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80240094d0 con 0x7f803413e360 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f8032a00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f803413df80 0x7f80341f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:53.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802be00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413e360 0x7f803413f100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f801c00d7d0 tx=0x7f801c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802aa00640 1 -- 172.21.15.141:0/2331919518 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c018070 con 0x7f803413e360 2024-01-16T14:57:53.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802aa00640 1 -- 172.21.15.141:0/2331919518 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c004020 con 0x7f803413e360 2024-01-16T14:57:53.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f802aa00640 1 -- 172.21.15.141:0/2331919518 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c0041b0 con 0x7f803413e360 2024-01-16T14:57:53.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80340a8a40 con 0x7f803413e360 2024-01-16T14:57:53.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.247+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80340a9000 con 0x7f803413e360 2024-01-16T14:57:53.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.248+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8034140890 con 0x7f803413e360 2024-01-16T14:57:53.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.249+0000 7f802aa00640 1 -- 172.21.15.141:0/2331919518 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f801c004310 con 0x7f803413e360 2024-01-16T14:57:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.251+0000 7f802aa00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06b2c0 0x7f800c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.251+0000 7f802aa00640 1 -- 172.21.15.141:0/2331919518 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f801c089af0 con 0x7f803413e360 2024-01-16T14:57:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.251+0000 7f8032a00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06b2c0 0x7f800c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.252+0000 7f8032a00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06b2c0 0x7f800c06d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8024036000 tx=0x7f802403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.255+0000 7f802aa00640 1 -- 172.21.15.141:0/2331919518 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f801c054950 con 0x7f803413e360 2024-01-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.454+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80340a9190 con 0x7f803413e360 2024-01-16T14:57:53.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.454+0000 7f802aa00640 1 -- 172.21.15.141:0/2331919518 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f801c057f70 con 0x7f803413e360 2024-01-16T14:57:53.437 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06b2c0 msgr2=0x7f800c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06b2c0 0x7f800c06d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8024036000 tx=0x7f802403c040 comp rx=0 tx=0).stop 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413e360 msgr2=0x7f803413f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413e360 0x7f803413f100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f801c00d7d0 tx=0x7f801c00dc90 comp rx=0 tx=0).stop 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 shutdown_connections 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06b2c0 0x7f800c06d780 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f803415f9c0 0x7f803413d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803413e360 0x7f803413f100 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 --2- 172.21.15.141:0/2331919518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f803413df80 0x7f80341f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 >> 172.21.15.141:0/2331919518 conn(0x7f803406a800 msgr2=0x7f80340afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.457+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 shutdown_connections 2024-01-16T14:57:53.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.458+0000 7f8039c00640 1 -- 172.21.15.141:0/2331919518 wait complete. 2024-01-16T14:57:53.458 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:53.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.687+0000 7f9102a00640 1 Processor -- start 2024-01-16T14:57:53.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.687+0000 7f9102a00640 1 -- start start 2024-01-16T14:57:53.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.688+0000 7f9102a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f90fc13df80 0x7f90fc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f9102a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f90fc13e360 0x7f90fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f9102a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc06c470 con 0x7f90fc15f9c0 2024-01-16T14:57:53.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f9102a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc06c750 con 0x7f90fc13e360 2024-01-16T14:57:53.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f9102a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc06ca30 con 0x7f90fc13df80 2024-01-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f90fc13e360 0x7f90fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f90fc13e360 0x7f90fc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42512/0 (socket says 172.21.15.141:42512) 2024-01-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fb400640 1 -- 172.21.15.141:0/1389654644 learned_addr learned my addr 172.21.15.141:0/1389654644 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fbe00640 1 --2- 172.21.15.141:0/1389654644 >> v2:172.21.15.188:3300/0 conn(0x7f90fc13df80 0x7f90fc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fb400640 1 -- 172.21.15.141:0/1389654644 >> v2:172.21.15.188:3300/0 conn(0x7f90fc13df80 msgr2=0x7f90fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fb400640 1 --2- 172.21.15.141:0/1389654644 >> v2:172.21.15.188:3300/0 conn(0x7f90fc13df80 0x7f90fc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fb400640 1 -- 172.21.15.141:0/1389654644 >> v1:172.21.15.188:6789/0 conn(0x7f90fc15f9c0 legacy=0x7f90fc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fb400640 1 -- 172.21.15.141:0/1389654644 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc06cd10 con 0x7f90fc13e360 2024-01-16T14:57:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.689+0000 7f90fbe00640 1 --2- 172.21.15.141:0/1389654644 >> v2:172.21.15.188:3300/0 conn(0x7f90fc13df80 0x7f90fc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.691+0000 7f90fb400640 1 --2- 172.21.15.141:0/1389654644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc0cf290 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f90fc072770 tx=0x7f90e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459c8743b877f203 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.691+0000 7f90faa00640 1 -- 172.21.15.141:0/1389654644 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4031870 con 0x7f90fc13e360 2024-01-16T14:57:53.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.691+0000 7f90faa00640 1 -- 172.21.15.141:0/1389654644 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e40319d0 con 0x7f90fc13e360 2024-01-16T14:57:53.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.691+0000 7f90faa00640 1 -- 172.21.15.141:0/1389654644 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4031d60 con 0x7f90fc13e360 2024-01-16T14:57:53.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 -- 172.21.15.141:0/1389654644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 msgr2=0x7f90fc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 --2- 172.21.15.141:0/1389654644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc0cf290 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f90fc072770 tx=0x7f90e402b910 comp rx=0 tx=0).stop 2024-01-16T14:57:53.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 -- 172.21.15.141:0/1389654644 shutdown_connections 2024-01-16T14:57:53.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 --2- 172.21.15.141:0/1389654644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc0cf290 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 --2- 172.21.15.141:0/1389654644 >> v2:172.21.15.188:3300/0 conn(0x7f90fc13df80 0x7f90fc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 -- 172.21.15.141:0/1389654644 >> 172.21.15.141:0/1389654644 conn(0x7f90fc06a800 msgr2=0x7f90fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 -- 172.21.15.141:0/1389654644 shutdown_connections 2024-01-16T14:57:53.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.692+0000 7f9102a00640 1 -- 172.21.15.141:0/1389654644 wait complete. 2024-01-16T14:57:53.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.693+0000 7f9102a00640 1 Processor -- start 2024-01-16T14:57:53.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.694+0000 7f9102a00640 1 -- start start 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f9102a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90fc13df80 0x7f90fc1f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f9102a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc1f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f9102a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90fc15f9c0 0x7f90fc13d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f9102a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc071170 con 0x7f90fc13e360 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f9102a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc1f33a0 con 0x7f90fc15f9c0 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f9102a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc1f3680 con 0x7f90fc13df80 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc1f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc1f3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42514/0 (socket says 172.21.15.141:42514) 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 -- 172.21.15.141:0/3823448371 learned_addr learned my addr 172.21.15.141:0/3823448371 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 -- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90fc13df80 msgr2=0x7f90fc1f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90fc13df80 0x7f90fc1f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 -- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90fc15f9c0 msgr2=0x7f90fc13d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90fc15f9c0 0x7f90fc13d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.695+0000 7f90fb400640 1 -- 172.21.15.141:0/3823448371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40094d0 con 0x7f90fc13e360 2024-01-16T14:57:53.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.696+0000 7f90fb400640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc1f3fa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f90e4000c00 tx=0x7f90e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.696+0000 7f90f3e00640 1 -- 172.21.15.141:0/3823448371 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4031840 con 0x7f90fc13e360 2024-01-16T14:57:53.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.696+0000 7f90f3e00640 1 -- 172.21.15.141:0/3823448371 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e40443f0 con 0x7f90fc13e360 2024-01-16T14:57:53.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.696+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc1f3960 con 0x7f90fc13e360 2024-01-16T14:57:53.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.696+0000 7f90f3e00640 1 -- 172.21.15.141:0/3823448371 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e40447a0 con 0x7f90fc13e360 2024-01-16T14:57:53.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.696+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc14c760 con 0x7f90fc13e360 2024-01-16T14:57:53.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.697+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc0712b0 con 0x7f90fc13e360 2024-01-16T14:57:53.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.698+0000 7f90f3e00640 1 -- 172.21.15.141:0/3823448371 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f90e404d450 con 0x7f90fc13e360 2024-01-16T14:57:53.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.699+0000 7f90f3e00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90d406b2c0 0x7f90d406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:53.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.699+0000 7f90f3e00640 1 -- 172.21.15.141:0/3823448371 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f90e402d040 con 0x7f90fc13e360 2024-01-16T14:57:53.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.699+0000 7f90fbe00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90d406b2c0 0x7f90d406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:53.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.701+0000 7f90fbe00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90d406b2c0 0x7f90d406d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f90ec006fd0 tx=0x7f90ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:53.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.703+0000 7f90f3e00640 1 -- 172.21.15.141:0/3823448371 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90fc0712b0 con 0x7f90fc13e360 2024-01-16T14:57:53.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.908+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90fc0712b0 con 0x7f90fc13e360 2024-01-16T14:57:53.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.908+0000 7f90f3e00640 1 -- 172.21.15.141:0/3823448371 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f90fc0712b0 con 0x7f90fc13e360 2024-01-16T14:57:53.891 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:57:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90d406b2c0 msgr2=0x7f90d406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90d406b2c0 0x7f90d406d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f90ec006fd0 tx=0x7f90ec008040 comp rx=0 tx=0).stop 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 msgr2=0x7f90fc1f3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc1f3fa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f90e4000c00 tx=0x7f90e40046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 shutdown_connections 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90d406b2c0 0x7f90d406d780 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90fc15f9c0 0x7f90fc13d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90fc13e360 0x7f90fc1f3fa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 --2- 172.21.15.141:0/3823448371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90fc13df80 0x7f90fc1f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 >> 172.21.15.141:0/3823448371 conn(0x7f90fc06a800 msgr2=0x7f90fc15e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 shutdown_connections 2024-01-16T14:57:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:53.911+0000 7f9102a00640 1 -- 172.21.15.141:0/3823448371 wait complete. 2024-01-16T14:57:53.912 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T14:57:53.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:53.913 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T14:57:54.004 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:54.004 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:57:54.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:54.005 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T14:57:54.073 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:57:54.074 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:54.130 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:57:54.130 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:54.178 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.416+0000 7fc650a00640 1 Processor -- start 2024-01-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.416+0000 7fc650a00640 1 -- start start 2024-01-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.417+0000 7fc650a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc6480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.417+0000 7fc650a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc6480cf7d0 0x7fc64815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.417+0000 7fc650a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc64815e1e0 0x7fc64815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.417+0000 7fc650a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6480cfc30 con 0x7fc64815e1e0 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.417+0000 7fc650a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6480b1ee0 con 0x7fc6480cf7d0 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.417+0000 7fc650a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6480b2780 con 0x7fc648143ae0 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc64815e1e0 0x7fc64815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc64815e1e0 0x7fc64815a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42526/0 (socket says 172.21.15.141:42526) 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 -- 172.21.15.141:0/1164369277 learned_addr learned my addr 172.21.15.141:0/1164369277 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 -- 172.21.15.141:0/1164369277 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 msgr2=0x7fc6480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 --2- 172.21.15.141:0/1164369277 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc6480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 -- 172.21.15.141:0/1164369277 >> v2:172.21.15.188:3300/0 conn(0x7fc6480cf7d0 msgr2=0x7fc64815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 --2- 172.21.15.141:0/1164369277 >> v2:172.21.15.188:3300/0 conn(0x7fc6480cf7d0 0x7fc64815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.418+0000 7fc64ea00640 1 -- 172.21.15.141:0/1164369277 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64815e990 con 0x7fc64815e1e0 2024-01-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.419+0000 7fc64ea00640 1 --2- 172.21.15.141:0/1164369277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64815a670 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc64806e350 tx=0x7fc64002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da156b22000c725 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc64cc00640 1 -- 172.21.15.141:0/1164369277 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64002ca80 con 0x7fc64815e1e0 2024-01-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc64cc00640 1 -- 172.21.15.141:0/1164369277 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64002cbe0 con 0x7fc64815e1e0 2024-01-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc64cc00640 1 -- 172.21.15.141:0/1164369277 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400343f0 con 0x7fc64815e1e0 2024-01-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc650a00640 1 -- 172.21.15.141:0/1164369277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 msgr2=0x7fc64815a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc650a00640 1 --2- 172.21.15.141:0/1164369277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64815a670 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc64806e350 tx=0x7fc64002bee0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc650a00640 1 -- 172.21.15.141:0/1164369277 shutdown_connections 2024-01-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc650a00640 1 --2- 172.21.15.141:0/1164369277 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64815a670 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc650a00640 1 --2- 172.21.15.141:0/1164369277 >> v2:172.21.15.188:3300/0 conn(0x7fc6480cf7d0 0x7fc64815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc650a00640 1 --2- 172.21.15.141:0/1164369277 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc6480cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.420+0000 7fc650a00640 1 -- 172.21.15.141:0/1164369277 >> 172.21.15.141:0/1164369277 conn(0x7fc64806a800 msgr2=0x7fc64806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.421+0000 7fc650a00640 1 -- 172.21.15.141:0/1164369277 shutdown_connections 2024-01-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.421+0000 7fc650a00640 1 -- 172.21.15.141:0/1164369277 wait complete. 2024-01-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.422+0000 7fc650a00640 1 Processor -- start 2024-01-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.422+0000 7fc650a00640 1 -- start start 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.423+0000 7fc650a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6480cf7d0 0x7fc648141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.423+0000 7fc650a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc648141cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.423+0000 7fc650a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64813be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.423+0000 7fc650a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6481f3b70 con 0x7fc64815e1e0 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.423+0000 7fc650a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64813ca20 con 0x7fc6480cf7d0 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.423+0000 7fc650a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6481f31d0 con 0x7fc648143ae0 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64813be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64813be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42530/0 (socket says 172.21.15.141:42530) 2024-01-16T14:57:54.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 -- 172.21.15.141:0/408821294 learned_addr learned my addr 172.21.15.141:0/408821294 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:54.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64e000640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6480cf7d0 0x7fc648141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64d600640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc648141cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 -- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 msgr2=0x7fc648141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc648141cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 -- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6480cf7d0 msgr2=0x7fc648141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6480cf7d0 0x7fc648141770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 -- 172.21.15.141:0/408821294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc640009b30 con 0x7fc64815e1e0 2024-01-16T14:57:54.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64e000640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6480cf7d0 0x7fc648141770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:54.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64d600640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc648141cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:54.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.424+0000 7fc64ea00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64813be20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc640036040 tx=0x7fc640004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.425+0000 7fc646000640 1 -- 172.21.15.141:0/408821294 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640040670 con 0x7fc64815e1e0 2024-01-16T14:57:54.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.425+0000 7fc646000640 1 -- 172.21.15.141:0/408821294 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6400407f0 con 0x7fc64815e1e0 2024-01-16T14:57:54.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.425+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6481f34b0 con 0x7fc64815e1e0 2024-01-16T14:57:54.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.425+0000 7fc646000640 1 -- 172.21.15.141:0/408821294 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640040ba0 con 0x7fc64815e1e0 2024-01-16T14:57:54.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.425+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64813fc10 con 0x7fc64815e1e0 2024-01-16T14:57:54.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.426+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6481f3cb0 con 0x7fc64815e1e0 2024-01-16T14:57:54.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.427+0000 7fc646000640 1 -- 172.21.15.141:0/408821294 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc64002cd50 con 0x7fc64815e1e0 2024-01-16T14:57:54.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.428+0000 7fc646000640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc62006b2f0 0x7fc62006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.428+0000 7fc646000640 1 -- 172.21.15.141:0/408821294 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fc6400ac820 con 0x7fc64815e1e0 2024-01-16T14:57:54.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.428+0000 7fc64e000640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc62006b2f0 0x7fc62006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.429+0000 7fc64e000640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc62006b2f0 0x7fc62006d7b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc638006fd0 tx=0x7fc638008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.432+0000 7fc646000640 1 -- 172.21.15.141:0/408821294 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6481f3cb0 con 0x7fc64815e1e0 2024-01-16T14:57:54.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.625+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6481f3cb0 con 0x7fc64815e1e0 2024-01-16T14:57:54.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.628+0000 7fc646000640 1 -- 172.21.15.141:0/408821294 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7fc6481f3cb0 con 0x7fc64815e1e0 2024-01-16T14:57:54.611 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:54.611 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc62006b2f0 msgr2=0x7fc62006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc62006b2f0 0x7fc62006d7b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc638006fd0 tx=0x7fc638008040 comp rx=0 tx=0).stop 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 msgr2=0x7fc64813be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64813be20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc640036040 tx=0x7fc640004a40 comp rx=0 tx=0).stop 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 shutdown_connections 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc62006b2f0 0x7fc62006d7b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc64815e1e0 0x7fc64813be20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc648143ae0 0x7fc648141cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 --2- 172.21.15.141:0/408821294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6480cf7d0 0x7fc648141770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 >> 172.21.15.141:0/408821294 conn(0x7fc64806a800 msgr2=0x7fc6480702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:54.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.631+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 shutdown_connections 2024-01-16T14:57:54.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.632+0000 7fc650a00640 1 -- 172.21.15.141:0/408821294 wait complete. 2024-01-16T14:57:54.632 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:54.632 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:54.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.853+0000 7fba0e800640 1 Processor -- start 2024-01-16T14:57:54.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.853+0000 7fba0e800640 1 -- start start 2024-01-16T14:57:54.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fba0e800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fba080d0b60 0x7fba080ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fba0e800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba08071de0 con 0x7fba080aede0 2024-01-16T14:57:54.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fba0e800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba080af270 con 0x7fba0806a3e0 2024-01-16T14:57:54.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fba0e800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba08072030 con 0x7fba080d0b60 2024-01-16T14:57:54.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fba07e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fba080aede0 0x7fba0806f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49590/0 (socket says 172.21.15.141:49590) 2024-01-16T14:57:54.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fba07e00640 1 -- 172.21.15.141:0/2907872629 learned_addr learned my addr 172.21.15.141:0/2907872629 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fb9ffe00640 1 --2- 172.21.15.141:0/2907872629 >> v2:172.21.15.188:3300/0 conn(0x7fba080d0b60 0x7fba080ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.855+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044329608 0 0) 0x7fba08071de0 con 0x7fba080aede0 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e8003520 con 0x7fba080aede0 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297493001 0 0) 0x7fb9e8003520 con 0x7fba080aede0 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 >> v2:172.21.15.188:3300/0 conn(0x7fba080d0b60 msgr2=0x7fba080ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 --2- 172.21.15.141:0/2907872629 >> v2:172.21.15.188:3300/0 conn(0x7fba080d0b60 0x7fba080ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 >> v1:172.21.15.188:6789/0 conn(0x7fba0806a3e0 legacy=0x7fba0806a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba08072310 con 0x7fba080aede0 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fba000019f0 con 0x7fba080aede0 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 >> v1:172.21.15.141:6789/0 conn(0x7fba080aede0 legacy=0x7fba0806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e80039c0 0x7fb9e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 0x7fb9e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8005180 0x7fb9e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba08072030 con 0x7fb9e80039c0 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e800c080 con 0x7fb9e80047c0 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.856+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e800c240 con 0x7fb9e8005180 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07400640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e80039c0 0x7fb9e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07e00640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8005180 0x7fb9e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fb9ffe00640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 0x7fb9e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07400640 1 -- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8005180 msgr2=0x7fb9e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07400640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8005180 0x7fb9e800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07400640 1 -- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 msgr2=0x7fb9e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07400640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 0x7fb9e8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07400640 1 -- 172.21.15.141:0/2907872629 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8003520 con 0x7fb9e80039c0 2024-01-16T14:57:54.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.857+0000 7fba07e00640 1 -- 172.21.15.141:0/2907872629 reap_dead start 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.858+0000 7fba07400640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e80039c0 0x7fb9e8003de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8001750 tx=0x7fb9e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0fa5e8c86d4459 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80052d0 con 0x7fb9e80039c0 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f802e050 con 0x7fb9e80039c0 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba06a00640 1 -- 172.21.15.141:0/2907872629 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8003830 con 0x7fb9e80039c0 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba0e800640 1 -- 172.21.15.141:0/2907872629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e80039c0 msgr2=0x7fb9e8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba0e800640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e80039c0 0x7fb9e8003de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8001750 tx=0x7fb9e80057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba0e800640 1 -- 172.21.15.141:0/2907872629 shutdown_connections 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba0e800640 1 --2- 172.21.15.141:0/2907872629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e80039c0 0x7fb9e8003de0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.859+0000 7fba0e800640 1 -- 172.21.15.141:0/2907872629 >> 172.21.15.141:0/2907872629 conn(0x7fba080cfef0 msgr2=0x7fba080d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.860+0000 7fba0e800640 1 -- 172.21.15.141:0/2907872629 shutdown_connections 2024-01-16T14:57:54.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.860+0000 7fba0e800640 1 -- 172.21.15.141:0/2907872629 wait complete. 2024-01-16T14:57:54.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.861+0000 7fba0e800640 1 Processor -- start 2024-01-16T14:57:54.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.861+0000 7fba0e800640 1 -- start start 2024-01-16T14:57:54.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.862+0000 7fba0e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80039c0 0x7fba081505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.862+0000 7fba0e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba0815fa70 0x7fba080ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.862+0000 7fba0e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba0814fe40 0x7fba080adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.862+0000 7fba0e800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0815fef0 con 0x7fba0814fe40 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.862+0000 7fba0e800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080ce9b0 con 0x7fb9e80039c0 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.862+0000 7fba0e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba080cdfe0 con 0x7fba0815fa70 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba0814fe40 0x7fba080adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba0814fe40 0x7fba080adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42542/0 (socket says 172.21.15.141:42542) 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 -- 172.21.15.141:0/3014172590 learned_addr learned my addr 172.21.15.141:0/3014172590 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fb9ffe00640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba0815fa70 0x7fba080ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07400640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80039c0 0x7fba081505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 -- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba0815fa70 msgr2=0x7fba080ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba0815fa70 0x7fba080ce470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 -- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80039c0 msgr2=0x7fba081505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80039c0 0x7fba081505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 -- 172.21.15.141:0/3014172590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba08159610 con 0x7fba0814fe40 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fb9ffe00640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba0815fa70 0x7fba080ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:54.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.863+0000 7fba07e00640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba0814fe40 0x7fba080adaf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fba0806cfa0 tx=0x7fba0000df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.864+0000 7fb9ff400640 1 -- 172.21.15.141:0/3014172590 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0000b800 con 0x7fba0814fe40 2024-01-16T14:57:54.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.864+0000 7fb9ff400640 1 -- 172.21.15.141:0/3014172590 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0000b960 con 0x7fba0814fe40 2024-01-16T14:57:54.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.864+0000 7fb9ff400640 1 -- 172.21.15.141:0/3014172590 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0000bd10 con 0x7fba0814fe40 2024-01-16T14:57:54.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.864+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba08159880 con 0x7fba0814fe40 2024-01-16T14:57:54.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.864+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba080cdc70 con 0x7fba0814fe40 2024-01-16T14:57:54.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.865+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba080cde30 con 0x7fba0814fe40 2024-01-16T14:57:54.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.866+0000 7fb9ff400640 1 -- 172.21.15.141:0/3014172590 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fba00007070 con 0x7fba0814fe40 2024-01-16T14:57:54.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.866+0000 7fb9ff400640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006b2c0 0x7fb9e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:54.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.867+0000 7fb9ff400640 1 -- 172.21.15.141:0/3014172590 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fba00089a50 con 0x7fba0814fe40 2024-01-16T14:57:54.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.867+0000 7fba07400640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006b2c0 0x7fb9e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:54.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.868+0000 7fba07400640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006b2c0 0x7fb9e006d780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8033000 tx=0x7fb9f8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:54.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:54.871+0000 7fb9ff400640 1 -- 172.21.15.141:0/3014172590 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba00054710 con 0x7fba0814fe40 2024-01-16T14:57:55.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.061+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba080675d0 con 0x7fba0814fe40 2024-01-16T14:57:55.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.062+0000 7fb9ff400640 1 -- 172.21.15.141:0/3014172590 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba00057d30 con 0x7fba0814fe40 2024-01-16T14:57:55.044 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:55.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006b2c0 msgr2=0x7fb9e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006b2c0 0x7fb9e006d780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8033000 tx=0x7fb9f8009030 comp rx=0 tx=0).stop 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba0814fe40 msgr2=0x7fba080adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba0814fe40 0x7fba080adaf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fba0806cfa0 tx=0x7fba0000df70 comp rx=0 tx=0).stop 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 shutdown_connections 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006b2c0 0x7fb9e006d780 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba0814fe40 0x7fba080adaf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba0815fa70 0x7fba080ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 --2- 172.21.15.141:0/3014172590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80039c0 0x7fba081505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 >> 172.21.15.141:0/3014172590 conn(0x7fba080cfef0 msgr2=0x7fba0815c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 shutdown_connections 2024-01-16T14:57:55.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.065+0000 7fba0e800640 1 -- 172.21.15.141:0/3014172590 wait complete. 2024-01-16T14:57:55.066 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:55.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.287+0000 7fc1f1600640 1 Processor -- start 2024-01-16T14:57:55.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.287+0000 7fc1f1600640 1 -- start start 2024-01-16T14:57:55.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1f1600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1ec13e360 0x7fc1ec0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1f1600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc1ec15f9c0 0x7fc1ec0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1f1600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ec15c5f0 con 0x7fc1ec13df80 2024-01-16T14:57:55.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1f1600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec06c470 con 0x7fc1ec13e360 2024-01-16T14:57:55.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1f1600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec06c750 con 0x7fc1ec15f9c0 2024-01-16T14:57:55.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1eaa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc1ec13df80 0x7fc1ec15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49606/0 (socket says 172.21.15.141:49606) 2024-01-16T14:57:55.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1eaa00640 1 -- 172.21.15.141:0/887223977 learned_addr learned my addr 172.21.15.141:0/887223977 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:55.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1e3e00640 1 --2- 172.21.15.141:0/887223977 >> v2:172.21.15.141:3300/0 conn(0x7fc1ec13e360 0x7fc1ec0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1e3e00640 1 -- 172.21.15.141:0/887223977 >> v2:172.21.15.188:3300/0 conn(0x7fc1ec15f9c0 msgr2=0x7fc1ec0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:55.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.289+0000 7fc1eb400640 1 --2- 172.21.15.141:0/887223977 >> v2:172.21.15.188:3300/0 conn(0x7fc1ec15f9c0 0x7fc1ec0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.290+0000 7fc1e3e00640 1 --2- 172.21.15.141:0/887223977 >> v2:172.21.15.188:3300/0 conn(0x7fc1ec15f9c0 0x7fc1ec0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.290+0000 7fc1e3e00640 1 -- 172.21.15.141:0/887223977 >> v1:172.21.15.141:6789/0 conn(0x7fc1ec13df80 legacy=0x7fc1ec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.290+0000 7fc1e3e00640 1 -- 172.21.15.141:0/887223977 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec06ca30 con 0x7fc1ec13e360 2024-01-16T14:57:55.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.290+0000 7fc1eb400640 1 --2- 172.21.15.141:0/887223977 >> v2:172.21.15.188:3300/0 conn(0x7fc1ec15f9c0 0x7fc1ec0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:55.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.290+0000 7fc1ea000640 1 -- 172.21.15.141:0/887223977 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36909627 0 0) 0x7fc1ec15c5f0 con 0x7fc1ec13df80 2024-01-16T14:57:55.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.291+0000 7fc1e3e00640 1 --2- 172.21.15.141:0/887223977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13e360 0x7fc1ec0cf290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0ccae0 tx=0x7fc1d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fdc9859b7d4f49 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.291+0000 7fc1ea000640 1 -- 172.21.15.141:0/887223977 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4031830 con 0x7fc1ec13e360 2024-01-16T14:57:55.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.291+0000 7fc1ea000640 1 -- 172.21.15.141:0/887223977 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d4031990 con 0x7fc1ec13e360 2024-01-16T14:57:55.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.291+0000 7fc1ea000640 1 -- 172.21.15.141:0/887223977 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4031d20 con 0x7fc1ec13e360 2024-01-16T14:57:55.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 -- 172.21.15.141:0/887223977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13e360 msgr2=0x7fc1ec0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 --2- 172.21.15.141:0/887223977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13e360 0x7fc1ec0cf290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0ccae0 tx=0x7fc1d402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 -- 172.21.15.141:0/887223977 shutdown_connections 2024-01-16T14:57:55.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 --2- 172.21.15.141:0/887223977 >> v2:172.21.15.188:3300/0 conn(0x7fc1ec15f9c0 0x7fc1ec0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 --2- 172.21.15.141:0/887223977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13e360 0x7fc1ec0cf290 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 -- 172.21.15.141:0/887223977 >> 172.21.15.141:0/887223977 conn(0x7fc1ec06a800 msgr2=0x7fc1ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:55.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 -- 172.21.15.141:0/887223977 shutdown_connections 2024-01-16T14:57:55.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.292+0000 7fc1f1600640 1 -- 172.21.15.141:0/887223977 wait complete. 2024-01-16T14:57:55.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.293+0000 7fc1f1600640 1 Processor -- start 2024-01-16T14:57:55.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.294+0000 7fc1f1600640 1 -- start start 2024-01-16T14:57:55.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.294+0000 7fc1f1600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13df80 0x7fc1ec1501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.294+0000 7fc1f1600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1ec13e360 0x7fc1ec1f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1f1600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1ec15f9c0 0x7fc1ec131b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1f1600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec131680 con 0x7fc1ec13df80 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1f1600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec131380 con 0x7fc1ec13e360 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1f1600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1506f0 con 0x7fc1ec15f9c0 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13df80 0x7fc1ec1501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13df80 0x7fc1ec1501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42568/0 (socket says 172.21.15.141:42568) 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 -- 172.21.15.141:0/1478725483 learned_addr learned my addr 172.21.15.141:0/1478725483 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 -- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1ec15f9c0 msgr2=0x7fc1ec131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1ec15f9c0 0x7fc1ec131b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 -- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1ec13e360 msgr2=0x7fc1ec1f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1ec13e360 0x7fc1ec1f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 -- 172.21.15.141:0/1478725483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d4009400 con 0x7fc1ec13df80 2024-01-16T14:57:55.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.295+0000 7fc1eaa00640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13df80 0x7fc1ec1501b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00c400 tx=0x7fc1dc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.296+0000 7fc1e2a00640 1 -- 172.21.15.141:0/1478725483 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc005ce0 con 0x7fc1ec13df80 2024-01-16T14:57:55.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.296+0000 7fc1e2a00640 1 -- 172.21.15.141:0/1478725483 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc005e40 con 0x7fc1ec13df80 2024-01-16T14:57:55.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.296+0000 7fc1e2a00640 1 -- 172.21.15.141:0/1478725483 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc0072d0 con 0x7fc1ec13df80 2024-01-16T14:57:55.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.296+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec1f6a90 con 0x7fc1ec13df80 2024-01-16T14:57:55.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.296+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec1f6780 con 0x7fc1ec13df80 2024-01-16T14:57:55.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.297+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec0675d0 con 0x7fc1ec13df80 2024-01-16T14:57:55.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.298+0000 7fc1e2a00640 1 -- 172.21.15.141:0/1478725483 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc1dc007430 con 0x7fc1ec13df80 2024-01-16T14:57:55.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.299+0000 7fc1e2a00640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1c406b2f0 0x7fc1c406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.299+0000 7fc1e2a00640 1 -- 172.21.15.141:0/1478725483 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fc1dc089230 con 0x7fc1ec13df80 2024-01-16T14:57:55.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.299+0000 7fc1e3e00640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1c406b2f0 0x7fc1c406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.300+0000 7fc1e3e00640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1c406b2f0 0x7fc1c406d7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4036000 tx=0x7fc1d4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.303+0000 7fc1e2a00640 1 -- 172.21.15.141:0/1478725483 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1dc054000 con 0x7fc1ec13df80 2024-01-16T14:57:55.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.500+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc1ec0cfad0 con 0x7fc1ec13df80 2024-01-16T14:57:55.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.500+0000 7fc1e2a00640 1 -- 172.21.15.141:0/1478725483 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc1dc057620 con 0x7fc1ec13df80 2024-01-16T14:57:55.483 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1c406b2f0 msgr2=0x7fc1c406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1c406b2f0 0x7fc1c406d7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4036000 tx=0x7fc1d4030000 comp rx=0 tx=0).stop 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13df80 msgr2=0x7fc1ec1501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13df80 0x7fc1ec1501b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00c400 tx=0x7fc1dc00c8c0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 shutdown_connections 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1c406b2f0 0x7fc1c406d7b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1ec15f9c0 0x7fc1ec131b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1ec13e360 0x7fc1ec1f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 --2- 172.21.15.141:0/1478725483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1ec13df80 0x7fc1ec1501b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 >> 172.21.15.141:0/1478725483 conn(0x7fc1ec06a800 msgr2=0x7fc1ec0b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 shutdown_connections 2024-01-16T14:57:55.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.503+0000 7fc1f1600640 1 -- 172.21.15.141:0/1478725483 wait complete. 2024-01-16T14:57:55.503 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:55.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.721+0000 7f7d70200640 1 Processor -- start 2024-01-16T14:57:55.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.721+0000 7f7d70200640 1 -- start start 2024-01-16T14:57:55.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d70200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7d6806a3e0 0x7f7d680ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d70200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d680d0b60 0x7f7d680aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d70200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7d6806a7c0 0x7f7d6806f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d70200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d680af320 con 0x7f7d6806a7c0 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d70200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d68071de0 con 0x7f7d6806a3e0 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d70200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68072030 con 0x7f7d680d0b60 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d6e200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7d6806a7c0 0x7f7d6806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d6e200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7d6806a7c0 0x7f7d6806f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42580/0 (socket says 172.21.15.141:42580) 2024-01-16T14:57:55.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.723+0000 7f7d6e200640 1 -- 172.21.15.141:0/2935415950 learned_addr learned my addr 172.21.15.141:0/2935415950 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.724+0000 7f7d6d800640 1 --2- 172.21.15.141:0/2935415950 >> v2:172.21.15.188:3300/0 conn(0x7f7d6806a3e0 0x7f7d680ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.724+0000 7f7d6ce00640 1 --2- 172.21.15.141:0/2935415950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d680d0b60 0x7f7d680aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.724+0000 7f7d6e200640 1 -- 172.21.15.141:0/2935415950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d680d0b60 msgr2=0x7f7d680aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.724+0000 7f7d6e200640 1 --2- 172.21.15.141:0/2935415950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d680d0b60 0x7f7d680aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.724+0000 7f7d6e200640 1 -- 172.21.15.141:0/2935415950 >> v2:172.21.15.188:3300/0 conn(0x7f7d6806a3e0 msgr2=0x7f7d680ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.724+0000 7f7d6e200640 1 --2- 172.21.15.141:0/2935415950 >> v2:172.21.15.188:3300/0 conn(0x7f7d6806a3e0 0x7f7d680ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.724+0000 7f7d6e200640 1 -- 172.21.15.141:0/2935415950 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68072310 con 0x7f7d6806a7c0 2024-01-16T14:57:55.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.725+0000 7f7d6e200640 1 --2- 172.21.15.141:0/2935415950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a7c0 0x7f7d6806f080 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7d68073fe0 tx=0x7f7d6002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fc22766085755 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.725+0000 7f7d67e00640 1 -- 172.21.15.141:0/2935415950 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6002ca70 con 0x7f7d6806a7c0 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.725+0000 7f7d67e00640 1 -- 172.21.15.141:0/2935415950 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6002cbd0 con 0x7f7d6806a7c0 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.725+0000 7f7d67e00640 1 -- 172.21.15.141:0/2935415950 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d600343f0 con 0x7f7d6806a7c0 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 -- 172.21.15.141:0/2935415950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a7c0 msgr2=0x7f7d6806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 --2- 172.21.15.141:0/2935415950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a7c0 0x7f7d6806f080 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7d68073fe0 tx=0x7f7d6002bed0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 -- 172.21.15.141:0/2935415950 shutdown_connections 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 --2- 172.21.15.141:0/2935415950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a7c0 0x7f7d6806f080 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 --2- 172.21.15.141:0/2935415950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d680d0b60 0x7f7d680aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 --2- 172.21.15.141:0/2935415950 >> v2:172.21.15.188:3300/0 conn(0x7f7d6806a3e0 0x7f7d680ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 -- 172.21.15.141:0/2935415950 >> 172.21.15.141:0/2935415950 conn(0x7f7d680cfef0 msgr2=0x7f7d680d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 -- 172.21.15.141:0/2935415950 shutdown_connections 2024-01-16T14:57:55.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.726+0000 7f7d70200640 1 -- 172.21.15.141:0/2935415950 wait complete. 2024-01-16T14:57:55.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.728+0000 7f7d70200640 1 Processor -- start 2024-01-16T14:57:55.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.728+0000 7f7d70200640 1 -- start start 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d70200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a3e0 0x7f7d6815dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d70200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d6806a7c0 0x7f7d681479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d70200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d680d0b60 0x7f7d680cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d70200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6815e510 con 0x7f7d6806a3e0 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d70200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68147f20 con 0x7f7d680d0b60 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d70200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68158f80 con 0x7f7d6806a7c0 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d6d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a3e0 0x7f7d6815dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d6d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a3e0 0x7f7d6815dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42596/0 (socket says 172.21.15.141:42596) 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.729+0000 7f7d6d800640 1 -- 172.21.15.141:0/879123076 learned_addr learned my addr 172.21.15.141:0/879123076 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:55.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6ce00640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d6806a7c0 0x7f7d681479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6e200640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d680d0b60 0x7f7d680cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6d800640 1 -- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d6806a7c0 msgr2=0x7f7d681479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6d800640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d6806a7c0 0x7f7d681479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6d800640 1 -- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d680d0b60 msgr2=0x7f7d680cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6d800640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d680d0b60 0x7f7d680cd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6d800640 1 -- 172.21.15.141:0/879123076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d60009b30 con 0x7f7d6806a3e0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d6d800640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a3e0 0x7f7d6815dfd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d830 tx=0x7f7d5800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d65600640 1 -- 172.21.15.141:0/879123076 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5800b6c0 con 0x7f7d6806a3e0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d65600640 1 -- 172.21.15.141:0/879123076 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5800b820 con 0x7f7d6806a3e0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d65600640 1 -- 172.21.15.141:0/879123076 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5800bbb0 con 0x7f7d6806a3e0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d68158670 con 0x7f7d6806a3e0 2024-01-16T14:57:55.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.730+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d68158930 con 0x7f7d6806a3e0 2024-01-16T14:57:55.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.732+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d680675d0 con 0x7f7d6806a3e0 2024-01-16T14:57:55.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.732+0000 7f7d65600640 1 -- 172.21.15.141:0/879123076 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d5801e460 con 0x7f7d6806a3e0 2024-01-16T14:57:55.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.734+0000 7f7d65600640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d4006b2c0 0x7f7d4006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:55.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.734+0000 7f7d65600640 1 -- 172.21.15.141:0/879123076 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f7d5808a3d0 con 0x7f7d6806a3e0 2024-01-16T14:57:55.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.734+0000 7f7d6ce00640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d4006b2c0 0x7f7d4006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:55.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.735+0000 7f7d6ce00640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d4006b2c0 0x7f7d4006d780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7d50006fd0 tx=0x7f7d50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:55.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.738+0000 7f7d65600640 1 -- 172.21.15.141:0/879123076 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d580551c0 con 0x7f7d6806a3e0 2024-01-16T14:57:55.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.928+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d680ad320 con 0x7f7d6806a3e0 2024-01-16T14:57:55.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.929+0000 7f7d65600640 1 -- 172.21.15.141:0/879123076 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d580587e0 con 0x7f7d6806a3e0 2024-01-16T14:57:55.912 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:55.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d4006b2c0 msgr2=0x7f7d4006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d4006b2c0 0x7f7d4006d780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7d50006fd0 tx=0x7f7d50008040 comp rx=0 tx=0).stop 2024-01-16T14:57:55.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a3e0 msgr2=0x7f7d6815dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:55.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a3e0 0x7f7d6815dfd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d830 tx=0x7f7d5800dcf0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 shutdown_connections 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d4006b2c0 0x7f7d4006d780 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d680d0b60 0x7f7d680cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d6806a7c0 0x7f7d681479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 --2- 172.21.15.141:0/879123076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d6806a3e0 0x7f7d6815dfd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 >> 172.21.15.141:0/879123076 conn(0x7f7d680cfef0 msgr2=0x7f7d6806d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 shutdown_connections 2024-01-16T14:57:55.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:55.932+0000 7f7d70200640 1 -- 172.21.15.141:0/879123076 wait complete. 2024-01-16T14:57:55.934 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:56.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.160+0000 7f913a000640 1 Processor -- start 2024-01-16T14:57:56.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.160+0000 7f913a000640 1 -- start start 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f913a000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9134157c30 0x7f91340cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f913a000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134071200 0x7f913406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f913a000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913415deb0 con 0x7f91341618f0 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f913a000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913415ea30 con 0x7f9134157c30 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f913a000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913406d860 con 0x7f9134071200 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f9133400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9134157c30 0x7f91340cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f9133400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9134157c30 0x7f91340cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42606/0 (socket says 172.21.15.141:42606) 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.162+0000 7f9133400640 1 -- 172.21.15.141:0/3271409957 learned_addr learned my addr 172.21.15.141:0/3271409957 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:56.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.163+0000 7f9133e00640 1 --2- 172.21.15.141:0/3271409957 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134071200 0x7f913406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.163+0000 7f9133400640 1 -- 172.21.15.141:0/3271409957 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134071200 msgr2=0x7f913406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:56.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.163+0000 7f9133400640 1 --2- 172.21.15.141:0/3271409957 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134071200 0x7f913406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.163+0000 7f9133400640 1 -- 172.21.15.141:0/3271409957 >> v1:172.21.15.188:6789/0 conn(0x7f91341618f0 legacy=0x7f9134070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:56.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.163+0000 7f9133400640 1 -- 172.21.15.141:0/3271409957 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913415e190 con 0x7f9134157c30 2024-01-16T14:57:56.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.164+0000 7f9133400640 1 --2- 172.21.15.141:0/3271409957 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134157c30 0x7f91340cd600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f91340723f0 tx=0x7f912402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3654d45e9d3279d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:56.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.164+0000 7f9132000640 1 -- 172.21.15.141:0/3271409957 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124031830 con 0x7f9134157c30 2024-01-16T14:57:56.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.164+0000 7f9132000640 1 -- 172.21.15.141:0/3271409957 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124031990 con 0x7f9134157c30 2024-01-16T14:57:56.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.164+0000 7f9132000640 1 -- 172.21.15.141:0/3271409957 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124031d20 con 0x7f9134157c30 2024-01-16T14:57:56.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 -- 172.21.15.141:0/3271409957 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134157c30 msgr2=0x7f91340cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 --2- 172.21.15.141:0/3271409957 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134157c30 0x7f91340cd600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f91340723f0 tx=0x7f912402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 -- 172.21.15.141:0/3271409957 shutdown_connections 2024-01-16T14:57:56.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 --2- 172.21.15.141:0/3271409957 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134071200 0x7f913406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 --2- 172.21.15.141:0/3271409957 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134157c30 0x7f91340cd600 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 -- 172.21.15.141:0/3271409957 >> 172.21.15.141:0/3271409957 conn(0x7f91340cf700 msgr2=0x7f91340cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:56.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 -- 172.21.15.141:0/3271409957 shutdown_connections 2024-01-16T14:57:56.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.165+0000 7f913a000640 1 -- 172.21.15.141:0/3271409957 wait complete. 2024-01-16T14:57:56.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.167+0000 7f913a000640 1 Processor -- start 2024-01-16T14:57:56.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.167+0000 7f913a000640 1 -- start start 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f913a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134071200 0x7f91340affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f913a000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134157c30 0x7f91341e7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f913a000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91341618f0 0x7f9134153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f913a000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91341e7cc0 con 0x7f9134071200 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f913a000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91341e7f70 con 0x7f91341618f0 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f913a000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91341e8250 con 0x7f9134157c30 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f9133400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134071200 0x7f91340affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f9133400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134071200 0x7f91340affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42618/0 (socket says 172.21.15.141:42618) 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.168+0000 7f9133400640 1 -- 172.21.15.141:0/2318883245 learned_addr learned my addr 172.21.15.141:0/2318883245 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:56.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9132a00640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134157c30 0x7f91341e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133e00640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91341618f0 0x7f9134153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133400640 1 -- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134157c30 msgr2=0x7f91341e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133400640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134157c30 0x7f91341e7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133400640 1 -- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91341618f0 msgr2=0x7f9134153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133400640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91341618f0 0x7f9134153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133400640 1 -- 172.21.15.141:0/2318883245 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91240094d0 con 0x7f9134071200 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133e00640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91341618f0 0x7f9134153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f9133400640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134071200 0x7f91340affe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9124000c00 tx=0x7f91240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f912b400640 1 -- 172.21.15.141:0/2318883245 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91240317d0 con 0x7f9134071200 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f912b400640 1 -- 172.21.15.141:0/2318883245 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912403aa60 con 0x7f9134071200 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91341e8530 con 0x7f9134071200 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.169+0000 7f912b400640 1 -- 172.21.15.141:0/2318883245 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912403ae10 con 0x7f9134071200 2024-01-16T14:57:56.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.170+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913414f1c0 con 0x7f9134071200 2024-01-16T14:57:56.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.171+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9134161d80 con 0x7f9134071200 2024-01-16T14:57:56.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.171+0000 7f912b400640 1 -- 172.21.15.141:0/2318883245 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f912404d450 con 0x7f9134071200 2024-01-16T14:57:56.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.172+0000 7f912b400640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f910c06b2c0 0x7f910c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.172+0000 7f912b400640 1 -- 172.21.15.141:0/2318883245 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f91240abd30 con 0x7f9134071200 2024-01-16T14:57:56.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.173+0000 7f9132a00640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f910c06b2c0 0x7f910c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.174+0000 7f9132a00640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f910c06b2c0 0x7f910c06d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f91340b0c20 tx=0x7f911c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:56.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.177+0000 7f912b400640 1 -- 172.21.15.141:0/2318883245 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9134161d80 con 0x7f9134071200 2024-01-16T14:57:56.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.376+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9134161d80 con 0x7f9134071200 2024-01-16T14:57:56.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.377+0000 7f912b400640 1 -- 172.21.15.141:0/2318883245 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9134161d80 con 0x7f9134071200 2024-01-16T14:57:56.360 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f910c06b2c0 msgr2=0x7f910c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f910c06b2c0 0x7f910c06d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f91340b0c20 tx=0x7f911c009040 comp rx=0 tx=0).stop 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134071200 msgr2=0x7f91340affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134071200 0x7f91340affe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9124000c00 tx=0x7f91240046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 shutdown_connections 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f910c06b2c0 0x7f910c06d780 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91341618f0 0x7f9134153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9134157c30 0x7f91341e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 --2- 172.21.15.141:0/2318883245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9134071200 0x7f91340affe0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 >> 172.21.15.141:0/2318883245 conn(0x7f91340cf700 msgr2=0x7f91340cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 shutdown_connections 2024-01-16T14:57:56.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.380+0000 7f913a000640 1 -- 172.21.15.141:0/2318883245 wait complete. 2024-01-16T14:57:56.381 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:57:56.381 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:56.438 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:56.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.670+0000 7f5823600640 1 Processor -- start 2024-01-16T14:57:56.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.670+0000 7f5823600640 1 -- start start 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5823600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f581c099a60 0x7f581c099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5823600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f581c06e7c0 0x7f581c06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5823600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c0cfa70 con 0x7f581c15e880 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5823600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f581c09a3e0 con 0x7f581c06e7c0 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5823600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f581c0cff00 con 0x7f581c099a60 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5821600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f581c06e7c0 0x7f581c06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5821600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f581c06e7c0 0x7f581c06ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42630/0 (socket says 172.21.15.141:42630) 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.672+0000 7f5821600640 1 -- 172.21.15.141:0/2614345392 learned_addr learned my addr 172.21.15.141:0/2614345392 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.673+0000 7f5813c00640 1 --2- 172.21.15.141:0/2614345392 >> v2:172.21.15.188:3300/0 conn(0x7f581c099a60 0x7f581c099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.673+0000 7f5821600640 1 -- 172.21.15.141:0/2614345392 >> v2:172.21.15.188:3300/0 conn(0x7f581c099a60 msgr2=0x7f581c099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:56.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.673+0000 7f5821600640 1 --2- 172.21.15.141:0/2614345392 >> v2:172.21.15.188:3300/0 conn(0x7f581c099a60 0x7f581c099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.673+0000 7f5821600640 1 -- 172.21.15.141:0/2614345392 >> v1:172.21.15.188:6789/0 conn(0x7f581c15e880 legacy=0x7f581c06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:56.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.673+0000 7f5821600640 1 -- 172.21.15.141:0/2614345392 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c06fc70 con 0x7f581c06e7c0 2024-01-16T14:57:56.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.674+0000 7f5821600640 1 --2- 172.21.15.141:0/2614345392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c06ec10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f581c071800 tx=0x7f581402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7015f684a380cb04 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:56.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.674+0000 7f581be00640 1 -- 172.21.15.141:0/2614345392 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5814030860 con 0x7f581c06e7c0 2024-01-16T14:57:56.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.674+0000 7f581be00640 1 -- 172.21.15.141:0/2614345392 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58140309c0 con 0x7f581c06e7c0 2024-01-16T14:57:56.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.674+0000 7f581be00640 1 -- 172.21.15.141:0/2614345392 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5814030d50 con 0x7f581c06e7c0 2024-01-16T14:57:56.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 -- 172.21.15.141:0/2614345392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 msgr2=0x7f581c06ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 --2- 172.21.15.141:0/2614345392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c06ec10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f581c071800 tx=0x7f581402b9a0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 -- 172.21.15.141:0/2614345392 shutdown_connections 2024-01-16T14:57:56.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 --2- 172.21.15.141:0/2614345392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c06ec10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 --2- 172.21.15.141:0/2614345392 >> v2:172.21.15.188:3300/0 conn(0x7f581c099a60 0x7f581c099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 -- 172.21.15.141:0/2614345392 >> 172.21.15.141:0/2614345392 conn(0x7f581c0b2d00 msgr2=0x7f581c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:56.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 -- 172.21.15.141:0/2614345392 shutdown_connections 2024-01-16T14:57:56.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.675+0000 7f5823600640 1 -- 172.21.15.141:0/2614345392 wait complete. 2024-01-16T14:57:56.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.677+0000 7f5823600640 1 Processor -- start 2024-01-16T14:57:56.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.677+0000 7f5823600640 1 -- start start 2024-01-16T14:57:56.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5823600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5823600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f581c099a60 0x7f581c0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5823600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f581c15e880 0x7f581c0b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5823600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c0c55a0 con 0x7f581c06e7c0 2024-01-16T14:57:56.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5823600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c0c5c50 con 0x7f581c099a60 2024-01-16T14:57:56.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5823600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581c0c9970 con 0x7f581c15e880 2024-01-16T14:57:56.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c15a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42640/0 (socket says 172.21.15.141:42640) 2024-01-16T14:57:56.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 -- 172.21.15.141:0/3688579142 learned_addr learned my addr 172.21.15.141:0/3688579142 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 -- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f581c15e880 msgr2=0x7f581c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f581c15e880 0x7f581c0b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 -- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f581c099a60 msgr2=0x7f581c0c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f581c099a60 0x7f581c0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.678+0000 7f5820c00640 1 -- 172.21.15.141:0/3688579142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58140094d0 con 0x7f581c06e7c0 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.679+0000 7f5820c00640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c15a0c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f580c00ba50 tx=0x7f580c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.679+0000 7f5819600640 1 -- 172.21.15.141:0/3688579142 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c00cbe0 con 0x7f581c06e7c0 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.679+0000 7f5819600640 1 -- 172.21.15.141:0/3688579142 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c00cd80 con 0x7f581c06e7c0 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.679+0000 7f5819600640 1 -- 172.21.15.141:0/3688579142 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c014680 con 0x7f581c06e7c0 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.679+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c1f2a60 con 0x7f581c06e7c0 2024-01-16T14:57:56.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.679+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c0c9040 con 0x7f581c06e7c0 2024-01-16T14:57:56.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.680+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c0675d0 con 0x7f581c06e7c0 2024-01-16T14:57:56.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.681+0000 7f5819600640 1 -- 172.21.15.141:0/3688579142 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f580c0147e0 con 0x7f581c06e7c0 2024-01-16T14:57:56.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.682+0000 7f5819600640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57f406b2c0 0x7f57f406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:56.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.682+0000 7f5819600640 1 -- 172.21.15.141:0/3688579142 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f580c089b60 con 0x7f581c06e7c0 2024-01-16T14:57:56.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.682+0000 7f5813c00640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57f406b2c0 0x7f57f406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:56.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.683+0000 7f5813c00640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57f406b2c0 0x7f57f406d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f581c0ccfd0 tx=0x7f5804008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:56.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.686+0000 7f5819600640 1 -- 172.21.15.141:0/3688579142 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f580c054960 con 0x7f581c06e7c0 2024-01-16T14:57:56.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.879+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f581c073d50 con 0x7f581c06e7c0 2024-01-16T14:57:56.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.882+0000 7f5819600640 1 -- 172.21.15.141:0/3688579142 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7f580c019020 con 0x7f581c06e7c0 2024-01-16T14:57:56.864 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:56.865 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:56.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57f406b2c0 msgr2=0x7f57f406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57f406b2c0 0x7f57f406d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f581c0ccfd0 tx=0x7f5804008040 comp rx=0 tx=0).stop 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 msgr2=0x7f581c15a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c15a0c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f580c00ba50 tx=0x7f580c00bf10 comp rx=0 tx=0).stop 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 shutdown_connections 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57f406b2c0 0x7f57f406d780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f581c15e880 0x7f581c0b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f581c099a60 0x7f581c0c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 --2- 172.21.15.141:0/3688579142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f581c06e7c0 0x7f581c15a0c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:56.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 >> 172.21.15.141:0/3688579142 conn(0x7f581c0b2d00 msgr2=0x7f581c06bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:56.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.884+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 shutdown_connections 2024-01-16T14:57:56.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:56.885+0000 7f5823600640 1 -- 172.21.15.141:0/3688579142 wait complete. 2024-01-16T14:57:56.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:56.885 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:57.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.111+0000 7fe5caa00640 1 Processor -- start 2024-01-16T14:57:57.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.111+0000 7fe5caa00640 1 -- start start 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5caa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe5c413df80 0x7fe5c415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5caa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5caa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c406c470 con 0x7fe5c413e360 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5caa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c406c750 con 0x7fe5c413df80 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5caa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c406ca30 con 0x7fe5c40cf9a0 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5c3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe5c413df80 0x7fe5c415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5c3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe5c413df80 0x7fe5c415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42648/0 (socket says 172.21.15.141:42648) 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5c3e00640 1 -- 172.21.15.141:0/2171985037 learned_addr learned my addr 172.21.15.141:0/2171985037 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:57.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.113+0000 7fe5bb200640 1 --1- 172.21.15.141:0/2171985037 >> v1:172.21.15.141:6789/0 conn(0x7fe5c413e360 0x7fe5c40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49608/0 (socket says 172.21.15.141:49608) 2024-01-16T14:57:57.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.114+0000 7fe5c8a00640 1 --2- 172.21.15.141:0/2171985037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.114+0000 7fe5c3e00640 1 -- 172.21.15.141:0/2171985037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 msgr2=0x7fe5c40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:57.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.114+0000 7fe5c3e00640 1 --2- 172.21.15.141:0/2171985037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c40cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.114+0000 7fe5c3e00640 1 -- 172.21.15.141:0/2171985037 >> v1:172.21.15.141:6789/0 conn(0x7fe5c413e360 legacy=0x7fe5c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.114+0000 7fe5c3e00640 1 -- 172.21.15.141:0/2171985037 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c406cd10 con 0x7fe5c413df80 2024-01-16T14:57:57.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.115+0000 7fe5c3e00640 1 --2- 172.21.15.141:0/2171985037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413df80 0x7fe5c415bee0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4009e50 tx=0x7fe5b402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb38482e0b37c83d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.116+0000 7fe5c3400640 1 -- 172.21.15.141:0/2171985037 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b402cb00 con 0x7fe5c413df80 2024-01-16T14:57:57.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.116+0000 7fe5c3400640 1 -- 172.21.15.141:0/2171985037 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b402cc60 con 0x7fe5c413df80 2024-01-16T14:57:57.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.116+0000 7fe5c3400640 1 -- 172.21.15.141:0/2171985037 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40343f0 con 0x7fe5c413df80 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.116+0000 7fe5caa00640 1 -- 172.21.15.141:0/2171985037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413df80 msgr2=0x7fe5c415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.116+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2171985037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413df80 0x7fe5c415bee0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4009e50 tx=0x7fe5b402bf60 comp rx=0 tx=0).stop 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.117+0000 7fe5caa00640 1 -- 172.21.15.141:0/2171985037 shutdown_connections 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.117+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2171985037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c40cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.117+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2171985037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413df80 0x7fe5c415bee0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.117+0000 7fe5caa00640 1 -- 172.21.15.141:0/2171985037 >> 172.21.15.141:0/2171985037 conn(0x7fe5c406a800 msgr2=0x7fe5c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.117+0000 7fe5caa00640 1 -- 172.21.15.141:0/2171985037 shutdown_connections 2024-01-16T14:57:57.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.117+0000 7fe5caa00640 1 -- 172.21.15.141:0/2171985037 wait complete. 2024-01-16T14:57:57.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.118+0000 7fe5caa00640 1 Processor -- start 2024-01-16T14:57:57.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.119+0000 7fe5caa00640 1 -- start start 2024-01-16T14:57:57.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.119+0000 7fe5caa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c4070540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.119+0000 7fe5caa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5c413df80 0x7fe5c41356c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.119+0000 7fe5caa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413e360 0x7fe5c4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.119+0000 7fe5caa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4071850 con 0x7fe5c413e360 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.119+0000 7fe5caa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4135c00 con 0x7fe5c413df80 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.119+0000 7fe5caa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4070c40 con 0x7fe5c40cf9a0 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413e360 0x7fe5c4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413e360 0x7fe5c4071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42664/0 (socket says 172.21.15.141:42664) 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 -- 172.21.15.141:0/2434575350 learned_addr learned my addr 172.21.15.141:0/2434575350 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c3e00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c4070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5bb200640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5c413df80 0x7fe5c41356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 -- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 msgr2=0x7fe5c4070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c4070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 -- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5c413df80 msgr2=0x7fe5c41356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5c413df80 0x7fe5c41356c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 -- 172.21.15.141:0/2434575350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b4009b30 con 0x7fe5c413e360 2024-01-16T14:57:57.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.120+0000 7fe5c8a00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413e360 0x7fe5c4071ce0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00e7f0 tx=0x7fe5bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.121+0000 7fe5c0c00640 1 -- 172.21.15.141:0/2434575350 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc0094b0 con 0x7fe5c413e360 2024-01-16T14:57:57.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.121+0000 7fe5c0c00640 1 -- 172.21.15.141:0/2434575350 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5bc00c730 con 0x7fe5c413e360 2024-01-16T14:57:57.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.121+0000 7fe5c0c00640 1 -- 172.21.15.141:0/2434575350 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00cae0 con 0x7fe5c413e360 2024-01-16T14:57:57.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.121+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c4070f20 con 0x7fe5c413e360 2024-01-16T14:57:57.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.121+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c40714e0 con 0x7fe5c413e360 2024-01-16T14:57:57.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.122+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c4071a00 con 0x7fe5c413e360 2024-01-16T14:57:57.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.123+0000 7fe5c0c00640 1 -- 172.21.15.141:0/2434575350 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe5bc00cc40 con 0x7fe5c413e360 2024-01-16T14:57:57.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.124+0000 7fe5c0c00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe59c06b2c0 0x7fe59c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.124+0000 7fe5c0c00640 1 -- 172.21.15.141:0/2434575350 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fe5bc089da0 con 0x7fe5c413e360 2024-01-16T14:57:57.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.125+0000 7fe5c3e00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe59c06b2c0 0x7fe59c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.125+0000 7fe5c3e00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe59c06b2c0 0x7fe59c06d780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4035000 tx=0x7fe5b4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.128+0000 7fe5c0c00640 1 -- 172.21.15.141:0/2434575350 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5c4071a00 con 0x7fe5c413e360 2024-01-16T14:57:57.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.316+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5c40675d0 con 0x7fe5c413e360 2024-01-16T14:57:57.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.317+0000 7fe5c0c00640 1 -- 172.21.15.141:0/2434575350 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5bc0170a0 con 0x7fe5c413e360 2024-01-16T14:57:57.300 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:57.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe59c06b2c0 msgr2=0x7fe59c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe59c06b2c0 0x7fe59c06d780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4035000 tx=0x7fe5b4030000 comp rx=0 tx=0).stop 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413e360 msgr2=0x7fe5c4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413e360 0x7fe5c4071ce0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00e7f0 tx=0x7fe5bc00ecb0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 shutdown_connections 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe59c06b2c0 0x7fe59c06d780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe5c413e360 0x7fe5c4071ce0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5c413df80 0x7fe5c41356c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 --2- 172.21.15.141:0/2434575350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5c40cf9a0 0x7fe5c4070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 >> 172.21.15.141:0/2434575350 conn(0x7fe5c406a800 msgr2=0x7fe5c40afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 shutdown_connections 2024-01-16T14:57:57.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.320+0000 7fe5caa00640 1 -- 172.21.15.141:0/2434575350 wait complete. 2024-01-16T14:57:57.320 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:57.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.548+0000 7fb6ca200640 1 Processor -- start 2024-01-16T14:57:57.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.548+0000 7fb6ca200640 1 -- start start 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6ca200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6ca200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb6c40cf9a0 0x7fb6c40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6ca200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c406c470 con 0x7fb6c413e360 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6ca200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c406c750 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6ca200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c406ca30 con 0x7fb6c413df80 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6c3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb6c40cf9a0 0x7fb6c40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6c3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb6c40cf9a0 0x7fb6c40cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42678/0 (socket says 172.21.15.141:42678) 2024-01-16T14:57:57.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.550+0000 7fb6c3e00640 1 -- 172.21.15.141:0/4103963199 learned_addr learned my addr 172.21.15.141:0/4103963199 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:57.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.551+0000 7fb6c3400640 1 --2- 172.21.15.141:0/4103963199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.551+0000 7fb6c3e00640 1 -- 172.21.15.141:0/4103963199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413df80 msgr2=0x7fb6c415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:57.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.551+0000 7fb6c3e00640 1 --2- 172.21.15.141:0/4103963199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.551+0000 7fb6c3e00640 1 -- 172.21.15.141:0/4103963199 >> v1:172.21.15.141:6789/0 conn(0x7fb6c413e360 legacy=0x7fb6c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.551+0000 7fb6c3e00640 1 -- 172.21.15.141:0/4103963199 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c406cd10 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.551+0000 7fb6c2a00640 1 -- 172.21.15.141:0/4103963199 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243133387 0 0) 0x7fb6c406c470 con 0x7fb6c413e360 2024-01-16T14:57:57.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.551+0000 7fb6c3400640 1 --2- 172.21.15.141:0/4103963199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:57.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.552+0000 7fb6c3e00640 1 --2- 172.21.15.141:0/4103963199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c40cd840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009e50 tx=0x7fb6bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99ac8b59b76964f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.552+0000 7fb6c2a00640 1 -- 172.21.15.141:0/4103963199 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc02cb00 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.552+0000 7fb6c2a00640 1 -- 172.21.15.141:0/4103963199 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc02cc60 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.552+0000 7fb6c2a00640 1 -- 172.21.15.141:0/4103963199 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0343f0 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 -- 172.21.15.141:0/4103963199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 msgr2=0x7fb6c40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 --2- 172.21.15.141:0/4103963199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c40cd840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009e50 tx=0x7fb6bc02bf60 comp rx=0 tx=0).stop 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 -- 172.21.15.141:0/4103963199 shutdown_connections 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 --2- 172.21.15.141:0/4103963199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c40cd840 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 --2- 172.21.15.141:0/4103963199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413df80 0x7fb6c415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 -- 172.21.15.141:0/4103963199 >> 172.21.15.141:0/4103963199 conn(0x7fb6c406a800 msgr2=0x7fb6c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 -- 172.21.15.141:0/4103963199 shutdown_connections 2024-01-16T14:57:57.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.553+0000 7fb6ca200640 1 -- 172.21.15.141:0/4103963199 wait complete. 2024-01-16T14:57:57.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.555+0000 7fb6ca200640 1 Processor -- start 2024-01-16T14:57:57.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.555+0000 7fb6ca200640 1 -- start start 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6ca200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c4071650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6ca200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c413df80 0x7fb6c40af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6ca200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413e360 0x7fb6c4152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6ca200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c40702c0 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6ca200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c40b3310 con 0x7fb6c413df80 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6ca200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c40704a0 con 0x7fb6c413e360 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c4071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c4071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42688/0 (socket says 172.21.15.141:42688) 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.556+0000 7fb6c3400640 1 -- 172.21.15.141:0/2078156463 learned_addr learned my addr 172.21.15.141:0/2078156463 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:57.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6ba800640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c413df80 0x7fb6c40af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6c3e00640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413e360 0x7fb6c4152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6c3400640 1 -- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413e360 msgr2=0x7fb6c4152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6c3400640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413e360 0x7fb6c4152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6c3400640 1 -- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c413df80 msgr2=0x7fb6c40af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6c3400640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c413df80 0x7fb6c40af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6c3400640 1 -- 172.21.15.141:0/2078156463 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc009b30 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6ba800640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c413df80 0x7fb6c40af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6c3400640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c4071650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb6b400c7f0 tx=0x7fb6b400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6bbe00640 1 -- 172.21.15.141:0/2078156463 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4007a30 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6bbe00640 1 -- 172.21.15.141:0/2078156463 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b4007b90 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6bbe00640 1 -- 172.21.15.141:0/2078156463 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b400ce40 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.557+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c4070780 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.558+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c40afac0 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.559+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c40afd10 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.559+0000 7fb6bbe00640 1 -- 172.21.15.141:0/2078156463 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb6b40113f0 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.560+0000 7fb6bbe00640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb69c06b2c0 0x7fb69c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.560+0000 7fb6bbe00640 1 -- 172.21.15.141:0/2078156463 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fb6b4089d60 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.561+0000 7fb6ba800640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb69c06b2c0 0x7fb69c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.562+0000 7fb6ba800640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb69c06b2c0 0x7fb69c06d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb6c415c0b0 tx=0x7fb6ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.565+0000 7fb6bbe00640 1 -- 172.21.15.141:0/2078156463 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6b4054be0 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.754+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb6c41f2f10 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.755+0000 7fb6bbe00640 1 -- 172.21.15.141:0/2078156463 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb6b401f020 con 0x7fb6c40cf9a0 2024-01-16T14:57:57.738 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:57:57.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb69c06b2c0 msgr2=0x7fb69c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb69c06b2c0 0x7fb69c06d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb6c415c0b0 tx=0x7fb6ac009040 comp rx=0 tx=0).stop 2024-01-16T14:57:57.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 msgr2=0x7fb6c4071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c4071650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb6b400c7f0 tx=0x7fb6b400cbf0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 shutdown_connections 2024-01-16T14:57:57.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb69c06b2c0 0x7fb69c06d780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6c413e360 0x7fb6c4152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c413df80 0x7fb6c40af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 --2- 172.21.15.141:0/2078156463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6c40cf9a0 0x7fb6c4071650 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 >> 172.21.15.141:0/2078156463 conn(0x7fb6c406a800 msgr2=0x7fb6c415df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:57.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 shutdown_connections 2024-01-16T14:57:57.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.758+0000 7fb6ca200640 1 -- 172.21.15.141:0/2078156463 wait complete. 2024-01-16T14:57:57.760 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:57:57.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.985+0000 7f9823200640 1 Processor -- start 2024-01-16T14:57:57.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.985+0000 7f9823200640 1 -- start start 2024-01-16T14:57:57.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.987+0000 7f9823200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f981c13df80 0x7f981c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.987+0000 7f9823200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c06c470 con 0x7f981c13e360 2024-01-16T14:57:57.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.987+0000 7f9823200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c06c750 con 0x7f981c0cf9a0 2024-01-16T14:57:57.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.987+0000 7f9823200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981c06ca30 con 0x7f981c13df80 2024-01-16T14:57:57.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.987+0000 7f981be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f981c13df80 0x7f981c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.987+0000 7f981b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f981c13e360 0x7f981c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49626/0 (socket says 172.21.15.141:49626) 2024-01-16T14:57:57.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.987+0000 7f981b400640 1 -- 172.21.15.141:0/3854069905 learned_addr learned my addr 172.21.15.141:0/3854069905 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:57.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.988+0000 7f981be00640 1 -- 172.21.15.141:0/3854069905 >> v1:172.21.15.188:6789/0 conn(0x7f981c0cf9a0 legacy=0x7f981c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:57.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.988+0000 7f981be00640 1 -- 172.21.15.141:0/3854069905 >> v1:172.21.15.141:6789/0 conn(0x7f981c13e360 legacy=0x7f981c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.988+0000 7f981be00640 1 -- 172.21.15.141:0/3854069905 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c06cd10 con 0x7f981c13df80 2024-01-16T14:57:57.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.989+0000 7f981be00640 1 --2- 172.21.15.141:0/3854069905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c13df80 0x7f981c15bee0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f980c0097a0 tx=0x7f980c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f5c2eb73ca353e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.989+0000 7f981aa00640 1 -- 172.21.15.141:0/3854069905 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c031830 con 0x7f981c13df80 2024-01-16T14:57:57.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f981aa00640 1 -- 172.21.15.141:0/3854069905 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c031990 con 0x7f981c13df80 2024-01-16T14:57:57.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f981aa00640 1 -- 172.21.15.141:0/3854069905 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c031d20 con 0x7f981c13df80 2024-01-16T14:57:57.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f9823200640 1 -- 172.21.15.141:0/3854069905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c13df80 msgr2=0x7f981c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f9823200640 1 --2- 172.21.15.141:0/3854069905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c13df80 0x7f981c15bee0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f980c0097a0 tx=0x7f980c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f9823200640 1 -- 172.21.15.141:0/3854069905 shutdown_connections 2024-01-16T14:57:57.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f9823200640 1 --2- 172.21.15.141:0/3854069905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c13df80 0x7f981c15bee0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f9823200640 1 -- 172.21.15.141:0/3854069905 >> 172.21.15.141:0/3854069905 conn(0x7f981c06a800 msgr2=0x7f981c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:57.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f9823200640 1 -- 172.21.15.141:0/3854069905 shutdown_connections 2024-01-16T14:57:57.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.990+0000 7f9823200640 1 -- 172.21.15.141:0/3854069905 wait complete. 2024-01-16T14:57:57.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.992+0000 7f9823200640 1 Processor -- start 2024-01-16T14:57:57.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.992+0000 7f9823200640 1 -- start start 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f9823200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c0cf9a0 0x7f981c1e73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f9823200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981c13df80 0x7f981c15a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f9823200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f981c13e360 0x7f981c1f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f9823200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c1e7910 con 0x7f981c0cf9a0 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f9823200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c070850 con 0x7f981c13e360 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f9823200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c070e80 con 0x7f981c13df80 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f981be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c0cf9a0 0x7f981c1e73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f981be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c0cf9a0 0x7f981c1e73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42710/0 (socket says 172.21.15.141:42710) 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.993+0000 7f981be00640 1 -- 172.21.15.141:0/2711074067 learned_addr learned my addr 172.21.15.141:0/2711074067 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f981b400640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981c13df80 0x7f981c15a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f9821200640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f981c13e360 0x7f981c1f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f981be00640 1 -- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981c13df80 msgr2=0x7f981c15a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f981be00640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981c13df80 0x7f981c15a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f981be00640 1 -- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f981c13e360 msgr2=0x7f981c1f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f981be00640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f981c13e360 0x7f981c1f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f981be00640 1 -- 172.21.15.141:0/2711074067 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c009400 con 0x7f981c0cf9a0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f981be00640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c0cf9a0 0x7f981c1e73d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f980c000c00 tx=0x7f980c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f9813e00640 1 -- 172.21.15.141:0/2711074067 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c03ba70 con 0x7f981c0cf9a0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f9813e00640 1 -- 172.21.15.141:0/2711074067 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c03bbf0 con 0x7f981c0cf9a0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f9813e00640 1 -- 172.21.15.141:0/2711074067 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c0453f0 con 0x7f981c0cf9a0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.994+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c071160 con 0x7f981c0cf9a0 2024-01-16T14:57:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.995+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c070490 con 0x7f981c0cf9a0 2024-01-16T14:57:57.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.996+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c0717b0 con 0x7f981c0cf9a0 2024-01-16T14:57:57.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.997+0000 7f9813e00640 1 -- 172.21.15.141:0/2711074067 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f980c04e450 con 0x7f981c0cf9a0 2024-01-16T14:57:57.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.997+0000 7f9813e00640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97f406b2c0 0x7f97f406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:57.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.998+0000 7f9813e00640 1 -- 172.21.15.141:0/2711074067 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f980c0abf80 con 0x7f981c0cf9a0 2024-01-16T14:57:57.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.998+0000 7f981b400640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97f406b2c0 0x7f97f406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:57.999+0000 7f981b400640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97f406b2c0 0x7f97f406d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f98040012e0 tx=0x7f9804009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.004+0000 7f9813e00640 1 -- 172.21.15.141:0/2711074067 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f980c076df0 con 0x7f981c0cf9a0 2024-01-16T14:57:58.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.186+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f981c0675d0 con 0x7f981c0cf9a0 2024-01-16T14:57:58.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.187+0000 7f9813e00640 1 -- 172.21.15.141:0/2711074067 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f980c07a410 con 0x7f981c0cf9a0 2024-01-16T14:57:58.170 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:57:58.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97f406b2c0 msgr2=0x7f97f406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97f406b2c0 0x7f97f406d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f98040012e0 tx=0x7f9804009040 comp rx=0 tx=0).stop 2024-01-16T14:57:58.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c0cf9a0 msgr2=0x7f981c1e73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c0cf9a0 0x7f981c1e73d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f980c000c00 tx=0x7f980c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 shutdown_connections 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97f406b2c0 0x7f97f406d780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f981c13e360 0x7f981c1f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981c13df80 0x7f981c15a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 --2- 172.21.15.141:0/2711074067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981c0cf9a0 0x7f981c1e73d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 >> 172.21.15.141:0/2711074067 conn(0x7f981c06a800 msgr2=0x7f981c15d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 shutdown_connections 2024-01-16T14:57:58.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.190+0000 7f9823200640 1 -- 172.21.15.141:0/2711074067 wait complete. 2024-01-16T14:57:58.190 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:57:58.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.413+0000 7f8de9200640 1 Processor -- start 2024-01-16T14:57:58.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.413+0000 7f8de9200640 1 -- start start 2024-01-16T14:57:58.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.415+0000 7f8de9200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8de414b2c0 0x7f8de40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:58.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.415+0000 7f8de9200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:58.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.415+0000 7f8de9200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40cfbf0 con 0x7f8de4070ea0 2024-01-16T14:57:58.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.415+0000 7f8de9200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de414b6a0 con 0x7f8de414b2c0 2024-01-16T14:57:58.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.415+0000 7f8de9200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de4071690 con 0x7f8de40cf7d0 2024-01-16T14:57:58.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8de414b2c0 0x7f8de40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:58.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:58.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8de414b2c0 0x7f8de40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:44700/0 (socket says 172.21.15.141:44700) 2024-01-16T14:57:58.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1e00640 1 -- 172.21.15.141:0/1941832310 learned_addr learned my addr 172.21.15.141:0/1941832310 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:58.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1e00640 1 -- 172.21.15.141:0/1941832310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 msgr2=0x7f8de4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:58.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1e00640 1 --2- 172.21.15.141:0/1941832310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1e00640 1 -- 172.21.15.141:0/1941832310 >> v1:172.21.15.188:6789/0 conn(0x7f8de4070ea0 legacy=0x7f8de40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.416+0000 7f8de1e00640 1 -- 172.21.15.141:0/1941832310 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4071970 con 0x7f8de414b2c0 2024-01-16T14:57:58.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.418+0000 7f8de1e00640 1 --2- 172.21.15.141:0/1941832310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de414b2c0 0x7f8de40cf290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009870 tx=0x7f8dd402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b2a374d74b8e9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:58.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de0a00640 1 -- 172.21.15.141:0/1941832310 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4030840 con 0x7f8de414b2c0 2024-01-16T14:57:58.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de0a00640 1 -- 172.21.15.141:0/1941832310 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd40309a0 con 0x7f8de414b2c0 2024-01-16T14:57:58.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de0a00640 1 -- 172.21.15.141:0/1941832310 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4030d30 con 0x7f8de414b2c0 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de9200640 1 -- 172.21.15.141:0/1941832310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de414b2c0 msgr2=0x7f8de40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de9200640 1 --2- 172.21.15.141:0/1941832310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de414b2c0 0x7f8de40cf290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009870 tx=0x7f8dd402b980 comp rx=0 tx=0).stop 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de9200640 1 -- 172.21.15.141:0/1941832310 shutdown_connections 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de9200640 1 --2- 172.21.15.141:0/1941832310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de9200640 1 --2- 172.21.15.141:0/1941832310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de414b2c0 0x7f8de40cf290 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de9200640 1 -- 172.21.15.141:0/1941832310 >> 172.21.15.141:0/1941832310 conn(0x7f8de406a800 msgr2=0x7f8de406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.419+0000 7f8de9200640 1 -- 172.21.15.141:0/1941832310 shutdown_connections 2024-01-16T14:57:58.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.420+0000 7f8de9200640 1 -- 172.21.15.141:0/1941832310 wait complete. 2024-01-16T14:57:58.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.422+0000 7f8de9200640 1 Processor -- start 2024-01-16T14:57:58.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.422+0000 7f8de9200640 1 -- start start 2024-01-16T14:57:58.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de9200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de4070ea0 0x7f8de415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de9200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de9200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de414b2c0 0x7f8de4163ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de9200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de4163900 con 0x7f8de414b2c0 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de9200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de41476c0 con 0x7f8de4070ea0 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de9200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de4146430 con 0x7f8de40cf7d0 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de414b2c0 0x7f8de4163ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de414b2c0 0x7f8de4163ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42718/0 (socket says 172.21.15.141:42718) 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de2800640 1 -- 172.21.15.141:0/2103373639 learned_addr learned my addr 172.21.15.141:0/2103373639 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.423+0000 7f8de1e00640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de4070ea0 0x7f8de415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de1400640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de2800640 1 -- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 msgr2=0x7f8de4147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de2800640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4147180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de2800640 1 -- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de4070ea0 msgr2=0x7f8de415dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de2800640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de4070ea0 0x7f8de415dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de2800640 1 -- 172.21.15.141:0/2103373639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd40094d0 con 0x7f8de414b2c0 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de2800640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de414b2c0 0x7f8de4163ad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00c7a0 tx=0x7f8ddc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8dda000640 1 -- 172.21.15.141:0/2103373639 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc003b70 con 0x7f8de414b2c0 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8dda000640 1 -- 172.21.15.141:0/2103373639 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ddc003cf0 con 0x7f8de414b2c0 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8dda000640 1 -- 172.21.15.141:0/2103373639 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc00aa90 con 0x7f8de414b2c0 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de41466a0 con 0x7f8de414b2c0 2024-01-16T14:57:58.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.424+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de4146c60 con 0x7f8de414b2c0 2024-01-16T14:57:58.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.426+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de40675d0 con 0x7f8de414b2c0 2024-01-16T14:57:58.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.427+0000 7f8dda000640 1 -- 172.21.15.141:0/2103373639 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8ddc00abf0 con 0x7f8de414b2c0 2024-01-16T14:57:58.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.427+0000 7f8dda000640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8dbc06b2c0 0x7f8dbc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:58.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.427+0000 7f8dda000640 1 -- 172.21.15.141:0/2103373639 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f8ddc089ab0 con 0x7f8de414b2c0 2024-01-16T14:57:58.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.428+0000 7f8de1e00640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8dbc06b2c0 0x7f8dbc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:58.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.429+0000 7f8de1e00640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8dbc06b2c0 0x7f8dbc06d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4036000 tx=0x7f8dd403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:58.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.432+0000 7f8dda000640 1 -- 172.21.15.141:0/2103373639 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ddc0548a0 con 0x7f8de414b2c0 2024-01-16T14:57:58.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.637+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8de415e980 con 0x7f8de414b2c0 2024-01-16T14:57:58.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.638+0000 7f8dda000640 1 -- 172.21.15.141:0/2103373639 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8ddc003e60 con 0x7f8de414b2c0 2024-01-16T14:57:58.620 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:57:58.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.640+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8dbc06b2c0 msgr2=0x7f8dbc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.640+0000 7f8de9200640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8dbc06b2c0 0x7f8dbc06d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4036000 tx=0x7f8dd403b040 comp rx=0 tx=0).stop 2024-01-16T14:57:58.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.640+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de414b2c0 msgr2=0x7f8de4163ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.640+0000 7f8de9200640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de414b2c0 0x7f8de4163ad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00c7a0 tx=0x7f8ddc00cc60 comp rx=0 tx=0).stop 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 shutdown_connections 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8dbc06b2c0 0x7f8dbc06d780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de414b2c0 0x7f8de4163ad0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8de40cf7d0 0x7f8de4147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 --2- 172.21.15.141:0/2103373639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de4070ea0 0x7f8de415dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 >> 172.21.15.141:0/2103373639 conn(0x7f8de406a800 msgr2=0x7f8de40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 shutdown_connections 2024-01-16T14:57:58.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:58.641+0000 7f8de9200640 1 -- 172.21.15.141:0/2103373639 wait complete. 2024-01-16T14:57:58.642 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T14:57:58.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:58.642 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T14:57:58.744 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:57:58.745 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:57:58.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:57:58.745 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T14:57:58.803 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:57:58.803 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:58.858 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:57:58.858 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:57:58.908 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:57:59.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.127+0000 7f4476a00640 1 Processor -- start 2024-01-16T14:57:59.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.127+0000 7f4476a00640 1 -- start start 2024-01-16T14:57:59.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.128+0000 7f4476a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f447013cfa0 0x7f447015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.128+0000 7f4476a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f44700cd530 0x7f447013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.128+0000 7f4476a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44700cdec0 con 0x7f44700cd9b0 2024-01-16T14:57:59.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.128+0000 7f4476a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44700ce1a0 con 0x7f44700cd530 2024-01-16T14:57:59.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.128+0000 7f4476a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44700ce480 con 0x7f447013cfa0 2024-01-16T14:57:59.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.129+0000 7f4476000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f44700cd9b0 0x7f447015d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49636/0 (socket says 172.21.15.141:49636) 2024-01-16T14:57:59.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.129+0000 7f4476000640 1 -- 172.21.15.141:0/240682397 learned_addr learned my addr 172.21.15.141:0/240682397 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:59.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.129+0000 7f4475600640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f447013cfa0 0x7f447015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.129+0000 7f4474c00640 1 --2- 172.21.15.141:0/240682397 >> v2:172.21.15.188:3300/0 conn(0x7f44700cd530 0x7f447013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.129+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600496497 0 0) 0x7f44700cdec0 con 0x7f44700cd9b0 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.129+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4450005730 con 0x7f44700cd9b0 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4029774734 0 0) 0x7f4450005730 con 0x7f44700cd9b0 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f447013cfa0 msgr2=0x7f447015afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f447013cfa0 0x7f447015afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 >> v2:172.21.15.188:3300/0 conn(0x7f44700cd530 msgr2=0x7f447013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 --2- 172.21.15.141:0/240682397 >> v2:172.21.15.188:3300/0 conn(0x7f44700cd530 0x7f447013f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44700ce760 con 0x7f44700cd9b0 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4468001690 con 0x7f44700cd9b0 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 >> v1:172.21.15.141:6789/0 conn(0x7f44700cd9b0 legacy=0x7f447015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4450005b80 0x7f4450005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f447013cfa0 0x7f44500069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4450006f10 0x7f4450007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700ce1a0 con 0x7f4450005b80 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700ce480 con 0x7f4450006f10 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445000db90 con 0x7f447013cfa0 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f4475600640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4450005b80 0x7f4450005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f4475600640 1 -- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f447013cfa0 msgr2=0x7f44500069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f4475600640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f447013cfa0 0x7f44500069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f4475600640 1 -- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4450006f10 msgr2=0x7f4450007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f4475600640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4450006f10 0x7f4450007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.130+0000 7f4475600640 1 -- 172.21.15.141:0/240682397 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450005730 con 0x7f4450005b80 2024-01-16T14:57:59.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.131+0000 7f4475600640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4450005b80 0x7f4450005fa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4460009fc0 tx=0x7f4450007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71ab7be39c2b3a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446002cc60 con 0x7f4450005b80 2024-01-16T14:57:59.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446000f040 con 0x7f4450005b80 2024-01-16T14:57:59.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f446fe00640 1 -- 172.21.15.141:0/240682397 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460034440 con 0x7f4450005b80 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f4476a00640 1 -- 172.21.15.141:0/240682397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4450005b80 msgr2=0x7f4450005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f4476a00640 1 --2- 172.21.15.141:0/240682397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4450005b80 0x7f4450005fa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4460009fc0 tx=0x7f4450007a00 comp rx=0 tx=0).stop 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f4476000640 1 -- 172.21.15.141:0/240682397 reap_dead start 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f4476a00640 1 -- 172.21.15.141:0/240682397 shutdown_connections 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f4476a00640 1 -- 172.21.15.141:0/240682397 >> 172.21.15.141:0/240682397 conn(0x7f447006a800 msgr2=0x7f447006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f4476a00640 1 -- 172.21.15.141:0/240682397 shutdown_connections 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.132+0000 7f4476a00640 1 -- 172.21.15.141:0/240682397 wait complete. 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.134+0000 7f4476a00640 1 Processor -- start 2024-01-16T14:57:59.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.134+0000 7f4476a00640 1 -- start start 2024-01-16T14:57:59.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4450005b80 0x7f4470158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447013c830 0x7f4470159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4470156600 0x7f4470156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700c84d0 con 0x7f4470156600 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700c8180 con 0x7f447013c830 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4470157330 con 0x7f4450005b80 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4470156600 0x7f4470156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4470156600 0x7f4470156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42736/0 (socket says 172.21.15.141:42736) 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.135+0000 7f4476000640 1 -- 172.21.15.141:0/3681173295 learned_addr learned my addr 172.21.15.141:0/3681173295 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4475600640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4450005b80 0x7f4470158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4474c00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447013c830 0x7f4470159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4476000640 1 -- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4450005b80 msgr2=0x7f4470158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4476000640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4450005b80 0x7f4470158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4476000640 1 -- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447013c830 msgr2=0x7f4470159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4476000640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447013c830 0x7f4470159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4476000640 1 -- 172.21.15.141:0/3681173295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470157610 con 0x7f4470156600 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4474c00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447013c830 0x7f4470159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4475600640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4450005b80 0x7f4470158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4476000640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4470156600 0x7f4470156a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f447006cf00 tx=0x7f446800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f446d600640 1 -- 172.21.15.141:0/3681173295 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4468007230 con 0x7f4470156600 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f446d600640 1 -- 172.21.15.141:0/3681173295 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446800ced0 con 0x7f4470156600 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f446d600640 1 -- 172.21.15.141:0/3681173295 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44680075f0 con 0x7f4470156600 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.136+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44701578f0 con 0x7f4470156600 2024-01-16T14:57:59.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.137+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44701f31a0 con 0x7f4470156600 2024-01-16T14:57:59.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.138+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4470067450 con 0x7f4470156600 2024-01-16T14:57:59.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.139+0000 7f446d600640 1 -- 172.21.15.141:0/3681173295 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f44680047e0 con 0x7f4470156600 2024-01-16T14:57:59.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.139+0000 7f446d600640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f444806b2c0 0x7f444806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.139+0000 7f446d600640 1 -- 172.21.15.141:0/3681173295 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f4468088e50 con 0x7f4470156600 2024-01-16T14:57:59.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.140+0000 7f4475600640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f444806b2c0 0x7f444806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.141+0000 7f4475600640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f444806b2c0 0x7f444806d780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4460036000 tx=0x7f446002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.143+0000 7f446d600640 1 -- 172.21.15.141:0/3681173295 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4468053bb0 con 0x7f4470156600 2024-01-16T14:57:59.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.319+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f447014d580 con 0x7f4470156600 2024-01-16T14:57:59.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.321+0000 7f446d600640 1 -- 172.21.15.141:0/3681173295 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7f44680571d0 con 0x7f4470156600 2024-01-16T14:57:59.304 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:57:59.304 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:57:59.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.323+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f444806b2c0 msgr2=0x7f444806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.323+0000 7f4476a00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f444806b2c0 0x7f444806d780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4460036000 tx=0x7f446002e000 comp rx=0 tx=0).stop 2024-01-16T14:57:59.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4470156600 msgr2=0x7f4470156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4470156600 0x7f4470156a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f447006cf00 tx=0x7f446800cb90 comp rx=0 tx=0).stop 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 shutdown_connections 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f444806b2c0 0x7f444806d780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4470156600 0x7f4470156a80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447013c830 0x7f4470159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 --2- 172.21.15.141:0/3681173295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4450005b80 0x7f4470158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 >> 172.21.15.141:0/3681173295 conn(0x7f447006a800 msgr2=0x7f447015cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 shutdown_connections 2024-01-16T14:57:59.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.324+0000 7f4476a00640 1 -- 172.21.15.141:0/3681173295 wait complete. 2024-01-16T14:57:59.324 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:57:59.324 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:57:59.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.552+0000 7fc019e00640 1 Processor -- start 2024-01-16T14:57:59.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.552+0000 7fc019e00640 1 -- start start 2024-01-16T14:57:59.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.553+0000 7fc019e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc014070a30 0x7fc01415bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.553+0000 7fc019e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01415f4e0 con 0x7fc0140ccb50 2024-01-16T14:57:59.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.554+0000 7fc019e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc014070ea0 con 0x7fc0140adf20 2024-01-16T14:57:59.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.554+0000 7fc019e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01415c450 con 0x7fc014070a30 2024-01-16T14:57:59.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.554+0000 7fc00bc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc0140ccb50 0x7fc014143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49644/0 (socket says 172.21.15.141:49644) 2024-01-16T14:57:59.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.554+0000 7fc00bc00640 1 -- 172.21.15.141:0/1657714224 learned_addr learned my addr 172.21.15.141:0/1657714224 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:57:59.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.554+0000 7fc019400640 1 --2- 172.21.15.141:0/1657714224 >> v2:172.21.15.188:3300/0 conn(0x7fc014070a30 0x7fc01415bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.554+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402207428 0 0) 0x7fc01415f4e0 con 0x7fc0140ccb50 2024-01-16T14:57:59.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.554+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff4003520 con 0x7fc0140ccb50 2024-01-16T14:57:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1682445575 0 0) 0x7fbff4003520 con 0x7fc0140ccb50 2024-01-16T14:57:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 >> v2:172.21.15.188:3300/0 conn(0x7fc014070a30 msgr2=0x7fc01415bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 --2- 172.21.15.141:0/1657714224 >> v2:172.21.15.188:3300/0 conn(0x7fc014070a30 0x7fc01415bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 >> v1:172.21.15.188:6789/0 conn(0x7fc0140adf20 legacy=0x7fc0140cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014071130 con 0x7fc0140ccb50 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fbffc001670 con 0x7fc0140ccb50 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 >> v1:172.21.15.141:6789/0 conn(0x7fc0140ccb50 legacy=0x7fc014143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbff4003970 0x7fbff4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff40047c0 0x7fbff4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4005180 0x7fbff400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01415c450 con 0x7fbff4003970 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff400c120 con 0x7fbff4005180 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff400c2e0 con 0x7fbff40047c0 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc018a00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbff4003970 0x7fbff4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc018a00640 1 -- 172.21.15.141:0/1657714224 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff40047c0 msgr2=0x7fbff4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.555+0000 7fc018a00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff40047c0 0x7fbff4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.556+0000 7fc018a00640 1 -- 172.21.15.141:0/1657714224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4005180 msgr2=0x7fbff400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.556+0000 7fc018a00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4005180 0x7fbff400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.556+0000 7fc018a00640 1 -- 172.21.15.141:0/1657714224 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff4003520 con 0x7fbff4003970 2024-01-16T14:57:59.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.556+0000 7fc019400640 1 -- 172.21.15.141:0/1657714224 reap_dead start 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.557+0000 7fc018a00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbff4003970 0x7fbff4003d90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc004001750 tx=0x7fbff40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7144f62707ea2e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.557+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0040052d0 con 0x7fbff4003970 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.557+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00402e050 con 0x7fbff4003970 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.557+0000 7fc013e00640 1 -- 172.21.15.141:0/1657714224 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004003830 con 0x7fbff4003970 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.558+0000 7fc019e00640 1 -- 172.21.15.141:0/1657714224 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbff4003970 msgr2=0x7fbff4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.558+0000 7fc019e00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbff4003970 0x7fbff4003d90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc004001750 tx=0x7fbff40057f0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.558+0000 7fc019e00640 1 -- 172.21.15.141:0/1657714224 shutdown_connections 2024-01-16T14:57:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.558+0000 7fc019e00640 1 --2- 172.21.15.141:0/1657714224 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbff4003970 0x7fbff4003d90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.558+0000 7fc019e00640 1 -- 172.21.15.141:0/1657714224 >> 172.21.15.141:0/1657714224 conn(0x7fc01406a7d0 msgr2=0x7fc01406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.558+0000 7fc019e00640 1 -- 172.21.15.141:0/1657714224 shutdown_connections 2024-01-16T14:57:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.558+0000 7fc019e00640 1 -- 172.21.15.141:0/1657714224 wait complete. 2024-01-16T14:57:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.560+0000 7fc019e00640 1 Processor -- start 2024-01-16T14:57:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.560+0000 7fc019e00640 1 -- start start 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff4003970 0x7fc01415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc014140510 0x7fc01413bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc01413abc0 0x7fc0140ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01413a7a0 con 0x7fc01413abc0 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01413c2e0 con 0x7fc014140510 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc014140990 con 0x7fbff4003970 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc01413abc0 0x7fc0140ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc01413abc0 0x7fc0140ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42760/0 (socket says 172.21.15.141:42760) 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.561+0000 7fc019400640 1 -- 172.21.15.141:0/45421308 learned_addr learned my addr 172.21.15.141:0/45421308 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc00bc00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc014140510 0x7fc01413bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc018a00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff4003970 0x7fc01415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc019400640 1 -- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff4003970 msgr2=0x7fc01415a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc019400640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff4003970 0x7fc01415a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc019400640 1 -- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc014140510 msgr2=0x7fc01413bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc019400640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc014140510 0x7fc01413bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc019400640 1 -- 172.21.15.141:0/45421308 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01413c510 con 0x7fc01413abc0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc019400640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc01413abc0 0x7fc0140ac8d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc01415fb10 tx=0x7fc00c00ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc011600640 1 -- 172.21.15.141:0/45421308 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c010040 con 0x7fc01413abc0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc011600640 1 -- 172.21.15.141:0/45421308 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c00bb30 con 0x7fc01413abc0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01413c7f0 con 0x7fc01413abc0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.562+0000 7fc011600640 1 -- 172.21.15.141:0/45421308 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c014400 con 0x7fc01413abc0 2024-01-16T14:57:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.563+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01413a3c0 con 0x7fc01413abc0 2024-01-16T14:57:59.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.564+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01413ca30 con 0x7fc01413abc0 2024-01-16T14:57:59.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.565+0000 7fc011600640 1 -- 172.21.15.141:0/45421308 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc00c014560 con 0x7fc01413abc0 2024-01-16T14:57:59.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.565+0000 7fc011600640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfec06b2c0 0x7fbfec06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.565+0000 7fc011600640 1 -- 172.21.15.141:0/45421308 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fc00c08a240 con 0x7fc01413abc0 2024-01-16T14:57:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.566+0000 7fc018a00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfec06b2c0 0x7fbfec06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.567+0000 7fc018a00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfec06b2c0 0x7fbfec06d780 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc004033000 tx=0x7fc004009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.570+0000 7fc011600640 1 -- 172.21.15.141:0/45421308 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc01413ca30 con 0x7fc01413abc0 2024-01-16T14:57:59.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.762+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc01413a8e0 con 0x7fc01413abc0 2024-01-16T14:57:59.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.763+0000 7fc011600640 1 -- 172.21.15.141:0/45421308 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc01413a8e0 con 0x7fc01413abc0 2024-01-16T14:57:59.745 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfec06b2c0 msgr2=0x7fbfec06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfec06b2c0 0x7fbfec06d780 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc004033000 tx=0x7fc004009030 comp rx=0 tx=0).stop 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc01413abc0 msgr2=0x7fc0140ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc01413abc0 0x7fc0140ac8d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc01415fb10 tx=0x7fc00c00ef50 comp rx=0 tx=0).stop 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 shutdown_connections 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfec06b2c0 0x7fbfec06d780 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc01413abc0 0x7fc0140ac8d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc014140510 0x7fc01413bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 --2- 172.21.15.141:0/45421308 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbff4003970 0x7fc01415a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.765+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 >> 172.21.15.141:0/45421308 conn(0x7fc01406a7d0 msgr2=0x7fc014143030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:59.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.766+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 shutdown_connections 2024-01-16T14:57:59.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.766+0000 7fc019e00640 1 -- 172.21.15.141:0/45421308 wait complete. 2024-01-16T14:57:59.767 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:57:59.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.995+0000 7f400ae00640 1 Processor -- start 2024-01-16T14:57:59.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.995+0000 7f400ae00640 1 -- start start 2024-01-16T14:57:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.996+0000 7f400ae00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f40040ccb50 0x7f4004070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.996+0000 7f400ae00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400415f4e0 con 0x7f400415bf10 2024-01-16T14:57:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.996+0000 7f400ae00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004071130 con 0x7f40040adf20 2024-01-16T14:57:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f400ae00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400415f970 con 0x7f40040ccb50 2024-01-16T14:57:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f4009000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f40040ccb50 0x7f4004070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f4009000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f40040ccb50 0x7f4004070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42770/0 (socket says 172.21.15.141:42770) 2024-01-16T14:57:59.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f4009000640 1 -- 172.21.15.141:0/168174458 learned_addr learned my addr 172.21.15.141:0/168174458 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f400a400640 1 --1- 172.21.15.141:0/168174458 >> v1:172.21.15.141:6789/0 conn(0x7f400415bf10 0x7f4004143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49650/0 (socket says 172.21.15.141:49650) 2024-01-16T14:57:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f4009000640 1 -- 172.21.15.141:0/168174458 >> v1:172.21.15.188:6789/0 conn(0x7f40040adf20 legacy=0x7f40040cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:57:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f4009000640 1 -- 172.21.15.141:0/168174458 >> v1:172.21.15.141:6789/0 conn(0x7f400415bf10 legacy=0x7f4004143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.997+0000 7f4009000640 1 -- 172.21.15.141:0/168174458 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400415ebb0 con 0x7f40040ccb50 2024-01-16T14:57:59.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.998+0000 7f4009000640 1 --2- 172.21.15.141:0/168174458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040ccb50 0x7f4004070a30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009870 tx=0x7f3fec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d87ed3837e1f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f4003e00640 1 -- 172.21.15.141:0/168174458 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec031870 con 0x7f40040ccb50 2024-01-16T14:57:59.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f4003e00640 1 -- 172.21.15.141:0/168174458 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fec0319d0 con 0x7f40040ccb50 2024-01-16T14:57:59.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f4003e00640 1 -- 172.21.15.141:0/168174458 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec031d60 con 0x7f40040ccb50 2024-01-16T14:57:59.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f400ae00640 1 -- 172.21.15.141:0/168174458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040ccb50 msgr2=0x7f4004070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f400ae00640 1 --2- 172.21.15.141:0/168174458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040ccb50 0x7f4004070a30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009870 tx=0x7f3fec02b910 comp rx=0 tx=0).stop 2024-01-16T14:57:59.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f400ae00640 1 -- 172.21.15.141:0/168174458 shutdown_connections 2024-01-16T14:57:59.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f400ae00640 1 --2- 172.21.15.141:0/168174458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040ccb50 0x7f4004070a30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:57:59.999+0000 7f400ae00640 1 -- 172.21.15.141:0/168174458 >> 172.21.15.141:0/168174458 conn(0x7f400406a7d0 msgr2=0x7f400406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:57:59.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.000+0000 7f400ae00640 1 -- 172.21.15.141:0/168174458 shutdown_connections 2024-01-16T14:57:59.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.000+0000 7f400ae00640 1 -- 172.21.15.141:0/168174458 wait complete. 2024-01-16T14:57:59.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.001+0000 7f400ae00640 1 Processor -- start 2024-01-16T14:57:59.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.001+0000 7f400ae00640 1 -- start start 2024-01-16T14:57:59.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.002+0000 7f400ae00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040adf20 0x7f4004156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.002+0000 7f400ae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40040ccb50 0x7f4004159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.002+0000 7f400ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f400415bf10 0x7f4004154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.002+0000 7f400ae00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4004155e20 con 0x7f40040adf20 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.002+0000 7f400ae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4004154f80 con 0x7f40040ccb50 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.002+0000 7f400ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4004155760 con 0x7f400415bf10 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.002+0000 7f4009a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040adf20 0x7f4004156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040adf20 0x7f4004156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42778/0 (socket says 172.21.15.141:42778) 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 -- 172.21.15.141:0/2026345339 learned_addr learned my addr 172.21.15.141:0/2026345339 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f400a400640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f400415bf10 0x7f4004154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009000640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40040ccb50 0x7f4004159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 -- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f400415bf10 msgr2=0x7f4004154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f400415bf10 0x7f4004154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 -- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40040ccb50 msgr2=0x7f4004159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:57:59.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40040ccb50 0x7f4004159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:57:59.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 -- 172.21.15.141:0/2026345339 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec0094d0 con 0x7f40040adf20 2024-01-16T14:57:59.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009000640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40040ccb50 0x7f4004159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:59.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f400a400640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f400415bf10 0x7f4004154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:57:59.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.003+0000 7f4009a00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040adf20 0x7f4004156240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400b3f0 tx=0x7f3ff400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.004+0000 7f4001600640 1 -- 172.21.15.141:0/2026345339 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4011830 con 0x7f40040adf20 2024-01-16T14:57:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.004+0000 7f4001600640 1 -- 172.21.15.141:0/2026345339 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff4011990 con 0x7f40040adf20 2024-01-16T14:57:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.004+0000 7f4001600640 1 -- 172.21.15.141:0/2026345339 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4011d40 con 0x7f40040adf20 2024-01-16T14:57:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.004+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4004155a40 con 0x7f40040adf20 2024-01-16T14:57:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.004+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4004155260 con 0x7f40040adf20 2024-01-16T14:57:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.005+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004155f60 con 0x7f40040adf20 2024-01-16T14:57:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.006+0000 7f4001600640 1 -- 172.21.15.141:0/2026345339 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3ff401a400 con 0x7f40040adf20 2024-01-16T14:57:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.007+0000 7f4001600640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3fdc06b2c0 0x7f3fdc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:57:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.007+0000 7f4001600640 1 -- 172.21.15.141:0/2026345339 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f3ff4089a80 con 0x7f40040adf20 2024-01-16T14:57:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.007+0000 7f4009000640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3fdc06b2c0 0x7f3fdc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:57:59.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.008+0000 7f4009000640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3fdc06b2c0 0x7f3fdc06d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3fec036000 tx=0x7f3fec03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:57:59.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.011+0000 7f4001600640 1 -- 172.21.15.141:0/2026345339 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4004155f60 con 0x7f40040adf20 2024-01-16T14:58:00.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.203+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4004155f60 con 0x7f40040adf20 2024-01-16T14:58:00.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.204+0000 7f4001600640 1 -- 172.21.15.141:0/2026345339 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ff4054860 con 0x7f40040adf20 2024-01-16T14:58:00.187 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:58:00.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3fdc06b2c0 msgr2=0x7f3fdc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3fdc06b2c0 0x7f3fdc06d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3fec036000 tx=0x7f3fec03c040 comp rx=0 tx=0).stop 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040adf20 msgr2=0x7f4004156240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040adf20 0x7f4004156240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400b3f0 tx=0x7f3ff400b8b0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 shutdown_connections 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3fdc06b2c0 0x7f3fdc06d780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f400415bf10 0x7f4004154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40040ccb50 0x7f4004159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 --2- 172.21.15.141:0/2026345339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40040adf20 0x7f4004156240 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 >> 172.21.15.141:0/2026345339 conn(0x7f400406a7d0 msgr2=0x7f400415b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 shutdown_connections 2024-01-16T14:58:00.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.207+0000 7f400ae00640 1 -- 172.21.15.141:0/2026345339 wait complete. 2024-01-16T14:58:00.208 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:58:00.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.433+0000 7f70f1000640 1 Processor -- start 2024-01-16T14:58:00.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.433+0000 7f70f1000640 1 -- start start 2024-01-16T14:58:00.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.435+0000 7f70f1000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f70ec15d4b0 0x7f70ec0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.435+0000 7f70f1000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0b2730 con 0x7f70ec0b41e0 2024-01-16T14:58:00.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.435+0000 7f70f1000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0d1ec0 con 0x7f70ec0cf9a0 2024-01-16T14:58:00.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.435+0000 7f70f1000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec0b29d0 con 0x7f70ec15d4b0 2024-01-16T14:58:00.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.435+0000 7f70ebe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f70ec0b41e0 0x7f70ec0ce500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49664/0 (socket says 172.21.15.141:49664) 2024-01-16T14:58:00.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.435+0000 7f70ebe00640 1 -- 172.21.15.141:0/660851200 learned_addr learned my addr 172.21.15.141:0/660851200 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:00.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eb400640 1 --2- 172.21.15.141:0/660851200 >> v2:172.21.15.188:3300/0 conn(0x7f70ec15d4b0 0x7f70ec0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438589390 0 0) 0x7f70ec0b2730 con 0x7f70ec0b41e0 2024-01-16T14:58:00.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70cc003520 con 0x7f70ec0b41e0 2024-01-16T14:58:00.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823703121 0 0) 0x7f70cc003520 con 0x7f70ec0b41e0 2024-01-16T14:58:00.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 >> v2:172.21.15.188:3300/0 conn(0x7f70ec15d4b0 msgr2=0x7f70ec0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:00.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 --2- 172.21.15.141:0/660851200 >> v2:172.21.15.188:3300/0 conn(0x7f70ec15d4b0 0x7f70ec0d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 >> v1:172.21.15.188:6789/0 conn(0x7f70ec0cf9a0 legacy=0x7f70ec0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec0b2cb0 con 0x7f70ec0b41e0 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f70e40019f0 con 0x7f70ec0b41e0 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 >> v1:172.21.15.141:6789/0 conn(0x7f70ec0b41e0 legacy=0x7f70ec0ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc003970 0x7f70cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70cc0047c0 0x7f70cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70cc005180 0x7f70cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0b29d0 con 0x7f70cc0047c0 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc00c020 con 0x7f70cc003970 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.436+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc00c1e0 con 0x7f70cc005180 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70e2800640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70cc0047c0 0x7f70cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70eb400640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc003970 0x7f70cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70e2800640 1 -- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70cc005180 msgr2=0x7f70cc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70ebe00640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70cc005180 0x7f70cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70e2800640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70cc005180 0x7f70cc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70e2800640 1 -- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc003970 msgr2=0x7f70cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:00.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70e2800640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc003970 0x7f70cc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70e2800640 1 -- 172.21.15.141:0/660851200 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70cc003520 con 0x7f70cc0047c0 2024-01-16T14:58:00.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70eb400640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc003970 0x7f70cc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:00.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70ebe00640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70cc005180 0x7f70cc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:00.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.437+0000 7f70ebe00640 1 -- 172.21.15.141:0/660851200 reap_dead start 2024-01-16T14:58:00.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.438+0000 7f70e2800640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70cc0047c0 0x7f70cc004c40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f70d4001750 tx=0x7f70cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9adb40ccf91797 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:00.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.438+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d40052d0 con 0x7f70cc0047c0 2024-01-16T14:58:00.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.438+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d402e050 con 0x7f70cc0047c0 2024-01-16T14:58:00.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.438+0000 7f70eaa00640 1 -- 172.21.15.141:0/660851200 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4003830 con 0x7f70cc0047c0 2024-01-16T14:58:00.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.439+0000 7f70f1000640 1 -- 172.21.15.141:0/660851200 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70cc0047c0 msgr2=0x7f70cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.439+0000 7f70f1000640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70cc0047c0 0x7f70cc004c40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f70d4001750 tx=0x7f70cc0057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.439+0000 7f70f1000640 1 -- 172.21.15.141:0/660851200 shutdown_connections 2024-01-16T14:58:00.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.439+0000 7f70f1000640 1 --2- 172.21.15.141:0/660851200 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70cc0047c0 0x7f70cc004c40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.439+0000 7f70f1000640 1 -- 172.21.15.141:0/660851200 >> 172.21.15.141:0/660851200 conn(0x7f70ec0d3000 msgr2=0x7f70ec0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:00.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.439+0000 7f70f1000640 1 -- 172.21.15.141:0/660851200 shutdown_connections 2024-01-16T14:58:00.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.439+0000 7f70f1000640 1 -- 172.21.15.141:0/660851200 wait complete. 2024-01-16T14:58:00.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.441+0000 7f70f1000640 1 Processor -- start 2024-01-16T14:58:00.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.441+0000 7f70f1000640 1 -- start start 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70f1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0047c0 0x7f70ec151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70f1000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70ec1520e0 0x7f70ec0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70f1000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70ec0cc970 0x7f70ec15dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70f1000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0ca830 con 0x7f70ec1520e0 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70f1000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0ccdf0 con 0x7f70cc0047c0 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70f1000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec15ed70 con 0x7f70ec0cc970 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70e2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70ec1520e0 0x7f70ec0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70e2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70ec1520e0 0x7f70ec0cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42798/0 (socket says 172.21.15.141:42798) 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.442+0000 7f70e2800640 1 -- 172.21.15.141:0/2932050504 learned_addr learned my addr 172.21.15.141:0/2932050504 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70ebe00640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70ec0cc970 0x7f70ec15dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70eb400640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0047c0 0x7f70ec151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e2800640 1 -- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70ec0cc970 msgr2=0x7f70ec15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e2800640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70ec0cc970 0x7f70ec15dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e2800640 1 -- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0047c0 msgr2=0x7f70ec151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e2800640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0047c0 0x7f70ec151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e2800640 1 -- 172.21.15.141:0/2932050504 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec15da10 con 0x7f70ec1520e0 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70eb400640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0047c0 0x7f70ec151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e2800640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70ec1520e0 0x7f70ec0cc430 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f70d402bc00 tx=0x7f70d4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e3e00640 1 -- 172.21.15.141:0/2932050504 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d40076c0 con 0x7f70ec1520e0 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e3e00640 1 -- 172.21.15.141:0/2932050504 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4007a30 con 0x7f70ec1520e0 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec1f2cd0 con 0x7f70ec1520e0 2024-01-16T14:58:00.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.443+0000 7f70e3e00640 1 -- 172.21.15.141:0/2932050504 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4006be0 con 0x7f70ec1520e0 2024-01-16T14:58:00.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.444+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec0ca170 con 0x7f70ec1520e0 2024-01-16T14:58:00.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.445+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ec0ca970 con 0x7f70ec1520e0 2024-01-16T14:58:00.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.446+0000 7f70e3e00640 1 -- 172.21.15.141:0/2932050504 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f70d4006d40 con 0x7f70ec1520e0 2024-01-16T14:58:00.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.446+0000 7f70e3e00640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70c406b2c0 0x7f70c406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.447+0000 7f70e3e00640 1 -- 172.21.15.141:0/2932050504 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f70d40ab8d0 con 0x7f70ec1520e0 2024-01-16T14:58:00.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.447+0000 7f70eb400640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70c406b2c0 0x7f70c406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.448+0000 7f70eb400640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70c406b2c0 0x7f70c406d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009b20 tx=0x7f70dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:00.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.451+0000 7f70e3e00640 1 -- 172.21.15.141:0/2932050504 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70ec0ca970 con 0x7f70ec1520e0 2024-01-16T14:58:00.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.644+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f70ec0ca970 con 0x7f70ec1520e0 2024-01-16T14:58:00.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.645+0000 7f70e3e00640 1 -- 172.21.15.141:0/2932050504 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f70d4041020 con 0x7f70ec1520e0 2024-01-16T14:58:00.627 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:58:00.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70c406b2c0 msgr2=0x7f70c406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70c406b2c0 0x7f70c406d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009b20 tx=0x7f70dc008040 comp rx=0 tx=0).stop 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70ec1520e0 msgr2=0x7f70ec0cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70ec1520e0 0x7f70ec0cc430 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f70d402bc00 tx=0x7f70d4001120 comp rx=0 tx=0).stop 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 shutdown_connections 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f70c406b2c0 0x7f70c406d780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70ec0cc970 0x7f70ec15dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70ec1520e0 0x7f70ec0cc430 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 --2- 172.21.15.141:0/2932050504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0047c0 0x7f70ec151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 >> 172.21.15.141:0/2932050504 conn(0x7f70ec0d3000 msgr2=0x7f70ec0b7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 shutdown_connections 2024-01-16T14:58:00.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.648+0000 7f70f1000640 1 -- 172.21.15.141:0/2932050504 wait complete. 2024-01-16T14:58:00.647 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:58:00.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 Processor -- start 2024-01-16T14:58:00.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 -- start start 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1ae8143ae0 0x7f1ae80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae80cf7d0 0x7f1ae815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1ae815e1e0 0x7f1ae815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80cfc30 con 0x7f1ae815e1e0 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80b1ee0 con 0x7f1ae8143ae0 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.872+0000 7f1aed000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80b2780 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1ae815e1e0 0x7f1ae815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1ae815e1e0 0x7f1ae815a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42814/0 (socket says 172.21.15.141:42814) 2024-01-16T14:58:00.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 -- 172.21.15.141:0/3545984933 learned_addr learned my addr 172.21.15.141:0/3545984933 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:00.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae5e00640 1 --2- 172.21.15.141:0/3545984933 >> v2:172.21.15.188:3300/0 conn(0x7f1ae8143ae0 0x7f1ae80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 -- 172.21.15.141:0/3545984933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae80cf7d0 msgr2=0x7f1ae815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:00.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 --2- 172.21.15.141:0/3545984933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae80cf7d0 0x7f1ae815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 -- 172.21.15.141:0/3545984933 >> v2:172.21.15.188:3300/0 conn(0x7f1ae8143ae0 msgr2=0x7f1ae80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:00.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 --2- 172.21.15.141:0/3545984933 >> v2:172.21.15.188:3300/0 conn(0x7f1ae8143ae0 0x7f1ae80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.873+0000 7f1ae6800640 1 -- 172.21.15.141:0/3545984933 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae815e990 con 0x7f1ae815e1e0 2024-01-16T14:58:00.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.874+0000 7f1ae6800640 1 --2- 172.21.15.141:0/3545984933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae815e1e0 0x7f1ae815a670 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80cfea0 tx=0x7f1ae002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c5808559dce928 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:00.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1ae5400640 1 -- 172.21.15.141:0/3545984933 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae002ca80 con 0x7f1ae815e1e0 2024-01-16T14:58:00.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1ae5400640 1 -- 172.21.15.141:0/3545984933 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae002cbe0 con 0x7f1ae815e1e0 2024-01-16T14:58:00.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1ae5400640 1 -- 172.21.15.141:0/3545984933 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae00343f0 con 0x7f1ae815e1e0 2024-01-16T14:58:00.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 -- 172.21.15.141:0/3545984933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae815e1e0 msgr2=0x7f1ae815a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 --2- 172.21.15.141:0/3545984933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae815e1e0 0x7f1ae815a670 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80cfea0 tx=0x7f1ae002bee0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 -- 172.21.15.141:0/3545984933 shutdown_connections 2024-01-16T14:58:00.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 --2- 172.21.15.141:0/3545984933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae815e1e0 0x7f1ae815a670 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 --2- 172.21.15.141:0/3545984933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae80cf7d0 0x7f1ae815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 --2- 172.21.15.141:0/3545984933 >> v2:172.21.15.188:3300/0 conn(0x7f1ae8143ae0 0x7f1ae80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 -- 172.21.15.141:0/3545984933 >> 172.21.15.141:0/3545984933 conn(0x7f1ae806a800 msgr2=0x7f1ae806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:00.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 -- 172.21.15.141:0/3545984933 shutdown_connections 2024-01-16T14:58:00.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.875+0000 7f1aed000640 1 -- 172.21.15.141:0/3545984933 wait complete. 2024-01-16T14:58:00.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.877+0000 7f1aed000640 1 Processor -- start 2024-01-16T14:58:00.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.877+0000 7f1aed000640 1 -- start start 2024-01-16T14:58:00.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1aed000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae80cf7d0 0x7f1ae80ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1aed000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae8143ae0 0x7f1ae81f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1aed000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ae815e1e0 0x7f1ae80afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1aed000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae81f08b0 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1aed000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8159a50 con 0x7f1ae815e1e0 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1aed000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae815bc70 con 0x7f1ae8143ae0 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1ae5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae80cf7d0 0x7f1ae80ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1ae5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae80cf7d0 0x7f1ae80ae410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42822/0 (socket says 172.21.15.141:42822) 2024-01-16T14:58:00.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.878+0000 7f1ae5e00640 1 -- 172.21.15.141:0/3669004075 learned_addr learned my addr 172.21.15.141:0/3669004075 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:00.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1adfe00640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae8143ae0 0x7f1ae81f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae6800640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ae815e1e0 0x7f1ae80afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae5e00640 1 -- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae8143ae0 msgr2=0x7f1ae81f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae5e00640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae8143ae0 0x7f1ae81f0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae5e00640 1 -- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ae815e1e0 msgr2=0x7f1ae80afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae5e00640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ae815e1e0 0x7f1ae80afa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae5e00640 1 -- 172.21.15.141:0/3669004075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae0009b30 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae6800640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ae815e1e0 0x7f1ae80afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ae5e00640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae80cf7d0 0x7f1ae80ae410 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c820 tx=0x7f1ad800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.879+0000 7f1ade000640 1 -- 172.21.15.141:0/3669004075 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8004220 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.880+0000 7f1ade000640 1 -- 172.21.15.141:0/3669004075 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad80043a0 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.880+0000 7f1ade000640 1 -- 172.21.15.141:0/3669004075 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8007a70 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.880+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae8159120 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.880+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae81596e0 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.881+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae80675d0 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.882+0000 7f1ade000640 1 -- 172.21.15.141:0/3669004075 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1ad8007bd0 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.883+0000 7f1ade000640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1ac006b2c0 0x7f1ac006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:00.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.883+0000 7f1ade000640 1 -- 172.21.15.141:0/3669004075 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f1ad80899e0 con 0x7f1ae80cf7d0 2024-01-16T14:58:00.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.883+0000 7f1adfe00640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1ac006b2c0 0x7f1ac006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:00.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.884+0000 7f1adfe00640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1ac006b2c0 0x7f1ac006d780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80b3b40 tx=0x7f1ad0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:00.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:00.887+0000 7f1ade000640 1 -- 172.21.15.141:0/3669004075 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ad80547c0 con 0x7f1ae80cf7d0 2024-01-16T14:58:01.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.090+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1ae8072230 con 0x7f1ae80cf7d0 2024-01-16T14:58:01.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.091+0000 7f1ade000640 1 -- 172.21.15.141:0/3669004075 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1ad801e020 con 0x7f1ae80cf7d0 2024-01-16T14:58:01.073 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1ac006b2c0 msgr2=0x7f1ac006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1ac006b2c0 0x7f1ac006d780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80b3b40 tx=0x7f1ad0008040 comp rx=0 tx=0).stop 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae80cf7d0 msgr2=0x7f1ae80ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae80cf7d0 0x7f1ae80ae410 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c820 tx=0x7f1ad800cce0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 shutdown_connections 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1ac006b2c0 0x7f1ac006d780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ae815e1e0 0x7f1ae80afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ae8143ae0 0x7f1ae81f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 --2- 172.21.15.141:0/3669004075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ae80cf7d0 0x7f1ae80ae410 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 >> 172.21.15.141:0/3669004075 conn(0x7f1ae806a800 msgr2=0x7f1ae815a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 shutdown_connections 2024-01-16T14:58:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.093+0000 7f1aed000640 1 -- 172.21.15.141:0/3669004075 wait complete. 2024-01-16T14:58:01.094 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:58:01.094 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:01.139 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:01.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.414+0000 7fd5d2e00640 1 Processor -- start 2024-01-16T14:58:01.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.414+0000 7fd5d2e00640 1 -- start start 2024-01-16T14:58:01.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5d2e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5cc13e360 0x7fd5cc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5d2e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5d2e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc15c5f0 con 0x7fd5cc13df80 2024-01-16T14:58:01.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5d2e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5cc06c470 con 0x7fd5cc13e360 2024-01-16T14:58:01.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5d2e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc06c750 con 0x7fd5cc15f9c0 2024-01-16T14:58:01.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5cbe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd5cc13df80 0x7fd5cc15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49674/0 (socket says 172.21.15.141:49674) 2024-01-16T14:58:01.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5cbe00640 1 -- 172.21.15.141:0/1320517291 learned_addr learned my addr 172.21.15.141:0/1320517291 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:01.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.416+0000 7fd5cb400640 1 --2- 172.21.15.141:0/1320517291 >> v2:172.21.15.141:3300/0 conn(0x7fd5cc13e360 0x7fd5cc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.417+0000 7fd5d0e00640 1 --2- 172.21.15.141:0/1320517291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.417+0000 7fd5cb400640 1 -- 172.21.15.141:0/1320517291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 msgr2=0x7fd5cc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:01.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.417+0000 7fd5cb400640 1 --2- 172.21.15.141:0/1320517291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.417+0000 7fd5cb400640 1 -- 172.21.15.141:0/1320517291 >> v1:172.21.15.141:6789/0 conn(0x7fd5cc13df80 legacy=0x7fd5cc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.417+0000 7fd5cb400640 1 -- 172.21.15.141:0/1320517291 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc06ca30 con 0x7fd5cc13e360 2024-01-16T14:58:01.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.417+0000 7fd5caa00640 1 -- 172.21.15.141:0/1320517291 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785197967 0 0) 0x7fd5cc15c5f0 con 0x7fd5cc13df80 2024-01-16T14:58:01.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.418+0000 7fd5cb400640 1 --2- 172.21.15.141:0/1320517291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc0cf290 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009d90 tx=0x7fd5b402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5669b52e92468a2d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:01.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.418+0000 7fd5caa00640 1 -- 172.21.15.141:0/1320517291 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b402cc50 con 0x7fd5cc13e360 2024-01-16T14:58:01.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.418+0000 7fd5caa00640 1 -- 172.21.15.141:0/1320517291 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b402cdb0 con 0x7fd5cc13e360 2024-01-16T14:58:01.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.418+0000 7fd5caa00640 1 -- 172.21.15.141:0/1320517291 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4034660 con 0x7fd5cc13e360 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 -- 172.21.15.141:0/1320517291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 msgr2=0x7fd5cc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/1320517291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc0cf290 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009d90 tx=0x7fd5b402bea0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 -- 172.21.15.141:0/1320517291 shutdown_connections 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/1320517291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/1320517291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc0cf290 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 -- 172.21.15.141:0/1320517291 >> 172.21.15.141:0/1320517291 conn(0x7fd5cc06a800 msgr2=0x7fd5cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 -- 172.21.15.141:0/1320517291 shutdown_connections 2024-01-16T14:58:01.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.419+0000 7fd5d2e00640 1 -- 172.21.15.141:0/1320517291 wait complete. 2024-01-16T14:58:01.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.421+0000 7fd5d2e00640 1 Processor -- start 2024-01-16T14:58:01.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.421+0000 7fd5d2e00640 1 -- start start 2024-01-16T14:58:01.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5d2e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5cc13df80 0x7fd5cc13d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5d2e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5d2e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5d2e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc13dda0 con 0x7fd5cc13e360 2024-01-16T14:58:01.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5d2e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc1502b0 con 0x7fd5cc13df80 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5d2e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc150590 con 0x7fd5cc15f9c0 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5cb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5cb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc14f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42838/0 (socket says 172.21.15.141:42838) 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.422+0000 7fd5cb400640 1 -- 172.21.15.141:0/708067346 learned_addr learned my addr 172.21.15.141:0/708067346 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cbe00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5cc13df80 0x7fd5cc13d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5d0e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cb400640 1 -- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 msgr2=0x7fd5cc0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cb400640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cb400640 1 -- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5cc13df80 msgr2=0x7fd5cc13d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cb400640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5cc13df80 0x7fd5cc13d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cb400640 1 -- 172.21.15.141:0/708067346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4009a70 con 0x7fd5cc13e360 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cbe00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5cc13df80 0x7fd5cc13d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5d0e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:01.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5cb400640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc14f8d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009d90 tx=0x7fd5b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:01.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5c3e00640 1 -- 172.21.15.141:0/708067346 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4040650 con 0x7fd5cc13e360 2024-01-16T14:58:01.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5c3e00640 1 -- 172.21.15.141:0/708067346 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b40407b0 con 0x7fd5cc13e360 2024-01-16T14:58:01.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.423+0000 7fd5c3e00640 1 -- 172.21.15.141:0/708067346 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4040b60 con 0x7fd5cc13e360 2024-01-16T14:58:01.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.424+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc150870 con 0x7fd5cc13e360 2024-01-16T14:58:01.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.424+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc150e30 con 0x7fd5cc13e360 2024-01-16T14:58:01.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.425+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc0675d0 con 0x7fd5cc13e360 2024-01-16T14:58:01.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.426+0000 7fd5c3e00640 1 -- 172.21.15.141:0/708067346 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd5b4040cc0 con 0x7fd5cc13e360 2024-01-16T14:58:01.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.427+0000 7fd5c3e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd5a406b2c0 0x7fd5a406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.427+0000 7fd5c3e00640 1 -- 172.21.15.141:0/708067346 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fd5b40acaf0 con 0x7fd5cc13e360 2024-01-16T14:58:01.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.427+0000 7fd5cbe00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd5a406b2c0 0x7fd5a406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.428+0000 7fd5cbe00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd5a406b2c0 0x7fd5a406d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc000f60 tx=0x7fd5bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:01.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.431+0000 7fd5c3e00640 1 -- 172.21.15.141:0/708067346 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5b4077880 con 0x7fd5cc13e360 2024-01-16T14:58:01.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.620+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5cc06df00 con 0x7fd5cc13e360 2024-01-16T14:58:01.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.622+0000 7fd5c3e00640 1 -- 172.21.15.141:0/708067346 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7fd5b407aea0 con 0x7fd5cc13e360 2024-01-16T14:58:01.605 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:01.605 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd5a406b2c0 msgr2=0x7fd5a406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd5a406b2c0 0x7fd5a406d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc000f60 tx=0x7fd5bc009040 comp rx=0 tx=0).stop 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 msgr2=0x7fd5cc14f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc14f8d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009d90 tx=0x7fd5b4004a40 comp rx=0 tx=0).stop 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 shutdown_connections 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd5a406b2c0 0x7fd5a406d780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5cc15f9c0 0x7fd5cc0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5cc13e360 0x7fd5cc14f8d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 --2- 172.21.15.141:0/708067346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5cc13df80 0x7fd5cc13d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 >> 172.21.15.141:0/708067346 conn(0x7fd5cc06a800 msgr2=0x7fd5cc153930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:01.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 shutdown_connections 2024-01-16T14:58:01.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.625+0000 7fd5d2e00640 1 -- 172.21.15.141:0/708067346 wait complete. 2024-01-16T14:58:01.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:58:01.628 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:58:01.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.853+0000 7f3a8e000640 1 Processor -- start 2024-01-16T14:58:01.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.853+0000 7f3a8e000640 1 -- start start 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.855+0000 7f3a8e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.855+0000 7f3a8e000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3a880cf290 0x7f3a880cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.855+0000 7f3a8e000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a88070c40 con 0x7f3a880b3ae0 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.855+0000 7f3a8e000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a88071450 con 0x7f3a880cf290 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.855+0000 7f3a8e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a880cfc60 con 0x7f3a880ae5a0 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33354/0 (socket says 172.21.15.141:33354) 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87400640 1 -- 172.21.15.141:0/3372923185 learned_addr learned my addr 172.21.15.141:0/3372923185 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:01.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87e00640 1 --2- 172.21.15.141:0/3372923185 >> v2:172.21.15.188:3300/0 conn(0x7f3a880cf290 0x7f3a880cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87e00640 1 -- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 msgr2=0x7f3a8815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:01.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87e00640 1 --2- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87e00640 1 -- 172.21.15.141:0/3372923185 >> v1:172.21.15.188:6789/0 conn(0x7f3a880b3ae0 legacy=0x7f3a8806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87e00640 1 -- 172.21.15.141:0/3372923185 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8806c470 con 0x7f3a880cf290 2024-01-16T14:58:01.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.856+0000 7f3a87400640 1 --2- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:01.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.858+0000 7f3a87e00640 1 --2- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880cf290 0x7f3a880cf720 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009d90 tx=0x7f3a8002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daae5aa8eece20a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.858+0000 7f3a86a00640 1 -- 172.21.15.141:0/3372923185 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8002cce0 con 0x7f3a880cf290 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.858+0000 7f3a86a00640 1 -- 172.21.15.141:0/3372923185 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8002ce40 con 0x7f3a880cf290 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.858+0000 7f3a86a00640 1 -- 172.21.15.141:0/3372923185 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80034660 con 0x7f3a880cf290 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 -- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880cf290 msgr2=0x7f3a880cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 --2- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880cf290 0x7f3a880cf720 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009d90 tx=0x7f3a8002bf30 comp rx=0 tx=0).stop 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 -- 172.21.15.141:0/3372923185 shutdown_connections 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 --2- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880cf290 0x7f3a880cf720 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 --2- 172.21.15.141:0/3372923185 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 -- 172.21.15.141:0/3372923185 >> 172.21.15.141:0/3372923185 conn(0x7f3a8806a800 msgr2=0x7f3a8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:01.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 -- 172.21.15.141:0/3372923185 shutdown_connections 2024-01-16T14:58:01.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.859+0000 7f3a8e000640 1 -- 172.21.15.141:0/3372923185 wait complete. 2024-01-16T14:58:01.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.861+0000 7f3a8e000640 1 Processor -- start 2024-01-16T14:58:01.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.861+0000 7f3a8e000640 1 -- start start 2024-01-16T14:58:01.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880b3ae0 0x7f3a88149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a880cf290 0x7f3a8815fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a88157240 con 0x7f3a880cf290 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a88149ed0 con 0x7f3a880b3ae0 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8814f3b0 con 0x7f3a880ae5a0 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a880cf290 0x7f3a8815fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a880cf290 0x7f3a8815fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42852/0 (socket says 172.21.15.141:42852) 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 -- 172.21.15.141:0/1351549161 learned_addr learned my addr 172.21.15.141:0/1351549161 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87400640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a7e800640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880b3ae0 0x7f3a88149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 -- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 msgr2=0x7f3a8815cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 -- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880b3ae0 msgr2=0x7f3a88149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880b3ae0 0x7f3a88149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 -- 172.21.15.141:0/1351549161 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a780094d0 con 0x7f3a880cf290 2024-01-16T14:58:01.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a7e800640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880b3ae0 0x7f3a88149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a87e00640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a880cf290 0x7f3a8815fba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009d90 tx=0x7f3a80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a7fe00640 1 -- 172.21.15.141:0/1351549161 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8003e670 con 0x7f3a880cf290 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a7fe00640 1 -- 172.21.15.141:0/1351549161 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8003e7f0 con 0x7f3a880cf290 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a7fe00640 1 -- 172.21.15.141:0/1351549161 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8003eba0 con 0x7f3a880cf290 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a80009a70 con 0x7f3a880cf290 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.863+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a88156ed0 con 0x7f3a880cf290 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.864+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8814f8d0 con 0x7f3a880cf290 2024-01-16T14:58:01.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.865+0000 7f3a7fe00640 1 -- 172.21.15.141:0/1351549161 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3a8003b040 con 0x7f3a880cf290 2024-01-16T14:58:01.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.866+0000 7f3a7fe00640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a6006b2c0 0x7f3a6006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:01.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.866+0000 7f3a7fe00640 1 -- 172.21.15.141:0/1351549161 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f3a800ac630 con 0x7f3a880cf290 2024-01-16T14:58:01.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.867+0000 7f3a87400640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a6006b2c0 0x7f3a6006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:01.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.868+0000 7f3a87400640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a6006b2c0 0x7f3a6006d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009cf0 tx=0x7f3a78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:01.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:01.870+0000 7f3a7fe00640 1 -- 172.21.15.141:0/1351549161 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a8814f8d0 con 0x7f3a880cf290 2024-01-16T14:58:02.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.062+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a88156d30 con 0x7f3a880cf290 2024-01-16T14:58:02.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.063+0000 7f3a7fe00640 1 -- 172.21.15.141:0/1351549161 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a88156d30 con 0x7f3a880cf290 2024-01-16T14:58:02.045 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:58:02.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a6006b2c0 msgr2=0x7f3a6006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a6006b2c0 0x7f3a6006d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009cf0 tx=0x7f3a78008040 comp rx=0 tx=0).stop 2024-01-16T14:58:02.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a880cf290 msgr2=0x7f3a8815fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a880cf290 0x7f3a8815fba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009d90 tx=0x7f3a80004a40 comp rx=0 tx=0).stop 2024-01-16T14:58:02.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 shutdown_connections 2024-01-16T14:58:02.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a6006b2c0 0x7f3a6006d780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a880cf290 0x7f3a8815fba0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a880b3ae0 0x7f3a88149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 --2- 172.21.15.141:0/1351549161 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a880ae5a0 0x7f3a8815cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 >> 172.21.15.141:0/1351549161 conn(0x7f3a8806a800 msgr2=0x7f3a8815a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:02.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 shutdown_connections 2024-01-16T14:58:02.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.065+0000 7f3a8e000640 1 -- 172.21.15.141:0/1351549161 wait complete. 2024-01-16T14:58:02.067 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:58:02.210 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:02.217 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:02.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.283+0000 7f673b200640 1 Processor -- start 2024-01-16T14:58:02.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.283+0000 7f673b200640 1 -- start start 2024-01-16T14:58:02.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.284+0000 7f673b200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f673413e360 0x7f67340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.284+0000 7f673b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.284+0000 7f673b200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673415c5f0 con 0x7f673413df80 2024-01-16T14:58:02.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.284+0000 7f673b200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673406c470 con 0x7f673413e360 2024-01-16T14:58:02.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.284+0000 7f673b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673406c750 con 0x7f673415f9c0 2024-01-16T14:58:02.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f673413e360 0x7f67340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6739200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f673413e360 0x7f67340cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:44846/0 (socket says 172.21.15.141:44846) 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733400640 1 -- 172.21.15.141:0/774192491 learned_addr learned my addr 172.21.15.141:0/774192491 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733e00640 1 --1- 172.21.15.141:0/774192491 >> v1:172.21.15.188:6789/0 conn(0x7f673413df80 0x7f673415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:47864/0 (socket says 172.21.15.141:47864) 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733400640 1 -- 172.21.15.141:0/774192491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 msgr2=0x7f67340cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733400640 1 --2- 172.21.15.141:0/774192491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733400640 1 -- 172.21.15.141:0/774192491 >> v1:172.21.15.188:6789/0 conn(0x7f673413df80 legacy=0x7f673415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.285+0000 7f6733400640 1 -- 172.21.15.141:0/774192491 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673406ca30 con 0x7f673413e360 2024-01-16T14:58:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.286+0000 7f6733400640 1 --2- 172.21.15.141:0/774192491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 0x7f67340cf290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f671c009870 tx=0x7f671c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c2ee27bc3746169 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f6732a00640 1 -- 172.21.15.141:0/774192491 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c030840 con 0x7f673413e360 2024-01-16T14:58:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f6732a00640 1 -- 172.21.15.141:0/774192491 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c0309a0 con 0x7f673413e360 2024-01-16T14:58:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f6732a00640 1 -- 172.21.15.141:0/774192491 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c030d30 con 0x7f673413e360 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f673b200640 1 -- 172.21.15.141:0/774192491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 msgr2=0x7f67340cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f673b200640 1 --2- 172.21.15.141:0/774192491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 0x7f67340cf290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f671c009870 tx=0x7f671c02b980 comp rx=0 tx=0).stop 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f673b200640 1 -- 172.21.15.141:0/774192491 shutdown_connections 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f673b200640 1 --2- 172.21.15.141:0/774192491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 0x7f67340cf290 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f673b200640 1 --2- 172.21.15.141:0/774192491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f673b200640 1 -- 172.21.15.141:0/774192491 >> 172.21.15.141:0/774192491 conn(0x7f673406a800 msgr2=0x7f673406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.287+0000 7f673b200640 1 -- 172.21.15.141:0/774192491 shutdown_connections 2024-01-16T14:58:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.288+0000 7f673b200640 1 -- 172.21.15.141:0/774192491 wait complete. 2024-01-16T14:58:02.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.289+0000 7f673b200640 1 Processor -- start 2024-01-16T14:58:02.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.289+0000 7f673b200640 1 -- start start 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.290+0000 7f673b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f673413df80 0x7f67341e5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.290+0000 7f673b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 0x7f673413cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.290+0000 7f673b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.290+0000 7f673b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67341f44c0 con 0x7f673413df80 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.290+0000 7f673b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67341f47a0 con 0x7f673413e360 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.290+0000 7f673b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67341f4a80 con 0x7f673415f9c0 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f673413df80 0x7f67341e5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f673413df80 0x7f67341e5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42856/0 (socket says 172.21.15.141:42856) 2024-01-16T14:58:02.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 -- 172.21.15.141:0/2410654435 learned_addr learned my addr 172.21.15.141:0/2410654435 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6739200640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733400640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 0x7f673413cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 -- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 msgr2=0x7f67340b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 -- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 msgr2=0x7f673413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 0x7f673413cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 -- 172.21.15.141:0/2410654435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0094d0 con 0x7f673413df80 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.291+0000 7f6733e00640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f673413df80 0x7f67341e5d40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f672400c3c0 tx=0x7f672400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.292+0000 7f672be00640 1 -- 172.21.15.141:0/2410654435 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67240074e0 con 0x7f673413df80 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.292+0000 7f672be00640 1 -- 172.21.15.141:0/2410654435 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6724007660 con 0x7f673413df80 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.292+0000 7f672be00640 1 -- 172.21.15.141:0/2410654435 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724004a20 con 0x7f673413df80 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.292+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67341f4d60 con 0x7f673413df80 2024-01-16T14:58:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.292+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67341e6280 con 0x7f673413df80 2024-01-16T14:58:02.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.293+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67340675d0 con 0x7f673413df80 2024-01-16T14:58:02.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.294+0000 7f672be00640 1 -- 172.21.15.141:0/2410654435 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6724004b80 con 0x7f673413df80 2024-01-16T14:58:02.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.295+0000 7f672be00640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f670c06b2c0 0x7f670c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.295+0000 7f672be00640 1 -- 172.21.15.141:0/2410654435 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f6724089620 con 0x7f673413df80 2024-01-16T14:58:02.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.295+0000 7f6733400640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f670c06b2c0 0x7f670c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.296+0000 7f6733400640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f670c06b2c0 0x7f670c06d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f671c036000 tx=0x7f671c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:02.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.299+0000 7f672be00640 1 -- 172.21.15.141:0/2410654435 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6724054370 con 0x7f673413df80 2024-01-16T14:58:02.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.481+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f673406df00 con 0x7f673413df80 2024-01-16T14:58:02.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.481+0000 7f672be00640 1 -- 172.21.15.141:0/2410654435 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6724057990 con 0x7f673413df80 2024-01-16T14:58:02.464 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:58:02.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f670c06b2c0 msgr2=0x7f670c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f670c06b2c0 0x7f670c06d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f671c036000 tx=0x7f671c008030 comp rx=0 tx=0).stop 2024-01-16T14:58:02.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f673413df80 msgr2=0x7f67341e5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f673413df80 0x7f67341e5d40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f672400c3c0 tx=0x7f672400c880 comp rx=0 tx=0).stop 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 shutdown_connections 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f670c06b2c0 0x7f670c06d780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f673415f9c0 0x7f67340b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f673413e360 0x7f673413cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 --2- 172.21.15.141:0/2410654435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f673413df80 0x7f67341e5d40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 >> 172.21.15.141:0/2410654435 conn(0x7f673406a800 msgr2=0x7f67340aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 shutdown_connections 2024-01-16T14:58:02.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.484+0000 7f673b200640 1 -- 172.21.15.141:0/2410654435 wait complete. 2024-01-16T14:58:02.485 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:58:02.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.722+0000 7fb634000640 1 Processor -- start 2024-01-16T14:58:02.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.722+0000 7fb634000640 1 -- start start 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb634000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb62c13e360 0x7fb62c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb634000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb62c15f9c0 0x7fb62c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb634000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62c15c5f0 con 0x7fb62c13df80 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb634000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c06c470 con 0x7fb62c13e360 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb634000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c06c750 con 0x7fb62c15f9c0 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb630c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb62c13e360 0x7fb62c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb630c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb62c13e360 0x7fb62c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42872/0 (socket says 172.21.15.141:42872) 2024-01-16T14:58:02.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb630c00640 1 -- 172.21.15.141:0/1059609903 learned_addr learned my addr 172.21.15.141:0/1059609903 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:02.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.724+0000 7fb631600640 1 --1- 172.21.15.141:0/1059609903 >> v1:172.21.15.141:6789/0 conn(0x7fb62c13df80 0x7fb62c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49684/0 (socket says 172.21.15.141:49684) 2024-01-16T14:58:02.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.725+0000 7fb632000640 1 --2- 172.21.15.141:0/1059609903 >> v2:172.21.15.188:3300/0 conn(0x7fb62c15f9c0 0x7fb62c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.725+0000 7fb630c00640 1 -- 172.21.15.141:0/1059609903 >> v2:172.21.15.188:3300/0 conn(0x7fb62c15f9c0 msgr2=0x7fb62c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:02.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.725+0000 7fb630c00640 1 --2- 172.21.15.141:0/1059609903 >> v2:172.21.15.188:3300/0 conn(0x7fb62c15f9c0 0x7fb62c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.725+0000 7fb630c00640 1 -- 172.21.15.141:0/1059609903 >> v1:172.21.15.141:6789/0 conn(0x7fb62c13df80 legacy=0x7fb62c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.725+0000 7fb630c00640 1 -- 172.21.15.141:0/1059609903 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c06ca30 con 0x7fb62c13e360 2024-01-16T14:58:02.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.725+0000 7fb62be00640 1 -- 172.21.15.141:0/1059609903 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480470289 0 0) 0x7fb62c15c5f0 con 0x7fb62c13df80 2024-01-16T14:58:02.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.726+0000 7fb630c00640 1 --2- 172.21.15.141:0/1059609903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13e360 0x7fb62c0cf290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0ccae0 tx=0x7fb61402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a734731c2c197a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:02.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.726+0000 7fb62be00640 1 -- 172.21.15.141:0/1059609903 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614030860 con 0x7fb62c13e360 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.726+0000 7fb62be00640 1 -- 172.21.15.141:0/1059609903 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6140309c0 con 0x7fb62c13e360 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.726+0000 7fb62be00640 1 -- 172.21.15.141:0/1059609903 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614030d50 con 0x7fb62c13e360 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 -- 172.21.15.141:0/1059609903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13e360 msgr2=0x7fb62c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 --2- 172.21.15.141:0/1059609903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13e360 0x7fb62c0cf290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0ccae0 tx=0x7fb61402b9a0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 -- 172.21.15.141:0/1059609903 shutdown_connections 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 --2- 172.21.15.141:0/1059609903 >> v2:172.21.15.188:3300/0 conn(0x7fb62c15f9c0 0x7fb62c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 --2- 172.21.15.141:0/1059609903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13e360 0x7fb62c0cf290 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 -- 172.21.15.141:0/1059609903 >> 172.21.15.141:0/1059609903 conn(0x7fb62c06a800 msgr2=0x7fb62c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 -- 172.21.15.141:0/1059609903 shutdown_connections 2024-01-16T14:58:02.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.727+0000 7fb634000640 1 -- 172.21.15.141:0/1059609903 wait complete. 2024-01-16T14:58:02.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.729+0000 7fb634000640 1 Processor -- start 2024-01-16T14:58:02.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.729+0000 7fb634000640 1 -- start start 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb634000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13df80 0x7fb62c1f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb634000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb62c13e360 0x7fb62c152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb634000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb62c15f9c0 0x7fb62c0a92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb634000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c153db0 con 0x7fb62c13df80 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb634000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c152f50 con 0x7fb62c15f9c0 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb634000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0a9830 con 0x7fb62c13e360 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb631600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13df80 0x7fb62c1f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb631600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13df80 0x7fb62c1f6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42876/0 (socket says 172.21.15.141:42876) 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.730+0000 7fb631600640 1 -- 172.21.15.141:0/1767986948 learned_addr learned my addr 172.21.15.141:0/1767986948 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:02.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb632000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb62c15f9c0 0x7fb62c0a92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb630c00640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb62c13e360 0x7fb62c152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb631600640 1 -- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb62c13e360 msgr2=0x7fb62c152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb631600640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb62c13e360 0x7fb62c152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb631600640 1 -- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb62c15f9c0 msgr2=0x7fb62c0a92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb631600640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb62c15f9c0 0x7fb62c0a92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb631600640 1 -- 172.21.15.141:0/1767986948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6140094d0 con 0x7fb62c13df80 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb632000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb62c15f9c0 0x7fb62c0a92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb630c00640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb62c13e360 0x7fb62c152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb631600640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13df80 0x7fb62c1f6110 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c410 tx=0x7fb61c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb629600640 1 -- 172.21.15.141:0/1767986948 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0074e0 con 0x7fb62c13df80 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb629600640 1 -- 172.21.15.141:0/1767986948 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c007640 con 0x7fb62c13df80 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c153130 con 0x7fb62c13df80 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.731+0000 7fb629600640 1 -- 172.21.15.141:0/1767986948 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c004a20 con 0x7fb62c13df80 2024-01-16T14:58:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.732+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c1536f0 con 0x7fb62c13df80 2024-01-16T14:58:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.733+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c0675d0 con 0x7fb62c13df80 2024-01-16T14:58:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.734+0000 7fb629600640 1 -- 172.21.15.141:0/1767986948 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb61c004b80 con 0x7fb62c13df80 2024-01-16T14:58:02.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.735+0000 7fb629600640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb60406b2c0 0x7fb60406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:02.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.735+0000 7fb629600640 1 -- 172.21.15.141:0/1767986948 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fb61c089670 con 0x7fb62c13df80 2024-01-16T14:58:02.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.735+0000 7fb630c00640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb60406b2c0 0x7fb60406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:02.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.736+0000 7fb630c00640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb60406b2c0 0x7fb60406d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb614036000 tx=0x7fb61403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:02.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.740+0000 7fb629600640 1 -- 172.21.15.141:0/1767986948 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb61c057ef0 con 0x7fb62c13df80 2024-01-16T14:58:02.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.927+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb62c06df00 con 0x7fb62c13df80 2024-01-16T14:58:02.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.928+0000 7fb629600640 1 -- 172.21.15.141:0/1767986948 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb61c057a80 con 0x7fb62c13df80 2024-01-16T14:58:02.910 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:58:02.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb60406b2c0 msgr2=0x7fb60406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb60406b2c0 0x7fb60406d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb614036000 tx=0x7fb61403b040 comp rx=0 tx=0).stop 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13df80 msgr2=0x7fb62c1f6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13df80 0x7fb62c1f6110 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c410 tx=0x7fb61c00c8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 shutdown_connections 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb60406b2c0 0x7fb60406d780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb62c15f9c0 0x7fb62c0a92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb62c13e360 0x7fb62c152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 --2- 172.21.15.141:0/1767986948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb62c13df80 0x7fb62c1f6110 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 >> 172.21.15.141:0/1767986948 conn(0x7fb62c06a800 msgr2=0x7fb62c0705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 shutdown_connections 2024-01-16T14:58:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:02.931+0000 7fb634000640 1 -- 172.21.15.141:0/1767986948 wait complete. 2024-01-16T14:58:02.931 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:58:03.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.145+0000 7f7b67600640 1 Processor -- start 2024-01-16T14:58:03.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.145+0000 7f7b67600640 1 -- start start 2024-01-16T14:58:03.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.146+0000 7f7b67600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7b60143ae0 0x7f7b600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.146+0000 7f7b67600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e3b0 0x7f7b600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.146+0000 7f7b67600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b600cfc30 con 0x7f7b600cf7d0 2024-01-16T14:58:03.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.146+0000 7f7b67600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b600b1ee0 con 0x7f7b60143ae0 2024-01-16T14:58:03.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.146+0000 7f7b67600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b600b2780 con 0x7f7b6015e3b0 2024-01-16T14:58:03.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.147+0000 7f7b57c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7b600cf7d0 0x7f7b6015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56948/0 (socket says 172.21.15.141:56948) 2024-01-16T14:58:03.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.147+0000 7f7b57c00640 1 -- 172.21.15.141:0/3801347916 learned_addr learned my addr 172.21.15.141:0/3801347916 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:03.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.147+0000 7f7b65600640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e3b0 0x7f7b600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.147+0000 7f7b64c00640 1 --2- 172.21.15.141:0/3801347916 >> v2:172.21.15.188:3300/0 conn(0x7f7b60143ae0 0x7f7b600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.147+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448075201 0 0) 0x7f7b600cfc30 con 0x7f7b600cf7d0 2024-01-16T14:58:03.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.147+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b40003520 con 0x7f7b600cf7d0 2024-01-16T14:58:03.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2083466876 0 0) 0x7f7b40003520 con 0x7f7b600cf7d0 2024-01-16T14:58:03.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e3b0 msgr2=0x7f7b600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e3b0 0x7f7b600702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 >> v2:172.21.15.188:3300/0 conn(0x7f7b60143ae0 msgr2=0x7f7b600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 --2- 172.21.15.141:0/3801347916 >> v2:172.21.15.188:3300/0 conn(0x7f7b60143ae0 0x7f7b600cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6015e990 con 0x7f7b600cf7d0 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7b48001670 con 0x7f7b600cf7d0 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 >> v1:172.21.15.141:6789/0 conn(0x7f7b600cf7d0 legacy=0x7f7b6015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b400047c0 0x7f7b40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e3b0 0x7f7b40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b600b1ee0 con 0x7f7b40003970 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b600b2780 con 0x7f7b400047c0 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4000db90 con 0x7f7b6015e3b0 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b64c00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b64c00640 1 -- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e3b0 msgr2=0x7f7b40005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b64c00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e3b0 0x7f7b40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b57c00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b400047c0 0x7f7b40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b64c00640 1 -- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b400047c0 msgr2=0x7f7b40004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b64c00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b400047c0 0x7f7b40004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.148+0000 7f7b64c00640 1 -- 172.21.15.141:0/3801347916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40003520 con 0x7f7b40003970 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b64c00640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b40003d90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009ac0 tx=0x7f7b400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80ed03128d3fabd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5002c710 con 0x7f7b40003970 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5002c870 con 0x7f7b40003970 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b5fe00640 1 -- 172.21.15.141:0/3801347916 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5002cc00 con 0x7f7b40003970 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b67600640 1 -- 172.21.15.141:0/3801347916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 msgr2=0x7f7b40003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b67600640 1 --2- 172.21.15.141:0/3801347916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b40003d90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009ac0 tx=0x7f7b400057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b65600640 1 -- 172.21.15.141:0/3801347916 reap_dead start 2024-01-16T14:58:03.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b67600640 1 -- 172.21.15.141:0/3801347916 shutdown_connections 2024-01-16T14:58:03.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b67600640 1 -- 172.21.15.141:0/3801347916 >> 172.21.15.141:0/3801347916 conn(0x7f7b6006a800 msgr2=0x7f7b6006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:03.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b67600640 1 -- 172.21.15.141:0/3801347916 shutdown_connections 2024-01-16T14:58:03.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.150+0000 7f7b67600640 1 -- 172.21.15.141:0/3801347916 wait complete. 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.152+0000 7f7b67600640 1 Processor -- start 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.152+0000 7f7b67600640 1 -- start start 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b67600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b601ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b67600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6012df90 0x7f7b6015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b67600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b60140460 0x7f7b601408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b67600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6012efc0 con 0x7f7b40003970 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b67600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b60140e30 con 0x7f7b60140460 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b67600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b601ee7f0 con 0x7f7b6012df90 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b64c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b601ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b64c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b601ee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45648/0 (socket says 172.21.15.141:45648) 2024-01-16T14:58:03.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.153+0000 7f7b64c00640 1 -- 172.21.15.141:0/1413727439 learned_addr learned my addr 172.21.15.141:0/1413727439 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b57c00640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6012df90 0x7f7b6015cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b65600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b60140460 0x7f7b601408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b64c00640 1 -- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6012df90 msgr2=0x7f7b6015cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b64c00640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6012df90 0x7f7b6015cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b64c00640 1 -- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b60140460 msgr2=0x7f7b601408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b64c00640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b60140460 0x7f7b601408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b64c00640 1 -- 172.21.15.141:0/1413727439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6012e440 con 0x7f7b40003970 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b65600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b60140460 0x7f7b601408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b64c00640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b601ee2b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7b500099e0 tx=0x7f7b50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b5d600640 1 -- 172.21.15.141:0/1413727439 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50003810 con 0x7f7b40003970 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b5d600640 1 -- 172.21.15.141:0/1413727439 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5002de90 con 0x7f7b40003970 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.154+0000 7f7b5d600640 1 -- 172.21.15.141:0/1413727439 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50003bb0 con 0x7f7b40003970 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.155+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6012e720 con 0x7f7b40003970 2024-01-16T14:58:03.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.155+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6012ece0 con 0x7f7b40003970 2024-01-16T14:58:03.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.156+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6012bf60 con 0x7f7b40003970 2024-01-16T14:58:03.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.157+0000 7f7b5d600640 1 -- 172.21.15.141:0/1413727439 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7b50003d10 con 0x7f7b40003970 2024-01-16T14:58:03.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.157+0000 7f7b5d600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b2f0 0x7f7b3806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.158+0000 7f7b5d600640 1 -- 172.21.15.141:0/1413727439 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f7b500abdc0 con 0x7f7b40003970 2024-01-16T14:58:03.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.158+0000 7f7b57c00640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b2f0 0x7f7b3806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.159+0000 7f7b57c00640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b2f0 0x7f7b3806d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7b6012e370 tx=0x7f7b48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.162+0000 7f7b5d600640 1 -- 172.21.15.141:0/1413727439 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b50076c30 con 0x7f7b40003970 2024-01-16T14:58:03.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.363+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7b600675d0 con 0x7f7b40003970 2024-01-16T14:58:03.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.364+0000 7f7b5d600640 1 -- 172.21.15.141:0/1413727439 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7b5007a250 con 0x7f7b40003970 2024-01-16T14:58:03.345 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:58:03.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.366+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b2f0 msgr2=0x7f7b3806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.366+0000 7f7b67600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b2f0 0x7f7b3806d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7b6012e370 tx=0x7f7b48009040 comp rx=0 tx=0).stop 2024-01-16T14:58:03.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.366+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 msgr2=0x7f7b601ee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.366+0000 7f7b67600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b601ee2b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7b500099e0 tx=0x7f7b50004730 comp rx=0 tx=0).stop 2024-01-16T14:58:03.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 shutdown_connections 2024-01-16T14:58:03.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b2f0 0x7f7b3806d7b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b60140460 0x7f7b601408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6012df90 0x7f7b6015cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 --2- 172.21.15.141:0/1413727439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b40003970 0x7f7b601ee2b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 >> 172.21.15.141:0/1413727439 conn(0x7f7b6006a800 msgr2=0x7f7b6006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:03.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 shutdown_connections 2024-01-16T14:58:03.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.367+0000 7f7b67600640 1 -- 172.21.15.141:0/1413727439 wait complete. 2024-01-16T14:58:03.368 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T14:58:03.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:03.368 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-16T14:58:03.453 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:58:03.453 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:58:03.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:03.454 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T14:58:03.502 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:58:03.502 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:03.553 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:58:03.553 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:03.638 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:03.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.869+0000 7faf63200640 1 Processor -- start 2024-01-16T14:58:03.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.870+0000 7faf63200640 1 -- start start 2024-01-16T14:58:03.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.871+0000 7faf63200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7faf5c160920 0x7faf5c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.871+0000 7faf63200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c0d0a10 con 0x7faf5c0cfea0 2024-01-16T14:58:03.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.871+0000 7faf63200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c161e30 con 0x7faf5c0cf9f0 2024-01-16T14:58:03.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.871+0000 7faf63200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf5c0d1f50 con 0x7faf5c160920 2024-01-16T14:58:03.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf62800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7faf5c0cfea0 0x7faf5c161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56964/0 (socket says 172.21.15.141:56964) 2024-01-16T14:58:03.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf62800640 1 -- 172.21.15.141:0/3030466371 learned_addr learned my addr 172.21.15.141:0/3030466371 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:03.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf61e00640 1 --2- 172.21.15.141:0/3030466371 >> v2:172.21.15.188:3300/0 conn(0x7faf5c160920 0x7faf5c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030341118 0 0) 0x7faf5c0d0a10 con 0x7faf5c0cfea0 2024-01-16T14:58:03.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf3c003520 con 0x7faf5c0cfea0 2024-01-16T14:58:03.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429466080 0 0) 0x7faf5c161e30 con 0x7faf5c0cf9f0 2024-01-16T14:58:03.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5c0d0a10 con 0x7faf5c0cf9f0 2024-01-16T14:58:03.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.872+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333101113 0 0) 0x7faf3c003520 con 0x7faf5c0cfea0 2024-01-16T14:58:03.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 >> v2:172.21.15.188:3300/0 conn(0x7faf5c160920 msgr2=0x7faf5c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 --2- 172.21.15.141:0/3030466371 >> v2:172.21.15.188:3300/0 conn(0x7faf5c160920 0x7faf5c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 >> v1:172.21.15.188:6789/0 conn(0x7faf5c0cf9f0 legacy=0x7faf5c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c0cf080 con 0x7faf5c0cfea0 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7faf540019f0 con 0x7faf5c0cfea0 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 >> v1:172.21.15.141:6789/0 conn(0x7faf5c0cfea0 legacy=0x7faf5c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf3c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf3c0047c0 0x7faf3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf3c005180 0x7faf3c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c0d1f50 con 0x7faf3c003b00 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c00c000 con 0x7faf3c005180 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c00c1c0 con 0x7faf3c0047c0 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf61e00640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf3c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf61400640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf3c0047c0 0x7faf3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf62800640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf3c005180 0x7faf3c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf61e00640 1 -- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf3c0047c0 msgr2=0x7faf3c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf61e00640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf3c0047c0 0x7faf3c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf61e00640 1 -- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf3c005180 msgr2=0x7faf3c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:03.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf61e00640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf3c005180 0x7faf3c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf62800640 1 -- 172.21.15.141:0/3030466371 reap_dead start 2024-01-16T14:58:03.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.873+0000 7faf61e00640 1 -- 172.21.15.141:0/3030466371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c003520 con 0x7faf3c003b00 2024-01-16T14:58:03.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.875+0000 7faf61e00640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf3c003f00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009ac0 tx=0x7faf3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fe8fad731f212b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.875+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c02c6b0 con 0x7faf3c003b00 2024-01-16T14:58:03.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.875+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c02c810 con 0x7faf3c003b00 2024-01-16T14:58:03.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.875+0000 7faf60a00640 1 -- 172.21.15.141:0/3030466371 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c02cba0 con 0x7faf3c003b00 2024-01-16T14:58:03.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.875+0000 7faf63200640 1 -- 172.21.15.141:0/3030466371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 msgr2=0x7faf3c003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.875+0000 7faf63200640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf3c003f00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009ac0 tx=0x7faf3c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.876+0000 7faf63200640 1 -- 172.21.15.141:0/3030466371 shutdown_connections 2024-01-16T14:58:03.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.876+0000 7faf63200640 1 --2- 172.21.15.141:0/3030466371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf3c003f00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.876+0000 7faf63200640 1 -- 172.21.15.141:0/3030466371 >> 172.21.15.141:0/3030466371 conn(0x7faf5c0d3090 msgr2=0x7faf5c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:03.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.876+0000 7faf63200640 1 -- 172.21.15.141:0/3030466371 shutdown_connections 2024-01-16T14:58:03.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.876+0000 7faf63200640 1 -- 172.21.15.141:0/3030466371 wait complete. 2024-01-16T14:58:03.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.877+0000 7faf63200640 1 Processor -- start 2024-01-16T14:58:03.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.878+0000 7faf63200640 1 -- start start 2024-01-16T14:58:03.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.878+0000 7faf63200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf5c15f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.878+0000 7faf63200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf5c15d2f0 0x7faf5c150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.878+0000 7faf63200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf5c15b050 0x7faf5c15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf63200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c15ba10 con 0x7faf3c003b00 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf63200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c15d8b0 con 0x7faf5c15b050 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf63200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c15db90 con 0x7faf5c15d2f0 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf5c15f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf5c15f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45668/0 (socket says 172.21.15.141:45668) 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 -- 172.21.15.141:0/749492323 learned_addr learned my addr 172.21.15.141:0/749492323 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61400640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf5c15d2f0 0x7faf5c150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf62800640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf5c15b050 0x7faf5c15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 -- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf5c15d2f0 msgr2=0x7faf5c150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf5c15d2f0 0x7faf5c150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 -- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf5c15b050 msgr2=0x7faf5c15b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf5c15b050 0x7faf5c15b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.879+0000 7faf61e00640 1 -- 172.21.15.141:0/749492323 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c0b7060 con 0x7faf3c003b00 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.880+0000 7faf61e00640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf5c15f700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009ac0 tx=0x7faf4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.880+0000 7faf5a000640 1 -- 172.21.15.141:0/749492323 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c003810 con 0x7faf3c003b00 2024-01-16T14:58:03.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.880+0000 7faf5a000640 1 -- 172.21.15.141:0/749492323 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c003970 con 0x7faf3c003b00 2024-01-16T14:58:03.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.880+0000 7faf5a000640 1 -- 172.21.15.141:0/749492323 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c003d00 con 0x7faf3c003b00 2024-01-16T14:58:03.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.880+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5c0b7340 con 0x7faf3c003b00 2024-01-16T14:58:03.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.880+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5c150bd0 con 0x7faf3c003b00 2024-01-16T14:58:03.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.881+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5c0b74d0 con 0x7faf3c003b00 2024-01-16T14:58:03.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.882+0000 7faf5a000640 1 -- 172.21.15.141:0/749492323 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faf4c040450 con 0x7faf3c003b00 2024-01-16T14:58:03.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.883+0000 7faf5a000640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf3406b2c0 0x7faf3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:03.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.884+0000 7faf5a000640 1 -- 172.21.15.141:0/749492323 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7faf4c0abd20 con 0x7faf3c003b00 2024-01-16T14:58:03.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.884+0000 7faf61400640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf3406b2c0 0x7faf3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:03.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.885+0000 7faf61400640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf3406b2c0 0x7faf3406d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf5c15bba0 tx=0x7faf44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:03.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:03.889+0000 7faf5a000640 1 -- 172.21.15.141:0/749492323 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf4c076a70 con 0x7faf3c003b00 2024-01-16T14:58:04.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.082+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf5c0d28c0 con 0x7faf3c003b00 2024-01-16T14:58:04.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.085+0000 7faf5a000640 1 -- 172.21.15.141:0/749492323 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7faf4c003e60 con 0x7faf3c003b00 2024-01-16T14:58:04.066 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:04.066 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:04.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf3406b2c0 msgr2=0x7faf3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf3406b2c0 0x7faf3406d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf5c15bba0 tx=0x7faf44009040 comp rx=0 tx=0).stop 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 msgr2=0x7faf5c15f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf5c15f700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009ac0 tx=0x7faf4c004730 comp rx=0 tx=0).stop 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 shutdown_connections 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf3406b2c0 0x7faf3406d780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf5c15b050 0x7faf5c15b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf5c15d2f0 0x7faf5c150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 --2- 172.21.15.141:0/749492323 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf3c003b00 0x7faf5c15f700 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 >> 172.21.15.141:0/749492323 conn(0x7faf5c0d3090 msgr2=0x7faf5c06c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 shutdown_connections 2024-01-16T14:58:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.088+0000 7faf63200640 1 -- 172.21.15.141:0/749492323 wait complete. 2024-01-16T14:58:04.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:58:04.089 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:58:04.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.320+0000 7f5a8a000640 1 Processor -- start 2024-01-16T14:58:04.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.320+0000 7f5a8a000640 1 -- start start 2024-01-16T14:58:04.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a8a000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5a84143ae0 0x7f5a840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a8a000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5a840cf7d0 0x7f5a8415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a8a000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840cfc30 con 0x7f5a8415e1e0 2024-01-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a8a000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a840b1ee0 con 0x7f5a84143ae0 2024-01-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a8a000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a840b2780 con 0x7f5a840cf7d0 2024-01-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a83e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5a8415e1e0 0x7f5a8415a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56980/0 (socket says 172.21.15.141:56980) 2024-01-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a83e00640 1 -- 172.21.15.141:0/3589920619 learned_addr learned my addr 172.21.15.141:0/3589920619 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:04.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.322+0000 7f5a83400640 1 --2- 172.21.15.141:0/3589920619 >> v2:172.21.15.141:3300/0 conn(0x7f5a84143ae0 0x7f5a840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.323+0000 7f5a7be00640 1 --2- 172.21.15.141:0/3589920619 >> v2:172.21.15.188:3300/0 conn(0x7f5a840cf7d0 0x7f5a8415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.323+0000 7f5a82a00640 1 -- 172.21.15.141:0/3589920619 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441046189 0 0) 0x7f5a840cfc30 con 0x7f5a8415e1e0 2024-01-16T14:58:04.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.323+0000 7f5a82a00640 1 -- 172.21.15.141:0/3589920619 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a64003520 con 0x7f5a8415e1e0 2024-01-16T14:58:04.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.323+0000 7f5a83400640 1 -- 172.21.15.141:0/3589920619 >> v2:172.21.15.188:3300/0 conn(0x7f5a840cf7d0 msgr2=0x7f5a8415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:04.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.323+0000 7f5a83400640 1 --2- 172.21.15.141:0/3589920619 >> v2:172.21.15.188:3300/0 conn(0x7f5a840cf7d0 0x7f5a8415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.323+0000 7f5a83400640 1 -- 172.21.15.141:0/3589920619 >> v1:172.21.15.141:6789/0 conn(0x7f5a8415e1e0 legacy=0x7f5a8415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.323+0000 7f5a83400640 1 -- 172.21.15.141:0/3589920619 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8415e990 con 0x7f5a84143ae0 2024-01-16T14:58:04.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.324+0000 7f5a83400640 1 --2- 172.21.15.141:0/3589920619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a84143ae0 0x7f5a840cf290 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5a840cb830 tx=0x7f5a7402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa895d23969980 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.324+0000 7f5a82a00640 1 -- 172.21.15.141:0/3589920619 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74031830 con 0x7f5a84143ae0 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a82a00640 1 -- 172.21.15.141:0/3589920619 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a74031990 con 0x7f5a84143ae0 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a82a00640 1 -- 172.21.15.141:0/3589920619 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74031d20 con 0x7f5a84143ae0 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 -- 172.21.15.141:0/3589920619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a84143ae0 msgr2=0x7f5a840cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 --2- 172.21.15.141:0/3589920619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a84143ae0 0x7f5a840cf290 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5a840cb830 tx=0x7f5a7402b8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 -- 172.21.15.141:0/3589920619 shutdown_connections 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 --2- 172.21.15.141:0/3589920619 >> v2:172.21.15.188:3300/0 conn(0x7f5a840cf7d0 0x7f5a8415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 --2- 172.21.15.141:0/3589920619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a84143ae0 0x7f5a840cf290 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 -- 172.21.15.141:0/3589920619 >> 172.21.15.141:0/3589920619 conn(0x7f5a8406a800 msgr2=0x7f5a8406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 -- 172.21.15.141:0/3589920619 shutdown_connections 2024-01-16T14:58:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.325+0000 7f5a8a000640 1 -- 172.21.15.141:0/3589920619 wait complete. 2024-01-16T14:58:04.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.327+0000 7f5a8a000640 1 Processor -- start 2024-01-16T14:58:04.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.327+0000 7f5a8a000640 1 -- start start 2024-01-16T14:58:04.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.328+0000 7f5a8a000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a840cf7d0 0x7f5a84142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.328+0000 7f5a8a000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a84143ae0 0x7f5a841677a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.328+0000 7f5a8a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a8415e1e0 0x7f5a84165f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.328+0000 7f5a8a000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a84166480 con 0x7f5a8415e1e0 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.328+0000 7f5a8a000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a84166700 con 0x7f5a840cf7d0 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.328+0000 7f5a8a000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a841669e0 con 0x7f5a84143ae0 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.328+0000 7f5a83e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a8415e1e0 0x7f5a84165f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a8415e1e0 0x7f5a84165f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45694/0 (socket says 172.21.15.141:45694) 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 -- 172.21.15.141:0/2554085398 learned_addr learned my addr 172.21.15.141:0/2554085398 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83400640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a840cf7d0 0x7f5a84142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a7be00640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a84143ae0 0x7f5a841677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 -- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a84143ae0 msgr2=0x7f5a841677a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a84143ae0 0x7f5a841677a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 -- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a840cf7d0 msgr2=0x7f5a84142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a840cf7d0 0x7f5a84142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 -- 172.21.15.141:0/2554085398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74009400 con 0x7f5a8415e1e0 2024-01-16T14:58:04.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.329+0000 7f5a83e00640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a8415e1e0 0x7f5a84165f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00cb40 tx=0x7f5a7c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:04.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.330+0000 7f5a7b400640 1 -- 172.21.15.141:0/2554085398 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00aca0 con 0x7f5a8415e1e0 2024-01-16T14:58:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.330+0000 7f5a7b400640 1 -- 172.21.15.141:0/2554085398 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00ae00 con 0x7f5a8415e1e0 2024-01-16T14:58:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.330+0000 7f5a7b400640 1 -- 172.21.15.141:0/2554085398 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c012620 con 0x7f5a8415e1e0 2024-01-16T14:58:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.330+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a84166cc0 con 0x7f5a8415e1e0 2024-01-16T14:58:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.330+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a84167280 con 0x7f5a8415e1e0 2024-01-16T14:58:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.331+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a840b2bf0 con 0x7f5a8415e1e0 2024-01-16T14:58:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.332+0000 7f5a7b400640 1 -- 172.21.15.141:0/2554085398 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5a7c007070 con 0x7f5a8415e1e0 2024-01-16T14:58:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.333+0000 7f5a7b400640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a5c06b2c0 0x7f5a5c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.333+0000 7f5a7b400640 1 -- 172.21.15.141:0/2554085398 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f5a7c089c90 con 0x7f5a8415e1e0 2024-01-16T14:58:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.334+0000 7f5a83400640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a5c06b2c0 0x7f5a5c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.335+0000 7f5a83400640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a5c06b2c0 0x7f5a5c06d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5a74036000 tx=0x7f5a7403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:04.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.337+0000 7f5a7b400640 1 -- 172.21.15.141:0/2554085398 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a840b2bf0 con 0x7f5a8415e1e0 2024-01-16T14:58:04.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.527+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a840b2bf0 con 0x7f5a8415e1e0 2024-01-16T14:58:04.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.528+0000 7f5a7b400640 1 -- 172.21.15.141:0/2554085398 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a840b2bf0 con 0x7f5a8415e1e0 2024-01-16T14:58:04.509 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a5c06b2c0 msgr2=0x7f5a5c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a5c06b2c0 0x7f5a5c06d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5a74036000 tx=0x7f5a7403c040 comp rx=0 tx=0).stop 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a8415e1e0 msgr2=0x7f5a84165f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a8415e1e0 0x7f5a84165f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00cb40 tx=0x7f5a7c00a060 comp rx=0 tx=0).stop 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 shutdown_connections 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a5c06b2c0 0x7f5a5c06d780 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a8415e1e0 0x7f5a84165f40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a84143ae0 0x7f5a841677a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 --2- 172.21.15.141:0/2554085398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a840cf7d0 0x7f5a84142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 >> 172.21.15.141:0/2554085398 conn(0x7f5a8406a800 msgr2=0x7f5a840b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:04.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 shutdown_connections 2024-01-16T14:58:04.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.530+0000 7f5a8a000640 1 -- 172.21.15.141:0/2554085398 wait complete. 2024-01-16T14:58:04.529 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:58:04.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.755+0000 7ff17c400640 1 Processor -- start 2024-01-16T14:58:04.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.756+0000 7ff17c400640 1 -- start start 2024-01-16T14:58:04.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.757+0000 7ff17c400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff1741610e0 0x7ff1740d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.757+0000 7ff17c400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff17415cc90 0x7ff17415d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.757+0000 7ff17c400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1740b2920 con 0x7ff1740cf4c0 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.757+0000 7ff17c400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1740cb670 con 0x7ff17415cc90 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.757+0000 7ff17c400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1740cb950 con 0x7ff1741610e0 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff17a400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff17415cc90 0x7ff17415d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff17a400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff17415cc90 0x7ff17415d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45700/0 (socket says 172.21.15.141:45700) 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff17a400640 1 -- 172.21.15.141:0/602039448 learned_addr learned my addr 172.21.15.141:0/602039448 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff179a00640 1 --2- 172.21.15.141:0/602039448 >> v2:172.21.15.188:3300/0 conn(0x7ff1741610e0 0x7ff1740d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff17a400640 1 -- 172.21.15.141:0/602039448 >> v2:172.21.15.188:3300/0 conn(0x7ff1741610e0 msgr2=0x7ff1740d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff17a400640 1 --2- 172.21.15.141:0/602039448 >> v2:172.21.15.188:3300/0 conn(0x7ff1741610e0 0x7ff1740d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff17a400640 1 -- 172.21.15.141:0/602039448 >> v1:172.21.15.188:6789/0 conn(0x7ff1740cf4c0 legacy=0x7ff1741614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:04.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.758+0000 7ff17a400640 1 -- 172.21.15.141:0/602039448 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1740cbc30 con 0x7ff17415cc90 2024-01-16T14:58:04.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.759+0000 7ff17a400640 1 --2- 172.21.15.141:0/602039448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff17415cc90 0x7ff17415d140 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442e5c4c6ea2de80 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:04.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff173e00640 1 -- 172.21.15.141:0/602039448 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c031870 con 0x7ff17415cc90 2024-01-16T14:58:04.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff173e00640 1 -- 172.21.15.141:0/602039448 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c0319d0 con 0x7ff17415cc90 2024-01-16T14:58:04.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff173e00640 1 -- 172.21.15.141:0/602039448 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c031d60 con 0x7ff17415cc90 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff17c400640 1 -- 172.21.15.141:0/602039448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff17415cc90 msgr2=0x7ff17415d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff17c400640 1 --2- 172.21.15.141:0/602039448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff17415cc90 0x7ff17415d140 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c02b910 comp rx=0 tx=0).stop 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff17c400640 1 -- 172.21.15.141:0/602039448 shutdown_connections 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff17c400640 1 --2- 172.21.15.141:0/602039448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff17415cc90 0x7ff17415d140 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff17c400640 1 --2- 172.21.15.141:0/602039448 >> v2:172.21.15.188:3300/0 conn(0x7ff1741610e0 0x7ff1740d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff17c400640 1 -- 172.21.15.141:0/602039448 >> 172.21.15.141:0/602039448 conn(0x7ff1740d1020 msgr2=0x7ff1740d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.760+0000 7ff17c400640 1 -- 172.21.15.141:0/602039448 shutdown_connections 2024-01-16T14:58:04.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.761+0000 7ff17c400640 1 -- 172.21.15.141:0/602039448 wait complete. 2024-01-16T14:58:04.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.763+0000 7ff17c400640 1 Processor -- start 2024-01-16T14:58:04.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.763+0000 7ff17c400640 1 -- start start 2024-01-16T14:58:04.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17c400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1740cf4c0 0x7ff1741515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17c400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff17415cc90 0x7ff17415efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17c400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1741610e0 0x7ff1741e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17c400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17415ebc0 con 0x7ff1741610e0 2024-01-16T14:58:04.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17c400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff174151b00 con 0x7ff1740cf4c0 2024-01-16T14:58:04.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17c400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17415df40 con 0x7ff17415cc90 2024-01-16T14:58:04.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1741610e0 0x7ff1741e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1741610e0 0x7ff1741e8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45710/0 (socket says 172.21.15.141:45710) 2024-01-16T14:58:04.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff17a400640 1 -- 172.21.15.141:0/3713433389 learned_addr learned my addr 172.21.15.141:0/3713433389 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:04.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.764+0000 7ff179a00640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1740cf4c0 0x7ff1741515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff179000640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff17415cc90 0x7ff17415efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff17a400640 1 -- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff17415cc90 msgr2=0x7ff17415efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff17a400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff17415cc90 0x7ff17415efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff17a400640 1 -- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1740cf4c0 msgr2=0x7ff1741515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff17a400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1740cf4c0 0x7ff1741515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff17a400640 1 -- 172.21.15.141:0/3713433389 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c0094d0 con 0x7ff1741610e0 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff17a400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1741610e0 0x7ff1741e8ba0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff16c02b950 tx=0x7ff16c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff171600640 1 -- 172.21.15.141:0/3713433389 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c03ad20 con 0x7ff1741610e0 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.765+0000 7ff171600640 1 -- 172.21.15.141:0/3713433389 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c0443f0 con 0x7ff1741610e0 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.766+0000 7ff171600640 1 -- 172.21.15.141:0/3713433389 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c03aea0 con 0x7ff1741610e0 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.766+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17415e220 con 0x7ff1741610e0 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.766+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17415e7e0 con 0x7ff1741610e0 2024-01-16T14:58:04.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.767+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17415ed00 con 0x7ff1741610e0 2024-01-16T14:58:04.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.768+0000 7ff171600640 1 -- 172.21.15.141:0/3713433389 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff16c0447a0 con 0x7ff1741610e0 2024-01-16T14:58:04.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.769+0000 7ff171600640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff14c06b2f0 0x7ff14c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:04.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.769+0000 7ff171600640 1 -- 172.21.15.141:0/3713433389 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7ff16c02d040 con 0x7ff1741610e0 2024-01-16T14:58:04.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.770+0000 7ff179a00640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff14c06b2f0 0x7ff14c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:04.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.771+0000 7ff179a00640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff14c06b2f0 0x7ff14c06d7b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff1640099c0 tx=0x7ff164003990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:04.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.774+0000 7ff171600640 1 -- 172.21.15.141:0/3713433389 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff17415ed00 con 0x7ff1741610e0 2024-01-16T14:58:04.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.948+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff17415ed00 con 0x7ff1741610e0 2024-01-16T14:58:04.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.949+0000 7ff171600640 1 -- 172.21.15.141:0/3713433389 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff17415ed00 con 0x7ff1741610e0 2024-01-16T14:58:04.930 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:58:04.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff14c06b2f0 msgr2=0x7ff14c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff14c06b2f0 0x7ff14c06d7b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff1640099c0 tx=0x7ff164003990 comp rx=0 tx=0).stop 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1741610e0 msgr2=0x7ff1741e8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1741610e0 0x7ff1741e8ba0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff16c02b950 tx=0x7ff16c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 shutdown_connections 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff14c06b2f0 0x7ff14c06d7b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1741610e0 0x7ff1741e8ba0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff17415cc90 0x7ff17415efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 --2- 172.21.15.141:0/3713433389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1740cf4c0 0x7ff1741515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 >> 172.21.15.141:0/3713433389 conn(0x7ff1740d1020 msgr2=0x7ff17406e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 shutdown_connections 2024-01-16T14:58:04.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:04.951+0000 7ff17c400640 1 -- 172.21.15.141:0/3713433389 wait complete. 2024-01-16T14:58:04.949 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:58:05.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.167+0000 7eff3ec00640 1 Processor -- start 2024-01-16T14:58:05.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.167+0000 7eff3ec00640 1 -- start start 2024-01-16T14:58:05.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.169+0000 7eff3ec00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7eff380b3b20 0x7eff3815a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.169+0000 7eff3ec00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff380b3fb0 con 0x7eff381505b0 2024-01-16T14:58:05.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.169+0000 7eff3ec00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff380b1e30 con 0x7eff3806d6d0 2024-01-16T14:58:05.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.169+0000 7eff3ec00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3815abd0 con 0x7eff380b3b20 2024-01-16T14:58:05.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.169+0000 7eff3d800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7eff381505b0 0x7eff3806cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56994/0 (socket says 172.21.15.141:56994) 2024-01-16T14:58:05.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.169+0000 7eff3d800640 1 -- 172.21.15.141:0/1528516400 learned_addr learned my addr 172.21.15.141:0/1528516400 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:05.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.169+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207016081 0 0) 0x7eff380b3fb0 con 0x7eff381505b0 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff18003440 con 0x7eff381505b0 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff3e200640 1 --2- 172.21.15.141:0/1528516400 >> v2:172.21.15.188:3300/0 conn(0x7eff380b3b20 0x7eff3815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98766593 0 0) 0x7eff18003440 con 0x7eff381505b0 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 >> v2:172.21.15.188:3300/0 conn(0x7eff380b3b20 msgr2=0x7eff3815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 --2- 172.21.15.141:0/1528516400 >> v2:172.21.15.188:3300/0 conn(0x7eff380b3b20 0x7eff3815a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 >> v1:172.21.15.188:6789/0 conn(0x7eff3806d6d0 legacy=0x7eff380ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380b20a0 con 0x7eff381505b0 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7eff20001670 con 0x7eff381505b0 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 >> v1:172.21.15.141:6789/0 conn(0x7eff381505b0 legacy=0x7eff3806cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff18003ab0 0x7eff18003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18008900 0x7eff1800ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff1800b5e0 0x7eff1800daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3815abd0 con 0x7eff1800b5e0 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1800e380 con 0x7eff18008900 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1800e540 con 0x7eff18003ab0 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.170+0000 7eff3e200640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff1800b5e0 0x7eff1800daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3d800640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff18003ab0 0x7eff18003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3ce00640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18008900 0x7eff1800ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3e200640 1 -- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff18003ab0 msgr2=0x7eff18003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3e200640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff18003ab0 0x7eff18003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3e200640 1 -- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18008900 msgr2=0x7eff1800ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3e200640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18008900 0x7eff1800ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3e200640 1 -- 172.21.15.141:0/1528516400 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18003440 con 0x7eff1800b5e0 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3d800640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff18003ab0 0x7eff18003ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:05.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.171+0000 7eff3ce00640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18008900 0x7eff1800ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.172+0000 7eff3e200640 1 -- 172.21.15.141:0/1528516400 reap_dead start 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.172+0000 7eff3e200640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff1800b5e0 0x7eff1800daa0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7eff30007360 tx=0x7eff180036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce69ed875f2e63e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.172+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3002ccf0 con 0x7eff1800b5e0 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.172+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3002ce90 con 0x7eff1800b5e0 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.172+0000 7eff37e00640 1 -- 172.21.15.141:0/1528516400 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30003bd0 con 0x7eff1800b5e0 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.173+0000 7eff3ec00640 1 -- 172.21.15.141:0/1528516400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff1800b5e0 msgr2=0x7eff1800daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.173+0000 7eff3ec00640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff1800b5e0 0x7eff1800daa0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7eff30007360 tx=0x7eff180036f0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.173+0000 7eff3ec00640 1 -- 172.21.15.141:0/1528516400 shutdown_connections 2024-01-16T14:58:05.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.173+0000 7eff3ec00640 1 --2- 172.21.15.141:0/1528516400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff1800b5e0 0x7eff1800daa0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.173+0000 7eff3ec00640 1 -- 172.21.15.141:0/1528516400 >> 172.21.15.141:0/1528516400 conn(0x7eff3806a7d0 msgr2=0x7eff3806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.173+0000 7eff3ec00640 1 -- 172.21.15.141:0/1528516400 shutdown_connections 2024-01-16T14:58:05.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.173+0000 7eff3ec00640 1 -- 172.21.15.141:0/1528516400 wait complete. 2024-01-16T14:58:05.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.175+0000 7eff3ec00640 1 Processor -- start 2024-01-16T14:58:05.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.175+0000 7eff3ec00640 1 -- start start 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.175+0000 7eff3ec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff1800b5e0 0x7eff38154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38141a50 0x7eff3813af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38140d70 0x7eff38141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff38141760 con 0x7eff38141a50 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff38071b10 con 0x7eff1800b5e0 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3813b860 con 0x7eff38140d70 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38141a50 0x7eff3813af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38141a50 0x7eff3813af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45732/0 (socket says 172.21.15.141:45732) 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 -- 172.21.15.141:0/2939390491 learned_addr learned my addr 172.21.15.141:0/2939390491 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 -- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38140d70 msgr2=0x7eff38141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38140d70 0x7eff38141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 -- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff1800b5e0 msgr2=0x7eff38154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3d800640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff1800b5e0 0x7eff38154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff1800b5e0 0x7eff38154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.176+0000 7eff3ce00640 1 -- 172.21.15.141:0/2939390491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff381547a0 con 0x7eff38141a50 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.177+0000 7eff3ce00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38141a50 0x7eff3813af10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7eff2800d570 tx=0x7eff2800da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.177+0000 7eff35600640 1 -- 172.21.15.141:0/2939390491 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28005480 con 0x7eff38141a50 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.177+0000 7eff35600640 1 -- 172.21.15.141:0/2939390491 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff28003610 con 0x7eff38141a50 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.177+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff38150fa0 con 0x7eff38141a50 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.177+0000 7eff35600640 1 -- 172.21.15.141:0/2939390491 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff280039a0 con 0x7eff38141a50 2024-01-16T14:58:05.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.177+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3813bab0 con 0x7eff38141a50 2024-01-16T14:58:05.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.178+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff38067450 con 0x7eff38141a50 2024-01-16T14:58:05.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.179+0000 7eff35600640 1 -- 172.21.15.141:0/2939390491 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff28003b00 con 0x7eff38141a50 2024-01-16T14:58:05.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.180+0000 7eff35600640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2f0 0x7eff1006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.180+0000 7eff35600640 1 -- 172.21.15.141:0/2939390491 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7eff28089a90 con 0x7eff38141a50 2024-01-16T14:58:05.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.181+0000 7eff3d800640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2f0 0x7eff1006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.182+0000 7eff3d800640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2f0 0x7eff1006d7b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7eff20001e70 tx=0x7eff20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.185+0000 7eff35600640 1 -- 172.21.15.141:0/2939390491 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff28054920 con 0x7eff38141a50 2024-01-16T14:58:05.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.373+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff380b3660 con 0x7eff38141a50 2024-01-16T14:58:05.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.374+0000 7eff35600640 1 -- 172.21.15.141:0/2939390491 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7eff28001230 con 0x7eff38141a50 2024-01-16T14:58:05.356 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2f0 msgr2=0x7eff1006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2f0 0x7eff1006d7b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7eff20001e70 tx=0x7eff20009040 comp rx=0 tx=0).stop 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38141a50 msgr2=0x7eff3813af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38141a50 0x7eff3813af10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7eff2800d570 tx=0x7eff2800da30 comp rx=0 tx=0).stop 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 shutdown_connections 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff1006b2f0 0x7eff1006d7b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff38140d70 0x7eff38141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff38141a50 0x7eff3813af10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 --2- 172.21.15.141:0/2939390491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff1800b5e0 0x7eff38154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.377+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 >> 172.21.15.141:0/2939390491 conn(0x7eff3806a7d0 msgr2=0x7eff3814cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.378+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 shutdown_connections 2024-01-16T14:58:05.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.378+0000 7eff3ec00640 1 -- 172.21.15.141:0/2939390491 wait complete. 2024-01-16T14:58:05.376 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:58:05.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.592+0000 7fc516200640 1 Processor -- start 2024-01-16T14:58:05.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.592+0000 7fc516200640 1 -- start start 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.593+0000 7fc516200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc51013df80 0x7fc51015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.593+0000 7fc516200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc5100cf9a0 0x7fc5100cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc516200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51006c470 con 0x7fc51013e360 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc516200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51006c750 con 0x7fc5100cf9a0 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc516200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51006ca30 con 0x7fc51013df80 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc5100cf9a0 0x7fc5100cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc5100cf9a0 0x7fc5100cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45734/0 (socket says 172.21.15.141:45734) 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50fe00640 1 -- 172.21.15.141:0/4043697538 learned_addr learned my addr 172.21.15.141:0/4043697538 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50f400640 1 --2- 172.21.15.141:0/4043697538 >> v2:172.21.15.188:3300/0 conn(0x7fc51013df80 0x7fc51015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50fe00640 1 -- 172.21.15.141:0/4043697538 >> v2:172.21.15.188:3300/0 conn(0x7fc51013df80 msgr2=0x7fc51015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50fe00640 1 --2- 172.21.15.141:0/4043697538 >> v2:172.21.15.188:3300/0 conn(0x7fc51013df80 0x7fc51015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50fe00640 1 -- 172.21.15.141:0/4043697538 >> v1:172.21.15.141:6789/0 conn(0x7fc51013e360 legacy=0x7fc5100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.594+0000 7fc50fe00640 1 -- 172.21.15.141:0/4043697538 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51006cd10 con 0x7fc5100cf9a0 2024-01-16T14:58:05.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.596+0000 7fc50fe00640 1 --2- 172.21.15.141:0/4043697538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5100cf9a0 0x7fc5100cd840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc510072770 tx=0x7fc50802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5614ca7b85a11918 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.596+0000 7fc50ea00640 1 -- 172.21.15.141:0/4043697538 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50802cb20 con 0x7fc5100cf9a0 2024-01-16T14:58:05.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.596+0000 7fc50ea00640 1 -- 172.21.15.141:0/4043697538 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50802cc80 con 0x7fc5100cf9a0 2024-01-16T14:58:05.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.596+0000 7fc50ea00640 1 -- 172.21.15.141:0/4043697538 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5080343f0 con 0x7fc5100cf9a0 2024-01-16T14:58:05.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.596+0000 7fc516200640 1 -- 172.21.15.141:0/4043697538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5100cf9a0 msgr2=0x7fc5100cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.596+0000 7fc516200640 1 --2- 172.21.15.141:0/4043697538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5100cf9a0 0x7fc5100cd840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc510072770 tx=0x7fc50802bf80 comp rx=0 tx=0).stop 2024-01-16T14:58:05.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.597+0000 7fc516200640 1 -- 172.21.15.141:0/4043697538 shutdown_connections 2024-01-16T14:58:05.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.597+0000 7fc516200640 1 --2- 172.21.15.141:0/4043697538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5100cf9a0 0x7fc5100cd840 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.597+0000 7fc516200640 1 --2- 172.21.15.141:0/4043697538 >> v2:172.21.15.188:3300/0 conn(0x7fc51013df80 0x7fc51015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.597+0000 7fc516200640 1 -- 172.21.15.141:0/4043697538 >> 172.21.15.141:0/4043697538 conn(0x7fc51006a800 msgr2=0x7fc51006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.597+0000 7fc516200640 1 -- 172.21.15.141:0/4043697538 shutdown_connections 2024-01-16T14:58:05.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.597+0000 7fc516200640 1 -- 172.21.15.141:0/4043697538 wait complete. 2024-01-16T14:58:05.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.598+0000 7fc516200640 1 Processor -- start 2024-01-16T14:58:05.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.598+0000 7fc516200640 1 -- start start 2024-01-16T14:58:05.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.599+0000 7fc516200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5100cf9a0 0x7fc510133790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.599+0000 7fc516200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc51013df80 0x7fc51014cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.599+0000 7fc516200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc51013e360 0x7fc510152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.599+0000 7fc516200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51014ac50 con 0x7fc51013e360 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.599+0000 7fc516200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51014ef10 con 0x7fc51013df80 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.599+0000 7fc516200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51014e600 con 0x7fc5100cf9a0 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc51013e360 0x7fc510152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc51013e360 0x7fc510152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45736/0 (socket says 172.21.15.141:45736) 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 -- 172.21.15.141:0/3850302140 learned_addr learned my addr 172.21.15.141:0/3850302140 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc506800640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc51013df80 0x7fc51014cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50f400640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5100cf9a0 0x7fc510133790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 -- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5100cf9a0 msgr2=0x7fc510133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5100cf9a0 0x7fc510133790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 -- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc51013df80 msgr2=0x7fc51014cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc51013df80 0x7fc51014cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 -- 172.21.15.141:0/3850302140 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc508009b30 con 0x7fc51013e360 2024-01-16T14:58:05.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.600+0000 7fc50fe00640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc51013e360 0x7fc510152c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc508036040 tx=0x7fc508004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.601+0000 7fc507e00640 1 -- 172.21.15.141:0/3850302140 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508007e70 con 0x7fc51013e360 2024-01-16T14:58:05.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.601+0000 7fc507e00640 1 -- 172.21.15.141:0/3850302140 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc508009510 con 0x7fc51013e360 2024-01-16T14:58:05.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.601+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51014e8c0 con 0x7fc51013e360 2024-01-16T14:58:05.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.601+0000 7fc507e00640 1 -- 172.21.15.141:0/3850302140 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50803e690 con 0x7fc51013e360 2024-01-16T14:58:05.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.601+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51014d8b0 con 0x7fc51013e360 2024-01-16T14:58:05.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.602+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51014f0c0 con 0x7fc51013e360 2024-01-16T14:58:05.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.603+0000 7fc507e00640 1 -- 172.21.15.141:0/3850302140 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc50803e7f0 con 0x7fc51013e360 2024-01-16T14:58:05.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.604+0000 7fc507e00640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc4e806b2f0 0x7fc4e806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:05.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.604+0000 7fc507e00640 1 -- 172.21.15.141:0/3850302140 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fc5080ac790 con 0x7fc51013e360 2024-01-16T14:58:05.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.604+0000 7fc50f400640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc4e806b2f0 0x7fc4e806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:05.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.605+0000 7fc50f400640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc4e806b2f0 0x7fc4e806d7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc500006fd0 tx=0x7fc500008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:05.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.608+0000 7fc507e00640 1 -- 172.21.15.141:0/3850302140 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc51014f0c0 con 0x7fc51013e360 2024-01-16T14:58:05.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.811+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc51014f080 con 0x7fc51013e360 2024-01-16T14:58:05.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.812+0000 7fc507e00640 1 -- 172.21.15.141:0/3850302140 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc508077570 con 0x7fc51013e360 2024-01-16T14:58:05.793 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:58:05.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc4e806b2f0 msgr2=0x7fc4e806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc4e806b2f0 0x7fc4e806d7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc500006fd0 tx=0x7fc500008040 comp rx=0 tx=0).stop 2024-01-16T14:58:05.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc51013e360 msgr2=0x7fc510152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:05.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc51013e360 0x7fc510152c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc508036040 tx=0x7fc508004a40 comp rx=0 tx=0).stop 2024-01-16T14:58:05.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 shutdown_connections 2024-01-16T14:58:05.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc4e806b2f0 0x7fc4e806d7b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc51013e360 0x7fc510152c60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc51013df80 0x7fc51014cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 --2- 172.21.15.141:0/3850302140 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5100cf9a0 0x7fc510133790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:05.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 >> 172.21.15.141:0/3850302140 conn(0x7fc51006a800 msgr2=0x7fc51015d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:05.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 shutdown_connections 2024-01-16T14:58:05.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:05.815+0000 7fc516200640 1 -- 172.21.15.141:0/3850302140 wait complete. 2024-01-16T14:58:05.815 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:58:05.815 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:05.871 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:06.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.104+0000 7f42bd000640 1 Processor -- start 2024-01-16T14:58:06.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.105+0000 7f42bd000640 1 -- start start 2024-01-16T14:58:06.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.106+0000 7f42bd000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b8143c40 0x7f42b8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.106+0000 7f42bd000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f42b80ccc10 0x7f42b806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.106+0000 7f42bd000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b8070e50 con 0x7f42b80717d0 2024-01-16T14:58:06.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.106+0000 7f42bd000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b80723c0 con 0x7f42b80ccc10 2024-01-16T14:58:06.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.106+0000 7f42bd000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b8144020 con 0x7f42b8143c40 2024-01-16T14:58:06.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.107+0000 7f42b7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b8143c40 0x7f42b8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.107+0000 7f42b7e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f42b80ccc10 0x7f42b806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.107+0000 7f42ae800640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f42b80717d0 0x7f42b80cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:39142/0 (socket says 172.21.15.141:39142) 2024-01-16T14:58:06.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.107+0000 7f42ae800640 1 -- 172.21.15.141:0/4177404468 learned_addr learned my addr 172.21.15.141:0/4177404468 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.108+0000 7f42b6a00640 1 -- 172.21.15.141:0/4177404468 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723785051 0 0) 0x7f42b8070e50 con 0x7f42b80717d0 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.108+0000 7f42b6a00640 1 -- 172.21.15.141:0/4177404468 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4298003520 con 0x7f42b80717d0 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.108+0000 7f42b7e00640 1 -- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b8143c40 msgr2=0x7f42b8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.108+0000 7f42b7e00640 1 --2- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b8143c40 0x7f42b8071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.108+0000 7f42b7e00640 1 -- 172.21.15.141:0/4177404468 >> v1:172.21.15.188:6789/0 conn(0x7f42b80717d0 legacy=0x7f42b80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.108+0000 7f42b7e00640 1 -- 172.21.15.141:0/4177404468 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b8071e00 con 0x7f42b80ccc10 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.108+0000 7f42b7400640 1 --2- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b8143c40 0x7f42b8071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:06.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.109+0000 7f42b7e00640 1 --2- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b80ccc10 0x7f42b806d160 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009d90 tx=0x7f42b002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d888972b7cda63ac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.110+0000 7f42b6a00640 1 -- 172.21.15.141:0/4177404468 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b002cbd0 con 0x7f42b80ccc10 2024-01-16T14:58:06.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.110+0000 7f42b6a00640 1 -- 172.21.15.141:0/4177404468 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b002cd30 con 0x7f42b80ccc10 2024-01-16T14:58:06.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.110+0000 7f42b6a00640 1 -- 172.21.15.141:0/4177404468 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b002cea0 con 0x7f42b80ccc10 2024-01-16T14:58:06.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.110+0000 7f42bd000640 1 -- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b80ccc10 msgr2=0x7f42b806d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.111+0000 7f42bd000640 1 --2- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b80ccc10 0x7f42b806d160 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009d90 tx=0x7f42b002be20 comp rx=0 tx=0).stop 2024-01-16T14:58:06.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.111+0000 7f42bd000640 1 -- 172.21.15.141:0/4177404468 shutdown_connections 2024-01-16T14:58:06.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.111+0000 7f42bd000640 1 --2- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b80ccc10 0x7f42b806d160 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.111+0000 7f42bd000640 1 --2- 172.21.15.141:0/4177404468 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b8143c40 0x7f42b8071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.111+0000 7f42bd000640 1 -- 172.21.15.141:0/4177404468 >> 172.21.15.141:0/4177404468 conn(0x7f42b806a800 msgr2=0x7f42b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.111+0000 7f42bd000640 1 -- 172.21.15.141:0/4177404468 shutdown_connections 2024-01-16T14:58:06.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.111+0000 7f42bd000640 1 -- 172.21.15.141:0/4177404468 wait complete. 2024-01-16T14:58:06.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.113+0000 7f42bd000640 1 Processor -- start 2024-01-16T14:58:06.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.113+0000 7f42bd000640 1 -- start start 2024-01-16T14:58:06.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.114+0000 7f42bd000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b80717d0 0x7f42b812f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.114+0000 7f42bd000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b80ccc10 0x7f42b80ab520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.114+0000 7f42bd000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b8143c40 0x7f42b80ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.114+0000 7f42bd000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b812fcb0 con 0x7f42b80717d0 2024-01-16T14:58:06.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.114+0000 7f42bd000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80ada50 con 0x7f42b8143c40 2024-01-16T14:58:06.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.114+0000 7f42bd000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b8133020 con 0x7f42b80ccc10 2024-01-16T14:58:06.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b80717d0 0x7f42b812f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b80717d0 0x7f42b812f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45742/0 (socket says 172.21.15.141:45742) 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 -- 172.21.15.141:0/3666773703 learned_addr learned my addr 172.21.15.141:0/3666773703 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7e00640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b8143c40 0x7f42b80ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42ae800640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b80ccc10 0x7f42b80ab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 -- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b80ccc10 msgr2=0x7f42b80ab520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b80ccc10 0x7f42b80ab520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 -- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b8143c40 msgr2=0x7f42b80ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b8143c40 0x7f42b80ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7400640 1 -- 172.21.15.141:0/3666773703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a8009400 con 0x7f42b80717d0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.115+0000 7f42b7e00640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b8143c40 0x7f42b80ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.116+0000 7f42b7400640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b80717d0 0x7f42b812f680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f42a800e960 tx=0x7f42a800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.116+0000 7f42afe00640 1 -- 172.21.15.141:0/3666773703 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a800ca00 con 0x7f42b80717d0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.116+0000 7f42afe00640 1 -- 172.21.15.141:0/3666773703 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a800cb60 con 0x7f42b80717d0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.116+0000 7f42afe00640 1 -- 172.21.15.141:0/3666773703 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8013470 con 0x7f42b80717d0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.116+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b0009a70 con 0x7f42b80717d0 2024-01-16T14:58:06.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.116+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b80ace10 con 0x7f42b80717d0 2024-01-16T14:58:06.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.117+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b80adcc0 con 0x7f42b80717d0 2024-01-16T14:58:06.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.118+0000 7f42afe00640 1 -- 172.21.15.141:0/3666773703 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f42a80135d0 con 0x7f42b80717d0 2024-01-16T14:58:06.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.119+0000 7f42afe00640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006b2c0 0x7f429006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.119+0000 7f42afe00640 1 -- 172.21.15.141:0/3666773703 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f42a8089e30 con 0x7f42b80717d0 2024-01-16T14:58:06.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.120+0000 7f42ae800640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006b2c0 0x7f429006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.121+0000 7f42ae800640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006b2c0 0x7f429006d780 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f42b813f470 tx=0x7f42a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.123+0000 7f42afe00640 1 -- 172.21.15.141:0/3666773703 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42a8054cb0 con 0x7f42b80717d0 2024-01-16T14:58:06.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.315+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42b8067450 con 0x7f42b80717d0 2024-01-16T14:58:06.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.318+0000 7f42afe00640 1 -- 172.21.15.141:0/3666773703 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7f42a8017090 con 0x7f42b80717d0 2024-01-16T14:58:06.300 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:06.300 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:06.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.320+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006b2c0 msgr2=0x7f429006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.320+0000 7f42bd000640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006b2c0 0x7f429006d780 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f42b813f470 tx=0x7f42a0009040 comp rx=0 tx=0).stop 2024-01-16T14:58:06.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.320+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b80717d0 msgr2=0x7f42b812f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b80717d0 0x7f42b812f680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f42a800e960 tx=0x7f42a800ee20 comp rx=0 tx=0).stop 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 shutdown_connections 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006b2c0 0x7f429006d780 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42b8143c40 0x7f42b80ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b80ccc10 0x7f42b80ab520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 --2- 172.21.15.141:0/3666773703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b80717d0 0x7f42b812f680 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 >> 172.21.15.141:0/3666773703 conn(0x7f42b806a800 msgr2=0x7f42b806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 shutdown_connections 2024-01-16T14:58:06.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.321+0000 7f42bd000640 1 -- 172.21.15.141:0/3666773703 wait complete. 2024-01-16T14:58:06.322 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:58:06.323 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:58:06.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.546+0000 7f7be6600640 1 Processor -- start 2024-01-16T14:58:06.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.546+0000 7f7be6600640 1 -- start start 2024-01-16T14:58:06.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.547+0000 7f7be6600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be0143ae0 0x7f7be00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.547+0000 7f7be6600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7be015e3b0 0x7f7be00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.547+0000 7f7be6600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be00cfc30 con 0x7f7be00cf7d0 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.547+0000 7f7be6600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be00b1ee0 con 0x7f7be015e3b0 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.547+0000 7f7be6600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00b2780 con 0x7f7be0143ae0 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7be015e3b0 0x7f7be00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7be015e3b0 0x7f7be00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45748/0 (socket says 172.21.15.141:45748) 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdfe00640 1 -- 172.21.15.141:0/1501613151 learned_addr learned my addr 172.21.15.141:0/1501613151 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdf400640 1 --2- 172.21.15.141:0/1501613151 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be0143ae0 0x7f7be00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdfe00640 1 -- 172.21.15.141:0/1501613151 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be0143ae0 msgr2=0x7f7be00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdfe00640 1 --2- 172.21.15.141:0/1501613151 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be0143ae0 0x7f7be00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdfe00640 1 -- 172.21.15.141:0/1501613151 >> v1:172.21.15.188:6789/0 conn(0x7f7be00cf7d0 legacy=0x7f7be015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:06.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.548+0000 7f7bdfe00640 1 -- 172.21.15.141:0/1501613151 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be015e990 con 0x7f7be015e3b0 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.549+0000 7f7bdfe00640 1 --2- 172.21.15.141:0/1501613151 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be015e3b0 0x7f7be00702c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009e50 tx=0x7f7bd802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d0ed1543b6ef1b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.549+0000 7f7bdea00640 1 -- 172.21.15.141:0/1501613151 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd802ec80 con 0x7f7be015e3b0 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7bdea00640 1 -- 172.21.15.141:0/1501613151 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd802ede0 con 0x7f7be015e3b0 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7bdea00640 1 -- 172.21.15.141:0/1501613151 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8036610 con 0x7f7be015e3b0 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 -- 172.21.15.141:0/1501613151 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be015e3b0 msgr2=0x7f7be00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 --2- 172.21.15.141:0/1501613151 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be015e3b0 0x7f7be00702c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009e50 tx=0x7f7bd802e0e0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 -- 172.21.15.141:0/1501613151 shutdown_connections 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 --2- 172.21.15.141:0/1501613151 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be015e3b0 0x7f7be00702c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 --2- 172.21.15.141:0/1501613151 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be0143ae0 0x7f7be00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 -- 172.21.15.141:0/1501613151 >> 172.21.15.141:0/1501613151 conn(0x7f7be006a800 msgr2=0x7f7be006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 -- 172.21.15.141:0/1501613151 shutdown_connections 2024-01-16T14:58:06.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.550+0000 7f7be6600640 1 -- 172.21.15.141:0/1501613151 wait complete. 2024-01-16T14:58:06.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.552+0000 7f7be6600640 1 Processor -- start 2024-01-16T14:58:06.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.552+0000 7f7be6600640 1 -- start start 2024-01-16T14:58:06.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7be6600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be00cf7d0 0x7f7be013b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7be6600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be0143ae0 0x7f7be015b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7be6600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7be015e3b0 0x7f7be015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7be6600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be013d100 con 0x7f7be0143ae0 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7be6600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be01f2e00 con 0x7f7be015e3b0 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7be6600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be013f880 con 0x7f7be00cf7d0 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7bd7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be0143ae0 0x7f7be015b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7bd7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be0143ae0 0x7f7be015b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45760/0 (socket says 172.21.15.141:45760) 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.553+0000 7f7bd7e00640 1 -- 172.21.15.141:0/2662927295 learned_addr learned my addr 172.21.15.141:0/2662927295 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bdf400640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be00cf7d0 0x7f7be013b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bdfe00640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7be015e3b0 0x7f7be015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7e00640 1 -- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be00cf7d0 msgr2=0x7f7be013b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7e00640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be00cf7d0 0x7f7be013b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7e00640 1 -- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7be015e3b0 msgr2=0x7f7be015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7e00640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7be015e3b0 0x7f7be015bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7e00640 1 -- 172.21.15.141:0/2662927295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8009b30 con 0x7f7be0143ae0 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7e00640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be0143ae0 0x7f7be015b870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800c430 tx=0x7f7bc800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7400640 1 -- 172.21.15.141:0/2662927295 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8011790 con 0x7f7be0143ae0 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7400640 1 -- 172.21.15.141:0/2662927295 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc80118f0 con 0x7f7be0143ae0 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7bd7400640 1 -- 172.21.15.141:0/2662927295 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8011ca0 con 0x7f7be0143ae0 2024-01-16T14:58:06.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.554+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be013ed20 con 0x7f7be0143ae0 2024-01-16T14:58:06.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.555+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be013f2e0 con 0x7f7be0143ae0 2024-01-16T14:58:06.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.556+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be013faa0 con 0x7f7be0143ae0 2024-01-16T14:58:06.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.557+0000 7f7bd7400640 1 -- 172.21.15.141:0/2662927295 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7bc801c460 con 0x7f7be0143ae0 2024-01-16T14:58:06.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.557+0000 7f7bd7400640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bb806b2c0 0x7f7bb806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.557+0000 7f7bd7400640 1 -- 172.21.15.141:0/2662927295 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f7bc8089ca0 con 0x7f7be0143ae0 2024-01-16T14:58:06.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.558+0000 7f7bdf400640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bb806b2c0 0x7f7bb806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.559+0000 7f7bdf400640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bb806b2c0 0x7f7bb806d780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0006fd0 tx=0x7f7bd0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.562+0000 7f7bd7400640 1 -- 172.21.15.141:0/2662927295 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7be013faa0 con 0x7f7be0143ae0 2024-01-16T14:58:06.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.753+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7be013fa40 con 0x7f7be0143ae0 2024-01-16T14:58:06.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.754+0000 7f7bd7400640 1 -- 172.21.15.141:0/2662927295 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7bc80549f0 con 0x7f7be0143ae0 2024-01-16T14:58:06.735 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bb806b2c0 msgr2=0x7f7bb806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bb806b2c0 0x7f7bb806d780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0006fd0 tx=0x7f7bd0008040 comp rx=0 tx=0).stop 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be0143ae0 msgr2=0x7f7be015b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be0143ae0 0x7f7be015b870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800c430 tx=0x7f7bc800c8f0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 shutdown_connections 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bb806b2c0 0x7f7bb806d780 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7be015e3b0 0x7f7be015bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7be0143ae0 0x7f7be015b870 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 --2- 172.21.15.141:0/2662927295 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7be00cf7d0 0x7f7be013b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 >> 172.21.15.141:0/2662927295 conn(0x7f7be006a800 msgr2=0x7f7be006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.756+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 shutdown_connections 2024-01-16T14:58:06.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.757+0000 7f7be6600640 1 -- 172.21.15.141:0/2662927295 wait complete. 2024-01-16T14:58:06.757 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:58:06.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.982+0000 7f51e6000640 1 Processor -- start 2024-01-16T14:58:06.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.982+0000 7f51e6000640 1 -- start start 2024-01-16T14:58:06.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.984+0000 7f51e6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e00ccb50 0x7f51e0070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.984+0000 7f51e6000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f51e015bf10 0x7f51e0143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.984+0000 7f51e6000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e015f4e0 con 0x7f51e00adf20 2024-01-16T14:58:06.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.984+0000 7f51e6000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e015f970 con 0x7f51e015bf10 2024-01-16T14:58:06.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.984+0000 7f51e6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e015c3c0 con 0x7f51e00ccb50 2024-01-16T14:58:06.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e00ccb50 0x7f51e0070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51e5600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f51e015bf10 0x7f51e0143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51e4c00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f51e00adf20 0x7f51e00cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:39152/0 (socket says 172.21.15.141:39152) 2024-01-16T14:58:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51e5600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f51e015bf10 0x7f51e0143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36088/0 (socket says 172.21.15.141:36088) 2024-01-16T14:58:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e00ccb50 0x7f51e0070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33772/0 (socket says 172.21.15.141:33772) 2024-01-16T14:58:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51e4c00640 1 -- 172.21.15.141:0/4040499541 learned_addr learned my addr 172.21.15.141:0/4040499541 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51e5600640 1 -- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e00ccb50 msgr2=0x7f51e0070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.985+0000 7f51e5600640 1 --2- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e00ccb50 0x7f51e0070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.986+0000 7f51e5600640 1 -- 172.21.15.141:0/4040499541 >> v1:172.21.15.188:6789/0 conn(0x7f51e00adf20 legacy=0x7f51e00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.986+0000 7f51e5600640 1 -- 172.21.15.141:0/4040499541 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e0071130 con 0x7f51e015bf10 2024-01-16T14:58:06.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.986+0000 7f51d7e00640 1 --2- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e00ccb50 0x7f51e0070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:06.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.987+0000 7f51e5600640 1 --2- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e015bf10 0x7f51e0143d30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009d90 tx=0x7f51d802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e521db5e907bae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.987+0000 7f51dfe00640 1 -- 172.21.15.141:0/4040499541 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d802cbc0 con 0x7f51e015bf10 2024-01-16T14:58:06.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51dfe00640 1 -- 172.21.15.141:0/4040499541 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d802cd20 con 0x7f51e015bf10 2024-01-16T14:58:06.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51dfe00640 1 -- 172.21.15.141:0/4040499541 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d8034660 con 0x7f51e015bf10 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 -- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e015bf10 msgr2=0x7f51e0143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 --2- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e015bf10 0x7f51e0143d30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009d90 tx=0x7f51d802be10 comp rx=0 tx=0).stop 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 -- 172.21.15.141:0/4040499541 shutdown_connections 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 --2- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e015bf10 0x7f51e0143d30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 --2- 172.21.15.141:0/4040499541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e00ccb50 0x7f51e0070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 -- 172.21.15.141:0/4040499541 >> 172.21.15.141:0/4040499541 conn(0x7f51e006a7d0 msgr2=0x7f51e006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 -- 172.21.15.141:0/4040499541 shutdown_connections 2024-01-16T14:58:06.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.988+0000 7f51e6000640 1 -- 172.21.15.141:0/4040499541 wait complete. 2024-01-16T14:58:06.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.990+0000 7f51e6000640 1 Processor -- start 2024-01-16T14:58:06.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.990+0000 7f51e6000640 1 -- start start 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.991+0000 7f51e6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e00adf20 0x7f51e01410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.991+0000 7f51e6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51e00ccb50 0x7f51e01f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.991+0000 7f51e6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e015bf10 0x7f51e01f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.991+0000 7f51e6000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e013f5e0 con 0x7f51e00ccb50 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.991+0000 7f51e6000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01f6a60 con 0x7f51e00adf20 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.991+0000 7f51e6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01416c0 con 0x7f51e015bf10 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.991+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51e00ccb50 0x7f51e01f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51e00ccb50 0x7f51e01f6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45772/0 (socket says 172.21.15.141:45772) 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 -- 172.21.15.141:0/2565234341 learned_addr learned my addr 172.21.15.141:0/2565234341 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51e4c00640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e00adf20 0x7f51e01410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51e5600640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e015bf10 0x7f51e01f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 -- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e015bf10 msgr2=0x7f51e01f36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e015bf10 0x7f51e01f36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 -- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e00adf20 msgr2=0x7f51e01410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e00adf20 0x7f51e01410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 -- 172.21.15.141:0/2565234341 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c8009400 con 0x7f51e00ccb50 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.992+0000 7f51d7e00640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51e00ccb50 0x7f51e01f6520 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f51c800e9e0 tx=0x7f51c800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.993+0000 7f51dd600640 1 -- 172.21.15.141:0/2565234341 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800ca30 con 0x7f51e00ccb50 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.993+0000 7f51dd600640 1 -- 172.21.15.141:0/2565234341 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c800cb90 con 0x7f51e00ccb50 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.993+0000 7f51dd600640 1 -- 172.21.15.141:0/2565234341 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8013470 con 0x7f51e00ccb50 2024-01-16T14:58:06.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.993+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d8009a70 con 0x7f51e00ccb50 2024-01-16T14:58:06.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.993+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e0141c80 con 0x7f51e00ccb50 2024-01-16T14:58:06.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.994+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e0067450 con 0x7f51e00ccb50 2024-01-16T14:58:06.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.995+0000 7f51dd600640 1 -- 172.21.15.141:0/2565234341 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f51c80135d0 con 0x7f51e00ccb50 2024-01-16T14:58:06.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.996+0000 7f51dd600640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51b806b2c0 0x7f51b806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:06.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.996+0000 7f51dd600640 1 -- 172.21.15.141:0/2565234341 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f51c8089b20 con 0x7f51e00ccb50 2024-01-16T14:58:06.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.996+0000 7f51e4c00640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51b806b2c0 0x7f51b806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:06.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:06.997+0000 7f51e4c00640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51b806b2c0 0x7f51b806d780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f51d0005880 tx=0x7f51d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:06.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.000+0000 7f51dd600640 1 -- 172.21.15.141:0/2565234341 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51c80549a0 con 0x7f51e00ccb50 2024-01-16T14:58:07.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.191+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f51e01f3c20 con 0x7f51e00ccb50 2024-01-16T14:58:07.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.192+0000 7f51dd600640 1 -- 172.21.15.141:0/2565234341 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f51c8057fc0 con 0x7f51e00ccb50 2024-01-16T14:58:07.173 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:58:07.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.194+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51b806b2c0 msgr2=0x7f51b806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.194+0000 7f51e6000640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51b806b2c0 0x7f51b806d780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f51d0005880 tx=0x7f51d0009040 comp rx=0 tx=0).stop 2024-01-16T14:58:07.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.194+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51e00ccb50 msgr2=0x7f51e01f6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.194+0000 7f51e6000640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51e00ccb50 0x7f51e01f6520 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f51c800e9e0 tx=0x7f51c800eea0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.194+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 shutdown_connections 2024-01-16T14:58:07.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.195+0000 7f51e6000640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f51b806b2c0 0x7f51b806d780 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.195+0000 7f51e6000640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51e015bf10 0x7f51e01f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.195+0000 7f51e6000640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51e00ccb50 0x7f51e01f6520 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.195+0000 7f51e6000640 1 --2- 172.21.15.141:0/2565234341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51e00adf20 0x7f51e01410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.195+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 >> 172.21.15.141:0/2565234341 conn(0x7f51e006a7d0 msgr2=0x7f51e0071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.195+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 shutdown_connections 2024-01-16T14:58:07.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.195+0000 7f51e6000640 1 -- 172.21.15.141:0/2565234341 wait complete. 2024-01-16T14:58:07.195 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:58:07.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.423+0000 7f7406c00640 1 Processor -- start 2024-01-16T14:58:07.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.423+0000 7f7406c00640 1 -- start start 2024-01-16T14:58:07.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.425+0000 7f7406c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f740013e360 0x7f74000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.425+0000 7f7406c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740015c5f0 con 0x7f740013df80 2024-01-16T14:58:07.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.425+0000 7f7406c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740006c470 con 0x7f740015f9c0 2024-01-16T14:58:07.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.425+0000 7f7406c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740006c750 con 0x7f740013e360 2024-01-16T14:58:07.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.425+0000 7f73ff400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f740013e360 0x7f74000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.425+0000 7f73ffe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f740013df80 0x7f740015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57016/0 (socket says 172.21.15.141:57016) 2024-01-16T14:58:07.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.425+0000 7f73ffe00640 1 -- 172.21.15.141:0/2605496535 learned_addr learned my addr 172.21.15.141:0/2605496535 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:07.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.426+0000 7f73ff400640 1 -- 172.21.15.141:0/2605496535 >> v1:172.21.15.188:6789/0 conn(0x7f740015f9c0 legacy=0x7f74000cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.426+0000 7f73ff400640 1 -- 172.21.15.141:0/2605496535 >> v1:172.21.15.141:6789/0 conn(0x7f740013df80 legacy=0x7f740015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.426+0000 7f73ff400640 1 -- 172.21.15.141:0/2605496535 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740006ca30 con 0x7f740013e360 2024-01-16T14:58:07.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.427+0000 7f73ff400640 1 --2- 172.21.15.141:0/2605496535 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740013e360 0x7f74000cf290 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009d90 tx=0x7f73e802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f9f5d91911d224 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.427+0000 7f73fea00640 1 -- 172.21.15.141:0/2605496535 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e802cc60 con 0x7f740013e360 2024-01-16T14:58:07.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.427+0000 7f73fea00640 1 -- 172.21.15.141:0/2605496535 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e802cdc0 con 0x7f740013e360 2024-01-16T14:58:07.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.427+0000 7f73fea00640 1 -- 172.21.15.141:0/2605496535 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8034660 con 0x7f740013e360 2024-01-16T14:58:07.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.428+0000 7f7406c00640 1 -- 172.21.15.141:0/2605496535 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740013e360 msgr2=0x7f74000cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.428+0000 7f7406c00640 1 --2- 172.21.15.141:0/2605496535 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740013e360 0x7f74000cf290 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009d90 tx=0x7f73e802beb0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.428+0000 7f7406c00640 1 -- 172.21.15.141:0/2605496535 shutdown_connections 2024-01-16T14:58:07.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.428+0000 7f7406c00640 1 --2- 172.21.15.141:0/2605496535 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740013e360 0x7f74000cf290 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.428+0000 7f7406c00640 1 -- 172.21.15.141:0/2605496535 >> 172.21.15.141:0/2605496535 conn(0x7f740006a800 msgr2=0x7f740006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.428+0000 7f7406c00640 1 -- 172.21.15.141:0/2605496535 shutdown_connections 2024-01-16T14:58:07.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.428+0000 7f7406c00640 1 -- 172.21.15.141:0/2605496535 wait complete. 2024-01-16T14:58:07.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.429+0000 7f7406c00640 1 Processor -- start 2024-01-16T14:58:07.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.430+0000 7f7406c00640 1 -- start start 2024-01-16T14:58:07.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.430+0000 7f7406c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f740013df80 0x7f74001e7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.430+0000 7f7406c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f740013e360 0x7f74001532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.430+0000 7f7406c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740015f9c0 0x7f7400153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.430+0000 7f7406c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74001e6fc0 con 0x7f740015f9c0 2024-01-16T14:58:07.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.430+0000 7f7406c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74001e63b0 con 0x7f740013df80 2024-01-16T14:58:07.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.430+0000 7f7406c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74001e6690 con 0x7f740013e360 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740015f9c0 0x7f7400153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740015f9c0 0x7f7400153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45800/0 (socket says 172.21.15.141:45800) 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 -- 172.21.15.141:0/1684263661 learned_addr learned my addr 172.21.15.141:0/1684263661 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f73ff400640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f740013e360 0x7f74001532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f73ffe00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f740013df80 0x7f74001e7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 -- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f740013e360 msgr2=0x7f74001532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f740013e360 0x7f74001532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 -- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f740013df80 msgr2=0x7f74001e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f740013df80 0x7f74001e7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.431+0000 7f7404c00640 1 -- 172.21.15.141:0/1684263661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8009a70 con 0x7f740015f9c0 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.432+0000 7f7404c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740015f9c0 0x7f7400153ca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f73f800c3e0 tx=0x7f73f800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.432+0000 7f73f7e00640 1 -- 172.21.15.141:0/1684263661 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8011790 con 0x7f740015f9c0 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.432+0000 7f73f7e00640 1 -- 172.21.15.141:0/1684263661 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f80118f0 con 0x7f740015f9c0 2024-01-16T14:58:07.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.432+0000 7f73f7e00640 1 -- 172.21.15.141:0/1684263661 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8011ce0 con 0x7f740015f9c0 2024-01-16T14:58:07.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.432+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74001e6970 con 0x7f740015f9c0 2024-01-16T14:58:07.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.432+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74001541e0 con 0x7f740015f9c0 2024-01-16T14:58:07.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.433+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74000b09e0 con 0x7f740015f9c0 2024-01-16T14:58:07.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.434+0000 7f73f7e00640 1 -- 172.21.15.141:0/1684263661 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f73f801c460 con 0x7f740015f9c0 2024-01-16T14:58:07.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.435+0000 7f73f7e00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f73d806b2f0 0x7f73d806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.435+0000 7f73f7e00640 1 -- 172.21.15.141:0/1684263661 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f73f8089b60 con 0x7f740015f9c0 2024-01-16T14:58:07.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.435+0000 7f73ffe00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f73d806b2f0 0x7f73d806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.436+0000 7f73ffe00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f73d806b2f0 0x7f73d806d7b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000f60 tx=0x7f73f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.439+0000 7f73f7e00640 1 -- 172.21.15.141:0/1684263661 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73f80548b0 con 0x7f740015f9c0 2024-01-16T14:58:07.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.625+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7400154370 con 0x7f740015f9c0 2024-01-16T14:58:07.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.626+0000 7f73f7e00640 1 -- 172.21.15.141:0/1684263661 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7400154370 con 0x7f740015f9c0 2024-01-16T14:58:07.608 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:58:07.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f73d806b2f0 msgr2=0x7f73d806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f73d806b2f0 0x7f73d806d7b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000f60 tx=0x7f73f0009040 comp rx=0 tx=0).stop 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740015f9c0 msgr2=0x7f7400153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740015f9c0 0x7f7400153ca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f73f800c3e0 tx=0x7f73f800c8a0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 shutdown_connections 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f73d806b2f0 0x7f73d806d7b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f740015f9c0 0x7f7400153ca0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f740013e360 0x7f74001532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 --2- 172.21.15.141:0/1684263661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f740013df80 0x7f74001e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 >> 172.21.15.141:0/1684263661 conn(0x7f740006a800 msgr2=0x7f740006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 shutdown_connections 2024-01-16T14:58:07.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.629+0000 7f7406c00640 1 -- 172.21.15.141:0/1684263661 wait complete. 2024-01-16T14:58:07.628 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:58:07.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.842+0000 7f594e600640 1 Processor -- start 2024-01-16T14:58:07.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.842+0000 7f594e600640 1 -- start start 2024-01-16T14:58:07.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.844+0000 7f594e600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f59480ae5a0 0x7f594815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.844+0000 7f594e600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5948070c40 con 0x7f59480b3ae0 2024-01-16T14:58:07.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.844+0000 7f594e600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594806c470 con 0x7f59480cf290 2024-01-16T14:58:07.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.844+0000 7f594e600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5948071450 con 0x7f59480ae5a0 2024-01-16T14:58:07.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.844+0000 7f5946a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f59480b3ae0 0x7f594806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57032/0 (socket says 172.21.15.141:57032) 2024-01-16T14:58:07.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.844+0000 7f5946a00640 1 -- 172.21.15.141:0/2183001159 learned_addr learned my addr 172.21.15.141:0/2183001159 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166266478 0 0) 0x7f5948070c40 con 0x7f59480b3ae0 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928003520 con 0x7f59480b3ae0 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5947400640 1 --2- 172.21.15.141:0/2183001159 >> v2:172.21.15.188:3300/0 conn(0x7f59480ae5a0 0x7f594815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160003830 0 0) 0x7f5928003520 con 0x7f59480b3ae0 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 >> v2:172.21.15.188:3300/0 conn(0x7f59480ae5a0 msgr2=0x7f594815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 --2- 172.21.15.141:0/2183001159 >> v2:172.21.15.188:3300/0 conn(0x7f59480ae5a0 0x7f594815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 >> v1:172.21.15.188:6789/0 conn(0x7f59480cf290 legacy=0x7f59480b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594806c750 con 0x7f59480b3ae0 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f5930001670 con 0x7f59480b3ae0 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 >> v1:172.21.15.141:6789/0 conn(0x7f59480b3ae0 legacy=0x7f594806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f5928003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59280047c0 0x7f5928004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5928005180 0x7f592800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5948071450 con 0x7f5928003970 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592800c0b0 con 0x7f5928005180 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592800c270 con 0x7f59280047c0 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.845+0000 7f5947400640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f5928003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947e00640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5928005180 0x7f592800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5946a00640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59280047c0 0x7f5928004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947400640 1 -- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59280047c0 msgr2=0x7f5928004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947400640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59280047c0 0x7f5928004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947400640 1 -- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5928005180 msgr2=0x7f592800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:07.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947400640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5928005180 0x7f592800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947400640 1 -- 172.21.15.141:0/2183001159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928003520 con 0x7f5928003970 2024-01-16T14:58:07.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947e00640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5928005180 0x7f592800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:07.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.846+0000 7f5947e00640 1 -- 172.21.15.141:0/2183001159 reap_dead start 2024-01-16T14:58:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.847+0000 7f5947400640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f5928003d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5938007360 tx=0x7f59280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9b7b14bef0d749 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593802ccd0 con 0x7f5928003970 2024-01-16T14:58:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593802ce30 con 0x7f5928003970 2024-01-16T14:58:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f5946000640 1 -- 172.21.15.141:0/2183001159 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938034660 con 0x7f5928003970 2024-01-16T14:58:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f594e600640 1 -- 172.21.15.141:0/2183001159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 msgr2=0x7f5928003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f594e600640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f5928003d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5938007360 tx=0x7f59280057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f594e600640 1 -- 172.21.15.141:0/2183001159 shutdown_connections 2024-01-16T14:58:07.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f594e600640 1 --2- 172.21.15.141:0/2183001159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f5928003d90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f594e600640 1 -- 172.21.15.141:0/2183001159 >> 172.21.15.141:0/2183001159 conn(0x7f594806a800 msgr2=0x7f594806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:07.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f594e600640 1 -- 172.21.15.141:0/2183001159 shutdown_connections 2024-01-16T14:58:07.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.848+0000 7f594e600640 1 -- 172.21.15.141:0/2183001159 wait complete. 2024-01-16T14:58:07.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.850+0000 7f594e600640 1 Processor -- start 2024-01-16T14:58:07.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.850+0000 7f594e600640 1 -- start start 2024-01-16T14:58:07.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f594e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f594815c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f594e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594814fa10 0x7f59481f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f594e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59481f6e70 0x7f59481f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f594e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594815d050 con 0x7f5928003970 2024-01-16T14:58:07.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f594e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59480cc650 con 0x7f594814fa10 2024-01-16T14:58:07.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f594e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5948150830 con 0x7f59481f6e70 2024-01-16T14:58:07.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f5947400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f594815c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f5947400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f594815c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45832/0 (socket says 172.21.15.141:45832) 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.851+0000 7f5947400640 1 -- 172.21.15.141:0/1260820551 learned_addr learned my addr 172.21.15.141:0/1260820551 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5947e00640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59481f6e70 0x7f59481f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5947400640 1 -- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59481f6e70 msgr2=0x7f59481f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5947400640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59481f6e70 0x7f59481f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5947400640 1 -- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594814fa10 msgr2=0x7f59481f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5946a00640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594814fa10 0x7f59481f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5947400640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594814fa10 0x7f59481f6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5947400640 1 -- 172.21.15.141:0/1260820551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594814fe90 con 0x7f5928003970 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f5947400640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f594815c3e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5938037040 tx=0x7f5938004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f593f400640 1 -- 172.21.15.141:0/1260820551 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938003b90 con 0x7f5928003970 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f593f400640 1 -- 172.21.15.141:0/1260820551 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5938003cf0 con 0x7f5928003970 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f593f400640 1 -- 172.21.15.141:0/1260820551 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938003ea0 con 0x7f5928003970 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59480cb9d0 con 0x7f5928003970 2024-01-16T14:58:07.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.852+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59480cbf90 con 0x7f5928003970 2024-01-16T14:58:07.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.853+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59480675d0 con 0x7f5928003970 2024-01-16T14:58:07.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.854+0000 7f593f400640 1 -- 172.21.15.141:0/1260820551 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f593803b040 con 0x7f5928003970 2024-01-16T14:58:07.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.855+0000 7f593f400640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f592006b2f0 0x7f592006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:07.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.855+0000 7f593f400640 1 -- 172.21.15.141:0/1260820551 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f59380ab940 con 0x7f5928003970 2024-01-16T14:58:07.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.856+0000 7f5946a00640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f592006b2f0 0x7f592006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:07.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.857+0000 7f5946a00640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f592006b2f0 0x7f592006d7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5930000f60 tx=0x7f5930009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:07.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:07.859+0000 7f593f400640 1 -- 172.21.15.141:0/1260820551 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59380768f0 con 0x7f5928003970 2024-01-16T14:58:08.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.046+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f594815cb50 con 0x7f5928003970 2024-01-16T14:58:08.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.047+0000 7f593f400640 1 -- 172.21.15.141:0/1260820551 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5938039070 con 0x7f5928003970 2024-01-16T14:58:08.029 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T14:58:08.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.049+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f592006b2f0 msgr2=0x7f592006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.049+0000 7f594e600640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f592006b2f0 0x7f592006d7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5930000f60 tx=0x7f5930009040 comp rx=0 tx=0).stop 2024-01-16T14:58:08.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.049+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 msgr2=0x7f594815c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.049+0000 7f594e600640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f594815c3e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5938037040 tx=0x7f5938004ab0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.049+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 shutdown_connections 2024-01-16T14:58:08.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.049+0000 7f594e600640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f592006b2f0 0x7f592006d7b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.050+0000 7f594e600640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59481f6e70 0x7f59481f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.050+0000 7f594e600640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594814fa10 0x7f59481f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.050+0000 7f594e600640 1 --2- 172.21.15.141:0/1260820551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5928003970 0x7f594815c3e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.050+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 >> 172.21.15.141:0/1260820551 conn(0x7f594806a800 msgr2=0x7f59480b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.050+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 shutdown_connections 2024-01-16T14:58:08.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.050+0000 7f594e600640 1 -- 172.21.15.141:0/1260820551 wait complete. 2024-01-16T14:58:08.048 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T14:58:08.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:08.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-16T14:58:08.144 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:58:08.144 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:58:08.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:08.145 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T14:58:08.196 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T14:58:08.196 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:08.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.420+0000 7f2e40000640 1 Processor -- start 2024-01-16T14:58:08.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.420+0000 7f2e40000640 1 -- start start 2024-01-16T14:58:08.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e40000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2e3813e360 0x7f2e380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e40000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 0x7f2e380cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e40000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3815c5f0 con 0x7f2e3813df80 2024-01-16T14:58:08.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e40000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3806c470 con 0x7f2e3813e360 2024-01-16T14:58:08.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e40000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3806c750 con 0x7f2e3815f9c0 2024-01-16T14:58:08.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e3d600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2e3813df80 0x7f2e3815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57042/0 (socket says 172.21.15.141:57042) 2024-01-16T14:58:08.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e3d600640 1 -- 172.21.15.141:0/3624445415 learned_addr learned my addr 172.21.15.141:0/3624445415 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670048971 0 0) 0x7f2e3815c5f0 con 0x7f2e3813df80 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e18003520 con 0x7f2e3813df80 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e3e000640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 0x7f2e380cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.422+0000 7f2e3cc00640 1 --2- 172.21.15.141:0/3624445415 >> v2:172.21.15.188:3300/0 conn(0x7f2e3813e360 0x7f2e380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2900343660 0 0) 0x7f2e18003520 con 0x7f2e3813df80 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 msgr2=0x7f2e380cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 0x7f2e380cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 >> v2:172.21.15.188:3300/0 conn(0x7f2e3813e360 msgr2=0x7f2e380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 --2- 172.21.15.141:0/3624445415 >> v2:172.21.15.188:3300/0 conn(0x7f2e3813e360 0x7f2e380cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3806ca30 con 0x7f2e3813df80 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2e28001690 con 0x7f2e3813df80 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 >> v1:172.21.15.141:6789/0 conn(0x7f2e3813df80 legacy=0x7f2e3815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e180047c0 0x7f2e18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 0x7f2e18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3806c470 con 0x7f2e18003970 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3806c750 con 0x7f2e180047c0 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1800ba80 con 0x7f2e3815f9c0 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e3d600640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.423+0000 7f2e3cc00640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e180047c0 0x7f2e18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.424+0000 7f2e3e000640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 0x7f2e18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.424+0000 7f2e3d600640 1 -- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 msgr2=0x7f2e18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.424+0000 7f2e3d600640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815f9c0 0x7f2e18005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.424+0000 7f2e3d600640 1 -- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e180047c0 msgr2=0x7f2e18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.424+0000 7f2e3d600640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e180047c0 0x7f2e18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.424+0000 7f2e3d600640 1 -- 172.21.15.141:0/3624445415 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18003520 con 0x7f2e18003970 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.424+0000 7f2e3cc00640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e180047c0 0x7f2e18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.425+0000 7f2e3d600640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e18003d90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2e3815c780 tx=0x7f2e180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cece27dd46014326 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.425+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28005ad0 con 0x7f2e18003970 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.425+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e28005c30 con 0x7f2e18003970 2024-01-16T14:58:08.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.425+0000 7f2e37e00640 1 -- 172.21.15.141:0/3624445415 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2802c7c0 con 0x7f2e18003970 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.425+0000 7f2e40000640 1 -- 172.21.15.141:0/3624445415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 msgr2=0x7f2e18003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.425+0000 7f2e40000640 1 --2- 172.21.15.141:0/3624445415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e18003d90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2e3815c780 tx=0x7f2e180057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.425+0000 7f2e3e000640 1 -- 172.21.15.141:0/3624445415 reap_dead start 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.426+0000 7f2e40000640 1 -- 172.21.15.141:0/3624445415 shutdown_connections 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.426+0000 7f2e40000640 1 -- 172.21.15.141:0/3624445415 >> 172.21.15.141:0/3624445415 conn(0x7f2e3806a800 msgr2=0x7f2e3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.426+0000 7f2e40000640 1 -- 172.21.15.141:0/3624445415 shutdown_connections 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.426+0000 7f2e40000640 1 -- 172.21.15.141:0/3624445415 wait complete. 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.427+0000 7f2e40000640 1 Processor -- start 2024-01-16T14:58:08.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.427+0000 7f2e40000640 1 -- start start 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.428+0000 7f2e40000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e380715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.428+0000 7f2e40000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e38070cd0 0x7f2e381ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.428+0000 7f2e40000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815c9e0 0x7f2e3815ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.428+0000 7f2e40000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3815d3d0 con 0x7f2e18003970 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.428+0000 7f2e40000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e380711b0 con 0x7f2e38070cd0 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.428+0000 7f2e40000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3814fbf0 con 0x7f2e3815c9e0 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e380715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e380715c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45862/0 (socket says 172.21.15.141:45862) 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 -- 172.21.15.141:0/98255931 learned_addr learned my addr 172.21.15.141:0/98255931 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3e000640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815c9e0 0x7f2e3815ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3cc00640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e38070cd0 0x7f2e381ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 -- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815c9e0 msgr2=0x7f2e3815ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815c9e0 0x7f2e3815ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 -- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e38070cd0 msgr2=0x7f2e381ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e38070cd0 0x7f2e381ee1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 -- 172.21.15.141:0/98255931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e38155ea0 con 0x7f2e18003970 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e3d600640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e380715c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800c250 tx=0x7f2e28001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.429+0000 7f2e35600640 1 -- 172.21.15.141:0/98255931 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2802cbd0 con 0x7f2e18003970 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.430+0000 7f2e35600640 1 -- 172.21.15.141:0/98255931 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2802cd30 con 0x7f2e18003970 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.430+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e38156180 con 0x7f2e18003970 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.430+0000 7f2e35600640 1 -- 172.21.15.141:0/98255931 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2802cea0 con 0x7f2e18003970 2024-01-16T14:58:08.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.430+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e38156460 con 0x7f2e18003970 2024-01-16T14:58:08.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.431+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e380675d0 con 0x7f2e18003970 2024-01-16T14:58:08.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.432+0000 7f2e35600640 1 -- 172.21.15.141:0/98255931 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2e28032070 con 0x7f2e18003970 2024-01-16T14:58:08.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.433+0000 7f2e35600640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e1006b2f0 0x7f2e1006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.433+0000 7f2e35600640 1 -- 172.21.15.141:0/98255931 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f2e280ab0d0 con 0x7f2e18003970 2024-01-16T14:58:08.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.433+0000 7f2e3cc00640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e1006b2f0 0x7f2e1006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.434+0000 7f2e3cc00640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e1006b2f0 0x7f2e1006d7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2e38068060 tx=0x7f2e20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.437+0000 7f2e35600640 1 -- 172.21.15.141:0/98255931 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e28075f50 con 0x7f2e18003970 2024-01-16T14:58:08.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.652+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e3814fd30 con 0x7f2e18003970 2024-01-16T14:58:08.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.654+0000 7f2e35600640 1 -- 172.21.15.141:0/98255931 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f2e28045090 con 0x7f2e18003970 2024-01-16T14:58:08.635 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:08.635 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:58:08.635 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:58:08.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.656+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e1006b2f0 msgr2=0x7f2e1006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.656+0000 7f2e40000640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e1006b2f0 0x7f2e1006d7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2e38068060 tx=0x7f2e20008040 comp rx=0 tx=0).stop 2024-01-16T14:58:08.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.656+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 msgr2=0x7f2e380715c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.656+0000 7f2e40000640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e380715c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800c250 tx=0x7f2e28001950 comp rx=0 tx=0).stop 2024-01-16T14:58:08.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 shutdown_connections 2024-01-16T14:58:08.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e1006b2f0 0x7f2e1006d7b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e3815c9e0 0x7f2e3815ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e38070cd0 0x7f2e381ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 --2- 172.21.15.141:0/98255931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e18003970 0x7f2e380715c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 >> 172.21.15.141:0/98255931 conn(0x7f2e3806a800 msgr2=0x7f2e3815de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 shutdown_connections 2024-01-16T14:58:08.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.657+0000 7f2e40000640 1 -- 172.21.15.141:0/98255931 wait complete. 2024-01-16T14:58:08.656 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:08.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.870+0000 7f72a2800640 1 Processor -- start 2024-01-16T14:58:08.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.870+0000 7f72a2800640 1 -- start start 2024-01-16T14:58:08.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f72a2800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f729c14b2c0 0x7f729c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f72a2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c0cf7d0 0x7f729c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f72a2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f729c070ea0 0x7f729c0b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f72a2800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729c14b6a0 con 0x7f729c070ea0 2024-01-16T14:58:08.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f72a2800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729c0cfbf0 con 0x7f729c14b2c0 2024-01-16T14:58:08.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f72a2800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c071690 con 0x7f729c0cf7d0 2024-01-16T14:58:08.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f729be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f729c070ea0 0x7f729c0b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f729be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f729c070ea0 0x7f729c0b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45876/0 (socket says 172.21.15.141:45876) 2024-01-16T14:58:08.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.872+0000 7f729be00640 1 -- 172.21.15.141:0/2322811900 learned_addr learned my addr 172.21.15.141:0/2322811900 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:08.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.873+0000 7f7292800640 1 --2- 172.21.15.141:0/2322811900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c0cf7d0 0x7f729c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.873+0000 7f729b400640 1 --2- 172.21.15.141:0/2322811900 >> v2:172.21.15.188:3300/0 conn(0x7f729c14b2c0 0x7f729c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.873+0000 7f729be00640 1 -- 172.21.15.141:0/2322811900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c0cf7d0 msgr2=0x7f729c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.873+0000 7f729be00640 1 --2- 172.21.15.141:0/2322811900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c0cf7d0 0x7f729c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.873+0000 7f729be00640 1 -- 172.21.15.141:0/2322811900 >> v2:172.21.15.188:3300/0 conn(0x7f729c14b2c0 msgr2=0x7f729c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:08.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.873+0000 7f729be00640 1 --2- 172.21.15.141:0/2322811900 >> v2:172.21.15.188:3300/0 conn(0x7f729c14b2c0 0x7f729c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.873+0000 7f729be00640 1 -- 172.21.15.141:0/2322811900 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c071970 con 0x7f729c070ea0 2024-01-16T14:58:08.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.874+0000 7f729be00640 1 --2- 172.21.15.141:0/2322811900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c070ea0 0x7f729c0b1ee0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f729c06e650 tx=0x7f729402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed064b053c3903b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f729aa00640 1 -- 172.21.15.141:0/2322811900 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729402ca70 con 0x7f729c070ea0 2024-01-16T14:58:08.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f729aa00640 1 -- 172.21.15.141:0/2322811900 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729402cbd0 con 0x7f729c070ea0 2024-01-16T14:58:08.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f729aa00640 1 -- 172.21.15.141:0/2322811900 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940343f0 con 0x7f729c070ea0 2024-01-16T14:58:08.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 -- 172.21.15.141:0/2322811900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c070ea0 msgr2=0x7f729c0b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 --2- 172.21.15.141:0/2322811900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c070ea0 0x7f729c0b1ee0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f729c06e650 tx=0x7f729402bed0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 -- 172.21.15.141:0/2322811900 shutdown_connections 2024-01-16T14:58:08.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 --2- 172.21.15.141:0/2322811900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c070ea0 0x7f729c0b1ee0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 --2- 172.21.15.141:0/2322811900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c0cf7d0 0x7f729c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 --2- 172.21.15.141:0/2322811900 >> v2:172.21.15.188:3300/0 conn(0x7f729c14b2c0 0x7f729c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 -- 172.21.15.141:0/2322811900 >> 172.21.15.141:0/2322811900 conn(0x7f729c06a800 msgr2=0x7f729c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:08.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 -- 172.21.15.141:0/2322811900 shutdown_connections 2024-01-16T14:58:08.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.875+0000 7f72a2800640 1 -- 172.21.15.141:0/2322811900 wait complete. 2024-01-16T14:58:08.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.877+0000 7f72a2800640 1 Processor -- start 2024-01-16T14:58:08.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.877+0000 7f72a2800640 1 -- start start 2024-01-16T14:58:08.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f72a2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c070ea0 0x7f729c0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f72a2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0cf7d0 0x7f729c0ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f72a2800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f729c14b2c0 0x7f729c1f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f72a2800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c0adc60 con 0x7f729c0cf7d0 2024-01-16T14:58:08.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f72a2800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c0adf40 con 0x7f729c14b2c0 2024-01-16T14:58:08.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f72a2800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c0ae220 con 0x7f729c070ea0 2024-01-16T14:58:08.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f7292800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0cf7d0 0x7f729c0ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f7292800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0cf7d0 0x7f729c0ab550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45890/0 (socket says 172.21.15.141:45890) 2024-01-16T14:58:08.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.878+0000 7f7292800640 1 -- 172.21.15.141:0/1688563729 learned_addr learned my addr 172.21.15.141:0/1688563729 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:08.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f729b400640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c070ea0 0x7f729c0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f729be00640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f729c14b2c0 0x7f729c1f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7292800640 1 -- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c070ea0 msgr2=0x7f729c0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7292800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c070ea0 0x7f729c0ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7292800640 1 -- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f729c14b2c0 msgr2=0x7f729c1f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7292800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f729c14b2c0 0x7f729c1f5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7292800640 1 -- 172.21.15.141:0/1688563729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7294009b30 con 0x7f729c0cf7d0 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f729b400640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c070ea0 0x7f729c0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:08.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f729be00640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f729c14b2c0 0x7f729c1f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:08.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7292800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0cf7d0 0x7f729c0ab550 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f728400d800 tx=0x7f728400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7293e00640 1 -- 172.21.15.141:0/1688563729 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72840096a0 con 0x7f729c0cf7d0 2024-01-16T14:58:08.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7293e00640 1 -- 172.21.15.141:0/1688563729 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7284009800 con 0x7f729c0cf7d0 2024-01-16T14:58:08.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f7293e00640 1 -- 172.21.15.141:0/1688563729 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7284009b90 con 0x7f729c0cf7d0 2024-01-16T14:58:08.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.879+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c0ae500 con 0x7f729c0cf7d0 2024-01-16T14:58:08.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.880+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c0ad850 con 0x7f729c0cf7d0 2024-01-16T14:58:08.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.881+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c0675d0 con 0x7f729c0cf7d0 2024-01-16T14:58:08.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.882+0000 7f7293e00640 1 -- 172.21.15.141:0/1688563729 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7284009cf0 con 0x7f729c0cf7d0 2024-01-16T14:58:08.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.882+0000 7f7293e00640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f727406b2c0 0x7f727406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:08.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.882+0000 7f7293e00640 1 -- 172.21.15.141:0/1688563729 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f728408a110 con 0x7f729c0cf7d0 2024-01-16T14:58:08.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.883+0000 7f729b400640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f727406b2c0 0x7f727406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:08.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.884+0000 7f729b400640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f727406b2c0 0x7f727406d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:08.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:08.887+0000 7f7293e00640 1 -- 172.21.15.141:0/1688563729 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7284054f00 con 0x7f729c0cf7d0 2024-01-16T14:58:09.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.061+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f729c06b3f0 con 0x7f729c0cf7d0 2024-01-16T14:58:09.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.063+0000 7f7293e00640 1 -- 172.21.15.141:0/1688563729 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7f728400b440 con 0x7f729c0cf7d0 2024-01-16T14:58:09.045 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:09.046 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:09.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f727406b2c0 msgr2=0x7f727406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f727406b2c0 0x7f727406d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c008040 comp rx=0 tx=0).stop 2024-01-16T14:58:09.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0cf7d0 msgr2=0x7f729c0ab550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0cf7d0 0x7f729c0ab550 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f728400d800 tx=0x7f728400dcc0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 shutdown_connections 2024-01-16T14:58:09.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f727406b2c0 0x7f727406d780 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f729c14b2c0 0x7f729c1f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0cf7d0 0x7f729c0ab550 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 --2- 172.21.15.141:0/1688563729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f729c070ea0 0x7f729c0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 >> 172.21.15.141:0/1688563729 conn(0x7f729c06a800 msgr2=0x7f729c0b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 shutdown_connections 2024-01-16T14:58:09.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.066+0000 7f72a2800640 1 -- 172.21.15.141:0/1688563729 wait complete. 2024-01-16T14:58:09.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:57:04.899427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:57:05.994953+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:58:09.066 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:58:09.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.285+0000 7fc2f9000640 1 Processor -- start 2024-01-16T14:58:09.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.285+0000 7fc2f9000640 1 -- start start 2024-01-16T14:58:09.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.287+0000 7fc2f9000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f413df80 0x7fc2f415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.287+0000 7fc2f9000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f406c470 con 0x7fc2f40cf9a0 2024-01-16T14:58:09.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.287+0000 7fc2f9000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f406c750 con 0x7fc2f413e360 2024-01-16T14:58:09.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.287+0000 7fc2f9000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f406ca30 con 0x7fc2f413df80 2024-01-16T14:58:09.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.287+0000 7fc2f2800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc2f40cf9a0 0x7fc2f40cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57054/0 (socket says 172.21.15.141:57054) 2024-01-16T14:58:09.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.287+0000 7fc2f2800640 1 -- 172.21.15.141:0/68957209 learned_addr learned my addr 172.21.15.141:0/68957209 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:09.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1e00640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f413df80 0x7fc2f415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219040465 0 0) 0x7fc2f406c470 con 0x7fc2f40cf9a0 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d4003520 con 0x7fc2f40cf9a0 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2224701300 0 0) 0x7fc2d4003520 con 0x7fc2f40cf9a0 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f413df80 msgr2=0x7fc2f415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f413df80 0x7fc2f415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 >> v1:172.21.15.188:6789/0 conn(0x7fc2f413e360 legacy=0x7fc2f40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f406cd10 con 0x7fc2f40cf9a0 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fc2ec0019f0 con 0x7fc2f40cf9a0 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.288+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 >> v1:172.21.15.141:6789/0 conn(0x7fc2f40cf9a0 legacy=0x7fc2f40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2f1400640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40039a0 0x7fc2d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2f1400640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2d40047c0 0x7fc2d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2f1400640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f413df80 0x7fc2d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f406ca30 con 0x7fc2d40047c0 2024-01-16T14:58:09.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d400fe40 con 0x7fc2d40039a0 2024-01-16T14:58:09.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4010000 con 0x7fc2f413df80 2024-01-16T14:58:09.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2ebe00640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2d40047c0 0x7fc2d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2f1e00640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40039a0 0x7fc2d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2ebe00640 1 -- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f413df80 msgr2=0x7fc2d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2ebe00640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f413df80 0x7fc2d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2ebe00640 1 -- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40039a0 msgr2=0x7fc2d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2ebe00640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40039a0 0x7fc2d4003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.289+0000 7fc2ebe00640 1 -- 172.21.15.141:0/68957209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d4003520 con 0x7fc2d40047c0 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.290+0000 7fc2ebe00640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2d40047c0 0x7fc2d4004c40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc001720 tx=0x7fc2d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a7d5a99ada38ce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0052d0 con 0x7fc2d40047c0 2024-01-16T14:58:09.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc02e050 con 0x7fc2d40047c0 2024-01-16T14:58:09.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f1400640 1 -- 172.21.15.141:0/68957209 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0035f0 con 0x7fc2d40047c0 2024-01-16T14:58:09.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f9000640 1 -- 172.21.15.141:0/68957209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2d40047c0 msgr2=0x7fc2d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f9000640 1 --2- 172.21.15.141:0/68957209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2d40047c0 0x7fc2d4004c40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc001720 tx=0x7fc2d40057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f2800640 1 -- 172.21.15.141:0/68957209 reap_dead start 2024-01-16T14:58:09.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f9000640 1 -- 172.21.15.141:0/68957209 shutdown_connections 2024-01-16T14:58:09.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f9000640 1 -- 172.21.15.141:0/68957209 >> 172.21.15.141:0/68957209 conn(0x7fc2f406a800 msgr2=0x7fc2f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f9000640 1 -- 172.21.15.141:0/68957209 shutdown_connections 2024-01-16T14:58:09.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.291+0000 7fc2f9000640 1 -- 172.21.15.141:0/68957209 wait complete. 2024-01-16T14:58:09.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.293+0000 7fc2f9000640 1 Processor -- start 2024-01-16T14:58:09.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.293+0000 7fc2f9000640 1 -- start start 2024-01-16T14:58:09.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2f9000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40047c0 0x7fc2f413d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2f9000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2f40b0670 0x7fc2f413cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2f9000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f4150710 0x7fc2f41560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2f9000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4150ca0 con 0x7fc2f40b0670 2024-01-16T14:58:09.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2f9000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4150f80 con 0x7fc2d40047c0 2024-01-16T14:58:09.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2f9000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4151260 con 0x7fc2f4150710 2024-01-16T14:58:09.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2f40b0670 0x7fc2f413cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.294+0000 7fc2ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2f40b0670 0x7fc2f413cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45902/0 (socket says 172.21.15.141:45902) 2024-01-16T14:58:09.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2ebe00640 1 -- 172.21.15.141:0/3463686257 learned_addr learned my addr 172.21.15.141:0/3463686257 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:09.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2f1e00640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40047c0 0x7fc2f413d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2f2800640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f4150710 0x7fc2f41560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2ebe00640 1 -- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f4150710 msgr2=0x7fc2f41560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2ebe00640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f4150710 0x7fc2f41560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2ebe00640 1 -- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40047c0 msgr2=0x7fc2f413d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2ebe00640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40047c0 0x7fc2f413d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2ebe00640 1 -- 172.21.15.141:0/3463686257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f4151540 con 0x7fc2f40b0670 2024-01-16T14:58:09.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.295+0000 7fc2ebe00640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2f40b0670 0x7fc2f413cda0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc008fd0 tx=0x7fc2dc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.296+0000 7fc2ea000640 1 -- 172.21.15.141:0/3463686257 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0076c0 con 0x7fc2f40b0670 2024-01-16T14:58:09.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.296+0000 7fc2ea000640 1 -- 172.21.15.141:0/3463686257 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc007a30 con 0x7fc2f40b0670 2024-01-16T14:58:09.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.296+0000 7fc2ea000640 1 -- 172.21.15.141:0/3463686257 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc006be0 con 0x7fc2f40b0670 2024-01-16T14:58:09.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.296+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f4151820 con 0x7fc2f40b0670 2024-01-16T14:58:09.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.296+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f41565f0 con 0x7fc2f40b0670 2024-01-16T14:58:09.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.297+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f40675d0 con 0x7fc2f40b0670 2024-01-16T14:58:09.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.298+0000 7fc2ea000640 1 -- 172.21.15.141:0/3463686257 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc2dc006d40 con 0x7fc2f40b0670 2024-01-16T14:58:09.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.299+0000 7fc2ea000640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2cc06b2f0 0x7fc2cc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.299+0000 7fc2ea000640 1 -- 172.21.15.141:0/3463686257 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fc2dc0abf90 con 0x7fc2f40b0670 2024-01-16T14:58:09.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.299+0000 7fc2f1e00640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2cc06b2f0 0x7fc2cc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.300+0000 7fc2f1e00640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2cc06b2f0 0x7fc2cc06d7b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4071de0 tx=0x7fc2e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.303+0000 7fc2ea000640 1 -- 172.21.15.141:0/3463686257 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2dc0dee90 con 0x7fc2f40b0670 2024-01-16T14:58:09.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.493+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc2f415e350 con 0x7fc2f40b0670 2024-01-16T14:58:09.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.493+0000 7fc2ea000640 1 -- 172.21.15.141:0/3463686257 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc2dc04b020 con 0x7fc2f40b0670 2024-01-16T14:58:09.475 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:58:09.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2cc06b2f0 msgr2=0x7fc2cc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2cc06b2f0 0x7fc2cc06d7b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4071de0 tx=0x7fc2e4008040 comp rx=0 tx=0).stop 2024-01-16T14:58:09.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2f40b0670 msgr2=0x7fc2f413cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2f40b0670 0x7fc2f413cda0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc008fd0 tx=0x7fc2dc001120 comp rx=0 tx=0).stop 2024-01-16T14:58:09.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 shutdown_connections 2024-01-16T14:58:09.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2cc06b2f0 0x7fc2cc06d7b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc2f4150710 0x7fc2f41560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2f40b0670 0x7fc2f413cda0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 --2- 172.21.15.141:0/3463686257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2d40047c0 0x7fc2f413d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 >> 172.21.15.141:0/3463686257 conn(0x7fc2f406a800 msgr2=0x7fc2f40cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 shutdown_connections 2024-01-16T14:58:09.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.496+0000 7fc2f9000640 1 -- 172.21.15.141:0/3463686257 wait complete. 2024-01-16T14:58:09.497 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:58:09.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.730+0000 7fd44f200640 1 Processor -- start 2024-01-16T14:58:09.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.730+0000 7fd44f200640 1 -- start start 2024-01-16T14:58:09.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.731+0000 7fd44f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd4480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd44f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd4480cf7d0 0x7fd44815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd44f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4480cfc30 con 0x7fd44815e1e0 2024-01-16T14:58:09.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd44f200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4480b1ee0 con 0x7fd4480cf7d0 2024-01-16T14:58:09.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd44f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4480b2780 con 0x7fd448143ae0 2024-01-16T14:58:09.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd447e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd4480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd447e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd4480cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33882/0 (socket says 172.21.15.141:33882) 2024-01-16T14:58:09.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd447e00640 1 -- 172.21.15.141:0/333695069 learned_addr learned my addr 172.21.15.141:0/333695069 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:09.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.732+0000 7fd43f200640 1 --2- 172.21.15.141:0/333695069 >> v2:172.21.15.188:3300/0 conn(0x7fd4480cf7d0 0x7fd44815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.733+0000 7fd43f200640 1 -- 172.21.15.141:0/333695069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 msgr2=0x7fd4480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:09.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.733+0000 7fd43f200640 1 --2- 172.21.15.141:0/333695069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd4480cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.733+0000 7fd43f200640 1 -- 172.21.15.141:0/333695069 >> v1:172.21.15.188:6789/0 conn(0x7fd44815e1e0 legacy=0x7fd44815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.733+0000 7fd43f200640 1 -- 172.21.15.141:0/333695069 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44815e990 con 0x7fd4480cf7d0 2024-01-16T14:58:09.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.734+0000 7fd43f200640 1 --2- 172.21.15.141:0/333695069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4480cf7d0 0x7fd44815dca0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd4300097a0 tx=0x7fd43002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=317e04d2cae6140f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.735+0000 7fd447400640 1 -- 172.21.15.141:0/333695069 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430031830 con 0x7fd4480cf7d0 2024-01-16T14:58:09.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.735+0000 7fd447400640 1 -- 172.21.15.141:0/333695069 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd430031990 con 0x7fd4480cf7d0 2024-01-16T14:58:09.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.735+0000 7fd447400640 1 -- 172.21.15.141:0/333695069 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430031d20 con 0x7fd4480cf7d0 2024-01-16T14:58:09.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.735+0000 7fd44f200640 1 -- 172.21.15.141:0/333695069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4480cf7d0 msgr2=0x7fd44815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.735+0000 7fd44f200640 1 --2- 172.21.15.141:0/333695069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4480cf7d0 0x7fd44815dca0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd4300097a0 tx=0x7fd43002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.736+0000 7fd44f200640 1 -- 172.21.15.141:0/333695069 shutdown_connections 2024-01-16T14:58:09.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.736+0000 7fd44f200640 1 --2- 172.21.15.141:0/333695069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4480cf7d0 0x7fd44815dca0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.736+0000 7fd44f200640 1 --2- 172.21.15.141:0/333695069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd4480cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.736+0000 7fd44f200640 1 -- 172.21.15.141:0/333695069 >> 172.21.15.141:0/333695069 conn(0x7fd44806a800 msgr2=0x7fd44806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.736+0000 7fd44f200640 1 -- 172.21.15.141:0/333695069 shutdown_connections 2024-01-16T14:58:09.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.736+0000 7fd44f200640 1 -- 172.21.15.141:0/333695069 wait complete. 2024-01-16T14:58:09.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.737+0000 7fd44f200640 1 Processor -- start 2024-01-16T14:58:09.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.738+0000 7fd44f200640 1 -- start start 2024-01-16T14:58:09.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.738+0000 7fd44f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4480cf7d0 0x7fd44814c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd44f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd448139b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd44f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd44815e1e0 0x7fd4480ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd44f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44813bcc0 con 0x7fd4480cf7d0 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd44f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4480ad800 con 0x7fd44815e1e0 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd44f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4480ad240 con 0x7fd448143ae0 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4480cf7d0 0x7fd44814c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4480cf7d0 0x7fd44814c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45916/0 (socket says 172.21.15.141:45916) 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 -- 172.21.15.141:0/1225323307 learned_addr learned my addr 172.21.15.141:0/1225323307 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 -- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 msgr2=0x7fd448139b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd448139b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 -- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd44815e1e0 msgr2=0x7fd4480ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd44815e1e0 0x7fd4480ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.739+0000 7fd447e00640 1 -- 172.21.15.141:0/1225323307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd430009400 con 0x7fd4480cf7d0 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.740+0000 7fd447e00640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4480cf7d0 0x7fd44814c520 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd438009fd0 tx=0x7fd43800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.740+0000 7fd444c00640 1 -- 172.21.15.141:0/1225323307 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43800c940 con 0x7fd4480cf7d0 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.740+0000 7fd444c00640 1 -- 172.21.15.141:0/1225323307 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43800caa0 con 0x7fd4480cf7d0 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.740+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4480ad520 con 0x7fd4480cf7d0 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.740+0000 7fd444c00640 1 -- 172.21.15.141:0/1225323307 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43800ce30 con 0x7fd4480cf7d0 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.740+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44814ca60 con 0x7fd4480cf7d0 2024-01-16T14:58:09.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.741+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44813b740 con 0x7fd4480cf7d0 2024-01-16T14:58:09.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.742+0000 7fd444c00640 1 -- 172.21.15.141:0/1225323307 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd438013470 con 0x7fd4480cf7d0 2024-01-16T14:58:09.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.743+0000 7fd444c00640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd42006b2c0 0x7fd42006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:09.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.743+0000 7fd444c00640 1 -- 172.21.15.141:0/1225323307 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fd438089e20 con 0x7fd4480cf7d0 2024-01-16T14:58:09.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.744+0000 7fd43f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd42006b2c0 0x7fd42006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:09.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.745+0000 7fd43f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd42006b2c0 0x7fd42006d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd430036000 tx=0x7fd430030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:09.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.748+0000 7fd444c00640 1 -- 172.21.15.141:0/1225323307 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd44813b740 con 0x7fd4480cf7d0 2024-01-16T14:58:09.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.935+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd44813b740 con 0x7fd4480cf7d0 2024-01-16T14:58:09.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.936+0000 7fd444c00640 1 -- 172.21.15.141:0/1225323307 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd438017020 con 0x7fd4480cf7d0 2024-01-16T14:58:09.918 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd42006b2c0 msgr2=0x7fd42006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd42006b2c0 0x7fd42006d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd430036000 tx=0x7fd430030000 comp rx=0 tx=0).stop 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4480cf7d0 msgr2=0x7fd44814c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4480cf7d0 0x7fd44814c520 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd438009fd0 tx=0x7fd43800ed40 comp rx=0 tx=0).stop 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 shutdown_connections 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd42006b2c0 0x7fd42006d780 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd44815e1e0 0x7fd4480ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd448143ae0 0x7fd448139b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.938+0000 7fd44f200640 1 --2- 172.21.15.141:0/1225323307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4480cf7d0 0x7fd44814c520 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:09.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.939+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 >> 172.21.15.141:0/1225323307 conn(0x7fd44806a800 msgr2=0x7fd44815ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:09.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.939+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 shutdown_connections 2024-01-16T14:58:09.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:09.939+0000 7fd44f200640 1 -- 172.21.15.141:0/1225323307 wait complete. 2024-01-16T14:58:09.937 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:58:10.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.169+0000 7fd40e800640 1 Processor -- start 2024-01-16T14:58:10.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.170+0000 7fd40e800640 1 -- start start 2024-01-16T14:58:10.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.171+0000 7fd40e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 0x7fd4080b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.171+0000 7fd40e800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd408071190 con 0x7fd40815da10 2024-01-16T14:58:10.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.171+0000 7fd40e800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd408071470 con 0x7fd408150760 2024-01-16T14:58:10.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.171+0000 7fd40e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd408071750 con 0x7fd408072230 2024-01-16T14:58:10.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.171+0000 7fd406a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd40815da10 0x7fd4080b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57066/0 (socket says 172.21.15.141:57066) 2024-01-16T14:58:10.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406a00640 1 -- 172.21.15.141:0/697424577 learned_addr learned my addr 172.21.15.141:0/697424577 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd407e00640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 0x7fd4080b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223866648 0 0) 0x7fd408071190 con 0x7fd40815da10 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e8003520 con 0x7fd40815da10 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3285627683 0 0) 0x7fd3e8003520 con 0x7fd40815da10 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 msgr2=0x7fd4080b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 0x7fd4080b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 >> v1:172.21.15.188:6789/0 conn(0x7fd408150760 legacy=0x7fd40815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd408071a30 con 0x7fd40815da10 2024-01-16T14:58:10.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd3f0001670 con 0x7fd40815da10 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 >> v1:172.21.15.141:6789/0 conn(0x7fd40815da10 legacy=0x7fd4080b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.172+0000 7fd406000640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 0x7fd3e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406000640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3e8003f70 0x7fd3e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406000640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8004d00 0x7fd3e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408071750 con 0x7fd3e8003f70 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e800dcd0 con 0x7fd3e8004d00 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e800de90 con 0x7fd408072230 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406a00640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3e8003f70 0x7fd3e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd407400640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 0x7fd3e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd407e00640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8004d00 0x7fd3e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406a00640 1 -- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 msgr2=0x7fd3e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406a00640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 0x7fd3e80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406a00640 1 -- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8004d00 msgr2=0x7fd3e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406a00640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8004d00 0x7fd3e8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd406a00640 1 -- 172.21.15.141:0/697424577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8003520 con 0x7fd3e8003f70 2024-01-16T14:58:10.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.173+0000 7fd407400640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408072230 0x7fd3e80039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.174+0000 7fd406a00640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3e8003f70 0x7fd3e80047c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000c0e0 tx=0x7fd3e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f17469fe5fa1c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0005ad0 con 0x7fd3e8003f70 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f0005c30 con 0x7fd3e8003f70 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd406000640 1 -- 172.21.15.141:0/697424577 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0007070 con 0x7fd3e8003f70 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd40e800640 1 -- 172.21.15.141:0/697424577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3e8003f70 msgr2=0x7fd3e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd40e800640 1 --2- 172.21.15.141:0/697424577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3e8003f70 0x7fd3e80047c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000c0e0 tx=0x7fd3e80057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd407e00640 1 -- 172.21.15.141:0/697424577 reap_dead start 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd40e800640 1 -- 172.21.15.141:0/697424577 shutdown_connections 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd40e800640 1 -- 172.21.15.141:0/697424577 >> 172.21.15.141:0/697424577 conn(0x7fd40806a800 msgr2=0x7fd40806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:10.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd40e800640 1 -- 172.21.15.141:0/697424577 shutdown_connections 2024-01-16T14:58:10.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.175+0000 7fd40e800640 1 -- 172.21.15.141:0/697424577 wait complete. 2024-01-16T14:58:10.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.177+0000 7fd40e800640 1 Processor -- start 2024-01-16T14:58:10.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.177+0000 7fd40e800640 1 -- start start 2024-01-16T14:58:10.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.178+0000 7fd40e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8003f70 0x7fd40815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.178+0000 7fd40e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4080ce460 0x7fd4080ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.178+0000 7fd40e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408140950 0x7fd408140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.178+0000 7fd40e800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b1a40 con 0x7fd4080ce460 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.178+0000 7fd40e800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080cc100 con 0x7fd3e8003f70 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.178+0000 7fd40e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b0e30 con 0x7fd408140950 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4080ce460 0x7fd4080ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4080ce460 0x7fd4080ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45930/0 (socket says 172.21.15.141:45930) 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 -- 172.21.15.141:0/3125049210 learned_addr learned my addr 172.21.15.141:0/3125049210 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd407400640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8003f70 0x7fd40815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 -- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408140950 msgr2=0x7fd408140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd407e00640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408140950 0x7fd408140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408140950 0x7fd408140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 -- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8003f70 msgr2=0x7fd40815ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8003f70 0x7fd40815ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 -- 172.21.15.141:0/3125049210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4080b1110 con 0x7fd4080ce460 2024-01-16T14:58:10.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.179+0000 7fd406a00640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4080ce460 0x7fd4080ce8e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000c0e0 tx=0x7fd40815ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.180+0000 7fd3ff400640 1 -- 172.21.15.141:0/3125049210 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0007500 con 0x7fd4080ce460 2024-01-16T14:58:10.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.180+0000 7fd3ff400640 1 -- 172.21.15.141:0/3125049210 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f0007660 con 0x7fd4080ce460 2024-01-16T14:58:10.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.180+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4080b13f0 con 0x7fd4080ce460 2024-01-16T14:58:10.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.180+0000 7fd3ff400640 1 -- 172.21.15.141:0/3125049210 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f002c960 con 0x7fd4080ce460 2024-01-16T14:58:10.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.180+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd408141310 con 0x7fd4080ce460 2024-01-16T14:58:10.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.181+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4080cc410 con 0x7fd4080ce460 2024-01-16T14:58:10.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.182+0000 7fd3ff400640 1 -- 172.21.15.141:0/3125049210 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd3f0032070 con 0x7fd4080ce460 2024-01-16T14:58:10.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.183+0000 7fd3ff400640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006b2c0 0x7fd3e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.183+0000 7fd3ff400640 1 -- 172.21.15.141:0/3125049210 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fd3f00ab250 con 0x7fd4080ce460 2024-01-16T14:58:10.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.183+0000 7fd407400640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006b2c0 0x7fd3e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.185+0000 7fd407400640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006b2c0 0x7fd3e006d780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8001e40 tx=0x7fd3f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.187+0000 7fd3ff400640 1 -- 172.21.15.141:0/3125049210 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4080cc410 con 0x7fd4080ce460 2024-01-16T14:58:10.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.375+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4080cc3f0 con 0x7fd4080ce460 2024-01-16T14:58:10.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.376+0000 7fd3ff400640 1 -- 172.21.15.141:0/3125049210 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3f0075fa0 con 0x7fd4080ce460 2024-01-16T14:58:10.358 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006b2c0 msgr2=0x7fd3e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006b2c0 0x7fd3e006d780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8001e40 tx=0x7fd3f8009040 comp rx=0 tx=0).stop 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4080ce460 msgr2=0x7fd4080ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4080ce460 0x7fd4080ce8e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000c0e0 tx=0x7fd40815ad20 comp rx=0 tx=0).stop 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 shutdown_connections 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006b2c0 0x7fd3e006d780 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd408140950 0x7fd408140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4080ce460 0x7fd4080ce8e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 --2- 172.21.15.141:0/3125049210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3e8003f70 0x7fd40815ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.379+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 >> 172.21.15.141:0/3125049210 conn(0x7fd40806a800 msgr2=0x7fd40815ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.380+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 shutdown_connections 2024-01-16T14:58:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.380+0000 7fd40e800640 1 -- 172.21.15.141:0/3125049210 wait complete. 2024-01-16T14:58:10.379 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:10.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.606+0000 7f866b000640 1 Processor -- start 2024-01-16T14:58:10.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.606+0000 7f866b000640 1 -- start start 2024-01-16T14:58:10.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.608+0000 7f866b000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 0x7f86640cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.608+0000 7f866b000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866415deb0 con 0x7f8664071200 2024-01-16T14:58:10.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.608+0000 7f866b000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866415e190 con 0x7f86641618f0 2024-01-16T14:58:10.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.608+0000 7f866b000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866415ea30 con 0x7f8664157c30 2024-01-16T14:58:10.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.608+0000 7f8669000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8664071200 0x7f866406dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57080/0 (socket says 172.21.15.141:57080) 2024-01-16T14:58:10.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.608+0000 7f8669000640 1 -- 172.21.15.141:0/63672529 learned_addr learned my addr 172.21.15.141:0/63672529 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:10.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8663e00640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 0x7f86640cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758999794 0 0) 0x7f866415deb0 con 0x7f8664071200 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8644003520 con 0x7f8664071200 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520111471 0 0) 0x7f8644003520 con 0x7f8664071200 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 msgr2=0x7f86640cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 0x7f86640cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 >> v1:172.21.15.188:6789/0 conn(0x7f86641618f0 legacy=0x7f8664070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866415e470 con 0x7f8664071200 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f865c001a10 con 0x7f8664071200 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 >> v1:172.21.15.141:6789/0 conn(0x7f8664071200 legacy=0x7f866406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 0x7f86440039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8644003f20 0x7f86440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8644004d00 0x7f8644005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866415ea30 con 0x7f8644003f20 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864400ba60 con 0x7f8644004d00 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864400bc20 con 0x7f8664157c30 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.609+0000 7f8663400640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8644003f20 0x7f86440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.610+0000 7f8663e00640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 0x7f86440039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.610+0000 7f8663400640 1 -- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 msgr2=0x7f86440039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.610+0000 7f8663400640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8664157c30 0x7f86440039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.610+0000 7f8663400640 1 -- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8644004d00 msgr2=0x7f8644005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.610+0000 7f8663400640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8644004d00 0x7f8644005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.610+0000 7f8663400640 1 -- 172.21.15.141:0/63672529 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8644003520 con 0x7f8644003f20 2024-01-16T14:58:10.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.611+0000 7f8663400640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8644003f20 0x7f86440047c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f864c001750 tx=0x7f86440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4245669cabc50ad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.611+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0052d0 con 0x7f8644003f20 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.611+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c02e050 con 0x7f8644003f20 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.611+0000 7f8662a00640 1 -- 172.21.15.141:0/63672529 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c003830 con 0x7f8644003f20 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.612+0000 7f866b000640 1 -- 172.21.15.141:0/63672529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8644003f20 msgr2=0x7f86440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.612+0000 7f866b000640 1 --2- 172.21.15.141:0/63672529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8644003f20 0x7f86440047c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f864c001750 tx=0x7f86440057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.612+0000 7f8669000640 1 -- 172.21.15.141:0/63672529 reap_dead start 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.612+0000 7f866b000640 1 -- 172.21.15.141:0/63672529 shutdown_connections 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.612+0000 7f866b000640 1 -- 172.21.15.141:0/63672529 >> 172.21.15.141:0/63672529 conn(0x7f86640cf700 msgr2=0x7f86640cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.612+0000 7f866b000640 1 -- 172.21.15.141:0/63672529 shutdown_connections 2024-01-16T14:58:10.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.612+0000 7f866b000640 1 -- 172.21.15.141:0/63672529 wait complete. 2024-01-16T14:58:10.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.614+0000 7f866b000640 1 Processor -- start 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.614+0000 7f866b000640 1 -- start start 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f866b000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8644003f20 0x7f866415a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f866b000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86640b44e0 0x7f866415c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f866b000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86640b23b0 0x7f86640b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f866b000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8664158560 con 0x7f86640b44e0 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f866b000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866415afc0 con 0x7f86640b23b0 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f866b000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86640b4940 con 0x7f8644003f20 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f8663400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86640b44e0 0x7f866415c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f8663400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86640b44e0 0x7f866415c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45956/0 (socket says 172.21.15.141:45956) 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.615+0000 7f8663400640 1 -- 172.21.15.141:0/2076935393 learned_addr learned my addr 172.21.15.141:0/2076935393 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8663e00640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8644003f20 0x7f866415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8663400640 1 -- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8644003f20 msgr2=0x7f866415a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8663400640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8644003f20 0x7f866415a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8669000640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86640b23b0 0x7f86640b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8663400640 1 -- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86640b23b0 msgr2=0x7f86640b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8663400640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86640b23b0 0x7f86640b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8663400640 1 -- 172.21.15.141:0/2076935393 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866415aa00 con 0x7f86640b44e0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f8663400640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86640b44e0 0x7f866415c770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f864c008fd0 tx=0x7f864c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f865be00640 1 -- 172.21.15.141:0/2076935393 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0076c0 con 0x7f86640b44e0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f865be00640 1 -- 172.21.15.141:0/2076935393 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c007a30 con 0x7f86640b44e0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f865be00640 1 -- 172.21.15.141:0/2076935393 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c006c00 con 0x7f86640b44e0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.616+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866415ace0 con 0x7f86640b44e0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.617+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866415ccb0 con 0x7f86640b44e0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.618+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86641586d0 con 0x7f86640b44e0 2024-01-16T14:58:10.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.618+0000 7f865be00640 1 -- 172.21.15.141:0/2076935393 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f864c0035f0 con 0x7f86640b44e0 2024-01-16T14:58:10.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.619+0000 7f865be00640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f863c06b2f0 0x7f863c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:10.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.619+0000 7f865be00640 1 -- 172.21.15.141:0/2076935393 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f864c0abdf0 con 0x7f86640b44e0 2024-01-16T14:58:10.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.620+0000 7f8663e00640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f863c06b2f0 0x7f863c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:10.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.621+0000 7f8663e00640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f863c06b2f0 0x7f863c06d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8654009910 tx=0x7f8654008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:10.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.624+0000 7f865be00640 1 -- 172.21.15.141:0/2076935393 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f864c07a540 con 0x7f86640b44e0 2024-01-16T14:58:10.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.843+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f866415ce40 con 0x7f86640b44e0 2024-01-16T14:58:10.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.845+0000 7f865be00640 1 -- 172.21.15.141:0/2076935393 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f864c04b020 con 0x7f86640b44e0 2024-01-16T14:58:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:10.826 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:58:10.826 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.847+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f863c06b2f0 msgr2=0x7f863c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.847+0000 7f866b000640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f863c06b2f0 0x7f863c06d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8654009910 tx=0x7f8654008040 comp rx=0 tx=0).stop 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.847+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86640b44e0 msgr2=0x7f866415c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.847+0000 7f866b000640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86640b44e0 0x7f866415c770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f864c008fd0 tx=0x7f864c001120 comp rx=0 tx=0).stop 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 shutdown_connections 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f863c06b2f0 0x7f863c06d7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86640b23b0 0x7f86640b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86640b44e0 0x7f866415c770 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 --2- 172.21.15.141:0/2076935393 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8644003f20 0x7f866415a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 >> 172.21.15.141:0/2076935393 conn(0x7f86640cf700 msgr2=0x7f8664160450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:10.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 shutdown_connections 2024-01-16T14:58:10.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:10.848+0000 7f866b000640 1 -- 172.21.15.141:0/2076935393 wait complete. 2024-01-16T14:58:10.847 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:11.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.073+0000 7fee35000640 1 Processor -- start 2024-01-16T14:58:11.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.073+0000 7fee35000640 1 -- start start 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.074+0000 7fee35000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fee300cf9a0 0x7fee30070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.074+0000 7fee35000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fee30070ea0 0x7fee300cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee35000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee30071690 con 0x7fee3014b2c0 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee35000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3014b6a0 con 0x7fee30070ea0 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee35000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee30071970 con 0x7fee300cf9a0 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fee3014b2c0 0x7fee300cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57094/0 (socket says 172.21.15.141:57094) 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2de00640 1 -- 172.21.15.141:0/3227975199 learned_addr learned my addr 172.21.15.141:0/3227975199 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2e800640 1 --2- 172.21.15.141:0/3227975199 >> v2:172.21.15.141:3300/0 conn(0x7fee30070ea0 0x7fee300cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2d400640 1 -- 172.21.15.141:0/3227975199 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562200783 0 0) 0x7fee30071690 con 0x7fee3014b2c0 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2d400640 1 -- 172.21.15.141:0/3227975199 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee10003520 con 0x7fee3014b2c0 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee27e00640 1 --2- 172.21.15.141:0/3227975199 >> v2:172.21.15.188:3300/0 conn(0x7fee300cf9a0 0x7fee30070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2e800640 1 -- 172.21.15.141:0/3227975199 >> v2:172.21.15.188:3300/0 conn(0x7fee300cf9a0 msgr2=0x7fee30070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2e800640 1 --2- 172.21.15.141:0/3227975199 >> v2:172.21.15.188:3300/0 conn(0x7fee300cf9a0 0x7fee30070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2e800640 1 -- 172.21.15.141:0/3227975199 >> v1:172.21.15.141:6789/0 conn(0x7fee3014b2c0 legacy=0x7fee300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.075+0000 7fee2e800640 1 -- 172.21.15.141:0/3227975199 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee30071c50 con 0x7fee30070ea0 2024-01-16T14:58:11.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.076+0000 7fee27e00640 1 --2- 172.21.15.141:0/3227975199 >> v2:172.21.15.188:3300/0 conn(0x7fee300cf9a0 0x7fee30070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.077+0000 7fee2e800640 1 --2- 172.21.15.141:0/3227975199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee30070ea0 0x7fee300cc160 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fee300cfea0 tx=0x7fee2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2547d75089f3184f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.077+0000 7fee2d400640 1 -- 172.21.15.141:0/3227975199 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2802cc80 con 0x7fee30070ea0 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.077+0000 7fee2d400640 1 -- 172.21.15.141:0/3227975199 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2802cde0 con 0x7fee30070ea0 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.077+0000 7fee2d400640 1 -- 172.21.15.141:0/3227975199 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28034660 con 0x7fee30070ea0 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 -- 172.21.15.141:0/3227975199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee30070ea0 msgr2=0x7fee300cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 --2- 172.21.15.141:0/3227975199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee30070ea0 0x7fee300cc160 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fee300cfea0 tx=0x7fee2802bed0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 -- 172.21.15.141:0/3227975199 shutdown_connections 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 --2- 172.21.15.141:0/3227975199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee30070ea0 0x7fee300cc160 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 --2- 172.21.15.141:0/3227975199 >> v2:172.21.15.188:3300/0 conn(0x7fee300cf9a0 0x7fee30070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 -- 172.21.15.141:0/3227975199 >> 172.21.15.141:0/3227975199 conn(0x7fee3006a800 msgr2=0x7fee3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:11.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 -- 172.21.15.141:0/3227975199 shutdown_connections 2024-01-16T14:58:11.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.078+0000 7fee35000640 1 -- 172.21.15.141:0/3227975199 wait complete. 2024-01-16T14:58:11.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.080+0000 7fee35000640 1 Processor -- start 2024-01-16T14:58:11.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.080+0000 7fee35000640 1 -- start start 2024-01-16T14:58:11.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee35000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee30070ea0 0x7fee30163ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee35000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee300cf9a0 0x7fee301fa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee35000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee3014b2c0 0x7fee3015eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee35000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee301fb550 con 0x7fee3014b2c0 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee35000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee301facb0 con 0x7fee300cf9a0 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee35000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee301faf90 con 0x7fee30070ea0 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee3014b2c0 0x7fee3015eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee3014b2c0 0x7fee3015eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45968/0 (socket says 172.21.15.141:45968) 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 -- 172.21.15.141:0/4222624925 learned_addr learned my addr 172.21.15.141:0/4222624925 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 -- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee30070ea0 msgr2=0x7fee30163ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee30070ea0 0x7fee30163ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 -- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee300cf9a0 msgr2=0x7fee301fa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee300cf9a0 0x7fee301fa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.081+0000 7fee2e800640 1 -- 172.21.15.141:0/4222624925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee28009a70 con 0x7fee3014b2c0 2024-01-16T14:58:11.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.082+0000 7fee2e800640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee3014b2c0 0x7fee3015eb80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fee28009e80 tx=0x7fee28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.082+0000 7fee26000640 1 -- 172.21.15.141:0/4222624925 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28040650 con 0x7fee3014b2c0 2024-01-16T14:58:11.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.082+0000 7fee26000640 1 -- 172.21.15.141:0/4222624925 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee280407b0 con 0x7fee3014b2c0 2024-01-16T14:58:11.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.082+0000 7fee26000640 1 -- 172.21.15.141:0/4222624925 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28040b60 con 0x7fee3014b2c0 2024-01-16T14:58:11.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.082+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee301fb270 con 0x7fee3014b2c0 2024-01-16T14:58:11.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.082+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee301409c0 con 0x7fee3014b2c0 2024-01-16T14:58:11.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.083+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee301f7410 con 0x7fee3014b2c0 2024-01-16T14:58:11.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.084+0000 7fee26000640 1 -- 172.21.15.141:0/4222624925 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fee28040cc0 con 0x7fee3014b2c0 2024-01-16T14:58:11.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.085+0000 7fee26000640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee0806b2c0 0x7fee0806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.085+0000 7fee26000640 1 -- 172.21.15.141:0/4222624925 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fee280ac920 con 0x7fee3014b2c0 2024-01-16T14:58:11.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.086+0000 7fee2de00640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee0806b2c0 0x7fee0806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.087+0000 7fee2de00640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee0806b2c0 0x7fee0806d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fee20001570 tx=0x7fee20007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.089+0000 7fee26000640 1 -- 172.21.15.141:0/4222624925 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee301f7410 con 0x7fee3014b2c0 2024-01-16T14:58:11.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.278+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee301f7410 con 0x7fee3014b2c0 2024-01-16T14:58:11.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.281+0000 7fee26000640 1 -- 172.21.15.141:0/4222624925 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7fee301f7410 con 0x7fee3014b2c0 2024-01-16T14:58:11.263 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:11.263 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:11.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee0806b2c0 msgr2=0x7fee0806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee0806b2c0 0x7fee0806d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fee20001570 tx=0x7fee20007930 comp rx=0 tx=0).stop 2024-01-16T14:58:11.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee3014b2c0 msgr2=0x7fee3015eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee3014b2c0 0x7fee3015eb80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fee28009e80 tx=0x7fee28004a40 comp rx=0 tx=0).stop 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 shutdown_connections 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee0806b2c0 0x7fee0806d780 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee3014b2c0 0x7fee3015eb80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee300cf9a0 0x7fee301fa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 --2- 172.21.15.141:0/4222624925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee30070ea0 0x7fee30163ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 >> 172.21.15.141:0/4222624925 conn(0x7fee3006a800 msgr2=0x7fee30147640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 shutdown_connections 2024-01-16T14:58:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.284+0000 7fee35000640 1 -- 172.21.15.141:0/4222624925 wait complete. 2024-01-16T14:58:11.285 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T14:58:11.285 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:11.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.513+0000 7fa326a00640 1 Processor -- start 2024-01-16T14:58:11.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.513+0000 7fa326a00640 1 -- start start 2024-01-16T14:58:11.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.515+0000 7fa326a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa3200cf9a0 0x7fa32015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.515+0000 7fa326a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3200b1ee0 con 0x7fa320143ae0 2024-01-16T14:58:11.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.515+0000 7fa326a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3200b2780 con 0x7fa32015e1e0 2024-01-16T14:58:11.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.515+0000 7fa326a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa32015e990 con 0x7fa3200cf9a0 2024-01-16T14:58:11.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.515+0000 7fa31fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa320143ae0 0x7fa3200cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57106/0 (socket says 172.21.15.141:57106) 2024-01-16T14:58:11.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.515+0000 7fa31fe00640 1 -- 172.21.15.141:0/3617112916 learned_addr learned my addr 172.21.15.141:0/3617112916 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:11.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.515+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696959513 0 0) 0x7fa3200b1ee0 con 0x7fa320143ae0 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa300005730 con 0x7fa320143ae0 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31f400640 1 --2- 172.21.15.141:0/3617112916 >> v2:172.21.15.188:3300/0 conn(0x7fa3200cf9a0 0x7fa32015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3249012222 0 0) 0x7fa300005730 con 0x7fa320143ae0 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 >> v2:172.21.15.188:3300/0 conn(0x7fa3200cf9a0 msgr2=0x7fa32015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 --2- 172.21.15.141:0/3617112916 >> v2:172.21.15.188:3300/0 conn(0x7fa3200cf9a0 0x7fa32015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 >> v1:172.21.15.188:6789/0 conn(0x7fa32015e1e0 legacy=0x7fa3200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3200b2130 con 0x7fa320143ae0 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa310001670 con 0x7fa320143ae0 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 >> v1:172.21.15.141:6789/0 conn(0x7fa320143ae0 legacy=0x7fa3200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa300005bd0 0x7fa300005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3000069d0 0x7fa300006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3000073e0 0x7fa30000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32015e990 con 0x7fa300005bd0 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30000e310 con 0x7fa3000073e0 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30000e4d0 con 0x7fa3000069d0 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.516+0000 7fa31fe00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa300005bd0 0x7fa300005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.517+0000 7fa31fe00640 1 -- 172.21.15.141:0/3617112916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3000069d0 msgr2=0x7fa300006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.517+0000 7fa31fe00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3000069d0 0x7fa300006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.517+0000 7fa31fe00640 1 -- 172.21.15.141:0/3617112916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3000073e0 msgr2=0x7fa30000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.517+0000 7fa31fe00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3000073e0 0x7fa30000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.517+0000 7fa31fe00640 1 -- 172.21.15.141:0/3617112916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa300005730 con 0x7fa300005bd0 2024-01-16T14:58:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.517+0000 7fa324a00640 1 -- 172.21.15.141:0/3617112916 reap_dead start 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.518+0000 7fa31fe00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa300005bd0 0x7fa300005fd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c160 tx=0x7fa300007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da61f4bf40922e5a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.518+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310005ad0 con 0x7fa300005bd0 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.518+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa310005c30 con 0x7fa300005bd0 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.518+0000 7fa31ea00640 1 -- 172.21.15.141:0/3617112916 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31002c7e0 con 0x7fa300005bd0 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.518+0000 7fa326a00640 1 -- 172.21.15.141:0/3617112916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa300005bd0 msgr2=0x7fa300005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.518+0000 7fa326a00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa300005bd0 0x7fa300005fd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c160 tx=0x7fa300007a00 comp rx=0 tx=0).stop 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.519+0000 7fa326a00640 1 -- 172.21.15.141:0/3617112916 shutdown_connections 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.519+0000 7fa326a00640 1 --2- 172.21.15.141:0/3617112916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa300005bd0 0x7fa300005fd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.519+0000 7fa326a00640 1 -- 172.21.15.141:0/3617112916 >> 172.21.15.141:0/3617112916 conn(0x7fa32006a800 msgr2=0x7fa32006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.519+0000 7fa326a00640 1 -- 172.21.15.141:0/3617112916 shutdown_connections 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.519+0000 7fa326a00640 1 -- 172.21.15.141:0/3617112916 wait complete. 2024-01-16T14:58:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.521+0000 7fa326a00640 1 Processor -- start 2024-01-16T14:58:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.521+0000 7fa326a00640 1 -- start start 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa326a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa300005bd0 0x7fa320159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa326a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa32013c220 0x7fa32013c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa326a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3200cdf80 0x7fa3200ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa326a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3201ef3c0 con 0x7fa32013c220 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa326a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200cecb0 con 0x7fa3200cdf80 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa326a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32013cbe0 con 0x7fa300005bd0 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa32013c220 0x7fa32013c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa32013c220 0x7fa32013c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45984/0 (socket says 172.21.15.141:45984) 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 -- 172.21.15.141:0/2192063060 learned_addr learned my addr 172.21.15.141:0/2192063060 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa324a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3200cdf80 0x7fa3200ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31fe00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa300005bd0 0x7fa320159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 -- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa300005bd0 msgr2=0x7fa320159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa300005bd0 0x7fa320159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 -- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3200cdf80 msgr2=0x7fa3200ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3200cdf80 0x7fa3200ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.522+0000 7fa31f400640 1 -- 172.21.15.141:0/2192063060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3201ee790 con 0x7fa32013c220 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.523+0000 7fa324a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3200cdf80 0x7fa3200ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.523+0000 7fa31f400640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa32013c220 0x7fa32013c6a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa308007fd0 tx=0x7fa30800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.523+0000 7fa317e00640 1 -- 172.21.15.141:0/2192063060 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3080125f0 con 0x7fa32013c220 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.523+0000 7fa317e00640 1 -- 172.21.15.141:0/2192063060 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa308012750 con 0x7fa32013c220 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.523+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3200cc6e0 con 0x7fa32013c220 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.523+0000 7fa317e00640 1 -- 172.21.15.141:0/2192063060 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308012b00 con 0x7fa32013c220 2024-01-16T14:58:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.523+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3201eed00 con 0x7fa32013c220 2024-01-16T14:58:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.524+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3201ef500 con 0x7fa32013c220 2024-01-16T14:58:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.525+0000 7fa317e00640 1 -- 172.21.15.141:0/2192063060 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa308012c60 con 0x7fa32013c220 2024-01-16T14:58:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.526+0000 7fa317e00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2f806b2f0 0x7fa2f806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.526+0000 7fa317e00640 1 -- 172.21.15.141:0/2192063060 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fa308089ea0 con 0x7fa32013c220 2024-01-16T14:58:11.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.527+0000 7fa31fe00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2f806b2f0 0x7fa2f806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.528+0000 7fa31fe00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2f806b2f0 0x7fa2f806d7b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa310034000 tx=0x7fa310009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.530+0000 7fa317e00640 1 -- 172.21.15.141:0/2192063060 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3201ef500 con 0x7fa32013c220 2024-01-16T14:58:11.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.761+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3201ef500 con 0x7fa32013c220 2024-01-16T14:58:11.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.762+0000 7fa317e00640 1 -- 172.21.15.141:0/2192063060 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fa3201ef500 con 0x7fa32013c220 2024-01-16T14:58:11.743 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:11.744 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:58:11.744 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2f806b2f0 msgr2=0x7fa2f806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2f806b2f0 0x7fa2f806d7b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa310034000 tx=0x7fa310009030 comp rx=0 tx=0).stop 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa32013c220 msgr2=0x7fa32013c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa32013c220 0x7fa32013c6a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa308007fd0 tx=0x7fa30800d9e0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 shutdown_connections 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2f806b2f0 0x7fa2f806d7b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3200cdf80 0x7fa3200ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa32013c220 0x7fa32013c6a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 --2- 172.21.15.141:0/2192063060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa300005bd0 0x7fa320159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 >> 172.21.15.141:0/2192063060 conn(0x7fa32006a800 msgr2=0x7fa32015da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 shutdown_connections 2024-01-16T14:58:11.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.765+0000 7fa326a00640 1 -- 172.21.15.141:0/2192063060 wait complete. 2024-01-16T14:58:11.765 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:11.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.991+0000 7fa34e600640 1 Processor -- start 2024-01-16T14:58:11.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.991+0000 7fa34e600640 1 -- start start 2024-01-16T14:58:11.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.992+0000 7fa34e600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.992+0000 7fa34e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 0x7fa3480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa34e600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa34815f9c0 0x7fa3480cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa34e600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34806c470 con 0x7fa34813df80 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa34e600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34806c750 con 0x7fa34815f9c0 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa34e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34806ca30 con 0x7fa34813e360 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45992/0 (socket says 172.21.15.141:45992) 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 -- 172.21.15.141:0/1447290809 learned_addr learned my addr 172.21.15.141:0/1447290809 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 -- 172.21.15.141:0/1447290809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 msgr2=0x7fa3480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 --2- 172.21.15.141:0/1447290809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 0x7fa3480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 -- 172.21.15.141:0/1447290809 >> v2:172.21.15.188:3300/0 conn(0x7fa34815f9c0 msgr2=0x7fa3480cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:11.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347e00640 1 --2- 172.21.15.141:0/1447290809 >> v2:172.21.15.188:3300/0 conn(0x7fa34815f9c0 0x7fa3480cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 --2- 172.21.15.141:0/1447290809 >> v2:172.21.15.188:3300/0 conn(0x7fa34815f9c0 0x7fa3480cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347400640 1 -- 172.21.15.141:0/1447290809 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34806cd10 con 0x7fa34813df80 2024-01-16T14:58:11.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.993+0000 7fa347e00640 1 --2- 172.21.15.141:0/1447290809 >> v2:172.21.15.188:3300/0 conn(0x7fa34815f9c0 0x7fa3480cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:11.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.994+0000 7fa347400640 1 --2- 172.21.15.141:0/1447290809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa34815bee0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa34806e6d0 tx=0x7fa33802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9623a9ffa64f47e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.995+0000 7fa346a00640 1 -- 172.21.15.141:0/1447290809 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338031860 con 0x7fa34813df80 2024-01-16T14:58:11.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.995+0000 7fa346a00640 1 -- 172.21.15.141:0/1447290809 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3380319c0 con 0x7fa34813df80 2024-01-16T14:58:11.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.995+0000 7fa346a00640 1 -- 172.21.15.141:0/1447290809 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338031d50 con 0x7fa34813df80 2024-01-16T14:58:11.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.995+0000 7fa34e600640 1 -- 172.21.15.141:0/1447290809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 msgr2=0x7fa34815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.995+0000 7fa34e600640 1 --2- 172.21.15.141:0/1447290809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa34815bee0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa34806e6d0 tx=0x7fa33802b900 comp rx=0 tx=0).stop 2024-01-16T14:58:11.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.996+0000 7fa34e600640 1 -- 172.21.15.141:0/1447290809 shutdown_connections 2024-01-16T14:58:11.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.996+0000 7fa34e600640 1 --2- 172.21.15.141:0/1447290809 >> v2:172.21.15.188:3300/0 conn(0x7fa34815f9c0 0x7fa3480cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.996+0000 7fa34e600640 1 --2- 172.21.15.141:0/1447290809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 0x7fa3480cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.996+0000 7fa34e600640 1 --2- 172.21.15.141:0/1447290809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.996+0000 7fa34e600640 1 -- 172.21.15.141:0/1447290809 >> 172.21.15.141:0/1447290809 conn(0x7fa34806a800 msgr2=0x7fa34806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:11.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.996+0000 7fa34e600640 1 -- 172.21.15.141:0/1447290809 shutdown_connections 2024-01-16T14:58:11.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.996+0000 7fa34e600640 1 -- 172.21.15.141:0/1447290809 wait complete. 2024-01-16T14:58:11.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.997+0000 7fa34e600640 1 Processor -- start 2024-01-16T14:58:11.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.998+0000 7fa34e600640 1 -- start start 2024-01-16T14:58:11.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.998+0000 7fa34e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa3480a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa34e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 0x7fa34813abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa34e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34815f9c0 0x7fa34815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa34e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480b37a0 con 0x7fa34813df80 2024-01-16T14:58:11.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa34e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480b3c30 con 0x7fa34815f9c0 2024-01-16T14:58:11.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa34e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480b2e70 con 0x7fa34813e360 2024-01-16T14:58:11.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa3480a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa3480a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46008/0 (socket says 172.21.15.141:46008) 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 -- 172.21.15.141:0/1669753000 learned_addr learned my addr 172.21.15.141:0/1669753000 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 -- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 msgr2=0x7fa34813abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 0x7fa34813abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 -- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34815f9c0 msgr2=0x7fa34815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34815f9c0 0x7fa34815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:11.999+0000 7fa347400640 1 -- 172.21.15.141:0/1669753000 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3380094d0 con 0x7fa34813df80 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.000+0000 7fa347400640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa3480a84c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa338000c00 tx=0x7fa3380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.000+0000 7fa33fe00640 1 -- 172.21.15.141:0/1669753000 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338031830 con 0x7fa34813df80 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.000+0000 7fa33fe00640 1 -- 172.21.15.141:0/1669753000 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3380443f0 con 0x7fa34813df80 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.000+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3480b3150 con 0x7fa34813df80 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.000+0000 7fa33fe00640 1 -- 172.21.15.141:0/1669753000 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3380447a0 con 0x7fa34813df80 2024-01-16T14:58:11.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.000+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3480add90 con 0x7fa34813df80 2024-01-16T14:58:11.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.001+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3480675d0 con 0x7fa34813df80 2024-01-16T14:58:11.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.002+0000 7fa33fe00640 1 -- 172.21.15.141:0/1669753000 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa33804d450 con 0x7fa34813df80 2024-01-16T14:58:11.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.003+0000 7fa33fe00640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa32006b2c0 0x7fa32006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:11.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.003+0000 7fa33fe00640 1 -- 172.21.15.141:0/1669753000 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fa3380ac300 con 0x7fa34813df80 2024-01-16T14:58:11.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.003+0000 7fa33e800640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa32006b2c0 0x7fa32006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:11.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.005+0000 7fa33e800640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa32006b2c0 0x7fa32006d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3481555d0 tx=0x7fa330006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:11.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.007+0000 7fa33fe00640 1 -- 172.21.15.141:0/1669753000 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa338077180 con 0x7fa34813df80 2024-01-16T14:58:12.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.234+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa34806df00 con 0x7fa34813df80 2024-01-16T14:58:12.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.235+0000 7fa33fe00640 1 -- 172.21.15.141:0/1669753000 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fa33807a7a0 con 0x7fa34813df80 2024-01-16T14:58:12.216 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:12.217 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/855022955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":855022955},{"type":"v1","addr":"172.21.15.188:6835","nonce":855022955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2020234717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2020234717},{"type":"v1","addr":"172.21.15.188:6837","nonce":2020234717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3158353041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3158353041},{"type":"v1","addr":"172.21.15.141:6835","nonce":3158353041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:57:06.988377+0000","modified":"2024-01-16T14:57:10.018720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6833/114446300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":114446300},{"type":"v1","addr":"172.21.15.141:6833","nonce":114446300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:58:12.217 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 8 2024-01-16T14:58:12.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa32006b2c0 msgr2=0x7fa32006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa32006b2c0 0x7fa32006d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3481555d0 tx=0x7fa330006c90 comp rx=0 tx=0).stop 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 msgr2=0x7fa3480a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa3480a84c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa338000c00 tx=0x7fa3380046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 shutdown_connections 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa32006b2c0 0x7fa32006d780 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34815f9c0 0x7fa34815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa34813e360 0x7fa34813abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 --2- 172.21.15.141:0/1669753000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa34813df80 0x7fa3480a84c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 >> 172.21.15.141:0/1669753000 conn(0x7fa34806a800 msgr2=0x7fa34813e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 shutdown_connections 2024-01-16T14:58:12.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.238+0000 7fa34e600640 1 -- 172.21.15.141:0/1669753000 wait complete. 2024-01-16T14:58:12.239 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.452+0000 7f68aec00640 1 Processor -- start 2024-01-16T14:58:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.453+0000 7f68aec00640 1 -- start start 2024-01-16T14:58:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.454+0000 7f68aec00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f68a8071f00 0x7f68a8146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.454+0000 7f68aec00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80b2e10 con 0x7f68a8071410 2024-01-16T14:58:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68aec00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a806cc10 con 0x7f68a806c7c0 2024-01-16T14:58:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68aec00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a806cec0 con 0x7f68a8071f00 2024-01-16T14:58:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68ad800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f68a8071410 0x7f68a80717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57110/0 (socket says 172.21.15.141:57110) 2024-01-16T14:58:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68ad800640 1 -- 172.21.15.141:0/1737688722 learned_addr learned my addr 172.21.15.141:0/1737688722 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608382079 0 0) 0x7f68a80b2e10 con 0x7f68a8071410 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6888003520 con 0x7f68a8071410 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68ace00640 1 --2- 172.21.15.141:0/1737688722 >> v2:172.21.15.188:3300/0 conn(0x7f68a8071f00 0x7f68a8146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930145017 0 0) 0x7f6888003520 con 0x7f68a8071410 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 >> v2:172.21.15.188:3300/0 conn(0x7f68a8071f00 msgr2=0x7f68a8146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 --2- 172.21.15.141:0/1737688722 >> v2:172.21.15.188:3300/0 conn(0x7f68a8071f00 0x7f68a8146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 >> v1:172.21.15.188:6789/0 conn(0x7f68a806c7c0 legacy=0x7f68a8166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a806d1a0 con 0x7f68a8071410 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.455+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6898001670 con 0x7f68a8071410 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 >> v1:172.21.15.141:6789/0 conn(0x7f68a8071410 legacy=0x7f68a80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68a7e00640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6888003970 0x7f6888003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68a7e00640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68880047c0 0x7f6888004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68a7e00640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68880052a0 0x7f688800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a806cec0 con 0x7f6888003970 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688800c150 con 0x7f68880047c0 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688800c310 con 0x7f68880052a0 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ad800640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6888003970 0x7f6888003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ace00640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68880047c0 0x7f6888004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ad800640 1 -- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68880052a0 msgr2=0x7f688800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ad800640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68880052a0 0x7f688800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ad800640 1 -- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68880047c0 msgr2=0x7f6888004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ad800640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68880047c0 0x7f6888004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ad800640 1 -- 172.21.15.141:0/1737688722 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6888003520 con 0x7f6888003970 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ace00640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68880047c0 0x7f6888004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:12.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.456+0000 7f68ae200640 1 -- 172.21.15.141:0/1737688722 reap_dead start 2024-01-16T14:58:12.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.457+0000 7f68ad800640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6888003970 0x7f6888003d70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6898001eb0 tx=0x7f68880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e09c8cb909583f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898005ad0 con 0x7f6888003970 2024-01-16T14:58:12.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6898005c50 con 0x7f6888003970 2024-01-16T14:58:12.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68a7e00640 1 -- 172.21.15.141:0/1737688722 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689802c890 con 0x7f6888003970 2024-01-16T14:58:12.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68aec00640 1 -- 172.21.15.141:0/1737688722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6888003970 msgr2=0x7f6888003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68aec00640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6888003970 0x7f6888003d70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6898001eb0 tx=0x7f68880057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68aec00640 1 -- 172.21.15.141:0/1737688722 shutdown_connections 2024-01-16T14:58:12.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68aec00640 1 --2- 172.21.15.141:0/1737688722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6888003970 0x7f6888003d70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68aec00640 1 -- 172.21.15.141:0/1737688722 >> 172.21.15.141:0/1737688722 conn(0x7f68a806a800 msgr2=0x7f68a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:12.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68aec00640 1 -- 172.21.15.141:0/1737688722 shutdown_connections 2024-01-16T14:58:12.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.458+0000 7f68aec00640 1 -- 172.21.15.141:0/1737688722 wait complete. 2024-01-16T14:58:12.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.460+0000 7f68aec00640 1 Processor -- start 2024-01-16T14:58:12.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.460+0000 7f68aec00640 1 -- start start 2024-01-16T14:58:12.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68aec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6888003970 0x7f68a80c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68aec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68a8142ed0 0x7f68a8131ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68aec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68a8143350 0x7f68a8132400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68aec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8144580 con 0x7f68a8143350 2024-01-16T14:58:12.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68aec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a81449a0 con 0x7f6888003970 2024-01-16T14:58:12.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68aec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8143900 con 0x7f68a8142ed0 2024-01-16T14:58:12.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68ae200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68a8143350 0x7f68a8132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68ae200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68a8143350 0x7f68a8132400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46030/0 (socket says 172.21.15.141:46030) 2024-01-16T14:58:12.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.461+0000 7f68ae200640 1 -- 172.21.15.141:0/178264806 learned_addr learned my addr 172.21.15.141:0/178264806 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:12.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ace00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68a8142ed0 0x7f68a8131ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ad800640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6888003970 0x7f68a80c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ae200640 1 -- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68a8142ed0 msgr2=0x7f68a8131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ae200640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68a8142ed0 0x7f68a8131ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ae200640 1 -- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6888003970 msgr2=0x7f68a80c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ae200640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6888003970 0x7f68a80c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ae200640 1 -- 172.21.15.141:0/178264806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8143be0 con 0x7f68a8143350 2024-01-16T14:58:12.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ace00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68a8142ed0 0x7f68a8131ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:12.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ad800640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6888003970 0x7f68a80c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:12.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68ae200640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68a8143350 0x7f68a8132400 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f68a8146ec0 tx=0x7f68a000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68a5600640 1 -- 172.21.15.141:0/178264806 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a00070d0 con 0x7f68a8143350 2024-01-16T14:58:12.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68a5600640 1 -- 172.21.15.141:0/178264806 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0007230 con 0x7f68a8143350 2024-01-16T14:58:12.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68a5600640 1 -- 172.21.15.141:0/178264806 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0007620 con 0x7f68a8143350 2024-01-16T14:58:12.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a8143ec0 con 0x7f68a8143350 2024-01-16T14:58:12.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.462+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a8132be0 con 0x7f68a8143350 2024-01-16T14:58:12.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.463+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a81446c0 con 0x7f68a8143350 2024-01-16T14:58:12.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.464+0000 7f68a5600640 1 -- 172.21.15.141:0/178264806 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f68a0007780 con 0x7f68a8143350 2024-01-16T14:58:12.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.465+0000 7f68a5600640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f688006b2c0 0x7f688006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.465+0000 7f68a5600640 1 -- 172.21.15.141:0/178264806 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f68a00890a0 con 0x7f68a8143350 2024-01-16T14:58:12.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.466+0000 7f68ad800640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f688006b2c0 0x7f688006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.467+0000 7f68ad800640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f688006b2c0 0x7f688006d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6898034000 tx=0x7f6898009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.470+0000 7f68a5600640 1 -- 172.21.15.141:0/178264806 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68a81446c0 con 0x7f68a8143350 2024-01-16T14:58:12.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.663+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68a81446c0 con 0x7f68a8143350 2024-01-16T14:58:12.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.665+0000 7f68a5600640 1 -- 172.21.15.141:0/178264806 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16820 (secure 0 0 0) 0x7f68a81446c0 con 0x7f68a8143350 2024-01-16T14:58:12.647 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:12.647 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":24,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:57:54.368771+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:57:04.899427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:57:05.994953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2},{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.668+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f688006b2c0 msgr2=0x7f688006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.668+0000 7f68aec00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f688006b2c0 0x7f688006d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6898034000 tx=0x7f6898009030 comp rx=0 tx=0).stop 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.668+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68a8143350 msgr2=0x7f68a8132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.668+0000 7f68aec00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68a8143350 0x7f68a8132400 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f68a8146ec0 tx=0x7f68a000c870 comp rx=0 tx=0).stop 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 shutdown_connections 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f688006b2c0 0x7f688006d780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68a8143350 0x7f68a8132400 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68a8142ed0 0x7f68a8131ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 --2- 172.21.15.141:0/178264806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6888003970 0x7f68a80c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 >> 172.21.15.141:0/178264806 conn(0x7f68a806a800 msgr2=0x7f68a80afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:12.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 shutdown_connections 2024-01-16T14:58:12.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.669+0000 7f68aec00640 1 -- 172.21.15.141:0/178264806 wait complete. 2024-01-16T14:58:12.669 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-16T14:58:12.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.889+0000 7f9536a00640 1 Processor -- start 2024-01-16T14:58:12.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.889+0000 7f9536a00640 1 -- start start 2024-01-16T14:58:12.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f9536a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f95300cc500 0x7f95300cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f9536a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9530070e50 con 0x7f9530143c40 2024-01-16T14:58:12.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f9536a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9530144020 con 0x7f953006d160 2024-01-16T14:58:12.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f9536a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95300723c0 con 0x7f95300cc500 2024-01-16T14:58:12.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f9535600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9530143c40 0x7f9530071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57118/0 (socket says 172.21.15.141:57118) 2024-01-16T14:58:12.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f9535600640 1 -- 172.21.15.141:0/2333755144 learned_addr learned my addr 172.21.15.141:0/2333755144 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:12.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956165624 0 0) 0x7f9530070e50 con 0x7f9530143c40 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510003520 con 0x7f9530143c40 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.891+0000 7f9534c00640 1 --2- 172.21.15.141:0/2333755144 >> v2:172.21.15.188:3300/0 conn(0x7f95300cc500 0x7f95300cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34147968 0 0) 0x7f9510003520 con 0x7f9530143c40 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 >> v2:172.21.15.188:3300/0 conn(0x7f95300cc500 msgr2=0x7f95300cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 --2- 172.21.15.141:0/2333755144 >> v2:172.21.15.188:3300/0 conn(0x7f95300cc500 0x7f95300cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 >> v1:172.21.15.188:6789/0 conn(0x7f953006d160 legacy=0x7f953006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530071e00 con 0x7f9530143c40 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9520001670 con 0x7f9530143c40 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 >> v1:172.21.15.141:6789/0 conn(0x7f9530143c40 legacy=0x7f9530071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95100039c0 0x7f9510003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f951000fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95300723c0 con 0x7f9510005250 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100105b0 con 0x7f95100047c0 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9510010770 con 0x7f95100039c0 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f9536000640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f951000fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f9535600640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95100039c0 0x7f9510003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.892+0000 7f9534c00640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.893+0000 7f9536000640 1 -- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95100039c0 msgr2=0x7f9510003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.893+0000 7f9536000640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95100039c0 0x7f9510003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.893+0000 7f9536000640 1 -- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 msgr2=0x7f9510004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.893+0000 7f9536000640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.893+0000 7f9536000640 1 -- 172.21.15.141:0/2333755144 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9510003520 con 0x7f9510005250 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.893+0000 7f9534c00640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.893+0000 7f9536000640 1 -- 172.21.15.141:0/2333755144 reap_dead start 2024-01-16T14:58:12.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.894+0000 7f9536000640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f951000fd90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f95280044b0 tx=0x7f95100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b27059a3126c033 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.894+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528006d50 con 0x7f9510005250 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.894+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9528007020 con 0x7f9510005250 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.894+0000 7f952fe00640 1 -- 172.21.15.141:0/2333755144 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95280073d0 con 0x7f9510005250 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.894+0000 7f9536a00640 1 -- 172.21.15.141:0/2333755144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 msgr2=0x7f951000fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.894+0000 7f9536a00640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f951000fd90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f95280044b0 tx=0x7f95100057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.895+0000 7f9536a00640 1 -- 172.21.15.141:0/2333755144 shutdown_connections 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.895+0000 7f9536a00640 1 --2- 172.21.15.141:0/2333755144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f951000fd90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.895+0000 7f9536a00640 1 -- 172.21.15.141:0/2333755144 >> 172.21.15.141:0/2333755144 conn(0x7f953006a800 msgr2=0x7f953006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.895+0000 7f9536a00640 1 -- 172.21.15.141:0/2333755144 shutdown_connections 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.895+0000 7f9536a00640 1 -- 172.21.15.141:0/2333755144 wait complete. 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.896+0000 7f9536a00640 1 Processor -- start 2024-01-16T14:58:12.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.896+0000 7f9536a00640 1 -- start start 2024-01-16T14:58:12.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.897+0000 7f9536a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f95301ee070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.897+0000 7f9536a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f2960 0x7f9530142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.897+0000 7f9536a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9530143500 0x7f95301f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.897+0000 7f9536a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530157650 con 0x7f9510005250 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.897+0000 7f9536a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95301ee5b0 con 0x7f95301f2960 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.897+0000 7f9536a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9530158f90 con 0x7f9530143500 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.897+0000 7f9535600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f95301ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f95301ee070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46046/0 (socket says 172.21.15.141:46046) 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 -- 172.21.15.141:0/3583539770 learned_addr learned my addr 172.21.15.141:0/3583539770 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9536000640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9530143500 0x7f95301f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9534c00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f2960 0x7f9530142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 -- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9530143500 msgr2=0x7f95301f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9530143500 0x7f95301f1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 -- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f2960 msgr2=0x7f9530142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:12.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f2960 0x7f9530142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:12.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 -- 172.21.15.141:0/3583539770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530159270 con 0x7f9510005250 2024-01-16T14:58:12.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f9535600640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f95301ee070 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f952000e760 tx=0x7f952000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.898+0000 7f952d600640 1 -- 172.21.15.141:0/3583539770 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952000c6c0 con 0x7f9510005250 2024-01-16T14:58:12.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.899+0000 7f952d600640 1 -- 172.21.15.141:0/3583539770 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952000c820 con 0x7f9510005250 2024-01-16T14:58:12.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.899+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95301572e0 con 0x7f9510005250 2024-01-16T14:58:12.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.899+0000 7f952d600640 1 -- 172.21.15.141:0/3583539770 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952000cbb0 con 0x7f9510005250 2024-01-16T14:58:12.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.899+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95301f21e0 con 0x7f9510005250 2024-01-16T14:58:12.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.900+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95301594b0 con 0x7f9510005250 2024-01-16T14:58:12.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.901+0000 7f952d600640 1 -- 172.21.15.141:0/3583539770 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f952000cd10 con 0x7f9510005250 2024-01-16T14:58:12.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.902+0000 7f952d600640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806b2c0 0x7f950806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:12.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.902+0000 7f952d600640 1 -- 172.21.15.141:0/3583539770 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f9520089aa0 con 0x7f9510005250 2024-01-16T14:58:12.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.903+0000 7f9534c00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806b2c0 0x7f950806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:12.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.904+0000 7f9534c00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806b2c0 0x7f950806d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f953013fc70 tx=0x7f9518008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:12.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:12.907+0000 7f952d600640 1 -- 172.21.15.141:0/3583539770 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95301594b0 con 0x7f9510005250 2024-01-16T14:58:13.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.117+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f95301594b0 con 0x7f9510005250 2024-01-16T14:58:13.099 INFO:tasks.ceph.mon.a.smithi141.stderr:2024-01-16T14:58:13.118+0000 7f723b200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T14:58:13.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.121+0000 7f952d600640 1 -- 172.21.15.141:0/3583539770 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f95301594b0 con 0x7f9510005250 2024-01-16T14:58:13.102 INFO:teuthology.orchestra.run.smithi141.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-16T14:58:13.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806b2c0 msgr2=0x7f950806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806b2c0 0x7f950806d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f953013fc70 tx=0x7f9518008040 comp rx=0 tx=0).stop 2024-01-16T14:58:13.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 msgr2=0x7f95301ee070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f95301ee070 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f952000e760 tx=0x7f952000ec20 comp rx=0 tx=0).stop 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 shutdown_connections 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806b2c0 0x7f950806d780 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9530143500 0x7f95301f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f2960 0x7f9530142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 --2- 172.21.15.141:0/3583539770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005250 0x7f95301ee070 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.124+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 >> 172.21.15.141:0/3583539770 conn(0x7f953006a800 msgr2=0x7f95300b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.125+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 shutdown_connections 2024-01-16T14:58:13.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.125+0000 7f9536a00640 1 -- 172.21.15.141:0/3583539770 wait complete. 2024-01-16T14:58:13.123 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-16T14:58:13.130 INFO:tasks.ceph.mds.a.smithi141.stderr: -3761> 2024-01-16T14:57:09.951+0000 7f7e9c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:58:13.206 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T14:58:13.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.330+0000 7fb0d7e00640 1 Processor -- start 2024-01-16T14:58:13.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.331+0000 7fb0d7e00640 1 -- start start 2024-01-16T14:58:13.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0d7e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb0d80adf20 0x7fb0d8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0d7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb0d80ccb50 0x7fb0d8143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0d7e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d815f4e0 con 0x7fb0d815bf10 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0d7e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d815f970 con 0x7fb0d80ccb50 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0d7e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d8071130 con 0x7fb0d80adf20 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0cde00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb0d815bf10 0x7fb0d80cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58386/0 (socket says 172.21.15.141:58386) 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0d7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb0d80ccb50 0x7fb0d8143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.332+0000 7fb0cde00640 1 -- 172.21.15.141:0/757982150 learned_addr learned my addr 172.21.15.141:0/757982150 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.333+0000 7fb0d6a00640 1 --2- 172.21.15.141:0/757982150 >> v2:172.21.15.188:3300/0 conn(0x7fb0d80adf20 0x7fb0d8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.333+0000 7fb0d7400640 1 -- 172.21.15.141:0/757982150 >> v2:172.21.15.188:3300/0 conn(0x7fb0d80adf20 msgr2=0x7fb0d8070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.333+0000 7fb0d7400640 1 --2- 172.21.15.141:0/757982150 >> v2:172.21.15.188:3300/0 conn(0x7fb0d80adf20 0x7fb0d8070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.333+0000 7fb0d6000640 1 -- 172.21.15.141:0/757982150 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027107450 0 0) 0x7fb0d815f4e0 con 0x7fb0d815bf10 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.333+0000 7fb0d7400640 1 -- 172.21.15.141:0/757982150 >> v1:172.21.15.141:6789/0 conn(0x7fb0d815bf10 legacy=0x7fb0d80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.333+0000 7fb0d7400640 1 -- 172.21.15.141:0/757982150 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d815c390 con 0x7fb0d80ccb50 2024-01-16T14:58:13.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.334+0000 7fb0d7400640 1 --2- 172.21.15.141:0/757982150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d80ccb50 0x7fb0d8143d30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80b4150 tx=0x7fb0d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6013d5b9ec54d03 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.334+0000 7fb0d6000640 1 -- 172.21.15.141:0/757982150 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d002e070 con 0x7fb0d80ccb50 2024-01-16T14:58:13.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.334+0000 7fb0d6000640 1 -- 172.21.15.141:0/757982150 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d00315a0 con 0x7fb0d80ccb50 2024-01-16T14:58:13.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.334+0000 7fb0d6000640 1 -- 172.21.15.141:0/757982150 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0031930 con 0x7fb0d80ccb50 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 -- 172.21.15.141:0/757982150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d80ccb50 msgr2=0x7fb0d8143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/757982150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d80ccb50 0x7fb0d8143d30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80b4150 tx=0x7fb0d002b840 comp rx=0 tx=0).stop 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 -- 172.21.15.141:0/757982150 shutdown_connections 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/757982150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d80ccb50 0x7fb0d8143d30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/757982150 >> v2:172.21.15.188:3300/0 conn(0x7fb0d80adf20 0x7fb0d8070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 -- 172.21.15.141:0/757982150 >> 172.21.15.141:0/757982150 conn(0x7fb0d806a7d0 msgr2=0x7fb0d806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 -- 172.21.15.141:0/757982150 shutdown_connections 2024-01-16T14:58:13.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.335+0000 7fb0d7e00640 1 -- 172.21.15.141:0/757982150 wait complete. 2024-01-16T14:58:13.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.337+0000 7fb0d7e00640 1 Processor -- start 2024-01-16T14:58:13.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.337+0000 7fb0d7e00640 1 -- start start 2024-01-16T14:58:13.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d80adf20 0x7fb0d81391f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0d80ccb50 0x7fb0d80ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d815bf10 0x7fb0d80ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d813ab60 con 0x7fb0d815bf10 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d814f5f0 con 0x7fb0d80adf20 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8139730 con 0x7fb0d80ccb50 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d815bf10 0x7fb0d80ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d815bf10 0x7fb0d80ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55656/0 (socket says 172.21.15.141:55656) 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.338+0000 7fb0d7400640 1 -- 172.21.15.141:0/716971359 learned_addr learned my addr 172.21.15.141:0/716971359 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d6a00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d80adf20 0x7fb0d81391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7400640 1 -- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0d80ccb50 msgr2=0x7fb0d80ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0cde00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0d80ccb50 0x7fb0d80ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7400640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0d80ccb50 0x7fb0d80ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7400640 1 -- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d80adf20 msgr2=0x7fb0d81391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:13.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7400640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d80adf20 0x7fb0d81391f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7400640 1 -- 172.21.15.141:0/716971359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0009400 con 0x7fb0d815bf10 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d6a00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d80adf20 0x7fb0d81391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7400640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d815bf10 0x7fb0d80ace10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0034040 tx=0x7fb0d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0cf400640 1 -- 172.21.15.141:0/716971359 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0004230 con 0x7fb0d815bf10 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0cf400640 1 -- 172.21.15.141:0/716971359 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0031f00 con 0x7fb0d815bf10 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0cf400640 1 -- 172.21.15.141:0/716971359 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0032640 con 0x7fb0d815bf10 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d814ed50 con 0x7fb0d815bf10 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.339+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d814f310 con 0x7fb0d815bf10 2024-01-16T14:58:13.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.341+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d8067450 con 0x7fb0d815bf10 2024-01-16T14:58:13.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.341+0000 7fb0cf400640 1 -- 172.21.15.141:0/716971359 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb0d00327a0 con 0x7fb0d815bf10 2024-01-16T14:58:13.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.342+0000 7fb0cf400640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0b006b2c0 0x7fb0b006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:13.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.342+0000 7fb0cf400640 1 -- 172.21.15.141:0/716971359 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5919+0+0 (secure 0 0 0) 0x7fb0d00abdd0 con 0x7fb0d815bf10 2024-01-16T14:58:13.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.343+0000 7fb0d6a00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0b006b2c0 0x7fb0b006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:13.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.344+0000 7fb0d6a00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0b006b2c0 0x7fb0b006d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8007330 tx=0x7fb0c800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:13.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.346+0000 7fb0cf400640 1 -- 172.21.15.141:0/716971359 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0d0076ad0 con 0x7fb0d815bf10 2024-01-16T14:58:13.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:13.570+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb0d81428a0 con 0x7fb0d815bf10 2024-01-16T14:58:14.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.125+0000 7fb0cf400640 1 -- 172.21.15.141:0/716971359 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb0d003ae60 con 0x7fb0d815bf10 2024-01-16T14:58:14.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.128+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0b006b2c0 msgr2=0x7fb0b006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.128+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0b006b2c0 0x7fb0b006d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8007330 tx=0x7fb0c800b040 comp rx=0 tx=0).stop 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d815bf10 msgr2=0x7fb0d80ace10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d815bf10 0x7fb0d80ace10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0034040 tx=0x7fb0d00046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 shutdown_connections 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb0b006b2c0 0x7fb0b006d780 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0d815bf10 0x7fb0d80ace10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0d80ccb50 0x7fb0d80ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 --2- 172.21.15.141:0/716971359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d80adf20 0x7fb0d81391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 >> 172.21.15.141:0/716971359 conn(0x7fb0d806a7d0 msgr2=0x7fb0d815b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 shutdown_connections 2024-01-16T14:58:14.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.129+0000 7fb0d7e00640 1 -- 172.21.15.141:0/716971359 wait complete. 2024-01-16T14:58:14.131 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-16T14:58:14.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.406+0000 7f52cf600640 1 Processor -- start 2024-01-16T14:58:14.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.406+0000 7f52cf600640 1 -- start start 2024-01-16T14:58:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.407+0000 7f52cf600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.407+0000 7f52cf600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52c80cf7d0 0x7f52c815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52cf600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c80cfc30 con 0x7f52c815e1e0 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52cf600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c80b1ee0 con 0x7f52c80cf7d0 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52cf600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80b2780 con 0x7f52c8143ae0 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52c7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52c80cf7d0 0x7f52c815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52cd600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f52c815e1e0 0x7f52c815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58394/0 (socket says 172.21.15.141:58394) 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52cd600640 1 -- 172.21.15.141:0/2881136742 learned_addr learned my addr 172.21.15.141:0/2881136742 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52ccc00640 1 --2- 172.21.15.141:0/2881136742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52c7e00640 1 -- 172.21.15.141:0/2881136742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 msgr2=0x7f52c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52c7e00640 1 --2- 172.21.15.141:0/2881136742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52c7e00640 1 -- 172.21.15.141:0/2881136742 >> v1:172.21.15.141:6789/0 conn(0x7f52c815e1e0 legacy=0x7f52c815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52c7e00640 1 -- 172.21.15.141:0/2881136742 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c815e990 con 0x7f52c80cf7d0 2024-01-16T14:58:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.408+0000 7f52ccc00640 1 --2- 172.21.15.141:0/2881136742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:14.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.409+0000 7f52c7e00640 1 --2- 172.21.15.141:0/2881136742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c80cf7d0 0x7f52c815dca0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f52b00097a0 tx=0x7f52b002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1204269354f0f545 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:14.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52c7400640 1 -- 172.21.15.141:0/2881136742 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b002d040 con 0x7f52c80cf7d0 2024-01-16T14:58:14.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52c7400640 1 -- 172.21.15.141:0/2881136742 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0031810 con 0x7f52c80cf7d0 2024-01-16T14:58:14.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52c7400640 1 -- 172.21.15.141:0/2881136742 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0031ba0 con 0x7f52c80cf7d0 2024-01-16T14:58:14.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52cf600640 1 -- 172.21.15.141:0/2881136742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c80cf7d0 msgr2=0x7f52c815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52cf600640 1 --2- 172.21.15.141:0/2881136742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c80cf7d0 0x7f52c815dca0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f52b00097a0 tx=0x7f52b002b8b0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52cf600640 1 -- 172.21.15.141:0/2881136742 shutdown_connections 2024-01-16T14:58:14.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52cf600640 1 --2- 172.21.15.141:0/2881136742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c80cf7d0 0x7f52c815dca0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52cf600640 1 --2- 172.21.15.141:0/2881136742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.410+0000 7f52cf600640 1 -- 172.21.15.141:0/2881136742 >> 172.21.15.141:0/2881136742 conn(0x7f52c806a800 msgr2=0x7f52c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:14.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.411+0000 7f52cf600640 1 -- 172.21.15.141:0/2881136742 shutdown_connections 2024-01-16T14:58:14.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.411+0000 7f52cf600640 1 -- 172.21.15.141:0/2881136742 wait complete. 2024-01-16T14:58:14.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.413+0000 7f52cf600640 1 Processor -- start 2024-01-16T14:58:14.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.413+0000 7f52cf600640 1 -- start start 2024-01-16T14:58:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cf600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52c80cf7d0 0x7f52c815cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cf600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c813c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cf600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c815e1e0 0x7f52c813ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cf600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c813dd70 con 0x7f52c815e1e0 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cf600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8139110 con 0x7f52c80cf7d0 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cf600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8138500 con 0x7f52c8143ae0 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c815e1e0 0x7f52c813ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c815e1e0 0x7f52c813ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55680/0 (socket says 172.21.15.141:55680) 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cd600640 1 -- 172.21.15.141:0/1440542994 learned_addr learned my addr 172.21.15.141:0/1440542994 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52ccc00640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52c80cf7d0 0x7f52c815cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52c7e00640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c813c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cd600640 1 -- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 msgr2=0x7f52c813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cd600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c813c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.414+0000 7f52cd600640 1 -- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52c80cf7d0 msgr2=0x7f52c815cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52cd600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52c80cf7d0 0x7f52c815cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52cd600640 1 -- 172.21.15.141:0/1440542994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0009400 con 0x7f52c815e1e0 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52ccc00640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52c80cf7d0 0x7f52c815cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52cd600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c815e1e0 0x7f52c813ca00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f52c000ca80 tx=0x7f52c000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52c4c00640 1 -- 172.21.15.141:0/1440542994 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c000ac30 con 0x7f52c815e1e0 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52c4c00640 1 -- 172.21.15.141:0/1440542994 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c000ad90 con 0x7f52c815e1e0 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c81387e0 con 0x7f52c815e1e0 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52c4c00640 1 -- 172.21.15.141:0/1440542994 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0012600 con 0x7f52c815e1e0 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.415+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c8138da0 con 0x7f52c815e1e0 2024-01-16T14:58:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.416+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c8139280 con 0x7f52c815e1e0 2024-01-16T14:58:14.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.417+0000 7f52c4c00640 1 -- 172.21.15.141:0/1440542994 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f52c0007070 con 0x7f52c815e1e0 2024-01-16T14:58:14.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.418+0000 7f52c4c00640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a006b2c0 0x7f52a006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:14.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.418+0000 7f52c4c00640 1 -- 172.21.15.141:0/1440542994 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5919+0+0 (secure 0 0 0) 0x7f52c0089b10 con 0x7f52c815e1e0 2024-01-16T14:58:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.419+0000 7f52ccc00640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a006b2c0 0x7f52a006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:14.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.420+0000 7f52ccc00640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a006b2c0 0x7f52a006d780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f52c815d470 tx=0x7f52b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:14.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.422+0000 7f52c4c00640 1 -- 172.21.15.141:0/1440542994 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52c00548c0 con 0x7f52c815e1e0 2024-01-16T14:58:14.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:14.613+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52c80675d0 con 0x7f52c815e1e0 2024-01-16T14:58:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.134+0000 7f52c4c00640 1 -- 172.21.15.141:0/1440542994 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7f52c0017020 con 0x7f52c815e1e0 2024-01-16T14:58:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' removed 2024-01-16T14:58:15.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.138+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a006b2c0 msgr2=0x7f52a006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.138+0000 7f52cf600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a006b2c0 0x7f52a006d780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f52c815d470 tx=0x7f52b8008040 comp rx=0 tx=0).stop 2024-01-16T14:58:15.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.138+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c815e1e0 msgr2=0x7f52c813ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.138+0000 7f52cf600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c815e1e0 0x7f52c813ca00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f52c000ca80 tx=0x7f52c000cf40 comp rx=0 tx=0).stop 2024-01-16T14:58:15.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.139+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 shutdown_connections 2024-01-16T14:58:15.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.139+0000 7f52cf600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a006b2c0 0x7f52a006d780 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.139+0000 7f52cf600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52c815e1e0 0x7f52c813ca00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.139+0000 7f52cf600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52c8143ae0 0x7f52c813c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.139+0000 7f52cf600640 1 --2- 172.21.15.141:0/1440542994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52c80cf7d0 0x7f52c815cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.139+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 >> 172.21.15.141:0/1440542994 conn(0x7f52c806a800 msgr2=0x7f52c806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:15.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.140+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 shutdown_connections 2024-01-16T14:58:15.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.140+0000 7f52cf600640 1 -- 172.21.15.141:0/1440542994 wait complete. 2024-01-16T14:58:15.144 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.367+0000 7f1c27200640 1 Processor -- start 2024-01-16T14:58:15.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.367+0000 7f1c27200640 1 -- start start 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c27200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1c200ad880 0x7f1c200717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c27200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c200ac8d0 0x7f1c2006b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c27200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2015e680 con 0x7f1c2006b930 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c27200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2006bdc0 con 0x7f1c200ad880 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c27200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c20071cf0 con 0x7f1c200ac8d0 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c1fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1c200ad880 0x7f1c200717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c25200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1c2006b930 0x7f1c20160ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58402/0 (socket says 172.21.15.141:58402) 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.369+0000 7f1c25200640 1 -- 172.21.15.141:0/1891996439 learned_addr learned my addr 172.21.15.141:0/1891996439 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.370+0000 7f1c1f400640 1 --2- 172.21.15.141:0/1891996439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c200ac8d0 0x7f1c2006b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.370+0000 7f1c1fe00640 1 -- 172.21.15.141:0/1891996439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c200ac8d0 msgr2=0x7f1c2006b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.370+0000 7f1c1fe00640 1 --2- 172.21.15.141:0/1891996439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c200ac8d0 0x7f1c2006b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.370+0000 7f1c1fe00640 1 -- 172.21.15.141:0/1891996439 >> v1:172.21.15.141:6789/0 conn(0x7f1c2006b930 legacy=0x7f1c20160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.370+0000 7f1c1fe00640 1 -- 172.21.15.141:0/1891996439 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2015dd50 con 0x7f1c200ad880 2024-01-16T14:58:15.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.370+0000 7f1c1ea00640 1 -- 172.21.15.141:0/1891996439 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529182349 0 0) 0x7f1c2015e680 con 0x7f1c2006b930 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.371+0000 7f1c1fe00640 1 --2- 172.21.15.141:0/1891996439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ad880 0x7f1c200717b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1c100097a0 tx=0x7f1c1002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557d1e24bc9903fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.371+0000 7f1c1ea00640 1 -- 172.21.15.141:0/1891996439 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10030610 con 0x7f1c200ad880 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.371+0000 7f1c1ea00640 1 -- 172.21.15.141:0/1891996439 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c10030770 con 0x7f1c200ad880 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.371+0000 7f1c1ea00640 1 -- 172.21.15.141:0/1891996439 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10030b00 con 0x7f1c200ad880 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 -- 172.21.15.141:0/1891996439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ad880 msgr2=0x7f1c200717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 --2- 172.21.15.141:0/1891996439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ad880 0x7f1c200717b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1c100097a0 tx=0x7f1c1002b960 comp rx=0 tx=0).stop 2024-01-16T14:58:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 -- 172.21.15.141:0/1891996439 shutdown_connections 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 --2- 172.21.15.141:0/1891996439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c200ac8d0 0x7f1c2006b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 --2- 172.21.15.141:0/1891996439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ad880 0x7f1c200717b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 -- 172.21.15.141:0/1891996439 >> 172.21.15.141:0/1891996439 conn(0x7f1c20070a80 msgr2=0x7f1c20070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 -- 172.21.15.141:0/1891996439 shutdown_connections 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.372+0000 7f1c27200640 1 -- 172.21.15.141:0/1891996439 wait complete. 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.374+0000 7f1c27200640 1 Processor -- start 2024-01-16T14:58:15.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.374+0000 7f1c27200640 1 -- start start 2024-01-16T14:58:15.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c27200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c2006b930 0x7f1c201f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c27200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ac8d0 0x7f1c200b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c27200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200ad880 0x7f1c2006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c27200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200cea70 con 0x7f1c200ac8d0 2024-01-16T14:58:15.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c27200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c201f2d60 con 0x7f1c200ad880 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c27200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200b0ce0 con 0x7f1c2006b930 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c1f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ac8d0 0x7f1c200b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c1f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ac8d0 0x7f1c200b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55704/0 (socket says 172.21.15.141:55704) 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c1f400640 1 -- 172.21.15.141:0/1177617104 learned_addr learned my addr 172.21.15.141:0/1177617104 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c1fe00640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c2006b930 0x7f1c201f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c1f400640 1 -- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c2006b930 msgr2=0x7f1c201f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c1f400640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c2006b930 0x7f1c201f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c25200640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200ad880 0x7f1c2006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.375+0000 7f1c1f400640 1 -- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200ad880 msgr2=0x7f1c2006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:15.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c1f400640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200ad880 0x7f1c2006dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c1f400640 1 -- 172.21.15.141:0/1177617104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10009400 con 0x7f1c200ac8d0 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c1fe00640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c2006b930 0x7f1c201f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c1f400640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ac8d0 0x7f1c200b3c30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800ce40 tx=0x7f1c0800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c17e00640 1 -- 172.21.15.141:0/1177617104 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08003b90 con 0x7f1c200ac8d0 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c17e00640 1 -- 172.21.15.141:0/1177617104 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c08003cf0 con 0x7f1c200ac8d0 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2014d800 con 0x7f1c200ac8d0 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c17e00640 1 -- 172.21.15.141:0/1177617104 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08004240 con 0x7f1c200ac8d0 2024-01-16T14:58:15.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.376+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c200ce4b0 con 0x7f1c200ac8d0 2024-01-16T14:58:15.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.377+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c200675d0 con 0x7f1c200ac8d0 2024-01-16T14:58:15.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.378+0000 7f1c17e00640 1 -- 172.21.15.141:0/1177617104 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1c080043a0 con 0x7f1c200ac8d0 2024-01-16T14:58:15.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.379+0000 7f1c17e00640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1bf806b2c0 0x7f1bf806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:15.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.379+0000 7f1c17e00640 1 -- 172.21.15.141:0/1177617104 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5481+0+0 (secure 0 0 0) 0x7f1c08012070 con 0x7f1c200ac8d0 2024-01-16T14:58:15.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.380+0000 7f1c1fe00640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1bf806b2c0 0x7f1bf806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:15.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.381+0000 7f1c1fe00640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1bf806b2c0 0x7f1bf806d780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1c10036000 tx=0x7f1c1003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:15.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.383+0000 7f1c17e00640 1 -- 172.21.15.141:0/1177617104 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c08055490 con 0x7f1c200ac8d0 2024-01-16T14:58:15.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:15.579+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1c200aeb60 con 0x7f1c200ac8d0 2024-01-16T14:58:16.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.145+0000 7f1c17e00640 1 -- 172.21.15.141:0/1177617104 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1c08003e60 con 0x7f1c200ac8d0 2024-01-16T14:58:16.127 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' removed 2024-01-16T14:58:16.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.150+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1bf806b2c0 msgr2=0x7f1bf806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.150+0000 7f1c27200640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1bf806b2c0 0x7f1bf806d780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1c10036000 tx=0x7f1c1003b040 comp rx=0 tx=0).stop 2024-01-16T14:58:16.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.150+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ac8d0 msgr2=0x7f1c200b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.150+0000 7f1c27200640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ac8d0 0x7f1c200b3c30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800ce40 tx=0x7f1c0800a660 comp rx=0 tx=0).stop 2024-01-16T14:58:16.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.152+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 shutdown_connections 2024-01-16T14:58:16.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.152+0000 7f1c27200640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1bf806b2c0 0x7f1bf806d780 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.152+0000 7f1c27200640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200ad880 0x7f1c2006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.152+0000 7f1c27200640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c200ac8d0 0x7f1c200b3c30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.152+0000 7f1c27200640 1 --2- 172.21.15.141:0/1177617104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c2006b930 0x7f1c201f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.152+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 >> 172.21.15.141:0/1177617104 conn(0x7f1c20070a80 msgr2=0x7f1c20160e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:16.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.153+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 shutdown_connections 2024-01-16T14:58:16.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.154+0000 7f1c27200640 1 -- 172.21.15.141:0/1177617104 wait complete. 2024-01-16T14:58:16.157 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T14:58:16.157 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T14:58:16.158 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T14:58:16.158 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:58:16.158 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T14:58:16.158 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T14:58:16.158 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T14:58:16.158 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:58:16.159 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T14:58:16.159 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T14:58:16.159 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T14:58:16.159 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:58:16.159 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T14:58:16.159 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T14:58:16.159 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T14:58:16.160 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:58:16.181 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T14:58:16.181 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T14:58:16.184 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T14:58:16.184 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T14:58:16.185 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T14:58:16.185 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T14:58:16.186 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T14:58:16.186 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T14:58:16.231 INFO:tasks.ceph.mds.a:Started 2024-01-16T14:58:16.231 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:16.233 INFO:tasks.ceph.mds.d:Started 2024-01-16T14:58:16.233 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:16.237 INFO:tasks.ceph.mds.c:Started 2024-01-16T14:58:16.237 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:16.239 INFO:tasks.ceph.mds.b:Started 2024-01-16T14:58:16.239 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:16.239 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T14:58:16.395 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T14:58:16.395 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T14:58:16.402 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T14:58:16.403 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T14:58:16.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.454+0000 7f09e4a00640 1 Processor -- start 2024-01-16T14:58:16.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.454+0000 7f09e4a00640 1 -- start start 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.455+0000 7f09e4a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09e0160920 0x7f09e00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.455+0000 7f09e4a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f09e00cf9f0 0x7f09e00cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.455+0000 7f09e4a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e00d0a10 con 0x7f09e0161010 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.455+0000 7f09e4a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e00d03e0 con 0x7f09e0160920 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.455+0000 7f09e4a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e00d1f50 con 0x7f09e00cf9f0 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.455+0000 7f09de800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09e0160920 0x7f09e00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09de800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09e0160920 0x7f09e00b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55730/0 (socket says 172.21.15.141:55730) 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09de800640 1 -- 172.21.15.141:0/1312352463 learned_addr learned my addr 172.21.15.141:0/1312352463 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09df200640 1 --1- 172.21.15.141:0/1312352463 >> v1:172.21.15.141:6789/0 conn(0x7f09e0161010 0x7f09e0161480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58406/0 (socket says 172.21.15.141:58406) 2024-01-16T14:58:16.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09d5c00640 1 --2- 172.21.15.141:0/1312352463 >> v2:172.21.15.188:3300/0 conn(0x7f09e00cf9f0 0x7f09e00cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09de800640 1 -- 172.21.15.141:0/1312352463 >> v2:172.21.15.188:3300/0 conn(0x7f09e00cf9f0 msgr2=0x7f09e00cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09de800640 1 --2- 172.21.15.141:0/1312352463 >> v2:172.21.15.188:3300/0 conn(0x7f09e00cf9f0 0x7f09e00cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09de800640 1 -- 172.21.15.141:0/1312352463 >> v1:172.21.15.141:6789/0 conn(0x7f09e0161010 legacy=0x7f09e0161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09dde00640 1 -- 172.21.15.141:0/1312352463 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211980436 0 0) 0x7f09e00d0a10 con 0x7f09e0161010 2024-01-16T14:58:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.456+0000 7f09de800640 1 -- 172.21.15.141:0/1312352463 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00cf080 con 0x7f09e0160920 2024-01-16T14:58:16.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.457+0000 7f09de800640 1 --2- 172.21.15.141:0/1312352463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0160920 0x7f09e00b3f30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f09e008c1f0 tx=0x7f09d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a590ab52e5f504 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.457+0000 7f09dde00640 1 -- 172.21.15.141:0/1312352463 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0030860 con 0x7f09e0160920 2024-01-16T14:58:16.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.457+0000 7f09dde00640 1 -- 172.21.15.141:0/1312352463 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d00309c0 con 0x7f09e0160920 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.457+0000 7f09dde00640 1 -- 172.21.15.141:0/1312352463 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0030d50 con 0x7f09e0160920 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 -- 172.21.15.141:0/1312352463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0160920 msgr2=0x7f09e00b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 --2- 172.21.15.141:0/1312352463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0160920 0x7f09e00b3f30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f09e008c1f0 tx=0x7f09d002b9a0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 -- 172.21.15.141:0/1312352463 shutdown_connections 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 --2- 172.21.15.141:0/1312352463 >> v2:172.21.15.188:3300/0 conn(0x7f09e00cf9f0 0x7f09e00cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 --2- 172.21.15.141:0/1312352463 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0160920 0x7f09e00b3f30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 -- 172.21.15.141:0/1312352463 >> 172.21.15.141:0/1312352463 conn(0x7f09e00d3090 msgr2=0x7f09e00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 -- 172.21.15.141:0/1312352463 shutdown_connections 2024-01-16T14:58:16.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.458+0000 7f09e4a00640 1 -- 172.21.15.141:0/1312352463 wait complete. 2024-01-16T14:58:16.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.460+0000 7f09e4a00640 1 Processor -- start 2024-01-16T14:58:16.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.460+0000 7f09e4a00640 1 -- start start 2024-01-16T14:58:16.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09e4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e00cf9f0 0x7f09e0157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09e4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09e0160920 0x7f09e00ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09e4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0161010 0x7f09e00c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09e4a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00cb360 con 0x7f09e0161010 2024-01-16T14:58:16.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09e4a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00b5c50 con 0x7f09e00cf9f0 2024-01-16T14:58:16.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09e4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00b5ea0 con 0x7f09e0160920 2024-01-16T14:58:16.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09df200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0161010 0x7f09e00c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09de800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e00cf9f0 0x7f09e0157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09d5c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09e0160920 0x7f09e00ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09d5c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09e0160920 0x7f09e00ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:42012/0 (socket says 172.21.15.141:42012) 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09de800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e00cf9f0 0x7f09e0157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:51232/0 (socket says 172.21.15.141:51232) 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09d5c00640 1 -- 172.21.15.141:0/2420965226 learned_addr learned my addr 172.21.15.141:0/2420965226 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.461+0000 7f09df200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0161010 0x7f09e00c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55738/0 (socket says 172.21.15.141:55738) 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09d5c00640 1 -- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e00cf9f0 msgr2=0x7f09e0157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09d5c00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e00cf9f0 0x7f09e0157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09d5c00640 1 -- 172.21.15.141:0/2420965226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0161010 msgr2=0x7f09e00c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09d5c00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0161010 0x7f09e00c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09d5c00640 1 -- 172.21.15.141:0/2420965226 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00094d0 con 0x7f09e0160920 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09df200640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0161010 0x7f09e00c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:16.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09d5c00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09e0160920 0x7f09e00ca8c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f09c800e870 tx=0x7f09c800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.462+0000 7f09d7400640 1 -- 172.21.15.141:0/2420965226 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80094f0 con 0x7f09e0160920 2024-01-16T14:58:16.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.463+0000 7f09d7400640 1 -- 172.21.15.141:0/2420965226 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c800c940 con 0x7f09e0160920 2024-01-16T14:58:16.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.463+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e00b6180 con 0x7f09e0160920 2024-01-16T14:58:16.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.463+0000 7f09d7400640 1 -- 172.21.15.141:0/2420965226 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c800ccf0 con 0x7f09e0160920 2024-01-16T14:58:16.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.463+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e00b6740 con 0x7f09e0160920 2024-01-16T14:58:16.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.464+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e0067450 con 0x7f09e0160920 2024-01-16T14:58:16.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.465+0000 7f09d7400640 1 -- 172.21.15.141:0/2420965226 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f09c8005990 con 0x7f09e0160920 2024-01-16T14:58:16.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.466+0000 7f09d7400640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09b806b2f0 0x7f09b806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:16.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.466+0000 7f09d7400640 1 -- 172.21.15.141:0/2420965226 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09c8089bf0 con 0x7f09e0160920 2024-01-16T14:58:16.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.466+0000 7f09de800640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09b806b2f0 0x7f09b806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:16.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.467+0000 7f09de800640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09b806b2f0 0x7f09b806d7b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f09d0036000 tx=0x7f09d0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:16.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.471+0000 7f09d7400640 1 -- 172.21.15.141:0/2420965226 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09c8054e80 con 0x7f09e0160920 2024-01-16T14:58:16.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:16.674+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f09e00ce790 con 0x7f09e0160920 2024-01-16T14:58:17.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.181+0000 7f09d7400640 1 -- 172.21.15.141:0/2420965226 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7f09c8021080 con 0x7f09e0160920 2024-01-16T14:58:17.163 INFO:teuthology.orchestra.run.smithi141.stderr: removed all blocklist entries 2024-01-16T14:58:17.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09b806b2f0 msgr2=0x7f09b806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09b806b2f0 0x7f09b806d7b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f09d0036000 tx=0x7f09d0008030 comp rx=0 tx=0).stop 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09e0160920 msgr2=0x7f09e00ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09e0160920 0x7f09e00ca8c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f09c800e870 tx=0x7f09c800ed30 comp rx=0 tx=0).stop 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 shutdown_connections 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09b806b2f0 0x7f09b806d7b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09e0161010 0x7f09e00c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09e0160920 0x7f09e00ca8c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 --2- 172.21.15.141:0/2420965226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e00cf9f0 0x7f09e0157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 >> 172.21.15.141:0/2420965226 conn(0x7f09e00d3090 msgr2=0x7f09e0161460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 shutdown_connections 2024-01-16T14:58:17.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.185+0000 7f09e4a00640 1 -- 172.21.15.141:0/2420965226 wait complete. 2024-01-16T14:58:17.188 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T14:58:17.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.417+0000 7fcc1f200640 1 Processor -- start 2024-01-16T14:58:17.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.417+0000 7fcc1f200640 1 -- start start 2024-01-16T14:58:17.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcc20160c70 0x7fcc200d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcc200c6be0 0x7fcc200cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc200cf6a0 con 0x7fcc20160890 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1f200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc200d1430 con 0x7fcc200c6be0 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1f200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc200b4370 con 0x7fcc20160c70 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1e800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcc200c6be0 0x7fcc200cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1e800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcc200c6be0 0x7fcc200cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55746/0 (socket says 172.21.15.141:55746) 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.419+0000 7fcc1e800640 1 -- 172.21.15.141:0/239920046 learned_addr learned my addr 172.21.15.141:0/239920046 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:17.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.420+0000 7fcc1d400640 1 --2- 172.21.15.141:0/239920046 >> v2:172.21.15.188:3300/0 conn(0x7fcc20160c70 0x7fcc200d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.420+0000 7fcc1e800640 1 -- 172.21.15.141:0/239920046 >> v2:172.21.15.188:3300/0 conn(0x7fcc20160c70 msgr2=0x7fcc200d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:17.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.420+0000 7fcc1e800640 1 --2- 172.21.15.141:0/239920046 >> v2:172.21.15.188:3300/0 conn(0x7fcc20160c70 0x7fcc200d3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.420+0000 7fcc1e800640 1 -- 172.21.15.141:0/239920046 >> v1:172.21.15.188:6789/0 conn(0x7fcc20160890 legacy=0x7fcc200d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:17.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.420+0000 7fcc1e800640 1 -- 172.21.15.141:0/239920046 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc200ce7c0 con 0x7fcc200c6be0 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.421+0000 7fcc1e800640 1 --2- 172.21.15.141:0/239920046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc200c6be0 0x7fcc200cfb30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcc200b5c80 tx=0x7fcc1802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4fb4389e434f151 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.421+0000 7fcc1ca00640 1 -- 172.21.15.141:0/239920046 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18031750 con 0x7fcc200c6be0 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.421+0000 7fcc1ca00640 1 -- 172.21.15.141:0/239920046 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc180318b0 con 0x7fcc200c6be0 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.421+0000 7fcc1ca00640 1 -- 172.21.15.141:0/239920046 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18031c40 con 0x7fcc200c6be0 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 -- 172.21.15.141:0/239920046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc200c6be0 msgr2=0x7fcc200cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 --2- 172.21.15.141:0/239920046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc200c6be0 0x7fcc200cfb30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcc200b5c80 tx=0x7fcc1802b7f0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 -- 172.21.15.141:0/239920046 shutdown_connections 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 --2- 172.21.15.141:0/239920046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc200c6be0 0x7fcc200cfb30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 --2- 172.21.15.141:0/239920046 >> v2:172.21.15.188:3300/0 conn(0x7fcc20160c70 0x7fcc200d3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 -- 172.21.15.141:0/239920046 >> 172.21.15.141:0/239920046 conn(0x7fcc200d3090 msgr2=0x7fcc200d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 -- 172.21.15.141:0/239920046 shutdown_connections 2024-01-16T14:58:17.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.422+0000 7fcc1f200640 1 -- 172.21.15.141:0/239920046 wait complete. 2024-01-16T14:58:17.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.424+0000 7fcc1f200640 1 Processor -- start 2024-01-16T14:58:17.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.424+0000 7fcc1f200640 1 -- start start 2024-01-16T14:58:17.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc200c6be0 0x7fcc200ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc20160890 0x7fcc200cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc20160c70 0x7fcc201589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2015db10 con 0x7fcc20160890 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc20158ee0 con 0x7fcc20160c70 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc200ca740 con 0x7fcc200c6be0 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc20160890 0x7fcc200cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc20160890 0x7fcc200cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55748/0 (socket says 172.21.15.141:55748) 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.425+0000 7fcc1d400640 1 -- 172.21.15.141:0/1795313524 learned_addr learned my addr 172.21.15.141:0/1795313524 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1e800640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc20160c70 0x7fcc201589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1d400640 1 -- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc200c6be0 msgr2=0x7fcc200ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1de00640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc200c6be0 0x7fcc200ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1d400640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc200c6be0 0x7fcc200ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1d400640 1 -- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc20160c70 msgr2=0x7fcc201589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1d400640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc20160c70 0x7fcc201589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1d400640 1 -- 172.21.15.141:0/1795313524 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc180094d0 con 0x7fcc20160890 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1e800640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc20160c70 0x7fcc201589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1d400640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc20160890 0x7fcc200cb7e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800ce40 tx=0x7fcc0800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc16000640 1 -- 172.21.15.141:0/1795313524 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08003b90 con 0x7fcc20160890 2024-01-16T14:58:17.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc16000640 1 -- 172.21.15.141:0/1795313524 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08003cf0 con 0x7fcc20160890 2024-01-16T14:58:17.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc16000640 1 -- 172.21.15.141:0/1795313524 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08004240 con 0x7fcc20160890 2024-01-16T14:58:17.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.426+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc200caa20 con 0x7fcc20160890 2024-01-16T14:58:17.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.427+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc200cafe0 con 0x7fcc20160890 2024-01-16T14:58:17.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.428+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2006b5a0 con 0x7fcc20160890 2024-01-16T14:58:17.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.429+0000 7fcc16000640 1 -- 172.21.15.141:0/1795313524 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcc080043a0 con 0x7fcc20160890 2024-01-16T14:58:17.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.429+0000 7fcc16000640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcbf806b2c0 0x7fcbf806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.429+0000 7fcc16000640 1 -- 172.21.15.141:0/1795313524 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcc08012070 con 0x7fcc20160890 2024-01-16T14:58:17.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.430+0000 7fcc1de00640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcbf806b2c0 0x7fcbf806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.431+0000 7fcc1de00640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcbf806b2c0 0x7fcbf806d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000a750 tx=0x7fcc10008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.434+0000 7fcc16000640 1 -- 172.21.15.141:0/1795313524 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc08055490 con 0x7fcc20160890 2024-01-16T14:58:17.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.690+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcc20067450 con 0x7fcc20160890 2024-01-16T14:58:17.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.691+0000 7fcc16000640 1 -- 172.21.15.141:0/1795313524 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fcc08003e60 con 0x7fcc20160890 2024-01-16T14:58:17.672 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:17.672 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:58:17.672 INFO:teuthology.orchestra.run.smithi141.stdout: "auth_dump": [ 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.a", 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ==", 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.673 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.b", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg==", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.c", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw==", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.674 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.d", 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg==", 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T14:58:17.675 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.0", 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA==", 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.676 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.1", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg==", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.2", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g==", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.677 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.3", 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w==", 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.4", 2024-01-16T14:58:17.678 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ==", 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.5", 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA==", 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.679 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.6", 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ==", 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.680 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.7", 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg==", 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.0", 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg==", 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.1", 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ==", 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T14:58:17.682 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.2", 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ==", 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.3", 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA==", 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.admin", 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAEmaZl0VgDMRAABOWZ0senIXHFvij8FQHiAg==", 2024-01-16T14:58:17.684 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.bootstrap-mds", 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAvmaZl3K94FRAA1NCxLl0JiVIfsG4JWxRPaA==", 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile bootstrap-mds" 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.bootstrap-mgr", 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAvmaZlQLl4FRAAJr9OCQ8SqIWEKQZBLFko2Q==", 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.bootstrap-osd", 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAvmaZlc8F4FRAAx036WqZsjYzhuIkmEf1+bQ==", 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.686 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile bootstrap-osd" 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.bootstrap-rbd", 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAvmaZl0cl4FRAAITkLWxPTokL+N8k3PzHIOQ==", 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-16T14:58:17.687 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAvmaZl49R4FRAAd8MT4yjWfD3hBS8wD8Y5Nw==", 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.bootstrap-rgw", 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAvmaZlv914FRAAYz0dDhocEe24stTPzNIKrA==", 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.x", 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ==", 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.y", 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA==", 2024-01-16T14:58:17.689 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T14:58:17.690 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T14:58:17.690 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T14:58:17.690 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T14:58:17.690 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.690 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:17.690 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2024-01-16T14:58:17.690 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:58:17.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcbf806b2c0 msgr2=0x7fcbf806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcbf806b2c0 0x7fcbf806d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000a750 tx=0x7fcc10008040 comp rx=0 tx=0).stop 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc20160890 msgr2=0x7fcc200cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc20160890 0x7fcc200cb7e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800ce40 tx=0x7fcc0800a660 comp rx=0 tx=0).stop 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 shutdown_connections 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcbf806b2c0 0x7fcbf806d780 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc20160c70 0x7fcc201589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc20160890 0x7fcc200cb7e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 --2- 172.21.15.141:0/1795313524 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc200c6be0 0x7fcc200ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.693+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 >> 172.21.15.141:0/1795313524 conn(0x7fcc200d3090 msgr2=0x7fcc2006ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.694+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 shutdown_connections 2024-01-16T14:58:17.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.694+0000 7fcc1f200640 1 -- 172.21.15.141:0/1795313524 wait complete. 2024-01-16T14:58:17.694 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-16T14:58:17.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.918+0000 7f20db600640 1 Processor -- start 2024-01-16T14:58:17.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.918+0000 7f20db600640 1 -- start start 2024-01-16T14:58:17.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.920+0000 7f20db600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f20d4143ae0 0x7f20d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.920+0000 7f20db600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f20d415e3b0 0x7f20d40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.920+0000 7f20db600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d40cfc30 con 0x7f20d40cf7d0 2024-01-16T14:58:17.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.920+0000 7f20db600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d40b1ee0 con 0x7f20d4143ae0 2024-01-16T14:58:17.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.920+0000 7f20db600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d40b2780 con 0x7f20d415e3b0 2024-01-16T14:58:17.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.920+0000 7f20d8c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f20d4143ae0 0x7f20d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.921+0000 7f20d8c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f20d4143ae0 0x7f20d40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55758/0 (socket says 172.21.15.141:55758) 2024-01-16T14:58:17.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.921+0000 7f20d8c00640 1 -- 172.21.15.141:0/1051295689 learned_addr learned my addr 172.21.15.141:0/1051295689 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:17.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.921+0000 7f20d8c00640 1 -- 172.21.15.141:0/1051295689 >> v2:172.21.15.188:3300/0 conn(0x7f20d415e3b0 msgr2=0x7f20d40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:17.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.921+0000 7f20d8c00640 1 --2- 172.21.15.141:0/1051295689 >> v2:172.21.15.188:3300/0 conn(0x7f20d415e3b0 0x7f20d40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.921+0000 7f20d8c00640 1 -- 172.21.15.141:0/1051295689 >> v1:172.21.15.141:6789/0 conn(0x7f20d40cf7d0 legacy=0x7f20d415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.921+0000 7f20d8c00640 1 -- 172.21.15.141:0/1051295689 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d415e990 con 0x7f20d4143ae0 2024-01-16T14:58:17.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.922+0000 7f20d8c00640 1 --2- 172.21.15.141:0/1051295689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d4143ae0 0x7f20d40cf290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f20d406e4f0 tx=0x7f20bc02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8f5712b8ffaf95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.922+0000 7f20d3400640 1 -- 172.21.15.141:0/1051295689 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc030880 con 0x7f20d4143ae0 2024-01-16T14:58:17.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.922+0000 7f20d3400640 1 -- 172.21.15.141:0/1051295689 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc0309e0 con 0x7f20d4143ae0 2024-01-16T14:58:17.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.922+0000 7f20d3400640 1 -- 172.21.15.141:0/1051295689 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc030d70 con 0x7f20d4143ae0 2024-01-16T14:58:17.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 -- 172.21.15.141:0/1051295689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d4143ae0 msgr2=0x7f20d40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 --2- 172.21.15.141:0/1051295689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d4143ae0 0x7f20d40cf290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f20d406e4f0 tx=0x7f20bc02b9c0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 -- 172.21.15.141:0/1051295689 shutdown_connections 2024-01-16T14:58:17.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 --2- 172.21.15.141:0/1051295689 >> v2:172.21.15.188:3300/0 conn(0x7f20d415e3b0 0x7f20d40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 --2- 172.21.15.141:0/1051295689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d4143ae0 0x7f20d40cf290 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 -- 172.21.15.141:0/1051295689 >> 172.21.15.141:0/1051295689 conn(0x7f20d406a800 msgr2=0x7f20d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:17.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 -- 172.21.15.141:0/1051295689 shutdown_connections 2024-01-16T14:58:17.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.923+0000 7f20db600640 1 -- 172.21.15.141:0/1051295689 wait complete. 2024-01-16T14:58:17.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.925+0000 7f20db600640 1 Processor -- start 2024-01-16T14:58:17.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.925+0000 7f20db600640 1 -- start start 2024-01-16T14:58:17.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20db600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d40cf7d0 0x7f20d413eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20db600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20d4143ae0 0x7f20d41f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20db600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d415e3b0 0x7f20d40adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20db600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d40cc960 con 0x7f20d40cf7d0 2024-01-16T14:58:17.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20db600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d413f190 con 0x7f20d415e3b0 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20db600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d413f470 con 0x7f20d4143ae0 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20d8c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d40cf7d0 0x7f20d413eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20d8c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d40cf7d0 0x7f20d413eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55768/0 (socket says 172.21.15.141:55768) 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.926+0000 7f20d8c00640 1 -- 172.21.15.141:0/2262350675 learned_addr learned my addr 172.21.15.141:0/2262350675 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d9600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d415e3b0 0x7f20d40adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d8c00640 1 -- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20d4143ae0 msgr2=0x7f20d41f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d3e00640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20d4143ae0 0x7f20d41f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d8c00640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20d4143ae0 0x7f20d41f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d8c00640 1 -- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d415e3b0 msgr2=0x7f20d40adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d8c00640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d415e3b0 0x7f20d40adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d8c00640 1 -- 172.21.15.141:0/2262350675 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc0094d0 con 0x7f20d40cf7d0 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d9600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d415e3b0 0x7f20d40adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:17.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d8c00640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d40cf7d0 0x7f20d413eb90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f20bc02ba00 tx=0x7f20bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.927+0000 7f20d0c00640 1 -- 172.21.15.141:0/2262350675 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc030870 con 0x7f20d40cf7d0 2024-01-16T14:58:17.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.928+0000 7f20d0c00640 1 -- 172.21.15.141:0/2262350675 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc043400 con 0x7f20d40cf7d0 2024-01-16T14:58:17.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.928+0000 7f20d0c00640 1 -- 172.21.15.141:0/2262350675 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc0437b0 con 0x7f20d40cf7d0 2024-01-16T14:58:17.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.928+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d413f750 con 0x7f20d40cf7d0 2024-01-16T14:58:17.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.928+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d40ae2e0 con 0x7f20d40cf7d0 2024-01-16T14:58:17.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.929+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d40ccb60 con 0x7f20d40cf7d0 2024-01-16T14:58:17.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.930+0000 7f20d0c00640 1 -- 172.21.15.141:0/2262350675 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f20bc043910 con 0x7f20d40cf7d0 2024-01-16T14:58:17.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.931+0000 7f20d0c00640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20ac06b2f0 0x7f20ac06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:17.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.931+0000 7f20d0c00640 1 -- 172.21.15.141:0/2262350675 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20bc03e070 con 0x7f20d40cf7d0 2024-01-16T14:58:17.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.931+0000 7f20d3e00640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20ac06b2f0 0x7f20ac06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:17.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.933+0000 7f20d3e00640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20ac06b2f0 0x7f20ac06d7b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20c4007fd0 tx=0x7f20c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:17.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:17.935+0000 7f20d0c00640 1 -- 172.21.15.141:0/2262350675 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20d40ccb60 con 0x7f20d40cf7d0 2024-01-16T14:58:18.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.202+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f20d40ccb60 con 0x7f20d40cf7d0 2024-01-16T14:58:18.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.206+0000 7f20d0c00640 1 -- 172.21.15.141:0/2262350675 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f20d40ccb60 con 0x7f20d40cf7d0 2024-01-16T14:58:18.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20ac06b2f0 msgr2=0x7f20ac06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20ac06b2f0 0x7f20ac06d7b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20c4007fd0 tx=0x7f20c4009040 comp rx=0 tx=0).stop 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d40cf7d0 msgr2=0x7f20d413eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d40cf7d0 0x7f20d413eb90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f20bc02ba00 tx=0x7f20bc004730 comp rx=0 tx=0).stop 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 shutdown_connections 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20ac06b2f0 0x7f20ac06d7b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d415e3b0 0x7f20d40adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20d4143ae0 0x7f20d41f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 --2- 172.21.15.141:0/2262350675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20d40cf7d0 0x7f20d413eb90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 >> 172.21.15.141:0/2262350675 conn(0x7f20d406a800 msgr2=0x7f20d415c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 shutdown_connections 2024-01-16T14:58:18.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.209+0000 7f20db600640 1 -- 172.21.15.141:0/2262350675 wait complete. 2024-01-16T14:58:18.209 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-16T14:58:18.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.430+0000 7f8a30c00640 1 Processor -- start 2024-01-16T14:58:18.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.430+0000 7f8a30c00640 1 -- start start 2024-01-16T14:58:18.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a30c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0ae5a0 0x7f8a2c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a30c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8a2c0b3ae0 0x7f8a2c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a30c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8a2c06d120 0x7f8a2c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a30c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2c071450 con 0x7f8a2c06d120 2024-01-16T14:58:18.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a30c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2c070c40 con 0x7f8a2c0b3ae0 2024-01-16T14:58:18.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a30c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c06c470 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a2a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8a2c06d120 0x7f8a2c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a2a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8a2c06d120 0x7f8a2c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55780/0 (socket says 172.21.15.141:55780) 2024-01-16T14:58:18.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a2a800640 1 -- 172.21.15.141:0/2446530136 learned_addr learned my addr 172.21.15.141:0/2446530136 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a2a800640 1 -- 172.21.15.141:0/2446530136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0ae5a0 msgr2=0x7f8a2c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a29e00640 1 --2- 172.21.15.141:0/2446530136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0ae5a0 0x7f8a2c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.432+0000 7f8a23e00640 1 --2- 172.21.15.141:0/2446530136 >> v2:172.21.15.188:3300/0 conn(0x7f8a2c0b3ae0 0x7f8a2c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.433+0000 7f8a2a800640 1 --2- 172.21.15.141:0/2446530136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0ae5a0 0x7f8a2c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.433+0000 7f8a2a800640 1 -- 172.21.15.141:0/2446530136 >> v2:172.21.15.188:3300/0 conn(0x7f8a2c0b3ae0 msgr2=0x7f8a2c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.433+0000 7f8a2a800640 1 --2- 172.21.15.141:0/2446530136 >> v2:172.21.15.188:3300/0 conn(0x7f8a2c0b3ae0 0x7f8a2c06cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.433+0000 7f8a2a800640 1 -- 172.21.15.141:0/2446530136 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c06c750 con 0x7f8a2c06d120 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.433+0000 7f8a29e00640 1 --2- 172.21.15.141:0/2446530136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0ae5a0 0x7f8a2c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.433+0000 7f8a23e00640 1 --2- 172.21.15.141:0/2446530136 >> v2:172.21.15.188:3300/0 conn(0x7f8a2c0b3ae0 0x7f8a2c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:18.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.434+0000 7f8a2a800640 1 --2- 172.21.15.141:0/2446530136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c06d120 0x7f8a2c0cf290 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0cfed0 tx=0x7f8a2402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9e4d40514c1f3e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.434+0000 7f8a29400640 1 -- 172.21.15.141:0/2446530136 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2402ca70 con 0x7f8a2c06d120 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.434+0000 7f8a29400640 1 -- 172.21.15.141:0/2446530136 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2402cbd0 con 0x7f8a2c06d120 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.434+0000 7f8a29400640 1 -- 172.21.15.141:0/2446530136 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a240343f0 con 0x7f8a2c06d120 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 -- 172.21.15.141:0/2446530136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c06d120 msgr2=0x7f8a2c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2446530136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c06d120 0x7f8a2c0cf290 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0cfed0 tx=0x7f8a2402bed0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 -- 172.21.15.141:0/2446530136 shutdown_connections 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2446530136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c06d120 0x7f8a2c0cf290 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2446530136 >> v2:172.21.15.188:3300/0 conn(0x7f8a2c0b3ae0 0x7f8a2c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2446530136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0ae5a0 0x7f8a2c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 -- 172.21.15.141:0/2446530136 >> 172.21.15.141:0/2446530136 conn(0x7f8a2c06a800 msgr2=0x7f8a2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 -- 172.21.15.141:0/2446530136 shutdown_connections 2024-01-16T14:58:18.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.435+0000 7f8a30c00640 1 -- 172.21.15.141:0/2446530136 wait complete. 2024-01-16T14:58:18.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.436+0000 7f8a30c00640 1 Processor -- start 2024-01-16T14:58:18.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.437+0000 7f8a30c00640 1 -- start start 2024-01-16T14:58:18.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.437+0000 7f8a30c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a2c06d120 0x7f8a2c0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.437+0000 7f8a30c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c0ae5a0 0x7f8a2c1f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a30c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0b3ae0 0x7f8a2c0c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a30c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c1582d0 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a30c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0ccde0 con 0x7f8a2c06d120 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a30c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0cd0c0 con 0x7f8a2c0b3ae0 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c0ae5a0 0x7f8a2c1f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c0ae5a0 0x7f8a2c1f6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55788/0 (socket says 172.21.15.141:55788) 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 -- 172.21.15.141:0/2873459026 learned_addr learned my addr 172.21.15.141:0/2873459026 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 -- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0b3ae0 msgr2=0x7f8a2c0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0b3ae0 0x7f8a2c0c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 -- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a2c06d120 msgr2=0x7f8a2c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a2c06d120 0x7f8a2c0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 -- 172.21.15.141:0/2873459026 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24009b30 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.438+0000 7f8a23e00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c0ae5a0 0x7f8a2c1f6aa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400c7f0 tx=0x7f8a1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.439+0000 7f8a22000640 1 -- 172.21.15.141:0/2873459026 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14007820 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.439+0000 7f8a22000640 1 -- 172.21.15.141:0/2873459026 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14007980 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.439+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c158520 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.439+0000 7f8a22000640 1 -- 172.21.15.141:0/2873459026 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14007cf0 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.439+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c0c4390 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.440+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c0675d0 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.441+0000 7f8a22000640 1 -- 172.21.15.141:0/2873459026 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8a140113f0 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.442+0000 7f8a22000640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a0406b2c0 0x7f8a0406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:18.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.442+0000 7f8a22000640 1 -- 172.21.15.141:0/2873459026 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8a14089bb0 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.442+0000 7f8a29e00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a0406b2c0 0x7f8a0406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:18.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.443+0000 7f8a29e00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a0406b2c0 0x7f8a0406d780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00a8b0 tx=0x7f8a1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:18.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.446+0000 7f8a22000640 1 -- 172.21.15.141:0/2873459026 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a14054d50 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.706+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f8a2c067c90 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.710+0000 7f8a22000640 1 -- 172.21.15.141:0/2873459026 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8a1401f0a0 con 0x7f8a2c0ae5a0 2024-01-16T14:58:18.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a0406b2c0 msgr2=0x7f8a0406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a0406b2c0 0x7f8a0406d780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00a8b0 tx=0x7f8a1c008040 comp rx=0 tx=0).stop 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c0ae5a0 msgr2=0x7f8a2c1f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c0ae5a0 0x7f8a2c1f6aa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400c7f0 tx=0x7f8a1400ccb0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 shutdown_connections 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a0406b2c0 0x7f8a0406d780 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a2c0b3ae0 0x7f8a2c0c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a2c0ae5a0 0x7f8a2c1f6aa0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 --2- 172.21.15.141:0/2873459026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a2c06d120 0x7f8a2c0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:18.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.713+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 >> 172.21.15.141:0/2873459026 conn(0x7f8a2c06a800 msgr2=0x7f8a2c0cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:18.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.714+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 shutdown_connections 2024-01-16T14:58:18.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:18.714+0000 7f8a30c00640 1 -- 172.21.15.141:0/2873459026 wait complete. 2024-01-16T14:58:18.818 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-16T14:58:19.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.057+0000 7f1d2aa00640 1 Processor -- start 2024-01-16T14:58:19.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.057+0000 7f1d2aa00640 1 -- start start 2024-01-16T14:58:19.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.058+0000 7f1d2aa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d241447f0 0x7f1d2415c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.058+0000 7f1d2aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1d240cccf0 0x7f1d2415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.058+0000 7f1d2aa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2415c5c0 con 0x7f1d2406c470 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.058+0000 7f1d2aa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d2406d690 con 0x7f1d240cccf0 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.058+0000 7f1d2aa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d240cebb0 con 0x7f1d241447f0 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d2a000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1d240cccf0 0x7f1d2415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d28c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1d2406c470 0x7f1d240cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58430/0 (socket says 172.21.15.141:58430) 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d28c00640 1 -- 172.21.15.141:0/2097688797 learned_addr learned my addr 172.21.15.141:0/2097688797 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d29600640 1 --2- 172.21.15.141:0/2097688797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d241447f0 0x7f1d2415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d2a000640 1 -- 172.21.15.141:0/2097688797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d241447f0 msgr2=0x7f1d2415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d2a000640 1 --2- 172.21.15.141:0/2097688797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d241447f0 0x7f1d2415c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d2a000640 1 -- 172.21.15.141:0/2097688797 >> v1:172.21.15.141:6789/0 conn(0x7f1d2406c470 legacy=0x7f1d240cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d2a000640 1 -- 172.21.15.141:0/2097688797 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2406ca10 con 0x7f1d240cccf0 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d23e00640 1 -- 172.21.15.141:0/2097688797 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724948544 0 0) 0x7f1d2415c5c0 con 0x7f1d2406c470 2024-01-16T14:58:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.059+0000 7f1d29600640 1 --2- 172.21.15.141:0/2097688797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d241447f0 0x7f1d2415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:19.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.061+0000 7f1d2a000640 1 --2- 172.21.15.141:0/2097688797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d240cccf0 0x7f1d2415c7c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1d240cecf0 tx=0x7f1d1c02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802334370de78be1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.061+0000 7f1d23e00640 1 -- 172.21.15.141:0/2097688797 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c02cbb0 con 0x7f1d240cccf0 2024-01-16T14:58:19.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.061+0000 7f1d23e00640 1 -- 172.21.15.141:0/2097688797 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1c02cd10 con 0x7f1d240cccf0 2024-01-16T14:58:19.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.061+0000 7f1d23e00640 1 -- 172.21.15.141:0/2097688797 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c034660 con 0x7f1d240cccf0 2024-01-16T14:58:19.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 -- 172.21.15.141:0/2097688797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d240cccf0 msgr2=0x7f1d2415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/2097688797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d240cccf0 0x7f1d2415c7c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1d240cecf0 tx=0x7f1d1c02be00 comp rx=0 tx=0).stop 2024-01-16T14:58:19.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 -- 172.21.15.141:0/2097688797 shutdown_connections 2024-01-16T14:58:19.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/2097688797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d240cccf0 0x7f1d2415c7c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/2097688797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d241447f0 0x7f1d2415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 -- 172.21.15.141:0/2097688797 >> 172.21.15.141:0/2097688797 conn(0x7f1d2406a3e0 msgr2=0x7f1d2406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 -- 172.21.15.141:0/2097688797 shutdown_connections 2024-01-16T14:58:19.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.062+0000 7f1d2aa00640 1 -- 172.21.15.141:0/2097688797 wait complete. 2024-01-16T14:58:19.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.064+0000 7f1d2aa00640 1 Processor -- start 2024-01-16T14:58:19.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.064+0000 7f1d2aa00640 1 -- start start 2024-01-16T14:58:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.065+0000 7f1d2aa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2406c470 0x7f1d241f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.065+0000 7f1d2aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d240cccf0 0x7f1d2413c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.065+0000 7f1d2aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d241447f0 0x7f1d240ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.065+0000 7f1d2aa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2413a350 con 0x7f1d241447f0 2024-01-16T14:58:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.065+0000 7f1d2aa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2413a040 con 0x7f1d240cccf0 2024-01-16T14:58:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.065+0000 7f1d2aa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2413cd70 con 0x7f1d2406c470 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d241447f0 0x7f1d240ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d241447f0 0x7f1d240ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55806/0 (socket says 172.21.15.141:55806) 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 -- 172.21.15.141:0/1004393222 learned_addr learned my addr 172.21.15.141:0/1004393222 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d29600640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2406c470 0x7f1d241f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 -- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2406c470 msgr2=0x7f1d241f6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2406c470 0x7f1d241f6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 -- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d240cccf0 msgr2=0x7f1d2413c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d28c00640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d240cccf0 0x7f1d2413c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d240cccf0 0x7f1d2413c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.066+0000 7f1d2a000640 1 -- 172.21.15.141:0/1004393222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c009a70 con 0x7f1d241447f0 2024-01-16T14:58:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.067+0000 7f1d2a000640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d241447f0 0x7f1d240ac8d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009e80 tx=0x7f1d1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.067+0000 7f1d21600640 1 -- 172.21.15.141:0/1004393222 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c040650 con 0x7f1d241447f0 2024-01-16T14:58:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.067+0000 7f1d21600640 1 -- 172.21.15.141:0/1004393222 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1c0407d0 con 0x7f1d241447f0 2024-01-16T14:58:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.067+0000 7f1d21600640 1 -- 172.21.15.141:0/1004393222 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c040b80 con 0x7f1d241447f0 2024-01-16T14:58:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.067+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d240ad120 con 0x7f1d241447f0 2024-01-16T14:58:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.067+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d240ad6e0 con 0x7f1d241447f0 2024-01-16T14:58:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.069+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d240ad8d0 con 0x7f1d241447f0 2024-01-16T14:58:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.070+0000 7f1d21600640 1 -- 172.21.15.141:0/1004393222 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1d1c040ce0 con 0x7f1d241447f0 2024-01-16T14:58:19.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.071+0000 7f1d21600640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cfc06b2f0 0x7f1cfc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.071+0000 7f1d21600640 1 -- 172.21.15.141:0/1004393222 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1d1c0ac730 con 0x7f1d241447f0 2024-01-16T14:58:19.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.071+0000 7f1d29600640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cfc06b2f0 0x7f1cfc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.072+0000 7f1d29600640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cfc06b2f0 0x7f1cfc06d7b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1d24142500 tx=0x7f1d14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.076+0000 7f1d21600640 1 -- 172.21.15.141:0/1004393222 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d240ad8d0 con 0x7f1d241447f0 2024-01-16T14:58:19.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.337+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f1d2413a490 con 0x7f1d241447f0 2024-01-16T14:58:19.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.342+0000 7f1d21600640 1 -- 172.21.15.141:0/1004393222 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1d2413a490 con 0x7f1d241447f0 2024-01-16T14:58:19.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cfc06b2f0 msgr2=0x7f1cfc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cfc06b2f0 0x7f1cfc06d7b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1d24142500 tx=0x7f1d14008040 comp rx=0 tx=0).stop 2024-01-16T14:58:19.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d241447f0 msgr2=0x7f1d240ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d241447f0 0x7f1d240ac8d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009e80 tx=0x7f1d1c004a40 comp rx=0 tx=0).stop 2024-01-16T14:58:19.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 shutdown_connections 2024-01-16T14:58:19.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cfc06b2f0 0x7f1cfc06d7b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d241447f0 0x7f1d240ac8d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d240cccf0 0x7f1d2413c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 --2- 172.21.15.141:0/1004393222 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2406c470 0x7f1d241f6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 >> 172.21.15.141:0/1004393222 conn(0x7f1d2406a3e0 msgr2=0x7f1d241599f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 shutdown_connections 2024-01-16T14:58:19.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.346+0000 7f1d2aa00640 1 -- 172.21.15.141:0/1004393222 wait complete. 2024-01-16T14:58:19.345 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-16T14:58:19.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.565+0000 7f09fe800640 1 Processor -- start 2024-01-16T14:58:19.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.565+0000 7f09fe800640 1 -- start start 2024-01-16T14:58:19.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.566+0000 7f09fe800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09f8143ae0 0x7f09f80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.566+0000 7f09fe800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09fe800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f80cfc30 con 0x7f09f815e1e0 2024-01-16T14:58:19.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09fe800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f80b1ee0 con 0x7f09f8143ae0 2024-01-16T14:58:19.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09fe800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80b2780 con 0x7f09f80cf7d0 2024-01-16T14:58:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09f8143ae0 0x7f09f80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09f8143ae0 0x7f09f80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55822/0 (socket says 172.21.15.141:55822) 2024-01-16T14:58:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7400640 1 -- 172.21.15.141:0/3807057847 learned_addr learned my addr 172.21.15.141:0/3807057847 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f09f815e1e0 0x7f09f815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58438/0 (socket says 172.21.15.141:58438) 2024-01-16T14:58:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f6a00640 1 --2- 172.21.15.141:0/3807057847 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7400640 1 -- 172.21.15.141:0/3807057847 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 msgr2=0x7f09f815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7400640 1 --2- 172.21.15.141:0/3807057847 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7400640 1 -- 172.21.15.141:0/3807057847 >> v1:172.21.15.141:6789/0 conn(0x7f09f815e1e0 legacy=0x7f09f815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.567+0000 7f09f7400640 1 -- 172.21.15.141:0/3807057847 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f815e990 con 0x7f09f8143ae0 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.569+0000 7f09f7400640 1 --2- 172.21.15.141:0/3807057847 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f8143ae0 0x7f09f80cf290 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009870 tx=0x7f09e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f5c6b4d3bc73a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.569+0000 7f09f6000640 1 -- 172.21.15.141:0/3807057847 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8030840 con 0x7f09f8143ae0 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.569+0000 7f09f6000640 1 -- 172.21.15.141:0/3807057847 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e80309a0 con 0x7f09f8143ae0 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.569+0000 7f09f6000640 1 -- 172.21.15.141:0/3807057847 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8030d30 con 0x7f09f8143ae0 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 -- 172.21.15.141:0/3807057847 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f8143ae0 msgr2=0x7f09f80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 --2- 172.21.15.141:0/3807057847 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f8143ae0 0x7f09f80cf290 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009870 tx=0x7f09e802b980 comp rx=0 tx=0).stop 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 -- 172.21.15.141:0/3807057847 shutdown_connections 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 --2- 172.21.15.141:0/3807057847 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 --2- 172.21.15.141:0/3807057847 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f8143ae0 0x7f09f80cf290 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 -- 172.21.15.141:0/3807057847 >> 172.21.15.141:0/3807057847 conn(0x7f09f806a800 msgr2=0x7f09f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 -- 172.21.15.141:0/3807057847 shutdown_connections 2024-01-16T14:58:19.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.570+0000 7f09fe800640 1 -- 172.21.15.141:0/3807057847 wait complete. 2024-01-16T14:58:19.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.571+0000 7f09fe800640 1 Processor -- start 2024-01-16T14:58:19.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.572+0000 7f09fe800640 1 -- start start 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.572+0000 7f09fe800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f8148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.572+0000 7f09fe800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8143ae0 0x7f09f814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.572+0000 7f09fe800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f815e1e0 0x7f09f814aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.572+0000 7f09fe800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f813c770 con 0x7f09f815e1e0 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.572+0000 7f09fe800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f815a5f0 con 0x7f09f8143ae0 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.572+0000 7f09fe800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f815a8d0 con 0x7f09f80cf7d0 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f815e1e0 0x7f09f814aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f815e1e0 0x7f09f814aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55826/0 (socket says 172.21.15.141:55826) 2024-01-16T14:58:19.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 -- 172.21.15.141:0/65070147 learned_addr learned my addr 172.21.15.141:0/65070147 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f6a00640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8143ae0 0x7f09f814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7400640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f8148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 -- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 msgr2=0x7f09f8148470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f8148470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 -- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8143ae0 msgr2=0x7f09f814a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8143ae0 0x7f09f814a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 -- 172.21.15.141:0/65070147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e80094d0 con 0x7f09f815e1e0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.573+0000 7f09f7e00640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f815e1e0 0x7f09f814aa40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c730 tx=0x7f09f000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.574+0000 7f09ef400640 1 -- 172.21.15.141:0/65070147 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f000ce40 con 0x7f09f815e1e0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.574+0000 7f09ef400640 1 -- 172.21.15.141:0/65070147 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f000a640 con 0x7f09f815e1e0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.574+0000 7f09ef400640 1 -- 172.21.15.141:0/65070147 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f000a9f0 con 0x7f09f815e1e0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.574+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f815abb0 con 0x7f09f815e1e0 2024-01-16T14:58:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.574+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f8159c60 con 0x7f09f815e1e0 2024-01-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.575+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f80b2bf0 con 0x7f09f815e1e0 2024-01-16T14:58:19.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.576+0000 7f09ef400640 1 -- 172.21.15.141:0/65070147 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f09f000ab50 con 0x7f09f815e1e0 2024-01-16T14:58:19.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.577+0000 7f09ef400640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09d006b2c0 0x7f09d006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:19.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.577+0000 7f09ef400640 1 -- 172.21.15.141:0/65070147 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09f0088ee0 con 0x7f09f815e1e0 2024-01-16T14:58:19.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.577+0000 7f09f7400640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09d006b2c0 0x7f09d006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:19.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.578+0000 7f09f7400640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09d006b2c0 0x7f09d006d780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f09e8036000 tx=0x7f09e803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:19.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.581+0000 7f09ef400640 1 -- 172.21.15.141:0/65070147 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09f80b2bf0 con 0x7f09f815e1e0 2024-01-16T14:58:19.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.841+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f09f80b2bf0 con 0x7f09f815e1e0 2024-01-16T14:58:19.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.847+0000 7f09ef400640 1 -- 172.21.15.141:0/65070147 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f09f80b2bf0 con 0x7f09f815e1e0 2024-01-16T14:58:19.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.849+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09d006b2c0 msgr2=0x7f09d006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.849+0000 7f09fe800640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09d006b2c0 0x7f09d006d780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f09e8036000 tx=0x7f09e803b040 comp rx=0 tx=0).stop 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.849+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f815e1e0 msgr2=0x7f09f814aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.849+0000 7f09fe800640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f815e1e0 0x7f09f814aa40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c730 tx=0x7f09f000cbf0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 shutdown_connections 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f09d006b2c0 0x7f09d006d780 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09f815e1e0 0x7f09f814aa40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8143ae0 0x7f09f814a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 --2- 172.21.15.141:0/65070147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09f80cf7d0 0x7f09f8148470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:19.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 >> 172.21.15.141:0/65070147 conn(0x7f09f806a800 msgr2=0x7f09f80aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:19.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 shutdown_connections 2024-01-16T14:58:19.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:19.850+0000 7f09fe800640 1 -- 172.21.15.141:0/65070147 wait complete. 2024-01-16T14:58:19.849 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-16T14:58:20.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.074+0000 7f7bfb200640 1 Processor -- start 2024-01-16T14:58:20.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.074+0000 7f7bfb200640 1 -- start start 2024-01-16T14:58:20.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.076+0000 7f7bfb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf40cc990 0x7f7bf406d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.076+0000 7f7bfb200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf4070e50 con 0x7f7bf40cc500 2024-01-16T14:58:20.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.076+0000 7f7bfb200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf4144020 con 0x7f7bf4143c40 2024-01-16T14:58:20.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.076+0000 7f7bfb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40723c0 con 0x7f7bf40cc990 2024-01-16T14:58:20.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.076+0000 7f7bf9400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7bf40cc500 0x7f7bf406d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58446/0 (socket says 172.21.15.141:58446) 2024-01-16T14:58:20.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.076+0000 7f7bf9400640 1 -- 172.21.15.141:0/2296479396 learned_addr learned my addr 172.21.15.141:0/2296479396 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bfa800640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf40cc990 0x7f7bf406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772482187 0 0) 0x7f7bf4070e50 con 0x7f7bf40cc500 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd4003520 con 0x7f7bf40cc500 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2903216167 0 0) 0x7f7bd4003520 con 0x7f7bf40cc500 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf40cc990 msgr2=0x7f7bf406d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf40cc990 0x7f7bf406d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 >> v1:172.21.15.188:6789/0 conn(0x7f7bf4143c40 legacy=0x7f7bf4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4071e00 con 0x7f7bf40cc500 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7bdc001690 con 0x7f7bf40cc500 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 >> v1:172.21.15.141:6789/0 conn(0x7f7bf40cc500 legacy=0x7f7bf406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd40039a0 0x7f7bd4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.077+0000 7f7bf8a00640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bd40047c0 0x7f7bd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf8a00640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf40cc990 0x7f7bd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40723c0 con 0x7f7bd40047c0 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd400fef0 con 0x7f7bd40039a0 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd40100b0 con 0x7f7bf40cc990 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9400640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bd40047c0 0x7f7bd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9400640 1 -- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf40cc990 msgr2=0x7f7bd40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9400640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf40cc990 0x7f7bd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9400640 1 -- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd40039a0 msgr2=0x7f7bd4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9e00640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd40039a0 0x7f7bd4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9400640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd40039a0 0x7f7bd4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9400640 1 -- 172.21.15.141:0/2296479396 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd4003520 con 0x7f7bd40047c0 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.078+0000 7f7bf9e00640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd40039a0 0x7f7bd4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.079+0000 7f7bf9400640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bd40047c0 0x7f7bd4004c40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40cbd00 tx=0x7f7bd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0e0b82f7f6ac4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:20.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc005ad0 con 0x7f7bd40047c0 2024-01-16T14:58:20.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bdc005c30 con 0x7f7bd40047c0 2024-01-16T14:58:20.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bf8a00640 1 -- 172.21.15.141:0/2296479396 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc02c890 con 0x7f7bd40047c0 2024-01-16T14:58:20.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bfb200640 1 -- 172.21.15.141:0/2296479396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bd40047c0 msgr2=0x7f7bd4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bfb200640 1 --2- 172.21.15.141:0/2296479396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bd40047c0 0x7f7bd4004c40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40cbd00 tx=0x7f7bd40057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bfa800640 1 -- 172.21.15.141:0/2296479396 reap_dead start 2024-01-16T14:58:20.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bfb200640 1 -- 172.21.15.141:0/2296479396 shutdown_connections 2024-01-16T14:58:20.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bfb200640 1 -- 172.21.15.141:0/2296479396 >> 172.21.15.141:0/2296479396 conn(0x7f7bf406a800 msgr2=0x7f7bf406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:20.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bfb200640 1 -- 172.21.15.141:0/2296479396 shutdown_connections 2024-01-16T14:58:20.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.080+0000 7f7bfb200640 1 -- 172.21.15.141:0/2296479396 wait complete. 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.082+0000 7f7bfb200640 1 Processor -- start 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.082+0000 7f7bfb200640 1 -- start start 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.083+0000 7f7bfb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bd40047c0 0x7f7bf415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.083+0000 7f7bfb200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4141d90 0x7f7bf4142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.083+0000 7f7bfb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40b28e0 0x7f7bf40b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.083+0000 7f7bfb200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40b32d0 con 0x7f7bf40b28e0 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.083+0000 7f7bfb200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4142830 con 0x7f7bf4141d90 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.083+0000 7f7bfb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4142b10 con 0x7f7bd40047c0 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40b28e0 0x7f7bf40b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40b28e0 0x7f7bf40b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55840/0 (socket says 172.21.15.141:55840) 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 -- 172.21.15.141:0/3569159342 learned_addr learned my addr 172.21.15.141:0/3569159342 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bf9400640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4141d90 0x7f7bf4142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 -- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bd40047c0 msgr2=0x7f7bf415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bf9e00640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bd40047c0 0x7f7bf415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bd40047c0 0x7f7bf415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 -- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4141d90 msgr2=0x7f7bf4142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4141d90 0x7f7bf4142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 -- 172.21.15.141:0/3569159342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4142df0 con 0x7f7bf40b28e0 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bf9400640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4141d90 0x7f7bf4142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.084+0000 7f7bfa800640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40b28e0 0x7f7bf40b2d90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40cc3f0 tx=0x7f7bec00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.085+0000 7f7bf2000640 1 -- 172.21.15.141:0/3569159342 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec010040 con 0x7f7bf40b28e0 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.085+0000 7f7bf2000640 1 -- 172.21.15.141:0/3569159342 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec00bae0 con 0x7f7bf40b28e0 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.085+0000 7f7bf2000640 1 -- 172.21.15.141:0/3569159342 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00be70 con 0x7f7bf40b28e0 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.085+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf41430d0 con 0x7f7bf40b28e0 2024-01-16T14:58:20.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.085+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf415ad30 con 0x7f7bf40b28e0 2024-01-16T14:58:20.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.086+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf4067450 con 0x7f7bf40b28e0 2024-01-16T14:58:20.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.087+0000 7f7bf2000640 1 -- 172.21.15.141:0/3569159342 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7bec01d400 con 0x7f7bf40b28e0 2024-01-16T14:58:20.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.088+0000 7f7bf2000640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06b2f0 0x7f7bcc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.088+0000 7f7bf2000640 1 -- 172.21.15.141:0/3569159342 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7bec089f60 con 0x7f7bf40b28e0 2024-01-16T14:58:20.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.089+0000 7f7bf9e00640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06b2f0 0x7f7bcc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.090+0000 7f7bf9e00640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06b2f0 0x7f7bcc06d7b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7be40015b0 tx=0x7f7be4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:20.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.092+0000 7f7bf2000640 1 -- 172.21.15.141:0/3569159342 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bec055110 con 0x7f7bf40b28e0 2024-01-16T14:58:20.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.341+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f7bf41eead0 con 0x7f7bf40b28e0 2024-01-16T14:58:20.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.345+0000 7f7bf2000640 1 -- 172.21.15.141:0/3569159342 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7bec058730 con 0x7f7bf40b28e0 2024-01-16T14:58:20.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06b2f0 msgr2=0x7f7bcc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06b2f0 0x7f7bcc06d7b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7be40015b0 tx=0x7f7be4009040 comp rx=0 tx=0).stop 2024-01-16T14:58:20.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40b28e0 msgr2=0x7f7bf40b2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40b28e0 0x7f7bf40b2d90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40cc3f0 tx=0x7f7bec00b480 comp rx=0 tx=0).stop 2024-01-16T14:58:20.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 shutdown_connections 2024-01-16T14:58:20.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06b2f0 0x7f7bcc06d7b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40b28e0 0x7f7bf40b2d90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4141d90 0x7f7bf4142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 --2- 172.21.15.141:0/3569159342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bd40047c0 0x7f7bf415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 >> 172.21.15.141:0/3569159342 conn(0x7f7bf406a800 msgr2=0x7f7bf40affa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:20.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.348+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 shutdown_connections 2024-01-16T14:58:20.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.349+0000 7f7bfb200640 1 -- 172.21.15.141:0/3569159342 wait complete. 2024-01-16T14:58:20.347 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-16T14:58:20.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.571+0000 7f2596c00640 1 Processor -- start 2024-01-16T14:58:20.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.571+0000 7f2596c00640 1 -- start start 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.573+0000 7f2596c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f259015f380 0x7f25900b3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.573+0000 7f2596c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900b2db0 con 0x7f259015efa0 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.573+0000 7f2596c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900b1ff0 con 0x7f25901628b0 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.573+0000 7f2596c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2590071890 con 0x7f259015f380 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.573+0000 7f2595800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f259015efa0 0x7f2590071180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58460/0 (socket says 172.21.15.141:58460) 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.573+0000 7f2595800640 1 -- 172.21.15.141:0/2511594361 learned_addr learned my addr 172.21.15.141:0/2511594361 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.574+0000 7f2594e00640 1 --2- 172.21.15.141:0/2511594361 >> v2:172.21.15.188:3300/0 conn(0x7f259015f380 0x7f25900b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.574+0000 7f258fe00640 1 -- 172.21.15.141:0/2511594361 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583828752 0 0) 0x7f25900b2db0 con 0x7f259015efa0 2024-01-16T14:58:20.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.574+0000 7f258fe00640 1 -- 172.21.15.141:0/2511594361 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2570003520 con 0x7f259015efa0 2024-01-16T14:58:20.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.574+0000 7f2594e00640 1 -- 172.21.15.141:0/2511594361 >> v1:172.21.15.188:6789/0 conn(0x7f25901628b0 legacy=0x7f259006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.574+0000 7f2594e00640 1 -- 172.21.15.141:0/2511594361 >> v1:172.21.15.141:6789/0 conn(0x7f259015efa0 legacy=0x7f2590071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.574+0000 7f2594e00640 1 -- 172.21.15.141:0/2511594361 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25900b22d0 con 0x7f259015f380 2024-01-16T14:58:20.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.576+0000 7f2594e00640 1 --2- 172.21.15.141:0/2511594361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f25900b3030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25780097d0 tx=0x7f257802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63223cf2ad176387 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:20.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.576+0000 7f258fe00640 1 -- 172.21.15.141:0/2511594361 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578030800 con 0x7f259015f380 2024-01-16T14:58:20.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.576+0000 7f258fe00640 1 -- 172.21.15.141:0/2511594361 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578030960 con 0x7f259015f380 2024-01-16T14:58:20.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.576+0000 7f258fe00640 1 -- 172.21.15.141:0/2511594361 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578030cd0 con 0x7f259015f380 2024-01-16T14:58:20.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.577+0000 7f2596c00640 1 -- 172.21.15.141:0/2511594361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 msgr2=0x7f25900b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.577+0000 7f2596c00640 1 --2- 172.21.15.141:0/2511594361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f25900b3030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25780097d0 tx=0x7f257802b940 comp rx=0 tx=0).stop 2024-01-16T14:58:20.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.577+0000 7f2596c00640 1 -- 172.21.15.141:0/2511594361 shutdown_connections 2024-01-16T14:58:20.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.577+0000 7f2596c00640 1 --2- 172.21.15.141:0/2511594361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f25900b3030 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.577+0000 7f2596c00640 1 -- 172.21.15.141:0/2511594361 >> 172.21.15.141:0/2511594361 conn(0x7f259006a3e0 msgr2=0x7f259006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:20.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.577+0000 7f2596c00640 1 -- 172.21.15.141:0/2511594361 shutdown_connections 2024-01-16T14:58:20.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.577+0000 7f2596c00640 1 -- 172.21.15.141:0/2511594361 wait complete. 2024-01-16T14:58:20.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.578+0000 7f2596c00640 1 Processor -- start 2024-01-16T14:58:20.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.579+0000 7f2596c00640 1 -- start start 2024-01-16T14:58:20.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.579+0000 7f2596c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259015efa0 0x7f25900cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.579+0000 7f2596c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f259015a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.579+0000 7f2596c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25901628b0 0x7f259006c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.579+0000 7f2596c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25900a95c0 con 0x7f259015efa0 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.579+0000 7f2596c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25900ae040 con 0x7f259015f380 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.579+0000 7f2596c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25900ae320 con 0x7f25901628b0 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259015efa0 0x7f25900cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259015efa0 0x7f25900cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55856/0 (socket says 172.21.15.141:55856) 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 -- 172.21.15.141:0/469104682 learned_addr learned my addr 172.21.15.141:0/469104682 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2594e00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f259015a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 -- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25901628b0 msgr2=0x7f259006c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2596200640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25901628b0 0x7f259006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25901628b0 0x7f259006c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 -- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 msgr2=0x7f259015a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f259015a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2595800640 1 -- 172.21.15.141:0/469104682 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578009400 con 0x7f259015efa0 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.580+0000 7f2594e00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f259015a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:20.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.581+0000 7f2595800640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259015efa0 0x7f25900cd500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f258000c500 tx=0x7f258000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:20.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.581+0000 7f258d600640 1 -- 172.21.15.141:0/469104682 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580005ce0 con 0x7f259015efa0 2024-01-16T14:58:20.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.581+0000 7f258d600640 1 -- 172.21.15.141:0/469104682 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580005e40 con 0x7f259015efa0 2024-01-16T14:58:20.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.581+0000 7f258d600640 1 -- 172.21.15.141:0/469104682 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580007330 con 0x7f259015efa0 2024-01-16T14:58:20.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.581+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25900a8f70 con 0x7f259015efa0 2024-01-16T14:58:20.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.581+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259015ab00 con 0x7f259015efa0 2024-01-16T14:58:20.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.582+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2590067450 con 0x7f259015efa0 2024-01-16T14:58:20.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.583+0000 7f258d600640 1 -- 172.21.15.141:0/469104682 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2580007550 con 0x7f259015efa0 2024-01-16T14:58:20.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.584+0000 7f258d600640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806b2c0 0x7f256806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:20.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.584+0000 7f258d600640 1 -- 172.21.15.141:0/469104682 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2580088f20 con 0x7f259015efa0 2024-01-16T14:58:20.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.584+0000 7f2594e00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806b2c0 0x7f256806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:20.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.586+0000 7f2594e00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806b2c0 0x7f256806d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2578036000 tx=0x7f257803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:20.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.589+0000 7f258d600640 1 -- 172.21.15.141:0/469104682 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25800541a0 con 0x7f259015efa0 2024-01-16T14:58:20.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.848+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f2590070f60 con 0x7f259015efa0 2024-01-16T14:58:20.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.853+0000 7f258d600640 1 -- 172.21.15.141:0/469104682 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f25800170a0 con 0x7f259015efa0 2024-01-16T14:58:20.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806b2c0 msgr2=0x7f256806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806b2c0 0x7f256806d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2578036000 tx=0x7f257803b040 comp rx=0 tx=0).stop 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259015efa0 msgr2=0x7f25900cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259015efa0 0x7f25900cd500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f258000c500 tx=0x7f258000c9c0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 shutdown_connections 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806b2c0 0x7f256806d780 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25901628b0 0x7f259006c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f259015f380 0x7f259015a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 --2- 172.21.15.141:0/469104682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259015efa0 0x7f25900cd500 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 >> 172.21.15.141:0/469104682 conn(0x7f259006a3e0 msgr2=0x7f259015f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 shutdown_connections 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:20.856+0000 7f2596c00640 1 -- 172.21.15.141:0/469104682 wait complete. 2024-01-16T14:58:20.856 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:21.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.086+0000 7f0044200640 1 Processor -- start 2024-01-16T14:58:21.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.086+0000 7f0044200640 1 -- start start 2024-01-16T14:58:21.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0044200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f003c0b3ae0 0x7f003c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0044200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f003c0cf290 0x7f003c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0044200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c070c40 con 0x7f003c0ae5a0 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0044200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c071450 con 0x7f003c0cf290 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0044200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c0cfc60 con 0x7f003c0b3ae0 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0041800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f003c0ae5a0 0x7f003c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58466/0 (socket says 172.21.15.141:58466) 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0041800640 1 -- 172.21.15.141:0/3912315101 learned_addr learned my addr 172.21.15.141:0/3912315101 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.088+0000 7f0042200640 1 --2- 172.21.15.141:0/3912315101 >> v2:172.21.15.141:3300/0 conn(0x7f003c0cf290 0x7f003c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.089+0000 7f0040e00640 1 --2- 172.21.15.141:0/3912315101 >> v2:172.21.15.188:3300/0 conn(0x7f003c0b3ae0 0x7f003c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.089+0000 7f003be00640 1 -- 172.21.15.141:0/3912315101 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579772734 0 0) 0x7f003c070c40 con 0x7f003c0ae5a0 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.089+0000 7f003be00640 1 -- 172.21.15.141:0/3912315101 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c003520 con 0x7f003c0ae5a0 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.089+0000 7f0042200640 1 -- 172.21.15.141:0/3912315101 >> v2:172.21.15.188:3300/0 conn(0x7f003c0b3ae0 msgr2=0x7f003c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.089+0000 7f0042200640 1 --2- 172.21.15.141:0/3912315101 >> v2:172.21.15.188:3300/0 conn(0x7f003c0b3ae0 0x7f003c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.089+0000 7f0042200640 1 -- 172.21.15.141:0/3912315101 >> v1:172.21.15.141:6789/0 conn(0x7f003c0ae5a0 legacy=0x7f003c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.089+0000 7f0042200640 1 -- 172.21.15.141:0/3912315101 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c06c470 con 0x7f003c0cf290 2024-01-16T14:58:21.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.090+0000 7f0042200640 1 --2- 172.21.15.141:0/3912315101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0cf290 0x7f003c0cf720 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f003c0cfea0 tx=0x7f003402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368d49b4918f66ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.090+0000 7f003be00640 1 -- 172.21.15.141:0/3912315101 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003402cd00 con 0x7f003c0cf290 2024-01-16T14:58:21.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.090+0000 7f003be00640 1 -- 172.21.15.141:0/3912315101 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003402ce60 con 0x7f003c0cf290 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.090+0000 7f003be00640 1 -- 172.21.15.141:0/3912315101 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0034034660 con 0x7f003c0cf290 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 -- 172.21.15.141:0/3912315101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0cf290 msgr2=0x7f003c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 --2- 172.21.15.141:0/3912315101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0cf290 0x7f003c0cf720 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f003c0cfea0 tx=0x7f003402bf50 comp rx=0 tx=0).stop 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 -- 172.21.15.141:0/3912315101 shutdown_connections 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 --2- 172.21.15.141:0/3912315101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0cf290 0x7f003c0cf720 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 --2- 172.21.15.141:0/3912315101 >> v2:172.21.15.188:3300/0 conn(0x7f003c0b3ae0 0x7f003c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 -- 172.21.15.141:0/3912315101 >> 172.21.15.141:0/3912315101 conn(0x7f003c06a800 msgr2=0x7f003c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 -- 172.21.15.141:0/3912315101 shutdown_connections 2024-01-16T14:58:21.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.091+0000 7f0044200640 1 -- 172.21.15.141:0/3912315101 wait complete. 2024-01-16T14:58:21.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.093+0000 7f0044200640 1 Processor -- start 2024-01-16T14:58:21.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.093+0000 7f0044200640 1 -- start start 2024-01-16T14:58:21.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0044200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f003c0ae5a0 0x7f003c14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0044200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0b3ae0 0x7f003c15fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0044200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f003c0cf290 0x7f003c1600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0044200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c0af160 con 0x7f003c0b3ae0 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0044200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c0aedc0 con 0x7f003c0ae5a0 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0044200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c1500e0 con 0x7f003c0cf290 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0040e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0b3ae0 0x7f003c15fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0040e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0b3ae0 0x7f003c15fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55886/0 (socket says 172.21.15.141:55886) 2024-01-16T14:58:21.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.094+0000 7f0040e00640 1 -- 172.21.15.141:0/2904117514 learned_addr learned my addr 172.21.15.141:0/2904117514 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:21.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0041800640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f003c0ae5a0 0x7f003c14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0042200640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f003c0cf290 0x7f003c1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0040e00640 1 -- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f003c0cf290 msgr2=0x7f003c1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0040e00640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f003c0cf290 0x7f003c1600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0040e00640 1 -- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f003c0ae5a0 msgr2=0x7f003c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0040e00640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f003c0ae5a0 0x7f003c14fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0040e00640 1 -- 172.21.15.141:0/2904117514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034009a70 con 0x7f003c0b3ae0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0040e00640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0b3ae0 0x7f003c15fb80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f002400d800 tx=0x7f002400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.095+0000 7f0039600640 1 -- 172.21.15.141:0/2904117514 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002400b6c0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.096+0000 7f0039600640 1 -- 172.21.15.141:0/2904117514 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002400b820 con 0x7f003c0b3ae0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.096+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c1586e0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.096+0000 7f0039600640 1 -- 172.21.15.141:0/2904117514 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002400bbb0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.096+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c158ca0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.097+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c0af3d0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.098+0000 7f0039600640 1 -- 172.21.15.141:0/2904117514 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f002400bd10 con 0x7f003c0b3ae0 2024-01-16T14:58:21.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.099+0000 7f0039600640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f001406b2c0 0x7f001406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.099+0000 7f0039600640 1 -- 172.21.15.141:0/2904117514 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0024089df0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.099+0000 7f0041800640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f001406b2c0 0x7f001406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.100+0000 7f0041800640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f001406b2c0 0x7f001406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f002c0011e0 tx=0x7f002c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.103+0000 7f0039600640 1 -- 172.21.15.141:0/2904117514 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0024054fa0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.293+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f003c0675d0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.294+0000 7f0039600640 1 -- 172.21.15.141:0/2904117514 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12905 (secure 0 0 0) 0x7f00240585c0 con 0x7f003c0b3ae0 2024-01-16T14:58:21.275 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:21.275 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":31,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:58:17.186700+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f001406b2c0 msgr2=0x7f001406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f001406b2c0 0x7f001406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f002c0011e0 tx=0x7f002c009040 comp rx=0 tx=0).stop 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0b3ae0 msgr2=0x7f003c15fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0b3ae0 0x7f003c15fb80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f002400d800 tx=0x7f002400dcc0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 shutdown_connections 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f001406b2c0 0x7f001406d780 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f003c0cf290 0x7f003c1600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f003c0b3ae0 0x7f003c15fb80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 --2- 172.21.15.141:0/2904117514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f003c0ae5a0 0x7f003c14fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.296+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 >> 172.21.15.141:0/2904117514 conn(0x7f003c06a800 msgr2=0x7f003c152e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.297+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 shutdown_connections 2024-01-16T14:58:21.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.297+0000 7f0044200640 1 -- 172.21.15.141:0/2904117514 wait complete. 2024-01-16T14:58:21.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:58:21.297 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:58:21.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.522+0000 7fc6f5000640 1 Processor -- start 2024-01-16T14:58:21.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.523+0000 7fc6f5000640 1 -- start start 2024-01-16T14:58:21.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.524+0000 7fc6f5000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013df80 0x7fc6f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.524+0000 7fc6f5000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc6f013e360 0x7fc6f00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.524+0000 7fc6f5000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f006c470 con 0x7fc6f015f9c0 2024-01-16T14:58:21.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.524+0000 7fc6f5000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f006c750 con 0x7fc6f013e360 2024-01-16T14:58:21.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.524+0000 7fc6f5000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f006ca30 con 0x7fc6f013df80 2024-01-16T14:58:21.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6e5200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc6f013e360 0x7fc6f00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6e5200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc6f013e360 0x7fc6f00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55898/0 (socket says 172.21.15.141:55898) 2024-01-16T14:58:21.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6e5200640 1 -- 172.21.15.141:0/2038284029 learned_addr learned my addr 172.21.15.141:0/2038284029 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:21.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6ede00640 1 --2- 172.21.15.141:0/2038284029 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013df80 0x7fc6f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6e5200640 1 -- 172.21.15.141:0/2038284029 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013df80 msgr2=0x7fc6f015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:21.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6e5200640 1 --2- 172.21.15.141:0/2038284029 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013df80 0x7fc6f015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6e5200640 1 -- 172.21.15.141:0/2038284029 >> v1:172.21.15.188:6789/0 conn(0x7fc6f015f9c0 legacy=0x7fc6f01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:21.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.525+0000 7fc6e5200640 1 -- 172.21.15.141:0/2038284029 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f006cd10 con 0x7fc6f013e360 2024-01-16T14:58:21.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.527+0000 7fc6e5200640 1 --2- 172.21.15.141:0/2038284029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013e360 0x7fc6f00cf290 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009870 tx=0x7fc6d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6075b10a1b9ef614 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.527+0000 7fc6ed400640 1 -- 172.21.15.141:0/2038284029 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8031640 con 0x7fc6f013e360 2024-01-16T14:58:21.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.527+0000 7fc6ed400640 1 -- 172.21.15.141:0/2038284029 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d80317a0 con 0x7fc6f013e360 2024-01-16T14:58:21.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.527+0000 7fc6ed400640 1 -- 172.21.15.141:0/2038284029 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8031b30 con 0x7fc6f013e360 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.527+0000 7fc6f5000640 1 -- 172.21.15.141:0/2038284029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013e360 msgr2=0x7fc6f00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.527+0000 7fc6f5000640 1 --2- 172.21.15.141:0/2038284029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013e360 0x7fc6f00cf290 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009870 tx=0x7fc6d802b8f0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.528+0000 7fc6f5000640 1 -- 172.21.15.141:0/2038284029 shutdown_connections 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.528+0000 7fc6f5000640 1 --2- 172.21.15.141:0/2038284029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013e360 0x7fc6f00cf290 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.528+0000 7fc6f5000640 1 --2- 172.21.15.141:0/2038284029 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013df80 0x7fc6f015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.528+0000 7fc6f5000640 1 -- 172.21.15.141:0/2038284029 >> 172.21.15.141:0/2038284029 conn(0x7fc6f006a800 msgr2=0x7fc6f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.528+0000 7fc6f5000640 1 -- 172.21.15.141:0/2038284029 shutdown_connections 2024-01-16T14:58:21.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.528+0000 7fc6f5000640 1 -- 172.21.15.141:0/2038284029 wait complete. 2024-01-16T14:58:21.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.529+0000 7fc6f5000640 1 Processor -- start 2024-01-16T14:58:21.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.530+0000 7fc6f5000640 1 -- start start 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.530+0000 7fc6f5000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013df80 0x7fc6f0135620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.530+0000 7fc6f5000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013e360 0x7fc6f01f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.530+0000 7fc6f5000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f015f9c0 0x7fc6f015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.530+0000 7fc6f5000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f01f43f0 con 0x7fc6f013df80 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.530+0000 7fc6f5000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f01f46d0 con 0x7fc6f015f9c0 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.530+0000 7fc6f5000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f01f49b0 con 0x7fc6f013e360 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013df80 0x7fc6f0135620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013df80 0x7fc6f0135620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55902/0 (socket says 172.21.15.141:55902) 2024-01-16T14:58:21.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 -- 172.21.15.141:0/361417624 learned_addr learned my addr 172.21.15.141:0/361417624 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:21.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6e5200640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013e360 0x7fc6f01f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 -- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013e360 msgr2=0x7fc6f01f3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013e360 0x7fc6f01f3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 -- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f015f9c0 msgr2=0x7fc6f015d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f015f9c0 0x7fc6f015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 -- 172.21.15.141:0/361417624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80094d0 con 0x7fc6f013df80 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.531+0000 7fc6ede00640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013df80 0x7fc6f0135620 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000d5f0 tx=0x7fc6e000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.532+0000 7fc6e6a00640 1 -- 172.21.15.141:0/361417624 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e00125f0 con 0x7fc6f013df80 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.532+0000 7fc6e6a00640 1 -- 172.21.15.141:0/361417624 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0012750 con 0x7fc6f013df80 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.532+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f01f4c90 con 0x7fc6f013df80 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.532+0000 7fc6e6a00640 1 -- 172.21.15.141:0/361417624 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0012b00 con 0x7fc6f013df80 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.532+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f0135b60 con 0x7fc6f013df80 2024-01-16T14:58:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.533+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f00675d0 con 0x7fc6f013df80 2024-01-16T14:58:21.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.534+0000 7fc6e6a00640 1 -- 172.21.15.141:0/361417624 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc6e0012c60 con 0x7fc6f013df80 2024-01-16T14:58:21.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.535+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6c806b2c0 0x7fc6c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.535+0000 7fc6e6a00640 1 -- 172.21.15.141:0/361417624 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc6e0089d70 con 0x7fc6f013df80 2024-01-16T14:58:21.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.536+0000 7fc6e5200640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6c806b2c0 0x7fc6c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.537+0000 7fc6e5200640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6c806b2c0 0x7fc6c806d780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8036000 tx=0x7fc6d803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.539+0000 7fc6e6a00640 1 -- 172.21.15.141:0/361417624 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6e0054f30 con 0x7fc6f013df80 2024-01-16T14:58:21.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.729+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6f006df00 con 0x7fc6f013df80 2024-01-16T14:58:21.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.730+0000 7fc6e6a00640 1 -- 172.21.15.141:0/361417624 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6e0058550 con 0x7fc6f013df80 2024-01-16T14:58:21.711 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:58:21.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.732+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6c806b2c0 msgr2=0x7fc6c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.732+0000 7fc6f5000640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6c806b2c0 0x7fc6c806d780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8036000 tx=0x7fc6d803c040 comp rx=0 tx=0).stop 2024-01-16T14:58:21.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013df80 msgr2=0x7fc6f0135620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013df80 0x7fc6f0135620 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000d5f0 tx=0x7fc6e000dab0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 shutdown_connections 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6c806b2c0 0x7fc6c806d780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f015f9c0 0x7fc6f015d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6f013e360 0x7fc6f01f3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 --2- 172.21.15.141:0/361417624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6f013df80 0x7fc6f0135620 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 >> 172.21.15.141:0/361417624 conn(0x7fc6f006a800 msgr2=0x7fc6f00aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 shutdown_connections 2024-01-16T14:58:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.733+0000 7fc6f5000640 1 -- 172.21.15.141:0/361417624 wait complete. 2024-01-16T14:58:21.733 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T14:58:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.960+0000 7f37fc800640 1 Processor -- start 2024-01-16T14:58:21.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.961+0000 7f37fc800640 1 -- start start 2024-01-16T14:58:21.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.962+0000 7f37fc800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f37f4159040 0x7f37f40cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.962+0000 7f37fc800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f40b34e0 0x7f37f40d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.962+0000 7f37fc800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f40c6060 con 0x7f37f415c300 2024-01-16T14:58:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.962+0000 7f37fc800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f40c62d0 con 0x7f37f4159040 2024-01-16T14:58:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.962+0000 7f37fc800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40c65b0 con 0x7f37f40b34e0 2024-01-16T14:58:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.962+0000 7f37f9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f37f4159040 0x7f37f40cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.963+0000 7f37f9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f37f4159040 0x7f37f40cf000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55908/0 (socket says 172.21.15.141:55908) 2024-01-16T14:58:21.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.963+0000 7f37f9e00640 1 -- 172.21.15.141:0/2696743248 learned_addr learned my addr 172.21.15.141:0/2696743248 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.963+0000 7f37fa800640 1 --2- 172.21.15.141:0/2696743248 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f40b34e0 0x7f37f40d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.963+0000 7f37f9e00640 1 -- 172.21.15.141:0/2696743248 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f40b34e0 msgr2=0x7f37f40d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.963+0000 7f37f9e00640 1 --2- 172.21.15.141:0/2696743248 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f40b34e0 0x7f37f40d0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.963+0000 7f37f9e00640 1 -- 172.21.15.141:0/2696743248 >> v1:172.21.15.188:6789/0 conn(0x7f37f415c300 legacy=0x7f37f415f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:21.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.963+0000 7f37f9e00640 1 -- 172.21.15.141:0/2696743248 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f415eec0 con 0x7f37f4159040 2024-01-16T14:58:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.964+0000 7f37f9e00640 1 --2- 172.21.15.141:0/2696743248 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f4159040 0x7f37f40cf000 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009870 tx=0x7f37e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3def000f37dc83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.964+0000 7f37f8a00640 1 -- 172.21.15.141:0/2696743248 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4031640 con 0x7f37f4159040 2024-01-16T14:58:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37f8a00640 1 -- 172.21.15.141:0/2696743248 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e40317a0 con 0x7f37f4159040 2024-01-16T14:58:21.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37f8a00640 1 -- 172.21.15.141:0/2696743248 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4031b30 con 0x7f37f4159040 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 -- 172.21.15.141:0/2696743248 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f4159040 msgr2=0x7f37f40cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 --2- 172.21.15.141:0/2696743248 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f4159040 0x7f37f40cf000 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009870 tx=0x7f37e402b8f0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 -- 172.21.15.141:0/2696743248 shutdown_connections 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 --2- 172.21.15.141:0/2696743248 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f40b34e0 0x7f37f40d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 --2- 172.21.15.141:0/2696743248 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f4159040 0x7f37f40cf000 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 -- 172.21.15.141:0/2696743248 >> 172.21.15.141:0/2696743248 conn(0x7f37f4066690 msgr2=0x7f37f406a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 -- 172.21.15.141:0/2696743248 shutdown_connections 2024-01-16T14:58:21.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.965+0000 7f37fc800640 1 -- 172.21.15.141:0/2696743248 wait complete. 2024-01-16T14:58:21.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.967+0000 7f37fc800640 1 Processor -- start 2024-01-16T14:58:21.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.967+0000 7f37fc800640 1 -- start start 2024-01-16T14:58:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fc800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37f40b34e0 0x7f37f415b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fc800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f4159040 0x7f37f40b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fc800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f415c300 0x7f37f4152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fc800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40c8140 con 0x7f37f415c300 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fc800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415ca70 con 0x7f37f40b34e0 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fc800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415a150 con 0x7f37f4159040 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f415c300 0x7f37f4152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f415c300 0x7f37f4152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55924/0 (socket says 172.21.15.141:55924) 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 -- 172.21.15.141:0/3110072930 learned_addr learned my addr 172.21.15.141:0/3110072930 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37f9e00640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37f40b34e0 0x7f37f415b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37f9400640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f4159040 0x7f37f40b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 -- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f4159040 msgr2=0x7f37f40b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f4159040 0x7f37f40b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 -- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37f40b34e0 msgr2=0x7f37f415b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37f40b34e0 0x7f37f415b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37fa800640 1 -- 172.21.15.141:0/3110072930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40094d0 con 0x7f37f415c300 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.969+0000 7f37f9400640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f4159040 0x7f37f40b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:21.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.970+0000 7f37fa800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f415c300 0x7f37f4152610 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00c540 tx=0x7f37ec00ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.970+0000 7f37f2000640 1 -- 172.21.15.141:0/3110072930 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec004220 con 0x7f37f415c300 2024-01-16T14:58:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.970+0000 7f37f2000640 1 -- 172.21.15.141:0/3110072930 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec004380 con 0x7f37f415c300 2024-01-16T14:58:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.970+0000 7f37f2000640 1 -- 172.21.15.141:0/3110072930 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec003850 con 0x7f37f415c300 2024-01-16T14:58:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.970+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f415a430 con 0x7f37f415c300 2024-01-16T14:58:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.970+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f415cd30 con 0x7f37f415c300 2024-01-16T14:58:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.972+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f4074a50 con 0x7f37f415c300 2024-01-16T14:58:21.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.973+0000 7f37f2000640 1 -- 172.21.15.141:0/3110072930 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f37ec0039b0 con 0x7f37f415c300 2024-01-16T14:58:21.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.973+0000 7f37f2000640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37cc06b2c0 0x7f37cc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:21.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.974+0000 7f37f2000640 1 -- 172.21.15.141:0/3110072930 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f37ec089590 con 0x7f37f415c300 2024-01-16T14:58:21.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.974+0000 7f37f9e00640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37cc06b2c0 0x7f37cc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:21.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.975+0000 7f37f9e00640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37cc06b2c0 0x7f37cc06d780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37e4036000 tx=0x7f37e403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:21.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:21.978+0000 7f37f2000640 1 -- 172.21.15.141:0/3110072930 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37ec054750 con 0x7f37f415c300 2024-01-16T14:58:22.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.169+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f37f415cf50 con 0x7f37f415c300 2024-01-16T14:58:22.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.170+0000 7f37f2000640 1 -- 172.21.15.141:0/3110072930 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7f37ec057d70 con 0x7f37f415c300 2024-01-16T14:58:22.151 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:22.151 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T14:58:22.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:22.152 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T14:58:22.152 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T14:58:22.152 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:22.152 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T14:58:22.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.172+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37cc06b2c0 msgr2=0x7f37cc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37cc06b2c0 0x7f37cc06d780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37e4036000 tx=0x7f37e403c040 comp rx=0 tx=0).stop 2024-01-16T14:58:22.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f415c300 msgr2=0x7f37f4152610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f415c300 0x7f37f4152610 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00c540 tx=0x7f37ec00ca00 comp rx=0 tx=0).stop 2024-01-16T14:58:22.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 shutdown_connections 2024-01-16T14:58:22.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37cc06b2c0 0x7f37cc06d780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37f415c300 0x7f37f4152610 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37f4159040 0x7f37f40b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 --2- 172.21.15.141:0/3110072930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37f40b34e0 0x7f37f415b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 >> 172.21.15.141:0/3110072930 conn(0x7f37f4066690 msgr2=0x7f37f40cf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:22.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 shutdown_connections 2024-01-16T14:58:22.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.173+0000 7f37fc800640 1 -- 172.21.15.141:0/3110072930 wait complete. 2024-01-16T14:58:22.173 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T14:58:22.173 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T14:58:22.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.408+0000 7fea4a000640 1 Processor -- start 2024-01-16T14:58:22.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.409+0000 7fea4a000640 1 -- start start 2024-01-16T14:58:22.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.410+0000 7fea4a000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fea440ae5a0 0x7fea4415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.410+0000 7fea4a000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea44070c40 con 0x7fea440cf290 2024-01-16T14:58:22.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.410+0000 7fea4a000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4406c470 con 0x7fea440b3ae0 2024-01-16T14:58:22.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.410+0000 7fea4a000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea44071450 con 0x7fea440ae5a0 2024-01-16T14:58:22.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.411+0000 7fea43400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fea440ae5a0 0x7fea4415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:22.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.411+0000 7fea43e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fea440cf290 0x7fea440b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58470/0 (socket says 172.21.15.141:58470) 2024-01-16T14:58:22.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.411+0000 7fea43e00640 1 -- 172.21.15.141:0/2966076998 learned_addr learned my addr 172.21.15.141:0/2966076998 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:22.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.411+0000 7fea43400640 1 -- 172.21.15.141:0/2966076998 >> v1:172.21.15.188:6789/0 conn(0x7fea440b3ae0 legacy=0x7fea4406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:22.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.411+0000 7fea43400640 1 -- 172.21.15.141:0/2966076998 >> v1:172.21.15.141:6789/0 conn(0x7fea440cf290 legacy=0x7fea440b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.411+0000 7fea43400640 1 -- 172.21.15.141:0/2966076998 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4406c750 con 0x7fea440ae5a0 2024-01-16T14:58:22.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.412+0000 7fea43400640 1 --2- 172.21.15.141:0/2966076998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4415e430 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fea340097a0 tx=0x7fea3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c04c19d487450e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:22.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea42a00640 1 -- 172.21.15.141:0/2966076998 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34030610 con 0x7fea440ae5a0 2024-01-16T14:58:22.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea42a00640 1 -- 172.21.15.141:0/2966076998 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea34030770 con 0x7fea440ae5a0 2024-01-16T14:58:22.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea42a00640 1 -- 172.21.15.141:0/2966076998 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34030b00 con 0x7fea440ae5a0 2024-01-16T14:58:22.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea4a000640 1 -- 172.21.15.141:0/2966076998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 msgr2=0x7fea4415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea4a000640 1 --2- 172.21.15.141:0/2966076998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4415e430 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fea340097a0 tx=0x7fea3402b960 comp rx=0 tx=0).stop 2024-01-16T14:58:22.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea4a000640 1 -- 172.21.15.141:0/2966076998 shutdown_connections 2024-01-16T14:58:22.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea4a000640 1 --2- 172.21.15.141:0/2966076998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4415e430 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea4a000640 1 -- 172.21.15.141:0/2966076998 >> 172.21.15.141:0/2966076998 conn(0x7fea4406a800 msgr2=0x7fea4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:22.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea4a000640 1 -- 172.21.15.141:0/2966076998 shutdown_connections 2024-01-16T14:58:22.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.413+0000 7fea4a000640 1 -- 172.21.15.141:0/2966076998 wait complete. 2024-01-16T14:58:22.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.415+0000 7fea4a000640 1 Processor -- start 2024-01-16T14:58:22.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.416+0000 7fea4a000640 1 -- start start 2024-01-16T14:58:22.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea4a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4414bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea4a000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea440b3ae0 0x7fea44147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea4a000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea440cf290 0x7fea44157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea4a000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440b07c0 con 0x7fea440ae5a0 2024-01-16T14:58:22.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea4a000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea44158040 con 0x7fea440cf290 2024-01-16T14:58:22.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea4a000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea440afb40 con 0x7fea440b3ae0 2024-01-16T14:58:22.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea43400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4414bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:22.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea43400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4414bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55944/0 (socket says 172.21.15.141:55944) 2024-01-16T14:58:22.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.417+0000 7fea43400640 1 -- 172.21.15.141:0/3107128612 learned_addr learned my addr 172.21.15.141:0/3107128612 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:22.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43e00640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea440cf290 0x7fea44157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea3a800640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea440b3ae0 0x7fea44147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43400640 1 -- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea440b3ae0 msgr2=0x7fea44147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43400640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea440b3ae0 0x7fea44147bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43400640 1 -- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea440cf290 msgr2=0x7fea44157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43400640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea440cf290 0x7fea44157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43400640 1 -- 172.21.15.141:0/3107128612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea34009400 con 0x7fea440ae5a0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43e00640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea440cf290 0x7fea44157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea43400640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4414bd30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fea340098c0 tx=0x7fea34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea3be00640 1 -- 172.21.15.141:0/3107128612 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3403e040 con 0x7fea440ae5a0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea3be00640 1 -- 172.21.15.141:0/3107128612 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea34039cb0 con 0x7fea440ae5a0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea3be00640 1 -- 172.21.15.141:0/3107128612 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34044630 con 0x7fea440ae5a0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.418+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea440afe20 con 0x7fea440ae5a0 2024-01-16T14:58:22.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.419+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea440b03e0 con 0x7fea440ae5a0 2024-01-16T14:58:22.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.420+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea440b0900 con 0x7fea440ae5a0 2024-01-16T14:58:22.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.420+0000 7fea3be00640 1 -- 172.21.15.141:0/3107128612 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fea3404d3b0 con 0x7fea440ae5a0 2024-01-16T14:58:22.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.421+0000 7fea3be00640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1c06b2c0 0x7fea1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:22.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.421+0000 7fea3be00640 1 -- 172.21.15.141:0/3107128612 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fea340abf10 con 0x7fea440ae5a0 2024-01-16T14:58:22.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.422+0000 7fea3a800640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1c06b2c0 0x7fea1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:22.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.423+0000 7fea3a800640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1c06b2c0 0x7fea1c06d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fea440b35a0 tx=0x7fea2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:22.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.426+0000 7fea3be00640 1 -- 172.21.15.141:0/3107128612 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea34077280 con 0x7fea440ae5a0 2024-01-16T14:58:22.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:22.618+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fea440675d0 con 0x7fea440ae5a0 2024-01-16T14:58:23.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.382+0000 7fea3be00640 1 -- 172.21.15.141:0/3107128612 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7fea3407a8a0 con 0x7fea440ae5a0 2024-01-16T14:58:23.364 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T14:58:23.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1c06b2c0 msgr2=0x7fea1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:23.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1c06b2c0 0x7fea1c06d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fea440b35a0 tx=0x7fea2c009040 comp rx=0 tx=0).stop 2024-01-16T14:58:23.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 msgr2=0x7fea4414bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:23.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4414bd30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fea340098c0 tx=0x7fea34004730 comp rx=0 tx=0).stop 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 shutdown_connections 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1c06b2c0 0x7fea1c06d780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea440cf290 0x7fea44157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea440b3ae0 0x7fea44147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 --2- 172.21.15.141:0/3107128612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea440ae5a0 0x7fea4414bd30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 >> 172.21.15.141:0/3107128612 conn(0x7fea4406a800 msgr2=0x7fea4406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 shutdown_connections 2024-01-16T14:58:23.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.386+0000 7fea4a000640 1 -- 172.21.15.141:0/3107128612 wait complete. 2024-01-16T14:58:23.393 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T14:58:23.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.605+0000 7f800b600640 1 Processor -- start 2024-01-16T14:58:23.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.605+0000 7f800b600640 1 -- start start 2024-01-16T14:58:23.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.606+0000 7f800b600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f800415ed20 0x7f8004141360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:23.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.606+0000 7f800b600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f800406c850 0x7f800406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:23.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.606+0000 7f800b600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800413e440 con 0x7f800415f100 2024-01-16T14:58:23.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.606+0000 7f800b600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040ce1d0 con 0x7f800406c850 2024-01-16T14:58:23.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.606+0000 7f800b600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040ce4b0 con 0x7f800415ed20 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.607+0000 7f8009600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f800406c850 0x7f800406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.607+0000 7f8009600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f800406c850 0x7f800406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53088/0 (socket says 172.21.15.141:53088) 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.607+0000 7f8009600640 1 -- 172.21.15.141:0/3150788698 learned_addr learned my addr 172.21.15.141:0/3150788698 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.607+0000 7f8008c00640 1 --2- 172.21.15.141:0/3150788698 >> v2:172.21.15.188:3300/0 conn(0x7f800415ed20 0x7f8004141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.608+0000 7f8009600640 1 -- 172.21.15.141:0/3150788698 >> v2:172.21.15.188:3300/0 conn(0x7f800415ed20 msgr2=0x7f8004141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.608+0000 7f8009600640 1 --2- 172.21.15.141:0/3150788698 >> v2:172.21.15.188:3300/0 conn(0x7f800415ed20 0x7f8004141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.608+0000 7f8009600640 1 -- 172.21.15.141:0/3150788698 >> v1:172.21.15.141:6789/0 conn(0x7f800415f100 legacy=0x7f80040cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.608+0000 7f8009600640 1 -- 172.21.15.141:0/3150788698 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80040ce790 con 0x7f800406c850 2024-01-16T14:58:23.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.608+0000 7f8003e00640 1 -- 172.21.15.141:0/3150788698 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135174792 0 0) 0x7f800413e440 con 0x7f800415f100 2024-01-16T14:58:23.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.609+0000 7f8009600640 1 --2- 172.21.15.141:0/3150788698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800406cc60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f80040cfee0 tx=0x7f7ffc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5db887a6887555 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:23.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.609+0000 7f8003e00640 1 -- 172.21.15.141:0/3150788698 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc02ca80 con 0x7f800406c850 2024-01-16T14:58:23.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.609+0000 7f8003e00640 1 -- 172.21.15.141:0/3150788698 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc02cbe0 con 0x7f800406c850 2024-01-16T14:58:23.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.609+0000 7f8003e00640 1 -- 172.21.15.141:0/3150788698 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0343f0 con 0x7f800406c850 2024-01-16T14:58:23.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 -- 172.21.15.141:0/3150788698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 msgr2=0x7f800406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:23.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 --2- 172.21.15.141:0/3150788698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800406cc60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f80040cfee0 tx=0x7f7ffc02bee0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 -- 172.21.15.141:0/3150788698 shutdown_connections 2024-01-16T14:58:23.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 --2- 172.21.15.141:0/3150788698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800406cc60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 --2- 172.21.15.141:0/3150788698 >> v2:172.21.15.188:3300/0 conn(0x7f800415ed20 0x7f8004141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 -- 172.21.15.141:0/3150788698 >> 172.21.15.141:0/3150788698 conn(0x7f80040b0be0 msgr2=0x7f80040b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:23.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 -- 172.21.15.141:0/3150788698 shutdown_connections 2024-01-16T14:58:23.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.610+0000 7f800b600640 1 -- 172.21.15.141:0/3150788698 wait complete. 2024-01-16T14:58:23.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.612+0000 7f800b600640 1 Processor -- start 2024-01-16T14:58:23.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.612+0000 7f800b600640 1 -- start start 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f800b600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800413da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f800b600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f800415ed20 0x7f800415d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f800b600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f800415f100 0x7f80040ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f800b600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040b3b70 con 0x7f800406c850 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f800b600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800413df50 con 0x7f800415f100 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f800b600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80040b3f90 con 0x7f800415ed20 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800413da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800413da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53094/0 (socket says 172.21.15.141:53094) 2024-01-16T14:58:23.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 -- 172.21.15.141:0/115842307 learned_addr learned my addr 172.21.15.141:0/115842307 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 -- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f800415ed20 msgr2=0x7f800415d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f800415ed20 0x7f800415d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 -- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f800415f100 msgr2=0x7f80040ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f800415f100 0x7f80040ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.613+0000 7f8008c00640 1 -- 172.21.15.141:0/115842307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc009b30 con 0x7f800406c850 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.614+0000 7f8008c00640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800413da10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400e9b0 tx=0x7f7ff400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.614+0000 7f8001600640 1 -- 172.21.15.141:0/115842307 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4009670 con 0x7f800406c850 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.614+0000 7f8001600640 1 -- 172.21.15.141:0/115842307 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff400c9a0 con 0x7f800406c850 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.614+0000 7f8001600640 1 -- 172.21.15.141:0/115842307 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff400cd50 con 0x7f800406c850 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.614+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80040b2ef0 con 0x7f800406c850 2024-01-16T14:58:23.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.614+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80040b34b0 con 0x7f800406c850 2024-01-16T14:58:23.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.615+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80040b3de0 con 0x7f800406c850 2024-01-16T14:58:23.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.616+0000 7f8001600640 1 -- 172.21.15.141:0/115842307 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7ff4013470 con 0x7f800406c850 2024-01-16T14:58:23.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.617+0000 7f8001600640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fdc06b2c0 0x7f7fdc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:23.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.617+0000 7f8001600640 1 -- 172.21.15.141:0/115842307 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7ff4089fe0 con 0x7f800406c850 2024-01-16T14:58:23.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.618+0000 7f7ffbc00640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fdc06b2c0 0x7f7fdc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:23.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.619+0000 7f7ffbc00640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fdc06b2c0 0x7f7fdc06d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7fec000f60 tx=0x7f7fec0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:23.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.621+0000 7f8001600640 1 -- 172.21.15.141:0/115842307 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ff4055100 con 0x7f800406c850 2024-01-16T14:58:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:23.812+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8004067670 con 0x7f800406c850 2024-01-16T14:58:24.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.389+0000 7f8001600640 1 -- 172.21.15.141:0/115842307 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7ff4058720 con 0x7f800406c850 2024-01-16T14:58:24.371 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.393+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fdc06b2c0 msgr2=0x7f7fdc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.393+0000 7f800b600640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fdc06b2c0 0x7f7fdc06d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7fec000f60 tx=0x7f7fec0079b0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.393+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 msgr2=0x7f800413da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.393+0000 7f800b600640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800413da10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400e9b0 tx=0x7f7ff400ee70 comp rx=0 tx=0).stop 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.393+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 shutdown_connections 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.393+0000 7f800b600640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fdc06b2c0 0x7f7fdc06d780 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.394+0000 7f800b600640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f800415f100 0x7f80040ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.394+0000 7f800b600640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f800415ed20 0x7f800415d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.394+0000 7f800b600640 1 --2- 172.21.15.141:0/115842307 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f800406c850 0x7f800413da10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.394+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 >> 172.21.15.141:0/115842307 conn(0x7f80040b0be0 msgr2=0x7f800415b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:24.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.394+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 shutdown_connections 2024-01-16T14:58:24.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.394+0000 7f800b600640 1 -- 172.21.15.141:0/115842307 wait complete. 2024-01-16T14:58:24.412 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T14:58:24.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.614+0000 7f6b04c00640 1 Processor -- start 2024-01-16T14:58:24.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.614+0000 7f6b04c00640 1 -- start start 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.615+0000 7f6b04c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.615+0000 7f6b04c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6b000cf9a0 0x7f6b000cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.615+0000 7f6b04c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0006c470 con 0x7f6b0013e360 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.615+0000 7f6b04c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0006c750 con 0x7f6b000cf9a0 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.615+0000 7f6b04c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0006ca30 con 0x7f6b0013df80 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.616+0000 7f6afde00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.616+0000 7f6afde00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:49272/0 (socket says 172.21.15.141:49272) 2024-01-16T14:58:24.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.616+0000 7f6afde00640 1 -- 172.21.15.141:0/57244741 learned_addr learned my addr 172.21.15.141:0/57244741 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.616+0000 7f6afe800640 1 --2- 172.21.15.141:0/57244741 >> v2:172.21.15.188:3300/0 conn(0x7f6b000cf9a0 0x7f6b000cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.617+0000 7f6afe800640 1 -- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 msgr2=0x7f6b0015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.617+0000 7f6afe800640 1 --2- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.617+0000 7f6afe800640 1 -- 172.21.15.141:0/57244741 >> v1:172.21.15.188:6789/0 conn(0x7f6b0013e360 legacy=0x7f6b000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.617+0000 7f6afe800640 1 -- 172.21.15.141:0/57244741 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0006cd10 con 0x7f6b000cf9a0 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.617+0000 7f6afde00640 1 --2- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.618+0000 7f6afe800640 1 --2- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 0x7f6b000cd840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6af80097a0 tx=0x7f6af802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af31e3b6abf4f726 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.618+0000 7f6afd400640 1 -- 172.21.15.141:0/57244741 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af802d040 con 0x7f6b000cf9a0 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6afd400640 1 -- 172.21.15.141:0/57244741 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8031810 con 0x7f6b000cf9a0 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6afd400640 1 -- 172.21.15.141:0/57244741 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8031ba0 con 0x7f6b000cf9a0 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 -- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 msgr2=0x7f6b000cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 --2- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 0x7f6b000cd840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6af80097a0 tx=0x7f6af802b8b0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 -- 172.21.15.141:0/57244741 shutdown_connections 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 --2- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 0x7f6b000cd840 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 --2- 172.21.15.141:0/57244741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 -- 172.21.15.141:0/57244741 >> 172.21.15.141:0/57244741 conn(0x7f6b0006a800 msgr2=0x7f6b0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 -- 172.21.15.141:0/57244741 shutdown_connections 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.619+0000 7f6b04c00640 1 -- 172.21.15.141:0/57244741 wait complete. 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.621+0000 7f6b04c00640 1 Processor -- start 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.621+0000 7f6b04c00640 1 -- start start 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.622+0000 7f6b04c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 0x7f6b000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.622+0000 7f6b04c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0013d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.622+0000 7f6b04c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0013e360 0x7f6b000b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.622+0000 7f6b04c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00071520 con 0x7f6b0013e360 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.622+0000 7f6b04c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0013d830 con 0x7f6b000cf9a0 2024-01-16T14:58:24.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.622+0000 7f6b04c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000708a0 con 0x7f6b0013df80 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0013e360 0x7f6b000b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0013e360 0x7f6b000b0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53102/0 (socket says 172.21.15.141:53102) 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 -- 172.21.15.141:0/129286718 learned_addr learned my addr 172.21.15.141:0/129286718 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6af5200640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0013d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afde00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 0x7f6b000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 -- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 msgr2=0x7f6b0013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0013d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 -- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 msgr2=0x7f6b000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 0x7f6b000702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.623+0000 7f6afe800640 1 -- 172.21.15.141:0/129286718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af00094d0 con 0x7f6b0013e360 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.624+0000 7f6afe800640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0013e360 0x7f6b000b0330 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6af80097a0 tx=0x7f6af80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.624+0000 7f6af6a00640 1 -- 172.21.15.141:0/129286718 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8006e10 con 0x7f6b0013e360 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.624+0000 7f6af6a00640 1 -- 172.21.15.141:0/129286718 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af803ace0 con 0x7f6b0013e360 2024-01-16T14:58:24.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.624+0000 7f6af6a00640 1 -- 172.21.15.141:0/129286718 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8044640 con 0x7f6b0013e360 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.624+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af8009400 con 0x7f6b0013e360 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.624+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b00070e60 con 0x7f6b0013e360 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.625+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b000675d0 con 0x7f6b0013e360 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.627+0000 7f6af6a00640 1 -- 172.21.15.141:0/129286718 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6af804d440 con 0x7f6b0013e360 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.627+0000 7f6af6a00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806b2c0 0x7f6ad806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.628+0000 7f6af6a00640 1 -- 172.21.15.141:0/129286718 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6af80abe40 con 0x7f6b0013e360 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.628+0000 7f6afde00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806b2c0 0x7f6ad806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.629+0000 7f6afde00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806b2c0 0x7f6ad806d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6b0015d350 tx=0x7f6af0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.633+0000 7f6af6a00640 1 -- 172.21.15.141:0/129286718 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6af8076c70 con 0x7f6b0013e360 2024-01-16T14:58:24.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:24.850+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6b0015cc80 con 0x7f6b0013e360 2024-01-16T14:58:25.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.429+0000 7f6af6a00640 1 -- 172.21.15.141:0/129286718 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6af807a290 con 0x7f6b0013e360 2024-01-16T14:58:25.414 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 4 and data pool 5 2024-01-16T14:58:25.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.437+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806b2c0 msgr2=0x7f6ad806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.437+0000 7f6b04c00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806b2c0 0x7f6ad806d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6b0015d350 tx=0x7f6af0008040 comp rx=0 tx=0).stop 2024-01-16T14:58:25.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.437+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0013e360 msgr2=0x7f6b000b0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.437+0000 7f6b04c00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0013e360 0x7f6b000b0330 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6af80097a0 tx=0x7f6af80046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.439+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 shutdown_connections 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.439+0000 7f6b04c00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806b2c0 0x7f6ad806d780 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.439+0000 7f6b04c00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0013e360 0x7f6b000b0330 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.439+0000 7f6b04c00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b0013df80 0x7f6b0013d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.439+0000 7f6b04c00640 1 --2- 172.21.15.141:0/129286718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b000cf9a0 0x7f6b000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.439+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 >> 172.21.15.141:0/129286718 conn(0x7f6b0006a800 msgr2=0x7f6b000ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.440+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 shutdown_connections 2024-01-16T14:58:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.441+0000 7f6b04c00640 1 -- 172.21.15.141:0/129286718 wait complete. 2024-01-16T14:58:25.456 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:25.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.644+0000 7fd1f5a00640 1 Processor -- start 2024-01-16T14:58:25.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.645+0000 7fd1f5a00640 1 -- start start 2024-01-16T14:58:25.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.646+0000 7fd1f5a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 0x7fd1f00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.646+0000 7fd1f5a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd1f015f9c0 0x7fd1f00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.646+0000 7fd1f5a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f015c5f0 con 0x7fd1f013df80 2024-01-16T14:58:25.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.646+0000 7fd1f5a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f006c470 con 0x7fd1f015f9c0 2024-01-16T14:58:25.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.646+0000 7fd1f5a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f006c750 con 0x7fd1f013e360 2024-01-16T14:58:25.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.646+0000 7fd1eea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd1f013df80 0x7fd1f015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42774/0 (socket says 172.21.15.141:42774) 2024-01-16T14:58:25.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.646+0000 7fd1eea00640 1 -- 172.21.15.141:0/305127746 learned_addr learned my addr 172.21.15.141:0/305127746 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:25.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544862441 0 0) 0x7fd1f015c5f0 con 0x7fd1f013df80 2024-01-16T14:58:25.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d0003520 con 0x7fd1f013df80 2024-01-16T14:58:25.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1e5e00640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 0x7fd1f00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ef400640 1 --2- 172.21.15.141:0/305127746 >> v2:172.21.15.188:3300/0 conn(0x7fd1f015f9c0 0x7fd1f00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043781471 0 0) 0x7fd1d0003520 con 0x7fd1f013df80 2024-01-16T14:58:25.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 msgr2=0x7fd1f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 0x7fd1f00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 >> v2:172.21.15.188:3300/0 conn(0x7fd1f015f9c0 msgr2=0x7fd1f00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 --2- 172.21.15.141:0/305127746 >> v2:172.21.15.188:3300/0 conn(0x7fd1f015f9c0 0x7fd1f00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f006ca30 con 0x7fd1f013df80 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd1e0001670 con 0x7fd1f013df80 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ef400640 1 --2- 172.21.15.141:0/305127746 >> v2:172.21.15.188:3300/0 conn(0x7fd1f015f9c0 0x7fd1f00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 >> v1:172.21.15.141:6789/0 conn(0x7fd1f013df80 legacy=0x7fd1f015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0003970 0x7fd1d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 0x7fd1d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1d0004d00 0x7fd1d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f006c470 con 0x7fd1d0004d00 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f006c750 con 0x7fd1d0003970 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d000b9b0 con 0x7fd1f013e360 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.647+0000 7fd1ef400640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1d0004d00 0x7fd1d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.648+0000 7fd1e5e00640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 0x7fd1d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.648+0000 7fd1ef400640 1 -- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 msgr2=0x7fd1d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.648+0000 7fd1eea00640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0003970 0x7fd1d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.648+0000 7fd1ef400640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f013e360 0x7fd1d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.648+0000 7fd1ef400640 1 -- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0003970 msgr2=0x7fd1d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.648+0000 7fd1ef400640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0003970 0x7fd1d0003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.648+0000 7fd1ef400640 1 -- 172.21.15.141:0/305127746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d0003520 con 0x7fd1d0004d00 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.649+0000 7fd1ef400640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1d0004d00 0x7fd1d0005180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80073e0 tx=0x7fd1d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904dc540826d65b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.649+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e802cdc0 con 0x7fd1d0004d00 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.649+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e8034440 con 0x7fd1d0004d00 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.649+0000 7fd1ee000640 1 -- 172.21.15.141:0/305127746 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80347b0 con 0x7fd1d0004d00 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.650+0000 7fd1f5a00640 1 -- 172.21.15.141:0/305127746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1d0004d00 msgr2=0x7fd1d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.650+0000 7fd1f5a00640 1 --2- 172.21.15.141:0/305127746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1d0004d00 0x7fd1d0005180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80073e0 tx=0x7fd1d00057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.650+0000 7fd1ef400640 1 -- 172.21.15.141:0/305127746 reap_dead start 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.650+0000 7fd1f5a00640 1 -- 172.21.15.141:0/305127746 shutdown_connections 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.650+0000 7fd1f5a00640 1 -- 172.21.15.141:0/305127746 >> 172.21.15.141:0/305127746 conn(0x7fd1f006a800 msgr2=0x7fd1f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.650+0000 7fd1f5a00640 1 -- 172.21.15.141:0/305127746 shutdown_connections 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.650+0000 7fd1f5a00640 1 -- 172.21.15.141:0/305127746 wait complete. 2024-01-16T14:58:25.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.652+0000 7fd1f5a00640 1 Processor -- start 2024-01-16T14:58:25.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.652+0000 7fd1f5a00640 1 -- start start 2024-01-16T14:58:25.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1f5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0004d00 0x7fd1f00addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1f5a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1f00af260 0x7fd1f00af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1f5a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f00a84c0 0x7fd1f00a8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1f5a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f01322c0 con 0x7fd1f00af260 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1f5a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00afc00 con 0x7fd1d0004d00 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1f5a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00a8e80 con 0x7fd1f00a84c0 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1f00af260 0x7fd1f00af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1f00af260 0x7fd1f00af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53118/0 (socket says 172.21.15.141:53118) 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 -- 172.21.15.141:0/8815582 learned_addr learned my addr 172.21.15.141:0/8815582 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 -- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f00a84c0 msgr2=0x7fd1f00a8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f00a84c0 0x7fd1f00a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 -- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0004d00 msgr2=0x7fd1f00addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0004d00 0x7fd1f00addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 -- 172.21.15.141:0/8815582 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f00ae310 con 0x7fd1f00af260 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.653+0000 7fd1e5e00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1f00af260 0x7fd1f00af6c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80115a0 tx=0x7fd1d8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.654+0000 7fd1e7400640 1 -- 172.21.15.141:0/8815582 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8015930 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.654+0000 7fd1e7400640 1 -- 172.21.15.141:0/8815582 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d8015a90 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.654+0000 7fd1e7400640 1 -- 172.21.15.141:0/8815582 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8015e20 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.654+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f01f7060 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.654+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f01f6580 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.655+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f00675d0 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.656+0000 7fd1e7400640 1 -- 172.21.15.141:0/8815582 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd1d801d400 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.657+0000 7fd1e7400640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1c806b2c0 0x7fd1c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.657+0000 7fd1e7400640 1 -- 172.21.15.141:0/8815582 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1d808a2f0 con 0x7fd1f00af260 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.657+0000 7fd1eea00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1c806b2c0 0x7fd1c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:25.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.658+0000 7fd1eea00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1c806b2c0 0x7fd1c806d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00018a0 tx=0x7fd1e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:25.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.662+0000 7fd1e7400640 1 -- 172.21.15.141:0/8815582 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1d80551f0 con 0x7fd1f00af260 2024-01-16T14:58:25.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.838+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1f006df00 con 0x7fd1f00af260 2024-01-16T14:58:25.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.841+0000 7fd1e7400640 1 -- 172.21.15.141:0/8815582 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16630 (secure 0 0 0) 0x7fd1d8058810 con 0x7fd1f00af260 2024-01-16T14:58:25.823 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:25.823 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":34,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:58:24.852215+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:58:22.670734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T14:58:23.861781+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.844+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1c806b2c0 msgr2=0x7fd1c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.844+0000 7fd1f5a00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1c806b2c0 0x7fd1c806d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00018a0 tx=0x7fd1e0009040 comp rx=0 tx=0).stop 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.844+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1f00af260 msgr2=0x7fd1f00af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.844+0000 7fd1f5a00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1f00af260 0x7fd1f00af6c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80115a0 tx=0x7fd1d8011a60 comp rx=0 tx=0).stop 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 shutdown_connections 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1c806b2c0 0x7fd1c806d780 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd1f00a84c0 0x7fd1f00a8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd1f00af260 0x7fd1f00af6c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 --2- 172.21.15.141:0/8815582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0004d00 0x7fd1f00addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:25.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 >> 172.21.15.141:0/8815582 conn(0x7fd1f006a800 msgr2=0x7fd1f00aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:25.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 shutdown_connections 2024-01-16T14:58:25.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:25.845+0000 7fd1f5a00640 1 -- 172.21.15.141:0/8815582 wait complete. 2024-01-16T14:58:25.845 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:26.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.060+0000 7f3c62000640 1 Processor -- start 2024-01-16T14:58:26.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.060+0000 7f3c62000640 1 -- start start 2024-01-16T14:58:26.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.061+0000 7f3c62000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3c5c143c40 0x7f3c5c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.061+0000 7f3c62000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 0x7f3c5c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.061+0000 7f3c62000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c070e50 con 0x7f3c5c0cca40 2024-01-16T14:58:26.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.061+0000 7f3c62000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5c0723c0 con 0x7f3c5c143c40 2024-01-16T14:58:26.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.061+0000 7f3c62000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0cce70 con 0x7f3c5c0717d0 2024-01-16T14:58:26.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.061+0000 7f3c61600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3c5c0cca40 0x7f3c5c15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42788/0 (socket says 172.21.15.141:42788) 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c61600640 1 -- 172.21.15.141:0/914181111 learned_addr learned my addr 172.21.15.141:0/914181111 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c60c00640 1 --2- 172.21.15.141:0/914181111 >> v2:172.21.15.188:3300/0 conn(0x7f3c5c143c40 0x7f3c5c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5be00640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 0x7f3c5c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433474922 0 0) 0x7f3c5c070e50 con 0x7f3c5c0cca40 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c3c003520 con 0x7f3c5c0cca40 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 780268717 0 0) 0x7f3c3c003520 con 0x7f3c5c0cca40 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 msgr2=0x7f3c5c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 0x7f3c5c0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 >> v2:172.21.15.188:3300/0 conn(0x7f3c5c143c40 msgr2=0x7f3c5c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 --2- 172.21.15.141:0/914181111 >> v2:172.21.15.188:3300/0 conn(0x7f3c5c143c40 0x7f3c5c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c06e170 con 0x7f3c5c0cca40 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3c540019f0 con 0x7f3c5c0cca40 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 >> v1:172.21.15.141:6789/0 conn(0x7f3c5c0cca40 legacy=0x7f3c5c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c003970 0x7f3c3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.062+0000 7f3c5b400640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 0x7f3c3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0723c0 con 0x7f3c3c0047c0 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0cce70 con 0x7f3c3c003970 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c00ba00 con 0x7f3c5c0717d0 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5be00640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c61600640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 0x7f3c3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5be00640 1 -- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 msgr2=0x7f3c3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5be00640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0717d0 0x7f3c3c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5be00640 1 -- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c003970 msgr2=0x7f3c3c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5be00640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c003970 0x7f3c3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.063+0000 7f3c5be00640 1 -- 172.21.15.141:0/914181111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c003520 con 0x7f3c3c0047c0 2024-01-16T14:58:26.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.064+0000 7f3c5be00640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c3c004c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009ac0 tx=0x7f3c3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15021c182732da0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4402c6b0 con 0x7f3c3c0047c0 2024-01-16T14:58:26.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4402c810 con 0x7f3c3c0047c0 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c5b400640 1 -- 172.21.15.141:0/914181111 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4402cba0 con 0x7f3c3c0047c0 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c62000640 1 -- 172.21.15.141:0/914181111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 msgr2=0x7f3c3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c62000640 1 --2- 172.21.15.141:0/914181111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c3c004c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009ac0 tx=0x7f3c3c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c61600640 1 -- 172.21.15.141:0/914181111 reap_dead start 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c62000640 1 -- 172.21.15.141:0/914181111 shutdown_connections 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c62000640 1 -- 172.21.15.141:0/914181111 >> 172.21.15.141:0/914181111 conn(0x7f3c5c06a800 msgr2=0x7f3c5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c62000640 1 -- 172.21.15.141:0/914181111 shutdown_connections 2024-01-16T14:58:26.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.065+0000 7f3c62000640 1 -- 172.21.15.141:0/914181111 wait complete. 2024-01-16T14:58:26.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.067+0000 7f3c62000640 1 Processor -- start 2024-01-16T14:58:26.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.067+0000 7f3c62000640 1 -- start start 2024-01-16T14:58:26.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c62000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c5c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c62000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0b1f00 0x7f3c5c1f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c62000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c5c13ea70 0x7f3c5c13eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c62000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c13f430 con 0x7f3c3c0047c0 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c62000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0c3de0 con 0x7f3c5c13ea70 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c62000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0ad5f0 con 0x7f3c5c0b1f00 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c60c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c5c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c60c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c5c159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53134/0 (socket says 172.21.15.141:53134) 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.068+0000 7f3c60c00640 1 -- 172.21.15.141:0/3029149802 learned_addr learned my addr 172.21.15.141:0/3029149802 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c60c00640 1 -- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0b1f00 msgr2=0x7f3c5c1f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c60c00640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0b1f00 0x7f3c5c1f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c61600640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c5c13ea70 0x7f3c5c13eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c60c00640 1 -- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c5c13ea70 msgr2=0x7f3c5c13eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c60c00640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c5c13ea70 0x7f3c5c13eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c60c00640 1 -- 172.21.15.141:0/3029149802 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0b26c0 con 0x7f3c3c0047c0 2024-01-16T14:58:26.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c60c00640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c5c159a90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00e8c0 tx=0x7f3c4c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c58c00640 1 -- 172.21.15.141:0/3029149802 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c00c940 con 0x7f3c3c0047c0 2024-01-16T14:58:26.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c58c00640 1 -- 172.21.15.141:0/3029149802 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4c00cac0 con 0x7f3c3c0047c0 2024-01-16T14:58:26.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c58c00640 1 -- 172.21.15.141:0/3029149802 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c00ce70 con 0x7f3c3c0047c0 2024-01-16T14:58:26.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.069+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c1f4080 con 0x7f3c3c0047c0 2024-01-16T14:58:26.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.070+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c0accc0 con 0x7f3c3c0047c0 2024-01-16T14:58:26.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.071+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c067450 con 0x7f3c3c0047c0 2024-01-16T14:58:26.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.072+0000 7f3c58c00640 1 -- 172.21.15.141:0/3029149802 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3c4c013470 con 0x7f3c3c0047c0 2024-01-16T14:58:26.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.072+0000 7f3c58c00640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c3406b2c0 0x7f3c3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.072+0000 7f3c58c00640 1 -- 172.21.15.141:0/3029149802 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c4c089e30 con 0x7f3c3c0047c0 2024-01-16T14:58:26.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.073+0000 7f3c5be00640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c3406b2c0 0x7f3c3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.074+0000 7f3c5be00640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c3406b2c0 0x7f3c3406d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009ac0 tx=0x7f3c4402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.077+0000 7f3c58c00640 1 -- 172.21.15.141:0/3029149802 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c4c058690 con 0x7f3c3c0047c0 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.265+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c5c06cc30 con 0x7f3c3c0047c0 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.267+0000 7f3c58c00640 1 -- 172.21.15.141:0/3029149802 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f3c4c017090 con 0x7f3c3c0047c0 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":34,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:58:24.852215+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:58:22.670734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T14:58:23.861781+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c3406b2c0 msgr2=0x7f3c3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c3406b2c0 0x7f3c3406d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009ac0 tx=0x7f3c4402e000 comp rx=0 tx=0).stop 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 msgr2=0x7f3c5c159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c5c159a90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00e8c0 tx=0x7f3c4c00ed80 comp rx=0 tx=0).stop 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 shutdown_connections 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c3406b2c0 0x7f3c3406d780 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c5c13ea70 0x7f3c5c13eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c5c0b1f00 0x7f3c5c1f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 --2- 172.21.15.141:0/3029149802 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c3c0047c0 0x7f3c5c159a90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.270+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 >> 172.21.15.141:0/3029149802 conn(0x7f3c5c06a800 msgr2=0x7f3c5c0afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.271+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 shutdown_connections 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.271+0000 7f3c62000640 1 -- 172.21.15.141:0/3029149802 wait complete. 2024-01-16T14:58:26.270 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T14:58:26.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.501+0000 7ff4d4c00640 1 Processor -- start 2024-01-16T14:58:26.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.501+0000 7ff4d4c00640 1 -- start start 2024-01-16T14:58:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.503+0000 7ff4d4c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff4d0157c30 0x7ff4d00cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.503+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0071200 0x7ff4d006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.503+0000 7ff4d4c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d015deb0 con 0x7ff4d01618f0 2024-01-16T14:58:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.503+0000 7ff4d4c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d015ea30 con 0x7ff4d0157c30 2024-01-16T14:58:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.503+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d006d860 con 0x7ff4d0071200 2024-01-16T14:58:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.503+0000 7ff4ce800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0071200 0x7ff4d006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.504+0000 7ff4ce800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0071200 0x7ff4d006dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:49342/0 (socket says 172.21.15.141:49342) 2024-01-16T14:58:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.504+0000 7ff4ce800640 1 -- 172.21.15.141:0/2674344812 learned_addr learned my addr 172.21.15.141:0/2674344812 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.504+0000 7ff4cde00640 1 --2- 172.21.15.141:0/2674344812 >> v2:172.21.15.188:3300/0 conn(0x7ff4d0157c30 0x7ff4d00cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.504+0000 7ff4ce800640 1 -- 172.21.15.141:0/2674344812 >> v2:172.21.15.188:3300/0 conn(0x7ff4d0157c30 msgr2=0x7ff4d00cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:26.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.504+0000 7ff4ce800640 1 --2- 172.21.15.141:0/2674344812 >> v2:172.21.15.188:3300/0 conn(0x7ff4d0157c30 0x7ff4d00cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.504+0000 7ff4ce800640 1 -- 172.21.15.141:0/2674344812 >> v1:172.21.15.188:6789/0 conn(0x7ff4d01618f0 legacy=0x7ff4d0070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.504+0000 7ff4ce800640 1 -- 172.21.15.141:0/2674344812 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d015e190 con 0x7ff4d0071200 2024-01-16T14:58:26.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.506+0000 7ff4ce800640 1 --2- 172.21.15.141:0/2674344812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0071200 0x7ff4d006dcb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009e50 tx=0x7ff4c802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cf31af245124e22 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.506+0000 7ff4cca00640 1 -- 172.21.15.141:0/2674344812 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c802cb00 con 0x7ff4d0071200 2024-01-16T14:58:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4cca00640 1 -- 172.21.15.141:0/2674344812 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c802cc60 con 0x7ff4d0071200 2024-01-16T14:58:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4cca00640 1 -- 172.21.15.141:0/2674344812 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c80343f0 con 0x7ff4d0071200 2024-01-16T14:58:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 -- 172.21.15.141:0/2674344812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0071200 msgr2=0x7ff4d006dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/2674344812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0071200 0x7ff4d006dcb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009e50 tx=0x7ff4c802bf60 comp rx=0 tx=0).stop 2024-01-16T14:58:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 -- 172.21.15.141:0/2674344812 shutdown_connections 2024-01-16T14:58:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/2674344812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0071200 0x7ff4d006dcb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/2674344812 >> v2:172.21.15.188:3300/0 conn(0x7ff4d0157c30 0x7ff4d00cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 -- 172.21.15.141:0/2674344812 >> 172.21.15.141:0/2674344812 conn(0x7ff4d00cf700 msgr2=0x7ff4d00cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 -- 172.21.15.141:0/2674344812 shutdown_connections 2024-01-16T14:58:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.507+0000 7ff4d4c00640 1 -- 172.21.15.141:0/2674344812 wait complete. 2024-01-16T14:58:26.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.509+0000 7ff4d4c00640 1 Processor -- start 2024-01-16T14:58:26.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.509+0000 7ff4d4c00640 1 -- start start 2024-01-16T14:58:26.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4d0071200 0x7ff4d01570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0157c30 0x7ff4d00b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4d01618f0 0x7ff4d00b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d0156770 con 0x7ff4d0071200 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00b23b0 con 0x7ff4d01618f0 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00b2690 con 0x7ff4d0157c30 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4cde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4d0071200 0x7ff4d01570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4cde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4d0071200 0x7ff4d01570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53150/0 (socket says 172.21.15.141:53150) 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.510+0000 7ff4cde00640 1 -- 172.21.15.141:0/520099955 learned_addr learned my addr 172.21.15.141:0/520099955 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cd400640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0157c30 0x7ff4d00b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4ce800640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4d01618f0 0x7ff4d00b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cde00640 1 -- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0157c30 msgr2=0x7ff4d00b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cde00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0157c30 0x7ff4d00b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cde00640 1 -- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4d01618f0 msgr2=0x7ff4d00b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cde00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4d01618f0 0x7ff4d00b0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cde00640 1 -- 172.21.15.141:0/520099955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c8009b30 con 0x7ff4d0071200 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cd400640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0157c30 0x7ff4d00b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4cde00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4d0071200 0x7ff4d01570b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e9b0 tx=0x7ff4b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4c6000640 1 -- 172.21.15.141:0/520099955 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8009670 con 0x7ff4d0071200 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4c6000640 1 -- 172.21.15.141:0/520099955 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b800c9a0 con 0x7ff4d0071200 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d00b2970 con 0x7ff4d0071200 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4c6000640 1 -- 172.21.15.141:0/520099955 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800cd50 con 0x7ff4d0071200 2024-01-16T14:58:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.511+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d00b2f30 con 0x7ff4d0071200 2024-01-16T14:58:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.512+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d00675d0 con 0x7ff4d0071200 2024-01-16T14:58:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.514+0000 7ff4c6000640 1 -- 172.21.15.141:0/520099955 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff4b8013470 con 0x7ff4d0071200 2024-01-16T14:58:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.514+0000 7ff4c6000640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4a806b2c0 0x7ff4a806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:26.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.515+0000 7ff4c6000640 1 -- 172.21.15.141:0/520099955 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4b8089f00 con 0x7ff4d0071200 2024-01-16T14:58:26.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.515+0000 7ff4cd400640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4a806b2c0 0x7ff4a806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:26.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.516+0000 7ff4cd400640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4a806b2c0 0x7ff4a806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00b1340 tx=0x7ff4c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:26.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.519+0000 7ff4c6000640 1 -- 172.21.15.141:0/520099955 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4b8054cf0 con 0x7ff4d0071200 2024-01-16T14:58:26.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:26.748+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff4d00b4970 con 0x7ff4d0071200 2024-01-16T14:58:27.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.452+0000 7ff4c6000640 1 -- 172.21.15.141:0/520099955 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff4b8058310 con 0x7ff4d0071200 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4a806b2c0 msgr2=0x7ff4a806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4a806b2c0 0x7ff4a806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00b1340 tx=0x7ff4c0009040 comp rx=0 tx=0).stop 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4d0071200 msgr2=0x7ff4d01570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4d0071200 0x7ff4d01570b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e9b0 tx=0x7ff4b800ee70 comp rx=0 tx=0).stop 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 shutdown_connections 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4a806b2c0 0x7ff4a806d780 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4d01618f0 0x7ff4d00b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4d0157c30 0x7ff4d00b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 --2- 172.21.15.141:0/520099955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4d0071200 0x7ff4d01570b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.455+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 >> 172.21.15.141:0/520099955 conn(0x7ff4d00cf700 msgr2=0x7ff4d015eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:27.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.456+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 shutdown_connections 2024-01-16T14:58:27.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.456+0000 7ff4d4c00640 1 -- 172.21.15.141:0/520099955 wait complete. 2024-01-16T14:58:27.456 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T14:58:27.457 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T14:58:27.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.689+0000 7fde1be00640 1 Processor -- start 2024-01-16T14:58:27.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.689+0000 7fde1be00640 1 -- start start 2024-01-16T14:58:27.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde1c0adf20 0x7fde1c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1be00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fde1c15bf10 0x7fde1c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c0cc980 0x7fde1c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1be00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c15f970 con 0x7fde1c0adf20 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1be00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c15f4e0 con 0x7fde1c15bf10 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1be00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0cce10 con 0x7fde1c0cc980 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde1c0adf20 0x7fde1c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde1c0adf20 0x7fde1c070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53164/0 (socket says 172.21.15.141:53164) 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.691+0000 7fde1aa00640 1 -- 172.21.15.141:0/3377523387 learned_addr learned my addr 172.21.15.141:0/3377523387 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.692+0000 7fde1a000640 1 --2- 172.21.15.141:0/3377523387 >> v2:172.21.15.188:3300/0 conn(0x7fde1c15bf10 0x7fde1c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.692+0000 7fde1aa00640 1 -- 172.21.15.141:0/3377523387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c0cc980 msgr2=0x7fde1c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.692+0000 7fde1aa00640 1 --2- 172.21.15.141:0/3377523387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c0cc980 0x7fde1c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.692+0000 7fde1aa00640 1 -- 172.21.15.141:0/3377523387 >> v2:172.21.15.188:3300/0 conn(0x7fde1c15bf10 msgr2=0x7fde1c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.692+0000 7fde1aa00640 1 --2- 172.21.15.141:0/3377523387 >> v2:172.21.15.188:3300/0 conn(0x7fde1c15bf10 0x7fde1c0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.692+0000 7fde1aa00640 1 -- 172.21.15.141:0/3377523387 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c071130 con 0x7fde1c0adf20 2024-01-16T14:58:27.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.692+0000 7fde1a000640 1 --2- 172.21.15.141:0/3377523387 >> v2:172.21.15.188:3300/0 conn(0x7fde1c15bf10 0x7fde1c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.693+0000 7fde1aa00640 1 --2- 172.21.15.141:0/3377523387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0adf20 0x7fde1c070a30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fde1c15e9e0 tx=0x7fde1402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7011925b636bb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.693+0000 7fde19600640 1 -- 172.21.15.141:0/3377523387 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde140317e0 con 0x7fde1c0adf20 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.693+0000 7fde19600640 1 -- 172.21.15.141:0/3377523387 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde14031940 con 0x7fde1c0adf20 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.693+0000 7fde19600640 1 -- 172.21.15.141:0/3377523387 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14031cd0 con 0x7fde1c0adf20 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 -- 172.21.15.141:0/3377523387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0adf20 msgr2=0x7fde1c070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 --2- 172.21.15.141:0/3377523387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0adf20 0x7fde1c070a30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fde1c15e9e0 tx=0x7fde1402b880 comp rx=0 tx=0).stop 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 -- 172.21.15.141:0/3377523387 shutdown_connections 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 --2- 172.21.15.141:0/3377523387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c0cc980 0x7fde1c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 --2- 172.21.15.141:0/3377523387 >> v2:172.21.15.188:3300/0 conn(0x7fde1c15bf10 0x7fde1c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 --2- 172.21.15.141:0/3377523387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0adf20 0x7fde1c070a30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 -- 172.21.15.141:0/3377523387 >> 172.21.15.141:0/3377523387 conn(0x7fde1c06a7d0 msgr2=0x7fde1c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:27.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 -- 172.21.15.141:0/3377523387 shutdown_connections 2024-01-16T14:58:27.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.694+0000 7fde1be00640 1 -- 172.21.15.141:0/3377523387 wait complete. 2024-01-16T14:58:27.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.696+0000 7fde1be00640 1 Processor -- start 2024-01-16T14:58:27.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.696+0000 7fde1be00640 1 -- start start 2024-01-16T14:58:27.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.696+0000 7fde1be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde1c0adf20 0x7fde1c0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:27.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0cc980 0x7fde1c13ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:27.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c15bf10 0x7fde1c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:27.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1be00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c13a100 con 0x7fde1c0cc980 2024-01-16T14:58:27.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1be00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0b3b70 con 0x7fde1c0adf20 2024-01-16T14:58:27.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1be00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0b2a40 con 0x7fde1c15bf10 2024-01-16T14:58:27.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0cc980 0x7fde1c13ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0cc980 0x7fde1c13ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53176/0 (socket says 172.21.15.141:53176) 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 -- 172.21.15.141:0/3792794663 learned_addr learned my addr 172.21.15.141:0/3792794663 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1aa00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde1c0adf20 0x7fde1c0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1b400640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c15bf10 0x7fde1c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 -- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c15bf10 msgr2=0x7fde1c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c15bf10 0x7fde1c0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 -- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde1c0adf20 msgr2=0x7fde1c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde1c0adf20 0x7fde1c0b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1a000640 1 -- 172.21.15.141:0/3792794663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde140094d0 con 0x7fde1c0cc980 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.697+0000 7fde1aa00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde1c0adf20 0x7fde1c0b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.698+0000 7fde1a000640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0cc980 0x7fde1c13ff40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fde0400cf60 tx=0x7fde0400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.698+0000 7fde13400640 1 -- 172.21.15.141:0/3792794663 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400d840 con 0x7fde1c0cc980 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.698+0000 7fde13400640 1 -- 172.21.15.141:0/3792794663 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0400d9a0 con 0x7fde1c0cc980 2024-01-16T14:58:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.698+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c0b2d20 con 0x7fde1c0cc980 2024-01-16T14:58:27.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.698+0000 7fde13400640 1 -- 172.21.15.141:0/3792794663 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400dd30 con 0x7fde1c0cc980 2024-01-16T14:58:27.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.698+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c0b32e0 con 0x7fde1c0cc980 2024-01-16T14:58:27.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.699+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c13a270 con 0x7fde1c0cc980 2024-01-16T14:58:27.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.700+0000 7fde13400640 1 -- 172.21.15.141:0/3792794663 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fde04015440 con 0x7fde1c0cc980 2024-01-16T14:58:27.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.701+0000 7fde13400640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fddf406b2c0 0x7fddf406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:27.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.701+0000 7fde13400640 1 -- 172.21.15.141:0/3792794663 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde0408a630 con 0x7fde1c0cc980 2024-01-16T14:58:27.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.702+0000 7fde1aa00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fddf406b2c0 0x7fddf406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:27.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.703+0000 7fde1aa00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fddf406b2c0 0x7fddf406d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fde14036000 tx=0x7fde1403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:27.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.708+0000 7fde13400640 1 -- 172.21.15.141:0/3792794663 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde040554b0 con 0x7fde1c0cc980 2024-01-16T14:58:27.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:27.937+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fde1c067450 con 0x7fde1c0cc980 2024-01-16T14:58:28.172 INFO:tasks.ceph.mds.a.smithi141.stderr:2024-01-16T14:58:28.190+0000 7f8a59400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:58:28.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.468+0000 7fde13400640 1 -- 172.21.15.141:0/3792794663 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fde04058ad0 con 0x7fde1c0cc980 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fddf406b2c0 msgr2=0x7fddf406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fddf406b2c0 0x7fddf406d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fde14036000 tx=0x7fde1403d040 comp rx=0 tx=0).stop 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0cc980 msgr2=0x7fde1c13ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0cc980 0x7fde1c13ff40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fde0400cf60 tx=0x7fde0400cf90 comp rx=0 tx=0).stop 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 shutdown_connections 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fddf406b2c0 0x7fddf406d780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde1c15bf10 0x7fde1c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde1c0cc980 0x7fde1c13ff40 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 --2- 172.21.15.141:0/3792794663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde1c0adf20 0x7fde1c0b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.471+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 >> 172.21.15.141:0/3792794663 conn(0x7fde1c06a7d0 msgr2=0x7fde1c06db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.472+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 shutdown_connections 2024-01-16T14:58:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.472+0000 7fde1be00640 1 -- 172.21.15.141:0/3792794663 wait complete. 2024-01-16T14:58:28.472 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T14:58:28.472 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:28.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.694+0000 7f37ada00640 1 Processor -- start 2024-01-16T14:58:28.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.695+0000 7f37ada00640 1 -- start start 2024-01-16T14:58:28.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.696+0000 7f37ada00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f37a80b3ae0 0x7f37a815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:28.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.696+0000 7f37ada00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80cf290 0x7f37a80cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:28.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.696+0000 7f37ada00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a8070c40 con 0x7f37a80ae5a0 2024-01-16T14:58:28.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.696+0000 7f37ada00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a8071450 con 0x7f37a80b3ae0 2024-01-16T14:58:28.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.696+0000 7f37ada00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a80cfc60 con 0x7f37a80cf290 2024-01-16T14:58:28.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.697+0000 7f37a7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80cf290 0x7f37a80cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:28.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.697+0000 7f37a7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80cf290 0x7f37a80cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:49366/0 (socket says 172.21.15.141:49366) 2024-01-16T14:58:28.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.697+0000 7f37a7400640 1 -- 172.21.15.141:0/60054949 learned_addr learned my addr 172.21.15.141:0/60054949 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:28.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.697+0000 7f37a6a00640 1 --1- 172.21.15.141:0/60054949 >> v1:172.21.15.188:6789/0 conn(0x7f37a80ae5a0 0x7f37a806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:59720/0 (socket says 172.21.15.141:59720) 2024-01-16T14:58:28.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.697+0000 7f379de00640 1 --2- 172.21.15.141:0/60054949 >> v2:172.21.15.188:3300/0 conn(0x7f37a80b3ae0 0x7f37a815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:28.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.698+0000 7f37a6000640 1 -- 172.21.15.141:0/60054949 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567318711 0 0) 0x7f37a8070c40 con 0x7f37a80ae5a0 2024-01-16T14:58:28.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.698+0000 7f37a6000640 1 -- 172.21.15.141:0/60054949 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3788003520 con 0x7f37a80ae5a0 2024-01-16T14:58:28.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.698+0000 7f379de00640 1 -- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80cf290 msgr2=0x7f37a80cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:28.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.698+0000 7f379de00640 1 --2- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80cf290 0x7f37a80cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.698+0000 7f379de00640 1 -- 172.21.15.141:0/60054949 >> v1:172.21.15.188:6789/0 conn(0x7f37a80ae5a0 legacy=0x7f37a806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.698+0000 7f379de00640 1 -- 172.21.15.141:0/60054949 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a806c470 con 0x7f37a80b3ae0 2024-01-16T14:58:28.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.698+0000 7f37a7400640 1 --2- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80cf290 0x7f37a80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:28.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.699+0000 7f379de00640 1 --2- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80b3ae0 0x7f37a815e430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f37900097a0 tx=0x7f379002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3f05d01e34e4a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:28.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.700+0000 7f37a6000640 1 -- 172.21.15.141:0/60054949 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790030800 con 0x7f37a80b3ae0 2024-01-16T14:58:28.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.700+0000 7f37a6000640 1 -- 172.21.15.141:0/60054949 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790030960 con 0x7f37a80b3ae0 2024-01-16T14:58:28.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.700+0000 7f37a6000640 1 -- 172.21.15.141:0/60054949 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790030cf0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.700+0000 7f37ada00640 1 -- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80b3ae0 msgr2=0x7f37a815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.700+0000 7f37ada00640 1 --2- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80b3ae0 0x7f37a815e430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f37900097a0 tx=0x7f379002b940 comp rx=0 tx=0).stop 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.701+0000 7f37ada00640 1 -- 172.21.15.141:0/60054949 shutdown_connections 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.701+0000 7f37ada00640 1 --2- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80b3ae0 0x7f37a815e430 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.701+0000 7f37ada00640 1 --2- 172.21.15.141:0/60054949 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80cf290 0x7f37a80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.701+0000 7f37ada00640 1 -- 172.21.15.141:0/60054949 >> 172.21.15.141:0/60054949 conn(0x7f37a806a800 msgr2=0x7f37a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.701+0000 7f37ada00640 1 -- 172.21.15.141:0/60054949 shutdown_connections 2024-01-16T14:58:28.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.701+0000 7f37ada00640 1 -- 172.21.15.141:0/60054949 wait complete. 2024-01-16T14:58:28.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.702+0000 7f37ada00640 1 Processor -- start 2024-01-16T14:58:28.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.703+0000 7f37ada00640 1 -- start start 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.703+0000 7f37ada00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80ae5a0 0x7f37a81600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f37ada00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37a80b3ae0 0x7f37a813e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f37ada00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80cf290 0x7f37a8141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f37ada00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a81448d0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f37ada00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a813e730 con 0x7f37a80cf290 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f37ada00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815f010 con 0x7f37a80ae5a0 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37a80b3ae0 0x7f37a813e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37a80b3ae0 0x7f37a813e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53182/0 (socket says 172.21.15.141:53182) 2024-01-16T14:58:28.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 -- 172.21.15.141:0/2027195231 learned_addr learned my addr 172.21.15.141:0/2027195231 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:28.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f37a6a00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80ae5a0 0x7f37a81600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:28.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 -- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80ae5a0 msgr2=0x7f37a81600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80ae5a0 0x7f37a81600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 -- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80cf290 msgr2=0x7f37a8141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80cf290 0x7f37a8141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f379de00640 1 -- 172.21.15.141:0/2027195231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a0009b30 con 0x7f37a80b3ae0 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.704+0000 7f37a6a00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80ae5a0 0x7f37a81600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.705+0000 7f379de00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37a80b3ae0 0x7f37a813e1f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f379002bf50 tx=0x7f3790004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.705+0000 7f379f400640 1 -- 172.21.15.141:0/2027195231 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379003e040 con 0x7f37a80b3ae0 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.705+0000 7f379f400640 1 -- 172.21.15.141:0/2027195231 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790039bb0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.705+0000 7f379f400640 1 -- 172.21.15.141:0/2027195231 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37900443f0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.705+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3790009400 con 0x7f37a80b3ae0 2024-01-16T14:58:28.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.705+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a815f5d0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.706+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a80675d0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.707+0000 7f379f400640 1 -- 172.21.15.141:0/2027195231 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3790043390 con 0x7f37a80b3ae0 2024-01-16T14:58:28.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.708+0000 7f379f400640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f378006b2c0 0x7f378006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:28.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.708+0000 7f379f400640 1 -- 172.21.15.141:0/2027195231 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37900abef0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.709+0000 7f37a6a00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f378006b2c0 0x7f378006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:28.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.710+0000 7f37a6a00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f378006b2c0 0x7f378006d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3798000f60 tx=0x7f3798009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:28.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.713+0000 7f379f400640 1 -- 172.21.15.141:0/2027195231 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3790076cc0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.946+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37a80cb620 con 0x7f37a80b3ae0 2024-01-16T14:58:28.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.947+0000 7f379f400640 1 -- 172.21.15.141:0/2027195231 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f379007a2e0 con 0x7f37a80b3ae0 2024-01-16T14:58:28.928 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:28.929 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/2661986774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":2661986774},{"type":"v1","addr":"172.21.15.188:6836","nonce":2661986774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4586,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2344909439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2344909439},{"type":"v1","addr":"172.21.15.188:6837","nonce":2344909439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/1479821799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":1479821799},{"type":"v1","addr":"172.21.15.141:6834","nonce":1479821799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:58:24.852726+0000","modified":"2024-01-16T14:58:27.938767+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4762":{"gid":4762,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.141:6835/176725955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":176725955},{"type":"v1","addr":"172.21.15.141:6835","nonce":176725955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T14:58:28.929 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 17 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.950+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f378006b2c0 msgr2=0x7f378006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.950+0000 7f37ada00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f378006b2c0 0x7f378006d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3798000f60 tx=0x7f3798009040 comp rx=0 tx=0).stop 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.950+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37a80b3ae0 msgr2=0x7f37a813e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.950+0000 7f37ada00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37a80b3ae0 0x7f37a813e1f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f379002bf50 tx=0x7f3790004730 comp rx=0 tx=0).stop 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 shutdown_connections 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f378006b2c0 0x7f378006d780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37a80cf290 0x7f37a8141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37a80b3ae0 0x7f37a813e1f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 --2- 172.21.15.141:0/2027195231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37a80ae5a0 0x7f37a81600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:28.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 >> 172.21.15.141:0/2027195231 conn(0x7f37a806a800 msgr2=0x7f37a815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:28.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 shutdown_connections 2024-01-16T14:58:28.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:28.951+0000 7f37ada00640 1 -- 172.21.15.141:0/2027195231 wait complete. 2024-01-16T14:58:28.956 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:58:29.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.184+0000 7f18e6800640 1 Processor -- start 2024-01-16T14:58:29.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.184+0000 7f18e6800640 1 -- start start 2024-01-16T14:58:29.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18e6800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f18e013df80 0x7f18e015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18e6800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e013e360 0x7f18e00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18e6800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18e015f9c0 0x7f18e00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18e6800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e006c470 con 0x7f18e015f9c0 2024-01-16T14:58:29.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18e6800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e006c750 con 0x7f18e013df80 2024-01-16T14:58:29.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18e6800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e006ca30 con 0x7f18e013e360 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18dfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18e015f9c0 0x7f18e00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18dfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18e015f9c0 0x7f18e00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53186/0 (socket says 172.21.15.141:53186) 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18dfe00640 1 -- 172.21.15.141:0/3783685649 learned_addr learned my addr 172.21.15.141:0/3783685649 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18df400640 1 --2- 172.21.15.141:0/3783685649 >> v2:172.21.15.188:3300/0 conn(0x7f18e013df80 0x7f18e015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.186+0000 7f18d7e00640 1 --2- 172.21.15.141:0/3783685649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e013e360 0x7f18e00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.187+0000 7f18dfe00640 1 -- 172.21.15.141:0/3783685649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e013e360 msgr2=0x7f18e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.187+0000 7f18dfe00640 1 --2- 172.21.15.141:0/3783685649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e013e360 0x7f18e00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.187+0000 7f18dfe00640 1 -- 172.21.15.141:0/3783685649 >> v2:172.21.15.188:3300/0 conn(0x7f18e013df80 msgr2=0x7f18e015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:29.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.187+0000 7f18dfe00640 1 --2- 172.21.15.141:0/3783685649 >> v2:172.21.15.188:3300/0 conn(0x7f18e013df80 0x7f18e015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.187+0000 7f18dfe00640 1 -- 172.21.15.141:0/3783685649 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e006cd10 con 0x7f18e015f9c0 2024-01-16T14:58:29.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.188+0000 7f18dfe00640 1 --2- 172.21.15.141:0/3783685649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e015f9c0 0x7f18e00cf7d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f18e0072770 tx=0x7f18d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684f48dab93491e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:29.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18dea00640 1 -- 172.21.15.141:0/3783685649 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d802ca70 con 0x7f18e015f9c0 2024-01-16T14:58:29.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18dea00640 1 -- 172.21.15.141:0/3783685649 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d802cbd0 con 0x7f18e015f9c0 2024-01-16T14:58:29.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18dea00640 1 -- 172.21.15.141:0/3783685649 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d80343f0 con 0x7f18e015f9c0 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 -- 172.21.15.141:0/3783685649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e015f9c0 msgr2=0x7f18e00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 --2- 172.21.15.141:0/3783685649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e015f9c0 0x7f18e00cf7d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f18e0072770 tx=0x7f18d802bed0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 -- 172.21.15.141:0/3783685649 shutdown_connections 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 --2- 172.21.15.141:0/3783685649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e015f9c0 0x7f18e00cf7d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 --2- 172.21.15.141:0/3783685649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e013e360 0x7f18e00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 --2- 172.21.15.141:0/3783685649 >> v2:172.21.15.188:3300/0 conn(0x7f18e013df80 0x7f18e015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 -- 172.21.15.141:0/3783685649 >> 172.21.15.141:0/3783685649 conn(0x7f18e006a800 msgr2=0x7f18e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 -- 172.21.15.141:0/3783685649 shutdown_connections 2024-01-16T14:58:29.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.189+0000 7f18e6800640 1 -- 172.21.15.141:0/3783685649 wait complete. 2024-01-16T14:58:29.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.191+0000 7f18e6800640 1 Processor -- start 2024-01-16T14:58:29.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.191+0000 7f18e6800640 1 -- start start 2024-01-16T14:58:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.192+0000 7f18e6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e013df80 0x7f18e013d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.192+0000 7f18e6800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e013e360 0x7f18e00a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.192+0000 7f18e6800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e015f9c0 0x7f18e015dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.192+0000 7f18e6800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e01f2a90 con 0x7f18e013df80 2024-01-16T14:58:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.192+0000 7f18e6800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00aabd0 con 0x7f18e013e360 2024-01-16T14:58:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.192+0000 7f18e6800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e01f2c80 con 0x7f18e015f9c0 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e013df80 0x7f18e013d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e013df80 0x7f18e013d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53192/0 (socket says 172.21.15.141:53192) 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 -- 172.21.15.141:0/1734486577 learned_addr learned my addr 172.21.15.141:0/1734486577 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18d7e00640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e013e360 0x7f18e00a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18dfe00640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e015f9c0 0x7f18e015dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 -- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e015f9c0 msgr2=0x7f18e015dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e015f9c0 0x7f18e015dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 -- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e013e360 msgr2=0x7f18e00a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e013e360 0x7f18e00a9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.193+0000 7f18df400640 1 -- 172.21.15.141:0/1734486577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8009b30 con 0x7f18e013df80 2024-01-16T14:58:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.194+0000 7f18df400640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e013df80 0x7f18e013d1e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f18d000d5f0 tx=0x7f18d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.194+0000 7f18d7400640 1 -- 172.21.15.141:0/1734486577 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d00125f0 con 0x7f18e013df80 2024-01-16T14:58:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.194+0000 7f18d7400640 1 -- 172.21.15.141:0/1734486577 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d0012750 con 0x7f18e013df80 2024-01-16T14:58:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.194+0000 7f18d7400640 1 -- 172.21.15.141:0/1734486577 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0012b00 con 0x7f18e013df80 2024-01-16T14:58:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.194+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e00aa8f0 con 0x7f18e013df80 2024-01-16T14:58:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.194+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e01f3220 con 0x7f18e013df80 2024-01-16T14:58:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.195+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e01f33e0 con 0x7f18e013df80 2024-01-16T14:58:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.196+0000 7f18d7400640 1 -- 172.21.15.141:0/1734486577 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f18d0012c60 con 0x7f18e013df80 2024-01-16T14:58:29.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.197+0000 7f18d7400640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18b806b2c0 0x7f18b806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.197+0000 7f18d7400640 1 -- 172.21.15.141:0/1734486577 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18d008a010 con 0x7f18e013df80 2024-01-16T14:58:29.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.198+0000 7f18d7e00640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18b806b2c0 0x7f18b806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.199+0000 7f18d7e00640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18b806b2c0 0x7f18b806d780 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f18e015b5b0 tx=0x7f18c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:29.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.202+0000 7f18d7400640 1 -- 172.21.15.141:0/1734486577 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18d0054f10 con 0x7f18e013df80 2024-01-16T14:58:29.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.392+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18e00675d0 con 0x7f18e013df80 2024-01-16T14:58:29.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.394+0000 7f18d7400640 1 -- 172.21.15.141:0/1734486577 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f18d0058530 con 0x7f18e013df80 2024-01-16T14:58:29.376 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:29.376 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":35,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:58:26.430609+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:58:22.670734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T14:58:23.861781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18b806b2c0 msgr2=0x7f18b806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18b806b2c0 0x7f18b806d780 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f18e015b5b0 tx=0x7f18c8008040 comp rx=0 tx=0).stop 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e013df80 msgr2=0x7f18e013d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e013df80 0x7f18e013d1e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f18d000d5f0 tx=0x7f18d000dab0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 shutdown_connections 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18b806b2c0 0x7f18b806d780 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e015f9c0 0x7f18e015dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e013e360 0x7f18e00a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 --2- 172.21.15.141:0/1734486577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e013df80 0x7f18e013d1e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.397+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 >> 172.21.15.141:0/1734486577 conn(0x7f18e006a800 msgr2=0x7f18e006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.398+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 shutdown_connections 2024-01-16T14:58:29.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.398+0000 7f18e6800640 1 -- 172.21.15.141:0/1734486577 wait complete. 2024-01-16T14:58:29.397 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T14:58:29.397 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:58:29.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.612+0000 7fbf7a000640 1 Processor -- start 2024-01-16T14:58:29.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.612+0000 7fbf7a000640 1 -- start start 2024-01-16T14:58:29.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.614+0000 7fbf7a000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf740cf9a0 0x7fbf74070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.614+0000 7fbf7a000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74071690 con 0x7fbf7414b2c0 2024-01-16T14:58:29.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.614+0000 7fbf7a000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74071970 con 0x7fbf74070ea0 2024-01-16T14:58:29.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.614+0000 7fbf7a000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7414b6a0 con 0x7fbf740cf9a0 2024-01-16T14:58:29.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.614+0000 7fbf73400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbf7414b2c0 0x7fbf740cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42796/0 (socket says 172.21.15.141:42796) 2024-01-16T14:58:29.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.614+0000 7fbf73400640 1 -- 172.21.15.141:0/3205032275 learned_addr learned my addr 172.21.15.141:0/3205032275 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:29.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf6be00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf740cf9a0 0x7fbf74070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465040408 0 0) 0x7fbf74071690 con 0x7fbf7414b2c0 2024-01-16T14:58:29.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54003520 con 0x7fbf7414b2c0 2024-01-16T14:58:29.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928005939 0 0) 0x7fbf54003520 con 0x7fbf7414b2c0 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf740cf9a0 msgr2=0x7fbf74070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf740cf9a0 0x7fbf74070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 >> v1:172.21.15.188:6789/0 conn(0x7fbf74070ea0 legacy=0x7fbf740cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf74071c50 con 0x7fbf7414b2c0 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fbf64001670 con 0x7fbf7414b2c0 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 >> v1:172.21.15.141:6789/0 conn(0x7fbf7414b2c0 legacy=0x7fbf740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf740cf9a0 0x7fbf540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54003f70 0x7fbf540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54004d00 0x7fbf54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7414b6a0 con 0x7fbf54003f70 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5400bac0 con 0x7fbf54004d00 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.615+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5400bc80 con 0x7fbf740cf9a0 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.616+0000 7fbf6be00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54003f70 0x7fbf540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.616+0000 7fbf6be00640 1 -- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf740cf9a0 msgr2=0x7fbf540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.616+0000 7fbf6be00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf740cf9a0 0x7fbf540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.616+0000 7fbf6be00640 1 -- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54004d00 msgr2=0x7fbf54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:29.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.616+0000 7fbf73e00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54004d00 0x7fbf54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.616+0000 7fbf6be00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54004d00 0x7fbf54005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.616+0000 7fbf6be00640 1 -- 172.21.15.141:0/3205032275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54003520 con 0x7fbf54003f70 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.617+0000 7fbf6be00640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54003f70 0x7fbf540047c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009ac0 tx=0x7fbf540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1cc7bd7087e9b4f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.617+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c02c630 con 0x7fbf54003f70 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.617+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c02c790 con 0x7fbf54003f70 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.617+0000 7fbf72a00640 1 -- 172.21.15.141:0/3205032275 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c02cb20 con 0x7fbf54003f70 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.618+0000 7fbf7a000640 1 -- 172.21.15.141:0/3205032275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54003f70 msgr2=0x7fbf540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.618+0000 7fbf7a000640 1 --2- 172.21.15.141:0/3205032275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54003f70 0x7fbf540047c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009ac0 tx=0x7fbf540057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.618+0000 7fbf73e00640 1 -- 172.21.15.141:0/3205032275 reap_dead start 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.618+0000 7fbf7a000640 1 -- 172.21.15.141:0/3205032275 shutdown_connections 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.618+0000 7fbf7a000640 1 -- 172.21.15.141:0/3205032275 >> 172.21.15.141:0/3205032275 conn(0x7fbf7406a800 msgr2=0x7fbf7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:29.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.618+0000 7fbf7a000640 1 -- 172.21.15.141:0/3205032275 shutdown_connections 2024-01-16T14:58:29.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.618+0000 7fbf7a000640 1 -- 172.21.15.141:0/3205032275 wait complete. 2024-01-16T14:58:29.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.620+0000 7fbf7a000640 1 Processor -- start 2024-01-16T14:58:29.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.620+0000 7fbf7a000640 1 -- start start 2024-01-16T14:58:29.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf7a000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54003f70 0x7fbf74163990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf7a000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf74148090 0x7fbf74148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf7a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf741f72e0 0x7fbf741f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf7a000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74148a50 con 0x7fbf741f72e0 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf7a000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74161d60 con 0x7fbf54003f70 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf7a000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74162020 con 0x7fbf74148090 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf73e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf741f72e0 0x7fbf741f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf73e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf741f72e0 0x7fbf741f7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53214/0 (socket says 172.21.15.141:53214) 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.621+0000 7fbf73e00640 1 -- 172.21.15.141:0/3664577623 learned_addr learned my addr 172.21.15.141:0/3664577623 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf73400640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54003f70 0x7fbf74163990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf6be00640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf74148090 0x7fbf74148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf73e00640 1 -- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf74148090 msgr2=0x7fbf74148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf73e00640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf74148090 0x7fbf74148510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf73e00640 1 -- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54003f70 msgr2=0x7fbf74163990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf73e00640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54003f70 0x7fbf74163990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf73e00640 1 -- 172.21.15.141:0/3664577623 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf74162300 con 0x7fbf741f72e0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf6be00640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf74148090 0x7fbf74148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf73e00640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf741f72e0 0x7fbf741f7770 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbf740cfe50 tx=0x7fbf6c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf6b400640 1 -- 172.21.15.141:0/3664577623 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003b70 con 0x7fbf741f72e0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf6b400640 1 -- 172.21.15.141:0/3664577623 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c003cd0 con 0x7fbf741f72e0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf741625e0 con 0x7fbf741f72e0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.622+0000 7fbf6b400640 1 -- 172.21.15.141:0/3664577623 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0052b0 con 0x7fbf741f72e0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.623+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf74162de0 con 0x7fbf741f72e0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.624+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf74162b00 con 0x7fbf741f72e0 2024-01-16T14:58:29.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.624+0000 7fbf6b400640 1 -- 172.21.15.141:0/3664577623 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbf6c005410 con 0x7fbf741f72e0 2024-01-16T14:58:29.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.625+0000 7fbf6b400640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbf4c06b2c0 0x7fbf4c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:29.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.625+0000 7fbf6b400640 1 -- 172.21.15.141:0/3664577623 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf6c0896d0 con 0x7fbf741f72e0 2024-01-16T14:58:29.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.626+0000 7fbf73400640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbf4c06b2c0 0x7fbf4c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:29.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.627+0000 7fbf73400640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbf4c06b2c0 0x7fbf4c06d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf740b2650 tx=0x7fbf640016a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:29.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.630+0000 7fbf6b400640 1 -- 172.21.15.141:0/3664577623 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf74162b00 con 0x7fbf741f72e0 2024-01-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.801+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf74162190 con 0x7fbf4c06b2c0 2024-01-16T14:58:29.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.829+0000 7fbf6b400640 1 -- 172.21.15.141:0/3664577623 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281181 (secure 0 0 0) 0x7fbf74162190 con 0x7fbf4c06b2c0 2024-01-16T14:58:29.814 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:29.815 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbf4c06b2c0 msgr2=0x7fbf4c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbf4c06b2c0 0x7fbf4c06d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf740b2650 tx=0x7fbf640016a0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf741f72e0 msgr2=0x7fbf741f7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf741f72e0 0x7fbf741f7770 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbf740cfe50 tx=0x7fbf6c00dc00 comp rx=0 tx=0).stop 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 shutdown_connections 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbf4c06b2c0 0x7fbf4c06d780 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf741f72e0 0x7fbf741f7770 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf74148090 0x7fbf74148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 --2- 172.21.15.141:0/3664577623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf54003f70 0x7fbf74163990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:29.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 >> 172.21.15.141:0/3664577623 conn(0x7fbf7406a800 msgr2=0x7fbf740aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:29.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 shutdown_connections 2024-01-16T14:58:29.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:29.835+0000 7fbf7a000640 1 -- 172.21.15.141:0/3664577623 wait complete. 2024-01-16T14:58:29.835 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-01-16T14:58:28.463542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220104,"kb_used_data":3944,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754616,"statfs":{"total":773094113280,"available":772868726784,"internally_reserved":0,"allocated":4038656,"data_stored":2261536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.077488"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409591+0000","last_change":"2024-01-16T14:58:24.409591+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409591+0000","last_undegraded":"2024-01-16T14:58:24.409591+0000","last_fullsized":"2024-01-16T14:58:24.409591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.406123+0000","last_change":"2024-01-16T14:58:24.406123+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.406123+0000","last_undegraded":"2024-01-16T14:58:24.406123+0000","last_fullsized":"2024-01-16T14:58:24.406123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403652+0000","last_change":"2024-01-16T14:58:24.403652+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403652+0000","last_undegraded":"2024-01-16T14:58:24.403652+0000","last_fullsized":"2024-01-16T14:58:24.403652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414133+0000","last_change":"2024-01-16T14:58:24.414133+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414133+0000","last_undegraded":"2024-01-16T14:58:24.414133+0000","last_fullsized":"2024-01-16T14:58:24.414133+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.410962+0000","last_change":"2024-01-16T14:58:24.410962+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.410962+0000","last_undegraded":"2024-01-16T14:58:24.410962+0000","last_fullsized":"2024-01-16T14:58:24.410962+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405581+0000","last_change":"2024-01-16T14:58:24.405581+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405581+0000","last_undegraded":"2024-01-16T14:58:24.405581+0000","last_fullsized":"2024-01-16T14:58:24.405581+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.415348+0000","last_change":"2024-01-16T14:58:24.415348+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.415348+0000","last_undegraded":"2024-01-16T14:58:24.415348+0000","last_fullsized":"2024-01-16T14:58:24.415348+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.424050+0000","last_change":"2024-01-16T14:58:24.424050+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.424050+0000","last_undegraded":"2024-01-16T14:58:24.424050+0000","last_fullsized":"2024-01-16T14:58:24.424050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409209+0000","last_change":"2024-01-16T14:58:24.409209+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409209+0000","last_undegraded":"2024-01-16T14:58:24.409209+0000","last_fullsized":"2024-01-16T14:58:24.409209+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414124+0000","last_change":"2024-01-16T14:58:24.414124+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414124+0000","last_undegraded":"2024-01-16T14:58:24.414124+0000","last_fullsized":"2024-01-16T14:58:24.414124+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.417717+0000","last_change":"2024-01-16T14:58:24.417717+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.417717+0000","last_undegraded":"2024-01-16T14:58:24.417717+0000","last_fullsized":"2024-01-16T14:58:24.417717+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.408501+0000","last_change":"2024-01-16T14:58:24.408501+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.408501+0000","last_undegraded":"2024-01-16T14:58:24.408501+0000","last_fullsized":"2024-01-16T14:58:24.408501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.410510+0000","last_change":"2024-01-16T14:58:24.410510+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.410510+0000","last_undegraded":"2024-01-16T14:58:24.410510+0000","last_fullsized":"2024-01-16T14:58:24.410510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404869+0000","last_change":"2024-01-16T14:58:24.404869+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404869+0000","last_undegraded":"2024-01-16T14:58:24.404869+0000","last_fullsized":"2024-01-16T14:58:24.404869+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401335+0000","last_change":"2024-01-16T14:58:24.401335+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401335+0000","last_undegraded":"2024-01-16T14:58:24.401335+0000","last_fullsized":"2024-01-16T14:58:24.401335+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.413375+0000","last_change":"2024-01-16T14:58:24.413375+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.413375+0000","last_undegraded":"2024-01-16T14:58:24.413375+0000","last_fullsized":"2024-01-16T14:58:24.413375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.413923+0000","last_change":"2024-01-16T14:58:24.413923+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.413923+0000","last_undegraded":"2024-01-16T14:58:24.413923+0000","last_fullsized":"2024-01-16T14:58:24.413923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404785+0000","last_change":"2024-01-16T14:58:24.404785+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404785+0000","last_undegraded":"2024-01-16T14:58:24.404785+0000","last_fullsized":"2024-01-16T14:58:24.404785+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404064+0000","last_change":"2024-01-16T14:58:24.404064+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404064+0000","last_undegraded":"2024-01-16T14:58:24.404064+0000","last_fullsized":"2024-01-16T14:58:24.404064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.407187+0000","last_change":"2024-01-16T14:58:24.407187+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.407187+0000","last_undegraded":"2024-01-16T14:58:24.407187+0000","last_fullsized":"2024-01-16T14:58:24.407187+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.413325+0000","last_change":"2024-01-16T14:58:24.413325+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.413325+0000","last_undegraded":"2024-01-16T14:58:24.413325+0000","last_fullsized":"2024-01-16T14:58:24.413325+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.407443+0000","last_change":"2024-01-16T14:58:24.407443+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.407443+0000","last_undegraded":"2024-01-16T14:58:24.407443+0000","last_fullsized":"2024-01-16T14:58:24.407443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403420+0000","last_change":"2024-01-16T14:58:24.403420+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403420+0000","last_undegraded":"2024-01-16T14:58:24.403420+0000","last_fullsized":"2024-01-16T14:58:24.403420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.399749+0000","last_change":"2024-01-16T14:58:24.399749+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.399749+0000","last_undegraded":"2024-01-16T14:58:24.399749+0000","last_fullsized":"2024-01-16T14:58:24.399749+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414535+0000","last_change":"2024-01-16T14:58:24.414535+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414535+0000","last_undegraded":"2024-01-16T14:58:24.414535+0000","last_fullsized":"2024-01-16T14:58:24.414535+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404079+0000","last_change":"2024-01-16T14:58:24.404079+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404079+0000","last_undegraded":"2024-01-16T14:58:24.404079+0000","last_fullsized":"2024-01-16T14:58:24.404079+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409991+0000","last_change":"2024-01-16T14:58:24.409991+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409991+0000","last_undegraded":"2024-01-16T14:58:24.409991+0000","last_fullsized":"2024-01-16T14:58:24.409991+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401405+0000","last_change":"2024-01-16T14:58:24.401405+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401405+0000","last_undegraded":"2024-01-16T14:58:24.401405+0000","last_fullsized":"2024-01-16T14:58:24.401405+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405298+0000","last_change":"2024-01-16T14:58:24.405298+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405298+0000","last_undegraded":"2024-01-16T14:58:24.405298+0000","last_fullsized":"2024-01-16T14:58:24.405298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.423789+0000","last_change":"2024-01-16T14:58:24.423789+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.423789+0000","last_undegraded":"2024-01-16T14:58:24.423789+0000","last_fullsized":"2024-01-16T14:58:24.423789+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.411277+0000","last_change":"2024-01-16T14:58:24.411277+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.411277+0000","last_undegraded":"2024-01-16T14:58:24.411277+0000","last_fullsized":"2024-01-16T14:58:24.411277+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.423000+0000","last_change":"2024-01-16T14:58:24.423000+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.423000+0000","last_undegraded":"2024-01-16T14:58:24.423000+0000","last_fullsized":"2024-01-16T14:58:24.423000+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.402573+0000","last_change":"2024-01-16T14:58:24.402573+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.402573+0000","last_undegraded":"2024-01-16T14:58:24.402573+0000","last_fullsized":"2024-01-16T14:58:24.402573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431192+0000","last_change":"2024-01-16T14:58:24.428745+0000","last_active":"2024-01-16T14:58:24.431192+0000","last_peered":"2024-01-16T14:58:24.431192+0000","last_clean":"2024-01-16T14:58:24.431192+0000","last_became_active":"2024-01-16T14:58:24.427170+0000","last_became_peered":"2024-01-16T14:58:24.427170+0000","last_unstale":"2024-01-16T14:58:24.431192+0000","last_undegraded":"2024-01-16T14:58:24.431192+0000","last_fullsized":"2024-01-16T14:58:24.431192+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:59.581331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.408928+0000","last_change":"2024-01-16T14:58:24.408928+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.408928+0000","last_undegraded":"2024-01-16T14:58:24.408928+0000","last_fullsized":"2024-01-16T14:58:24.408928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431329+0000","last_change":"2024-01-16T14:58:24.429636+0000","last_active":"2024-01-16T14:58:24.431329+0000","last_peered":"2024-01-16T14:58:24.431329+0000","last_clean":"2024-01-16T14:58:24.431329+0000","last_became_active":"2024-01-16T14:58:24.428619+0000","last_became_peered":"2024-01-16T14:58:24.428619+0000","last_unstale":"2024-01-16T14:58:24.431329+0000","last_undegraded":"2024-01-16T14:58:24.431329+0000","last_fullsized":"2024-01-16T14:58:24.431329+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:39.718130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.411759+0000","last_change":"2024-01-16T14:58:24.411759+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.411759+0000","last_undegraded":"2024-01-16T14:58:24.411759+0000","last_fullsized":"2024-01-16T14:58:24.411759+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.422992+0000","last_change":"2024-01-16T14:58:24.421162+0000","last_active":"2024-01-16T14:58:24.422992+0000","last_peered":"2024-01-16T14:58:24.422992+0000","last_clean":"2024-01-16T14:58:24.422992+0000","last_became_active":"2024-01-16T14:58:24.420858+0000","last_became_peered":"2024-01-16T14:58:24.420858+0000","last_unstale":"2024-01-16T14:58:24.422992+0000","last_undegraded":"2024-01-16T14:58:24.422992+0000","last_fullsized":"2024-01-16T14:58:24.422992+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:37.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409558+0000","last_change":"2024-01-16T14:58:24.409558+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409558+0000","last_undegraded":"2024-01-16T14:58:24.409558+0000","last_fullsized":"2024-01-16T14:58:24.409558+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.428852+0000","last_change":"2024-01-16T14:58:24.426642+0000","last_active":"2024-01-16T14:58:24.428852+0000","last_peered":"2024-01-16T14:58:24.428852+0000","last_clean":"2024-01-16T14:58:24.428852+0000","last_became_active":"2024-01-16T14:58:24.426379+0000","last_became_peered":"2024-01-16T14:58:24.426379+0000","last_unstale":"2024-01-16T14:58:24.428852+0000","last_undegraded":"2024-01-16T14:58:24.428852+0000","last_fullsized":"2024-01-16T14:58:24.428852+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:24.961936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.419238+0000","last_change":"2024-01-16T14:58:24.419238+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.419238+0000","last_undegraded":"2024-01-16T14:58:24.419238+0000","last_fullsized":"2024-01-16T14:58:24.419238+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.423647+0000","last_change":"2024-01-16T14:58:24.419926+0000","last_active":"2024-01-16T14:58:24.423647+0000","last_peered":"2024-01-16T14:58:24.423647+0000","last_clean":"2024-01-16T14:58:24.423647+0000","last_became_active":"2024-01-16T14:58:24.419550+0000","last_became_peered":"2024-01-16T14:58:24.419550+0000","last_unstale":"2024-01-16T14:58:24.423647+0000","last_undegraded":"2024-01-16T14:58:24.423647+0000","last_fullsized":"2024-01-16T14:58:24.423647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:25.305571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401594+0000","last_change":"2024-01-16T14:58:24.401594+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401594+0000","last_undegraded":"2024-01-16T14:58:24.401594+0000","last_fullsized":"2024-01-16T14:58:24.401594+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.422958+0000","last_change":"2024-01-16T14:58:24.420225+0000","last_active":"2024-01-16T14:58:24.422958+0000","last_peered":"2024-01-16T14:58:24.422958+0000","last_clean":"2024-01-16T14:58:24.422958+0000","last_became_active":"2024-01-16T14:58:24.419237+0000","last_became_peered":"2024-01-16T14:58:24.419237+0000","last_unstale":"2024-01-16T14:58:24.422958+0000","last_undegraded":"2024-01-16T14:58:24.422958+0000","last_fullsized":"2024-01-16T14:58:24.422958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:27.083704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.399035+0000","last_change":"2024-01-16T14:58:24.399035+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.399035+0000","last_undegraded":"2024-01-16T14:58:24.399035+0000","last_fullsized":"2024-01-16T14:58:24.399035+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.418782+0000","last_change":"2024-01-16T14:58:24.416771+0000","last_active":"2024-01-16T14:58:24.418782+0000","last_peered":"2024-01-16T14:58:24.418782+0000","last_clean":"2024-01-16T14:58:24.418782+0000","last_became_active":"2024-01-16T14:58:24.416479+0000","last_became_peered":"2024-01-16T14:58:24.416479+0000","last_unstale":"2024-01-16T14:58:24.418782+0000","last_undegraded":"2024-01-16T14:58:24.418782+0000","last_fullsized":"2024-01-16T14:58:24.418782+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:40.868803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414498+0000","last_change":"2024-01-16T14:58:24.414498+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414498+0000","last_undegraded":"2024-01-16T14:58:24.414498+0000","last_fullsized":"2024-01-16T14:58:24.414498+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.428247+0000","last_change":"2024-01-16T14:58:24.426653+0000","last_active":"2024-01-16T14:58:24.428247+0000","last_peered":"2024-01-16T14:58:24.428247+0000","last_clean":"2024-01-16T14:58:24.428247+0000","last_became_active":"2024-01-16T14:58:24.421492+0000","last_became_peered":"2024-01-16T14:58:24.421492+0000","last_unstale":"2024-01-16T14:58:24.428247+0000","last_undegraded":"2024-01-16T14:58:24.428247+0000","last_fullsized":"2024-01-16T14:58:24.428247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:02.111575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409360+0000","last_change":"2024-01-16T14:58:24.409360+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409360+0000","last_undegraded":"2024-01-16T14:58:24.409360+0000","last_fullsized":"2024-01-16T14:58:24.409360+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431265+0000","last_change":"2024-01-16T14:58:24.426750+0000","last_active":"2024-01-16T14:58:24.431265+0000","last_peered":"2024-01-16T14:58:24.431265+0000","last_clean":"2024-01-16T14:58:24.431265+0000","last_became_active":"2024-01-16T14:58:24.426534+0000","last_became_peered":"2024-01-16T14:58:24.426534+0000","last_unstale":"2024-01-16T14:58:24.431265+0000","last_undegraded":"2024-01-16T14:58:24.431265+0000","last_fullsized":"2024-01-16T14:58:24.431265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:19.956837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404018+0000","last_change":"2024-01-16T14:58:24.404018+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404018+0000","last_undegraded":"2024-01-16T14:58:24.404018+0000","last_fullsized":"2024-01-16T14:58:24.404018+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.418883+0000","last_change":"2024-01-16T14:58:24.414545+0000","last_active":"2024-01-16T14:58:24.418883+0000","last_peered":"2024-01-16T14:58:24.418883+0000","last_clean":"2024-01-16T14:58:24.418883+0000","last_became_active":"2024-01-16T14:58:24.412623+0000","last_became_peered":"2024-01-16T14:58:24.412623+0000","last_unstale":"2024-01-16T14:58:24.418883+0000","last_undegraded":"2024-01-16T14:58:24.418883+0000","last_fullsized":"2024-01-16T14:58:24.418883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:57.374401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401402+0000","last_change":"2024-01-16T14:58:24.401402+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401402+0000","last_undegraded":"2024-01-16T14:58:24.401402+0000","last_fullsized":"2024-01-16T14:58:24.401402+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431145+0000","last_change":"2024-01-16T14:58:24.428533+0000","last_active":"2024-01-16T14:58:24.431145+0000","last_peered":"2024-01-16T14:58:24.431145+0000","last_clean":"2024-01-16T14:58:24.431145+0000","last_became_active":"2024-01-16T14:58:24.428239+0000","last_became_peered":"2024-01-16T14:58:24.428239+0000","last_unstale":"2024-01-16T14:58:24.431145+0000","last_undegraded":"2024-01-16T14:58:24.431145+0000","last_fullsized":"2024-01-16T14:58:24.431145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409536+0000","last_change":"2024-01-16T14:58:24.409536+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409536+0000","last_undegraded":"2024-01-16T14:58:24.409536+0000","last_fullsized":"2024-01-16T14:58:24.409536+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430608+0000","last_change":"2024-01-16T14:58:24.426130+0000","last_active":"2024-01-16T14:58:24.430608+0000","last_peered":"2024-01-16T14:58:24.430608+0000","last_clean":"2024-01-16T14:58:24.430608+0000","last_became_active":"2024-01-16T14:58:24.425829+0000","last_became_peered":"2024-01-16T14:58:24.425829+0000","last_unstale":"2024-01-16T14:58:24.430608+0000","last_undegraded":"2024-01-16T14:58:24.430608+0000","last_fullsized":"2024-01-16T14:58:24.430608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:31.332098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414087+0000","last_change":"2024-01-16T14:58:24.414087+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414087+0000","last_undegraded":"2024-01-16T14:58:24.414087+0000","last_fullsized":"2024-01-16T14:58:24.414087+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.427923+0000","last_change":"2024-01-16T14:58:24.423758+0000","last_active":"2024-01-16T14:58:24.427923+0000","last_peered":"2024-01-16T14:58:24.427923+0000","last_clean":"2024-01-16T14:58:24.427923+0000","last_became_active":"2024-01-16T14:58:24.423458+0000","last_became_peered":"2024-01-16T14:58:24.423458+0000","last_unstale":"2024-01-16T14:58:24.427923+0000","last_undegraded":"2024-01-16T14:58:24.427923+0000","last_fullsized":"2024-01-16T14:58:24.427923+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:24.961936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401369+0000","last_change":"2024-01-16T14:58:24.401369+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401369+0000","last_undegraded":"2024-01-16T14:58:24.401369+0000","last_fullsized":"2024-01-16T14:58:24.401369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430379+0000","last_change":"2024-01-16T14:58:24.427569+0000","last_active":"2024-01-16T14:58:24.430379+0000","last_peered":"2024-01-16T14:58:24.430379+0000","last_clean":"2024-01-16T14:58:24.430379+0000","last_became_active":"2024-01-16T14:58:24.427309+0000","last_became_peered":"2024-01-16T14:58:24.427309+0000","last_unstale":"2024-01-16T14:58:24.430379+0000","last_undegraded":"2024-01-16T14:58:24.430379+0000","last_fullsized":"2024-01-16T14:58:24.430379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:24.349485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.421465+0000","last_change":"2024-01-16T14:58:24.421465+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.421465+0000","last_undegraded":"2024-01-16T14:58:24.421465+0000","last_fullsized":"2024-01-16T14:58:24.421465+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431377+0000","last_change":"2024-01-16T14:58:24.427259+0000","last_active":"2024-01-16T14:58:24.431377+0000","last_peered":"2024-01-16T14:58:24.431377+0000","last_clean":"2024-01-16T14:58:24.431377+0000","last_became_active":"2024-01-16T14:58:24.426959+0000","last_became_peered":"2024-01-16T14:58:24.426959+0000","last_unstale":"2024-01-16T14:58:24.431377+0000","last_undegraded":"2024-01-16T14:58:24.431377+0000","last_fullsized":"2024-01-16T14:58:24.431377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:43.995208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.410505+0000","last_change":"2024-01-16T14:58:24.410505+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.410505+0000","last_undegraded":"2024-01-16T14:58:24.410505+0000","last_fullsized":"2024-01-16T14:58:24.410505+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.418803+0000","last_change":"2024-01-16T14:58:24.416038+0000","last_active":"2024-01-16T14:58:24.418803+0000","last_peered":"2024-01-16T14:58:24.418803+0000","last_clean":"2024-01-16T14:58:24.418803+0000","last_became_active":"2024-01-16T14:58:24.415718+0000","last_became_peered":"2024-01-16T14:58:24.415718+0000","last_unstale":"2024-01-16T14:58:24.418803+0000","last_undegraded":"2024-01-16T14:58:24.418803+0000","last_fullsized":"2024-01-16T14:58:24.418803+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:20.089624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403800+0000","last_change":"2024-01-16T14:58:24.403800+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403800+0000","last_undegraded":"2024-01-16T14:58:24.403800+0000","last_fullsized":"2024-01-16T14:58:24.403800+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430252+0000","last_change":"2024-01-16T14:58:24.424464+0000","last_active":"2024-01-16T14:58:24.430252+0000","last_peered":"2024-01-16T14:58:24.430252+0000","last_clean":"2024-01-16T14:58:24.430252+0000","last_became_active":"2024-01-16T14:58:24.424200+0000","last_became_peered":"2024-01-16T14:58:24.424200+0000","last_unstale":"2024-01-16T14:58:24.430252+0000","last_undegraded":"2024-01-16T14:58:24.430252+0000","last_fullsized":"2024-01-16T14:58:24.430252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:39.251923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.412816+0000","last_change":"2024-01-16T14:58:24.412816+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.412816+0000","last_undegraded":"2024-01-16T14:58:24.412816+0000","last_fullsized":"2024-01-16T14:58:24.412816+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.427996+0000","last_change":"2024-01-16T14:58:24.420965+0000","last_active":"2024-01-16T14:58:24.427996+0000","last_peered":"2024-01-16T14:58:24.427996+0000","last_clean":"2024-01-16T14:58:24.427996+0000","last_became_active":"2024-01-16T14:58:24.420644+0000","last_became_peered":"2024-01-16T14:58:24.420644+0000","last_unstale":"2024-01-16T14:58:24.427996+0000","last_undegraded":"2024-01-16T14:58:24.427996+0000","last_fullsized":"2024-01-16T14:58:24.427996+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:40.305191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.408800+0000","last_change":"2024-01-16T14:58:24.408800+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.408800+0000","last_undegraded":"2024-01-16T14:58:24.408800+0000","last_fullsized":"2024-01-16T14:58:24.408800+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.428969+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:24.428969+0000","last_peered":"2024-01-16T14:58:24.428969+0000","last_clean":"2024-01-16T14:58:24.428969+0000","last_became_active":"2024-01-16T14:58:24.421626+0000","last_became_peered":"2024-01-16T14:58:24.421626+0000","last_unstale":"2024-01-16T14:58:24.428969+0000","last_undegraded":"2024-01-16T14:58:24.428969+0000","last_fullsized":"2024-01-16T14:58:24.428969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:57:30.780004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409045+0000","last_change":"2024-01-16T14:58:24.409045+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409045+0000","last_undegraded":"2024-01-16T14:58:24.409045+0000","last_fullsized":"2024-01-16T14:58:24.409045+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.419706+0000","last_change":"2024-01-16T14:58:24.413866+0000","last_active":"2024-01-16T14:58:24.419706+0000","last_peered":"2024-01-16T14:58:24.419706+0000","last_clean":"2024-01-16T14:58:24.419706+0000","last_became_active":"2024-01-16T14:58:24.413577+0000","last_became_peered":"2024-01-16T14:58:24.413577+0000","last_unstale":"2024-01-16T14:58:24.419706+0000","last_undegraded":"2024-01-16T14:58:24.419706+0000","last_fullsized":"2024-01-16T14:58:24.419706+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:43.604145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405624+0000","last_change":"2024-01-16T14:58:24.405624+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405624+0000","last_undegraded":"2024-01-16T14:58:24.405624+0000","last_fullsized":"2024-01-16T14:58:24.405624+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.421985+0000","last_change":"2024-01-16T14:58:24.417730+0000","last_active":"2024-01-16T14:58:24.421985+0000","last_peered":"2024-01-16T14:58:24.421985+0000","last_clean":"2024-01-16T14:58:24.421985+0000","last_became_active":"2024-01-16T14:58:24.417446+0000","last_became_peered":"2024-01-16T14:58:24.417446+0000","last_unstale":"2024-01-16T14:58:24.421985+0000","last_undegraded":"2024-01-16T14:58:24.421985+0000","last_fullsized":"2024-01-16T14:58:24.421985+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:09.948145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405266+0000","last_change":"2024-01-16T14:58:24.405266+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405266+0000","last_undegraded":"2024-01-16T14:58:24.405266+0000","last_fullsized":"2024-01-16T14:58:24.405266+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431017+0000","last_change":"2024-01-16T14:58:24.426266+0000","last_active":"2024-01-16T14:58:24.431017+0000","last_peered":"2024-01-16T14:58:24.431017+0000","last_clean":"2024-01-16T14:58:24.431017+0000","last_became_active":"2024-01-16T14:58:24.425944+0000","last_became_peered":"2024-01-16T14:58:24.425944+0000","last_unstale":"2024-01-16T14:58:24.431017+0000","last_undegraded":"2024-01-16T14:58:24.431017+0000","last_fullsized":"2024-01-16T14:58:24.431017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:57.374401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403031+0000","last_change":"2024-01-16T14:58:24.403031+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403031+0000","last_undegraded":"2024-01-16T14:58:24.403031+0000","last_fullsized":"2024-01-16T14:58:24.403031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.421182+0000","last_change":"2024-01-16T14:58:24.416410+0000","last_active":"2024-01-16T14:58:24.421182+0000","last_peered":"2024-01-16T14:58:24.421182+0000","last_clean":"2024-01-16T14:58:24.421182+0000","last_became_active":"2024-01-16T14:58:24.416121+0000","last_became_peered":"2024-01-16T14:58:24.416121+0000","last_unstale":"2024-01-16T14:58:24.421182+0000","last_undegraded":"2024-01-16T14:58:24.421182+0000","last_fullsized":"2024-01-16T14:58:24.421182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:45.988189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.422583+0000","last_change":"2024-01-16T14:58:24.422583+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.422583+0000","last_undegraded":"2024-01-16T14:58:24.422583+0000","last_fullsized":"2024-01-16T14:58:24.422583+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431715+0000","last_change":"2024-01-16T14:58:24.429805+0000","last_active":"2024-01-16T14:58:24.431715+0000","last_peered":"2024-01-16T14:58:24.431715+0000","last_clean":"2024-01-16T14:58:24.431715+0000","last_became_active":"2024-01-16T14:58:24.429510+0000","last_became_peered":"2024-01-16T14:58:24.429510+0000","last_unstale":"2024-01-16T14:58:24.431715+0000","last_undegraded":"2024-01-16T14:58:24.431715+0000","last_fullsized":"2024-01-16T14:58:24.431715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:56.142999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.402339+0000","last_change":"2024-01-16T14:58:24.402339+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.402339+0000","last_undegraded":"2024-01-16T14:58:24.402339+0000","last_fullsized":"2024-01-16T14:58:24.402339+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430778+0000","last_change":"2024-01-16T14:58:24.428471+0000","last_active":"2024-01-16T14:58:24.430778+0000","last_peered":"2024-01-16T14:58:24.430778+0000","last_clean":"2024-01-16T14:58:24.430778+0000","last_became_active":"2024-01-16T14:58:24.427946+0000","last_became_peered":"2024-01-16T14:58:24.427946+0000","last_unstale":"2024-01-16T14:58:24.430778+0000","last_undegraded":"2024-01-16T14:58:24.430778+0000","last_fullsized":"2024-01-16T14:58:24.430778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.340235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.407501+0000","last_change":"2024-01-16T14:58:24.407501+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.407501+0000","last_undegraded":"2024-01-16T14:58:24.407501+0000","last_fullsized":"2024-01-16T14:58:24.407501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.423929+0000","last_change":"2024-01-16T14:58:24.420850+0000","last_active":"2024-01-16T14:58:24.423929+0000","last_peered":"2024-01-16T14:58:24.423929+0000","last_clean":"2024-01-16T14:58:24.423929+0000","last_became_active":"2024-01-16T14:58:24.420553+0000","last_became_peered":"2024-01-16T14:58:24.420553+0000","last_unstale":"2024-01-16T14:58:24.423929+0000","last_undegraded":"2024-01-16T14:58:24.423929+0000","last_fullsized":"2024-01-16T14:58:24.423929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:36.056013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403443+0000","last_change":"2024-01-16T14:58:24.403443+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403443+0000","last_undegraded":"2024-01-16T14:58:24.403443+0000","last_fullsized":"2024-01-16T14:58:24.403443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430222+0000","last_change":"2024-01-16T14:58:24.423543+0000","last_active":"2024-01-16T14:58:24.430222+0000","last_peered":"2024-01-16T14:58:24.430222+0000","last_clean":"2024-01-16T14:58:24.430222+0000","last_became_active":"2024-01-16T14:58:24.420295+0000","last_became_peered":"2024-01-16T14:58:24.420295+0000","last_unstale":"2024-01-16T14:58:24.430222+0000","last_undegraded":"2024-01-16T14:58:24.430222+0000","last_fullsized":"2024-01-16T14:58:24.430222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:51.956244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.411919+0000","last_change":"2024-01-16T14:58:24.411919+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.411919+0000","last_undegraded":"2024-01-16T14:58:24.411919+0000","last_fullsized":"2024-01-16T14:58:24.411919+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.427918+0000","last_change":"2024-01-16T14:58:24.424952+0000","last_active":"2024-01-16T14:58:24.427918+0000","last_peered":"2024-01-16T14:58:24.427918+0000","last_clean":"2024-01-16T14:58:24.427918+0000","last_became_active":"2024-01-16T14:58:24.423308+0000","last_became_peered":"2024-01-16T14:58:24.423308+0000","last_unstale":"2024-01-16T14:58:24.427918+0000","last_undegraded":"2024-01-16T14:58:24.427918+0000","last_fullsized":"2024-01-16T14:58:24.427918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:59.581331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403031+0000","last_change":"2024-01-16T14:58:24.403031+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403031+0000","last_undegraded":"2024-01-16T14:58:24.403031+0000","last_fullsized":"2024-01-16T14:58:24.403031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430359+0000","last_change":"2024-01-16T14:58:24.427373+0000","last_active":"2024-01-16T14:58:24.430359+0000","last_peered":"2024-01-16T14:58:24.430359+0000","last_clean":"2024-01-16T14:58:24.430359+0000","last_became_active":"2024-01-16T14:58:24.426937+0000","last_became_peered":"2024-01-16T14:58:24.426937+0000","last_unstale":"2024-01-16T14:58:24.430359+0000","last_undegraded":"2024-01-16T14:58:24.430359+0000","last_fullsized":"2024-01-16T14:58:24.430359+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:34.799937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401387+0000","last_change":"2024-01-16T14:58:24.401387+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401387+0000","last_undegraded":"2024-01-16T14:58:24.401387+0000","last_fullsized":"2024-01-16T14:58:24.401387+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.426842+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:24.426842+0000","last_peered":"2024-01-16T14:58:24.426842+0000","last_clean":"2024-01-16T14:58:24.426842+0000","last_became_active":"2024-01-16T14:58:24.421586+0000","last_became_peered":"2024-01-16T14:58:24.421586+0000","last_unstale":"2024-01-16T14:58:24.426842+0000","last_undegraded":"2024-01-16T14:58:24.426842+0000","last_fullsized":"2024-01-16T14:58:24.426842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:45.630256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.400069+0000","last_change":"2024-01-16T14:58:24.400069+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.400069+0000","last_undegraded":"2024-01-16T14:58:24.400069+0000","last_fullsized":"2024-01-16T14:58:24.400069+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431307+0000","last_change":"2024-01-16T14:58:24.427165+0000","last_active":"2024-01-16T14:58:24.431307+0000","last_peered":"2024-01-16T14:58:24.431307+0000","last_clean":"2024-01-16T14:58:24.431307+0000","last_became_active":"2024-01-16T14:58:24.426927+0000","last_became_peered":"2024-01-16T14:58:24.426927+0000","last_unstale":"2024-01-16T14:58:24.431307+0000","last_undegraded":"2024-01-16T14:58:24.431307+0000","last_fullsized":"2024-01-16T14:58:24.431307+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414028+0000","last_change":"2024-01-16T14:58:24.414028+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414028+0000","last_undegraded":"2024-01-16T14:58:24.414028+0000","last_fullsized":"2024-01-16T14:58:24.414028+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430061+0000","last_change":"2024-01-16T14:58:24.425447+0000","last_active":"2024-01-16T14:58:24.430061+0000","last_peered":"2024-01-16T14:58:24.430061+0000","last_clean":"2024-01-16T14:58:24.430061+0000","last_became_active":"2024-01-16T14:58:24.425035+0000","last_became_peered":"2024-01-16T14:58:24.425035+0000","last_unstale":"2024-01-16T14:58:24.430061+0000","last_undegraded":"2024-01-16T14:58:24.430061+0000","last_fullsized":"2024-01-16T14:58:24.430061+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:36.056013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":175,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T14:58:23.400116+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:58:23.400116+0000","last_peered":"2024-01-16T14:58:23.400116+0000","last_clean":"2024-01-16T14:58:23.400116+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:58:23.400116+0000","last_undegraded":"2024-01-16T14:58:23.400116+0000","last_fullsized":"2024-01-16T14:58:23.400116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.2589999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.875},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.78800000000000003}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.097}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.21},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.1659999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.306},{"interface":"front","average":{"1min":1.45,"5min":1.45,"15min":1.45},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.589}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.109,"5min":1.109,"15min":1.109},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.357},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"max":{"1min":3.3530000000000002,"5min":3.3530000000000002,"15min":3.3530000000000002},"last":1.5309999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":3.258,"5min":3.258,"15min":3.258},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.617,"5min":1.617,"15min":1.617},"min":{"1min":1.139,"5min":1.139,"15min":1.139},"max":{"1min":3.101,"5min":3.101,"15min":3.101},"last":1.6519999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.76000000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.083},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.226}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.335},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.1830000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.2869999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.615},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.647}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.401},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.6919999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.90600000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.254},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.0700000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.369},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.1279999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.2}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.6259999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":2.9540000000000002},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.5}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"max":{"1min":3.2389999999999999,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":1.579}]}]},{"osd":4,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.085},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1359999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.70499999999999996}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":1.327}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.7490000000000001,"5min":2.7490000000000001,"15min":2.7490000000000001},"last":1.224},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":1.0549999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.2849999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.079,"5min":1.079,"15min":1.079},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.444},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":1.407}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":3.1880000000000002,"5min":3.1880000000000002,"15min":3.1880000000000002},"last":1.381},{"interface":"front","average":{"1min":1.788,"5min":1.788,"15min":1.788},"min":{"1min":1.232,"5min":1.232,"15min":1.232},"max":{"1min":3.2290000000000001,"5min":3.2290000000000001,"15min":3.2290000000000001},"last":1.258}]}]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.625},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.008,"5min":4.008,"15min":4.008},"last":0.754},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.79900000000000004}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.81899999999999995}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.88},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.028}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.97199999999999998}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.78000000000000003}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.56799999999999995}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.86799999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.3140000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.131},{"interface":"front","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.3799999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.236},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.0740000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.4119999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.3620000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.177},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.2150000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":1.298}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.038},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":0.93200000000000005}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.47},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.2549999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.649,"5min":2.649,"15min":2.649},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":1.4339999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.7290000000000001,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.0800000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.383},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.194,"5min":4.194,"15min":4.194},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.76700000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.2359999999999998,"5min":4.2359999999999998,"15min":4.2359999999999998},"last":0.96999999999999997}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.69799999999999995}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":4.5529999999999999,"5min":4.5529999999999999,"15min":4.5529999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":0.93000000000000005}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.4880000000000004,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.026},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.052}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":1.2470000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.325},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.524,"5min":4.524,"15min":4.524},"last":1.2010000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:58:29.839 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.101+0000 7f3563e00640 1 Processor -- start 2024-01-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.102+0000 7f3563e00640 1 -- start start 2024-01-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3563e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f355c0cf9a0 0x7f355c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3563e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f355c15e1e0 0x7f355c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3563e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0b1ee0 con 0x7f355c143ae0 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3563e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c0b2780 con 0x7f355c0cf9a0 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3563e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c15e990 con 0x7f355c15e1e0 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3560a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f355c0cf9a0 0x7f355c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3560a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f355c0cf9a0 0x7f355c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53230/0 (socket says 172.21.15.141:53230) 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.103+0000 7f3560a00640 1 -- 172.21.15.141:0/2507597921 learned_addr learned my addr 172.21.15.141:0/2507597921 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.104+0000 7f3560a00640 1 -- 172.21.15.141:0/2507597921 >> v2:172.21.15.188:3300/0 conn(0x7f355c15e1e0 msgr2=0x7f355c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.104+0000 7f3560a00640 1 --2- 172.21.15.141:0/2507597921 >> v2:172.21.15.188:3300/0 conn(0x7f355c15e1e0 0x7f355c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.104+0000 7f3560a00640 1 -- 172.21.15.141:0/2507597921 >> v1:172.21.15.188:6789/0 conn(0x7f355c143ae0 legacy=0x7f355c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.104+0000 7f3560a00640 1 -- 172.21.15.141:0/2507597921 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0b2130 con 0x7f355c0cf9a0 2024-01-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.105+0000 7f3560a00640 1 --2- 172.21.15.141:0/2507597921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c15dca0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f355c0ae9d0 tx=0x7f354402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994fae8d175d048f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.105+0000 7f355be00640 1 -- 172.21.15.141:0/2507597921 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544031870 con 0x7f355c0cf9a0 2024-01-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.105+0000 7f355be00640 1 -- 172.21.15.141:0/2507597921 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35440319d0 con 0x7f355c0cf9a0 2024-01-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.105+0000 7f355be00640 1 -- 172.21.15.141:0/2507597921 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544031d60 con 0x7f355c0cf9a0 2024-01-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 -- 172.21.15.141:0/2507597921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 msgr2=0x7f355c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 --2- 172.21.15.141:0/2507597921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c15dca0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f355c0ae9d0 tx=0x7f354402b910 comp rx=0 tx=0).stop 2024-01-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 -- 172.21.15.141:0/2507597921 shutdown_connections 2024-01-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 --2- 172.21.15.141:0/2507597921 >> v2:172.21.15.188:3300/0 conn(0x7f355c15e1e0 0x7f355c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 --2- 172.21.15.141:0/2507597921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c15dca0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 -- 172.21.15.141:0/2507597921 >> 172.21.15.141:0/2507597921 conn(0x7f355c06a800 msgr2=0x7f355c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 -- 172.21.15.141:0/2507597921 shutdown_connections 2024-01-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.106+0000 7f3563e00640 1 -- 172.21.15.141:0/2507597921 wait complete. 2024-01-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.108+0000 7f3563e00640 1 Processor -- start 2024-01-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.108+0000 7f3563e00640 1 -- start start 2024-01-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3563e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3563e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f355c143ae0 0x7f355c0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3563e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f355c15e1e0 0x7f355c13dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3563e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c159f40 con 0x7f355c0cf9a0 2024-01-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3563e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c13fbb0 con 0x7f355c15e1e0 2024-01-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3563e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c13ad20 con 0x7f355c143ae0 2024-01-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3561400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3561400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53232/0 (socket says 172.21.15.141:53232) 2024-01-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.109+0000 7f3561400640 1 -- 172.21.15.141:0/3564969203 learned_addr learned my addr 172.21.15.141:0/3564969203 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3560a00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f355c143ae0 0x7f355c0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f355c15e1e0 0x7f355c13dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561400640 1 -- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f355c143ae0 msgr2=0x7f355c0a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561400640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f355c143ae0 0x7f355c0a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561400640 1 -- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f355c15e1e0 msgr2=0x7f355c13dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561400640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f355c15e1e0 0x7f355c13dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561400640 1 -- 172.21.15.141:0/3564969203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35440094d0 con 0x7f355c0cf9a0 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f355c15e1e0 0x7f355c13dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3560a00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f355c143ae0 0x7f355c0a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3561400640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c159a00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c7a0 tx=0x7f354c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3559600640 1 -- 172.21.15.141:0/3564969203 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c003d80 con 0x7f355c0cf9a0 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3559600640 1 -- 172.21.15.141:0/3564969203 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c005070 con 0x7f355c0cf9a0 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3559600640 1 -- 172.21.15.141:0/3564969203 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c005420 con 0x7f355c0cf9a0 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.110+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c13bb80 con 0x7f355c0cf9a0 2024-01-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.111+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c0a8a00 con 0x7f355c0cf9a0 2024-01-16T14:58:30.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.112+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c0675d0 con 0x7f355c0cf9a0 2024-01-16T14:58:30.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.113+0000 7f3559600640 1 -- 172.21.15.141:0/3564969203 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f354c005580 con 0x7f355c0cf9a0 2024-01-16T14:58:30.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.114+0000 7f3559600640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f353406b2c0 0x7f353406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.114+0000 7f3559600640 1 -- 172.21.15.141:0/3564969203 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f354c089090 con 0x7f355c0cf9a0 2024-01-16T14:58:30.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.114+0000 7f3560a00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f353406b2c0 0x7f353406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.115+0000 7f3560a00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f353406b2c0 0x7f353406d780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3544036000 tx=0x7f354403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:30.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.118+0000 7f3559600640 1 -- 172.21.15.141:0/3564969203 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f354c053f90 con 0x7f355c0cf9a0 2024-01-16T14:58:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.305+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f355c15e670 con 0x7f353406b2c0 2024-01-16T14:58:30.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.327+0000 7f3559600640 1 -- 172.21.15.141:0/3564969203 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281181 (secure 0 0 0) 0x7f3544031d10 con 0x7f353406b2c0 2024-01-16T14:58:30.312 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:30.313 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f353406b2c0 msgr2=0x7f353406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f353406b2c0 0x7f353406d780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3544036000 tx=0x7f354403c040 comp rx=0 tx=0).stop 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 msgr2=0x7f355c159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c159a00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c7a0 tx=0x7f354c00cc60 comp rx=0 tx=0).stop 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 shutdown_connections 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f353406b2c0 0x7f353406d780 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f355c15e1e0 0x7f355c13dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f355c143ae0 0x7f355c0a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 --2- 172.21.15.141:0/3564969203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f355c0cf9a0 0x7f355c159a00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 >> 172.21.15.141:0/3564969203 conn(0x7f355c06a800 msgr2=0x7f355c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:30.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 shutdown_connections 2024-01-16T14:58:30.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.332+0000 7f3563e00640 1 -- 172.21.15.141:0/3564969203 wait complete. 2024-01-16T14:58:30.333 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-01-16T14:58:28.463542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220104,"kb_used_data":3944,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754616,"statfs":{"total":773094113280,"available":772868726784,"internally_reserved":0,"allocated":4038656,"data_stored":2261536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.077488"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409591+0000","last_change":"2024-01-16T14:58:24.409591+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409591+0000","last_undegraded":"2024-01-16T14:58:24.409591+0000","last_fullsized":"2024-01-16T14:58:24.409591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.406123+0000","last_change":"2024-01-16T14:58:24.406123+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.406123+0000","last_undegraded":"2024-01-16T14:58:24.406123+0000","last_fullsized":"2024-01-16T14:58:24.406123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403652+0000","last_change":"2024-01-16T14:58:24.403652+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403652+0000","last_undegraded":"2024-01-16T14:58:24.403652+0000","last_fullsized":"2024-01-16T14:58:24.403652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414133+0000","last_change":"2024-01-16T14:58:24.414133+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414133+0000","last_undegraded":"2024-01-16T14:58:24.414133+0000","last_fullsized":"2024-01-16T14:58:24.414133+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.410962+0000","last_change":"2024-01-16T14:58:24.410962+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.410962+0000","last_undegraded":"2024-01-16T14:58:24.410962+0000","last_fullsized":"2024-01-16T14:58:24.410962+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405581+0000","last_change":"2024-01-16T14:58:24.405581+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405581+0000","last_undegraded":"2024-01-16T14:58:24.405581+0000","last_fullsized":"2024-01-16T14:58:24.405581+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.415348+0000","last_change":"2024-01-16T14:58:24.415348+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.415348+0000","last_undegraded":"2024-01-16T14:58:24.415348+0000","last_fullsized":"2024-01-16T14:58:24.415348+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.424050+0000","last_change":"2024-01-16T14:58:24.424050+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.424050+0000","last_undegraded":"2024-01-16T14:58:24.424050+0000","last_fullsized":"2024-01-16T14:58:24.424050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409209+0000","last_change":"2024-01-16T14:58:24.409209+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409209+0000","last_undegraded":"2024-01-16T14:58:24.409209+0000","last_fullsized":"2024-01-16T14:58:24.409209+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414124+0000","last_change":"2024-01-16T14:58:24.414124+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414124+0000","last_undegraded":"2024-01-16T14:58:24.414124+0000","last_fullsized":"2024-01-16T14:58:24.414124+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.417717+0000","last_change":"2024-01-16T14:58:24.417717+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.417717+0000","last_undegraded":"2024-01-16T14:58:24.417717+0000","last_fullsized":"2024-01-16T14:58:24.417717+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.408501+0000","last_change":"2024-01-16T14:58:24.408501+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.408501+0000","last_undegraded":"2024-01-16T14:58:24.408501+0000","last_fullsized":"2024-01-16T14:58:24.408501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.410510+0000","last_change":"2024-01-16T14:58:24.410510+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.410510+0000","last_undegraded":"2024-01-16T14:58:24.410510+0000","last_fullsized":"2024-01-16T14:58:24.410510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404869+0000","last_change":"2024-01-16T14:58:24.404869+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404869+0000","last_undegraded":"2024-01-16T14:58:24.404869+0000","last_fullsized":"2024-01-16T14:58:24.404869+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401335+0000","last_change":"2024-01-16T14:58:24.401335+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401335+0000","last_undegraded":"2024-01-16T14:58:24.401335+0000","last_fullsized":"2024-01-16T14:58:24.401335+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.413375+0000","last_change":"2024-01-16T14:58:24.413375+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.413375+0000","last_undegraded":"2024-01-16T14:58:24.413375+0000","last_fullsized":"2024-01-16T14:58:24.413375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.413923+0000","last_change":"2024-01-16T14:58:24.413923+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.413923+0000","last_undegraded":"2024-01-16T14:58:24.413923+0000","last_fullsized":"2024-01-16T14:58:24.413923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404785+0000","last_change":"2024-01-16T14:58:24.404785+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404785+0000","last_undegraded":"2024-01-16T14:58:24.404785+0000","last_fullsized":"2024-01-16T14:58:24.404785+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404064+0000","last_change":"2024-01-16T14:58:24.404064+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404064+0000","last_undegraded":"2024-01-16T14:58:24.404064+0000","last_fullsized":"2024-01-16T14:58:24.404064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.407187+0000","last_change":"2024-01-16T14:58:24.407187+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.407187+0000","last_undegraded":"2024-01-16T14:58:24.407187+0000","last_fullsized":"2024-01-16T14:58:24.407187+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.413325+0000","last_change":"2024-01-16T14:58:24.413325+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.413325+0000","last_undegraded":"2024-01-16T14:58:24.413325+0000","last_fullsized":"2024-01-16T14:58:24.413325+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.407443+0000","last_change":"2024-01-16T14:58:24.407443+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.407443+0000","last_undegraded":"2024-01-16T14:58:24.407443+0000","last_fullsized":"2024-01-16T14:58:24.407443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403420+0000","last_change":"2024-01-16T14:58:24.403420+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403420+0000","last_undegraded":"2024-01-16T14:58:24.403420+0000","last_fullsized":"2024-01-16T14:58:24.403420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.399749+0000","last_change":"2024-01-16T14:58:24.399749+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.399749+0000","last_undegraded":"2024-01-16T14:58:24.399749+0000","last_fullsized":"2024-01-16T14:58:24.399749+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414535+0000","last_change":"2024-01-16T14:58:24.414535+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414535+0000","last_undegraded":"2024-01-16T14:58:24.414535+0000","last_fullsized":"2024-01-16T14:58:24.414535+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404079+0000","last_change":"2024-01-16T14:58:24.404079+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404079+0000","last_undegraded":"2024-01-16T14:58:24.404079+0000","last_fullsized":"2024-01-16T14:58:24.404079+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409991+0000","last_change":"2024-01-16T14:58:24.409991+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409991+0000","last_undegraded":"2024-01-16T14:58:24.409991+0000","last_fullsized":"2024-01-16T14:58:24.409991+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401405+0000","last_change":"2024-01-16T14:58:24.401405+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401405+0000","last_undegraded":"2024-01-16T14:58:24.401405+0000","last_fullsized":"2024-01-16T14:58:24.401405+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405298+0000","last_change":"2024-01-16T14:58:24.405298+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405298+0000","last_undegraded":"2024-01-16T14:58:24.405298+0000","last_fullsized":"2024-01-16T14:58:24.405298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.423789+0000","last_change":"2024-01-16T14:58:24.423789+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.423789+0000","last_undegraded":"2024-01-16T14:58:24.423789+0000","last_fullsized":"2024-01-16T14:58:24.423789+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.411277+0000","last_change":"2024-01-16T14:58:24.411277+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.411277+0000","last_undegraded":"2024-01-16T14:58:24.411277+0000","last_fullsized":"2024-01-16T14:58:24.411277+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.423000+0000","last_change":"2024-01-16T14:58:24.423000+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.423000+0000","last_undegraded":"2024-01-16T14:58:24.423000+0000","last_fullsized":"2024-01-16T14:58:24.423000+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.402573+0000","last_change":"2024-01-16T14:58:24.402573+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.402573+0000","last_undegraded":"2024-01-16T14:58:24.402573+0000","last_fullsized":"2024-01-16T14:58:24.402573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431192+0000","last_change":"2024-01-16T14:58:24.428745+0000","last_active":"2024-01-16T14:58:24.431192+0000","last_peered":"2024-01-16T14:58:24.431192+0000","last_clean":"2024-01-16T14:58:24.431192+0000","last_became_active":"2024-01-16T14:58:24.427170+0000","last_became_peered":"2024-01-16T14:58:24.427170+0000","last_unstale":"2024-01-16T14:58:24.431192+0000","last_undegraded":"2024-01-16T14:58:24.431192+0000","last_fullsized":"2024-01-16T14:58:24.431192+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:59.581331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.408928+0000","last_change":"2024-01-16T14:58:24.408928+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.408928+0000","last_undegraded":"2024-01-16T14:58:24.408928+0000","last_fullsized":"2024-01-16T14:58:24.408928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431329+0000","last_change":"2024-01-16T14:58:24.429636+0000","last_active":"2024-01-16T14:58:24.431329+0000","last_peered":"2024-01-16T14:58:24.431329+0000","last_clean":"2024-01-16T14:58:24.431329+0000","last_became_active":"2024-01-16T14:58:24.428619+0000","last_became_peered":"2024-01-16T14:58:24.428619+0000","last_unstale":"2024-01-16T14:58:24.431329+0000","last_undegraded":"2024-01-16T14:58:24.431329+0000","last_fullsized":"2024-01-16T14:58:24.431329+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:39.718130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.411759+0000","last_change":"2024-01-16T14:58:24.411759+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.411759+0000","last_undegraded":"2024-01-16T14:58:24.411759+0000","last_fullsized":"2024-01-16T14:58:24.411759+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.422992+0000","last_change":"2024-01-16T14:58:24.421162+0000","last_active":"2024-01-16T14:58:24.422992+0000","last_peered":"2024-01-16T14:58:24.422992+0000","last_clean":"2024-01-16T14:58:24.422992+0000","last_became_active":"2024-01-16T14:58:24.420858+0000","last_became_peered":"2024-01-16T14:58:24.420858+0000","last_unstale":"2024-01-16T14:58:24.422992+0000","last_undegraded":"2024-01-16T14:58:24.422992+0000","last_fullsized":"2024-01-16T14:58:24.422992+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:37.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409558+0000","last_change":"2024-01-16T14:58:24.409558+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409558+0000","last_undegraded":"2024-01-16T14:58:24.409558+0000","last_fullsized":"2024-01-16T14:58:24.409558+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.428852+0000","last_change":"2024-01-16T14:58:24.426642+0000","last_active":"2024-01-16T14:58:24.428852+0000","last_peered":"2024-01-16T14:58:24.428852+0000","last_clean":"2024-01-16T14:58:24.428852+0000","last_became_active":"2024-01-16T14:58:24.426379+0000","last_became_peered":"2024-01-16T14:58:24.426379+0000","last_unstale":"2024-01-16T14:58:24.428852+0000","last_undegraded":"2024-01-16T14:58:24.428852+0000","last_fullsized":"2024-01-16T14:58:24.428852+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:24.961936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.419238+0000","last_change":"2024-01-16T14:58:24.419238+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.419238+0000","last_undegraded":"2024-01-16T14:58:24.419238+0000","last_fullsized":"2024-01-16T14:58:24.419238+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.423647+0000","last_change":"2024-01-16T14:58:24.419926+0000","last_active":"2024-01-16T14:58:24.423647+0000","last_peered":"2024-01-16T14:58:24.423647+0000","last_clean":"2024-01-16T14:58:24.423647+0000","last_became_active":"2024-01-16T14:58:24.419550+0000","last_became_peered":"2024-01-16T14:58:24.419550+0000","last_unstale":"2024-01-16T14:58:24.423647+0000","last_undegraded":"2024-01-16T14:58:24.423647+0000","last_fullsized":"2024-01-16T14:58:24.423647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:25.305571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401594+0000","last_change":"2024-01-16T14:58:24.401594+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401594+0000","last_undegraded":"2024-01-16T14:58:24.401594+0000","last_fullsized":"2024-01-16T14:58:24.401594+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.422958+0000","last_change":"2024-01-16T14:58:24.420225+0000","last_active":"2024-01-16T14:58:24.422958+0000","last_peered":"2024-01-16T14:58:24.422958+0000","last_clean":"2024-01-16T14:58:24.422958+0000","last_became_active":"2024-01-16T14:58:24.419237+0000","last_became_peered":"2024-01-16T14:58:24.419237+0000","last_unstale":"2024-01-16T14:58:24.422958+0000","last_undegraded":"2024-01-16T14:58:24.422958+0000","last_fullsized":"2024-01-16T14:58:24.422958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:27.083704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.399035+0000","last_change":"2024-01-16T14:58:24.399035+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.399035+0000","last_undegraded":"2024-01-16T14:58:24.399035+0000","last_fullsized":"2024-01-16T14:58:24.399035+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.418782+0000","last_change":"2024-01-16T14:58:24.416771+0000","last_active":"2024-01-16T14:58:24.418782+0000","last_peered":"2024-01-16T14:58:24.418782+0000","last_clean":"2024-01-16T14:58:24.418782+0000","last_became_active":"2024-01-16T14:58:24.416479+0000","last_became_peered":"2024-01-16T14:58:24.416479+0000","last_unstale":"2024-01-16T14:58:24.418782+0000","last_undegraded":"2024-01-16T14:58:24.418782+0000","last_fullsized":"2024-01-16T14:58:24.418782+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:40.868803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414498+0000","last_change":"2024-01-16T14:58:24.414498+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414498+0000","last_undegraded":"2024-01-16T14:58:24.414498+0000","last_fullsized":"2024-01-16T14:58:24.414498+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.428247+0000","last_change":"2024-01-16T14:58:24.426653+0000","last_active":"2024-01-16T14:58:24.428247+0000","last_peered":"2024-01-16T14:58:24.428247+0000","last_clean":"2024-01-16T14:58:24.428247+0000","last_became_active":"2024-01-16T14:58:24.421492+0000","last_became_peered":"2024-01-16T14:58:24.421492+0000","last_unstale":"2024-01-16T14:58:24.428247+0000","last_undegraded":"2024-01-16T14:58:24.428247+0000","last_fullsized":"2024-01-16T14:58:24.428247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:02.111575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409360+0000","last_change":"2024-01-16T14:58:24.409360+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409360+0000","last_undegraded":"2024-01-16T14:58:24.409360+0000","last_fullsized":"2024-01-16T14:58:24.409360+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431265+0000","last_change":"2024-01-16T14:58:24.426750+0000","last_active":"2024-01-16T14:58:24.431265+0000","last_peered":"2024-01-16T14:58:24.431265+0000","last_clean":"2024-01-16T14:58:24.431265+0000","last_became_active":"2024-01-16T14:58:24.426534+0000","last_became_peered":"2024-01-16T14:58:24.426534+0000","last_unstale":"2024-01-16T14:58:24.431265+0000","last_undegraded":"2024-01-16T14:58:24.431265+0000","last_fullsized":"2024-01-16T14:58:24.431265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:19.956837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.404018+0000","last_change":"2024-01-16T14:58:24.404018+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.404018+0000","last_undegraded":"2024-01-16T14:58:24.404018+0000","last_fullsized":"2024-01-16T14:58:24.404018+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.418883+0000","last_change":"2024-01-16T14:58:24.414545+0000","last_active":"2024-01-16T14:58:24.418883+0000","last_peered":"2024-01-16T14:58:24.418883+0000","last_clean":"2024-01-16T14:58:24.418883+0000","last_became_active":"2024-01-16T14:58:24.412623+0000","last_became_peered":"2024-01-16T14:58:24.412623+0000","last_unstale":"2024-01-16T14:58:24.418883+0000","last_undegraded":"2024-01-16T14:58:24.418883+0000","last_fullsized":"2024-01-16T14:58:24.418883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:57.374401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401402+0000","last_change":"2024-01-16T14:58:24.401402+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401402+0000","last_undegraded":"2024-01-16T14:58:24.401402+0000","last_fullsized":"2024-01-16T14:58:24.401402+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431145+0000","last_change":"2024-01-16T14:58:24.428533+0000","last_active":"2024-01-16T14:58:24.431145+0000","last_peered":"2024-01-16T14:58:24.431145+0000","last_clean":"2024-01-16T14:58:24.431145+0000","last_became_active":"2024-01-16T14:58:24.428239+0000","last_became_peered":"2024-01-16T14:58:24.428239+0000","last_unstale":"2024-01-16T14:58:24.431145+0000","last_undegraded":"2024-01-16T14:58:24.431145+0000","last_fullsized":"2024-01-16T14:58:24.431145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409536+0000","last_change":"2024-01-16T14:58:24.409536+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409536+0000","last_undegraded":"2024-01-16T14:58:24.409536+0000","last_fullsized":"2024-01-16T14:58:24.409536+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430608+0000","last_change":"2024-01-16T14:58:24.426130+0000","last_active":"2024-01-16T14:58:24.430608+0000","last_peered":"2024-01-16T14:58:24.430608+0000","last_clean":"2024-01-16T14:58:24.430608+0000","last_became_active":"2024-01-16T14:58:24.425829+0000","last_became_peered":"2024-01-16T14:58:24.425829+0000","last_unstale":"2024-01-16T14:58:24.430608+0000","last_undegraded":"2024-01-16T14:58:24.430608+0000","last_fullsized":"2024-01-16T14:58:24.430608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:31.332098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414087+0000","last_change":"2024-01-16T14:58:24.414087+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414087+0000","last_undegraded":"2024-01-16T14:58:24.414087+0000","last_fullsized":"2024-01-16T14:58:24.414087+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.427923+0000","last_change":"2024-01-16T14:58:24.423758+0000","last_active":"2024-01-16T14:58:24.427923+0000","last_peered":"2024-01-16T14:58:24.427923+0000","last_clean":"2024-01-16T14:58:24.427923+0000","last_became_active":"2024-01-16T14:58:24.423458+0000","last_became_peered":"2024-01-16T14:58:24.423458+0000","last_unstale":"2024-01-16T14:58:24.427923+0000","last_undegraded":"2024-01-16T14:58:24.427923+0000","last_fullsized":"2024-01-16T14:58:24.427923+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:24.961936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401369+0000","last_change":"2024-01-16T14:58:24.401369+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401369+0000","last_undegraded":"2024-01-16T14:58:24.401369+0000","last_fullsized":"2024-01-16T14:58:24.401369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430379+0000","last_change":"2024-01-16T14:58:24.427569+0000","last_active":"2024-01-16T14:58:24.430379+0000","last_peered":"2024-01-16T14:58:24.430379+0000","last_clean":"2024-01-16T14:58:24.430379+0000","last_became_active":"2024-01-16T14:58:24.427309+0000","last_became_peered":"2024-01-16T14:58:24.427309+0000","last_unstale":"2024-01-16T14:58:24.430379+0000","last_undegraded":"2024-01-16T14:58:24.430379+0000","last_fullsized":"2024-01-16T14:58:24.430379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:24.349485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.421465+0000","last_change":"2024-01-16T14:58:24.421465+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.421465+0000","last_undegraded":"2024-01-16T14:58:24.421465+0000","last_fullsized":"2024-01-16T14:58:24.421465+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431377+0000","last_change":"2024-01-16T14:58:24.427259+0000","last_active":"2024-01-16T14:58:24.431377+0000","last_peered":"2024-01-16T14:58:24.431377+0000","last_clean":"2024-01-16T14:58:24.431377+0000","last_became_active":"2024-01-16T14:58:24.426959+0000","last_became_peered":"2024-01-16T14:58:24.426959+0000","last_unstale":"2024-01-16T14:58:24.431377+0000","last_undegraded":"2024-01-16T14:58:24.431377+0000","last_fullsized":"2024-01-16T14:58:24.431377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:43.995208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.410505+0000","last_change":"2024-01-16T14:58:24.410505+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.410505+0000","last_undegraded":"2024-01-16T14:58:24.410505+0000","last_fullsized":"2024-01-16T14:58:24.410505+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.418803+0000","last_change":"2024-01-16T14:58:24.416038+0000","last_active":"2024-01-16T14:58:24.418803+0000","last_peered":"2024-01-16T14:58:24.418803+0000","last_clean":"2024-01-16T14:58:24.418803+0000","last_became_active":"2024-01-16T14:58:24.415718+0000","last_became_peered":"2024-01-16T14:58:24.415718+0000","last_unstale":"2024-01-16T14:58:24.418803+0000","last_undegraded":"2024-01-16T14:58:24.418803+0000","last_fullsized":"2024-01-16T14:58:24.418803+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:20.089624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403800+0000","last_change":"2024-01-16T14:58:24.403800+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403800+0000","last_undegraded":"2024-01-16T14:58:24.403800+0000","last_fullsized":"2024-01-16T14:58:24.403800+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430252+0000","last_change":"2024-01-16T14:58:24.424464+0000","last_active":"2024-01-16T14:58:24.430252+0000","last_peered":"2024-01-16T14:58:24.430252+0000","last_clean":"2024-01-16T14:58:24.430252+0000","last_became_active":"2024-01-16T14:58:24.424200+0000","last_became_peered":"2024-01-16T14:58:24.424200+0000","last_unstale":"2024-01-16T14:58:24.430252+0000","last_undegraded":"2024-01-16T14:58:24.430252+0000","last_fullsized":"2024-01-16T14:58:24.430252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:39.251923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.412816+0000","last_change":"2024-01-16T14:58:24.412816+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.412816+0000","last_undegraded":"2024-01-16T14:58:24.412816+0000","last_fullsized":"2024-01-16T14:58:24.412816+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.427996+0000","last_change":"2024-01-16T14:58:24.420965+0000","last_active":"2024-01-16T14:58:24.427996+0000","last_peered":"2024-01-16T14:58:24.427996+0000","last_clean":"2024-01-16T14:58:24.427996+0000","last_became_active":"2024-01-16T14:58:24.420644+0000","last_became_peered":"2024-01-16T14:58:24.420644+0000","last_unstale":"2024-01-16T14:58:24.427996+0000","last_undegraded":"2024-01-16T14:58:24.427996+0000","last_fullsized":"2024-01-16T14:58:24.427996+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:40.305191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.408800+0000","last_change":"2024-01-16T14:58:24.408800+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.408800+0000","last_undegraded":"2024-01-16T14:58:24.408800+0000","last_fullsized":"2024-01-16T14:58:24.408800+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.428969+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:24.428969+0000","last_peered":"2024-01-16T14:58:24.428969+0000","last_clean":"2024-01-16T14:58:24.428969+0000","last_became_active":"2024-01-16T14:58:24.421626+0000","last_became_peered":"2024-01-16T14:58:24.421626+0000","last_unstale":"2024-01-16T14:58:24.428969+0000","last_undegraded":"2024-01-16T14:58:24.428969+0000","last_fullsized":"2024-01-16T14:58:24.428969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:57:30.780004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.409045+0000","last_change":"2024-01-16T14:58:24.409045+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.409045+0000","last_undegraded":"2024-01-16T14:58:24.409045+0000","last_fullsized":"2024-01-16T14:58:24.409045+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.419706+0000","last_change":"2024-01-16T14:58:24.413866+0000","last_active":"2024-01-16T14:58:24.419706+0000","last_peered":"2024-01-16T14:58:24.419706+0000","last_clean":"2024-01-16T14:58:24.419706+0000","last_became_active":"2024-01-16T14:58:24.413577+0000","last_became_peered":"2024-01-16T14:58:24.413577+0000","last_unstale":"2024-01-16T14:58:24.419706+0000","last_undegraded":"2024-01-16T14:58:24.419706+0000","last_fullsized":"2024-01-16T14:58:24.419706+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:43.604145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405624+0000","last_change":"2024-01-16T14:58:24.405624+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405624+0000","last_undegraded":"2024-01-16T14:58:24.405624+0000","last_fullsized":"2024-01-16T14:58:24.405624+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.421985+0000","last_change":"2024-01-16T14:58:24.417730+0000","last_active":"2024-01-16T14:58:24.421985+0000","last_peered":"2024-01-16T14:58:24.421985+0000","last_clean":"2024-01-16T14:58:24.421985+0000","last_became_active":"2024-01-16T14:58:24.417446+0000","last_became_peered":"2024-01-16T14:58:24.417446+0000","last_unstale":"2024-01-16T14:58:24.421985+0000","last_undegraded":"2024-01-16T14:58:24.421985+0000","last_fullsized":"2024-01-16T14:58:24.421985+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:09.948145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.405266+0000","last_change":"2024-01-16T14:58:24.405266+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.405266+0000","last_undegraded":"2024-01-16T14:58:24.405266+0000","last_fullsized":"2024-01-16T14:58:24.405266+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431017+0000","last_change":"2024-01-16T14:58:24.426266+0000","last_active":"2024-01-16T14:58:24.431017+0000","last_peered":"2024-01-16T14:58:24.431017+0000","last_clean":"2024-01-16T14:58:24.431017+0000","last_became_active":"2024-01-16T14:58:24.425944+0000","last_became_peered":"2024-01-16T14:58:24.425944+0000","last_unstale":"2024-01-16T14:58:24.431017+0000","last_undegraded":"2024-01-16T14:58:24.431017+0000","last_fullsized":"2024-01-16T14:58:24.431017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:57.374401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403031+0000","last_change":"2024-01-16T14:58:24.403031+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403031+0000","last_undegraded":"2024-01-16T14:58:24.403031+0000","last_fullsized":"2024-01-16T14:58:24.403031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.421182+0000","last_change":"2024-01-16T14:58:24.416410+0000","last_active":"2024-01-16T14:58:24.421182+0000","last_peered":"2024-01-16T14:58:24.421182+0000","last_clean":"2024-01-16T14:58:24.421182+0000","last_became_active":"2024-01-16T14:58:24.416121+0000","last_became_peered":"2024-01-16T14:58:24.416121+0000","last_unstale":"2024-01-16T14:58:24.421182+0000","last_undegraded":"2024-01-16T14:58:24.421182+0000","last_fullsized":"2024-01-16T14:58:24.421182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:45.988189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.422583+0000","last_change":"2024-01-16T14:58:24.422583+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.422583+0000","last_undegraded":"2024-01-16T14:58:24.422583+0000","last_fullsized":"2024-01-16T14:58:24.422583+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431715+0000","last_change":"2024-01-16T14:58:24.429805+0000","last_active":"2024-01-16T14:58:24.431715+0000","last_peered":"2024-01-16T14:58:24.431715+0000","last_clean":"2024-01-16T14:58:24.431715+0000","last_became_active":"2024-01-16T14:58:24.429510+0000","last_became_peered":"2024-01-16T14:58:24.429510+0000","last_unstale":"2024-01-16T14:58:24.431715+0000","last_undegraded":"2024-01-16T14:58:24.431715+0000","last_fullsized":"2024-01-16T14:58:24.431715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:56.142999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.402339+0000","last_change":"2024-01-16T14:58:24.402339+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.402339+0000","last_undegraded":"2024-01-16T14:58:24.402339+0000","last_fullsized":"2024-01-16T14:58:24.402339+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430778+0000","last_change":"2024-01-16T14:58:24.428471+0000","last_active":"2024-01-16T14:58:24.430778+0000","last_peered":"2024-01-16T14:58:24.430778+0000","last_clean":"2024-01-16T14:58:24.430778+0000","last_became_active":"2024-01-16T14:58:24.427946+0000","last_became_peered":"2024-01-16T14:58:24.427946+0000","last_unstale":"2024-01-16T14:58:24.430778+0000","last_undegraded":"2024-01-16T14:58:24.430778+0000","last_fullsized":"2024-01-16T14:58:24.430778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:08.340235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.407501+0000","last_change":"2024-01-16T14:58:24.407501+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.407501+0000","last_undegraded":"2024-01-16T14:58:24.407501+0000","last_fullsized":"2024-01-16T14:58:24.407501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.423929+0000","last_change":"2024-01-16T14:58:24.420850+0000","last_active":"2024-01-16T14:58:24.423929+0000","last_peered":"2024-01-16T14:58:24.423929+0000","last_clean":"2024-01-16T14:58:24.423929+0000","last_became_active":"2024-01-16T14:58:24.420553+0000","last_became_peered":"2024-01-16T14:58:24.420553+0000","last_unstale":"2024-01-16T14:58:24.423929+0000","last_undegraded":"2024-01-16T14:58:24.423929+0000","last_fullsized":"2024-01-16T14:58:24.423929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:36.056013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403443+0000","last_change":"2024-01-16T14:58:24.403443+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403443+0000","last_undegraded":"2024-01-16T14:58:24.403443+0000","last_fullsized":"2024-01-16T14:58:24.403443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430222+0000","last_change":"2024-01-16T14:58:24.423543+0000","last_active":"2024-01-16T14:58:24.430222+0000","last_peered":"2024-01-16T14:58:24.430222+0000","last_clean":"2024-01-16T14:58:24.430222+0000","last_became_active":"2024-01-16T14:58:24.420295+0000","last_became_peered":"2024-01-16T14:58:24.420295+0000","last_unstale":"2024-01-16T14:58:24.430222+0000","last_undegraded":"2024-01-16T14:58:24.430222+0000","last_fullsized":"2024-01-16T14:58:24.430222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:51.956244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.411919+0000","last_change":"2024-01-16T14:58:24.411919+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.411919+0000","last_undegraded":"2024-01-16T14:58:24.411919+0000","last_fullsized":"2024-01-16T14:58:24.411919+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.427918+0000","last_change":"2024-01-16T14:58:24.424952+0000","last_active":"2024-01-16T14:58:24.427918+0000","last_peered":"2024-01-16T14:58:24.427918+0000","last_clean":"2024-01-16T14:58:24.427918+0000","last_became_active":"2024-01-16T14:58:24.423308+0000","last_became_peered":"2024-01-16T14:58:24.423308+0000","last_unstale":"2024-01-16T14:58:24.427918+0000","last_undegraded":"2024-01-16T14:58:24.427918+0000","last_fullsized":"2024-01-16T14:58:24.427918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:59.581331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.403031+0000","last_change":"2024-01-16T14:58:24.403031+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.403031+0000","last_undegraded":"2024-01-16T14:58:24.403031+0000","last_fullsized":"2024-01-16T14:58:24.403031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430359+0000","last_change":"2024-01-16T14:58:24.427373+0000","last_active":"2024-01-16T14:58:24.430359+0000","last_peered":"2024-01-16T14:58:24.430359+0000","last_clean":"2024-01-16T14:58:24.430359+0000","last_became_active":"2024-01-16T14:58:24.426937+0000","last_became_peered":"2024-01-16T14:58:24.426937+0000","last_unstale":"2024-01-16T14:58:24.430359+0000","last_undegraded":"2024-01-16T14:58:24.430359+0000","last_fullsized":"2024-01-16T14:58:24.430359+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:34.799937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.401387+0000","last_change":"2024-01-16T14:58:24.401387+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.401387+0000","last_undegraded":"2024-01-16T14:58:24.401387+0000","last_fullsized":"2024-01-16T14:58:24.401387+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.426842+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:24.426842+0000","last_peered":"2024-01-16T14:58:24.426842+0000","last_clean":"2024-01-16T14:58:24.426842+0000","last_became_active":"2024-01-16T14:58:24.421586+0000","last_became_peered":"2024-01-16T14:58:24.421586+0000","last_unstale":"2024-01-16T14:58:24.426842+0000","last_undegraded":"2024-01-16T14:58:24.426842+0000","last_fullsized":"2024-01-16T14:58:24.426842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:00:45.630256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.400069+0000","last_change":"2024-01-16T14:58:24.400069+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.400069+0000","last_undegraded":"2024-01-16T14:58:24.400069+0000","last_fullsized":"2024-01-16T14:58:24.400069+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.431307+0000","last_change":"2024-01-16T14:58:24.427165+0000","last_active":"2024-01-16T14:58:24.431307+0000","last_peered":"2024-01-16T14:58:24.431307+0000","last_clean":"2024-01-16T14:58:24.431307+0000","last_became_active":"2024-01-16T14:58:24.426927+0000","last_became_peered":"2024-01-16T14:58:24.426927+0000","last_unstale":"2024-01-16T14:58:24.431307+0000","last_undegraded":"2024-01-16T14:58:24.431307+0000","last_fullsized":"2024-01-16T14:58:24.431307+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-16T14:58:24.414028+0000","last_change":"2024-01-16T14:58:24.414028+0000","last_active":"2024-01-16T14:58:24.382919+0000","last_peered":"2024-01-16T14:58:24.382919+0000","last_clean":"2024-01-16T14:58:24.382919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T14:58:24.414028+0000","last_undegraded":"2024-01-16T14:58:24.414028+0000","last_fullsized":"2024-01-16T14:58:24.414028+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T14:58:24.430061+0000","last_change":"2024-01-16T14:58:24.425447+0000","last_active":"2024-01-16T14:58:24.430061+0000","last_peered":"2024-01-16T14:58:24.430061+0000","last_clean":"2024-01-16T14:58:24.430061+0000","last_became_active":"2024-01-16T14:58:24.425035+0000","last_became_peered":"2024-01-16T14:58:24.425035+0000","last_unstale":"2024-01-16T14:58:24.430061+0000","last_undegraded":"2024-01-16T14:58:24.430061+0000","last_fullsized":"2024-01-16T14:58:24.430061+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:36.056013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":175,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T14:58:23.400116+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:58:23.400116+0000","last_peered":"2024-01-16T14:58:23.400116+0000","last_clean":"2024-01-16T14:58:23.400116+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:58:23.400116+0000","last_undegraded":"2024-01-16T14:58:23.400116+0000","last_fullsized":"2024-01-16T14:58:23.400116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.2589999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.875},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.78800000000000003}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.097}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.21},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.1659999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.306},{"interface":"front","average":{"1min":1.45,"5min":1.45,"15min":1.45},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.589}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.109,"5min":1.109,"15min":1.109},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.357},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"max":{"1min":3.3530000000000002,"5min":3.3530000000000002,"15min":3.3530000000000002},"last":1.5309999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":3.258,"5min":3.258,"15min":3.258},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.617,"5min":1.617,"15min":1.617},"min":{"1min":1.139,"5min":1.139,"15min":1.139},"max":{"1min":3.101,"5min":3.101,"15min":3.101},"last":1.6519999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.76000000000000001}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.083},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.226}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.335},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.1830000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.2869999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.615},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.647}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.401},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.6919999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.90600000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.254},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.0700000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.369},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.1279999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.2}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.6259999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":2.9540000000000002},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.5}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"max":{"1min":3.2389999999999999,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":1.579}]}]},{"osd":4,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.085},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1359999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.70499999999999996}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":1.327}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.7490000000000001,"5min":2.7490000000000001,"15min":2.7490000000000001},"last":1.224},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":1.0549999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.2849999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.079,"5min":1.079,"15min":1.079},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.444},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":1.407}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":3.1880000000000002,"5min":3.1880000000000002,"15min":3.1880000000000002},"last":1.381},{"interface":"front","average":{"1min":1.788,"5min":1.788,"15min":1.788},"min":{"1min":1.232,"5min":1.232,"15min":1.232},"max":{"1min":3.2290000000000001,"5min":3.2290000000000001,"15min":3.2290000000000001},"last":1.258}]}]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.625},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.008,"5min":4.008,"15min":4.008},"last":0.754},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.79900000000000004}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.81899999999999995}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.88},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.028}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.97199999999999998}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.78000000000000003}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.56799999999999995}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.86799999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.3140000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.131},{"interface":"front","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.3799999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.236},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.0740000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.4119999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.3620000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.177},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.2150000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":1.298}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.038},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":0.93200000000000005}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.47},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.2549999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.649,"5min":2.649,"15min":2.649},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":1.4339999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.7290000000000001,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.0800000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.383},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.194,"5min":4.194,"15min":4.194},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.76700000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.2359999999999998,"5min":4.2359999999999998,"15min":4.2359999999999998},"last":0.96999999999999997}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.69799999999999995}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":4.5529999999999999,"5min":4.5529999999999999,"15min":4.5529999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":0.93000000000000005}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.4880000000000004,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.026},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.052}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":1.2470000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.325},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.524,"5min":4.524,"15min":4.524},"last":1.2010000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:58:30.338 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:58:30.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.607+0000 7feb47400640 1 Processor -- start 2024-01-16T14:58:30.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.607+0000 7feb47400640 1 -- start start 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb47400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7feb4013e360 0x7feb400cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb47400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feb4015f9c0 0x7feb400cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb47400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4015c5f0 con 0x7feb4013df80 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb47400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4006c470 con 0x7feb4015f9c0 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb47400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4006c750 con 0x7feb4013e360 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb45400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feb4015f9c0 0x7feb400cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb45400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feb4015f9c0 0x7feb400cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53244/0 (socket says 172.21.15.141:53244) 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb45400640 1 -- 172.21.15.141:0/4145707801 learned_addr learned my addr 172.21.15.141:0/4145707801 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.609+0000 7feb45400640 1 -- 172.21.15.141:0/4145707801 >> v2:172.21.15.188:3300/0 conn(0x7feb4013e360 msgr2=0x7feb400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.610+0000 7feb45400640 1 --2- 172.21.15.141:0/4145707801 >> v2:172.21.15.188:3300/0 conn(0x7feb4013e360 0x7feb400cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.610+0000 7feb45400640 1 -- 172.21.15.141:0/4145707801 >> v1:172.21.15.188:6789/0 conn(0x7feb4013df80 legacy=0x7feb4015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:30.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.610+0000 7feb45400640 1 -- 172.21.15.141:0/4145707801 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4006ca30 con 0x7feb4015f9c0 2024-01-16T14:58:30.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.611+0000 7feb45400640 1 --2- 172.21.15.141:0/4145707801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4015f9c0 0x7feb400cf7d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7feb400ccae0 tx=0x7feb3802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5d570b6a886966 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:30.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.611+0000 7feb3f400640 1 -- 172.21.15.141:0/4145707801 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3802ca50 con 0x7feb4015f9c0 2024-01-16T14:58:30.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.611+0000 7feb3f400640 1 -- 172.21.15.141:0/4145707801 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3802cbb0 con 0x7feb4015f9c0 2024-01-16T14:58:30.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.611+0000 7feb3f400640 1 -- 172.21.15.141:0/4145707801 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380343f0 con 0x7feb4015f9c0 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 -- 172.21.15.141:0/4145707801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4015f9c0 msgr2=0x7feb400cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 --2- 172.21.15.141:0/4145707801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4015f9c0 0x7feb400cf7d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7feb400ccae0 tx=0x7feb3802beb0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 -- 172.21.15.141:0/4145707801 shutdown_connections 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 --2- 172.21.15.141:0/4145707801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4015f9c0 0x7feb400cf7d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 --2- 172.21.15.141:0/4145707801 >> v2:172.21.15.188:3300/0 conn(0x7feb4013e360 0x7feb400cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 -- 172.21.15.141:0/4145707801 >> 172.21.15.141:0/4145707801 conn(0x7feb4006a800 msgr2=0x7feb4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 -- 172.21.15.141:0/4145707801 shutdown_connections 2024-01-16T14:58:30.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.612+0000 7feb47400640 1 -- 172.21.15.141:0/4145707801 wait complete. 2024-01-16T14:58:30.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.614+0000 7feb47400640 1 Processor -- start 2024-01-16T14:58:30.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.614+0000 7feb47400640 1 -- start start 2024-01-16T14:58:30.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb47400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4013df80 0x7feb4015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb47400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb4013e360 0x7feb4014aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb47400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb4015f9c0 0x7feb4014b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb47400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4015af00 con 0x7feb4013df80 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb47400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400b28c0 con 0x7feb4015f9c0 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb47400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4014afc0 con 0x7feb4013e360 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4013df80 0x7feb4015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4013df80 0x7feb4015a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53258/0 (socket says 172.21.15.141:53258) 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 -- 172.21.15.141:0/1312934013 learned_addr learned my addr 172.21.15.141:0/1312934013 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb45400640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb4015f9c0 0x7feb4014b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 -- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb4013e360 msgr2=0x7feb4014aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb4013e360 0x7feb4014aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 -- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb4015f9c0 msgr2=0x7feb4014b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb4015f9c0 0x7feb4014b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.615+0000 7feb44a00640 1 -- 172.21.15.141:0/1312934013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38009b30 con 0x7feb4013df80 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.616+0000 7feb44a00640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4013df80 0x7feb4015a330 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feb3000b3f0 tx=0x7feb3000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.616+0000 7feb3cc00640 1 -- 172.21.15.141:0/1312934013 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30011830 con 0x7feb4013df80 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.616+0000 7feb3cc00640 1 -- 172.21.15.141:0/1312934013 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb30011990 con 0x7feb4013df80 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.616+0000 7feb3cc00640 1 -- 172.21.15.141:0/1312934013 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30011d40 con 0x7feb4013df80 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.616+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4015a940 con 0x7feb4013df80 2024-01-16T14:58:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.616+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb400b2550 con 0x7feb4013df80 2024-01-16T14:58:30.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.617+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb400b23b0 con 0x7feb4013df80 2024-01-16T14:58:30.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.618+0000 7feb3cc00640 1 -- 172.21.15.141:0/1312934013 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feb3001a400 con 0x7feb4013df80 2024-01-16T14:58:30.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.620+0000 7feb3cc00640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb1806b2c0 0x7feb1806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:30.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.620+0000 7feb3cc00640 1 -- 172.21.15.141:0/1312934013 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb30089be0 con 0x7feb4013df80 2024-01-16T14:58:30.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.620+0000 7feb3fe00640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb1806b2c0 0x7feb1806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:30.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.621+0000 7feb3fe00640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb1806b2c0 0x7feb1806d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feb401f6560 tx=0x7feb28006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:30.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.625+0000 7feb3cc00640 1 -- 172.21.15.141:0/1312934013 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb400b23b0 con 0x7feb4013df80 2024-01-16T14:58:30.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.812+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb4006e520 con 0x7feb1806b2c0 2024-01-16T14:58:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.836+0000 7feb3cc00640 1 -- 172.21.15.141:0/1312934013 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286529 (secure 0 0 0) 0x7feb4006e520 con 0x7feb1806b2c0 2024-01-16T14:58:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:30.823 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.842+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb1806b2c0 msgr2=0x7feb1806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.842+0000 7feb47400640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb1806b2c0 0x7feb1806d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feb401f6560 tx=0x7feb28006c90 comp rx=0 tx=0).stop 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.842+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4013df80 msgr2=0x7feb4015a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.842+0000 7feb47400640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4013df80 0x7feb4015a330 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feb3000b3f0 tx=0x7feb3000b8b0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 shutdown_connections 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb1806b2c0 0x7feb1806d780 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb4015f9c0 0x7feb4014b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb4013e360 0x7feb4014aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 --2- 172.21.15.141:0/1312934013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb4013df80 0x7feb4015a330 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:30.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 >> 172.21.15.141:0/1312934013 conn(0x7feb4006a800 msgr2=0x7feb4015cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:30.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 shutdown_connections 2024-01-16T14:58:30.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:30.843+0000 7feb47400640 1 -- 172.21.15.141:0/1312934013 wait complete. 2024-01-16T14:58:30.841 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-01-16T14:58:30.464293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224632,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750088,"statfs":{"total":773094113280,"available":772864090112,"internally_reserved":0,"allocated":4399104,"data_stored":2421092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.071246"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232198+0000","last_change":"2024-01-16T14:58:25.453436+0000","last_active":"2024-01-16T14:58:28.232198+0000","last_peered":"2024-01-16T14:58:28.232198+0000","last_clean":"2024-01-16T14:58:28.232198+0000","last_became_active":"2024-01-16T14:58:25.453170+0000","last_became_peered":"2024-01-16T14:58:25.453170+0000","last_unstale":"2024-01-16T14:58:28.232198+0000","last_undegraded":"2024-01-16T14:58:28.232198+0000","last_fullsized":"2024-01-16T14:58:28.232198+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:19.680397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089898+0000","last_change":"2024-01-16T14:58:25.450853+0000","last_active":"2024-01-16T14:58:28.089898+0000","last_peered":"2024-01-16T14:58:28.089898+0000","last_clean":"2024-01-16T14:58:28.089898+0000","last_became_active":"2024-01-16T14:58:25.447306+0000","last_became_peered":"2024-01-16T14:58:25.447306+0000","last_unstale":"2024-01-16T14:58:28.089898+0000","last_undegraded":"2024-01-16T14:58:28.089898+0000","last_fullsized":"2024-01-16T14:58:28.089898+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:58.177859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451614+0000","last_change":"2024-01-16T14:58:25.467441+0000","last_active":"2024-01-16T14:58:26.451614+0000","last_peered":"2024-01-16T14:58:26.451614+0000","last_clean":"2024-01-16T14:58:26.451614+0000","last_became_active":"2024-01-16T14:58:25.466836+0000","last_became_peered":"2024-01-16T14:58:25.466836+0000","last_unstale":"2024-01-16T14:58:26.451614+0000","last_undegraded":"2024-01-16T14:58:26.451614+0000","last_fullsized":"2024-01-16T14:58:26.451614+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:22.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726970+0000","last_change":"2024-01-16T14:58:25.459481+0000","last_active":"2024-01-16T14:58:27.726970+0000","last_peered":"2024-01-16T14:58:27.726970+0000","last_clean":"2024-01-16T14:58:27.726970+0000","last_became_active":"2024-01-16T14:58:25.459193+0000","last_became_peered":"2024-01-16T14:58:25.459193+0000","last_unstale":"2024-01-16T14:58:27.726970+0000","last_undegraded":"2024-01-16T14:58:27.726970+0000","last_fullsized":"2024-01-16T14:58:27.726970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.719820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.862896+0000","last_change":"2024-01-16T14:58:25.453879+0000","last_active":"2024-01-16T14:58:27.862896+0000","last_peered":"2024-01-16T14:58:27.862896+0000","last_clean":"2024-01-16T14:58:27.862896+0000","last_became_active":"2024-01-16T14:58:25.453454+0000","last_became_peered":"2024-01-16T14:58:25.453454+0000","last_unstale":"2024-01-16T14:58:27.862896+0000","last_undegraded":"2024-01-16T14:58:27.862896+0000","last_fullsized":"2024-01-16T14:58:27.862896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:25.362250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.088902+0000","last_change":"2024-01-16T14:58:25.454154+0000","last_active":"2024-01-16T14:58:28.088902+0000","last_peered":"2024-01-16T14:58:28.088902+0000","last_clean":"2024-01-16T14:58:28.088902+0000","last_became_active":"2024-01-16T14:58:25.453809+0000","last_became_peered":"2024-01-16T14:58:25.453809+0000","last_unstale":"2024-01-16T14:58:28.088902+0000","last_undegraded":"2024-01-16T14:58:28.088902+0000","last_fullsized":"2024-01-16T14:58:28.088902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:09.724971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448873+0000","last_change":"2024-01-16T14:58:25.449577+0000","last_active":"2024-01-16T14:58:26.448873+0000","last_peered":"2024-01-16T14:58:26.448873+0000","last_clean":"2024-01-16T14:58:26.448873+0000","last_became_active":"2024-01-16T14:58:25.449289+0000","last_became_peered":"2024-01-16T14:58:25.449289+0000","last_unstale":"2024-01-16T14:58:26.448873+0000","last_undegraded":"2024-01-16T14:58:26.448873+0000","last_fullsized":"2024-01-16T14:58:26.448873+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:22.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447326+0000","last_change":"2024-01-16T14:58:25.447662+0000","last_active":"2024-01-16T14:58:26.447326+0000","last_peered":"2024-01-16T14:58:26.447326+0000","last_clean":"2024-01-16T14:58:26.447326+0000","last_became_active":"2024-01-16T14:58:25.447359+0000","last_became_peered":"2024-01-16T14:58:25.447359+0000","last_unstale":"2024-01-16T14:58:26.447326+0000","last_undegraded":"2024-01-16T14:58:26.447326+0000","last_fullsized":"2024-01-16T14:58:26.447326+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.719820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454085+0000","last_change":"2024-01-16T14:58:25.445625+0000","last_active":"2024-01-16T14:58:26.454085+0000","last_peered":"2024-01-16T14:58:26.454085+0000","last_clean":"2024-01-16T14:58:26.454085+0000","last_became_active":"2024-01-16T14:58:25.445288+0000","last_became_peered":"2024-01-16T14:58:25.445288+0000","last_unstale":"2024-01-16T14:58:26.454085+0000","last_undegraded":"2024-01-16T14:58:26.454085+0000","last_fullsized":"2024-01-16T14:58:26.454085+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:48.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727416+0000","last_change":"2024-01-16T14:58:25.446611+0000","last_active":"2024-01-16T14:58:27.727416+0000","last_peered":"2024-01-16T14:58:27.727416+0000","last_clean":"2024-01-16T14:58:27.727416+0000","last_became_active":"2024-01-16T14:58:25.445802+0000","last_became_peered":"2024-01-16T14:58:25.445802+0000","last_unstale":"2024-01-16T14:58:27.727416+0000","last_undegraded":"2024-01-16T14:58:27.727416+0000","last_fullsized":"2024-01-16T14:58:27.727416+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448582+0000","last_change":"2024-01-16T14:58:25.457171+0000","last_active":"2024-01-16T14:58:26.448582+0000","last_peered":"2024-01-16T14:58:26.448582+0000","last_clean":"2024-01-16T14:58:26.448582+0000","last_became_active":"2024-01-16T14:58:25.456898+0000","last_became_peered":"2024-01-16T14:58:25.456898+0000","last_unstale":"2024-01-16T14:58:26.448582+0000","last_undegraded":"2024-01-16T14:58:26.448582+0000","last_fullsized":"2024-01-16T14:58:26.448582+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:11.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451319+0000","last_change":"2024-01-16T14:58:25.454493+0000","last_active":"2024-01-16T14:58:26.451319+0000","last_peered":"2024-01-16T14:58:26.451319+0000","last_clean":"2024-01-16T14:58:26.451319+0000","last_became_active":"2024-01-16T14:58:25.454215+0000","last_became_peered":"2024-01-16T14:58:25.454215+0000","last_unstale":"2024-01-16T14:58:26.451319+0000","last_undegraded":"2024-01-16T14:58:26.451319+0000","last_fullsized":"2024-01-16T14:58:26.451319+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:11.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863116+0000","last_change":"2024-01-16T14:58:25.447370+0000","last_active":"2024-01-16T14:58:27.863116+0000","last_peered":"2024-01-16T14:58:27.863116+0000","last_clean":"2024-01-16T14:58:27.863116+0000","last_became_active":"2024-01-16T14:58:25.447100+0000","last_became_peered":"2024-01-16T14:58:25.447100+0000","last_unstale":"2024-01-16T14:58:27.863116+0000","last_undegraded":"2024-01-16T14:58:27.863116+0000","last_fullsized":"2024-01-16T14:58:27.863116+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:40.264688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089314+0000","last_change":"2024-01-16T14:58:25.454014+0000","last_active":"2024-01-16T14:58:28.089314+0000","last_peered":"2024-01-16T14:58:28.089314+0000","last_clean":"2024-01-16T14:58:28.089314+0000","last_became_active":"2024-01-16T14:58:25.453284+0000","last_became_peered":"2024-01-16T14:58:25.453284+0000","last_unstale":"2024-01-16T14:58:28.089314+0000","last_undegraded":"2024-01-16T14:58:28.089314+0000","last_fullsized":"2024-01-16T14:58:28.089314+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:08.961898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451019+0000","last_change":"2024-01-16T14:58:25.450821+0000","last_active":"2024-01-16T14:58:26.451019+0000","last_peered":"2024-01-16T14:58:26.451019+0000","last_clean":"2024-01-16T14:58:26.451019+0000","last_became_active":"2024-01-16T14:58:25.450270+0000","last_became_peered":"2024-01-16T14:58:25.450270+0000","last_unstale":"2024-01-16T14:58:26.451019+0000","last_undegraded":"2024-01-16T14:58:26.451019+0000","last_fullsized":"2024-01-16T14:58:26.451019+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:37.439565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448105+0000","last_change":"2024-01-16T14:58:25.449064+0000","last_active":"2024-01-16T14:58:26.448105+0000","last_peered":"2024-01-16T14:58:26.448105+0000","last_clean":"2024-01-16T14:58:26.448105+0000","last_became_active":"2024-01-16T14:58:25.448777+0000","last_became_peered":"2024-01-16T14:58:25.448777+0000","last_unstale":"2024-01-16T14:58:26.448105+0000","last_undegraded":"2024-01-16T14:58:26.448105+0000","last_fullsized":"2024-01-16T14:58:26.448105+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449556+0000","last_change":"2024-01-16T14:58:25.452550+0000","last_active":"2024-01-16T14:58:26.449556+0000","last_peered":"2024-01-16T14:58:26.449556+0000","last_clean":"2024-01-16T14:58:26.449556+0000","last_became_active":"2024-01-16T14:58:25.452267+0000","last_became_peered":"2024-01-16T14:58:25.452267+0000","last_unstale":"2024-01-16T14:58:26.449556+0000","last_undegraded":"2024-01-16T14:58:26.449556+0000","last_fullsized":"2024-01-16T14:58:26.449556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:51.577006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089919+0000","last_change":"2024-01-16T14:58:25.447596+0000","last_active":"2024-01-16T14:58:28.089919+0000","last_peered":"2024-01-16T14:58:28.089919+0000","last_clean":"2024-01-16T14:58:28.089919+0000","last_became_active":"2024-01-16T14:58:25.446849+0000","last_became_peered":"2024-01-16T14:58:25.446849+0000","last_unstale":"2024-01-16T14:58:28.089919+0000","last_undegraded":"2024-01-16T14:58:28.089919+0000","last_fullsized":"2024-01-16T14:58:28.089919+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:25:27.806945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089503+0000","last_change":"2024-01-16T14:58:25.452291+0000","last_active":"2024-01-16T14:58:28.089503+0000","last_peered":"2024-01-16T14:58:28.089503+0000","last_clean":"2024-01-16T14:58:28.089503+0000","last_became_active":"2024-01-16T14:58:25.452017+0000","last_became_peered":"2024-01-16T14:58:25.452017+0000","last_unstale":"2024-01-16T14:58:28.089503+0000","last_undegraded":"2024-01-16T14:58:28.089503+0000","last_fullsized":"2024-01-16T14:58:28.089503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:31.087496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863458+0000","last_change":"2024-01-16T14:58:25.451264+0000","last_active":"2024-01-16T14:58:27.863458+0000","last_peered":"2024-01-16T14:58:27.863458+0000","last_clean":"2024-01-16T14:58:27.863458+0000","last_became_active":"2024-01-16T14:58:25.450973+0000","last_became_peered":"2024-01-16T14:58:25.450973+0000","last_unstale":"2024-01-16T14:58:27.863458+0000","last_undegraded":"2024-01-16T14:58:27.863458+0000","last_fullsized":"2024-01-16T14:58:27.863458+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:29.912804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452736+0000","last_change":"2024-01-16T14:58:25.445306+0000","last_active":"2024-01-16T14:58:26.452736+0000","last_peered":"2024-01-16T14:58:26.452736+0000","last_clean":"2024-01-16T14:58:26.452736+0000","last_became_active":"2024-01-16T14:58:25.442633+0000","last_became_peered":"2024-01-16T14:58:25.442633+0000","last_unstale":"2024-01-16T14:58:26.452736+0000","last_undegraded":"2024-01-16T14:58:26.452736+0000","last_fullsized":"2024-01-16T14:58:26.452736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:11.731137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452224+0000","last_change":"2024-01-16T14:58:25.448740+0000","last_active":"2024-01-16T14:58:26.452224+0000","last_peered":"2024-01-16T14:58:26.452224+0000","last_clean":"2024-01-16T14:58:26.452224+0000","last_became_active":"2024-01-16T14:58:25.448233+0000","last_became_peered":"2024-01-16T14:58:25.448233+0000","last_unstale":"2024-01-16T14:58:26.452224+0000","last_undegraded":"2024-01-16T14:58:26.452224+0000","last_fullsized":"2024-01-16T14:58:26.452224+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:36.906207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.527930+0000","last_change":"2024-01-16T14:58:25.445600+0000","last_active":"2024-01-16T14:58:26.527930+0000","last_peered":"2024-01-16T14:58:26.527930+0000","last_clean":"2024-01-16T14:58:26.527930+0000","last_became_active":"2024-01-16T14:58:25.445302+0000","last_became_peered":"2024-01-16T14:58:25.445302+0000","last_unstale":"2024-01-16T14:58:26.527930+0000","last_undegraded":"2024-01-16T14:58:26.527930+0000","last_fullsized":"2024-01-16T14:58:26.527930+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:45.333764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.450450+0000","last_change":"2024-01-16T14:58:25.452393+0000","last_active":"2024-01-16T14:58:26.450450+0000","last_peered":"2024-01-16T14:58:26.450450+0000","last_clean":"2024-01-16T14:58:26.450450+0000","last_became_active":"2024-01-16T14:58:25.451586+0000","last_became_peered":"2024-01-16T14:58:25.451586+0000","last_unstale":"2024-01-16T14:58:26.450450+0000","last_undegraded":"2024-01-16T14:58:26.450450+0000","last_fullsized":"2024-01-16T14:58:26.450450+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727298+0000","last_change":"2024-01-16T14:58:25.448882+0000","last_active":"2024-01-16T14:58:27.727298+0000","last_peered":"2024-01-16T14:58:27.727298+0000","last_clean":"2024-01-16T14:58:27.727298+0000","last_became_active":"2024-01-16T14:58:25.448300+0000","last_became_peered":"2024-01-16T14:58:25.448300+0000","last_unstale":"2024-01-16T14:58:27.727298+0000","last_undegraded":"2024-01-16T14:58:27.727298+0000","last_fullsized":"2024-01-16T14:58:27.727298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.090087+0000","last_change":"2024-01-16T14:58:25.458035+0000","last_active":"2024-01-16T14:58:28.090087+0000","last_peered":"2024-01-16T14:58:28.090087+0000","last_clean":"2024-01-16T14:58:28.090087+0000","last_became_active":"2024-01-16T14:58:25.457804+0000","last_became_peered":"2024-01-16T14:58:25.457804+0000","last_unstale":"2024-01-16T14:58:28.090087+0000","last_undegraded":"2024-01-16T14:58:28.090087+0000","last_fullsized":"2024-01-16T14:58:28.090087+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:39.016885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727478+0000","last_change":"2024-01-16T14:58:25.465239+0000","last_active":"2024-01-16T14:58:27.727478+0000","last_peered":"2024-01-16T14:58:27.727478+0000","last_clean":"2024-01-16T14:58:27.727478+0000","last_became_active":"2024-01-16T14:58:25.462123+0000","last_became_peered":"2024-01-16T14:58:25.462123+0000","last_unstale":"2024-01-16T14:58:27.727478+0000","last_undegraded":"2024-01-16T14:58:27.727478+0000","last_fullsized":"2024-01-16T14:58:27.727478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451320+0000","last_change":"2024-01-16T14:58:25.457821+0000","last_active":"2024-01-16T14:58:26.451320+0000","last_peered":"2024-01-16T14:58:26.451320+0000","last_clean":"2024-01-16T14:58:26.451320+0000","last_became_active":"2024-01-16T14:58:25.457526+0000","last_became_peered":"2024-01-16T14:58:25.457526+0000","last_unstale":"2024-01-16T14:58:26.451320+0000","last_undegraded":"2024-01-16T14:58:26.451320+0000","last_fullsized":"2024-01-16T14:58:26.451320+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452858+0000","last_change":"2024-01-16T14:58:25.445985+0000","last_active":"2024-01-16T14:58:26.452858+0000","last_peered":"2024-01-16T14:58:26.452858+0000","last_clean":"2024-01-16T14:58:26.452858+0000","last_became_active":"2024-01-16T14:58:25.443290+0000","last_became_peered":"2024-01-16T14:58:25.443290+0000","last_unstale":"2024-01-16T14:58:26.452858+0000","last_undegraded":"2024-01-16T14:58:26.452858+0000","last_fullsized":"2024-01-16T14:58:26.452858+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:40.619757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449725+0000","last_change":"2024-01-16T14:58:25.448385+0000","last_active":"2024-01-16T14:58:26.449725+0000","last_peered":"2024-01-16T14:58:26.449725+0000","last_clean":"2024-01-16T14:58:26.449725+0000","last_became_active":"2024-01-16T14:58:25.448118+0000","last_became_peered":"2024-01-16T14:58:25.448118+0000","last_unstale":"2024-01-16T14:58:26.449725+0000","last_undegraded":"2024-01-16T14:58:26.449725+0000","last_fullsized":"2024-01-16T14:58:26.449725+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:22.409645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.231762+0000","last_change":"2024-01-16T14:58:25.454801+0000","last_active":"2024-01-16T14:58:28.231762+0000","last_peered":"2024-01-16T14:58:28.231762+0000","last_clean":"2024-01-16T14:58:28.231762+0000","last_became_active":"2024-01-16T14:58:25.454281+0000","last_became_peered":"2024-01-16T14:58:25.454281+0000","last_unstale":"2024-01-16T14:58:28.231762+0000","last_undegraded":"2024-01-16T14:58:28.231762+0000","last_fullsized":"2024-01-16T14:58:28.231762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:41.881567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447551+0000","last_change":"2024-01-16T14:58:25.458893+0000","last_active":"2024-01-16T14:58:26.447551+0000","last_peered":"2024-01-16T14:58:26.447551+0000","last_clean":"2024-01-16T14:58:26.447551+0000","last_became_active":"2024-01-16T14:58:25.458300+0000","last_became_peered":"2024-01-16T14:58:25.458300+0000","last_unstale":"2024-01-16T14:58:26.447551+0000","last_undegraded":"2024-01-16T14:58:26.447551+0000","last_fullsized":"2024-01-16T14:58:26.447551+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528093+0000","last_change":"2024-01-16T14:58:25.448792+0000","last_active":"2024-01-16T14:58:26.528093+0000","last_peered":"2024-01-16T14:58:26.528093+0000","last_clean":"2024-01-16T14:58:26.528093+0000","last_became_active":"2024-01-16T14:58:25.448490+0000","last_became_peered":"2024-01-16T14:58:25.448490+0000","last_unstale":"2024-01-16T14:58:26.528093+0000","last_undegraded":"2024-01-16T14:58:26.528093+0000","last_fullsized":"2024-01-16T14:58:26.528093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:45.048906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474893+0000","last_change":"2024-01-16T14:58:24.428745+0000","last_active":"2024-01-16T14:58:25.474893+0000","last_peered":"2024-01-16T14:58:25.474893+0000","last_clean":"2024-01-16T14:58:25.474893+0000","last_became_active":"2024-01-16T14:58:24.427170+0000","last_became_peered":"2024-01-16T14:58:24.427170+0000","last_unstale":"2024-01-16T14:58:25.474893+0000","last_undegraded":"2024-01-16T14:58:25.474893+0000","last_fullsized":"2024-01-16T14:58:25.474893+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:43.995208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232212+0000","last_change":"2024-01-16T14:58:25.455325+0000","last_active":"2024-01-16T14:58:28.232212+0000","last_peered":"2024-01-16T14:58:28.232212+0000","last_clean":"2024-01-16T14:58:28.232212+0000","last_became_active":"2024-01-16T14:58:25.454992+0000","last_became_peered":"2024-01-16T14:58:25.454992+0000","last_unstale":"2024-01-16T14:58:28.232212+0000","last_undegraded":"2024-01-16T14:58:28.232212+0000","last_fullsized":"2024-01-16T14:58:28.232212+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:36.367808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.469108+0000","last_change":"2024-01-16T14:58:24.429636+0000","last_active":"2024-01-16T14:58:25.469108+0000","last_peered":"2024-01-16T14:58:25.469108+0000","last_clean":"2024-01-16T14:58:25.469108+0000","last_became_active":"2024-01-16T14:58:24.428619+0000","last_became_peered":"2024-01-16T14:58:24.428619+0000","last_unstale":"2024-01-16T14:58:25.469108+0000","last_undegraded":"2024-01-16T14:58:25.469108+0000","last_fullsized":"2024-01-16T14:58:25.469108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:40.868803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.453290+0000","last_change":"2024-01-16T14:58:25.450473+0000","last_active":"2024-01-16T14:58:26.453290+0000","last_peered":"2024-01-16T14:58:26.453290+0000","last_clean":"2024-01-16T14:58:26.453290+0000","last_became_active":"2024-01-16T14:58:25.450169+0000","last_became_peered":"2024-01-16T14:58:25.450169+0000","last_unstale":"2024-01-16T14:58:26.453290+0000","last_undegraded":"2024-01-16T14:58:26.453290+0000","last_fullsized":"2024-01-16T14:58:26.453290+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:43.043403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474307+0000","last_change":"2024-01-16T14:58:24.421162+0000","last_active":"2024-01-16T14:58:25.474307+0000","last_peered":"2024-01-16T14:58:25.474307+0000","last_clean":"2024-01-16T14:58:25.474307+0000","last_became_active":"2024-01-16T14:58:24.420858+0000","last_became_peered":"2024-01-16T14:58:24.420858+0000","last_unstale":"2024-01-16T14:58:25.474307+0000","last_undegraded":"2024-01-16T14:58:25.474307+0000","last_fullsized":"2024-01-16T14:58:25.474307+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:48.834908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727604+0000","last_change":"2024-01-16T14:58:25.465488+0000","last_active":"2024-01-16T14:58:27.727604+0000","last_peered":"2024-01-16T14:58:27.727604+0000","last_clean":"2024-01-16T14:58:27.727604+0000","last_became_active":"2024-01-16T14:58:25.460624+0000","last_became_peered":"2024-01-16T14:58:25.460624+0000","last_unstale":"2024-01-16T14:58:27.727604+0000","last_undegraded":"2024-01-16T14:58:27.727604+0000","last_fullsized":"2024-01-16T14:58:27.727604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.473676+0000","last_change":"2024-01-16T14:58:24.426642+0000","last_active":"2024-01-16T14:58:25.473676+0000","last_peered":"2024-01-16T14:58:25.473676+0000","last_clean":"2024-01-16T14:58:25.473676+0000","last_became_active":"2024-01-16T14:58:24.426379+0000","last_became_peered":"2024-01-16T14:58:24.426379+0000","last_unstale":"2024-01-16T14:58:25.473676+0000","last_undegraded":"2024-01-16T14:58:25.473676+0000","last_fullsized":"2024-01-16T14:58:25.473676+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:29:09.809503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449031+0000","last_change":"2024-01-16T14:58:25.458055+0000","last_active":"2024-01-16T14:58:26.449031+0000","last_peered":"2024-01-16T14:58:26.449031+0000","last_clean":"2024-01-16T14:58:26.449031+0000","last_became_active":"2024-01-16T14:58:25.457556+0000","last_became_peered":"2024-01-16T14:58:25.457556+0000","last_unstale":"2024-01-16T14:58:26.449031+0000","last_undegraded":"2024-01-16T14:58:26.449031+0000","last_fullsized":"2024-01-16T14:58:26.449031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:37.439565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.471816+0000","last_change":"2024-01-16T14:58:24.419926+0000","last_active":"2024-01-16T14:58:25.471816+0000","last_peered":"2024-01-16T14:58:25.471816+0000","last_clean":"2024-01-16T14:58:25.471816+0000","last_became_active":"2024-01-16T14:58:24.419550+0000","last_became_peered":"2024-01-16T14:58:24.419550+0000","last_unstale":"2024-01-16T14:58:25.471816+0000","last_undegraded":"2024-01-16T14:58:25.471816+0000","last_fullsized":"2024-01-16T14:58:25.471816+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:09.094139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451316+0000","last_change":"2024-01-16T14:58:25.453757+0000","last_active":"2024-01-16T14:58:26.451316+0000","last_peered":"2024-01-16T14:58:26.451316+0000","last_clean":"2024-01-16T14:58:26.451316+0000","last_became_active":"2024-01-16T14:58:25.453492+0000","last_became_peered":"2024-01-16T14:58:25.453492+0000","last_unstale":"2024-01-16T14:58:26.451316+0000","last_undegraded":"2024-01-16T14:58:26.451316+0000","last_fullsized":"2024-01-16T14:58:26.451316+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.471378+0000","last_change":"2024-01-16T14:58:24.420225+0000","last_active":"2024-01-16T14:58:25.471378+0000","last_peered":"2024-01-16T14:58:25.471378+0000","last_clean":"2024-01-16T14:58:25.471378+0000","last_became_active":"2024-01-16T14:58:24.419237+0000","last_became_peered":"2024-01-16T14:58:24.419237+0000","last_unstale":"2024-01-16T14:58:25.471378+0000","last_undegraded":"2024-01-16T14:58:25.471378+0000","last_fullsized":"2024-01-16T14:58:25.471378+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:21.201884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451519+0000","last_change":"2024-01-16T14:58:25.451171+0000","last_active":"2024-01-16T14:58:26.451519+0000","last_peered":"2024-01-16T14:58:26.451519+0000","last_clean":"2024-01-16T14:58:26.451519+0000","last_became_active":"2024-01-16T14:58:25.450320+0000","last_became_peered":"2024-01-16T14:58:25.450320+0000","last_unstale":"2024-01-16T14:58:26.451519+0000","last_undegraded":"2024-01-16T14:58:26.451519+0000","last_fullsized":"2024-01-16T14:58:26.451519+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:57:49.471035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437180+0000","last_change":"2024-01-16T14:58:24.416771+0000","last_active":"2024-01-16T14:58:25.437180+0000","last_peered":"2024-01-16T14:58:25.437180+0000","last_clean":"2024-01-16T14:58:25.437180+0000","last_became_active":"2024-01-16T14:58:24.416479+0000","last_became_peered":"2024-01-16T14:58:24.416479+0000","last_unstale":"2024-01-16T14:58:25.437180+0000","last_undegraded":"2024-01-16T14:58:25.437180+0000","last_fullsized":"2024-01-16T14:58:25.437180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:15.420130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726309+0000","last_change":"2024-01-16T14:58:25.446441+0000","last_active":"2024-01-16T14:58:27.726309+0000","last_peered":"2024-01-16T14:58:27.726309+0000","last_clean":"2024-01-16T14:58:27.726309+0000","last_became_active":"2024-01-16T14:58:25.446153+0000","last_became_peered":"2024-01-16T14:58:25.446153+0000","last_unstale":"2024-01-16T14:58:27.726309+0000","last_undegraded":"2024-01-16T14:58:27.726309+0000","last_fullsized":"2024-01-16T14:58:27.726309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:59.978419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.424174+0000","last_change":"2024-01-16T14:58:24.426653+0000","last_active":"2024-01-16T14:58:25.424174+0000","last_peered":"2024-01-16T14:58:25.424174+0000","last_clean":"2024-01-16T14:58:25.424174+0000","last_became_active":"2024-01-16T14:58:24.421492+0000","last_became_peered":"2024-01-16T14:58:24.421492+0000","last_unstale":"2024-01-16T14:58:25.424174+0000","last_undegraded":"2024-01-16T14:58:25.424174+0000","last_fullsized":"2024-01-16T14:58:25.424174+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:51.956244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454109+0000","last_change":"2024-01-16T14:58:25.455256+0000","last_active":"2024-01-16T14:58:26.454109+0000","last_peered":"2024-01-16T14:58:26.454109+0000","last_clean":"2024-01-16T14:58:26.454109+0000","last_became_active":"2024-01-16T14:58:25.454696+0000","last_became_peered":"2024-01-16T14:58:25.454696+0000","last_unstale":"2024-01-16T14:58:26.454109+0000","last_undegraded":"2024-01-16T14:58:26.454109+0000","last_fullsized":"2024-01-16T14:58:26.454109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:11.932103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.443769+0000","last_change":"2024-01-16T14:58:24.426750+0000","last_active":"2024-01-16T14:58:25.443769+0000","last_peered":"2024-01-16T14:58:25.443769+0000","last_clean":"2024-01-16T14:58:25.443769+0000","last_became_active":"2024-01-16T14:58:24.426534+0000","last_became_peered":"2024-01-16T14:58:24.426534+0000","last_unstale":"2024-01-16T14:58:25.443769+0000","last_undegraded":"2024-01-16T14:58:25.443769+0000","last_fullsized":"2024-01-16T14:58:25.443769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:32.501031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089695+0000","last_change":"2024-01-16T14:58:25.471298+0000","last_active":"2024-01-16T14:58:28.089695+0000","last_peered":"2024-01-16T14:58:28.089695+0000","last_clean":"2024-01-16T14:58:28.089695+0000","last_became_active":"2024-01-16T14:58:25.470921+0000","last_became_peered":"2024-01-16T14:58:25.470921+0000","last_unstale":"2024-01-16T14:58:28.089695+0000","last_undegraded":"2024-01-16T14:58:28.089695+0000","last_fullsized":"2024-01-16T14:58:28.089695+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:18.391832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.421251+0000","last_change":"2024-01-16T14:58:24.414545+0000","last_active":"2024-01-16T14:58:25.421251+0000","last_peered":"2024-01-16T14:58:25.421251+0000","last_clean":"2024-01-16T14:58:25.421251+0000","last_became_active":"2024-01-16T14:58:24.412623+0000","last_became_peered":"2024-01-16T14:58:24.412623+0000","last_unstale":"2024-01-16T14:58:25.421251+0000","last_undegraded":"2024-01-16T14:58:25.421251+0000","last_fullsized":"2024-01-16T14:58:25.421251+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:37.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089315+0000","last_change":"2024-01-16T14:58:25.456962+0000","last_active":"2024-01-16T14:58:28.089315+0000","last_peered":"2024-01-16T14:58:28.089315+0000","last_clean":"2024-01-16T14:58:28.089315+0000","last_became_active":"2024-01-16T14:58:25.456712+0000","last_became_peered":"2024-01-16T14:58:25.456712+0000","last_unstale":"2024-01-16T14:58:28.089315+0000","last_undegraded":"2024-01-16T14:58:28.089315+0000","last_fullsized":"2024-01-16T14:58:28.089315+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:10.429909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.473767+0000","last_change":"2024-01-16T14:58:24.428533+0000","last_active":"2024-01-16T14:58:25.473767+0000","last_peered":"2024-01-16T14:58:25.473767+0000","last_clean":"2024-01-16T14:58:25.473767+0000","last_became_active":"2024-01-16T14:58:24.428239+0000","last_became_peered":"2024-01-16T14:58:24.428239+0000","last_unstale":"2024-01-16T14:58:25.473767+0000","last_undegraded":"2024-01-16T14:58:25.473767+0000","last_fullsized":"2024-01-16T14:58:25.473767+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:32.544123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451239+0000","last_change":"2024-01-16T14:58:25.467675+0000","last_active":"2024-01-16T14:58:26.451239+0000","last_peered":"2024-01-16T14:58:26.451239+0000","last_clean":"2024-01-16T14:58:26.451239+0000","last_became_active":"2024-01-16T14:58:25.467220+0000","last_became_peered":"2024-01-16T14:58:25.467220+0000","last_unstale":"2024-01-16T14:58:26.451239+0000","last_undegraded":"2024-01-16T14:58:26.451239+0000","last_fullsized":"2024-01-16T14:58:26.451239+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.477630+0000","last_change":"2024-01-16T14:58:24.426130+0000","last_active":"2024-01-16T14:58:25.477630+0000","last_peered":"2024-01-16T14:58:25.477630+0000","last_clean":"2024-01-16T14:58:25.477630+0000","last_became_active":"2024-01-16T14:58:24.425829+0000","last_became_peered":"2024-01-16T14:58:24.425829+0000","last_unstale":"2024-01-16T14:58:25.477630+0000","last_undegraded":"2024-01-16T14:58:25.477630+0000","last_fullsized":"2024-01-16T14:58:25.477630+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:11.348392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726576+0000","last_change":"2024-01-16T14:58:25.446712+0000","last_active":"2024-01-16T14:58:27.726576+0000","last_peered":"2024-01-16T14:58:27.726576+0000","last_clean":"2024-01-16T14:58:27.726576+0000","last_became_active":"2024-01-16T14:58:25.446349+0000","last_became_peered":"2024-01-16T14:58:25.446349+0000","last_unstale":"2024-01-16T14:58:27.726576+0000","last_undegraded":"2024-01-16T14:58:27.726576+0000","last_fullsized":"2024-01-16T14:58:27.726576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:33.556888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474223+0000","last_change":"2024-01-16T14:58:24.423758+0000","last_active":"2024-01-16T14:58:25.474223+0000","last_peered":"2024-01-16T14:58:25.474223+0000","last_clean":"2024-01-16T14:58:25.474223+0000","last_became_active":"2024-01-16T14:58:24.423458+0000","last_became_peered":"2024-01-16T14:58:24.423458+0000","last_unstale":"2024-01-16T14:58:25.474223+0000","last_undegraded":"2024-01-16T14:58:25.474223+0000","last_fullsized":"2024-01-16T14:58:25.474223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:56.142999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089168+0000","last_change":"2024-01-16T14:58:25.447562+0000","last_active":"2024-01-16T14:58:28.089168+0000","last_peered":"2024-01-16T14:58:28.089168+0000","last_clean":"2024-01-16T14:58:28.089168+0000","last_became_active":"2024-01-16T14:58:25.447243+0000","last_became_peered":"2024-01-16T14:58:25.447243+0000","last_unstale":"2024-01-16T14:58:28.089168+0000","last_undegraded":"2024-01-16T14:58:28.089168+0000","last_fullsized":"2024-01-16T14:58:28.089168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.429141+0000","last_change":"2024-01-16T14:58:24.427569+0000","last_active":"2024-01-16T14:58:25.429141+0000","last_peered":"2024-01-16T14:58:25.429141+0000","last_clean":"2024-01-16T14:58:25.429141+0000","last_became_active":"2024-01-16T14:58:24.427309+0000","last_became_peered":"2024-01-16T14:58:24.427309+0000","last_unstale":"2024-01-16T14:58:25.429141+0000","last_undegraded":"2024-01-16T14:58:25.429141+0000","last_fullsized":"2024-01-16T14:58:25.429141+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:19.412624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447681+0000","last_change":"2024-01-16T14:58:25.458315+0000","last_active":"2024-01-16T14:58:26.447681+0000","last_peered":"2024-01-16T14:58:26.447681+0000","last_clean":"2024-01-16T14:58:26.447681+0000","last_became_active":"2024-01-16T14:58:25.458030+0000","last_became_peered":"2024-01-16T14:58:25.458030+0000","last_unstale":"2024-01-16T14:58:26.447681+0000","last_undegraded":"2024-01-16T14:58:26.447681+0000","last_fullsized":"2024-01-16T14:58:26.447681+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437070+0000","last_change":"2024-01-16T14:58:24.427259+0000","last_active":"2024-01-16T14:58:25.437070+0000","last_peered":"2024-01-16T14:58:25.437070+0000","last_clean":"2024-01-16T14:58:25.437070+0000","last_became_active":"2024-01-16T14:58:24.426959+0000","last_became_peered":"2024-01-16T14:58:24.426959+0000","last_unstale":"2024-01-16T14:58:25.437070+0000","last_undegraded":"2024-01-16T14:58:25.437070+0000","last_fullsized":"2024-01-16T14:58:25.437070+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:10.096980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232161+0000","last_change":"2024-01-16T14:58:25.460308+0000","last_active":"2024-01-16T14:58:28.232161+0000","last_peered":"2024-01-16T14:58:28.232161+0000","last_clean":"2024-01-16T14:58:28.232161+0000","last_became_active":"2024-01-16T14:58:25.458176+0000","last_became_peered":"2024-01-16T14:58:25.458176+0000","last_unstale":"2024-01-16T14:58:28.232161+0000","last_undegraded":"2024-01-16T14:58:28.232161+0000","last_fullsized":"2024-01-16T14:58:28.232161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:20.969602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.475331+0000","last_change":"2024-01-16T14:58:24.416038+0000","last_active":"2024-01-16T14:58:25.475331+0000","last_peered":"2024-01-16T14:58:25.475331+0000","last_clean":"2024-01-16T14:58:25.475331+0000","last_became_active":"2024-01-16T14:58:24.415718+0000","last_became_peered":"2024-01-16T14:58:24.415718+0000","last_unstale":"2024-01-16T14:58:25.475331+0000","last_undegraded":"2024-01-16T14:58:25.475331+0000","last_fullsized":"2024-01-16T14:58:25.475331+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:03.937440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.453721+0000","last_change":"2024-01-16T14:58:25.451447+0000","last_active":"2024-01-16T14:58:26.453721+0000","last_peered":"2024-01-16T14:58:26.453721+0000","last_clean":"2024-01-16T14:58:26.453721+0000","last_became_active":"2024-01-16T14:58:25.446471+0000","last_became_peered":"2024-01-16T14:58:25.446471+0000","last_unstale":"2024-01-16T14:58:26.453721+0000","last_undegraded":"2024-01-16T14:58:26.453721+0000","last_fullsized":"2024-01-16T14:58:26.453721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:31.965127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.476243+0000","last_change":"2024-01-16T14:58:24.424464+0000","last_active":"2024-01-16T14:58:25.476243+0000","last_peered":"2024-01-16T14:58:25.476243+0000","last_clean":"2024-01-16T14:58:25.476243+0000","last_became_active":"2024-01-16T14:58:24.424200+0000","last_became_peered":"2024-01-16T14:58:24.424200+0000","last_unstale":"2024-01-16T14:58:25.476243+0000","last_undegraded":"2024-01-16T14:58:25.476243+0000","last_fullsized":"2024-01-16T14:58:25.476243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:37.904616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454392+0000","last_change":"2024-01-16T14:58:25.449719+0000","last_active":"2024-01-16T14:58:26.454392+0000","last_peered":"2024-01-16T14:58:26.454392+0000","last_clean":"2024-01-16T14:58:26.454392+0000","last_became_active":"2024-01-16T14:58:25.447359+0000","last_became_peered":"2024-01-16T14:58:25.447359+0000","last_unstale":"2024-01-16T14:58:26.454392+0000","last_undegraded":"2024-01-16T14:58:26.454392+0000","last_fullsized":"2024-01-16T14:58:26.454392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:22.409645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474319+0000","last_change":"2024-01-16T14:58:24.420965+0000","last_active":"2024-01-16T14:58:25.474319+0000","last_peered":"2024-01-16T14:58:25.474319+0000","last_clean":"2024-01-16T14:58:25.474319+0000","last_became_active":"2024-01-16T14:58:24.420644+0000","last_became_peered":"2024-01-16T14:58:24.420644+0000","last_unstale":"2024-01-16T14:58:25.474319+0000","last_undegraded":"2024-01-16T14:58:25.474319+0000","last_fullsized":"2024-01-16T14:58:25.474319+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:45.513448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726936+0000","last_change":"2024-01-16T14:58:25.455406+0000","last_active":"2024-01-16T14:58:27.726936+0000","last_peered":"2024-01-16T14:58:27.726936+0000","last_clean":"2024-01-16T14:58:27.726936+0000","last_became_active":"2024-01-16T14:58:25.455154+0000","last_became_peered":"2024-01-16T14:58:25.455154+0000","last_unstale":"2024-01-16T14:58:27.726936+0000","last_undegraded":"2024-01-16T14:58:27.726936+0000","last_fullsized":"2024-01-16T14:58:27.726936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:49.965022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.470656+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:25.470656+0000","last_peered":"2024-01-16T14:58:25.470656+0000","last_clean":"2024-01-16T14:58:25.470656+0000","last_became_active":"2024-01-16T14:58:24.421626+0000","last_became_peered":"2024-01-16T14:58:24.421626+0000","last_unstale":"2024-01-16T14:58:25.470656+0000","last_undegraded":"2024-01-16T14:58:25.470656+0000","last_fullsized":"2024-01-16T14:58:25.470656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:06.171691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863332+0000","last_change":"2024-01-16T14:58:25.456831+0000","last_active":"2024-01-16T14:58:27.863332+0000","last_peered":"2024-01-16T14:58:27.863332+0000","last_clean":"2024-01-16T14:58:27.863332+0000","last_became_active":"2024-01-16T14:58:25.456558+0000","last_became_peered":"2024-01-16T14:58:25.456558+0000","last_unstale":"2024-01-16T14:58:27.863332+0000","last_undegraded":"2024-01-16T14:58:27.863332+0000","last_fullsized":"2024-01-16T14:58:27.863332+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:46.526212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426019+0000","last_change":"2024-01-16T14:58:24.413866+0000","last_active":"2024-01-16T14:58:25.426019+0000","last_peered":"2024-01-16T14:58:25.426019+0000","last_clean":"2024-01-16T14:58:25.426019+0000","last_became_active":"2024-01-16T14:58:24.413577+0000","last_became_peered":"2024-01-16T14:58:24.413577+0000","last_unstale":"2024-01-16T14:58:25.426019+0000","last_undegraded":"2024-01-16T14:58:25.426019+0000","last_fullsized":"2024-01-16T14:58:25.426019+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:34.799937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.862924+0000","last_change":"2024-01-16T14:58:25.452034+0000","last_active":"2024-01-16T14:58:27.862924+0000","last_peered":"2024-01-16T14:58:27.862924+0000","last_clean":"2024-01-16T14:58:27.862924+0000","last_became_active":"2024-01-16T14:58:25.451687+0000","last_became_peered":"2024-01-16T14:58:25.451687+0000","last_unstale":"2024-01-16T14:58:27.862924+0000","last_undegraded":"2024-01-16T14:58:27.862924+0000","last_fullsized":"2024-01-16T14:58:27.862924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:52.969008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.472079+0000","last_change":"2024-01-16T14:58:24.417730+0000","last_active":"2024-01-16T14:58:25.472079+0000","last_peered":"2024-01-16T14:58:25.472079+0000","last_clean":"2024-01-16T14:58:25.472079+0000","last_became_active":"2024-01-16T14:58:24.417446+0000","last_became_peered":"2024-01-16T14:58:24.417446+0000","last_unstale":"2024-01-16T14:58:25.472079+0000","last_undegraded":"2024-01-16T14:58:25.472079+0000","last_fullsized":"2024-01-16T14:58:25.472079+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:22.607082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863443+0000","last_change":"2024-01-16T14:58:25.453154+0000","last_active":"2024-01-16T14:58:27.863443+0000","last_peered":"2024-01-16T14:58:27.863443+0000","last_clean":"2024-01-16T14:58:27.863443+0000","last_became_active":"2024-01-16T14:58:25.452818+0000","last_became_peered":"2024-01-16T14:58:25.452818+0000","last_unstale":"2024-01-16T14:58:27.863443+0000","last_undegraded":"2024-01-16T14:58:27.863443+0000","last_fullsized":"2024-01-16T14:58:27.863443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:10.106904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.435646+0000","last_change":"2024-01-16T14:58:24.426266+0000","last_active":"2024-01-16T14:58:25.435646+0000","last_peered":"2024-01-16T14:58:25.435646+0000","last_clean":"2024-01-16T14:58:25.435646+0000","last_became_active":"2024-01-16T14:58:24.425944+0000","last_became_peered":"2024-01-16T14:58:24.425944+0000","last_unstale":"2024-01-16T14:58:25.435646+0000","last_undegraded":"2024-01-16T14:58:25.435646+0000","last_fullsized":"2024-01-16T14:58:25.435646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863304+0000","last_change":"2024-01-16T14:58:25.456158+0000","last_active":"2024-01-16T14:58:27.863304+0000","last_peered":"2024-01-16T14:58:27.863304+0000","last_clean":"2024-01-16T14:58:27.863304+0000","last_became_active":"2024-01-16T14:58:25.455879+0000","last_became_peered":"2024-01-16T14:58:25.455879+0000","last_unstale":"2024-01-16T14:58:27.863304+0000","last_undegraded":"2024-01-16T14:58:27.863304+0000","last_fullsized":"2024-01-16T14:58:27.863304+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:35.812702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426368+0000","last_change":"2024-01-16T14:58:24.416410+0000","last_active":"2024-01-16T14:58:25.426368+0000","last_peered":"2024-01-16T14:58:25.426368+0000","last_clean":"2024-01-16T14:58:25.426368+0000","last_became_active":"2024-01-16T14:58:24.416121+0000","last_became_peered":"2024-01-16T14:58:24.416121+0000","last_unstale":"2024-01-16T14:58:25.426368+0000","last_undegraded":"2024-01-16T14:58:25.426368+0000","last_fullsized":"2024-01-16T14:58:25.426368+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:28.900039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449256+0000","last_change":"2024-01-16T14:58:25.452830+0000","last_active":"2024-01-16T14:58:26.449256+0000","last_peered":"2024-01-16T14:58:26.449256+0000","last_clean":"2024-01-16T14:58:26.449256+0000","last_became_active":"2024-01-16T14:58:25.449825+0000","last_became_peered":"2024-01-16T14:58:25.449825+0000","last_unstale":"2024-01-16T14:58:26.449256+0000","last_undegraded":"2024-01-16T14:58:26.449256+0000","last_fullsized":"2024-01-16T14:58:26.449256+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:11.932103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.432600+0000","last_change":"2024-01-16T14:58:24.429805+0000","last_active":"2024-01-16T14:58:25.432600+0000","last_peered":"2024-01-16T14:58:25.432600+0000","last_clean":"2024-01-16T14:58:25.432600+0000","last_became_active":"2024-01-16T14:58:24.429510+0000","last_became_peered":"2024-01-16T14:58:24.429510+0000","last_unstale":"2024-01-16T14:58:25.432600+0000","last_undegraded":"2024-01-16T14:58:25.432600+0000","last_fullsized":"2024-01-16T14:58:25.432600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:39.606992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863109+0000","last_change":"2024-01-16T14:58:25.451255+0000","last_active":"2024-01-16T14:58:27.863109+0000","last_peered":"2024-01-16T14:58:27.863109+0000","last_clean":"2024-01-16T14:58:27.863109+0000","last_became_active":"2024-01-16T14:58:25.450936+0000","last_became_peered":"2024-01-16T14:58:25.450936+0000","last_unstale":"2024-01-16T14:58:27.863109+0000","last_undegraded":"2024-01-16T14:58:27.863109+0000","last_fullsized":"2024-01-16T14:58:27.863109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:09.352999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474590+0000","last_change":"2024-01-16T14:58:24.428471+0000","last_active":"2024-01-16T14:58:25.474590+0000","last_peered":"2024-01-16T14:58:25.474590+0000","last_clean":"2024-01-16T14:58:25.474590+0000","last_became_active":"2024-01-16T14:58:24.427946+0000","last_became_peered":"2024-01-16T14:58:24.427946+0000","last_unstale":"2024-01-16T14:58:25.474590+0000","last_undegraded":"2024-01-16T14:58:25.474590+0000","last_fullsized":"2024-01-16T14:58:25.474590+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454398+0000","last_change":"2024-01-16T14:58:25.466478+0000","last_active":"2024-01-16T14:58:26.454398+0000","last_peered":"2024-01-16T14:58:26.454398+0000","last_clean":"2024-01-16T14:58:26.454398+0000","last_became_active":"2024-01-16T14:58:25.466179+0000","last_became_peered":"2024-01-16T14:58:25.466179+0000","last_unstale":"2024-01-16T14:58:26.454398+0000","last_undegraded":"2024-01-16T14:58:26.454398+0000","last_fullsized":"2024-01-16T14:58:26.454398+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:51.577006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.421288+0000","last_change":"2024-01-16T14:58:24.420850+0000","last_active":"2024-01-16T14:58:25.421288+0000","last_peered":"2024-01-16T14:58:25.421288+0000","last_clean":"2024-01-16T14:58:25.421288+0000","last_became_active":"2024-01-16T14:58:24.420553+0000","last_became_peered":"2024-01-16T14:58:24.420553+0000","last_unstale":"2024-01-16T14:58:25.421288+0000","last_undegraded":"2024-01-16T14:58:25.421288+0000","last_fullsized":"2024-01-16T14:58:25.421288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:19.956837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528092+0000","last_change":"2024-01-16T14:58:25.440870+0000","last_active":"2024-01-16T14:58:26.528092+0000","last_peered":"2024-01-16T14:58:26.528092+0000","last_clean":"2024-01-16T14:58:26.528092+0000","last_became_active":"2024-01-16T14:58:25.440562+0000","last_became_peered":"2024-01-16T14:58:25.440562+0000","last_unstale":"2024-01-16T14:58:26.528092+0000","last_undegraded":"2024-01-16T14:58:26.528092+0000","last_fullsized":"2024-01-16T14:58:26.528092+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:52.505975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.435229+0000","last_change":"2024-01-16T14:58:24.423543+0000","last_active":"2024-01-16T14:58:25.435229+0000","last_peered":"2024-01-16T14:58:25.435229+0000","last_clean":"2024-01-16T14:58:25.435229+0000","last_became_active":"2024-01-16T14:58:24.420295+0000","last_became_peered":"2024-01-16T14:58:24.420295+0000","last_unstale":"2024-01-16T14:58:25.435229+0000","last_undegraded":"2024-01-16T14:58:25.435229+0000","last_fullsized":"2024-01-16T14:58:25.435229+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:30.952362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.231899+0000","last_change":"2024-01-16T14:58:25.449151+0000","last_active":"2024-01-16T14:58:28.231899+0000","last_peered":"2024-01-16T14:58:28.231899+0000","last_clean":"2024-01-16T14:58:28.231899+0000","last_became_active":"2024-01-16T14:58:25.448796+0000","last_became_peered":"2024-01-16T14:58:25.448796+0000","last_unstale":"2024-01-16T14:58:28.231899+0000","last_undegraded":"2024-01-16T14:58:28.231899+0000","last_fullsized":"2024-01-16T14:58:28.231899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:38.606880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.420062+0000","last_change":"2024-01-16T14:58:24.424952+0000","last_active":"2024-01-16T14:58:25.420062+0000","last_peered":"2024-01-16T14:58:25.420062+0000","last_clean":"2024-01-16T14:58:25.420062+0000","last_became_active":"2024-01-16T14:58:24.423308+0000","last_became_peered":"2024-01-16T14:58:24.423308+0000","last_unstale":"2024-01-16T14:58:25.420062+0000","last_undegraded":"2024-01-16T14:58:25.420062+0000","last_fullsized":"2024-01-16T14:58:25.420062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528278+0000","last_change":"2024-01-16T14:58:25.451666+0000","last_active":"2024-01-16T14:58:26.528278+0000","last_peered":"2024-01-16T14:58:26.528278+0000","last_clean":"2024-01-16T14:58:26.528278+0000","last_became_active":"2024-01-16T14:58:25.451394+0000","last_became_peered":"2024-01-16T14:58:25.451394+0000","last_unstale":"2024-01-16T14:58:26.528278+0000","last_undegraded":"2024-01-16T14:58:26.528278+0000","last_fullsized":"2024-01-16T14:58:26.528278+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:50.425785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.429562+0000","last_change":"2024-01-16T14:58:24.427373+0000","last_active":"2024-01-16T14:58:25.429562+0000","last_peered":"2024-01-16T14:58:25.429562+0000","last_clean":"2024-01-16T14:58:25.429562+0000","last_became_active":"2024-01-16T14:58:24.426937+0000","last_became_peered":"2024-01-16T14:58:24.426937+0000","last_unstale":"2024-01-16T14:58:25.429562+0000","last_undegraded":"2024-01-16T14:58:25.429562+0000","last_fullsized":"2024-01-16T14:58:25.429562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:10.718372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.088446+0000","last_change":"2024-01-16T14:58:25.449369+0000","last_active":"2024-01-16T14:58:28.088446+0000","last_peered":"2024-01-16T14:58:28.088446+0000","last_clean":"2024-01-16T14:58:28.088446+0000","last_became_active":"2024-01-16T14:58:25.449078+0000","last_became_peered":"2024-01-16T14:58:25.449078+0000","last_unstale":"2024-01-16T14:58:28.088446+0000","last_undegraded":"2024-01-16T14:58:28.088446+0000","last_fullsized":"2024-01-16T14:58:28.088446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:23.851919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426752+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:25.426752+0000","last_peered":"2024-01-16T14:58:25.426752+0000","last_clean":"2024-01-16T14:58:25.426752+0000","last_became_active":"2024-01-16T14:58:24.421586+0000","last_became_peered":"2024-01-16T14:58:24.421586+0000","last_unstale":"2024-01-16T14:58:25.426752+0000","last_undegraded":"2024-01-16T14:58:25.426752+0000","last_fullsized":"2024-01-16T14:58:25.426752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452369+0000","last_change":"2024-01-16T14:58:25.449615+0000","last_active":"2024-01-16T14:58:26.452369+0000","last_peered":"2024-01-16T14:58:26.452369+0000","last_clean":"2024-01-16T14:58:26.452369+0000","last_became_active":"2024-01-16T14:58:25.447805+0000","last_became_peered":"2024-01-16T14:58:25.447805+0000","last_unstale":"2024-01-16T14:58:26.452369+0000","last_undegraded":"2024-01-16T14:58:26.452369+0000","last_fullsized":"2024-01-16T14:58:26.452369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:20.425389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437045+0000","last_change":"2024-01-16T14:58:24.427165+0000","last_active":"2024-01-16T14:58:25.437045+0000","last_peered":"2024-01-16T14:58:25.437045+0000","last_clean":"2024-01-16T14:58:25.437045+0000","last_became_active":"2024-01-16T14:58:24.426927+0000","last_became_peered":"2024-01-16T14:58:24.426927+0000","last_unstale":"2024-01-16T14:58:25.437045+0000","last_undegraded":"2024-01-16T14:58:25.437045+0000","last_fullsized":"2024-01-16T14:58:25.437045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:43.919514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448298+0000","last_change":"2024-01-16T14:58:25.448033+0000","last_active":"2024-01-16T14:58:26.448298+0000","last_peered":"2024-01-16T14:58:26.448298+0000","last_clean":"2024-01-16T14:58:26.448298+0000","last_became_active":"2024-01-16T14:58:25.447748+0000","last_became_peered":"2024-01-16T14:58:25.447748+0000","last_unstale":"2024-01-16T14:58:26.448298+0000","last_undegraded":"2024-01-16T14:58:26.448298+0000","last_fullsized":"2024-01-16T14:58:26.448298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.436751+0000","last_change":"2024-01-16T14:58:24.425447+0000","last_active":"2024-01-16T14:58:25.436751+0000","last_peered":"2024-01-16T14:58:25.436751+0000","last_clean":"2024-01-16T14:58:25.436751+0000","last_became_active":"2024-01-16T14:58:24.425035+0000","last_became_peered":"2024-01-16T14:58:24.425035+0000","last_unstale":"2024-01-16T14:58:25.436751+0000","last_undegraded":"2024-01-16T14:58:25.436751+0000","last_fullsized":"2024-01-16T14:58:25.436751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:20.089624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":177,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.436147+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:58:25.436147+0000","last_peered":"2024-01-16T14:58:25.436147+0000","last_clean":"2024-01-16T14:58:25.436147+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:58:25.436147+0000","last_undegraded":"2024-01-16T14:58:25.436147+0000","last_fullsized":"2024-01-16T14:58:25.436147+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":991232,"data_stored":744900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1519999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":1.0089999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.256},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.90400000000000003}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.605}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.45,"15min":1.45},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.337}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.109,"5min":1.109,"15min":1.109},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"max":{"1min":3.3530000000000002,"5min":3.3530000000000002,"15min":3.3530000000000002},"last":1.3879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":3.258,"5min":3.258,"15min":3.258},"last":1.573},{"interface":"front","average":{"1min":1.617,"5min":1.617,"15min":1.617},"min":{"1min":1.139,"5min":1.139,"15min":1.139},"max":{"1min":3.101,"5min":3.101,"15min":3.101},"last":1.542}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":1.04},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":1.109}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.294},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.5900000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.415}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.244}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.95799999999999996}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.625},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.4570000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.5469999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":155114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.331},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.78000000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.64300000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.234}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.1910000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.038},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":2.9540000000000002},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.431}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.149},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"max":{"1min":3.2389999999999999,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":1.407}]}]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":154546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.65900000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.96299999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.7490000000000001,"5min":2.7490000000000001,"15min":2.7490000000000001},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":1.1799999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":1.163},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.522}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.079,"5min":1.079,"15min":1.079},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.038},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":1.091}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":3.1880000000000002,"5min":3.1880000000000002,"15min":3.1880000000000002},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.788,"5min":1.788,"15min":1.788},"min":{"1min":1.232,"5min":1.232,"15min":1.232},"max":{"1min":3.2290000000000001,"5min":3.2290000000000001,"15min":3.2290000000000001},"last":1.2829999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.80900000000000005}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.196},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.008,"5min":4.008,"15min":4.008},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.54300000000000004}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.745},{"interface":"front","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.498},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.0269999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.872},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5549999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.0389999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":154512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.1419999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.4279999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.55},{"interface":"front","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.8779999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.514}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.7210000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.3680000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.76700000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.89900000000000002}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":1.016}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.115},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.0860000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.649,"5min":2.649,"15min":2.649},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":1.319}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.7290000000000001,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.3480000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.377},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.5529999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":744810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.194,"5min":4.194,"15min":4.194},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.4299999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.2359999999999998,"5min":4.2359999999999998,"15min":4.2359999999999998},"last":1.147}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.026}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":4.5529999999999999,"5min":4.5529999999999999,"15min":4.5529999999999999},"last":1.339},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.089}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.4880000000000004,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.276},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.2250000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.375},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.524,"5min":4.524,"15min":4.524},"last":1.4470000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:58:30.847 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-16T14:58:31.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.111+0000 7f6c8b200640 1 Processor -- start 2024-01-16T14:58:31.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.112+0000 7f6c8b200640 1 -- start start 2024-01-16T14:58:31.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.113+0000 7f6c8b200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6c8c0d0150 0x7f6c8c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.113+0000 7f6c8b200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c0b2c70 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.113+0000 7f6c8b200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c0cf6d0 con 0x7f6c8c15d3c0 2024-01-16T14:58:31.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.113+0000 7f6c8b200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c0cf3f0 con 0x7f6c8c0d0150 2024-01-16T14:58:31.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.113+0000 7f6c89400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6c8c0d0150 0x7f6c8c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:31.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.113+0000 7f6c89400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6c8c0d0150 0x7f6c8c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53270/0 (socket says 172.21.15.141:53270) 2024-01-16T14:58:31.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.114+0000 7f6c89400640 1 -- 172.21.15.141:0/4270503502 learned_addr learned my addr 172.21.15.141:0/4270503502 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:31.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.114+0000 7f6c89400640 1 -- 172.21.15.141:0/4270503502 >> v1:172.21.15.188:6789/0 conn(0x7f6c8c15d3c0 legacy=0x7f6c8c0cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:31.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.114+0000 7f6c89400640 1 -- 172.21.15.141:0/4270503502 >> v1:172.21.15.141:6789/0 conn(0x7f6c8c0ceea0 legacy=0x7f6c8c0b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.114+0000 7f6c89400640 1 -- 172.21.15.141:0/4270503502 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c0b2620 con 0x7f6c8c0d0150 2024-01-16T14:58:31.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.115+0000 7f6c89400640 1 --2- 172.21.15.141:0/4270503502 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0d0150 0x7f6c8c0d1f20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009870 tx=0x7f6c7402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce64a0c32762c69 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:31.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.115+0000 7f6c88a00640 1 -- 172.21.15.141:0/4270503502 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74030850 con 0x7f6c8c0d0150 2024-01-16T14:58:31.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.115+0000 7f6c88a00640 1 -- 172.21.15.141:0/4270503502 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c740309b0 con 0x7f6c8c0d0150 2024-01-16T14:58:31.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.115+0000 7f6c88a00640 1 -- 172.21.15.141:0/4270503502 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74030d40 con 0x7f6c8c0d0150 2024-01-16T14:58:31.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.116+0000 7f6c8b200640 1 -- 172.21.15.141:0/4270503502 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0d0150 msgr2=0x7f6c8c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.116+0000 7f6c8b200640 1 --2- 172.21.15.141:0/4270503502 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0d0150 0x7f6c8c0d1f20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009870 tx=0x7f6c7402b990 comp rx=0 tx=0).stop 2024-01-16T14:58:31.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.116+0000 7f6c8b200640 1 -- 172.21.15.141:0/4270503502 shutdown_connections 2024-01-16T14:58:31.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.116+0000 7f6c8b200640 1 --2- 172.21.15.141:0/4270503502 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0d0150 0x7f6c8c0d1f20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.116+0000 7f6c8b200640 1 -- 172.21.15.141:0/4270503502 >> 172.21.15.141:0/4270503502 conn(0x7f6c8c0d3050 msgr2=0x7f6c8c0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:31.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.116+0000 7f6c8b200640 1 -- 172.21.15.141:0/4270503502 shutdown_connections 2024-01-16T14:58:31.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.116+0000 7f6c8b200640 1 -- 172.21.15.141:0/4270503502 wait complete. 2024-01-16T14:58:31.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.118+0000 7f6c8b200640 1 Processor -- start 2024-01-16T14:58:31.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.118+0000 7f6c8b200640 1 -- start start 2024-01-16T14:58:31.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c8b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0ceea0 0x7f6c8c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c8b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c8c0d0150 0x7f6c8c15ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c8b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c8c15d3c0 0x7f6c8c159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c8b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0b7470 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c8b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c15f790 con 0x7f6c8c15d3c0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c8b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c15fa70 con 0x7f6c8c0d0150 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c89e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0ceea0 0x7f6c8c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c89e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0ceea0 0x7f6c8c0ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53280/0 (socket says 172.21.15.141:53280) 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c89e00640 1 -- 172.21.15.141:0/1066148022 learned_addr learned my addr 172.21.15.141:0/1066148022 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c89e00640 1 -- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c8c0d0150 msgr2=0x7f6c8c15ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c89e00640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c8c0d0150 0x7f6c8c15ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c89e00640 1 -- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c8c15d3c0 msgr2=0x7f6c8c159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.119+0000 7f6c89e00640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c8c15d3c0 0x7f6c8c159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.120+0000 7f6c89e00640 1 -- 172.21.15.141:0/1066148022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740094d0 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.120+0000 7f6c89e00640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0ceea0 0x7f6c8c0ca3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00b3f0 tx=0x7f6c7c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.120+0000 7f6c82000640 1 -- 172.21.15.141:0/1066148022 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c011830 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.120+0000 7f6c82000640 1 -- 172.21.15.141:0/1066148022 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c011990 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.120+0000 7f6c82000640 1 -- 172.21.15.141:0/1066148022 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c011d40 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.120+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c0b6db0 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.120+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c1603b0 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.121+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c08ba80 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.122+0000 7f6c82000640 1 -- 172.21.15.141:0/1066148022 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6c7c01a400 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.123+0000 7f6c82000640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6c6406b2c0 0x7f6c6406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.123+0000 7f6c82000640 1 -- 172.21.15.141:0/1066148022 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c7c089960 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.124+0000 7f6c89400640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6c6406b2c0 0x7f6c6406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:31.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.125+0000 7f6c89400640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6c6406b2c0 0x7f6c6406d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c74036000 tx=0x7f6c7403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:31.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.127+0000 7f6c82000640 1 -- 172.21.15.141:0/1066148022 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c7c054860 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.385+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6c8c067450 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.386+0000 7f6c82000640 1 -- 172.21.15.141:0/1066148022 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1012 (secure 0 0 0) 0x7f6c7c015410 con 0x7f6c8c0ceea0 2024-01-16T14:58:31.367 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:31.368 INFO:teuthology.orchestra.run.smithi141.stdout:{"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":104,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":8,"num_up_osds":8,"osd_up_since":1705417019,"num_in_osds":8,"osd_in_since":1705417011,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":230023168,"bytes_avail":772864090112,"bytes_total":773094113280,"write_bytes_sec":2361,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":4762}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-01-16T14:58:10.458693+0000","services":{}},"progress_events":{}} 2024-01-16T14:58:31.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6c6406b2c0 msgr2=0x7f6c6406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6c6406b2c0 0x7f6c6406d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c74036000 tx=0x7f6c7403b040 comp rx=0 tx=0).stop 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0ceea0 msgr2=0x7f6c8c0ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0ceea0 0x7f6c8c0ca3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00b3f0 tx=0x7f6c7c00b8b0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 shutdown_connections 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6c6406b2c0 0x7f6c6406d780 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c8c15d3c0 0x7f6c8c159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c8c0d0150 0x7f6c8c15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 --2- 172.21.15.141:0/1066148022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c8c0ceea0 0x7f6c8c0ca3f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 >> 172.21.15.141:0/1066148022 conn(0x7f6c8c0d3050 msgr2=0x7f6c8c0d1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 shutdown_connections 2024-01-16T14:58:31.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.389+0000 7f6c8b200640 1 -- 172.21.15.141:0/1066148022 wait complete. 2024-01-16T14:58:31.390 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-16T14:58:31.390 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:58:31.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.618+0000 7f236c800640 1 Processor -- start 2024-01-16T14:58:31.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.618+0000 7f236c800640 1 -- start start 2024-01-16T14:58:31.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f236c800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f236415d590 0x7f23640cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f236c800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f23640d00b0 0x7f23640ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f236c800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23640b6750 con 0x7f23640ce6f0 2024-01-16T14:58:31.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f236c800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23640d0530 con 0x7f236415d590 2024-01-16T14:58:31.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f236c800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23640cf1f0 con 0x7f23640d00b0 2024-01-16T14:58:31.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f2369e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f236415d590 0x7f23640cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:31.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f2369e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f236415d590 0x7f23640cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53288/0 (socket says 172.21.15.141:53288) 2024-01-16T14:58:31.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.620+0000 7f2369e00640 1 -- 172.21.15.141:0/1708338484 learned_addr learned my addr 172.21.15.141:0/1708338484 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:31.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.621+0000 7f2369e00640 1 -- 172.21.15.141:0/1708338484 >> v2:172.21.15.188:3300/0 conn(0x7f23640d00b0 msgr2=0x7f23640ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:31.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.621+0000 7f2369e00640 1 --2- 172.21.15.141:0/1708338484 >> v2:172.21.15.188:3300/0 conn(0x7f23640d00b0 0x7f23640ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.621+0000 7f2369e00640 1 -- 172.21.15.141:0/1708338484 >> v1:172.21.15.188:6789/0 conn(0x7f23640ce6f0 legacy=0x7f23640d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:31.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.621+0000 7f2369e00640 1 -- 172.21.15.141:0/1708338484 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23640cef10 con 0x7f236415d590 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.622+0000 7f2369e00640 1 --2- 172.21.15.141:0/1708338484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f236415d590 0x7f23640cfb70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f236408d970 tx=0x7f235402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75ea69f50a23fd4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.622+0000 7f2368a00640 1 -- 172.21.15.141:0/1708338484 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354031870 con 0x7f236415d590 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.622+0000 7f2368a00640 1 -- 172.21.15.141:0/1708338484 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23540319d0 con 0x7f236415d590 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.622+0000 7f2368a00640 1 -- 172.21.15.141:0/1708338484 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354031d60 con 0x7f236415d590 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 -- 172.21.15.141:0/1708338484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f236415d590 msgr2=0x7f23640cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 --2- 172.21.15.141:0/1708338484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f236415d590 0x7f23640cfb70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f236408d970 tx=0x7f235402b910 comp rx=0 tx=0).stop 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 -- 172.21.15.141:0/1708338484 shutdown_connections 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 --2- 172.21.15.141:0/1708338484 >> v2:172.21.15.188:3300/0 conn(0x7f23640d00b0 0x7f23640ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 --2- 172.21.15.141:0/1708338484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f236415d590 0x7f23640cfb70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 -- 172.21.15.141:0/1708338484 >> 172.21.15.141:0/1708338484 conn(0x7f236409c540 msgr2=0x7f23640d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:31.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 -- 172.21.15.141:0/1708338484 shutdown_connections 2024-01-16T14:58:31.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.623+0000 7f236c800640 1 -- 172.21.15.141:0/1708338484 wait complete. 2024-01-16T14:58:31.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.625+0000 7f236c800640 1 Processor -- start 2024-01-16T14:58:31.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.625+0000 7f236c800640 1 -- start start 2024-01-16T14:58:31.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f236c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23640ce6f0 0x7f236409e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f236c800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23640d00b0 0x7f23641e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f236c800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236415d590 0x7f2364160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f236c800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23641608a0 con 0x7f23640d00b0 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f236c800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23641e6400 con 0x7f236415d590 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f236c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23641e66e0 con 0x7f23640ce6f0 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f2369400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23640d00b0 0x7f23641e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f2369400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23640d00b0 0x7f23641e5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53294/0 (socket says 172.21.15.141:53294) 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.626+0000 7f2369400640 1 -- 172.21.15.141:0/2383242940 learned_addr learned my addr 172.21.15.141:0/2383242940 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2369400640 1 -- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23640ce6f0 msgr2=0x7f236409e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2369400640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23640ce6f0 0x7f236409e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2369400640 1 -- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236415d590 msgr2=0x7f2364160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2369400640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236415d590 0x7f2364160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2369400640 1 -- 172.21.15.141:0/2383242940 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23540094d0 con 0x7f23640d00b0 2024-01-16T14:58:31.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2369400640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23640d00b0 0x7f23641e5d90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f234c00b4a0 tx=0x7f234c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:31.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2362000640 1 -- 172.21.15.141:0/2383242940 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c010610 con 0x7f23640d00b0 2024-01-16T14:58:31.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2362000640 1 -- 172.21.15.141:0/2383242940 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234c010770 con 0x7f23640d00b0 2024-01-16T14:58:31.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f2362000640 1 -- 172.21.15.141:0/2383242940 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c010b20 con 0x7f23640d00b0 2024-01-16T14:58:31.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.627+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23641e69c0 con 0x7f23640d00b0 2024-01-16T14:58:31.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.628+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23641e6f80 con 0x7f23640d00b0 2024-01-16T14:58:31.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.629+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23641e71a0 con 0x7f23640d00b0 2024-01-16T14:58:31.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.630+0000 7f2362000640 1 -- 172.21.15.141:0/2383242940 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f234c010c80 con 0x7f23640d00b0 2024-01-16T14:58:31.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.630+0000 7f2362000640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f233c06b2c0 0x7f233c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:31.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.631+0000 7f2362000640 1 -- 172.21.15.141:0/2383242940 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f234c0897e0 con 0x7f23640d00b0 2024-01-16T14:58:31.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.631+0000 7f2369e00640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f233c06b2c0 0x7f233c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:31.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.632+0000 7f2369e00640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f233c06b2c0 0x7f233c06d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2354036000 tx=0x7f2354030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:31.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.635+0000 7f2362000640 1 -- 172.21.15.141:0/2383242940 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f234c054640 con 0x7f23640d00b0 2024-01-16T14:58:31.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.821+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f236415d970 con 0x7f233c06b2c0 2024-01-16T14:58:31.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.843+0000 7f2362000640 1 -- 172.21.15.141:0/2383242940 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286529 (secure 0 0 0) 0x7f2354031c40 con 0x7f233c06b2c0 2024-01-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.848+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f233c06b2c0 msgr2=0x7f233c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.848+0000 7f236c800640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f233c06b2c0 0x7f233c06d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2354036000 tx=0x7f2354030000 comp rx=0 tx=0).stop 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.848+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23640d00b0 msgr2=0x7f23641e5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.848+0000 7f236c800640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23640d00b0 0x7f23641e5d90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f234c00b4a0 tx=0x7f234c00b960 comp rx=0 tx=0).stop 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 shutdown_connections 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f233c06b2c0 0x7f233c06d780 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236415d590 0x7f2364160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23640d00b0 0x7f23641e5d90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 --2- 172.21.15.141:0/2383242940 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23640ce6f0 0x7f236409e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 >> 172.21.15.141:0/2383242940 conn(0x7f236409c540 msgr2=0x7f236406c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 shutdown_connections 2024-01-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:31.849+0000 7f236c800640 1 -- 172.21.15.141:0/2383242940 wait complete. 2024-01-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-01-16T14:58:30.464293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224632,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750088,"statfs":{"total":773094113280,"available":772864090112,"internally_reserved":0,"allocated":4399104,"data_stored":2421092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.071246"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232198+0000","last_change":"2024-01-16T14:58:25.453436+0000","last_active":"2024-01-16T14:58:28.232198+0000","last_peered":"2024-01-16T14:58:28.232198+0000","last_clean":"2024-01-16T14:58:28.232198+0000","last_became_active":"2024-01-16T14:58:25.453170+0000","last_became_peered":"2024-01-16T14:58:25.453170+0000","last_unstale":"2024-01-16T14:58:28.232198+0000","last_undegraded":"2024-01-16T14:58:28.232198+0000","last_fullsized":"2024-01-16T14:58:28.232198+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:19.680397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089898+0000","last_change":"2024-01-16T14:58:25.450853+0000","last_active":"2024-01-16T14:58:28.089898+0000","last_peered":"2024-01-16T14:58:28.089898+0000","last_clean":"2024-01-16T14:58:28.089898+0000","last_became_active":"2024-01-16T14:58:25.447306+0000","last_became_peered":"2024-01-16T14:58:25.447306+0000","last_unstale":"2024-01-16T14:58:28.089898+0000","last_undegraded":"2024-01-16T14:58:28.089898+0000","last_fullsized":"2024-01-16T14:58:28.089898+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:58.177859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451614+0000","last_change":"2024-01-16T14:58:25.467441+0000","last_active":"2024-01-16T14:58:26.451614+0000","last_peered":"2024-01-16T14:58:26.451614+0000","last_clean":"2024-01-16T14:58:26.451614+0000","last_became_active":"2024-01-16T14:58:25.466836+0000","last_became_peered":"2024-01-16T14:58:25.466836+0000","last_unstale":"2024-01-16T14:58:26.451614+0000","last_undegraded":"2024-01-16T14:58:26.451614+0000","last_fullsized":"2024-01-16T14:58:26.451614+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:22.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726970+0000","last_change":"2024-01-16T14:58:25.459481+0000","last_active":"2024-01-16T14:58:27.726970+0000","last_peered":"2024-01-16T14:58:27.726970+0000","last_clean":"2024-01-16T14:58:27.726970+0000","last_became_active":"2024-01-16T14:58:25.459193+0000","last_became_peered":"2024-01-16T14:58:25.459193+0000","last_unstale":"2024-01-16T14:58:27.726970+0000","last_undegraded":"2024-01-16T14:58:27.726970+0000","last_fullsized":"2024-01-16T14:58:27.726970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.719820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.862896+0000","last_change":"2024-01-16T14:58:25.453879+0000","last_active":"2024-01-16T14:58:27.862896+0000","last_peered":"2024-01-16T14:58:27.862896+0000","last_clean":"2024-01-16T14:58:27.862896+0000","last_became_active":"2024-01-16T14:58:25.453454+0000","last_became_peered":"2024-01-16T14:58:25.453454+0000","last_unstale":"2024-01-16T14:58:27.862896+0000","last_undegraded":"2024-01-16T14:58:27.862896+0000","last_fullsized":"2024-01-16T14:58:27.862896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:25.362250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.088902+0000","last_change":"2024-01-16T14:58:25.454154+0000","last_active":"2024-01-16T14:58:28.088902+0000","last_peered":"2024-01-16T14:58:28.088902+0000","last_clean":"2024-01-16T14:58:28.088902+0000","last_became_active":"2024-01-16T14:58:25.453809+0000","last_became_peered":"2024-01-16T14:58:25.453809+0000","last_unstale":"2024-01-16T14:58:28.088902+0000","last_undegraded":"2024-01-16T14:58:28.088902+0000","last_fullsized":"2024-01-16T14:58:28.088902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:09.724971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448873+0000","last_change":"2024-01-16T14:58:25.449577+0000","last_active":"2024-01-16T14:58:26.448873+0000","last_peered":"2024-01-16T14:58:26.448873+0000","last_clean":"2024-01-16T14:58:26.448873+0000","last_became_active":"2024-01-16T14:58:25.449289+0000","last_became_peered":"2024-01-16T14:58:25.449289+0000","last_unstale":"2024-01-16T14:58:26.448873+0000","last_undegraded":"2024-01-16T14:58:26.448873+0000","last_fullsized":"2024-01-16T14:58:26.448873+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:22.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447326+0000","last_change":"2024-01-16T14:58:25.447662+0000","last_active":"2024-01-16T14:58:26.447326+0000","last_peered":"2024-01-16T14:58:26.447326+0000","last_clean":"2024-01-16T14:58:26.447326+0000","last_became_active":"2024-01-16T14:58:25.447359+0000","last_became_peered":"2024-01-16T14:58:25.447359+0000","last_unstale":"2024-01-16T14:58:26.447326+0000","last_undegraded":"2024-01-16T14:58:26.447326+0000","last_fullsized":"2024-01-16T14:58:26.447326+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.719820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454085+0000","last_change":"2024-01-16T14:58:25.445625+0000","last_active":"2024-01-16T14:58:26.454085+0000","last_peered":"2024-01-16T14:58:26.454085+0000","last_clean":"2024-01-16T14:58:26.454085+0000","last_became_active":"2024-01-16T14:58:25.445288+0000","last_became_peered":"2024-01-16T14:58:25.445288+0000","last_unstale":"2024-01-16T14:58:26.454085+0000","last_undegraded":"2024-01-16T14:58:26.454085+0000","last_fullsized":"2024-01-16T14:58:26.454085+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:48.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727416+0000","last_change":"2024-01-16T14:58:25.446611+0000","last_active":"2024-01-16T14:58:27.727416+0000","last_peered":"2024-01-16T14:58:27.727416+0000","last_clean":"2024-01-16T14:58:27.727416+0000","last_became_active":"2024-01-16T14:58:25.445802+0000","last_became_peered":"2024-01-16T14:58:25.445802+0000","last_unstale":"2024-01-16T14:58:27.727416+0000","last_undegraded":"2024-01-16T14:58:27.727416+0000","last_fullsized":"2024-01-16T14:58:27.727416+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448582+0000","last_change":"2024-01-16T14:58:25.457171+0000","last_active":"2024-01-16T14:58:26.448582+0000","last_peered":"2024-01-16T14:58:26.448582+0000","last_clean":"2024-01-16T14:58:26.448582+0000","last_became_active":"2024-01-16T14:58:25.456898+0000","last_became_peered":"2024-01-16T14:58:25.456898+0000","last_unstale":"2024-01-16T14:58:26.448582+0000","last_undegraded":"2024-01-16T14:58:26.448582+0000","last_fullsized":"2024-01-16T14:58:26.448582+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:11.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451319+0000","last_change":"2024-01-16T14:58:25.454493+0000","last_active":"2024-01-16T14:58:26.451319+0000","last_peered":"2024-01-16T14:58:26.451319+0000","last_clean":"2024-01-16T14:58:26.451319+0000","last_became_active":"2024-01-16T14:58:25.454215+0000","last_became_peered":"2024-01-16T14:58:25.454215+0000","last_unstale":"2024-01-16T14:58:26.451319+0000","last_undegraded":"2024-01-16T14:58:26.451319+0000","last_fullsized":"2024-01-16T14:58:26.451319+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:11.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863116+0000","last_change":"2024-01-16T14:58:25.447370+0000","last_active":"2024-01-16T14:58:27.863116+0000","last_peered":"2024-01-16T14:58:27.863116+0000","last_clean":"2024-01-16T14:58:27.863116+0000","last_became_active":"2024-01-16T14:58:25.447100+0000","last_became_peered":"2024-01-16T14:58:25.447100+0000","last_unstale":"2024-01-16T14:58:27.863116+0000","last_undegraded":"2024-01-16T14:58:27.863116+0000","last_fullsized":"2024-01-16T14:58:27.863116+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:40.264688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089314+0000","last_change":"2024-01-16T14:58:25.454014+0000","last_active":"2024-01-16T14:58:28.089314+0000","last_peered":"2024-01-16T14:58:28.089314+0000","last_clean":"2024-01-16T14:58:28.089314+0000","last_became_active":"2024-01-16T14:58:25.453284+0000","last_became_peered":"2024-01-16T14:58:25.453284+0000","last_unstale":"2024-01-16T14:58:28.089314+0000","last_undegraded":"2024-01-16T14:58:28.089314+0000","last_fullsized":"2024-01-16T14:58:28.089314+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:08.961898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451019+0000","last_change":"2024-01-16T14:58:25.450821+0000","last_active":"2024-01-16T14:58:26.451019+0000","last_peered":"2024-01-16T14:58:26.451019+0000","last_clean":"2024-01-16T14:58:26.451019+0000","last_became_active":"2024-01-16T14:58:25.450270+0000","last_became_peered":"2024-01-16T14:58:25.450270+0000","last_unstale":"2024-01-16T14:58:26.451019+0000","last_undegraded":"2024-01-16T14:58:26.451019+0000","last_fullsized":"2024-01-16T14:58:26.451019+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:37.439565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448105+0000","last_change":"2024-01-16T14:58:25.449064+0000","last_active":"2024-01-16T14:58:26.448105+0000","last_peered":"2024-01-16T14:58:26.448105+0000","last_clean":"2024-01-16T14:58:26.448105+0000","last_became_active":"2024-01-16T14:58:25.448777+0000","last_became_peered":"2024-01-16T14:58:25.448777+0000","last_unstale":"2024-01-16T14:58:26.448105+0000","last_undegraded":"2024-01-16T14:58:26.448105+0000","last_fullsized":"2024-01-16T14:58:26.448105+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449556+0000","last_change":"2024-01-16T14:58:25.452550+0000","last_active":"2024-01-16T14:58:26.449556+0000","last_peered":"2024-01-16T14:58:26.449556+0000","last_clean":"2024-01-16T14:58:26.449556+0000","last_became_active":"2024-01-16T14:58:25.452267+0000","last_became_peered":"2024-01-16T14:58:25.452267+0000","last_unstale":"2024-01-16T14:58:26.449556+0000","last_undegraded":"2024-01-16T14:58:26.449556+0000","last_fullsized":"2024-01-16T14:58:26.449556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:51.577006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089919+0000","last_change":"2024-01-16T14:58:25.447596+0000","last_active":"2024-01-16T14:58:28.089919+0000","last_peered":"2024-01-16T14:58:28.089919+0000","last_clean":"2024-01-16T14:58:28.089919+0000","last_became_active":"2024-01-16T14:58:25.446849+0000","last_became_peered":"2024-01-16T14:58:25.446849+0000","last_unstale":"2024-01-16T14:58:28.089919+0000","last_undegraded":"2024-01-16T14:58:28.089919+0000","last_fullsized":"2024-01-16T14:58:28.089919+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:25:27.806945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089503+0000","last_change":"2024-01-16T14:58:25.452291+0000","last_active":"2024-01-16T14:58:28.089503+0000","last_peered":"2024-01-16T14:58:28.089503+0000","last_clean":"2024-01-16T14:58:28.089503+0000","last_became_active":"2024-01-16T14:58:25.452017+0000","last_became_peered":"2024-01-16T14:58:25.452017+0000","last_unstale":"2024-01-16T14:58:28.089503+0000","last_undegraded":"2024-01-16T14:58:28.089503+0000","last_fullsized":"2024-01-16T14:58:28.089503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:31.087496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863458+0000","last_change":"2024-01-16T14:58:25.451264+0000","last_active":"2024-01-16T14:58:27.863458+0000","last_peered":"2024-01-16T14:58:27.863458+0000","last_clean":"2024-01-16T14:58:27.863458+0000","last_became_active":"2024-01-16T14:58:25.450973+0000","last_became_peered":"2024-01-16T14:58:25.450973+0000","last_unstale":"2024-01-16T14:58:27.863458+0000","last_undegraded":"2024-01-16T14:58:27.863458+0000","last_fullsized":"2024-01-16T14:58:27.863458+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:29.912804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452736+0000","last_change":"2024-01-16T14:58:25.445306+0000","last_active":"2024-01-16T14:58:26.452736+0000","last_peered":"2024-01-16T14:58:26.452736+0000","last_clean":"2024-01-16T14:58:26.452736+0000","last_became_active":"2024-01-16T14:58:25.442633+0000","last_became_peered":"2024-01-16T14:58:25.442633+0000","last_unstale":"2024-01-16T14:58:26.452736+0000","last_undegraded":"2024-01-16T14:58:26.452736+0000","last_fullsized":"2024-01-16T14:58:26.452736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:11.731137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452224+0000","last_change":"2024-01-16T14:58:25.448740+0000","last_active":"2024-01-16T14:58:26.452224+0000","last_peered":"2024-01-16T14:58:26.452224+0000","last_clean":"2024-01-16T14:58:26.452224+0000","last_became_active":"2024-01-16T14:58:25.448233+0000","last_became_peered":"2024-01-16T14:58:25.448233+0000","last_unstale":"2024-01-16T14:58:26.452224+0000","last_undegraded":"2024-01-16T14:58:26.452224+0000","last_fullsized":"2024-01-16T14:58:26.452224+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:36.906207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.527930+0000","last_change":"2024-01-16T14:58:25.445600+0000","last_active":"2024-01-16T14:58:26.527930+0000","last_peered":"2024-01-16T14:58:26.527930+0000","last_clean":"2024-01-16T14:58:26.527930+0000","last_became_active":"2024-01-16T14:58:25.445302+0000","last_became_peered":"2024-01-16T14:58:25.445302+0000","last_unstale":"2024-01-16T14:58:26.527930+0000","last_undegraded":"2024-01-16T14:58:26.527930+0000","last_fullsized":"2024-01-16T14:58:26.527930+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:45.333764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.450450+0000","last_change":"2024-01-16T14:58:25.452393+0000","last_active":"2024-01-16T14:58:26.450450+0000","last_peered":"2024-01-16T14:58:26.450450+0000","last_clean":"2024-01-16T14:58:26.450450+0000","last_became_active":"2024-01-16T14:58:25.451586+0000","last_became_peered":"2024-01-16T14:58:25.451586+0000","last_unstale":"2024-01-16T14:58:26.450450+0000","last_undegraded":"2024-01-16T14:58:26.450450+0000","last_fullsized":"2024-01-16T14:58:26.450450+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727298+0000","last_change":"2024-01-16T14:58:25.448882+0000","last_active":"2024-01-16T14:58:27.727298+0000","last_peered":"2024-01-16T14:58:27.727298+0000","last_clean":"2024-01-16T14:58:27.727298+0000","last_became_active":"2024-01-16T14:58:25.448300+0000","last_became_peered":"2024-01-16T14:58:25.448300+0000","last_unstale":"2024-01-16T14:58:27.727298+0000","last_undegraded":"2024-01-16T14:58:27.727298+0000","last_fullsized":"2024-01-16T14:58:27.727298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.090087+0000","last_change":"2024-01-16T14:58:25.458035+0000","last_active":"2024-01-16T14:58:28.090087+0000","last_peered":"2024-01-16T14:58:28.090087+0000","last_clean":"2024-01-16T14:58:28.090087+0000","last_became_active":"2024-01-16T14:58:25.457804+0000","last_became_peered":"2024-01-16T14:58:25.457804+0000","last_unstale":"2024-01-16T14:58:28.090087+0000","last_undegraded":"2024-01-16T14:58:28.090087+0000","last_fullsized":"2024-01-16T14:58:28.090087+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:39.016885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727478+0000","last_change":"2024-01-16T14:58:25.465239+0000","last_active":"2024-01-16T14:58:27.727478+0000","last_peered":"2024-01-16T14:58:27.727478+0000","last_clean":"2024-01-16T14:58:27.727478+0000","last_became_active":"2024-01-16T14:58:25.462123+0000","last_became_peered":"2024-01-16T14:58:25.462123+0000","last_unstale":"2024-01-16T14:58:27.727478+0000","last_undegraded":"2024-01-16T14:58:27.727478+0000","last_fullsized":"2024-01-16T14:58:27.727478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451320+0000","last_change":"2024-01-16T14:58:25.457821+0000","last_active":"2024-01-16T14:58:26.451320+0000","last_peered":"2024-01-16T14:58:26.451320+0000","last_clean":"2024-01-16T14:58:26.451320+0000","last_became_active":"2024-01-16T14:58:25.457526+0000","last_became_peered":"2024-01-16T14:58:25.457526+0000","last_unstale":"2024-01-16T14:58:26.451320+0000","last_undegraded":"2024-01-16T14:58:26.451320+0000","last_fullsized":"2024-01-16T14:58:26.451320+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452858+0000","last_change":"2024-01-16T14:58:25.445985+0000","last_active":"2024-01-16T14:58:26.452858+0000","last_peered":"2024-01-16T14:58:26.452858+0000","last_clean":"2024-01-16T14:58:26.452858+0000","last_became_active":"2024-01-16T14:58:25.443290+0000","last_became_peered":"2024-01-16T14:58:25.443290+0000","last_unstale":"2024-01-16T14:58:26.452858+0000","last_undegraded":"2024-01-16T14:58:26.452858+0000","last_fullsized":"2024-01-16T14:58:26.452858+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:40.619757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449725+0000","last_change":"2024-01-16T14:58:25.448385+0000","last_active":"2024-01-16T14:58:26.449725+0000","last_peered":"2024-01-16T14:58:26.449725+0000","last_clean":"2024-01-16T14:58:26.449725+0000","last_became_active":"2024-01-16T14:58:25.448118+0000","last_became_peered":"2024-01-16T14:58:25.448118+0000","last_unstale":"2024-01-16T14:58:26.449725+0000","last_undegraded":"2024-01-16T14:58:26.449725+0000","last_fullsized":"2024-01-16T14:58:26.449725+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:22.409645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.231762+0000","last_change":"2024-01-16T14:58:25.454801+0000","last_active":"2024-01-16T14:58:28.231762+0000","last_peered":"2024-01-16T14:58:28.231762+0000","last_clean":"2024-01-16T14:58:28.231762+0000","last_became_active":"2024-01-16T14:58:25.454281+0000","last_became_peered":"2024-01-16T14:58:25.454281+0000","last_unstale":"2024-01-16T14:58:28.231762+0000","last_undegraded":"2024-01-16T14:58:28.231762+0000","last_fullsized":"2024-01-16T14:58:28.231762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:41.881567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447551+0000","last_change":"2024-01-16T14:58:25.458893+0000","last_active":"2024-01-16T14:58:26.447551+0000","last_peered":"2024-01-16T14:58:26.447551+0000","last_clean":"2024-01-16T14:58:26.447551+0000","last_became_active":"2024-01-16T14:58:25.458300+0000","last_became_peered":"2024-01-16T14:58:25.458300+0000","last_unstale":"2024-01-16T14:58:26.447551+0000","last_undegraded":"2024-01-16T14:58:26.447551+0000","last_fullsized":"2024-01-16T14:58:26.447551+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528093+0000","last_change":"2024-01-16T14:58:25.448792+0000","last_active":"2024-01-16T14:58:26.528093+0000","last_peered":"2024-01-16T14:58:26.528093+0000","last_clean":"2024-01-16T14:58:26.528093+0000","last_became_active":"2024-01-16T14:58:25.448490+0000","last_became_peered":"2024-01-16T14:58:25.448490+0000","last_unstale":"2024-01-16T14:58:26.528093+0000","last_undegraded":"2024-01-16T14:58:26.528093+0000","last_fullsized":"2024-01-16T14:58:26.528093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:45.048906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474893+0000","last_change":"2024-01-16T14:58:24.428745+0000","last_active":"2024-01-16T14:58:25.474893+0000","last_peered":"2024-01-16T14:58:25.474893+0000","last_clean":"2024-01-16T14:58:25.474893+0000","last_became_active":"2024-01-16T14:58:24.427170+0000","last_became_peered":"2024-01-16T14:58:24.427170+0000","last_unstale":"2024-01-16T14:58:25.474893+0000","last_undegraded":"2024-01-16T14:58:25.474893+0000","last_fullsized":"2024-01-16T14:58:25.474893+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:43.995208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232212+0000","last_change":"2024-01-16T14:58:25.455325+0000","last_active":"2024-01-16T14:58:28.232212+0000","last_peered":"2024-01-16T14:58:28.232212+0000","last_clean":"2024-01-16T14:58:28.232212+0000","last_became_active":"2024-01-16T14:58:25.454992+0000","last_became_peered":"2024-01-16T14:58:25.454992+0000","last_unstale":"2024-01-16T14:58:28.232212+0000","last_undegraded":"2024-01-16T14:58:28.232212+0000","last_fullsized":"2024-01-16T14:58:28.232212+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:36.367808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.469108+0000","last_change":"2024-01-16T14:58:24.429636+0000","last_active":"2024-01-16T14:58:25.469108+0000","last_peered":"2024-01-16T14:58:25.469108+0000","last_clean":"2024-01-16T14:58:25.469108+0000","last_became_active":"2024-01-16T14:58:24.428619+0000","last_became_peered":"2024-01-16T14:58:24.428619+0000","last_unstale":"2024-01-16T14:58:25.469108+0000","last_undegraded":"2024-01-16T14:58:25.469108+0000","last_fullsized":"2024-01-16T14:58:25.469108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:40.868803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.453290+0000","last_change":"2024-01-16T14:58:25.450473+0000","last_active":"2024-01-16T14:58:26.453290+0000","last_peered":"2024-01-16T14:58:26.453290+0000","last_clean":"2024-01-16T14:58:26.453290+0000","last_became_active":"2024-01-16T14:58:25.450169+0000","last_became_peered":"2024-01-16T14:58:25.450169+0000","last_unstale":"2024-01-16T14:58:26.453290+0000","last_undegraded":"2024-01-16T14:58:26.453290+0000","last_fullsized":"2024-01-16T14:58:26.453290+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:43.043403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474307+0000","last_change":"2024-01-16T14:58:24.421162+0000","last_active":"2024-01-16T14:58:25.474307+0000","last_peered":"2024-01-16T14:58:25.474307+0000","last_clean":"2024-01-16T14:58:25.474307+0000","last_became_active":"2024-01-16T14:58:24.420858+0000","last_became_peered":"2024-01-16T14:58:24.420858+0000","last_unstale":"2024-01-16T14:58:25.474307+0000","last_undegraded":"2024-01-16T14:58:25.474307+0000","last_fullsized":"2024-01-16T14:58:25.474307+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:48.834908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727604+0000","last_change":"2024-01-16T14:58:25.465488+0000","last_active":"2024-01-16T14:58:27.727604+0000","last_peered":"2024-01-16T14:58:27.727604+0000","last_clean":"2024-01-16T14:58:27.727604+0000","last_became_active":"2024-01-16T14:58:25.460624+0000","last_became_peered":"2024-01-16T14:58:25.460624+0000","last_unstale":"2024-01-16T14:58:27.727604+0000","last_undegraded":"2024-01-16T14:58:27.727604+0000","last_fullsized":"2024-01-16T14:58:27.727604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.473676+0000","last_change":"2024-01-16T14:58:24.426642+0000","last_active":"2024-01-16T14:58:25.473676+0000","last_peered":"2024-01-16T14:58:25.473676+0000","last_clean":"2024-01-16T14:58:25.473676+0000","last_became_active":"2024-01-16T14:58:24.426379+0000","last_became_peered":"2024-01-16T14:58:24.426379+0000","last_unstale":"2024-01-16T14:58:25.473676+0000","last_undegraded":"2024-01-16T14:58:25.473676+0000","last_fullsized":"2024-01-16T14:58:25.473676+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:29:09.809503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449031+0000","last_change":"2024-01-16T14:58:25.458055+0000","last_active":"2024-01-16T14:58:26.449031+0000","last_peered":"2024-01-16T14:58:26.449031+0000","last_clean":"2024-01-16T14:58:26.449031+0000","last_became_active":"2024-01-16T14:58:25.457556+0000","last_became_peered":"2024-01-16T14:58:25.457556+0000","last_unstale":"2024-01-16T14:58:26.449031+0000","last_undegraded":"2024-01-16T14:58:26.449031+0000","last_fullsized":"2024-01-16T14:58:26.449031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:37.439565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.471816+0000","last_change":"2024-01-16T14:58:24.419926+0000","last_active":"2024-01-16T14:58:25.471816+0000","last_peered":"2024-01-16T14:58:25.471816+0000","last_clean":"2024-01-16T14:58:25.471816+0000","last_became_active":"2024-01-16T14:58:24.419550+0000","last_became_peered":"2024-01-16T14:58:24.419550+0000","last_unstale":"2024-01-16T14:58:25.471816+0000","last_undegraded":"2024-01-16T14:58:25.471816+0000","last_fullsized":"2024-01-16T14:58:25.471816+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:09.094139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451316+0000","last_change":"2024-01-16T14:58:25.453757+0000","last_active":"2024-01-16T14:58:26.451316+0000","last_peered":"2024-01-16T14:58:26.451316+0000","last_clean":"2024-01-16T14:58:26.451316+0000","last_became_active":"2024-01-16T14:58:25.453492+0000","last_became_peered":"2024-01-16T14:58:25.453492+0000","last_unstale":"2024-01-16T14:58:26.451316+0000","last_undegraded":"2024-01-16T14:58:26.451316+0000","last_fullsized":"2024-01-16T14:58:26.451316+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.471378+0000","last_change":"2024-01-16T14:58:24.420225+0000","last_active":"2024-01-16T14:58:25.471378+0000","last_peered":"2024-01-16T14:58:25.471378+0000","last_clean":"2024-01-16T14:58:25.471378+0000","last_became_active":"2024-01-16T14:58:24.419237+0000","last_became_peered":"2024-01-16T14:58:24.419237+0000","last_unstale":"2024-01-16T14:58:25.471378+0000","last_undegraded":"2024-01-16T14:58:25.471378+0000","last_fullsized":"2024-01-16T14:58:25.471378+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:21.201884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451519+0000","last_change":"2024-01-16T14:58:25.451171+0000","last_active":"2024-01-16T14:58:26.451519+0000","last_peered":"2024-01-16T14:58:26.451519+0000","last_clean":"2024-01-16T14:58:26.451519+0000","last_became_active":"2024-01-16T14:58:25.450320+0000","last_became_peered":"2024-01-16T14:58:25.450320+0000","last_unstale":"2024-01-16T14:58:26.451519+0000","last_undegraded":"2024-01-16T14:58:26.451519+0000","last_fullsized":"2024-01-16T14:58:26.451519+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:57:49.471035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437180+0000","last_change":"2024-01-16T14:58:24.416771+0000","last_active":"2024-01-16T14:58:25.437180+0000","last_peered":"2024-01-16T14:58:25.437180+0000","last_clean":"2024-01-16T14:58:25.437180+0000","last_became_active":"2024-01-16T14:58:24.416479+0000","last_became_peered":"2024-01-16T14:58:24.416479+0000","last_unstale":"2024-01-16T14:58:25.437180+0000","last_undegraded":"2024-01-16T14:58:25.437180+0000","last_fullsized":"2024-01-16T14:58:25.437180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:15.420130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726309+0000","last_change":"2024-01-16T14:58:25.446441+0000","last_active":"2024-01-16T14:58:27.726309+0000","last_peered":"2024-01-16T14:58:27.726309+0000","last_clean":"2024-01-16T14:58:27.726309+0000","last_became_active":"2024-01-16T14:58:25.446153+0000","last_became_peered":"2024-01-16T14:58:25.446153+0000","last_unstale":"2024-01-16T14:58:27.726309+0000","last_undegraded":"2024-01-16T14:58:27.726309+0000","last_fullsized":"2024-01-16T14:58:27.726309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:59.978419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.424174+0000","last_change":"2024-01-16T14:58:24.426653+0000","last_active":"2024-01-16T14:58:25.424174+0000","last_peered":"2024-01-16T14:58:25.424174+0000","last_clean":"2024-01-16T14:58:25.424174+0000","last_became_active":"2024-01-16T14:58:24.421492+0000","last_became_peered":"2024-01-16T14:58:24.421492+0000","last_unstale":"2024-01-16T14:58:25.424174+0000","last_undegraded":"2024-01-16T14:58:25.424174+0000","last_fullsized":"2024-01-16T14:58:25.424174+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:51.956244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454109+0000","last_change":"2024-01-16T14:58:25.455256+0000","last_active":"2024-01-16T14:58:26.454109+0000","last_peered":"2024-01-16T14:58:26.454109+0000","last_clean":"2024-01-16T14:58:26.454109+0000","last_became_active":"2024-01-16T14:58:25.454696+0000","last_became_peered":"2024-01-16T14:58:25.454696+0000","last_unstale":"2024-01-16T14:58:26.454109+0000","last_undegraded":"2024-01-16T14:58:26.454109+0000","last_fullsized":"2024-01-16T14:58:26.454109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:11.932103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.443769+0000","last_change":"2024-01-16T14:58:24.426750+0000","last_active":"2024-01-16T14:58:25.443769+0000","last_peered":"2024-01-16T14:58:25.443769+0000","last_clean":"2024-01-16T14:58:25.443769+0000","last_became_active":"2024-01-16T14:58:24.426534+0000","last_became_peered":"2024-01-16T14:58:24.426534+0000","last_unstale":"2024-01-16T14:58:25.443769+0000","last_undegraded":"2024-01-16T14:58:25.443769+0000","last_fullsized":"2024-01-16T14:58:25.443769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:32.501031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089695+0000","last_change":"2024-01-16T14:58:25.471298+0000","last_active":"2024-01-16T14:58:28.089695+0000","last_peered":"2024-01-16T14:58:28.089695+0000","last_clean":"2024-01-16T14:58:28.089695+0000","last_became_active":"2024-01-16T14:58:25.470921+0000","last_became_peered":"2024-01-16T14:58:25.470921+0000","last_unstale":"2024-01-16T14:58:28.089695+0000","last_undegraded":"2024-01-16T14:58:28.089695+0000","last_fullsized":"2024-01-16T14:58:28.089695+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:18.391832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.421251+0000","last_change":"2024-01-16T14:58:24.414545+0000","last_active":"2024-01-16T14:58:25.421251+0000","last_peered":"2024-01-16T14:58:25.421251+0000","last_clean":"2024-01-16T14:58:25.421251+0000","last_became_active":"2024-01-16T14:58:24.412623+0000","last_became_peered":"2024-01-16T14:58:24.412623+0000","last_unstale":"2024-01-16T14:58:25.421251+0000","last_undegraded":"2024-01-16T14:58:25.421251+0000","last_fullsized":"2024-01-16T14:58:25.421251+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:37.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089315+0000","last_change":"2024-01-16T14:58:25.456962+0000","last_active":"2024-01-16T14:58:28.089315+0000","last_peered":"2024-01-16T14:58:28.089315+0000","last_clean":"2024-01-16T14:58:28.089315+0000","last_became_active":"2024-01-16T14:58:25.456712+0000","last_became_peered":"2024-01-16T14:58:25.456712+0000","last_unstale":"2024-01-16T14:58:28.089315+0000","last_undegraded":"2024-01-16T14:58:28.089315+0000","last_fullsized":"2024-01-16T14:58:28.089315+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:10.429909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.473767+0000","last_change":"2024-01-16T14:58:24.428533+0000","last_active":"2024-01-16T14:58:25.473767+0000","last_peered":"2024-01-16T14:58:25.473767+0000","last_clean":"2024-01-16T14:58:25.473767+0000","last_became_active":"2024-01-16T14:58:24.428239+0000","last_became_peered":"2024-01-16T14:58:24.428239+0000","last_unstale":"2024-01-16T14:58:25.473767+0000","last_undegraded":"2024-01-16T14:58:25.473767+0000","last_fullsized":"2024-01-16T14:58:25.473767+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:32.544123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451239+0000","last_change":"2024-01-16T14:58:25.467675+0000","last_active":"2024-01-16T14:58:26.451239+0000","last_peered":"2024-01-16T14:58:26.451239+0000","last_clean":"2024-01-16T14:58:26.451239+0000","last_became_active":"2024-01-16T14:58:25.467220+0000","last_became_peered":"2024-01-16T14:58:25.467220+0000","last_unstale":"2024-01-16T14:58:26.451239+0000","last_undegraded":"2024-01-16T14:58:26.451239+0000","last_fullsized":"2024-01-16T14:58:26.451239+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.477630+0000","last_change":"2024-01-16T14:58:24.426130+0000","last_active":"2024-01-16T14:58:25.477630+0000","last_peered":"2024-01-16T14:58:25.477630+0000","last_clean":"2024-01-16T14:58:25.477630+0000","last_became_active":"2024-01-16T14:58:24.425829+0000","last_became_peered":"2024-01-16T14:58:24.425829+0000","last_unstale":"2024-01-16T14:58:25.477630+0000","last_undegraded":"2024-01-16T14:58:25.477630+0000","last_fullsized":"2024-01-16T14:58:25.477630+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:11.348392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726576+0000","last_change":"2024-01-16T14:58:25.446712+0000","last_active":"2024-01-16T14:58:27.726576+0000","last_peered":"2024-01-16T14:58:27.726576+0000","last_clean":"2024-01-16T14:58:27.726576+0000","last_became_active":"2024-01-16T14:58:25.446349+0000","last_became_peered":"2024-01-16T14:58:25.446349+0000","last_unstale":"2024-01-16T14:58:27.726576+0000","last_undegraded":"2024-01-16T14:58:27.726576+0000","last_fullsized":"2024-01-16T14:58:27.726576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:33.556888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474223+0000","last_change":"2024-01-16T14:58:24.423758+0000","last_active":"2024-01-16T14:58:25.474223+0000","last_peered":"2024-01-16T14:58:25.474223+0000","last_clean":"2024-01-16T14:58:25.474223+0000","last_became_active":"2024-01-16T14:58:24.423458+0000","last_became_peered":"2024-01-16T14:58:24.423458+0000","last_unstale":"2024-01-16T14:58:25.474223+0000","last_undegraded":"2024-01-16T14:58:25.474223+0000","last_fullsized":"2024-01-16T14:58:25.474223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:56.142999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089168+0000","last_change":"2024-01-16T14:58:25.447562+0000","last_active":"2024-01-16T14:58:28.089168+0000","last_peered":"2024-01-16T14:58:28.089168+0000","last_clean":"2024-01-16T14:58:28.089168+0000","last_became_active":"2024-01-16T14:58:25.447243+0000","last_became_peered":"2024-01-16T14:58:25.447243+0000","last_unstale":"2024-01-16T14:58:28.089168+0000","last_undegraded":"2024-01-16T14:58:28.089168+0000","last_fullsized":"2024-01-16T14:58:28.089168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.429141+0000","last_change":"2024-01-16T14:58:24.427569+0000","last_active":"2024-01-16T14:58:25.429141+0000","last_peered":"2024-01-16T14:58:25.429141+0000","last_clean":"2024-01-16T14:58:25.429141+0000","last_became_active":"2024-01-16T14:58:24.427309+0000","last_became_peered":"2024-01-16T14:58:24.427309+0000","last_unstale":"2024-01-16T14:58:25.429141+0000","last_undegraded":"2024-01-16T14:58:25.429141+0000","last_fullsized":"2024-01-16T14:58:25.429141+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:19.412624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447681+0000","last_change":"2024-01-16T14:58:25.458315+0000","last_active":"2024-01-16T14:58:26.447681+0000","last_peered":"2024-01-16T14:58:26.447681+0000","last_clean":"2024-01-16T14:58:26.447681+0000","last_became_active":"2024-01-16T14:58:25.458030+0000","last_became_peered":"2024-01-16T14:58:25.458030+0000","last_unstale":"2024-01-16T14:58:26.447681+0000","last_undegraded":"2024-01-16T14:58:26.447681+0000","last_fullsized":"2024-01-16T14:58:26.447681+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437070+0000","last_change":"2024-01-16T14:58:24.427259+0000","last_active":"2024-01-16T14:58:25.437070+0000","last_peered":"2024-01-16T14:58:25.437070+0000","last_clean":"2024-01-16T14:58:25.437070+0000","last_became_active":"2024-01-16T14:58:24.426959+0000","last_became_peered":"2024-01-16T14:58:24.426959+0000","last_unstale":"2024-01-16T14:58:25.437070+0000","last_undegraded":"2024-01-16T14:58:25.437070+0000","last_fullsized":"2024-01-16T14:58:25.437070+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:10.096980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232161+0000","last_change":"2024-01-16T14:58:25.460308+0000","last_active":"2024-01-16T14:58:28.232161+0000","last_peered":"2024-01-16T14:58:28.232161+0000","last_clean":"2024-01-16T14:58:28.232161+0000","last_became_active":"2024-01-16T14:58:25.458176+0000","last_became_peered":"2024-01-16T14:58:25.458176+0000","last_unstale":"2024-01-16T14:58:28.232161+0000","last_undegraded":"2024-01-16T14:58:28.232161+0000","last_fullsized":"2024-01-16T14:58:28.232161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:20.969602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.475331+0000","last_change":"2024-01-16T14:58:24.416038+0000","last_active":"2024-01-16T14:58:25.475331+0000","last_peered":"2024-01-16T14:58:25.475331+0000","last_clean":"2024-01-16T14:58:25.475331+0000","last_became_active":"2024-01-16T14:58:24.415718+0000","last_became_peered":"2024-01-16T14:58:24.415718+0000","last_unstale":"2024-01-16T14:58:25.475331+0000","last_undegraded":"2024-01-16T14:58:25.475331+0000","last_fullsized":"2024-01-16T14:58:25.475331+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:03.937440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.453721+0000","last_change":"2024-01-16T14:58:25.451447+0000","last_active":"2024-01-16T14:58:26.453721+0000","last_peered":"2024-01-16T14:58:26.453721+0000","last_clean":"2024-01-16T14:58:26.453721+0000","last_became_active":"2024-01-16T14:58:25.446471+0000","last_became_peered":"2024-01-16T14:58:25.446471+0000","last_unstale":"2024-01-16T14:58:26.453721+0000","last_undegraded":"2024-01-16T14:58:26.453721+0000","last_fullsized":"2024-01-16T14:58:26.453721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:31.965127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.476243+0000","last_change":"2024-01-16T14:58:24.424464+0000","last_active":"2024-01-16T14:58:25.476243+0000","last_peered":"2024-01-16T14:58:25.476243+0000","last_clean":"2024-01-16T14:58:25.476243+0000","last_became_active":"2024-01-16T14:58:24.424200+0000","last_became_peered":"2024-01-16T14:58:24.424200+0000","last_unstale":"2024-01-16T14:58:25.476243+0000","last_undegraded":"2024-01-16T14:58:25.476243+0000","last_fullsized":"2024-01-16T14:58:25.476243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:37.904616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454392+0000","last_change":"2024-01-16T14:58:25.449719+0000","last_active":"2024-01-16T14:58:26.454392+0000","last_peered":"2024-01-16T14:58:26.454392+0000","last_clean":"2024-01-16T14:58:26.454392+0000","last_became_active":"2024-01-16T14:58:25.447359+0000","last_became_peered":"2024-01-16T14:58:25.447359+0000","last_unstale":"2024-01-16T14:58:26.454392+0000","last_undegraded":"2024-01-16T14:58:26.454392+0000","last_fullsized":"2024-01-16T14:58:26.454392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:22.409645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474319+0000","last_change":"2024-01-16T14:58:24.420965+0000","last_active":"2024-01-16T14:58:25.474319+0000","last_peered":"2024-01-16T14:58:25.474319+0000","last_clean":"2024-01-16T14:58:25.474319+0000","last_became_active":"2024-01-16T14:58:24.420644+0000","last_became_peered":"2024-01-16T14:58:24.420644+0000","last_unstale":"2024-01-16T14:58:25.474319+0000","last_undegraded":"2024-01-16T14:58:25.474319+0000","last_fullsized":"2024-01-16T14:58:25.474319+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:45.513448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726936+0000","last_change":"2024-01-16T14:58:25.455406+0000","last_active":"2024-01-16T14:58:27.726936+0000","last_peered":"2024-01-16T14:58:27.726936+0000","last_clean":"2024-01-16T14:58:27.726936+0000","last_became_active":"2024-01-16T14:58:25.455154+0000","last_became_peered":"2024-01-16T14:58:25.455154+0000","last_unstale":"2024-01-16T14:58:27.726936+0000","last_undegraded":"2024-01-16T14:58:27.726936+0000","last_fullsized":"2024-01-16T14:58:27.726936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:49.965022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.470656+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:25.470656+0000","last_peered":"2024-01-16T14:58:25.470656+0000","last_clean":"2024-01-16T14:58:25.470656+0000","last_became_active":"2024-01-16T14:58:24.421626+0000","last_became_peered":"2024-01-16T14:58:24.421626+0000","last_unstale":"2024-01-16T14:58:25.470656+0000","last_undegraded":"2024-01-16T14:58:25.470656+0000","last_fullsized":"2024-01-16T14:58:25.470656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:06.171691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863332+0000","last_change":"2024-01-16T14:58:25.456831+0000","last_active":"2024-01-16T14:58:27.863332+0000","last_peered":"2024-01-16T14:58:27.863332+0000","last_clean":"2024-01-16T14:58:27.863332+0000","last_became_active":"2024-01-16T14:58:25.456558+0000","last_became_peered":"2024-01-16T14:58:25.456558+0000","last_unstale":"2024-01-16T14:58:27.863332+0000","last_undegraded":"2024-01-16T14:58:27.863332+0000","last_fullsized":"2024-01-16T14:58:27.863332+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:46.526212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426019+0000","last_change":"2024-01-16T14:58:24.413866+0000","last_active":"2024-01-16T14:58:25.426019+0000","last_peered":"2024-01-16T14:58:25.426019+0000","last_clean":"2024-01-16T14:58:25.426019+0000","last_became_active":"2024-01-16T14:58:24.413577+0000","last_became_peered":"2024-01-16T14:58:24.413577+0000","last_unstale":"2024-01-16T14:58:25.426019+0000","last_undegraded":"2024-01-16T14:58:25.426019+0000","last_fullsized":"2024-01-16T14:58:25.426019+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:34.799937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.862924+0000","last_change":"2024-01-16T14:58:25.452034+0000","last_active":"2024-01-16T14:58:27.862924+0000","last_peered":"2024-01-16T14:58:27.862924+0000","last_clean":"2024-01-16T14:58:27.862924+0000","last_became_active":"2024-01-16T14:58:25.451687+0000","last_became_peered":"2024-01-16T14:58:25.451687+0000","last_unstale":"2024-01-16T14:58:27.862924+0000","last_undegraded":"2024-01-16T14:58:27.862924+0000","last_fullsized":"2024-01-16T14:58:27.862924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:52.969008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.472079+0000","last_change":"2024-01-16T14:58:24.417730+0000","last_active":"2024-01-16T14:58:25.472079+0000","last_peered":"2024-01-16T14:58:25.472079+0000","last_clean":"2024-01-16T14:58:25.472079+0000","last_became_active":"2024-01-16T14:58:24.417446+0000","last_became_peered":"2024-01-16T14:58:24.417446+0000","last_unstale":"2024-01-16T14:58:25.472079+0000","last_undegraded":"2024-01-16T14:58:25.472079+0000","last_fullsized":"2024-01-16T14:58:25.472079+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:22.607082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863443+0000","last_change":"2024-01-16T14:58:25.453154+0000","last_active":"2024-01-16T14:58:27.863443+0000","last_peered":"2024-01-16T14:58:27.863443+0000","last_clean":"2024-01-16T14:58:27.863443+0000","last_became_active":"2024-01-16T14:58:25.452818+0000","last_became_peered":"2024-01-16T14:58:25.452818+0000","last_unstale":"2024-01-16T14:58:27.863443+0000","last_undegraded":"2024-01-16T14:58:27.863443+0000","last_fullsized":"2024-01-16T14:58:27.863443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:10.106904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.435646+0000","last_change":"2024-01-16T14:58:24.426266+0000","last_active":"2024-01-16T14:58:25.435646+0000","last_peered":"2024-01-16T14:58:25.435646+0000","last_clean":"2024-01-16T14:58:25.435646+0000","last_became_active":"2024-01-16T14:58:24.425944+0000","last_became_peered":"2024-01-16T14:58:24.425944+0000","last_unstale":"2024-01-16T14:58:25.435646+0000","last_undegraded":"2024-01-16T14:58:25.435646+0000","last_fullsized":"2024-01-16T14:58:25.435646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863304+0000","last_change":"2024-01-16T14:58:25.456158+0000","last_active":"2024-01-16T14:58:27.863304+0000","last_peered":"2024-01-16T14:58:27.863304+0000","last_clean":"2024-01-16T14:58:27.863304+0000","last_became_active":"2024-01-16T14:58:25.455879+0000","last_became_peered":"2024-01-16T14:58:25.455879+0000","last_unstale":"2024-01-16T14:58:27.863304+0000","last_undegraded":"2024-01-16T14:58:27.863304+0000","last_fullsized":"2024-01-16T14:58:27.863304+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:35.812702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426368+0000","last_change":"2024-01-16T14:58:24.416410+0000","last_active":"2024-01-16T14:58:25.426368+0000","last_peered":"2024-01-16T14:58:25.426368+0000","last_clean":"2024-01-16T14:58:25.426368+0000","last_became_active":"2024-01-16T14:58:24.416121+0000","last_became_peered":"2024-01-16T14:58:24.416121+0000","last_unstale":"2024-01-16T14:58:25.426368+0000","last_undegraded":"2024-01-16T14:58:25.426368+0000","last_fullsized":"2024-01-16T14:58:25.426368+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:28.900039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449256+0000","last_change":"2024-01-16T14:58:25.452830+0000","last_active":"2024-01-16T14:58:26.449256+0000","last_peered":"2024-01-16T14:58:26.449256+0000","last_clean":"2024-01-16T14:58:26.449256+0000","last_became_active":"2024-01-16T14:58:25.449825+0000","last_became_peered":"2024-01-16T14:58:25.449825+0000","last_unstale":"2024-01-16T14:58:26.449256+0000","last_undegraded":"2024-01-16T14:58:26.449256+0000","last_fullsized":"2024-01-16T14:58:26.449256+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:11.932103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.432600+0000","last_change":"2024-01-16T14:58:24.429805+0000","last_active":"2024-01-16T14:58:25.432600+0000","last_peered":"2024-01-16T14:58:25.432600+0000","last_clean":"2024-01-16T14:58:25.432600+0000","last_became_active":"2024-01-16T14:58:24.429510+0000","last_became_peered":"2024-01-16T14:58:24.429510+0000","last_unstale":"2024-01-16T14:58:25.432600+0000","last_undegraded":"2024-01-16T14:58:25.432600+0000","last_fullsized":"2024-01-16T14:58:25.432600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:39.606992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863109+0000","last_change":"2024-01-16T14:58:25.451255+0000","last_active":"2024-01-16T14:58:27.863109+0000","last_peered":"2024-01-16T14:58:27.863109+0000","last_clean":"2024-01-16T14:58:27.863109+0000","last_became_active":"2024-01-16T14:58:25.450936+0000","last_became_peered":"2024-01-16T14:58:25.450936+0000","last_unstale":"2024-01-16T14:58:27.863109+0000","last_undegraded":"2024-01-16T14:58:27.863109+0000","last_fullsized":"2024-01-16T14:58:27.863109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:09.352999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474590+0000","last_change":"2024-01-16T14:58:24.428471+0000","last_active":"2024-01-16T14:58:25.474590+0000","last_peered":"2024-01-16T14:58:25.474590+0000","last_clean":"2024-01-16T14:58:25.474590+0000","last_became_active":"2024-01-16T14:58:24.427946+0000","last_became_peered":"2024-01-16T14:58:24.427946+0000","last_unstale":"2024-01-16T14:58:25.474590+0000","last_undegraded":"2024-01-16T14:58:25.474590+0000","last_fullsized":"2024-01-16T14:58:25.474590+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454398+0000","last_change":"2024-01-16T14:58:25.466478+0000","last_active":"2024-01-16T14:58:26.454398+0000","last_peered":"2024-01-16T14:58:26.454398+0000","last_clean":"2024-01-16T14:58:26.454398+0000","last_became_active":"2024-01-16T14:58:25.466179+0000","last_became_peered":"2024-01-16T14:58:25.466179+0000","last_unstale":"2024-01-16T14:58:26.454398+0000","last_undegraded":"2024-01-16T14:58:26.454398+0000","last_fullsized":"2024-01-16T14:58:26.454398+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:51.577006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.421288+0000","last_change":"2024-01-16T14:58:24.420850+0000","last_active":"2024-01-16T14:58:25.421288+0000","last_peered":"2024-01-16T14:58:25.421288+0000","last_clean":"2024-01-16T14:58:25.421288+0000","last_became_active":"2024-01-16T14:58:24.420553+0000","last_became_peered":"2024-01-16T14:58:24.420553+0000","last_unstale":"2024-01-16T14:58:25.421288+0000","last_undegraded":"2024-01-16T14:58:25.421288+0000","last_fullsized":"2024-01-16T14:58:25.421288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:19.956837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528092+0000","last_change":"2024-01-16T14:58:25.440870+0000","last_active":"2024-01-16T14:58:26.528092+0000","last_peered":"2024-01-16T14:58:26.528092+0000","last_clean":"2024-01-16T14:58:26.528092+0000","last_became_active":"2024-01-16T14:58:25.440562+0000","last_became_peered":"2024-01-16T14:58:25.440562+0000","last_unstale":"2024-01-16T14:58:26.528092+0000","last_undegraded":"2024-01-16T14:58:26.528092+0000","last_fullsized":"2024-01-16T14:58:26.528092+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:52.505975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.435229+0000","last_change":"2024-01-16T14:58:24.423543+0000","last_active":"2024-01-16T14:58:25.435229+0000","last_peered":"2024-01-16T14:58:25.435229+0000","last_clean":"2024-01-16T14:58:25.435229+0000","last_became_active":"2024-01-16T14:58:24.420295+0000","last_became_peered":"2024-01-16T14:58:24.420295+0000","last_unstale":"2024-01-16T14:58:25.435229+0000","last_undegraded":"2024-01-16T14:58:25.435229+0000","last_fullsized":"2024-01-16T14:58:25.435229+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:30.952362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.231899+0000","last_change":"2024-01-16T14:58:25.449151+0000","last_active":"2024-01-16T14:58:28.231899+0000","last_peered":"2024-01-16T14:58:28.231899+0000","last_clean":"2024-01-16T14:58:28.231899+0000","last_became_active":"2024-01-16T14:58:25.448796+0000","last_became_peered":"2024-01-16T14:58:25.448796+0000","last_unstale":"2024-01-16T14:58:28.231899+0000","last_undegraded":"2024-01-16T14:58:28.231899+0000","last_fullsized":"2024-01-16T14:58:28.231899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:38.606880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.420062+0000","last_change":"2024-01-16T14:58:24.424952+0000","last_active":"2024-01-16T14:58:25.420062+0000","last_peered":"2024-01-16T14:58:25.420062+0000","last_clean":"2024-01-16T14:58:25.420062+0000","last_became_active":"2024-01-16T14:58:24.423308+0000","last_became_peered":"2024-01-16T14:58:24.423308+0000","last_unstale":"2024-01-16T14:58:25.420062+0000","last_undegraded":"2024-01-16T14:58:25.420062+0000","last_fullsized":"2024-01-16T14:58:25.420062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528278+0000","last_change":"2024-01-16T14:58:25.451666+0000","last_active":"2024-01-16T14:58:26.528278+0000","last_peered":"2024-01-16T14:58:26.528278+0000","last_clean":"2024-01-16T14:58:26.528278+0000","last_became_active":"2024-01-16T14:58:25.451394+0000","last_became_peered":"2024-01-16T14:58:25.451394+0000","last_unstale":"2024-01-16T14:58:26.528278+0000","last_undegraded":"2024-01-16T14:58:26.528278+0000","last_fullsized":"2024-01-16T14:58:26.528278+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:50.425785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.429562+0000","last_change":"2024-01-16T14:58:24.427373+0000","last_active":"2024-01-16T14:58:25.429562+0000","last_peered":"2024-01-16T14:58:25.429562+0000","last_clean":"2024-01-16T14:58:25.429562+0000","last_became_active":"2024-01-16T14:58:24.426937+0000","last_became_peered":"2024-01-16T14:58:24.426937+0000","last_unstale":"2024-01-16T14:58:25.429562+0000","last_undegraded":"2024-01-16T14:58:25.429562+0000","last_fullsized":"2024-01-16T14:58:25.429562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:10.718372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.088446+0000","last_change":"2024-01-16T14:58:25.449369+0000","last_active":"2024-01-16T14:58:28.088446+0000","last_peered":"2024-01-16T14:58:28.088446+0000","last_clean":"2024-01-16T14:58:28.088446+0000","last_became_active":"2024-01-16T14:58:25.449078+0000","last_became_peered":"2024-01-16T14:58:25.449078+0000","last_unstale":"2024-01-16T14:58:28.088446+0000","last_undegraded":"2024-01-16T14:58:28.088446+0000","last_fullsized":"2024-01-16T14:58:28.088446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:23.851919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426752+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:25.426752+0000","last_peered":"2024-01-16T14:58:25.426752+0000","last_clean":"2024-01-16T14:58:25.426752+0000","last_became_active":"2024-01-16T14:58:24.421586+0000","last_became_peered":"2024-01-16T14:58:24.421586+0000","last_unstale":"2024-01-16T14:58:25.426752+0000","last_undegraded":"2024-01-16T14:58:25.426752+0000","last_fullsized":"2024-01-16T14:58:25.426752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452369+0000","last_change":"2024-01-16T14:58:25.449615+0000","last_active":"2024-01-16T14:58:26.452369+0000","last_peered":"2024-01-16T14:58:26.452369+0000","last_clean":"2024-01-16T14:58:26.452369+0000","last_became_active":"2024-01-16T14:58:25.447805+0000","last_became_peered":"2024-01-16T14:58:25.447805+0000","last_unstale":"2024-01-16T14:58:26.452369+0000","last_undegraded":"2024-01-16T14:58:26.452369+0000","last_fullsized":"2024-01-16T14:58:26.452369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:20.425389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437045+0000","last_change":"2024-01-16T14:58:24.427165+0000","last_active":"2024-01-16T14:58:25.437045+0000","last_peered":"2024-01-16T14:58:25.437045+0000","last_clean":"2024-01-16T14:58:25.437045+0000","last_became_active":"2024-01-16T14:58:24.426927+0000","last_became_peered":"2024-01-16T14:58:24.426927+0000","last_unstale":"2024-01-16T14:58:25.437045+0000","last_undegraded":"2024-01-16T14:58:25.437045+0000","last_fullsized":"2024-01-16T14:58:25.437045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:43.919514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448298+0000","last_change":"2024-01-16T14:58:25.448033+0000","last_active":"2024-01-16T14:58:26.448298+0000","last_peered":"2024-01-16T14:58:26.448298+0000","last_clean":"2024-01-16T14:58:26.448298+0000","last_became_active":"2024-01-16T14:58:25.447748+0000","last_became_peered":"2024-01-16T14:58:25.447748+0000","last_unstale":"2024-01-16T14:58:26.448298+0000","last_undegraded":"2024-01-16T14:58:26.448298+0000","last_fullsized":"2024-01-16T14:58:26.448298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.436751+0000","last_change":"2024-01-16T14:58:24.425447+0000","last_active":"2024-01-16T14:58:25.436751+0000","last_peered":"2024-01-16T14:58:25.436751+0000","last_clean":"2024-01-16T14:58:25.436751+0000","last_became_active":"2024-01-16T14:58:24.425035+0000","last_became_peered":"2024-01-16T14:58:24.425035+0000","last_unstale":"2024-01-16T14:58:25.436751+0000","last_undegraded":"2024-01-16T14:58:25.436751+0000","last_fullsized":"2024-01-16T14:58:25.436751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:20.089624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":177,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.436147+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:58:25.436147+0000","last_peered":"2024-01-16T14:58:25.436147+0000","last_clean":"2024-01-16T14:58:25.436147+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:58:25.436147+0000","last_undegraded":"2024-01-16T14:58:25.436147+0000","last_fullsized":"2024-01-16T14:58:25.436147+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":991232,"data_stored":744900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1519999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":1.0089999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.256},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.90400000000000003}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.605}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.45,"15min":1.45},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.337}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.109,"5min":1.109,"15min":1.109},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"max":{"1min":3.3530000000000002,"5min":3.3530000000000002,"15min":3.3530000000000002},"last":1.3879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":3.258,"5min":3.258,"15min":3.258},"last":1.573},{"interface":"front","average":{"1min":1.617,"5min":1.617,"15min":1.617},"min":{"1min":1.139,"5min":1.139,"15min":1.139},"max":{"1min":3.101,"5min":3.101,"15min":3.101},"last":1.542}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":1.04},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":1.109}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.294},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.5900000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.415}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.244}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.95799999999999996}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.625},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.4570000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.5469999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":155114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.331},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.78000000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.64300000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.234}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.1910000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.038},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":2.9540000000000002},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.431}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.149},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"max":{"1min":3.2389999999999999,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":1.407}]}]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":154546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.65900000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.96299999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.7490000000000001,"5min":2.7490000000000001,"15min":2.7490000000000001},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":1.1799999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":1.163},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.522}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.079,"5min":1.079,"15min":1.079},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.038},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":1.091}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":3.1880000000000002,"5min":3.1880000000000002,"15min":3.1880000000000002},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.788,"5min":1.788,"15min":1.788},"min":{"1min":1.232,"5min":1.232,"15min":1.232},"max":{"1min":3.2290000000000001,"5min":3.2290000000000001,"15min":3.2290000000000001},"last":1.2829999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.80900000000000005}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.196},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.008,"5min":4.008,"15min":4.008},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.54300000000000004}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.745},{"interface":"front","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.498},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.0269999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.872},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5549999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.0389999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":154512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.1419999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.4279999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.55},{"interface":"front","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.8779999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.514}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.7210000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.3680000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.76700000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.89900000000000002}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":1.016}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.115},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.0860000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.649,"5min":2.649,"15min":2.649},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":1.319}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.7290000000000001,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.3480000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.377},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.5529999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":744810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.194,"5min":4.194,"15min":4.194},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.4299999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.2359999999999998,"5min":4.2359999999999998,"15min":4.2359999999999998},"last":1.147}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.026}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":4.5529999999999999,"5min":4.5529999999999999,"15min":4.5529999999999999},"last":1.339},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.089}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.4880000000000004,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.276},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.2250000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.375},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.524,"5min":4.524,"15min":4.524},"last":1.4470000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:58:32.355 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:32.363 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:34.855 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:58:35.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.087+0000 7ff4b6a00640 1 Processor -- start 2024-01-16T14:58:35.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.087+0000 7ff4b6a00640 1 -- start start 2024-01-16T14:58:35.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4b6a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b00d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4b6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff4b00cf4c0 0x7ff4b01614c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4b6a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b00b2920 con 0x7ff4b015cc90 2024-01-16T14:58:35.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4b6a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b0161a00 con 0x7ff4b00cf4c0 2024-01-16T14:58:35.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4b6a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b00cb670 con 0x7ff4b01610e0 2024-01-16T14:58:35.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4a7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff4b00cf4c0 0x7ff4b01614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4a7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff4b00cf4c0 0x7ff4b01614c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44234/0 (socket says 172.21.15.141:44234) 2024-01-16T14:58:35.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.089+0000 7ff4a7e00640 1 -- 172.21.15.141:0/4072310936 learned_addr learned my addr 172.21.15.141:0/4072310936 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:35.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.090+0000 7ff4afe00640 1 --2- 172.21.15.141:0/4072310936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b00d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.090+0000 7ff4a7e00640 1 -- 172.21.15.141:0/4072310936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 msgr2=0x7ff4b00d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:35.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.090+0000 7ff4a7e00640 1 --2- 172.21.15.141:0/4072310936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b00d0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.090+0000 7ff4a7e00640 1 -- 172.21.15.141:0/4072310936 >> v1:172.21.15.188:6789/0 conn(0x7ff4b015cc90 legacy=0x7ff4b015d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:35.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.090+0000 7ff4a7e00640 1 -- 172.21.15.141:0/4072310936 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b00cb950 con 0x7ff4b00cf4c0 2024-01-16T14:58:35.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.091+0000 7ff4a7e00640 1 --2- 172.21.15.141:0/4072310936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b01614c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff498009870 tx=0x7ff49802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9c36a2951f4e3f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:35.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.091+0000 7ff4af400640 1 -- 172.21.15.141:0/4072310936 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498031640 con 0x7ff4b00cf4c0 2024-01-16T14:58:35.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.091+0000 7ff4af400640 1 -- 172.21.15.141:0/4072310936 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4980317a0 con 0x7ff4b00cf4c0 2024-01-16T14:58:35.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.091+0000 7ff4af400640 1 -- 172.21.15.141:0/4072310936 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498031b30 con 0x7ff4b00cf4c0 2024-01-16T14:58:35.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 -- 172.21.15.141:0/4072310936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b00cf4c0 msgr2=0x7ff4b01614c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/4072310936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b01614c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff498009870 tx=0x7ff49802b8f0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 -- 172.21.15.141:0/4072310936 shutdown_connections 2024-01-16T14:58:35.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/4072310936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b01614c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/4072310936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b00d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 -- 172.21.15.141:0/4072310936 >> 172.21.15.141:0/4072310936 conn(0x7ff4b00d1020 msgr2=0x7ff4b00d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:35.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 -- 172.21.15.141:0/4072310936 shutdown_connections 2024-01-16T14:58:35.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.092+0000 7ff4b6a00640 1 -- 172.21.15.141:0/4072310936 wait complete. 2024-01-16T14:58:35.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.093+0000 7ff4b6a00640 1 Processor -- start 2024-01-16T14:58:35.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.094+0000 7ff4b6a00640 1 -- start start 2024-01-16T14:58:35.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.094+0000 7ff4b6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b014b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.094+0000 7ff4b6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b015cc90 0x7ff4b01e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.094+0000 7ff4b6a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b01e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4b6a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b00c5a50 con 0x7ff4b015cc90 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4b6a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b014b870 con 0x7ff4b00cf4c0 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4b6a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b01f30b0 con 0x7ff4b01610e0 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b015cc90 0x7ff4b01e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b015cc90 0x7ff4b01e8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44244/0 (socket says 172.21.15.141:44244) 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 -- 172.21.15.141:0/1305286464 learned_addr learned my addr 172.21.15.141:0/1305286464 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4afe00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b014b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 -- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 msgr2=0x7ff4b01e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4b4a00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b01e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b01e6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 -- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4b00cf4c0 msgr2=0x7ff4b014b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b014b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4a7e00640 1 -- 172.21.15.141:0/1305286464 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4980094d0 con 0x7ff4b015cc90 2024-01-16T14:58:35.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.095+0000 7ff4afe00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b014b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:35.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.096+0000 7ff4a7e00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b015cc90 0x7ff4b01e8ae0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff498000c00 tx=0x7ff4980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:35.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.096+0000 7ff4acc00640 1 -- 172.21.15.141:0/1305286464 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49803ac60 con 0x7ff4b015cc90 2024-01-16T14:58:35.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.096+0000 7ff4acc00640 1 -- 172.21.15.141:0/1305286464 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49803ade0 con 0x7ff4b015cc90 2024-01-16T14:58:35.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.096+0000 7ff4acc00640 1 -- 172.21.15.141:0/1305286464 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498044630 con 0x7ff4b015cc90 2024-01-16T14:58:35.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.096+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b01e94e0 con 0x7ff4b015cc90 2024-01-16T14:58:35.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.096+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b00c56e0 con 0x7ff4b015cc90 2024-01-16T14:58:35.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.097+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b01e9a00 con 0x7ff4b015cc90 2024-01-16T14:58:35.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.098+0000 7ff4acc00640 1 -- 172.21.15.141:0/1305286464 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff49802f050 con 0x7ff4b015cc90 2024-01-16T14:58:35.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.099+0000 7ff4acc00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff48806b2c0 0x7ff48806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.099+0000 7ff4acc00640 1 -- 172.21.15.141:0/1305286464 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4980abd90 con 0x7ff4b015cc90 2024-01-16T14:58:35.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.100+0000 7ff4afe00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff48806b2c0 0x7ff48806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.101+0000 7ff4afe00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff48806b2c0 0x7ff48806d780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0008890 tx=0x7ff4a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:35.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.104+0000 7ff4acc00640 1 -- 172.21.15.141:0/1305286464 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4b01e9a00 con 0x7ff4b015cc90 2024-01-16T14:58:35.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.290+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4b00cc1f0 con 0x7ff48806b2c0 2024-01-16T14:58:35.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.313+0000 7ff4acc00640 1 -- 172.21.15.141:0/1305286464 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286529 (secure 0 0 0) 0x7ff4a000c040 con 0x7ff48806b2c0 2024-01-16T14:58:35.299 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:35.300 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T14:58:35.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff48806b2c0 msgr2=0x7ff48806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff48806b2c0 0x7ff48806d780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0008890 tx=0x7ff4a0008040 comp rx=0 tx=0).stop 2024-01-16T14:58:35.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b015cc90 msgr2=0x7ff4b01e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b015cc90 0x7ff4b01e8ae0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff498000c00 tx=0x7ff4980046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 shutdown_connections 2024-01-16T14:58:35.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff48806b2c0 0x7ff48806d780 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4b01610e0 0x7ff4b01e6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4b015cc90 0x7ff4b01e8ae0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 --2- 172.21.15.141:0/1305286464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4b00cf4c0 0x7ff4b014b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.318+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 >> 172.21.15.141:0/1305286464 conn(0x7ff4b00d1020 msgr2=0x7ff4b00cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:35.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.319+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 shutdown_connections 2024-01-16T14:58:35.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.319+0000 7ff4b6a00640 1 -- 172.21.15.141:0/1305286464 wait complete. 2024-01-16T14:58:35.319 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-01-16T14:58:34.465952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224632,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750088,"statfs":{"total":773094113280,"available":772864090112,"internally_reserved":0,"allocated":4399104,"data_stored":2421092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.049088"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232198+0000","last_change":"2024-01-16T14:58:25.453436+0000","last_active":"2024-01-16T14:58:28.232198+0000","last_peered":"2024-01-16T14:58:28.232198+0000","last_clean":"2024-01-16T14:58:28.232198+0000","last_became_active":"2024-01-16T14:58:25.453170+0000","last_became_peered":"2024-01-16T14:58:25.453170+0000","last_unstale":"2024-01-16T14:58:28.232198+0000","last_undegraded":"2024-01-16T14:58:28.232198+0000","last_fullsized":"2024-01-16T14:58:28.232198+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:19.680397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089898+0000","last_change":"2024-01-16T14:58:25.450853+0000","last_active":"2024-01-16T14:58:28.089898+0000","last_peered":"2024-01-16T14:58:28.089898+0000","last_clean":"2024-01-16T14:58:28.089898+0000","last_became_active":"2024-01-16T14:58:25.447306+0000","last_became_peered":"2024-01-16T14:58:25.447306+0000","last_unstale":"2024-01-16T14:58:28.089898+0000","last_undegraded":"2024-01-16T14:58:28.089898+0000","last_fullsized":"2024-01-16T14:58:28.089898+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:58.177859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451614+0000","last_change":"2024-01-16T14:58:25.467441+0000","last_active":"2024-01-16T14:58:26.451614+0000","last_peered":"2024-01-16T14:58:26.451614+0000","last_clean":"2024-01-16T14:58:26.451614+0000","last_became_active":"2024-01-16T14:58:25.466836+0000","last_became_peered":"2024-01-16T14:58:25.466836+0000","last_unstale":"2024-01-16T14:58:26.451614+0000","last_undegraded":"2024-01-16T14:58:26.451614+0000","last_fullsized":"2024-01-16T14:58:26.451614+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:22.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726970+0000","last_change":"2024-01-16T14:58:25.459481+0000","last_active":"2024-01-16T14:58:27.726970+0000","last_peered":"2024-01-16T14:58:27.726970+0000","last_clean":"2024-01-16T14:58:27.726970+0000","last_became_active":"2024-01-16T14:58:25.459193+0000","last_became_peered":"2024-01-16T14:58:25.459193+0000","last_unstale":"2024-01-16T14:58:27.726970+0000","last_undegraded":"2024-01-16T14:58:27.726970+0000","last_fullsized":"2024-01-16T14:58:27.726970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.719820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.862896+0000","last_change":"2024-01-16T14:58:25.453879+0000","last_active":"2024-01-16T14:58:27.862896+0000","last_peered":"2024-01-16T14:58:27.862896+0000","last_clean":"2024-01-16T14:58:27.862896+0000","last_became_active":"2024-01-16T14:58:25.453454+0000","last_became_peered":"2024-01-16T14:58:25.453454+0000","last_unstale":"2024-01-16T14:58:27.862896+0000","last_undegraded":"2024-01-16T14:58:27.862896+0000","last_fullsized":"2024-01-16T14:58:27.862896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:25.362250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.088902+0000","last_change":"2024-01-16T14:58:25.454154+0000","last_active":"2024-01-16T14:58:28.088902+0000","last_peered":"2024-01-16T14:58:28.088902+0000","last_clean":"2024-01-16T14:58:28.088902+0000","last_became_active":"2024-01-16T14:58:25.453809+0000","last_became_peered":"2024-01-16T14:58:25.453809+0000","last_unstale":"2024-01-16T14:58:28.088902+0000","last_undegraded":"2024-01-16T14:58:28.088902+0000","last_fullsized":"2024-01-16T14:58:28.088902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:09.724971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448873+0000","last_change":"2024-01-16T14:58:25.449577+0000","last_active":"2024-01-16T14:58:26.448873+0000","last_peered":"2024-01-16T14:58:26.448873+0000","last_clean":"2024-01-16T14:58:26.448873+0000","last_became_active":"2024-01-16T14:58:25.449289+0000","last_became_peered":"2024-01-16T14:58:25.449289+0000","last_unstale":"2024-01-16T14:58:26.448873+0000","last_undegraded":"2024-01-16T14:58:26.448873+0000","last_fullsized":"2024-01-16T14:58:26.448873+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:22.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447326+0000","last_change":"2024-01-16T14:58:25.447662+0000","last_active":"2024-01-16T14:58:26.447326+0000","last_peered":"2024-01-16T14:58:26.447326+0000","last_clean":"2024-01-16T14:58:26.447326+0000","last_became_active":"2024-01-16T14:58:25.447359+0000","last_became_peered":"2024-01-16T14:58:25.447359+0000","last_unstale":"2024-01-16T14:58:26.447326+0000","last_undegraded":"2024-01-16T14:58:26.447326+0000","last_fullsized":"2024-01-16T14:58:26.447326+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:38.719820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454085+0000","last_change":"2024-01-16T14:58:25.445625+0000","last_active":"2024-01-16T14:58:26.454085+0000","last_peered":"2024-01-16T14:58:26.454085+0000","last_clean":"2024-01-16T14:58:26.454085+0000","last_became_active":"2024-01-16T14:58:25.445288+0000","last_became_peered":"2024-01-16T14:58:25.445288+0000","last_unstale":"2024-01-16T14:58:26.454085+0000","last_undegraded":"2024-01-16T14:58:26.454085+0000","last_fullsized":"2024-01-16T14:58:26.454085+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:48.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727416+0000","last_change":"2024-01-16T14:58:25.446611+0000","last_active":"2024-01-16T14:58:27.727416+0000","last_peered":"2024-01-16T14:58:27.727416+0000","last_clean":"2024-01-16T14:58:27.727416+0000","last_became_active":"2024-01-16T14:58:25.445802+0000","last_became_peered":"2024-01-16T14:58:25.445802+0000","last_unstale":"2024-01-16T14:58:27.727416+0000","last_undegraded":"2024-01-16T14:58:27.727416+0000","last_fullsized":"2024-01-16T14:58:27.727416+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448582+0000","last_change":"2024-01-16T14:58:25.457171+0000","last_active":"2024-01-16T14:58:26.448582+0000","last_peered":"2024-01-16T14:58:26.448582+0000","last_clean":"2024-01-16T14:58:26.448582+0000","last_became_active":"2024-01-16T14:58:25.456898+0000","last_became_peered":"2024-01-16T14:58:25.456898+0000","last_unstale":"2024-01-16T14:58:26.448582+0000","last_undegraded":"2024-01-16T14:58:26.448582+0000","last_fullsized":"2024-01-16T14:58:26.448582+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:11.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451319+0000","last_change":"2024-01-16T14:58:25.454493+0000","last_active":"2024-01-16T14:58:26.451319+0000","last_peered":"2024-01-16T14:58:26.451319+0000","last_clean":"2024-01-16T14:58:26.451319+0000","last_became_active":"2024-01-16T14:58:25.454215+0000","last_became_peered":"2024-01-16T14:58:25.454215+0000","last_unstale":"2024-01-16T14:58:26.451319+0000","last_undegraded":"2024-01-16T14:58:26.451319+0000","last_fullsized":"2024-01-16T14:58:26.451319+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:11.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863116+0000","last_change":"2024-01-16T14:58:25.447370+0000","last_active":"2024-01-16T14:58:27.863116+0000","last_peered":"2024-01-16T14:58:27.863116+0000","last_clean":"2024-01-16T14:58:27.863116+0000","last_became_active":"2024-01-16T14:58:25.447100+0000","last_became_peered":"2024-01-16T14:58:25.447100+0000","last_unstale":"2024-01-16T14:58:27.863116+0000","last_undegraded":"2024-01-16T14:58:27.863116+0000","last_fullsized":"2024-01-16T14:58:27.863116+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:40.264688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089314+0000","last_change":"2024-01-16T14:58:25.454014+0000","last_active":"2024-01-16T14:58:28.089314+0000","last_peered":"2024-01-16T14:58:28.089314+0000","last_clean":"2024-01-16T14:58:28.089314+0000","last_became_active":"2024-01-16T14:58:25.453284+0000","last_became_peered":"2024-01-16T14:58:25.453284+0000","last_unstale":"2024-01-16T14:58:28.089314+0000","last_undegraded":"2024-01-16T14:58:28.089314+0000","last_fullsized":"2024-01-16T14:58:28.089314+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:08.961898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451019+0000","last_change":"2024-01-16T14:58:25.450821+0000","last_active":"2024-01-16T14:58:26.451019+0000","last_peered":"2024-01-16T14:58:26.451019+0000","last_clean":"2024-01-16T14:58:26.451019+0000","last_became_active":"2024-01-16T14:58:25.450270+0000","last_became_peered":"2024-01-16T14:58:25.450270+0000","last_unstale":"2024-01-16T14:58:26.451019+0000","last_undegraded":"2024-01-16T14:58:26.451019+0000","last_fullsized":"2024-01-16T14:58:26.451019+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:37.439565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448105+0000","last_change":"2024-01-16T14:58:25.449064+0000","last_active":"2024-01-16T14:58:26.448105+0000","last_peered":"2024-01-16T14:58:26.448105+0000","last_clean":"2024-01-16T14:58:26.448105+0000","last_became_active":"2024-01-16T14:58:25.448777+0000","last_became_peered":"2024-01-16T14:58:25.448777+0000","last_unstale":"2024-01-16T14:58:26.448105+0000","last_undegraded":"2024-01-16T14:58:26.448105+0000","last_fullsized":"2024-01-16T14:58:26.448105+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449556+0000","last_change":"2024-01-16T14:58:25.452550+0000","last_active":"2024-01-16T14:58:26.449556+0000","last_peered":"2024-01-16T14:58:26.449556+0000","last_clean":"2024-01-16T14:58:26.449556+0000","last_became_active":"2024-01-16T14:58:25.452267+0000","last_became_peered":"2024-01-16T14:58:25.452267+0000","last_unstale":"2024-01-16T14:58:26.449556+0000","last_undegraded":"2024-01-16T14:58:26.449556+0000","last_fullsized":"2024-01-16T14:58:26.449556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:51.577006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089919+0000","last_change":"2024-01-16T14:58:25.447596+0000","last_active":"2024-01-16T14:58:28.089919+0000","last_peered":"2024-01-16T14:58:28.089919+0000","last_clean":"2024-01-16T14:58:28.089919+0000","last_became_active":"2024-01-16T14:58:25.446849+0000","last_became_peered":"2024-01-16T14:58:25.446849+0000","last_unstale":"2024-01-16T14:58:28.089919+0000","last_undegraded":"2024-01-16T14:58:28.089919+0000","last_fullsized":"2024-01-16T14:58:28.089919+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:25:27.806945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089503+0000","last_change":"2024-01-16T14:58:25.452291+0000","last_active":"2024-01-16T14:58:28.089503+0000","last_peered":"2024-01-16T14:58:28.089503+0000","last_clean":"2024-01-16T14:58:28.089503+0000","last_became_active":"2024-01-16T14:58:25.452017+0000","last_became_peered":"2024-01-16T14:58:25.452017+0000","last_unstale":"2024-01-16T14:58:28.089503+0000","last_undegraded":"2024-01-16T14:58:28.089503+0000","last_fullsized":"2024-01-16T14:58:28.089503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:31.087496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863458+0000","last_change":"2024-01-16T14:58:25.451264+0000","last_active":"2024-01-16T14:58:27.863458+0000","last_peered":"2024-01-16T14:58:27.863458+0000","last_clean":"2024-01-16T14:58:27.863458+0000","last_became_active":"2024-01-16T14:58:25.450973+0000","last_became_peered":"2024-01-16T14:58:25.450973+0000","last_unstale":"2024-01-16T14:58:27.863458+0000","last_undegraded":"2024-01-16T14:58:27.863458+0000","last_fullsized":"2024-01-16T14:58:27.863458+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:29.912804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452736+0000","last_change":"2024-01-16T14:58:25.445306+0000","last_active":"2024-01-16T14:58:26.452736+0000","last_peered":"2024-01-16T14:58:26.452736+0000","last_clean":"2024-01-16T14:58:26.452736+0000","last_became_active":"2024-01-16T14:58:25.442633+0000","last_became_peered":"2024-01-16T14:58:25.442633+0000","last_unstale":"2024-01-16T14:58:26.452736+0000","last_undegraded":"2024-01-16T14:58:26.452736+0000","last_fullsized":"2024-01-16T14:58:26.452736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:11.731137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452224+0000","last_change":"2024-01-16T14:58:25.448740+0000","last_active":"2024-01-16T14:58:26.452224+0000","last_peered":"2024-01-16T14:58:26.452224+0000","last_clean":"2024-01-16T14:58:26.452224+0000","last_became_active":"2024-01-16T14:58:25.448233+0000","last_became_peered":"2024-01-16T14:58:25.448233+0000","last_unstale":"2024-01-16T14:58:26.452224+0000","last_undegraded":"2024-01-16T14:58:26.452224+0000","last_fullsized":"2024-01-16T14:58:26.452224+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:36.906207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.527930+0000","last_change":"2024-01-16T14:58:25.445600+0000","last_active":"2024-01-16T14:58:26.527930+0000","last_peered":"2024-01-16T14:58:26.527930+0000","last_clean":"2024-01-16T14:58:26.527930+0000","last_became_active":"2024-01-16T14:58:25.445302+0000","last_became_peered":"2024-01-16T14:58:25.445302+0000","last_unstale":"2024-01-16T14:58:26.527930+0000","last_undegraded":"2024-01-16T14:58:26.527930+0000","last_fullsized":"2024-01-16T14:58:26.527930+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:45.333764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.450450+0000","last_change":"2024-01-16T14:58:25.452393+0000","last_active":"2024-01-16T14:58:26.450450+0000","last_peered":"2024-01-16T14:58:26.450450+0000","last_clean":"2024-01-16T14:58:26.450450+0000","last_became_active":"2024-01-16T14:58:25.451586+0000","last_became_peered":"2024-01-16T14:58:25.451586+0000","last_unstale":"2024-01-16T14:58:26.450450+0000","last_undegraded":"2024-01-16T14:58:26.450450+0000","last_fullsized":"2024-01-16T14:58:26.450450+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727298+0000","last_change":"2024-01-16T14:58:25.448882+0000","last_active":"2024-01-16T14:58:27.727298+0000","last_peered":"2024-01-16T14:58:27.727298+0000","last_clean":"2024-01-16T14:58:27.727298+0000","last_became_active":"2024-01-16T14:58:25.448300+0000","last_became_peered":"2024-01-16T14:58:25.448300+0000","last_unstale":"2024-01-16T14:58:27.727298+0000","last_undegraded":"2024-01-16T14:58:27.727298+0000","last_fullsized":"2024-01-16T14:58:27.727298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.090087+0000","last_change":"2024-01-16T14:58:25.458035+0000","last_active":"2024-01-16T14:58:28.090087+0000","last_peered":"2024-01-16T14:58:28.090087+0000","last_clean":"2024-01-16T14:58:28.090087+0000","last_became_active":"2024-01-16T14:58:25.457804+0000","last_became_peered":"2024-01-16T14:58:25.457804+0000","last_unstale":"2024-01-16T14:58:28.090087+0000","last_undegraded":"2024-01-16T14:58:28.090087+0000","last_fullsized":"2024-01-16T14:58:28.090087+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:39.016885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727478+0000","last_change":"2024-01-16T14:58:25.465239+0000","last_active":"2024-01-16T14:58:27.727478+0000","last_peered":"2024-01-16T14:58:27.727478+0000","last_clean":"2024-01-16T14:58:27.727478+0000","last_became_active":"2024-01-16T14:58:25.462123+0000","last_became_peered":"2024-01-16T14:58:25.462123+0000","last_unstale":"2024-01-16T14:58:27.727478+0000","last_undegraded":"2024-01-16T14:58:27.727478+0000","last_fullsized":"2024-01-16T14:58:27.727478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451320+0000","last_change":"2024-01-16T14:58:25.457821+0000","last_active":"2024-01-16T14:58:26.451320+0000","last_peered":"2024-01-16T14:58:26.451320+0000","last_clean":"2024-01-16T14:58:26.451320+0000","last_became_active":"2024-01-16T14:58:25.457526+0000","last_became_peered":"2024-01-16T14:58:25.457526+0000","last_unstale":"2024-01-16T14:58:26.451320+0000","last_undegraded":"2024-01-16T14:58:26.451320+0000","last_fullsized":"2024-01-16T14:58:26.451320+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:25.159168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452858+0000","last_change":"2024-01-16T14:58:25.445985+0000","last_active":"2024-01-16T14:58:26.452858+0000","last_peered":"2024-01-16T14:58:26.452858+0000","last_clean":"2024-01-16T14:58:26.452858+0000","last_became_active":"2024-01-16T14:58:25.443290+0000","last_became_peered":"2024-01-16T14:58:25.443290+0000","last_unstale":"2024-01-16T14:58:26.452858+0000","last_undegraded":"2024-01-16T14:58:26.452858+0000","last_fullsized":"2024-01-16T14:58:26.452858+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:40.619757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449725+0000","last_change":"2024-01-16T14:58:25.448385+0000","last_active":"2024-01-16T14:58:26.449725+0000","last_peered":"2024-01-16T14:58:26.449725+0000","last_clean":"2024-01-16T14:58:26.449725+0000","last_became_active":"2024-01-16T14:58:25.448118+0000","last_became_peered":"2024-01-16T14:58:25.448118+0000","last_unstale":"2024-01-16T14:58:26.449725+0000","last_undegraded":"2024-01-16T14:58:26.449725+0000","last_fullsized":"2024-01-16T14:58:26.449725+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:22.409645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.231762+0000","last_change":"2024-01-16T14:58:25.454801+0000","last_active":"2024-01-16T14:58:28.231762+0000","last_peered":"2024-01-16T14:58:28.231762+0000","last_clean":"2024-01-16T14:58:28.231762+0000","last_became_active":"2024-01-16T14:58:25.454281+0000","last_became_peered":"2024-01-16T14:58:25.454281+0000","last_unstale":"2024-01-16T14:58:28.231762+0000","last_undegraded":"2024-01-16T14:58:28.231762+0000","last_fullsized":"2024-01-16T14:58:28.231762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:41.881567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447551+0000","last_change":"2024-01-16T14:58:25.458893+0000","last_active":"2024-01-16T14:58:26.447551+0000","last_peered":"2024-01-16T14:58:26.447551+0000","last_clean":"2024-01-16T14:58:26.447551+0000","last_became_active":"2024-01-16T14:58:25.458300+0000","last_became_peered":"2024-01-16T14:58:25.458300+0000","last_unstale":"2024-01-16T14:58:26.447551+0000","last_undegraded":"2024-01-16T14:58:26.447551+0000","last_fullsized":"2024-01-16T14:58:26.447551+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:59.140128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528093+0000","last_change":"2024-01-16T14:58:25.448792+0000","last_active":"2024-01-16T14:58:26.528093+0000","last_peered":"2024-01-16T14:58:26.528093+0000","last_clean":"2024-01-16T14:58:26.528093+0000","last_became_active":"2024-01-16T14:58:25.448490+0000","last_became_peered":"2024-01-16T14:58:25.448490+0000","last_unstale":"2024-01-16T14:58:26.528093+0000","last_undegraded":"2024-01-16T14:58:26.528093+0000","last_fullsized":"2024-01-16T14:58:26.528093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:45.048906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474893+0000","last_change":"2024-01-16T14:58:24.428745+0000","last_active":"2024-01-16T14:58:25.474893+0000","last_peered":"2024-01-16T14:58:25.474893+0000","last_clean":"2024-01-16T14:58:25.474893+0000","last_became_active":"2024-01-16T14:58:24.427170+0000","last_became_peered":"2024-01-16T14:58:24.427170+0000","last_unstale":"2024-01-16T14:58:25.474893+0000","last_undegraded":"2024-01-16T14:58:25.474893+0000","last_fullsized":"2024-01-16T14:58:25.474893+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:43.995208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232212+0000","last_change":"2024-01-16T14:58:25.455325+0000","last_active":"2024-01-16T14:58:28.232212+0000","last_peered":"2024-01-16T14:58:28.232212+0000","last_clean":"2024-01-16T14:58:28.232212+0000","last_became_active":"2024-01-16T14:58:25.454992+0000","last_became_peered":"2024-01-16T14:58:25.454992+0000","last_unstale":"2024-01-16T14:58:28.232212+0000","last_undegraded":"2024-01-16T14:58:28.232212+0000","last_fullsized":"2024-01-16T14:58:28.232212+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:36.367808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.469108+0000","last_change":"2024-01-16T14:58:24.429636+0000","last_active":"2024-01-16T14:58:25.469108+0000","last_peered":"2024-01-16T14:58:25.469108+0000","last_clean":"2024-01-16T14:58:25.469108+0000","last_became_active":"2024-01-16T14:58:24.428619+0000","last_became_peered":"2024-01-16T14:58:24.428619+0000","last_unstale":"2024-01-16T14:58:25.469108+0000","last_undegraded":"2024-01-16T14:58:25.469108+0000","last_fullsized":"2024-01-16T14:58:25.469108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:06:40.868803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.453290+0000","last_change":"2024-01-16T14:58:25.450473+0000","last_active":"2024-01-16T14:58:26.453290+0000","last_peered":"2024-01-16T14:58:26.453290+0000","last_clean":"2024-01-16T14:58:26.453290+0000","last_became_active":"2024-01-16T14:58:25.450169+0000","last_became_peered":"2024-01-16T14:58:25.450169+0000","last_unstale":"2024-01-16T14:58:26.453290+0000","last_undegraded":"2024-01-16T14:58:26.453290+0000","last_fullsized":"2024-01-16T14:58:26.453290+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:43.043403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474307+0000","last_change":"2024-01-16T14:58:24.421162+0000","last_active":"2024-01-16T14:58:25.474307+0000","last_peered":"2024-01-16T14:58:25.474307+0000","last_clean":"2024-01-16T14:58:25.474307+0000","last_became_active":"2024-01-16T14:58:24.420858+0000","last_became_peered":"2024-01-16T14:58:24.420858+0000","last_unstale":"2024-01-16T14:58:25.474307+0000","last_undegraded":"2024-01-16T14:58:25.474307+0000","last_fullsized":"2024-01-16T14:58:25.474307+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:48.834908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.727604+0000","last_change":"2024-01-16T14:58:25.465488+0000","last_active":"2024-01-16T14:58:27.727604+0000","last_peered":"2024-01-16T14:58:27.727604+0000","last_clean":"2024-01-16T14:58:27.727604+0000","last_became_active":"2024-01-16T14:58:25.460624+0000","last_became_peered":"2024-01-16T14:58:25.460624+0000","last_unstale":"2024-01-16T14:58:27.727604+0000","last_undegraded":"2024-01-16T14:58:27.727604+0000","last_fullsized":"2024-01-16T14:58:27.727604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.473676+0000","last_change":"2024-01-16T14:58:24.426642+0000","last_active":"2024-01-16T14:58:25.473676+0000","last_peered":"2024-01-16T14:58:25.473676+0000","last_clean":"2024-01-16T14:58:25.473676+0000","last_became_active":"2024-01-16T14:58:24.426379+0000","last_became_peered":"2024-01-16T14:58:24.426379+0000","last_unstale":"2024-01-16T14:58:25.473676+0000","last_undegraded":"2024-01-16T14:58:25.473676+0000","last_fullsized":"2024-01-16T14:58:25.473676+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:29:09.809503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449031+0000","last_change":"2024-01-16T14:58:25.458055+0000","last_active":"2024-01-16T14:58:26.449031+0000","last_peered":"2024-01-16T14:58:26.449031+0000","last_clean":"2024-01-16T14:58:26.449031+0000","last_became_active":"2024-01-16T14:58:25.457556+0000","last_became_peered":"2024-01-16T14:58:25.457556+0000","last_unstale":"2024-01-16T14:58:26.449031+0000","last_undegraded":"2024-01-16T14:58:26.449031+0000","last_fullsized":"2024-01-16T14:58:26.449031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:37.439565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.471816+0000","last_change":"2024-01-16T14:58:24.419926+0000","last_active":"2024-01-16T14:58:25.471816+0000","last_peered":"2024-01-16T14:58:25.471816+0000","last_clean":"2024-01-16T14:58:25.471816+0000","last_became_active":"2024-01-16T14:58:24.419550+0000","last_became_peered":"2024-01-16T14:58:24.419550+0000","last_unstale":"2024-01-16T14:58:25.471816+0000","last_undegraded":"2024-01-16T14:58:25.471816+0000","last_fullsized":"2024-01-16T14:58:25.471816+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:09.094139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451316+0000","last_change":"2024-01-16T14:58:25.453757+0000","last_active":"2024-01-16T14:58:26.451316+0000","last_peered":"2024-01-16T14:58:26.451316+0000","last_clean":"2024-01-16T14:58:26.451316+0000","last_became_active":"2024-01-16T14:58:25.453492+0000","last_became_peered":"2024-01-16T14:58:25.453492+0000","last_unstale":"2024-01-16T14:58:26.451316+0000","last_undegraded":"2024-01-16T14:58:26.451316+0000","last_fullsized":"2024-01-16T14:58:26.451316+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.471378+0000","last_change":"2024-01-16T14:58:24.420225+0000","last_active":"2024-01-16T14:58:25.471378+0000","last_peered":"2024-01-16T14:58:25.471378+0000","last_clean":"2024-01-16T14:58:25.471378+0000","last_became_active":"2024-01-16T14:58:24.419237+0000","last_became_peered":"2024-01-16T14:58:24.419237+0000","last_unstale":"2024-01-16T14:58:25.471378+0000","last_undegraded":"2024-01-16T14:58:25.471378+0000","last_fullsized":"2024-01-16T14:58:25.471378+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:21.201884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451519+0000","last_change":"2024-01-16T14:58:25.451171+0000","last_active":"2024-01-16T14:58:26.451519+0000","last_peered":"2024-01-16T14:58:26.451519+0000","last_clean":"2024-01-16T14:58:26.451519+0000","last_became_active":"2024-01-16T14:58:25.450320+0000","last_became_peered":"2024-01-16T14:58:25.450320+0000","last_unstale":"2024-01-16T14:58:26.451519+0000","last_undegraded":"2024-01-16T14:58:26.451519+0000","last_fullsized":"2024-01-16T14:58:26.451519+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:57:49.471035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437180+0000","last_change":"2024-01-16T14:58:24.416771+0000","last_active":"2024-01-16T14:58:25.437180+0000","last_peered":"2024-01-16T14:58:25.437180+0000","last_clean":"2024-01-16T14:58:25.437180+0000","last_became_active":"2024-01-16T14:58:24.416479+0000","last_became_peered":"2024-01-16T14:58:24.416479+0000","last_unstale":"2024-01-16T14:58:25.437180+0000","last_undegraded":"2024-01-16T14:58:25.437180+0000","last_fullsized":"2024-01-16T14:58:25.437180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:15.420130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726309+0000","last_change":"2024-01-16T14:58:25.446441+0000","last_active":"2024-01-16T14:58:27.726309+0000","last_peered":"2024-01-16T14:58:27.726309+0000","last_clean":"2024-01-16T14:58:27.726309+0000","last_became_active":"2024-01-16T14:58:25.446153+0000","last_became_peered":"2024-01-16T14:58:25.446153+0000","last_unstale":"2024-01-16T14:58:27.726309+0000","last_undegraded":"2024-01-16T14:58:27.726309+0000","last_fullsized":"2024-01-16T14:58:27.726309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:59.978419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.424174+0000","last_change":"2024-01-16T14:58:24.426653+0000","last_active":"2024-01-16T14:58:25.424174+0000","last_peered":"2024-01-16T14:58:25.424174+0000","last_clean":"2024-01-16T14:58:25.424174+0000","last_became_active":"2024-01-16T14:58:24.421492+0000","last_became_peered":"2024-01-16T14:58:24.421492+0000","last_unstale":"2024-01-16T14:58:25.424174+0000","last_undegraded":"2024-01-16T14:58:25.424174+0000","last_fullsized":"2024-01-16T14:58:25.424174+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:51.956244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454109+0000","last_change":"2024-01-16T14:58:25.455256+0000","last_active":"2024-01-16T14:58:26.454109+0000","last_peered":"2024-01-16T14:58:26.454109+0000","last_clean":"2024-01-16T14:58:26.454109+0000","last_became_active":"2024-01-16T14:58:25.454696+0000","last_became_peered":"2024-01-16T14:58:25.454696+0000","last_unstale":"2024-01-16T14:58:26.454109+0000","last_undegraded":"2024-01-16T14:58:26.454109+0000","last_fullsized":"2024-01-16T14:58:26.454109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:11.932103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.443769+0000","last_change":"2024-01-16T14:58:24.426750+0000","last_active":"2024-01-16T14:58:25.443769+0000","last_peered":"2024-01-16T14:58:25.443769+0000","last_clean":"2024-01-16T14:58:25.443769+0000","last_became_active":"2024-01-16T14:58:24.426534+0000","last_became_peered":"2024-01-16T14:58:24.426534+0000","last_unstale":"2024-01-16T14:58:25.443769+0000","last_undegraded":"2024-01-16T14:58:25.443769+0000","last_fullsized":"2024-01-16T14:58:25.443769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:32.501031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089695+0000","last_change":"2024-01-16T14:58:25.471298+0000","last_active":"2024-01-16T14:58:28.089695+0000","last_peered":"2024-01-16T14:58:28.089695+0000","last_clean":"2024-01-16T14:58:28.089695+0000","last_became_active":"2024-01-16T14:58:25.470921+0000","last_became_peered":"2024-01-16T14:58:25.470921+0000","last_unstale":"2024-01-16T14:58:28.089695+0000","last_undegraded":"2024-01-16T14:58:28.089695+0000","last_fullsized":"2024-01-16T14:58:28.089695+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:18.391832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.421251+0000","last_change":"2024-01-16T14:58:24.414545+0000","last_active":"2024-01-16T14:58:25.421251+0000","last_peered":"2024-01-16T14:58:25.421251+0000","last_clean":"2024-01-16T14:58:25.421251+0000","last_became_active":"2024-01-16T14:58:24.412623+0000","last_became_peered":"2024-01-16T14:58:24.412623+0000","last_unstale":"2024-01-16T14:58:25.421251+0000","last_undegraded":"2024-01-16T14:58:25.421251+0000","last_fullsized":"2024-01-16T14:58:25.421251+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:37.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089315+0000","last_change":"2024-01-16T14:58:25.456962+0000","last_active":"2024-01-16T14:58:28.089315+0000","last_peered":"2024-01-16T14:58:28.089315+0000","last_clean":"2024-01-16T14:58:28.089315+0000","last_became_active":"2024-01-16T14:58:25.456712+0000","last_became_peered":"2024-01-16T14:58:25.456712+0000","last_unstale":"2024-01-16T14:58:28.089315+0000","last_undegraded":"2024-01-16T14:58:28.089315+0000","last_fullsized":"2024-01-16T14:58:28.089315+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:10.429909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.473767+0000","last_change":"2024-01-16T14:58:24.428533+0000","last_active":"2024-01-16T14:58:25.473767+0000","last_peered":"2024-01-16T14:58:25.473767+0000","last_clean":"2024-01-16T14:58:25.473767+0000","last_became_active":"2024-01-16T14:58:24.428239+0000","last_became_peered":"2024-01-16T14:58:24.428239+0000","last_unstale":"2024-01-16T14:58:25.473767+0000","last_undegraded":"2024-01-16T14:58:25.473767+0000","last_fullsized":"2024-01-16T14:58:25.473767+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:32.544123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.451239+0000","last_change":"2024-01-16T14:58:25.467675+0000","last_active":"2024-01-16T14:58:26.451239+0000","last_peered":"2024-01-16T14:58:26.451239+0000","last_clean":"2024-01-16T14:58:26.451239+0000","last_became_active":"2024-01-16T14:58:25.467220+0000","last_became_peered":"2024-01-16T14:58:25.467220+0000","last_unstale":"2024-01-16T14:58:26.451239+0000","last_undegraded":"2024-01-16T14:58:26.451239+0000","last_fullsized":"2024-01-16T14:58:26.451239+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.477630+0000","last_change":"2024-01-16T14:58:24.426130+0000","last_active":"2024-01-16T14:58:25.477630+0000","last_peered":"2024-01-16T14:58:25.477630+0000","last_clean":"2024-01-16T14:58:25.477630+0000","last_became_active":"2024-01-16T14:58:24.425829+0000","last_became_peered":"2024-01-16T14:58:24.425829+0000","last_unstale":"2024-01-16T14:58:25.477630+0000","last_undegraded":"2024-01-16T14:58:25.477630+0000","last_fullsized":"2024-01-16T14:58:25.477630+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:11.348392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726576+0000","last_change":"2024-01-16T14:58:25.446712+0000","last_active":"2024-01-16T14:58:27.726576+0000","last_peered":"2024-01-16T14:58:27.726576+0000","last_clean":"2024-01-16T14:58:27.726576+0000","last_became_active":"2024-01-16T14:58:25.446349+0000","last_became_peered":"2024-01-16T14:58:25.446349+0000","last_unstale":"2024-01-16T14:58:27.726576+0000","last_undegraded":"2024-01-16T14:58:27.726576+0000","last_fullsized":"2024-01-16T14:58:27.726576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:33.556888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474223+0000","last_change":"2024-01-16T14:58:24.423758+0000","last_active":"2024-01-16T14:58:25.474223+0000","last_peered":"2024-01-16T14:58:25.474223+0000","last_clean":"2024-01-16T14:58:25.474223+0000","last_became_active":"2024-01-16T14:58:24.423458+0000","last_became_peered":"2024-01-16T14:58:24.423458+0000","last_unstale":"2024-01-16T14:58:25.474223+0000","last_undegraded":"2024-01-16T14:58:25.474223+0000","last_fullsized":"2024-01-16T14:58:25.474223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:56.142999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.089168+0000","last_change":"2024-01-16T14:58:25.447562+0000","last_active":"2024-01-16T14:58:28.089168+0000","last_peered":"2024-01-16T14:58:28.089168+0000","last_clean":"2024-01-16T14:58:28.089168+0000","last_became_active":"2024-01-16T14:58:25.447243+0000","last_became_peered":"2024-01-16T14:58:25.447243+0000","last_unstale":"2024-01-16T14:58:28.089168+0000","last_undegraded":"2024-01-16T14:58:28.089168+0000","last_fullsized":"2024-01-16T14:58:28.089168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:41.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.429141+0000","last_change":"2024-01-16T14:58:24.427569+0000","last_active":"2024-01-16T14:58:25.429141+0000","last_peered":"2024-01-16T14:58:25.429141+0000","last_clean":"2024-01-16T14:58:25.429141+0000","last_became_active":"2024-01-16T14:58:24.427309+0000","last_became_peered":"2024-01-16T14:58:24.427309+0000","last_unstale":"2024-01-16T14:58:25.429141+0000","last_undegraded":"2024-01-16T14:58:25.429141+0000","last_fullsized":"2024-01-16T14:58:25.429141+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:19.412624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.447681+0000","last_change":"2024-01-16T14:58:25.458315+0000","last_active":"2024-01-16T14:58:26.447681+0000","last_peered":"2024-01-16T14:58:26.447681+0000","last_clean":"2024-01-16T14:58:26.447681+0000","last_became_active":"2024-01-16T14:58:25.458030+0000","last_became_peered":"2024-01-16T14:58:25.458030+0000","last_unstale":"2024-01-16T14:58:26.447681+0000","last_undegraded":"2024-01-16T14:58:26.447681+0000","last_fullsized":"2024-01-16T14:58:26.447681+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:16.981434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437070+0000","last_change":"2024-01-16T14:58:24.427259+0000","last_active":"2024-01-16T14:58:25.437070+0000","last_peered":"2024-01-16T14:58:25.437070+0000","last_clean":"2024-01-16T14:58:25.437070+0000","last_became_active":"2024-01-16T14:58:24.426959+0000","last_became_peered":"2024-01-16T14:58:24.426959+0000","last_unstale":"2024-01-16T14:58:25.437070+0000","last_undegraded":"2024-01-16T14:58:25.437070+0000","last_fullsized":"2024-01-16T14:58:25.437070+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:10.096980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.232161+0000","last_change":"2024-01-16T14:58:25.460308+0000","last_active":"2024-01-16T14:58:28.232161+0000","last_peered":"2024-01-16T14:58:28.232161+0000","last_clean":"2024-01-16T14:58:28.232161+0000","last_became_active":"2024-01-16T14:58:25.458176+0000","last_became_peered":"2024-01-16T14:58:25.458176+0000","last_unstale":"2024-01-16T14:58:28.232161+0000","last_undegraded":"2024-01-16T14:58:28.232161+0000","last_fullsized":"2024-01-16T14:58:28.232161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:20.969602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.475331+0000","last_change":"2024-01-16T14:58:24.416038+0000","last_active":"2024-01-16T14:58:25.475331+0000","last_peered":"2024-01-16T14:58:25.475331+0000","last_clean":"2024-01-16T14:58:25.475331+0000","last_became_active":"2024-01-16T14:58:24.415718+0000","last_became_peered":"2024-01-16T14:58:24.415718+0000","last_unstale":"2024-01-16T14:58:25.475331+0000","last_undegraded":"2024-01-16T14:58:25.475331+0000","last_fullsized":"2024-01-16T14:58:25.475331+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:03.937440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.453721+0000","last_change":"2024-01-16T14:58:25.451447+0000","last_active":"2024-01-16T14:58:26.453721+0000","last_peered":"2024-01-16T14:58:26.453721+0000","last_clean":"2024-01-16T14:58:26.453721+0000","last_became_active":"2024-01-16T14:58:25.446471+0000","last_became_peered":"2024-01-16T14:58:25.446471+0000","last_unstale":"2024-01-16T14:58:26.453721+0000","last_undegraded":"2024-01-16T14:58:26.453721+0000","last_fullsized":"2024-01-16T14:58:26.453721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:31.965127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.476243+0000","last_change":"2024-01-16T14:58:24.424464+0000","last_active":"2024-01-16T14:58:25.476243+0000","last_peered":"2024-01-16T14:58:25.476243+0000","last_clean":"2024-01-16T14:58:25.476243+0000","last_became_active":"2024-01-16T14:58:24.424200+0000","last_became_peered":"2024-01-16T14:58:24.424200+0000","last_unstale":"2024-01-16T14:58:25.476243+0000","last_undegraded":"2024-01-16T14:58:25.476243+0000","last_fullsized":"2024-01-16T14:58:25.476243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:37.904616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454392+0000","last_change":"2024-01-16T14:58:25.449719+0000","last_active":"2024-01-16T14:58:26.454392+0000","last_peered":"2024-01-16T14:58:26.454392+0000","last_clean":"2024-01-16T14:58:26.454392+0000","last_became_active":"2024-01-16T14:58:25.447359+0000","last_became_peered":"2024-01-16T14:58:25.447359+0000","last_unstale":"2024-01-16T14:58:26.454392+0000","last_undegraded":"2024-01-16T14:58:26.454392+0000","last_fullsized":"2024-01-16T14:58:26.454392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:22.409645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474319+0000","last_change":"2024-01-16T14:58:24.420965+0000","last_active":"2024-01-16T14:58:25.474319+0000","last_peered":"2024-01-16T14:58:25.474319+0000","last_clean":"2024-01-16T14:58:25.474319+0000","last_became_active":"2024-01-16T14:58:24.420644+0000","last_became_peered":"2024-01-16T14:58:24.420644+0000","last_unstale":"2024-01-16T14:58:25.474319+0000","last_undegraded":"2024-01-16T14:58:25.474319+0000","last_fullsized":"2024-01-16T14:58:25.474319+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:45.513448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.726936+0000","last_change":"2024-01-16T14:58:25.455406+0000","last_active":"2024-01-16T14:58:27.726936+0000","last_peered":"2024-01-16T14:58:27.726936+0000","last_clean":"2024-01-16T14:58:27.726936+0000","last_became_active":"2024-01-16T14:58:25.455154+0000","last_became_peered":"2024-01-16T14:58:25.455154+0000","last_unstale":"2024-01-16T14:58:27.726936+0000","last_undegraded":"2024-01-16T14:58:27.726936+0000","last_fullsized":"2024-01-16T14:58:27.726936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:49.965022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.470656+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:25.470656+0000","last_peered":"2024-01-16T14:58:25.470656+0000","last_clean":"2024-01-16T14:58:25.470656+0000","last_became_active":"2024-01-16T14:58:24.421626+0000","last_became_peered":"2024-01-16T14:58:24.421626+0000","last_unstale":"2024-01-16T14:58:25.470656+0000","last_undegraded":"2024-01-16T14:58:25.470656+0000","last_fullsized":"2024-01-16T14:58:25.470656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:06.171691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863332+0000","last_change":"2024-01-16T14:58:25.456831+0000","last_active":"2024-01-16T14:58:27.863332+0000","last_peered":"2024-01-16T14:58:27.863332+0000","last_clean":"2024-01-16T14:58:27.863332+0000","last_became_active":"2024-01-16T14:58:25.456558+0000","last_became_peered":"2024-01-16T14:58:25.456558+0000","last_unstale":"2024-01-16T14:58:27.863332+0000","last_undegraded":"2024-01-16T14:58:27.863332+0000","last_fullsized":"2024-01-16T14:58:27.863332+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:46.526212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426019+0000","last_change":"2024-01-16T14:58:24.413866+0000","last_active":"2024-01-16T14:58:25.426019+0000","last_peered":"2024-01-16T14:58:25.426019+0000","last_clean":"2024-01-16T14:58:25.426019+0000","last_became_active":"2024-01-16T14:58:24.413577+0000","last_became_peered":"2024-01-16T14:58:24.413577+0000","last_unstale":"2024-01-16T14:58:25.426019+0000","last_undegraded":"2024-01-16T14:58:25.426019+0000","last_fullsized":"2024-01-16T14:58:25.426019+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:34.799937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.862924+0000","last_change":"2024-01-16T14:58:25.452034+0000","last_active":"2024-01-16T14:58:27.862924+0000","last_peered":"2024-01-16T14:58:27.862924+0000","last_clean":"2024-01-16T14:58:27.862924+0000","last_became_active":"2024-01-16T14:58:25.451687+0000","last_became_peered":"2024-01-16T14:58:25.451687+0000","last_unstale":"2024-01-16T14:58:27.862924+0000","last_undegraded":"2024-01-16T14:58:27.862924+0000","last_fullsized":"2024-01-16T14:58:27.862924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:52.969008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.472079+0000","last_change":"2024-01-16T14:58:24.417730+0000","last_active":"2024-01-16T14:58:25.472079+0000","last_peered":"2024-01-16T14:58:25.472079+0000","last_clean":"2024-01-16T14:58:25.472079+0000","last_became_active":"2024-01-16T14:58:24.417446+0000","last_became_peered":"2024-01-16T14:58:24.417446+0000","last_unstale":"2024-01-16T14:58:25.472079+0000","last_undegraded":"2024-01-16T14:58:25.472079+0000","last_fullsized":"2024-01-16T14:58:25.472079+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:22.607082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863443+0000","last_change":"2024-01-16T14:58:25.453154+0000","last_active":"2024-01-16T14:58:27.863443+0000","last_peered":"2024-01-16T14:58:27.863443+0000","last_clean":"2024-01-16T14:58:27.863443+0000","last_became_active":"2024-01-16T14:58:25.452818+0000","last_became_peered":"2024-01-16T14:58:25.452818+0000","last_unstale":"2024-01-16T14:58:27.863443+0000","last_undegraded":"2024-01-16T14:58:27.863443+0000","last_fullsized":"2024-01-16T14:58:27.863443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:10.106904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.435646+0000","last_change":"2024-01-16T14:58:24.426266+0000","last_active":"2024-01-16T14:58:25.435646+0000","last_peered":"2024-01-16T14:58:25.435646+0000","last_clean":"2024-01-16T14:58:25.435646+0000","last_became_active":"2024-01-16T14:58:24.425944+0000","last_became_peered":"2024-01-16T14:58:24.425944+0000","last_unstale":"2024-01-16T14:58:25.435646+0000","last_undegraded":"2024-01-16T14:58:25.435646+0000","last_fullsized":"2024-01-16T14:58:25.435646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863304+0000","last_change":"2024-01-16T14:58:25.456158+0000","last_active":"2024-01-16T14:58:27.863304+0000","last_peered":"2024-01-16T14:58:27.863304+0000","last_clean":"2024-01-16T14:58:27.863304+0000","last_became_active":"2024-01-16T14:58:25.455879+0000","last_became_peered":"2024-01-16T14:58:25.455879+0000","last_unstale":"2024-01-16T14:58:27.863304+0000","last_undegraded":"2024-01-16T14:58:27.863304+0000","last_fullsized":"2024-01-16T14:58:27.863304+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:01:35.812702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426368+0000","last_change":"2024-01-16T14:58:24.416410+0000","last_active":"2024-01-16T14:58:25.426368+0000","last_peered":"2024-01-16T14:58:25.426368+0000","last_clean":"2024-01-16T14:58:25.426368+0000","last_became_active":"2024-01-16T14:58:24.416121+0000","last_became_peered":"2024-01-16T14:58:24.416121+0000","last_unstale":"2024-01-16T14:58:25.426368+0000","last_undegraded":"2024-01-16T14:58:25.426368+0000","last_fullsized":"2024-01-16T14:58:25.426368+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:28.900039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.449256+0000","last_change":"2024-01-16T14:58:25.452830+0000","last_active":"2024-01-16T14:58:26.449256+0000","last_peered":"2024-01-16T14:58:26.449256+0000","last_clean":"2024-01-16T14:58:26.449256+0000","last_became_active":"2024-01-16T14:58:25.449825+0000","last_became_peered":"2024-01-16T14:58:25.449825+0000","last_unstale":"2024-01-16T14:58:26.449256+0000","last_undegraded":"2024-01-16T14:58:26.449256+0000","last_fullsized":"2024-01-16T14:58:26.449256+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:11.932103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.432600+0000","last_change":"2024-01-16T14:58:24.429805+0000","last_active":"2024-01-16T14:58:25.432600+0000","last_peered":"2024-01-16T14:58:25.432600+0000","last_clean":"2024-01-16T14:58:25.432600+0000","last_became_active":"2024-01-16T14:58:24.429510+0000","last_became_peered":"2024-01-16T14:58:24.429510+0000","last_unstale":"2024-01-16T14:58:25.432600+0000","last_undegraded":"2024-01-16T14:58:25.432600+0000","last_fullsized":"2024-01-16T14:58:25.432600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:39.606992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:27.863109+0000","last_change":"2024-01-16T14:58:25.451255+0000","last_active":"2024-01-16T14:58:27.863109+0000","last_peered":"2024-01-16T14:58:27.863109+0000","last_clean":"2024-01-16T14:58:27.863109+0000","last_became_active":"2024-01-16T14:58:25.450936+0000","last_became_peered":"2024-01-16T14:58:25.450936+0000","last_unstale":"2024-01-16T14:58:27.863109+0000","last_undegraded":"2024-01-16T14:58:27.863109+0000","last_fullsized":"2024-01-16T14:58:27.863109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:09.352999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.474590+0000","last_change":"2024-01-16T14:58:24.428471+0000","last_active":"2024-01-16T14:58:25.474590+0000","last_peered":"2024-01-16T14:58:25.474590+0000","last_clean":"2024-01-16T14:58:25.474590+0000","last_became_active":"2024-01-16T14:58:24.427946+0000","last_became_peered":"2024-01-16T14:58:24.427946+0000","last_unstale":"2024-01-16T14:58:25.474590+0000","last_undegraded":"2024-01-16T14:58:25.474590+0000","last_fullsized":"2024-01-16T14:58:25.474590+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.454398+0000","last_change":"2024-01-16T14:58:25.466478+0000","last_active":"2024-01-16T14:58:26.454398+0000","last_peered":"2024-01-16T14:58:26.454398+0000","last_clean":"2024-01-16T14:58:26.454398+0000","last_became_active":"2024-01-16T14:58:25.466179+0000","last_became_peered":"2024-01-16T14:58:25.466179+0000","last_unstale":"2024-01-16T14:58:26.454398+0000","last_undegraded":"2024-01-16T14:58:26.454398+0000","last_fullsized":"2024-01-16T14:58:26.454398+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:51.577006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.421288+0000","last_change":"2024-01-16T14:58:24.420850+0000","last_active":"2024-01-16T14:58:25.421288+0000","last_peered":"2024-01-16T14:58:25.421288+0000","last_clean":"2024-01-16T14:58:25.421288+0000","last_became_active":"2024-01-16T14:58:24.420553+0000","last_became_peered":"2024-01-16T14:58:24.420553+0000","last_unstale":"2024-01-16T14:58:25.421288+0000","last_undegraded":"2024-01-16T14:58:25.421288+0000","last_fullsized":"2024-01-16T14:58:25.421288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:19.956837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528092+0000","last_change":"2024-01-16T14:58:25.440870+0000","last_active":"2024-01-16T14:58:26.528092+0000","last_peered":"2024-01-16T14:58:26.528092+0000","last_clean":"2024-01-16T14:58:26.528092+0000","last_became_active":"2024-01-16T14:58:25.440562+0000","last_became_peered":"2024-01-16T14:58:25.440562+0000","last_unstale":"2024-01-16T14:58:26.528092+0000","last_undegraded":"2024-01-16T14:58:26.528092+0000","last_fullsized":"2024-01-16T14:58:26.528092+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:52.505975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.435229+0000","last_change":"2024-01-16T14:58:24.423543+0000","last_active":"2024-01-16T14:58:25.435229+0000","last_peered":"2024-01-16T14:58:25.435229+0000","last_clean":"2024-01-16T14:58:25.435229+0000","last_became_active":"2024-01-16T14:58:24.420295+0000","last_became_peered":"2024-01-16T14:58:24.420295+0000","last_unstale":"2024-01-16T14:58:25.435229+0000","last_undegraded":"2024-01-16T14:58:25.435229+0000","last_fullsized":"2024-01-16T14:58:25.435229+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:30.952362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.231899+0000","last_change":"2024-01-16T14:58:25.449151+0000","last_active":"2024-01-16T14:58:28.231899+0000","last_peered":"2024-01-16T14:58:28.231899+0000","last_clean":"2024-01-16T14:58:28.231899+0000","last_became_active":"2024-01-16T14:58:25.448796+0000","last_became_peered":"2024-01-16T14:58:25.448796+0000","last_unstale":"2024-01-16T14:58:28.231899+0000","last_undegraded":"2024-01-16T14:58:28.231899+0000","last_fullsized":"2024-01-16T14:58:28.231899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:38.606880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.420062+0000","last_change":"2024-01-16T14:58:24.424952+0000","last_active":"2024-01-16T14:58:25.420062+0000","last_peered":"2024-01-16T14:58:25.420062+0000","last_clean":"2024-01-16T14:58:25.420062+0000","last_became_active":"2024-01-16T14:58:24.423308+0000","last_became_peered":"2024-01-16T14:58:24.423308+0000","last_unstale":"2024-01-16T14:58:25.420062+0000","last_undegraded":"2024-01-16T14:58:25.420062+0000","last_fullsized":"2024-01-16T14:58:25.420062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:12.586116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.528278+0000","last_change":"2024-01-16T14:58:25.451666+0000","last_active":"2024-01-16T14:58:26.528278+0000","last_peered":"2024-01-16T14:58:26.528278+0000","last_clean":"2024-01-16T14:58:26.528278+0000","last_became_active":"2024-01-16T14:58:25.451394+0000","last_became_peered":"2024-01-16T14:58:25.451394+0000","last_unstale":"2024-01-16T14:58:26.528278+0000","last_undegraded":"2024-01-16T14:58:26.528278+0000","last_fullsized":"2024-01-16T14:58:26.528278+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:50.425785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.429562+0000","last_change":"2024-01-16T14:58:24.427373+0000","last_active":"2024-01-16T14:58:25.429562+0000","last_peered":"2024-01-16T14:58:25.429562+0000","last_clean":"2024-01-16T14:58:25.429562+0000","last_became_active":"2024-01-16T14:58:24.426937+0000","last_became_peered":"2024-01-16T14:58:24.426937+0000","last_unstale":"2024-01-16T14:58:25.429562+0000","last_undegraded":"2024-01-16T14:58:25.429562+0000","last_fullsized":"2024-01-16T14:58:25.429562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:10.718372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:28.088446+0000","last_change":"2024-01-16T14:58:25.449369+0000","last_active":"2024-01-16T14:58:28.088446+0000","last_peered":"2024-01-16T14:58:28.088446+0000","last_clean":"2024-01-16T14:58:28.088446+0000","last_became_active":"2024-01-16T14:58:25.449078+0000","last_became_peered":"2024-01-16T14:58:25.449078+0000","last_unstale":"2024-01-16T14:58:28.088446+0000","last_undegraded":"2024-01-16T14:58:28.088446+0000","last_fullsized":"2024-01-16T14:58:28.088446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:23.851919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.426752+0000","last_change":"2024-01-16T14:58:24.421864+0000","last_active":"2024-01-16T14:58:25.426752+0000","last_peered":"2024-01-16T14:58:25.426752+0000","last_clean":"2024-01-16T14:58:25.426752+0000","last_became_active":"2024-01-16T14:58:24.421586+0000","last_became_peered":"2024-01-16T14:58:24.421586+0000","last_unstale":"2024-01-16T14:58:25.426752+0000","last_undegraded":"2024-01-16T14:58:25.426752+0000","last_fullsized":"2024-01-16T14:58:25.426752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:15.083056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.452369+0000","last_change":"2024-01-16T14:58:25.449615+0000","last_active":"2024-01-16T14:58:26.452369+0000","last_peered":"2024-01-16T14:58:26.452369+0000","last_clean":"2024-01-16T14:58:26.452369+0000","last_became_active":"2024-01-16T14:58:25.447805+0000","last_became_peered":"2024-01-16T14:58:25.447805+0000","last_unstale":"2024-01-16T14:58:26.452369+0000","last_undegraded":"2024-01-16T14:58:26.452369+0000","last_fullsized":"2024-01-16T14:58:26.452369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:20.425389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.437045+0000","last_change":"2024-01-16T14:58:24.427165+0000","last_active":"2024-01-16T14:58:25.437045+0000","last_peered":"2024-01-16T14:58:25.437045+0000","last_clean":"2024-01-16T14:58:25.437045+0000","last_became_active":"2024-01-16T14:58:24.426927+0000","last_became_peered":"2024-01-16T14:58:24.426927+0000","last_unstale":"2024-01-16T14:58:25.437045+0000","last_undegraded":"2024-01-16T14:58:25.437045+0000","last_fullsized":"2024-01-16T14:58:25.437045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:43.919514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T14:58:26.448298+0000","last_change":"2024-01-16T14:58:25.448033+0000","last_active":"2024-01-16T14:58:26.448298+0000","last_peered":"2024-01-16T14:58:26.448298+0000","last_clean":"2024-01-16T14:58:26.448298+0000","last_became_active":"2024-01-16T14:58:25.447748+0000","last_became_peered":"2024-01-16T14:58:25.447748+0000","last_unstale":"2024-01-16T14:58:26.448298+0000","last_undegraded":"2024-01-16T14:58:26.448298+0000","last_fullsized":"2024-01-16T14:58:26.448298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:24.382919+0000","last_clean_scrub_stamp":"2024-01-16T14:58:24.382919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:41.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.436751+0000","last_change":"2024-01-16T14:58:24.425447+0000","last_active":"2024-01-16T14:58:25.436751+0000","last_peered":"2024-01-16T14:58:25.436751+0000","last_clean":"2024-01-16T14:58:25.436751+0000","last_became_active":"2024-01-16T14:58:24.425035+0000","last_became_peered":"2024-01-16T14:58:24.425035+0000","last_unstale":"2024-01-16T14:58:25.436751+0000","last_undegraded":"2024-01-16T14:58:25.436751+0000","last_fullsized":"2024-01-16T14:58:25.436751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:58:23.370155+0000","last_clean_scrub_stamp":"2024-01-16T14:58:23.370155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:20.089624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":177,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T14:58:25.436147+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T14:58:25.436147+0000","last_peered":"2024-01-16T14:58:25.436147+0000","last_clean":"2024-01-16T14:58:25.436147+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T14:58:25.436147+0000","last_undegraded":"2024-01-16T14:58:25.436147+0000","last_fullsized":"2024-01-16T14:58:25.436147+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":991232,"data_stored":744900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1519999999999999}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":1.0089999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.256},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.90400000000000003}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.605}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.45,"15min":1.45},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.337}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.109,"5min":1.109,"15min":1.109},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"max":{"1min":3.3530000000000002,"5min":3.3530000000000002,"15min":3.3530000000000002},"last":1.3879999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":3.258,"5min":3.258,"15min":3.258},"last":1.573},{"interface":"front","average":{"1min":1.617,"5min":1.617,"15min":1.617},"min":{"1min":1.139,"5min":1.139,"15min":1.139},"max":{"1min":3.101,"5min":3.101,"15min":3.101},"last":1.542}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":1.04},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":1.109}]},{"osd":1,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.294},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.5900000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.415}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.244}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.95799999999999996}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.625},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.4570000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.5469999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":155114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.331},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.78000000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.64300000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.234}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.1910000000000001}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.038},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":2.9540000000000002},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.431}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.149},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"max":{"1min":3.2389999999999999,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":1.407}]}]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":154546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.65900000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.96299999999999997}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.7490000000000001,"5min":2.7490000000000001,"15min":2.7490000000000001},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":1.1799999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":1.163},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.522}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.079,"5min":1.079,"15min":1.079},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.038},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":1.091}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":3.1880000000000002,"5min":3.1880000000000002,"15min":3.1880000000000002},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.788,"5min":1.788,"15min":1.788},"min":{"1min":1.232,"5min":1.232,"15min":1.232},"max":{"1min":3.2290000000000001,"5min":3.2290000000000001,"15min":3.2290000000000001},"last":1.2829999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.80900000000000005}]},{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.196},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.008,"5min":4.008,"15min":4.008},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.54300000000000004}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.745},{"interface":"front","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.498},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.0269999999999999}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.872},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5549999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.0389999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":154512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.1419999999999999}]},{"osd":3,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.4279999999999999}]},{"osd":4,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.55},{"interface":"front","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.8779999999999999}]},{"osd":5,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.514}]},{"osd":6,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.7210000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.3680000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.76700000000000002}]},{"osd":2,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.89900000000000002}]},{"osd":3,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":1.016}]},{"osd":4,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.115},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.0860000000000001}]},{"osd":5,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.649,"5min":2.649,"15min":2.649},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":1.319}]},{"osd":6,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.7290000000000001,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.3480000000000001}]},{"osd":7,"last update":"Tue Jan 16 14:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.377},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.5529999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":744810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.194,"5min":4.194,"15min":4.194},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.4299999999999999}]},{"osd":2,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.2359999999999998,"5min":4.2359999999999998,"15min":4.2359999999999998},"last":1.147}]},{"osd":3,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.026}]},{"osd":4,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":4.5529999999999999,"5min":4.5529999999999999,"15min":4.5529999999999999},"last":1.339},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.089}]},{"osd":5,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.4880000000000004,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.276},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.2250000000000001}]},{"osd":6,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Tue Jan 16 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.375},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.524,"5min":4.524,"15min":4.524},"last":1.4470000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:58:35.324 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T14:58:35.324 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T14:58:35.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.597+0000 7f2e16000640 1 Processor -- start 2024-01-16T14:58:35.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.597+0000 7f2e16000640 1 -- start start 2024-01-16T14:58:35.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e16000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2e1015dd40 0x7f2e1015e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e16000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1015e6e0 con 0x7f2e1015bed0 2024-01-16T14:58:35.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e16000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1015ba40 con 0x7f2e10157dc0 2024-01-16T14:58:35.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e16000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e101581a0 con 0x7f2e1015dd40 2024-01-16T14:58:35.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e0fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2e1015bed0 0x7f2e1015c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44576/0 (socket says 172.21.15.141:44576) 2024-01-16T14:58:35.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e0fe00640 1 -- 172.21.15.141:0/1305470667 learned_addr learned my addr 172.21.15.141:0/1305470667 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e06800640 1 --2- 172.21.15.141:0/1305470667 >> v2:172.21.15.188:3300/0 conn(0x7f2e1015dd40 0x7f2e1015e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131309190 0 0) 0x7f2e1015e6e0 con 0x7f2e1015bed0 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.599+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0003520 con 0x7f2e1015bed0 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222793858 0 0) 0x7f2df0003520 con 0x7f2e1015bed0 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 >> v2:172.21.15.188:3300/0 conn(0x7f2e1015dd40 msgr2=0x7f2e1015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 --2- 172.21.15.141:0/1305470667 >> v2:172.21.15.188:3300/0 conn(0x7f2e1015dd40 0x7f2e1015e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 >> v1:172.21.15.188:6789/0 conn(0x7f2e10157dc0 legacy=0x7f2e10070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1015eb10 con 0x7f2e1015bed0 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2e080019f0 con 0x7f2e1015bed0 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 >> v1:172.21.15.141:6789/0 conn(0x7f2e1015bed0 legacy=0x7f2e1015c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2df0003970 0x7f2df0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2df00047c0 0x7f2df0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2df0005180 0x7f2df000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e101581a0 con 0x7f2df0003970 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df000c020 con 0x7f2df0005180 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df000c1e0 con 0x7f2df00047c0 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.600+0000 7f2e0f400640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2df0003970 0x7f2df0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.601+0000 7f2e0f400640 1 -- 172.21.15.141:0/1305470667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2df00047c0 msgr2=0x7f2df0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.601+0000 7f2e0f400640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2df00047c0 0x7f2df0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.601+0000 7f2e0f400640 1 -- 172.21.15.141:0/1305470667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2df0005180 msgr2=0x7f2df000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:35.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.601+0000 7f2e0f400640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2df0005180 0x7f2df000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.601+0000 7f2e0f400640 1 -- 172.21.15.141:0/1305470667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df0003520 con 0x7f2df0003970 2024-01-16T14:58:35.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.601+0000 7f2e0fe00640 1 -- 172.21.15.141:0/1305470667 reap_dead start 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.602+0000 7f2e0f400640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2df0003970 0x7f2df0003d90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2e00001750 tx=0x7f2df00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85195d76f42b3d78 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.602+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e000052d0 con 0x7f2df0003970 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.602+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0002e050 con 0x7f2df0003970 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.602+0000 7f2e0ea00640 1 -- 172.21.15.141:0/1305470667 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00003830 con 0x7f2df0003970 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.602+0000 7f2e16000640 1 -- 172.21.15.141:0/1305470667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2df0003970 msgr2=0x7f2df0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.602+0000 7f2e16000640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2df0003970 0x7f2df0003d90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2e00001750 tx=0x7f2df00057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.603+0000 7f2e16000640 1 -- 172.21.15.141:0/1305470667 shutdown_connections 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.603+0000 7f2e16000640 1 --2- 172.21.15.141:0/1305470667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2df0003970 0x7f2df0003d90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.603+0000 7f2e16000640 1 -- 172.21.15.141:0/1305470667 >> 172.21.15.141:0/1305470667 conn(0x7f2e100b0be0 msgr2=0x7f2e100b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.603+0000 7f2e16000640 1 -- 172.21.15.141:0/1305470667 shutdown_connections 2024-01-16T14:58:35.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.603+0000 7f2e16000640 1 -- 172.21.15.141:0/1305470667 wait complete. 2024-01-16T14:58:35.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.604+0000 7f2e16000640 1 Processor -- start 2024-01-16T14:58:35.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.605+0000 7f2e16000640 1 -- start start 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.605+0000 7f2e16000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2df0003970 0x7f2e1014a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.605+0000 7f2e16000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e100b2e90 0x7f2e100b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.605+0000 7f2e16000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e100c8d80 0x7f2e100c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.605+0000 7f2e16000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100b3850 con 0x7f2e100c8d80 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.605+0000 7f2e16000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100cabd0 con 0x7f2df0003970 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.605+0000 7f2e16000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1014af20 con 0x7f2e100b2e90 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e100c8d80 0x7f2e100c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e100c8d80 0x7f2e100c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44254/0 (socket says 172.21.15.141:44254) 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 -- 172.21.15.141:0/3051720302 learned_addr learned my addr 172.21.15.141:0/3051720302 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 -- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e100b2e90 msgr2=0x7f2e100b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e100b2e90 0x7f2e100b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 -- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2df0003970 msgr2=0x7f2e1014a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2df0003970 0x7f2e1014a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 -- 172.21.15.141:0/3051720302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e100c9b30 con 0x7f2e100c8d80 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.606+0000 7f2e0fe00640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e100c8d80 0x7f2e100c9230 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2e10067280 tx=0x7f2e0800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.607+0000 7f2e07e00640 1 -- 172.21.15.141:0/3051720302 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08004d60 con 0x7f2e100c8d80 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.607+0000 7f2e07e00640 1 -- 172.21.15.141:0/3051720302 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0800a5b0 con 0x7f2e100c8d80 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.607+0000 7f2e07e00640 1 -- 172.21.15.141:0/3051720302 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0800a960 con 0x7f2e100c8d80 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.607+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e100c9e10 con 0x7f2e100c8d80 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.607+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e100ca3d0 con 0x7f2e100c8d80 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.608+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e10073c00 con 0x7f2e100c8d80 2024-01-16T14:58:35.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.609+0000 7f2e07e00640 1 -- 172.21.15.141:0/3051720302 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2e0800aac0 con 0x7f2e100c8d80 2024-01-16T14:58:35.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.610+0000 7f2e07e00640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2de806b2f0 0x7f2de806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:35.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.610+0000 7f2e07e00640 1 -- 172.21.15.141:0/3051720302 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e080898a0 con 0x7f2e100c8d80 2024-01-16T14:58:35.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.610+0000 7f2e0f400640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2de806b2f0 0x7f2de806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:35.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.611+0000 7f2e0f400640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2de806b2f0 0x7f2de806d7b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e00033000 tx=0x7f2e0002f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:35.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.614+0000 7f2e07e00640 1 -- 172.21.15.141:0/3051720302 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e08054700 con 0x7f2e100c8d80 2024-01-16T14:58:35.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.851+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2e100739c0 con 0x7f2e100c8d80 2024-01-16T14:58:35.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.856+0000 7f2e07e00640 1 -- 172.21.15.141:0/3051720302 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2e08057d20 con 0x7f2e100c8d80 2024-01-16T14:58:35.839 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T14:58:35.839 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T14:58:35.839 INFO:teuthology.orchestra.run.smithi141.stdout: caps mds = "allow" 2024-01-16T14:58:35.839 INFO:teuthology.orchestra.run.smithi141.stdout: caps mon = "allow r" 2024-01-16T14:58:35.839 INFO:teuthology.orchestra.run.smithi141.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T14:58:35.839 INFO:teuthology.orchestra.run.smithi141.stderr:updated caps for client.0 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.859+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2de806b2f0 msgr2=0x7f2de806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.859+0000 7f2e16000640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2de806b2f0 0x7f2de806d7b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e00033000 tx=0x7f2e0002f000 comp rx=0 tx=0).stop 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.859+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e100c8d80 msgr2=0x7f2e100c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.859+0000 7f2e16000640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e100c8d80 0x7f2e100c9230 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2e10067280 tx=0x7f2e0800cb90 comp rx=0 tx=0).stop 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 shutdown_connections 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2de806b2f0 0x7f2de806d7b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e100c8d80 0x7f2e100c9230 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e100b2e90 0x7f2e100b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 --2- 172.21.15.141:0/3051720302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2df0003970 0x7f2e1014a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:35.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 >> 172.21.15.141:0/3051720302 conn(0x7f2e100b0be0 msgr2=0x7f2e10156670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:35.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 shutdown_connections 2024-01-16T14:58:35.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:35.860+0000 7f2e16000640 1 -- 172.21.15.141:0/3051720302 wait complete. 2024-01-16T14:58:35.859 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:58:36.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.083+0000 7fec5d600640 1 Processor -- start 2024-01-16T14:58:36.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.083+0000 7fec5d600640 1 -- start start 2024-01-16T14:58:36.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec5d600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fec580b3c20 0x7fec58070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec5d600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec58071790 con 0x7fec58159590 2024-01-16T14:58:36.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec5d600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec58071100 con 0x7fec5806d050 2024-01-16T14:58:36.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec5d600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec58070860 con 0x7fec580b3c20 2024-01-16T14:58:36.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec57e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fec58159590 0x7fec580ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44584/0 (socket says 172.21.15.141:44584) 2024-01-16T14:58:36.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec57e00640 1 -- 172.21.15.141:0/1968915889 learned_addr learned my addr 172.21.15.141:0/1968915889 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:36.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec4f200640 1 --2- 172.21.15.141:0/1968915889 >> v2:172.21.15.141:3300/0 conn(0x7fec580b3c20 0x7fec58070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.085+0000 7fec57400640 1 -- 172.21.15.141:0/1968915889 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826743200 0 0) 0x7fec58071790 con 0x7fec58159590 2024-01-16T14:58:36.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.086+0000 7fec57400640 1 -- 172.21.15.141:0/1968915889 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec38005730 con 0x7fec58159590 2024-01-16T14:58:36.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.086+0000 7fec4f200640 1 -- 172.21.15.141:0/1968915889 >> v1:172.21.15.188:6789/0 conn(0x7fec5806d050 legacy=0x7fec5806d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.086+0000 7fec4f200640 1 -- 172.21.15.141:0/1968915889 >> v1:172.21.15.141:6789/0 conn(0x7fec58159590 legacy=0x7fec580ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.086+0000 7fec4f200640 1 -- 172.21.15.141:0/1968915889 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58070b40 con 0x7fec580b3c20 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.087+0000 7fec4f200640 1 --2- 172.21.15.141:0/1968915889 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec580b3c20 0x7fec58070160 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fec400097a0 tx=0x7fec4002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba62f199023ded9a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.087+0000 7fec57400640 1 -- 172.21.15.141:0/1968915889 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4002e070 con 0x7fec580b3c20 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.087+0000 7fec57400640 1 -- 172.21.15.141:0/1968915889 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec400315a0 con 0x7fec580b3c20 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.087+0000 7fec57400640 1 -- 172.21.15.141:0/1968915889 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40031930 con 0x7fec580b3c20 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.087+0000 7fec5d600640 1 -- 172.21.15.141:0/1968915889 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec580b3c20 msgr2=0x7fec58070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.087+0000 7fec5d600640 1 --2- 172.21.15.141:0/1968915889 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec580b3c20 0x7fec58070160 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fec400097a0 tx=0x7fec4002b840 comp rx=0 tx=0).stop 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.088+0000 7fec5d600640 1 -- 172.21.15.141:0/1968915889 shutdown_connections 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.088+0000 7fec5d600640 1 --2- 172.21.15.141:0/1968915889 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec580b3c20 0x7fec58070160 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.088+0000 7fec5d600640 1 -- 172.21.15.141:0/1968915889 >> 172.21.15.141:0/1968915889 conn(0x7fec5806a800 msgr2=0x7fec5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.088+0000 7fec5d600640 1 -- 172.21.15.141:0/1968915889 shutdown_connections 2024-01-16T14:58:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.088+0000 7fec5d600640 1 -- 172.21.15.141:0/1968915889 wait complete. 2024-01-16T14:58:36.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.089+0000 7fec5d600640 1 Processor -- start 2024-01-16T14:58:36.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.090+0000 7fec5d600640 1 -- start start 2024-01-16T14:58:36.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.090+0000 7fec5d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec5806d050 0x7fec58152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.090+0000 7fec5d600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec580b3c20 0x7fec58154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.090+0000 7fec5d600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec58159590 0x7fec5814e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.090+0000 7fec5d600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec581f3cd0 con 0x7fec5806d050 2024-01-16T14:58:36.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.090+0000 7fec5d600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec581f2e00 con 0x7fec58159590 2024-01-16T14:58:36.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.090+0000 7fec5d600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec581f3050 con 0x7fec580b3c20 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec5806d050 0x7fec58152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec5806d050 0x7fec58152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44272/0 (socket says 172.21.15.141:44272) 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 -- 172.21.15.141:0/2896279945 learned_addr learned my addr 172.21.15.141:0/2896279945 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 -- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec580b3c20 msgr2=0x7fec58154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec580b3c20 0x7fec58154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 -- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec58159590 msgr2=0x7fec5814e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec58159590 0x7fec5814e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 -- 172.21.15.141:0/2896279945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec40009400 con 0x7fec5806d050 2024-01-16T14:58:36.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.091+0000 7fec57e00640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec5806d050 0x7fec58152aa0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fec4800c430 tx=0x7fec4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.092+0000 7fec54c00640 1 -- 172.21.15.141:0/2896279945 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48005ce0 con 0x7fec5806d050 2024-01-16T14:58:36.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.092+0000 7fec54c00640 1 -- 172.21.15.141:0/2896279945 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec48005e40 con 0x7fec5806d050 2024-01-16T14:58:36.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.092+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec581f3330 con 0x7fec5806d050 2024-01-16T14:58:36.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.092+0000 7fec54c00640 1 -- 172.21.15.141:0/2896279945 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec480072d0 con 0x7fec5806d050 2024-01-16T14:58:36.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.092+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec581f38f0 con 0x7fec5806d050 2024-01-16T14:58:36.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.093+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec58150080 con 0x7fec5806d050 2024-01-16T14:58:36.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.094+0000 7fec54c00640 1 -- 172.21.15.141:0/2896279945 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fec480074f0 con 0x7fec5806d050 2024-01-16T14:58:36.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.095+0000 7fec54c00640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec3006b2c0 0x7fec3006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.095+0000 7fec54c00640 1 -- 172.21.15.141:0/2896279945 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec48089120 con 0x7fec5806d050 2024-01-16T14:58:36.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.096+0000 7fec4f200640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec3006b2c0 0x7fec3006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.097+0000 7fec4f200640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec3006b2c0 0x7fec3006d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fec40036000 tx=0x7fec40030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.099+0000 7fec54c00640 1 -- 172.21.15.141:0/2896279945 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec48054030 con 0x7fec5806d050 2024-01-16T14:58:36.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.325+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec580cde30 con 0x7fec5806d050 2024-01-16T14:58:36.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.327+0000 7fec54c00640 1 -- 172.21.15.141:0/2896279945 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fec48057650 con 0x7fec5806d050 2024-01-16T14:58:36.309 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:58:36.309 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/2661986774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":2661986774},{"type":"v1","addr":"172.21.15.188:6836","nonce":2661986774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4586,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2344909439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2344909439},{"type":"v1","addr":"172.21.15.188:6837","nonce":2344909439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/1479821799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":1479821799},{"type":"v1","addr":"172.21.15.141:6834","nonce":1479821799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:58:24.852726+0000","modified":"2024-01-16T14:58:27.938767+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4762":{"gid":4762,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.141:6835/176725955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":176725955},{"type":"v1","addr":"172.21.15.141:6835","nonce":176725955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T14:58:36.309 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 17 2024-01-16T14:58:36.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec3006b2c0 msgr2=0x7fec3006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec3006b2c0 0x7fec3006d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fec40036000 tx=0x7fec40030000 comp rx=0 tx=0).stop 2024-01-16T14:58:36.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec5806d050 msgr2=0x7fec58152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec5806d050 0x7fec58152aa0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fec4800c430 tx=0x7fec4800c8f0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 shutdown_connections 2024-01-16T14:58:36.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec3006b2c0 0x7fec3006d780 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec58159590 0x7fec5814e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec580b3c20 0x7fec58154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 --2- 172.21.15.141:0/2896279945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec5806d050 0x7fec58152aa0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.329+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 >> 172.21.15.141:0/2896279945 conn(0x7fec5806a800 msgr2=0x7fec58071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:36.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.330+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 shutdown_connections 2024-01-16T14:58:36.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.330+0000 7fec5d600640 1 -- 172.21.15.141:0/2896279945 wait complete. 2024-01-16T14:58:36.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T14:58:24.852726+0000', 'modified': '2024-01-16T14:58:27.938767+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4762}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4762': {'gid': 4762, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.141:6835/176725955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6833', 'nonce': 176725955}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 176725955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T14:58:36.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T14:58:36.733 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T14:58:36.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.969+0000 7fca62e00640 1 Processor -- start 2024-01-16T14:58:36.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.969+0000 7fca62e00640 1 -- start start 2024-01-16T14:58:36.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca62e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c13df80 0x7fca5c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca62e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fca5c0cf9a0 0x7fca5c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca62e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5c06c470 con 0x7fca5c13e360 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca62e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5c06c750 con 0x7fca5c0cf9a0 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca62e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c06ca30 con 0x7fca5c13df80 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca60e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fca5c0cf9a0 0x7fca5c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca60e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fca5c0cf9a0 0x7fca5c0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44284/0 (socket says 172.21.15.141:44284) 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.971+0000 7fca60e00640 1 -- 172.21.15.141:0/2167158318 learned_addr learned my addr 172.21.15.141:0/2167158318 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.972+0000 7fca60e00640 1 -- 172.21.15.141:0/2167158318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c13df80 msgr2=0x7fca5c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.972+0000 7fca60e00640 1 --2- 172.21.15.141:0/2167158318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c13df80 0x7fca5c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.972+0000 7fca60e00640 1 -- 172.21.15.141:0/2167158318 >> v1:172.21.15.188:6789/0 conn(0x7fca5c13e360 legacy=0x7fca5c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.972+0000 7fca60e00640 1 -- 172.21.15.141:0/2167158318 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c06cd10 con 0x7fca5c0cf9a0 2024-01-16T14:58:36.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.973+0000 7fca60e00640 1 --2- 172.21.15.141:0/2167158318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c0cf9a0 0x7fca5c0cd840 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fca54009e50 tx=0x7fca5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd39c611c3291fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.973+0000 7fca5aa00640 1 -- 172.21.15.141:0/2167158318 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5402ca70 con 0x7fca5c0cf9a0 2024-01-16T14:58:36.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.973+0000 7fca5aa00640 1 -- 172.21.15.141:0/2167158318 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5402cbd0 con 0x7fca5c0cf9a0 2024-01-16T14:58:36.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.973+0000 7fca5aa00640 1 -- 172.21.15.141:0/2167158318 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca540343f0 con 0x7fca5c0cf9a0 2024-01-16T14:58:36.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 -- 172.21.15.141:0/2167158318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c0cf9a0 msgr2=0x7fca5c0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 --2- 172.21.15.141:0/2167158318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c0cf9a0 0x7fca5c0cd840 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fca54009e50 tx=0x7fca5402bed0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 -- 172.21.15.141:0/2167158318 shutdown_connections 2024-01-16T14:58:36.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 --2- 172.21.15.141:0/2167158318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c0cf9a0 0x7fca5c0cd840 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 --2- 172.21.15.141:0/2167158318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c13df80 0x7fca5c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 -- 172.21.15.141:0/2167158318 >> 172.21.15.141:0/2167158318 conn(0x7fca5c06a800 msgr2=0x7fca5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:36.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 -- 172.21.15.141:0/2167158318 shutdown_connections 2024-01-16T14:58:36.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.974+0000 7fca62e00640 1 -- 172.21.15.141:0/2167158318 wait complete. 2024-01-16T14:58:36.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.976+0000 7fca62e00640 1 Processor -- start 2024-01-16T14:58:36.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.976+0000 7fca62e00640 1 -- start start 2024-01-16T14:58:36.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca62e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c0cf9a0 0x7fca5c070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca62e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca5c13df80 0x7fca5c1f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca62e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c13e360 0x7fca5c0a9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca62e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c1f6490 con 0x7fca5c13e360 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca62e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c0a97d0 con 0x7fca5c13df80 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca62e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c0713d0 con 0x7fca5c0cf9a0 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca60e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c13e360 0x7fca5c0a9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca60e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c13e360 0x7fca5c0a9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44300/0 (socket says 172.21.15.141:44300) 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.977+0000 7fca60e00640 1 -- 172.21.15.141:0/3883461624 learned_addr learned my addr 172.21.15.141:0/3883461624 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca5be00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c0cf9a0 0x7fca5c070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca5b400640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca5c13df80 0x7fca5c1f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca60e00640 1 -- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c0cf9a0 msgr2=0x7fca5c070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca60e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c0cf9a0 0x7fca5c070cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca60e00640 1 -- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca5c13df80 msgr2=0x7fca5c1f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca60e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca5c13df80 0x7fca5c1f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca60e00640 1 -- 172.21.15.141:0/3883461624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca54009b30 con 0x7fca5c13e360 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca5b400640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca5c13df80 0x7fca5c1f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca60e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c13e360 0x7fca5c0a9290 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fca54036040 tx=0x7fca54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca53e00640 1 -- 172.21.15.141:0/3883461624 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5403e450 con 0x7fca5c13e360 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca53e00640 1 -- 172.21.15.141:0/3883461624 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5403e5b0 con 0x7fca5c13e360 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca53e00640 1 -- 172.21.15.141:0/3883461624 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5403e960 con 0x7fca5c13e360 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.978+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5c0716b0 con 0x7fca5c13e360 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.979+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5c071c70 con 0x7fca5c13e360 2024-01-16T14:58:36.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.980+0000 7fca53e00640 1 -- 172.21.15.141:0/3883461624 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fca5403eac0 con 0x7fca5c13e360 2024-01-16T14:58:36.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.981+0000 7fca53e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca3406b2f0 0x7fca3406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.981+0000 7fca53e00640 1 -- 172.21.15.141:0/3883461624 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca540ac760 con 0x7fca5c13e360 2024-01-16T14:58:36.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.982+0000 7fca5be00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca3406b2f0 0x7fca3406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.982+0000 7fca65b93740 1 Processor -- start 2024-01-16T14:58:36.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.982+0000 7fca65b93740 1 -- start start 2024-01-16T14:58:36.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.983+0000 7fca5be00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca3406b2f0 0x7fca3406d7b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00a810 tx=0x7fca4c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.983+0000 7fca65b93740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680416ea0 0x55c680417280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.983+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680411ba0 0x55c6804160a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.983+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 0x55c68045a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.983+0000 7fca65b93740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c680375880 con 0x55c680416ea0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.983+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c680384ae0 con 0x55c6804165e0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.983+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6803821d0 con 0x55c680411ba0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680416ea0 0x55c680417280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680416ea0 0x55c680417280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44306/0 (socket says 172.21.15.141:44306) 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 -- 172.21.15.141:0/2949889585 learned_addr learned my addr 172.21.15.141:0/2949889585 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 -- 172.21.15.141:0/2949889585 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680411ba0 msgr2=0x55c6804160a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 --2- 172.21.15.141:0/2949889585 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680411ba0 0x55c6804160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 -- 172.21.15.141:0/2949889585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 msgr2=0x55c68045a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 --2- 172.21.15.141:0/2949889585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 0x55c68045a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 -- 172.21.15.141:0/2949889585 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c680345860 con 0x55c680416ea0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca5be00640 1 --2- 172.21.15.141:0/2949889585 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680416ea0 0x55c680417280 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00cac0 tx=0x7fca4c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ffba0ea426aae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.984+0000 7fca50c00640 1 -- 172.21.15.141:0/2949889585 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c013db0 con 0x55c680416ea0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca50c00640 1 -- 172.21.15.141:0/2949889585 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4c01e400 con 0x55c680416ea0 2024-01-16T14:58:36.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca50c00640 1 -- 172.21.15.141:0/2949889585 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c01e790 con 0x55c680416ea0 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 -- 172.21.15.141:0/2949889585 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680416ea0 msgr2=0x55c680417280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 --2- 172.21.15.141:0/2949889585 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680416ea0 0x55c680417280 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00cac0 tx=0x7fca4c00f000 comp rx=0 tx=0).stop 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 -- 172.21.15.141:0/2949889585 shutdown_connections 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 --2- 172.21.15.141:0/2949889585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 0x55c68045a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 --2- 172.21.15.141:0/2949889585 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680411ba0 0x55c6804160a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 --2- 172.21.15.141:0/2949889585 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680416ea0 0x55c680417280 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 -- 172.21.15.141:0/2949889585 >> 172.21.15.141:0/2949889585 conn(0x55c6803d8c70 msgr2=0x55c68045c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 -- 172.21.15.141:0/2949889585 shutdown_connections 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.985+0000 7fca65b93740 1 -- 172.21.15.141:0/2949889585 wait complete. 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.986+0000 7fca65b93740 1 -- start start 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.986+0000 7fca65b93740 1 Processor -- start 2024-01-16T14:58:36.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.987+0000 7fca65b93740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680464570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.987+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 0x55c680455030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.987+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680416ea0 0x55c680463920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.987+0000 7fca65b93740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68037dac0 con 0x55c680411ba0 2024-01-16T14:58:36.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.987+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68031a2f0 con 0x55c6804165e0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.987+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c680337c70 con 0x55c680416ea0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.987+0000 7fca65b93740 10 client.0 _ll_register_callbacks cb 0x55c68033c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680464570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680464570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44322/0 (socket says 172.21.15.141:44322) 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 -- 172.21.15.141:0/3315087608 learned_addr learned my addr 172.21.15.141:0/3315087608 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca60e00640 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680416ea0 0x55c680463920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5b400640 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 0x55c680455030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 -- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680416ea0 msgr2=0x55c680463920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680416ea0 0x55c680463920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 -- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 msgr2=0x55c680455030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 0x55c680455030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 -- 172.21.15.141:0/3315087608 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c00c7d0 con 0x55c680411ba0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca5be00640 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680464570 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fca4c021040 tx=0x7fca4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8139b065dff2bab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.988+0000 7fca49600640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca49600640 1 -- 172.21.15.141:0/3315087608 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c01eb40 con 0x55c680411ba0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c680376aa0 con 0x55c680411ba0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca49600640 1 -- 172.21.15.141:0/3315087608 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca4c01eca0 con 0x55c680411ba0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca49600640 1 -- 172.21.15.141:0/3315087608 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca4c01ee60 con 0x55c680411ba0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 10 client.5451 fetch_fsmap learned FSMap version 17 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55c68037c140 con 0x55c680411ba0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca49600640 1 -- 172.21.15.141:0/3315087608 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c680376aa0 con 0x55c680411ba0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca49600640 1 -- 172.21.15.141:0/3315087608 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fca4c027400 con 0x55c680411ba0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 10 client.5451 fetch_fsmap finished waiting for FSMap version 17 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 10 client.5451 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 20 client.5451 populate_metadata read hostname 'smithi141' 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c68043d0a0 0x55c680461060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 4 client.5451 mds_command: new command op to 4762 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.989+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 --> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6803767c0 con 0x55c68043d0a0 2024-01-16T14:58:36.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.990+0000 7fca5b400640 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c68043d0a0 0x55c680461060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.990+0000 7fca5b400640 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c68043d0a0 0x55c680461060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af72562bb927ec3a server_cookie=826a9df0c1a60a56 in_seq=0 out_seq=0 2024-01-16T14:58:36.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.990+0000 7fca49600640 10 client.5451 ms_handle_connect on v2:172.21.15.141:6833/176725955 2024-01-16T14:58:36.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca49600640 1 -- 172.21.15.141:0/3315087608 <== mds.0 v2:172.21.15.141:6833/176725955 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c6803767c0 con 0x55c68043d0a0 2024-01-16T14:58:36.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca49600640 10 client.5451 handle_command_reply: tid=0 2024-01-16T14:58:36.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 client.5451 shutdown 2024-01-16T14:58:36.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 msgr2=0x55c680464570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680464570 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fca4c021040 tx=0x7fca4c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 shutdown_connections 2024-01-16T14:58:36.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c68043d0a0 0x55c680461060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c680416ea0 0x55c680463920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6804165e0 0x55c680455030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca49600640 0 client.5451 ms_handle_reset on v2:172.21.15.141:6833/176725955 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 --2- 172.21.15.141:0/3315087608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680464570 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 >> 172.21.15.141:0/3315087608 conn(0x55c6803d8c70 msgr2=0x55c680410b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 shutdown_connections 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.991+0000 7fca65b93740 1 -- 172.21.15.141:0/3315087608 wait complete. 2024-01-16T14:58:36.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:36.992+0000 7fca65b93740 20 client.5451 trim_cache size 0 max 16384 2024-01-16T14:58:36.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 Processor -- start 2024-01-16T14:58:36.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 -- start start 2024-01-16T14:58:36.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680455e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 0x55c680456390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c680410b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c680414100 con 0x55c680411ba0 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6800e0f20 con 0x55c680416ea0 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.009+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c680463790 con 0x55c6804165e0 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680455e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680455e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44324/0 (socket says 172.21.15.141:44324) 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 -- 172.21.15.141:0/268823554 learned_addr learned my addr 172.21.15.141:0/268823554 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca60e00640 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c680410b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5b400640 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 0x55c680456390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 -- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 msgr2=0x55c680456390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 0x55c680456390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 -- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 msgr2=0x55c680410b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c680410b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.010+0000 7fca5be00640 1 -- 172.21.15.141:0/268823554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c680319b80 con 0x55c680411ba0 2024-01-16T14:58:36.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca5be00640 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680455e50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fca4c013920 tx=0x7fca4c02b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b405eb5d112e037 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca51600640 1 -- 172.21.15.141:0/268823554 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c040070 con 0x55c680411ba0 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca51600640 1 -- 172.21.15.141:0/268823554 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4c017070 con 0x55c680411ba0 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca51600640 1 -- 172.21.15.141:0/268823554 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c00c960 con 0x55c680411ba0 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca65b93740 1 -- 172.21.15.141:0/268823554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 msgr2=0x55c680455e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca65b93740 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680455e50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fca4c013920 tx=0x7fca4c02b000 comp rx=0 tx=0).stop 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca65b93740 1 -- 172.21.15.141:0/268823554 shutdown_connections 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca65b93740 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c680410b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca65b93740 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 0x55c680456390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca65b93740 1 --2- 172.21.15.141:0/268823554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c680455e50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.011+0000 7fca65b93740 1 -- 172.21.15.141:0/268823554 >> 172.21.15.141:0/268823554 conn(0x55c6803d8c70 msgr2=0x55c68045c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:36.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.012+0000 7fca65b93740 1 -- 172.21.15.141:0/268823554 shutdown_connections 2024-01-16T14:58:36.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.012+0000 7fca65b93740 1 -- 172.21.15.141:0/268823554 wait complete. 2024-01-16T14:58:36.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.012+0000 7fca65b93740 1 -- start start 2024-01-16T14:58:36.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.012+0000 7fca65b93740 1 Processor -- start 2024-01-16T14:58:36.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.013+0000 7fca65b93740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c6804c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:36.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.013+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 0x55c680440bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.013+0000 7fca65b93740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c68043c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.013+0000 7fca65b93740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68043b480 con 0x55c680411ba0 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.013+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6804415c0 con 0x55c680416ea0 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.013+0000 7fca65b93740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6804c94e0 con 0x55c6804165e0 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.013+0000 7fca65b93740 10 client.0 _ll_register_callbacks cb 0x7fca4c021070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c6804c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c6804c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44336/0 (socket says 172.21.15.141:44336) 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 -- 172.21.15.141:0/812132345 learned_addr learned my addr 172.21.15.141:0/812132345 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 -- 172.21.15.141:0/812132345 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 msgr2=0x55c680440bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 0x55c680440bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 -- 172.21.15.141:0/812132345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 msgr2=0x55c68043c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:37.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca60e00640 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c68043c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c68043c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 -- 172.21.15.141:0/812132345 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c68037c140 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca5be00640 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c6804c8fa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fca4c0137f0 tx=0x7fca4c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ff1e3eca3e923 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca49600640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca49600640 1 -- 172.21.15.141:0/812132345 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c040070 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca49600640 1 -- 172.21.15.141:0/812132345 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca4c027400 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.014+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca4c01e400 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca49600640 1 -- 172.21.15.141:0/812132345 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca4c0059c0 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 10 client.5457 fetch_fsmap learned FSMap version 17 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55c68033f170 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca49600640 1 -- 172.21.15.141:0/812132345 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4c01e400 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca49600640 1 -- 172.21.15.141:0/812132345 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fca4c005b20 con 0x55c680411ba0 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 10 client.5457 fetch_fsmap finished waiting for FSMap version 17 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 10 client.5457 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T14:58:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 20 client.5457 populate_metadata read hostname 'smithi141' 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c6804631e0 0x55c680410b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 4 client.5457 mds_command: new command op to 4762 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 --> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c68007e6c0 con 0x55c6804631e0 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.015+0000 7fca5b400640 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c6804631e0 0x55c680410b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca5b400640 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c6804631e0 0x55c680410b80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa58fb42e64fbd4c server_cookie=3a76eed442988632 in_seq=0 out_seq=0 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca49600640 10 client.5457 ms_handle_connect on v2:172.21.15.141:6833/176725955 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca49600640 1 -- 172.21.15.141:0/812132345 <== mds.0 v2:172.21.15.141:6833/176725955 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c68007e6c0 con 0x55c6804631e0 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca49600640 10 client.5457 handle_command_reply: tid=0 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 client.5457 shutdown 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 msgr2=0x55c6804c8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c6804c8fa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fca4c0137f0 tx=0x7fca4c012000 comp rx=0 tx=0).stop 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 shutdown_connections 2024-01-16T14:58:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:6833/176725955,v1:172.21.15.141:6835/176725955] conn(0x55c6804631e0 0x55c680410b80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c680416ea0 0x55c68043c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55c6804165e0 0x55c680440bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca49600640 0 client.5457 ms_handle_reset on v2:172.21.15.141:6833/176725955 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 --2- 172.21.15.141:0/812132345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c680411ba0 0x55c6804c8fa0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.016+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 >> 172.21.15.141:0/812132345 conn(0x55c6803d8c70 msgr2=0x55c680412030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 shutdown_connections 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca65b93740 1 -- 172.21.15.141:0/812132345 wait complete. 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca65b93740 20 client.5457 trim_cache size 0 max 16384 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T14:58:37.003 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4762, 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 11.579536641000001, 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 17, 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 35, 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 35, 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 20.591568898999999 2024-01-16T14:58:37.004 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:58:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca3406b2f0 msgr2=0x7fca3406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca3406b2f0 0x7fca3406d7b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00a810 tx=0x7fca4c008040 comp rx=0 tx=0).stop 2024-01-16T14:58:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c13e360 msgr2=0x7fca5c0a9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c13e360 0x7fca5c0a9290 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fca54036040 tx=0x7fca54004a40 comp rx=0 tx=0).stop 2024-01-16T14:58:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 shutdown_connections 2024-01-16T14:58:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca3406b2f0 0x7fca3406d7b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca5c13e360 0x7fca5c0a9290 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca5c13df80 0x7fca5c1f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 --2- 172.21.15.141:0/3883461624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca5c0cf9a0 0x7fca5c070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 >> 172.21.15.141:0/3883461624 conn(0x7fca5c06a800 msgr2=0x7fca5c15e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.017+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 shutdown_connections 2024-01-16T14:58:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.018+0000 7fca62e00640 1 -- 172.21.15.141:0/3883461624 wait complete. 2024-01-16T14:58:37.016 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:58:37.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.225+0000 7fad89e00640 1 Processor -- start 2024-01-16T14:58:37.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.225+0000 7fad89e00640 1 -- start start 2024-01-16T14:58:37.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.226+0000 7fad89e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 0x7fad84166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad89e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8415c5f0 con 0x7fad8413e360 2024-01-16T14:58:37.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad89e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8406c470 con 0x7fad8413df80 2024-01-16T14:58:37.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad89e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8406c750 con 0x7fad840cf9a0 2024-01-16T14:58:37.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad82000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fad8413e360 0x7fad840cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44592/0 (socket says 172.21.15.141:44592) 2024-01-16T14:58:37.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad82000640 1 -- 172.21.15.141:0/2837757458 learned_addr learned my addr 172.21.15.141:0/2837757458 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428483817 0 0) 0x7fad8415c5f0 con 0x7fad8413e360 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad64003520 con 0x7fad8413e360 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.227+0000 7fad83400640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 0x7fad84166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 707782314 0 0) 0x7fad64003520 con 0x7fad8413e360 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 msgr2=0x7fad84166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 0x7fad84166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 >> v1:172.21.15.188:6789/0 conn(0x7fad8413df80 legacy=0x7fad8415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8406ca30 con 0x7fad8413e360 2024-01-16T14:58:37.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fad6c001670 con 0x7fad8413e360 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 >> v1:172.21.15.141:6789/0 conn(0x7fad8413e360 legacy=0x7fad840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad640039a0 0x7fad64003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 0x7fad640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad64004d00 0x7fad64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8406c750 con 0x7fad64004d00 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6400ba60 con 0x7fad640039a0 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6400bc20 con 0x7fad840cf9a0 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad83400640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad64004d00 0x7fad64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.228+0000 7fad82000640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 0x7fad640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.229+0000 7fad83400640 1 -- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 msgr2=0x7fad640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.229+0000 7fad83400640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 0x7fad640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.229+0000 7fad83400640 1 -- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad640039a0 msgr2=0x7fad64003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.229+0000 7fad83400640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad640039a0 0x7fad64003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.229+0000 7fad83400640 1 -- 172.21.15.141:0/2837757458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad64003520 con 0x7fad64004d00 2024-01-16T14:58:37.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.229+0000 7fad82000640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad840cf9a0 0x7fad640047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:37.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad83400640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad64004d00 0x7fad64005180 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fad84155d40 tx=0x7fad640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854b9582b55d537e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:37.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c02cb90 con 0x7fad64004d00 2024-01-16T14:58:37.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c00f040 con 0x7fad64004d00 2024-01-16T14:58:37.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad81600640 1 -- 172.21.15.141:0/2837757458 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c034440 con 0x7fad64004d00 2024-01-16T14:58:37.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad89e00640 1 -- 172.21.15.141:0/2837757458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad64004d00 msgr2=0x7fad64005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad89e00640 1 --2- 172.21.15.141:0/2837757458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad64004d00 0x7fad64005180 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fad84155d40 tx=0x7fad640057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad83400640 1 -- 172.21.15.141:0/2837757458 reap_dead start 2024-01-16T14:58:37.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad89e00640 1 -- 172.21.15.141:0/2837757458 shutdown_connections 2024-01-16T14:58:37.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad89e00640 1 -- 172.21.15.141:0/2837757458 >> 172.21.15.141:0/2837757458 conn(0x7fad8406a800 msgr2=0x7fad8406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:37.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.230+0000 7fad89e00640 1 -- 172.21.15.141:0/2837757458 shutdown_connections 2024-01-16T14:58:37.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.231+0000 7fad89e00640 1 -- 172.21.15.141:0/2837757458 wait complete. 2024-01-16T14:58:37.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.232+0000 7fad89e00640 1 Processor -- start 2024-01-16T14:58:37.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.232+0000 7fad89e00640 1 -- start start 2024-01-16T14:58:37.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.233+0000 7fad89e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad64004d00 0x7fad84154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.233+0000 7fad89e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad841f31b0 0x7fad840b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.233+0000 7fad89e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad841f3610 0x7fad840b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.233+0000 7fad89e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840aa410 con 0x7fad841f3610 2024-01-16T14:58:37.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.233+0000 7fad89e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840aa6f0 con 0x7fad64004d00 2024-01-16T14:58:37.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.233+0000 7fad89e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad840aa9d0 con 0x7fad841f31b0 2024-01-16T14:58:37.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.233+0000 7fad83400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad841f3610 0x7fad840b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad841f3610 0x7fad840b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44340/0 (socket says 172.21.15.141:44340) 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 -- 172.21.15.141:0/207516733 learned_addr learned my addr 172.21.15.141:0/207516733 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad82000640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad841f31b0 0x7fad840b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad82a00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad64004d00 0x7fad84154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 -- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad841f31b0 msgr2=0x7fad840b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad841f31b0 0x7fad840b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 -- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad64004d00 msgr2=0x7fad84154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad64004d00 0x7fad84154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 -- 172.21.15.141:0/207516733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad840aacb0 con 0x7fad841f3610 2024-01-16T14:58:37.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad82a00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad64004d00 0x7fad84154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:37.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.234+0000 7fad83400640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad841f3610 0x7fad840b3230 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fad7c037040 tx=0x7fad7c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:37.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.235+0000 7fad7b400640 1 -- 172.21.15.141:0/207516733 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c034cd0 con 0x7fad841f3610 2024-01-16T14:58:37.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.235+0000 7fad7b400640 1 -- 172.21.15.141:0/207516733 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c034e30 con 0x7fad841f3610 2024-01-16T14:58:37.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.235+0000 7fad7b400640 1 -- 172.21.15.141:0/207516733 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c003bd0 con 0x7fad841f3610 2024-01-16T14:58:37.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.235+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad840aaf90 con 0x7fad841f3610 2024-01-16T14:58:37.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.235+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad840a9770 con 0x7fad841f3610 2024-01-16T14:58:37.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.236+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad840ce7a0 con 0x7fad841f3610 2024-01-16T14:58:37.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.237+0000 7fad7b400640 1 -- 172.21.15.141:0/207516733 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fad7c003d30 con 0x7fad841f3610 2024-01-16T14:58:37.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.237+0000 7fad7b400640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad5c06b2c0 0x7fad5c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:37.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.238+0000 7fad7b400640 1 -- 172.21.15.141:0/207516733 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad7c0ab220 con 0x7fad841f3610 2024-01-16T14:58:37.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.238+0000 7fad82a00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad5c06b2c0 0x7fad5c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:37.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.239+0000 7fad82a00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad5c06b2c0 0x7fad5c06d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fad740016a0 tx=0x7fad74009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:37.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.242+0000 7fad7b400640 1 -- 172.21.15.141:0/207516733 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad7c0760b0 con 0x7fad841f3610 2024-01-16T14:58:37.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.469+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fad840ab180 con 0x7fad841f3610 2024-01-16T14:58:37.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.470+0000 7fad7b400640 1 -- 172.21.15.141:0/207516733 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fad7c045060 con 0x7fad841f3610 2024-01-16T14:58:37.452 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:58:37.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.472+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad5c06b2c0 msgr2=0x7fad5c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.472+0000 7fad89e00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad5c06b2c0 0x7fad5c06d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fad740016a0 tx=0x7fad74009040 comp rx=0 tx=0).stop 2024-01-16T14:58:37.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad841f3610 msgr2=0x7fad840b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:37.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad841f3610 0x7fad840b3230 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fad7c037040 tx=0x7fad7c009750 comp rx=0 tx=0).stop 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 shutdown_connections 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad5c06b2c0 0x7fad5c06d780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad841f3610 0x7fad840b3230 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad841f31b0 0x7fad840b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 --2- 172.21.15.141:0/207516733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad64004d00 0x7fad84154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 >> 172.21.15.141:0/207516733 conn(0x7fad8406a800 msgr2=0x7fad840ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 shutdown_connections 2024-01-16T14:58:37.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:37.473+0000 7fad89e00640 1 -- 172.21.15.141:0/207516733 wait complete. 2024-01-16T14:58:37.472 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:58:37.472 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:58:37.473 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T14:58:37.473 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T14:58:37.473 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:37.473 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:58:37.473 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:58:37.473 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:58:37.473 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:58:37.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:37.473 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:37.520 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42546sec preferred_lft 42546sec 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:37.521 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::882:e9ff:fe92:c54c/64 scope link 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:37.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:37.522 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:58:37.647 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T14:58:37.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:37.652 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T14:58:37.706 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T14:58:37.706 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:58:37.706 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:58:37.706 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:58:37.706 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:58:37.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:37.707 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T14:58:37.707 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T14:58:37.707 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:58:37.707 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:58:37.707 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:58:37.707 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T14:58:37.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:37.886 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T14:58:37.937 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T14:58:37.937 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T14:58:37.938 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:58:37.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:37.938 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T14:58:38.030 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T14:58:38.030 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:38.030 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:38.085 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:58:38.085 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:38.085 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:38.135 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T14:58:38.229 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T14:58:38.284 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T14:58:38.285 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:38.286 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:38.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:38.287 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T14:58:38.336 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:58:38.336 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T14:58:38.341 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T14:58:38.415 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:38.459 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T14:58:38.459 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T14:58:38.459 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T14:58:38.460 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:38.461 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:58:38.462 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:38.462 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:58:38.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:38.463 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T14:58:38.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T14:58:38.539+0000 7fad3a64a480 -1 init, newargv = 0x5559192835f0 newargc=15 2024-01-16T14:58:38.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[49299]: starting ceph client 2024-01-16T14:58:38.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[49299]: starting fuse 2024-01-16T14:58:38.547 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T14:58:38.547 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T14:58:38.547 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:58:38.547 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T14:58:38.547 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T14:58:38.548 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T14:58:38.549 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T14:58:38.740 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.49299.asok 2024-01-16T14:58:38.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:58:38.757 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.49299.asok 2024-01-16T14:58:38.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:38.758 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.49299.asok status 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "49299", 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5472, 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 5472 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 1207727132 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 1207727132 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.5472 192.168.0.1:0/1207727132", 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/1207727132", 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 17, 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 35, 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:58:39.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:39.008 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T14:58:39.008 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:39.059 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T14:58:39.059 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:39.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:39.059 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:58:39.152 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:58:39.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.377+0000 7efef8600640 1 Processor -- start 2024-01-16T14:58:39.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.377+0000 7efef8600640 1 -- start start 2024-01-16T14:58:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef8600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7efef00ae5a0 0x7efef015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef8600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efef00cf290 0x7efef00cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef8600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef0070c40 con 0x7efef00b3ae0 2024-01-16T14:58:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef8600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef0071450 con 0x7efef00ae5a0 2024-01-16T14:58:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef8600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef00cfc60 con 0x7efef00cf290 2024-01-16T14:58:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef5200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7efef00b3ae0 0x7efef006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44606/0 (socket says 172.21.15.141:44606) 2024-01-16T14:58:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef5200640 1 -- 172.21.15.141:0/2909490436 learned_addr learned my addr 172.21.15.141:0/2909490436 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef5c00640 1 --2- 172.21.15.141:0/2909490436 >> v2:172.21.15.188:3300/0 conn(0x7efef00ae5a0 0x7efef015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.379+0000 7efef6600640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efef00cf290 0x7efef00cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758302504 0 0) 0x7efef0070c40 con 0x7efef00b3ae0 2024-01-16T14:58:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0003520 con 0x7efef00b3ae0 2024-01-16T14:58:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1336586397 0 0) 0x7efed0003520 con 0x7efef00b3ae0 2024-01-16T14:58:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efef00cf290 msgr2=0x7efef00cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efef00cf290 0x7efef00cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 >> v2:172.21.15.188:3300/0 conn(0x7efef00ae5a0 msgr2=0x7efef015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 --2- 172.21.15.141:0/2909490436 >> v2:172.21.15.188:3300/0 conn(0x7efef00ae5a0 0x7efef015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef006c470 con 0x7efef00b3ae0 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7efed8001670 con 0x7efef00b3ae0 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 >> v1:172.21.15.141:6789/0 conn(0x7efef00b3ae0 legacy=0x7efef006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efef00cf290 0x7efed0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efed0003ef0 0x7efed00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efed0004d00 0x7efed0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef0071450 con 0x7efed0004d00 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00cfc60 con 0x7efed0003ef0 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed000b9b0 con 0x7efef00cf290 2024-01-16T14:58:39.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.380+0000 7efef6600640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efed0004d00 0x7efed0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:39.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.381+0000 7efef6600640 1 -- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efef00cf290 msgr2=0x7efed0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:39.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.381+0000 7efef6600640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efef00cf290 0x7efed0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.381+0000 7efef6600640 1 -- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efed0003ef0 msgr2=0x7efed00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:39.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.381+0000 7efef6600640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efed0003ef0 0x7efed00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.381+0000 7efef6600640 1 -- 172.21.15.141:0/2909490436 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0003520 con 0x7efed0004d00 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.382+0000 7efef6600640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efed0004d00 0x7efed0005180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efee8009fc0 tx=0x7efed00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e429d758dd4e4541 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.382+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee802cc60 con 0x7efed0004d00 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.382+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee800f040 con 0x7efed0004d00 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.382+0000 7efeefe00640 1 -- 172.21.15.141:0/2909490436 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8034440 con 0x7efed0004d00 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.383+0000 7efef8600640 1 -- 172.21.15.141:0/2909490436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efed0004d00 msgr2=0x7efed0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.383+0000 7efef8600640 1 --2- 172.21.15.141:0/2909490436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efed0004d00 0x7efed0005180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efee8009fc0 tx=0x7efed00057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.383+0000 7efef6600640 1 -- 172.21.15.141:0/2909490436 reap_dead start 2024-01-16T14:58:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.383+0000 7efef8600640 1 -- 172.21.15.141:0/2909490436 shutdown_connections 2024-01-16T14:58:39.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.383+0000 7efef8600640 1 -- 172.21.15.141:0/2909490436 >> 172.21.15.141:0/2909490436 conn(0x7efef006a800 msgr2=0x7efef006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:39.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.383+0000 7efef8600640 1 -- 172.21.15.141:0/2909490436 shutdown_connections 2024-01-16T14:58:39.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.383+0000 7efef8600640 1 -- 172.21.15.141:0/2909490436 wait complete. 2024-01-16T14:58:39.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.385+0000 7efef8600640 1 Processor -- start 2024-01-16T14:58:39.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.385+0000 7efef8600640 1 -- start start 2024-01-16T14:58:39.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef8600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efed0004d00 0x7efef00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef8600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efef0158300 0x7efef0158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef8600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efef00b22c0 0x7efef00b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef8600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00cad50 con 0x7efef0158300 2024-01-16T14:58:39.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef8600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00b2c80 con 0x7efef00b22c0 2024-01-16T14:58:39.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef8600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef0157330 con 0x7efed0004d00 2024-01-16T14:58:39.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efef0158300 0x7efef0158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efef0158300 0x7efef0158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44346/0 (socket says 172.21.15.141:44346) 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef5200640 1 -- 172.21.15.141:0/2255570238 learned_addr learned my addr 172.21.15.141:0/2255570238 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef5c00640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efed0004d00 0x7efef00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef5200640 1 -- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efed0004d00 msgr2=0x7efef00ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef6600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efef00b22c0 0x7efef00b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef5200640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efed0004d00 0x7efef00ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.386+0000 7efef5200640 1 -- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efef00b22c0 msgr2=0x7efef00b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efef5200640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efef00b22c0 0x7efef00b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efef5200640 1 -- 172.21.15.141:0/2255570238 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0158dc0 con 0x7efef0158300 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efef5200640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efef0158300 0x7efef0158760 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7efed800c820 tx=0x7efed800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efeed600640 1 -- 172.21.15.141:0/2255570238 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed80074e0 con 0x7efef0158300 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efeed600640 1 -- 172.21.15.141:0/2255570238 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed8007640 con 0x7efef0158300 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efeed600640 1 -- 172.21.15.141:0/2255570238 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8004a00 con 0x7efef0158300 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef01590a0 con 0x7efef0158300 2024-01-16T14:58:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.387+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef0159660 con 0x7efef0158300 2024-01-16T14:58:39.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.388+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef00675d0 con 0x7efef0158300 2024-01-16T14:58:39.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.389+0000 7efeed600640 1 -- 172.21.15.141:0/2255570238 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efed8004b60 con 0x7efef0158300 2024-01-16T14:58:39.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.390+0000 7efeed600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efec806b2c0 0x7efec806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:39.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.390+0000 7efeed600640 1 -- 172.21.15.141:0/2255570238 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efed8089140 con 0x7efef0158300 2024-01-16T14:58:39.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.391+0000 7efef5c00640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efec806b2c0 0x7efec806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:39.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.392+0000 7efef5c00640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efec806b2c0 0x7efec806d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efee0004770 tx=0x7efee0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:39.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.395+0000 7efeed600640 1 -- 172.21.15.141:0/2255570238 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efed8053f50 con 0x7efef0158300 2024-01-16T14:58:39.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.623+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efef00cf8b0 con 0x7efef0158300 2024-01-16T14:58:39.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.624+0000 7efeed600640 1 -- 172.21.15.141:0/2255570238 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7efed8057570 con 0x7efef0158300 2024-01-16T14:58:39.606 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:58:39.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efec806b2c0 msgr2=0x7efec806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:39.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efec806b2c0 0x7efec806d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efee0004770 tx=0x7efee0008040 comp rx=0 tx=0).stop 2024-01-16T14:58:39.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efef0158300 msgr2=0x7efef0158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:39.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efef0158300 0x7efef0158760 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7efed800c820 tx=0x7efed800cce0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 shutdown_connections 2024-01-16T14:58:39.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efec806b2c0 0x7efec806d780 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efef00b22c0 0x7efef00b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efef0158300 0x7efef0158760 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 --2- 172.21.15.141:0/2255570238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efed0004d00 0x7efef00ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:39.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 >> 172.21.15.141:0/2255570238 conn(0x7efef006a800 msgr2=0x7efef00cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:39.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 shutdown_connections 2024-01-16T14:58:39.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:39.627+0000 7efef8600640 1 -- 172.21.15.141:0/2255570238 wait complete. 2024-01-16T14:58:39.626 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:58:39.626 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:58:39.626 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T14:58:39.626 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T14:58:39.626 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:39.626 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:58:39.627 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:58:39.627 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:58:39.627 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:58:39.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:39.627 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T14:58:39.684 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:58:39.684 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:58:39.684 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:58:39.684 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:39.684 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42546sec preferred_lft 42546sec 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:39.685 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::a40a:23ff:fed8:ea96/64 scope link 2024-01-16T14:58:39.686 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:58:39.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:39.687 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:58:39.778 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T14:58:39.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:39.782 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T14:58:39.835 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T14:58:39.835 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:58:39.835 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:58:39.836 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:58:39.836 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:58:39.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:39.836 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T14:58:39.836 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T14:58:39.836 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:58:39.836 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:58:39.836 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:58:39.836 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T14:58:40.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:40.014 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T14:58:40.062 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T14:58:40.062 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T14:58:40.063 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:58:40.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:40.063 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T14:58:40.150 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T14:58:40.150 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:40.151 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:40.205 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:58:40.205 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:58:40.206 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:40.254 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:58:40.344 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:58:40.403 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:58:40.404 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:40.405 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:40.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:40.406 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:58:40.456 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:58:40.456 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T14:58:40.461 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:58:40.538 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:40.585 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:58:40.585 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:58:40.585 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:58:40.585 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:58:40.585 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:58:40.586 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:40.587 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:40.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:40.588 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:58:40.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T14:58:40.661+0000 7fa433731480 -1 init, newargv = 0x560a421c1a90 newargc=15 2024-01-16T14:58:40.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[42607]: starting ceph client 2024-01-16T14:58:40.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[42607]: starting fuse 2024-01-16T14:58:41.728 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T14:58:41.829 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:58:41.885 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T14:58:41.886 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:58:41.887 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:58:41.888 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:41.889 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:58:41.889 INFO:teuthology.orchestra.run.smithi188.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:41.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:41.890 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T14:58:41.937 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T14:58:41.937 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T14:58:41.937 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:58:41.937 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T14:58:41.937 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T14:58:41.938 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T14:58:41.939 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T14:58:42.158 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.42607.asok 2024-01-16T14:58:42.163 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T14:58:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:58:42.176 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42607.asok 2024-01-16T14:58:42.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:42.177 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42607.asok status 2024-01-16T14:58:42.397 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "42607", 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 4840, 2024-01-16T14:58:42.398 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 4840 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 379408434 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:58:42.399 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 379408434 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.4840 192.168.0.1:0/379408434", 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/379408434", 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 17, 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 35, 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T14:58:42.400 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T14:58:42.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:42.412 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T14:58:42.412 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:42.463 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T14:58:42.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:42.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:42.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:58:42.556 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T14:58:42.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.800+0000 7f5947400640 1 Processor -- start 2024-01-16T14:58:42.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.800+0000 7f5947400640 1 -- start start 2024-01-16T14:58:42.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.802+0000 7f5947400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 0x7f59400cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.802+0000 7f5947400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594015c5f0 con 0x7f594013df80 2024-01-16T14:58:42.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.802+0000 7f5947400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594006c470 con 0x7f594015f9c0 2024-01-16T14:58:42.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.802+0000 7f5947400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594006c750 con 0x7f594013e360 2024-01-16T14:58:42.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.802+0000 7f5944a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f594013df80 0x7f594015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44608/0 (socket says 172.21.15.141:44608) 2024-01-16T14:58:42.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.802+0000 7f5944a00640 1 -- 172.21.15.141:0/3435077997 learned_addr learned my addr 172.21.15.141:0/3435077997 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:42.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593fe00640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 0x7f59400cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:42.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909982641 0 0) 0x7f594015c5f0 con 0x7f594013df80 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5920003520 con 0x7f594013df80 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632923342 0 0) 0x7f5920003520 con 0x7f594013df80 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 msgr2=0x7f59400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 0x7f59400cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 >> v1:172.21.15.188:6789/0 conn(0x7f594015f9c0 legacy=0x7f59400cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594006ca30 con 0x7f594013df80 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f5938001670 con 0x7f594013df80 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 >> v1:172.21.15.141:6789/0 conn(0x7f594013df80 legacy=0x7f594015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 0x7f59200039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5920003f20 0x7f59200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5920004d00 0x7f5920005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594006c750 con 0x7f5920003f20 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592000ba60 con 0x7f5920004d00 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.803+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592000bc20 con 0x7f594013e360 2024-01-16T14:58:42.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f593fe00640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5920003f20 0x7f59200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:42.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f5944a00640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 0x7f59200039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:42.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f593fe00640 1 -- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 msgr2=0x7f59200039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:42.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f593fe00640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 0x7f59200039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:42.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f593fe00640 1 -- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5920004d00 msgr2=0x7f5920005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:42.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f593fe00640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5920004d00 0x7f5920005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:42.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f593fe00640 1 -- 172.21.15.141:0/3435077997 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5920003520 con 0x7f5920003f20 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.804+0000 7f5944a00640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f594013e360 0x7f59200039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.805+0000 7f593fe00640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5920003f20 0x7f59200047c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f59200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2fa83034da3bb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.805+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592802c6b0 con 0x7f5920003f20 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.805+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592802c810 con 0x7f5920003f20 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.805+0000 7f593f400640 1 -- 172.21.15.141:0/3435077997 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592802cba0 con 0x7f5920003f20 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.806+0000 7f5947400640 1 -- 172.21.15.141:0/3435077997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5920003f20 msgr2=0x7f59200047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.806+0000 7f5947400640 1 --2- 172.21.15.141:0/3435077997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5920003f20 0x7f59200047c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f59200057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.806+0000 7f5945400640 1 -- 172.21.15.141:0/3435077997 reap_dead start 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.806+0000 7f5947400640 1 -- 172.21.15.141:0/3435077997 shutdown_connections 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.806+0000 7f5947400640 1 -- 172.21.15.141:0/3435077997 >> 172.21.15.141:0/3435077997 conn(0x7f594006a800 msgr2=0x7f594006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.806+0000 7f5947400640 1 -- 172.21.15.141:0/3435077997 shutdown_connections 2024-01-16T14:58:42.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.806+0000 7f5947400640 1 -- 172.21.15.141:0/3435077997 wait complete. 2024-01-16T14:58:42.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.808+0000 7f5947400640 1 Processor -- start 2024-01-16T14:58:42.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.808+0000 7f5947400640 1 -- start start 2024-01-16T14:58:42.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.809+0000 7f5947400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5920003f20 0x7f59400b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.809+0000 7f5947400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594013d410 0x7f594013d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.809+0000 7f5947400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f594014eec0 0x7f594014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.809+0000 7f5947400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594014db20 con 0x7f594014eec0 2024-01-16T14:58:42.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.809+0000 7f5947400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594013ddd0 con 0x7f594013d410 2024-01-16T14:58:42.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.809+0000 7f5947400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594014d180 con 0x7f5920003f20 2024-01-16T14:58:42.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f594014eec0 0x7f594014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f594014eec0 0x7f594014f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44370/0 (socket says 172.21.15.141:44370) 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 -- 172.21.15.141:0/332049192 learned_addr learned my addr 172.21.15.141:0/332049192 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f593fe00640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594013d410 0x7f594013d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5944a00640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5920003f20 0x7f59400b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 -- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5920003f20 msgr2=0x7f59400b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5920003f20 0x7f59400b31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 -- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594013d410 msgr2=0x7f594013d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594013d410 0x7f594013d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 -- 172.21.15.141:0/332049192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594014d460 con 0x7f594014eec0 2024-01-16T14:58:42.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.810+0000 7f5945400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f594014eec0 0x7f594014f350 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f594015fda0 tx=0x7f593000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:42.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.811+0000 7f593cc00640 1 -- 172.21.15.141:0/332049192 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930005d40 con 0x7f594014eec0 2024-01-16T14:58:42.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.811+0000 7f593cc00640 1 -- 172.21.15.141:0/332049192 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593000ced0 con 0x7f594014eec0 2024-01-16T14:58:42.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.811+0000 7f593cc00640 1 -- 172.21.15.141:0/332049192 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930005ea0 con 0x7f594014eec0 2024-01-16T14:58:42.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.811+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594014d740 con 0x7f594014eec0 2024-01-16T14:58:42.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.811+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594014f890 con 0x7f594014eec0 2024-01-16T14:58:42.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.812+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594014dc60 con 0x7f594014eec0 2024-01-16T14:58:42.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.813+0000 7f593cc00640 1 -- 172.21.15.141:0/332049192 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f593000a850 con 0x7f594014eec0 2024-01-16T14:58:42.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.814+0000 7f593cc00640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591806b2c0 0x7f591806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:42.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.814+0000 7f593cc00640 1 -- 172.21.15.141:0/332049192 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5930088cf0 con 0x7f594014eec0 2024-01-16T14:58:42.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.814+0000 7f5944a00640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591806b2c0 0x7f591806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:42.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.816+0000 7f5944a00640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591806b2c0 0x7f591806d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5938001200 tx=0x7f5938009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:42.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:42.819+0000 7f593cc00640 1 -- 172.21.15.141:0/332049192 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f594014dc60 con 0x7f594014eec0 2024-01-16T14:58:43.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.045+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f594014dc60 con 0x7f594014eec0 2024-01-16T14:58:43.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.388+0000 7f593cc00640 1 -- 172.21.15.141:0/332049192 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 136+0+0 (secure 0 0 0) 0x7f594014dc60 con 0x7f594014eec0 2024-01-16T14:58:43.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591806b2c0 msgr2=0x7f591806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591806b2c0 0x7f591806d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5938001200 tx=0x7f5938009040 comp rx=0 tx=0).stop 2024-01-16T14:58:43.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f594014eec0 msgr2=0x7f594014f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f594014eec0 0x7f594014f350 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f594015fda0 tx=0x7f593000cb90 comp rx=0 tx=0).stop 2024-01-16T14:58:43.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 shutdown_connections 2024-01-16T14:58:43.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591806b2c0 0x7f591806d780 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f594014eec0 0x7f594014f350 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f594013d410 0x7f594013d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 --2- 172.21.15.141:0/332049192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5920003f20 0x7f59400b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 >> 172.21.15.141:0/332049192 conn(0x7f594006a800 msgr2=0x7f594015c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:43.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 shutdown_connections 2024-01-16T14:58:43.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.392+0000 7f5947400640 1 -- 172.21.15.141:0/332049192 wait complete. 2024-01-16T14:58:43.396 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T14:58:43.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.623+0000 7f1026e00640 1 Processor -- start 2024-01-16T14:58:43.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.623+0000 7f1026e00640 1 -- start start 2024-01-16T14:58:43.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.625+0000 7f1026e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 0x7f10200d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.625+0000 7f1026e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10200d0a10 con 0x7f10200d0100 2024-01-16T14:58:43.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.625+0000 7f1026e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10200cf080 con 0x7f1020160920 2024-01-16T14:58:43.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.625+0000 7f1026e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10200c6be0 con 0x7f10200b3f30 2024-01-16T14:58:43.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.625+0000 7f1025000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f10200d0100 0x7f10200d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50534/0 (socket says 172.21.15.141:50534) 2024-01-16T14:58:43.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.625+0000 7f1025000640 1 -- 172.21.15.141:0/1036474417 learned_addr learned my addr 172.21.15.141:0/1036474417 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:43.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159862586 0 0) 0x7f10200d0a10 con 0x7f10200d0100 2024-01-16T14:58:43.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f1026400640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 0x7f10200d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1000003520 con 0x7f10200d0100 2024-01-16T14:58:43.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799580048 0 0) 0x7f1000003520 con 0x7f10200d0100 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 msgr2=0x7f10200d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 0x7f10200d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 >> v1:172.21.15.188:6789/0 conn(0x7f1020160920 legacy=0x7f10200cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10200ce7e0 con 0x7f10200d0100 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f10080019f0 con 0x7f10200d0100 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 >> v1:172.21.15.141:6789/0 conn(0x7f10200d0100 legacy=0x7f10200d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 0x7f10000039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f10000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1000004d00 0x7f1000005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10200c6be0 con 0x7f1000003fd0 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100000bb40 con 0x7f1000004d00 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100000bd00 con 0x7f10200b3f30 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.626+0000 7f1025000640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f10000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.627+0000 7f1025a00640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 0x7f10000039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.627+0000 7f1025000640 1 -- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 msgr2=0x7f10000039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:43.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.627+0000 7f1026400640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1000004d00 0x7f1000005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.627+0000 7f1025000640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f10200b3f30 0x7f10000039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.627+0000 7f1025000640 1 -- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1000004d00 msgr2=0x7f1000005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:43.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.627+0000 7f1025000640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1000004d00 0x7f1000005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.627+0000 7f1025000640 1 -- 172.21.15.141:0/1036474417 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1000003520 con 0x7f1000003fd0 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.628+0000 7f1025000640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f10000047c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f100800a830 tx=0x7f10000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba95a6db83a75a87 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.628+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080072d0 con 0x7f1000003fd0 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.628+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100802f070 con 0x7f1000003fd0 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.628+0000 7f101fe00640 1 -- 172.21.15.141:0/1036474417 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080076f0 con 0x7f1000003fd0 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.629+0000 7f1026e00640 1 -- 172.21.15.141:0/1036474417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 msgr2=0x7f10000047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.629+0000 7f1026e00640 1 --2- 172.21.15.141:0/1036474417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f10000047c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f100800a830 tx=0x7f10000057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.629+0000 7f1026400640 1 -- 172.21.15.141:0/1036474417 reap_dead start 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.629+0000 7f1026e00640 1 -- 172.21.15.141:0/1036474417 shutdown_connections 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.629+0000 7f1026e00640 1 -- 172.21.15.141:0/1036474417 >> 172.21.15.141:0/1036474417 conn(0x7f10200d3090 msgr2=0x7f10200d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.629+0000 7f1026e00640 1 -- 172.21.15.141:0/1036474417 shutdown_connections 2024-01-16T14:58:43.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.629+0000 7f1026e00640 1 -- 172.21.15.141:0/1036474417 wait complete. 2024-01-16T14:58:43.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.631+0000 7f1026e00640 1 Processor -- start 2024-01-16T14:58:43.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.631+0000 7f1026e00640 1 -- start start 2024-01-16T14:58:43.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1026e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f102015a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1026e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10200cbc40 0x7f10200cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1026e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f102015f680 0x7f102015fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1026e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10200cc600 con 0x7f1000003fd0 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1026e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020160040 con 0x7f10200cbc40 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1026e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10201f6800 con 0x7f102015f680 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1025a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f102015a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1025a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f102015a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34928/0 (socket says 172.21.15.141:34928) 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1025a00640 1 -- 172.21.15.141:0/2251827522 learned_addr learned my addr 172.21.15.141:0/2251827522 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.632+0000 7f1025000640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10200cbc40 0x7f10200cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1026400640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f102015f680 0x7f102015fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1025a00640 1 -- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f102015f680 msgr2=0x7f102015fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1025a00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f102015f680 0x7f102015fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1025a00640 1 -- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10200cbc40 msgr2=0x7f10200cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1025a00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10200cbc40 0x7f10200cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1025a00640 1 -- 172.21.15.141:0/2251827522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10201f6ae0 con 0x7f1000003fd0 2024-01-16T14:58:43.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1025000640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10200cbc40 0x7f10200cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:43.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1025a00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f102015a450 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f101000d570 tx=0x7f101000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:43.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f101d600640 1 -- 172.21.15.141:0/2251827522 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010005480 con 0x7f1000003fd0 2024-01-16T14:58:43.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f101d600640 1 -- 172.21.15.141:0/2251827522 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010003610 con 0x7f1000003fd0 2024-01-16T14:58:43.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10201f6dc0 con 0x7f1000003fd0 2024-01-16T14:58:43.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f101d600640 1 -- 172.21.15.141:0/2251827522 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010003980 con 0x7f1000003fd0 2024-01-16T14:58:43.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.633+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10201f7380 con 0x7f1000003fd0 2024-01-16T14:58:43.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.634+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10201f7540 con 0x7f1000003fd0 2024-01-16T14:58:43.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.636+0000 7f101d600640 1 -- 172.21.15.141:0/2251827522 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1010003ae0 con 0x7f1000003fd0 2024-01-16T14:58:43.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.637+0000 7f101d600640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ff806b2c0 0x7f0ff806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:43.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.637+0000 7f101d600640 1 -- 172.21.15.141:0/2251827522 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1010089bd0 con 0x7f1000003fd0 2024-01-16T14:58:43.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.637+0000 7f1025000640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ff806b2c0 0x7f0ff806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:43.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.639+0000 7f1025000640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ff806b2c0 0x7f0ff806d780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1008034000 tx=0x7f1008007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.641+0000 7f101d600640 1 -- 172.21.15.141:0/2251827522 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10100549a0 con 0x7f1000003fd0 2024-01-16T14:58:43.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.849+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1020160d00 con 0x7f0ff806b2c0 2024-01-16T14:58:43.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.853+0000 7f101d600640 1 -- 172.21.15.141:0/2251827522 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1008001980 con 0x7f0ff806b2c0 2024-01-16T14:58:43.835 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T14:58:43.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:58:43.835 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T14:58:43.835 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:58:43.835 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T14:58:43.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ff806b2c0 msgr2=0x7f0ff806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ff806b2c0 0x7f0ff806d780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1008034000 tx=0x7f1008007000 comp rx=0 tx=0).stop 2024-01-16T14:58:43.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 msgr2=0x7f102015a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:43.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f102015a450 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f101000d570 tx=0x7f101000da30 comp rx=0 tx=0).stop 2024-01-16T14:58:43.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 shutdown_connections 2024-01-16T14:58:43.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ff806b2c0 0x7f0ff806d780 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f102015f680 0x7f102015fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10200cbc40 0x7f10200cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 --2- 172.21.15.141:0/2251827522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1000003fd0 0x7f102015a450 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:43.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 >> 172.21.15.141:0/2251827522 conn(0x7f10200d3090 msgr2=0x7f10200d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:43.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 shutdown_connections 2024-01-16T14:58:43.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:43.855+0000 7f1026e00640 1 -- 172.21.15.141:0/2251827522 wait complete. 2024-01-16T14:58:43.856 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T14:58:44.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.074+0000 7f7dab400640 1 Processor -- start 2024-01-16T14:58:44.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.074+0000 7f7dab400640 1 -- start start 2024-01-16T14:58:44.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.076+0000 7f7dab400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7da4143ae0 0x7f7da40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.076+0000 7f7dab400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.076+0000 7f7dab400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da40cfc30 con 0x7f7da40cf7d0 2024-01-16T14:58:44.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.076+0000 7f7dab400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da40b1ee0 con 0x7f7da4143ae0 2024-01-16T14:58:44.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.076+0000 7f7dab400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40b2780 con 0x7f7da415e3b0 2024-01-16T14:58:44.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.076+0000 7f7da8a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7da4143ae0 0x7f7da40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.076+0000 7f7da9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.077+0000 7f7da9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:59444/0 (socket says 172.21.15.141:59444) 2024-01-16T14:58:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.077+0000 7f7da9400640 1 -- 172.21.15.141:0/4151592563 learned_addr learned my addr 172.21.15.141:0/4151592563 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.077+0000 7f7da9400640 1 -- 172.21.15.141:0/4151592563 >> v2:172.21.15.188:3300/0 conn(0x7f7da4143ae0 msgr2=0x7f7da40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.077+0000 7f7da9400640 1 --2- 172.21.15.141:0/4151592563 >> v2:172.21.15.188:3300/0 conn(0x7f7da4143ae0 0x7f7da40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.077+0000 7f7da9400640 1 -- 172.21.15.141:0/4151592563 >> v1:172.21.15.188:6789/0 conn(0x7f7da40cf7d0 legacy=0x7f7da415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.077+0000 7f7da9400640 1 -- 172.21.15.141:0/4151592563 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da415e990 con 0x7f7da415e3b0 2024-01-16T14:58:44.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.077+0000 7f7da8a00640 1 --2- 172.21.15.141:0/4151592563 >> v2:172.21.15.188:3300/0 conn(0x7f7da4143ae0 0x7f7da40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:58:44.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.079+0000 7f7da9400640 1 --2- 172.21.15.141:0/4151592563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da40702c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009e50 tx=0x7f7d9c02e170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b47590b25315fee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:44.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.079+0000 7f7da3400640 1 -- 172.21.15.141:0/4151592563 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c02ed10 con 0x7f7da415e3b0 2024-01-16T14:58:44.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.079+0000 7f7da3400640 1 -- 172.21.15.141:0/4151592563 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c02ee70 con 0x7f7da415e3b0 2024-01-16T14:58:44.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.079+0000 7f7da3400640 1 -- 172.21.15.141:0/4151592563 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c0365f0 con 0x7f7da415e3b0 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 -- 172.21.15.141:0/4151592563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 msgr2=0x7f7da40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 --2- 172.21.15.141:0/4151592563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da40702c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009e50 tx=0x7f7d9c02e170 comp rx=0 tx=0).stop 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 -- 172.21.15.141:0/4151592563 shutdown_connections 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 --2- 172.21.15.141:0/4151592563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da40702c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 --2- 172.21.15.141:0/4151592563 >> v2:172.21.15.188:3300/0 conn(0x7f7da4143ae0 0x7f7da40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 -- 172.21.15.141:0/4151592563 >> 172.21.15.141:0/4151592563 conn(0x7f7da406a800 msgr2=0x7f7da406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 -- 172.21.15.141:0/4151592563 shutdown_connections 2024-01-16T14:58:44.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.080+0000 7f7dab400640 1 -- 172.21.15.141:0/4151592563 wait complete. 2024-01-16T14:58:44.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.082+0000 7f7dab400640 1 Processor -- start 2024-01-16T14:58:44.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.082+0000 7f7dab400640 1 -- start start 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.083+0000 7f7dab400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7da40cf7d0 0x7f7da413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.083+0000 7f7dab400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7da4143ae0 0x7f7da415c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.083+0000 7f7dab400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7dab400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40cdc90 con 0x7f7da40cf7d0 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7dab400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40cd010 con 0x7f7da4143ae0 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7dab400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40cd2f0 con 0x7f7da415e3b0 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7da40cf7d0 0x7f7da413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7da40cf7d0 0x7f7da413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34942/0 (socket says 172.21.15.141:34942) 2024-01-16T14:58:44.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 -- 172.21.15.141:0/2358868252 learned_addr learned my addr 172.21.15.141:0/2358868252 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:44.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da3e00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7da4143ae0 0x7f7da415c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da9400640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 -- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 msgr2=0x7f7da415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da415cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 -- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7da4143ae0 msgr2=0x7f7da415c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7da4143ae0 0x7f7da415c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da8a00640 1 -- 172.21.15.141:0/2358868252 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94009400 con 0x7f7da40cf7d0 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.084+0000 7f7da3e00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7da4143ae0 0x7f7da415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.085+0000 7f7da8a00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7da40cf7d0 0x7f7da413d7f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400ebe0 tx=0x7f7d9400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.085+0000 7f7da0c00640 1 -- 172.21.15.141:0/2358868252 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94009760 con 0x7f7da40cf7d0 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.085+0000 7f7da0c00640 1 -- 172.21.15.141:0/2358868252 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9400cc30 con 0x7f7da40cf7d0 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.085+0000 7f7da0c00640 1 -- 172.21.15.141:0/2358868252 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9400eee0 con 0x7f7da40cf7d0 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.085+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c009b30 con 0x7f7da40cf7d0 2024-01-16T14:58:44.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.085+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da40cd8b0 con 0x7f7da40cf7d0 2024-01-16T14:58:44.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.086+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da40675d0 con 0x7f7da40cf7d0 2024-01-16T14:58:44.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.087+0000 7f7da0c00640 1 -- 172.21.15.141:0/2358868252 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d94013470 con 0x7f7da40cf7d0 2024-01-16T14:58:44.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.089+0000 7f7da0c00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d7c06b2c0 0x7f7d7c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.089+0000 7f7da0c00640 1 -- 172.21.15.141:0/2358868252 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d94089ec0 con 0x7f7da40cf7d0 2024-01-16T14:58:44.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.089+0000 7f7da3e00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d7c06b2c0 0x7f7d7c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.090+0000 7f7da3e00640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d7c06b2c0 0x7f7d7c06d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7da41f2140 tx=0x7f7d8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:44.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.093+0000 7f7da0c00640 1 -- 172.21.15.141:0/2358868252 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d94054d20 con 0x7f7da40cf7d0 2024-01-16T14:58:44.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.268+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7da40b13d0 con 0x7f7da40cf7d0 2024-01-16T14:58:44.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.269+0000 7f7da0c00640 1 -- 172.21.15.141:0/2358868252 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7d94058340 con 0x7f7da40cf7d0 2024-01-16T14:58:44.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d7c06b2c0 msgr2=0x7f7d7c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d7c06b2c0 0x7f7d7c06d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7da41f2140 tx=0x7f7d8c009040 comp rx=0 tx=0).stop 2024-01-16T14:58:44.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7da40cf7d0 msgr2=0x7f7da413d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7da40cf7d0 0x7f7da413d7f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400ebe0 tx=0x7f7d9400c520 comp rx=0 tx=0).stop 2024-01-16T14:58:44.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 shutdown_connections 2024-01-16T14:58:44.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7d7c06b2c0 0x7f7d7c06d780 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7da415e3b0 0x7f7da415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7da4143ae0 0x7f7da415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 --2- 172.21.15.141:0/2358868252 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7da40cf7d0 0x7f7da413d7f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 >> 172.21.15.141:0/2358868252 conn(0x7f7da406a800 msgr2=0x7f7da4142af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:44.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 shutdown_connections 2024-01-16T14:58:44.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.271+0000 7f7dab400640 1 -- 172.21.15.141:0/2358868252 wait complete. 2024-01-16T14:58:44.270 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T14:58:44.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.500+0000 7fd4b5800640 1 Processor -- start 2024-01-16T14:58:44.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.500+0000 7fd4b5800640 1 -- start start 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.501+0000 7fd4b5800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd4b013df80 0x7fd4b015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4b5800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013e360 0x7fd4b00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4b5800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b006c470 con 0x7fd4b015f9c0 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4b5800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b006c750 con 0x7fd4b013df80 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4b5800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b006ca30 con 0x7fd4b013e360 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4af400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd4b015f9c0 0x7fd4b01408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50546/0 (socket says 172.21.15.141:50546) 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4aea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd4b013df80 0x7fd4b015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4af400640 1 -- 172.21.15.141:0/1981680137 learned_addr learned my addr 172.21.15.141:0/1981680137 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:44.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4aea00640 1 -- 172.21.15.141:0/1981680137 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013e360 msgr2=0x7fd4b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:44.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4ae000640 1 -- 172.21.15.141:0/1981680137 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865429161 0 0) 0x7fd4b006c470 con 0x7fd4b015f9c0 2024-01-16T14:58:44.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4aea00640 1 --2- 172.21.15.141:0/1981680137 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013e360 0x7fd4b00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4aea00640 1 -- 172.21.15.141:0/1981680137 >> v1:172.21.15.141:6789/0 conn(0x7fd4b015f9c0 legacy=0x7fd4b01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.502+0000 7fd4aea00640 1 -- 172.21.15.141:0/1981680137 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b006cd10 con 0x7fd4b013df80 2024-01-16T14:58:44.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.504+0000 7fd4aea00640 1 --2- 172.21.15.141:0/1981680137 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013df80 0x7fd4b015bee0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00097a0 tx=0x7fd4a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7504deaf52281 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:44.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.504+0000 7fd4ae000640 1 -- 172.21.15.141:0/1981680137 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0031830 con 0x7fd4b013df80 2024-01-16T14:58:44.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.504+0000 7fd4ae000640 1 -- 172.21.15.141:0/1981680137 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a0031990 con 0x7fd4b013df80 2024-01-16T14:58:44.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.504+0000 7fd4ae000640 1 -- 172.21.15.141:0/1981680137 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0031d20 con 0x7fd4b013df80 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 -- 172.21.15.141:0/1981680137 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013df80 msgr2=0x7fd4b015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1981680137 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013df80 0x7fd4b015bee0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00097a0 tx=0x7fd4a002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 -- 172.21.15.141:0/1981680137 shutdown_connections 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1981680137 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013e360 0x7fd4b00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1981680137 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013df80 0x7fd4b015bee0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 -- 172.21.15.141:0/1981680137 >> 172.21.15.141:0/1981680137 conn(0x7fd4b006a800 msgr2=0x7fd4b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 -- 172.21.15.141:0/1981680137 shutdown_connections 2024-01-16T14:58:44.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.505+0000 7fd4b5800640 1 -- 172.21.15.141:0/1981680137 wait complete. 2024-01-16T14:58:44.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.507+0000 7fd4b5800640 1 Processor -- start 2024-01-16T14:58:44.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.507+0000 7fd4b5800640 1 -- start start 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4b5800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013df80 0x7fd4b01502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4b5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013e360 0x7fd4b0150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4b5800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4b015f9c0 0x7fd4b00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4b5800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0151620 con 0x7fd4b013e360 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4b5800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0151900 con 0x7fd4b015f9c0 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4b5800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0151be0 con 0x7fd4b013df80 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013e360 0x7fd4b0150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013e360 0x7fd4b0150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34946/0 (socket says 172.21.15.141:34946) 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 -- 172.21.15.141:0/1214010531 learned_addr learned my addr 172.21.15.141:0/1214010531 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 -- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013df80 msgr2=0x7fd4b01502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013df80 0x7fd4b01502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 -- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4b015f9c0 msgr2=0x7fd4b00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4b015f9c0 0x7fd4b00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.508+0000 7fd4a7e00640 1 -- 172.21.15.141:0/1214010531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a0009400 con 0x7fd4b013e360 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.509+0000 7fd4a7e00640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013e360 0x7fd4b0150c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd49800b4a0 tx=0x7fd49800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.509+0000 7fd4a6a00640 1 -- 172.21.15.141:0/1214010531 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498010610 con 0x7fd4b013e360 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.509+0000 7fd4a6a00640 1 -- 172.21.15.141:0/1214010531 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd498010770 con 0x7fd4b013e360 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.509+0000 7fd4a6a00640 1 -- 172.21.15.141:0/1214010531 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498010b20 con 0x7fd4b013e360 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.509+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b0151ec0 con 0x7fd4b013e360 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.509+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b01507f0 con 0x7fd4b013e360 2024-01-16T14:58:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.510+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b01523e0 con 0x7fd4b013e360 2024-01-16T14:58:44.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.511+0000 7fd4a6a00640 1 -- 172.21.15.141:0/1214010531 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd498010c80 con 0x7fd4b013e360 2024-01-16T14:58:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.512+0000 7fd4a6a00640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd48806b2c0 0x7fd48806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.512+0000 7fd4a6a00640 1 -- 172.21.15.141:0/1214010531 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd498089840 con 0x7fd4b013e360 2024-01-16T14:58:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.512+0000 7fd4aea00640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd48806b2c0 0x7fd48806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:44.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.513+0000 7fd4aea00640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd48806b2c0 0x7fd48806d780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0036000 tx=0x7fd4a0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.516+0000 7fd4a6a00640 1 -- 172.21.15.141:0/1214010531 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4b01523e0 con 0x7fd4b013e360 2024-01-16T14:58:44.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.735+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4b00ce850 con 0x7fd48806b2c0 2024-01-16T14:58:44.730 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:58:44.749+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:44.730 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:58:44.749+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:44.730 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:58:44.749+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:44.730 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:58:44.749+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:44.730 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:58:44.749+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:58:44.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.830+0000 7fd4a6a00640 1 -- 172.21.15.141:0/1214010531 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd4a0031b80 con 0x7fd48806b2c0 2024-01-16T14:58:44.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd48806b2c0 msgr2=0x7fd48806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd48806b2c0 0x7fd48806d780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0036000 tx=0x7fd4a0030000 comp rx=0 tx=0).stop 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013e360 msgr2=0x7fd4b0150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013e360 0x7fd4b0150c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd49800b4a0 tx=0x7fd49800b960 comp rx=0 tx=0).stop 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 shutdown_connections 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd48806b2c0 0x7fd48806d780 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4b015f9c0 0x7fd4b00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4b013e360 0x7fd4b0150c40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 --2- 172.21.15.141:0/1214010531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4b013df80 0x7fd4b01502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 >> 172.21.15.141:0/1214010531 conn(0x7fd4b006a800 msgr2=0x7fd4b00706a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.833+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 shutdown_connections 2024-01-16T14:58:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:44.834+0000 7fd4b5800640 1 -- 172.21.15.141:0/1214010531 wait complete. 2024-01-16T14:58:44.837 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T14:58:45.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.063+0000 7ff317e00640 1 Processor -- start 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.063+0000 7ff317e00640 1 -- start start 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.064+0000 7ff317e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff31813cf10 0x7ff31813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.064+0000 7ff317e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3180ce440 con 0x7ff31813fae0 2024-01-16T14:58:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.064+0000 7ff317e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3180ce720 con 0x7ff3180cddf0 2024-01-16T14:58:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.064+0000 7ff317e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31813ff90 con 0x7ff31813cf10 2024-01-16T14:58:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.065+0000 7ff30de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff31813fae0 0x7ff3180cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50558/0 (socket says 172.21.15.141:50558) 2024-01-16T14:58:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.065+0000 7ff30de00640 1 -- 172.21.15.141:0/2356576697 learned_addr learned my addr 172.21.15.141:0/2356576697 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.065+0000 7ff316a00640 1 --2- 172.21.15.141:0/2356576697 >> v2:172.21.15.188:3300/0 conn(0x7ff31813cf10 0x7ff31813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.065+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177900988 0 0) 0x7ff3180ce440 con 0x7ff31813fae0 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.065+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8003520 con 0x7ff31813fae0 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4006333812 0 0) 0x7ff2f8003520 con 0x7ff31813fae0 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 >> v2:172.21.15.188:3300/0 conn(0x7ff31813cf10 msgr2=0x7ff31813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 --2- 172.21.15.141:0/2356576697 >> v2:172.21.15.188:3300/0 conn(0x7ff31813cf10 0x7ff31813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 >> v1:172.21.15.188:6789/0 conn(0x7ff3180cddf0 legacy=0x7ff3180b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3180cea00 con 0x7ff31813fae0 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ff300001670 con 0x7ff31813fae0 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 >> v1:172.21.15.141:6789/0 conn(0x7ff31813fae0 legacy=0x7ff3180cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:45.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80039c0 0x7ff2f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2f80047c0 0x7ff2f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2f8005180 0x7ff2f800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31813ff90 con 0x7ff2f80047c0 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f800e290 con 0x7ff2f80039c0 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f800e450 con 0x7ff2f8005180 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff30de00640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2f80047c0 0x7ff2f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.066+0000 7ff316a00640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80039c0 0x7ff2f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.067+0000 7ff30de00640 1 -- 172.21.15.141:0/2356576697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2f8005180 msgr2=0x7ff2f800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.067+0000 7ff30de00640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2f8005180 0x7ff2f800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.067+0000 7ff30de00640 1 -- 172.21.15.141:0/2356576697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80039c0 msgr2=0x7ff2f8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.067+0000 7ff30de00640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80039c0 0x7ff2f8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.067+0000 7ff30de00640 1 -- 172.21.15.141:0/2356576697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f8003520 con 0x7ff2f80047c0 2024-01-16T14:58:45.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.067+0000 7ff317400640 1 -- 172.21.15.141:0/2356576697 reap_dead start 2024-01-16T14:58:45.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.068+0000 7ff30de00640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2f80047c0 0x7ff2f8004c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c0e0 tx=0x7ff2f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660344b5a91c6dd9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:45.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.068+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300005ad0 con 0x7ff2f80047c0 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.068+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300005c50 con 0x7ff2f80047c0 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.068+0000 7ff316000640 1 -- 172.21.15.141:0/2356576697 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30002c810 con 0x7ff2f80047c0 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.068+0000 7ff317e00640 1 -- 172.21.15.141:0/2356576697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2f80047c0 msgr2=0x7ff2f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.068+0000 7ff317e00640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2f80047c0 0x7ff2f8004c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c0e0 tx=0x7ff2f80057f0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.069+0000 7ff317e00640 1 -- 172.21.15.141:0/2356576697 shutdown_connections 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.069+0000 7ff317e00640 1 --2- 172.21.15.141:0/2356576697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2f80047c0 0x7ff2f8004c40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.069+0000 7ff317e00640 1 -- 172.21.15.141:0/2356576697 >> 172.21.15.141:0/2356576697 conn(0x7ff31806a7d0 msgr2=0x7ff31806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.069+0000 7ff317e00640 1 -- 172.21.15.141:0/2356576697 shutdown_connections 2024-01-16T14:58:45.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.069+0000 7ff317e00640 1 -- 172.21.15.141:0/2356576697 wait complete. 2024-01-16T14:58:45.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.070+0000 7ff317e00640 1 Processor -- start 2024-01-16T14:58:45.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.071+0000 7ff317e00640 1 -- start start 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.071+0000 7ff317e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80047c0 0x7ff31814f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.071+0000 7ff317e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff318158ec0 0x7ff318159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.071+0000 7ff317e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff31813e8f0 0x7ff31813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.071+0000 7ff317e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff318159880 con 0x7ff318158ec0 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.071+0000 7ff317e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31814f630 con 0x7ff2f80047c0 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.071+0000 7ff317e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff318070160 con 0x7ff31813e8f0 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff318158ec0 0x7ff318159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff318158ec0 0x7ff318159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34964/0 (socket says 172.21.15.141:34964) 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 -- 172.21.15.141:0/4277405670 learned_addr learned my addr 172.21.15.141:0/4277405670 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff317400640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff31813e8f0 0x7ff31813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff316a00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80047c0 0x7ff31814f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 -- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff31813e8f0 msgr2=0x7ff31813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff31813e8f0 0x7ff31813ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 -- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80047c0 msgr2=0x7ff31814f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80047c0 0x7ff31814f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.072+0000 7ff30de00640 1 -- 172.21.15.141:0/4277405670 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31815c160 con 0x7ff318158ec0 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.073+0000 7ff30de00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff318158ec0 0x7ff318159340 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff300000c00 tx=0x7ff30002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.073+0000 7ff30fe00640 1 -- 172.21.15.141:0/4277405670 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30003c040 con 0x7ff318158ec0 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.073+0000 7ff30fe00640 1 -- 172.21.15.141:0/4277405670 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30002cce0 con 0x7ff318158ec0 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.073+0000 7ff30fe00640 1 -- 172.21.15.141:0/4277405670 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3000072b0 con 0x7ff318158ec0 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.073+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31815c440 con 0x7ff318158ec0 2024-01-16T14:58:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.073+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31815ca00 con 0x7ff318158ec0 2024-01-16T14:58:45.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.074+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3180b19d0 con 0x7ff318158ec0 2024-01-16T14:58:45.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.075+0000 7ff30fe00640 1 -- 172.21.15.141:0/4277405670 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff300032070 con 0x7ff318158ec0 2024-01-16T14:58:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.076+0000 7ff30fe00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff2f006b2c0 0x7ff2f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:58:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.076+0000 7ff30fe00640 1 -- 172.21.15.141:0/4277405670 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3000ab0d0 con 0x7ff318158ec0 2024-01-16T14:58:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.077+0000 7ff316a00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff2f006b2c0 0x7ff2f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:58:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.078+0000 7ff316a00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff2f006b2c0 0x7ff2f006d780 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff308007380 tx=0x7ff30800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:58:45.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.080+0000 7ff30fe00640 1 -- 172.21.15.141:0/4277405670 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff300075fd0 con 0x7ff318158ec0 2024-01-16T14:58:45.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.298+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3180b2070 con 0x7ff2f006b2c0 2024-01-16T14:58:45.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.311+0000 7ff30fe00640 1 -- 172.21.15.141:0/4277405670 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7ff308009c90 con 0x7ff2f006b2c0 2024-01-16T14:58:45.293 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6 2024-01-16T14:58:45.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff2f006b2c0 msgr2=0x7ff2f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff2f006b2c0 0x7ff2f006d780 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff308007380 tx=0x7ff30800b040 comp rx=0 tx=0).stop 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff318158ec0 msgr2=0x7ff318159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff318158ec0 0x7ff318159340 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff300000c00 tx=0x7ff30002e070 comp rx=0 tx=0).stop 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 shutdown_connections 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff2f006b2c0 0x7ff2f006d780 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff308007380 tx=0x7ff30800b040 comp rx=0 tx=0).stop 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff31813e8f0 0x7ff31813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff318158ec0 0x7ff318159340 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 --2- 172.21.15.141:0/4277405670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2f80047c0 0x7ff31814f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 >> 172.21.15.141:0/4277405670 conn(0x7ff31806a7d0 msgr2=0x7ff318159ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.313+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 shutdown_connections 2024-01-16T14:58:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:58:45.314+0000 7ff317e00640 1 -- 172.21.15.141:0/4277405670 wait complete. 2024-01-16T14:58:45.313 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6 2024-01-16T14:58:45.313 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:58:45.313 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:45.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:45.361 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T14:58:59.924 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T14:58:59.924 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T14:58:59.924 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.4423 s, 74.3 MB/s 2024-01-16T14:58:59.926 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:58:59.926 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:58:59.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:58:59.979 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T14:59:02.500 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:02.508 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:11.214 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T14:59:11.214 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T14:59:11.214 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.1813 s, 96.0 MB/s 2024-01-16T14:59:11.216 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:59:11.216 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:11.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:11.270 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T14:59:22.121 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T14:59:22.121 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T14:59:22.121 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.7993 s, 99.4 MB/s 2024-01-16T14:59:22.123 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T14:59:22.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.355+0000 7fbb95600640 1 Processor -- start 2024-01-16T14:59:22.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.355+0000 7fbb95600640 1 -- start start 2024-01-16T14:59:22.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb95600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb900adf20 0x7fbb90070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb95600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb900ccb50 0x7fbb90143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb95600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9015f4e0 con 0x7fbb9015bf10 2024-01-16T14:59:22.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb95600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9015f970 con 0x7fbb900ccb50 2024-01-16T14:59:22.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb95600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90071130 con 0x7fbb900adf20 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb94c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb900ccb50 0x7fbb90143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb94c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb900ccb50 0x7fbb90143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60502/0 (socket says 172.21.15.141:60502) 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb94c00640 1 -- 172.21.15.141:0/4100951483 learned_addr learned my addr 172.21.15.141:0/4100951483 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.357+0000 7fbb8fe00640 1 --2- 172.21.15.141:0/4100951483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb900adf20 0x7fbb90070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.358+0000 7fbb94c00640 1 -- 172.21.15.141:0/4100951483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb900adf20 msgr2=0x7fbb90070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.358+0000 7fbb94c00640 1 --2- 172.21.15.141:0/4100951483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb900adf20 0x7fbb90070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.358+0000 7fbb94c00640 1 -- 172.21.15.141:0/4100951483 >> v1:172.21.15.141:6789/0 conn(0x7fbb9015bf10 legacy=0x7fbb900cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.358+0000 7fbb94c00640 1 -- 172.21.15.141:0/4100951483 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9015c390 con 0x7fbb900ccb50 2024-01-16T14:59:22.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.359+0000 7fbb94c00640 1 --2- 172.21.15.141:0/4100951483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb90143d30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009e50 tx=0x7fbb8802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82300259b316cb74 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.359+0000 7fbb8ea00640 1 -- 172.21.15.141:0/4100951483 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8802ca80 con 0x7fbb900ccb50 2024-01-16T14:59:22.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.359+0000 7fbb8ea00640 1 -- 172.21.15.141:0/4100951483 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8802cbe0 con 0x7fbb900ccb50 2024-01-16T14:59:22.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.359+0000 7fbb8ea00640 1 -- 172.21.15.141:0/4100951483 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb880343f0 con 0x7fbb900ccb50 2024-01-16T14:59:22.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 -- 172.21.15.141:0/4100951483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 msgr2=0x7fbb90143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 --2- 172.21.15.141:0/4100951483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb90143d30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009e50 tx=0x7fbb8802bee0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 -- 172.21.15.141:0/4100951483 shutdown_connections 2024-01-16T14:59:22.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 --2- 172.21.15.141:0/4100951483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb90143d30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 --2- 172.21.15.141:0/4100951483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb900adf20 0x7fbb90070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 -- 172.21.15.141:0/4100951483 >> 172.21.15.141:0/4100951483 conn(0x7fbb9006a7d0 msgr2=0x7fbb9006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:22.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 -- 172.21.15.141:0/4100951483 shutdown_connections 2024-01-16T14:59:22.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.360+0000 7fbb95600640 1 -- 172.21.15.141:0/4100951483 wait complete. 2024-01-16T14:59:22.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.362+0000 7fbb95600640 1 Processor -- start 2024-01-16T14:59:22.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.362+0000 7fbb95600640 1 -- start start 2024-01-16T14:59:22.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb95600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb900adf20 0x7fbb900b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb95600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb9015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb95600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb9015bf10 0x7fbb901f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb95600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9015ae50 con 0x7fbb900ccb50 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb95600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9013f480 con 0x7fbb900adf20 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb95600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9013f760 con 0x7fbb9015bf10 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb8f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb9015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb8f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb9015a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60518/0 (socket says 172.21.15.141:60518) 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.363+0000 7fbb8f400640 1 -- 172.21.15.141:0/3427368968 learned_addr learned my addr 172.21.15.141:0/3427368968 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb8fe00640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb900adf20 0x7fbb900b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb8f400640 1 -- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb9015bf10 msgr2=0x7fbb901f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb8f400640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb9015bf10 0x7fbb901f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb8f400640 1 -- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb900adf20 msgr2=0x7fbb900b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb8f400640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb900adf20 0x7fbb900b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb8f400640 1 -- 172.21.15.141:0/3427368968 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb88009b30 con 0x7fbb900ccb50 2024-01-16T14:59:22.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb8f400640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb9015a910 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800c400 tx=0x7fbb7800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb87e00640 1 -- 172.21.15.141:0/3427368968 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb780052d0 con 0x7fbb900ccb50 2024-01-16T14:59:22.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb87e00640 1 -- 172.21.15.141:0/3427368968 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb78005430 con 0x7fbb900ccb50 2024-01-16T14:59:22.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9013fa40 con 0x7fbb900ccb50 2024-01-16T14:59:22.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.364+0000 7fbb87e00640 1 -- 172.21.15.141:0/3427368968 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78003830 con 0x7fbb900ccb50 2024-01-16T14:59:22.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.365+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb90140000 con 0x7fbb900ccb50 2024-01-16T14:59:22.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.366+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb900ce1a0 con 0x7fbb900ccb50 2024-01-16T14:59:22.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.367+0000 7fbb87e00640 1 -- 172.21.15.141:0/3427368968 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fbb78003990 con 0x7fbb900ccb50 2024-01-16T14:59:22.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.367+0000 7fbb87e00640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb6806b350 0x7fbb6806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.368+0000 7fbb87e00640 1 -- 172.21.15.141:0/3427368968 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb78089170 con 0x7fbb900ccb50 2024-01-16T14:59:22.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.368+0000 7fbb8fe00640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb6806b350 0x7fbb6806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.369+0000 7fbb8fe00640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb6806b350 0x7fbb6806d810 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbb80006fd0 tx=0x7fbb80008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.372+0000 7fbb87e00640 1 -- 172.21.15.141:0/3427368968 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb78053ff0 con 0x7fbb900ccb50 2024-01-16T14:59:22.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.565+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fbb90067450 con 0x7fbb900ccb50 2024-01-16T14:59:22.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.570+0000 7fbb87e00640 1 -- 172.21.15.141:0/3427368968 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7fbb78057610 con 0x7fbb900ccb50 2024-01-16T14:59:22.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb6806b350 msgr2=0x7fbb6806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb6806b350 0x7fbb6806d810 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbb80006fd0 tx=0x7fbb80008040 comp rx=0 tx=0).stop 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 msgr2=0x7fbb9015a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb9015a910 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800c400 tx=0x7fbb7800c8c0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 shutdown_connections 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb6806b350 0x7fbb6806d810 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb9015bf10 0x7fbb901f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb900ccb50 0x7fbb9015a910 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 --2- 172.21.15.141:0/3427368968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb900adf20 0x7fbb900b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 >> 172.21.15.141:0/3427368968 conn(0x7fbb9006a7d0 msgr2=0x7fbb9015b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 shutdown_connections 2024-01-16T14:59:22.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.573+0000 7fbb95600640 1 -- 172.21.15.141:0/3427368968 wait complete. 2024-01-16T14:59:22.572 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T14:59:22.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.804+0000 7fce4cc00640 1 Processor -- start 2024-01-16T14:59:22.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.804+0000 7fce4cc00640 1 -- start start 2024-01-16T14:59:22.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.805+0000 7fce4cc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fce48143c40 0x7fce48071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.805+0000 7fce4cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480717d0 0x7fce480cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.805+0000 7fce4cc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce48070e50 con 0x7fce480cca40 2024-01-16T14:59:22.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.806+0000 7fce4cc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce480723c0 con 0x7fce48143c40 2024-01-16T14:59:22.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.806+0000 7fce4cc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce480cce70 con 0x7fce480717d0 2024-01-16T14:59:22.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.806+0000 7fce47400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fce48143c40 0x7fce48071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.806+0000 7fce46a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480717d0 0x7fce480cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.806+0000 7fce46a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480717d0 0x7fce480cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:54394/0 (socket says 172.21.15.141:54394) 2024-01-16T14:59:22.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.806+0000 7fce46a00640 1 -- 172.21.15.141:0/4003655878 learned_addr learned my addr 172.21.15.141:0/4003655878 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:22.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.806+0000 7fce47400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fce48143c40 0x7fce48071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:54536/0 (socket says 172.21.15.141:54536) 2024-01-16T14:59:22.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.807+0000 7fce46a00640 1 -- 172.21.15.141:0/4003655878 >> v2:172.21.15.188:3300/0 conn(0x7fce48143c40 msgr2=0x7fce48071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:22.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.807+0000 7fce46a00640 1 --2- 172.21.15.141:0/4003655878 >> v2:172.21.15.188:3300/0 conn(0x7fce48143c40 0x7fce48071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.807+0000 7fce46a00640 1 -- 172.21.15.141:0/4003655878 >> v1:172.21.15.188:6789/0 conn(0x7fce480cca40 legacy=0x7fce4815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.807+0000 7fce46a00640 1 -- 172.21.15.141:0/4003655878 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4806e170 con 0x7fce480717d0 2024-01-16T14:59:22.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.807+0000 7fce47400640 1 --2- 172.21.15.141:0/4003655878 >> v2:172.21.15.188:3300/0 conn(0x7fce48143c40 0x7fce48071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:22.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.809+0000 7fce46a00640 1 --2- 172.21.15.141:0/4003655878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480717d0 0x7fce480cc500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce30009870 tx=0x7fce3002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d42aaaf1a9d2686d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.809+0000 7fce46000640 1 -- 172.21.15.141:0/4003655878 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30031640 con 0x7fce480717d0 2024-01-16T14:59:22.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.809+0000 7fce46000640 1 -- 172.21.15.141:0/4003655878 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce300317a0 con 0x7fce480717d0 2024-01-16T14:59:22.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.809+0000 7fce46000640 1 -- 172.21.15.141:0/4003655878 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30031b30 con 0x7fce480717d0 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 -- 172.21.15.141:0/4003655878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480717d0 msgr2=0x7fce480cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 --2- 172.21.15.141:0/4003655878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480717d0 0x7fce480cc500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce30009870 tx=0x7fce3002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 -- 172.21.15.141:0/4003655878 shutdown_connections 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 --2- 172.21.15.141:0/4003655878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480717d0 0x7fce480cc500 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 --2- 172.21.15.141:0/4003655878 >> v2:172.21.15.188:3300/0 conn(0x7fce48143c40 0x7fce48071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 -- 172.21.15.141:0/4003655878 >> 172.21.15.141:0/4003655878 conn(0x7fce4806a800 msgr2=0x7fce4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 -- 172.21.15.141:0/4003655878 shutdown_connections 2024-01-16T14:59:22.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.810+0000 7fce4cc00640 1 -- 172.21.15.141:0/4003655878 wait complete. 2024-01-16T14:59:22.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.812+0000 7fce4cc00640 1 Processor -- start 2024-01-16T14:59:22.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.813+0000 7fce4cc00640 1 -- start start 2024-01-16T14:59:22.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce4cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce480717d0 0x7fce4813d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce4cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480cca40 0x7fce480b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce4cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48143c40 0x7fce481e54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce4cc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480b1eb0 con 0x7fce480717d0 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce4cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480b2d10 con 0x7fce48143c40 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce4cc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce480b2100 con 0x7fce480cca40 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce47400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce480717d0 0x7fce4813d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce47400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce480717d0 0x7fce4813d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60522/0 (socket says 172.21.15.141:60522) 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.814+0000 7fce47400640 1 -- 172.21.15.141:0/3456130268 learned_addr learned my addr 172.21.15.141:0/3456130268 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce46a00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480cca40 0x7fce480b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce47e00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48143c40 0x7fce481e54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce47400640 1 -- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480cca40 msgr2=0x7fce480b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce47400640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480cca40 0x7fce480b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce47400640 1 -- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48143c40 msgr2=0x7fce481e54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce47400640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48143c40 0x7fce481e54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:22.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce47400640 1 -- 172.21.15.141:0/3456130268 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38009400 con 0x7fce480717d0 2024-01-16T14:59:22.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce47400640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce480717d0 0x7fce4813d340 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fce3800e990 tx=0x7fce3800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce3f400640 1 -- 172.21.15.141:0/3456130268 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3800c9e0 con 0x7fce480717d0 2024-01-16T14:59:22.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.815+0000 7fce3f400640 1 -- 172.21.15.141:0/3456130268 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3800cb40 con 0x7fce480717d0 2024-01-16T14:59:22.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.816+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce300094d0 con 0x7fce480717d0 2024-01-16T14:59:22.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.816+0000 7fce3f400640 1 -- 172.21.15.141:0/3456130268 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3800ced0 con 0x7fce480717d0 2024-01-16T14:59:22.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.816+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce480b26c0 con 0x7fce480717d0 2024-01-16T14:59:22.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.817+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce48067450 con 0x7fce480717d0 2024-01-16T14:59:22.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.818+0000 7fce3f400640 1 -- 172.21.15.141:0/3456130268 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fce38013470 con 0x7fce480717d0 2024-01-16T14:59:22.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.819+0000 7fce3f400640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce2006b350 0x7fce2006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:22.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.819+0000 7fce3f400640 1 -- 172.21.15.141:0/3456130268 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce38089ed0 con 0x7fce480717d0 2024-01-16T14:59:22.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.820+0000 7fce46a00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce2006b350 0x7fce2006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:22.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.821+0000 7fce46a00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce2006b350 0x7fce2006d810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce30036000 tx=0x7fce3003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:22.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:22.824+0000 7fce3f400640 1 -- 172.21.15.141:0/3456130268 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce38054d50 con 0x7fce480717d0 2024-01-16T14:59:23.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.046+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fce480720e0 con 0x7fce2006b350 2024-01-16T14:59:23.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.078+0000 7fce3f400640 1 -- 172.21.15.141:0/3456130268 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fce30031a80 con 0x7fce2006b350 2024-01-16T14:59:23.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce2006b350 msgr2=0x7fce2006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce2006b350 0x7fce2006d810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce30036000 tx=0x7fce3003c040 comp rx=0 tx=0).stop 2024-01-16T14:59:23.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce480717d0 msgr2=0x7fce4813d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce480717d0 0x7fce4813d340 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fce3800e990 tx=0x7fce3800ee50 comp rx=0 tx=0).stop 2024-01-16T14:59:23.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 shutdown_connections 2024-01-16T14:59:23.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce2006b350 0x7fce2006d810 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48143c40 0x7fce481e54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce480cca40 0x7fce480b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 --2- 172.21.15.141:0/3456130268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce480717d0 0x7fce4813d340 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.081+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 >> 172.21.15.141:0/3456130268 conn(0x7fce4806a800 msgr2=0x7fce48141230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.082+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 shutdown_connections 2024-01-16T14:59:23.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.082+0000 7fce4cc00640 1 -- 172.21.15.141:0/3456130268 wait complete. 2024-01-16T14:59:23.082 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T14:59:23.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.303+0000 7f58ff600640 1 Processor -- start 2024-01-16T14:59:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.303+0000 7f58ff600640 1 -- start start 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.304+0000 7f58ff600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8157c30 0x7f58f80cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.304+0000 7f58ff600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f58f81618f0 0x7f58f8070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.304+0000 7f58ff600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f58f8071030 0x7f58f806dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58ff600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f815ea30 con 0x7f58f8071030 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58ff600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f806d860 con 0x7f58f81618f0 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58ff600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f8071460 con 0x7f58f8157c30 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f58f8071030 0x7f58f806dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f58f8071030 0x7f58f806dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55858/0 (socket says 172.21.15.141:55858) 2024-01-16T14:59:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 -- 172.21.15.141:0/2124046945 learned_addr learned my addr 172.21.15.141:0/2124046945 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fcc00640 1 --2- 172.21.15.141:0/2124046945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8157c30 0x7f58f80cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58efc00640 1 --2- 172.21.15.141:0/2124046945 >> v2:172.21.15.188:3300/0 conn(0x7f58f81618f0 0x7f58f8070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 -- 172.21.15.141:0/2124046945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8157c30 msgr2=0x7f58f80cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 --2- 172.21.15.141:0/2124046945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8157c30 0x7f58f80cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 -- 172.21.15.141:0/2124046945 >> v2:172.21.15.188:3300/0 conn(0x7f58f81618f0 msgr2=0x7f58f8070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 --2- 172.21.15.141:0/2124046945 >> v2:172.21.15.188:3300/0 conn(0x7f58f81618f0 0x7f58f8070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.305+0000 7f58fd600640 1 -- 172.21.15.141:0/2124046945 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f815deb0 con 0x7f58f8071030 2024-01-16T14:59:23.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.307+0000 7f58fd600640 1 --2- 172.21.15.141:0/2124046945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f8071030 0x7f58f806dcb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f58f8072450 tx=0x7f58f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316b5e70ba5dd5a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.307+0000 7f58f7e00640 1 -- 172.21.15.141:0/2124046945 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0031870 con 0x7f58f8071030 2024-01-16T14:59:23.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.307+0000 7f58f7e00640 1 -- 172.21.15.141:0/2124046945 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f00319d0 con 0x7f58f8071030 2024-01-16T14:59:23.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.307+0000 7f58f7e00640 1 -- 172.21.15.141:0/2124046945 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0031d60 con 0x7f58f8071030 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 -- 172.21.15.141:0/2124046945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f8071030 msgr2=0x7f58f806dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 --2- 172.21.15.141:0/2124046945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f8071030 0x7f58f806dcb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f58f8072450 tx=0x7f58f002b910 comp rx=0 tx=0).stop 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 -- 172.21.15.141:0/2124046945 shutdown_connections 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 --2- 172.21.15.141:0/2124046945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f8071030 0x7f58f806dcb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 --2- 172.21.15.141:0/2124046945 >> v2:172.21.15.188:3300/0 conn(0x7f58f81618f0 0x7f58f8070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 --2- 172.21.15.141:0/2124046945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8157c30 0x7f58f80cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 -- 172.21.15.141:0/2124046945 >> 172.21.15.141:0/2124046945 conn(0x7f58f80cf700 msgr2=0x7f58f80cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 -- 172.21.15.141:0/2124046945 shutdown_connections 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.308+0000 7f58ff600640 1 -- 172.21.15.141:0/2124046945 wait complete. 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.310+0000 7f58ff600640 1 Processor -- start 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.310+0000 7f58ff600640 1 -- start start 2024-01-16T14:59:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58ff600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8071030 0x7f58f81f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58ff600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f8157c30 0x7f58f8156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58ff600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f81618f0 0x7f58f8157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58ff600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f81f7a20 con 0x7f58f81618f0 2024-01-16T14:59:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58ff600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f81f7d00 con 0x7f58f8157c30 2024-01-16T14:59:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58ff600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f81f7fe0 con 0x7f58f8071030 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58fd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f81618f0 0x7f58f8157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58fd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f81618f0 0x7f58f8157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55868/0 (socket says 172.21.15.141:55868) 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.311+0000 7f58fd600640 1 -- 172.21.15.141:0/1489785362 learned_addr learned my addr 172.21.15.141:0/1489785362 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58fcc00640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8071030 0x7f58f81f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58efc00640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f8157c30 0x7f58f8156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58fd600640 1 -- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8071030 msgr2=0x7f58f81f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58fd600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8071030 0x7f58f81f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58fd600640 1 -- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f8157c30 msgr2=0x7f58f8156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58fd600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f8157c30 0x7f58f8156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58fd600640 1 -- 172.21.15.141:0/1489785362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00094d0 con 0x7f58f81618f0 2024-01-16T14:59:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58fd600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f81618f0 0x7f58f8157180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f58f00090b0 tx=0x7f58f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58f5600640 1 -- 172.21.15.141:0/1489785362 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f003ac60 con 0x7f58f81618f0 2024-01-16T14:59:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.312+0000 7f58f5600640 1 -- 172.21.15.141:0/1489785362 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f003adc0 con 0x7f58f81618f0 2024-01-16T14:59:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.313+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f8146ae0 con 0x7f58f81618f0 2024-01-16T14:59:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.313+0000 7f58f5600640 1 -- 172.21.15.141:0/1489785362 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0044630 con 0x7f58f81618f0 2024-01-16T14:59:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.313+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f81f7660 con 0x7f58f81618f0 2024-01-16T14:59:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.314+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f8147000 con 0x7f58f81618f0 2024-01-16T14:59:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.315+0000 7f58f5600640 1 -- 172.21.15.141:0/1489785362 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f58f004d450 con 0x7f58f81618f0 2024-01-16T14:59:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.315+0000 7f58f5600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f58d006b350 0x7f58d006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.316+0000 7f58f5600640 1 -- 172.21.15.141:0/1489785362 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58f002d040 con 0x7f58f81618f0 2024-01-16T14:59:23.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.316+0000 7f58fcc00640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f58d006b350 0x7f58d006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.317+0000 7f58fcc00640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f58d006b350 0x7f58d006d810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f58f815db80 tx=0x7f58e8003a50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.321+0000 7f58f5600640 1 -- 172.21.15.141:0/1489785362 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58f8147000 con 0x7f58f81618f0 2024-01-16T14:59:23.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.543+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f58f815d3f0 con 0x7f58d006b350 2024-01-16T14:59:23.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.554+0000 7f58f5600640 1 -- 172.21.15.141:0/1489785362 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f58f815d3f0 con 0x7f58d006b350 2024-01-16T14:59:23.536 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6 2024-01-16T14:59:23.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f58d006b350 msgr2=0x7f58d006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f58d006b350 0x7f58d006d810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f58f815db80 tx=0x7f58e8003a50 comp rx=0 tx=0).stop 2024-01-16T14:59:23.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f81618f0 msgr2=0x7f58f8157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f81618f0 0x7f58f8157180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f58f00090b0 tx=0x7f58f00046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 shutdown_connections 2024-01-16T14:59:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f58d006b350 0x7f58d006d810 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f58f81618f0 0x7f58f8157180 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f8157c30 0x7f58f8156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 --2- 172.21.15.141:0/1489785362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f58f8071030 0x7f58f81f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 >> 172.21.15.141:0/1489785362 conn(0x7f58f80cf700 msgr2=0x7f58f8160470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 shutdown_connections 2024-01-16T14:59:23.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:23.557+0000 7f58ff600640 1 -- 172.21.15.141:0/1489785362 wait complete. 2024-01-16T14:59:23.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:23.557 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6) 2024-01-16T14:59:23.613 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6 2024-01-16T14:59:23.613 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T14:59:23.613 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:23.614 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T14:59:25.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:25.615 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6) 2024-01-16T14:59:25.680 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6 2024-01-16T14:59:25.680 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T14:59:25.680 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:25.681 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T14:59:27.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:27.683 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6) 2024-01-16T14:59:27.748 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/aff0cdb7-c92f-4275-8065-8fbb0430f7f6 2024-01-16T14:59:27.748 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T14:59:27.749 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:27.749 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T14:59:27.750 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.986+0000 7f05c3000640 1 Processor -- start 2024-01-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.986+0000 7f05c3000640 1 -- start start 2024-01-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c3000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f05bc0cf9a0 0x7f05bc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c3000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f05bc15e1e0 0x7f05bc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c3000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc0b1ee0 con 0x7f05bc143ae0 2024-01-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c3000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc0b2780 con 0x7f05bc15e1e0 2024-01-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c3000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc15e990 con 0x7f05bc0cf9a0 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f05bc15e1e0 0x7f05bc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f05bc15e1e0 0x7f05bc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55870/0 (socket says 172.21.15.141:55870) 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05c1000640 1 -- 172.21.15.141:0/4284963072 learned_addr learned my addr 172.21.15.141:0/4284963072 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.988+0000 7f05bb400640 1 --2- 172.21.15.141:0/4284963072 >> v2:172.21.15.188:3300/0 conn(0x7f05bc0cf9a0 0x7f05bc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.989+0000 7f05c1000640 1 -- 172.21.15.141:0/4284963072 >> v2:172.21.15.188:3300/0 conn(0x7f05bc0cf9a0 msgr2=0x7f05bc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.989+0000 7f05c1000640 1 --2- 172.21.15.141:0/4284963072 >> v2:172.21.15.188:3300/0 conn(0x7f05bc0cf9a0 0x7f05bc15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.989+0000 7f05c1000640 1 -- 172.21.15.141:0/4284963072 >> v1:172.21.15.188:6789/0 conn(0x7f05bc143ae0 legacy=0x7f05bc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.989+0000 7f05c1000640 1 -- 172.21.15.141:0/4284963072 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc0b2130 con 0x7f05bc15e1e0 2024-01-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.990+0000 7f05c1000640 1 --2- 172.21.15.141:0/4284963072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc0702c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0cb830 tx=0x7f05b402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676fd6bc2138bdb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.990+0000 7f05baa00640 1 -- 172.21.15.141:0/4284963072 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b402ca80 con 0x7f05bc15e1e0 2024-01-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.990+0000 7f05baa00640 1 -- 172.21.15.141:0/4284963072 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b402cbe0 con 0x7f05bc15e1e0 2024-01-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.990+0000 7f05baa00640 1 -- 172.21.15.141:0/4284963072 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b40343f0 con 0x7f05bc15e1e0 2024-01-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 -- 172.21.15.141:0/4284963072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 msgr2=0x7f05bc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 --2- 172.21.15.141:0/4284963072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc0702c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0cb830 tx=0x7f05b402bee0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 -- 172.21.15.141:0/4284963072 shutdown_connections 2024-01-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 --2- 172.21.15.141:0/4284963072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc0702c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 --2- 172.21.15.141:0/4284963072 >> v2:172.21.15.188:3300/0 conn(0x7f05bc0cf9a0 0x7f05bc15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 -- 172.21.15.141:0/4284963072 >> 172.21.15.141:0/4284963072 conn(0x7f05bc06a800 msgr2=0x7f05bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 -- 172.21.15.141:0/4284963072 shutdown_connections 2024-01-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.991+0000 7f05c3000640 1 -- 172.21.15.141:0/4284963072 wait complete. 2024-01-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.993+0000 7f05c3000640 1 Processor -- start 2024-01-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.993+0000 7f05c3000640 1 -- start start 2024-01-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.994+0000 7f05c3000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05bc0cf9a0 0x7f05bc0adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.994+0000 7f05c3000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05bc143ae0 0x7f05bc139860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.994+0000 7f05c3000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.994+0000 7f05c3000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0ae290 con 0x7f05bc15e1e0 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.994+0000 7f05c3000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc1f6ae0 con 0x7f05bc0cf9a0 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.994+0000 7f05c3000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc1f6f70 con 0x7f05bc143ae0 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55884/0 (socket says 172.21.15.141:55884) 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 -- 172.21.15.141:0/645793044 learned_addr learned my addr 172.21.15.141:0/645793044 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05bbe00640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05bc0cf9a0 0x7f05bc0adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 -- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05bc143ae0 msgr2=0x7f05bc139860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05bb400640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05bc143ae0 0x7f05bc139860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05bc143ae0 0x7f05bc139860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 -- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05bc0cf9a0 msgr2=0x7f05bc0adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05bc0cf9a0 0x7f05bc0adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 -- 172.21.15.141:0/645793044 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b4009b30 con 0x7f05bc15e1e0 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05bbe00640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05bc0cf9a0 0x7f05bc0adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.995+0000 7f05c1000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc159010 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009f40 tx=0x7f05b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.996+0000 7f05b3e00640 1 -- 172.21.15.141:0/645793044 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4009510 con 0x7f05bc15e1e0 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.996+0000 7f05b3e00640 1 -- 172.21.15.141:0/645793044 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b403e450 con 0x7f05bc15e1e0 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.996+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc139da0 con 0x7f05bc15e1e0 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.996+0000 7f05b3e00640 1 -- 172.21.15.141:0/645793044 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b403e7e0 con 0x7f05bc15e1e0 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.996+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc1f6490 con 0x7f05bc15e1e0 2024-01-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.997+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc1f6c70 con 0x7f05bc15e1e0 2024-01-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.998+0000 7f05b3e00640 1 -- 172.21.15.141:0/645793044 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f05b402cd50 con 0x7f05bc15e1e0 2024-01-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.998+0000 7f05b3e00640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f059406b380 0x7f059406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.999+0000 7f05b3e00640 1 -- 172.21.15.141:0/645793044 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05b40ac450 con 0x7f05bc15e1e0 2024-01-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:27.999+0000 7f05bbe00640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f059406b380 0x7f059406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.000+0000 7f05bbe00640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f059406b380 0x7f059406d840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f05ac003eb0 tx=0x7f05ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.003+0000 7f05b3e00640 1 -- 172.21.15.141:0/645793044 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05b40772e0 con 0x7f05bc15e1e0 2024-01-16T14:59:28.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.227+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f05bc15e670 con 0x7f059406b380 2024-01-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.381+0000 7f05b3e00640 1 -- 172.21.15.141:0/645793044 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f05ac005030 con 0x7f059406b380 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.383+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f059406b380 msgr2=0x7f059406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.383+0000 7f05c3000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f059406b380 0x7f059406d840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f05ac003eb0 tx=0x7f05ac009040 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.383+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 msgr2=0x7f05bc159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.383+0000 7f05c3000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc159010 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009f40 tx=0x7f05b4004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 shutdown_connections 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f059406b380 0x7f059406d840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05bc15e1e0 0x7f05bc159010 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05bc143ae0 0x7f05bc139860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 --2- 172.21.15.141:0/645793044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05bc0cf9a0 0x7f05bc0adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 >> 172.21.15.141:0/645793044 conn(0x7f05bc06a800 msgr2=0x7f05bc15c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 shutdown_connections 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.384+0000 7f05c3000640 1 -- 172.21.15.141:0/645793044 wait complete. 2024-01-16T14:59:28.369 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.388+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.370 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.388+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.370 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.388+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.370 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.388+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.370 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.388+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.390 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T14:59:28.390 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:28.400 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.419+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.400 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.419+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.400 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.419+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.400 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.419+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.401 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:28.419+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.617+0000 7f9fe5e00640 1 Processor -- start 2024-01-16T14:59:28.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.618+0000 7f9fe5e00640 1 -- start start 2024-01-16T14:59:28.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.619+0000 7f9fe5e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe01608b0 0x7f9fe0161970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.619+0000 7f9fe5e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe00d04e0 con 0x7f9fe00cf150 2024-01-16T14:59:28.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.619+0000 7f9fe5e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe0160c90 con 0x7f9fe00ce000 2024-01-16T14:59:28.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.619+0000 7f9fe5e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0161eb0 con 0x7f9fe01608b0 2024-01-16T14:59:28.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdfe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9fe00cf150 0x7f9fe00cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49328/0 (socket says 172.21.15.141:49328) 2024-01-16T14:59:28.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdfe00640 1 -- 172.21.15.141:0/248254841 learned_addr learned my addr 172.21.15.141:0/248254841 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fe4a00640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe01608b0 0x7f9fe0161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938457761 0 0) 0x7f9fe00d04e0 con 0x7f9fe00cf150 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc0003520 con 0x7f9fe00cf150 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728041662 0 0) 0x7f9fc0003520 con 0x7f9fe00cf150 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe01608b0 msgr2=0x7f9fe0161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdf400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe01608b0 0x7f9fe0161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.620+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 >> v1:172.21.15.188:6789/0 conn(0x7f9fe00ce000 legacy=0x7f9fe015cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe00cf8d0 con 0x7f9fe00cf150 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9fc80019f0 con 0x7f9fe00cf150 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 >> v1:172.21.15.141:6789/0 conn(0x7f9fe00cf150 legacy=0x7f9fe00cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fc00039a0 0x7f9fc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe01608b0 0x7f9fc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0161eb0 con 0x7f9fc0004d00 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc000ba30 con 0x7f9fc00039a0 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc000bbf0 con 0x7f9fe01608b0 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fe5400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fe4a00640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fc00039a0 0x7f9fc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fe5400640 1 -- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe01608b0 msgr2=0x7f9fc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fe5400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe01608b0 0x7f9fc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fe5400640 1 -- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fc00039a0 msgr2=0x7f9fc0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fe5400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fc00039a0 0x7f9fc0003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.621+0000 7f9fe5400640 1 -- 172.21.15.141:0/248254841 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc0003520 con 0x7f9fc0004d00 2024-01-16T14:59:28.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fe5400640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0071a60 tx=0x7f9fc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3599d0d1348c819 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd80052d0 con 0x7f9fc0004d00 2024-01-16T14:59:28.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd802e050 con 0x7f9fc0004d00 2024-01-16T14:59:28.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fdf400640 1 -- 172.21.15.141:0/248254841 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd80035f0 con 0x7f9fc0004d00 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fe5e00640 1 -- 172.21.15.141:0/248254841 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 msgr2=0x7f9fc0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fe5e00640 1 --2- 172.21.15.141:0/248254841 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0071a60 tx=0x7f9fc00057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fe5400640 1 -- 172.21.15.141:0/248254841 reap_dead start 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fe5e00640 1 -- 172.21.15.141:0/248254841 shutdown_connections 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.623+0000 7f9fe5e00640 1 -- 172.21.15.141:0/248254841 >> 172.21.15.141:0/248254841 conn(0x7f9fe00b2a60 msgr2=0x7f9fe00b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.624+0000 7f9fe5e00640 1 -- 172.21.15.141:0/248254841 shutdown_connections 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.624+0000 7f9fe5e00640 1 -- 172.21.15.141:0/248254841 wait complete. 2024-01-16T14:59:28.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.625+0000 7f9fe5e00640 1 Processor -- start 2024-01-16T14:59:28.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.625+0000 7f9fe5e00640 1 -- start start 2024-01-16T14:59:28.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.626+0000 7f9fe5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fe0144600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.626+0000 7f9fe5e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe0156170 0x7f9fe01565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.626+0000 7f9fe5e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe0142370 0x7f9fe01427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.626+0000 7f9fe5e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0144180 con 0x7f9fc0004d00 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.626+0000 7f9fe5e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0156b10 con 0x7f9fe0156170 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.626+0000 7f9fe5e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01430e0 con 0x7f9fe0142370 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fe0144600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fe0144600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55912/0 (socket says 172.21.15.141:55912) 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 -- 172.21.15.141:0/1163522178 learned_addr learned my addr 172.21.15.141:0/1163522178 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe5400640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe0142370 0x7f9fe01427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fdfe00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe0156170 0x7f9fe01565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 -- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe0142370 msgr2=0x7f9fe01427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe0142370 0x7f9fe01427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 -- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe0156170 msgr2=0x7f9fe01565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe0156170 0x7f9fe01565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 -- 172.21.15.141:0/1163522178 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe01433c0 con 0x7f9fc0004d00 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.627+0000 7f9fe4a00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fe0144600 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0010890 tx=0x7f9fd0010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.628+0000 7f9fdcc00640 1 -- 172.21.15.141:0/1163522178 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0009570 con 0x7f9fc0004d00 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.628+0000 7f9fdcc00640 1 -- 172.21.15.141:0/1163522178 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd000eb00 con 0x7f9fc0004d00 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.628+0000 7f9fdcc00640 1 -- 172.21.15.141:0/1163522178 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000eeb0 con 0x7f9fc0004d00 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.628+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe01436a0 con 0x7f9fc0004d00 2024-01-16T14:59:28.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.628+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe0143c60 con 0x7f9fc0004d00 2024-01-16T14:59:28.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.629+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe0067b60 con 0x7f9fc0004d00 2024-01-16T14:59:28.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.630+0000 7f9fdcc00640 1 -- 172.21.15.141:0/1163522178 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f9fd001c400 con 0x7f9fc0004d00 2024-01-16T14:59:28.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.631+0000 7f9fdcc00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb806b4a0 0x7f9fb806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.631+0000 7f9fdcc00640 1 -- 172.21.15.141:0/1163522178 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fd008a4d0 con 0x7f9fc0004d00 2024-01-16T14:59:28.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.632+0000 7f9fdfe00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb806b4a0 0x7f9fb806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.633+0000 7f9fdfe00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb806b4a0 0x7f9fb806d960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8001560 tx=0x7f9fc8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.636+0000 7f9fdcc00640 1 -- 172.21.15.141:0/1163522178 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fd0054f90 con 0x7f9fc0004d00 2024-01-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:28.862+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fe00d0170 con 0x7f9fb806b4a0 2024-01-16T14:59:31.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.460+0000 7f9fdcc00640 1 -- 172.21.15.141:0/1163522178 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f9fc80019d0 con 0x7f9fb806b4a0 2024-01-16T14:59:31.442 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:31.443 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:31.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.463+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb806b4a0 msgr2=0x7f9fb806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.463+0000 7f9fe5e00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb806b4a0 0x7f9fb806d960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8001560 tx=0x7f9fc8009040 comp rx=0 tx=0).stop 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.463+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 msgr2=0x7f9fe0144600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.463+0000 7f9fe5e00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fe0144600 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0010890 tx=0x7f9fd0010d50 comp rx=0 tx=0).stop 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 shutdown_connections 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fb806b4a0 0x7f9fb806d960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fe0142370 0x7f9fe01427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe0156170 0x7f9fe01565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 --2- 172.21.15.141:0/1163522178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fc0004d00 0x7f9fe0144600 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 >> 172.21.15.141:0/1163522178 conn(0x7f9fe00b2a60 msgr2=0x7f9fe015ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 shutdown_connections 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.464+0000 7f9fe5e00640 1 -- 172.21.15.141:0/1163522178 wait complete. 2024-01-16T14:59:31.466 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T14:59:31.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.684+0000 7ff705200640 1 Processor -- start 2024-01-16T14:59:31.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.684+0000 7ff705200640 1 -- start start 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.685+0000 7ff705200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000c9870 0x7ff7000ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.685+0000 7ff705200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff7000d0560 0x7ff700161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.685+0000 7ff705200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7000cf4c0 con 0x7ff7000d2260 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.685+0000 7ff705200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7000d2ef0 con 0x7ff7000d0560 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.685+0000 7ff705200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000c9c50 con 0x7ff7000c9870 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fd400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff7000d0560 0x7ff700161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fd400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff7000d0560 0x7ff700161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55920/0 (socket says 172.21.15.141:55920) 2024-01-16T14:59:31.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fd400640 1 -- 172.21.15.141:0/4179699913 learned_addr learned my addr 172.21.15.141:0/4179699913 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:31.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fde00640 1 --2- 172.21.15.141:0/4179699913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000c9870 0x7ff7000ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fd400640 1 -- 172.21.15.141:0/4179699913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000c9870 msgr2=0x7ff7000ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:31.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fd400640 1 --2- 172.21.15.141:0/4179699913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000c9870 0x7ff7000ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fd400640 1 -- 172.21.15.141:0/4179699913 >> v1:172.21.15.188:6789/0 conn(0x7ff7000d2260 legacy=0x7ff7000d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:31.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.686+0000 7ff6fd400640 1 -- 172.21.15.141:0/4179699913 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7000cf7a0 con 0x7ff7000d0560 2024-01-16T14:59:31.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.688+0000 7ff6fd400640 1 --2- 172.21.15.141:0/4179699913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000d0560 0x7ff700161030 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009e50 tx=0x7ff6e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a916245eb5059972 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.688+0000 7ff6fca00640 1 -- 172.21.15.141:0/4179699913 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e802ca70 con 0x7ff7000d0560 2024-01-16T14:59:31.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.688+0000 7ff6fca00640 1 -- 172.21.15.141:0/4179699913 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e802cbd0 con 0x7ff7000d0560 2024-01-16T14:59:31.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.688+0000 7ff6fca00640 1 -- 172.21.15.141:0/4179699913 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80343f0 con 0x7ff7000d0560 2024-01-16T14:59:31.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.688+0000 7ff705200640 1 -- 172.21.15.141:0/4179699913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000d0560 msgr2=0x7ff700161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.688+0000 7ff705200640 1 --2- 172.21.15.141:0/4179699913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000d0560 0x7ff700161030 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009e50 tx=0x7ff6e802bed0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.689+0000 7ff705200640 1 -- 172.21.15.141:0/4179699913 shutdown_connections 2024-01-16T14:59:31.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.689+0000 7ff705200640 1 --2- 172.21.15.141:0/4179699913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000d0560 0x7ff700161030 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.689+0000 7ff705200640 1 --2- 172.21.15.141:0/4179699913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000c9870 0x7ff7000ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.689+0000 7ff705200640 1 -- 172.21.15.141:0/4179699913 >> 172.21.15.141:0/4179699913 conn(0x7ff70009c560 msgr2=0x7ff7000d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:31.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.689+0000 7ff705200640 1 -- 172.21.15.141:0/4179699913 shutdown_connections 2024-01-16T14:59:31.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.689+0000 7ff705200640 1 -- 172.21.15.141:0/4179699913 wait complete. 2024-01-16T14:59:31.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.690+0000 7ff705200640 1 Processor -- start 2024-01-16T14:59:31.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.690+0000 7ff705200640 1 -- start start 2024-01-16T14:59:31.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.691+0000 7ff705200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000c9870 0x7ff7000afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.691+0000 7ff705200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000d0560 0x7ff700150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.691+0000 7ff705200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7000d2260 0x7ff7000c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.691+0000 7ff705200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000b5de0 con 0x7ff7000c9870 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.691+0000 7ff705200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000b60c0 con 0x7ff7000d2260 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.691+0000 7ff705200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000b63a0 con 0x7ff7000d0560 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000c9870 0x7ff7000afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000c9870 0x7ff7000afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55928/0 (socket says 172.21.15.141:55928) 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 -- 172.21.15.141:0/325730521 learned_addr learned my addr 172.21.15.141:0/325730521 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fd400640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000d0560 0x7ff700150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fe800640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7000d2260 0x7ff7000c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 -- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000d0560 msgr2=0x7ff700150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000d0560 0x7ff700150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 -- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7000d2260 msgr2=0x7ff7000c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7000d2260 0x7ff7000c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 -- 172.21.15.141:0/325730521 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8009b30 con 0x7ff7000c9870 2024-01-16T14:59:31.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fe800640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7000d2260 0x7ff7000c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.692+0000 7ff6fde00640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000c9870 0x7ff7000afdc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000d5f0 tx=0x7ff6f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.693+0000 7ff6f6000640 1 -- 172.21.15.141:0/325730521 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f00125f0 con 0x7ff7000c9870 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.693+0000 7ff6f6000640 1 -- 172.21.15.141:0/325730521 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f0012750 con 0x7ff7000c9870 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.693+0000 7ff6f6000640 1 -- 172.21.15.141:0/325730521 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0012b00 con 0x7ff7000c9870 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.693+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7000b6680 con 0x7ff7000c9870 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.693+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7000b0300 con 0x7ff7000c9870 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.694+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7000675d0 con 0x7ff7000c9870 2024-01-16T14:59:31.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.695+0000 7ff6f6000640 1 -- 172.21.15.141:0/325730521 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7ff6f0012c60 con 0x7ff7000c9870 2024-01-16T14:59:31.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.696+0000 7ff6f6000640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6d806b470 0x7ff6d806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.696+0000 7ff6f6000640 1 -- 172.21.15.141:0/325730521 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6f008a320 con 0x7ff7000c9870 2024-01-16T14:59:31.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.696+0000 7ff6fd400640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6d806b470 0x7ff6d806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.698+0000 7ff6fd400640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6d806b470 0x7ff6d806d930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8035000 tx=0x7ff6e8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.700+0000 7ff6f6000640 1 -- 172.21.15.141:0/325730521 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6f0054fa0 con 0x7ff7000c9870 2024-01-16T14:59:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.957+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff7000d0a20 con 0x7ff7000c9870 2024-01-16T14:59:31.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.957+0000 7ff6f6000640 1 -- 172.21.15.141:0/325730521 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1855 (secure 0 0 0) 0x7ff6f00585c0 con 0x7ff7000c9870 2024-01-16T14:59:31.939 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:31.939 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:31.939 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T14:59:31.939 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T14:59:31.939 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T14:59:31.939 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T14:59:31.940 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 164, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 35, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T14:59:31.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 796, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222103727, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6685802496, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766408310784, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T14:59:31.942 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 85, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 64298623, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 17 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 18, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 2, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 2, 2024-01-16T14:59:31.943 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 4762 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T14:59:31.944 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 7, 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T14:59:28.486818+0000", 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": {} 2024-01-16T14:59:31.945 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:31.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6d806b470 msgr2=0x7ff6d806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6d806b470 0x7ff6d806d930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8035000 tx=0x7ff6e8030000 comp rx=0 tx=0).stop 2024-01-16T14:59:31.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000c9870 msgr2=0x7ff7000afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000c9870 0x7ff7000afdc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000d5f0 tx=0x7ff6f000dab0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 shutdown_connections 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6d806b470 0x7ff6d806d930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7000d2260 0x7ff7000c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff7000d0560 0x7ff700150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 --2- 172.21.15.141:0/325730521 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff7000c9870 0x7ff7000afdc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 >> 172.21.15.141:0/325730521 conn(0x7ff70009c560 msgr2=0x7ff700160800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 shutdown_connections 2024-01-16T14:59:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:31.960+0000 7ff705200640 1 -- 172.21.15.141:0/325730521 wait complete. 2024-01-16T14:59:31.961 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T14:59:31.961 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:32.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.196+0000 7f8d3dc00640 1 Processor -- start 2024-01-16T14:59:32.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.197+0000 7f8d3dc00640 1 -- start start 2024-01-16T14:59:32.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.198+0000 7f8d3dc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8d3813fcb0 0x7f8d380cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.198+0000 7f8d3dc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8d380cdc20 0x7f8d380b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.198+0000 7f8d3dc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380ce440 con 0x7f8d3813cf10 2024-01-16T14:59:32.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.198+0000 7f8d3dc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380ce720 con 0x7f8d3813fcb0 2024-01-16T14:59:32.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.198+0000 7f8d3dc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380cea00 con 0x7f8d380cdc20 2024-01-16T14:59:32.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.198+0000 7f8d37e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8d3813cf10 0x7f8d3813f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49330/0 (socket says 172.21.15.141:49330) 2024-01-16T14:59:32.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d37e00640 1 -- 172.21.15.141:0/3352476561 learned_addr learned my addr 172.21.15.141:0/3352476561 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:32.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d37400640 1 --2- 172.21.15.141:0/3352476561 >> v2:172.21.15.141:3300/0 conn(0x7f8d3813fcb0 0x7f8d380cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d3d200640 1 --2- 172.21.15.141:0/3352476561 >> v2:172.21.15.188:3300/0 conn(0x7f8d380cdc20 0x7f8d380b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d36a00640 1 -- 172.21.15.141:0/3352476561 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590824138 0 0) 0x7f8d380ce440 con 0x7f8d3813cf10 2024-01-16T14:59:32.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d36a00640 1 -- 172.21.15.141:0/3352476561 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d18003520 con 0x7f8d3813cf10 2024-01-16T14:59:32.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d37400640 1 -- 172.21.15.141:0/3352476561 >> v2:172.21.15.188:3300/0 conn(0x7f8d380cdc20 msgr2=0x7f8d380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:32.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d37400640 1 --2- 172.21.15.141:0/3352476561 >> v2:172.21.15.188:3300/0 conn(0x7f8d380cdc20 0x7f8d380b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d37400640 1 -- 172.21.15.141:0/3352476561 >> v1:172.21.15.141:6789/0 conn(0x7f8d3813cf10 legacy=0x7f8d3813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d37400640 1 -- 172.21.15.141:0/3352476561 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d380b1d90 con 0x7f8d3813fcb0 2024-01-16T14:59:32.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.199+0000 7f8d3d200640 1 --2- 172.21.15.141:0/3352476561 >> v2:172.21.15.188:3300/0 conn(0x7f8d380cdc20 0x7f8d380b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:32.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.201+0000 7f8d37400640 1 --2- 172.21.15.141:0/3352476561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813fcb0 0x7f8d380cd6e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8d38154a30 tx=0x7f8d2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20cbbd93bd20152 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.201+0000 7f8d36a00640 1 -- 172.21.15.141:0/3352476561 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20031830 con 0x7f8d3813fcb0 2024-01-16T14:59:32.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.201+0000 7f8d36a00640 1 -- 172.21.15.141:0/3352476561 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d20031990 con 0x7f8d3813fcb0 2024-01-16T14:59:32.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.201+0000 7f8d36a00640 1 -- 172.21.15.141:0/3352476561 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20031d00 con 0x7f8d3813fcb0 2024-01-16T14:59:32.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.201+0000 7f8d3dc00640 1 -- 172.21.15.141:0/3352476561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813fcb0 msgr2=0x7f8d380cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.201+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/3352476561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813fcb0 0x7f8d380cd6e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8d38154a30 tx=0x7f8d2002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.202+0000 7f8d3dc00640 1 -- 172.21.15.141:0/3352476561 shutdown_connections 2024-01-16T14:59:32.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.202+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/3352476561 >> v2:172.21.15.188:3300/0 conn(0x7f8d380cdc20 0x7f8d380b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.202+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/3352476561 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813fcb0 0x7f8d380cd6e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.202+0000 7f8d3dc00640 1 -- 172.21.15.141:0/3352476561 >> 172.21.15.141:0/3352476561 conn(0x7f8d3806a7d0 msgr2=0x7f8d3806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:32.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.202+0000 7f8d3dc00640 1 -- 172.21.15.141:0/3352476561 shutdown_connections 2024-01-16T14:59:32.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.202+0000 7f8d3dc00640 1 -- 172.21.15.141:0/3352476561 wait complete. 2024-01-16T14:59:32.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.203+0000 7f8d3dc00640 1 Processor -- start 2024-01-16T14:59:32.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.204+0000 7f8d3dc00640 1 -- start start 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.204+0000 7f8d3dc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d380cdc20 0x7f8d38070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.204+0000 7f8d3dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813cf10 0x7f8d3814f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.204+0000 7f8d3dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d3813fcb0 0x7f8d3814c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.204+0000 7f8d3dc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d38072100 con 0x7f8d3813cf10 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.204+0000 7f8d3dc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3814fed0 con 0x7f8d380cdc20 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.204+0000 7f8d3dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3814cb50 con 0x7f8d3813fcb0 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813cf10 0x7f8d3814f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813cf10 0x7f8d3814f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55952/0 (socket says 172.21.15.141:55952) 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 -- 172.21.15.141:0/1056712359 learned_addr learned my addr 172.21.15.141:0/1056712359 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 -- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d3813fcb0 msgr2=0x7f8d3814c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d3813fcb0 0x7f8d3814c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 -- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d380cdc20 msgr2=0x7f8d38070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d380cdc20 0x7f8d38070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 -- 172.21.15.141:0/1056712359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d20009400 con 0x7f8d3813cf10 2024-01-16T14:59:32.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.205+0000 7f8d37400640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813cf10 0x7f8d3814f990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8d20000c00 tx=0x7f8d200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.206+0000 7f8d2fe00640 1 -- 172.21.15.141:0/1056712359 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2003a830 con 0x7f8d3813cf10 2024-01-16T14:59:32.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.206+0000 7f8d2fe00640 1 -- 172.21.15.141:0/1056712359 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2003a990 con 0x7f8d3813cf10 2024-01-16T14:59:32.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.206+0000 7f8d2fe00640 1 -- 172.21.15.141:0/1056712359 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2003ad20 con 0x7f8d3813cf10 2024-01-16T14:59:32.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.206+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d38071480 con 0x7f8d3813cf10 2024-01-16T14:59:32.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.206+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d38071a40 con 0x7f8d3813cf10 2024-01-16T14:59:32.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.207+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d38067450 con 0x7f8d3813cf10 2024-01-16T14:59:32.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.208+0000 7f8d2fe00640 1 -- 172.21.15.141:0/1056712359 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f8d2004d440 con 0x7f8d3813cf10 2024-01-16T14:59:32.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.209+0000 7f8d2fe00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d1006b470 0x7f8d1006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.209+0000 7f8d2fe00640 1 -- 172.21.15.141:0/1056712359 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d200ac250 con 0x7f8d3813cf10 2024-01-16T14:59:32.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.209+0000 7f8d37e00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d1006b470 0x7f8d1006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.210+0000 7f8d37e00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d1006b470 0x7f8d1006d930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8d28000f60 tx=0x7f8d280078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.214+0000 7f8d2fe00640 1 -- 172.21.15.141:0/1056712359 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d20076c00 con 0x7f8d3813cf10 2024-01-16T14:59:32.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.442+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d380b2ad0 con 0x7f8d1006b470 2024-01-16T14:59:32.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.460+0000 7f8d2fe00640 1 -- 172.21.15.141:0/1056712359 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f8d28005890 con 0x7f8d1006b470 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 0.0, 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "0 B/3 GB", 2024-01-16T14:59:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "0/3", 2024-01-16T14:59:32.442 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:32.442 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:32.442 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:32.442 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:32.442 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d1006b470 msgr2=0x7f8d1006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d1006b470 0x7f8d1006d930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8d28000f60 tx=0x7f8d280078f0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813cf10 msgr2=0x7f8d3814f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813cf10 0x7f8d3814f990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8d20000c00 tx=0x7f8d200046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 shutdown_connections 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d1006b470 0x7f8d1006d930 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d3813fcb0 0x7f8d3814c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d3813cf10 0x7f8d3814f990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 --2- 172.21.15.141:0/1056712359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d380cdc20 0x7f8d38070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 >> 172.21.15.141:0/1056712359 conn(0x7f8d3806a7d0 msgr2=0x7f8d381528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:32.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 shutdown_connections 2024-01-16T14:59:32.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:32.461+0000 7f8d3dc00640 1 -- 172.21.15.141:0/1056712359 wait complete. 2024-01-16T14:59:32.462 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:32.650 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:32.658 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:33.463 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:33.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.694+0000 7feeb7200640 1 Processor -- start 2024-01-16T14:59:33.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.694+0000 7feeb7200640 1 -- start start 2024-01-16T14:59:33.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb7200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013cf10 0x7feeb013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb7200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feeb00cddf0 0x7feeb00b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb7200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb00ce440 con 0x7feeb013fae0 2024-01-16T14:59:33.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb7200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb013ff60 con 0x7feeb00cddf0 2024-01-16T14:59:33.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb7200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00ce720 con 0x7feeb013cf10 2024-01-16T14:59:33.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb6800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feeb00cddf0 0x7feeb00b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb5400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7feeb013fae0 0x7feeb00cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60176/0 (socket says 172.21.15.141:60176) 2024-01-16T14:59:33.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.696+0000 7feeb5400640 1 -- 172.21.15.141:0/3738180076 learned_addr learned my addr 172.21.15.141:0/3738180076 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:33.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.697+0000 7feeb6800640 1 -- 172.21.15.141:0/3738180076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013cf10 msgr2=0x7feeb013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.697+0000 7feeb6800640 1 --2- 172.21.15.141:0/3738180076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013cf10 0x7feeb013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.697+0000 7feeb6800640 1 -- 172.21.15.141:0/3738180076 >> v1:172.21.15.141:6789/0 conn(0x7feeb013fae0 legacy=0x7feeb00cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.697+0000 7feeb6800640 1 -- 172.21.15.141:0/3738180076 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb00cea00 con 0x7feeb00cddf0 2024-01-16T14:59:33.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.698+0000 7feeb6800640 1 --2- 172.21.15.141:0/3738180076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb00cddf0 0x7feeb00b2590 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7feea8009d90 tx=0x7feea802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985ddc4612dbe611 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.698+0000 7feeb4a00640 1 -- 172.21.15.141:0/3738180076 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea802cc60 con 0x7feeb00cddf0 2024-01-16T14:59:33.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.698+0000 7feeb4a00640 1 -- 172.21.15.141:0/3738180076 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea802cdc0 con 0x7feeb00cddf0 2024-01-16T14:59:33.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.698+0000 7feeb4a00640 1 -- 172.21.15.141:0/3738180076 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8034660 con 0x7feeb00cddf0 2024-01-16T14:59:33.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 -- 172.21.15.141:0/3738180076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb00cddf0 msgr2=0x7feeb00b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 --2- 172.21.15.141:0/3738180076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb00cddf0 0x7feeb00b2590 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7feea8009d90 tx=0x7feea802beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 -- 172.21.15.141:0/3738180076 shutdown_connections 2024-01-16T14:59:33.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 --2- 172.21.15.141:0/3738180076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb00cddf0 0x7feeb00b2590 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 --2- 172.21.15.141:0/3738180076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013cf10 0x7feeb013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 -- 172.21.15.141:0/3738180076 >> 172.21.15.141:0/3738180076 conn(0x7feeb006a7d0 msgr2=0x7feeb006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 -- 172.21.15.141:0/3738180076 shutdown_connections 2024-01-16T14:59:33.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.699+0000 7feeb7200640 1 -- 172.21.15.141:0/3738180076 wait complete. 2024-01-16T14:59:33.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.701+0000 7feeb7200640 1 Processor -- start 2024-01-16T14:59:33.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.701+0000 7feeb7200640 1 -- start start 2024-01-16T14:59:33.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.702+0000 7feeb7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00cddf0 0x7feeb013af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.702+0000 7feeb7200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb013cf10 0x7feeb0070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.702+0000 7feeb7200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013fae0 0x7feeb0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.702+0000 7feeb7200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb013dbb0 con 0x7feeb013cf10 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.702+0000 7feeb7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb013b5a0 con 0x7feeb00cddf0 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.702+0000 7feeb7200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb013b880 con 0x7feeb013fae0 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.702+0000 7feeb5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb013cf10 0x7feeb0070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb013cf10 0x7feeb0070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33904/0 (socket says 172.21.15.141:33904) 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 -- 172.21.15.141:0/2561906280 learned_addr learned my addr 172.21.15.141:0/2561906280 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 -- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013fae0 msgr2=0x7feeb0071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013fae0 0x7feeb0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 -- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00cddf0 msgr2=0x7feeb013af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00cddf0 0x7feeb013af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 -- 172.21.15.141:0/2561906280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea8009a70 con 0x7feeb013cf10 2024-01-16T14:59:33.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.703+0000 7feeb5400640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb013cf10 0x7feeb0070ae0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c3c0 tx=0x7fee9800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.704+0000 7feeae000640 1 -- 172.21.15.141:0/2561906280 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee98005ce0 con 0x7feeb013cf10 2024-01-16T14:59:33.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.704+0000 7feeae000640 1 -- 172.21.15.141:0/2561906280 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee98005e40 con 0x7feeb013cf10 2024-01-16T14:59:33.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.704+0000 7feeae000640 1 -- 172.21.15.141:0/2561906280 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980072d0 con 0x7feeb013cf10 2024-01-16T14:59:33.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.704+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb013bb60 con 0x7feeb013cf10 2024-01-16T14:59:33.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.704+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb013be40 con 0x7feeb013cf10 2024-01-16T14:59:33.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.705+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb0067450 con 0x7feeb013cf10 2024-01-16T14:59:33.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.707+0000 7feeae000640 1 -- 172.21.15.141:0/2561906280 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fee980074f0 con 0x7feeb013cf10 2024-01-16T14:59:33.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.708+0000 7feeae000640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806b470 0x7fee8806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.708+0000 7feeae000640 1 -- 172.21.15.141:0/2561906280 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee98089460 con 0x7feeb013cf10 2024-01-16T14:59:33.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.709+0000 7feeb5e00640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806b470 0x7fee8806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.711+0000 7feeb5e00640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806b470 0x7fee8806d930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feea0009710 tx=0x7feea0006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:33.714+0000 7feeae000640 1 -- 172.21.15.141:0/2561906280 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee98054050 con 0x7feeb013cf10 2024-01-16T14:59:34.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.030+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7feeb00b2070 con 0x7fee8806b470 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.044+0000 7feeae000640 1 -- 172.21.15.141:0/2561906280 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7feea000b480 con 0x7fee8806b470 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:34.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 0.0, 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "0 B/3 GB", 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "0/3", 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:34.026 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:34.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.045+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806b470 msgr2=0x7fee8806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.045+0000 7feeb7200640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806b470 0x7fee8806d930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feea0009710 tx=0x7feea0006c90 comp rx=0 tx=0).stop 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb013cf10 msgr2=0x7feeb0070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb013cf10 0x7feeb0070ae0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c3c0 tx=0x7fee9800c880 comp rx=0 tx=0).stop 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 shutdown_connections 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806b470 0x7fee8806d930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb013fae0 0x7feeb0071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb013cf10 0x7feeb0070ae0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 --2- 172.21.15.141:0/2561906280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00cddf0 0x7feeb013af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 >> 172.21.15.141:0/2561906280 conn(0x7feeb006a7d0 msgr2=0x7feeb00adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 shutdown_connections 2024-01-16T14:59:34.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:34.046+0000 7feeb7200640 1 -- 172.21.15.141:0/2561906280 wait complete. 2024-01-16T14:59:34.046 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:35.046 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.356+0000 7f7e94c00640 1 Processor -- start 2024-01-16T14:59:35.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.356+0000 7f7e94c00640 1 -- start start 2024-01-16T14:59:35.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.357+0000 7f7e94c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e90143ae0 0x7f7e900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.357+0000 7f7e94c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e900cf7d0 0x7f7e9015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.357+0000 7f7e94c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e900cfc30 con 0x7f7e9015e1e0 2024-01-16T14:59:35.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.357+0000 7f7e94c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e900b1ee0 con 0x7f7e90143ae0 2024-01-16T14:59:35.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.357+0000 7f7e94c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900b2780 con 0x7f7e900cf7d0 2024-01-16T14:59:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e8de00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e90143ae0 0x7f7e900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e8de00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e90143ae0 0x7f7e900cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33918/0 (socket says 172.21.15.141:33918) 2024-01-16T14:59:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e8de00640 1 -- 172.21.15.141:0/3208636442 learned_addr learned my addr 172.21.15.141:0/3208636442 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e87e00640 1 --2- 172.21.15.141:0/3208636442 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e900cf7d0 0x7f7e9015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e8de00640 1 -- 172.21.15.141:0/3208636442 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e900cf7d0 msgr2=0x7f7e9015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e8de00640 1 --2- 172.21.15.141:0/3208636442 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e900cf7d0 0x7f7e9015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e8de00640 1 -- 172.21.15.141:0/3208636442 >> v1:172.21.15.188:6789/0 conn(0x7f7e9015e1e0 legacy=0x7f7e9015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.358+0000 7f7e8de00640 1 -- 172.21.15.141:0/3208636442 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9015e990 con 0x7f7e90143ae0 2024-01-16T14:59:35.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.360+0000 7f7e8de00640 1 --2- 172.21.15.141:0/3208636442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e90143ae0 0x7f7e900cf290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009870 tx=0x7f7e8002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cac091f927bcc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.360+0000 7f7e8d400640 1 -- 172.21.15.141:0/3208636442 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80030840 con 0x7f7e90143ae0 2024-01-16T14:59:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.360+0000 7f7e8d400640 1 -- 172.21.15.141:0/3208636442 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e800309a0 con 0x7f7e90143ae0 2024-01-16T14:59:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.360+0000 7f7e8d400640 1 -- 172.21.15.141:0/3208636442 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80030d30 con 0x7f7e90143ae0 2024-01-16T14:59:35.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.361+0000 7f7e94c00640 1 -- 172.21.15.141:0/3208636442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e90143ae0 msgr2=0x7f7e900cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.361+0000 7f7e94c00640 1 --2- 172.21.15.141:0/3208636442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e90143ae0 0x7f7e900cf290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009870 tx=0x7f7e8002b980 comp rx=0 tx=0).stop 2024-01-16T14:59:35.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.362+0000 7f7e94c00640 1 -- 172.21.15.141:0/3208636442 shutdown_connections 2024-01-16T14:59:35.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.362+0000 7f7e94c00640 1 --2- 172.21.15.141:0/3208636442 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e900cf7d0 0x7f7e9015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.362+0000 7f7e94c00640 1 --2- 172.21.15.141:0/3208636442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e90143ae0 0x7f7e900cf290 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.362+0000 7f7e94c00640 1 -- 172.21.15.141:0/3208636442 >> 172.21.15.141:0/3208636442 conn(0x7f7e9006a800 msgr2=0x7f7e9006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.362+0000 7f7e94c00640 1 -- 172.21.15.141:0/3208636442 shutdown_connections 2024-01-16T14:59:35.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.362+0000 7f7e94c00640 1 -- 172.21.15.141:0/3208636442 wait complete. 2024-01-16T14:59:35.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.365+0000 7f7e94c00640 1 Processor -- start 2024-01-16T14:59:35.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.365+0000 7f7e94c00640 1 -- start start 2024-01-16T14:59:35.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.367+0000 7f7e94c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e900cf7d0 0x7f7e901328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.367+0000 7f7e94c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e90143ae0 0x7f7e900ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.367+0000 7f7e94c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9015e1e0 0x7f7e900ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.367+0000 7f7e94c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9015d660 con 0x7f7e900cf7d0 2024-01-16T14:59:35.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.367+0000 7f7e94c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9015f0e0 con 0x7f7e90143ae0 2024-01-16T14:59:35.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.367+0000 7f7e94c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e901678d0 con 0x7f7e9015e1e0 2024-01-16T14:59:35.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e8de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e900cf7d0 0x7f7e901328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e8de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e900cf7d0 0x7f7e901328a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33928/0 (socket says 172.21.15.141:33928) 2024-01-16T14:59:35.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e8de00640 1 -- 172.21.15.141:0/113654292 learned_addr learned my addr 172.21.15.141:0/113654292 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e87e00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e90143ae0 0x7f7e900ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e8e800640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9015e1e0 0x7f7e900ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e87e00640 1 -- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9015e1e0 msgr2=0x7f7e900ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e87e00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9015e1e0 0x7f7e900ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e87e00640 1 -- 172.21.15.141:0/113654292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e900cf7d0 msgr2=0x7f7e901328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e87e00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e900cf7d0 0x7f7e901328a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.368+0000 7f7e87e00640 1 -- 172.21.15.141:0/113654292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800094d0 con 0x7f7e90143ae0 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.369+0000 7f7e8de00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e900cf7d0 0x7f7e901328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:35.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.369+0000 7f7e87e00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e90143ae0 0x7f7e900ad920 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800d5c0 tx=0x7f7e7800da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.369+0000 7f7e86000640 1 -- 172.21.15.141:0/113654292 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e780125f0 con 0x7f7e90143ae0 2024-01-16T14:59:35.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.369+0000 7f7e86000640 1 -- 172.21.15.141:0/113654292 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e78012750 con 0x7f7e90143ae0 2024-01-16T14:59:35.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.369+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e90166fa0 con 0x7f7e90143ae0 2024-01-16T14:59:35.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.370+0000 7f7e86000640 1 -- 172.21.15.141:0/113654292 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78012ae0 con 0x7f7e90143ae0 2024-01-16T14:59:35.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.370+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e90167560 con 0x7f7e90143ae0 2024-01-16T14:59:35.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.371+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e900b2bf0 con 0x7f7e90143ae0 2024-01-16T14:59:35.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.374+0000 7f7e86000640 1 -- 172.21.15.141:0/113654292 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f7e78012c40 con 0x7f7e90143ae0 2024-01-16T14:59:35.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.376+0000 7f7e86000640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e6806b470 0x7f7e6806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.376+0000 7f7e86000640 1 -- 172.21.15.141:0/113654292 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e7808abf0 con 0x7f7e90143ae0 2024-01-16T14:59:35.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.377+0000 7f7e8de00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e6806b470 0x7f7e6806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.382+0000 7f7e8de00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e6806b470 0x7f7e6806d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009870 tx=0x7f7e80008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.384+0000 7f7e86000640 1 -- 172.21.15.141:0/113654292 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e78055870 con 0x7f7e90143ae0 2024-01-16T14:59:35.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.765+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e9015e640 con 0x7f7e6806b470 2024-01-16T14:59:35.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.776+0000 7f7e86000640 1 -- 172.21.15.141:0/113654292 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f7e80039550 con 0x7f7e6806b470 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 33.33333333333333, 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "1/3", 2024-01-16T14:59:35.757 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:35.758 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:35.758 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:35.758 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:35.758 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:35.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.778+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e6806b470 msgr2=0x7f7e6806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.778+0000 7f7e94c00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e6806b470 0x7f7e6806d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009870 tx=0x7f7e80008030 comp rx=0 tx=0).stop 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.778+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e90143ae0 msgr2=0x7f7e900ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.778+0000 7f7e94c00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e90143ae0 0x7f7e900ad920 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800d5c0 tx=0x7f7e7800da80 comp rx=0 tx=0).stop 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 shutdown_connections 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e6806b470 0x7f7e6806d930 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e9015e1e0 0x7f7e900ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e90143ae0 0x7f7e900ad920 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 --2- 172.21.15.141:0/113654292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e900cf7d0 0x7f7e901328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 >> 172.21.15.141:0/113654292 conn(0x7f7e9006a800 msgr2=0x7f7e900ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 shutdown_connections 2024-01-16T14:59:35.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:35.779+0000 7f7e94c00640 1 -- 172.21.15.141:0/113654292 wait complete. 2024-01-16T14:59:35.777 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:36.778 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.020+0000 7f24ebe00640 1 Processor -- start 2024-01-16T14:59:37.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.020+0000 7f24ebe00640 1 -- start start 2024-01-16T14:59:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.021+0000 7f24ebe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f24ec15d4b0 0x7f24ec0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24ebe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f24ec0cf9a0 0x7f24ec0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24ebe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec0d0320 con 0x7f24ec0cddf0 2024-01-16T14:59:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24ebe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec0b2730 con 0x7f24ec15d4b0 2024-01-16T14:59:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24ebe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec0d1ec0 con 0x7f24ec0cf9a0 2024-01-16T14:59:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eaa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f24ec15d4b0 0x7f24ec0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eb400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f24ec0cddf0 0x7f24ec0ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60184/0 (socket says 172.21.15.141:60184) 2024-01-16T14:59:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eaa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f24ec15d4b0 0x7f24ec0d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33942/0 (socket says 172.21.15.141:33942) 2024-01-16T14:59:37.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eb400640 1 -- 172.21.15.141:0/1030432953 learned_addr learned my addr 172.21.15.141:0/1030432953 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eaa00640 1 -- 172.21.15.141:0/1030432953 >> v2:172.21.15.188:3300/0 conn(0x7f24ec0cf9a0 msgr2=0x7f24ec0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24ea000640 1 --2- 172.21.15.141:0/1030432953 >> v2:172.21.15.188:3300/0 conn(0x7f24ec0cf9a0 0x7f24ec0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eaa00640 1 --2- 172.21.15.141:0/1030432953 >> v2:172.21.15.188:3300/0 conn(0x7f24ec0cf9a0 0x7f24ec0cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24e9600640 1 -- 172.21.15.141:0/1030432953 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133010549 0 0) 0x7f24ec0d0320 con 0x7f24ec0cddf0 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eaa00640 1 -- 172.21.15.141:0/1030432953 >> v1:172.21.15.141:6789/0 conn(0x7f24ec0cddf0 legacy=0x7f24ec0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.022+0000 7f24eaa00640 1 -- 172.21.15.141:0/1030432953 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec0b29d0 con 0x7f24ec15d4b0 2024-01-16T14:59:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24eaa00640 1 --2- 172.21.15.141:0/1030432953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec0d1550 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f24ec08b580 tx=0x7f24dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4affbcc68bb8735 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24e9600640 1 -- 172.21.15.141:0/1030432953 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc031830 con 0x7f24ec15d4b0 2024-01-16T14:59:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24e9600640 1 -- 172.21.15.141:0/1030432953 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc031990 con 0x7f24ec15d4b0 2024-01-16T14:59:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24e9600640 1 -- 172.21.15.141:0/1030432953 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc031d20 con 0x7f24ec15d4b0 2024-01-16T14:59:37.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24ebe00640 1 -- 172.21.15.141:0/1030432953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 msgr2=0x7f24ec0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1030432953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec0d1550 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f24ec08b580 tx=0x7f24dc02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24ebe00640 1 -- 172.21.15.141:0/1030432953 shutdown_connections 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1030432953 >> v2:172.21.15.188:3300/0 conn(0x7f24ec0cf9a0 0x7f24ec0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1030432953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec0d1550 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.024+0000 7f24ebe00640 1 -- 172.21.15.141:0/1030432953 >> 172.21.15.141:0/1030432953 conn(0x7f24ec0d3000 msgr2=0x7f24ec0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.025+0000 7f24ebe00640 1 -- 172.21.15.141:0/1030432953 shutdown_connections 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.025+0000 7f24ebe00640 1 -- 172.21.15.141:0/1030432953 wait complete. 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.026+0000 7f24ebe00640 1 Processor -- start 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.026+0000 7f24ebe00640 1 -- start start 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.027+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24ec0cddf0 0x7f24ec153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.027+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec0cf9a0 0x7f24ec14f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.027+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec09e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.027+0000 7f24ebe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec0cace0 con 0x7f24ec15d4b0 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.027+0000 7f24ebe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec0ca0d0 con 0x7f24ec0cf9a0 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.027+0000 7f24ebe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec0ca3b0 con 0x7f24ec0cddf0 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec09e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec09e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33948/0 (socket says 172.21.15.141:33948) 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 -- 172.21.15.141:0/1972061291 learned_addr learned my addr 172.21.15.141:0/1972061291 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eaa00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24ec0cddf0 0x7f24ec153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24ea000640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec0cf9a0 0x7f24ec14f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 -- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24ec0cddf0 msgr2=0x7f24ec153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24ec0cddf0 0x7f24ec153330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 -- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec0cf9a0 msgr2=0x7f24ec14f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec0cf9a0 0x7f24ec14f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 -- 172.21.15.141:0/1972061291 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc009400 con 0x7f24ec15d4b0 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.028+0000 7f24eb400640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec09e900 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c7e0 tx=0x7f24e400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.029+0000 7f24e2a00640 1 -- 172.21.15.141:0/1972061291 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e400a640 con 0x7f24ec15d4b0 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.029+0000 7f24e2a00640 1 -- 172.21.15.141:0/1972061291 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e400a7c0 con 0x7f24ec15d4b0 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.029+0000 7f24e2a00640 1 -- 172.21.15.141:0/1972061291 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e400ab70 con 0x7f24ec15d4b0 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.029+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec15da10 con 0x7f24ec15d4b0 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.029+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec0ca970 con 0x7f24ec15d4b0 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.030+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec0cae90 con 0x7f24ec15d4b0 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.031+0000 7f24e2a00640 1 -- 172.21.15.141:0/1972061291 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f24e400acd0 con 0x7f24ec15d4b0 2024-01-16T14:59:37.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.032+0000 7f24e2a00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f24c406b4a0 0x7f24c406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.032+0000 7f24e2a00640 1 -- 172.21.15.141:0/1972061291 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24e40893e0 con 0x7f24ec15d4b0 2024-01-16T14:59:37.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.033+0000 7f24eaa00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f24c406b4a0 0x7f24c406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.034+0000 7f24eaa00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f24c406b4a0 0x7f24c406d960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f24dc036000 tx=0x7f24dc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.036+0000 7f24e2a00640 1 -- 172.21.15.141:0/1972061291 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24ec0cae90 con 0x7f24ec15d4b0 2024-01-16T14:59:37.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.265+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f24ec15d890 con 0x7f24c406b4a0 2024-01-16T14:59:37.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.279+0000 7f24e2a00640 1 -- 172.21.15.141:0/1972061291 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f24dc03a420 con 0x7f24c406b4a0 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 33.33333333333333, 2024-01-16T14:59:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T14:59:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "1/3", 2024-01-16T14:59:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:37.261 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.281+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f24c406b4a0 msgr2=0x7f24c406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.281+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f24c406b4a0 0x7f24c406d960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f24dc036000 tx=0x7f24dc03c040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.281+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 msgr2=0x7f24ec09e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.281+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec09e900 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c7e0 tx=0x7f24e400cca0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 shutdown_connections 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f24c406b4a0 0x7f24c406d960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24ec15d4b0 0x7f24ec09e900 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec0cf9a0 0x7f24ec14f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 --2- 172.21.15.141:0/1972061291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24ec0cddf0 0x7f24ec153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 >> 172.21.15.141:0/1972061291 conn(0x7f24ec0d3000 msgr2=0x7f24ec06c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 shutdown_connections 2024-01-16T14:59:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:37.282+0000 7f24ebe00640 1 -- 172.21.15.141:0/1972061291 wait complete. 2024-01-16T14:59:37.282 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:38.283 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:38.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.556+0000 7ff781400640 1 Processor -- start 2024-01-16T14:59:38.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.556+0000 7ff781400640 1 -- start start 2024-01-16T14:59:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.557+0000 7ff781400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff77c13df80 0x7ff77c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.557+0000 7ff781400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff77c0cf9a0 0x7ff77c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.557+0000 7ff781400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c06c470 con 0x7ff77c13e360 2024-01-16T14:59:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.557+0000 7ff781400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c06c750 con 0x7ff77c13df80 2024-01-16T14:59:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.557+0000 7ff781400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c06ca30 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.558+0000 7ff77a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff77c13df80 0x7ff77c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.558+0000 7ff77a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff77c13df80 0x7ff77c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33964/0 (socket says 172.21.15.141:33964) 2024-01-16T14:59:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.558+0000 7ff77a800640 1 -- 172.21.15.141:0/2487485519 learned_addr learned my addr 172.21.15.141:0/2487485519 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.558+0000 7ff77a800640 1 -- 172.21.15.141:0/2487485519 >> v2:172.21.15.188:3300/0 conn(0x7ff77c0cf9a0 msgr2=0x7ff77c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.558+0000 7ff77a800640 1 --2- 172.21.15.141:0/2487485519 >> v2:172.21.15.188:3300/0 conn(0x7ff77c0cf9a0 0x7ff77c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.558+0000 7ff77a800640 1 -- 172.21.15.141:0/2487485519 >> v1:172.21.15.141:6789/0 conn(0x7ff77c13e360 legacy=0x7ff77c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.558+0000 7ff77a800640 1 -- 172.21.15.141:0/2487485519 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c06cd10 con 0x7ff77c13df80 2024-01-16T14:59:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.559+0000 7ff77a800640 1 --2- 172.21.15.141:0/2487485519 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c13df80 0x7ff77c15bee0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0ccb10 tx=0x7ff76c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a56063ba247071 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.560+0000 7ff779400640 1 -- 172.21.15.141:0/2487485519 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c030870 con 0x7ff77c13df80 2024-01-16T14:59:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.560+0000 7ff779400640 1 -- 172.21.15.141:0/2487485519 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c0309d0 con 0x7ff77c13df80 2024-01-16T14:59:38.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.560+0000 7ff779400640 1 -- 172.21.15.141:0/2487485519 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c030d60 con 0x7ff77c13df80 2024-01-16T14:59:38.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 -- 172.21.15.141:0/2487485519 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c13df80 msgr2=0x7ff77c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 --2- 172.21.15.141:0/2487485519 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c13df80 0x7ff77c15bee0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0ccb10 tx=0x7ff76c02b9b0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 -- 172.21.15.141:0/2487485519 shutdown_connections 2024-01-16T14:59:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 --2- 172.21.15.141:0/2487485519 >> v2:172.21.15.188:3300/0 conn(0x7ff77c0cf9a0 0x7ff77c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 --2- 172.21.15.141:0/2487485519 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c13df80 0x7ff77c15bee0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 -- 172.21.15.141:0/2487485519 >> 172.21.15.141:0/2487485519 conn(0x7ff77c06a800 msgr2=0x7ff77c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 -- 172.21.15.141:0/2487485519 shutdown_connections 2024-01-16T14:59:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.561+0000 7ff781400640 1 -- 172.21.15.141:0/2487485519 wait complete. 2024-01-16T14:59:38.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.563+0000 7ff781400640 1 Processor -- start 2024-01-16T14:59:38.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.564+0000 7ff781400640 1 -- start start 2024-01-16T14:59:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff781400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c0cf9a0 0x7ff77c1e6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff781400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77c13df80 0x7ff77c1e6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff781400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff77c13e360 0x7ff77c1e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff781400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c1e6ed0 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff781400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c1e71b0 con 0x7ff77c13df80 2024-01-16T14:59:38.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff781400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c1e7490 con 0x7ff77c13e360 2024-01-16T14:59:38.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff77a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c0cf9a0 0x7ff77c1e6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff77a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c0cf9a0 0x7ff77c1e6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33968/0 (socket says 172.21.15.141:33968) 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.565+0000 7ff77a800640 1 -- 172.21.15.141:0/3710456407 learned_addr learned my addr 172.21.15.141:0/3710456407 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff779e00640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77c13df80 0x7ff77c1e6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff77a800640 1 -- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff77c13e360 msgr2=0x7ff77c1e8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff77a800640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff77c13e360 0x7ff77c1e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff77a800640 1 -- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77c13df80 msgr2=0x7ff77c1e6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff77a800640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77c13df80 0x7ff77c1e6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff77a800640 1 -- 172.21.15.141:0/3710456407 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0094d0 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff77a800640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c0cf9a0 0x7ff77c1e6860 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff76c000c00 tx=0x7ff76c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.566+0000 7ff772a00640 1 -- 172.21.15.141:0/3710456407 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c039d40 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.567+0000 7ff772a00640 1 -- 172.21.15.141:0/3710456407 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c043400 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.567+0000 7ff772a00640 1 -- 172.21.15.141:0/3710456407 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c039ea0 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.567+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c1e7770 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.567+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c1e90c0 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.568+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c1e7c90 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.569+0000 7ff772a00640 1 -- 172.21.15.141:0/3710456407 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7ff76c0437b0 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.569+0000 7ff772a00640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff75406b470 0x7ff75406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.570+0000 7ff772a00640 1 -- 172.21.15.141:0/3710456407 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff76c03e070 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.575+0000 7ff772a00640 1 -- 172.21.15.141:0/3710456407 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff77c1e7c90 con 0x7ff77c0cf9a0 2024-01-16T14:59:38.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.577+0000 7ff779e00640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff75406b470 0x7ff75406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.578+0000 7ff779e00640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff75406b470 0x7ff75406d930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0cc010 tx=0x7ff764009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.904+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff77c15f9c0 con 0x7ff75406b470 2024-01-16T14:59:38.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.934+0000 7ff772a00640 1 -- 172.21.15.141:0/3710456407 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7ff764001360 con 0x7ff75406b470 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:38.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 33.33333333333333, 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "1/3", 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:38.916 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:38.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff75406b470 msgr2=0x7ff75406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff75406b470 0x7ff75406d930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0cc010 tx=0x7ff764009040 comp rx=0 tx=0).stop 2024-01-16T14:59:38.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c0cf9a0 msgr2=0x7ff77c1e6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c0cf9a0 0x7ff77c1e6860 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff76c000c00 tx=0x7ff76c004730 comp rx=0 tx=0).stop 2024-01-16T14:59:38.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 shutdown_connections 2024-01-16T14:59:38.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff75406b470 0x7ff75406d930 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff77c13e360 0x7ff77c1e8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77c13df80 0x7ff77c1e6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 --2- 172.21.15.141:0/3710456407 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff77c0cf9a0 0x7ff77c1e6860 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 >> 172.21.15.141:0/3710456407 conn(0x7ff77c06a800 msgr2=0x7ff77c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.939+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 shutdown_connections 2024-01-16T14:59:38.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:38.940+0000 7ff781400640 1 -- 172.21.15.141:0/3710456407 wait complete. 2024-01-16T14:59:38.956 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:39.958 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:40.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.188+0000 7f27e7c00640 1 Processor -- start 2024-01-16T14:59:40.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.189+0000 7f27e7c00640 1 -- start start 2024-01-16T14:59:40.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.190+0000 7f27e7c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27e013e360 0x7f27e00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.190+0000 7f27e7c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f27e015f9c0 0x7f27e00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.190+0000 7f27e7c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e015c5f0 con 0x7f27e013df80 2024-01-16T14:59:40.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.190+0000 7f27e7c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e006c470 con 0x7f27e013e360 2024-01-16T14:59:40.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.190+0000 7f27e7c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e006c750 con 0x7f27e015f9c0 2024-01-16T14:59:40.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27dfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27e013e360 0x7f27e00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27dfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27e013e360 0x7f27e00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33980/0 (socket says 172.21.15.141:33980) 2024-01-16T14:59:40.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27dfe00640 1 -- 172.21.15.141:0/665786045 learned_addr learned my addr 172.21.15.141:0/665786045 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:40.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27dfe00640 1 -- 172.21.15.141:0/665786045 >> v2:172.21.15.188:3300/0 conn(0x7f27e015f9c0 msgr2=0x7f27e00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27e5c00640 1 --2- 172.21.15.141:0/665786045 >> v2:172.21.15.188:3300/0 conn(0x7f27e015f9c0 0x7f27e00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27dfe00640 1 --2- 172.21.15.141:0/665786045 >> v2:172.21.15.188:3300/0 conn(0x7f27e015f9c0 0x7f27e00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27dfe00640 1 -- 172.21.15.141:0/665786045 >> v1:172.21.15.188:6789/0 conn(0x7f27e013df80 legacy=0x7f27e015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27dfe00640 1 -- 172.21.15.141:0/665786045 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e006ca30 con 0x7f27e013e360 2024-01-16T14:59:40.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.191+0000 7f27e5c00640 1 --2- 172.21.15.141:0/665786045 >> v2:172.21.15.188:3300/0 conn(0x7f27e015f9c0 0x7f27e00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:40.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.193+0000 7f27dfe00640 1 --2- 172.21.15.141:0/665786045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013e360 0x7f27e00cf290 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f27e006ded0 tx=0x7f27c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1c7d90947e91f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.193+0000 7f27df400640 1 -- 172.21.15.141:0/665786045 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8031640 con 0x7f27e013e360 2024-01-16T14:59:40.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.193+0000 7f27df400640 1 -- 172.21.15.141:0/665786045 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c80317a0 con 0x7f27e013e360 2024-01-16T14:59:40.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.193+0000 7f27df400640 1 -- 172.21.15.141:0/665786045 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8031b30 con 0x7f27e013e360 2024-01-16T14:59:40.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 -- 172.21.15.141:0/665786045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013e360 msgr2=0x7f27e00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 --2- 172.21.15.141:0/665786045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013e360 0x7f27e00cf290 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f27e006ded0 tx=0x7f27c802b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 -- 172.21.15.141:0/665786045 shutdown_connections 2024-01-16T14:59:40.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 --2- 172.21.15.141:0/665786045 >> v2:172.21.15.188:3300/0 conn(0x7f27e015f9c0 0x7f27e00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 --2- 172.21.15.141:0/665786045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013e360 0x7f27e00cf290 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 -- 172.21.15.141:0/665786045 >> 172.21.15.141:0/665786045 conn(0x7f27e006a800 msgr2=0x7f27e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 -- 172.21.15.141:0/665786045 shutdown_connections 2024-01-16T14:59:40.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.194+0000 7f27e7c00640 1 -- 172.21.15.141:0/665786045 wait complete. 2024-01-16T14:59:40.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.196+0000 7f27e7c00640 1 Processor -- start 2024-01-16T14:59:40.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.196+0000 7f27e7c00640 1 -- start start 2024-01-16T14:59:40.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e7c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013df80 0x7f27e013da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e7c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e013e360 0x7f27e014b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e7c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e015f9c0 0x7f27e014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e7c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e014af70 con 0x7f27e013df80 2024-01-16T14:59:40.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e7c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e0070990 con 0x7f27e015f9c0 2024-01-16T14:59:40.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e7c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e0070c70 con 0x7f27e013e360 2024-01-16T14:59:40.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013df80 0x7f27e013da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013df80 0x7f27e013da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33990/0 (socket says 172.21.15.141:33990) 2024-01-16T14:59:40.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 -- 172.21.15.141:0/3246028828 learned_addr learned my addr 172.21.15.141:0/3246028828 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:40.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 -- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e013e360 msgr2=0x7f27e014b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e013e360 0x7f27e014b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 -- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e015f9c0 msgr2=0x7f27e014bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e015f9c0 0x7f27e014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.198+0000 7f27e5200640 1 -- 172.21.15.141:0/3246028828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80094d0 con 0x7f27e013df80 2024-01-16T14:59:40.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.199+0000 7f27e5200640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013df80 0x7f27e013da20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f27d000b3f0 tx=0x7f27d000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.199+0000 7f27dcc00640 1 -- 172.21.15.141:0/3246028828 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0011830 con 0x7f27e013df80 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.199+0000 7f27dcc00640 1 -- 172.21.15.141:0/3246028828 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d0011990 con 0x7f27e013df80 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.199+0000 7f27dcc00640 1 -- 172.21.15.141:0/3246028828 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0011d40 con 0x7f27e013df80 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.199+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e0070f50 con 0x7f27e013df80 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.199+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e014dbf0 con 0x7f27e013df80 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.200+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e00675d0 con 0x7f27e013df80 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.201+0000 7f27dcc00640 1 -- 172.21.15.141:0/3246028828 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f27d001a400 con 0x7f27e013df80 2024-01-16T14:59:40.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.203+0000 7f27dcc00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27b806b470 0x7f27b806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.203+0000 7f27dcc00640 1 -- 172.21.15.141:0/3246028828 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27d0089df0 con 0x7f27e013df80 2024-01-16T14:59:40.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.204+0000 7f27dfe00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27b806b470 0x7f27b806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.205+0000 7f27dfe00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27b806b470 0x7f27b806d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27c8036000 tx=0x7f27c8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.206+0000 7f27dcc00640 1 -- 172.21.15.141:0/3246028828 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27d0054940 con 0x7f27e013df80 2024-01-16T14:59:40.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.436+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e006daf0 con 0x7f27b806b470 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.450+0000 7f27dcc00640 1 -- 172.21.15.141:0/3246028828 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f27e006daf0 con 0x7f27b806b470 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:40.432 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:40.432 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T14:59:40.432 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T14:59:40.432 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T14:59:40.432 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:40.432 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:40.435 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:40.436 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:40.436 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.453+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27b806b470 msgr2=0x7f27b806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.453+0000 7f27e7c00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27b806b470 0x7f27b806d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27c8036000 tx=0x7f27c8030000 comp rx=0 tx=0).stop 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.453+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013df80 msgr2=0x7f27e013da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.453+0000 7f27e7c00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013df80 0x7f27e013da20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f27d000b3f0 tx=0x7f27d000b8b0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 shutdown_connections 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27b806b470 0x7f27b806d930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e015f9c0 0x7f27e014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e013e360 0x7f27e014b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 --2- 172.21.15.141:0/3246028828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e013df80 0x7f27e013da20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 >> 172.21.15.141:0/3246028828 conn(0x7f27e006a800 msgr2=0x7f27e015d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 shutdown_connections 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:40.454+0000 7f27e7c00640 1 -- 172.21.15.141:0/3246028828 wait complete. 2024-01-16T14:59:40.453 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:41.454 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:41.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.694+0000 7f79d7e00640 1 Processor -- start 2024-01-16T14:59:41.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.694+0000 7f79d7e00640 1 -- start start 2024-01-16T14:59:41.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d815d4b0 0x7f79d80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f79d80b41e0 0x7f79d80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d80d0320 con 0x7f79d80cf9a0 2024-01-16T14:59:41.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d80b2730 con 0x7f79d80b41e0 2024-01-16T14:59:41.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d80d1ec0 con 0x7f79d815d4b0 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f79d80b41e0 0x7f79d80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f79d80b41e0 0x7f79d80cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33996/0 (socket says 172.21.15.141:33996) 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.696+0000 7f79d7400640 1 -- 172.21.15.141:0/1746969563 learned_addr learned my addr 172.21.15.141:0/1746969563 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.697+0000 7f79d6a00640 1 --2- 172.21.15.141:0/1746969563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d815d4b0 0x7f79d80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.697+0000 7f79d7400640 1 -- 172.21.15.141:0/1746969563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d815d4b0 msgr2=0x7f79d80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.697+0000 7f79d7400640 1 --2- 172.21.15.141:0/1746969563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d815d4b0 0x7f79d80d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.697+0000 7f79d7400640 1 -- 172.21.15.141:0/1746969563 >> v1:172.21.15.188:6789/0 conn(0x7f79d80cf9a0 legacy=0x7f79d80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.697+0000 7f79d7400640 1 -- 172.21.15.141:0/1746969563 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d80b29d0 con 0x7f79d80b41e0 2024-01-16T14:59:41.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.698+0000 7f79d7400640 1 --2- 172.21.15.141:0/1746969563 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d80b41e0 0x7f79d80cfde0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009e80 tx=0x7f79d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbd3bc1044964b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.698+0000 7f79d6000640 1 -- 172.21.15.141:0/1746969563 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d002ca50 con 0x7f79d80b41e0 2024-01-16T14:59:41.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.698+0000 7f79d6000640 1 -- 172.21.15.141:0/1746969563 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d002cbb0 con 0x7f79d80b41e0 2024-01-16T14:59:41.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.698+0000 7f79d6000640 1 -- 172.21.15.141:0/1746969563 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d00343f0 con 0x7f79d80b41e0 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 -- 172.21.15.141:0/1746969563 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d80b41e0 msgr2=0x7f79d80cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 --2- 172.21.15.141:0/1746969563 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d80b41e0 0x7f79d80cfde0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009e80 tx=0x7f79d002beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 -- 172.21.15.141:0/1746969563 shutdown_connections 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 --2- 172.21.15.141:0/1746969563 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d80b41e0 0x7f79d80cfde0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 --2- 172.21.15.141:0/1746969563 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d815d4b0 0x7f79d80d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 -- 172.21.15.141:0/1746969563 >> 172.21.15.141:0/1746969563 conn(0x7f79d80d3000 msgr2=0x7f79d80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 -- 172.21.15.141:0/1746969563 shutdown_connections 2024-01-16T14:59:41.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.699+0000 7f79d7e00640 1 -- 172.21.15.141:0/1746969563 wait complete. 2024-01-16T14:59:41.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.701+0000 7f79d7e00640 1 Processor -- start 2024-01-16T14:59:41.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.701+0000 7f79d7e00640 1 -- start start 2024-01-16T14:59:41.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79d80b41e0 0x7f79d8160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d80cf9a0 0x7f79d8156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d815d4b0 0x7f79d80b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d80ca0d0 con 0x7f79d815d4b0 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d80caf30 con 0x7f79d80b41e0 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d80ca320 con 0x7f79d80cf9a0 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d815d4b0 0x7f79d80b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d815d4b0 0x7f79d80b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34008/0 (socket says 172.21.15.141:34008) 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.702+0000 7f79d7400640 1 -- 172.21.15.141:0/4256096694 learned_addr learned my addr 172.21.15.141:0/4256096694 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79d6a00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79d80b41e0 0x7f79d8160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79cde00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d80cf9a0 0x7f79d8156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79d7400640 1 -- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d80cf9a0 msgr2=0x7f79d8156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79d7400640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d80cf9a0 0x7f79d8156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79d7400640 1 -- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79d80b41e0 msgr2=0x7f79d8160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79d7400640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79d80b41e0 0x7f79d8160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79d7400640 1 -- 172.21.15.141:0/4256096694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d0009b30 con 0x7f79d815d4b0 2024-01-16T14:59:41.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79cde00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d80cf9a0 0x7f79d8156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.703+0000 7f79d7400640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d815d4b0 0x7f79d80b04d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f79d002c4c0 tx=0x7f79d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.704+0000 7f79cf400640 1 -- 172.21.15.141:0/4256096694 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0009510 con 0x7f79d815d4b0 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.704+0000 7f79cf400640 1 -- 172.21.15.141:0/4256096694 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d003e450 con 0x7f79d815d4b0 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.704+0000 7f79cf400640 1 -- 172.21.15.141:0/4256096694 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d003e800 con 0x7f79d815d4b0 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.704+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d815da10 con 0x7f79d815d4b0 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.704+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d80cabc0 con 0x7f79d815d4b0 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.705+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d80ca710 con 0x7f79d815d4b0 2024-01-16T14:59:41.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.706+0000 7f79cf400640 1 -- 172.21.15.141:0/4256096694 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f79d003e960 con 0x7f79d815d4b0 2024-01-16T14:59:41.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.707+0000 7f79cf400640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79b006b470 0x7f79b006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.707+0000 7f79cf400640 1 -- 172.21.15.141:0/4256096694 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79d00acc40 con 0x7f79d815d4b0 2024-01-16T14:59:41.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.707+0000 7f79d6a00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79b006b470 0x7f79b006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.708+0000 7f79d6a00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79b006b470 0x7f79b006d930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f79c8006fd0 tx=0x7f79c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.711+0000 7f79cf400640 1 -- 172.21.15.141:0/4256096694 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79d0077830 con 0x7f79d815d4b0 2024-01-16T14:59:41.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.956+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f79d815d890 con 0x7f79b006b470 2024-01-16T14:59:41.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.980+0000 7f79cf400640 1 -- 172.21.15.141:0/4256096694 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f79c800f860 con 0x7f79b006b470 2024-01-16T14:59:41.961 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:41.961 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:41.961 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:41.962 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79b006b470 msgr2=0x7f79b006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79b006b470 0x7f79b006d930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f79c8006fd0 tx=0x7f79c8008040 comp rx=0 tx=0).stop 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d815d4b0 msgr2=0x7f79d80b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d815d4b0 0x7f79d80b04d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f79d002c4c0 tx=0x7f79d0004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 shutdown_connections 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79b006b470 0x7f79b006d930 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79d815d4b0 0x7f79d80b04d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79d80cf9a0 0x7f79d8156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 --2- 172.21.15.141:0/4256096694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79d80b41e0 0x7f79d8160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 >> 172.21.15.141:0/4256096694 conn(0x7f79d80d3000 msgr2=0x7f79d80d33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 shutdown_connections 2024-01-16T14:59:41.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:41.984+0000 7f79d7e00640 1 -- 172.21.15.141:0/4256096694 wait complete. 2024-01-16T14:59:41.988 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:42.989 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:43.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.305+0000 7f4b1c400640 1 Processor -- start 2024-01-16T14:59:43.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.305+0000 7f4b1c400640 1 -- start start 2024-01-16T14:59:43.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.307+0000 7f4b1c400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b140cf9a0 0x7f4b14166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.307+0000 7f4b1c400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1415c5f0 con 0x7f4b1413df80 2024-01-16T14:59:43.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.307+0000 7f4b1c400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1406c470 con 0x7f4b1413e360 2024-01-16T14:59:43.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.307+0000 7f4b1c400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1406c750 con 0x7f4b140cf9a0 2024-01-16T14:59:43.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.307+0000 7f4b19a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4b1413df80 0x7f4b1415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49612/0 (socket says 172.21.15.141:49612) 2024-01-16T14:59:43.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.307+0000 7f4b19a00640 1 -- 172.21.15.141:0/3820029089 learned_addr learned my addr 172.21.15.141:0/3820029089 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:43.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.309+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171515317 0 0) 0x7f4b1406c470 con 0x7f4b1413e360 2024-01-16T14:59:43.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.309+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af4003520 con 0x7f4b1413e360 2024-01-16T14:59:43.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000405515 0 0) 0x7f4af4003520 con 0x7f4b1413e360 2024-01-16T14:59:43.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 >> v2:172.21.15.141:3300/0 conn(0x7f4b140cf9a0 msgr2=0x7f4b14166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 --2- 172.21.15.141:0/3820029089 >> v2:172.21.15.141:3300/0 conn(0x7f4b140cf9a0 0x7f4b14166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 >> v1:172.21.15.141:6789/0 conn(0x7f4b1413df80 legacy=0x7f4b1415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1406ca30 con 0x7f4b1413e360 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4afc001670 con 0x7f4b1413e360 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 >> v1:172.21.15.188:6789/0 conn(0x7f4b1413e360 legacy=0x7f4b140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4003970 0x7f4af4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4af4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4af40052a0 0x7f4af400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1406c750 con 0x7f4af40052a0 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af400c1a0 con 0x7f4af4003970 2024-01-16T14:59:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.310+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af400c360 con 0x7f4af40047c0 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.311+0000 7f4b19000640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4af4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.312+0000 7f4b19a00640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4003970 0x7f4af4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.312+0000 7f4b1a400640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4af40052a0 0x7f4af400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.313+0000 7f4b19000640 1 -- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4003970 msgr2=0x7f4af4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.313+0000 7f4b19000640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4003970 0x7f4af4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.313+0000 7f4b19000640 1 -- 172.21.15.141:0/3820029089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4af40052a0 msgr2=0x7f4af400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.313+0000 7f4b19000640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4af40052a0 0x7f4af400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.313+0000 7f4b19000640 1 -- 172.21.15.141:0/3820029089 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1406ca30 con 0x7f4af40047c0 2024-01-16T14:59:43.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.315+0000 7f4b1a400640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4af40052a0 0x7f4af400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:43.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.315+0000 7f4b1a400640 1 -- 172.21.15.141:0/3820029089 reap_dead start 2024-01-16T14:59:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.316+0000 7f4b19000640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4af4004c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b140cc790 tx=0x7f4af40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4f82bb88e4194 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.317+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc030e20 con 0x7f4af40047c0 2024-01-16T14:59:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.317+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc007050 con 0x7f4af40047c0 2024-01-16T14:59:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.317+0000 7f4b13e00640 1 -- 172.21.15.141:0/3820029089 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc007440 con 0x7f4af40047c0 2024-01-16T14:59:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.318+0000 7f4b1c400640 1 -- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 msgr2=0x7f4af4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.318+0000 7f4b1c400640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4af4004c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b140cc790 tx=0x7f4af40057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.318+0000 7f4b1c400640 1 -- 172.21.15.141:0/3820029089 shutdown_connections 2024-01-16T14:59:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.318+0000 7f4b1c400640 1 --2- 172.21.15.141:0/3820029089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4af4004c40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.318+0000 7f4b1c400640 1 -- 172.21.15.141:0/3820029089 >> 172.21.15.141:0/3820029089 conn(0x7f4b1406a800 msgr2=0x7f4b1406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.318+0000 7f4b1c400640 1 -- 172.21.15.141:0/3820029089 shutdown_connections 2024-01-16T14:59:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.318+0000 7f4b1c400640 1 -- 172.21.15.141:0/3820029089 wait complete. 2024-01-16T14:59:43.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.322+0000 7f4b1c400640 1 Processor -- start 2024-01-16T14:59:43.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.322+0000 7f4b1c400640 1 -- start start 2024-01-16T14:59:43.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.324+0000 7f4b1c400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4b141eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b1c400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b141f1db0 0x7f4b14154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b1c400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1413bc40 0x7f4b1413c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b1c400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1413c600 con 0x7f4b141f1db0 2024-01-16T14:59:43.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b1c400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b14152010 con 0x7f4b1413bc40 2024-01-16T14:59:43.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b1c400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140b29f0 con 0x7f4af40047c0 2024-01-16T14:59:43.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b19000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b141f1db0 0x7f4b14154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b19000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b141f1db0 0x7f4b14154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56150/0 (socket says 172.21.15.141:56150) 2024-01-16T14:59:43.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b19000640 1 -- 172.21.15.141:0/2831494999 learned_addr learned my addr 172.21.15.141:0/2831494999 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:43.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.325+0000 7f4b19a00640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4b141eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.326+0000 7f4b19000640 1 -- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 msgr2=0x7f4b141eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.326+0000 7f4b19000640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4b141eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.326+0000 7f4b19000640 1 -- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1413bc40 msgr2=0x7f4b1413c0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:59:43.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.326+0000 7f4b19000640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1413bc40 0x7f4b1413c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.326+0000 7f4b19000640 1 -- 172.21.15.141:0/2831494999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b140b2cd0 con 0x7f4b141f1db0 2024-01-16T14:59:43.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.327+0000 7f4b19000640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b141f1db0 0x7f4b14154aa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00c220 tx=0x7f4afc001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.327+0000 7f4b11600640 1 -- 172.21.15.141:0/2831494999 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc0067d0 con 0x7f4b141f1db0 2024-01-16T14:59:43.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.327+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b140b2fb0 con 0x7f4b141f1db0 2024-01-16T14:59:43.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.327+0000 7f4b11600640 1 -- 172.21.15.141:0/2831494999 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc006930 con 0x7f4b141f1db0 2024-01-16T14:59:43.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.327+0000 7f4b11600640 1 -- 172.21.15.141:0/2831494999 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc006cc0 con 0x7f4b141f1db0 2024-01-16T14:59:43.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.327+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b14154fe0 con 0x7f4b141f1db0 2024-01-16T14:59:43.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.329+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b140b2850 con 0x7f4b141f1db0 2024-01-16T14:59:43.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.332+0000 7f4b11600640 1 -- 172.21.15.141:0/2831494999 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f4afc0035f0 con 0x7f4b141f1db0 2024-01-16T14:59:43.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.333+0000 7f4b11600640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4aec06b470 0x7f4aec06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.333+0000 7f4b11600640 1 -- 172.21.15.141:0/2831494999 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4afc0b0eb0 con 0x7f4b141f1db0 2024-01-16T14:59:43.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.334+0000 7f4b19a00640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4aec06b470 0x7f4aec06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.342+0000 7f4b11600640 1 -- 172.21.15.141:0/2831494999 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4afc07bca0 con 0x7f4b141f1db0 2024-01-16T14:59:43.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.347+0000 7f4b19a00640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4aec06b470 0x7f4aec06d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4b04007250 tx=0x7f4b04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.676+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b1406cff0 con 0x7f4aec06b470 2024-01-16T14:59:43.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.690+0000 7f4b11600640 1 -- 172.21.15.141:0/2831494999 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f4b040077d0 con 0x7f4aec06b470 2024-01-16T14:59:43.671 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:43.671 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:43.671 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T14:59:43.671 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T14:59:43.671 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T14:59:43.671 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:43.672 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4aec06b470 msgr2=0x7f4aec06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4aec06b470 0x7f4aec06d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4b04007250 tx=0x7f4b04009040 comp rx=0 tx=0).stop 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b141f1db0 msgr2=0x7f4b14154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b141f1db0 0x7f4b14154aa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00c220 tx=0x7f4afc001930 comp rx=0 tx=0).stop 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 shutdown_connections 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4aec06b470 0x7f4aec06d930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1413bc40 0x7f4b1413c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b141f1db0 0x7f4b14154aa0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 --2- 172.21.15.141:0/2831494999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4af40047c0 0x7f4b141eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 >> 172.21.15.141:0/2831494999 conn(0x7f4b1406a800 msgr2=0x7f4b1415d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 shutdown_connections 2024-01-16T14:59:43.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:43.693+0000 7f4b1c400640 1 -- 172.21.15.141:0/2831494999 wait complete. 2024-01-16T14:59:43.695 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T14:59:44.696 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T14:59:44.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.941+0000 7f0340c00640 1 Processor -- start 2024-01-16T14:59:44.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.941+0000 7f0340c00640 1 -- start start 2024-01-16T14:59:44.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.943+0000 7f0340c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c13e360 0x7f033c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.943+0000 7f0340c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c15c5f0 con 0x7f033c13df80 2024-01-16T14:59:44.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.943+0000 7f0340c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c06c470 con 0x7f033c15f9c0 2024-01-16T14:59:44.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.943+0000 7f0340c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c06c750 con 0x7f033c13e360 2024-01-16T14:59:44.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.944+0000 7f0339e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f033c13df80 0x7f033c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49620/0 (socket says 172.21.15.141:49620) 2024-01-16T14:59:44.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.944+0000 7f0339e00640 1 -- 172.21.15.141:0/3617452281 learned_addr learned my addr 172.21.15.141:0/3617452281 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:44.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.944+0000 7f0339400640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c13e360 0x7f033c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.944+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306113716 0 0) 0x7f033c15c5f0 con 0x7f033c13df80 2024-01-16T14:59:44.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.944+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c003520 con 0x7f033c13df80 2024-01-16T14:59:44.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3386866718 0 0) 0x7f031c003520 con 0x7f033c13df80 2024-01-16T14:59:44.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c13e360 msgr2=0x7f033c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c13e360 0x7f033c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 >> v1:172.21.15.188:6789/0 conn(0x7f033c15f9c0 legacy=0x7f033c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c06ca30 con 0x7f033c13df80 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f032c001670 con 0x7f033c13df80 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 >> v1:172.21.15.141:6789/0 conn(0x7f033c13df80 legacy=0x7f033c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f031c0039a0 0x7f031c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f031c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c13e360 0x7f031c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c06c750 con 0x7f031c0047c0 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c00ba90 con 0x7f031c0039a0 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031c00bc50 con 0x7f033c13e360 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0339400640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f031c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0339e00640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f031c0039a0 0x7f031c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0339400640 1 -- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c13e360 msgr2=0x7f031c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0339400640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c13e360 0x7f031c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0339400640 1 -- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f031c0039a0 msgr2=0x7f031c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0339400640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f031c0039a0 0x7f031c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.945+0000 7f0339400640 1 -- 172.21.15.141:0/3617452281 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c003520 con 0x7f031c0047c0 2024-01-16T14:59:44.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.946+0000 7f0339e00640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f031c0039a0 0x7f031c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0339400640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f031c004c40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0324009f90 tx=0x7f031c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8227b24befb87d5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032402cbf0 con 0x7f031c0047c0 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032400f040 con 0x7f031c0047c0 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0338a00640 1 -- 172.21.15.141:0/3617452281 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324034440 con 0x7f031c0047c0 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0340c00640 1 -- 172.21.15.141:0/3617452281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 msgr2=0x7f031c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0340c00640 1 --2- 172.21.15.141:0/3617452281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f031c004c40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0324009f90 tx=0x7f031c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f033a800640 1 -- 172.21.15.141:0/3617452281 reap_dead start 2024-01-16T14:59:44.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0340c00640 1 -- 172.21.15.141:0/3617452281 shutdown_connections 2024-01-16T14:59:44.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.947+0000 7f0340c00640 1 -- 172.21.15.141:0/3617452281 >> 172.21.15.141:0/3617452281 conn(0x7f033c06a800 msgr2=0x7f033c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:44.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.948+0000 7f0340c00640 1 -- 172.21.15.141:0/3617452281 shutdown_connections 2024-01-16T14:59:44.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.948+0000 7f0340c00640 1 -- 172.21.15.141:0/3617452281 wait complete. 2024-01-16T14:59:44.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.949+0000 7f0340c00640 1 Processor -- start 2024-01-16T14:59:44.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.950+0000 7f0340c00640 1 -- start start 2024-01-16T14:59:44.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.950+0000 7f0340c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f033c13d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.950+0000 7f0340c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f033c070cd0 0x7f033c071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.950+0000 7f0340c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c14b660 0x7f033c14bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.950+0000 7f0340c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c14c050 con 0x7f031c0047c0 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.950+0000 7f0340c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c071d50 con 0x7f033c070cd0 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.950+0000 7f0340c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c071790 con 0x7f033c14b660 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f033c13d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f033c13d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56170/0 (socket says 172.21.15.141:56170) 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 -- 172.21.15.141:0/617920352 learned_addr learned my addr 172.21.15.141:0/617920352 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f033a800640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c14b660 0x7f033c14bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339400640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f033c070cd0 0x7f033c071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 -- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c14b660 msgr2=0x7f033c14bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c14b660 0x7f033c14bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 -- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f033c070cd0 msgr2=0x7f033c071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f033c070cd0 0x7f033c071150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.951+0000 7f0339e00640 1 -- 172.21.15.141:0/617920352 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c071a70 con 0x7f031c0047c0 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.952+0000 7f0339400640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f033c070cd0 0x7f033c071150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.952+0000 7f0339e00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f033c13d730 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f032c00d840 tx=0x7f032c00dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.952+0000 7f0332000640 1 -- 172.21.15.141:0/617920352 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c0074e0 con 0x7f031c0047c0 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.952+0000 7f0332000640 1 -- 172.21.15.141:0/617920352 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c007640 con 0x7f031c0047c0 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.952+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c0b2230 con 0x7f031c0047c0 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.952+0000 7f0332000640 1 -- 172.21.15.141:0/617920352 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c0049e0 con 0x7f031c0047c0 2024-01-16T14:59:44.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.952+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c14d980 con 0x7f031c0047c0 2024-01-16T14:59:44.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.953+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c0675d0 con 0x7f031c0047c0 2024-01-16T14:59:44.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.954+0000 7f0332000640 1 -- 172.21.15.141:0/617920352 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f032c004b40 con 0x7f031c0047c0 2024-01-16T14:59:44.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.955+0000 7f0332000640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031406b4a0 0x7f031406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.955+0000 7f0332000640 1 -- 172.21.15.141:0/617920352 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f032c0898e0 con 0x7f031c0047c0 2024-01-16T14:59:44.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.956+0000 7f0339400640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031406b4a0 0x7f031406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.957+0000 7f0339400640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031406b4a0 0x7f031406d960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0324036000 tx=0x7f032402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:44.959+0000 7f0332000640 1 -- 172.21.15.141:0/617920352 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f032c054500 con 0x7f031c0047c0 2024-01-16T14:59:45.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:45.193+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f033c06cff0 con 0x7f031406b4a0 2024-01-16T14:59:46.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.453+0000 7f0332000640 1 -- 172.21.15.141:0/617920352 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f03240450c0 con 0x7f031406b4a0 2024-01-16T14:59:46.434 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T14:59:46.434 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T14:59:46.434 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T14:59:46.434 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:46.434 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.456+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031406b4a0 msgr2=0x7f031406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031406b4a0 0x7f031406d960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0324036000 tx=0x7f032402e000 comp rx=0 tx=0).stop 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 msgr2=0x7f033c13d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f033c13d730 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f032c00d840 tx=0x7f032c00dd00 comp rx=0 tx=0).stop 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 shutdown_connections 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031406b4a0 0x7f031406d960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f033c14b660 0x7f033c14bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f033c070cd0 0x7f033c071150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 --2- 172.21.15.141:0/617920352 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f031c0047c0 0x7f033c13d730 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 >> 172.21.15.141:0/617920352 conn(0x7f033c06a800 msgr2=0x7f033c15c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 shutdown_connections 2024-01-16T14:59:46.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.457+0000 7f0340c00640 1 -- 172.21.15.141:0/617920352 wait complete. 2024-01-16T14:59:46.458 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T14:59:46.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.683+0000 7f0ae2800640 1 Processor -- start 2024-01-16T14:59:46.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.683+0000 7f0ae2800640 1 -- start start 2024-01-16T14:59:46.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.685+0000 7f0ae2800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0adc15e3b0 0x7f0adc164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.685+0000 7f0ae2800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0adc0b1ee0 con 0x7f0adc0cf9a0 2024-01-16T14:59:46.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.685+0000 7f0ae2800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0adc0b2780 con 0x7f0adc143ae0 2024-01-16T14:59:46.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.685+0000 7f0ae2800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0adc15e990 con 0x7f0adc15e3b0 2024-01-16T14:59:46.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.685+0000 7f0ad3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0adc0cf9a0 0x7f0adc15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49628/0 (socket says 172.21.15.141:49628) 2024-01-16T14:59:46.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.685+0000 7f0ad3e00640 1 -- 172.21.15.141:0/2926274925 learned_addr learned my addr 172.21.15.141:0/2926274925 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adbe00640 1 --2- 172.21.15.141:0/2926274925 >> v2:172.21.15.188:3300/0 conn(0x7f0adc15e3b0 0x7f0adc164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249545459 0 0) 0x7f0adc0b1ee0 con 0x7f0adc0cf9a0 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0abc003520 con 0x7f0adc0cf9a0 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131626501 0 0) 0x7f0abc003520 con 0x7f0adc0cf9a0 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 >> v2:172.21.15.188:3300/0 conn(0x7f0adc15e3b0 msgr2=0x7f0adc164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 --2- 172.21.15.141:0/2926274925 >> v2:172.21.15.188:3300/0 conn(0x7f0adc15e3b0 0x7f0adc164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 >> v1:172.21.15.188:6789/0 conn(0x7f0adc143ae0 legacy=0x7f0adc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc0b2130 con 0x7f0adc0cf9a0 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0ac4001670 con 0x7f0adc0cf9a0 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 >> v1:172.21.15.141:6789/0 conn(0x7f0adc0cf9a0 legacy=0x7f0adc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0abc0039c0 0x7f0abc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0047c0 0x7f0abc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0abc005180 0x7f0abc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc15e990 con 0x7f0abc0039c0 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc00c080 con 0x7f0abc0047c0 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc00e450 con 0x7f0abc005180 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.686+0000 7f0adb400640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0abc0039c0 0x7f0abc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.687+0000 7f0adb400640 1 -- 172.21.15.141:0/2926274925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0abc005180 msgr2=0x7f0abc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.687+0000 7f0adb400640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0abc005180 0x7f0abc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.687+0000 7f0adb400640 1 -- 172.21.15.141:0/2926274925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0047c0 msgr2=0x7f0abc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.687+0000 7f0adb400640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0047c0 0x7f0abc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.687+0000 7f0adb400640 1 -- 172.21.15.141:0/2926274925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc003520 con 0x7f0abc0039c0 2024-01-16T14:59:46.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.687+0000 7f0adbe00640 1 -- 172.21.15.141:0/2926274925 reap_dead start 2024-01-16T14:59:46.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.688+0000 7f0adb400640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0abc0039c0 0x7f0abc003de0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0acc001750 tx=0x7f0abc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf524925c974cfe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:46.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.688+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0052d0 con 0x7f0abc0039c0 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.688+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc02e050 con 0x7f0abc0039c0 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.688+0000 7f0adaa00640 1 -- 172.21.15.141:0/2926274925 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc003830 con 0x7f0abc0039c0 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.689+0000 7f0ae2800640 1 -- 172.21.15.141:0/2926274925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0abc0039c0 msgr2=0x7f0abc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.689+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0abc0039c0 0x7f0abc003de0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0acc001750 tx=0x7f0abc0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.689+0000 7f0ae2800640 1 -- 172.21.15.141:0/2926274925 shutdown_connections 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.689+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2926274925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0abc0039c0 0x7f0abc003de0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.689+0000 7f0ae2800640 1 -- 172.21.15.141:0/2926274925 >> 172.21.15.141:0/2926274925 conn(0x7f0adc06a800 msgr2=0x7f0adc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.689+0000 7f0ae2800640 1 -- 172.21.15.141:0/2926274925 shutdown_connections 2024-01-16T14:59:46.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.689+0000 7f0ae2800640 1 -- 172.21.15.141:0/2926274925 wait complete. 2024-01-16T14:59:46.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.691+0000 7f0ae2800640 1 Processor -- start 2024-01-16T14:59:46.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.691+0000 7f0ae2800640 1 -- start start 2024-01-16T14:59:46.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ae2800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0039c0 0x7f0adc15c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ae2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0adc15cb80 0x7f0adc134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ae2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0adc1349c0 0x7f0adc141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ae2800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc15cfe0 con 0x7f0adc15cb80 2024-01-16T14:59:46.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ae2800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc132570 con 0x7f0abc0039c0 2024-01-16T14:59:46.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ae2800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc135170 con 0x7f0adc1349c0 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ad3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0adc15cb80 0x7f0adc134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ad3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0adc15cb80 0x7f0adc134480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56176/0 (socket says 172.21.15.141:56176) 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.692+0000 7f0ad3e00640 1 -- 172.21.15.141:0/2628977722 learned_addr learned my addr 172.21.15.141:0/2628977722 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0adb400640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0039c0 0x7f0adc15c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0adbe00640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0adc1349c0 0x7f0adc141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3e00640 1 -- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0adc1349c0 msgr2=0x7f0adc141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3e00640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0adc1349c0 0x7f0adc141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3e00640 1 -- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0039c0 msgr2=0x7f0adc15c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3e00640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0039c0 0x7f0adc15c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3e00640 1 -- 172.21.15.141:0/2628977722 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc132830 con 0x7f0adc15cb80 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3e00640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0adc15cb80 0x7f0adc134480 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400c700 tx=0x7f0ac400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3400640 1 -- 172.21.15.141:0/2628977722 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac40072d0 con 0x7f0adc15cb80 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3400640 1 -- 172.21.15.141:0/2628977722 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac4007450 con 0x7f0adc15cb80 2024-01-16T14:59:46.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ad3400640 1 -- 172.21.15.141:0/2628977722 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac40047e0 con 0x7f0adc15cb80 2024-01-16T14:59:46.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.693+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0adc132b10 con 0x7f0adc15cb80 2024-01-16T14:59:46.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.694+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0adc0caba0 con 0x7f0adc15cb80 2024-01-16T14:59:46.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.695+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0adc0cadc0 con 0x7f0adc15cb80 2024-01-16T14:59:46.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.696+0000 7f0ad3400640 1 -- 172.21.15.141:0/2628977722 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f0ac4004960 con 0x7f0adc15cb80 2024-01-16T14:59:46.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.697+0000 7f0ad3400640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ab406b4a0 0x7f0ab406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:46.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.697+0000 7f0ad3400640 1 -- 172.21.15.141:0/2628977722 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ac4089280 con 0x7f0adc15cb80 2024-01-16T14:59:46.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.698+0000 7f0adb400640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ab406b4a0 0x7f0ab406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:46.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.699+0000 7f0adb400640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ab406b4a0 0x7f0ab406d960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0acc033000 tx=0x7f0acc009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:46.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.701+0000 7f0ad3400640 1 -- 172.21.15.141:0/2628977722 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ac4053dd0 con 0x7f0adc15cb80 2024-01-16T14:59:46.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.914+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0adc1323f0 con 0x7f0ab406b4a0 2024-01-16T14:59:46.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.918+0000 7f0ad3400640 1 -- 172.21.15.141:0/2628977722 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0acc005490 con 0x7f0ab406b4a0 2024-01-16T14:59:46.899 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T14:59:46.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:59:46.899 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T14:59:46.899 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:46.899 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T14:59:46.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.920+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ab406b4a0 msgr2=0x7f0ab406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.920+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ab406b4a0 0x7f0ab406d960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0acc033000 tx=0x7f0acc009030 comp rx=0 tx=0).stop 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.920+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0adc15cb80 msgr2=0x7f0adc134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.920+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0adc15cb80 0x7f0adc134480 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400c700 tx=0x7f0ac400cbc0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 shutdown_connections 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ab406b4a0 0x7f0ab406d960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0adc1349c0 0x7f0adc141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0adc15cb80 0x7f0adc134480 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 --2- 172.21.15.141:0/2628977722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0abc0039c0 0x7f0adc15c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 >> 172.21.15.141:0/2628977722 conn(0x7f0adc06a800 msgr2=0x7f0adc070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 shutdown_connections 2024-01-16T14:59:46.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:46.921+0000 7f0ae2800640 1 -- 172.21.15.141:0/2628977722 wait complete. 2024-01-16T14:59:46.919 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T14:59:47.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.150+0000 7f2eda600640 1 Processor -- start 2024-01-16T14:59:47.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.150+0000 7f2eda600640 1 -- start start 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.151+0000 7f2eda600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed414b2c0 0x7f2ed40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.151+0000 7f2eda600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ed4071070 0x7f2ed40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.151+0000 7f2eda600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40cfbf0 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.151+0000 7f2eda600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed414b6a0 con 0x7f2ed4071070 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.151+0000 7f2eda600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4071690 con 0x7f2ed414b2c0 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ed4071070 0x7f2ed40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ed4071070 0x7f2ed40cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56178/0 (socket says 172.21.15.141:56178) 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3e00640 1 -- 172.21.15.141:0/3736487296 learned_addr learned my addr 172.21.15.141:0/3736487296 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3400640 1 --2- 172.21.15.141:0/3736487296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed414b2c0 0x7f2ed40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3e00640 1 -- 172.21.15.141:0/3736487296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed414b2c0 msgr2=0x7f2ed40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3e00640 1 --2- 172.21.15.141:0/3736487296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed414b2c0 0x7f2ed40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3e00640 1 -- 172.21.15.141:0/3736487296 >> v1:172.21.15.188:6789/0 conn(0x7f2ed40cf7d0 legacy=0x7f2ed4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:47.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.152+0000 7f2ed3e00640 1 -- 172.21.15.141:0/3736487296 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed4071970 con 0x7f2ed4071070 2024-01-16T14:59:47.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.153+0000 7f2ed3e00640 1 --2- 172.21.15.141:0/3736487296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed4071070 0x7f2ed40cc160 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009e50 tx=0x7f2ecc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89180ec2708423e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2ed2000640 1 -- 172.21.15.141:0/3736487296 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc02ca70 con 0x7f2ed4071070 2024-01-16T14:59:47.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2ed2000640 1 -- 172.21.15.141:0/3736487296 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc02cbd0 con 0x7f2ed4071070 2024-01-16T14:59:47.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2ed2000640 1 -- 172.21.15.141:0/3736487296 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc0343f0 con 0x7f2ed4071070 2024-01-16T14:59:47.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 -- 172.21.15.141:0/3736487296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed4071070 msgr2=0x7f2ed40cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 --2- 172.21.15.141:0/3736487296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed4071070 0x7f2ed40cc160 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009e50 tx=0x7f2ecc02bed0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 -- 172.21.15.141:0/3736487296 shutdown_connections 2024-01-16T14:59:47.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 --2- 172.21.15.141:0/3736487296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed4071070 0x7f2ed40cc160 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 --2- 172.21.15.141:0/3736487296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed414b2c0 0x7f2ed40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 -- 172.21.15.141:0/3736487296 >> 172.21.15.141:0/3736487296 conn(0x7f2ed406a800 msgr2=0x7f2ed406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:47.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 -- 172.21.15.141:0/3736487296 shutdown_connections 2024-01-16T14:59:47.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.154+0000 7f2eda600640 1 -- 172.21.15.141:0/3736487296 wait complete. 2024-01-16T14:59:47.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.156+0000 7f2eda600640 1 Processor -- start 2024-01-16T14:59:47.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.156+0000 7f2eda600640 1 -- start start 2024-01-16T14:59:47.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.157+0000 7f2eda600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed4071070 0x7f2ed41457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.157+0000 7f2eda600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed40cf7d0 0x7f2ed40b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.157+0000 7f2eda600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ed414b2c0 0x7f2ed415fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.157+0000 7f2eda600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4145360 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.157+0000 7f2eda600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4144790 con 0x7f2ed414b2c0 2024-01-16T14:59:47.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.157+0000 7f2eda600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4144a30 con 0x7f2ed4071070 2024-01-16T14:59:47.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed40cf7d0 0x7f2ed40b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed40cf7d0 0x7f2ed40b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56190/0 (socket says 172.21.15.141:56190) 2024-01-16T14:59:47.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 -- 172.21.15.141:0/2543731449 learned_addr learned my addr 172.21.15.141:0/2543731449 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:47.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed3400640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed4071070 0x7f2ed41457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed3e00640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ed414b2c0 0x7f2ed415fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 -- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed4071070 msgr2=0x7f2ed41457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed4071070 0x7f2ed41457f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 -- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ed414b2c0 msgr2=0x7f2ed415fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ed414b2c0 0x7f2ed415fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 -- 172.21.15.141:0/2543731449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc009b30 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.158+0000 7f2ed2a00640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed40cf7d0 0x7f2ed40b01c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00c430 tx=0x7f2ebc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.159+0000 7f2ecb400640 1 -- 172.21.15.141:0/2543731449 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc011790 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.159+0000 7f2ecb400640 1 -- 172.21.15.141:0/2543731449 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ebc0118f0 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.159+0000 7f2ecb400640 1 -- 172.21.15.141:0/2543731449 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc011ce0 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.159+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed4144d10 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.159+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed40b0700 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.160+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed40675d0 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.161+0000 7f2ecb400640 1 -- 172.21.15.141:0/2543731449 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f2ebc01c460 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.162+0000 7f2ecb400640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2eac06b470 0x7f2eac06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.162+0000 7f2ecb400640 1 -- 172.21.15.141:0/2543731449 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ebc089eb0 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.162+0000 7f2ed3400640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2eac06b470 0x7f2eac06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.164+0000 7f2ed3400640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2eac06b470 0x7f2eac06d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.166+0000 7f2ecb400640 1 -- 172.21.15.141:0/2543731449 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ebc054a90 con 0x7f2ed40cf7d0 2024-01-16T14:59:47.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.385+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ed41451d0 con 0x7f2eac06b470 2024-01-16T14:59:47.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.396+0000 7f2ecb400640 1 -- 172.21.15.141:0/2543731449 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f2ed41451d0 con 0x7f2eac06b470 2024-01-16T14:59:47.377 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:47.377 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T14:59:47.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:59:47.377 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "sv1" 2024-01-16T14:59:47.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T14:59:47.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:59:47.378 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1clone1" 2024-01-16T14:59:47.378 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:47.378 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T14:59:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2eac06b470 msgr2=0x7f2eac06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2eac06b470 0x7f2eac06d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).stop 2024-01-16T14:59:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed40cf7d0 msgr2=0x7f2ed40b01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed40cf7d0 0x7f2ed40b01c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00c430 tx=0x7f2ebc00c8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 shutdown_connections 2024-01-16T14:59:47.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2eac06b470 0x7f2eac06d930 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ed414b2c0 0x7f2ed415fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ed40cf7d0 0x7f2ed40b01c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 --2- 172.21.15.141:0/2543731449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ed4071070 0x7f2ed41457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.399+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 >> 172.21.15.141:0/2543731449 conn(0x7f2ed406a800 msgr2=0x7f2ed4149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:47.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.400+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 shutdown_connections 2024-01-16T14:59:47.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.400+0000 7f2eda600640 1 -- 172.21.15.141:0/2543731449 wait complete. 2024-01-16T14:59:47.398 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T14:59:47.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.632+0000 7f1c44c00640 1 Processor -- start 2024-01-16T14:59:47.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.632+0000 7f1c44c00640 1 -- start start 2024-01-16T14:59:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.633+0000 7f1c44c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 0x7f1c400d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.633+0000 7f1c44c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c400b2920 con 0x7f1c401610e0 2024-01-16T14:59:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.633+0000 7f1c44c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c400cb670 con 0x7f1c4015cc90 2024-01-16T14:59:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.633+0000 7f1c44c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c400cb950 con 0x7f1c400cf4c0 2024-01-16T14:59:47.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1c401610e0 0x7f1c401614c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49632/0 (socket says 172.21.15.141:49632) 2024-01-16T14:59:47.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3de00640 1 -- 172.21.15.141:0/2784800238 learned_addr learned my addr 172.21.15.141:0/2784800238 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:47.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599264264 0 0) 0x7f1c400b2920 con 0x7f1c401610e0 2024-01-16T14:59:47.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c20003520 con 0x7f1c401610e0 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3d400640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 0x7f1c400d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1456055489 0 0) 0x7f1c20003520 con 0x7f1c401610e0 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 msgr2=0x7f1c400d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.634+0000 7f1c3ca00640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 0x7f1c400d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 >> v1:172.21.15.188:6789/0 conn(0x7f1c4015cc90 legacy=0x7f1c4015d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c400cbc30 con 0x7f1c401610e0 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1c30001670 con 0x7f1c401610e0 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 >> v1:172.21.15.141:6789/0 conn(0x7f1c401610e0 legacy=0x7f1c401614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200039a0 0x7f1c20003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 0x7f1c200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c400cb950 con 0x7f1c20004d00 2024-01-16T14:59:47.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2000bb90 con 0x7f1c200039a0 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2000bd50 con 0x7f1c400cf4c0 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3e800640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3de00640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200039a0 0x7f1c20003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3d400640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 0x7f1c200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3e800640 1 -- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 msgr2=0x7f1c200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3e800640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c400cf4c0 0x7f1c200047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3e800640 1 -- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200039a0 msgr2=0x7f1c20003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3e800640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200039a0 0x7f1c20003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3e800640 1 -- 172.21.15.141:0/2784800238 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c20003520 con 0x7f1c20004d00 2024-01-16T14:59:47.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.635+0000 7f1c3de00640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c200039a0 0x7f1c20003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.636+0000 7f1c3e800640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c20005180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1c38001750 tx=0x7f1c200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e395973dd5a5ebdb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c380052d0 con 0x7f1c20004d00 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3802e050 con 0x7f1c20004d00 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c3ca00640 1 -- 172.21.15.141:0/2784800238 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c380035f0 con 0x7f1c20004d00 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c44c00640 1 -- 172.21.15.141:0/2784800238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 msgr2=0x7f1c20005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c44c00640 1 --2- 172.21.15.141:0/2784800238 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c20005180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1c38001750 tx=0x7f1c200057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c3e800640 1 -- 172.21.15.141:0/2784800238 reap_dead start 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c44c00640 1 -- 172.21.15.141:0/2784800238 shutdown_connections 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c44c00640 1 -- 172.21.15.141:0/2784800238 >> 172.21.15.141:0/2784800238 conn(0x7f1c400d1020 msgr2=0x7f1c400d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c44c00640 1 -- 172.21.15.141:0/2784800238 shutdown_connections 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.637+0000 7f1c44c00640 1 -- 172.21.15.141:0/2784800238 wait complete. 2024-01-16T14:59:47.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.639+0000 7f1c44c00640 1 Processor -- start 2024-01-16T14:59:47.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.639+0000 7f1c44c00640 1 -- start start 2024-01-16T14:59:47.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c44c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c40144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c44c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c4015e300 0x7f1c401e6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c44c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c401e6e60 0x7f1c400c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c44c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c401e72e0 con 0x7f1c20004d00 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c44c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c401455a0 con 0x7f1c401e6e60 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c44c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c40145880 con 0x7f1c4015e300 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c3de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c40144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c3de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c40144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56214/0 (socket says 172.21.15.141:56214) 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.640+0000 7f1c3de00640 1 -- 172.21.15.141:0/81502667 learned_addr learned my addr 172.21.15.141:0/81502667 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3d400640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c4015e300 0x7f1c401e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3de00640 1 -- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c4015e300 msgr2=0x7f1c401e6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3e800640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c401e6e60 0x7f1c400c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3de00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c4015e300 0x7f1c401e6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3de00640 1 -- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c401e6e60 msgr2=0x7f1c400c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3de00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c401e6e60 0x7f1c400c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3de00640 1 -- 172.21.15.141:0/81502667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c40145b60 con 0x7f1c20004d00 2024-01-16T14:59:47.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c3de00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c40144d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000d810 tx=0x7f1c3000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.641+0000 7f1c36000640 1 -- 172.21.15.141:0/81502667 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300074e0 con 0x7f1c20004d00 2024-01-16T14:59:47.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.642+0000 7f1c36000640 1 -- 172.21.15.141:0/81502667 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c30007640 con 0x7f1c20004d00 2024-01-16T14:59:47.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.642+0000 7f1c36000640 1 -- 172.21.15.141:0/81502667 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300049e0 con 0x7f1c20004d00 2024-01-16T14:59:47.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.642+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c40145e40 con 0x7f1c20004d00 2024-01-16T14:59:47.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.642+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4015e760 con 0x7f1c20004d00 2024-01-16T14:59:47.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.643+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c40074520 con 0x7f1c20004d00 2024-01-16T14:59:47.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.644+0000 7f1c36000640 1 -- 172.21.15.141:0/81502667 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f1c30004b40 con 0x7f1c20004d00 2024-01-16T14:59:47.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.644+0000 7f1c36000640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c1806b470 0x7f1c1806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:47.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.645+0000 7f1c36000640 1 -- 172.21.15.141:0/81502667 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c30089b30 con 0x7f1c20004d00 2024-01-16T14:59:47.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.645+0000 7f1c3d400640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c1806b470 0x7f1c1806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:47.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.646+0000 7f1c3d400640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c1806b470 0x7f1c1806d930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1c401f6980 tx=0x7f1c2800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:47.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.649+0000 7f1c36000640 1 -- 172.21.15.141:0/81502667 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c30054580 con 0x7f1c20004d00 2024-01-16T14:59:47.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.869+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c400cc4d0 con 0x7f1c1806b470 2024-01-16T14:59:47.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.899+0000 7f1c36000640 1 -- 172.21.15.141:0/81502667 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1c2800f040 con 0x7f1c1806b470 2024-01-16T14:59:47.880 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:47.880 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T14:59:47.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T14:59:47.880 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1" 2024-01-16T14:59:47.880 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T14:59:47.880 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c1806b470 msgr2=0x7f1c1806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c1806b470 0x7f1c1806d930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1c401f6980 tx=0x7f1c2800b040 comp rx=0 tx=0).stop 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 msgr2=0x7f1c40144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c40144d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000d810 tx=0x7f1c3000dcd0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 shutdown_connections 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c1806b470 0x7f1c1806d930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c401e6e60 0x7f1c400c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c4015e300 0x7f1c401e6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 --2- 172.21.15.141:0/81502667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c20004d00 0x7f1c40144d80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:47.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 >> 172.21.15.141:0/81502667 conn(0x7f1c400d1020 msgr2=0x7f1c4006e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:47.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.902+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 shutdown_connections 2024-01-16T14:59:47.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:47.903+0000 7f1c44c00640 1 -- 172.21.15.141:0/81502667 wait complete. 2024-01-16T14:59:47.903 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T14:59:48.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.137+0000 7f33c5200640 1 Processor -- start 2024-01-16T14:59:48.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.138+0000 7f33c5200640 1 -- start start 2024-01-16T14:59:48.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.139+0000 7f33c5200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f33c00d0100 0x7f33c00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.139+0000 7f33c5200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f33c00d0f40 0x7f33c00d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.139+0000 7f33c5200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c00d0a10 con 0x7f33c0160920 2024-01-16T14:59:48.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.139+0000 7f33c5200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00cf080 con 0x7f33c00d0100 2024-01-16T14:59:48.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.139+0000 7f33c5200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00c6be0 con 0x7f33c00d0f40 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f33c00d0100 0x7f33c00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f33c00d0100 0x7f33c00b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56230/0 (socket says 172.21.15.141:56230) 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bea00640 1 -- 172.21.15.141:0/3986585296 learned_addr learned my addr 172.21.15.141:0/3986585296 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bfe00640 1 --2- 172.21.15.141:0/3986585296 >> v2:172.21.15.188:3300/0 conn(0x7f33c00d0f40 0x7f33c00d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bea00640 1 -- 172.21.15.141:0/3986585296 >> v2:172.21.15.188:3300/0 conn(0x7f33c00d0f40 msgr2=0x7f33c00d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bea00640 1 --2- 172.21.15.141:0/3986585296 >> v2:172.21.15.188:3300/0 conn(0x7f33c00d0f40 0x7f33c00d1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bea00640 1 -- 172.21.15.141:0/3986585296 >> v1:172.21.15.141:6789/0 conn(0x7f33c0160920 legacy=0x7f33c00cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.140+0000 7f33bea00640 1 -- 172.21.15.141:0/3986585296 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00ce7e0 con 0x7f33c00d0100 2024-01-16T14:59:48.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.141+0000 7f33bea00640 1 --2- 172.21.15.141:0/3986585296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0100 0x7f33c00b3f30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f33c01620b0 tx=0x7f33a802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d413259199c5ba95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.141+0000 7f33be000640 1 -- 172.21.15.141:0/3986585296 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a802cb10 con 0x7f33c00d0100 2024-01-16T14:59:48.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33be000640 1 -- 172.21.15.141:0/3986585296 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a802cc70 con 0x7f33c00d0100 2024-01-16T14:59:48.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33be000640 1 -- 172.21.15.141:0/3986585296 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a80343f0 con 0x7f33c00d0100 2024-01-16T14:59:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 -- 172.21.15.141:0/3986585296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0100 msgr2=0x7f33c00b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 --2- 172.21.15.141:0/3986585296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0100 0x7f33c00b3f30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f33c01620b0 tx=0x7f33a802bf70 comp rx=0 tx=0).stop 2024-01-16T14:59:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 -- 172.21.15.141:0/3986585296 shutdown_connections 2024-01-16T14:59:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 --2- 172.21.15.141:0/3986585296 >> v2:172.21.15.188:3300/0 conn(0x7f33c00d0f40 0x7f33c00d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 --2- 172.21.15.141:0/3986585296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0100 0x7f33c00b3f30 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 -- 172.21.15.141:0/3986585296 >> 172.21.15.141:0/3986585296 conn(0x7f33c00d3090 msgr2=0x7f33c00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 -- 172.21.15.141:0/3986585296 shutdown_connections 2024-01-16T14:59:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.142+0000 7f33c5200640 1 -- 172.21.15.141:0/3986585296 wait complete. 2024-01-16T14:59:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.144+0000 7f33c5200640 1 Processor -- start 2024-01-16T14:59:48.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.144+0000 7f33c5200640 1 -- start start 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33c5200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33c00d0100 0x7f33c00cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33c5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0f40 0x7f33c00ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33c5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33c0160920 0x7f33c01f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33c5200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00ca6a0 con 0x7f33c00d0f40 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33c5200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01f3030 con 0x7f33c0160920 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33c5200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01f3310 con 0x7f33c00d0100 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33bea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0f40 0x7f33c00ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33bea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0f40 0x7f33c00ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56238/0 (socket says 172.21.15.141:56238) 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.145+0000 7f33bea00640 1 -- 172.21.15.141:0/1616678639 learned_addr learned my addr 172.21.15.141:0/1616678639 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bea00640 1 -- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33c00d0100 msgr2=0x7f33c00cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bf400640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33c00d0100 0x7f33c00cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bfe00640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33c0160920 0x7f33c01f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bea00640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33c00d0100 0x7f33c00cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bea00640 1 -- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33c0160920 msgr2=0x7f33c01f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bea00640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33c0160920 0x7f33c01f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bea00640 1 -- 172.21.15.141:0/1616678639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a8009b30 con 0x7f33c00d0f40 2024-01-16T14:59:48.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bf400640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33c00d0100 0x7f33c00cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:48.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33bea00640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0f40 0x7f33c00ca160 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f33a8000f80 tx=0x7f33a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.146+0000 7f33b7400640 1 -- 172.21.15.141:0/1616678639 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a803e450 con 0x7f33c00d0f40 2024-01-16T14:59:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.147+0000 7f33b7400640 1 -- 172.21.15.141:0/1616678639 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a803e5f0 con 0x7f33c00d0f40 2024-01-16T14:59:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.147+0000 7f33b7400640 1 -- 172.21.15.141:0/1616678639 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a803e9a0 con 0x7f33c00d0f40 2024-01-16T14:59:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.147+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c01f35f0 con 0x7f33c00d0f40 2024-01-16T14:59:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.147+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c01f2d20 con 0x7f33c00d0f40 2024-01-16T14:59:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.148+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c01f3ab0 con 0x7f33c00d0f40 2024-01-16T14:59:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.149+0000 7f33b7400640 1 -- 172.21.15.141:0/1616678639 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f33a803eb00 con 0x7f33c00d0f40 2024-01-16T14:59:48.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.149+0000 7f33b7400640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f339806b470 0x7f339806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.150+0000 7f33b7400640 1 -- 172.21.15.141:0/1616678639 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33a80ace50 con 0x7f33c00d0f40 2024-01-16T14:59:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.150+0000 7f33bf400640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f339806b470 0x7f339806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.151+0000 7f33bf400640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f339806b470 0x7f339806d930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f33b0000f60 tx=0x7f33b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.154+0000 7f33b7400640 1 -- 172.21.15.141:0/1616678639 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33a8077b50 con 0x7f33c00d0f40 2024-01-16T14:59:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.378+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33c0160d00 con 0x7f339806b470 2024-01-16T14:59:48.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.447+0000 7f33b7400640 1 -- 172.21.15.141:0/1616678639 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f33b0001c90 con 0x7f339806b470 2024-01-16T14:59:48.428 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:48.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f339806b470 msgr2=0x7f339806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f339806b470 0x7f339806d930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f33b0000f60 tx=0x7f33b0009040 comp rx=0 tx=0).stop 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0f40 msgr2=0x7f33c00ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0f40 0x7f33c00ca160 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f33a8000f80 tx=0x7f33a8004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 shutdown_connections 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f339806b470 0x7f339806d930 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33c0160920 0x7f33c01f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33c00d0f40 0x7f33c00ca160 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 --2- 172.21.15.141:0/1616678639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33c00d0100 0x7f33c00cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 >> 172.21.15.141:0/1616678639 conn(0x7f33c00d3090 msgr2=0x7f33c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:48.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 shutdown_connections 2024-01-16T14:59:48.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.450+0000 7f33c5200640 1 -- 172.21.15.141:0/1616678639 wait complete. 2024-01-16T14:59:48.452 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T14:59:48.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.678+0000 7f8dd5c00640 1 Processor -- start 2024-01-16T14:59:48.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.678+0000 7f8dd5c00640 1 -- start start 2024-01-16T14:59:48.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.679+0000 7f8dd5c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8dd00cfcf0 0x7f8dd0160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.679+0000 7f8dd5c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00d3910 0x7f8dd0160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.679+0000 7f8dd5c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd00d4160 con 0x7f8dd00d00d0 2024-01-16T14:59:48.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.679+0000 7f8dd5c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00ceff0 con 0x7f8dd00cfcf0 2024-01-16T14:59:48.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.679+0000 7f8dd5c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00cf410 con 0x7f8dd00d3910 2024-01-16T14:59:48.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.680+0000 7f8dcea00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8dd00cfcf0 0x7f8dd0160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.680+0000 7f8dcf400640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f8dd00d00d0 0x7f8dd00b2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:55330/0 (socket says 172.21.15.141:55330) 2024-01-16T14:59:48.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.680+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00d3910 0x7f8dd0160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.680+0000 7f8dcf400640 1 -- 172.21.15.141:0/3859671708 learned_addr learned my addr 172.21.15.141:0/3859671708 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:48.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.681+0000 7f8dc7e00640 1 -- 172.21.15.141:0/3859671708 >> v2:172.21.15.188:3300/0 conn(0x7f8dd00cfcf0 msgr2=0x7f8dd0160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:48.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.681+0000 7f8dc7e00640 1 --2- 172.21.15.141:0/3859671708 >> v2:172.21.15.188:3300/0 conn(0x7f8dd00cfcf0 0x7f8dd0160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.681+0000 7f8dc7e00640 1 -- 172.21.15.141:0/3859671708 >> v1:172.21.15.188:6789/0 conn(0x7f8dd00d00d0 legacy=0x7f8dd00b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.681+0000 7f8dc7e00640 1 -- 172.21.15.141:0/3859671708 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00b2420 con 0x7f8dd00d3910 2024-01-16T14:59:48.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.681+0000 7f8dcea00640 1 --2- 172.21.15.141:0/3859671708 >> v2:172.21.15.188:3300/0 conn(0x7f8dd00cfcf0 0x7f8dd0160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:48.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.682+0000 7f8dc7e00640 1 --2- 172.21.15.141:0/3859671708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00d3910 0x7f8dd0160c40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097a0 tx=0x7f8db802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576a852bc6f162fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.683+0000 7f8dce000640 1 -- 172.21.15.141:0/3859671708 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8031780 con 0x7f8dd00d3910 2024-01-16T14:59:48.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.683+0000 7f8dce000640 1 -- 172.21.15.141:0/3859671708 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80318e0 con 0x7f8dd00d3910 2024-01-16T14:59:48.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.683+0000 7f8dce000640 1 -- 172.21.15.141:0/3859671708 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8031c70 con 0x7f8dd00d3910 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3859671708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00d3910 msgr2=0x7f8dd0160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3859671708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00d3910 0x7f8dd0160c40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097a0 tx=0x7f8db802b820 comp rx=0 tx=0).stop 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3859671708 shutdown_connections 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3859671708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00d3910 0x7f8dd0160c40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3859671708 >> v2:172.21.15.188:3300/0 conn(0x7f8dd00cfcf0 0x7f8dd0160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3859671708 >> 172.21.15.141:0/3859671708 conn(0x7f8dd009c530 msgr2=0x7f8dd00d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3859671708 shutdown_connections 2024-01-16T14:59:48.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.684+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3859671708 wait complete. 2024-01-16T14:59:48.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.686+0000 7f8dd5c00640 1 Processor -- start 2024-01-16T14:59:48.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.686+0000 7f8dd5c00640 1 -- start start 2024-01-16T14:59:48.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dd5c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00cfcf0 0x7f8dd00b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dd5c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd00d00d0 0x7f8dd0154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dd5c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8dd00d3910 0x7f8dd00b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dd5c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00b69d0 con 0x7f8dd00d3910 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dd5c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00b6cb0 con 0x7f8dd00d00d0 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dd5c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00b6f90 con 0x7f8dd00cfcf0 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dcf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8dd00d3910 0x7f8dd00b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dcf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8dd00d3910 0x7f8dd00b6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56244/0 (socket says 172.21.15.141:56244) 2024-01-16T14:59:48.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.687+0000 7f8dcf400640 1 -- 172.21.15.141:0/3056149142 learned_addr learned my addr 172.21.15.141:0/3056149142 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:48.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dcf400640 1 -- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00cfcf0 msgr2=0x7f8dd00b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dcf400640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00cfcf0 0x7f8dd00b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dcf400640 1 -- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd00d00d0 msgr2=0x7f8dd0154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dc7e00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd00d00d0 0x7f8dd0154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dcf400640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd00d00d0 0x7f8dd0154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dcf400640 1 -- 172.21.15.141:0/3056149142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc0009400 con 0x7f8dd00d3910 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dcf400640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8dd00d3910 0x7f8dd00b6050 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800cbc0 tx=0x7f8dc800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dc6a00640 1 -- 172.21.15.141:0/3056149142 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc800ad40 con 0x7f8dd00d3910 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dc6a00640 1 -- 172.21.15.141:0/3056149142 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8012400 con 0x7f8dd00d3910 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db8009400 con 0x7f8dd00d3910 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dc6a00640 1 -- 172.21.15.141:0/3056149142 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc800aea0 con 0x7f8dd00d3910 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.688+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd00b7550 con 0x7f8dd00d3910 2024-01-16T14:59:48.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.690+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd00675d0 con 0x7f8dd00d3910 2024-01-16T14:59:48.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.690+0000 7f8dc6a00640 1 -- 172.21.15.141:0/3056149142 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f8dc8007070 con 0x7f8dd00d3910 2024-01-16T14:59:48.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.691+0000 7f8dc6a00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8da806b4a0 0x7f8da806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:48.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.691+0000 7f8dc6a00640 1 -- 172.21.15.141:0/3056149142 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dc8089d10 con 0x7f8dd00d3910 2024-01-16T14:59:48.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.692+0000 7f8dcea00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8da806b4a0 0x7f8da806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:48.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.693+0000 7f8dcea00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8da806b4a0 0x7f8da806d960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0006e80 tx=0x7f8dc0006dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:48.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.696+0000 7f8dc6a00640 1 -- 172.21.15.141:0/3056149142 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8dc808f020 con 0x7f8dd00d3910 2024-01-16T14:59:48.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.914+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dd00cec10 con 0x7f8da806b4a0 2024-01-16T14:59:48.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.957+0000 7f8dc6a00640 1 -- 172.21.15.141:0/3056149142 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8dc000e920 con 0x7f8da806b4a0 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8da806b4a0 msgr2=0x7f8da806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8da806b4a0 0x7f8da806d960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0006e80 tx=0x7f8dc0006dd0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8dd00d3910 msgr2=0x7f8dd00b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8dd00d3910 0x7f8dd00b6050 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800cbc0 tx=0x7f8dc800a430 comp rx=0 tx=0).stop 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 shutdown_connections 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8da806b4a0 0x7f8da806d960 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8dd00d3910 0x7f8dd00b6050 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd00d00d0 0x7f8dd0154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 --2- 172.21.15.141:0/3056149142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8dd00cfcf0 0x7f8dd00b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:48.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 >> 172.21.15.141:0/3056149142 conn(0x7f8dd009c530 msgr2=0x7f8dd006b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:48.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 shutdown_connections 2024-01-16T14:59:48.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:48.960+0000 7f8dd5c00640 1 -- 172.21.15.141:0/3056149142 wait complete. 2024-01-16T14:59:48.945 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.964+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.945 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.964+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.945 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.964+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.945 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.964+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.946 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.964+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.960 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T14:59:48.972 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.991+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.972 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.991+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.972 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.991+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.972 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.991+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:48.973 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T14:59:48.991+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T14:59:49.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.176+0000 7f27efe00640 1 Processor -- start 2024-01-16T14:59:49.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.177+0000 7f27efe00640 1 -- start start 2024-01-16T14:59:49.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.178+0000 7f27efe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27e8143ae0 0x7f27e80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.178+0000 7f27efe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f27e815e3b0 0x7f27e80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.178+0000 7f27efe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e80cfc30 con 0x7f27e80cf7d0 2024-01-16T14:59:49.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.178+0000 7f27efe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e80b1ee0 con 0x7f27e8143ae0 2024-01-16T14:59:49.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.178+0000 7f27efe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e80b2780 con 0x7f27e815e3b0 2024-01-16T14:59:49.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.179+0000 7f27ed400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27e8143ae0 0x7f27e80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.179+0000 7f27ed400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27e8143ae0 0x7f27e80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56254/0 (socket says 172.21.15.141:56254) 2024-01-16T14:59:49.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.179+0000 7f27ed400640 1 -- 172.21.15.141:0/2770351814 learned_addr learned my addr 172.21.15.141:0/2770351814 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:49.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.179+0000 7f27ed400640 1 -- 172.21.15.141:0/2770351814 >> v2:172.21.15.188:3300/0 conn(0x7f27e815e3b0 msgr2=0x7f27e80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:49.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.179+0000 7f27ed400640 1 --2- 172.21.15.141:0/2770351814 >> v2:172.21.15.188:3300/0 conn(0x7f27e815e3b0 0x7f27e80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.179+0000 7f27ed400640 1 -- 172.21.15.141:0/2770351814 >> v1:172.21.15.141:6789/0 conn(0x7f27e80cf7d0 legacy=0x7f27e815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.179+0000 7f27ed400640 1 -- 172.21.15.141:0/2770351814 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e815e990 con 0x7f27e8143ae0 2024-01-16T14:59:49.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.180+0000 7f27ed400640 1 --2- 172.21.15.141:0/2770351814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e8143ae0 0x7f27e80cf290 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f27e806e4f0 tx=0x7f27d802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141e5b29d96d2e6e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.180+0000 7f27e7e00640 1 -- 172.21.15.141:0/2770351814 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d802cb10 con 0x7f27e8143ae0 2024-01-16T14:59:49.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27e7e00640 1 -- 172.21.15.141:0/2770351814 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d802cc70 con 0x7f27e8143ae0 2024-01-16T14:59:49.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27e7e00640 1 -- 172.21.15.141:0/2770351814 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d80343f0 con 0x7f27e8143ae0 2024-01-16T14:59:49.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 -- 172.21.15.141:0/2770351814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e8143ae0 msgr2=0x7f27e80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 --2- 172.21.15.141:0/2770351814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e8143ae0 0x7f27e80cf290 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f27e806e4f0 tx=0x7f27d802bf70 comp rx=0 tx=0).stop 2024-01-16T14:59:49.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 -- 172.21.15.141:0/2770351814 shutdown_connections 2024-01-16T14:59:49.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 --2- 172.21.15.141:0/2770351814 >> v2:172.21.15.188:3300/0 conn(0x7f27e815e3b0 0x7f27e80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 --2- 172.21.15.141:0/2770351814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e8143ae0 0x7f27e80cf290 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 -- 172.21.15.141:0/2770351814 >> 172.21.15.141:0/2770351814 conn(0x7f27e806a800 msgr2=0x7f27e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 -- 172.21.15.141:0/2770351814 shutdown_connections 2024-01-16T14:59:49.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.181+0000 7f27efe00640 1 -- 172.21.15.141:0/2770351814 wait complete. 2024-01-16T14:59:49.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.183+0000 7f27efe00640 1 Processor -- start 2024-01-16T14:59:49.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.183+0000 7f27efe00640 1 -- start start 2024-01-16T14:59:49.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.184+0000 7f27efe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e80cf7d0 0x7f27e80adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.184+0000 7f27efe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e8143ae0 0x7f27e813bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.184+0000 7f27efe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e815e3b0 0x7f27e813f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.184+0000 7f27efe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e813acf0 con 0x7f27e815e3b0 2024-01-16T14:59:49.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.184+0000 7f27efe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e813afd0 con 0x7f27e80cf7d0 2024-01-16T14:59:49.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.184+0000 7f27efe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e813b2b0 con 0x7f27e8143ae0 2024-01-16T14:59:49.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e815e3b0 0x7f27e813f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e815e3b0 0x7f27e813f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56268/0 (socket says 172.21.15.141:56268) 2024-01-16T14:59:49.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 -- 172.21.15.141:0/1025279208 learned_addr learned my addr 172.21.15.141:0/1025279208 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:49.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ed400640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e80cf7d0 0x7f27e80adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27eca00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e8143ae0 0x7f27e813bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 -- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e8143ae0 msgr2=0x7f27e813bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e8143ae0 0x7f27e813bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 -- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e80cf7d0 msgr2=0x7f27e80adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e80cf7d0 0x7f27e80adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ede00640 1 -- 172.21.15.141:0/1025279208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d8009b30 con 0x7f27e815e3b0 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.185+0000 7f27ed400640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e80cf7d0 0x7f27e80adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.186+0000 7f27ede00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e815e3b0 0x7f27e813f790 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b410 tx=0x7f27e000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.186+0000 7f27e5600640 1 -- 172.21.15.141:0/1025279208 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0011830 con 0x7f27e815e3b0 2024-01-16T14:59:49.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.186+0000 7f27e5600640 1 -- 172.21.15.141:0/1025279208 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e00119b0 con 0x7f27e815e3b0 2024-01-16T14:59:49.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.186+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e813b590 con 0x7f27e815e3b0 2024-01-16T14:59:49.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.186+0000 7f27e5600640 1 -- 172.21.15.141:0/1025279208 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0011d60 con 0x7f27e815e3b0 2024-01-16T14:59:49.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.186+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e813a910 con 0x7f27e815e3b0 2024-01-16T14:59:49.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.187+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e80675d0 con 0x7f27e815e3b0 2024-01-16T14:59:49.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.189+0000 7f27e5600640 1 -- 172.21.15.141:0/1025279208 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f27e001a400 con 0x7f27e815e3b0 2024-01-16T14:59:49.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.190+0000 7f27e5600640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27c006b470 0x7f27c006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.190+0000 7f27e5600640 1 -- 172.21.15.141:0/1025279208 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27e008a0d0 con 0x7f27e815e3b0 2024-01-16T14:59:49.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.191+0000 7f27ed400640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27c006b470 0x7f27c006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.192+0000 7f27ed400640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27c006b470 0x7f27c006d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f27d8035000 tx=0x7f27d8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.194+0000 7f27e5600640 1 -- 172.21.15.141:0/1025279208 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27e0054c20 con 0x7f27e815e3b0 2024-01-16T14:59:49.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.415+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e80b2410 con 0x7f27c006b470 2024-01-16T14:59:49.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.452+0000 7f27e5600640 1 -- 172.21.15.141:0/1025279208 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f27d80440c0 con 0x7f27c006b470 2024-01-16T14:59:49.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27c006b470 msgr2=0x7f27c006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27c006b470 0x7f27c006d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f27d8035000 tx=0x7f27d8030000 comp rx=0 tx=0).stop 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e815e3b0 msgr2=0x7f27e813f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e815e3b0 0x7f27e813f790 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b410 tx=0x7f27e000b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 shutdown_connections 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f27c006b470 0x7f27c006d930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27e815e3b0 0x7f27e813f790 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27e8143ae0 0x7f27e813bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 --2- 172.21.15.141:0/1025279208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27e80cf7d0 0x7f27e80adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 >> 172.21.15.141:0/1025279208 conn(0x7f27e806a800 msgr2=0x7f27e815da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 shutdown_connections 2024-01-16T14:59:49.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.455+0000 7f27efe00640 1 -- 172.21.15.141:0/1025279208 wait complete. 2024-01-16T14:59:49.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:49.456 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T14:59:49.507 INFO:teuthology.orchestra.run.smithi141.stdout:2 2024-01-16T14:59:49.508 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T14:59:49.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.709+0000 7f9e22a00640 1 Processor -- start 2024-01-16T14:59:49.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.709+0000 7f9e22a00640 1 -- start start 2024-01-16T14:59:49.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.710+0000 7f9e22a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9e1c0b02b0 0x7f9e1c06c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.710+0000 7f9e22a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9e1c0cb640 0x7f9e1c0cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.710+0000 7f9e22a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c0d0530 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.710+0000 7f9e22a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c0b3a80 con 0x7f9e1c0cb640 2024-01-16T14:59:49.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.710+0000 7f9e22a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c0b0070 con 0x7f9e1c0b02b0 2024-01-16T14:59:49.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e20a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9e1c0cb640 0x7f9e1c0cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e20a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9e1c0cb640 0x7f9e1c0cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56270/0 (socket says 172.21.15.141:56270) 2024-01-16T14:59:49.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e20a00640 1 -- 172.21.15.141:0/2382619773 learned_addr learned my addr 172.21.15.141:0/2382619773 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:49.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e1b400640 1 --2- 172.21.15.141:0/2382619773 >> v2:172.21.15.188:3300/0 conn(0x7f9e1c0b02b0 0x7f9e1c06c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e20a00640 1 -- 172.21.15.141:0/2382619773 >> v2:172.21.15.188:3300/0 conn(0x7f9e1c0b02b0 msgr2=0x7f9e1c06c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:49.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e20a00640 1 --2- 172.21.15.141:0/2382619773 >> v2:172.21.15.188:3300/0 conn(0x7f9e1c0b02b0 0x7f9e1c06c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e20a00640 1 -- 172.21.15.141:0/2382619773 >> v1:172.21.15.188:6789/0 conn(0x7f9e1c15d6e0 legacy=0x7f9e1c06b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:49.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.711+0000 7f9e20a00640 1 -- 172.21.15.141:0/2382619773 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c0b37a0 con 0x7f9e1c0cb640 2024-01-16T14:59:49.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.713+0000 7f9e20a00640 1 --2- 172.21.15.141:0/2382619773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c0cb640 0x7f9e1c0cbad0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c161220 tx=0x7f9e1402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389d242ec6f13ff5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.713+0000 7f9e1aa00640 1 -- 172.21.15.141:0/2382619773 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1402ca80 con 0x7f9e1c0cb640 2024-01-16T14:59:49.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.713+0000 7f9e1aa00640 1 -- 172.21.15.141:0/2382619773 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1402cbe0 con 0x7f9e1c0cb640 2024-01-16T14:59:49.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.713+0000 7f9e1aa00640 1 -- 172.21.15.141:0/2382619773 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e140343f0 con 0x7f9e1c0cb640 2024-01-16T14:59:49.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 -- 172.21.15.141:0/2382619773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c0cb640 msgr2=0x7f9e1c0cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 --2- 172.21.15.141:0/2382619773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c0cb640 0x7f9e1c0cbad0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c161220 tx=0x7f9e1402bee0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 -- 172.21.15.141:0/2382619773 shutdown_connections 2024-01-16T14:59:49.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 --2- 172.21.15.141:0/2382619773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c0cb640 0x7f9e1c0cbad0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 --2- 172.21.15.141:0/2382619773 >> v2:172.21.15.188:3300/0 conn(0x7f9e1c0b02b0 0x7f9e1c06c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 -- 172.21.15.141:0/2382619773 >> 172.21.15.141:0/2382619773 conn(0x7f9e1c0b2ae0 msgr2=0x7f9e1c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:49.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 -- 172.21.15.141:0/2382619773 shutdown_connections 2024-01-16T14:59:49.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.714+0000 7f9e22a00640 1 -- 172.21.15.141:0/2382619773 wait complete. 2024-01-16T14:59:49.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.715+0000 7f9e22a00640 1 Processor -- start 2024-01-16T14:59:49.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.716+0000 7f9e22a00640 1 -- start start 2024-01-16T14:59:49.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e22a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e1c0b02b0 0x7f9e1c155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e22a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1c0cb640 0x7f9e1c0ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e22a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c15d6e0 0x7f9e1c1e3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e22a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0c5430 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e22a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c156360 con 0x7f9e1c0cb640 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e22a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c1e6130 con 0x7f9e1c0b02b0 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c15d6e0 0x7f9e1c1e3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c15d6e0 0x7f9e1c1e3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56278/0 (socket says 172.21.15.141:56278) 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 -- 172.21.15.141:0/1577780473 learned_addr learned my addr 172.21.15.141:0/1577780473 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e1be00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e1c0b02b0 0x7f9e1c155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 -- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e1c0b02b0 msgr2=0x7f9e1c155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e1c0b02b0 0x7f9e1c155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e1b400640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1c0cb640 0x7f9e1c0ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 -- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1c0cb640 msgr2=0x7f9e1c0ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1c0cb640 0x7f9e1c0ca430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.717+0000 7f9e20a00640 1 -- 172.21.15.141:0/1577780473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14009b30 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.718+0000 7f9e20a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c15d6e0 0x7f9e1c1e3b20 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9e14036040 tx=0x7f9e14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.718+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1403e450 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.718+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1403e5d0 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.718+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1403e980 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.718+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c0caf30 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.718+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c0c4de0 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.719+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c0c55e0 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.720+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f9e1402cd50 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.721+0000 7f9e13e00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9df406b470 0x7f9df406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:49.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.721+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e140acbb0 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.722+0000 7f9e1be00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9df406b470 0x7f9df406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:49.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.723+0000 7f9e1be00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9df406b470 0x7f9df406d930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00a730 tx=0x7f9e0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.725+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e1c0c55e0 con 0x7f9e1c15d6e0 2024-01-16T14:59:49.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:49.929+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e1c0d01c0 con 0x7f9df406b470 2024-01-16T14:59:49.954 INFO:tasks.ceph.mon.a.smithi141.stderr:2024-01-16T14:59:49.971+0000 7f723b200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T14:59:50.044 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T14:59:50.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:51.004+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9e14077ad0 con 0x7f9e1c15d6e0 2024-01-16T14:59:52.999 INFO:teuthology.orchestra.run.smithi141.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T14:59:53.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.015+0000 7f9e13e00640 1 -- 172.21.15.141:0/1577780473 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f9e0c00c520 con 0x7f9df406b470 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.020+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9df406b470 msgr2=0x7f9df406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.020+0000 7f9e22a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9df406b470 0x7f9df406d930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00a730 tx=0x7f9e0c008040 comp rx=0 tx=0).stop 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.020+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c15d6e0 msgr2=0x7f9e1c1e3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.020+0000 7f9e22a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c15d6e0 0x7f9e1c1e3b20 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9e14036040 tx=0x7f9e14004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.020+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 shutdown_connections 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.020+0000 7f9e22a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9df406b470 0x7f9df406d930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.021+0000 7f9e22a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e1c15d6e0 0x7f9e1c1e3b20 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.021+0000 7f9e22a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1c0cb640 0x7f9e1c0ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.021+0000 7f9e22a00640 1 --2- 172.21.15.141:0/1577780473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9e1c0b02b0 0x7f9e1c155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.021+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 >> 172.21.15.141:0/1577780473 conn(0x7f9e1c0b2ae0 msgr2=0x7f9e1c06ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.021+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 shutdown_connections 2024-01-16T14:59:53.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.021+0000 7f9e22a00640 1 -- 172.21.15.141:0/1577780473 wait complete. 2024-01-16T14:59:53.057 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T14:59:53.057 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:59:53.181 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T14:59:53.282 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T14:59:53.282 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:59:53.282 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:53.337 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:53.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.567+0000 7f65c4000640 1 Processor -- start 2024-01-16T14:59:53.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.567+0000 7f65c4000640 1 -- start start 2024-01-16T14:59:53.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.568+0000 7f65c4000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f65bc0cf9a0 0x7f65bc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.568+0000 7f65c4000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f65bc15e1e0 0x7f65bc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.568+0000 7f65c4000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc0b1ee0 con 0x7f65bc143ae0 2024-01-16T14:59:53.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.568+0000 7f65c4000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc0b2780 con 0x7f65bc0cf9a0 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.568+0000 7f65c4000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc15e990 con 0x7f65bc15e1e0 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.568+0000 7f65c0c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f65bc0cf9a0 0x7f65bc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.569+0000 7f65c0c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f65bc0cf9a0 0x7f65bc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46776/0 (socket says 172.21.15.141:46776) 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.569+0000 7f65c0c00640 1 -- 172.21.15.141:0/2432838479 learned_addr learned my addr 172.21.15.141:0/2432838479 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.569+0000 7f65c0c00640 1 -- 172.21.15.141:0/2432838479 >> v2:172.21.15.188:3300/0 conn(0x7f65bc15e1e0 msgr2=0x7f65bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.569+0000 7f65c0c00640 1 --2- 172.21.15.141:0/2432838479 >> v2:172.21.15.188:3300/0 conn(0x7f65bc15e1e0 0x7f65bc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.569+0000 7f65c0c00640 1 -- 172.21.15.141:0/2432838479 >> v1:172.21.15.188:6789/0 conn(0x7f65bc143ae0 legacy=0x7f65bc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.569+0000 7f65c0c00640 1 -- 172.21.15.141:0/2432838479 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc0b2130 con 0x7f65bc0cf9a0 2024-01-16T14:59:53.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.570+0000 7f65c0c00640 1 --2- 172.21.15.141:0/2432838479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc0cf9a0 0x7f65bc15dca0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0ae9d0 tx=0x7f65a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b7a6f9ff898374 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.570+0000 7f65bbe00640 1 -- 172.21.15.141:0/2432838479 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4031870 con 0x7f65bc0cf9a0 2024-01-16T14:59:53.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.570+0000 7f65bbe00640 1 -- 172.21.15.141:0/2432838479 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a40319d0 con 0x7f65bc0cf9a0 2024-01-16T14:59:53.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.570+0000 7f65bbe00640 1 -- 172.21.15.141:0/2432838479 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4031d60 con 0x7f65bc0cf9a0 2024-01-16T14:59:53.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 -- 172.21.15.141:0/2432838479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc0cf9a0 msgr2=0x7f65bc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 --2- 172.21.15.141:0/2432838479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc0cf9a0 0x7f65bc15dca0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0ae9d0 tx=0x7f65a402b910 comp rx=0 tx=0).stop 2024-01-16T14:59:53.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 -- 172.21.15.141:0/2432838479 shutdown_connections 2024-01-16T14:59:53.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 --2- 172.21.15.141:0/2432838479 >> v2:172.21.15.188:3300/0 conn(0x7f65bc15e1e0 0x7f65bc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 --2- 172.21.15.141:0/2432838479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc0cf9a0 0x7f65bc15dca0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 -- 172.21.15.141:0/2432838479 >> 172.21.15.141:0/2432838479 conn(0x7f65bc06a800 msgr2=0x7f65bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 -- 172.21.15.141:0/2432838479 shutdown_connections 2024-01-16T14:59:53.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.571+0000 7f65c4000640 1 -- 172.21.15.141:0/2432838479 wait complete. 2024-01-16T14:59:53.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.573+0000 7f65c4000640 1 Processor -- start 2024-01-16T14:59:53.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.573+0000 7f65c4000640 1 -- start start 2024-01-16T14:59:53.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c4000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65bc0cf9a0 0x7f65bc138a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c4000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc143ae0 0x7f65bc13e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c4000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65bc15e1e0 0x7f65bc139d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c4000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc13da40 con 0x7f65bc143ae0 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c4000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc13ad90 con 0x7f65bc0cf9a0 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c4000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc138fc0 con 0x7f65bc15e1e0 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc143ae0 0x7f65bc13e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc143ae0 0x7f65bc13e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46786/0 (socket says 172.21.15.141:46786) 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 -- 172.21.15.141:0/2613952234 learned_addr learned my addr 172.21.15.141:0/2613952234 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 -- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65bc15e1e0 msgr2=0x7f65bc139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65bc15e1e0 0x7f65bc139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 -- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65bc0cf9a0 msgr2=0x7f65bc138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65bc0cf9a0 0x7f65bc138a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.574+0000 7f65c0c00640 1 -- 172.21.15.141:0/2613952234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a40094d0 con 0x7f65bc143ae0 2024-01-16T14:59:53.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.575+0000 7f65c0c00640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc143ae0 0x7f65bc13e470 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f65a40097f0 tx=0x7f65a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.575+0000 7f65b9600640 1 -- 172.21.15.141:0/2613952234 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a403ad30 con 0x7f65bc143ae0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.575+0000 7f65b9600640 1 -- 172.21.15.141:0/2613952234 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a403ae90 con 0x7f65bc143ae0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.575+0000 7f65b9600640 1 -- 172.21.15.141:0/2613952234 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4044630 con 0x7f65bc143ae0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.575+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc13a7d0 con 0x7f65bc143ae0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.575+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc13dc70 con 0x7f65bc143ae0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.576+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc0675d0 con 0x7f65bc143ae0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.577+0000 7f65b9600640 1 -- 172.21.15.141:0/2613952234 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f65a404d450 con 0x7f65bc143ae0 2024-01-16T14:59:53.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.578+0000 7f65b9600640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f659406b500 0x7f659406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.578+0000 7f65b9600640 1 -- 172.21.15.141:0/2613952234 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f65a402d040 con 0x7f65bc143ae0 2024-01-16T14:59:53.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.579+0000 7f65c1600640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f659406b500 0x7f659406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.580+0000 7f65c1600640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f659406b500 0x7f659406d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65ac001560 tx=0x7f65ac007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.583+0000 7f65b9600640 1 -- 172.21.15.141:0/2613952234 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65a4077260 con 0x7f65bc143ae0 2024-01-16T14:59:53.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.834+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65bc06aff0 con 0x7f65bc143ae0 2024-01-16T14:59:53.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.836+0000 7f65b9600640 1 -- 172.21.15.141:0/2613952234 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13034 (secure 0 0 0) 0x7f65a407a880 con 0x7f65bc143ae0 2024-01-16T14:59:53.817 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:53.817 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":38,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:59:52.987816+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/176725955":"2024-01-17T14:59:49.971845+0000","172.21.15.141:6833/176725955":"2024-01-17T14:59:49.971845+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f659406b500 msgr2=0x7f659406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f659406b500 0x7f659406d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65ac001560 tx=0x7f65ac007990 comp rx=0 tx=0).stop 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc143ae0 msgr2=0x7f65bc13e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc143ae0 0x7f65bc13e470 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f65a40097f0 tx=0x7f65a40046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 shutdown_connections 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f659406b500 0x7f659406d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65bc15e1e0 0x7f65bc139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65bc143ae0 0x7f65bc13e470 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 --2- 172.21.15.141:0/2613952234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65bc0cf9a0 0x7f65bc138a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 >> 172.21.15.141:0/2613952234 conn(0x7f65bc06a800 msgr2=0x7f65bc0caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 shutdown_connections 2024-01-16T14:59:53.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:53.839+0000 7f65c4000640 1 -- 172.21.15.141:0/2613952234 wait complete. 2024-01-16T14:59:53.838 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:53.838 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:54.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.052+0000 7fc366600640 1 Processor -- start 2024-01-16T14:59:54.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.052+0000 7fc366600640 1 -- start start 2024-01-16T14:59:54.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.053+0000 7fc366600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc360143810 0x7fc360071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.053+0000 7fc366600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 0x7fc36015e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.053+0000 7fc366600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc36015d770 0x7fc36015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.053+0000 7fc366600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc360143bf0 con 0x7fc36015d770 2024-01-16T14:59:54.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.053+0000 7fc366600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36015e140 con 0x7fc360143810 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.053+0000 7fc366600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36006c730 con 0x7fc3600ae170 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc36015d770 0x7fc36015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc36015d770 0x7fc36015dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46802/0 (socket says 172.21.15.141:46802) 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 -- 172.21.15.141:0/3494796488 learned_addr learned my addr 172.21.15.141:0/3494796488 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365200640 1 --2- 172.21.15.141:0/3494796488 >> v2:172.21.15.188:3300/0 conn(0x7fc360143810 0x7fc360071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 -- 172.21.15.141:0/3494796488 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 msgr2=0x7fc36015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 --2- 172.21.15.141:0/3494796488 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 0x7fc36015e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 -- 172.21.15.141:0/3494796488 >> v2:172.21.15.188:3300/0 conn(0x7fc360143810 msgr2=0x7fc360071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 --2- 172.21.15.141:0/3494796488 >> v2:172.21.15.188:3300/0 conn(0x7fc360143810 0x7fc360071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.054+0000 7fc365c00640 1 -- 172.21.15.141:0/3494796488 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600ae5f0 con 0x7fc36015d770 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.055+0000 7fc365c00640 1 --2- 172.21.15.141:0/3494796488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc36015dc00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc3600cec40 tx=0x7fc35802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc03ef1afc96db1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc35f400640 1 -- 172.21.15.141:0/3494796488 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35802c9e0 con 0x7fc36015d770 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc35f400640 1 -- 172.21.15.141:0/3494796488 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35802cb40 con 0x7fc36015d770 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc35f400640 1 -- 172.21.15.141:0/3494796488 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580343f0 con 0x7fc36015d770 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc366600640 1 -- 172.21.15.141:0/3494796488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 msgr2=0x7fc36015dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc366600640 1 --2- 172.21.15.141:0/3494796488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc36015dc00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc3600cec40 tx=0x7fc35802be40 comp rx=0 tx=0).stop 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc366600640 1 -- 172.21.15.141:0/3494796488 shutdown_connections 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc366600640 1 --2- 172.21.15.141:0/3494796488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc36015dc00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc366600640 1 --2- 172.21.15.141:0/3494796488 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 0x7fc36015e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc366600640 1 --2- 172.21.15.141:0/3494796488 >> v2:172.21.15.188:3300/0 conn(0x7fc360143810 0x7fc360071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.056+0000 7fc366600640 1 -- 172.21.15.141:0/3494796488 >> 172.21.15.141:0/3494796488 conn(0x7fc36006a7d0 msgr2=0x7fc36006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.057+0000 7fc366600640 1 -- 172.21.15.141:0/3494796488 shutdown_connections 2024-01-16T14:59:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.057+0000 7fc366600640 1 -- 172.21.15.141:0/3494796488 wait complete. 2024-01-16T14:59:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.058+0000 7fc366600640 1 Processor -- start 2024-01-16T14:59:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.058+0000 7fc366600640 1 -- start start 2024-01-16T14:59:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.059+0000 7fc366600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 0x7fc3601377e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.059+0000 7fc366600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc360143810 0x7fc3601382e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.059+0000 7fc366600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc360138820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.059+0000 7fc366600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600b12b0 con 0x7fc36015d770 2024-01-16T14:59:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.059+0000 7fc366600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36013a600 con 0x7fc360143810 2024-01-16T14:59:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.059+0000 7fc366600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600b0630 con 0x7fc3600ae170 2024-01-16T14:59:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc360138820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc360138820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46814/0 (socket says 172.21.15.141:46814) 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 -- 172.21.15.141:0/641859861 learned_addr learned my addr 172.21.15.141:0/641859861 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc35fe00640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc360143810 0x7fc3601382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365200640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 0x7fc3601377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 -- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 msgr2=0x7fc3601377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 0x7fc3601377e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 -- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc360143810 msgr2=0x7fc3601382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc360143810 0x7fc3601382e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.060+0000 7fc365c00640 1 -- 172.21.15.141:0/641859861 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358009b30 con 0x7fc36015d770 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.061+0000 7fc365c00640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc360138820 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc358036040 tx=0x7fc358004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.061+0000 7fc35cc00640 1 -- 172.21.15.141:0/641859861 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358009510 con 0x7fc36015d770 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.061+0000 7fc35cc00640 1 -- 172.21.15.141:0/641859861 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35803e450 con 0x7fc36015d770 2024-01-16T14:59:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.061+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3600b0910 con 0x7fc36015d770 2024-01-16T14:59:54.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.061+0000 7fc35cc00640 1 -- 172.21.15.141:0/641859861 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35803e7e0 con 0x7fc36015d770 2024-01-16T14:59:54.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.061+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3600b0ed0 con 0x7fc36015d770 2024-01-16T14:59:54.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.062+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc360067450 con 0x7fc36015d770 2024-01-16T14:59:54.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.063+0000 7fc35cc00640 1 -- 172.21.15.141:0/641859861 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc35803e940 con 0x7fc36015d770 2024-01-16T14:59:54.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.064+0000 7fc35cc00640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc33806b500 0x7fc33806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.064+0000 7fc35cc00640 1 -- 172.21.15.141:0/641859861 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc3580acad0 con 0x7fc36015d770 2024-01-16T14:59:54.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.065+0000 7fc365200640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc33806b500 0x7fc33806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.066+0000 7fc365200640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc33806b500 0x7fc33806d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc350006fd0 tx=0x7fc350008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.068+0000 7fc35cc00640 1 -- 172.21.15.141:0/641859861 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc358077970 con 0x7fc36015d770 2024-01-16T14:59:54.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.256+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc360070c30 con 0x7fc36015d770 2024-01-16T14:59:54.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.257+0000 7fc35cc00640 1 -- 172.21.15.141:0/641859861 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc3580430a0 con 0x7fc36015d770 2024-01-16T14:59:54.237 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:59:54.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.259+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc33806b500 msgr2=0x7fc33806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.259+0000 7fc366600640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc33806b500 0x7fc33806d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc350006fd0 tx=0x7fc350008040 comp rx=0 tx=0).stop 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.259+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 msgr2=0x7fc360138820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.259+0000 7fc366600640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc360138820 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc358036040 tx=0x7fc358004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 shutdown_connections 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc33806b500 0x7fc33806d9c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc36015d770 0x7fc360138820 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc360143810 0x7fc3601382e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 --2- 172.21.15.141:0/641859861 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3600ae170 0x7fc3601377e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 >> 172.21.15.141:0/641859861 conn(0x7fc36006a7d0 msgr2=0x7fc36015f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 shutdown_connections 2024-01-16T14:59:54.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.260+0000 7fc366600640 1 -- 172.21.15.141:0/641859861 wait complete. 2024-01-16T14:59:54.260 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:59:54.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.485+0000 7f8670200640 1 Processor -- start 2024-01-16T14:59:54.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.486+0000 7f8670200640 1 -- start start 2024-01-16T14:59:54.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.487+0000 7f8670200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f866813df80 0x7f866815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.487+0000 7f8670200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f866813e360 0x7f86680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.487+0000 7f8670200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866806c470 con 0x7f866815f9c0 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.487+0000 7f8670200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866806c750 con 0x7f866813df80 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.487+0000 7f8670200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866806ca30 con 0x7f866813e360 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.487+0000 7f866d800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f866813df80 0x7f866815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866d800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f866813df80 0x7f866815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46824/0 (socket says 172.21.15.141:46824) 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866e200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f866815f9c0 0x7f86681408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50714/0 (socket says 172.21.15.141:50714) 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866d800640 1 -- 172.21.15.141:0/3376427349 learned_addr learned my addr 172.21.15.141:0/3376427349 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866ce00640 1 --2- 172.21.15.141:0/3376427349 >> v2:172.21.15.188:3300/0 conn(0x7f866813e360 0x7f86680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866d800640 1 -- 172.21.15.141:0/3376427349 >> v2:172.21.15.188:3300/0 conn(0x7f866813e360 msgr2=0x7f86680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866d800640 1 --2- 172.21.15.141:0/3376427349 >> v2:172.21.15.188:3300/0 conn(0x7f866813e360 0x7f86680cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866d800640 1 -- 172.21.15.141:0/3376427349 >> v1:172.21.15.141:6789/0 conn(0x7f866815f9c0 legacy=0x7f86681408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866d800640 1 -- 172.21.15.141:0/3376427349 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866806cd10 con 0x7f866813df80 2024-01-16T14:59:54.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.488+0000 7f866ce00640 1 --2- 172.21.15.141:0/3376427349 >> v2:172.21.15.188:3300/0 conn(0x7f866813e360 0x7f86680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.489+0000 7f866d800640 1 --2- 172.21.15.141:0/3376427349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813df80 0x7f866815bee0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f866806ded0 tx=0x7f865802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6bd505512ec770 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.489+0000 7f8667e00640 1 -- 172.21.15.141:0/3376427349 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8658030870 con 0x7f866813df80 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.489+0000 7f8667e00640 1 -- 172.21.15.141:0/3376427349 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86580309d0 con 0x7f866813df80 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.489+0000 7f8667e00640 1 -- 172.21.15.141:0/3376427349 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8658030d60 con 0x7f866813df80 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 -- 172.21.15.141:0/3376427349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813df80 msgr2=0x7f866815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 --2- 172.21.15.141:0/3376427349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813df80 0x7f866815bee0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f866806ded0 tx=0x7f865802b9b0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 -- 172.21.15.141:0/3376427349 shutdown_connections 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 --2- 172.21.15.141:0/3376427349 >> v2:172.21.15.188:3300/0 conn(0x7f866813e360 0x7f86680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 --2- 172.21.15.141:0/3376427349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813df80 0x7f866815bee0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 -- 172.21.15.141:0/3376427349 >> 172.21.15.141:0/3376427349 conn(0x7f866806a800 msgr2=0x7f866806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 -- 172.21.15.141:0/3376427349 shutdown_connections 2024-01-16T14:59:54.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.490+0000 7f8670200640 1 -- 172.21.15.141:0/3376427349 wait complete. 2024-01-16T14:59:54.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.492+0000 7f8670200640 1 Processor -- start 2024-01-16T14:59:54.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.492+0000 7f8670200640 1 -- start start 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f8670200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f866813df80 0x7f86680b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f8670200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813e360 0x7f86680b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f8670200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f866815f9c0 0x7f866812f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f8670200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8668150a80 con 0x7f866813e360 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f8670200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86681300a0 con 0x7f866813df80 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f8670200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8668130380 con 0x7f866815f9c0 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f866ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813e360 0x7f86680b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f866ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813e360 0x7f86680b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46826/0 (socket says 172.21.15.141:46826) 2024-01-16T14:59:54.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.493+0000 7f866ce00640 1 -- 172.21.15.141:0/1379652875 learned_addr learned my addr 172.21.15.141:0/1379652875 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:54.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866d800640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f866813df80 0x7f86680b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866ce00640 1 -- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f866815f9c0 msgr2=0x7f866812f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866ce00640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f866815f9c0 0x7f866812f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866ce00640 1 -- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f866813df80 msgr2=0x7f86680b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866ce00640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f866813df80 0x7f86680b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866ce00640 1 -- 172.21.15.141:0/1379652875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580094d0 con 0x7f866813e360 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866d800640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f866813df80 0x7f86680b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f866ce00640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813e360 0x7f86680b3230 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f865000c7f0 tx=0x7f865000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f8665600640 1 -- 172.21.15.141:0/1379652875 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865000cea0 con 0x7f866813e360 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f8665600640 1 -- 172.21.15.141:0/1379652875 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8650004230 con 0x7f866813e360 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8668130660 con 0x7f866813e360 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f8665600640 1 -- 172.21.15.141:0/1379652875 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650007830 con 0x7f866813e360 2024-01-16T14:59:54.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.494+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8668130c20 con 0x7f866813e360 2024-01-16T14:59:54.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.496+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86680675d0 con 0x7f866813e360 2024-01-16T14:59:54.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.497+0000 7f8665600640 1 -- 172.21.15.141:0/1379652875 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f8650007990 con 0x7f866813e360 2024-01-16T14:59:54.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.498+0000 7f8665600640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864006b500 0x7f864006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.498+0000 7f8665600640 1 -- 172.21.15.141:0/1379652875 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8650089e90 con 0x7f866813e360 2024-01-16T14:59:54.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.498+0000 7f866d800640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864006b500 0x7f864006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.500+0000 7f866d800640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864006b500 0x7f864006d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8658036000 tx=0x7f865803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.502+0000 7f8665600640 1 -- 172.21.15.141:0/1379652875 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8650054af0 con 0x7f866813e360 2024-01-16T14:59:54.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.828+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f866806df00 con 0x7f866813e360 2024-01-16T14:59:54.809 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T14:59:54.810 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/176725955 2024-01-17T14:59:49.971845+0000 2024-01-16T14:59:54.810 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6833/176725955 2024-01-17T14:59:49.971845+0000 2024-01-16T14:59:54.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.828+0000 7f8665600640 1 -- 172.21.15.141:0/1379652875 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v38) v1 ==== 80+0+122 (secure 0 0 0) 0x7f865008b090 con 0x7f866813e360 2024-01-16T14:59:54.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.832+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864006b500 msgr2=0x7f864006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.832+0000 7f8670200640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864006b500 0x7f864006d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8658036000 tx=0x7f865803b040 comp rx=0 tx=0).stop 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.832+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813e360 msgr2=0x7f86680b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.832+0000 7f8670200640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813e360 0x7f86680b3230 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f865000c7f0 tx=0x7f865000ccb0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 shutdown_connections 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864006b500 0x7f864006d9c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f866815f9c0 0x7f866812f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f866813e360 0x7f86680b3230 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 --2- 172.21.15.141:0/1379652875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f866813df80 0x7f86680b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 >> 172.21.15.141:0/1379652875 conn(0x7f866806a800 msgr2=0x7f866815a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 shutdown_connections 2024-01-16T14:59:54.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:54.833+0000 7f8670200640 1 -- 172.21.15.141:0/1379652875 wait complete. 2024-01-16T14:59:54.839 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T14:59:54.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:54.839 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:54.908 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:59:54.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:54.909 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T14:59:54.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[49299]: fuse finished with error 0 and tester_r 0 2024-01-16T14:59:55.101 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:59:55.101 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:55.159 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:55.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.385+0000 7f449ce00640 1 Processor -- start 2024-01-16T14:59:55.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.385+0000 7f449ce00640 1 -- start start 2024-01-16T14:59:55.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f449ce00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4498143ae0 0x7f44980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f449ce00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f44980cf7d0 0x7f449815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f449ce00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f449815e1e0 0x7f449815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f449ce00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44980cfc30 con 0x7f44980cf7d0 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f449ce00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44980b1ee0 con 0x7f449815e1e0 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f449ce00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44980b2780 con 0x7f4498143ae0 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f4495400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f44980cf7d0 0x7f449815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f4495400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f44980cf7d0 0x7f449815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46838/0 (socket says 172.21.15.141:46838) 2024-01-16T14:59:55.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.387+0000 7f4495400640 1 -- 172.21.15.141:0/3183444697 learned_addr learned my addr 172.21.15.141:0/3183444697 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4495e00640 1 --2- 172.21.15.141:0/3183444697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4498143ae0 0x7f44980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4496800640 1 --2- 172.21.15.141:0/3183444697 >> v2:172.21.15.188:3300/0 conn(0x7f449815e1e0 0x7f449815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4495400640 1 -- 172.21.15.141:0/3183444697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4498143ae0 msgr2=0x7f44980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4495400640 1 --2- 172.21.15.141:0/3183444697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4498143ae0 0x7f44980cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4495400640 1 -- 172.21.15.141:0/3183444697 >> v2:172.21.15.188:3300/0 conn(0x7f449815e1e0 msgr2=0x7f449815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4495400640 1 --2- 172.21.15.141:0/3183444697 >> v2:172.21.15.188:3300/0 conn(0x7f449815e1e0 0x7f449815a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4495400640 1 -- 172.21.15.141:0/3183444697 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449815e990 con 0x7f44980cf7d0 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.388+0000 7f4495e00640 1 --2- 172.21.15.141:0/3183444697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4498143ae0 0x7f44980cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:55.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.389+0000 7f4495400640 1 --2- 172.21.15.141:0/3183444697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f449815dca0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4498072160 tx=0x7f448002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec12178b7f7065d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.389+0000 7f4494a00640 1 -- 172.21.15.141:0/3183444697 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480031870 con 0x7f44980cf7d0 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.389+0000 7f4494a00640 1 -- 172.21.15.141:0/3183444697 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44800319d0 con 0x7f44980cf7d0 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.389+0000 7f4494a00640 1 -- 172.21.15.141:0/3183444697 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480031d60 con 0x7f44980cf7d0 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 -- 172.21.15.141:0/3183444697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 msgr2=0x7f449815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 --2- 172.21.15.141:0/3183444697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f449815dca0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4498072160 tx=0x7f448002b910 comp rx=0 tx=0).stop 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 -- 172.21.15.141:0/3183444697 shutdown_connections 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 --2- 172.21.15.141:0/3183444697 >> v2:172.21.15.188:3300/0 conn(0x7f449815e1e0 0x7f449815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 --2- 172.21.15.141:0/3183444697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f449815dca0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 --2- 172.21.15.141:0/3183444697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4498143ae0 0x7f44980cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 -- 172.21.15.141:0/3183444697 >> 172.21.15.141:0/3183444697 conn(0x7f449806a800 msgr2=0x7f449806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 -- 172.21.15.141:0/3183444697 shutdown_connections 2024-01-16T14:59:55.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.390+0000 7f449ce00640 1 -- 172.21.15.141:0/3183444697 wait complete. 2024-01-16T14:59:55.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.392+0000 7f449ce00640 1 Processor -- start 2024-01-16T14:59:55.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.392+0000 7f449ce00640 1 -- start start 2024-01-16T14:59:55.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f449ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f4498140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f449ce00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4498143ae0 0x7f4498141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f449ce00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f449815e1e0 0x7f44981f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f449ce00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44981f18e0 con 0x7f44980cf7d0 2024-01-16T14:59:55.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f449ce00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4498141ba0 con 0x7f4498143ae0 2024-01-16T14:59:55.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f449ce00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44980aca30 con 0x7f449815e1e0 2024-01-16T14:59:55.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f4498140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f4498140e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46842/0 (socket says 172.21.15.141:46842) 2024-01-16T14:59:55.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 -- 172.21.15.141:0/3610121366 learned_addr learned my addr 172.21.15.141:0/3610121366 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:55.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495400640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4498143ae0 0x7f4498141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 -- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f449815e1e0 msgr2=0x7f44981f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4496800640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f449815e1e0 0x7f44981f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f449815e1e0 0x7f44981f0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 -- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4498143ae0 msgr2=0x7f4498141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4498143ae0 0x7f4498141360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.393+0000 7f4495e00640 1 -- 172.21.15.141:0/3610121366 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44800094d0 con 0x7f44980cf7d0 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.394+0000 7f4495e00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f4498140e20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f448800cee0 tx=0x7f448800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.394+0000 7f448e000640 1 -- 172.21.15.141:0/3610121366 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448800d840 con 0x7f44980cf7d0 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.394+0000 7f448e000640 1 -- 172.21.15.141:0/3610121366 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448800d9a0 con 0x7f44980cf7d0 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.394+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44980acd10 con 0x7f44980cf7d0 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.394+0000 7f448e000640 1 -- 172.21.15.141:0/3610121366 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448800dd30 con 0x7f44980cf7d0 2024-01-16T14:59:55.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.394+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44981418a0 con 0x7f44980cf7d0 2024-01-16T14:59:55.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.395+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44980ad230 con 0x7f44980cf7d0 2024-01-16T14:59:55.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.396+0000 7f448e000640 1 -- 172.21.15.141:0/3610121366 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4488015440 con 0x7f44980cf7d0 2024-01-16T14:59:55.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.397+0000 7f448e000640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f447006b530 0x7f447006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.397+0000 7f448e000640 1 -- 172.21.15.141:0/3610121366 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f448808a5f0 con 0x7f44980cf7d0 2024-01-16T14:59:55.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.398+0000 7f4495400640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f447006b530 0x7f447006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.399+0000 7f4495400640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f447006b530 0x7f447006d9f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4480036000 tx=0x7f448003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.401+0000 7f448e000640 1 -- 172.21.15.141:0/3610121366 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44980ad230 con 0x7f44980cf7d0 2024-01-16T14:59:55.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.592+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44980ad230 con 0x7f44980cf7d0 2024-01-16T14:59:55.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.594+0000 7f448e000640 1 -- 172.21.15.141:0/3610121366 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13034 (secure 0 0 0) 0x7f44980ad230 con 0x7f44980cf7d0 2024-01-16T14:59:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:55.576 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":38,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:59:52.987816+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/176725955":"2024-01-17T14:59:49.971845+0000","172.21.15.141:6833/176725955":"2024-01-17T14:59:49.971845+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:55.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f447006b530 msgr2=0x7f447006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f447006b530 0x7f447006d9f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4480036000 tx=0x7f448003c040 comp rx=0 tx=0).stop 2024-01-16T14:59:55.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 msgr2=0x7f4498140e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f4498140e20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f448800cee0 tx=0x7f448800a710 comp rx=0 tx=0).stop 2024-01-16T14:59:55.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 shutdown_connections 2024-01-16T14:59:55.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f447006b530 0x7f447006d9f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f449815e1e0 0x7f44981f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4498143ae0 0x7f4498141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 --2- 172.21.15.141:0/3610121366 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f44980cf7d0 0x7f4498140e20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 >> 172.21.15.141:0/3610121366 conn(0x7f449806a800 msgr2=0x7f44980b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 shutdown_connections 2024-01-16T14:59:55.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.597+0000 7f449ce00640 1 -- 172.21.15.141:0/3610121366 wait complete. 2024-01-16T14:59:55.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:55.596 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:55.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.824+0000 7f494ee00640 1 Processor -- start 2024-01-16T14:59:55.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.824+0000 7f494ee00640 1 -- start start 2024-01-16T14:59:55.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.826+0000 7f494ee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 0x7f49480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.826+0000 7f494ee00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494815c5f0 con 0x7f494813df80 2024-01-16T14:59:55.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.826+0000 7f494ee00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494806c470 con 0x7f494815f9c0 2024-01-16T14:59:55.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.826+0000 7f494ee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494806c750 con 0x7f494813e360 2024-01-16T14:59:55.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.826+0000 7f4947e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f494813df80 0x7f494815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50728/0 (socket says 172.21.15.141:50728) 2024-01-16T14:59:55.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.826+0000 7f4947e00640 1 -- 172.21.15.141:0/4112852027 learned_addr learned my addr 172.21.15.141:0/4112852027 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.826+0000 7f4947400640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 0x7f49480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828531154 0 0) 0x7f494815c5f0 con 0x7f494813df80 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4928003520 con 0x7f494813df80 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284016677 0 0) 0x7f4928003520 con 0x7f494813df80 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 msgr2=0x7f49480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 0x7f49480cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 >> v1:172.21.15.188:6789/0 conn(0x7f494815f9c0 legacy=0x7f49480cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494806ca30 con 0x7f494813df80 2024-01-16T14:59:55.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4938001670 con 0x7f494813df80 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 >> v1:172.21.15.141:6789/0 conn(0x7f494813df80 legacy=0x7f494815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49280039a0 0x7f4928003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 0x7f49280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4928004d00 0x7f4928005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494806c750 con 0x7f49280039a0 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492800ba90 con 0x7f4928004d00 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492800bc50 con 0x7f494813e360 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.827+0000 7f4947e00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49280039a0 0x7f4928003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.828+0000 7f4947400640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 0x7f49280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.828+0000 7f4947e00640 1 -- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 msgr2=0x7f49280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.828+0000 7f494ce00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4928004d00 0x7f4928005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.828+0000 7f4947e00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813e360 0x7f49280047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.828+0000 7f4947e00640 1 -- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4928004d00 msgr2=0x7f4928005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.828+0000 7f4947e00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4928004d00 0x7f4928005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.828+0000 7f4947e00640 1 -- 172.21.15.141:0/4112852027 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4928003520 con 0x7f49280039a0 2024-01-16T14:59:55.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.829+0000 7f4947e00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49280039a0 0x7f4928003dc0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4938001eb0 tx=0x7f49280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc32692c3297f58 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938005ad0 con 0x7f49280039a0 2024-01-16T14:59:55.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938005c30 con 0x7f49280039a0 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f4946a00640 1 -- 172.21.15.141:0/4112852027 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493802c7c0 con 0x7f49280039a0 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f494ee00640 1 -- 172.21.15.141:0/4112852027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49280039a0 msgr2=0x7f4928003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f494ee00640 1 --2- 172.21.15.141:0/4112852027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49280039a0 0x7f4928003dc0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4938001eb0 tx=0x7f49280057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f494ce00640 1 -- 172.21.15.141:0/4112852027 reap_dead start 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f494ee00640 1 -- 172.21.15.141:0/4112852027 shutdown_connections 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f494ee00640 1 -- 172.21.15.141:0/4112852027 >> 172.21.15.141:0/4112852027 conn(0x7f494806a800 msgr2=0x7f494806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f494ee00640 1 -- 172.21.15.141:0/4112852027 shutdown_connections 2024-01-16T14:59:55.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.830+0000 7f494ee00640 1 -- 172.21.15.141:0/4112852027 wait complete. 2024-01-16T14:59:55.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.832+0000 7f494ee00640 1 Processor -- start 2024-01-16T14:59:55.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.832+0000 7f494ee00640 1 -- start start 2024-01-16T14:59:55.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49280039a0 0x7f49481f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813a8e0 0x7f494813ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ee00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f494813b430 0x7f494813b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ee00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494813b2a0 con 0x7f494813b430 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494813cd00 con 0x7f49280039a0 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494813c0f0 con 0x7f494813a8e0 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f494813b430 0x7f494813b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f494813b430 0x7f494813b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46864/0 (socket says 172.21.15.141:46864) 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.833+0000 7f494ce00640 1 -- 172.21.15.141:0/4115917778 learned_addr learned my addr 172.21.15.141:0/4115917778 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f4947400640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813a8e0 0x7f494813ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f4947e00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49280039a0 0x7f49481f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f494ce00640 1 -- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813a8e0 msgr2=0x7f494813ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f494ce00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813a8e0 0x7f494813ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f494ce00640 1 -- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49280039a0 msgr2=0x7f49481f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f494ce00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49280039a0 0x7f49481f0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f494ce00640 1 -- 172.21.15.141:0/4115917778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494813c3d0 con 0x7f494813b430 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f494ce00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f494813b430 0x7f494813b8b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f494815b6b0 tx=0x7f494000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f493fe00640 1 -- 172.21.15.141:0/4115917778 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940003b70 con 0x7f494813b430 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f493fe00640 1 -- 172.21.15.141:0/4115917778 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4940003cd0 con 0x7f494813b430 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494813c6b0 con 0x7f494813b430 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.834+0000 7f493fe00640 1 -- 172.21.15.141:0/4115917778 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49400052b0 con 0x7f494813b430 2024-01-16T14:59:55.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.835+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494813bdf0 con 0x7f494813b430 2024-01-16T14:59:55.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.836+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49480675d0 con 0x7f494813b430 2024-01-16T14:59:55.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.836+0000 7f493fe00640 1 -- 172.21.15.141:0/4115917778 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4940005410 con 0x7f494813b430 2024-01-16T14:59:55.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.837+0000 7f493fe00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492006b530 0x7f492006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.837+0000 7f493fe00640 1 -- 172.21.15.141:0/4115917778 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f49400897c0 con 0x7f494813b430 2024-01-16T14:59:55.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.838+0000 7f4947e00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492006b530 0x7f492006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.839+0000 7f4947e00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492006b530 0x7f492006d9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4938034000 tx=0x7f4938009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:55.842+0000 7f493fe00640 1 -- 172.21.15.141:0/4115917778 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49400544a0 con 0x7f494813b430 2024-01-16T14:59:56.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.031+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f494806df00 con 0x7f494813b430 2024-01-16T14:59:56.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.032+0000 7f493fe00640 1 -- 172.21.15.141:0/4115917778 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4940018020 con 0x7f494813b430 2024-01-16T14:59:56.013 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T14:59:56.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492006b530 msgr2=0x7f492006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492006b530 0x7f492006d9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4938034000 tx=0x7f4938009030 comp rx=0 tx=0).stop 2024-01-16T14:59:56.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f494813b430 msgr2=0x7f494813b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f494813b430 0x7f494813b8b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f494815b6b0 tx=0x7f494000dc00 comp rx=0 tx=0).stop 2024-01-16T14:59:56.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 shutdown_connections 2024-01-16T14:59:56.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492006b530 0x7f492006d9f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f494813b430 0x7f494813b8b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494813a8e0 0x7f494813ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 --2- 172.21.15.141:0/4115917778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49280039a0 0x7f49481f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 >> 172.21.15.141:0/4115917778 conn(0x7f494806a800 msgr2=0x7f494806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:56.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 shutdown_connections 2024-01-16T14:59:56.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.035+0000 7f494ee00640 1 -- 172.21.15.141:0/4115917778 wait complete. 2024-01-16T14:59:56.036 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T14:59:56.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.267+0000 7f3bd2c00640 1 Processor -- start 2024-01-16T14:59:56.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.267+0000 7f3bd2c00640 1 -- start start 2024-01-16T14:59:56.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd2c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc13cfa0 0x7f3bcc15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd2c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3bcc0cd530 0x7f3bcc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd2c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bcc0cdec0 con 0x7f3bcc0cd9b0 2024-01-16T14:59:56.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd2c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bcc0ce1a0 con 0x7f3bcc0cd530 2024-01-16T14:59:56.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd2c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0ce480 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd0e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3bcc0cd530 0x7f3bcc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd0e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3bcc0cd530 0x7f3bcc13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46876/0 (socket says 172.21.15.141:46876) 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd0e00640 1 -- 172.21.15.141:0/3752874480 learned_addr learned my addr 172.21.15.141:0/3752874480 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd2200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3bcc0cd9b0 0x7f3bcc15d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50738/0 (socket says 172.21.15.141:50738) 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.269+0000 7f3bd1800640 1 --2- 172.21.15.141:0/3752874480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc13cfa0 0x7f3bcc15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.270+0000 7f3bd0e00640 1 -- 172.21.15.141:0/3752874480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc13cfa0 msgr2=0x7f3bcc15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.270+0000 7f3bd0e00640 1 --2- 172.21.15.141:0/3752874480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc13cfa0 0x7f3bcc15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.270+0000 7f3bd0e00640 1 -- 172.21.15.141:0/3752874480 >> v1:172.21.15.141:6789/0 conn(0x7f3bcc0cd9b0 legacy=0x7f3bcc15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.270+0000 7f3bd0e00640 1 -- 172.21.15.141:0/3752874480 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc0ce760 con 0x7f3bcc0cd530 2024-01-16T14:59:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.270+0000 7f3bcbe00640 1 -- 172.21.15.141:0/3752874480 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376922463 0 0) 0x7f3bcc0cdec0 con 0x7f3bcc0cd9b0 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.271+0000 7f3bd0e00640 1 --2- 172.21.15.141:0/3752874480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc0cd530 0x7f3bcc13f5a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009870 tx=0x7f3bb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa84050cd8f7b31 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.271+0000 7f3bcbe00640 1 -- 172.21.15.141:0/3752874480 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4031640 con 0x7f3bcc0cd530 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.271+0000 7f3bcbe00640 1 -- 172.21.15.141:0/3752874480 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb40317a0 con 0x7f3bcc0cd530 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.271+0000 7f3bcbe00640 1 -- 172.21.15.141:0/3752874480 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4031b30 con 0x7f3bcc0cd530 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 -- 172.21.15.141:0/3752874480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc0cd530 msgr2=0x7f3bcc13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/3752874480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc0cd530 0x7f3bcc13f5a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009870 tx=0x7f3bb402b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 -- 172.21.15.141:0/3752874480 shutdown_connections 2024-01-16T14:59:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/3752874480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc0cd530 0x7f3bcc13f5a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/3752874480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc13cfa0 0x7f3bcc15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 -- 172.21.15.141:0/3752874480 >> 172.21.15.141:0/3752874480 conn(0x7f3bcc06a800 msgr2=0x7f3bcc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 -- 172.21.15.141:0/3752874480 shutdown_connections 2024-01-16T14:59:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.272+0000 7f3bd2c00640 1 -- 172.21.15.141:0/3752874480 wait complete. 2024-01-16T14:59:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.274+0000 7f3bd2c00640 1 Processor -- start 2024-01-16T14:59:56.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.274+0000 7f3bd2c00640 1 -- start start 2024-01-16T14:59:56.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc0cd530 0x7f3bcc1f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bcc0cd9b0 0x7f3bcc159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc13cfa0 0x7f3bcc14f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc1588c0 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc158ba0 con 0x7f3bcc0cd9b0 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc158e80 con 0x7f3bcc0cd530 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc13cfa0 0x7f3bcc14f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc13cfa0 0x7f3bcc14f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46884/0 (socket says 172.21.15.141:46884) 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.275+0000 7f3bd2200640 1 -- 172.21.15.141:0/2805170362 learned_addr learned my addr 172.21.15.141:0/2805170362 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd1800640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc0cd530 0x7f3bcc1f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd0e00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bcc0cd9b0 0x7f3bcc159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd2200640 1 -- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc0cd530 msgr2=0x7f3bcc1f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd2200640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc0cd530 0x7f3bcc1f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd2200640 1 -- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bcc0cd9b0 msgr2=0x7f3bcc159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd2200640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bcc0cd9b0 0x7f3bcc159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd2200640 1 -- 172.21.15.141:0/2805170362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40094d0 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd0e00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bcc0cd9b0 0x7f3bcc159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd2200640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc13cfa0 0x7f3bcc14f340 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4001560 tx=0x7f3bc400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bc9600640 1 -- 172.21.15.141:0/2805170362 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc400a640 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bc9600640 1 -- 172.21.15.141:0/2805170362 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc400a7a0 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bc9600640 1 -- 172.21.15.141:0/2805170362 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc400ab30 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.276+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bcc159160 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.277+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bcc1f33e0 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.278+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bcc1f35d0 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.278+0000 7f3bc9600640 1 -- 172.21.15.141:0/2805170362 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f3bc400ac90 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.279+0000 7f3bc9600640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ba406b530 0x7f3ba406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.279+0000 7f3bc9600640 1 -- 172.21.15.141:0/2805170362 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3bc40896a0 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.280+0000 7f3bd1800640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ba406b530 0x7f3ba406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.281+0000 7f3bd1800640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ba406b530 0x7f3ba406d9f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc006fd0 tx=0x7f3bbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:56.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.284+0000 7f3bc9600640 1 -- 172.21.15.141:0/2805170362 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bc4054380 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.487+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3bcc067450 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.488+0000 7f3bc9600640 1 -- 172.21.15.141:0/2805170362 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v38) v1 ==== 80+0+122 (secure 0 0 0) 0x7f3bc4017020 con 0x7f3bcc13cfa0 2024-01-16T14:59:56.469 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/176725955 2024-01-17T14:59:49.971845+0000 2024-01-16T14:59:56.469 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6833/176725955 2024-01-17T14:59:49.971845+0000 2024-01-16T14:59:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T14:59:56.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.490+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ba406b530 msgr2=0x7f3ba406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.490+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ba406b530 0x7f3ba406d9f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc006fd0 tx=0x7f3bbc008040 comp rx=0 tx=0).stop 2024-01-16T14:59:56.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.490+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc13cfa0 msgr2=0x7f3bcc14f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.490+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc13cfa0 0x7f3bcc14f340 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4001560 tx=0x7f3bc400cc70 comp rx=0 tx=0).stop 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 shutdown_connections 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ba406b530 0x7f3ba406d9f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bcc13cfa0 0x7f3bcc14f340 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bcc0cd9b0 0x7f3bcc159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 --2- 172.21.15.141:0/2805170362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bcc0cd530 0x7f3bcc1f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 >> 172.21.15.141:0/2805170362 conn(0x7f3bcc06a800 msgr2=0x7f3bcc13c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 shutdown_connections 2024-01-16T14:59:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:56.491+0000 7f3bd2c00640 1 -- 172.21.15.141:0/2805170362 wait complete. 2024-01-16T14:59:56.490 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T14:59:56.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:56.490 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:56.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[42607]: fuse finished with error 0 and tester_r 0 2024-01-16T14:59:56.596 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:59:56.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:56.596 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T14:59:56.784 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T14:59:56.785 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:56.843 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T14:59:56.843 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T14:59:56.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:56.843 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T14:59:56.897 INFO:teuthology.orchestra.run.smithi141.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T14:59:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:59:56.899 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T14:59:56.899 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T14:59:56.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T14:59:56.956 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T14:59:56.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:56.956 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T14:59:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:59:57.009 INFO:teuthology.orchestra.run.smithi188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T14:59:57.009 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:57.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.244+0000 7fb3baa00640 1 Processor -- start 2024-01-16T14:59:57.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.244+0000 7fb3baa00640 1 -- start start 2024-01-16T14:59:57.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.246+0000 7fb3baa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb3b40cf9a0 0x7fb3b4166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.246+0000 7fb3baa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b415c5f0 con 0x7fb3b413e360 2024-01-16T14:59:57.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.246+0000 7fb3baa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b406c470 con 0x7fb3b413df80 2024-01-16T14:59:57.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.246+0000 7fb3baa00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b406c750 con 0x7fb3b40cf9a0 2024-01-16T14:59:57.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.246+0000 7fb3ab200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb3b413e360 0x7fb3b40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50750/0 (socket says 172.21.15.141:50750) 2024-01-16T14:59:57.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.246+0000 7fb3ab200640 1 -- 172.21.15.141:0/2045022598 learned_addr learned my addr 172.21.15.141:0/2045022598 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T14:59:57.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b8a00640 1 --2- 172.21.15.141:0/2045022598 >> v2:172.21.15.188:3300/0 conn(0x7fb3b40cf9a0 0x7fb3b4166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062471242 0 0) 0x7fb3b415c5f0 con 0x7fb3b413e360 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394003520 con 0x7fb3b413e360 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1557681918 0 0) 0x7fb394003520 con 0x7fb3b413e360 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 >> v2:172.21.15.188:3300/0 conn(0x7fb3b40cf9a0 msgr2=0x7fb3b4166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 --2- 172.21.15.141:0/2045022598 >> v2:172.21.15.188:3300/0 conn(0x7fb3b40cf9a0 0x7fb3b4166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 >> v1:172.21.15.188:6789/0 conn(0x7fb3b413df80 legacy=0x7fb3b415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b406ca30 con 0x7fb3b413e360 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb39c001670 con 0x7fb3b413e360 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 >> v1:172.21.15.141:6789/0 conn(0x7fb3b413e360 legacy=0x7fb3b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb394003970 0x7fb394003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3940047c0 0x7fb394004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb39400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b406c750 con 0x7fb394005180 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400c000 con 0x7fb3940047c0 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.247+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400c1c0 con 0x7fb394003970 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3b8a00640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb39400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3b3e00640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb394003970 0x7fb394003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3b8a00640 1 -- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb394003970 msgr2=0x7fb394003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3ab200640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3940047c0 0x7fb394004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3b8a00640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb394003970 0x7fb394003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3b8a00640 1 -- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3940047c0 msgr2=0x7fb394004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3b8a00640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3940047c0 0x7fb394004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3b8a00640 1 -- 172.21.15.141:0/2045022598 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394003520 con 0x7fb394005180 2024-01-16T14:59:57.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.248+0000 7fb3ab200640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3940047c0 0x7fb394004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:57.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.249+0000 7fb3b8a00640 1 -- 172.21.15.141:0/2045022598 reap_dead start 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.249+0000 7fb3b8a00640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb39400b980 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009f90 tx=0x7fb3940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9957f99c5a7662 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.249+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac02cc20 con 0x7fb394005180 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.249+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac00f040 con 0x7fb394005180 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.249+0000 7fb3b3400640 1 -- 172.21.15.141:0/2045022598 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac003990 con 0x7fb394005180 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.250+0000 7fb3baa00640 1 -- 172.21.15.141:0/2045022598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 msgr2=0x7fb39400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.250+0000 7fb3baa00640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb39400b980 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009f90 tx=0x7fb3940057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.250+0000 7fb3baa00640 1 -- 172.21.15.141:0/2045022598 shutdown_connections 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.250+0000 7fb3baa00640 1 --2- 172.21.15.141:0/2045022598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb39400b980 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.250+0000 7fb3baa00640 1 -- 172.21.15.141:0/2045022598 >> 172.21.15.141:0/2045022598 conn(0x7fb3b406a800 msgr2=0x7fb3b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.250+0000 7fb3baa00640 1 -- 172.21.15.141:0/2045022598 shutdown_connections 2024-01-16T14:59:57.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.250+0000 7fb3baa00640 1 -- 172.21.15.141:0/2045022598 wait complete. 2024-01-16T14:59:57.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.252+0000 7fb3baa00640 1 Processor -- start 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.252+0000 7fb3baa00640 1 -- start start 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3baa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb3b414a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3baa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3b40b3e90 0x7fb3b413d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3baa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3b41f2e40 0x7fb3b415a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3baa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b40b4330 con 0x7fb394005180 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3baa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b413de10 con 0x7fb3b41f2e40 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3baa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b414a910 con 0x7fb3b40b3e90 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3b3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb3b414a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3b3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb3b414a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46894/0 (socket says 172.21.15.141:46894) 2024-01-16T14:59:57.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3b3e00640 1 -- 172.21.15.141:0/1029588286 learned_addr learned my addr 172.21.15.141:0/1029588286 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:57.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3b3e00640 1 -- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3b40b3e90 msgr2=0x7fb3b413d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3b3e00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3b40b3e90 0x7fb3b413d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.253+0000 7fb3b3e00640 1 -- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3b41f2e40 msgr2=0x7fb3b415a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3b3e00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3b41f2e40 0x7fb3b415a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3b3e00640 1 -- 172.21.15.141:0/1029588286 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b414abf0 con 0x7fb394005180 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3b3e00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb3b414a1f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400d4e0 tx=0x7fb3a400d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3b0c00640 1 -- 172.21.15.141:0/1029588286 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4012800 con 0x7fb394005180 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3b0c00640 1 -- 172.21.15.141:0/1029588286 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a4012960 con 0x7fb394005180 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b414aed0 con 0x7fb394005180 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3b0c00640 1 -- 172.21.15.141:0/1029588286 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4012d10 con 0x7fb394005180 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.254+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b4149e60 con 0x7fb394005180 2024-01-16T14:59:57.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.255+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b414b3f0 con 0x7fb394005180 2024-01-16T14:59:57.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.257+0000 7fb3b0c00640 1 -- 172.21.15.141:0/1029588286 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb3a4019360 con 0x7fb394005180 2024-01-16T14:59:57.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.257+0000 7fb3b0c00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb38c06b530 0x7fb38c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.258+0000 7fb3b0c00640 1 -- 172.21.15.141:0/1029588286 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb3a4089ef0 con 0x7fb394005180 2024-01-16T14:59:57.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.258+0000 7fb3ab200640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb38c06b530 0x7fb38c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.259+0000 7fb3ab200640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb38c06b530 0x7fb38c06d9f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0018e0 tx=0x7fb39c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.262+0000 7fb3b0c00640 1 -- 172.21.15.141:0/1029588286 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3a4054b50 con 0x7fb394005180 2024-01-16T14:59:57.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.490+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3b414aa50 con 0x7fb394005180 2024-01-16T14:59:57.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.491+0000 7fb3b0c00640 1 -- 172.21.15.141:0/1029588286 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3711 (secure 0 0 0) 0x7fb3b414aa50 con 0x7fb394005180 2024-01-16T14:59:57.472 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:57.472 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/2661986774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":2661986774},{"type":"v1","addr":"172.21.15.188:6836","nonce":2661986774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4586,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2344909439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2344909439},{"type":"v1","addr":"172.21.15.188:6837","nonce":2344909439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/1479821799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":1479821799},{"type":"v1","addr":"172.21.15.141:6834","nonce":1479821799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5691,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/4229845994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":4229845994},{"type":"v1","addr":"172.21.15.141:6835","nonce":4229845994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-01-16T14:59:57.472 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 20 2024-01-16T14:59:57.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb38c06b530 msgr2=0x7fb38c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb38c06b530 0x7fb38c06d9f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0018e0 tx=0x7fb39c009040 comp rx=0 tx=0).stop 2024-01-16T14:59:57.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 msgr2=0x7fb3b414a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb3b414a1f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400d4e0 tx=0x7fb3a400d9a0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 shutdown_connections 2024-01-16T14:59:57.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb38c06b530 0x7fb38c06d9f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3b41f2e40 0x7fb3b415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3b40b3e90 0x7fb3b413d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 --2- 172.21.15.141:0/1029588286 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb394005180 0x7fb3b414a1f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.494+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 >> 172.21.15.141:0/1029588286 conn(0x7fb3b406a800 msgr2=0x7fb3b415cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.495+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 shutdown_connections 2024-01-16T14:59:57.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.495+0000 7fb3baa00640 1 -- 172.21.15.141:0/1029588286 wait complete. 2024-01-16T14:59:57.494 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T14:59:57.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.720+0000 7f64df200640 1 Processor -- start 2024-01-16T14:59:57.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.720+0000 7f64df200640 1 -- start start 2024-01-16T14:59:57.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64df200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64df200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f64d815d300 0x7f64d815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64df200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64d80b2bd0 0x7f64d80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64df200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d815dc80 con 0x7f64d80b2bd0 2024-01-16T14:59:57.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64df200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d8071190 con 0x7f64d815d300 2024-01-16T14:59:57.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64df200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d8071470 con 0x7f64d8150760 2024-01-16T14:59:57.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64dd200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64d80b2bd0 0x7f64d80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64dd200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64d80b2bd0 0x7f64d80b3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46904/0 (socket says 172.21.15.141:46904) 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64dd200640 1 -- 172.21.15.141:0/2496000268 learned_addr learned my addr 172.21.15.141:0/2496000268 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64d7400640 1 --2- 172.21.15.141:0/2496000268 >> v2:172.21.15.188:3300/0 conn(0x7f64d815d300 0x7f64d815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.722+0000 7f64d7e00640 1 --2- 172.21.15.141:0/2496000268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.723+0000 7f64dd200640 1 -- 172.21.15.141:0/2496000268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 msgr2=0x7f64d8072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.723+0000 7f64dd200640 1 --2- 172.21.15.141:0/2496000268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.723+0000 7f64dd200640 1 -- 172.21.15.141:0/2496000268 >> v2:172.21.15.188:3300/0 conn(0x7f64d815d300 msgr2=0x7f64d815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.723+0000 7f64dd200640 1 --2- 172.21.15.141:0/2496000268 >> v2:172.21.15.188:3300/0 conn(0x7f64d815d300 0x7f64d815d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.723+0000 7f64dd200640 1 -- 172.21.15.141:0/2496000268 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d8071750 con 0x7f64d80b2bd0 2024-01-16T14:59:57.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.724+0000 7f64dd200640 1 --2- 172.21.15.141:0/2496000268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d80b3060 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f64d806e0c0 tx=0x7f64d002cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef4627681077b28 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.724+0000 7f64d6a00640 1 -- 172.21.15.141:0/2496000268 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d002db90 con 0x7f64d80b2bd0 2024-01-16T14:59:57.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.724+0000 7f64d6a00640 1 -- 172.21.15.141:0/2496000268 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d002dcf0 con 0x7f64d80b2bd0 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.724+0000 7f64d6a00640 1 -- 172.21.15.141:0/2496000268 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0035660 con 0x7f64d80b2bd0 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 -- 172.21.15.141:0/2496000268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 msgr2=0x7f64d80b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 --2- 172.21.15.141:0/2496000268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d80b3060 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f64d806e0c0 tx=0x7f64d002cde0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 -- 172.21.15.141:0/2496000268 shutdown_connections 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 --2- 172.21.15.141:0/2496000268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d80b3060 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 --2- 172.21.15.141:0/2496000268 >> v2:172.21.15.188:3300/0 conn(0x7f64d815d300 0x7f64d815d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 --2- 172.21.15.141:0/2496000268 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 -- 172.21.15.141:0/2496000268 >> 172.21.15.141:0/2496000268 conn(0x7f64d806a800 msgr2=0x7f64d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 -- 172.21.15.141:0/2496000268 shutdown_connections 2024-01-16T14:59:57.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.725+0000 7f64df200640 1 -- 172.21.15.141:0/2496000268 wait complete. 2024-01-16T14:59:57.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.727+0000 7f64df200640 1 Processor -- start 2024-01-16T14:59:57.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.727+0000 7f64df200640 1 -- start start 2024-01-16T14:59:57.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64df200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d8147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64df200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64df200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64d815d300 0x7f64d8159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64df200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80cda00 con 0x7f64d80b2bd0 2024-01-16T14:59:57.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64df200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d8147f40 con 0x7f64d815d300 2024-01-16T14:59:57.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64df200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80cd060 con 0x7f64d8150760 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d8147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d8147a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46914/0 (socket says 172.21.15.141:46914) 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64d7e00640 1 -- 172.21.15.141:0/932466011 learned_addr learned my addr 172.21.15.141:0/932466011 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64d7400640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.728+0000 7f64d7e00640 1 -- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 msgr2=0x7f64d8158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64d7e00640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8158f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64d7e00640 1 -- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64d815d300 msgr2=0x7f64d8159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64d7e00640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64d815d300 0x7f64d8159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64d7e00640 1 -- 172.21.15.141:0/932466011 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d000a670 con 0x7f64d80b2bd0 2024-01-16T14:59:57.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64d7e00640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d8147a00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f64c000d800 tx=0x7f64c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64cfe00640 1 -- 172.21.15.141:0/932466011 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c00096a0 con 0x7f64d80b2bd0 2024-01-16T14:59:57.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64cfe00640 1 -- 172.21.15.141:0/932466011 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c0009800 con 0x7f64d80b2bd0 2024-01-16T14:59:57.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64cfe00640 1 -- 172.21.15.141:0/932466011 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c0009b90 con 0x7f64d80b2bd0 2024-01-16T14:59:57.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d80cd340 con 0x7f64d80b2bd0 2024-01-16T14:59:57.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.729+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d815b520 con 0x7f64d80b2bd0 2024-01-16T14:59:57.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.730+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d80675d0 con 0x7f64d80b2bd0 2024-01-16T14:59:57.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.732+0000 7f64cfe00640 1 -- 172.21.15.141:0/932466011 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f64c0009cf0 con 0x7f64d80b2bd0 2024-01-16T14:59:57.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.733+0000 7f64cfe00640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64b006b530 0x7f64b006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.733+0000 7f64cfe00640 1 -- 172.21.15.141:0/932466011 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f64c008a1f0 con 0x7f64d80b2bd0 2024-01-16T14:59:57.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.734+0000 7f64d7400640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64b006b530 0x7f64b006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.735+0000 7f64d7400640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64b006b530 0x7f64b006d9f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f64d81574b0 tx=0x7f64c800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.738+0000 7f64cfe00640 1 -- 172.21.15.141:0/932466011 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64c0054f70 con 0x7f64d80b2bd0 2024-01-16T14:59:57.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.927+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f64d8067d00 con 0x7f64d80b2bd0 2024-01-16T14:59:57.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.932+0000 7f64cfe00640 1 -- 172.21.15.141:0/932466011 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v12) v1 ==== 120+0+0 (secure 0 0 0) 0x7f64c000b440 con 0x7f64d80b2bd0 2024-01-16T14:59:57.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64b006b530 msgr2=0x7f64b006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64b006b530 0x7f64b006d9f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f64d81574b0 tx=0x7f64c800b040 comp rx=0 tx=0).stop 2024-01-16T14:59:57.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 msgr2=0x7f64d8147a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d8147a00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f64c000d800 tx=0x7f64c000dcc0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 shutdown_connections 2024-01-16T14:59:57.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64b006b530 0x7f64b006d9f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64d815d300 0x7f64d8159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64d8150760 0x7f64d8158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 --2- 172.21.15.141:0/932466011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64d80b2bd0 0x7f64d8147a00 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.935+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 >> 172.21.15.141:0/932466011 conn(0x7f64d806a800 msgr2=0x7f64d814f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.936+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 shutdown_connections 2024-01-16T14:59:57.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:57.936+0000 7f64df200640 1 -- 172.21.15.141:0/932466011 wait complete. 2024-01-16T14:59:57.936 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T14:59:58.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.165+0000 7fb1cf000640 1 Processor -- start 2024-01-16T14:59:58.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.165+0000 7fb1cf000640 1 -- start start 2024-01-16T14:59:58.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.166+0000 7fb1cf000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb1c813e360 0x7fb1c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.166+0000 7fb1cf000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c815f9c0 0x7fb1c80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.166+0000 7fb1cf000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c815c5f0 con 0x7fb1c813df80 2024-01-16T14:59:58.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.166+0000 7fb1cf000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c806c470 con 0x7fb1c813e360 2024-01-16T14:59:58.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.166+0000 7fb1cf000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c806c750 con 0x7fb1c815f9c0 2024-01-16T14:59:58.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.167+0000 7fb1c7400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb1c813e360 0x7fb1c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.167+0000 7fb1cd000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c815f9c0 0x7fb1c80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.167+0000 7fb1c7400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb1c813e360 0x7fb1c80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:49334/0 (socket says 172.21.15.141:49334) 2024-01-16T14:59:58.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.167+0000 7fb1c7400640 1 -- 172.21.15.141:0/4243202267 learned_addr learned my addr 172.21.15.141:0/4243202267 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:58.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.168+0000 7fb1cd000640 1 -- 172.21.15.141:0/4243202267 >> v2:172.21.15.188:3300/0 conn(0x7fb1c813e360 msgr2=0x7fb1c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.168+0000 7fb1cd000640 1 --2- 172.21.15.141:0/4243202267 >> v2:172.21.15.188:3300/0 conn(0x7fb1c813e360 0x7fb1c80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.168+0000 7fb1cd000640 1 -- 172.21.15.141:0/4243202267 >> v1:172.21.15.188:6789/0 conn(0x7fb1c813df80 legacy=0x7fb1c815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.168+0000 7fb1cd000640 1 -- 172.21.15.141:0/4243202267 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c806ca30 con 0x7fb1c815f9c0 2024-01-16T14:59:58.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.168+0000 7fb1c7400640 1 --2- 172.21.15.141:0/4243202267 >> v2:172.21.15.188:3300/0 conn(0x7fb1c813e360 0x7fb1c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:58.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.169+0000 7fb1cd000640 1 --2- 172.21.15.141:0/4243202267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c815f9c0 0x7fb1c80cf7d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009d90 tx=0x7fb1c002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=869c957809c1f24f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.170+0000 7fb1c6a00640 1 -- 172.21.15.141:0/4243202267 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c002cc60 con 0x7fb1c815f9c0 2024-01-16T14:59:58.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.170+0000 7fb1c6a00640 1 -- 172.21.15.141:0/4243202267 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c002cdc0 con 0x7fb1c815f9c0 2024-01-16T14:59:58.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.170+0000 7fb1c6a00640 1 -- 172.21.15.141:0/4243202267 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0034660 con 0x7fb1c815f9c0 2024-01-16T14:59:58.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.170+0000 7fb1cf000640 1 -- 172.21.15.141:0/4243202267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c815f9c0 msgr2=0x7fb1c80cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.170+0000 7fb1cf000640 1 --2- 172.21.15.141:0/4243202267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c815f9c0 0x7fb1c80cf7d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009d90 tx=0x7fb1c002beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.171+0000 7fb1cf000640 1 -- 172.21.15.141:0/4243202267 shutdown_connections 2024-01-16T14:59:58.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.171+0000 7fb1cf000640 1 --2- 172.21.15.141:0/4243202267 >> v2:172.21.15.188:3300/0 conn(0x7fb1c813e360 0x7fb1c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.171+0000 7fb1cf000640 1 --2- 172.21.15.141:0/4243202267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c815f9c0 0x7fb1c80cf7d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.171+0000 7fb1cf000640 1 -- 172.21.15.141:0/4243202267 >> 172.21.15.141:0/4243202267 conn(0x7fb1c806a800 msgr2=0x7fb1c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.171+0000 7fb1cf000640 1 -- 172.21.15.141:0/4243202267 shutdown_connections 2024-01-16T14:59:58.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.171+0000 7fb1cf000640 1 -- 172.21.15.141:0/4243202267 wait complete. 2024-01-16T14:59:58.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.172+0000 7fb1cf000640 1 Processor -- start 2024-01-16T14:59:58.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.173+0000 7fb1cf000640 1 -- start start 2024-01-16T14:59:58.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.173+0000 7fb1cf000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1c813df80 0x7fb1c80b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.173+0000 7fb1cf000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c813e360 0x7fb1c80a88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.173+0000 7fb1cf000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1c815f9c0 0x7fb1c813d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.173+0000 7fb1cf000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c81527d0 con 0x7fb1c813df80 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.173+0000 7fb1cf000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c8152a90 con 0x7fb1c815f9c0 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.173+0000 7fb1cf000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c8152d70 con 0x7fb1c813e360 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1c813df80 0x7fb1c80b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1c813df80 0x7fb1c80b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46920/0 (socket says 172.21.15.141:46920) 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 -- 172.21.15.141:0/3391166791 learned_addr learned my addr 172.21.15.141:0/3391166791 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1cd000640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1c815f9c0 0x7fb1c813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7400640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c813e360 0x7fb1c80a88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 -- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c813e360 msgr2=0x7fb1c80a88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c813e360 0x7fb1c80a88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 -- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1c815f9c0 msgr2=0x7fb1c813d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1c815f9c0 0x7fb1c813d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7e00640 1 -- 172.21.15.141:0/3391166791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b00094d0 con 0x7fb1c813df80 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.174+0000 7fb1c7400640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c813e360 0x7fb1c80a88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.175+0000 7fb1c7e00640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1c813df80 0x7fb1c80b2f60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c340 tx=0x7fb1b800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.175+0000 7fb1bfe00640 1 -- 172.21.15.141:0/3391166791 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80052d0 con 0x7fb1c813df80 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.175+0000 7fb1bfe00640 1 -- 172.21.15.141:0/3391166791 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8005430 con 0x7fb1c813df80 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.175+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c0009a70 con 0x7fb1c813df80 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.175+0000 7fb1bfe00640 1 -- 172.21.15.141:0/3391166791 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8003850 con 0x7fb1c813df80 2024-01-16T14:59:58.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.175+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c8153330 con 0x7fb1c813df80 2024-01-16T14:59:58.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.176+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c8155420 con 0x7fb1c813df80 2024-01-16T14:59:58.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.177+0000 7fb1bfe00640 1 -- 172.21.15.141:0/3391166791 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb1b80039b0 con 0x7fb1c813df80 2024-01-16T14:59:58.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.178+0000 7fb1bfe00640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1a006b530 0x7fb1a006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.178+0000 7fb1bfe00640 1 -- 172.21.15.141:0/3391166791 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb1b80891c0 con 0x7fb1c813df80 2024-01-16T14:59:58.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.178+0000 7fb1c7400640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1a006b530 0x7fb1a006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.179+0000 7fb1c7400640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1a006b530 0x7fb1a006d9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80aa4d0 tx=0x7fb1b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.182+0000 7fb1bfe00640 1 -- 172.21.15.141:0/3391166791 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1b8054000 con 0x7fb1c813df80 2024-01-16T14:59:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.374+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb1c80675d0 con 0x7fb1c813df80 2024-01-16T14:59:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.382+0000 7fb1bfe00640 1 -- 172.21.15.141:0/3391166791 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb1b8057620 con 0x7fb1c813df80 2024-01-16T14:59:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1a006b530 msgr2=0x7fb1a006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1a006b530 0x7fb1a006d9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80aa4d0 tx=0x7fb1b0008040 comp rx=0 tx=0).stop 2024-01-16T14:59:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1c813df80 msgr2=0x7fb1c80b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1c813df80 0x7fb1c80b2f60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c340 tx=0x7fb1b800c800 comp rx=0 tx=0).stop 2024-01-16T14:59:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 shutdown_connections 2024-01-16T14:59:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1a006b530 0x7fb1a006d9f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1c815f9c0 0x7fb1c813d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb1c813e360 0x7fb1c80a88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 --2- 172.21.15.141:0/3391166791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1c813df80 0x7fb1c80b2f60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 >> 172.21.15.141:0/3391166791 conn(0x7fb1c806a800 msgr2=0x7fb1c806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 shutdown_connections 2024-01-16T14:59:58.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.385+0000 7fb1cf000640 1 -- 172.21.15.141:0/3391166791 wait complete. 2024-01-16T14:59:58.384 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T14:59:58.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.605+0000 7f4e16200640 1 Processor -- start 2024-01-16T14:59:58.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.605+0000 7f4e16200640 1 -- start start 2024-01-16T14:59:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e16200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4e1015b030 0x7f4e100b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e16200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e100b31b0 0x7f4e100b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e16200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e100b3610 con 0x7f4e100b5380 2024-01-16T14:59:58.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e16200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e100d0780 con 0x7f4e100b31b0 2024-01-16T14:59:58.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e16200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e100cf8f0 con 0x7f4e1015b030 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e0ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e100b31b0 0x7f4e100b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e0ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e100b31b0 0x7f4e100b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46926/0 (socket says 172.21.15.141:46926) 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.607+0000 7f4e0ea00640 1 -- 172.21.15.141:0/4294924031 learned_addr learned my addr 172.21.15.141:0/4294924031 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.608+0000 7f4e0f400640 1 --2- 172.21.15.141:0/4294924031 >> v2:172.21.15.188:3300/0 conn(0x7f4e1015b030 0x7f4e100b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.608+0000 7f4e0ea00640 1 -- 172.21.15.141:0/4294924031 >> v2:172.21.15.188:3300/0 conn(0x7f4e1015b030 msgr2=0x7f4e100b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.608+0000 7f4e0ea00640 1 --2- 172.21.15.141:0/4294924031 >> v2:172.21.15.188:3300/0 conn(0x7f4e1015b030 0x7f4e100b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.608+0000 7f4e0ea00640 1 -- 172.21.15.141:0/4294924031 >> v1:172.21.15.188:6789/0 conn(0x7f4e100b5380 legacy=0x7f4e1015ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.608+0000 7f4e0ea00640 1 -- 172.21.15.141:0/4294924031 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100b5810 con 0x7f4e100b31b0 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.609+0000 7f4e0ea00640 1 --2- 172.21.15.141:0/4294924031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e100b31b0 0x7f4e100b4e40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4e100b61d0 tx=0x7f4df802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0b6085d9afdbb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.609+0000 7f4e0e000640 1 -- 172.21.15.141:0/4294924031 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8031870 con 0x7f4e100b31b0 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.609+0000 7f4e0e000640 1 -- 172.21.15.141:0/4294924031 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df80319d0 con 0x7f4e100b31b0 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.609+0000 7f4e0e000640 1 -- 172.21.15.141:0/4294924031 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8031d60 con 0x7f4e100b31b0 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 -- 172.21.15.141:0/4294924031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e100b31b0 msgr2=0x7f4e100b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 --2- 172.21.15.141:0/4294924031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e100b31b0 0x7f4e100b4e40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4e100b61d0 tx=0x7f4df802b910 comp rx=0 tx=0).stop 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 -- 172.21.15.141:0/4294924031 shutdown_connections 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 --2- 172.21.15.141:0/4294924031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e100b31b0 0x7f4e100b4e40 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 --2- 172.21.15.141:0/4294924031 >> v2:172.21.15.188:3300/0 conn(0x7f4e1015b030 0x7f4e100b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 -- 172.21.15.141:0/4294924031 >> 172.21.15.141:0/4294924031 conn(0x7f4e1006a3e0 msgr2=0x7f4e1006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 -- 172.21.15.141:0/4294924031 shutdown_connections 2024-01-16T14:59:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.610+0000 7f4e16200640 1 -- 172.21.15.141:0/4294924031 wait complete. 2024-01-16T14:59:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.612+0000 7f4e16200640 1 Processor -- start 2024-01-16T14:59:58.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.612+0000 7f4e16200640 1 -- start start 2024-01-16T14:59:58.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.613+0000 7f4e16200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e100b31b0 0x7f4e100cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.613+0000 7f4e16200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e100b5380 0x7f4e1014cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.613+0000 7f4e16200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e1015b030 0x7f4e1015e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.613+0000 7f4e16200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100cbde0 con 0x7f4e1015b030 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.613+0000 7f4e16200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1014d090 con 0x7f4e100b31b0 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.613+0000 7f4e16200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e101f32d0 con 0x7f4e100b5380 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e1015b030 0x7f4e1015e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e1015b030 0x7f4e1015e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46928/0 (socket says 172.21.15.141:46928) 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 -- 172.21.15.141:0/2056663604 learned_addr learned my addr 172.21.15.141:0/2056663604 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0f400640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e100b31b0 0x7f4e100cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0ea00640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e100b5380 0x7f4e1014cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 -- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e100b5380 msgr2=0x7f4e1014cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e100b5380 0x7f4e1014cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 -- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e100b31b0 msgr2=0x7f4e100cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e100b31b0 0x7f4e100cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.614+0000 7f4e0fe00640 1 -- 172.21.15.141:0/2056663604 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80094d0 con 0x7f4e1015b030 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.615+0000 7f4e0fe00640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e1015b030 0x7f4e1015e0e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c410 tx=0x7f4e0800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.615+0000 7f4e07400640 1 -- 172.21.15.141:0/2056663604 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08011790 con 0x7f4e1015b030 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.615+0000 7f4e07400640 1 -- 172.21.15.141:0/2056663604 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e080118f0 con 0x7f4e1015b030 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.615+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e101f35b0 con 0x7f4e1015b030 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.615+0000 7f4e07400640 1 -- 172.21.15.141:0/2056663604 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08011ca0 con 0x7f4e1015b030 2024-01-16T14:59:58.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.615+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e101f3b70 con 0x7f4e1015b030 2024-01-16T14:59:58.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.616+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e100675d0 con 0x7f4e1015b030 2024-01-16T14:59:58.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.617+0000 7f4e07400640 1 -- 172.21.15.141:0/2056663604 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4e0801c460 con 0x7f4e1015b030 2024-01-16T14:59:58.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.618+0000 7f4e07400640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4de806b500 0x7f4de806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.618+0000 7f4e07400640 1 -- 172.21.15.141:0/2056663604 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4e080898d0 con 0x7f4e1015b030 2024-01-16T14:59:58.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.619+0000 7f4e0f400640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4de806b500 0x7f4de806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.620+0000 7f4e0f400640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4de806b500 0x7f4de806d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4e1015b760 tx=0x7f4e00003990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.622+0000 7f4e07400640 1 -- 172.21.15.141:0/2056663604 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e080547a0 con 0x7f4e1015b030 2024-01-16T14:59:58.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:58.880+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f4e10070a00 con 0x7f4e1015b030 2024-01-16T14:59:59.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.418+0000 7f4e07400640 1 -- 172.21.15.141:0/2056663604 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v155) v1 ==== 152+0+0 (secure 0 0 0) 0x7f4e08057dc0 con 0x7f4e1015b030 2024-01-16T14:59:59.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.421+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4de806b500 msgr2=0x7f4de806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.421+0000 7f4e16200640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4de806b500 0x7f4de806d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4e1015b760 tx=0x7f4e00003990 comp rx=0 tx=0).stop 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e1015b030 msgr2=0x7f4e1015e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e1015b030 0x7f4e1015e0e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c410 tx=0x7f4e0800c8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 shutdown_connections 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4de806b500 0x7f4de806d9c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e1015b030 0x7f4e1015e0e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e100b5380 0x7f4e1014cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 --2- 172.21.15.141:0/2056663604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e100b31b0 0x7f4e100cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 >> 172.21.15.141:0/2056663604 conn(0x7f4e1006a3e0 msgr2=0x7f4e100b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 shutdown_connections 2024-01-16T14:59:59.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.422+0000 7f4e16200640 1 -- 172.21.15.141:0/2056663604 wait complete. 2024-01-16T14:59:59.425 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ok 2024-01-16T14:59:59.425 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T14:59:59.425 INFO:tasks.cephfs_test_runner:test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:59:59.426 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:59.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.658+0000 7f766f200640 1 Processor -- start 2024-01-16T14:59:59.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.659+0000 7f766f200640 1 -- start start 2024-01-16T14:59:59.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.660+0000 7f766f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f766813e360 0x7f76680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.660+0000 7f766f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766815f9c0 0x7f76680cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.660+0000 7f766f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766815c5f0 con 0x7f766813df80 2024-01-16T14:59:59.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.660+0000 7f766f200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766806c470 con 0x7f766813e360 2024-01-16T14:59:59.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.660+0000 7f766f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766806c750 con 0x7f766815f9c0 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.661+0000 7f766d200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766815f9c0 0x7f76680cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.661+0000 7f765f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f766813e360 0x7f76680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.661+0000 7f765f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f766813e360 0x7f76680cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:49370/0 (socket says 172.21.15.141:49370) 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.661+0000 7f766d200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766815f9c0 0x7f76680cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:41146/0 (socket says 172.21.15.141:41146) 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.661+0000 7f765f200640 1 -- 172.21.15.141:0/1700065762 learned_addr learned my addr 172.21.15.141:0/1700065762 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.662+0000 7f765f200640 1 -- 172.21.15.141:0/1700065762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766815f9c0 msgr2=0x7f76680cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.662+0000 7f765f200640 1 --2- 172.21.15.141:0/1700065762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766815f9c0 0x7f76680cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.662+0000 7f765f200640 1 -- 172.21.15.141:0/1700065762 >> v1:172.21.15.188:6789/0 conn(0x7f766813df80 legacy=0x7f766815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.662+0000 7f765f200640 1 -- 172.21.15.141:0/1700065762 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766806ca30 con 0x7f766813e360 2024-01-16T14:59:59.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.663+0000 7f765f200640 1 --2- 172.21.15.141:0/1700065762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813e360 0x7f76680cf290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7650009870 tx=0x7f765002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1afc0dc6c7e96f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.664+0000 7f7667400640 1 -- 172.21.15.141:0/1700065762 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650030840 con 0x7f766813e360 2024-01-16T14:59:59.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.664+0000 7f7667400640 1 -- 172.21.15.141:0/1700065762 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76500309a0 con 0x7f766813e360 2024-01-16T14:59:59.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.664+0000 7f7667400640 1 -- 172.21.15.141:0/1700065762 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650030d30 con 0x7f766813e360 2024-01-16T14:59:59.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 -- 172.21.15.141:0/1700065762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813e360 msgr2=0x7f76680cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 --2- 172.21.15.141:0/1700065762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813e360 0x7f76680cf290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7650009870 tx=0x7f765002b980 comp rx=0 tx=0).stop 2024-01-16T14:59:59.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 -- 172.21.15.141:0/1700065762 shutdown_connections 2024-01-16T14:59:59.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 --2- 172.21.15.141:0/1700065762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813e360 0x7f76680cf290 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 --2- 172.21.15.141:0/1700065762 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766815f9c0 0x7f76680cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 -- 172.21.15.141:0/1700065762 >> 172.21.15.141:0/1700065762 conn(0x7f766806a800 msgr2=0x7f766806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 -- 172.21.15.141:0/1700065762 shutdown_connections 2024-01-16T14:59:59.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.665+0000 7f766f200640 1 -- 172.21.15.141:0/1700065762 wait complete. 2024-01-16T14:59:59.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.667+0000 7f766f200640 1 Processor -- start 2024-01-16T14:59:59.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.667+0000 7f766f200640 1 -- start start 2024-01-16T14:59:59.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.668+0000 7f766f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813df80 0x7f76680ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.668+0000 7f766f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766813e360 0x7f766813c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.668+0000 7f766f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f766815f9c0 0x7f7668153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.668+0000 7f766f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766813dd10 con 0x7f766815f9c0 2024-01-16T14:59:59.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.668+0000 7f766f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766813cea0 con 0x7f766813df80 2024-01-16T14:59:59.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.668+0000 7f766f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7668154250 con 0x7f766813e360 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f766815f9c0 0x7f7668153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f766815f9c0 0x7f7668153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46930/0 (socket says 172.21.15.141:46930) 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 -- 172.21.15.141:0/1105255823 learned_addr learned my addr 172.21.15.141:0/1105255823 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f7667e00640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813df80 0x7f76680ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f765f200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766813e360 0x7f766813c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 -- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766813e360 msgr2=0x7f766813c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766813e360 0x7f766813c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 -- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813df80 msgr2=0x7f76680ac470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813df80 0x7f76680ac470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 -- 172.21.15.141:0/1105255823 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500094d0 con 0x7f766815f9c0 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.669+0000 7f766d200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f766815f9c0 0x7f7668153d10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f766000efc0 tx=0x7f766000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.670+0000 7f7664c00640 1 -- 172.21.15.141:0/1105255823 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660009ca0 con 0x7f766815f9c0 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.670+0000 7f7664c00640 1 -- 172.21.15.141:0/1105255823 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766000f040 con 0x7f766815f9c0 2024-01-16T14:59:59.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.670+0000 7f7664c00640 1 -- 172.21.15.141:0/1105255823 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76600146c0 con 0x7f766815f9c0 2024-01-16T14:59:59.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.670+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766813d100 con 0x7f766815f9c0 2024-01-16T14:59:59.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.670+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766813d6c0 con 0x7f766815f9c0 2024-01-16T14:59:59.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.671+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76680a84c0 con 0x7f766815f9c0 2024-01-16T14:59:59.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.672+0000 7f7664c00640 1 -- 172.21.15.141:0/1105255823 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f7660014820 con 0x7f766815f9c0 2024-01-16T14:59:59.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.673+0000 7f7664c00640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764006b500 0x7f764006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.673+0000 7f7664c00640 1 -- 172.21.15.141:0/1105255823 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f766008ad70 con 0x7f766815f9c0 2024-01-16T14:59:59.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.673+0000 7f7667e00640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764006b500 0x7f764006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.675+0000 7f7667e00640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764006b500 0x7f764006d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7658000f60 tx=0x7f7658009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.677+0000 7f7664c00640 1 -- 172.21.15.141:0/1105255823 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76680a84c0 con 0x7f766815f9c0 2024-01-16T14:59:59.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.869+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76680a84c0 con 0x7f766815f9c0 2024-01-16T14:59:59.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.871+0000 7f7664c00640 1 -- 172.21.15.141:0/1105255823 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13034 (secure 0 0 0) 0x7f76680a84c0 con 0x7f766815f9c0 2024-01-16T14:59:59.852 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T14:59:59.852 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":38,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T14:59:52.987816+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/176725955":"2024-01-17T14:59:49.971845+0000","172.21.15.141:6833/176725955":"2024-01-17T14:59:49.971845+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764006b500 msgr2=0x7f764006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764006b500 0x7f764006d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7658000f60 tx=0x7f7658009040 comp rx=0 tx=0).stop 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f766815f9c0 msgr2=0x7f7668153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f766815f9c0 0x7f7668153d10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f766000efc0 tx=0x7f766000c490 comp rx=0 tx=0).stop 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 shutdown_connections 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764006b500 0x7f764006d9c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f766815f9c0 0x7f7668153d10 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f766813e360 0x7f766813c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 --2- 172.21.15.141:0/1105255823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f766813df80 0x7f76680ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 >> 172.21.15.141:0/1105255823 conn(0x7f766806a800 msgr2=0x7f76680b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 shutdown_connections 2024-01-16T14:59:59.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T14:59:59.874+0000 7f766f200640 1 -- 172.21.15.141:0/1105255823 wait complete. 2024-01-16T14:59:59.874 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:59.875 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:00.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.105+0000 7f579c600640 1 Processor -- start 2024-01-16T15:00:00.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.105+0000 7f579c600640 1 -- start start 2024-01-16T15:00:00.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.106+0000 7f579c600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 0x7f57940b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.107+0000 7f579c600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579406cea0 con 0x7f579406c6b0 2024-01-16T15:00:00.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.107+0000 7f579c600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579406cbc0 con 0x7f579409a170 2024-01-16T15:00:00.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.107+0000 7f579c600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579406f350 con 0x7f57940cd240 2024-01-16T15:00:00.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.107+0000 7f5799200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f579406c6b0 0x7f5794099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50752/0 (socket says 172.21.15.141:50752) 2024-01-16T15:00:00.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.107+0000 7f5799200640 1 -- 172.21.15.141:0/1098926661 learned_addr learned my addr 172.21.15.141:0/1098926661 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:00.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.107+0000 7f5799c00640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 0x7f57940b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.107+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262836435 0 0) 0x7f579406cea0 con 0x7f579406c6b0 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5774003520 con 0x7f579406c6b0 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36467284 0 0) 0x7f5774003520 con 0x7f579406c6b0 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 msgr2=0x7f57940b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 0x7f57940b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 >> v1:172.21.15.188:6789/0 conn(0x7f579409a170 legacy=0x7f57940cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579406f630 con 0x7f579406c6b0 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f577c001670 con 0x7f579406c6b0 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 >> v1:172.21.15.141:6789/0 conn(0x7f579406c6b0 legacy=0x7f5794099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57740039a0 0x7f5774003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 0x7f57740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5774005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579406f350 con 0x7f5774004d00 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577400ba60 con 0x7f57740039a0 2024-01-16T15:00:00.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.108+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577400bc20 con 0x7f57940cd240 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f579a600640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5774005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f5799c00640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57740039a0 0x7f5774003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f5799200640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 0x7f57740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f579a600640 1 -- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 msgr2=0x7f57740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f579a600640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940cd240 0x7f57740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f579a600640 1 -- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57740039a0 msgr2=0x7f5774003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f579a600640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57740039a0 0x7f5774003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.109+0000 7f579a600640 1 -- 172.21.15.141:0/1098926661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5774003520 con 0x7f5774004d00 2024-01-16T15:00:00.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.110+0000 7f579a600640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5774005180 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f578c001e20 tx=0x7f57740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af92776ab3758a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.110+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c003b70 con 0x7f5774004d00 2024-01-16T15:00:00.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.110+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c003cd0 con 0x7f5774004d00 2024-01-16T15:00:00.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.110+0000 7f5793e00640 1 -- 172.21.15.141:0/1098926661 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c005310 con 0x7f5774004d00 2024-01-16T15:00:00.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.111+0000 7f579c600640 1 -- 172.21.15.141:0/1098926661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 msgr2=0x7f5774005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.111+0000 7f579c600640 1 --2- 172.21.15.141:0/1098926661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5774005180 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f578c001e20 tx=0x7f57740057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.111+0000 7f579a600640 1 -- 172.21.15.141:0/1098926661 reap_dead start 2024-01-16T15:00:00.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.111+0000 7f579c600640 1 -- 172.21.15.141:0/1098926661 shutdown_connections 2024-01-16T15:00:00.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.111+0000 7f579c600640 1 -- 172.21.15.141:0/1098926661 >> 172.21.15.141:0/1098926661 conn(0x7f57940b2d00 msgr2=0x7f57940b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.111+0000 7f579c600640 1 -- 172.21.15.141:0/1098926661 shutdown_connections 2024-01-16T15:00:00.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.111+0000 7f579c600640 1 -- 172.21.15.141:0/1098926661 wait complete. 2024-01-16T15:00:00.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.113+0000 7f579c600640 1 Processor -- start 2024-01-16T15:00:00.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.113+0000 7f579c600640 1 -- start start 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f579c600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5794154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f579c600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940b4060 0x7f57940c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f579c600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57940cffa0 0x7f57940d0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f579c600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940b5170 con 0x7f5774004d00 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f579c600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940d0990 con 0x7f57940cffa0 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f579c600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5794154c90 con 0x7f57940b4060 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f5799c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5794154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f5799c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5794154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46942/0 (socket says 172.21.15.141:46942) 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.114+0000 7f5799c00640 1 -- 172.21.15.141:0/949098686 learned_addr learned my addr 172.21.15.141:0/949098686 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5799200640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940b4060 0x7f57940c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f579a600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57940cffa0 0x7f57940d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5799c00640 1 -- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940b4060 msgr2=0x7f57940c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5799c00640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940b4060 0x7f57940c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5799c00640 1 -- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57940cffa0 msgr2=0x7f57940d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5799c00640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57940cffa0 0x7f57940d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5799c00640 1 -- 172.21.15.141:0/949098686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940b44f0 con 0x7f5774004d00 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f579a600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57940cffa0 0x7f57940d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5799c00640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5794154750 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5784011600 tx=0x7f5784011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5791600640 1 -- 172.21.15.141:0/949098686 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578400f920 con 0x7f5774004d00 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5791600640 1 -- 172.21.15.141:0/949098686 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578400fa80 con 0x7f5774004d00 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f5791600640 1 -- 172.21.15.141:0/949098686 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578400fdf0 con 0x7f5774004d00 2024-01-16T15:00:00.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.115+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57940b47d0 con 0x7f5774004d00 2024-01-16T15:00:00.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.116+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57940b4d90 con 0x7f5774004d00 2024-01-16T15:00:00.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.117+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57940675d0 con 0x7f5774004d00 2024-01-16T15:00:00.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.118+0000 7f5791600640 1 -- 172.21.15.141:0/949098686 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f5784015460 con 0x7f5774004d00 2024-01-16T15:00:00.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.118+0000 7f5791600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f576c06b500 0x7f576c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.119+0000 7f5791600640 1 -- 172.21.15.141:0/949098686 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f578408a270 con 0x7f5774004d00 2024-01-16T15:00:00.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.119+0000 7f5799200640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f576c06b500 0x7f576c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.120+0000 7f5799200640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f576c06b500 0x7f576c06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f57940cb510 tx=0x7f577c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.123+0000 7f5791600640 1 -- 172.21.15.141:0/949098686 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5784054fe0 con 0x7f5774004d00 2024-01-16T15:00:00.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.313+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57940b53e0 con 0x7f5774004d00 2024-01-16T15:00:00.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.314+0000 7f5791600640 1 -- 172.21.15.141:0/949098686 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5784058600 con 0x7f5774004d00 2024-01-16T15:00:00.295 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:00:00.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f576c06b500 msgr2=0x7f576c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f576c06b500 0x7f576c06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f57940cb510 tx=0x7f577c009040 comp rx=0 tx=0).stop 2024-01-16T15:00:00.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 msgr2=0x7f5794154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5794154750 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5784011600 tx=0x7f5784011ac0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 shutdown_connections 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f576c06b500 0x7f576c06d9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57940cffa0 0x7f57940d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57940b4060 0x7f57940c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 --2- 172.21.15.141:0/949098686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5774004d00 0x7f5794154750 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.317+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 >> 172.21.15.141:0/949098686 conn(0x7f57940b2d00 msgr2=0x7f57940b0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.318+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 shutdown_connections 2024-01-16T15:00:00.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.318+0000 7f579c600640 1 -- 172.21.15.141:0/949098686 wait complete. 2024-01-16T15:00:00.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T15:00:00.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.546+0000 7f0bd4c00640 1 Processor -- start 2024-01-16T15:00:00.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.547+0000 7f0bd4c00640 1 -- start start 2024-01-16T15:00:00.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bd4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 0x7f0bd0161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bd4c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0bd00ced50 0x7f0bd00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bd4c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00cf1b0 con 0x7f0bd00ce8f0 2024-01-16T15:00:00.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bd4c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd01618a0 con 0x7f0bd00ced50 2024-01-16T15:00:00.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bd4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00d0de0 con 0x7f0bd0160f80 2024-01-16T15:00:00.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bc5200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0bd00ce8f0 0x7f0bd00cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50754/0 (socket says 172.21.15.141:50754) 2024-01-16T15:00:00.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bc5200640 1 -- 172.21.15.141:0/4087942179 learned_addr learned my addr 172.21.15.141:0/4087942179 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bce800640 1 --2- 172.21.15.141:0/4087942179 >> v2:172.21.15.188:3300/0 conn(0x7f0bd00ced50 0x7f0bd00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bcde00640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 0x7f0bd0161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.548+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701461871 0 0) 0x7f0bd00cf1b0 con 0x7f0bd00ce8f0 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb0003520 con 0x7f0bd00ce8f0 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699603371 0 0) 0x7f0bb0003520 con 0x7f0bd00ce8f0 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 msgr2=0x7f0bd0161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 0x7f0bd0161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 >> v2:172.21.15.188:3300/0 conn(0x7f0bd00ced50 msgr2=0x7f0bd00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 --2- 172.21.15.141:0/4087942179 >> v2:172.21.15.188:3300/0 conn(0x7f0bd00ced50 0x7f0bd00d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd00d10c0 con 0x7f0bd00ce8f0 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0bb8001670 con 0x7f0bd00ce8f0 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 >> v1:172.21.15.141:6789/0 conn(0x7f0bd00ce8f0 legacy=0x7f0bd00cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 0x7f0bb0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bb0003f40 0x7f0bb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0004d00 0x7f0bb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd01618a0 con 0x7f0bb0003f40 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00d0de0 con 0x7f0bb0004d00 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb000b980 con 0x7f0bd0160f80 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.549+0000 7f0bc5200640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bb0003f40 0x7f0bb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.550+0000 7f0bce800640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0004d00 0x7f0bb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.550+0000 7f0bcde00640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 0x7f0bb0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.550+0000 7f0bc5200640 1 -- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 msgr2=0x7f0bb0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.550+0000 7f0bc5200640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd0160f80 0x7f0bb0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.550+0000 7f0bc5200640 1 -- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0004d00 msgr2=0x7f0bb0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.550+0000 7f0bc5200640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0004d00 0x7f0bb0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.550+0000 7f0bc5200640 1 -- 172.21.15.141:0/4087942179 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb0003520 con 0x7f0bb0003f40 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.551+0000 7f0bc5200640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bb0003f40 0x7f0bb00047c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800c140 tx=0x7f0bb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddab38aeb1a643b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.551+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8005ad0 con 0x7f0bb0003f40 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.551+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb8005c50 con 0x7f0bb0003f40 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.551+0000 7f0bcd400640 1 -- 172.21.15.141:0/4087942179 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb802c810 con 0x7f0bb0003f40 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.552+0000 7f0bd4c00640 1 -- 172.21.15.141:0/4087942179 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bb0003f40 msgr2=0x7f0bb00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.552+0000 7f0bd4c00640 1 --2- 172.21.15.141:0/4087942179 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bb0003f40 0x7f0bb00047c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800c140 tx=0x7f0bb00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.552+0000 7f0bce800640 1 -- 172.21.15.141:0/4087942179 reap_dead start 2024-01-16T15:00:00.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.552+0000 7f0bd4c00640 1 -- 172.21.15.141:0/4087942179 shutdown_connections 2024-01-16T15:00:00.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.552+0000 7f0bd4c00640 1 -- 172.21.15.141:0/4087942179 >> 172.21.15.141:0/4087942179 conn(0x7f0bd009c550 msgr2=0x7f0bd00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.552+0000 7f0bd4c00640 1 -- 172.21.15.141:0/4087942179 shutdown_connections 2024-01-16T15:00:00.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.552+0000 7f0bd4c00640 1 -- 172.21.15.141:0/4087942179 wait complete. 2024-01-16T15:00:00.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.555+0000 7f0bd4c00640 1 Processor -- start 2024-01-16T15:00:00.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.555+0000 7f0bd4c00640 1 -- start start 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.556+0000 7f0bd4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0003f40 0x7f0bd00ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.556+0000 7f0bd4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd00b0e90 0x7f0bd01f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.556+0000 7f0bd4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bd01f6940 0x7f0bd00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.556+0000 7f0bd4c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00cd320 con 0x7f0bd01f6940 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.556+0000 7f0bd4c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd01f6dc0 con 0x7f0bb0003f40 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.556+0000 7f0bd4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00cb3f0 con 0x7f0bd00b0e90 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bd01f6940 0x7f0bd00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bd01f6940 0x7f0bd00ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46960/0 (socket says 172.21.15.141:46960) 2024-01-16T15:00:00.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 -- 172.21.15.141:0/3697769226 learned_addr learned my addr 172.21.15.141:0/3697769226 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bc5200640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd00b0e90 0x7f0bd01f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bcde00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0003f40 0x7f0bd00ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 -- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd00b0e90 msgr2=0x7f0bd01f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd00b0e90 0x7f0bd01f6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 -- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0003f40 msgr2=0x7f0bd00ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0003f40 0x7f0bd00ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bce800640 1 -- 172.21.15.141:0/3697769226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd00c7e90 con 0x7f0bd01f6940 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.557+0000 7f0bc5200640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd00b0e90 0x7f0bd01f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.558+0000 7f0bce800640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bd01f6940 0x7f0bd00ca810 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00d0f20 tx=0x7f0bc8011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.558+0000 7f0bc6a00640 1 -- 172.21.15.141:0/3697769226 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8004500 con 0x7f0bd01f6940 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.558+0000 7f0bc6a00640 1 -- 172.21.15.141:0/3697769226 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc8003610 con 0x7f0bd01f6940 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.558+0000 7f0bc6a00640 1 -- 172.21.15.141:0/3697769226 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc80039a0 con 0x7f0bd01f6940 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.558+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd00c8170 con 0x7f0bd01f6940 2024-01-16T15:00:00.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.558+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd00cb6d0 con 0x7f0bd01f6940 2024-01-16T15:00:00.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.559+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd00675d0 con 0x7f0bd01f6940 2024-01-16T15:00:00.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.560+0000 7f0bc6a00640 1 -- 172.21.15.141:0/3697769226 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f0bc8003b00 con 0x7f0bd01f6940 2024-01-16T15:00:00.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.561+0000 7f0bc6a00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba806b530 0x7f0ba806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.561+0000 7f0bc6a00640 1 -- 172.21.15.141:0/3697769226 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0bc808a050 con 0x7f0bd01f6940 2024-01-16T15:00:00.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.562+0000 7f0bcde00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba806b530 0x7f0ba806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.563+0000 7f0bcde00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba806b530 0x7f0ba806d9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0bd006d050 tx=0x7f0bc000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.568+0000 7f0bc6a00640 1 -- 172.21.15.141:0/3697769226 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bc8054fa0 con 0x7f0bd01f6940 2024-01-16T15:00:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:00.831+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f0bd00d3920 con 0x7f0bd01f6940 2024-01-16T15:00:01.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.433+0000 7f0bc6a00640 1 -- 172.21.15.141:0/3697769226 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v157) v1 ==== 156+0+0 (secure 0 0 0) 0x7f0bc8003db0 con 0x7f0bd01f6940 2024-01-16T15:00:01.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba806b530 msgr2=0x7f0ba806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba806b530 0x7f0ba806d9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0bd006d050 tx=0x7f0bc000b040 comp rx=0 tx=0).stop 2024-01-16T15:00:01.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bd01f6940 msgr2=0x7f0bd00ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bd01f6940 0x7f0bd00ca810 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00d0f20 tx=0x7f0bc8011920 comp rx=0 tx=0).stop 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 shutdown_connections 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba806b530 0x7f0ba806d9f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bd01f6940 0x7f0bd00ca810 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bd00b0e90 0x7f0bd01f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 --2- 172.21.15.141:0/3697769226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bb0003f40 0x7f0bd00ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 >> 172.21.15.141:0/3697769226 conn(0x7f0bd009c550 msgr2=0x7f0bd00b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 shutdown_connections 2024-01-16T15:00:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.438+0000 7f0bd4c00640 1 -- 172.21.15.141:0/3697769226 wait complete. 2024-01-16T15:00:01.442 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:00:01.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.675+0000 7f540ba00640 1 Processor -- start 2024-01-16T15:00:01.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.675+0000 7f540ba00640 1 -- start start 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f540ba00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f54040cf9a0 0x7f540415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f540ba00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f540415e1e0 0x7f54040702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f540ba00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54040b1ee0 con 0x7f5404143ae0 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f540ba00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54040b2780 con 0x7f54040cf9a0 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f540ba00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540415e990 con 0x7f540415e1e0 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f53fbc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f54040cf9a0 0x7f540415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f5409000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5404143ae0 0x7f54040cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50762/0 (socket says 172.21.15.141:50762) 2024-01-16T15:00:01.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.677+0000 7f5409000640 1 -- 172.21.15.141:0/1830503548 learned_addr learned my addr 172.21.15.141:0/1830503548 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:01.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.678+0000 7f5409a00640 1 --2- 172.21.15.141:0/1830503548 >> v2:172.21.15.188:3300/0 conn(0x7f540415e1e0 0x7f54040702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.678+0000 7f53fbc00640 1 -- 172.21.15.141:0/1830503548 >> v2:172.21.15.188:3300/0 conn(0x7f540415e1e0 msgr2=0x7f54040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.678+0000 7f53fbc00640 1 --2- 172.21.15.141:0/1830503548 >> v2:172.21.15.188:3300/0 conn(0x7f540415e1e0 0x7f54040702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.678+0000 7f53fbc00640 1 -- 172.21.15.141:0/1830503548 >> v1:172.21.15.141:6789/0 conn(0x7f5404143ae0 legacy=0x7f54040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.678+0000 7f53fbc00640 1 -- 172.21.15.141:0/1830503548 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54040b2130 con 0x7f54040cf9a0 2024-01-16T15:00:01.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.679+0000 7f53fbc00640 1 --2- 172.21.15.141:0/1830503548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f540415dca0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f54040ae9d0 tx=0x7f53ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1349718a2854c553 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.679+0000 7f5403e00640 1 -- 172.21.15.141:0/1830503548 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec031830 con 0x7f54040cf9a0 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.679+0000 7f5403e00640 1 -- 172.21.15.141:0/1830503548 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ec031990 con 0x7f54040cf9a0 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.679+0000 7f5403e00640 1 -- 172.21.15.141:0/1830503548 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec031d20 con 0x7f54040cf9a0 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 -- 172.21.15.141:0/1830503548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 msgr2=0x7f540415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 --2- 172.21.15.141:0/1830503548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f540415dca0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f54040ae9d0 tx=0x7f53ec02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 -- 172.21.15.141:0/1830503548 shutdown_connections 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 --2- 172.21.15.141:0/1830503548 >> v2:172.21.15.188:3300/0 conn(0x7f540415e1e0 0x7f54040702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 --2- 172.21.15.141:0/1830503548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f540415dca0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 -- 172.21.15.141:0/1830503548 >> 172.21.15.141:0/1830503548 conn(0x7f540406a800 msgr2=0x7f540406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 -- 172.21.15.141:0/1830503548 shutdown_connections 2024-01-16T15:00:01.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.680+0000 7f540ba00640 1 -- 172.21.15.141:0/1830503548 wait complete. 2024-01-16T15:00:01.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.682+0000 7f540ba00640 1 Processor -- start 2024-01-16T15:00:01.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.682+0000 7f540ba00640 1 -- start start 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f540ba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f54040ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f540ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5404143ae0 0x7f54040c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f540ba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f540415e1e0 0x7f540413eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f540ba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54040cc610 con 0x7f54040cf9a0 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f540ba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54040adb30 con 0x7f540415e1e0 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f540ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54040ade10 con 0x7f5404143ae0 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f5409000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f54040ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f5409000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f54040ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46990/0 (socket says 172.21.15.141:46990) 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f5409000640 1 -- 172.21.15.141:0/3981505406 learned_addr learned my addr 172.21.15.141:0/3981505406 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.683+0000 7f5409a00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f540415e1e0 0x7f540413eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f53fbc00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5404143ae0 0x7f54040c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5409000640 1 -- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5404143ae0 msgr2=0x7f54040c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5409000640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5404143ae0 0x7f54040c4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5409000640 1 -- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f540415e1e0 msgr2=0x7f540413eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5409000640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f540415e1e0 0x7f540413eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5409000640 1 -- 172.21.15.141:0/3981505406 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec009400 con 0x7f54040cf9a0 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5409a00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f540415e1e0 0x7f540413eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5409000640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f54040ad2e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c4a0 tx=0x7f53f400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5401600640 1 -- 172.21.15.141:0/3981505406 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4005ce0 con 0x7f54040cf9a0 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5401600640 1 -- 172.21.15.141:0/3981505406 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f4005e40 con 0x7f54040cf9a0 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f5401600640 1 -- 172.21.15.141:0/3981505406 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4007270 con 0x7f54040cf9a0 2024-01-16T15:00:01.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54040ae0f0 con 0x7f54040cf9a0 2024-01-16T15:00:01.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.684+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54040ad820 con 0x7f54040cf9a0 2024-01-16T15:00:01.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.686+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54040ae280 con 0x7f54040cf9a0 2024-01-16T15:00:01.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.686+0000 7f5401600640 1 -- 172.21.15.141:0/3981505406 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f53f4007490 con 0x7f54040cf9a0 2024-01-16T15:00:01.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.687+0000 7f5401600640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53dc06b500 0x7f53dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.687+0000 7f5401600640 1 -- 172.21.15.141:0/3981505406 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f53f4089250 con 0x7f54040cf9a0 2024-01-16T15:00:01.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.688+0000 7f53fbc00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53dc06b500 0x7f53dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.689+0000 7f53fbc00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53dc06b500 0x7f53dc06d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f53ec036000 tx=0x7f53ec03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.692+0000 7f5401600640 1 -- 172.21.15.141:0/3981505406 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53f4054070 con 0x7f54040cf9a0 2024-01-16T15:00:01.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.881+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f54040cbf50 con 0x7f54040cf9a0 2024-01-16T15:00:01.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.887+0000 7f5401600640 1 -- 172.21.15.141:0/3981505406 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f53f4057690 con 0x7f54040cf9a0 2024-01-16T15:00:01.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.890+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53dc06b500 msgr2=0x7f53dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.890+0000 7f540ba00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53dc06b500 0x7f53dc06d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f53ec036000 tx=0x7f53ec03c040 comp rx=0 tx=0).stop 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.890+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 msgr2=0x7f54040ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.890+0000 7f540ba00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f54040ad2e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c4a0 tx=0x7f53f400c960 comp rx=0 tx=0).stop 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 shutdown_connections 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53dc06b500 0x7f53dc06d9c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f540415e1e0 0x7f540413eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5404143ae0 0x7f54040c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 --2- 172.21.15.141:0/3981505406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54040cf9a0 0x7f54040ad2e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 >> 172.21.15.141:0/3981505406 conn(0x7f540406a800 msgr2=0x7f5404142cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 shutdown_connections 2024-01-16T15:00:01.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:01.891+0000 7f540ba00640 1 -- 172.21.15.141:0/3981505406 wait complete. 2024-01-16T15:00:01.890 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:00:01.890 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:00:01.995 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:00:02.134 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:00:02.135 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:00:02.135 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:02.191 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:00:02.191 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:00:02.191 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:02.239 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:00:02.239 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:00:02.239 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:02.294 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:00:02.295 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:00:02.295 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:02.336 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:00:02.336 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:00:02.336 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:02.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.572+0000 7f1d09800640 1 Processor -- start 2024-01-16T15:00:02.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.572+0000 7f1d09800640 1 -- start start 2024-01-16T15:00:02.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.574+0000 7f1d09800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 0x7f1d040aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.574+0000 7f1d09800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d04071de0 con 0x7f1d0406a3e0 2024-01-16T15:00:02.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.574+0000 7f1d09800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d04072030 con 0x7f1d040d0b60 2024-01-16T15:00:02.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.574+0000 7f1d09800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040acd60 con 0x7f1d040ac8d0 2024-01-16T15:00:02.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.574+0000 7f1d02a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1d0406a3e0 0x7f1d0406a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50764/0 (socket says 172.21.15.141:50764) 2024-01-16T15:00:02.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.574+0000 7f1d02a00640 1 -- 172.21.15.141:0/1416284962 learned_addr learned my addr 172.21.15.141:0/1416284962 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182630756 0 0) 0x7f1d04071de0 con 0x7f1d0406a3e0 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce4003520 con 0x7f1d0406a3e0 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d03400640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 0x7f1d040aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186666198 0 0) 0x7f1ce4003520 con 0x7f1d0406a3e0 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 msgr2=0x7f1d040aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 0x7f1d040aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 >> v1:172.21.15.188:6789/0 conn(0x7f1d040d0b60 legacy=0x7f1d0406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d04072310 con 0x7f1d0406a3e0 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1cf40019f0 con 0x7f1d0406a3e0 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 >> v1:172.21.15.141:6789/0 conn(0x7f1d0406a3e0 legacy=0x7f1d0406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 0x7f1ce40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4003fd0 0x7f1ce40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1ce4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040acd60 con 0x7f1ce4004d00 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce400bb40 con 0x7f1ce4003fd0 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.575+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce400bd00 con 0x7f1d040ac8d0 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1d03400640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1ce4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1cf9e00640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4003fd0 0x7f1ce40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1d02a00640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 0x7f1ce40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1d03400640 1 -- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 msgr2=0x7f1ce40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1d03400640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d040ac8d0 0x7f1ce40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1d03400640 1 -- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4003fd0 msgr2=0x7f1ce40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1d03400640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4003fd0 0x7f1ce40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1d03400640 1 -- 172.21.15.141:0/1416284962 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4003520 con 0x7f1ce4004d00 2024-01-16T15:00:02.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.576+0000 7f1cf9e00640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4003fd0 0x7f1ce40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.577+0000 7f1d03400640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1ce4005180 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00ba30 tx=0x7f1ce40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2acbe4c2b533e25 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.577+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc032640 con 0x7f1ce4004d00 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.577+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cfc0327a0 con 0x7f1ce4004d00 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.577+0000 7f1d02000640 1 -- 172.21.15.141:0/1416284962 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc032b30 con 0x7f1ce4004d00 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.578+0000 7f1d09800640 1 -- 172.21.15.141:0/1416284962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 msgr2=0x7f1ce4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.578+0000 7f1d09800640 1 --2- 172.21.15.141:0/1416284962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1ce4005180 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00ba30 tx=0x7f1ce40057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.578+0000 7f1d03400640 1 -- 172.21.15.141:0/1416284962 reap_dead start 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.578+0000 7f1d09800640 1 -- 172.21.15.141:0/1416284962 shutdown_connections 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.578+0000 7f1d09800640 1 -- 172.21.15.141:0/1416284962 >> 172.21.15.141:0/1416284962 conn(0x7f1d040cfef0 msgr2=0x7f1d040d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.578+0000 7f1d09800640 1 -- 172.21.15.141:0/1416284962 shutdown_connections 2024-01-16T15:00:02.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.578+0000 7f1d09800640 1 -- 172.21.15.141:0/1416284962 wait complete. 2024-01-16T15:00:02.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.580+0000 7f1d09800640 1 Processor -- start 2024-01-16T15:00:02.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.580+0000 7f1d09800640 1 -- start start 2024-01-16T15:00:02.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d09800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1d04150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d09800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d0415e300 0x7f1d041602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d09800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d040ce340 0x7f1d040ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d09800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d04155130 con 0x7f1ce4004d00 2024-01-16T15:00:02.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d09800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040ced00 con 0x7f1d040ce340 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d09800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d041552c0 con 0x7f1d0415e300 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d02a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1d04150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d02a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1d04150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46996/0 (socket says 172.21.15.141:46996) 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.581+0000 7f1d02a00640 1 -- 172.21.15.141:0/3265849639 learned_addr learned my addr 172.21.15.141:0/3265849639 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d02a00640 1 -- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d0415e300 msgr2=0x7f1d041602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d02a00640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d0415e300 0x7f1d041602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d02a00640 1 -- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d040ce340 msgr2=0x7f1d040ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d02a00640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d040ce340 0x7f1d040ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d02a00640 1 -- 172.21.15.141:0/3265849639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0415e780 con 0x7f1ce4004d00 2024-01-16T15:00:02.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d02a00640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1d04150b70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4007ab0 tx=0x7f1cf400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1cfb400640 1 -- 172.21.15.141:0/3265849639 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf400ee90 con 0x7f1ce4004d00 2024-01-16T15:00:02.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1cfb400640 1 -- 172.21.15.141:0/3265849639 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf4013420 con 0x7f1ce4004d00 2024-01-16T15:00:02.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1cfb400640 1 -- 172.21.15.141:0/3265849639 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40137b0 con 0x7f1ce4004d00 2024-01-16T15:00:02.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0415ea00 con 0x7f1ce4004d00 2024-01-16T15:00:02.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.582+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0415efc0 con 0x7f1ce4004d00 2024-01-16T15:00:02.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.584+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d040675d0 con 0x7f1ce4004d00 2024-01-16T15:00:02.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.584+0000 7f1cfb400640 1 -- 172.21.15.141:0/3265849639 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f1cf4013910 con 0x7f1ce4004d00 2024-01-16T15:00:02.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.585+0000 7f1cfb400640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cdc06b500 0x7f1cdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.585+0000 7f1cfb400640 1 -- 172.21.15.141:0/3265849639 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1cf408a520 con 0x7f1ce4004d00 2024-01-16T15:00:02.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.586+0000 7f1cf9e00640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cdc06b500 0x7f1cdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.587+0000 7f1cf9e00640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cdc06b500 0x7f1cdc06d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d0415baa0 tx=0x7f1cec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.590+0000 7f1cfb400640 1 -- 172.21.15.141:0/3265849639 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1cf4055360 con 0x7f1ce4004d00 2024-01-16T15:00:02.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.815+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d0406dcd0 con 0x7f1ce4004d00 2024-01-16T15:00:02.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.816+0000 7f1cfb400640 1 -- 172.21.15.141:0/3265849639 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f1cf4058980 con 0x7f1ce4004d00 2024-01-16T15:00:02.797 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:02.797 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/2661986774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":2661986774},{"type":"v1","addr":"172.21.15.188:6836","nonce":2661986774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4586,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2344909439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2344909439},{"type":"v1","addr":"172.21.15.188:6837","nonce":2344909439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/1479821799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":1479821799},{"type":"v1","addr":"172.21.15.141:6834","nonce":1479821799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5691,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/4229845994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":4229845994},{"type":"v1","addr":"172.21.15.141:6835","nonce":4229845994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-01-16T15:00:02.797 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 20 2024-01-16T15:00:02.799 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:02.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cdc06b500 msgr2=0x7f1cdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cdc06b500 0x7f1cdc06d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d0415baa0 tx=0x7f1cec009040 comp rx=0 tx=0).stop 2024-01-16T15:00:02.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 msgr2=0x7f1d04150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1d04150b70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4007ab0 tx=0x7f1cf400e530 comp rx=0 tx=0).stop 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 shutdown_connections 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1cdc06b500 0x7f1cdc06d9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d040ce340 0x7f1d040ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d0415e300 0x7f1d041602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 --2- 172.21.15.141:0/3265849639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ce4004d00 0x7f1d04150b70 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 >> 172.21.15.141:0/3265849639 conn(0x7f1d040cfef0 msgr2=0x7f1d0406d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.819+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 shutdown_connections 2024-01-16T15:00:02.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:02.820+0000 7f1d09800640 1 -- 172.21.15.141:0/3265849639 wait complete. 2024-01-16T15:00:02.804 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:02.819 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:00:02.820 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:00:02.820 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:00:02.820 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:02.820 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:00:02.820 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:00:02.820 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:00:02.820 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:02.821 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:00:02.821 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:00:02.821 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:02.821 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:00:02.821 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:00:02.821 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:02.873 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:00:02.873 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:00:02.878 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:00:02.878 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:00:02.879 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:00:02.879 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:00:02.880 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:00:02.880 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:00:02.926 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:00:02.926 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:02.927 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:00:02.927 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:02.932 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:00:02.932 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:02.933 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:00:02.933 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:02.934 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:00:03.077 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T15:00:03.080 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:00:03.090 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T15:00:03.091 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T15:00:03.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.180+0000 7f6575000640 1 Processor -- start 2024-01-16T15:00:03.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.180+0000 7f6575000640 1 -- start start 2024-01-16T15:00:03.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.181+0000 7f6575000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f65700cf7a0 0x7f65700cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.181+0000 7f6575000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657006d040 con 0x7f657013e140 2024-01-16T15:00:03.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.181+0000 7f6575000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657006c470 con 0x7f65700cf310 2024-01-16T15:00:03.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.181+0000 7f6575000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657015c4c0 con 0x7f65700cf7a0 2024-01-16T15:00:03.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.181+0000 7f656de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f657013e140 0x7f657015bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55168/0 (socket says 172.21.15.141:55168) 2024-01-16T15:00:03.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.181+0000 7f656de00640 1 -- 172.21.15.141:0/2493897039 learned_addr learned my addr 172.21.15.141:0/2493897039 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:03.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.181+0000 7f656e800640 1 --2- 172.21.15.141:0/2493897039 >> v2:172.21.15.141:3300/0 conn(0x7f65700cf7a0 0x7f65700cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.182+0000 7f656d400640 1 -- 172.21.15.141:0/2493897039 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048707151 0 0) 0x7f657006d040 con 0x7f657013e140 2024-01-16T15:00:03.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.182+0000 7f656d400640 1 -- 172.21.15.141:0/2493897039 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6550003520 con 0x7f657013e140 2024-01-16T15:00:03.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.182+0000 7f656e800640 1 -- 172.21.15.141:0/2493897039 >> v1:172.21.15.188:6789/0 conn(0x7f65700cf310 legacy=0x7f65700cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.182+0000 7f656e800640 1 -- 172.21.15.141:0/2493897039 >> v1:172.21.15.141:6789/0 conn(0x7f657013e140 legacy=0x7f657015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.182+0000 7f656e800640 1 -- 172.21.15.141:0/2493897039 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700cfc30 con 0x7f65700cf7a0 2024-01-16T15:00:03.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.183+0000 7f656e800640 1 --2- 172.21.15.141:0/2493897039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f65700cdf50 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6568009d90 tx=0x7f656802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76674a8d9503b719 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.183+0000 7f656d400640 1 -- 172.21.15.141:0/2493897039 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656802cc60 con 0x7f65700cf7a0 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.183+0000 7f656d400640 1 -- 172.21.15.141:0/2493897039 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656802cdc0 con 0x7f65700cf7a0 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.183+0000 7f656d400640 1 -- 172.21.15.141:0/2493897039 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568034640 con 0x7f65700cf7a0 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.184+0000 7f6575000640 1 -- 172.21.15.141:0/2493897039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 msgr2=0x7f65700cdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.184+0000 7f6575000640 1 --2- 172.21.15.141:0/2493897039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f65700cdf50 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6568009d90 tx=0x7f656802beb0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.184+0000 7f6575000640 1 -- 172.21.15.141:0/2493897039 shutdown_connections 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.184+0000 7f6575000640 1 --2- 172.21.15.141:0/2493897039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f65700cdf50 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.184+0000 7f6575000640 1 -- 172.21.15.141:0/2493897039 >> 172.21.15.141:0/2493897039 conn(0x7f657006a800 msgr2=0x7f657006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.184+0000 7f6575000640 1 -- 172.21.15.141:0/2493897039 shutdown_connections 2024-01-16T15:00:03.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.184+0000 7f6575000640 1 -- 172.21.15.141:0/2493897039 wait complete. 2024-01-16T15:00:03.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.186+0000 7f6575000640 1 Processor -- start 2024-01-16T15:00:03.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.186+0000 7f6575000640 1 -- start start 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6575000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65700cf310 0x7f6570139a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6575000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f6570136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6575000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f657013e140 0x7f65701407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6575000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657013a750 con 0x7f65700cf7a0 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6575000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6570140ce0 con 0x7f65700cf310 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6575000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6570140f80 con 0x7f657013e140 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6567e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f6570136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6567e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f6570136fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39116/0 (socket says 172.21.15.141:39116) 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f6567e00640 1 -- 172.21.15.141:0/2927397749 learned_addr learned my addr 172.21.15.141:0/2927397749 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f656de00640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65700cf310 0x7f6570139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.187+0000 7f656e800640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f657013e140 0x7f65701407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6567e00640 1 -- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f657013e140 msgr2=0x7f65701407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6567e00640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f657013e140 0x7f65701407a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6567e00640 1 -- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65700cf310 msgr2=0x7f6570139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6567e00640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65700cf310 0x7f6570139a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6567e00640 1 -- 172.21.15.141:0/2927397749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568009a70 con 0x7f65700cf7a0 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f656de00640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65700cf310 0x7f6570139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6567e00640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f6570136fa0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f655800c430 tx=0x7f655800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6566000640 1 -- 172.21.15.141:0/2927397749 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6558011790 con 0x7f65700cf7a0 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6566000640 1 -- 172.21.15.141:0/2927397749 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65580118f0 con 0x7f65700cf7a0 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6570141260 con 0x7f65700cf7a0 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6566000640 1 -- 172.21.15.141:0/2927397749 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6558011ce0 con 0x7f65700cf7a0 2024-01-16T15:00:03.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.188+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6570144060 con 0x7f65700cf7a0 2024-01-16T15:00:03.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.189+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6570141780 con 0x7f65700cf7a0 2024-01-16T15:00:03.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.190+0000 7f6566000640 1 -- 172.21.15.141:0/2927397749 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f655801c460 con 0x7f65700cf7a0 2024-01-16T15:00:03.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.191+0000 7f6566000640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f654806b500 0x7f654806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.191+0000 7f6566000640 1 -- 172.21.15.141:0/2927397749 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6558089ae0 con 0x7f65700cf7a0 2024-01-16T15:00:03.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.192+0000 7f656de00640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f654806b500 0x7f654806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.193+0000 7f656de00640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f654806b500 0x7f654806d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6560007fd0 tx=0x7f6560009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.196+0000 7f6566000640 1 -- 172.21.15.141:0/2927397749 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6570141780 con 0x7f65700cf7a0 2024-01-16T15:00:03.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.397+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f657013a520 con 0x7f65700cf7a0 2024-01-16T15:00:03.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.457+0000 7f6566000640 1 -- 172.21.15.141:0/2927397749 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v40) v1 ==== 97+0+0 (secure 0 0 0) 0x7f657013a520 con 0x7f65700cf7a0 2024-01-16T15:00:03.437 INFO:teuthology.orchestra.run.smithi141.stderr: removed all blocklist entries 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f654806b500 msgr2=0x7f654806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f654806b500 0x7f654806d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6560007fd0 tx=0x7f6560009040 comp rx=0 tx=0).stop 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 msgr2=0x7f6570136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f6570136fa0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f655800c430 tx=0x7f655800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 shutdown_connections 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f654806b500 0x7f654806d9c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f657013e140 0x7f65701407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65700cf7a0 0x7f6570136fa0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 --2- 172.21.15.141:0/2927397749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65700cf310 0x7f6570139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 >> 172.21.15.141:0/2927397749 conn(0x7f657006a800 msgr2=0x7f6570154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 shutdown_connections 2024-01-16T15:00:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.460+0000 7f6575000640 1 -- 172.21.15.141:0/2927397749 wait complete. 2024-01-16T15:00:03.463 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:00:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.682+0000 7f9eca400640 1 Processor -- start 2024-01-16T15:00:03.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.682+0000 7f9eca400640 1 -- start start 2024-01-16T15:00:03.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.683+0000 7f9eca400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9ec40ae5a0 0x7f9ec415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.683+0000 7f9eca400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec40cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.683+0000 7f9eca400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec4070c40 con 0x7f9ec40b3ae0 2024-01-16T15:00:03.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.683+0000 7f9eca400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec4071450 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.683+0000 7f9eca400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec40cfc60 con 0x7f9ec40cf290 2024-01-16T15:00:03.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.684+0000 7f9ec3400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9ec40ae5a0 0x7f9ec415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.684+0000 7f9ec3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec40cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.684+0000 7f9ec3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec40cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:41382/0 (socket says 172.21.15.141:41382) 2024-01-16T15:00:03.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.684+0000 7f9ec3400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9ec40ae5a0 0x7f9ec415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:35270/0 (socket says 172.21.15.141:35270) 2024-01-16T15:00:03.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.684+0000 7f9ec3e00640 1 -- 172.21.15.141:0/1629274553 learned_addr learned my addr 172.21.15.141:0/1629274553 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:03.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.685+0000 7f9ec3400640 1 -- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 msgr2=0x7f9ec40cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.685+0000 7f9ec3400640 1 --2- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec40cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.685+0000 7f9ec3400640 1 -- 172.21.15.141:0/1629274553 >> v1:172.21.15.188:6789/0 conn(0x7f9ec40b3ae0 legacy=0x7f9ec406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.685+0000 7f9ec3400640 1 -- 172.21.15.141:0/1629274553 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec406c470 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.685+0000 7f9ec3e00640 1 --2- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:03.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.686+0000 7f9ec3400640 1 --2- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec415e430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40097a0 tx=0x7f9eb402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d8d18137e1cb92a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.687+0000 7f9ec2000640 1 -- 172.21.15.141:0/1629274553 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4030610 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.687+0000 7f9ec2000640 1 -- 172.21.15.141:0/1629274553 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4030770 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.687+0000 7f9ec2000640 1 -- 172.21.15.141:0/1629274553 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4030b00 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 -- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40ae5a0 msgr2=0x7f9ec415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 --2- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec415e430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40097a0 tx=0x7f9eb402b960 comp rx=0 tx=0).stop 2024-01-16T15:00:03.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 -- 172.21.15.141:0/1629274553 shutdown_connections 2024-01-16T15:00:03.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 --2- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 --2- 172.21.15.141:0/1629274553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec415e430 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 -- 172.21.15.141:0/1629274553 >> 172.21.15.141:0/1629274553 conn(0x7f9ec406a800 msgr2=0x7f9ec406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 -- 172.21.15.141:0/1629274553 shutdown_connections 2024-01-16T15:00:03.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.688+0000 7f9eca400640 1 -- 172.21.15.141:0/1629274553 wait complete. 2024-01-16T15:00:03.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.690+0000 7f9eca400640 1 Processor -- start 2024-01-16T15:00:03.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.690+0000 7f9eca400640 1 -- start start 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.691+0000 7f9eca400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec414f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.691+0000 7f9eca400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40b3ae0 0x7f9ec41f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.691+0000 7f9eca400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec415c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.691+0000 7f9eca400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec415c350 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.691+0000 7f9eca400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec415b6d0 con 0x7f9ec40b3ae0 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.691+0000 7f9eca400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec415b9b0 con 0x7f9ec40cf290 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.691+0000 7f9ec3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec414f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec414f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39124/0 (socket says 172.21.15.141:39124) 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 -- 172.21.15.141:0/3427917181 learned_addr learned my addr 172.21.15.141:0/3427917181 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3e00640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec415c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec2a00640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40b3ae0 0x7f9ec41f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 -- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 msgr2=0x7f9ec415c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec415c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 -- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40b3ae0 msgr2=0x7f9ec41f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40b3ae0 0x7f9ec41f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 -- 172.21.15.141:0/3427917181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009b30 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec2a00640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40b3ae0 0x7f9ec41f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:03.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3e00640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec415c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:03.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.692+0000 7f9ec3400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec414f960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9eb402bf70 tx=0x7f9eb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.693+0000 7f9ebb400640 1 -- 172.21.15.141:0/3427917181 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb403e040 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.693+0000 7f9ebb400640 1 -- 172.21.15.141:0/3427917181 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4039a40 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.693+0000 7f9ebb400640 1 -- 172.21.15.141:0/3427917181 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4039df0 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.693+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb4009400 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.693+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec415bf70 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.694+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec40675d0 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.695+0000 7f9ebb400640 1 -- 172.21.15.141:0/3427917181 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9eb4043390 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.696+0000 7f9ebb400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.696+0000 7f9ebb400640 1 -- 172.21.15.141:0/3427917181 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9eb40abee0 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.696+0000 7f9ec2a00640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.698+0000 7f9ec2a00640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4160c60 tx=0x7f9eac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.700+0000 7f9ebb400640 1 -- 172.21.15.141:0/3427917181 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9eb4076d40 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.955+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9ec415acf0 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.956+0000 7f9ebb400640 1 -- 172.21.15.141:0/3427917181 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9eb407a360 con 0x7f9ec40ae5a0 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: "auth_dump": [ 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.a", 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ==", 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.b", 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg==", 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.c", 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw==", 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.d", 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg==", 2024-01-16T15:00:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.0", 2024-01-16T15:00:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA==", 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.1", 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg==", 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.2", 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g==", 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.3", 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w==", 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.944 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.4", 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ==", 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.5", 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA==", 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.947 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.947 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.947 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.947 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.948 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.6", 2024-01-16T15:00:03.948 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ==", 2024-01-16T15:00:03.948 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.948 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.948 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.948 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.7", 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg==", 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.949 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.0", 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg==", 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow r", 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:00:03.950 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.1", 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ==", 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:00:03.951 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.952 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.2", 2024-01-16T15:00:03.952 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ==", 2024-01-16T15:00:03.952 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.952 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.952 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.3", 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA==", 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:00:03.953 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.admin", 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAEmaZl0VgDMRAABOWZ0senIXHFvij8FQHiAg==", 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.954 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.x", 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ==", 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.955 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.y", 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA==", 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:00:03.956 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:00:03.957 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:00:03.957 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:00:03.957 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.957 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:03.957 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2024-01-16T15:00:03.957 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:00:03.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9e9c06b500 msgr2=0x7f9e9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4160c60 tx=0x7f9eac009040 comp rx=0 tx=0).stop 2024-01-16T15:00:03.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ec40ae5a0 msgr2=0x7f9ec414f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec414f960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9eb402bf70 tx=0x7f9eb4004730 comp rx=0 tx=0).stop 2024-01-16T15:00:03.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 shutdown_connections 2024-01-16T15:00:03.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ec40cf290 0x7f9ec415c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec40b3ae0 0x7f9ec41f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 --2- 172.21.15.141:0/3427917181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ec40ae5a0 0x7f9ec414f960 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 >> 172.21.15.141:0/3427917181 conn(0x7f9ec406a800 msgr2=0x7f9ec4160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.958+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 shutdown_connections 2024-01-16T15:00:03.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:03.959+0000 7f9eca400640 1 -- 172.21.15.141:0/3427917181 wait complete. 2024-01-16T15:00:03.961 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:04.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.189+0000 7fdba7600640 1 Processor -- start 2024-01-16T15:00:04.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.189+0000 7fdba7600640 1 -- start start 2024-01-16T15:00:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba7600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdba006a3e0 0x7fdba00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba7600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00aede0 0x7fdba006f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba7600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba0071de0 con 0x7fdba00d0b60 2024-01-16T15:00:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba7600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00af270 con 0x7fdba006a3e0 2024-01-16T15:00:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba7600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0072030 con 0x7fdba00aede0 2024-01-16T15:00:04.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba4c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdba006a3e0 0x7fdba00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba5600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00aede0 0x7fdba006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba5600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00aede0 0x7fdba006f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:41396/0 (socket says 172.21.15.141:41396) 2024-01-16T15:00:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.191+0000 7fdba5600640 1 -- 172.21.15.141:0/421826635 learned_addr learned my addr 172.21.15.141:0/421826635 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.192+0000 7fdba5600640 1 -- 172.21.15.141:0/421826635 >> v2:172.21.15.188:3300/0 conn(0x7fdba006a3e0 msgr2=0x7fdba00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.192+0000 7fdba5600640 1 --2- 172.21.15.141:0/421826635 >> v2:172.21.15.188:3300/0 conn(0x7fdba006a3e0 0x7fdba00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.192+0000 7fdba5600640 1 -- 172.21.15.141:0/421826635 >> v1:172.21.15.188:6789/0 conn(0x7fdba00d0b60 legacy=0x7fdba006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.192+0000 7fdba5600640 1 -- 172.21.15.141:0/421826635 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba0072310 con 0x7fdba00aede0 2024-01-16T15:00:04.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.193+0000 7fdba5600640 1 --2- 172.21.15.141:0/421826635 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00aede0 0x7fdba006f080 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009e50 tx=0x7fdb9802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4814a74767c7d3b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.194+0000 7fdb9f400640 1 -- 172.21.15.141:0/421826635 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9802cb00 con 0x7fdba00aede0 2024-01-16T15:00:04.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.194+0000 7fdb9f400640 1 -- 172.21.15.141:0/421826635 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9802cc60 con 0x7fdba00aede0 2024-01-16T15:00:04.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.194+0000 7fdb9f400640 1 -- 172.21.15.141:0/421826635 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb980343f0 con 0x7fdba00aede0 2024-01-16T15:00:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.194+0000 7fdba7600640 1 -- 172.21.15.141:0/421826635 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00aede0 msgr2=0x7fdba006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.194+0000 7fdba7600640 1 --2- 172.21.15.141:0/421826635 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00aede0 0x7fdba006f080 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009e50 tx=0x7fdb9802bf60 comp rx=0 tx=0).stop 2024-01-16T15:00:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.195+0000 7fdba7600640 1 -- 172.21.15.141:0/421826635 shutdown_connections 2024-01-16T15:00:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.195+0000 7fdba7600640 1 --2- 172.21.15.141:0/421826635 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00aede0 0x7fdba006f080 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.195+0000 7fdba7600640 1 --2- 172.21.15.141:0/421826635 >> v2:172.21.15.188:3300/0 conn(0x7fdba006a3e0 0x7fdba00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.195+0000 7fdba7600640 1 -- 172.21.15.141:0/421826635 >> 172.21.15.141:0/421826635 conn(0x7fdba00cfef0 msgr2=0x7fdba00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.195+0000 7fdba7600640 1 -- 172.21.15.141:0/421826635 shutdown_connections 2024-01-16T15:00:04.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.195+0000 7fdba7600640 1 -- 172.21.15.141:0/421826635 wait complete. 2024-01-16T15:00:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.196+0000 7fdba7600640 1 Processor -- start 2024-01-16T15:00:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.197+0000 7fdba7600640 1 -- start start 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.197+0000 7fdba7600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba006a3e0 0x7fdba0141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.197+0000 7fdba7600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba00aede0 0x7fdba0144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.197+0000 7fdba7600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00d0b60 0x7fdba0142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.197+0000 7fdba7600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0141560 con 0x7fdba006a3e0 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.197+0000 7fdba7600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0141bc0 con 0x7fdba00aede0 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.197+0000 7fdba7600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0141ea0 con 0x7fdba00d0b60 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba006a3e0 0x7fdba0141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba006a3e0 0x7fdba0141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39128/0 (socket says 172.21.15.141:39128) 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 -- 172.21.15.141:0/3589723087 learned_addr learned my addr 172.21.15.141:0/3589723087 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 -- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00d0b60 msgr2=0x7fdba0142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba5600640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00d0b60 0x7fdba0142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdb9fe00640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba00aede0 0x7fdba0144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00d0b60 0x7fdba0142c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 -- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba00aede0 msgr2=0x7fdba0144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba00aede0 0x7fdba0144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.198+0000 7fdba4c00640 1 -- 172.21.15.141:0/3589723087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb98009b30 con 0x7fdba006a3e0 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.199+0000 7fdba4c00640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba006a3e0 0x7fdba0141020 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000e9b0 tx=0x7fdb9000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.199+0000 7fdb9cc00640 1 -- 172.21.15.141:0/3589723087 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb90009670 con 0x7fdba006a3e0 2024-01-16T15:00:04.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.199+0000 7fdb9cc00640 1 -- 172.21.15.141:0/3589723087 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9000c9a0 con 0x7fdba006a3e0 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.199+0000 7fdb9cc00640 1 -- 172.21.15.141:0/3589723087 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9000cd50 con 0x7fdba006a3e0 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.199+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba0142180 con 0x7fdba006a3e0 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.199+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba0142740 con 0x7fdba006a3e0 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.200+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba00ac8d0 con 0x7fdba006a3e0 2024-01-16T15:00:04.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.201+0000 7fdb9cc00640 1 -- 172.21.15.141:0/3589723087 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fdb90013470 con 0x7fdba006a3e0 2024-01-16T15:00:04.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.202+0000 7fdb9cc00640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7806b530 0x7fdb7806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.202+0000 7fdb9cc00640 1 -- 172.21.15.141:0/3589723087 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdb9008a390 con 0x7fdba006a3e0 2024-01-16T15:00:04.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.202+0000 7fdb9fe00640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7806b530 0x7fdb7806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.204+0000 7fdb9fe00640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7806b530 0x7fdb7806d9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdba00cc080 tx=0x7fdb88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.206+0000 7fdb9cc00640 1 -- 172.21.15.141:0/3589723087 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb900550c0 con 0x7fdba006a3e0 2024-01-16T15:00:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.393+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdba00675d0 con 0x7fdba006a3e0 2024-01-16T15:00:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.395+0000 7fdb9cc00640 1 -- 172.21.15.141:0/3589723087 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+12905 (secure 0 0 0) 0x7fdb900586e0 con 0x7fdba006a3e0 2024-01-16T15:00:04.376 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:04.377 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":42,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:00:03.468983+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:04.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.397+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7806b530 msgr2=0x7fdb7806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.397+0000 7fdba7600640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7806b530 0x7fdb7806d9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdba00cc080 tx=0x7fdb88009040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.397+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba006a3e0 msgr2=0x7fdba0141020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.397+0000 7fdba7600640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba006a3e0 0x7fdba0141020 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000e9b0 tx=0x7fdb9000ee70 comp rx=0 tx=0).stop 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.397+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 shutdown_connections 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.397+0000 7fdba7600640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7806b530 0x7fdb7806d9f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.398+0000 7fdba7600640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba00d0b60 0x7fdba0142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.398+0000 7fdba7600640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba00aede0 0x7fdba0144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.398+0000 7fdba7600640 1 --2- 172.21.15.141:0/3589723087 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba006a3e0 0x7fdba0141020 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.398+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 >> 172.21.15.141:0/3589723087 conn(0x7fdba00cfef0 msgr2=0x7fdba00ae5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.398+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 shutdown_connections 2024-01-16T15:00:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.398+0000 7fdba7600640 1 -- 172.21.15.141:0/3589723087 wait complete. 2024-01-16T15:00:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:04.400 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:04.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.629+0000 7fa62a600640 1 Processor -- start 2024-01-16T15:00:04.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.629+0000 7fa62a600640 1 -- start start 2024-01-16T15:00:04.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.630+0000 7fa62a600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa62413e360 0x7fa6240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa62a600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62415c5f0 con 0x7fa62413df80 2024-01-16T15:00:04.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa62a600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62406c470 con 0x7fa62415f9c0 2024-01-16T15:00:04.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa62a600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62406c750 con 0x7fa62413e360 2024-01-16T15:00:04.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa623400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa62413df80 0x7fa62415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55172/0 (socket says 172.21.15.141:55172) 2024-01-16T15:00:04.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa623400640 1 -- 172.21.15.141:0/1182894954 learned_addr learned my addr 172.21.15.141:0/1182894954 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360635026 0 0) 0x7fa62415c5f0 con 0x7fa62413df80 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa622a00640 1 --2- 172.21.15.141:0/1182894954 >> v2:172.21.15.188:3300/0 conn(0x7fa62413e360 0x7fa6240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.631+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa604003520 con 0x7fa62413df80 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 841878667 0 0) 0x7fa604003520 con 0x7fa62413df80 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 >> v2:172.21.15.188:3300/0 conn(0x7fa62413e360 msgr2=0x7fa6240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 --2- 172.21.15.141:0/1182894954 >> v2:172.21.15.188:3300/0 conn(0x7fa62413e360 0x7fa6240cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 >> v1:172.21.15.188:6789/0 conn(0x7fa62415f9c0 legacy=0x7fa6240cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62406ca30 con 0x7fa62413df80 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa61c001a10 con 0x7fa62413df80 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 >> v1:172.21.15.141:6789/0 conn(0x7fa62413df80 legacy=0x7fa62415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa604003970 0x7fa604003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6040047c0 0x7fa604004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa604005180 0x7fa60400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62406c750 con 0x7fa604003970 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c020 con 0x7fa6040047c0 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c1e0 con 0x7fa604005180 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa623400640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa604003970 0x7fa604003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa622a00640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6040047c0 0x7fa604004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.632+0000 7fa623400640 1 -- 172.21.15.141:0/1182894954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa604005180 msgr2=0x7fa60400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.633+0000 7fa623400640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa604005180 0x7fa60400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.633+0000 7fa623400640 1 -- 172.21.15.141:0/1182894954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6040047c0 msgr2=0x7fa604004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.633+0000 7fa623400640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6040047c0 0x7fa604004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.633+0000 7fa623400640 1 -- 172.21.15.141:0/1182894954 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604003520 con 0x7fa604003970 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.633+0000 7fa623e00640 1 -- 172.21.15.141:0/1182894954 reap_dead start 2024-01-16T15:00:04.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa623400640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa604003970 0x7fa604003d90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa62415c780 tx=0x7fa6040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c283588b704f323 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c034630 con 0x7fa604003970 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c02f070 con 0x7fa604003970 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa622000640 1 -- 172.21.15.141:0/1182894954 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c034a30 con 0x7fa604003970 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa62a600640 1 -- 172.21.15.141:0/1182894954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa604003970 msgr2=0x7fa604003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa62a600640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa604003970 0x7fa604003d90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa62415c780 tx=0x7fa6040057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa62a600640 1 -- 172.21.15.141:0/1182894954 shutdown_connections 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa62a600640 1 --2- 172.21.15.141:0/1182894954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa604003970 0x7fa604003d90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.634+0000 7fa62a600640 1 -- 172.21.15.141:0/1182894954 >> 172.21.15.141:0/1182894954 conn(0x7fa62406a800 msgr2=0x7fa62406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.635+0000 7fa62a600640 1 -- 172.21.15.141:0/1182894954 shutdown_connections 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.635+0000 7fa62a600640 1 -- 172.21.15.141:0/1182894954 wait complete. 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.636+0000 7fa62a600640 1 Processor -- start 2024-01-16T15:00:04.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.636+0000 7fa62a600640 1 -- start start 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.637+0000 7fa62a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa604003970 0x7fa62413c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.637+0000 7fa62a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6240ad270 0x7fa6240ab320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.637+0000 7fa62a600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa624152cd0 0x7fa624153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.637+0000 7fa62a600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240b1ca0 con 0x7fa624152cd0 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.637+0000 7fa62a600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6241536c0 con 0x7fa604003970 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.637+0000 7fa62a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62413d1d0 con 0x7fa6240ad270 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa624152cd0 0x7fa624153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa624152cd0 0x7fa624153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39152/0 (socket says 172.21.15.141:39152) 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 -- 172.21.15.141:0/3378492241 learned_addr learned my addr 172.21.15.141:0/3378492241 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:04.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623400640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa604003970 0x7fa62413c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa622a00640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6240ad270 0x7fa6240ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 -- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6240ad270 msgr2=0x7fa6240ab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6240ad270 0x7fa6240ab320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 -- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa604003970 msgr2=0x7fa62413c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa604003970 0x7fa62413c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 -- 172.21.15.141:0/3378492241 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62413d4b0 con 0x7fa624152cd0 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa622a00640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6240ad270 0x7fa6240ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623400640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa604003970 0x7fa62413c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.638+0000 7fa623e00640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa624152cd0 0x7fa624153180 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa6240cfea0 tx=0x7fa61400c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.639+0000 7fa61b400640 1 -- 172.21.15.141:0/3378492241 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa614004220 con 0x7fa624152cd0 2024-01-16T15:00:04.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.639+0000 7fa61b400640 1 -- 172.21.15.141:0/3378492241 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6140043a0 con 0x7fa624152cd0 2024-01-16T15:00:04.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.639+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62413d790 con 0x7fa624152cd0 2024-01-16T15:00:04.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.639+0000 7fa61b400640 1 -- 172.21.15.141:0/3378492241 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa614007310 con 0x7fa624152cd0 2024-01-16T15:00:04.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.639+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62413ce10 con 0x7fa624152cd0 2024-01-16T15:00:04.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.640+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6240b1e50 con 0x7fa624152cd0 2024-01-16T15:00:04.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.641+0000 7fa61b400640 1 -- 172.21.15.141:0/3378492241 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa614007470 con 0x7fa624152cd0 2024-01-16T15:00:04.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.641+0000 7fa61b400640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5fc06b530 0x7fa5fc06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.642+0000 7fa61b400640 1 -- 172.21.15.141:0/3378492241 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa614088ba0 con 0x7fa624152cd0 2024-01-16T15:00:04.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.642+0000 7fa623400640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5fc06b530 0x7fa5fc06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.643+0000 7fa623400640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5fc06b530 0x7fa5fc06d9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa61c034000 tx=0x7fa61c02f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.646+0000 7fa61b400640 1 -- 172.21.15.141:0/3378492241 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6240b1e50 con 0x7fa624152cd0 2024-01-16T15:00:04.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.837+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa62413cfa0 con 0x7fa624152cd0 2024-01-16T15:00:04.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.838+0000 7fa61b400640 1 -- 172.21.15.141:0/3378492241 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa614053a90 con 0x7fa624152cd0 2024-01-16T15:00:04.820 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5fc06b530 msgr2=0x7fa5fc06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5fc06b530 0x7fa5fc06d9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa61c034000 tx=0x7fa61c02f000 comp rx=0 tx=0).stop 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa624152cd0 msgr2=0x7fa624153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa624152cd0 0x7fa624153180 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa6240cfea0 tx=0x7fa61400c5f0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 shutdown_connections 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5fc06b530 0x7fa5fc06d9f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa624152cd0 0x7fa624153180 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6240ad270 0x7fa6240ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 --2- 172.21.15.141:0/3378492241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa604003970 0x7fa62413c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 >> 172.21.15.141:0/3378492241 conn(0x7fa62406a800 msgr2=0x7fa62415dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 shutdown_connections 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:04.841+0000 7fa62a600640 1 -- 172.21.15.141:0/3378492241 wait complete. 2024-01-16T15:00:04.842 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:00:05.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.061+0000 7fd28f800640 1 Processor -- start 2024-01-16T15:00:05.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.061+0000 7fd28f800640 1 -- start start 2024-01-16T15:00:05.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd28f800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd2880ae5a0 0x7fd28815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd28f800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd28806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd28f800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd288070c40 con 0x7fd28806d120 2024-01-16T15:00:05.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd28f800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd288071450 con 0x7fd2880ae5a0 2024-01-16T15:00:05.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd28f800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28806c470 con 0x7fd2880b3ae0 2024-01-16T15:00:05.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd287e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd28806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd28ce00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd2880ae5a0 0x7fd28815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd287e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd28806cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:41426/0 (socket says 172.21.15.141:41426) 2024-01-16T15:00:05.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd28ce00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd2880ae5a0 0x7fd28815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:35330/0 (socket says 172.21.15.141:35330) 2024-01-16T15:00:05.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.063+0000 7fd287e00640 1 -- 172.21.15.141:0/3391444777 learned_addr learned my addr 172.21.15.141:0/3391444777 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:05.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.064+0000 7fd287e00640 1 -- 172.21.15.141:0/3391444777 >> v2:172.21.15.188:3300/0 conn(0x7fd2880ae5a0 msgr2=0x7fd28815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.064+0000 7fd287e00640 1 --2- 172.21.15.141:0/3391444777 >> v2:172.21.15.188:3300/0 conn(0x7fd2880ae5a0 0x7fd28815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.064+0000 7fd287e00640 1 -- 172.21.15.141:0/3391444777 >> v1:172.21.15.188:6789/0 conn(0x7fd28806d120 legacy=0x7fd2880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.064+0000 7fd287e00640 1 -- 172.21.15.141:0/3391444777 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28806c750 con 0x7fd2880b3ae0 2024-01-16T15:00:05.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.064+0000 7fd28ce00640 1 --2- 172.21.15.141:0/3391444777 >> v2:172.21.15.188:3300/0 conn(0x7fd2880ae5a0 0x7fd28815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:05.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.066+0000 7fd287e00640 1 --2- 172.21.15.141:0/3391444777 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd28806cbe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd270009870 tx=0x7fd27002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2603f7bf04ed049 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.066+0000 7fd287400640 1 -- 172.21.15.141:0/3391444777 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270030840 con 0x7fd2880b3ae0 2024-01-16T15:00:05.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.066+0000 7fd287400640 1 -- 172.21.15.141:0/3391444777 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2700309a0 con 0x7fd2880b3ae0 2024-01-16T15:00:05.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.066+0000 7fd287400640 1 -- 172.21.15.141:0/3391444777 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270030d30 con 0x7fd2880b3ae0 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 -- 172.21.15.141:0/3391444777 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 msgr2=0x7fd28806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 --2- 172.21.15.141:0/3391444777 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd28806cbe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd270009870 tx=0x7fd27002b980 comp rx=0 tx=0).stop 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 -- 172.21.15.141:0/3391444777 shutdown_connections 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 --2- 172.21.15.141:0/3391444777 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd28806cbe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 --2- 172.21.15.141:0/3391444777 >> v2:172.21.15.188:3300/0 conn(0x7fd2880ae5a0 0x7fd28815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 -- 172.21.15.141:0/3391444777 >> 172.21.15.141:0/3391444777 conn(0x7fd28806a800 msgr2=0x7fd28806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 -- 172.21.15.141:0/3391444777 shutdown_connections 2024-01-16T15:00:05.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.067+0000 7fd28f800640 1 -- 172.21.15.141:0/3391444777 wait complete. 2024-01-16T15:00:05.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.069+0000 7fd28f800640 1 Processor -- start 2024-01-16T15:00:05.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.069+0000 7fd28f800640 1 -- start start 2024-01-16T15:00:05.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd28f800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd28806d120 0x7fd2881559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd28f800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2880ae5a0 0x7fd288156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd28f800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd2881567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd28f800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880cbef0 con 0x7fd2880ae5a0 2024-01-16T15:00:05.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd28f800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880cbc10 con 0x7fd28806d120 2024-01-16T15:00:05.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd28f800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880cde00 con 0x7fd2880b3ae0 2024-01-16T15:00:05.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd287e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2880ae5a0 0x7fd288156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd287e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2880ae5a0 0x7fd288156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39158/0 (socket says 172.21.15.141:39158) 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd287e00640 1 -- 172.21.15.141:0/2882057189 learned_addr learned my addr 172.21.15.141:0/2882057189 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.070+0000 7fd28d800640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd2881567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd287e00640 1 -- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 msgr2=0x7fd2881567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd28ce00640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd28806d120 0x7fd2881559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd287e00640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd2881567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd287e00640 1 -- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd28806d120 msgr2=0x7fd2881559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd287e00640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd28806d120 0x7fd2881559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd287e00640 1 -- 172.21.15.141:0/2882057189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd278009a70 con 0x7fd2880ae5a0 2024-01-16T15:00:05.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd287e00640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2880ae5a0 0x7fd288156290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd270009870 tx=0x7fd270004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd284c00640 1 -- 172.21.15.141:0/2882057189 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270039a90 con 0x7fd2880ae5a0 2024-01-16T15:00:05.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd284c00640 1 -- 172.21.15.141:0/2882057189 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd270039c10 con 0x7fd2880ae5a0 2024-01-16T15:00:05.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2700094d0 con 0x7fd2880ae5a0 2024-01-16T15:00:05.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd284c00640 1 -- 172.21.15.141:0/2882057189 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2700433f0 con 0x7fd2880ae5a0 2024-01-16T15:00:05.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.071+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2880ce3c0 con 0x7fd2880ae5a0 2024-01-16T15:00:05.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.073+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2880cdc60 con 0x7fd2880ae5a0 2024-01-16T15:00:05.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.074+0000 7fd284c00640 1 -- 172.21.15.141:0/2882057189 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd27004c3b0 con 0x7fd2880ae5a0 2024-01-16T15:00:05.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.074+0000 7fd284c00640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd26006b500 0x7fd26006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.075+0000 7fd284c00640 1 -- 172.21.15.141:0/2882057189 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd2700ac1d0 con 0x7fd2880ae5a0 2024-01-16T15:00:05.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.075+0000 7fd28ce00640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd26006b500 0x7fd26006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.076+0000 7fd28ce00640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd26006b500 0x7fd26006d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2880b1770 tx=0x7fd27800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.079+0000 7fd284c00640 1 -- 172.21.15.141:0/2882057189 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2880cdc60 con 0x7fd2880ae5a0 2024-01-16T15:00:05.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.273+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd2880cdc60 con 0x7fd2880ae5a0 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.274+0000 7fd284c00640 1 -- 172.21.15.141:0/2882057189 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd2880cdc60 con 0x7fd2880ae5a0 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T15:00:05.257 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:05.257 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:00:05.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd26006b500 msgr2=0x7fd26006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd26006b500 0x7fd26006d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2880b1770 tx=0x7fd27800b040 comp rx=0 tx=0).stop 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2880ae5a0 msgr2=0x7fd288156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2880ae5a0 0x7fd288156290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd270009870 tx=0x7fd270004730 comp rx=0 tx=0).stop 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 shutdown_connections 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd26006b500 0x7fd26006d9c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2880b3ae0 0x7fd2881567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2880ae5a0 0x7fd288156290 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 --2- 172.21.15.141:0/2882057189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd28806d120 0x7fd2881559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 >> 172.21.15.141:0/2882057189 conn(0x7fd28806a800 msgr2=0x7fd2880cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.277+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 shutdown_connections 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.278+0000 7fd28f800640 1 -- 172.21.15.141:0/2882057189 wait complete. 2024-01-16T15:00:05.276 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:00:05.276 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:00:05.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.503+0000 7fc02d000640 1 Processor -- start 2024-01-16T15:00:05.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.503+0000 7fc02d000640 1 -- start start 2024-01-16T15:00:05.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.505+0000 7fc02d000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc02813df80 0x7fc02815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.505+0000 7fc02d000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02806c470 con 0x7fc02813e360 2024-01-16T15:00:05.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.505+0000 7fc02d000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02806c750 con 0x7fc0280cf9a0 2024-01-16T15:00:05.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.505+0000 7fc02d000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02806ca30 con 0x7fc02813df80 2024-01-16T15:00:05.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc01d200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc02813e360 0x7fc0280cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55184/0 (socket says 172.21.15.141:55184) 2024-01-16T15:00:05.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc01d200640 1 -- 172.21.15.141:0/1468232817 learned_addr learned my addr 172.21.15.141:0/1468232817 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:05.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc025e00640 1 --2- 172.21.15.141:0/1468232817 >> v2:172.21.15.141:3300/0 conn(0x7fc02813df80 0x7fc02815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc025400640 1 -- 172.21.15.141:0/1468232817 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877094882 0 0) 0x7fc02806c470 con 0x7fc02813e360 2024-01-16T15:00:05.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc025400640 1 -- 172.21.15.141:0/1468232817 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc008003520 con 0x7fc02813e360 2024-01-16T15:00:05.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc025e00640 1 -- 172.21.15.141:0/1468232817 >> v1:172.21.15.188:6789/0 conn(0x7fc0280cf9a0 legacy=0x7fc0280cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc025e00640 1 -- 172.21.15.141:0/1468232817 >> v1:172.21.15.141:6789/0 conn(0x7fc02813e360 legacy=0x7fc0280cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.506+0000 7fc025e00640 1 -- 172.21.15.141:0/1468232817 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02806cd10 con 0x7fc02813df80 2024-01-16T15:00:05.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.508+0000 7fc025e00640 1 --2- 172.21.15.141:0/1468232817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813df80 0x7fc02815bee0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc0180097a0 tx=0x7fc01802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0930100c4048f9a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc025400640 1 -- 172.21.15.141:0/1468232817 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018030610 con 0x7fc02813df80 2024-01-16T15:00:05.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc025400640 1 -- 172.21.15.141:0/1468232817 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc018030770 con 0x7fc02813df80 2024-01-16T15:00:05.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc025400640 1 -- 172.21.15.141:0/1468232817 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018030b00 con 0x7fc02813df80 2024-01-16T15:00:05.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc02d000640 1 -- 172.21.15.141:0/1468232817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813df80 msgr2=0x7fc02815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc02d000640 1 --2- 172.21.15.141:0/1468232817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813df80 0x7fc02815bee0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc0180097a0 tx=0x7fc01802b960 comp rx=0 tx=0).stop 2024-01-16T15:00:05.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc02d000640 1 -- 172.21.15.141:0/1468232817 shutdown_connections 2024-01-16T15:00:05.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc02d000640 1 --2- 172.21.15.141:0/1468232817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813df80 0x7fc02815bee0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc02d000640 1 -- 172.21.15.141:0/1468232817 >> 172.21.15.141:0/1468232817 conn(0x7fc02806a800 msgr2=0x7fc02806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc02d000640 1 -- 172.21.15.141:0/1468232817 shutdown_connections 2024-01-16T15:00:05.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.509+0000 7fc02d000640 1 -- 172.21.15.141:0/1468232817 wait complete. 2024-01-16T15:00:05.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.511+0000 7fc02d000640 1 Processor -- start 2024-01-16T15:00:05.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.511+0000 7fc02d000640 1 -- start start 2024-01-16T15:00:05.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc02d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0280cf9a0 0x7fc0281e6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc02d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc02813df80 0x7fc0280b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc02d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813e360 0x7fc02813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc02d000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280709e0 con 0x7fc02813e360 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc02d000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280b34a0 con 0x7fc0280cf9a0 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc02d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc028070360 con 0x7fc02813df80 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc026800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813e360 0x7fc02813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.512+0000 7fc026800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813e360 0x7fc02813d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39176/0 (socket says 172.21.15.141:39176) 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc026800640 1 -- 172.21.15.141:0/3033377410 learned_addr learned my addr 172.21.15.141:0/3033377410 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc025e00640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0280cf9a0 0x7fc0281e6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc01d200640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc02813df80 0x7fc0280b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc026800640 1 -- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc02813df80 msgr2=0x7fc0280b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc026800640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc02813df80 0x7fc0280b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc026800640 1 -- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0280cf9a0 msgr2=0x7fc0281e6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc026800640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0280cf9a0 0x7fc0281e6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc026800640 1 -- 172.21.15.141:0/3033377410 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc018009400 con 0x7fc02813e360 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc025e00640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0280cf9a0 0x7fc0281e6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc01d200640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc02813df80 0x7fc0280b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc026800640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813e360 0x7fc02813d730 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c680 tx=0x7fc02000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc01ea00640 1 -- 172.21.15.141:0/3033377410 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020003b70 con 0x7fc02813e360 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc01ea00640 1 -- 172.21.15.141:0/3033377410 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc020003cd0 con 0x7fc02813e360 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.513+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0280b3730 con 0x7fc02813e360 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.514+0000 7fc01ea00640 1 -- 172.21.15.141:0/3033377410 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020005310 con 0x7fc02813e360 2024-01-16T15:00:05.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.514+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02813dc70 con 0x7fc02813e360 2024-01-16T15:00:05.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.515+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc028070b20 con 0x7fc02813e360 2024-01-16T15:00:05.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.515+0000 7fc01ea00640 1 -- 172.21.15.141:0/3033377410 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc020005470 con 0x7fc02813e360 2024-01-16T15:00:05.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.516+0000 7fc01ea00640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc00006b500 0x7fc00006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.516+0000 7fc01ea00640 1 -- 172.21.15.141:0/3033377410 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc020089a80 con 0x7fc02813e360 2024-01-16T15:00:05.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.517+0000 7fc025e00640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc00006b500 0x7fc00006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.518+0000 7fc025e00640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc00006b500 0x7fc00006d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc018036000 tx=0x7fc01803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.521+0000 7fc01ea00640 1 -- 172.21.15.141:0/3033377410 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc028070b20 con 0x7fc02813e360 2024-01-16T15:00:05.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:05.710+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc028070b20 con 0x7fc02813e360 2024-01-16T15:00:06.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.484+0000 7fc01ea00640 1 -- 172.21.15.141:0/3033377410 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v43) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc028070b20 con 0x7fc02813e360 2024-01-16T15:00:06.466 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T15:00:06.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc00006b500 msgr2=0x7fc00006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc00006b500 0x7fc00006d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc018036000 tx=0x7fc01803b040 comp rx=0 tx=0).stop 2024-01-16T15:00:06.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813e360 msgr2=0x7fc02813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813e360 0x7fc02813d730 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c680 tx=0x7fc02000cb40 comp rx=0 tx=0).stop 2024-01-16T15:00:06.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 shutdown_connections 2024-01-16T15:00:06.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc00006b500 0x7fc00006d9c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02813e360 0x7fc02813d730 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc02813df80 0x7fc0280b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 --2- 172.21.15.141:0/3033377410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0280cf9a0 0x7fc0281e6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.489+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 >> 172.21.15.141:0/3033377410 conn(0x7fc02806a800 msgr2=0x7fc0280ce530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.490+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 shutdown_connections 2024-01-16T15:00:06.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.490+0000 7fc02d000640 1 -- 172.21.15.141:0/3033377410 wait complete. 2024-01-16T15:00:06.496 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:00:06.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.716+0000 7f7721600640 1 Processor -- start 2024-01-16T15:00:06.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.716+0000 7f7721600640 1 -- start start 2024-01-16T15:00:06.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.717+0000 7f7721600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f771c0ce490 0x7f771c071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.717+0000 7f7721600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c160fe0 0x7f771c161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.718+0000 7f7721600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c161980 con 0x7f771c0b1f50 2024-01-16T15:00:06.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.718+0000 7f7721600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c070bf0 con 0x7f771c0ce490 2024-01-16T15:00:06.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.718+0000 7f7721600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0702e0 con 0x7f771c160fe0 2024-01-16T15:00:06.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.718+0000 7f771aa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f771c0ce490 0x7f771c071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.718+0000 7f771aa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f771c0ce490 0x7f771c071080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:35368/0 (socket says 172.21.15.141:35368) 2024-01-16T15:00:06.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.718+0000 7f771aa00640 1 -- 172.21.15.141:0/1077881898 learned_addr learned my addr 172.21.15.141:0/1077881898 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:06.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.718+0000 7f7711e00640 1 --2- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c160fe0 0x7f771c161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.719+0000 7f771aa00640 1 -- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c160fe0 msgr2=0x7f771c161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.719+0000 7f771aa00640 1 --2- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c160fe0 0x7f771c161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.719+0000 7f771aa00640 1 -- 172.21.15.141:0/1077881898 >> v1:172.21.15.188:6789/0 conn(0x7f771c0b1f50 legacy=0x7f771c0b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.719+0000 7f771aa00640 1 -- 172.21.15.141:0/1077881898 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c0b2cc0 con 0x7f771c0ce490 2024-01-16T15:00:06.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.719+0000 7f771a000640 1 -- 172.21.15.141:0/1077881898 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064539602 0 0) 0x7f771c161980 con 0x7f771c0b1f50 2024-01-16T15:00:06.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.721+0000 7f771aa00640 1 --2- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c0ce490 0x7f771c071080 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f770c0098a0 tx=0x7f770c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea2c38465197d892 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.721+0000 7f771a000640 1 -- 172.21.15.141:0/1077881898 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c030840 con 0x7f771c0ce490 2024-01-16T15:00:06.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.721+0000 7f771a000640 1 -- 172.21.15.141:0/1077881898 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c0309a0 con 0x7f771c0ce490 2024-01-16T15:00:06.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.721+0000 7f771a000640 1 -- 172.21.15.141:0/1077881898 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c030d30 con 0x7f771c0ce490 2024-01-16T15:00:06.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 -- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c0ce490 msgr2=0x7f771c071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 --2- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c0ce490 0x7f771c071080 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f770c0098a0 tx=0x7f770c02b980 comp rx=0 tx=0).stop 2024-01-16T15:00:06.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 -- 172.21.15.141:0/1077881898 shutdown_connections 2024-01-16T15:00:06.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 --2- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c160fe0 0x7f771c161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 --2- 172.21.15.141:0/1077881898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c0ce490 0x7f771c071080 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 -- 172.21.15.141:0/1077881898 >> 172.21.15.141:0/1077881898 conn(0x7f771c0b0be0 msgr2=0x7f771c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 -- 172.21.15.141:0/1077881898 shutdown_connections 2024-01-16T15:00:06.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.722+0000 7f7721600640 1 -- 172.21.15.141:0/1077881898 wait complete. 2024-01-16T15:00:06.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.724+0000 7f7721600640 1 Processor -- start 2024-01-16T15:00:06.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.724+0000 7f7721600640 1 -- start start 2024-01-16T15:00:06.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.725+0000 7f7721600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771c0b1f50 0x7f771c14cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.725+0000 7f7721600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c0ce490 0x7f771c15d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.725+0000 7f7721600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c160fe0 0x7f771c0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.725+0000 7f7721600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1f7530 con 0x7f771c0b1f50 2024-01-16T15:00:06.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.725+0000 7f7721600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1f32c0 con 0x7f771c160fe0 2024-01-16T15:00:06.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.725+0000 7f7721600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1f35a0 con 0x7f771c0ce490 2024-01-16T15:00:06.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771c0b1f50 0x7f771c14cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771c0b1f50 0x7f771c14cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39190/0 (socket says 172.21.15.141:39190) 2024-01-16T15:00:06.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 -- 172.21.15.141:0/152210689 learned_addr learned my addr 172.21.15.141:0/152210689 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:06.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f7711e00640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c0ce490 0x7f771c15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771b400640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c160fe0 0x7f771c0afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 -- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c0ce490 msgr2=0x7f771c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c0ce490 0x7f771c15d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 -- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c160fe0 msgr2=0x7f771c0afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c160fe0 0x7f771c0afaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.726+0000 7f771aa00640 1 -- 172.21.15.141:0/152210689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c0094d0 con 0x7f771c0b1f50 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.727+0000 7f771aa00640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771c0b1f50 0x7f771c14cd90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f770c000c00 tx=0x7f770c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.727+0000 7f7713400640 1 -- 172.21.15.141:0/152210689 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c039c90 con 0x7f771c0b1f50 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.727+0000 7f7713400640 1 -- 172.21.15.141:0/152210689 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c039df0 con 0x7f771c0b1f50 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.727+0000 7f7713400640 1 -- 172.21.15.141:0/152210689 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c043640 con 0x7f771c0b1f50 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.727+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c1f3880 con 0x7f771c0b1f50 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.727+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1f3e40 con 0x7f771c0b1f50 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.728+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c1543b0 con 0x7f771c0b1f50 2024-01-16T15:00:06.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.729+0000 7f7713400640 1 -- 172.21.15.141:0/152210689 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f770c0437a0 con 0x7f771c0b1f50 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.730+0000 7f7713400640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76f406b530 0x7f76f406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.730+0000 7f7713400640 1 -- 172.21.15.141:0/152210689 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f770c03e070 con 0x7f771c0b1f50 2024-01-16T15:00:06.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.731+0000 7f7711e00640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76f406b530 0x7f76f406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.732+0000 7f7711e00640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76f406b530 0x7f76f406d9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f771c1540e0 tx=0x7f7704008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.735+0000 7f7713400640 1 -- 172.21.15.141:0/152210689 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f770c077b20 con 0x7f771c0b1f50 2024-01-16T15:00:06.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:06.919+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f771c067670 con 0x7f771c0b1f50 2024-01-16T15:00:07.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.490+0000 7f7713400640 1 -- 172.21.15.141:0/152210689 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v44) v1 ==== 170+0+0 (secure 0 0 0) 0x7f770c0340d0 con 0x7f771c0b1f50 2024-01-16T15:00:07.472 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T15:00:07.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76f406b530 msgr2=0x7f76f406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76f406b530 0x7f76f406d9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f771c1540e0 tx=0x7f7704008040 comp rx=0 tx=0).stop 2024-01-16T15:00:07.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771c0b1f50 msgr2=0x7f771c14cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771c0b1f50 0x7f771c14cd90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f770c000c00 tx=0x7f770c004730 comp rx=0 tx=0).stop 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 shutdown_connections 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76f406b530 0x7f76f406d9f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771c160fe0 0x7f771c0afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771c0ce490 0x7f771c15d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 --2- 172.21.15.141:0/152210689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771c0b1f50 0x7f771c14cd90 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 >> 172.21.15.141:0/152210689 conn(0x7f771c0b0be0 msgr2=0x7f771c15a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 shutdown_connections 2024-01-16T15:00:07.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.494+0000 7f7721600640 1 -- 172.21.15.141:0/152210689 wait complete. 2024-01-16T15:00:07.507 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:00:07.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.703+0000 7feb07800640 1 Processor -- start 2024-01-16T15:00:07.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.703+0000 7feb07800640 1 -- start start 2024-01-16T15:00:07.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb07800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb0015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb07800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7feb000cf9a0 0x7feb000cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb07800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0006c470 con 0x7feb0013e360 2024-01-16T15:00:07.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb07800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0006c750 con 0x7feb000cf9a0 2024-01-16T15:00:07.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb07800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0006ca30 con 0x7feb0013df80 2024-01-16T15:00:07.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb05800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7feb000cf9a0 0x7feb000cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb04e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb0015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb05800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7feb000cf9a0 0x7feb000cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:35380/0 (socket says 172.21.15.141:35380) 2024-01-16T15:00:07.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.705+0000 7feb05800640 1 -- 172.21.15.141:0/4123840098 learned_addr learned my addr 172.21.15.141:0/4123840098 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:07.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.706+0000 7feb04e00640 1 -- 172.21.15.141:0/4123840098 >> v2:172.21.15.188:3300/0 conn(0x7feb000cf9a0 msgr2=0x7feb000cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.706+0000 7feb04e00640 1 --2- 172.21.15.141:0/4123840098 >> v2:172.21.15.188:3300/0 conn(0x7feb000cf9a0 0x7feb000cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.706+0000 7feb04e00640 1 -- 172.21.15.141:0/4123840098 >> v1:172.21.15.188:6789/0 conn(0x7feb0013e360 legacy=0x7feb000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.706+0000 7feb04e00640 1 -- 172.21.15.141:0/4123840098 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0006cd10 con 0x7feb0013df80 2024-01-16T15:00:07.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.708+0000 7feb04e00640 1 --2- 172.21.15.141:0/4123840098 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb0015bee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009d90 tx=0x7feaf002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec0456567172fcbc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.708+0000 7feaff400640 1 -- 172.21.15.141:0/4123840098 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf002cc50 con 0x7feb0013df80 2024-01-16T15:00:07.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.708+0000 7feaff400640 1 -- 172.21.15.141:0/4123840098 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf002cdb0 con 0x7feb0013df80 2024-01-16T15:00:07.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.708+0000 7feaff400640 1 -- 172.21.15.141:0/4123840098 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0034660 con 0x7feb0013df80 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 -- 172.21.15.141:0/4123840098 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 msgr2=0x7feb0015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 --2- 172.21.15.141:0/4123840098 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb0015bee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009d90 tx=0x7feaf002bea0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 -- 172.21.15.141:0/4123840098 shutdown_connections 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 --2- 172.21.15.141:0/4123840098 >> v2:172.21.15.188:3300/0 conn(0x7feb000cf9a0 0x7feb000cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 --2- 172.21.15.141:0/4123840098 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb0015bee0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 -- 172.21.15.141:0/4123840098 >> 172.21.15.141:0/4123840098 conn(0x7feb0006a800 msgr2=0x7feb0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 -- 172.21.15.141:0/4123840098 shutdown_connections 2024-01-16T15:00:07.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.709+0000 7feb07800640 1 -- 172.21.15.141:0/4123840098 wait complete. 2024-01-16T15:00:07.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.711+0000 7feb07800640 1 Processor -- start 2024-01-16T15:00:07.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.711+0000 7feb07800640 1 -- start start 2024-01-16T15:00:07.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb07800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb000cf9a0 0x7feb001381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb07800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb001e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb07800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb0013e360 0x7feb0015dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb07800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb000710e0 con 0x7feb0013e360 2024-01-16T15:00:07.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb07800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001e5480 con 0x7feb000cf9a0 2024-01-16T15:00:07.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb07800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb001e5760 con 0x7feb0013df80 2024-01-16T15:00:07.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb0013e360 0x7feb0015dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb0013e360 0x7feb0015dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39206/0 (socket says 172.21.15.141:39206) 2024-01-16T15:00:07.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 -- 172.21.15.141:0/1279916503 learned_addr learned my addr 172.21.15.141:0/1279916503 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 -- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 msgr2=0x7feb001e4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb001e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 -- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb000cf9a0 msgr2=0x7feb001381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb000cf9a0 0x7feb001381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.712+0000 7feb05800640 1 -- 172.21.15.141:0/1279916503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf0009a70 con 0x7feb0013e360 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.713+0000 7feb05800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb0013e360 0x7feb0015dbe0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7feaf800e870 tx=0x7feaf800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.713+0000 7feafcc00640 1 -- 172.21.15.141:0/1279916503 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf800ee60 con 0x7feb0013e360 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.713+0000 7feafcc00640 1 -- 172.21.15.141:0/1279916503 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf800c940 con 0x7feb0013e360 2024-01-16T15:00:07.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.713+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb001e5a40 con 0x7feb0013e360 2024-01-16T15:00:07.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.713+0000 7feafcc00640 1 -- 172.21.15.141:0/1279916503 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf800ccf0 con 0x7feb0013e360 2024-01-16T15:00:07.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.713+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb001e4f60 con 0x7feb0013e360 2024-01-16T15:00:07.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.714+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb000675d0 con 0x7feb0013e360 2024-01-16T15:00:07.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.715+0000 7feafcc00640 1 -- 172.21.15.141:0/1279916503 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7feaf8013470 con 0x7feb0013e360 2024-01-16T15:00:07.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.716+0000 7feafcc00640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fead806b500 0x7fead806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.716+0000 7feafcc00640 1 -- 172.21.15.141:0/1279916503 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5609+0+0 (secure 0 0 0) 0x7feaf808a800 con 0x7feb0013e360 2024-01-16T15:00:07.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.717+0000 7feb04e00640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fead806b500 0x7fead806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.718+0000 7feb04e00640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fead806b500 0x7fead806d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feaf0036000 tx=0x7feaf002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.720+0000 7feafcc00640 1 -- 172.21.15.141:0/1279916503 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feaf80553e0 con 0x7feb0013e360 2024-01-16T15:00:07.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:07.941+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feb00071220 con 0x7feb0013e360 2024-01-16T15:00:08.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.531+0000 7feafcc00640 1 -- 172.21.15.141:0/1279916503 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v23) v1 ==== 172+0+0 (secure 0 0 0) 0x7feb00071220 con 0x7feb0013e360 2024-01-16T15:00:08.515 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 6 and data pool 7 2024-01-16T15:00:08.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.539+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fead806b500 msgr2=0x7fead806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.539+0000 7feb07800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fead806b500 0x7fead806d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feaf0036000 tx=0x7feaf002e000 comp rx=0 tx=0).stop 2024-01-16T15:00:08.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.539+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb0013e360 msgr2=0x7feb0015dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.539+0000 7feb07800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb0013e360 0x7feb0015dbe0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7feaf800e870 tx=0x7feaf800ed30 comp rx=0 tx=0).stop 2024-01-16T15:00:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.540+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 shutdown_connections 2024-01-16T15:00:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.540+0000 7feb07800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fead806b500 0x7fead806d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.540+0000 7feb07800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feb0013e360 0x7feb0015dbe0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.540+0000 7feb07800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feb0013df80 0x7feb001e4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.540+0000 7feb07800640 1 --2- 172.21.15.141:0/1279916503 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb000cf9a0 0x7feb001381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.540+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 >> 172.21.15.141:0/1279916503 conn(0x7feb0006a800 msgr2=0x7feb000aeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.542+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 shutdown_connections 2024-01-16T15:00:08.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.542+0000 7feb07800640 1 -- 172.21.15.141:0/1279916503 wait complete. 2024-01-16T15:00:08.556 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:08.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.749+0000 7f237b200640 1 Processor -- start 2024-01-16T15:00:08.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.749+0000 7f237b200640 1 -- start start 2024-01-16T15:00:08.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.750+0000 7f237b200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f237c0cc500 0x7f237c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.750+0000 7f237b200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c070e50 con 0x7f237c143c40 2024-01-16T15:00:08.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f237b200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c144020 con 0x7f237c06d160 2024-01-16T15:00:08.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f237b200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0723c0 con 0x7f237c0cc500 2024-01-16T15:00:08.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f2379400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f237c0cc500 0x7f237c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f2379e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f237c143c40 0x7f237c071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55198/0 (socket says 172.21.15.141:55198) 2024-01-16T15:00:08.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f2379e00640 1 -- 172.21.15.141:0/2826460403 learned_addr learned my addr 172.21.15.141:0/2826460403 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:08.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f2379400640 1 -- 172.21.15.141:0/2826460403 >> v1:172.21.15.188:6789/0 conn(0x7f237c06d160 legacy=0x7f237c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f2379400640 1 -- 172.21.15.141:0/2826460403 >> v1:172.21.15.141:6789/0 conn(0x7f237c143c40 legacy=0x7f237c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f2379400640 1 -- 172.21.15.141:0/2826460403 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c071e00 con 0x7f237c0cc500 2024-01-16T15:00:08.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.751+0000 7f2378a00640 1 -- 172.21.15.141:0/2826460403 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029656364 0 0) 0x7f237c070e50 con 0x7f237c143c40 2024-01-16T15:00:08.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.752+0000 7f2379400640 1 --2- 172.21.15.141:0/2826460403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c0cc500 0x7f237c0cc990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f23640097a0 tx=0x7f236402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f28b330b5ab040 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f2378a00640 1 -- 172.21.15.141:0/2826460403 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236402e070 con 0x7f237c0cc500 2024-01-16T15:00:08.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f2378a00640 1 -- 172.21.15.141:0/2826460403 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23640315a0 con 0x7f237c0cc500 2024-01-16T15:00:08.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f2378a00640 1 -- 172.21.15.141:0/2826460403 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364031930 con 0x7f237c0cc500 2024-01-16T15:00:08.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f237b200640 1 -- 172.21.15.141:0/2826460403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c0cc500 msgr2=0x7f237c0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f237b200640 1 --2- 172.21.15.141:0/2826460403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c0cc500 0x7f237c0cc990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f23640097a0 tx=0x7f236402b840 comp rx=0 tx=0).stop 2024-01-16T15:00:08.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f237b200640 1 -- 172.21.15.141:0/2826460403 shutdown_connections 2024-01-16T15:00:08.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f237b200640 1 --2- 172.21.15.141:0/2826460403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c0cc500 0x7f237c0cc990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.753+0000 7f237b200640 1 -- 172.21.15.141:0/2826460403 >> 172.21.15.141:0/2826460403 conn(0x7f237c06a800 msgr2=0x7f237c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.754+0000 7f237b200640 1 -- 172.21.15.141:0/2826460403 shutdown_connections 2024-01-16T15:00:08.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.754+0000 7f237b200640 1 -- 172.21.15.141:0/2826460403 wait complete. 2024-01-16T15:00:08.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.755+0000 7f237b200640 1 Processor -- start 2024-01-16T15:00:08.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.755+0000 7f237b200640 1 -- start start 2024-01-16T15:00:08.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.756+0000 7f237b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c06d160 0x7f237c1f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.756+0000 7f237b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f237c0cc500 0x7f237c13b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.756+0000 7f237b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f237c143c40 0x7f237c1419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.756+0000 7f237b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c13b0d0 con 0x7f237c06d160 2024-01-16T15:00:08.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.756+0000 7f237b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1f35e0 con 0x7f237c143c40 2024-01-16T15:00:08.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.756+0000 7f237b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1f38c0 con 0x7f237c0cc500 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.756+0000 7f2379e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c06d160 0x7f237c1f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c06d160 0x7f237c1f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39224/0 (socket says 172.21.15.141:39224) 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 -- 172.21.15.141:0/314045070 learned_addr learned my addr 172.21.15.141:0/314045070 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 -- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f237c0cc500 msgr2=0x7f237c13b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379400640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f237c0cc500 0x7f237c13b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f237c0cc500 0x7f237c13b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 -- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f237c143c40 msgr2=0x7f237c1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f237a800640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f237c143c40 0x7f237c1419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f237c143c40 0x7f237c1419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 -- 172.21.15.141:0/314045070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009400 con 0x7f237c06d160 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.757+0000 7f2379e00640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c06d160 0x7f237c1f2d10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c890 tx=0x7f236c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.758+0000 7f2372000640 1 -- 172.21.15.141:0/314045070 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c00a640 con 0x7f237c06d160 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.758+0000 7f2372000640 1 -- 172.21.15.141:0/314045070 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c00a7c0 con 0x7f237c06d160 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.758+0000 7f2372000640 1 -- 172.21.15.141:0/314045070 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c00ab70 con 0x7f237c06d160 2024-01-16T15:00:08.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.758+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c1f3ba0 con 0x7f237c06d160 2024-01-16T15:00:08.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.758+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c1f3250 con 0x7f237c06d160 2024-01-16T15:00:08.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.759+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c067450 con 0x7f237c06d160 2024-01-16T15:00:08.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.760+0000 7f2372000640 1 -- 172.21.15.141:0/314045070 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f236c00acd0 con 0x7f237c06d160 2024-01-16T15:00:08.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.761+0000 7f2372000640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f235406b500 0x7f235406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.761+0000 7f2372000640 1 -- 172.21.15.141:0/314045070 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f236c089520 con 0x7f237c06d160 2024-01-16T15:00:08.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.762+0000 7f2379400640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f235406b500 0x7f235406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.763+0000 7f2379400640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f235406b500 0x7f235406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2364036000 tx=0x7f236403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.766+0000 7f2372000640 1 -- 172.21.15.141:0/314045070 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f236c054020 con 0x7f237c06d160 2024-01-16T15:00:08.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.952+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f237c0b3410 con 0x7f237c06d160 2024-01-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.955+0000 7f2372000640 1 -- 172.21.15.141:0/314045070 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16574 (secure 0 0 0) 0x7f236c057640 con 0x7f237c06d160 2024-01-16T15:00:08.937 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:08.937 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":45,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:00:07.943885+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:00:05.761243+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:00:06.970207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f235406b500 msgr2=0x7f235406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f235406b500 0x7f235406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2364036000 tx=0x7f236403d040 comp rx=0 tx=0).stop 2024-01-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c06d160 msgr2=0x7f237c1f2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c06d160 0x7f237c1f2d10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c890 tx=0x7f236c00cd50 comp rx=0 tx=0).stop 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 shutdown_connections 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f235406b500 0x7f235406d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f237c143c40 0x7f237c1419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f237c0cc500 0x7f237c13b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 --2- 172.21.15.141:0/314045070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f237c06d160 0x7f237c1f2d10 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.957+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 >> 172.21.15.141:0/314045070 conn(0x7f237c06a800 msgr2=0x7f237c13e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.958+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 shutdown_connections 2024-01-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:08.958+0000 7f237b200640 1 -- 172.21.15.141:0/314045070 wait complete. 2024-01-16T15:00:08.957 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:09.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.180+0000 7f8e64200640 1 Processor -- start 2024-01-16T15:00:09.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.180+0000 7f8e64200640 1 -- start start 2024-01-16T15:00:09.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.182+0000 7f8e64200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 0x7f8e5c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.182+0000 7f8e64200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8e5c0cf9a0 0x7f8e5c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.182+0000 7f8e64200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c06c470 con 0x7f8e5c13e360 2024-01-16T15:00:09.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.182+0000 7f8e64200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c06c750 con 0x7f8e5c0cf9a0 2024-01-16T15:00:09.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.182+0000 7f8e64200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c06ca30 con 0x7f8e5c13df80 2024-01-16T15:00:09.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.182+0000 7f8e60e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8e5c13e360 0x7f8e5c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55214/0 (socket says 172.21.15.141:55214) 2024-01-16T15:00:09.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.182+0000 7f8e60e00640 1 -- 172.21.15.141:0/1040662441 learned_addr learned my addr 172.21.15.141:0/1040662441 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e61800640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 0x7f8e5c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e62200640 1 --2- 172.21.15.141:0/1040662441 >> v2:172.21.15.188:3300/0 conn(0x7f8e5c0cf9a0 0x7f8e5c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908590023 0 0) 0x7f8e5c06c470 con 0x7f8e5c13e360 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3c003520 con 0x7f8e5c13e360 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2322643589 0 0) 0x7f8e3c003520 con 0x7f8e5c13e360 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 msgr2=0x7f8e5c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 0x7f8e5c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 >> v2:172.21.15.188:3300/0 conn(0x7f8e5c0cf9a0 msgr2=0x7f8e5c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 --2- 172.21.15.141:0/1040662441 >> v2:172.21.15.188:3300/0 conn(0x7f8e5c0cf9a0 0x7f8e5c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c06cd10 con 0x7f8e5c13e360 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8e4c001670 con 0x7f8e5c13e360 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 >> v1:172.21.15.141:6789/0 conn(0x7f8e5c13e360 legacy=0x7f8e5c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3c003970 0x7f8e3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 0x7f8e3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c004d00 0x7f8e3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c06c750 con 0x7f8e3c003970 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c06ca30 con 0x7f8e3c004d00 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.183+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c00b980 con 0x7f8e5c13df80 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.184+0000 7f8e61800640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3c003970 0x7f8e3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.184+0000 7f8e60e00640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 0x7f8e3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.184+0000 7f8e61800640 1 -- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 msgr2=0x7f8e3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.184+0000 7f8e61800640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c13df80 0x7f8e3c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.184+0000 7f8e61800640 1 -- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c004d00 msgr2=0x7f8e3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.184+0000 7f8e61800640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c004d00 0x7f8e3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.184+0000 7f8e61800640 1 -- 172.21.15.141:0/1040662441 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c003520 con 0x7f8e3c003970 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.185+0000 7f8e61800640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3c003970 0x7f8e3c003d90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009ac0 tx=0x7f8e3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e306b9ba1106020 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.185+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4402c6b0 con 0x7f8e3c003970 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.185+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4402c810 con 0x7f8e3c003970 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.185+0000 7f8e5be00640 1 -- 172.21.15.141:0/1040662441 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4402cba0 con 0x7f8e3c003970 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.186+0000 7f8e64200640 1 -- 172.21.15.141:0/1040662441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3c003970 msgr2=0x7f8e3c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.186+0000 7f8e64200640 1 --2- 172.21.15.141:0/1040662441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3c003970 0x7f8e3c003d90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009ac0 tx=0x7f8e3c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.186+0000 7f8e62200640 1 -- 172.21.15.141:0/1040662441 reap_dead start 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.186+0000 7f8e64200640 1 -- 172.21.15.141:0/1040662441 shutdown_connections 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.186+0000 7f8e64200640 1 -- 172.21.15.141:0/1040662441 >> 172.21.15.141:0/1040662441 conn(0x7f8e5c06a800 msgr2=0x7f8e5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.186+0000 7f8e64200640 1 -- 172.21.15.141:0/1040662441 shutdown_connections 2024-01-16T15:00:09.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.186+0000 7f8e64200640 1 -- 172.21.15.141:0/1040662441 wait complete. 2024-01-16T15:00:09.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.188+0000 7f8e64200640 1 Processor -- start 2024-01-16T15:00:09.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.188+0000 7f8e64200640 1 -- start start 2024-01-16T15:00:09.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e64200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c003970 0x7f8e5c0ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e64200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e5c13d7b0 0x7f8e5c0b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e64200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c0b3d70 0x7f8e5c1f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e64200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c070940 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e64200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0b41f0 con 0x7f8e3c003970 2024-01-16T15:00:09.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e64200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c156580 con 0x7f8e5c0b3d70 2024-01-16T15:00:09.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e5c13d7b0 0x7f8e5c0b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e5c13d7b0 0x7f8e5c0b3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39242/0 (socket says 172.21.15.141:39242) 2024-01-16T15:00:09.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e60e00640 1 -- 172.21.15.141:0/3112992833 learned_addr learned my addr 172.21.15.141:0/3112992833 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:09.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e61800640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c003970 0x7f8e5c0ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.189+0000 7f8e62200640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c0b3d70 0x7f8e5c1f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e60e00640 1 -- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c0b3d70 msgr2=0x7f8e5c1f2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e60e00640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c0b3d70 0x7f8e5c1f2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e60e00640 1 -- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c003970 msgr2=0x7f8e5c0ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e60e00640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c003970 0x7f8e5c0ad2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e60e00640 1 -- 172.21.15.141:0/3112992833 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0702c0 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e60e00640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e5c13d7b0 0x7f8e5c0b3830 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00d7c0 tx=0x7f8e4c00dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e59600640 1 -- 172.21.15.141:0/3112992833 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0074e0 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e59600640 1 -- 172.21.15.141:0/3112992833 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c007640 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e59600640 1 -- 172.21.15.141:0/3112992833 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c004a00 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.190+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c1567d0 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.191+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c070560 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.192+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c070a80 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.192+0000 7f8e59600640 1 -- 172.21.15.141:0/3112992833 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f8e4c004b60 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.193+0000 7f8e59600640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e3406b530 0x7f8e3406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.193+0000 7f8e59600640 1 -- 172.21.15.141:0/3112992833 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e4c0899d0 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.194+0000 7f8e61800640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e3406b530 0x7f8e3406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.195+0000 7f8e61800640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e3406b530 0x7f8e3406d9f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8e44036000 tx=0x7f8e44030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.197+0000 7f8e59600640 1 -- 172.21.15.141:0/3112992833 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e5c070a80 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.389+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e5c070a80 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.391+0000 7f8e59600640 1 -- 172.21.15.141:0/3112992833 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16574 (secure 0 0 0) 0x7f8e5c070a80 con 0x7f8e5c13d7b0 2024-01-16T15:00:09.373 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:09.373 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":45,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:00:07.943885+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:00:05.761243+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:00:06.970207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e3406b530 msgr2=0x7f8e3406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e3406b530 0x7f8e3406d9f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8e44036000 tx=0x7f8e44030000 comp rx=0 tx=0).stop 2024-01-16T15:00:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e5c13d7b0 msgr2=0x7f8e5c0b3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e5c13d7b0 0x7f8e5c0b3830 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00d7c0 tx=0x7f8e4c00dc80 comp rx=0 tx=0).stop 2024-01-16T15:00:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 shutdown_connections 2024-01-16T15:00:09.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e3406b530 0x7f8e3406d9f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e5c0b3d70 0x7f8e5c1f2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e5c13d7b0 0x7f8e5c0b3830 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 --2- 172.21.15.141:0/3112992833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e3c003970 0x7f8e5c0ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.394+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 >> 172.21.15.141:0/3112992833 conn(0x7f8e5c06a800 msgr2=0x7f8e5c0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.395+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 shutdown_connections 2024-01-16T15:00:09.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.395+0000 7f8e64200640 1 -- 172.21.15.141:0/3112992833 wait complete. 2024-01-16T15:00:09.395 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:00:09.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.632+0000 7f2903c00640 1 Processor -- start 2024-01-16T15:00:09.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.632+0000 7f2903c00640 1 -- start start 2024-01-16T15:00:09.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.634+0000 7f2903c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.634+0000 7f2903c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f28fc13e360 0x7f28fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.634+0000 7f2903c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc06c470 con 0x7f28fc15f9c0 2024-01-16T15:00:09.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.634+0000 7f2903c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc06c750 con 0x7f28fc13e360 2024-01-16T15:00:09.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.634+0000 7f2903c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc06ca30 con 0x7f28fc13df80 2024-01-16T15:00:09.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.634+0000 7f28fbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f28fc13e360 0x7f28fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.634+0000 7f2901c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f28fc15f9c0 0x7f28fc1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55226/0 (socket says 172.21.15.141:55226) 2024-01-16T15:00:09.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.635+0000 7f2901c00640 1 -- 172.21.15.141:0/1175722900 learned_addr learned my addr 172.21.15.141:0/1175722900 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:09.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.635+0000 7f2901200640 1 --2- 172.21.15.141:0/1175722900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.635+0000 7f28fbe00640 1 -- 172.21.15.141:0/1175722900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 msgr2=0x7f28fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.635+0000 7f28fbe00640 1 --2- 172.21.15.141:0/1175722900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.635+0000 7f28fbe00640 1 -- 172.21.15.141:0/1175722900 >> v1:172.21.15.141:6789/0 conn(0x7f28fc15f9c0 legacy=0x7f28fc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.635+0000 7f28fbe00640 1 -- 172.21.15.141:0/1175722900 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc06cd10 con 0x7f28fc13e360 2024-01-16T15:00:09.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.636+0000 7f28fbe00640 1 --2- 172.21.15.141:0/1175722900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc0cf290 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f28e40097a0 tx=0x7f28e402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7930d1a9050998 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.636+0000 7f28fb400640 1 -- 172.21.15.141:0/1175722900 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e402d040 con 0x7f28fc13e360 2024-01-16T15:00:09.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.636+0000 7f28fb400640 1 -- 172.21.15.141:0/1175722900 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e4031810 con 0x7f28fc13e360 2024-01-16T15:00:09.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.636+0000 7f28fb400640 1 -- 172.21.15.141:0/1175722900 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4031ba0 con 0x7f28fc13e360 2024-01-16T15:00:09.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 -- 172.21.15.141:0/1175722900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 msgr2=0x7f28fc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 --2- 172.21.15.141:0/1175722900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc0cf290 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f28e40097a0 tx=0x7f28e402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 -- 172.21.15.141:0/1175722900 shutdown_connections 2024-01-16T15:00:09.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 --2- 172.21.15.141:0/1175722900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc0cf290 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 --2- 172.21.15.141:0/1175722900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 -- 172.21.15.141:0/1175722900 >> 172.21.15.141:0/1175722900 conn(0x7f28fc06a800 msgr2=0x7f28fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 -- 172.21.15.141:0/1175722900 shutdown_connections 2024-01-16T15:00:09.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.637+0000 7f2903c00640 1 -- 172.21.15.141:0/1175722900 wait complete. 2024-01-16T15:00:09.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.639+0000 7f2903c00640 1 Processor -- start 2024-01-16T15:00:09.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.639+0000 7f2903c00640 1 -- start start 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f2903c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc132180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f2903c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc14d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f2903c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28fc15f9c0 0x7f28fc14d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f2903c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc150050 con 0x7f28fc13e360 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f2903c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc150330 con 0x7f28fc15f9c0 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f2903c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc15c9f0 con 0x7f28fc13df80 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc14d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc14d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39268/0 (socket says 172.21.15.141:39268) 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 -- 172.21.15.141:0/3858861439 learned_addr learned my addr 172.21.15.141:0/3858861439 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:09.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f2901200640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc132180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 -- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 msgr2=0x7f28fc132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc132180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 -- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28fc15f9c0 msgr2=0x7f28fc14d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28fc15f9c0 0x7f28fc14d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.640+0000 7f28fbe00640 1 -- 172.21.15.141:0/3858861439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4009400 con 0x7f28fc13e360 2024-01-16T15:00:09.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.641+0000 7f28fbe00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc14d1c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f28e40098c0 tx=0x7f28e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.641+0000 7f28f8c00640 1 -- 172.21.15.141:0/3858861439 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e403ad60 con 0x7f28fc13e360 2024-01-16T15:00:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.641+0000 7f28f8c00640 1 -- 172.21.15.141:0/3858861439 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e4044400 con 0x7f28fc13e360 2024-01-16T15:00:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.641+0000 7f28f8c00640 1 -- 172.21.15.141:0/3858861439 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e40447b0 con 0x7f28fc13e360 2024-01-16T15:00:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.641+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc15ccd0 con 0x7f28fc13e360 2024-01-16T15:00:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.641+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc14f290 con 0x7f28fc13e360 2024-01-16T15:00:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.642+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc15c850 con 0x7f28fc13e360 2024-01-16T15:00:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.643+0000 7f28f8c00640 1 -- 172.21.15.141:0/3858861439 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f28e404d440 con 0x7f28fc13e360 2024-01-16T15:00:09.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.644+0000 7f28f8c00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f28d406b500 0x7f28d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.644+0000 7f28f8c00640 1 -- 172.21.15.141:0/3858861439 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28e40ac560 con 0x7f28fc13e360 2024-01-16T15:00:09.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.645+0000 7f2901200640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f28d406b500 0x7f28d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.646+0000 7f2901200640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f28d406b500 0x7f28d406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006fd0 tx=0x7f28ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.648+0000 7f28f8c00640 1 -- 172.21.15.141:0/3858861439 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28fc15c850 con 0x7f28fc13e360 2024-01-16T15:00:09.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:09.867+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f28fc15c850 con 0x7f28fc13e360 2024-01-16T15:00:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.546+0000 7f28f8c00640 1 -- 172.21.15.141:0/3858861439 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7f28fc15c850 con 0x7f28fc13e360 2024-01-16T15:00:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f28d406b500 msgr2=0x7f28d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f28d406b500 0x7f28d406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006fd0 tx=0x7f28ec008040 comp rx=0 tx=0).stop 2024-01-16T15:00:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 msgr2=0x7f28fc14d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc14d1c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f28e40098c0 tx=0x7f28e40046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 shutdown_connections 2024-01-16T15:00:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f28d406b500 0x7f28d406d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28fc15f9c0 0x7f28fc14d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28fc13e360 0x7f28fc14d1c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 --2- 172.21.15.141:0/3858861439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28fc13df80 0x7f28fc132180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.550+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 >> 172.21.15.141:0/3858861439 conn(0x7f28fc06a800 msgr2=0x7f28fc0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:10.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.551+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 shutdown_connections 2024-01-16T15:00:10.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.551+0000 7f2903c00640 1 -- 172.21.15.141:0/3858861439 wait complete. 2024-01-16T15:00:10.554 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:00:10.554 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:00:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.780+0000 7ff90b000640 1 Processor -- start 2024-01-16T15:00:10.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.780+0000 7ff90b000640 1 -- start start 2024-01-16T15:00:10.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.781+0000 7ff90b000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff90413cf10 0x7ff90413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.781+0000 7ff90b000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff90413fae0 0x7ff9040cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.781+0000 7ff90b000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9040ce440 con 0x7ff9040cdc20 2024-01-16T15:00:10.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.781+0000 7ff90b000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90413ff90 con 0x7ff90413cf10 2024-01-16T15:00:10.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.781+0000 7ff90b000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9040ce720 con 0x7ff90413fae0 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff909c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff90413cf10 0x7ff90413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff90a600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff9040cdc20 0x7ff90415d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55230/0 (socket says 172.21.15.141:55230) 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff90a600640 1 -- 172.21.15.141:0/1676617509 learned_addr learned my addr 172.21.15.141:0/1676617509 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff909200640 1 --2- 172.21.15.141:0/1676617509 >> v2:172.21.15.188:3300/0 conn(0x7ff90413fae0 0x7ff9040cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff909c00640 1 -- 172.21.15.141:0/1676617509 >> v2:172.21.15.188:3300/0 conn(0x7ff90413fae0 msgr2=0x7ff9040cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff909c00640 1 --2- 172.21.15.141:0/1676617509 >> v2:172.21.15.188:3300/0 conn(0x7ff90413fae0 0x7ff9040cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff909c00640 1 -- 172.21.15.141:0/1676617509 >> v1:172.21.15.141:6789/0 conn(0x7ff9040cdc20 legacy=0x7ff90415d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff909c00640 1 -- 172.21.15.141:0/1676617509 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9040cea00 con 0x7ff90413cf10 2024-01-16T15:00:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.782+0000 7ff903e00640 1 -- 172.21.15.141:0/1676617509 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535137027 0 0) 0x7ff9040ce440 con 0x7ff9040cdc20 2024-01-16T15:00:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.783+0000 7ff909c00640 1 --2- 172.21.15.141:0/1676617509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff90413f5a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff90406ffb0 tx=0x7ff8f402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355a83babd937f15 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.783+0000 7ff903e00640 1 -- 172.21.15.141:0/1676617509 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f402cc80 con 0x7ff90413cf10 2024-01-16T15:00:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.783+0000 7ff903e00640 1 -- 172.21.15.141:0/1676617509 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f402cde0 con 0x7ff90413cf10 2024-01-16T15:00:10.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.783+0000 7ff903e00640 1 -- 172.21.15.141:0/1676617509 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4034660 con 0x7ff90413cf10 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 -- 172.21.15.141:0/1676617509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 msgr2=0x7ff90413f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 --2- 172.21.15.141:0/1676617509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff90413f5a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff90406ffb0 tx=0x7ff8f402bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 -- 172.21.15.141:0/1676617509 shutdown_connections 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 --2- 172.21.15.141:0/1676617509 >> v2:172.21.15.188:3300/0 conn(0x7ff90413fae0 0x7ff9040cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 --2- 172.21.15.141:0/1676617509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff90413f5a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 -- 172.21.15.141:0/1676617509 >> 172.21.15.141:0/1676617509 conn(0x7ff90406a7d0 msgr2=0x7ff90406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 -- 172.21.15.141:0/1676617509 shutdown_connections 2024-01-16T15:00:10.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.784+0000 7ff90b000640 1 -- 172.21.15.141:0/1676617509 wait complete. 2024-01-16T15:00:10.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.786+0000 7ff90b000640 1 Processor -- start 2024-01-16T15:00:10.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.786+0000 7ff90b000640 1 -- start start 2024-01-16T15:00:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff90b000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9040cdc20 0x7ff90413bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff90b000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff9041e69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff90b000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff90413fae0 0x7ff9041e6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff90b000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90413e6f0 con 0x7ff90413cf10 2024-01-16T15:00:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff90b000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90413e9d0 con 0x7ff9040cdc20 2024-01-16T15:00:10.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff90b000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90413ecb0 con 0x7ff90413fae0 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff909200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff9041e69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff909200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff9041e69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39286/0 (socket says 172.21.15.141:39286) 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.787+0000 7ff909200640 1 -- 172.21.15.141:0/1089428147 learned_addr learned my addr 172.21.15.141:0/1089428147 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff909c00640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9040cdc20 0x7ff90413bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff90a600640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff90413fae0 0x7ff9041e6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff909200640 1 -- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff90413fae0 msgr2=0x7ff9041e6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff909200640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff90413fae0 0x7ff9041e6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff909200640 1 -- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9040cdc20 msgr2=0x7ff90413bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff909200640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9040cdc20 0x7ff90413bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff909200640 1 -- 172.21.15.141:0/1089428147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4009a70 con 0x7ff90413cf10 2024-01-16T15:00:10.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff909200640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff9041e69f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00c7f0 tx=0x7ff8ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff901600640 1 -- 172.21.15.141:0/1089428147 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec004220 con 0x7ff90413cf10 2024-01-16T15:00:10.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff901600640 1 -- 172.21.15.141:0/1089428147 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec004380 con 0x7ff90413cf10 2024-01-16T15:00:10.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff901600640 1 -- 172.21.15.141:0/1089428147 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec007840 con 0x7ff90413cf10 2024-01-16T15:00:10.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.788+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90413ef90 con 0x7ff90413cf10 2024-01-16T15:00:10.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.789+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90413d970 con 0x7ff90413cf10 2024-01-16T15:00:10.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.790+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9041f2e60 con 0x7ff90413cf10 2024-01-16T15:00:10.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.790+0000 7ff901600640 1 -- 172.21.15.141:0/1089428147 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff8ec007bb0 con 0x7ff90413cf10 2024-01-16T15:00:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.792+0000 7ff901600640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.792+0000 7ff901600640 1 -- 172.21.15.141:0/1089428147 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8ec089fd0 con 0x7ff90413cf10 2024-01-16T15:00:10.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.792+0000 7ff909c00640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.793+0000 7ff909c00640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4036000 tx=0x7ff8f4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:10.796+0000 7ff901600640 1 -- 172.21.15.141:0/1089428147 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8ec054a60 con 0x7ff90413cf10 2024-01-16T15:00:11.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.022+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff904067450 con 0x7ff90413cf10 2024-01-16T15:00:11.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.557+0000 7ff901600640 1 -- 172.21.15.141:0/1089428147 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff8ec01e0c0 con 0x7ff90413cf10 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8dc06b500 msgr2=0x7ff8dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4036000 tx=0x7ff8f4030000 comp rx=0 tx=0).stop 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 msgr2=0x7ff9041e69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff9041e69f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00c7f0 tx=0x7ff8ec00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 shutdown_connections 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff90413fae0 0x7ff9041e6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff90413cf10 0x7ff9041e69f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 --2- 172.21.15.141:0/1089428147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9040cdc20 0x7ff90413bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 >> 172.21.15.141:0/1089428147 conn(0x7ff90406a7d0 msgr2=0x7ff90406cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 shutdown_connections 2024-01-16T15:00:11.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.561+0000 7ff90b000640 1 -- 172.21.15.141:0/1089428147 wait complete. 2024-01-16T15:00:11.562 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:00:11.562 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:11.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.789+0000 7f17c7400640 1 Processor -- start 2024-01-16T15:00:11.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.789+0000 7f17c7400640 1 -- start start 2024-01-16T15:00:11.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.790+0000 7f17c7400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f17c013e360 0x7f17c00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17c7400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c015c5f0 con 0x7f17c015f9c0 2024-01-16T15:00:11.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17c7400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c006c470 con 0x7f17c013df80 2024-01-16T15:00:11.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17c7400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c006c750 con 0x7f17c013e360 2024-01-16T15:00:11.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17c5400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f17c015f9c0 0x7f17c00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55234/0 (socket says 172.21.15.141:55234) 2024-01-16T15:00:11.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17c5400640 1 -- 172.21.15.141:0/3580502919 learned_addr learned my addr 172.21.15.141:0/3580502919 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:11.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17b7e00640 1 --2- 172.21.15.141:0/3580502919 >> v2:172.21.15.188:3300/0 conn(0x7f17c013e360 0x7f17c00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951332698 0 0) 0x7f17c015c5f0 con 0x7f17c015f9c0 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.791+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a0003520 con 0x7f17c015f9c0 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248011111 0 0) 0x7f17a0003520 con 0x7f17c015f9c0 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 >> v2:172.21.15.188:3300/0 conn(0x7f17c013e360 msgr2=0x7f17c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 --2- 172.21.15.141:0/3580502919 >> v2:172.21.15.188:3300/0 conn(0x7f17c013e360 0x7f17c00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 >> v1:172.21.15.188:6789/0 conn(0x7f17c013df80 legacy=0x7f17c015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c006ca30 con 0x7f17c015f9c0 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f17b8001a10 con 0x7f17c015f9c0 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 >> v1:172.21.15.141:6789/0 conn(0x7f17c015f9c0 legacy=0x7f17c00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17a00047c0 0x7f17a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17a0005180 0x7f17a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c006c750 con 0x7f17a00039c0 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a000c050 con 0x7f17a0005180 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a000c210 con 0x7f17a00047c0 2024-01-16T15:00:11.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.792+0000 7f17c4a00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17b7e00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17a00047c0 0x7f17a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17c5400640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17a0005180 0x7f17a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17c4a00640 1 -- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17a00047c0 msgr2=0x7f17a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17c4a00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17a00047c0 0x7f17a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17c4a00640 1 -- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17a0005180 msgr2=0x7f17a000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17c4a00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17a0005180 0x7f17a000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17c5400640 1 -- 172.21.15.141:0/3580502919 reap_dead start 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17c4a00640 1 -- 172.21.15.141:0/3580502919 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a0003520 con 0x7f17a00039c0 2024-01-16T15:00:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.793+0000 7f17b7e00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17a00047c0 0x7f17a0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:11.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.794+0000 7f17c4a00640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17a0003dc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f17b0003db0 tx=0x7f17a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea48b8b8efc6f56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.794+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0032600 con 0x7f17a00039c0 2024-01-16T15:00:11.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.794+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b002e050 con 0x7f17a00039c0 2024-01-16T15:00:11.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.794+0000 7f17bfe00640 1 -- 172.21.15.141:0/3580502919 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b00329e0 con 0x7f17a00039c0 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.795+0000 7f17c7400640 1 -- 172.21.15.141:0/3580502919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 msgr2=0x7f17a0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.795+0000 7f17c7400640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17a0003dc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f17b0003db0 tx=0x7f17a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.795+0000 7f17c7400640 1 -- 172.21.15.141:0/3580502919 shutdown_connections 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.795+0000 7f17c7400640 1 --2- 172.21.15.141:0/3580502919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17a0003dc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.795+0000 7f17c7400640 1 -- 172.21.15.141:0/3580502919 >> 172.21.15.141:0/3580502919 conn(0x7f17c006a800 msgr2=0x7f17c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.795+0000 7f17c7400640 1 -- 172.21.15.141:0/3580502919 shutdown_connections 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.795+0000 7f17c7400640 1 -- 172.21.15.141:0/3580502919 wait complete. 2024-01-16T15:00:11.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.797+0000 7f17c7400640 1 Processor -- start 2024-01-16T15:00:11.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.797+0000 7f17c7400640 1 -- start start 2024-01-16T15:00:11.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17c01f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c0071820 0x7f17c014b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17c0159f80 0x7f17c015a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c7400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c015a940 con 0x7f17a00039c0 2024-01-16T15:00:11.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c7400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c014bc10 con 0x7f17c0071820 2024-01-16T15:00:11.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c7400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c014bef0 con 0x7f17c0159f80 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17c01f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17c01f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39300/0 (socket says 172.21.15.141:39300) 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.798+0000 7f17c4a00640 1 -- 172.21.15.141:0/3621536576 learned_addr learned my addr 172.21.15.141:0/3621536576 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c5400640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17c0159f80 0x7f17c015a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17b7e00640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c0071820 0x7f17c014b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c4a00640 1 -- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17c0159f80 msgr2=0x7f17c015a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c4a00640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17c0159f80 0x7f17c015a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c4a00640 1 -- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c0071820 msgr2=0x7f17c014b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c4a00640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c0071820 0x7f17c014b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c4a00640 1 -- 172.21.15.141:0/3621536576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c014c1d0 con 0x7f17a00039c0 2024-01-16T15:00:11.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c4a00640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17c01f60b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f17b0000c00 tx=0x7f17b00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17bd600640 1 -- 172.21.15.141:0/3621536576 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b003c410 con 0x7f17a00039c0 2024-01-16T15:00:11.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17bd600640 1 -- 172.21.15.141:0/3621536576 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b003c570 con 0x7f17a00039c0 2024-01-16T15:00:11.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17bd600640 1 -- 172.21.15.141:0/3621536576 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b003c8e0 con 0x7f17a00039c0 2024-01-16T15:00:11.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c014c4b0 con 0x7f17a00039c0 2024-01-16T15:00:11.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.799+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c01f65f0 con 0x7f17a00039c0 2024-01-16T15:00:11.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.800+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c01f32a0 con 0x7f17a00039c0 2024-01-16T15:00:11.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.801+0000 7f17bd600640 1 -- 172.21.15.141:0/3621536576 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f17b0039050 con 0x7f17a00039c0 2024-01-16T15:00:11.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.802+0000 7f17bd600640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f179806b530 0x7f179806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.802+0000 7f17bd600640 1 -- 172.21.15.141:0/3621536576 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17b00ab560 con 0x7f17a00039c0 2024-01-16T15:00:11.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.803+0000 7f17b7e00640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f179806b530 0x7f179806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.804+0000 7f17b7e00640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f179806b530 0x7f179806d9f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f17c00b0770 tx=0x7f17a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:11.807+0000 7f17bd600640 1 -- 172.21.15.141:0/3621536576 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17c01f32a0 con 0x7f17a00039c0 2024-01-16T15:00:12.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.032+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17c01f32a0 con 0x7f17a00039c0 2024-01-16T15:00:12.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.033+0000 7f17bd600640 1 -- 172.21.15.141:0/3621536576 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f17c01f32a0 con 0x7f17a00039c0 2024-01-16T15:00:12.015 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:12.015 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2118095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2118095418},{"type":"v1","addr":"172.21.15.188:6837","nonce":2118095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/3680808697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3680808697},{"type":"v1","addr":"172.21.15.188:6836","nonce":3680808697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5793,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/275409001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":275409001},{"type":"v1","addr":"172.21.15.141:6833","nonce":275409001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:00:07.944739+0000","modified":"2024-01-16T15:00:11.025175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5796":{"gid":5796,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/2933218811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":2933218811},{"type":"v1","addr":"172.21.15.141:6835","nonce":2933218811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T15:00:12.015 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 27 2024-01-16T15:00:12.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f179806b530 msgr2=0x7f179806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f179806b530 0x7f179806d9f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f17c00b0770 tx=0x7f17a8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:12.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 msgr2=0x7f17c01f60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17c01f60b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f17b0000c00 tx=0x7f17b00036f0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 shutdown_connections 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f179806b530 0x7f179806d9f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17c0159f80 0x7f17c015a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c0071820 0x7f17c014b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 --2- 172.21.15.141:0/3621536576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17a00039c0 0x7f17c01f60b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 >> 172.21.15.141:0/3621536576 conn(0x7f17c006a800 msgr2=0x7f17c015c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 shutdown_connections 2024-01-16T15:00:12.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.036+0000 7f17c7400640 1 -- 172.21.15.141:0/3621536576 wait complete. 2024-01-16T15:00:12.035 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:12.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.256+0000 7fcf34400640 1 Processor -- start 2024-01-16T15:00:12.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.256+0000 7fcf34400640 1 -- start start 2024-01-16T15:00:12.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.257+0000 7fcf34400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcf2c0ae5a0 0x7fcf2c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.257+0000 7fcf34400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0b3ae0 0x7fcf2c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf34400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcf2c06d120 0x7fcf2c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf34400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c071450 con 0x7fcf2c0ae5a0 2024-01-16T15:00:12.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf34400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c070c40 con 0x7fcf2c06d120 2024-01-16T15:00:12.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf34400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c06c470 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcf2c0ae5a0 0x7fcf2c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcf2c0ae5a0 0x7fcf2c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39304/0 (socket says 172.21.15.141:39304) 2024-01-16T15:00:12.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31a00640 1 -- 172.21.15.141:0/2713460822 learned_addr learned my addr 172.21.15.141:0/2713460822 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31000640 1 --2- 172.21.15.141:0/2713460822 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0b3ae0 0x7fcf2c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31a00640 1 -- 172.21.15.141:0/2713460822 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0b3ae0 msgr2=0x7fcf2c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf32400640 1 --2- 172.21.15.141:0/2713460822 >> v2:172.21.15.188:3300/0 conn(0x7fcf2c06d120 0x7fcf2c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31a00640 1 --2- 172.21.15.141:0/2713460822 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0b3ae0 0x7fcf2c06cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31a00640 1 -- 172.21.15.141:0/2713460822 >> v2:172.21.15.188:3300/0 conn(0x7fcf2c06d120 msgr2=0x7fcf2c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.258+0000 7fcf31a00640 1 --2- 172.21.15.141:0/2713460822 >> v2:172.21.15.188:3300/0 conn(0x7fcf2c06d120 0x7fcf2c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.259+0000 7fcf31a00640 1 -- 172.21.15.141:0/2713460822 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c06c750 con 0x7fcf2c0ae5a0 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.259+0000 7fcf31000640 1 --2- 172.21.15.141:0/2713460822 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0b3ae0 0x7fcf2c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.259+0000 7fcf32400640 1 --2- 172.21.15.141:0/2713460822 >> v2:172.21.15.188:3300/0 conn(0x7fcf2c06d120 0x7fcf2c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.260+0000 7fcf31a00640 1 --2- 172.21.15.141:0/2713460822 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0ae5a0 0x7fcf2c15e430 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c06e230 tx=0x7fcf1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe86c34a88ec8b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.260+0000 7fcf2be00640 1 -- 172.21.15.141:0/2713460822 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c031870 con 0x7fcf2c0ae5a0 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.260+0000 7fcf2be00640 1 -- 172.21.15.141:0/2713460822 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c0319d0 con 0x7fcf2c0ae5a0 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.260+0000 7fcf2be00640 1 -- 172.21.15.141:0/2713460822 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c031d60 con 0x7fcf2c0ae5a0 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 -- 172.21.15.141:0/2713460822 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0ae5a0 msgr2=0x7fcf2c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 --2- 172.21.15.141:0/2713460822 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0ae5a0 0x7fcf2c15e430 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c06e230 tx=0x7fcf1c02b910 comp rx=0 tx=0).stop 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 -- 172.21.15.141:0/2713460822 shutdown_connections 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 --2- 172.21.15.141:0/2713460822 >> v2:172.21.15.188:3300/0 conn(0x7fcf2c06d120 0x7fcf2c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 --2- 172.21.15.141:0/2713460822 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0b3ae0 0x7fcf2c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 --2- 172.21.15.141:0/2713460822 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0ae5a0 0x7fcf2c15e430 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 -- 172.21.15.141:0/2713460822 >> 172.21.15.141:0/2713460822 conn(0x7fcf2c06a800 msgr2=0x7fcf2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 -- 172.21.15.141:0/2713460822 shutdown_connections 2024-01-16T15:00:12.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.261+0000 7fcf34400640 1 -- 172.21.15.141:0/2713460822 wait complete. 2024-01-16T15:00:12.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.263+0000 7fcf34400640 1 Processor -- start 2024-01-16T15:00:12.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.263+0000 7fcf34400640 1 -- start start 2024-01-16T15:00:12.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.263+0000 7fcf34400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c06d120 0x7fcf2c14ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf34400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0ae5a0 0x7fcf2c0b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf34400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0b3ae0 0x7fcf2c15d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf34400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0cdfb0 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf34400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c14b170 con 0x7fcf2c06d120 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf34400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0ce780 con 0x7fcf2c0ae5a0 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0b3ae0 0x7fcf2c15d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0b3ae0 0x7fcf2c15d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39320/0 (socket says 172.21.15.141:39320) 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 -- 172.21.15.141:0/2967119005 learned_addr learned my addr 172.21.15.141:0/2967119005 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf31000640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0ae5a0 0x7fcf2c0b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf31a00640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c06d120 0x7fcf2c14ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 -- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0ae5a0 msgr2=0x7fcf2c0b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0ae5a0 0x7fcf2c0b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 -- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c06d120 msgr2=0x7fcf2c14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c06d120 0x7fcf2c14ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf32400640 1 -- 172.21.15.141:0/2967119005 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0094d0 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.264+0000 7fcf31a00640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c06d120 0x7fcf2c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.265+0000 7fcf32400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0b3ae0 0x7fcf2c15d820 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400cef0 tx=0x7fcf2400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.265+0000 7fcf29600640 1 -- 172.21.15.141:0/2967119005 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2400d840 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.265+0000 7fcf29600640 1 -- 172.21.15.141:0/2967119005 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2400d9a0 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.265+0000 7fcf29600640 1 -- 172.21.15.141:0/2967119005 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2400dd50 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.265+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c0ce1c0 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.265+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c156230 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.266+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c0675d0 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.267+0000 7fcf29600640 1 -- 172.21.15.141:0/2967119005 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fcf2401e3b0 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.268+0000 7fcf29600640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf0406b500 0x7fcf0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.268+0000 7fcf29600640 1 -- 172.21.15.141:0/2967119005 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf2408aae0 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.269+0000 7fcf31a00640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf0406b500 0x7fcf0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.270+0000 7fcf31a00640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf0406b500 0x7fcf0406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c036000 tx=0x7fcf1c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.273+0000 7fcf29600640 1 -- 172.21.15.141:0/2967119005 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf24055550 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.469+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf2c0cbb40 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.470+0000 7fcf29600640 1 -- 172.21.15.141:0/2967119005 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16561 (secure 0 0 0) 0x7fcf2401a020 con 0x7fcf2c0b3ae0 2024-01-16T15:00:12.452 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:12.452 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":46,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:00:09.532582+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:00:05.761243+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:00:06.970207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf0406b500 msgr2=0x7fcf0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf0406b500 0x7fcf0406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c036000 tx=0x7fcf1c03c040 comp rx=0 tx=0).stop 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0b3ae0 msgr2=0x7fcf2c15d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0b3ae0 0x7fcf2c15d820 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400cef0 tx=0x7fcf2400a710 comp rx=0 tx=0).stop 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 shutdown_connections 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcf0406b500 0x7fcf0406d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf2c0b3ae0 0x7fcf2c15d820 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcf2c0ae5a0 0x7fcf2c0b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 --2- 172.21.15.141:0/2967119005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c06d120 0x7fcf2c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 >> 172.21.15.141:0/2967119005 conn(0x7fcf2c06a800 msgr2=0x7fcf2c15a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 shutdown_connections 2024-01-16T15:00:12.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.473+0000 7fcf34400640 1 -- 172.21.15.141:0/2967119005 wait complete. 2024-01-16T15:00:12.475 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:00:12.475 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:12.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.703+0000 7fd8c9800640 1 Processor -- start 2024-01-16T15:00:12.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.703+0000 7fd8c9800640 1 -- start start 2024-01-16T15:00:12.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.704+0000 7fd8c9800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd8c40ae5a0 0x7fd8c415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.704+0000 7fd8c9800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd8c40b3ae0 0x7fd8c406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.704+0000 7fd8c9800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c4070c40 con 0x7fd8c406d120 2024-01-16T15:00:12.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.704+0000 7fd8c9800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c4071450 con 0x7fd8c40b3ae0 2024-01-16T15:00:12.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.704+0000 7fd8c9800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c406c470 con 0x7fd8c40ae5a0 2024-01-16T15:00:12.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8b9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd8c40b3ae0 0x7fd8c406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8c3400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd8c406d120 0x7fd8c40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55248/0 (socket says 172.21.15.141:55248) 2024-01-16T15:00:12.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8c3400640 1 -- 172.21.15.141:0/2797864499 learned_addr learned my addr 172.21.15.141:0/2797864499 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:12.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8c2a00640 1 --2- 172.21.15.141:0/2797864499 >> v2:172.21.15.188:3300/0 conn(0x7fd8c40ae5a0 0x7fd8c415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8c2000640 1 -- 172.21.15.141:0/2797864499 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349130915 0 0) 0x7fd8c4070c40 con 0x7fd8c406d120 2024-01-16T15:00:12.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8c2000640 1 -- 172.21.15.141:0/2797864499 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4003520 con 0x7fd8c406d120 2024-01-16T15:00:12.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8b9e00640 1 -- 172.21.15.141:0/2797864499 >> v2:172.21.15.188:3300/0 conn(0x7fd8c40ae5a0 msgr2=0x7fd8c415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8b9e00640 1 --2- 172.21.15.141:0/2797864499 >> v2:172.21.15.188:3300/0 conn(0x7fd8c40ae5a0 0x7fd8c415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8b9e00640 1 -- 172.21.15.141:0/2797864499 >> v1:172.21.15.141:6789/0 conn(0x7fd8c406d120 legacy=0x7fd8c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.705+0000 7fd8b9e00640 1 -- 172.21.15.141:0/2797864499 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c406c750 con 0x7fd8c40b3ae0 2024-01-16T15:00:12.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8b9e00640 1 --2- 172.21.15.141:0/2797864499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c40b3ae0 0x7fd8c406cbe0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40cfe20 tx=0x7fd8ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e954f723f001ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c2000640 1 -- 172.21.15.141:0/2797864499 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac031830 con 0x7fd8c40b3ae0 2024-01-16T15:00:12.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c2000640 1 -- 172.21.15.141:0/2797864499 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac031990 con 0x7fd8c40b3ae0 2024-01-16T15:00:12.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c2000640 1 -- 172.21.15.141:0/2797864499 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac031d20 con 0x7fd8c40b3ae0 2024-01-16T15:00:12.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c9800640 1 -- 172.21.15.141:0/2797864499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c40b3ae0 msgr2=0x7fd8c406cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c9800640 1 --2- 172.21.15.141:0/2797864499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c40b3ae0 0x7fd8c406cbe0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40cfe20 tx=0x7fd8ac02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c9800640 1 -- 172.21.15.141:0/2797864499 shutdown_connections 2024-01-16T15:00:12.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c9800640 1 --2- 172.21.15.141:0/2797864499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c40b3ae0 0x7fd8c406cbe0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c9800640 1 --2- 172.21.15.141:0/2797864499 >> v2:172.21.15.188:3300/0 conn(0x7fd8c40ae5a0 0x7fd8c415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.707+0000 7fd8c9800640 1 -- 172.21.15.141:0/2797864499 >> 172.21.15.141:0/2797864499 conn(0x7fd8c406a800 msgr2=0x7fd8c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.708+0000 7fd8c9800640 1 -- 172.21.15.141:0/2797864499 shutdown_connections 2024-01-16T15:00:12.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.708+0000 7fd8c9800640 1 -- 172.21.15.141:0/2797864499 wait complete. 2024-01-16T15:00:12.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.709+0000 7fd8c9800640 1 Processor -- start 2024-01-16T15:00:12.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.710+0000 7fd8c9800640 1 -- start start 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c9800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c406d120 0x7fd8c4153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8c40ae5a0 0x7fd8c4156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd8c40b3ae0 0x7fd8c40afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c9800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40cb940 con 0x7fd8c406d120 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c9800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40cbc20 con 0x7fd8c40ae5a0 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40cbf00 con 0x7fd8c40b3ae0 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c406d120 0x7fd8c4153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c406d120 0x7fd8c4153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39340/0 (socket says 172.21.15.141:39340) 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 -- 172.21.15.141:0/121208777 learned_addr learned my addr 172.21.15.141:0/121208777 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 -- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd8c40b3ae0 msgr2=0x7fd8c40afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd8c40b3ae0 0x7fd8c40afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 -- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8c40ae5a0 msgr2=0x7fd8c4156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8c40ae5a0 0x7fd8c4156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.711+0000 7fd8c2a00640 1 -- 172.21.15.141:0/121208777 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac009400 con 0x7fd8c406d120 2024-01-16T15:00:12.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.712+0000 7fd8c2a00640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c406d120 0x7fd8c4153120 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400d800 tx=0x7fd8b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.712+0000 7fd8bb400640 1 -- 172.21.15.141:0/121208777 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b400b6c0 con 0x7fd8c406d120 2024-01-16T15:00:12.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.712+0000 7fd8bb400640 1 -- 172.21.15.141:0/121208777 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b400b820 con 0x7fd8c406d120 2024-01-16T15:00:12.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.712+0000 7fd8bb400640 1 -- 172.21.15.141:0/121208777 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b400bbb0 con 0x7fd8c406d120 2024-01-16T15:00:12.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.712+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c40cc1e0 con 0x7fd8c406d120 2024-01-16T15:00:12.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.712+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c4153660 con 0x7fd8c406d120 2024-01-16T15:00:12.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.713+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c40675d0 con 0x7fd8c406d120 2024-01-16T15:00:12.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.714+0000 7fd8bb400640 1 -- 172.21.15.141:0/121208777 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd8b400bd10 con 0x7fd8c406d120 2024-01-16T15:00:12.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.715+0000 7fd8bb400640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd89c06b500 0x7fd89c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.715+0000 7fd8bb400640 1 -- 172.21.15.141:0/121208777 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8b408a5a0 con 0x7fd8c406d120 2024-01-16T15:00:12.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.716+0000 7fd8b9e00640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd89c06b500 0x7fd89c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.717+0000 7fd8b9e00640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd89c06b500 0x7fd89c06d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac036000 tx=0x7fd8ac030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.720+0000 7fd8bb400640 1 -- 172.21.15.141:0/121208777 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8b40550a0 con 0x7fd8c406d120 2024-01-16T15:00:12.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.908+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8c4070550 con 0x7fd89c06b500 2024-01-16T15:00:12.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.937+0000 7fd8bb400640 1 -- 172.21.15.141:0/121208777 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286650 (secure 0 0 0) 0x7fd8ac03a420 con 0x7fd89c06b500 2024-01-16T15:00:12.922 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:12.924 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:00:12.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.943+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd89c06b500 msgr2=0x7fd89c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.943+0000 7fd8c9800640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd89c06b500 0x7fd89c06d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac036000 tx=0x7fd8ac030000 comp rx=0 tx=0).stop 2024-01-16T15:00:12.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.943+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c406d120 msgr2=0x7fd8c4153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.943+0000 7fd8c9800640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c406d120 0x7fd8c4153120 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400d800 tx=0x7fd8b400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 shutdown_connections 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd89c06b500 0x7fd89c06d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd8c40b3ae0 0x7fd8c40afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8c40ae5a0 0x7fd8c4156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 --2- 172.21.15.141:0/121208777 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8c406d120 0x7fd8c4153120 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 >> 172.21.15.141:0/121208777 conn(0x7fd8c406a800 msgr2=0x7fd8c40b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 shutdown_connections 2024-01-16T15:00:12.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:12.944+0000 7fd8c9800640 1 -- 172.21.15.141:0/121208777 wait complete. 2024-01-16T15:00:12.945 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-01-16T15:00:12.502899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254784,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719936,"statfs":{"total":773094113280,"available":772833214464,"internally_reserved":0,"allocated":5447680,"data_stored":2876228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002279"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.543229+0000","last_change":"2024-01-16T15:00:08.536867+0000","last_active":"2024-01-16T15:00:08.543229+0000","last_peered":"2024-01-16T15:00:08.543229+0000","last_clean":"2024-01-16T15:00:08.543229+0000","last_became_active":"2024-01-16T15:00:08.536584+0000","last_became_peered":"2024-01-16T15:00:08.536584+0000","last_unstale":"2024-01-16T15:00:08.543229+0000","last_undegraded":"2024-01-16T15:00:08.543229+0000","last_fullsized":"2024-01-16T15:00:08.543229+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:40.608354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563267+0000","last_change":"2024-01-16T15:00:08.555740+0000","last_active":"2024-01-16T15:00:08.563267+0000","last_peered":"2024-01-16T15:00:08.563267+0000","last_clean":"2024-01-16T15:00:08.563267+0000","last_became_active":"2024-01-16T15:00:08.555299+0000","last_became_peered":"2024-01-16T15:00:08.555299+0000","last_unstale":"2024-01-16T15:00:08.563267+0000","last_undegraded":"2024-01-16T15:00:08.563267+0000","last_fullsized":"2024-01-16T15:00:08.563267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:32.978431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547013+0000","last_change":"2024-01-16T15:00:08.544321+0000","last_active":"2024-01-16T15:00:08.547013+0000","last_peered":"2024-01-16T15:00:08.547013+0000","last_clean":"2024-01-16T15:00:08.547013+0000","last_became_active":"2024-01-16T15:00:08.544004+0000","last_became_peered":"2024-01-16T15:00:08.544004+0000","last_unstale":"2024-01-16T15:00:08.547013+0000","last_undegraded":"2024-01-16T15:00:08.547013+0000","last_fullsized":"2024-01-16T15:00:08.547013+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:12.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.557359+0000","last_change":"2024-01-16T15:00:08.541565+0000","last_active":"2024-01-16T15:00:08.557359+0000","last_peered":"2024-01-16T15:00:08.557359+0000","last_clean":"2024-01-16T15:00:08.557359+0000","last_became_active":"2024-01-16T15:00:08.541299+0000","last_became_peered":"2024-01-16T15:00:08.541299+0000","last_unstale":"2024-01-16T15:00:08.557359+0000","last_undegraded":"2024-01-16T15:00:08.557359+0000","last_fullsized":"2024-01-16T15:00:08.557359+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:00.980436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563180+0000","last_change":"2024-01-16T15:00:08.558164+0000","last_active":"2024-01-16T15:00:08.563180+0000","last_peered":"2024-01-16T15:00:08.563180+0000","last_clean":"2024-01-16T15:00:08.563180+0000","last_became_active":"2024-01-16T15:00:08.557839+0000","last_became_peered":"2024-01-16T15:00:08.557839+0000","last_unstale":"2024-01-16T15:00:08.563180+0000","last_undegraded":"2024-01-16T15:00:08.563180+0000","last_fullsized":"2024-01-16T15:00:08.563180+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:47.964823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.551947+0000","last_change":"2024-01-16T15:00:08.547938+0000","last_active":"2024-01-16T15:00:08.551947+0000","last_peered":"2024-01-16T15:00:08.551947+0000","last_clean":"2024-01-16T15:00:08.551947+0000","last_became_active":"2024-01-16T15:00:08.547625+0000","last_became_peered":"2024-01-16T15:00:08.547625+0000","last_unstale":"2024-01-16T15:00:08.551947+0000","last_undegraded":"2024-01-16T15:00:08.551947+0000","last_fullsized":"2024-01-16T15:00:08.551947+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:14.919862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.560956+0000","last_change":"2024-01-16T15:00:08.555571+0000","last_active":"2024-01-16T15:00:08.560956+0000","last_peered":"2024-01-16T15:00:08.560956+0000","last_clean":"2024-01-16T15:00:08.560956+0000","last_became_active":"2024-01-16T15:00:08.554593+0000","last_became_peered":"2024-01-16T15:00:08.554593+0000","last_unstale":"2024-01-16T15:00:08.560956+0000","last_undegraded":"2024-01-16T15:00:08.560956+0000","last_fullsized":"2024-01-16T15:00:08.560956+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:47.964823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568817+0000","last_change":"2024-01-16T15:00:08.559549+0000","last_active":"2024-01-16T15:00:08.568817+0000","last_peered":"2024-01-16T15:00:08.568817+0000","last_clean":"2024-01-16T15:00:08.568817+0000","last_became_active":"2024-01-16T15:00:08.557229+0000","last_became_peered":"2024-01-16T15:00:08.557229+0000","last_unstale":"2024-01-16T15:00:08.568817+0000","last_undegraded":"2024-01-16T15:00:08.568817+0000","last_fullsized":"2024-01-16T15:00:08.568817+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:11.355514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571087+0000","last_change":"2024-01-16T15:00:08.567557+0000","last_active":"2024-01-16T15:00:08.571087+0000","last_peered":"2024-01-16T15:00:08.571087+0000","last_clean":"2024-01-16T15:00:08.571087+0000","last_became_active":"2024-01-16T15:00:08.567392+0000","last_became_peered":"2024-01-16T15:00:08.567392+0000","last_unstale":"2024-01-16T15:00:08.571087+0000","last_undegraded":"2024-01-16T15:00:08.571087+0000","last_fullsized":"2024-01-16T15:00:08.571087+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:30.390941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568935+0000","last_change":"2024-01-16T15:00:08.556952+0000","last_active":"2024-01-16T15:00:08.568935+0000","last_peered":"2024-01-16T15:00:08.568935+0000","last_clean":"2024-01-16T15:00:08.568935+0000","last_became_active":"2024-01-16T15:00:08.556684+0000","last_became_peered":"2024-01-16T15:00:08.556684+0000","last_unstale":"2024-01-16T15:00:08.568935+0000","last_undegraded":"2024-01-16T15:00:08.568935+0000","last_fullsized":"2024-01-16T15:00:08.568935+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:26.068944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571091+0000","last_change":"2024-01-16T15:00:08.569546+0000","last_active":"2024-01-16T15:00:08.571091+0000","last_peered":"2024-01-16T15:00:08.571091+0000","last_clean":"2024-01-16T15:00:08.571091+0000","last_became_active":"2024-01-16T15:00:08.569124+0000","last_became_peered":"2024-01-16T15:00:08.569124+0000","last_unstale":"2024-01-16T15:00:08.571091+0000","last_undegraded":"2024-01-16T15:00:08.571091+0000","last_fullsized":"2024-01-16T15:00:08.571091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:15.756996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.546825+0000","last_change":"2024-01-16T15:00:08.544704+0000","last_active":"2024-01-16T15:00:08.546825+0000","last_peered":"2024-01-16T15:00:08.546825+0000","last_clean":"2024-01-16T15:00:08.546825+0000","last_became_active":"2024-01-16T15:00:08.543395+0000","last_became_peered":"2024-01-16T15:00:08.543395+0000","last_unstale":"2024-01-16T15:00:08.546825+0000","last_undegraded":"2024-01-16T15:00:08.546825+0000","last_fullsized":"2024-01-16T15:00:08.546825+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:56.333632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571053+0000","last_change":"2024-01-16T15:00:08.564033+0000","last_active":"2024-01-16T15:00:08.571053+0000","last_peered":"2024-01-16T15:00:08.571053+0000","last_clean":"2024-01-16T15:00:08.571053+0000","last_became_active":"2024-01-16T15:00:08.563588+0000","last_became_peered":"2024-01-16T15:00:08.563588+0000","last_unstale":"2024-01-16T15:00:08.571053+0000","last_undegraded":"2024-01-16T15:00:08.571053+0000","last_fullsized":"2024-01-16T15:00:08.571053+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:51.709344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.570692+0000","last_change":"2024-01-16T15:00:08.567255+0000","last_active":"2024-01-16T15:00:08.570692+0000","last_peered":"2024-01-16T15:00:08.570692+0000","last_clean":"2024-01-16T15:00:08.570692+0000","last_became_active":"2024-01-16T15:00:08.566997+0000","last_became_peered":"2024-01-16T15:00:08.566997+0000","last_unstale":"2024-01-16T15:00:08.570692+0000","last_undegraded":"2024-01-16T15:00:08.570692+0000","last_fullsized":"2024-01-16T15:00:08.570692+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:21.748083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.553205+0000","last_change":"2024-01-16T15:00:08.549815+0000","last_active":"2024-01-16T15:00:08.553205+0000","last_peered":"2024-01-16T15:00:08.553205+0000","last_clean":"2024-01-16T15:00:08.553205+0000","last_became_active":"2024-01-16T15:00:08.548946+0000","last_became_peered":"2024-01-16T15:00:08.548946+0000","last_unstale":"2024-01-16T15:00:08.553205+0000","last_undegraded":"2024-01-16T15:00:08.553205+0000","last_fullsized":"2024-01-16T15:00:08.553205+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:50.725282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563564+0000","last_change":"2024-01-16T15:00:08.559961+0000","last_active":"2024-01-16T15:00:08.563564+0000","last_peered":"2024-01-16T15:00:08.563564+0000","last_clean":"2024-01-16T15:00:08.563564+0000","last_became_active":"2024-01-16T15:00:08.559686+0000","last_became_peered":"2024-01-16T15:00:08.559686+0000","last_unstale":"2024-01-16T15:00:08.563564+0000","last_undegraded":"2024-01-16T15:00:08.563564+0000","last_fullsized":"2024-01-16T15:00:08.563564+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:55:05.241468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.545403+0000","last_change":"2024-01-16T15:00:08.536701+0000","last_active":"2024-01-16T15:00:08.545403+0000","last_peered":"2024-01-16T15:00:08.545403+0000","last_clean":"2024-01-16T15:00:08.545403+0000","last_became_active":"2024-01-16T15:00:08.536370+0000","last_became_peered":"2024-01-16T15:00:08.536370+0000","last_unstale":"2024-01-16T15:00:08.545403+0000","last_undegraded":"2024-01-16T15:00:08.545403+0000","last_fullsized":"2024-01-16T15:00:08.545403+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:20.095673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563431+0000","last_change":"2024-01-16T15:00:08.558699+0000","last_active":"2024-01-16T15:00:08.563431+0000","last_peered":"2024-01-16T15:00:08.563431+0000","last_clean":"2024-01-16T15:00:08.563431+0000","last_became_active":"2024-01-16T15:00:08.558421+0000","last_became_peered":"2024-01-16T15:00:08.558421+0000","last_unstale":"2024-01-16T15:00:08.563431+0000","last_undegraded":"2024-01-16T15:00:08.563431+0000","last_fullsized":"2024-01-16T15:00:08.563431+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:25.495238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563498+0000","last_change":"2024-01-16T15:00:08.559361+0000","last_active":"2024-01-16T15:00:08.563498+0000","last_peered":"2024-01-16T15:00:08.563498+0000","last_clean":"2024-01-16T15:00:08.563498+0000","last_became_active":"2024-01-16T15:00:08.559057+0000","last_became_peered":"2024-01-16T15:00:08.559057+0000","last_unstale":"2024-01-16T15:00:08.563498+0000","last_undegraded":"2024-01-16T15:00:08.563498+0000","last_fullsized":"2024-01-16T15:00:08.563498+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:57.357255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569657+0000","last_change":"2024-01-16T15:00:08.558429+0000","last_active":"2024-01-16T15:00:08.569657+0000","last_peered":"2024-01-16T15:00:08.569657+0000","last_clean":"2024-01-16T15:00:08.569657+0000","last_became_active":"2024-01-16T15:00:08.558160+0000","last_became_peered":"2024-01-16T15:00:08.558160+0000","last_unstale":"2024-01-16T15:00:08.569657+0000","last_undegraded":"2024-01-16T15:00:08.569657+0000","last_fullsized":"2024-01-16T15:00:08.569657+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:40.521526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554510+0000","last_change":"2024-01-16T15:00:08.548394+0000","last_active":"2024-01-16T15:00:08.554510+0000","last_peered":"2024-01-16T15:00:08.554510+0000","last_clean":"2024-01-16T15:00:08.554510+0000","last_became_active":"2024-01-16T15:00:08.548075+0000","last_became_peered":"2024-01-16T15:00:08.548075+0000","last_unstale":"2024-01-16T15:00:08.554510+0000","last_undegraded":"2024-01-16T15:00:08.554510+0000","last_fullsized":"2024-01-16T15:00:08.554510+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:22.075114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.557388+0000","last_change":"2024-01-16T15:00:08.545364+0000","last_active":"2024-01-16T15:00:08.557388+0000","last_peered":"2024-01-16T15:00:08.557388+0000","last_clean":"2024-01-16T15:00:08.557388+0000","last_became_active":"2024-01-16T15:00:08.545096+0000","last_became_peered":"2024-01-16T15:00:08.545096+0000","last_unstale":"2024-01-16T15:00:08.557388+0000","last_undegraded":"2024-01-16T15:00:08.557388+0000","last_fullsized":"2024-01-16T15:00:08.557388+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:35.701093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552607+0000","last_change":"2024-01-16T15:00:08.546916+0000","last_active":"2024-01-16T15:00:08.552607+0000","last_peered":"2024-01-16T15:00:08.552607+0000","last_clean":"2024-01-16T15:00:08.552607+0000","last_became_active":"2024-01-16T15:00:08.546663+0000","last_became_peered":"2024-01-16T15:00:08.546663+0000","last_unstale":"2024-01-16T15:00:08.552607+0000","last_undegraded":"2024-01-16T15:00:08.552607+0000","last_fullsized":"2024-01-16T15:00:08.552607+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:07:21.928549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547360+0000","last_change":"2024-01-16T15:00:08.544924+0000","last_active":"2024-01-16T15:00:08.547360+0000","last_peered":"2024-01-16T15:00:08.547360+0000","last_clean":"2024-01-16T15:00:08.547360+0000","last_became_active":"2024-01-16T15:00:08.544446+0000","last_became_peered":"2024-01-16T15:00:08.544446+0000","last_unstale":"2024-01-16T15:00:08.547360+0000","last_undegraded":"2024-01-16T15:00:08.547360+0000","last_fullsized":"2024-01-16T15:00:08.547360+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:14.742590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566162+0000","last_change":"2024-01-16T15:00:08.561486+0000","last_active":"2024-01-16T15:00:08.566162+0000","last_peered":"2024-01-16T15:00:08.566162+0000","last_clean":"2024-01-16T15:00:08.566162+0000","last_became_active":"2024-01-16T15:00:08.561192+0000","last_became_peered":"2024-01-16T15:00:08.561192+0000","last_unstale":"2024-01-16T15:00:08.566162+0000","last_undegraded":"2024-01-16T15:00:08.566162+0000","last_fullsized":"2024-01-16T15:00:08.566162+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:10.767361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571003+0000","last_change":"2024-01-16T15:00:08.562984+0000","last_active":"2024-01-16T15:00:08.571003+0000","last_peered":"2024-01-16T15:00:08.571003+0000","last_clean":"2024-01-16T15:00:08.571003+0000","last_became_active":"2024-01-16T15:00:08.562708+0000","last_became_peered":"2024-01-16T15:00:08.562708+0000","last_unstale":"2024-01-16T15:00:08.571003+0000","last_undegraded":"2024-01-16T15:00:08.571003+0000","last_fullsized":"2024-01-16T15:00:08.571003+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:20:03.366522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.545478+0000","last_change":"2024-01-16T15:00:08.537951+0000","last_active":"2024-01-16T15:00:08.545478+0000","last_peered":"2024-01-16T15:00:08.545478+0000","last_clean":"2024-01-16T15:00:08.545478+0000","last_became_active":"2024-01-16T15:00:08.537603+0000","last_became_peered":"2024-01-16T15:00:08.537603+0000","last_unstale":"2024-01-16T15:00:08.545478+0000","last_undegraded":"2024-01-16T15:00:08.545478+0000","last_fullsized":"2024-01-16T15:00:08.545478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:51.295894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.544332+0000","last_change":"2024-01-16T15:00:08.539930+0000","last_active":"2024-01-16T15:00:08.544332+0000","last_peered":"2024-01-16T15:00:08.544332+0000","last_clean":"2024-01-16T15:00:08.544332+0000","last_became_active":"2024-01-16T15:00:08.539657+0000","last_became_peered":"2024-01-16T15:00:08.539657+0000","last_unstale":"2024-01-16T15:00:08.544332+0000","last_undegraded":"2024-01-16T15:00:08.544332+0000","last_fullsized":"2024-01-16T15:00:08.544332+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:12.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566226+0000","last_change":"2024-01-16T15:00:08.562005+0000","last_active":"2024-01-16T15:00:08.566226+0000","last_peered":"2024-01-16T15:00:08.566226+0000","last_clean":"2024-01-16T15:00:08.566226+0000","last_became_active":"2024-01-16T15:00:08.561689+0000","last_became_peered":"2024-01-16T15:00:08.561689+0000","last_unstale":"2024-01-16T15:00:08.566226+0000","last_undegraded":"2024-01-16T15:00:08.566226+0000","last_fullsized":"2024-01-16T15:00:08.566226+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:44.207622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.546400+0000","last_change":"2024-01-16T15:00:08.539760+0000","last_active":"2024-01-16T15:00:08.546400+0000","last_peered":"2024-01-16T15:00:08.546400+0000","last_clean":"2024-01-16T15:00:08.546400+0000","last_became_active":"2024-01-16T15:00:08.538972+0000","last_became_peered":"2024-01-16T15:00:08.538972+0000","last_unstale":"2024-01-16T15:00:08.546400+0000","last_undegraded":"2024-01-16T15:00:08.546400+0000","last_fullsized":"2024-01-16T15:00:08.546400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:07:41.025992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.553186+0000","last_change":"2024-01-16T15:00:08.545311+0000","last_active":"2024-01-16T15:00:08.553186+0000","last_peered":"2024-01-16T15:00:08.553186+0000","last_clean":"2024-01-16T15:00:08.553186+0000","last_became_active":"2024-01-16T15:00:08.545033+0000","last_became_peered":"2024-01-16T15:00:08.545033+0000","last_unstale":"2024-01-16T15:00:08.553186+0000","last_undegraded":"2024-01-16T15:00:08.553186+0000","last_fullsized":"2024-01-16T15:00:08.553186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:25.113062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.562469+0000","last_change":"2024-01-16T15:00:08.552664+0000","last_active":"2024-01-16T15:00:08.562469+0000","last_peered":"2024-01-16T15:00:08.562469+0000","last_clean":"2024-01-16T15:00:08.562469+0000","last_became_active":"2024-01-16T15:00:08.552340+0000","last_became_peered":"2024-01-16T15:00:08.552340+0000","last_unstale":"2024-01-16T15:00:08.562469+0000","last_undegraded":"2024-01-16T15:00:08.562469+0000","last_fullsized":"2024-01-16T15:00:08.562469+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:18:16.018305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569478+0000","last_change":"2024-01-16T15:00:08.566380+0000","last_active":"2024-01-16T15:00:08.569478+0000","last_peered":"2024-01-16T15:00:08.569478+0000","last_clean":"2024-01-16T15:00:08.569478+0000","last_became_active":"2024-01-16T15:00:08.566093+0000","last_became_peered":"2024-01-16T15:00:08.566093+0000","last_unstale":"2024-01-16T15:00:08.569478+0000","last_undegraded":"2024-01-16T15:00:08.569478+0000","last_fullsized":"2024-01-16T15:00:08.569478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:51.295894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571412+0000","last_change":"2024-01-16T15:00:07.527871+0000","last_active":"2024-01-16T15:00:08.571412+0000","last_peered":"2024-01-16T15:00:08.571412+0000","last_clean":"2024-01-16T15:00:08.571412+0000","last_became_active":"2024-01-16T15:00:07.527410+0000","last_became_peered":"2024-01-16T15:00:07.527410+0000","last_unstale":"2024-01-16T15:00:08.571412+0000","last_undegraded":"2024-01-16T15:00:08.571412+0000","last_fullsized":"2024-01-16T15:00:08.571412+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:08.792642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552641+0000","last_change":"2024-01-16T15:00:08.547896+0000","last_active":"2024-01-16T15:00:08.552641+0000","last_peered":"2024-01-16T15:00:08.552641+0000","last_clean":"2024-01-16T15:00:08.552641+0000","last_became_active":"2024-01-16T15:00:08.547555+0000","last_became_peered":"2024-01-16T15:00:08.547555+0000","last_unstale":"2024-01-16T15:00:08.552641+0000","last_undegraded":"2024-01-16T15:00:08.552641+0000","last_fullsized":"2024-01-16T15:00:08.552641+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:37.924541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.555671+0000","last_change":"2024-01-16T15:00:07.525831+0000","last_active":"2024-01-16T15:00:08.555671+0000","last_peered":"2024-01-16T15:00:08.555671+0000","last_clean":"2024-01-16T15:00:08.555671+0000","last_became_active":"2024-01-16T15:00:07.524073+0000","last_became_peered":"2024-01-16T15:00:07.524073+0000","last_unstale":"2024-01-16T15:00:08.555671+0000","last_undegraded":"2024-01-16T15:00:08.555671+0000","last_fullsized":"2024-01-16T15:00:08.555671+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:01.203257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554345+0000","last_change":"2024-01-16T15:00:08.548698+0000","last_active":"2024-01-16T15:00:08.554345+0000","last_peered":"2024-01-16T15:00:08.554345+0000","last_clean":"2024-01-16T15:00:08.554345+0000","last_became_active":"2024-01-16T15:00:08.548300+0000","last_became_peered":"2024-01-16T15:00:08.548300+0000","last_unstale":"2024-01-16T15:00:08.554345+0000","last_undegraded":"2024-01-16T15:00:08.554345+0000","last_fullsized":"2024-01-16T15:00:08.554345+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:20.095673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.574328+0000","last_change":"2024-01-16T15:00:07.524362+0000","last_active":"2024-01-16T15:00:08.574328+0000","last_peered":"2024-01-16T15:00:08.574328+0000","last_clean":"2024-01-16T15:00:08.574328+0000","last_became_active":"2024-01-16T15:00:07.524070+0000","last_became_peered":"2024-01-16T15:00:07.524070+0000","last_unstale":"2024-01-16T15:00:08.574328+0000","last_undegraded":"2024-01-16T15:00:08.574328+0000","last_fullsized":"2024-01-16T15:00:08.574328+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:11.875064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569368+0000","last_change":"2024-01-16T15:00:08.560285+0000","last_active":"2024-01-16T15:00:08.569368+0000","last_peered":"2024-01-16T15:00:08.569368+0000","last_clean":"2024-01-16T15:00:08.569368+0000","last_became_active":"2024-01-16T15:00:08.555077+0000","last_became_peered":"2024-01-16T15:00:08.555077+0000","last_unstale":"2024-01-16T15:00:08.569368+0000","last_undegraded":"2024-01-16T15:00:08.569368+0000","last_fullsized":"2024-01-16T15:00:08.569368+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:23.521963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563928+0000","last_change":"2024-01-16T15:00:07.527507+0000","last_active":"2024-01-16T15:00:08.563928+0000","last_peered":"2024-01-16T15:00:08.563928+0000","last_clean":"2024-01-16T15:00:08.563928+0000","last_became_active":"2024-01-16T15:00:07.526378+0000","last_became_peered":"2024-01-16T15:00:07.526378+0000","last_unstale":"2024-01-16T15:00:08.563928+0000","last_undegraded":"2024-01-16T15:00:08.563928+0000","last_fullsized":"2024-01-16T15:00:08.563928+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:55.319675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547249+0000","last_change":"2024-01-16T15:00:08.541357+0000","last_active":"2024-01-16T15:00:08.547249+0000","last_peered":"2024-01-16T15:00:08.547249+0000","last_clean":"2024-01-16T15:00:08.547249+0000","last_became_active":"2024-01-16T15:00:08.541070+0000","last_became_peered":"2024-01-16T15:00:08.541070+0000","last_unstale":"2024-01-16T15:00:08.547249+0000","last_undegraded":"2024-01-16T15:00:08.547249+0000","last_fullsized":"2024-01-16T15:00:08.547249+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:12.438613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571320+0000","last_change":"2024-01-16T15:00:07.524641+0000","last_active":"2024-01-16T15:00:08.571320+0000","last_peered":"2024-01-16T15:00:08.571320+0000","last_clean":"2024-01-16T15:00:08.571320+0000","last_became_active":"2024-01-16T15:00:07.524317+0000","last_became_peered":"2024-01-16T15:00:07.524317+0000","last_unstale":"2024-01-16T15:00:08.571320+0000","last_undegraded":"2024-01-16T15:00:08.571320+0000","last_fullsized":"2024-01-16T15:00:08.571320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:06.500463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568904+0000","last_change":"2024-01-16T15:00:08.556634+0000","last_active":"2024-01-16T15:00:08.568904+0000","last_peered":"2024-01-16T15:00:08.568904+0000","last_clean":"2024-01-16T15:00:08.568904+0000","last_became_active":"2024-01-16T15:00:08.556291+0000","last_became_peered":"2024-01-16T15:00:08.556291+0000","last_unstale":"2024-01-16T15:00:08.568904+0000","last_undegraded":"2024-01-16T15:00:08.568904+0000","last_fullsized":"2024-01-16T15:00:08.568904+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:53.707528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.555517+0000","last_change":"2024-01-16T15:00:07.527016+0000","last_active":"2024-01-16T15:00:08.555517+0000","last_peered":"2024-01-16T15:00:08.555517+0000","last_clean":"2024-01-16T15:00:08.555517+0000","last_became_active":"2024-01-16T15:00:07.526723+0000","last_became_peered":"2024-01-16T15:00:07.526723+0000","last_unstale":"2024-01-16T15:00:08.555517+0000","last_undegraded":"2024-01-16T15:00:08.555517+0000","last_fullsized":"2024-01-16T15:00:08.555517+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:12.674764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.561890+0000","last_change":"2024-01-16T15:00:08.550819+0000","last_active":"2024-01-16T15:00:08.561890+0000","last_peered":"2024-01-16T15:00:08.561890+0000","last_clean":"2024-01-16T15:00:08.561890+0000","last_became_active":"2024-01-16T15:00:08.550551+0000","last_became_peered":"2024-01-16T15:00:08.550551+0000","last_unstale":"2024-01-16T15:00:08.561890+0000","last_undegraded":"2024-01-16T15:00:08.561890+0000","last_fullsized":"2024-01-16T15:00:08.561890+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:34.775356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.524180+0000","last_change":"2024-01-16T15:00:07.523327+0000","last_active":"2024-01-16T15:00:08.524180+0000","last_peered":"2024-01-16T15:00:08.524180+0000","last_clean":"2024-01-16T15:00:08.524180+0000","last_became_active":"2024-01-16T15:00:07.523011+0000","last_became_peered":"2024-01-16T15:00:07.523011+0000","last_unstale":"2024-01-16T15:00:08.524180+0000","last_undegraded":"2024-01-16T15:00:08.524180+0000","last_fullsized":"2024-01-16T15:00:08.524180+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:39.594396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552587+0000","last_change":"2024-01-16T15:00:08.547054+0000","last_active":"2024-01-16T15:00:08.552587+0000","last_peered":"2024-01-16T15:00:08.552587+0000","last_clean":"2024-01-16T15:00:08.552587+0000","last_became_active":"2024-01-16T15:00:08.546741+0000","last_became_peered":"2024-01-16T15:00:08.546741+0000","last_unstale":"2024-01-16T15:00:08.552587+0000","last_undegraded":"2024-01-16T15:00:08.552587+0000","last_fullsized":"2024-01-16T15:00:08.552587+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:25.902778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.540369+0000","last_change":"2024-01-16T15:00:07.520711+0000","last_active":"2024-01-16T15:00:08.540369+0000","last_peered":"2024-01-16T15:00:08.540369+0000","last_clean":"2024-01-16T15:00:08.540369+0000","last_became_active":"2024-01-16T15:00:07.520440+0000","last_became_peered":"2024-01-16T15:00:07.520440+0000","last_unstale":"2024-01-16T15:00:08.540369+0000","last_undegraded":"2024-01-16T15:00:08.540369+0000","last_fullsized":"2024-01-16T15:00:08.540369+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:39.974432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.560765+0000","last_change":"2024-01-16T15:00:08.558301+0000","last_active":"2024-01-16T15:00:08.560765+0000","last_peered":"2024-01-16T15:00:08.560765+0000","last_clean":"2024-01-16T15:00:08.560765+0000","last_became_active":"2024-01-16T15:00:08.558040+0000","last_became_peered":"2024-01-16T15:00:08.558040+0000","last_unstale":"2024-01-16T15:00:08.560765+0000","last_undegraded":"2024-01-16T15:00:08.560765+0000","last_fullsized":"2024-01-16T15:00:08.560765+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:26.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.530143+0000","last_change":"2024-01-16T15:00:07.509925+0000","last_active":"2024-01-16T15:00:08.530143+0000","last_peered":"2024-01-16T15:00:08.530143+0000","last_clean":"2024-01-16T15:00:08.530143+0000","last_became_active":"2024-01-16T15:00:07.509361+0000","last_became_peered":"2024-01-16T15:00:07.509361+0000","last_unstale":"2024-01-16T15:00:08.530143+0000","last_undegraded":"2024-01-16T15:00:08.530143+0000","last_fullsized":"2024-01-16T15:00:08.530143+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:46.950866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.543208+0000","last_change":"2024-01-16T15:00:08.536213+0000","last_active":"2024-01-16T15:00:08.543208+0000","last_peered":"2024-01-16T15:00:08.543208+0000","last_clean":"2024-01-16T15:00:08.543208+0000","last_became_active":"2024-01-16T15:00:08.532964+0000","last_became_peered":"2024-01-16T15:00:08.532964+0000","last_unstale":"2024-01-16T15:00:08.543208+0000","last_undegraded":"2024-01-16T15:00:08.543208+0000","last_fullsized":"2024-01-16T15:00:08.543208+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:13.644323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547080+0000","last_change":"2024-01-16T15:00:07.522394+0000","last_active":"2024-01-16T15:00:08.547080+0000","last_peered":"2024-01-16T15:00:08.547080+0000","last_clean":"2024-01-16T15:00:08.547080+0000","last_became_active":"2024-01-16T15:00:07.522116+0000","last_became_peered":"2024-01-16T15:00:07.522116+0000","last_unstale":"2024-01-16T15:00:08.547080+0000","last_undegraded":"2024-01-16T15:00:08.547080+0000","last_fullsized":"2024-01-16T15:00:08.547080+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:12.769085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.559511+0000","last_change":"2024-01-16T15:00:08.555601+0000","last_active":"2024-01-16T15:00:08.559511+0000","last_peered":"2024-01-16T15:00:08.559511+0000","last_clean":"2024-01-16T15:00:08.559511+0000","last_became_active":"2024-01-16T15:00:08.555315+0000","last_became_peered":"2024-01-16T15:00:08.555315+0000","last_unstale":"2024-01-16T15:00:08.559511+0000","last_undegraded":"2024-01-16T15:00:08.559511+0000","last_fullsized":"2024-01-16T15:00:08.559511+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:01:52.394343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566256+0000","last_change":"2024-01-16T15:00:07.527997+0000","last_active":"2024-01-16T15:00:08.566256+0000","last_peered":"2024-01-16T15:00:08.566256+0000","last_clean":"2024-01-16T15:00:08.566256+0000","last_became_active":"2024-01-16T15:00:07.527734+0000","last_became_peered":"2024-01-16T15:00:07.527734+0000","last_unstale":"2024-01-16T15:00:08.566256+0000","last_undegraded":"2024-01-16T15:00:08.566256+0000","last_fullsized":"2024-01-16T15:00:08.566256+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:42.035002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563827+0000","last_change":"2024-01-16T15:00:08.561750+0000","last_active":"2024-01-16T15:00:08.563827+0000","last_peered":"2024-01-16T15:00:08.563827+0000","last_clean":"2024-01-16T15:00:08.563827+0000","last_became_active":"2024-01-16T15:00:08.561473+0000","last_became_peered":"2024-01-16T15:00:08.561473+0000","last_unstale":"2024-01-16T15:00:08.563827+0000","last_undegraded":"2024-01-16T15:00:08.563827+0000","last_fullsized":"2024-01-16T15:00:08.563827+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:04.210496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.574246+0000","last_change":"2024-01-16T15:00:07.515751+0000","last_active":"2024-01-16T15:00:08.574246+0000","last_peered":"2024-01-16T15:00:08.574246+0000","last_clean":"2024-01-16T15:00:08.574246+0000","last_became_active":"2024-01-16T15:00:07.514812+0000","last_became_peered":"2024-01-16T15:00:07.514812+0000","last_unstale":"2024-01-16T15:00:08.574246+0000","last_undegraded":"2024-01-16T15:00:08.574246+0000","last_fullsized":"2024-01-16T15:00:08.574246+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:17:19.391074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563636+0000","last_change":"2024-01-16T15:00:08.560164+0000","last_active":"2024-01-16T15:00:08.563636+0000","last_peered":"2024-01-16T15:00:08.563636+0000","last_clean":"2024-01-16T15:00:08.563636+0000","last_became_active":"2024-01-16T15:00:08.559883+0000","last_became_peered":"2024-01-16T15:00:08.559883+0000","last_unstale":"2024-01-16T15:00:08.563636+0000","last_undegraded":"2024-01-16T15:00:08.563636+0000","last_fullsized":"2024-01-16T15:00:08.563636+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:58.168006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571982+0000","last_change":"2024-01-16T15:00:07.518437+0000","last_active":"2024-01-16T15:00:08.571982+0000","last_peered":"2024-01-16T15:00:08.571982+0000","last_clean":"2024-01-16T15:00:08.571982+0000","last_became_active":"2024-01-16T15:00:07.518169+0000","last_became_peered":"2024-01-16T15:00:07.518169+0000","last_unstale":"2024-01-16T15:00:08.571982+0000","last_undegraded":"2024-01-16T15:00:08.571982+0000","last_fullsized":"2024-01-16T15:00:08.571982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:43.595100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.556142+0000","last_change":"2024-01-16T15:00:08.536009+0000","last_active":"2024-01-16T15:00:08.556142+0000","last_peered":"2024-01-16T15:00:08.556142+0000","last_clean":"2024-01-16T15:00:08.556142+0000","last_became_active":"2024-01-16T15:00:08.535704+0000","last_became_peered":"2024-01-16T15:00:08.535704+0000","last_unstale":"2024-01-16T15:00:08.556142+0000","last_undegraded":"2024-01-16T15:00:08.556142+0000","last_fullsized":"2024-01-16T15:00:08.556142+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:31.059425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.526683+0000","last_change":"2024-01-16T15:00:07.526795+0000","last_active":"2024-01-16T15:00:08.526683+0000","last_peered":"2024-01-16T15:00:08.526683+0000","last_clean":"2024-01-16T15:00:08.526683+0000","last_became_active":"2024-01-16T15:00:07.526461+0000","last_became_peered":"2024-01-16T15:00:07.526461+0000","last_unstale":"2024-01-16T15:00:08.526683+0000","last_undegraded":"2024-01-16T15:00:08.526683+0000","last_fullsized":"2024-01-16T15:00:08.526683+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:26.258788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.551777+0000","last_change":"2024-01-16T15:00:08.548663+0000","last_active":"2024-01-16T15:00:08.551777+0000","last_peered":"2024-01-16T15:00:08.551777+0000","last_clean":"2024-01-16T15:00:08.551777+0000","last_became_active":"2024-01-16T15:00:08.548390+0000","last_became_peered":"2024-01-16T15:00:08.548390+0000","last_unstale":"2024-01-16T15:00:08.551777+0000","last_undegraded":"2024-01-16T15:00:08.551777+0000","last_fullsized":"2024-01-16T15:00:08.551777+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:29:19.904575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.527301+0000","last_change":"2024-01-16T15:00:07.518411+0000","last_active":"2024-01-16T15:00:08.527301+0000","last_peered":"2024-01-16T15:00:08.527301+0000","last_clean":"2024-01-16T15:00:08.527301+0000","last_became_active":"2024-01-16T15:00:07.518124+0000","last_became_peered":"2024-01-16T15:00:07.518124+0000","last_unstale":"2024-01-16T15:00:08.527301+0000","last_undegraded":"2024-01-16T15:00:08.527301+0000","last_fullsized":"2024-01-16T15:00:08.527301+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:52.845621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547130+0000","last_change":"2024-01-16T15:00:08.542930+0000","last_active":"2024-01-16T15:00:08.547130+0000","last_peered":"2024-01-16T15:00:08.547130+0000","last_clean":"2024-01-16T15:00:08.547130+0000","last_became_active":"2024-01-16T15:00:08.542625+0000","last_became_peered":"2024-01-16T15:00:08.542625+0000","last_unstale":"2024-01-16T15:00:08.547130+0000","last_undegraded":"2024-01-16T15:00:08.547130+0000","last_fullsized":"2024-01-16T15:00:08.547130+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:59:17.485656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":19,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.573734+0000","last_change":"2024-01-16T15:00:07.525232+0000","last_active":"2024-01-16T15:00:08.573734+0000","last_peered":"2024-01-16T15:00:08.573734+0000","last_clean":"2024-01-16T15:00:08.573734+0000","last_became_active":"2024-01-16T15:00:07.524943+0000","last_became_peered":"2024-01-16T15:00:07.524943+0000","last_unstale":"2024-01-16T15:00:08.573734+0000","last_undegraded":"2024-01-16T15:00:08.573734+0000","last_fullsized":"2024-01-16T15:00:08.573734+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:40.597612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.548468+0000","last_change":"2024-01-16T15:00:08.537997+0000","last_active":"2024-01-16T15:00:08.548468+0000","last_peered":"2024-01-16T15:00:08.548468+0000","last_clean":"2024-01-16T15:00:08.548468+0000","last_became_active":"2024-01-16T15:00:08.537674+0000","last_became_peered":"2024-01-16T15:00:08.537674+0000","last_unstale":"2024-01-16T15:00:08.548468+0000","last_undegraded":"2024-01-16T15:00:08.548468+0000","last_fullsized":"2024-01-16T15:00:08.548468+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:56.758344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.572186+0000","last_change":"2024-01-16T15:00:07.515769+0000","last_active":"2024-01-16T15:00:08.572186+0000","last_peered":"2024-01-16T15:00:08.572186+0000","last_clean":"2024-01-16T15:00:08.572186+0000","last_became_active":"2024-01-16T15:00:07.514876+0000","last_became_peered":"2024-01-16T15:00:07.514876+0000","last_unstale":"2024-01-16T15:00:08.572186+0000","last_undegraded":"2024-01-16T15:00:08.572186+0000","last_fullsized":"2024-01-16T15:00:08.572186+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:11:09.934752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.559775+0000","last_change":"2024-01-16T15:00:08.549642+0000","last_active":"2024-01-16T15:00:08.559775+0000","last_peered":"2024-01-16T15:00:08.559775+0000","last_clean":"2024-01-16T15:00:08.559775+0000","last_became_active":"2024-01-16T15:00:08.549325+0000","last_became_peered":"2024-01-16T15:00:08.549325+0000","last_unstale":"2024-01-16T15:00:08.559775+0000","last_undegraded":"2024-01-16T15:00:08.559775+0000","last_fullsized":"2024-01-16T15:00:08.559775+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:43.578946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566624+0000","last_change":"2024-01-16T15:00:07.516841+0000","last_active":"2024-01-16T15:00:08.566624+0000","last_peered":"2024-01-16T15:00:08.566624+0000","last_clean":"2024-01-16T15:00:08.566624+0000","last_became_active":"2024-01-16T15:00:07.516522+0000","last_became_peered":"2024-01-16T15:00:07.516522+0000","last_unstale":"2024-01-16T15:00:08.566624+0000","last_undegraded":"2024-01-16T15:00:08.566624+0000","last_fullsized":"2024-01-16T15:00:08.566624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:22.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552779+0000","last_change":"2024-01-16T15:00:08.549017+0000","last_active":"2024-01-16T15:00:08.552779+0000","last_peered":"2024-01-16T15:00:08.552779+0000","last_clean":"2024-01-16T15:00:08.552779+0000","last_became_active":"2024-01-16T15:00:08.548730+0000","last_became_peered":"2024-01-16T15:00:08.548730+0000","last_unstale":"2024-01-16T15:00:08.552779+0000","last_undegraded":"2024-01-16T15:00:08.552779+0000","last_fullsized":"2024-01-16T15:00:08.552779+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:05.699896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.572786+0000","last_change":"2024-01-16T15:00:07.523324+0000","last_active":"2024-01-16T15:00:08.572786+0000","last_peered":"2024-01-16T15:00:08.572786+0000","last_clean":"2024-01-16T15:00:08.572786+0000","last_became_active":"2024-01-16T15:00:07.523035+0000","last_became_peered":"2024-01-16T15:00:07.523035+0000","last_unstale":"2024-01-16T15:00:08.572786+0000","last_undegraded":"2024-01-16T15:00:08.572786+0000","last_fullsized":"2024-01-16T15:00:08.572786+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:03.852422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566309+0000","last_change":"2024-01-16T15:00:08.561723+0000","last_active":"2024-01-16T15:00:08.566309+0000","last_peered":"2024-01-16T15:00:08.566309+0000","last_clean":"2024-01-16T15:00:08.566309+0000","last_became_active":"2024-01-16T15:00:08.561379+0000","last_became_peered":"2024-01-16T15:00:08.561379+0000","last_unstale":"2024-01-16T15:00:08.566309+0000","last_undegraded":"2024-01-16T15:00:08.566309+0000","last_fullsized":"2024-01-16T15:00:08.566309+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:18.717365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.531908+0000","last_change":"2024-01-16T15:00:07.525994+0000","last_active":"2024-01-16T15:00:08.531908+0000","last_peered":"2024-01-16T15:00:08.531908+0000","last_clean":"2024-01-16T15:00:08.531908+0000","last_became_active":"2024-01-16T15:00:07.525765+0000","last_became_peered":"2024-01-16T15:00:07.525765+0000","last_unstale":"2024-01-16T15:00:08.531908+0000","last_undegraded":"2024-01-16T15:00:08.531908+0000","last_fullsized":"2024-01-16T15:00:08.531908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:44.875264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563116+0000","last_change":"2024-01-16T15:00:08.557781+0000","last_active":"2024-01-16T15:00:08.563116+0000","last_peered":"2024-01-16T15:00:08.563116+0000","last_clean":"2024-01-16T15:00:08.563116+0000","last_became_active":"2024-01-16T15:00:08.557413+0000","last_became_peered":"2024-01-16T15:00:08.557413+0000","last_unstale":"2024-01-16T15:00:08.563116+0000","last_undegraded":"2024-01-16T15:00:08.563116+0000","last_fullsized":"2024-01-16T15:00:08.563116+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:28.697891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.572033+0000","last_change":"2024-01-16T15:00:07.527222+0000","last_active":"2024-01-16T15:00:08.572033+0000","last_peered":"2024-01-16T15:00:08.572033+0000","last_clean":"2024-01-16T15:00:08.572033+0000","last_became_active":"2024-01-16T15:00:07.526918+0000","last_became_peered":"2024-01-16T15:00:07.526918+0000","last_unstale":"2024-01-16T15:00:08.572033+0000","last_undegraded":"2024-01-16T15:00:08.572033+0000","last_fullsized":"2024-01-16T15:00:08.572033+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:12.165083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569445+0000","last_change":"2024-01-16T15:00:08.564185+0000","last_active":"2024-01-16T15:00:08.569445+0000","last_peered":"2024-01-16T15:00:08.569445+0000","last_clean":"2024-01-16T15:00:08.569445+0000","last_became_active":"2024-01-16T15:00:08.560525+0000","last_became_peered":"2024-01-16T15:00:08.560525+0000","last_unstale":"2024-01-16T15:00:08.569445+0000","last_undegraded":"2024-01-16T15:00:08.569445+0000","last_fullsized":"2024-01-16T15:00:08.569445+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:02.217215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.525908+0000","last_change":"2024-01-16T15:00:07.522096+0000","last_active":"2024-01-16T15:00:08.525908+0000","last_peered":"2024-01-16T15:00:08.525908+0000","last_clean":"2024-01-16T15:00:08.525908+0000","last_became_active":"2024-01-16T15:00:07.521466+0000","last_became_peered":"2024-01-16T15:00:07.521466+0000","last_unstale":"2024-01-16T15:00:08.525908+0000","last_undegraded":"2024-01-16T15:00:08.525908+0000","last_fullsized":"2024-01-16T15:00:08.525908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:12.630365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552523+0000","last_change":"2024-01-16T15:00:08.544028+0000","last_active":"2024-01-16T15:00:08.552523+0000","last_peered":"2024-01-16T15:00:08.552523+0000","last_clean":"2024-01-16T15:00:08.552523+0000","last_became_active":"2024-01-16T15:00:08.543635+0000","last_became_peered":"2024-01-16T15:00:08.543635+0000","last_unstale":"2024-01-16T15:00:08.552523+0000","last_undegraded":"2024-01-16T15:00:08.552523+0000","last_fullsized":"2024-01-16T15:00:08.552523+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:33.048860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.524361+0000","last_change":"2024-01-16T15:00:07.523438+0000","last_active":"2024-01-16T15:00:08.524361+0000","last_peered":"2024-01-16T15:00:08.524361+0000","last_clean":"2024-01-16T15:00:08.524361+0000","last_became_active":"2024-01-16T15:00:07.523084+0000","last_became_peered":"2024-01-16T15:00:07.523084+0000","last_unstale":"2024-01-16T15:00:08.524361+0000","last_undegraded":"2024-01-16T15:00:08.524361+0000","last_fullsized":"2024-01-16T15:00:08.524361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:15.387500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563195+0000","last_change":"2024-01-16T15:00:08.556154+0000","last_active":"2024-01-16T15:00:08.563195+0000","last_peered":"2024-01-16T15:00:08.563195+0000","last_clean":"2024-01-16T15:00:08.563195+0000","last_became_active":"2024-01-16T15:00:08.555728+0000","last_became_peered":"2024-01-16T15:00:08.555728+0000","last_unstale":"2024-01-16T15:00:08.563195+0000","last_undegraded":"2024-01-16T15:00:08.563195+0000","last_fullsized":"2024-01-16T15:00:08.563195+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:35:36.232094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.523201+0000","last_change":"2024-01-16T15:00:07.526002+0000","last_active":"2024-01-16T15:00:08.523201+0000","last_peered":"2024-01-16T15:00:08.523201+0000","last_clean":"2024-01-16T15:00:08.523201+0000","last_became_active":"2024-01-16T15:00:07.525647+0000","last_became_peered":"2024-01-16T15:00:07.525647+0000","last_unstale":"2024-01-16T15:00:08.523201+0000","last_undegraded":"2024-01-16T15:00:08.523201+0000","last_fullsized":"2024-01-16T15:00:08.523201+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:19.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566096+0000","last_change":"2024-01-16T15:00:08.560698+0000","last_active":"2024-01-16T15:00:08.566096+0000","last_peered":"2024-01-16T15:00:08.566096+0000","last_clean":"2024-01-16T15:00:08.566096+0000","last_became_active":"2024-01-16T15:00:08.560052+0000","last_became_peered":"2024-01-16T15:00:08.560052+0000","last_unstale":"2024-01-16T15:00:08.566096+0000","last_undegraded":"2024-01-16T15:00:08.566096+0000","last_fullsized":"2024-01-16T15:00:08.566096+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:57.480817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.531062+0000","last_change":"2024-01-16T15:00:07.524975+0000","last_active":"2024-01-16T15:00:08.531062+0000","last_peered":"2024-01-16T15:00:08.531062+0000","last_clean":"2024-01-16T15:00:08.531062+0000","last_became_active":"2024-01-16T15:00:07.524369+0000","last_became_peered":"2024-01-16T15:00:07.524369+0000","last_unstale":"2024-01-16T15:00:08.531062+0000","last_undegraded":"2024-01-16T15:00:08.531062+0000","last_fullsized":"2024-01-16T15:00:08.531062+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:44.886009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554528+0000","last_change":"2024-01-16T15:00:08.537143+0000","last_active":"2024-01-16T15:00:08.554528+0000","last_peered":"2024-01-16T15:00:08.554528+0000","last_clean":"2024-01-16T15:00:08.554528+0000","last_became_active":"2024-01-16T15:00:08.536676+0000","last_became_peered":"2024-01-16T15:00:08.536676+0000","last_unstale":"2024-01-16T15:00:08.554528+0000","last_undegraded":"2024-01-16T15:00:08.554528+0000","last_fullsized":"2024-01-16T15:00:08.554528+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:26.566875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.526360+0000","last_change":"2024-01-16T15:00:07.528578+0000","last_active":"2024-01-16T15:00:08.526360+0000","last_peered":"2024-01-16T15:00:08.526360+0000","last_clean":"2024-01-16T15:00:08.526360+0000","last_became_active":"2024-01-16T15:00:07.528259+0000","last_became_peered":"2024-01-16T15:00:07.528259+0000","last_unstale":"2024-01-16T15:00:08.526360+0000","last_undegraded":"2024-01-16T15:00:08.526360+0000","last_fullsized":"2024-01-16T15:00:08.526360+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:01.903132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.546878+0000","last_change":"2024-01-16T15:00:08.545558+0000","last_active":"2024-01-16T15:00:08.546878+0000","last_peered":"2024-01-16T15:00:08.546878+0000","last_clean":"2024-01-16T15:00:08.546878+0000","last_became_active":"2024-01-16T15:00:08.545257+0000","last_became_peered":"2024-01-16T15:00:08.545257+0000","last_unstale":"2024-01-16T15:00:08.546878+0000","last_undegraded":"2024-01-16T15:00:08.546878+0000","last_fullsized":"2024-01-16T15:00:08.546878+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:04.866379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.551512+0000","last_change":"2024-01-16T15:00:07.521641+0000","last_active":"2024-01-16T15:00:08.551512+0000","last_peered":"2024-01-16T15:00:08.551512+0000","last_clean":"2024-01-16T15:00:08.551512+0000","last_became_active":"2024-01-16T15:00:07.521431+0000","last_became_peered":"2024-01-16T15:00:07.521431+0000","last_unstale":"2024-01-16T15:00:08.551512+0000","last_undegraded":"2024-01-16T15:00:08.551512+0000","last_fullsized":"2024-01-16T15:00:08.551512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:19.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571400+0000","last_change":"2024-01-16T15:00:08.565783+0000","last_active":"2024-01-16T15:00:08.571400+0000","last_peered":"2024-01-16T15:00:08.571400+0000","last_clean":"2024-01-16T15:00:08.571400+0000","last_became_active":"2024-01-16T15:00:08.565524+0000","last_became_peered":"2024-01-16T15:00:08.565524+0000","last_unstale":"2024-01-16T15:00:08.571400+0000","last_undegraded":"2024-01-16T15:00:08.571400+0000","last_fullsized":"2024-01-16T15:00:08.571400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:43:36.337639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.526138+0000","last_change":"2024-01-16T15:00:07.528974+0000","last_active":"2024-01-16T15:00:08.526138+0000","last_peered":"2024-01-16T15:00:08.526138+0000","last_clean":"2024-01-16T15:00:08.526138+0000","last_became_active":"2024-01-16T15:00:07.528466+0000","last_became_peered":"2024-01-16T15:00:07.528466+0000","last_unstale":"2024-01-16T15:00:08.526138+0000","last_undegraded":"2024-01-16T15:00:08.526138+0000","last_fullsized":"2024-01-16T15:00:08.526138+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:59.733387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.567905+0000","last_change":"2024-01-16T15:00:08.559998+0000","last_active":"2024-01-16T15:00:08.567905+0000","last_peered":"2024-01-16T15:00:08.567905+0000","last_clean":"2024-01-16T15:00:08.567905+0000","last_became_active":"2024-01-16T15:00:08.559664+0000","last_became_peered":"2024-01-16T15:00:08.559664+0000","last_unstale":"2024-01-16T15:00:08.567905+0000","last_undegraded":"2024-01-16T15:00:08.567905+0000","last_fullsized":"2024-01-16T15:00:08.567905+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:43.478443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.521948+0000","last_change":"2024-01-16T15:00:07.519669+0000","last_active":"2024-01-16T15:00:08.521948+0000","last_peered":"2024-01-16T15:00:08.521948+0000","last_clean":"2024-01-16T15:00:08.521948+0000","last_became_active":"2024-01-16T15:00:07.517915+0000","last_became_peered":"2024-01-16T15:00:07.517915+0000","last_unstale":"2024-01-16T15:00:08.521948+0000","last_undegraded":"2024-01-16T15:00:08.521948+0000","last_fullsized":"2024-01-16T15:00:08.521948+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:07.206101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.555206+0000","last_change":"2024-01-16T15:00:08.552602+0000","last_active":"2024-01-16T15:00:08.555206+0000","last_peered":"2024-01-16T15:00:08.555206+0000","last_clean":"2024-01-16T15:00:08.555206+0000","last_became_active":"2024-01-16T15:00:08.552271+0000","last_became_peered":"2024-01-16T15:00:08.552271+0000","last_unstale":"2024-01-16T15:00:08.555206+0000","last_undegraded":"2024-01-16T15:00:08.555206+0000","last_fullsized":"2024-01-16T15:00:08.555206+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:59:17.485656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.550736+0000","last_change":"2024-01-16T15:00:07.526284+0000","last_active":"2024-01-16T15:00:08.550736+0000","last_peered":"2024-01-16T15:00:08.550736+0000","last_clean":"2024-01-16T15:00:08.550736+0000","last_became_active":"2024-01-16T15:00:07.525987+0000","last_became_peered":"2024-01-16T15:00:07.525987+0000","last_unstale":"2024-01-16T15:00:08.550736+0000","last_undegraded":"2024-01-16T15:00:08.550736+0000","last_fullsized":"2024-01-16T15:00:08.550736+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:31.964473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554431+0000","last_change":"2024-01-16T15:00:08.550156+0000","last_active":"2024-01-16T15:00:08.554431+0000","last_peered":"2024-01-16T15:00:08.554431+0000","last_clean":"2024-01-16T15:00:08.554431+0000","last_became_active":"2024-01-16T15:00:08.549857+0000","last_became_peered":"2024-01-16T15:00:08.549857+0000","last_unstale":"2024-01-16T15:00:08.554431+0000","last_undegraded":"2024-01-16T15:00:08.554431+0000","last_fullsized":"2024-01-16T15:00:08.554431+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:32.978431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.540145+0000","last_change":"2024-01-16T15:00:07.516846+0000","last_active":"2024-01-16T15:00:08.540145+0000","last_peered":"2024-01-16T15:00:08.540145+0000","last_clean":"2024-01-16T15:00:08.540145+0000","last_became_active":"2024-01-16T15:00:07.516509+0000","last_became_peered":"2024-01-16T15:00:07.516509+0000","last_unstale":"2024-01-16T15:00:08.540145+0000","last_undegraded":"2024-01-16T15:00:08.540145+0000","last_fullsized":"2024-01-16T15:00:08.540145+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:11.646032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.556965+0000","last_change":"2024-01-16T15:00:08.547704+0000","last_active":"2024-01-16T15:00:08.556965+0000","last_peered":"2024-01-16T15:00:08.556965+0000","last_clean":"2024-01-16T15:00:08.556965+0000","last_became_active":"2024-01-16T15:00:08.547287+0000","last_became_peered":"2024-01-16T15:00:08.547287+0000","last_unstale":"2024-01-16T15:00:08.556965+0000","last_undegraded":"2024-01-16T15:00:08.556965+0000","last_fullsized":"2024-01-16T15:00:08.556965+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:50.283502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568862+0000","last_change":"2024-01-16T15:00:07.524923+0000","last_active":"2024-01-16T15:00:08.568862+0000","last_peered":"2024-01-16T15:00:08.568862+0000","last_clean":"2024-01-16T15:00:08.568862+0000","last_became_active":"2024-01-16T15:00:07.524609+0000","last_became_peered":"2024-01-16T15:00:07.524609+0000","last_unstale":"2024-01-16T15:00:08.568862+0000","last_undegraded":"2024-01-16T15:00:08.568862+0000","last_fullsized":"2024-01-16T15:00:08.568862+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:42.035002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":187,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T15:00:07.517425+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:00:07.517425+0000","last_peered":"2024-01-16T15:00:07.517425+0000","last_clean":"2024-01-16T15:00:07.517425+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:00:07.517425+0000","last_undegraded":"2024-01-16T15:00:07.517425+0000","last_fullsized":"2024-01-16T15:00:07.517425+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":801792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.105,"15min":1.073},"min":{"1min":0.746,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.236,"15min":1.1759999999999999},"min":{"1min":0.755,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.056,"5min":2.774,"15min":2.774},"last":1.3879999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.083,"15min":1.042},"min":{"1min":0.69199999999999995,"5min":0.63,"15min":0.63},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.724,"5min":1.272,"15min":1.167},"min":{"1min":0.64900000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":8.3360000000000003,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":0.95699999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.2290000000000001,"15min":1.177},"min":{"1min":1.0249999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.262,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.603,"5min":1.2629999999999999,"15min":1.2310000000000001},"min":{"1min":0.91600000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":6.1559999999999997,"5min":6.1559999999999997,"15min":6.1559999999999997},"last":0.89900000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.304,"15min":1.29},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.464,"5min":2.464,"15min":2.464},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.45,"15min":1.3979999999999999},"min":{"1min":1.117,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.5710000000000002,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.101}]},{"osd":4,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3560000000000001,"15min":1.3540000000000001},"min":{"1min":0.78900000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.431,"15min":1.4430000000000001},"min":{"1min":0.98499999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.6990000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.524,"15min":1.5289999999999999},"min":{"1min":0.93999999999999995,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.1219999999999999,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.645,"5min":1.595,"15min":1.627},"min":{"1min":1.214,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.601,"5min":3.3530000000000002,"15min":3.3530000000000002},"last":1.0149999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.5720000000000001,"15min":1.579},"min":{"1min":1.1020000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.7709999999999999,"5min":3.258,"15min":3.258},"last":1.26},{"interface":"front","average":{"1min":1.601,"5min":1.5649999999999999,"15min":1.599},"min":{"1min":1.1379999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.2749999999999999,"5min":3.101,"15min":3.101},"last":1.4370000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":212006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.099,"15min":1.093},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.554,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.2010000000000001,"15min":1.22},"min":{"1min":0.71199999999999997,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.0710000000000002,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":1.528}]},{"osd":1,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.0980000000000001,"15min":1.046},"min":{"1min":0.873,"5min":0.624,"15min":0.624},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.423,"5min":1.3939999999999999,"15min":1.3680000000000001},"min":{"1min":0.69299999999999995,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":4.5380000000000003,"5min":6.8029999999999999,"15min":6.8029999999999999},"last":1.4379999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.286,"15min":1.2},"min":{"1min":0.871,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.764,"5min":8.0449999999999999,"15min":8.0449999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.28,"15min":1.2869999999999999},"min":{"1min":0.98899999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.7150000000000001,"5min":1.95,"15min":1.95},"last":1.143}]},{"osd":3,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.331,"15min":1.343},"min":{"1min":0.72699999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.387,"15min":1.379},"min":{"1min":0.95399999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.0779999999999998,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.6519999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.458,"15min":1.486},"min":{"1min":0.97699999999999998,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.9650000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.24},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4930000000000001,"15min":1.5149999999999999},"min":{"1min":0.94299999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.2679999999999998,"5min":2.331,"15min":2.331},"last":1.776}]},{"osd":5,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.452,"15min":1.47},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.623,"15min":1.6579999999999999},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":2.206,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.5740000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.6359999999999999,"15min":1.663},"min":{"1min":1.2470000000000001,"5min":1.232,"15min":1.232},"max":{"1min":2.3490000000000002,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.623,"15min":1.661},"min":{"1min":1.048,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.488}]}]},{"osd":5,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":211404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1619999999999999,"15min":1.1819999999999999},"min":{"1min":0.70699999999999996,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.0249999999999999,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.163,"5min":1.0860000000000001,"15min":1.0820000000000001},"min":{"1min":0.67900000000000005,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.5489999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.22,"15min":1.266},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5980000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.2969999999999999,"15min":1.1919999999999999},"min":{"1min":0.80800000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":11.345000000000001,"5min":11.345000000000001,"15min":11.345000000000001},"last":1.367}]},{"osd":2,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":1.397,"15min":1.3069999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":11.445,"5min":11.445,"15min":11.445},"last":1.161},{"interface":"front","average":{"1min":1.339,"5min":1.2549999999999999,"15min":1.264},"min":{"1min":0.83999999999999997,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":1.452}]},{"osd":3,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.337},"min":{"1min":0.96599999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.1339999999999999,"5min":2.323,"15min":2.323},"last":1.4099999999999999},{"interface":"front","average":{"1min":2.1320000000000001,"5min":1.5289999999999999,"15min":1.4410000000000001},"min":{"1min":1.125,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":11.396000000000001,"5min":11.396000000000001,"15min":11.396000000000001},"last":1.5960000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.367,"15min":1.4399999999999999},"min":{"1min":0.63500000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.476,"15min":1.538},"min":{"1min":0.69799999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.8279999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.0449999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":1.645,"15min":1.6180000000000001},"min":{"1min":1.0189999999999999,"5min":0.499,"15min":0.499},"max":{"1min":11.512,"5min":11.512,"15min":11.512},"last":1.266},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5860000000000001,"15min":1.6539999999999999},"min":{"1min":0.83299999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0510000000000002,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.581,"15min":1.617},"min":{"1min":0.65700000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":1.627},{"interface":"front","average":{"1min":2.1640000000000001,"5min":1.768,"15min":1.746},"min":{"1min":1.123,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":11.476000000000001,"5min":11.476000000000001,"15min":11.476000000000001},"last":1.508}]}]},{"osd":4,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":212946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1220000000000001,"15min":1.089},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":1.109},{"interface":"front","average":{"1min":1.577,"5min":1.2869999999999999,"15min":1.24},"min":{"1min":0.69699999999999995,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":6.1790000000000003,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.0620000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.266,"15min":1.202},"min":{"1min":0.73499999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":6.2610000000000001,"5min":6.2610000000000001,"15min":6.2610000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":3.6659999999999999,"5min":3.6659999999999999,"15min":3.6659999999999999},"last":1.216}]},{"osd":2,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.4430000000000001,"15min":1.3680000000000001},"min":{"1min":0.72799999999999998,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.742,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.393,"15min":1.4470000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.4319999999999999,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":0.95799999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.5609999999999999,"15min":1.579},"min":{"1min":0.79300000000000004,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":6.4219999999999997,"5min":6.4219999999999997,"15min":6.4219999999999997},"last":1.48},{"interface":"front","average":{"1min":2.4409999999999998,"5min":1.5940000000000001,"15min":1.4910000000000001},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":21.706,"5min":21.706,"15min":21.706},"last":1.012}]},{"osd":5,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.5249999999999999,"15min":1.605},"min":{"1min":0.75700000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.2149999999999999,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.607,"15min":1.6140000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":6.3129999999999997,"5min":6.3129999999999997,"15min":6.3129999999999997},"last":1.4079999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.5089999999999999,"15min":1.5940000000000001},"min":{"1min":0.81299999999999994,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.6459999999999999,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.53,"5min":1.6299999999999999,"15min":1.6970000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.145,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":1.3540000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.883,"5min":1.718,"15min":1.722},"min":{"1min":0.94999999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":6.3659999999999997,"5min":6.3659999999999997,"15min":6.3659999999999997},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.6559999999999999,"15min":1.744},"min":{"1min":1.0149999999999999,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.8359999999999999,"5min":3.2290000000000001,"15min":3.2290000000000001},"last":1.6479999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":211972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":0.93899999999999995,"15min":0.90300000000000002},"min":{"1min":0.53800000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.2889999999999999,"5min":0.95999999999999996,"15min":0.874},"min":{"1min":0.56100000000000005,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.90500000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85599999999999998,"15min":0.83699999999999997},"min":{"1min":0.47099999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.232,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.02,"15min":1.002},"min":{"1min":0.53700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":2.819,"5min":2.819,"15min":2.819},"last":1.3480000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.056,"15min":1.0680000000000001},"min":{"1min":0.439,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.706,"5min":4.008,"15min":4.008},"last":1.49},{"interface":"front","average":{"1min":1.423,"5min":1.1319999999999999,"15min":1.056},"min":{"1min":0.58199999999999996,"5min":0.497,"15min":0.497},"max":{"1min":3.4199999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.163}]},{"osd":4,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.131,"15min":1.091},"min":{"1min":0.83099999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":4.093,"5min":4.093,"15min":4.093},"last":1.383},{"interface":"front","average":{"1min":1.302,"5min":1.236,"15min":1.228},"min":{"1min":0.83799999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8959999999999999,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.319}]},{"osd":5,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.27,"15min":1.254},"min":{"1min":0.873,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.857,"5min":2.149,"15min":2.149},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.385,"15min":1.3720000000000001},"min":{"1min":0.75600000000000001,"5min":0.747,"15min":0.747},"max":{"1min":3.4700000000000002,"5min":3.4700000000000002,"15min":3.4700000000000002},"last":1.2949999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.2929999999999999,"15min":1.2689999999999999},"min":{"1min":0.80000000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.383,"15min":1.3540000000000001},"min":{"1min":0.89800000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.9470000000000001,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.4319999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.427,"15min":1.403},"min":{"1min":0.90100000000000002,"5min":0.755,"15min":0.755},"max":{"1min":3.3700000000000001,"5min":3.3700000000000001,"15min":3.3700000000000001},"last":1.532},{"interface":"front","average":{"1min":1.607,"5min":1.526,"15min":1.3799999999999999},"min":{"1min":0.82099999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":4.4050000000000002,"5min":18.846,"15min":18.846},"last":1.647}]}]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":212968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.91300000000000003,"15min":0.89900000000000002},"min":{"1min":0.625,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.746},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.97099999999999997,"15min":0.92500000000000004},"min":{"1min":0.59599999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.5010000000000003,"5min":4.5010000000000003,"15min":4.5010000000000003},"last":0.65100000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.0329999999999999,"15min":1.0229999999999999},"min":{"1min":0.74299999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.0249999999999999,"15min":0.98099999999999998},"min":{"1min":0.51100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":4.8250000000000002,"5min":4.8250000000000002,"15min":4.8250000000000002},"last":0.85199999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":0.98899999999999999,"15min":0.95599999999999996},"min":{"1min":0.51200000000000001,"5min":0.436,"15min":0.436},"max":{"1min":5.0540000000000003,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1220000000000001,"15min":1.1060000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":3.4340000000000002,"5min":3.4340000000000002,"15min":3.4340000000000002},"last":0.97099999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.3759999999999999,"15min":1.3360000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.875},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.389,"15min":1.361},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":5.0860000000000003,"5min":5.0860000000000003,"15min":5.0860000000000003},"last":1.0580000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4239999999999999,"15min":1.3480000000000001},"min":{"1min":0.85899999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":5.3570000000000002,"5min":8.9689999999999994,"15min":8.9689999999999994},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3340000000000001,"15min":1.323},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":4.391,"5min":4.391,"15min":4.391},"last":0.98999999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.359,"15min":1.365},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.5310000000000001,"5min":2.5310000000000001,"15min":2.5310000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.401,"15min":1.389},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":3.4790000000000001,"5min":3.4790000000000001,"15min":3.4790000000000001},"last":1.274}]},{"osd":7,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.518,"15min":1.484},"min":{"1min":0.90500000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.472,"15min":1.458},"min":{"1min":0.99099999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"last":1.337}]}]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":211404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.89400000000000002,"15min":0.91800000000000004},"min":{"1min":0.42899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0289999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.016},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.99199999999999999,"15min":1.004},"min":{"1min":0.56100000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.4239999999999999,"5min":2.073,"15min":2.073},"last":0.73999999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91900000000000004,"15min":0.91600000000000004},"min":{"1min":0.45700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.456,"5min":2.952,"15min":2.952},"last":1.224},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.95499999999999996,"15min":0.95899999999999996},"min":{"1min":0.52500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.502,"5min":2.129,"15min":2.129},"last":0.90200000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.05,"15min":1.0629999999999999},"min":{"1min":0.54000000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.3160000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.157},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.018,"15min":1.008},"min":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6970000000000001,"5min":2.641,"15min":2.641},"last":0.95999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2,"15min":1.177},"min":{"1min":0.89700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.548,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.2310000000000001,"15min":1.2470000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.498,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.476}]},{"osd":5,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3480000000000001,"15min":1.3859999999999999},"min":{"1min":0.81499999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.621,"5min":2.649,"15min":2.649},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.403,"15min":1.423},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.6739999999999999,"5min":2.766,"15min":2.766},"last":1.3700000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.4019999999999999,"15min":1.4259999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.9059999999999999,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.097},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.3779999999999999,"15min":1.4079999999999999},"min":{"1min":0.748,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.6819999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.3160000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.4079999999999999,"15min":1.4039999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":3.0779999999999998,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":1.284},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.4059999999999999,"15min":1.427},"min":{"1min":0.68400000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.8160000000000001,"5min":2.879,"15min":2.879},"last":1.2629999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":801736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97899999999999998,"15min":0.97499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.3069999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":3.6469999999999998},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.004,"15min":0.98399999999999999},"min":{"1min":0.55300000000000005,"5min":0.371,"15min":0.371},"max":{"1min":1.321,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":3.0990000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1259999999999999,"15min":1.107},"min":{"1min":0.56999999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.524,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":3.2200000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.26,"15min":1.244},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":2.395,"5min":5.5970000000000004,"15min":5.5970000000000004},"last":5.0380000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.93100000000000005,"15min":0.88700000000000001},"min":{"1min":0.628,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.597,"5min":3.597,"15min":3.597},"last":4.8899999999999997},{"interface":"front","average":{"1min":1.052,"5min":1.0649999999999999,"15min":1.0409999999999999},"min":{"1min":0.51000000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.478,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":6.1079999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.5309999999999999,"15min":1.4630000000000001},"min":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.532,"5min":14.297000000000001,"15min":14.297000000000001},"last":3.7519999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.298,"15min":1.284},"min":{"1min":0.88200000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.4489999999999998,"5min":4.5170000000000003,"15min":4.5170000000000003},"last":4.1159999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.5680000000000001,"15min":1.472},"min":{"1min":1.0529999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.4889999999999999,"5min":14.212999999999999,"15min":14.212999999999999},"last":3.9729999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3069999999999999,"15min":1.2909999999999999},"min":{"1min":0.85199999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":3.9350000000000001,"5min":3.9350000000000001,"15min":3.9350000000000001},"last":4.8399999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.417,"15min":1.4039999999999999},"min":{"1min":0.97799999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.925,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":3.1520000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6100000000000001,"15min":1.573},"min":{"1min":0.96899999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.593,"5min":5.7759999999999998,"15min":5.7759999999999998},"last":3.79}]},{"osd":7,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.552,"15min":1.454},"min":{"1min":0.88700000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":15.945,"5min":15.945,"15min":15.945},"last":3.1869999999999998},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.647,"15min":1.6140000000000001},"min":{"1min":1.1080000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.0590000000000002,"5min":5.9269999999999996,"15min":5.9269999999999996},"last":4.0129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:12.955 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:13.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.213+0000 7fa398200640 1 Processor -- start 2024-01-16T15:00:13.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.213+0000 7fa398200640 1 -- start start 2024-01-16T15:00:13.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.214+0000 7fa398200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa390143ae0 0x7fa3900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.214+0000 7fa398200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa39015e3b0 0x7fa3900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa398200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3900cfc30 con 0x7fa3900cf7d0 2024-01-16T15:00:13.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa398200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3900b1ee0 con 0x7fa39015e3b0 2024-01-16T15:00:13.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa398200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3900b2780 con 0x7fa390143ae0 2024-01-16T15:00:13.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa394e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa3900cf7d0 0x7fa39015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58148/0 (socket says 172.21.15.141:58148) 2024-01-16T15:00:13.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa394e00640 1 -- 172.21.15.141:0/2169498064 learned_addr learned my addr 172.21.15.141:0/2169498064 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:13.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa396200640 1 --2- 172.21.15.141:0/2169498064 >> v2:172.21.15.141:3300/0 conn(0x7fa39015e3b0 0x7fa3900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa395800640 1 --2- 172.21.15.141:0/2169498064 >> v2:172.21.15.188:3300/0 conn(0x7fa390143ae0 0x7fa3900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa38fe00640 1 -- 172.21.15.141:0/2169498064 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315825229 0 0) 0x7fa3900cfc30 con 0x7fa3900cf7d0 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa38fe00640 1 -- 172.21.15.141:0/2169498064 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa370005730 con 0x7fa3900cf7d0 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa396200640 1 -- 172.21.15.141:0/2169498064 >> v2:172.21.15.188:3300/0 conn(0x7fa390143ae0 msgr2=0x7fa3900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa396200640 1 --2- 172.21.15.141:0/2169498064 >> v2:172.21.15.188:3300/0 conn(0x7fa390143ae0 0x7fa3900cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa396200640 1 -- 172.21.15.141:0/2169498064 >> v1:172.21.15.141:6789/0 conn(0x7fa3900cf7d0 legacy=0x7fa39015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa396200640 1 -- 172.21.15.141:0/2169498064 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39015e990 con 0x7fa39015e3b0 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.215+0000 7fa395800640 1 --2- 172.21.15.141:0/2169498064 >> v2:172.21.15.188:3300/0 conn(0x7fa390143ae0 0x7fa3900cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:13.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa396200640 1 --2- 172.21.15.141:0/2169498064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa3900702c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa3900cfea0 tx=0x7fa38802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acf991c5b5e31ab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa38fe00640 1 -- 172.21.15.141:0/2169498064 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388031830 con 0x7fa39015e3b0 2024-01-16T15:00:13.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa38fe00640 1 -- 172.21.15.141:0/2169498064 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa388031990 con 0x7fa39015e3b0 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa38fe00640 1 -- 172.21.15.141:0/2169498064 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388031d20 con 0x7fa39015e3b0 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa398200640 1 -- 172.21.15.141:0/2169498064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 msgr2=0x7fa3900702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa398200640 1 --2- 172.21.15.141:0/2169498064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa3900702c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa3900cfea0 tx=0x7fa38802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa398200640 1 -- 172.21.15.141:0/2169498064 shutdown_connections 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa398200640 1 --2- 172.21.15.141:0/2169498064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa3900702c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa398200640 1 --2- 172.21.15.141:0/2169498064 >> v2:172.21.15.188:3300/0 conn(0x7fa390143ae0 0x7fa3900cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.217+0000 7fa398200640 1 -- 172.21.15.141:0/2169498064 >> 172.21.15.141:0/2169498064 conn(0x7fa39006a800 msgr2=0x7fa39006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.218+0000 7fa398200640 1 -- 172.21.15.141:0/2169498064 shutdown_connections 2024-01-16T15:00:13.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.218+0000 7fa398200640 1 -- 172.21.15.141:0/2169498064 wait complete. 2024-01-16T15:00:13.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.219+0000 7fa398200640 1 Processor -- start 2024-01-16T15:00:13.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.219+0000 7fa398200640 1 -- start start 2024-01-16T15:00:13.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.220+0000 7fa398200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3900cf7d0 0x7fa3900ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.220+0000 7fa398200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa390143ae0 0x7fa3901ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.220+0000 7fa398200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa39013ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.220+0000 7fa398200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa390140050 con 0x7fa39015e3b0 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.220+0000 7fa398200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3900a8660 con 0x7fa390143ae0 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.220+0000 7fa398200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3900acbc0 con 0x7fa3900cf7d0 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa39013ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa39013ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48968/0 (socket says 172.21.15.141:48968) 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 -- 172.21.15.141:0/2970839296 learned_addr learned my addr 172.21.15.141:0/2970839296 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa395800640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3900cf7d0 0x7fa3900ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 -- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3900cf7d0 msgr2=0x7fa3900ac680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3900cf7d0 0x7fa3900ac680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 -- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa390143ae0 msgr2=0x7fa3901ee350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa390143ae0 0x7fa3901ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 -- 172.21.15.141:0/2970839296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa388009400 con 0x7fa39015e3b0 2024-01-16T15:00:13.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.221+0000 7fa396200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa39013ee40 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa388034040 tx=0x7fa3880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.222+0000 7fa38d600640 1 -- 172.21.15.141:0/2970839296 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38803a830 con 0x7fa39015e3b0 2024-01-16T15:00:13.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.222+0000 7fa38d600640 1 -- 172.21.15.141:0/2970839296 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38803a9b0 con 0x7fa39015e3b0 2024-01-16T15:00:13.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.222+0000 7fa38d600640 1 -- 172.21.15.141:0/2970839296 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38803ad60 con 0x7fa39015e3b0 2024-01-16T15:00:13.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.222+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3901ee890 con 0x7fa39015e3b0 2024-01-16T15:00:13.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.222+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39013fa90 con 0x7fa39015e3b0 2024-01-16T15:00:13.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.223+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3900a84c0 con 0x7fa39015e3b0 2024-01-16T15:00:13.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.224+0000 7fa38d600640 1 -- 172.21.15.141:0/2970839296 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa3880443f0 con 0x7fa39015e3b0 2024-01-16T15:00:13.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.225+0000 7fa38d600640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa36806b500 0x7fa36806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.226+0000 7fa38d600640 1 -- 172.21.15.141:0/2970839296 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3880ac2c0 con 0x7fa39015e3b0 2024-01-16T15:00:13.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.226+0000 7fa395800640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa36806b500 0x7fa36806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.227+0000 7fa395800640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa36806b500 0x7fa36806d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa380008f80 tx=0x7fa3800084d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.230+0000 7fa38d600640 1 -- 172.21.15.141:0/2970839296 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3900a84c0 con 0x7fa39015e3b0 2024-01-16T15:00:13.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.416+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3900b2410 con 0x7fa36806b500 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.441+0000 7fa38d600640 1 -- 172.21.15.141:0/2970839296 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286650 (secure 0 0 0) 0x7fa380008b90 con 0x7fa36806b500 2024-01-16T15:00:13.427 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:13.428 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:00:13.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa36806b500 msgr2=0x7fa36806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa36806b500 0x7fa36806d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa380008f80 tx=0x7fa3800084d0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 msgr2=0x7fa39013ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa39013ee40 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa388034040 tx=0x7fa3880046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 shutdown_connections 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa36806b500 0x7fa36806d9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa39015e3b0 0x7fa39013ee40 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa390143ae0 0x7fa3901ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.446+0000 7fa398200640 1 --2- 172.21.15.141:0/2970839296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3900cf7d0 0x7fa3900ac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.447+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 >> 172.21.15.141:0/2970839296 conn(0x7fa39006a800 msgr2=0x7fa39015a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.447+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 shutdown_connections 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.447+0000 7fa398200640 1 -- 172.21.15.141:0/2970839296 wait complete. 2024-01-16T15:00:13.447 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-01-16T15:00:12.502899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254784,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719936,"statfs":{"total":773094113280,"available":772833214464,"internally_reserved":0,"allocated":5447680,"data_stored":2876228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002279"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.543229+0000","last_change":"2024-01-16T15:00:08.536867+0000","last_active":"2024-01-16T15:00:08.543229+0000","last_peered":"2024-01-16T15:00:08.543229+0000","last_clean":"2024-01-16T15:00:08.543229+0000","last_became_active":"2024-01-16T15:00:08.536584+0000","last_became_peered":"2024-01-16T15:00:08.536584+0000","last_unstale":"2024-01-16T15:00:08.543229+0000","last_undegraded":"2024-01-16T15:00:08.543229+0000","last_fullsized":"2024-01-16T15:00:08.543229+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:40.608354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563267+0000","last_change":"2024-01-16T15:00:08.555740+0000","last_active":"2024-01-16T15:00:08.563267+0000","last_peered":"2024-01-16T15:00:08.563267+0000","last_clean":"2024-01-16T15:00:08.563267+0000","last_became_active":"2024-01-16T15:00:08.555299+0000","last_became_peered":"2024-01-16T15:00:08.555299+0000","last_unstale":"2024-01-16T15:00:08.563267+0000","last_undegraded":"2024-01-16T15:00:08.563267+0000","last_fullsized":"2024-01-16T15:00:08.563267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:32.978431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547013+0000","last_change":"2024-01-16T15:00:08.544321+0000","last_active":"2024-01-16T15:00:08.547013+0000","last_peered":"2024-01-16T15:00:08.547013+0000","last_clean":"2024-01-16T15:00:08.547013+0000","last_became_active":"2024-01-16T15:00:08.544004+0000","last_became_peered":"2024-01-16T15:00:08.544004+0000","last_unstale":"2024-01-16T15:00:08.547013+0000","last_undegraded":"2024-01-16T15:00:08.547013+0000","last_fullsized":"2024-01-16T15:00:08.547013+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:12.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.557359+0000","last_change":"2024-01-16T15:00:08.541565+0000","last_active":"2024-01-16T15:00:08.557359+0000","last_peered":"2024-01-16T15:00:08.557359+0000","last_clean":"2024-01-16T15:00:08.557359+0000","last_became_active":"2024-01-16T15:00:08.541299+0000","last_became_peered":"2024-01-16T15:00:08.541299+0000","last_unstale":"2024-01-16T15:00:08.557359+0000","last_undegraded":"2024-01-16T15:00:08.557359+0000","last_fullsized":"2024-01-16T15:00:08.557359+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:00.980436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563180+0000","last_change":"2024-01-16T15:00:08.558164+0000","last_active":"2024-01-16T15:00:08.563180+0000","last_peered":"2024-01-16T15:00:08.563180+0000","last_clean":"2024-01-16T15:00:08.563180+0000","last_became_active":"2024-01-16T15:00:08.557839+0000","last_became_peered":"2024-01-16T15:00:08.557839+0000","last_unstale":"2024-01-16T15:00:08.563180+0000","last_undegraded":"2024-01-16T15:00:08.563180+0000","last_fullsized":"2024-01-16T15:00:08.563180+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:47.964823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.551947+0000","last_change":"2024-01-16T15:00:08.547938+0000","last_active":"2024-01-16T15:00:08.551947+0000","last_peered":"2024-01-16T15:00:08.551947+0000","last_clean":"2024-01-16T15:00:08.551947+0000","last_became_active":"2024-01-16T15:00:08.547625+0000","last_became_peered":"2024-01-16T15:00:08.547625+0000","last_unstale":"2024-01-16T15:00:08.551947+0000","last_undegraded":"2024-01-16T15:00:08.551947+0000","last_fullsized":"2024-01-16T15:00:08.551947+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:14.919862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.560956+0000","last_change":"2024-01-16T15:00:08.555571+0000","last_active":"2024-01-16T15:00:08.560956+0000","last_peered":"2024-01-16T15:00:08.560956+0000","last_clean":"2024-01-16T15:00:08.560956+0000","last_became_active":"2024-01-16T15:00:08.554593+0000","last_became_peered":"2024-01-16T15:00:08.554593+0000","last_unstale":"2024-01-16T15:00:08.560956+0000","last_undegraded":"2024-01-16T15:00:08.560956+0000","last_fullsized":"2024-01-16T15:00:08.560956+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:47.964823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568817+0000","last_change":"2024-01-16T15:00:08.559549+0000","last_active":"2024-01-16T15:00:08.568817+0000","last_peered":"2024-01-16T15:00:08.568817+0000","last_clean":"2024-01-16T15:00:08.568817+0000","last_became_active":"2024-01-16T15:00:08.557229+0000","last_became_peered":"2024-01-16T15:00:08.557229+0000","last_unstale":"2024-01-16T15:00:08.568817+0000","last_undegraded":"2024-01-16T15:00:08.568817+0000","last_fullsized":"2024-01-16T15:00:08.568817+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:11.355514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571087+0000","last_change":"2024-01-16T15:00:08.567557+0000","last_active":"2024-01-16T15:00:08.571087+0000","last_peered":"2024-01-16T15:00:08.571087+0000","last_clean":"2024-01-16T15:00:08.571087+0000","last_became_active":"2024-01-16T15:00:08.567392+0000","last_became_peered":"2024-01-16T15:00:08.567392+0000","last_unstale":"2024-01-16T15:00:08.571087+0000","last_undegraded":"2024-01-16T15:00:08.571087+0000","last_fullsized":"2024-01-16T15:00:08.571087+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:30.390941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568935+0000","last_change":"2024-01-16T15:00:08.556952+0000","last_active":"2024-01-16T15:00:08.568935+0000","last_peered":"2024-01-16T15:00:08.568935+0000","last_clean":"2024-01-16T15:00:08.568935+0000","last_became_active":"2024-01-16T15:00:08.556684+0000","last_became_peered":"2024-01-16T15:00:08.556684+0000","last_unstale":"2024-01-16T15:00:08.568935+0000","last_undegraded":"2024-01-16T15:00:08.568935+0000","last_fullsized":"2024-01-16T15:00:08.568935+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:26.068944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571091+0000","last_change":"2024-01-16T15:00:08.569546+0000","last_active":"2024-01-16T15:00:08.571091+0000","last_peered":"2024-01-16T15:00:08.571091+0000","last_clean":"2024-01-16T15:00:08.571091+0000","last_became_active":"2024-01-16T15:00:08.569124+0000","last_became_peered":"2024-01-16T15:00:08.569124+0000","last_unstale":"2024-01-16T15:00:08.571091+0000","last_undegraded":"2024-01-16T15:00:08.571091+0000","last_fullsized":"2024-01-16T15:00:08.571091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:15.756996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.546825+0000","last_change":"2024-01-16T15:00:08.544704+0000","last_active":"2024-01-16T15:00:08.546825+0000","last_peered":"2024-01-16T15:00:08.546825+0000","last_clean":"2024-01-16T15:00:08.546825+0000","last_became_active":"2024-01-16T15:00:08.543395+0000","last_became_peered":"2024-01-16T15:00:08.543395+0000","last_unstale":"2024-01-16T15:00:08.546825+0000","last_undegraded":"2024-01-16T15:00:08.546825+0000","last_fullsized":"2024-01-16T15:00:08.546825+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:56.333632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571053+0000","last_change":"2024-01-16T15:00:08.564033+0000","last_active":"2024-01-16T15:00:08.571053+0000","last_peered":"2024-01-16T15:00:08.571053+0000","last_clean":"2024-01-16T15:00:08.571053+0000","last_became_active":"2024-01-16T15:00:08.563588+0000","last_became_peered":"2024-01-16T15:00:08.563588+0000","last_unstale":"2024-01-16T15:00:08.571053+0000","last_undegraded":"2024-01-16T15:00:08.571053+0000","last_fullsized":"2024-01-16T15:00:08.571053+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:51.709344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.570692+0000","last_change":"2024-01-16T15:00:08.567255+0000","last_active":"2024-01-16T15:00:08.570692+0000","last_peered":"2024-01-16T15:00:08.570692+0000","last_clean":"2024-01-16T15:00:08.570692+0000","last_became_active":"2024-01-16T15:00:08.566997+0000","last_became_peered":"2024-01-16T15:00:08.566997+0000","last_unstale":"2024-01-16T15:00:08.570692+0000","last_undegraded":"2024-01-16T15:00:08.570692+0000","last_fullsized":"2024-01-16T15:00:08.570692+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:21.748083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.553205+0000","last_change":"2024-01-16T15:00:08.549815+0000","last_active":"2024-01-16T15:00:08.553205+0000","last_peered":"2024-01-16T15:00:08.553205+0000","last_clean":"2024-01-16T15:00:08.553205+0000","last_became_active":"2024-01-16T15:00:08.548946+0000","last_became_peered":"2024-01-16T15:00:08.548946+0000","last_unstale":"2024-01-16T15:00:08.553205+0000","last_undegraded":"2024-01-16T15:00:08.553205+0000","last_fullsized":"2024-01-16T15:00:08.553205+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:50.725282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563564+0000","last_change":"2024-01-16T15:00:08.559961+0000","last_active":"2024-01-16T15:00:08.563564+0000","last_peered":"2024-01-16T15:00:08.563564+0000","last_clean":"2024-01-16T15:00:08.563564+0000","last_became_active":"2024-01-16T15:00:08.559686+0000","last_became_peered":"2024-01-16T15:00:08.559686+0000","last_unstale":"2024-01-16T15:00:08.563564+0000","last_undegraded":"2024-01-16T15:00:08.563564+0000","last_fullsized":"2024-01-16T15:00:08.563564+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:55:05.241468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.545403+0000","last_change":"2024-01-16T15:00:08.536701+0000","last_active":"2024-01-16T15:00:08.545403+0000","last_peered":"2024-01-16T15:00:08.545403+0000","last_clean":"2024-01-16T15:00:08.545403+0000","last_became_active":"2024-01-16T15:00:08.536370+0000","last_became_peered":"2024-01-16T15:00:08.536370+0000","last_unstale":"2024-01-16T15:00:08.545403+0000","last_undegraded":"2024-01-16T15:00:08.545403+0000","last_fullsized":"2024-01-16T15:00:08.545403+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:20.095673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563431+0000","last_change":"2024-01-16T15:00:08.558699+0000","last_active":"2024-01-16T15:00:08.563431+0000","last_peered":"2024-01-16T15:00:08.563431+0000","last_clean":"2024-01-16T15:00:08.563431+0000","last_became_active":"2024-01-16T15:00:08.558421+0000","last_became_peered":"2024-01-16T15:00:08.558421+0000","last_unstale":"2024-01-16T15:00:08.563431+0000","last_undegraded":"2024-01-16T15:00:08.563431+0000","last_fullsized":"2024-01-16T15:00:08.563431+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:25.495238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563498+0000","last_change":"2024-01-16T15:00:08.559361+0000","last_active":"2024-01-16T15:00:08.563498+0000","last_peered":"2024-01-16T15:00:08.563498+0000","last_clean":"2024-01-16T15:00:08.563498+0000","last_became_active":"2024-01-16T15:00:08.559057+0000","last_became_peered":"2024-01-16T15:00:08.559057+0000","last_unstale":"2024-01-16T15:00:08.563498+0000","last_undegraded":"2024-01-16T15:00:08.563498+0000","last_fullsized":"2024-01-16T15:00:08.563498+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:57.357255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569657+0000","last_change":"2024-01-16T15:00:08.558429+0000","last_active":"2024-01-16T15:00:08.569657+0000","last_peered":"2024-01-16T15:00:08.569657+0000","last_clean":"2024-01-16T15:00:08.569657+0000","last_became_active":"2024-01-16T15:00:08.558160+0000","last_became_peered":"2024-01-16T15:00:08.558160+0000","last_unstale":"2024-01-16T15:00:08.569657+0000","last_undegraded":"2024-01-16T15:00:08.569657+0000","last_fullsized":"2024-01-16T15:00:08.569657+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:40.521526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554510+0000","last_change":"2024-01-16T15:00:08.548394+0000","last_active":"2024-01-16T15:00:08.554510+0000","last_peered":"2024-01-16T15:00:08.554510+0000","last_clean":"2024-01-16T15:00:08.554510+0000","last_became_active":"2024-01-16T15:00:08.548075+0000","last_became_peered":"2024-01-16T15:00:08.548075+0000","last_unstale":"2024-01-16T15:00:08.554510+0000","last_undegraded":"2024-01-16T15:00:08.554510+0000","last_fullsized":"2024-01-16T15:00:08.554510+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:22.075114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.557388+0000","last_change":"2024-01-16T15:00:08.545364+0000","last_active":"2024-01-16T15:00:08.557388+0000","last_peered":"2024-01-16T15:00:08.557388+0000","last_clean":"2024-01-16T15:00:08.557388+0000","last_became_active":"2024-01-16T15:00:08.545096+0000","last_became_peered":"2024-01-16T15:00:08.545096+0000","last_unstale":"2024-01-16T15:00:08.557388+0000","last_undegraded":"2024-01-16T15:00:08.557388+0000","last_fullsized":"2024-01-16T15:00:08.557388+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:35.701093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552607+0000","last_change":"2024-01-16T15:00:08.546916+0000","last_active":"2024-01-16T15:00:08.552607+0000","last_peered":"2024-01-16T15:00:08.552607+0000","last_clean":"2024-01-16T15:00:08.552607+0000","last_became_active":"2024-01-16T15:00:08.546663+0000","last_became_peered":"2024-01-16T15:00:08.546663+0000","last_unstale":"2024-01-16T15:00:08.552607+0000","last_undegraded":"2024-01-16T15:00:08.552607+0000","last_fullsized":"2024-01-16T15:00:08.552607+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:07:21.928549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547360+0000","last_change":"2024-01-16T15:00:08.544924+0000","last_active":"2024-01-16T15:00:08.547360+0000","last_peered":"2024-01-16T15:00:08.547360+0000","last_clean":"2024-01-16T15:00:08.547360+0000","last_became_active":"2024-01-16T15:00:08.544446+0000","last_became_peered":"2024-01-16T15:00:08.544446+0000","last_unstale":"2024-01-16T15:00:08.547360+0000","last_undegraded":"2024-01-16T15:00:08.547360+0000","last_fullsized":"2024-01-16T15:00:08.547360+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:14.742590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566162+0000","last_change":"2024-01-16T15:00:08.561486+0000","last_active":"2024-01-16T15:00:08.566162+0000","last_peered":"2024-01-16T15:00:08.566162+0000","last_clean":"2024-01-16T15:00:08.566162+0000","last_became_active":"2024-01-16T15:00:08.561192+0000","last_became_peered":"2024-01-16T15:00:08.561192+0000","last_unstale":"2024-01-16T15:00:08.566162+0000","last_undegraded":"2024-01-16T15:00:08.566162+0000","last_fullsized":"2024-01-16T15:00:08.566162+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:10.767361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571003+0000","last_change":"2024-01-16T15:00:08.562984+0000","last_active":"2024-01-16T15:00:08.571003+0000","last_peered":"2024-01-16T15:00:08.571003+0000","last_clean":"2024-01-16T15:00:08.571003+0000","last_became_active":"2024-01-16T15:00:08.562708+0000","last_became_peered":"2024-01-16T15:00:08.562708+0000","last_unstale":"2024-01-16T15:00:08.571003+0000","last_undegraded":"2024-01-16T15:00:08.571003+0000","last_fullsized":"2024-01-16T15:00:08.571003+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:20:03.366522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.545478+0000","last_change":"2024-01-16T15:00:08.537951+0000","last_active":"2024-01-16T15:00:08.545478+0000","last_peered":"2024-01-16T15:00:08.545478+0000","last_clean":"2024-01-16T15:00:08.545478+0000","last_became_active":"2024-01-16T15:00:08.537603+0000","last_became_peered":"2024-01-16T15:00:08.537603+0000","last_unstale":"2024-01-16T15:00:08.545478+0000","last_undegraded":"2024-01-16T15:00:08.545478+0000","last_fullsized":"2024-01-16T15:00:08.545478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:51.295894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.544332+0000","last_change":"2024-01-16T15:00:08.539930+0000","last_active":"2024-01-16T15:00:08.544332+0000","last_peered":"2024-01-16T15:00:08.544332+0000","last_clean":"2024-01-16T15:00:08.544332+0000","last_became_active":"2024-01-16T15:00:08.539657+0000","last_became_peered":"2024-01-16T15:00:08.539657+0000","last_unstale":"2024-01-16T15:00:08.544332+0000","last_undegraded":"2024-01-16T15:00:08.544332+0000","last_fullsized":"2024-01-16T15:00:08.544332+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:12.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566226+0000","last_change":"2024-01-16T15:00:08.562005+0000","last_active":"2024-01-16T15:00:08.566226+0000","last_peered":"2024-01-16T15:00:08.566226+0000","last_clean":"2024-01-16T15:00:08.566226+0000","last_became_active":"2024-01-16T15:00:08.561689+0000","last_became_peered":"2024-01-16T15:00:08.561689+0000","last_unstale":"2024-01-16T15:00:08.566226+0000","last_undegraded":"2024-01-16T15:00:08.566226+0000","last_fullsized":"2024-01-16T15:00:08.566226+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:44.207622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.546400+0000","last_change":"2024-01-16T15:00:08.539760+0000","last_active":"2024-01-16T15:00:08.546400+0000","last_peered":"2024-01-16T15:00:08.546400+0000","last_clean":"2024-01-16T15:00:08.546400+0000","last_became_active":"2024-01-16T15:00:08.538972+0000","last_became_peered":"2024-01-16T15:00:08.538972+0000","last_unstale":"2024-01-16T15:00:08.546400+0000","last_undegraded":"2024-01-16T15:00:08.546400+0000","last_fullsized":"2024-01-16T15:00:08.546400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:07:41.025992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.553186+0000","last_change":"2024-01-16T15:00:08.545311+0000","last_active":"2024-01-16T15:00:08.553186+0000","last_peered":"2024-01-16T15:00:08.553186+0000","last_clean":"2024-01-16T15:00:08.553186+0000","last_became_active":"2024-01-16T15:00:08.545033+0000","last_became_peered":"2024-01-16T15:00:08.545033+0000","last_unstale":"2024-01-16T15:00:08.553186+0000","last_undegraded":"2024-01-16T15:00:08.553186+0000","last_fullsized":"2024-01-16T15:00:08.553186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:53:25.113062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.562469+0000","last_change":"2024-01-16T15:00:08.552664+0000","last_active":"2024-01-16T15:00:08.562469+0000","last_peered":"2024-01-16T15:00:08.562469+0000","last_clean":"2024-01-16T15:00:08.562469+0000","last_became_active":"2024-01-16T15:00:08.552340+0000","last_became_peered":"2024-01-16T15:00:08.552340+0000","last_unstale":"2024-01-16T15:00:08.562469+0000","last_undegraded":"2024-01-16T15:00:08.562469+0000","last_fullsized":"2024-01-16T15:00:08.562469+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:18:16.018305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569478+0000","last_change":"2024-01-16T15:00:08.566380+0000","last_active":"2024-01-16T15:00:08.569478+0000","last_peered":"2024-01-16T15:00:08.569478+0000","last_clean":"2024-01-16T15:00:08.569478+0000","last_became_active":"2024-01-16T15:00:08.566093+0000","last_became_peered":"2024-01-16T15:00:08.566093+0000","last_unstale":"2024-01-16T15:00:08.569478+0000","last_undegraded":"2024-01-16T15:00:08.569478+0000","last_fullsized":"2024-01-16T15:00:08.569478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:51.295894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571412+0000","last_change":"2024-01-16T15:00:07.527871+0000","last_active":"2024-01-16T15:00:08.571412+0000","last_peered":"2024-01-16T15:00:08.571412+0000","last_clean":"2024-01-16T15:00:08.571412+0000","last_became_active":"2024-01-16T15:00:07.527410+0000","last_became_peered":"2024-01-16T15:00:07.527410+0000","last_unstale":"2024-01-16T15:00:08.571412+0000","last_undegraded":"2024-01-16T15:00:08.571412+0000","last_fullsized":"2024-01-16T15:00:08.571412+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:08.792642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552641+0000","last_change":"2024-01-16T15:00:08.547896+0000","last_active":"2024-01-16T15:00:08.552641+0000","last_peered":"2024-01-16T15:00:08.552641+0000","last_clean":"2024-01-16T15:00:08.552641+0000","last_became_active":"2024-01-16T15:00:08.547555+0000","last_became_peered":"2024-01-16T15:00:08.547555+0000","last_unstale":"2024-01-16T15:00:08.552641+0000","last_undegraded":"2024-01-16T15:00:08.552641+0000","last_fullsized":"2024-01-16T15:00:08.552641+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:37.924541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.555671+0000","last_change":"2024-01-16T15:00:07.525831+0000","last_active":"2024-01-16T15:00:08.555671+0000","last_peered":"2024-01-16T15:00:08.555671+0000","last_clean":"2024-01-16T15:00:08.555671+0000","last_became_active":"2024-01-16T15:00:07.524073+0000","last_became_peered":"2024-01-16T15:00:07.524073+0000","last_unstale":"2024-01-16T15:00:08.555671+0000","last_undegraded":"2024-01-16T15:00:08.555671+0000","last_fullsized":"2024-01-16T15:00:08.555671+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:01.203257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554345+0000","last_change":"2024-01-16T15:00:08.548698+0000","last_active":"2024-01-16T15:00:08.554345+0000","last_peered":"2024-01-16T15:00:08.554345+0000","last_clean":"2024-01-16T15:00:08.554345+0000","last_became_active":"2024-01-16T15:00:08.548300+0000","last_became_peered":"2024-01-16T15:00:08.548300+0000","last_unstale":"2024-01-16T15:00:08.554345+0000","last_undegraded":"2024-01-16T15:00:08.554345+0000","last_fullsized":"2024-01-16T15:00:08.554345+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:20.095673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.574328+0000","last_change":"2024-01-16T15:00:07.524362+0000","last_active":"2024-01-16T15:00:08.574328+0000","last_peered":"2024-01-16T15:00:08.574328+0000","last_clean":"2024-01-16T15:00:08.574328+0000","last_became_active":"2024-01-16T15:00:07.524070+0000","last_became_peered":"2024-01-16T15:00:07.524070+0000","last_unstale":"2024-01-16T15:00:08.574328+0000","last_undegraded":"2024-01-16T15:00:08.574328+0000","last_fullsized":"2024-01-16T15:00:08.574328+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:11.875064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569368+0000","last_change":"2024-01-16T15:00:08.560285+0000","last_active":"2024-01-16T15:00:08.569368+0000","last_peered":"2024-01-16T15:00:08.569368+0000","last_clean":"2024-01-16T15:00:08.569368+0000","last_became_active":"2024-01-16T15:00:08.555077+0000","last_became_peered":"2024-01-16T15:00:08.555077+0000","last_unstale":"2024-01-16T15:00:08.569368+0000","last_undegraded":"2024-01-16T15:00:08.569368+0000","last_fullsized":"2024-01-16T15:00:08.569368+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:23.521963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563928+0000","last_change":"2024-01-16T15:00:07.527507+0000","last_active":"2024-01-16T15:00:08.563928+0000","last_peered":"2024-01-16T15:00:08.563928+0000","last_clean":"2024-01-16T15:00:08.563928+0000","last_became_active":"2024-01-16T15:00:07.526378+0000","last_became_peered":"2024-01-16T15:00:07.526378+0000","last_unstale":"2024-01-16T15:00:08.563928+0000","last_undegraded":"2024-01-16T15:00:08.563928+0000","last_fullsized":"2024-01-16T15:00:08.563928+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:55.319675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547249+0000","last_change":"2024-01-16T15:00:08.541357+0000","last_active":"2024-01-16T15:00:08.547249+0000","last_peered":"2024-01-16T15:00:08.547249+0000","last_clean":"2024-01-16T15:00:08.547249+0000","last_became_active":"2024-01-16T15:00:08.541070+0000","last_became_peered":"2024-01-16T15:00:08.541070+0000","last_unstale":"2024-01-16T15:00:08.547249+0000","last_undegraded":"2024-01-16T15:00:08.547249+0000","last_fullsized":"2024-01-16T15:00:08.547249+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:12.438613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571320+0000","last_change":"2024-01-16T15:00:07.524641+0000","last_active":"2024-01-16T15:00:08.571320+0000","last_peered":"2024-01-16T15:00:08.571320+0000","last_clean":"2024-01-16T15:00:08.571320+0000","last_became_active":"2024-01-16T15:00:07.524317+0000","last_became_peered":"2024-01-16T15:00:07.524317+0000","last_unstale":"2024-01-16T15:00:08.571320+0000","last_undegraded":"2024-01-16T15:00:08.571320+0000","last_fullsized":"2024-01-16T15:00:08.571320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:06.500463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568904+0000","last_change":"2024-01-16T15:00:08.556634+0000","last_active":"2024-01-16T15:00:08.568904+0000","last_peered":"2024-01-16T15:00:08.568904+0000","last_clean":"2024-01-16T15:00:08.568904+0000","last_became_active":"2024-01-16T15:00:08.556291+0000","last_became_peered":"2024-01-16T15:00:08.556291+0000","last_unstale":"2024-01-16T15:00:08.568904+0000","last_undegraded":"2024-01-16T15:00:08.568904+0000","last_fullsized":"2024-01-16T15:00:08.568904+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:53.707528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.555517+0000","last_change":"2024-01-16T15:00:07.527016+0000","last_active":"2024-01-16T15:00:08.555517+0000","last_peered":"2024-01-16T15:00:08.555517+0000","last_clean":"2024-01-16T15:00:08.555517+0000","last_became_active":"2024-01-16T15:00:07.526723+0000","last_became_peered":"2024-01-16T15:00:07.526723+0000","last_unstale":"2024-01-16T15:00:08.555517+0000","last_undegraded":"2024-01-16T15:00:08.555517+0000","last_fullsized":"2024-01-16T15:00:08.555517+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:12.674764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.561890+0000","last_change":"2024-01-16T15:00:08.550819+0000","last_active":"2024-01-16T15:00:08.561890+0000","last_peered":"2024-01-16T15:00:08.561890+0000","last_clean":"2024-01-16T15:00:08.561890+0000","last_became_active":"2024-01-16T15:00:08.550551+0000","last_became_peered":"2024-01-16T15:00:08.550551+0000","last_unstale":"2024-01-16T15:00:08.561890+0000","last_undegraded":"2024-01-16T15:00:08.561890+0000","last_fullsized":"2024-01-16T15:00:08.561890+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:34.775356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.524180+0000","last_change":"2024-01-16T15:00:07.523327+0000","last_active":"2024-01-16T15:00:08.524180+0000","last_peered":"2024-01-16T15:00:08.524180+0000","last_clean":"2024-01-16T15:00:08.524180+0000","last_became_active":"2024-01-16T15:00:07.523011+0000","last_became_peered":"2024-01-16T15:00:07.523011+0000","last_unstale":"2024-01-16T15:00:08.524180+0000","last_undegraded":"2024-01-16T15:00:08.524180+0000","last_fullsized":"2024-01-16T15:00:08.524180+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:04:39.594396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552587+0000","last_change":"2024-01-16T15:00:08.547054+0000","last_active":"2024-01-16T15:00:08.552587+0000","last_peered":"2024-01-16T15:00:08.552587+0000","last_clean":"2024-01-16T15:00:08.552587+0000","last_became_active":"2024-01-16T15:00:08.546741+0000","last_became_peered":"2024-01-16T15:00:08.546741+0000","last_unstale":"2024-01-16T15:00:08.552587+0000","last_undegraded":"2024-01-16T15:00:08.552587+0000","last_fullsized":"2024-01-16T15:00:08.552587+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:25.902778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.540369+0000","last_change":"2024-01-16T15:00:07.520711+0000","last_active":"2024-01-16T15:00:08.540369+0000","last_peered":"2024-01-16T15:00:08.540369+0000","last_clean":"2024-01-16T15:00:08.540369+0000","last_became_active":"2024-01-16T15:00:07.520440+0000","last_became_peered":"2024-01-16T15:00:07.520440+0000","last_unstale":"2024-01-16T15:00:08.540369+0000","last_undegraded":"2024-01-16T15:00:08.540369+0000","last_fullsized":"2024-01-16T15:00:08.540369+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:39.974432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.560765+0000","last_change":"2024-01-16T15:00:08.558301+0000","last_active":"2024-01-16T15:00:08.560765+0000","last_peered":"2024-01-16T15:00:08.560765+0000","last_clean":"2024-01-16T15:00:08.560765+0000","last_became_active":"2024-01-16T15:00:08.558040+0000","last_became_peered":"2024-01-16T15:00:08.558040+0000","last_unstale":"2024-01-16T15:00:08.560765+0000","last_undegraded":"2024-01-16T15:00:08.560765+0000","last_fullsized":"2024-01-16T15:00:08.560765+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:26.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.530143+0000","last_change":"2024-01-16T15:00:07.509925+0000","last_active":"2024-01-16T15:00:08.530143+0000","last_peered":"2024-01-16T15:00:08.530143+0000","last_clean":"2024-01-16T15:00:08.530143+0000","last_became_active":"2024-01-16T15:00:07.509361+0000","last_became_peered":"2024-01-16T15:00:07.509361+0000","last_unstale":"2024-01-16T15:00:08.530143+0000","last_undegraded":"2024-01-16T15:00:08.530143+0000","last_fullsized":"2024-01-16T15:00:08.530143+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:46.950866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.543208+0000","last_change":"2024-01-16T15:00:08.536213+0000","last_active":"2024-01-16T15:00:08.543208+0000","last_peered":"2024-01-16T15:00:08.543208+0000","last_clean":"2024-01-16T15:00:08.543208+0000","last_became_active":"2024-01-16T15:00:08.532964+0000","last_became_peered":"2024-01-16T15:00:08.532964+0000","last_unstale":"2024-01-16T15:00:08.543208+0000","last_undegraded":"2024-01-16T15:00:08.543208+0000","last_fullsized":"2024-01-16T15:00:08.543208+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:13.644323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547080+0000","last_change":"2024-01-16T15:00:07.522394+0000","last_active":"2024-01-16T15:00:08.547080+0000","last_peered":"2024-01-16T15:00:08.547080+0000","last_clean":"2024-01-16T15:00:08.547080+0000","last_became_active":"2024-01-16T15:00:07.522116+0000","last_became_peered":"2024-01-16T15:00:07.522116+0000","last_unstale":"2024-01-16T15:00:08.547080+0000","last_undegraded":"2024-01-16T15:00:08.547080+0000","last_fullsized":"2024-01-16T15:00:08.547080+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:12.769085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.559511+0000","last_change":"2024-01-16T15:00:08.555601+0000","last_active":"2024-01-16T15:00:08.559511+0000","last_peered":"2024-01-16T15:00:08.559511+0000","last_clean":"2024-01-16T15:00:08.559511+0000","last_became_active":"2024-01-16T15:00:08.555315+0000","last_became_peered":"2024-01-16T15:00:08.555315+0000","last_unstale":"2024-01-16T15:00:08.559511+0000","last_undegraded":"2024-01-16T15:00:08.559511+0000","last_fullsized":"2024-01-16T15:00:08.559511+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:01:52.394343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566256+0000","last_change":"2024-01-16T15:00:07.527997+0000","last_active":"2024-01-16T15:00:08.566256+0000","last_peered":"2024-01-16T15:00:08.566256+0000","last_clean":"2024-01-16T15:00:08.566256+0000","last_became_active":"2024-01-16T15:00:07.527734+0000","last_became_peered":"2024-01-16T15:00:07.527734+0000","last_unstale":"2024-01-16T15:00:08.566256+0000","last_undegraded":"2024-01-16T15:00:08.566256+0000","last_fullsized":"2024-01-16T15:00:08.566256+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:42.035002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563827+0000","last_change":"2024-01-16T15:00:08.561750+0000","last_active":"2024-01-16T15:00:08.563827+0000","last_peered":"2024-01-16T15:00:08.563827+0000","last_clean":"2024-01-16T15:00:08.563827+0000","last_became_active":"2024-01-16T15:00:08.561473+0000","last_became_peered":"2024-01-16T15:00:08.561473+0000","last_unstale":"2024-01-16T15:00:08.563827+0000","last_undegraded":"2024-01-16T15:00:08.563827+0000","last_fullsized":"2024-01-16T15:00:08.563827+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:04.210496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.574246+0000","last_change":"2024-01-16T15:00:07.515751+0000","last_active":"2024-01-16T15:00:08.574246+0000","last_peered":"2024-01-16T15:00:08.574246+0000","last_clean":"2024-01-16T15:00:08.574246+0000","last_became_active":"2024-01-16T15:00:07.514812+0000","last_became_peered":"2024-01-16T15:00:07.514812+0000","last_unstale":"2024-01-16T15:00:08.574246+0000","last_undegraded":"2024-01-16T15:00:08.574246+0000","last_fullsized":"2024-01-16T15:00:08.574246+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:17:19.391074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563636+0000","last_change":"2024-01-16T15:00:08.560164+0000","last_active":"2024-01-16T15:00:08.563636+0000","last_peered":"2024-01-16T15:00:08.563636+0000","last_clean":"2024-01-16T15:00:08.563636+0000","last_became_active":"2024-01-16T15:00:08.559883+0000","last_became_peered":"2024-01-16T15:00:08.559883+0000","last_unstale":"2024-01-16T15:00:08.563636+0000","last_undegraded":"2024-01-16T15:00:08.563636+0000","last_fullsized":"2024-01-16T15:00:08.563636+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:58.168006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571982+0000","last_change":"2024-01-16T15:00:07.518437+0000","last_active":"2024-01-16T15:00:08.571982+0000","last_peered":"2024-01-16T15:00:08.571982+0000","last_clean":"2024-01-16T15:00:08.571982+0000","last_became_active":"2024-01-16T15:00:07.518169+0000","last_became_peered":"2024-01-16T15:00:07.518169+0000","last_unstale":"2024-01-16T15:00:08.571982+0000","last_undegraded":"2024-01-16T15:00:08.571982+0000","last_fullsized":"2024-01-16T15:00:08.571982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:43.595100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.556142+0000","last_change":"2024-01-16T15:00:08.536009+0000","last_active":"2024-01-16T15:00:08.556142+0000","last_peered":"2024-01-16T15:00:08.556142+0000","last_clean":"2024-01-16T15:00:08.556142+0000","last_became_active":"2024-01-16T15:00:08.535704+0000","last_became_peered":"2024-01-16T15:00:08.535704+0000","last_unstale":"2024-01-16T15:00:08.556142+0000","last_undegraded":"2024-01-16T15:00:08.556142+0000","last_fullsized":"2024-01-16T15:00:08.556142+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:31.059425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.526683+0000","last_change":"2024-01-16T15:00:07.526795+0000","last_active":"2024-01-16T15:00:08.526683+0000","last_peered":"2024-01-16T15:00:08.526683+0000","last_clean":"2024-01-16T15:00:08.526683+0000","last_became_active":"2024-01-16T15:00:07.526461+0000","last_became_peered":"2024-01-16T15:00:07.526461+0000","last_unstale":"2024-01-16T15:00:08.526683+0000","last_undegraded":"2024-01-16T15:00:08.526683+0000","last_fullsized":"2024-01-16T15:00:08.526683+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:26.258788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.551777+0000","last_change":"2024-01-16T15:00:08.548663+0000","last_active":"2024-01-16T15:00:08.551777+0000","last_peered":"2024-01-16T15:00:08.551777+0000","last_clean":"2024-01-16T15:00:08.551777+0000","last_became_active":"2024-01-16T15:00:08.548390+0000","last_became_peered":"2024-01-16T15:00:08.548390+0000","last_unstale":"2024-01-16T15:00:08.551777+0000","last_undegraded":"2024-01-16T15:00:08.551777+0000","last_fullsized":"2024-01-16T15:00:08.551777+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:29:19.904575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.527301+0000","last_change":"2024-01-16T15:00:07.518411+0000","last_active":"2024-01-16T15:00:08.527301+0000","last_peered":"2024-01-16T15:00:08.527301+0000","last_clean":"2024-01-16T15:00:08.527301+0000","last_became_active":"2024-01-16T15:00:07.518124+0000","last_became_peered":"2024-01-16T15:00:07.518124+0000","last_unstale":"2024-01-16T15:00:08.527301+0000","last_undegraded":"2024-01-16T15:00:08.527301+0000","last_fullsized":"2024-01-16T15:00:08.527301+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:52.845621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.547130+0000","last_change":"2024-01-16T15:00:08.542930+0000","last_active":"2024-01-16T15:00:08.547130+0000","last_peered":"2024-01-16T15:00:08.547130+0000","last_clean":"2024-01-16T15:00:08.547130+0000","last_became_active":"2024-01-16T15:00:08.542625+0000","last_became_peered":"2024-01-16T15:00:08.542625+0000","last_unstale":"2024-01-16T15:00:08.547130+0000","last_undegraded":"2024-01-16T15:00:08.547130+0000","last_fullsized":"2024-01-16T15:00:08.547130+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:59:17.485656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":19,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.573734+0000","last_change":"2024-01-16T15:00:07.525232+0000","last_active":"2024-01-16T15:00:08.573734+0000","last_peered":"2024-01-16T15:00:08.573734+0000","last_clean":"2024-01-16T15:00:08.573734+0000","last_became_active":"2024-01-16T15:00:07.524943+0000","last_became_peered":"2024-01-16T15:00:07.524943+0000","last_unstale":"2024-01-16T15:00:08.573734+0000","last_undegraded":"2024-01-16T15:00:08.573734+0000","last_fullsized":"2024-01-16T15:00:08.573734+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:40.597612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.548468+0000","last_change":"2024-01-16T15:00:08.537997+0000","last_active":"2024-01-16T15:00:08.548468+0000","last_peered":"2024-01-16T15:00:08.548468+0000","last_clean":"2024-01-16T15:00:08.548468+0000","last_became_active":"2024-01-16T15:00:08.537674+0000","last_became_peered":"2024-01-16T15:00:08.537674+0000","last_unstale":"2024-01-16T15:00:08.548468+0000","last_undegraded":"2024-01-16T15:00:08.548468+0000","last_fullsized":"2024-01-16T15:00:08.548468+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:56.758344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.572186+0000","last_change":"2024-01-16T15:00:07.515769+0000","last_active":"2024-01-16T15:00:08.572186+0000","last_peered":"2024-01-16T15:00:08.572186+0000","last_clean":"2024-01-16T15:00:08.572186+0000","last_became_active":"2024-01-16T15:00:07.514876+0000","last_became_peered":"2024-01-16T15:00:07.514876+0000","last_unstale":"2024-01-16T15:00:08.572186+0000","last_undegraded":"2024-01-16T15:00:08.572186+0000","last_fullsized":"2024-01-16T15:00:08.572186+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:11:09.934752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.559775+0000","last_change":"2024-01-16T15:00:08.549642+0000","last_active":"2024-01-16T15:00:08.559775+0000","last_peered":"2024-01-16T15:00:08.559775+0000","last_clean":"2024-01-16T15:00:08.559775+0000","last_became_active":"2024-01-16T15:00:08.549325+0000","last_became_peered":"2024-01-16T15:00:08.549325+0000","last_unstale":"2024-01-16T15:00:08.559775+0000","last_undegraded":"2024-01-16T15:00:08.559775+0000","last_fullsized":"2024-01-16T15:00:08.559775+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:43.578946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566624+0000","last_change":"2024-01-16T15:00:07.516841+0000","last_active":"2024-01-16T15:00:08.566624+0000","last_peered":"2024-01-16T15:00:08.566624+0000","last_clean":"2024-01-16T15:00:08.566624+0000","last_became_active":"2024-01-16T15:00:07.516522+0000","last_became_peered":"2024-01-16T15:00:07.516522+0000","last_unstale":"2024-01-16T15:00:08.566624+0000","last_undegraded":"2024-01-16T15:00:08.566624+0000","last_fullsized":"2024-01-16T15:00:08.566624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:22.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552779+0000","last_change":"2024-01-16T15:00:08.549017+0000","last_active":"2024-01-16T15:00:08.552779+0000","last_peered":"2024-01-16T15:00:08.552779+0000","last_clean":"2024-01-16T15:00:08.552779+0000","last_became_active":"2024-01-16T15:00:08.548730+0000","last_became_peered":"2024-01-16T15:00:08.548730+0000","last_unstale":"2024-01-16T15:00:08.552779+0000","last_undegraded":"2024-01-16T15:00:08.552779+0000","last_fullsized":"2024-01-16T15:00:08.552779+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:05.699896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.572786+0000","last_change":"2024-01-16T15:00:07.523324+0000","last_active":"2024-01-16T15:00:08.572786+0000","last_peered":"2024-01-16T15:00:08.572786+0000","last_clean":"2024-01-16T15:00:08.572786+0000","last_became_active":"2024-01-16T15:00:07.523035+0000","last_became_peered":"2024-01-16T15:00:07.523035+0000","last_unstale":"2024-01-16T15:00:08.572786+0000","last_undegraded":"2024-01-16T15:00:08.572786+0000","last_fullsized":"2024-01-16T15:00:08.572786+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:03.852422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566309+0000","last_change":"2024-01-16T15:00:08.561723+0000","last_active":"2024-01-16T15:00:08.566309+0000","last_peered":"2024-01-16T15:00:08.566309+0000","last_clean":"2024-01-16T15:00:08.566309+0000","last_became_active":"2024-01-16T15:00:08.561379+0000","last_became_peered":"2024-01-16T15:00:08.561379+0000","last_unstale":"2024-01-16T15:00:08.566309+0000","last_undegraded":"2024-01-16T15:00:08.566309+0000","last_fullsized":"2024-01-16T15:00:08.566309+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:18.717365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.531908+0000","last_change":"2024-01-16T15:00:07.525994+0000","last_active":"2024-01-16T15:00:08.531908+0000","last_peered":"2024-01-16T15:00:08.531908+0000","last_clean":"2024-01-16T15:00:08.531908+0000","last_became_active":"2024-01-16T15:00:07.525765+0000","last_became_peered":"2024-01-16T15:00:07.525765+0000","last_unstale":"2024-01-16T15:00:08.531908+0000","last_undegraded":"2024-01-16T15:00:08.531908+0000","last_fullsized":"2024-01-16T15:00:08.531908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:44.875264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563116+0000","last_change":"2024-01-16T15:00:08.557781+0000","last_active":"2024-01-16T15:00:08.563116+0000","last_peered":"2024-01-16T15:00:08.563116+0000","last_clean":"2024-01-16T15:00:08.563116+0000","last_became_active":"2024-01-16T15:00:08.557413+0000","last_became_peered":"2024-01-16T15:00:08.557413+0000","last_unstale":"2024-01-16T15:00:08.563116+0000","last_undegraded":"2024-01-16T15:00:08.563116+0000","last_fullsized":"2024-01-16T15:00:08.563116+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:28.697891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.572033+0000","last_change":"2024-01-16T15:00:07.527222+0000","last_active":"2024-01-16T15:00:08.572033+0000","last_peered":"2024-01-16T15:00:08.572033+0000","last_clean":"2024-01-16T15:00:08.572033+0000","last_became_active":"2024-01-16T15:00:07.526918+0000","last_became_peered":"2024-01-16T15:00:07.526918+0000","last_unstale":"2024-01-16T15:00:08.572033+0000","last_undegraded":"2024-01-16T15:00:08.572033+0000","last_fullsized":"2024-01-16T15:00:08.572033+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:12.165083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.569445+0000","last_change":"2024-01-16T15:00:08.564185+0000","last_active":"2024-01-16T15:00:08.569445+0000","last_peered":"2024-01-16T15:00:08.569445+0000","last_clean":"2024-01-16T15:00:08.569445+0000","last_became_active":"2024-01-16T15:00:08.560525+0000","last_became_peered":"2024-01-16T15:00:08.560525+0000","last_unstale":"2024-01-16T15:00:08.569445+0000","last_undegraded":"2024-01-16T15:00:08.569445+0000","last_fullsized":"2024-01-16T15:00:08.569445+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:02.217215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.525908+0000","last_change":"2024-01-16T15:00:07.522096+0000","last_active":"2024-01-16T15:00:08.525908+0000","last_peered":"2024-01-16T15:00:08.525908+0000","last_clean":"2024-01-16T15:00:08.525908+0000","last_became_active":"2024-01-16T15:00:07.521466+0000","last_became_peered":"2024-01-16T15:00:07.521466+0000","last_unstale":"2024-01-16T15:00:08.525908+0000","last_undegraded":"2024-01-16T15:00:08.525908+0000","last_fullsized":"2024-01-16T15:00:08.525908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:12.630365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.552523+0000","last_change":"2024-01-16T15:00:08.544028+0000","last_active":"2024-01-16T15:00:08.552523+0000","last_peered":"2024-01-16T15:00:08.552523+0000","last_clean":"2024-01-16T15:00:08.552523+0000","last_became_active":"2024-01-16T15:00:08.543635+0000","last_became_peered":"2024-01-16T15:00:08.543635+0000","last_unstale":"2024-01-16T15:00:08.552523+0000","last_undegraded":"2024-01-16T15:00:08.552523+0000","last_fullsized":"2024-01-16T15:00:08.552523+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:33.048860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.524361+0000","last_change":"2024-01-16T15:00:07.523438+0000","last_active":"2024-01-16T15:00:08.524361+0000","last_peered":"2024-01-16T15:00:08.524361+0000","last_clean":"2024-01-16T15:00:08.524361+0000","last_became_active":"2024-01-16T15:00:07.523084+0000","last_became_peered":"2024-01-16T15:00:07.523084+0000","last_unstale":"2024-01-16T15:00:08.524361+0000","last_undegraded":"2024-01-16T15:00:08.524361+0000","last_fullsized":"2024-01-16T15:00:08.524361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:15.387500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.563195+0000","last_change":"2024-01-16T15:00:08.556154+0000","last_active":"2024-01-16T15:00:08.563195+0000","last_peered":"2024-01-16T15:00:08.563195+0000","last_clean":"2024-01-16T15:00:08.563195+0000","last_became_active":"2024-01-16T15:00:08.555728+0000","last_became_peered":"2024-01-16T15:00:08.555728+0000","last_unstale":"2024-01-16T15:00:08.563195+0000","last_undegraded":"2024-01-16T15:00:08.563195+0000","last_fullsized":"2024-01-16T15:00:08.563195+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:35:36.232094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.523201+0000","last_change":"2024-01-16T15:00:07.526002+0000","last_active":"2024-01-16T15:00:08.523201+0000","last_peered":"2024-01-16T15:00:08.523201+0000","last_clean":"2024-01-16T15:00:08.523201+0000","last_became_active":"2024-01-16T15:00:07.525647+0000","last_became_peered":"2024-01-16T15:00:07.525647+0000","last_unstale":"2024-01-16T15:00:08.523201+0000","last_undegraded":"2024-01-16T15:00:08.523201+0000","last_fullsized":"2024-01-16T15:00:08.523201+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:19.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.566096+0000","last_change":"2024-01-16T15:00:08.560698+0000","last_active":"2024-01-16T15:00:08.566096+0000","last_peered":"2024-01-16T15:00:08.566096+0000","last_clean":"2024-01-16T15:00:08.566096+0000","last_became_active":"2024-01-16T15:00:08.560052+0000","last_became_peered":"2024-01-16T15:00:08.560052+0000","last_unstale":"2024-01-16T15:00:08.566096+0000","last_undegraded":"2024-01-16T15:00:08.566096+0000","last_fullsized":"2024-01-16T15:00:08.566096+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:57.480817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.531062+0000","last_change":"2024-01-16T15:00:07.524975+0000","last_active":"2024-01-16T15:00:08.531062+0000","last_peered":"2024-01-16T15:00:08.531062+0000","last_clean":"2024-01-16T15:00:08.531062+0000","last_became_active":"2024-01-16T15:00:07.524369+0000","last_became_peered":"2024-01-16T15:00:07.524369+0000","last_unstale":"2024-01-16T15:00:08.531062+0000","last_undegraded":"2024-01-16T15:00:08.531062+0000","last_fullsized":"2024-01-16T15:00:08.531062+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:44.886009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554528+0000","last_change":"2024-01-16T15:00:08.537143+0000","last_active":"2024-01-16T15:00:08.554528+0000","last_peered":"2024-01-16T15:00:08.554528+0000","last_clean":"2024-01-16T15:00:08.554528+0000","last_became_active":"2024-01-16T15:00:08.536676+0000","last_became_peered":"2024-01-16T15:00:08.536676+0000","last_unstale":"2024-01-16T15:00:08.554528+0000","last_undegraded":"2024-01-16T15:00:08.554528+0000","last_fullsized":"2024-01-16T15:00:08.554528+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:26.566875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.526360+0000","last_change":"2024-01-16T15:00:07.528578+0000","last_active":"2024-01-16T15:00:08.526360+0000","last_peered":"2024-01-16T15:00:08.526360+0000","last_clean":"2024-01-16T15:00:08.526360+0000","last_became_active":"2024-01-16T15:00:07.528259+0000","last_became_peered":"2024-01-16T15:00:07.528259+0000","last_unstale":"2024-01-16T15:00:08.526360+0000","last_undegraded":"2024-01-16T15:00:08.526360+0000","last_fullsized":"2024-01-16T15:00:08.526360+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:01.903132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.546878+0000","last_change":"2024-01-16T15:00:08.545558+0000","last_active":"2024-01-16T15:00:08.546878+0000","last_peered":"2024-01-16T15:00:08.546878+0000","last_clean":"2024-01-16T15:00:08.546878+0000","last_became_active":"2024-01-16T15:00:08.545257+0000","last_became_peered":"2024-01-16T15:00:08.545257+0000","last_unstale":"2024-01-16T15:00:08.546878+0000","last_undegraded":"2024-01-16T15:00:08.546878+0000","last_fullsized":"2024-01-16T15:00:08.546878+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:04.866379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.551512+0000","last_change":"2024-01-16T15:00:07.521641+0000","last_active":"2024-01-16T15:00:08.551512+0000","last_peered":"2024-01-16T15:00:08.551512+0000","last_clean":"2024-01-16T15:00:08.551512+0000","last_became_active":"2024-01-16T15:00:07.521431+0000","last_became_peered":"2024-01-16T15:00:07.521431+0000","last_unstale":"2024-01-16T15:00:08.551512+0000","last_undegraded":"2024-01-16T15:00:08.551512+0000","last_fullsized":"2024-01-16T15:00:08.551512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:19.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.571400+0000","last_change":"2024-01-16T15:00:08.565783+0000","last_active":"2024-01-16T15:00:08.571400+0000","last_peered":"2024-01-16T15:00:08.571400+0000","last_clean":"2024-01-16T15:00:08.571400+0000","last_became_active":"2024-01-16T15:00:08.565524+0000","last_became_peered":"2024-01-16T15:00:08.565524+0000","last_unstale":"2024-01-16T15:00:08.571400+0000","last_undegraded":"2024-01-16T15:00:08.571400+0000","last_fullsized":"2024-01-16T15:00:08.571400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:43:36.337639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.526138+0000","last_change":"2024-01-16T15:00:07.528974+0000","last_active":"2024-01-16T15:00:08.526138+0000","last_peered":"2024-01-16T15:00:08.526138+0000","last_clean":"2024-01-16T15:00:08.526138+0000","last_became_active":"2024-01-16T15:00:07.528466+0000","last_became_peered":"2024-01-16T15:00:07.528466+0000","last_unstale":"2024-01-16T15:00:08.526138+0000","last_undegraded":"2024-01-16T15:00:08.526138+0000","last_fullsized":"2024-01-16T15:00:08.526138+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:59.733387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.567905+0000","last_change":"2024-01-16T15:00:08.559998+0000","last_active":"2024-01-16T15:00:08.567905+0000","last_peered":"2024-01-16T15:00:08.567905+0000","last_clean":"2024-01-16T15:00:08.567905+0000","last_became_active":"2024-01-16T15:00:08.559664+0000","last_became_peered":"2024-01-16T15:00:08.559664+0000","last_unstale":"2024-01-16T15:00:08.567905+0000","last_undegraded":"2024-01-16T15:00:08.567905+0000","last_fullsized":"2024-01-16T15:00:08.567905+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:43.478443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.521948+0000","last_change":"2024-01-16T15:00:07.519669+0000","last_active":"2024-01-16T15:00:08.521948+0000","last_peered":"2024-01-16T15:00:08.521948+0000","last_clean":"2024-01-16T15:00:08.521948+0000","last_became_active":"2024-01-16T15:00:07.517915+0000","last_became_peered":"2024-01-16T15:00:07.517915+0000","last_unstale":"2024-01-16T15:00:08.521948+0000","last_undegraded":"2024-01-16T15:00:08.521948+0000","last_fullsized":"2024-01-16T15:00:08.521948+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:44:07.206101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.555206+0000","last_change":"2024-01-16T15:00:08.552602+0000","last_active":"2024-01-16T15:00:08.555206+0000","last_peered":"2024-01-16T15:00:08.555206+0000","last_clean":"2024-01-16T15:00:08.555206+0000","last_became_active":"2024-01-16T15:00:08.552271+0000","last_became_peered":"2024-01-16T15:00:08.552271+0000","last_unstale":"2024-01-16T15:00:08.555206+0000","last_undegraded":"2024-01-16T15:00:08.555206+0000","last_fullsized":"2024-01-16T15:00:08.555206+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:59:17.485656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.550736+0000","last_change":"2024-01-16T15:00:07.526284+0000","last_active":"2024-01-16T15:00:08.550736+0000","last_peered":"2024-01-16T15:00:08.550736+0000","last_clean":"2024-01-16T15:00:08.550736+0000","last_became_active":"2024-01-16T15:00:07.525987+0000","last_became_peered":"2024-01-16T15:00:07.525987+0000","last_unstale":"2024-01-16T15:00:08.550736+0000","last_undegraded":"2024-01-16T15:00:08.550736+0000","last_fullsized":"2024-01-16T15:00:08.550736+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:31.964473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.554431+0000","last_change":"2024-01-16T15:00:08.550156+0000","last_active":"2024-01-16T15:00:08.554431+0000","last_peered":"2024-01-16T15:00:08.554431+0000","last_clean":"2024-01-16T15:00:08.554431+0000","last_became_active":"2024-01-16T15:00:08.549857+0000","last_became_peered":"2024-01-16T15:00:08.549857+0000","last_unstale":"2024-01-16T15:00:08.554431+0000","last_undegraded":"2024-01-16T15:00:08.554431+0000","last_fullsized":"2024-01-16T15:00:08.554431+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:32.978431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.540145+0000","last_change":"2024-01-16T15:00:07.516846+0000","last_active":"2024-01-16T15:00:08.540145+0000","last_peered":"2024-01-16T15:00:08.540145+0000","last_clean":"2024-01-16T15:00:08.540145+0000","last_became_active":"2024-01-16T15:00:07.516509+0000","last_became_peered":"2024-01-16T15:00:07.516509+0000","last_unstale":"2024-01-16T15:00:08.540145+0000","last_undegraded":"2024-01-16T15:00:08.540145+0000","last_fullsized":"2024-01-16T15:00:08.540145+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:11.646032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.556965+0000","last_change":"2024-01-16T15:00:08.547704+0000","last_active":"2024-01-16T15:00:08.556965+0000","last_peered":"2024-01-16T15:00:08.556965+0000","last_clean":"2024-01-16T15:00:08.556965+0000","last_became_active":"2024-01-16T15:00:08.547287+0000","last_became_peered":"2024-01-16T15:00:08.547287+0000","last_unstale":"2024-01-16T15:00:08.556965+0000","last_undegraded":"2024-01-16T15:00:08.556965+0000","last_fullsized":"2024-01-16T15:00:08.556965+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:07.485558+0000","last_clean_scrub_stamp":"2024-01-16T15:00:07.485558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:50.283502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-16T15:00:08.568862+0000","last_change":"2024-01-16T15:00:07.524923+0000","last_active":"2024-01-16T15:00:08.568862+0000","last_peered":"2024-01-16T15:00:08.568862+0000","last_clean":"2024-01-16T15:00:08.568862+0000","last_became_active":"2024-01-16T15:00:07.524609+0000","last_became_peered":"2024-01-16T15:00:07.524609+0000","last_unstale":"2024-01-16T15:00:08.568862+0000","last_undegraded":"2024-01-16T15:00:08.568862+0000","last_fullsized":"2024-01-16T15:00:08.568862+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:06.471600+0000","last_clean_scrub_stamp":"2024-01-16T15:00:06.471600+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:07:42.035002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":187,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-16T15:00:07.517425+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:00:07.517425+0000","last_peered":"2024-01-16T15:00:07.517425+0000","last_clean":"2024-01-16T15:00:07.517425+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:00:07.517425+0000","last_undegraded":"2024-01-16T15:00:07.517425+0000","last_fullsized":"2024-01-16T15:00:07.517425+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":801792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.105,"15min":1.073},"min":{"1min":0.746,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.236,"15min":1.1759999999999999},"min":{"1min":0.755,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.056,"5min":2.774,"15min":2.774},"last":1.3879999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.083,"15min":1.042},"min":{"1min":0.69199999999999995,"5min":0.63,"15min":0.63},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.724,"5min":1.272,"15min":1.167},"min":{"1min":0.64900000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":8.3360000000000003,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":0.95699999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.2290000000000001,"15min":1.177},"min":{"1min":1.0249999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.262,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.603,"5min":1.2629999999999999,"15min":1.2310000000000001},"min":{"1min":0.91600000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":6.1559999999999997,"5min":6.1559999999999997,"15min":6.1559999999999997},"last":0.89900000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.304,"15min":1.29},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.464,"5min":2.464,"15min":2.464},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.45,"15min":1.3979999999999999},"min":{"1min":1.117,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.5710000000000002,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.101}]},{"osd":4,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3560000000000001,"15min":1.3540000000000001},"min":{"1min":0.78900000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.431,"15min":1.4430000000000001},"min":{"1min":0.98499999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.6990000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.524,"15min":1.5289999999999999},"min":{"1min":0.93999999999999995,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.1219999999999999,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.645,"5min":1.595,"15min":1.627},"min":{"1min":1.214,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.601,"5min":3.3530000000000002,"15min":3.3530000000000002},"last":1.0149999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.5720000000000001,"15min":1.579},"min":{"1min":1.1020000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.7709999999999999,"5min":3.258,"15min":3.258},"last":1.26},{"interface":"front","average":{"1min":1.601,"5min":1.5649999999999999,"15min":1.599},"min":{"1min":1.1379999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.2749999999999999,"5min":3.101,"15min":3.101},"last":1.4370000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":212006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.099,"15min":1.093},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.554,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.2010000000000001,"15min":1.22},"min":{"1min":0.71199999999999997,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.0710000000000002,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":1.528}]},{"osd":1,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.0980000000000001,"15min":1.046},"min":{"1min":0.873,"5min":0.624,"15min":0.624},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.423,"5min":1.3939999999999999,"15min":1.3680000000000001},"min":{"1min":0.69299999999999995,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":4.5380000000000003,"5min":6.8029999999999999,"15min":6.8029999999999999},"last":1.4379999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.286,"15min":1.2},"min":{"1min":0.871,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.764,"5min":8.0449999999999999,"15min":8.0449999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.28,"15min":1.2869999999999999},"min":{"1min":0.98899999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.7150000000000001,"5min":1.95,"15min":1.95},"last":1.143}]},{"osd":3,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.331,"15min":1.343},"min":{"1min":0.72699999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.387,"15min":1.379},"min":{"1min":0.95399999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.0779999999999998,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.6519999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.458,"15min":1.486},"min":{"1min":0.97699999999999998,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.9650000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.24},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4930000000000001,"15min":1.5149999999999999},"min":{"1min":0.94299999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.2679999999999998,"5min":2.331,"15min":2.331},"last":1.776}]},{"osd":5,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.452,"15min":1.47},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.623,"15min":1.6579999999999999},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":2.206,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.5740000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.6359999999999999,"15min":1.663},"min":{"1min":1.2470000000000001,"5min":1.232,"15min":1.232},"max":{"1min":2.3490000000000002,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.623,"15min":1.661},"min":{"1min":1.048,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.488}]}]},{"osd":5,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":211404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1619999999999999,"15min":1.1819999999999999},"min":{"1min":0.70699999999999996,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.0249999999999999,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.163,"5min":1.0860000000000001,"15min":1.0820000000000001},"min":{"1min":0.67900000000000005,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.5489999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.22,"15min":1.266},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5980000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.2969999999999999,"15min":1.1919999999999999},"min":{"1min":0.80800000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":11.345000000000001,"5min":11.345000000000001,"15min":11.345000000000001},"last":1.367}]},{"osd":2,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":1.397,"15min":1.3069999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":11.445,"5min":11.445,"15min":11.445},"last":1.161},{"interface":"front","average":{"1min":1.339,"5min":1.2549999999999999,"15min":1.264},"min":{"1min":0.83999999999999997,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":1.452}]},{"osd":3,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.337},"min":{"1min":0.96599999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.1339999999999999,"5min":2.323,"15min":2.323},"last":1.4099999999999999},{"interface":"front","average":{"1min":2.1320000000000001,"5min":1.5289999999999999,"15min":1.4410000000000001},"min":{"1min":1.125,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":11.396000000000001,"5min":11.396000000000001,"15min":11.396000000000001},"last":1.5960000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.367,"15min":1.4399999999999999},"min":{"1min":0.63500000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.476,"15min":1.538},"min":{"1min":0.69799999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.8279999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.0449999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":1.645,"15min":1.6180000000000001},"min":{"1min":1.0189999999999999,"5min":0.499,"15min":0.499},"max":{"1min":11.512,"5min":11.512,"15min":11.512},"last":1.266},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5860000000000001,"15min":1.6539999999999999},"min":{"1min":0.83299999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0510000000000002,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.581,"15min":1.617},"min":{"1min":0.65700000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":1.627},{"interface":"front","average":{"1min":2.1640000000000001,"5min":1.768,"15min":1.746},"min":{"1min":1.123,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":11.476000000000001,"5min":11.476000000000001,"15min":11.476000000000001},"last":1.508}]}]},{"osd":4,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":212946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1220000000000001,"15min":1.089},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":1.109},{"interface":"front","average":{"1min":1.577,"5min":1.2869999999999999,"15min":1.24},"min":{"1min":0.69699999999999995,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":6.1790000000000003,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":1.0620000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.266,"15min":1.202},"min":{"1min":0.73499999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":6.2610000000000001,"5min":6.2610000000000001,"15min":6.2610000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":3.6659999999999999,"5min":3.6659999999999999,"15min":3.6659999999999999},"last":1.216}]},{"osd":2,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.4430000000000001,"15min":1.3680000000000001},"min":{"1min":0.72799999999999998,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.742,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.393,"15min":1.4470000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.4319999999999999,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":0.95799999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.5609999999999999,"15min":1.579},"min":{"1min":0.79300000000000004,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":6.4219999999999997,"5min":6.4219999999999997,"15min":6.4219999999999997},"last":1.48},{"interface":"front","average":{"1min":2.4409999999999998,"5min":1.5940000000000001,"15min":1.4910000000000001},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":21.706,"5min":21.706,"15min":21.706},"last":1.012}]},{"osd":5,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.5249999999999999,"15min":1.605},"min":{"1min":0.75700000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.2149999999999999,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.607,"15min":1.6140000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":6.3129999999999997,"5min":6.3129999999999997,"15min":6.3129999999999997},"last":1.4079999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.5089999999999999,"15min":1.5940000000000001},"min":{"1min":0.81299999999999994,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.6459999999999999,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.53,"5min":1.6299999999999999,"15min":1.6970000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.145,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":1.3540000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:02 2024","interfaces":[{"interface":"back","average":{"1min":1.883,"5min":1.718,"15min":1.722},"min":{"1min":0.94999999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":6.3659999999999997,"5min":6.3659999999999997,"15min":6.3659999999999997},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.6559999999999999,"15min":1.744},"min":{"1min":1.0149999999999999,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.8359999999999999,"5min":3.2290000000000001,"15min":3.2290000000000001},"last":1.6479999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":211972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":0.93899999999999995,"15min":0.90300000000000002},"min":{"1min":0.53800000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.2889999999999999,"5min":0.95999999999999996,"15min":0.874},"min":{"1min":0.56100000000000005,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.90500000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85599999999999998,"15min":0.83699999999999997},"min":{"1min":0.47099999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.232,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.02,"15min":1.002},"min":{"1min":0.53700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":2.819,"5min":2.819,"15min":2.819},"last":1.3480000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.056,"15min":1.0680000000000001},"min":{"1min":0.439,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.706,"5min":4.008,"15min":4.008},"last":1.49},{"interface":"front","average":{"1min":1.423,"5min":1.1319999999999999,"15min":1.056},"min":{"1min":0.58199999999999996,"5min":0.497,"15min":0.497},"max":{"1min":3.4199999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.163}]},{"osd":4,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.131,"15min":1.091},"min":{"1min":0.83099999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":4.093,"5min":4.093,"15min":4.093},"last":1.383},{"interface":"front","average":{"1min":1.302,"5min":1.236,"15min":1.228},"min":{"1min":0.83799999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8959999999999999,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.319}]},{"osd":5,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.27,"15min":1.254},"min":{"1min":0.873,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.857,"5min":2.149,"15min":2.149},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.385,"15min":1.3720000000000001},"min":{"1min":0.75600000000000001,"5min":0.747,"15min":0.747},"max":{"1min":3.4700000000000002,"5min":3.4700000000000002,"15min":3.4700000000000002},"last":1.2949999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.2929999999999999,"15min":1.2689999999999999},"min":{"1min":0.80000000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.383,"15min":1.3540000000000001},"min":{"1min":0.89800000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.9470000000000001,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.4319999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.427,"15min":1.403},"min":{"1min":0.90100000000000002,"5min":0.755,"15min":0.755},"max":{"1min":3.3700000000000001,"5min":3.3700000000000001,"15min":3.3700000000000001},"last":1.532},{"interface":"front","average":{"1min":1.607,"5min":1.526,"15min":1.3799999999999999},"min":{"1min":0.82099999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":4.4050000000000002,"5min":18.846,"15min":18.846},"last":1.647}]}]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":212968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.91300000000000003,"15min":0.89900000000000002},"min":{"1min":0.625,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.746},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.97099999999999997,"15min":0.92500000000000004},"min":{"1min":0.59599999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.5010000000000003,"5min":4.5010000000000003,"15min":4.5010000000000003},"last":0.65100000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.0329999999999999,"15min":1.0229999999999999},"min":{"1min":0.74299999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.0249999999999999,"15min":0.98099999999999998},"min":{"1min":0.51100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":4.8250000000000002,"5min":4.8250000000000002,"15min":4.8250000000000002},"last":0.85199999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":0.98899999999999999,"15min":0.95599999999999996},"min":{"1min":0.51200000000000001,"5min":0.436,"15min":0.436},"max":{"1min":5.0540000000000003,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1220000000000001,"15min":1.1060000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":3.4340000000000002,"5min":3.4340000000000002,"15min":3.4340000000000002},"last":0.97099999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.3759999999999999,"15min":1.3360000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.875},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.389,"15min":1.361},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":5.0860000000000003,"5min":5.0860000000000003,"15min":5.0860000000000003},"last":1.0580000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4239999999999999,"15min":1.3480000000000001},"min":{"1min":0.85899999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":5.3570000000000002,"5min":8.9689999999999994,"15min":8.9689999999999994},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3340000000000001,"15min":1.323},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":4.391,"5min":4.391,"15min":4.391},"last":0.98999999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.359,"15min":1.365},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.5310000000000001,"5min":2.5310000000000001,"15min":2.5310000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.401,"15min":1.389},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":3.4790000000000001,"5min":3.4790000000000001,"15min":3.4790000000000001},"last":1.274}]},{"osd":7,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.518,"15min":1.484},"min":{"1min":0.90500000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.472,"15min":1.458},"min":{"1min":0.99099999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"last":1.337}]}]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":211404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.89400000000000002,"15min":0.91800000000000004},"min":{"1min":0.42899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0289999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.016},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.99199999999999999,"15min":1.004},"min":{"1min":0.56100000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.4239999999999999,"5min":2.073,"15min":2.073},"last":0.73999999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91900000000000004,"15min":0.91600000000000004},"min":{"1min":0.45700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.456,"5min":2.952,"15min":2.952},"last":1.224},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.95499999999999996,"15min":0.95899999999999996},"min":{"1min":0.52500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.502,"5min":2.129,"15min":2.129},"last":0.90200000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.05,"15min":1.0629999999999999},"min":{"1min":0.54000000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.3160000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.157},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.018,"15min":1.008},"min":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6970000000000001,"5min":2.641,"15min":2.641},"last":0.95999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2,"15min":1.177},"min":{"1min":0.89700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.548,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.2310000000000001,"15min":1.2470000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.498,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.476}]},{"osd":5,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3480000000000001,"15min":1.3859999999999999},"min":{"1min":0.81499999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.621,"5min":2.649,"15min":2.649},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.403,"15min":1.423},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.6739999999999999,"5min":2.766,"15min":2.766},"last":1.3700000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.4019999999999999,"15min":1.4259999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.9059999999999999,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.097},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.3779999999999999,"15min":1.4079999999999999},"min":{"1min":0.748,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.6819999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.3160000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:08 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.4079999999999999,"15min":1.4039999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":3.0779999999999998,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":1.284},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.4059999999999999,"15min":1.427},"min":{"1min":0.68400000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.8160000000000001,"5min":2.879,"15min":2.879},"last":1.2629999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":801736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97899999999999998,"15min":0.97499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.3069999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":3.6469999999999998},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.004,"15min":0.98399999999999999},"min":{"1min":0.55300000000000005,"5min":0.371,"15min":0.371},"max":{"1min":1.321,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":3.0990000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1259999999999999,"15min":1.107},"min":{"1min":0.56999999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.524,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":3.2200000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.26,"15min":1.244},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":2.395,"5min":5.5970000000000004,"15min":5.5970000000000004},"last":5.0380000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.93100000000000005,"15min":0.88700000000000001},"min":{"1min":0.628,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.597,"5min":3.597,"15min":3.597},"last":4.8899999999999997},{"interface":"front","average":{"1min":1.052,"5min":1.0649999999999999,"15min":1.0409999999999999},"min":{"1min":0.51000000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.478,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":6.1079999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.5309999999999999,"15min":1.4630000000000001},"min":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.532,"5min":14.297000000000001,"15min":14.297000000000001},"last":3.7519999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.298,"15min":1.284},"min":{"1min":0.88200000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.4489999999999998,"5min":4.5170000000000003,"15min":4.5170000000000003},"last":4.1159999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.5680000000000001,"15min":1.472},"min":{"1min":1.0529999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.4889999999999999,"5min":14.212999999999999,"15min":14.212999999999999},"last":3.9729999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3069999999999999,"15min":1.2909999999999999},"min":{"1min":0.85199999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":3.9350000000000001,"5min":3.9350000000000001,"15min":3.9350000000000001},"last":4.8399999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.417,"15min":1.4039999999999999},"min":{"1min":0.97799999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.925,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":3.1520000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6100000000000001,"15min":1.573},"min":{"1min":0.96899999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.593,"5min":5.7759999999999998,"15min":5.7759999999999998},"last":3.79}]},{"osd":7,"last update":"Tue Jan 16 15:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.552,"15min":1.454},"min":{"1min":0.88700000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":15.945,"5min":15.945,"15min":15.945},"last":3.1869999999999998},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.647,"15min":1.6140000000000001},"min":{"1min":1.1080000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.0590000000000002,"5min":5.9269999999999996,"15min":5.9269999999999996},"last":4.0129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:13.451 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:00:13.452 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:00:13.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.730+0000 7f2afc600640 1 Processor -- start 2024-01-16T15:00:13.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.730+0000 7f2afc600640 1 -- start start 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2afc600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2af415ed20 0x7f2af4141360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2afc600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af406c850 0x7f2af406cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2afc600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af413e440 con 0x7f2af415f100 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2afc600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af40ce1d0 con 0x7f2af415ed20 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2afc600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af40ce4b0 con 0x7f2af406c850 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2af9200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2af415f100 0x7f2af40cd780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58162/0 (socket says 172.21.15.141:58162) 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2af9c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2af415ed20 0x7f2af4141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.732+0000 7f2af9200640 1 -- 172.21.15.141:0/2499271843 learned_addr learned my addr 172.21.15.141:0/2499271843 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.733+0000 7f2afa600640 1 --2- 172.21.15.141:0/2499271843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af406c850 0x7f2af406cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.733+0000 7f2af9c00640 1 -- 172.21.15.141:0/2499271843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af406c850 msgr2=0x7f2af406cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.733+0000 7f2af9c00640 1 --2- 172.21.15.141:0/2499271843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af406c850 0x7f2af406cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.733+0000 7f2af9c00640 1 -- 172.21.15.141:0/2499271843 >> v1:172.21.15.141:6789/0 conn(0x7f2af415f100 legacy=0x7f2af40cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.733+0000 7f2af3e00640 1 -- 172.21.15.141:0/2499271843 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415409659 0 0) 0x7f2af413e440 con 0x7f2af415f100 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.733+0000 7f2af9c00640 1 -- 172.21.15.141:0/2499271843 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af40ce790 con 0x7f2af415ed20 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.733+0000 7f2afa600640 1 --2- 172.21.15.141:0/2499271843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af406c850 0x7f2af406cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:13.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.734+0000 7f2af9c00640 1 --2- 172.21.15.141:0/2499271843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af4141360 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40097a0 tx=0x7f2ae402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd457f67864c3de0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.734+0000 7f2af3e00640 1 -- 172.21.15.141:0/2499271843 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4031780 con 0x7f2af415ed20 2024-01-16T15:00:13.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.734+0000 7f2af3e00640 1 -- 172.21.15.141:0/2499271843 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae40318e0 con 0x7f2af415ed20 2024-01-16T15:00:13.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.734+0000 7f2af3e00640 1 -- 172.21.15.141:0/2499271843 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4031c70 con 0x7f2af415ed20 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 -- 172.21.15.141:0/2499271843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 msgr2=0x7f2af4141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 --2- 172.21.15.141:0/2499271843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af4141360 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40097a0 tx=0x7f2ae402b820 comp rx=0 tx=0).stop 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 -- 172.21.15.141:0/2499271843 shutdown_connections 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 --2- 172.21.15.141:0/2499271843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af406c850 0x7f2af406cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 --2- 172.21.15.141:0/2499271843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af4141360 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 -- 172.21.15.141:0/2499271843 >> 172.21.15.141:0/2499271843 conn(0x7f2af40b0be0 msgr2=0x7f2af40b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 -- 172.21.15.141:0/2499271843 shutdown_connections 2024-01-16T15:00:13.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.735+0000 7f2afc600640 1 -- 172.21.15.141:0/2499271843 wait complete. 2024-01-16T15:00:13.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.737+0000 7f2afc600640 1 Processor -- start 2024-01-16T15:00:13.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.737+0000 7f2afc600640 1 -- start start 2024-01-16T15:00:13.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2afc600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af406c850 0x7f2af40c6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2afc600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2afc600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af415f100 0x7f2af40c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2afc600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af4071fa0 con 0x7f2af415ed20 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2afc600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af413f5d0 con 0x7f2af406c850 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2afc600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af413eca0 con 0x7f2af415f100 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2af9200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2af9200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48994/0 (socket says 172.21.15.141:48994) 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.738+0000 7f2af9200640 1 -- 172.21.15.141:0/344817611 learned_addr learned my addr 172.21.15.141:0/344817611 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:13.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af9c00640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af406c850 0x7f2af40c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2afa600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af415f100 0x7f2af40c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af9200640 1 -- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af415f100 msgr2=0x7f2af40c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af9200640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af415f100 0x7f2af40c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af9200640 1 -- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af406c850 msgr2=0x7f2af40c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af9200640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af406c850 0x7f2af40c6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af9200640 1 -- 172.21.15.141:0/344817611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae4009400 con 0x7f2af415ed20 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af9200640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af413d7f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00c430 tx=0x7f2adc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af1600640 1 -- 172.21.15.141:0/344817611 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc0074e0 con 0x7f2af415ed20 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af1600640 1 -- 172.21.15.141:0/344817611 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc007660 con 0x7f2af415ed20 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af413ef80 con 0x7f2af415ed20 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.739+0000 7f2af1600640 1 -- 172.21.15.141:0/344817611 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc004a20 con 0x7f2af415ed20 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.740+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af413dd30 con 0x7f2af415ed20 2024-01-16T15:00:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.741+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af413f760 con 0x7f2af415ed20 2024-01-16T15:00:13.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.742+0000 7f2af1600640 1 -- 172.21.15.141:0/344817611 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2adc004b80 con 0x7f2af415ed20 2024-01-16T15:00:13.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.742+0000 7f2af1600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.743+0000 7f2af1600640 1 -- 172.21.15.141:0/344817611 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2adc089e90 con 0x7f2af415ed20 2024-01-16T15:00:13.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.743+0000 7f2af9c00640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.744+0000 7f2af9c00640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2acc06b500 0x7f2acc06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4036000 tx=0x7f2ae403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:13.747+0000 7f2af1600640 1 -- 172.21.15.141:0/344817611 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2adc054990 con 0x7f2af415ed20 2024-01-16T15:00:13.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.004+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2af4067670 con 0x7f2af415ed20 2024-01-16T15:00:13.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.008+0000 7f2af1600640 1 -- 172.21.15.141:0/344817611 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2adc057fb0 con 0x7f2af415ed20 2024-01-16T15:00:13.990 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T15:00:13.990 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T15:00:13.990 INFO:teuthology.orchestra.run.smithi141.stdout: caps mds = "allow" 2024-01-16T15:00:13.990 INFO:teuthology.orchestra.run.smithi141.stdout: caps mon = "allow r" 2024-01-16T15:00:13.990 INFO:teuthology.orchestra.run.smithi141.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:00:13.991 INFO:teuthology.orchestra.run.smithi141.stderr:updated caps for client.0 2024-01-16T15:00:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.011+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2acc06b500 msgr2=0x7f2acc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.011+0000 7f2afc600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2acc06b500 0x7f2acc06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4036000 tx=0x7f2ae403d040 comp rx=0 tx=0).stop 2024-01-16T15:00:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 msgr2=0x7f2af413d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af413d7f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00c430 tx=0x7f2adc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 shutdown_connections 2024-01-16T15:00:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af415f100 0x7f2af40c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af415ed20 0x7f2af413d7f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 --2- 172.21.15.141:0/344817611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af406c850 0x7f2af40c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 >> 172.21.15.141:0/344817611 conn(0x7f2af40b0be0 msgr2=0x7f2af40cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 shutdown_connections 2024-01-16T15:00:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.012+0000 7f2afc600640 1 -- 172.21.15.141:0/344817611 wait complete. 2024-01-16T15:00:14.011 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:14.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.236+0000 7f7e85400640 1 Processor -- start 2024-01-16T15:00:14.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.236+0000 7f7e85400640 1 -- start start 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.238+0000 7f7e85400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7e8013fcb0 0x7f7e80164d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.238+0000 7f7e85400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e800cdc40 con 0x7f7e8013cfa0 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.238+0000 7f7e85400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e800cdec0 con 0x7f7e8015afb0 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.238+0000 7f7e85400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e800ce1a0 con 0x7f7e8013fcb0 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.238+0000 7f7e7fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7e8013cfa0 0x7f7e800cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58174/0 (socket says 172.21.15.141:58174) 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.238+0000 7f7e7fe00640 1 -- 172.21.15.141:0/2370791942 learned_addr learned my addr 172.21.15.141:0/2370791942 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.238+0000 7f7e84a00640 1 --2- 172.21.15.141:0/2370791942 >> v2:172.21.15.188:3300/0 conn(0x7f7e8013fcb0 0x7f7e80164d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069456781 0 0) 0x7f7e800cdc40 con 0x7f7e8013cfa0 2024-01-16T15:00:14.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e60003520 con 0x7f7e8013cfa0 2024-01-16T15:00:14.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913393312 0 0) 0x7f7e60003520 con 0x7f7e8013cfa0 2024-01-16T15:00:14.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 >> v2:172.21.15.188:3300/0 conn(0x7f7e8013fcb0 msgr2=0x7f7e80164d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 --2- 172.21.15.141:0/2370791942 >> v2:172.21.15.188:3300/0 conn(0x7f7e8013fcb0 0x7f7e80164d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 >> v1:172.21.15.188:6789/0 conn(0x7f7e8015afb0 legacy=0x7f7e8013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800ce480 con 0x7f7e8013cfa0 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7e70001670 con 0x7f7e8013cfa0 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 >> v1:172.21.15.141:6789/0 conn(0x7f7e8013cfa0 legacy=0x7f7e800cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e60003970 0x7f7e60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e600047c0 0x7f7e60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e60005180 0x7f7e6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800ce1a0 con 0x7f7e600047c0 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6000c120 con 0x7f7e60003970 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6000c2e0 con 0x7f7e60005180 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.239+0000 7f7e7f400640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e600047c0 0x7f7e60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.240+0000 7f7e7fe00640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e60003970 0x7f7e60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.240+0000 7f7e7f400640 1 -- 172.21.15.141:0/2370791942 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e60005180 msgr2=0x7f7e6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.240+0000 7f7e7f400640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e60005180 0x7f7e6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.240+0000 7f7e7f400640 1 -- 172.21.15.141:0/2370791942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e60003970 msgr2=0x7f7e60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.240+0000 7f7e7f400640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e60003970 0x7f7e60003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.240+0000 7f7e7f400640 1 -- 172.21.15.141:0/2370791942 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e60003520 con 0x7f7e600047c0 2024-01-16T15:00:14.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.240+0000 7f7e84a00640 1 -- 172.21.15.141:0/2370791942 reap_dead start 2024-01-16T15:00:14.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.241+0000 7f7e7f400640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e600047c0 0x7f7e60004c40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7e68001750 tx=0x7f7e600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f681ba2423c40965 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.241+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680052d0 con 0x7f7e600047c0 2024-01-16T15:00:14.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.241+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6802e050 con 0x7f7e600047c0 2024-01-16T15:00:14.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.241+0000 7f7e7ea00640 1 -- 172.21.15.141:0/2370791942 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68003830 con 0x7f7e600047c0 2024-01-16T15:00:14.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.241+0000 7f7e85400640 1 -- 172.21.15.141:0/2370791942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e600047c0 msgr2=0x7f7e60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.242+0000 7f7e85400640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e600047c0 0x7f7e60004c40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7e68001750 tx=0x7f7e600057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.242+0000 7f7e85400640 1 -- 172.21.15.141:0/2370791942 shutdown_connections 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.242+0000 7f7e85400640 1 --2- 172.21.15.141:0/2370791942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e600047c0 0x7f7e60004c40 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.242+0000 7f7e85400640 1 -- 172.21.15.141:0/2370791942 >> 172.21.15.141:0/2370791942 conn(0x7f7e8006a800 msgr2=0x7f7e8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.242+0000 7f7e85400640 1 -- 172.21.15.141:0/2370791942 shutdown_connections 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.242+0000 7f7e85400640 1 -- 172.21.15.141:0/2370791942 wait complete. 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.244+0000 7f7e85400640 1 Processor -- start 2024-01-16T15:00:14.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.244+0000 7f7e85400640 1 -- start start 2024-01-16T15:00:14.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e85400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e600047c0 0x7f7e80158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e85400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e801e3830 0x7f7e8013b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e85400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e8013b560 0x7f7e80139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e85400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e801e3cb0 con 0x7f7e8013b560 2024-01-16T15:00:14.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e85400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8013b9e0 con 0x7f7e600047c0 2024-01-16T15:00:14.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e85400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e80158b60 con 0x7f7e801e3830 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e84a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e8013b560 0x7f7e80139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e84a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e8013b560 0x7f7e80139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49016/0 (socket says 172.21.15.141:49016) 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e84a00640 1 -- 172.21.15.141:0/1318177040 learned_addr learned my addr 172.21.15.141:0/1318177040 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e7f400640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e801e3830 0x7f7e8013b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.245+0000 7f7e7fe00640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e600047c0 0x7f7e80158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e84a00640 1 -- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e801e3830 msgr2=0x7f7e8013b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e84a00640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e801e3830 0x7f7e8013b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e84a00640 1 -- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e600047c0 msgr2=0x7f7e80158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e84a00640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e600047c0 0x7f7e80158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e84a00640 1 -- 172.21.15.141:0/1318177040 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e801e3ec0 con 0x7f7e8013b560 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e84a00640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e8013b560 0x7f7e80139720 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7e8006e270 tx=0x7f7e7800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e77e00640 1 -- 172.21.15.141:0/1318177040 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78010040 con 0x7f7e8013b560 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e77e00640 1 -- 172.21.15.141:0/1318177040 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7800bc80 con 0x7f7e8013b560 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e77e00640 1 -- 172.21.15.141:0/1318177040 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78014400 con 0x7f7e8013b560 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e801e41a0 con 0x7f7e8013b560 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.246+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e80154b30 con 0x7f7e8013b560 2024-01-16T15:00:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.248+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e80067450 con 0x7f7e8013b560 2024-01-16T15:00:14.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.248+0000 7f7e77e00640 1 -- 172.21.15.141:0/1318177040 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f7e78014560 con 0x7f7e8013b560 2024-01-16T15:00:14.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.249+0000 7f7e77e00640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5806b500 0x7f7e5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.249+0000 7f7e77e00640 1 -- 172.21.15.141:0/1318177040 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e7808a690 con 0x7f7e8013b560 2024-01-16T15:00:14.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.250+0000 7f7e7fe00640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5806b500 0x7f7e5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.251+0000 7f7e7fe00640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5806b500 0x7f7e5806d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e70001200 tx=0x7f7e70001530 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.254+0000 7f7e77e00640 1 -- 172.21.15.141:0/1318177040 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e780550c0 con 0x7f7e8013b560 2024-01-16T15:00:14.450 INFO:tasks.ceph.mds.c.smithi141.stderr:2024-01-16T15:00:14.469+0000 7fa1b5200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:00:14.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.481+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e8015a210 con 0x7f7e8013b560 2024-01-16T15:00:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.482+0000 7f7e77e00640 1 -- 172.21.15.141:0/1318177040 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f7e780586e0 con 0x7f7e8013b560 2024-01-16T15:00:14.464 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:00:14.464 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2118095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2118095418},{"type":"v1","addr":"172.21.15.188:6837","nonce":2118095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/3680808697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3680808697},{"type":"v1","addr":"172.21.15.188:6836","nonce":3680808697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5793,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/275409001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":275409001},{"type":"v1","addr":"172.21.15.141:6833","nonce":275409001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:00:07.944739+0000","modified":"2024-01-16T15:00:11.025175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5796":{"gid":5796,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/2933218811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":2933218811},{"type":"v1","addr":"172.21.15.141:6835","nonce":2933218811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T15:00:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 27 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5806b500 msgr2=0x7f7e5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5806b500 0x7f7e5806d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e70001200 tx=0x7f7e70001530 comp rx=0 tx=0).stop 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e8013b560 msgr2=0x7f7e80139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e8013b560 0x7f7e80139720 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7e8006e270 tx=0x7f7e7800b480 comp rx=0 tx=0).stop 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 shutdown_connections 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5806b500 0x7f7e5806d9c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e8013b560 0x7f7e80139720 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e801e3830 0x7f7e8013b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 --2- 172.21.15.141:0/1318177040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e600047c0 0x7f7e80158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.485+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 >> 172.21.15.141:0/1318177040 conn(0x7f7e8006a800 msgr2=0x7f7e80070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.486+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 shutdown_connections 2024-01-16T15:00:14.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:14.486+0000 7f7e85400640 1 -- 172.21.15.141:0/1318177040 wait complete. 2024-01-16T15:00:14.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:00:07.944739+0000', 'modified': '2024-01-16T15:00:11.025175+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5796': {'gid': 5796, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6835/2933218811', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6834', 'nonce': 2933218811}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 2933218811}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:00:14.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:00:14.889 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:00:15.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.125+0000 7f87e7e00640 1 Processor -- start 2024-01-16T15:00:15.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.125+0000 7f87e7e00640 1 -- start start 2024-01-16T15:00:15.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.127+0000 7f87e7e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f87e00d0b60 0x7f87e00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.127+0000 7f87e7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f87e00aede0 0x7f87e006f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.127+0000 7f87e7e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e0071de0 con 0x7f87e006a3e0 2024-01-16T15:00:15.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.127+0000 7f87e7e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e00af270 con 0x7f87e00aede0 2024-01-16T15:00:15.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.127+0000 7f87e7e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e0072030 con 0x7f87e00d0b60 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.127+0000 7f87e5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f87e00aede0 0x7f87e006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.128+0000 7f87e5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f87e00aede0 0x7f87e006f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49026/0 (socket says 172.21.15.141:49026) 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.128+0000 7f87e5e00640 1 -- 172.21.15.141:0/121796757 learned_addr learned my addr 172.21.15.141:0/121796757 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.128+0000 7f87e4a00640 1 --2- 172.21.15.141:0/121796757 >> v2:172.21.15.188:3300/0 conn(0x7f87e00d0b60 0x7f87e00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.128+0000 7f87e5e00640 1 -- 172.21.15.141:0/121796757 >> v2:172.21.15.188:3300/0 conn(0x7f87e00d0b60 msgr2=0x7f87e00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.128+0000 7f87e5e00640 1 --2- 172.21.15.141:0/121796757 >> v2:172.21.15.188:3300/0 conn(0x7f87e00d0b60 0x7f87e00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.128+0000 7f87e5e00640 1 -- 172.21.15.141:0/121796757 >> v1:172.21.15.188:6789/0 conn(0x7f87e006a3e0 legacy=0x7f87e006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.128+0000 7f87e5e00640 1 -- 172.21.15.141:0/121796757 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0072310 con 0x7f87e00aede0 2024-01-16T15:00:15.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.129+0000 7f87e5e00640 1 --2- 172.21.15.141:0/121796757 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e00aede0 0x7f87e006f080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f87e00b6120 tx=0x7f87d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fc7b956c3a9b97 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87dfe00640 1 -- 172.21.15.141:0/121796757 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8031870 con 0x7f87e00aede0 2024-01-16T15:00:15.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87dfe00640 1 -- 172.21.15.141:0/121796757 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d80319d0 con 0x7f87e00aede0 2024-01-16T15:00:15.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87dfe00640 1 -- 172.21.15.141:0/121796757 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8031d60 con 0x7f87e00aede0 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 -- 172.21.15.141:0/121796757 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e00aede0 msgr2=0x7f87e006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 --2- 172.21.15.141:0/121796757 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e00aede0 0x7f87e006f080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f87e00b6120 tx=0x7f87d802b910 comp rx=0 tx=0).stop 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 -- 172.21.15.141:0/121796757 shutdown_connections 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 --2- 172.21.15.141:0/121796757 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e00aede0 0x7f87e006f080 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 --2- 172.21.15.141:0/121796757 >> v2:172.21.15.188:3300/0 conn(0x7f87e00d0b60 0x7f87e00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 -- 172.21.15.141:0/121796757 >> 172.21.15.141:0/121796757 conn(0x7f87e00cfef0 msgr2=0x7f87e00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 -- 172.21.15.141:0/121796757 shutdown_connections 2024-01-16T15:00:15.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.130+0000 7f87e7e00640 1 -- 172.21.15.141:0/121796757 wait complete. 2024-01-16T15:00:15.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.132+0000 7f87e7e00640 1 Processor -- start 2024-01-16T15:00:15.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.132+0000 7f87e7e00640 1 -- start start 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.133+0000 7f87e7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e006a3e0 0x7f87e0149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.133+0000 7f87e7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87e00aede0 0x7f87e01498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.133+0000 7f87e7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87e00d0b60 0x7f87e0143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.133+0000 7f87e7e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00ce2d0 con 0x7f87e006a3e0 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.133+0000 7f87e7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00cd6c0 con 0x7f87e00aede0 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.133+0000 7f87e7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00cd9a0 con 0x7f87e00d0b60 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e006a3e0 0x7f87e0149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e006a3e0 0x7f87e0149360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49030/0 (socket says 172.21.15.141:49030) 2024-01-16T15:00:15.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 -- 172.21.15.141:0/2227517275 learned_addr learned my addr 172.21.15.141:0/2227517275 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e4a00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87e00aede0 0x7f87e01498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 -- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87e00d0b60 msgr2=0x7f87e0143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87e00d0b60 0x7f87e0143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 -- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87e00aede0 msgr2=0x7f87e01498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87e00aede0 0x7f87e01498a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 -- 172.21.15.141:0/2227517275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d80094d0 con 0x7f87e006a3e0 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.134+0000 7f87e5400640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e006a3e0 0x7f87e0149360 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f87c800b3f0 tx=0x7f87c800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.135+0000 7f87dd600640 1 -- 172.21.15.141:0/2227517275 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8011830 con 0x7f87e006a3e0 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.135+0000 7f87dd600640 1 -- 172.21.15.141:0/2227517275 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8011990 con 0x7f87e006a3e0 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.135+0000 7f87dd600640 1 -- 172.21.15.141:0/2227517275 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8011d40 con 0x7f87e006a3e0 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.135+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e00cdc80 con 0x7f87e006a3e0 2024-01-16T15:00:15.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.135+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e00cd350 con 0x7f87e006a3e0 2024-01-16T15:00:15.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.137+0000 7f87dd600640 1 -- 172.21.15.141:0/2227517275 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f87c801a400 con 0x7f87e006a3e0 2024-01-16T15:00:15.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.138+0000 7f87dd600640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87b806b500 0x7f87b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.138+0000 7f87dd600640 1 -- 172.21.15.141:0/2227517275 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87c8089ef0 con 0x7f87e006a3e0 2024-01-16T15:00:15.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.138+0000 7f87e4a00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87b806b500 0x7f87b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.139+0000 7f87ea9a9740 1 Processor -- start 2024-01-16T15:00:15.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.139+0000 7f87ea9a9740 1 -- start start 2024-01-16T15:00:15.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.139+0000 7f87e4a00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87b806b500 0x7f87b806d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f87e00add00 tx=0x7f87d000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa1d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1cbba0 0x555faa1d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1d05e0 0x555faa214560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x555faa12f880 con 0x555faa1d05e0 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555faa13eae0 con 0x555faa1d0ea0 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x555faa13c1d0 con 0x555faa1cbba0 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87e5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1d05e0 0x555faa214560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87e5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1d05e0 0x555faa214560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49042/0 (socket says 172.21.15.141:49042) 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87e5e00640 1 -- 172.21.15.141:0/3805729644 learned_addr learned my addr 172.21.15.141:0/3805729644 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87e5400640 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa1d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.140+0000 7f87e5e00640 1 -- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1cbba0 msgr2=0x555faa1d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87e4a00640 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1cbba0 0x555faa1d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87e5e00640 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1cbba0 0x555faa1d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87e5e00640 1 -- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 msgr2=0x555faa1d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87e5e00640 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa1d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87e5e00640 1 -- 172.21.15.141:0/3805729644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555faa0ff860 con 0x555faa1d05e0 2024-01-16T15:00:15.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87e5400640 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa1d1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:15.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87e5e00640 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1d05e0 0x555faa214560 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f87d802bf20 tx=0x7f87d802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118890537354c646 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87d6000640 1 -- 172.21.15.141:0/3805729644 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8031b40 con 0x555faa1d05e0 2024-01-16T15:00:15.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87d6000640 1 -- 172.21.15.141:0/3805729644 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8031ca0 con 0x555faa1d05e0 2024-01-16T15:00:15.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.141+0000 7f87d6000640 1 -- 172.21.15.141:0/3805729644 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d803a640 con 0x555faa1d05e0 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3805729644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1d05e0 msgr2=0x555faa214560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1d05e0 0x555faa214560 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f87d802bf20 tx=0x7f87d802e000 comp rx=0 tx=0).stop 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3805729644 shutdown_connections 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1d05e0 0x555faa214560 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1cbba0 0x555faa1d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3805729644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa1d1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3805729644 >> 172.21.15.141:0/3805729644 conn(0x555faa192c70 msgr2=0x555faa216990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3805729644 shutdown_connections 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.142+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3805729644 wait complete. 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.143+0000 7f87ea9a9740 1 -- start start 2024-01-16T15:00:15.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.143+0000 7f87ea9a9740 1 Processor -- start 2024-01-16T15:00:15.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1fc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d05e0 0x555faa21e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa214d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x555faa137ac0 con 0x555faa1cbba0 2024-01-16T15:00:15.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555faa0d42f0 con 0x555faa1d0ea0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x555faa0f1c70 con 0x555faa1d05e0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87ea9a9740 10 client.0 _ll_register_callbacks cb 0x555faa0f63b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1fc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49052/0 (socket says 172.21.15.141:49052) 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.144+0000 7f87e5400640 1 -- 172.21.15.141:0/4103924231 learned_addr learned my addr 172.21.15.141:0/4103924231 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e5400640 1 -- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d05e0 msgr2=0x555faa21e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e5e00640 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa214d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e4a00640 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d05e0 0x555faa21e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e5400640 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d05e0 0x555faa21e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e5400640 1 -- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 msgr2=0x555faa214d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e5400640 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa214d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e5400640 1 -- 172.21.15.141:0/4103924231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e0072310 con 0x555faa1cbba0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87e5400640 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1fc670 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f87c80585b0 tx=0x7f87c8058760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28583d58d0ea074 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87cea00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87cea00640 1 -- 172.21.15.141:0/4103924231 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8060690 con 0x555faa1cbba0 2024-01-16T15:00:15.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87cea00640 1 -- 172.21.15.141:0/4103924231 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87c8088250 con 0x555faa1cbba0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555faa130aa0 con 0x555faa1cbba0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87cea00640 1 -- 172.21.15.141:0/4103924231 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87c8088410 con 0x555faa1cbba0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87ea9a9740 10 client.5940 fetch_fsmap learned FSMap version 27 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.145+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x555faa136140 con 0x555faa1cbba0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87cea00640 1 -- 172.21.15.141:0/4103924231 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555faa130aa0 con 0x555faa1cbba0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87cea00640 1 -- 172.21.15.141:0/4103924231 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f87c8074080 con 0x555faa1cbba0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87ea9a9740 10 client.5940 fetch_fsmap finished waiting for FSMap version 27 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87ea9a9740 10 client.5940 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87ea9a9740 20 client.5940 populate_metadata read hostname 'smithi141' 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21c0b0 0x555faa21c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87ea9a9740 4 client.5940 mds_command: new command op to 5796 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 --> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555faa1307c0 con 0x555faa21c0b0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.146+0000 7f87e4a00640 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21c0b0 0x555faa21c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.147+0000 7f87cea00640 10 client.5940 ms_handle_connect on v2:172.21.15.141:6834/2933218811 2024-01-16T15:00:15.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.147+0000 7f87e4a00640 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21c0b0 0x555faa21c490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=687b0d3cab597c67 server_cookie=a387d0c8c51ba863 in_seq=0 out_seq=0 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.147+0000 7f87cea00640 1 -- 172.21.15.141:0/4103924231 <== mds.0 v2:172.21.15.141:6834/2933218811 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555faa1307c0 con 0x555faa21c0b0 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.147+0000 7f87cea00640 10 client.5940 handle_command_reply: tid=0 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 client.5940 shutdown 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 msgr2=0x555faa1fc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1fc670 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f87c80585b0 tx=0x7f87c8058760 comp rx=0 tx=0).stop 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 shutdown_connections 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21c0b0 0x555faa21c490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d0ea0 0x555faa214d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d05e0 0x555faa21e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/4103924231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1fc670 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87cea00640 0 client.5940 ms_handle_reset on v2:172.21.15.141:6834/2933218811 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 >> 172.21.15.141:0/4103924231 conn(0x555faa192c70 msgr2=0x555faa1cab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 shutdown_connections 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 1 -- 172.21.15.141:0/4103924231 wait complete. 2024-01-16T15:00:15.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.148+0000 7f87ea9a9740 20 client.5940 trim_cache size 0 max 16384 2024-01-16T15:00:15.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.165+0000 7f87ea9a9740 1 Processor -- start 2024-01-16T15:00:15.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.165+0000 7f87ea9a9740 1 -- start start 2024-01-16T15:00:15.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa1f8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 0x555faa214b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fa9e9af20 con 0x555faa1cbba0 2024-01-16T15:00:15.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555faa1cba30 con 0x555faa1d05e0 2024-01-16T15:00:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x555faa2172c0 con 0x555faa1d0ea0 2024-01-16T15:00:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49062/0 (socket says 172.21.15.141:49062) 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.166+0000 7f87e5400640 1 -- 172.21.15.141:0/3830931290 learned_addr learned my addr 172.21.15.141:0/3830931290 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e4a00640 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa1f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e5e00640 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 0x555faa214b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e5400640 1 -- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 msgr2=0x555faa214b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e5400640 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 0x555faa214b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e5400640 1 -- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 msgr2=0x555faa1f8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e5400640 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa1f8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e5400640 1 -- 172.21.15.141:0/3830931290 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555faa0d3b80 con 0x555faa1cbba0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e4a00640 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa1f8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87e5400640 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1faef0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f87c80603f0 tx=0x7f87c805a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f8bf7d2fe641a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87d6a00640 1 -- 172.21.15.141:0/3830931290 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8060690 con 0x555faa1cbba0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87d6a00640 1 -- 172.21.15.141:0/3830931290 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8071070 con 0x555faa1cbba0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.167+0000 7f87d6a00640 1 -- 172.21.15.141:0/3830931290 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8088470 con 0x555faa1cbba0 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3830931290 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 msgr2=0x555faa1faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1faef0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f87c80603f0 tx=0x7f87c805a000 comp rx=0 tx=0).stop 2024-01-16T15:00:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3830931290 shutdown_connections 2024-01-16T15:00:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 0x555faa214b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa1f8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3830931290 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa1faef0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3830931290 >> 172.21.15.141:0/3830931290 conn(0x555faa192c70 msgr2=0x555faa1fb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3830931290 shutdown_connections 2024-01-16T15:00:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.168+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3830931290 wait complete. 2024-01-16T15:00:15.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.169+0000 7f87ea9a9740 1 -- start start 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.169+0000 7f87ea9a9740 1 Processor -- start 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa229000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa2803e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87ea9a9740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 0x555faa1cab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x555faa229540 con 0x555faa1cbba0 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555faa1f7860 con 0x555faa1d05e0 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87ea9a9740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x555faa226360 con 0x555faa1d0ea0 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87ea9a9740 10 client.0 _ll_register_callbacks cb 0x7f87c80583f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa229000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa229000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49072/0 (socket says 172.21.15.141:49072) 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 -- 172.21.15.141:0/3479396834 learned_addr learned my addr 172.21.15.141:0/3479396834 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 -- 172.21.15.141:0/3479396834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 msgr2=0x555faa1cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 0x555faa1cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 -- 172.21.15.141:0/3479396834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 msgr2=0x555faa2803e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa2803e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.170+0000 7f87e5400640 1 -- 172.21.15.141:0/3479396834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555faa136140 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87e5400640 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa229000 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f87c8058320 tx=0x7f87c80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2804c60582b6ba70 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87cea00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87cea00640 1 -- 172.21.15.141:0/3479396834 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8054960 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87cea00640 1 -- 172.21.15.141:0/3479396834 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87c80544b0 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f87d8031ca0 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87cea00640 1 -- 172.21.15.141:0/3479396834 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87c8060920 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87ea9a9740 10 client.5946 fetch_fsmap learned FSMap version 27 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x555faa0f9170 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87cea00640 1 -- 172.21.15.141:0/3479396834 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8031ca0 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.171+0000 7f87cea00640 1 -- 172.21.15.141:0/3479396834 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f87c8074070 con 0x555faa1cbba0 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87ea9a9740 10 client.5946 fetch_fsmap finished waiting for FSMap version 27 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87ea9a9740 10 client.5946 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:00:15.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87ea9a9740 20 client.5946 populate_metadata read hostname 'smithi141' 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21dbf0 0x555faa21dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87ea9a9740 4 client.5946 mds_command: new command op to 5796 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 --> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555fa9e386c0 con 0x555faa21dbf0 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87e4a00640 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21dbf0 0x555faa21dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87cea00640 10 client.5946 ms_handle_connect on v2:172.21.15.141:6834/2933218811 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.172+0000 7f87e4a00640 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21dbf0 0x555faa21dfd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=642fc55767cc3f28 server_cookie=8c2968eeb4acbbc1 in_seq=0 out_seq=0 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87cea00640 1 -- 172.21.15.141:0/3479396834 <== mds.0 v2:172.21.15.141:6834/2933218811 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x555fa9e386c0 con 0x555faa21dbf0 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87cea00640 10 client.5946 handle_command_reply: tid=0 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 client.5946 shutdown 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 msgr2=0x555faa229000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa229000 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f87c8058320 tx=0x7f87c80046d0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 shutdown_connections 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:6834/2933218811,v1:172.21.15.141:6835/2933218811] conn(0x555faa21dbf0 0x555faa21dfd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x555faa1d0ea0 0x555faa1cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555faa1d05e0 0x555faa2803e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87cea00640 0 client.5946 ms_handle_reset on v2:172.21.15.141:6834/2933218811 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 --2- 172.21.15.141:0/3479396834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x555faa1cbba0 0x555faa229000 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 >> 172.21.15.141:0/3479396834 conn(0x555faa192c70 msgr2=0x555faa1fb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 shutdown_connections 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 1 -- 172.21.15.141:0/3479396834 wait complete. 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.173+0000 7f87ea9a9740 20 client.5946 trim_cache size 0 max 16384 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T15:00:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5796, 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 6.6372631010000003, 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 27, 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 46, 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 46, 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 12.069419622 2024-01-16T15:00:15.159 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87b806b500 msgr2=0x7f87b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87b806b500 0x7f87b806d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f87e00add00 tx=0x7f87d000b040 comp rx=0 tx=0).stop 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e006a3e0 msgr2=0x7f87e0149360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e006a3e0 0x7f87e0149360 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f87c800b3f0 tx=0x7f87c800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 shutdown_connections 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87b806b500 0x7f87b806d9c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87e00d0b60 0x7f87e0143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87e00aede0 0x7f87e01498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 --2- 172.21.15.141:0/2227517275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87e006a3e0 0x7f87e0149360 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 >> 172.21.15.141:0/2227517275 conn(0x7f87e00cfef0 msgr2=0x7f87e0151990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 shutdown_connections 2024-01-16T15:00:15.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.174+0000 7f87e7e00640 1 -- 172.21.15.141:0/2227517275 wait complete. 2024-01-16T15:00:15.172 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:15.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.394+0000 7fd7ae000640 1 Processor -- start 2024-01-16T15:00:15.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.395+0000 7fd7ae000640 1 -- start start 2024-01-16T15:00:15.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7ae000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd7a813df80 0x7fd7a815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 0x7fd7a80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7ae000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a806c470 con 0x7fd7a815f9c0 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7ae000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a806c750 con 0x7fd7a813df80 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7ae000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a806ca30 con 0x7fd7a813e360 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7a7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd7a813df80 0x7fd7a815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7a7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd7a815f9c0 0x7fd7a81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58176/0 (socket says 172.21.15.141:58176) 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.396+0000 7fd7a7e00640 1 -- 172.21.15.141:0/221567531 learned_addr learned my addr 172.21.15.141:0/221567531 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:15.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.397+0000 7fd7a7400640 1 -- 172.21.15.141:0/221567531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 msgr2=0x7fd7a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.397+0000 7fd7a7400640 1 --2- 172.21.15.141:0/221567531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 0x7fd7a80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.397+0000 7fd7a7400640 1 -- 172.21.15.141:0/221567531 >> v1:172.21.15.141:6789/0 conn(0x7fd7a815f9c0 legacy=0x7fd7a81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.397+0000 7fd7a7400640 1 -- 172.21.15.141:0/221567531 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a806cd10 con 0x7fd7a813df80 2024-01-16T15:00:15.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.398+0000 7fd7a7400640 1 --2- 172.21.15.141:0/221567531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a815bee0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd7980097a0 tx=0x7fd79802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394f18c80a8c3fe5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.398+0000 7fd7a6a00640 1 -- 172.21.15.141:0/221567531 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79802d040 con 0x7fd7a813df80 2024-01-16T15:00:15.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.398+0000 7fd7a6a00640 1 -- 172.21.15.141:0/221567531 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd798031810 con 0x7fd7a813df80 2024-01-16T15:00:15.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.398+0000 7fd7a6a00640 1 -- 172.21.15.141:0/221567531 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798031ba0 con 0x7fd7a813df80 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 -- 172.21.15.141:0/221567531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 msgr2=0x7fd7a815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 --2- 172.21.15.141:0/221567531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a815bee0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd7980097a0 tx=0x7fd79802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 -- 172.21.15.141:0/221567531 shutdown_connections 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 --2- 172.21.15.141:0/221567531 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 0x7fd7a80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 --2- 172.21.15.141:0/221567531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a815bee0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 -- 172.21.15.141:0/221567531 >> 172.21.15.141:0/221567531 conn(0x7fd7a806a800 msgr2=0x7fd7a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 -- 172.21.15.141:0/221567531 shutdown_connections 2024-01-16T15:00:15.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.399+0000 7fd7ae000640 1 -- 172.21.15.141:0/221567531 wait complete. 2024-01-16T15:00:15.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.401+0000 7fd7ae000640 1 Processor -- start 2024-01-16T15:00:15.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.401+0000 7fd7ae000640 1 -- start start 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a814da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 0x7fd7a814fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7a815f9c0 0x7fd7a813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7ae000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8150f40 con 0x7fd7a813df80 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7ae000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8151360 con 0x7fd7a815f9c0 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7ae000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a814ad40 con 0x7fd7a813e360 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a814da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a814da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49090/0 (socket says 172.21.15.141:49090) 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 -- 172.21.15.141:0/2395692135 learned_addr learned my addr 172.21.15.141:0/2395692135 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 -- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 msgr2=0x7fd7a814fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 0x7fd7a814fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 -- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7a815f9c0 msgr2=0x7fd7a813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7a815f9c0 0x7fd7a813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.402+0000 7fd7a7400640 1 -- 172.21.15.141:0/2395692135 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798009400 con 0x7fd7a813df80 2024-01-16T15:00:15.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.403+0000 7fd7a7400640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a814da10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd798000c00 tx=0x7fd7980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.403+0000 7fd79fe00640 1 -- 172.21.15.141:0/2395692135 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79803ad60 con 0x7fd7a813df80 2024-01-16T15:00:15.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.403+0000 7fd79fe00640 1 -- 172.21.15.141:0/2395692135 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7980443f0 con 0x7fd7a813df80 2024-01-16T15:00:15.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.403+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a81502c0 con 0x7fd7a813df80 2024-01-16T15:00:15.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.403+0000 7fd79fe00640 1 -- 172.21.15.141:0/2395692135 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7980447a0 con 0x7fd7a813df80 2024-01-16T15:00:15.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.403+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a8150880 con 0x7fd7a813df80 2024-01-16T15:00:15.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.404+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a80675d0 con 0x7fd7a813df80 2024-01-16T15:00:15.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.405+0000 7fd79fe00640 1 -- 172.21.15.141:0/2395692135 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd79804d450 con 0x7fd7a813df80 2024-01-16T15:00:15.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.407+0000 7fd79fe00640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd78006b500 0x7fd78006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.407+0000 7fd79fe00640 1 -- 172.21.15.141:0/2395692135 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7980ac460 con 0x7fd7a813df80 2024-01-16T15:00:15.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.407+0000 7fd79e800640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd78006b500 0x7fd78006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.408+0000 7fd79e800640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd78006b500 0x7fd78006d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd7a814aab0 tx=0x7fd790006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.411+0000 7fd79fe00640 1 -- 172.21.15.141:0/2395692135 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd798076e50 con 0x7fd7a813df80 2024-01-16T15:00:15.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.644+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd7a815b0f0 con 0x7fd7a813df80 2024-01-16T15:00:15.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.645+0000 7fd79fe00640 1 -- 172.21.15.141:0/2395692135 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd79807a470 con 0x7fd7a813df80 2024-01-16T15:00:15.626 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.647+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd78006b500 msgr2=0x7fd78006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.647+0000 7fd7ae000640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd78006b500 0x7fd78006d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd7a814aab0 tx=0x7fd790006c90 comp rx=0 tx=0).stop 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.647+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 msgr2=0x7fd7a814da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.647+0000 7fd7ae000640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a814da10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd798000c00 tx=0x7fd7980046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 shutdown_connections 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd78006b500 0x7fd78006d9c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7a815f9c0 0x7fd7a813d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7a813e360 0x7fd7a814fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 --2- 172.21.15.141:0/2395692135 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7a813df80 0x7fd7a814da10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 >> 172.21.15.141:0/2395692135 conn(0x7fd7a806a800 msgr2=0x7fd7a815d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 shutdown_connections 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:15.648+0000 7fd7ae000640 1 -- 172.21.15.141:0/2395692135 wait complete. 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:15.649 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:15.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.650 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T15:00:15.697 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:15.698 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42448sec preferred_lft 42448sec 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T15:00:15.699 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:15.700 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:15.700 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:15.700 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::882:e9ff:fe92:c54c/64 scope link 2024-01-16T15:00:15.700 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:15.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.701 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:15.782 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T15:00:15.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.786 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T15:00:15.835 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T15:00:15.835 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:15.836 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:00:15.836 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:00:15.836 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:15.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.836 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T15:00:15.836 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T15:00:15.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:00:15.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:00:15.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:00:15.837 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T15:00:16.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.005 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T15:00:16.055 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:00:16.056 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:00:16.057 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:16.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.057 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T15:00:16.149 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:00:16.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:16.149 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:16.203 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:16.204 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:16.204 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:16.253 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:00:16.343 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:00:16.400 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:16.401 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:00:16.402 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:16.403 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:16.403 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:16.403 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:16.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.404 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:00:16.456 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:16.457 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:00:16.462 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:00:16.537 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:16.578 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:00:16.579 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:16.580 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:16.581 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:00:16.581 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:16.581 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:16.581 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:16.581 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:16.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.582 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:00:16.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T15:00:16.665+0000 7f62dee38480 -1 init, newargv = 0x560fe7d2f900 newargc=15 2024-01-16T15:00:16.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[52836]: starting ceph client 2024-01-16T15:00:16.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[52836]: starting fuse 2024-01-16T15:00:16.665 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T15:00:16.666 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:00:16.666 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T15:00:16.667 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T15:00:16.668 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:16.669 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:00:16.669 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T15:00:16.669 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T15:00:16.857 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.52836.asok 2024-01-16T15:00:16.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:16.874 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52836.asok 2024-01-16T15:00:16.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.874 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52836.asok status 2024-01-16T15:00:17.107 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:00:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T15:00:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T15:00:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T15:00:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "52836", 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5964, 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 5964 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:00:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 1850929204 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 1850929204 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.5964 192.168.0.1:0/1850929204", 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/1850929204", 2024-01-16T15:00:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T15:00:17.203 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 27, 2024-01-16T15:00:17.203 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 46, 2024-01-16T15:00:17.203 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:17.203 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T15:00:17.203 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T15:00:17.203 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:00:17.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.204 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T15:00:17.205 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:17.258 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T15:00:17.258 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:17.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.258 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:17.352 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:17.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.580+0000 7fc1a0e00640 1 Processor -- start 2024-01-16T15:00:17.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.580+0000 7fc1a0e00640 1 -- start start 2024-01-16T15:00:17.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.581+0000 7fc1a0e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.581+0000 7fc1a0e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc19c0d0640 0x7fc19c166f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.581+0000 7fc1a0e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0b1430 con 0x7fc19c0cd820 2024-01-16T15:00:17.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.581+0000 7fc1a0e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19c0d0c40 con 0x7fc19c0d0640 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.581+0000 7fc1a0e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0d0f20 con 0x7fc19c0ce310 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc19be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc19c0d0640 0x7fc19c166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc19be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc19c0d0640 0x7fc19c166f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49100/0 (socket says 172.21.15.141:49100) 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc19be00640 1 -- 172.21.15.141:0/406945196 learned_addr learned my addr 172.21.15.141:0/406945196 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc193e00640 1 --2- 172.21.15.141:0/406945196 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc19be00640 1 -- 172.21.15.141:0/406945196 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 msgr2=0x7fc19c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc19be00640 1 --2- 172.21.15.141:0/406945196 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc19be00640 1 -- 172.21.15.141:0/406945196 >> v1:172.21.15.188:6789/0 conn(0x7fc19c0cd820 legacy=0x7fc19c0cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.582+0000 7fc19be00640 1 -- 172.21.15.141:0/406945196 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0d1200 con 0x7fc19c0d0640 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.583+0000 7fc19be00640 1 --2- 172.21.15.141:0/406945196 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c166f20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc194009e50 tx=0x7fc19402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e2136cb74acf12 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc19aa00640 1 -- 172.21.15.141:0/406945196 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19402ca70 con 0x7fc19c0d0640 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc19aa00640 1 -- 172.21.15.141:0/406945196 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19402cbd0 con 0x7fc19c0d0640 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc19aa00640 1 -- 172.21.15.141:0/406945196 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1940343f0 con 0x7fc19c0d0640 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 -- 172.21.15.141:0/406945196 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 msgr2=0x7fc19c166f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/406945196 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c166f20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc194009e50 tx=0x7fc19402bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 -- 172.21.15.141:0/406945196 shutdown_connections 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/406945196 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c166f20 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/406945196 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 -- 172.21.15.141:0/406945196 >> 172.21.15.141:0/406945196 conn(0x7fc19c099a60 msgr2=0x7fc19c099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 -- 172.21.15.141:0/406945196 shutdown_connections 2024-01-16T15:00:17.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.584+0000 7fc1a0e00640 1 -- 172.21.15.141:0/406945196 wait complete. 2024-01-16T15:00:17.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.586+0000 7fc1a0e00640 1 Processor -- start 2024-01-16T15:00:17.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.586+0000 7fc1a0e00640 1 -- start start 2024-01-16T15:00:17.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.587+0000 7fc1a0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19c0cd820 0x7fc19c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.587+0000 7fc1a0e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.587+0000 7fc1a0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c0cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.587+0000 7fc1a0e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c15d940 con 0x7fc19c0d0640 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.587+0000 7fc1a0e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0ccc70 con 0x7fc19c0cd820 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.587+0000 7fc1a0e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0cbc50 con 0x7fc19c0ce310 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c0cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c0cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49116/0 (socket says 172.21.15.141:49116) 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 -- 172.21.15.141:0/1986636376 learned_addr learned my addr 172.21.15.141:0/1986636376 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc193e00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19b400640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19c0cd820 0x7fc19c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 -- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 msgr2=0x7fc19c0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 -- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19c0cd820 msgr2=0x7fc19c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19c0cd820 0x7fc19c15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 -- 172.21.15.141:0/1986636376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194009b30 con 0x7fc19c0d0640 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19b400640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19c0cd820 0x7fc19c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:17.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.588+0000 7fc19be00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c0cb610 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc194036040 tx=0x7fc194004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.589+0000 7fc193400640 1 -- 172.21.15.141:0/1986636376 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194009510 con 0x7fc19c0d0640 2024-01-16T15:00:17.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.589+0000 7fc193400640 1 -- 172.21.15.141:0/1986636376 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19403e450 con 0x7fc19c0d0640 2024-01-16T15:00:17.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.589+0000 7fc193400640 1 -- 172.21.15.141:0/1986636376 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19403e7e0 con 0x7fc19c0d0640 2024-01-16T15:00:17.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.589+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c0cbf30 con 0x7fc19c0d0640 2024-01-16T15:00:17.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.589+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c15de70 con 0x7fc19c0d0640 2024-01-16T15:00:17.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.590+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c0b7020 con 0x7fc19c0d0640 2024-01-16T15:00:17.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.591+0000 7fc193400640 1 -- 172.21.15.141:0/1986636376 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc19403e940 con 0x7fc19c0d0640 2024-01-16T15:00:17.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.592+0000 7fc193400640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc17406b500 0x7fc17406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.592+0000 7fc193400640 1 -- 172.21.15.141:0/1986636376 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1940acdb0 con 0x7fc19c0d0640 2024-01-16T15:00:17.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.592+0000 7fc19b400640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc17406b500 0x7fc17406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.594+0000 7fc19b400640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc17406b500 0x7fc17406d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00a830 tx=0x7fc18c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.597+0000 7fc193400640 1 -- 172.21.15.141:0/1986636376 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1940b1070 con 0x7fc19c0d0640 2024-01-16T15:00:17.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.825+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc19c15d520 con 0x7fc19c0d0640 2024-01-16T15:00:17.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.826+0000 7fc193400640 1 -- 172.21.15.141:0/1986636376 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc194043070 con 0x7fc19c0d0640 2024-01-16T15:00:17.807 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:17.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc17406b500 msgr2=0x7fc17406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc17406b500 0x7fc17406d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00a830 tx=0x7fc18c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:17.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 msgr2=0x7fc19c0cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c0cb610 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc194036040 tx=0x7fc194004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 shutdown_connections 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc17406b500 0x7fc17406d9c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19c0d0640 0x7fc19c0cb610 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19c0ce310 0x7fc19c0cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 --2- 172.21.15.141:0/1986636376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19c0cd820 0x7fc19c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 >> 172.21.15.141:0/1986636376 conn(0x7fc19c099a60 msgr2=0x7fc19c15cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:17.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.828+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 shutdown_connections 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:17.829+0000 7fc1a0e00640 1 -- 172.21.15.141:0/1986636376 wait complete. 2024-01-16T15:00:17.829 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:17.829 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:17.829 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T15:00:17.830 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:00:17.830 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:17.830 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:17.830 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:17.830 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:17.830 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:17.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.830 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T15:00:17.889 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:17.889 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:17.889 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42447sec preferred_lft 42447sec 2024-01-16T15:00:17.890 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::a40a:23ff:fed8:ea96/64 scope link 2024-01-16T15:00:17.891 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:17.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.892 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:17.986 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T15:00:17.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.990 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T15:00:18.041 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T15:00:18.041 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:18.042 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:00:18.042 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:00:18.042 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:18.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.043 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T15:00:18.043 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T15:00:18.043 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:00:18.043 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:00:18.043 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:00:18.043 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T15:00:18.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.217 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T15:00:18.265 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:00:18.265 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:00:18.266 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:18.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.266 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T15:00:18.356 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:00:18.357 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:18.357 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:18.406 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:18.407 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:18.407 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:18.455 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:00:18.547 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:00:18.602 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:18.603 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:18.604 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:18.604 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:18.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.604 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:00:18.654 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:18.654 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:00:18.658 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:00:18.730 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:00:18.774 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:18.775 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:18.776 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:18.776 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:18.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.777 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:00:18.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T15:00:18.864+0000 7f5f8e80b480 -1 init, newargv = 0x561598fc2860 newargc=15 2024-01-16T15:00:18.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[43380]: starting ceph client 2024-01-16T15:00:18.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[43380]: starting fuse 2024-01-16T15:00:18.860 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T15:00:18.861 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:00:18.861 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:18.862 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:00:18.863 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T15:00:18.863 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T15:00:19.051 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.43380.asok 2024-01-16T15:00:19.056 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T15:00:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:19.068 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43380.asok 2024-01-16T15:00:19.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:19.069 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43380.asok status 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:00:19.302 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "43380", 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 5107, 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 5107 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 745677825 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T15:00:19.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 745677825 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.5107 192.168.0.1:0/745677825", 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/745677825", 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 27, 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 46, 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T15:00:19.304 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T15:00:19.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:19.317 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T15:00:19.317 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:19.366 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T15:00:19.366 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:19.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:19.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:19.463 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:00:19.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.707+0000 7f3c86c00640 1 Processor -- start 2024-01-16T15:00:19.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.707+0000 7f3c86c00640 1 -- start start 2024-01-16T15:00:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c86c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3c8013df80 0x7f3c8015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c86c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3c8013e360 0x7f3c800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c86c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8006c470 con 0x7f3c8015f9c0 2024-01-16T15:00:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c86c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8006c750 con 0x7f3c8013df80 2024-01-16T15:00:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c86c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8006ca30 con 0x7f3c8013e360 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c7fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3c8013df80 0x7f3c8015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c84c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3c8015f9c0 0x7f3c801408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58178/0 (socket says 172.21.15.141:58178) 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c84c00640 1 -- 172.21.15.141:0/3020693890 learned_addr learned my addr 172.21.15.141:0/3020693890 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c7f400640 1 --2- 172.21.15.141:0/3020693890 >> v2:172.21.15.188:3300/0 conn(0x7f3c8013e360 0x7f3c800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c7fe00640 1 -- 172.21.15.141:0/3020693890 >> v2:172.21.15.188:3300/0 conn(0x7f3c8013e360 msgr2=0x7f3c800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c7fe00640 1 --2- 172.21.15.141:0/3020693890 >> v2:172.21.15.188:3300/0 conn(0x7f3c8013e360 0x7f3c800cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c7fe00640 1 -- 172.21.15.141:0/3020693890 >> v1:172.21.15.141:6789/0 conn(0x7f3c8015f9c0 legacy=0x7f3c801408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c7fe00640 1 -- 172.21.15.141:0/3020693890 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8006cd10 con 0x7f3c8013df80 2024-01-16T15:00:19.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.709+0000 7f3c7ea00640 1 -- 172.21.15.141:0/3020693890 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885598447 0 0) 0x7f3c8006c470 con 0x7f3c8015f9c0 2024-01-16T15:00:19.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.710+0000 7f3c7fe00640 1 --2- 172.21.15.141:0/3020693890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c8015bee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3c8006ded0 tx=0x7f3c7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b4753edc2b3755 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.710+0000 7f3c7ea00640 1 -- 172.21.15.141:0/3020693890 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70031830 con 0x7f3c8013df80 2024-01-16T15:00:19.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.710+0000 7f3c7ea00640 1 -- 172.21.15.141:0/3020693890 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70031990 con 0x7f3c8013df80 2024-01-16T15:00:19.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.710+0000 7f3c7ea00640 1 -- 172.21.15.141:0/3020693890 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70031d20 con 0x7f3c8013df80 2024-01-16T15:00:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 -- 172.21.15.141:0/3020693890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 msgr2=0x7f3c8015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3020693890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c8015bee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3c8006ded0 tx=0x7f3c7002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 -- 172.21.15.141:0/3020693890 shutdown_connections 2024-01-16T15:00:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3020693890 >> v2:172.21.15.188:3300/0 conn(0x7f3c8013e360 0x7f3c800cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3020693890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c8015bee0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 -- 172.21.15.141:0/3020693890 >> 172.21.15.141:0/3020693890 conn(0x7f3c8006a800 msgr2=0x7f3c8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:19.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 -- 172.21.15.141:0/3020693890 shutdown_connections 2024-01-16T15:00:19.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.711+0000 7f3c86c00640 1 -- 172.21.15.141:0/3020693890 wait complete. 2024-01-16T15:00:19.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.712+0000 7f3c86c00640 1 Processor -- start 2024-01-16T15:00:19.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.713+0000 7f3c86c00640 1 -- start start 2024-01-16T15:00:19.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c86c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c80139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c86c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c8013e360 0x7f3c800b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c86c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c8015f9c0 0x7f3c800b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c86c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8014c2d0 con 0x7f3c8013df80 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c86c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8014ef90 con 0x7f3c8013e360 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c86c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8014ecb0 con 0x7f3c8015f9c0 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c80139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c80139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49134/0 (socket says 172.21.15.141:49134) 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 -- 172.21.15.141:0/3590460428 learned_addr learned my addr 172.21.15.141:0/3590460428 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 -- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c8015f9c0 msgr2=0x7f3c800b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c8015f9c0 0x7f3c800b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 -- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c8013e360 msgr2=0x7f3c800b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c8013e360 0x7f3c800b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.714+0000 7f3c7fe00640 1 -- 172.21.15.141:0/3590460428 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70009400 con 0x7f3c8013df80 2024-01-16T15:00:19.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.715+0000 7f3c7fe00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c80139f40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3c70000c00 tx=0x7f3c700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.715+0000 7f3c77e00640 1 -- 172.21.15.141:0/3590460428 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7003a9e0 con 0x7f3c8013df80 2024-01-16T15:00:19.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.715+0000 7f3c77e00640 1 -- 172.21.15.141:0/3590460428 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7003ab60 con 0x7f3c8013df80 2024-01-16T15:00:19.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.715+0000 7f3c77e00640 1 -- 172.21.15.141:0/3590460428 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c700443f0 con 0x7f3c8013df80 2024-01-16T15:00:19.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.715+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8014bc80 con 0x7f3c8013df80 2024-01-16T15:00:19.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.715+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8013a480 con 0x7f3c8013df80 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.716+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8014bae0 con 0x7f3c8013df80 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.717+0000 7f3c77e00640 1 -- 172.21.15.141:0/3590460428 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f3c7004d450 con 0x7f3c8013df80 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.718+0000 7f3c77e00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c5806b500 0x7f3c5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.718+0000 7f3c77e00640 1 -- 172.21.15.141:0/3590460428 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c700ac330 con 0x7f3c8013df80 2024-01-16T15:00:19.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.719+0000 7f3c7f400640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c5806b500 0x7f3c5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.720+0000 7f3c7f400640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c5806b500 0x7f3c5806d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3c80159bb0 tx=0x7f3c68008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.722+0000 7f3c77e00640 1 -- 172.21.15.141:0/3590460428 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c8014bae0 con 0x7f3c8013df80 2024-01-16T15:00:19.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:19.952+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3c8014bae0 con 0x7f3c8013df80 2024-01-16T15:00:20.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.605+0000 7f3c77e00640 1 -- 172.21.15.141:0/3590460428 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3c8014bae0 con 0x7f3c8013df80 2024-01-16T15:00:20.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c5806b500 msgr2=0x7f3c5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c5806b500 0x7f3c5806d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3c80159bb0 tx=0x7f3c68008040 comp rx=0 tx=0).stop 2024-01-16T15:00:20.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 msgr2=0x7f3c80139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c80139f40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3c70000c00 tx=0x7f3c700046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 shutdown_connections 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c5806b500 0x7f3c5806d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c8015f9c0 0x7f3c800b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.608+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c8013e360 0x7f3c800b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.609+0000 7f3c86c00640 1 --2- 172.21.15.141:0/3590460428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c8013df80 0x7f3c80139f40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.609+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 >> 172.21.15.141:0/3590460428 conn(0x7f3c8006a800 msgr2=0x7f3c800b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.609+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 shutdown_connections 2024-01-16T15:00:20.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.609+0000 7f3c86c00640 1 -- 172.21.15.141:0/3590460428 wait complete. 2024-01-16T15:00:20.612 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:00:20.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.842+0000 7ff110e00640 1 Processor -- start 2024-01-16T15:00:20.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.842+0000 7ff110e00640 1 -- start start 2024-01-16T15:00:20.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.843+0000 7ff110e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff10c0b3ae0 0x7ff10c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.843+0000 7ff110e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff10c0cf290 0x7ff10c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff110e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10c070c40 con 0x7ff10c0ae5a0 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff110e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10c071450 con 0x7ff10c0cf290 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff110e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10c0cfc60 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff10a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff10c0cf290 0x7ff10c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff10a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff10c0cf290 0x7ff10c0cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49140/0 (socket says 172.21.15.141:49140) 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff10a800640 1 -- 172.21.15.141:0/3901183549 learned_addr learned my addr 172.21.15.141:0/3901183549 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff10a800640 1 -- 172.21.15.141:0/3901183549 >> v2:172.21.15.188:3300/0 conn(0x7ff10c0b3ae0 msgr2=0x7ff10c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff10a800640 1 --2- 172.21.15.141:0/3901183549 >> v2:172.21.15.188:3300/0 conn(0x7ff10c0b3ae0 0x7ff10c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff10a800640 1 -- 172.21.15.141:0/3901183549 >> v1:172.21.15.188:6789/0 conn(0x7ff10c0ae5a0 legacy=0x7ff10c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.844+0000 7ff10a800640 1 -- 172.21.15.141:0/3901183549 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c06c470 con 0x7ff10c0cf290 2024-01-16T15:00:20.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.845+0000 7ff10a800640 1 --2- 172.21.15.141:0/3901183549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0cf290 0x7ff10c0cf720 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0cfea0 tx=0x7ff10402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3af84acc1ae6bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff109400640 1 -- 172.21.15.141:0/3901183549 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10402cb20 con 0x7ff10c0cf290 2024-01-16T15:00:20.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff109400640 1 -- 172.21.15.141:0/3901183549 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10402cc80 con 0x7ff10c0cf290 2024-01-16T15:00:20.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff109400640 1 -- 172.21.15.141:0/3901183549 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040343f0 con 0x7ff10c0cf290 2024-01-16T15:00:20.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff110e00640 1 -- 172.21.15.141:0/3901183549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0cf290 msgr2=0x7ff10c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff110e00640 1 --2- 172.21.15.141:0/3901183549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0cf290 0x7ff10c0cf720 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0cfea0 tx=0x7ff10402bf80 comp rx=0 tx=0).stop 2024-01-16T15:00:20.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff110e00640 1 -- 172.21.15.141:0/3901183549 shutdown_connections 2024-01-16T15:00:20.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff110e00640 1 --2- 172.21.15.141:0/3901183549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0cf290 0x7ff10c0cf720 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff110e00640 1 --2- 172.21.15.141:0/3901183549 >> v2:172.21.15.188:3300/0 conn(0x7ff10c0b3ae0 0x7ff10c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.846+0000 7ff110e00640 1 -- 172.21.15.141:0/3901183549 >> 172.21.15.141:0/3901183549 conn(0x7ff10c06a800 msgr2=0x7ff10c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:20.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.847+0000 7ff110e00640 1 -- 172.21.15.141:0/3901183549 shutdown_connections 2024-01-16T15:00:20.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.847+0000 7ff110e00640 1 -- 172.21.15.141:0/3901183549 wait complete. 2024-01-16T15:00:20.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.849+0000 7ff110e00640 1 Processor -- start 2024-01-16T15:00:20.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.849+0000 7ff110e00640 1 -- start start 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.850+0000 7ff110e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff10c0ae5a0 0x7ff10c159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.850+0000 7ff110e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0b3ae0 0x7ff10c15c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.850+0000 7ff110e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff10c0cf290 0x7ff10c15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.850+0000 7ff110e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c14dc30 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.850+0000 7ff110e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c13c1e0 con 0x7ff10c0ae5a0 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.850+0000 7ff110e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c14d390 con 0x7ff10c0cf290 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.850+0000 7ff103e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0b3ae0 0x7ff10c15c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0b3ae0 0x7ff10c15c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49154/0 (socket says 172.21.15.141:49154) 2024-01-16T15:00:20.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 -- 172.21.15.141:0/2994500691 learned_addr learned my addr 172.21.15.141:0/2994500691 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:20.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff109e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff10c0ae5a0 0x7ff10c159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff10a800640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff10c0cf290 0x7ff10c15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 -- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff10c0cf290 msgr2=0x7ff10c15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff10c0cf290 0x7ff10c15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 -- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff10c0ae5a0 msgr2=0x7ff10c159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff10c0ae5a0 0x7ff10c159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 -- 172.21.15.141:0/2994500691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff104009b30 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff10a800640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff10c0cf290 0x7ff10c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff109e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff10c0ae5a0 0x7ff10c159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff103e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0b3ae0 0x7ff10c15c720 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b4a0 tx=0x7ff0f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff102000640 1 -- 172.21.15.141:0/2994500691 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4010610 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.851+0000 7ff102000640 1 -- 172.21.15.141:0/2994500691 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f4010770 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.852+0000 7ff102000640 1 -- 172.21.15.141:0/2994500691 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4010b20 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.852+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c14d670 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.852+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c15a140 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.853+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10c13c400 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.854+0000 7ff102000640 1 -- 172.21.15.141:0/2994500691 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff0f4010c80 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.855+0000 7ff102000640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0e406b500 0x7ff0e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.855+0000 7ff102000640 1 -- 172.21.15.141:0/2994500691 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0f4089cb0 con 0x7ff10c0b3ae0 2024-01-16T15:00:20.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.856+0000 7ff109e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0e406b500 0x7ff0e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.857+0000 7ff109e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0e406b500 0x7ff0e406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc007fd0 tx=0x7ff0fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:20.859+0000 7ff102000640 1 -- 172.21.15.141:0/2994500691 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff10c13c400 con 0x7ff10c0b3ae0 2024-01-16T15:00:21.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.068+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff10c13c060 con 0x7ff0e406b500 2024-01-16T15:00:21.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.072+0000 7ff102000640 1 -- 172.21.15.141:0/2994500691 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff0fc001360 con 0x7ff0e406b500 2024-01-16T15:00:21.052 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:00:21.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:00:21.053 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:00:21.053 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:00:21.053 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:00:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.074+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0e406b500 msgr2=0x7ff0e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.074+0000 7ff110e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0e406b500 0x7ff0e406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc007fd0 tx=0x7ff0fc009040 comp rx=0 tx=0).stop 2024-01-16T15:00:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.074+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0b3ae0 msgr2=0x7ff10c15c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.074+0000 7ff110e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0b3ae0 0x7ff10c15c720 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b4a0 tx=0x7ff0f400b960 comp rx=0 tx=0).stop 2024-01-16T15:00:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 shutdown_connections 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0e406b500 0x7ff0e406d9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff10c0cf290 0x7ff10c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff10c0b3ae0 0x7ff10c15c720 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 --2- 172.21.15.141:0/2994500691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff10c0ae5a0 0x7ff10c159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 >> 172.21.15.141:0/2994500691 conn(0x7ff10c06a800 msgr2=0x7ff10c15d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 shutdown_connections 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.075+0000 7ff110e00640 1 -- 172.21.15.141:0/2994500691 wait complete. 2024-01-16T15:00:21.073 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:00:21.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.299+0000 7f4c06800640 1 Processor -- start 2024-01-16T15:00:21.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.300+0000 7f4c06800640 1 -- start start 2024-01-16T15:00:21.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.301+0000 7f4c06800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4c0013e360 0x7f4c000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.301+0000 7f4c06800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.301+0000 7f4c06800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0015c5f0 con 0x7f4c0013df80 2024-01-16T15:00:21.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.301+0000 7f4c06800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0006c470 con 0x7f4c0013e360 2024-01-16T15:00:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.301+0000 7f4c06800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0006c750 con 0x7f4c0015f9c0 2024-01-16T15:00:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.302+0000 7f4bffe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.302+0000 7f4bffe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33344/0 (socket says 172.21.15.141:33344) 2024-01-16T15:00:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.302+0000 7f4bffe00640 1 -- 172.21.15.141:0/2419363900 learned_addr learned my addr 172.21.15.141:0/2419363900 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.302+0000 7f4bf6800640 1 --2- 172.21.15.141:0/2419363900 >> v2:172.21.15.188:3300/0 conn(0x7f4c0013e360 0x7f4c000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.303+0000 7f4bf6800640 1 -- 172.21.15.141:0/2419363900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 msgr2=0x7f4c000cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.303+0000 7f4bf6800640 1 --2- 172.21.15.141:0/2419363900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.303+0000 7f4bf6800640 1 -- 172.21.15.141:0/2419363900 >> v1:172.21.15.188:6789/0 conn(0x7f4c0013df80 legacy=0x7f4c0015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.303+0000 7f4bf6800640 1 -- 172.21.15.141:0/2419363900 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0006ca30 con 0x7f4c0013e360 2024-01-16T15:00:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.304+0000 7f4bf6800640 1 --2- 172.21.15.141:0/2419363900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013e360 0x7f4c000cf290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4be80097a0 tx=0x7f4be802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=842d8ec1853c258 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.304+0000 7f4bfea00640 1 -- 172.21.15.141:0/2419363900 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8031830 con 0x7f4c0013e360 2024-01-16T15:00:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4bfea00640 1 -- 172.21.15.141:0/2419363900 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be8031990 con 0x7f4c0013e360 2024-01-16T15:00:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4bfea00640 1 -- 172.21.15.141:0/2419363900 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8031d20 con 0x7f4c0013e360 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 -- 172.21.15.141:0/2419363900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013e360 msgr2=0x7f4c000cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 --2- 172.21.15.141:0/2419363900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013e360 0x7f4c000cf290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4be80097a0 tx=0x7f4be802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 -- 172.21.15.141:0/2419363900 shutdown_connections 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 --2- 172.21.15.141:0/2419363900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013e360 0x7f4c000cf290 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 --2- 172.21.15.141:0/2419363900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 -- 172.21.15.141:0/2419363900 >> 172.21.15.141:0/2419363900 conn(0x7f4c0006a800 msgr2=0x7f4c0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 -- 172.21.15.141:0/2419363900 shutdown_connections 2024-01-16T15:00:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.305+0000 7f4c06800640 1 -- 172.21.15.141:0/2419363900 wait complete. 2024-01-16T15:00:21.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.307+0000 7f4c06800640 1 Processor -- start 2024-01-16T15:00:21.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.307+0000 7f4c06800640 1 -- start start 2024-01-16T15:00:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4c06800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013df80 0x7f4c0013da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4c06800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c0013e360 0x7f4c0014dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4c06800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4c06800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c00151390 con 0x7f4c0013e360 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4c06800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0014cf10 con 0x7f4c0013df80 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4c06800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0014e6a0 con 0x7f4c0015f9c0 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4bf6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c0013e360 0x7f4c0014dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4bf6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c0013e360 0x7f4c0014dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49162/0 (socket says 172.21.15.141:49162) 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.308+0000 7f4bf6800640 1 -- 172.21.15.141:0/3458477167 learned_addr learned my addr 172.21.15.141:0/3458477167 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bff400640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013df80 0x7f4c0013da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bffe00640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bf6800640 1 -- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 msgr2=0x7f4c000b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bf6800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bf6800640 1 -- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013df80 msgr2=0x7f4c0013da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bf6800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013df80 0x7f4c0013da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bf6800640 1 -- 172.21.15.141:0/3458477167 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be8009400 con 0x7f4c0013e360 2024-01-16T15:00:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.309+0000 7f4bf6800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c0013e360 0x7f4c0014dfa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4be802bee0 tx=0x7f4be80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.310+0000 7f4bf7e00640 1 -- 172.21.15.141:0/3458477167 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be803a6f0 con 0x7f4c0013e360 2024-01-16T15:00:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.310+0000 7f4bf7e00640 1 -- 172.21.15.141:0/3458477167 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be803a870 con 0x7f4c0013e360 2024-01-16T15:00:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.310+0000 7f4bf7e00640 1 -- 172.21.15.141:0/3458477167 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be803ac20 con 0x7f4c0013e360 2024-01-16T15:00:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.310+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0014c5e0 con 0x7f4c0013e360 2024-01-16T15:00:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.310+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0014cba0 con 0x7f4c0013e360 2024-01-16T15:00:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.311+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0014d080 con 0x7f4c0013e360 2024-01-16T15:00:21.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.312+0000 7f4bf7e00640 1 -- 172.21.15.141:0/3458477167 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4be8030380 con 0x7f4c0013e360 2024-01-16T15:00:21.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.313+0000 7f4bf7e00640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bd806b500 0x7f4bd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.313+0000 7f4bf7e00640 1 -- 172.21.15.141:0/3458477167 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4be80aba80 con 0x7f4c0013e360 2024-01-16T15:00:21.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.313+0000 7f4bff400640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bd806b500 0x7f4bd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.314+0000 7f4bff400640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bd806b500 0x7f4bd806d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0000f60 tx=0x7f4bf0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.317+0000 7f4bf7e00640 1 -- 172.21.15.141:0/3458477167 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4be8076470 con 0x7f4c0013e360 2024-01-16T15:00:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.506+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4c0014cd30 con 0x7f4c0013e360 2024-01-16T15:00:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.507+0000 7f4bf7e00640 1 -- 172.21.15.141:0/3458477167 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4be8079a90 con 0x7f4c0013e360 2024-01-16T15:00:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.509+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bd806b500 msgr2=0x7f4bd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.509+0000 7f4c06800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bd806b500 0x7f4bd806d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0000f60 tx=0x7f4bf0009040 comp rx=0 tx=0).stop 2024-01-16T15:00:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.509+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c0013e360 msgr2=0x7f4c0014dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.509+0000 7f4c06800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c0013e360 0x7f4c0014dfa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4be802bee0 tx=0x7f4be80046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 shutdown_connections 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bd806b500 0x7f4bd806d9c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c0015f9c0 0x7f4c000b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c0013e360 0x7f4c0014dfa0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 --2- 172.21.15.141:0/3458477167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0013df80 0x7f4c0013da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 >> 172.21.15.141:0/3458477167 conn(0x7f4c0006a800 msgr2=0x7f4c0015def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 shutdown_connections 2024-01-16T15:00:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.510+0000 7f4c06800640 1 -- 172.21.15.141:0/3458477167 wait complete. 2024-01-16T15:00:21.508 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:00:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.735+0000 7fcdb3200640 1 Processor -- start 2024-01-16T15:00:21.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.735+0000 7fcdb3200640 1 -- start start 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.736+0000 7fcdb3200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcdb40cc500 0x7fcdb40cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.736+0000 7fcdb3200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcdb406d160 0x7fcdb406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.736+0000 7fcdb3200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb4070e50 con 0x7fcdb4143c40 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.736+0000 7fcdb3200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb40723c0 con 0x7fcdb406d160 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.736+0000 7fcdb3200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb40cced0 con 0x7fcdb40cc500 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.737+0000 7fcdb2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcdb406d160 0x7fcdb406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.737+0000 7fcdb2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcdb406d160 0x7fcdb406d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49172/0 (socket says 172.21.15.141:49172) 2024-01-16T15:00:21.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.737+0000 7fcdb2800640 1 -- 172.21.15.141:0/2839144264 learned_addr learned my addr 172.21.15.141:0/2839144264 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.737+0000 7fcdb2800640 1 -- 172.21.15.141:0/2839144264 >> v2:172.21.15.188:3300/0 conn(0x7fcdb40cc500 msgr2=0x7fcdb40cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.737+0000 7fcdb2800640 1 --2- 172.21.15.141:0/2839144264 >> v2:172.21.15.188:3300/0 conn(0x7fcdb40cc500 0x7fcdb40cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.737+0000 7fcdb2800640 1 -- 172.21.15.141:0/2839144264 >> v1:172.21.15.188:6789/0 conn(0x7fcdb4143c40 legacy=0x7fcdb4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.737+0000 7fcdb2800640 1 -- 172.21.15.141:0/2839144264 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4144020 con 0x7fcdb406d160 2024-01-16T15:00:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.738+0000 7fcdb2800640 1 --2- 172.21.15.141:0/2839144264 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb406d160 0x7fcdb406d560 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40b44a0 tx=0x7fcdac02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1858423e339829 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb1400640 1 -- 172.21.15.141:0/2839144264 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac02c9f0 con 0x7fcdb406d160 2024-01-16T15:00:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb1400640 1 -- 172.21.15.141:0/2839144264 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdac02cb50 con 0x7fcdb406d160 2024-01-16T15:00:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb1400640 1 -- 172.21.15.141:0/2839144264 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0343f0 con 0x7fcdb406d160 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb3200640 1 -- 172.21.15.141:0/2839144264 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb406d160 msgr2=0x7fcdb406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb3200640 1 --2- 172.21.15.141:0/2839144264 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb406d160 0x7fcdb406d560 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40b44a0 tx=0x7fcdac02be50 comp rx=0 tx=0).stop 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb3200640 1 -- 172.21.15.141:0/2839144264 shutdown_connections 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb3200640 1 --2- 172.21.15.141:0/2839144264 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb406d160 0x7fcdb406d560 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb3200640 1 --2- 172.21.15.141:0/2839144264 >> v2:172.21.15.188:3300/0 conn(0x7fcdb40cc500 0x7fcdb40cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb3200640 1 -- 172.21.15.141:0/2839144264 >> 172.21.15.141:0/2839144264 conn(0x7fcdb406a800 msgr2=0x7fcdb406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.739+0000 7fcdb3200640 1 -- 172.21.15.141:0/2839144264 shutdown_connections 2024-01-16T15:00:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.740+0000 7fcdb3200640 1 -- 172.21.15.141:0/2839144264 wait complete. 2024-01-16T15:00:21.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.742+0000 7fcdb3200640 1 Processor -- start 2024-01-16T15:00:21.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.742+0000 7fcdb3200640 1 -- start start 2024-01-16T15:00:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb3200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdb406d160 0x7fcdb40a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb3200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcdb40cc500 0x7fcdb41577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb3200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb4143c40 0x7fcdb413c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb3200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb4158280 con 0x7fcdb4143c40 2024-01-16T15:00:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb3200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb4158560 con 0x7fcdb406d160 2024-01-16T15:00:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb3200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb4158840 con 0x7fcdb40cc500 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb4143c40 0x7fcdb413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb4143c40 0x7fcdb413c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49188/0 (socket says 172.21.15.141:49188) 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.743+0000 7fcdb2800640 1 -- 172.21.15.141:0/99303799 learned_addr learned my addr 172.21.15.141:0/99303799 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcda9200640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcdb40cc500 0x7fcdb41577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdb2800640 1 -- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcdb40cc500 msgr2=0x7fcdb41577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdb2800640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcdb40cc500 0x7fcdb41577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdb2800640 1 -- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdb406d160 msgr2=0x7fcdb40a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdb2800640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdb406d160 0x7fcdb40a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdb2800640 1 -- 172.21.15.141:0/99303799 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac009b30 con 0x7fcdb4143c40 2024-01-16T15:00:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdb2800640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb4143c40 0x7fcdb413c380 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcdac036040 tx=0x7fcdac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdaaa00640 1 -- 172.21.15.141:0/99303799 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac03e450 con 0x7fcdb4143c40 2024-01-16T15:00:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdaaa00640 1 -- 172.21.15.141:0/99303799 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdac03e5d0 con 0x7fcdb4143c40 2024-01-16T15:00:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdaaa00640 1 -- 172.21.15.141:0/99303799 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac03e980 con 0x7fcdb4143c40 2024-01-16T15:00:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.744+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb4158b20 con 0x7fcdb4143c40 2024-01-16T15:00:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.745+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb40a93d0 con 0x7fcdb4143c40 2024-01-16T15:00:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.746+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb40a9560 con 0x7fcdb4143c40 2024-01-16T15:00:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.747+0000 7fcdaaa00640 1 -- 172.21.15.141:0/99303799 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fcdac03eae0 con 0x7fcdb4143c40 2024-01-16T15:00:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.747+0000 7fcdaaa00640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcd8c06b500 0x7fcd8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.748+0000 7fcdaaa00640 1 -- 172.21.15.141:0/99303799 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdac0acf40 con 0x7fcdb4143c40 2024-01-16T15:00:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.748+0000 7fcdb1e00640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcd8c06b500 0x7fcd8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.749+0000 7fcdb1e00640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcd8c06b500 0x7fcd8c06d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009650 tx=0x7fcda4006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.752+0000 7fcdaaa00640 1 -- 172.21.15.141:0/99303799 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcdac077a40 con 0x7fcdb4143c40 2024-01-16T15:00:21.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:21.970+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdb41586d0 con 0x7fcd8c06b500 2024-01-16T15:00:21.961 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:00:21.980+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:21.961 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:00:21.980+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:21.961 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:00:21.980+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:21.961 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:00:21.980+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:21.961 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:00:21.980+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:00:22.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.057+0000 7fcdaaa00640 1 -- 172.21.15.141:0/99303799 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcdb41586d0 con 0x7fcd8c06b500 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcd8c06b500 msgr2=0x7fcd8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcd8c06b500 0x7fcd8c06d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009650 tx=0x7fcda4006ba0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb4143c40 msgr2=0x7fcdb413c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb4143c40 0x7fcdb413c380 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcdac036040 tx=0x7fcdac004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 shutdown_connections 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcd8c06b500 0x7fcd8c06d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcdb4143c40 0x7fcdb413c380 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcdb40cc500 0x7fcdb41577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 --2- 172.21.15.141:0/99303799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdb406d160 0x7fcdb40a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 >> 172.21.15.141:0/99303799 conn(0x7fcdb406a800 msgr2=0x7fcdb406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 shutdown_connections 2024-01-16T15:00:22.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.061+0000 7fcdb3200640 1 -- 172.21.15.141:0/99303799 wait complete. 2024-01-16T15:00:22.063 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:00:22.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.295+0000 7f9381800640 1 Processor -- start 2024-01-16T15:00:22.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.295+0000 7f9381800640 1 -- start start 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.297+0000 7f9381800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f937c15de80 0x7f937c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.297+0000 7f9381800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c06c730 con 0x7f937c071e40 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.297+0000 7f9381800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0724d0 con 0x7f937c143810 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.297+0000 7f9381800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937c143bf0 con 0x7f937c15de80 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.297+0000 7f937b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f937c071e40 0x7f937c15d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58194/0 (socket says 172.21.15.141:58194) 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.297+0000 7f937b400640 1 -- 172.21.15.141:0/3257866686 learned_addr learned my addr 172.21.15.141:0/3257866686 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.297+0000 7f9380e00640 1 --2- 172.21.15.141:0/3257866686 >> v2:172.21.15.141:3300/0 conn(0x7f937c15de80 0x7f937c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.298+0000 7f937aa00640 1 -- 172.21.15.141:0/3257866686 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883294684 0 0) 0x7f937c06c730 con 0x7f937c071e40 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.298+0000 7f937aa00640 1 -- 172.21.15.141:0/3257866686 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c003520 con 0x7f937c071e40 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.298+0000 7f9380e00640 1 -- 172.21.15.141:0/3257866686 >> v1:172.21.15.188:6789/0 conn(0x7f937c143810 legacy=0x7f937c15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.298+0000 7f9380e00640 1 -- 172.21.15.141:0/3257866686 >> v1:172.21.15.141:6789/0 conn(0x7f937c071e40 legacy=0x7f937c15d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.298+0000 7f9380e00640 1 -- 172.21.15.141:0/3257866686 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c06c9f0 con 0x7f937c15de80 2024-01-16T15:00:22.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.299+0000 7f9380e00640 1 --2- 172.21.15.141:0/3257866686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c0ae170 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9374009d90 tx=0x7f937402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f213c7761918ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.299+0000 7f937aa00640 1 -- 172.21.15.141:0/3257866686 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937402cbd0 con 0x7f937c15de80 2024-01-16T15:00:22.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.299+0000 7f937aa00640 1 -- 172.21.15.141:0/3257866686 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937402cd30 con 0x7f937c15de80 2024-01-16T15:00:22.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.299+0000 7f937aa00640 1 -- 172.21.15.141:0/3257866686 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937402cea0 con 0x7f937c15de80 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.300+0000 7f9381800640 1 -- 172.21.15.141:0/3257866686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 msgr2=0x7f937c0ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.300+0000 7f9381800640 1 --2- 172.21.15.141:0/3257866686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c0ae170 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9374009d90 tx=0x7f937402be20 comp rx=0 tx=0).stop 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.300+0000 7f9381800640 1 -- 172.21.15.141:0/3257866686 shutdown_connections 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.300+0000 7f9381800640 1 --2- 172.21.15.141:0/3257866686 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c0ae170 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.300+0000 7f9381800640 1 -- 172.21.15.141:0/3257866686 >> 172.21.15.141:0/3257866686 conn(0x7f937c06a7d0 msgr2=0x7f937c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.300+0000 7f9381800640 1 -- 172.21.15.141:0/3257866686 shutdown_connections 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.300+0000 7f9381800640 1 -- 172.21.15.141:0/3257866686 wait complete. 2024-01-16T15:00:22.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.302+0000 7f9381800640 1 Processor -- start 2024-01-16T15:00:22.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.302+0000 7f9381800640 1 -- start start 2024-01-16T15:00:22.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9381800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f937c071e40 0x7f937c1344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9381800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f937c143810 0x7f937c0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9381800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9381800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0722f0 con 0x7f937c15de80 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9381800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c1462c0 con 0x7f937c071e40 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9381800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c1349f0 con 0x7f937c143810 2024-01-16T15:00:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9380e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9380e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c1466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49198/0 (socket says 172.21.15.141:49198) 2024-01-16T15:00:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.303+0000 7f9380e00640 1 -- 172.21.15.141:0/3981026651 learned_addr learned my addr 172.21.15.141:0/3981026651 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f937b400640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f937c143810 0x7f937c0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f937be00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f937c071e40 0x7f937c1344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9380e00640 1 -- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f937c143810 msgr2=0x7f937c0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9380e00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f937c143810 0x7f937c0ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9380e00640 1 -- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f937c071e40 msgr2=0x7f937c1344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9380e00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f937c071e40 0x7f937c1344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9380e00640 1 -- 172.21.15.141:0/3981026651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9374009a70 con 0x7f937c15de80 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f937be00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f937c071e40 0x7f937c1344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9380e00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c1466e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9374009d90 tx=0x7f9374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9373e00640 1 -- 172.21.15.141:0/3981026651 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937402cea0 con 0x7f937c15de80 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9373e00640 1 -- 172.21.15.141:0/3981026651 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9374040650 con 0x7f937c15de80 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c0cb890 con 0x7f937c15de80 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9373e00640 1 -- 172.21.15.141:0/3981026651 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374040a00 con 0x7f937c15de80 2024-01-16T15:00:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.304+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c0cea10 con 0x7f937c15de80 2024-01-16T15:00:22.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.306+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c146400 con 0x7f937c15de80 2024-01-16T15:00:22.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.306+0000 7f9373e00640 1 -- 172.21.15.141:0/3981026651 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9374040b60 con 0x7f937c15de80 2024-01-16T15:00:22.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.307+0000 7f9373e00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f935406b500 0x7f935406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.308+0000 7f9373e00640 1 -- 172.21.15.141:0/3981026651 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93740acca0 con 0x7f937c15de80 2024-01-16T15:00:22.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.308+0000 7f937be00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f935406b500 0x7f935406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.309+0000 7f937be00640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f935406b500 0x7f935406d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9364000f60 tx=0x7f9364009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.312+0000 7f9373e00640 1 -- 172.21.15.141:0/3981026651 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f937c146400 con 0x7f937c15de80 2024-01-16T15:00:22.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.532+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f937c146400 con 0x7f935406b500 2024-01-16T15:00:22.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.544+0000 7f9373e00640 1 -- 172.21.15.141:0/3981026651 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f937c146400 con 0x7f935406b500 2024-01-16T15:00:22.524 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.546+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f935406b500 msgr2=0x7f935406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.546+0000 7f9381800640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f935406b500 0x7f935406d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9364000f60 tx=0x7f9364009040 comp rx=0 tx=0).stop 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.546+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 msgr2=0x7f937c1466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.546+0000 7f9381800640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c1466e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9374009d90 tx=0x7f9374004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 shutdown_connections 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f935406b500 0x7f935406d9c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c15de80 0x7f937c1466e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f937c143810 0x7f937c0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 --2- 172.21.15.141:0/3981026651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f937c071e40 0x7f937c1344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 >> 172.21.15.141:0/3981026651 conn(0x7f937c06a7d0 msgr2=0x7f937c160ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 shutdown_connections 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:22.547+0000 7f9381800640 1 -- 172.21.15.141:0/3981026651 wait complete. 2024-01-16T15:00:22.546 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0 2024-01-16T15:00:22.547 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:00:22.547 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:22.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:22.601 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:00:32.936 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:32.943 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:36.015 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:00:36.015 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:00:36.015 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.2863 s, 80.8 MB/s 2024-01-16T15:00:36.017 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:00:36.017 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:36.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:36.065 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:00:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:00:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:00:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.0395 s, 89.2 MB/s 2024-01-16T15:00:48.203 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:00:48.203 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:48.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:48.254 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:00:59.321 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:00:59.321 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:00:59.321 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.9618 s, 98.0 MB/s 2024-01-16T15:00:59.322 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:00:59.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.554+0000 7f1b1e600640 1 Processor -- start 2024-01-16T15:00:59.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.554+0000 7f1b1e600640 1 -- start start 2024-01-16T15:00:59.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b1e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b1e600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1b180d0b60 0x7f1b180aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b1e600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b18071de0 con 0x7f1b1806a7c0 2024-01-16T15:00:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b1e600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b180af320 con 0x7f1b180d0b60 2024-01-16T15:00:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b1e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18072030 con 0x7f1b1806a3e0 2024-01-16T15:00:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b0fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1b180d0b60 0x7f1b180aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b0fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1b180d0b60 0x7f1b180aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54460/0 (socket says 172.21.15.141:54460) 2024-01-16T15:00:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b0fe00640 1 -- 172.21.15.141:0/2799534431 learned_addr learned my addr 172.21.15.141:0/2799534431 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.556+0000 7f1b17400640 1 --2- 172.21.15.141:0/2799534431 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.557+0000 7f1b0fe00640 1 -- 172.21.15.141:0/2799534431 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 msgr2=0x7f1b180ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.557+0000 7f1b0fe00640 1 --2- 172.21.15.141:0/2799534431 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.557+0000 7f1b0fe00640 1 -- 172.21.15.141:0/2799534431 >> v1:172.21.15.188:6789/0 conn(0x7f1b1806a7c0 legacy=0x7f1b1806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.557+0000 7f1b0fe00640 1 -- 172.21.15.141:0/2799534431 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18072310 con 0x7f1b180d0b60 2024-01-16T15:00:59.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.558+0000 7f1b0fe00640 1 --2- 172.21.15.141:0/2799534431 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b180aede0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009870 tx=0x7f1b0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9923a4318156bd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.558+0000 7f1b16a00640 1 -- 172.21.15.141:0/2799534431 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00031640 con 0x7f1b180d0b60 2024-01-16T15:00:59.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.558+0000 7f1b16a00640 1 -- 172.21.15.141:0/2799534431 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b000317a0 con 0x7f1b180d0b60 2024-01-16T15:00:59.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.558+0000 7f1b16a00640 1 -- 172.21.15.141:0/2799534431 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00031b30 con 0x7f1b180d0b60 2024-01-16T15:00:59.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 -- 172.21.15.141:0/2799534431 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 msgr2=0x7f1b180aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 --2- 172.21.15.141:0/2799534431 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b180aede0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009870 tx=0x7f1b0002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 -- 172.21.15.141:0/2799534431 shutdown_connections 2024-01-16T15:00:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 --2- 172.21.15.141:0/2799534431 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b180aede0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 --2- 172.21.15.141:0/2799534431 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 -- 172.21.15.141:0/2799534431 >> 172.21.15.141:0/2799534431 conn(0x7f1b180cfef0 msgr2=0x7f1b180d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 -- 172.21.15.141:0/2799534431 shutdown_connections 2024-01-16T15:00:59.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.559+0000 7f1b1e600640 1 -- 172.21.15.141:0/2799534431 wait complete. 2024-01-16T15:00:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.561+0000 7f1b1e600640 1 Processor -- start 2024-01-16T15:00:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.561+0000 7f1b1e600640 1 -- start start 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.562+0000 7f1b1e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.562+0000 7f1b1e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b1806a7c0 0x7f1b181482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.562+0000 7f1b1e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b18144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.562+0000 7f1b1e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b180cf030 con 0x7f1b180d0b60 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.562+0000 7f1b1e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18160000 con 0x7f1b1806a7c0 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.562+0000 7f1b1e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b180cf990 con 0x7f1b1806a3e0 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.562+0000 7f1b17e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b18144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b18144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54474/0 (socket says 172.21.15.141:54474) 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 -- 172.21.15.141:0/473806562 learned_addr learned my addr 172.21.15.141:0/473806562 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17400640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b0fe00640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b1806a7c0 0x7f1b181482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 -- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 msgr2=0x7f1b180cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 -- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b1806a7c0 msgr2=0x7f1b181482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b1806a7c0 0x7f1b181482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 -- 172.21.15.141:0/473806562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000094d0 con 0x7f1b180d0b60 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b0fe00640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b1806a7c0 0x7f1b181482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.563+0000 7f1b17e00640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b18144f90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000c780 tx=0x7f1b1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.564+0000 7f1b0f400640 1 -- 172.21.15.141:0/473806562 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b10004b60 con 0x7f1b180d0b60 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.564+0000 7f1b0f400640 1 -- 172.21.15.141:0/473806562 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1000ce90 con 0x7f1b180d0b60 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.564+0000 7f1b0f400640 1 -- 172.21.15.141:0/473806562 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1000a850 con 0x7f1b180d0b60 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.564+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b18148800 con 0x7f1b180d0b60 2024-01-16T15:00:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.564+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b180ce690 con 0x7f1b180d0b60 2024-01-16T15:00:59.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.565+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b180cf170 con 0x7f1b180d0b60 2024-01-16T15:00:59.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.566+0000 7f1b0f400640 1 -- 172.21.15.141:0/473806562 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f1b1000a9b0 con 0x7f1b180d0b60 2024-01-16T15:00:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.567+0000 7f1b0f400640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1af006b590 0x7f1af006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.567+0000 7f1b0f400640 1 -- 172.21.15.141:0/473806562 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1b1008a190 con 0x7f1b180d0b60 2024-01-16T15:00:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.568+0000 7f1b17400640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1af006b590 0x7f1af006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.569+0000 7f1b17400640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1af006b590 0x7f1af006da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1b08006fd0 tx=0x7f1b08008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.572+0000 7f1b0f400640 1 -- 172.21.15.141:0/473806562 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b180cf170 con 0x7f1b180d0b60 2024-01-16T15:00:59.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.767+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f1b180cf170 con 0x7f1b180d0b60 2024-01-16T15:00:59.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.773+0000 7f1b0f400640 1 -- 172.21.15.141:0/473806562 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7f1b180cf170 con 0x7f1b180d0b60 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1af006b590 msgr2=0x7f1af006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1af006b590 0x7f1af006da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1b08006fd0 tx=0x7f1b08008040 comp rx=0 tx=0).stop 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 msgr2=0x7f1b18144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b18144f90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000c780 tx=0x7f1b1000cc40 comp rx=0 tx=0).stop 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 shutdown_connections 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1af006b590 0x7f1af006da50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b180d0b60 0x7f1b18144f90 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b1806a7c0 0x7f1b181482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 --2- 172.21.15.141:0/473806562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b1806a3e0 0x7f1b180cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 >> 172.21.15.141:0/473806562 conn(0x7f1b180cfef0 msgr2=0x7f1b180b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 shutdown_connections 2024-01-16T15:00:59.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:00:59.776+0000 7f1b1e600640 1 -- 172.21.15.141:0/473806562 wait complete. 2024-01-16T15:00:59.776 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:00:59.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.006+0000 7fa0da600640 1 Processor -- start 2024-01-16T15:00:59.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.006+0000 7fa0da600640 1 -- start start 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.008+0000 7fa0da600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa0d4143ae0 0x7fa0d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.008+0000 7fa0da600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.008+0000 7fa0da600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d40cfc30 con 0x7fa0d415e1e0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.008+0000 7fa0da600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d40b1ee0 con 0x7fa0d4143ae0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.008+0000 7fa0da600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d40b2780 con 0x7fa0d40cf7d0 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d3400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa0d4143ae0 0x7fa0d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d3400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa0d4143ae0 0x7fa0d40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:53292/0 (socket says 172.21.15.141:53292) 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d3400640 1 -- 172.21.15.141:0/1928563827 learned_addr learned my addr 172.21.15.141:0/1928563827 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:59.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d3400640 1 -- 172.21.15.141:0/1928563827 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 msgr2=0x7fa0d415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d3400640 1 --2- 172.21.15.141:0/1928563827 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d3400640 1 -- 172.21.15.141:0/1928563827 >> v1:172.21.15.188:6789/0 conn(0x7fa0d415e1e0 legacy=0x7fa0d415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.009+0000 7fa0d3400640 1 -- 172.21.15.141:0/1928563827 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d415e990 con 0x7fa0d4143ae0 2024-01-16T15:00:59.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.011+0000 7fa0d3400640 1 --2- 172.21.15.141:0/1928563827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d4143ae0 0x7fa0d40cf290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400bb10 tx=0x7fa0c402e150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65ffae73522ef71c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.011+0000 7fa0d2000640 1 -- 172.21.15.141:0/1928563827 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4036440 con 0x7fa0d4143ae0 2024-01-16T15:00:59.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.011+0000 7fa0d2000640 1 -- 172.21.15.141:0/1928563827 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c40365a0 con 0x7fa0d4143ae0 2024-01-16T15:00:59.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0d2000640 1 -- 172.21.15.141:0/1928563827 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4036930 con 0x7fa0d4143ae0 2024-01-16T15:00:59.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 -- 172.21.15.141:0/1928563827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d4143ae0 msgr2=0x7fa0d40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 --2- 172.21.15.141:0/1928563827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d4143ae0 0x7fa0d40cf290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400bb10 tx=0x7fa0c402e150 comp rx=0 tx=0).stop 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 -- 172.21.15.141:0/1928563827 shutdown_connections 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 --2- 172.21.15.141:0/1928563827 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 --2- 172.21.15.141:0/1928563827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d4143ae0 0x7fa0d40cf290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 -- 172.21.15.141:0/1928563827 >> 172.21.15.141:0/1928563827 conn(0x7fa0d406a800 msgr2=0x7fa0d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 -- 172.21.15.141:0/1928563827 shutdown_connections 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.012+0000 7fa0da600640 1 -- 172.21.15.141:0/1928563827 wait complete. 2024-01-16T15:00:59.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.014+0000 7fa0da600640 1 Processor -- start 2024-01-16T15:00:59.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.014+0000 7fa0da600640 1 -- start start 2024-01-16T15:00:59.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0da600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d413e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0da600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0d4143ae0 0x7fa0d41f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0da600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d415e1e0 0x7fa0d40ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0da600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d40ad580 con 0x7fa0d4143ae0 2024-01-16T15:00:59.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0da600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d40ac970 con 0x7fa0d415e1e0 2024-01-16T15:00:59.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0da600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d40acc50 con 0x7fa0d40cf7d0 2024-01-16T15:00:59.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0d2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0d4143ae0 0x7fa0d41f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0d2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0d4143ae0 0x7fa0d41f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54484/0 (socket says 172.21.15.141:54484) 2024-01-16T15:00:59.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.015+0000 7fa0d2a00640 1 -- 172.21.15.141:0/1159515522 learned_addr learned my addr 172.21.15.141:0/1159515522 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:00:59.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d3400640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d413e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d3e00640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d415e1e0 0x7fa0d40ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d2a00640 1 -- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 msgr2=0x7fa0d413e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d2a00640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d413e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d2a00640 1 -- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d415e1e0 msgr2=0x7fa0d40ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d2a00640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d415e1e0 0x7fa0d40ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d2a00640 1 -- 172.21.15.141:0/1159515522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c400b770 con 0x7fa0d4143ae0 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0d2a00640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0d4143ae0 0x7fa0d41f2b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00e9b0 tx=0x7fa0bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.016+0000 7fa0cb400640 1 -- 172.21.15.141:0/1159515522 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc009670 con 0x7fa0d4143ae0 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.017+0000 7fa0cb400640 1 -- 172.21.15.141:0/1159515522 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc00c9a0 con 0x7fa0d4143ae0 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.017+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d40acf30 con 0x7fa0d4143ae0 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.017+0000 7fa0cb400640 1 -- 172.21.15.141:0/1159515522 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00cd50 con 0x7fa0d4143ae0 2024-01-16T15:00:59.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.017+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d413ee90 con 0x7fa0d4143ae0 2024-01-16T15:01:00.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.018+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d40b2bf0 con 0x7fa0d4143ae0 2024-01-16T15:01:00.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.019+0000 7fa0cb400640 1 -- 172.21.15.141:0/1159515522 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fa0bc013470 con 0x7fa0d4143ae0 2024-01-16T15:01:00.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.020+0000 7fa0cb400640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa0ac06b590 0x7fa0ac06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.020+0000 7fa0cb400640 1 -- 172.21.15.141:0/1159515522 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0bc08a350 con 0x7fa0d4143ae0 2024-01-16T15:01:00.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.020+0000 7fa0d3400640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa0ac06b590 0x7fa0ac06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.021+0000 7fa0d3400640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa0ac06b590 0x7fa0ac06da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4038000 tx=0x7fa0c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.024+0000 7fa0cb400640 1 -- 172.21.15.141:0/1159515522 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0d40b2bf0 con 0x7fa0d4143ae0 2024-01-16T15:01:00.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.215+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fa0d40b2bf0 con 0x7fa0d4143ae0 2024-01-16T15:01:00.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.222+0000 7fa0cb400640 1 -- 172.21.15.141:0/1159515522 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v16)=0 v16) v1 ==== 135+0+0 (secure 0 0 0) 0x7fa0d40b2bf0 con 0x7fa0d4143ae0 2024-01-16T15:01:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa0ac06b590 msgr2=0x7fa0ac06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa0ac06b590 0x7fa0ac06da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4038000 tx=0x7fa0c4008040 comp rx=0 tx=0).stop 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0d4143ae0 msgr2=0x7fa0d41f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0d4143ae0 0x7fa0d41f2b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00e9b0 tx=0x7fa0bc00ee70 comp rx=0 tx=0).stop 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 shutdown_connections 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa0ac06b590 0x7fa0ac06da50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0d415e1e0 0x7fa0d40ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0d4143ae0 0x7fa0d41f2b80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 --2- 172.21.15.141:0/1159515522 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0d40cf7d0 0x7fa0d413e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 >> 172.21.15.141:0/1159515522 conn(0x7fa0d406a800 msgr2=0x7fa0d40b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 shutdown_connections 2024-01-16T15:01:00.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.225+0000 7fa0da600640 1 -- 172.21.15.141:0/1159515522 wait complete. 2024-01-16T15:01:00.225 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.459+0000 7fee13c00640 1 Processor -- start 2024-01-16T15:01:00.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.459+0000 7fee13c00640 1 -- start start 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.461+0000 7fee13c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 0x7fee0c166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.461+0000 7fee13c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c15c5f0 con 0x7fee0c13df80 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.461+0000 7fee13c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c06c470 con 0x7fee0c13e360 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.461+0000 7fee13c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c06c750 con 0x7fee0c0cf9a0 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.461+0000 7fee11200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fee0c13df80 0x7fee0c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:51094/0 (socket says 172.21.15.141:51094) 2024-01-16T15:01:00.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.461+0000 7fee11200640 1 -- 172.21.15.141:0/717359675 learned_addr learned my addr 172.21.15.141:0/717359675 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee11c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 0x7fee0c166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168100699 0 0) 0x7fee0c15c5f0 con 0x7fee0c13df80 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedec003520 con 0x7fee0c13df80 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503669279 0 0) 0x7fedec003520 con 0x7fee0c13df80 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 msgr2=0x7fee0c166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 0x7fee0c166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 >> v1:172.21.15.188:6789/0 conn(0x7fee0c13e360 legacy=0x7fee0c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c06ca30 con 0x7fee0c13df80 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fedfc0019f0 con 0x7fee0c13df80 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 >> v1:172.21.15.141:6789/0 conn(0x7fee0c13df80 legacy=0x7fee0c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.462+0000 7fee0be00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedec0039a0 0x7fedec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee0be00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 0x7fedec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee0be00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedec004d00 0x7fedec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c06c750 con 0x7fedec004d00 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec00ba60 con 0x7fedec0039a0 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec00bc20 con 0x7fee0c0cf9a0 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee11c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedec004d00 0x7fedec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee11200640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedec0039a0 0x7fedec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee03c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 0x7fedec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee11c00640 1 -- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 msgr2=0x7fedec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee11c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 0x7fedec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee11c00640 1 -- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedec0039a0 msgr2=0x7fedec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee11c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedec0039a0 0x7fedec003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee11c00640 1 -- 172.21.15.141:0/717359675 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec003520 con 0x7fedec004d00 2024-01-16T15:01:00.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.463+0000 7fee03c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee0c0cf9a0 0x7fedec0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:00.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.464+0000 7fee11c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedec004d00 0x7fedec005180 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fee0c155d40 tx=0x7fedec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637c6156e7c514cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0402c660 con 0x7fedec004d00 2024-01-16T15:01:00.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0402c7c0 con 0x7fedec004d00 2024-01-16T15:01:00.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee0be00640 1 -- 172.21.15.141:0/717359675 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0402cb50 con 0x7fedec004d00 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee13c00640 1 -- 172.21.15.141:0/717359675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedec004d00 msgr2=0x7fedec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee13c00640 1 --2- 172.21.15.141:0/717359675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedec004d00 0x7fedec005180 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fee0c155d40 tx=0x7fedec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee11c00640 1 -- 172.21.15.141:0/717359675 reap_dead start 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee13c00640 1 -- 172.21.15.141:0/717359675 shutdown_connections 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee13c00640 1 -- 172.21.15.141:0/717359675 >> 172.21.15.141:0/717359675 conn(0x7fee0c06a800 msgr2=0x7fee0c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee13c00640 1 -- 172.21.15.141:0/717359675 shutdown_connections 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.465+0000 7fee13c00640 1 -- 172.21.15.141:0/717359675 wait complete. 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.467+0000 7fee13c00640 1 Processor -- start 2024-01-16T15:01:00.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.467+0000 7fee13c00640 1 -- start start 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.468+0000 7fee13c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fedec004d00 0x7fee0c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.468+0000 7fee13c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee0c0a84c0 0x7fee0c154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.468+0000 7fee13c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee0c15d8a0 0x7fee0c15dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.468+0000 7fee13c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0ac8a0 con 0x7fee0c15d8a0 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.468+0000 7fee13c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c15e260 con 0x7fee0c0a84c0 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.468+0000 7fee13c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c12f1c0 con 0x7fedec004d00 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee0c15d8a0 0x7fee0c15dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee0c15d8a0 0x7fee0c15dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54502/0 (socket says 172.21.15.141:54502) 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 -- 172.21.15.141:0/3441329667 learned_addr learned my addr 172.21.15.141:0/3441329667 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee03c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee0c0a84c0 0x7fee0c154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 -- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fedec004d00 msgr2=0x7fee0c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11200640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fedec004d00 0x7fee0c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fedec004d00 0x7fee0c131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 -- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee0c0a84c0 msgr2=0x7fee0c154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee0c0a84c0 0x7fee0c154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 -- 172.21.15.141:0/3441329667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c1321a0 con 0x7fee0c15d8a0 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee03c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee0c0a84c0 0x7fee0c154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:00.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11200640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fedec004d00 0x7fee0c131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.469+0000 7fee11c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee0c15d8a0 0x7fee0c15dd20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fee0402c2d0 tx=0x7fee04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.470+0000 7fee09600640 1 -- 172.21.15.141:0/3441329667 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0402c660 con 0x7fee0c15d8a0 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.470+0000 7fee09600640 1 -- 172.21.15.141:0/3441329667 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04003830 con 0x7fee0c15d8a0 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.470+0000 7fee09600640 1 -- 172.21.15.141:0/3441329667 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04003be0 con 0x7fee0c15d8a0 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.470+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c0a8a30 con 0x7fee0c15d8a0 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.470+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c0a8ff0 con 0x7fee0c15d8a0 2024-01-16T15:01:00.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.471+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c0675d0 con 0x7fee0c15d8a0 2024-01-16T15:01:00.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.472+0000 7fee09600640 1 -- 172.21.15.141:0/3441329667 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fee04003d40 con 0x7fee0c15d8a0 2024-01-16T15:01:00.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.473+0000 7fee09600640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fede406b5c0 0x7fede406da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.473+0000 7fee09600640 1 -- 172.21.15.141:0/3441329667 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fee040ab410 con 0x7fee0c15d8a0 2024-01-16T15:01:00.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.473+0000 7fee11200640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fede406b5c0 0x7fede406da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.474+0000 7fee11200640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fede406b5c0 0x7fede406da80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fedfc002010 tx=0x7fedfc001840 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.477+0000 7fee09600640 1 -- 172.21.15.141:0/3441329667 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee04075eb0 con 0x7fee0c15d8a0 2024-01-16T15:01:00.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.698+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fee0c06cff0 con 0x7fede406b5c0 2024-01-16T15:01:00.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.734+0000 7fee09600640 1 -- 172.21.15.141:0/3441329667 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fedfc005d10 con 0x7fede406b5c0 2024-01-16T15:01:00.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fede406b5c0 msgr2=0x7fede406da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fede406b5c0 0x7fede406da80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fedfc002010 tx=0x7fedfc001840 comp rx=0 tx=0).stop 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee0c15d8a0 msgr2=0x7fee0c15dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee0c15d8a0 0x7fee0c15dd20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fee0402c2d0 tx=0x7fee04004730 comp rx=0 tx=0).stop 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 shutdown_connections 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fede406b5c0 0x7fede406da80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee0c15d8a0 0x7fee0c15dd20 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee0c0a84c0 0x7fee0c154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 --2- 172.21.15.141:0/3441329667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fedec004d00 0x7fee0c131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 >> 172.21.15.141:0/3441329667 conn(0x7fee0c06a800 msgr2=0x7fee0c0cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 shutdown_connections 2024-01-16T15:01:00.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.737+0000 7fee13c00640 1 -- 172.21.15.141:0/3441329667 wait complete. 2024-01-16T15:01:00.736 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:01:00.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.952+0000 7fc8fbe00640 1 Processor -- start 2024-01-16T15:01:00.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.953+0000 7fc8fbe00640 1 -- start start 2024-01-16T15:01:00.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.954+0000 7fc8fbe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 0x7fc8f415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8fbe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f4070c40 con 0x7fc8f40cf290 2024-01-16T15:01:00.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8fbe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f406c470 con 0x7fc8f40ae5a0 2024-01-16T15:01:00.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8fbe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4071450 con 0x7fc8f40b3ae0 2024-01-16T15:01:00.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8f9e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc8f40cf290 0x7fc8f40b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:51106/0 (socket says 172.21.15.141:51106) 2024-01-16T15:01:00.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8f9e00640 1 -- 172.21.15.141:0/1995975310 learned_addr learned my addr 172.21.15.141:0/1995975310 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:00.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8f8a00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 0x7fc8f415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669877524 0 0) 0x7fc8f4070c40 con 0x7fc8f40cf290 2024-01-16T15:01:00.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.955+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d4003520 con 0x7fc8f40cf290 2024-01-16T15:01:00.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83137844 0 0) 0x7fc8d4003520 con 0x7fc8f40cf290 2024-01-16T15:01:00.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 msgr2=0x7fc8f415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 0x7fc8f415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 >> v1:172.21.15.188:6789/0 conn(0x7fc8f40ae5a0 legacy=0x7fc8f406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f406c750 con 0x7fc8f40cf290 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fc8ec0019f0 con 0x7fc8f40cf290 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 >> v1:172.21.15.141:6789/0 conn(0x7fc8f40cf290 legacy=0x7fc8f40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d40039a0 0x7fc8d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 0x7fc8d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8d4004d00 0x7fc8d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4071450 con 0x7fc8d4004d00 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d400bac0 con 0x7fc8d40039a0 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d400bc80 con 0x7fc8f40b3ae0 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f9e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8d4004d00 0x7fc8d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.956+0000 7fc8f9400640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d40039a0 0x7fc8d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.957+0000 7fc8f8a00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 0x7fc8d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.957+0000 7fc8f9e00640 1 -- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 msgr2=0x7fc8d40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.957+0000 7fc8f9e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f40b3ae0 0x7fc8d40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.957+0000 7fc8f9e00640 1 -- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d40039a0 msgr2=0x7fc8d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.957+0000 7fc8f9e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d40039a0 0x7fc8d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.957+0000 7fc8f9e00640 1 -- 172.21.15.141:0/1995975310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d4003520 con 0x7fc8d4004d00 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.958+0000 7fc8f9e00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8d4004d00 0x7fc8d4005180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec002230 tx=0x7fc8d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db51a0fa2afc6c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.958+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec034630 con 0x7fc8d4004d00 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.958+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec02f070 con 0x7fc8d4004d00 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.958+0000 7fc8f3e00640 1 -- 172.21.15.141:0/1995975310 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec034a10 con 0x7fc8d4004d00 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.959+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1995975310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8d4004d00 msgr2=0x7fc8d4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.959+0000 7fc8fbe00640 1 --2- 172.21.15.141:0/1995975310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8d4004d00 0x7fc8d4005180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec002230 tx=0x7fc8d40057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.959+0000 7fc8f9e00640 1 -- 172.21.15.141:0/1995975310 reap_dead start 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.959+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1995975310 shutdown_connections 2024-01-16T15:01:00.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.959+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1995975310 >> 172.21.15.141:0/1995975310 conn(0x7fc8f406a800 msgr2=0x7fc8f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.959+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1995975310 shutdown_connections 2024-01-16T15:01:00.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.959+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1995975310 wait complete. 2024-01-16T15:01:00.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.961+0000 7fc8fbe00640 1 Processor -- start 2024-01-16T15:01:00.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.961+0000 7fc8fbe00640 1 -- start start 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8fbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d4004d00 0x7fc8f4157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8f40cde30 0x7fc8f40ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8fbe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f4159310 0x7fc8f4159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8fbe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4159cd0 con 0x7fc8f40cde30 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8fbe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4157ed0 con 0x7fc8d4004d00 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8fbe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40cc590 con 0x7fc8f4159310 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8f8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8f40cde30 0x7fc8f40ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8f8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8f40cde30 0x7fc8f40ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54518/0 (socket says 172.21.15.141:54518) 2024-01-16T15:01:00.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.962+0000 7fc8f8a00640 1 -- 172.21.15.141:0/1491793936 learned_addr learned my addr 172.21.15.141:0/1491793936 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f9e00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f4159310 0x7fc8f4159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f9400640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d4004d00 0x7fc8f4157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f8a00640 1 -- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f4159310 msgr2=0x7fc8f4159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f8a00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f4159310 0x7fc8f4159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f8a00640 1 -- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d4004d00 msgr2=0x7fc8f4157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f8a00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d4004d00 0x7fc8f4157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f8a00640 1 -- 172.21.15.141:0/1491793936 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40cc870 con 0x7fc8f40cde30 2024-01-16T15:01:00.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f8a00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8f40cde30 0x7fc8f40ce2b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc010860 tx=0x7fc8dc010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f1600640 1 -- 172.21.15.141:0/1491793936 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc009610 con 0x7fc8f40cde30 2024-01-16T15:01:00.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f1600640 1 -- 172.21.15.141:0/1491793936 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc00eb60 con 0x7fc8f40cde30 2024-01-16T15:01:00.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8f1600640 1 -- 172.21.15.141:0/1491793936 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc01c400 con 0x7fc8f40cde30 2024-01-16T15:01:00.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.963+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f40ccb50 con 0x7fc8f40cde30 2024-01-16T15:01:00.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.964+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f40cd110 con 0x7fc8f40cde30 2024-01-16T15:01:00.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.965+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f40675d0 con 0x7fc8f40cde30 2024-01-16T15:01:00.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.965+0000 7fc8f1600640 1 -- 172.21.15.141:0/1491793936 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fc8dc01c560 con 0x7fc8f40cde30 2024-01-16T15:01:00.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.966+0000 7fc8f1600640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8cc06b590 0x7fc8cc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.966+0000 7fc8f1600640 1 -- 172.21.15.141:0/1491793936 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc8dc08a5b0 con 0x7fc8f40cde30 2024-01-16T15:01:00.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.967+0000 7fc8f9400640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8cc06b590 0x7fc8cc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.968+0000 7fc8f9400640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8cc06b590 0x7fc8cc06da50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4007fd0 tx=0x7fc8e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:00.971+0000 7fc8f1600640 1 -- 172.21.15.141:0/1491793936 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8dc055190 con 0x7fc8f40cde30 2024-01-16T15:01:01.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.189+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8f4070550 con 0x7fc8cc06b590 2024-01-16T15:01:01.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.200+0000 7fc8f1600640 1 -- 172.21.15.141:0/1491793936 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fc8e400d050 con 0x7fc8cc06b590 2024-01-16T15:01:01.181 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0 2024-01-16T15:01:01.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.202+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8cc06b590 msgr2=0x7fc8cc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.202+0000 7fc8fbe00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8cc06b590 0x7fc8cc06da50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4007fd0 tx=0x7fc8e4009040 comp rx=0 tx=0).stop 2024-01-16T15:01:01.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.202+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8f40cde30 msgr2=0x7fc8f40ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.202+0000 7fc8fbe00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8f40cde30 0x7fc8f40ce2b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc010860 tx=0x7fc8dc010d20 comp rx=0 tx=0).stop 2024-01-16T15:01:01.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 shutdown_connections 2024-01-16T15:01:01.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8cc06b590 0x7fc8cc06da50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc8f4159310 0x7fc8f4159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc8f40cde30 0x7fc8f40ce2b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 --2- 172.21.15.141:0/1491793936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8d4004d00 0x7fc8f4157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 >> 172.21.15.141:0/1491793936 conn(0x7fc8f406a800 msgr2=0x7fc8f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 shutdown_connections 2024-01-16T15:01:01.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:01.203+0000 7fc8fbe00640 1 -- 172.21.15.141:0/1491793936 wait complete. 2024-01-16T15:01:01.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:01.201 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0) 2024-01-16T15:01:01.258 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0 2024-01-16T15:01:01.258 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:01:01.258 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:01.259 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:01:03.065 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:03.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:03.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:03.260 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0) 2024-01-16T15:01:03.326 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/82ea8062-acef-4edf-a874-903a7a6528b0 2024-01-16T15:01:03.326 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:01:03.326 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:03.327 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:01:03.328 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone29511 2024-01-16T15:01:03.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.565+0000 7f0bf6000640 1 Processor -- start 2024-01-16T15:01:03.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.566+0000 7f0bf6000640 1 -- start start 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.567+0000 7f0bf6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf013df80 0x7f0bf015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.567+0000 7f0bf6000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0bf00cf9a0 0x7f0bf00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.567+0000 7f0bf6000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf006c470 con 0x7f0bf013e360 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.567+0000 7f0bf6000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf006c750 con 0x7f0bf00cf9a0 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.567+0000 7f0bf6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf006ca30 con 0x7f0bf013df80 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0befe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0bf00cf9a0 0x7f0bf00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0befe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0bf00cf9a0 0x7f0bf00cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57638/0 (socket says 172.21.15.141:57638) 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0befe00640 1 -- 172.21.15.141:0/714056408 learned_addr learned my addr 172.21.15.141:0/714056408 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:03.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0bef400640 1 --2- 172.21.15.141:0/714056408 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf013df80 0x7f0bf015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0befe00640 1 -- 172.21.15.141:0/714056408 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf013df80 msgr2=0x7f0bf015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:03.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0befe00640 1 --2- 172.21.15.141:0/714056408 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf013df80 0x7f0bf015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0befe00640 1 -- 172.21.15.141:0/714056408 >> v1:172.21.15.188:6789/0 conn(0x7f0bf013e360 legacy=0x7f0bf00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:03.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.568+0000 7f0befe00640 1 -- 172.21.15.141:0/714056408 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf006cd10 con 0x7f0bf00cf9a0 2024-01-16T15:01:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.570+0000 7f0befe00640 1 --2- 172.21.15.141:0/714056408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf00cf9a0 0x7f0bf00cd840 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009870 tx=0x7f0be802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e0cddf1c555a06 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.570+0000 7f0beea00640 1 -- 172.21.15.141:0/714056408 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8031640 con 0x7f0bf00cf9a0 2024-01-16T15:01:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.570+0000 7f0beea00640 1 -- 172.21.15.141:0/714056408 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be80317a0 con 0x7f0bf00cf9a0 2024-01-16T15:01:03.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.570+0000 7f0beea00640 1 -- 172.21.15.141:0/714056408 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8031b30 con 0x7f0bf00cf9a0 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 -- 172.21.15.141:0/714056408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf00cf9a0 msgr2=0x7f0bf00cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 --2- 172.21.15.141:0/714056408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf00cf9a0 0x7f0bf00cd840 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009870 tx=0x7f0be802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 -- 172.21.15.141:0/714056408 shutdown_connections 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 --2- 172.21.15.141:0/714056408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf00cf9a0 0x7f0bf00cd840 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 --2- 172.21.15.141:0/714056408 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf013df80 0x7f0bf015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 -- 172.21.15.141:0/714056408 >> 172.21.15.141:0/714056408 conn(0x7f0bf006a800 msgr2=0x7f0bf006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 -- 172.21.15.141:0/714056408 shutdown_connections 2024-01-16T15:01:03.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.571+0000 7f0bf6000640 1 -- 172.21.15.141:0/714056408 wait complete. 2024-01-16T15:01:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.573+0000 7f0bf6000640 1 Processor -- start 2024-01-16T15:01:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.573+0000 7f0bf6000640 1 -- start start 2024-01-16T15:01:03.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0bf6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf00cf9a0 0x7f0bf013cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0bf6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bf013df80 0x7f0bf015d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0bf6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf013e360 0x7f0bf015d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0bf6000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf013dd00 con 0x7f0bf013e360 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0bf6000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf01e7030 con 0x7f0bf013df80 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0bf6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00b1d40 con 0x7f0bf00cf9a0 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf013e360 0x7f0bf015d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf013e360 0x7f0bf015d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57652/0 (socket says 172.21.15.141:57652) 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 -- 172.21.15.141:0/1500309760 learned_addr learned my addr 172.21.15.141:0/1500309760 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 -- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf00cf9a0 msgr2=0x7f0bf013cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf00cf9a0 0x7f0bf013cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 -- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bf013df80 msgr2=0x7f0bf015d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bf013df80 0x7f0bf015d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.574+0000 7f0befe00640 1 -- 172.21.15.141:0/1500309760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80094d0 con 0x7f0bf013e360 2024-01-16T15:01:03.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.575+0000 7f0befe00640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf013e360 0x7f0bf015d8c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0be802b930 tx=0x7f0be80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.575+0000 7f0be7e00640 1 -- 172.21.15.141:0/1500309760 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be803ac60 con 0x7f0bf013e360 2024-01-16T15:01:03.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.575+0000 7f0be7e00640 1 -- 172.21.15.141:0/1500309760 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be803adc0 con 0x7f0bf013e360 2024-01-16T15:01:03.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.575+0000 7f0be7e00640 1 -- 172.21.15.141:0/1500309760 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8044630 con 0x7f0bf013e360 2024-01-16T15:01:03.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.575+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf00b2020 con 0x7f0bf013e360 2024-01-16T15:01:03.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.575+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf00b25e0 con 0x7f0bf013e360 2024-01-16T15:01:03.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.576+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf00675d0 con 0x7f0bf013e360 2024-01-16T15:01:03.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.577+0000 7f0be7e00640 1 -- 172.21.15.141:0/1500309760 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f0be802f050 con 0x7f0bf013e360 2024-01-16T15:01:03.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.578+0000 7f0be7e00640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0bc806b590 0x7f0bc806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.578+0000 7f0be7e00640 1 -- 172.21.15.141:0/1500309760 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0be80ac000 con 0x7f0bf013e360 2024-01-16T15:01:03.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.578+0000 7f0bef400640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0bc806b590 0x7f0bc806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.579+0000 7f0bef400640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0bc806b590 0x7f0bc806da50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0be00070b0 tx=0x7f0be0007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.583+0000 7f0be7e00640 1 -- 172.21.15.141:0/1500309760 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0be8076a20 con 0x7f0bf013e360 2024-01-16T15:01:03.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.806+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone29511", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bf00b1800 con 0x7f0bc806b590 2024-01-16T15:01:03.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.942+0000 7f0be7e00640 1 -- 172.21.15.141:0/1500309760 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0bf00b1800 con 0x7f0bc806b590 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.945+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0bc806b590 msgr2=0x7f0bc806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.945+0000 7f0bf6000640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0bc806b590 0x7f0bc806da50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0be00070b0 tx=0x7f0be0007040 comp rx=0 tx=0).stop 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.945+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf013e360 msgr2=0x7f0bf015d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.945+0000 7f0bf6000640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf013e360 0x7f0bf015d8c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0be802b930 tx=0x7f0be80046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 shutdown_connections 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0bc806b590 0x7f0bc806da50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bf013e360 0x7f0bf015d8c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bf013df80 0x7f0bf015d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 --2- 172.21.15.141:0/1500309760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bf00cf9a0 0x7f0bf013cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 >> 172.21.15.141:0/1500309760 conn(0x7f0bf006a800 msgr2=0x7f0bf00ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:03.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 shutdown_connections 2024-01-16T15:01:03.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:03.946+0000 7f0bf6000640 1 -- 172.21.15.141:0/1500309760 wait complete. 2024-01-16T15:01:03.936 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.953+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.936 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.953+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.936 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.953+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.936 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.953+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.936 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.953+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.946 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone29512 2024-01-16T15:01:03.956 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.977+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.956 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.977+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.956 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.977+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.956 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.977+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:03.956 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:03.977+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:04.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.174+0000 7f7b68600640 1 Processor -- start 2024-01-16T15:01:04.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.174+0000 7f7b68600640 1 -- start start 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.175+0000 7f7b68600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7b60143ae0 0x7f7b600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.175+0000 7f7b68600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7b600cf7d0 0x7f7b6015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b68600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b600cfc30 con 0x7f7b6015e1e0 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b68600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b600b1ee0 con 0x7f7b60143ae0 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b68600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b600b2780 con 0x7f7b600cf7d0 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7b60143ae0 0x7f7b600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7b60143ae0 0x7f7b600cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57654/0 (socket says 172.21.15.141:57654) 2024-01-16T15:01:04.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65c00640 1 -- 172.21.15.141:0/92312567 learned_addr learned my addr 172.21.15.141:0/92312567 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:04.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65200640 1 --2- 172.21.15.141:0/92312567 >> v2:172.21.15.188:3300/0 conn(0x7f7b600cf7d0 0x7f7b6015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65c00640 1 -- 172.21.15.141:0/92312567 >> v2:172.21.15.188:3300/0 conn(0x7f7b600cf7d0 msgr2=0x7f7b6015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65c00640 1 --2- 172.21.15.141:0/92312567 >> v2:172.21.15.188:3300/0 conn(0x7f7b600cf7d0 0x7f7b6015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65c00640 1 -- 172.21.15.141:0/92312567 >> v1:172.21.15.188:6789/0 conn(0x7f7b6015e1e0 legacy=0x7f7b6015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.176+0000 7f7b65c00640 1 -- 172.21.15.141:0/92312567 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6015e990 con 0x7f7b60143ae0 2024-01-16T15:01:04.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.177+0000 7f7b65c00640 1 --2- 172.21.15.141:0/92312567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b600cf290 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7b600cb830 tx=0x7f7b5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524acf3491a91daf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.178+0000 7f7b5fe00640 1 -- 172.21.15.141:0/92312567 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5002ca80 con 0x7f7b60143ae0 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.178+0000 7f7b5fe00640 1 -- 172.21.15.141:0/92312567 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5002cbe0 con 0x7f7b60143ae0 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.178+0000 7f7b5fe00640 1 -- 172.21.15.141:0/92312567 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b500343f0 con 0x7f7b60143ae0 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.178+0000 7f7b68600640 1 -- 172.21.15.141:0/92312567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 msgr2=0x7f7b600cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.178+0000 7f7b68600640 1 --2- 172.21.15.141:0/92312567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b600cf290 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7b600cb830 tx=0x7f7b5002bee0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.178+0000 7f7b68600640 1 -- 172.21.15.141:0/92312567 shutdown_connections 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.178+0000 7f7b68600640 1 --2- 172.21.15.141:0/92312567 >> v2:172.21.15.188:3300/0 conn(0x7f7b600cf7d0 0x7f7b6015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.179+0000 7f7b68600640 1 --2- 172.21.15.141:0/92312567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b600cf290 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.179+0000 7f7b68600640 1 -- 172.21.15.141:0/92312567 >> 172.21.15.141:0/92312567 conn(0x7f7b6006a800 msgr2=0x7f7b6006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.179+0000 7f7b68600640 1 -- 172.21.15.141:0/92312567 shutdown_connections 2024-01-16T15:01:04.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.179+0000 7f7b68600640 1 -- 172.21.15.141:0/92312567 wait complete. 2024-01-16T15:01:04.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.180+0000 7f7b68600640 1 Processor -- start 2024-01-16T15:01:04.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.180+0000 7f7b68600640 1 -- start start 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.181+0000 7f7b68600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b600cf7d0 0x7f7b600ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.181+0000 7f7b68600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b60165ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.181+0000 7f7b68600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e1e0 0x7f7b60142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.181+0000 7f7b68600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b60140010 con 0x7f7b60143ae0 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.181+0000 7f7b68600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6013f440 con 0x7f7b600cf7d0 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.181+0000 7f7b68600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6013f6e0 con 0x7f7b6015e1e0 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b60165ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b60165ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57658/0 (socket says 172.21.15.141:57658) 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 -- 172.21.15.141:0/2903851186 learned_addr learned my addr 172.21.15.141:0/2903851186 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65c00640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b600cf7d0 0x7f7b600ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 -- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e1e0 msgr2=0x7f7b60142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b66600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e1e0 0x7f7b60142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e1e0 0x7f7b60142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 -- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b600cf7d0 msgr2=0x7f7b600ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b600cf7d0 0x7f7b600ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 -- 172.21.15.141:0/2903851186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50009b30 con 0x7f7b60143ae0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.182+0000 7f7b65200640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b60165ba0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c7f0 tx=0x7f7b4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.183+0000 7f7b5d600640 1 -- 172.21.15.141:0/2903851186 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48004220 con 0x7f7b60143ae0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.183+0000 7f7b5d600640 1 -- 172.21.15.141:0/2903851186 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b480043a0 con 0x7f7b60143ae0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.183+0000 7f7b5d600640 1 -- 172.21.15.141:0/2903851186 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48007860 con 0x7f7b60143ae0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.183+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6013f9c0 con 0x7f7b60143ae0 2024-01-16T15:01:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.183+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b600ae730 con 0x7f7b60143ae0 2024-01-16T15:01:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.184+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b600b2bf0 con 0x7f7b60143ae0 2024-01-16T15:01:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.185+0000 7f7b5d600640 1 -- 172.21.15.141:0/2903851186 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f7b48007bd0 con 0x7f7b60143ae0 2024-01-16T15:01:04.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.186+0000 7f7b5d600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b590 0x7f7b3806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.186+0000 7f7b5d600640 1 -- 172.21.15.141:0/2903851186 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7b4808a0f0 con 0x7f7b60143ae0 2024-01-16T15:01:04.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.186+0000 7f7b65c00640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b590 0x7f7b3806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.187+0000 7f7b65c00640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b590 0x7f7b3806da50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7b50035000 tx=0x7f7b50030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.191+0000 7f7b5d600640 1 -- 172.21.15.141:0/2903851186 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b600b2bf0 con 0x7f7b60143ae0 2024-01-16T15:01:04.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.415+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone29512", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b6015e670 con 0x7f7b3806b590 2024-01-16T15:01:04.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.547+0000 7f7b5d600640 1 -- 172.21.15.141:0/2903851186 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7b50034470 con 0x7f7b3806b590 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b590 msgr2=0x7f7b3806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b590 0x7f7b3806da50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7b50035000 tx=0x7f7b50030000 comp rx=0 tx=0).stop 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 msgr2=0x7f7b60165ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b60165ba0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c7f0 tx=0x7f7b4800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 shutdown_connections 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b3806b590 0x7f7b3806da50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b6015e1e0 0x7f7b60142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b60143ae0 0x7f7b60165ba0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 --2- 172.21.15.141:0/2903851186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b600cf7d0 0x7f7b600ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.551+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 >> 172.21.15.141:0/2903851186 conn(0x7f7b6006a800 msgr2=0x7f7b600ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.552+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 shutdown_connections 2024-01-16T15:01:04.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.552+0000 7f7b68600640 1 -- 172.21.15.141:0/2903851186 wait complete. 2024-01-16T15:01:04.557 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone29513 2024-01-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.779+0000 7fb2db200640 1 Processor -- start 2024-01-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.780+0000 7fb2db200640 1 -- start start 2024-01-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.781+0000 7fb2db200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2d4143ae0 0x7fb2d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.781+0000 7fb2db200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb2d415e3b0 0x7fb2d40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.781+0000 7fb2db200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40cfc30 con 0x7fb2d40cf7d0 2024-01-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.781+0000 7fb2db200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d40b1ee0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.781+0000 7fb2db200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d40b2780 con 0x7fb2d415e3b0 2024-01-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2d4143ae0 0x7fb2d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2d4143ae0 0x7fb2d40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57666/0 (socket says 172.21.15.141:57666) 2024-01-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d3e00640 1 -- 172.21.15.141:0/837201637 learned_addr learned my addr 172.21.15.141:0/837201637 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d9200640 1 --2- 172.21.15.141:0/837201637 >> v2:172.21.15.188:3300/0 conn(0x7fb2d415e3b0 0x7fb2d40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d3e00640 1 -- 172.21.15.141:0/837201637 >> v2:172.21.15.188:3300/0 conn(0x7fb2d415e3b0 msgr2=0x7fb2d40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d3e00640 1 --2- 172.21.15.141:0/837201637 >> v2:172.21.15.188:3300/0 conn(0x7fb2d415e3b0 0x7fb2d40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d3e00640 1 -- 172.21.15.141:0/837201637 >> v1:172.21.15.188:6789/0 conn(0x7fb2d40cf7d0 legacy=0x7fb2d415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.782+0000 7fb2d3e00640 1 -- 172.21.15.141:0/837201637 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d415e990 con 0x7fb2d4143ae0 2024-01-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.783+0000 7fb2d3e00640 1 --2- 172.21.15.141:0/837201637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d40cf290 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb2d406e4f0 tx=0x7fb2c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1a6a2b4df94f5b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2d3400640 1 -- 172.21.15.141:0/837201637 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4031870 con 0x7fb2d4143ae0 2024-01-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2d3400640 1 -- 172.21.15.141:0/837201637 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c40319d0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2d3400640 1 -- 172.21.15.141:0/837201637 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4031d60 con 0x7fb2d4143ae0 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2db200640 1 -- 172.21.15.141:0/837201637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 msgr2=0x7fb2d40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2db200640 1 --2- 172.21.15.141:0/837201637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d40cf290 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb2d406e4f0 tx=0x7fb2c402b910 comp rx=0 tx=0).stop 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2db200640 1 -- 172.21.15.141:0/837201637 shutdown_connections 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2db200640 1 --2- 172.21.15.141:0/837201637 >> v2:172.21.15.188:3300/0 conn(0x7fb2d415e3b0 0x7fb2d40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2db200640 1 --2- 172.21.15.141:0/837201637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d40cf290 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.784+0000 7fb2db200640 1 -- 172.21.15.141:0/837201637 >> 172.21.15.141:0/837201637 conn(0x7fb2d406a800 msgr2=0x7fb2d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.785+0000 7fb2db200640 1 -- 172.21.15.141:0/837201637 shutdown_connections 2024-01-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.785+0000 7fb2db200640 1 -- 172.21.15.141:0/837201637 wait complete. 2024-01-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.786+0000 7fb2db200640 1 Processor -- start 2024-01-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.786+0000 7fb2db200640 1 -- start start 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.787+0000 7fb2db200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2d40cf7d0 0x7fb2d4139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.787+0000 7fb2db200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d413bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.787+0000 7fb2db200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2d415e3b0 0x7fb2d4159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.787+0000 7fb2db200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d4138c30 con 0x7fb2d4143ae0 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.787+0000 7fb2db200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d4138f10 con 0x7fb2d415e3b0 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.787+0000 7fb2db200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d41391f0 con 0x7fb2d40cf7d0 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d413bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d413bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57682/0 (socket says 172.21.15.141:57682) 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 -- 172.21.15.141:0/2372609288 learned_addr learned my addr 172.21.15.141:0/2372609288 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2d9200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2d415e3b0 0x7fb2d4159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2d3e00640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2d40cf7d0 0x7fb2d4139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 -- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2d40cf7d0 msgr2=0x7fb2d4139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2d40cf7d0 0x7fb2d4139cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 -- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2d415e3b0 msgr2=0x7fb2d4159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2d415e3b0 0x7fb2d4159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 -- 172.21.15.141:0/2372609288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40094d0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.788+0000 7fb2cb200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d413bb80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00c430 tx=0x7fb2bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.789+0000 7fb2d0c00640 1 -- 172.21.15.141:0/2372609288 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc011790 con 0x7fb2d4143ae0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.789+0000 7fb2d0c00640 1 -- 172.21.15.141:0/2372609288 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc0118f0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.789+0000 7fb2d0c00640 1 -- 172.21.15.141:0/2372609288 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc011ce0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.789+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d41394d0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.789+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d413c0c0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.790+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d41399f0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.791+0000 7fb2d0c00640 1 -- 172.21.15.141:0/2372609288 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7fb2bc01c460 con 0x7fb2d4143ae0 2024-01-16T15:01:04.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.792+0000 7fb2d0c00640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2ac06b6b0 0x7fb2ac06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.792+0000 7fb2d0c00640 1 -- 172.21.15.141:0/2372609288 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb2bc08a2b0 con 0x7fb2d4143ae0 2024-01-16T15:01:04.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.793+0000 7fb2d3e00640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2ac06b6b0 0x7fb2ac06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.794+0000 7fb2d3e00640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2ac06b6b0 0x7fb2ac06db70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4036000 tx=0x7fb2c403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:04.797+0000 7fb2d0c00640 1 -- 172.21.15.141:0/2372609288 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2bc054b00 con 0x7fb2d4143ae0 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:05.023+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone29513", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2d40b2410 con 0x7fb2ac06b6b0 2024-01-16T15:01:08.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.299+0000 7fb2d0c00640 1 -- 172.21.15.141:0/2372609288 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb2c4031bc0 con 0x7fb2ac06b6b0 2024-01-16T15:01:08.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.303+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2ac06b6b0 msgr2=0x7fb2ac06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.303+0000 7fb2db200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2ac06b6b0 0x7fb2ac06db70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4036000 tx=0x7fb2c403c040 comp rx=0 tx=0).stop 2024-01-16T15:01:08.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.303+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 msgr2=0x7fb2d413bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.303+0000 7fb2db200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d413bb80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00c430 tx=0x7fb2bc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 shutdown_connections 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2ac06b6b0 0x7fb2ac06db70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2d415e3b0 0x7fb2d4159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2d4143ae0 0x7fb2d413bb80 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 --2- 172.21.15.141:0/2372609288 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2d40cf7d0 0x7fb2d4139cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 >> 172.21.15.141:0/2372609288 conn(0x7fb2d406a800 msgr2=0x7fb2d4070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 shutdown_connections 2024-01-16T15:01:08.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.304+0000 7fb2db200640 1 -- 172.21.15.141:0/2372609288 wait complete. 2024-01-16T15:01:08.305 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone29514 2024-01-16T15:01:08.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.533+0000 7f982e400640 1 Processor -- start 2024-01-16T15:01:08.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.534+0000 7f982e400640 1 -- start start 2024-01-16T15:01:08.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.535+0000 7f982e400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9828160920 0x7f98280b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.535+0000 7f982e400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f98280cf9f0 0x7f98280cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.535+0000 7f982e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9828161010 0x7f98281614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.535+0000 7f982e400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98280d03e0 con 0x7f98280cf9f0 2024-01-16T15:01:08.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.535+0000 7f982e400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98280d0a10 con 0x7f9828160920 2024-01-16T15:01:08.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.535+0000 7f982e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98280cf080 con 0x7f9828161010 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f98280cf9f0 0x7f98280cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f98280cf9f0 0x7f98280cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57688/0 (socket says 172.21.15.141:57688) 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 -- 172.21.15.141:0/1812489629 learned_addr learned my addr 172.21.15.141:0/1812489629 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f982d000640 1 --2- 172.21.15.141:0/1812489629 >> v2:172.21.15.188:3300/0 conn(0x7f9828160920 0x7f98280b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f982da00640 1 --2- 172.21.15.141:0/1812489629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9828161010 0x7f98281614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 -- 172.21.15.141:0/1812489629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9828161010 msgr2=0x7f98281614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 --2- 172.21.15.141:0/1812489629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9828161010 0x7f98281614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 -- 172.21.15.141:0/1812489629 >> v2:172.21.15.188:3300/0 conn(0x7f9828160920 msgr2=0x7f98280b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 --2- 172.21.15.141:0/1812489629 >> v2:172.21.15.188:3300/0 conn(0x7f9828160920 0x7f98280b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.536+0000 7f9827e00640 1 -- 172.21.15.141:0/1812489629 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98280c6be0 con 0x7f98280cf9f0 2024-01-16T15:01:08.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.537+0000 7f9827e00640 1 --2- 172.21.15.141:0/1812489629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98280cf9f0 0x7f98280cfea0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f982808c060 tx=0x7f981002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b290d0e5b989cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.537+0000 7f9827400640 1 -- 172.21.15.141:0/1812489629 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810031860 con 0x7f98280cf9f0 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.537+0000 7f9827400640 1 -- 172.21.15.141:0/1812489629 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100319c0 con 0x7f98280cf9f0 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.537+0000 7f9827400640 1 -- 172.21.15.141:0/1812489629 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810031d50 con 0x7f98280cf9f0 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 -- 172.21.15.141:0/1812489629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98280cf9f0 msgr2=0x7f98280cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 --2- 172.21.15.141:0/1812489629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98280cf9f0 0x7f98280cfea0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f982808c060 tx=0x7f981002b900 comp rx=0 tx=0).stop 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 -- 172.21.15.141:0/1812489629 shutdown_connections 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 --2- 172.21.15.141:0/1812489629 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9828161010 0x7f98281614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 --2- 172.21.15.141:0/1812489629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98280cf9f0 0x7f98280cfea0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 --2- 172.21.15.141:0/1812489629 >> v2:172.21.15.188:3300/0 conn(0x7f9828160920 0x7f98280b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 -- 172.21.15.141:0/1812489629 >> 172.21.15.141:0/1812489629 conn(0x7f98280d3090 msgr2=0x7f98280d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 -- 172.21.15.141:0/1812489629 shutdown_connections 2024-01-16T15:01:08.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.538+0000 7f982e400640 1 -- 172.21.15.141:0/1812489629 wait complete. 2024-01-16T15:01:08.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.540+0000 7f982e400640 1 Processor -- start 2024-01-16T15:01:08.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.540+0000 7f982e400640 1 -- start start 2024-01-16T15:01:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f982e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98280cf9f0 0x7f98280b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f982e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9828160920 0x7f98280cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f982e400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9828161010 0x7f98280cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f982e400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98280b8c80 con 0x7f9828160920 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f982e400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98280b6a00 con 0x7f9828161010 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f982e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98280b5df0 con 0x7f98280cf9f0 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f9827e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9828160920 0x7f98280cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f9827e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9828160920 0x7f98280cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57704/0 (socket says 172.21.15.141:57704) 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f9827e00640 1 -- 172.21.15.141:0/328275395 learned_addr learned my addr 172.21.15.141:0/328275395 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.541+0000 7f982d000640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98280cf9f0 0x7f98280b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f982da00640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9828161010 0x7f98280cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9827e00640 1 -- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98280cf9f0 msgr2=0x7f98280b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9827e00640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98280cf9f0 0x7f98280b6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9827e00640 1 -- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9828161010 msgr2=0x7f98280cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9827e00640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9828161010 0x7f98280cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9827e00640 1 -- 172.21.15.141:0/328275395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100094d0 con 0x7f9828160920 2024-01-16T15:01:08.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f982da00640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9828161010 0x7f98280cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:08.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9827e00640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9828160920 0x7f98280cab70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f981002b940 tx=0x7f98100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9824c00640 1 -- 172.21.15.141:0/328275395 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810031850 con 0x7f9828160920 2024-01-16T15:01:08.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9824c00640 1 -- 172.21.15.141:0/328275395 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100443f0 con 0x7f9828160920 2024-01-16T15:01:08.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f9824c00640 1 -- 172.21.15.141:0/328275395 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100447a0 con 0x7f9828160920 2024-01-16T15:01:08.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.542+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98280b60d0 con 0x7f9828160920 2024-01-16T15:01:08.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.543+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98280b6690 con 0x7f9828160920 2024-01-16T15:01:08.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.544+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98280b6b70 con 0x7f9828160920 2024-01-16T15:01:08.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.545+0000 7f9824c00640 1 -- 172.21.15.141:0/328275395 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f9810044900 con 0x7f9828160920 2024-01-16T15:01:08.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.545+0000 7f9824c00640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f980006b6e0 0x7f980006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.546+0000 7f9824c00640 1 -- 172.21.15.141:0/328275395 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f98100acb20 con 0x7f9828160920 2024-01-16T15:01:08.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.546+0000 7f982d000640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f980006b6e0 0x7f980006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.547+0000 7f982d000640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f980006b6e0 0x7f980006dba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9818006fd0 tx=0x7f9818008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.550+0000 7f9824c00640 1 -- 172.21.15.141:0/328275395 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98100773d0 con 0x7f9828160920 2024-01-16T15:01:08.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:08.777+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone29514", "target": ["mon-mgr", ""]}) v1 -- 0x7f98280ceda0 con 0x7f980006b6e0 2024-01-16T15:01:13.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.300+0000 7f9824c00640 1 -- 172.21.15.141:0/328275395 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f981800c020 con 0x7f980006b6e0 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f980006b6e0 msgr2=0x7f980006dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f980006b6e0 0x7f980006dba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9818006fd0 tx=0x7f9818008040 comp rx=0 tx=0).stop 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9828160920 msgr2=0x7f98280cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9828160920 0x7f98280cab70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f981002b940 tx=0x7f98100046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 shutdown_connections 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f980006b6e0 0x7f980006dba0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9828161010 0x7f98280cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9828160920 0x7f98280cab70 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 --2- 172.21.15.141:0/328275395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98280cf9f0 0x7f98280b6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.303+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 >> 172.21.15.141:0/328275395 conn(0x7f98280d3090 msgr2=0x7f982806b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:13.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.304+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 shutdown_connections 2024-01-16T15:01:13.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.304+0000 7f982e400640 1 -- 172.21.15.141:0/328275395 wait complete. 2024-01-16T15:01:13.306 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:01:13.306 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone29511 2024-01-16T15:01:13.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.527+0000 7f2b5fa00640 1 Processor -- start 2024-01-16T15:01:13.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.527+0000 7f2b5fa00640 1 -- start start 2024-01-16T15:01:13.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.528+0000 7f2b5fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b5815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.528+0000 7f2b5fa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2b5815e1e0 0x7f2b580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.528+0000 7f2b5fa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580b1ee0 con 0x7f2b58143ae0 2024-01-16T15:01:13.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.528+0000 7f2b5fa00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b580b2780 con 0x7f2b5815e1e0 2024-01-16T15:01:13.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.528+0000 7f2b5fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5815e990 con 0x7f2b580cf9a0 2024-01-16T15:01:13.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.529+0000 7f2b5da00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2b5815e1e0 0x7f2b580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.529+0000 7f2b4fc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b5815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.529+0000 7f2b5d000640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f2b58143ae0 0x7f2b580cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:43890/0 (socket says 172.21.15.141:43890) 2024-01-16T15:01:13.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.529+0000 7f2b5d000640 1 -- 172.21.15.141:0/2860118997 learned_addr learned my addr 172.21.15.141:0/2860118997 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:13.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.530+0000 7f2b5da00640 1 -- 172.21.15.141:0/2860118997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 msgr2=0x7f2b5815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:13.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.530+0000 7f2b5da00640 1 --2- 172.21.15.141:0/2860118997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b5815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.530+0000 7f2b5da00640 1 -- 172.21.15.141:0/2860118997 >> v1:172.21.15.188:6789/0 conn(0x7f2b58143ae0 legacy=0x7f2b580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.530+0000 7f2b5da00640 1 -- 172.21.15.141:0/2860118997 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b580b2130 con 0x7f2b5815e1e0 2024-01-16T15:01:13.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.531+0000 7f2b5da00640 1 --2- 172.21.15.141:0/2860118997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b5815e1e0 0x7f2b580702c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b500097a0 tx=0x7f2b5002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3dd630fc7b39ee8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.531+0000 7f2b57e00640 1 -- 172.21.15.141:0/2860118997 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5002d040 con 0x7f2b5815e1e0 2024-01-16T15:01:13.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b57e00640 1 -- 172.21.15.141:0/2860118997 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b50031810 con 0x7f2b5815e1e0 2024-01-16T15:01:13.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b57e00640 1 -- 172.21.15.141:0/2860118997 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50031ba0 con 0x7f2b5815e1e0 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 -- 172.21.15.141:0/2860118997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b5815e1e0 msgr2=0x7f2b580702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/2860118997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b5815e1e0 0x7f2b580702c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b500097a0 tx=0x7f2b5002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 -- 172.21.15.141:0/2860118997 shutdown_connections 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/2860118997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b5815e1e0 0x7f2b580702c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/2860118997 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b5815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 -- 172.21.15.141:0/2860118997 >> 172.21.15.141:0/2860118997 conn(0x7f2b5806a800 msgr2=0x7f2b5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 -- 172.21.15.141:0/2860118997 shutdown_connections 2024-01-16T15:01:13.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.532+0000 7f2b5fa00640 1 -- 172.21.15.141:0/2860118997 wait complete. 2024-01-16T15:01:13.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.534+0000 7f2b5fa00640 1 Processor -- start 2024-01-16T15:01:13.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.534+0000 7f2b5fa00640 1 -- start start 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b58159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5fa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b58143ae0 0x7f2b58132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5fa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b5815e1e0 0x7f2b580a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5fa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b58159f40 con 0x7f2b5815e1e0 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5fa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b580a9170 con 0x7f2b58143ae0 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b580a8560 con 0x7f2b580cf9a0 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5da00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b5815e1e0 0x7f2b580a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5da00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b5815e1e0 0x7f2b580a9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38850/0 (socket says 172.21.15.141:38850) 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.535+0000 7f2b5da00640 1 -- 172.21.15.141:0/51829993 learned_addr learned my addr 172.21.15.141:0/51829993 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b5d000640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b58159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b4fc00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b58143ae0 0x7f2b58132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b5da00640 1 -- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 msgr2=0x7f2b58159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b5da00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b58159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b5da00640 1 -- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b58143ae0 msgr2=0x7f2b58132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b5da00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b58143ae0 0x7f2b58132af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b5da00640 1 -- 172.21.15.141:0/51829993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50009400 con 0x7f2b5815e1e0 2024-01-16T15:01:13.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.536+0000 7f2b5da00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b5815e1e0 0x7f2b580a9600 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2b5002b8f0 tx=0x7f2b500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.537+0000 7f2b55600640 1 -- 172.21.15.141:0/51829993 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5003ac60 con 0x7f2b5815e1e0 2024-01-16T15:01:13.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.537+0000 7f2b55600640 1 -- 172.21.15.141:0/51829993 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5003ade0 con 0x7f2b5815e1e0 2024-01-16T15:01:13.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.537+0000 7f2b55600640 1 -- 172.21.15.141:0/51829993 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50042630 con 0x7f2b5815e1e0 2024-01-16T15:01:13.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.537+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b580a8840 con 0x7f2b5815e1e0 2024-01-16T15:01:13.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.537+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b580a8e00 con 0x7f2b5815e1e0 2024-01-16T15:01:13.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.538+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b580675d0 con 0x7f2b5815e1e0 2024-01-16T15:01:13.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.539+0000 7f2b55600640 1 -- 172.21.15.141:0/51829993 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f2b5004b3b0 con 0x7f2b5815e1e0 2024-01-16T15:01:13.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.540+0000 7f2b55600640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b3006b6b0 0x7f2b3006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.540+0000 7f2b55600640 1 -- 172.21.15.141:0/51829993 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b500ac540 con 0x7f2b5815e1e0 2024-01-16T15:01:13.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.540+0000 7f2b5d000640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b3006b6b0 0x7f2b3006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.541+0000 7f2b5d000640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b3006b6b0 0x7f2b3006db70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2b48005c00 tx=0x7f2b48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.545+0000 7f2b55600640 1 -- 172.21.15.141:0/51829993 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b50076c40 con 0x7f2b5815e1e0 2024-01-16T15:01:13.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:13.772+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone29511", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b5815e640 con 0x7f2b3006b6b0 2024-01-16T15:01:18.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.149+0000 7f2b55600640 1 -- 172.21.15.141:0/51829993 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f2b4800ec40 con 0x7f2b3006b6b0 2024-01-16T15:01:18.130 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:18.130 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:01:18.130 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:01:18.130 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:01:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:01:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:01:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:01:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:18.131 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:18.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b3006b6b0 msgr2=0x7f2b3006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:18.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b3006b6b0 0x7f2b3006db70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2b48005c00 tx=0x7f2b48009040 comp rx=0 tx=0).stop 2024-01-16T15:01:18.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b5815e1e0 msgr2=0x7f2b580a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:18.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b5815e1e0 0x7f2b580a9600 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2b5002b8f0 tx=0x7f2b500046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 shutdown_connections 2024-01-16T15:01:18.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b3006b6b0 0x7f2b3006db70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b5815e1e0 0x7f2b580a9600 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b58143ae0 0x7f2b58132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 --2- 172.21.15.141:0/51829993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b580cf9a0 0x7f2b58159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.153+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 >> 172.21.15.141:0/51829993 conn(0x7f2b5806a800 msgr2=0x7f2b5815ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:18.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.154+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 shutdown_connections 2024-01-16T15:01:18.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:18.154+0000 7f2b5fa00640 1 -- 172.21.15.141:0/51829993 wait complete. 2024-01-16T15:01:18.155 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:01:19.156 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone29511 2024-01-16T15:01:19.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.397+0000 7f851ca00640 1 Processor -- start 2024-01-16T15:01:19.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.397+0000 7f851ca00640 1 -- start start 2024-01-16T15:01:19.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f851ca00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85140d0b60 0x7f85140ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f851ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f851406f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f851ca00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8514071de0 con 0x7f851406a3e0 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f851ca00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85140af270 con 0x7f85140d0b60 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f851ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8514072030 con 0x7f85140aede0 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f8519600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85140d0b60 0x7f85140ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f8519600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85140d0b60 0x7f85140ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38854/0 (socket says 172.21.15.141:38854) 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f8519600640 1 -- 172.21.15.141:0/1321415655 learned_addr learned my addr 172.21.15.141:0/1321415655 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f851aa00640 1 --2- 172.21.15.141:0/1321415655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f851406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f8519600640 1 -- 172.21.15.141:0/1321415655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 msgr2=0x7f851406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f8519600640 1 --2- 172.21.15.141:0/1321415655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f851406f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.399+0000 7f8519600640 1 -- 172.21.15.141:0/1321415655 >> v1:172.21.15.188:6789/0 conn(0x7f851406a3e0 legacy=0x7f851406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.400+0000 7f8519600640 1 -- 172.21.15.141:0/1321415655 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8514072310 con 0x7f85140d0b60 2024-01-16T15:01:19.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.401+0000 7f8519600640 1 --2- 172.21.15.141:0/1321415655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85140d0b60 0x7f85140ac8d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009870 tx=0x7f84fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2169e308d7f4f62 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.401+0000 7f8518c00640 1 -- 172.21.15.141:0/1321415655 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc030840 con 0x7f85140d0b60 2024-01-16T15:01:19.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.401+0000 7f8518c00640 1 -- 172.21.15.141:0/1321415655 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc0309a0 con 0x7f85140d0b60 2024-01-16T15:01:19.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.401+0000 7f8518c00640 1 -- 172.21.15.141:0/1321415655 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc030d30 con 0x7f85140d0b60 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.401+0000 7f851ca00640 1 -- 172.21.15.141:0/1321415655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85140d0b60 msgr2=0x7f85140ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.401+0000 7f851ca00640 1 --2- 172.21.15.141:0/1321415655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85140d0b60 0x7f85140ac8d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009870 tx=0x7f84fc02b980 comp rx=0 tx=0).stop 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.402+0000 7f851ca00640 1 -- 172.21.15.141:0/1321415655 shutdown_connections 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.402+0000 7f851ca00640 1 --2- 172.21.15.141:0/1321415655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85140d0b60 0x7f85140ac8d0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.402+0000 7f851ca00640 1 --2- 172.21.15.141:0/1321415655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f851406f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.402+0000 7f851ca00640 1 -- 172.21.15.141:0/1321415655 >> 172.21.15.141:0/1321415655 conn(0x7f85140cfef0 msgr2=0x7f85140d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.402+0000 7f851ca00640 1 -- 172.21.15.141:0/1321415655 shutdown_connections 2024-01-16T15:01:19.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.402+0000 7f851ca00640 1 -- 172.21.15.141:0/1321415655 wait complete. 2024-01-16T15:01:19.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.403+0000 7f851ca00640 1 Processor -- start 2024-01-16T15:01:19.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.404+0000 7f851ca00640 1 -- start start 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.404+0000 7f851ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f851406a3e0 0x7f85140cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.404+0000 7f851ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f85141f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.404+0000 7f851ca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85140d0b60 0x7f85140cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.404+0000 7f851ca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85141e79b0 con 0x7f851406a3e0 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.404+0000 7f851ca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8514159640 con 0x7f85140d0b60 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.404+0000 7f851ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8514159920 con 0x7f85140aede0 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f851406a3e0 0x7f85140cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f851406a3e0 0x7f85140cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38858/0 (socket says 172.21.15.141:38858) 2024-01-16T15:01:19.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 -- 172.21.15.141:0/940978642 learned_addr learned my addr 172.21.15.141:0/940978642 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851aa00640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85140d0b60 0x7f85140cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f8519600640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f85141f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 -- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 msgr2=0x7f85141f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f85141f2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 -- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85140d0b60 msgr2=0x7f85140cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85140d0b60 0x7f85140cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 -- 172.21.15.141:0/940978642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc0094d0 con 0x7f851406a3e0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.405+0000 7f851a000640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f851406a3e0 0x7f85140cd9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c870 tx=0x7f850c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.406+0000 7f8512000640 1 -- 172.21.15.141:0/940978642 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c003b70 con 0x7f851406a3e0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.406+0000 7f8512000640 1 -- 172.21.15.141:0/940978642 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c003cf0 con 0x7f851406a3e0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.406+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8514159c00 con 0x7f851406a3e0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.406+0000 7f8512000640 1 -- 172.21.15.141:0/940978642 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c00aa90 con 0x7f851406a3e0 2024-01-16T15:01:19.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.406+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85140cdf00 con 0x7f851406a3e0 2024-01-16T15:01:19.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.407+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85141e7780 con 0x7f851406a3e0 2024-01-16T15:01:19.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.408+0000 7f8512000640 1 -- 172.21.15.141:0/940978642 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f850c00abf0 con 0x7f851406a3e0 2024-01-16T15:01:19.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.409+0000 7f8512000640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84ec06b6b0 0x7f84ec06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.409+0000 7f8512000640 1 -- 172.21.15.141:0/940978642 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f850c08a1d0 con 0x7f851406a3e0 2024-01-16T15:01:19.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.409+0000 7f8519600640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84ec06b6b0 0x7f84ec06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.411+0000 7f8519600640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84ec06b6b0 0x7f84ec06db70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f84fc036000 tx=0x7f84fc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.413+0000 7f8512000640 1 -- 172.21.15.141:0/940978642 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f850c054a70 con 0x7f851406a3e0 2024-01-16T15:01:19.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.643+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone29511", "target": ["mon-mgr", ""]}) v1 -- 0x7f851406eb60 con 0x7f84ec06b6b0 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.666+0000 7f8512000640 1 -- 172.21.15.141:0/940978642 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f84fc030b90 con 0x7f84ec06b6b0 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:01:19.647 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 0.0, 2024-01-16T15:01:19.648 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:01:19.648 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "0/3", 2024-01-16T15:01:19.648 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:01:19.648 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:01:19.648 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:19.648 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:19.648 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:19.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84ec06b6b0 msgr2=0x7f84ec06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84ec06b6b0 0x7f84ec06db70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f84fc036000 tx=0x7f84fc03b040 comp rx=0 tx=0).stop 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f851406a3e0 msgr2=0x7f85140cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f851406a3e0 0x7f85140cd9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c870 tx=0x7f850c00cd30 comp rx=0 tx=0).stop 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 shutdown_connections 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f84ec06b6b0 0x7f84ec06db70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85140d0b60 0x7f85140cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85140aede0 0x7f85141f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 --2- 172.21.15.141:0/940978642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f851406a3e0 0x7f85140cd9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 >> 172.21.15.141:0/940978642 conn(0x7f85140cfef0 msgr2=0x7f851406d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.669+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 shutdown_connections 2024-01-16T15:01:19.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.670+0000 7f851ca00640 1 -- 172.21.15.141:0/940978642 wait complete. 2024-01-16T15:01:19.668 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:19.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.886+0000 7f7f7d000640 1 Processor -- start 2024-01-16T15:01:19.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.886+0000 7f7f7d000640 1 -- start start 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f7d000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7f78157c30 0x7f7f780cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f7d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f78070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f7d000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7f78071030 0x7f7f7806dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f7d000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7815ea30 con 0x7f7f78071030 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f7d000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7806d860 con 0x7f7f78157c30 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f7d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f78071460 con 0x7f7f781618f0 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f76800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7f78071030 0x7f7f7806dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f76800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7f78071030 0x7f7f7806dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38874/0 (socket says 172.21.15.141:38874) 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f76800640 1 -- 172.21.15.141:0/3903584797 learned_addr learned my addr 172.21.15.141:0/3903584797 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.888+0000 7f7f6fe00640 1 --2- 172.21.15.141:0/3903584797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f78070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.889+0000 7f7f76800640 1 -- 172.21.15.141:0/3903584797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 msgr2=0x7f7f78070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.889+0000 7f7f76800640 1 --2- 172.21.15.141:0/3903584797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f78070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.889+0000 7f7f75e00640 1 --2- 172.21.15.141:0/3903584797 >> v2:172.21.15.188:3300/0 conn(0x7f7f78157c30 0x7f7f780cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.889+0000 7f7f76800640 1 -- 172.21.15.141:0/3903584797 >> v2:172.21.15.188:3300/0 conn(0x7f7f78157c30 msgr2=0x7f7f780cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:19.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.889+0000 7f7f76800640 1 --2- 172.21.15.141:0/3903584797 >> v2:172.21.15.188:3300/0 conn(0x7f7f78157c30 0x7f7f780cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.889+0000 7f7f76800640 1 -- 172.21.15.141:0/3903584797 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7815deb0 con 0x7f7f78071030 2024-01-16T15:01:19.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.889+0000 7f7f75e00640 1 --2- 172.21.15.141:0/3903584797 >> v2:172.21.15.188:3300/0 conn(0x7f7f78157c30 0x7f7f780cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:19.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.890+0000 7f7f76800640 1 --2- 172.21.15.141:0/3903584797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f7806dcb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7f780723f0 tx=0x7f7f7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd7b5a18e4ee2a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.890+0000 7f7f75400640 1 -- 172.21.15.141:0/3903584797 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7002ca70 con 0x7f7f78071030 2024-01-16T15:01:19.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.890+0000 7f7f75400640 1 -- 172.21.15.141:0/3903584797 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7002cbd0 con 0x7f7f78071030 2024-01-16T15:01:19.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.890+0000 7f7f75400640 1 -- 172.21.15.141:0/3903584797 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f700343f0 con 0x7f7f78071030 2024-01-16T15:01:19.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 -- 172.21.15.141:0/3903584797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 msgr2=0x7f7f7806dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 --2- 172.21.15.141:0/3903584797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f7806dcb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7f780723f0 tx=0x7f7f7002bed0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 -- 172.21.15.141:0/3903584797 shutdown_connections 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 --2- 172.21.15.141:0/3903584797 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f7806dcb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 --2- 172.21.15.141:0/3903584797 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f78070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 --2- 172.21.15.141:0/3903584797 >> v2:172.21.15.188:3300/0 conn(0x7f7f78157c30 0x7f7f780cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 -- 172.21.15.141:0/3903584797 >> 172.21.15.141:0/3903584797 conn(0x7f7f780cf700 msgr2=0x7f7f780cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 -- 172.21.15.141:0/3903584797 shutdown_connections 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.891+0000 7f7f7d000640 1 -- 172.21.15.141:0/3903584797 wait complete. 2024-01-16T15:01:19.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.893+0000 7f7f7d000640 1 Processor -- start 2024-01-16T15:01:19.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.893+0000 7f7f7d000640 1 -- start start 2024-01-16T15:01:19.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f7d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f781589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f7d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f78157c30 0x7f7f781567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f7d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f780aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f7d000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f78158670 con 0x7f7f78071030 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f7d000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f781580b0 con 0x7f7f78157c30 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f7d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f78158390 con 0x7f7f781618f0 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f75e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f781589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f75e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f781589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38882/0 (socket says 172.21.15.141:38882) 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.894+0000 7f7f75e00640 1 -- 172.21.15.141:0/2069274706 learned_addr learned my addr 172.21.15.141:0/2069274706 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f76800640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f780aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f6fe00640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f78157c30 0x7f7f781567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f75e00640 1 -- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 msgr2=0x7f7f780aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f75e00640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f780aee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f75e00640 1 -- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f78157c30 msgr2=0x7f7f781567c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f75e00640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f78157c30 0x7f7f781567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f75e00640 1 -- 172.21.15.141:0/2069274706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f70009b30 con 0x7f7f78071030 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f75e00640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f781589a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800c7f0 tx=0x7f7f6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f6e000640 1 -- 172.21.15.141:0/2069274706 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68007820 con 0x7f7f78071030 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f6e000640 1 -- 172.21.15.141:0/2069274706 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f68007980 con 0x7f7f78071030 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f6e000640 1 -- 172.21.15.141:0/2069274706 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68007d30 con 0x7f7f78071030 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f780afa60 con 0x7f7f78071030 2024-01-16T15:01:19.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.895+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f78158ee0 con 0x7f7f78071030 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.897+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f780d0060 con 0x7f7f78071030 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.897+0000 7f7f6e000640 1 -- 172.21.15.141:0/2069274706 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f7f680113f0 con 0x7f7f78071030 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.899+0000 7f7f6e000640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f5006b6b0 0x7f7f5006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.899+0000 7f7f6e000640 1 -- 172.21.15.141:0/2069274706 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f6808a760 con 0x7f7f78071030 2024-01-16T15:01:19.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.899+0000 7f7f6fe00640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f5006b6b0 0x7f7f5006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.900+0000 7f7f6fe00640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f5006b6b0 0x7f7f5006db70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7f780b45a0 tx=0x7f7f60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:19.903+0000 7f7f6e000640 1 -- 172.21.15.141:0/2069274706 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f68054e40 con 0x7f7f78071030 2024-01-16T15:01:20.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.166+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7f780675d0 con 0x7f7f78071030 2024-01-16T15:01:20.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.166+0000 7f7f6e000640 1 -- 172.21.15.141:0/2069274706 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2092 (secure 0 0 0) 0x7f7f68058460 con 0x7f7f78071030 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:20.147 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 272, 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:20.148 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:20.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 801, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222213053, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6704439296, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766389673984, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 426, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 12197, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 3 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:20.150 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:20.151 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:01:20.152 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:20.153 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:20.153 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:20.153 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:20.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f5006b6b0 msgr2=0x7f7f5006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f5006b6b0 0x7f7f5006db70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7f780b45a0 tx=0x7f7f60008040 comp rx=0 tx=0).stop 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 msgr2=0x7f7f781589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f781589a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800c7f0 tx=0x7f7f6800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 shutdown_connections 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f5006b6b0 0x7f7f5006db70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f781618f0 0x7f7f780aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f78157c30 0x7f7f781567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 --2- 172.21.15.141:0/2069274706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f78071030 0x7f7f781589a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 >> 172.21.15.141:0/2069274706 conn(0x7f7f780cf700 msgr2=0x7f7f7815f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 shutdown_connections 2024-01-16T15:01:20.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:20.169+0000 7f7f7d000640 1 -- 172.21.15.141:0/2069274706 wait complete. 2024-01-16T15:01:22.167 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:22.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.432+0000 7fec87200640 1 Processor -- start 2024-01-16T15:01:22.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.432+0000 7fec87200640 1 -- start start 2024-01-16T15:01:22.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.434+0000 7fec87200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec80161740 0x7fec80168380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.434+0000 7fec87200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800cf260 con 0x7fec800d0560 2024-01-16T15:01:22.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.434+0000 7fec87200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800cf4c0 con 0x7fec800c9870 2024-01-16T15:01:22.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.434+0000 7fec87200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec800d2ef0 con 0x7fec80161740 2024-01-16T15:01:22.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.435+0000 7fec7f400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fec800d0560 0x7fec80161030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52918/0 (socket says 172.21.15.141:52918) 2024-01-16T15:01:22.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.435+0000 7fec7f400640 1 -- 172.21.15.141:0/1476947697 learned_addr learned my addr 172.21.15.141:0/1476947697 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:22.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.435+0000 7fec85200640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec80161740 0x7fec80168380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.435+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362847247 0 0) 0x7fec800cf260 con 0x7fec800d0560 2024-01-16T15:01:22.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec60003520 con 0x7fec800d0560 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 272853111 0 0) 0x7fec60003520 con 0x7fec800d0560 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec80161740 msgr2=0x7fec80168380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec80161740 0x7fec80168380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 >> v1:172.21.15.188:6789/0 conn(0x7fec800c9870 legacy=0x7fec800ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec800cf7a0 con 0x7fec800d0560 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fec68001690 con 0x7fec800d0560 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 >> v1:172.21.15.141:6789/0 conn(0x7fec800d0560 legacy=0x7fec80161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec600039a0 0x7fec60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec600047c0 0x7fec60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec60005180 0x7fec6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec800d2ef0 con 0x7fec600039a0 2024-01-16T15:01:22.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6000bfa0 con 0x7fec60005180 2024-01-16T15:01:22.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.436+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6000c160 con 0x7fec600047c0 2024-01-16T15:01:22.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.437+0000 7fec7fe00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec600039a0 0x7fec60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.437+0000 7fec7fe00640 1 -- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec600047c0 msgr2=0x7fec60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.437+0000 7fec85200640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec60005180 0x7fec6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.437+0000 7fec7fe00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec600047c0 0x7fec60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.437+0000 7fec7fe00640 1 -- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec60005180 msgr2=0x7fec6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.437+0000 7fec7fe00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec60005180 0x7fec6000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.437+0000 7fec7fe00640 1 -- 172.21.15.141:0/1476947697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec800cf7a0 con 0x7fec600039a0 2024-01-16T15:01:22.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.439+0000 7fec7fe00640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec600039a0 0x7fec60003dc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fec70001720 tx=0x7fec600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50099a360cb18e4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.462+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec700052d0 con 0x7fec600039a0 2024-01-16T15:01:22.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.462+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7002e050 con 0x7fec600039a0 2024-01-16T15:01:22.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.462+0000 7fec7ea00640 1 -- 172.21.15.141:0/1476947697 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70003610 con 0x7fec600039a0 2024-01-16T15:01:22.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.463+0000 7fec87200640 1 -- 172.21.15.141:0/1476947697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec600039a0 msgr2=0x7fec60003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.463+0000 7fec87200640 1 --2- 172.21.15.141:0/1476947697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec600039a0 0x7fec60003dc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fec70001720 tx=0x7fec600057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.463+0000 7fec85200640 1 -- 172.21.15.141:0/1476947697 reap_dead start 2024-01-16T15:01:22.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.463+0000 7fec87200640 1 -- 172.21.15.141:0/1476947697 shutdown_connections 2024-01-16T15:01:22.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.463+0000 7fec87200640 1 -- 172.21.15.141:0/1476947697 >> 172.21.15.141:0/1476947697 conn(0x7fec8009c560 msgr2=0x7fec800d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:22.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.463+0000 7fec87200640 1 -- 172.21.15.141:0/1476947697 shutdown_connections 2024-01-16T15:01:22.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.463+0000 7fec87200640 1 -- 172.21.15.141:0/1476947697 wait complete. 2024-01-16T15:01:22.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.466+0000 7fec87200640 1 Processor -- start 2024-01-16T15:01:22.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.466+0000 7fec87200640 1 -- start start 2024-01-16T15:01:22.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.467+0000 7fec87200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec600039a0 0x7fec800cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.467+0000 7fec87200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec801554f0 0x7fec801f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.467+0000 7fec87200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec801f6ea0 0x7fec801f7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.467+0000 7fec87200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015fdd0 con 0x7fec801f6ea0 2024-01-16T15:01:22.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.467+0000 7fec87200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015fa30 con 0x7fec600039a0 2024-01-16T15:01:22.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.467+0000 7fec87200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec801598a0 con 0x7fec801554f0 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec801f6ea0 0x7fec801f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec801f6ea0 0x7fec801f7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38902/0 (socket says 172.21.15.141:38902) 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 -- 172.21.15.141:0/1886398611 learned_addr learned my addr 172.21.15.141:0/1886398611 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec7fe00640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec600039a0 0x7fec800cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 -- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec801554f0 msgr2=0x7fec801f6960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec801554f0 0x7fec801f6960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 -- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec600039a0 msgr2=0x7fec800cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec600039a0 0x7fec800cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.468+0000 7fec85200640 1 -- 172.21.15.141:0/1886398611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80159b80 con 0x7fec801f6ea0 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.469+0000 7fec85200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec801f6ea0 0x7fec801f7320 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fec8009cb60 tx=0x7fec78009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.469+0000 7fec77e00640 1 -- 172.21.15.141:0/1886398611 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78017400 con 0x7fec801f6ea0 2024-01-16T15:01:22.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.469+0000 7fec77e00640 1 -- 172.21.15.141:0/1886398611 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec78011040 con 0x7fec801f6ea0 2024-01-16T15:01:22.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.469+0000 7fec77e00640 1 -- 172.21.15.141:0/1886398611 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec780177a0 con 0x7fec801f6ea0 2024-01-16T15:01:22.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.469+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec80159e60 con 0x7fec801f6ea0 2024-01-16T15:01:22.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.470+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec800cce80 con 0x7fec801f6ea0 2024-01-16T15:01:22.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.471+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8015ff10 con 0x7fec801f6ea0 2024-01-16T15:01:22.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.472+0000 7fec77e00640 1 -- 172.21.15.141:0/1886398611 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7fec780203b0 con 0x7fec801f6ea0 2024-01-16T15:01:22.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.473+0000 7fec77e00640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec5806b6b0 0x7fec5806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.473+0000 7fec77e00640 1 -- 172.21.15.141:0/1886398611 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fec7808a440 con 0x7fec801f6ea0 2024-01-16T15:01:22.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.476+0000 7fec7fe00640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec5806b6b0 0x7fec5806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.478+0000 7fec7fe00640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec5806b6b0 0x7fec5806db70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec70033000 tx=0x7fec70009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.482+0000 7fec77e00640 1 -- 172.21.15.141:0/1886398611 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec8015ff10 con 0x7fec801f6ea0 2024-01-16T15:01:22.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.962+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fec800cce80 con 0x7fec801f6ea0 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.962+0000 7fec77e00640 1 -- 172.21.15.141:0/1886398611 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2092 (secure 0 0 0) 0x7fec800cce80 con 0x7fec801f6ea0 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:22.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 275, 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:22.944 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:22.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 802, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222254263, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6704529408, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766389583872, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 852, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 16547, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:01:22.946 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 5 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:22.947 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:22.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:22.949 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:22.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.967+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec5806b6b0 msgr2=0x7fec5806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.967+0000 7fec87200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec5806b6b0 0x7fec5806db70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec70033000 tx=0x7fec70009030 comp rx=0 tx=0).stop 2024-01-16T15:01:22.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.967+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec801f6ea0 msgr2=0x7fec801f7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.967+0000 7fec87200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec801f6ea0 0x7fec801f7320 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fec8009cb60 tx=0x7fec78009f90 comp rx=0 tx=0).stop 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 shutdown_connections 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec5806b6b0 0x7fec5806db70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec801f6ea0 0x7fec801f7320 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec801554f0 0x7fec801f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 --2- 172.21.15.141:0/1886398611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec600039a0 0x7fec800cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 >> 172.21.15.141:0/1886398611 conn(0x7fec8009c560 msgr2=0x7fec800d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 shutdown_connections 2024-01-16T15:01:22.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:22.968+0000 7fec87200640 1 -- 172.21.15.141:0/1886398611 wait complete. 2024-01-16T15:01:23.165 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:23.184+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:23.165 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:23.184+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:23.165 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:23.184+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:23.165 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:23.184+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:23.165 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:23.184+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:24.972 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:25.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.208+0000 7ff95d400640 1 Processor -- start 2024-01-16T15:01:25.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.208+0000 7ff95d400640 1 -- start start 2024-01-16T15:01:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff95d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 0x7ff95815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff95d400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff958070c40 con 0x7ff9580b3ae0 2024-01-16T15:01:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff95d400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95806c470 con 0x7ff9580cf290 2024-01-16T15:01:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff95d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff958071450 con 0x7ff9580ae5a0 2024-01-16T15:01:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff955e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff9580b3ae0 0x7ff95806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53002/0 (socket says 172.21.15.141:53002) 2024-01-16T15:01:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff955e00640 1 -- 172.21.15.141:0/889423231 learned_addr learned my addr 172.21.15.141:0/889423231 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771310174 0 0) 0x7ff958070c40 con 0x7ff9580b3ae0 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.210+0000 7ff956800640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 0x7ff95815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff938003520 con 0x7ff9580b3ae0 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1551240831 0 0) 0x7ff938003520 con 0x7ff9580b3ae0 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 msgr2=0x7ff95815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 0x7ff95815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 >> v1:172.21.15.188:6789/0 conn(0x7ff9580cf290 legacy=0x7ff9580b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95806c750 con 0x7ff9580b3ae0 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ff940001670 con 0x7ff9580b3ae0 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 >> v1:172.21.15.141:6789/0 conn(0x7ff9580b3ae0 legacy=0x7ff95806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9380039a0 0x7ff938003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 0x7ff9380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff938005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff958071450 con 0x7ff938004d00 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93800baf0 con 0x7ff9380039a0 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93800bcb0 con 0x7ff9580ae5a0 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.211+0000 7ff957200640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff938005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.212+0000 7ff956800640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9380039a0 0x7ff938003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.212+0000 7ff955e00640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 0x7ff9380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.212+0000 7ff957200640 1 -- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 msgr2=0x7ff9380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.212+0000 7ff957200640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ae5a0 0x7ff9380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.212+0000 7ff957200640 1 -- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9380039a0 msgr2=0x7ff938003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.212+0000 7ff957200640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9380039a0 0x7ff938003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.212+0000 7ff957200640 1 -- 172.21.15.141:0/889423231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938003520 con 0x7ff938004d00 2024-01-16T15:01:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.213+0000 7ff957200640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff938005180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff950001e20 tx=0x7ff9380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640fe1eb0d343dd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.213+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950003b70 con 0x7ff938004d00 2024-01-16T15:01:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.213+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950003cd0 con 0x7ff938004d00 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.213+0000 7ff955400640 1 -- 172.21.15.141:0/889423231 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950005310 con 0x7ff938004d00 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.214+0000 7ff95d400640 1 -- 172.21.15.141:0/889423231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 msgr2=0x7ff938005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.214+0000 7ff95d400640 1 --2- 172.21.15.141:0/889423231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff938005180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff950001e20 tx=0x7ff9380057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.214+0000 7ff957200640 1 -- 172.21.15.141:0/889423231 reap_dead start 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.214+0000 7ff95d400640 1 -- 172.21.15.141:0/889423231 shutdown_connections 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.214+0000 7ff95d400640 1 -- 172.21.15.141:0/889423231 >> 172.21.15.141:0/889423231 conn(0x7ff95806a800 msgr2=0x7ff95806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.214+0000 7ff95d400640 1 -- 172.21.15.141:0/889423231 shutdown_connections 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.214+0000 7ff95d400640 1 -- 172.21.15.141:0/889423231 wait complete. 2024-01-16T15:01:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.215+0000 7ff95d400640 1 Processor -- start 2024-01-16T15:01:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.216+0000 7ff95d400640 1 -- start start 2024-01-16T15:01:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.216+0000 7ff95d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff9581f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff95d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ce5d0 0x7ff9580ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff95d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9580cab00 0x7ff9580caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff95d400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9581f3ed0 con 0x7ff938004d00 2024-01-16T15:01:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff95d400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9581f72d0 con 0x7ff9580cab00 2024-01-16T15:01:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff95d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9581f66c0 con 0x7ff9580ce5d0 2024-01-16T15:01:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff9581f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff9581f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47240/0 (socket says 172.21.15.141:47240) 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 -- 172.21.15.141:0/3636597417 learned_addr learned my addr 172.21.15.141:0/3636597417 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 -- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ce5d0 msgr2=0x7ff9580ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ce5d0 0x7ff9580ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 -- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9580cab00 msgr2=0x7ff9580caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9580cab00 0x7ff9580caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 -- 172.21.15.141:0/3636597417 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9581f69a0 con 0x7ff938004d00 2024-01-16T15:01:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.217+0000 7ff956800640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff9581f3990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff948011630 tx=0x7ff948011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.218+0000 7ff94ea00640 1 -- 172.21.15.141:0/3636597417 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948015870 con 0x7ff938004d00 2024-01-16T15:01:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.218+0000 7ff94ea00640 1 -- 172.21.15.141:0/3636597417 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9480159d0 con 0x7ff938004d00 2024-01-16T15:01:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.218+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9581f6c80 con 0x7ff938004d00 2024-01-16T15:01:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.218+0000 7ff94ea00640 1 -- 172.21.15.141:0/3636597417 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948015d40 con 0x7ff938004d00 2024-01-16T15:01:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.218+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9580cb4c0 con 0x7ff938004d00 2024-01-16T15:01:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.219+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9580675d0 con 0x7ff938004d00 2024-01-16T15:01:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.220+0000 7ff94ea00640 1 -- 172.21.15.141:0/3636597417 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 11) v1 ==== 89947+0+0 (secure 0 0 0) 0x7ff94800f490 con 0x7ff938004d00 2024-01-16T15:01:25.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.220+0000 7ff94ea00640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff93006b770 0x7ff93006dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.221+0000 7ff94ea00640 1 -- 172.21.15.141:0/3636597417 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff94808ace0 con 0x7ff938004d00 2024-01-16T15:01:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.221+0000 7ff955e00640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff93006b770 0x7ff93006dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.222+0000 7ff955e00640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff93006b770 0x7ff93006dc30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff958159800 tx=0x7ff940000ef0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.225+0000 7ff94ea00640 1 -- 172.21.15.141:0/3636597417 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff948055410 con 0x7ff938004d00 2024-01-16T15:01:25.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.482+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff958160050 con 0x7ff938004d00 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.483+0000 7ff94ea00640 1 -- 172.21.15.141:0/3636597417 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2128 (secure 0 0 0) 0x7ff948058a30 con 0x7ff938004d00 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:25.463 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 278, 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:25.464 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:25.465 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 802, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222254263, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6704529408, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766389583872, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 597, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 10235, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 3 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:25.466 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:25.467 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:25.468 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.22220000624656677, 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:25.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff93006b770 msgr2=0x7ff93006dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff93006b770 0x7ff93006dc30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff958159800 tx=0x7ff940000ef0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 msgr2=0x7ff9581f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff9581f3990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff948011630 tx=0x7ff948011af0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 shutdown_connections 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff93006b770 0x7ff93006dc30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9580cab00 0x7ff9580caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9580ce5d0 0x7ff9580ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 --2- 172.21.15.141:0/3636597417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff938004d00 0x7ff9581f3990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.485+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 >> 172.21.15.141:0/3636597417 conn(0x7ff95806a800 msgr2=0x7ff95815e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.486+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 shutdown_connections 2024-01-16T15:01:25.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:25.486+0000 7ff95d400640 1 -- 172.21.15.141:0/3636597417 wait complete. 2024-01-16T15:01:27.484 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.786+0000 7ff072400640 1 Processor -- start 2024-01-16T15:01:27.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.787+0000 7ff072400640 1 -- start start 2024-01-16T15:01:27.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.788+0000 7ff072400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0cfcf0 0x7ff06c160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:27.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.788+0000 7ff072400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff06c0d00d0 0x7ff06c160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:27.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.788+0000 7ff072400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06c0d4160 con 0x7ff06c0d3910 2024-01-16T15:01:27.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.788+0000 7ff072400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06c0ceff0 con 0x7ff06c0d00d0 2024-01-16T15:01:27.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.788+0000 7ff072400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c0cf410 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.789+0000 7ff06aa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff06c0d3910 0x7ff06c0b2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53018/0 (socket says 172.21.15.141:53018) 2024-01-16T15:01:27.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.790+0000 7ff06aa00640 1 -- 172.21.15.141:0/266583344 learned_addr learned my addr 172.21.15.141:0/266583344 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:27.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.790+0000 7ff06be00640 1 --2- 172.21.15.141:0/266583344 >> v2:172.21.15.188:3300/0 conn(0x7ff06c0d00d0 0x7ff06c160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:27.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.791+0000 7ff06b400640 1 --2- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0cfcf0 0x7ff06c160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:27.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.792+0000 7ff06be00640 1 -- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0cfcf0 msgr2=0x7ff06c160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:27.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.792+0000 7ff06be00640 1 --2- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0cfcf0 0x7ff06c160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:27.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.792+0000 7ff06be00640 1 -- 172.21.15.141:0/266583344 >> v1:172.21.15.141:6789/0 conn(0x7ff06c0d3910 legacy=0x7ff06c0b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:27.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.792+0000 7ff06be00640 1 -- 172.21.15.141:0/266583344 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c0b2420 con 0x7ff06c0d00d0 2024-01-16T15:01:27.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.795+0000 7ff06be00640 1 --2- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d00d0 0x7ff06c160c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff064009dc0 tx=0x7ff06402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeac54ebad47ceef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.796+0000 7ff06a000640 1 -- 172.21.15.141:0/266583344 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06402c960 con 0x7ff06c0d00d0 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.797+0000 7ff06a000640 1 -- 172.21.15.141:0/266583344 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06402cac0 con 0x7ff06c0d00d0 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.798+0000 7ff06a000640 1 -- 172.21.15.141:0/266583344 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06402ce50 con 0x7ff06c0d00d0 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.798+0000 7ff072400640 1 -- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d00d0 msgr2=0x7ff06c160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.798+0000 7ff072400640 1 --2- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d00d0 0x7ff06c160c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff064009dc0 tx=0x7ff06402bc30 comp rx=0 tx=0).stop 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.799+0000 7ff072400640 1 -- 172.21.15.141:0/266583344 shutdown_connections 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.799+0000 7ff072400640 1 --2- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d00d0 0x7ff06c160c40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.799+0000 7ff072400640 1 --2- 172.21.15.141:0/266583344 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0cfcf0 0x7ff06c160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.799+0000 7ff072400640 1 -- 172.21.15.141:0/266583344 >> 172.21.15.141:0/266583344 conn(0x7ff06c09c530 msgr2=0x7ff06c0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.799+0000 7ff072400640 1 -- 172.21.15.141:0/266583344 shutdown_connections 2024-01-16T15:01:27.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.799+0000 7ff072400640 1 -- 172.21.15.141:0/266583344 wait complete. 2024-01-16T15:01:27.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.802+0000 7ff072400640 1 Processor -- start 2024-01-16T15:01:27.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.802+0000 7ff072400640 1 -- start start 2024-01-16T15:01:27.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.804+0000 7ff072400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff06c0cfcf0 0x7ff06c0c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:27.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.804+0000 7ff072400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0d00d0 0x7ff06c0b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:27.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.804+0000 7ff072400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d3910 0x7ff06c0b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:27.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.805+0000 7ff072400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c0cb920 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.805+0000 7ff072400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c0b78a0 con 0x7ff06c0d3910 2024-01-16T15:01:27.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.805+0000 7ff072400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c0c85e0 con 0x7ff06c0d00d0 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.806+0000 7ff06b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff06c0cfcf0 0x7ff06c0c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.806+0000 7ff06b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff06c0cfcf0 0x7ff06c0c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47246/0 (socket says 172.21.15.141:47246) 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.806+0000 7ff06b400640 1 -- 172.21.15.141:0/567443906 learned_addr learned my addr 172.21.15.141:0/567443906 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.806+0000 7ff06aa00640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0d00d0 0x7ff06c0b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.807+0000 7ff06b400640 1 -- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0d00d0 msgr2=0x7ff06c0b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.807+0000 7ff06b400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0d00d0 0x7ff06c0b7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.807+0000 7ff06b400640 1 -- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d3910 msgr2=0x7ff06c0b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.807+0000 7ff06b400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d3910 0x7ff06c0b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.807+0000 7ff06b400640 1 -- 172.21.15.141:0/567443906 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064009a70 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.807+0000 7ff06b400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff06c0cfcf0 0x7ff06c0c7630 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff05400d800 tx=0x7ff05400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.808+0000 7ff063400640 1 -- 172.21.15.141:0/567443906 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054016be0 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.808+0000 7ff063400640 1 -- 172.21.15.141:0/567443906 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05401a070 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.808+0000 7ff063400640 1 -- 172.21.15.141:0/567443906 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05400b6c0 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.808+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06c0c7d40 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.809+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06c0c8300 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.811+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06c08c9f0 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.814+0000 7ff063400640 1 -- 172.21.15.141:0/567443906 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 11) v1 ==== 89947+0+0 (secure 0 0 0) 0x7ff05400b820 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.817+0000 7ff063400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff04406b740 0x7ff04406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:27.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.819+0000 7ff063400640 1 -- 172.21.15.141:0/567443906 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff05401c070 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.829+0000 7ff06aa00640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff04406b740 0x7ff04406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:27.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.835+0000 7ff063400640 1 -- 172.21.15.141:0/567443906 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff05405cfa0 con 0x7ff06c0cfcf0 2024-01-16T15:01:27.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:27.845+0000 7ff06aa00640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff04406b740 0x7ff04406dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff06c1f6ca0 tx=0x7ff05c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:28.146 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:28.164+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:28.146 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:28.164+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:28.146 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:28.164+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:28.146 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:28.164+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:28.146 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:01:28.164+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.355+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff06c0675d0 con 0x7ff06c0cfcf0 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.356+0000 7ff063400640 1 -- 172.21.15.141:0/567443906 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2142 (secure 0 0 0) 0x7ff054026090 con 0x7ff06c0cfcf0 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:28.337 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 281, 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:28.338 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:28.339 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1313, 2024-01-16T15:01:28.340 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5369831131, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10999668736, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762094444544, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 178861330, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 178879411, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 49 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:28.341 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:28.342 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:28.343 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-16T15:01:28.344 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.22220000624656677, 2024-01-16T15:01:28.345 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:28.345 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:28.345 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:28.345 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:28.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff04406b740 msgr2=0x7ff04406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff04406b740 0x7ff04406dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff06c1f6ca0 tx=0x7ff05c009040 comp rx=0 tx=0).stop 2024-01-16T15:01:28.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff06c0cfcf0 msgr2=0x7ff06c0c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff06c0cfcf0 0x7ff06c0c7630 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff05400d800 tx=0x7ff05400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 shutdown_connections 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff04406b740 0x7ff04406dc00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff06c0d3910 0x7ff06c0b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff06c0d00d0 0x7ff06c0b7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 --2- 172.21.15.141:0/567443906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff06c0cfcf0 0x7ff06c0c7630 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 >> 172.21.15.141:0/567443906 conn(0x7ff06c09c530 msgr2=0x7ff06c0b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 shutdown_connections 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:28.359+0000 7ff072400640 1 -- 172.21.15.141:0/567443906 wait complete. 2024-01-16T15:01:30.366 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:30.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.577+0000 7faf51000640 1 Processor -- start 2024-01-16T15:01:30.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.577+0000 7faf51000640 1 -- start start 2024-01-16T15:01:30.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.578+0000 7faf51000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c13cfa0 0x7faf4c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.578+0000 7faf51000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7faf4c13fcb0 0x7faf4c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.578+0000 7faf51000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0cdec0 con 0x7faf4c0cd530 2024-01-16T15:01:30.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.578+0000 7faf51000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf4c0ce1a0 con 0x7faf4c13fcb0 2024-01-16T15:01:30.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.578+0000 7faf51000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0ce480 con 0x7faf4c13cfa0 2024-01-16T15:01:30.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.579+0000 7faf42800640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7faf4c0cd530 0x7faf4c13f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:54880/0 (socket says 172.21.15.141:54880) 2024-01-16T15:01:30.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.579+0000 7faf42800640 1 -- 172.21.15.141:0/113891787 learned_addr learned my addr 172.21.15.141:0/113891787 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.579+0000 7faf4b400640 1 --2- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c13cfa0 0x7faf4c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.579+0000 7faf4be00640 1 --2- 172.21.15.141:0/113891787 >> v2:172.21.15.188:3300/0 conn(0x7faf4c13fcb0 0x7faf4c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.579+0000 7faf4aa00640 1 -- 172.21.15.141:0/113891787 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435751757 0 0) 0x7faf4c0cdec0 con 0x7faf4c0cd530 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.579+0000 7faf4aa00640 1 -- 172.21.15.141:0/113891787 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c003520 con 0x7faf4c0cd530 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.580+0000 7faf4be00640 1 -- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c13cfa0 msgr2=0x7faf4c15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.580+0000 7faf4be00640 1 --2- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c13cfa0 0x7faf4c15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.580+0000 7faf4be00640 1 -- 172.21.15.141:0/113891787 >> v1:172.21.15.188:6789/0 conn(0x7faf4c0cd530 legacy=0x7faf4c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.580+0000 7faf4be00640 1 -- 172.21.15.141:0/113891787 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c0ce760 con 0x7faf4c13fcb0 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.580+0000 7faf4b400640 1 --2- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c13cfa0 0x7faf4c15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:01:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.581+0000 7faf4be00640 1 --2- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 0x7faf4c0b2040 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faf44009d90 tx=0x7faf4402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c709b725818e56c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.581+0000 7faf4aa00640 1 -- 172.21.15.141:0/113891787 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4402cbd0 con 0x7faf4c13fcb0 2024-01-16T15:01:30.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.581+0000 7faf4aa00640 1 -- 172.21.15.141:0/113891787 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4402cd30 con 0x7faf4c13fcb0 2024-01-16T15:01:30.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.581+0000 7faf4aa00640 1 -- 172.21.15.141:0/113891787 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4402cea0 con 0x7faf4c13fcb0 2024-01-16T15:01:30.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 -- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 msgr2=0x7faf4c0b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 --2- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 0x7faf4c0b2040 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faf44009d90 tx=0x7faf4402be20 comp rx=0 tx=0).stop 2024-01-16T15:01:30.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 -- 172.21.15.141:0/113891787 shutdown_connections 2024-01-16T15:01:30.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 --2- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 0x7faf4c0b2040 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 --2- 172.21.15.141:0/113891787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c13cfa0 0x7faf4c15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 -- 172.21.15.141:0/113891787 >> 172.21.15.141:0/113891787 conn(0x7faf4c06a800 msgr2=0x7faf4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:30.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 -- 172.21.15.141:0/113891787 shutdown_connections 2024-01-16T15:01:30.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.582+0000 7faf51000640 1 -- 172.21.15.141:0/113891787 wait complete. 2024-01-16T15:01:30.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.584+0000 7faf51000640 1 Processor -- start 2024-01-16T15:01:30.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.584+0000 7faf51000640 1 -- start start 2024-01-16T15:01:30.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf51000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c0cd530 0x7faf4c1679c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf51000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf4c13cfa0 0x7faf4c131d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf51000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 0x7faf4c153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf51000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c070160 con 0x7faf4c13cfa0 2024-01-16T15:01:30.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf51000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c070440 con 0x7faf4c13fcb0 2024-01-16T15:01:30.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf51000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c159f30 con 0x7faf4c0cd530 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf4c13cfa0 0x7faf4c131d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf4c13cfa0 0x7faf4c131d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47254/0 (socket says 172.21.15.141:47254) 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 -- 172.21.15.141:0/1837064620 learned_addr learned my addr 172.21.15.141:0/1837064620 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf4b400640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c0cd530 0x7faf4c1679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 -- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c0cd530 msgr2=0x7faf4c1679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c0cd530 0x7faf4c1679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 -- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 msgr2=0x7faf4c153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 0x7faf4c153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.585+0000 7faf42800640 1 -- 172.21.15.141:0/1837064620 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c009400 con 0x7faf4c13cfa0 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.586+0000 7faf4b400640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c0cd530 0x7faf4c1679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.586+0000 7faf42800640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf4c13cfa0 0x7faf4c131d40 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7faf3400c460 tx=0x7faf3400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.586+0000 7faf43e00640 1 -- 172.21.15.141:0/1837064620 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34005ce0 con 0x7faf4c13cfa0 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.586+0000 7faf43e00640 1 -- 172.21.15.141:0/1837064620 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf34005e40 con 0x7faf4c13cfa0 2024-01-16T15:01:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.586+0000 7faf43e00640 1 -- 172.21.15.141:0/1837064620 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34007290 con 0x7faf4c13cfa0 2024-01-16T15:01:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.586+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf44009a70 con 0x7faf4c13cfa0 2024-01-16T15:01:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.586+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c158480 con 0x7faf4c13cfa0 2024-01-16T15:01:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.587+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c1586a0 con 0x7faf4c13cfa0 2024-01-16T15:01:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.588+0000 7faf43e00640 1 -- 172.21.15.141:0/1837064620 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7faf340073f0 con 0x7faf4c13cfa0 2024-01-16T15:01:30.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.589+0000 7faf43e00640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf2406b7d0 0x7faf2406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.589+0000 7faf43e00640 1 -- 172.21.15.141:0/1837064620 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faf340899e0 con 0x7faf4c13cfa0 2024-01-16T15:01:30.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.590+0000 7faf4b400640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf2406b7d0 0x7faf2406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.591+0000 7faf4b400640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf2406b7d0 0x7faf2406dc90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009c50 tx=0x7faf3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.594+0000 7faf43e00640 1 -- 172.21.15.141:0/1837064620 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf340541c0 con 0x7faf4c13cfa0 2024-01-16T15:01:30.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.835+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faf4c067450 con 0x7faf4c13cfa0 2024-01-16T15:01:30.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.836+0000 7faf43e00640 1 -- 172.21.15.141:0/1837064620 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2145 (secure 0 0 0) 0x7faf34017020 con 0x7faf4c13cfa0 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:30.817 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 283, 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:30.818 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:30.819 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1940, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7999784761, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 15685001216, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 757409112064, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 409685492, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 397495503, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 98, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 103 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:30.820 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:30.821 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.41667500138282776, 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:30.822 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:30.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf2406b7d0 msgr2=0x7faf2406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf2406b7d0 0x7faf2406dc90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009c50 tx=0x7faf3c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf4c13cfa0 msgr2=0x7faf4c131d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf4c13cfa0 0x7faf4c131d40 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7faf3400c460 tx=0x7faf3400c920 comp rx=0 tx=0).stop 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 shutdown_connections 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf2406b7d0 0x7faf2406dc90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf4c13fcb0 0x7faf4c153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf4c13cfa0 0x7faf4c131d40 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 --2- 172.21.15.141:0/1837064620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf4c0cd530 0x7faf4c1679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 >> 172.21.15.141:0/1837064620 conn(0x7faf4c06a800 msgr2=0x7faf4c136160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 shutdown_connections 2024-01-16T15:01:30.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:30.839+0000 7faf51000640 1 -- 172.21.15.141:0/1837064620 wait complete. 2024-01-16T15:01:32.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:33.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.113+0000 7f1623600640 1 Processor -- start 2024-01-16T15:01:33.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.113+0000 7f1623600640 1 -- start start 2024-01-16T15:01:33.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1623600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f161c0cd240 0x7f161c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1623600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c06c6b0 0x7f161c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1623600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161c06cea0 con 0x7f161c099fa0 2024-01-16T15:01:33.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1623600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f161c09a430 con 0x7f161c0cd240 2024-01-16T15:01:33.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1623600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161c06cbc0 con 0x7f161c06c6b0 2024-01-16T15:01:33.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1621600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f161c099fa0 0x7f161c06b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50404/0 (socket says 172.21.15.141:50404) 2024-01-16T15:01:33.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1621600640 1 -- 172.21.15.141:0/2766589633 learned_addr learned my addr 172.21.15.141:0/2766589633 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:33.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1620c00640 1 --2- 172.21.15.141:0/2766589633 >> v2:172.21.15.188:3300/0 conn(0x7f161c0cd240 0x7f161c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.115+0000 7f1613e00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c06c6b0 0x7f161c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028106041 0 0) 0x7f161c06cea0 con 0x7f161c099fa0 2024-01-16T15:01:33.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc003520 con 0x7f161c099fa0 2024-01-16T15:01:33.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 671411796 0 0) 0x7f15fc003520 con 0x7f161c099fa0 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c06c6b0 msgr2=0x7f161c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c06c6b0 0x7f161c099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 >> v2:172.21.15.188:3300/0 conn(0x7f161c0cd240 msgr2=0x7f161c0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 --2- 172.21.15.141:0/2766589633 >> v2:172.21.15.188:3300/0 conn(0x7f161c0cd240 0x7f161c0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c06f350 con 0x7f161c099fa0 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f16140019f0 con 0x7f161c099fa0 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 >> v1:172.21.15.141:6789/0 conn(0x7f161c099fa0 legacy=0x7f161c06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15fc003970 0x7f15fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f15fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c06c6b0 0x7f15fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c09a430 con 0x7f15fc0047c0 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c06cbc0 con 0x7f15fc003970 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc00b980 con 0x7f161c06c6b0 2024-01-16T15:01:33.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.116+0000 7f1613e00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f15fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.117+0000 7f1613e00640 1 -- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c06c6b0 msgr2=0x7f15fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.117+0000 7f1613e00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c06c6b0 0x7f15fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.117+0000 7f1613e00640 1 -- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15fc003970 msgr2=0x7f15fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.117+0000 7f1613e00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15fc003970 0x7f15fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.117+0000 7f1613e00640 1 -- 172.21.15.141:0/2766589633 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc003520 con 0x7f15fc0047c0 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.118+0000 7f1613e00640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f15fc004c40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1604009a90 tx=0x7f15fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cce4df0a586a2ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.119+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160402c6b0 con 0x7f15fc0047c0 2024-01-16T15:01:33.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.119+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160402c810 con 0x7f15fc0047c0 2024-01-16T15:01:33.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.119+0000 7f161be00640 1 -- 172.21.15.141:0/2766589633 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160402cba0 con 0x7f15fc0047c0 2024-01-16T15:01:33.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.120+0000 7f1623600640 1 -- 172.21.15.141:0/2766589633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 msgr2=0x7f15fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.120+0000 7f1623600640 1 --2- 172.21.15.141:0/2766589633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f15fc004c40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1604009a90 tx=0x7f15fc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.120+0000 7f1621600640 1 -- 172.21.15.141:0/2766589633 reap_dead start 2024-01-16T15:01:33.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.120+0000 7f1623600640 1 -- 172.21.15.141:0/2766589633 shutdown_connections 2024-01-16T15:01:33.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.120+0000 7f1623600640 1 -- 172.21.15.141:0/2766589633 >> 172.21.15.141:0/2766589633 conn(0x7f161c0b2d00 msgr2=0x7f161c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:33.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.120+0000 7f1623600640 1 -- 172.21.15.141:0/2766589633 shutdown_connections 2024-01-16T15:01:33.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.120+0000 7f1623600640 1 -- 172.21.15.141:0/2766589633 wait complete. 2024-01-16T15:01:33.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.123+0000 7f1623600640 1 Processor -- start 2024-01-16T15:01:33.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.126+0000 7f1623600640 1 -- start start 2024-01-16T15:01:33.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.127+0000 7f1623600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f161c1f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.127+0000 7f1623600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f161c0cf4d0 0x7f161c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.128+0000 7f1623600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c15c290 0x7f161c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.128+0000 7f1623600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c0cfe90 con 0x7f15fc0047c0 2024-01-16T15:01:33.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.128+0000 7f1623600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c1f09f0 con 0x7f161c0cf4d0 2024-01-16T15:01:33.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.128+0000 7f1623600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161c15cc80 con 0x7f161c15c290 2024-01-16T15:01:33.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f161c1f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f161c1f04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53474/0 (socket says 172.21.15.141:53474) 2024-01-16T15:01:33.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 -- 172.21.15.141:0/3500298899 learned_addr learned my addr 172.21.15.141:0/3500298899 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:33.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 -- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c15c290 msgr2=0x7f161c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c15c290 0x7f161c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 -- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f161c0cf4d0 msgr2=0x7f161c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f161c0cf4d0 0x7f161c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.129+0000 7f1620c00640 1 -- 172.21.15.141:0/3500298899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c1f32b0 con 0x7f15fc0047c0 2024-01-16T15:01:33.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.130+0000 7f1620c00640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f161c1f04b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f160c00e890 tx=0x7f160c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.130+0000 7f1619600640 1 -- 172.21.15.141:0/3500298899 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c00c940 con 0x7f15fc0047c0 2024-01-16T15:01:33.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.130+0000 7f1619600640 1 -- 172.21.15.141:0/3500298899 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c00cac0 con 0x7f15fc0047c0 2024-01-16T15:01:33.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.130+0000 7f1619600640 1 -- 172.21.15.141:0/3500298899 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c00ce70 con 0x7f15fc0047c0 2024-01-16T15:01:33.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.132+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c15cf00 con 0x7f15fc0047c0 2024-01-16T15:01:33.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.132+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c15d4c0 con 0x7f15fc0047c0 2024-01-16T15:01:33.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.134+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c0675d0 con 0x7f15fc0047c0 2024-01-16T15:01:33.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.136+0000 7f1619600640 1 -- 172.21.15.141:0/3500298899 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f160c013470 con 0x7f15fc0047c0 2024-01-16T15:01:33.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.138+0000 7f1619600640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f15f406b850 0x7f15f406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.139+0000 7f1619600640 1 -- 172.21.15.141:0/3500298899 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f160c08a540 con 0x7f15fc0047c0 2024-01-16T15:01:33.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.155+0000 7f1619600640 1 -- 172.21.15.141:0/3500298899 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f160c054d60 con 0x7f15fc0047c0 2024-01-16T15:01:33.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.158+0000 7f1613e00640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f15f406b850 0x7f15f406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.170+0000 7f1613e00640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f15f406b850 0x7f15f406dd10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1604036000 tx=0x7f160402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.201 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:33.207 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:33.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.619+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f161c0c8010 con 0x7f15fc0047c0 2024-01-16T15:01:33.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.620+0000 7f1619600640 1 -- 172.21.15.141:0/3500298899 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2145 (secure 0 0 0) 0x7f160c058380 con 0x7f15fc0047c0 2024-01-16T15:01:33.600 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:33.600 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:33.600 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:33.600 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:33.600 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 286, 2024-01-16T15:01:33.601 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:33.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1940, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7999784761, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 15685001216, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 757409112064, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 409683404, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 397489545, 2024-01-16T15:01:33.603 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 98, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 101 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:33.604 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:33.605 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.41667500138282776, 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:33.606 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.625+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f15f406b850 msgr2=0x7f15f406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.625+0000 7f1623600640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f15f406b850 0x7f15f406dd10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1604036000 tx=0x7f160402e000 comp rx=0 tx=0).stop 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 msgr2=0x7f161c1f04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f161c1f04b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f160c00e890 tx=0x7f160c00ed50 comp rx=0 tx=0).stop 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 shutdown_connections 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f15f406b850 0x7f15f406dd10 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1604036000 tx=0x7f160402e000 comp rx=0 tx=0).stop 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f161c15c290 0x7f161c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f161c0cf4d0 0x7f161c0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 --2- 172.21.15.141:0/3500298899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15fc0047c0 0x7f161c1f04b0 secure :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f160c00e890 tx=0x7f160c00ed50 comp rx=0 tx=0).stop 2024-01-16T15:01:33.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.626+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 >> 172.21.15.141:0/3500298899 conn(0x7f161c0b2d00 msgr2=0x7f161c0aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:33.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.627+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 shutdown_connections 2024-01-16T15:01:33.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:33.627+0000 7f1623600640 1 -- 172.21.15.141:0/3500298899 wait complete. 2024-01-16T15:01:35.640 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.940+0000 7f778f200640 1 Processor -- start 2024-01-16T15:01:35.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.941+0000 7f778f200640 1 -- start start 2024-01-16T15:01:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.943+0000 7f778f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f77880b3c20 0x7f7788070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.943+0000 7f778f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f778806d050 0x7f778806d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.943+0000 7f778f200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7788071790 con 0x7f7788159590 2024-01-16T15:01:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.943+0000 7f778f200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778806d990 con 0x7f77880b3c20 2024-01-16T15:01:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.943+0000 7f778f200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7788071100 con 0x7f778806d050 2024-01-16T15:01:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.945+0000 7f778e800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f778806d050 0x7f778806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.945+0000 7f778e800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f778806d050 0x7f778806d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:39882/0 (socket says 172.21.15.141:39882) 2024-01-16T15:01:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.945+0000 7f778e800640 1 -- 172.21.15.141:0/1187638254 learned_addr learned my addr 172.21.15.141:0/1187638254 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:35.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.945+0000 7f778d400640 1 --2- 172.21.15.141:0/1187638254 >> v2:172.21.15.141:3300/0 conn(0x7f77880b3c20 0x7f7788070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.946+0000 7f778e800640 1 -- 172.21.15.141:0/1187638254 >> v2:172.21.15.141:3300/0 conn(0x7f77880b3c20 msgr2=0x7f7788070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.946+0000 7f778e800640 1 --2- 172.21.15.141:0/1187638254 >> v2:172.21.15.141:3300/0 conn(0x7f77880b3c20 0x7f7788070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.946+0000 7f778e800640 1 -- 172.21.15.141:0/1187638254 >> v1:172.21.15.141:6789/0 conn(0x7f7788159590 legacy=0x7f77880ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:35.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.947+0000 7f778e800640 1 -- 172.21.15.141:0/1187638254 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788070860 con 0x7f778806d050 2024-01-16T15:01:35.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.949+0000 7f778e800640 1 --2- 172.21.15.141:0/1187638254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f778806d050 0x7f778806d450 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f77880b3770 tx=0x7f778002cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5690d321a1e88b93 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:35.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.952+0000 7f778ca00640 1 -- 172.21.15.141:0/1187638254 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778002db20 con 0x7f778806d050 2024-01-16T15:01:35.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.952+0000 7f778ca00640 1 -- 172.21.15.141:0/1187638254 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778002dc80 con 0x7f778806d050 2024-01-16T15:01:35.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.952+0000 7f778ca00640 1 -- 172.21.15.141:0/1187638254 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77800353f0 con 0x7f778806d050 2024-01-16T15:01:35.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.957+0000 7f778f200640 1 -- 172.21.15.141:0/1187638254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f778806d050 msgr2=0x7f778806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:35.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.957+0000 7f778f200640 1 --2- 172.21.15.141:0/1187638254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f778806d050 0x7f778806d450 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f77880b3770 tx=0x7f778002cf80 comp rx=0 tx=0).stop 2024-01-16T15:01:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.958+0000 7f778f200640 1 -- 172.21.15.141:0/1187638254 shutdown_connections 2024-01-16T15:01:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.958+0000 7f778f200640 1 --2- 172.21.15.141:0/1187638254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f778806d050 0x7f778806d450 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.958+0000 7f778f200640 1 --2- 172.21.15.141:0/1187638254 >> v2:172.21.15.141:3300/0 conn(0x7f77880b3c20 0x7f7788070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.958+0000 7f778f200640 1 -- 172.21.15.141:0/1187638254 >> 172.21.15.141:0/1187638254 conn(0x7f778806a800 msgr2=0x7f778806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:35.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.959+0000 7f778f200640 1 -- 172.21.15.141:0/1187638254 shutdown_connections 2024-01-16T15:01:35.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.960+0000 7f778f200640 1 -- 172.21.15.141:0/1187638254 wait complete. 2024-01-16T15:01:35.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.965+0000 7f778f200640 1 Processor -- start 2024-01-16T15:01:35.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.965+0000 7f778f200640 1 -- start start 2024-01-16T15:01:35.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.967+0000 7f778f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f778806d050 0x7f778814cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:35.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.967+0000 7f778f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77880b3c20 0x7f778814eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:35.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.967+0000 7f778f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7788159590 0x7f7788152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:35.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.967+0000 7f778f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778814d450 con 0x7f7788159590 2024-01-16T15:01:35.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.967+0000 7f778f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778814d7d0 con 0x7f77880b3c20 2024-01-16T15:01:35.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.967+0000 7f778f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778814da20 con 0x7f778806d050 2024-01-16T15:01:35.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.969+0000 7f778de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f778806d050 0x7f778814cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:35.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.969+0000 7f778de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f778806d050 0x7f778814cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:37978/0 (socket says 172.21.15.141:37978) 2024-01-16T15:01:35.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.969+0000 7f778de00640 1 -- 172.21.15.141:0/2691136681 learned_addr learned my addr 172.21.15.141:0/2691136681 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:35.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.969+0000 7f778d400640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77880b3c20 0x7f778814eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:35.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.970+0000 7f778e800640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7788159590 0x7f7788152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.970+0000 7f778de00640 1 -- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77880b3c20 msgr2=0x7f778814eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.970+0000 7f778de00640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77880b3c20 0x7f778814eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.970+0000 7f778de00640 1 -- 172.21.15.141:0/2691136681 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7788159590 msgr2=0x7f7788152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.970+0000 7f778de00640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7788159590 0x7f7788152870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.970+0000 7f778de00640 1 -- 172.21.15.141:0/2691136681 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778000d040 con 0x7f778806d050 2024-01-16T15:01:35.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.973+0000 7f778de00640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f778806d050 0x7f778814cf10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f777800b3e0 tx=0x7f777800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:35.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.978+0000 7f7786000640 1 -- 172.21.15.141:0/2691136681 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778011830 con 0x7f778806d050 2024-01-16T15:01:35.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.979+0000 7f7786000640 1 -- 172.21.15.141:0/2691136681 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77780119b0 con 0x7f778806d050 2024-01-16T15:01:35.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.979+0000 7f7786000640 1 -- 172.21.15.141:0/2691136681 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778011d60 con 0x7f778806d050 2024-01-16T15:01:35.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.981+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778814dd00 con 0x7f778806d050 2024-01-16T15:01:35.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.981+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778814e2c0 con 0x7f778806d050 2024-01-16T15:01:35.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.983+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77880ceb60 con 0x7f778806d050 2024-01-16T15:01:35.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.991+0000 7f7786000640 1 -- 172.21.15.141:0/2691136681 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f777801a400 con 0x7f778806d050 2024-01-16T15:01:35.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.992+0000 7f7786000640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f776006b820 0x7f776006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:35.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:35.992+0000 7f7786000640 1 -- 172.21.15.141:0/2691136681 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f777808aa00 con 0x7f778806d050 2024-01-16T15:01:35.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.005+0000 7f778d400640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f776006b820 0x7f776006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:35.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.013+0000 7f778d400640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f776006b820 0x7f776006dce0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f778815ca30 tx=0x7f7770008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:35.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.013+0000 7f7786000640 1 -- 172.21.15.141:0/2691136681 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7778055370 con 0x7f778806d050 2024-01-16T15:01:36.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.423+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7788067450 con 0x7f778806d050 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.426+0000 7f7786000640 1 -- 172.21.15.141:0/2691136681 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2145 (secure 0 0 0) 0x7f7778058990 con 0x7f778806d050 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:36.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 289, 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:36.409 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:36.410 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1940, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7999784761, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 15685001216, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 757409112064, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 409643212, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 397450549, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 98, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 101 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:36.411 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:36.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:36.413 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:36.414 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:36.414 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:01:36.414 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.41667500138282776, 2024-01-16T15:01:36.414 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:36.414 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:36.414 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:36.414 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:36.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.435+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f776006b820 msgr2=0x7f776006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:36.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f776006b820 0x7f776006dce0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f778815ca30 tx=0x7f7770008040 comp rx=0 tx=0).stop 2024-01-16T15:01:36.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f778806d050 msgr2=0x7f778814cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:36.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f778806d050 0x7f778814cf10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f777800b3e0 tx=0x7f777800b8a0 comp rx=0 tx=0).stop 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 shutdown_connections 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f776006b820 0x7f776006dce0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7788159590 0x7f7788152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77880b3c20 0x7f778814eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 --2- 172.21.15.141:0/2691136681 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f778806d050 0x7f778814cf10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.436+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 >> 172.21.15.141:0/2691136681 conn(0x7f778806a800 msgr2=0x7f77881561c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.437+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 shutdown_connections 2024-01-16T15:01:36.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:36.437+0000 7f778f200640 1 -- 172.21.15.141:0/2691136681 wait complete. 2024-01-16T15:01:38.470 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:38.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.707+0000 7f4e67e00640 1 Processor -- start 2024-01-16T15:01:38.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.707+0000 7f4e67e00640 1 -- start start 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e67e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e600d0b60 0x7f4e600ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e67e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4e600aede0 0x7f4e6006f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e67e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e60071de0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e67e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e600af270 con 0x7f4e600d0b60 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e67e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e60072030 con 0x7f4e600aede0 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e64a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e600d0b60 0x7f4e600ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e65400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4e6006a3e0 0x7f4e6006a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50422/0 (socket says 172.21.15.141:50422) 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.709+0000 7f4e65400640 1 -- 172.21.15.141:0/835573617 learned_addr learned my addr 172.21.15.141:0/835573617 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.710+0000 7f4e65e00640 1 --2- 172.21.15.141:0/835573617 >> v2:172.21.15.188:3300/0 conn(0x7f4e600aede0 0x7f4e6006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.710+0000 7f4e64a00640 1 -- 172.21.15.141:0/835573617 >> v2:172.21.15.188:3300/0 conn(0x7f4e600aede0 msgr2=0x7f4e6006f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.710+0000 7f4e64a00640 1 --2- 172.21.15.141:0/835573617 >> v2:172.21.15.188:3300/0 conn(0x7f4e600aede0 0x7f4e6006f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.710+0000 7f4e64a00640 1 -- 172.21.15.141:0/835573617 >> v1:172.21.15.141:6789/0 conn(0x7f4e6006a3e0 legacy=0x7f4e6006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.710+0000 7f4e64a00640 1 -- 172.21.15.141:0/835573617 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e60072310 con 0x7f4e600d0b60 2024-01-16T15:01:38.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.710+0000 7f4e5fe00640 1 -- 172.21.15.141:0/835573617 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943545026 0 0) 0x7f4e60071de0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.711+0000 7f4e64a00640 1 --2- 172.21.15.141:0/835573617 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e600d0b60 0x7f4e600ac8d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4e60071220 tx=0x7f4e4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af07bcef8e06cc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.711+0000 7f4e5fe00640 1 -- 172.21.15.141:0/835573617 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48031830 con 0x7f4e600d0b60 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.711+0000 7f4e5fe00640 1 -- 172.21.15.141:0/835573617 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e48031990 con 0x7f4e600d0b60 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.711+0000 7f4e5fe00640 1 -- 172.21.15.141:0/835573617 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48031d20 con 0x7f4e600d0b60 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 -- 172.21.15.141:0/835573617 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e600d0b60 msgr2=0x7f4e600ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 --2- 172.21.15.141:0/835573617 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e600d0b60 0x7f4e600ac8d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4e60071220 tx=0x7f4e4802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 -- 172.21.15.141:0/835573617 shutdown_connections 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 --2- 172.21.15.141:0/835573617 >> v2:172.21.15.188:3300/0 conn(0x7f4e600aede0 0x7f4e6006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 --2- 172.21.15.141:0/835573617 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e600d0b60 0x7f4e600ac8d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 -- 172.21.15.141:0/835573617 >> 172.21.15.141:0/835573617 conn(0x7f4e600cfef0 msgr2=0x7f4e600d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:38.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 -- 172.21.15.141:0/835573617 shutdown_connections 2024-01-16T15:01:38.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.712+0000 7f4e67e00640 1 -- 172.21.15.141:0/835573617 wait complete. 2024-01-16T15:01:38.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.714+0000 7f4e67e00640 1 Processor -- start 2024-01-16T15:01:38.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.714+0000 7f4e67e00640 1 -- start start 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e67e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e6006a3e0 0x7f4e6014d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e67e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e600aede0 0x7f4e601468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e67e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e600d0b60 0x7f4e60158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e67e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6014d670 con 0x7f4e6006a3e0 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e67e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60154160 con 0x7f4e600aede0 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e67e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600ccf00 con 0x7f4e600d0b60 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e6006a3e0 0x7f4e6014d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e6006a3e0 0x7f4e6014d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53498/0 (socket says 172.21.15.141:53498) 2024-01-16T15:01:38.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 -- 172.21.15.141:0/3917307236 learned_addr learned my addr 172.21.15.141:0/3917307236 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 -- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e600d0b60 msgr2=0x7f4e60158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e600d0b60 0x7f4e60158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 -- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e600aede0 msgr2=0x7f4e601468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e600aede0 0x7f4e601468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.715+0000 7f4e65400640 1 -- 172.21.15.141:0/3917307236 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e48009400 con 0x7f4e6006a3e0 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.716+0000 7f4e65400640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e6006a3e0 0x7f4e6014d130 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000c7c0 tx=0x7f4e5000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.716+0000 7f4e5d600640 1 -- 172.21.15.141:0/3917307236 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000a640 con 0x7f4e6006a3e0 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.716+0000 7f4e5d600640 1 -- 172.21.15.141:0/3917307236 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5000a7c0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.716+0000 7f4e5d600640 1 -- 172.21.15.141:0/3917307236 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000ab70 con 0x7f4e6006a3e0 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.716+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e600cd1e0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.716+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e600cd7a0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.717+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e60154380 con 0x7f4e6006a3e0 2024-01-16T15:01:38.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.718+0000 7f4e5d600640 1 -- 172.21.15.141:0/3917307236 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f4e5000acd0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.719+0000 7f4e5d600640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e3806b800 0x7f4e3806dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:38.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.719+0000 7f4e5d600640 1 -- 172.21.15.141:0/3917307236 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4e50089880 con 0x7f4e6006a3e0 2024-01-16T15:01:38.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.720+0000 7f4e64a00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e3806b800 0x7f4e3806dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:38.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.721+0000 7f4e64a00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e3806b800 0x7f4e3806dcc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e48036000 tx=0x7f4e48030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:38.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.724+0000 7f4e5d600640 1 -- 172.21.15.141:0/3917307236 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e500540b0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.974+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4e600675d0 con 0x7f4e6006a3e0 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.975+0000 7f4e5d600640 1 -- 172.21.15.141:0/3917307236 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2112 (secure 0 0 0) 0x7f4e50018850 con 0x7f4e6006a3e0 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:38.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 291, 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:38.957 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:38.958 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 2544, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 10537450985, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 20626194432, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 752467918848, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 438637579, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 429934987, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 104, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 109 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:38.959 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:38.960 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:38.961 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (0s)\n [==============..............] ", 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.5, 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:38.962 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:38.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e3806b800 msgr2=0x7f4e3806dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e3806b800 0x7f4e3806dcc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e48036000 tx=0x7f4e48030000 comp rx=0 tx=0).stop 2024-01-16T15:01:38.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e6006a3e0 msgr2=0x7f4e6014d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:38.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e6006a3e0 0x7f4e6014d130 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000c7c0 tx=0x7f4e5000cc80 comp rx=0 tx=0).stop 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 shutdown_connections 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e3806b800 0x7f4e3806dcc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e600d0b60 0x7f4e60158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e600aede0 0x7f4e601468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 --2- 172.21.15.141:0/3917307236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e6006a3e0 0x7f4e6014d130 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 >> 172.21.15.141:0/3917307236 conn(0x7f4e600cfef0 msgr2=0x7f4e6015c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 shutdown_connections 2024-01-16T15:01:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:38.978+0000 7f4e67e00640 1 -- 172.21.15.141:0/3917307236 wait complete. 2024-01-16T15:01:40.976 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:41.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.319+0000 7f0639c00640 1 Processor -- start 2024-01-16T15:01:41.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.319+0000 7f0639c00640 1 -- start start 2024-01-16T15:01:41.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.324+0000 7f0639c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0634143ae0 0x7f06340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:41.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.324+0000 7f0639c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06340cf7d0 0x7f063415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:41.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.324+0000 7f0639c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340cfc30 con 0x7f063415e1e0 2024-01-16T15:01:41.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.324+0000 7f0639c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340b1ee0 con 0x7f0634143ae0 2024-01-16T15:01:41.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.324+0000 7f0639c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340b2780 con 0x7f06340cf7d0 2024-01-16T15:01:41.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.326+0000 7f0632a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0634143ae0 0x7f06340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:41.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.326+0000 7f0632a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0634143ae0 0x7f06340cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53508/0 (socket says 172.21.15.141:53508) 2024-01-16T15:01:41.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.326+0000 7f0632a00640 1 -- 172.21.15.141:0/3387445187 learned_addr learned my addr 172.21.15.141:0/3387445187 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:41.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.327+0000 7f0632000640 1 --2- 172.21.15.141:0/3387445187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06340cf7d0 0x7f063415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:41.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.328+0000 7f0632a00640 1 -- 172.21.15.141:0/3387445187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06340cf7d0 msgr2=0x7f063415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:41.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.328+0000 7f0632a00640 1 --2- 172.21.15.141:0/3387445187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06340cf7d0 0x7f063415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.328+0000 7f0632a00640 1 -- 172.21.15.141:0/3387445187 >> v1:172.21.15.188:6789/0 conn(0x7f063415e1e0 legacy=0x7f063415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:41.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.328+0000 7f0632a00640 1 -- 172.21.15.141:0/3387445187 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063415e990 con 0x7f0634143ae0 2024-01-16T15:01:41.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.330+0000 7f0632a00640 1 --2- 172.21.15.141:0/3387445187 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0634143ae0 0x7f06340cf290 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f062c009e50 tx=0x7f062c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54dc26432f665b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:41.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.330+0000 7f0631600640 1 -- 172.21.15.141:0/3387445187 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c02ca70 con 0x7f0634143ae0 2024-01-16T15:01:41.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.330+0000 7f0631600640 1 -- 172.21.15.141:0/3387445187 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c02cbd0 con 0x7f0634143ae0 2024-01-16T15:01:41.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.330+0000 7f0631600640 1 -- 172.21.15.141:0/3387445187 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c0343f0 con 0x7f0634143ae0 2024-01-16T15:01:41.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.333+0000 7f0639c00640 1 -- 172.21.15.141:0/3387445187 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0634143ae0 msgr2=0x7f06340cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:41.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.333+0000 7f0639c00640 1 --2- 172.21.15.141:0/3387445187 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0634143ae0 0x7f06340cf290 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f062c009e50 tx=0x7f062c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.334+0000 7f0639c00640 1 -- 172.21.15.141:0/3387445187 shutdown_connections 2024-01-16T15:01:41.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.334+0000 7f0639c00640 1 --2- 172.21.15.141:0/3387445187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f06340cf7d0 0x7f063415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.334+0000 7f0639c00640 1 --2- 172.21.15.141:0/3387445187 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0634143ae0 0x7f06340cf290 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.334+0000 7f0639c00640 1 -- 172.21.15.141:0/3387445187 >> 172.21.15.141:0/3387445187 conn(0x7f063406a800 msgr2=0x7f063406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:41.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.335+0000 7f0639c00640 1 -- 172.21.15.141:0/3387445187 shutdown_connections 2024-01-16T15:01:41.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.335+0000 7f0639c00640 1 -- 172.21.15.141:0/3387445187 wait complete. 2024-01-16T15:01:41.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.337+0000 7f0639c00640 1 Processor -- start 2024-01-16T15:01:41.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.337+0000 7f0639c00640 1 -- start start 2024-01-16T15:01:41.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.339+0000 7f0639c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06340cf7d0 0x7f063415d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:41.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.339+0000 7f0639c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0634143ae0 0x7f0634140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:41.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.339+0000 7f0639c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f063415e1e0 0x7f063413e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:41.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.339+0000 7f0639c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063415cc90 con 0x7f06340cf7d0 2024-01-16T15:01:41.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.339+0000 7f0639c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063413ee90 con 0x7f0634143ae0 2024-01-16T15:01:41.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.339+0000 7f0639c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340ce4e0 con 0x7f063415e1e0 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06340cf7d0 0x7f063415d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06340cf7d0 0x7f063415d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53512/0 (socket says 172.21.15.141:53512) 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 -- 172.21.15.141:0/1828505932 learned_addr learned my addr 172.21.15.141:0/1828505932 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632000640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0634143ae0 0x7f0634140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0633400640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f063415e1e0 0x7f063413e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 -- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f063415e1e0 msgr2=0x7f063413e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f063415e1e0 0x7f063413e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 -- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0634143ae0 msgr2=0x7f0634140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0634143ae0 0x7f0634140550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.340+0000 7f0632a00640 1 -- 172.21.15.141:0/1828505932 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c009b30 con 0x7f06340cf7d0 2024-01-16T15:01:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.341+0000 7f0632a00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06340cf7d0 0x7f063415d120 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f062c009e50 tx=0x7f062c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.342+0000 7f062aa00640 1 -- 172.21.15.141:0/1828505932 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c009510 con 0x7f06340cf7d0 2024-01-16T15:01:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.342+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063415c360 con 0x7f06340cf7d0 2024-01-16T15:01:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.342+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063415c920 con 0x7f06340cf7d0 2024-01-16T15:01:41.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.342+0000 7f062aa00640 1 -- 172.21.15.141:0/1828505932 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c03e450 con 0x7f06340cf7d0 2024-01-16T15:01:41.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.342+0000 7f062aa00640 1 -- 172.21.15.141:0/1828505932 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c03e800 con 0x7f06340cf7d0 2024-01-16T15:01:41.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.344+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0604005250 con 0x7f06340cf7d0 2024-01-16T15:01:41.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.346+0000 7f062aa00640 1 -- 172.21.15.141:0/1828505932 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f062c03e960 con 0x7f06340cf7d0 2024-01-16T15:01:41.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.347+0000 7f062aa00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f060c06b7d0 0x7f060c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:41.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.347+0000 7f062aa00640 1 -- 172.21.15.141:0/1828505932 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f062c0ad190 con 0x7f06340cf7d0 2024-01-16T15:01:41.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.350+0000 7f0632000640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f060c06b7d0 0x7f060c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:41.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.352+0000 7f0632000640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f060c06b7d0 0x7f060c06dc90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06340cf650 tx=0x7f061c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:41.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.354+0000 7f062aa00640 1 -- 172.21.15.141:0/1828505932 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f062c077aa0 con 0x7f06340cf7d0 2024-01-16T15:01:41.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.777+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f06040057d0 con 0x7f06340cf7d0 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.778+0000 7f062aa00640 1 -- 172.21.15.141:0/1828505932 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2112 (secure 0 0 0) 0x7f062c07b0c0 con 0x7f06340cf7d0 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:41.759 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 294, 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:41.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:41.761 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 3115, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 12907399382, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 26054205440, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 747039907840, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 629821714, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 627762390, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 150, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 160 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:41.762 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:41.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:41.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (0s)\n [==============..............] ", 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.5, 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:41.765 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f060c06b7d0 msgr2=0x7f060c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f060c06b7d0 0x7f060c06dc90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06340cf650 tx=0x7f061c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06340cf7d0 msgr2=0x7f063415d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06340cf7d0 0x7f063415d120 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f062c009e50 tx=0x7f062c004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 shutdown_connections 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f060c06b7d0 0x7f060c06dc90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f063415e1e0 0x7f063413e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0634143ae0 0x7f0634140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 --2- 172.21.15.141:0/1828505932 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f06340cf7d0 0x7f063415d120 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.780+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 >> 172.21.15.141:0/1828505932 conn(0x7f063406a800 msgr2=0x7f06340aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:41.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.781+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 shutdown_connections 2024-01-16T15:01:41.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:41.781+0000 7f0639c00640 1 -- 172.21.15.141:0/1828505932 wait complete. 2024-01-16T15:01:43.786 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:43.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.013+0000 7f5284e00640 1 Processor -- start 2024-01-16T15:01:43.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.014+0000 7f5284e00640 1 -- start start 2024-01-16T15:01:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.015+0000 7f5284e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f528013df80 0x7f528015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.015+0000 7f5284e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f528013e360 0x7f52800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.015+0000 7f5284e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528006c470 con 0x7f528015f9c0 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.015+0000 7f5284e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528006c750 con 0x7f528013e360 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.015+0000 7f5284e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528006ca30 con 0x7f528013df80 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527d400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f528013e360 0x7f52800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f528015f9c0 0x7f52801408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44076/0 (socket says 172.21.15.141:44076) 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527d400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f528013e360 0x7f52800cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47370/0 (socket says 172.21.15.141:47370) 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527e800640 1 -- 172.21.15.141:0/3839430930 learned_addr learned my addr 172.21.15.141:0/3839430930 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527de00640 1 --2- 172.21.15.141:0/3839430930 >> v2:172.21.15.188:3300/0 conn(0x7f528013df80 0x7f528015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527d400640 1 -- 172.21.15.141:0/3839430930 >> v2:172.21.15.188:3300/0 conn(0x7f528013df80 msgr2=0x7f528015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527d400640 1 --2- 172.21.15.141:0/3839430930 >> v2:172.21.15.188:3300/0 conn(0x7f528013df80 0x7f528015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527d400640 1 -- 172.21.15.141:0/3839430930 >> v1:172.21.15.141:6789/0 conn(0x7f528015f9c0 legacy=0x7f52801408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527d400640 1 -- 172.21.15.141:0/3839430930 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528006cd10 con 0x7f528013e360 2024-01-16T15:01:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.016+0000 7f527de00640 1 --2- 172.21.15.141:0/3839430930 >> v2:172.21.15.188:3300/0 conn(0x7f528013df80 0x7f528015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.017+0000 7f527d400640 1 --2- 172.21.15.141:0/3839430930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52800cf290 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5280072770 tx=0x7f527002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8ec29098d7e4ab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f527ca00640 1 -- 172.21.15.141:0/3839430930 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527002cc70 con 0x7f528013e360 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f527ca00640 1 -- 172.21.15.141:0/3839430930 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527002cdd0 con 0x7f528013e360 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f527ca00640 1 -- 172.21.15.141:0/3839430930 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270034660 con 0x7f528013e360 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f5284e00640 1 -- 172.21.15.141:0/3839430930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 msgr2=0x7f52800cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f5284e00640 1 --2- 172.21.15.141:0/3839430930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52800cf290 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5280072770 tx=0x7f527002bec0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f5284e00640 1 -- 172.21.15.141:0/3839430930 shutdown_connections 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f5284e00640 1 --2- 172.21.15.141:0/3839430930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52800cf290 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f5284e00640 1 --2- 172.21.15.141:0/3839430930 >> v2:172.21.15.188:3300/0 conn(0x7f528013df80 0x7f528015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f5284e00640 1 -- 172.21.15.141:0/3839430930 >> 172.21.15.141:0/3839430930 conn(0x7f528006a800 msgr2=0x7f528006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.018+0000 7f5284e00640 1 -- 172.21.15.141:0/3839430930 shutdown_connections 2024-01-16T15:01:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.019+0000 7f5284e00640 1 -- 172.21.15.141:0/3839430930 wait complete. 2024-01-16T15:01:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.020+0000 7f5284e00640 1 Processor -- start 2024-01-16T15:01:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.020+0000 7f5284e00640 1 -- start start 2024-01-16T15:01:44.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.021+0000 7f5284e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f528013df80 0x7f528015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.021+0000 7f5284e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52801f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.021+0000 7f5284e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f528015f9c0 0x7f52801508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.021+0000 7f5284e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528015cd90 con 0x7f528013e360 2024-01-16T15:01:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.021+0000 7f5284e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528015cfc0 con 0x7f528015f9c0 2024-01-16T15:01:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.021+0000 7f5284e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528015d2a0 con 0x7f528013df80 2024-01-16T15:01:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.021+0000 7f527d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52801f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52801f5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47380/0 (socket says 172.21.15.141:47380) 2024-01-16T15:01:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 -- 172.21.15.141:0/1232969651 learned_addr learned my addr 172.21.15.141:0/1232969651 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527e800640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f528015f9c0 0x7f52801508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 -- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f528013df80 msgr2=0x7f528015c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f528013df80 0x7f528015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 -- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f528015f9c0 msgr2=0x7f52801508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f528015f9c0 0x7f52801508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 -- 172.21.15.141:0/1232969651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5270009a70 con 0x7f528013e360 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527e800640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f528015f9c0 0x7f52801508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f527d400640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52801f5e80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5270037040 tx=0x7f5270004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.022+0000 7f5276000640 1 -- 172.21.15.141:0/1232969651 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270040650 con 0x7f528013e360 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.023+0000 7f5276000640 1 -- 172.21.15.141:0/1232969651 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52700407b0 con 0x7f528013e360 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.023+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528015d580 con 0x7f528013e360 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.023+0000 7f5276000640 1 -- 172.21.15.141:0/1232969651 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270040b60 con 0x7f528013e360 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.023+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528015db40 con 0x7f528013e360 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.024+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52800675d0 con 0x7f528013e360 2024-01-16T15:01:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.025+0000 7f5276000640 1 -- 172.21.15.141:0/1232969651 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f5270040cc0 con 0x7f528013e360 2024-01-16T15:01:44.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.026+0000 7f5276000640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f525806b7d0 0x7f525806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:44.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.026+0000 7f5276000640 1 -- 172.21.15.141:0/1232969651 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f52700ad2f0 con 0x7f528013e360 2024-01-16T15:01:44.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.026+0000 7f527de00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f525806b7d0 0x7f525806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:44.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.027+0000 7f527de00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f525806b7d0 0x7f525806dc90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5268006fd0 tx=0x7f5268008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:44.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.031+0000 7f5276000640 1 -- 172.21.15.141:0/1232969651 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5270077ad0 con 0x7f528013e360 2024-01-16T15:01:44.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.290+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f528006df00 con 0x7f528013e360 2024-01-16T15:01:44.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.291+0000 7f5276000640 1 -- 172.21.15.141:0/1232969651 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2111 (secure 0 0 0) 0x7f527007b0f0 con 0x7f528013e360 2024-01-16T15:01:44.272 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:44.273 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 296, 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:44.274 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:44.275 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 3115, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 12907399382, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 26054205440, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 747039907840, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 398508395, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 408662150, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 95, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 104 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:44.276 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:44.277 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:44.278 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:01:44.279 INFO:teuthology.orchestra.run.smithi141.stdout: "e6aa9702-3a17-470b-a5cd-8314d1555657": { 2024-01-16T15:01:44.279 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (0s)\n [==============..............] ", 2024-01-16T15:01:44.279 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.5, 2024-01-16T15:01:44.279 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:01:44.279 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:44.279 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:44.279 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f525806b7d0 msgr2=0x7f525806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f525806b7d0 0x7f525806dc90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5268006fd0 tx=0x7f5268008040 comp rx=0 tx=0).stop 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 msgr2=0x7f52801f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52801f5e80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5270037040 tx=0x7f5270004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 shutdown_connections 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f525806b7d0 0x7f525806dc90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f528015f9c0 0x7f52801508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f528013e360 0x7f52801f5e80 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 --2- 172.21.15.141:0/1232969651 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f528013df80 0x7f528015c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:44.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.294+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 >> 172.21.15.141:0/1232969651 conn(0x7f528006a800 msgr2=0x7f52800b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:44.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.295+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 shutdown_connections 2024-01-16T15:01:44.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:44.295+0000 7f5284e00640 1 -- 172.21.15.141:0/1232969651 wait complete. 2024-01-16T15:01:46.294 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:46.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.574+0000 7f564ee00640 1 Processor -- start 2024-01-16T15:01:46.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.574+0000 7f564ee00640 1 -- start start 2024-01-16T15:01:46.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.576+0000 7f564ee00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5648070c00 0x7f56480d1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.576+0000 7f564ee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56480c5f50 0x7f56480c6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.576+0000 7f564ee00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56480cf430 con 0x7f564815b130 2024-01-16T15:01:46.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.576+0000 7f564ee00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56480cec60 con 0x7f5648070c00 2024-01-16T15:01:46.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.576+0000 7f564ee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56480d0890 con 0x7f56480c5f50 2024-01-16T15:01:46.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.578+0000 7f563f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5648070c00 0x7f56480d1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.578+0000 7f5647e00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f564815b130 0x7f564815f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:41328/0 (socket says 172.21.15.141:41328) 2024-01-16T15:01:46.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.578+0000 7f5647e00640 1 -- 172.21.15.141:0/2608613945 learned_addr learned my addr 172.21.15.141:0/2608613945 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:46.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.579+0000 7f563f200640 1 -- 172.21.15.141:0/2608613945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56480c5f50 msgr2=0x7f56480c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:46.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.579+0000 7f563f200640 1 --2- 172.21.15.141:0/2608613945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56480c5f50 0x7f56480c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.579+0000 7f563f200640 1 -- 172.21.15.141:0/2608613945 >> v1:172.21.15.188:6789/0 conn(0x7f564815b130 legacy=0x7f564815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:46.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.579+0000 7f563f200640 1 -- 172.21.15.141:0/2608613945 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56480cee70 con 0x7f5648070c00 2024-01-16T15:01:46.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.580+0000 7f563f200640 1 --2- 172.21.15.141:0/2608613945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5648070c00 0x7f56480d1330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f56300097a0 tx=0x7f563002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f7c8a6c353126 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:46.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.581+0000 7f5647400640 1 -- 172.21.15.141:0/2608613945 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630031780 con 0x7f5648070c00 2024-01-16T15:01:46.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.581+0000 7f5647400640 1 -- 172.21.15.141:0/2608613945 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56300318e0 con 0x7f5648070c00 2024-01-16T15:01:46.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.581+0000 7f5647400640 1 -- 172.21.15.141:0/2608613945 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630031c70 con 0x7f5648070c00 2024-01-16T15:01:46.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 -- 172.21.15.141:0/2608613945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5648070c00 msgr2=0x7f56480d1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 --2- 172.21.15.141:0/2608613945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5648070c00 0x7f56480d1330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f56300097a0 tx=0x7f563002b820 comp rx=0 tx=0).stop 2024-01-16T15:01:46.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 -- 172.21.15.141:0/2608613945 shutdown_connections 2024-01-16T15:01:46.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 --2- 172.21.15.141:0/2608613945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5648070c00 0x7f56480d1330 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 --2- 172.21.15.141:0/2608613945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56480c5f50 0x7f56480c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 -- 172.21.15.141:0/2608613945 >> 172.21.15.141:0/2608613945 conn(0x7f56480b2ce0 msgr2=0x7f56480b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:46.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 -- 172.21.15.141:0/2608613945 shutdown_connections 2024-01-16T15:01:46.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.582+0000 7f564ee00640 1 -- 172.21.15.141:0/2608613945 wait complete. 2024-01-16T15:01:46.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.584+0000 7f564ee00640 1 Processor -- start 2024-01-16T15:01:46.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.584+0000 7f564ee00640 1 -- start start 2024-01-16T15:01:46.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f564ee00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5648070c00 0x7f5648155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f564ee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56480c5f50 0x7f5648157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f564ee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f564815b130 0x7f5648157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f564ee00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481f4280 con 0x7f5648070c00 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f564ee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481f3600 con 0x7f56480c5f50 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f564ee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56481f38e0 con 0x7f564815b130 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f5647e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5648070c00 0x7f5648155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f5647e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5648070c00 0x7f5648155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47386/0 (socket says 172.21.15.141:47386) 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f5647e00640 1 -- 172.21.15.141:0/2324700756 learned_addr learned my addr 172.21.15.141:0/2324700756 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.585+0000 7f564ce00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f564815b130 0x7f5648157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f563f200640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56480c5f50 0x7f5648157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f5647e00640 1 -- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f564815b130 msgr2=0x7f5648157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f5647e00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f564815b130 0x7f5648157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f5647e00640 1 -- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56480c5f50 msgr2=0x7f5648157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f5647e00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56480c5f50 0x7f5648157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f5647e00640 1 -- 172.21.15.141:0/2324700756 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630009400 con 0x7f5648070c00 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f5647e00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5648070c00 0x7f5648155c90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f563800ba50 tx=0x7f563800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.586+0000 7f5644c00640 1 -- 172.21.15.141:0/2324700756 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800cbe0 con 0x7f5648070c00 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.587+0000 7f5644c00640 1 -- 172.21.15.141:0/2324700756 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800cd60 con 0x7f5648070c00 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.587+0000 7f5644c00640 1 -- 172.21.15.141:0/2324700756 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638014680 con 0x7f5648070c00 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.587+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56481f3bc0 con 0x7f5648070c00 2024-01-16T15:01:46.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.587+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56481561d0 con 0x7f5648070c00 2024-01-16T15:01:46.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.588+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56481f43c0 con 0x7f5648070c00 2024-01-16T15:01:46.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.589+0000 7f5644c00640 1 -- 172.21.15.141:0/2324700756 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f563801d3b0 con 0x7f5648070c00 2024-01-16T15:01:46.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.590+0000 7f5644c00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f562006b800 0x7f562006dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.590+0000 7f5644c00640 1 -- 172.21.15.141:0/2324700756 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f563808a300 con 0x7f5648070c00 2024-01-16T15:01:46.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.593+0000 7f563f200640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f562006b800 0x7f562006dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.594+0000 7f5644c00640 1 -- 172.21.15.141:0/2324700756 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56481f43c0 con 0x7f5648070c00 2024-01-16T15:01:46.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.597+0000 7f563f200640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f562006b800 0x7f562006dcc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5630036000 tx=0x7f563003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:46.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.990+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f56481f43c0 con 0x7f5648070c00 2024-01-16T15:01:46.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.991+0000 7f5644c00640 1 -- 172.21.15.141:0/2324700756 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1869 (secure 0 0 0) 0x7f5638019760 con 0x7f5648070c00 2024-01-16T15:01:46.972 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:01:46.972 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:46.973 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 299, 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:01:46.974 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 48, 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:01:46.975 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 3619, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 15034049284, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 30353670144, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 742740443136, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 574279190, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 585844371, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 137, 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 151 2024-01-16T15:01:46.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 28, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 3, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 3, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:01:46.977 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 5796 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:46.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:46.979 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:01:46.979 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 9, 2024-01-16T15:01:46.979 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:01:14.527633+0000", 2024-01-16T15:01:46.979 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:01:46.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:01:46.979 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": {} 2024-01-16T15:01:46.979 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:46.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f562006b800 msgr2=0x7f562006dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f562006b800 0x7f562006dcc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5630036000 tx=0x7f563003d040 comp rx=0 tx=0).stop 2024-01-16T15:01:46.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5648070c00 msgr2=0x7f5648155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5648070c00 0x7f5648155c90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f563800ba50 tx=0x7f563800bf10 comp rx=0 tx=0).stop 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 shutdown_connections 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f562006b800 0x7f562006dcc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f564815b130 0x7f5648157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56480c5f50 0x7f5648157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 --2- 172.21.15.141:0/2324700756 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5648070c00 0x7f5648155c90 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 >> 172.21.15.141:0/2324700756 conn(0x7f56480b2ce0 msgr2=0x7f56480b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 shutdown_connections 2024-01-16T15:01:46.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:46.995+0000 7f564ee00640 1 -- 172.21.15.141:0/2324700756 wait complete. 2024-01-16T15:01:46.998 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:46.999 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone29511 2024-01-16T15:01:47.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.195+0000 7fea86e00640 1 Processor -- start 2024-01-16T15:01:47.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.195+0000 7fea86e00640 1 -- start start 2024-01-16T15:01:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.196+0000 7fea86e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fea80157c30 0x7fea800cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.196+0000 7fea86e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8015deb0 con 0x7fea80071200 2024-01-16T15:01:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.196+0000 7fea86e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8015e190 con 0x7fea801618f0 2024-01-16T15:01:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.196+0000 7fea86e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea8015ea30 con 0x7fea80157c30 2024-01-16T15:01:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.197+0000 7fea84e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fea80071200 0x7fea8006dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44088/0 (socket says 172.21.15.141:44088) 2024-01-16T15:01:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.197+0000 7fea84e00640 1 -- 172.21.15.141:0/3986070724 learned_addr learned my addr 172.21.15.141:0/3986070724 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:47.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.197+0000 7fea7fe00640 1 --2- 172.21.15.141:0/3986070724 >> v2:172.21.15.188:3300/0 conn(0x7fea80157c30 0x7fea800cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.197+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193672252 0 0) 0x7fea8015deb0 con 0x7fea80071200 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.197+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea60003520 con 0x7fea80071200 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3276302694 0 0) 0x7fea60003520 con 0x7fea80071200 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 >> v2:172.21.15.188:3300/0 conn(0x7fea80157c30 msgr2=0x7fea800cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 --2- 172.21.15.141:0/3986070724 >> v2:172.21.15.188:3300/0 conn(0x7fea80157c30 0x7fea800cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 >> v1:172.21.15.188:6789/0 conn(0x7fea801618f0 legacy=0x7fea80070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8015e470 con 0x7fea80071200 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fea70001670 con 0x7fea80071200 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 >> v1:172.21.15.141:6789/0 conn(0x7fea80071200 legacy=0x7fea8006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600039c0 0x7fea60003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea600047c0 0x7fea60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea60005180 0x7fea6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8015ea30 con 0x7fea600047c0 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6000c080 con 0x7fea600039c0 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6000c240 con 0x7fea60005180 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7f400640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea600047c0 0x7fea60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7f400640 1 -- 172.21.15.141:0/3986070724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea60005180 msgr2=0x7fea6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7f400640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea60005180 0x7fea6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7f400640 1 -- 172.21.15.141:0/3986070724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600039c0 msgr2=0x7fea60003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7f400640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600039c0 0x7fea60003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea7f400640 1 -- 172.21.15.141:0/3986070724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea60003520 con 0x7fea600047c0 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.198+0000 7fea84e00640 1 -- 172.21.15.141:0/3986070724 reap_dead start 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.199+0000 7fea7f400640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea600047c0 0x7fea60004c40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fea68001e50 tx=0x7fea600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a23b0e1b1d5d854 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68003b70 con 0x7fea600047c0 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea68003cd0 con 0x7fea600047c0 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea7ea00640 1 -- 172.21.15.141:0/3986070724 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68005310 con 0x7fea600047c0 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea86e00640 1 -- 172.21.15.141:0/3986070724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea600047c0 msgr2=0x7fea60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea86e00640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea600047c0 0x7fea60004c40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fea68001e50 tx=0x7fea600057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea86e00640 1 -- 172.21.15.141:0/3986070724 shutdown_connections 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea86e00640 1 --2- 172.21.15.141:0/3986070724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea600047c0 0x7fea60004c40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea86e00640 1 -- 172.21.15.141:0/3986070724 >> 172.21.15.141:0/3986070724 conn(0x7fea800cf700 msgr2=0x7fea800cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.200+0000 7fea86e00640 1 -- 172.21.15.141:0/3986070724 shutdown_connections 2024-01-16T15:01:47.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.201+0000 7fea86e00640 1 -- 172.21.15.141:0/3986070724 wait complete. 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.202+0000 7fea86e00640 1 Processor -- start 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.202+0000 7fea86e00640 1 -- start start 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea86e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600047c0 0x7fea80145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea86e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea8015cd50 0x7fea8015d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea86e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea800b2fc0 0x7fea800b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea86e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea80146e80 con 0x7fea800b2fc0 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea86e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea800b3d20 con 0x7fea600047c0 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea86e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea801464e0 con 0x7fea8015cd50 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea84e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea800b2fc0 0x7fea800b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea84e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea800b2fc0 0x7fea800b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47392/0 (socket says 172.21.15.141:47392) 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.203+0000 7fea84e00640 1 -- 172.21.15.141:0/336518294 learned_addr learned my addr 172.21.15.141:0/336518294 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea7fe00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600047c0 0x7fea80145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea7f400640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea8015cd50 0x7fea8015d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea84e00640 1 -- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea8015cd50 msgr2=0x7fea8015d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea84e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea8015cd50 0x7fea8015d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea84e00640 1 -- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600047c0 msgr2=0x7fea80145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea84e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600047c0 0x7fea80145b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea84e00640 1 -- 172.21.15.141:0/336518294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea801467c0 con 0x7fea800b2fc0 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea7f400640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea8015cd50 0x7fea8015d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea84e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea800b2fc0 0x7fea800b3440 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fea80161cd0 tx=0x7fea7000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea77e00640 1 -- 172.21.15.141:0/336518294 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea70007440 con 0x7fea800b2fc0 2024-01-16T15:01:47.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea77e00640 1 -- 172.21.15.141:0/336518294 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea700075a0 con 0x7fea800b2fc0 2024-01-16T15:01:47.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea80146aa0 con 0x7fea800b2fc0 2024-01-16T15:01:47.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea77e00640 1 -- 172.21.15.141:0/336518294 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea70004a00 con 0x7fea800b2fc0 2024-01-16T15:01:47.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.204+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea80146050 con 0x7fea800b2fc0 2024-01-16T15:01:47.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.205+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea800675d0 con 0x7fea800b2fc0 2024-01-16T15:01:47.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.206+0000 7fea77e00640 1 -- 172.21.15.141:0/336518294 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fea70004b60 con 0x7fea800b2fc0 2024-01-16T15:01:47.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.207+0000 7fea77e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea5806b7d0 0x7fea5806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.207+0000 7fea77e00640 1 -- 172.21.15.141:0/336518294 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea70089a60 con 0x7fea800b2fc0 2024-01-16T15:01:47.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.208+0000 7fea7fe00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea5806b7d0 0x7fea5806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.209+0000 7fea7fe00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea5806b7d0 0x7fea5806dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fea780047b0 tx=0x7fea78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.211+0000 7fea77e00640 1 -- 172.21.15.141:0/336518294 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea70054200 con 0x7fea800b2fc0 2024-01-16T15:01:47.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:47.435+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone29511", "target": ["mon-mgr", ""]}) v1 -- 0x7fea80071ae0 con 0x7fea5806b7d0 2024-01-16T15:01:48.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.138+0000 7fea77e00640 1 -- 172.21.15.141:0/336518294 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fea80071ae0 con 0x7fea5806b7d0 2024-01-16T15:01:48.120 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:48.120 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:01:48.120 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:01:48.120 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:48.120 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea5806b7d0 msgr2=0x7fea5806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea5806b7d0 0x7fea5806dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fea780047b0 tx=0x7fea78008040 comp rx=0 tx=0).stop 2024-01-16T15:01:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea800b2fc0 msgr2=0x7fea800b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea800b2fc0 0x7fea800b3440 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fea80161cd0 tx=0x7fea7000c890 comp rx=0 tx=0).stop 2024-01-16T15:01:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 shutdown_connections 2024-01-16T15:01:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea5806b7d0 0x7fea5806dc90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea800b2fc0 0x7fea800b3440 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea8015cd50 0x7fea8015d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 --2- 172.21.15.141:0/336518294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea600047c0 0x7fea80145b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.141+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 >> 172.21.15.141:0/336518294 conn(0x7fea800cf700 msgr2=0x7fea8006e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.142+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 shutdown_connections 2024-01-16T15:01:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.142+0000 7fea86e00640 1 -- 172.21.15.141:0/336518294 wait complete. 2024-01-16T15:01:48.142 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:48.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone29512 2024-01-16T15:01:48.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.366+0000 7fee18c00640 1 Processor -- start 2024-01-16T15:01:48.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.366+0000 7fee18c00640 1 -- start start 2024-01-16T15:01:48.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee18c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fee140cf9a0 0x7fee1415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:48.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee18c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee1415e1e0 0x7fee140702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:48.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee18c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140b1ee0 con 0x7fee14143ae0 2024-01-16T15:01:48.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee18c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee140b2780 con 0x7fee140cf9a0 2024-01-16T15:01:48.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee18c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415e990 con 0x7fee1415e1e0 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee0be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fee140cf9a0 0x7fee1415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee0be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fee140cf9a0 0x7fee1415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47406/0 (socket says 172.21.15.141:47406) 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee0be00640 1 -- 172.21.15.141:0/215205152 learned_addr learned my addr 172.21.15.141:0/215205152 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.368+0000 7fee11e00640 1 --1- 172.21.15.141:0/215205152 >> v1:172.21.15.141:6789/0 conn(0x7fee14143ae0 0x7fee140cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44098/0 (socket says 172.21.15.141:44098) 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.369+0000 7fee0be00640 1 -- 172.21.15.141:0/215205152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee1415e1e0 msgr2=0x7fee140702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.369+0000 7fee12800640 1 --2- 172.21.15.141:0/215205152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee1415e1e0 0x7fee140702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.369+0000 7fee0be00640 1 --2- 172.21.15.141:0/215205152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee1415e1e0 0x7fee140702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.369+0000 7fee11400640 1 -- 172.21.15.141:0/215205152 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191320382 0 0) 0x7fee140b1ee0 con 0x7fee14143ae0 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.369+0000 7fee0be00640 1 -- 172.21.15.141:0/215205152 >> v1:172.21.15.141:6789/0 conn(0x7fee14143ae0 legacy=0x7fee140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.369+0000 7fee0be00640 1 -- 172.21.15.141:0/215205152 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140b2130 con 0x7fee140cf9a0 2024-01-16T15:01:48.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.369+0000 7fee12800640 1 --2- 172.21.15.141:0/215205152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee1415e1e0 0x7fee140702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:48.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.370+0000 7fee0be00640 1 --2- 172.21.15.141:0/215205152 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee1415dca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bb10 tx=0x7fedfc02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c416284a0683e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:48.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.370+0000 7fee11400640 1 -- 172.21.15.141:0/215205152 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc036440 con 0x7fee140cf9a0 2024-01-16T15:01:48.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.370+0000 7fee11400640 1 -- 172.21.15.141:0/215205152 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc0365a0 con 0x7fee140cf9a0 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.370+0000 7fee11400640 1 -- 172.21.15.141:0/215205152 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc036930 con 0x7fee140cf9a0 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 -- 172.21.15.141:0/215205152 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 msgr2=0x7fee1415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 --2- 172.21.15.141:0/215205152 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee1415dca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bb10 tx=0x7fedfc02e150 comp rx=0 tx=0).stop 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 -- 172.21.15.141:0/215205152 shutdown_connections 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 --2- 172.21.15.141:0/215205152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee1415e1e0 0x7fee140702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 --2- 172.21.15.141:0/215205152 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee1415dca0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 -- 172.21.15.141:0/215205152 >> 172.21.15.141:0/215205152 conn(0x7fee1406a800 msgr2=0x7fee1406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 -- 172.21.15.141:0/215205152 shutdown_connections 2024-01-16T15:01:48.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.371+0000 7fee18c00640 1 -- 172.21.15.141:0/215205152 wait complete. 2024-01-16T15:01:48.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.373+0000 7fee18c00640 1 Processor -- start 2024-01-16T15:01:48.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.373+0000 7fee18c00640 1 -- start start 2024-01-16T15:01:48.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee18c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee141419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:48.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee18c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee14143ae0 0x7fee140c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee18c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1415e1e0 0x7fee1415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee18c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140c9190 con 0x7fee140cf9a0 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee18c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140c88a0 con 0x7fee1415e1e0 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee18c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140c8b40 con 0x7fee14143ae0 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee11e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee141419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee11e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee141419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47418/0 (socket says 172.21.15.141:47418) 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.374+0000 7fee11e00640 1 -- 172.21.15.141:0/820192475 learned_addr learned my addr 172.21.15.141:0/820192475 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee0be00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee14143ae0 0x7fee140c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee12800640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1415e1e0 0x7fee1415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee11e00640 1 -- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee14143ae0 msgr2=0x7fee140c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee11e00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee14143ae0 0x7fee140c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee11e00640 1 -- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1415e1e0 msgr2=0x7fee1415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee11e00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1415e1e0 0x7fee1415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee11e00640 1 -- 172.21.15.141:0/820192475 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc00b770 con 0x7fee140cf9a0 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee12800640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1415e1e0 0x7fee1415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee11e00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee141419a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c4d0 tx=0x7fee0400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee0a000640 1 -- 172.21.15.141:0/820192475 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee040074e0 con 0x7fee140cf9a0 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee0a000640 1 -- 172.21.15.141:0/820192475 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04007660 con 0x7fee140cf9a0 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee0a000640 1 -- 172.21.15.141:0/820192475 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04004a20 con 0x7fee140cf9a0 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.375+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee140c8e20 con 0x7fee140cf9a0 2024-01-16T15:01:48.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.376+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee140ae680 con 0x7fee140cf9a0 2024-01-16T15:01:48.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.377+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee140675d0 con 0x7fee140cf9a0 2024-01-16T15:01:48.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.378+0000 7fee0a000640 1 -- 172.21.15.141:0/820192475 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fee04004b80 con 0x7fee140cf9a0 2024-01-16T15:01:48.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.378+0000 7fee0a000640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fedec06b7d0 0x7fedec06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:48.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.379+0000 7fee0a000640 1 -- 172.21.15.141:0/820192475 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fee04089650 con 0x7fee140cf9a0 2024-01-16T15:01:48.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.379+0000 7fee0be00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fedec06b7d0 0x7fedec06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:48.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.380+0000 7fee0be00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fedec06b7d0 0x7fedec06dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fedfc038000 tx=0x7fedfc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:48.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.383+0000 7fee0a000640 1 -- 172.21.15.141:0/820192475 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee04053df0 con 0x7fee140cf9a0 2024-01-16T15:01:48.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:48.613+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone29512", "target": ["mon-mgr", ""]}) v1 -- 0x7fee1415e670 con 0x7fedec06b7d0 2024-01-16T15:01:53.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.134+0000 7fee0a000640 1 -- 172.21.15.141:0/820192475 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fedfc036710 con 0x7fedec06b7d0 2024-01-16T15:01:53.115 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:53.116 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:01:53.116 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:01:53.116 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:53.116 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fedec06b7d0 msgr2=0x7fedec06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fedec06b7d0 0x7fedec06dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fedfc038000 tx=0x7fedfc008040 comp rx=0 tx=0).stop 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 msgr2=0x7fee141419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee141419a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c4d0 tx=0x7fee0400c990 comp rx=0 tx=0).stop 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 shutdown_connections 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fedec06b7d0 0x7fedec06dc90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1415e1e0 0x7fee1415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee14143ae0 0x7fee140c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 --2- 172.21.15.141:0/820192475 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee140cf9a0 0x7fee141419a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 >> 172.21.15.141:0/820192475 conn(0x7fee1406a800 msgr2=0x7fee14142db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:53.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 shutdown_connections 2024-01-16T15:01:53.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.138+0000 7fee18c00640 1 -- 172.21.15.141:0/820192475 wait complete. 2024-01-16T15:01:53.138 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:53.138 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone29513 2024-01-16T15:01:53.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.373+0000 7f2b73c00640 1 Processor -- start 2024-01-16T15:01:53.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.373+0000 7f2b73c00640 1 -- start start 2024-01-16T15:01:53.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.374+0000 7f2b73c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2b6c0cf9a0 0x7f2b6c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.374+0000 7f2b73c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6c070c40 con 0x7f2b6c0b3ae0 2024-01-16T15:01:53.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b73c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6c06c470 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b73c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6c071450 con 0x7f2b6c0cf9a0 2024-01-16T15:01:53.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b63c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2b6c0b3ae0 0x7f2b6c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41894/0 (socket says 172.21.15.141:41894) 2024-01-16T15:01:53.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b63c00640 1 -- 172.21.15.141:0/371506380 learned_addr learned my addr 172.21.15.141:0/371506380 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:01:53.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b71c00640 1 --2- 172.21.15.141:0/371506380 >> v2:172.21.15.141:3300/0 conn(0x7f2b6c0cf9a0 0x7f2b6c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b71c00640 1 -- 172.21.15.141:0/371506380 >> v1:172.21.15.188:6789/0 conn(0x7f2b6c0ae5a0 legacy=0x7f2b6c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:53.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b71c00640 1 -- 172.21.15.141:0/371506380 >> v1:172.21.15.141:6789/0 conn(0x7f2b6c0b3ae0 legacy=0x7f2b6c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.375+0000 7f2b71c00640 1 -- 172.21.15.141:0/371506380 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c06c750 con 0x7f2b6c0cf9a0 2024-01-16T15:01:53.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.376+0000 7f2b71c00640 1 --2- 172.21.15.141:0/371506380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0cf9a0 0x7f2b6c15e430 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009d90 tx=0x7f2b6402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55509c22e96c50a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:53.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.377+0000 7f2b6be00640 1 -- 172.21.15.141:0/371506380 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6402ccf0 con 0x7f2b6c0cf9a0 2024-01-16T15:01:53.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.377+0000 7f2b6be00640 1 -- 172.21.15.141:0/371506380 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6402ce50 con 0x7f2b6c0cf9a0 2024-01-16T15:01:53.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.377+0000 7f2b6be00640 1 -- 172.21.15.141:0/371506380 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64034660 con 0x7f2b6c0cf9a0 2024-01-16T15:01:53.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.377+0000 7f2b73c00640 1 -- 172.21.15.141:0/371506380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0cf9a0 msgr2=0x7f2b6c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.377+0000 7f2b73c00640 1 --2- 172.21.15.141:0/371506380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0cf9a0 0x7f2b6c15e430 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009d90 tx=0x7f2b6402bf40 comp rx=0 tx=0).stop 2024-01-16T15:01:53.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.378+0000 7f2b73c00640 1 -- 172.21.15.141:0/371506380 shutdown_connections 2024-01-16T15:01:53.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.378+0000 7f2b73c00640 1 --2- 172.21.15.141:0/371506380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0cf9a0 0x7f2b6c15e430 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.378+0000 7f2b73c00640 1 -- 172.21.15.141:0/371506380 >> 172.21.15.141:0/371506380 conn(0x7f2b6c06a800 msgr2=0x7f2b6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:53.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.378+0000 7f2b73c00640 1 -- 172.21.15.141:0/371506380 shutdown_connections 2024-01-16T15:01:53.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.378+0000 7f2b73c00640 1 -- 172.21.15.141:0/371506380 wait complete. 2024-01-16T15:01:53.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.379+0000 7f2b73c00640 1 Processor -- start 2024-01-16T15:01:53.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.379+0000 7f2b73c00640 1 -- start start 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.380+0000 7f2b73c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0ae5a0 0x7f2b6c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.380+0000 7f2b73c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b6c0b3ae0 0x7f2b6c147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.380+0000 7f2b73c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b6c0cf9a0 0x7f2b6c147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.380+0000 7f2b73c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c15f5f0 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.380+0000 7f2b73c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c15f8d0 con 0x7f2b6c0b3ae0 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.380+0000 7f2b73c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c15fbb0 con 0x7f2b6c0cf9a0 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0ae5a0 0x7f2b6c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0ae5a0 0x7f2b6c15eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54014/0 (socket says 172.21.15.141:54014) 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 -- 172.21.15.141:0/861908089 learned_addr learned my addr 172.21.15.141:0/861908089 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 -- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b6c0cf9a0 msgr2=0x7f2b6c147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b6c0cf9a0 0x7f2b6c147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 -- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b6c0b3ae0 msgr2=0x7f2b6c147010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b6c0b3ae0 0x7f2b6c147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 -- 172.21.15.141:0/861908089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b64009a70 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.381+0000 7f2b71200640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0ae5a0 0x7f2b6c15eb50 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00c430 tx=0x7f2b5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:53.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.382+0000 7f2b69600640 1 -- 172.21.15.141:0/861908089 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c011790 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.382+0000 7f2b69600640 1 -- 172.21.15.141:0/861908089 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c0118f0 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.382+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c15fe90 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.382+0000 7f2b69600640 1 -- 172.21.15.141:0/861908089 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c011ce0 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.382+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c14dea0 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.383+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6c1603b0 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.384+0000 7f2b69600640 1 -- 172.21.15.141:0/861908089 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f2b5c01c460 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.385+0000 7f2b69600640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b4406b7d0 0x7f2b4406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:53.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.385+0000 7f2b69600640 1 -- 172.21.15.141:0/861908089 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b5c08a330 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.385+0000 7f2b63c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b4406b7d0 0x7f2b4406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:53.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.387+0000 7f2b63c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b4406b7d0 0x7f2b4406dc90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c160c40 tx=0x7f2b54007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:53.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.389+0000 7f2b69600640 1 -- 172.21.15.141:0/861908089 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b6c1603b0 con 0x7f2b6c0ae5a0 2024-01-16T15:01:53.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:53.613+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone29513", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b6c070550 con 0x7f2b4406b7d0 2024-01-16T15:01:58.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.135+0000 7f2b69600640 1 -- 172.21.15.141:0/861908089 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f2b540012b0 con 0x7f2b4406b7d0 2024-01-16T15:01:58.115 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:01:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:01:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:01:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:01:58.116 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:01:58.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b4406b7d0 msgr2=0x7f2b4406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b4406b7d0 0x7f2b4406dc90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c160c40 tx=0x7f2b54007a40 comp rx=0 tx=0).stop 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0ae5a0 msgr2=0x7f2b6c15eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0ae5a0 0x7f2b6c15eb50 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00c430 tx=0x7f2b5c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 shutdown_connections 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b4406b7d0 0x7f2b4406dc90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b6c0cf9a0 0x7f2b6c147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b6c0b3ae0 0x7f2b6c147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 --2- 172.21.15.141:0/861908089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b6c0ae5a0 0x7f2b6c15eb50 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.138+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 >> 172.21.15.141:0/861908089 conn(0x7f2b6c06a800 msgr2=0x7f2b6c160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.139+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 shutdown_connections 2024-01-16T15:01:58.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.139+0000 7f2b73c00640 1 -- 172.21.15.141:0/861908089 wait complete. 2024-01-16T15:01:58.141 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:58.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone29514 2024-01-16T15:01:58.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.372+0000 7fe052e00640 1 Processor -- start 2024-01-16T15:01:58.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.372+0000 7fe052e00640 1 -- start start 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe052e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe04c143ae0 0x7fe04c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe052e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c0cf7d0 0x7fe04c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe052e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe04c15e1e0 0x7fe04c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe052e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04c0cfc30 con 0x7fe04c143ae0 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe052e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04c0b1ee0 con 0x7fe04c15e1e0 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe052e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0b2780 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe04be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe04c143ae0 0x7fe04c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe04be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe04c143ae0 0x7fe04c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54026/0 (socket says 172.21.15.141:54026) 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe04be00640 1 -- 172.21.15.141:0/2017390107 learned_addr learned my addr 172.21.15.141:0/2017390107 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe050e00640 1 --2- 172.21.15.141:0/2017390107 >> v2:172.21.15.188:3300/0 conn(0x7fe04c15e1e0 0x7fe04c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe043e00640 1 --2- 172.21.15.141:0/2017390107 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c0cf7d0 0x7fe04c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe04be00640 1 -- 172.21.15.141:0/2017390107 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c0cf7d0 msgr2=0x7fe04c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:58.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.374+0000 7fe04be00640 1 --2- 172.21.15.141:0/2017390107 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c0cf7d0 0x7fe04c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.375+0000 7fe04be00640 1 -- 172.21.15.141:0/2017390107 >> v2:172.21.15.188:3300/0 conn(0x7fe04c15e1e0 msgr2=0x7fe04c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:58.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.375+0000 7fe04be00640 1 --2- 172.21.15.141:0/2017390107 >> v2:172.21.15.188:3300/0 conn(0x7fe04c15e1e0 0x7fe04c15a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.375+0000 7fe04be00640 1 -- 172.21.15.141:0/2017390107 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c15e990 con 0x7fe04c143ae0 2024-01-16T15:01:58.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.375+0000 7fe043e00640 1 --2- 172.21.15.141:0/2017390107 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c0cf7d0 0x7fe04c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:58.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.376+0000 7fe04be00640 1 --2- 172.21.15.141:0/2017390107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c143ae0 0x7fe04c0cf290 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe04c072160 tx=0x7fe03c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f1867aafa391c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:58.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.376+0000 7fe04b400640 1 -- 172.21.15.141:0/2017390107 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c031870 con 0x7fe04c143ae0 2024-01-16T15:01:58.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.376+0000 7fe04b400640 1 -- 172.21.15.141:0/2017390107 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c0319d0 con 0x7fe04c143ae0 2024-01-16T15:01:58.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.376+0000 7fe04b400640 1 -- 172.21.15.141:0/2017390107 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c031d60 con 0x7fe04c143ae0 2024-01-16T15:01:58.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 -- 172.21.15.141:0/2017390107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c143ae0 msgr2=0x7fe04c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 --2- 172.21.15.141:0/2017390107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c143ae0 0x7fe04c0cf290 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe04c072160 tx=0x7fe03c02b910 comp rx=0 tx=0).stop 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 -- 172.21.15.141:0/2017390107 shutdown_connections 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 --2- 172.21.15.141:0/2017390107 >> v2:172.21.15.188:3300/0 conn(0x7fe04c15e1e0 0x7fe04c15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 --2- 172.21.15.141:0/2017390107 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c0cf7d0 0x7fe04c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 --2- 172.21.15.141:0/2017390107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c143ae0 0x7fe04c0cf290 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 -- 172.21.15.141:0/2017390107 >> 172.21.15.141:0/2017390107 conn(0x7fe04c06a800 msgr2=0x7fe04c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 -- 172.21.15.141:0/2017390107 shutdown_connections 2024-01-16T15:01:58.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.377+0000 7fe052e00640 1 -- 172.21.15.141:0/2017390107 wait complete. 2024-01-16T15:01:58.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.379+0000 7fe052e00640 1 Processor -- start 2024-01-16T15:01:58.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.379+0000 7fe052e00640 1 -- start start 2024-01-16T15:01:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.380+0000 7fe052e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c0cf7d0 0x7fe04c0ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.380+0000 7fe052e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c143ae0 0x7fe04c12fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.380+0000 7fe052e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe04c15e1e0 0x7fe04c0aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.380+0000 7fe052e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0aeac0 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.380+0000 7fe052e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c12e790 con 0x7fe04c15e1e0 2024-01-16T15:01:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.380+0000 7fe052e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0ae1b0 con 0x7fe04c143ae0 2024-01-16T15:01:58.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c0cf7d0 0x7fe04c0ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c0cf7d0 0x7fe04c0ad950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:54028/0 (socket says 172.21.15.141:54028) 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 -- 172.21.15.141:0/556575425 learned_addr learned my addr 172.21.15.141:0/556575425 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe050e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe04c15e1e0 0x7fe04c0aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe043e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c143ae0 0x7fe04c12fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 -- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c143ae0 msgr2=0x7fe04c12fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c143ae0 0x7fe04c12fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 -- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe04c15e1e0 msgr2=0x7fe04c0aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe04c15e1e0 0x7fe04c0aae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 -- 172.21.15.141:0/556575425 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c0094d0 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.381+0000 7fe04be00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c0cf7d0 0x7fe04c0ad950 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe03c02b950 tx=0x7fe03c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:58.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.382+0000 7fe048c00640 1 -- 172.21.15.141:0/556575425 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c03ad30 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.382+0000 7fe048c00640 1 -- 172.21.15.141:0/556575425 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c0443f0 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.382+0000 7fe048c00640 1 -- 172.21.15.141:0/556575425 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0447a0 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.382+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04c0ae490 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.382+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04c0ae770 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.383+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04c0aef40 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.384+0000 7fe048c00640 1 -- 172.21.15.141:0/556575425 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fe03c04d450 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.385+0000 7fe048c00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe02406b800 0x7fe02406dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:58.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.385+0000 7fe048c00640 1 -- 172.21.15.141:0/556575425 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe03c02d040 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.386+0000 7fe043e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe02406b800 0x7fe02406dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:58.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.387+0000 7fe043e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe02406b800 0x7fe02406dcc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe034006fd0 tx=0x7fe034008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:58.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.390+0000 7fe048c00640 1 -- 172.21.15.141:0/556575425 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe03c0771c0 con 0x7fe04c0cf7d0 2024-01-16T15:01:58.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:01:58.618+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone29514", "target": ["mon-mgr", ""]}) v1 -- 0x7fe04c15e640 con 0x7fe02406b800 2024-01-16T15:02:03.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.130+0000 7fe048c00640 1 -- 172.21.15.141:0/556575425 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fe03400c040 con 0x7fe02406b800 2024-01-16T15:02:03.110 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:02:03.110 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:02:03.110 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:02:03.111 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:03.111 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe02406b800 msgr2=0x7fe02406dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe02406b800 0x7fe02406dcc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe034006fd0 tx=0x7fe034008040 comp rx=0 tx=0).stop 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c0cf7d0 msgr2=0x7fe04c0ad950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c0cf7d0 0x7fe04c0ad950 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe03c02b950 tx=0x7fe03c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 shutdown_connections 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe02406b800 0x7fe02406dcc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe04c15e1e0 0x7fe04c0aae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe04c143ae0 0x7fe04c12fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 --2- 172.21.15.141:0/556575425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe04c0cf7d0 0x7fe04c0ad950 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 >> 172.21.15.141:0/556575425 conn(0x7fe04c06a800 msgr2=0x7fe04c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 shutdown_connections 2024-01-16T15:02:03.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.134+0000 7fe052e00640 1 -- 172.21.15.141:0/556575425 wait complete. 2024-01-16T15:02:03.138 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:02:03.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.369+0000 7fae4be00640 1 Processor -- start 2024-01-16T15:02:03.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.370+0000 7fae4be00640 1 -- start start 2024-01-16T15:02:03.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.371+0000 7fae4be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 0x7fae4c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.371+0000 7fae4be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fae4c0717d0 0x7fae4c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.371+0000 7fae4be00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4c070e50 con 0x7fae4c0cca40 2024-01-16T15:02:03.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.371+0000 7fae4be00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0723c0 con 0x7fae4c0717d0 2024-01-16T15:02:03.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.371+0000 7fae4be00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0cce70 con 0x7fae4c143c40 2024-01-16T15:02:03.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae41e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fae4c0717d0 0x7fae4c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae41e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fae4c0717d0 0x7fae4c0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57948/0 (socket says 172.21.15.141:57948) 2024-01-16T15:02:03.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae41e00640 1 -- 172.21.15.141:0/32470569 learned_addr learned my addr 172.21.15.141:0/32470569 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:03.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae4b400640 1 --1- 172.21.15.141:0/32470569 >> v1:172.21.15.141:6789/0 conn(0x7fae4c0cca40 0x7fae4c15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41614/0 (socket says 172.21.15.141:41614) 2024-01-16T15:02:03.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae4aa00640 1 --2- 172.21.15.141:0/32470569 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 0x7fae4c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae41e00640 1 -- 172.21.15.141:0/32470569 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 msgr2=0x7fae4c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:03.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae41e00640 1 --2- 172.21.15.141:0/32470569 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 0x7fae4c071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae41e00640 1 -- 172.21.15.141:0/32470569 >> v1:172.21.15.141:6789/0 conn(0x7fae4c0cca40 legacy=0x7fae4c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.372+0000 7fae41e00640 1 -- 172.21.15.141:0/32470569 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c06e170 con 0x7fae4c0717d0 2024-01-16T15:02:03.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.373+0000 7fae41e00640 1 --2- 172.21.15.141:0/32470569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c0cc500 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fae34009870 tx=0x7fae3402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2287f5510dd60322 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.373+0000 7fae4a000640 1 -- 172.21.15.141:0/32470569 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34031640 con 0x7fae4c0717d0 2024-01-16T15:02:03.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4a000640 1 -- 172.21.15.141:0/32470569 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae340317a0 con 0x7fae4c0717d0 2024-01-16T15:02:03.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4a000640 1 -- 172.21.15.141:0/32470569 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34031b30 con 0x7fae4c0717d0 2024-01-16T15:02:03.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 -- 172.21.15.141:0/32470569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 msgr2=0x7fae4c0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 --2- 172.21.15.141:0/32470569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c0cc500 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fae34009870 tx=0x7fae3402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 -- 172.21.15.141:0/32470569 shutdown_connections 2024-01-16T15:02:03.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 --2- 172.21.15.141:0/32470569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c0cc500 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 --2- 172.21.15.141:0/32470569 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 0x7fae4c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 -- 172.21.15.141:0/32470569 >> 172.21.15.141:0/32470569 conn(0x7fae4c06a800 msgr2=0x7fae4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:03.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 -- 172.21.15.141:0/32470569 shutdown_connections 2024-01-16T15:02:03.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.374+0000 7fae4be00640 1 -- 172.21.15.141:0/32470569 wait complete. 2024-01-16T15:02:03.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.376+0000 7fae4be00640 1 Processor -- start 2024-01-16T15:02:03.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.376+0000 7fae4be00640 1 -- start start 2024-01-16T15:02:03.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c1395e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae4c0cca40 0x7fae4c1f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 0x7fae4c1423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4be00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0b1f00 con 0x7fae4c0717d0 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4be00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c1f35a0 con 0x7fae4c0cca40 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4be00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c1f3880 con 0x7fae4c143c40 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c1395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c1395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57952/0 (socket says 172.21.15.141:57952) 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.377+0000 7fae4aa00640 1 -- 172.21.15.141:0/1459204491 learned_addr learned my addr 172.21.15.141:0/1459204491 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae4aa00640 1 -- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 msgr2=0x7fae4c1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae4aa00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 0x7fae4c1423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae4aa00640 1 -- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae4c0cca40 msgr2=0x7fae4c1f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae4aa00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae4c0cca40 0x7fae4c1f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae4aa00640 1 -- 172.21.15.141:0/1459204491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae340094d0 con 0x7fae4c0717d0 2024-01-16T15:02:03.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae4aa00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c1395e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00c7f0 tx=0x7fae3c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae43e00640 1 -- 172.21.15.141:0/1459204491 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c004220 con 0x7fae4c0717d0 2024-01-16T15:02:03.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae43e00640 1 -- 172.21.15.141:0/1459204491 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c004380 con 0x7fae4c0717d0 2024-01-16T15:02:03.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae43e00640 1 -- 172.21.15.141:0/1459204491 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c007840 con 0x7fae4c0717d0 2024-01-16T15:02:03.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.378+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c0b2110 con 0x7fae4c0717d0 2024-01-16T15:02:03.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.379+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c0adff0 con 0x7fae4c0717d0 2024-01-16T15:02:03.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.380+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c1f3aa0 con 0x7fae4c0717d0 2024-01-16T15:02:03.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.381+0000 7fae43e00640 1 -- 172.21.15.141:0/1459204491 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fae3c007bb0 con 0x7fae4c0717d0 2024-01-16T15:02:03.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.382+0000 7fae43e00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae2406b7d0 0x7fae2406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.382+0000 7fae43e00640 1 -- 172.21.15.141:0/1459204491 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fae3c08a3e0 con 0x7fae4c0717d0 2024-01-16T15:02:03.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.382+0000 7fae41e00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae2406b7d0 0x7fae2406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.383+0000 7fae41e00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae2406b7d0 0x7fae2406dc90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae34036000 tx=0x7fae34030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.386+0000 7fae43e00640 1 -- 172.21.15.141:0/1459204491 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae3c054b00 con 0x7fae4c0717d0 2024-01-16T15:02:03.395 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:03.401 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:03.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.606+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fae4c0720e0 con 0x7fae2406b7d0 2024-01-16T15:02:03.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.609+0000 7fae43e00640 1 -- 172.21.15.141:0/1459204491 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fae34031a10 con 0x7fae2406b7d0 2024-01-16T15:02:03.589 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:02:03.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:03.590 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:02:03.590 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:03.590 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:02:03.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae2406b7d0 msgr2=0x7fae2406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae2406b7d0 0x7fae2406dc90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae34036000 tx=0x7fae34030000 comp rx=0 tx=0).stop 2024-01-16T15:02:03.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 msgr2=0x7fae4c1395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c1395e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00c7f0 tx=0x7fae3c00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 shutdown_connections 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae2406b7d0 0x7fae2406dc90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae4c143c40 0x7fae4c1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae4c0cca40 0x7fae4c1f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 --2- 172.21.15.141:0/1459204491 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae4c0717d0 0x7fae4c1395e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 >> 172.21.15.141:0/1459204491 conn(0x7fae4c06a800 msgr2=0x7fae4c0b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 shutdown_connections 2024-01-16T15:02:03.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.612+0000 7fae4be00640 1 -- 172.21.15.141:0/1459204491 wait complete. 2024-01-16T15:02:03.610 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:02:03.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.840+0000 7fdbab000640 1 Processor -- start 2024-01-16T15:02:03.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.841+0000 7fdbab000640 1 -- start start 2024-01-16T15:02:03.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdbab000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdba415d1d0 0x7fdba40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdbab000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b11d0 0x7fdba40d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdbab000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40d0090 con 0x7fdba40b1f70 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdbab000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba40d02e0 con 0x7fdba415d1d0 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdbab000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40d05c0 con 0x7fdba40b11d0 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdba9c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdba415d1d0 0x7fdba40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdba9c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdba415d1d0 0x7fdba40d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57968/0 (socket says 172.21.15.141:57968) 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdba9c00640 1 -- 172.21.15.141:0/3694076457 learned_addr learned my addr 172.21.15.141:0/3694076457 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdba9200640 1 --1- 172.21.15.141:0/3694076457 >> v1:172.21.15.141:6789/0 conn(0x7fdba40b1f70 0x7fdba40cdb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41620/0 (socket says 172.21.15.141:41620) 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdbaa600640 1 --2- 172.21.15.141:0/3694076457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b11d0 0x7fdba40d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdba9c00640 1 -- 172.21.15.141:0/3694076457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b11d0 msgr2=0x7fdba40d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.842+0000 7fdba9c00640 1 --2- 172.21.15.141:0/3694076457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b11d0 0x7fdba40d1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.843+0000 7fdba9c00640 1 -- 172.21.15.141:0/3694076457 >> v1:172.21.15.141:6789/0 conn(0x7fdba40b1f70 legacy=0x7fdba40cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.843+0000 7fdba9c00640 1 -- 172.21.15.141:0/3694076457 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40d08a0 con 0x7fdba415d1d0 2024-01-16T15:02:03.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.843+0000 7fdbaa600640 1 --2- 172.21.15.141:0/3694076457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b11d0 0x7fdba40d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:03.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.843+0000 7fdba3e00640 1 -- 172.21.15.141:0/3694076457 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361194071 0 0) 0x7fdba40d0090 con 0x7fdba40b1f70 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.844+0000 7fdba9c00640 1 --2- 172.21.15.141:0/3694076457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba415d1d0 0x7fdba40d1630 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009870 tx=0x7fdb9402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b417cc823c8aab25 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.844+0000 7fdba3e00640 1 -- 172.21.15.141:0/3694076457 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94030840 con 0x7fdba415d1d0 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.844+0000 7fdba3e00640 1 -- 172.21.15.141:0/3694076457 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb940309a0 con 0x7fdba415d1d0 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.844+0000 7fdba3e00640 1 -- 172.21.15.141:0/3694076457 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94030d30 con 0x7fdba415d1d0 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.844+0000 7fdbab000640 1 -- 172.21.15.141:0/3694076457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba415d1d0 msgr2=0x7fdba40d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.844+0000 7fdbab000640 1 --2- 172.21.15.141:0/3694076457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba415d1d0 0x7fdba40d1630 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009870 tx=0x7fdb9402b980 comp rx=0 tx=0).stop 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.845+0000 7fdbab000640 1 -- 172.21.15.141:0/3694076457 shutdown_connections 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.845+0000 7fdbab000640 1 --2- 172.21.15.141:0/3694076457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b11d0 0x7fdba40d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.845+0000 7fdbab000640 1 --2- 172.21.15.141:0/3694076457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba415d1d0 0x7fdba40d1630 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.845+0000 7fdbab000640 1 -- 172.21.15.141:0/3694076457 >> 172.21.15.141:0/3694076457 conn(0x7fdba40d3050 msgr2=0x7fdba40d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:03.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.845+0000 7fdbab000640 1 -- 172.21.15.141:0/3694076457 shutdown_connections 2024-01-16T15:02:03.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.845+0000 7fdbab000640 1 -- 172.21.15.141:0/3694076457 wait complete. 2024-01-16T15:02:03.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.846+0000 7fdbab000640 1 Processor -- start 2024-01-16T15:02:03.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.846+0000 7fdbab000640 1 -- start start 2024-01-16T15:02:03.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.847+0000 7fdbab000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40b11d0 0x7fdba4152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.847+0000 7fdbab000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b1f70 0x7fdba40cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.847+0000 7fdbab000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba415d1d0 0x7fdba40b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.847+0000 7fdbab000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40b2980 con 0x7fdba40b11d0 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.847+0000 7fdbab000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40b2c20 con 0x7fdba415d1d0 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.847+0000 7fdbab000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40b2f00 con 0x7fdba40b1f70 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40b11d0 0x7fdba4152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40b11d0 0x7fdba4152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57982/0 (socket says 172.21.15.141:57982) 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 -- 172.21.15.141:0/2548896048 learned_addr learned my addr 172.21.15.141:0/2548896048 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 -- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b1f70 msgr2=0x7fdba40cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b1f70 0x7fdba40cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 -- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba415d1d0 msgr2=0x7fdba40b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba415d1d0 0x7fdba40b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 -- 172.21.15.141:0/2548896048 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb940094d0 con 0x7fdba40b11d0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.848+0000 7fdba9c00640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40b11d0 0x7fdba4152090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000c00 tx=0x7fdb94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.849+0000 7fdba1600640 1 -- 172.21.15.141:0/2548896048 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94030810 con 0x7fdba40b11d0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.849+0000 7fdba1600640 1 -- 172.21.15.141:0/2548896048 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb94043400 con 0x7fdba40b11d0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.849+0000 7fdba1600640 1 -- 172.21.15.141:0/2548896048 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb940437d0 con 0x7fdba40b11d0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.849+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba40b31e0 con 0x7fdba40b11d0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.849+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba40b39e0 con 0x7fdba40b11d0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.850+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba40b3700 con 0x7fdba40b11d0 2024-01-16T15:02:03.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.851+0000 7fdba1600640 1 -- 172.21.15.141:0/2548896048 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fdb94043930 con 0x7fdba40b11d0 2024-01-16T15:02:03.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.852+0000 7fdba1600640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06b7d0 0x7fdb7c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:03.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.852+0000 7fdba1600640 1 -- 172.21.15.141:0/2548896048 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdb9403e070 con 0x7fdba40b11d0 2024-01-16T15:02:03.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.853+0000 7fdba9200640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06b7d0 0x7fdb7c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:03.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.854+0000 7fdba9200640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06b7d0 0x7fdb7c06dc90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c001560 tx=0x7fdb8c007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:03.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:03.857+0000 7fdba1600640 1 -- 172.21.15.141:0/2548896048 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdba40b3700 con 0x7fdba40b11d0 2024-01-16T15:02:04.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.074+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdba40b2d90 con 0x7fdb7c06b7d0 2024-01-16T15:02:04.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.090+0000 7fdba1600640 1 -- 172.21.15.141:0/2548896048 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+234 (secure 0 0 0) 0x7fdba40b2d90 con 0x7fdb7c06b7d0 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone29514" 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "sv1" 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone29512" 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone29511" 2024-01-16T15:02:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone29513" 2024-01-16T15:02:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:04.072 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:02:04.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06b7d0 msgr2=0x7fdb7c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06b7d0 0x7fdb7c06dc90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c001560 tx=0x7fdb8c007ca0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40b11d0 msgr2=0x7fdba4152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40b11d0 0x7fdba4152090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000c00 tx=0x7fdb94004730 comp rx=0 tx=0).stop 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 shutdown_connections 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06b7d0 0x7fdb7c06dc90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba415d1d0 0x7fdba40b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40b1f70 0x7fdba40cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 --2- 172.21.15.141:0/2548896048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40b11d0 0x7fdba4152090 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.093+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 >> 172.21.15.141:0/2548896048 conn(0x7fdba40d3050 msgr2=0x7fdba415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.094+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 shutdown_connections 2024-01-16T15:02:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.094+0000 7fdbab000640 1 -- 172.21.15.141:0/2548896048 wait complete. 2024-01-16T15:02:04.091 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone29514 2024-01-16T15:02:04.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.328+0000 7f8345400640 1 Processor -- start 2024-01-16T15:02:04.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.329+0000 7f8345400640 1 -- start start 2024-01-16T15:02:04.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.330+0000 7f8345400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f83400cf9a0 0x7f834015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.330+0000 7f8345400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400b1ee0 con 0x7f8340143ae0 2024-01-16T15:02:04.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.330+0000 7f8345400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400b2780 con 0x7f834015e1e0 2024-01-16T15:02:04.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.330+0000 7f8345400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834015e990 con 0x7f83400cf9a0 2024-01-16T15:02:04.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.331+0000 7f833e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8340143ae0 0x7f83400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41626/0 (socket says 172.21.15.141:41626) 2024-01-16T15:02:04.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.331+0000 7f833e800640 1 -- 172.21.15.141:0/3281836809 learned_addr learned my addr 172.21.15.141:0/3281836809 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:04.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.331+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324250016 0 0) 0x7f83400b1ee0 con 0x7f8340143ae0 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.331+0000 7f8335c00640 1 --2- 172.21.15.141:0/3281836809 >> v2:172.21.15.188:3300/0 conn(0x7f83400cf9a0 0x7f834015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.331+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320003520 con 0x7f8340143ae0 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83801059 0 0) 0x7f8320003520 con 0x7f8340143ae0 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 >> v2:172.21.15.188:3300/0 conn(0x7f83400cf9a0 msgr2=0x7f834015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 --2- 172.21.15.141:0/3281836809 >> v2:172.21.15.188:3300/0 conn(0x7f83400cf9a0 0x7f834015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 >> v1:172.21.15.188:6789/0 conn(0x7f834015e1e0 legacy=0x7f83400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400b2130 con 0x7f8340143ae0 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8330001670 con 0x7f8340143ae0 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 >> v1:172.21.15.141:6789/0 conn(0x7f8340143ae0 legacy=0x7f83400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8320003970 0x7f8320003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83200047c0 0x7f8320004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8320005180 0x7f832000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834015e990 con 0x7f8320005180 2024-01-16T15:02:04.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832000e2e0 con 0x7f83200047c0 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832000e4a0 con 0x7f8320003970 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833f200640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8320005180 0x7f832000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f8335c00640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83200047c0 0x7f8320004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.332+0000 7f833e800640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8320003970 0x7f8320003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.333+0000 7f833f200640 1 -- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8320003970 msgr2=0x7f8320003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.333+0000 7f833f200640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8320003970 0x7f8320003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.333+0000 7f833f200640 1 -- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83200047c0 msgr2=0x7f8320004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.333+0000 7f833f200640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83200047c0 0x7f8320004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.333+0000 7f833f200640 1 -- 172.21.15.141:0/3281836809 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8320003520 con 0x7f8320005180 2024-01-16T15:02:04.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.333+0000 7f833e800640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8320003970 0x7f8320003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:04.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.333+0000 7f833f200640 1 -- 172.21.15.141:0/3281836809 reap_dead start 2024-01-16T15:02:04.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f833f200640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8320005180 0x7f832000db90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8338001e50 tx=0x7f83200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556bbf6654182559 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338003b70 con 0x7f8320005180 2024-01-16T15:02:04.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8338003cf0 con 0x7f8320005180 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f833de00640 1 -- 172.21.15.141:0/3281836809 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338005310 con 0x7f8320005180 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f8345400640 1 -- 172.21.15.141:0/3281836809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8320005180 msgr2=0x7f832000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f8345400640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8320005180 0x7f832000db90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8338001e50 tx=0x7f83200057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f8345400640 1 -- 172.21.15.141:0/3281836809 shutdown_connections 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f8345400640 1 --2- 172.21.15.141:0/3281836809 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8320005180 0x7f832000db90 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.334+0000 7f8345400640 1 -- 172.21.15.141:0/3281836809 >> 172.21.15.141:0/3281836809 conn(0x7f834006a800 msgr2=0x7f834006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.335+0000 7f8345400640 1 -- 172.21.15.141:0/3281836809 shutdown_connections 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.335+0000 7f8345400640 1 -- 172.21.15.141:0/3281836809 wait complete. 2024-01-16T15:02:04.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.336+0000 7f8345400640 1 Processor -- start 2024-01-16T15:02:04.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.336+0000 7f8345400640 1 -- start start 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.337+0000 7f8345400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8320005180 0x7f8340159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.337+0000 7f8345400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83401ee5c0 0x7f83400ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.337+0000 7f8345400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f83401eea20 0x7f83400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.337+0000 7f8345400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834015a4e0 con 0x7f83401eea20 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.337+0000 7f8345400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83400ce730 con 0x7f8320005180 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.337+0000 7f8345400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834015a730 con 0x7f83401ee5c0 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f83401eea20 0x7f83400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f83401eea20 0x7f83400702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58000/0 (socket says 172.21.15.141:58000) 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 -- 172.21.15.141:0/786579386 learned_addr learned my addr 172.21.15.141:0/786579386 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f8335c00640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83401ee5c0 0x7f83400ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833e800640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8320005180 0x7f8340159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 -- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83401ee5c0 msgr2=0x7f83400ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83401ee5c0 0x7f83400ae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 -- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8320005180 msgr2=0x7f8340159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8320005180 0x7f8340159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 -- 172.21.15.141:0/786579386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834015aa10 con 0x7f83401eea20 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.338+0000 7f833f200640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f83401eea20 0x7f83400702c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f833802c060 tx=0x7f83380014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.339+0000 7f8337400640 1 -- 172.21.15.141:0/786579386 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338006d50 con 0x7f83401eea20 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.339+0000 7f8337400640 1 -- 172.21.15.141:0/786579386 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8338003b80 con 0x7f83401eea20 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.339+0000 7f8337400640 1 -- 172.21.15.141:0/786579386 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83380396a0 con 0x7f83401eea20 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.339+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834015acf0 con 0x7f83401eea20 2024-01-16T15:02:04.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.339+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834015b2b0 con 0x7f83401eea20 2024-01-16T15:02:04.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.340+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83400675d0 con 0x7f83401eea20 2024-01-16T15:02:04.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.341+0000 7f8337400640 1 -- 172.21.15.141:0/786579386 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f8338039800 con 0x7f83401eea20 2024-01-16T15:02:04.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.342+0000 7f8337400640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f831806b800 0x7f831806dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.342+0000 7f8337400640 1 -- 172.21.15.141:0/786579386 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f83380ac450 con 0x7f83401eea20 2024-01-16T15:02:04.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.343+0000 7f833e800640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f831806b800 0x7f831806dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.344+0000 7f833e800640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f831806b800 0x7f831806dcc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f83300018e0 tx=0x7f8330009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.347+0000 7f8337400640 1 -- 172.21.15.141:0/786579386 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8338076a50 con 0x7f83401eea20 2024-01-16T15:02:04.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.568+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone29514", "target": ["mon-mgr", ""]}) v1 -- 0x7f8340073c60 con 0x7f831806b800 2024-01-16T15:02:04.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.604+0000 7f8337400640 1 -- 172.21.15.141:0/786579386 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8340073c60 con 0x7f831806b800 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f831806b800 msgr2=0x7f831806dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f831806b800 0x7f831806dcc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f83300018e0 tx=0x7f8330009040 comp rx=0 tx=0).stop 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f83401eea20 msgr2=0x7f83400702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f83401eea20 0x7f83400702c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f833802c060 tx=0x7f83380014a0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 shutdown_connections 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f831806b800 0x7f831806dcc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f83401eea20 0x7f83400702c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83401ee5c0 0x7f83400ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 --2- 172.21.15.141:0/786579386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8320005180 0x7f8340159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 >> 172.21.15.141:0/786579386 conn(0x7f834006a800 msgr2=0x7f83400aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 shutdown_connections 2024-01-16T15:02:04.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.607+0000 7f8345400640 1 -- 172.21.15.141:0/786579386 wait complete. 2024-01-16T15:02:04.594 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.615+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.594 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.615+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.594 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.615+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.594 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.615+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.594 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.615+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.608 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:02:04.615 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.636+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.615 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.636+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.616 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.636+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.616 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.636+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.616 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:04.636+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:04.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.832+0000 7f6394e00640 1 Processor -- start 2024-01-16T15:02:04.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.833+0000 7f6394e00640 1 -- start start 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.834+0000 7f6394e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63900b3ae0 0x7f639015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.834+0000 7f6394e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f63900cf290 0x7f63900cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.834+0000 7f6394e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6390070c40 con 0x7f63900ae5a0 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.834+0000 7f6394e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6390071450 con 0x7f63900cf290 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.834+0000 7f6394e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63900cfc60 con 0x7f63900b3ae0 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.834+0000 7f638de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f63900ae5a0 0x7f639006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41636/0 (socket says 172.21.15.141:41636) 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.834+0000 7f638de00640 1 -- 172.21.15.141:0/1961776036 learned_addr learned my addr 172.21.15.141:0/1961776036 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f6385200640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63900b3ae0 0x7f639015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466244516 0 0) 0x7f6390070c40 con 0x7f63900ae5a0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638e800640 1 --2- 172.21.15.141:0/1961776036 >> v2:172.21.15.188:3300/0 conn(0x7f63900cf290 0x7f63900cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6370003520 con 0x7f63900ae5a0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3070265500 0 0) 0x7f6370003520 con 0x7f63900ae5a0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63900b3ae0 msgr2=0x7f639015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63900b3ae0 0x7f639015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 >> v2:172.21.15.188:3300/0 conn(0x7f63900cf290 msgr2=0x7f63900cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 --2- 172.21.15.141:0/1961776036 >> v2:172.21.15.188:3300/0 conn(0x7f63900cf290 0x7f63900cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639006c470 con 0x7f63900ae5a0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6380001670 con 0x7f63900ae5a0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 >> v1:172.21.15.141:6789/0 conn(0x7f63900ae5a0 legacy=0x7f639006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.835+0000 7f638d400640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6370003970 0x7f6370003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638d400640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63700047c0 0x7f6370004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638d400640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63900b3ae0 0x7f6370005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390071450 con 0x7f6370003970 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900cfc60 con 0x7f63700047c0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637000ba30 con 0x7f63900b3ae0 2024-01-16T15:02:04.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638de00640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6370003970 0x7f6370003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638de00640 1 -- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63900b3ae0 msgr2=0x7f6370005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638de00640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63900b3ae0 0x7f6370005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638de00640 1 -- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63700047c0 msgr2=0x7f6370004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638de00640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63700047c0 0x7f6370004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.836+0000 7f638de00640 1 -- 172.21.15.141:0/1961776036 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6370003520 con 0x7f6370003970 2024-01-16T15:02:04.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.837+0000 7f638de00640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6370003970 0x7f6370003d90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f638000c240 tx=0x7f63700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9662f8f81e07946 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380005ad0 con 0x7f6370003970 2024-01-16T15:02:04.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6380005c50 con 0x7f6370003970 2024-01-16T15:02:04.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f638d400640 1 -- 172.21.15.141:0/1961776036 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638002c8d0 con 0x7f6370003970 2024-01-16T15:02:04.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f6394e00640 1 -- 172.21.15.141:0/1961776036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6370003970 msgr2=0x7f6370003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f6394e00640 1 --2- 172.21.15.141:0/1961776036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6370003970 0x7f6370003d90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f638000c240 tx=0x7f63700057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f638e800640 1 -- 172.21.15.141:0/1961776036 reap_dead start 2024-01-16T15:02:04.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f6394e00640 1 -- 172.21.15.141:0/1961776036 shutdown_connections 2024-01-16T15:02:04.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f6394e00640 1 -- 172.21.15.141:0/1961776036 >> 172.21.15.141:0/1961776036 conn(0x7f639006a800 msgr2=0x7f639006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f6394e00640 1 -- 172.21.15.141:0/1961776036 shutdown_connections 2024-01-16T15:02:04.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.838+0000 7f6394e00640 1 -- 172.21.15.141:0/1961776036 wait complete. 2024-01-16T15:02:04.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.840+0000 7f6394e00640 1 Processor -- start 2024-01-16T15:02:04.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.840+0000 7f6394e00640 1 -- start start 2024-01-16T15:02:04.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f6394e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6370003970 0x7f6390144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f6394e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63901f2e80 0x7f63901529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f6394e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6390152f30 0x7f639015c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f6394e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390144d60 con 0x7f6390152f30 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f6394e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390155f70 con 0x7f63901f2e80 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f6394e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6390155d20 con 0x7f6370003970 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f638e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6390152f30 0x7f639015c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f638e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6390152f30 0x7f639015c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58018/0 (socket says 172.21.15.141:58018) 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.841+0000 7f638e800640 1 -- 172.21.15.141:0/2529307526 learned_addr learned my addr 172.21.15.141:0/2529307526 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f638de00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6370003970 0x7f6390144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f6385200640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63901f2e80 0x7f63901529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f638e800640 1 -- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6370003970 msgr2=0x7f6390144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f638e800640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6370003970 0x7f6390144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f638e800640 1 -- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63901f2e80 msgr2=0x7f63901529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f638e800640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63901f2e80 0x7f63901529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f638e800640 1 -- 172.21.15.141:0/2529307526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6390153560 con 0x7f6390152f30 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f6385200640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63901f2e80 0x7f63901529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f638e800640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6390152f30 0x7f639015c750 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f63900cf670 tx=0x7f638800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.842+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638800ed90 con 0x7f6390152f30 2024-01-16T15:02:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.843+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6388014420 con 0x7f6390152f30 2024-01-16T15:02:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.843+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6390156210 con 0x7f6390152f30 2024-01-16T15:02:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.843+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63880147d0 con 0x7f6390152f30 2024-01-16T15:02:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.843+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63901567d0 con 0x7f6390152f30 2024-01-16T15:02:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.844+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63901569f0 con 0x7f6390152f30 2024-01-16T15:02:04.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.845+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f6388014930 con 0x7f6390152f30 2024-01-16T15:02:04.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.845+0000 7f6386a00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f636806b7d0 0x7f636806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.846+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f638808a6e0 con 0x7f6390152f30 2024-01-16T15:02:04.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.846+0000 7f638de00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f636806b7d0 0x7f636806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.847+0000 7f638de00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f636806b7d0 0x7f636806dc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6380034000 tx=0x7f6380007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:04.850+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6388054ec0 con 0x7f6390152f30 2024-01-16T15:02:05.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:05.071+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6390071170 con 0x7f636806b7d0 2024-01-16T15:02:07.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:07.543+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f6388012dc0 con 0x7f6390152f30 2024-01-16T15:02:08.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.138+0000 7f6386a00640 1 -- 172.21.15.141:0/2529307526 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f6380005dc0 con 0x7f636806b7d0 2024-01-16T15:02:08.118 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:08.118 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:02:08.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:08.118 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1" 2024-01-16T15:02:08.118 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:08.118 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f636806b7d0 msgr2=0x7f636806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f636806b7d0 0x7f636806dc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6380034000 tx=0x7f6380007000 comp rx=0 tx=0).stop 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6390152f30 msgr2=0x7f639015c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6390152f30 0x7f639015c750 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f63900cf670 tx=0x7f638800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 shutdown_connections 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f636806b7d0 0x7f636806dc90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6390152f30 0x7f639015c750 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63901f2e80 0x7f63901529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 --2- 172.21.15.141:0/2529307526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6370003970 0x7f6390144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.141+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 >> 172.21.15.141:0/2529307526 conn(0x7f639006a800 msgr2=0x7f63900b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.142+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 shutdown_connections 2024-01-16T15:02:08.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.142+0000 7f6394e00640 1 -- 172.21.15.141:0/2529307526 wait complete. 2024-01-16T15:02:08.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:02:08.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.374+0000 7efd96e00640 1 Processor -- start 2024-01-16T15:02:08.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.375+0000 7efd96e00640 1 -- start start 2024-01-16T15:02:08.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.376+0000 7efd96e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efd90160920 0x7efd900b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.376+0000 7efd96e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd900d0a10 con 0x7efd900cf9f0 2024-01-16T15:02:08.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.376+0000 7efd96e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd90161e30 con 0x7efd900cfea0 2024-01-16T15:02:08.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.376+0000 7efd96e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd900d1f50 con 0x7efd90160920 2024-01-16T15:02:08.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.377+0000 7efd95a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efd90160920 0x7efd900b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.377+0000 7efd95000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7efd900cf9f0 0x7efd90161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41644/0 (socket says 172.21.15.141:41644) 2024-01-16T15:02:08.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.377+0000 7efd95000640 1 -- 172.21.15.141:0/2102795534 learned_addr learned my addr 172.21.15.141:0/2102795534 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:08.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.377+0000 7efd95a00640 1 -- 172.21.15.141:0/2102795534 >> v1:172.21.15.188:6789/0 conn(0x7efd900cfea0 legacy=0x7efd90161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.377+0000 7efd95a00640 1 -- 172.21.15.141:0/2102795534 >> v1:172.21.15.141:6789/0 conn(0x7efd900cf9f0 legacy=0x7efd90161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.377+0000 7efd95a00640 1 -- 172.21.15.141:0/2102795534 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd900cf080 con 0x7efd90160920 2024-01-16T15:02:08.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.377+0000 7efd8fe00640 1 -- 172.21.15.141:0/2102795534 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715551106 0 0) 0x7efd900d0a10 con 0x7efd900cf9f0 2024-01-16T15:02:08.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.378+0000 7efd95a00640 1 --2- 172.21.15.141:0/2102795534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd900b3f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7efd780097a0 tx=0x7efd7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc566825284819b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd8fe00640 1 -- 172.21.15.141:0/2102795534 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78031830 con 0x7efd90160920 2024-01-16T15:02:08.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd8fe00640 1 -- 172.21.15.141:0/2102795534 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78031990 con 0x7efd90160920 2024-01-16T15:02:08.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd8fe00640 1 -- 172.21.15.141:0/2102795534 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78031d20 con 0x7efd90160920 2024-01-16T15:02:08.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd96e00640 1 -- 172.21.15.141:0/2102795534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 msgr2=0x7efd900b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd96e00640 1 --2- 172.21.15.141:0/2102795534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd900b3f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7efd780097a0 tx=0x7efd7802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd96e00640 1 -- 172.21.15.141:0/2102795534 shutdown_connections 2024-01-16T15:02:08.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd96e00640 1 --2- 172.21.15.141:0/2102795534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd900b3f30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd96e00640 1 -- 172.21.15.141:0/2102795534 >> 172.21.15.141:0/2102795534 conn(0x7efd900d3090 msgr2=0x7efd900d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:08.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd96e00640 1 -- 172.21.15.141:0/2102795534 shutdown_connections 2024-01-16T15:02:08.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.379+0000 7efd96e00640 1 -- 172.21.15.141:0/2102795534 wait complete. 2024-01-16T15:02:08.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.381+0000 7efd96e00640 1 Processor -- start 2024-01-16T15:02:08.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.381+0000 7efd96e00640 1 -- start start 2024-01-16T15:02:08.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd900cf9f0 0x7efd900cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd900cfea0 0x7efd901516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd901538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd900cc0b0 con 0x7efd90160920 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd90152b00 con 0x7efd900cfea0 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd90152de0 con 0x7efd900cf9f0 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd901538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd901538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58050/0 (socket says 172.21.15.141:58050) 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.382+0000 7efd96400640 1 -- 172.21.15.141:0/3102501878 learned_addr learned my addr 172.21.15.141:0/3102501878 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd95a00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd900cf9f0 0x7efd900cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd95000640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd900cfea0 0x7efd901516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd96400640 1 -- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd900cf9f0 msgr2=0x7efd900cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd96400640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd900cf9f0 0x7efd900cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd96400640 1 -- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd900cfea0 msgr2=0x7efd901516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd96400640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd900cfea0 0x7efd901516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd96400640 1 -- 172.21.15.141:0/3102501878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd78009400 con 0x7efd90160920 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd95000640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd900cfea0 0x7efd901516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd96400640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd901538c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7efd8800c410 tx=0x7efd8800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd8d600640 1 -- 172.21.15.141:0/3102501878 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88011790 con 0x7efd90160920 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.383+0000 7efd8d600640 1 -- 172.21.15.141:0/3102501878 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd880118f0 con 0x7efd90160920 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.384+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd901530c0 con 0x7efd90160920 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.384+0000 7efd8d600640 1 -- 172.21.15.141:0/3102501878 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88011ce0 con 0x7efd90160920 2024-01-16T15:02:08.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.384+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd900cce10 con 0x7efd90160920 2024-01-16T15:02:08.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.385+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd900ccfd0 con 0x7efd90160920 2024-01-16T15:02:08.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.386+0000 7efd8d600640 1 -- 172.21.15.141:0/3102501878 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7efd8801c460 con 0x7efd90160920 2024-01-16T15:02:08.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.387+0000 7efd8d600640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd6806b8f0 0x7efd6806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.387+0000 7efd8d600640 1 -- 172.21.15.141:0/3102501878 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efd8808a470 con 0x7efd90160920 2024-01-16T15:02:08.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.387+0000 7efd95a00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd6806b8f0 0x7efd6806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.389+0000 7efd95a00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd6806b8f0 0x7efd6806ddb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efd78036000 tx=0x7efd7803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.392+0000 7efd8d600640 1 -- 172.21.15.141:0/3102501878 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd88054960 con 0x7efd90160920 2024-01-16T15:02:08.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.621+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd900ce7e0 con 0x7efd6806b8f0 2024-01-16T15:02:08.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.689+0000 7efd8d600640 1 -- 172.21.15.141:0/3102501878 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efd7803a420 con 0x7efd6806b8f0 2024-01-16T15:02:08.670 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:08.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.692+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd6806b8f0 msgr2=0x7efd6806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.692+0000 7efd96e00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd6806b8f0 0x7efd6806ddb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efd78036000 tx=0x7efd7803c040 comp rx=0 tx=0).stop 2024-01-16T15:02:08.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.692+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 msgr2=0x7efd901538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.692+0000 7efd96e00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd901538c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7efd8800c410 tx=0x7efd8800c8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 shutdown_connections 2024-01-16T15:02:08.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd6806b8f0 0x7efd6806ddb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd90160920 0x7efd901538c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd900cfea0 0x7efd901516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 --2- 172.21.15.141:0/3102501878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd900cf9f0 0x7efd900cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 >> 172.21.15.141:0/3102501878 conn(0x7efd900d3090 msgr2=0x7efd900cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:08.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 shutdown_connections 2024-01-16T15:02:08.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.693+0000 7efd96e00640 1 -- 172.21.15.141:0/3102501878 wait complete. 2024-01-16T15:02:08.693 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:02:08.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.923+0000 7f279fe00640 1 Processor -- start 2024-01-16T15:02:08.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.923+0000 7f279fe00640 1 -- start start 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 0x7f27a00ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279fe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f27a015d770 0x7f27a015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a015e140 con 0x7f27a0143810 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279fe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a0143bf0 con 0x7f27a015d770 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00722c0 con 0x7f27a0071e40 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f27a0143810 0x7f27a015e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41654/0 (socket says 172.21.15.141:41654) 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279ea00640 1 -- 172.21.15.141:0/1074890634 learned_addr learned my addr 172.21.15.141:0/1074890634 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f2795e00640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 0x7f27a00ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368911883 0 0) 0x7f27a015e140 con 0x7f27a0143810 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279f400640 1 --2- 172.21.15.141:0/1074890634 >> v2:172.21.15.188:3300/0 conn(0x7f27a015d770 0x7f27a015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.925+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2780003520 con 0x7f27a0143810 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3303154583 0 0) 0x7f2780003520 con 0x7f27a0143810 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 msgr2=0x7f27a00ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 0x7f27a00ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 >> v2:172.21.15.188:3300/0 conn(0x7f27a015d770 msgr2=0x7f27a015dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 --2- 172.21.15.141:0/1074890634 >> v2:172.21.15.188:3300/0 conn(0x7f27a015d770 0x7f27a015dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a006c730 con 0x7f27a0143810 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2790001670 con 0x7f27a0143810 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 >> v1:172.21.15.141:6789/0 conn(0x7f27a0143810 legacy=0x7f27a015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2780003970 0x7f2780003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f2780004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 0x7f2780005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0143bf0 con 0x7f27800047c0 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00722c0 con 0x7f2780003970 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278000ba60 con 0x7f27a0071e40 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.926+0000 7f2795e00640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f2780004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.927+0000 7f279f400640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 0x7f2780005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.927+0000 7f2795e00640 1 -- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 msgr2=0x7f2780005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.927+0000 7f2795e00640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 0x7f2780005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.927+0000 7f2795e00640 1 -- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2780003970 msgr2=0x7f2780003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:08.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.927+0000 7f2795e00640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2780003970 0x7f2780003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.927+0000 7f2795e00640 1 -- 172.21.15.141:0/1074890634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780003520 con 0x7f27800047c0 2024-01-16T15:02:08.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.927+0000 7f279f400640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a0071e40 0x7f2780005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:08.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.928+0000 7f2795e00640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f2780004c40 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2788009a90 tx=0x7f27800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da18b0f32db33117 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.928+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278802c660 con 0x7f27800047c0 2024-01-16T15:02:08.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.928+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278802c7c0 con 0x7f27800047c0 2024-01-16T15:02:08.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.928+0000 7f279e000640 1 -- 172.21.15.141:0/1074890634 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278802cb50 con 0x7f27800047c0 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.928+0000 7f279fe00640 1 -- 172.21.15.141:0/1074890634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 msgr2=0x7f2780004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.928+0000 7f279fe00640 1 --2- 172.21.15.141:0/1074890634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f2780004c40 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2788009a90 tx=0x7f27800057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.928+0000 7f279f400640 1 -- 172.21.15.141:0/1074890634 reap_dead start 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.929+0000 7f279fe00640 1 -- 172.21.15.141:0/1074890634 shutdown_connections 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.929+0000 7f279fe00640 1 -- 172.21.15.141:0/1074890634 >> 172.21.15.141:0/1074890634 conn(0x7f27a006a7d0 msgr2=0x7f27a006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.929+0000 7f279fe00640 1 -- 172.21.15.141:0/1074890634 shutdown_connections 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.929+0000 7f279fe00640 1 -- 172.21.15.141:0/1074890634 wait complete. 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.930+0000 7f279fe00640 1 Processor -- start 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.931+0000 7f279fe00640 1 -- start start 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f27a0144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27a0141fc0 0x7f27a01f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a01f2190 0x7f27a01f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0144860 con 0x7f27800047c0 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0141590 con 0x7f27a0141fc0 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a013f0a0 con 0x7f27a01f2190 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f27a0144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f27a0144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58072/0 (socket says 172.21.15.141:58072) 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 -- 172.21.15.141:0/2683321687 learned_addr learned my addr 172.21.15.141:0/2683321687 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f2795e00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27a0141fc0 0x7f27a01f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279f400640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a01f2190 0x7f27a01f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 -- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a01f2190 msgr2=0x7f27a01f6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a01f2190 0x7f27a01f6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 -- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27a0141fc0 msgr2=0x7f27a01f1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27a0141fc0 0x7f27a01f1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f279ea00640 1 -- 172.21.15.141:0/2683321687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a013f380 con 0x7f27800047c0 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.932+0000 7f2795e00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27a0141fc0 0x7f27a01f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.933+0000 7f279ea00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f27a0144320 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f279000c700 tx=0x7f279000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.933+0000 7f2797400640 1 -- 172.21.15.141:0/2683321687 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27900072d0 con 0x7f27800047c0 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.933+0000 7f2797400640 1 -- 172.21.15.141:0/2683321687 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2790007430 con 0x7f27800047c0 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.933+0000 7f2797400640 1 -- 172.21.15.141:0/2683321687 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27900047e0 con 0x7f27800047c0 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.933+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a01417c0 con 0x7f27800047c0 2024-01-16T15:02:08.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.933+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a013f660 con 0x7f27800047c0 2024-01-16T15:02:08.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.934+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a0141ce0 con 0x7f27800047c0 2024-01-16T15:02:08.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.935+0000 7f2797400640 1 -- 172.21.15.141:0/2683321687 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f2790004940 con 0x7f27800047c0 2024-01-16T15:02:08.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.936+0000 7f2797400640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277806b920 0x7f277806dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:08.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.936+0000 7f2797400640 1 -- 172.21.15.141:0/2683321687 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2790089970 con 0x7f27800047c0 2024-01-16T15:02:08.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.936+0000 7f2795e00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277806b920 0x7f277806dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:08.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.938+0000 7f2795e00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277806b920 0x7f277806dde0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2788036000 tx=0x7f2788030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:08.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:08.940+0000 7f2797400640 1 -- 172.21.15.141:0/2683321687 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27a0141ce0 con 0x7f27800047c0 2024-01-16T15:02:09.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.139+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f27a0140c10 con 0x7f277806b920 2024-01-16T15:02:09.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.177+0000 7f2797400640 1 -- 172.21.15.141:0/2683321687 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f27a0140c10 con 0x7f277806b920 2024-01-16T15:02:09.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277806b920 msgr2=0x7f277806dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277806b920 0x7f277806dde0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2788036000 tx=0x7f2788030000 comp rx=0 tx=0).stop 2024-01-16T15:02:09.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 msgr2=0x7f27a0144320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f27a0144320 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f279000c700 tx=0x7f279000cbc0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 shutdown_connections 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277806b920 0x7f277806dde0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f27a01f2190 0x7f27a01f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27a0141fc0 0x7f27a01f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 --2- 172.21.15.141:0/2683321687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27800047c0 0x7f27a0144320 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 >> 172.21.15.141:0/2683321687 conn(0x7f27a006a7d0 msgr2=0x7f27a006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 shutdown_connections 2024-01-16T15:02:09.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.179+0000 7f279fe00640 1 -- 172.21.15.141:0/2683321687 wait complete. 2024-01-16T15:02:09.179 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone29512 2024-01-16T15:02:09.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.390+0000 7f97b9600640 1 Processor -- start 2024-01-16T15:02:09.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.391+0000 7f97b9600640 1 -- start start 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.392+0000 7f97b9600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f97b40d0100 0x7f97b40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.392+0000 7f97b9600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0f40 0x7f97b40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.392+0000 7f97b9600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40d0a10 con 0x7f97b4160920 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.392+0000 7f97b9600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b40cf080 con 0x7f97b40d0100 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.392+0000 7f97b9600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40c6be0 con 0x7f97b40d0f40 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97ab200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f97b40d0100 0x7f97b40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97ab200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f97b40d0100 0x7f97b40b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58080/0 (socket says 172.21.15.141:58080) 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97ab200640 1 -- 172.21.15.141:0/3124157068 learned_addr learned my addr 172.21.15.141:0/3124157068 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97b8c00640 1 --2- 172.21.15.141:0/3124157068 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0f40 0x7f97b40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97ab200640 1 -- 172.21.15.141:0/3124157068 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0f40 msgr2=0x7f97b40d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97ab200640 1 --2- 172.21.15.141:0/3124157068 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0f40 0x7f97b40d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97ab200640 1 -- 172.21.15.141:0/3124157068 >> v1:172.21.15.188:6789/0 conn(0x7f97b4160920 legacy=0x7f97b40cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.393+0000 7f97ab200640 1 -- 172.21.15.141:0/3124157068 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b40ce7e0 con 0x7f97b40d0100 2024-01-16T15:02:09.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.394+0000 7f97ab200640 1 --2- 172.21.15.141:0/3124157068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0100 0x7f97b40b3f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f979c009870 tx=0x7f979c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adedbeaa6f4d1c62 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b3400640 1 -- 172.21.15.141:0/3124157068 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c030840 con 0x7f97b40d0100 2024-01-16T15:02:09.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b3400640 1 -- 172.21.15.141:0/3124157068 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c0309a0 con 0x7f97b40d0100 2024-01-16T15:02:09.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b3400640 1 -- 172.21.15.141:0/3124157068 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c030d30 con 0x7f97b40d0100 2024-01-16T15:02:09.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 -- 172.21.15.141:0/3124157068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0100 msgr2=0x7f97b40b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 --2- 172.21.15.141:0/3124157068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0100 0x7f97b40b3f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f979c009870 tx=0x7f979c02b980 comp rx=0 tx=0).stop 2024-01-16T15:02:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 -- 172.21.15.141:0/3124157068 shutdown_connections 2024-01-16T15:02:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 --2- 172.21.15.141:0/3124157068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0100 0x7f97b40b3f30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 --2- 172.21.15.141:0/3124157068 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0f40 0x7f97b40d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 -- 172.21.15.141:0/3124157068 >> 172.21.15.141:0/3124157068 conn(0x7f97b40d3090 msgr2=0x7f97b40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 -- 172.21.15.141:0/3124157068 shutdown_connections 2024-01-16T15:02:09.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.395+0000 7f97b9600640 1 -- 172.21.15.141:0/3124157068 wait complete. 2024-01-16T15:02:09.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.398+0000 7f97b9600640 1 Processor -- start 2024-01-16T15:02:09.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.398+0000 7f97b9600640 1 -- start start 2024-01-16T15:02:09.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b9600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0100 0x7f97b41492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0f40 0x7f97b40b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97b4160920 0x7f97b40b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b9600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4146730 con 0x7f97b40d0f40 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b9600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40b83d0 con 0x7f97b4160920 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b9600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40b86b0 con 0x7f97b40d0100 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97ab200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0f40 0x7f97b40b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97ab200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0f40 0x7f97b40b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58092/0 (socket says 172.21.15.141:58092) 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97ab200640 1 -- 172.21.15.141:0/1379482910 learned_addr learned my addr 172.21.15.141:0/1379482910 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b8c00640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97b4160920 0x7f97b40b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.400+0000 7f97b3e00640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0100 0x7f97b41492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97ab200640 1 -- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0100 msgr2=0x7f97b41492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97ab200640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0100 0x7f97b41492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97ab200640 1 -- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97b4160920 msgr2=0x7f97b40b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97ab200640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97b4160920 0x7f97b40b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97ab200640 1 -- 172.21.15.141:0/1379482910 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c0094d0 con 0x7f97b40d0f40 2024-01-16T15:02:09.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97ab200640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0f40 0x7f97b40b7b60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f979c000c00 tx=0x7f979c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97b0c00640 1 -- 172.21.15.141:0/1379482910 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c039c90 con 0x7f97b40d0f40 2024-01-16T15:02:09.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97b0c00640 1 -- 172.21.15.141:0/1379482910 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c039df0 con 0x7f97b40d0f40 2024-01-16T15:02:09.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97b0c00640 1 -- 172.21.15.141:0/1379482910 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c043640 con 0x7f97b40d0f40 2024-01-16T15:02:09.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b40b8990 con 0x7f97b40d0f40 2024-01-16T15:02:09.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.401+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b40b80a0 con 0x7f97b40d0f40 2024-01-16T15:02:09.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.402+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b41468a0 con 0x7f97b40d0f40 2024-01-16T15:02:09.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.403+0000 7f97b0c00640 1 -- 172.21.15.141:0/1379482910 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f979c0437a0 con 0x7f97b40d0f40 2024-01-16T15:02:09.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.404+0000 7f97b0c00640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f978c06b8f0 0x7f978c06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.404+0000 7f97b0c00640 1 -- 172.21.15.141:0/1379482910 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f979c03e070 con 0x7f97b40d0f40 2024-01-16T15:02:09.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.405+0000 7f97b3e00640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f978c06b8f0 0x7f978c06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.406+0000 7f97b3e00640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f978c06b8f0 0x7f978c06ddb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f97a4000f60 tx=0x7f97a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.409+0000 7f97b0c00640 1 -- 172.21.15.141:0/1379482910 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97b41468a0 con 0x7f97b40d0f40 2024-01-16T15:02:09.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.620+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone29512", "target": ["mon-mgr", ""]}) v1 -- 0x7f97b40b8eb0 con 0x7f978c06b8f0 2024-01-16T15:02:09.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.656+0000 7f97b0c00640 1 -- 172.21.15.141:0/1379482910 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f97b40b8eb0 con 0x7f978c06b8f0 2024-01-16T15:02:09.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f978c06b8f0 msgr2=0x7f978c06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f978c06b8f0 0x7f978c06ddb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f97a4000f60 tx=0x7f97a4009040 comp rx=0 tx=0).stop 2024-01-16T15:02:09.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0f40 msgr2=0x7f97b40b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0f40 0x7f97b40b7b60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f979c000c00 tx=0x7f979c004730 comp rx=0 tx=0).stop 2024-01-16T15:02:09.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 shutdown_connections 2024-01-16T15:02:09.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f978c06b8f0 0x7f978c06ddb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97b4160920 0x7f97b40b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97b40d0f40 0x7f97b40b7b60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 --2- 172.21.15.141:0/1379482910 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97b40d0100 0x7f97b41492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 >> 172.21.15.141:0/1379482910 conn(0x7f97b40d3090 msgr2=0x7f97b40cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 shutdown_connections 2024-01-16T15:02:09.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.659+0000 7f97b9600640 1 -- 172.21.15.141:0/1379482910 wait complete. 2024-01-16T15:02:09.657 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone29511 2024-01-16T15:02:09.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.858+0000 7f0d0f400640 1 Processor -- start 2024-01-16T15:02:09.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.858+0000 7f0d0f400640 1 -- start start 2024-01-16T15:02:09.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.860+0000 7f0d0f400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0d08159590 0x7f0d08070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.860+0000 7f0d0f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 0x7f0d080ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.860+0000 7f0d0f400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d08071790 con 0x7f0d080cd220 2024-01-16T15:02:09.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.860+0000 7f0d0f400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d08071100 con 0x7f0d08159590 2024-01-16T15:02:09.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.860+0000 7f0d0f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08070860 con 0x7f0d080b3c20 2024-01-16T15:02:09.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.860+0000 7f0d0ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0d080cd220 0x7f0d0806d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41658/0 (socket says 172.21.15.141:41658) 2024-01-16T15:02:09.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.860+0000 7f0d0ea00640 1 -- 172.21.15.141:0/4127085652 learned_addr learned my addr 172.21.15.141:0/4127085652 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:09.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0e000640 1 --2- 172.21.15.141:0/4127085652 >> v2:172.21.15.188:3300/0 conn(0x7f0d08159590 0x7f0d08070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0d600640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 0x7f0d080ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270285346 0 0) 0x7f0d08071790 con 0x7f0d080cd220 2024-01-16T15:02:09.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce8005730 con 0x7f0d080cd220 2024-01-16T15:02:09.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 779983121 0 0) 0x7f0ce8005730 con 0x7f0d080cd220 2024-01-16T15:02:09.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 msgr2=0x7f0d080ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 0x7f0d080ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 >> v2:172.21.15.188:3300/0 conn(0x7f0d08159590 msgr2=0x7f0d08070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 --2- 172.21.15.141:0/4127085652 >> v2:172.21.15.188:3300/0 conn(0x7f0d08159590 0x7f0d08070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08070b40 con 0x7f0d080cd220 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0d000019f0 con 0x7f0d080cd220 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 >> v1:172.21.15.141:6789/0 conn(0x7f0d080cd220 legacy=0x7f0d0806d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce8005b80 0x7f0ce8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 0x7f0ce80069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ce8006f10 0x7f0ce8007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08071100 con 0x7f0ce8006f10 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08070860 con 0x7f0ce8005b80 2024-01-16T15:02:09.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.861+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce800db90 con 0x7f0d080b3c20 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ce8006f10 0x7f0ce8007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0d600640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 0x7f0ce80069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0e000640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce8005b80 0x7f0ce8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0ea00640 1 -- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 msgr2=0x7f0ce80069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d080b3c20 0x7f0ce80069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0ea00640 1 -- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce8005b80 msgr2=0x7f0ce8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce8005b80 0x7f0ce8005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.862+0000 7f0d0ea00640 1 -- 172.21.15.141:0/4127085652 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8005730 con 0x7f0ce8006f10 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.863+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ce8006f10 0x7f0ce8007390 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000a860 tx=0x7f0ce8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d27699618ab3a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.863+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00034630 con 0x7f0ce8006f10 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.863+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0002f070 con 0x7f0ce8006f10 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.863+0000 7f0d0cc00640 1 -- 172.21.15.141:0/4127085652 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d000349f0 con 0x7f0ce8006f10 2024-01-16T15:02:09.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.864+0000 7f0d0f400640 1 -- 172.21.15.141:0/4127085652 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ce8006f10 msgr2=0x7f0ce8007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.864+0000 7f0d0f400640 1 --2- 172.21.15.141:0/4127085652 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ce8006f10 0x7f0ce8007390 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000a860 tx=0x7f0ce8007a00 comp rx=0 tx=0).stop 2024-01-16T15:02:09.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.864+0000 7f0d0ea00640 1 -- 172.21.15.141:0/4127085652 reap_dead start 2024-01-16T15:02:09.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.864+0000 7f0d0f400640 1 -- 172.21.15.141:0/4127085652 shutdown_connections 2024-01-16T15:02:09.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.864+0000 7f0d0f400640 1 -- 172.21.15.141:0/4127085652 >> 172.21.15.141:0/4127085652 conn(0x7f0d0806a800 msgr2=0x7f0d0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.864+0000 7f0d0f400640 1 -- 172.21.15.141:0/4127085652 shutdown_connections 2024-01-16T15:02:09.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.864+0000 7f0d0f400640 1 -- 172.21.15.141:0/4127085652 wait complete. 2024-01-16T15:02:09.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.866+0000 7f0d0f400640 1 Processor -- start 2024-01-16T15:02:09.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.866+0000 7f0d0f400640 1 -- start start 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.866+0000 7f0d0f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ce8006f10 0x7f0d08155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.866+0000 7f0d0f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d08142670 0x7f0d08157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d080cdb50 0x7f0d080cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0f400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080ce510 con 0x7f0d08142670 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0f400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0814ff80 con 0x7f0d080cdb50 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d081585d0 con 0x7f0ce8006f10 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d08142670 0x7f0d08157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d08142670 0x7f0d08157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58110/0 (socket says 172.21.15.141:58110) 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 -- 172.21.15.141:0/3305284024 learned_addr learned my addr 172.21.15.141:0/3305284024 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 -- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ce8006f10 msgr2=0x7f0d08155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0e000640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ce8006f10 0x7f0d08155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ce8006f10 0x7f0d08155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 -- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d080cdb50 msgr2=0x7f0d080cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d080cdb50 0x7f0d080cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 -- 172.21.15.141:0/3305284024 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d081588b0 con 0x7f0d08142670 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.867+0000 7f0d0d600640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d08142670 0x7f0d08157d80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0010860 tx=0x7f0cf0010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.868+0000 7f0d06000640 1 -- 172.21.15.141:0/3305284024 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0009610 con 0x7f0d08142670 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.868+0000 7f0d06000640 1 -- 172.21.15.141:0/3305284024 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf000eae0 con 0x7f0d08142670 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.868+0000 7f0d06000640 1 -- 172.21.15.141:0/3305284024 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000ee70 con 0x7f0d08142670 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.868+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d08158b90 con 0x7f0d08142670 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.868+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d081582c0 con 0x7f0d08142670 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.869+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d08067450 con 0x7f0d08142670 2024-01-16T15:02:09.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.870+0000 7f0d06000640 1 -- 172.21.15.141:0/3305284024 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f0cf001c400 con 0x7f0d08142670 2024-01-16T15:02:09.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.871+0000 7f0d06000640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce006b920 0x7f0ce006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.871+0000 7f0d06000640 1 -- 172.21.15.141:0/3305284024 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0cf008ac90 con 0x7f0d08142670 2024-01-16T15:02:09.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.872+0000 7f0d0e000640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce006b920 0x7f0ce006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.873+0000 7f0d0e000640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce006b920 0x7f0ce006dde0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006e80 tx=0x7f0cf8006dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:09.876+0000 7f0d06000640 1 -- 172.21.15.141:0/3305284024 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cf0055140 con 0x7f0d08142670 2024-01-16T15:02:10.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.088+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone29511", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d08159970 con 0x7f0ce006b920 2024-01-16T15:02:10.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.133+0000 7f0d06000640 1 -- 172.21.15.141:0/3305284024 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0cf800a6f0 con 0x7f0ce006b920 2024-01-16T15:02:10.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce006b920 msgr2=0x7f0ce006dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce006b920 0x7f0ce006dde0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006e80 tx=0x7f0cf8006dd0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d08142670 msgr2=0x7f0d08157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d08142670 0x7f0d08157d80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0010860 tx=0x7f0cf0010d20 comp rx=0 tx=0).stop 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 shutdown_connections 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce006b920 0x7f0ce006dde0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d080cdb50 0x7f0d080cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d08142670 0x7f0d08157d80 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 --2- 172.21.15.141:0/3305284024 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ce8006f10 0x7f0d08155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 >> 172.21.15.141:0/3305284024 conn(0x7f0d0806a800 msgr2=0x7f0d081535e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.136+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 shutdown_connections 2024-01-16T15:02:10.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.137+0000 7f0d0f400640 1 -- 172.21.15.141:0/3305284024 wait complete. 2024-01-16T15:02:10.134 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone29513 2024-01-16T15:02:10.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.357+0000 7fed0ec00640 1 Processor -- start 2024-01-16T15:02:10.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.357+0000 7fed0ec00640 1 -- start start 2024-01-16T15:02:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.359+0000 7fed0ec00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fed0815e3b0 0x7fed08164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.359+0000 7fed0ec00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed080b1ee0 con 0x7fed08143ae0 2024-01-16T15:02:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.359+0000 7fed0ec00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed080b2780 con 0x7fed080cf9a0 2024-01-16T15:02:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.359+0000 7fed0ec00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0815e990 con 0x7fed0815e3b0 2024-01-16T15:02:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.359+0000 7fed07e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fed08143ae0 0x7fed080cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41664/0 (socket says 172.21.15.141:41664) 2024-01-16T15:02:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.359+0000 7fed07e00640 1 -- 172.21.15.141:0/276385638 learned_addr learned my addr 172.21.15.141:0/276385638 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed0cc00640 1 --2- 172.21.15.141:0/276385638 >> v2:172.21.15.188:3300/0 conn(0x7fed0815e3b0 0x7fed08164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122626132 0 0) 0x7fed080b1ee0 con 0x7fed08143ae0 2024-01-16T15:02:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece8003520 con 0x7fed08143ae0 2024-01-16T15:02:10.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105734050 0 0) 0x7fece8003520 con 0x7fed08143ae0 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 >> v2:172.21.15.188:3300/0 conn(0x7fed0815e3b0 msgr2=0x7fed08164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 --2- 172.21.15.141:0/276385638 >> v2:172.21.15.188:3300/0 conn(0x7fed0815e3b0 0x7fed08164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 >> v1:172.21.15.188:6789/0 conn(0x7fed080cf9a0 legacy=0x7fed0815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080b2130 con 0x7fed08143ae0 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fecf8001670 con 0x7fed08143ae0 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 >> v1:172.21.15.141:6789/0 conn(0x7fed08143ae0 legacy=0x7fed080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fece80039c0 0x7fece8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece80047c0 0x7fece8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fece8005180 0x7fece800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0815e990 con 0x7fece8005180 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece800c080 con 0x7fece80039c0 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.360+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece800c240 con 0x7fece80047c0 2024-01-16T15:02:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.361+0000 7fed0cc00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fece8005180 0x7fece800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.361+0000 7fed0cc00640 1 -- 172.21.15.141:0/276385638 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece80047c0 msgr2=0x7fece8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.361+0000 7fed0cc00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece80047c0 0x7fece8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.361+0000 7fed0cc00640 1 -- 172.21.15.141:0/276385638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fece80039c0 msgr2=0x7fece8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.361+0000 7fed0cc00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fece80039c0 0x7fece8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.361+0000 7fed0cc00640 1 -- 172.21.15.141:0/276385638 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8003520 con 0x7fece8005180 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.362+0000 7fed0cc00640 1 -- 172.21.15.141:0/276385638 reap_dead start 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.362+0000 7fed0cc00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fece8005180 0x7fece800b980 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fed00009fc0 tx=0x7fece80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eed370d93bed7e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.362+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0002cc60 con 0x7fece8005180 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.362+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0000f040 con 0x7fece8005180 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.362+0000 7fed06a00640 1 -- 172.21.15.141:0/276385638 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00003990 con 0x7fece8005180 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.363+0000 7fed0ec00640 1 -- 172.21.15.141:0/276385638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fece8005180 msgr2=0x7fece800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.363+0000 7fed0ec00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fece8005180 0x7fece800b980 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fed00009fc0 tx=0x7fece80057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.363+0000 7fed0ec00640 1 -- 172.21.15.141:0/276385638 shutdown_connections 2024-01-16T15:02:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.363+0000 7fed0ec00640 1 --2- 172.21.15.141:0/276385638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fece8005180 0x7fece800b980 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.363+0000 7fed0ec00640 1 -- 172.21.15.141:0/276385638 >> 172.21.15.141:0/276385638 conn(0x7fed0806a800 msgr2=0x7fed0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.363+0000 7fed0ec00640 1 -- 172.21.15.141:0/276385638 shutdown_connections 2024-01-16T15:02:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.363+0000 7fed0ec00640 1 -- 172.21.15.141:0/276385638 wait complete. 2024-01-16T15:02:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.365+0000 7fed0ec00640 1 Processor -- start 2024-01-16T15:02:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.365+0000 7fed0ec00640 1 -- start start 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.366+0000 7fed0ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece8005180 0x7fed081f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.366+0000 7fed0ec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed080ac3b0 0x7fed0806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.366+0000 7fed0ec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed0806c9b0 0x7fed08140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.366+0000 7fed0ec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0806ce60 con 0x7fed080ac3b0 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.366+0000 7fed0ec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed081f0820 con 0x7fed0806c9b0 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.366+0000 7fed0ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0812e950 con 0x7fece8005180 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed080ac3b0 0x7fed0806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed080ac3b0 0x7fed0806c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58130/0 (socket says 172.21.15.141:58130) 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 -- 172.21.15.141:0/1014890709 learned_addr learned my addr 172.21.15.141:0/1014890709 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07e00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece8005180 0x7fed081f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed0cc00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed0806c9b0 0x7fed08140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 -- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece8005180 msgr2=0x7fed081f02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece8005180 0x7fed081f02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 -- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed0806c9b0 msgr2=0x7fed08140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed0806c9b0 0x7fed08140250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed07400640 1 -- 172.21.15.141:0/1014890709 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08140ff0 con 0x7fed080ac3b0 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.367+0000 7fed0cc00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed0806c9b0 0x7fed08140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.368+0000 7fed07400640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed080ac3b0 0x7fed0806c470 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c450 tx=0x7fecf000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.368+0000 7fecffe00640 1 -- 172.21.15.141:0/1014890709 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf00052d0 con 0x7fed080ac3b0 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.368+0000 7fecffe00640 1 -- 172.21.15.141:0/1014890709 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf0005430 con 0x7fed080ac3b0 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.368+0000 7fecffe00640 1 -- 172.21.15.141:0/1014890709 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0012870 con 0x7fed080ac3b0 2024-01-16T15:02:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.368+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0812e020 con 0x7fed080ac3b0 2024-01-16T15:02:10.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.368+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0812e5e0 con 0x7fed080ac3b0 2024-01-16T15:02:10.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.369+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed080675d0 con 0x7fed080ac3b0 2024-01-16T15:02:10.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.370+0000 7fecffe00640 1 -- 172.21.15.141:0/1014890709 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fecf00129d0 con 0x7fed080ac3b0 2024-01-16T15:02:10.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.371+0000 7fecffe00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fece006b8f0 0x7fece006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.371+0000 7fecffe00640 1 -- 172.21.15.141:0/1014890709 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fecf008a2e0 con 0x7fed080ac3b0 2024-01-16T15:02:10.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.371+0000 7fed07e00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fece006b8f0 0x7fece006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.372+0000 7fed07e00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fece006b8f0 0x7fece006ddb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fecf8001860 tx=0x7fecf8001670 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.375+0000 7fecffe00640 1 -- 172.21.15.141:0/1014890709 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fecf0054790 con 0x7fed080ac3b0 2024-01-16T15:02:10.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.590+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone29513", "target": ["mon-mgr", ""]}) v1 -- 0x7fed0815dca0 con 0x7fece006b8f0 2024-01-16T15:02:10.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.626+0000 7fecffe00640 1 -- 172.21.15.141:0/1014890709 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fecf800e7b0 con 0x7fece006b8f0 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fece006b8f0 msgr2=0x7fece006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fece006b8f0 0x7fece006ddb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fecf8001860 tx=0x7fecf8001670 comp rx=0 tx=0).stop 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed080ac3b0 msgr2=0x7fed0806c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed080ac3b0 0x7fed0806c470 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c450 tx=0x7fecf000c910 comp rx=0 tx=0).stop 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 shutdown_connections 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fece006b8f0 0x7fece006ddb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed0806c9b0 0x7fed08140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed080ac3b0 0x7fed0806c470 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 --2- 172.21.15.141:0/1014890709 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fece8005180 0x7fed081f02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 >> 172.21.15.141:0/1014890709 conn(0x7fed0806a800 msgr2=0x7fed0806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.629+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 shutdown_connections 2024-01-16T15:02:10.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:10.630+0000 7fed0ec00640 1 -- 172.21.15.141:0/1014890709 wait complete. 2024-01-16T15:02:10.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:10.629 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:02:10.699 INFO:teuthology.orchestra.run.smithi141.stdout:2 2024-01-16T15:02:10.701 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:02:10.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.011+0000 7f68f0c00640 1 Processor -- start 2024-01-16T15:02:10.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.011+0000 7f68f0c00640 1 -- start start 2024-01-16T15:02:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.013+0000 7f68f0c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f68ec0ae5a0 0x7f68ec15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.013+0000 7f68f0c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec0b3ae0 0x7f68ec06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.013+0000 7f68f0c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec070c40 con 0x7f68ec06d120 2024-01-16T15:02:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.013+0000 7f68f0c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ec071450 con 0x7f68ec0ae5a0 2024-01-16T15:02:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.013+0000 7f68f0c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec06c470 con 0x7f68ec0b3ae0 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68e9e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f68ec0ae5a0 0x7f68ec15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68e9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec0b3ae0 0x7f68ec06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68e9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec0b3ae0 0x7f68ec06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:38508/0 (socket says 172.21.15.141:38508) 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68e9400640 1 -- 172.21.15.141:0/1568314704 learned_addr learned my addr 172.21.15.141:0/1568314704 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68ea800640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f68ec06d120 0x7f68ec0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:54112/0 (socket says 172.21.15.141:54112) 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68e9400640 1 -- 172.21.15.141:0/1568314704 >> v2:172.21.15.188:3300/0 conn(0x7f68ec0ae5a0 msgr2=0x7f68ec15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68e9400640 1 --2- 172.21.15.141:0/1568314704 >> v2:172.21.15.188:3300/0 conn(0x7f68ec0ae5a0 0x7f68ec15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.014+0000 7f68e9400640 1 -- 172.21.15.141:0/1568314704 >> v1:172.21.15.188:6789/0 conn(0x7f68ec06d120 legacy=0x7f68ec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.015+0000 7f68e9400640 1 -- 172.21.15.141:0/1568314704 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec06c750 con 0x7f68ec0b3ae0 2024-01-16T15:02:10.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.016+0000 7f68e9400640 1 --2- 172.21.15.141:0/1568314704 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec0b3ae0 0x7f68ec06cbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009870 tx=0x7f68d402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd308a0306aa3d1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.017+0000 7f68e8a00640 1 -- 172.21.15.141:0/1568314704 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4030840 con 0x7f68ec0b3ae0 2024-01-16T15:02:10.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.017+0000 7f68e8a00640 1 -- 172.21.15.141:0/1568314704 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d40309a0 con 0x7f68ec0b3ae0 2024-01-16T15:02:10.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.017+0000 7f68e8a00640 1 -- 172.21.15.141:0/1568314704 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4030d30 con 0x7f68ec0b3ae0 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.018+0000 7f68f0c00640 1 -- 172.21.15.141:0/1568314704 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec0b3ae0 msgr2=0x7f68ec06cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.018+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1568314704 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec0b3ae0 0x7f68ec06cbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009870 tx=0x7f68d402b980 comp rx=0 tx=0).stop 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.018+0000 7f68f0c00640 1 -- 172.21.15.141:0/1568314704 shutdown_connections 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.018+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1568314704 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec0b3ae0 0x7f68ec06cbe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.018+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1568314704 >> v2:172.21.15.188:3300/0 conn(0x7f68ec0ae5a0 0x7f68ec15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.018+0000 7f68f0c00640 1 -- 172.21.15.141:0/1568314704 >> 172.21.15.141:0/1568314704 conn(0x7f68ec06a800 msgr2=0x7f68ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.019+0000 7f68f0c00640 1 -- 172.21.15.141:0/1568314704 shutdown_connections 2024-01-16T15:02:10.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.019+0000 7f68f0c00640 1 -- 172.21.15.141:0/1568314704 wait complete. 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.021+0000 7f68f0c00640 1 Processor -- start 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.021+0000 7f68f0c00640 1 -- start start 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.022+0000 7f68f0c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec06d120 0x7f68ec1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.022+0000 7f68f0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68ec0ae5a0 0x7f68ec0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.023+0000 7f68e9e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec06d120 0x7f68ec1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.023+0000 7f68e9e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec06d120 0x7f68ec1588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:38518/0 (socket says 172.21.15.141:38518) 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.023+0000 7f68f0c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68ec0b3ae0 0x7f68ec159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.023+0000 7f68e9e00640 1 -- 172.21.15.141:0/1423942605 learned_addr learned my addr 172.21.15.141:0/1423942605 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.023+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec158df0 con 0x7f68ec0ae5a0 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.023+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec0ce5c0 con 0x7f68ec0b3ae0 2024-01-16T15:02:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.023+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec1fa4e0 con 0x7f68ec06d120 2024-01-16T15:02:11.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68ea800640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68ec0b3ae0 0x7f68ec159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68e9400640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68ec0ae5a0 0x7f68ec0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68e9e00640 1 -- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68ec0b3ae0 msgr2=0x7f68ec159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68e9e00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68ec0b3ae0 0x7f68ec159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68e9e00640 1 -- 172.21.15.141:0/1423942605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68ec0ae5a0 msgr2=0x7f68ec0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68e9e00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68ec0ae5a0 0x7f68ec0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68e9e00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40094d0 con 0x7f68ec06d120 2024-01-16T15:02:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.024+0000 7f68e9400640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68ec0ae5a0 0x7f68ec0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.025+0000 7f68e9e00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec06d120 0x7f68ec1588b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00e9b0 tx=0x7f68dc00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.026+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc009670 con 0x7f68ec06d120 2024-01-16T15:02:11.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.026+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc00c9a0 con 0x7f68ec06d120 2024-01-16T15:02:11.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.026+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc00cd50 con 0x7f68ec06d120 2024-01-16T15:02:11.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.026+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec1fa7c0 con 0x7f68ec06d120 2024-01-16T15:02:11.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.028+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec1fad80 con 0x7f68ec06d120 2024-01-16T15:02:11.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.031+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f68dc013470 con 0x7f68ec06d120 2024-01-16T15:02:11.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.032+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec1fafa0 con 0x7f68ec06d120 2024-01-16T15:02:11.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.033+0000 7f68e2000640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68c406b940 0x7f68c406de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.033+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f68dc08a940 con 0x7f68ec06d120 2024-01-16T15:02:11.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.034+0000 7f68e9400640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68c406b940 0x7f68c406de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.035+0000 7f68e9400640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68c406b940 0x7f68c406de00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f68d4036000 tx=0x7f68d403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.043+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68dc054f20 con 0x7f68ec06d120 2024-01-16T15:02:11.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:11.286+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f68ec070550 con 0x7f68c406b940 2024-01-16T15:02:11.316 INFO:tasks.ceph.mon.a.smithi141.stderr:2024-01-16T15:02:11.334+0000 7f723b200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:02:11.409 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:02:13.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:13.594+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f68dc0552f0 con 0x7f68ec06d120 2024-01-16T15:02:14.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.365+0000 7f68e2000640 1 -- 172.21.15.141:0/1423942605 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f68d4030d00 con 0x7f68c406b940 2024-01-16T15:02:14.345 INFO:teuthology.orchestra.run.smithi141.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:02:14.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68c406b940 msgr2=0x7f68c406de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68c406b940 0x7f68c406de00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f68d4036000 tx=0x7f68d403b040 comp rx=0 tx=0).stop 2024-01-16T15:02:14.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec06d120 msgr2=0x7f68ec1588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec06d120 0x7f68ec1588b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00e9b0 tx=0x7f68dc00ee70 comp rx=0 tx=0).stop 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 shutdown_connections 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68c406b940 0x7f68c406de00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68ec0b3ae0 0x7f68ec159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68ec0ae5a0 0x7f68ec0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 --2- 172.21.15.141:0/1423942605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68ec06d120 0x7f68ec1588b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 >> 172.21.15.141:0/1423942605 conn(0x7f68ec06a800 msgr2=0x7f68ec0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.369+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 shutdown_connections 2024-01-16T15:02:14.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.370+0000 7f68f0c00640 1 -- 172.21.15.141:0/1423942605 wait complete. 2024-01-16T15:02:14.379 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:02:14.379 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:14.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:14.646 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:02:14.646 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:02:14.647 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:14.707 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:14.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.905+0000 7fce80e00640 1 Processor -- start 2024-01-16T15:02:14.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.905+0000 7fce80e00640 1 -- start start 2024-01-16T15:02:14.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce80e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fce7c0cf9a0 0x7fce7c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce80e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fce7c15e1e0 0x7fce7c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce80e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce7c0b1ee0 con 0x7fce7c143ae0 2024-01-16T15:02:14.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce80e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7c0b2780 con 0x7fce7c15e1e0 2024-01-16T15:02:14.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce80e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7c15e990 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce7a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fce7c15e1e0 0x7fce7c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce7a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fce7c15e1e0 0x7fce7c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33612/0 (socket says 172.21.15.141:33612) 2024-01-16T15:02:14.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce7a800640 1 -- 172.21.15.141:0/2162355949 learned_addr learned my addr 172.21.15.141:0/2162355949 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:14.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce79400640 1 --2- 172.21.15.141:0/2162355949 >> v2:172.21.15.188:3300/0 conn(0x7fce7c0cf9a0 0x7fce7c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce7a800640 1 -- 172.21.15.141:0/2162355949 >> v2:172.21.15.188:3300/0 conn(0x7fce7c0cf9a0 msgr2=0x7fce7c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:14.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce7a800640 1 --2- 172.21.15.141:0/2162355949 >> v2:172.21.15.188:3300/0 conn(0x7fce7c0cf9a0 0x7fce7c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce7a800640 1 -- 172.21.15.141:0/2162355949 >> v1:172.21.15.141:6789/0 conn(0x7fce7c143ae0 legacy=0x7fce7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.908+0000 7fce7a800640 1 -- 172.21.15.141:0/2162355949 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7c0b2130 con 0x7fce7c15e1e0 2024-01-16T15:02:14.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.907+0000 7fce78a00640 1 -- 172.21.15.141:0/2162355949 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990652220 0 0) 0x7fce7c0b1ee0 con 0x7fce7c143ae0 2024-01-16T15:02:14.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.908+0000 7fce7a800640 1 --2- 172.21.15.141:0/2162355949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c15e1e0 0x7fce7c0702c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0cb830 tx=0x7fce7402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca85ed5d133ff8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:14.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce78a00640 1 -- 172.21.15.141:0/2162355949 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7402cb20 con 0x7fce7c15e1e0 2024-01-16T15:02:14.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce78a00640 1 -- 172.21.15.141:0/2162355949 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7402cc80 con 0x7fce7c15e1e0 2024-01-16T15:02:14.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce78a00640 1 -- 172.21.15.141:0/2162355949 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce740343f0 con 0x7fce7c15e1e0 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce80e00640 1 -- 172.21.15.141:0/2162355949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c15e1e0 msgr2=0x7fce7c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce80e00640 1 --2- 172.21.15.141:0/2162355949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c15e1e0 0x7fce7c0702c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0cb830 tx=0x7fce7402bf80 comp rx=0 tx=0).stop 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce80e00640 1 -- 172.21.15.141:0/2162355949 shutdown_connections 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce80e00640 1 --2- 172.21.15.141:0/2162355949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c15e1e0 0x7fce7c0702c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce80e00640 1 --2- 172.21.15.141:0/2162355949 >> v2:172.21.15.188:3300/0 conn(0x7fce7c0cf9a0 0x7fce7c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce80e00640 1 -- 172.21.15.141:0/2162355949 >> 172.21.15.141:0/2162355949 conn(0x7fce7c06a800 msgr2=0x7fce7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.909+0000 7fce80e00640 1 -- 172.21.15.141:0/2162355949 shutdown_connections 2024-01-16T15:02:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.910+0000 7fce80e00640 1 -- 172.21.15.141:0/2162355949 wait complete. 2024-01-16T15:02:14.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.911+0000 7fce80e00640 1 Processor -- start 2024-01-16T15:02:14.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.911+0000 7fce80e00640 1 -- start start 2024-01-16T15:02:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce80e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c0cf9a0 0x7fce7c159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce80e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce7c143ae0 0x7fce7c132030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce80e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce7c15e1e0 0x7fce7c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce80e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c12ffa0 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce80e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c132ad0 con 0x7fce7c143ae0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce80e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c132db0 con 0x7fce7c15e1e0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce79e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c0cf9a0 0x7fce7c159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce79e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c0cf9a0 0x7fce7c159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33616/0 (socket says 172.21.15.141:33616) 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.912+0000 7fce79e00640 1 -- 172.21.15.141:0/500005703 learned_addr learned my addr 172.21.15.141:0/500005703 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce79400640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce7c143ae0 0x7fce7c132030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce79e00640 1 -- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce7c15e1e0 msgr2=0x7fce7c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce79e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce7c15e1e0 0x7fce7c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce79e00640 1 -- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce7c143ae0 msgr2=0x7fce7c132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce79e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce7c143ae0 0x7fce7c132030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce79e00640 1 -- 172.21.15.141:0/500005703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74009b30 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce79e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c0cf9a0 0x7fce7c159970 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00c4b0 tx=0x7fce6c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce72000640 1 -- 172.21.15.141:0/500005703 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c005480 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce72000640 1 -- 172.21.15.141:0/500005703 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c0035f0 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce72000640 1 -- 172.21.15.141:0/500005703 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c0039e0 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.913+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7c1301b0 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.914+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7c15d3f0 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.915+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7c1306d0 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.915+0000 7fce72000640 1 -- 172.21.15.141:0/500005703 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fce6c003b40 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.917+0000 7fce72000640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce5406b860 0x7fce5406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:14.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.917+0000 7fce72000640 1 -- 172.21.15.141:0/500005703 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fce6c089930 con 0x7fce7c0cf9a0 2024-01-16T15:02:14.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.917+0000 7fce79400640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce5406b860 0x7fce5406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:14.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.919+0000 7fce79400640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce5406b860 0x7fce5406dd20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce7c15d150 tx=0x7fce64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:14.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:14.922+0000 7fce72000640 1 -- 172.21.15.141:0/500005703 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce7c1306d0 con 0x7fce7c0cf9a0 2024-01-16T15:02:15.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.099+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce7c1306d0 con 0x7fce7c0cf9a0 2024-01-16T15:02:15.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.101+0000 7fce72000640 1 -- 172.21.15.141:0/500005703 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+13036 (secure 0 0 0) 0x7fce7c1306d0 con 0x7fce7c0cf9a0 2024-01-16T15:02:15.081 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:15.082 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":51,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:02:14.349088+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/2933218811":"2024-01-17T15:02:11.334365+0000","172.21.15.141:6834/2933218811":"2024-01-17T15:02:11.334365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:15.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce5406b860 msgr2=0x7fce5406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce5406b860 0x7fce5406dd20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce7c15d150 tx=0x7fce64008040 comp rx=0 tx=0).stop 2024-01-16T15:02:15.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c0cf9a0 msgr2=0x7fce7c159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c0cf9a0 0x7fce7c159970 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00c4b0 tx=0x7fce6c00c970 comp rx=0 tx=0).stop 2024-01-16T15:02:15.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 shutdown_connections 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fce5406b860 0x7fce5406dd20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fce7c15e1e0 0x7fce7c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce7c143ae0 0x7fce7c132030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 --2- 172.21.15.141:0/500005703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce7c0cf9a0 0x7fce7c159970 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 >> 172.21.15.141:0/500005703 conn(0x7fce7c06a800 msgr2=0x7fce7c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 shutdown_connections 2024-01-16T15:02:15.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.104+0000 7fce80e00640 1 -- 172.21.15.141:0/500005703 wait complete. 2024-01-16T15:02:15.102 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:15.122 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:15.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.349+0000 7fde6f400640 1 Processor -- start 2024-01-16T15:02:15.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.349+0000 7fde6f400640 1 -- start start 2024-01-16T15:02:15.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.350+0000 7fde6f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde6806c730 0x7fde680b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6f400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fde6806cb10 0x7fde6806cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6f400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680708d0 con 0x7fde68070160 2024-01-16T15:02:15.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6f400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde68070bb0 con 0x7fde6806c730 2024-01-16T15:02:15.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6f400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde68070e90 con 0x7fde6806cb10 2024-01-16T15:02:15.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde6806c730 0x7fde680b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde6806c730 0x7fde680b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33620/0 (socket says 172.21.15.141:33620) 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6e000640 1 -- 172.21.15.141:0/1896664635 learned_addr learned my addr 172.21.15.141:0/1896664635 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6e000640 1 -- 172.21.15.141:0/1896664635 >> v2:172.21.15.188:3300/0 conn(0x7fde6806cb10 msgr2=0x7fde6806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6ea00640 1 --2- 172.21.15.141:0/1896664635 >> v2:172.21.15.188:3300/0 conn(0x7fde6806cb10 0x7fde6806cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6e000640 1 --2- 172.21.15.141:0/1896664635 >> v2:172.21.15.188:3300/0 conn(0x7fde6806cb10 0x7fde6806cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.351+0000 7fde6e000640 1 -- 172.21.15.141:0/1896664635 >> v1:172.21.15.188:6789/0 conn(0x7fde68070160 legacy=0x7fde68071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.352+0000 7fde6e000640 1 -- 172.21.15.141:0/1896664635 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde68071170 con 0x7fde6806c730 2024-01-16T15:02:15.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.352+0000 7fde6ea00640 1 --2- 172.21.15.141:0/1896664635 >> v2:172.21.15.188:3300/0 conn(0x7fde6806cb10 0x7fde6806cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:15.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.353+0000 7fde6e000640 1 --2- 172.21.15.141:0/1896664635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806c730 0x7fde680b2e00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fde6806e530 tx=0x7fde5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5cf153965d8825 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:15.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.353+0000 7fde6cc00640 1 -- 172.21.15.141:0/1896664635 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58031870 con 0x7fde6806c730 2024-01-16T15:02:15.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.353+0000 7fde6cc00640 1 -- 172.21.15.141:0/1896664635 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde580319d0 con 0x7fde6806c730 2024-01-16T15:02:15.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.353+0000 7fde6cc00640 1 -- 172.21.15.141:0/1896664635 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58031d60 con 0x7fde6806c730 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 -- 172.21.15.141:0/1896664635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806c730 msgr2=0x7fde680b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 --2- 172.21.15.141:0/1896664635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806c730 0x7fde680b2e00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fde6806e530 tx=0x7fde5802b910 comp rx=0 tx=0).stop 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 -- 172.21.15.141:0/1896664635 shutdown_connections 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 --2- 172.21.15.141:0/1896664635 >> v2:172.21.15.188:3300/0 conn(0x7fde6806cb10 0x7fde6806cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 --2- 172.21.15.141:0/1896664635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806c730 0x7fde680b2e00 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 -- 172.21.15.141:0/1896664635 >> 172.21.15.141:0/1896664635 conn(0x7fde6806a7d0 msgr2=0x7fde6806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 -- 172.21.15.141:0/1896664635 shutdown_connections 2024-01-16T15:02:15.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.354+0000 7fde6f400640 1 -- 172.21.15.141:0/1896664635 wait complete. 2024-01-16T15:02:15.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.356+0000 7fde6f400640 1 Processor -- start 2024-01-16T15:02:15.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.356+0000 7fde6f400640 1 -- start start 2024-01-16T15:02:15.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde6806c730 0x7fde68160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806cb10 0x7fde6813c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde68070160 0x7fde680abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6f400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde680ac6b0 con 0x7fde6806cb10 2024-01-16T15:02:15.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6f400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde680ac990 con 0x7fde6806c730 2024-01-16T15:02:15.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde680acc70 con 0x7fde68070160 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806cb10 0x7fde6813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806cb10 0x7fde6813c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33636/0 (socket says 172.21.15.141:33636) 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.357+0000 7fde6d600640 1 -- 172.21.15.141:0/2820594257 learned_addr learned my addr 172.21.15.141:0/2820594257 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6e000640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde6806c730 0x7fde68160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6d600640 1 -- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde68070160 msgr2=0x7fde680abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6d600640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde68070160 0x7fde680abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6d600640 1 -- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde6806c730 msgr2=0x7fde68160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6d600640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde6806c730 0x7fde68160f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6d600640 1 -- 172.21.15.141:0/2820594257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580094d0 con 0x7fde6806cb10 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6d600640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806cb10 0x7fde6813c890 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fde5000b3e0 tx=0x7fde5000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde66000640 1 -- 172.21.15.141:0/2820594257 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50011830 con 0x7fde6806cb10 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde66000640 1 -- 172.21.15.141:0/2820594257 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde500119b0 con 0x7fde6806cb10 2024-01-16T15:02:15.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde66000640 1 -- 172.21.15.141:0/2820594257 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50011d60 con 0x7fde6806cb10 2024-01-16T15:02:15.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.358+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde680acf50 con 0x7fde6806cb10 2024-01-16T15:02:15.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.359+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68161490 con 0x7fde6806cb10 2024-01-16T15:02:15.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.360+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde680ad470 con 0x7fde6806cb10 2024-01-16T15:02:15.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.360+0000 7fde66000640 1 -- 172.21.15.141:0/2820594257 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fde5001a400 con 0x7fde6806cb10 2024-01-16T15:02:15.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.362+0000 7fde66000640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde4006b860 0x7fde4006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.362+0000 7fde66000640 1 -- 172.21.15.141:0/2820594257 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fde5008a580 con 0x7fde6806cb10 2024-01-16T15:02:15.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.362+0000 7fde6e000640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde4006b860 0x7fde4006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.364+0000 7fde6e000640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde4006b860 0x7fde4006dd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fde58036000 tx=0x7fde5803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:15.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.366+0000 7fde66000640 1 -- 172.21.15.141:0/2820594257 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde50054db0 con 0x7fde6806cb10 2024-01-16T15:02:15.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.542+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde68067450 con 0x7fde6806cb10 2024-01-16T15:02:15.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.542+0000 7fde66000640 1 -- 172.21.15.141:0/2820594257 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde50015020 con 0x7fde6806cb10 2024-01-16T15:02:15.522 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde4006b860 msgr2=0x7fde4006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde4006b860 0x7fde4006dd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fde58036000 tx=0x7fde5803c040 comp rx=0 tx=0).stop 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806cb10 msgr2=0x7fde6813c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806cb10 0x7fde6813c890 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fde5000b3e0 tx=0x7fde5000b8a0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 shutdown_connections 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde4006b860 0x7fde4006dd20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde68070160 0x7fde680abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6806cb10 0x7fde6813c890 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 --2- 172.21.15.141:0/2820594257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde6806c730 0x7fde68160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 >> 172.21.15.141:0/2820594257 conn(0x7fde6806a7d0 msgr2=0x7fde68147ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:15.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 shutdown_connections 2024-01-16T15:02:15.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.545+0000 7fde6f400640 1 -- 172.21.15.141:0/2820594257 wait complete. 2024-01-16T15:02:15.541 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:02:15.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.758+0000 7f8898000640 1 Processor -- start 2024-01-16T15:02:15.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.758+0000 7f8898000640 1 -- start start 2024-01-16T15:02:15.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.759+0000 7f8898000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8890143ae0 0x7f88900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.759+0000 7f8898000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f88900cf7d0 0x7f889015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.759+0000 7f8898000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f889015e1e0 0x7f889015a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.759+0000 7f8898000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88900cfc30 con 0x7f88900cf7d0 2024-01-16T15:02:15.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.759+0000 7f8898000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88900b1ee0 con 0x7f889015e1e0 2024-01-16T15:02:15.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.759+0000 7f8898000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88900b2780 con 0x7f8890143ae0 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f88900cf7d0 0x7f889015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f88900cf7d0 0x7f889015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33648/0 (socket says 172.21.15.141:33648) 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 -- 172.21.15.141:0/797439214 learned_addr learned my addr 172.21.15.141:0/797439214 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8895600640 1 --2- 172.21.15.141:0/797439214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8890143ae0 0x7f88900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8896000640 1 --2- 172.21.15.141:0/797439214 >> v2:172.21.15.188:3300/0 conn(0x7f889015e1e0 0x7f889015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 -- 172.21.15.141:0/797439214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8890143ae0 msgr2=0x7f88900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 --2- 172.21.15.141:0/797439214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8890143ae0 0x7f88900cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 -- 172.21.15.141:0/797439214 >> v2:172.21.15.188:3300/0 conn(0x7f889015e1e0 msgr2=0x7f889015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 --2- 172.21.15.141:0/797439214 >> v2:172.21.15.188:3300/0 conn(0x7f889015e1e0 0x7f889015a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.760+0000 7f8894c00640 1 -- 172.21.15.141:0/797439214 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889015e990 con 0x7f88900cf7d0 2024-01-16T15:02:15.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.761+0000 7f8894c00640 1 --2- 172.21.15.141:0/797439214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88900cf7d0 0x7f889015dca0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8890072160 tx=0x7f887802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100814d144fd8b28 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f888fe00640 1 -- 172.21.15.141:0/797439214 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887802ca70 con 0x7f88900cf7d0 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f888fe00640 1 -- 172.21.15.141:0/797439214 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887802cbd0 con 0x7f88900cf7d0 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f888fe00640 1 -- 172.21.15.141:0/797439214 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88780343f0 con 0x7f88900cf7d0 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 -- 172.21.15.141:0/797439214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88900cf7d0 msgr2=0x7f889015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 --2- 172.21.15.141:0/797439214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88900cf7d0 0x7f889015dca0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8890072160 tx=0x7f887802bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 -- 172.21.15.141:0/797439214 shutdown_connections 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 --2- 172.21.15.141:0/797439214 >> v2:172.21.15.188:3300/0 conn(0x7f889015e1e0 0x7f889015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 --2- 172.21.15.141:0/797439214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88900cf7d0 0x7f889015dca0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 --2- 172.21.15.141:0/797439214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8890143ae0 0x7f88900cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 -- 172.21.15.141:0/797439214 >> 172.21.15.141:0/797439214 conn(0x7f889006a800 msgr2=0x7f889006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 -- 172.21.15.141:0/797439214 shutdown_connections 2024-01-16T15:02:15.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.762+0000 7f8898000640 1 -- 172.21.15.141:0/797439214 wait complete. 2024-01-16T15:02:15.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.764+0000 7f8898000640 1 Processor -- start 2024-01-16T15:02:15.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.764+0000 7f8898000640 1 -- start start 2024-01-16T15:02:15.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8898000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88900cf7d0 0x7f8890160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8898000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8890143ae0 0x7f88901421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8898000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f889015e1e0 0x7f88900af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8898000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88901617c0 con 0x7f8890143ae0 2024-01-16T15:02:15.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8898000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8890161aa0 con 0x7f88900cf7d0 2024-01-16T15:02:15.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8898000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8890161d80 con 0x7f889015e1e0 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8894c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8890143ae0 0x7f88901421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8894c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8890143ae0 0x7f88901421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33654/0 (socket says 172.21.15.141:33654) 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.765+0000 7f8894c00640 1 -- 172.21.15.141:0/4061630786 learned_addr learned my addr 172.21.15.141:0/4061630786 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8895600640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88900cf7d0 0x7f8890160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8894c00640 1 -- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f889015e1e0 msgr2=0x7f88900af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8894c00640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f889015e1e0 0x7f88900af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8894c00640 1 -- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88900cf7d0 msgr2=0x7f8890160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8894c00640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88900cf7d0 0x7f8890160ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8894c00640 1 -- 172.21.15.141:0/4061630786 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8878009b30 con 0x7f8890143ae0 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8894c00640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8890143ae0 0x7f88901421a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8878000f80 tx=0x7f8878004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f888d600640 1 -- 172.21.15.141:0/4061630786 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8878009510 con 0x7f8890143ae0 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f888d600640 1 -- 172.21.15.141:0/4061630786 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887803e450 con 0x7f8890143ae0 2024-01-16T15:02:15.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f888d600640 1 -- 172.21.15.141:0/4061630786 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887803e800 con 0x7f8890143ae0 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8890162060 con 0x7f8890143ae0 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.766+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8890161400 con 0x7f8890143ae0 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.767+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890161590 con 0x7f8890143ae0 2024-01-16T15:02:15.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.768+0000 7f888d600640 1 -- 172.21.15.141:0/4061630786 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f887803e960 con 0x7f8890143ae0 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.769+0000 7f888d600640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f886806b860 0x7f886806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.769+0000 7f888d600640 1 -- 172.21.15.141:0/4061630786 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f88780ad170 con 0x7f8890143ae0 2024-01-16T15:02:15.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.770+0000 7f8895600640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f886806b860 0x7f886806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:15.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.771+0000 7f8895600640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f886806b860 0x7f886806dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f888000b7e0 tx=0x7f8880008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:15.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.774+0000 7f888d600640 1 -- 172.21.15.141:0/4061630786 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8878077b90 con 0x7f8890143ae0 2024-01-16T15:02:15.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.971+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8890162520 con 0x7f8890143ae0 2024-01-16T15:02:15.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.971+0000 7f888d600640 1 -- 172.21.15.141:0/4061630786 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v51) v1 ==== 80+0+124 (secure 0 0 0) 0x7f887807b1b0 con 0x7f8890143ae0 2024-01-16T15:02:15.952 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/2933218811 2024-01-17T15:02:11.334365+0000 2024-01-16T15:02:15.952 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/2933218811 2024-01-17T15:02:11.334365+0000 2024-01-16T15:02:15.952 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:02:15.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.975+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f886806b860 msgr2=0x7f886806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.975+0000 7f8898000640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f886806b860 0x7f886806dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f888000b7e0 tx=0x7f8880008040 comp rx=0 tx=0).stop 2024-01-16T15:02:15.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.975+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8890143ae0 msgr2=0x7f88901421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.975+0000 7f8898000640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8890143ae0 0x7f88901421a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8878000f80 tx=0x7f8878004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:15.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 shutdown_connections 2024-01-16T15:02:15.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f886806b860 0x7f886806dd20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f889015e1e0 0x7f88900af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8890143ae0 0x7f88901421a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 --2- 172.21.15.141:0/4061630786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88900cf7d0 0x7f8890160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 >> 172.21.15.141:0/4061630786 conn(0x7f889006a800 msgr2=0x7f889006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:15.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 shutdown_connections 2024-01-16T15:02:15.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:15.977+0000 7f8898000640 1 -- 172.21.15.141:0/4061630786 wait complete. 2024-01-16T15:02:15.986 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T15:02:15.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:15.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:16.097 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:02:16.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:16.097 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:02:16.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[52836]: fuse finished with error 0 and tester_r 0 2024-01-16T15:02:16.333 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:02:16.333 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:16.369 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:16.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.556+0000 7f7060800640 1 Processor -- start 2024-01-16T15:02:16.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.556+0000 7f7060800640 1 -- start start 2024-01-16T15:02:16.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.558+0000 7f7060800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f705815e3b0 0x7f7058164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.558+0000 7f7060800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580b1ee0 con 0x7f70580cf9a0 2024-01-16T15:02:16.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.558+0000 7f7060800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580b2780 con 0x7f7058143ae0 2024-01-16T15:02:16.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.558+0000 7f7060800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705815e990 con 0x7f705815e3b0 2024-01-16T15:02:16.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.558+0000 7f705d400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f70580cf9a0 0x7f705815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58874/0 (socket says 172.21.15.141:58874) 2024-01-16T15:02:16.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.558+0000 7f705d400640 1 -- 172.21.15.141:0/885810114 learned_addr learned my addr 172.21.15.141:0/885810114 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.558+0000 7f705e800640 1 --2- 172.21.15.141:0/885810114 >> v2:172.21.15.188:3300/0 conn(0x7f705815e3b0 0x7f7058164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879435681 0 0) 0x7f70580b1ee0 con 0x7f70580cf9a0 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7038003520 con 0x7f70580cf9a0 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 972686756 0 0) 0x7f7038003520 con 0x7f70580cf9a0 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 >> v2:172.21.15.188:3300/0 conn(0x7f705815e3b0 msgr2=0x7f7058164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 --2- 172.21.15.141:0/885810114 >> v2:172.21.15.188:3300/0 conn(0x7f705815e3b0 0x7f7058164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 >> v1:172.21.15.188:6789/0 conn(0x7f7058143ae0 legacy=0x7f70580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70580b2130 con 0x7f70580cf9a0 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7040001670 con 0x7f70580cf9a0 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 >> v1:172.21.15.141:6789/0 conn(0x7f70580cf9a0 legacy=0x7f705815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70380039c0 0x7f7038003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70380047c0 0x7f7038004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7038005180 0x7f703800fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705815e990 con 0x7f70380039c0 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038010490 con 0x7f7038005180 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7038010650 con 0x7f70380047c0 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.559+0000 7f705de00640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70380039c0 0x7f7038003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.560+0000 7f705de00640 1 -- 172.21.15.141:0/885810114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70380047c0 msgr2=0x7f7038004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.560+0000 7f705de00640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70380047c0 0x7f7038004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.560+0000 7f705de00640 1 -- 172.21.15.141:0/885810114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7038005180 msgr2=0x7f703800fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.560+0000 7f705de00640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7038005180 0x7f703800fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.560+0000 7f705de00640 1 -- 172.21.15.141:0/885810114 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038003520 con 0x7f70380039c0 2024-01-16T15:02:16.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.560+0000 7f705e800640 1 -- 172.21.15.141:0/885810114 reap_dead start 2024-01-16T15:02:16.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.561+0000 7f705de00640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70380039c0 0x7f7038003de0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7048001e50 tx=0x7f70380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bca0ff60e9535eb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.561+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048003b70 con 0x7f70380039c0 2024-01-16T15:02:16.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.561+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7048003cd0 con 0x7f70380039c0 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.561+0000 7f705ca00640 1 -- 172.21.15.141:0/885810114 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048005310 con 0x7f70380039c0 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.562+0000 7f7060800640 1 -- 172.21.15.141:0/885810114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70380039c0 msgr2=0x7f7038003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.562+0000 7f7060800640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70380039c0 0x7f7038003de0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7048001e50 tx=0x7f70380057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.562+0000 7f7060800640 1 -- 172.21.15.141:0/885810114 shutdown_connections 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.562+0000 7f7060800640 1 --2- 172.21.15.141:0/885810114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70380039c0 0x7f7038003de0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.562+0000 7f7060800640 1 -- 172.21.15.141:0/885810114 >> 172.21.15.141:0/885810114 conn(0x7f705806a800 msgr2=0x7f705806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.562+0000 7f7060800640 1 -- 172.21.15.141:0/885810114 shutdown_connections 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.562+0000 7f7060800640 1 -- 172.21.15.141:0/885810114 wait complete. 2024-01-16T15:02:16.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.564+0000 7f7060800640 1 Processor -- start 2024-01-16T15:02:16.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.564+0000 7f7060800640 1 -- start start 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.565+0000 7f7060800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70380039c0 0x7f70581ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.565+0000 7f7060800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70580cd840 0x7f70580adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.565+0000 7f7060800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70581efbf0 0x7f705815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.565+0000 7f7060800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70581ef9f0 con 0x7f70580cd840 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.565+0000 7f7060800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70580a84c0 con 0x7f70380039c0 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.565+0000 7f7060800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70580a8760 con 0x7f70581efbf0 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70580cd840 0x7f70580adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705de00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70380039c0 0x7f70581ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70580cd840 0x7f70580adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33664/0 (socket says 172.21.15.141:33664) 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 -- 172.21.15.141:0/2385715313 learned_addr learned my addr 172.21.15.141:0/2385715313 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:16.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705e800640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70581efbf0 0x7f705815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 -- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70581efbf0 msgr2=0x7f705815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70581efbf0 0x7f705815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 -- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70380039c0 msgr2=0x7f70581ef4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70380039c0 0x7f70581ef4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 -- 172.21.15.141:0/2385715313 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70580a8a40 con 0x7f70580cd840 2024-01-16T15:02:16.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.566+0000 7f705d400640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70580cd840 0x7f70580adeb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f704000c730 tx=0x7f704000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.567+0000 7f7056000640 1 -- 172.21.15.141:0/2385715313 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70400072d0 con 0x7f70580cd840 2024-01-16T15:02:16.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.567+0000 7f7056000640 1 -- 172.21.15.141:0/2385715313 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7040007430 con 0x7f70580cd840 2024-01-16T15:02:16.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.567+0000 7f7056000640 1 -- 172.21.15.141:0/2385715313 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70400047e0 con 0x7f70580cd840 2024-01-16T15:02:16.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.567+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70580a8d20 con 0x7f70580cd840 2024-01-16T15:02:16.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.567+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70580a92e0 con 0x7f70580cd840 2024-01-16T15:02:16.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.568+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70580675d0 con 0x7f70580cd840 2024-01-16T15:02:16.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.569+0000 7f7056000640 1 -- 172.21.15.141:0/2385715313 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7040004940 con 0x7f70580cd840 2024-01-16T15:02:16.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.569+0000 7f7056000640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f703006b860 0x7f703006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.570+0000 7f7056000640 1 -- 172.21.15.141:0/2385715313 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7040089720 con 0x7f70580cd840 2024-01-16T15:02:16.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.570+0000 7f705de00640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f703006b860 0x7f703006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.571+0000 7f705de00640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f703006b860 0x7f703006dd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7048034000 tx=0x7f704803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.574+0000 7f7056000640 1 -- 172.21.15.141:0/2385715313 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7040054090 con 0x7f70580cd840 2024-01-16T15:02:16.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.752+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70580a94a0 con 0x7f70580cd840 2024-01-16T15:02:16.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.754+0000 7f7056000640 1 -- 172.21.15.141:0/2385715313 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f7040018090 con 0x7f70580cd840 2024-01-16T15:02:16.734 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:16.734 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":51,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:02:14.349088+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/2933218811":"2024-01-17T15:02:11.334365+0000","172.21.15.141:6834/2933218811":"2024-01-17T15:02:11.334365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:16.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.756+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f703006b860 msgr2=0x7f703006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.756+0000 7f7060800640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f703006b860 0x7f703006dd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7048034000 tx=0x7f704803b040 comp rx=0 tx=0).stop 2024-01-16T15:02:16.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70580cd840 msgr2=0x7f70580adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70580cd840 0x7f70580adeb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f704000c730 tx=0x7f704000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 shutdown_connections 2024-01-16T15:02:16.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f703006b860 0x7f703006dd20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f70581efbf0 0x7f705815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70580cd840 0x7f70580adeb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 --2- 172.21.15.141:0/2385715313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70380039c0 0x7f70581ef4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 >> 172.21.15.141:0/2385715313 conn(0x7f705806a800 msgr2=0x7f705815d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 shutdown_connections 2024-01-16T15:02:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.757+0000 7f7060800640 1 -- 172.21.15.141:0/2385715313 wait complete. 2024-01-16T15:02:16.754 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:16.754 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:16.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.982+0000 7f1e16800640 1 Processor -- start 2024-01-16T15:02:16.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.982+0000 7f1e16800640 1 -- start start 2024-01-16T15:02:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.984+0000 7f1e16800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1e1013cf10 0x7f1e1013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.984+0000 7f1e16800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e100ce440 con 0x7f1e1013fae0 2024-01-16T15:02:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.984+0000 7f1e16800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e100ce720 con 0x7f1e100cddf0 2024-01-16T15:02:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.984+0000 7f1e16800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1013ff90 con 0x7f1e1013cf10 2024-01-16T15:02:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.984+0000 7f1e15400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1e1013cf10 0x7f1e1013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.984+0000 7f1e14a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1e1013fae0 0x7f1e100cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58876/0 (socket says 172.21.15.141:58876) 2024-01-16T15:02:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.984+0000 7f1e14a00640 1 -- 172.21.15.141:0/2353952386 learned_addr learned my addr 172.21.15.141:0/2353952386 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:16.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.985+0000 7f1e0fe00640 1 -- 172.21.15.141:0/2353952386 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165200093 0 0) 0x7f1e100ce440 con 0x7f1e1013fae0 2024-01-16T15:02:16.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.985+0000 7f1e15400640 1 -- 172.21.15.141:0/2353952386 >> v1:172.21.15.188:6789/0 conn(0x7f1e100cddf0 legacy=0x7f1e100b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:16.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.985+0000 7f1e15400640 1 -- 172.21.15.141:0/2353952386 >> v1:172.21.15.141:6789/0 conn(0x7f1e1013fae0 legacy=0x7f1e100cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.985+0000 7f1e15400640 1 -- 172.21.15.141:0/2353952386 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e100cea00 con 0x7f1e1013cf10 2024-01-16T15:02:16.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.986+0000 7f1e15400640 1 --2- 172.21.15.141:0/2353952386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e1013f5a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097a0 tx=0x7f1e0002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd83c915b826610 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.986+0000 7f1e0fe00640 1 -- 172.21.15.141:0/2353952386 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0002f8a0 con 0x7f1e1013cf10 2024-01-16T15:02:16.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.986+0000 7f1e0fe00640 1 -- 172.21.15.141:0/2353952386 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0002fa00 con 0x7f1e1013cf10 2024-01-16T15:02:16.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.986+0000 7f1e0fe00640 1 -- 172.21.15.141:0/2353952386 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0002fd90 con 0x7f1e1013cf10 2024-01-16T15:02:16.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.987+0000 7f1e16800640 1 -- 172.21.15.141:0/2353952386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 msgr2=0x7f1e1013f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.987+0000 7f1e16800640 1 --2- 172.21.15.141:0/2353952386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e1013f5a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097a0 tx=0x7f1e0002dae0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.987+0000 7f1e16800640 1 -- 172.21.15.141:0/2353952386 shutdown_connections 2024-01-16T15:02:16.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.987+0000 7f1e16800640 1 --2- 172.21.15.141:0/2353952386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e1013f5a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.987+0000 7f1e16800640 1 -- 172.21.15.141:0/2353952386 >> 172.21.15.141:0/2353952386 conn(0x7f1e1006a7d0 msgr2=0x7f1e1006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.987+0000 7f1e16800640 1 -- 172.21.15.141:0/2353952386 shutdown_connections 2024-01-16T15:02:16.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.987+0000 7f1e16800640 1 -- 172.21.15.141:0/2353952386 wait complete. 2024-01-16T15:02:16.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.989+0000 7f1e16800640 1 Processor -- start 2024-01-16T15:02:16.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.989+0000 7f1e16800640 1 -- start start 2024-01-16T15:02:16.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e16800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e100cddf0 0x7f1e101f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e16800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e10151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e16800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e1013fae0 0x7f1e10152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e16800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101f7330 con 0x7f1e1013cf10 2024-01-16T15:02:16.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e16800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e10135e50 con 0x7f1e100cddf0 2024-01-16T15:02:16.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e16800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e100ae5e0 con 0x7f1e1013fae0 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e14a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e10151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e14a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e10151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33680/0 (socket says 172.21.15.141:33680) 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e14a00640 1 -- 172.21.15.141:0/3792584907 learned_addr learned my addr 172.21.15.141:0/3792584907 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e14a00640 1 -- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e1013fae0 msgr2=0x7f1e10152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e14a00640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e1013fae0 0x7f1e10152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e14a00640 1 -- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e100cddf0 msgr2=0x7f1e101f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.990+0000 7f1e15400640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e100cddf0 0x7f1e101f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e14a00640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e100cddf0 0x7f1e101f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e14a00640 1 -- 172.21.15.141:0/3792584907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00009400 con 0x7f1e1013cf10 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e14a00640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e10151b00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1df800c430 tx=0x7f1df800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e0d600640 1 -- 172.21.15.141:0/3792584907 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df80074e0 con 0x7f1e1013cf10 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e0d600640 1 -- 172.21.15.141:0/3792584907 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df8007660 con 0x7f1e1013cf10 2024-01-16T15:02:16.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e101f6990 con 0x7f1e1013cf10 2024-01-16T15:02:16.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e0d600640 1 -- 172.21.15.141:0/3792584907 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8004a20 con 0x7f1e1013cf10 2024-01-16T15:02:16.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.991+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e101f6f50 con 0x7f1e1013cf10 2024-01-16T15:02:16.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.992+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e101f7470 con 0x7f1e1013cf10 2024-01-16T15:02:16.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.993+0000 7f1e0d600640 1 -- 172.21.15.141:0/3792584907 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1df8004b80 con 0x7f1e1013cf10 2024-01-16T15:02:16.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.994+0000 7f1e0d600640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1de806b860 0x7f1de806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.994+0000 7f1e0d600640 1 -- 172.21.15.141:0/3792584907 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1df808a210 con 0x7f1e1013cf10 2024-01-16T15:02:16.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.995+0000 7f1e15400640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1de806b860 0x7f1de806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.996+0000 7f1e15400640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1de806b860 0x7f1de806dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1e00038000 tx=0x7f1e00008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:16.998+0000 7f1e0d600640 1 -- 172.21.15.141:0/3792584907 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e101f7470 con 0x7f1e1013cf10 2024-01-16T15:02:17.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.174+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e101f7470 con 0x7f1e1013cf10 2024-01-16T15:02:17.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.175+0000 7f1e0d600640 1 -- 172.21.15.141:0/3792584907 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e101f7470 con 0x7f1e1013cf10 2024-01-16T15:02:17.155 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:02:17.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1de806b860 msgr2=0x7f1de806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1de806b860 0x7f1de806dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1e00038000 tx=0x7f1e00008030 comp rx=0 tx=0).stop 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 msgr2=0x7f1e10151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e10151b00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1df800c430 tx=0x7f1df800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 shutdown_connections 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1de806b860 0x7f1de806dd20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e1013fae0 0x7f1e10152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e1013cf10 0x7f1e10151b00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 --2- 172.21.15.141:0/3792584907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e100cddf0 0x7f1e101f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.177+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 >> 172.21.15.141:0/3792584907 conn(0x7f1e1006a7d0 msgr2=0x7f1e100b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.178+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 shutdown_connections 2024-01-16T15:02:17.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.178+0000 7f1e16800640 1 -- 172.21.15.141:0/3792584907 wait complete. 2024-01-16T15:02:17.176 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:02:17.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.402+0000 7f12be600640 1 Processor -- start 2024-01-16T15:02:17.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.402+0000 7f12be600640 1 -- start start 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12be600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f12b813df80 0x7f12b815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12be600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b80cf9a0 0x7f12b80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12be600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b806c470 con 0x7f12b813e360 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12be600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b806c750 con 0x7f12b813df80 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12be600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b806ca30 con 0x7f12b80cf9a0 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f12b813df80 0x7f12b815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f12b813df80 0x7f12b815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33682/0 (socket says 172.21.15.141:33682) 2024-01-16T15:02:17.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7400640 1 -- 172.21.15.141:0/1089101056 learned_addr learned my addr 172.21.15.141:0/1089101056 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:17.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7400640 1 -- 172.21.15.141:0/1089101056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b80cf9a0 msgr2=0x7f12b80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7e00640 1 --2- 172.21.15.141:0/1089101056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b80cf9a0 0x7f12b80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7400640 1 --2- 172.21.15.141:0/1089101056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b80cf9a0 0x7f12b80cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7400640 1 -- 172.21.15.141:0/1089101056 >> v1:172.21.15.188:6789/0 conn(0x7f12b813e360 legacy=0x7f12b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.404+0000 7f12b7400640 1 -- 172.21.15.141:0/1089101056 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b806cd10 con 0x7f12b813df80 2024-01-16T15:02:17.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.405+0000 7f12b7400640 1 --2- 172.21.15.141:0/1089101056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b815bee0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009870 tx=0x7f12a802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c518fddcc69e29 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12b6000640 1 -- 172.21.15.141:0/1089101056 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8031640 con 0x7f12b813df80 2024-01-16T15:02:17.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12b6000640 1 -- 172.21.15.141:0/1089101056 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a80317a0 con 0x7f12b813df80 2024-01-16T15:02:17.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12b6000640 1 -- 172.21.15.141:0/1089101056 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8031b30 con 0x7f12b813df80 2024-01-16T15:02:17.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12be600640 1 -- 172.21.15.141:0/1089101056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 msgr2=0x7f12b815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12be600640 1 --2- 172.21.15.141:0/1089101056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b815bee0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009870 tx=0x7f12a802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12be600640 1 -- 172.21.15.141:0/1089101056 shutdown_connections 2024-01-16T15:02:17.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12be600640 1 --2- 172.21.15.141:0/1089101056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b80cf9a0 0x7f12b80cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12be600640 1 --2- 172.21.15.141:0/1089101056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b815bee0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.406+0000 7f12be600640 1 -- 172.21.15.141:0/1089101056 >> 172.21.15.141:0/1089101056 conn(0x7f12b806a800 msgr2=0x7f12b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.407+0000 7f12be600640 1 -- 172.21.15.141:0/1089101056 shutdown_connections 2024-01-16T15:02:17.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.407+0000 7f12be600640 1 -- 172.21.15.141:0/1089101056 wait complete. 2024-01-16T15:02:17.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.408+0000 7f12be600640 1 Processor -- start 2024-01-16T15:02:17.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.408+0000 7f12be600640 1 -- start start 2024-01-16T15:02:17.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.409+0000 7f12be600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12b80cf9a0 0x7f12b815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.409+0000 7f12be600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b8152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.409+0000 7f12be600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b813e360 0x7f12b8152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.409+0000 7f12be600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8156180 con 0x7f12b813df80 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.409+0000 7f12be600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b81f3f20 con 0x7f12b80cf9a0 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.409+0000 7f12be600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b81f4200 con 0x7f12b813e360 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b8152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b8152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33692/0 (socket says 172.21.15.141:33692) 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 -- 172.21.15.141:0/122788207 learned_addr learned my addr 172.21.15.141:0/122788207 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b7e00640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b813e360 0x7f12b8152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 -- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b813e360 msgr2=0x7f12b8152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b813e360 0x7f12b8152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 -- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12b80cf9a0 msgr2=0x7f12b815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b7400640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12b80cf9a0 0x7f12b815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12b80cf9a0 0x7f12b815cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 -- 172.21.15.141:0/122788207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80094d0 con 0x7f12b813df80 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.410+0000 7f12b6a00640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b8152880 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f12a000c400 tx=0x7f12a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.411+0000 7f12af400640 1 -- 172.21.15.141:0/122788207 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0011790 con 0x7f12b813df80 2024-01-16T15:02:17.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.411+0000 7f12af400640 1 -- 172.21.15.141:0/122788207 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a0011930 con 0x7f12b813df80 2024-01-16T15:02:17.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.411+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b8155e10 con 0x7f12b813df80 2024-01-16T15:02:17.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.411+0000 7f12af400640 1 -- 172.21.15.141:0/122788207 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0011ce0 con 0x7f12b813df80 2024-01-16T15:02:17.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.411+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b815d000 con 0x7f12b813df80 2024-01-16T15:02:17.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.412+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b815d1f0 con 0x7f12b813df80 2024-01-16T15:02:17.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.413+0000 7f12af400640 1 -- 172.21.15.141:0/122788207 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f12a001c460 con 0x7f12b813df80 2024-01-16T15:02:17.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.413+0000 7f12af400640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129006b890 0x7f129006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.414+0000 7f12af400640 1 -- 172.21.15.141:0/122788207 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f12a008a2a0 con 0x7f12b813df80 2024-01-16T15:02:17.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.414+0000 7f12b7400640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129006b890 0x7f129006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.415+0000 7f12b7400640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129006b890 0x7f129006dd50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f12a8036000 tx=0x7f12a803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.418+0000 7f12af400640 1 -- 172.21.15.141:0/122788207 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12a0054b60 con 0x7f12b813df80 2024-01-16T15:02:17.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.614+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f12b80675d0 con 0x7f12b813df80 2024-01-16T15:02:17.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.615+0000 7f12af400640 1 -- 172.21.15.141:0/122788207 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v51) v1 ==== 80+0+124 (secure 0 0 0) 0x7f12a0058180 con 0x7f12b813df80 2024-01-16T15:02:17.594 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/2933218811 2024-01-17T15:02:11.334365+0000 2024-01-16T15:02:17.595 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/2933218811 2024-01-17T15:02:11.334365+0000 2024-01-16T15:02:17.595 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:02:17.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.617+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129006b890 msgr2=0x7f129006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.617+0000 7f12be600640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129006b890 0x7f129006dd50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f12a8036000 tx=0x7f12a803c040 comp rx=0 tx=0).stop 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.617+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 msgr2=0x7f12b8152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.617+0000 7f12be600640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b8152880 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f12a000c400 tx=0x7f12a000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 shutdown_connections 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129006b890 0x7f129006dd50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12b813e360 0x7f12b8152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12b813df80 0x7f12b8152880 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 --2- 172.21.15.141:0/122788207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12b80cf9a0 0x7f12b815cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 >> 172.21.15.141:0/122788207 conn(0x7f12b806a800 msgr2=0x7f12b80afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 shutdown_connections 2024-01-16T15:02:17.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:17.618+0000 7f12be600640 1 -- 172.21.15.141:0/122788207 wait complete. 2024-01-16T15:02:17.615 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T15:02:17.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:17.615 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:17.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[43380]: fuse finished with error 0 and tester_r 0 2024-01-16T15:02:17.711 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:02:17.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:17.711 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:02:17.892 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:02:17.892 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:17.946 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:17.946 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:02:17.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:17.947 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:02:17.997 INFO:teuthology.orchestra.run.smithi141.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:02:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:02:17.998 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:02:17.998 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:18.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:18.054 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:02:18.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:18.055 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:02:18.108 INFO:teuthology.orchestra.run.smithi188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:02:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:02:18.109 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:18.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.342+0000 7f4b7e400640 1 Processor -- start 2024-01-16T15:02:18.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.343+0000 7f4b7e400640 1 -- start start 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813df80 0x7f4b7815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b7e400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4b7813e360 0x7f4b780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b7e400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b7815f9c0 0x7f4b780cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b7e400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7806c470 con 0x7f4b7815f9c0 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b7e400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7806c750 con 0x7f4b7813e360 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b7e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7806ca30 con 0x7f4b7813df80 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b77e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b7815f9c0 0x7f4b780cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b77e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b7815f9c0 0x7f4b780cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33702/0 (socket says 172.21.15.141:33702) 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.344+0000 7f4b77e00640 1 -- 172.21.15.141:0/2252707771 learned_addr learned my addr 172.21.15.141:0/2252707771 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.345+0000 7f4b77400640 1 --2- 172.21.15.141:0/2252707771 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813df80 0x7f4b7815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.345+0000 7f4b76a00640 1 --2- 172.21.15.141:0/2252707771 >> v2:172.21.15.188:3300/0 conn(0x7f4b7813e360 0x7f4b780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.345+0000 7f4b77e00640 1 -- 172.21.15.141:0/2252707771 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813df80 msgr2=0x7f4b7815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.345+0000 7f4b77e00640 1 --2- 172.21.15.141:0/2252707771 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813df80 0x7f4b7815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.345+0000 7f4b77e00640 1 -- 172.21.15.141:0/2252707771 >> v2:172.21.15.188:3300/0 conn(0x7f4b7813e360 msgr2=0x7f4b780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.345+0000 7f4b77e00640 1 --2- 172.21.15.141:0/2252707771 >> v2:172.21.15.188:3300/0 conn(0x7f4b7813e360 0x7f4b780cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.345+0000 7f4b77e00640 1 -- 172.21.15.141:0/2252707771 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7806cd10 con 0x7f4b7815f9c0 2024-01-16T15:02:18.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.346+0000 7f4b77e00640 1 --2- 172.21.15.141:0/2252707771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7815f9c0 0x7f4b780cf7d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4b780cfed0 tx=0x7f4b7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c732a1407d3bacc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b76000640 1 -- 172.21.15.141:0/2252707771 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7002ca70 con 0x7f4b7815f9c0 2024-01-16T15:02:18.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b76000640 1 -- 172.21.15.141:0/2252707771 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7002cbd0 con 0x7f4b7815f9c0 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b76000640 1 -- 172.21.15.141:0/2252707771 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b700343f0 con 0x7f4b7815f9c0 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 -- 172.21.15.141:0/2252707771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7815f9c0 msgr2=0x7f4b780cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 --2- 172.21.15.141:0/2252707771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7815f9c0 0x7f4b780cf7d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4b780cfed0 tx=0x7f4b7002bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 -- 172.21.15.141:0/2252707771 shutdown_connections 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 --2- 172.21.15.141:0/2252707771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7815f9c0 0x7f4b780cf7d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 --2- 172.21.15.141:0/2252707771 >> v2:172.21.15.188:3300/0 conn(0x7f4b7813e360 0x7f4b780cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 --2- 172.21.15.141:0/2252707771 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813df80 0x7f4b7815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 -- 172.21.15.141:0/2252707771 >> 172.21.15.141:0/2252707771 conn(0x7f4b7806a800 msgr2=0x7f4b7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 -- 172.21.15.141:0/2252707771 shutdown_connections 2024-01-16T15:02:18.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.347+0000 7f4b7e400640 1 -- 172.21.15.141:0/2252707771 wait complete. 2024-01-16T15:02:18.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.349+0000 7f4b7e400640 1 Processor -- start 2024-01-16T15:02:18.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.349+0000 7f4b7e400640 1 -- start start 2024-01-16T15:02:18.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.350+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b7813df80 0x7f4b7813eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.350+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813e360 0x7f4b78158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.350+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7815f9c0 0x7f4b7813ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.350+0000 7f4b7e400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780b2370 con 0x7f4b7813e360 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.350+0000 7f4b7e400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780b25c0 con 0x7f4b7813df80 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.350+0000 7f4b7e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780b28a0 con 0x7f4b7815f9c0 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813e360 0x7f4b78158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813e360 0x7f4b78158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33710/0 (socket says 172.21.15.141:33710) 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 -- 172.21.15.141:0/3449931386 learned_addr learned my addr 172.21.15.141:0/3449931386 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b77400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b7813df80 0x7f4b7813eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b77e00640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7815f9c0 0x7f4b7813ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 -- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7815f9c0 msgr2=0x7f4b7813ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7815f9c0 0x7f4b7813ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 -- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b7813df80 msgr2=0x7f4b7813eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b7813df80 0x7f4b7813eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 -- 172.21.15.141:0/3449931386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70009b30 con 0x7f4b7813e360 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b77400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b7813df80 0x7f4b7813eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b76a00640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813e360 0x7f4b78158650 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000d800 tx=0x7f4b6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.351+0000 7f4b6f400640 1 -- 172.21.15.141:0/3449931386 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000b6c0 con 0x7f4b7813e360 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.352+0000 7f4b6f400640 1 -- 172.21.15.141:0/3449931386 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6000b820 con 0x7f4b7813e360 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.352+0000 7f4b6f400640 1 -- 172.21.15.141:0/3449931386 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000bbb0 con 0x7f4b7813e360 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.352+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b780b2b80 con 0x7f4b7813e360 2024-01-16T15:02:18.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.352+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b780b3140 con 0x7f4b7813e360 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.353+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b780b3300 con 0x7f4b7813e360 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.354+0000 7f4b6f400640 1 -- 172.21.15.141:0/3449931386 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f4b6000bd10 con 0x7f4b7813e360 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.355+0000 7f4b6f400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006b860 0x7f4b5006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.355+0000 7f4b6f400640 1 -- 172.21.15.141:0/3449931386 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4b6008a880 con 0x7f4b7813e360 2024-01-16T15:02:18.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.355+0000 7f4b77400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006b860 0x7f4b5006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.357+0000 7f4b77400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006b860 0x7f4b5006dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006fd0 tx=0x7f4b68008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.360+0000 7f4b6f400640 1 -- 172.21.15.141:0/3449931386 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b60055180 con 0x7f4b7813e360 2024-01-16T15:02:18.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.586+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b780675d0 con 0x7f4b7813e360 2024-01-16T15:02:18.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.588+0000 7f4b6f400640 1 -- 172.21.15.141:0/3449931386 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f4b600587a0 con 0x7f4b7813e360 2024-01-16T15:02:18.568 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:18.568 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2118095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2118095418},{"type":"v1","addr":"172.21.15.188:6837","nonce":2118095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/3680808697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3680808697},{"type":"v1","addr":"172.21.15.188:6836","nonce":3680808697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5793,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/275409001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":275409001},{"type":"v1","addr":"172.21.15.141:6833","nonce":275409001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3757287180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3757287180},{"type":"v1","addr":"172.21.15.141:6835","nonce":3757287180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-01-16T15:02:18.568 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 30 2024-01-16T15:02:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006b860 msgr2=0x7f4b5006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006b860 0x7f4b5006dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006fd0 tx=0x7f4b68008040 comp rx=0 tx=0).stop 2024-01-16T15:02:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813e360 msgr2=0x7f4b78158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813e360 0x7f4b78158650 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000d800 tx=0x7f4b6000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 shutdown_connections 2024-01-16T15:02:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006b860 0x7f4b5006dd20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7815f9c0 0x7f4b7813ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813e360 0x7f4b78158650 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 --2- 172.21.15.141:0/3449931386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b7813df80 0x7f4b7813eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.590+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 >> 172.21.15.141:0/3449931386 conn(0x7f4b7806a800 msgr2=0x7f4b7806d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.591+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 shutdown_connections 2024-01-16T15:02:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.591+0000 7f4b7e400640 1 -- 172.21.15.141:0/3449931386 wait complete. 2024-01-16T15:02:18.590 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:02:18.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.807+0000 7fab7c800640 1 Processor -- start 2024-01-16T15:02:18.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.807+0000 7fab7c800640 1 -- start start 2024-01-16T15:02:18.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab7c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 0x7fab74164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab7c800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab740b1ee0 con 0x7fab740cf9a0 2024-01-16T15:02:18.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab7c800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab740b2780 con 0x7fab74143ae0 2024-01-16T15:02:18.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab7c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7415e990 con 0x7fab7415e3b0 2024-01-16T15:02:18.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab79400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fab740cf9a0 0x7fab7415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58886/0 (socket says 172.21.15.141:58886) 2024-01-16T15:02:18.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab79400640 1 -- 172.21.15.141:0/1736690294 learned_addr learned my addr 172.21.15.141:0/1736690294 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:18.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab7a800640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 0x7fab74164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629998671 0 0) 0x7fab740b1ee0 con 0x7fab740cf9a0 2024-01-16T15:02:18.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.809+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab54003520 con 0x7fab740cf9a0 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223648241 0 0) 0x7fab54003520 con 0x7fab740cf9a0 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 msgr2=0x7fab74164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 0x7fab74164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 >> v1:172.21.15.188:6789/0 conn(0x7fab74143ae0 legacy=0x7fab740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab740b2130 con 0x7fab740cf9a0 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fab5c001670 con 0x7fab740cf9a0 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 >> v1:172.21.15.141:6789/0 conn(0x7fab740cf9a0 legacy=0x7fab7415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab540039a0 0x7fab54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 0x7fab540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab54004d00 0x7fab54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7415e990 con 0x7fab54004d00 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5400bb90 con 0x7fab540039a0 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5400bd50 con 0x7fab7415e3b0 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab7a800640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab54004d00 0x7fab54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab79400640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 0x7fab540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.810+0000 7fab79e00640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab540039a0 0x7fab54003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.811+0000 7fab7a800640 1 -- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 msgr2=0x7fab540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.811+0000 7fab7a800640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415e3b0 0x7fab540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.811+0000 7fab7a800640 1 -- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab540039a0 msgr2=0x7fab54003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.811+0000 7fab7a800640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab540039a0 0x7fab54003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.811+0000 7fab7a800640 1 -- 172.21.15.141:0/1736690294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab54003520 con 0x7fab54004d00 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.812+0000 7fab7a800640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab54004d00 0x7fab54005180 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fab6c007360 tx=0x7fab540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa60dd13d1239e17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.812+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c02ccb0 con 0x7fab54004d00 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.812+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c02ce10 con 0x7fab54004d00 2024-01-16T15:02:18.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.812+0000 7fab78a00640 1 -- 172.21.15.141:0/1736690294 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c034660 con 0x7fab54004d00 2024-01-16T15:02:18.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.813+0000 7fab7c800640 1 -- 172.21.15.141:0/1736690294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab54004d00 msgr2=0x7fab54005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.813+0000 7fab7c800640 1 --2- 172.21.15.141:0/1736690294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab54004d00 0x7fab54005180 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fab6c007360 tx=0x7fab540057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.813+0000 7fab7a800640 1 -- 172.21.15.141:0/1736690294 reap_dead start 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.813+0000 7fab7c800640 1 -- 172.21.15.141:0/1736690294 shutdown_connections 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.813+0000 7fab7c800640 1 -- 172.21.15.141:0/1736690294 >> 172.21.15.141:0/1736690294 conn(0x7fab7406a800 msgr2=0x7fab7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.813+0000 7fab7c800640 1 -- 172.21.15.141:0/1736690294 shutdown_connections 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.813+0000 7fab7c800640 1 -- 172.21.15.141:0/1736690294 wait complete. 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.815+0000 7fab7c800640 1 Processor -- start 2024-01-16T15:02:18.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.815+0000 7fab7c800640 1 -- start start 2024-01-16T15:02:18.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab7c800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab54004d00 0x7fab740af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab7c800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab741421d0 0x7fab740b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab7c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415a8c0 0x7fab7415ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab7c800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab740b1880 con 0x7fab741421d0 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab7c800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7415b280 con 0x7fab54004d00 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab7c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab740b09d0 con 0x7fab7415a8c0 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab741421d0 0x7fab740b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab741421d0 0x7fab740b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33716/0 (socket says 172.21.15.141:33716) 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 -- 172.21.15.141:0/2822799305 learned_addr learned my addr 172.21.15.141:0/2822799305 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 -- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415a8c0 msgr2=0x7fab7415ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415a8c0 0x7fab7415ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 -- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab54004d00 msgr2=0x7fab740af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab54004d00 0x7fab740af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.816+0000 7fab79400640 1 -- 172.21.15.141:0/2822799305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab740b0c00 con 0x7fab741421d0 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.817+0000 7fab79400640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab741421d0 0x7fab740b0490 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00e790 tx=0x7fab5c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.817+0000 7fab72000640 1 -- 172.21.15.141:0/2822799305 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c00eea0 con 0x7fab741421d0 2024-01-16T15:02:18.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.817+0000 7fab72000640 1 -- 172.21.15.141:0/2822799305 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c00c850 con 0x7fab741421d0 2024-01-16T15:02:18.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.817+0000 7fab72000640 1 -- 172.21.15.141:0/2822799305 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c00cc00 con 0x7fab741421d0 2024-01-16T15:02:18.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.817+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab740b0ee0 con 0x7fab741421d0 2024-01-16T15:02:18.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.817+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab740b14a0 con 0x7fab741421d0 2024-01-16T15:02:18.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.818+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7406c3f0 con 0x7fab741421d0 2024-01-16T15:02:18.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.819+0000 7fab72000640 1 -- 172.21.15.141:0/2822799305 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fab5c013420 con 0x7fab741421d0 2024-01-16T15:02:18.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.820+0000 7fab72000640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab4c06b890 0x7fab4c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.820+0000 7fab72000640 1 -- 172.21.15.141:0/2822799305 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fab5c089f90 con 0x7fab741421d0 2024-01-16T15:02:18.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.820+0000 7fab79e00640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab4c06b890 0x7fab4c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.822+0000 7fab79e00640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab4c06b890 0x7fab4c06dd50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fab64007fd0 tx=0x7fab64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:18.824+0000 7fab72000640 1 -- 172.21.15.141:0/2822799305 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab5c0582e0 con 0x7fab741421d0 2024-01-16T15:02:18.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.013+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fab740675d0 con 0x7fab741421d0 2024-01-16T15:02:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.019+0000 7fab72000640 1 -- 172.21.15.141:0/2822799305 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v17) v1 ==== 115+0+0 (secure 0 0 0) 0x7fab5c001540 con 0x7fab741421d0 2024-01-16T15:02:19.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab4c06b890 msgr2=0x7fab4c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab4c06b890 0x7fab4c06dd50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fab64007fd0 tx=0x7fab64009040 comp rx=0 tx=0).stop 2024-01-16T15:02:19.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab741421d0 msgr2=0x7fab740b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab741421d0 0x7fab740b0490 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00e790 tx=0x7fab5c00ec50 comp rx=0 tx=0).stop 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 shutdown_connections 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab4c06b890 0x7fab4c06dd50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab7415a8c0 0x7fab7415ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab741421d0 0x7fab740b0490 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 --2- 172.21.15.141:0/2822799305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab54004d00 0x7fab740af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.021+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 >> 172.21.15.141:0/2822799305 conn(0x7fab7406a800 msgr2=0x7fab7415c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.022+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 shutdown_connections 2024-01-16T15:02:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.022+0000 7fab7c800640 1 -- 172.21.15.141:0/2822799305 wait complete. 2024-01-16T15:02:19.018 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:02:19.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.250+0000 7ff00ec00640 1 Processor -- start 2024-01-16T15:02:19.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.250+0000 7ff00ec00640 1 -- start start 2024-01-16T15:02:19.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.252+0000 7ff00ec00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff008144c60 0x7ff0080cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.252+0000 7ff00ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff00806a3e0 0x7ff00806a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.252+0000 7ff00ec00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff008071a20 con 0x7ff0080cf780 2024-01-16T15:02:19.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.252+0000 7ff00ec00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00806adb0 con 0x7ff008144c60 2024-01-16T15:02:19.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.252+0000 7ff00ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00806eef0 con 0x7ff00806a3e0 2024-01-16T15:02:19.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.252+0000 7ff007400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff0080cf780 0x7ff00806e7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58888/0 (socket says 172.21.15.141:58888) 2024-01-16T15:02:19.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.252+0000 7ff007400640 1 -- 172.21.15.141:0/1435507489 learned_addr learned my addr 172.21.15.141:0/1435507489 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:19.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff00cc00640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff00806a3e0 0x7ff00806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff007e00640 1 --2- 172.21.15.141:0/1435507489 >> v2:172.21.15.188:3300/0 conn(0x7ff008144c60 0x7ff0080cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893708150 0 0) 0x7ff008071a20 con 0x7ff0080cf780 2024-01-16T15:02:19.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe8003520 con 0x7ff0080cf780 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155059009 0 0) 0x7fefe8003520 con 0x7ff0080cf780 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff00806a3e0 msgr2=0x7ff00806a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff00806a3e0 0x7ff00806a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 >> v2:172.21.15.188:3300/0 conn(0x7ff008144c60 msgr2=0x7ff0080cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 --2- 172.21.15.141:0/1435507489 >> v2:172.21.15.188:3300/0 conn(0x7ff008144c60 0x7ff0080cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0080ccfd0 con 0x7ff0080cf780 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7feff0001670 con 0x7ff0080cf780 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 >> v1:172.21.15.141:6789/0 conn(0x7ff0080cf780 legacy=0x7ff00806e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe8003970 0x7fefe8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.253+0000 7ff006a00640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff00806a3e0 0x7fefe8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00806adb0 con 0x7fefe80047c0 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00806eef0 con 0x7fefe8003970 2024-01-16T15:02:19.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe800b980 con 0x7ff00806a3e0 2024-01-16T15:02:19.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff007400640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff007e00640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe8003970 0x7fefe8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff007400640 1 -- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff00806a3e0 msgr2=0x7fefe8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff007400640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff00806a3e0 0x7fefe8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff007400640 1 -- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe8003970 msgr2=0x7fefe8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff007400640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe8003970 0x7fefe8003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.254+0000 7ff007400640 1 -- 172.21.15.141:0/1435507489 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8003520 con 0x7fefe80047c0 2024-01-16T15:02:19.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.255+0000 7ff007400640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7feff000c240 tx=0x7fefe80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2f96285f9b8324 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff002cb00 con 0x7fefe80047c0 2024-01-16T15:02:19.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff002cc60 con 0x7fefe80047c0 2024-01-16T15:02:19.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff006a00640 1 -- 172.21.15.141:0/1435507489 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00058d0 con 0x7fefe80047c0 2024-01-16T15:02:19.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff00ec00640 1 -- 172.21.15.141:0/1435507489 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80047c0 msgr2=0x7fefe8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff00ec00640 1 --2- 172.21.15.141:0/1435507489 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7feff000c240 tx=0x7fefe80057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff00cc00640 1 -- 172.21.15.141:0/1435507489 reap_dead start 2024-01-16T15:02:19.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff00ec00640 1 -- 172.21.15.141:0/1435507489 shutdown_connections 2024-01-16T15:02:19.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff00ec00640 1 -- 172.21.15.141:0/1435507489 >> 172.21.15.141:0/1435507489 conn(0x7ff0080cfef0 msgr2=0x7ff0080d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff00ec00640 1 -- 172.21.15.141:0/1435507489 shutdown_connections 2024-01-16T15:02:19.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.256+0000 7ff00ec00640 1 -- 172.21.15.141:0/1435507489 wait complete. 2024-01-16T15:02:19.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.258+0000 7ff00ec00640 1 Processor -- start 2024-01-16T15:02:19.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.258+0000 7ff00ec00640 1 -- start start 2024-01-16T15:02:19.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff00ec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80047c0 0x7ff008141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff00ec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff008143af0 0x7ff00815a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff00ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff008142850 0x7ff008142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff00ec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080b4280 con 0x7ff008143af0 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff00ec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080b4560 con 0x7fefe80047c0 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff00ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080b4840 con 0x7ff008142850 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff007400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff008143af0 0x7ff00815a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff007400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff008143af0 0x7ff00815a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33734/0 (socket says 172.21.15.141:33734) 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.259+0000 7ff007400640 1 -- 172.21.15.141:0/489663794 learned_addr learned my addr 172.21.15.141:0/489663794 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007e00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80047c0 0x7ff008141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007400640 1 -- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff008142850 msgr2=0x7ff008142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff00cc00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff008142850 0x7ff008142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007400640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff008142850 0x7ff008142d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007400640 1 -- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80047c0 msgr2=0x7ff008141ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007400640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80047c0 0x7ff008141ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007400640 1 -- 172.21.15.141:0/489663794 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0081435d0 con 0x7ff008143af0 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007e00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80047c0 0x7ff008141ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7ff007400640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff008143af0 0x7ff00815a9f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7feff000c240 tx=0x7feff0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.260+0000 7fefffe00640 1 -- 172.21.15.141:0/489663794 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0039070 con 0x7ff008143af0 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.261+0000 7fefffe00640 1 -- 172.21.15.141:0/489663794 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0001630 con 0x7ff008143af0 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.261+0000 7fefffe00640 1 -- 172.21.15.141:0/489663794 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff003c040 con 0x7ff008143af0 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.261+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff008143240 con 0x7ff008143af0 2024-01-16T15:02:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.261+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff008144190 con 0x7ff008143af0 2024-01-16T15:02:19.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.262+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff008143760 con 0x7ff008143af0 2024-01-16T15:02:19.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.263+0000 7fefffe00640 1 -- 172.21.15.141:0/489663794 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7feff0032070 con 0x7ff008143af0 2024-01-16T15:02:19.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.263+0000 7fefffe00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefe006b860 0x7fefe006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.264+0000 7fefffe00640 1 -- 172.21.15.141:0/489663794 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feff00abac0 con 0x7ff008143af0 2024-01-16T15:02:19.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.264+0000 7ff007e00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefe006b860 0x7fefe006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.265+0000 7ff007e00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefe006b860 0x7fefe006dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7feff8007460 tx=0x7feff800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.268+0000 7fefffe00640 1 -- 172.21.15.141:0/489663794 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feff0076550 con 0x7ff008143af0 2024-01-16T15:02:19.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.462+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7ff008143430 con 0x7ff008143af0 2024-01-16T15:02:19.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.467+0000 7fefffe00640 1 -- 172.21.15.141:0/489663794 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v18) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff008143430 con 0x7ff008143af0 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.469+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefe006b860 msgr2=0x7fefe006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.469+0000 7ff00ec00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefe006b860 0x7fefe006dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7feff8007460 tx=0x7feff800b040 comp rx=0 tx=0).stop 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.469+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff008143af0 msgr2=0x7ff00815a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.469+0000 7ff00ec00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff008143af0 0x7ff00815a9f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7feff000c240 tx=0x7feff0001a00 comp rx=0 tx=0).stop 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 shutdown_connections 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fefe006b860 0x7fefe006dd20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff008142850 0x7ff008142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff008143af0 0x7ff00815a9f0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 --2- 172.21.15.141:0/489663794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefe80047c0 0x7ff008141ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 >> 172.21.15.141:0/489663794 conn(0x7ff0080cfef0 msgr2=0x7ff00815ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 shutdown_connections 2024-01-16T15:02:19.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.470+0000 7ff00ec00640 1 -- 172.21.15.141:0/489663794 wait complete. 2024-01-16T15:02:19.469 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:02:19.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.700+0000 7fdad4a00640 1 Processor -- start 2024-01-16T15:02:19.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.700+0000 7fdad4a00640 1 -- start start 2024-01-16T15:02:19.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad4a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdacc143ae0 0x7fdacc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad4a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdacc15e3b0 0x7fdacc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad4a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0cfc30 con 0x7fdacc0cf7d0 2024-01-16T15:02:19.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad4a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdacc0b1ee0 con 0x7fdacc15e3b0 2024-01-16T15:02:19.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad4a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdacc0b2780 con 0x7fdacc143ae0 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad1600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdacc0cf7d0 0x7fdacc15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58902/0 (socket says 172.21.15.141:58902) 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad1600640 1 -- 172.21.15.141:0/2988517429 learned_addr learned my addr 172.21.15.141:0/2988517429 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad2a00640 1 --2- 172.21.15.141:0/2988517429 >> v2:172.21.15.141:3300/0 conn(0x7fdacc15e3b0 0x7fdacc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad2000640 1 --2- 172.21.15.141:0/2988517429 >> v2:172.21.15.188:3300/0 conn(0x7fdacc143ae0 0x7fdacc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.702+0000 7fdad0c00640 1 -- 172.21.15.141:0/2988517429 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287727723 0 0) 0x7fdacc0cfc30 con 0x7fdacc0cf7d0 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.703+0000 7fdad0c00640 1 -- 172.21.15.141:0/2988517429 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaac003520 con 0x7fdacc0cf7d0 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.703+0000 7fdad2a00640 1 -- 172.21.15.141:0/2988517429 >> v2:172.21.15.188:3300/0 conn(0x7fdacc143ae0 msgr2=0x7fdacc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.703+0000 7fdad2a00640 1 --2- 172.21.15.141:0/2988517429 >> v2:172.21.15.188:3300/0 conn(0x7fdacc143ae0 0x7fdacc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.703+0000 7fdad2a00640 1 -- 172.21.15.141:0/2988517429 >> v1:172.21.15.141:6789/0 conn(0x7fdacc0cf7d0 legacy=0x7fdacc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.703+0000 7fdad2a00640 1 -- 172.21.15.141:0/2988517429 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc15e990 con 0x7fdacc15e3b0 2024-01-16T15:02:19.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.704+0000 7fdad2a00640 1 --2- 172.21.15.141:0/2988517429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc0702c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0cfea0 tx=0x7fdac402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6598cfda908c1f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.704+0000 7fdad0c00640 1 -- 172.21.15.141:0/2988517429 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac402cc80 con 0x7fdacc15e3b0 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.704+0000 7fdad0c00640 1 -- 172.21.15.141:0/2988517429 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac402cde0 con 0x7fdacc15e3b0 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.704+0000 7fdad0c00640 1 -- 172.21.15.141:0/2988517429 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4034660 con 0x7fdacc15e3b0 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 -- 172.21.15.141:0/2988517429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 msgr2=0x7fdacc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 --2- 172.21.15.141:0/2988517429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc0702c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0cfea0 tx=0x7fdac402bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 -- 172.21.15.141:0/2988517429 shutdown_connections 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 --2- 172.21.15.141:0/2988517429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc0702c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 --2- 172.21.15.141:0/2988517429 >> v2:172.21.15.188:3300/0 conn(0x7fdacc143ae0 0x7fdacc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 -- 172.21.15.141:0/2988517429 >> 172.21.15.141:0/2988517429 conn(0x7fdacc06a800 msgr2=0x7fdacc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 -- 172.21.15.141:0/2988517429 shutdown_connections 2024-01-16T15:02:19.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.705+0000 7fdad4a00640 1 -- 172.21.15.141:0/2988517429 wait complete. 2024-01-16T15:02:19.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.707+0000 7fdad4a00640 1 Processor -- start 2024-01-16T15:02:19.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.707+0000 7fdad4a00640 1 -- start start 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdacc0cf7d0 0x7fdacc13d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdacc143ae0 0x7fdacc15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc15aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad4a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc13d7a0 con 0x7fdacc15e3b0 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad4a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc13da80 con 0x7fdacc143ae0 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc13dd60 con 0x7fdacc0cf7d0 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc15aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc15aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33762/0 (socket says 172.21.15.141:33762) 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 -- 172.21.15.141:0/1675412539 learned_addr learned my addr 172.21.15.141:0/1675412539 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 -- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdacc0cf7d0 msgr2=0x7fdacc13d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdacc0cf7d0 0x7fdacc13d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 -- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdacc143ae0 msgr2=0x7fdacc15a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdacc143ae0 0x7fdacc15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.708+0000 7fdad2a00640 1 -- 172.21.15.141:0/1675412539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4009a70 con 0x7fdacc15e3b0 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.709+0000 7fdad2a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc15aea0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e80 tx=0x7fdac4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.709+0000 7fdaca000640 1 -- 172.21.15.141:0/1675412539 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4040650 con 0x7fdacc15e3b0 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.709+0000 7fdaca000640 1 -- 172.21.15.141:0/1675412539 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac40407b0 con 0x7fdacc15e3b0 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.709+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc13e040 con 0x7fdacc15e3b0 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.709+0000 7fdaca000640 1 -- 172.21.15.141:0/1675412539 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4040b60 con 0x7fdacc15e3b0 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.709+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc13e840 con 0x7fdacc15e3b0 2024-01-16T15:02:19.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.710+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc0675d0 con 0x7fdacc15e3b0 2024-01-16T15:02:19.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.711+0000 7fdaca000640 1 -- 172.21.15.141:0/1675412539 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fdac4040cc0 con 0x7fdacc15e3b0 2024-01-16T15:02:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.712+0000 7fdaca000640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdaa406b890 0x7fdaa406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.713+0000 7fdaca000640 1 -- 172.21.15.141:0/1675412539 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdac40acd00 con 0x7fdacc15e3b0 2024-01-16T15:02:19.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.713+0000 7fdad2000640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdaa406b890 0x7fdaa406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.714+0000 7fdad2000640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdaa406b890 0x7fdaa406dd50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdacc1f6240 tx=0x7fdabc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.717+0000 7fdaca000640 1 -- 172.21.15.141:0/1675412539 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdac4077690 con 0x7fdacc15e3b0 2024-01-16T15:02:19.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.910+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdacc13eec0 con 0x7fdacc15e3b0 2024-01-16T15:02:19.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.917+0000 7fdaca000640 1 -- 172.21.15.141:0/1675412539 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdac4009450 con 0x7fdacc15e3b0 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.919+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdaa406b890 msgr2=0x7fdaa406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdaa406b890 0x7fdaa406dd50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdacc1f6240 tx=0x7fdabc008040 comp rx=0 tx=0).stop 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 msgr2=0x7fdacc15aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc15aea0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e80 tx=0x7fdac4004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 shutdown_connections 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdaa406b890 0x7fdaa406dd50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdacc15e3b0 0x7fdacc15aea0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdacc143ae0 0x7fdacc15a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 --2- 172.21.15.141:0/1675412539 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdacc0cf7d0 0x7fdacc13d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 >> 172.21.15.141:0/1675412539 conn(0x7fdacc06a800 msgr2=0x7fdacc15b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 shutdown_connections 2024-01-16T15:02:19.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:19.920+0000 7fdad4a00640 1 -- 172.21.15.141:0/1675412539 wait complete. 2024-01-16T15:02:19.918 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T15:02:20.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.141+0000 7f7693800640 1 Processor -- start 2024-01-16T15:02:20.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.142+0000 7f7693800640 1 -- start start 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7693800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f768c0cc1e0 0x7f768c0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7693800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f768c0ae750 0x7f768c0ac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7693800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c162100 con 0x7f768c074520 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7693800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0ccf20 con 0x7f768c0cc1e0 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7693800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0cd200 con 0x7f768c0ae750 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7690e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f768c0cc1e0 0x7f768c0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7690e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f768c0cc1e0 0x7f768c0cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33770/0 (socket says 172.21.15.141:33770) 2024-01-16T15:02:20.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7690e00640 1 -- 172.21.15.141:0/388644213 learned_addr learned my addr 172.21.15.141:0/388644213 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:20.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7690e00640 1 -- 172.21.15.141:0/388644213 >> v2:172.21.15.188:3300/0 conn(0x7f768c0ae750 msgr2=0x7f768c0ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7690e00640 1 --2- 172.21.15.141:0/388644213 >> v2:172.21.15.188:3300/0 conn(0x7f768c0ae750 0x7f768c0ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.143+0000 7f7690e00640 1 -- 172.21.15.141:0/388644213 >> v1:172.21.15.188:6789/0 conn(0x7f768c074520 legacy=0x7f768c0ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.144+0000 7f7690e00640 1 -- 172.21.15.141:0/388644213 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c06daa0 con 0x7f768c0cc1e0 2024-01-16T15:02:20.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.145+0000 7f7690e00640 1 --2- 172.21.15.141:0/388644213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c0cc1e0 0x7f768c0cea50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f768c0716e0 tx=0x7f767c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb341ff9a977525 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.145+0000 7f768b400640 1 -- 172.21.15.141:0/388644213 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c031750 con 0x7f768c0cc1e0 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.145+0000 7f768b400640 1 -- 172.21.15.141:0/388644213 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c0318b0 con 0x7f768c0cc1e0 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.145+0000 7f768b400640 1 -- 172.21.15.141:0/388644213 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c031c40 con 0x7f768c0cc1e0 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.145+0000 7f7693800640 1 -- 172.21.15.141:0/388644213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c0cc1e0 msgr2=0x7f768c0cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.145+0000 7f7693800640 1 --2- 172.21.15.141:0/388644213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c0cc1e0 0x7f768c0cea50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f768c0716e0 tx=0x7f767c02b7f0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.146+0000 7f7693800640 1 -- 172.21.15.141:0/388644213 shutdown_connections 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.146+0000 7f7693800640 1 --2- 172.21.15.141:0/388644213 >> v2:172.21.15.188:3300/0 conn(0x7f768c0ae750 0x7f768c0ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.146+0000 7f7693800640 1 --2- 172.21.15.141:0/388644213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c0cc1e0 0x7f768c0cea50 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.146+0000 7f7693800640 1 -- 172.21.15.141:0/388644213 >> 172.21.15.141:0/388644213 conn(0x7f768c0b0d70 msgr2=0x7f768c0b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.146+0000 7f7693800640 1 -- 172.21.15.141:0/388644213 shutdown_connections 2024-01-16T15:02:20.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.146+0000 7f7693800640 1 -- 172.21.15.141:0/388644213 wait complete. 2024-01-16T15:02:20.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.147+0000 7f7693800640 1 Processor -- start 2024-01-16T15:02:20.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.148+0000 7f7693800640 1 -- start start 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.148+0000 7f7693800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c074520 0x7f768c0c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.148+0000 7f7693800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f768c0ae750 0x7f768c15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.148+0000 7f7693800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f768c0cc1e0 0x7f768c0b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.148+0000 7f7693800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0c8bd0 con 0x7f768c074520 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.148+0000 7f7693800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0c46f0 con 0x7f768c0ae750 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.148+0000 7f7693800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0c5ba0 con 0x7f768c0cc1e0 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c074520 0x7f768c0c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c074520 0x7f768c0c4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33782/0 (socket says 172.21.15.141:33782) 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 -- 172.21.15.141:0/2206528280 learned_addr learned my addr 172.21.15.141:0/2206528280 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7691800640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f768c0cc1e0 0x7f768c0b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f768be00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f768c0ae750 0x7f768c15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 -- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f768c0cc1e0 msgr2=0x7f768c0b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f768c0cc1e0 0x7f768c0b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 -- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f768c0ae750 msgr2=0x7f768c15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f768c0ae750 0x7f768c15a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f7690e00640 1 -- 172.21.15.141:0/2206528280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0094d0 con 0x7f768c074520 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.149+0000 7f768be00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f768c0ae750 0x7f768c15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.150+0000 7f7690e00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c074520 0x7f768c0c4b10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f767c000c00 tx=0x7f767c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.150+0000 7f7688c00640 1 -- 172.21.15.141:0/2206528280 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c033c60 con 0x7f768c074520 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.150+0000 7f7688c00640 1 -- 172.21.15.141:0/2206528280 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c033de0 con 0x7f768c074520 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.150+0000 7f7688c00640 1 -- 172.21.15.141:0/2206528280 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c03d6b0 con 0x7f768c074520 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.150+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c0c3ab0 con 0x7f768c074520 2024-01-16T15:02:20.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.150+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c0c4030 con 0x7f768c074520 2024-01-16T15:02:20.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.151+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c0675d0 con 0x7f768c074520 2024-01-16T15:02:20.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.153+0000 7f7688c00640 1 -- 172.21.15.141:0/2206528280 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f767c03a040 con 0x7f768c074520 2024-01-16T15:02:20.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.154+0000 7f7688c00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f766406b860 0x7f766406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.154+0000 7f7688c00640 1 -- 172.21.15.141:0/2206528280 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f767c0ac320 con 0x7f768c074520 2024-01-16T15:02:20.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.155+0000 7f768be00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f766406b860 0x7f766406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.156+0000 7f768be00640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f766406b860 0x7f766406dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f768c0b08c0 tx=0x7f7674009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.158+0000 7f7688c00640 1 -- 172.21.15.141:0/2206528280 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f767c076db0 con 0x7f768c074520 2024-01-16T15:02:20.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.417+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f768c0b05f0 con 0x7f768c074520 2024-01-16T15:02:20.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.608+0000 7f7688c00640 1 -- 172.21.15.141:0/2206528280 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v259) v1 ==== 153+0+0 (secure 0 0 0) 0x7f767c07a3d0 con 0x7f768c074520 2024-01-16T15:02:20.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f766406b860 msgr2=0x7f766406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f766406b860 0x7f766406dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f768c0b08c0 tx=0x7f7674009040 comp rx=0 tx=0).stop 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c074520 msgr2=0x7f768c0c4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c074520 0x7f768c0c4b10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f767c000c00 tx=0x7f767c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 shutdown_connections 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f766406b860 0x7f766406dd20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f768c0cc1e0 0x7f768c0b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f768c0ae750 0x7f768c15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 --2- 172.21.15.141:0/2206528280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f768c074520 0x7f768c0c4b10 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.611+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 >> 172.21.15.141:0/2206528280 conn(0x7f768c0b0d70 msgr2=0x7f768c0cb200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.612+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 shutdown_connections 2024-01-16T15:02:20.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.612+0000 7f7693800640 1 -- 172.21.15.141:0/2206528280 wait complete. 2024-01-16T15:02:20.615 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:02:20.615 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:02:20.615 INFO:tasks.cephfs_test_runner:test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:02:20.615 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:20.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.855+0000 7f634bc00640 1 Processor -- start 2024-01-16T15:02:20.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.855+0000 7f634bc00640 1 -- start start 2024-01-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.856+0000 7f634bc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f634413e360 0x7f63440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.856+0000 7f634bc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634415f9c0 0x7f63440cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f634bc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634415c5f0 con 0x7f634413df80 2024-01-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f634bc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634406c470 con 0x7f634413e360 2024-01-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f634bc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634406c750 con 0x7f634415f9c0 2024-01-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f6349200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f634413df80 0x7f634415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58910/0 (socket says 172.21.15.141:58910) 2024-01-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f6349200640 1 -- 172.21.15.141:0/1654402770 learned_addr learned my addr 172.21.15.141:0/1654402770 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:20.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f633be00640 1 --2- 172.21.15.141:0/1654402770 >> v2:172.21.15.188:3300/0 conn(0x7f634413e360 0x7f63440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659702688 0 0) 0x7f634415c5f0 con 0x7f634413df80 2024-01-16T15:02:20.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f6349c00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634415f9c0 0x7f63440cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.857+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6324003520 con 0x7f634413df80 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2462521750 0 0) 0x7f6324003520 con 0x7f634413df80 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634415f9c0 msgr2=0x7f63440cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634415f9c0 0x7f63440cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 >> v2:172.21.15.188:3300/0 conn(0x7f634413e360 msgr2=0x7f63440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 --2- 172.21.15.141:0/1654402770 >> v2:172.21.15.188:3300/0 conn(0x7f634413e360 0x7f63440cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634406ca30 con 0x7f634413df80 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6334001690 con 0x7f634413df80 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 >> v1:172.21.15.141:6789/0 conn(0x7f634413df80 legacy=0x7f634415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6324003970 0x7f6324003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634415f9c0 0x7f63240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6324004d00 0x7f6324005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634406c470 con 0x7f6324003970 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634406c750 con 0x7f6324004d00 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632400b980 con 0x7f634415f9c0 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.858+0000 7f6349200640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6324003970 0x7f6324003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.859+0000 7f6349200640 1 -- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634415f9c0 msgr2=0x7f63240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.859+0000 7f6349c00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6324004d00 0x7f6324005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.859+0000 7f6349200640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634415f9c0 0x7f63240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.859+0000 7f6349200640 1 -- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6324004d00 msgr2=0x7f6324005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.859+0000 7f6349200640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6324004d00 0x7f6324005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.859+0000 7f6349200640 1 -- 172.21.15.141:0/1654402770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324003520 con 0x7f6324003970 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.859+0000 7f6349c00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6324004d00 0x7f6324005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:20.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.860+0000 7f6349200640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6324003970 0x7f6324003d90 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f633400c210 tx=0x7f63240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cd291897582d20 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.860+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334005ad0 con 0x7f6324003970 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.860+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6334005c30 con 0x7f6324003970 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.860+0000 7f6343e00640 1 -- 172.21.15.141:0/1654402770 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633402c8b0 con 0x7f6324003970 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.861+0000 7f634bc00640 1 -- 172.21.15.141:0/1654402770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6324003970 msgr2=0x7f6324003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.861+0000 7f634bc00640 1 --2- 172.21.15.141:0/1654402770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6324003970 0x7f6324003d90 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f633400c210 tx=0x7f63240057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.861+0000 7f6349c00640 1 -- 172.21.15.141:0/1654402770 reap_dead start 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.861+0000 7f634bc00640 1 -- 172.21.15.141:0/1654402770 shutdown_connections 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.861+0000 7f634bc00640 1 -- 172.21.15.141:0/1654402770 >> 172.21.15.141:0/1654402770 conn(0x7f634406a800 msgr2=0x7f634406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.861+0000 7f634bc00640 1 -- 172.21.15.141:0/1654402770 shutdown_connections 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.861+0000 7f634bc00640 1 -- 172.21.15.141:0/1654402770 wait complete. 2024-01-16T15:02:20.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.863+0000 7f634bc00640 1 Processor -- start 2024-01-16T15:02:20.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.863+0000 7f634bc00640 1 -- start start 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.864+0000 7f634bc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6324003970 0x7f6344137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.864+0000 7f634bc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6344142f20 0x7f63441459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.864+0000 7f634bc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6344144560 0x7f63440b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.864+0000 7f634bc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63441440d0 con 0x7f6344144560 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.864+0000 7f634bc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6344145f00 con 0x7f6344142f20 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.864+0000 7f634bc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63441434c0 con 0x7f6324003970 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.864+0000 7f6349c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6344144560 0x7f63440b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6344144560 0x7f63440b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33806/0 (socket says 172.21.15.141:33806) 2024-01-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 -- 172.21.15.141:0/3864159508 learned_addr learned my addr 172.21.15.141:0/3864159508 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f633be00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6344142f20 0x7f63441459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349200640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6324003970 0x7f6344137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 -- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6324003970 msgr2=0x7f6344137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6324003970 0x7f6344137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 -- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6344142f20 msgr2=0x7f63441459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6344142f20 0x7f63441459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 -- 172.21.15.141:0/3864159508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63441437a0 con 0x7f6344144560 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f633be00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6344142f20 0x7f63441459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6349c00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6344144560 0x7f63440b2cf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6344155d40 tx=0x7f633c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6341600640 1 -- 172.21.15.141:0/3864159508 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c00fec0 con 0x7f6344144560 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f6341600640 1 -- 172.21.15.141:0/3864159508 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c011040 con 0x7f6344144560 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.865+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6344143a80 con 0x7f6344144560 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.866+0000 7f6341600640 1 -- 172.21.15.141:0/3864159508 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c017640 con 0x7f6344144560 2024-01-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.866+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63441449e0 con 0x7f6344144560 2024-01-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.867+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6344067fd0 con 0x7f6344144560 2024-01-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.868+0000 7f6341600640 1 -- 172.21.15.141:0/3864159508 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f633c0203b0 con 0x7f6344144560 2024-01-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.868+0000 7f6341600640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f631c06b890 0x7f631c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.869+0000 7f6341600640 1 -- 172.21.15.141:0/3864159508 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f633c08a600 con 0x7f6344144560 2024-01-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.869+0000 7f6349200640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f631c06b890 0x7f631c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.870+0000 7f6349200640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f631c06b890 0x7f631c06dd50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6334034000 tx=0x7f6334037040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:20.873+0000 7f6341600640 1 -- 172.21.15.141:0/3864159508 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f633c054db0 con 0x7f6344144560 2024-01-16T15:02:21.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.063+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63440675d0 con 0x7f6344144560 2024-01-16T15:02:21.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.065+0000 7f6341600640 1 -- 172.21.15.141:0/3864159508 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f633c0583d0 con 0x7f6344144560 2024-01-16T15:02:21.045 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:21.045 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":51,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:02:14.349088+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/2933218811":"2024-01-17T15:02:11.334365+0000","172.21.15.141:6834/2933218811":"2024-01-17T15:02:11.334365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:21.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f631c06b890 msgr2=0x7f631c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f631c06b890 0x7f631c06dd50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6334034000 tx=0x7f6334037040 comp rx=0 tx=0).stop 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6344144560 msgr2=0x7f63440b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6344144560 0x7f63440b2cf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6344155d40 tx=0x7f633c00f4a0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 shutdown_connections 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f631c06b890 0x7f631c06dd50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6344144560 0x7f63440b2cf0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6344142f20 0x7f63441459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 --2- 172.21.15.141:0/3864159508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6324003970 0x7f6344137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 >> 172.21.15.141:0/3864159508 conn(0x7f634406a800 msgr2=0x7f634415c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 shutdown_connections 2024-01-16T15:02:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.068+0000 7f634bc00640 1 -- 172.21.15.141:0/3864159508 wait complete. 2024-01-16T15:02:21.068 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:21.068 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.302+0000 7fd6b7200640 1 Processor -- start 2024-01-16T15:02:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.302+0000 7fd6b7200640 1 -- start start 2024-01-16T15:02:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.304+0000 7fd6b7200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd6b00ae5a0 0x7fd6b015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.304+0000 7fd6b7200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b00cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.304+0000 7fd6b7200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b0070c40 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.304+0000 7fd6b7200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b0071450 con 0x7fd6b00ae5a0 2024-01-16T15:02:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.304+0000 7fd6b7200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b00cfc60 con 0x7fd6b00cf290 2024-01-16T15:02:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6b5200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b00cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6b5200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b00cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:45424/0 (socket says 172.21.15.141:45424) 2024-01-16T15:02:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6b5200640 1 -- 172.21.15.141:0/3680501790 learned_addr learned my addr 172.21.15.141:0/3680501790 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6afe00640 1 --2- 172.21.15.141:0/3680501790 >> v2:172.21.15.188:3300/0 conn(0x7fd6b00ae5a0 0x7fd6b015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6b5200640 1 -- 172.21.15.141:0/3680501790 >> v2:172.21.15.188:3300/0 conn(0x7fd6b00ae5a0 msgr2=0x7fd6b015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6b5200640 1 --2- 172.21.15.141:0/3680501790 >> v2:172.21.15.188:3300/0 conn(0x7fd6b00ae5a0 0x7fd6b015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6b5200640 1 -- 172.21.15.141:0/3680501790 >> v1:172.21.15.188:6789/0 conn(0x7fd6b00b3ae0 legacy=0x7fd6b006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.306+0000 7fd6b5200640 1 -- 172.21.15.141:0/3680501790 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b006c470 con 0x7fd6b00cf290 2024-01-16T15:02:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.305+0000 7fd6aea00640 1 -- 172.21.15.141:0/3680501790 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259237829 0 0) 0x7fd6b0070c40 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.307+0000 7fd6b5200640 1 --2- 172.21.15.141:0/3680501790 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b00cf720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009e50 tx=0x7fd6a802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c7e427058802b7e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.307+0000 7fd6aea00640 1 -- 172.21.15.141:0/3680501790 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a802cb90 con 0x7fd6b00cf290 2024-01-16T15:02:21.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.307+0000 7fd6aea00640 1 -- 172.21.15.141:0/3680501790 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a802ccf0 con 0x7fd6b00cf290 2024-01-16T15:02:21.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.307+0000 7fd6aea00640 1 -- 172.21.15.141:0/3680501790 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8034610 con 0x7fd6b00cf290 2024-01-16T15:02:21.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 -- 172.21.15.141:0/3680501790 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 msgr2=0x7fd6b00cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 --2- 172.21.15.141:0/3680501790 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b00cf720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009e50 tx=0x7fd6a802bff0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 -- 172.21.15.141:0/3680501790 shutdown_connections 2024-01-16T15:02:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 --2- 172.21.15.141:0/3680501790 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b00cf720 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 --2- 172.21.15.141:0/3680501790 >> v2:172.21.15.188:3300/0 conn(0x7fd6b00ae5a0 0x7fd6b015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 -- 172.21.15.141:0/3680501790 >> 172.21.15.141:0/3680501790 conn(0x7fd6b006a800 msgr2=0x7fd6b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 -- 172.21.15.141:0/3680501790 shutdown_connections 2024-01-16T15:02:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.308+0000 7fd6b7200640 1 -- 172.21.15.141:0/3680501790 wait complete. 2024-01-16T15:02:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.310+0000 7fd6b7200640 1 Processor -- start 2024-01-16T15:02:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.310+0000 7fd6b7200640 1 -- start start 2024-01-16T15:02:21.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.311+0000 7fd6b7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6b00ae5a0 0x7fd6b014f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.311+0000 7fd6b7200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6b00b3ae0 0x7fd6b0151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.311+0000 7fd6b7200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.311+0000 7fd6b7200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b015ca90 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.311+0000 7fd6b7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b015cf20 con 0x7fd6b00ae5a0 2024-01-16T15:02:21.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.311+0000 7fd6b7200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b015be80 con 0x7fd6b00cf290 2024-01-16T15:02:21.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6b00b3ae0 0x7fd6b0151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6b5200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6b00b3ae0 0x7fd6b0151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33818/0 (socket says 172.21.15.141:33818) 2024-01-16T15:02:21.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6afe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6b00ae5a0 0x7fd6b014f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 -- 172.21.15.141:0/1155918925 learned_addr learned my addr 172.21.15.141:0/1155918925 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:21.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 -- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 msgr2=0x7fd6b015b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b015b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 -- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6b00ae5a0 msgr2=0x7fd6b014f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6b00ae5a0 0x7fd6b014f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 -- 172.21.15.141:0/1155918925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8009b30 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.312+0000 7fd6af400640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6b00b3ae0 0x7fd6b0151880 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c4b0 tx=0x7fd69800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.313+0000 7fd6a7e00640 1 -- 172.21.15.141:0/1155918925 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698005480 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.313+0000 7fd6a7e00640 1 -- 172.21.15.141:0/1155918925 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6980035f0 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.313+0000 7fd6a7e00640 1 -- 172.21.15.141:0/1155918925 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980039e0 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.313+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b015c160 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.313+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b015c720 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.314+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b015cc40 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.315+0000 7fd6a7e00640 1 -- 172.21.15.141:0/1155918925 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fd698003b40 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.316+0000 7fd6a7e00640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd68806b890 0x7fd68806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.316+0000 7fd6a7e00640 1 -- 172.21.15.141:0/1155918925 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd698089840 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.316+0000 7fd6afe00640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd68806b890 0x7fd68806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.317+0000 7fd6afe00640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd68806b890 0x7fd68806dd50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009b40 tx=0x7fd6a0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.320+0000 7fd6a7e00640 1 -- 172.21.15.141:0/1155918925 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6b015cc40 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.507+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6b00675d0 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.508+0000 7fd6a7e00640 1 -- 172.21.15.141:0/1155918925 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6980170a0 con 0x7fd6b00b3ae0 2024-01-16T15:02:21.488 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.510+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd68806b890 msgr2=0x7fd68806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.510+0000 7fd6b7200640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd68806b890 0x7fd68806dd50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009b40 tx=0x7fd6a0009190 comp rx=0 tx=0).stop 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6b00b3ae0 msgr2=0x7fd6b0151880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6b00b3ae0 0x7fd6b0151880 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c4b0 tx=0x7fd69800c970 comp rx=0 tx=0).stop 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 shutdown_connections 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd68806b890 0x7fd68806dd50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6b00cf290 0x7fd6b015b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6b00b3ae0 0x7fd6b0151880 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 --2- 172.21.15.141:0/1155918925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6b00ae5a0 0x7fd6b014f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 >> 172.21.15.141:0/1155918925 conn(0x7fd6b006a800 msgr2=0x7fd6b00b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 shutdown_connections 2024-01-16T15:02:21.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.511+0000 7fd6b7200640 1 -- 172.21.15.141:0/1155918925 wait complete. 2024-01-16T15:02:21.512 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:02:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.741+0000 7f2bdbc00640 1 Processor -- start 2024-01-16T15:02:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.741+0000 7f2bdbc00640 1 -- start start 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bdbc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2bd40cf9a0 0x7f2bd415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bdbc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd4070c40 con 0x7f2bd40b3ae0 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bdbc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd406c470 con 0x7f2bd40ae5a0 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bdbc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd4071450 con 0x7f2bd40cf9a0 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bcbc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2bd40b3ae0 0x7f2bd40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58916/0 (socket says 172.21.15.141:58916) 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bcbc00640 1 -- 172.21.15.141:0/768306310 learned_addr learned my addr 172.21.15.141:0/768306310 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490810532 0 0) 0x7f2bd4070c40 con 0x7f2bd40b3ae0 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb4003520 con 0x7f2bd40b3ae0 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.743+0000 7f2bd9c00640 1 --2- 172.21.15.141:0/768306310 >> v2:172.21.15.188:3300/0 conn(0x7f2bd40cf9a0 0x7f2bd415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2771505525 0 0) 0x7f2bb4003520 con 0x7f2bd40b3ae0 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 >> v2:172.21.15.188:3300/0 conn(0x7f2bd40cf9a0 msgr2=0x7f2bd415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 --2- 172.21.15.141:0/768306310 >> v2:172.21.15.188:3300/0 conn(0x7f2bd40cf9a0 0x7f2bd415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 >> v1:172.21.15.188:6789/0 conn(0x7f2bd40ae5a0 legacy=0x7f2bd406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd406c750 con 0x7f2bd40b3ae0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd9c00640 1 --2- 172.21.15.141:0/768306310 >> v2:172.21.15.188:3300/0 conn(0x7f2bd40cf9a0 0x7f2bd415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2bbc001670 con 0x7f2bd40b3ae0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 >> v1:172.21.15.141:6789/0 conn(0x7f2bd40b3ae0 legacy=0x7f2bd40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bb40039c0 0x7f2bb4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bb40051d0 0x7f2bb400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4071450 con 0x7f2bb40047c0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb400c110 con 0x7f2bb40051d0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb400c2d0 con 0x7f2bb40039c0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.744+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bd9c00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bb40051d0 0x7f2bb400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bd9200640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bb40039c0 0x7f2bb4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bcbc00640 1 -- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bb40039c0 msgr2=0x7f2bb4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bb40039c0 0x7f2bb4003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bcbc00640 1 -- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bb40051d0 msgr2=0x7f2bb400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bb40051d0 0x7f2bb400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bcbc00640 1 -- 172.21.15.141:0/768306310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4003520 con 0x7f2bb40047c0 2024-01-16T15:02:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.745+0000 7f2bd9c00640 1 -- 172.21.15.141:0/768306310 reap_dead start 2024-01-16T15:02:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.746+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bb4004c40 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c0e0 tx=0x7f2bb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f09c80eda34d902 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.746+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc005ad0 con 0x7f2bb40047c0 2024-01-16T15:02:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.746+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc005c30 con 0x7f2bb40047c0 2024-01-16T15:02:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.746+0000 7f2bd3e00640 1 -- 172.21.15.141:0/768306310 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc02c7d0 con 0x7f2bb40047c0 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.747+0000 7f2bdbc00640 1 -- 172.21.15.141:0/768306310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 msgr2=0x7f2bb4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.747+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bb4004c40 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c0e0 tx=0x7f2bb40057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.747+0000 7f2bdbc00640 1 -- 172.21.15.141:0/768306310 shutdown_connections 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.747+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/768306310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bb4004c40 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.747+0000 7f2bdbc00640 1 -- 172.21.15.141:0/768306310 >> 172.21.15.141:0/768306310 conn(0x7f2bd406a800 msgr2=0x7f2bd406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.747+0000 7f2bdbc00640 1 -- 172.21.15.141:0/768306310 shutdown_connections 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.747+0000 7f2bdbc00640 1 -- 172.21.15.141:0/768306310 wait complete. 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.749+0000 7f2bdbc00640 1 Processor -- start 2024-01-16T15:02:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.749+0000 7f2bdbc00640 1 -- start start 2024-01-16T15:02:21.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bdbc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bd41563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bdbc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bd406cd50 0x7f2bd40cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bdbc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bd406d1d0 0x7f2bd40ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bdbc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40cc850 con 0x7f2bb40047c0 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bdbc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41f29e0 con 0x7f2bd406cd50 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bdbc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41f2ca0 con 0x7f2bd406d1d0 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bd9200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bd41563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bd9200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bd41563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33842/0 (socket says 172.21.15.141:33842) 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bd9200640 1 -- 172.21.15.141:0/2911931687 learned_addr learned my addr 172.21.15.141:0/2911931687 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bd406cd50 0x7f2bd40cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.750+0000 7f2bd9c00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bd406d1d0 0x7f2bd40ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd9200640 1 -- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bd406d1d0 msgr2=0x7f2bd40ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd9200640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bd406d1d0 0x7f2bd40ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd9200640 1 -- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bd406cd50 msgr2=0x7f2bd40cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd9200640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bd406cd50 0x7f2bd40cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd9200640 1 -- 172.21.15.141:0/2911931687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd41f2f80 con 0x7f2bb40047c0 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd9c00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bd406d1d0 0x7f2bd40ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bd406cd50 0x7f2bd40cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd9200640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bd41563b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d480 tx=0x7f2bc400d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd1600640 1 -- 172.21.15.141:0/2911931687 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4005480 con 0x7f2bb40047c0 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd1600640 1 -- 172.21.15.141:0/2911931687 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4003610 con 0x7f2bb40047c0 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bd1600640 1 -- 172.21.15.141:0/2911931687 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40039c0 con 0x7f2bb40047c0 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd41f3260 con 0x7f2bb40047c0 2024-01-16T15:02:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.751+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd41f3820 con 0x7f2bb40047c0 2024-01-16T15:02:21.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.753+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd40675d0 con 0x7f2bb40047c0 2024-01-16T15:02:21.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.754+0000 7f2bd1600640 1 -- 172.21.15.141:0/2911931687 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f2bc4003b20 con 0x7f2bb40047c0 2024-01-16T15:02:21.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.754+0000 7f2bd1600640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bac06b860 0x7f2bac06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.754+0000 7f2bd1600640 1 -- 172.21.15.141:0/2911931687 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2bc408a280 con 0x7f2bb40047c0 2024-01-16T15:02:21.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.755+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bac06b860 0x7f2bac06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.756+0000 7f2bcbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bac06b860 0x7f2bac06dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc034000 tx=0x7f2bbc009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:21.759+0000 7f2bd1600640 1 -- 172.21.15.141:0/2911931687 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bc4054bc0 con 0x7f2bb40047c0 2024-01-16T15:02:21.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.017+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f2bd415e400 con 0x7f2bb40047c0 2024-01-16T15:02:22.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.614+0000 7f2bd1600640 1 -- 172.21.15.141:0/2911931687 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v261) v1 ==== 157+0+0 (secure 0 0 0) 0x7f2bc40581e0 con 0x7f2bb40047c0 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bac06b860 msgr2=0x7f2bac06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bac06b860 0x7f2bac06dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc034000 tx=0x7f2bbc009030 comp rx=0 tx=0).stop 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 msgr2=0x7f2bd41563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bd41563b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d480 tx=0x7f2bc400d940 comp rx=0 tx=0).stop 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 shutdown_connections 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bac06b860 0x7f2bac06dd20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bd406d1d0 0x7f2bd40ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bd406cd50 0x7f2bd40cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 --2- 172.21.15.141:0/2911931687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bb40047c0 0x7f2bd41563b0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 >> 172.21.15.141:0/2911931687 conn(0x7f2bd406a800 msgr2=0x7f2bd41617a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:22.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 shutdown_connections 2024-01-16T15:02:22.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.618+0000 7f2bdbc00640 1 -- 172.21.15.141:0/2911931687 wait complete. 2024-01-16T15:02:22.619 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:02:22.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.846+0000 7fb06d000640 1 Processor -- start 2024-01-16T15:02:22.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.846+0000 7fb06d000640 1 -- start start 2024-01-16T15:02:22.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.848+0000 7fb06d000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb068146850 0x7fb0680715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.848+0000 7fb06d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb068071b10 0x7fb0680ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.848+0000 7fb06d000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06806d480 con 0x7fb0680adb00 2024-01-16T15:02:22.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.848+0000 7fb06d000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb068071fa0 con 0x7fb068146850 2024-01-16T15:02:22.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.848+0000 7fb06d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680b3490 con 0x7fb068071b10 2024-01-16T15:02:22.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.848+0000 7fb05fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb068071b10 0x7fb0680ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.849+0000 7fb05fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb068071b10 0x7fb0680ad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:45466/0 (socket says 172.21.15.141:45466) 2024-01-16T15:02:22.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.849+0000 7fb05fe00640 1 -- 172.21.15.141:0/897668409 learned_addr learned my addr 172.21.15.141:0/897668409 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:22.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.849+0000 7fb067400640 1 --2- 172.21.15.141:0/897668409 >> v2:172.21.15.188:3300/0 conn(0x7fb068146850 0x7fb0680715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.849+0000 7fb05fe00640 1 -- 172.21.15.141:0/897668409 >> v2:172.21.15.188:3300/0 conn(0x7fb068146850 msgr2=0x7fb0680715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:22.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.849+0000 7fb05fe00640 1 --2- 172.21.15.141:0/897668409 >> v2:172.21.15.188:3300/0 conn(0x7fb068146850 0x7fb0680715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.849+0000 7fb05fe00640 1 -- 172.21.15.141:0/897668409 >> v1:172.21.15.188:6789/0 conn(0x7fb0680adb00 legacy=0x7fb0680700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.849+0000 7fb05fe00640 1 -- 172.21.15.141:0/897668409 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0680b3770 con 0x7fb068071b10 2024-01-16T15:02:22.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.851+0000 7fb05fe00640 1 --2- 172.21.15.141:0/897668409 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb068071b10 0x7fb0680ad5c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb050009870 tx=0x7fb05002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4cf5e1f01414c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.851+0000 7fb066a00640 1 -- 172.21.15.141:0/897668409 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050030840 con 0x7fb068071b10 2024-01-16T15:02:22.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.851+0000 7fb066a00640 1 -- 172.21.15.141:0/897668409 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0500309a0 con 0x7fb068071b10 2024-01-16T15:02:22.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.851+0000 7fb066a00640 1 -- 172.21.15.141:0/897668409 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050030d30 con 0x7fb068071b10 2024-01-16T15:02:22.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 -- 172.21.15.141:0/897668409 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb068071b10 msgr2=0x7fb0680ad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 --2- 172.21.15.141:0/897668409 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb068071b10 0x7fb0680ad5c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb050009870 tx=0x7fb05002b980 comp rx=0 tx=0).stop 2024-01-16T15:02:22.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 -- 172.21.15.141:0/897668409 shutdown_connections 2024-01-16T15:02:22.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 --2- 172.21.15.141:0/897668409 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb068071b10 0x7fb0680ad5c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 --2- 172.21.15.141:0/897668409 >> v2:172.21.15.188:3300/0 conn(0x7fb068146850 0x7fb0680715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 -- 172.21.15.141:0/897668409 >> 172.21.15.141:0/897668409 conn(0x7fb06806a3e0 msgr2=0x7fb06806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:22.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 -- 172.21.15.141:0/897668409 shutdown_connections 2024-01-16T15:02:22.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.852+0000 7fb06d000640 1 -- 172.21.15.141:0/897668409 wait complete. 2024-01-16T15:02:22.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.854+0000 7fb06d000640 1 Processor -- start 2024-01-16T15:02:22.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.855+0000 7fb06d000640 1 -- start start 2024-01-16T15:02:22.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb06d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb068071b10 0x7fb06815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb06d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0680adb00 0x7fb0681efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb06d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb068146850 0x7fb0681effa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb06d000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06815bac0 con 0x7fb068071b10 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb06d000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06815aeb0 con 0x7fb068146850 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb06d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06815b190 con 0x7fb0680adb00 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb067400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb068071b10 0x7fb06815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb067400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb068071b10 0x7fb06815bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33848/0 (socket says 172.21.15.141:33848) 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb067400640 1 -- 172.21.15.141:0/3401353566 learned_addr learned my addr 172.21.15.141:0/3401353566 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.856+0000 7fb05fe00640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0680adb00 0x7fb0681efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb067400640 1 -- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0680adb00 msgr2=0x7fb0681efa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb067400640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0680adb00 0x7fb0681efa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb067400640 1 -- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb068146850 msgr2=0x7fb0681effa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb067e00640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb068146850 0x7fb0681effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb067400640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb068146850 0x7fb0681effa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb067400640 1 -- 172.21.15.141:0/3401353566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0500094d0 con 0x7fb068071b10 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb05fe00640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0680adb00 0x7fb0681efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb067400640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb068071b10 0x7fb06815bf90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb05800e9b0 tx=0x7fb05800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb05f400640 1 -- 172.21.15.141:0/3401353566 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058009670 con 0x7fb068071b10 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb05f400640 1 -- 172.21.15.141:0/3401353566 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05800c9a0 con 0x7fb068071b10 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06815b470 con 0x7fb068071b10 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.857+0000 7fb05f400640 1 -- 172.21.15.141:0/3401353566 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800cd50 con 0x7fb068071b10 2024-01-16T15:02:22.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.858+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06815c4d0 con 0x7fb068071b10 2024-01-16T15:02:22.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.859+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06815c6c0 con 0x7fb068071b10 2024-01-16T15:02:22.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.859+0000 7fb05f400640 1 -- 172.21.15.141:0/3401353566 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb058013470 con 0x7fb068071b10 2024-01-16T15:02:22.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.860+0000 7fb05f400640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb04006b860 0x7fb04006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.860+0000 7fb05f400640 1 -- 172.21.15.141:0/3401353566 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb05808a620 con 0x7fb068071b10 2024-01-16T15:02:22.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.861+0000 7fb05fe00640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb04006b860 0x7fb04006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.862+0000 7fb05fe00640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb04006b860 0x7fb04006dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb050036000 tx=0x7fb05003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:22.865+0000 7fb05f400640 1 -- 172.21.15.141:0/3401353566 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb058054ee0 con 0x7fb068071b10 2024-01-16T15:02:23.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.040+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb068067450 con 0x7fb068071b10 2024-01-16T15:02:23.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.047+0000 7fb05f400640 1 -- 172.21.15.141:0/3401353566 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb058058500 con 0x7fb068071b10 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.049+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb04006b860 msgr2=0x7fb04006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.049+0000 7fb06d000640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb04006b860 0x7fb04006dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb050036000 tx=0x7fb05003b040 comp rx=0 tx=0).stop 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.049+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb068071b10 msgr2=0x7fb06815bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.049+0000 7fb06d000640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb068071b10 0x7fb06815bf90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb05800e9b0 tx=0x7fb05800ee70 comp rx=0 tx=0).stop 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 shutdown_connections 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb04006b860 0x7fb04006dd20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb068146850 0x7fb0681effa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb0680adb00 0x7fb0681efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 --2- 172.21.15.141:0/3401353566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb068071b10 0x7fb06815bf90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 >> 172.21.15.141:0/3401353566 conn(0x7fb06806a3e0 msgr2=0x7fb0680afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:23.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 shutdown_connections 2024-01-16T15:02:23.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.050+0000 7fb06d000640 1 -- 172.21.15.141:0/3401353566 wait complete. 2024-01-16T15:02:23.046 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:02:23.046 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:23.156 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:23.273 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:02:23.273 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:02:23.273 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:23.328 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:23.328 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:02:23.328 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:23.382 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:23.382 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:02:23.382 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:23.441 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:23.441 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:02:23.441 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:23.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:23.494 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:02:23.495 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:23.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.725+0000 7f368d400640 1 Processor -- start 2024-01-16T15:02:23.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.725+0000 7f368d400640 1 -- start start 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.726+0000 7f368d400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3688143c40 0x7f3688071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.726+0000 7f368d400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f36880717d0 0x7f36880cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.726+0000 7f368d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880cca40 0x7f368815eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.726+0000 7f368d400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36880723c0 con 0x7f3688143c40 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.726+0000 7f368d400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3688070e50 con 0x7f36880717d0 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.726+0000 7f368d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36880cce40 con 0x7f36880cca40 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3688143c40 0x7f3688071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3688143c40 0x7f3688071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34880/0 (socket says 172.21.15.141:34880) 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 -- 172.21.15.141:0/3229630853 learned_addr learned my addr 172.21.15.141:0/3229630853 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:23.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 -- 172.21.15.141:0/3229630853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880cca40 msgr2=0x7f368815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:23.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 --2- 172.21.15.141:0/3229630853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880cca40 0x7f368815eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 -- 172.21.15.141:0/3229630853 >> v2:172.21.15.188:3300/0 conn(0x7f36880717d0 msgr2=0x7f36880cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:23.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 --2- 172.21.15.141:0/3229630853 >> v2:172.21.15.188:3300/0 conn(0x7f36880717d0 0x7f36880cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.727+0000 7f3687e00640 1 -- 172.21.15.141:0/3229630853 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3688144020 con 0x7f3688143c40 2024-01-16T15:02:23.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.728+0000 7f3687e00640 1 --2- 172.21.15.141:0/3229630853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3688143c40 0x7f3688071290 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f36880b44a0 tx=0x7f367802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801129104cfb5d88 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:23.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.728+0000 7f3687400640 1 -- 172.21.15.141:0/3229630853 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36780317d0 con 0x7f3688143c40 2024-01-16T15:02:23.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f3687400640 1 -- 172.21.15.141:0/3229630853 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3678031930 con 0x7f3688143c40 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f3687400640 1 -- 172.21.15.141:0/3229630853 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3678031cc0 con 0x7f3688143c40 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 -- 172.21.15.141:0/3229630853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3688143c40 msgr2=0x7f3688071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 --2- 172.21.15.141:0/3229630853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3688143c40 0x7f3688071290 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f36880b44a0 tx=0x7f367802b870 comp rx=0 tx=0).stop 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 -- 172.21.15.141:0/3229630853 shutdown_connections 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 --2- 172.21.15.141:0/3229630853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880cca40 0x7f368815eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 --2- 172.21.15.141:0/3229630853 >> v2:172.21.15.188:3300/0 conn(0x7f36880717d0 0x7f36880cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 --2- 172.21.15.141:0/3229630853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3688143c40 0x7f3688071290 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 -- 172.21.15.141:0/3229630853 >> 172.21.15.141:0/3229630853 conn(0x7f368806a800 msgr2=0x7f368806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 -- 172.21.15.141:0/3229630853 shutdown_connections 2024-01-16T15:02:23.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.729+0000 7f368d400640 1 -- 172.21.15.141:0/3229630853 wait complete. 2024-01-16T15:02:23.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.731+0000 7f368d400640 1 Processor -- start 2024-01-16T15:02:23.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.731+0000 7f368d400640 1 -- start start 2024-01-16T15:02:23.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.732+0000 7f368d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880717d0 0x7f36880b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:23.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.732+0000 7f368d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36880cca40 0x7f36880aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:23.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.732+0000 7f368d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688143c40 0x7f36881f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.732+0000 7f368d400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880b32f0 con 0x7f36880cca40 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.732+0000 7f368d400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880b2440 con 0x7f3688143c40 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.732+0000 7f368d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36880b2670 con 0x7f36880717d0 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.732+0000 7f367f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36880cca40 0x7f36880aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36880cca40 0x7f36880aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34882/0 (socket says 172.21.15.141:34882) 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 -- 172.21.15.141:0/1446567512 learned_addr learned my addr 172.21.15.141:0/1446567512 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f3687e00640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880717d0 0x7f36880b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 -- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880717d0 msgr2=0x7f36880b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f368ca00640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688143c40 0x7f36881f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880717d0 0x7f36880b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 -- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688143c40 msgr2=0x7f36881f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688143c40 0x7f36881f53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 -- 172.21.15.141:0/1446567512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36780094d0 con 0x7f36880cca40 2024-01-16T15:02:23.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f367f200640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36880cca40 0x7f36880aece0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f367000b410 tx=0x7f367000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:23.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.733+0000 7f3684c00640 1 -- 172.21.15.141:0/1446567512 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670011830 con 0x7f36880cca40 2024-01-16T15:02:23.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.734+0000 7f3684c00640 1 -- 172.21.15.141:0/1446567512 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36700119b0 con 0x7f36880cca40 2024-01-16T15:02:23.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.734+0000 7f3684c00640 1 -- 172.21.15.141:0/1446567512 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670011d60 con 0x7f36880cca40 2024-01-16T15:02:23.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.734+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36880b2950 con 0x7f36880cca40 2024-01-16T15:02:23.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.734+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36880b2f10 con 0x7f36880cca40 2024-01-16T15:02:23.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.735+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3688067450 con 0x7f36880cca40 2024-01-16T15:02:23.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.736+0000 7f3684c00640 1 -- 172.21.15.141:0/1446567512 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f367001a400 con 0x7f36880cca40 2024-01-16T15:02:23.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.737+0000 7f3684c00640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f366006b860 0x7f366006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:23.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.737+0000 7f3684c00640 1 -- 172.21.15.141:0/1446567512 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f367008a530 con 0x7f36880cca40 2024-01-16T15:02:23.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.738+0000 7f3687e00640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f366006b860 0x7f366006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:23.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.739+0000 7f3687e00640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f366006b860 0x7f366006dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3678036000 tx=0x7f367803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:23.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.742+0000 7f3684c00640 1 -- 172.21.15.141:0/1446567512 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3670054df0 con 0x7f36880cca40 2024-01-16T15:02:23.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.951+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36880ceb30 con 0x7f36880cca40 2024-01-16T15:02:23.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.952+0000 7f3684c00640 1 -- 172.21.15.141:0/1446567512 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f3670015090 con 0x7f36880cca40 2024-01-16T15:02:23.933 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:23.933 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2118095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":2118095418},{"type":"v1","addr":"172.21.15.188:6837","nonce":2118095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/3680808697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3680808697},{"type":"v1","addr":"172.21.15.188:6836","nonce":3680808697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5793,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/275409001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":275409001},{"type":"v1","addr":"172.21.15.141:6833","nonce":275409001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3757287180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3757287180},{"type":"v1","addr":"172.21.15.141:6835","nonce":3757287180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-01-16T15:02:23.934 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 30 2024-01-16T15:02:23.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f366006b860 msgr2=0x7f366006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f366006b860 0x7f366006dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3678036000 tx=0x7f367803d040 comp rx=0 tx=0).stop 2024-01-16T15:02:23.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36880cca40 msgr2=0x7f36880aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36880cca40 0x7f36880aece0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f367000b410 tx=0x7f367000b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 shutdown_connections 2024-01-16T15:02:23.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f366006b860 0x7f366006dd20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688143c40 0x7f36881f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36880cca40 0x7f36880aece0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 --2- 172.21.15.141:0/1446567512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36880717d0 0x7f36880b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 >> 172.21.15.141:0/1446567512 conn(0x7f368806a800 msgr2=0x7f36880b02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:23.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 shutdown_connections 2024-01-16T15:02:23.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:23.955+0000 7f368d400640 1 -- 172.21.15.141:0/1446567512 wait complete. 2024-01-16T15:02:23.955 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:02:23.955 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:02:23.955 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:02:23.955 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:23.955 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:02:23.956 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:02:23.956 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:02:23.956 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:23.956 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:02:23.956 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:02:23.959 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:02:23.959 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:23.959 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:02:23.959 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:02:23.959 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:02:23.959 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:23.975 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:02:23.975 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:02:23.976 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:02:23.976 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:02:23.988 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:02:23.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:02:23.994 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:02:23.994 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:02:24.022 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:02:24.022 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:24.028 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:02:24.028 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:24.041 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:02:24.041 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:24.047 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:02:24.047 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:24.047 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:02:24.168 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:02:24.177 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T15:02:24.198 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T15:02:24.213 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T15:02:24.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.248+0000 7fa4f0600640 1 Processor -- start 2024-01-16T15:02:24.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.248+0000 7fa4f0600640 1 -- start start 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4f0600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa4e81540b0 0x7fa4e8159f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4f0600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e806b690 0x7fa4e815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4f0600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa4e815e270 0x7fa4e8070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4f0600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e8154490 con 0x7fa4e815e270 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4f0600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e815d550 con 0x7fa4e81540b0 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4f0600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e815a7b0 con 0x7fa4e806b690 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4ee600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa4e815e270 0x7fa4e8070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4ee600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa4e815e270 0x7fa4e8070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34916/0 (socket says 172.21.15.141:34916) 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.249+0000 7fa4ee600640 1 -- 172.21.15.141:0/699342245 learned_addr learned my addr 172.21.15.141:0/699342245 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4edc00640 1 --2- 172.21.15.141:0/699342245 >> v2:172.21.15.188:3300/0 conn(0x7fa4e81540b0 0x7fa4e8159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4ed200640 1 --2- 172.21.15.141:0/699342245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e806b690 0x7fa4e815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4ee600640 1 -- 172.21.15.141:0/699342245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e806b690 msgr2=0x7fa4e815dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4ee600640 1 --2- 172.21.15.141:0/699342245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e806b690 0x7fa4e815dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4ee600640 1 -- 172.21.15.141:0/699342245 >> v2:172.21.15.188:3300/0 conn(0x7fa4e81540b0 msgr2=0x7fa4e8159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:24.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4ee600640 1 --2- 172.21.15.141:0/699342245 >> v2:172.21.15.188:3300/0 conn(0x7fa4e81540b0 0x7fa4e8159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4ee600640 1 -- 172.21.15.141:0/699342245 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e815aa90 con 0x7fa4e815e270 2024-01-16T15:02:24.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.250+0000 7fa4ed200640 1 --2- 172.21.15.141:0/699342245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e806b690 0x7fa4e815dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.251+0000 7fa4ee600640 1 --2- 172.21.15.141:0/699342245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e815e270 0x7fa4e8070af0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80704d0 tx=0x7fa4e002e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ca36197c261aa1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.251+0000 7fa4e7e00640 1 -- 172.21.15.141:0/699342245 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e002ee10 con 0x7fa4e815e270 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.251+0000 7fa4e7e00640 1 -- 172.21.15.141:0/699342245 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e0032070 con 0x7fa4e815e270 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.251+0000 7fa4e7e00640 1 -- 172.21.15.141:0/699342245 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0038620 con 0x7fa4e815e270 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 -- 172.21.15.141:0/699342245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e815e270 msgr2=0x7fa4e8070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 --2- 172.21.15.141:0/699342245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e815e270 0x7fa4e8070af0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80704d0 tx=0x7fa4e002e060 comp rx=0 tx=0).stop 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 -- 172.21.15.141:0/699342245 shutdown_connections 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 --2- 172.21.15.141:0/699342245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e815e270 0x7fa4e8070af0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 --2- 172.21.15.141:0/699342245 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e806b690 0x7fa4e815dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 --2- 172.21.15.141:0/699342245 >> v2:172.21.15.188:3300/0 conn(0x7fa4e81540b0 0x7fa4e8159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 -- 172.21.15.141:0/699342245 >> 172.21.15.141:0/699342245 conn(0x7fa4e80cf700 msgr2=0x7fa4e80cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:24.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 -- 172.21.15.141:0/699342245 shutdown_connections 2024-01-16T15:02:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.252+0000 7fa4f0600640 1 -- 172.21.15.141:0/699342245 wait complete. 2024-01-16T15:02:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.254+0000 7fa4f0600640 1 Processor -- start 2024-01-16T15:02:24.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.254+0000 7fa4f0600640 1 -- start start 2024-01-16T15:02:24.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4f0600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4e806b690 0x7fa4e8154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4f0600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e81540b0 0x7fa4e8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4f0600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e8157210 0x7fa4e8158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4f0600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e8142030 con 0x7fa4e81540b0 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4f0600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e8142310 con 0x7fa4e806b690 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4f0600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e81425f0 con 0x7fa4e8157210 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e81540b0 0x7fa4e8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e81540b0 0x7fa4e8156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34928/0 (socket says 172.21.15.141:34928) 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 -- 172.21.15.141:0/50987782 learned_addr learned my addr 172.21.15.141:0/50987782 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4edc00640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4e806b690 0x7fa4e8154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ee600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e8157210 0x7fa4e8158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 -- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e8157210 msgr2=0x7fa4e8158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e8157210 0x7fa4e8158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 -- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4e806b690 msgr2=0x7fa4e8154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4e806b690 0x7fa4e8154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4ed200640 1 -- 172.21.15.141:0/50987782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e000bd40 con 0x7fa4e81540b0 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.255+0000 7fa4edc00640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4e806b690 0x7fa4e8154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.256+0000 7fa4ed200640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e81540b0 0x7fa4e8156cd0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000e7e0 tx=0x7fa4d000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.256+0000 7fa4e5600640 1 -- 172.21.15.141:0/50987782 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0004030 con 0x7fa4e81540b0 2024-01-16T15:02:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.256+0000 7fa4e5600640 1 -- 172.21.15.141:0/50987782 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d0004190 con 0x7fa4e81540b0 2024-01-16T15:02:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.256+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e81428d0 con 0x7fa4e81540b0 2024-01-16T15:02:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.256+0000 7fa4e5600640 1 -- 172.21.15.141:0/50987782 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0004520 con 0x7fa4e81540b0 2024-01-16T15:02:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.256+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e8151650 con 0x7fa4e81540b0 2024-01-16T15:02:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.257+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e80675d0 con 0x7fa4e81540b0 2024-01-16T15:02:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.258+0000 7fa4e5600640 1 -- 172.21.15.141:0/50987782 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fa4d000c6d0 con 0x7fa4e81540b0 2024-01-16T15:02:24.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.259+0000 7fa4e5600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c006b860 0x7fa4c006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.259+0000 7fa4e5600640 1 -- 172.21.15.141:0/50987782 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa4d008a280 con 0x7fa4e81540b0 2024-01-16T15:02:24.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.259+0000 7fa4edc00640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c006b860 0x7fa4c006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.260+0000 7fa4edc00640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c006b860 0x7fa4c006dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009b10 tx=0x7fa4d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.263+0000 7fa4e5600640 1 -- 172.21.15.141:0/50987782 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4d00549a0 con 0x7fa4e81540b0 2024-01-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.469+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa4e80715e0 con 0x7fa4e81540b0 2024-01-16T15:02:24.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.638+0000 7fa4e5600640 1 -- 172.21.15.141:0/50987782 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v53) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa4d0057fc0 con 0x7fa4e81540b0 2024-01-16T15:02:24.618 INFO:teuthology.orchestra.run.smithi141.stderr: removed all blocklist entries 2024-01-16T15:02:24.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c006b860 msgr2=0x7fa4c006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c006b860 0x7fa4c006dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009b10 tx=0x7fa4d8008040 comp rx=0 tx=0).stop 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e81540b0 msgr2=0x7fa4e8156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e81540b0 0x7fa4e8156cd0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000e7e0 tx=0x7fa4d000eca0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 shutdown_connections 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c006b860 0x7fa4c006dd20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4e8157210 0x7fa4e8158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4e81540b0 0x7fa4e8156cd0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 --2- 172.21.15.141:0/50987782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4e806b690 0x7fa4e8154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 >> 172.21.15.141:0/50987782 conn(0x7fa4e80cf700 msgr2=0x7fa4e80cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:24.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.642+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 shutdown_connections 2024-01-16T15:02:24.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.643+0000 7fa4f0600640 1 -- 172.21.15.141:0/50987782 wait complete. 2024-01-16T15:02:24.643 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:02:24.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.865+0000 7f8360400640 1 Processor -- start 2024-01-16T15:02:24.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.865+0000 7f8360400640 1 -- start start 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.867+0000 7f8360400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f835813df80 0x7f835815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.867+0000 7f8360400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f83580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.867+0000 7f8360400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835806c470 con 0x7f835815f9c0 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f8360400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835806c750 con 0x7f835813df80 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f8360400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835806ca30 con 0x7f835813e360 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f835da00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f835813df80 0x7f835815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f835e400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f835815f9c0 0x7f83581408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42762/0 (socket says 172.21.15.141:42762) 2024-01-16T15:02:24.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f835e400640 1 -- 172.21.15.141:0/3211643316 learned_addr learned my addr 172.21.15.141:0/3211643316 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:24.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f835da00640 1 -- 172.21.15.141:0/3211643316 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 msgr2=0x7f83580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:24.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f835da00640 1 --2- 172.21.15.141:0/3211643316 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f83580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f835da00640 1 -- 172.21.15.141:0/3211643316 >> v1:172.21.15.141:6789/0 conn(0x7f835815f9c0 legacy=0x7f83581408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f835da00640 1 -- 172.21.15.141:0/3211643316 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835806cd10 con 0x7f835813df80 2024-01-16T15:02:24.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.868+0000 7f8357e00640 1 -- 172.21.15.141:0/3211643316 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824186764 0 0) 0x7f835806c470 con 0x7f835815f9c0 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.869+0000 7f835da00640 1 --2- 172.21.15.141:0/3211643316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835813df80 0x7f835815bee0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f83480097a0 tx=0x7f834802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826ff4c9fea271bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8357e00640 1 -- 172.21.15.141:0/3211643316 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348031830 con 0x7f835813df80 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8357e00640 1 -- 172.21.15.141:0/3211643316 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8348031990 con 0x7f835813df80 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8357e00640 1 -- 172.21.15.141:0/3211643316 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348031d20 con 0x7f835813df80 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 -- 172.21.15.141:0/3211643316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835813df80 msgr2=0x7f835815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 --2- 172.21.15.141:0/3211643316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835813df80 0x7f835815bee0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f83480097a0 tx=0x7f834802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 -- 172.21.15.141:0/3211643316 shutdown_connections 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 --2- 172.21.15.141:0/3211643316 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f83580cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 --2- 172.21.15.141:0/3211643316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835813df80 0x7f835815bee0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 -- 172.21.15.141:0/3211643316 >> 172.21.15.141:0/3211643316 conn(0x7f835806a800 msgr2=0x7f835806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:24.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 -- 172.21.15.141:0/3211643316 shutdown_connections 2024-01-16T15:02:24.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.870+0000 7f8360400640 1 -- 172.21.15.141:0/3211643316 wait complete. 2024-01-16T15:02:24.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.872+0000 7f8360400640 1 Processor -- start 2024-01-16T15:02:24.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.872+0000 7f8360400640 1 -- start start 2024-01-16T15:02:24.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f8360400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f835813df80 0x7f835814df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f8360400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f835813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f8360400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835815f9c0 0x7f835814d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f8360400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835813dda0 con 0x7f835815f9c0 2024-01-16T15:02:24.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f8360400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83581f22b0 con 0x7f835813df80 2024-01-16T15:02:24.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f8360400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83581f2590 con 0x7f835813e360 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f835e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835815f9c0 0x7f835814d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.873+0000 7f835e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835815f9c0 0x7f835814d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34954/0 (socket says 172.21.15.141:34954) 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835e400640 1 -- 172.21.15.141:0/2768552576 learned_addr learned my addr 172.21.15.141:0/2768552576 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835d000640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f835813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835e400640 1 -- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 msgr2=0x7f835813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835e400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f835813d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835e400640 1 -- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f835813df80 msgr2=0x7f835814df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835da00640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f835813df80 0x7f835814df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835e400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f835813df80 0x7f835814df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835e400640 1 -- 172.21.15.141:0/2768552576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8348009400 con 0x7f835815f9c0 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835d000640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f835813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f835e400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835815f9c0 0x7f835814d610 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f835000c7a0 tx=0x7f835000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f8355600640 1 -- 172.21.15.141:0/2768552576 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835000a640 con 0x7f835815f9c0 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f8355600640 1 -- 172.21.15.141:0/2768552576 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835000a7a0 con 0x7f835815f9c0 2024-01-16T15:02:24.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.874+0000 7f8355600640 1 -- 172.21.15.141:0/2768552576 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835000ab90 con 0x7f835815f9c0 2024-01-16T15:02:24.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.875+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83581f2870 con 0x7f835815f9c0 2024-01-16T15:02:24.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.875+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83581f2e30 con 0x7f835815f9c0 2024-01-16T15:02:24.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.876+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83581f3040 con 0x7f835815f9c0 2024-01-16T15:02:24.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.877+0000 7f8355600640 1 -- 172.21.15.141:0/2768552576 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f835000acf0 con 0x7f835815f9c0 2024-01-16T15:02:24.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.877+0000 7f8355600640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f833006b860 0x7f833006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.878+0000 7f8355600640 1 -- 172.21.15.141:0/2768552576 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f83500898a0 con 0x7f835815f9c0 2024-01-16T15:02:24.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.878+0000 7f835da00640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f833006b860 0x7f833006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.879+0000 7f835da00640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f833006b860 0x7f833006dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8348036000 tx=0x7f8348008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:24.881+0000 7f8355600640 1 -- 172.21.15.141:0/2768552576 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83500541b0 con 0x7f835815f9c0 2024-01-16T15:02:25.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.139+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f83580675d0 con 0x7f835815f9c0 2024-01-16T15:02:25.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.140+0000 7f8355600640 1 -- 172.21.15.141:0/2768552576 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8350018090 con 0x7f835815f9c0 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "auth_dump": [ 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.a", 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ==", 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.b", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg==", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.c", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw==", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:02:25.122 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.d", 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg==", 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.123 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.0", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA==", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.1", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg==", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.124 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.2", 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g==", 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.3", 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w==", 2024-01-16T15:02:25.125 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.4", 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ==", 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.126 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.5", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA==", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.6", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ==", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.127 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.7", 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg==", 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.0", 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg==", 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.128 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow r", 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.1", 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ==", 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.129 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.2", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ==", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.3", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA==", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:02:25.130 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.admin", 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAEmaZl0VgDMRAABOWZ0senIXHFvij8FQHiAg==", 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.x", 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ==", 2024-01-16T15:02:25.131 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.y", 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA==", 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:25.132 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2024-01-16T15:02:25.133 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:02:25.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.142+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f833006b860 msgr2=0x7f833006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.142+0000 7f8360400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f833006b860 0x7f833006dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8348036000 tx=0x7f8348008030 comp rx=0 tx=0).stop 2024-01-16T15:02:25.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.142+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835815f9c0 msgr2=0x7f835814d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.142+0000 7f8360400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835815f9c0 0x7f835814d610 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f835000c7a0 tx=0x7f835000cc60 comp rx=0 tx=0).stop 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 shutdown_connections 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f833006b860 0x7f833006dd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f835815f9c0 0x7f835814d610 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f835813e360 0x7f835813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 --2- 172.21.15.141:0/2768552576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f835813df80 0x7f835814df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 >> 172.21.15.141:0/2768552576 conn(0x7f835806a800 msgr2=0x7f835806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 shutdown_connections 2024-01-16T15:02:25.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.143+0000 7f8360400640 1 -- 172.21.15.141:0/2768552576 wait complete. 2024-01-16T15:02:25.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:25.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.368+0000 7f7e5f000640 1 Processor -- start 2024-01-16T15:02:25.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.368+0000 7f7e5f000640 1 -- start start 2024-01-16T15:02:25.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e5f000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7e58143ae0 0x7f7e580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e5f000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e580cf7d0 0x7f7e5815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e5f000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e580cfc30 con 0x7f7e5815e1e0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e5f000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e580b1ee0 con 0x7f7e580cf7d0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e5f000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e580b2780 con 0x7f7e58143ae0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e4f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e580cf7d0 0x7f7e5815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e4f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7e580cf7d0 0x7f7e5815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34956/0 (socket says 172.21.15.141:34956) 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e4f200640 1 -- 172.21.15.141:0/2247173391 learned_addr learned my addr 172.21.15.141:0/2247173391 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e5d000640 1 --1- 172.21.15.141:0/2247173391 >> v1:172.21.15.141:6789/0 conn(0x7f7e5815e1e0 0x7f7e5815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42764/0 (socket says 172.21.15.141:42764) 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e57e00640 1 --2- 172.21.15.141:0/2247173391 >> v2:172.21.15.188:3300/0 conn(0x7f7e58143ae0 0x7f7e580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e4f200640 1 -- 172.21.15.141:0/2247173391 >> v2:172.21.15.188:3300/0 conn(0x7f7e58143ae0 msgr2=0x7f7e580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.370+0000 7f7e4f200640 1 --2- 172.21.15.141:0/2247173391 >> v2:172.21.15.188:3300/0 conn(0x7f7e58143ae0 0x7f7e580cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.371+0000 7f7e4f200640 1 -- 172.21.15.141:0/2247173391 >> v1:172.21.15.141:6789/0 conn(0x7f7e5815e1e0 legacy=0x7f7e5815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.371+0000 7f7e4f200640 1 -- 172.21.15.141:0/2247173391 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5815e990 con 0x7f7e580cf7d0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.371+0000 7f7e57400640 1 -- 172.21.15.141:0/2247173391 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743399138 0 0) 0x7f7e580cfc30 con 0x7f7e5815e1e0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.372+0000 7f7e4f200640 1 --2- 172.21.15.141:0/2247173391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e580cf7d0 0x7f7e5815dca0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7e580cfea0 tx=0x7f7e4002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c26d505423d2baf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.372+0000 7f7e57400640 1 -- 172.21.15.141:0/2247173391 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40030860 con 0x7f7e580cf7d0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.372+0000 7f7e57400640 1 -- 172.21.15.141:0/2247173391 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e400309c0 con 0x7f7e580cf7d0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.372+0000 7f7e57400640 1 -- 172.21.15.141:0/2247173391 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40030d50 con 0x7f7e580cf7d0 2024-01-16T15:02:25.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 -- 172.21.15.141:0/2247173391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e580cf7d0 msgr2=0x7f7e5815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 --2- 172.21.15.141:0/2247173391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e580cf7d0 0x7f7e5815dca0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7e580cfea0 tx=0x7f7e4002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 -- 172.21.15.141:0/2247173391 shutdown_connections 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 --2- 172.21.15.141:0/2247173391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e580cf7d0 0x7f7e5815dca0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 --2- 172.21.15.141:0/2247173391 >> v2:172.21.15.188:3300/0 conn(0x7f7e58143ae0 0x7f7e580cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 -- 172.21.15.141:0/2247173391 >> 172.21.15.141:0/2247173391 conn(0x7f7e5806a800 msgr2=0x7f7e5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 -- 172.21.15.141:0/2247173391 shutdown_connections 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.373+0000 7f7e5f000640 1 -- 172.21.15.141:0/2247173391 wait complete. 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.375+0000 7f7e5f000640 1 Processor -- start 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.375+0000 7f7e5f000640 1 -- start start 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5f000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580cf7d0 0x7f7e58141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5f000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e58143ae0 0x7f7e581f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5f000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e5815e1e0 0x7f7e5813d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5f000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580b1730 con 0x7f7e5815e1e0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5f000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581f3120 con 0x7f7e580cf7d0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5f000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581f3400 con 0x7f7e58143ae0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e5815e1e0 0x7f7e5813d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e5815e1e0 0x7f7e5813d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34972/0 (socket says 172.21.15.141:34972) 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e5d000640 1 -- 172.21.15.141:0/3438182421 learned_addr learned my addr 172.21.15.141:0/3438182421 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e4f200640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e58143ae0 0x7f7e581f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.377+0000 7f7e57e00640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580cf7d0 0x7f7e58141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5d000640 1 -- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e58143ae0 msgr2=0x7f7e581f2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5d000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e58143ae0 0x7f7e581f2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5d000640 1 -- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580cf7d0 msgr2=0x7f7e58141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5d000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580cf7d0 0x7f7e58141b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5d000640 1 -- 172.21.15.141:0/3438182421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e400094d0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5d000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e5815e1e0 0x7f7e5813d7f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000c7e0 tx=0x7f7e5000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e54c00640 1 -- 172.21.15.141:0/3438182421 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000a640 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e54c00640 1 -- 172.21.15.141:0/3438182421 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5000a7c0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e54c00640 1 -- 172.21.15.141:0/3438182421 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000ab70 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e581f36e0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.378+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e581f39c0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.380+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e581f63c0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.380+0000 7f7e54c00640 1 -- 172.21.15.141:0/3438182421 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7e5000acd0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.381+0000 7f7e54c00640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e3006b860 0x7f7e3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.381+0000 7f7e54c00640 1 -- 172.21.15.141:0/3438182421 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e50089950 con 0x7f7e5815e1e0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.382+0000 7f7e57e00640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e3006b860 0x7f7e3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.383+0000 7f7e57e00640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e3006b860 0x7f7e3006dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7e48006fd0 tx=0x7f7e48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.385+0000 7f7e54c00640 1 -- 172.21.15.141:0/3438182421 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e581f63c0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.576+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e581f63c0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.578+0000 7f7e54c00640 1 -- 172.21.15.141:0/3438182421 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+12905 (secure 0 0 0) 0x7f7e581f63c0 con 0x7f7e5815e1e0 2024-01-16T15:02:25.559 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:25.559 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":55,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:02:24.651278+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:25.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e3006b860 msgr2=0x7f7e3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e3006b860 0x7f7e3006dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7e48006fd0 tx=0x7f7e48008040 comp rx=0 tx=0).stop 2024-01-16T15:02:25.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e5815e1e0 msgr2=0x7f7e5813d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e5815e1e0 0x7f7e5813d7f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000c7e0 tx=0x7f7e5000cca0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 shutdown_connections 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e3006b860 0x7f7e3006dd20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e5815e1e0 0x7f7e5813d7f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e58143ae0 0x7f7e581f2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 --2- 172.21.15.141:0/3438182421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580cf7d0 0x7f7e58141b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.580+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 >> 172.21.15.141:0/3438182421 conn(0x7f7e5806a800 msgr2=0x7f7e58142bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.581+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 shutdown_connections 2024-01-16T15:02:25.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.581+0000 7f7e5f000640 1 -- 172.21.15.141:0/3438182421 wait complete. 2024-01-16T15:02:25.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:25.580 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:25.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.809+0000 7fe407400640 1 Processor -- start 2024-01-16T15:02:25.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.810+0000 7fe407400640 1 -- start start 2024-01-16T15:02:25.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.811+0000 7fe407400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe4000cf9a0 0x7fe400166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.811+0000 7fe407400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40015c5f0 con 0x7fe40013e360 2024-01-16T15:02:25.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.811+0000 7fe407400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40006c470 con 0x7fe40013df80 2024-01-16T15:02:25.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.811+0000 7fe407400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40006c750 con 0x7fe4000cf9a0 2024-01-16T15:02:25.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.812+0000 7fe405400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe4000cf9a0 0x7fe400166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.812+0000 7fe3f7c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe40013e360 0x7fe4000cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42768/0 (socket says 172.21.15.141:42768) 2024-01-16T15:02:25.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.812+0000 7fe3f7c00640 1 -- 172.21.15.141:0/85999882 learned_addr learned my addr 172.21.15.141:0/85999882 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:25.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.812+0000 7fe405400640 1 -- 172.21.15.141:0/85999882 >> v1:172.21.15.188:6789/0 conn(0x7fe40013df80 legacy=0x7fe40015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:25.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.812+0000 7fe405400640 1 -- 172.21.15.141:0/85999882 >> v1:172.21.15.141:6789/0 conn(0x7fe40013e360 legacy=0x7fe4000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.812+0000 7fe405400640 1 -- 172.21.15.141:0/85999882 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40006ca30 con 0x7fe4000cf9a0 2024-01-16T15:02:25.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.813+0000 7fe405400640 1 --2- 172.21.15.141:0/85999882 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400166030 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009d90 tx=0x7fe3f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87285c9ab9102a57 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.813+0000 7fe3ffe00640 1 -- 172.21.15.141:0/85999882 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f802cc60 con 0x7fe4000cf9a0 2024-01-16T15:02:25.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.813+0000 7fe3ffe00640 1 -- 172.21.15.141:0/85999882 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f802cdc0 con 0x7fe4000cf9a0 2024-01-16T15:02:25.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.813+0000 7fe3ffe00640 1 -- 172.21.15.141:0/85999882 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8034660 con 0x7fe4000cf9a0 2024-01-16T15:02:25.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.814+0000 7fe407400640 1 -- 172.21.15.141:0/85999882 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 msgr2=0x7fe400166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.814+0000 7fe407400640 1 --2- 172.21.15.141:0/85999882 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400166030 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009d90 tx=0x7fe3f802beb0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.814+0000 7fe407400640 1 -- 172.21.15.141:0/85999882 shutdown_connections 2024-01-16T15:02:25.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.814+0000 7fe407400640 1 --2- 172.21.15.141:0/85999882 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400166030 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.814+0000 7fe407400640 1 -- 172.21.15.141:0/85999882 >> 172.21.15.141:0/85999882 conn(0x7fe40006a800 msgr2=0x7fe40006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.814+0000 7fe407400640 1 -- 172.21.15.141:0/85999882 shutdown_connections 2024-01-16T15:02:25.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.814+0000 7fe407400640 1 -- 172.21.15.141:0/85999882 wait complete. 2024-01-16T15:02:25.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.816+0000 7fe407400640 1 Processor -- start 2024-01-16T15:02:25.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.816+0000 7fe407400640 1 -- start start 2024-01-16T15:02:25.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe407400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe407400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe40013df80 0x7fe400158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe407400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe40013e360 0x7fe400133470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe407400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe400159af0 con 0x7fe4000cf9a0 2024-01-16T15:02:25.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe407400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe400153a20 con 0x7fe40013df80 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe407400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe400158e70 con 0x7fe40013e360 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe404a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe404a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400159f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34996/0 (socket says 172.21.15.141:34996) 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.817+0000 7fe404a00640 1 -- 172.21.15.141:0/77597601 learned_addr learned my addr 172.21.15.141:0/77597601 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe404a00640 1 -- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe40013e360 msgr2=0x7fe400133470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe404a00640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe40013e360 0x7fe400133470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe404a00640 1 -- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe40013df80 msgr2=0x7fe400158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe3f7c00640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe40013df80 0x7fe400158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe404a00640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe40013df80 0x7fe400158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe404a00640 1 -- 172.21.15.141:0/77597601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8009a70 con 0x7fe4000cf9a0 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe404a00640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400159f10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000b3f0 tx=0x7fe3f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe3fd600640 1 -- 172.21.15.141:0/77597601 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0011830 con 0x7fe4000cf9a0 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe3fd600640 1 -- 172.21.15.141:0/77597601 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f0011990 con 0x7fe4000cf9a0 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe3fd600640 1 -- 172.21.15.141:0/77597601 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0011d40 con 0x7fe4000cf9a0 2024-01-16T15:02:25.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.818+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe400159150 con 0x7fe4000cf9a0 2024-01-16T15:02:25.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.819+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe400159710 con 0x7fe4000cf9a0 2024-01-16T15:02:25.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.820+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe400159c30 con 0x7fe4000cf9a0 2024-01-16T15:02:25.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.821+0000 7fe3fd600640 1 -- 172.21.15.141:0/77597601 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe3f001a400 con 0x7fe4000cf9a0 2024-01-16T15:02:25.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.822+0000 7fe3fd600640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe3d806b890 0x7fe3d806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.822+0000 7fe3fd600640 1 -- 172.21.15.141:0/77597601 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe3f008a0e0 con 0x7fe4000cf9a0 2024-01-16T15:02:25.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.822+0000 7fe3f7c00640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe3d806b890 0x7fe3d806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.823+0000 7fe3f7c00640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe3d806b890 0x7fe3d806dd50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe4001548f0 tx=0x7fe3e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:25.826+0000 7fe3fd600640 1 -- 172.21.15.141:0/77597601 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe400159c30 con 0x7fe4000cf9a0 2024-01-16T15:02:25.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.015+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe400159c30 con 0x7fe4000cf9a0 2024-01-16T15:02:25.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.016+0000 7fe3fd600640 1 -- 172.21.15.141:0/77597601 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3f0054aa0 con 0x7fe4000cf9a0 2024-01-16T15:02:25.999 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:02:25.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe3d806b890 msgr2=0x7fe3d806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe3d806b890 0x7fe3d806dd50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe4001548f0 tx=0x7fe3e8009040 comp rx=0 tx=0).stop 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 msgr2=0x7fe400159f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400159f10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000b3f0 tx=0x7fe3f000b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 shutdown_connections 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe3d806b890 0x7fe3d806dd50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe40013e360 0x7fe400133470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe40013df80 0x7fe400158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.018+0000 7fe407400640 1 --2- 172.21.15.141:0/77597601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4000cf9a0 0x7fe400159f10 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.019+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 >> 172.21.15.141:0/77597601 conn(0x7fe40006a800 msgr2=0x7fe40006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.019+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 shutdown_connections 2024-01-16T15:02:26.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.019+0000 7fe407400640 1 -- 172.21.15.141:0/77597601 wait complete. 2024-01-16T15:02:26.020 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:02:26.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.231+0000 7f1d54000640 1 Processor -- start 2024-01-16T15:02:26.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.231+0000 7f1d54000640 1 -- start start 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.232+0000 7f1d54000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1d4c0ae5a0 0x7f1d4c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d54000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 0x7f1d4c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d54000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4c070c40 con 0x7f1d4c0b3ae0 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d54000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4c071450 con 0x7f1d4c0ae5a0 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d54000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c0cfc60 con 0x7f1d4c0cf290 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d50c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1d4c0b3ae0 0x7f1d4c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42776/0 (socket says 172.21.15.141:42776) 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d50c00640 1 -- 172.21.15.141:0/1651493718 learned_addr learned my addr 172.21.15.141:0/1651493718 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d51600640 1 --2- 172.21.15.141:0/1651493718 >> v2:172.21.15.188:3300/0 conn(0x7f1d4c0ae5a0 0x7f1d4c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d52000640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 0x7f1d4c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416555153 0 0) 0x7f1d4c070c40 con 0x7f1d4c0b3ae0 2024-01-16T15:02:26.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.233+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c003520 con 0x7f1d4c0b3ae0 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2661676908 0 0) 0x7f1d2c003520 con 0x7f1d4c0b3ae0 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 msgr2=0x7f1d4c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 0x7f1d4c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 >> v2:172.21.15.188:3300/0 conn(0x7f1d4c0ae5a0 msgr2=0x7f1d4c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 --2- 172.21.15.141:0/1651493718 >> v2:172.21.15.188:3300/0 conn(0x7f1d4c0ae5a0 0x7f1d4c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4c06c470 con 0x7f1d4c0b3ae0 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1d34001670 con 0x7f1d4c0b3ae0 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 >> v1:172.21.15.141:6789/0 conn(0x7f1d4c0b3ae0 legacy=0x7f1d4c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d2c003970 0x7f1d2c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 0x7f1d2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c071450 con 0x7f1d2c003970 2024-01-16T15:02:26.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c0cfc60 con 0x7f1d2c0047c0 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c00ba00 con 0x7f1d4c0cf290 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d51600640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d2c003970 0x7f1d2c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.234+0000 7f1d52000640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 0x7f1d2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.235+0000 7f1d50c00640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.235+0000 7f1d51600640 1 -- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 msgr2=0x7f1d2c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.235+0000 7f1d51600640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d4c0cf290 0x7f1d2c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.235+0000 7f1d51600640 1 -- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d2c0047c0 msgr2=0x7f1d2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.235+0000 7f1d51600640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.235+0000 7f1d51600640 1 -- 172.21.15.141:0/1651493718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c003520 con 0x7f1d2c003970 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.235+0000 7f1d50c00640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.236+0000 7f1d51600640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d2c003970 0x7f1d2c003d90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009ac0 tx=0x7f1d2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800a91c26ba6b6f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.236+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c02c710 con 0x7f1d2c003970 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.236+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c02c870 con 0x7f1d2c003970 2024-01-16T15:02:26.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.236+0000 7f1d4be00640 1 -- 172.21.15.141:0/1651493718 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c02cc00 con 0x7f1d2c003970 2024-01-16T15:02:26.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.237+0000 7f1d54000640 1 -- 172.21.15.141:0/1651493718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d2c003970 msgr2=0x7f1d2c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.237+0000 7f1d54000640 1 --2- 172.21.15.141:0/1651493718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d2c003970 0x7f1d2c003d90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009ac0 tx=0x7f1d2c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.237+0000 7f1d52000640 1 -- 172.21.15.141:0/1651493718 reap_dead start 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.237+0000 7f1d54000640 1 -- 172.21.15.141:0/1651493718 shutdown_connections 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.237+0000 7f1d54000640 1 -- 172.21.15.141:0/1651493718 >> 172.21.15.141:0/1651493718 conn(0x7f1d4c06a800 msgr2=0x7f1d4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.237+0000 7f1d54000640 1 -- 172.21.15.141:0/1651493718 shutdown_connections 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.237+0000 7f1d54000640 1 -- 172.21.15.141:0/1651493718 wait complete. 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.239+0000 7f1d54000640 1 Processor -- start 2024-01-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.239+0000 7f1d54000640 1 -- start start 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d54000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2c003970 0x7f1d4c156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d54000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d4c06c900 0x7f1d4c06cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d54000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d4c15ed40 0x7f1d4c15f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d54000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c06d2a0 con 0x7f1d4c15ed40 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d54000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c15c7c0 con 0x7f1d4c06c900 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d54000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c15ca60 con 0x7f1d2c003970 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d4c15ed40 0x7f1d4c15f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d4c15ed40 0x7f1d4c15f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35014/0 (socket says 172.21.15.141:35014) 2024-01-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 -- 172.21.15.141:0/2821769545 learned_addr learned my addr 172.21.15.141:0/2821769545 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 -- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2c003970 msgr2=0x7f1d4c156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2c003970 0x7f1d4c156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 -- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d4c06c900 msgr2=0x7f1d4c06cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d4c06c900 0x7f1d4c06cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 -- 172.21.15.141:0/2821769545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4c15cd40 con 0x7f1d4c15ed40 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.240+0000 7f1d52000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d4c15ed40 0x7f1d4c15f1c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c0cf670 tx=0x7f1d440115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.241+0000 7f1d49600640 1 -- 172.21.15.141:0/2821769545 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4400e070 con 0x7f1d4c15ed40 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.241+0000 7f1d49600640 1 -- 172.21.15.141:0/2821769545 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d44011e10 con 0x7f1d4c15ed40 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.241+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4c15d020 con 0x7f1d4c15ed40 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.241+0000 7f1d49600640 1 -- 172.21.15.141:0/2821769545 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d44003990 con 0x7f1d4c15ed40 2024-01-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.241+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4c156fa0 con 0x7f1d4c15ed40 2024-01-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.242+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4c0675d0 con 0x7f1d4c15ed40 2024-01-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.243+0000 7f1d49600640 1 -- 172.21.15.141:0/2821769545 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1d44003af0 con 0x7f1d4c15ed40 2024-01-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.244+0000 7f1d49600640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1d2406b860 0x7f1d2406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.244+0000 7f1d49600640 1 -- 172.21.15.141:0/2821769545 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1d4408ad30 con 0x7f1d4c15ed40 2024-01-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.245+0000 7f1d51600640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1d2406b860 0x7f1d2406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.246+0000 7f1d51600640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1d2406b860 0x7f1d2406dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c036000 tx=0x7f1d3c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.248+0000 7f1d49600640 1 -- 172.21.15.141:0/2821769545 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d440555f0 con 0x7f1d4c15ed40 2024-01-16T15:02:26.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.440+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1d4c1f2ef0 con 0x7f1d4c15ed40 2024-01-16T15:02:26.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.440+0000 7f1d49600640 1 -- 172.21.15.141:0/2821769545 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v55) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1d44058c10 con 0x7f1d4c15ed40 2024-01-16T15:02:26.421 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:26.421 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:02:26.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:26.421 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T15:02:26.422 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T15:02:26.422 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:26.422 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:02:26.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1d2406b860 msgr2=0x7f1d2406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1d2406b860 0x7f1d2406dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c036000 tx=0x7f1d3c02e000 comp rx=0 tx=0).stop 2024-01-16T15:02:26.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d4c15ed40 msgr2=0x7f1d4c15f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d4c15ed40 0x7f1d4c15f1c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c0cf670 tx=0x7f1d440115a0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 shutdown_connections 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1d2406b860 0x7f1d2406dd20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1d4c15ed40 0x7f1d4c15f1c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d4c06c900 0x7f1d4c06cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 --2- 172.21.15.141:0/2821769545 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1d2c003970 0x7f1d4c156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.443+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 >> 172.21.15.141:0/2821769545 conn(0x7f1d4c06a800 msgr2=0x7f1d4c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.444+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 shutdown_connections 2024-01-16T15:02:26.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.444+0000 7f1d54000640 1 -- 172.21.15.141:0/2821769545 wait complete. 2024-01-16T15:02:26.444 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:02:26.444 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:02:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.680+0000 7f2e2fe00640 1 Processor -- start 2024-01-16T15:02:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.680+0000 7f2e2fe00640 1 -- start start 2024-01-16T15:02:26.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.681+0000 7f2e2fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2e300b3550 0x7f2e300c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.681+0000 7f2e2fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300c43f0 0x7f2e300ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.681+0000 7f2e2fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e30161bf0 con 0x7f2e30161810 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.681+0000 7f2e2fe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300c48a0 con 0x7f2e300b3550 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.681+0000 7f2e2fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3006b620 con 0x7f2e300c43f0 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2e300b3550 0x7f2e300c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2e300b3550 0x7f2e300c3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35024/0 (socket says 172.21.15.141:35024) 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2e30161810 0x7f2e300ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42792/0 (socket says 172.21.15.141:42792) 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2e000640 1 -- 172.21.15.141:0/2856130028 learned_addr learned my addr 172.21.15.141:0/2856130028 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2e000640 1 -- 172.21.15.141:0/2856130028 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300c43f0 msgr2=0x7f2e300ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2e000640 1 --2- 172.21.15.141:0/2856130028 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300c43f0 0x7f2e300ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2e000640 1 -- 172.21.15.141:0/2856130028 >> v1:172.21.15.141:6789/0 conn(0x7f2e30161810 legacy=0x7f2e300ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.682+0000 7f2e2e000640 1 -- 172.21.15.141:0/2856130028 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3006b900 con 0x7f2e300b3550 2024-01-16T15:02:26.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.683+0000 7f2e2e000640 1 --2- 172.21.15.141:0/2856130028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e300b3550 0x7f2e300c3eb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009e50 tx=0x7f2e1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf13860e3964a21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.684+0000 7f2e2d600640 1 -- 172.21.15.141:0/2856130028 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1802ca80 con 0x7f2e300b3550 2024-01-16T15:02:26.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.684+0000 7f2e2d600640 1 -- 172.21.15.141:0/2856130028 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1802cbe0 con 0x7f2e300b3550 2024-01-16T15:02:26.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.684+0000 7f2e2d600640 1 -- 172.21.15.141:0/2856130028 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e180343f0 con 0x7f2e300b3550 2024-01-16T15:02:26.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.684+0000 7f2e2fe00640 1 -- 172.21.15.141:0/2856130028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e300b3550 msgr2=0x7f2e300c3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.684+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/2856130028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e300b3550 0x7f2e300c3eb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009e50 tx=0x7f2e1802bee0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.685+0000 7f2e2fe00640 1 -- 172.21.15.141:0/2856130028 shutdown_connections 2024-01-16T15:02:26.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.685+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/2856130028 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300c43f0 0x7f2e300ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.685+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/2856130028 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e300b3550 0x7f2e300c3eb0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.685+0000 7f2e2fe00640 1 -- 172.21.15.141:0/2856130028 >> 172.21.15.141:0/2856130028 conn(0x7f2e300b2a30 msgr2=0x7f2e300b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.685+0000 7f2e2fe00640 1 -- 172.21.15.141:0/2856130028 shutdown_connections 2024-01-16T15:02:26.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.685+0000 7f2e2fe00640 1 -- 172.21.15.141:0/2856130028 wait complete. 2024-01-16T15:02:26.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.686+0000 7f2e2fe00640 1 Processor -- start 2024-01-16T15:02:26.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.687+0000 7f2e2fe00640 1 -- start start 2024-01-16T15:02:26.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300b3550 0x7f2e3015c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e300c43f0 0x7f2e300b5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e30161810 0x7f2e300b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3015cec0 con 0x7f2e30161810 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300c5790 con 0x7f2e300c43f0 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300c5a70 con 0x7f2e300b3550 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e30161810 0x7f2e300b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e30161810 0x7f2e300b0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35038/0 (socket says 172.21.15.141:35038) 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 -- 172.21.15.141:0/1607632076 learned_addr learned my addr 172.21.15.141:0/1607632076 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2ea00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300b3550 0x7f2e3015c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2e000640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e300c43f0 0x7f2e300b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 -- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300b3550 msgr2=0x7f2e3015c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300b3550 0x7f2e3015c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 -- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e300c43f0 msgr2=0x7f2e300b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e300c43f0 0x7f2e300b5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.688+0000 7f2e2f400640 1 -- 172.21.15.141:0/1607632076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009b30 con 0x7f2e30161810 2024-01-16T15:02:26.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.689+0000 7f2e2f400640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e30161810 0x7f2e300b0730 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009fd0 tx=0x7f2e2800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.689+0000 7f2e26a00640 1 -- 172.21.15.141:0/1607632076 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28011830 con 0x7f2e30161810 2024-01-16T15:02:26.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.689+0000 7f2e26a00640 1 -- 172.21.15.141:0/1607632076 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e28011990 con 0x7f2e30161810 2024-01-16T15:02:26.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.689+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300c5d50 con 0x7f2e30161810 2024-01-16T15:02:26.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.689+0000 7f2e26a00640 1 -- 172.21.15.141:0/1607632076 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28011d20 con 0x7f2e30161810 2024-01-16T15:02:26.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.689+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300c6310 con 0x7f2e30161810 2024-01-16T15:02:26.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.690+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e300c6500 con 0x7f2e30161810 2024-01-16T15:02:26.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.691+0000 7f2e26a00640 1 -- 172.21.15.141:0/1607632076 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f2e2801b460 con 0x7f2e30161810 2024-01-16T15:02:26.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.692+0000 7f2e26a00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e0806b860 0x7f2e0806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.692+0000 7f2e26a00640 1 -- 172.21.15.141:0/1607632076 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2e2808a1e0 con 0x7f2e30161810 2024-01-16T15:02:26.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.692+0000 7f2e2ea00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e0806b860 0x7f2e0806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.694+0000 7f2e2ea00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e0806b860 0x7f2e0806dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2e20007fd0 tx=0x7f2e20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.696+0000 7f2e26a00640 1 -- 172.21.15.141:0/1607632076 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e280584f0 con 0x7f2e30161810 2024-01-16T15:02:26.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:26.887+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2e30067b60 con 0x7f2e30161810 2024-01-16T15:02:27.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.653+0000 7f2e26a00640 1 -- 172.21.15.141:0/1607632076 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v56) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2e28058080 con 0x7f2e30161810 2024-01-16T15:02:27.634 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T15:02:27.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.656+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e0806b860 msgr2=0x7f2e0806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.656+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e0806b860 0x7f2e0806dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2e20007fd0 tx=0x7f2e20009040 comp rx=0 tx=0).stop 2024-01-16T15:02:27.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.656+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e30161810 msgr2=0x7f2e300b0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.656+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e30161810 0x7f2e300b0730 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009fd0 tx=0x7f2e2800b8e0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 shutdown_connections 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e0806b860 0x7f2e0806dd20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e30161810 0x7f2e300b0730 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e300c43f0 0x7f2e300b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 --2- 172.21.15.141:0/1607632076 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e300b3550 0x7f2e3015c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 >> 172.21.15.141:0/1607632076 conn(0x7f2e300b2a30 msgr2=0x7f2e3006f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 shutdown_connections 2024-01-16T15:02:27.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.657+0000 7f2e2fe00640 1 -- 172.21.15.141:0/1607632076 wait complete. 2024-01-16T15:02:27.664 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:02:27.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.883+0000 7fe08f200640 1 Processor -- start 2024-01-16T15:02:27.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.883+0000 7fe08f200640 1 -- start start 2024-01-16T15:02:27.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe0900d0100 0x7fe0900b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08f200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0900d0a10 con 0x7fe090160920 2024-01-16T15:02:27.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0900cf080 con 0x7fe0900d0f40 2024-01-16T15:02:27.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08f200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0900c6be0 con 0x7fe0900d0100 2024-01-16T15:02:27.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe090160920 0x7fe0900cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42798/0 (socket says 172.21.15.141:42798) 2024-01-16T15:02:27.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08de00640 1 -- 172.21.15.141:0/2180984061 learned_addr learned my addr 172.21.15.141:0/2180984061 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08d400640 1 --2- 172.21.15.141:0/2180984061 >> v2:172.21.15.188:3300/0 conn(0x7fe0900d0100 0x7fe0900b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.885+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240705136 0 0) 0x7fe0900d0a10 con 0x7fe090160920 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe070003520 con 0x7fe090160920 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377172139 0 0) 0x7fe070003520 con 0x7fe090160920 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 >> v2:172.21.15.188:3300/0 conn(0x7fe0900d0100 msgr2=0x7fe0900b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 --2- 172.21.15.141:0/2180984061 >> v2:172.21.15.188:3300/0 conn(0x7fe0900d0100 0x7fe0900b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 >> v1:172.21.15.188:6789/0 conn(0x7fe0900d0f40 legacy=0x7fe0900d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0900ce7e0 con 0x7fe090160920 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe0800019f0 con 0x7fe090160920 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 >> v1:172.21.15.141:6789/0 conn(0x7fe090160920 legacy=0x7fe0900cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe070003970 0x7fe070003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0700047c0 0x7fe070004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe070005180 0x7fe07000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900c6be0 con 0x7fe0700047c0 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07000c020 con 0x7fe070003970 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07000c1e0 con 0x7fe070005180 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.886+0000 7fe08d400640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0700047c0 0x7fe070004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.887+0000 7fe08de00640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe070003970 0x7fe070003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.887+0000 7fe08d400640 1 -- 172.21.15.141:0/2180984061 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe070005180 msgr2=0x7fe07000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.887+0000 7fe08d400640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe070005180 0x7fe07000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.887+0000 7fe08d400640 1 -- 172.21.15.141:0/2180984061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe070003970 msgr2=0x7fe070003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.887+0000 7fe08d400640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe070003970 0x7fe070003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.887+0000 7fe08e800640 1 -- 172.21.15.141:0/2180984061 reap_dead start 2024-01-16T15:02:27.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.887+0000 7fe08d400640 1 -- 172.21.15.141:0/2180984061 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070003520 con 0x7fe0700047c0 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.888+0000 7fe08d400640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0700047c0 0x7fe070004c40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe078009ac0 tx=0x7fe0700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0d5ffde9fd794b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.888+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07802c6b0 con 0x7fe0700047c0 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.888+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07802c810 con 0x7fe0700047c0 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.888+0000 7fe08ca00640 1 -- 172.21.15.141:0/2180984061 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07802cba0 con 0x7fe0700047c0 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.889+0000 7fe08f200640 1 -- 172.21.15.141:0/2180984061 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0700047c0 msgr2=0x7fe070004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.889+0000 7fe08f200640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0700047c0 0x7fe070004c40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe078009ac0 tx=0x7fe0700057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.889+0000 7fe08f200640 1 -- 172.21.15.141:0/2180984061 shutdown_connections 2024-01-16T15:02:27.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.889+0000 7fe08f200640 1 --2- 172.21.15.141:0/2180984061 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0700047c0 0x7fe070004c40 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.889+0000 7fe08f200640 1 -- 172.21.15.141:0/2180984061 >> 172.21.15.141:0/2180984061 conn(0x7fe0900d3090 msgr2=0x7fe0900d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:27.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.889+0000 7fe08f200640 1 -- 172.21.15.141:0/2180984061 shutdown_connections 2024-01-16T15:02:27.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.889+0000 7fe08f200640 1 -- 172.21.15.141:0/2180984061 wait complete. 2024-01-16T15:02:27.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.891+0000 7fe08f200640 1 Processor -- start 2024-01-16T15:02:27.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.891+0000 7fe08f200640 1 -- start start 2024-01-16T15:02:27.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0700047c0 0x7fe0900b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0900b5c50 0x7fe0900cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0900b79b0 0x7fe0900b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe090152080 con 0x7fe0900b5c50 2024-01-16T15:02:27.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900b8370 con 0x7fe0900b79b0 2024-01-16T15:02:27.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900b6180 con 0x7fe0700047c0 2024-01-16T15:02:27.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0900b5c50 0x7fe0900cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0900b5c50 0x7fe0900cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35064/0 (socket says 172.21.15.141:35064) 2024-01-16T15:02:27.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.892+0000 7fe08d400640 1 -- 172.21.15.141:0/3297810102 learned_addr learned my addr 172.21.15.141:0/3297810102 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:27.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08d400640 1 -- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0700047c0 msgr2=0x7fe0900b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08d400640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0700047c0 0x7fe0900b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08d400640 1 -- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0900b79b0 msgr2=0x7fe0900b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08d400640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0900b79b0 0x7fe0900b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08d400640 1 -- 172.21.15.141:0/3297810102 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0900b6460 con 0x7fe0900b5c50 2024-01-16T15:02:27.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08d400640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0900b5c50 0x7fe0900cbc20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe07802c250 tx=0x7fe078004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe086000640 1 -- 172.21.15.141:0/3297810102 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078003810 con 0x7fe0900b5c50 2024-01-16T15:02:27.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe086000640 1 -- 172.21.15.141:0/3297810102 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe078003970 con 0x7fe0900b5c50 2024-01-16T15:02:27.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe086000640 1 -- 172.21.15.141:0/3297810102 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078003d00 con 0x7fe0900b5c50 2024-01-16T15:02:27.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0900b6740 con 0x7fe0900b5c50 2024-01-16T15:02:27.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.893+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0900b6d00 con 0x7fe0900b5c50 2024-01-16T15:02:27.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.895+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0900b6ec0 con 0x7fe0900b5c50 2024-01-16T15:02:27.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.896+0000 7fe086000640 1 -- 172.21.15.141:0/3297810102 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe07803e400 con 0x7fe0900b5c50 2024-01-16T15:02:27.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.897+0000 7fe086000640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe06806b890 0x7fe06806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.897+0000 7fe086000640 1 -- 172.21.15.141:0/3297810102 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe0780ac8a0 con 0x7fe0900b5c50 2024-01-16T15:02:27.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.897+0000 7fe08de00640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe06806b890 0x7fe06806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.899+0000 7fe08de00640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe06806b890 0x7fe06806dd50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe080001b20 tx=0x7fe080009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:27.901+0000 7fe086000640 1 -- 172.21.15.141:0/3297810102 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe078077170 con 0x7fe0900b5c50 2024-01-16T15:02:28.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.091+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe090067450 con 0x7fe0900b5c50 2024-01-16T15:02:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.659+0000 7fe086000640 1 -- 172.21.15.141:0/3297810102 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v57) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe078003e60 con 0x7fe0900b5c50 2024-01-16T15:02:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe06806b890 msgr2=0x7fe06806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe06806b890 0x7fe06806dd50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe080001b20 tx=0x7fe080009040 comp rx=0 tx=0).stop 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0900b5c50 msgr2=0x7fe0900cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0900b5c50 0x7fe0900cbc20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe07802c250 tx=0x7fe078004730 comp rx=0 tx=0).stop 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 shutdown_connections 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe06806b890 0x7fe06806dd50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0900b79b0 0x7fe0900b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0900b5c50 0x7fe0900cbc20 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 --2- 172.21.15.141:0/3297810102 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0700047c0 0x7fe0900b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 >> 172.21.15.141:0/3297810102 conn(0x7fe0900d3090 msgr2=0x7fe09006aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 shutdown_connections 2024-01-16T15:02:28.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.663+0000 7fe08f200640 1 -- 172.21.15.141:0/3297810102 wait complete. 2024-01-16T15:02:28.674 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:02:28.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.875+0000 7fe2d3a00640 1 Processor -- start 2024-01-16T15:02:28.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.876+0000 7fe2d3a00640 1 -- start start 2024-01-16T15:02:28.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d3a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2cc0c9870 0x7fe2cc0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d3a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc0cf4c0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d3a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc0d2ef0 con 0x7fe2cc0c9870 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c9c50 con 0x7fe2cc0d0560 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2cc0c9870 0x7fe2cc0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2cc0c9870 0x7fe2cc0ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35072/0 (socket says 172.21.15.141:35072) 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d1000640 1 -- 172.21.15.141:0/282843213 learned_addr learned my addr 172.21.15.141:0/282843213 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.877+0000 7fe2d1a00640 1 --1- 172.21.15.141:0/282843213 >> v1:172.21.15.141:6789/0 conn(0x7fe2cc0d2260 0x7fe2cc0d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42802/0 (socket says 172.21.15.141:42802) 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.878+0000 7fe2cbe00640 1 --2- 172.21.15.141:0/282843213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.878+0000 7fe2d1000640 1 -- 172.21.15.141:0/282843213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 msgr2=0x7fe2cc161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.878+0000 7fe2d1000640 1 --2- 172.21.15.141:0/282843213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.878+0000 7fe2d1000640 1 -- 172.21.15.141:0/282843213 >> v1:172.21.15.141:6789/0 conn(0x7fe2cc0d2260 legacy=0x7fe2cc0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.878+0000 7fe2cb400640 1 -- 172.21.15.141:0/282843213 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967200711 0 0) 0x7fe2cc0cf4c0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.878+0000 7fe2d1000640 1 -- 172.21.15.141:0/282843213 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc0cf7a0 con 0x7fe2cc0c9870 2024-01-16T15:02:28.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.878+0000 7fe2cbe00640 1 --2- 172.21.15.141:0/282843213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:28.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.879+0000 7fe2d1000640 1 --2- 172.21.15.141:0/282843213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0c9870 0x7fe2cc0ceb50 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009870 tx=0x7fe2bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190c33f1fc1c354c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.879+0000 7fe2cb400640 1 -- 172.21.15.141:0/282843213 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc030840 con 0x7fe2cc0c9870 2024-01-16T15:02:28.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.879+0000 7fe2cb400640 1 -- 172.21.15.141:0/282843213 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc0309a0 con 0x7fe2cc0c9870 2024-01-16T15:02:28.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.879+0000 7fe2cb400640 1 -- 172.21.15.141:0/282843213 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc030d30 con 0x7fe2cc0c9870 2024-01-16T15:02:28.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 -- 172.21.15.141:0/282843213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0c9870 msgr2=0x7fe2cc0ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/282843213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0c9870 0x7fe2cc0ceb50 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009870 tx=0x7fe2bc02b980 comp rx=0 tx=0).stop 2024-01-16T15:02:28.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 -- 172.21.15.141:0/282843213 shutdown_connections 2024-01-16T15:02:28.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/282843213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/282843213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0c9870 0x7fe2cc0ceb50 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 -- 172.21.15.141:0/282843213 >> 172.21.15.141:0/282843213 conn(0x7fe2cc09c560 msgr2=0x7fe2cc0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:28.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 -- 172.21.15.141:0/282843213 shutdown_connections 2024-01-16T15:02:28.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.880+0000 7fe2d3a00640 1 -- 172.21.15.141:0/282843213 wait complete. 2024-01-16T15:02:28.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.882+0000 7fe2d3a00640 1 Processor -- start 2024-01-16T15:02:28.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.882+0000 7fe2d3a00640 1 -- start start 2024-01-16T15:02:28.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.883+0000 7fe2d3a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc0c9870 0x7fe2cc15e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.883+0000 7fe2d3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc0b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.883+0000 7fe2d3a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0d2260 0x7fe2cc0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.883+0000 7fe2d3a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc09ed00 con 0x7fe2cc0d2260 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.883+0000 7fe2d3a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0b3a60 con 0x7fe2cc0c9870 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.883+0000 7fe2d3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0b5f10 con 0x7fe2cc0d0560 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.883+0000 7fe2d1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0d2260 0x7fe2cc0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0d2260 0x7fe2cc0b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35088/0 (socket says 172.21.15.141:35088) 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 -- 172.21.15.141:0/3603530209 learned_addr learned my addr 172.21.15.141:0/3603530209 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1000640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc0c9870 0x7fe2cc15e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2cbe00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc0b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 -- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 msgr2=0x7fe2cc0b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc0b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 -- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc0c9870 msgr2=0x7fe2cc15e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc0c9870 0x7fe2cc15e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 -- 172.21.15.141:0/3603530209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0094d0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.884+0000 7fe2d1a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0d2260 0x7fe2cc0b86f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400c480 tx=0x7fe2c400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.885+0000 7fe2c8c00640 1 -- 172.21.15.141:0/3603530209 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40074e0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.885+0000 7fe2c8c00640 1 -- 172.21.15.141:0/3603530209 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c4007640 con 0x7fe2cc0d2260 2024-01-16T15:02:28.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.885+0000 7fe2c8c00640 1 -- 172.21.15.141:0/3603530209 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40049e0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.885+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc0b61f0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.885+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc0b67b0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.886+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc0675d0 con 0x7fe2cc0d2260 2024-01-16T15:02:28.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.887+0000 7fe2c8c00640 1 -- 172.21.15.141:0/3603530209 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe2c4004b40 con 0x7fe2cc0d2260 2024-01-16T15:02:28.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.888+0000 7fe2c8c00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.888+0000 7fe2c8c00640 1 -- 172.21.15.141:0/3603530209 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe2c4089790 con 0x7fe2cc0d2260 2024-01-16T15:02:28.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.888+0000 7fe2d1000640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.889+0000 7fe2d1000640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc036000 tx=0x7fe2bc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:28.893+0000 7fe2c8c00640 1 -- 172.21.15.141:0/3603530209 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2c4057910 con 0x7fe2cc0d2260 2024-01-16T15:02:29.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.118+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe2cc0d0a20 con 0x7fe2cc0d2260 2024-01-16T15:02:29.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.703+0000 7fe2c8c00640 1 -- 172.21.15.141:0/3603530209 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v33) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe2c4018020 con 0x7fe2cc0d2260 2024-01-16T15:02:29.686 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 8 and data pool 9 2024-01-16T15:02:29.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.717+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 msgr2=0x7fe2a406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.717+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc036000 tx=0x7fe2bc008030 comp rx=0 tx=0).stop 2024-01-16T15:02:29.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.717+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0d2260 msgr2=0x7fe2cc0b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.718+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0d2260 0x7fe2cc0b86f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400c480 tx=0x7fe2c400c940 comp rx=0 tx=0).stop 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 shutdown_connections 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0d2260 0x7fe2cc0b86f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0d0560 0x7fe2cc0b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 --2- 172.21.15.141:0/3603530209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc0c9870 0x7fe2cc15e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 >> 172.21.15.141:0/3603530209 conn(0x7fe2cc09c560 msgr2=0x7fe2cc0c6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 shutdown_connections 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.719+0000 7fe2d3a00640 1 -- 172.21.15.141:0/3603530209 wait complete. 2024-01-16T15:02:29.728 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:29.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.921+0000 7f8677e00640 1 Processor -- start 2024-01-16T15:02:29.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.921+0000 7f8677e00640 1 -- start start 2024-01-16T15:02:29.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.922+0000 7f8677e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 0x7f8670164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.922+0000 7f8677e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86700b1ee0 con 0x7f86700cf9a0 2024-01-16T15:02:29.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.922+0000 7f8677e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86700b2780 con 0x7f8670143ae0 2024-01-16T15:02:29.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.922+0000 7f8677e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867015e990 con 0x7f867015e3b0 2024-01-16T15:02:29.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f8674a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f86700cf9a0 0x7f867015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42818/0 (socket says 172.21.15.141:42818) 2024-01-16T15:02:29.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f8674a00640 1 -- 172.21.15.141:0/634102811 learned_addr learned my addr 172.21.15.141:0/634102811 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209262437 0 0) 0x7f86700b1ee0 con 0x7f86700cf9a0 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f8675e00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 0x7f8670164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8650005730 con 0x7f86700cf9a0 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4179870195 0 0) 0x7f8650005730 con 0x7f86700cf9a0 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 msgr2=0x7f8670164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 0x7f8670164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 >> v1:172.21.15.188:6789/0 conn(0x7f8670143ae0 legacy=0x7f86700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86700b2130 con 0x7f86700cf9a0 2024-01-16T15:02:29.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8658001670 con 0x7f86700cf9a0 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.923+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 >> v1:172.21.15.141:6789/0 conn(0x7f86700cf9a0 legacy=0x7f867015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f866fe00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8650005bb0 0x7f8650005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f866fe00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86500069d0 0x7f8650006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f866fe00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 0x7f8650007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867015e990 con 0x7f8650005bb0 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865000dc70 con 0x7f86500069d0 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865000de30 con 0x7f867015e3b0 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8675400640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8650005bb0 0x7f8650005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8674a00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86500069d0 0x7f8650006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8675400640 1 -- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 msgr2=0x7f8650007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8675e00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 0x7f8650007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8675400640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f867015e3b0 0x7f8650007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8675400640 1 -- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86500069d0 msgr2=0x7f8650006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:29.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8675400640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86500069d0 0x7f8650006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.924+0000 7f8675400640 1 -- 172.21.15.141:0/634102811 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650005730 con 0x7f8650005bb0 2024-01-16T15:02:29.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.925+0000 7f8675400640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8650005bb0 0x7f8650005fd0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8660001720 tx=0x7f8650007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73fd9ab24db7dd4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600052d0 con 0x7f8650005bb0 2024-01-16T15:02:29.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866002e050 con 0x7f8650005bb0 2024-01-16T15:02:29.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f866fe00640 1 -- 172.21.15.141:0/634102811 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600035f0 con 0x7f8650005bb0 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f8677e00640 1 -- 172.21.15.141:0/634102811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8650005bb0 msgr2=0x7f8650005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f8677e00640 1 --2- 172.21.15.141:0/634102811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8650005bb0 0x7f8650005fd0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8660001720 tx=0x7f8650007a00 comp rx=0 tx=0).stop 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f8675e00640 1 -- 172.21.15.141:0/634102811 reap_dead start 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f8677e00640 1 -- 172.21.15.141:0/634102811 shutdown_connections 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f8677e00640 1 -- 172.21.15.141:0/634102811 >> 172.21.15.141:0/634102811 conn(0x7f867006a800 msgr2=0x7f867006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f8677e00640 1 -- 172.21.15.141:0/634102811 shutdown_connections 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.926+0000 7f8677e00640 1 -- 172.21.15.141:0/634102811 wait complete. 2024-01-16T15:02:29.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.928+0000 7f8677e00640 1 Processor -- start 2024-01-16T15:02:29.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.928+0000 7f8677e00640 1 -- start start 2024-01-16T15:02:29.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8677e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8650005bb0 0x7f86701f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8677e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86700b0110 0x7f86700cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8677e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86700cdc30 0x7f86700af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8677e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86700acf90 con 0x7f86700cdc30 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8677e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86700ce0c0 con 0x7f86700b0110 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8677e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867015a240 con 0x7f8650005bb0 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8675e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86700cdc30 0x7f86700af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8675e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86700cdc30 0x7f86700af510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35100/0 (socket says 172.21.15.141:35100) 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.929+0000 7f8675e00640 1 -- 172.21.15.141:0/1557778649 learned_addr learned my addr 172.21.15.141:0/1557778649 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675400640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8650005bb0 0x7f86701f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8674a00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86700b0110 0x7f86700cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675e00640 1 -- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8650005bb0 msgr2=0x7f86701f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8650005bb0 0x7f86701f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675e00640 1 -- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86700b0110 msgr2=0x7f86700cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86700b0110 0x7f86700cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675e00640 1 -- 172.21.15.141:0/1557778649 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86700b0590 con 0x7f86700cdc30 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675400640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8650005bb0 0x7f86701f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8675e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86700cdc30 0x7f86700af510 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f8670071f10 tx=0x7f866800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f866d600640 1 -- 172.21.15.141:0/1557778649 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668017400 con 0x7f86700cdc30 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f866d600640 1 -- 172.21.15.141:0/1557778649 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8668011040 con 0x7f86700cdc30 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f866d600640 1 -- 172.21.15.141:0/1557778649 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86680177c0 con 0x7f86700cdc30 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.930+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86700b0810 con 0x7f86700cdc30 2024-01-16T15:02:29.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.931+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86700ac8d0 con 0x7f86700cdc30 2024-01-16T15:02:29.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.932+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86700ad0d0 con 0x7f86700cdc30 2024-01-16T15:02:29.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.933+0000 7f866d600640 1 -- 172.21.15.141:0/1557778649 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8668017920 con 0x7f86700cdc30 2024-01-16T15:02:29.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.934+0000 7f866d600640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864806b860 0x7f864806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.934+0000 7f866d600640 1 -- 172.21.15.141:0/1557778649 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f866808ab50 con 0x7f86700cdc30 2024-01-16T15:02:29.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.934+0000 7f8675400640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864806b860 0x7f864806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.936+0000 7f8675400640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864806b860 0x7f864806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8660033000 tx=0x7f8660009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:29.939+0000 7f866d600640 1 -- 172.21.15.141:0/1557778649 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86700ad0d0 con 0x7f86700cdc30 2024-01-16T15:02:30.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.124+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86700ad0d0 con 0x7f86700cdc30 2024-01-16T15:02:30.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.126+0000 7f866d600640 1 -- 172.21.15.141:0/1557778649 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16626 (secure 0 0 0) 0x7f86700ad0d0 con 0x7f86700cdc30 2024-01-16T15:02:30.107 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:30.107 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":58,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:02:29.120541+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:26.939197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:28.143123+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:30.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864806b860 msgr2=0x7f864806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864806b860 0x7f864806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8660033000 tx=0x7f8660009030 comp rx=0 tx=0).stop 2024-01-16T15:02:30.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86700cdc30 msgr2=0x7f86700af510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86700cdc30 0x7f86700af510 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f8670071f10 tx=0x7f866800f4a0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 shutdown_connections 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864806b860 0x7f864806dd20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86700cdc30 0x7f86700af510 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86700b0110 0x7f86700cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.128+0000 7f8677e00640 1 --2- 172.21.15.141:0/1557778649 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8650005bb0 0x7f86701f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.129+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 >> 172.21.15.141:0/1557778649 conn(0x7f867006a800 msgr2=0x7f8670140640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.129+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 shutdown_connections 2024-01-16T15:02:30.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.129+0000 7f8677e00640 1 -- 172.21.15.141:0/1557778649 wait complete. 2024-01-16T15:02:30.127 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:30.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.348+0000 7faa93000640 1 Processor -- start 2024-01-16T15:02:30.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.348+0000 7faa93000640 1 -- start start 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa93000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7faa8c13df80 0x7faa8c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa93000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7faa8c13e360 0x7faa8c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa93000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c06c470 con 0x7faa8c15f9c0 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa93000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c06c750 con 0x7faa8c13e360 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa93000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c06ca30 con 0x7faa8c13df80 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa8b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7faa8c13e360 0x7faa8c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa91000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7faa8c15f9c0 0x7faa8c1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42826/0 (socket says 172.21.15.141:42826) 2024-01-16T15:02:30.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa91000640 1 -- 172.21.15.141:0/2570973090 learned_addr learned my addr 172.21.15.141:0/2570973090 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:30.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.350+0000 7faa8be00640 1 --2- 172.21.15.141:0/2570973090 >> v2:172.21.15.188:3300/0 conn(0x7faa8c13df80 0x7faa8c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.351+0000 7faa8aa00640 1 -- 172.21.15.141:0/2570973090 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771419608 0 0) 0x7faa8c06c470 con 0x7faa8c15f9c0 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.351+0000 7faa8b400640 1 -- 172.21.15.141:0/2570973090 >> v2:172.21.15.188:3300/0 conn(0x7faa8c13df80 msgr2=0x7faa8c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.351+0000 7faa8b400640 1 --2- 172.21.15.141:0/2570973090 >> v2:172.21.15.188:3300/0 conn(0x7faa8c13df80 0x7faa8c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.351+0000 7faa8b400640 1 -- 172.21.15.141:0/2570973090 >> v1:172.21.15.141:6789/0 conn(0x7faa8c15f9c0 legacy=0x7faa8c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.351+0000 7faa8b400640 1 -- 172.21.15.141:0/2570973090 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c06cd10 con 0x7faa8c13e360 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.352+0000 7faa8b400640 1 --2- 172.21.15.141:0/2570973090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c13e360 0x7faa8c0cf290 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7faa8c072770 tx=0x7faa7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25cf6bc9921deed4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.352+0000 7faa8aa00640 1 -- 172.21.15.141:0/2570973090 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c031830 con 0x7faa8c13e360 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.352+0000 7faa8aa00640 1 -- 172.21.15.141:0/2570973090 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c031990 con 0x7faa8c13e360 2024-01-16T15:02:30.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.352+0000 7faa8aa00640 1 -- 172.21.15.141:0/2570973090 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c031d20 con 0x7faa8c13e360 2024-01-16T15:02:30.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 -- 172.21.15.141:0/2570973090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c13e360 msgr2=0x7faa8c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 --2- 172.21.15.141:0/2570973090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c13e360 0x7faa8c0cf290 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7faa8c072770 tx=0x7faa7c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 -- 172.21.15.141:0/2570973090 shutdown_connections 2024-01-16T15:02:30.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 --2- 172.21.15.141:0/2570973090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c13e360 0x7faa8c0cf290 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 --2- 172.21.15.141:0/2570973090 >> v2:172.21.15.188:3300/0 conn(0x7faa8c13df80 0x7faa8c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 -- 172.21.15.141:0/2570973090 >> 172.21.15.141:0/2570973090 conn(0x7faa8c06a800 msgr2=0x7faa8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:30.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 -- 172.21.15.141:0/2570973090 shutdown_connections 2024-01-16T15:02:30.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.353+0000 7faa93000640 1 -- 172.21.15.141:0/2570973090 wait complete. 2024-01-16T15:02:30.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.354+0000 7faa93000640 1 Processor -- start 2024-01-16T15:02:30.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.355+0000 7faa93000640 1 -- start start 2024-01-16T15:02:30.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.355+0000 7faa93000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faa8c13df80 0x7faa8c133590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.355+0000 7faa93000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa8c13e360 0x7faa8c14ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.355+0000 7faa93000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c15f9c0 0x7faa8c15d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.355+0000 7faa93000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c14d4e0 con 0x7faa8c15f9c0 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa93000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c14e5d0 con 0x7faa8c13e360 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa93000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c14e8b0 con 0x7faa8c13df80 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c15f9c0 0x7faa8c15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c15f9c0 0x7faa8c15d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35110/0 (socket says 172.21.15.141:35110) 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 -- 172.21.15.141:0/4250659523 learned_addr learned my addr 172.21.15.141:0/4250659523 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa8be00640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faa8c13df80 0x7faa8c133590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 -- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faa8c13df80 msgr2=0x7faa8c133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faa8c13df80 0x7faa8c133590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa8b400640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa8c13e360 0x7faa8c14ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 -- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa8c13e360 msgr2=0x7faa8c14ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa8c13e360 0x7faa8c14ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.356+0000 7faa91000640 1 -- 172.21.15.141:0/4250659523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c009400 con 0x7faa8c15f9c0 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.357+0000 7faa91000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c15f9c0 0x7faa8c15d290 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faa8400c780 tx=0x7faa8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.357+0000 7faa83e00640 1 -- 172.21.15.141:0/4250659523 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8400ce90 con 0x7faa8c15f9c0 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.357+0000 7faa83e00640 1 -- 172.21.15.141:0/4250659523 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8400a640 con 0x7faa8c15f9c0 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.357+0000 7faa83e00640 1 -- 172.21.15.141:0/4250659523 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8400aa70 con 0x7faa8c15f9c0 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.357+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c14eb90 con 0x7faa8c15f9c0 2024-01-16T15:02:30.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.357+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c133ad0 con 0x7faa8c15f9c0 2024-01-16T15:02:30.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.358+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c14f0b0 con 0x7faa8c15f9c0 2024-01-16T15:02:30.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.359+0000 7faa83e00640 1 -- 172.21.15.141:0/4250659523 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7faa8400abd0 con 0x7faa8c15f9c0 2024-01-16T15:02:30.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.360+0000 7faa83e00640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faa6406b860 0x7faa6406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.360+0000 7faa83e00640 1 -- 172.21.15.141:0/4250659523 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa84089c20 con 0x7faa8c15f9c0 2024-01-16T15:02:30.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.361+0000 7faa8be00640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faa6406b860 0x7faa6406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.362+0000 7faa8be00640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faa6406b860 0x7faa6406dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faa74006fd0 tx=0x7faa74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.364+0000 7faa83e00640 1 -- 172.21.15.141:0/4250659523 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa8c14f0b0 con 0x7faa8c15f9c0 2024-01-16T15:02:30.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.555+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa8c14f0b0 con 0x7faa8c15f9c0 2024-01-16T15:02:30.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.556+0000 7faa83e00640 1 -- 172.21.15.141:0/4250659523 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16626 (secure 0 0 0) 0x7faa8c14f0b0 con 0x7faa8c15f9c0 2024-01-16T15:02:30.537 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:30.538 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":58,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:02:29.120541+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:26.939197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:28.143123+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:30.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faa6406b860 msgr2=0x7faa6406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faa6406b860 0x7faa6406dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faa74006fd0 tx=0x7faa74008040 comp rx=0 tx=0).stop 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c15f9c0 msgr2=0x7faa8c15d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c15f9c0 0x7faa8c15d290 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faa8400c780 tx=0x7faa8400cc40 comp rx=0 tx=0).stop 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 shutdown_connections 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faa6406b860 0x7faa6406dd20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa8c15f9c0 0x7faa8c15d290 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa8c13e360 0x7faa8c14ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 --2- 172.21.15.141:0/4250659523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faa8c13df80 0x7faa8c133590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 >> 172.21.15.141:0/4250659523 conn(0x7faa8c06a800 msgr2=0x7faa8c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 shutdown_connections 2024-01-16T15:02:30.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.559+0000 7faa93000640 1 -- 172.21.15.141:0/4250659523 wait complete. 2024-01-16T15:02:30.559 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:02:30.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.788+0000 7fad48800640 1 Processor -- start 2024-01-16T15:02:30.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.788+0000 7fad48800640 1 -- start start 2024-01-16T15:02:30.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.790+0000 7fad48800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 0x7fad4015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.790+0000 7fad48800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400b1ee0 con 0x7fad4015e1e0 2024-01-16T15:02:30.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.790+0000 7fad48800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400b2780 con 0x7fad40143ae0 2024-01-16T15:02:30.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.790+0000 7fad48800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4015e990 con 0x7fad400cf9a0 2024-01-16T15:02:30.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.790+0000 7fad46800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fad4015e1e0 0x7fad400702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42830/0 (socket says 172.21.15.141:42830) 2024-01-16T15:02:30.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.790+0000 7fad46800640 1 -- 172.21.15.141:0/1429217318 learned_addr learned my addr 172.21.15.141:0/1429217318 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:30.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad45400640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 0x7fad4015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987563263 0 0) 0x7fad400b1ee0 con 0x7fad4015e1e0 2024-01-16T15:02:30.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad20003520 con 0x7fad4015e1e0 2024-01-16T15:02:30.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2647732074 0 0) 0x7fad20003520 con 0x7fad4015e1e0 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 msgr2=0x7fad4015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 0x7fad4015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 >> v1:172.21.15.188:6789/0 conn(0x7fad40143ae0 legacy=0x7fad400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400b2130 con 0x7fad4015e1e0 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fad380019f0 con 0x7fad4015e1e0 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 >> v1:172.21.15.141:6789/0 conn(0x7fad4015e1e0 legacy=0x7fad400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 0x7fad200039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad20003f70 0x7fad200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20004d00 0x7fad20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4015e990 con 0x7fad20003f70 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.791+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2000bac0 con 0x7fad20004d00 2024-01-16T15:02:30.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2000de90 con 0x7fad400cf9a0 2024-01-16T15:02:30.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad45400640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad20003f70 0x7fad200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad45e00640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 0x7fad200039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad45400640 1 -- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 msgr2=0x7fad200039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:30.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad45400640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad400cf9a0 0x7fad200039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad45400640 1 -- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20004d00 msgr2=0x7fad20005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:30.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad45400640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20004d00 0x7fad20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.792+0000 7fad45400640 1 -- 172.21.15.141:0/1429217318 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad20003520 con 0x7fad20003f70 2024-01-16T15:02:30.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.793+0000 7fad45400640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad20003f70 0x7fad200047c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fad28009ac0 tx=0x7fad200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a24e9705235a0f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.793+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2802c600 con 0x7fad20003f70 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2802c760 con 0x7fad20003f70 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad44a00640 1 -- 172.21.15.141:0/1429217318 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2802caf0 con 0x7fad20003f70 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad48800640 1 -- 172.21.15.141:0/1429217318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad20003f70 msgr2=0x7fad200047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad48800640 1 --2- 172.21.15.141:0/1429217318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad20003f70 0x7fad200047c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fad28009ac0 tx=0x7fad200057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad46800640 1 -- 172.21.15.141:0/1429217318 reap_dead start 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad48800640 1 -- 172.21.15.141:0/1429217318 shutdown_connections 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad48800640 1 -- 172.21.15.141:0/1429217318 >> 172.21.15.141:0/1429217318 conn(0x7fad4006a800 msgr2=0x7fad4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad48800640 1 -- 172.21.15.141:0/1429217318 shutdown_connections 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.794+0000 7fad48800640 1 -- 172.21.15.141:0/1429217318 wait complete. 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.796+0000 7fad48800640 1 Processor -- start 2024-01-16T15:02:30.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.796+0000 7fad48800640 1 -- start start 2024-01-16T15:02:30.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad48800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20003f70 0x7fad4013e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad48800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad400ce1a0 0x7fad4014d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad48800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad4015c660 0x7fad4015cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad48800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad401401a0 con 0x7fad400ce1a0 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad48800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4015d020 con 0x7fad20003f70 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad48800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4013fb20 con 0x7fad4015c660 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad45400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad400ce1a0 0x7fad4014d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad45400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad400ce1a0 0x7fad4014d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35120/0 (socket says 172.21.15.141:35120) 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.797+0000 7fad45400640 1 -- 172.21.15.141:0/1642554435 learned_addr learned my addr 172.21.15.141:0/1642554435 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad46800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad4015c660 0x7fad4015cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad45e00640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20003f70 0x7fad4013e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad45400640 1 -- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad4015c660 msgr2=0x7fad4015cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad45400640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad4015c660 0x7fad4015cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad45400640 1 -- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20003f70 msgr2=0x7fad4013e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad45400640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20003f70 0x7fad4013e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad45400640 1 -- 172.21.15.141:0/1642554435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400b1520 con 0x7fad400ce1a0 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad46800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad4015c660 0x7fad4015cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad45400640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad400ce1a0 0x7fad4014d350 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fad28009ac0 tx=0x7fad28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.798+0000 7fad3e000640 1 -- 172.21.15.141:0/1642554435 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28003810 con 0x7fad400ce1a0 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.799+0000 7fad3e000640 1 -- 172.21.15.141:0/1642554435 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad28003970 con 0x7fad400ce1a0 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.799+0000 7fad3e000640 1 -- 172.21.15.141:0/1642554435 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28003ce0 con 0x7fad400ce1a0 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.799+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad400b1800 con 0x7fad400ce1a0 2024-01-16T15:02:30.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.799+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad400ce620 con 0x7fad400ce1a0 2024-01-16T15:02:30.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.800+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad400675d0 con 0x7fad400ce1a0 2024-01-16T15:02:30.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.801+0000 7fad3e000640 1 -- 172.21.15.141:0/1642554435 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fad28040450 con 0x7fad400ce1a0 2024-01-16T15:02:30.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.802+0000 7fad3e000640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad1806b860 0x7fad1806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.802+0000 7fad3e000640 1 -- 172.21.15.141:0/1642554435 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad280ac690 con 0x7fad400ce1a0 2024-01-16T15:02:30.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.802+0000 7fad45e00640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad1806b860 0x7fad1806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.804+0000 7fad45e00640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad1806b860 0x7fad1806dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fad30000c00 tx=0x7fad30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:30.806+0000 7fad3e000640 1 -- 172.21.15.141:0/1642554435 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad28076e00 con 0x7fad400ce1a0 2024-01-16T15:02:31.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.036+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fad400b1990 con 0x7fad400ce1a0 2024-01-16T15:02:31.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.719+0000 7fad3e000640 1 -- 172.21.15.141:0/1642554435 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v36) v1 ==== 118+0+0 (secure 0 0 0) 0x7fad280044e0 con 0x7fad400ce1a0 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad1806b860 msgr2=0x7fad1806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad1806b860 0x7fad1806dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fad30000c00 tx=0x7fad30009040 comp rx=0 tx=0).stop 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad400ce1a0 msgr2=0x7fad4014d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad400ce1a0 0x7fad4014d350 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fad28009ac0 tx=0x7fad28004730 comp rx=0 tx=0).stop 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 shutdown_connections 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fad1806b860 0x7fad1806dd20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fad4015c660 0x7fad4015cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad400ce1a0 0x7fad4014d350 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 --2- 172.21.15.141:0/1642554435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad20003f70 0x7fad4013e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 >> 172.21.15.141:0/1642554435 conn(0x7fad4006a800 msgr2=0x7fad40142d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:31.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 shutdown_connections 2024-01-16T15:02:31.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.723+0000 7fad48800640 1 -- 172.21.15.141:0/1642554435 wait complete. 2024-01-16T15:02:31.725 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:02:31.725 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:02:31.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.951+0000 7f2539000640 1 Processor -- start 2024-01-16T15:02:31.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.951+0000 7f2539000640 1 -- start start 2024-01-16T15:02:31.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.953+0000 7f2539000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 0x7f25340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.953+0000 7f2539000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f253415f9c0 0x7f25340cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.953+0000 7f2539000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253415c5f0 con 0x7f253413df80 2024-01-16T15:02:31.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.953+0000 7f2539000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253406c470 con 0x7f253415f9c0 2024-01-16T15:02:31.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.953+0000 7f2539000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253406c750 con 0x7f253413e360 2024-01-16T15:02:31.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.954+0000 7f2531e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f253413df80 0x7f253415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42836/0 (socket says 172.21.15.141:42836) 2024-01-16T15:02:31.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.954+0000 7f2531e00640 1 -- 172.21.15.141:0/1380301851 learned_addr learned my addr 172.21.15.141:0/1380301851 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.954+0000 7f2531400640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 0x7f25340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.954+0000 7f2532800640 1 --2- 172.21.15.141:0/1380301851 >> v2:172.21.15.188:3300/0 conn(0x7f253415f9c0 0x7f25340cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.954+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477336938 0 0) 0x7f253415c5f0 con 0x7f253413df80 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.954+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2514003520 con 0x7f253413df80 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3982980032 0 0) 0x7f2514003520 con 0x7f253413df80 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 msgr2=0x7f25340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 0x7f25340cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 >> v2:172.21.15.188:3300/0 conn(0x7f253415f9c0 msgr2=0x7f25340cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 --2- 172.21.15.141:0/1380301851 >> v2:172.21.15.188:3300/0 conn(0x7f253415f9c0 0x7f25340cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253406ca30 con 0x7f253413df80 2024-01-16T15:02:31.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f25240019f0 con 0x7f253413df80 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 >> v1:172.21.15.141:6789/0 conn(0x7f253413df80 legacy=0x7f253415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 0x7f2514003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2514003f40 0x7f25140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2514004d00 0x7f2514005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253406c470 con 0x7f2514004d00 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253406c750 con 0x7f2514003f40 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251400b980 con 0x7f253413e360 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2532800640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2514004d00 0x7f2514005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.955+0000 7f2531e00640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 0x7f2514003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.956+0000 7f2531400640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2514003f40 0x7f25140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.956+0000 7f2532800640 1 -- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 msgr2=0x7f2514003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.956+0000 7f2532800640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 0x7f2514003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.956+0000 7f2532800640 1 -- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2514003f40 msgr2=0x7f25140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:31.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.956+0000 7f2532800640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2514003f40 0x7f25140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.956+0000 7f2532800640 1 -- 172.21.15.141:0/1380301851 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2514003520 con 0x7f2514004d00 2024-01-16T15:02:31.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.956+0000 7f2531e00640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f253413e360 0x7f2514003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.957+0000 7f2532800640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2514004d00 0x7f2514005180 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f252c009af0 tx=0x7f25140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9a12ee503c44bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.957+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c02c710 con 0x7f2514004d00 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252c02c870 con 0x7f2514004d00 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2530a00640 1 -- 172.21.15.141:0/1380301851 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c02cc00 con 0x7f2514004d00 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2539000640 1 -- 172.21.15.141:0/1380301851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2514004d00 msgr2=0x7f2514005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2539000640 1 --2- 172.21.15.141:0/1380301851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2514004d00 0x7f2514005180 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f252c009af0 tx=0x7f25140057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2532800640 1 -- 172.21.15.141:0/1380301851 reap_dead start 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2539000640 1 -- 172.21.15.141:0/1380301851 shutdown_connections 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2539000640 1 -- 172.21.15.141:0/1380301851 >> 172.21.15.141:0/1380301851 conn(0x7f253406a800 msgr2=0x7f253406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2539000640 1 -- 172.21.15.141:0/1380301851 shutdown_connections 2024-01-16T15:02:31.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.958+0000 7f2539000640 1 -- 172.21.15.141:0/1380301851 wait complete. 2024-01-16T15:02:31.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.960+0000 7f2539000640 1 Processor -- start 2024-01-16T15:02:31.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.960+0000 7f2539000640 1 -- start start 2024-01-16T15:02:31.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.961+0000 7f2539000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2514004d00 0x7f253415e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.961+0000 7f2539000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2534150ed0 0x7f253413d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.961+0000 7f2539000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f253414a990 0x7f253415a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.961+0000 7f2539000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253415ea90 con 0x7f2534150ed0 2024-01-16T15:02:31.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.961+0000 7f2539000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253415e0c0 con 0x7f253414a990 2024-01-16T15:02:31.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.961+0000 7f2539000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253413dda0 con 0x7f2514004d00 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2534150ed0 0x7f253413d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2534150ed0 0x7f253413d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35138/0 (socket says 172.21.15.141:35138) 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 -- 172.21.15.141:0/157258381 learned_addr learned my addr 172.21.15.141:0/157258381 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 -- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2514004d00 msgr2=0x7f253415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2514004d00 0x7f253415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 -- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f253414a990 msgr2=0x7f253415a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f253414a990 0x7f253415a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 -- 172.21.15.141:0/157258381 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253414b490 con 0x7f2534150ed0 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f2531400640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2534150ed0 0x7f253413d860 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f251c011540 tx=0x7f251c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.962+0000 7f252a000640 1 -- 172.21.15.141:0/157258381 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c015930 con 0x7f2534150ed0 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.963+0000 7f252a000640 1 -- 172.21.15.141:0/157258381 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c015a90 con 0x7f2534150ed0 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.963+0000 7f252a000640 1 -- 172.21.15.141:0/157258381 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c015e60 con 0x7f2534150ed0 2024-01-16T15:02:31.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.963+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253414b770 con 0x7f2534150ed0 2024-01-16T15:02:31.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.963+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253415dd50 con 0x7f2534150ed0 2024-01-16T15:02:31.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.964+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25340675d0 con 0x7f2534150ed0 2024-01-16T15:02:31.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.965+0000 7f252a000640 1 -- 172.21.15.141:0/157258381 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f251c01d400 con 0x7f2534150ed0 2024-01-16T15:02:31.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.966+0000 7f252a000640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f250c06b890 0x7f250c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.966+0000 7f252a000640 1 -- 172.21.15.141:0/157258381 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f251c08ad50 con 0x7f2534150ed0 2024-01-16T15:02:31.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.966+0000 7f2531e00640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f250c06b890 0x7f250c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.967+0000 7f2531e00640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f250c06b890 0x7f250c06dd50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2524000f80 tx=0x7f2524001f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:31.970+0000 7f252a000640 1 -- 172.21.15.141:0/157258381 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f251c091080 con 0x7f2534150ed0 2024-01-16T15:02:32.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.203+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f253415e200 con 0x7f2534150ed0 2024-01-16T15:02:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.730+0000 7f252a000640 1 -- 172.21.15.141:0/157258381 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v37) v1 ==== 122+0+0 (secure 0 0 0) 0x7f253415e200 con 0x7f2534150ed0 2024-01-16T15:02:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f250c06b890 msgr2=0x7f250c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f250c06b890 0x7f250c06dd50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2524000f80 tx=0x7f2524001f00 comp rx=0 tx=0).stop 2024-01-16T15:02:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2534150ed0 msgr2=0x7f253413d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2534150ed0 0x7f253413d860 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f251c011540 tx=0x7f251c011a00 comp rx=0 tx=0).stop 2024-01-16T15:02:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 shutdown_connections 2024-01-16T15:02:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f250c06b890 0x7f250c06dd50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f253414a990 0x7f253415a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2534150ed0 0x7f253413d860 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 --2- 172.21.15.141:0/157258381 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2514004d00 0x7f253415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 >> 172.21.15.141:0/157258381 conn(0x7f253406a800 msgr2=0x7f253415cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.735+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 shutdown_connections 2024-01-16T15:02:32.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.736+0000 7f2539000640 1 -- 172.21.15.141:0/157258381 wait complete. 2024-01-16T15:02:32.738 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:02:32.738 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:32.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.973+0000 7fb46e600640 1 Processor -- start 2024-01-16T15:02:32.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.973+0000 7fb46e600640 1 -- start start 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb46e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb468159590 0x7fb468070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb46e600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb46806d050 0x7fb46806d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb46e600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46806d9a0 con 0x7fb4680b3c20 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb46e600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb468071790 con 0x7fb46806d050 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb46e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb468071100 con 0x7fb468159590 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb46dc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb46806d050 0x7fb46806d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb467e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb4680b3c20 0x7fb4680ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42852/0 (socket says 172.21.15.141:42852) 2024-01-16T15:02:32.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.975+0000 7fb467e00640 1 -- 172.21.15.141:0/3514278004 learned_addr learned my addr 172.21.15.141:0/3514278004 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:32.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.976+0000 7fb46d200640 1 --2- 172.21.15.141:0/3514278004 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb468159590 0x7fb468070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.976+0000 7fb46dc00640 1 -- 172.21.15.141:0/3514278004 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb468159590 msgr2=0x7fb468070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:32.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.976+0000 7fb46dc00640 1 --2- 172.21.15.141:0/3514278004 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb468159590 0x7fb468070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.976+0000 7fb46dc00640 1 -- 172.21.15.141:0/3514278004 >> v1:172.21.15.141:6789/0 conn(0x7fb4680b3c20 legacy=0x7fb4680ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.976+0000 7fb46dc00640 1 -- 172.21.15.141:0/3514278004 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb468070860 con 0x7fb46806d050 2024-01-16T15:02:32.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.977+0000 7fb46dc00640 1 --2- 172.21.15.141:0/3514278004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb46806d460 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb460009d90 tx=0x7fb46002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d3f17d39d7c044 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.977+0000 7fb467400640 1 -- 172.21.15.141:0/3514278004 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46002cbc0 con 0x7fb46806d050 2024-01-16T15:02:32.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.977+0000 7fb467400640 1 -- 172.21.15.141:0/3514278004 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46002cd20 con 0x7fb46806d050 2024-01-16T15:02:32.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.977+0000 7fb467400640 1 -- 172.21.15.141:0/3514278004 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460034660 con 0x7fb46806d050 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 -- 172.21.15.141:0/3514278004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 msgr2=0x7fb46806d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 --2- 172.21.15.141:0/3514278004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb46806d460 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb460009d90 tx=0x7fb46002be10 comp rx=0 tx=0).stop 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 -- 172.21.15.141:0/3514278004 shutdown_connections 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 --2- 172.21.15.141:0/3514278004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb46806d460 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 --2- 172.21.15.141:0/3514278004 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb468159590 0x7fb468070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 -- 172.21.15.141:0/3514278004 >> 172.21.15.141:0/3514278004 conn(0x7fb46806a800 msgr2=0x7fb46806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 -- 172.21.15.141:0/3514278004 shutdown_connections 2024-01-16T15:02:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.978+0000 7fb46e600640 1 -- 172.21.15.141:0/3514278004 wait complete. 2024-01-16T15:02:32.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.980+0000 7fb46e600640 1 Processor -- start 2024-01-16T15:02:32.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.980+0000 7fb46e600640 1 -- start start 2024-01-16T15:02:32.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb468158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4680b3c20 0x7fb46815b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb468159590 0x7fb46815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680ac8d0 con 0x7fb46806d050 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb468155ba0 con 0x7fb468159590 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb468155e80 con 0x7fb4680b3c20 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb468158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb468158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35150/0 (socket says 172.21.15.141:35150) 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.981+0000 7fb46d200640 1 -- 172.21.15.141:0/565907254 learned_addr learned my addr 172.21.15.141:0/565907254 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb467e00640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4680b3c20 0x7fb46815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46d200640 1 -- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4680b3c20 msgr2=0x7fb46815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46d200640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4680b3c20 0x7fb46815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46dc00640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb468159590 0x7fb46815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46d200640 1 -- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb468159590 msgr2=0x7fb46815b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46d200640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb468159590 0x7fb46815b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46d200640 1 -- 172.21.15.141:0/565907254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb460009a70 con 0x7fb46806d050 2024-01-16T15:02:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46d200640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb468158480 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb45800d5f0 tx=0x7fb45800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb464c00640 1 -- 172.21.15.141:0/565907254 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4580125f0 con 0x7fb46806d050 2024-01-16T15:02:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb464c00640 1 -- 172.21.15.141:0/565907254 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb458012750 con 0x7fb46806d050 2024-01-16T15:02:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb464c00640 1 -- 172.21.15.141:0/565907254 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458012b00 con 0x7fb46806d050 2024-01-16T15:02:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.982+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb468156160 con 0x7fb46806d050 2024-01-16T15:02:32.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.983+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46814eb50 con 0x7fb46806d050 2024-01-16T15:02:32.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.984+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb468067450 con 0x7fb46806d050 2024-01-16T15:02:32.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.984+0000 7fb464c00640 1 -- 172.21.15.141:0/565907254 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb458012c60 con 0x7fb46806d050 2024-01-16T15:02:32.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.985+0000 7fb464c00640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb44006b860 0x7fb44006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.985+0000 7fb464c00640 1 -- 172.21.15.141:0/565907254 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb45808aa90 con 0x7fb46806d050 2024-01-16T15:02:32.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.986+0000 7fb467e00640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb44006b860 0x7fb44006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.987+0000 7fb467e00640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb44006b860 0x7fb44006dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb4680cb5c0 tx=0x7fb450009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:32.990+0000 7fb464c00640 1 -- 172.21.15.141:0/565907254 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4580550e0 con 0x7fb46806d050 2024-01-16T15:02:33.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.220+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb46815ced0 con 0x7fb46806d050 2024-01-16T15:02:33.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.222+0000 7fb464c00640 1 -- 172.21.15.141:0/565907254 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fb458058700 con 0x7fb46806d050 2024-01-16T15:02:33.203 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:33.203 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/916741426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":916741426},{"type":"v1","addr":"172.21.15.188:6837","nonce":916741426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5332,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3966594812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3966594812},{"type":"v1","addr":"172.21.15.141:6833","nonce":3966594812}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3648732517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3648732517},{"type":"v1","addr":"172.21.15.188:6835","nonce":3648732517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:02:29.121311+0000","modified":"2024-01-16T15:02:32.205611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1757492142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":1757492142},{"type":"v1","addr":"172.21.15.141:6835","nonce":1757492142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:02:33.203 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 37 2024-01-16T15:02:33.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.224+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb44006b860 msgr2=0x7fb44006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.224+0000 7fb46e600640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb44006b860 0x7fb44006dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb4680cb5c0 tx=0x7fb450009040 comp rx=0 tx=0).stop 2024-01-16T15:02:33.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.224+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 msgr2=0x7fb468158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.224+0000 7fb46e600640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb468158480 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb45800d5f0 tx=0x7fb45800dab0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 shutdown_connections 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb44006b860 0x7fb44006dd20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb468159590 0x7fb46815b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4680b3c20 0x7fb46815b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 --2- 172.21.15.141:0/565907254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb46806d050 0x7fb468158480 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 >> 172.21.15.141:0/565907254 conn(0x7fb46806a800 msgr2=0x7fb468152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 shutdown_connections 2024-01-16T15:02:33.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.225+0000 7fb46e600640 1 -- 172.21.15.141:0/565907254 wait complete. 2024-01-16T15:02:33.225 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:33.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.447+0000 7f63e7400640 1 Processor -- start 2024-01-16T15:02:33.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.447+0000 7f63e7400640 1 -- start start 2024-01-16T15:02:33.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.448+0000 7f63e7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 0x7f63e006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.448+0000 7f63e7400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00cfa70 con 0x7f63e0099a60 2024-01-16T15:02:33.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.448+0000 7f63e7400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00cff00 con 0x7f63e015e880 2024-01-16T15:02:33.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.448+0000 7f63e7400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006fc70 con 0x7f63e0099ea0 2024-01-16T15:02:33.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63d7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f63e0099a60 0x7f63e006e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41638/0 (socket says 172.21.15.141:41638) 2024-01-16T15:02:33.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63d7e00640 1 -- 172.21.15.141:0/269397174 learned_addr learned my addr 172.21.15.141:0/269397174 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:33.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63e5400640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 0x7f63e006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731239056 0 0) 0x7f63e00cfa70 con 0x7f63e0099a60 2024-01-16T15:02:33.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c0003520 con 0x7f63e0099a60 2024-01-16T15:02:33.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521135845 0 0) 0x7f63c0003520 con 0x7f63e0099a60 2024-01-16T15:02:33.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 msgr2=0x7f63e006eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 0x7f63e006eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 >> v1:172.21.15.188:6789/0 conn(0x7f63e015e880 legacy=0x7f63e006ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00cf420 con 0x7f63e0099a60 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.449+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f63c8001670 con 0x7f63e0099a60 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 >> v1:172.21.15.141:6789/0 conn(0x7f63e0099a60 legacy=0x7f63e006e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63dfe00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63c00039a0 0x7f63c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63dfe00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63dfe00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 0x7f63c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006fc70 con 0x7f63c00039a0 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c000baf0 con 0x7f63c00047c0 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c000bcb0 con 0x7f63e0099ea0 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63e4a00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63c00039a0 0x7f63c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63d7e00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.450+0000 7f63e5400640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 0x7f63c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.451+0000 7f63e4a00640 1 -- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 msgr2=0x7f63c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:33.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.451+0000 7f63e4a00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 0x7f63c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.451+0000 7f63e4a00640 1 -- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00047c0 msgr2=0x7f63c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:33.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.451+0000 7f63e4a00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.451+0000 7f63e4a00640 1 -- 172.21.15.141:0/269397174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0003520 con 0x7f63c00039a0 2024-01-16T15:02:33.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.451+0000 7f63e5400640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e0099ea0 0x7f63c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.452+0000 7f63e4a00640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63c00039a0 0x7f63c0003dc0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f63d0001720 tx=0x7f63c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af893756f4fe4565 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.452+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00052d0 con 0x7f63c00039a0 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.452+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d002e050 con 0x7f63c00039a0 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.452+0000 7f63dfe00640 1 -- 172.21.15.141:0/269397174 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00035f0 con 0x7f63c00039a0 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.453+0000 7f63e7400640 1 -- 172.21.15.141:0/269397174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63c00039a0 msgr2=0x7f63c0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.453+0000 7f63e7400640 1 --2- 172.21.15.141:0/269397174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63c00039a0 0x7f63c0003dc0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f63d0001720 tx=0x7f63c00057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.453+0000 7f63e5400640 1 -- 172.21.15.141:0/269397174 reap_dead start 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.453+0000 7f63e7400640 1 -- 172.21.15.141:0/269397174 shutdown_connections 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.453+0000 7f63e7400640 1 -- 172.21.15.141:0/269397174 >> 172.21.15.141:0/269397174 conn(0x7f63e00b2d00 msgr2=0x7f63e00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:33.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.453+0000 7f63e7400640 1 -- 172.21.15.141:0/269397174 shutdown_connections 2024-01-16T15:02:33.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.453+0000 7f63e7400640 1 -- 172.21.15.141:0/269397174 wait complete. 2024-01-16T15:02:33.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.455+0000 7f63e7400640 1 Processor -- start 2024-01-16T15:02:33.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.455+0000 7f63e7400640 1 -- start start 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63e7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00039a0 0x7f63e006c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63e7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63e00c89b0 0x7f63e00c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63e7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e01593f0 0x7f63e0159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63e7400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e01f3920 con 0x7f63e00c89b0 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63e7400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0159db0 con 0x7f63c00039a0 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63e7400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e01f2ca0 con 0x7f63e01593f0 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63e00c89b0 0x7f63e00c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63d7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63e00c89b0 0x7f63e00c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59738/0 (socket says 172.21.15.141:59738) 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.456+0000 7f63d7e00640 1 -- 172.21.15.141:0/2034720966 learned_addr learned my addr 172.21.15.141:0/2034720966 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:33.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63e5400640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e01593f0 0x7f63e0159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63e4a00640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00039a0 0x7f63e006c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63d7e00640 1 -- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e01593f0 msgr2=0x7f63e0159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63d7e00640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e01593f0 0x7f63e0159870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63d7e00640 1 -- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00039a0 msgr2=0x7f63e006c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63d7e00640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00039a0 0x7f63e006c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63d7e00640 1 -- 172.21.15.141:0/2034720966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e01f2f80 con 0x7f63e00c89b0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63d7e00640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63e00c89b0 0x7f63e00c8e10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f63c8007290 tx=0x7f63c8007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63dd600640 1 -- 172.21.15.141:0/2034720966 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8001650 con 0x7f63e00c89b0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63dd600640 1 -- 172.21.15.141:0/2034720966 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c800f9c0 con 0x7f63e00c89b0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e01f3260 con 0x7f63e00c89b0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.457+0000 7f63dd600640 1 -- 172.21.15.141:0/2034720966 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c800fd50 con 0x7f63e00c89b0 2024-01-16T15:02:33.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.458+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e00b4a80 con 0x7f63e00c89b0 2024-01-16T15:02:33.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.459+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e01f3a60 con 0x7f63e00c89b0 2024-01-16T15:02:33.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.460+0000 7f63dd600640 1 -- 172.21.15.141:0/2034720966 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f63c801d460 con 0x7f63e00c89b0 2024-01-16T15:02:33.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.460+0000 7f63dd600640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63b806b860 0x7f63b806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.461+0000 7f63dd600640 1 -- 172.21.15.141:0/2034720966 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63c808a650 con 0x7f63e00c89b0 2024-01-16T15:02:33.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.461+0000 7f63e4a00640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63b806b860 0x7f63b806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.462+0000 7f63e4a00640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63b806b860 0x7f63b806dd20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f63d0033000 tx=0x7f63d0009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.465+0000 7f63dd600640 1 -- 172.21.15.141:0/2034720966 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63e01f3a60 con 0x7f63e00c89b0 2024-01-16T15:02:33.534 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:33.541 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:33.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.662+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63e01f3a60 con 0x7f63e00c89b0 2024-01-16T15:02:33.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.664+0000 7f63dd600640 1 -- 172.21.15.141:0/2034720966 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16613 (secure 0 0 0) 0x7f63e01f3a60 con 0x7f63e00c89b0 2024-01-16T15:02:33.644 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:33.645 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":59,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:02:30.705153+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:26.939197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:28.143123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:33.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63b806b860 msgr2=0x7f63b806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63b806b860 0x7f63b806dd20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f63d0033000 tx=0x7f63d0009030 comp rx=0 tx=0).stop 2024-01-16T15:02:33.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63e00c89b0 msgr2=0x7f63e00c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63e00c89b0 0x7f63e00c8e10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f63c8007290 tx=0x7f63c8007750 comp rx=0 tx=0).stop 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 shutdown_connections 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63b806b860 0x7f63b806dd20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f63e01593f0 0x7f63e0159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63e00c89b0 0x7f63e00c8e10 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 --2- 172.21.15.141:0/2034720966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c00039a0 0x7f63e006c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.666+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 >> 172.21.15.141:0/2034720966 conn(0x7f63e00b2d00 msgr2=0x7f63e015da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.667+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 shutdown_connections 2024-01-16T15:02:33.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.667+0000 7f63e7400640 1 -- 172.21.15.141:0/2034720966 wait complete. 2024-01-16T15:02:33.665 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:02:33.665 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:33.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.867+0000 7f5605e00640 1 Processor -- start 2024-01-16T15:02:33.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.867+0000 7f5605e00640 1 -- start start 2024-01-16T15:02:33.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.868+0000 7f5605e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f5605e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f560015f9c0 0x7f56000cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f5605e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560015c5f0 con 0x7f560013df80 2024-01-16T15:02:33.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f5605e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560006c470 con 0x7f560015f9c0 2024-01-16T15:02:33.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f5605e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560006c750 con 0x7f560013e360 2024-01-16T15:02:33.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55ff400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f560015f9c0 0x7f56000cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55fea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f560013df80 0x7f560015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41650/0 (socket says 172.21.15.141:41650) 2024-01-16T15:02:33.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55fea00640 1 -- 172.21.15.141:0/2979895567 learned_addr learned my addr 172.21.15.141:0/2979895567 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:33.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55f7e00640 1 --2- 172.21.15.141:0/2979895567 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55ff400640 1 -- 172.21.15.141:0/2979895567 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 msgr2=0x7f56000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:33.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55ff400640 1 --2- 172.21.15.141:0/2979895567 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55ff400640 1 -- 172.21.15.141:0/2979895567 >> v1:172.21.15.141:6789/0 conn(0x7f560013df80 legacy=0x7f560015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55ff400640 1 -- 172.21.15.141:0/2979895567 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560006ca30 con 0x7f560015f9c0 2024-01-16T15:02:33.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55fe000640 1 -- 172.21.15.141:0/2979895567 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221876532 0 0) 0x7f560015c5f0 con 0x7f560013df80 2024-01-16T15:02:33.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.869+0000 7f55f7e00640 1 --2- 172.21.15.141:0/2979895567 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:33.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.870+0000 7f55ff400640 1 --2- 172.21.15.141:0/2979895567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000cf7d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009d90 tx=0x7f55f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9b0cfbd07767f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.871+0000 7f55fe000640 1 -- 172.21.15.141:0/2979895567 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f802cc60 con 0x7f560015f9c0 2024-01-16T15:02:33.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.871+0000 7f55fe000640 1 -- 172.21.15.141:0/2979895567 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f802cdc0 con 0x7f560015f9c0 2024-01-16T15:02:33.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.871+0000 7f55fe000640 1 -- 172.21.15.141:0/2979895567 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8034660 con 0x7f560015f9c0 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.871+0000 7f5605e00640 1 -- 172.21.15.141:0/2979895567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 msgr2=0x7f56000cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.871+0000 7f5605e00640 1 --2- 172.21.15.141:0/2979895567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000cf7d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009d90 tx=0x7f55f802beb0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.872+0000 7f5605e00640 1 -- 172.21.15.141:0/2979895567 shutdown_connections 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.872+0000 7f5605e00640 1 --2- 172.21.15.141:0/2979895567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000cf7d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.872+0000 7f5605e00640 1 --2- 172.21.15.141:0/2979895567 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.872+0000 7f5605e00640 1 -- 172.21.15.141:0/2979895567 >> 172.21.15.141:0/2979895567 conn(0x7f560006a800 msgr2=0x7f560006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.872+0000 7f5605e00640 1 -- 172.21.15.141:0/2979895567 shutdown_connections 2024-01-16T15:02:33.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.872+0000 7f5605e00640 1 -- 172.21.15.141:0/2979895567 wait complete. 2024-01-16T15:02:33.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.873+0000 7f5605e00640 1 Processor -- start 2024-01-16T15:02:33.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.873+0000 7f5605e00640 1 -- start start 2024-01-16T15:02:33.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.874+0000 7f5605e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560013df80 0x7f5600136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.874+0000 7f5605e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.874+0000 7f5605e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.874+0000 7f5605e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560014e4d0 con 0x7f560015f9c0 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.874+0000 7f5605e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560014e1f0 con 0x7f560013df80 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.874+0000 7f5605e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560014bb90 con 0x7f560013e360 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59756/0 (socket says 172.21.15.141:59756) 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 -- 172.21.15.141:0/1072400894 learned_addr learned my addr 172.21.15.141:0/1072400894 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 -- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 msgr2=0x7f56000b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 -- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560013df80 msgr2=0x7f5600136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560013df80 0x7f5600136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 -- 172.21.15.141:0/1072400894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f8009a70 con 0x7f560015f9c0 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55ff400640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000b3230 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009d90 tx=0x7f55f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.875+0000 7f55f7400640 1 -- 172.21.15.141:0/1072400894 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8040650 con 0x7f560015f9c0 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.876+0000 7f55f7400640 1 -- 172.21.15.141:0/1072400894 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f80407d0 con 0x7f560015f9c0 2024-01-16T15:02:33.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.876+0000 7f55f7400640 1 -- 172.21.15.141:0/1072400894 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8040b80 con 0x7f560015f9c0 2024-01-16T15:02:33.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.876+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560014be70 con 0x7f560015f9c0 2024-01-16T15:02:33.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.876+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5600136fe0 con 0x7f560015f9c0 2024-01-16T15:02:33.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.877+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560014c390 con 0x7f560015f9c0 2024-01-16T15:02:33.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.878+0000 7f55f7400640 1 -- 172.21.15.141:0/1072400894 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f55f8040ce0 con 0x7f560015f9c0 2024-01-16T15:02:33.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.878+0000 7f55f7400640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55d806b860 0x7f55d806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.879+0000 7f55f7400640 1 -- 172.21.15.141:0/1072400894 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55f80ad520 con 0x7f560015f9c0 2024-01-16T15:02:33.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.879+0000 7f55fea00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55d806b860 0x7f55d806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.880+0000 7f55fea00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55d806b860 0x7f55d806dd20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f55f0007fd0 tx=0x7f55f0007960 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:33.883+0000 7f55f7400640 1 -- 172.21.15.141:0/1072400894 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55f8077b70 con 0x7f560015f9c0 2024-01-16T15:02:34.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.061+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f560006cff0 con 0x7f55d806b860 2024-01-16T15:02:34.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.089+0000 7f55f7400640 1 -- 172.21.15.141:0/1072400894 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282293 (secure 0 0 0) 0x7f55f0007070 con 0x7f55d806b860 2024-01-16T15:02:34.074 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:34.075 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:02:34.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55d806b860 msgr2=0x7f55d806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55d806b860 0x7f55d806dd20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f55f0007fd0 tx=0x7f55f0007960 comp rx=0 tx=0).stop 2024-01-16T15:02:34.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 msgr2=0x7f56000b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000b3230 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009d90 tx=0x7f55f8004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 shutdown_connections 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55d806b860 0x7f55d806dd20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f560015f9c0 0x7f56000b3230 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f560013e360 0x7f56000b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 --2- 172.21.15.141:0/1072400894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560013df80 0x7f5600136aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 >> 172.21.15.141:0/1072400894 conn(0x7f560006a800 msgr2=0x7f560015d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 shutdown_connections 2024-01-16T15:02:34.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.094+0000 7f5605e00640 1 -- 172.21.15.141:0/1072400894 wait complete. 2024-01-16T15:02:34.092 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-01-16T15:02:32.575663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289216,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685504,"statfs":{"total":773094113280,"available":772797956096,"internally_reserved":0,"allocated":6627328,"data_stored":3419944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.917962"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.691258+0000","last_change":"2024-01-16T15:02:28.691258+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.691258+0000","last_undegraded":"2024-01-16T15:02:28.691258+0000","last_fullsized":"2024-01-16T15:02:28.691258+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674881+0000","last_change":"2024-01-16T15:02:28.674881+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674881+0000","last_undegraded":"2024-01-16T15:02:28.674881+0000","last_fullsized":"2024-01-16T15:02:28.674881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.685084+0000","last_change":"2024-01-16T15:02:28.685084+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.685084+0000","last_undegraded":"2024-01-16T15:02:28.685084+0000","last_fullsized":"2024-01-16T15:02:28.685084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682154+0000","last_change":"2024-01-16T15:02:28.682154+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682154+0000","last_undegraded":"2024-01-16T15:02:28.682154+0000","last_fullsized":"2024-01-16T15:02:28.682154+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687284+0000","last_change":"2024-01-16T15:02:28.687284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687284+0000","last_undegraded":"2024-01-16T15:02:28.687284+0000","last_fullsized":"2024-01-16T15:02:28.687284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680550+0000","last_change":"2024-01-16T15:02:28.680550+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680550+0000","last_undegraded":"2024-01-16T15:02:28.680550+0000","last_fullsized":"2024-01-16T15:02:28.680550+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677104+0000","last_change":"2024-01-16T15:02:28.677104+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677104+0000","last_undegraded":"2024-01-16T15:02:28.677104+0000","last_fullsized":"2024-01-16T15:02:28.677104+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677904+0000","last_change":"2024-01-16T15:02:28.677904+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677904+0000","last_undegraded":"2024-01-16T15:02:28.677904+0000","last_fullsized":"2024-01-16T15:02:28.677904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671452+0000","last_change":"2024-01-16T15:02:28.671452+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671452+0000","last_undegraded":"2024-01-16T15:02:28.671452+0000","last_fullsized":"2024-01-16T15:02:28.671452+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740996+0000","last_change":"2024-01-16T15:02:29.726552+0000","last_active":"2024-01-16T15:02:29.740996+0000","last_peered":"2024-01-16T15:02:29.740996+0000","last_clean":"2024-01-16T15:02:29.740996+0000","last_became_active":"2024-01-16T15:02:29.726242+0000","last_became_peered":"2024-01-16T15:02:29.726242+0000","last_unstale":"2024-01-16T15:02:29.740996+0000","last_undegraded":"2024-01-16T15:02:29.740996+0000","last_fullsized":"2024-01-16T15:02:29.740996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:15.816925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.743873+0000","last_change":"2024-01-16T15:02:29.735792+0000","last_active":"2024-01-16T15:02:29.743873+0000","last_peered":"2024-01-16T15:02:29.743873+0000","last_clean":"2024-01-16T15:02:29.743873+0000","last_became_active":"2024-01-16T15:02:29.735504+0000","last_became_peered":"2024-01-16T15:02:29.735504+0000","last_unstale":"2024-01-16T15:02:29.743873+0000","last_undegraded":"2024-01-16T15:02:29.743873+0000","last_fullsized":"2024-01-16T15:02:29.743873+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:09.173541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681639+0000","last_change":"2024-01-16T15:02:28.681639+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681639+0000","last_undegraded":"2024-01-16T15:02:28.681639+0000","last_fullsized":"2024-01-16T15:02:28.681639+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726359+0000","last_change":"2024-01-16T15:02:29.721939+0000","last_active":"2024-01-16T15:02:29.726359+0000","last_peered":"2024-01-16T15:02:29.726359+0000","last_clean":"2024-01-16T15:02:29.726359+0000","last_became_active":"2024-01-16T15:02:29.721563+0000","last_became_peered":"2024-01-16T15:02:29.721563+0000","last_unstale":"2024-01-16T15:02:29.726359+0000","last_undegraded":"2024-01-16T15:02:29.726359+0000","last_fullsized":"2024-01-16T15:02:29.726359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:35:52.362185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677387+0000","last_change":"2024-01-16T15:02:28.677387+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677387+0000","last_undegraded":"2024-01-16T15:02:28.677387+0000","last_fullsized":"2024-01-16T15:02:28.677387+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671228+0000","last_change":"2024-01-16T15:02:28.671228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671228+0000","last_undegraded":"2024-01-16T15:02:28.671228+0000","last_fullsized":"2024-01-16T15:02:28.671228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679199+0000","last_change":"2024-01-16T15:02:28.679199+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679199+0000","last_undegraded":"2024-01-16T15:02:28.679199+0000","last_fullsized":"2024-01-16T15:02:28.679199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674612+0000","last_change":"2024-01-16T15:02:28.674612+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674612+0000","last_undegraded":"2024-01-16T15:02:28.674612+0000","last_fullsized":"2024-01-16T15:02:28.674612+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.688365+0000","last_change":"2024-01-16T15:02:28.688365+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.688365+0000","last_undegraded":"2024-01-16T15:02:28.688365+0000","last_fullsized":"2024-01-16T15:02:28.688365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684341+0000","last_change":"2024-01-16T15:02:28.684341+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684341+0000","last_undegraded":"2024-01-16T15:02:28.684341+0000","last_fullsized":"2024-01-16T15:02:28.684341+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677483+0000","last_change":"2024-01-16T15:02:28.677483+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677483+0000","last_undegraded":"2024-01-16T15:02:28.677483+0000","last_fullsized":"2024-01-16T15:02:28.677483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680887+0000","last_change":"2024-01-16T15:02:28.680887+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680887+0000","last_undegraded":"2024-01-16T15:02:28.680887+0000","last_fullsized":"2024-01-16T15:02:28.680887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675926+0000","last_change":"2024-01-16T15:02:28.675926+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675926+0000","last_undegraded":"2024-01-16T15:02:28.675926+0000","last_fullsized":"2024-01-16T15:02:28.675926+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680236+0000","last_change":"2024-01-16T15:02:28.680236+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680236+0000","last_undegraded":"2024-01-16T15:02:28.680236+0000","last_fullsized":"2024-01-16T15:02:28.680236+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726074+0000","last_change":"2024-01-16T15:02:29.721477+0000","last_active":"2024-01-16T15:02:29.726074+0000","last_peered":"2024-01-16T15:02:29.726074+0000","last_clean":"2024-01-16T15:02:29.726074+0000","last_became_active":"2024-01-16T15:02:29.721137+0000","last_became_peered":"2024-01-16T15:02:29.721137+0000","last_unstale":"2024-01-16T15:02:29.726074+0000","last_undegraded":"2024-01-16T15:02:29.726074+0000","last_fullsized":"2024-01-16T15:02:29.726074+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:31.998282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.690190+0000","last_change":"2024-01-16T15:02:28.690190+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.690190+0000","last_undegraded":"2024-01-16T15:02:28.690190+0000","last_fullsized":"2024-01-16T15:02:28.690190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683203+0000","last_change":"2024-01-16T15:02:28.683203+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683203+0000","last_undegraded":"2024-01-16T15:02:28.683203+0000","last_fullsized":"2024-01-16T15:02:28.683203+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678571+0000","last_change":"2024-01-16T15:02:28.678571+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678571+0000","last_undegraded":"2024-01-16T15:02:28.678571+0000","last_fullsized":"2024-01-16T15:02:28.678571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687166+0000","last_change":"2024-01-16T15:02:28.687166+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687166+0000","last_undegraded":"2024-01-16T15:02:28.687166+0000","last_fullsized":"2024-01-16T15:02:28.687166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674298+0000","last_change":"2024-01-16T15:02:28.674298+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674298+0000","last_undegraded":"2024-01-16T15:02:28.674298+0000","last_fullsized":"2024-01-16T15:02:28.674298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682909+0000","last_change":"2024-01-16T15:02:28.682909+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682909+0000","last_undegraded":"2024-01-16T15:02:28.682909+0000","last_fullsized":"2024-01-16T15:02:28.682909+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683833+0000","last_change":"2024-01-16T15:02:28.683833+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683833+0000","last_undegraded":"2024-01-16T15:02:28.683833+0000","last_fullsized":"2024-01-16T15:02:28.683833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675958+0000","last_change":"2024-01-16T15:02:28.675958+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675958+0000","last_undegraded":"2024-01-16T15:02:28.675958+0000","last_fullsized":"2024-01-16T15:02:28.675958+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673472+0000","last_change":"2024-01-16T15:02:28.673472+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673472+0000","last_undegraded":"2024-01-16T15:02:28.673472+0000","last_fullsized":"2024-01-16T15:02:28.673472+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700420+0000","last_change":"2024-01-16T15:02:28.696840+0000","last_active":"2024-01-16T15:02:28.700420+0000","last_peered":"2024-01-16T15:02:28.700420+0000","last_clean":"2024-01-16T15:02:28.700420+0000","last_became_active":"2024-01-16T15:02:28.695989+0000","last_became_peered":"2024-01-16T15:02:28.695989+0000","last_unstale":"2024-01-16T15:02:28.700420+0000","last_undegraded":"2024-01-16T15:02:28.700420+0000","last_fullsized":"2024-01-16T15:02:28.700420+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:36.619091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.741340+0000","last_change":"2024-01-16T15:02:29.722881+0000","last_active":"2024-01-16T15:02:29.741340+0000","last_peered":"2024-01-16T15:02:29.741340+0000","last_clean":"2024-01-16T15:02:29.741340+0000","last_became_active":"2024-01-16T15:02:29.722447+0000","last_became_peered":"2024-01-16T15:02:29.722447+0000","last_unstale":"2024-01-16T15:02:29.741340+0000","last_undegraded":"2024-01-16T15:02:29.741340+0000","last_fullsized":"2024-01-16T15:02:29.741340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:52.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700061+0000","last_change":"2024-01-16T15:02:28.697447+0000","last_active":"2024-01-16T15:02:28.700061+0000","last_peered":"2024-01-16T15:02:28.700061+0000","last_clean":"2024-01-16T15:02:28.700061+0000","last_became_active":"2024-01-16T15:02:28.697122+0000","last_became_peered":"2024-01-16T15:02:28.697122+0000","last_unstale":"2024-01-16T15:02:28.700061+0000","last_undegraded":"2024-01-16T15:02:28.700061+0000","last_fullsized":"2024-01-16T15:02:28.700061+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:09.273766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672137+0000","last_change":"2024-01-16T15:02:28.672137+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672137+0000","last_undegraded":"2024-01-16T15:02:28.672137+0000","last_fullsized":"2024-01-16T15:02:28.672137+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696624+0000","last_change":"2024-01-16T15:02:28.689593+0000","last_active":"2024-01-16T15:02:28.696624+0000","last_peered":"2024-01-16T15:02:28.696624+0000","last_clean":"2024-01-16T15:02:28.696624+0000","last_became_active":"2024-01-16T15:02:28.689343+0000","last_became_peered":"2024-01-16T15:02:28.689343+0000","last_unstale":"2024-01-16T15:02:28.696624+0000","last_undegraded":"2024-01-16T15:02:28.696624+0000","last_fullsized":"2024-01-16T15:02:28.696624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:51.159242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673332+0000","last_change":"2024-01-16T15:02:28.673332+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673332+0000","last_undegraded":"2024-01-16T15:02:28.673332+0000","last_fullsized":"2024-01-16T15:02:28.673332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698069+0000","last_change":"2024-01-16T15:02:28.694539+0000","last_active":"2024-01-16T15:02:28.698069+0000","last_peered":"2024-01-16T15:02:28.698069+0000","last_clean":"2024-01-16T15:02:28.698069+0000","last_became_active":"2024-01-16T15:02:28.694233+0000","last_became_peered":"2024-01-16T15:02:28.694233+0000","last_unstale":"2024-01-16T15:02:28.698069+0000","last_undegraded":"2024-01-16T15:02:28.698069+0000","last_fullsized":"2024-01-16T15:02:28.698069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675366+0000","last_change":"2024-01-16T15:02:28.675366+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675366+0000","last_undegraded":"2024-01-16T15:02:28.675366+0000","last_fullsized":"2024-01-16T15:02:28.675366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696756+0000","last_change":"2024-01-16T15:02:28.693560+0000","last_active":"2024-01-16T15:02:28.696756+0000","last_peered":"2024-01-16T15:02:28.696756+0000","last_clean":"2024-01-16T15:02:28.696756+0000","last_became_active":"2024-01-16T15:02:28.693211+0000","last_became_peered":"2024-01-16T15:02:28.693211+0000","last_unstale":"2024-01-16T15:02:28.696756+0000","last_undegraded":"2024-01-16T15:02:28.696756+0000","last_fullsized":"2024-01-16T15:02:28.696756+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676930+0000","last_change":"2024-01-16T15:02:28.676930+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676930+0000","last_undegraded":"2024-01-16T15:02:28.676930+0000","last_fullsized":"2024-01-16T15:02:28.676930+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.691282+0000","last_change":"2024-01-16T15:02:28.683014+0000","last_active":"2024-01-16T15:02:28.691282+0000","last_peered":"2024-01-16T15:02:28.691282+0000","last_clean":"2024-01-16T15:02:28.691282+0000","last_became_active":"2024-01-16T15:02:28.682735+0000","last_became_peered":"2024-01-16T15:02:28.682735+0000","last_unstale":"2024-01-16T15:02:28.691282+0000","last_undegraded":"2024-01-16T15:02:28.691282+0000","last_fullsized":"2024-01-16T15:02:28.691282+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:55.974417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726389+0000","last_change":"2024-01-16T15:02:29.721954+0000","last_active":"2024-01-16T15:02:29.726389+0000","last_peered":"2024-01-16T15:02:29.726389+0000","last_clean":"2024-01-16T15:02:29.726389+0000","last_became_active":"2024-01-16T15:02:29.721647+0000","last_became_peered":"2024-01-16T15:02:29.721647+0000","last_unstale":"2024-01-16T15:02:29.726389+0000","last_undegraded":"2024-01-16T15:02:29.726389+0000","last_fullsized":"2024-01-16T15:02:29.726389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:16.064389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697872+0000","last_change":"2024-01-16T15:02:28.694096+0000","last_active":"2024-01-16T15:02:28.697872+0000","last_peered":"2024-01-16T15:02:28.697872+0000","last_clean":"2024-01-16T15:02:28.697872+0000","last_became_active":"2024-01-16T15:02:28.693789+0000","last_became_peered":"2024-01-16T15:02:28.693789+0000","last_unstale":"2024-01-16T15:02:28.697872+0000","last_undegraded":"2024-01-16T15:02:28.697872+0000","last_fullsized":"2024-01-16T15:02:28.697872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681431+0000","last_change":"2024-01-16T15:02:28.681431+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681431+0000","last_undegraded":"2024-01-16T15:02:28.681431+0000","last_fullsized":"2024-01-16T15:02:28.681431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696269+0000","last_change":"2024-01-16T15:02:28.693108+0000","last_active":"2024-01-16T15:02:28.696269+0000","last_peered":"2024-01-16T15:02:28.696269+0000","last_clean":"2024-01-16T15:02:28.696269+0000","last_became_active":"2024-01-16T15:02:28.691074+0000","last_became_peered":"2024-01-16T15:02:28.691074+0000","last_unstale":"2024-01-16T15:02:28.696269+0000","last_undegraded":"2024-01-16T15:02:28.696269+0000","last_fullsized":"2024-01-16T15:02:28.696269+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672957+0000","last_change":"2024-01-16T15:02:28.672957+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672957+0000","last_undegraded":"2024-01-16T15:02:28.672957+0000","last_fullsized":"2024-01-16T15:02:28.672957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702078+0000","last_change":"2024-01-16T15:02:28.700517+0000","last_active":"2024-01-16T15:02:28.702078+0000","last_peered":"2024-01-16T15:02:28.702078+0000","last_clean":"2024-01-16T15:02:28.702078+0000","last_became_active":"2024-01-16T15:02:28.700187+0000","last_became_peered":"2024-01-16T15:02:28.700187+0000","last_unstale":"2024-01-16T15:02:28.702078+0000","last_undegraded":"2024-01-16T15:02:28.702078+0000","last_fullsized":"2024-01-16T15:02:28.702078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.686150+0000","last_change":"2024-01-16T15:02:28.686150+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.686150+0000","last_undegraded":"2024-01-16T15:02:28.686150+0000","last_fullsized":"2024-01-16T15:02:28.686150+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699635+0000","last_change":"2024-01-16T15:02:28.694170+0000","last_active":"2024-01-16T15:02:28.699635+0000","last_peered":"2024-01-16T15:02:28.699635+0000","last_clean":"2024-01-16T15:02:28.699635+0000","last_became_active":"2024-01-16T15:02:28.693951+0000","last_became_peered":"2024-01-16T15:02:28.693951+0000","last_unstale":"2024-01-16T15:02:28.699635+0000","last_undegraded":"2024-01-16T15:02:28.699635+0000","last_fullsized":"2024-01-16T15:02:28.699635+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:05.345755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740967+0000","last_change":"2024-01-16T15:02:29.728569+0000","last_active":"2024-01-16T15:02:29.740967+0000","last_peered":"2024-01-16T15:02:29.740967+0000","last_clean":"2024-01-16T15:02:29.740967+0000","last_became_active":"2024-01-16T15:02:29.728240+0000","last_became_peered":"2024-01-16T15:02:29.728240+0000","last_unstale":"2024-01-16T15:02:29.740967+0000","last_undegraded":"2024-01-16T15:02:29.740967+0000","last_fullsized":"2024-01-16T15:02:29.740967+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:49.783058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702091+0000","last_change":"2024-01-16T15:02:28.700598+0000","last_active":"2024-01-16T15:02:28.702091+0000","last_peered":"2024-01-16T15:02:28.702091+0000","last_clean":"2024-01-16T15:02:28.702091+0000","last_became_active":"2024-01-16T15:02:28.700296+0000","last_became_peered":"2024-01-16T15:02:28.700296+0000","last_unstale":"2024-01-16T15:02:28.702091+0000","last_undegraded":"2024-01-16T15:02:28.702091+0000","last_fullsized":"2024-01-16T15:02:28.702091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684658+0000","last_change":"2024-01-16T15:02:28.684658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684658+0000","last_undegraded":"2024-01-16T15:02:28.684658+0000","last_fullsized":"2024-01-16T15:02:28.684658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701099+0000","last_change":"2024-01-16T15:02:28.697153+0000","last_active":"2024-01-16T15:02:28.701099+0000","last_peered":"2024-01-16T15:02:28.701099+0000","last_clean":"2024-01-16T15:02:28.701099+0000","last_became_active":"2024-01-16T15:02:28.693885+0000","last_became_peered":"2024-01-16T15:02:28.693885+0000","last_unstale":"2024-01-16T15:02:28.701099+0000","last_undegraded":"2024-01-16T15:02:28.701099+0000","last_fullsized":"2024-01-16T15:02:28.701099+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683793+0000","last_change":"2024-01-16T15:02:28.683793+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683793+0000","last_undegraded":"2024-01-16T15:02:28.683793+0000","last_fullsized":"2024-01-16T15:02:28.683793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699827+0000","last_change":"2024-01-16T15:02:28.695099+0000","last_active":"2024-01-16T15:02:28.699827+0000","last_peered":"2024-01-16T15:02:28.699827+0000","last_clean":"2024-01-16T15:02:28.699827+0000","last_became_active":"2024-01-16T15:02:28.694828+0000","last_became_peered":"2024-01-16T15:02:28.694828+0000","last_unstale":"2024-01-16T15:02:28.699827+0000","last_undegraded":"2024-01-16T15:02:28.699827+0000","last_fullsized":"2024-01-16T15:02:28.699827+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:29.254715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680264+0000","last_change":"2024-01-16T15:02:28.680264+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680264+0000","last_undegraded":"2024-01-16T15:02:28.680264+0000","last_fullsized":"2024-01-16T15:02:28.680264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706804+0000","last_change":"2024-01-16T15:02:28.688577+0000","last_active":"2024-01-16T15:02:29.706804+0000","last_peered":"2024-01-16T15:02:29.706804+0000","last_clean":"2024-01-16T15:02:29.706804+0000","last_became_active":"2024-01-16T15:02:28.687999+0000","last_became_peered":"2024-01-16T15:02:28.687999+0000","last_unstale":"2024-01-16T15:02:29.706804+0000","last_undegraded":"2024-01-16T15:02:29.706804+0000","last_fullsized":"2024-01-16T15:02:29.706804+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:17.380916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673658+0000","last_change":"2024-01-16T15:02:28.673658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673658+0000","last_undegraded":"2024-01-16T15:02:28.673658+0000","last_fullsized":"2024-01-16T15:02:28.673658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697893+0000","last_change":"2024-01-16T15:02:28.694910+0000","last_active":"2024-01-16T15:02:28.697893+0000","last_peered":"2024-01-16T15:02:28.697893+0000","last_clean":"2024-01-16T15:02:28.697893+0000","last_became_active":"2024-01-16T15:02:28.690704+0000","last_became_peered":"2024-01-16T15:02:28.690704+0000","last_unstale":"2024-01-16T15:02:28.697893+0000","last_undegraded":"2024-01-16T15:02:28.697893+0000","last_fullsized":"2024-01-16T15:02:28.697893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678221+0000","last_change":"2024-01-16T15:02:28.678221+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678221+0000","last_undegraded":"2024-01-16T15:02:28.678221+0000","last_fullsized":"2024-01-16T15:02:28.678221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701998+0000","last_change":"2024-01-16T15:02:28.699544+0000","last_active":"2024-01-16T15:02:28.701998+0000","last_peered":"2024-01-16T15:02:28.701998+0000","last_clean":"2024-01-16T15:02:28.701998+0000","last_became_active":"2024-01-16T15:02:28.699169+0000","last_became_peered":"2024-01-16T15:02:28.699169+0000","last_unstale":"2024-01-16T15:02:28.701998+0000","last_undegraded":"2024-01-16T15:02:28.701998+0000","last_fullsized":"2024-01-16T15:02:28.701998+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687323+0000","last_change":"2024-01-16T15:02:28.687323+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687323+0000","last_undegraded":"2024-01-16T15:02:28.687323+0000","last_fullsized":"2024-01-16T15:02:28.687323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699992+0000","last_change":"2024-01-16T15:02:28.695124+0000","last_active":"2024-01-16T15:02:28.699992+0000","last_peered":"2024-01-16T15:02:28.699992+0000","last_clean":"2024-01-16T15:02:28.699992+0000","last_became_active":"2024-01-16T15:02:28.694847+0000","last_became_peered":"2024-01-16T15:02:28.694847+0000","last_unstale":"2024-01-16T15:02:28.699992+0000","last_undegraded":"2024-01-16T15:02:28.699992+0000","last_fullsized":"2024-01-16T15:02:28.699992+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:56.293921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684228+0000","last_change":"2024-01-16T15:02:28.684228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684228+0000","last_undegraded":"2024-01-16T15:02:28.684228+0000","last_fullsized":"2024-01-16T15:02:28.684228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700477+0000","last_change":"2024-01-16T15:02:28.698762+0000","last_active":"2024-01-16T15:02:28.700477+0000","last_peered":"2024-01-16T15:02:28.700477+0000","last_clean":"2024-01-16T15:02:28.700477+0000","last_became_active":"2024-01-16T15:02:28.698337+0000","last_became_peered":"2024-01-16T15:02:28.698337+0000","last_unstale":"2024-01-16T15:02:28.700477+0000","last_undegraded":"2024-01-16T15:02:28.700477+0000","last_fullsized":"2024-01-16T15:02:28.700477+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:40.971035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676992+0000","last_change":"2024-01-16T15:02:28.676992+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676992+0000","last_undegraded":"2024-01-16T15:02:28.676992+0000","last_fullsized":"2024-01-16T15:02:28.676992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698294+0000","last_change":"2024-01-16T15:02:28.695728+0000","last_active":"2024-01-16T15:02:28.698294+0000","last_peered":"2024-01-16T15:02:28.698294+0000","last_clean":"2024-01-16T15:02:28.698294+0000","last_became_active":"2024-01-16T15:02:28.695438+0000","last_became_peered":"2024-01-16T15:02:28.695438+0000","last_unstale":"2024-01-16T15:02:28.698294+0000","last_undegraded":"2024-01-16T15:02:28.698294+0000","last_fullsized":"2024-01-16T15:02:28.698294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679870+0000","last_change":"2024-01-16T15:02:28.679870+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679870+0000","last_undegraded":"2024-01-16T15:02:28.679870+0000","last_fullsized":"2024-01-16T15:02:28.679870+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696703+0000","last_change":"2024-01-16T15:02:28.690359+0000","last_active":"2024-01-16T15:02:28.696703+0000","last_peered":"2024-01-16T15:02:28.696703+0000","last_clean":"2024-01-16T15:02:28.696703+0000","last_became_active":"2024-01-16T15:02:28.689968+0000","last_became_peered":"2024-01-16T15:02:28.689968+0000","last_unstale":"2024-01-16T15:02:28.696703+0000","last_undegraded":"2024-01-16T15:02:28.696703+0000","last_fullsized":"2024-01-16T15:02:28.696703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:06.195041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683284+0000","last_change":"2024-01-16T15:02:28.683284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683284+0000","last_undegraded":"2024-01-16T15:02:28.683284+0000","last_fullsized":"2024-01-16T15:02:28.683284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689306+0000","last_change":"2024-01-16T15:02:28.687104+0000","last_active":"2024-01-16T15:02:28.689306+0000","last_peered":"2024-01-16T15:02:28.689306+0000","last_clean":"2024-01-16T15:02:28.689306+0000","last_became_active":"2024-01-16T15:02:28.686179+0000","last_became_peered":"2024-01-16T15:02:28.686179+0000","last_unstale":"2024-01-16T15:02:28.689306+0000","last_undegraded":"2024-01-16T15:02:28.689306+0000","last_fullsized":"2024-01-16T15:02:28.689306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:39.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672340+0000","last_change":"2024-01-16T15:02:28.672340+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672340+0000","last_undegraded":"2024-01-16T15:02:28.672340+0000","last_fullsized":"2024-01-16T15:02:28.672340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.722558+0000","last_change":"2024-01-16T15:02:28.693574+0000","last_active":"2024-01-16T15:02:29.722558+0000","last_peered":"2024-01-16T15:02:29.722558+0000","last_clean":"2024-01-16T15:02:29.722558+0000","last_became_active":"2024-01-16T15:02:28.692884+0000","last_became_peered":"2024-01-16T15:02:28.692884+0000","last_unstale":"2024-01-16T15:02:29.722558+0000","last_undegraded":"2024-01-16T15:02:29.722558+0000","last_fullsized":"2024-01-16T15:02:29.722558+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672000+0000","last_change":"2024-01-16T15:02:28.672000+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672000+0000","last_undegraded":"2024-01-16T15:02:28.672000+0000","last_fullsized":"2024-01-16T15:02:28.672000+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696515+0000","last_change":"2024-01-16T15:02:28.692170+0000","last_active":"2024-01-16T15:02:28.696515+0000","last_peered":"2024-01-16T15:02:28.696515+0000","last_clean":"2024-01-16T15:02:28.696515+0000","last_became_active":"2024-01-16T15:02:28.691779+0000","last_became_peered":"2024-01-16T15:02:28.691779+0000","last_unstale":"2024-01-16T15:02:28.696515+0000","last_undegraded":"2024-01-16T15:02:28.696515+0000","last_fullsized":"2024-01-16T15:02:28.696515+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681494+0000","last_change":"2024-01-16T15:02:28.681494+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681494+0000","last_undegraded":"2024-01-16T15:02:28.681494+0000","last_fullsized":"2024-01-16T15:02:28.681494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.737753+0000","last_change":"2024-01-16T15:02:28.689348+0000","last_active":"2024-01-16T15:02:29.737753+0000","last_peered":"2024-01-16T15:02:29.737753+0000","last_clean":"2024-01-16T15:02:29.737753+0000","last_became_active":"2024-01-16T15:02:28.689030+0000","last_became_peered":"2024-01-16T15:02:28.689030+0000","last_unstale":"2024-01-16T15:02:29.737753+0000","last_undegraded":"2024-01-16T15:02:29.737753+0000","last_fullsized":"2024-01-16T15:02:29.737753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:24.816323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683770+0000","last_change":"2024-01-16T15:02:28.683770+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683770+0000","last_undegraded":"2024-01-16T15:02:28.683770+0000","last_fullsized":"2024-01-16T15:02:28.683770+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689249+0000","last_change":"2024-01-16T15:02:28.681603+0000","last_active":"2024-01-16T15:02:28.689249+0000","last_peered":"2024-01-16T15:02:28.689249+0000","last_clean":"2024-01-16T15:02:28.689249+0000","last_became_active":"2024-01-16T15:02:28.681322+0000","last_became_peered":"2024-01-16T15:02:28.681322+0000","last_unstale":"2024-01-16T15:02:28.689249+0000","last_undegraded":"2024-01-16T15:02:28.689249+0000","last_fullsized":"2024-01-16T15:02:28.689249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:13.938868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740947+0000","last_change":"2024-01-16T15:02:29.728538+0000","last_active":"2024-01-16T15:02:29.740947+0000","last_peered":"2024-01-16T15:02:29.740947+0000","last_clean":"2024-01-16T15:02:29.740947+0000","last_became_active":"2024-01-16T15:02:29.728211+0000","last_became_peered":"2024-01-16T15:02:29.728211+0000","last_unstale":"2024-01-16T15:02:29.740947+0000","last_undegraded":"2024-01-16T15:02:29.740947+0000","last_fullsized":"2024-01-16T15:02:29.740947+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:32.872029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706859+0000","last_change":"2024-01-16T15:02:28.681634+0000","last_active":"2024-01-16T15:02:29.706859+0000","last_peered":"2024-01-16T15:02:29.706859+0000","last_clean":"2024-01-16T15:02:29.706859+0000","last_became_active":"2024-01-16T15:02:28.680098+0000","last_became_peered":"2024-01-16T15:02:28.680098+0000","last_unstale":"2024-01-16T15:02:29.706859+0000","last_undegraded":"2024-01-16T15:02:29.706859+0000","last_fullsized":"2024-01-16T15:02:29.706859+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:45.583344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671790+0000","last_change":"2024-01-16T15:02:28.671790+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671790+0000","last_undegraded":"2024-01-16T15:02:28.671790+0000","last_fullsized":"2024-01-16T15:02:28.671790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697195+0000","last_change":"2024-01-16T15:02:28.692899+0000","last_active":"2024-01-16T15:02:28.697195+0000","last_peered":"2024-01-16T15:02:28.697195+0000","last_clean":"2024-01-16T15:02:28.697195+0000","last_became_active":"2024-01-16T15:02:28.692607+0000","last_became_peered":"2024-01-16T15:02:28.692607+0000","last_unstale":"2024-01-16T15:02:28.697195+0000","last_undegraded":"2024-01-16T15:02:28.697195+0000","last_fullsized":"2024-01-16T15:02:28.697195+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674849+0000","last_change":"2024-01-16T15:02:28.674849+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674849+0000","last_undegraded":"2024-01-16T15:02:28.674849+0000","last_fullsized":"2024-01-16T15:02:28.674849+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.728464+0000","last_change":"2024-01-16T15:02:28.695802+0000","last_active":"2024-01-16T15:02:29.728464+0000","last_peered":"2024-01-16T15:02:29.728464+0000","last_clean":"2024-01-16T15:02:29.728464+0000","last_became_active":"2024-01-16T15:02:28.695516+0000","last_became_peered":"2024-01-16T15:02:28.695516+0000","last_unstale":"2024-01-16T15:02:29.728464+0000","last_undegraded":"2024-01-16T15:02:29.728464+0000","last_fullsized":"2024-01-16T15:02:29.728464+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:00.682347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679260+0000","last_change":"2024-01-16T15:02:28.679260+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679260+0000","last_undegraded":"2024-01-16T15:02:28.679260+0000","last_fullsized":"2024-01-16T15:02:28.679260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696398+0000","last_change":"2024-01-16T15:02:28.691759+0000","last_active":"2024-01-16T15:02:28.696398+0000","last_peered":"2024-01-16T15:02:28.696398+0000","last_clean":"2024-01-16T15:02:28.696398+0000","last_became_active":"2024-01-16T15:02:28.691457+0000","last_became_peered":"2024-01-16T15:02:28.691457+0000","last_unstale":"2024-01-16T15:02:28.696398+0000","last_undegraded":"2024-01-16T15:02:28.696398+0000","last_fullsized":"2024-01-16T15:02:28.696398+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:31.867536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.738586+0000","last_change":"2024-01-16T15:02:29.719394+0000","last_active":"2024-01-16T15:02:29.738586+0000","last_peered":"2024-01-16T15:02:29.738586+0000","last_clean":"2024-01-16T15:02:29.738586+0000","last_became_active":"2024-01-16T15:02:29.719030+0000","last_became_peered":"2024-01-16T15:02:29.719030+0000","last_unstale":"2024-01-16T15:02:29.738586+0000","last_undegraded":"2024-01-16T15:02:29.738586+0000","last_fullsized":"2024-01-16T15:02:29.738586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:51.117200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.717109+0000","last_change":"2024-01-16T15:02:28.692453+0000","last_active":"2024-01-16T15:02:29.717109+0000","last_peered":"2024-01-16T15:02:29.717109+0000","last_clean":"2024-01-16T15:02:29.717109+0000","last_became_active":"2024-01-16T15:02:28.692132+0000","last_became_peered":"2024-01-16T15:02:28.692132+0000","last_unstale":"2024-01-16T15:02:29.717109+0000","last_undegraded":"2024-01-16T15:02:29.717109+0000","last_fullsized":"2024-01-16T15:02:29.717109+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674276+0000","last_change":"2024-01-16T15:02:28.674276+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674276+0000","last_undegraded":"2024-01-16T15:02:28.674276+0000","last_fullsized":"2024-01-16T15:02:28.674276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700041+0000","last_change":"2024-01-16T15:02:28.696929+0000","last_active":"2024-01-16T15:02:28.700041+0000","last_peered":"2024-01-16T15:02:28.700041+0000","last_clean":"2024-01-16T15:02:28.700041+0000","last_became_active":"2024-01-16T15:02:28.696616+0000","last_became_peered":"2024-01-16T15:02:28.696616+0000","last_unstale":"2024-01-16T15:02:28.700041+0000","last_undegraded":"2024-01-16T15:02:28.700041+0000","last_fullsized":"2024-01-16T15:02:28.700041+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:29.321021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675396+0000","last_change":"2024-01-16T15:02:28.675396+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675396+0000","last_undegraded":"2024-01-16T15:02:28.675396+0000","last_fullsized":"2024-01-16T15:02:28.675396+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.708768+0000","last_change":"2024-01-16T15:02:28.695669+0000","last_active":"2024-01-16T15:02:29.708768+0000","last_peered":"2024-01-16T15:02:29.708768+0000","last_clean":"2024-01-16T15:02:29.708768+0000","last_became_active":"2024-01-16T15:02:28.695350+0000","last_became_peered":"2024-01-16T15:02:28.695350+0000","last_unstale":"2024-01-16T15:02:29.708768+0000","last_undegraded":"2024-01-16T15:02:29.708768+0000","last_fullsized":"2024-01-16T15:02:29.708768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:36.266476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":199,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-16T15:02:27.660341+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:02:27.660341+0000","last_peered":"2024-01-16T15:02:27.660341+0000","last_clean":"2024-01-16T15:02:27.660341+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:02:27.660341+0000","last_undegraded":"2024-01-16T15:02:27.660341+0000","last_fullsized":"2024-01-16T15:02:27.660341+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.0329999999999999,"5min":1.3480000000000001,"15min":1.1539999999999999},"min":{"1min":0.73899999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3089999999999999,"15min":1.2010000000000001},"min":{"1min":0.76700000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9319999999999999,"5min":2.774,"15min":2.774},"last":1.1080000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.2909999999999999,"15min":1.111},"min":{"1min":0.76400000000000001,"5min":0.63,"15min":0.63},"max":{"1min":11.487,"5min":11.487,"15min":11.487},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3380000000000001,"15min":1.1890000000000001},"min":{"1min":0.89000000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.8109999999999999,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":0.82199999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.288,"15min":1.1970000000000001},"min":{"1min":0.85599999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.7269999999999999,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.28},{"interface":"front","average":{"1min":2.8940000000000001,"5min":1.6160000000000001,"15min":1.349},"min":{"1min":0.84199999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":20.577000000000002,"5min":20.577000000000002,"15min":20.577000000000002},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2639999999999998,"5min":1.6259999999999999,"15min":1.3979999999999999},"min":{"1min":1.0569999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":11.872,"5min":12.861000000000001,"15min":12.861000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.486,"15min":1.4099999999999999},"min":{"1min":0.81100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.1379999999999999,"5min":4.0720000000000001,"15min":4.0720000000000001},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.8439999999999999,"5min":1.631,"15min":1.446},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":20.649999999999999,"5min":20.649999999999999,"15min":20.649999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.387,"5min":1.633,"15min":1.5109999999999999},"min":{"1min":0.94399999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":11.702,"5min":11.702,"15min":11.702},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.514,"15min":1.526},"min":{"1min":0.70499999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.8039999999999998,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.73,"15min":1.6719999999999999},"min":{"1min":1.0149999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":11.225,"5min":11.225,"15min":11.225},"last":1.0649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":4.1589999999999998,"5min":2.0630000000000002,"15min":1.7430000000000001},"min":{"1min":1.306,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":41.206000000000003,"5min":41.206000000000003,"15min":41.206000000000003},"last":1.155},{"interface":"front","average":{"1min":4.125,"5min":2.0449999999999999,"15min":1.7589999999999999},"min":{"1min":1.0940000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":21.908000000000001,"5min":21.908000000000001,"15min":21.908000000000001},"last":1.3280000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.2609999999999999,"15min":1.147},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":9.1890000000000001,"5min":9.1890000000000001,"15min":9.1890000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":5.2039999999999997,"5min":1.976,"15min":1.478},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":50.472000000000001,"5min":50.472000000000001,"15min":50.472000000000001},"last":0.80600000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":1.3979999999999999,"15min":1.1459999999999999},"min":{"1min":0.66300000000000003,"5min":0.624,"15min":0.624},"max":{"1min":10.629,"5min":10.629,"15min":10.629},"last":0.98399999999999999},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.5509999999999999,"15min":1.421},"min":{"1min":0.82699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":9.3279999999999994,"5min":9.3279999999999994,"15min":9.3279999999999994},"last":1.0569999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.3789999999999996,"5min":1.9450000000000001,"15min":1.4199999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":50.643999999999998,"5min":50.643999999999998,"15min":50.643999999999998},"last":1.385},{"interface":"front","average":{"1min":2.4169999999999998,"5min":1.4970000000000001,"15min":1.359},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":10.843,"5min":10.843,"15min":10.843},"last":1.607}]},{"osd":3,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":1.6100000000000001,"15min":1.4359999999999999},"min":{"1min":0.82399999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":15.675000000000001,"5min":15.675000000000001,"15min":15.675000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":4.3760000000000003,"5min":1.99,"15min":1.5800000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":50.561,"5min":50.561,"15min":50.561},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.8289999999999997,"5min":2.1840000000000002,"15min":1.728},"min":{"1min":0.92700000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":47.488,"5min":47.488,"15min":47.488},"last":1.542},{"interface":"front","average":{"1min":5.2519999999999998,"5min":2.2330000000000001,"15min":1.7609999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":50.381,"5min":50.381,"15min":50.381},"last":1.4850000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.5419999999999998,"5min":1.6659999999999999,"15min":1.5409999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":10.654,"5min":10.654,"15min":10.654},"last":1.7450000000000001},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.8280000000000001,"15min":1.726},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":9.3729999999999993,"5min":9.3729999999999993,"15min":9.3729999999999993},"last":1.7789999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.9820000000000002,"5min":2.2799999999999998,"15min":1.8779999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":52.912999999999997,"5min":52.912999999999997,"15min":52.912999999999997},"last":1.581},{"interface":"front","average":{"1min":2.5270000000000001,"5min":1.7769999999999999,"15min":1.7130000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":10.581,"5min":10.581,"15min":10.581},"last":1.8600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3109999999999999,"5min":1.3580000000000001,"15min":1.248},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":13.938000000000001,"5min":13.938000000000001,"15min":13.938000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":2.9500000000000002,"5min":1.478,"15min":1.212},"min":{"1min":0.73199999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":16.896000000000001,"5min":16.896000000000001,"15min":16.896000000000001},"last":1.591}]},{"osd":1,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.482,"15min":1.353},"min":{"1min":0.77500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":3.4550000000000001,"5min":1.7549999999999999,"15min":1.345},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":35.328000000000003,"5min":35.328000000000003,"15min":35.328000000000003},"last":1.554}]},{"osd":2,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.7639999999999998,"5min":1.8859999999999999,"15min":1.47},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":35.398000000000003,"5min":35.398000000000003,"15min":35.398000000000003},"last":1.091},{"interface":"front","average":{"1min":1.782,"5min":1.3340000000000001,"15min":1.29},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.883,"5min":4.883,"15min":4.883},"last":1.7150000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8210000000000002,"5min":1.667,"15min":1.4430000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":13.202,"5min":13.202,"15min":13.202},"last":1.4810000000000001},{"interface":"front","average":{"1min":3.6669999999999998,"5min":1.9670000000000001,"15min":1.587},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":35.268000000000001,"5min":35.268000000000001,"15min":35.268000000000001},"last":1.6799999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.5470000000000002,"5min":1.784,"15min":1.579},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":15.757,"5min":15.757,"15min":15.757},"last":1.0069999999999999},{"interface":"front","average":{"1min":4.3479999999999999,"5min":2.004,"15min":1.714},"min":{"1min":0.63300000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":26.166,"5min":26.166,"15min":26.166},"last":1.7709999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":4.0350000000000001,"5min":2.0950000000000002,"15min":1.768},"min":{"1min":0.53400000000000003,"5min":0.499,"15min":0.499},"max":{"1min":35.427999999999997,"5min":35.427999999999997,"15min":35.427999999999997},"last":1.7410000000000001},{"interface":"front","average":{"1min":3.4740000000000002,"5min":1.903,"15min":1.7589999999999999},"min":{"1min":0.53100000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":1.6259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.9350000000000001,"5min":2.0379999999999998,"15min":1.7689999999999999},"min":{"1min":0.58099999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":19.364999999999998,"5min":19.364999999999998,"15min":19.364999999999998},"last":1.837},{"interface":"front","average":{"1min":4.0419999999999998,"5min":2.1829999999999998,"15min":1.8839999999999999},"min":{"1min":0.71699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":35.363,"5min":35.363,"15min":35.363},"last":1.994}]}]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.1779999999999999,"15min":1.1080000000000001},"min":{"1min":0.751,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":4.2869999999999999,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":1.012},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.4059999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":8.0440000000000005,"5min":8.0440000000000005,"15min":8.0440000000000005},"last":0.90300000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.4359999999999999,"15min":1.258},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.1050000000000004,"5min":7.1050000000000004,"15min":7.1050000000000004},"last":1.391},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.2889999999999999,"15min":1.234},"min":{"1min":0.84499999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":5.7290000000000001,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":1.137}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.583,"15min":1.415},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.4860000000000002,"5min":15.986000000000001,"15min":15.986000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.554,"5min":1.3520000000000001,"15min":1.4330000000000001},"min":{"1min":0.83699999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":3.762,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":1.232}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":3.0049999999999999,"5min":1.871,"15min":1.6819999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5600000000000001,"15min":1.48},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":4.0490000000000004,"5min":21.706,"15min":21.706},"last":1.538}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4370000000000001,"15min":1.575},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":4.327,"5min":4.327,"15min":4.327},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.649,"15min":1.6279999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":10.831,"5min":10.831,"15min":10.831},"last":1.504}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4530000000000001,"15min":1.575},"min":{"1min":0.66000000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":1.476},{"interface":"front","average":{"1min":2.468,"5min":1.734,"15min":1.732},"min":{"1min":0.74199999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":20.939,"5min":20.939,"15min":20.939},"last":1.4399999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.8,"15min":1.7490000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":7.0110000000000001,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":1.591},{"interface":"front","average":{"1min":1.976,"5min":1.6399999999999999,"15min":1.738},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":10.978999999999999,"5min":10.978999999999999,"15min":10.978999999999999},"last":1.6279999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.98499999999999999,"15min":0.91900000000000004},"min":{"1min":0.433,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1459999999999999,"15min":0.93600000000000005},"min":{"1min":0.56799999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":1.099}]},{"osd":1,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.92500000000000004,"15min":0.85999999999999999},"min":{"1min":0.47099999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.752,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.121,"5min":1.0620000000000001,"15min":1.016},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.4590000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.075,"15min":1.075},"min":{"1min":0.59299999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9339999999999999,"5min":4.008,"15min":4.008},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.0820000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.2069999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.2929999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.3500000000000001,"15min":1.1639999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":7.7439999999999998,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.492,"15min":1.3129999999999999},"min":{"1min":0.80500000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":11.068,"5min":11.068,"15min":11.068},"last":1.208}]},{"osd":5,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4630000000000001,"15min":1.3180000000000001},"min":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":12.791,"5min":12.791,"15min":12.791},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.508,"15min":1.413},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":12.106,"5min":12.106,"15min":12.106},"last":1.5389999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":1.546,"15min":1.3540000000000001},"min":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":8.1039999999999992,"5min":8.1039999999999992,"15min":8.1039999999999992},"last":1.502},{"interface":"front","average":{"1min":2.3980000000000001,"5min":1.6359999999999999,"15min":1.4379999999999999},"min":{"1min":0.91900000000000004,"5min":0.752,"15min":0.752},"max":{"1min":6.891,"5min":6.891,"15min":6.891},"last":1.4890000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.532,"15min":1.4379999999999999},"min":{"1min":0.82599999999999996,"5min":0.755,"15min":0.755},"max":{"1min":3.7269999999999999,"5min":3.7269999999999999,"15min":3.7269999999999999},"last":1.599},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.677,"15min":1.4299999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":7.6159999999999997,"5min":18.846,"15min":18.846},"last":1.5640000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":0.91600000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.839,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.378},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0049999999999999,"15min":0.93600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.353,"5min":4.5010000000000003,"15min":4.5010000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0609999999999999,"15min":1.0329999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.133,"5min":3.798,"15min":3.798},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.125,"15min":1.014},"min":{"1min":0.45800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.8069999999999999,"5min":4.8250000000000002,"15min":4.8250000000000002},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0620000000000001,"15min":0.97999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.9870000000000001,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.224,"15min":1.1399999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":8.4610000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.5489999999999999,"15min":1.3939999999999999},"min":{"1min":0.67700000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":11.843,"5min":11.843,"15min":11.843},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.847,"5min":1.54,"15min":1.411},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":11.768000000000001,"5min":11.768000000000001,"15min":11.768000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":1.72,"15min":1.4470000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":26.501000000000001,"5min":26.501000000000001,"15min":26.501000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.389,"15min":1.341},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.8640000000000001,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.593,"15min":1.4430000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.5059999999999998,"5min":15.144,"15min":15.144},"last":0.56799999999999995},{"interface":"front","average":{"1min":5.2089999999999996,"5min":2.214,"15min":1.6599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":66.061999999999998,"5min":66.061999999999998,"15min":66.061999999999998},"last":0.63300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.581,"15min":1.5049999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":4.0750000000000002,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.554,"5min":1.5509999999999999,"15min":1.4850000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":4.0519999999999996,"5min":6.0590000000000002,"15min":6.0590000000000002},"last":0.5}]}]},{"osd":1,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.92800000000000005,"15min":0.93000000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.089,"5min":1.026,"15min":1.0149999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":1.298}]},{"osd":2,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.0369999999999999,"15min":0.95499999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":14.817,"5min":14.817,"15min":14.817},"last":0.996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.028,"15min":0.98299999999999998},"min":{"1min":0.47199999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.3460000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.075,"15min":1.071},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.8680000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.1839999999999999,"15min":1.0640000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":20.439,"5min":20.439,"15min":20.439},"last":1.0469999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.4359999999999999,"15min":1.256},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":13.641999999999999,"5min":13.641999999999999,"15min":13.641999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.4670000000000001,"15min":1.3260000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":13.789999999999999,"5min":13.789999999999999,"15min":13.789999999999999},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.516,"15min":1.4419999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":14.129,"5min":14.129,"15min":14.129},"last":1.377},{"interface":"front","average":{"1min":3.399,"5min":1.8160000000000001,"15min":1.5609999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":35.747,"5min":35.747,"15min":35.747},"last":1.466}]},{"osd":6,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9359999999999999,"5min":1.5669999999999999,"15min":1.4810000000000001},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":11.545,"5min":11.545,"15min":11.545},"last":1.5920000000000001},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.5600000000000001,"15min":1.4690000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":13.56,"5min":13.56,"15min":13.56},"last":1.101}]},{"osd":7,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.629,"15min":1.4770000000000001},"min":{"1min":0.70699999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":6.2969999999999997,"5min":12.494999999999999,"15min":12.494999999999999},"last":1.405},{"interface":"front","average":{"1min":2.7120000000000002,"5min":1.6970000000000001,"15min":1.524},"min":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":31.044,"5min":31.044,"15min":31.044},"last":1.6870000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.98399999999999999,"15min":0.97599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.4379999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.2230000000000001,"15min":1.0569999999999999},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":4.3550000000000004,"5min":17.876000000000001,"15min":17.876000000000001},"last":0.79700000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.357,"15min":1.1839999999999999},"min":{"1min":0.54200000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":5.9080000000000004,"5min":17.760999999999999,"15min":17.760999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.226,"15min":1.232},"min":{"1min":0.56100000000000005,"5min":0.38,"15min":0.38},"max":{"1min":1.9139999999999999,"5min":5.5970000000000004,"15min":5.5970000000000004},"last":0.875}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.401,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.3520000000000001,"15min":1.137},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.4169999999999998,"5min":17.981000000000002,"15min":17.981000000000002},"last":0.95599999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.6040000000000001,"15min":1.4870000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":7.0389999999999997,"5min":14.297000000000001,"15min":14.297000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":2.9910000000000001,"5min":1.8939999999999999,"15min":1.4830000000000001},"min":{"1min":0.83999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":18.347999999999999,"5min":19.422000000000001,"15min":19.422000000000001},"last":1.1919999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":1.716,"15min":1.5209999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":10.933,"5min":14.212999999999999,"15min":14.212999999999999},"last":1.169},{"interface":"front","average":{"1min":2.319,"5min":1.79,"15min":1.452},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":16.446999999999999,"5min":19.315000000000001,"15min":19.315000000000001},"last":1.292}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":4.1150000000000002,"5min":2.2349999999999999,"15min":1.677},"min":{"1min":0.81499999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":27.442,"5min":27.442,"15min":27.442},"last":1.321},{"interface":"front","average":{"1min":3.9689999999999999,"5min":2.1800000000000002,"15min":1.7629999999999999},"min":{"1min":0.70899999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":27.411999999999999,"5min":27.411999999999999,"15min":27.411999999999999},"last":1.3440000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":1.8819999999999999,"15min":1.5640000000000001},"min":{"1min":0.78700000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":7.4130000000000003,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.6819999999999999,"15min":1.6259999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":5.335,"5min":5.9269999999999996,"15min":5.9269999999999996},"last":1.262}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:34.097 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:34.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.353+0000 7f85f2800640 1 Processor -- start 2024-01-16T15:02:34.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.353+0000 7f85f2800640 1 -- start start 2024-01-16T15:02:34.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85f2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85ec15b090 0x7f85ec0b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85f2800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f85ec06bb00 0x7f85ec0cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85f2800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec0ce770 con 0x7f85ec0b2be0 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85f2800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec0ccd40 con 0x7f85ec15b090 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85f2800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec0ce180 con 0x7f85ec06bb00 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85eb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85ec15b090 0x7f85ec0b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85eb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85ec15b090 0x7f85ec0b26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59760/0 (socket says 172.21.15.141:59760) 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.355+0000 7f85eb400640 1 -- 172.21.15.141:0/4207691695 learned_addr learned my addr 172.21.15.141:0/4207691695 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.356+0000 7f85ebe00640 1 --2- 172.21.15.141:0/4207691695 >> v2:172.21.15.188:3300/0 conn(0x7f85ec06bb00 0x7f85ec0cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.356+0000 7f85eb400640 1 -- 172.21.15.141:0/4207691695 >> v2:172.21.15.188:3300/0 conn(0x7f85ec06bb00 msgr2=0x7f85ec0cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.356+0000 7f85eb400640 1 --2- 172.21.15.141:0/4207691695 >> v2:172.21.15.188:3300/0 conn(0x7f85ec06bb00 0x7f85ec0cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.356+0000 7f85eb400640 1 -- 172.21.15.141:0/4207691695 >> v1:172.21.15.188:6789/0 conn(0x7f85ec0b2be0 legacy=0x7f85ec06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.356+0000 7f85eb400640 1 -- 172.21.15.141:0/4207691695 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec0715a0 con 0x7f85ec15b090 2024-01-16T15:02:34.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.357+0000 7f85eb400640 1 --2- 172.21.15.141:0/4207691695 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec0b26a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0b4b40 tx=0x7f85dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b89eac128ff4ebf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.357+0000 7f85eaa00640 1 -- 172.21.15.141:0/4207691695 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc031870 con 0x7f85ec15b090 2024-01-16T15:02:34.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.357+0000 7f85eaa00640 1 -- 172.21.15.141:0/4207691695 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc0319d0 con 0x7f85ec15b090 2024-01-16T15:02:34.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.357+0000 7f85eaa00640 1 -- 172.21.15.141:0/4207691695 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc031d60 con 0x7f85ec15b090 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 -- 172.21.15.141:0/4207691695 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 msgr2=0x7f85ec0b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 --2- 172.21.15.141:0/4207691695 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec0b26a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0b4b40 tx=0x7f85dc02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 -- 172.21.15.141:0/4207691695 shutdown_connections 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 --2- 172.21.15.141:0/4207691695 >> v2:172.21.15.188:3300/0 conn(0x7f85ec06bb00 0x7f85ec0cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 --2- 172.21.15.141:0/4207691695 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec0b26a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 -- 172.21.15.141:0/4207691695 >> 172.21.15.141:0/4207691695 conn(0x7f85ec070a80 msgr2=0x7f85ec070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 -- 172.21.15.141:0/4207691695 shutdown_connections 2024-01-16T15:02:34.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.358+0000 7f85f2800640 1 -- 172.21.15.141:0/4207691695 wait complete. 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.360+0000 7f85f2800640 1 Processor -- start 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.360+0000 7f85f2800640 1 -- start start 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85f2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85ec06bb00 0x7f85ec14efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85f2800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85ec0b2be0 0x7f85ec14fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85f2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85f2800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec15c150 con 0x7f85ec15b090 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85f2800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec15c710 con 0x7f85ec0b2be0 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85f2800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec14e220 con 0x7f85ec06bb00 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec148320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59774/0 (socket says 172.21.15.141:59774) 2024-01-16T15:02:34.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 -- 172.21.15.141:0/1355114826 learned_addr learned my addr 172.21.15.141:0/1355114826 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85eb400640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85ec06bb00 0x7f85ec14efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85e3e00640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85ec0b2be0 0x7f85ec14fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 -- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85ec06bb00 msgr2=0x7f85ec14efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85ec06bb00 0x7f85ec14efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 -- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85ec0b2be0 msgr2=0x7f85ec14fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85ec0b2be0 0x7f85ec14fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.361+0000 7f85ebe00640 1 -- 172.21.15.141:0/1355114826 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0094d0 con 0x7f85ec15b090 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.362+0000 7f85ebe00640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec148320 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f85e400cda0 tx=0x7f85e400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.362+0000 7f85e3400640 1 -- 172.21.15.141:0/1355114826 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4003b90 con 0x7f85ec15b090 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.362+0000 7f85e3400640 1 -- 172.21.15.141:0/1355114826 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e4003cf0 con 0x7f85ec15b090 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.362+0000 7f85e3400640 1 -- 172.21.15.141:0/1355114826 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4004240 con 0x7f85ec15b090 2024-01-16T15:02:34.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.362+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec14e500 con 0x7f85ec15b090 2024-01-16T15:02:34.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.362+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec14de80 con 0x7f85ec15b090 2024-01-16T15:02:34.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.363+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec0675d0 con 0x7f85ec15b090 2024-01-16T15:02:34.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.364+0000 7f85e3400640 1 -- 172.21.15.141:0/1355114826 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f85e40043a0 con 0x7f85ec15b090 2024-01-16T15:02:34.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.366+0000 7f85e3400640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f85c406b860 0x7f85c406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.366+0000 7f85e3400640 1 -- 172.21.15.141:0/1355114826 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85e4012070 con 0x7f85ec15b090 2024-01-16T15:02:34.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.366+0000 7f85eb400640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f85c406b860 0x7f85c406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.367+0000 7f85eb400640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f85c406b860 0x7f85c406dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f85dc036000 tx=0x7f85dc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.370+0000 7f85e3400640 1 -- 172.21.15.141:0/1355114826 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85e40556a0 con 0x7f85ec15b090 2024-01-16T15:02:34.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.559+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85ec0ce400 con 0x7f85c406b860 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.586+0000 7f85e3400640 1 -- 172.21.15.141:0/1355114826 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282293 (secure 0 0 0) 0x7f85dc031cc0 con 0x7f85c406b860 2024-01-16T15:02:34.571 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:34.573 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:02:34.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f85c406b860 msgr2=0x7f85c406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f85c406b860 0x7f85c406dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f85dc036000 tx=0x7f85dc03c040 comp rx=0 tx=0).stop 2024-01-16T15:02:34.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 msgr2=0x7f85ec148320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec148320 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f85e400cda0 tx=0x7f85e400a660 comp rx=0 tx=0).stop 2024-01-16T15:02:34.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 shutdown_connections 2024-01-16T15:02:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f85c406b860 0x7f85c406dd20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85ec15b090 0x7f85ec148320 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85ec0b2be0 0x7f85ec14fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 --2- 172.21.15.141:0/1355114826 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85ec06bb00 0x7f85ec14efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.590+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 >> 172.21.15.141:0/1355114826 conn(0x7f85ec070a80 msgr2=0x7f85ec0ae070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.591+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 shutdown_connections 2024-01-16T15:02:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.591+0000 7f85f2800640 1 -- 172.21.15.141:0/1355114826 wait complete. 2024-01-16T15:02:34.590 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-01-16T15:02:32.575663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289216,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685504,"statfs":{"total":773094113280,"available":772797956096,"internally_reserved":0,"allocated":6627328,"data_stored":3419944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.917962"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.691258+0000","last_change":"2024-01-16T15:02:28.691258+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.691258+0000","last_undegraded":"2024-01-16T15:02:28.691258+0000","last_fullsized":"2024-01-16T15:02:28.691258+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674881+0000","last_change":"2024-01-16T15:02:28.674881+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674881+0000","last_undegraded":"2024-01-16T15:02:28.674881+0000","last_fullsized":"2024-01-16T15:02:28.674881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.685084+0000","last_change":"2024-01-16T15:02:28.685084+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.685084+0000","last_undegraded":"2024-01-16T15:02:28.685084+0000","last_fullsized":"2024-01-16T15:02:28.685084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682154+0000","last_change":"2024-01-16T15:02:28.682154+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682154+0000","last_undegraded":"2024-01-16T15:02:28.682154+0000","last_fullsized":"2024-01-16T15:02:28.682154+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687284+0000","last_change":"2024-01-16T15:02:28.687284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687284+0000","last_undegraded":"2024-01-16T15:02:28.687284+0000","last_fullsized":"2024-01-16T15:02:28.687284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680550+0000","last_change":"2024-01-16T15:02:28.680550+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680550+0000","last_undegraded":"2024-01-16T15:02:28.680550+0000","last_fullsized":"2024-01-16T15:02:28.680550+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677104+0000","last_change":"2024-01-16T15:02:28.677104+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677104+0000","last_undegraded":"2024-01-16T15:02:28.677104+0000","last_fullsized":"2024-01-16T15:02:28.677104+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677904+0000","last_change":"2024-01-16T15:02:28.677904+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677904+0000","last_undegraded":"2024-01-16T15:02:28.677904+0000","last_fullsized":"2024-01-16T15:02:28.677904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671452+0000","last_change":"2024-01-16T15:02:28.671452+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671452+0000","last_undegraded":"2024-01-16T15:02:28.671452+0000","last_fullsized":"2024-01-16T15:02:28.671452+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740996+0000","last_change":"2024-01-16T15:02:29.726552+0000","last_active":"2024-01-16T15:02:29.740996+0000","last_peered":"2024-01-16T15:02:29.740996+0000","last_clean":"2024-01-16T15:02:29.740996+0000","last_became_active":"2024-01-16T15:02:29.726242+0000","last_became_peered":"2024-01-16T15:02:29.726242+0000","last_unstale":"2024-01-16T15:02:29.740996+0000","last_undegraded":"2024-01-16T15:02:29.740996+0000","last_fullsized":"2024-01-16T15:02:29.740996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:15.816925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.743873+0000","last_change":"2024-01-16T15:02:29.735792+0000","last_active":"2024-01-16T15:02:29.743873+0000","last_peered":"2024-01-16T15:02:29.743873+0000","last_clean":"2024-01-16T15:02:29.743873+0000","last_became_active":"2024-01-16T15:02:29.735504+0000","last_became_peered":"2024-01-16T15:02:29.735504+0000","last_unstale":"2024-01-16T15:02:29.743873+0000","last_undegraded":"2024-01-16T15:02:29.743873+0000","last_fullsized":"2024-01-16T15:02:29.743873+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:09.173541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681639+0000","last_change":"2024-01-16T15:02:28.681639+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681639+0000","last_undegraded":"2024-01-16T15:02:28.681639+0000","last_fullsized":"2024-01-16T15:02:28.681639+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726359+0000","last_change":"2024-01-16T15:02:29.721939+0000","last_active":"2024-01-16T15:02:29.726359+0000","last_peered":"2024-01-16T15:02:29.726359+0000","last_clean":"2024-01-16T15:02:29.726359+0000","last_became_active":"2024-01-16T15:02:29.721563+0000","last_became_peered":"2024-01-16T15:02:29.721563+0000","last_unstale":"2024-01-16T15:02:29.726359+0000","last_undegraded":"2024-01-16T15:02:29.726359+0000","last_fullsized":"2024-01-16T15:02:29.726359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:35:52.362185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677387+0000","last_change":"2024-01-16T15:02:28.677387+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677387+0000","last_undegraded":"2024-01-16T15:02:28.677387+0000","last_fullsized":"2024-01-16T15:02:28.677387+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671228+0000","last_change":"2024-01-16T15:02:28.671228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671228+0000","last_undegraded":"2024-01-16T15:02:28.671228+0000","last_fullsized":"2024-01-16T15:02:28.671228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679199+0000","last_change":"2024-01-16T15:02:28.679199+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679199+0000","last_undegraded":"2024-01-16T15:02:28.679199+0000","last_fullsized":"2024-01-16T15:02:28.679199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674612+0000","last_change":"2024-01-16T15:02:28.674612+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674612+0000","last_undegraded":"2024-01-16T15:02:28.674612+0000","last_fullsized":"2024-01-16T15:02:28.674612+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.688365+0000","last_change":"2024-01-16T15:02:28.688365+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.688365+0000","last_undegraded":"2024-01-16T15:02:28.688365+0000","last_fullsized":"2024-01-16T15:02:28.688365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684341+0000","last_change":"2024-01-16T15:02:28.684341+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684341+0000","last_undegraded":"2024-01-16T15:02:28.684341+0000","last_fullsized":"2024-01-16T15:02:28.684341+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677483+0000","last_change":"2024-01-16T15:02:28.677483+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677483+0000","last_undegraded":"2024-01-16T15:02:28.677483+0000","last_fullsized":"2024-01-16T15:02:28.677483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680887+0000","last_change":"2024-01-16T15:02:28.680887+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680887+0000","last_undegraded":"2024-01-16T15:02:28.680887+0000","last_fullsized":"2024-01-16T15:02:28.680887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675926+0000","last_change":"2024-01-16T15:02:28.675926+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675926+0000","last_undegraded":"2024-01-16T15:02:28.675926+0000","last_fullsized":"2024-01-16T15:02:28.675926+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680236+0000","last_change":"2024-01-16T15:02:28.680236+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680236+0000","last_undegraded":"2024-01-16T15:02:28.680236+0000","last_fullsized":"2024-01-16T15:02:28.680236+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726074+0000","last_change":"2024-01-16T15:02:29.721477+0000","last_active":"2024-01-16T15:02:29.726074+0000","last_peered":"2024-01-16T15:02:29.726074+0000","last_clean":"2024-01-16T15:02:29.726074+0000","last_became_active":"2024-01-16T15:02:29.721137+0000","last_became_peered":"2024-01-16T15:02:29.721137+0000","last_unstale":"2024-01-16T15:02:29.726074+0000","last_undegraded":"2024-01-16T15:02:29.726074+0000","last_fullsized":"2024-01-16T15:02:29.726074+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:31.998282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.690190+0000","last_change":"2024-01-16T15:02:28.690190+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.690190+0000","last_undegraded":"2024-01-16T15:02:28.690190+0000","last_fullsized":"2024-01-16T15:02:28.690190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683203+0000","last_change":"2024-01-16T15:02:28.683203+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683203+0000","last_undegraded":"2024-01-16T15:02:28.683203+0000","last_fullsized":"2024-01-16T15:02:28.683203+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678571+0000","last_change":"2024-01-16T15:02:28.678571+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678571+0000","last_undegraded":"2024-01-16T15:02:28.678571+0000","last_fullsized":"2024-01-16T15:02:28.678571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687166+0000","last_change":"2024-01-16T15:02:28.687166+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687166+0000","last_undegraded":"2024-01-16T15:02:28.687166+0000","last_fullsized":"2024-01-16T15:02:28.687166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674298+0000","last_change":"2024-01-16T15:02:28.674298+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674298+0000","last_undegraded":"2024-01-16T15:02:28.674298+0000","last_fullsized":"2024-01-16T15:02:28.674298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682909+0000","last_change":"2024-01-16T15:02:28.682909+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682909+0000","last_undegraded":"2024-01-16T15:02:28.682909+0000","last_fullsized":"2024-01-16T15:02:28.682909+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683833+0000","last_change":"2024-01-16T15:02:28.683833+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683833+0000","last_undegraded":"2024-01-16T15:02:28.683833+0000","last_fullsized":"2024-01-16T15:02:28.683833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675958+0000","last_change":"2024-01-16T15:02:28.675958+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675958+0000","last_undegraded":"2024-01-16T15:02:28.675958+0000","last_fullsized":"2024-01-16T15:02:28.675958+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673472+0000","last_change":"2024-01-16T15:02:28.673472+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673472+0000","last_undegraded":"2024-01-16T15:02:28.673472+0000","last_fullsized":"2024-01-16T15:02:28.673472+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700420+0000","last_change":"2024-01-16T15:02:28.696840+0000","last_active":"2024-01-16T15:02:28.700420+0000","last_peered":"2024-01-16T15:02:28.700420+0000","last_clean":"2024-01-16T15:02:28.700420+0000","last_became_active":"2024-01-16T15:02:28.695989+0000","last_became_peered":"2024-01-16T15:02:28.695989+0000","last_unstale":"2024-01-16T15:02:28.700420+0000","last_undegraded":"2024-01-16T15:02:28.700420+0000","last_fullsized":"2024-01-16T15:02:28.700420+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:36.619091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.741340+0000","last_change":"2024-01-16T15:02:29.722881+0000","last_active":"2024-01-16T15:02:29.741340+0000","last_peered":"2024-01-16T15:02:29.741340+0000","last_clean":"2024-01-16T15:02:29.741340+0000","last_became_active":"2024-01-16T15:02:29.722447+0000","last_became_peered":"2024-01-16T15:02:29.722447+0000","last_unstale":"2024-01-16T15:02:29.741340+0000","last_undegraded":"2024-01-16T15:02:29.741340+0000","last_fullsized":"2024-01-16T15:02:29.741340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:52.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700061+0000","last_change":"2024-01-16T15:02:28.697447+0000","last_active":"2024-01-16T15:02:28.700061+0000","last_peered":"2024-01-16T15:02:28.700061+0000","last_clean":"2024-01-16T15:02:28.700061+0000","last_became_active":"2024-01-16T15:02:28.697122+0000","last_became_peered":"2024-01-16T15:02:28.697122+0000","last_unstale":"2024-01-16T15:02:28.700061+0000","last_undegraded":"2024-01-16T15:02:28.700061+0000","last_fullsized":"2024-01-16T15:02:28.700061+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:09.273766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672137+0000","last_change":"2024-01-16T15:02:28.672137+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672137+0000","last_undegraded":"2024-01-16T15:02:28.672137+0000","last_fullsized":"2024-01-16T15:02:28.672137+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696624+0000","last_change":"2024-01-16T15:02:28.689593+0000","last_active":"2024-01-16T15:02:28.696624+0000","last_peered":"2024-01-16T15:02:28.696624+0000","last_clean":"2024-01-16T15:02:28.696624+0000","last_became_active":"2024-01-16T15:02:28.689343+0000","last_became_peered":"2024-01-16T15:02:28.689343+0000","last_unstale":"2024-01-16T15:02:28.696624+0000","last_undegraded":"2024-01-16T15:02:28.696624+0000","last_fullsized":"2024-01-16T15:02:28.696624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:51.159242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673332+0000","last_change":"2024-01-16T15:02:28.673332+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673332+0000","last_undegraded":"2024-01-16T15:02:28.673332+0000","last_fullsized":"2024-01-16T15:02:28.673332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698069+0000","last_change":"2024-01-16T15:02:28.694539+0000","last_active":"2024-01-16T15:02:28.698069+0000","last_peered":"2024-01-16T15:02:28.698069+0000","last_clean":"2024-01-16T15:02:28.698069+0000","last_became_active":"2024-01-16T15:02:28.694233+0000","last_became_peered":"2024-01-16T15:02:28.694233+0000","last_unstale":"2024-01-16T15:02:28.698069+0000","last_undegraded":"2024-01-16T15:02:28.698069+0000","last_fullsized":"2024-01-16T15:02:28.698069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675366+0000","last_change":"2024-01-16T15:02:28.675366+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675366+0000","last_undegraded":"2024-01-16T15:02:28.675366+0000","last_fullsized":"2024-01-16T15:02:28.675366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696756+0000","last_change":"2024-01-16T15:02:28.693560+0000","last_active":"2024-01-16T15:02:28.696756+0000","last_peered":"2024-01-16T15:02:28.696756+0000","last_clean":"2024-01-16T15:02:28.696756+0000","last_became_active":"2024-01-16T15:02:28.693211+0000","last_became_peered":"2024-01-16T15:02:28.693211+0000","last_unstale":"2024-01-16T15:02:28.696756+0000","last_undegraded":"2024-01-16T15:02:28.696756+0000","last_fullsized":"2024-01-16T15:02:28.696756+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676930+0000","last_change":"2024-01-16T15:02:28.676930+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676930+0000","last_undegraded":"2024-01-16T15:02:28.676930+0000","last_fullsized":"2024-01-16T15:02:28.676930+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.691282+0000","last_change":"2024-01-16T15:02:28.683014+0000","last_active":"2024-01-16T15:02:28.691282+0000","last_peered":"2024-01-16T15:02:28.691282+0000","last_clean":"2024-01-16T15:02:28.691282+0000","last_became_active":"2024-01-16T15:02:28.682735+0000","last_became_peered":"2024-01-16T15:02:28.682735+0000","last_unstale":"2024-01-16T15:02:28.691282+0000","last_undegraded":"2024-01-16T15:02:28.691282+0000","last_fullsized":"2024-01-16T15:02:28.691282+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:55.974417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726389+0000","last_change":"2024-01-16T15:02:29.721954+0000","last_active":"2024-01-16T15:02:29.726389+0000","last_peered":"2024-01-16T15:02:29.726389+0000","last_clean":"2024-01-16T15:02:29.726389+0000","last_became_active":"2024-01-16T15:02:29.721647+0000","last_became_peered":"2024-01-16T15:02:29.721647+0000","last_unstale":"2024-01-16T15:02:29.726389+0000","last_undegraded":"2024-01-16T15:02:29.726389+0000","last_fullsized":"2024-01-16T15:02:29.726389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:16.064389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697872+0000","last_change":"2024-01-16T15:02:28.694096+0000","last_active":"2024-01-16T15:02:28.697872+0000","last_peered":"2024-01-16T15:02:28.697872+0000","last_clean":"2024-01-16T15:02:28.697872+0000","last_became_active":"2024-01-16T15:02:28.693789+0000","last_became_peered":"2024-01-16T15:02:28.693789+0000","last_unstale":"2024-01-16T15:02:28.697872+0000","last_undegraded":"2024-01-16T15:02:28.697872+0000","last_fullsized":"2024-01-16T15:02:28.697872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681431+0000","last_change":"2024-01-16T15:02:28.681431+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681431+0000","last_undegraded":"2024-01-16T15:02:28.681431+0000","last_fullsized":"2024-01-16T15:02:28.681431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696269+0000","last_change":"2024-01-16T15:02:28.693108+0000","last_active":"2024-01-16T15:02:28.696269+0000","last_peered":"2024-01-16T15:02:28.696269+0000","last_clean":"2024-01-16T15:02:28.696269+0000","last_became_active":"2024-01-16T15:02:28.691074+0000","last_became_peered":"2024-01-16T15:02:28.691074+0000","last_unstale":"2024-01-16T15:02:28.696269+0000","last_undegraded":"2024-01-16T15:02:28.696269+0000","last_fullsized":"2024-01-16T15:02:28.696269+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672957+0000","last_change":"2024-01-16T15:02:28.672957+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672957+0000","last_undegraded":"2024-01-16T15:02:28.672957+0000","last_fullsized":"2024-01-16T15:02:28.672957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702078+0000","last_change":"2024-01-16T15:02:28.700517+0000","last_active":"2024-01-16T15:02:28.702078+0000","last_peered":"2024-01-16T15:02:28.702078+0000","last_clean":"2024-01-16T15:02:28.702078+0000","last_became_active":"2024-01-16T15:02:28.700187+0000","last_became_peered":"2024-01-16T15:02:28.700187+0000","last_unstale":"2024-01-16T15:02:28.702078+0000","last_undegraded":"2024-01-16T15:02:28.702078+0000","last_fullsized":"2024-01-16T15:02:28.702078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.686150+0000","last_change":"2024-01-16T15:02:28.686150+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.686150+0000","last_undegraded":"2024-01-16T15:02:28.686150+0000","last_fullsized":"2024-01-16T15:02:28.686150+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699635+0000","last_change":"2024-01-16T15:02:28.694170+0000","last_active":"2024-01-16T15:02:28.699635+0000","last_peered":"2024-01-16T15:02:28.699635+0000","last_clean":"2024-01-16T15:02:28.699635+0000","last_became_active":"2024-01-16T15:02:28.693951+0000","last_became_peered":"2024-01-16T15:02:28.693951+0000","last_unstale":"2024-01-16T15:02:28.699635+0000","last_undegraded":"2024-01-16T15:02:28.699635+0000","last_fullsized":"2024-01-16T15:02:28.699635+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:05.345755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740967+0000","last_change":"2024-01-16T15:02:29.728569+0000","last_active":"2024-01-16T15:02:29.740967+0000","last_peered":"2024-01-16T15:02:29.740967+0000","last_clean":"2024-01-16T15:02:29.740967+0000","last_became_active":"2024-01-16T15:02:29.728240+0000","last_became_peered":"2024-01-16T15:02:29.728240+0000","last_unstale":"2024-01-16T15:02:29.740967+0000","last_undegraded":"2024-01-16T15:02:29.740967+0000","last_fullsized":"2024-01-16T15:02:29.740967+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:49.783058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702091+0000","last_change":"2024-01-16T15:02:28.700598+0000","last_active":"2024-01-16T15:02:28.702091+0000","last_peered":"2024-01-16T15:02:28.702091+0000","last_clean":"2024-01-16T15:02:28.702091+0000","last_became_active":"2024-01-16T15:02:28.700296+0000","last_became_peered":"2024-01-16T15:02:28.700296+0000","last_unstale":"2024-01-16T15:02:28.702091+0000","last_undegraded":"2024-01-16T15:02:28.702091+0000","last_fullsized":"2024-01-16T15:02:28.702091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684658+0000","last_change":"2024-01-16T15:02:28.684658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684658+0000","last_undegraded":"2024-01-16T15:02:28.684658+0000","last_fullsized":"2024-01-16T15:02:28.684658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701099+0000","last_change":"2024-01-16T15:02:28.697153+0000","last_active":"2024-01-16T15:02:28.701099+0000","last_peered":"2024-01-16T15:02:28.701099+0000","last_clean":"2024-01-16T15:02:28.701099+0000","last_became_active":"2024-01-16T15:02:28.693885+0000","last_became_peered":"2024-01-16T15:02:28.693885+0000","last_unstale":"2024-01-16T15:02:28.701099+0000","last_undegraded":"2024-01-16T15:02:28.701099+0000","last_fullsized":"2024-01-16T15:02:28.701099+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683793+0000","last_change":"2024-01-16T15:02:28.683793+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683793+0000","last_undegraded":"2024-01-16T15:02:28.683793+0000","last_fullsized":"2024-01-16T15:02:28.683793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699827+0000","last_change":"2024-01-16T15:02:28.695099+0000","last_active":"2024-01-16T15:02:28.699827+0000","last_peered":"2024-01-16T15:02:28.699827+0000","last_clean":"2024-01-16T15:02:28.699827+0000","last_became_active":"2024-01-16T15:02:28.694828+0000","last_became_peered":"2024-01-16T15:02:28.694828+0000","last_unstale":"2024-01-16T15:02:28.699827+0000","last_undegraded":"2024-01-16T15:02:28.699827+0000","last_fullsized":"2024-01-16T15:02:28.699827+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:29.254715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680264+0000","last_change":"2024-01-16T15:02:28.680264+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680264+0000","last_undegraded":"2024-01-16T15:02:28.680264+0000","last_fullsized":"2024-01-16T15:02:28.680264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706804+0000","last_change":"2024-01-16T15:02:28.688577+0000","last_active":"2024-01-16T15:02:29.706804+0000","last_peered":"2024-01-16T15:02:29.706804+0000","last_clean":"2024-01-16T15:02:29.706804+0000","last_became_active":"2024-01-16T15:02:28.687999+0000","last_became_peered":"2024-01-16T15:02:28.687999+0000","last_unstale":"2024-01-16T15:02:29.706804+0000","last_undegraded":"2024-01-16T15:02:29.706804+0000","last_fullsized":"2024-01-16T15:02:29.706804+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:17.380916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673658+0000","last_change":"2024-01-16T15:02:28.673658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673658+0000","last_undegraded":"2024-01-16T15:02:28.673658+0000","last_fullsized":"2024-01-16T15:02:28.673658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697893+0000","last_change":"2024-01-16T15:02:28.694910+0000","last_active":"2024-01-16T15:02:28.697893+0000","last_peered":"2024-01-16T15:02:28.697893+0000","last_clean":"2024-01-16T15:02:28.697893+0000","last_became_active":"2024-01-16T15:02:28.690704+0000","last_became_peered":"2024-01-16T15:02:28.690704+0000","last_unstale":"2024-01-16T15:02:28.697893+0000","last_undegraded":"2024-01-16T15:02:28.697893+0000","last_fullsized":"2024-01-16T15:02:28.697893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678221+0000","last_change":"2024-01-16T15:02:28.678221+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678221+0000","last_undegraded":"2024-01-16T15:02:28.678221+0000","last_fullsized":"2024-01-16T15:02:28.678221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701998+0000","last_change":"2024-01-16T15:02:28.699544+0000","last_active":"2024-01-16T15:02:28.701998+0000","last_peered":"2024-01-16T15:02:28.701998+0000","last_clean":"2024-01-16T15:02:28.701998+0000","last_became_active":"2024-01-16T15:02:28.699169+0000","last_became_peered":"2024-01-16T15:02:28.699169+0000","last_unstale":"2024-01-16T15:02:28.701998+0000","last_undegraded":"2024-01-16T15:02:28.701998+0000","last_fullsized":"2024-01-16T15:02:28.701998+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687323+0000","last_change":"2024-01-16T15:02:28.687323+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687323+0000","last_undegraded":"2024-01-16T15:02:28.687323+0000","last_fullsized":"2024-01-16T15:02:28.687323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699992+0000","last_change":"2024-01-16T15:02:28.695124+0000","last_active":"2024-01-16T15:02:28.699992+0000","last_peered":"2024-01-16T15:02:28.699992+0000","last_clean":"2024-01-16T15:02:28.699992+0000","last_became_active":"2024-01-16T15:02:28.694847+0000","last_became_peered":"2024-01-16T15:02:28.694847+0000","last_unstale":"2024-01-16T15:02:28.699992+0000","last_undegraded":"2024-01-16T15:02:28.699992+0000","last_fullsized":"2024-01-16T15:02:28.699992+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:56.293921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684228+0000","last_change":"2024-01-16T15:02:28.684228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684228+0000","last_undegraded":"2024-01-16T15:02:28.684228+0000","last_fullsized":"2024-01-16T15:02:28.684228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700477+0000","last_change":"2024-01-16T15:02:28.698762+0000","last_active":"2024-01-16T15:02:28.700477+0000","last_peered":"2024-01-16T15:02:28.700477+0000","last_clean":"2024-01-16T15:02:28.700477+0000","last_became_active":"2024-01-16T15:02:28.698337+0000","last_became_peered":"2024-01-16T15:02:28.698337+0000","last_unstale":"2024-01-16T15:02:28.700477+0000","last_undegraded":"2024-01-16T15:02:28.700477+0000","last_fullsized":"2024-01-16T15:02:28.700477+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:40.971035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676992+0000","last_change":"2024-01-16T15:02:28.676992+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676992+0000","last_undegraded":"2024-01-16T15:02:28.676992+0000","last_fullsized":"2024-01-16T15:02:28.676992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698294+0000","last_change":"2024-01-16T15:02:28.695728+0000","last_active":"2024-01-16T15:02:28.698294+0000","last_peered":"2024-01-16T15:02:28.698294+0000","last_clean":"2024-01-16T15:02:28.698294+0000","last_became_active":"2024-01-16T15:02:28.695438+0000","last_became_peered":"2024-01-16T15:02:28.695438+0000","last_unstale":"2024-01-16T15:02:28.698294+0000","last_undegraded":"2024-01-16T15:02:28.698294+0000","last_fullsized":"2024-01-16T15:02:28.698294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679870+0000","last_change":"2024-01-16T15:02:28.679870+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679870+0000","last_undegraded":"2024-01-16T15:02:28.679870+0000","last_fullsized":"2024-01-16T15:02:28.679870+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696703+0000","last_change":"2024-01-16T15:02:28.690359+0000","last_active":"2024-01-16T15:02:28.696703+0000","last_peered":"2024-01-16T15:02:28.696703+0000","last_clean":"2024-01-16T15:02:28.696703+0000","last_became_active":"2024-01-16T15:02:28.689968+0000","last_became_peered":"2024-01-16T15:02:28.689968+0000","last_unstale":"2024-01-16T15:02:28.696703+0000","last_undegraded":"2024-01-16T15:02:28.696703+0000","last_fullsized":"2024-01-16T15:02:28.696703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:06.195041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683284+0000","last_change":"2024-01-16T15:02:28.683284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683284+0000","last_undegraded":"2024-01-16T15:02:28.683284+0000","last_fullsized":"2024-01-16T15:02:28.683284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689306+0000","last_change":"2024-01-16T15:02:28.687104+0000","last_active":"2024-01-16T15:02:28.689306+0000","last_peered":"2024-01-16T15:02:28.689306+0000","last_clean":"2024-01-16T15:02:28.689306+0000","last_became_active":"2024-01-16T15:02:28.686179+0000","last_became_peered":"2024-01-16T15:02:28.686179+0000","last_unstale":"2024-01-16T15:02:28.689306+0000","last_undegraded":"2024-01-16T15:02:28.689306+0000","last_fullsized":"2024-01-16T15:02:28.689306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:39.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672340+0000","last_change":"2024-01-16T15:02:28.672340+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672340+0000","last_undegraded":"2024-01-16T15:02:28.672340+0000","last_fullsized":"2024-01-16T15:02:28.672340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.722558+0000","last_change":"2024-01-16T15:02:28.693574+0000","last_active":"2024-01-16T15:02:29.722558+0000","last_peered":"2024-01-16T15:02:29.722558+0000","last_clean":"2024-01-16T15:02:29.722558+0000","last_became_active":"2024-01-16T15:02:28.692884+0000","last_became_peered":"2024-01-16T15:02:28.692884+0000","last_unstale":"2024-01-16T15:02:29.722558+0000","last_undegraded":"2024-01-16T15:02:29.722558+0000","last_fullsized":"2024-01-16T15:02:29.722558+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672000+0000","last_change":"2024-01-16T15:02:28.672000+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672000+0000","last_undegraded":"2024-01-16T15:02:28.672000+0000","last_fullsized":"2024-01-16T15:02:28.672000+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696515+0000","last_change":"2024-01-16T15:02:28.692170+0000","last_active":"2024-01-16T15:02:28.696515+0000","last_peered":"2024-01-16T15:02:28.696515+0000","last_clean":"2024-01-16T15:02:28.696515+0000","last_became_active":"2024-01-16T15:02:28.691779+0000","last_became_peered":"2024-01-16T15:02:28.691779+0000","last_unstale":"2024-01-16T15:02:28.696515+0000","last_undegraded":"2024-01-16T15:02:28.696515+0000","last_fullsized":"2024-01-16T15:02:28.696515+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681494+0000","last_change":"2024-01-16T15:02:28.681494+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681494+0000","last_undegraded":"2024-01-16T15:02:28.681494+0000","last_fullsized":"2024-01-16T15:02:28.681494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.737753+0000","last_change":"2024-01-16T15:02:28.689348+0000","last_active":"2024-01-16T15:02:29.737753+0000","last_peered":"2024-01-16T15:02:29.737753+0000","last_clean":"2024-01-16T15:02:29.737753+0000","last_became_active":"2024-01-16T15:02:28.689030+0000","last_became_peered":"2024-01-16T15:02:28.689030+0000","last_unstale":"2024-01-16T15:02:29.737753+0000","last_undegraded":"2024-01-16T15:02:29.737753+0000","last_fullsized":"2024-01-16T15:02:29.737753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:24.816323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683770+0000","last_change":"2024-01-16T15:02:28.683770+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683770+0000","last_undegraded":"2024-01-16T15:02:28.683770+0000","last_fullsized":"2024-01-16T15:02:28.683770+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689249+0000","last_change":"2024-01-16T15:02:28.681603+0000","last_active":"2024-01-16T15:02:28.689249+0000","last_peered":"2024-01-16T15:02:28.689249+0000","last_clean":"2024-01-16T15:02:28.689249+0000","last_became_active":"2024-01-16T15:02:28.681322+0000","last_became_peered":"2024-01-16T15:02:28.681322+0000","last_unstale":"2024-01-16T15:02:28.689249+0000","last_undegraded":"2024-01-16T15:02:28.689249+0000","last_fullsized":"2024-01-16T15:02:28.689249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:13.938868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740947+0000","last_change":"2024-01-16T15:02:29.728538+0000","last_active":"2024-01-16T15:02:29.740947+0000","last_peered":"2024-01-16T15:02:29.740947+0000","last_clean":"2024-01-16T15:02:29.740947+0000","last_became_active":"2024-01-16T15:02:29.728211+0000","last_became_peered":"2024-01-16T15:02:29.728211+0000","last_unstale":"2024-01-16T15:02:29.740947+0000","last_undegraded":"2024-01-16T15:02:29.740947+0000","last_fullsized":"2024-01-16T15:02:29.740947+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:32.872029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706859+0000","last_change":"2024-01-16T15:02:28.681634+0000","last_active":"2024-01-16T15:02:29.706859+0000","last_peered":"2024-01-16T15:02:29.706859+0000","last_clean":"2024-01-16T15:02:29.706859+0000","last_became_active":"2024-01-16T15:02:28.680098+0000","last_became_peered":"2024-01-16T15:02:28.680098+0000","last_unstale":"2024-01-16T15:02:29.706859+0000","last_undegraded":"2024-01-16T15:02:29.706859+0000","last_fullsized":"2024-01-16T15:02:29.706859+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:45.583344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671790+0000","last_change":"2024-01-16T15:02:28.671790+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671790+0000","last_undegraded":"2024-01-16T15:02:28.671790+0000","last_fullsized":"2024-01-16T15:02:28.671790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697195+0000","last_change":"2024-01-16T15:02:28.692899+0000","last_active":"2024-01-16T15:02:28.697195+0000","last_peered":"2024-01-16T15:02:28.697195+0000","last_clean":"2024-01-16T15:02:28.697195+0000","last_became_active":"2024-01-16T15:02:28.692607+0000","last_became_peered":"2024-01-16T15:02:28.692607+0000","last_unstale":"2024-01-16T15:02:28.697195+0000","last_undegraded":"2024-01-16T15:02:28.697195+0000","last_fullsized":"2024-01-16T15:02:28.697195+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674849+0000","last_change":"2024-01-16T15:02:28.674849+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674849+0000","last_undegraded":"2024-01-16T15:02:28.674849+0000","last_fullsized":"2024-01-16T15:02:28.674849+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.728464+0000","last_change":"2024-01-16T15:02:28.695802+0000","last_active":"2024-01-16T15:02:29.728464+0000","last_peered":"2024-01-16T15:02:29.728464+0000","last_clean":"2024-01-16T15:02:29.728464+0000","last_became_active":"2024-01-16T15:02:28.695516+0000","last_became_peered":"2024-01-16T15:02:28.695516+0000","last_unstale":"2024-01-16T15:02:29.728464+0000","last_undegraded":"2024-01-16T15:02:29.728464+0000","last_fullsized":"2024-01-16T15:02:29.728464+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:00.682347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679260+0000","last_change":"2024-01-16T15:02:28.679260+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679260+0000","last_undegraded":"2024-01-16T15:02:28.679260+0000","last_fullsized":"2024-01-16T15:02:28.679260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696398+0000","last_change":"2024-01-16T15:02:28.691759+0000","last_active":"2024-01-16T15:02:28.696398+0000","last_peered":"2024-01-16T15:02:28.696398+0000","last_clean":"2024-01-16T15:02:28.696398+0000","last_became_active":"2024-01-16T15:02:28.691457+0000","last_became_peered":"2024-01-16T15:02:28.691457+0000","last_unstale":"2024-01-16T15:02:28.696398+0000","last_undegraded":"2024-01-16T15:02:28.696398+0000","last_fullsized":"2024-01-16T15:02:28.696398+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:31.867536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.738586+0000","last_change":"2024-01-16T15:02:29.719394+0000","last_active":"2024-01-16T15:02:29.738586+0000","last_peered":"2024-01-16T15:02:29.738586+0000","last_clean":"2024-01-16T15:02:29.738586+0000","last_became_active":"2024-01-16T15:02:29.719030+0000","last_became_peered":"2024-01-16T15:02:29.719030+0000","last_unstale":"2024-01-16T15:02:29.738586+0000","last_undegraded":"2024-01-16T15:02:29.738586+0000","last_fullsized":"2024-01-16T15:02:29.738586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:51.117200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.717109+0000","last_change":"2024-01-16T15:02:28.692453+0000","last_active":"2024-01-16T15:02:29.717109+0000","last_peered":"2024-01-16T15:02:29.717109+0000","last_clean":"2024-01-16T15:02:29.717109+0000","last_became_active":"2024-01-16T15:02:28.692132+0000","last_became_peered":"2024-01-16T15:02:28.692132+0000","last_unstale":"2024-01-16T15:02:29.717109+0000","last_undegraded":"2024-01-16T15:02:29.717109+0000","last_fullsized":"2024-01-16T15:02:29.717109+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674276+0000","last_change":"2024-01-16T15:02:28.674276+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674276+0000","last_undegraded":"2024-01-16T15:02:28.674276+0000","last_fullsized":"2024-01-16T15:02:28.674276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700041+0000","last_change":"2024-01-16T15:02:28.696929+0000","last_active":"2024-01-16T15:02:28.700041+0000","last_peered":"2024-01-16T15:02:28.700041+0000","last_clean":"2024-01-16T15:02:28.700041+0000","last_became_active":"2024-01-16T15:02:28.696616+0000","last_became_peered":"2024-01-16T15:02:28.696616+0000","last_unstale":"2024-01-16T15:02:28.700041+0000","last_undegraded":"2024-01-16T15:02:28.700041+0000","last_fullsized":"2024-01-16T15:02:28.700041+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:29.321021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675396+0000","last_change":"2024-01-16T15:02:28.675396+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675396+0000","last_undegraded":"2024-01-16T15:02:28.675396+0000","last_fullsized":"2024-01-16T15:02:28.675396+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.708768+0000","last_change":"2024-01-16T15:02:28.695669+0000","last_active":"2024-01-16T15:02:29.708768+0000","last_peered":"2024-01-16T15:02:29.708768+0000","last_clean":"2024-01-16T15:02:29.708768+0000","last_became_active":"2024-01-16T15:02:28.695350+0000","last_became_peered":"2024-01-16T15:02:28.695350+0000","last_unstale":"2024-01-16T15:02:29.708768+0000","last_undegraded":"2024-01-16T15:02:29.708768+0000","last_fullsized":"2024-01-16T15:02:29.708768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:36.266476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":199,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-16T15:02:27.660341+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:02:27.660341+0000","last_peered":"2024-01-16T15:02:27.660341+0000","last_clean":"2024-01-16T15:02:27.660341+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:02:27.660341+0000","last_undegraded":"2024-01-16T15:02:27.660341+0000","last_fullsized":"2024-01-16T15:02:27.660341+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.0329999999999999,"5min":1.3480000000000001,"15min":1.1539999999999999},"min":{"1min":0.73899999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3089999999999999,"15min":1.2010000000000001},"min":{"1min":0.76700000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9319999999999999,"5min":2.774,"15min":2.774},"last":1.1080000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.2909999999999999,"15min":1.111},"min":{"1min":0.76400000000000001,"5min":0.63,"15min":0.63},"max":{"1min":11.487,"5min":11.487,"15min":11.487},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3380000000000001,"15min":1.1890000000000001},"min":{"1min":0.89000000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.8109999999999999,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":0.82199999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.288,"15min":1.1970000000000001},"min":{"1min":0.85599999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.7269999999999999,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.28},{"interface":"front","average":{"1min":2.8940000000000001,"5min":1.6160000000000001,"15min":1.349},"min":{"1min":0.84199999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":20.577000000000002,"5min":20.577000000000002,"15min":20.577000000000002},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2639999999999998,"5min":1.6259999999999999,"15min":1.3979999999999999},"min":{"1min":1.0569999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":11.872,"5min":12.861000000000001,"15min":12.861000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.486,"15min":1.4099999999999999},"min":{"1min":0.81100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.1379999999999999,"5min":4.0720000000000001,"15min":4.0720000000000001},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.8439999999999999,"5min":1.631,"15min":1.446},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":20.649999999999999,"5min":20.649999999999999,"15min":20.649999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.387,"5min":1.633,"15min":1.5109999999999999},"min":{"1min":0.94399999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":11.702,"5min":11.702,"15min":11.702},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.514,"15min":1.526},"min":{"1min":0.70499999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.8039999999999998,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.73,"15min":1.6719999999999999},"min":{"1min":1.0149999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":11.225,"5min":11.225,"15min":11.225},"last":1.0649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":4.1589999999999998,"5min":2.0630000000000002,"15min":1.7430000000000001},"min":{"1min":1.306,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":41.206000000000003,"5min":41.206000000000003,"15min":41.206000000000003},"last":1.155},{"interface":"front","average":{"1min":4.125,"5min":2.0449999999999999,"15min":1.7589999999999999},"min":{"1min":1.0940000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":21.908000000000001,"5min":21.908000000000001,"15min":21.908000000000001},"last":1.3280000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.2609999999999999,"15min":1.147},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":9.1890000000000001,"5min":9.1890000000000001,"15min":9.1890000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":5.2039999999999997,"5min":1.976,"15min":1.478},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":50.472000000000001,"5min":50.472000000000001,"15min":50.472000000000001},"last":0.80600000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":1.3979999999999999,"15min":1.1459999999999999},"min":{"1min":0.66300000000000003,"5min":0.624,"15min":0.624},"max":{"1min":10.629,"5min":10.629,"15min":10.629},"last":0.98399999999999999},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.5509999999999999,"15min":1.421},"min":{"1min":0.82699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":9.3279999999999994,"5min":9.3279999999999994,"15min":9.3279999999999994},"last":1.0569999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.3789999999999996,"5min":1.9450000000000001,"15min":1.4199999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":50.643999999999998,"5min":50.643999999999998,"15min":50.643999999999998},"last":1.385},{"interface":"front","average":{"1min":2.4169999999999998,"5min":1.4970000000000001,"15min":1.359},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":10.843,"5min":10.843,"15min":10.843},"last":1.607}]},{"osd":3,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":1.6100000000000001,"15min":1.4359999999999999},"min":{"1min":0.82399999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":15.675000000000001,"5min":15.675000000000001,"15min":15.675000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":4.3760000000000003,"5min":1.99,"15min":1.5800000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":50.561,"5min":50.561,"15min":50.561},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.8289999999999997,"5min":2.1840000000000002,"15min":1.728},"min":{"1min":0.92700000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":47.488,"5min":47.488,"15min":47.488},"last":1.542},{"interface":"front","average":{"1min":5.2519999999999998,"5min":2.2330000000000001,"15min":1.7609999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":50.381,"5min":50.381,"15min":50.381},"last":1.4850000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.5419999999999998,"5min":1.6659999999999999,"15min":1.5409999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":10.654,"5min":10.654,"15min":10.654},"last":1.7450000000000001},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.8280000000000001,"15min":1.726},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":9.3729999999999993,"5min":9.3729999999999993,"15min":9.3729999999999993},"last":1.7789999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.9820000000000002,"5min":2.2799999999999998,"15min":1.8779999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":52.912999999999997,"5min":52.912999999999997,"15min":52.912999999999997},"last":1.581},{"interface":"front","average":{"1min":2.5270000000000001,"5min":1.7769999999999999,"15min":1.7130000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":10.581,"5min":10.581,"15min":10.581},"last":1.8600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3109999999999999,"5min":1.3580000000000001,"15min":1.248},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":13.938000000000001,"5min":13.938000000000001,"15min":13.938000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":2.9500000000000002,"5min":1.478,"15min":1.212},"min":{"1min":0.73199999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":16.896000000000001,"5min":16.896000000000001,"15min":16.896000000000001},"last":1.591}]},{"osd":1,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.482,"15min":1.353},"min":{"1min":0.77500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":3.4550000000000001,"5min":1.7549999999999999,"15min":1.345},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":35.328000000000003,"5min":35.328000000000003,"15min":35.328000000000003},"last":1.554}]},{"osd":2,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.7639999999999998,"5min":1.8859999999999999,"15min":1.47},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":35.398000000000003,"5min":35.398000000000003,"15min":35.398000000000003},"last":1.091},{"interface":"front","average":{"1min":1.782,"5min":1.3340000000000001,"15min":1.29},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.883,"5min":4.883,"15min":4.883},"last":1.7150000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8210000000000002,"5min":1.667,"15min":1.4430000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":13.202,"5min":13.202,"15min":13.202},"last":1.4810000000000001},{"interface":"front","average":{"1min":3.6669999999999998,"5min":1.9670000000000001,"15min":1.587},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":35.268000000000001,"5min":35.268000000000001,"15min":35.268000000000001},"last":1.6799999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.5470000000000002,"5min":1.784,"15min":1.579},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":15.757,"5min":15.757,"15min":15.757},"last":1.0069999999999999},{"interface":"front","average":{"1min":4.3479999999999999,"5min":2.004,"15min":1.714},"min":{"1min":0.63300000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":26.166,"5min":26.166,"15min":26.166},"last":1.7709999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":4.0350000000000001,"5min":2.0950000000000002,"15min":1.768},"min":{"1min":0.53400000000000003,"5min":0.499,"15min":0.499},"max":{"1min":35.427999999999997,"5min":35.427999999999997,"15min":35.427999999999997},"last":1.7410000000000001},{"interface":"front","average":{"1min":3.4740000000000002,"5min":1.903,"15min":1.7589999999999999},"min":{"1min":0.53100000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":1.6259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.9350000000000001,"5min":2.0379999999999998,"15min":1.7689999999999999},"min":{"1min":0.58099999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":19.364999999999998,"5min":19.364999999999998,"15min":19.364999999999998},"last":1.837},{"interface":"front","average":{"1min":4.0419999999999998,"5min":2.1829999999999998,"15min":1.8839999999999999},"min":{"1min":0.71699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":35.363,"5min":35.363,"15min":35.363},"last":1.994}]}]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.1779999999999999,"15min":1.1080000000000001},"min":{"1min":0.751,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":4.2869999999999999,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":1.012},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.4059999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":8.0440000000000005,"5min":8.0440000000000005,"15min":8.0440000000000005},"last":0.90300000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.4359999999999999,"15min":1.258},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.1050000000000004,"5min":7.1050000000000004,"15min":7.1050000000000004},"last":1.391},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.2889999999999999,"15min":1.234},"min":{"1min":0.84499999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":5.7290000000000001,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":1.137}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.583,"15min":1.415},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.4860000000000002,"5min":15.986000000000001,"15min":15.986000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.554,"5min":1.3520000000000001,"15min":1.4330000000000001},"min":{"1min":0.83699999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":3.762,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":1.232}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":3.0049999999999999,"5min":1.871,"15min":1.6819999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5600000000000001,"15min":1.48},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":4.0490000000000004,"5min":21.706,"15min":21.706},"last":1.538}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4370000000000001,"15min":1.575},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":4.327,"5min":4.327,"15min":4.327},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.649,"15min":1.6279999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":10.831,"5min":10.831,"15min":10.831},"last":1.504}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4530000000000001,"15min":1.575},"min":{"1min":0.66000000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":1.476},{"interface":"front","average":{"1min":2.468,"5min":1.734,"15min":1.732},"min":{"1min":0.74199999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":20.939,"5min":20.939,"15min":20.939},"last":1.4399999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.8,"15min":1.7490000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":7.0110000000000001,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":1.591},{"interface":"front","average":{"1min":1.976,"5min":1.6399999999999999,"15min":1.738},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":10.978999999999999,"5min":10.978999999999999,"15min":10.978999999999999},"last":1.6279999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.98499999999999999,"15min":0.91900000000000004},"min":{"1min":0.433,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1459999999999999,"15min":0.93600000000000005},"min":{"1min":0.56799999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":1.099}]},{"osd":1,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.92500000000000004,"15min":0.85999999999999999},"min":{"1min":0.47099999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.752,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.121,"5min":1.0620000000000001,"15min":1.016},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.4590000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.075,"15min":1.075},"min":{"1min":0.59299999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9339999999999999,"5min":4.008,"15min":4.008},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.0820000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.2069999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.2929999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.3500000000000001,"15min":1.1639999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":7.7439999999999998,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.492,"15min":1.3129999999999999},"min":{"1min":0.80500000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":11.068,"5min":11.068,"15min":11.068},"last":1.208}]},{"osd":5,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4630000000000001,"15min":1.3180000000000001},"min":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":12.791,"5min":12.791,"15min":12.791},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.508,"15min":1.413},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":12.106,"5min":12.106,"15min":12.106},"last":1.5389999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":1.546,"15min":1.3540000000000001},"min":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":8.1039999999999992,"5min":8.1039999999999992,"15min":8.1039999999999992},"last":1.502},{"interface":"front","average":{"1min":2.3980000000000001,"5min":1.6359999999999999,"15min":1.4379999999999999},"min":{"1min":0.91900000000000004,"5min":0.752,"15min":0.752},"max":{"1min":6.891,"5min":6.891,"15min":6.891},"last":1.4890000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.532,"15min":1.4379999999999999},"min":{"1min":0.82599999999999996,"5min":0.755,"15min":0.755},"max":{"1min":3.7269999999999999,"5min":3.7269999999999999,"15min":3.7269999999999999},"last":1.599},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.677,"15min":1.4299999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":7.6159999999999997,"5min":18.846,"15min":18.846},"last":1.5640000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":0.91600000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.839,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.378},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0049999999999999,"15min":0.93600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.353,"5min":4.5010000000000003,"15min":4.5010000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0609999999999999,"15min":1.0329999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.133,"5min":3.798,"15min":3.798},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.125,"15min":1.014},"min":{"1min":0.45800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.8069999999999999,"5min":4.8250000000000002,"15min":4.8250000000000002},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0620000000000001,"15min":0.97999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.9870000000000001,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.224,"15min":1.1399999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":8.4610000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.5489999999999999,"15min":1.3939999999999999},"min":{"1min":0.67700000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":11.843,"5min":11.843,"15min":11.843},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.847,"5min":1.54,"15min":1.411},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":11.768000000000001,"5min":11.768000000000001,"15min":11.768000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":1.72,"15min":1.4470000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":26.501000000000001,"5min":26.501000000000001,"15min":26.501000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.389,"15min":1.341},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.8640000000000001,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.593,"15min":1.4430000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.5059999999999998,"5min":15.144,"15min":15.144},"last":0.56799999999999995},{"interface":"front","average":{"1min":5.2089999999999996,"5min":2.214,"15min":1.6599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":66.061999999999998,"5min":66.061999999999998,"15min":66.061999999999998},"last":0.63300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.581,"15min":1.5049999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":4.0750000000000002,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.554,"5min":1.5509999999999999,"15min":1.4850000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":4.0519999999999996,"5min":6.0590000000000002,"15min":6.0590000000000002},"last":0.5}]}]},{"osd":1,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.92800000000000005,"15min":0.93000000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.089,"5min":1.026,"15min":1.0149999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":1.298}]},{"osd":2,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.0369999999999999,"15min":0.95499999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":14.817,"5min":14.817,"15min":14.817},"last":0.996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.028,"15min":0.98299999999999998},"min":{"1min":0.47199999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.3460000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.075,"15min":1.071},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.8680000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.1839999999999999,"15min":1.0640000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":20.439,"5min":20.439,"15min":20.439},"last":1.0469999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.4359999999999999,"15min":1.256},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":13.641999999999999,"5min":13.641999999999999,"15min":13.641999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.4670000000000001,"15min":1.3260000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":13.789999999999999,"5min":13.789999999999999,"15min":13.789999999999999},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.516,"15min":1.4419999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":14.129,"5min":14.129,"15min":14.129},"last":1.377},{"interface":"front","average":{"1min":3.399,"5min":1.8160000000000001,"15min":1.5609999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":35.747,"5min":35.747,"15min":35.747},"last":1.466}]},{"osd":6,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9359999999999999,"5min":1.5669999999999999,"15min":1.4810000000000001},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":11.545,"5min":11.545,"15min":11.545},"last":1.5920000000000001},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.5600000000000001,"15min":1.4690000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":13.56,"5min":13.56,"15min":13.56},"last":1.101}]},{"osd":7,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.629,"15min":1.4770000000000001},"min":{"1min":0.70699999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":6.2969999999999997,"5min":12.494999999999999,"15min":12.494999999999999},"last":1.405},{"interface":"front","average":{"1min":2.7120000000000002,"5min":1.6970000000000001,"15min":1.524},"min":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":31.044,"5min":31.044,"15min":31.044},"last":1.6870000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.98399999999999999,"15min":0.97599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.4379999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.2230000000000001,"15min":1.0569999999999999},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":4.3550000000000004,"5min":17.876000000000001,"15min":17.876000000000001},"last":0.79700000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.357,"15min":1.1839999999999999},"min":{"1min":0.54200000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":5.9080000000000004,"5min":17.760999999999999,"15min":17.760999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.226,"15min":1.232},"min":{"1min":0.56100000000000005,"5min":0.38,"15min":0.38},"max":{"1min":1.9139999999999999,"5min":5.5970000000000004,"15min":5.5970000000000004},"last":0.875}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.401,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.3520000000000001,"15min":1.137},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.4169999999999998,"5min":17.981000000000002,"15min":17.981000000000002},"last":0.95599999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.6040000000000001,"15min":1.4870000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":7.0389999999999997,"5min":14.297000000000001,"15min":14.297000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":2.9910000000000001,"5min":1.8939999999999999,"15min":1.4830000000000001},"min":{"1min":0.83999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":18.347999999999999,"5min":19.422000000000001,"15min":19.422000000000001},"last":1.1919999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":1.716,"15min":1.5209999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":10.933,"5min":14.212999999999999,"15min":14.212999999999999},"last":1.169},{"interface":"front","average":{"1min":2.319,"5min":1.79,"15min":1.452},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":16.446999999999999,"5min":19.315000000000001,"15min":19.315000000000001},"last":1.292}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":4.1150000000000002,"5min":2.2349999999999999,"15min":1.677},"min":{"1min":0.81499999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":27.442,"5min":27.442,"15min":27.442},"last":1.321},{"interface":"front","average":{"1min":3.9689999999999999,"5min":2.1800000000000002,"15min":1.7629999999999999},"min":{"1min":0.70899999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":27.411999999999999,"5min":27.411999999999999,"15min":27.411999999999999},"last":1.3440000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":1.8819999999999999,"15min":1.5640000000000001},"min":{"1min":0.78700000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":7.4130000000000003,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.6819999999999999,"15min":1.6259999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":5.335,"5min":5.9269999999999996,"15min":5.9269999999999996},"last":1.262}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:34.595 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:34.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.858+0000 7f8eb3a00640 1 Processor -- start 2024-01-16T15:02:34.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.858+0000 7f8eb3a00640 1 -- start start 2024-01-16T15:02:34.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.859+0000 7f8eb3a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8eac13df80 0x7f8eac15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.859+0000 7f8eb3a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8eac13e360 0x7f8eac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.859+0000 7f8eb3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac15f9c0 0x7f8eac0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.859+0000 7f8eb3a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac06c470 con 0x7f8eac13df80 2024-01-16T15:02:34.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb3a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac06c750 con 0x7f8eac13e360 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac06ca30 con 0x7f8eac15f9c0 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8eac13df80 0x7f8eac15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8eac13df80 0x7f8eac15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59788/0 (socket says 172.21.15.141:59788) 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 -- 172.21.15.141:0/2618251708 learned_addr learned my addr 172.21.15.141:0/2618251708 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 -- 172.21.15.141:0/2618251708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac15f9c0 msgr2=0x7f8eac0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 --2- 172.21.15.141:0/2618251708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac15f9c0 0x7f8eac0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 -- 172.21.15.141:0/2618251708 >> v2:172.21.15.188:3300/0 conn(0x7f8eac13e360 msgr2=0x7f8eac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 --2- 172.21.15.141:0/2618251708 >> v2:172.21.15.188:3300/0 conn(0x7f8eac13e360 0x7f8eac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.860+0000 7f8eb1000640 1 -- 172.21.15.141:0/2618251708 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac06cd10 con 0x7f8eac13df80 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.861+0000 7f8eb1000640 1 --2- 172.21.15.141:0/2618251708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac13df80 0x7f8eac15bee0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8eac06ded0 tx=0x7f8e9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f451007192a8020d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.861+0000 7f8eabe00640 1 -- 172.21.15.141:0/2618251708 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c02ca70 con 0x7f8eac13df80 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eabe00640 1 -- 172.21.15.141:0/2618251708 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c02cbd0 con 0x7f8eac13df80 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eabe00640 1 -- 172.21.15.141:0/2618251708 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c0343f0 con 0x7f8eac13df80 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 -- 172.21.15.141:0/2618251708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac13df80 msgr2=0x7f8eac15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/2618251708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac13df80 0x7f8eac15bee0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8eac06ded0 tx=0x7f8e9c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 -- 172.21.15.141:0/2618251708 shutdown_connections 2024-01-16T15:02:34.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/2618251708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac15f9c0 0x7f8eac0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/2618251708 >> v2:172.21.15.188:3300/0 conn(0x7f8eac13e360 0x7f8eac0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/2618251708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac13df80 0x7f8eac15bee0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 -- 172.21.15.141:0/2618251708 >> 172.21.15.141:0/2618251708 conn(0x7f8eac06a800 msgr2=0x7f8eac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 -- 172.21.15.141:0/2618251708 shutdown_connections 2024-01-16T15:02:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.862+0000 7f8eb3a00640 1 -- 172.21.15.141:0/2618251708 wait complete. 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.864+0000 7f8eb3a00640 1 Processor -- start 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.864+0000 7f8eb3a00640 1 -- start start 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.865+0000 7f8eb3a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8eac13df80 0x7f8eac071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.865+0000 7f8eb3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac13e360 0x7f8eac1f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.865+0000 7f8eb3a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac15f9c0 0x7f8eac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.865+0000 7f8eb3a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac071890 con 0x7f8eac15f9c0 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.865+0000 7f8eb3a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac1f3410 con 0x7f8eac13df80 2024-01-16T15:02:34.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.865+0000 7f8eb3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac070c80 con 0x7f8eac13e360 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.865+0000 7f8eb1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac15f9c0 0x7f8eac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac15f9c0 0x7f8eac0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59792/0 (socket says 172.21.15.141:59792) 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 -- 172.21.15.141:0/3935395962 learned_addr learned my addr 172.21.15.141:0/3935395962 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1000640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8eac13df80 0x7f8eac071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8ea3c00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac13e360 0x7f8eac1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 -- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac13e360 msgr2=0x7f8eac1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac13e360 0x7f8eac1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 -- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8eac13df80 msgr2=0x7f8eac071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8eac13df80 0x7f8eac071d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 -- 172.21.15.141:0/3935395962 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c009b30 con 0x7f8eac15f9c0 2024-01-16T15:02:34.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1000640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8eac13df80 0x7f8eac071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:34.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.866+0000 7f8eb1a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac15f9c0 0x7f8eac0702c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400b420 tx=0x7f8ea400b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.867+0000 7f8ea9600640 1 -- 172.21.15.141:0/3935395962 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4011830 con 0x7f8eac15f9c0 2024-01-16T15:02:34.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.867+0000 7f8ea9600640 1 -- 172.21.15.141:0/3935395962 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea40119b0 con 0x7f8eac15f9c0 2024-01-16T15:02:34.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.867+0000 7f8ea9600640 1 -- 172.21.15.141:0/3935395962 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4011d60 con 0x7f8eac15f9c0 2024-01-16T15:02:34.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.867+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac070f60 con 0x7f8eac15f9c0 2024-01-16T15:02:34.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.867+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac071520 con 0x7f8eac15f9c0 2024-01-16T15:02:34.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.868+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac071a40 con 0x7f8eac15f9c0 2024-01-16T15:02:34.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.869+0000 7f8ea9600640 1 -- 172.21.15.141:0/3935395962 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8ea401a400 con 0x7f8eac15f9c0 2024-01-16T15:02:34.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.870+0000 7f8ea9600640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e8406b860 0x7f8e8406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.870+0000 7f8ea9600640 1 -- 172.21.15.141:0/3935395962 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ea408a6a0 con 0x7f8eac15f9c0 2024-01-16T15:02:34.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.870+0000 7f8eb1000640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e8406b860 0x7f8e8406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.871+0000 7f8eb1000640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e8406b860 0x7f8e8406dd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c035000 tx=0x7f8e9c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:34.874+0000 7f8ea9600640 1 -- 172.21.15.141:0/3935395962 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8eac071a40 con 0x7f8eac15f9c0 2024-01-16T15:02:35.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.057+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8eac1408d0 con 0x7f8e8406b860 2024-01-16T15:02:35.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.078+0000 7f8ea9600640 1 -- 172.21.15.141:0/3935395962 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282293 (secure 0 0 0) 0x7f8e9c044090 con 0x7f8e8406b860 2024-01-16T15:02:35.063 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:35.064 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:02:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e8406b860 msgr2=0x7f8e8406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e8406b860 0x7f8e8406dd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c035000 tx=0x7f8e9c030000 comp rx=0 tx=0).stop 2024-01-16T15:02:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac15f9c0 msgr2=0x7f8eac0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac15f9c0 0x7f8eac0702c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400b420 tx=0x7f8ea400b820 comp rx=0 tx=0).stop 2024-01-16T15:02:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 shutdown_connections 2024-01-16T15:02:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e8406b860 0x7f8e8406dd20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8eac15f9c0 0x7f8eac0702c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8eac13e360 0x7f8eac1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 --2- 172.21.15.141:0/3935395962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8eac13df80 0x7f8eac071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 >> 172.21.15.141:0/3935395962 conn(0x7f8eac06a800 msgr2=0x7f8eac13faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 shutdown_connections 2024-01-16T15:02:35.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.083+0000 7f8eb3a00640 1 -- 172.21.15.141:0/3935395962 wait complete. 2024-01-16T15:02:35.081 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-01-16T15:02:34.586289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289216,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685504,"statfs":{"total":773094113280,"available":772797956096,"internally_reserved":0,"allocated":6627328,"data_stored":3419944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.011975"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.691258+0000","last_change":"2024-01-16T15:02:28.691258+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.691258+0000","last_undegraded":"2024-01-16T15:02:28.691258+0000","last_fullsized":"2024-01-16T15:02:28.691258+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674881+0000","last_change":"2024-01-16T15:02:28.674881+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674881+0000","last_undegraded":"2024-01-16T15:02:28.674881+0000","last_fullsized":"2024-01-16T15:02:28.674881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.685084+0000","last_change":"2024-01-16T15:02:28.685084+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.685084+0000","last_undegraded":"2024-01-16T15:02:28.685084+0000","last_fullsized":"2024-01-16T15:02:28.685084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682154+0000","last_change":"2024-01-16T15:02:28.682154+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682154+0000","last_undegraded":"2024-01-16T15:02:28.682154+0000","last_fullsized":"2024-01-16T15:02:28.682154+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687284+0000","last_change":"2024-01-16T15:02:28.687284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687284+0000","last_undegraded":"2024-01-16T15:02:28.687284+0000","last_fullsized":"2024-01-16T15:02:28.687284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680550+0000","last_change":"2024-01-16T15:02:28.680550+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680550+0000","last_undegraded":"2024-01-16T15:02:28.680550+0000","last_fullsized":"2024-01-16T15:02:28.680550+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677104+0000","last_change":"2024-01-16T15:02:28.677104+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677104+0000","last_undegraded":"2024-01-16T15:02:28.677104+0000","last_fullsized":"2024-01-16T15:02:28.677104+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677904+0000","last_change":"2024-01-16T15:02:28.677904+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677904+0000","last_undegraded":"2024-01-16T15:02:28.677904+0000","last_fullsized":"2024-01-16T15:02:28.677904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671452+0000","last_change":"2024-01-16T15:02:28.671452+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671452+0000","last_undegraded":"2024-01-16T15:02:28.671452+0000","last_fullsized":"2024-01-16T15:02:28.671452+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740996+0000","last_change":"2024-01-16T15:02:29.726552+0000","last_active":"2024-01-16T15:02:29.740996+0000","last_peered":"2024-01-16T15:02:29.740996+0000","last_clean":"2024-01-16T15:02:29.740996+0000","last_became_active":"2024-01-16T15:02:29.726242+0000","last_became_peered":"2024-01-16T15:02:29.726242+0000","last_unstale":"2024-01-16T15:02:29.740996+0000","last_undegraded":"2024-01-16T15:02:29.740996+0000","last_fullsized":"2024-01-16T15:02:29.740996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:15.816925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.743873+0000","last_change":"2024-01-16T15:02:29.735792+0000","last_active":"2024-01-16T15:02:29.743873+0000","last_peered":"2024-01-16T15:02:29.743873+0000","last_clean":"2024-01-16T15:02:29.743873+0000","last_became_active":"2024-01-16T15:02:29.735504+0000","last_became_peered":"2024-01-16T15:02:29.735504+0000","last_unstale":"2024-01-16T15:02:29.743873+0000","last_undegraded":"2024-01-16T15:02:29.743873+0000","last_fullsized":"2024-01-16T15:02:29.743873+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:09.173541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681639+0000","last_change":"2024-01-16T15:02:28.681639+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681639+0000","last_undegraded":"2024-01-16T15:02:28.681639+0000","last_fullsized":"2024-01-16T15:02:28.681639+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726359+0000","last_change":"2024-01-16T15:02:29.721939+0000","last_active":"2024-01-16T15:02:29.726359+0000","last_peered":"2024-01-16T15:02:29.726359+0000","last_clean":"2024-01-16T15:02:29.726359+0000","last_became_active":"2024-01-16T15:02:29.721563+0000","last_became_peered":"2024-01-16T15:02:29.721563+0000","last_unstale":"2024-01-16T15:02:29.726359+0000","last_undegraded":"2024-01-16T15:02:29.726359+0000","last_fullsized":"2024-01-16T15:02:29.726359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:35:52.362185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677387+0000","last_change":"2024-01-16T15:02:28.677387+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677387+0000","last_undegraded":"2024-01-16T15:02:28.677387+0000","last_fullsized":"2024-01-16T15:02:28.677387+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671228+0000","last_change":"2024-01-16T15:02:28.671228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671228+0000","last_undegraded":"2024-01-16T15:02:28.671228+0000","last_fullsized":"2024-01-16T15:02:28.671228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679199+0000","last_change":"2024-01-16T15:02:28.679199+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679199+0000","last_undegraded":"2024-01-16T15:02:28.679199+0000","last_fullsized":"2024-01-16T15:02:28.679199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674612+0000","last_change":"2024-01-16T15:02:28.674612+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674612+0000","last_undegraded":"2024-01-16T15:02:28.674612+0000","last_fullsized":"2024-01-16T15:02:28.674612+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.688365+0000","last_change":"2024-01-16T15:02:28.688365+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.688365+0000","last_undegraded":"2024-01-16T15:02:28.688365+0000","last_fullsized":"2024-01-16T15:02:28.688365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684341+0000","last_change":"2024-01-16T15:02:28.684341+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684341+0000","last_undegraded":"2024-01-16T15:02:28.684341+0000","last_fullsized":"2024-01-16T15:02:28.684341+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677483+0000","last_change":"2024-01-16T15:02:28.677483+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677483+0000","last_undegraded":"2024-01-16T15:02:28.677483+0000","last_fullsized":"2024-01-16T15:02:28.677483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680887+0000","last_change":"2024-01-16T15:02:28.680887+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680887+0000","last_undegraded":"2024-01-16T15:02:28.680887+0000","last_fullsized":"2024-01-16T15:02:28.680887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675926+0000","last_change":"2024-01-16T15:02:28.675926+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675926+0000","last_undegraded":"2024-01-16T15:02:28.675926+0000","last_fullsized":"2024-01-16T15:02:28.675926+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680236+0000","last_change":"2024-01-16T15:02:28.680236+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680236+0000","last_undegraded":"2024-01-16T15:02:28.680236+0000","last_fullsized":"2024-01-16T15:02:28.680236+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726074+0000","last_change":"2024-01-16T15:02:29.721477+0000","last_active":"2024-01-16T15:02:29.726074+0000","last_peered":"2024-01-16T15:02:29.726074+0000","last_clean":"2024-01-16T15:02:29.726074+0000","last_became_active":"2024-01-16T15:02:29.721137+0000","last_became_peered":"2024-01-16T15:02:29.721137+0000","last_unstale":"2024-01-16T15:02:29.726074+0000","last_undegraded":"2024-01-16T15:02:29.726074+0000","last_fullsized":"2024-01-16T15:02:29.726074+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:31.998282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.690190+0000","last_change":"2024-01-16T15:02:28.690190+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.690190+0000","last_undegraded":"2024-01-16T15:02:28.690190+0000","last_fullsized":"2024-01-16T15:02:28.690190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683203+0000","last_change":"2024-01-16T15:02:28.683203+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683203+0000","last_undegraded":"2024-01-16T15:02:28.683203+0000","last_fullsized":"2024-01-16T15:02:28.683203+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678571+0000","last_change":"2024-01-16T15:02:28.678571+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678571+0000","last_undegraded":"2024-01-16T15:02:28.678571+0000","last_fullsized":"2024-01-16T15:02:28.678571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687166+0000","last_change":"2024-01-16T15:02:28.687166+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687166+0000","last_undegraded":"2024-01-16T15:02:28.687166+0000","last_fullsized":"2024-01-16T15:02:28.687166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674298+0000","last_change":"2024-01-16T15:02:28.674298+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674298+0000","last_undegraded":"2024-01-16T15:02:28.674298+0000","last_fullsized":"2024-01-16T15:02:28.674298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682909+0000","last_change":"2024-01-16T15:02:28.682909+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682909+0000","last_undegraded":"2024-01-16T15:02:28.682909+0000","last_fullsized":"2024-01-16T15:02:28.682909+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683833+0000","last_change":"2024-01-16T15:02:28.683833+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683833+0000","last_undegraded":"2024-01-16T15:02:28.683833+0000","last_fullsized":"2024-01-16T15:02:28.683833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675958+0000","last_change":"2024-01-16T15:02:28.675958+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675958+0000","last_undegraded":"2024-01-16T15:02:28.675958+0000","last_fullsized":"2024-01-16T15:02:28.675958+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673472+0000","last_change":"2024-01-16T15:02:28.673472+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673472+0000","last_undegraded":"2024-01-16T15:02:28.673472+0000","last_fullsized":"2024-01-16T15:02:28.673472+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700420+0000","last_change":"2024-01-16T15:02:28.696840+0000","last_active":"2024-01-16T15:02:28.700420+0000","last_peered":"2024-01-16T15:02:28.700420+0000","last_clean":"2024-01-16T15:02:28.700420+0000","last_became_active":"2024-01-16T15:02:28.695989+0000","last_became_peered":"2024-01-16T15:02:28.695989+0000","last_unstale":"2024-01-16T15:02:28.700420+0000","last_undegraded":"2024-01-16T15:02:28.700420+0000","last_fullsized":"2024-01-16T15:02:28.700420+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:36.619091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.741340+0000","last_change":"2024-01-16T15:02:29.722881+0000","last_active":"2024-01-16T15:02:29.741340+0000","last_peered":"2024-01-16T15:02:29.741340+0000","last_clean":"2024-01-16T15:02:29.741340+0000","last_became_active":"2024-01-16T15:02:29.722447+0000","last_became_peered":"2024-01-16T15:02:29.722447+0000","last_unstale":"2024-01-16T15:02:29.741340+0000","last_undegraded":"2024-01-16T15:02:29.741340+0000","last_fullsized":"2024-01-16T15:02:29.741340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:52.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700061+0000","last_change":"2024-01-16T15:02:28.697447+0000","last_active":"2024-01-16T15:02:28.700061+0000","last_peered":"2024-01-16T15:02:28.700061+0000","last_clean":"2024-01-16T15:02:28.700061+0000","last_became_active":"2024-01-16T15:02:28.697122+0000","last_became_peered":"2024-01-16T15:02:28.697122+0000","last_unstale":"2024-01-16T15:02:28.700061+0000","last_undegraded":"2024-01-16T15:02:28.700061+0000","last_fullsized":"2024-01-16T15:02:28.700061+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:09.273766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672137+0000","last_change":"2024-01-16T15:02:28.672137+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672137+0000","last_undegraded":"2024-01-16T15:02:28.672137+0000","last_fullsized":"2024-01-16T15:02:28.672137+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696624+0000","last_change":"2024-01-16T15:02:28.689593+0000","last_active":"2024-01-16T15:02:28.696624+0000","last_peered":"2024-01-16T15:02:28.696624+0000","last_clean":"2024-01-16T15:02:28.696624+0000","last_became_active":"2024-01-16T15:02:28.689343+0000","last_became_peered":"2024-01-16T15:02:28.689343+0000","last_unstale":"2024-01-16T15:02:28.696624+0000","last_undegraded":"2024-01-16T15:02:28.696624+0000","last_fullsized":"2024-01-16T15:02:28.696624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:51.159242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673332+0000","last_change":"2024-01-16T15:02:28.673332+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673332+0000","last_undegraded":"2024-01-16T15:02:28.673332+0000","last_fullsized":"2024-01-16T15:02:28.673332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698069+0000","last_change":"2024-01-16T15:02:28.694539+0000","last_active":"2024-01-16T15:02:28.698069+0000","last_peered":"2024-01-16T15:02:28.698069+0000","last_clean":"2024-01-16T15:02:28.698069+0000","last_became_active":"2024-01-16T15:02:28.694233+0000","last_became_peered":"2024-01-16T15:02:28.694233+0000","last_unstale":"2024-01-16T15:02:28.698069+0000","last_undegraded":"2024-01-16T15:02:28.698069+0000","last_fullsized":"2024-01-16T15:02:28.698069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675366+0000","last_change":"2024-01-16T15:02:28.675366+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675366+0000","last_undegraded":"2024-01-16T15:02:28.675366+0000","last_fullsized":"2024-01-16T15:02:28.675366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696756+0000","last_change":"2024-01-16T15:02:28.693560+0000","last_active":"2024-01-16T15:02:28.696756+0000","last_peered":"2024-01-16T15:02:28.696756+0000","last_clean":"2024-01-16T15:02:28.696756+0000","last_became_active":"2024-01-16T15:02:28.693211+0000","last_became_peered":"2024-01-16T15:02:28.693211+0000","last_unstale":"2024-01-16T15:02:28.696756+0000","last_undegraded":"2024-01-16T15:02:28.696756+0000","last_fullsized":"2024-01-16T15:02:28.696756+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676930+0000","last_change":"2024-01-16T15:02:28.676930+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676930+0000","last_undegraded":"2024-01-16T15:02:28.676930+0000","last_fullsized":"2024-01-16T15:02:28.676930+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.691282+0000","last_change":"2024-01-16T15:02:28.683014+0000","last_active":"2024-01-16T15:02:28.691282+0000","last_peered":"2024-01-16T15:02:28.691282+0000","last_clean":"2024-01-16T15:02:28.691282+0000","last_became_active":"2024-01-16T15:02:28.682735+0000","last_became_peered":"2024-01-16T15:02:28.682735+0000","last_unstale":"2024-01-16T15:02:28.691282+0000","last_undegraded":"2024-01-16T15:02:28.691282+0000","last_fullsized":"2024-01-16T15:02:28.691282+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:55.974417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726389+0000","last_change":"2024-01-16T15:02:29.721954+0000","last_active":"2024-01-16T15:02:29.726389+0000","last_peered":"2024-01-16T15:02:29.726389+0000","last_clean":"2024-01-16T15:02:29.726389+0000","last_became_active":"2024-01-16T15:02:29.721647+0000","last_became_peered":"2024-01-16T15:02:29.721647+0000","last_unstale":"2024-01-16T15:02:29.726389+0000","last_undegraded":"2024-01-16T15:02:29.726389+0000","last_fullsized":"2024-01-16T15:02:29.726389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:16.064389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697872+0000","last_change":"2024-01-16T15:02:28.694096+0000","last_active":"2024-01-16T15:02:28.697872+0000","last_peered":"2024-01-16T15:02:28.697872+0000","last_clean":"2024-01-16T15:02:28.697872+0000","last_became_active":"2024-01-16T15:02:28.693789+0000","last_became_peered":"2024-01-16T15:02:28.693789+0000","last_unstale":"2024-01-16T15:02:28.697872+0000","last_undegraded":"2024-01-16T15:02:28.697872+0000","last_fullsized":"2024-01-16T15:02:28.697872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681431+0000","last_change":"2024-01-16T15:02:28.681431+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681431+0000","last_undegraded":"2024-01-16T15:02:28.681431+0000","last_fullsized":"2024-01-16T15:02:28.681431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696269+0000","last_change":"2024-01-16T15:02:28.693108+0000","last_active":"2024-01-16T15:02:28.696269+0000","last_peered":"2024-01-16T15:02:28.696269+0000","last_clean":"2024-01-16T15:02:28.696269+0000","last_became_active":"2024-01-16T15:02:28.691074+0000","last_became_peered":"2024-01-16T15:02:28.691074+0000","last_unstale":"2024-01-16T15:02:28.696269+0000","last_undegraded":"2024-01-16T15:02:28.696269+0000","last_fullsized":"2024-01-16T15:02:28.696269+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672957+0000","last_change":"2024-01-16T15:02:28.672957+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672957+0000","last_undegraded":"2024-01-16T15:02:28.672957+0000","last_fullsized":"2024-01-16T15:02:28.672957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702078+0000","last_change":"2024-01-16T15:02:28.700517+0000","last_active":"2024-01-16T15:02:28.702078+0000","last_peered":"2024-01-16T15:02:28.702078+0000","last_clean":"2024-01-16T15:02:28.702078+0000","last_became_active":"2024-01-16T15:02:28.700187+0000","last_became_peered":"2024-01-16T15:02:28.700187+0000","last_unstale":"2024-01-16T15:02:28.702078+0000","last_undegraded":"2024-01-16T15:02:28.702078+0000","last_fullsized":"2024-01-16T15:02:28.702078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.686150+0000","last_change":"2024-01-16T15:02:28.686150+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.686150+0000","last_undegraded":"2024-01-16T15:02:28.686150+0000","last_fullsized":"2024-01-16T15:02:28.686150+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699635+0000","last_change":"2024-01-16T15:02:28.694170+0000","last_active":"2024-01-16T15:02:28.699635+0000","last_peered":"2024-01-16T15:02:28.699635+0000","last_clean":"2024-01-16T15:02:28.699635+0000","last_became_active":"2024-01-16T15:02:28.693951+0000","last_became_peered":"2024-01-16T15:02:28.693951+0000","last_unstale":"2024-01-16T15:02:28.699635+0000","last_undegraded":"2024-01-16T15:02:28.699635+0000","last_fullsized":"2024-01-16T15:02:28.699635+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:05.345755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740967+0000","last_change":"2024-01-16T15:02:29.728569+0000","last_active":"2024-01-16T15:02:29.740967+0000","last_peered":"2024-01-16T15:02:29.740967+0000","last_clean":"2024-01-16T15:02:29.740967+0000","last_became_active":"2024-01-16T15:02:29.728240+0000","last_became_peered":"2024-01-16T15:02:29.728240+0000","last_unstale":"2024-01-16T15:02:29.740967+0000","last_undegraded":"2024-01-16T15:02:29.740967+0000","last_fullsized":"2024-01-16T15:02:29.740967+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:49.783058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702091+0000","last_change":"2024-01-16T15:02:28.700598+0000","last_active":"2024-01-16T15:02:28.702091+0000","last_peered":"2024-01-16T15:02:28.702091+0000","last_clean":"2024-01-16T15:02:28.702091+0000","last_became_active":"2024-01-16T15:02:28.700296+0000","last_became_peered":"2024-01-16T15:02:28.700296+0000","last_unstale":"2024-01-16T15:02:28.702091+0000","last_undegraded":"2024-01-16T15:02:28.702091+0000","last_fullsized":"2024-01-16T15:02:28.702091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684658+0000","last_change":"2024-01-16T15:02:28.684658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684658+0000","last_undegraded":"2024-01-16T15:02:28.684658+0000","last_fullsized":"2024-01-16T15:02:28.684658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701099+0000","last_change":"2024-01-16T15:02:28.697153+0000","last_active":"2024-01-16T15:02:28.701099+0000","last_peered":"2024-01-16T15:02:28.701099+0000","last_clean":"2024-01-16T15:02:28.701099+0000","last_became_active":"2024-01-16T15:02:28.693885+0000","last_became_peered":"2024-01-16T15:02:28.693885+0000","last_unstale":"2024-01-16T15:02:28.701099+0000","last_undegraded":"2024-01-16T15:02:28.701099+0000","last_fullsized":"2024-01-16T15:02:28.701099+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683793+0000","last_change":"2024-01-16T15:02:28.683793+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683793+0000","last_undegraded":"2024-01-16T15:02:28.683793+0000","last_fullsized":"2024-01-16T15:02:28.683793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699827+0000","last_change":"2024-01-16T15:02:28.695099+0000","last_active":"2024-01-16T15:02:28.699827+0000","last_peered":"2024-01-16T15:02:28.699827+0000","last_clean":"2024-01-16T15:02:28.699827+0000","last_became_active":"2024-01-16T15:02:28.694828+0000","last_became_peered":"2024-01-16T15:02:28.694828+0000","last_unstale":"2024-01-16T15:02:28.699827+0000","last_undegraded":"2024-01-16T15:02:28.699827+0000","last_fullsized":"2024-01-16T15:02:28.699827+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:29.254715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680264+0000","last_change":"2024-01-16T15:02:28.680264+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680264+0000","last_undegraded":"2024-01-16T15:02:28.680264+0000","last_fullsized":"2024-01-16T15:02:28.680264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706804+0000","last_change":"2024-01-16T15:02:28.688577+0000","last_active":"2024-01-16T15:02:29.706804+0000","last_peered":"2024-01-16T15:02:29.706804+0000","last_clean":"2024-01-16T15:02:29.706804+0000","last_became_active":"2024-01-16T15:02:28.687999+0000","last_became_peered":"2024-01-16T15:02:28.687999+0000","last_unstale":"2024-01-16T15:02:29.706804+0000","last_undegraded":"2024-01-16T15:02:29.706804+0000","last_fullsized":"2024-01-16T15:02:29.706804+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:17.380916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673658+0000","last_change":"2024-01-16T15:02:28.673658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673658+0000","last_undegraded":"2024-01-16T15:02:28.673658+0000","last_fullsized":"2024-01-16T15:02:28.673658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697893+0000","last_change":"2024-01-16T15:02:28.694910+0000","last_active":"2024-01-16T15:02:28.697893+0000","last_peered":"2024-01-16T15:02:28.697893+0000","last_clean":"2024-01-16T15:02:28.697893+0000","last_became_active":"2024-01-16T15:02:28.690704+0000","last_became_peered":"2024-01-16T15:02:28.690704+0000","last_unstale":"2024-01-16T15:02:28.697893+0000","last_undegraded":"2024-01-16T15:02:28.697893+0000","last_fullsized":"2024-01-16T15:02:28.697893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678221+0000","last_change":"2024-01-16T15:02:28.678221+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678221+0000","last_undegraded":"2024-01-16T15:02:28.678221+0000","last_fullsized":"2024-01-16T15:02:28.678221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701998+0000","last_change":"2024-01-16T15:02:28.699544+0000","last_active":"2024-01-16T15:02:28.701998+0000","last_peered":"2024-01-16T15:02:28.701998+0000","last_clean":"2024-01-16T15:02:28.701998+0000","last_became_active":"2024-01-16T15:02:28.699169+0000","last_became_peered":"2024-01-16T15:02:28.699169+0000","last_unstale":"2024-01-16T15:02:28.701998+0000","last_undegraded":"2024-01-16T15:02:28.701998+0000","last_fullsized":"2024-01-16T15:02:28.701998+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687323+0000","last_change":"2024-01-16T15:02:28.687323+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687323+0000","last_undegraded":"2024-01-16T15:02:28.687323+0000","last_fullsized":"2024-01-16T15:02:28.687323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699992+0000","last_change":"2024-01-16T15:02:28.695124+0000","last_active":"2024-01-16T15:02:28.699992+0000","last_peered":"2024-01-16T15:02:28.699992+0000","last_clean":"2024-01-16T15:02:28.699992+0000","last_became_active":"2024-01-16T15:02:28.694847+0000","last_became_peered":"2024-01-16T15:02:28.694847+0000","last_unstale":"2024-01-16T15:02:28.699992+0000","last_undegraded":"2024-01-16T15:02:28.699992+0000","last_fullsized":"2024-01-16T15:02:28.699992+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:56.293921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684228+0000","last_change":"2024-01-16T15:02:28.684228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684228+0000","last_undegraded":"2024-01-16T15:02:28.684228+0000","last_fullsized":"2024-01-16T15:02:28.684228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700477+0000","last_change":"2024-01-16T15:02:28.698762+0000","last_active":"2024-01-16T15:02:28.700477+0000","last_peered":"2024-01-16T15:02:28.700477+0000","last_clean":"2024-01-16T15:02:28.700477+0000","last_became_active":"2024-01-16T15:02:28.698337+0000","last_became_peered":"2024-01-16T15:02:28.698337+0000","last_unstale":"2024-01-16T15:02:28.700477+0000","last_undegraded":"2024-01-16T15:02:28.700477+0000","last_fullsized":"2024-01-16T15:02:28.700477+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:40.971035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676992+0000","last_change":"2024-01-16T15:02:28.676992+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676992+0000","last_undegraded":"2024-01-16T15:02:28.676992+0000","last_fullsized":"2024-01-16T15:02:28.676992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698294+0000","last_change":"2024-01-16T15:02:28.695728+0000","last_active":"2024-01-16T15:02:28.698294+0000","last_peered":"2024-01-16T15:02:28.698294+0000","last_clean":"2024-01-16T15:02:28.698294+0000","last_became_active":"2024-01-16T15:02:28.695438+0000","last_became_peered":"2024-01-16T15:02:28.695438+0000","last_unstale":"2024-01-16T15:02:28.698294+0000","last_undegraded":"2024-01-16T15:02:28.698294+0000","last_fullsized":"2024-01-16T15:02:28.698294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679870+0000","last_change":"2024-01-16T15:02:28.679870+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679870+0000","last_undegraded":"2024-01-16T15:02:28.679870+0000","last_fullsized":"2024-01-16T15:02:28.679870+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696703+0000","last_change":"2024-01-16T15:02:28.690359+0000","last_active":"2024-01-16T15:02:28.696703+0000","last_peered":"2024-01-16T15:02:28.696703+0000","last_clean":"2024-01-16T15:02:28.696703+0000","last_became_active":"2024-01-16T15:02:28.689968+0000","last_became_peered":"2024-01-16T15:02:28.689968+0000","last_unstale":"2024-01-16T15:02:28.696703+0000","last_undegraded":"2024-01-16T15:02:28.696703+0000","last_fullsized":"2024-01-16T15:02:28.696703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:06.195041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683284+0000","last_change":"2024-01-16T15:02:28.683284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683284+0000","last_undegraded":"2024-01-16T15:02:28.683284+0000","last_fullsized":"2024-01-16T15:02:28.683284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689306+0000","last_change":"2024-01-16T15:02:28.687104+0000","last_active":"2024-01-16T15:02:28.689306+0000","last_peered":"2024-01-16T15:02:28.689306+0000","last_clean":"2024-01-16T15:02:28.689306+0000","last_became_active":"2024-01-16T15:02:28.686179+0000","last_became_peered":"2024-01-16T15:02:28.686179+0000","last_unstale":"2024-01-16T15:02:28.689306+0000","last_undegraded":"2024-01-16T15:02:28.689306+0000","last_fullsized":"2024-01-16T15:02:28.689306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:39.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672340+0000","last_change":"2024-01-16T15:02:28.672340+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672340+0000","last_undegraded":"2024-01-16T15:02:28.672340+0000","last_fullsized":"2024-01-16T15:02:28.672340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.722558+0000","last_change":"2024-01-16T15:02:28.693574+0000","last_active":"2024-01-16T15:02:29.722558+0000","last_peered":"2024-01-16T15:02:29.722558+0000","last_clean":"2024-01-16T15:02:29.722558+0000","last_became_active":"2024-01-16T15:02:28.692884+0000","last_became_peered":"2024-01-16T15:02:28.692884+0000","last_unstale":"2024-01-16T15:02:29.722558+0000","last_undegraded":"2024-01-16T15:02:29.722558+0000","last_fullsized":"2024-01-16T15:02:29.722558+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672000+0000","last_change":"2024-01-16T15:02:28.672000+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672000+0000","last_undegraded":"2024-01-16T15:02:28.672000+0000","last_fullsized":"2024-01-16T15:02:28.672000+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696515+0000","last_change":"2024-01-16T15:02:28.692170+0000","last_active":"2024-01-16T15:02:28.696515+0000","last_peered":"2024-01-16T15:02:28.696515+0000","last_clean":"2024-01-16T15:02:28.696515+0000","last_became_active":"2024-01-16T15:02:28.691779+0000","last_became_peered":"2024-01-16T15:02:28.691779+0000","last_unstale":"2024-01-16T15:02:28.696515+0000","last_undegraded":"2024-01-16T15:02:28.696515+0000","last_fullsized":"2024-01-16T15:02:28.696515+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681494+0000","last_change":"2024-01-16T15:02:28.681494+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681494+0000","last_undegraded":"2024-01-16T15:02:28.681494+0000","last_fullsized":"2024-01-16T15:02:28.681494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.737753+0000","last_change":"2024-01-16T15:02:28.689348+0000","last_active":"2024-01-16T15:02:29.737753+0000","last_peered":"2024-01-16T15:02:29.737753+0000","last_clean":"2024-01-16T15:02:29.737753+0000","last_became_active":"2024-01-16T15:02:28.689030+0000","last_became_peered":"2024-01-16T15:02:28.689030+0000","last_unstale":"2024-01-16T15:02:29.737753+0000","last_undegraded":"2024-01-16T15:02:29.737753+0000","last_fullsized":"2024-01-16T15:02:29.737753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:24.816323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683770+0000","last_change":"2024-01-16T15:02:28.683770+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683770+0000","last_undegraded":"2024-01-16T15:02:28.683770+0000","last_fullsized":"2024-01-16T15:02:28.683770+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689249+0000","last_change":"2024-01-16T15:02:28.681603+0000","last_active":"2024-01-16T15:02:28.689249+0000","last_peered":"2024-01-16T15:02:28.689249+0000","last_clean":"2024-01-16T15:02:28.689249+0000","last_became_active":"2024-01-16T15:02:28.681322+0000","last_became_peered":"2024-01-16T15:02:28.681322+0000","last_unstale":"2024-01-16T15:02:28.689249+0000","last_undegraded":"2024-01-16T15:02:28.689249+0000","last_fullsized":"2024-01-16T15:02:28.689249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:13.938868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740947+0000","last_change":"2024-01-16T15:02:29.728538+0000","last_active":"2024-01-16T15:02:29.740947+0000","last_peered":"2024-01-16T15:02:29.740947+0000","last_clean":"2024-01-16T15:02:29.740947+0000","last_became_active":"2024-01-16T15:02:29.728211+0000","last_became_peered":"2024-01-16T15:02:29.728211+0000","last_unstale":"2024-01-16T15:02:29.740947+0000","last_undegraded":"2024-01-16T15:02:29.740947+0000","last_fullsized":"2024-01-16T15:02:29.740947+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:32.872029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706859+0000","last_change":"2024-01-16T15:02:28.681634+0000","last_active":"2024-01-16T15:02:29.706859+0000","last_peered":"2024-01-16T15:02:29.706859+0000","last_clean":"2024-01-16T15:02:29.706859+0000","last_became_active":"2024-01-16T15:02:28.680098+0000","last_became_peered":"2024-01-16T15:02:28.680098+0000","last_unstale":"2024-01-16T15:02:29.706859+0000","last_undegraded":"2024-01-16T15:02:29.706859+0000","last_fullsized":"2024-01-16T15:02:29.706859+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:45.583344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671790+0000","last_change":"2024-01-16T15:02:28.671790+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671790+0000","last_undegraded":"2024-01-16T15:02:28.671790+0000","last_fullsized":"2024-01-16T15:02:28.671790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697195+0000","last_change":"2024-01-16T15:02:28.692899+0000","last_active":"2024-01-16T15:02:28.697195+0000","last_peered":"2024-01-16T15:02:28.697195+0000","last_clean":"2024-01-16T15:02:28.697195+0000","last_became_active":"2024-01-16T15:02:28.692607+0000","last_became_peered":"2024-01-16T15:02:28.692607+0000","last_unstale":"2024-01-16T15:02:28.697195+0000","last_undegraded":"2024-01-16T15:02:28.697195+0000","last_fullsized":"2024-01-16T15:02:28.697195+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674849+0000","last_change":"2024-01-16T15:02:28.674849+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674849+0000","last_undegraded":"2024-01-16T15:02:28.674849+0000","last_fullsized":"2024-01-16T15:02:28.674849+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.728464+0000","last_change":"2024-01-16T15:02:28.695802+0000","last_active":"2024-01-16T15:02:29.728464+0000","last_peered":"2024-01-16T15:02:29.728464+0000","last_clean":"2024-01-16T15:02:29.728464+0000","last_became_active":"2024-01-16T15:02:28.695516+0000","last_became_peered":"2024-01-16T15:02:28.695516+0000","last_unstale":"2024-01-16T15:02:29.728464+0000","last_undegraded":"2024-01-16T15:02:29.728464+0000","last_fullsized":"2024-01-16T15:02:29.728464+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:00.682347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679260+0000","last_change":"2024-01-16T15:02:28.679260+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679260+0000","last_undegraded":"2024-01-16T15:02:28.679260+0000","last_fullsized":"2024-01-16T15:02:28.679260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696398+0000","last_change":"2024-01-16T15:02:28.691759+0000","last_active":"2024-01-16T15:02:28.696398+0000","last_peered":"2024-01-16T15:02:28.696398+0000","last_clean":"2024-01-16T15:02:28.696398+0000","last_became_active":"2024-01-16T15:02:28.691457+0000","last_became_peered":"2024-01-16T15:02:28.691457+0000","last_unstale":"2024-01-16T15:02:28.696398+0000","last_undegraded":"2024-01-16T15:02:28.696398+0000","last_fullsized":"2024-01-16T15:02:28.696398+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:31.867536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.738586+0000","last_change":"2024-01-16T15:02:29.719394+0000","last_active":"2024-01-16T15:02:29.738586+0000","last_peered":"2024-01-16T15:02:29.738586+0000","last_clean":"2024-01-16T15:02:29.738586+0000","last_became_active":"2024-01-16T15:02:29.719030+0000","last_became_peered":"2024-01-16T15:02:29.719030+0000","last_unstale":"2024-01-16T15:02:29.738586+0000","last_undegraded":"2024-01-16T15:02:29.738586+0000","last_fullsized":"2024-01-16T15:02:29.738586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:51.117200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.717109+0000","last_change":"2024-01-16T15:02:28.692453+0000","last_active":"2024-01-16T15:02:29.717109+0000","last_peered":"2024-01-16T15:02:29.717109+0000","last_clean":"2024-01-16T15:02:29.717109+0000","last_became_active":"2024-01-16T15:02:28.692132+0000","last_became_peered":"2024-01-16T15:02:28.692132+0000","last_unstale":"2024-01-16T15:02:29.717109+0000","last_undegraded":"2024-01-16T15:02:29.717109+0000","last_fullsized":"2024-01-16T15:02:29.717109+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674276+0000","last_change":"2024-01-16T15:02:28.674276+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674276+0000","last_undegraded":"2024-01-16T15:02:28.674276+0000","last_fullsized":"2024-01-16T15:02:28.674276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700041+0000","last_change":"2024-01-16T15:02:28.696929+0000","last_active":"2024-01-16T15:02:28.700041+0000","last_peered":"2024-01-16T15:02:28.700041+0000","last_clean":"2024-01-16T15:02:28.700041+0000","last_became_active":"2024-01-16T15:02:28.696616+0000","last_became_peered":"2024-01-16T15:02:28.696616+0000","last_unstale":"2024-01-16T15:02:28.700041+0000","last_undegraded":"2024-01-16T15:02:28.700041+0000","last_fullsized":"2024-01-16T15:02:28.700041+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:29.321021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675396+0000","last_change":"2024-01-16T15:02:28.675396+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675396+0000","last_undegraded":"2024-01-16T15:02:28.675396+0000","last_fullsized":"2024-01-16T15:02:28.675396+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.708768+0000","last_change":"2024-01-16T15:02:28.695669+0000","last_active":"2024-01-16T15:02:29.708768+0000","last_peered":"2024-01-16T15:02:29.708768+0000","last_clean":"2024-01-16T15:02:29.708768+0000","last_became_active":"2024-01-16T15:02:28.695350+0000","last_became_peered":"2024-01-16T15:02:28.695350+0000","last_unstale":"2024-01-16T15:02:29.708768+0000","last_undegraded":"2024-01-16T15:02:29.708768+0000","last_fullsized":"2024-01-16T15:02:29.708768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:36.266476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":199,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-16T15:02:27.660341+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:02:27.660341+0000","last_peered":"2024-01-16T15:02:27.660341+0000","last_clean":"2024-01-16T15:02:27.660341+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:02:27.660341+0000","last_undegraded":"2024-01-16T15:02:27.660341+0000","last_fullsized":"2024-01-16T15:02:27.660341+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.0329999999999999,"5min":1.3480000000000001,"15min":1.1539999999999999},"min":{"1min":0.73899999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3089999999999999,"15min":1.2010000000000001},"min":{"1min":0.76700000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9319999999999999,"5min":2.774,"15min":2.774},"last":1.1080000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.2909999999999999,"15min":1.111},"min":{"1min":0.76400000000000001,"5min":0.63,"15min":0.63},"max":{"1min":11.487,"5min":11.487,"15min":11.487},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3380000000000001,"15min":1.1890000000000001},"min":{"1min":0.89000000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.8109999999999999,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":0.82199999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.288,"15min":1.1970000000000001},"min":{"1min":0.85599999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.7269999999999999,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.28},{"interface":"front","average":{"1min":2.8940000000000001,"5min":1.6160000000000001,"15min":1.349},"min":{"1min":0.84199999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":20.577000000000002,"5min":20.577000000000002,"15min":20.577000000000002},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2639999999999998,"5min":1.6259999999999999,"15min":1.3979999999999999},"min":{"1min":1.0569999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":11.872,"5min":12.861000000000001,"15min":12.861000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.486,"15min":1.4099999999999999},"min":{"1min":0.81100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.1379999999999999,"5min":4.0720000000000001,"15min":4.0720000000000001},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.8439999999999999,"5min":1.631,"15min":1.446},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":20.649999999999999,"5min":20.649999999999999,"15min":20.649999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.387,"5min":1.633,"15min":1.5109999999999999},"min":{"1min":0.94399999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":11.702,"5min":11.702,"15min":11.702},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.514,"15min":1.526},"min":{"1min":0.70499999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.8039999999999998,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.73,"15min":1.6719999999999999},"min":{"1min":1.0149999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":11.225,"5min":11.225,"15min":11.225},"last":1.0649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":4.1589999999999998,"5min":2.0630000000000002,"15min":1.7430000000000001},"min":{"1min":1.306,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":41.206000000000003,"5min":41.206000000000003,"15min":41.206000000000003},"last":1.155},{"interface":"front","average":{"1min":4.125,"5min":2.0449999999999999,"15min":1.7589999999999999},"min":{"1min":1.0940000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":21.908000000000001,"5min":21.908000000000001,"15min":21.908000000000001},"last":1.3280000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.2609999999999999,"15min":1.147},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":9.1890000000000001,"5min":9.1890000000000001,"15min":9.1890000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":5.2039999999999997,"5min":1.976,"15min":1.478},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":50.472000000000001,"5min":50.472000000000001,"15min":50.472000000000001},"last":0.80600000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":1.3979999999999999,"15min":1.1459999999999999},"min":{"1min":0.66300000000000003,"5min":0.624,"15min":0.624},"max":{"1min":10.629,"5min":10.629,"15min":10.629},"last":0.98399999999999999},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.5509999999999999,"15min":1.421},"min":{"1min":0.82699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":9.3279999999999994,"5min":9.3279999999999994,"15min":9.3279999999999994},"last":1.0569999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.3789999999999996,"5min":1.9450000000000001,"15min":1.4199999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":50.643999999999998,"5min":50.643999999999998,"15min":50.643999999999998},"last":1.385},{"interface":"front","average":{"1min":2.4169999999999998,"5min":1.4970000000000001,"15min":1.359},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":10.843,"5min":10.843,"15min":10.843},"last":1.607}]},{"osd":3,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":1.6100000000000001,"15min":1.4359999999999999},"min":{"1min":0.82399999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":15.675000000000001,"5min":15.675000000000001,"15min":15.675000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":4.3760000000000003,"5min":1.99,"15min":1.5800000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":50.561,"5min":50.561,"15min":50.561},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.8289999999999997,"5min":2.1840000000000002,"15min":1.728},"min":{"1min":0.92700000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":47.488,"5min":47.488,"15min":47.488},"last":1.542},{"interface":"front","average":{"1min":5.2519999999999998,"5min":2.2330000000000001,"15min":1.7609999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":50.381,"5min":50.381,"15min":50.381},"last":1.4850000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.5419999999999998,"5min":1.6659999999999999,"15min":1.5409999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":10.654,"5min":10.654,"15min":10.654},"last":1.7450000000000001},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.8280000000000001,"15min":1.726},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":9.3729999999999993,"5min":9.3729999999999993,"15min":9.3729999999999993},"last":1.7789999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.9820000000000002,"5min":2.2799999999999998,"15min":1.8779999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":52.912999999999997,"5min":52.912999999999997,"15min":52.912999999999997},"last":1.581},{"interface":"front","average":{"1min":2.5270000000000001,"5min":1.7769999999999999,"15min":1.7130000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":10.581,"5min":10.581,"15min":10.581},"last":1.8600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3109999999999999,"5min":1.3580000000000001,"15min":1.248},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":13.938000000000001,"5min":13.938000000000001,"15min":13.938000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":2.9500000000000002,"5min":1.478,"15min":1.212},"min":{"1min":0.73199999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":16.896000000000001,"5min":16.896000000000001,"15min":16.896000000000001},"last":1.591}]},{"osd":1,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.482,"15min":1.353},"min":{"1min":0.77500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":3.4550000000000001,"5min":1.7549999999999999,"15min":1.345},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":35.328000000000003,"5min":35.328000000000003,"15min":35.328000000000003},"last":1.554}]},{"osd":2,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.7639999999999998,"5min":1.8859999999999999,"15min":1.47},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":35.398000000000003,"5min":35.398000000000003,"15min":35.398000000000003},"last":1.091},{"interface":"front","average":{"1min":1.782,"5min":1.3340000000000001,"15min":1.29},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.883,"5min":4.883,"15min":4.883},"last":1.7150000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8210000000000002,"5min":1.667,"15min":1.4430000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":13.202,"5min":13.202,"15min":13.202},"last":1.4810000000000001},{"interface":"front","average":{"1min":3.6669999999999998,"5min":1.9670000000000001,"15min":1.587},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":35.268000000000001,"5min":35.268000000000001,"15min":35.268000000000001},"last":1.6799999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.5470000000000002,"5min":1.784,"15min":1.579},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":15.757,"5min":15.757,"15min":15.757},"last":1.0069999999999999},{"interface":"front","average":{"1min":4.3479999999999999,"5min":2.004,"15min":1.714},"min":{"1min":0.63300000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":26.166,"5min":26.166,"15min":26.166},"last":1.7709999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":4.0350000000000001,"5min":2.0950000000000002,"15min":1.768},"min":{"1min":0.53400000000000003,"5min":0.499,"15min":0.499},"max":{"1min":35.427999999999997,"5min":35.427999999999997,"15min":35.427999999999997},"last":1.7410000000000001},{"interface":"front","average":{"1min":3.4740000000000002,"5min":1.903,"15min":1.7589999999999999},"min":{"1min":0.53100000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":1.6259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.9350000000000001,"5min":2.0379999999999998,"15min":1.7689999999999999},"min":{"1min":0.58099999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":19.364999999999998,"5min":19.364999999999998,"15min":19.364999999999998},"last":1.837},{"interface":"front","average":{"1min":4.0419999999999998,"5min":2.1829999999999998,"15min":1.8839999999999999},"min":{"1min":0.71699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":35.363,"5min":35.363,"15min":35.363},"last":1.994}]}]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.1779999999999999,"15min":1.1080000000000001},"min":{"1min":0.751,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":4.2869999999999999,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":1.012},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.4059999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":8.0440000000000005,"5min":8.0440000000000005,"15min":8.0440000000000005},"last":0.90300000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.4359999999999999,"15min":1.258},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.1050000000000004,"5min":7.1050000000000004,"15min":7.1050000000000004},"last":1.391},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.2889999999999999,"15min":1.234},"min":{"1min":0.84499999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":5.7290000000000001,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":1.137}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.583,"15min":1.415},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.4860000000000002,"5min":15.986000000000001,"15min":15.986000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.554,"5min":1.3520000000000001,"15min":1.4330000000000001},"min":{"1min":0.83699999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":3.762,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":1.232}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":3.0049999999999999,"5min":1.871,"15min":1.6819999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5600000000000001,"15min":1.48},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":4.0490000000000004,"5min":21.706,"15min":21.706},"last":1.538}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4370000000000001,"15min":1.575},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":4.327,"5min":4.327,"15min":4.327},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.649,"15min":1.6279999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":10.831,"5min":10.831,"15min":10.831},"last":1.504}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4530000000000001,"15min":1.575},"min":{"1min":0.66000000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":1.476},{"interface":"front","average":{"1min":2.468,"5min":1.734,"15min":1.732},"min":{"1min":0.74199999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":20.939,"5min":20.939,"15min":20.939},"last":1.4399999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.8,"15min":1.7490000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":7.0110000000000001,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":1.591},{"interface":"front","average":{"1min":1.976,"5min":1.6399999999999999,"15min":1.738},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":10.978999999999999,"5min":10.978999999999999,"15min":10.978999999999999},"last":1.6279999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.98499999999999999,"15min":0.91900000000000004},"min":{"1min":0.433,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1459999999999999,"15min":0.93600000000000005},"min":{"1min":0.56799999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":1.099}]},{"osd":1,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.92500000000000004,"15min":0.85999999999999999},"min":{"1min":0.47099999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.752,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.121,"5min":1.0620000000000001,"15min":1.016},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.4590000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.075,"15min":1.075},"min":{"1min":0.59299999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9339999999999999,"5min":4.008,"15min":4.008},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.0820000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.2069999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.2929999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.3500000000000001,"15min":1.1639999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":7.7439999999999998,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.492,"15min":1.3129999999999999},"min":{"1min":0.80500000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":11.068,"5min":11.068,"15min":11.068},"last":1.208}]},{"osd":5,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4630000000000001,"15min":1.3180000000000001},"min":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":12.791,"5min":12.791,"15min":12.791},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.508,"15min":1.413},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":12.106,"5min":12.106,"15min":12.106},"last":1.5389999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":1.546,"15min":1.3540000000000001},"min":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":8.1039999999999992,"5min":8.1039999999999992,"15min":8.1039999999999992},"last":1.502},{"interface":"front","average":{"1min":2.3980000000000001,"5min":1.6359999999999999,"15min":1.4379999999999999},"min":{"1min":0.91900000000000004,"5min":0.752,"15min":0.752},"max":{"1min":6.891,"5min":6.891,"15min":6.891},"last":1.4890000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.532,"15min":1.4379999999999999},"min":{"1min":0.82599999999999996,"5min":0.755,"15min":0.755},"max":{"1min":3.7269999999999999,"5min":3.7269999999999999,"15min":3.7269999999999999},"last":1.599},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.677,"15min":1.4299999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":7.6159999999999997,"5min":18.846,"15min":18.846},"last":1.5640000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":0.91600000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.839,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.378},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0049999999999999,"15min":0.93600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.353,"5min":4.5010000000000003,"15min":4.5010000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0609999999999999,"15min":1.0329999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.133,"5min":3.798,"15min":3.798},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.125,"15min":1.014},"min":{"1min":0.45800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.8069999999999999,"5min":4.8250000000000002,"15min":4.8250000000000002},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0620000000000001,"15min":0.97999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.9870000000000001,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.224,"15min":1.1399999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":8.4610000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.5489999999999999,"15min":1.3939999999999999},"min":{"1min":0.67700000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":11.843,"5min":11.843,"15min":11.843},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.847,"5min":1.54,"15min":1.411},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":11.768000000000001,"5min":11.768000000000001,"15min":11.768000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":1.72,"15min":1.4470000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":26.501000000000001,"5min":26.501000000000001,"15min":26.501000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.389,"15min":1.341},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.8640000000000001,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.593,"15min":1.4430000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.5059999999999998,"5min":15.144,"15min":15.144},"last":0.56799999999999995},{"interface":"front","average":{"1min":5.2089999999999996,"5min":2.214,"15min":1.6599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":66.061999999999998,"5min":66.061999999999998,"15min":66.061999999999998},"last":0.63300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.581,"15min":1.5049999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":4.0750000000000002,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.554,"5min":1.5509999999999999,"15min":1.4850000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":4.0519999999999996,"5min":6.0590000000000002,"15min":6.0590000000000002},"last":0.5}]}]},{"osd":1,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.92800000000000005,"15min":0.93000000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.089,"5min":1.026,"15min":1.0149999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":1.298}]},{"osd":2,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.0369999999999999,"15min":0.95499999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":14.817,"5min":14.817,"15min":14.817},"last":0.996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.028,"15min":0.98299999999999998},"min":{"1min":0.47199999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.3460000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.075,"15min":1.071},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.8680000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.1839999999999999,"15min":1.0640000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":20.439,"5min":20.439,"15min":20.439},"last":1.0469999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.4359999999999999,"15min":1.256},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":13.641999999999999,"5min":13.641999999999999,"15min":13.641999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.4670000000000001,"15min":1.3260000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":13.789999999999999,"5min":13.789999999999999,"15min":13.789999999999999},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.516,"15min":1.4419999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":14.129,"5min":14.129,"15min":14.129},"last":1.377},{"interface":"front","average":{"1min":3.399,"5min":1.8160000000000001,"15min":1.5609999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":35.747,"5min":35.747,"15min":35.747},"last":1.466}]},{"osd":6,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9359999999999999,"5min":1.5669999999999999,"15min":1.4810000000000001},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":11.545,"5min":11.545,"15min":11.545},"last":1.5920000000000001},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.5600000000000001,"15min":1.4690000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":13.56,"5min":13.56,"15min":13.56},"last":1.101}]},{"osd":7,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.629,"15min":1.4770000000000001},"min":{"1min":0.70699999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":6.2969999999999997,"5min":12.494999999999999,"15min":12.494999999999999},"last":1.405},{"interface":"front","average":{"1min":2.7120000000000002,"5min":1.6970000000000001,"15min":1.524},"min":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":31.044,"5min":31.044,"15min":31.044},"last":1.6870000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.98399999999999999,"15min":0.97599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.4379999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.2230000000000001,"15min":1.0569999999999999},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":4.3550000000000004,"5min":17.876000000000001,"15min":17.876000000000001},"last":0.79700000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.357,"15min":1.1839999999999999},"min":{"1min":0.54200000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":5.9080000000000004,"5min":17.760999999999999,"15min":17.760999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.226,"15min":1.232},"min":{"1min":0.56100000000000005,"5min":0.38,"15min":0.38},"max":{"1min":1.9139999999999999,"5min":5.5970000000000004,"15min":5.5970000000000004},"last":0.875}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.401,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.3520000000000001,"15min":1.137},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.4169999999999998,"5min":17.981000000000002,"15min":17.981000000000002},"last":0.95599999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.6040000000000001,"15min":1.4870000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":7.0389999999999997,"5min":14.297000000000001,"15min":14.297000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":2.9910000000000001,"5min":1.8939999999999999,"15min":1.4830000000000001},"min":{"1min":0.83999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":18.347999999999999,"5min":19.422000000000001,"15min":19.422000000000001},"last":1.1919999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":1.716,"15min":1.5209999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":10.933,"5min":14.212999999999999,"15min":14.212999999999999},"last":1.169},{"interface":"front","average":{"1min":2.319,"5min":1.79,"15min":1.452},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":16.446999999999999,"5min":19.315000000000001,"15min":19.315000000000001},"last":1.292}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":4.1150000000000002,"5min":2.2349999999999999,"15min":1.677},"min":{"1min":0.81499999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":27.442,"5min":27.442,"15min":27.442},"last":1.321},{"interface":"front","average":{"1min":3.9689999999999999,"5min":2.1800000000000002,"15min":1.7629999999999999},"min":{"1min":0.70899999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":27.411999999999999,"5min":27.411999999999999,"15min":27.411999999999999},"last":1.3440000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":1.8819999999999999,"15min":1.5640000000000001},"min":{"1min":0.78700000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":7.4130000000000003,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.6819999999999999,"15min":1.6259999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":5.335,"5min":5.9269999999999996,"15min":5.9269999999999996},"last":1.262}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3f is not active+clean 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.691258+0000', 'last_change': '2024-01-16T15:02:28.691258+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.691258+0000', 'last_undegraded': '2024-01-16T15:02:28.691258+0000', 'last_fullsized': '2024-01-16T15:02:28.691258+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.674881+0000', 'last_change': '2024-01-16T15:02:28.674881+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.674881+0000', 'last_undegraded': '2024-01-16T15:02:28.674881+0000', 'last_fullsized': '2024-01-16T15:02:28.674881+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3d is not active+clean 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.685084+0000', 'last_change': '2024-01-16T15:02:28.685084+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.685084+0000', 'last_undegraded': '2024-01-16T15:02:28.685084+0000', 'last_fullsized': '2024-01-16T15:02:28.685084+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-01-16T15:02:35.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.682154+0000', 'last_change': '2024-01-16T15:02:28.682154+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.682154+0000', 'last_undegraded': '2024-01-16T15:02:28.682154+0000', 'last_fullsized': '2024-01-16T15:02:28.682154+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3b is not active+clean 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.687284+0000', 'last_change': '2024-01-16T15:02:28.687284+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.687284+0000', 'last_undegraded': '2024-01-16T15:02:28.687284+0000', 'last_fullsized': '2024-01-16T15:02:28.687284+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.680550+0000', 'last_change': '2024-01-16T15:02:28.680550+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.680550+0000', 'last_undegraded': '2024-01-16T15:02:28.680550+0000', 'last_fullsized': '2024-01-16T15:02:28.680550+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.677104+0000', 'last_change': '2024-01-16T15:02:28.677104+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.677104+0000', 'last_undegraded': '2024-01-16T15:02:28.677104+0000', 'last_fullsized': '2024-01-16T15:02:28.677104+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.677904+0000', 'last_change': '2024-01-16T15:02:28.677904+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.677904+0000', 'last_undegraded': '2024-01-16T15:02:28.677904+0000', 'last_fullsized': '2024-01-16T15:02:28.677904+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.671452+0000', 'last_change': '2024-01-16T15:02:28.671452+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.671452+0000', 'last_undegraded': '2024-01-16T15:02:28.671452+0000', 'last_fullsized': '2024-01-16T15:02:28.671452+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:02:35.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.681639+0000', 'last_change': '2024-01-16T15:02:28.681639+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.681639+0000', 'last_undegraded': '2024-01-16T15:02:28.681639+0000', 'last_fullsized': '2024-01-16T15:02:28.681639+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.677387+0000', 'last_change': '2024-01-16T15:02:28.677387+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.677387+0000', 'last_undegraded': '2024-01-16T15:02:28.677387+0000', 'last_fullsized': '2024-01-16T15:02:28.677387+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.671228+0000', 'last_change': '2024-01-16T15:02:28.671228+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.671228+0000', 'last_undegraded': '2024-01-16T15:02:28.671228+0000', 'last_fullsized': '2024-01-16T15:02:28.671228+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.679199+0000', 'last_change': '2024-01-16T15:02:28.679199+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.679199+0000', 'last_undegraded': '2024-01-16T15:02:28.679199+0000', 'last_fullsized': '2024-01-16T15:02:28.679199+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.674612+0000', 'last_change': '2024-01-16T15:02:28.674612+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.674612+0000', 'last_undegraded': '2024-01-16T15:02:28.674612+0000', 'last_fullsized': '2024-01-16T15:02:28.674612+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2e is not active+clean 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.688365+0000', 'last_change': '2024-01-16T15:02:28.688365+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.688365+0000', 'last_undegraded': '2024-01-16T15:02:28.688365+0000', 'last_fullsized': '2024-01-16T15:02:28.688365+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2d is not active+clean 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.684341+0000', 'last_change': '2024-01-16T15:02:28.684341+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.684341+0000', 'last_undegraded': '2024-01-16T15:02:28.684341+0000', 'last_fullsized': '2024-01-16T15:02:28.684341+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.677483+0000', 'last_change': '2024-01-16T15:02:28.677483+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.677483+0000', 'last_undegraded': '2024-01-16T15:02:28.677483+0000', 'last_fullsized': '2024-01-16T15:02:28.677483+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2b is not active+clean 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.680887+0000', 'last_change': '2024-01-16T15:02:28.680887+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.680887+0000', 'last_undegraded': '2024-01-16T15:02:28.680887+0000', 'last_fullsized': '2024-01-16T15:02:28.680887+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.675926+0000', 'last_change': '2024-01-16T15:02:28.675926+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.675926+0000', 'last_undegraded': '2024-01-16T15:02:28.675926+0000', 'last_fullsized': '2024-01-16T15:02:28.675926+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.29 is not active+clean 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.680236+0000', 'last_change': '2024-01-16T15:02:28.680236+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.680236+0000', 'last_undegraded': '2024-01-16T15:02:28.680236+0000', 'last_fullsized': '2024-01-16T15:02:28.680236+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.27 is not active+clean 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.690190+0000', 'last_change': '2024-01-16T15:02:28.690190+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.690190+0000', 'last_undegraded': '2024-01-16T15:02:28.690190+0000', 'last_fullsized': '2024-01-16T15:02:28.690190+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.683203+0000', 'last_change': '2024-01-16T15:02:28.683203+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.683203+0000', 'last_undegraded': '2024-01-16T15:02:28.683203+0000', 'last_fullsized': '2024-01-16T15:02:28.683203+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.678571+0000', 'last_change': '2024-01-16T15:02:28.678571+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.678571+0000', 'last_undegraded': '2024-01-16T15:02:28.678571+0000', 'last_fullsized': '2024-01-16T15:02:28.678571+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.24 is not active+clean 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.687166+0000', 'last_change': '2024-01-16T15:02:28.687166+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.687166+0000', 'last_undegraded': '2024-01-16T15:02:28.687166+0000', 'last_fullsized': '2024-01-16T15:02:28.687166+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:02:35.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.674298+0000', 'last_change': '2024-01-16T15:02:28.674298+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.674298+0000', 'last_undegraded': '2024-01-16T15:02:28.674298+0000', 'last_fullsized': '2024-01-16T15:02:28.674298+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.22 is not active+clean 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.682909+0000', 'last_change': '2024-01-16T15:02:28.682909+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.682909+0000', 'last_undegraded': '2024-01-16T15:02:28.682909+0000', 'last_fullsized': '2024-01-16T15:02:28.682909+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.21 is not active+clean 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.683833+0000', 'last_change': '2024-01-16T15:02:28.683833+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.683833+0000', 'last_undegraded': '2024-01-16T15:02:28.683833+0000', 'last_fullsized': '2024-01-16T15:02:28.683833+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.675958+0000', 'last_change': '2024-01-16T15:02:28.675958+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.675958+0000', 'last_undegraded': '2024-01-16T15:02:28.675958+0000', 'last_fullsized': '2024-01-16T15:02:28.675958+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.673472+0000', 'last_change': '2024-01-16T15:02:28.673472+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.673472+0000', 'last_undegraded': '2024-01-16T15:02:28.673472+0000', 'last_fullsized': '2024-01-16T15:02:28.673472+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-01-16T15:02:35.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.672137+0000', 'last_change': '2024-01-16T15:02:28.672137+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.672137+0000', 'last_undegraded': '2024-01-16T15:02:28.672137+0000', 'last_fullsized': '2024-01-16T15:02:28.672137+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.673332+0000', 'last_change': '2024-01-16T15:02:28.673332+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.673332+0000', 'last_undegraded': '2024-01-16T15:02:28.673332+0000', 'last_fullsized': '2024-01-16T15:02:28.673332+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.675366+0000', 'last_change': '2024-01-16T15:02:28.675366+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.675366+0000', 'last_undegraded': '2024-01-16T15:02:28.675366+0000', 'last_fullsized': '2024-01-16T15:02:28.675366+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.676930+0000', 'last_change': '2024-01-16T15:02:28.676930+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.676930+0000', 'last_undegraded': '2024-01-16T15:02:28.676930+0000', 'last_fullsized': '2024-01-16T15:02:28.676930+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.681431+0000', 'last_change': '2024-01-16T15:02:28.681431+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.681431+0000', 'last_undegraded': '2024-01-16T15:02:28.681431+0000', 'last_fullsized': '2024-01-16T15:02:28.681431+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-01-16T15:02:35.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.672957+0000', 'last_change': '2024-01-16T15:02:28.672957+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.672957+0000', 'last_undegraded': '2024-01-16T15:02:28.672957+0000', 'last_fullsized': '2024-01-16T15:02:28.672957+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.686150+0000', 'last_change': '2024-01-16T15:02:28.686150+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.686150+0000', 'last_undegraded': '2024-01-16T15:02:28.686150+0000', 'last_fullsized': '2024-01-16T15:02:28.686150+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.684658+0000', 'last_change': '2024-01-16T15:02:28.684658+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.684658+0000', 'last_undegraded': '2024-01-16T15:02:28.684658+0000', 'last_fullsized': '2024-01-16T15:02:28.684658+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.683793+0000', 'last_change': '2024-01-16T15:02:28.683793+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.683793+0000', 'last_undegraded': '2024-01-16T15:02:28.683793+0000', 'last_fullsized': '2024-01-16T15:02:28.683793+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.680264+0000', 'last_change': '2024-01-16T15:02:28.680264+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.680264+0000', 'last_undegraded': '2024-01-16T15:02:28.680264+0000', 'last_fullsized': '2024-01-16T15:02:28.680264+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.673658+0000', 'last_change': '2024-01-16T15:02:28.673658+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.673658+0000', 'last_undegraded': '2024-01-16T15:02:28.673658+0000', 'last_fullsized': '2024-01-16T15:02:28.673658+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-01-16T15:02:35.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.678221+0000', 'last_change': '2024-01-16T15:02:28.678221+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.678221+0000', 'last_undegraded': '2024-01-16T15:02:28.678221+0000', 'last_fullsized': '2024-01-16T15:02:28.678221+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.687323+0000', 'last_change': '2024-01-16T15:02:28.687323+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.687323+0000', 'last_undegraded': '2024-01-16T15:02:28.687323+0000', 'last_fullsized': '2024-01-16T15:02:28.687323+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.684228+0000', 'last_change': '2024-01-16T15:02:28.684228+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.684228+0000', 'last_undegraded': '2024-01-16T15:02:28.684228+0000', 'last_fullsized': '2024-01-16T15:02:28.684228+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.676992+0000', 'last_change': '2024-01-16T15:02:28.676992+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.676992+0000', 'last_undegraded': '2024-01-16T15:02:28.676992+0000', 'last_fullsized': '2024-01-16T15:02:28.676992+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.679870+0000', 'last_change': '2024-01-16T15:02:28.679870+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.679870+0000', 'last_undegraded': '2024-01-16T15:02:28.679870+0000', 'last_fullsized': '2024-01-16T15:02:28.679870+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.683284+0000', 'last_change': '2024-01-16T15:02:28.683284+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.683284+0000', 'last_undegraded': '2024-01-16T15:02:28.683284+0000', 'last_fullsized': '2024-01-16T15:02:28.683284+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.672340+0000', 'last_change': '2024-01-16T15:02:28.672340+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.672340+0000', 'last_undegraded': '2024-01-16T15:02:28.672340+0000', 'last_fullsized': '2024-01-16T15:02:28.672340+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.672000+0000', 'last_change': '2024-01-16T15:02:28.672000+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.672000+0000', 'last_undegraded': '2024-01-16T15:02:28.672000+0000', 'last_fullsized': '2024-01-16T15:02:28.672000+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.681494+0000', 'last_change': '2024-01-16T15:02:28.681494+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.681494+0000', 'last_undegraded': '2024-01-16T15:02:28.681494+0000', 'last_fullsized': '2024-01-16T15:02:28.681494+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.683770+0000', 'last_change': '2024-01-16T15:02:28.683770+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.683770+0000', 'last_undegraded': '2024-01-16T15:02:28.683770+0000', 'last_fullsized': '2024-01-16T15:02:28.683770+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.671790+0000', 'last_change': '2024-01-16T15:02:28.671790+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.671790+0000', 'last_undegraded': '2024-01-16T15:02:28.671790+0000', 'last_fullsized': '2024-01-16T15:02:28.671790+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-01-16T15:02:35.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.674849+0000', 'last_change': '2024-01-16T15:02:28.674849+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.674849+0000', 'last_undegraded': '2024-01-16T15:02:28.674849+0000', 'last_fullsized': '2024-01-16T15:02:28.674849+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-01-16T15:02:35.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.679260+0000', 'last_change': '2024-01-16T15:02:28.679260+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.679260+0000', 'last_undegraded': '2024-01-16T15:02:28.679260+0000', 'last_fullsized': '2024-01-16T15:02:28.679260+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:02:35.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-01-16T15:02:35.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.674276+0000', 'last_change': '2024-01-16T15:02:28.674276+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.674276+0000', 'last_undegraded': '2024-01-16T15:02:28.674276+0000', 'last_fullsized': '2024-01-16T15:02:28.674276+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-01-16T15:02:35.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:02:28.675396+0000', 'last_change': '2024-01-16T15:02:28.675396+0000', 'last_active': '2024-01-16T15:02:28.654459+0000', 'last_peered': '2024-01-16T15:02:28.654459+0000', 'last_clean': '2024-01-16T15:02:28.654459+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:02:28.675396+0000', 'last_undegraded': '2024-01-16T15:02:28.675396+0000', 'last_fullsized': '2024-01-16T15:02:28.675396+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'last_clean_scrub_stamp': '2024-01-16T15:02:28.654459+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:02:35.096 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-16T15:02:35.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.349+0000 7fe2d2e00640 1 Processor -- start 2024-01-16T15:02:35.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.350+0000 7fe2d2e00640 1 -- start start 2024-01-16T15:02:35.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.351+0000 7fe2d2e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0cd240 0x7fe2cc0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.351+0000 7fe2d2e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2cc09a170 0x7fe2cc0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.351+0000 7fe2d2e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc06cea0 con 0x7fe2cc06c6b0 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.351+0000 7fe2d2e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc06cbc0 con 0x7fe2cc09a170 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.351+0000 7fe2d2e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc06f350 con 0x7fe2cc0cd240 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.351+0000 7fe2d0e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2cc09a170 0x7fe2cc0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2d0e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2cc09a170 0x7fe2cc0cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59798/0 (socket says 172.21.15.141:59798) 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2cb400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe2cc06c6b0 0x7fe2cc099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41666/0 (socket says 172.21.15.141:41666) 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2d0e00640 1 -- 172.21.15.141:0/1598431228 learned_addr learned my addr 172.21.15.141:0/1598431228 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2cbe00640 1 --2- 172.21.15.141:0/1598431228 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0cd240 0x7fe2cc0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2d0e00640 1 -- 172.21.15.141:0/1598431228 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0cd240 msgr2=0x7fe2cc0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2d0e00640 1 --2- 172.21.15.141:0/1598431228 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0cd240 0x7fe2cc0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2d0e00640 1 -- 172.21.15.141:0/1598431228 >> v1:172.21.15.141:6789/0 conn(0x7fe2cc06c6b0 legacy=0x7fe2cc099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.352+0000 7fe2d0e00640 1 -- 172.21.15.141:0/1598431228 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc06f630 con 0x7fe2cc09a170 2024-01-16T15:02:35.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.353+0000 7fe2d0e00640 1 --2- 172.21.15.141:0/1598431228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc09a170 0x7fe2cc0cf640 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009e50 tx=0x7fe2c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331de56b930b0b4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.353+0000 7fe2caa00640 1 -- 172.21.15.141:0/1598431228 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c402cb00 con 0x7fe2cc09a170 2024-01-16T15:02:35.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.353+0000 7fe2caa00640 1 -- 172.21.15.141:0/1598431228 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c402cc60 con 0x7fe2cc09a170 2024-01-16T15:02:35.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.353+0000 7fe2caa00640 1 -- 172.21.15.141:0/1598431228 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40343f0 con 0x7fe2cc09a170 2024-01-16T15:02:35.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 -- 172.21.15.141:0/1598431228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc09a170 msgr2=0x7fe2cc0cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/1598431228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc09a170 0x7fe2cc0cf640 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009e50 tx=0x7fe2c402bf60 comp rx=0 tx=0).stop 2024-01-16T15:02:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 -- 172.21.15.141:0/1598431228 shutdown_connections 2024-01-16T15:02:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/1598431228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc09a170 0x7fe2cc0cf640 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/1598431228 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc0cd240 0x7fe2cc0b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 -- 172.21.15.141:0/1598431228 >> 172.21.15.141:0/1598431228 conn(0x7fe2cc0b2d00 msgr2=0x7fe2cc0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 -- 172.21.15.141:0/1598431228 shutdown_connections 2024-01-16T15:02:35.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.354+0000 7fe2d2e00640 1 -- 172.21.15.141:0/1598431228 wait complete. 2024-01-16T15:02:35.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.356+0000 7fe2d2e00640 1 Processor -- start 2024-01-16T15:02:35.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.356+0000 7fe2d2e00640 1 -- start start 2024-01-16T15:02:35.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d2e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc06c6b0 0x7fe2cc15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d2e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc09a170 0x7fe2cc0b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d2e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0cd240 0x7fe2cc0cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d2e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c9a50 con 0x7fe2cc0cd240 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d2e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c9d30 con 0x7fe2cc06c6b0 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d2e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0ca010 con 0x7fe2cc09a170 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0cd240 0x7fe2cc0cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0cd240 0x7fe2cc0cfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59804/0 (socket says 172.21.15.141:59804) 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.357+0000 7fe2d0e00640 1 -- 172.21.15.141:0/3376747394 learned_addr learned my addr 172.21.15.141:0/3376747394 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2cbe00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc06c6b0 0x7fe2cc15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2cb400640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc09a170 0x7fe2cc0b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2d0e00640 1 -- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc09a170 msgr2=0x7fe2cc0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2d0e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc09a170 0x7fe2cc0b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2d0e00640 1 -- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc06c6b0 msgr2=0x7fe2cc15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2d0e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc06c6b0 0x7fe2cc15aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2d0e00640 1 -- 172.21.15.141:0/3376747394 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c4009b30 con 0x7fe2cc0cd240 2024-01-16T15:02:35.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2d0e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0cd240 0x7fe2cc0cfee0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009e50 tx=0x7fe2c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2c3e00640 1 -- 172.21.15.141:0/3376747394 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4009510 con 0x7fe2cc0cd240 2024-01-16T15:02:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2c3e00640 1 -- 172.21.15.141:0/3376747394 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c403e450 con 0x7fe2cc0cd240 2024-01-16T15:02:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.358+0000 7fe2c3e00640 1 -- 172.21.15.141:0/3376747394 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c403e800 con 0x7fe2cc0cd240 2024-01-16T15:02:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.359+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc0ca2f0 con 0x7fe2cc0cd240 2024-01-16T15:02:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.359+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc0ca8b0 con 0x7fe2cc0cd240 2024-01-16T15:02:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.360+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc15b7b0 con 0x7fe2cc0cd240 2024-01-16T15:02:35.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.361+0000 7fe2c3e00640 1 -- 172.21.15.141:0/3376747394 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe2c403e960 con 0x7fe2cc0cd240 2024-01-16T15:02:35.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.362+0000 7fe2c3e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.362+0000 7fe2c3e00640 1 -- 172.21.15.141:0/3376747394 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2c40ad4d0 con 0x7fe2cc0cd240 2024-01-16T15:02:35.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.362+0000 7fe2cbe00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.363+0000 7fe2cbe00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc006fd0 tx=0x7fe2bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.366+0000 7fe2c3e00640 1 -- 172.21.15.141:0/3376747394 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2c4077a00 con 0x7fe2cc0cd240 2024-01-16T15:02:35.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.629+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe2cc0675d0 con 0x7fe2cc0cd240 2024-01-16T15:02:35.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.630+0000 7fe2c3e00640 1 -- 172.21.15.141:0/3376747394 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1094 (secure 0 0 0) 0x7fe2c407b020 con 0x7fe2cc0cd240 2024-01-16T15:02:35.635 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:35.635 INFO:teuthology.orchestra.run.smithi141.stdout:{"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":348,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1705417019,"num_in_osds":8,"osd_in_since":1705417011,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":55},{"state_name":"active+clean","count":42}],"num_pgs":97,"num_pools":3,"num_objects":3,"data_bytes":590368,"bytes_used":296157184,"bytes_avail":772797956096,"bytes_total":773094113280,"inactive_pgs_ratio":0.56701028347015381,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":37,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"a","status":"up:active","gid":6360}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-01-16T15:01:14.527633+0000","services":{}},"progress_events":{}} 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 msgr2=0x7fe2a406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc006fd0 tx=0x7fe2bc008040 comp rx=0 tx=0).stop 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0cd240 msgr2=0x7fe2cc0cfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0cd240 0x7fe2cc0cfee0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009e50 tx=0x7fe2c4004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 shutdown_connections 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2a406b860 0x7fe2a406dd20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2cc0cd240 0x7fe2cc0cfee0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2cc09a170 0x7fe2cc0b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 --2- 172.21.15.141:0/3376747394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2cc06c6b0 0x7fe2cc15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 >> 172.21.15.141:0/3376747394 conn(0x7fe2cc0b2d00 msgr2=0x7fe2cc099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 shutdown_connections 2024-01-16T15:02:35.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.633+0000 7fe2d2e00640 1 -- 172.21.15.141:0/3376747394 wait complete. 2024-01-16T15:02:35.638 INFO:tasks.ceph.mds.a.smithi141.stderr:2024-01-16T15:02:35.648+0000 7f3bfce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:02:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-16T15:02:35.639 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:35.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.868+0000 7f53e9a00640 1 Processor -- start 2024-01-16T15:02:35.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.868+0000 7f53e9a00640 1 -- start start 2024-01-16T15:02:35.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53e9a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f53e415da10 0x7f53e4072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53e9a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e40b3730 con 0x7f53e4150760 2024-01-16T15:02:35.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53e9a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e4071190 con 0x7f53e40b2bd0 2024-01-16T15:02:35.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53e9a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e4071470 con 0x7f53e415da10 2024-01-16T15:02:35.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53e2a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f53e4150760 0x7f53e415d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41672/0 (socket says 172.21.15.141:41672) 2024-01-16T15:02:35.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53e2a00640 1 -- 172.21.15.141:0/2125019710 learned_addr learned my addr 172.21.15.141:0/2125019710 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:35.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53d9e00640 1 --2- 172.21.15.141:0/2125019710 >> v2:172.21.15.141:3300/0 conn(0x7f53e415da10 0x7f53e4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.870+0000 7f53e2000640 1 -- 172.21.15.141:0/2125019710 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395704827 0 0) 0x7f53e40b3730 con 0x7f53e4150760 2024-01-16T15:02:35.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.871+0000 7f53d9e00640 1 -- 172.21.15.141:0/2125019710 >> v1:172.21.15.188:6789/0 conn(0x7f53e40b2bd0 legacy=0x7f53e40b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:35.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.871+0000 7f53d9e00640 1 -- 172.21.15.141:0/2125019710 >> v1:172.21.15.141:6789/0 conn(0x7f53e4150760 legacy=0x7f53e415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.871+0000 7f53d9e00640 1 -- 172.21.15.141:0/2125019710 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e4071750 con 0x7f53e415da10 2024-01-16T15:02:35.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53d9e00640 1 --2- 172.21.15.141:0/2125019710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e4072230 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009d90 tx=0x7f53cc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f5db19b032c185 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e2000640 1 -- 172.21.15.141:0/2125019710 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc02cc60 con 0x7f53e415da10 2024-01-16T15:02:35.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e2000640 1 -- 172.21.15.141:0/2125019710 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53cc02cdc0 con 0x7f53e415da10 2024-01-16T15:02:35.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e2000640 1 -- 172.21.15.141:0/2125019710 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc034660 con 0x7f53e415da10 2024-01-16T15:02:35.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e9a00640 1 -- 172.21.15.141:0/2125019710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 msgr2=0x7f53e4072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e9a00640 1 --2- 172.21.15.141:0/2125019710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e4072230 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009d90 tx=0x7f53cc02beb0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e9a00640 1 -- 172.21.15.141:0/2125019710 shutdown_connections 2024-01-16T15:02:35.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e9a00640 1 --2- 172.21.15.141:0/2125019710 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e4072230 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.872+0000 7f53e9a00640 1 -- 172.21.15.141:0/2125019710 >> 172.21.15.141:0/2125019710 conn(0x7f53e406a800 msgr2=0x7f53e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.873+0000 7f53e9a00640 1 -- 172.21.15.141:0/2125019710 shutdown_connections 2024-01-16T15:02:35.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.873+0000 7f53e9a00640 1 -- 172.21.15.141:0/2125019710 wait complete. 2024-01-16T15:02:35.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.874+0000 7f53e9a00640 1 Processor -- start 2024-01-16T15:02:35.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.874+0000 7f53e9a00640 1 -- start start 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.875+0000 7f53e9a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53e40b2bd0 0x7f53e4157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.875+0000 7f53e9a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53e4150760 0x7f53e4145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.875+0000 7f53e9a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e40adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.875+0000 7f53e9a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40ceba0 con 0x7f53e415da10 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.875+0000 7f53e9a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40cdf90 con 0x7f53e4150760 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.875+0000 7f53e9a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40ce270 con 0x7f53e40b2bd0 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.875+0000 7f53e3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e40adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e40adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59826/0 (socket says 172.21.15.141:59826) 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 -- 172.21.15.141:0/143605586 learned_addr learned my addr 172.21.15.141:0/143605586 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:35.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e2a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53e40b2bd0 0x7f53e4157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53d9e00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53e4150760 0x7f53e4145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 -- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53e40b2bd0 msgr2=0x7f53e4157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53e40b2bd0 0x7f53e4157880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 -- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53e4150760 msgr2=0x7f53e4145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53e4150760 0x7f53e4145850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 -- 172.21.15.141:0/143605586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc009a70 con 0x7f53e415da10 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53e3400640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e40adb80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c410 tx=0x7f53dc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53db400640 1 -- 172.21.15.141:0/143605586 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc011790 con 0x7f53e415da10 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53db400640 1 -- 172.21.15.141:0/143605586 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc0118f0 con 0x7f53e415da10 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.876+0000 7f53db400640 1 -- 172.21.15.141:0/143605586 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc011ce0 con 0x7f53e415da10 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.877+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e40ce550 con 0x7f53e415da10 2024-01-16T15:02:35.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.877+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e4145d90 con 0x7f53e415da10 2024-01-16T15:02:35.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.878+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e40675d0 con 0x7f53e415da10 2024-01-16T15:02:35.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.879+0000 7f53db400640 1 -- 172.21.15.141:0/143605586 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f53dc01c460 con 0x7f53e415da10 2024-01-16T15:02:35.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.879+0000 7f53db400640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53bc06b890 0x7f53bc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.879+0000 7f53db400640 1 -- 172.21.15.141:0/143605586 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53dc08a2c0 con 0x7f53e415da10 2024-01-16T15:02:35.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.880+0000 7f53e2a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53bc06b890 0x7f53bc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.881+0000 7f53e2a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53bc06b890 0x7f53bc06dd50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f53d4007fd0 tx=0x7f53d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:35.884+0000 7f53db400640 1 -- 172.21.15.141:0/143605586 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53dc0548c0 con 0x7f53e415da10 2024-01-16T15:02:36.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.068+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53e4070d70 con 0x7f53bc06b890 2024-01-16T15:02:36.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.090+0000 7f53db400640 1 -- 172.21.15.141:0/143605586 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282293 (secure 0 0 0) 0x7f53e4070d70 con 0x7f53bc06b890 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:36.076 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.096+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53bc06b890 msgr2=0x7f53bc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.096+0000 7f53e9a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53bc06b890 0x7f53bc06dd50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f53d4007fd0 tx=0x7f53d4009040 comp rx=0 tx=0).stop 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.096+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 msgr2=0x7f53e40adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.096+0000 7f53e9a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e40adb80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c410 tx=0x7f53dc00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.096+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 shutdown_connections 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.097+0000 7f53e9a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53bc06b890 0x7f53bc06dd50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.097+0000 7f53e9a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53e415da10 0x7f53e40adb80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.097+0000 7f53e9a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53e4150760 0x7f53e4145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.097+0000 7f53e9a00640 1 --2- 172.21.15.141:0/143605586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f53e40b2bd0 0x7f53e4157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.097+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 >> 172.21.15.141:0/143605586 conn(0x7f53e406a800 msgr2=0x7f53e406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:36.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.097+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 shutdown_connections 2024-01-16T15:02:36.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:36.097+0000 7f53e9a00640 1 -- 172.21.15.141:0/143605586 wait complete. 2024-01-16T15:02:36.097 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-01-16T15:02:34.586289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289216,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685504,"statfs":{"total":773094113280,"available":772797956096,"internally_reserved":0,"allocated":6627328,"data_stored":3419944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.011975"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.691258+0000","last_change":"2024-01-16T15:02:28.691258+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.691258+0000","last_undegraded":"2024-01-16T15:02:28.691258+0000","last_fullsized":"2024-01-16T15:02:28.691258+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674881+0000","last_change":"2024-01-16T15:02:28.674881+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674881+0000","last_undegraded":"2024-01-16T15:02:28.674881+0000","last_fullsized":"2024-01-16T15:02:28.674881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.685084+0000","last_change":"2024-01-16T15:02:28.685084+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.685084+0000","last_undegraded":"2024-01-16T15:02:28.685084+0000","last_fullsized":"2024-01-16T15:02:28.685084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682154+0000","last_change":"2024-01-16T15:02:28.682154+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682154+0000","last_undegraded":"2024-01-16T15:02:28.682154+0000","last_fullsized":"2024-01-16T15:02:28.682154+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687284+0000","last_change":"2024-01-16T15:02:28.687284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687284+0000","last_undegraded":"2024-01-16T15:02:28.687284+0000","last_fullsized":"2024-01-16T15:02:28.687284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680550+0000","last_change":"2024-01-16T15:02:28.680550+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680550+0000","last_undegraded":"2024-01-16T15:02:28.680550+0000","last_fullsized":"2024-01-16T15:02:28.680550+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677104+0000","last_change":"2024-01-16T15:02:28.677104+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677104+0000","last_undegraded":"2024-01-16T15:02:28.677104+0000","last_fullsized":"2024-01-16T15:02:28.677104+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677904+0000","last_change":"2024-01-16T15:02:28.677904+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677904+0000","last_undegraded":"2024-01-16T15:02:28.677904+0000","last_fullsized":"2024-01-16T15:02:28.677904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671452+0000","last_change":"2024-01-16T15:02:28.671452+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671452+0000","last_undegraded":"2024-01-16T15:02:28.671452+0000","last_fullsized":"2024-01-16T15:02:28.671452+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740996+0000","last_change":"2024-01-16T15:02:29.726552+0000","last_active":"2024-01-16T15:02:29.740996+0000","last_peered":"2024-01-16T15:02:29.740996+0000","last_clean":"2024-01-16T15:02:29.740996+0000","last_became_active":"2024-01-16T15:02:29.726242+0000","last_became_peered":"2024-01-16T15:02:29.726242+0000","last_unstale":"2024-01-16T15:02:29.740996+0000","last_undegraded":"2024-01-16T15:02:29.740996+0000","last_fullsized":"2024-01-16T15:02:29.740996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:15.816925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.743873+0000","last_change":"2024-01-16T15:02:29.735792+0000","last_active":"2024-01-16T15:02:29.743873+0000","last_peered":"2024-01-16T15:02:29.743873+0000","last_clean":"2024-01-16T15:02:29.743873+0000","last_became_active":"2024-01-16T15:02:29.735504+0000","last_became_peered":"2024-01-16T15:02:29.735504+0000","last_unstale":"2024-01-16T15:02:29.743873+0000","last_undegraded":"2024-01-16T15:02:29.743873+0000","last_fullsized":"2024-01-16T15:02:29.743873+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:09.173541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681639+0000","last_change":"2024-01-16T15:02:28.681639+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681639+0000","last_undegraded":"2024-01-16T15:02:28.681639+0000","last_fullsized":"2024-01-16T15:02:28.681639+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726359+0000","last_change":"2024-01-16T15:02:29.721939+0000","last_active":"2024-01-16T15:02:29.726359+0000","last_peered":"2024-01-16T15:02:29.726359+0000","last_clean":"2024-01-16T15:02:29.726359+0000","last_became_active":"2024-01-16T15:02:29.721563+0000","last_became_peered":"2024-01-16T15:02:29.721563+0000","last_unstale":"2024-01-16T15:02:29.726359+0000","last_undegraded":"2024-01-16T15:02:29.726359+0000","last_fullsized":"2024-01-16T15:02:29.726359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:35:52.362185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677387+0000","last_change":"2024-01-16T15:02:28.677387+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677387+0000","last_undegraded":"2024-01-16T15:02:28.677387+0000","last_fullsized":"2024-01-16T15:02:28.677387+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671228+0000","last_change":"2024-01-16T15:02:28.671228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671228+0000","last_undegraded":"2024-01-16T15:02:28.671228+0000","last_fullsized":"2024-01-16T15:02:28.671228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679199+0000","last_change":"2024-01-16T15:02:28.679199+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679199+0000","last_undegraded":"2024-01-16T15:02:28.679199+0000","last_fullsized":"2024-01-16T15:02:28.679199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674612+0000","last_change":"2024-01-16T15:02:28.674612+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674612+0000","last_undegraded":"2024-01-16T15:02:28.674612+0000","last_fullsized":"2024-01-16T15:02:28.674612+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.688365+0000","last_change":"2024-01-16T15:02:28.688365+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.688365+0000","last_undegraded":"2024-01-16T15:02:28.688365+0000","last_fullsized":"2024-01-16T15:02:28.688365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684341+0000","last_change":"2024-01-16T15:02:28.684341+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684341+0000","last_undegraded":"2024-01-16T15:02:28.684341+0000","last_fullsized":"2024-01-16T15:02:28.684341+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.677483+0000","last_change":"2024-01-16T15:02:28.677483+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.677483+0000","last_undegraded":"2024-01-16T15:02:28.677483+0000","last_fullsized":"2024-01-16T15:02:28.677483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680887+0000","last_change":"2024-01-16T15:02:28.680887+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680887+0000","last_undegraded":"2024-01-16T15:02:28.680887+0000","last_fullsized":"2024-01-16T15:02:28.680887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675926+0000","last_change":"2024-01-16T15:02:28.675926+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675926+0000","last_undegraded":"2024-01-16T15:02:28.675926+0000","last_fullsized":"2024-01-16T15:02:28.675926+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680236+0000","last_change":"2024-01-16T15:02:28.680236+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680236+0000","last_undegraded":"2024-01-16T15:02:28.680236+0000","last_fullsized":"2024-01-16T15:02:28.680236+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726074+0000","last_change":"2024-01-16T15:02:29.721477+0000","last_active":"2024-01-16T15:02:29.726074+0000","last_peered":"2024-01-16T15:02:29.726074+0000","last_clean":"2024-01-16T15:02:29.726074+0000","last_became_active":"2024-01-16T15:02:29.721137+0000","last_became_peered":"2024-01-16T15:02:29.721137+0000","last_unstale":"2024-01-16T15:02:29.726074+0000","last_undegraded":"2024-01-16T15:02:29.726074+0000","last_fullsized":"2024-01-16T15:02:29.726074+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:31.998282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.690190+0000","last_change":"2024-01-16T15:02:28.690190+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.690190+0000","last_undegraded":"2024-01-16T15:02:28.690190+0000","last_fullsized":"2024-01-16T15:02:28.690190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683203+0000","last_change":"2024-01-16T15:02:28.683203+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683203+0000","last_undegraded":"2024-01-16T15:02:28.683203+0000","last_fullsized":"2024-01-16T15:02:28.683203+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678571+0000","last_change":"2024-01-16T15:02:28.678571+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678571+0000","last_undegraded":"2024-01-16T15:02:28.678571+0000","last_fullsized":"2024-01-16T15:02:28.678571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687166+0000","last_change":"2024-01-16T15:02:28.687166+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687166+0000","last_undegraded":"2024-01-16T15:02:28.687166+0000","last_fullsized":"2024-01-16T15:02:28.687166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674298+0000","last_change":"2024-01-16T15:02:28.674298+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674298+0000","last_undegraded":"2024-01-16T15:02:28.674298+0000","last_fullsized":"2024-01-16T15:02:28.674298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.682909+0000","last_change":"2024-01-16T15:02:28.682909+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.682909+0000","last_undegraded":"2024-01-16T15:02:28.682909+0000","last_fullsized":"2024-01-16T15:02:28.682909+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683833+0000","last_change":"2024-01-16T15:02:28.683833+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683833+0000","last_undegraded":"2024-01-16T15:02:28.683833+0000","last_fullsized":"2024-01-16T15:02:28.683833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675958+0000","last_change":"2024-01-16T15:02:28.675958+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675958+0000","last_undegraded":"2024-01-16T15:02:28.675958+0000","last_fullsized":"2024-01-16T15:02:28.675958+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673472+0000","last_change":"2024-01-16T15:02:28.673472+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673472+0000","last_undegraded":"2024-01-16T15:02:28.673472+0000","last_fullsized":"2024-01-16T15:02:28.673472+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700420+0000","last_change":"2024-01-16T15:02:28.696840+0000","last_active":"2024-01-16T15:02:28.700420+0000","last_peered":"2024-01-16T15:02:28.700420+0000","last_clean":"2024-01-16T15:02:28.700420+0000","last_became_active":"2024-01-16T15:02:28.695989+0000","last_became_peered":"2024-01-16T15:02:28.695989+0000","last_unstale":"2024-01-16T15:02:28.700420+0000","last_undegraded":"2024-01-16T15:02:28.700420+0000","last_fullsized":"2024-01-16T15:02:28.700420+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:36.619091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.741340+0000","last_change":"2024-01-16T15:02:29.722881+0000","last_active":"2024-01-16T15:02:29.741340+0000","last_peered":"2024-01-16T15:02:29.741340+0000","last_clean":"2024-01-16T15:02:29.741340+0000","last_became_active":"2024-01-16T15:02:29.722447+0000","last_became_peered":"2024-01-16T15:02:29.722447+0000","last_unstale":"2024-01-16T15:02:29.741340+0000","last_undegraded":"2024-01-16T15:02:29.741340+0000","last_fullsized":"2024-01-16T15:02:29.741340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:52.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700061+0000","last_change":"2024-01-16T15:02:28.697447+0000","last_active":"2024-01-16T15:02:28.700061+0000","last_peered":"2024-01-16T15:02:28.700061+0000","last_clean":"2024-01-16T15:02:28.700061+0000","last_became_active":"2024-01-16T15:02:28.697122+0000","last_became_peered":"2024-01-16T15:02:28.697122+0000","last_unstale":"2024-01-16T15:02:28.700061+0000","last_undegraded":"2024-01-16T15:02:28.700061+0000","last_fullsized":"2024-01-16T15:02:28.700061+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:09.273766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672137+0000","last_change":"2024-01-16T15:02:28.672137+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672137+0000","last_undegraded":"2024-01-16T15:02:28.672137+0000","last_fullsized":"2024-01-16T15:02:28.672137+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696624+0000","last_change":"2024-01-16T15:02:28.689593+0000","last_active":"2024-01-16T15:02:28.696624+0000","last_peered":"2024-01-16T15:02:28.696624+0000","last_clean":"2024-01-16T15:02:28.696624+0000","last_became_active":"2024-01-16T15:02:28.689343+0000","last_became_peered":"2024-01-16T15:02:28.689343+0000","last_unstale":"2024-01-16T15:02:28.696624+0000","last_undegraded":"2024-01-16T15:02:28.696624+0000","last_fullsized":"2024-01-16T15:02:28.696624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:51.159242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673332+0000","last_change":"2024-01-16T15:02:28.673332+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673332+0000","last_undegraded":"2024-01-16T15:02:28.673332+0000","last_fullsized":"2024-01-16T15:02:28.673332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698069+0000","last_change":"2024-01-16T15:02:28.694539+0000","last_active":"2024-01-16T15:02:28.698069+0000","last_peered":"2024-01-16T15:02:28.698069+0000","last_clean":"2024-01-16T15:02:28.698069+0000","last_became_active":"2024-01-16T15:02:28.694233+0000","last_became_peered":"2024-01-16T15:02:28.694233+0000","last_unstale":"2024-01-16T15:02:28.698069+0000","last_undegraded":"2024-01-16T15:02:28.698069+0000","last_fullsized":"2024-01-16T15:02:28.698069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675366+0000","last_change":"2024-01-16T15:02:28.675366+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675366+0000","last_undegraded":"2024-01-16T15:02:28.675366+0000","last_fullsized":"2024-01-16T15:02:28.675366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696756+0000","last_change":"2024-01-16T15:02:28.693560+0000","last_active":"2024-01-16T15:02:28.696756+0000","last_peered":"2024-01-16T15:02:28.696756+0000","last_clean":"2024-01-16T15:02:28.696756+0000","last_became_active":"2024-01-16T15:02:28.693211+0000","last_became_peered":"2024-01-16T15:02:28.693211+0000","last_unstale":"2024-01-16T15:02:28.696756+0000","last_undegraded":"2024-01-16T15:02:28.696756+0000","last_fullsized":"2024-01-16T15:02:28.696756+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676930+0000","last_change":"2024-01-16T15:02:28.676930+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676930+0000","last_undegraded":"2024-01-16T15:02:28.676930+0000","last_fullsized":"2024-01-16T15:02:28.676930+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.691282+0000","last_change":"2024-01-16T15:02:28.683014+0000","last_active":"2024-01-16T15:02:28.691282+0000","last_peered":"2024-01-16T15:02:28.691282+0000","last_clean":"2024-01-16T15:02:28.691282+0000","last_became_active":"2024-01-16T15:02:28.682735+0000","last_became_peered":"2024-01-16T15:02:28.682735+0000","last_unstale":"2024-01-16T15:02:28.691282+0000","last_undegraded":"2024-01-16T15:02:28.691282+0000","last_fullsized":"2024-01-16T15:02:28.691282+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:55.974417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.726389+0000","last_change":"2024-01-16T15:02:29.721954+0000","last_active":"2024-01-16T15:02:29.726389+0000","last_peered":"2024-01-16T15:02:29.726389+0000","last_clean":"2024-01-16T15:02:29.726389+0000","last_became_active":"2024-01-16T15:02:29.721647+0000","last_became_peered":"2024-01-16T15:02:29.721647+0000","last_unstale":"2024-01-16T15:02:29.726389+0000","last_undegraded":"2024-01-16T15:02:29.726389+0000","last_fullsized":"2024-01-16T15:02:29.726389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:16.064389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697872+0000","last_change":"2024-01-16T15:02:28.694096+0000","last_active":"2024-01-16T15:02:28.697872+0000","last_peered":"2024-01-16T15:02:28.697872+0000","last_clean":"2024-01-16T15:02:28.697872+0000","last_became_active":"2024-01-16T15:02:28.693789+0000","last_became_peered":"2024-01-16T15:02:28.693789+0000","last_unstale":"2024-01-16T15:02:28.697872+0000","last_undegraded":"2024-01-16T15:02:28.697872+0000","last_fullsized":"2024-01-16T15:02:28.697872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681431+0000","last_change":"2024-01-16T15:02:28.681431+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681431+0000","last_undegraded":"2024-01-16T15:02:28.681431+0000","last_fullsized":"2024-01-16T15:02:28.681431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696269+0000","last_change":"2024-01-16T15:02:28.693108+0000","last_active":"2024-01-16T15:02:28.696269+0000","last_peered":"2024-01-16T15:02:28.696269+0000","last_clean":"2024-01-16T15:02:28.696269+0000","last_became_active":"2024-01-16T15:02:28.691074+0000","last_became_peered":"2024-01-16T15:02:28.691074+0000","last_unstale":"2024-01-16T15:02:28.696269+0000","last_undegraded":"2024-01-16T15:02:28.696269+0000","last_fullsized":"2024-01-16T15:02:28.696269+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672957+0000","last_change":"2024-01-16T15:02:28.672957+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672957+0000","last_undegraded":"2024-01-16T15:02:28.672957+0000","last_fullsized":"2024-01-16T15:02:28.672957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702078+0000","last_change":"2024-01-16T15:02:28.700517+0000","last_active":"2024-01-16T15:02:28.702078+0000","last_peered":"2024-01-16T15:02:28.702078+0000","last_clean":"2024-01-16T15:02:28.702078+0000","last_became_active":"2024-01-16T15:02:28.700187+0000","last_became_peered":"2024-01-16T15:02:28.700187+0000","last_unstale":"2024-01-16T15:02:28.702078+0000","last_undegraded":"2024-01-16T15:02:28.702078+0000","last_fullsized":"2024-01-16T15:02:28.702078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.686150+0000","last_change":"2024-01-16T15:02:28.686150+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.686150+0000","last_undegraded":"2024-01-16T15:02:28.686150+0000","last_fullsized":"2024-01-16T15:02:28.686150+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699635+0000","last_change":"2024-01-16T15:02:28.694170+0000","last_active":"2024-01-16T15:02:28.699635+0000","last_peered":"2024-01-16T15:02:28.699635+0000","last_clean":"2024-01-16T15:02:28.699635+0000","last_became_active":"2024-01-16T15:02:28.693951+0000","last_became_peered":"2024-01-16T15:02:28.693951+0000","last_unstale":"2024-01-16T15:02:28.699635+0000","last_undegraded":"2024-01-16T15:02:28.699635+0000","last_fullsized":"2024-01-16T15:02:28.699635+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:05.345755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740967+0000","last_change":"2024-01-16T15:02:29.728569+0000","last_active":"2024-01-16T15:02:29.740967+0000","last_peered":"2024-01-16T15:02:29.740967+0000","last_clean":"2024-01-16T15:02:29.740967+0000","last_became_active":"2024-01-16T15:02:29.728240+0000","last_became_peered":"2024-01-16T15:02:29.728240+0000","last_unstale":"2024-01-16T15:02:29.740967+0000","last_undegraded":"2024-01-16T15:02:29.740967+0000","last_fullsized":"2024-01-16T15:02:29.740967+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:49.783058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.702091+0000","last_change":"2024-01-16T15:02:28.700598+0000","last_active":"2024-01-16T15:02:28.702091+0000","last_peered":"2024-01-16T15:02:28.702091+0000","last_clean":"2024-01-16T15:02:28.702091+0000","last_became_active":"2024-01-16T15:02:28.700296+0000","last_became_peered":"2024-01-16T15:02:28.700296+0000","last_unstale":"2024-01-16T15:02:28.702091+0000","last_undegraded":"2024-01-16T15:02:28.702091+0000","last_fullsized":"2024-01-16T15:02:28.702091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:25:14.812431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684658+0000","last_change":"2024-01-16T15:02:28.684658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684658+0000","last_undegraded":"2024-01-16T15:02:28.684658+0000","last_fullsized":"2024-01-16T15:02:28.684658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701099+0000","last_change":"2024-01-16T15:02:28.697153+0000","last_active":"2024-01-16T15:02:28.701099+0000","last_peered":"2024-01-16T15:02:28.701099+0000","last_clean":"2024-01-16T15:02:28.701099+0000","last_became_active":"2024-01-16T15:02:28.693885+0000","last_became_peered":"2024-01-16T15:02:28.693885+0000","last_unstale":"2024-01-16T15:02:28.701099+0000","last_undegraded":"2024-01-16T15:02:28.701099+0000","last_fullsized":"2024-01-16T15:02:28.701099+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683793+0000","last_change":"2024-01-16T15:02:28.683793+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683793+0000","last_undegraded":"2024-01-16T15:02:28.683793+0000","last_fullsized":"2024-01-16T15:02:28.683793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699827+0000","last_change":"2024-01-16T15:02:28.695099+0000","last_active":"2024-01-16T15:02:28.699827+0000","last_peered":"2024-01-16T15:02:28.699827+0000","last_clean":"2024-01-16T15:02:28.699827+0000","last_became_active":"2024-01-16T15:02:28.694828+0000","last_became_peered":"2024-01-16T15:02:28.694828+0000","last_unstale":"2024-01-16T15:02:28.699827+0000","last_undegraded":"2024-01-16T15:02:28.699827+0000","last_fullsized":"2024-01-16T15:02:28.699827+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:29.254715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.680264+0000","last_change":"2024-01-16T15:02:28.680264+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.680264+0000","last_undegraded":"2024-01-16T15:02:28.680264+0000","last_fullsized":"2024-01-16T15:02:28.680264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706804+0000","last_change":"2024-01-16T15:02:28.688577+0000","last_active":"2024-01-16T15:02:29.706804+0000","last_peered":"2024-01-16T15:02:29.706804+0000","last_clean":"2024-01-16T15:02:29.706804+0000","last_became_active":"2024-01-16T15:02:28.687999+0000","last_became_peered":"2024-01-16T15:02:28.687999+0000","last_unstale":"2024-01-16T15:02:29.706804+0000","last_undegraded":"2024-01-16T15:02:29.706804+0000","last_fullsized":"2024-01-16T15:02:29.706804+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:17.380916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.673658+0000","last_change":"2024-01-16T15:02:28.673658+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.673658+0000","last_undegraded":"2024-01-16T15:02:28.673658+0000","last_fullsized":"2024-01-16T15:02:28.673658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697893+0000","last_change":"2024-01-16T15:02:28.694910+0000","last_active":"2024-01-16T15:02:28.697893+0000","last_peered":"2024-01-16T15:02:28.697893+0000","last_clean":"2024-01-16T15:02:28.697893+0000","last_became_active":"2024-01-16T15:02:28.690704+0000","last_became_peered":"2024-01-16T15:02:28.690704+0000","last_unstale":"2024-01-16T15:02:28.697893+0000","last_undegraded":"2024-01-16T15:02:28.697893+0000","last_fullsized":"2024-01-16T15:02:28.697893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.678221+0000","last_change":"2024-01-16T15:02:28.678221+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.678221+0000","last_undegraded":"2024-01-16T15:02:28.678221+0000","last_fullsized":"2024-01-16T15:02:28.678221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.701998+0000","last_change":"2024-01-16T15:02:28.699544+0000","last_active":"2024-01-16T15:02:28.701998+0000","last_peered":"2024-01-16T15:02:28.701998+0000","last_clean":"2024-01-16T15:02:28.701998+0000","last_became_active":"2024-01-16T15:02:28.699169+0000","last_became_peered":"2024-01-16T15:02:28.699169+0000","last_unstale":"2024-01-16T15:02:28.701998+0000","last_undegraded":"2024-01-16T15:02:28.701998+0000","last_fullsized":"2024-01-16T15:02:28.701998+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:51.361029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.687323+0000","last_change":"2024-01-16T15:02:28.687323+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.687323+0000","last_undegraded":"2024-01-16T15:02:28.687323+0000","last_fullsized":"2024-01-16T15:02:28.687323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.699992+0000","last_change":"2024-01-16T15:02:28.695124+0000","last_active":"2024-01-16T15:02:28.699992+0000","last_peered":"2024-01-16T15:02:28.699992+0000","last_clean":"2024-01-16T15:02:28.699992+0000","last_became_active":"2024-01-16T15:02:28.694847+0000","last_became_peered":"2024-01-16T15:02:28.694847+0000","last_unstale":"2024-01-16T15:02:28.699992+0000","last_undegraded":"2024-01-16T15:02:28.699992+0000","last_fullsized":"2024-01-16T15:02:28.699992+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:56.293921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.684228+0000","last_change":"2024-01-16T15:02:28.684228+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.684228+0000","last_undegraded":"2024-01-16T15:02:28.684228+0000","last_fullsized":"2024-01-16T15:02:28.684228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700477+0000","last_change":"2024-01-16T15:02:28.698762+0000","last_active":"2024-01-16T15:02:28.700477+0000","last_peered":"2024-01-16T15:02:28.700477+0000","last_clean":"2024-01-16T15:02:28.700477+0000","last_became_active":"2024-01-16T15:02:28.698337+0000","last_became_peered":"2024-01-16T15:02:28.698337+0000","last_unstale":"2024-01-16T15:02:28.700477+0000","last_undegraded":"2024-01-16T15:02:28.700477+0000","last_fullsized":"2024-01-16T15:02:28.700477+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:40.971035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.676992+0000","last_change":"2024-01-16T15:02:28.676992+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.676992+0000","last_undegraded":"2024-01-16T15:02:28.676992+0000","last_fullsized":"2024-01-16T15:02:28.676992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.698294+0000","last_change":"2024-01-16T15:02:28.695728+0000","last_active":"2024-01-16T15:02:28.698294+0000","last_peered":"2024-01-16T15:02:28.698294+0000","last_clean":"2024-01-16T15:02:28.698294+0000","last_became_active":"2024-01-16T15:02:28.695438+0000","last_became_peered":"2024-01-16T15:02:28.695438+0000","last_unstale":"2024-01-16T15:02:28.698294+0000","last_undegraded":"2024-01-16T15:02:28.698294+0000","last_fullsized":"2024-01-16T15:02:28.698294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679870+0000","last_change":"2024-01-16T15:02:28.679870+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679870+0000","last_undegraded":"2024-01-16T15:02:28.679870+0000","last_fullsized":"2024-01-16T15:02:28.679870+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696703+0000","last_change":"2024-01-16T15:02:28.690359+0000","last_active":"2024-01-16T15:02:28.696703+0000","last_peered":"2024-01-16T15:02:28.696703+0000","last_clean":"2024-01-16T15:02:28.696703+0000","last_became_active":"2024-01-16T15:02:28.689968+0000","last_became_peered":"2024-01-16T15:02:28.689968+0000","last_unstale":"2024-01-16T15:02:28.696703+0000","last_undegraded":"2024-01-16T15:02:28.696703+0000","last_fullsized":"2024-01-16T15:02:28.696703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:06.195041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683284+0000","last_change":"2024-01-16T15:02:28.683284+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683284+0000","last_undegraded":"2024-01-16T15:02:28.683284+0000","last_fullsized":"2024-01-16T15:02:28.683284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689306+0000","last_change":"2024-01-16T15:02:28.687104+0000","last_active":"2024-01-16T15:02:28.689306+0000","last_peered":"2024-01-16T15:02:28.689306+0000","last_clean":"2024-01-16T15:02:28.689306+0000","last_became_active":"2024-01-16T15:02:28.686179+0000","last_became_peered":"2024-01-16T15:02:28.686179+0000","last_unstale":"2024-01-16T15:02:28.689306+0000","last_undegraded":"2024-01-16T15:02:28.689306+0000","last_fullsized":"2024-01-16T15:02:28.689306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:39.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672340+0000","last_change":"2024-01-16T15:02:28.672340+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672340+0000","last_undegraded":"2024-01-16T15:02:28.672340+0000","last_fullsized":"2024-01-16T15:02:28.672340+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.722558+0000","last_change":"2024-01-16T15:02:28.693574+0000","last_active":"2024-01-16T15:02:29.722558+0000","last_peered":"2024-01-16T15:02:29.722558+0000","last_clean":"2024-01-16T15:02:29.722558+0000","last_became_active":"2024-01-16T15:02:28.692884+0000","last_became_peered":"2024-01-16T15:02:28.692884+0000","last_unstale":"2024-01-16T15:02:29.722558+0000","last_undegraded":"2024-01-16T15:02:29.722558+0000","last_fullsized":"2024-01-16T15:02:29.722558+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.672000+0000","last_change":"2024-01-16T15:02:28.672000+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.672000+0000","last_undegraded":"2024-01-16T15:02:28.672000+0000","last_fullsized":"2024-01-16T15:02:28.672000+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696515+0000","last_change":"2024-01-16T15:02:28.692170+0000","last_active":"2024-01-16T15:02:28.696515+0000","last_peered":"2024-01-16T15:02:28.696515+0000","last_clean":"2024-01-16T15:02:28.696515+0000","last_became_active":"2024-01-16T15:02:28.691779+0000","last_became_peered":"2024-01-16T15:02:28.691779+0000","last_unstale":"2024-01-16T15:02:28.696515+0000","last_undegraded":"2024-01-16T15:02:28.696515+0000","last_fullsized":"2024-01-16T15:02:28.696515+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:48.778565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.681494+0000","last_change":"2024-01-16T15:02:28.681494+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.681494+0000","last_undegraded":"2024-01-16T15:02:28.681494+0000","last_fullsized":"2024-01-16T15:02:28.681494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.737753+0000","last_change":"2024-01-16T15:02:28.689348+0000","last_active":"2024-01-16T15:02:29.737753+0000","last_peered":"2024-01-16T15:02:29.737753+0000","last_clean":"2024-01-16T15:02:29.737753+0000","last_became_active":"2024-01-16T15:02:28.689030+0000","last_became_peered":"2024-01-16T15:02:28.689030+0000","last_unstale":"2024-01-16T15:02:29.737753+0000","last_undegraded":"2024-01-16T15:02:29.737753+0000","last_fullsized":"2024-01-16T15:02:29.737753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:24.816323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.683770+0000","last_change":"2024-01-16T15:02:28.683770+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.683770+0000","last_undegraded":"2024-01-16T15:02:28.683770+0000","last_fullsized":"2024-01-16T15:02:28.683770+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.689249+0000","last_change":"2024-01-16T15:02:28.681603+0000","last_active":"2024-01-16T15:02:28.689249+0000","last_peered":"2024-01-16T15:02:28.689249+0000","last_clean":"2024-01-16T15:02:28.689249+0000","last_became_active":"2024-01-16T15:02:28.681322+0000","last_became_peered":"2024-01-16T15:02:28.681322+0000","last_unstale":"2024-01-16T15:02:28.689249+0000","last_undegraded":"2024-01-16T15:02:28.689249+0000","last_fullsized":"2024-01-16T15:02:28.689249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:13.938868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740947+0000","last_change":"2024-01-16T15:02:29.728538+0000","last_active":"2024-01-16T15:02:29.740947+0000","last_peered":"2024-01-16T15:02:29.740947+0000","last_clean":"2024-01-16T15:02:29.740947+0000","last_became_active":"2024-01-16T15:02:29.728211+0000","last_became_peered":"2024-01-16T15:02:29.728211+0000","last_unstale":"2024-01-16T15:02:29.740947+0000","last_undegraded":"2024-01-16T15:02:29.740947+0000","last_fullsized":"2024-01-16T15:02:29.740947+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:32.872029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.706859+0000","last_change":"2024-01-16T15:02:28.681634+0000","last_active":"2024-01-16T15:02:29.706859+0000","last_peered":"2024-01-16T15:02:29.706859+0000","last_clean":"2024-01-16T15:02:29.706859+0000","last_became_active":"2024-01-16T15:02:28.680098+0000","last_became_peered":"2024-01-16T15:02:28.680098+0000","last_unstale":"2024-01-16T15:02:29.706859+0000","last_undegraded":"2024-01-16T15:02:29.706859+0000","last_fullsized":"2024-01-16T15:02:29.706859+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:45.583344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.671790+0000","last_change":"2024-01-16T15:02:28.671790+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.671790+0000","last_undegraded":"2024-01-16T15:02:28.671790+0000","last_fullsized":"2024-01-16T15:02:28.671790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.697195+0000","last_change":"2024-01-16T15:02:28.692899+0000","last_active":"2024-01-16T15:02:28.697195+0000","last_peered":"2024-01-16T15:02:28.697195+0000","last_clean":"2024-01-16T15:02:28.697195+0000","last_became_active":"2024-01-16T15:02:28.692607+0000","last_became_peered":"2024-01-16T15:02:28.692607+0000","last_unstale":"2024-01-16T15:02:28.697195+0000","last_undegraded":"2024-01-16T15:02:28.697195+0000","last_fullsized":"2024-01-16T15:02:28.697195+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:08.169048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674849+0000","last_change":"2024-01-16T15:02:28.674849+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674849+0000","last_undegraded":"2024-01-16T15:02:28.674849+0000","last_fullsized":"2024-01-16T15:02:28.674849+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.728464+0000","last_change":"2024-01-16T15:02:28.695802+0000","last_active":"2024-01-16T15:02:29.728464+0000","last_peered":"2024-01-16T15:02:29.728464+0000","last_clean":"2024-01-16T15:02:29.728464+0000","last_became_active":"2024-01-16T15:02:28.695516+0000","last_became_peered":"2024-01-16T15:02:28.695516+0000","last_unstale":"2024-01-16T15:02:29.728464+0000","last_undegraded":"2024-01-16T15:02:29.728464+0000","last_fullsized":"2024-01-16T15:02:29.728464+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:00.682347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.679260+0000","last_change":"2024-01-16T15:02:28.679260+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.679260+0000","last_undegraded":"2024-01-16T15:02:28.679260+0000","last_fullsized":"2024-01-16T15:02:28.679260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.696398+0000","last_change":"2024-01-16T15:02:28.691759+0000","last_active":"2024-01-16T15:02:28.696398+0000","last_peered":"2024-01-16T15:02:28.696398+0000","last_clean":"2024-01-16T15:02:28.696398+0000","last_became_active":"2024-01-16T15:02:28.691457+0000","last_became_peered":"2024-01-16T15:02:28.691457+0000","last_unstale":"2024-01-16T15:02:28.696398+0000","last_undegraded":"2024-01-16T15:02:28.696398+0000","last_fullsized":"2024-01-16T15:02:28.696398+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:31.867536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.738586+0000","last_change":"2024-01-16T15:02:29.719394+0000","last_active":"2024-01-16T15:02:29.738586+0000","last_peered":"2024-01-16T15:02:29.738586+0000","last_clean":"2024-01-16T15:02:29.738586+0000","last_became_active":"2024-01-16T15:02:29.719030+0000","last_became_peered":"2024-01-16T15:02:29.719030+0000","last_unstale":"2024-01-16T15:02:29.738586+0000","last_undegraded":"2024-01-16T15:02:29.738586+0000","last_fullsized":"2024-01-16T15:02:29.738586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:51.117200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.717109+0000","last_change":"2024-01-16T15:02:28.692453+0000","last_active":"2024-01-16T15:02:29.717109+0000","last_peered":"2024-01-16T15:02:29.717109+0000","last_clean":"2024-01-16T15:02:29.717109+0000","last_became_active":"2024-01-16T15:02:28.692132+0000","last_became_peered":"2024-01-16T15:02:28.692132+0000","last_unstale":"2024-01-16T15:02:29.717109+0000","last_undegraded":"2024-01-16T15:02:29.717109+0000","last_fullsized":"2024-01-16T15:02:29.717109+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.674276+0000","last_change":"2024-01-16T15:02:28.674276+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.674276+0000","last_undegraded":"2024-01-16T15:02:28.674276+0000","last_fullsized":"2024-01-16T15:02:28.674276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-16T15:02:28.700041+0000","last_change":"2024-01-16T15:02:28.696929+0000","last_active":"2024-01-16T15:02:28.700041+0000","last_peered":"2024-01-16T15:02:28.700041+0000","last_clean":"2024-01-16T15:02:28.700041+0000","last_became_active":"2024-01-16T15:02:28.696616+0000","last_became_peered":"2024-01-16T15:02:28.696616+0000","last_unstale":"2024-01-16T15:02:28.700041+0000","last_undegraded":"2024-01-16T15:02:28.700041+0000","last_fullsized":"2024-01-16T15:02:28.700041+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:29.321021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-01-16T15:02:28.675396+0000","last_change":"2024-01-16T15:02:28.675396+0000","last_active":"2024-01-16T15:02:28.654459+0000","last_peered":"2024-01-16T15:02:28.654459+0000","last_clean":"2024-01-16T15:02:28.654459+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:28.675396+0000","last_undegraded":"2024-01-16T15:02:28.675396+0000","last_fullsized":"2024-01-16T15:02:28.675396+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.708768+0000","last_change":"2024-01-16T15:02:28.695669+0000","last_active":"2024-01-16T15:02:29.708768+0000","last_peered":"2024-01-16T15:02:29.708768+0000","last_clean":"2024-01-16T15:02:29.708768+0000","last_became_active":"2024-01-16T15:02:28.695350+0000","last_became_peered":"2024-01-16T15:02:28.695350+0000","last_unstale":"2024-01-16T15:02:29.708768+0000","last_undegraded":"2024-01-16T15:02:29.708768+0000","last_fullsized":"2024-01-16T15:02:29.708768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:36.266476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":199,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-16T15:02:27.660341+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:02:27.660341+0000","last_peered":"2024-01-16T15:02:27.660341+0000","last_clean":"2024-01-16T15:02:27.660341+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:02:27.660341+0000","last_undegraded":"2024-01-16T15:02:27.660341+0000","last_fullsized":"2024-01-16T15:02:27.660341+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.0329999999999999,"5min":1.3480000000000001,"15min":1.1539999999999999},"min":{"1min":0.73899999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3089999999999999,"15min":1.2010000000000001},"min":{"1min":0.76700000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9319999999999999,"5min":2.774,"15min":2.774},"last":1.1080000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.2909999999999999,"15min":1.111},"min":{"1min":0.76400000000000001,"5min":0.63,"15min":0.63},"max":{"1min":11.487,"5min":11.487,"15min":11.487},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3380000000000001,"15min":1.1890000000000001},"min":{"1min":0.89000000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.8109999999999999,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":0.82199999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.288,"15min":1.1970000000000001},"min":{"1min":0.85599999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.7269999999999999,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.28},{"interface":"front","average":{"1min":2.8940000000000001,"5min":1.6160000000000001,"15min":1.349},"min":{"1min":0.84199999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":20.577000000000002,"5min":20.577000000000002,"15min":20.577000000000002},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2639999999999998,"5min":1.6259999999999999,"15min":1.3979999999999999},"min":{"1min":1.0569999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":11.872,"5min":12.861000000000001,"15min":12.861000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.486,"15min":1.4099999999999999},"min":{"1min":0.81100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.1379999999999999,"5min":4.0720000000000001,"15min":4.0720000000000001},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.8439999999999999,"5min":1.631,"15min":1.446},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":20.649999999999999,"5min":20.649999999999999,"15min":20.649999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.387,"5min":1.633,"15min":1.5109999999999999},"min":{"1min":0.94399999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":11.702,"5min":11.702,"15min":11.702},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.514,"15min":1.526},"min":{"1min":0.70499999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.8039999999999998,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.73,"15min":1.6719999999999999},"min":{"1min":1.0149999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":11.225,"5min":11.225,"15min":11.225},"last":1.0649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":4.1589999999999998,"5min":2.0630000000000002,"15min":1.7430000000000001},"min":{"1min":1.306,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":41.206000000000003,"5min":41.206000000000003,"15min":41.206000000000003},"last":1.155},{"interface":"front","average":{"1min":4.125,"5min":2.0449999999999999,"15min":1.7589999999999999},"min":{"1min":1.0940000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":21.908000000000001,"5min":21.908000000000001,"15min":21.908000000000001},"last":1.3280000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.2609999999999999,"15min":1.147},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":9.1890000000000001,"5min":9.1890000000000001,"15min":9.1890000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":5.2039999999999997,"5min":1.976,"15min":1.478},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":50.472000000000001,"5min":50.472000000000001,"15min":50.472000000000001},"last":0.80600000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":1.3979999999999999,"15min":1.1459999999999999},"min":{"1min":0.66300000000000003,"5min":0.624,"15min":0.624},"max":{"1min":10.629,"5min":10.629,"15min":10.629},"last":0.98399999999999999},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.5509999999999999,"15min":1.421},"min":{"1min":0.82699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":9.3279999999999994,"5min":9.3279999999999994,"15min":9.3279999999999994},"last":1.0569999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.3789999999999996,"5min":1.9450000000000001,"15min":1.4199999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":50.643999999999998,"5min":50.643999999999998,"15min":50.643999999999998},"last":1.385},{"interface":"front","average":{"1min":2.4169999999999998,"5min":1.4970000000000001,"15min":1.359},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":10.843,"5min":10.843,"15min":10.843},"last":1.607}]},{"osd":3,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":1.6100000000000001,"15min":1.4359999999999999},"min":{"1min":0.82399999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":15.675000000000001,"5min":15.675000000000001,"15min":15.675000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":4.3760000000000003,"5min":1.99,"15min":1.5800000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":50.561,"5min":50.561,"15min":50.561},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.8289999999999997,"5min":2.1840000000000002,"15min":1.728},"min":{"1min":0.92700000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":47.488,"5min":47.488,"15min":47.488},"last":1.542},{"interface":"front","average":{"1min":5.2519999999999998,"5min":2.2330000000000001,"15min":1.7609999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":50.381,"5min":50.381,"15min":50.381},"last":1.4850000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.5419999999999998,"5min":1.6659999999999999,"15min":1.5409999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":10.654,"5min":10.654,"15min":10.654},"last":1.7450000000000001},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.8280000000000001,"15min":1.726},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":9.3729999999999993,"5min":9.3729999999999993,"15min":9.3729999999999993},"last":1.7789999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.9820000000000002,"5min":2.2799999999999998,"15min":1.8779999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":52.912999999999997,"5min":52.912999999999997,"15min":52.912999999999997},"last":1.581},{"interface":"front","average":{"1min":2.5270000000000001,"5min":1.7769999999999999,"15min":1.7130000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":10.581,"5min":10.581,"15min":10.581},"last":1.8600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3109999999999999,"5min":1.3580000000000001,"15min":1.248},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":13.938000000000001,"5min":13.938000000000001,"15min":13.938000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":2.9500000000000002,"5min":1.478,"15min":1.212},"min":{"1min":0.73199999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":16.896000000000001,"5min":16.896000000000001,"15min":16.896000000000001},"last":1.591}]},{"osd":1,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.482,"15min":1.353},"min":{"1min":0.77500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":3.4550000000000001,"5min":1.7549999999999999,"15min":1.345},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":35.328000000000003,"5min":35.328000000000003,"15min":35.328000000000003},"last":1.554}]},{"osd":2,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.7639999999999998,"5min":1.8859999999999999,"15min":1.47},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":35.398000000000003,"5min":35.398000000000003,"15min":35.398000000000003},"last":1.091},{"interface":"front","average":{"1min":1.782,"5min":1.3340000000000001,"15min":1.29},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.883,"5min":4.883,"15min":4.883},"last":1.7150000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8210000000000002,"5min":1.667,"15min":1.4430000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":13.202,"5min":13.202,"15min":13.202},"last":1.4810000000000001},{"interface":"front","average":{"1min":3.6669999999999998,"5min":1.9670000000000001,"15min":1.587},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":35.268000000000001,"5min":35.268000000000001,"15min":35.268000000000001},"last":1.6799999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.5470000000000002,"5min":1.784,"15min":1.579},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":15.757,"5min":15.757,"15min":15.757},"last":1.0069999999999999},{"interface":"front","average":{"1min":4.3479999999999999,"5min":2.004,"15min":1.714},"min":{"1min":0.63300000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":26.166,"5min":26.166,"15min":26.166},"last":1.7709999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":4.0350000000000001,"5min":2.0950000000000002,"15min":1.768},"min":{"1min":0.53400000000000003,"5min":0.499,"15min":0.499},"max":{"1min":35.427999999999997,"5min":35.427999999999997,"15min":35.427999999999997},"last":1.7410000000000001},{"interface":"front","average":{"1min":3.4740000000000002,"5min":1.903,"15min":1.7589999999999999},"min":{"1min":0.53100000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":1.6259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.9350000000000001,"5min":2.0379999999999998,"15min":1.7689999999999999},"min":{"1min":0.58099999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":19.364999999999998,"5min":19.364999999999998,"15min":19.364999999999998},"last":1.837},{"interface":"front","average":{"1min":4.0419999999999998,"5min":2.1829999999999998,"15min":1.8839999999999999},"min":{"1min":0.71699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":35.363,"5min":35.363,"15min":35.363},"last":1.994}]}]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.1779999999999999,"15min":1.1080000000000001},"min":{"1min":0.751,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":4.2869999999999999,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":1.012},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.4059999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":8.0440000000000005,"5min":8.0440000000000005,"15min":8.0440000000000005},"last":0.90300000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.4359999999999999,"15min":1.258},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.1050000000000004,"5min":7.1050000000000004,"15min":7.1050000000000004},"last":1.391},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.2889999999999999,"15min":1.234},"min":{"1min":0.84499999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":5.7290000000000001,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":1.137}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.583,"15min":1.415},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.4860000000000002,"5min":15.986000000000001,"15min":15.986000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.554,"5min":1.3520000000000001,"15min":1.4330000000000001},"min":{"1min":0.83699999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":3.762,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":1.232}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":3.0049999999999999,"5min":1.871,"15min":1.6819999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5600000000000001,"15min":1.48},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":4.0490000000000004,"5min":21.706,"15min":21.706},"last":1.538}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4370000000000001,"15min":1.575},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":4.327,"5min":4.327,"15min":4.327},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.649,"15min":1.6279999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":10.831,"5min":10.831,"15min":10.831},"last":1.504}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4530000000000001,"15min":1.575},"min":{"1min":0.66000000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":1.476},{"interface":"front","average":{"1min":2.468,"5min":1.734,"15min":1.732},"min":{"1min":0.74199999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":20.939,"5min":20.939,"15min":20.939},"last":1.4399999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.8,"15min":1.7490000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":7.0110000000000001,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":1.591},{"interface":"front","average":{"1min":1.976,"5min":1.6399999999999999,"15min":1.738},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":10.978999999999999,"5min":10.978999999999999,"15min":10.978999999999999},"last":1.6279999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.98499999999999999,"15min":0.91900000000000004},"min":{"1min":0.433,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1459999999999999,"15min":0.93600000000000005},"min":{"1min":0.56799999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":1.099}]},{"osd":1,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.92500000000000004,"15min":0.85999999999999999},"min":{"1min":0.47099999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.752,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.121,"5min":1.0620000000000001,"15min":1.016},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.4590000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.075,"15min":1.075},"min":{"1min":0.59299999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9339999999999999,"5min":4.008,"15min":4.008},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.0820000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.2069999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.2929999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.3500000000000001,"15min":1.1639999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":7.7439999999999998,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.492,"15min":1.3129999999999999},"min":{"1min":0.80500000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":11.068,"5min":11.068,"15min":11.068},"last":1.208}]},{"osd":5,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4630000000000001,"15min":1.3180000000000001},"min":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":12.791,"5min":12.791,"15min":12.791},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.508,"15min":1.413},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":12.106,"5min":12.106,"15min":12.106},"last":1.5389999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":1.546,"15min":1.3540000000000001},"min":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":8.1039999999999992,"5min":8.1039999999999992,"15min":8.1039999999999992},"last":1.502},{"interface":"front","average":{"1min":2.3980000000000001,"5min":1.6359999999999999,"15min":1.4379999999999999},"min":{"1min":0.91900000000000004,"5min":0.752,"15min":0.752},"max":{"1min":6.891,"5min":6.891,"15min":6.891},"last":1.4890000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.532,"15min":1.4379999999999999},"min":{"1min":0.82599999999999996,"5min":0.755,"15min":0.755},"max":{"1min":3.7269999999999999,"5min":3.7269999999999999,"15min":3.7269999999999999},"last":1.599},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.677,"15min":1.4299999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":7.6159999999999997,"5min":18.846,"15min":18.846},"last":1.5640000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36004,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335836,"statfs":{"total":96636764160,"available":96599896064,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":0.91600000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.839,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.378},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0049999999999999,"15min":0.93600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.353,"5min":4.5010000000000003,"15min":4.5010000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0609999999999999,"15min":1.0329999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.133,"5min":3.798,"15min":3.798},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.125,"15min":1.014},"min":{"1min":0.45800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.8069999999999999,"5min":4.8250000000000002,"15min":4.8250000000000002},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0620000000000001,"15min":0.97999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.9870000000000001,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.224,"15min":1.1399999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":8.4610000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.5489999999999999,"15min":1.3939999999999999},"min":{"1min":0.67700000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":11.843,"5min":11.843,"15min":11.843},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.847,"5min":1.54,"15min":1.411},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":11.768000000000001,"5min":11.768000000000001,"15min":11.768000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":1.72,"15min":1.4470000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":26.501000000000001,"5min":26.501000000000001,"15min":26.501000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.389,"15min":1.341},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.8640000000000001,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.593,"15min":1.4430000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.5059999999999998,"5min":15.144,"15min":15.144},"last":0.56799999999999995},{"interface":"front","average":{"1min":5.2089999999999996,"5min":2.214,"15min":1.6599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":66.061999999999998,"5min":66.061999999999998,"15min":66.061999999999998},"last":0.63300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.581,"15min":1.5049999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":4.0750000000000002,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.554,"5min":1.5509999999999999,"15min":1.4850000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":4.0519999999999996,"5min":6.0590000000000002,"15min":6.0590000000000002},"last":0.5}]}]},{"osd":1,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":679936,"data_stored":279901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.92800000000000005,"15min":0.93000000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.089,"5min":1.026,"15min":1.0149999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":1.298}]},{"osd":2,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.0369999999999999,"15min":0.95499999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":14.817,"5min":14.817,"15min":14.817},"last":0.996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.028,"15min":0.98299999999999998},"min":{"1min":0.47199999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.3460000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.075,"15min":1.071},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.8680000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.1839999999999999,"15min":1.0640000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":20.439,"5min":20.439,"15min":20.439},"last":1.0469999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.4359999999999999,"15min":1.256},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":13.641999999999999,"5min":13.641999999999999,"15min":13.641999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.4670000000000001,"15min":1.3260000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":13.789999999999999,"5min":13.789999999999999,"15min":13.789999999999999},"last":1.4419999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.516,"15min":1.4419999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":14.129,"5min":14.129,"15min":14.129},"last":1.377},{"interface":"front","average":{"1min":3.399,"5min":1.8160000000000001,"15min":1.5609999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":35.747,"5min":35.747,"15min":35.747},"last":1.466}]},{"osd":6,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9359999999999999,"5min":1.5669999999999999,"15min":1.4810000000000001},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":11.545,"5min":11.545,"15min":11.545},"last":1.5920000000000001},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.5600000000000001,"15min":1.4690000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":13.56,"5min":13.56,"15min":13.56},"last":1.101}]},{"osd":7,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.629,"15min":1.4770000000000001},"min":{"1min":0.70699999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":6.2969999999999997,"5min":12.494999999999999,"15min":12.494999999999999},"last":1.405},{"interface":"front","average":{"1min":2.7120000000000002,"5min":1.6970000000000001,"15min":1.524},"min":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":31.044,"5min":31.044,"15min":31.044},"last":1.6870000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1273856,"data_stored":870269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.98399999999999999,"15min":0.97599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.4379999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.2230000000000001,"15min":1.0569999999999999},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":4.3550000000000004,"5min":17.876000000000001,"15min":17.876000000000001},"last":0.79700000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.357,"15min":1.1839999999999999},"min":{"1min":0.54200000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":5.9080000000000004,"5min":17.760999999999999,"15min":17.760999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.226,"15min":1.232},"min":{"1min":0.56100000000000005,"5min":0.38,"15min":0.38},"max":{"1min":1.9139999999999999,"5min":5.5970000000000004,"15min":5.5970000000000004},"last":0.875}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.401,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.3520000000000001,"15min":1.137},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.4169999999999998,"5min":17.981000000000002,"15min":17.981000000000002},"last":0.95599999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.6040000000000001,"15min":1.4870000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":7.0389999999999997,"5min":14.297000000000001,"15min":14.297000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":2.9910000000000001,"5min":1.8939999999999999,"15min":1.4830000000000001},"min":{"1min":0.83999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":18.347999999999999,"5min":19.422000000000001,"15min":19.422000000000001},"last":1.1919999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":1.716,"15min":1.5209999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":10.933,"5min":14.212999999999999,"15min":14.212999999999999},"last":1.169},{"interface":"front","average":{"1min":2.319,"5min":1.79,"15min":1.452},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":16.446999999999999,"5min":19.315000000000001,"15min":19.315000000000001},"last":1.292}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":4.1150000000000002,"5min":2.2349999999999999,"15min":1.677},"min":{"1min":0.81499999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":27.442,"5min":27.442,"15min":27.442},"last":1.321},{"interface":"front","average":{"1min":3.9689999999999999,"5min":2.1800000000000002,"15min":1.7629999999999999},"min":{"1min":0.70899999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":27.411999999999999,"5min":27.411999999999999,"15min":27.411999999999999},"last":1.3440000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":1.8819999999999999,"15min":1.5640000000000001},"min":{"1min":0.78700000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":7.4130000000000003,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.6819999999999999,"15min":1.6259999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":5.335,"5min":5.9269999999999996,"15min":5.9269999999999996},"last":1.262}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:39.103 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:39.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.342+0000 7f366fa00640 1 Processor -- start 2024-01-16T15:02:39.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.342+0000 7f366fa00640 1 -- start start 2024-01-16T15:02:39.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.342+0000 7f366fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 0x7f36680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f366fa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680cfc30 con 0x7f36680cf7d0 2024-01-16T15:02:39.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f366fa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680b1ee0 con 0x7f366815e3b0 2024-01-16T15:02:39.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f366fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36680b2780 con 0x7f3668143ae0 2024-01-16T15:02:39.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f365fc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f36680cf7d0 0x7f366815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41686/0 (socket says 172.21.15.141:41686) 2024-01-16T15:02:39.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f365fc00640 1 -- 172.21.15.141:0/2739096603 learned_addr learned my addr 172.21.15.141:0/2739096603 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f366d000640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 0x7f36680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021603162 0 0) 0x7f36680cfc30 con 0x7f36680cf7d0 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.343+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3648003520 con 0x7f36680cf7d0 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401244410 0 0) 0x7f3648003520 con 0x7f36680cf7d0 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 msgr2=0x7f36680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 0x7f36680cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 >> v1:172.21.15.188:6789/0 conn(0x7f366815e3b0 legacy=0x7f36680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366815e990 con 0x7f36680cf7d0 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3650001670 con 0x7f36680cf7d0 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 >> v1:172.21.15.141:6789/0 conn(0x7f36680cf7d0 legacy=0x7f366815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 0x7f36480039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3648003f20 0x7f36480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f3648005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36680b2780 con 0x7f3648004d00 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364800dc70 con 0x7f3648003f20 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364800de30 con 0x7f3668143ae0 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.344+0000 7f366da00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f3648005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f366d000640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 0x7f36480039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f365fc00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3648003f20 0x7f36480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f366da00640 1 -- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 msgr2=0x7f36480039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f366da00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 0x7f36480039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f366da00640 1 -- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3648003f20 msgr2=0x7f36480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f366da00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3648003f20 0x7f36480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f366da00640 1 -- 172.21.15.141:0/2739096603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648003520 con 0x7f3648004d00 2024-01-16T15:02:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.345+0000 7f366d000640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3668143ae0 0x7f36480039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.346+0000 7f366da00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f3648005180 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3660001e50 tx=0x7f36480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70307fd33fa74d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.346+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660003b70 con 0x7f3648004d00 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.346+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3660003cd0 con 0x7f3648004d00 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.346+0000 7f3667e00640 1 -- 172.21.15.141:0/2739096603 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660005310 con 0x7f3648004d00 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.347+0000 7f366fa00640 1 -- 172.21.15.141:0/2739096603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 msgr2=0x7f3648005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.347+0000 7f366fa00640 1 --2- 172.21.15.141:0/2739096603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f3648005180 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3660001e50 tx=0x7f36480057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.347+0000 7f366da00640 1 -- 172.21.15.141:0/2739096603 reap_dead start 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.347+0000 7f366fa00640 1 -- 172.21.15.141:0/2739096603 shutdown_connections 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.347+0000 7f366fa00640 1 -- 172.21.15.141:0/2739096603 >> 172.21.15.141:0/2739096603 conn(0x7f366806a800 msgr2=0x7f366806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.347+0000 7f366fa00640 1 -- 172.21.15.141:0/2739096603 shutdown_connections 2024-01-16T15:02:39.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.347+0000 7f366fa00640 1 -- 172.21.15.141:0/2739096603 wait complete. 2024-01-16T15:02:39.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.349+0000 7f366fa00640 1 Processor -- start 2024-01-16T15:02:39.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.349+0000 7f366fa00640 1 -- start start 2024-01-16T15:02:39.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366fa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f36680cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366fa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f366813c440 0x7f366813e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36681f3dc0 0x7f36681f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366fa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366813c9b0 con 0x7f3648004d00 2024-01-16T15:02:39.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366fa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366813cc90 con 0x7f366813c440 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366813cf70 con 0x7f36681f3dc0 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f36680cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f36680cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59842/0 (socket says 172.21.15.141:59842) 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366d000640 1 -- 172.21.15.141:0/2027007159 learned_addr learned my addr 172.21.15.141:0/2027007159 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f365fc00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f366813c440 0x7f366813e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366d000640 1 -- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36681f3dc0 msgr2=0x7f36681f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366d000640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36681f3dc0 0x7f36681f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366d000640 1 -- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f366813c440 msgr2=0x7f366813e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.350+0000 7f366d000640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f366813c440 0x7f366813e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.351+0000 7f366d000640 1 -- 172.21.15.141:0/2027007159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36681f2d30 con 0x7f3648004d00 2024-01-16T15:02:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.351+0000 7f366d000640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f36680cab50 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f36580108b0 tx=0x7f3658010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.351+0000 7f3665600640 1 -- 172.21.15.141:0/2027007159 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365800eae0 con 0x7f3648004d00 2024-01-16T15:02:39.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.351+0000 7f3665600640 1 -- 172.21.15.141:0/2027007159 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365800ec40 con 0x7f3648004d00 2024-01-16T15:02:39.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.351+0000 7f3665600640 1 -- 172.21.15.141:0/2027007159 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365801c400 con 0x7f3648004d00 2024-01-16T15:02:39.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.351+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366813dfb0 con 0x7f3648004d00 2024-01-16T15:02:39.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.351+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36681f4240 con 0x7f3648004d00 2024-01-16T15:02:39.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.352+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36680675d0 con 0x7f3648004d00 2024-01-16T15:02:39.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.353+0000 7f3665600640 1 -- 172.21.15.141:0/2027007159 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f365801c560 con 0x7f3648004d00 2024-01-16T15:02:39.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.354+0000 7f3665600640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f364006b860 0x7f364006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.354+0000 7f3665600640 1 -- 172.21.15.141:0/2027007159 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f365808ac10 con 0x7f3648004d00 2024-01-16T15:02:39.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.355+0000 7f365fc00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f364006b860 0x7f364006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.356+0000 7f365fc00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f364006b860 0x7f364006dd20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3650001c80 tx=0x7f3650001d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.359+0000 7f3665600640 1 -- 172.21.15.141:0/2027007159 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3658055310 con 0x7f3648004d00 2024-01-16T15:02:39.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.546+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f366813ce00 con 0x7f364006b860 2024-01-16T15:02:39.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.569+0000 7f3665600640 1 -- 172.21.15.141:0/2027007159 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286838 (secure 0 0 0) 0x7f366813ce00 con 0x7f364006b860 2024-01-16T15:02:39.554 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:02:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f364006b860 msgr2=0x7f364006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f364006b860 0x7f364006dd20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3650001c80 tx=0x7f3650001d50 comp rx=0 tx=0).stop 2024-01-16T15:02:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 msgr2=0x7f36680cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f36680cab50 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f36580108b0 tx=0x7f3658010d70 comp rx=0 tx=0).stop 2024-01-16T15:02:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 shutdown_connections 2024-01-16T15:02:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f364006b860 0x7f364006dd20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36681f3dc0 0x7f36681f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f366813c440 0x7f366813e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 --2- 172.21.15.141:0/2027007159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3648004d00 0x7f36680cab50 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 >> 172.21.15.141:0/2027007159 conn(0x7f366806a800 msgr2=0x7f36680702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:39.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 shutdown_connections 2024-01-16T15:02:39.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.576+0000 7f366fa00640 1 -- 172.21.15.141:0/2027007159 wait complete. 2024-01-16T15:02:39.577 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-01-16T15:02:38.587961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289464,"kb_used_data":6728,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685256,"statfs":{"total":773094113280,"available":772797702144,"internally_reserved":0,"allocated":6889472,"data_stored":3529364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.898728"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.818781+0000","last_change":"2024-01-16T15:02:29.726011+0000","last_active":"2024-01-16T15:02:31.818781+0000","last_peered":"2024-01-16T15:02:31.818781+0000","last_clean":"2024-01-16T15:02:31.818781+0000","last_became_active":"2024-01-16T15:02:29.725792+0000","last_became_peered":"2024-01-16T15:02:29.725792+0000","last_unstale":"2024-01-16T15:02:31.818781+0000","last_undegraded":"2024-01-16T15:02:31.818781+0000","last_fullsized":"2024-01-16T15:02:31.818781+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:22.758055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152259+0000","last_change":"2024-01-16T15:02:29.739850+0000","last_active":"2024-01-16T15:02:31.152259+0000","last_peered":"2024-01-16T15:02:31.152259+0000","last_clean":"2024-01-16T15:02:31.152259+0000","last_became_active":"2024-01-16T15:02:29.739589+0000","last_became_peered":"2024-01-16T15:02:29.739589+0000","last_unstale":"2024-01-16T15:02:31.152259+0000","last_undegraded":"2024-01-16T15:02:31.152259+0000","last_fullsized":"2024-01-16T15:02:31.152259+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:18.969372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727948+0000","last_change":"2024-01-16T15:02:29.717703+0000","last_active":"2024-01-16T15:02:30.727948+0000","last_peered":"2024-01-16T15:02:30.727948+0000","last_clean":"2024-01-16T15:02:30.727948+0000","last_became_active":"2024-01-16T15:02:29.716028+0000","last_became_peered":"2024-01-16T15:02:29.716028+0000","last_unstale":"2024-01-16T15:02:30.727948+0000","last_undegraded":"2024-01-16T15:02:30.727948+0000","last_fullsized":"2024-01-16T15:02:30.727948+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:05.876541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.140619+0000","last_change":"2024-01-16T15:02:29.718778+0000","last_active":"2024-01-16T15:02:31.140619+0000","last_peered":"2024-01-16T15:02:31.140619+0000","last_clean":"2024-01-16T15:02:31.140619+0000","last_became_active":"2024-01-16T15:02:29.718514+0000","last_became_peered":"2024-01-16T15:02:29.718514+0000","last_unstale":"2024-01-16T15:02:31.140619+0000","last_undegraded":"2024-01-16T15:02:31.140619+0000","last_fullsized":"2024-01-16T15:02:31.140619+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:04.026537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.818821+0000","last_change":"2024-01-16T15:02:29.713999+0000","last_active":"2024-01-16T15:02:31.818821+0000","last_peered":"2024-01-16T15:02:31.818821+0000","last_clean":"2024-01-16T15:02:31.818821+0000","last_became_active":"2024-01-16T15:02:29.713686+0000","last_became_peered":"2024-01-16T15:02:29.713686+0000","last_unstale":"2024-01-16T15:02:31.818821+0000","last_undegraded":"2024-01-16T15:02:31.818821+0000","last_fullsized":"2024-01-16T15:02:31.818821+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:15.732755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.985954+0000","last_change":"2024-01-16T15:02:29.720190+0000","last_active":"2024-01-16T15:02:31.985954+0000","last_peered":"2024-01-16T15:02:31.985954+0000","last_clean":"2024-01-16T15:02:31.985954+0000","last_became_active":"2024-01-16T15:02:29.719883+0000","last_became_peered":"2024-01-16T15:02:29.719883+0000","last_unstale":"2024-01-16T15:02:31.985954+0000","last_undegraded":"2024-01-16T15:02:31.985954+0000","last_fullsized":"2024-01-16T15:02:31.985954+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:50.012072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.724953+0000","last_change":"2024-01-16T15:02:29.717759+0000","last_active":"2024-01-16T15:02:30.724953+0000","last_peered":"2024-01-16T15:02:30.724953+0000","last_clean":"2024-01-16T15:02:30.724953+0000","last_became_active":"2024-01-16T15:02:29.717374+0000","last_became_peered":"2024-01-16T15:02:29.717374+0000","last_unstale":"2024-01-16T15:02:30.724953+0000","last_undegraded":"2024-01-16T15:02:30.724953+0000","last_fullsized":"2024-01-16T15:02:30.724953+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:47.322380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.151768+0000","last_change":"2024-01-16T15:02:29.723895+0000","last_active":"2024-01-16T15:02:31.151768+0000","last_peered":"2024-01-16T15:02:31.151768+0000","last_clean":"2024-01-16T15:02:31.151768+0000","last_became_active":"2024-01-16T15:02:29.723372+0000","last_became_peered":"2024-01-16T15:02:29.723372+0000","last_unstale":"2024-01-16T15:02:31.151768+0000","last_undegraded":"2024-01-16T15:02:31.151768+0000","last_fullsized":"2024-01-16T15:02:31.151768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:58.665311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.725904+0000","last_change":"2024-01-16T15:02:29.714212+0000","last_active":"2024-01-16T15:02:30.725904+0000","last_peered":"2024-01-16T15:02:30.725904+0000","last_clean":"2024-01-16T15:02:30.725904+0000","last_became_active":"2024-01-16T15:02:29.713918+0000","last_became_peered":"2024-01-16T15:02:29.713918+0000","last_unstale":"2024-01-16T15:02:30.725904+0000","last_undegraded":"2024-01-16T15:02:30.725904+0000","last_fullsized":"2024-01-16T15:02:30.725904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:46.224223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.490223+0000","last_change":"2024-01-16T15:02:29.726552+0000","last_active":"2024-01-16T15:02:31.490223+0000","last_peered":"2024-01-16T15:02:31.490223+0000","last_clean":"2024-01-16T15:02:31.490223+0000","last_became_active":"2024-01-16T15:02:29.726242+0000","last_became_peered":"2024-01-16T15:02:29.726242+0000","last_unstale":"2024-01-16T15:02:31.490223+0000","last_undegraded":"2024-01-16T15:02:31.490223+0000","last_fullsized":"2024-01-16T15:02:31.490223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:40.190180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.490158+0000","last_change":"2024-01-16T15:02:29.735792+0000","last_active":"2024-01-16T15:02:31.490158+0000","last_peered":"2024-01-16T15:02:31.490158+0000","last_clean":"2024-01-16T15:02:31.490158+0000","last_became_active":"2024-01-16T15:02:29.735504+0000","last_became_peered":"2024-01-16T15:02:29.735504+0000","last_unstale":"2024-01-16T15:02:31.490158+0000","last_undegraded":"2024-01-16T15:02:31.490158+0000","last_fullsized":"2024-01-16T15:02:31.490158+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:39.219185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.140638+0000","last_change":"2024-01-16T15:02:29.723315+0000","last_active":"2024-01-16T15:02:31.140638+0000","last_peered":"2024-01-16T15:02:31.140638+0000","last_clean":"2024-01-16T15:02:31.140638+0000","last_became_active":"2024-01-16T15:02:29.722945+0000","last_became_peered":"2024-01-16T15:02:29.722945+0000","last_unstale":"2024-01-16T15:02:31.140638+0000","last_undegraded":"2024-01-16T15:02:31.140638+0000","last_fullsized":"2024-01-16T15:02:31.140638+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:22.191278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.730012+0000","last_change":"2024-01-16T15:02:29.721939+0000","last_active":"2024-01-16T15:02:30.730012+0000","last_peered":"2024-01-16T15:02:30.730012+0000","last_clean":"2024-01-16T15:02:30.730012+0000","last_became_active":"2024-01-16T15:02:29.721563+0000","last_became_peered":"2024-01-16T15:02:29.721563+0000","last_unstale":"2024-01-16T15:02:30.730012+0000","last_undegraded":"2024-01-16T15:02:30.730012+0000","last_fullsized":"2024-01-16T15:02:30.730012+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:29:18.012133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.725647+0000","last_change":"2024-01-16T15:02:29.713474+0000","last_active":"2024-01-16T15:02:30.725647+0000","last_peered":"2024-01-16T15:02:30.725647+0000","last_clean":"2024-01-16T15:02:30.725647+0000","last_became_active":"2024-01-16T15:02:29.713165+0000","last_became_peered":"2024-01-16T15:02:29.713165+0000","last_unstale":"2024-01-16T15:02:30.725647+0000","last_undegraded":"2024-01-16T15:02:30.725647+0000","last_fullsized":"2024-01-16T15:02:30.725647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:06.874950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.725527+0000","last_change":"2024-01-16T15:02:29.724741+0000","last_active":"2024-01-16T15:02:30.725527+0000","last_peered":"2024-01-16T15:02:30.725527+0000","last_clean":"2024-01-16T15:02:30.725527+0000","last_became_active":"2024-01-16T15:02:29.724198+0000","last_became_peered":"2024-01-16T15:02:29.724198+0000","last_unstale":"2024-01-16T15:02:30.725527+0000","last_undegraded":"2024-01-16T15:02:30.725527+0000","last_fullsized":"2024-01-16T15:02:30.725527+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:56.917062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.985987+0000","last_change":"2024-01-16T15:02:29.720236+0000","last_active":"2024-01-16T15:02:31.985987+0000","last_peered":"2024-01-16T15:02:31.985987+0000","last_clean":"2024-01-16T15:02:31.985987+0000","last_became_active":"2024-01-16T15:02:29.719855+0000","last_became_peered":"2024-01-16T15:02:29.719855+0000","last_unstale":"2024-01-16T15:02:31.985987+0000","last_undegraded":"2024-01-16T15:02:31.985987+0000","last_fullsized":"2024-01-16T15:02:31.985987+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:27.117492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152614+0000","last_change":"2024-01-16T15:02:29.727419+0000","last_active":"2024-01-16T15:02:31.152614+0000","last_peered":"2024-01-16T15:02:31.152614+0000","last_clean":"2024-01-16T15:02:31.152614+0000","last_became_active":"2024-01-16T15:02:29.727158+0000","last_became_peered":"2024-01-16T15:02:29.727158+0000","last_unstale":"2024-01-16T15:02:31.152614+0000","last_undegraded":"2024-01-16T15:02:31.152614+0000","last_fullsized":"2024-01-16T15:02:31.152614+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:20.574101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727984+0000","last_change":"2024-01-16T15:02:29.722809+0000","last_active":"2024-01-16T15:02:30.727984+0000","last_peered":"2024-01-16T15:02:30.727984+0000","last_clean":"2024-01-16T15:02:30.727984+0000","last_became_active":"2024-01-16T15:02:29.722472+0000","last_became_peered":"2024-01-16T15:02:29.722472+0000","last_unstale":"2024-01-16T15:02:30.727984+0000","last_undegraded":"2024-01-16T15:02:30.727984+0000","last_fullsized":"2024-01-16T15:02:30.727984+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:04.026537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.818316+0000","last_change":"2024-01-16T15:02:29.731289+0000","last_active":"2024-01-16T15:02:31.818316+0000","last_peered":"2024-01-16T15:02:31.818316+0000","last_clean":"2024-01-16T15:02:31.818316+0000","last_became_active":"2024-01-16T15:02:29.730989+0000","last_became_peered":"2024-01-16T15:02:29.730989+0000","last_unstale":"2024-01-16T15:02:31.818316+0000","last_undegraded":"2024-01-16T15:02:31.818316+0000","last_fullsized":"2024-01-16T15:02:31.818316+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:55.945547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.151867+0000","last_change":"2024-01-16T15:02:29.736628+0000","last_active":"2024-01-16T15:02:31.151867+0000","last_peered":"2024-01-16T15:02:31.151867+0000","last_clean":"2024-01-16T15:02:31.151867+0000","last_became_active":"2024-01-16T15:02:29.735200+0000","last_became_peered":"2024-01-16T15:02:29.735200+0000","last_unstale":"2024-01-16T15:02:31.151867+0000","last_undegraded":"2024-01-16T15:02:31.151867+0000","last_fullsized":"2024-01-16T15:02:31.151867+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:27.266627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727074+0000","last_change":"2024-01-16T15:02:29.716691+0000","last_active":"2024-01-16T15:02:30.727074+0000","last_peered":"2024-01-16T15:02:30.727074+0000","last_clean":"2024-01-16T15:02:30.727074+0000","last_became_active":"2024-01-16T15:02:29.716366+0000","last_became_peered":"2024-01-16T15:02:29.716366+0000","last_unstale":"2024-01-16T15:02:30.727074+0000","last_undegraded":"2024-01-16T15:02:30.727074+0000","last_fullsized":"2024-01-16T15:02:30.727074+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:14:19.169249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152273+0000","last_change":"2024-01-16T15:02:29.726608+0000","last_active":"2024-01-16T15:02:31.152273+0000","last_peered":"2024-01-16T15:02:31.152273+0000","last_clean":"2024-01-16T15:02:31.152273+0000","last_became_active":"2024-01-16T15:02:29.726328+0000","last_became_peered":"2024-01-16T15:02:29.726328+0000","last_unstale":"2024-01-16T15:02:31.152273+0000","last_undegraded":"2024-01-16T15:02:31.152273+0000","last_fullsized":"2024-01-16T15:02:31.152273+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:49:37.534362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727028+0000","last_change":"2024-01-16T15:02:29.733176+0000","last_active":"2024-01-16T15:02:30.727028+0000","last_peered":"2024-01-16T15:02:30.727028+0000","last_clean":"2024-01-16T15:02:30.727028+0000","last_became_active":"2024-01-16T15:02:29.732867+0000","last_became_peered":"2024-01-16T15:02:29.732867+0000","last_unstale":"2024-01-16T15:02:30.727028+0000","last_undegraded":"2024-01-16T15:02:30.727028+0000","last_fullsized":"2024-01-16T15:02:30.727028+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:52.181001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.729433+0000","last_change":"2024-01-16T15:02:29.721477+0000","last_active":"2024-01-16T15:02:30.729433+0000","last_peered":"2024-01-16T15:02:30.729433+0000","last_clean":"2024-01-16T15:02:30.729433+0000","last_became_active":"2024-01-16T15:02:29.721137+0000","last_became_peered":"2024-01-16T15:02:29.721137+0000","last_unstale":"2024-01-16T15:02:30.729433+0000","last_undegraded":"2024-01-16T15:02:30.729433+0000","last_fullsized":"2024-01-16T15:02:30.729433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:54:15.439919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.819062+0000","last_change":"2024-01-16T15:02:29.724757+0000","last_active":"2024-01-16T15:02:31.819062+0000","last_peered":"2024-01-16T15:02:31.819062+0000","last_clean":"2024-01-16T15:02:31.819062+0000","last_became_active":"2024-01-16T15:02:29.723683+0000","last_became_peered":"2024-01-16T15:02:29.723683+0000","last_unstale":"2024-01-16T15:02:31.819062+0000","last_undegraded":"2024-01-16T15:02:31.819062+0000","last_fullsized":"2024-01-16T15:02:31.819062+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:47.814936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.140309+0000","last_change":"2024-01-16T15:02:29.719568+0000","last_active":"2024-01-16T15:02:31.140309+0000","last_peered":"2024-01-16T15:02:31.140309+0000","last_clean":"2024-01-16T15:02:31.140309+0000","last_became_active":"2024-01-16T15:02:29.719266+0000","last_became_peered":"2024-01-16T15:02:29.719266+0000","last_unstale":"2024-01-16T15:02:31.140309+0000","last_undegraded":"2024-01-16T15:02:31.140309+0000","last_fullsized":"2024-01-16T15:02:31.140309+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:37.350378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986454+0000","last_change":"2024-01-16T15:02:29.715578+0000","last_active":"2024-01-16T15:02:31.986454+0000","last_peered":"2024-01-16T15:02:31.986454+0000","last_clean":"2024-01-16T15:02:31.986454+0000","last_became_active":"2024-01-16T15:02:29.714548+0000","last_became_peered":"2024-01-16T15:02:29.714548+0000","last_unstale":"2024-01-16T15:02:31.986454+0000","last_undegraded":"2024-01-16T15:02:31.986454+0000","last_fullsized":"2024-01-16T15:02:31.986454+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:35:52.289727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.818474+0000","last_change":"2024-01-16T15:02:29.713816+0000","last_active":"2024-01-16T15:02:31.818474+0000","last_peered":"2024-01-16T15:02:31.818474+0000","last_clean":"2024-01-16T15:02:31.818474+0000","last_became_active":"2024-01-16T15:02:29.713498+0000","last_became_peered":"2024-01-16T15:02:29.713498+0000","last_unstale":"2024-01-16T15:02:31.818474+0000","last_undegraded":"2024-01-16T15:02:31.818474+0000","last_fullsized":"2024-01-16T15:02:31.818474+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:51.090017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152996+0000","last_change":"2024-01-16T15:02:29.729452+0000","last_active":"2024-01-16T15:02:31.152996+0000","last_peered":"2024-01-16T15:02:31.152996+0000","last_clean":"2024-01-16T15:02:31.152996+0000","last_became_active":"2024-01-16T15:02:29.729149+0000","last_became_peered":"2024-01-16T15:02:29.729149+0000","last_unstale":"2024-01-16T15:02:31.152996+0000","last_undegraded":"2024-01-16T15:02:31.152996+0000","last_fullsized":"2024-01-16T15:02:31.152996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:55.280824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727068+0000","last_change":"2024-01-16T15:02:29.723319+0000","last_active":"2024-01-16T15:02:30.727068+0000","last_peered":"2024-01-16T15:02:30.727068+0000","last_clean":"2024-01-16T15:02:30.727068+0000","last_became_active":"2024-01-16T15:02:29.723050+0000","last_became_peered":"2024-01-16T15:02:29.723050+0000","last_unstale":"2024-01-16T15:02:30.727068+0000","last_undegraded":"2024-01-16T15:02:30.727068+0000","last_fullsized":"2024-01-16T15:02:30.727068+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:02:53.483995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.818139+0000","last_change":"2024-01-16T15:02:29.725282+0000","last_active":"2024-01-16T15:02:31.818139+0000","last_peered":"2024-01-16T15:02:31.818139+0000","last_clean":"2024-01-16T15:02:31.818139+0000","last_became_active":"2024-01-16T15:02:29.723794+0000","last_became_peered":"2024-01-16T15:02:29.723794+0000","last_unstale":"2024-01-16T15:02:31.818139+0000","last_undegraded":"2024-01-16T15:02:31.818139+0000","last_fullsized":"2024-01-16T15:02:31.818139+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:27.326504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.985960+0000","last_change":"2024-01-16T15:02:29.736869+0000","last_active":"2024-01-16T15:02:31.985960+0000","last_peered":"2024-01-16T15:02:31.985960+0000","last_clean":"2024-01-16T15:02:31.985960+0000","last_became_active":"2024-01-16T15:02:29.735803+0000","last_became_peered":"2024-01-16T15:02:29.735803+0000","last_unstale":"2024-01-16T15:02:31.985960+0000","last_undegraded":"2024-01-16T15:02:31.985960+0000","last_fullsized":"2024-01-16T15:02:31.985960+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:30.259209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986548+0000","last_change":"2024-01-16T15:02:29.719680+0000","last_active":"2024-01-16T15:02:31.986548+0000","last_peered":"2024-01-16T15:02:31.986548+0000","last_clean":"2024-01-16T15:02:31.986548+0000","last_became_active":"2024-01-16T15:02:29.719395+0000","last_became_peered":"2024-01-16T15:02:29.719395+0000","last_unstale":"2024-01-16T15:02:31.986548+0000","last_undegraded":"2024-01-16T15:02:31.986548+0000","last_fullsized":"2024-01-16T15:02:31.986548+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:53.444325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"58'4","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.740223+0000","last_change":"2024-01-16T15:02:28.696840+0000","last_active":"2024-01-16T15:02:29.740223+0000","last_peered":"2024-01-16T15:02:29.740223+0000","last_clean":"2024-01-16T15:02:29.740223+0000","last_became_active":"2024-01-16T15:02:28.695989+0000","last_became_peered":"2024-01-16T15:02:28.695989+0000","last_unstale":"2024-01-16T15:02:29.740223+0000","last_undegraded":"2024-01-16T15:02:29.740223+0000","last_fullsized":"2024-01-16T15:02:29.740223+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:56.293921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.490348+0000","last_change":"2024-01-16T15:02:29.722881+0000","last_active":"2024-01-16T15:02:31.490348+0000","last_peered":"2024-01-16T15:02:31.490348+0000","last_clean":"2024-01-16T15:02:31.490348+0000","last_became_active":"2024-01-16T15:02:29.722447+0000","last_became_peered":"2024-01-16T15:02:29.722447+0000","last_unstale":"2024-01-16T15:02:31.490348+0000","last_undegraded":"2024-01-16T15:02:31.490348+0000","last_fullsized":"2024-01-16T15:02:31.490348+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:23:58.379661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.742000+0000","last_change":"2024-01-16T15:02:28.697447+0000","last_active":"2024-01-16T15:02:29.742000+0000","last_peered":"2024-01-16T15:02:29.742000+0000","last_clean":"2024-01-16T15:02:29.742000+0000","last_became_active":"2024-01-16T15:02:28.697122+0000","last_became_peered":"2024-01-16T15:02:28.697122+0000","last_unstale":"2024-01-16T15:02:29.742000+0000","last_undegraded":"2024-01-16T15:02:29.742000+0000","last_fullsized":"2024-01-16T15:02:29.742000+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:19:56.196522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986456+0000","last_change":"2024-01-16T15:02:29.731438+0000","last_active":"2024-01-16T15:02:31.986456+0000","last_peered":"2024-01-16T15:02:31.986456+0000","last_clean":"2024-01-16T15:02:31.986456+0000","last_became_active":"2024-01-16T15:02:29.729477+0000","last_became_peered":"2024-01-16T15:02:29.729477+0000","last_unstale":"2024-01-16T15:02:31.986456+0000","last_undegraded":"2024-01-16T15:02:31.986456+0000","last_fullsized":"2024-01-16T15:02:31.986456+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:35.213731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.742370+0000","last_change":"2024-01-16T15:02:28.689593+0000","last_active":"2024-01-16T15:02:29.742370+0000","last_peered":"2024-01-16T15:02:29.742370+0000","last_clean":"2024-01-16T15:02:29.742370+0000","last_became_active":"2024-01-16T15:02:28.689343+0000","last_became_peered":"2024-01-16T15:02:28.689343+0000","last_unstale":"2024-01-16T15:02:29.742370+0000","last_undegraded":"2024-01-16T15:02:29.742370+0000","last_fullsized":"2024-01-16T15:02:29.742370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:29.321021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986533+0000","last_change":"2024-01-16T15:02:29.723633+0000","last_active":"2024-01-16T15:02:31.986533+0000","last_peered":"2024-01-16T15:02:31.986533+0000","last_clean":"2024-01-16T15:02:31.986533+0000","last_became_active":"2024-01-16T15:02:29.723349+0000","last_became_peered":"2024-01-16T15:02:29.723349+0000","last_unstale":"2024-01-16T15:02:31.986533+0000","last_undegraded":"2024-01-16T15:02:31.986533+0000","last_fullsized":"2024-01-16T15:02:31.986533+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:30:28.117041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.733937+0000","last_change":"2024-01-16T15:02:28.694539+0000","last_active":"2024-01-16T15:02:29.733937+0000","last_peered":"2024-01-16T15:02:29.733937+0000","last_clean":"2024-01-16T15:02:29.733937+0000","last_became_active":"2024-01-16T15:02:28.694233+0000","last_became_peered":"2024-01-16T15:02:28.694233+0000","last_unstale":"2024-01-16T15:02:29.733937+0000","last_undegraded":"2024-01-16T15:02:29.733937+0000","last_fullsized":"2024-01-16T15:02:29.733937+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:46.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.151840+0000","last_change":"2024-01-16T15:02:29.722942+0000","last_active":"2024-01-16T15:02:31.151840+0000","last_peered":"2024-01-16T15:02:31.151840+0000","last_clean":"2024-01-16T15:02:31.151840+0000","last_became_active":"2024-01-16T15:02:29.722662+0000","last_became_peered":"2024-01-16T15:02:29.722662+0000","last_unstale":"2024-01-16T15:02:31.151840+0000","last_undegraded":"2024-01-16T15:02:31.151840+0000","last_fullsized":"2024-01-16T15:02:31.151840+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:50.363252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.739401+0000","last_change":"2024-01-16T15:02:28.693560+0000","last_active":"2024-01-16T15:02:29.739401+0000","last_peered":"2024-01-16T15:02:29.739401+0000","last_clean":"2024-01-16T15:02:29.739401+0000","last_became_active":"2024-01-16T15:02:28.693211+0000","last_became_peered":"2024-01-16T15:02:28.693211+0000","last_unstale":"2024-01-16T15:02:29.739401+0000","last_undegraded":"2024-01-16T15:02:29.739401+0000","last_fullsized":"2024-01-16T15:02:29.739401+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:53.457968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152750+0000","last_change":"2024-01-16T15:02:29.729985+0000","last_active":"2024-01-16T15:02:31.152750+0000","last_peered":"2024-01-16T15:02:31.152750+0000","last_clean":"2024-01-16T15:02:31.152750+0000","last_became_active":"2024-01-16T15:02:29.729662+0000","last_became_peered":"2024-01-16T15:02:29.729662+0000","last_unstale":"2024-01-16T15:02:31.152750+0000","last_undegraded":"2024-01-16T15:02:31.152750+0000","last_fullsized":"2024-01-16T15:02:31.152750+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:58.891955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.727424+0000","last_change":"2024-01-16T15:02:28.683014+0000","last_active":"2024-01-16T15:02:29.727424+0000","last_peered":"2024-01-16T15:02:29.727424+0000","last_clean":"2024-01-16T15:02:29.727424+0000","last_became_active":"2024-01-16T15:02:28.682735+0000","last_became_peered":"2024-01-16T15:02:28.682735+0000","last_unstale":"2024-01-16T15:02:29.727424+0000","last_undegraded":"2024-01-16T15:02:29.727424+0000","last_fullsized":"2024-01-16T15:02:29.727424+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:18.560295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.729490+0000","last_change":"2024-01-16T15:02:29.721954+0000","last_active":"2024-01-16T15:02:30.729490+0000","last_peered":"2024-01-16T15:02:30.729490+0000","last_clean":"2024-01-16T15:02:30.729490+0000","last_became_active":"2024-01-16T15:02:29.721647+0000","last_became_peered":"2024-01-16T15:02:29.721647+0000","last_unstale":"2024-01-16T15:02:30.729490+0000","last_undegraded":"2024-01-16T15:02:30.729490+0000","last_fullsized":"2024-01-16T15:02:30.729490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:56:15.674634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.702919+0000","last_change":"2024-01-16T15:02:28.694096+0000","last_active":"2024-01-16T15:02:29.702919+0000","last_peered":"2024-01-16T15:02:29.702919+0000","last_clean":"2024-01-16T15:02:29.702919+0000","last_became_active":"2024-01-16T15:02:28.693789+0000","last_became_peered":"2024-01-16T15:02:28.693789+0000","last_unstale":"2024-01-16T15:02:29.702919+0000","last_undegraded":"2024-01-16T15:02:29.702919+0000","last_fullsized":"2024-01-16T15:02:29.702919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:39.185686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.151883+0000","last_change":"2024-01-16T15:02:29.729503+0000","last_active":"2024-01-16T15:02:31.151883+0000","last_peered":"2024-01-16T15:02:31.151883+0000","last_clean":"2024-01-16T15:02:31.151883+0000","last_became_active":"2024-01-16T15:02:29.729123+0000","last_became_peered":"2024-01-16T15:02:29.729123+0000","last_unstale":"2024-01-16T15:02:31.151883+0000","last_undegraded":"2024-01-16T15:02:31.151883+0000","last_fullsized":"2024-01-16T15:02:31.151883+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:41.273582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.693451+0000","last_change":"2024-01-16T15:02:28.693108+0000","last_active":"2024-01-16T15:02:29.693451+0000","last_peered":"2024-01-16T15:02:29.693451+0000","last_clean":"2024-01-16T15:02:29.693451+0000","last_became_active":"2024-01-16T15:02:28.691074+0000","last_became_peered":"2024-01-16T15:02:28.691074+0000","last_unstale":"2024-01-16T15:02:29.693451+0000","last_undegraded":"2024-01-16T15:02:29.693451+0000","last_fullsized":"2024-01-16T15:02:29.693451+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:59.354392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.725732+0000","last_change":"2024-01-16T15:02:29.732853+0000","last_active":"2024-01-16T15:02:30.725732+0000","last_peered":"2024-01-16T15:02:30.725732+0000","last_clean":"2024-01-16T15:02:30.725732+0000","last_became_active":"2024-01-16T15:02:29.732508+0000","last_became_peered":"2024-01-16T15:02:29.732508+0000","last_unstale":"2024-01-16T15:02:30.725732+0000","last_undegraded":"2024-01-16T15:02:30.725732+0000","last_fullsized":"2024-01-16T15:02:30.725732+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:26.902157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.709237+0000","last_change":"2024-01-16T15:02:28.700517+0000","last_active":"2024-01-16T15:02:29.709237+0000","last_peered":"2024-01-16T15:02:29.709237+0000","last_clean":"2024-01-16T15:02:29.709237+0000","last_became_active":"2024-01-16T15:02:28.700187+0000","last_became_peered":"2024-01-16T15:02:28.700187+0000","last_unstale":"2024-01-16T15:02:29.709237+0000","last_undegraded":"2024-01-16T15:02:29.709237+0000","last_fullsized":"2024-01-16T15:02:29.709237+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:25.897663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.728494+0000","last_change":"2024-01-16T15:02:29.725064+0000","last_active":"2024-01-16T15:02:30.728494+0000","last_peered":"2024-01-16T15:02:30.728494+0000","last_clean":"2024-01-16T15:02:30.728494+0000","last_became_active":"2024-01-16T15:02:29.724732+0000","last_became_peered":"2024-01-16T15:02:29.724732+0000","last_unstale":"2024-01-16T15:02:30.728494+0000","last_undegraded":"2024-01-16T15:02:30.728494+0000","last_fullsized":"2024-01-16T15:02:30.728494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:43.319857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.713007+0000","last_change":"2024-01-16T15:02:28.694170+0000","last_active":"2024-01-16T15:02:29.713007+0000","last_peered":"2024-01-16T15:02:29.713007+0000","last_clean":"2024-01-16T15:02:29.713007+0000","last_became_active":"2024-01-16T15:02:28.693951+0000","last_became_peered":"2024-01-16T15:02:28.693951+0000","last_unstale":"2024-01-16T15:02:29.713007+0000","last_undegraded":"2024-01-16T15:02:29.713007+0000","last_fullsized":"2024-01-16T15:02:29.713007+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:24:43.955279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.489767+0000","last_change":"2024-01-16T15:02:29.728569+0000","last_active":"2024-01-16T15:02:31.489767+0000","last_peered":"2024-01-16T15:02:31.489767+0000","last_clean":"2024-01-16T15:02:31.489767+0000","last_became_active":"2024-01-16T15:02:29.728240+0000","last_became_peered":"2024-01-16T15:02:29.728240+0000","last_unstale":"2024-01-16T15:02:31.489767+0000","last_undegraded":"2024-01-16T15:02:31.489767+0000","last_fullsized":"2024-01-16T15:02:31.489767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:53.111904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"58'3","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.744942+0000","last_change":"2024-01-16T15:02:28.700598+0000","last_active":"2024-01-16T15:02:29.744942+0000","last_peered":"2024-01-16T15:02:29.744942+0000","last_clean":"2024-01-16T15:02:29.744942+0000","last_became_active":"2024-01-16T15:02:28.700296+0000","last_became_peered":"2024-01-16T15:02:28.700296+0000","last_unstale":"2024-01-16T15:02:29.744942+0000","last_undegraded":"2024-01-16T15:02:29.744942+0000","last_fullsized":"2024-01-16T15:02:29.744942+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:23:57.375167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.728213+0000","last_change":"2024-01-16T15:02:29.741879+0000","last_active":"2024-01-16T15:02:30.728213+0000","last_peered":"2024-01-16T15:02:30.728213+0000","last_clean":"2024-01-16T15:02:30.728213+0000","last_became_active":"2024-01-16T15:02:29.741643+0000","last_became_peered":"2024-01-16T15:02:29.741643+0000","last_unstale":"2024-01-16T15:02:30.728213+0000","last_undegraded":"2024-01-16T15:02:30.728213+0000","last_fullsized":"2024-01-16T15:02:30.728213+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:22.191278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"58'6","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.747087+0000","last_change":"2024-01-16T15:02:28.697153+0000","last_active":"2024-01-16T15:02:29.747087+0000","last_peered":"2024-01-16T15:02:29.747087+0000","last_clean":"2024-01-16T15:02:29.747087+0000","last_became_active":"2024-01-16T15:02:28.693885+0000","last_became_peered":"2024-01-16T15:02:28.693885+0000","last_unstale":"2024-01-16T15:02:29.747087+0000","last_undegraded":"2024-01-16T15:02:29.747087+0000","last_fullsized":"2024-01-16T15:02:29.747087+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:24.816323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.140057+0000","last_change":"2024-01-16T15:02:29.740817+0000","last_active":"2024-01-16T15:02:31.140057+0000","last_peered":"2024-01-16T15:02:31.140057+0000","last_clean":"2024-01-16T15:02:31.140057+0000","last_became_active":"2024-01-16T15:02:29.740574+0000","last_became_peered":"2024-01-16T15:02:29.740574+0000","last_unstale":"2024-01-16T15:02:31.140057+0000","last_undegraded":"2024-01-16T15:02:31.140057+0000","last_fullsized":"2024-01-16T15:02:31.140057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:14:19.169249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.745110+0000","last_change":"2024-01-16T15:02:28.695099+0000","last_active":"2024-01-16T15:02:29.745110+0000","last_peered":"2024-01-16T15:02:29.745110+0000","last_clean":"2024-01-16T15:02:29.745110+0000","last_became_active":"2024-01-16T15:02:28.694828+0000","last_became_peered":"2024-01-16T15:02:28.694828+0000","last_unstale":"2024-01-16T15:02:29.745110+0000","last_undegraded":"2024-01-16T15:02:29.745110+0000","last_fullsized":"2024-01-16T15:02:29.745110+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:03.223725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727189+0000","last_change":"2024-01-16T15:02:29.718662+0000","last_active":"2024-01-16T15:02:30.727189+0000","last_peered":"2024-01-16T15:02:30.727189+0000","last_clean":"2024-01-16T15:02:30.727189+0000","last_became_active":"2024-01-16T15:02:29.718373+0000","last_became_peered":"2024-01-16T15:02:29.718373+0000","last_unstale":"2024-01-16T15:02:30.727189+0000","last_undegraded":"2024-01-16T15:02:30.727189+0000","last_fullsized":"2024-01-16T15:02:30.727189+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:37.350378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.042253+0000","last_change":"2024-01-16T15:02:28.688577+0000","last_active":"2024-01-16T15:02:30.042253+0000","last_peered":"2024-01-16T15:02:30.042253+0000","last_clean":"2024-01-16T15:02:30.042253+0000","last_became_active":"2024-01-16T15:02:28.687999+0000","last_became_peered":"2024-01-16T15:02:28.687999+0000","last_unstale":"2024-01-16T15:02:30.042253+0000","last_undegraded":"2024-01-16T15:02:30.042253+0000","last_fullsized":"2024-01-16T15:02:30.042253+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:17.380916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.724915+0000","last_change":"2024-01-16T15:02:29.720475+0000","last_active":"2024-01-16T15:02:30.724915+0000","last_peered":"2024-01-16T15:02:30.724915+0000","last_clean":"2024-01-16T15:02:30.724915+0000","last_became_active":"2024-01-16T15:02:29.720093+0000","last_became_peered":"2024-01-16T15:02:29.720093+0000","last_unstale":"2024-01-16T15:02:30.724915+0000","last_undegraded":"2024-01-16T15:02:30.724915+0000","last_fullsized":"2024-01-16T15:02:30.724915+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:23:58.379661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.703489+0000","last_change":"2024-01-16T15:02:28.694910+0000","last_active":"2024-01-16T15:02:29.703489+0000","last_peered":"2024-01-16T15:02:29.703489+0000","last_clean":"2024-01-16T15:02:29.703489+0000","last_became_active":"2024-01-16T15:02:28.690704+0000","last_became_peered":"2024-01-16T15:02:28.690704+0000","last_unstale":"2024-01-16T15:02:29.703489+0000","last_undegraded":"2024-01-16T15:02:29.703489+0000","last_fullsized":"2024-01-16T15:02:29.703489+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:55.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727070+0000","last_change":"2024-01-16T15:02:29.733921+0000","last_active":"2024-01-16T15:02:30.727070+0000","last_peered":"2024-01-16T15:02:30.727070+0000","last_clean":"2024-01-16T15:02:30.727070+0000","last_became_active":"2024-01-16T15:02:29.733406+0000","last_became_peered":"2024-01-16T15:02:29.733406+0000","last_unstale":"2024-01-16T15:02:30.727070+0000","last_undegraded":"2024-01-16T15:02:30.727070+0000","last_fullsized":"2024-01-16T15:02:30.727070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:16.464948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"58'8","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.741445+0000","last_change":"2024-01-16T15:02:28.699544+0000","last_active":"2024-01-16T15:02:29.741445+0000","last_peered":"2024-01-16T15:02:29.741445+0000","last_clean":"2024-01-16T15:02:29.741445+0000","last_became_active":"2024-01-16T15:02:28.699169+0000","last_became_peered":"2024-01-16T15:02:28.699169+0000","last_unstale":"2024-01-16T15:02:29.741445+0000","last_undegraded":"2024-01-16T15:02:29.741445+0000","last_fullsized":"2024-01-16T15:02:29.741445+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:53.457968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.728246+0000","last_change":"2024-01-16T15:02:29.716433+0000","last_active":"2024-01-16T15:02:30.728246+0000","last_peered":"2024-01-16T15:02:30.728246+0000","last_clean":"2024-01-16T15:02:30.728246+0000","last_became_active":"2024-01-16T15:02:29.715492+0000","last_became_peered":"2024-01-16T15:02:29.715492+0000","last_unstale":"2024-01-16T15:02:30.728246+0000","last_undegraded":"2024-01-16T15:02:30.728246+0000","last_fullsized":"2024-01-16T15:02:30.728246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:28.339262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.744747+0000","last_change":"2024-01-16T15:02:28.695124+0000","last_active":"2024-01-16T15:02:29.744747+0000","last_peered":"2024-01-16T15:02:29.744747+0000","last_clean":"2024-01-16T15:02:29.744747+0000","last_became_active":"2024-01-16T15:02:28.694847+0000","last_became_peered":"2024-01-16T15:02:28.694847+0000","last_unstale":"2024-01-16T15:02:29.744747+0000","last_undegraded":"2024-01-16T15:02:29.744747+0000","last_fullsized":"2024-01-16T15:02:29.744747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:22.370575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.140361+0000","last_change":"2024-01-16T15:02:29.725472+0000","last_active":"2024-01-16T15:02:31.140361+0000","last_peered":"2024-01-16T15:02:31.140361+0000","last_clean":"2024-01-16T15:02:31.140361+0000","last_became_active":"2024-01-16T15:02:29.725153+0000","last_became_peered":"2024-01-16T15:02:29.725153+0000","last_unstale":"2024-01-16T15:02:31.140361+0000","last_undegraded":"2024-01-16T15:02:31.140361+0000","last_fullsized":"2024-01-16T15:02:31.140361+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:05.876541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.739197+0000","last_change":"2024-01-16T15:02:28.698762+0000","last_active":"2024-01-16T15:02:29.739197+0000","last_peered":"2024-01-16T15:02:29.739197+0000","last_clean":"2024-01-16T15:02:29.739197+0000","last_became_active":"2024-01-16T15:02:28.698337+0000","last_became_peered":"2024-01-16T15:02:28.698337+0000","last_unstale":"2024-01-16T15:02:29.739197+0000","last_undegraded":"2024-01-16T15:02:29.739197+0000","last_fullsized":"2024-01-16T15:02:29.739197+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:09.273766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152396+0000","last_change":"2024-01-16T15:02:29.730092+0000","last_active":"2024-01-16T15:02:31.152396+0000","last_peered":"2024-01-16T15:02:31.152396+0000","last_clean":"2024-01-16T15:02:31.152396+0000","last_became_active":"2024-01-16T15:02:29.729794+0000","last_became_peered":"2024-01-16T15:02:29.729794+0000","last_unstale":"2024-01-16T15:02:31.152396+0000","last_undegraded":"2024-01-16T15:02:31.152396+0000","last_fullsized":"2024-01-16T15:02:31.152396+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:56.817811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.733464+0000","last_change":"2024-01-16T15:02:28.695728+0000","last_active":"2024-01-16T15:02:29.733464+0000","last_peered":"2024-01-16T15:02:29.733464+0000","last_clean":"2024-01-16T15:02:29.733464+0000","last_became_active":"2024-01-16T15:02:28.695438+0000","last_became_peered":"2024-01-16T15:02:28.695438+0000","last_unstale":"2024-01-16T15:02:29.733464+0000","last_undegraded":"2024-01-16T15:02:29.733464+0000","last_fullsized":"2024-01-16T15:02:29.733464+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:25.897663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.728311+0000","last_change":"2024-01-16T15:02:29.730112+0000","last_active":"2024-01-16T15:02:30.728311+0000","last_peered":"2024-01-16T15:02:30.728311+0000","last_clean":"2024-01-16T15:02:30.728311+0000","last_became_active":"2024-01-16T15:02:29.729823+0000","last_became_peered":"2024-01-16T15:02:29.729823+0000","last_unstale":"2024-01-16T15:02:30.728311+0000","last_undegraded":"2024-01-16T15:02:30.728311+0000","last_fullsized":"2024-01-16T15:02:30.728311+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:34:08.244127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.707484+0000","last_change":"2024-01-16T15:02:28.690359+0000","last_active":"2024-01-16T15:02:29.707484+0000","last_peered":"2024-01-16T15:02:29.707484+0000","last_clean":"2024-01-16T15:02:29.707484+0000","last_became_active":"2024-01-16T15:02:28.689968+0000","last_became_peered":"2024-01-16T15:02:28.689968+0000","last_unstale":"2024-01-16T15:02:29.707484+0000","last_undegraded":"2024-01-16T15:02:29.707484+0000","last_fullsized":"2024-01-16T15:02:29.707484+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:39.963348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.139911+0000","last_change":"2024-01-16T15:02:29.725216+0000","last_active":"2024-01-16T15:02:31.139911+0000","last_peered":"2024-01-16T15:02:31.139911+0000","last_clean":"2024-01-16T15:02:31.139911+0000","last_became_active":"2024-01-16T15:02:29.724868+0000","last_became_peered":"2024-01-16T15:02:29.724868+0000","last_unstale":"2024-01-16T15:02:31.139911+0000","last_undegraded":"2024-01-16T15:02:31.139911+0000","last_fullsized":"2024-01-16T15:02:31.139911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:16.464948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.742305+0000","last_change":"2024-01-16T15:02:28.687104+0000","last_active":"2024-01-16T15:02:29.742305+0000","last_peered":"2024-01-16T15:02:29.742305+0000","last_clean":"2024-01-16T15:02:29.742305+0000","last_became_active":"2024-01-16T15:02:28.686179+0000","last_became_peered":"2024-01-16T15:02:28.686179+0000","last_unstale":"2024-01-16T15:02:29.742305+0000","last_undegraded":"2024-01-16T15:02:29.742305+0000","last_fullsized":"2024-01-16T15:02:29.742305+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:16.381808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986394+0000","last_change":"2024-01-16T15:02:29.731507+0000","last_active":"2024-01-16T15:02:31.986394+0000","last_peered":"2024-01-16T15:02:31.986394+0000","last_clean":"2024-01-16T15:02:31.986394+0000","last_became_active":"2024-01-16T15:02:29.720427+0000","last_became_peered":"2024-01-16T15:02:29.720427+0000","last_unstale":"2024-01-16T15:02:31.986394+0000","last_undegraded":"2024-01-16T15:02:31.986394+0000","last_fullsized":"2024-01-16T15:02:31.986394+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:02:07.002558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.745156+0000","last_change":"2024-01-16T15:02:28.693574+0000","last_active":"2024-01-16T15:02:29.745156+0000","last_peered":"2024-01-16T15:02:29.745156+0000","last_clean":"2024-01-16T15:02:29.745156+0000","last_became_active":"2024-01-16T15:02:28.692884+0000","last_became_peered":"2024-01-16T15:02:28.692884+0000","last_unstale":"2024-01-16T15:02:29.745156+0000","last_undegraded":"2024-01-16T15:02:29.745156+0000","last_fullsized":"2024-01-16T15:02:29.745156+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:46.605381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986063+0000","last_change":"2024-01-16T15:02:29.712151+0000","last_active":"2024-01-16T15:02:31.986063+0000","last_peered":"2024-01-16T15:02:31.986063+0000","last_clean":"2024-01-16T15:02:31.986063+0000","last_became_active":"2024-01-16T15:02:29.711793+0000","last_became_peered":"2024-01-16T15:02:29.711793+0000","last_unstale":"2024-01-16T15:02:31.986063+0000","last_undegraded":"2024-01-16T15:02:31.986063+0000","last_fullsized":"2024-01-16T15:02:31.986063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:43:54.197031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.701105+0000","last_change":"2024-01-16T15:02:28.692170+0000","last_active":"2024-01-16T15:02:29.701105+0000","last_peered":"2024-01-16T15:02:29.701105+0000","last_clean":"2024-01-16T15:02:29.701105+0000","last_became_active":"2024-01-16T15:02:28.691779+0000","last_became_peered":"2024-01-16T15:02:28.691779+0000","last_unstale":"2024-01-16T15:02:29.701105+0000","last_undegraded":"2024-01-16T15:02:29.701105+0000","last_fullsized":"2024-01-16T15:02:29.701105+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:53.457968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.140349+0000","last_change":"2024-01-16T15:02:29.717779+0000","last_active":"2024-01-16T15:02:31.140349+0000","last_peered":"2024-01-16T15:02:31.140349+0000","last_clean":"2024-01-16T15:02:31.140349+0000","last_became_active":"2024-01-16T15:02:29.717463+0000","last_became_peered":"2024-01-16T15:02:29.717463+0000","last_unstale":"2024-01-16T15:02:31.140349+0000","last_undegraded":"2024-01-16T15:02:31.140349+0000","last_fullsized":"2024-01-16T15:02:31.140349+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:07.843067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.745563+0000","last_change":"2024-01-16T15:02:28.689348+0000","last_active":"2024-01-16T15:02:29.745563+0000","last_peered":"2024-01-16T15:02:29.745563+0000","last_clean":"2024-01-16T15:02:29.745563+0000","last_became_active":"2024-01-16T15:02:28.689030+0000","last_became_peered":"2024-01-16T15:02:28.689030+0000","last_unstale":"2024-01-16T15:02:29.745563+0000","last_undegraded":"2024-01-16T15:02:29.745563+0000","last_fullsized":"2024-01-16T15:02:29.745563+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:24.816323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.140808+0000","last_change":"2024-01-16T15:02:29.716477+0000","last_active":"2024-01-16T15:02:31.140808+0000","last_peered":"2024-01-16T15:02:31.140808+0000","last_clean":"2024-01-16T15:02:31.140808+0000","last_became_active":"2024-01-16T15:02:29.716119+0000","last_became_peered":"2024-01-16T15:02:29.716119+0000","last_unstale":"2024-01-16T15:02:31.140808+0000","last_undegraded":"2024-01-16T15:02:31.140808+0000","last_fullsized":"2024-01-16T15:02:31.140808+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:28.339262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986819+0000","last_change":"2024-01-16T15:02:28.681603+0000","last_active":"2024-01-16T15:02:31.986819+0000","last_peered":"2024-01-16T15:02:31.986819+0000","last_clean":"2024-01-16T15:02:31.986819+0000","last_became_active":"2024-01-16T15:02:28.681322+0000","last_became_peered":"2024-01-16T15:02:28.681322+0000","last_unstale":"2024-01-16T15:02:31.986819+0000","last_undegraded":"2024-01-16T15:02:31.986819+0000","last_fullsized":"2024-01-16T15:02:31.986819+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:05.345755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.490183+0000","last_change":"2024-01-16T15:02:29.728538+0000","last_active":"2024-01-16T15:02:31.490183+0000","last_peered":"2024-01-16T15:02:31.490183+0000","last_clean":"2024-01-16T15:02:31.490183+0000","last_became_active":"2024-01-16T15:02:29.728211+0000","last_became_peered":"2024-01-16T15:02:29.728211+0000","last_unstale":"2024-01-16T15:02:31.490183+0000","last_undegraded":"2024-01-16T15:02:31.490183+0000","last_fullsized":"2024-01-16T15:02:31.490183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:54.462461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.042325+0000","last_change":"2024-01-16T15:02:28.681634+0000","last_active":"2024-01-16T15:02:30.042325+0000","last_peered":"2024-01-16T15:02:30.042325+0000","last_clean":"2024-01-16T15:02:30.042325+0000","last_became_active":"2024-01-16T15:02:28.680098+0000","last_became_peered":"2024-01-16T15:02:28.680098+0000","last_unstale":"2024-01-16T15:02:30.042325+0000","last_undegraded":"2024-01-16T15:02:30.042325+0000","last_fullsized":"2024-01-16T15:02:30.042325+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:45.583344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.986660+0000","last_change":"2024-01-16T15:02:29.731590+0000","last_active":"2024-01-16T15:02:31.986660+0000","last_peered":"2024-01-16T15:02:31.986660+0000","last_clean":"2024-01-16T15:02:31.986660+0000","last_became_active":"2024-01-16T15:02:29.720016+0000","last_became_peered":"2024-01-16T15:02:29.720016+0000","last_unstale":"2024-01-16T15:02:31.986660+0000","last_undegraded":"2024-01-16T15:02:31.986660+0000","last_fullsized":"2024-01-16T15:02:31.986660+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:55:30.834802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.711607+0000","last_change":"2024-01-16T15:02:28.692899+0000","last_active":"2024-01-16T15:02:29.711607+0000","last_peered":"2024-01-16T15:02:29.711607+0000","last_clean":"2024-01-16T15:02:29.711607+0000","last_became_active":"2024-01-16T15:02:28.692607+0000","last_became_peered":"2024-01-16T15:02:28.692607+0000","last_unstale":"2024-01-16T15:02:29.711607+0000","last_undegraded":"2024-01-16T15:02:29.711607+0000","last_fullsized":"2024-01-16T15:02:29.711607+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:45.219729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152244+0000","last_change":"2024-01-16T15:02:29.727241+0000","last_active":"2024-01-16T15:02:31.152244+0000","last_peered":"2024-01-16T15:02:31.152244+0000","last_clean":"2024-01-16T15:02:31.152244+0000","last_became_active":"2024-01-16T15:02:29.726906+0000","last_became_peered":"2024-01-16T15:02:29.726906+0000","last_unstale":"2024-01-16T15:02:31.152244+0000","last_undegraded":"2024-01-16T15:02:31.152244+0000","last_fullsized":"2024-01-16T15:02:31.152244+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:08.890407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.745165+0000","last_change":"2024-01-16T15:02:28.695802+0000","last_active":"2024-01-16T15:02:29.745165+0000","last_peered":"2024-01-16T15:02:29.745165+0000","last_clean":"2024-01-16T15:02:29.745165+0000","last_became_active":"2024-01-16T15:02:28.695516+0000","last_became_peered":"2024-01-16T15:02:28.695516+0000","last_unstale":"2024-01-16T15:02:29.745165+0000","last_undegraded":"2024-01-16T15:02:29.745165+0000","last_fullsized":"2024-01-16T15:02:29.745165+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:00.682347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.727805+0000","last_change":"2024-01-16T15:02:29.733231+0000","last_active":"2024-01-16T15:02:30.727805+0000","last_peered":"2024-01-16T15:02:30.727805+0000","last_clean":"2024-01-16T15:02:30.727805+0000","last_became_active":"2024-01-16T15:02:29.732954+0000","last_became_peered":"2024-01-16T15:02:29.732954+0000","last_unstale":"2024-01-16T15:02:30.727805+0000","last_undegraded":"2024-01-16T15:02:30.727805+0000","last_fullsized":"2024-01-16T15:02:30.727805+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:07.843067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.693553+0000","last_change":"2024-01-16T15:02:28.691759+0000","last_active":"2024-01-16T15:02:29.693553+0000","last_peered":"2024-01-16T15:02:29.693553+0000","last_clean":"2024-01-16T15:02:29.693553+0000","last_became_active":"2024-01-16T15:02:28.691457+0000","last_became_peered":"2024-01-16T15:02:28.691457+0000","last_unstale":"2024-01-16T15:02:29.693553+0000","last_undegraded":"2024-01-16T15:02:29.693553+0000","last_fullsized":"2024-01-16T15:02:29.693553+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:52.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.489318+0000","last_change":"2024-01-16T15:02:29.719394+0000","last_active":"2024-01-16T15:02:31.489318+0000","last_peered":"2024-01-16T15:02:31.489318+0000","last_clean":"2024-01-16T15:02:31.489318+0000","last_became_active":"2024-01-16T15:02:29.719030+0000","last_became_peered":"2024-01-16T15:02:29.719030+0000","last_unstale":"2024-01-16T15:02:31.489318+0000","last_undegraded":"2024-01-16T15:02:31.489318+0000","last_fullsized":"2024-01-16T15:02:31.489318+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:00.358886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.745145+0000","last_change":"2024-01-16T15:02:28.692453+0000","last_active":"2024-01-16T15:02:29.745145+0000","last_peered":"2024-01-16T15:02:29.745145+0000","last_clean":"2024-01-16T15:02:29.745145+0000","last_became_active":"2024-01-16T15:02:28.692132+0000","last_became_peered":"2024-01-16T15:02:28.692132+0000","last_unstale":"2024-01-16T15:02:29.745145+0000","last_undegraded":"2024-01-16T15:02:29.745145+0000","last_fullsized":"2024-01-16T15:02:29.745145+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:21:33.088637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.152252+0000","last_change":"2024-01-16T15:02:29.722517+0000","last_active":"2024-01-16T15:02:31.152252+0000","last_peered":"2024-01-16T15:02:31.152252+0000","last_clean":"2024-01-16T15:02:31.152252+0000","last_became_active":"2024-01-16T15:02:29.722220+0000","last_became_peered":"2024-01-16T15:02:29.722220+0000","last_unstale":"2024-01-16T15:02:31.152252+0000","last_undegraded":"2024-01-16T15:02:31.152252+0000","last_fullsized":"2024-01-16T15:02:31.152252+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:16:55.146736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.712883+0000","last_change":"2024-01-16T15:02:28.696929+0000","last_active":"2024-01-16T15:02:29.712883+0000","last_peered":"2024-01-16T15:02:29.712883+0000","last_clean":"2024-01-16T15:02:29.712883+0000","last_became_active":"2024-01-16T15:02:28.696616+0000","last_became_peered":"2024-01-16T15:02:28.696616+0000","last_unstale":"2024-01-16T15:02:29.712883+0000","last_undegraded":"2024-01-16T15:02:29.712883+0000","last_fullsized":"2024-01-16T15:02:29.712883+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:46.748679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.151871+0000","last_change":"2024-01-16T15:02:29.726584+0000","last_active":"2024-01-16T15:02:31.151871+0000","last_peered":"2024-01-16T15:02:31.151871+0000","last_clean":"2024-01-16T15:02:31.151871+0000","last_became_active":"2024-01-16T15:02:29.726300+0000","last_became_peered":"2024-01-16T15:02:29.726300+0000","last_unstale":"2024-01-16T15:02:31.151871+0000","last_undegraded":"2024-01-16T15:02:31.151871+0000","last_fullsized":"2024-01-16T15:02:31.151871+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:28.654459+0000","last_clean_scrub_stamp":"2024-01-16T15:02:28.654459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:42.833353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:30.042231+0000","last_change":"2024-01-16T15:02:28.695669+0000","last_active":"2024-01-16T15:02:30.042231+0000","last_peered":"2024-01-16T15:02:30.042231+0000","last_clean":"2024-01-16T15:02:30.042231+0000","last_became_active":"2024-01-16T15:02:28.695350+0000","last_became_peered":"2024-01-16T15:02:28.695350+0000","last_unstale":"2024-01-16T15:02:30.042231+0000","last_undegraded":"2024-01-16T15:02:30.042231+0000","last_fullsized":"2024-01-16T15:02:30.042231+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:27.649966+0000","last_clean_scrub_stamp":"2024-01-16T15:02:27.649966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:36.266476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":201,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:02:29.705043+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:02:29.705043+0000","last_peered":"2024-01-16T15:02:29.705043+0000","last_clean":"2024-01-16T15:02:29.705043+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:02:29.705043+0000","last_undegraded":"2024-01-16T15:02:29.705043+0000","last_fullsized":"2024-01-16T15:02:29.705043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542214,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1306624,"data_stored":883504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.0329999999999999,"5min":1.3480000000000001,"15min":1.1539999999999999},"min":{"1min":0.73899999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"last":1.375},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3089999999999999,"15min":1.2010000000000001},"min":{"1min":0.76700000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9319999999999999,"5min":2.774,"15min":2.774},"last":1.044}]},{"osd":1,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.2909999999999999,"15min":1.111},"min":{"1min":0.76400000000000001,"5min":0.63,"15min":0.63},"max":{"1min":11.487,"5min":11.487,"15min":11.487},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3380000000000001,"15min":1.1890000000000001},"min":{"1min":0.89000000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.8109999999999999,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":0.92800000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.288,"15min":1.1970000000000001},"min":{"1min":0.85599999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.7269999999999999,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":2.8940000000000001,"5min":1.6160000000000001,"15min":1.349},"min":{"1min":0.84199999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":20.577000000000002,"5min":20.577000000000002,"15min":20.577000000000002},"last":1.2809999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2639999999999998,"5min":1.6259999999999999,"15min":1.3979999999999999},"min":{"1min":1.0569999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":11.872,"5min":12.861000000000001,"15min":12.861000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.486,"15min":1.4099999999999999},"min":{"1min":0.81100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.1379999999999999,"5min":4.0720000000000001,"15min":4.0720000000000001},"last":1.1439999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.8439999999999999,"5min":1.631,"15min":1.446},"min":{"1min":0.60199999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":20.649999999999999,"5min":20.649999999999999,"15min":20.649999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":2.387,"5min":1.633,"15min":1.5109999999999999},"min":{"1min":0.94399999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":11.702,"5min":11.702,"15min":11.702},"last":1.7410000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.514,"15min":1.526},"min":{"1min":0.70499999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.8039999999999998,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.244},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.73,"15min":1.6719999999999999},"min":{"1min":1.0149999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":11.225,"5min":11.225,"15min":11.225},"last":1.5569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:05 2024","interfaces":[{"interface":"back","average":{"1min":4.1589999999999998,"5min":2.0630000000000002,"15min":1.7430000000000001},"min":{"1min":1.306,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":41.206000000000003,"5min":41.206000000000003,"15min":41.206000000000003},"last":1.4650000000000001},{"interface":"front","average":{"1min":4.125,"5min":2.0449999999999999,"15min":1.7589999999999999},"min":{"1min":1.0940000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":21.908000000000001,"5min":21.908000000000001,"15min":21.908000000000001},"last":1.6870000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542213,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36028,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335812,"statfs":{"total":96636764160,"available":96599871488,"internally_reserved":0,"allocated":704512,"data_stored":292976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.2609999999999999,"15min":1.147},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":9.1890000000000001,"5min":9.1890000000000001,"15min":9.1890000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":5.2039999999999997,"5min":1.976,"15min":1.478},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":50.472000000000001,"5min":50.472000000000001,"15min":50.472000000000001},"last":1.456}]},{"osd":1,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":1.3979999999999999,"15min":1.1459999999999999},"min":{"1min":0.66300000000000003,"5min":0.624,"15min":0.624},"max":{"1min":10.629,"5min":10.629,"15min":10.629},"last":0.95799999999999996},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.5509999999999999,"15min":1.421},"min":{"1min":0.82699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":9.3279999999999994,"5min":9.3279999999999994,"15min":9.3279999999999994},"last":1.1599999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.3789999999999996,"5min":1.9450000000000001,"15min":1.4199999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":50.643999999999998,"5min":50.643999999999998,"15min":50.643999999999998},"last":0.91100000000000003},{"interface":"front","average":{"1min":2.4169999999999998,"5min":1.4970000000000001,"15min":1.359},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":10.843,"5min":10.843,"15min":10.843},"last":1.101}]},{"osd":3,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":1.6100000000000001,"15min":1.4359999999999999},"min":{"1min":0.82399999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":15.675000000000001,"5min":15.675000000000001,"15min":15.675000000000001},"last":1.278},{"interface":"front","average":{"1min":4.3760000000000003,"5min":1.99,"15min":1.5800000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":50.561,"5min":50.561,"15min":50.561},"last":1.6080000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.8289999999999997,"5min":2.1840000000000002,"15min":1.728},"min":{"1min":0.92700000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":47.488,"5min":47.488,"15min":47.488},"last":1.3759999999999999},{"interface":"front","average":{"1min":5.2519999999999998,"5min":2.2330000000000001,"15min":1.7609999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":50.381,"5min":50.381,"15min":50.381},"last":1.738}]},{"osd":5,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":2.5419999999999998,"5min":1.6659999999999999,"15min":1.5409999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":10.654,"5min":10.654,"15min":10.654},"last":1.768},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.8280000000000001,"15min":1.726},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":9.3729999999999993,"5min":9.3729999999999993,"15min":9.3729999999999993},"last":1.5389999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:11 2024","interfaces":[{"interface":"back","average":{"1min":4.9820000000000002,"5min":2.2799999999999998,"15min":1.8779999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":52.912999999999997,"5min":52.912999999999997,"15min":52.912999999999997},"last":1.681},{"interface":"front","average":{"1min":2.5270000000000001,"5min":1.7769999999999999,"15min":1.7130000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":10.581,"5min":10.581,"15min":10.581},"last":1.879}]}]},{"osd":5,"up_from":14,"seq":60129542213,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36040,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335800,"statfs":{"total":96636764160,"available":96599859200,"internally_reserved":0,"allocated":712704,"data_stored":293628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3109999999999999,"5min":1.3580000000000001,"15min":1.248},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":13.938000000000001,"5min":13.938000000000001,"15min":13.938000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":2.9500000000000002,"5min":1.478,"15min":1.212},"min":{"1min":0.73199999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":16.896000000000001,"5min":16.896000000000001,"15min":16.896000000000001},"last":1.419}]},{"osd":1,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.482,"15min":1.353},"min":{"1min":0.77500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":3.4550000000000001,"5min":1.7549999999999999,"15min":1.345},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":35.328000000000003,"5min":35.328000000000003,"15min":35.328000000000003},"last":1.26}]},{"osd":2,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.7639999999999998,"5min":1.8859999999999999,"15min":1.47},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":35.398000000000003,"5min":35.398000000000003,"15min":35.398000000000003},"last":1.544},{"interface":"front","average":{"1min":1.782,"5min":1.3340000000000001,"15min":1.29},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.883,"5min":4.883,"15min":4.883},"last":1.3520000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8210000000000002,"5min":1.667,"15min":1.4430000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":13.202,"5min":13.202,"15min":13.202},"last":1.113},{"interface":"front","average":{"1min":3.6669999999999998,"5min":1.9670000000000001,"15min":1.587},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":35.268000000000001,"5min":35.268000000000001,"15min":35.268000000000001},"last":1.47}]},{"osd":4,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.5470000000000002,"5min":1.784,"15min":1.579},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":15.757,"5min":15.757,"15min":15.757},"last":1.0409999999999999},{"interface":"front","average":{"1min":4.3479999999999999,"5min":2.004,"15min":1.714},"min":{"1min":0.63300000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":26.166,"5min":26.166,"15min":26.166},"last":1.319}]},{"osd":6,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":4.0350000000000001,"5min":2.0950000000000002,"15min":1.768},"min":{"1min":0.53400000000000003,"5min":0.499,"15min":0.499},"max":{"1min":35.427999999999997,"5min":35.427999999999997,"15min":35.427999999999997},"last":1.8140000000000001},{"interface":"front","average":{"1min":3.4740000000000002,"5min":1.903,"15min":1.7589999999999999},"min":{"1min":0.53100000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":1.6379999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":3.9350000000000001,"5min":2.0379999999999998,"15min":1.7689999999999999},"min":{"1min":0.58099999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":19.364999999999998,"5min":19.364999999999998,"15min":19.364999999999998},"last":1.226},{"interface":"front","average":{"1min":4.0419999999999998,"5min":2.1829999999999998,"15min":1.8839999999999999},"min":{"1min":0.71699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":35.363,"5min":35.363,"15min":35.363},"last":1.7370000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542214,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36040,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335800,"statfs":{"total":96636764160,"available":96599859200,"internally_reserved":0,"allocated":716800,"data_stored":294588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.1779999999999999,"15min":1.1080000000000001},"min":{"1min":0.751,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":4.2869999999999999,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":1.111},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.4059999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":8.0440000000000005,"5min":8.0440000000000005,"15min":8.0440000000000005},"last":0.85499999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.4359999999999999,"15min":1.258},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.1050000000000004,"5min":7.1050000000000004,"15min":7.1050000000000004},"last":1.482},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.2889999999999999,"15min":1.234},"min":{"1min":0.84499999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":5.7290000000000001,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":1.0049999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.583,"15min":1.415},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.4860000000000002,"5min":15.986000000000001,"15min":15.986000000000001},"last":1.252},{"interface":"front","average":{"1min":1.554,"5min":1.3520000000000001,"15min":1.4330000000000001},"min":{"1min":0.83699999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":3.762,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":1.3100000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":3.0049999999999999,"5min":1.871,"15min":1.6819999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5600000000000001,"15min":1.48},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":4.0490000000000004,"5min":21.706,"15min":21.706},"last":1.5680000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4370000000000001,"15min":1.575},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":4.327,"5min":4.327,"15min":4.327},"last":1.224},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.649,"15min":1.6279999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":10.831,"5min":10.831,"15min":10.831},"last":1.641}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4530000000000001,"15min":1.575},"min":{"1min":0.66000000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":1.6719999999999999},{"interface":"front","average":{"1min":2.468,"5min":1.734,"15min":1.732},"min":{"1min":0.74199999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":20.939,"5min":20.939,"15min":20.939},"last":1.4259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.8,"15min":1.7490000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":7.0110000000000001,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":1.726},{"interface":"front","average":{"1min":1.976,"5min":1.6399999999999999,"15min":1.738},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":10.978999999999999,"5min":10.978999999999999,"15min":10.978999999999999},"last":1.7629999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542213,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36036,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335804,"statfs":{"total":96636764160,"available":96599863296,"internally_reserved":0,"allocated":712704,"data_stored":293032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.98499999999999999,"15min":0.91900000000000004},"min":{"1min":0.433,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1459999999999999,"15min":0.93600000000000005},"min":{"1min":0.56799999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.96899999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.92500000000000004,"15min":0.85999999999999999},"min":{"1min":0.47099999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.752,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.121,"5min":1.0620000000000001,"15min":1.016},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.075,"15min":1.075},"min":{"1min":0.59299999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9339999999999999,"5min":4.008,"15min":4.008},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.0820000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.2069999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.073}]},{"osd":4,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.3500000000000001,"15min":1.1639999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":7.7439999999999998,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.52},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.492,"15min":1.3129999999999999},"min":{"1min":0.80500000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":11.068,"5min":11.068,"15min":11.068},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4630000000000001,"15min":1.3180000000000001},"min":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":12.791,"5min":12.791,"15min":12.791},"last":1.212},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.508,"15min":1.413},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":12.106,"5min":12.106,"15min":12.106},"last":1.4630000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":1.546,"15min":1.3540000000000001},"min":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":8.1039999999999992,"5min":8.1039999999999992,"15min":8.1039999999999992},"last":1.1639999999999999},{"interface":"front","average":{"1min":2.3980000000000001,"5min":1.6359999999999999,"15min":1.4379999999999999},"min":{"1min":0.91900000000000004,"5min":0.752,"15min":0.752},"max":{"1min":6.891,"5min":6.891,"15min":6.891},"last":1.3380000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.532,"15min":1.4379999999999999},"min":{"1min":0.82599999999999996,"5min":0.755,"15min":0.755},"max":{"1min":3.7269999999999999,"5min":3.7269999999999999,"15min":3.7269999999999999},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.677,"15min":1.4299999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":7.6159999999999997,"5min":18.846,"15min":18.846},"last":1.288}]}]},{"osd":2,"up_from":14,"seq":60129542213,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36036,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335804,"statfs":{"total":96636764160,"available":96599863296,"internally_reserved":0,"allocated":712704,"data_stored":293032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":0.91600000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.839,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0049999999999999,"15min":0.93600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.353,"5min":4.5010000000000003,"15min":4.5010000000000003},"last":0.97299999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0609999999999999,"15min":1.0329999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.133,"5min":3.798,"15min":3.798},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.125,"15min":1.014},"min":{"1min":0.45800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.8069999999999999,"5min":4.8250000000000002,"15min":4.8250000000000002},"last":0.92500000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0620000000000001,"15min":0.97999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.9870000000000001,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.224,"15min":1.1399999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":8.4610000000000003},"last":1.284}]},{"osd":4,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.5489999999999999,"15min":1.3939999999999999},"min":{"1min":0.67700000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":11.843,"5min":11.843,"15min":11.843},"last":1.647},{"interface":"front","average":{"1min":1.847,"5min":1.54,"15min":1.411},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":11.768000000000001,"5min":11.768000000000001,"15min":11.768000000000001},"last":1.171}]},{"osd":5,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":1.72,"15min":1.4470000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":26.501000000000001,"5min":26.501000000000001,"15min":26.501000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.389,"15min":1.341},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.8640000000000001,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":1.349}]},{"osd":6,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.593,"15min":1.4430000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.5059999999999998,"5min":15.144,"15min":15.144},"last":1.069},{"interface":"front","average":{"1min":5.2089999999999996,"5min":2.214,"15min":1.6599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":66.061999999999998,"5min":66.061999999999998,"15min":66.061999999999998},"last":1.7270000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.581,"15min":1.5049999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":4.0750000000000002,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.5509999999999999,"15min":1.4850000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":4.0519999999999996,"5min":6.0590000000000002,"15min":6.0590000000000002},"last":1.204}]}]},{"osd":1,"up_from":14,"seq":60129542213,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36036,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335804,"statfs":{"total":96636764160,"available":96599863296,"internally_reserved":0,"allocated":708608,"data_stored":293066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.92800000000000005,"15min":0.93000000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.026,"15min":1.0149999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.999}]},{"osd":2,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.0369999999999999,"15min":0.95499999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":14.817,"5min":14.817,"15min":14.817},"last":1.726},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.028,"15min":0.98299999999999998},"min":{"1min":0.47199999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.7839999999999998,"5min":2.7839999999999998,"15min":2.7839999999999998},"last":1.0780000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.075,"15min":1.071},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.8680000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.1839999999999999,"15min":1.0640000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":20.439,"5min":20.439,"15min":20.439},"last":1.3160000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.4359999999999999,"15min":1.256},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":13.641999999999999,"5min":13.641999999999999,"15min":13.641999999999999},"last":1.879},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.4670000000000001,"15min":1.3260000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":13.789999999999999,"5min":13.789999999999999,"15min":13.789999999999999},"last":1.1990000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.516,"15min":1.4419999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":14.129,"5min":14.129,"15min":14.129},"last":1.4390000000000001},{"interface":"front","average":{"1min":3.399,"5min":1.8160000000000001,"15min":1.5609999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":35.747,"5min":35.747,"15min":35.747},"last":1.609}]},{"osd":6,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9359999999999999,"5min":1.5669999999999999,"15min":1.4810000000000001},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":11.545,"5min":11.545,"15min":11.545},"last":1.6559999999999999},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.5600000000000001,"15min":1.4690000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":13.56,"5min":13.56,"15min":13.56},"last":1.5669999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.629,"15min":1.4770000000000001},"min":{"1min":0.70699999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":6.2969999999999997,"5min":12.494999999999999,"15min":12.494999999999999},"last":1.6930000000000001},{"interface":"front","average":{"1min":2.7120000000000002,"5min":1.6970000000000001,"15min":1.524},"min":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":31.044,"5min":31.044,"15min":31.044},"last":1.5029999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542214,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":885538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.98399999999999999,"15min":0.97599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.4379999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.2230000000000001,"15min":1.0569999999999999},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":4.3550000000000004,"5min":17.876000000000001,"15min":17.876000000000001},"last":0.71799999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.357,"15min":1.1839999999999999},"min":{"1min":0.54200000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":5.9080000000000004,"5min":17.760999999999999,"15min":17.760999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.036,"5min":1.226,"15min":1.232},"min":{"1min":0.56100000000000005,"5min":0.38,"15min":0.38},"max":{"1min":1.9139999999999999,"5min":5.5970000000000004,"15min":5.5970000000000004},"last":0.64900000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.401,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.232,"5min":1.3520000000000001,"15min":1.137},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.4169999999999998,"5min":17.981000000000002,"15min":17.981000000000002},"last":0.92700000000000005}]},{"osd":4,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.6040000000000001,"15min":1.4870000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":7.0389999999999997,"5min":14.297000000000001,"15min":14.297000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":2.9910000000000001,"5min":1.8939999999999999,"15min":1.4830000000000001},"min":{"1min":0.83999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":18.347999999999999,"5min":19.422000000000001,"15min":19.422000000000001},"last":1.1000000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":1.716,"15min":1.5209999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":10.933,"5min":14.212999999999999,"15min":14.212999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":2.319,"5min":1.79,"15min":1.452},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":16.446999999999999,"5min":19.315000000000001,"15min":19.315000000000001},"last":1.161}]},{"osd":6,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":4.1150000000000002,"5min":2.2349999999999999,"15min":1.677},"min":{"1min":0.81499999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":27.442,"5min":27.442,"15min":27.442},"last":1.1319999999999999},{"interface":"front","average":{"1min":3.9689999999999999,"5min":2.1800000000000002,"15min":1.7629999999999999},"min":{"1min":0.70899999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":27.411999999999999,"5min":27.411999999999999,"15min":27.411999999999999},"last":1.006}]},{"osd":7,"last update":"Tue Jan 16 15:02:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":1.8819999999999999,"15min":1.5640000000000001},"min":{"1min":0.78700000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":7.4130000000000003,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.6819999999999999,"15min":1.6259999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":5.335,"5min":5.9269999999999996,"15min":5.9269999999999996},"last":0.86599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:39.582 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:02:39.582 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:02:39.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.865+0000 7f541cc00640 1 Processor -- start 2024-01-16T15:02:39.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.865+0000 7f541cc00640 1 -- start start 2024-01-16T15:02:39.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.866+0000 7f541cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5418159d50 0x7f54180cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.866+0000 7f541cc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f54180b3df0 0x7f54180cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.866+0000 7f541cc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f54180cfa50 0x7f54180713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.866+0000 7f541cc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54180b48b0 con 0x7f54180b3df0 2024-01-16T15:02:39.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.866+0000 7f541cc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54180cd3c0 con 0x7f54180cfa50 2024-01-16T15:02:39.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.866+0000 7f541cc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54180cfec0 con 0x7f5418159d50 2024-01-16T15:02:39.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f54180b3df0 0x7f54180cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f54180b3df0 0x7f54180cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59854/0 (socket says 172.21.15.141:59854) 2024-01-16T15:02:39.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 -- 172.21.15.141:0/3918244461 learned_addr learned my addr 172.21.15.141:0/3918244461 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:39.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5417e00640 1 --2- 172.21.15.141:0/3918244461 >> v2:172.21.15.188:3300/0 conn(0x7f54180cfa50 0x7f54180713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5417400640 1 --2- 172.21.15.141:0/3918244461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5418159d50 0x7f54180cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 -- 172.21.15.141:0/3918244461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5418159d50 msgr2=0x7f54180cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:39.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 --2- 172.21.15.141:0/3918244461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5418159d50 0x7f54180cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 -- 172.21.15.141:0/3918244461 >> v2:172.21.15.188:3300/0 conn(0x7f54180cfa50 msgr2=0x7f54180713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:39.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 --2- 172.21.15.141:0/3918244461 >> v2:172.21.15.188:3300/0 conn(0x7f54180cfa50 0x7f54180713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.867+0000 7f5416a00640 1 -- 172.21.15.141:0/3918244461 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54180ce790 con 0x7f54180b3df0 2024-01-16T15:02:39.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.868+0000 7f5416a00640 1 --2- 172.21.15.141:0/3918244461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f54180cf510 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f54180b4e00 tx=0x7f540002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b279212792ffdf1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f5416000640 1 -- 172.21.15.141:0/3918244461 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540002ca70 con 0x7f54180b3df0 2024-01-16T15:02:39.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f5416000640 1 -- 172.21.15.141:0/3918244461 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540002cbd0 con 0x7f54180b3df0 2024-01-16T15:02:39.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f5416000640 1 -- 172.21.15.141:0/3918244461 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54000343f0 con 0x7f54180b3df0 2024-01-16T15:02:39.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f541cc00640 1 -- 172.21.15.141:0/3918244461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 msgr2=0x7f54180cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f541cc00640 1 --2- 172.21.15.141:0/3918244461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f54180cf510 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f54180b4e00 tx=0x7f540002bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f541cc00640 1 -- 172.21.15.141:0/3918244461 shutdown_connections 2024-01-16T15:02:39.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f541cc00640 1 --2- 172.21.15.141:0/3918244461 >> v2:172.21.15.188:3300/0 conn(0x7f54180cfa50 0x7f54180713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f541cc00640 1 --2- 172.21.15.141:0/3918244461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f54180cf510 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f541cc00640 1 --2- 172.21.15.141:0/3918244461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5418159d50 0x7f54180cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.869+0000 7f541cc00640 1 -- 172.21.15.141:0/3918244461 >> 172.21.15.141:0/3918244461 conn(0x7f54180b1dd0 msgr2=0x7f54180b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:39.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.870+0000 7f541cc00640 1 -- 172.21.15.141:0/3918244461 shutdown_connections 2024-01-16T15:02:39.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.870+0000 7f541cc00640 1 -- 172.21.15.141:0/3918244461 wait complete. 2024-01-16T15:02:39.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.871+0000 7f541cc00640 1 Processor -- start 2024-01-16T15:02:39.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.871+0000 7f541cc00640 1 -- start start 2024-01-16T15:02:39.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.872+0000 7f541cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f5418145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.872+0000 7f541cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54180cfa50 0x7f5418144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.872+0000 7f541cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5418159d50 0x7f5418143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.872+0000 7f541cc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5418145840 con 0x7f54180b3df0 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.872+0000 7f541cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5418146210 con 0x7f5418159d50 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.872+0000 7f541cc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5418144c30 con 0x7f54180cfa50 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f5418145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f5418145cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59870/0 (socket says 172.21.15.141:59870) 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 -- 172.21.15.141:0/1568449320 learned_addr learned my addr 172.21.15.141:0/1568449320 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 -- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54180cfa50 msgr2=0x7f5418144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54180cfa50 0x7f5418144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417e00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5418159d50 0x7f5418143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 -- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5418159d50 msgr2=0x7f5418143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5418159d50 0x7f5418143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 -- 172.21.15.141:0/1568449320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5400009b30 con 0x7f54180b3df0 2024-01-16T15:02:39.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.873+0000 7f5417400640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f5418145cd0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f540800d5f0 tx=0x7f540800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.874+0000 7f540f400640 1 -- 172.21.15.141:0/1568449320 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54080125f0 con 0x7f54180b3df0 2024-01-16T15:02:39.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.874+0000 7f540f400640 1 -- 172.21.15.141:0/1568449320 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5408012750 con 0x7f54180b3df0 2024-01-16T15:02:39.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.874+0000 7f540f400640 1 -- 172.21.15.141:0/1568449320 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408012b00 con 0x7f54180b3df0 2024-01-16T15:02:39.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.874+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5418144f10 con 0x7f54180b3df0 2024-01-16T15:02:39.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.874+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54181454d0 con 0x7f54180b3df0 2024-01-16T15:02:39.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.875+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5418067b60 con 0x7f54180b3df0 2024-01-16T15:02:39.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.876+0000 7f540f400640 1 -- 172.21.15.141:0/1568449320 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f5408012c60 con 0x7f54180b3df0 2024-01-16T15:02:39.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.877+0000 7f540f400640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53f006b890 0x7f53f006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.877+0000 7f540f400640 1 -- 172.21.15.141:0/1568449320 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f540808aac0 con 0x7f54180b3df0 2024-01-16T15:02:39.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.877+0000 7f5416a00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53f006b890 0x7f53f006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.878+0000 7f5416a00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53f006b890 0x7f53f006dd50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5400035000 tx=0x7f540002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:39.882+0000 7f540f400640 1 -- 172.21.15.141:0/1568449320 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5408055080 con 0x7f54180b3df0 2024-01-16T15:02:40.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.137+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f54180b4600 con 0x7f54180b3df0 2024-01-16T15:02:40.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.141+0000 7f540f400640 1 -- 172.21.15.141:0/1568449320 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f54080586a0 con 0x7f54180b3df0 2024-01-16T15:02:40.122 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T15:02:40.122 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T15:02:40.122 INFO:teuthology.orchestra.run.smithi141.stdout: caps mds = "allow" 2024-01-16T15:02:40.122 INFO:teuthology.orchestra.run.smithi141.stdout: caps mon = "allow r" 2024-01-16T15:02:40.122 INFO:teuthology.orchestra.run.smithi141.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:02:40.122 INFO:teuthology.orchestra.run.smithi141.stderr:updated caps for client.0 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53f006b890 msgr2=0x7f53f006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53f006b890 0x7f53f006dd50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5400035000 tx=0x7f540002e000 comp rx=0 tx=0).stop 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 msgr2=0x7f5418145cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f5418145cd0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f540800d5f0 tx=0x7f540800dab0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 shutdown_connections 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53f006b890 0x7f53f006dd50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5418159d50 0x7f5418143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54180cfa50 0x7f5418144400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 --2- 172.21.15.141:0/1568449320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54180b3df0 0x7f5418145cd0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 >> 172.21.15.141:0/1568449320 conn(0x7f54180b1dd0 msgr2=0x7f5418071b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 shutdown_connections 2024-01-16T15:02:40.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.144+0000 7f541cc00640 1 -- 172.21.15.141:0/1568449320 wait complete. 2024-01-16T15:02:40.145 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:40.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.371+0000 7f4e34a00640 1 Processor -- start 2024-01-16T15:02:40.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.371+0000 7f4e34a00640 1 -- start start 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.372+0000 7f4e34a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e300cc500 0x7f4e300cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.372+0000 7f4e34a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e3006d160 0x7f4e3006d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.372+0000 7f4e34a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e30070e50 con 0x7f4e30143c40 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.372+0000 7f4e34a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e300723c0 con 0x7f4e3006d160 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.372+0000 7f4e34a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30144020 con 0x7f4e300cc500 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.373+0000 7f4e2fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e3006d160 0x7f4e3006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.373+0000 7f4e2fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4e3006d160 0x7f4e3006d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59874/0 (socket says 172.21.15.141:59874) 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.373+0000 7f4e2fe00640 1 -- 172.21.15.141:0/2006440666 learned_addr learned my addr 172.21.15.141:0/2006440666 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.373+0000 7f4e2fe00640 1 -- 172.21.15.141:0/2006440666 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e300cc500 msgr2=0x7f4e300cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.373+0000 7f4e2fe00640 1 --2- 172.21.15.141:0/2006440666 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e300cc500 0x7f4e300cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.373+0000 7f4e2fe00640 1 -- 172.21.15.141:0/2006440666 >> v1:172.21.15.188:6789/0 conn(0x7f4e30143c40 legacy=0x7f4e30071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.373+0000 7f4e2fe00640 1 -- 172.21.15.141:0/2006440666 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e300ccea0 con 0x7f4e3006d160 2024-01-16T15:02:40.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.375+0000 7f4e2fe00640 1 --2- 172.21.15.141:0/2006440666 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e3006d590 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009e50 tx=0x7f4e2802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb8cf49757fdc04 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:40.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.375+0000 7f4e2ea00640 1 -- 172.21.15.141:0/2006440666 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2802c9e0 con 0x7f4e3006d160 2024-01-16T15:02:40.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.375+0000 7f4e2ea00640 1 -- 172.21.15.141:0/2006440666 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2802cb40 con 0x7f4e3006d160 2024-01-16T15:02:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.375+0000 7f4e2ea00640 1 -- 172.21.15.141:0/2006440666 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e280343f0 con 0x7f4e3006d160 2024-01-16T15:02:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.375+0000 7f4e34a00640 1 -- 172.21.15.141:0/2006440666 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 msgr2=0x7f4e3006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.375+0000 7f4e34a00640 1 --2- 172.21.15.141:0/2006440666 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e3006d590 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009e50 tx=0x7f4e2802be40 comp rx=0 tx=0).stop 2024-01-16T15:02:40.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.376+0000 7f4e34a00640 1 -- 172.21.15.141:0/2006440666 shutdown_connections 2024-01-16T15:02:40.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.376+0000 7f4e34a00640 1 --2- 172.21.15.141:0/2006440666 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e3006d590 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.376+0000 7f4e34a00640 1 --2- 172.21.15.141:0/2006440666 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e300cc500 0x7f4e300cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.376+0000 7f4e34a00640 1 -- 172.21.15.141:0/2006440666 >> 172.21.15.141:0/2006440666 conn(0x7f4e3006a800 msgr2=0x7f4e3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:40.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.376+0000 7f4e34a00640 1 -- 172.21.15.141:0/2006440666 shutdown_connections 2024-01-16T15:02:40.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.376+0000 7f4e34a00640 1 -- 172.21.15.141:0/2006440666 wait complete. 2024-01-16T15:02:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.377+0000 7f4e34a00640 1 Processor -- start 2024-01-16T15:02:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.378+0000 7f4e34a00640 1 -- start start 2024-01-16T15:02:40.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.378+0000 7f4e34a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e300ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.378+0000 7f4e34a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e300cc500 0x7f4e300acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.378+0000 7f4e34a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e30143c40 0x7f4e3013d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.378+0000 7f4e34a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300ae370 con 0x7f4e3006d160 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.378+0000 7f4e34a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300ad470 con 0x7f4e300cc500 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.378+0000 7f4e34a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301f0e80 con 0x7f4e30143c40 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e300ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e300ab800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59880/0 (socket says 172.21.15.141:59880) 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 -- 172.21.15.141:0/830939605 learned_addr learned my addr 172.21.15.141:0/830939605 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 -- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e30143c40 msgr2=0x7f4e3013d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e30143c40 0x7f4e3013d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 -- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e300cc500 msgr2=0x7f4e300acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e300cc500 0x7f4e300acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 -- 172.21.15.141:0/830939605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e28009b30 con 0x7f4e3006d160 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.379+0000 7f4e2f400640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e300ab800 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000b3f0 tx=0x7f4e2000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.380+0000 7f4e27e00640 1 -- 172.21.15.141:0/830939605 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e20011830 con 0x7f4e3006d160 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.380+0000 7f4e27e00640 1 -- 172.21.15.141:0/830939605 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e20011990 con 0x7f4e3006d160 2024-01-16T15:02:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.380+0000 7f4e27e00640 1 -- 172.21.15.141:0/830939605 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e20011d40 con 0x7f4e3006d160 2024-01-16T15:02:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.380+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e300ad6f0 con 0x7f4e3006d160 2024-01-16T15:02:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.380+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e300adcb0 con 0x7f4e3006d160 2024-01-16T15:02:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.381+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e300ae810 con 0x7f4e3006d160 2024-01-16T15:02:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.382+0000 7f4e27e00640 1 -- 172.21.15.141:0/830939605 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f4e2001a400 con 0x7f4e3006d160 2024-01-16T15:02:40.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.383+0000 7f4e27e00640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e0806b860 0x7f4e0806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.383+0000 7f4e27e00640 1 -- 172.21.15.141:0/830939605 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e2008a530 con 0x7f4e3006d160 2024-01-16T15:02:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.383+0000 7f4e26800640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e0806b860 0x7f4e0806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.385+0000 7f4e26800640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e0806b860 0x7f4e0806dd20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4e300c6420 tx=0x7f4e18006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:40.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.387+0000 7f4e27e00640 1 -- 172.21.15.141:0/830939605 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e20054af0 con 0x7f4e3006d160 2024-01-16T15:02:40.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.617+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e30067450 con 0x7f4e3006d160 2024-01-16T15:02:40.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.618+0000 7f4e27e00640 1 -- 172.21.15.141:0/830939605 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f4e20058110 con 0x7f4e3006d160 2024-01-16T15:02:40.599 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:02:40.599 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/916741426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":916741426},{"type":"v1","addr":"172.21.15.188:6837","nonce":916741426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5332,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3966594812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3966594812},{"type":"v1","addr":"172.21.15.141:6833","nonce":3966594812}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3648732517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3648732517},{"type":"v1","addr":"172.21.15.188:6835","nonce":3648732517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:02:29.121311+0000","modified":"2024-01-16T15:02:32.205611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1757492142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":1757492142},{"type":"v1","addr":"172.21.15.141:6835","nonce":1757492142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:02:40.599 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 37 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e0806b860 msgr2=0x7f4e0806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e0806b860 0x7f4e0806dd20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4e300c6420 tx=0x7f4e18006c90 comp rx=0 tx=0).stop 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 msgr2=0x7f4e300ab800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e300ab800 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000b3f0 tx=0x7f4e2000b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 shutdown_connections 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4e0806b860 0x7f4e0806dd20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4e30143c40 0x7f4e3013d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e300cc500 0x7f4e300acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 --2- 172.21.15.141:0/830939605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e3006d160 0x7f4e300ab800 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.621+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 >> 172.21.15.141:0/830939605 conn(0x7f4e3006a800 msgr2=0x7f4e300aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:40.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.622+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 shutdown_connections 2024-01-16T15:02:40.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:40.622+0000 7f4e34a00640 1 -- 172.21.15.141:0/830939605 wait complete. 2024-01-16T15:02:41.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:02:29.121311+0000', 'modified': '2024-01-16T15:02:32.205611+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6360}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6360': {'gid': 6360, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6835/1757492142', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6834', 'nonce': 1757492142}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 1757492142}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:02:41.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:02:41.024 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T15:02:41.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.263+0000 7fddbbe00640 1 Processor -- start 2024-01-16T15:02:41.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.263+0000 7fddbbe00640 1 -- start start 2024-01-16T15:02:41.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddbbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fddbc15afb0 0x7fddbc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddbbe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc13ff30 con 0x7fddbc13fae0 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddbbe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0cdc40 con 0x7fddbc13cfa0 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddbbe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddbc0cdec0 con 0x7fddbc15afb0 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddb3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fddbc15afb0 0x7fddbc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddb3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fddbc15afb0 0x7fddbc13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59882/0 (socket says 172.21.15.141:59882) 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddbb400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fddbc13fae0 0x7fddbc0b2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:41696/0 (socket says 172.21.15.141:41696) 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.265+0000 7fddb3e00640 1 -- 172.21.15.141:0/1944861186 learned_addr learned my addr 172.21.15.141:0/1944861186 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.266+0000 7fddb3e00640 1 -- 172.21.15.141:0/1944861186 >> v1:172.21.15.188:6789/0 conn(0x7fddbc13cfa0 legacy=0x7fddbc0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.266+0000 7fddb3e00640 1 -- 172.21.15.141:0/1944861186 >> v1:172.21.15.141:6789/0 conn(0x7fddbc13fae0 legacy=0x7fddbc0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.266+0000 7fddb3e00640 1 -- 172.21.15.141:0/1944861186 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc0ce1a0 con 0x7fddbc15afb0 2024-01-16T15:02:41.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.267+0000 7fddb3e00640 1 --2- 172.21.15.141:0/1944861186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc15afb0 0x7fddbc13f5a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009870 tx=0x7fdda402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d9feeb0a78b69a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.267+0000 7fddba000640 1 -- 172.21.15.141:0/1944861186 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4031870 con 0x7fddbc15afb0 2024-01-16T15:02:41.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.267+0000 7fddba000640 1 -- 172.21.15.141:0/1944861186 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda40319d0 con 0x7fddbc15afb0 2024-01-16T15:02:41.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.267+0000 7fddba000640 1 -- 172.21.15.141:0/1944861186 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4031d60 con 0x7fddbc15afb0 2024-01-16T15:02:41.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.267+0000 7fddbbe00640 1 -- 172.21.15.141:0/1944861186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc15afb0 msgr2=0x7fddbc13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.267+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1944861186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc15afb0 0x7fddbc13f5a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009870 tx=0x7fdda402b910 comp rx=0 tx=0).stop 2024-01-16T15:02:41.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.268+0000 7fddbbe00640 1 -- 172.21.15.141:0/1944861186 shutdown_connections 2024-01-16T15:02:41.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.268+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1944861186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc15afb0 0x7fddbc13f5a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.268+0000 7fddbbe00640 1 -- 172.21.15.141:0/1944861186 >> 172.21.15.141:0/1944861186 conn(0x7fddbc06a800 msgr2=0x7fddbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.268+0000 7fddbbe00640 1 -- 172.21.15.141:0/1944861186 shutdown_connections 2024-01-16T15:02:41.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.268+0000 7fddbbe00640 1 -- 172.21.15.141:0/1944861186 wait complete. 2024-01-16T15:02:41.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.269+0000 7fddbbe00640 1 Processor -- start 2024-01-16T15:02:41.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.270+0000 7fddbbe00640 1 -- start start 2024-01-16T15:02:41.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.270+0000 7fddbbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc13cfa0 0x7fddbc0ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.270+0000 7fddbbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc13fae0 0x7fddbc15c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbbe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fddbc15afb0 0x7fddbc15cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbbe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc15bcc0 con 0x7fddbc13cfa0 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbbe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc15b640 con 0x7fddbc13fae0 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbbe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc15d070 con 0x7fddbc15afb0 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc13cfa0 0x7fddbc0ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc13cfa0 0x7fddbc0ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59890/0 (socket says 172.21.15.141:59890) 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 -- 172.21.15.141:0/1584981696 learned_addr learned my addr 172.21.15.141:0/1584981696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 -- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fddbc15afb0 msgr2=0x7fddbc15cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fddbc15afb0 0x7fddbc15cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 -- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc13fae0 msgr2=0x7fddbc15c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc13fae0 0x7fddbc15c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 -- 172.21.15.141:0/1584981696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda40094d0 con 0x7fddbc13cfa0 2024-01-16T15:02:41.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.271+0000 7fddbaa00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc13cfa0 0x7fddbc0ae760 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c430 tx=0x7fddac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.272+0000 7fddb2a00640 1 -- 172.21.15.141:0/1584981696 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac011790 con 0x7fddbc13cfa0 2024-01-16T15:02:41.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.272+0000 7fddb2a00640 1 -- 172.21.15.141:0/1584981696 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac0118f0 con 0x7fddbc13cfa0 2024-01-16T15:02:41.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.272+0000 7fddb2a00640 1 -- 172.21.15.141:0/1584981696 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac011ca0 con 0x7fddbc13cfa0 2024-01-16T15:02:41.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.272+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc15d310 con 0x7fddbc13cfa0 2024-01-16T15:02:41.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.272+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc15d5f0 con 0x7fddbc13cfa0 2024-01-16T15:02:41.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.275+0000 7fddb2a00640 1 -- 172.21.15.141:0/1584981696 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fddac01c460 con 0x7fddbc13cfa0 2024-01-16T15:02:41.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.275+0000 7fddb2a00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdd9406b860 0x7fdd9406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.276+0000 7fddb2a00640 1 -- 172.21.15.141:0/1584981696 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddac08a560 con 0x7fddbc13cfa0 2024-01-16T15:02:41.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.276+0000 7fddb3e00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdd9406b860 0x7fdd9406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.276+0000 7fddc444b740 1 Processor -- start 2024-01-16T15:02:41.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.277+0000 7fddc444b740 1 -- start start 2024-01-16T15:02:41.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.277+0000 7fddc444b740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b0a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b09270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 0x558ef6b4d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddc444b740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef6a68880 con 0x558ef6b0a070 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef6a77ae0 con 0x558ef6b04d70 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ef6a751d0 con 0x558ef6b097b0 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddb3e00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdd9406b860 0x7fdd9406dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b0a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b0a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59898/0 (socket says 172.21.15.141:59898) 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 -- 172.21.15.141:0/2526344188 learned_addr learned my addr 172.21.15.141:0/2526344188 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 -- 172.21.15.141:0/2526344188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 msgr2=0x558ef6b4d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddb3e00640 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b09270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 0x558ef6b4d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 -- 172.21.15.141:0/2526344188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 msgr2=0x558ef6b09270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b09270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.278+0000 7fddbaa00640 1 -- 172.21.15.141:0/2526344188 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ef6a38860 con 0x558ef6b0a070 2024-01-16T15:02:41.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.279+0000 7fddbaa00640 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b0a450 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fddac058840 tx=0x7fddac0587b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e74de10bb6b5538 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.279+0000 7fddab400640 1 -- 172.21.15.141:0/2526344188 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac088180 con 0x558ef6b0a070 2024-01-16T15:02:41.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.279+0000 7fddab400640 1 -- 172.21.15.141:0/2526344188 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac070180 con 0x558ef6b0a070 2024-01-16T15:02:41.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.279+0000 7fddab400640 1 -- 172.21.15.141:0/2526344188 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac054fc0 con 0x558ef6b0a070 2024-01-16T15:02:41.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.279+0000 7fddc444b740 1 -- 172.21.15.141:0/2526344188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 msgr2=0x558ef6b0a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.279+0000 7fddc444b740 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b0a450 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fddac058840 tx=0x7fddac0587b0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.279+0000 7fddc444b740 1 -- 172.21.15.141:0/2526344188 shutdown_connections 2024-01-16T15:02:41.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.280+0000 7fddc444b740 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 0x558ef6b4d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.280+0000 7fddc444b740 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b09270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.280+0000 7fddc444b740 1 --2- 172.21.15.141:0/2526344188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b0a450 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.280+0000 7fddc444b740 1 -- 172.21.15.141:0/2526344188 >> 172.21.15.141:0/2526344188 conn(0x558ef6acbc70 msgr2=0x558ef6b4fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.280+0000 7fddc444b740 1 -- 172.21.15.141:0/2526344188 shutdown_connections 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.280+0000 7fddc444b740 1 -- 172.21.15.141:0/2526344188 wait complete. 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.280+0000 7fddc444b740 1 -- start start 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.281+0000 7fddc444b740 1 Processor -- start 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.281+0000 7fddc444b740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6b48390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b46c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 0x558ef6b2ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddc444b740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef6a70ac0 con 0x558ef6b04d70 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef6a0d2f0 con 0x558ef6b097b0 2024-01-16T15:02:41.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ef6a2ac70 con 0x558ef6b0a070 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddc444b740 10 client.0 _ll_register_callbacks cb 0x558ef6a2f3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6b48390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6b48390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59902/0 (socket says 172.21.15.141:59902) 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 -- 172.21.15.141:0/3373299715 learned_addr learned my addr 172.21.15.141:0/3373299715 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddb3e00640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b46c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbb400640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 0x558ef6b2ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 -- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 msgr2=0x558ef6b2ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 0x558ef6b2ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 -- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 msgr2=0x558ef6b46c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b46c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddbaa00640 1 -- 172.21.15.141:0/3373299715 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc15d5f0 con 0x558ef6b04d70 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.282+0000 7fddb3e00640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b46c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fdda3e00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddbaa00640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6b48390 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fddac058840 tx=0x7fddac01e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f418c5e93cd782 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fdda3e00640 1 -- 172.21.15.141:0/3373299715 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac059de0 con 0x558ef6b04d70 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ef6a69aa0 con 0x558ef6b04d70 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fdda3e00640 1 -- 172.21.15.141:0/3373299715 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddac057da0 con 0x558ef6b04d70 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fdda3e00640 1 -- 172.21.15.141:0/3373299715 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddac077270 con 0x558ef6b04d70 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 10 client.6552 fetch_fsmap learned FSMap version 37 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x558ef6a6f140 con 0x558ef6b04d70 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fdda3e00640 1 -- 172.21.15.141:0/3373299715 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558ef6a69aa0 con 0x558ef6b04d70 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fdda3e00640 1 -- 172.21.15.141:0/3373299715 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fddac060720 con 0x558ef6b04d70 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 10 client.6552 fetch_fsmap finished waiting for FSMap version 37 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 10 client.6552 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 20 client.6552 populate_metadata read hostname 'smithi141' 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b47250 0x558ef6b53420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 4 client.6552 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.283+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 --> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ef6a697c0 con 0x558ef6b47250 2024-01-16T15:02:41.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.284+0000 7fddb3e00640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b47250 0x558ef6b53420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.284+0000 7fddb3e00640 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b47250 0x558ef6b53420 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fd04051231d62e1 server_cookie=190e4b248e95c167 in_seq=0 out_seq=0 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.284+0000 7fdda3e00640 10 client.6552 ms_handle_connect on v2:172.21.15.141:6834/1757492142 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.285+0000 7fdda3e00640 1 -- 172.21.15.141:0/3373299715 <== mds.0 v2:172.21.15.141:6834/1757492142 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558ef6a697c0 con 0x558ef6b47250 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.285+0000 7fdda3e00640 10 client.6552 handle_command_reply: tid=0 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.285+0000 7fddc444b740 1 client.6552 shutdown 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 msgr2=0x558ef6b48390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6b48390 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fddac058840 tx=0x7fddac01e070 comp rx=0 tx=0).stop 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 shutdown_connections 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b47250 0x558ef6b53420 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 0x558ef6b2ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b46c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 --2- 172.21.15.141:0/3373299715 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6b48390 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fdda3e00640 0 client.6552 ms_handle_reset on v2:172.21.15.141:6834/1757492142 2024-01-16T15:02:41.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 >> 172.21.15.141:0/3373299715 conn(0x558ef6acbc70 msgr2=0x558ef6b03dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 shutdown_connections 2024-01-16T15:02:41.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 1 -- 172.21.15.141:0/3373299715 wait complete. 2024-01-16T15:02:41.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.286+0000 7fddc444b740 20 client.6552 trim_cache size 0 max 16384 2024-01-16T15:02:41.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.303+0000 7fddc444b740 1 Processor -- start 2024-01-16T15:02:41.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.303+0000 7fddc444b740 1 -- start start 2024-01-16T15:02:41.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b4ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 0x558ef6b4f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddc444b740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b03dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddc444b740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef67d3f20 con 0x558ef6b0a070 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef6a24e30 con 0x558ef6b04d70 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ef6b08de0 con 0x558ef6b097b0 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddbb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b03dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddbb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b03dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59910/0 (socket says 172.21.15.141:59910) 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.304+0000 7fddbb400640 1 -- 172.21.15.141:0/1554244525 learned_addr learned my addr 172.21.15.141:0/1554244525 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddbaa00640 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b4ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddb3e00640 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 0x558ef6b4f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddbb400640 1 -- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 msgr2=0x558ef6b4f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddbb400640 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 0x558ef6b4f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddbb400640 1 -- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 msgr2=0x558ef6b4ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddbb400640 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b4ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddbb400640 1 -- 172.21.15.141:0/1554244525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ef6a0cb80 con 0x558ef6b0a070 2024-01-16T15:02:41.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddbb400640 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b03dd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x558ef6b051d0 tx=0x7fddb4001fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7e87e770211d20 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddabe00640 1 -- 172.21.15.141:0/1554244525 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb400d6a0 con 0x558ef6b0a070 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddabe00640 1 -- 172.21.15.141:0/1554244525 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb400d800 con 0x558ef6b0a070 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.305+0000 7fddabe00640 1 -- 172.21.15.141:0/1554244525 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb400dbb0 con 0x558ef6b0a070 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 -- 172.21.15.141:0/1554244525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 msgr2=0x558ef6b03dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b03dd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x558ef6b051d0 tx=0x7fddb4001fc0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 -- 172.21.15.141:0/1554244525 shutdown_connections 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b0a070 0x558ef6b03dd0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b097b0 0x558ef6b4f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 --2- 172.21.15.141:0/1554244525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b04d70 0x558ef6b4ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 -- 172.21.15.141:0/1554244525 >> 172.21.15.141:0/1554244525 conn(0x558ef6acbc70 msgr2=0x558ef6b725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 -- 172.21.15.141:0/1554244525 shutdown_connections 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.306+0000 7fddc444b740 1 -- 172.21.15.141:0/1554244525 wait complete. 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.307+0000 7fddc444b740 1 -- start start 2024-01-16T15:02:41.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.307+0000 7fddc444b740 1 Processor -- start 2024-01-16T15:02:41.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddc444b740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6bc2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b30980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddc444b740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 0x558ef6b03dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddc444b740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef6bc2420 con 0x558ef6b04d70 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ef6b31520 con 0x558ef6b097b0 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddc444b740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ef6b46b60 con 0x558ef6b0a070 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddc444b740 10 client.0 _ll_register_callbacks cb 0x7fddac059080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6bc2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddbaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6bc2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59922/0 (socket says 172.21.15.141:59922) 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.308+0000 7fddbaa00640 1 -- 172.21.15.141:0/393854933 learned_addr learned my addr 172.21.15.141:0/393854933 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddb3e00640 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b30980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddbaa00640 1 -- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 msgr2=0x558ef6b03dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddbaa00640 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 0x558ef6b03dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddbaa00640 1 -- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 msgr2=0x558ef6b30980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddbaa00640 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b30980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddbaa00640 1 -- 172.21.15.141:0/393854933 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb4001860 con 0x558ef6b04d70 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddb3e00640 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b30980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddbaa00640 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6bc2840 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fddac060680 tx=0x7fddac08a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4222d458f0f7325 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fdda3e00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fdda3e00640 1 -- 172.21.15.141:0/393854933 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00e040 con 0x558ef6b04d70 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fdda3e00640 1 -- 172.21.15.141:0/393854933 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddac08a0b0 con 0x558ef6b04d70 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddac070180 con 0x558ef6b04d70 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fdda3e00640 1 -- 172.21.15.141:0/393854933 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddac077270 con 0x558ef6b04d70 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddc444b740 10 client.6558 fetch_fsmap learned FSMap version 37 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.309+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x558ef6a32170 con 0x558ef6b04d70 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fdda3e00640 1 -- 172.21.15.141:0/393854933 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac070180 con 0x558ef6b04d70 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fdda3e00640 1 -- 172.21.15.141:0/393854933 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fddac060720 con 0x558ef6b04d70 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fddc444b740 10 client.6558 fetch_fsmap finished waiting for FSMap version 37 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fddc444b740 10 client.6558 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fddc444b740 20 client.6558 populate_metadata read hostname 'smithi141' 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fddc444b740 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b38e30 0x558ef6b04310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fddc444b740 4 client.6558 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 --> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558ef67716c0 con 0x558ef6b38e30 2024-01-16T15:02:41.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.310+0000 7fddb3e00640 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b38e30 0x558ef6b04310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddb3e00640 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b38e30 0x558ef6b04310 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a62021967695d3d server_cookie=7467ef16d565a14c in_seq=0 out_seq=0 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fdda3e00640 10 client.6558 ms_handle_connect on v2:172.21.15.141:6834/1757492142 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fdda3e00640 1 -- 172.21.15.141:0/393854933 <== mds.0 v2:172.21.15.141:6834/1757492142 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x558ef67716c0 con 0x558ef6b38e30 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fdda3e00640 10 client.6558 handle_command_reply: tid=0 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 client.6558 shutdown 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 msgr2=0x558ef6bc2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6bc2840 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fddac060680 tx=0x7fddac08a040 comp rx=0 tx=0).stop 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 shutdown_connections 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:6834/1757492142,v1:172.21.15.141:6835/1757492142] conn(0x558ef6b38e30 0x558ef6b04310 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x558ef6b0a070 0x558ef6b03dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ef6b097b0 0x558ef6b30980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 --2- 172.21.15.141:0/393854933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558ef6b04d70 0x558ef6bc2840 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fdda3e00640 0 client.6558 ms_handle_reset on v2:172.21.15.141:6834/1757492142 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 >> 172.21.15.141:0/393854933 conn(0x558ef6acbc70 msgr2=0x558ef6b4ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 shutdown_connections 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.311+0000 7fddc444b740 1 -- 172.21.15.141:0/393854933 wait complete. 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddc444b740 20 client.6558 trim_cache size 0 max 16384 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 6360, 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 11.602234661000001, 2024-01-16T15:02:41.295 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 37, 2024-01-16T15:02:41.296 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 59, 2024-01-16T15:02:41.296 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 59, 2024-01-16T15:02:41.296 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 17.108611181000001 2024-01-16T15:02:41.296 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdd9406b860 msgr2=0x7fdd9406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdd9406b860 0x7fdd9406dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda4030000 comp rx=0 tx=0).stop 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc13cfa0 msgr2=0x7fddbc0ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc13cfa0 0x7fddbc0ae760 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c430 tx=0x7fddac00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 shutdown_connections 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdd9406b860 0x7fdd9406dd20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fddbc15afb0 0x7fddbc15cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc13fae0 0x7fddbc15c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 --2- 172.21.15.141:0/1584981696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fddbc13cfa0 0x7fddbc0ae760 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 >> 172.21.15.141:0/1584981696 conn(0x7fddbc06a800 msgr2=0x7fddbc153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 shutdown_connections 2024-01-16T15:02:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.312+0000 7fddbbe00640 1 -- 172.21.15.141:0/1584981696 wait complete. 2024-01-16T15:02:41.308 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:02:41.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.534+0000 7fb448200640 1 Processor -- start 2024-01-16T15:02:41.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.534+0000 7fb448200640 1 -- start start 2024-01-16T15:02:41.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.535+0000 7fb448200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb440150760 0x7fb440072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.535+0000 7fb448200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.535+0000 7fb448200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4400b3750 con 0x7fb44015d300 2024-01-16T15:02:41.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.535+0000 7fb448200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44015dc80 con 0x7fb440150760 2024-01-16T15:02:41.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.535+0000 7fb448200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb440071190 con 0x7fb4400b32e0 2024-01-16T15:02:41.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb445800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb440150760 0x7fb440072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb445800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb440150760 0x7fb440072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59924/0 (socket says 172.21.15.141:59924) 2024-01-16T15:02:41.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb445800640 1 -- 172.21.15.141:0/3183189982 learned_addr learned my addr 172.21.15.141:0/3183189982 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb446200640 1 --2- 172.21.15.141:0/3183189982 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb445800640 1 -- 172.21.15.141:0/3183189982 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 msgr2=0x7fb44015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb445800640 1 --2- 172.21.15.141:0/3183189982 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44015d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb445800640 1 -- 172.21.15.141:0/3183189982 >> v1:172.21.15.188:6789/0 conn(0x7fb44015d300 legacy=0x7fb4400b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.536+0000 7fb445800640 1 -- 172.21.15.141:0/3183189982 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb440071470 con 0x7fb440150760 2024-01-16T15:02:41.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.537+0000 7fb445800640 1 --2- 172.21.15.141:0/3183189982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb440150760 0x7fb440072230 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb43000aa10 tx=0x7fb43002cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36965c26b89f53a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb43fe00640 1 -- 172.21.15.141:0/3183189982 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43002db80 con 0x7fb440150760 2024-01-16T15:02:41.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb43fe00640 1 -- 172.21.15.141:0/3183189982 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43002dce0 con 0x7fb440150760 2024-01-16T15:02:41.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb43fe00640 1 -- 172.21.15.141:0/3183189982 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430035660 con 0x7fb440150760 2024-01-16T15:02:41.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb448200640 1 -- 172.21.15.141:0/3183189982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb440150760 msgr2=0x7fb440072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb448200640 1 --2- 172.21.15.141:0/3183189982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb440150760 0x7fb440072230 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb43000aa10 tx=0x7fb43002cdd0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb448200640 1 -- 172.21.15.141:0/3183189982 shutdown_connections 2024-01-16T15:02:41.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb448200640 1 --2- 172.21.15.141:0/3183189982 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb448200640 1 --2- 172.21.15.141:0/3183189982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb440150760 0x7fb440072230 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.538+0000 7fb448200640 1 -- 172.21.15.141:0/3183189982 >> 172.21.15.141:0/3183189982 conn(0x7fb44006a800 msgr2=0x7fb44006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.539+0000 7fb448200640 1 -- 172.21.15.141:0/3183189982 shutdown_connections 2024-01-16T15:02:41.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.539+0000 7fb448200640 1 -- 172.21.15.141:0/3183189982 wait complete. 2024-01-16T15:02:41.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.540+0000 7fb448200640 1 Processor -- start 2024-01-16T15:02:41.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.541+0000 7fb448200640 1 -- start start 2024-01-16T15:02:41.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.541+0000 7fb448200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44014d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.541+0000 7fb448200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb440150760 0x7fb4400cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.541+0000 7fb448200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb44015d300 0x7fb4400ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.541+0000 7fb448200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400ceae0 con 0x7fb44015d300 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.541+0000 7fb448200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440157660 con 0x7fb440150760 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.541+0000 7fb448200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4401583e0 con 0x7fb4400b32e0 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb44015d300 0x7fb4400ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb44015d300 0x7fb4400ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59926/0 (socket says 172.21.15.141:59926) 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 -- 172.21.15.141:0/1434250136 learned_addr learned my addr 172.21.15.141:0/1434250136 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb445800640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44014d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 -- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 msgr2=0x7fb44014d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44014d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 -- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb440150760 msgr2=0x7fb4400cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb440150760 0x7fb4400cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 -- 172.21.15.141:0/1434250136 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43000a670 con 0x7fb44015d300 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.542+0000 7fb446200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb44015d300 0x7fb4400ce3f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb43800cd70 tx=0x7fb43800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.543+0000 7fb43d600640 1 -- 172.21.15.141:0/1434250136 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438003b90 con 0x7fb44015d300 2024-01-16T15:02:41.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.543+0000 7fb43d600640 1 -- 172.21.15.141:0/1434250136 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb438003cf0 con 0x7fb44015d300 2024-01-16T15:02:41.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.543+0000 7fb43d600640 1 -- 172.21.15.141:0/1434250136 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438004240 con 0x7fb44015d300 2024-01-16T15:02:41.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.543+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb440157380 con 0x7fb44015d300 2024-01-16T15:02:41.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.543+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb440158960 con 0x7fb44015d300 2024-01-16T15:02:41.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.544+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb440158b80 con 0x7fb44015d300 2024-01-16T15:02:41.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.545+0000 7fb43d600640 1 -- 172.21.15.141:0/1434250136 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb4380043a0 con 0x7fb44015d300 2024-01-16T15:02:41.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.546+0000 7fb43d600640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb41806b860 0x7fb41806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.546+0000 7fb43d600640 1 -- 172.21.15.141:0/1434250136 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb438012070 con 0x7fb44015d300 2024-01-16T15:02:41.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.546+0000 7fb445800640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb41806b860 0x7fb41806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.548+0000 7fb445800640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb41806b860 0x7fb41806dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb430037000 tx=0x7fb430008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.550+0000 7fb43d600640 1 -- 172.21.15.141:0/1434250136 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb43808e030 con 0x7fb44015d300 2024-01-16T15:02:41.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.776+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb4400675d0 con 0x7fb44015d300 2024-01-16T15:02:41.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.777+0000 7fb43d600640 1 -- 172.21.15.141:0/1434250136 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb4380555e0 con 0x7fb44015d300 2024-01-16T15:02:41.757 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:02:41.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.779+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb41806b860 msgr2=0x7fb41806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.779+0000 7fb448200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb41806b860 0x7fb41806dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb430037000 tx=0x7fb430008040 comp rx=0 tx=0).stop 2024-01-16T15:02:41.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.779+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb44015d300 msgr2=0x7fb4400ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.779+0000 7fb448200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb44015d300 0x7fb4400ce3f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb43800cd70 tx=0x7fb43800a660 comp rx=0 tx=0).stop 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 shutdown_connections 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb41806b860 0x7fb41806dd20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb44015d300 0x7fb4400ce3f0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb440150760 0x7fb4400cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 --2- 172.21.15.141:0/1434250136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4400b32e0 0x7fb44014d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 >> 172.21.15.141:0/1434250136 conn(0x7fb44006a800 msgr2=0x7fb44015c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 shutdown_connections 2024-01-16T15:02:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:41.780+0000 7fb448200640 1 -- 172.21.15.141:0/1434250136 wait complete. 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:02:41.777 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:02:41.778 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:41.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:41.778 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T15:02:41.829 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:02:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:02:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:02:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T15:02:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:41.829 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42302sec preferred_lft 42302sec 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:41.830 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::882:e9ff:fe92:c54c/64 scope link 2024-01-16T15:02:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:41.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:41.832 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:02:41.919 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T15:02:41.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:41.923 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T15:02:41.976 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T15:02:41.976 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:02:41.976 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:02:41.976 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:02:41.976 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:02:41.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:41.977 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T15:02:41.977 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T15:02:41.977 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:02:41.977 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:02:41.977 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:02:41.977 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T15:02:42.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.159 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T15:02:42.209 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:02:42.209 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:02:42.209 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:42.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.210 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T15:02:42.299 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:02:42.300 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:42.300 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:42.352 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:42.353 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:42.353 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:42.403 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:02:42.495 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:42.550 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:02:42.551 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:02:42.552 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:42.552 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:42.552 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:42.552 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:42.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.553 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:02:42.602 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:02:42.603 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:02:42.607 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:02:42.679 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:02:42.721 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:02:42.721 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:42.721 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:42.721 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:42.721 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:42.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.722 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:02:42.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T15:02:42.807+0000 7fdb1603f480 -1 init, newargv = 0x5560fe360cf0 newargc=15 2024-01-16T15:02:42.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[57243]: starting ceph client 2024-01-16T15:02:42.796 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[57243]: starting fuse 2024-01-16T15:02:42.808 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T15:02:42.808 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T15:02:42.808 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:02:42.809 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:02:42.810 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T15:02:42.810 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T15:02:43.002 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.57243.asok 2024-01-16T15:02:43.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:02:43.019 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57243.asok 2024-01-16T15:02:43.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.019 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57243.asok status 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "57243", 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T15:02:43.251 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 6573, 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 6573 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 3949256676 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:43.252 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 3949256676 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.6573 192.168.0.1:0/3949256676", 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/3949256676", 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 37, 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 59, 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T15:02:43.253 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:02:43.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.266 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T15:02:43.267 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:43.318 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T15:02:43.318 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:43.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:43.408 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:02:43.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.638+0000 7f2212600640 1 Processor -- start 2024-01-16T15:02:43.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.639+0000 7f2212600640 1 -- start start 2024-01-16T15:02:43.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.640+0000 7f2212600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f220c143ae0 0x7f220c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.640+0000 7f2212600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c0cf7d0 0x7f220c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.640+0000 7f2212600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f220c15e1e0 0x7f220c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.640+0000 7f2212600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220c0cfc30 con 0x7f220c143ae0 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.640+0000 7f2212600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220c0b1ee0 con 0x7f220c15e1e0 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.640+0000 7f2212600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0b2780 con 0x7f220c0cf7d0 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.640+0000 7f220b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f220c143ae0 0x7f220c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.641+0000 7f220b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f220c143ae0 0x7f220c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58338/0 (socket says 172.21.15.141:58338) 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.641+0000 7f220b400640 1 -- 172.21.15.141:0/1778925303 learned_addr learned my addr 172.21.15.141:0/1778925303 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.641+0000 7f220b400640 1 -- 172.21.15.141:0/1778925303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c0cf7d0 msgr2=0x7f220c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.641+0000 7f220b400640 1 --2- 172.21.15.141:0/1778925303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c0cf7d0 0x7f220c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.641+0000 7f220b400640 1 -- 172.21.15.141:0/1778925303 >> v2:172.21.15.188:3300/0 conn(0x7f220c15e1e0 msgr2=0x7f220c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:43.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.641+0000 7f220b400640 1 --2- 172.21.15.141:0/1778925303 >> v2:172.21.15.188:3300/0 conn(0x7f220c15e1e0 0x7f220c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.641+0000 7f220b400640 1 -- 172.21.15.141:0/1778925303 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c15e990 con 0x7f220c143ae0 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.642+0000 7f220b400640 1 --2- 172.21.15.141:0/1778925303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c143ae0 0x7f220c0cf290 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f220c072160 tx=0x7f21fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bae27b35edc1d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.642+0000 7f220a000640 1 -- 172.21.15.141:0/1778925303 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc031870 con 0x7f220c143ae0 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.642+0000 7f220a000640 1 -- 172.21.15.141:0/1778925303 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc0319d0 con 0x7f220c143ae0 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.642+0000 7f220a000640 1 -- 172.21.15.141:0/1778925303 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc031d60 con 0x7f220c143ae0 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 -- 172.21.15.141:0/1778925303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c143ae0 msgr2=0x7f220c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 --2- 172.21.15.141:0/1778925303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c143ae0 0x7f220c0cf290 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f220c072160 tx=0x7f21fc02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 -- 172.21.15.141:0/1778925303 shutdown_connections 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 --2- 172.21.15.141:0/1778925303 >> v2:172.21.15.188:3300/0 conn(0x7f220c15e1e0 0x7f220c15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 --2- 172.21.15.141:0/1778925303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c0cf7d0 0x7f220c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 --2- 172.21.15.141:0/1778925303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c143ae0 0x7f220c0cf290 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 -- 172.21.15.141:0/1778925303 >> 172.21.15.141:0/1778925303 conn(0x7f220c06a800 msgr2=0x7f220c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:43.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 -- 172.21.15.141:0/1778925303 shutdown_connections 2024-01-16T15:02:43.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.643+0000 7f2212600640 1 -- 172.21.15.141:0/1778925303 wait complete. 2024-01-16T15:02:43.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.645+0000 7f2212600640 1 Processor -- start 2024-01-16T15:02:43.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.645+0000 7f2212600640 1 -- start start 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f2212600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c0cf7d0 0x7f220c165320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f2212600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c143ae0 0x7f220c1414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f2212600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f220c15e1e0 0x7f220c163430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f2212600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c166610 con 0x7f220c0cf7d0 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f2212600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c141a20 con 0x7f220c15e1e0 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f2212600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c165a00 con 0x7f220c143ae0 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f220b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c0cf7d0 0x7f220c165320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f220b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c0cf7d0 0x7f220c165320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58342/0 (socket says 172.21.15.141:58342) 2024-01-16T15:02:43.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.646+0000 7f220b400640 1 -- 172.21.15.141:0/97308171 learned_addr learned my addr 172.21.15.141:0/97308171 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220b400640 1 -- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c143ae0 msgr2=0x7f220c1414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220be00640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f220c15e1e0 0x7f220c163430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220aa00640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c143ae0 0x7f220c1414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220b400640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c143ae0 0x7f220c1414e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220b400640 1 -- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f220c15e1e0 msgr2=0x7f220c163430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220b400640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f220c15e1e0 0x7f220c163430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220b400640 1 -- 172.21.15.141:0/97308171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0094d0 con 0x7f220c0cf7d0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f220b400640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c0cf7d0 0x7f220c165320 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0097c0 tx=0x7f21fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f2203400640 1 -- 172.21.15.141:0/97308171 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc03ad30 con 0x7f220c0cf7d0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f2203400640 1 -- 172.21.15.141:0/97308171 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc03ae90 con 0x7f220c0cf7d0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c165ce0 con 0x7f220c0cf7d0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.647+0000 7f2203400640 1 -- 172.21.15.141:0/97308171 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc044610 con 0x7f220c0cf7d0 2024-01-16T15:02:43.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.648+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c1662a0 con 0x7f220c0cf7d0 2024-01-16T15:02:43.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.649+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c0675b0 con 0x7f220c0cf7d0 2024-01-16T15:02:43.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.650+0000 7f2203400640 1 -- 172.21.15.141:0/97308171 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f21fc04d450 con 0x7f220c0cf7d0 2024-01-16T15:02:43.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.651+0000 7f2203400640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f21e406b860 0x7f21e406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.651+0000 7f2203400640 1 -- 172.21.15.141:0/97308171 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21fc02d040 con 0x7f220c0cf7d0 2024-01-16T15:02:43.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.652+0000 7f220aa00640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f21e406b860 0x7f21e406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.653+0000 7f220aa00640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f21e406b860 0x7f21e406dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f21f400a770 tx=0x7f21f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:43.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.656+0000 7f2203400640 1 -- 172.21.15.141:0/97308171 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21fc0772f0 con 0x7f220c0cf7d0 2024-01-16T15:02:43.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.883+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f220c06cfa0 con 0x7f220c0cf7d0 2024-01-16T15:02:43.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.884+0000 7f2203400640 1 -- 172.21.15.141:0/97308171 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) v1 ==== 53+0+73 (secure 0 0 0) 0x7f21fc07a910 con 0x7f220c0cf7d0 2024-01-16T15:02:43.864 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f21e406b860 msgr2=0x7f21e406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f21e406b860 0x7f21e406dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f21f400a770 tx=0x7f21f4008040 comp rx=0 tx=0).stop 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c0cf7d0 msgr2=0x7f220c165320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c0cf7d0 0x7f220c165320 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0097c0 tx=0x7f21fc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 shutdown_connections 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f21e406b860 0x7f21e406dd20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f220c15e1e0 0x7f220c163430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f220c143ae0 0x7f220c1414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 --2- 172.21.15.141:0/97308171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f220c0cf7d0 0x7f220c165320 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 >> 172.21.15.141:0/97308171 conn(0x7f220c06a800 msgr2=0x7f220c15b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 shutdown_connections 2024-01-16T15:02:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:43.887+0000 7f2212600640 1 -- 172.21.15.141:0/97308171 wait complete. 2024-01-16T15:02:43.884 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:02:43.884 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:02:43.884 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T15:02:43.885 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:02:43.885 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:43.885 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:02:43.885 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:02:43.885 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:02:43.885 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:43.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.885 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T15:02:43.942 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:02:43.942 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:02:43.942 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:02:43.942 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42301sec preferred_lft 42301sec 2024-01-16T15:02:43.943 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::a40a:23ff:fed8:ea96/64 scope link 2024-01-16T15:02:43.944 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:43.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.945 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:02:44.032 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T15:02:44.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.037 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T15:02:44.089 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T15:02:44.090 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:02:44.090 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:02:44.090 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:02:44.090 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:02:44.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.090 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T15:02:44.090 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T15:02:44.090 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:02:44.090 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:02:44.090 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:02:44.090 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T15:02:44.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.271 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T15:02:44.319 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:02:44.319 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:02:44.320 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:44.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.320 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T15:02:44.408 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:02:44.408 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:44.408 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:44.460 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:44.460 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:44.460 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:44.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:02:44.606 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:44.660 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:02:44.661 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:44.663 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:44.663 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:44.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.663 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:02:44.709 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:02:44.709 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:02:44.755 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:02:44.834 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:44.875 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:02:44.875 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:02:44.875 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:02:44.875 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:02:44.875 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:44.876 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:44.877 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:44.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.878 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:02:44.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T15:02:44.962+0000 7fe78fc36480 -1 init, newargv = 0x55998fc02a40 newargc=15 2024-01-16T15:02:44.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[44195]: starting ceph client 2024-01-16T15:02:44.952 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[44195]: starting fuse 2024-01-16T15:02:44.962 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T15:02:44.963 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T15:02:44.964 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T15:02:44.965 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T15:02:44.965 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:02:44.965 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:02:44.965 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T15:02:44.965 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T15:02:45.159 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.44195.asok 2024-01-16T15:02:45.163 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T15:02:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:02:45.176 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44195.asok 2024-01-16T15:02:45.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:45.176 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44195.asok status 2024-01-16T15:02:45.410 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T15:02:45.410 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T15:02:45.410 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:02:45.410 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "44195", 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 5401, 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 5401 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:02:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 51112688 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 51112688 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.5401 192.168.0.1:0/51112688", 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/51112688", 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 37, 2024-01-16T15:02:45.412 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 59, 2024-01-16T15:02:45.413 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:02:45.413 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T15:02:45.413 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T15:02:45.413 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T15:02:45.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:45.425 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T15:02:45.425 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:45.475 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T15:02:45.475 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:45.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:45.476 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:45.558 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:02:45.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.805+0000 7f79e9e00640 1 Processor -- start 2024-01-16T15:02:45.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.806+0000 7f79e9e00640 1 -- start start 2024-01-16T15:02:45.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.807+0000 7f79e9e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f79e4143ae0 0x7f79e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:45.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f79e415e3b0 0x7f79e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:45.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e9e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e40cfc30 con 0x7f79e40cf7d0 2024-01-16T15:02:45.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e9e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e40b1ee0 con 0x7f79e415e3b0 2024-01-16T15:02:45.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e9e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e40b2780 con 0x7f79e4143ae0 2024-01-16T15:02:45.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79dbe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f79e40cf7d0 0x7f79e415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39756/0 (socket says 172.21.15.141:39756) 2024-01-16T15:02:45.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79dbe00640 1 -- 172.21.15.141:0/472560211 learned_addr learned my addr 172.21.15.141:0/472560211 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:45.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e3400640 1 --2- 172.21.15.141:0/472560211 >> v2:172.21.15.141:3300/0 conn(0x7f79e415e3b0 0x7f79e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:45.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e2a00640 1 --2- 172.21.15.141:0/472560211 >> v2:172.21.15.188:3300/0 conn(0x7f79e4143ae0 0x7f79e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:45.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e2000640 1 -- 172.21.15.141:0/472560211 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512396218 0 0) 0x7f79e40cfc30 con 0x7f79e40cf7d0 2024-01-16T15:02:45.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e2000640 1 -- 172.21.15.141:0/472560211 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c4003520 con 0x7f79e40cf7d0 2024-01-16T15:02:45.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e3400640 1 -- 172.21.15.141:0/472560211 >> v2:172.21.15.188:3300/0 conn(0x7f79e4143ae0 msgr2=0x7f79e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:45.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.808+0000 7f79e3400640 1 --2- 172.21.15.141:0/472560211 >> v2:172.21.15.188:3300/0 conn(0x7f79e4143ae0 0x7f79e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.809+0000 7f79e3400640 1 -- 172.21.15.141:0/472560211 >> v1:172.21.15.141:6789/0 conn(0x7f79e40cf7d0 legacy=0x7f79e415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:45.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.809+0000 7f79e3400640 1 -- 172.21.15.141:0/472560211 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e415e990 con 0x7f79e415e3b0 2024-01-16T15:02:45.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.810+0000 7f79e3400640 1 --2- 172.21.15.141:0/472560211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e415e3b0 0x7f79e40702c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f79e40cfea0 tx=0x7f79dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5ceb8ed569a56d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:45.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.810+0000 7f79e2000640 1 -- 172.21.15.141:0/472560211 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc02cc80 con 0x7f79e415e3b0 2024-01-16T15:02:45.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.810+0000 7f79e2000640 1 -- 172.21.15.141:0/472560211 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc02cde0 con 0x7f79e415e3b0 2024-01-16T15:02:45.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.810+0000 7f79e2000640 1 -- 172.21.15.141:0/472560211 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc034660 con 0x7f79e415e3b0 2024-01-16T15:02:45.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.810+0000 7f79e9e00640 1 -- 172.21.15.141:0/472560211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e415e3b0 msgr2=0x7f79e40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:45.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.810+0000 7f79e9e00640 1 --2- 172.21.15.141:0/472560211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e415e3b0 0x7f79e40702c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f79e40cfea0 tx=0x7f79dc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.811+0000 7f79e9e00640 1 -- 172.21.15.141:0/472560211 shutdown_connections 2024-01-16T15:02:45.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.811+0000 7f79e9e00640 1 --2- 172.21.15.141:0/472560211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e415e3b0 0x7f79e40702c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.811+0000 7f79e9e00640 1 --2- 172.21.15.141:0/472560211 >> v2:172.21.15.188:3300/0 conn(0x7f79e4143ae0 0x7f79e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.811+0000 7f79e9e00640 1 -- 172.21.15.141:0/472560211 >> 172.21.15.141:0/472560211 conn(0x7f79e406a800 msgr2=0x7f79e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:45.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.811+0000 7f79e9e00640 1 -- 172.21.15.141:0/472560211 shutdown_connections 2024-01-16T15:02:45.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.811+0000 7f79e9e00640 1 -- 172.21.15.141:0/472560211 wait complete. 2024-01-16T15:02:45.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.812+0000 7f79e9e00640 1 Processor -- start 2024-01-16T15:02:45.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.813+0000 7f79e9e00640 1 -- start start 2024-01-16T15:02:45.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.813+0000 7f79e9e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e40cf7d0 0x7f79e40ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:45.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.813+0000 7f79e9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e4143ae0 0x7f79e40ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:45.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79e9e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79e415e3b0 0x7f79e40adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:45.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79e9e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40ad340 con 0x7f79e4143ae0 2024-01-16T15:02:45.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79e9e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40cd870 con 0x7f79e40cf7d0 2024-01-16T15:02:45.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79e9e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40cdb50 con 0x7f79e415e3b0 2024-01-16T15:02:45.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e4143ae0 0x7f79e40ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:45.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e4143ae0 0x7f79e40ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58360/0 (socket says 172.21.15.141:58360) 2024-01-16T15:02:45.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 -- 172.21.15.141:0/3722883157 learned_addr learned my addr 172.21.15.141:0/3722883157 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 -- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79e415e3b0 msgr2=0x7f79e40adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79e415e3b0 0x7f79e40adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 -- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e40cf7d0 msgr2=0x7f79e40ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79e2a00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e40cf7d0 0x7f79e40ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e40cf7d0 0x7f79e40ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.814+0000 7f79dbe00640 1 -- 172.21.15.141:0/3722883157 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc009a70 con 0x7f79e4143ae0 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.815+0000 7f79dbe00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e4143ae0 0x7f79e40ad5b0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00c500 tx=0x7f79cc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:45.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.815+0000 7f79db400640 1 -- 172.21.15.141:0/3722883157 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc005ce0 con 0x7f79e4143ae0 2024-01-16T15:02:45.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.815+0000 7f79db400640 1 -- 172.21.15.141:0/3722883157 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc005e40 con 0x7f79e4143ae0 2024-01-16T15:02:45.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.815+0000 7f79db400640 1 -- 172.21.15.141:0/3722883157 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc007330 con 0x7f79e4143ae0 2024-01-16T15:02:45.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.815+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e40cde30 con 0x7f79e4143ae0 2024-01-16T15:02:45.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.815+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e40ce3f0 con 0x7f79e4143ae0 2024-01-16T15:02:45.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.816+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e40ce5e0 con 0x7f79e4143ae0 2024-01-16T15:02:45.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.817+0000 7f79db400640 1 -- 172.21.15.141:0/3722883157 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f79cc007550 con 0x7f79e4143ae0 2024-01-16T15:02:45.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.818+0000 7f79db400640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79bc06b860 0x7f79bc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:45.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.818+0000 7f79db400640 1 -- 172.21.15.141:0/3722883157 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79cc089c90 con 0x7f79e4143ae0 2024-01-16T15:02:45.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.819+0000 7f79e2a00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79bc06b860 0x7f79bc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:45.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.820+0000 7f79e2a00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79bc06b860 0x7f79bc06dd20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79d4006fd0 tx=0x7f79d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:45.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:45.822+0000 7f79db400640 1 -- 172.21.15.141:0/3722883157 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79cc0543a0 con 0x7f79e4143ae0 2024-01-16T15:02:46.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.052+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f79e40675d0 con 0x7f79e4143ae0 2024-01-16T15:02:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.543+0000 7f79db400640 1 -- 172.21.15.141:0/3722883157 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 136+0+0 (secure 0 0 0) 0x7f79cc0579c0 con 0x7f79e4143ae0 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79bc06b860 msgr2=0x7f79bc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79bc06b860 0x7f79bc06dd20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79d4006fd0 tx=0x7f79d4008040 comp rx=0 tx=0).stop 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e4143ae0 msgr2=0x7f79e40ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e4143ae0 0x7f79e40ad5b0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00c500 tx=0x7f79cc00c9c0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 shutdown_connections 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79bc06b860 0x7f79bc06dd20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79e415e3b0 0x7f79e40adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79e4143ae0 0x7f79e40ad5b0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 --2- 172.21.15.141:0/3722883157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e40cf7d0 0x7f79e40ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.546+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 >> 172.21.15.141:0/3722883157 conn(0x7f79e406a800 msgr2=0x7f79e40b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.547+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 shutdown_connections 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.547+0000 7f79e9e00640 1 -- 172.21.15.141:0/3722883157 wait complete. 2024-01-16T15:02:46.549 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:02:46.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.785+0000 7f9dc6200640 1 Processor -- start 2024-01-16T15:02:46.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.785+0000 7f9dc6200640 1 -- start start 2024-01-16T15:02:46.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.787+0000 7f9dc6200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9dc013e360 0x7f9dc00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.787+0000 7f9dc6200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc015c5f0 con 0x7f9dc015f9c0 2024-01-16T15:02:46.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.787+0000 7f9dc6200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc006c470 con 0x7f9dc013df80 2024-01-16T15:02:46.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.787+0000 7f9dc6200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc006c750 con 0x7f9dc013e360 2024-01-16T15:02:46.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.787+0000 7f9dbfe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9dc015f9c0 0x7f9dc00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39770/0 (socket says 172.21.15.141:39770) 2024-01-16T15:02:46.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.787+0000 7f9db6800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9dc013e360 0x7f9dc00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.787+0000 7f9dbfe00640 1 -- 172.21.15.141:0/453558657 learned_addr learned my addr 172.21.15.141:0/453558657 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:46.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.788+0000 7f9db6800640 1 -- 172.21.15.141:0/453558657 >> v1:172.21.15.188:6789/0 conn(0x7f9dc013df80 legacy=0x7f9dc015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:46.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.788+0000 7f9db6800640 1 -- 172.21.15.141:0/453558657 >> v1:172.21.15.141:6789/0 conn(0x7f9dc015f9c0 legacy=0x7f9dc00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.788+0000 7f9db6800640 1 -- 172.21.15.141:0/453558657 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc006ca30 con 0x7f9dc013e360 2024-01-16T15:02:46.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.788+0000 7f9dbea00640 1 -- 172.21.15.141:0/453558657 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337116268 0 0) 0x7f9dc015c5f0 con 0x7f9dc015f9c0 2024-01-16T15:02:46.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.789+0000 7f9db6800640 1 --2- 172.21.15.141:0/453558657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013e360 0x7f9dc00cf290 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9da80097a0 tx=0x7f9da802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7243b98baf87c246 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.789+0000 7f9dbea00640 1 -- 172.21.15.141:0/453558657 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8030610 con 0x7f9dc013e360 2024-01-16T15:02:46.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.789+0000 7f9dbea00640 1 -- 172.21.15.141:0/453558657 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da8030770 con 0x7f9dc013e360 2024-01-16T15:02:46.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.789+0000 7f9dbea00640 1 -- 172.21.15.141:0/453558657 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8030b00 con 0x7f9dc013e360 2024-01-16T15:02:46.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.790+0000 7f9dc6200640 1 -- 172.21.15.141:0/453558657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013e360 msgr2=0x7f9dc00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.790+0000 7f9dc6200640 1 --2- 172.21.15.141:0/453558657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013e360 0x7f9dc00cf290 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9da80097a0 tx=0x7f9da802b960 comp rx=0 tx=0).stop 2024-01-16T15:02:46.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.790+0000 7f9dc6200640 1 -- 172.21.15.141:0/453558657 shutdown_connections 2024-01-16T15:02:46.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.790+0000 7f9dc6200640 1 --2- 172.21.15.141:0/453558657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013e360 0x7f9dc00cf290 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.790+0000 7f9dc6200640 1 -- 172.21.15.141:0/453558657 >> 172.21.15.141:0/453558657 conn(0x7f9dc006a800 msgr2=0x7f9dc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.790+0000 7f9dc6200640 1 -- 172.21.15.141:0/453558657 shutdown_connections 2024-01-16T15:02:46.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.790+0000 7f9dc6200640 1 -- 172.21.15.141:0/453558657 wait complete. 2024-01-16T15:02:46.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.791+0000 7f9dc6200640 1 Processor -- start 2024-01-16T15:02:46.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.792+0000 7f9dc6200640 1 -- start start 2024-01-16T15:02:46.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.792+0000 7f9dc6200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013df80 0x7f9dc01e5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.792+0000 7f9dc6200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dc013e360 0x7f9dc01e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dc6200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9dc015f9c0 0x7f9dc013d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dc6200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc01e7ff0 con 0x7f9dc013df80 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dc6200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc01e92c0 con 0x7f9dc013e360 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dc6200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc01e8220 con 0x7f9dc015f9c0 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013df80 0x7f9dc01e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013df80 0x7f9dc01e5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58378/0 (socket says 172.21.15.141:58378) 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 -- 172.21.15.141:0/1863144590 learned_addr learned my addr 172.21.15.141:0/1863144590 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 -- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9dc015f9c0 msgr2=0x7f9dc013d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9dc015f9c0 0x7f9dc013d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 -- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dc013e360 msgr2=0x7f9dc01e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dc013e360 0x7f9dc01e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 -- 172.21.15.141:0/1863144590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8009400 con 0x7f9dc013df80 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.793+0000 7f9dbf400640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013df80 0x7f9dc01e5680 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9db000c400 tx=0x7f9db000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.794+0000 7f9db7e00640 1 -- 172.21.15.141:0/1863144590 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db00052d0 con 0x7f9dc013df80 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.794+0000 7f9db7e00640 1 -- 172.21.15.141:0/1863144590 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db0005430 con 0x7f9dc013df80 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.794+0000 7f9db7e00640 1 -- 172.21.15.141:0/1863144590 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0003830 con 0x7f9dc013df80 2024-01-16T15:02:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.794+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc01e8500 con 0x7f9dc013df80 2024-01-16T15:02:46.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.794+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc01e8ac0 con 0x7f9dc013df80 2024-01-16T15:02:46.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.795+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc00cd420 con 0x7f9dc013df80 2024-01-16T15:02:46.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.796+0000 7f9db7e00640 1 -- 172.21.15.141:0/1863144590 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f9db0003990 con 0x7f9dc013df80 2024-01-16T15:02:46.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.797+0000 7f9db7e00640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d9806b860 0x7f9d9806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.797+0000 7f9db7e00640 1 -- 172.21.15.141:0/1863144590 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9db0089b50 con 0x7f9dc013df80 2024-01-16T15:02:46.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.797+0000 7f9db6800640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d9806b860 0x7f9d9806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.798+0000 7f9db6800640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d9806b860 0x7f9d9806dd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9da8036000 tx=0x7f9da803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:46.803+0000 7f9db7e00640 1 -- 172.21.15.141:0/1863144590 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9db00541a0 con 0x7f9dc013df80 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.013+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dc006cff0 con 0x7f9d9806b860 2024-01-16T15:02:46.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.016+0000 7f9db7e00640 1 -- 172.21.15.141:0/1863144590 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9da8030960 con 0x7f9d9806b860 2024-01-16T15:02:46.996 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:02:46.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:02:46.996 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:02:46.996 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:02:46.996 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d9806b860 msgr2=0x7f9d9806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d9806b860 0x7f9d9806dd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9da8036000 tx=0x7f9da803b040 comp rx=0 tx=0).stop 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013df80 msgr2=0x7f9dc01e5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013df80 0x7f9dc01e5680 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9db000c400 tx=0x7f9db000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 shutdown_connections 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9d9806b860 0x7f9d9806dd20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9dc015f9c0 0x7f9dc013d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dc013e360 0x7f9dc01e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 --2- 172.21.15.141:0/1863144590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9dc013df80 0x7f9dc01e5680 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 >> 172.21.15.141:0/1863144590 conn(0x7f9dc006a800 msgr2=0x7f9dc015a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 shutdown_connections 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.019+0000 7f9dc6200640 1 -- 172.21.15.141:0/1863144590 wait complete. 2024-01-16T15:02:47.017 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:02:47.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.239+0000 7f8a3ea00640 1 Processor -- start 2024-01-16T15:02:47.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.239+0000 7f8a3ea00640 1 -- start start 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.240+0000 7f8a3ea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a3815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a3ea00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8a3815e1e0 0x7f8a380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a3ea00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a380b1ee0 con 0x7f8a38143ae0 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a3ea00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a380b2780 con 0x7f8a3815e1e0 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a3ea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3815e990 con 0x7f8a380cf9a0 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a2f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a3815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a37e00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f8a38143ae0 0x7f8a380cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:56776/0 (socket says 172.21.15.141:56776) 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a37e00640 1 -- 172.21.15.141:0/133580187 learned_addr learned my addr 172.21.15.141:0/133580187 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.241+0000 7f8a3ca00640 1 --2- 172.21.15.141:0/133580187 >> v2:172.21.15.188:3300/0 conn(0x7f8a3815e1e0 0x7f8a380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.242+0000 7f8a37400640 1 -- 172.21.15.141:0/133580187 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154799687 0 0) 0x7f8a380b1ee0 con 0x7f8a38143ae0 2024-01-16T15:02:47.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.242+0000 7f8a37400640 1 -- 172.21.15.141:0/133580187 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a18003520 con 0x7f8a38143ae0 2024-01-16T15:02:47.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.242+0000 7f8a2f200640 1 -- 172.21.15.141:0/133580187 >> v2:172.21.15.188:3300/0 conn(0x7f8a3815e1e0 msgr2=0x7f8a380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:47.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.242+0000 7f8a2f200640 1 --2- 172.21.15.141:0/133580187 >> v2:172.21.15.188:3300/0 conn(0x7f8a3815e1e0 0x7f8a380702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.242+0000 7f8a2f200640 1 -- 172.21.15.141:0/133580187 >> v1:172.21.15.188:6789/0 conn(0x7f8a38143ae0 legacy=0x7f8a380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.242+0000 7f8a2f200640 1 -- 172.21.15.141:0/133580187 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a380b2130 con 0x7f8a380cf9a0 2024-01-16T15:02:47.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.242+0000 7f8a3ca00640 1 --2- 172.21.15.141:0/133580187 >> v2:172.21.15.188:3300/0 conn(0x7f8a3815e1e0 0x7f8a380702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:02:47.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.244+0000 7f8a2f200640 1 --2- 172.21.15.141:0/133580187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a3815dca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8a200097a0 tx=0x7f8a2002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9892c01ab2bbc0da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.244+0000 7f8a37400640 1 -- 172.21.15.141:0/133580187 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2002d040 con 0x7f8a380cf9a0 2024-01-16T15:02:47.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a37400640 1 -- 172.21.15.141:0/133580187 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a20031810 con 0x7f8a380cf9a0 2024-01-16T15:02:47.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a37400640 1 -- 172.21.15.141:0/133580187 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20031b80 con 0x7f8a380cf9a0 2024-01-16T15:02:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a3ea00640 1 -- 172.21.15.141:0/133580187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 msgr2=0x7f8a3815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/133580187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a3815dca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8a200097a0 tx=0x7f8a2002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a3ea00640 1 -- 172.21.15.141:0/133580187 shutdown_connections 2024-01-16T15:02:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/133580187 >> v2:172.21.15.188:3300/0 conn(0x7f8a3815e1e0 0x7f8a380702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/133580187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a3815dca0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a3ea00640 1 -- 172.21.15.141:0/133580187 >> 172.21.15.141:0/133580187 conn(0x7f8a3806a800 msgr2=0x7f8a3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.245+0000 7f8a3ea00640 1 -- 172.21.15.141:0/133580187 shutdown_connections 2024-01-16T15:02:47.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.246+0000 7f8a3ea00640 1 -- 172.21.15.141:0/133580187 wait complete. 2024-01-16T15:02:47.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.247+0000 7f8a3ea00640 1 Processor -- start 2024-01-16T15:02:47.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.247+0000 7f8a3ea00640 1 -- start start 2024-01-16T15:02:47.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.248+0000 7f8a3ea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a380cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.248+0000 7f8a3ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a38143ae0 0x7f8a380adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.248+0000 7f8a3ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a3815e1e0 0x7f8a38154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.248+0000 7f8a3ea00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3813f5e0 con 0x7f8a38143ae0 2024-01-16T15:02:47.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.248+0000 7f8a3ea00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3813ce90 con 0x7f8a3815e1e0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.248+0000 7f8a3ea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a380cdd70 con 0x7f8a380cf9a0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a38143ae0 0x7f8a380adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a38143ae0 0x7f8a380adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58388/0 (socket says 172.21.15.141:58388) 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 -- 172.21.15.141:0/3706121156 learned_addr learned my addr 172.21.15.141:0/3706121156 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a3ca00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a3815e1e0 0x7f8a38154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 -- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 msgr2=0x7f8a380cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a380cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 -- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a3815e1e0 msgr2=0x7f8a38154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a3815e1e0 0x7f8a38154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.249+0000 7f8a2f200640 1 -- 172.21.15.141:0/3706121156 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a30009a70 con 0x7f8a38143ae0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.250+0000 7f8a2f200640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a38143ae0 0x7f8a380adb90 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8a200098c0 tx=0x7f8a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.250+0000 7f8a34c00640 1 -- 172.21.15.141:0/3706121156 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20003820 con 0x7f8a38143ae0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.250+0000 7f8a34c00640 1 -- 172.21.15.141:0/3706121156 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2003f070 con 0x7f8a38143ae0 2024-01-16T15:02:47.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.250+0000 7f8a34c00640 1 -- 172.21.15.141:0/3706121156 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20003bc0 con 0x7f8a38143ae0 2024-01-16T15:02:47.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.250+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a20009400 con 0x7f8a38143ae0 2024-01-16T15:02:47.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.250+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a380ce620 con 0x7f8a38143ae0 2024-01-16T15:02:47.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.251+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a380ce0c0 con 0x7f8a38143ae0 2024-01-16T15:02:47.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.252+0000 7f8a34c00640 1 -- 172.21.15.141:0/3706121156 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8a20003d20 con 0x7f8a38143ae0 2024-01-16T15:02:47.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.253+0000 7f8a34c00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a1006b860 0x7f8a1006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.253+0000 7f8a34c00640 1 -- 172.21.15.141:0/3706121156 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a20041070 con 0x7f8a38143ae0 2024-01-16T15:02:47.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.253+0000 7f8a37e00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a1006b860 0x7f8a1006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.254+0000 7f8a37e00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a1006b860 0x7f8a1006dd20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8a28000f60 tx=0x7f8a280078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.257+0000 7f8a34c00640 1 -- 172.21.15.141:0/3706121156 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a20076de0 con 0x7f8a38143ae0 2024-01-16T15:02:47.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.443+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8a380675d0 con 0x7f8a38143ae0 2024-01-16T15:02:47.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.443+0000 7f8a34c00640 1 -- 172.21.15.141:0/3706121156 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8a2007a400 con 0x7f8a38143ae0 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a1006b860 msgr2=0x7f8a1006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a1006b860 0x7f8a1006dd20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8a28000f60 tx=0x7f8a280078f0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a38143ae0 msgr2=0x7f8a380adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a38143ae0 0x7f8a380adb90 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8a200098c0 tx=0x7f8a200046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 shutdown_connections 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a1006b860 0x7f8a1006dd20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a3815e1e0 0x7f8a38154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a38143ae0 0x7f8a380adb90 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 --2- 172.21.15.141:0/3706121156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8a380cf9a0 0x7f8a380cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 >> 172.21.15.141:0/3706121156 conn(0x7f8a3806a800 msgr2=0x7f8a38070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 shutdown_connections 2024-01-16T15:02:47.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.446+0000 7f8a3ea00640 1 -- 172.21.15.141:0/3706121156 wait complete. 2024-01-16T15:02:47.444 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:02:47.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.686+0000 7f9032a00640 1 Processor -- start 2024-01-16T15:02:47.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.686+0000 7f9032a00640 1 -- start start 2024-01-16T15:02:47.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f9032a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f902c0d0b60 0x7f902c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f9032a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f902c0aede0 0x7f902c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f9032a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c071de0 con 0x7f902c06a3e0 2024-01-16T15:02:47.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f9032a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c0af270 con 0x7f902c0d0b60 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f9032a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c072030 con 0x7f902c0aede0 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f902be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f902c06a3e0 0x7f902c06a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39780/0 (socket says 172.21.15.141:39780) 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f902be00640 1 -- 172.21.15.141:0/4162795959 learned_addr learned my addr 172.21.15.141:0/4162795959 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.688+0000 7f902b400640 1 --2- 172.21.15.141:0/4162795959 >> v2:172.21.15.141:3300/0 conn(0x7f902c0d0b60 0x7f902c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.689+0000 7f9030a00640 1 --2- 172.21.15.141:0/4162795959 >> v2:172.21.15.188:3300/0 conn(0x7f902c0aede0 0x7f902c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.689+0000 7f902aa00640 1 -- 172.21.15.141:0/4162795959 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168202475 0 0) 0x7f902c071de0 con 0x7f902c06a3e0 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.689+0000 7f902aa00640 1 -- 172.21.15.141:0/4162795959 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c003520 con 0x7f902c06a3e0 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.689+0000 7f902b400640 1 -- 172.21.15.141:0/4162795959 >> v2:172.21.15.188:3300/0 conn(0x7f902c0aede0 msgr2=0x7f902c06f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.689+0000 7f902b400640 1 --2- 172.21.15.141:0/4162795959 >> v2:172.21.15.188:3300/0 conn(0x7f902c0aede0 0x7f902c06f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.689+0000 7f902b400640 1 -- 172.21.15.141:0/4162795959 >> v1:172.21.15.141:6789/0 conn(0x7f902c06a3e0 legacy=0x7f902c06a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.689+0000 7f902b400640 1 -- 172.21.15.141:0/4162795959 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c072310 con 0x7f902c0d0b60 2024-01-16T15:02:47.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.690+0000 7f902b400640 1 --2- 172.21.15.141:0/4162795959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c0d0b60 0x7f902c0ac8d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f902c071220 tx=0x7f901402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1120d658c131297 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.690+0000 7f902aa00640 1 -- 172.21.15.141:0/4162795959 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014031830 con 0x7f902c0d0b60 2024-01-16T15:02:47.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.690+0000 7f902aa00640 1 -- 172.21.15.141:0/4162795959 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014031990 con 0x7f902c0d0b60 2024-01-16T15:02:47.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.690+0000 7f902aa00640 1 -- 172.21.15.141:0/4162795959 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014031d20 con 0x7f902c0d0b60 2024-01-16T15:02:47.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 -- 172.21.15.141:0/4162795959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c0d0b60 msgr2=0x7f902c0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 --2- 172.21.15.141:0/4162795959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c0d0b60 0x7f902c0ac8d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f902c071220 tx=0x7f901402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 -- 172.21.15.141:0/4162795959 shutdown_connections 2024-01-16T15:02:47.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 --2- 172.21.15.141:0/4162795959 >> v2:172.21.15.188:3300/0 conn(0x7f902c0aede0 0x7f902c06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 --2- 172.21.15.141:0/4162795959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c0d0b60 0x7f902c0ac8d0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 -- 172.21.15.141:0/4162795959 >> 172.21.15.141:0/4162795959 conn(0x7f902c0cfef0 msgr2=0x7f902c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 -- 172.21.15.141:0/4162795959 shutdown_connections 2024-01-16T15:02:47.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.691+0000 7f9032a00640 1 -- 172.21.15.141:0/4162795959 wait complete. 2024-01-16T15:02:47.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.693+0000 7f9032a00640 1 Processor -- start 2024-01-16T15:02:47.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.693+0000 7f9032a00640 1 -- start start 2024-01-16T15:02:47.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f9032a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c06a3e0 0x7f902c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f9032a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f902c0aede0 0x7f902c1e5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f9032a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f902c0d0b60 0x7f902c1e6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f9032a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c1e74b0 con 0x7f902c06a3e0 2024-01-16T15:02:47.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f9032a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c1e6c10 con 0x7f902c0d0b60 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f9032a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c1e6ef0 con 0x7f902c0aede0 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f902be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c06a3e0 0x7f902c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f902be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c06a3e0 0x7f902c0cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58406/0 (socket says 172.21.15.141:58406) 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.694+0000 7f902be00640 1 -- 172.21.15.141:0/1691217999 learned_addr learned my addr 172.21.15.141:0/1691217999 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f902be00640 1 -- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f902c0aede0 msgr2=0x7f902c1e5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f902be00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f902c0aede0 0x7f902c1e5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f902be00640 1 -- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f902c0d0b60 msgr2=0x7f902c1e6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f902be00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f902c0d0b60 0x7f902c1e6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f902be00640 1 -- 172.21.15.141:0/1691217999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9014009400 con 0x7f902c06a3e0 2024-01-16T15:02:47.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f902be00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c06a3e0 0x7f902c0cf130 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f901c00c410 tx=0x7f901c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f9023e00640 1 -- 172.21.15.141:0/1691217999 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c005ce0 con 0x7f902c06a3e0 2024-01-16T15:02:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f9023e00640 1 -- 172.21.15.141:0/1691217999 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c005e40 con 0x7f902c06a3e0 2024-01-16T15:02:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f9023e00640 1 -- 172.21.15.141:0/1691217999 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c0072f0 con 0x7f902c06a3e0 2024-01-16T15:02:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c1e71d0 con 0x7f902c06a3e0 2024-01-16T15:02:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.695+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c15a500 con 0x7f902c06a3e0 2024-01-16T15:02:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.696+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c15af40 con 0x7f902c06a3e0 2024-01-16T15:02:47.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.697+0000 7f9023e00640 1 -- 172.21.15.141:0/1691217999 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f901c007510 con 0x7f902c06a3e0 2024-01-16T15:02:47.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.698+0000 7f9023e00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f900406b860 0x7f900406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.698+0000 7f9023e00640 1 -- 172.21.15.141:0/1691217999 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f901c089ca0 con 0x7f902c06a3e0 2024-01-16T15:02:47.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.699+0000 7f902b400640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f900406b860 0x7f900406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.700+0000 7f902b400640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f900406b860 0x7f900406dd20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9014036000 tx=0x7f9014030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.703+0000 7f9023e00640 1 -- 172.21.15.141:0/1691217999 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f902c15af40 con 0x7f902c06a3e0 2024-01-16T15:02:47.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.901+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f902c06eb60 con 0x7f900406b860 2024-01-16T15:02:47.892 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:47.913+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.893 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:47.913+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.893 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:47.914+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.893 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:47.914+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.893 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:02:47.914+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.979+0000 7f9023e00640 1 -- 172.21.15.141:0/1691217999 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9014031630 con 0x7f900406b860 2024-01-16T15:02:47.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f900406b860 msgr2=0x7f900406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f900406b860 0x7f900406dd20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9014036000 tx=0x7f9014030000 comp rx=0 tx=0).stop 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c06a3e0 msgr2=0x7f902c0cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c06a3e0 0x7f902c0cf130 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f901c00c410 tx=0x7f901c00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 shutdown_connections 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f900406b860 0x7f900406dd20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f902c0d0b60 0x7f902c1e6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f902c0aede0 0x7f902c1e5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 --2- 172.21.15.141:0/1691217999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f902c06a3e0 0x7f902c0cf130 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 >> 172.21.15.141:0/1691217999 conn(0x7f902c0cfef0 msgr2=0x7f902c1e4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 shutdown_connections 2024-01-16T15:02:47.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:47.982+0000 7f9032a00640 1 -- 172.21.15.141:0/1691217999 wait complete. 2024-01-16T15:02:47.981 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:02:48.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.214+0000 7f9fade00640 1 Processor -- start 2024-01-16T15:02:48.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.214+0000 7f9fade00640 1 -- start start 2024-01-16T15:02:48.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fade00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa814b2c0 0x7f9fa80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fade00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9fa80cf7d0 0x7f9fa8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fade00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9fa8070ea0 0x7f9fa80b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fade00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa814b6a0 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fade00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa80cfbf0 con 0x7f9fa8070ea0 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fade00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8071690 con 0x7f9fa814b2c0 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fa6000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9fa80cf7d0 0x7f9fa8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fa6000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9fa80cf7d0 0x7f9fa8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58408/0 (socket says 172.21.15.141:58408) 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.216+0000 7f9fa6000640 1 -- 172.21.15.141:0/1504206558 learned_addr learned my addr 172.21.15.141:0/1504206558 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.217+0000 7f9fa6000640 1 -- 172.21.15.141:0/1504206558 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa814b2c0 msgr2=0x7f9fa80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.217+0000 7f9fa6000640 1 --2- 172.21.15.141:0/1504206558 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa814b2c0 0x7f9fa80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.217+0000 7f9fa6000640 1 -- 172.21.15.141:0/1504206558 >> v2:172.21.15.188:3300/0 conn(0x7f9fa8070ea0 msgr2=0x7f9fa80b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.217+0000 7f9fa6000640 1 --2- 172.21.15.141:0/1504206558 >> v2:172.21.15.188:3300/0 conn(0x7f9fa8070ea0 0x7f9fa80b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.217+0000 7f9fa6000640 1 -- 172.21.15.141:0/1504206558 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8071970 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.218+0000 7f9fa6000640 1 --2- 172.21.15.141:0/1504206558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8070960 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9fa806e2a0 tx=0x7f9f9002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e3ebd9fd961a98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.218+0000 7f9fa5600640 1 -- 172.21.15.141:0/1504206558 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90031860 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.218+0000 7f9fa5600640 1 -- 172.21.15.141:0/1504206558 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f900319c0 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.218+0000 7f9fa5600640 1 -- 172.21.15.141:0/1504206558 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90031d50 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 -- 172.21.15.141:0/1504206558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 msgr2=0x7f9fa8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 --2- 172.21.15.141:0/1504206558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8070960 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9fa806e2a0 tx=0x7f9f9002b900 comp rx=0 tx=0).stop 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 -- 172.21.15.141:0/1504206558 shutdown_connections 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 --2- 172.21.15.141:0/1504206558 >> v2:172.21.15.188:3300/0 conn(0x7f9fa8070ea0 0x7f9fa80b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 --2- 172.21.15.141:0/1504206558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8070960 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 --2- 172.21.15.141:0/1504206558 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa814b2c0 0x7f9fa80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 -- 172.21.15.141:0/1504206558 >> 172.21.15.141:0/1504206558 conn(0x7f9fa806a800 msgr2=0x7f9fa806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 -- 172.21.15.141:0/1504206558 shutdown_connections 2024-01-16T15:02:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.219+0000 7f9fade00640 1 -- 172.21.15.141:0/1504206558 wait complete. 2024-01-16T15:02:48.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.221+0000 7f9fade00640 1 Processor -- start 2024-01-16T15:02:48.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.221+0000 7f9fade00640 1 -- start start 2024-01-16T15:02:48.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fade00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa8070ea0 0x7f9fa8154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fade00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fade00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fa814b2c0 0x7f9fa8145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fade00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8144350 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fade00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8154580 con 0x7f9fa814b2c0 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fade00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8143ab0 con 0x7f9fa8070ea0 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58410/0 (socket says 172.21.15.141:58410) 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 -- 172.21.15.141:0/1323353908 learned_addr learned my addr 172.21.15.141:0/1323353908 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 -- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa8070ea0 msgr2=0x7f9fa8154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa8070ea0 0x7f9fa8154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 -- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fa814b2c0 msgr2=0x7f9fa8145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fa814b2c0 0x7f9fa8145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.222+0000 7f9fa6000640 1 -- 172.21.15.141:0/1323353908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f900094d0 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.223+0000 7f9fa6000640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8144cc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000c00 tx=0x7f9f900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.223+0000 7f9f9f400640 1 -- 172.21.15.141:0/1323353908 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9003ad10 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.223+0000 7f9f9f400640 1 -- 172.21.15.141:0/1323353908 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9003ae70 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.223+0000 7f9f9f400640 1 -- 172.21.15.141:0/1323353908 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90044610 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.223+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa8143d90 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.223+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa8161f40 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.224+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8072610 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.225+0000 7f9f9f400640 1 -- 172.21.15.141:0/1323353908 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f9f9004d450 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.226+0000 7f9f9f400640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f8006b860 0x7f9f8006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.226+0000 7f9f9f400640 1 -- 172.21.15.141:0/1323353908 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f900acde0 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.227+0000 7f9fa6a00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f8006b860 0x7f9f8006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.228+0000 7f9fa6a00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f8006b860 0x7f9f8006dd20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009710 tx=0x7f9f98006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.230+0000 7f9f9f400640 1 -- 172.21.15.141:0/1323353908 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f900b2020 con 0x7f9fa80cf7d0 2024-01-16T15:02:48.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.450+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fa80cbc40 con 0x7f9f8006b860 2024-01-16T15:02:48.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.462+0000 7f9f9f400640 1 -- 172.21.15.141:0/1323353908 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9f9800b480 con 0x7f9f8006b860 2024-01-16T15:02:48.442 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f8006b860 msgr2=0x7f9f8006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f8006b860 0x7f9f8006dd20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009710 tx=0x7f9f98006c90 comp rx=0 tx=0).stop 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 msgr2=0x7f9fa8144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8144cc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000c00 tx=0x7f9f900046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 shutdown_connections 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f8006b860 0x7f9f8006dd20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fa814b2c0 0x7f9fa8145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9fa80cf7d0 0x7f9fa8144cc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 --2- 172.21.15.141:0/1323353908 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9fa8070ea0 0x7f9fa8154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 >> 172.21.15.141:0/1323353908 conn(0x7f9fa806a800 msgr2=0x7f9fa80ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 shutdown_connections 2024-01-16T15:02:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:02:48.465+0000 7f9fade00640 1 -- 172.21.15.141:0/1323353908 wait complete. 2024-01-16T15:02:48.463 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c 2024-01-16T15:02:48.463 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:02:48.463 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:48.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:48.510 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:03.377 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:03:03.377 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:03:03.377 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.7419 s, 72.8 MB/s 2024-01-16T15:03:03.379 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:03:03.379 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:03:03.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:03.430 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:03.674 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:03.682 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:14.783 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:03:14.783 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:03:14.784 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.2525 s, 95.4 MB/s 2024-01-16T15:03:14.785 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:03:14.785 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:03:14.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:14.840 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:25.679 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:03:25.679 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:03:25.679 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.782 s, 99.6 MB/s 2024-01-16T15:03:25.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:03:25.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.913+0000 7f7673e00640 1 Processor -- start 2024-01-16T15:03:25.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.913+0000 7f7673e00640 1 -- start start 2024-01-16T15:03:25.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.914+0000 7f7673e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f76741608b0 0x7f76740d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7673e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76740cf6c0 con 0x7f76740c6c00 2024-01-16T15:03:25.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7673e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76740d1450 con 0x7f7674160c90 2024-01-16T15:03:25.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7673e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76740b4390 con 0x7f76741608b0 2024-01-16T15:03:25.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7673400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f76740c6c00 0x7f76740cfb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36520/0 (socket says 172.21.15.141:36520) 2024-01-16T15:03:25.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7673400640 1 -- 172.21.15.141:0/3221619867 learned_addr learned my addr 172.21.15.141:0/3221619867 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7672a00640 1 --2- 172.21.15.141:0/3221619867 >> v2:172.21.15.188:3300/0 conn(0x7f76741608b0 0x7f76740d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254129518 0 0) 0x7f76740cf6c0 con 0x7f76740c6c00 2024-01-16T15:03:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.915+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7654003520 con 0x7f76740c6c00 2024-01-16T15:03:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2008818829 0 0) 0x7f7654003520 con 0x7f76740c6c00 2024-01-16T15:03:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 >> v2:172.21.15.188:3300/0 conn(0x7f76741608b0 msgr2=0x7f76740d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 --2- 172.21.15.141:0/3221619867 >> v2:172.21.15.188:3300/0 conn(0x7f76741608b0 0x7f76740d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 >> v1:172.21.15.188:6789/0 conn(0x7f7674160c90 legacy=0x7f76740d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76740ce7e0 con 0x7f76740c6c00 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f766c0019f0 con 0x7f76740c6c00 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 >> v1:172.21.15.141:6789/0 conn(0x7f76740c6c00 legacy=0x7f76740cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540039c0 0x7f7654003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76540047c0 0x7f7654004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76540052a0 0x7f765400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76740b4390 con 0x7f76540052a0 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765400c1a0 con 0x7f76540039c0 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765400c360 con 0x7f76540047c0 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7673400640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76540052a0 0x7f765400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.916+0000 7f7672a00640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540039c0 0x7f7654003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.917+0000 7f766be00640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76540047c0 0x7f7654004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.917+0000 7f7673400640 1 -- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76540047c0 msgr2=0x7f7654004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.917+0000 7f7673400640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76540047c0 0x7f7654004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.917+0000 7f7673400640 1 -- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540039c0 msgr2=0x7f7654003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.917+0000 7f7673400640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540039c0 0x7f7654003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.917+0000 7f7673400640 1 -- 172.21.15.141:0/3221619867 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654003520 con 0x7f76540052a0 2024-01-16T15:03:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.917+0000 7f7673400640 1 -- 172.21.15.141:0/3221619867 reap_dead start 2024-01-16T15:03:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.918+0000 7f7673400640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76540052a0 0x7f765400b980 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f766c00a760 tx=0x7f76540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39fa5e41e0d6bf0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.918+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c034630 con 0x7f76540052a0 2024-01-16T15:03:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.918+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c02f070 con 0x7f76540052a0 2024-01-16T15:03:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.918+0000 7f7672000640 1 -- 172.21.15.141:0/3221619867 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c034a10 con 0x7f76540052a0 2024-01-16T15:03:25.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.918+0000 7f7673e00640 1 -- 172.21.15.141:0/3221619867 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76540052a0 msgr2=0x7f765400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:25.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.918+0000 7f7673e00640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76540052a0 0x7f765400b980 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f766c00a760 tx=0x7f76540057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:25.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.919+0000 7f7673e00640 1 -- 172.21.15.141:0/3221619867 shutdown_connections 2024-01-16T15:03:25.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.919+0000 7f7673e00640 1 --2- 172.21.15.141:0/3221619867 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76540052a0 0x7f765400b980 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:25.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.919+0000 7f7673e00640 1 -- 172.21.15.141:0/3221619867 >> 172.21.15.141:0/3221619867 conn(0x7f76740d30b0 msgr2=0x7f76740d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:25.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.919+0000 7f7673e00640 1 -- 172.21.15.141:0/3221619867 shutdown_connections 2024-01-16T15:03:25.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.919+0000 7f7673e00640 1 -- 172.21.15.141:0/3221619867 wait complete. 2024-01-16T15:03:25.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.920+0000 7f7673e00640 1 Processor -- start 2024-01-16T15:03:25.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.921+0000 7f7673e00640 1 -- start start 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.921+0000 7f7673e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540052a0 0x7f767414fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.921+0000 7f7673e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76740ccdc0 0x7f7674152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.921+0000 7f7673e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f767415a180 0x7f767415a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.921+0000 7f7673e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767415ab40 con 0x7f767415a180 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.921+0000 7f7673e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7674150170 con 0x7f76540052a0 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.921+0000 7f7673e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7674153680 con 0x7f76740ccdc0 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f767415a180 0x7f767415a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f767415a180 0x7f767415a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35784/0 (socket says 172.21.15.141:35784) 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 -- 172.21.15.141:0/195088139 learned_addr learned my addr 172.21.15.141:0/195088139 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 -- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76740ccdc0 msgr2=0x7f7674152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76740ccdc0 0x7f7674152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 -- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540052a0 msgr2=0x7f767414fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540052a0 0x7f767414fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 -- 172.21.15.141:0/195088139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7674153960 con 0x7f767415a180 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.922+0000 7f7673400640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f767415a180 0x7f767415a600 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f766c00a760 tx=0x7f766c001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.923+0000 7f766aa00640 1 -- 172.21.15.141:0/195088139 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c0019f0 con 0x7f767415a180 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.923+0000 7f766aa00640 1 -- 172.21.15.141:0/195088139 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c004b60 con 0x7f767415a180 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.923+0000 7f766aa00640 1 -- 172.21.15.141:0/195088139 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c0347b0 con 0x7f767415a180 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.923+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7674153c40 con 0x7f767415a180 2024-01-16T15:03:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.923+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7674154200 con 0x7f767415a180 2024-01-16T15:03:25.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.924+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7674067450 con 0x7f767415a180 2024-01-16T15:03:25.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.925+0000 7f766aa00640 1 -- 172.21.15.141:0/195088139 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f766c031030 con 0x7f767415a180 2024-01-16T15:03:25.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.926+0000 7f766aa00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764c06b920 0x7f764c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:25.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.926+0000 7f766aa00640 1 -- 172.21.15.141:0/195088139 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f766c0abf80 con 0x7f767415a180 2024-01-16T15:03:25.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.926+0000 7f7672a00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764c06b920 0x7f764c06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:25.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.928+0000 7f7672a00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764c06b920 0x7f764c06dde0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7664009890 tx=0x7f7664008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:25.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:25.931+0000 7f766aa00640 1 -- 172.21.15.141:0/195088139 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f766c0764e0 con 0x7f767415a180 2024-01-16T15:03:26.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.120+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f767408c800 con 0x7f767415a180 2024-01-16T15:03:26.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.125+0000 7f766aa00640 1 -- 172.21.15.141:0/195088139 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v21)=0 v21) v1 ==== 131+0+0 (secure 0 0 0) 0x7f766c0440c0 con 0x7f767415a180 2024-01-16T15:03:26.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.128+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764c06b920 msgr2=0x7f764c06dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.128+0000 7f7673e00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764c06b920 0x7f764c06dde0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7664009890 tx=0x7f7664008040 comp rx=0 tx=0).stop 2024-01-16T15:03:26.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.128+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f767415a180 msgr2=0x7f767415a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.128+0000 7f7673e00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f767415a180 0x7f767415a600 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f766c00a760 tx=0x7f766c001cb0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 shutdown_connections 2024-01-16T15:03:26.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f764c06b920 0x7f764c06dde0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f767415a180 0x7f767415a600 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76740ccdc0 0x7f7674152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 --2- 172.21.15.141:0/195088139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76540052a0 0x7f767414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 >> 172.21.15.141:0/195088139 conn(0x7f76740d30b0 msgr2=0x7f76740b4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:26.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 shutdown_connections 2024-01-16T15:03:26.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.129+0000 7f7673e00640 1 -- 172.21.15.141:0/195088139 wait complete. 2024-01-16T15:03:26.128 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:03:26.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.365+0000 7f3118e00640 1 Processor -- start 2024-01-16T15:03:26.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.365+0000 7f3118e00640 1 -- start start 2024-01-16T15:03:26.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.367+0000 7f3118e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 0x7f3114166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.367+0000 7f3118e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311415c5f0 con 0x7f311413e360 2024-01-16T15:03:26.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.367+0000 7f3118e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311406c470 con 0x7f311413df80 2024-01-16T15:03:26.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.367+0000 7f3118e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311406c750 con 0x7f31140cf9a0 2024-01-16T15:03:26.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.367+0000 7f310be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f311413e360 0x7f31140cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36536/0 (socket says 172.21.15.141:36536) 2024-01-16T15:03:26.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.367+0000 7f310be00640 1 -- 172.21.15.141:0/784627340 learned_addr learned my addr 172.21.15.141:0/784627340 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:26.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3112800640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 0x7f3114166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740848179 0 0) 0x7f311415c5f0 con 0x7f311413e360 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f4003520 con 0x7f311413e360 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393338400 0 0) 0x7f30f4003520 con 0x7f311413e360 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 msgr2=0x7f3114166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 0x7f3114166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 >> v1:172.21.15.188:6789/0 conn(0x7f311413df80 legacy=0x7f311415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311406ca30 con 0x7f311413e360 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f30fc001670 con 0x7f311413e360 2024-01-16T15:03:26.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 >> v1:172.21.15.141:6789/0 conn(0x7f311413e360 legacy=0x7f31140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.368+0000 7f3111400640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30f40039a0 0x7f30f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111400640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f40047c0 0x7f30f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111400640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 0x7f30f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311406c750 con 0x7f30f40039a0 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f400ba60 con 0x7f30f40047c0 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f400bc20 con 0x7f31140cf9a0 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111e00640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30f40039a0 0x7f30f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f310be00640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f40047c0 0x7f30f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111e00640 1 -- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 msgr2=0x7f30f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3112800640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 0x7f30f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111e00640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31140cf9a0 0x7f30f4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111e00640 1 -- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f40047c0 msgr2=0x7f30f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:26.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111e00640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f40047c0 0x7f30f4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.369+0000 7f3111e00640 1 -- 172.21.15.141:0/784627340 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f4003520 con 0x7f30f40039a0 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.370+0000 7f3111e00640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30f40039a0 0x7f30f4003dc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3104001720 tx=0x7f30f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2de3f54ced51a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.370+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040052d0 con 0x7f30f40039a0 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310402e050 con 0x7f30f40039a0 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3111400640 1 -- 172.21.15.141:0/784627340 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040035f0 con 0x7f30f40039a0 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3118e00640 1 -- 172.21.15.141:0/784627340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30f40039a0 msgr2=0x7f30f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3118e00640 1 --2- 172.21.15.141:0/784627340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30f40039a0 0x7f30f4003dc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3104001720 tx=0x7f30f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3112800640 1 -- 172.21.15.141:0/784627340 reap_dead start 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3118e00640 1 -- 172.21.15.141:0/784627340 shutdown_connections 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3118e00640 1 -- 172.21.15.141:0/784627340 >> 172.21.15.141:0/784627340 conn(0x7f311406a800 msgr2=0x7f311406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3118e00640 1 -- 172.21.15.141:0/784627340 shutdown_connections 2024-01-16T15:03:26.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.371+0000 7f3118e00640 1 -- 172.21.15.141:0/784627340 wait complete. 2024-01-16T15:03:26.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.373+0000 7f3118e00640 1 Processor -- start 2024-01-16T15:03:26.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.373+0000 7f3118e00640 1 -- start start 2024-01-16T15:03:26.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f3118e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30f40039a0 0x7f3114150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f3118e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f311413cfc0 0x7f311414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f3118e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31141532f0 0x7f3114153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f3118e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3114155d70 con 0x7f311413cfc0 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f3118e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3114153cb0 con 0x7f31141532f0 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f3118e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311413d440 con 0x7f30f40039a0 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f310be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f311413cfc0 0x7f311414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f310be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f311413cfc0 0x7f311414d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35800/0 (socket says 172.21.15.141:35800) 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.374+0000 7f310be00640 1 -- 172.21.15.141:0/522448379 learned_addr learned my addr 172.21.15.141:0/522448379 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f3112800640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31141532f0 0x7f3114153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f3111e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30f40039a0 0x7f3114150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310be00640 1 -- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30f40039a0 msgr2=0x7f3114150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310be00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30f40039a0 0x7f3114150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310be00640 1 -- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31141532f0 msgr2=0x7f3114153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310be00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31141532f0 0x7f3114153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310be00640 1 -- 172.21.15.141:0/522448379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311414f370 con 0x7f311413cfc0 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f3112800640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31141532f0 0x7f3114153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310be00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f311413cfc0 0x7f311414d390 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00d7a0 tx=0x7f30fc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310a000640 1 -- 172.21.15.141:0/522448379 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc0074e0 con 0x7f311413cfc0 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310a000640 1 -- 172.21.15.141:0/522448379 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30fc007640 con 0x7f311413cfc0 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f310a000640 1 -- 172.21.15.141:0/522448379 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc004a00 con 0x7f311413cfc0 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.375+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311414f650 con 0x7f311413cfc0 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.376+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311414fc10 con 0x7f311413cfc0 2024-01-16T15:03:26.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.377+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31140675d0 con 0x7f311413cfc0 2024-01-16T15:03:26.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.377+0000 7f310a000640 1 -- 172.21.15.141:0/522448379 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f30fc004b60 con 0x7f311413cfc0 2024-01-16T15:03:26.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.378+0000 7f310a000640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f30ec06b8f0 0x7f30ec06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.379+0000 7f310a000640 1 -- 172.21.15.141:0/522448379 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30fc08a120 con 0x7f311413cfc0 2024-01-16T15:03:26.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.379+0000 7f3111e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f30ec06b8f0 0x7f30ec06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.380+0000 7f3111e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f30ec06b8f0 0x7f30ec06ddb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3104033000 tx=0x7f3104009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.383+0000 7f310a000640 1 -- 172.21.15.141:0/522448379 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30fc057f90 con 0x7f311413cfc0 2024-01-16T15:03:26.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.573+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f311406df00 con 0x7f311413cfc0 2024-01-16T15:03:26.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.579+0000 7f310a000640 1 -- 172.21.15.141:0/522448379 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f30fc0190c0 con 0x7f311413cfc0 2024-01-16T15:03:26.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f30ec06b8f0 msgr2=0x7f30ec06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f30ec06b8f0 0x7f30ec06ddb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3104033000 tx=0x7f3104009030 comp rx=0 tx=0).stop 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f311413cfc0 msgr2=0x7f311414d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f311413cfc0 0x7f311414d390 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00d7a0 tx=0x7f30fc00dc60 comp rx=0 tx=0).stop 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 shutdown_connections 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f30ec06b8f0 0x7f30ec06ddb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31141532f0 0x7f3114153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f311413cfc0 0x7f311414d390 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 --2- 172.21.15.141:0/522448379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30f40039a0 0x7f3114150410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 >> 172.21.15.141:0/522448379 conn(0x7f311406a800 msgr2=0x7f31140cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 shutdown_connections 2024-01-16T15:03:26.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.582+0000 7f3118e00640 1 -- 172.21.15.141:0/522448379 wait complete. 2024-01-16T15:03:26.582 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:03:26.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.815+0000 7f7b77800640 1 Processor -- start 2024-01-16T15:03:26.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.815+0000 7f7b77800640 1 -- start start 2024-01-16T15:03:26.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b77800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7b7013e360 0x7f7b700cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b77800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7b7015f9c0 0x7f7b700cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b77800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7015c5f0 con 0x7f7b7013df80 2024-01-16T15:03:26.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b77800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7006c470 con 0x7f7b7015f9c0 2024-01-16T15:03:26.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b77800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7006c750 con 0x7f7b7013e360 2024-01-16T15:03:26.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b74e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7b7013df80 0x7f7b7015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36540/0 (socket says 172.21.15.141:36540) 2024-01-16T15:03:26.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b74e00640 1 -- 172.21.15.141:0/4101691763 learned_addr learned my addr 172.21.15.141:0/4101691763 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:26.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.817+0000 7f7b75800640 1 --2- 172.21.15.141:0/4101691763 >> v2:172.21.15.141:3300/0 conn(0x7f7b7015f9c0 0x7f7b700cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b67c00640 1 --2- 172.21.15.141:0/4101691763 >> v2:172.21.15.188:3300/0 conn(0x7f7b7013e360 0x7f7b700cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b6fe00640 1 -- 172.21.15.141:0/4101691763 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896326215 0 0) 0x7f7b7015c5f0 con 0x7f7b7013df80 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b6fe00640 1 -- 172.21.15.141:0/4101691763 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b50003520 con 0x7f7b7013df80 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b75800640 1 -- 172.21.15.141:0/4101691763 >> v2:172.21.15.188:3300/0 conn(0x7f7b7013e360 msgr2=0x7f7b700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b75800640 1 --2- 172.21.15.141:0/4101691763 >> v2:172.21.15.188:3300/0 conn(0x7f7b7013e360 0x7f7b700cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b75800640 1 -- 172.21.15.141:0/4101691763 >> v1:172.21.15.141:6789/0 conn(0x7f7b7013df80 legacy=0x7f7b7015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b75800640 1 -- 172.21.15.141:0/4101691763 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7006ca30 con 0x7f7b7015f9c0 2024-01-16T15:03:26.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.818+0000 7f7b67c00640 1 --2- 172.21.15.141:0/4101691763 >> v2:172.21.15.188:3300/0 conn(0x7f7b7013e360 0x7f7b700cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:26.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.819+0000 7f7b75800640 1 --2- 172.21.15.141:0/4101691763 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7015f9c0 0x7f7b700cf7d0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7b7006ded0 tx=0x7f7b6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefcfa16c91d7475 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b6fe00640 1 -- 172.21.15.141:0/4101691763 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68031830 con 0x7f7b7015f9c0 2024-01-16T15:03:26.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b6fe00640 1 -- 172.21.15.141:0/4101691763 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b68031990 con 0x7f7b7015f9c0 2024-01-16T15:03:26.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b6fe00640 1 -- 172.21.15.141:0/4101691763 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68031d20 con 0x7f7b7015f9c0 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b77800640 1 -- 172.21.15.141:0/4101691763 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7015f9c0 msgr2=0x7f7b700cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b77800640 1 --2- 172.21.15.141:0/4101691763 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7015f9c0 0x7f7b700cf7d0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7b7006ded0 tx=0x7f7b6802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b77800640 1 -- 172.21.15.141:0/4101691763 shutdown_connections 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b77800640 1 --2- 172.21.15.141:0/4101691763 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7015f9c0 0x7f7b700cf7d0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b77800640 1 --2- 172.21.15.141:0/4101691763 >> v2:172.21.15.188:3300/0 conn(0x7f7b7013e360 0x7f7b700cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b77800640 1 -- 172.21.15.141:0/4101691763 >> 172.21.15.141:0/4101691763 conn(0x7f7b7006a800 msgr2=0x7f7b7006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.820+0000 7f7b77800640 1 -- 172.21.15.141:0/4101691763 shutdown_connections 2024-01-16T15:03:26.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.821+0000 7f7b77800640 1 -- 172.21.15.141:0/4101691763 wait complete. 2024-01-16T15:03:26.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.822+0000 7f7b77800640 1 Processor -- start 2024-01-16T15:03:26.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.822+0000 7f7b77800640 1 -- start start 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.823+0000 7f7b77800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7013df80 0x7f7b7013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.823+0000 7f7b77800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7013e360 0x7f7b7015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.823+0000 7f7b77800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b7015f9c0 0x7f7b7015dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.823+0000 7f7b77800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b701f2d00 con 0x7f7b7013df80 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.823+0000 7f7b77800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b700a8660 con 0x7f7b7013e360 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.823+0000 7f7b77800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b700a8940 con 0x7f7b7015f9c0 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7013df80 0x7f7b7013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7013df80 0x7f7b7013d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35818/0 (socket says 172.21.15.141:35818) 2024-01-16T15:03:26.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 -- 172.21.15.141:0/1749491208 learned_addr learned my addr 172.21.15.141:0/1749491208 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 -- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b7015f9c0 msgr2=0x7f7b7015dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b75800640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b7015f9c0 0x7f7b7015dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b67c00640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7013e360 0x7f7b7015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b7015f9c0 0x7f7b7015dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 -- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7013e360 msgr2=0x7f7b7015d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7013e360 0x7f7b7015d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 -- 172.21.15.141:0/1749491208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68009400 con 0x7f7b7013df80 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.824+0000 7f7b74e00640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7013df80 0x7f7b7013d8d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000cbb0 tx=0x7f7b6000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.825+0000 7f7b6d600640 1 -- 172.21.15.141:0/1749491208 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000ad40 con 0x7f7b7013df80 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.825+0000 7f7b6d600640 1 -- 172.21.15.141:0/1749491208 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b60012400 con 0x7f7b7013df80 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.825+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b700a8c20 con 0x7f7b7013df80 2024-01-16T15:03:26.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.825+0000 7f7b6d600640 1 -- 172.21.15.141:0/1749491208 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000aea0 con 0x7f7b7013df80 2024-01-16T15:03:26.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.825+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b700a91e0 con 0x7f7b7013df80 2024-01-16T15:03:26.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.826+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b700675d0 con 0x7f7b7013df80 2024-01-16T15:03:26.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.827+0000 7f7b6d600640 1 -- 172.21.15.141:0/1749491208 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f7b60012770 con 0x7f7b7013df80 2024-01-16T15:03:26.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.828+0000 7f7b6d600640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b4806b8f0 0x7f7b4806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:26.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.828+0000 7f7b6d600640 1 -- 172.21.15.141:0/1749491208 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b6008a5a0 con 0x7f7b7013df80 2024-01-16T15:03:26.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.829+0000 7f7b67c00640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b4806b8f0 0x7f7b4806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:26.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.830+0000 7f7b67c00640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b4806b8f0 0x7f7b4806ddb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7b58006fd0 tx=0x7f7b58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:26.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:26.833+0000 7f7b6d600640 1 -- 172.21.15.141:0/1749491208 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b60054b80 con 0x7f7b7013df80 2024-01-16T15:03:27.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.057+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b7006cff0 con 0x7f7b4806b8f0 2024-01-16T15:03:27.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.091+0000 7f7b6d600640 1 -- 172.21.15.141:0/1749491208 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7b580094e0 con 0x7f7b4806b8f0 2024-01-16T15:03:27.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.094+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b4806b8f0 msgr2=0x7f7b4806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:27.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.094+0000 7f7b77800640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b4806b8f0 0x7f7b4806ddb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7b58006fd0 tx=0x7f7b58008040 comp rx=0 tx=0).stop 2024-01-16T15:03:27.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.094+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7013df80 msgr2=0x7f7b7013d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:27.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.094+0000 7f7b77800640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7013df80 0x7f7b7013d8d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000cbb0 tx=0x7f7b6000a430 comp rx=0 tx=0).stop 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 shutdown_connections 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7b4806b8f0 0x7f7b4806ddb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7b7015f9c0 0x7f7b7015dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7013e360 0x7f7b7015d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 --2- 172.21.15.141:0/1749491208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b7013df80 0x7f7b7013d8d0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 >> 172.21.15.141:0/1749491208 conn(0x7f7b7006a800 msgr2=0x7f7b700b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 shutdown_connections 2024-01-16T15:03:27.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.095+0000 7f7b77800640 1 -- 172.21.15.141:0/1749491208 wait complete. 2024-01-16T15:03:27.094 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:03:27.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.323+0000 7f3ed5200640 1 Processor -- start 2024-01-16T15:03:27.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.324+0000 7f3ed5200640 1 -- start start 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.325+0000 7f3ed5200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3ed00cf9a0 0x7f3ed00cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.325+0000 7f3ed5200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3ed00b31d0 0x7f3ed00b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.325+0000 7f3ed5200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed00d0320 con 0x7f3ed015d4b0 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.325+0000 7f3ed5200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed00d1c60 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.325+0000 7f3ed5200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed00b2730 con 0x7f3ed00b31d0 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.325+0000 7f3ecea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3ed00cf9a0 0x7f3ed00cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3ed00cf9a0 0x7f3ed00cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35832/0 (socket says 172.21.15.141:35832) 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecea00640 1 -- 172.21.15.141:0/3183079297 learned_addr learned my addr 172.21.15.141:0/3183079297 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecfe00640 1 --2- 172.21.15.141:0/3183079297 >> v2:172.21.15.188:3300/0 conn(0x7f3ed00b31d0 0x7f3ed00b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecea00640 1 -- 172.21.15.141:0/3183079297 >> v2:172.21.15.188:3300/0 conn(0x7f3ed00b31d0 msgr2=0x7f3ed00b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:27.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecea00640 1 --2- 172.21.15.141:0/3183079297 >> v2:172.21.15.188:3300/0 conn(0x7f3ed00b31d0 0x7f3ed00b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecea00640 1 -- 172.21.15.141:0/3183079297 >> v1:172.21.15.188:6789/0 conn(0x7f3ed015d4b0 legacy=0x7f3ed00d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:27.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecea00640 1 -- 172.21.15.141:0/3183079297 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed00d1ec0 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.326+0000 7f3ecfe00640 1 --2- 172.21.15.141:0/3183079297 >> v2:172.21.15.188:3300/0 conn(0x7f3ed00b31d0 0x7f3ed00b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:27.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.327+0000 7f3ecea00640 1 --2- 172.21.15.141:0/3183079297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed00cfde0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0072a30 tx=0x7f3eb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed79596dbe0432f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:27.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.327+0000 7f3ece000640 1 -- 172.21.15.141:0/3183079297 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8031870 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ece000640 1 -- 172.21.15.141:0/3183079297 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb80319d0 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ece000640 1 -- 172.21.15.141:0/3183079297 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8031d60 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 -- 172.21.15.141:0/3183079297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 msgr2=0x7f3ed00cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:27.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 --2- 172.21.15.141:0/3183079297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed00cfde0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0072a30 tx=0x7f3eb802b910 comp rx=0 tx=0).stop 2024-01-16T15:03:27.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 -- 172.21.15.141:0/3183079297 shutdown_connections 2024-01-16T15:03:27.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 --2- 172.21.15.141:0/3183079297 >> v2:172.21.15.188:3300/0 conn(0x7f3ed00b31d0 0x7f3ed00b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 --2- 172.21.15.141:0/3183079297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed00cfde0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 -- 172.21.15.141:0/3183079297 >> 172.21.15.141:0/3183079297 conn(0x7f3ed00d3000 msgr2=0x7f3ed00d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:27.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 -- 172.21.15.141:0/3183079297 shutdown_connections 2024-01-16T15:03:27.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.328+0000 7f3ed5200640 1 -- 172.21.15.141:0/3183079297 wait complete. 2024-01-16T15:03:27.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.330+0000 7f3ed5200640 1 Processor -- start 2024-01-16T15:03:27.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.330+0000 7f3ed5200640 1 -- start start 2024-01-16T15:03:27.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.331+0000 7f3ed5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ed00b31d0 0x7f3ed00c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:27.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.331+0000 7f3ed5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:27.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.331+0000 7f3ed5200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ed015d4b0 0x7f3ed015b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:27.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.331+0000 7f3ed5200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed00a0ae0 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.331+0000 7f3ed5200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed00c9580 con 0x7f3ed00b31d0 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.331+0000 7f3ed5200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed015ab70 con 0x7f3ed015d4b0 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed009cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35844/0 (socket says 172.21.15.141:35844) 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 -- 172.21.15.141:0/1106402680 learned_addr learned my addr 172.21.15.141:0/1106402680 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecf400640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ed00b31d0 0x7f3ed00c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecfe00640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ed015d4b0 0x7f3ed015b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 -- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ed015d4b0 msgr2=0x7f3ed015b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ed015d4b0 0x7f3ed015b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 -- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ed00b31d0 msgr2=0x7f3ed00c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ed00b31d0 0x7f3ed00c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 -- 172.21.15.141:0/1106402680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80094d0 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecf400640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ed00b31d0 0x7f3ed00c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.332+0000 7f3ecea00640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed009cce0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8000c00 tx=0x7f3eb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.333+0000 7f3ec7400640 1 -- 172.21.15.141:0/1106402680 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8031840 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.333+0000 7f3ec7400640 1 -- 172.21.15.141:0/1106402680 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb80443f0 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.333+0000 7f3ec7400640 1 -- 172.21.15.141:0/1106402680 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80447a0 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.333+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed009fed0 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.333+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed00a0490 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.334+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed00a0c90 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.335+0000 7f3ec7400640 1 -- 172.21.15.141:0/1106402680 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f3eb804d450 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.336+0000 7f3ec7400640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ea806b8f0 0x7f3ea806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:27.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.336+0000 7f3ec7400640 1 -- 172.21.15.141:0/1106402680 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3eb802d040 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.336+0000 7f3ecf400640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ea806b8f0 0x7f3ea806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:27.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.337+0000 7f3ecf400640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ea806b8f0 0x7f3ea806ddb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000a810 tx=0x7f3ec0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:27.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.340+0000 7f3ec7400640 1 -- 172.21.15.141:0/1106402680 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3eb8077410 con 0x7f3ed00cf9a0 2024-01-16T15:03:27.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.558+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ed00d21a0 con 0x7f3ea806b8f0 2024-01-16T15:03:27.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.570+0000 7f3ec7400640 1 -- 172.21.15.141:0/1106402680 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f3ec000ad90 con 0x7f3ea806b8f0 2024-01-16T15:03:27.550 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c 2024-01-16T15:03:27.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.572+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ea806b8f0 msgr2=0x7f3ea806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:27.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.572+0000 7f3ed5200640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ea806b8f0 0x7f3ea806ddb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000a810 tx=0x7f3ec0008040 comp rx=0 tx=0).stop 2024-01-16T15:03:27.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.572+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 msgr2=0x7f3ed009cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:27.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.572+0000 7f3ed5200640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed009cce0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8000c00 tx=0x7f3eb80046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.572+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 shutdown_connections 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.573+0000 7f3ed5200640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3ea806b8f0 0x7f3ea806ddb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.573+0000 7f3ed5200640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ed015d4b0 0x7f3ed015b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.573+0000 7f3ed5200640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ed00cf9a0 0x7f3ed009cce0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.573+0000 7f3ed5200640 1 --2- 172.21.15.141:0/1106402680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ed00b31d0 0x7f3ed00c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.573+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 >> 172.21.15.141:0/1106402680 conn(0x7f3ed00d3000 msgr2=0x7f3ed006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.573+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 shutdown_connections 2024-01-16T15:03:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:27.573+0000 7f3ed5200640 1 -- 172.21.15.141:0/1106402680 wait complete. 2024-01-16T15:03:27.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:27.573 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c) 2024-01-16T15:03:27.629 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c 2024-01-16T15:03:27.629 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:03:27.629 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:03:27.630 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:03:29.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:29.632 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c) 2024-01-16T15:03:29.700 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/05ae3cb7-be11-4346-a128-ec7ffe03672c 2024-01-16T15:03:29.700 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:03:29.700 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:03:29.702 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:03:29.702 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66721 2024-01-16T15:03:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.949+0000 7fe7afc00640 1 Processor -- start 2024-01-16T15:03:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.949+0000 7fe7afc00640 1 -- start start 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7afc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe7a80ae5a0 0x7fe7a815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7afc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe7a80b3ae0 0x7fe7a806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7afc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a8070c40 con 0x7fe7a806d120 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7afc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a8071450 con 0x7fe7a80ae5a0 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7afc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a806c470 con 0x7fe7a80b3ae0 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7adc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe7a806d120 0x7fe7a80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36548/0 (socket says 172.21.15.141:36548) 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7adc00640 1 -- 172.21.15.141:0/2226594077 learned_addr learned my addr 172.21.15.141:0/2226594077 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.951+0000 7fe7ad200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe7a80ae5a0 0x7fe7a815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.952+0000 7fe7a7e00640 1 --2- 172.21.15.141:0/2226594077 >> v2:172.21.15.188:3300/0 conn(0x7fe7a80b3ae0 0x7fe7a806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.952+0000 7fe7a7400640 1 -- 172.21.15.141:0/2226594077 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656763710 0 0) 0x7fe7a8070c40 con 0x7fe7a806d120 2024-01-16T15:03:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.952+0000 7fe7a7400640 1 -- 172.21.15.141:0/2226594077 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe788003520 con 0x7fe7a806d120 2024-01-16T15:03:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.952+0000 7fe7ad200640 1 -- 172.21.15.141:0/2226594077 >> v2:172.21.15.188:3300/0 conn(0x7fe7a80b3ae0 msgr2=0x7fe7a806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.952+0000 7fe7ad200640 1 --2- 172.21.15.141:0/2226594077 >> v2:172.21.15.188:3300/0 conn(0x7fe7a80b3ae0 0x7fe7a806cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.952+0000 7fe7ad200640 1 -- 172.21.15.141:0/2226594077 >> v1:172.21.15.141:6789/0 conn(0x7fe7a806d120 legacy=0x7fe7a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.952+0000 7fe7ad200640 1 -- 172.21.15.141:0/2226594077 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a806c750 con 0x7fe7a80ae5a0 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.953+0000 7fe7ad200640 1 --2- 172.21.15.141:0/2226594077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a80ae5a0 0x7fe7a815e430 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80cfea0 tx=0x7fe79802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b76cf70bcab016 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7a7400640 1 -- 172.21.15.141:0/2226594077 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe798031830 con 0x7fe7a80ae5a0 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7a7400640 1 -- 172.21.15.141:0/2226594077 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe798031990 con 0x7fe7a80ae5a0 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7a7400640 1 -- 172.21.15.141:0/2226594077 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe798031d20 con 0x7fe7a80ae5a0 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 -- 172.21.15.141:0/2226594077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a80ae5a0 msgr2=0x7fe7a815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 --2- 172.21.15.141:0/2226594077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a80ae5a0 0x7fe7a815e430 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80cfea0 tx=0x7fe79802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 -- 172.21.15.141:0/2226594077 shutdown_connections 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 --2- 172.21.15.141:0/2226594077 >> v2:172.21.15.188:3300/0 conn(0x7fe7a80b3ae0 0x7fe7a806cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 --2- 172.21.15.141:0/2226594077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a80ae5a0 0x7fe7a815e430 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 -- 172.21.15.141:0/2226594077 >> 172.21.15.141:0/2226594077 conn(0x7fe7a806a800 msgr2=0x7fe7a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 -- 172.21.15.141:0/2226594077 shutdown_connections 2024-01-16T15:03:29.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.954+0000 7fe7afc00640 1 -- 172.21.15.141:0/2226594077 wait complete. 2024-01-16T15:03:29.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.956+0000 7fe7afc00640 1 Processor -- start 2024-01-16T15:03:29.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.956+0000 7fe7afc00640 1 -- start start 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.957+0000 7fe7afc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a806d120 0x7fe7a81f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.957+0000 7fe7afc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe7a80ae5a0 0x7fe7a815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.957+0000 7fe7afc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7a80b3ae0 0x7fe7a80ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.957+0000 7fe7afc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a80cd680 con 0x7fe7a806d120 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.957+0000 7fe7afc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a81f6f70 con 0x7fe7a80b3ae0 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.957+0000 7fe7afc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a80cdf00 con 0x7fe7a80ae5a0 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.957+0000 7fe7ad200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a806d120 0x7fe7a81f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a806d120 0x7fe7a81f2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35856/0 (socket says 172.21.15.141:35856) 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 -- 172.21.15.141:0/3087924678 learned_addr learned my addr 172.21.15.141:0/3087924678 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 -- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe7a80ae5a0 msgr2=0x7fe7a815b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe7a80ae5a0 0x7fe7a815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 -- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7a80b3ae0 msgr2=0x7fe7a80ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7a80b3ae0 0x7fe7a80ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 -- 172.21.15.141:0/3087924678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798009400 con 0x7fe7a806d120 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7ad200640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a806d120 0x7fe7a81f2d70 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe798009890 tx=0x7fe7980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.958+0000 7fe7a4c00640 1 -- 172.21.15.141:0/3087924678 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79803a6f0 con 0x7fe7a806d120 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.959+0000 7fe7a4c00640 1 -- 172.21.15.141:0/3087924678 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79803a870 con 0x7fe7a806d120 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.959+0000 7fe7a4c00640 1 -- 172.21.15.141:0/3087924678 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79803ac20 con 0x7fe7a806d120 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.959+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a81f6c90 con 0x7fe7a806d120 2024-01-16T15:03:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.959+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a80cdb90 con 0x7fe7a806d120 2024-01-16T15:03:29.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.960+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a806cf10 con 0x7fe7a806d120 2024-01-16T15:03:29.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.961+0000 7fe7a4c00640 1 -- 172.21.15.141:0/3087924678 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fe798043410 con 0x7fe7a806d120 2024-01-16T15:03:29.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.961+0000 7fe7a4c00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe78006b8f0 0x7fe78006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:29.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.962+0000 7fe7a4c00640 1 -- 172.21.15.141:0/3087924678 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7980ac450 con 0x7fe7a806d120 2024-01-16T15:03:29.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.962+0000 7fe7a7e00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe78006b8f0 0x7fe78006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:29.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.963+0000 7fe7a7e00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe78006b8f0 0x7fe78006ddb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe790006fd0 tx=0x7fe790008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:29.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:29.966+0000 7fe7a4c00640 1 -- 172.21.15.141:0/3087924678 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe798076830 con 0x7fe7a806d120 2024-01-16T15:03:30.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.194+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66721", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7a8070550 con 0x7fe78006b8f0 2024-01-16T15:03:30.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.361+0000 7fe7a4c00640 1 -- 172.21.15.141:0/3087924678 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe79000be20 con 0x7fe78006b8f0 2024-01-16T15:03:30.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.364+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe78006b8f0 msgr2=0x7fe78006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe78006b8f0 0x7fe78006ddb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe790006fd0 tx=0x7fe790008040 comp rx=0 tx=0).stop 2024-01-16T15:03:30.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a806d120 msgr2=0x7fe7a81f2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a806d120 0x7fe7a81f2d70 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe798009890 tx=0x7fe7980046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 shutdown_connections 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe78006b8f0 0x7fe78006ddb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7a80b3ae0 0x7fe7a80ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe7a80ae5a0 0x7fe7a815b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 --2- 172.21.15.141:0/3087924678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7a806d120 0x7fe7a81f2d70 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 >> 172.21.15.141:0/3087924678 conn(0x7fe7a806a800 msgr2=0x7fe7a80cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 shutdown_connections 2024-01-16T15:03:30.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.365+0000 7fe7afc00640 1 -- 172.21.15.141:0/3087924678 wait complete. 2024-01-16T15:03:30.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.370+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.370+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.370+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.370+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.370+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.364 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66722 2024-01-16T15:03:30.377 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.397+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.377 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.397+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.377 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.397+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.377 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.397+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.377 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:30.397+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:30.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.585+0000 7fbe2ca00640 1 Processor -- start 2024-01-16T15:03:30.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.585+0000 7fbe2ca00640 1 -- start start 2024-01-16T15:03:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.587+0000 7fbe2ca00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fbe2413df80 0x7fbe2415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.587+0000 7fbe2ca00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbe2413e360 0x7fbe240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.587+0000 7fbe2ca00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2406c470 con 0x7fbe2415f9c0 2024-01-16T15:03:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.587+0000 7fbe2ca00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2406c750 con 0x7fbe2413e360 2024-01-16T15:03:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.587+0000 7fbe2ca00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2406ca30 con 0x7fbe2413df80 2024-01-16T15:03:30.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.587+0000 7fbe29600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbe2413e360 0x7fbe240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.588+0000 7fbe29600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbe2413e360 0x7fbe240cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35872/0 (socket says 172.21.15.141:35872) 2024-01-16T15:03:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.588+0000 7fbe29600640 1 -- 172.21.15.141:0/2069333670 learned_addr learned my addr 172.21.15.141:0/2069333670 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.588+0000 7fbe2a000640 1 --2- 172.21.15.141:0/2069333670 >> v2:172.21.15.188:3300/0 conn(0x7fbe2413df80 0x7fbe2415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.588+0000 7fbe29600640 1 -- 172.21.15.141:0/2069333670 >> v2:172.21.15.188:3300/0 conn(0x7fbe2413df80 msgr2=0x7fbe2415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.588+0000 7fbe29600640 1 --2- 172.21.15.141:0/2069333670 >> v2:172.21.15.188:3300/0 conn(0x7fbe2413df80 0x7fbe2415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.588+0000 7fbe29600640 1 -- 172.21.15.141:0/2069333670 >> v1:172.21.15.188:6789/0 conn(0x7fbe2415f9c0 legacy=0x7fbe241408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:30.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.588+0000 7fbe29600640 1 -- 172.21.15.141:0/2069333670 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2406cd10 con 0x7fbe2413e360 2024-01-16T15:03:30.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.589+0000 7fbe29600640 1 --2- 172.21.15.141:0/2069333670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe240cf290 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbe24072770 tx=0x7fbe1402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f11319ac75c0feb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:30.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.590+0000 7fbe28c00640 1 -- 172.21.15.141:0/2069333670 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1402ca80 con 0x7fbe2413e360 2024-01-16T15:03:30.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.590+0000 7fbe28c00640 1 -- 172.21.15.141:0/2069333670 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1402cbe0 con 0x7fbe2413e360 2024-01-16T15:03:30.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.590+0000 7fbe28c00640 1 -- 172.21.15.141:0/2069333670 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe140343f0 con 0x7fbe2413e360 2024-01-16T15:03:30.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 -- 172.21.15.141:0/2069333670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 msgr2=0x7fbe240cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/2069333670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe240cf290 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbe24072770 tx=0x7fbe1402bee0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 -- 172.21.15.141:0/2069333670 shutdown_connections 2024-01-16T15:03:30.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/2069333670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe240cf290 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/2069333670 >> v2:172.21.15.188:3300/0 conn(0x7fbe2413df80 0x7fbe2415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 -- 172.21.15.141:0/2069333670 >> 172.21.15.141:0/2069333670 conn(0x7fbe2406a800 msgr2=0x7fbe2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:30.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 -- 172.21.15.141:0/2069333670 shutdown_connections 2024-01-16T15:03:30.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.591+0000 7fbe2ca00640 1 -- 172.21.15.141:0/2069333670 wait complete. 2024-01-16T15:03:30.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.593+0000 7fbe2ca00640 1 Processor -- start 2024-01-16T15:03:30.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.593+0000 7fbe2ca00640 1 -- start start 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe2413df80 0x7fbe241f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe241505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2ca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe2415f9c0 0x7fbe24150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2ca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2414e3b0 con 0x7fbe2413e360 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2ca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2414dda0 con 0x7fbe2415f9c0 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe241f3390 con 0x7fbe2413df80 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe29600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe241505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe29600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe241505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35880/0 (socket says 172.21.15.141:35880) 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe29600640 1 -- 172.21.15.141:0/247606439 learned_addr learned my addr 172.21.15.141:0/247606439 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2aa00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe2415f9c0 0x7fbe24150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe2a000640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe2413df80 0x7fbe241f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe29600640 1 -- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe2413df80 msgr2=0x7fbe241f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.594+0000 7fbe29600640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe2413df80 0x7fbe241f2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe29600640 1 -- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe2415f9c0 msgr2=0x7fbe24150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe29600640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe2415f9c0 0x7fbe24150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe29600640 1 -- 172.21.15.141:0/247606439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14009b30 con 0x7fbe2413e360 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe2aa00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe2415f9c0 0x7fbe24150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe29600640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe241505f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fbe14000f80 tx=0x7fbe14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14009510 con 0x7fbe2413e360 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1403e450 con 0x7fbe2413e360 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1403e800 con 0x7fbe2413e360 2024-01-16T15:03:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe240b3350 con 0x7fbe2413e360 2024-01-16T15:03:30.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.595+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe240b3630 con 0x7fbe2413e360 2024-01-16T15:03:30.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.596+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2414e520 con 0x7fbe2413e360 2024-01-16T15:03:30.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.597+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fbe1402cd50 con 0x7fbe2413e360 2024-01-16T15:03:30.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.599+0000 7fbe22000640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbdfc06b8f0 0x7fbdfc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:30.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.599+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe140ad2c0 con 0x7fbe2413e360 2024-01-16T15:03:30.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.599+0000 7fbe2a000640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbdfc06b8f0 0x7fbdfc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:30.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.600+0000 7fbe2a000640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbdfc06b8f0 0x7fbdfc06ddb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c006fd0 tx=0x7fbe0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:30.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.603+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe140777f0 con 0x7fbe2413e360 2024-01-16T15:03:30.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.716+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fbe14077bc0 con 0x7fbe2413e360 2024-01-16T15:03:30.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:30.836+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66722", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe240ce850 con 0x7fbdfc06b8f0 2024-01-16T15:03:30.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.001+0000 7fbe22000640 1 -- 172.21.15.141:0/247606439 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe0c009480 con 0x7fbdfc06b8f0 2024-01-16T15:03:30.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbdfc06b8f0 msgr2=0x7fbdfc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbdfc06b8f0 0x7fbdfc06ddb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c006fd0 tx=0x7fbe0c008040 comp rx=0 tx=0).stop 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 msgr2=0x7fbe241505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe241505f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fbe14000f80 tx=0x7fbe14004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 shutdown_connections 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbdfc06b8f0 0x7fbdfc06ddb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe2415f9c0 0x7fbe24150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe2413e360 0x7fbe241505f0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 --2- 172.21.15.141:0/247606439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe2413df80 0x7fbe241f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.004+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 >> 172.21.15.141:0/247606439 conn(0x7fbe2406a800 msgr2=0x7fbe240b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:30.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.005+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 shutdown_connections 2024-01-16T15:03:30.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.005+0000 7fbe2ca00640 1 -- 172.21.15.141:0/247606439 wait complete. 2024-01-16T15:03:31.005 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66723 2024-01-16T15:03:31.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.241+0000 7fd75fe00640 1 Processor -- start 2024-01-16T15:03:31.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.241+0000 7fd75fe00640 1 -- start start 2024-01-16T15:03:31.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.242+0000 7fd75fe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd760143c40 0x7fd760071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd75fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd760070e50 con 0x7fd7600717d0 2024-01-16T15:03:31.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd75fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd760144020 con 0x7fd7600ccc10 2024-01-16T15:03:31.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd75fe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600723c0 con 0x7fd760143c40 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd755e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd7600717d0 0x7fd7600cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36562/0 (socket says 172.21.15.141:36562) 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd755e00640 1 -- 172.21.15.141:0/1756242768 learned_addr learned my addr 172.21.15.141:0/1756242768 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707377274 0 0) 0x7fd760070e50 con 0x7fd7600717d0 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd740003520 con 0x7fd7600717d0 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.243+0000 7fd75ea00640 1 --2- 172.21.15.141:0/1756242768 >> v2:172.21.15.188:3300/0 conn(0x7fd760143c40 0x7fd760071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2803512986 0 0) 0x7fd740003520 con 0x7fd7600717d0 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 >> v2:172.21.15.188:3300/0 conn(0x7fd760143c40 msgr2=0x7fd760071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 --2- 172.21.15.141:0/1756242768 >> v2:172.21.15.188:3300/0 conn(0x7fd760143c40 0x7fd760071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 >> v1:172.21.15.188:6789/0 conn(0x7fd7600ccc10 legacy=0x7fd76006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd760071e00 con 0x7fd7600717d0 2024-01-16T15:03:31.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd748001670 con 0x7fd7600717d0 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 >> v1:172.21.15.141:6789/0 conn(0x7fd7600717d0 legacy=0x7fd7600cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd740003970 0x7fd740003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd740004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7400052a0 0x7fd74000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600723c0 con 0x7fd7400047c0 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74000c170 con 0x7fd7400052a0 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74000c330 con 0x7fd740003970 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd755e00640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd740004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd755e00640 1 -- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd740003970 msgr2=0x7fd740003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.244+0000 7fd75f400640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7400052a0 0x7fd74000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:31.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.245+0000 7fd75ea00640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd740003970 0x7fd740003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:31.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.245+0000 7fd755e00640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd740003970 0x7fd740003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:31.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.245+0000 7fd755e00640 1 -- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7400052a0 msgr2=0x7fd74000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:31.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.245+0000 7fd755e00640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7400052a0 0x7fd74000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:31.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.245+0000 7fd75f400640 1 -- 172.21.15.141:0/1756242768 reap_dead start 2024-01-16T15:03:31.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.245+0000 7fd755e00640 1 -- 172.21.15.141:0/1756242768 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd740003520 con 0x7fd7400047c0 2024-01-16T15:03:31.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.246+0000 7fd755e00640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd740004c40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd74800c210 tx=0x7fd7400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9269c91474b8843 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.246+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748005ad0 con 0x7fd7400047c0 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.246+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd748005c30 con 0x7fd7400047c0 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.246+0000 7fd75e000640 1 -- 172.21.15.141:0/1756242768 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74802cad0 con 0x7fd7400047c0 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.247+0000 7fd75fe00640 1 -- 172.21.15.141:0/1756242768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 msgr2=0x7fd740004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.247+0000 7fd75fe00640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd740004c40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd74800c210 tx=0x7fd7400057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.247+0000 7fd75fe00640 1 -- 172.21.15.141:0/1756242768 shutdown_connections 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.247+0000 7fd75fe00640 1 --2- 172.21.15.141:0/1756242768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd740004c40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.247+0000 7fd75fe00640 1 -- 172.21.15.141:0/1756242768 >> 172.21.15.141:0/1756242768 conn(0x7fd76006a800 msgr2=0x7fd76006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:31.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.247+0000 7fd75fe00640 1 -- 172.21.15.141:0/1756242768 shutdown_connections 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.247+0000 7fd75fe00640 1 -- 172.21.15.141:0/1756242768 wait complete. 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.248+0000 7fd75fe00640 1 Processor -- start 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.249+0000 7fd75fe00640 1 -- start start 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.249+0000 7fd75fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd7600aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.249+0000 7fd75fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd76013b8b0 0x7fd76015a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.249+0000 7fd75fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd76015d930 0x7fd76015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.249+0000 7fd75fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76015e2f0 con 0x7fd7400047c0 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600b0440 con 0x7fd76015d930 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76013be40 con 0x7fd76013b8b0 2024-01-16T15:03:31.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd7600aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd7600aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35900/0 (socket says 172.21.15.141:35900) 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 -- 172.21.15.141:0/2171715362 learned_addr learned my addr 172.21.15.141:0/2171715362 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 -- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd76013b8b0 msgr2=0x7fd76015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd76013b8b0 0x7fd76015a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 -- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd76015d930 msgr2=0x7fd76015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd76015d930 0x7fd76015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 -- 172.21.15.141:0/2171715362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76013c120 con 0x7fd7400047c0 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.250+0000 7fd75ea00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd7600aff00 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd750007e10 tx=0x7fd75000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.251+0000 7fd757e00640 1 -- 172.21.15.141:0/2171715362 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7500145f0 con 0x7fd7400047c0 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.251+0000 7fd757e00640 1 -- 172.21.15.141:0/2171715362 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd750014750 con 0x7fd7400047c0 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.251+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76013c400 con 0x7fd7400047c0 2024-01-16T15:03:31.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.251+0000 7fd757e00640 1 -- 172.21.15.141:0/2171715362 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750014ae0 con 0x7fd7400047c0 2024-01-16T15:03:31.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.251+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7601f0b20 con 0x7fd7400047c0 2024-01-16T15:03:31.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.252+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7601f0d40 con 0x7fd7400047c0 2024-01-16T15:03:31.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.253+0000 7fd757e00640 1 -- 172.21.15.141:0/2171715362 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fd750014c40 con 0x7fd7400047c0 2024-01-16T15:03:31.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.254+0000 7fd757e00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd73806ba10 0x7fd73806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:31.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.254+0000 7fd757e00640 1 -- 172.21.15.141:0/2171715362 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd75008bb80 con 0x7fd7400047c0 2024-01-16T15:03:31.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.254+0000 7fd755e00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd73806ba10 0x7fd73806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:31.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.256+0000 7fd755e00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd73806ba10 0x7fd73806ded0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd748034000 tx=0x7fd748039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:31.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.258+0000 7fd757e00640 1 -- 172.21.15.141:0/2171715362 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7601f0d40 con 0x7fd7400047c0 2024-01-16T15:03:31.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:31.481+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66723", "target": ["mon-mgr", ""]}) v1 -- 0x7fd76006cc40 con 0x7fd73806ba10 2024-01-16T15:03:33.809 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:33.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:34.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.393+0000 7fd757e00640 1 -- 172.21.15.141:0/2171715362 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd748001430 con 0x7fd73806ba10 2024-01-16T15:03:34.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd73806ba10 msgr2=0x7fd73806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd73806ba10 0x7fd73806ded0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd748034000 tx=0x7fd748039040 comp rx=0 tx=0).stop 2024-01-16T15:03:34.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 msgr2=0x7fd7600aff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd7600aff00 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd750007e10 tx=0x7fd75000f520 comp rx=0 tx=0).stop 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 shutdown_connections 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd73806ba10 0x7fd73806ded0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd76015d930 0x7fd76015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd76013b8b0 0x7fd76015a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 --2- 172.21.15.141:0/2171715362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7400047c0 0x7fd7600aff00 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.396+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 >> 172.21.15.141:0/2171715362 conn(0x7fd76006a800 msgr2=0x7fd76013fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.397+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 shutdown_connections 2024-01-16T15:03:34.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.397+0000 7fd75fe00640 1 -- 172.21.15.141:0/2171715362 wait complete. 2024-01-16T15:03:34.398 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66724 2024-01-16T15:03:34.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.630+0000 7f570b600640 1 Processor -- start 2024-01-16T15:03:34.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.630+0000 7f570b600640 1 -- start start 2024-01-16T15:03:34.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.632+0000 7f570b600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57040ce8f0 0x7f57040ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.632+0000 7f570b600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f57040b2a00 0x7f57040b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.632+0000 7f570b600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040d0de0 con 0x7f5704160f80 2024-01-16T15:03:34.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.632+0000 7f570b600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57040d10c0 con 0x7f57040b2a00 2024-01-16T15:03:34.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.632+0000 7f570b600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57040d13a0 con 0x7f57040ce8f0 2024-01-16T15:03:34.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.633+0000 7f5703e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57040ce8f0 0x7f57040ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.633+0000 7f5709600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f57040b2a00 0x7f57040b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.633+0000 7f5709600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f57040b2a00 0x7f57040b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:36226/0 (socket says 172.21.15.141:36226) 2024-01-16T15:03:34.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.633+0000 7f5709600640 1 -- 172.21.15.141:0/1243195120 learned_addr learned my addr 172.21.15.141:0/1243195120 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:34.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.633+0000 7f5709600640 1 -- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57040ce8f0 msgr2=0x7f57040ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:34.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.633+0000 7f5709600640 1 --2- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57040ce8f0 0x7f57040ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.633+0000 7f5709600640 1 -- 172.21.15.141:0/1243195120 >> v1:172.21.15.188:6789/0 conn(0x7f5704160f80 legacy=0x7f5704161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.634+0000 7f5709600640 1 -- 172.21.15.141:0/1243195120 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57040d1680 con 0x7f57040b2a00 2024-01-16T15:03:34.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.634+0000 7f5703e00640 1 --2- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57040ce8f0 0x7f57040ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:03:34.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.635+0000 7f5709600640 1 --2- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040b2a00 0x7f57040b2e60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009e50 tx=0x7f56fc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79d2a42200e29d08 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:34.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.635+0000 7f5703400640 1 -- 172.21.15.141:0/1243195120 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc02cb00 con 0x7f57040b2a00 2024-01-16T15:03:34.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.635+0000 7f5703400640 1 -- 172.21.15.141:0/1243195120 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc02cc60 con 0x7f57040b2a00 2024-01-16T15:03:34.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.635+0000 7f5703400640 1 -- 172.21.15.141:0/1243195120 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0343f0 con 0x7f57040b2a00 2024-01-16T15:03:34.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 -- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040b2a00 msgr2=0x7f57040b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 --2- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040b2a00 0x7f57040b2e60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009e50 tx=0x7f56fc02bf60 comp rx=0 tx=0).stop 2024-01-16T15:03:34.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 -- 172.21.15.141:0/1243195120 shutdown_connections 2024-01-16T15:03:34.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 --2- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040b2a00 0x7f57040b2e60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 --2- 172.21.15.141:0/1243195120 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57040ce8f0 0x7f57040ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 -- 172.21.15.141:0/1243195120 >> 172.21.15.141:0/1243195120 conn(0x7f570409c550 msgr2=0x7f57040d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:34.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 -- 172.21.15.141:0/1243195120 shutdown_connections 2024-01-16T15:03:34.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.636+0000 7f570b600640 1 -- 172.21.15.141:0/1243195120 wait complete. 2024-01-16T15:03:34.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.639+0000 7f570b600640 1 Processor -- start 2024-01-16T15:03:34.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.639+0000 7f570b600640 1 -- start start 2024-01-16T15:03:34.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f570b600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57040b2a00 0x7f57041f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f570b600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040ce8f0 0x7f57040b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f570b600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5704160f80 0x7f570415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f570b600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570415d7f0 con 0x7f57040b2a00 2024-01-16T15:03:34.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f570b600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57040b7830 con 0x7f57040ce8f0 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f570b600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570415f000 con 0x7f5704160f80 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f5708c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57040b2a00 0x7f57041f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f5708c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57040b2a00 0x7f57041f2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56492/0 (socket says 172.21.15.141:56492) 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.640+0000 7f5708c00640 1 -- 172.21.15.141:0/3624270267 learned_addr learned my addr 172.21.15.141:0/3624270267 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5703e00640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040ce8f0 0x7f57040b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5709600640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5704160f80 0x7f570415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5708c00640 1 -- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5704160f80 msgr2=0x7f570415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5708c00640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5704160f80 0x7f570415eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5708c00640 1 -- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040ce8f0 msgr2=0x7f57040b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5708c00640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040ce8f0 0x7f57040b72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5708c00640 1 -- 172.21.15.141:0/3624270267 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec009400 con 0x7f57040b2a00 2024-01-16T15:03:34.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5708c00640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57040b2a00 0x7f57041f2f80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c3f0 tx=0x7f56f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:34.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5700c00640 1 -- 172.21.15.141:0/3624270267 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f40052d0 con 0x7f57040b2a00 2024-01-16T15:03:34.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5700c00640 1 -- 172.21.15.141:0/3624270267 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f4005450 con 0x7f57040b2a00 2024-01-16T15:03:34.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.641+0000 7f5700c00640 1 -- 172.21.15.141:0/3624270267 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4007260 con 0x7f57040b2a00 2024-01-16T15:03:34.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.642+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc009b30 con 0x7f57040b2a00 2024-01-16T15:03:34.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.642+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57041f38b0 con 0x7f57040b2a00 2024-01-16T15:03:34.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.643+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57040675d0 con 0x7f57040b2a00 2024-01-16T15:03:34.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.644+0000 7f5700c00640 1 -- 172.21.15.141:0/3624270267 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f56f40073c0 con 0x7f57040b2a00 2024-01-16T15:03:34.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.645+0000 7f5700c00640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f56dc06ba10 0x7f56dc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.646+0000 7f5700c00640 1 -- 172.21.15.141:0/3624270267 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56f40878a0 con 0x7f57040b2a00 2024-01-16T15:03:34.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.646+0000 7f5703e00640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f56dc06ba10 0x7f56dc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.647+0000 7f5703e00640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f56dc06ba10 0x7f56dc06ded0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f57040b3550 tx=0x7f56ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:34.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.651+0000 7f5700c00640 1 -- 172.21.15.141:0/3624270267 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56f40875b0 con 0x7f57040b2a00 2024-01-16T15:03:34.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:34.882+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66724", "target": ["mon-mgr", ""]}) v1 -- 0x7f57040d1c40 con 0x7f56dc06ba10 2024-01-16T15:03:39.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.406+0000 7f5700c00640 1 -- 172.21.15.141:0/3624270267 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f56ec0098b0 con 0x7f56dc06ba10 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f56dc06ba10 msgr2=0x7f56dc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f56dc06ba10 0x7f56dc06ded0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f57040b3550 tx=0x7f56ec008040 comp rx=0 tx=0).stop 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57040b2a00 msgr2=0x7f57041f2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57040b2a00 0x7f57041f2f80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c3f0 tx=0x7f56f400c8b0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 shutdown_connections 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f56dc06ba10 0x7f56dc06ded0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5704160f80 0x7f570415eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57040ce8f0 0x7f57040b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 --2- 172.21.15.141:0/3624270267 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57040b2a00 0x7f57041f2f80 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 >> 172.21.15.141:0/3624270267 conn(0x7f570409c550 msgr2=0x7f57040d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 shutdown_connections 2024-01-16T15:03:39.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.409+0000 7f570b600640 1 -- 172.21.15.141:0/3624270267 wait complete. 2024-01-16T15:03:39.408 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66725 2024-01-16T15:03:39.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.644+0000 7f9971e00640 1 Processor -- start 2024-01-16T15:03:39.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.644+0000 7f9971e00640 1 -- start start 2024-01-16T15:03:39.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f9971e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c0cf9a0 0x7f996c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:39.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f9971e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f996c15e1e0 0x7f996c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:39.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f9971e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996c0b1ee0 con 0x7f996c143ae0 2024-01-16T15:03:39.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f9971e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c0b2780 con 0x7f996c15e1e0 2024-01-16T15:03:39.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f9971e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c15e990 con 0x7f996c0cf9a0 2024-01-16T15:03:39.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f996aa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f996c143ae0 0x7f996c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42966/0 (socket says 172.21.15.141:42966) 2024-01-16T15:03:39.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f996aa00640 1 -- 172.21.15.141:0/2390058472 learned_addr learned my addr 172.21.15.141:0/2390058472 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:39.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.646+0000 7f996b400640 1 --2- 172.21.15.141:0/2390058472 >> v2:172.21.15.141:3300/0 conn(0x7f996c15e1e0 0x7f996c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:39.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.647+0000 7f9961e00640 1 --2- 172.21.15.141:0/2390058472 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c0cf9a0 0x7f996c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:39.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.647+0000 7f996a000640 1 -- 172.21.15.141:0/2390058472 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685517274 0 0) 0x7f996c0b1ee0 con 0x7f996c143ae0 2024-01-16T15:03:39.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.647+0000 7f996a000640 1 -- 172.21.15.141:0/2390058472 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c005730 con 0x7f996c143ae0 2024-01-16T15:03:39.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.647+0000 7f996b400640 1 -- 172.21.15.141:0/2390058472 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c0cf9a0 msgr2=0x7f996c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:39.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.647+0000 7f996b400640 1 --2- 172.21.15.141:0/2390058472 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c0cf9a0 0x7f996c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.647+0000 7f996b400640 1 -- 172.21.15.141:0/2390058472 >> v1:172.21.15.141:6789/0 conn(0x7f996c143ae0 legacy=0x7f996c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:39.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.647+0000 7f996b400640 1 -- 172.21.15.141:0/2390058472 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c0b2130 con 0x7f996c15e1e0 2024-01-16T15:03:39.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.648+0000 7f996b400640 1 --2- 172.21.15.141:0/2390058472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c15e1e0 0x7f996c0702c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f9964009d90 tx=0x7f996402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d64551a9696a38 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:39.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.648+0000 7f996a000640 1 -- 172.21.15.141:0/2390058472 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996402cc50 con 0x7f996c15e1e0 2024-01-16T15:03:39.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.648+0000 7f996a000640 1 -- 172.21.15.141:0/2390058472 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996402cdb0 con 0x7f996c15e1e0 2024-01-16T15:03:39.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.648+0000 7f996a000640 1 -- 172.21.15.141:0/2390058472 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964034660 con 0x7f996c15e1e0 2024-01-16T15:03:39.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 -- 172.21.15.141:0/2390058472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c15e1e0 msgr2=0x7f996c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:39.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 --2- 172.21.15.141:0/2390058472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c15e1e0 0x7f996c0702c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f9964009d90 tx=0x7f996402bea0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 -- 172.21.15.141:0/2390058472 shutdown_connections 2024-01-16T15:03:39.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 --2- 172.21.15.141:0/2390058472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c15e1e0 0x7f996c0702c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 --2- 172.21.15.141:0/2390058472 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c0cf9a0 0x7f996c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 -- 172.21.15.141:0/2390058472 >> 172.21.15.141:0/2390058472 conn(0x7f996c06a800 msgr2=0x7f996c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:39.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 -- 172.21.15.141:0/2390058472 shutdown_connections 2024-01-16T15:03:39.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.649+0000 7f9971e00640 1 -- 172.21.15.141:0/2390058472 wait complete. 2024-01-16T15:03:39.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.652+0000 7f9971e00640 1 Processor -- start 2024-01-16T15:03:39.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.652+0000 7f9971e00640 1 -- start start 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9971e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c0cf9a0 0x7f996c0c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9971e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c143ae0 0x7f996c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9971e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c15e1e0 0x7f996c159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9971e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1f3690 con 0x7f996c143ae0 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9971e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c48b0 con 0x7f996c0cf9a0 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9971e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c3f80 con 0x7f996c15e1e0 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9961e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c143ae0 0x7f996c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9961e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c143ae0 0x7f996c0b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56508/0 (socket says 172.21.15.141:56508) 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.653+0000 7f9961e00640 1 -- 172.21.15.141:0/1097990240 learned_addr learned my addr 172.21.15.141:0/1097990240 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f996aa00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c0cf9a0 0x7f996c0c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:39.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f996b400640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c15e1e0 0x7f996c159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:39.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9961e00640 1 -- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c15e1e0 msgr2=0x7f996c159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:39.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9961e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c15e1e0 0x7f996c159560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9961e00640 1 -- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c0cf9a0 msgr2=0x7f996c0c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:39.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9961e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c0cf9a0 0x7f996c0c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:39.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9961e00640 1 -- 172.21.15.141:0/1097990240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9964009a70 con 0x7f996c143ae0 2024-01-16T15:03:39.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9961e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c143ae0 0x7f996c0b0e30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f995400d800 tx=0x7f995400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:39.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9963e00640 1 -- 172.21.15.141:0/1097990240 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995400b6c0 con 0x7f996c143ae0 2024-01-16T15:03:39.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9963e00640 1 -- 172.21.15.141:0/1097990240 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995400b820 con 0x7f996c143ae0 2024-01-16T15:03:39.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9963e00640 1 -- 172.21.15.141:0/1097990240 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995400bbb0 con 0x7f996c143ae0 2024-01-16T15:03:39.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.654+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c0c4260 con 0x7f996c143ae0 2024-01-16T15:03:39.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.655+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c0c62c0 con 0x7f996c143ae0 2024-01-16T15:03:39.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.656+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996c0c4a20 con 0x7f996c143ae0 2024-01-16T15:03:39.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.657+0000 7f9963e00640 1 -- 172.21.15.141:0/1097990240 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f995400bd10 con 0x7f996c143ae0 2024-01-16T15:03:39.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.658+0000 7f9963e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406ba10 0x7f994406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:39.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.658+0000 7f9963e00640 1 -- 172.21.15.141:0/1097990240 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f995408b290 con 0x7f996c143ae0 2024-01-16T15:03:39.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.659+0000 7f996aa00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406ba10 0x7f994406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:39.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.660+0000 7f996aa00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406ba10 0x7f994406ded0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f995c0011e0 tx=0x7f995c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:39.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.663+0000 7f9963e00640 1 -- 172.21.15.141:0/1097990240 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99540887e0 con 0x7f996c143ae0 2024-01-16T15:03:39.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:39.891+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f996c0c40f0 con 0x7f994406ba10 2024-01-16T15:03:44.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.398+0000 7f9963e00640 1 -- 172.21.15.141:0/1097990240 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f996c0c40f0 con 0x7f994406ba10 2024-01-16T15:03:44.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.402+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406ba10 msgr2=0x7f994406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.402+0000 7f9971e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406ba10 0x7f994406ded0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f995c0011e0 tx=0x7f995c009040 comp rx=0 tx=0).stop 2024-01-16T15:03:44.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.402+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c143ae0 msgr2=0x7f996c0b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.402+0000 7f9971e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c143ae0 0x7f996c0b0e30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f995400d800 tx=0x7f995400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 shutdown_connections 2024-01-16T15:03:44.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406ba10 0x7f994406ded0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c15e1e0 0x7f996c159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c143ae0 0x7f996c0b0e30 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 --2- 172.21.15.141:0/1097990240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c0cf9a0 0x7f996c0c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 >> 172.21.15.141:0/1097990240 conn(0x7f996c06a800 msgr2=0x7f996c0cb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:44.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 shutdown_connections 2024-01-16T15:03:44.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.403+0000 7f9971e00640 1 -- 172.21.15.141:0/1097990240 wait complete. 2024-01-16T15:03:44.406 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66726 2024-01-16T15:03:44.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.639+0000 7fc6e8a00640 1 Processor -- start 2024-01-16T15:03:44.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.639+0000 7fc6e8a00640 1 -- start start 2024-01-16T15:03:44.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.641+0000 7fc6e8a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc6e00ac8d0 0x7fc6e00aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.641+0000 7fc6e8a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e0071de0 con 0x7fc6e00d0b60 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.641+0000 7fc6e8a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e0072030 con 0x7fc6e006a3e0 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.641+0000 7fc6e8a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e00acd60 con 0x7fc6e00ac8d0 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.641+0000 7fc6e5600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc6e00d0b60 0x7fc6e006f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43028/0 (socket says 172.21.15.141:43028) 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.641+0000 7fc6e5600640 1 -- 172.21.15.141:0/492574007 learned_addr learned my addr 172.21.15.141:0/492574007 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/492574007 >> v2:172.21.15.188:3300/0 conn(0x7fc6e00ac8d0 0x7fc6e00aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224492795 0 0) 0x7fc6e0071de0 con 0x7fc6e00d0b60 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6bc003520 con 0x7fc6e00d0b60 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570834255 0 0) 0x7fc6bc003520 con 0x7fc6e00d0b60 2024-01-16T15:03:44.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 >> v2:172.21.15.188:3300/0 conn(0x7fc6e00ac8d0 msgr2=0x7fc6e00aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 --2- 172.21.15.141:0/492574007 >> v2:172.21.15.188:3300/0 conn(0x7fc6e00ac8d0 0x7fc6e00aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 >> v1:172.21.15.188:6789/0 conn(0x7fc6e006a3e0 legacy=0x7fc6e006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e0072310 con 0x7fc6e00d0b60 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fc6c4001670 con 0x7fc6e00d0b60 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 >> v1:172.21.15.141:6789/0 conn(0x7fc6e00d0b60 legacy=0x7fc6e006f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc003970 0x7fc6bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc0047c0 0x7fc6bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6bc005180 0x7fc6bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e00acd60 con 0x7fc6bc005180 2024-01-16T15:03:44.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc00c020 con 0x7fc6bc003970 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.642+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc00c1e0 con 0x7fc6bc0047c0 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6bc005180 0x7fc6bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e6000640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc003970 0x7fc6bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e5600640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc0047c0 0x7fc6bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e6a00640 1 -- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc0047c0 msgr2=0x7fc6bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc0047c0 0x7fc6bc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e6a00640 1 -- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc003970 msgr2=0x7fc6bc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc003970 0x7fc6bc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.643+0000 7fc6e6a00640 1 -- 172.21.15.141:0/492574007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc003520 con 0x7fc6bc005180 2024-01-16T15:03:44.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.644+0000 7fc6e6a00640 1 -- 172.21.15.141:0/492574007 reap_dead start 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.644+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6bc005180 0x7fc6bc00b980 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009ac0 tx=0x7fc6bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d01084f49a3f59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.644+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc004020 con 0x7fc6bc005180 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.644+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc004180 con 0x7fc6bc005180 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.644+0000 7fc6e4c00640 1 -- 172.21.15.141:0/492574007 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc02c710 con 0x7fc6bc005180 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.645+0000 7fc6e8a00640 1 -- 172.21.15.141:0/492574007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6bc005180 msgr2=0x7fc6bc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.645+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6bc005180 0x7fc6bc00b980 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009ac0 tx=0x7fc6bc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.645+0000 7fc6e8a00640 1 -- 172.21.15.141:0/492574007 shutdown_connections 2024-01-16T15:03:44.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.645+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/492574007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6bc005180 0x7fc6bc00b980 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.645+0000 7fc6e8a00640 1 -- 172.21.15.141:0/492574007 >> 172.21.15.141:0/492574007 conn(0x7fc6e00cfef0 msgr2=0x7fc6e00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:44.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.645+0000 7fc6e8a00640 1 -- 172.21.15.141:0/492574007 shutdown_connections 2024-01-16T15:03:44.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.645+0000 7fc6e8a00640 1 -- 172.21.15.141:0/492574007 wait complete. 2024-01-16T15:03:44.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.647+0000 7fc6e8a00640 1 Processor -- start 2024-01-16T15:03:44.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.647+0000 7fc6e8a00640 1 -- start start 2024-01-16T15:03:44.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.648+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc005180 0x7fc6e014fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6e01465a0 0x7fc6e00cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6e0150bb0 0x7fc6e00b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e8a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e0146fe0 con 0x7fc6e0150bb0 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e00cce80 con 0x7fc6e01465a0 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e015f9a0 con 0x7fc6bc005180 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6e0150bb0 0x7fc6e00b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6e0150bb0 0x7fc6e00b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44482/0 (socket says 172.21.15.141:44482) 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 -- 172.21.15.141:0/947608008 learned_addr learned my addr 172.21.15.141:0/947608008 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6000640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc005180 0x7fc6e014fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e5600640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6e01465a0 0x7fc6e00cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 -- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc005180 msgr2=0x7fc6e014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc005180 0x7fc6e014fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 -- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6e01465a0 msgr2=0x7fc6e00cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6e01465a0 0x7fc6e00cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e6a00640 1 -- 172.21.15.141:0/947608008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e0146a20 con 0x7fc6e0150bb0 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.649+0000 7fc6e5600640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6e01465a0 0x7fc6e00cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.650+0000 7fc6e6a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6e0150bb0 0x7fc6e00b08e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009bb0 tx=0x7fc6cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.650+0000 7fc6de000640 1 -- 172.21.15.141:0/947608008 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc03a070 con 0x7fc6e0150bb0 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.650+0000 7fc6de000640 1 -- 172.21.15.141:0/947608008 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc00f0c0 con 0x7fc6e0150bb0 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.650+0000 7fc6de000640 1 -- 172.21.15.141:0/947608008 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc003870 con 0x7fc6e0150bb0 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.650+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e00cd0b0 con 0x7fc6e0150bb0 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.650+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e015f350 con 0x7fc6e0150bb0 2024-01-16T15:03:44.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.651+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e015fb10 con 0x7fc6e0150bb0 2024-01-16T15:03:44.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.652+0000 7fc6de000640 1 -- 172.21.15.141:0/947608008 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fc6cc0039d0 con 0x7fc6e0150bb0 2024-01-16T15:03:44.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.653+0000 7fc6de000640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6b406ba40 0x7fc6b406df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:44.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.653+0000 7fc6de000640 1 -- 172.21.15.141:0/947608008 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6cc0abd60 con 0x7fc6e0150bb0 2024-01-16T15:03:44.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.654+0000 7fc6e6000640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6b406ba40 0x7fc6b406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:44.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.655+0000 7fc6e6000640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6b406ba40 0x7fc6b406df00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4007fd0 tx=0x7fc6d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:44.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.658+0000 7fc6de000640 1 -- 172.21.15.141:0/947608008 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6cc0a92b0 con 0x7fc6e0150bb0 2024-01-16T15:03:44.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:44.887+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66726", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6e006eb60 con 0x7fc6b406ba40 2024-01-16T15:03:45.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.075+0000 7fc6de000640 1 -- 172.21.15.141:0/947608008 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc6d4003e50 con 0x7fc6b406ba40 2024-01-16T15:03:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.078+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6b406ba40 msgr2=0x7fc6b406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.078+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6b406ba40 0x7fc6b406df00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4007fd0 tx=0x7fc6d4009040 comp rx=0 tx=0).stop 2024-01-16T15:03:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.078+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6e0150bb0 msgr2=0x7fc6e00b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.078+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6e0150bb0 0x7fc6e00b08e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009bb0 tx=0x7fc6cc004730 comp rx=0 tx=0).stop 2024-01-16T15:03:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 shutdown_connections 2024-01-16T15:03:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6b406ba40 0x7fc6b406df00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6e0150bb0 0x7fc6e00b08e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6e01465a0 0x7fc6e00cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 --2- 172.21.15.141:0/947608008 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6bc005180 0x7fc6e014fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 >> 172.21.15.141:0/947608008 conn(0x7fc6e00cfef0 msgr2=0x7fc6e00733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 shutdown_connections 2024-01-16T15:03:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.079+0000 7fc6e8a00640 1 -- 172.21.15.141:0/947608008 wait complete. 2024-01-16T15:03:45.078 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66727 2024-01-16T15:03:45.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.319+0000 7fab7ca00640 1 Processor -- start 2024-01-16T15:03:45.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.320+0000 7fab7ca00640 1 -- start start 2024-01-16T15:03:45.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab7ca00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fab78160920 0x7fab780b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab7ca00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fab780cf9f0 0x7fab780cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab7ca00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780d0a10 con 0x7fab78161010 2024-01-16T15:03:45.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab7ca00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab780d03e0 con 0x7fab78160920 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab7ca00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab780d1f50 con 0x7fab780cf9f0 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab76800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fab78160920 0x7fab780b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab77200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fab78161010 0x7fab78161480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43036/0 (socket says 172.21.15.141:43036) 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.321+0000 7fab77200640 1 -- 172.21.15.141:0/3663315068 learned_addr learned my addr 172.21.15.141:0/3663315068 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.322+0000 7fab75e00640 1 --2- 172.21.15.141:0/3663315068 >> v2:172.21.15.188:3300/0 conn(0x7fab780cf9f0 0x7fab780cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.322+0000 7fab76800640 1 -- 172.21.15.141:0/3663315068 >> v2:172.21.15.188:3300/0 conn(0x7fab780cf9f0 msgr2=0x7fab780cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.322+0000 7fab76800640 1 --2- 172.21.15.141:0/3663315068 >> v2:172.21.15.188:3300/0 conn(0x7fab780cf9f0 0x7fab780cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.322+0000 7fab76800640 1 -- 172.21.15.141:0/3663315068 >> v1:172.21.15.141:6789/0 conn(0x7fab78161010 legacy=0x7fab78161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.322+0000 7fab76800640 1 -- 172.21.15.141:0/3663315068 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab780cf080 con 0x7fab78160920 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.323+0000 7fab76800640 1 --2- 172.21.15.141:0/3663315068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78160920 0x7fab780b3f30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fab7808c1f0 tx=0x7fab6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b7052f7890903f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.323+0000 7fab75400640 1 -- 172.21.15.141:0/3663315068 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68031830 con 0x7fab78160920 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.323+0000 7fab75400640 1 -- 172.21.15.141:0/3663315068 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab68031990 con 0x7fab78160920 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.323+0000 7fab75400640 1 -- 172.21.15.141:0/3663315068 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68031d20 con 0x7fab78160920 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 -- 172.21.15.141:0/3663315068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78160920 msgr2=0x7fab780b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 --2- 172.21.15.141:0/3663315068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78160920 0x7fab780b3f30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fab7808c1f0 tx=0x7fab6802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 -- 172.21.15.141:0/3663315068 shutdown_connections 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 --2- 172.21.15.141:0/3663315068 >> v2:172.21.15.188:3300/0 conn(0x7fab780cf9f0 0x7fab780cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 --2- 172.21.15.141:0/3663315068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78160920 0x7fab780b3f30 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 -- 172.21.15.141:0/3663315068 >> 172.21.15.141:0/3663315068 conn(0x7fab780d3090 msgr2=0x7fab780d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:45.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 -- 172.21.15.141:0/3663315068 shutdown_connections 2024-01-16T15:03:45.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.324+0000 7fab7ca00640 1 -- 172.21.15.141:0/3663315068 wait complete. 2024-01-16T15:03:45.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.326+0000 7fab7ca00640 1 Processor -- start 2024-01-16T15:03:45.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.326+0000 7fab7ca00640 1 -- start start 2024-01-16T15:03:45.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab7ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab780cf9f0 0x7fab7814d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab7ca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab78160920 0x7fab781597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab7ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78161010 0x7fab78159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab7ca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab780b7800 con 0x7fab78161010 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab7ca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7814d9d0 con 0x7fab78160920 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab7ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab780b6b80 con 0x7fab780cf9f0 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab77200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78161010 0x7fab78159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab77200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78161010 0x7fab78159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44502/0 (socket says 172.21.15.141:44502) 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab77200640 1 -- 172.21.15.141:0/1759566871 learned_addr learned my addr 172.21.15.141:0/1759566871 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab76800640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab780cf9f0 0x7fab7814d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab75e00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab78160920 0x7fab781597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.327+0000 7fab77200640 1 -- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab780cf9f0 msgr2=0x7fab7814d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab77200640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab780cf9f0 0x7fab7814d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab77200640 1 -- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab78160920 msgr2=0x7fab781597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab77200640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab78160920 0x7fab781597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab77200640 1 -- 172.21.15.141:0/1759566871 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68009400 con 0x7fab78161010 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab77200640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78161010 0x7fab78159cf0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fab7000c420 tx=0x7fab7000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab6ea00640 1 -- 172.21.15.141:0/1759566871 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70005ce0 con 0x7fab78161010 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab6ea00640 1 -- 172.21.15.141:0/1759566871 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab70005e40 con 0x7fab78161010 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab780b6e60 con 0x7fab78161010 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab6ea00640 1 -- 172.21.15.141:0/1759566871 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab700072d0 con 0x7fab78161010 2024-01-16T15:03:45.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.328+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab780b7420 con 0x7fab78161010 2024-01-16T15:03:45.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.330+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab781f3060 con 0x7fab78161010 2024-01-16T15:03:45.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.330+0000 7fab6ea00640 1 -- 172.21.15.141:0/1759566871 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fab700074f0 con 0x7fab78161010 2024-01-16T15:03:45.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.331+0000 7fab6ea00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab5006ba10 0x7fab5006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.331+0000 7fab6ea00640 1 -- 172.21.15.141:0/1759566871 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab70087990 con 0x7fab78161010 2024-01-16T15:03:45.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.332+0000 7fab76800640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab5006ba10 0x7fab5006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.333+0000 7fab76800640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab5006ba10 0x7fab5006ded0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fab68036000 tx=0x7fab68008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.336+0000 7fab6ea00640 1 -- 172.21.15.141:0/1759566871 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab781f3060 con 0x7fab78161010 2024-01-16T15:03:45.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.563+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66727", "target": ["mon-mgr", ""]}) v1 -- 0x7fab780ceac0 con 0x7fab5006ba10 2024-01-16T15:03:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.751+0000 7fab6ea00640 1 -- 172.21.15.141:0/1759566871 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fab68031b00 con 0x7fab5006ba10 2024-01-16T15:03:45.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab5006ba10 msgr2=0x7fab5006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab5006ba10 0x7fab5006ded0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fab68036000 tx=0x7fab68008030 comp rx=0 tx=0).stop 2024-01-16T15:03:45.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78161010 msgr2=0x7fab78159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78161010 0x7fab78159cf0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fab7000c420 tx=0x7fab7000c8e0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 shutdown_connections 2024-01-16T15:03:45.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fab5006ba10 0x7fab5006ded0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fab78161010 0x7fab78159cf0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab78160920 0x7fab781597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 --2- 172.21.15.141:0/1759566871 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fab780cf9f0 0x7fab7814d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.754+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 >> 172.21.15.141:0/1759566871 conn(0x7fab780d3090 msgr2=0x7fab7806c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:45.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.755+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 shutdown_connections 2024-01-16T15:03:45.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.755+0000 7fab7ca00640 1 -- 172.21.15.141:0/1759566871 wait complete. 2024-01-16T15:03:45.753 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:03:45.753 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66721 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.980+0000 7f1337e00640 1 Processor -- start 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.980+0000 7f1337e00640 1 -- start start 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.981+0000 7f1337e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f133815d4b0 0x7f13380d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.981+0000 7f1337e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f13380cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.981+0000 7f1337e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13380d0320 con 0x7f13380cf9a0 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.981+0000 7f1337e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13380b2730 con 0x7f133815d4b0 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.981+0000 7f1337e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13380d1ec0 con 0x7f13380b41e0 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1336a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f133815d4b0 0x7f13380d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f132de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f13380cf9a0 0x7f13380cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43040/0 (socket says 172.21.15.141:43040) 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f132de00640 1 -- 172.21.15.141:0/212912922 learned_addr learned my addr 172.21.15.141:0/212912922 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1337400640 1 --2- 172.21.15.141:0/212912922 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f13380cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1336000640 1 -- 172.21.15.141:0/212912922 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161624754 0 0) 0x7f13380d0320 con 0x7f13380cf9a0 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1336000640 1 -- 172.21.15.141:0/212912922 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1318003520 con 0x7f13380cf9a0 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1336a00640 1 -- 172.21.15.141:0/212912922 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 msgr2=0x7f13380cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1336a00640 1 --2- 172.21.15.141:0/212912922 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f13380cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1336a00640 1 -- 172.21.15.141:0/212912922 >> v1:172.21.15.141:6789/0 conn(0x7f13380cf9a0 legacy=0x7f13380cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.982+0000 7f1336a00640 1 -- 172.21.15.141:0/212912922 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13380b29d0 con 0x7f133815d4b0 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.983+0000 7f1336a00640 1 --2- 172.21.15.141:0/212912922 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133815d4b0 0x7f13380d1550 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f13280097a0 tx=0x7f132802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af36dc32388bfd5e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.984+0000 7f1336000640 1 -- 172.21.15.141:0/212912922 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328031830 con 0x7f133815d4b0 2024-01-16T15:03:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.984+0000 7f1336000640 1 -- 172.21.15.141:0/212912922 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1328031990 con 0x7f133815d4b0 2024-01-16T15:03:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.984+0000 7f1336000640 1 -- 172.21.15.141:0/212912922 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328031d20 con 0x7f133815d4b0 2024-01-16T15:03:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.984+0000 7f1337e00640 1 -- 172.21.15.141:0/212912922 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133815d4b0 msgr2=0x7f13380d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.984+0000 7f1337e00640 1 --2- 172.21.15.141:0/212912922 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133815d4b0 0x7f13380d1550 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f13280097a0 tx=0x7f132802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.985+0000 7f1337e00640 1 -- 172.21.15.141:0/212912922 shutdown_connections 2024-01-16T15:03:45.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.985+0000 7f1337e00640 1 --2- 172.21.15.141:0/212912922 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f13380cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.985+0000 7f1337e00640 1 --2- 172.21.15.141:0/212912922 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f133815d4b0 0x7f13380d1550 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.985+0000 7f1337e00640 1 -- 172.21.15.141:0/212912922 >> 172.21.15.141:0/212912922 conn(0x7f13380d3000 msgr2=0x7f13380d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:45.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.985+0000 7f1337e00640 1 -- 172.21.15.141:0/212912922 shutdown_connections 2024-01-16T15:03:45.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.985+0000 7f1337e00640 1 -- 172.21.15.141:0/212912922 wait complete. 2024-01-16T15:03:45.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.987+0000 7f1337e00640 1 Processor -- start 2024-01-16T15:03:45.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.987+0000 7f1337e00640 1 -- start start 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f1337e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f1338155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f1337e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13380cf9a0 0x7f13380cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f1337e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133815d4b0 0x7f133809e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f1337e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13380cb130 con 0x7f13380cf9a0 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f1337e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13380ca4b0 con 0x7f133815d4b0 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f1337e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13380ca790 con 0x7f13380b41e0 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f132de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13380cf9a0 0x7f13380cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f132de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13380cf9a0 0x7f13380cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44514/0 (socket says 172.21.15.141:44514) 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.988+0000 7f132de00640 1 -- 172.21.15.141:0/1060135473 learned_addr learned my addr 172.21.15.141:0/1060135473 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f1336a00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f1338155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f1337400640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133815d4b0 0x7f133809e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f132de00640 1 -- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 msgr2=0x7f1338155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f132de00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f1338155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f132de00640 1 -- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133815d4b0 msgr2=0x7f133809e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f132de00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133815d4b0 0x7f133809e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f132de00640 1 -- 172.21.15.141:0/1060135473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328009400 con 0x7f13380cf9a0 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f132de00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13380cf9a0 0x7f13380cb550 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f132000c500 tx=0x7f132000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.989+0000 7f132f400640 1 -- 172.21.15.141:0/1060135473 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1320005ce0 con 0x7f13380cf9a0 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.990+0000 7f132f400640 1 -- 172.21.15.141:0/1060135473 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1320005e40 con 0x7f13380cf9a0 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.990+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133815da10 con 0x7f13380cf9a0 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.990+0000 7f132f400640 1 -- 172.21.15.141:0/1060135473 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1320007330 con 0x7f13380cf9a0 2024-01-16T15:03:45.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.990+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13380cad50 con 0x7f13380cf9a0 2024-01-16T15:03:45.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.991+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338067450 con 0x7f13380cf9a0 2024-01-16T15:03:45.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.992+0000 7f132f400640 1 -- 172.21.15.141:0/1060135473 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f1320007490 con 0x7f13380cf9a0 2024-01-16T15:03:45.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.992+0000 7f132f400640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f131006ba10 0x7f131006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.993+0000 7f132f400640 1 -- 172.21.15.141:0/1060135473 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f132008a330 con 0x7f13380cf9a0 2024-01-16T15:03:45.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.993+0000 7f1336a00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f131006ba10 0x7f131006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.994+0000 7f1336a00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f131006ba10 0x7f131006ded0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1328036000 tx=0x7f132803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:45.997+0000 7f132f400640 1 -- 172.21.15.141:0/1060135473 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1320087880 con 0x7f13380cf9a0 2024-01-16T15:03:46.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.222+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66721", "target": ["mon-mgr", ""]}) v1 -- 0x7f133815d890 con 0x7f131006ba10 2024-01-16T15:03:46.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.248+0000 7f132f400640 1 -- 172.21.15.141:0/1060135473 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f1328031b00 con 0x7f131006ba10 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 0.0, 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "0/3", 2024-01-16T15:03:46.228 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:03:46.229 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:03:46.229 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:46.229 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:46.229 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:03:46.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f131006ba10 msgr2=0x7f131006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f131006ba10 0x7f131006ded0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1328036000 tx=0x7f132803c040 comp rx=0 tx=0).stop 2024-01-16T15:03:46.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13380cf9a0 msgr2=0x7f13380cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13380cf9a0 0x7f13380cb550 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f132000c500 tx=0x7f132000c9c0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 shutdown_connections 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f131006ba10 0x7f131006ded0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133815d4b0 0x7f133809e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13380cf9a0 0x7f13380cb550 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 --2- 172.21.15.141:0/1060135473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13380b41e0 0x7f1338155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 >> 172.21.15.141:0/1060135473 conn(0x7f13380d3000 msgr2=0x7f133806c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 shutdown_connections 2024-01-16T15:03:46.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.251+0000 7f1337e00640 1 -- 172.21.15.141:0/1060135473 wait complete. 2024-01-16T15:03:46.250 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:03:46.250 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66727 2024-01-16T15:03:46.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.480+0000 7f9306800640 1 Processor -- start 2024-01-16T15:03:46.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.480+0000 7f9306800640 1 -- start start 2024-01-16T15:03:46.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.482+0000 7f9306800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f93000cf9a0 0x7f930015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.482+0000 7f9306800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93000b1ee0 con 0x7f930015e1e0 2024-01-16T15:03:46.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.482+0000 7f9306800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93000b2780 con 0x7f9300143ae0 2024-01-16T15:03:46.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.482+0000 7f9306800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930015e990 con 0x7f93000cf9a0 2024-01-16T15:03:46.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.482+0000 7f92f6800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f93000cf9a0 0x7f930015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.482+0000 7f92ffe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f930015e1e0 0x7f93000702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43044/0 (socket says 172.21.15.141:43044) 2024-01-16T15:03:46.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.482+0000 7f92ffe00640 1 -- 172.21.15.141:0/3099799391 learned_addr learned my addr 172.21.15.141:0/3099799391 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:46.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.483+0000 7f92f6800640 1 -- 172.21.15.141:0/3099799391 >> v1:172.21.15.188:6789/0 conn(0x7f9300143ae0 legacy=0x7f93000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:46.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.483+0000 7f92f6800640 1 -- 172.21.15.141:0/3099799391 >> v1:172.21.15.141:6789/0 conn(0x7f930015e1e0 legacy=0x7f93000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.483+0000 7f92f6800640 1 -- 172.21.15.141:0/3099799391 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93000b2130 con 0x7f93000cf9a0 2024-01-16T15:03:46.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.483+0000 7f92fea00640 1 -- 172.21.15.141:0/3099799391 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104749620 0 0) 0x7f93000b1ee0 con 0x7f930015e1e0 2024-01-16T15:03:46.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.484+0000 7f92f6800640 1 --2- 172.21.15.141:0/3099799391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f930015dca0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f92e80097a0 tx=0x7f92e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3731a9875e1c1395 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:46.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.484+0000 7f92fea00640 1 -- 172.21.15.141:0/3099799391 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8031830 con 0x7f93000cf9a0 2024-01-16T15:03:46.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.484+0000 7f92fea00640 1 -- 172.21.15.141:0/3099799391 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e8031990 con 0x7f93000cf9a0 2024-01-16T15:03:46.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.484+0000 7f92fea00640 1 -- 172.21.15.141:0/3099799391 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8031d20 con 0x7f93000cf9a0 2024-01-16T15:03:46.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.485+0000 7f9306800640 1 -- 172.21.15.141:0/3099799391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 msgr2=0x7f930015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.485+0000 7f9306800640 1 --2- 172.21.15.141:0/3099799391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f930015dca0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f92e80097a0 tx=0x7f92e802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.485+0000 7f9306800640 1 -- 172.21.15.141:0/3099799391 shutdown_connections 2024-01-16T15:03:46.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.485+0000 7f9306800640 1 --2- 172.21.15.141:0/3099799391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f930015dca0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.485+0000 7f9306800640 1 -- 172.21.15.141:0/3099799391 >> 172.21.15.141:0/3099799391 conn(0x7f930006a800 msgr2=0x7f930006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:46.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.485+0000 7f9306800640 1 -- 172.21.15.141:0/3099799391 shutdown_connections 2024-01-16T15:03:46.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.485+0000 7f9306800640 1 -- 172.21.15.141:0/3099799391 wait complete. 2024-01-16T15:03:46.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.487+0000 7f9306800640 1 Processor -- start 2024-01-16T15:03:46.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.487+0000 7f9306800640 1 -- start start 2024-01-16T15:03:46.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f9306800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f93000ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f9306800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9300143ae0 0x7f93001f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f9306800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f930015e1e0 0x7f930015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f9306800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93001f42e0 con 0x7f93000cf9a0 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f9306800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93001f3a40 con 0x7f930015e1e0 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f9306800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93001f3d20 con 0x7f9300143ae0 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f92ff400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f93000ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f92ff400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f93000ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44524/0 (socket says 172.21.15.141:44524) 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.488+0000 7f92ff400640 1 -- 172.21.15.141:0/4290775989 learned_addr learned my addr 172.21.15.141:0/4290775989 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92ff400640 1 -- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9300143ae0 msgr2=0x7f93001f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92ff400640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9300143ae0 0x7f93001f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92ff400640 1 -- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f930015e1e0 msgr2=0x7f930015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92ffe00640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f930015e1e0 0x7f930015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92ff400640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f930015e1e0 0x7f930015a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92ff400640 1 -- 172.21.15.141:0/4290775989 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8009400 con 0x7f93000cf9a0 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92ff400640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f93000ae070 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c7c0 tx=0x7f92f000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92f7e00640 1 -- 172.21.15.141:0/4290775989 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f000aa50 con 0x7f93000cf9a0 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92f7e00640 1 -- 172.21.15.141:0/4290775989 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f000abb0 con 0x7f93000cf9a0 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f92f7e00640 1 -- 172.21.15.141:0/4290775989 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0012420 con 0x7f93000cf9a0 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93001f4000 con 0x7f93000cf9a0 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.489+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93001f36c0 con 0x7f93000cf9a0 2024-01-16T15:03:46.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.491+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93000cca50 con 0x7f93000cf9a0 2024-01-16T15:03:46.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.491+0000 7f92f7e00640 1 -- 172.21.15.141:0/4290775989 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f92f0012580 con 0x7f93000cf9a0 2024-01-16T15:03:46.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.492+0000 7f92f7e00640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92d806ba10 0x7f92d806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:46.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.492+0000 7f92f7e00640 1 -- 172.21.15.141:0/4290775989 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92f008ac80 con 0x7f93000cf9a0 2024-01-16T15:03:46.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.493+0000 7f92f6800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92d806ba10 0x7f92d806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:46.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.494+0000 7f92f6800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92d806ba10 0x7f92d806ded0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f92e8036000 tx=0x7f92e8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:46.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.497+0000 7f92f7e00640 1 -- 172.21.15.141:0/4290775989 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93000cca50 con 0x7f93000cf9a0 2024-01-16T15:03:46.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.747+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66727", "target": ["mon-mgr", ""]}) v1 -- 0x7f930015e670 con 0x7f92d806ba10 2024-01-16T15:03:46.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.785+0000 7f92f7e00640 1 -- 172.21.15.141:0/4290775989 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f92e8031b00 con 0x7f92d806ba10 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:46.765 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:03:46.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.792+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92d806ba10 msgr2=0x7f92d806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.792+0000 7f9306800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92d806ba10 0x7f92d806ded0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f92e8036000 tx=0x7f92e8030000 comp rx=0 tx=0).stop 2024-01-16T15:03:46.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.792+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 msgr2=0x7f93000ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:46.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.792+0000 7f9306800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f93000ae070 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c7c0 tx=0x7f92f000cc80 comp rx=0 tx=0).stop 2024-01-16T15:03:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.795+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 shutdown_connections 2024-01-16T15:03:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.795+0000 7f9306800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92d806ba10 0x7f92d806ded0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.795+0000 7f9306800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f930015e1e0 0x7f930015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.795+0000 7f9306800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9300143ae0 0x7f93001f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.795+0000 7f9306800640 1 --2- 172.21.15.141:0/4290775989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93000cf9a0 0x7f93000ae070 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:46.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.795+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 >> 172.21.15.141:0/4290775989 conn(0x7f930006a800 msgr2=0x7f930006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:46.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.796+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 shutdown_connections 2024-01-16T15:03:46.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:46.796+0000 7f9306800640 1 -- 172.21.15.141:0/4290775989 wait complete. 2024-01-16T15:03:46.813 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:47.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.135+0000 7fe930a00640 1 Processor -- start 2024-01-16T15:03:47.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.135+0000 7fe930a00640 1 -- start start 2024-01-16T15:03:47.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.137+0000 7fe930a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe92814b2c0 0x7fe9280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:47.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.137+0000 7fe930a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9280cf7d0 0x7fe928070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:47.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.137+0000 7fe930a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe928070ea0 0x7fe9280b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:47.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.137+0000 7fe930a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92814b6a0 con 0x7fe92814b2c0 2024-01-16T15:03:47.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.137+0000 7fe930a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9280cfbf0 con 0x7fe928070ea0 2024-01-16T15:03:47.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.137+0000 7fe930a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe928071690 con 0x7fe9280cf7d0 2024-01-16T15:03:47.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.137+0000 7fe92e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe92814b2c0 0x7fe9280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:47.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.138+0000 7fe92e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe92814b2c0 0x7fe9280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44540/0 (socket says 172.21.15.141:44540) 2024-01-16T15:03:47.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.138+0000 7fe92e000640 1 -- 172.21.15.141:0/1238757310 learned_addr learned my addr 172.21.15.141:0/1238757310 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:47.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.138+0000 7fe92e000640 1 -- 172.21.15.141:0/1238757310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9280cf7d0 msgr2=0x7fe928070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:47.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.138+0000 7fe92e000640 1 --2- 172.21.15.141:0/1238757310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9280cf7d0 0x7fe928070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.138+0000 7fe92e000640 1 -- 172.21.15.141:0/1238757310 >> v2:172.21.15.188:3300/0 conn(0x7fe928070ea0 msgr2=0x7fe9280b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:47.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.138+0000 7fe92e000640 1 --2- 172.21.15.141:0/1238757310 >> v2:172.21.15.188:3300/0 conn(0x7fe928070ea0 0x7fe9280b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.139+0000 7fe92e000640 1 -- 172.21.15.141:0/1238757310 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928071970 con 0x7fe92814b2c0 2024-01-16T15:03:47.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.140+0000 7fe92e000640 1 --2- 172.21.15.141:0/1238757310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe92814b2c0 0x7fe9280cf290 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe92806e2a0 tx=0x7fe91802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c4fe4f43153f06 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:47.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.140+0000 7fe92cc00640 1 -- 172.21.15.141:0/1238757310 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918031870 con 0x7fe92814b2c0 2024-01-16T15:03:47.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.140+0000 7fe92cc00640 1 -- 172.21.15.141:0/1238757310 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9180319d0 con 0x7fe92814b2c0 2024-01-16T15:03:47.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.140+0000 7fe92cc00640 1 -- 172.21.15.141:0/1238757310 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918031d60 con 0x7fe92814b2c0 2024-01-16T15:03:47.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.141+0000 7fe930a00640 1 -- 172.21.15.141:0/1238757310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe92814b2c0 msgr2=0x7fe9280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:47.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.141+0000 7fe930a00640 1 --2- 172.21.15.141:0/1238757310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe92814b2c0 0x7fe9280cf290 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe92806e2a0 tx=0x7fe91802b910 comp rx=0 tx=0).stop 2024-01-16T15:03:47.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.141+0000 7fe930a00640 1 -- 172.21.15.141:0/1238757310 shutdown_connections 2024-01-16T15:03:47.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.141+0000 7fe930a00640 1 --2- 172.21.15.141:0/1238757310 >> v2:172.21.15.188:3300/0 conn(0x7fe928070ea0 0x7fe9280b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.141+0000 7fe930a00640 1 --2- 172.21.15.141:0/1238757310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9280cf7d0 0x7fe928070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.141+0000 7fe930a00640 1 --2- 172.21.15.141:0/1238757310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe92814b2c0 0x7fe9280cf290 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.141+0000 7fe930a00640 1 -- 172.21.15.141:0/1238757310 >> 172.21.15.141:0/1238757310 conn(0x7fe92806a800 msgr2=0x7fe92806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:47.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.142+0000 7fe930a00640 1 -- 172.21.15.141:0/1238757310 shutdown_connections 2024-01-16T15:03:47.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.142+0000 7fe930a00640 1 -- 172.21.15.141:0/1238757310 wait complete. 2024-01-16T15:03:47.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.144+0000 7fe930a00640 1 Processor -- start 2024-01-16T15:03:47.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.144+0000 7fe930a00640 1 -- start start 2024-01-16T15:03:47.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.146+0000 7fe930a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe928070ea0 0x7fe92815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:47.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.146+0000 7fe930a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9280cf7d0 0x7fe928164740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:47.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.146+0000 7fe930a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe92814b2c0 0x7fe9280b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:47.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.146+0000 7fe930a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928164c80 con 0x7fe928070ea0 2024-01-16T15:03:47.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.146+0000 7fe930a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92815dc90 con 0x7fe9280cf7d0 2024-01-16T15:03:47.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.146+0000 7fe930a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92815d080 con 0x7fe92814b2c0 2024-01-16T15:03:47.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.147+0000 7fe92e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe928070ea0 0x7fe92815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:47.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.147+0000 7fe92e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe928070ea0 0x7fe92815ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44548/0 (socket says 172.21.15.141:44548) 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.147+0000 7fe92e000640 1 -- 172.21.15.141:0/2168792750 learned_addr learned my addr 172.21.15.141:0/2168792750 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.147+0000 7fe92ea00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe92814b2c0 0x7fe9280b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.147+0000 7fe92e000640 1 -- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe92814b2c0 msgr2=0x7fe9280b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.147+0000 7fe92e000640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe92814b2c0 0x7fe9280b1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe92e000640 1 -- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9280cf7d0 msgr2=0x7fe928164740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe92e000640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9280cf7d0 0x7fe928164740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe92e000640 1 -- 172.21.15.141:0/2168792750 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9180094d0 con 0x7fe928070ea0 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe92ea00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe92814b2c0 0x7fe9280b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe92e000640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe928070ea0 0x7fe92815ca10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe918000c00 tx=0x7fe9180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe926000640 1 -- 172.21.15.141:0/2168792750 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91803ad10 con 0x7fe928070ea0 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe926000640 1 -- 172.21.15.141:0/2168792750 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91803ae70 con 0x7fe928070ea0 2024-01-16T15:03:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.148+0000 7fe926000640 1 -- 172.21.15.141:0/2168792750 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918044610 con 0x7fe928070ea0 2024-01-16T15:03:47.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.149+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92815d360 con 0x7fe928070ea0 2024-01-16T15:03:47.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.149+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92815d920 con 0x7fe928070ea0 2024-01-16T15:03:47.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.150+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f8005250 con 0x7fe928070ea0 2024-01-16T15:03:47.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.151+0000 7fe926000640 1 -- 172.21.15.141:0/2168792750 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fe91804d450 con 0x7fe928070ea0 2024-01-16T15:03:47.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.152+0000 7fe926000640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe90006ba10 0x7fe90006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:47.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.153+0000 7fe926000640 1 -- 172.21.15.141:0/2168792750 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe91802d040 con 0x7fe928070ea0 2024-01-16T15:03:47.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.154+0000 7fe92d600640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe90006ba10 0x7fe90006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:47.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.155+0000 7fe92d600640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe90006ba10 0x7fe90006ded0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe9281618d0 tx=0x7fe910006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:47.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.157+0000 7fe926000640 1 -- 172.21.15.141:0/2168792750 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9180aa970 con 0x7fe928070ea0 2024-01-16T15:03:47.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.532+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe8f80057d0 con 0x7fe928070ea0 2024-01-16T15:03:47.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.557+0000 7fe926000640 1 -- 172.21.15.141:0/2168792750 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7fe9180773e0 con 0x7fe928070ea0 2024-01-16T15:03:47.537 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:03:47.538 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 420, 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:03:47.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:03:47.540 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 804, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222280523, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6752432128, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766341681152, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 938, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 25334, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 7 2024-01-16T15:03:47.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:03:47.542 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:03:47.543 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:03:47.544 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.563+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe90006ba10 msgr2=0x7fe90006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.563+0000 7fe930a00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe90006ba10 0x7fe90006ded0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe9281618d0 tx=0x7fe910006c90 comp rx=0 tx=0).stop 2024-01-16T15:03:47.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.563+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe928070ea0 msgr2=0x7fe92815ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:47.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.563+0000 7fe930a00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe928070ea0 0x7fe92815ca10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe918000c00 tx=0x7fe9180046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.564+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 shutdown_connections 2024-01-16T15:03:47.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.564+0000 7fe930a00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe90006ba10 0x7fe90006ded0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.564+0000 7fe930a00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe92814b2c0 0x7fe9280b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.564+0000 7fe930a00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9280cf7d0 0x7fe928164740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.564+0000 7fe930a00640 1 --2- 172.21.15.141:0/2168792750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe928070ea0 0x7fe92815ca10 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:47.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.564+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 >> 172.21.15.141:0/2168792750 conn(0x7fe92806a800 msgr2=0x7fe9280afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:47.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.567+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 shutdown_connections 2024-01-16T15:03:47.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:47.567+0000 7fe930a00640 1 -- 172.21.15.141:0/2168792750 wait complete. 2024-01-16T15:03:48.579 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.933+0000 7fa81fc00640 1 Processor -- start 2024-01-16T15:03:48.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.933+0000 7fa81fc00640 1 -- start start 2024-01-16T15:03:48.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.935+0000 7fa81fc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa818143ae0 0x7fa8180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.935+0000 7fa81fc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8180cfc30 con 0x7fa8180cf7d0 2024-01-16T15:03:48.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.935+0000 7fa81fc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8180b1ee0 con 0x7fa81815e3b0 2024-01-16T15:03:48.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.935+0000 7fa81fc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8180b2780 con 0x7fa818143ae0 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.935+0000 7fa80fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa8180cf7d0 0x7fa81815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43058/0 (socket says 172.21.15.141:43058) 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.935+0000 7fa80fe00640 1 -- 172.21.15.141:0/1436003339 learned_addr learned my addr 172.21.15.141:0/1436003339 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563790738 0 0) 0x7fa8180cfc30 con 0x7fa8180cf7d0 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f8003520 con 0x7fa8180cf7d0 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3818740613 0 0) 0x7fa7f8003520 con 0x7fa8180cf7d0 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 >> v2:172.21.15.188:3300/0 conn(0x7fa818143ae0 msgr2=0x7fa8180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 --2- 172.21.15.141:0/1436003339 >> v2:172.21.15.188:3300/0 conn(0x7fa818143ae0 0x7fa8180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 >> v1:172.21.15.188:6789/0 conn(0x7fa81815e3b0 legacy=0x7fa81806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81815e990 con 0x7fa8180cf7d0 2024-01-16T15:03:48.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa800001670 con 0x7fa8180cf7d0 2024-01-16T15:03:48.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 >> v1:172.21.15.141:6789/0 conn(0x7fa8180cf7d0 legacy=0x7fa81815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:48.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7f80039c0 0x7fa7f8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7f80047c0 0x7fa7f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80052a0 0x7fa7f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8180b2780 con 0x7fa7f80047c0 2024-01-16T15:03:48.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f8010600 con 0x7fa7f80039c0 2024-01-16T15:03:48.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.936+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80107c0 con 0x7fa7f80052a0 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.937+0000 7fa80fe00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7f80047c0 0x7fa7f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.937+0000 7fa80fe00640 1 -- 172.21.15.141:0/1436003339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80052a0 msgr2=0x7fa7f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.937+0000 7fa80fe00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80052a0 0x7fa7f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.937+0000 7fa80fe00640 1 -- 172.21.15.141:0/1436003339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7f80039c0 msgr2=0x7fa7f8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.937+0000 7fa80fe00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7f80039c0 0x7fa7f8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.937+0000 7fa80fe00640 1 -- 172.21.15.141:0/1436003339 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8003520 con 0x7fa7f80047c0 2024-01-16T15:03:48.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.937+0000 7fa81dc00640 1 -- 172.21.15.141:0/1436003339 reap_dead start 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.939+0000 7fa80fe00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7f80047c0 0x7fa7f8004c40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa800007fd0 tx=0x7fa7f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7263e742306c64c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.939+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800005ad0 con 0x7fa7f80047c0 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.939+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa800005c30 con 0x7fa7f80047c0 2024-01-16T15:03:48.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.939+0000 7fa817e00640 1 -- 172.21.15.141:0/1436003339 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80002c7c0 con 0x7fa7f80047c0 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.941+0000 7fa81fc00640 1 -- 172.21.15.141:0/1436003339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7f80047c0 msgr2=0x7fa7f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.941+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7f80047c0 0x7fa7f8004c40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa800007fd0 tx=0x7fa7f80057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.941+0000 7fa81fc00640 1 -- 172.21.15.141:0/1436003339 shutdown_connections 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.941+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1436003339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7f80047c0 0x7fa7f8004c40 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.941+0000 7fa81fc00640 1 -- 172.21.15.141:0/1436003339 >> 172.21.15.141:0/1436003339 conn(0x7fa81806a800 msgr2=0x7fa81806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:48.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.943+0000 7fa81fc00640 1 -- 172.21.15.141:0/1436003339 shutdown_connections 2024-01-16T15:03:48.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.943+0000 7fa81fc00640 1 -- 172.21.15.141:0/1436003339 wait complete. 2024-01-16T15:03:48.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.946+0000 7fa81fc00640 1 Processor -- start 2024-01-16T15:03:48.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.947+0000 7fa81fc00640 1 -- start start 2024-01-16T15:03:48.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.948+0000 7fa81fc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80047c0 0x7fa818136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa81fc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa818159a90 0x7fa818137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa81fc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8181399e0 0x7fa818139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa81fc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa818138550 con 0x7fa818159a90 2024-01-16T15:03:48.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa81fc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa818159f10 con 0x7fa8181399e0 2024-01-16T15:03:48.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa81fc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8181382b0 con 0x7fa7f80047c0 2024-01-16T15:03:48.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa80fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa818159a90 0x7fa818137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa80fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa818159a90 0x7fa818137d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44562/0 (socket says 172.21.15.141:44562) 2024-01-16T15:03:48.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.949+0000 7fa80fe00640 1 -- 172.21.15.141:0/1649581506 learned_addr learned my addr 172.21.15.141:0/1649581506 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:48.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.950+0000 7fa81d200640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80047c0 0x7fa818136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.951+0000 7fa80fe00640 1 -- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80047c0 msgr2=0x7fa818136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:48.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.951+0000 7fa80fe00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80047c0 0x7fa818136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.951+0000 7fa80fe00640 1 -- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8181399e0 msgr2=0x7fa818139e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:03:48.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.951+0000 7fa80fe00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8181399e0 0x7fa818139e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:48.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.951+0000 7fa80fe00640 1 -- 172.21.15.141:0/1649581506 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81815b750 con 0x7fa818159a90 2024-01-16T15:03:48.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.951+0000 7fa80fe00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa818159a90 0x7fa818137d70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fa800009000 tx=0x7fa80002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:48.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.952+0000 7fa815600640 1 -- 172.21.15.141:0/1649581506 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80003c040 con 0x7fa818159a90 2024-01-16T15:03:48.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.953+0000 7fa815600640 1 -- 172.21.15.141:0/1649581506 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80002cc90 con 0x7fa818159a90 2024-01-16T15:03:48.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.953+0000 7fa815600640 1 -- 172.21.15.141:0/1649581506 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800007070 con 0x7fa818159a90 2024-01-16T15:03:48.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.953+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81815ba30 con 0x7fa818159a90 2024-01-16T15:03:48.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.954+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81815bff0 con 0x7fa818159a90 2024-01-16T15:03:48.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.956+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8180675d0 con 0x7fa818159a90 2024-01-16T15:03:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.959+0000 7fa815600640 1 -- 172.21.15.141:0/1649581506 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fa800032070 con 0x7fa818159a90 2024-01-16T15:03:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.960+0000 7fa815600640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f006ba10 0x7fa7f006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.962+0000 7fa815600640 1 -- 172.21.15.141:0/1649581506 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8000795e0 con 0x7fa818159a90 2024-01-16T15:03:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.962+0000 7fa81d200640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f006ba10 0x7fa7f006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:48.971+0000 7fa815600640 1 -- 172.21.15.141:0/1649581506 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8000b0060 con 0x7fa818159a90 2024-01-16T15:03:49.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.034+0000 7fa81d200640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f006ba10 0x7fa7f006ded0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa818071c20 tx=0x7fa808008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:49.279 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:49.300+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:49.279 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:49.300+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:49.279 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:49.300+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:49.279 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:49.300+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:49.279 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:49.300+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:49.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.407+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa8180b0540 con 0x7fa818159a90 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.408+0000 7fa815600640 1 -- 172.21.15.141:0/1649581506 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7fa800046070 con 0x7fa818159a90 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:03:49.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 422, 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:03:49.388 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:03:49.389 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 804, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222280523, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6752432128, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766341681152, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 597, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 13135, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 3 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:03:49.390 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:03:49.391 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.392 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:49.393 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.412+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f006ba10 msgr2=0x7fa7f006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.412+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f006ba10 0x7fa7f006ded0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa818071c20 tx=0x7fa808008040 comp rx=0 tx=0).stop 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.412+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa818159a90 msgr2=0x7fa818137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa818159a90 0x7fa818137d70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fa800009000 tx=0x7fa80002e070 comp rx=0 tx=0).stop 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 shutdown_connections 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f006ba10 0x7fa7f006ded0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8181399e0 0x7fa818139e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa818159a90 0x7fa818137d70 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 --2- 172.21.15.141:0/1649581506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7f80047c0 0x7fa818136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 >> 172.21.15.141:0/1649581506 conn(0x7fa81806a800 msgr2=0x7fa8180aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:49.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 shutdown_connections 2024-01-16T15:03:49.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:49.413+0000 7fa81fc00640 1 -- 172.21.15.141:0/1649581506 wait complete. 2024-01-16T15:03:51.416 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.648+0000 7f4bf9000640 1 Processor -- start 2024-01-16T15:03:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.648+0000 7f4bf9000640 1 -- start start 2024-01-16T15:03:51.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.650+0000 7f4bf9000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4bf41607a0 0x7f4bf40b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.650+0000 7f4bf9000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf4161ee0 con 0x7f4bf4160b80 2024-01-16T15:03:51.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.650+0000 7f4bf9000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf40b2b90 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.650+0000 7f4bf9000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf40d21d0 con 0x7f4bf41607a0 2024-01-16T15:03:51.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.650+0000 7f4bf3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4bf4160b80 0x7f4bf41617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43070/0 (socket says 172.21.15.141:43070) 2024-01-16T15:03:51.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.650+0000 7f4bf3e00640 1 -- 172.21.15.141:0/3862155568 learned_addr learned my addr 172.21.15.141:0/3862155568 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:51.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.650+0000 7f4bf3400640 1 --2- 172.21.15.141:0/3862155568 >> v2:172.21.15.141:3300/0 conn(0x7f4bf41607a0 0x7f4bf40b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.651+0000 7f4bf3400640 1 -- 172.21.15.141:0/3862155568 >> v1:172.21.15.188:6789/0 conn(0x7f4bf40b3eb0 legacy=0x7f4bf41610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.651+0000 7f4bf3400640 1 -- 172.21.15.141:0/3862155568 >> v1:172.21.15.141:6789/0 conn(0x7f4bf4160b80 legacy=0x7f4bf41617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.651+0000 7f4bf3400640 1 -- 172.21.15.141:0/3862155568 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf40b2e30 con 0x7f4bf41607a0 2024-01-16T15:03:51.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.651+0000 7f4bf2000640 1 -- 172.21.15.141:0/3862155568 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947395464 0 0) 0x7f4bf4161ee0 con 0x7f4bf4160b80 2024-01-16T15:03:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.652+0000 7f4bf3400640 1 --2- 172.21.15.141:0/3862155568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf41607a0 0x7f4bf40b4330 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4be40097a0 tx=0x7f4be402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba5d50739f483cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.652+0000 7f4bf2000640 1 -- 172.21.15.141:0/3862155568 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be4031830 con 0x7f4bf41607a0 2024-01-16T15:03:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.652+0000 7f4bf2000640 1 -- 172.21.15.141:0/3862155568 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be4031990 con 0x7f4bf41607a0 2024-01-16T15:03:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.652+0000 7f4bf2000640 1 -- 172.21.15.141:0/3862155568 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be4031d20 con 0x7f4bf41607a0 2024-01-16T15:03:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.653+0000 7f4bf9000640 1 -- 172.21.15.141:0/3862155568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf41607a0 msgr2=0x7f4bf40b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.653+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3862155568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf41607a0 0x7f4bf40b4330 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4be40097a0 tx=0x7f4be402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.653+0000 7f4bf9000640 1 -- 172.21.15.141:0/3862155568 shutdown_connections 2024-01-16T15:03:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.653+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3862155568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf41607a0 0x7f4bf40b4330 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.653+0000 7f4bf9000640 1 -- 172.21.15.141:0/3862155568 >> 172.21.15.141:0/3862155568 conn(0x7f4bf409c590 msgr2=0x7f4bf40d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.653+0000 7f4bf9000640 1 -- 172.21.15.141:0/3862155568 shutdown_connections 2024-01-16T15:03:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.653+0000 7f4bf9000640 1 -- 172.21.15.141:0/3862155568 wait complete. 2024-01-16T15:03:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.655+0000 7f4bf9000640 1 Processor -- start 2024-01-16T15:03:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.655+0000 7f4bf9000640 1 -- start start 2024-01-16T15:03:51.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.655+0000 7f4bf9000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf40b3eb0 0x7f4bf4151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf9000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4bf41607a0 0x7f4bf40cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf9000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4bf4160b80 0x7f4bf40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf9000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf40c8c00 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf9000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf4151fc0 con 0x7f4bf41607a0 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf9000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf40c7b60 con 0x7f4bf4160b80 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf40b3eb0 0x7f4bf4151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf40b3eb0 0x7f4bf4151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44578/0 (socket says 172.21.15.141:44578) 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 -- 172.21.15.141:0/3849041525 learned_addr learned my addr 172.21.15.141:0/3849041525 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 -- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4bf4160b80 msgr2=0x7f4bf40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4bf4160b80 0x7f4bf40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 -- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4bf41607a0 msgr2=0x7f4bf40cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf2a00640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4bf41607a0 0x7f4bf40cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4bf41607a0 0x7f4bf40cee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.656+0000 7f4bf3400640 1 -- 172.21.15.141:0/3849041525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be4009400 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.657+0000 7f4bf3400640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf40b3eb0 0x7f4bf4151a80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4be40098c0 tx=0x7f4be40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.657+0000 7f4beb400640 1 -- 172.21.15.141:0/3849041525 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be403ba70 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.657+0000 7f4beb400640 1 -- 172.21.15.141:0/3849041525 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be403bbd0 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.657+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf40c7e40 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.657+0000 7f4beb400640 1 -- 172.21.15.141:0/3849041525 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be40453f0 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.657+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf40c8400 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.658+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf40b3990 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.659+0000 7f4beb400640 1 -- 172.21.15.141:0/3849041525 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f4be404e450 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.660+0000 7f4beb400640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bcc06baa0 0x7f4bcc06df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.661+0000 7f4beb400640 1 -- 172.21.15.141:0/3849041525 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4be40aa550 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.661+0000 7f4bf2a00640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bcc06baa0 0x7f4bcc06df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.662+0000 7f4bf2a00640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bcc06baa0 0x7f4bcc06df60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4bf41f6f60 tx=0x7f4bdc0072d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.665+0000 7f4beb400640 1 -- 172.21.15.141:0/3849041525 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bf40b3990 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.973+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4bf40b3990 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.974+0000 7f4beb400640 1 -- 172.21.15.141:0/3849041525 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2424 (secure 0 0 0) 0x7f4bf40b3990 con 0x7f4bf40b3eb0 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:03:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 424, 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:03:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:03:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1299, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5298623858, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10097500160, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762996613120, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 178533597, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 172971601, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 50 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:03:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:03:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 49s)", 2024-01-16T15:03:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.22220000624656677, 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:51.960 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bcc06baa0 msgr2=0x7f4bcc06df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bcc06baa0 0x7f4bcc06df60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4bf41f6f60 tx=0x7f4bdc0072d0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf40b3eb0 msgr2=0x7f4bf4151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf40b3eb0 0x7f4bf4151a80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4be40098c0 tx=0x7f4be40046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 shutdown_connections 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4bcc06baa0 0x7f4bcc06df60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4bf4160b80 0x7f4bf40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4bf41607a0 0x7f4bf40cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 --2- 172.21.15.141:0/3849041525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4bf40b3eb0 0x7f4bf4151a80 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.977+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 >> 172.21.15.141:0/3849041525 conn(0x7f4bf409c590 msgr2=0x7f4bf406b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:51.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.978+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 shutdown_connections 2024-01-16T15:03:51.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:51.978+0000 7f4bf9000640 1 -- 172.21.15.141:0/3849041525 wait complete. 2024-01-16T15:03:53.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:54.295 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:54.316+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:54.295 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:54.316+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:54.295 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:54.316+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:54.295 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:54.316+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:54.295 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:03:54.316+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:54.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.322+0000 7fdd01600640 1 Processor -- start 2024-01-16T15:03:54.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.322+0000 7fdd01600640 1 -- start start 2024-01-16T15:03:54.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.324+0000 7fdd01600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdcfc0cf9a0 0x7fdcfc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.324+0000 7fdd01600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdcfc15e1e0 0x7fdcfc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.324+0000 7fdd01600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc0b1ee0 con 0x7fdcfc143ae0 2024-01-16T15:03:54.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.324+0000 7fdd01600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcfc0b2780 con 0x7fdcfc15e1e0 2024-01-16T15:03:54.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.324+0000 7fdd01600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcfc15e990 con 0x7fdcfc0cf9a0 2024-01-16T15:03:54.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.325+0000 7fdcfb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdcfc15e1e0 0x7fdcfc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.325+0000 7fdcfb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdcfc15e1e0 0x7fdcfc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53422/0 (socket says 172.21.15.141:53422) 2024-01-16T15:03:54.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.325+0000 7fdcfb400640 1 -- 172.21.15.141:0/1627641496 learned_addr learned my addr 172.21.15.141:0/1627641496 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:54.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.325+0000 7fdcfa000640 1 --2- 172.21.15.141:0/1627641496 >> v2:172.21.15.188:3300/0 conn(0x7fdcfc0cf9a0 0x7fdcfc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.326+0000 7fdcfb400640 1 -- 172.21.15.141:0/1627641496 >> v2:172.21.15.188:3300/0 conn(0x7fdcfc0cf9a0 msgr2=0x7fdcfc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:54.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.326+0000 7fdcfb400640 1 --2- 172.21.15.141:0/1627641496 >> v2:172.21.15.188:3300/0 conn(0x7fdcfc0cf9a0 0x7fdcfc15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.326+0000 7fdcfb400640 1 -- 172.21.15.141:0/1627641496 >> v1:172.21.15.141:6789/0 conn(0x7fdcfc143ae0 legacy=0x7fdcfc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:54.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.326+0000 7fdcfb400640 1 -- 172.21.15.141:0/1627641496 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc0b2130 con 0x7fdcfc15e1e0 2024-01-16T15:03:54.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.328+0000 7fdcfb400640 1 --2- 172.21.15.141:0/1627641496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc15e1e0 0x7fdcfc0702c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc0cb830 tx=0x7fdcf40302f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792ddb2afe6a1ff9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:54.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.329+0000 7fdcf9600640 1 -- 172.21.15.141:0/1627641496 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4030e90 con 0x7fdcfc15e1e0 2024-01-16T15:03:54.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.329+0000 7fdcf9600640 1 -- 172.21.15.141:0/1627641496 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf40383f0 con 0x7fdcfc15e1e0 2024-01-16T15:03:54.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.329+0000 7fdcf9600640 1 -- 172.21.15.141:0/1627641496 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4038780 con 0x7fdcfc15e1e0 2024-01-16T15:03:54.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.331+0000 7fdd01600640 1 -- 172.21.15.141:0/1627641496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc15e1e0 msgr2=0x7fdcfc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.331+0000 7fdd01600640 1 --2- 172.21.15.141:0/1627641496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc15e1e0 0x7fdcfc0702c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc0cb830 tx=0x7fdcf40302f0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.334+0000 7fdd01600640 1 -- 172.21.15.141:0/1627641496 shutdown_connections 2024-01-16T15:03:54.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.334+0000 7fdd01600640 1 --2- 172.21.15.141:0/1627641496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc15e1e0 0x7fdcfc0702c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.334+0000 7fdd01600640 1 --2- 172.21.15.141:0/1627641496 >> v2:172.21.15.188:3300/0 conn(0x7fdcfc0cf9a0 0x7fdcfc15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.334+0000 7fdd01600640 1 -- 172.21.15.141:0/1627641496 >> 172.21.15.141:0/1627641496 conn(0x7fdcfc06a800 msgr2=0x7fdcfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.335+0000 7fdd01600640 1 -- 172.21.15.141:0/1627641496 shutdown_connections 2024-01-16T15:03:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.335+0000 7fdd01600640 1 -- 172.21.15.141:0/1627641496 wait complete. 2024-01-16T15:03:54.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.338+0000 7fdd01600640 1 Processor -- start 2024-01-16T15:03:54.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.338+0000 7fdd01600640 1 -- start start 2024-01-16T15:03:54.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.339+0000 7fdd01600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcfc0cf9a0 0x7fdcfc130640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.339+0000 7fdd01600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc143ae0 0x7fdcfc0cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.339+0000 7fdd01600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdcfc15e1e0 0x7fdcfc131dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.339+0000 7fdd01600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc142020 con 0x7fdcfc143ae0 2024-01-16T15:03:54.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.339+0000 7fdd01600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc130d20 con 0x7fdcfc0cf9a0 2024-01-16T15:03:54.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.339+0000 7fdd01600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc131000 con 0x7fdcfc15e1e0 2024-01-16T15:03:54.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc143ae0 0x7fdcfc0cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc143ae0 0x7fdcfc0cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53436/0 (socket says 172.21.15.141:53436) 2024-01-16T15:03:54.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 -- 172.21.15.141:0/1436679565 learned_addr learned my addr 172.21.15.141:0/1436679565 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:54.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 -- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdcfc15e1e0 msgr2=0x7fdcfc131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdcfc15e1e0 0x7fdcfc131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 -- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcfc0cf9a0 msgr2=0x7fdcfc130640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcfc0cf9a0 0x7fdcfc130640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 -- 172.21.15.141:0/1436679565 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf4009b30 con 0x7fdcfc143ae0 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.340+0000 7fdcfa000640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc143ae0 0x7fdcfc0cb160 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fdce400d990 tx=0x7fdce400de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.341+0000 7fdcf2a00640 1 -- 172.21.15.141:0/1436679565 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4007a10 con 0x7fdcfc143ae0 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.341+0000 7fdcf2a00640 1 -- 172.21.15.141:0/1436679565 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce4007b70 con 0x7fdcfc143ae0 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.341+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc1312e0 con 0x7fdcfc143ae0 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.341+0000 7fdcf2a00640 1 -- 172.21.15.141:0/1436679565 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4010470 con 0x7fdcfc143ae0 2024-01-16T15:03:54.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.341+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc1318a0 con 0x7fdcfc143ae0 2024-01-16T15:03:54.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.342+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc06e0d0 con 0x7fdcfc143ae0 2024-01-16T15:03:54.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.343+0000 7fdcf2a00640 1 -- 172.21.15.141:0/1436679565 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 90253+0+0 (secure 0 0 0) 0x7fdce40105d0 con 0x7fdcfc143ae0 2024-01-16T15:03:54.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.344+0000 7fdcf2a00640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdcd406baa0 0x7fdcd406df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:54.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.344+0000 7fdcf2a00640 1 -- 172.21.15.141:0/1436679565 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdce40885b0 con 0x7fdcfc143ae0 2024-01-16T15:03:54.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.349+0000 7fdcf2a00640 1 -- 172.21.15.141:0/1436679565 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdce4055020 con 0x7fdcfc143ae0 2024-01-16T15:03:54.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.406+0000 7fdcfaa00640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdcd406baa0 0x7fdcd406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:54.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.443+0000 7fdcfaa00640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdcd406baa0 0x7fdcd406df60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdcec003cc0 tx=0x7fdcec0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:54.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.810+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdcfc0675d0 con 0x7fdcfc143ae0 2024-01-16T15:03:54.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.811+0000 7fdcf2a00640 1 -- 172.21.15.141:0/1436679565 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2424 (secure 0 0 0) 0x7fdce401f020 con 0x7fdcfc143ae0 2024-01-16T15:03:54.790 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:03:54.791 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 427, 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:03:54.792 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1299, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5298623858, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10097500160, 2024-01-16T15:03:54.793 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762996613120, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 178485880, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 172919738, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 48 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:54.794 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:03:54.795 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 49s)", 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-16T15:03:54.796 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.22220000624656677, 2024-01-16T15:03:54.797 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:54.797 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:54.797 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:54.797 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:03:54.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.815+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdcd406baa0 msgr2=0x7fdcd406df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.815+0000 7fdd01600640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdcd406baa0 0x7fdcd406df60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdcec003cc0 tx=0x7fdcec0079b0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.815+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc143ae0 msgr2=0x7fdcfc0cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.815+0000 7fdd01600640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc143ae0 0x7fdcfc0cb160 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fdce400d990 tx=0x7fdce400de50 comp rx=0 tx=0).stop 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 shutdown_connections 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdcd406baa0 0x7fdcd406df60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdcfc15e1e0 0x7fdcfc131dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdcfc143ae0 0x7fdcfc0cb160 secure :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fdce400d990 tx=0x7fdce400de50 comp rx=0 tx=0).stop 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 --2- 172.21.15.141:0/1436679565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcfc0cf9a0 0x7fdcfc130640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 >> 172.21.15.141:0/1436679565 conn(0x7fdcfc06a800 msgr2=0x7fdcfc0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 shutdown_connections 2024-01-16T15:03:54.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:54.816+0000 7fdd01600640 1 -- 172.21.15.141:0/1436679565 wait complete. 2024-01-16T15:03:56.818 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:57.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.052+0000 7fa01a200640 1 Processor -- start 2024-01-16T15:03:57.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.052+0000 7fa01a200640 1 -- start start 2024-01-16T15:03:57.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa01a200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa01415da10 0x7fa014072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa01a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0140b2bd0 0x7fa0140b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa01a200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa014071190 con 0x7fa014150760 2024-01-16T15:03:57.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa01a200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa014071470 con 0x7fa01415da10 2024-01-16T15:03:57.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa01a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa014071750 con 0x7fa0140b2bd0 2024-01-16T15:03:57.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa013e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0140b2bd0 0x7fa0140b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa013400640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa014150760 0x7fa01415d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:49008/0 (socket says 172.21.15.141:49008) 2024-01-16T15:03:57.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.054+0000 7fa013400640 1 -- 172.21.15.141:0/1156954334 learned_addr learned my addr 172.21.15.141:0/1156954334 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:03:57.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.055+0000 7fa012a00640 1 --2- 172.21.15.141:0/1156954334 >> v2:172.21.15.188:3300/0 conn(0x7fa01415da10 0x7fa014072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.055+0000 7fa013e00640 1 -- 172.21.15.141:0/1156954334 >> v2:172.21.15.188:3300/0 conn(0x7fa01415da10 msgr2=0x7fa014072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:57.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.055+0000 7fa013e00640 1 --2- 172.21.15.141:0/1156954334 >> v2:172.21.15.188:3300/0 conn(0x7fa01415da10 0x7fa014072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.055+0000 7fa013e00640 1 -- 172.21.15.141:0/1156954334 >> v1:172.21.15.188:6789/0 conn(0x7fa014150760 legacy=0x7fa01415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.055+0000 7fa013e00640 1 -- 172.21.15.141:0/1156954334 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014071a30 con 0x7fa0140b2bd0 2024-01-16T15:03:57.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.055+0000 7fa012a00640 1 --2- 172.21.15.141:0/1156954334 >> v2:172.21.15.188:3300/0 conn(0x7fa01415da10 0x7fa014072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:03:57.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.057+0000 7fa013e00640 1 --2- 172.21.15.141:0/1156954334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0140b2bd0 0x7fa0140b3060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d90 tx=0x7fa00c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c01927b9669c7c02 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:57.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.057+0000 7fa012000640 1 -- 172.21.15.141:0/1156954334 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c02cc60 con 0x7fa0140b2bd0 2024-01-16T15:03:57.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.057+0000 7fa012000640 1 -- 172.21.15.141:0/1156954334 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c02cdc0 con 0x7fa0140b2bd0 2024-01-16T15:03:57.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.057+0000 7fa012000640 1 -- 172.21.15.141:0/1156954334 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c034660 con 0x7fa0140b2bd0 2024-01-16T15:03:57.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 -- 172.21.15.141:0/1156954334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0140b2bd0 msgr2=0x7fa0140b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 --2- 172.21.15.141:0/1156954334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0140b2bd0 0x7fa0140b3060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d90 tx=0x7fa00c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 -- 172.21.15.141:0/1156954334 shutdown_connections 2024-01-16T15:03:57.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 --2- 172.21.15.141:0/1156954334 >> v2:172.21.15.188:3300/0 conn(0x7fa01415da10 0x7fa014072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 --2- 172.21.15.141:0/1156954334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa0140b2bd0 0x7fa0140b3060 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 -- 172.21.15.141:0/1156954334 >> 172.21.15.141:0/1156954334 conn(0x7fa01406a800 msgr2=0x7fa01406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:57.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 -- 172.21.15.141:0/1156954334 shutdown_connections 2024-01-16T15:03:57.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.058+0000 7fa01a200640 1 -- 172.21.15.141:0/1156954334 wait complete. 2024-01-16T15:03:57.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.060+0000 7fa01a200640 1 Processor -- start 2024-01-16T15:03:57.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.060+0000 7fa01a200640 1 -- start start 2024-01-16T15:03:57.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa01a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0140b2bd0 0x7fa0140cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa01a200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa014150760 0x7fa014141b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa01a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa01415da10 0x7fa0140cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa01a200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa014142090 con 0x7fa014150760 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa01a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140cc4b0 con 0x7fa0140b2bd0 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa01a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140cc790 con 0x7fa01415da10 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa012a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa014150760 0x7fa014141b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa012a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa014150760 0x7fa014141b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53440/0 (socket says 172.21.15.141:53440) 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa012a00640 1 -- 172.21.15.141:0/2233799874 learned_addr learned my addr 172.21.15.141:0/2233799874 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa013400640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0140b2bd0 0x7fa0140cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa013e00640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa01415da10 0x7fa0140cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa012a00640 1 -- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa01415da10 msgr2=0x7fa0140cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa012a00640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa01415da10 0x7fa0140cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa012a00640 1 -- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0140b2bd0 msgr2=0x7fa0140cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.062+0000 7fa012a00640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0140b2bd0 0x7fa0140cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa012a00640 1 -- 172.21.15.141:0/2233799874 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa004009400 con 0x7fa014150760 2024-01-16T15:03:57.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa013400640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0140b2bd0 0x7fa0140cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:57.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa012a00640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa014150760 0x7fa014141b50 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa00400e960 tx=0x7fa00400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:57.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa00b400640 1 -- 172.21.15.141:0/2233799874 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00400ca00 con 0x7fa014150760 2024-01-16T15:03:57.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa00b400640 1 -- 172.21.15.141:0/2233799874 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00400cb60 con 0x7fa014150760 2024-01-16T15:03:57.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa00b400640 1 -- 172.21.15.141:0/2233799874 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004013470 con 0x7fa014150760 2024-01-16T15:03:57.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c009a70 con 0x7fa014150760 2024-01-16T15:03:57.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.063+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0140ccd50 con 0x7fa014150760 2024-01-16T15:03:57.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.065+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0140675d0 con 0x7fa014150760 2024-01-16T15:03:57.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.066+0000 7fa00b400640 1 -- 172.21.15.141:0/2233799874 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa0040135d0 con 0x7fa014150760 2024-01-16T15:03:57.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.067+0000 7fa00b400640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fec06bb60 0x7f9fec06e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:57.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.067+0000 7fa00b400640 1 -- 172.21.15.141:0/2233799874 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa004088810 con 0x7fa014150760 2024-01-16T15:03:57.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.067+0000 7fa013400640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fec06bb60 0x7f9fec06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:57.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.069+0000 7fa013400640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fec06bb60 0x7f9fec06e020 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc005880 tx=0x7f9ffc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:57.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.071+0000 7fa00b400640 1 -- 172.21.15.141:0/2233799874 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0040bde90 con 0x7fa014150760 2024-01-16T15:03:57.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.335+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa0140b2640 con 0x7fa014150760 2024-01-16T15:03:57.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.336+0000 7fa00b400640 1 -- 172.21.15.141:0/2233799874 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fa004017990 con 0x7fa014150760 2024-01-16T15:03:57.315 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:03:57.315 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:03:57.316 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 429, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:03:57.317 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1793, 2024-01-16T15:03:57.318 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7370730693, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 14289625088, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 758804488192, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 357325430, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 345482894, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 86, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 93 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:03:57.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:03:57.320 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (12s)\n [======......................] (remaining: 38s)", 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.23810000717639923, 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:03:57.321 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:03:57.322 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:03:57.322 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.41667500138282776, 2024-01-16T15:03:57.322 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:03:57.322 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:57.322 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:03:57.322 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fec06bb60 msgr2=0x7f9fec06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fec06bb60 0x7f9fec06e020 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc005880 tx=0x7f9ffc009040 comp rx=0 tx=0).stop 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa014150760 msgr2=0x7fa014141b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa014150760 0x7fa014141b50 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa00400e960 tx=0x7fa00400ee20 comp rx=0 tx=0).stop 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 shutdown_connections 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fec06bb60 0x7f9fec06e020 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa01415da10 0x7fa0140cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa014150760 0x7fa014141b50 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 --2- 172.21.15.141:0/2233799874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0140b2bd0 0x7fa0140cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 >> 172.21.15.141:0/2233799874 conn(0x7fa01406a800 msgr2=0x7fa01415ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 shutdown_connections 2024-01-16T15:03:57.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:57.339+0000 7fa01a200640 1 -- 172.21.15.141:0/2233799874 wait complete. 2024-01-16T15:03:59.338 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.710+0000 7f87d6200640 1 Processor -- start 2024-01-16T15:03:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.710+0000 7f87d6200640 1 -- start start 2024-01-16T15:03:59.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.713+0000 7f87d6200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f87d016dce0 0x7f87d006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:59.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.713+0000 7f87d6200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f87d0070cd0 0x7f87d008a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.714+0000 7f87d6200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d008ae30 con 0x7f87d006d430 2024-01-16T15:03:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.714+0000 7f87d6200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d00711b0 con 0x7f87d0070cd0 2024-01-16T15:03:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.714+0000 7f87d6200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d0071490 con 0x7f87d016dce0 2024-01-16T15:03:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.714+0000 7f87cfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f87d0070cd0 0x7f87d008a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.714+0000 7f87cf400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f87d016dce0 0x7f87d006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.714+0000 7f87cfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f87d0070cd0 0x7f87d008a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53448/0 (socket says 172.21.15.141:53448) 2024-01-16T15:03:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.714+0000 7f87cfe00640 1 -- 172.21.15.141:0/3421297106 learned_addr learned my addr 172.21.15.141:0/3421297106 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.716+0000 7f87cfe00640 1 -- 172.21.15.141:0/3421297106 >> v2:172.21.15.188:3300/0 conn(0x7f87d016dce0 msgr2=0x7f87d006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.716+0000 7f87cfe00640 1 --2- 172.21.15.141:0/3421297106 >> v2:172.21.15.188:3300/0 conn(0x7f87d016dce0 0x7f87d006cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.716+0000 7f87cfe00640 1 -- 172.21.15.141:0/3421297106 >> v1:172.21.15.141:6789/0 conn(0x7f87d006d430 legacy=0x7f87d008a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.716+0000 7f87cfe00640 1 -- 172.21.15.141:0/3421297106 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d0071770 con 0x7f87d0070cd0 2024-01-16T15:03:59.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.717+0000 7f87cfe00640 1 --2- 172.21.15.141:0/3421297106 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d0070cd0 0x7f87d008a8f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f87d00b1860 tx=0x7f87c8034aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ffeb64f1727841 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:59.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.718+0000 7f87ce000640 1 -- 172.21.15.141:0/3421297106 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c80114e0 con 0x7f87d0070cd0 2024-01-16T15:03:59.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.718+0000 7f87ce000640 1 -- 172.21.15.141:0/3421297106 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8011640 con 0x7f87d0070cd0 2024-01-16T15:03:59.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.718+0000 7f87ce000640 1 -- 172.21.15.141:0/3421297106 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c80119d0 con 0x7f87d0070cd0 2024-01-16T15:03:59.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.719+0000 7f87d6200640 1 -- 172.21.15.141:0/3421297106 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d0070cd0 msgr2=0x7f87d008a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:59.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.719+0000 7f87d6200640 1 --2- 172.21.15.141:0/3421297106 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d0070cd0 0x7f87d008a8f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f87d00b1860 tx=0x7f87c8034aa0 comp rx=0 tx=0).stop 2024-01-16T15:03:59.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.722+0000 7f87d6200640 1 -- 172.21.15.141:0/3421297106 shutdown_connections 2024-01-16T15:03:59.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.722+0000 7f87d6200640 1 --2- 172.21.15.141:0/3421297106 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d0070cd0 0x7f87d008a8f0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:59.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.722+0000 7f87d6200640 1 --2- 172.21.15.141:0/3421297106 >> v2:172.21.15.188:3300/0 conn(0x7f87d016dce0 0x7f87d006cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:59.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.722+0000 7f87d6200640 1 -- 172.21.15.141:0/3421297106 >> 172.21.15.141:0/3421297106 conn(0x7f87d00b0be0 msgr2=0x7f87d00b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:59.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.722+0000 7f87d6200640 1 -- 172.21.15.141:0/3421297106 shutdown_connections 2024-01-16T15:03:59.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.722+0000 7f87d6200640 1 -- 172.21.15.141:0/3421297106 wait complete. 2024-01-16T15:03:59.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.726+0000 7f87d6200640 1 Processor -- start 2024-01-16T15:03:59.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.726+0000 7f87d6200640 1 -- start start 2024-01-16T15:03:59.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.732+0000 7f87d6200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d006d430 0x7f87d015b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.732+0000 7f87d6200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87d0070cd0 0x7f87d016c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.732+0000 7f87d6200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87d016dce0 0x7f87d00ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.733+0000 7f87d6200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00ac2b0 con 0x7f87d006d430 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.733+0000 7f87d6200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00ab980 con 0x7f87d016dce0 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.733+0000 7f87d6200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00abc60 con 0x7f87d0070cd0 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.733+0000 7f87cf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d006d430 0x7f87d015b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.733+0000 7f87cf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d006d430 0x7f87d015b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53456/0 (socket says 172.21.15.141:53456) 2024-01-16T15:03:59.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.733+0000 7f87cf400640 1 -- 172.21.15.141:0/30879950 learned_addr learned my addr 172.21.15.141:0/30879950 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:03:59.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.735+0000 7f87cf400640 1 -- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87d0070cd0 msgr2=0x7f87d016c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:59.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.736+0000 7f87cfe00640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87d016dce0 0x7f87d00ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:59.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.736+0000 7f87cea00640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87d0070cd0 0x7f87d016c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:59.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.738+0000 7f87cf400640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87d0070cd0 0x7f87d016c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:59.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.738+0000 7f87cf400640 1 -- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87d016dce0 msgr2=0x7f87d00ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:59.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.738+0000 7f87cf400640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87d016dce0 0x7f87d00ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:59.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.738+0000 7f87cf400640 1 -- 172.21.15.141:0/30879950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c8014660 con 0x7f87d006d430 2024-01-16T15:03:59.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.738+0000 7f87cfe00640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87d016dce0 0x7f87d00ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:59.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.738+0000 7f87cea00640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87d0070cd0 0x7f87d016c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:59.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.741+0000 7f87cf400640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d006d430 0x7f87d015b930 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f87b800c7f0 tx=0x7f87b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:59.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.742+0000 7f87c7400640 1 -- 172.21.15.141:0/30879950 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8004220 con 0x7f87d006d430 2024-01-16T15:03:59.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.742+0000 7f87c7400640 1 -- 172.21.15.141:0/30879950 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b80043a0 con 0x7f87d006d430 2024-01-16T15:03:59.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.742+0000 7f87c7400640 1 -- 172.21.15.141:0/30879950 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8007860 con 0x7f87d006d430 2024-01-16T15:03:59.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.742+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d00abf40 con 0x7f87d006d430 2024-01-16T15:03:59.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.744+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d00acc80 con 0x7f87d006d430 2024-01-16T15:03:59.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.747+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d00675d0 con 0x7f87d006d430 2024-01-16T15:03:59.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.751+0000 7f87c7400640 1 -- 172.21.15.141:0/30879950 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f87b8007bd0 con 0x7f87d006d430 2024-01-16T15:03:59.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.753+0000 7f87c7400640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87a806bb30 0x7f87a806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:59.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.753+0000 7f87c7400640 1 -- 172.21.15.141:0/30879950 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87b8088610 con 0x7f87d006d430 2024-01-16T15:03:59.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.757+0000 7f87cea00640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87a806bb30 0x7f87a806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:59.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.762+0000 7f87c7400640 1 -- 172.21.15.141:0/30879950 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87b8054d00 con 0x7f87d006d430 2024-01-16T15:03:59.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:03:59.785+0000 7f87cea00640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87a806bb30 0x7f87a806dff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f87d00cd3b0 tx=0x7f87c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:00.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.218+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f87d00cd3e0 con 0x7f87d006d430 2024-01-16T15:04:00.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.218+0000 7f87c7400640 1 -- 172.21.15.141:0/30879950 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f87b801e090 con 0x7f87d006d430 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:04:00.198 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 432, 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.199 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:00.200 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1793, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7370730693, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 14289625088, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 758804488192, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 357293683, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 345445278, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 85, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 91 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:04:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:04:00.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.203 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (12s)\n [======......................] (remaining: 38s)", 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.23810000717639923, 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.41667500138282776, 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:00.204 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:00.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87a806bb30 msgr2=0x7f87a806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:00.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87a806bb30 0x7f87a806dff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f87d00cd3b0 tx=0x7f87c0009040 comp rx=0 tx=0).stop 2024-01-16T15:04:00.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d006d430 msgr2=0x7f87d015b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d006d430 0x7f87d015b930 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f87b800c7f0 tx=0x7f87b800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 shutdown_connections 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f87a806bb30 0x7f87a806dff0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87d016dce0 0x7f87d00ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f87d0070cd0 0x7f87d016c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 --2- 172.21.15.141:0/30879950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f87d006d430 0x7f87d015b930 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 >> 172.21.15.141:0/30879950 conn(0x7f87d00b0be0 msgr2=0x7f87d00cc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 shutdown_connections 2024-01-16T15:04:00.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:00.221+0000 7f87d6200640 1 -- 172.21.15.141:0/30879950 wait complete. 2024-01-16T15:04:02.229 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:02.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.506+0000 7f7d05a00640 1 Processor -- start 2024-01-16T15:04:02.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.506+0000 7f7d05a00640 1 -- start start 2024-01-16T15:04:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.509+0000 7f7d05a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7d00072230 0x7f7d000b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.509+0000 7f7d05a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d00071190 con 0x7f7d0015da10 2024-01-16T15:04:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.509+0000 7f7d05a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d00071470 con 0x7f7d00150760 2024-01-16T15:04:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.509+0000 7f7d05a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d00071750 con 0x7f7d00072230 2024-01-16T15:04:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.509+0000 7f7cf7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7d0015da10 0x7f7d000b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42332/0 (socket says 172.21.15.141:42332) 2024-01-16T15:04:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.509+0000 7f7cf7e00640 1 -- 172.21.15.141:0/221137029 learned_addr learned my addr 172.21.15.141:0/221137029 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.509+0000 7f7cff400640 1 --2- 172.21.15.141:0/221137029 >> v2:172.21.15.188:3300/0 conn(0x7f7d00072230 0x7f7d000b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249364769 0 0) 0x7f7d00071190 con 0x7f7d0015da10 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce0005730 con 0x7f7d0015da10 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2024598541 0 0) 0x7f7ce0005730 con 0x7f7d0015da10 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 >> v2:172.21.15.188:3300/0 conn(0x7f7d00072230 msgr2=0x7f7d000b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 --2- 172.21.15.141:0/221137029 >> v2:172.21.15.188:3300/0 conn(0x7f7d00072230 0x7f7d000b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 >> v1:172.21.15.188:6789/0 conn(0x7f7d00150760 legacy=0x7f7d0015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d00071a30 con 0x7f7d0015da10 2024-01-16T15:04:02.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7ce8001670 con 0x7f7d0015da10 2024-01-16T15:04:02.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 >> v1:172.21.15.141:6789/0 conn(0x7f7d0015da10 legacy=0x7f7d000b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ce0005b80 0x7f7ce0005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ce00069d0 0x7f7ce0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7ce000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00071750 con 0x7f7ce0007390 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce000e290 con 0x7f7ce0005b80 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.510+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce000e450 con 0x7f7ce00069d0 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.511+0000 7f7cff400640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7ce000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.511+0000 7f7cff400640 1 -- 172.21.15.141:0/221137029 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ce00069d0 msgr2=0x7f7ce0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.511+0000 7f7cff400640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ce00069d0 0x7f7ce0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.511+0000 7f7cff400640 1 -- 172.21.15.141:0/221137029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ce0005b80 msgr2=0x7f7ce0005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.511+0000 7f7cff400640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ce0005b80 0x7f7ce0005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.511+0000 7f7cff400640 1 -- 172.21.15.141:0/221137029 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d00071a30 con 0x7f7ce0007390 2024-01-16T15:04:02.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.512+0000 7f7cff400640 1 -- 172.21.15.141:0/221137029 reap_dead start 2024-01-16T15:04:02.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.513+0000 7f7cff400640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7ce000db90 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8007360 tx=0x7f7ce0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5bf5e4756abaff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.513+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf802ccf0 con 0x7f7ce0007390 2024-01-16T15:04:02.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.513+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf802ce90 con 0x7f7ce0007390 2024-01-16T15:04:02.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.513+0000 7f7cfe000640 1 -- 172.21.15.141:0/221137029 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8003bd0 con 0x7f7ce0007390 2024-01-16T15:04:02.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.514+0000 7f7d05a00640 1 -- 172.21.15.141:0/221137029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 msgr2=0x7f7ce000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.514+0000 7f7d05a00640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7ce000db90 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8007360 tx=0x7f7ce0007a00 comp rx=0 tx=0).stop 2024-01-16T15:04:02.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.515+0000 7f7d05a00640 1 -- 172.21.15.141:0/221137029 shutdown_connections 2024-01-16T15:04:02.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.515+0000 7f7d05a00640 1 --2- 172.21.15.141:0/221137029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7ce000db90 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.515+0000 7f7d05a00640 1 -- 172.21.15.141:0/221137029 >> 172.21.15.141:0/221137029 conn(0x7f7d0006a800 msgr2=0x7f7d0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:02.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.515+0000 7f7d05a00640 1 -- 172.21.15.141:0/221137029 shutdown_connections 2024-01-16T15:04:02.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.515+0000 7f7d05a00640 1 -- 172.21.15.141:0/221137029 wait complete. 2024-01-16T15:04:02.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.518+0000 7f7d05a00640 1 Processor -- start 2024-01-16T15:04:02.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.518+0000 7f7d05a00640 1 -- start start 2024-01-16T15:04:02.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.519+0000 7f7d05a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7d0013e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.519+0000 7f7d05a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0013fea0 0x7f7d0014c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.519+0000 7f7d05a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d0014ccc0 0x7f7d001e3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.519+0000 7f7d05a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0014c2f0 con 0x7f7ce0007390 2024-01-16T15:04:02.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.519+0000 7f7d05a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0014ba30 con 0x7f7d0013fea0 2024-01-16T15:04:02.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.519+0000 7f7d05a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0014bca0 con 0x7f7d0014ccc0 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.520+0000 7f7cfea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7d0013e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.520+0000 7f7cfea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7d0013e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53474/0 (socket says 172.21.15.141:53474) 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.520+0000 7f7cfea00640 1 -- 172.21.15.141:0/2933524007 learned_addr learned my addr 172.21.15.141:0/2933524007 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.520+0000 7f7cf7e00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0013fea0 0x7f7d0014c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.520+0000 7f7cfea00640 1 -- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d0014ccc0 msgr2=0x7f7d001e3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.520+0000 7f7cff400640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d0014ccc0 0x7f7d001e3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.520+0000 7f7cfea00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d0014ccc0 0x7f7d001e3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7cfea00640 1 -- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0013fea0 msgr2=0x7f7d0014c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7cfea00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0013fea0 0x7f7d0014c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7cfea00640 1 -- 172.21.15.141:0/2933524007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0014bf80 con 0x7f7ce0007390 2024-01-16T15:04:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7cfea00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7d0013e180 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000c4b0 tx=0x7f7cf000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7cf6a00640 1 -- 172.21.15.141:0/2933524007 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0012830 con 0x7f7ce0007390 2024-01-16T15:04:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7cf6a00640 1 -- 172.21.15.141:0/2933524007 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf0012990 con 0x7f7ce0007390 2024-01-16T15:04:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7cf6a00640 1 -- 172.21.15.141:0/2933524007 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0012d40 con 0x7f7ce0007390 2024-01-16T15:04:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.521+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d00149330 con 0x7f7ce0007390 2024-01-16T15:04:02.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.522+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d001404e0 con 0x7f7ce0007390 2024-01-16T15:04:02.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.524+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d000675d0 con 0x7f7ce0007390 2024-01-16T15:04:02.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.525+0000 7f7cf6a00640 1 -- 172.21.15.141:0/2933524007 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f7cf001b400 con 0x7f7ce0007390 2024-01-16T15:04:02.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.526+0000 7f7cf6a00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cd806bb30 0x7f7cd806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:02.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.526+0000 7f7cf6a00640 1 -- 172.21.15.141:0/2933524007 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cf00884d0 con 0x7f7ce0007390 2024-01-16T15:04:02.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.527+0000 7f7cf7e00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cd806bb30 0x7f7cd806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:02.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.531+0000 7f7cf6a00640 1 -- 172.21.15.141:0/2933524007 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7cf0054b90 con 0x7f7ce0007390 2024-01-16T15:04:02.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.533+0000 7f7cf7e00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cd806bb30 0x7f7cd806dff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8000fb0 tx=0x7f7ce8001660 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:02.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.928+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7d0014c110 con 0x7f7ce0007390 2024-01-16T15:04:02.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.932+0000 7f7cf6a00640 1 -- 172.21.15.141:0/2933524007 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2429 (secure 0 0 0) 0x7f7cf0016070 con 0x7f7ce0007390 2024-01-16T15:04:02.914 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:02.914 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:02.914 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:04:02.914 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:04:02.914 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:02.914 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:04:02.914 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:04:02.915 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 435, 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:04:02.916 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:04:02.917 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 2421, 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 10009076404, 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 19642744832, 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 753451368448, 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:02.918 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 578941247, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 565011730, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 138, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 147 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:04:02.919 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:04:02.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:02.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (12s)\n [======......................] (remaining: 38s)", 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.23810000717639923, 2024-01-16T15:04:02.922 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.41667500138282776, 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:02.923 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:02.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.939+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cd806bb30 msgr2=0x7f7cd806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.939+0000 7f7d05a00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cd806bb30 0x7f7cd806dff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8000fb0 tx=0x7f7ce8001660 comp rx=0 tx=0).stop 2024-01-16T15:04:02.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.939+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 msgr2=0x7f7d0013e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.939+0000 7f7d05a00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7d0013e180 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000c4b0 tx=0x7f7cf000c970 comp rx=0 tx=0).stop 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 shutdown_connections 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cd806bb30 0x7f7cd806dff0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d0014ccc0 0x7f7d001e3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0013fea0 0x7f7d0014c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 --2- 172.21.15.141:0/2933524007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ce0007390 0x7f7d0013e180 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 >> 172.21.15.141:0/2933524007 conn(0x7f7d0006a800 msgr2=0x7f7d000b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 shutdown_connections 2024-01-16T15:04:02.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:02.941+0000 7f7d05a00640 1 -- 172.21.15.141:0/2933524007 wait complete. 2024-01-16T15:04:03.953 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:03.962 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:04.963 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:05.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.267+0000 7f968fe00640 1 Processor -- start 2024-01-16T15:04:05.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.268+0000 7f968fe00640 1 -- start start 2024-01-16T15:04:05.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.269+0000 7f968fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9688144aa0 0x7f968806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.269+0000 7f968fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9688144e80 con 0x7f968806f2e0 2024-01-16T15:04:05.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.269+0000 7f968fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968815bcc0 con 0x7f96880cd9f0 2024-01-16T15:04:05.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.269+0000 7f968fe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968806f780 con 0x7f9688144aa0 2024-01-16T15:04:05.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.270+0000 7f968d400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9688144aa0 0x7f968806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.270+0000 7f968d400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9688144aa0 0x7f968806eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33160/0 (socket says 172.21.15.141:33160) 2024-01-16T15:04:05.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.270+0000 7f968d400640 1 -- 172.21.15.141:0/1664765582 learned_addr learned my addr 172.21.15.141:0/1664765582 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:05.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.270+0000 7f968d400640 1 -- 172.21.15.141:0/1664765582 >> v1:172.21.15.188:6789/0 conn(0x7f96880cd9f0 legacy=0x7f96880b5450 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:04:05.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.270+0000 7f968d400640 1 -- 172.21.15.141:0/1664765582 >> v1:172.21.15.141:6789/0 conn(0x7f968806f2e0 legacy=0x7f968815c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.271+0000 7f968d400640 1 -- 172.21.15.141:0/1664765582 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968806fa60 con 0x7f9688144aa0 2024-01-16T15:04:05.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.270+0000 7f9687e00640 1 -- 172.21.15.141:0/1664765582 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458594492 0 0) 0x7f9680001490 con 0x7f968806f2e0 2024-01-16T15:04:05.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.272+0000 7f968d400640 1 --2- 172.21.15.141:0/1664765582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9688144aa0 0x7f968806eda0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9678009870 tx=0x7f967802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd31b19d3e441159 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.272+0000 7f9687e00640 1 -- 172.21.15.141:0/1664765582 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678030880 con 0x7f9688144aa0 2024-01-16T15:04:05.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.272+0000 7f9687e00640 1 -- 172.21.15.141:0/1664765582 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96780309e0 con 0x7f9688144aa0 2024-01-16T15:04:05.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.272+0000 7f9687e00640 1 -- 172.21.15.141:0/1664765582 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678030d70 con 0x7f9688144aa0 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.273+0000 7f968fe00640 1 -- 172.21.15.141:0/1664765582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9688144aa0 msgr2=0x7f968806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.273+0000 7f968fe00640 1 --2- 172.21.15.141:0/1664765582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9688144aa0 0x7f968806eda0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9678009870 tx=0x7f967802b9c0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.273+0000 7f968fe00640 1 -- 172.21.15.141:0/1664765582 shutdown_connections 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.273+0000 7f968fe00640 1 --2- 172.21.15.141:0/1664765582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9688144aa0 0x7f968806eda0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.273+0000 7f968fe00640 1 -- 172.21.15.141:0/1664765582 >> 172.21.15.141:0/1664765582 conn(0x7f968806d3e0 msgr2=0x7f968806d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.274+0000 7f968fe00640 1 -- 172.21.15.141:0/1664765582 shutdown_connections 2024-01-16T15:04:05.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.274+0000 7f968fe00640 1 -- 172.21.15.141:0/1664765582 wait complete. 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.276+0000 7f968fe00640 1 Processor -- start 2024-01-16T15:04:05.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.277+0000 7f968fe00640 1 -- start start 2024-01-16T15:04:05.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.279+0000 7f968fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f968806f2e0 0x7f96881f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.279+0000 7f968fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96880cd9f0 0x7f968815dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.279+0000 7f968fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9688144aa0 0x7f968815e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.279+0000 7f968fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96880ac860 con 0x7f968806f2e0 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.279+0000 7f968fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96880ab7c0 con 0x7f96880cd9f0 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.279+0000 7f968fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96880abaa0 con 0x7f9688144aa0 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.280+0000 7f968de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9688144aa0 0x7f968815e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.280+0000 7f968de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9688144aa0 0x7f968815e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:49270/0 (socket says 172.21.15.141:49270) 2024-01-16T15:04:05.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.280+0000 7f968de00640 1 -- 172.21.15.141:0/3553341270 learned_addr learned my addr 172.21.15.141:0/3553341270 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:05.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.281+0000 7f968d400640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f968806f2e0 0x7f96881f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.281+0000 7f968ca00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96880cd9f0 0x7f968815dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.281+0000 7f968de00640 1 -- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96880cd9f0 msgr2=0x7f968815dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.282+0000 7f968de00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96880cd9f0 0x7f968815dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.282+0000 7f968de00640 1 -- 172.21.15.141:0/3553341270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f968806f2e0 msgr2=0x7f96881f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.282+0000 7f968de00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f968806f2e0 0x7f96881f71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.282+0000 7f968de00640 1 -- 172.21.15.141:0/3553341270 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780094d0 con 0x7f9688144aa0 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.286+0000 7f968de00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9688144aa0 0x7f968815e190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f968000c7c0 tx=0x7f968000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.286+0000 7f9685600640 1 -- 172.21.15.141:0/3553341270 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680003da0 con 0x7f9688144aa0 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.286+0000 7f9685600640 1 -- 172.21.15.141:0/3553341270 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9680005070 con 0x7f9688144aa0 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.286+0000 7f9685600640 1 -- 172.21.15.141:0/3553341270 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680005400 con 0x7f9688144aa0 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.287+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96880abd80 con 0x7f9688144aa0 2024-01-16T15:04:05.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.287+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96880ac340 con 0x7f9688144aa0 2024-01-16T15:04:05.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.289+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96880ce660 con 0x7f9688144aa0 2024-01-16T15:04:05.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.291+0000 7f9685600640 1 -- 172.21.15.141:0/3553341270 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f9680005560 con 0x7f9688144aa0 2024-01-16T15:04:05.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.292+0000 7f9685600640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f966006bb80 0x7f966006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.293+0000 7f9685600640 1 -- 172.21.15.141:0/3553341270 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9680088170 con 0x7f9688144aa0 2024-01-16T15:04:05.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.294+0000 7f968d400640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f966006bb80 0x7f966006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.297+0000 7f968d400640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f966006bb80 0x7f966006e040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9678036000 tx=0x7f967803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.299+0000 7f9685600640 1 -- 172.21.15.141:0/3553341270 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96800548a0 con 0x7f9688144aa0 2024-01-16T15:04:05.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.727+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9688144e80 con 0x7f9688144aa0 2024-01-16T15:04:05.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.728+0000 7f9685600640 1 -- 172.21.15.141:0/3553341270 <== mon.2 v2:172.21.15.188:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2410 (secure 0 0 0) 0x7f9688144e80 con 0x7f9688144aa0 2024-01-16T15:04:05.712 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:05.712 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:04:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 438, 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:04:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:04:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 2421, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 10009076404, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 19642744832, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 753451368448, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 400612242, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 392251951, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 95, 2024-01-16T15:04:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 101 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:04:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 42.857% (19s)\n [============................] (remaining: 25s)", 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.4285714328289032, 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:04:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (19s)\n [==============..............] (remaining: 19s)", 2024-01-16T15:04:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.5, 2024-01-16T15:04:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:05.720 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:05.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.733+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f966006bb80 msgr2=0x7f966006e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.733+0000 7f968fe00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f966006bb80 0x7f966006e040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9678036000 tx=0x7f967803b040 comp rx=0 tx=0).stop 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.733+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9688144aa0 msgr2=0x7f968815e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.733+0000 7f968fe00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9688144aa0 0x7f968815e190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f968000c7c0 tx=0x7f968000cc80 comp rx=0 tx=0).stop 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 shutdown_connections 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f966006bb80 0x7f966006e040 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9688144aa0 0x7f968815e190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96880cd9f0 0x7f968815dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 --2- 172.21.15.141:0/3553341270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f968806f2e0 0x7f96881f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 >> 172.21.15.141:0/3553341270 conn(0x7f968806d3e0 msgr2=0x7f96880afdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 shutdown_connections 2024-01-16T15:04:05.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:05.734+0000 7f968fe00640 1 -- 172.21.15.141:0/3553341270 wait complete. 2024-01-16T15:04:07.754 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:07.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.980+0000 7fb862c00640 1 Processor -- start 2024-01-16T15:04:07.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.980+0000 7fb862c00640 1 -- start start 2024-01-16T15:04:07.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb862c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb85c150760 0x7fb85c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb862c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c15d300 0x7fb85c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb862c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c15dc80 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb862c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c0b3770 con 0x7fb85c150760 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb862c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c071190 con 0x7fb85c15d300 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb85be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb85c150760 0x7fb85c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb85be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb85c150760 0x7fb85c072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33182/0 (socket says 172.21.15.141:33182) 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.982+0000 7fb85be00640 1 -- 172.21.15.141:0/455862099 learned_addr learned my addr 172.21.15.141:0/455862099 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.983+0000 7fb853200640 1 --2- 172.21.15.141:0/455862099 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c15d300 0x7fb85c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.983+0000 7fb85be00640 1 -- 172.21.15.141:0/455862099 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c15d300 msgr2=0x7fb85c15d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.983+0000 7fb85be00640 1 --2- 172.21.15.141:0/455862099 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c15d300 0x7fb85c15d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.983+0000 7fb85be00640 1 -- 172.21.15.141:0/455862099 >> v1:172.21.15.188:6789/0 conn(0x7fb85c0b2bd0 legacy=0x7fb85c0b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.983+0000 7fb85be00640 1 -- 172.21.15.141:0/455862099 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c071470 con 0x7fb85c150760 2024-01-16T15:04:07.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.983+0000 7fb853200640 1 --2- 172.21.15.141:0/455862099 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c15d300 0x7fb85c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:07.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.984+0000 7fb85be00640 1 --2- 172.21.15.141:0/455862099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c150760 0x7fb85c072230 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00aa10 tx=0x7fb84c02cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137386f0bd72aa45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.984+0000 7fb85b400640 1 -- 172.21.15.141:0/455862099 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c02db80 con 0x7fb85c150760 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.984+0000 7fb85b400640 1 -- 172.21.15.141:0/455862099 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c02dce0 con 0x7fb85c150760 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.984+0000 7fb85b400640 1 -- 172.21.15.141:0/455862099 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c035660 con 0x7fb85c150760 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.984+0000 7fb862c00640 1 -- 172.21.15.141:0/455862099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c150760 msgr2=0x7fb85c072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.985+0000 7fb862c00640 1 --2- 172.21.15.141:0/455862099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c150760 0x7fb85c072230 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00aa10 tx=0x7fb84c02cdd0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.985+0000 7fb862c00640 1 -- 172.21.15.141:0/455862099 shutdown_connections 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.985+0000 7fb862c00640 1 --2- 172.21.15.141:0/455862099 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c15d300 0x7fb85c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.985+0000 7fb862c00640 1 --2- 172.21.15.141:0/455862099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c150760 0x7fb85c072230 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.985+0000 7fb862c00640 1 -- 172.21.15.141:0/455862099 >> 172.21.15.141:0/455862099 conn(0x7fb85c06a800 msgr2=0x7fb85c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.985+0000 7fb862c00640 1 -- 172.21.15.141:0/455862099 shutdown_connections 2024-01-16T15:04:07.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.985+0000 7fb862c00640 1 -- 172.21.15.141:0/455862099 wait complete. 2024-01-16T15:04:07.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.986+0000 7fb862c00640 1 Processor -- start 2024-01-16T15:04:07.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.987+0000 7fb862c00640 1 -- start start 2024-01-16T15:04:07.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.987+0000 7fb862c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c0b2bd0 0x7fb85c0aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.987+0000 7fb862c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c150760 0x7fb85c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.987+0000 7fb862c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb85c15d300 0x7fb85c15bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb862c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c157de0 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb862c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c158080 con 0x7fb85c15d300 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb862c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c158360 con 0x7fb85c150760 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c0b2bd0 0x7fb85c0aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c0b2bd0 0x7fb85c0aaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33188/0 (socket says 172.21.15.141:33188) 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 -- 172.21.15.141:0/3560027966 learned_addr learned my addr 172.21.15.141:0/3560027966 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 -- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c150760 msgr2=0x7fb85c1578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c150760 0x7fb85c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 -- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb85c15d300 msgr2=0x7fb85c15bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb85c15d300 0x7fb85c15bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 -- 172.21.15.141:0/3560027966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c00a670 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.988+0000 7fb85be00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c0b2bd0 0x7fb85c0aaf10 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00a1c0 tx=0x7fb84c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.989+0000 7fb858c00640 1 -- 172.21.15.141:0/3560027966 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c040650 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.989+0000 7fb858c00640 1 -- 172.21.15.141:0/3560027966 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c0407d0 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.989+0000 7fb858c00640 1 -- 172.21.15.141:0/3560027966 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c040b80 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.989+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c158640 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.989+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c0a84c0 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.990+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c0675d0 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.991+0000 7fb858c00640 1 -- 172.21.15.141:0/3560027966 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb84c03c070 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.992+0000 7fb858c00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb83406bb30 0x7fb83406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:07.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.992+0000 7fb858c00640 1 -- 172.21.15.141:0/3560027966 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb84c0aaee0 con 0x7fb85c0b2bd0 2024-01-16T15:04:07.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.992+0000 7fb853200640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb83406bb30 0x7fb83406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:07.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.993+0000 7fb853200640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb83406bb30 0x7fb83406dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb844006fd0 tx=0x7fb844008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:07.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:07.996+0000 7fb858c00640 1 -- 172.21.15.141:0/3560027966 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb84c0775d0 con 0x7fb85c0b2bd0 2024-01-16T15:04:08.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.256+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb85c06b3f0 con 0x7fb85c0b2bd0 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.257+0000 7fb858c00640 1 -- 172.21.15.141:0/3560027966 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7fb84c03a990 con 0x7fb85c0b2bd0 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:04:08.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 440, 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:08.238 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.239 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 3165, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 13104659677, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 26240118784, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 746853994496, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 651758405, 2024-01-16T15:04:08.240 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 650050972, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 156, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 166 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:04:08.241 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:04:08.242 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 47.62% (21s)\n [=============...............] (remaining: 24s)", 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.47620001435279846, 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 66.67% (21s)\n [==================..........] (remaining: 10s)", 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.66670000553131104, 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:08.243 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:08.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.259+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb83406bb30 msgr2=0x7fb83406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.259+0000 7fb862c00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb83406bb30 0x7fb83406dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb844006fd0 tx=0x7fb844008040 comp rx=0 tx=0).stop 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.259+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c0b2bd0 msgr2=0x7fb85c0aaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.259+0000 7fb862c00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c0b2bd0 0x7fb85c0aaf10 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00a1c0 tx=0x7fb84c004730 comp rx=0 tx=0).stop 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 shutdown_connections 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb83406bb30 0x7fb83406dff0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb85c15d300 0x7fb85c15bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb85c150760 0x7fb85c1578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 --2- 172.21.15.141:0/3560027966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb85c0b2bd0 0x7fb85c0aaf10 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 >> 172.21.15.141:0/3560027966 conn(0x7fb85c06a800 msgr2=0x7fb85c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 shutdown_connections 2024-01-16T15:04:08.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:08.260+0000 7fb862c00640 1 -- 172.21.15.141:0/3560027966 wait complete. 2024-01-16T15:04:10.258 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.545+0000 7fbb09800640 1 Processor -- start 2024-01-16T15:04:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.546+0000 7fbb09800640 1 -- start start 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb09800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb0415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb09800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb040cf290 0x7fbb040cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb09800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb04070c40 con 0x7fbb040b3ae0 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb09800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb04071450 con 0x7fbb040cf290 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb09800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040cfc60 con 0x7fbb040ae5a0 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb03400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb040cf290 0x7fbb040cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb02000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbb040b3ae0 0x7fbb0406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47468/0 (socket says 172.21.15.141:47468) 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb02000640 1 -- 172.21.15.141:0/632624465 learned_addr learned my addr 172.21.15.141:0/632624465 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.547+0000 7fbb02a00640 1 --2- 172.21.15.141:0/632624465 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb0415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.548+0000 7fbb02a00640 1 -- 172.21.15.141:0/632624465 >> v2:172.21.15.141:3300/0 conn(0x7fbb040cf290 msgr2=0x7fbb040cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.548+0000 7fbb02a00640 1 --2- 172.21.15.141:0/632624465 >> v2:172.21.15.141:3300/0 conn(0x7fbb040cf290 0x7fbb040cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.548+0000 7fbb02a00640 1 -- 172.21.15.141:0/632624465 >> v1:172.21.15.141:6789/0 conn(0x7fbb040b3ae0 legacy=0x7fbb0406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.548+0000 7fbb02a00640 1 -- 172.21.15.141:0/632624465 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0406c470 con 0x7fbb040ae5a0 2024-01-16T15:04:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.549+0000 7fbb02a00640 1 --2- 172.21.15.141:0/632624465 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb0415e430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40097a0 tx=0x7fbaf402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e95bdc7d768919d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.549+0000 7fbb01600640 1 -- 172.21.15.141:0/632624465 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4030800 con 0x7fbb040ae5a0 2024-01-16T15:04:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.549+0000 7fbb01600640 1 -- 172.21.15.141:0/632624465 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf4030960 con 0x7fbb040ae5a0 2024-01-16T15:04:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.549+0000 7fbb01600640 1 -- 172.21.15.141:0/632624465 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4030cf0 con 0x7fbb040ae5a0 2024-01-16T15:04:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 -- 172.21.15.141:0/632624465 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 msgr2=0x7fbb0415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 --2- 172.21.15.141:0/632624465 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb0415e430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40097a0 tx=0x7fbaf402b940 comp rx=0 tx=0).stop 2024-01-16T15:04:10.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 -- 172.21.15.141:0/632624465 shutdown_connections 2024-01-16T15:04:10.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 --2- 172.21.15.141:0/632624465 >> v2:172.21.15.141:3300/0 conn(0x7fbb040cf290 0x7fbb040cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 --2- 172.21.15.141:0/632624465 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb0415e430 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 -- 172.21.15.141:0/632624465 >> 172.21.15.141:0/632624465 conn(0x7fbb0406a800 msgr2=0x7fbb0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:10.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 -- 172.21.15.141:0/632624465 shutdown_connections 2024-01-16T15:04:10.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.550+0000 7fbb09800640 1 -- 172.21.15.141:0/632624465 wait complete. 2024-01-16T15:04:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.552+0000 7fbb09800640 1 Processor -- start 2024-01-16T15:04:10.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.552+0000 7fbb09800640 1 -- start start 2024-01-16T15:04:10.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.553+0000 7fbb09800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb04158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.553+0000 7fbb09800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb040b3ae0 0x7fbb0414f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.553+0000 7fbb09800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb040cf290 0x7fbb0415fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.553+0000 7fbb09800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040b0330 con 0x7fbb040b3ae0 2024-01-16T15:04:10.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.553+0000 7fbb09800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb04158d80 con 0x7fbb040cf290 2024-01-16T15:04:10.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.553+0000 7fbb09800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb04153670 con 0x7fbb040ae5a0 2024-01-16T15:04:10.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb040b3ae0 0x7fbb0414f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb04158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb040b3ae0 0x7fbb0414f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33204/0 (socket says 172.21.15.141:33204) 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 -- 172.21.15.141:0/4185832606 learned_addr learned my addr 172.21.15.141:0/4185832606 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb03400640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb040cf290 0x7fbb0415fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 -- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 msgr2=0x7fbb04158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb04158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 -- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb040cf290 msgr2=0x7fbb0415fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb040cf290 0x7fbb0415fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 -- 172.21.15.141:0/4185832606 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf4009400 con 0x7fbb040b3ae0 2024-01-16T15:04:10.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.554+0000 7fbb02000640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb040b3ae0 0x7fbb0414f980 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00c4d0 tx=0x7fbaec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.555+0000 7fbafaa00640 1 -- 172.21.15.141:0/4185832606 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec005ce0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.555+0000 7fbafaa00640 1 -- 172.21.15.141:0/4185832606 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec005e40 con 0x7fbb040b3ae0 2024-01-16T15:04:10.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.555+0000 7fbafaa00640 1 -- 172.21.15.141:0/4185832606 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec007270 con 0x7fbb040b3ae0 2024-01-16T15:04:10.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.555+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb040afd70 con 0x7fbb040b3ae0 2024-01-16T15:04:10.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.555+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb041518a0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.556+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb040675d0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.557+0000 7fbafaa00640 1 -- 172.21.15.141:0/4185832606 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fbaec0073d0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.558+0000 7fbafaa00640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbadc06bb30 0x7fbadc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.558+0000 7fbafaa00640 1 -- 172.21.15.141:0/4185832606 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbaec0879b0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.558+0000 7fbb02a00640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbadc06bb30 0x7fbadc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.559+0000 7fbb02a00640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbadc06bb30 0x7fbadc06dff0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4036000 tx=0x7fbaf4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.563+0000 7fbafaa00640 1 -- 172.21.15.141:0/4185832606 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbaec0541b0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.842+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbb040b1fe0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.844+0000 7fbafaa00640 1 -- 172.21.15.141:0/4185832606 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2404 (secure 0 0 0) 0x7fbaec0577d0 con 0x7fbb040b3ae0 2024-01-16T15:04:10.824 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:10.824 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:10.824 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:04:10.825 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 443, 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 59, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 3762, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 15621386718, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 31152087040, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 741942026240, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 680442960, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 687120733, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 163, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 176 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 38, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 4, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 4, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "a", 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 6360 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 11, 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:03:36.613107+0000", 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "59fbab69-9c69-401e-a549-ca264c7408ee": { 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (25s)\n [================............] (remaining: 19s)", 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.57142859697341919, 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi141.stdout: "76a9c111-faf1-4c71-903e-47efa92b9196": { 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "Subvolume \"subvol_clone66727\" has been 100.0% cloned (25s)\n [============================] ", 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 1, 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbadc06bb30 msgr2=0x7fbadc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbadc06bb30 0x7fbadc06dff0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4036000 tx=0x7fbaf4008030 comp rx=0 tx=0).stop 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb040b3ae0 msgr2=0x7fbb0414f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb040b3ae0 0x7fbb0414f980 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00c4d0 tx=0x7fbaec00c990 comp rx=0 tx=0).stop 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 shutdown_connections 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbadc06bb30 0x7fbadc06dff0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb040cf290 0x7fbb0415fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb040b3ae0 0x7fbb0414f980 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 --2- 172.21.15.141:0/4185832606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb040ae5a0 0x7fbb04158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 >> 172.21.15.141:0/4185832606 conn(0x7fbb0406a800 msgr2=0x7fbb0415d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:10.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 shutdown_connections 2024-01-16T15:04:10.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:10.847+0000 7fbb09800640 1 -- 172.21.15.141:0/4185832606 wait complete. 2024-01-16T15:04:12.847 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:12.847 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66721 2024-01-16T15:04:13.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.093+0000 7f9a82200640 1 Processor -- start 2024-01-16T15:04:13.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.093+0000 7f9a82200640 1 -- start start 2024-01-16T15:04:13.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.095+0000 7f9a82200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9a7c146ee0 0x7f9a7c071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.095+0000 7f9a82200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c1472c0 con 0x7f9a7c06a3e0 2024-01-16T15:04:13.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.095+0000 7f9a82200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c072510 con 0x7f9a7c0cd170 2024-01-16T15:04:13.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.095+0000 7f9a82200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c0cd580 con 0x7f9a7c146ee0 2024-01-16T15:04:13.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.095+0000 7f9a73e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9a7c06a3e0 0x7f9a7c0cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47096/0 (socket says 172.21.15.141:47096) 2024-01-16T15:04:13.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.095+0000 7f9a73e00640 1 -- 172.21.15.141:0/2172846458 learned_addr learned my addr 172.21.15.141:0/2172846458 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:13.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7b400640 1 --2- 172.21.15.141:0/2172846458 >> v2:172.21.15.188:3300/0 conn(0x7f9a7c146ee0 0x7f9a7c071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564485414 0 0) 0x7f9a7c1472c0 con 0x7f9a7c06a3e0 2024-01-16T15:04:13.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a5c003520 con 0x7f9a7c06a3e0 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151324918 0 0) 0x7f9a5c003520 con 0x7f9a7c06a3e0 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 >> v2:172.21.15.188:3300/0 conn(0x7f9a7c146ee0 msgr2=0x7f9a7c071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 --2- 172.21.15.141:0/2172846458 >> v2:172.21.15.188:3300/0 conn(0x7f9a7c146ee0 0x7f9a7c071fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 >> v1:172.21.15.188:6789/0 conn(0x7f9a7c0cd170 legacy=0x7f9a7c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0cd790 con 0x7f9a7c06a3e0 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9a6c001670 con 0x7f9a7c06a3e0 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 >> v1:172.21.15.141:6789/0 conn(0x7f9a7c06a3e0 legacy=0x7f9a7c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a5c0039c0 0x7f9a5c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a5c005180 0x7f9a5c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0cd580 con 0x7f9a5c0047c0 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c00c080 con 0x7f9a5c005180 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c00c240 con 0x7f9a5c0039c0 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.096+0000 7f9a73e00640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.097+0000 7f9a7b400640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a5c0039c0 0x7f9a5c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.097+0000 7f9a73e00640 1 -- 172.21.15.141:0/2172846458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a5c0039c0 msgr2=0x7f9a5c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.097+0000 7f9a73e00640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a5c0039c0 0x7f9a5c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.097+0000 7f9a73e00640 1 -- 172.21.15.141:0/2172846458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a5c005180 msgr2=0x7f9a5c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.097+0000 7f9a73e00640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a5c005180 0x7f9a5c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.097+0000 7f9a73e00640 1 -- 172.21.15.141:0/2172846458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c003520 con 0x7f9a5c0047c0 2024-01-16T15:04:13.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.097+0000 7f9a7be00640 1 -- 172.21.15.141:0/2172846458 reap_dead start 2024-01-16T15:04:13.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.098+0000 7f9a73e00640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a5c004c40 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c0e0 tx=0x7f9a5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec542211ddc4d3c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:13.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.098+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c005ad0 con 0x7f9a5c0047c0 2024-01-16T15:04:13.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.098+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c005c50 con 0x7f9a5c0047c0 2024-01-16T15:04:13.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.098+0000 7f9a7aa00640 1 -- 172.21.15.141:0/2172846458 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c02c7f0 con 0x7f9a5c0047c0 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.099+0000 7f9a82200640 1 -- 172.21.15.141:0/2172846458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 msgr2=0x7f9a5c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.099+0000 7f9a82200640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a5c004c40 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c0e0 tx=0x7f9a5c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.099+0000 7f9a82200640 1 -- 172.21.15.141:0/2172846458 shutdown_connections 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.099+0000 7f9a82200640 1 --2- 172.21.15.141:0/2172846458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a5c004c40 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.099+0000 7f9a82200640 1 -- 172.21.15.141:0/2172846458 >> 172.21.15.141:0/2172846458 conn(0x7f9a7c0cff90 msgr2=0x7f9a7c0d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.099+0000 7f9a82200640 1 -- 172.21.15.141:0/2172846458 shutdown_connections 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.099+0000 7f9a82200640 1 -- 172.21.15.141:0/2172846458 wait complete. 2024-01-16T15:04:13.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.101+0000 7f9a82200640 1 Processor -- start 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.101+0000 7f9a82200640 1 -- start start 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a82200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a7c0a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a82200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c0af200 0x7f9a7c0acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a82200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a7c15fa70 0x7f9a7c15ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a82200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c135330 con 0x7f9a5c0047c0 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a82200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c1355d0 con 0x7f9a7c15fa70 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a82200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c160440 con 0x7f9a7c0af200 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a7b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a7c0a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a7b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a7c0a8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58438/0 (socket says 172.21.15.141:58438) 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.102+0000 7f9a7b400640 1 -- 172.21.15.141:0/3052376387 learned_addr learned my addr 172.21.15.141:0/3052376387 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:13.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a7b400640 1 -- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c0af200 msgr2=0x7f9a7c0acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a7b400640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c0af200 0x7f9a7c0acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a7b400640 1 -- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a7c15fa70 msgr2=0x7f9a7c15ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a7b400640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a7c15fa70 0x7f9a7c15ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a7b400640 1 -- 172.21.15.141:0/3052376387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0a9800 con 0x7f9a5c0047c0 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a7b400640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a7c0a8ed0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9a640107c0 tx=0x7f9a64010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a73400640 1 -- 172.21.15.141:0/3052376387 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400eb60 con 0x7f9a5c0047c0 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a73400640 1 -- 172.21.15.141:0/3052376387 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6400ecc0 con 0x7f9a5c0047c0 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a73400640 1 -- 172.21.15.141:0/3052376387 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64010e70 con 0x7f9a5c0047c0 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c0a9ae0 con 0x7f9a5c0047c0 2024-01-16T15:04:13.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.103+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c0a9410 con 0x7f9a5c0047c0 2024-01-16T15:04:13.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.105+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c0aa000 con 0x7f9a5c0047c0 2024-01-16T15:04:13.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.105+0000 7f9a73400640 1 -- 172.21.15.141:0/3052376387 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f9a6401c400 con 0x7f9a5c0047c0 2024-01-16T15:04:13.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.106+0000 7f9a73400640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a5406bb30 0x7f9a5406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.106+0000 7f9a73400640 1 -- 172.21.15.141:0/3052376387 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a64088ae0 con 0x7f9a5c0047c0 2024-01-16T15:04:13.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.107+0000 7f9a73e00640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a5406bb30 0x7f9a5406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.108+0000 7f9a73e00640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a5406bb30 0x7f9a5406dff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c034000 tx=0x7f9a6c039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:13.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.111+0000 7f9a73400640 1 -- 172.21.15.141:0/3052376387 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a640552e0 con 0x7f9a5c0047c0 2024-01-16T15:04:13.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:13.338+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66721", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a7c0717d0 con 0x7f9a5406bb30 2024-01-16T15:04:14.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.228+0000 7f9a73400640 1 -- 172.21.15.141:0/3052376387 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f9a6c001470 con 0x7f9a5406bb30 2024-01-16T15:04:14.209 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:14.209 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:14.210 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:04:14.210 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:14.210 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:14.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a5406bb30 msgr2=0x7f9a5406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a5406bb30 0x7f9a5406dff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c034000 tx=0x7f9a6c039040 comp rx=0 tx=0).stop 2024-01-16T15:04:14.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 msgr2=0x7f9a7c0a8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a7c0a8ed0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9a640107c0 tx=0x7f9a64010c80 comp rx=0 tx=0).stop 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 shutdown_connections 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a5406bb30 0x7f9a5406dff0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a7c15fa70 0x7f9a7c15ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c0af200 0x7f9a7c0acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 --2- 172.21.15.141:0/3052376387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a5c0047c0 0x7f9a7c0a8ed0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 >> 172.21.15.141:0/3052376387 conn(0x7f9a7c0cff90 msgr2=0x7f9a7c0b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 shutdown_connections 2024-01-16T15:04:14.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.233+0000 7f9a82200640 1 -- 172.21.15.141:0/3052376387 wait complete. 2024-01-16T15:04:14.237 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:14.237 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66722 2024-01-16T15:04:14.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.464+0000 7f5e83c00640 1 Processor -- start 2024-01-16T15:04:14.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.464+0000 7f5e83c00640 1 -- start start 2024-01-16T15:04:14.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.466+0000 7f5e83c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5e7c13e360 0x7f5e7c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.466+0000 7f5e83c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5e7c15f9c0 0x7f5e7c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.466+0000 7f5e83c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7c15c5f0 con 0x7f5e7c13df80 2024-01-16T15:04:14.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.466+0000 7f5e83c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7c06c470 con 0x7f5e7c13e360 2024-01-16T15:04:14.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.466+0000 7f5e83c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7c06c750 con 0x7f5e7c15f9c0 2024-01-16T15:04:14.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.466+0000 7f5e81200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5e7c13df80 0x7f5e7c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47098/0 (socket says 172.21.15.141:47098) 2024-01-16T15:04:14.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.466+0000 7f5e81200640 1 -- 172.21.15.141:0/1597818848 learned_addr learned my addr 172.21.15.141:0/1597818848 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e7be00640 1 --2- 172.21.15.141:0/1597818848 >> v2:172.21.15.141:3300/0 conn(0x7f5e7c13e360 0x7f5e7c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e81c00640 1 --2- 172.21.15.141:0/1597818848 >> v2:172.21.15.188:3300/0 conn(0x7f5e7c15f9c0 0x7f5e7c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e7b400640 1 -- 172.21.15.141:0/1597818848 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547704506 0 0) 0x7f5e7c15c5f0 con 0x7f5e7c13df80 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e7b400640 1 -- 172.21.15.141:0/1597818848 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c003520 con 0x7f5e7c13df80 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e7be00640 1 -- 172.21.15.141:0/1597818848 >> v2:172.21.15.188:3300/0 conn(0x7f5e7c15f9c0 msgr2=0x7f5e7c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e7be00640 1 --2- 172.21.15.141:0/1597818848 >> v2:172.21.15.188:3300/0 conn(0x7f5e7c15f9c0 0x7f5e7c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e7be00640 1 -- 172.21.15.141:0/1597818848 >> v1:172.21.15.141:6789/0 conn(0x7f5e7c13df80 legacy=0x7f5e7c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e7be00640 1 -- 172.21.15.141:0/1597818848 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c06ca30 con 0x7f5e7c13e360 2024-01-16T15:04:14.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.467+0000 7f5e81c00640 1 --2- 172.21.15.141:0/1597818848 >> v2:172.21.15.188:3300/0 conn(0x7f5e7c15f9c0 0x7f5e7c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.468+0000 7f5e7be00640 1 --2- 172.21.15.141:0/1597818848 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13e360 0x7f5e7c0cf290 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0ccae0 tx=0x7f5e6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b0c783db33a710 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.468+0000 7f5e7b400640 1 -- 172.21.15.141:0/1597818848 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64031830 con 0x7f5e7c13e360 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.468+0000 7f5e7b400640 1 -- 172.21.15.141:0/1597818848 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e64031990 con 0x7f5e7c13e360 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.468+0000 7f5e7b400640 1 -- 172.21.15.141:0/1597818848 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64031d20 con 0x7f5e7c13e360 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 -- 172.21.15.141:0/1597818848 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13e360 msgr2=0x7f5e7c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 --2- 172.21.15.141:0/1597818848 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13e360 0x7f5e7c0cf290 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0ccae0 tx=0x7f5e6402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 -- 172.21.15.141:0/1597818848 shutdown_connections 2024-01-16T15:04:14.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 --2- 172.21.15.141:0/1597818848 >> v2:172.21.15.188:3300/0 conn(0x7f5e7c15f9c0 0x7f5e7c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 --2- 172.21.15.141:0/1597818848 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13e360 0x7f5e7c0cf290 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 -- 172.21.15.141:0/1597818848 >> 172.21.15.141:0/1597818848 conn(0x7f5e7c06a800 msgr2=0x7f5e7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:14.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 -- 172.21.15.141:0/1597818848 shutdown_connections 2024-01-16T15:04:14.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.469+0000 7f5e83c00640 1 -- 172.21.15.141:0/1597818848 wait complete. 2024-01-16T15:04:14.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.471+0000 7f5e83c00640 1 Processor -- start 2024-01-16T15:04:14.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.471+0000 7f5e83c00640 1 -- start start 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.472+0000 7f5e83c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13df80 0x7f5e7c152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.472+0000 7f5e83c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e7c13e360 0x7f5e7c071460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.472+0000 7f5e83c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c15f9c0 0x7f5e7c070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.472+0000 7f5e83c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c072200 con 0x7f5e7c13df80 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.472+0000 7f5e83c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c0703f0 con 0x7f5e7c15f9c0 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.472+0000 7f5e83c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c0719a0 con 0x7f5e7c13e360 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.472+0000 7f5e81200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13df80 0x7f5e7c152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13df80 0x7f5e7c152640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58452/0 (socket says 172.21.15.141:58452) 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 -- 172.21.15.141:0/3453326891 learned_addr learned my addr 172.21.15.141:0/3453326891 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e7be00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e7c13e360 0x7f5e7c071460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c15f9c0 0x7f5e7c070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 -- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e7c13e360 msgr2=0x7f5e7c071460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e7c13e360 0x7f5e7c071460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 -- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c15f9c0 msgr2=0x7f5e7c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c15f9c0 0x7f5e7c070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 -- 172.21.15.141:0/3453326891 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e64009400 con 0x7f5e7c13df80 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.473+0000 7f5e81200640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13df80 0x7f5e7c152640 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00c400 tx=0x7f5e6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.474+0000 7f5e78c00640 1 -- 172.21.15.141:0/3453326891 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c005ce0 con 0x7f5e7c13df80 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.474+0000 7f5e78c00640 1 -- 172.21.15.141:0/3453326891 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c005e40 con 0x7f5e7c13df80 2024-01-16T15:04:14.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.474+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7c071c40 con 0x7f5e7c13df80 2024-01-16T15:04:14.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.474+0000 7f5e78c00640 1 -- 172.21.15.141:0/3453326891 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0072d0 con 0x7f5e7c13df80 2024-01-16T15:04:14.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.474+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7c070690 con 0x7f5e7c13df80 2024-01-16T15:04:14.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.475+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7c0675d0 con 0x7f5e7c13df80 2024-01-16T15:04:14.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.476+0000 7f5e78c00640 1 -- 172.21.15.141:0/3453326891 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f5e6c0074b0 con 0x7f5e7c13df80 2024-01-16T15:04:14.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.477+0000 7f5e78c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e5406bb30 0x7f5e5406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.477+0000 7f5e78c00640 1 -- 172.21.15.141:0/3453326891 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e6c087b30 con 0x7f5e7c13df80 2024-01-16T15:04:14.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.477+0000 7f5e7be00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e5406bb30 0x7f5e5406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.479+0000 7f5e7be00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e5406bb30 0x7f5e5406dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5e64036000 tx=0x7f5e6403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.481+0000 7f5e78c00640 1 -- 172.21.15.141:0/3453326891 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e6c0542d0 con 0x7f5e7c13df80 2024-01-16T15:04:14.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:14.717+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66722", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e7c067bf0 con 0x7f5e5406bb30 2024-01-16T15:04:16.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.708+0000 7f5e78c00640 1 -- 172.21.15.141:0/3453326891 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f5e7c067bf0 con 0x7f5e5406bb30 2024-01-16T15:04:16.689 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:04:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:16.689 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:16.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.713+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e5406bb30 msgr2=0x7f5e5406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.713+0000 7f5e83c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e5406bb30 0x7f5e5406dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5e64036000 tx=0x7f5e6403c040 comp rx=0 tx=0).stop 2024-01-16T15:04:16.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.713+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13df80 msgr2=0x7f5e7c152640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.713+0000 7f5e83c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13df80 0x7f5e7c152640 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00c400 tx=0x7f5e6c00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 shutdown_connections 2024-01-16T15:04:16.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5e5406bb30 0x7f5e5406dff0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c15f9c0 0x7f5e7c070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e7c13e360 0x7f5e7c071460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 --2- 172.21.15.141:0/3453326891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e7c13df80 0x7f5e7c152640 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 >> 172.21.15.141:0/3453326891 conn(0x7f5e7c06a800 msgr2=0x7f5e7c0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:16.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 shutdown_connections 2024-01-16T15:04:16.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.714+0000 7f5e83c00640 1 -- 172.21.15.141:0/3453326891 wait complete. 2024-01-16T15:04:16.717 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:16.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66723 2024-01-16T15:04:16.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.993+0000 7f1909e00640 1 Processor -- start 2024-01-16T15:04:16.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.993+0000 7f1909e00640 1 -- start start 2024-01-16T15:04:16.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.995+0000 7f1909e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f190413e360 0x7f19040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.995+0000 7f1909e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190415c5f0 con 0x7f190413df80 2024-01-16T15:04:16.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.995+0000 7f1909e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190406c470 con 0x7f190415f9c0 2024-01-16T15:04:16.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.995+0000 7f1909e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190406c750 con 0x7f190413e360 2024-01-16T15:04:16.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.995+0000 7f1902a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f190413df80 0x7f190415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47110/0 (socket says 172.21.15.141:47110) 2024-01-16T15:04:16.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.995+0000 7f1902a00640 1 -- 172.21.15.141:0/591877362 learned_addr learned my addr 172.21.15.141:0/591877362 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:16.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623737056 0 0) 0x7f190415c5f0 con 0x7f190413df80 2024-01-16T15:04:16.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e4003520 con 0x7f190413df80 2024-01-16T15:04:16.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906480053 0 0) 0x7f18e4003520 con 0x7f190413df80 2024-01-16T15:04:16.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f190413e360 msgr2=0x7f19040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:16.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f190413e360 0x7f19040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 >> v1:172.21.15.188:6789/0 conn(0x7f190415f9c0 legacy=0x7f19040cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190406ca30 con 0x7f190413df80 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f18f4001670 con 0x7f190413df80 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 >> v1:172.21.15.141:6789/0 conn(0x7f190413df80 legacy=0x7f190415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40039a0 0x7f18e4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e40047c0 0x7f18e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f190413e360 0x7f18e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190406c750 con 0x7f18e40047c0 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e4005510 con 0x7f18e40039a0 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.996+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e400bec0 con 0x7f190413e360 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.997+0000 7f18fbe00640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e40047c0 0x7f18e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.997+0000 7f1902a00640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40039a0 0x7f18e4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.997+0000 7f1903400640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f190413e360 0x7f18e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.998+0000 7f18fbe00640 1 -- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f190413e360 msgr2=0x7f18e400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:16.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.998+0000 7f18fbe00640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f190413e360 0x7f18e400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.998+0000 7f18fbe00640 1 -- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40039a0 msgr2=0x7f18e4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:16.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.998+0000 7f18fbe00640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40039a0 0x7f18e4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.998+0000 7f18fbe00640 1 -- 172.21.15.141:0/591877362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e4003520 con 0x7f18e40047c0 2024-01-16T15:04:16.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.999+0000 7f18fbe00640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e40047c0 0x7f18e4004c40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009860 tx=0x7f18e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec45e53e10bb546 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:16.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.999+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec032600 con 0x7f18e40047c0 2024-01-16T15:04:16.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.999+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec02e050 con 0x7f18e40047c0 2024-01-16T15:04:16.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:16.999+0000 7f1902000640 1 -- 172.21.15.141:0/591877362 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0329c0 con 0x7f18e40047c0 2024-01-16T15:04:16.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.000+0000 7f1909e00640 1 -- 172.21.15.141:0/591877362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e40047c0 msgr2=0x7f18e4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.000+0000 7f1909e00640 1 --2- 172.21.15.141:0/591877362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18e40047c0 0x7f18e4004c40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009860 tx=0x7f18e40057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.000+0000 7f1903400640 1 -- 172.21.15.141:0/591877362 reap_dead start 2024-01-16T15:04:16.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.000+0000 7f1909e00640 1 -- 172.21.15.141:0/591877362 shutdown_connections 2024-01-16T15:04:16.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.000+0000 7f1909e00640 1 -- 172.21.15.141:0/591877362 >> 172.21.15.141:0/591877362 conn(0x7f190406a800 msgr2=0x7f190406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:16.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.001+0000 7f1909e00640 1 -- 172.21.15.141:0/591877362 shutdown_connections 2024-01-16T15:04:16.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.001+0000 7f1909e00640 1 -- 172.21.15.141:0/591877362 wait complete. 2024-01-16T15:04:16.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.004+0000 7f1909e00640 1 Processor -- start 2024-01-16T15:04:16.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.004+0000 7f1909e00640 1 -- start start 2024-01-16T15:04:16.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.006+0000 7f1909e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e40047c0 0x7f190414a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.006+0000 7f1909e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19041f28e0 0x7f190413cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.006+0000 7f1909e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f190415a140 0x7f190415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.006+0000 7f1909e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190415ab00 con 0x7f19041f28e0 2024-01-16T15:04:16.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.006+0000 7f1909e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190413d150 con 0x7f190415a140 2024-01-16T15:04:16.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.006+0000 7f1909e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190413d3d0 con 0x7f18e40047c0 2024-01-16T15:04:16.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.006+0000 7f18fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19041f28e0 0x7f190413cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f190415a140 0x7f190415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f190415a140 0x7f190415a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:56746/0 (socket says 172.21.15.141:56746) 2024-01-16T15:04:16.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 -- 172.21.15.141:0/2797625037 learned_addr learned my addr 172.21.15.141:0/2797625037 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:16.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1902a00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e40047c0 0x7f190414a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 -- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e40047c0 msgr2=0x7f190414a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e40047c0 0x7f190414a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 -- 172.21.15.141:0/2797625037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19041f28e0 msgr2=0x7f190413cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19041f28e0 0x7f190413cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f1903400640 1 -- 172.21.15.141:0/2797625037 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190413d6b0 con 0x7f190415a140 2024-01-16T15:04:16.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.007+0000 7f18fbe00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19041f28e0 0x7f190413cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:16.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.008+0000 7f1903400640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f190415a140 0x7f190415a5c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f19040cfea0 tx=0x7f18fc00ea00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:16.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.008+0000 7f18fb400640 1 -- 172.21.15.141:0/2797625037 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc003df0 con 0x7f190415a140 2024-01-16T15:04:16.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.009+0000 7f18fb400640 1 -- 172.21.15.141:0/2797625037 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc013420 con 0x7f190415a140 2024-01-16T15:04:16.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.009+0000 7f18fb400640 1 -- 172.21.15.141:0/2797625037 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc0137d0 con 0x7f190415a140 2024-01-16T15:04:16.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.009+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190413d990 con 0x7f190415a140 2024-01-16T15:04:16.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.009+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1904070970 con 0x7f190415a140 2024-01-16T15:04:16.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.011+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d4005250 con 0x7f190415a140 2024-01-16T15:04:16.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.013+0000 7f18fb400640 1 -- 172.21.15.141:0/2797625037 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f18fc03b020 con 0x7f190415a140 2024-01-16T15:04:16.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.014+0000 7f18fb400640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18dc06bb30 0x7f18dc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.014+0000 7f18fb400640 1 -- 172.21.15.141:0/2797625037 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18fc088340 con 0x7f190415a140 2024-01-16T15:04:16.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.019+0000 7f1902a00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18dc06bb30 0x7f18dc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.020+0000 7f18fb400640 1 -- 172.21.15.141:0/2797625037 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18fc054b20 con 0x7f190415a140 2024-01-16T15:04:17.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.021+0000 7f1902a00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18dc06bb30 0x7f18dc06dff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f18f4005a70 tx=0x7f18f4007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:17.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.383+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66723", "target": ["mon-mgr", ""]}) v1 -- 0x7f18d4000bc0 con 0x7f18dc06bb30 2024-01-16T15:04:17.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.700+0000 7f18fb400640 1 -- 172.21.15.141:0/2797625037 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f18d4000bc0 con 0x7f18dc06bb30 2024-01-16T15:04:17.680 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:04:17.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:17.681 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:17.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.705+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18dc06bb30 msgr2=0x7f18dc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.705+0000 7f1909e00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18dc06bb30 0x7f18dc06dff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f18f4005a70 tx=0x7f18f4007040 comp rx=0 tx=0).stop 2024-01-16T15:04:17.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.705+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f190415a140 msgr2=0x7f190415a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.705+0000 7f1909e00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f190415a140 0x7f190415a5c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f19040cfea0 tx=0x7f18fc00ea00 comp rx=0 tx=0).stop 2024-01-16T15:04:17.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 shutdown_connections 2024-01-16T15:04:17.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18dc06bb30 0x7f18dc06dff0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f190415a140 0x7f190415a5c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19041f28e0 0x7f190413cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 --2- 172.21.15.141:0/2797625037 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18e40047c0 0x7f190414a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 >> 172.21.15.141:0/2797625037 conn(0x7f190406a800 msgr2=0x7f1904154ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:17.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 shutdown_connections 2024-01-16T15:04:17.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:17.707+0000 7f1909e00640 1 -- 172.21.15.141:0/2797625037 wait complete. 2024-01-16T15:04:17.724 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:17.724 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66724 2024-01-16T15:04:17.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.018+0000 7f1ab6e00640 1 Processor -- start 2024-01-16T15:04:17.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.018+0000 7f1ab6e00640 1 -- start start 2024-01-16T15:04:18.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.020+0000 7f1ab6e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1ab014b2c0 0x7f1ab00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.020+0000 7f1ab6e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab00cfbf0 con 0x7f1ab00cf7d0 2024-01-16T15:04:18.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.020+0000 7f1ab6e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab0071690 con 0x7f1ab0071070 2024-01-16T15:04:18.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.020+0000 7f1ab6e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab014b6a0 con 0x7f1ab014b2c0 2024-01-16T15:04:18.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.021+0000 7f1aafe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1ab014b2c0 0x7f1ab00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.021+0000 7f1aaf400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1ab00cf7d0 0x7f1ab0070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47124/0 (socket says 172.21.15.141:47124) 2024-01-16T15:04:18.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.021+0000 7f1aaf400640 1 -- 172.21.15.141:0/434560571 learned_addr learned my addr 172.21.15.141:0/434560571 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:18.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.021+0000 7f1aafe00640 1 -- 172.21.15.141:0/434560571 >> v1:172.21.15.188:6789/0 conn(0x7f1ab0071070 legacy=0x7f1ab00cc160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:04:18.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.021+0000 7f1aafe00640 1 -- 172.21.15.141:0/434560571 >> v1:172.21.15.141:6789/0 conn(0x7f1ab00cf7d0 legacy=0x7f1ab0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.021+0000 7f1aafe00640 1 -- 172.21.15.141:0/434560571 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab0071970 con 0x7f1ab014b2c0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.023+0000 7f1aafe00640 1 --2- 172.21.15.141:0/434560571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 0x7f1ab00cf290 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00097a0 tx=0x7f1aa002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3fc73d19641acc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.024+0000 7f1aaea00640 1 -- 172.21.15.141:0/434560571 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0031830 con 0x7f1ab014b2c0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.024+0000 7f1aaea00640 1 -- 172.21.15.141:0/434560571 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa0031990 con 0x7f1ab014b2c0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.024+0000 7f1aaea00640 1 -- 172.21.15.141:0/434560571 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0031d20 con 0x7f1ab014b2c0 2024-01-16T15:04:18.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.025+0000 7f1ab6e00640 1 -- 172.21.15.141:0/434560571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 msgr2=0x7f1ab00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.025+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/434560571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 0x7f1ab00cf290 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00097a0 tx=0x7f1aa002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.027+0000 7f1ab6e00640 1 -- 172.21.15.141:0/434560571 shutdown_connections 2024-01-16T15:04:18.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.027+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/434560571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 0x7f1ab00cf290 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.027+0000 7f1ab6e00640 1 -- 172.21.15.141:0/434560571 >> 172.21.15.141:0/434560571 conn(0x7f1ab006a800 msgr2=0x7f1ab006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:18.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.028+0000 7f1ab6e00640 1 -- 172.21.15.141:0/434560571 shutdown_connections 2024-01-16T15:04:18.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.029+0000 7f1ab6e00640 1 -- 172.21.15.141:0/434560571 wait complete. 2024-01-16T15:04:18.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.032+0000 7f1ab6e00640 1 Processor -- start 2024-01-16T15:04:18.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.032+0000 7f1ab6e00640 1 -- start start 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.035+0000 7f1ab6e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ab0071070 0x7f1ab01642b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.036+0000 7f1ab6e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ab00cf7d0 0x7f1ab0144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.036+0000 7f1ab6e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 0x7f1ab0154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.036+0000 7f1ab6e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab0156390 con 0x7f1ab0071070 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.036+0000 7f1ab6e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab0155780 con 0x7f1ab014b2c0 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.036+0000 7f1ab6e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab0155a60 con 0x7f1ab00cf7d0 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.037+0000 7f1aafe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ab0071070 0x7f1ab01642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.037+0000 7f1aafe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ab0071070 0x7f1ab01642b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58466/0 (socket says 172.21.15.141:58466) 2024-01-16T15:04:18.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.037+0000 7f1aafe00640 1 -- 172.21.15.141:0/2140819251 learned_addr learned my addr 172.21.15.141:0/2140819251 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:18.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.038+0000 7f1aafe00640 1 -- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ab00cf7d0 msgr2=0x7f1ab0144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.038+0000 7f1aafe00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ab00cf7d0 0x7f1ab0144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.038+0000 7f1aafe00640 1 -- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 msgr2=0x7f1ab0154d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:04:18.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.038+0000 7f1aafe00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 0x7f1ab0154d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.038+0000 7f1aafe00640 1 -- 172.21.15.141:0/2140819251 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa0009400 con 0x7f1ab0071070 2024-01-16T15:04:18.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.039+0000 7f1aafe00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ab0071070 0x7f1ab01642b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0000c00 tx=0x7f1aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.042+0000 7f1aa7e00640 1 -- 172.21.15.141:0/2140819251 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa003aa60 con 0x7f1ab0071070 2024-01-16T15:04:18.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.042+0000 7f1aa7e00640 1 -- 172.21.15.141:0/2140819251 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa003abc0 con 0x7f1ab0071070 2024-01-16T15:04:18.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.043+0000 7f1aa7e00640 1 -- 172.21.15.141:0/2140819251 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa00443f0 con 0x7f1ab0071070 2024-01-16T15:04:18.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.044+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab0155d40 con 0x7f1ab0071070 2024-01-16T15:04:18.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.044+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab0157550 con 0x7f1ab0071070 2024-01-16T15:04:18.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.047+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab00675d0 con 0x7f1ab0071070 2024-01-16T15:04:18.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.048+0000 7f1aa7e00640 1 -- 172.21.15.141:0/2140819251 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f1aa004d450 con 0x7f1ab0071070 2024-01-16T15:04:18.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.049+0000 7f1aa7e00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a8806bb80 0x7f1a8806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.050+0000 7f1aa7e00640 1 -- 172.21.15.141:0/2140819251 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1aa00aa7d0 con 0x7f1ab0071070 2024-01-16T15:04:18.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.050+0000 7f1aaf400640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a8806bb80 0x7f1a8806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.053+0000 7f1aaf400640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a8806bb80 0x7f1a8806e040 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0143fe0 tx=0x7f1a98007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.059+0000 7f1aa7e00640 1 -- 172.21.15.141:0/2140819251 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1aa0076f20 con 0x7f1ab0071070 2024-01-16T15:04:18.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.391+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66724", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ab00cbc40 con 0x7f1a8806bb80 2024-01-16T15:04:18.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.665+0000 7f1aa7e00640 1 -- 172.21.15.141:0/2140819251 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f1a980012b0 con 0x7f1a8806bb80 2024-01-16T15:04:18.645 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:04:18.646 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:18.646 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.669+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a8806bb80 msgr2=0x7f1a8806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.669+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a8806bb80 0x7f1a8806e040 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0143fe0 tx=0x7f1a98007a40 comp rx=0 tx=0).stop 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.669+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ab0071070 msgr2=0x7f1ab01642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.669+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ab0071070 0x7f1ab01642b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0000c00 tx=0x7f1aa00046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 shutdown_connections 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a8806bb80 0x7f1a8806e040 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ab014b2c0 0x7f1ab0154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1ab00cf7d0 0x7f1ab0144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 --2- 172.21.15.141:0/2140819251 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1ab0071070 0x7f1ab01642b0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 >> 172.21.15.141:0/2140819251 conn(0x7f1ab006a800 msgr2=0x7f1ab006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:18.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 shutdown_connections 2024-01-16T15:04:18.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.670+0000 7f1ab6e00640 1 -- 172.21.15.141:0/2140819251 wait complete. 2024-01-16T15:04:18.668 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:18.668 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66725 2024-01-16T15:04:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.874+0000 7f097b200640 1 Processor -- start 2024-01-16T15:04:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.874+0000 7f097b200640 1 -- start start 2024-01-16T15:04:18.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.875+0000 7f097b200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f097415d610 0x7f09740ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.875+0000 7f097b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09740ced60 0x7f09740d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.875+0000 7f097b200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09740d1d20 0x7f09740cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.875+0000 7f097b200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09740d1390 con 0x7f09740d1d20 2024-01-16T15:04:18.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.875+0000 7f097b200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09740d21b0 con 0x7f097415d610 2024-01-16T15:04:18.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.875+0000 7f097b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09740b3400 con 0x7f09740ced60 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09740d1d20 0x7f09740cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f09740d1d20 0x7f09740cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58470/0 (socket says 172.21.15.141:58470) 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 -- 172.21.15.141:0/3005757317 learned_addr learned my addr 172.21.15.141:0/3005757317 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 -- 172.21.15.141:0/3005757317 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09740ced60 msgr2=0x7f09740d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f096be00640 1 --2- 172.21.15.141:0/3005757317 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09740ced60 0x7f09740d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 --2- 172.21.15.141:0/3005757317 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09740ced60 0x7f09740d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 -- 172.21.15.141:0/3005757317 >> v2:172.21.15.188:3300/0 conn(0x7f097415d610 msgr2=0x7f09740ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0973e00640 1 --2- 172.21.15.141:0/3005757317 >> v2:172.21.15.188:3300/0 conn(0x7f097415d610 0x7f09740ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 --2- 172.21.15.141:0/3005757317 >> v2:172.21.15.188:3300/0 conn(0x7f097415d610 0x7f09740ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.876+0000 7f0979200640 1 -- 172.21.15.141:0/3005757317 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09740b27f0 con 0x7f09740d1d20 2024-01-16T15:04:18.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.877+0000 7f0979200640 1 --2- 172.21.15.141:0/3005757317 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740d1d20 0x7f09740cf830 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f09740b5dc0 tx=0x7f096c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fbffb8bcb6ab46 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.878+0000 7f0973400640 1 -- 172.21.15.141:0/3005757317 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c02ca70 con 0x7f09740d1d20 2024-01-16T15:04:18.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.878+0000 7f0973400640 1 -- 172.21.15.141:0/3005757317 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c02cbd0 con 0x7f09740d1d20 2024-01-16T15:04:18.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.878+0000 7f0973400640 1 -- 172.21.15.141:0/3005757317 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c0343f0 con 0x7f09740d1d20 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.878+0000 7f097b200640 1 -- 172.21.15.141:0/3005757317 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740d1d20 msgr2=0x7f09740cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.878+0000 7f097b200640 1 --2- 172.21.15.141:0/3005757317 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740d1d20 0x7f09740cf830 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f09740b5dc0 tx=0x7f096c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.879+0000 7f097b200640 1 -- 172.21.15.141:0/3005757317 shutdown_connections 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.879+0000 7f097b200640 1 --2- 172.21.15.141:0/3005757317 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740d1d20 0x7f09740cf830 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.879+0000 7f097b200640 1 --2- 172.21.15.141:0/3005757317 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f09740ced60 0x7f09740d17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.879+0000 7f097b200640 1 --2- 172.21.15.141:0/3005757317 >> v2:172.21.15.188:3300/0 conn(0x7f097415d610 0x7f09740ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.879+0000 7f097b200640 1 -- 172.21.15.141:0/3005757317 >> 172.21.15.141:0/3005757317 conn(0x7f097409c540 msgr2=0x7f09740d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.879+0000 7f097b200640 1 -- 172.21.15.141:0/3005757317 shutdown_connections 2024-01-16T15:04:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.879+0000 7f097b200640 1 -- 172.21.15.141:0/3005757317 wait complete. 2024-01-16T15:04:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.880+0000 7f097b200640 1 Processor -- start 2024-01-16T15:04:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.881+0000 7f097b200640 1 -- start start 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.881+0000 7f097b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740ced60 0x7f09740ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.881+0000 7f097b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09740d1d20 0x7f09740cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.881+0000 7f097b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f097415d610 0x7f09740b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.881+0000 7f097b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09740cb4c0 con 0x7f09740ced60 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.881+0000 7f097b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09740ca8b0 con 0x7f09740d1d20 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.881+0000 7f097b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09740cab90 con 0x7f097415d610 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740ced60 0x7f09740ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740ced60 0x7f09740ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58472/0 (socket says 172.21.15.141:58472) 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 -- 172.21.15.141:0/1326680595 learned_addr learned my addr 172.21.15.141:0/1326680595 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 -- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f097415d610 msgr2=0x7f09740b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0979200640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f097415d610 0x7f09740b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f097415d610 0x7f09740b73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f096be00640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09740d1d20 0x7f09740cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 -- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09740d1d20 msgr2=0x7f09740cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09740d1d20 0x7f09740cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.882+0000 7f0973e00640 1 -- 172.21.15.141:0/1326680595 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c009b30 con 0x7f09740ced60 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.883+0000 7f0973e00640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740ced60 0x7f09740ca2d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f096400c7f0 tx=0x7f096400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.883+0000 7f0970c00640 1 -- 172.21.15.141:0/1326680595 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964004220 con 0x7f09740ced60 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.883+0000 7f0970c00640 1 -- 172.21.15.141:0/1326680595 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0964004380 con 0x7f09740ced60 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.883+0000 7f0970c00640 1 -- 172.21.15.141:0/1326680595 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964007840 con 0x7f09740ced60 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.883+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09740cae70 con 0x7f09740ced60 2024-01-16T15:04:18.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.883+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09740b7910 con 0x7f09740ced60 2024-01-16T15:04:18.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.884+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09740675d0 con 0x7f09740ced60 2024-01-16T15:04:18.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.885+0000 7f0970c00640 1 -- 172.21.15.141:0/1326680595 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f0964007bb0 con 0x7f09740ced60 2024-01-16T15:04:18.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.886+0000 7f0970c00640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f094c06bb60 0x7f094c06e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.886+0000 7f0970c00640 1 -- 172.21.15.141:0/1326680595 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0964088450 con 0x7f09740ced60 2024-01-16T15:04:18.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.887+0000 7f096be00640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f094c06bb60 0x7f094c06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.888+0000 7f096be00640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f094c06bb60 0x7f094c06e020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09740cd3d0 tx=0x7f095c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:18.891+0000 7f0970c00640 1 -- 172.21.15.141:0/1326680595 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0964054ba0 con 0x7f09740ced60 2024-01-16T15:04:19.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.217+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f09740d0390 con 0x7f094c06bb60 2024-01-16T15:04:19.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.244+0000 7f0970c00640 1 -- 172.21.15.141:0/1326680595 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f09740d0390 con 0x7f094c06bb60 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:19.224 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "1/3", 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:19.225 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:19.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f094c06bb60 msgr2=0x7f094c06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:19.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f094c06bb60 0x7f094c06e020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09740cd3d0 tx=0x7f095c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:19.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740ced60 msgr2=0x7f09740ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:19.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740ced60 0x7f09740ca2d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f096400c7f0 tx=0x7f096400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:04:19.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 shutdown_connections 2024-01-16T15:04:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f094c06bb60 0x7f094c06e020 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f097415d610 0x7f09740b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09740d1d20 0x7f09740cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 --2- 172.21.15.141:0/1326680595 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09740ced60 0x7f09740ca2d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 >> 172.21.15.141:0/1326680595 conn(0x7f097409c540 msgr2=0x7f09740b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 shutdown_connections 2024-01-16T15:04:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:19.248+0000 7f097b200640 1 -- 172.21.15.141:0/1326680595 wait complete. 2024-01-16T15:04:19.254 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:20.255 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66725 2024-01-16T15:04:20.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.547+0000 7f4362400640 1 Processor -- start 2024-01-16T15:04:20.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.547+0000 7f4362400640 1 -- start start 2024-01-16T15:04:20.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.549+0000 7f4362400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 0x7f435c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.549+0000 7f4362400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c06c470 con 0x7f435c0cf9a0 2024-01-16T15:04:20.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.549+0000 7f4362400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c06c750 con 0x7f435c13e360 2024-01-16T15:04:20.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.549+0000 7f4362400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c06ca30 con 0x7f435c13df80 2024-01-16T15:04:20.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.549+0000 7f435be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f435c0cf9a0 0x7f435c0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47128/0 (socket says 172.21.15.141:47128) 2024-01-16T15:04:20.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.549+0000 7f435be00640 1 -- 172.21.15.141:0/3846646901 learned_addr learned my addr 172.21.15.141:0/3846646901 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:20.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.549+0000 7f435b400640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 0x7f435c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429697137 0 0) 0x7f435c06c470 con 0x7f435c0cf9a0 2024-01-16T15:04:20.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c003520 con 0x7f435c0cf9a0 2024-01-16T15:04:20.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289774999 0 0) 0x7f435c06c750 con 0x7f435c13e360 2024-01-16T15:04:20.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c06c470 con 0x7f435c13e360 2024-01-16T15:04:20.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2458081963 0 0) 0x7f433c003520 con 0x7f435c0cf9a0 2024-01-16T15:04:20.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 msgr2=0x7f435c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 0x7f435c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 >> v1:172.21.15.188:6789/0 conn(0x7f435c13e360 legacy=0x7f435c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c06cd10 con 0x7f435c0cf9a0 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f43540019f0 con 0x7f435c0cf9a0 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 >> v1:172.21.15.141:6789/0 conn(0x7f435c0cf9a0 legacy=0x7f435c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f433c003b30 0x7f433c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f433c0047c0 0x7f433c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 0x7f433c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c06ca30 con 0x7f433c003b30 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c00ba30 con 0x7f433c0047c0 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.550+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c00bbf0 con 0x7f435c13df80 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.551+0000 7f435aa00640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f433c0047c0 0x7f433c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.551+0000 7f435be00640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 0x7f433c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.551+0000 7f435aa00640 1 -- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 msgr2=0x7f433c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.551+0000 7f435aa00640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435c13df80 0x7f433c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.551+0000 7f435aa00640 1 -- 172.21.15.141:0/3846646901 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f433c003b30 msgr2=0x7f433c003f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:04:20.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.551+0000 7f435aa00640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f433c003b30 0x7f433c003f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.551+0000 7f435aa00640 1 -- 172.21.15.141:0/3846646901 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c003520 con 0x7f433c0047c0 2024-01-16T15:04:20.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.553+0000 7f435aa00640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f433c0047c0 0x7f433c004c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f43440018d0 tx=0x7f433c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=482d128bed86511a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.554+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43440052d0 con 0x7f433c0047c0 2024-01-16T15:04:20.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.554+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4344005430 con 0x7f433c0047c0 2024-01-16T15:04:20.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.554+0000 7f435a000640 1 -- 172.21.15.141:0/3846646901 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344003810 con 0x7f433c0047c0 2024-01-16T15:04:20.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.555+0000 7f4362400640 1 -- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f433c0047c0 msgr2=0x7f433c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.555+0000 7f4362400640 1 --2- 172.21.15.141:0/3846646901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f433c0047c0 0x7f433c004c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f43440018d0 tx=0x7f433c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.555+0000 7f435be00640 1 -- 172.21.15.141:0/3846646901 reap_dead start 2024-01-16T15:04:20.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.555+0000 7f4362400640 1 -- 172.21.15.141:0/3846646901 shutdown_connections 2024-01-16T15:04:20.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.555+0000 7f4362400640 1 -- 172.21.15.141:0/3846646901 >> 172.21.15.141:0/3846646901 conn(0x7f435c06a800 msgr2=0x7f435c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:20.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.555+0000 7f4362400640 1 -- 172.21.15.141:0/3846646901 shutdown_connections 2024-01-16T15:04:20.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.555+0000 7f4362400640 1 -- 172.21.15.141:0/3846646901 wait complete. 2024-01-16T15:04:20.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.557+0000 7f4362400640 1 Processor -- start 2024-01-16T15:04:20.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.557+0000 7f4362400640 1 -- start start 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f4362400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f433c0047c0 0x7f435c1f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f4362400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f435c0ac1a0 0x7f435c0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f4362400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f435c0ac600 0x7f435c0b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f4362400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c0cc390 con 0x7f435c0ac600 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f4362400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1f72a0 con 0x7f435c0ac1a0 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f4362400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c0abd10 con 0x7f433c0047c0 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f435be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f435c0ac600 0x7f435c0b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f435be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f435c0ac600 0x7f435c0b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58480/0 (socket says 172.21.15.141:58480) 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.559+0000 7f435be00640 1 -- 172.21.15.141:0/3627040915 learned_addr learned my addr 172.21.15.141:0/3627040915 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f435be00640 1 -- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f433c0047c0 msgr2=0x7f435c1f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f435be00640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f433c0047c0 0x7f435c1f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f435be00640 1 -- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f435c0ac1a0 msgr2=0x7f435c0b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f435be00640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f435c0ac1a0 0x7f435c0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f435be00640 1 -- 172.21.15.141:0/3627040915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c0ab100 con 0x7f435c0ac600 2024-01-16T15:04:20.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f435be00640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f435c0ac600 0x7f435c0b3670 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4354002230 tx=0x7f435400a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f4353400640 1 -- 172.21.15.141:0/3627040915 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435400ac30 con 0x7f435c0ac600 2024-01-16T15:04:20.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f4353400640 1 -- 172.21.15.141:0/3627040915 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435400ad90 con 0x7f435c0ac600 2024-01-16T15:04:20.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f4353400640 1 -- 172.21.15.141:0/3627040915 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354013630 con 0x7f435c0ac600 2024-01-16T15:04:20.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.560+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c0ab3e0 con 0x7f435c0ac600 2024-01-16T15:04:20.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.561+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c0ab9a0 con 0x7f435c0ac600 2024-01-16T15:04:20.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.562+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c0abb60 con 0x7f435c0ac600 2024-01-16T15:04:20.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.563+0000 7f4353400640 1 -- 172.21.15.141:0/3627040915 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4354013790 con 0x7f435c0ac600 2024-01-16T15:04:20.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.564+0000 7f4353400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f433406bb60 0x7f433406e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.564+0000 7f4353400640 1 -- 172.21.15.141:0/3627040915 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f435408b680 con 0x7f435c0ac600 2024-01-16T15:04:20.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.565+0000 7f435b400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f433406bb60 0x7f433406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.569+0000 7f4353400640 1 -- 172.21.15.141:0/3627040915 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4354055470 con 0x7f435c0ac600 2024-01-16T15:04:20.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.570+0000 7f435b400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f433406bb60 0x7f433406e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f434c009970 tx=0x7f434c009d70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.909+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f435c06d410 con 0x7f433406bb60 2024-01-16T15:04:20.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.931+0000 7f4353400640 1 -- 172.21.15.141:0/3627040915 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f435c06d410 con 0x7f433406bb60 2024-01-16T15:04:20.910 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:20.910 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:20.910 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:20.910 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:20.910 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:20.910 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "1/3", 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:20.911 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f433406bb60 msgr2=0x7f433406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f433406bb60 0x7f433406e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f434c009970 tx=0x7f434c009d70 comp rx=0 tx=0).stop 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f435c0ac600 msgr2=0x7f435c0b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f435c0ac600 0x7f435c0b3670 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4354002230 tx=0x7f435400a600 comp rx=0 tx=0).stop 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 shutdown_connections 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f433406bb60 0x7f433406e020 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f435c0ac600 0x7f435c0b3670 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f435c0ac1a0 0x7f435c0b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 --2- 172.21.15.141:0/3627040915 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f433c0047c0 0x7f435c1f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.933+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 >> 172.21.15.141:0/3627040915 conn(0x7f435c06a800 msgr2=0x7f435c15e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:20.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.934+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 shutdown_connections 2024-01-16T15:04:20.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:20.934+0000 7f4362400640 1 -- 172.21.15.141:0/3627040915 wait complete. 2024-01-16T15:04:20.936 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:21.936 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66725 2024-01-16T15:04:22.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.297+0000 7f6b06200640 1 Processor -- start 2024-01-16T15:04:22.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.298+0000 7f6b06200640 1 -- start start 2024-01-16T15:04:22.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.301+0000 7f6b06200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00143ae0 0x7f6b000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.301+0000 7f6b06200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000cfc30 con 0x7f6b0015e3b0 2024-01-16T15:04:22.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.301+0000 7f6b06200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000b1ee0 con 0x7f6b000cf7d0 2024-01-16T15:04:22.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.301+0000 7f6b06200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000b2780 con 0x7f6b00143ae0 2024-01-16T15:04:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.301+0000 7f6affe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6b0015e3b0 0x7f6b000702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47140/0 (socket says 172.21.15.141:47140) 2024-01-16T15:04:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.302+0000 7f6affe00640 1 -- 172.21.15.141:0/2657142156 learned_addr learned my addr 172.21.15.141:0/2657142156 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.302+0000 7f6aff400640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00143ae0 0x7f6b000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.302+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082564481 0 0) 0x7f6b000cfc30 con 0x7f6b0015e3b0 2024-01-16T15:04:22.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.302+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae0003520 con 0x7f6b0015e3b0 2024-01-16T15:04:22.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.303+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1114772533 0 0) 0x7f6ae0003520 con 0x7f6b0015e3b0 2024-01-16T15:04:22.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.303+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00143ae0 msgr2=0x7f6b000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:22.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.303+0000 7f6afe000640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00143ae0 0x7f6b000cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.303+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 >> v1:172.21.15.188:6789/0 conn(0x7f6b000cf7d0 legacy=0x7f6b0015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0015e990 con 0x7f6b0015e3b0 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6af80019f0 con 0x7f6b0015e3b0 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 >> v1:172.21.15.141:6789/0 conn(0x7f6b0015e3b0 legacy=0x7f6b000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00143ae0 0x7f6ae00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ae0003f50 0x7f6ae00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae0004d00 0x7f6ae0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000b2780 con 0x7f6ae0003f50 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae0007b70 con 0x7f6ae0004d00 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.304+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae0007d30 con 0x7f6b00143ae0 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.305+0000 7f6affe00640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae0004d00 0x7f6ae0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.306+0000 7f6afea00640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ae0003f50 0x7f6ae00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.307+0000 7f6afea00640 1 -- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00143ae0 msgr2=0x7f6ae00039a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:04:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.307+0000 7f6afea00640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00143ae0 0x7f6ae00039a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.307+0000 7f6afea00640 1 -- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae0004d00 msgr2=0x7f6ae0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.307+0000 7f6afea00640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae0004d00 0x7f6ae0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.307+0000 7f6afea00640 1 -- 172.21.15.141:0/2657142156 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae0003520 con 0x7f6ae0003f50 2024-01-16T15:04:22.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.309+0000 7f6afea00640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ae0003f50 0x7f6ae00047c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8003db0 tx=0x7f6ae00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664bd872e1b0d9c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.309+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8032600 con 0x7f6ae0003f50 2024-01-16T15:04:22.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.310+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae802e050 con 0x7f6ae0003f50 2024-01-16T15:04:22.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.310+0000 7f6afe000640 1 -- 172.21.15.141:0/2657142156 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8032a40 con 0x7f6ae0003f50 2024-01-16T15:04:22.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.311+0000 7f6b06200640 1 -- 172.21.15.141:0/2657142156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ae0003f50 msgr2=0x7f6ae00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.311+0000 7f6b06200640 1 --2- 172.21.15.141:0/2657142156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ae0003f50 0x7f6ae00047c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8003db0 tx=0x7f6ae00057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.311+0000 7f6affe00640 1 -- 172.21.15.141:0/2657142156 reap_dead start 2024-01-16T15:04:22.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.314+0000 7f6b06200640 1 -- 172.21.15.141:0/2657142156 shutdown_connections 2024-01-16T15:04:22.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.314+0000 7f6b06200640 1 -- 172.21.15.141:0/2657142156 >> 172.21.15.141:0/2657142156 conn(0x7f6b0006a800 msgr2=0x7f6b0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:22.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.314+0000 7f6b06200640 1 -- 172.21.15.141:0/2657142156 shutdown_connections 2024-01-16T15:04:22.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.314+0000 7f6b06200640 1 -- 172.21.15.141:0/2657142156 wait complete. 2024-01-16T15:04:22.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.317+0000 7f6b06200640 1 Processor -- start 2024-01-16T15:04:22.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.317+0000 7f6b06200640 1 -- start start 2024-01-16T15:04:22.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6b06200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b00159060 0x7f6b000ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6b06200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00159440 0x7f6b0013d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6b06200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0015cb10 0x7f6b0015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6b06200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0015d500 con 0x7f6b0015cb10 2024-01-16T15:04:22.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6b06200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00159b70 con 0x7f6b00159060 2024-01-16T15:04:22.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6b06200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0013ae40 con 0x7f6b00159440 2024-01-16T15:04:22.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6affe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0015cb10 0x7f6b0015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6affe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0015cb10 0x7f6b0015cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58492/0 (socket says 172.21.15.141:58492) 2024-01-16T15:04:22.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6affe00640 1 -- 172.21.15.141:0/840041064 learned_addr learned my addr 172.21.15.141:0/840041064 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:22.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.318+0000 7f6afea00640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00159440 0x7f6b0013d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.319+0000 7f6affe00640 1 -- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00159440 msgr2=0x7f6b0013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.319+0000 7f6affe00640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00159440 0x7f6b0013d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.319+0000 7f6affe00640 1 -- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b00159060 msgr2=0x7f6b000ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.319+0000 7f6affe00640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b00159060 0x7f6b000ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.319+0000 7f6affe00640 1 -- 172.21.15.141:0/840041064 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0013b120 con 0x7f6b0015cb10 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.319+0000 7f6affe00640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0015cb10 0x7f6b0015cfc0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6b000cfe50 tx=0x7f6af800a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.319+0000 7f6af7400640 1 -- 172.21.15.141:0/840041064 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af80019b0 con 0x7f6b0015cb10 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.320+0000 7f6af7400640 1 -- 172.21.15.141:0/840041064 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8012560 con 0x7f6b0015cb10 2024-01-16T15:04:22.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.320+0000 7f6af7400640 1 -- 172.21.15.141:0/840041064 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af80128f0 con 0x7f6b0015cb10 2024-01-16T15:04:22.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.320+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0013b400 con 0x7f6b0015cb10 2024-01-16T15:04:22.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.320+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0013bc00 con 0x7f6b0015cb10 2024-01-16T15:04:22.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.322+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0013b920 con 0x7f6b0015cb10 2024-01-16T15:04:22.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.323+0000 7f6af7400640 1 -- 172.21.15.141:0/840041064 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6af8012a50 con 0x7f6b0015cb10 2024-01-16T15:04:22.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.324+0000 7f6af7400640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806bb30 0x7f6ad806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.324+0000 7f6af7400640 1 -- 172.21.15.141:0/840041064 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6af80891a0 con 0x7f6b0015cb10 2024-01-16T15:04:22.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.326+0000 7f6aff400640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806bb30 0x7f6ad806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.334+0000 7f6af7400640 1 -- 172.21.15.141:0/840041064 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b0013b920 con 0x7f6b0015cb10 2024-01-16T15:04:22.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.340+0000 7f6aff400640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806bb30 0x7f6ad806dff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b000b1ff0 tx=0x7f6af0009500 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.704+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b0013afb0 con 0x7f6ad806bb30 2024-01-16T15:04:22.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.733+0000 7f6af7400640 1 -- 172.21.15.141:0/840041064 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f6b0013afb0 con 0x7f6ad806bb30 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:04:22.713 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "1/3", 2024-01-16T15:04:22.714 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:22.714 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:22.714 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:22.714 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:22.714 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806bb30 msgr2=0x7f6ad806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806bb30 0x7f6ad806dff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b000b1ff0 tx=0x7f6af0009500 comp rx=0 tx=0).stop 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0015cb10 msgr2=0x7f6b0015cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0015cb10 0x7f6b0015cfc0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6b000cfe50 tx=0x7f6af800a910 comp rx=0 tx=0).stop 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 shutdown_connections 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ad806bb30 0x7f6ad806dff0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b0015cb10 0x7f6b0015cfc0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6b00159440 0x7f6b0013d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 --2- 172.21.15.141:0/840041064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b00159060 0x7f6b000ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 >> 172.21.15.141:0/840041064 conn(0x7f6b0006a800 msgr2=0x7f6b0015b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:22.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 shutdown_connections 2024-01-16T15:04:22.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:22.736+0000 7f6b06200640 1 -- 172.21.15.141:0/840041064 wait complete. 2024-01-16T15:04:22.749 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:23.750 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66725 2024-01-16T15:04:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.050+0000 7f31f9400640 1 Processor -- start 2024-01-16T15:04:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.051+0000 7f31f9400640 1 -- start start 2024-01-16T15:04:24.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.052+0000 7f31f9400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f31f40cfcf0 0x7f31f4160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.052+0000 7f31f9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 0x7f31f4160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.052+0000 7f31f9400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f40d4160 con 0x7f31f40d00d0 2024-01-16T15:04:24.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.052+0000 7f31f9400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f40ceff0 con 0x7f31f40cfcf0 2024-01-16T15:04:24.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.052+0000 7f31f9400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40cf410 con 0x7f31f40d3910 2024-01-16T15:04:24.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.052+0000 7f31f3200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f31f40d00d0 0x7f31f40b2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54230/0 (socket says 172.21.15.141:54230) 2024-01-16T15:04:24.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.052+0000 7f31f3200640 1 -- 172.21.15.141:0/1192705787 learned_addr learned my addr 172.21.15.141:0/1192705787 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31ebe00640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 0x7f31f4160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901687906 0 0) 0x7f31f40d4160 con 0x7f31f40d00d0 2024-01-16T15:04:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d4003520 con 0x7f31f40d00d0 2024-01-16T15:04:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f2800640 1 --2- 172.21.15.141:0/1192705787 >> v2:172.21.15.188:3300/0 conn(0x7f31f40cfcf0 0x7f31f4160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3238235842 0 0) 0x7f31d4003520 con 0x7f31f40d00d0 2024-01-16T15:04:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 msgr2=0x7f31f4160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 0x7f31f4160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 >> v2:172.21.15.188:3300/0 conn(0x7f31f40cfcf0 msgr2=0x7f31f4160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 --2- 172.21.15.141:0/1192705787 >> v2:172.21.15.188:3300/0 conn(0x7f31f40cfcf0 0x7f31f4160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f40b2420 con 0x7f31f40d00d0 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f31ec001670 con 0x7f31f40d00d0 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.053+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 >> v1:172.21.15.141:6789/0 conn(0x7f31f40d00d0 legacy=0x7f31f40b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.054+0000 7f31f1e00640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 0x7f31d40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.054+0000 7f31f1e00640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31d4003f70 0x7f31d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.054+0000 7f31f1e00640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31d4004d00 0x7f31d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.054+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40ceff0 con 0x7f31d4004d00 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.054+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40cf410 con 0x7f31d4003f70 2024-01-16T15:04:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.054+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d400bb10 con 0x7f31f40d3910 2024-01-16T15:04:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.055+0000 7f31f3200640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31d4004d00 0x7f31d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.055+0000 7f31f2800640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 0x7f31d40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.055+0000 7f31ebe00640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31d4003f70 0x7f31d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.056+0000 7f31f2800640 1 -- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31d4003f70 msgr2=0x7f31d40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.056+0000 7f31f2800640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31d4003f70 0x7f31d40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.056+0000 7f31f2800640 1 -- 172.21.15.141:0/1192705787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31d4004d00 msgr2=0x7f31d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.056+0000 7f31f2800640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31d4004d00 0x7f31d4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.056+0000 7f31f2800640 1 -- 172.21.15.141:0/1192705787 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4003520 con 0x7f31f40d3910 2024-01-16T15:04:24.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.061+0000 7f31f2800640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 0x7f31d40039a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f31e4007360 tx=0x7f31d40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b9a68cc5e5bdc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.063+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e402ccf0 con 0x7f31f40d3910 2024-01-16T15:04:24.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.063+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e402ce50 con 0x7f31f40d3910 2024-01-16T15:04:24.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.063+0000 7f31f1e00640 1 -- 172.21.15.141:0/1192705787 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e4034660 con 0x7f31f40d3910 2024-01-16T15:04:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.063+0000 7f31f9400640 1 -- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 msgr2=0x7f31d40039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.063+0000 7f31f9400640 1 --2- 172.21.15.141:0/1192705787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f40d3910 0x7f31d40039a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f31e4007360 tx=0x7f31d40057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.063+0000 7f31f3200640 1 -- 172.21.15.141:0/1192705787 reap_dead start 2024-01-16T15:04:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.064+0000 7f31f9400640 1 -- 172.21.15.141:0/1192705787 shutdown_connections 2024-01-16T15:04:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.064+0000 7f31f9400640 1 -- 172.21.15.141:0/1192705787 >> 172.21.15.141:0/1192705787 conn(0x7f31f409c530 msgr2=0x7f31f40d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.064+0000 7f31f9400640 1 -- 172.21.15.141:0/1192705787 shutdown_connections 2024-01-16T15:04:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.064+0000 7f31f9400640 1 -- 172.21.15.141:0/1192705787 wait complete. 2024-01-16T15:04:24.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.066+0000 7f31f9400640 1 Processor -- start 2024-01-16T15:04:24.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.066+0000 7f31f9400640 1 -- start start 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.069+0000 7f31f9400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31f40d3910 0x7f31f4159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.069+0000 7f31f9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31f4153730 0x7f31f40b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.069+0000 7f31f9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f415dc50 0x7f31f415e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.069+0000 7f31f9400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f415e610 con 0x7f31f4153730 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.069+0000 7f31f9400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f415c480 con 0x7f31f40d3910 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.069+0000 7f31f9400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f415a470 con 0x7f31f415dc50 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31f4153730 0x7f31f40b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31f4153730 0x7f31f40b6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38414/0 (socket says 172.21.15.141:38414) 2024-01-16T15:04:24.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 -- 172.21.15.141:0/2405101501 learned_addr learned my addr 172.21.15.141:0/2405101501 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:24.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31f2800640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31f40d3910 0x7f31f4159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 -- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f415dc50 msgr2=0x7f31f415e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:04:24.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f415dc50 0x7f31f415e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 -- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31f40d3910 msgr2=0x7f31f4159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31f40d3910 0x7f31f4159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.070+0000 7f31ebe00640 1 -- 172.21.15.141:0/2405101501 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f415be30 con 0x7f31f4153730 2024-01-16T15:04:24.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.071+0000 7f31ebe00640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31f4153730 0x7f31f40b6050 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f31dc011960 tx=0x7f31dc011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.073+0000 7f31eaa00640 1 -- 172.21.15.141:0/2405101501 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc00fa90 con 0x7f31f4153730 2024-01-16T15:04:24.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.073+0000 7f31eaa00640 1 -- 172.21.15.141:0/2405101501 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc00fbf0 con 0x7f31f4153730 2024-01-16T15:04:24.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.073+0000 7f31eaa00640 1 -- 172.21.15.141:0/2405101501 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc014440 con 0x7f31f4153730 2024-01-16T15:04:24.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.073+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f415c110 con 0x7f31f4153730 2024-01-16T15:04:24.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.073+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f40c6280 con 0x7f31f4153730 2024-01-16T15:04:24.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.076+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f415bc90 con 0x7f31f4153730 2024-01-16T15:04:24.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.085+0000 7f31eaa00640 1 -- 172.21.15.141:0/2405101501 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f31dc0263f0 con 0x7f31f4153730 2024-01-16T15:04:24.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.086+0000 7f31eaa00640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31cc06bc00 0x7f31cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.086+0000 7f31eaa00640 1 -- 172.21.15.141:0/2405101501 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31dc088f60 con 0x7f31f4153730 2024-01-16T15:04:24.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.087+0000 7f31f2800640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31cc06bc00 0x7f31cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.088+0000 7f31eaa00640 1 -- 172.21.15.141:0/2405101501 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31dc055650 con 0x7f31f4153730 2024-01-16T15:04:24.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.099+0000 7f31f2800640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31cc06bc00 0x7f31cc06e0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f31e4036000 tx=0x7f31e4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.541+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f31f40b5b60 con 0x7f31cc06bc00 2024-01-16T15:04:24.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.574+0000 7f31eaa00640 1 -- 172.21.15.141:0/2405101501 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f31f40b5b60 con 0x7f31cc06bc00 2024-01-16T15:04:24.553 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:24.553 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:24.553 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:24.553 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:24.553 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:24.554 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:24.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31cc06bc00 msgr2=0x7f31cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31cc06bc00 0x7f31cc06e0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f31e4036000 tx=0x7f31e4030000 comp rx=0 tx=0).stop 2024-01-16T15:04:24.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31f4153730 msgr2=0x7f31f40b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31f4153730 0x7f31f40b6050 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f31dc011960 tx=0x7f31dc011e20 comp rx=0 tx=0).stop 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 shutdown_connections 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31cc06bc00 0x7f31cc06e0c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31f415dc50 0x7f31f415e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31f4153730 0x7f31f40b6050 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 --2- 172.21.15.141:0/2405101501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31f40d3910 0x7f31f4159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 >> 172.21.15.141:0/2405101501 conn(0x7f31f409c530 msgr2=0x7f31f406c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 shutdown_connections 2024-01-16T15:04:24.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:24.577+0000 7f31f9400640 1 -- 172.21.15.141:0/2405101501 wait complete. 2024-01-16T15:04:24.580 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:25.581 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66725 2024-01-16T15:04:25.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.910+0000 7fc916800640 1 Processor -- start 2024-01-16T15:04:25.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.911+0000 7fc916800640 1 -- start start 2024-01-16T15:04:25.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.912+0000 7fc916800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc910143ae0 0x7fc9100cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.912+0000 7fc916800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc91015e3b0 0x7fc9100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.912+0000 7fc916800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100cfc30 con 0x7fc9100cf7d0 2024-01-16T15:04:25.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.912+0000 7fc916800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9100b1ee0 con 0x7fc91015e3b0 2024-01-16T15:04:25.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.912+0000 7fc916800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100b2780 con 0x7fc910143ae0 2024-01-16T15:04:25.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.913+0000 7fc90ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc9100cf7d0 0x7fc91015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54242/0 (socket says 172.21.15.141:54242) 2024-01-16T15:04:25.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.913+0000 7fc90ea00640 1 -- 172.21.15.141:0/2507944646 learned_addr learned my addr 172.21.15.141:0/2507944646 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:25.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.913+0000 7fc90f400640 1 --2- 172.21.15.141:0/2507944646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc910143ae0 0x7fc9100cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.913+0000 7fc90e000640 1 -- 172.21.15.141:0/2507944646 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292923299 0 0) 0x7fc9100cfc30 con 0x7fc9100cf7d0 2024-01-16T15:04:25.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.913+0000 7fc90e000640 1 -- 172.21.15.141:0/2507944646 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f0003420 con 0x7fc9100cf7d0 2024-01-16T15:04:25.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.914+0000 7fc90f400640 1 -- 172.21.15.141:0/2507944646 >> v2:172.21.15.188:3300/0 conn(0x7fc91015e3b0 msgr2=0x7fc9100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:25.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.914+0000 7fc90f400640 1 --2- 172.21.15.141:0/2507944646 >> v2:172.21.15.188:3300/0 conn(0x7fc91015e3b0 0x7fc9100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.914+0000 7fc90f400640 1 -- 172.21.15.141:0/2507944646 >> v1:172.21.15.141:6789/0 conn(0x7fc9100cf7d0 legacy=0x7fc91015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.914+0000 7fc90e000640 1 -- 172.21.15.141:0/2507944646 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344666659 0 0) 0x7fc8f0003420 con 0x7fc9100cf7d0 2024-01-16T15:04:25.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.914+0000 7fc90f400640 1 -- 172.21.15.141:0/2507944646 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91015e990 con 0x7fc910143ae0 2024-01-16T15:04:25.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.917+0000 7fc90f400640 1 --2- 172.21.15.141:0/2507944646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc910143ae0 0x7fc9100cf290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc9000097a0 tx=0x7fc900033fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e20659cc15cd12f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.917+0000 7fc90e000640 1 -- 172.21.15.141:0/2507944646 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8001670 con 0x7fc910143ae0 2024-01-16T15:04:25.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.917+0000 7fc90e000640 1 -- 172.21.15.141:0/2507944646 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc900034ad0 con 0x7fc910143ae0 2024-01-16T15:04:25.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.917+0000 7fc90e000640 1 -- 172.21.15.141:0/2507944646 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900034e60 con 0x7fc910143ae0 2024-01-16T15:04:25.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.918+0000 7fc916800640 1 -- 172.21.15.141:0/2507944646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc910143ae0 msgr2=0x7fc9100cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.918+0000 7fc916800640 1 --2- 172.21.15.141:0/2507944646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc910143ae0 0x7fc9100cf290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc9000097a0 tx=0x7fc900033fa0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.919+0000 7fc916800640 1 -- 172.21.15.141:0/2507944646 shutdown_connections 2024-01-16T15:04:25.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.919+0000 7fc916800640 1 --2- 172.21.15.141:0/2507944646 >> v2:172.21.15.188:3300/0 conn(0x7fc91015e3b0 0x7fc9100702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.919+0000 7fc916800640 1 --2- 172.21.15.141:0/2507944646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc910143ae0 0x7fc9100cf290 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.919+0000 7fc916800640 1 -- 172.21.15.141:0/2507944646 >> 172.21.15.141:0/2507944646 conn(0x7fc91006a800 msgr2=0x7fc91006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.919+0000 7fc916800640 1 -- 172.21.15.141:0/2507944646 shutdown_connections 2024-01-16T15:04:25.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.919+0000 7fc916800640 1 -- 172.21.15.141:0/2507944646 wait complete. 2024-01-16T15:04:25.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.921+0000 7fc916800640 1 Processor -- start 2024-01-16T15:04:25.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.921+0000 7fc916800640 1 -- start start 2024-01-16T15:04:25.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.922+0000 7fc916800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9100cf7d0 0x7fc9100ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.922+0000 7fc916800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc910143ae0 0x7fc9100a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.922+0000 7fc916800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc91015e3b0 0x7fc9100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.922+0000 7fc916800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100a94b0 con 0x7fc910143ae0 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.922+0000 7fc916800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100a8ef0 con 0x7fc9100cf7d0 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.922+0000 7fc916800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100a91d0 con 0x7fc91015e3b0 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc910143ae0 0x7fc9100a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc910143ae0 0x7fc9100a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38416/0 (socket says 172.21.15.141:38416) 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 -- 172.21.15.141:0/45789905 learned_addr learned my addr 172.21.15.141:0/45789905 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90f400640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9100cf7d0 0x7fc9100ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 -- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc91015e3b0 msgr2=0x7fc9100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc91015e3b0 0x7fc9100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 -- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9100cf7d0 msgr2=0x7fc9100ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9100cf7d0 0x7fc9100ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.923+0000 7fc90ea00640 1 -- 172.21.15.141:0/45789905 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900009400 con 0x7fc910143ae0 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.924+0000 7fc90ea00640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc910143ae0 0x7fc9100a84c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800caf0 tx=0x7fc8f800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.924+0000 7fc907400640 1 -- 172.21.15.141:0/45789905 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8005ad0 con 0x7fc910143ae0 2024-01-16T15:04:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.924+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91012f960 con 0x7fc910143ae0 2024-01-16T15:04:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.924+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9100acc50 con 0x7fc910143ae0 2024-01-16T15:04:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.924+0000 7fc907400640 1 -- 172.21.15.141:0/45789905 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f8005c30 con 0x7fc910143ae0 2024-01-16T15:04:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.924+0000 7fc907400640 1 -- 172.21.15.141:0/45789905 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8007070 con 0x7fc910143ae0 2024-01-16T15:04:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.926+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e0005250 con 0x7fc910143ae0 2024-01-16T15:04:25.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.927+0000 7fc907400640 1 -- 172.21.15.141:0/45789905 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fc8f80071d0 con 0x7fc910143ae0 2024-01-16T15:04:25.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.928+0000 7fc907400640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e806bb30 0x7fc8e806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.928+0000 7fc907400640 1 -- 172.21.15.141:0/45789905 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8f80883b0 con 0x7fc910143ae0 2024-01-16T15:04:25.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.931+0000 7fc90f400640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e806bb30 0x7fc8e806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.932+0000 7fc90f400640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e806bb30 0x7fc8e806dff0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc90003e000 tx=0x7fc900038000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:25.935+0000 7fc907400640 1 -- 172.21.15.141:0/45789905 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8f8054b00 con 0x7fc910143ae0 2024-01-16T15:04:26.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.316+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8e0000bc0 con 0x7fc8e806bb30 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.339+0000 7fc907400640 1 -- 172.21.15.141:0/45789905 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fc8e0000bc0 con 0x7fc8e806bb30 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:26.318 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:26.319 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:26.319 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:26.319 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:26.319 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:26.319 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:26.319 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:26.319 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.341+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e806bb30 msgr2=0x7fc8e806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.341+0000 7fc916800640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e806bb30 0x7fc8e806dff0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc90003e000 tx=0x7fc900038000 comp rx=0 tx=0).stop 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.341+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc910143ae0 msgr2=0x7fc9100a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.341+0000 7fc916800640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc910143ae0 0x7fc9100a84c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800caf0 tx=0x7fc8f800cef0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 shutdown_connections 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e806bb30 0x7fc8e806dff0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc91015e3b0 0x7fc9100702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc910143ae0 0x7fc9100a84c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 --2- 172.21.15.141:0/45789905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9100cf7d0 0x7fc9100ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 >> 172.21.15.141:0/45789905 conn(0x7fc91006a800 msgr2=0x7fc91006cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:26.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 shutdown_connections 2024-01-16T15:04:26.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:26.342+0000 7fc916800640 1 -- 172.21.15.141:0/45789905 wait complete. 2024-01-16T15:04:26.350 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:27.351 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66725 2024-01-16T15:04:27.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.672+0000 7f2755200640 1 Processor -- start 2024-01-16T15:04:27.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.672+0000 7f2755200640 1 -- start start 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.674+0000 7f2755200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27500ce480 0x7f27500ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.674+0000 7f2755200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f275015dd50 0x7f275015e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.674+0000 7f2755200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500cf500 con 0x7f2750143c00 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.674+0000 7f2755200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f275015c030 con 0x7f27500ce480 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.674+0000 7f2755200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f275015e700 con 0x7f275015dd50 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2750143c00 0x7f275015b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54258/0 (socket says 172.21.15.141:54258) 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274d400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f27500ce480 0x7f27500ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274de00640 1 -- 172.21.15.141:0/3402404054 learned_addr learned my addr 172.21.15.141:0/3402404054 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:27.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274d400640 1 -- 172.21.15.141:0/3402404054 >> v2:172.21.15.188:3300/0 conn(0x7f275015dd50 msgr2=0x7f275015e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:27.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274d400640 1 --2- 172.21.15.141:0/3402404054 >> v2:172.21.15.188:3300/0 conn(0x7f275015dd50 0x7f275015e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:27.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274d400640 1 -- 172.21.15.141:0/3402404054 >> v1:172.21.15.141:6789/0 conn(0x7f2750143c00 legacy=0x7f275015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:27.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274d400640 1 -- 172.21.15.141:0/3402404054 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500cef40 con 0x7f27500ce480 2024-01-16T15:04:27.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.675+0000 7f274ca00640 1 -- 172.21.15.141:0/3402404054 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925716549 0 0) 0x7f27500cf500 con 0x7f2750143c00 2024-01-16T15:04:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.676+0000 7f274d400640 1 --2- 172.21.15.141:0/3402404054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27500ce480 0x7f27500ce930 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f275015dc90 tx=0x7f273802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf998e78f656a3c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f274ca00640 1 -- 172.21.15.141:0/3402404054 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738031830 con 0x7f27500ce480 2024-01-16T15:04:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f274ca00640 1 -- 172.21.15.141:0/3402404054 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2738031990 con 0x7f27500ce480 2024-01-16T15:04:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f274ca00640 1 -- 172.21.15.141:0/3402404054 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738031d20 con 0x7f27500ce480 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f2755200640 1 -- 172.21.15.141:0/3402404054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27500ce480 msgr2=0x7f27500ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f2755200640 1 --2- 172.21.15.141:0/3402404054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27500ce480 0x7f27500ce930 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f275015dc90 tx=0x7f273802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f2755200640 1 -- 172.21.15.141:0/3402404054 shutdown_connections 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f2755200640 1 --2- 172.21.15.141:0/3402404054 >> v2:172.21.15.188:3300/0 conn(0x7f275015dd50 0x7f275015e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f2755200640 1 --2- 172.21.15.141:0/3402404054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27500ce480 0x7f27500ce930 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.677+0000 7f2755200640 1 -- 172.21.15.141:0/3402404054 >> 172.21.15.141:0/3402404054 conn(0x7f27500cff90 msgr2=0x7f27500d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.678+0000 7f2755200640 1 -- 172.21.15.141:0/3402404054 shutdown_connections 2024-01-16T15:04:27.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.678+0000 7f2755200640 1 -- 172.21.15.141:0/3402404054 wait complete. 2024-01-16T15:04:27.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.679+0000 7f2755200640 1 Processor -- start 2024-01-16T15:04:27.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.679+0000 7f2755200640 1 -- start start 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.680+0000 7f2755200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27500ce480 0x7f275013f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.680+0000 7f2755200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2750143c00 0x7f27500c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.680+0000 7f2755200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f275015dd50 0x7f27501f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.680+0000 7f2755200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27500c7a90 con 0x7f2750143c00 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.680+0000 7f2755200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27500c8af0 con 0x7f27500ce480 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.680+0000 7f2755200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27500c7d30 con 0x7f275015dd50 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2750143c00 0x7f27500c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2750143c00 0x7f27500c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38432/0 (socket says 172.21.15.141:38432) 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 -- 172.21.15.141:0/175820189 learned_addr learned my addr 172.21.15.141:0/175820189 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274de00640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27500ce480 0x7f275013f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274e800640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f275015dd50 0x7f27501f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 -- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f275015dd50 msgr2=0x7f27501f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f275015dd50 0x7f27501f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 -- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27500ce480 msgr2=0x7f275013f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27500ce480 0x7f275013f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 -- 172.21.15.141:0/175820189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2738009400 con 0x7f2750143c00 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.681+0000 7f274d400640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2750143c00 0x7f27500c9670 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2738009720 tx=0x7f27380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.682+0000 7f2746000640 1 -- 172.21.15.141:0/175820189 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273803aa60 con 0x7f2750143c00 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.682+0000 7f2746000640 1 -- 172.21.15.141:0/175820189 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273803abc0 con 0x7f2750143c00 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.682+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27500c8010 con 0x7f2750143c00 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.682+0000 7f2746000640 1 -- 172.21.15.141:0/175820189 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27380443f0 con 0x7f2750143c00 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.682+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27500c85d0 con 0x7f2750143c00 2024-01-16T15:04:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.683+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27500c87c0 con 0x7f2750143c00 2024-01-16T15:04:27.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.684+0000 7f2746000640 1 -- 172.21.15.141:0/175820189 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f273804d450 con 0x7f2750143c00 2024-01-16T15:04:27.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.685+0000 7f2746000640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f272806bb30 0x7f272806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:27.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.685+0000 7f2746000640 1 -- 172.21.15.141:0/175820189 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27380aa870 con 0x7f2750143c00 2024-01-16T15:04:27.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.688+0000 7f274de00640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f272806bb30 0x7f272806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:27.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.689+0000 7f274de00640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f272806bb30 0x7f272806dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2740000f60 tx=0x7f2740007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:27.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:27.693+0000 7f2746000640 1 -- 172.21.15.141:0/175820189 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2738076f60 con 0x7f2750143c00 2024-01-16T15:04:28.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.076+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f275006df50 con 0x7f272806bb30 2024-01-16T15:04:28.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.124+0000 7f2746000640 1 -- 172.21.15.141:0/175820189 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f2740007270 con 0x7f272806bb30 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:28.104 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:28.105 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:28.105 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:28.105 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:28.105 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:28.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f272806bb30 msgr2=0x7f272806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f272806bb30 0x7f272806dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2740000f60 tx=0x7f2740007c00 comp rx=0 tx=0).stop 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2750143c00 msgr2=0x7f27500c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2750143c00 0x7f27500c9670 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2738009720 tx=0x7f27380046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 shutdown_connections 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f272806bb30 0x7f272806dff0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f275015dd50 0x7f27501f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2750143c00 0x7f27500c9670 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 --2- 172.21.15.141:0/175820189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27500ce480 0x7f275013f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 >> 172.21.15.141:0/175820189 conn(0x7f27500cff90 msgr2=0x7f275006e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:28.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 shutdown_connections 2024-01-16T15:04:28.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:28.128+0000 7f2755200640 1 -- 172.21.15.141:0/175820189 wait complete. 2024-01-16T15:04:28.130 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:29.130 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66725 2024-01-16T15:04:29.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.440+0000 7f9675e00640 1 Processor -- start 2024-01-16T15:04:29.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.441+0000 7f9675e00640 1 -- start start 2024-01-16T15:04:29.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.443+0000 7f9675e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f967015e880 0x7f967006ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.443+0000 7f9675e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f967009a170 0x7f967006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.445+0000 7f966e000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9670070490 0x7f9670099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54270/0 (socket says 172.21.15.141:54270) 2024-01-16T15:04:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.445+0000 7f966e000640 1 -- 172.21.15.141:0/737555032 learned_addr learned my addr 172.21.15.141:0/737555032 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.445+0000 7f966ea00640 1 --2- 172.21.15.141:0/737555032 >> v2:172.21.15.141:3300/0 conn(0x7f967015e880 0x7f967006ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.445+0000 7f966f400640 1 --2- 172.21.15.141:0/737555032 >> v2:172.21.15.188:3300/0 conn(0x7f967009a170 0x7f967006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.446+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96700cfa70 con 0x7f9670070490 2024-01-16T15:04:29.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.446+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96700cff00 con 0x7f967015e880 2024-01-16T15:04:29.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.446+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967006fc70 con 0x7f967009a170 2024-01-16T15:04:29.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.447+0000 7f966ea00640 1 -- 172.21.15.141:0/737555032 >> v2:172.21.15.188:3300/0 conn(0x7f967009a170 msgr2=0x7f967006e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:29.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.447+0000 7f966d600640 1 -- 172.21.15.141:0/737555032 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090292177 0 0) 0x7f96700cfa70 con 0x7f9670070490 2024-01-16T15:04:29.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.447+0000 7f966ea00640 1 --2- 172.21.15.141:0/737555032 >> v2:172.21.15.188:3300/0 conn(0x7f967009a170 0x7f967006e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.447+0000 7f966ea00640 1 -- 172.21.15.141:0/737555032 >> v1:172.21.15.141:6789/0 conn(0x7f9670070490 legacy=0x7f9670099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.447+0000 7f966ea00640 1 -- 172.21.15.141:0/737555032 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96700cf420 con 0x7f967015e880 2024-01-16T15:04:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.451+0000 7f966ea00640 1 --2- 172.21.15.141:0/737555032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967015e880 0x7f967006ff50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f96700b72c0 tx=0x7f966002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b1b3b33e5594fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:29.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.453+0000 7f966d600640 1 -- 172.21.15.141:0/737555032 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660030610 con 0x7f967015e880 2024-01-16T15:04:29.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.453+0000 7f966d600640 1 -- 172.21.15.141:0/737555032 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9660030770 con 0x7f967015e880 2024-01-16T15:04:29.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.453+0000 7f966d600640 1 -- 172.21.15.141:0/737555032 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660030b00 con 0x7f967015e880 2024-01-16T15:04:29.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.457+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967015e880 msgr2=0x7f967006ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.457+0000 7f9675e00640 1 --2- 172.21.15.141:0/737555032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967015e880 0x7f967006ff50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f96700b72c0 tx=0x7f966002b960 comp rx=0 tx=0).stop 2024-01-16T15:04:29.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.458+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 shutdown_connections 2024-01-16T15:04:29.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.458+0000 7f9675e00640 1 --2- 172.21.15.141:0/737555032 >> v2:172.21.15.188:3300/0 conn(0x7f967009a170 0x7f967006e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.458+0000 7f9675e00640 1 --2- 172.21.15.141:0/737555032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967015e880 0x7f967006ff50 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.458+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 >> 172.21.15.141:0/737555032 conn(0x7f96700b2d00 msgr2=0x7f96700b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:29.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.459+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 shutdown_connections 2024-01-16T15:04:29.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.460+0000 7f9675e00640 1 -- 172.21.15.141:0/737555032 wait complete. 2024-01-16T15:04:29.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.463+0000 7f9675e00640 1 Processor -- start 2024-01-16T15:04:29.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.464+0000 7f9675e00640 1 -- start start 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.465+0000 7f9675e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9670070490 0x7f96701f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.465+0000 7f9675e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967009a170 0x7f96700b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.465+0000 7f9675e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f967015e880 0x7f96700b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f9675e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96700b6300 con 0x7f967009a170 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f9675e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96700b5fb0 con 0x7f967015e880 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f9675e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96701f7740 con 0x7f9670070490 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f966e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967009a170 0x7f96700b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f966e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967009a170 0x7f96700b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38448/0 (socket says 172.21.15.141:38448) 2024-01-16T15:04:29.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f966e000640 1 -- 172.21.15.141:0/1096100772 learned_addr learned my addr 172.21.15.141:0/1096100772 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:29.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f966ea00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9670070490 0x7f96701f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.466+0000 7f966f400640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f967015e880 0x7f96700b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.467+0000 7f966e000640 1 -- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9670070490 msgr2=0x7f96701f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.467+0000 7f966e000640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9670070490 0x7f96701f6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.467+0000 7f966e000640 1 -- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f967015e880 msgr2=0x7f96700b6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.467+0000 7f966e000640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f967015e880 0x7f96700b6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.467+0000 7f966e000640 1 -- 172.21.15.141:0/1096100772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660009400 con 0x7f967009a170 2024-01-16T15:04:29.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.467+0000 7f966e000640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967009a170 0x7f96700b6720 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f965800c500 tx=0x7f965800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:29.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.468+0000 7f9667400640 1 -- 172.21.15.141:0/1096100772 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658007560 con 0x7f967009a170 2024-01-16T15:04:29.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.468+0000 7f9667400640 1 -- 172.21.15.141:0/1096100772 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96580047e0 con 0x7f967009a170 2024-01-16T15:04:29.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.468+0000 7f9667400640 1 -- 172.21.15.141:0/1096100772 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658004b90 con 0x7f967009a170 2024-01-16T15:04:29.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.468+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96701f7a20 con 0x7f967009a170 2024-01-16T15:04:29.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.468+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96701f7450 con 0x7f967009a170 2024-01-16T15:04:29.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.471+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96700675b0 con 0x7f967009a170 2024-01-16T15:04:29.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.471+0000 7f9667400640 1 -- 172.21.15.141:0/1096100772 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f9658004cf0 con 0x7f967009a170 2024-01-16T15:04:29.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.472+0000 7f9667400640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f964806bb60 0x7f964806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.473+0000 7f9667400640 1 -- 172.21.15.141:0/1096100772 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96580877b0 con 0x7f967009a170 2024-01-16T15:04:29.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.474+0000 7f966ea00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f964806bb60 0x7f964806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.476+0000 7f966ea00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f964806bb60 0x7f964806e020 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9660036000 tx=0x7f966003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:29.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.477+0000 7f9667400640 1 -- 172.21.15.141:0/1096100772 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9658053ea0 con 0x7f967009a170 2024-01-16T15:04:29.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:29.799+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f96700ccb30 con 0x7f964806bb60 2024-01-16T15:04:30.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.042+0000 7f9667400640 1 -- 172.21.15.141:0/1096100772 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f96600308e0 con 0x7f964806bb60 2024-01-16T15:04:30.024 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:30.025 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:30.025 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:04:30.025 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:30.025 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:30.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.049+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f964806bb60 msgr2=0x7f964806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.049+0000 7f9675e00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f964806bb60 0x7f964806e020 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9660036000 tx=0x7f966003b040 comp rx=0 tx=0).stop 2024-01-16T15:04:30.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.049+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967009a170 msgr2=0x7f96700b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.049+0000 7f9675e00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967009a170 0x7f96700b6720 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f965800c500 tx=0x7f965800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 shutdown_connections 2024-01-16T15:04:30.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f964806bb60 0x7f964806e020 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f967015e880 0x7f96700b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f967009a170 0x7f96700b6720 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 --2- 172.21.15.141:0/1096100772 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9670070490 0x7f96701f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 >> 172.21.15.141:0/1096100772 conn(0x7f96700b2d00 msgr2=0x7f967015b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 shutdown_connections 2024-01-16T15:04:30.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.051+0000 7f9675e00640 1 -- 172.21.15.141:0/1096100772 wait complete. 2024-01-16T15:04:30.071 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:30.071 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66726 2024-01-16T15:04:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.386+0000 7f7c51600640 1 Processor -- start 2024-01-16T15:04:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.386+0000 7f7c51600640 1 -- start start 2024-01-16T15:04:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.388+0000 7f7c51600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7c4c0b22b0 0x7f7c4c0d3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.388+0000 7f7c51600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0d1a90 0x7f7c4c0ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.388+0000 7f7c51600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4c0d22e0 con 0x7f7c4c0b2690 2024-01-16T15:04:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.388+0000 7f7c51600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4c0ce680 con 0x7f7c4c0b22b0 2024-01-16T15:04:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.388+0000 7f7c51600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0cdd90 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.389+0000 7f7c4a000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7c4c0b2690 0x7f7c4c0b2b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54286/0 (socket says 172.21.15.141:54286) 2024-01-16T15:04:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.389+0000 7f7c4a000640 1 -- 172.21.15.141:0/1504825721 learned_addr learned my addr 172.21.15.141:0/1504825721 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:30.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.390+0000 7f7c4aa00640 1 --2- 172.21.15.141:0/1504825721 >> v2:172.21.15.188:3300/0 conn(0x7f7c4c0b22b0 0x7f7c4c0d3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.390+0000 7f7c4b400640 1 --2- 172.21.15.141:0/1504825721 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0d1a90 0x7f7c4c0ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.390+0000 7f7c4b400640 1 -- 172.21.15.141:0/1504825721 >> v2:172.21.15.188:3300/0 conn(0x7f7c4c0b22b0 msgr2=0x7f7c4c0d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:30.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.391+0000 7f7c4b400640 1 --2- 172.21.15.141:0/1504825721 >> v2:172.21.15.188:3300/0 conn(0x7f7c4c0b22b0 0x7f7c4c0d3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.391+0000 7f7c4b400640 1 -- 172.21.15.141:0/1504825721 >> v1:172.21.15.141:6789/0 conn(0x7f7c4c0b2690 legacy=0x7f7c4c0b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:30.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.391+0000 7f7c4b400640 1 -- 172.21.15.141:0/1504825721 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0d2000 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.393+0000 7f7c4b400640 1 --2- 172.21.15.141:0/1504825721 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0d1a90 0x7f7c4c0ceb10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c340097a0 tx=0x7f7c3402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4df8baeb518de127 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.394+0000 7f7c49600640 1 -- 172.21.15.141:0/1504825721 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3402e070 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.394+0000 7f7c49600640 1 -- 172.21.15.141:0/1504825721 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c340315a0 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.394+0000 7f7c49600640 1 -- 172.21.15.141:0/1504825721 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34031910 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.395+0000 7f7c51600640 1 -- 172.21.15.141:0/1504825721 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0d1a90 msgr2=0x7f7c4c0ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.395+0000 7f7c51600640 1 --2- 172.21.15.141:0/1504825721 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0d1a90 0x7f7c4c0ceb10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c340097a0 tx=0x7f7c3402b840 comp rx=0 tx=0).stop 2024-01-16T15:04:30.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.399+0000 7f7c51600640 1 -- 172.21.15.141:0/1504825721 shutdown_connections 2024-01-16T15:04:30.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.399+0000 7f7c51600640 1 --2- 172.21.15.141:0/1504825721 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0d1a90 0x7f7c4c0ceb10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.399+0000 7f7c51600640 1 --2- 172.21.15.141:0/1504825721 >> v2:172.21.15.188:3300/0 conn(0x7f7c4c0b22b0 0x7f7c4c0d3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.399+0000 7f7c51600640 1 -- 172.21.15.141:0/1504825721 >> 172.21.15.141:0/1504825721 conn(0x7f7c4c09c550 msgr2=0x7f7c4c0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.399+0000 7f7c51600640 1 -- 172.21.15.141:0/1504825721 shutdown_connections 2024-01-16T15:04:30.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.399+0000 7f7c51600640 1 -- 172.21.15.141:0/1504825721 wait complete. 2024-01-16T15:04:30.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.402+0000 7f7c51600640 1 Processor -- start 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.403+0000 7f7c51600640 1 -- start start 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.404+0000 7f7c51600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c4c0b22b0 0x7f7c4c15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.404+0000 7f7c51600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0b2690 0x7f7c4c0b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.404+0000 7f7c51600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c4c0d1a90 0x7f7c4c0b8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.404+0000 7f7c51600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0cc4c0 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.404+0000 7f7c51600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0b8ea0 con 0x7f7c4c0b22b0 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.404+0000 7f7c51600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0b6c20 con 0x7f7c4c0b2690 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.405+0000 7f7c4aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c4c0b22b0 0x7f7c4c15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.405+0000 7f7c4aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c4c0b22b0 0x7f7c4c15b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:55916/0 (socket says 172.21.15.141:55916) 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.405+0000 7f7c4aa00640 1 -- 172.21.15.141:0/493805803 learned_addr learned my addr 172.21.15.141:0/493805803 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.405+0000 7f7c4b400640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c4c0d1a90 0x7f7c4c0b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.405+0000 7f7c4a000640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0b2690 0x7f7c4c0b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.406+0000 7f7c4b400640 1 -- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0b2690 msgr2=0x7f7c4c0b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.406+0000 7f7c4b400640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0b2690 0x7f7c4c0b8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.406+0000 7f7c4b400640 1 -- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c4c0b22b0 msgr2=0x7f7c4c15b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.406+0000 7f7c4b400640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c4c0b22b0 0x7f7c4c15b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.407+0000 7f7c4b400640 1 -- 172.21.15.141:0/493805803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c34009400 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.407+0000 7f7c4aa00640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c4c0b22b0 0x7f7c4c15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:30.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.407+0000 7f7c4a000640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0b2690 0x7f7c4c0b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:30.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.408+0000 7f7c4b400640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c4c0d1a90 0x7f7c4c0b8960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0c65c0 tx=0x7f7c340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.408+0000 7f7c42a00640 1 -- 172.21.15.141:0/493805803 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34006e10 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.408+0000 7f7c42a00640 1 -- 172.21.15.141:0/493805803 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c34041070 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.409+0000 7f7c42a00640 1 -- 172.21.15.141:0/493805803 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34006e10 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.409+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4c0cbe70 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.410+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4c0b6f00 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.412+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4c0cc630 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.412+0000 7f7c42a00640 1 -- 172.21.15.141:0/493805803 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f7c34043030 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.413+0000 7f7c42a00640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c2406bb90 0x7f7c2406e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.414+0000 7f7c42a00640 1 -- 172.21.15.141:0/493805803 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c3407afc0 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.415+0000 7f7c4aa00640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c2406bb90 0x7f7c2406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.420+0000 7f7c42a00640 1 -- 172.21.15.141:0/493805803 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c34077390 con 0x7f7c4c0d1a90 2024-01-16T15:04:30.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.429+0000 7f7c4aa00640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c2406bb90 0x7f7c2406e050 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009a40 tx=0x7f7c4400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.738+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66726", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c4c0d4080 con 0x7f7c2406bb90 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:30.758 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:30.759 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:30.759 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:30.759 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:30.759 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:30.759 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:30.759 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:30.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.779+0000 7f7c42a00640 1 -- 172.21.15.141:0/493805803 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f7c4400f040 con 0x7f7c2406bb90 2024-01-16T15:04:30.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c2406bb90 msgr2=0x7f7c2406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c2406bb90 0x7f7c2406e050 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009a40 tx=0x7f7c4400b040 comp rx=0 tx=0).stop 2024-01-16T15:04:30.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c4c0d1a90 msgr2=0x7f7c4c0b8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c4c0d1a90 0x7f7c4c0b8960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0c65c0 tx=0x7f7c340046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 shutdown_connections 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c2406bb90 0x7f7c2406e050 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c4c0d1a90 0x7f7c4c0b8960 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c4c0b2690 0x7f7c4c0b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 --2- 172.21.15.141:0/493805803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c4c0b22b0 0x7f7c4c15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 >> 172.21.15.141:0/493805803 conn(0x7f7c4c09c550 msgr2=0x7f7c4c0cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 shutdown_connections 2024-01-16T15:04:30.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:30.784+0000 7f7c51600640 1 -- 172.21.15.141:0/493805803 wait complete. 2024-01-16T15:04:30.793 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:31.794 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66726 2024-01-16T15:04:32.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.039+0000 7f2096400640 1 Processor -- start 2024-01-16T15:04:32.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.039+0000 7f2096400640 1 -- start start 2024-01-16T15:04:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.041+0000 7f2096400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 0x7f209015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.041+0000 7f2096400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209006c470 con 0x7f20900cf9a0 2024-01-16T15:04:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.041+0000 7f2096400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209006c750 con 0x7f209013e360 2024-01-16T15:04:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.041+0000 7f2096400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209006ca30 con 0x7f209013df80 2024-01-16T15:04:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.041+0000 7f208fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f20900cf9a0 0x7f20900cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54296/0 (socket says 172.21.15.141:54296) 2024-01-16T15:04:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.041+0000 7f208fe00640 1 -- 172.21.15.141:0/3399736438 learned_addr learned my addr 172.21.15.141:0/3399736438 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:32.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208f400640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 0x7f209015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047933561 0 0) 0x7f209006c470 con 0x7f20900cf9a0 2024-01-16T15:04:32.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2070003520 con 0x7f20900cf9a0 2024-01-16T15:04:32.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 359585331 0 0) 0x7f2070003520 con 0x7f20900cf9a0 2024-01-16T15:04:32.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 msgr2=0x7f209015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 0x7f209015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 >> v1:172.21.15.188:6789/0 conn(0x7f209013e360 legacy=0x7f20900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209006cd10 con 0x7f20900cf9a0 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f20880019f0 con 0x7f20900cf9a0 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208f400640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 0x7f209015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 >> v1:172.21.15.141:6789/0 conn(0x7f20900cf9a0 legacy=0x7f20900cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20700039a0 0x7f2070003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 0x7f20700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f2070005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209006ca30 con 0x7f2070004d00 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207000ba90 con 0x7f20700039a0 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.042+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207000bc50 con 0x7f209013df80 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208fe00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f2070005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208f400640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20700039a0 0x7f2070003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f2086800640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 0x7f20700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208fe00640 1 -- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 msgr2=0x7f20700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208fe00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f209013df80 0x7f20700047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208fe00640 1 -- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20700039a0 msgr2=0x7f2070003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208fe00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20700039a0 0x7f2070003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208fe00640 1 -- 172.21.15.141:0/3399736438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2070003520 con 0x7f2070004d00 2024-01-16T15:04:32.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.043+0000 7f208f400640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20700039a0 0x7f2070003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.044+0000 7f208fe00640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f2070005180 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f208800a800 tx=0x7f20700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5ad98cb999dfc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.044+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880072d0 con 0x7f2070004d00 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.044+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208802f070 con 0x7f2070004d00 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.044+0000 7f208ea00640 1 -- 172.21.15.141:0/3399736438 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880076b0 con 0x7f2070004d00 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.045+0000 7f2096400640 1 -- 172.21.15.141:0/3399736438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 msgr2=0x7f2070005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.045+0000 7f2096400640 1 --2- 172.21.15.141:0/3399736438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f2070005180 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f208800a800 tx=0x7f20700057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.045+0000 7f208fe00640 1 -- 172.21.15.141:0/3399736438 reap_dead start 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.045+0000 7f2096400640 1 -- 172.21.15.141:0/3399736438 shutdown_connections 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.045+0000 7f2096400640 1 -- 172.21.15.141:0/3399736438 >> 172.21.15.141:0/3399736438 conn(0x7f209006a800 msgr2=0x7f209006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.045+0000 7f2096400640 1 -- 172.21.15.141:0/3399736438 shutdown_connections 2024-01-16T15:04:32.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.045+0000 7f2096400640 1 -- 172.21.15.141:0/3399736438 wait complete. 2024-01-16T15:04:32.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.047+0000 7f2096400640 1 Processor -- start 2024-01-16T15:04:32.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.047+0000 7f2096400640 1 -- start start 2024-01-16T15:04:32.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f2096400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f20901f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f2096400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20900b3470 0x7f2090156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f2096400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20900b38f0 0x7f2090156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f2096400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20901f32c0 con 0x7f2070004d00 2024-01-16T15:04:32.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f2096400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20901f6e10 con 0x7f20900b3470 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f2096400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2090153520 con 0x7f20900b38f0 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f208f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f20901f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f208f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f20901f68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38486/0 (socket says 172.21.15.141:38486) 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.048+0000 7f208f400640 1 -- 172.21.15.141:0/1851184321 learned_addr learned my addr 172.21.15.141:0/1851184321 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f2086800640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20900b3470 0x7f2090156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f208fe00640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20900b38f0 0x7f2090156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f208f400640 1 -- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20900b38f0 msgr2=0x7f2090156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f208f400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20900b38f0 0x7f2090156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f208f400640 1 -- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20900b3470 msgr2=0x7f2090156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f208f400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20900b3470 0x7f2090156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f208f400640 1 -- 172.21.15.141:0/1851184321 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2090152c80 con 0x7f2070004d00 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f2086800640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20900b3470 0x7f2090156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f208f400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f20901f68d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f208000f580 tx=0x7f208000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f2087e00640 1 -- 172.21.15.141:0/1851184321 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080014890 con 0x7f2070004d00 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f2087e00640 1 -- 172.21.15.141:0/1851184321 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208000c030 con 0x7f2070004d00 2024-01-16T15:04:32.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f2087e00640 1 -- 172.21.15.141:0/1851184321 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080014bf0 con 0x7f2070004d00 2024-01-16T15:04:32.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.049+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2090152f60 con 0x7f2070004d00 2024-01-16T15:04:32.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.050+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20901528a0 con 0x7f2070004d00 2024-01-16T15:04:32.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.051+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209015d290 con 0x7f2070004d00 2024-01-16T15:04:32.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.051+0000 7f2087e00640 1 -- 172.21.15.141:0/1851184321 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f2080014d50 con 0x7f2070004d00 2024-01-16T15:04:32.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.052+0000 7f2087e00640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f206806bb30 0x7f206806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.052+0000 7f2087e00640 1 -- 172.21.15.141:0/1851184321 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2080088990 con 0x7f2070004d00 2024-01-16T15:04:32.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.053+0000 7f2086800640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f206806bb30 0x7f206806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.054+0000 7f2086800640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f206806bb30 0x7f206806dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f20900b0720 tx=0x7f2078009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.057+0000 7f2087e00640 1 -- 172.21.15.141:0/1851184321 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2080055060 con 0x7f2070004d00 2024-01-16T15:04:32.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.281+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66726", "target": ["mon-mgr", ""]}) v1 -- 0x7f209015bee0 con 0x7f206806bb30 2024-01-16T15:04:32.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.556+0000 7f2087e00640 1 -- 172.21.15.141:0/1851184321 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f207800d040 con 0x7f206806bb30 2024-01-16T15:04:32.535 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:32.535 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:32.536 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:04:32.536 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:32.536 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:32.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f206806bb30 msgr2=0x7f206806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f206806bb30 0x7f206806dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f20900b0720 tx=0x7f2078009040 comp rx=0 tx=0).stop 2024-01-16T15:04:32.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 msgr2=0x7f20901f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f20901f68d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f208000f580 tx=0x7f208000fa40 comp rx=0 tx=0).stop 2024-01-16T15:04:32.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 shutdown_connections 2024-01-16T15:04:32.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f206806bb30 0x7f206806dff0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20900b38f0 0x7f2090156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20900b3470 0x7f2090156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 --2- 172.21.15.141:0/1851184321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2070004d00 0x7f20901f68d0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 >> 172.21.15.141:0/1851184321 conn(0x7f209006a800 msgr2=0x7f209015cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 shutdown_connections 2024-01-16T15:04:32.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.560+0000 7f2096400640 1 -- 172.21.15.141:0/1851184321 wait complete. 2024-01-16T15:04:32.560 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:32.561 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66727 2024-01-16T15:04:32.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.855+0000 7f04bf600640 1 Processor -- start 2024-01-16T15:04:32.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.855+0000 7f04bf600640 1 -- start start 2024-01-16T15:04:32.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.857+0000 7f04bf600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f04b80cf9a0 0x7f04b8166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.857+0000 7f04bf600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b815c5f0 con 0x7f04b813df80 2024-01-16T15:04:32.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.857+0000 7f04bf600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b806c470 con 0x7f04b813e360 2024-01-16T15:04:32.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.857+0000 7f04bf600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b806c750 con 0x7f04b80cf9a0 2024-01-16T15:04:32.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.857+0000 7f04bcc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f04b813df80 0x7f04b815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54300/0 (socket says 172.21.15.141:54300) 2024-01-16T15:04:32.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.857+0000 7f04bcc00640 1 -- 172.21.15.141:0/3273726138 learned_addr learned my addr 172.21.15.141:0/3273726138 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:32.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.858+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658834537 0 0) 0x7f04b815c5f0 con 0x7f04b813df80 2024-01-16T15:04:32.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.858+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0498003520 con 0x7f04b813df80 2024-01-16T15:04:32.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.858+0000 7f04bd600640 1 --2- 172.21.15.141:0/3273726138 >> v2:172.21.15.188:3300/0 conn(0x7f04b80cf9a0 0x7f04b8166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.858+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029075301 0 0) 0x7f0498003520 con 0x7f04b813df80 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.858+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 >> v2:172.21.15.188:3300/0 conn(0x7f04b80cf9a0 msgr2=0x7f04b8166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 --2- 172.21.15.141:0/3273726138 >> v2:172.21.15.188:3300/0 conn(0x7f04b80cf9a0 0x7f04b8166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 >> v1:172.21.15.188:6789/0 conn(0x7f04b813e360 legacy=0x7f04b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b806ca30 con 0x7f04b813df80 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f04a8001670 con 0x7f04b813df80 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 >> v1:172.21.15.141:6789/0 conn(0x7f04b813df80 legacy=0x7f04b815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0498003970 0x7f0498003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04980047c0 0x7f0498004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f049800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b806c750 con 0x7f0498005180 2024-01-16T15:04:32.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04980104e0 con 0x7f04980047c0 2024-01-16T15:04:32.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.859+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04980106a0 con 0x7f0498003970 2024-01-16T15:04:32.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.860+0000 7f04bd600640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f049800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.860+0000 7f04bd600640 1 -- 172.21.15.141:0/3273726138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0498003970 msgr2=0x7f0498003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.860+0000 7f04bd600640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0498003970 0x7f0498003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.860+0000 7f04bd600640 1 -- 172.21.15.141:0/3273726138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04980047c0 msgr2=0x7f0498004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:04:32.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.860+0000 7f04bd600640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04980047c0 0x7f0498004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.860+0000 7f04bd600640 1 -- 172.21.15.141:0/3273726138 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0498003520 con 0x7f0498005180 2024-01-16T15:04:32.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.861+0000 7f04bd600640 1 -- 172.21.15.141:0/3273726138 reap_dead start 2024-01-16T15:04:32.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.861+0000 7f04bd600640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f049800b980 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f04b0007330 tx=0x7f04980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec06ebb9c4d074 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.862+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b002ccd0 con 0x7f0498005180 2024-01-16T15:04:32.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.862+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b002ce30 con 0x7f0498005180 2024-01-16T15:04:32.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.862+0000 7f04b7400640 1 -- 172.21.15.141:0/3273726138 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0003bd0 con 0x7f0498005180 2024-01-16T15:04:32.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.863+0000 7f04bf600640 1 -- 172.21.15.141:0/3273726138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 msgr2=0x7f049800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.863+0000 7f04bf600640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f049800b980 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f04b0007330 tx=0x7f04980057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.863+0000 7f04bf600640 1 -- 172.21.15.141:0/3273726138 shutdown_connections 2024-01-16T15:04:32.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.863+0000 7f04bf600640 1 --2- 172.21.15.141:0/3273726138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f049800b980 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.863+0000 7f04bf600640 1 -- 172.21.15.141:0/3273726138 >> 172.21.15.141:0/3273726138 conn(0x7f04b806a800 msgr2=0x7f04b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.863+0000 7f04bf600640 1 -- 172.21.15.141:0/3273726138 shutdown_connections 2024-01-16T15:04:32.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.863+0000 7f04bf600640 1 -- 172.21.15.141:0/3273726138 wait complete. 2024-01-16T15:04:32.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.865+0000 7f04bf600640 1 Processor -- start 2024-01-16T15:04:32.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.866+0000 7f04bf600640 1 -- start start 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bf600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f04b813b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bf600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b8152d90 0x7f04b813bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bf600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04b814e440 0x7f04b80afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bf600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b814f610 con 0x7f0498005180 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bf600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b814ed70 con 0x7f04b8152d90 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bf600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b814f050 con 0x7f04b814e440 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bcc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f04b813b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bcc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f04b813b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38508/0 (socket says 172.21.15.141:38508) 2024-01-16T15:04:32.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.867+0000 7f04bcc00640 1 -- 172.21.15.141:0/2179644031 learned_addr learned my addr 172.21.15.141:0/2179644031 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:32.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.868+0000 7f04bd600640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04b814e440 0x7f04b80afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.868+0000 7f04bcc00640 1 -- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04b814e440 msgr2=0x7f04b80afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.868+0000 7f04bcc00640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04b814e440 0x7f04b80afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.868+0000 7f04bcc00640 1 -- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b8152d90 msgr2=0x7f04b813bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.868+0000 7f04bcc00640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b8152d90 0x7f04b813bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.868+0000 7f04bcc00640 1 -- 172.21.15.141:0/2179644031 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b814f330 con 0x7f0498005180 2024-01-16T15:04:32.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.868+0000 7f04bcc00640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f04b813b7b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f04a800c770 tx=0x7f04a800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.869+0000 7f04b4c00640 1 -- 172.21.15.141:0/2179644031 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80074e0 con 0x7f0498005180 2024-01-16T15:04:32.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.869+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b81535d0 con 0x7f0498005180 2024-01-16T15:04:32.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.869+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b813c4f0 con 0x7f0498005180 2024-01-16T15:04:32.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.871+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b8153810 con 0x7f0498005180 2024-01-16T15:04:32.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.871+0000 7f04b4c00640 1 -- 172.21.15.141:0/2179644031 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a800ce80 con 0x7f0498005180 2024-01-16T15:04:32.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.871+0000 7f04b4c00640 1 -- 172.21.15.141:0/2179644031 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80049e0 con 0x7f0498005180 2024-01-16T15:04:32.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.873+0000 7f04b4c00640 1 -- 172.21.15.141:0/2179644031 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f04a8004b80 con 0x7f0498005180 2024-01-16T15:04:32.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.874+0000 7f04b4c00640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f049006bb30 0x7f049006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.875+0000 7f04b7e00640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f049006bb30 0x7f049006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.875+0000 7f04b4c00640 1 -- 172.21.15.141:0/2179644031 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04a8007640 con 0x7f0498005180 2024-01-16T15:04:32.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.876+0000 7f04b7e00640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f049006bb30 0x7f049006dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f04a0000f60 tx=0x7f04a00079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:32.882+0000 7f04b4c00640 1 -- 172.21.15.141:0/2179644031 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04b8153810 con 0x7f0498005180 2024-01-16T15:04:33.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.232+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66727", "target": ["mon-mgr", ""]}) v1 -- 0x7f04b806cff0 con 0x7f049006bb30 2024-01-16T15:04:33.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.258+0000 7f04b4c00640 1 -- 172.21.15.141:0/2179644031 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f04a0005080 con 0x7f049006bb30 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:33.237 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:33.238 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:33.238 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:33.238 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:33.238 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:33.238 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:33.238 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.262+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f049006bb30 msgr2=0x7f049006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.262+0000 7f04bf600640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f049006bb30 0x7f049006dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f04a0000f60 tx=0x7f04a00079d0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.262+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 msgr2=0x7f04b813b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.262+0000 7f04bf600640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f04b813b7b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f04a800c770 tx=0x7f04a800cc30 comp rx=0 tx=0).stop 2024-01-16T15:04:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 shutdown_connections 2024-01-16T15:04:33.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f049006bb30 0x7f049006dff0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04b814e440 0x7f04b80afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b8152d90 0x7f04b813bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 --2- 172.21.15.141:0/2179644031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0498005180 0x7f04b813b7b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 >> 172.21.15.141:0/2179644031 conn(0x7f04b806a800 msgr2=0x7f04b8155400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:33.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 shutdown_connections 2024-01-16T15:04:33.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:33.263+0000 7f04bf600640 1 -- 172.21.15.141:0/2179644031 wait complete. 2024-01-16T15:04:33.283 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:34.090 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:34.102 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:34.284 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66727 2024-01-16T15:04:34.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.581+0000 7fa4f7e00640 1 Processor -- start 2024-01-16T15:04:34.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.581+0000 7fa4f7e00640 1 -- start start 2024-01-16T15:04:34.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.582+0000 7fa4f7e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa4f80cf9a0 0x7fa4f80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.582+0000 7fa4f7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f80b31d0 0x7fa4f80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.582+0000 7fa4f7e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80d0320 con 0x7fa4f815d4b0 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.582+0000 7fa4f7e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f80d1c60 con 0x7fa4f80cf9a0 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.582+0000 7fa4f7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f80b2730 con 0x7fa4f80b31d0 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa4f80cf9a0 0x7fa4f80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f80b31d0 0x7fa4f80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa4f80cf9a0 0x7fa4f80cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:42212/0 (socket says 172.21.15.141:42212) 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6000640 1 -- 172.21.15.141:0/2109518787 learned_addr learned my addr 172.21.15.141:0/2109518787 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:34.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6a00640 1 --1- 172.21.15.141:0/2109518787 >> v1:172.21.15.188:6789/0 conn(0x7fa4f815d4b0 0x7fa4f80d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:48372/0 (socket says 172.21.15.141:48372) 2024-01-16T15:04:34.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6000640 1 -- 172.21.15.141:0/2109518787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f80b31d0 msgr2=0x7fa4f80b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6000640 1 --2- 172.21.15.141:0/2109518787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f80b31d0 0x7fa4f80b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6000640 1 -- 172.21.15.141:0/2109518787 >> v1:172.21.15.188:6789/0 conn(0x7fa4f815d4b0 legacy=0x7fa4f80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.583+0000 7fa4f6000640 1 -- 172.21.15.141:0/2109518787 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80d1ec0 con 0x7fa4f80cf9a0 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.584+0000 7fa4f6000640 1 --2- 172.21.15.141:0/2109518787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 0x7fa4f80cfde0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009870 tx=0x7fa4e002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5186001b22f3b78 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.585+0000 7fa4f5600640 1 -- 172.21.15.141:0/2109518787 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0030840 con 0x7fa4f80cf9a0 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.585+0000 7fa4f5600640 1 -- 172.21.15.141:0/2109518787 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e00309a0 con 0x7fa4f80cf9a0 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.585+0000 7fa4f5600640 1 -- 172.21.15.141:0/2109518787 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0030d30 con 0x7fa4f80cf9a0 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.585+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2109518787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 msgr2=0x7fa4f80cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.585+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2109518787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 0x7fa4f80cfde0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009870 tx=0x7fa4e002b980 comp rx=0 tx=0).stop 2024-01-16T15:04:34.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.586+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2109518787 shutdown_connections 2024-01-16T15:04:34.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.586+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2109518787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 0x7fa4f80cfde0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.586+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2109518787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f80b31d0 0x7fa4f80b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.586+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2109518787 >> 172.21.15.141:0/2109518787 conn(0x7fa4f80d3000 msgr2=0x7fa4f80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:34.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.586+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2109518787 shutdown_connections 2024-01-16T15:04:34.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.586+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2109518787 wait complete. 2024-01-16T15:04:34.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.587+0000 7fa4f7e00640 1 Processor -- start 2024-01-16T15:04:34.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.588+0000 7fa4f7e00640 1 -- start start 2024-01-16T15:04:34.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.588+0000 7fa4f7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4f80b31d0 0x7fa4f814c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.588+0000 7fa4f7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 0x7fa4f809e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.588+0000 7fa4f7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f815d4b0 0x7fa4f809fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.588+0000 7fa4f7e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f809ec10 con 0x7fa4f80b31d0 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.588+0000 7fa4f7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f809eef0 con 0x7fa4f80cf9a0 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.588+0000 7fa4f7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f809f1d0 con 0x7fa4f815d4b0 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4f80b31d0 0x7fa4f814c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4f80b31d0 0x7fa4f814c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57108/0 (socket says 172.21.15.141:57108) 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 -- 172.21.15.141:0/2467751127 learned_addr learned my addr 172.21.15.141:0/2467751127 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f7400640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f815d4b0 0x7fa4f809fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6000640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 0x7fa4f809e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 -- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f815d4b0 msgr2=0x7fa4f809fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f815d4b0 0x7fa4f809fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 -- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 msgr2=0x7fa4f809e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 0x7fa4f809e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.589+0000 7fa4f6a00640 1 -- 172.21.15.141:0/2467751127 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e00094d0 con 0x7fa4f80b31d0 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.590+0000 7fa4f6a00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4f80b31d0 0x7fa4f814c300 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800c400 tx=0x7fa4e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.590+0000 7fa4eea00640 1 -- 172.21.15.141:0/2467751127 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e80074e0 con 0x7fa4f80b31d0 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.590+0000 7fa4eea00640 1 -- 172.21.15.141:0/2467751127 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e8007640 con 0x7fa4f80b31d0 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.590+0000 7fa4eea00640 1 -- 172.21.15.141:0/2467751127 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8004a00 con 0x7fa4f80b31d0 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.590+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f809f4b0 con 0x7fa4f80b31d0 2024-01-16T15:04:34.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.590+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f814c840 con 0x7fa4f80b31d0 2024-01-16T15:04:34.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.591+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f80d21a0 con 0x7fa4f80b31d0 2024-01-16T15:04:34.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.592+0000 7fa4eea00640 1 -- 172.21.15.141:0/2467751127 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa4e8004b60 con 0x7fa4f80b31d0 2024-01-16T15:04:34.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.593+0000 7fa4eea00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4d006bb30 0x7fa4d006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.593+0000 7fa4eea00640 1 -- 172.21.15.141:0/2467751127 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4e8087e60 con 0x7fa4f80b31d0 2024-01-16T15:04:34.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.594+0000 7fa4f6000640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4d006bb30 0x7fa4d006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.595+0000 7fa4f6000640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4d006bb30 0x7fa4d006dff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0036000 tx=0x7fa4e0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.598+0000 7fa4eea00640 1 -- 172.21.15.141:0/2467751127 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4e8054660 con 0x7fa4f80b31d0 2024-01-16T15:04:34.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:34.838+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66727", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4f814c9d0 con 0x7fa4d006bb30 2024-01-16T15:04:35.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.704+0000 7fa4eea00640 1 -- 172.21.15.141:0/2467751127 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fa4e0030b10 con 0x7fa4d006bb30 2024-01-16T15:04:35.683 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:04:35.684 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:04:35.684 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:04:35.684 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:35.684 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4d006bb30 msgr2=0x7fa4d006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4d006bb30 0x7fa4d006dff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0036000 tx=0x7fa4e0008030 comp rx=0 tx=0).stop 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4f80b31d0 msgr2=0x7fa4f814c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4f80b31d0 0x7fa4f814c300 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800c400 tx=0x7fa4e800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 shutdown_connections 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4d006bb30 0x7fa4d006dff0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4f815d4b0 0x7fa4f809fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4f80cf9a0 0x7fa4f809e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 --2- 172.21.15.141:0/2467751127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4f80b31d0 0x7fa4f814c300 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 >> 172.21.15.141:0/2467751127 conn(0x7fa4f80d3000 msgr2=0x7fa4f80b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.707+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 shutdown_connections 2024-01-16T15:04:35.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.708+0000 7fa4f7e00640 1 -- 172.21.15.141:0/2467751127 wait complete. 2024-01-16T15:04:35.707 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:04:35.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.935+0000 7f2121600640 1 Processor -- start 2024-01-16T15:04:35.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.935+0000 7f2121600640 1 -- start start 2024-01-16T15:04:35.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.937+0000 7f2121600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f211c071290 0x7f211c1539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.937+0000 7f2121600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c06aea0 con 0x7f211c0cef70 2024-01-16T15:04:35.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.937+0000 7f2121600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c06b180 con 0x7f211c161890 2024-01-16T15:04:35.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.937+0000 7f2121600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211c0cd390 con 0x7f211c071290 2024-01-16T15:04:35.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.937+0000 7f211be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f211c0cef70 0x7f211c06a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33248/0 (socket says 172.21.15.141:33248) 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.937+0000 7f211be00640 1 -- 172.21.15.141:0/3929621334 learned_addr learned my addr 172.21.15.141:0/3929621334 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912915966 0 0) 0x7f211c06aea0 con 0x7f211c0cef70 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc003520 con 0x7f211c0cef70 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f2120c00640 1 --2- 172.21.15.141:0/3929621334 >> v2:172.21.15.188:3300/0 conn(0x7f211c071290 0x7f211c1539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 112511667 0 0) 0x7f20fc003520 con 0x7f211c0cef70 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 >> v2:172.21.15.188:3300/0 conn(0x7f211c071290 msgr2=0x7f211c1539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 --2- 172.21.15.141:0/3929621334 >> v2:172.21.15.188:3300/0 conn(0x7f211c071290 0x7f211c1539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 >> v1:172.21.15.188:6789/0 conn(0x7f211c161890 legacy=0x7f211c070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c160db0 con 0x7f211c0cef70 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f210c001670 con 0x7f211c0cef70 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 >> v1:172.21.15.141:6789/0 conn(0x7f211c0cef70 legacy=0x7f211c06a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0039c0 0x7f20fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20fc0047c0 0x7f20fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20fc0052a0 0x7f20fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc00c190 con 0x7f20fc0047c0 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc00c300 con 0x7f20fc0039c0 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.938+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc00c4c0 con 0x7f20fc0052a0 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f211b400640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20fc0047c0 0x7f20fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f211be00640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0039c0 0x7f20fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f2120c00640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20fc0052a0 0x7f20fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f211b400640 1 -- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20fc0052a0 msgr2=0x7f20fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f211b400640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f20fc0052a0 0x7f20fc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f211b400640 1 -- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0039c0 msgr2=0x7f20fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f211b400640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0039c0 0x7f20fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f211b400640 1 -- 172.21.15.141:0/3929621334 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc003520 con 0x7f20fc0047c0 2024-01-16T15:04:35.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.939+0000 7f2120c00640 1 -- 172.21.15.141:0/3929621334 reap_dead start 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.940+0000 7f211b400640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20fc0047c0 0x7f20fc004c40 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2104001770 tx=0x7f20fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0cf2e6060b4e87 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.940+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040052d0 con 0x7f20fc0047c0 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.940+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210402e050 con 0x7f20fc0047c0 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.940+0000 7f211aa00640 1 -- 172.21.15.141:0/3929621334 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2104003640 con 0x7f20fc0047c0 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.941+0000 7f2121600640 1 -- 172.21.15.141:0/3929621334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20fc0047c0 msgr2=0x7f20fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.941+0000 7f2121600640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20fc0047c0 0x7f20fc004c40 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2104001770 tx=0x7f20fc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.941+0000 7f2121600640 1 -- 172.21.15.141:0/3929621334 shutdown_connections 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.941+0000 7f2121600640 1 --2- 172.21.15.141:0/3929621334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20fc0047c0 0x7f20fc004c40 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.941+0000 7f2121600640 1 -- 172.21.15.141:0/3929621334 >> 172.21.15.141:0/3929621334 conn(0x7f211c0cc1d0 msgr2=0x7f211c0cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.941+0000 7f2121600640 1 -- 172.21.15.141:0/3929621334 shutdown_connections 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.941+0000 7f2121600640 1 -- 172.21.15.141:0/3929621334 wait complete. 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.943+0000 7f2121600640 1 Processor -- start 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.943+0000 7f2121600640 1 -- start start 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2121600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0047c0 0x7f211c160330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2121600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f211c15cfb0 0x7f211c14c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2121600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f211c1f3260 0x7f211c1f3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2121600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c160870 con 0x7f211c1f3260 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2121600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c14c630 con 0x7f20fc0047c0 2024-01-16T15:04:35.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2121600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c15d4b0 con 0x7f211c15cfb0 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f211c1f3260 0x7f211c1f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f211c1f3260 0x7f211c1f3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57132/0 (socket says 172.21.15.141:57132) 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 -- 172.21.15.141:0/1567659806 learned_addr learned my addr 172.21.15.141:0/1567659806 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f211b400640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f211c15cfb0 0x7f211c14c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 -- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f211c15cfb0 msgr2=0x7f211c14c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f211be00640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0047c0 0x7f211c160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f211c15cfb0 0x7f211c14c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 -- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0047c0 msgr2=0x7f211c160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0047c0 0x7f211c160330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.944+0000 7f2120c00640 1 -- 172.21.15.141:0/1567659806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c15d790 con 0x7f211c1f3260 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.945+0000 7f2120c00640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f211c1f3260 0x7f211c1f3710 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f211c06bfb0 tx=0x7f2114009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.945+0000 7f2113e00640 1 -- 172.21.15.141:0/1567659806 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114007cf0 con 0x7f211c1f3260 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.945+0000 7f2113e00640 1 -- 172.21.15.141:0/1567659806 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2114007e70 con 0x7f211c1f3260 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.945+0000 7f2113e00640 1 -- 172.21.15.141:0/1567659806 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114014620 con 0x7f211c1f3260 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.945+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c15da70 con 0x7f211c1f3260 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.945+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c15e030 con 0x7f211c1f3260 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.946+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c15e220 con 0x7f211c1f3260 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.947+0000 7f2113e00640 1 -- 172.21.15.141:0/1567659806 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f2114014780 con 0x7f211c1f3260 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.948+0000 7f2113e00640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20f406bb30 0x7f20f406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.948+0000 7f2113e00640 1 -- 172.21.15.141:0/1567659806 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2114088290 con 0x7f211c1f3260 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.949+0000 7f211be00640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20f406bb30 0x7f20f406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.950+0000 7f211be00640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20f406bb30 0x7f20f406dff0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f210c001050 tx=0x7f210c001170 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:35.952+0000 7f2113e00640 1 -- 172.21.15.141:0/1567659806 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2114054a30 con 0x7f211c1f3260 2024-01-16T15:04:36.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.169+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f211c161370 con 0x7f20f406bb30 2024-01-16T15:04:36.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.172+0000 7f2113e00640 1 -- 172.21.15.141:0/1567659806 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f210c005c80 con 0x7f20f406bb30 2024-01-16T15:04:36.151 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:04:36.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.151 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:04:36.151 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:36.151 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20f406bb30 msgr2=0x7f20f406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20f406bb30 0x7f20f406dff0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f210c001050 tx=0x7f210c001170 comp rx=0 tx=0).stop 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f211c1f3260 msgr2=0x7f211c1f3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f211c1f3260 0x7f211c1f3710 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f211c06bfb0 tx=0x7f2114009d30 comp rx=0 tx=0).stop 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 shutdown_connections 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f20f406bb30 0x7f20f406dff0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f211c1f3260 0x7f211c1f3710 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f211c15cfb0 0x7f211c14c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 --2- 172.21.15.141:0/1567659806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc0047c0 0x7f211c160330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 >> 172.21.15.141:0/1567659806 conn(0x7f211c0cc1d0 msgr2=0x7f211c06a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 shutdown_connections 2024-01-16T15:04:36.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.175+0000 7f2121600640 1 -- 172.21.15.141:0/1567659806 wait complete. 2024-01-16T15:04:36.174 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:04:36.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.410+0000 7f47a4e00640 1 Processor -- start 2024-01-16T15:04:36.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.410+0000 7f47a4e00640 1 -- start start 2024-01-16T15:04:36.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a00adf20 0x7f47a0070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f47a4e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f47a015bf10 0x7f47a00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f47a4e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a015f4e0 con 0x7f47a00cc980 2024-01-16T15:04:36.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f47a4e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a015f970 con 0x7f47a015bf10 2024-01-16T15:04:36.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f47a4e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00cce10 con 0x7f47a00adf20 2024-01-16T15:04:36.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f479fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f47a00cc980 0x7f47a0143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33258/0 (socket says 172.21.15.141:33258) 2024-01-16T15:04:36.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f479fe00640 1 -- 172.21.15.141:0/6776966 learned_addr learned my addr 172.21.15.141:0/6776966 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:36.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.412+0000 7f4797e00640 1 --2- 172.21.15.141:0/6776966 >> v2:172.21.15.141:3300/0 conn(0x7f47a015bf10 0x7f47a00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.413+0000 7f479f400640 1 --2- 172.21.15.141:0/6776966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a00adf20 0x7f47a0070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.413+0000 7f479ea00640 1 -- 172.21.15.141:0/6776966 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693280788 0 0) 0x7f47a015f4e0 con 0x7f47a00cc980 2024-01-16T15:04:36.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.413+0000 7f479ea00640 1 -- 172.21.15.141:0/6776966 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4780003520 con 0x7f47a00cc980 2024-01-16T15:04:36.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.413+0000 7f4797e00640 1 -- 172.21.15.141:0/6776966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a00adf20 msgr2=0x7f47a0070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.413+0000 7f4797e00640 1 --2- 172.21.15.141:0/6776966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a00adf20 0x7f47a0070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.413+0000 7f4797e00640 1 -- 172.21.15.141:0/6776966 >> v1:172.21.15.141:6789/0 conn(0x7f47a00cc980 legacy=0x7f47a0143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.413+0000 7f4797e00640 1 -- 172.21.15.141:0/6776966 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a0071130 con 0x7f47a015bf10 2024-01-16T15:04:36.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.414+0000 7f4797e00640 1 --2- 172.21.15.141:0/6776966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a015bf10 0x7f47a00cc440 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f47880097a0 tx=0x7f478802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74191b556840668c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.414+0000 7f479ea00640 1 -- 172.21.15.141:0/6776966 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478802e070 con 0x7f47a015bf10 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.414+0000 7f479ea00640 1 -- 172.21.15.141:0/6776966 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47880315a0 con 0x7f47a015bf10 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.414+0000 7f479ea00640 1 -- 172.21.15.141:0/6776966 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788031930 con 0x7f47a015bf10 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 -- 172.21.15.141:0/6776966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a015bf10 msgr2=0x7f47a00cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 --2- 172.21.15.141:0/6776966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a015bf10 0x7f47a00cc440 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f47880097a0 tx=0x7f478802b840 comp rx=0 tx=0).stop 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 -- 172.21.15.141:0/6776966 shutdown_connections 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 --2- 172.21.15.141:0/6776966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a015bf10 0x7f47a00cc440 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 --2- 172.21.15.141:0/6776966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a00adf20 0x7f47a0070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 -- 172.21.15.141:0/6776966 >> 172.21.15.141:0/6776966 conn(0x7f47a006a7d0 msgr2=0x7f47a006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 -- 172.21.15.141:0/6776966 shutdown_connections 2024-01-16T15:04:36.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.415+0000 7f47a4e00640 1 -- 172.21.15.141:0/6776966 wait complete. 2024-01-16T15:04:36.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.417+0000 7f47a4e00640 1 Processor -- start 2024-01-16T15:04:36.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.417+0000 7f47a4e00640 1 -- start start 2024-01-16T15:04:36.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a00adf20 0x7f47a015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47a00cc980 0x7f47a00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a015bf10 0x7f47a00ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f47a4e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a013b410 con 0x7f47a00adf20 2024-01-16T15:04:36.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f47a4e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01f7440 con 0x7f47a00cc980 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f47a4e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01f7160 con 0x7f47a015bf10 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f479f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a00adf20 0x7f47a015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f479f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a00adf20 0x7f47a015a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57150/0 (socket says 172.21.15.141:57150) 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.418+0000 7f479f400640 1 -- 172.21.15.141:0/2290772656 learned_addr learned my addr 172.21.15.141:0/2290772656 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f4797e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47a00cc980 0x7f47a00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f479fe00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a015bf10 0x7f47a00ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f479f400640 1 -- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a015bf10 msgr2=0x7f47a00ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f479f400640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a015bf10 0x7f47a00ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f479f400640 1 -- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47a00cc980 msgr2=0x7f47a00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f479f400640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47a00cc980 0x7f47a00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f479f400640 1 -- 172.21.15.141:0/2290772656 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4788009400 con 0x7f47a00adf20 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f4797e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47a00cc980 0x7f47a00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f479f400640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a00adf20 0x7f47a015a4e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f479000d5f0 tx=0x7f479000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f4797400640 1 -- 172.21.15.141:0/2290772656 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47900125f0 con 0x7f47a00adf20 2024-01-16T15:04:36.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f4797400640 1 -- 172.21.15.141:0/2290772656 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790012750 con 0x7f47a00adf20 2024-01-16T15:04:36.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a00b20c0 con 0x7f47a00adf20 2024-01-16T15:04:36.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f4797400640 1 -- 172.21.15.141:0/2290772656 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790012b00 con 0x7f47a00adf20 2024-01-16T15:04:36.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.419+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a00b2680 con 0x7f47a00adf20 2024-01-16T15:04:36.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.421+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a00b2810 con 0x7f47a00adf20 2024-01-16T15:04:36.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.422+0000 7f4797400640 1 -- 172.21.15.141:0/2290772656 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4790012c60 con 0x7f47a00adf20 2024-01-16T15:04:36.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.422+0000 7f4797400640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f477806bb60 0x7f477806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.423+0000 7f4797400640 1 -- 172.21.15.141:0/2290772656 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4790088a50 con 0x7f47a00adf20 2024-01-16T15:04:36.402 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.423+0000 7f4797e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f477806bb60 0x7f477806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.424+0000 7f4797e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f477806bb60 0x7f477806e020 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4788036000 tx=0x7f478803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.427+0000 7f4797400640 1 -- 172.21.15.141:0/2290772656 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47900551a0 con 0x7f47a00adf20 2024-01-16T15:04:36.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.643+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f47a015ee90 con 0x7f477806bb60 2024-01-16T15:04:36.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.667+0000 7f4797400640 1 -- 172.21.15.141:0/2290772656 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7f4788031c20 con 0x7f477806bb60 2024-01-16T15:04:36.645 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone66727" 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone66724" 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone66726" 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "sv1" 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:36.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone66722" 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone66721" 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone66723" 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone66725" 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:36.647 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.669+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f477806bb60 msgr2=0x7f477806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.669+0000 7f47a4e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f477806bb60 0x7f477806e020 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4788036000 tx=0x7f478803d040 comp rx=0 tx=0).stop 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.669+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a00adf20 msgr2=0x7f47a015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.669+0000 7f47a4e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a00adf20 0x7f47a015a4e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f479000d5f0 tx=0x7f479000dab0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 shutdown_connections 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f477806bb60 0x7f477806e020 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f47a015bf10 0x7f47a00ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47a00cc980 0x7f47a00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 --2- 172.21.15.141:0/2290772656 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f47a00adf20 0x7f47a015a4e0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 >> 172.21.15.141:0/2290772656 conn(0x7f47a006a7d0 msgr2=0x7f47a00af280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 shutdown_connections 2024-01-16T15:04:36.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.670+0000 7f47a4e00640 1 -- 172.21.15.141:0/2290772656 wait complete. 2024-01-16T15:04:36.667 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66727 2024-01-16T15:04:36.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.905+0000 7f3e58a00640 1 Processor -- start 2024-01-16T15:04:36.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.905+0000 7f3e58a00640 1 -- start start 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e58a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3e540726f0 0x7f3e540b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e58a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e5415e030 0x7f3e5415e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e58a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5415dba0 con 0x7f3e54146650 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e58a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540cd130 con 0x7f3e5415e030 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e58a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5406f800 con 0x7f3e540726f0 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e53e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e5415e030 0x7f3e5415e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e53e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e5415e030 0x7f3e5415e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57154/0 (socket says 172.21.15.141:57154) 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e53e00640 1 -- 172.21.15.141:0/742663567 learned_addr learned my addr 172.21.15.141:0/742663567 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e53e00640 1 -- 172.21.15.141:0/742663567 >> v2:172.21.15.188:3300/0 conn(0x7f3e540726f0 msgr2=0x7f3e540b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e53e00640 1 --2- 172.21.15.141:0/742663567 >> v2:172.21.15.188:3300/0 conn(0x7f3e540726f0 0x7f3e540b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e53e00640 1 -- 172.21.15.141:0/742663567 >> v1:172.21.15.188:6789/0 conn(0x7f3e54146650 legacy=0x7f3e540cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:36.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.907+0000 7f3e53e00640 1 -- 172.21.15.141:0/742663567 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5406f240 con 0x7f3e5415e030 2024-01-16T15:04:36.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.909+0000 7f3e53e00640 1 --2- 172.21.15.141:0/742663567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e5415e030 0x7f3e5415e4e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3e5404f860 tx=0x7f3e4c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9465cba07ca261a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.909+0000 7f3e52a00640 1 -- 172.21.15.141:0/742663567 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c02ca80 con 0x7f3e5415e030 2024-01-16T15:04:36.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.909+0000 7f3e52a00640 1 -- 172.21.15.141:0/742663567 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c02cbe0 con 0x7f3e5415e030 2024-01-16T15:04:36.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.909+0000 7f3e52a00640 1 -- 172.21.15.141:0/742663567 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0343f0 con 0x7f3e5415e030 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 -- 172.21.15.141:0/742663567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e5415e030 msgr2=0x7f3e5415e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 --2- 172.21.15.141:0/742663567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e5415e030 0x7f3e5415e4e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3e5404f860 tx=0x7f3e4c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 -- 172.21.15.141:0/742663567 shutdown_connections 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 --2- 172.21.15.141:0/742663567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e5415e030 0x7f3e5415e4e0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 --2- 172.21.15.141:0/742663567 >> v2:172.21.15.188:3300/0 conn(0x7f3e540726f0 0x7f3e540b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 -- 172.21.15.141:0/742663567 >> 172.21.15.141:0/742663567 conn(0x7f3e540b1d40 msgr2=0x7f3e540b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 -- 172.21.15.141:0/742663567 shutdown_connections 2024-01-16T15:04:36.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.910+0000 7f3e58a00640 1 -- 172.21.15.141:0/742663567 wait complete. 2024-01-16T15:04:36.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.912+0000 7f3e58a00640 1 Processor -- start 2024-01-16T15:04:36.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.912+0000 7f3e58a00640 1 -- start start 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e58a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e540726f0 0x7f3e5415ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e58a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e54146650 0x7f3e541f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e58a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e5415e030 0x7f3e540b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e58a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54134140 con 0x7f3e540726f0 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e58a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e541334c0 con 0x7f3e54146650 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e58a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e541337a0 con 0x7f3e5415e030 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e53400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e540726f0 0x7f3e5415ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e53400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e540726f0 0x7f3e5415ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57158/0 (socket says 172.21.15.141:57158) 2024-01-16T15:04:36.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.913+0000 7f3e53400640 1 -- 172.21.15.141:0/33393680 learned_addr learned my addr 172.21.15.141:0/33393680 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:36.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e53e00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e5415e030 0x7f3e540b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e4be00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e54146650 0x7f3e541f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e53400640 1 -- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e5415e030 msgr2=0x7f3e540b0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e53400640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e5415e030 0x7f3e540b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e53400640 1 -- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e54146650 msgr2=0x7f3e541f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e53400640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e54146650 0x7f3e541f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e53400640 1 -- 172.21.15.141:0/33393680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c009b30 con 0x7f3e540726f0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e4be00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e54146650 0x7f3e541f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e53400640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e540726f0 0x7f3e5415ce30 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400b3e0 tx=0x7f3e4400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e4b400640 1 -- 172.21.15.141:0/33393680 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44011830 con 0x7f3e540726f0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e4b400640 1 -- 172.21.15.141:0/33393680 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e440119b0 con 0x7f3e540726f0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e4b400640 1 -- 172.21.15.141:0/33393680 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44011d60 con 0x7f3e540726f0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e54133a80 con 0x7f3e540726f0 2024-01-16T15:04:36.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.914+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5415d370 con 0x7f3e540726f0 2024-01-16T15:04:36.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.915+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e540ce600 con 0x7f3e540726f0 2024-01-16T15:04:36.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.916+0000 7f3e4b400640 1 -- 172.21.15.141:0/33393680 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f3e4401a400 con 0x7f3e540726f0 2024-01-16T15:04:36.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.918+0000 7f3e4b400640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e2c06bb30 0x7f3e2c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.918+0000 7f3e4b400640 1 -- 172.21.15.141:0/33393680 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e44088820 con 0x7f3e540726f0 2024-01-16T15:04:36.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.918+0000 7f3e4be00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e2c06bb30 0x7f3e2c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.919+0000 7f3e4be00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e2c06bb30 0x7f3e2c06dff0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3e5406af60 tx=0x7f3e3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:36.922+0000 7f3e4b400640 1 -- 172.21.15.141:0/33393680 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e440bde90 con 0x7f3e540726f0 2024-01-16T15:04:37.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.139+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66727", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e54161dc0 con 0x7f3e2c06bb30 2024-01-16T15:04:37.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.172+0000 7f3e4b400640 1 -- 172.21.15.141:0/33393680 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3e3c00adc0 con 0x7f3e2c06bb30 2024-01-16T15:04:37.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e2c06bb30 msgr2=0x7f3e2c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e2c06bb30 0x7f3e2c06dff0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3e5406af60 tx=0x7f3e3c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:37.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e540726f0 msgr2=0x7f3e5415ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e540726f0 0x7f3e5415ce30 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400b3e0 tx=0x7f3e4400b8a0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 shutdown_connections 2024-01-16T15:04:37.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e2c06bb30 0x7f3e2c06dff0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e5415e030 0x7f3e540b0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e54146650 0x7f3e541f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 --2- 172.21.15.141:0/33393680 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e540726f0 0x7f3e5415ce30 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 >> 172.21.15.141:0/33393680 conn(0x7f3e540b1d40 msgr2=0x7f3e54145210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 shutdown_connections 2024-01-16T15:04:37.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.175+0000 7f3e58a00640 1 -- 172.21.15.141:0/33393680 wait complete. 2024-01-16T15:04:37.162 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.183+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.162 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.183+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.162 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.183+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.162 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.184+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.162 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.184+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.174 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66724 2024-01-16T15:04:37.195 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.217+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.195 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.217+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.195 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.217+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.195 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.217+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.195 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:04:37.217+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:37.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.391+0000 7f24bde00640 1 Processor -- start 2024-01-16T15:04:37.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.392+0000 7f24bde00640 1 -- start start 2024-01-16T15:04:37.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.393+0000 7f24bde00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f24b8162810 0x7f24b8071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.393+0000 7f24bde00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f24b8071780 0x7f24b806c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.393+0000 7f24bde00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b8071c10 con 0x7f24b8162430 2024-01-16T15:04:37.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bde00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b80702e0 con 0x7f24b8071780 2024-01-16T15:04:37.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bde00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b80705c0 con 0x7f24b8162810 2024-01-16T15:04:37.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bd400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f24b8071780 0x7f24b806c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bd400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f24b8071780 0x7f24b806c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57174/0 (socket says 172.21.15.141:57174) 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bd400640 1 -- 172.21.15.141:0/2910478560 learned_addr learned my addr 172.21.15.141:0/2910478560 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24afc00640 1 --2- 172.21.15.141:0/2910478560 >> v2:172.21.15.188:3300/0 conn(0x7f24b8162810 0x7f24b8071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bd400640 1 -- 172.21.15.141:0/2910478560 >> v2:172.21.15.188:3300/0 conn(0x7f24b8162810 msgr2=0x7f24b8071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bd400640 1 --2- 172.21.15.141:0/2910478560 >> v2:172.21.15.188:3300/0 conn(0x7f24b8162810 0x7f24b8071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bd400640 1 -- 172.21.15.141:0/2910478560 >> v1:172.21.15.188:6789/0 conn(0x7f24b8162430 legacy=0x7f24b80b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24bd400640 1 -- 172.21.15.141:0/2910478560 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b80708a0 con 0x7f24b8071780 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.394+0000 7f24afc00640 1 --2- 172.21.15.141:0/2910478560 >> v2:172.21.15.188:3300/0 conn(0x7f24b8162810 0x7f24b8071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:37.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.395+0000 7f24bd400640 1 --2- 172.21.15.141:0/2910478560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b806c8c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f24b815c090 tx=0x7f24b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42418da4b2a68c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.396+0000 7f24b7e00640 1 -- 172.21.15.141:0/2910478560 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b002ca80 con 0x7f24b8071780 2024-01-16T15:04:37.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.396+0000 7f24b7e00640 1 -- 172.21.15.141:0/2910478560 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b002cbe0 con 0x7f24b8071780 2024-01-16T15:04:37.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.396+0000 7f24b7e00640 1 -- 172.21.15.141:0/2910478560 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b00343f0 con 0x7f24b8071780 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 -- 172.21.15.141:0/2910478560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 msgr2=0x7f24b806c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 --2- 172.21.15.141:0/2910478560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b806c8c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f24b815c090 tx=0x7f24b002bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 -- 172.21.15.141:0/2910478560 shutdown_connections 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 --2- 172.21.15.141:0/2910478560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b806c8c0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 --2- 172.21.15.141:0/2910478560 >> v2:172.21.15.188:3300/0 conn(0x7f24b8162810 0x7f24b8071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 -- 172.21.15.141:0/2910478560 >> 172.21.15.141:0/2910478560 conn(0x7f24b806a800 msgr2=0x7f24b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 -- 172.21.15.141:0/2910478560 shutdown_connections 2024-01-16T15:04:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.397+0000 7f24bde00640 1 -- 172.21.15.141:0/2910478560 wait complete. 2024-01-16T15:04:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.399+0000 7f24bde00640 1 Processor -- start 2024-01-16T15:04:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.399+0000 7f24bde00640 1 -- start start 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b8153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bde00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24b8162430 0x7f24b8159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bde00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24b8159cc0 0x7f24b80b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bde00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b8154450 con 0x7f24b8071780 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bde00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b81f7240 con 0x7f24b8162430 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bde00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b815aa00 con 0x7f24b8159cc0 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b8153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.400+0000 7f24bca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b8153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57180/0 (socket says 172.21.15.141:57180) 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bca00640 1 -- 172.21.15.141:0/1152993457 learned_addr learned my addr 172.21.15.141:0/1152993457 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bca00640 1 -- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24b8159cc0 msgr2=0x7f24b80b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bca00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24b8159cc0 0x7f24b80b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bca00640 1 -- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24b8162430 msgr2=0x7f24b8159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bca00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24b8162430 0x7f24b8159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bca00640 1 -- 172.21.15.141:0/1152993457 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b0009b30 con 0x7f24b8071780 2024-01-16T15:04:37.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bca00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b8153f10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c430 tx=0x7f24a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24b5600640 1 -- 172.21.15.141:0/1152993457 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8011790 con 0x7f24b8071780 2024-01-16T15:04:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24b5600640 1 -- 172.21.15.141:0/1152993457 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a80118f0 con 0x7f24b8071780 2024-01-16T15:04:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24b5600640 1 -- 172.21.15.141:0/1152993457 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8011ce0 con 0x7f24b8071780 2024-01-16T15:04:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.401+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b815a160 con 0x7f24b8071780 2024-01-16T15:04:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.402+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b815a720 con 0x7f24b8071780 2024-01-16T15:04:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.403+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b8067450 con 0x7f24b8071780 2024-01-16T15:04:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.404+0000 7f24b5600640 1 -- 172.21.15.141:0/1152993457 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f24a801c460 con 0x7f24b8071780 2024-01-16T15:04:37.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.405+0000 7f24b5600640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f249006bb30 0x7f249006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.405+0000 7f24b5600640 1 -- 172.21.15.141:0/1152993457 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24a8088580 con 0x7f24b8071780 2024-01-16T15:04:37.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.406+0000 7f24afc00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f249006bb30 0x7f249006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.407+0000 7f24afc00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f249006bb30 0x7f249006dff0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f24a0006fd0 tx=0x7f24a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.410+0000 7f24b5600640 1 -- 172.21.15.141:0/1152993457 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24a8054c70 con 0x7f24b8071780 2024-01-16T15:04:37.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.663+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66724", "target": ["mon-mgr", ""]}) v1 -- 0x7f24b80b3820 con 0x7f249006bb30 2024-01-16T15:04:37.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.702+0000 7f24b5600640 1 -- 172.21.15.141:0/1152993457 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f24a000ae20 con 0x7f249006bb30 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.705+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f249006bb30 msgr2=0x7f249006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.705+0000 7f24bde00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f249006bb30 0x7f249006dff0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f24a0006fd0 tx=0x7f24a0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.705+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 msgr2=0x7f24b8153f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.705+0000 7f24bde00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b8153f10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c430 tx=0x7f24a800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 shutdown_connections 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f249006bb30 0x7f249006dff0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f24b8159cc0 0x7f24b80b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24b8162430 0x7f24b8159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 --2- 172.21.15.141:0/1152993457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f24b8071780 0x7f24b8153f10 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 >> 172.21.15.141:0/1152993457 conn(0x7f24b806a800 msgr2=0x7f24b806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 shutdown_connections 2024-01-16T15:04:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.706+0000 7f24bde00640 1 -- 172.21.15.141:0/1152993457 wait complete. 2024-01-16T15:04:37.706 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66726 2024-01-16T15:04:37.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.927+0000 7f30aee00640 1 Processor -- start 2024-01-16T15:04:37.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.927+0000 7f30aee00640 1 -- start start 2024-01-16T15:04:37.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.928+0000 7f30aee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 0x7f30a8070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30aee00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a8071790 con 0x7f30a8159590 2024-01-16T15:04:37.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30aee00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a8071100 con 0x7f30a806d050 2024-01-16T15:04:37.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30aee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a8070860 con 0x7f30a80b3c20 2024-01-16T15:04:37.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30ada00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f30a8159590 0x7f30a80ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33266/0 (socket says 172.21.15.141:33266) 2024-01-16T15:04:37.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30ada00640 1 -- 172.21.15.141:0/3409488209 learned_addr learned my addr 172.21.15.141:0/3409488209 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:37.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979779238 0 0) 0x7f30a8071790 con 0x7f30a8159590 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30ad000640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 0x7f30a8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.929+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3088004630 con 0x7f30a8159590 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 950977111 0 0) 0x7f3088004630 con 0x7f30a8159590 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 msgr2=0x7f30a8070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 0x7f30a8070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 >> v1:172.21.15.188:6789/0 conn(0x7f30a806d050 legacy=0x7f30a806d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a8070b40 con 0x7f30a8159590 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f30980019f0 con 0x7f30a8159590 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 >> v1:172.21.15.141:6789/0 conn(0x7f30a8159590 legacy=0x7f30a80ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004ab0 0x7f3088004ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30880058d0 0x7f3088005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 0x7f3088006340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a8070860 con 0x7f3088004ab0 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308800cc20 con 0x7f30880058d0 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30880034c0 con 0x7f30a80b3c20 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30ada00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004ab0 0x7f3088004ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30ad000640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30880058d0 0x7f3088005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.930+0000 7f30ae400640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 0x7f3088006340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.931+0000 7f30ada00640 1 -- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 msgr2=0x7f3088006340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.931+0000 7f30ada00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a80b3c20 0x7f3088006340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.931+0000 7f30ada00640 1 -- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30880058d0 msgr2=0x7f3088005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.931+0000 7f30ada00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30880058d0 0x7f3088005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.931+0000 7f30ada00640 1 -- 172.21.15.141:0/3409488209 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3088004630 con 0x7f3088004ab0 2024-01-16T15:04:37.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.932+0000 7f30ada00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004ab0 0x7f3088004ed0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f309800a760 tx=0x7f3088006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c963d82d4e7ee312 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.932+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098034630 con 0x7f3088004ab0 2024-01-16T15:04:37.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.932+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309802f070 con 0x7f3088004ab0 2024-01-16T15:04:37.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.932+0000 7f30a7e00640 1 -- 172.21.15.141:0/3409488209 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098034a10 con 0x7f3088004ab0 2024-01-16T15:04:37.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.933+0000 7f30aee00640 1 -- 172.21.15.141:0/3409488209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004ab0 msgr2=0x7f3088004ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.933+0000 7f30aee00640 1 --2- 172.21.15.141:0/3409488209 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004ab0 0x7f3088004ed0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f309800a760 tx=0x7f3088006900 comp rx=0 tx=0).stop 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.933+0000 7f30ae400640 1 -- 172.21.15.141:0/3409488209 reap_dead start 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.933+0000 7f30aee00640 1 -- 172.21.15.141:0/3409488209 shutdown_connections 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.933+0000 7f30aee00640 1 -- 172.21.15.141:0/3409488209 >> 172.21.15.141:0/3409488209 conn(0x7f30a806a800 msgr2=0x7f30a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.933+0000 7f30aee00640 1 -- 172.21.15.141:0/3409488209 shutdown_connections 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.933+0000 7f30aee00640 1 -- 172.21.15.141:0/3409488209 wait complete. 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.935+0000 7f30aee00640 1 Processor -- start 2024-01-16T15:04:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.935+0000 7f30aee00640 1 -- start start 2024-01-16T15:04:37.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30aee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3088004ab0 0x7f30a813d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30aee00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30a80a98e0 0x7f30a813cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30aee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a813ea90 0x7f30a80aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30aee00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a813e670 con 0x7f30a80a98e0 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30aee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a813d9f0 con 0x7f3088004ab0 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30aee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a813dcd0 con 0x7f30a813ea90 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30a80a98e0 0x7f30a813cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30a80a98e0 0x7f30a813cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57198/0 (socket says 172.21.15.141:57198) 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 -- 172.21.15.141:0/1319302126 learned_addr learned my addr 172.21.15.141:0/1319302126 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ada00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3088004ab0 0x7f30a813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 -- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a813ea90 msgr2=0x7f30a80aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ae400640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a813ea90 0x7f30a80aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a813ea90 0x7f30a80aa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 -- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3088004ab0 msgr2=0x7f30a813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3088004ab0 0x7f30a813d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.936+0000 7f30ad000640 1 -- 172.21.15.141:0/1319302126 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a813dfb0 con 0x7f30a80a98e0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.937+0000 7f30ad000640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30a80a98e0 0x7f30a813cad0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3090011600 tx=0x7f3090011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.937+0000 7f30a5600640 1 -- 172.21.15.141:0/1319302126 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309000f920 con 0x7f30a80a98e0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.937+0000 7f30a5600640 1 -- 172.21.15.141:0/1319302126 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309000fa80 con 0x7f30a80a98e0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.937+0000 7f30a5600640 1 -- 172.21.15.141:0/1319302126 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309000fdf0 con 0x7f30a80a98e0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.937+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a813e290 con 0x7f30a80a98e0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.937+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a80a9d60 con 0x7f30a80a98e0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.938+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a8067450 con 0x7f30a80a98e0 2024-01-16T15:04:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.939+0000 7f30a5600640 1 -- 172.21.15.141:0/1319302126 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f3090015460 con 0x7f30a80a98e0 2024-01-16T15:04:37.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.940+0000 7f30a5600640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f308006bb30 0x7f308006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.940+0000 7f30a5600640 1 -- 172.21.15.141:0/1319302126 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3090088a50 con 0x7f30a80a98e0 2024-01-16T15:04:37.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.940+0000 7f30ada00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f308006bb30 0x7f308006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.941+0000 7f30ada00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f308006bb30 0x7f308006dff0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f309800a760 tx=0x7f309802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:37.944+0000 7f30a5600640 1 -- 172.21.15.141:0/1319302126 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3090055140 con 0x7f30a80a98e0 2024-01-16T15:04:38.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.160+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66726", "target": ["mon-mgr", ""]}) v1 -- 0x7f30a813de40 con 0x7f308006bb30 2024-01-16T15:04:38.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.202+0000 7f30a5600640 1 -- 172.21.15.141:0/1319302126 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f30a813de40 con 0x7f308006bb30 2024-01-16T15:04:38.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.204+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f308006bb30 msgr2=0x7f308006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f308006bb30 0x7f308006dff0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f309800a760 tx=0x7f309802f000 comp rx=0 tx=0).stop 2024-01-16T15:04:38.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30a80a98e0 msgr2=0x7f30a813cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30a80a98e0 0x7f30a813cad0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3090011600 tx=0x7f3090011ac0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 shutdown_connections 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f308006bb30 0x7f308006dff0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f30a813ea90 0x7f30a80aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30a80a98e0 0x7f30a813cad0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 --2- 172.21.15.141:0/1319302126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3088004ab0 0x7f30a813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 >> 172.21.15.141:0/1319302126 conn(0x7f30a806a800 msgr2=0x7f30a814e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 shutdown_connections 2024-01-16T15:04:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.205+0000 7f30aee00640 1 -- 172.21.15.141:0/1319302126 wait complete. 2024-01-16T15:04:38.205 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:04:38.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.439+0000 7fba22600640 1 Processor -- start 2024-01-16T15:04:38.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.440+0000 7fba22600640 1 -- start start 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.441+0000 7fba22600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fba1c0aeac0 0x7fba1c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.441+0000 7fba22600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c06f040 0x7fba1c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.441+0000 7fba22600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c0ce770 con 0x7fba1c15b090 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.441+0000 7fba22600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c0ccd40 con 0x7fba1c0aeac0 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.441+0000 7fba22600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0ce180 con 0x7fba1c06f040 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fba1c15b090 0x7fba1c0b26a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33282/0 (socket says 172.21.15.141:33282) 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1b400640 1 -- 172.21.15.141:0/2853958640 learned_addr learned my addr 172.21.15.141:0/2853958640 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1aa00640 1 --2- 172.21.15.141:0/2853958640 >> v2:172.21.15.141:3300/0 conn(0x7fba1c0aeac0 0x7fba1c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1be00640 1 --2- 172.21.15.141:0/2853958640 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c06f040 0x7fba1c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1a000640 1 -- 172.21.15.141:0/2853958640 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341805242 0 0) 0x7fba1c0ce770 con 0x7fba1c15b090 2024-01-16T15:04:38.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1a000640 1 -- 172.21.15.141:0/2853958640 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9fc003520 con 0x7fba1c15b090 2024-01-16T15:04:38.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1aa00640 1 -- 172.21.15.141:0/2853958640 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c06f040 msgr2=0x7fba1c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:38.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1aa00640 1 --2- 172.21.15.141:0/2853958640 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c06f040 0x7fba1c06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1aa00640 1 -- 172.21.15.141:0/2853958640 >> v1:172.21.15.141:6789/0 conn(0x7fba1c15b090 legacy=0x7fba1c0b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.442+0000 7fba1aa00640 1 -- 172.21.15.141:0/2853958640 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0715a0 con 0x7fba1c0aeac0 2024-01-16T15:04:38.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.444+0000 7fba1aa00640 1 --2- 172.21.15.141:0/2853958640 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c06b3f0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e896b9dc5fd689 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:38.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.444+0000 7fba1a000640 1 -- 172.21.15.141:0/2853958640 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c02cc50 con 0x7fba1c0aeac0 2024-01-16T15:04:38.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.444+0000 7fba1a000640 1 -- 172.21.15.141:0/2853958640 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c02cdb0 con 0x7fba1c0aeac0 2024-01-16T15:04:38.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.444+0000 7fba1a000640 1 -- 172.21.15.141:0/2853958640 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c034660 con 0x7fba1c0aeac0 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 -- 172.21.15.141:0/2853958640 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 msgr2=0x7fba1c06b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 --2- 172.21.15.141:0/2853958640 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c06b3f0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c02bea0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 -- 172.21.15.141:0/2853958640 shutdown_connections 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 --2- 172.21.15.141:0/2853958640 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c06f040 0x7fba1c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 --2- 172.21.15.141:0/2853958640 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c06b3f0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 -- 172.21.15.141:0/2853958640 >> 172.21.15.141:0/2853958640 conn(0x7fba1c070a80 msgr2=0x7fba1c070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 -- 172.21.15.141:0/2853958640 shutdown_connections 2024-01-16T15:04:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.445+0000 7fba22600640 1 -- 172.21.15.141:0/2853958640 wait complete. 2024-01-16T15:04:38.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.447+0000 7fba22600640 1 Processor -- start 2024-01-16T15:04:38.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.447+0000 7fba22600640 1 -- start start 2024-01-16T15:04:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba22600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba1c06f040 0x7fba1c14e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba22600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba22600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c15b090 0x7fba1c14f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba22600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c14c510 con 0x7fba1c0aeac0 2024-01-16T15:04:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba22600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c06dce0 con 0x7fba1c06f040 2024-01-16T15:04:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba22600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c14e710 con 0x7fba1c15b090 2024-01-16T15:04:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c15bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57226/0 (socket says 172.21.15.141:57226) 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 -- 172.21.15.141:0/1473340335 learned_addr learned my addr 172.21.15.141:0/1473340335 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1b400640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba1c06f040 0x7fba1c14e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1be00640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c15b090 0x7fba1c14f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 -- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c15b090 msgr2=0x7fba1c14f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c15b090 0x7fba1c14f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 -- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba1c06f040 msgr2=0x7fba1c14e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba1c06f040 0x7fba1c14e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.449+0000 7fba1aa00640 1 -- 172.21.15.141:0/1473340335 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c009a70 con 0x7fba1c0aeac0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.450+0000 7fba1aa00640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c15bff0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.450+0000 7fba13400640 1 -- 172.21.15.141:0/1473340335 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c040650 con 0x7fba1c0aeac0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.450+0000 7fba13400640 1 -- 172.21.15.141:0/1473340335 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c0407b0 con 0x7fba1c0aeac0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.450+0000 7fba13400640 1 -- 172.21.15.141:0/1473340335 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c040b60 con 0x7fba1c0aeac0 2024-01-16T15:04:38.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.450+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c14e9f0 con 0x7fba1c0aeac0 2024-01-16T15:04:38.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.450+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c14efb0 con 0x7fba1c0aeac0 2024-01-16T15:04:38.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.451+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c04ed60 con 0x7fba1c0aeac0 2024-01-16T15:04:38.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.452+0000 7fba13400640 1 -- 172.21.15.141:0/1473340335 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fba0c040cc0 con 0x7fba1c0aeac0 2024-01-16T15:04:38.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.453+0000 7fba13400640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9f406bb30 0x7fb9f406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.453+0000 7fba13400640 1 -- 172.21.15.141:0/1473340335 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba0c0ab470 con 0x7fba1c0aeac0 2024-01-16T15:04:38.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.454+0000 7fba1b400640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9f406bb30 0x7fb9f406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.455+0000 7fba1b400640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9f406bb30 0x7fb9f406dff0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba040011e0 tx=0x7fba04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:38.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.458+0000 7fba13400640 1 -- 172.21.15.141:0/1473340335 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba0c077b60 con 0x7fba1c0aeac0 2024-01-16T15:04:38.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:38.685+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba1c0ce400 con 0x7fb9f406bb30 2024-01-16T15:04:39.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.235+0000 7fba13400640 1 -- 172.21.15.141:0/1473340335 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fba04001350 con 0x7fb9f406bb30 2024-01-16T15:04:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:39.214 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:04:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:04:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1" 2024-01-16T15:04:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:04:39.215 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:04:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9f406bb30 msgr2=0x7fb9f406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9f406bb30 0x7fb9f406dff0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba040011e0 tx=0x7fba04009040 comp rx=0 tx=0).stop 2024-01-16T15:04:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 msgr2=0x7fba1c15bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c15bff0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 shutdown_connections 2024-01-16T15:04:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9f406bb30 0x7fb9f406dff0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba1c15b090 0x7fba1c14f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba1c0aeac0 0x7fba1c15bff0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 --2- 172.21.15.141:0/1473340335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba1c06f040 0x7fba1c14e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 >> 172.21.15.141:0/1473340335 conn(0x7fba1c070a80 msgr2=0x7fba1c070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:39.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 shutdown_connections 2024-01-16T15:04:39.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.239+0000 7fba22600640 1 -- 172.21.15.141:0/1473340335 wait complete. 2024-01-16T15:04:39.238 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:04:39.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.479+0000 7f90a1000640 1 Processor -- start 2024-01-16T15:04:39.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.479+0000 7f90a1000640 1 -- start start 2024-01-16T15:04:39.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.480+0000 7f90a1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f909c15d700 0x7f909c0ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f90a1000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c0cf240 con 0x7f909c15d320 2024-01-16T15:04:39.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f90a1000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c0d1c50 con 0x7f909c0cedd0 2024-01-16T15:04:39.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f90a1000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c0d1f30 con 0x7f909c15d700 2024-01-16T15:04:39.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f9099400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f909c15d700 0x7f909c0ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f9099e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f909c15d320 0x7f909c0c6d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33296/0 (socket says 172.21.15.141:33296) 2024-01-16T15:04:39.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f9099e00640 1 -- 172.21.15.141:0/3032424743 learned_addr learned my addr 172.21.15.141:0/3032424743 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:39.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f9099400640 1 -- 172.21.15.141:0/3032424743 >> v1:172.21.15.188:6789/0 conn(0x7f909c0cedd0 legacy=0x7f909c0d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:39.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f9099400640 1 -- 172.21.15.141:0/3032424743 >> v1:172.21.15.141:6789/0 conn(0x7f909c15d320 legacy=0x7f909c0c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f9098a00640 1 -- 172.21.15.141:0/3032424743 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219135858 0 0) 0x7f909c0cf240 con 0x7f909c15d320 2024-01-16T15:04:39.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.481+0000 7f9099400640 1 -- 172.21.15.141:0/3032424743 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0ce090 con 0x7f909c15d700 2024-01-16T15:04:39.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f9099400640 1 --2- 172.21.15.141:0/3032424743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c0ce890 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f908402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43f047219810927 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:39.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f9098a00640 1 -- 172.21.15.141:0/3032424743 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908402e070 con 0x7f909c15d700 2024-01-16T15:04:39.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f9098a00640 1 -- 172.21.15.141:0/3032424743 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90840315a0 con 0x7f909c15d700 2024-01-16T15:04:39.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f9098a00640 1 -- 172.21.15.141:0/3032424743 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084031930 con 0x7f909c15d700 2024-01-16T15:04:39.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f90a1000640 1 -- 172.21.15.141:0/3032424743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 msgr2=0x7f909c0ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f90a1000640 1 --2- 172.21.15.141:0/3032424743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c0ce890 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f908402b840 comp rx=0 tx=0).stop 2024-01-16T15:04:39.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f90a1000640 1 -- 172.21.15.141:0/3032424743 shutdown_connections 2024-01-16T15:04:39.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f90a1000640 1 --2- 172.21.15.141:0/3032424743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c0ce890 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.483+0000 7f90a1000640 1 -- 172.21.15.141:0/3032424743 >> 172.21.15.141:0/3032424743 conn(0x7f909c08cc50 msgr2=0x7f909c09c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:39.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.484+0000 7f90a1000640 1 -- 172.21.15.141:0/3032424743 shutdown_connections 2024-01-16T15:04:39.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.484+0000 7f90a1000640 1 -- 172.21.15.141:0/3032424743 wait complete. 2024-01-16T15:04:39.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.485+0000 7f90a1000640 1 Processor -- start 2024-01-16T15:04:39.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.485+0000 7f90a1000640 1 -- start start 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.486+0000 7f90a1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f909c0cedd0 0x7f909c0cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.486+0000 7f90a1000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f909c15d320 0x7f909c157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.486+0000 7f90a1000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.486+0000 7f90a1000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1f3f30 con 0x7f909c15d700 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.486+0000 7f90a1000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1f41f0 con 0x7f909c0cedd0 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.486+0000 7f90a1000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1f44b0 con 0x7f909c15d320 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57236/0 (socket says 172.21.15.141:57236) 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 -- 172.21.15.141:0/3554186996 learned_addr learned my addr 172.21.15.141:0/3554186996 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:39.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f9099400640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f909c15d320 0x7f909c157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f9099e00640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f909c0cedd0 0x7f909c0cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 -- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f909c15d320 msgr2=0x7f909c157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f909c15d320 0x7f909c157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 -- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f909c0cedd0 msgr2=0x7f909c0cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f909c0cedd0 0x7f909c0cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 -- 172.21.15.141:0/3554186996 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084009400 con 0x7f909c15d700 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f9099e00640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f909c0cedd0 0x7f909c0cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.487+0000 7f909a800640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c157a70 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f909400c780 tx=0x7f909400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:39.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.488+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094004b60 con 0x7f909c15d700 2024-01-16T15:04:39.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.488+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909400ce90 con 0x7f909c15d700 2024-01-16T15:04:39.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.488+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909400a850 con 0x7f909c15d700 2024-01-16T15:04:39.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.488+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c1f4790 con 0x7f909c15d700 2024-01-16T15:04:39.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.488+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c06cac0 con 0x7f909c15d700 2024-01-16T15:04:39.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.489+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c0675d0 con 0x7f909c15d700 2024-01-16T15:04:39.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.490+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f909400a9b0 con 0x7f909c15d700 2024-01-16T15:04:39.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.491+0000 7f9092000640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f907406bb30 0x7f907406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.491+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9094088800 con 0x7f909c15d700 2024-01-16T15:04:39.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.492+0000 7f9099e00640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f907406bb30 0x7f907406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.493+0000 7f9099e00640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f907406bb30 0x7f907406dff0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f908c000f60 tx=0x7f908c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:39.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.496+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9094054ef0 con 0x7f909c15d700 2024-01-16T15:04:39.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.699+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f909408e030 con 0x7f909c15d700 2024-01-16T15:04:39.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.724+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f909c0cdd40 con 0x7f907406bb30 2024-01-16T15:04:39.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.797+0000 7f9092000640 1 -- 172.21.15.141:0/3554186996 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f908c0013d0 con 0x7f907406bb30 2024-01-16T15:04:39.776 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:39.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f907406bb30 msgr2=0x7f907406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f907406bb30 0x7f907406dff0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f908c000f60 tx=0x7f908c009040 comp rx=0 tx=0).stop 2024-01-16T15:04:39.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 msgr2=0x7f909c157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c157a70 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f909400c780 tx=0x7f909400cc40 comp rx=0 tx=0).stop 2024-01-16T15:04:39.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 shutdown_connections 2024-01-16T15:04:39.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f907406bb30 0x7f907406dff0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f909c15d700 0x7f909c157a70 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f909c15d320 0x7f909c157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 --2- 172.21.15.141:0/3554186996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f909c0cedd0 0x7f909c0cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 >> 172.21.15.141:0/3554186996 conn(0x7f909c08cc50 msgr2=0x7f909c0cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:39.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 shutdown_connections 2024-01-16T15:04:39.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:39.801+0000 7f90a1000640 1 -- 172.21.15.141:0/3554186996 wait complete. 2024-01-16T15:04:39.799 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:04:40.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.082+0000 7f5cfb400640 1 Processor -- start 2024-01-16T15:04:40.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.082+0000 7f5cfb400640 1 -- start start 2024-01-16T15:04:40.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.084+0000 7f5cfb400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf414b2c0 0x7f5cf40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.084+0000 7f5cfb400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5cf40cf7d0 0x7f5cf4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.084+0000 7f5cfb400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40cfbf0 con 0x7f5cf4070ea0 2024-01-16T15:04:40.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.084+0000 7f5cfb400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf414b6a0 con 0x7f5cf40cf7d0 2024-01-16T15:04:40.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.084+0000 7f5cfb400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf4071690 con 0x7f5cf414b2c0 2024-01-16T15:04:40.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.085+0000 7f5cf8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf414b2c0 0x7f5cf40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.085+0000 7f5cf9400640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f5cf4070ea0 0x7f5cf40b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:48412/0 (socket says 172.21.15.141:48412) 2024-01-16T15:04:40.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.085+0000 7f5cf9400640 1 -- 172.21.15.141:0/1023206889 learned_addr learned my addr 172.21.15.141:0/1023206889 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:40.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.085+0000 7f5cf3e00640 1 --2- 172.21.15.141:0/1023206889 >> v2:172.21.15.141:3300/0 conn(0x7f5cf40cf7d0 0x7f5cf4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.086+0000 7f5cf8a00640 1 -- 172.21.15.141:0/1023206889 >> v2:172.21.15.141:3300/0 conn(0x7f5cf40cf7d0 msgr2=0x7f5cf4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.086+0000 7f5cf8a00640 1 --2- 172.21.15.141:0/1023206889 >> v2:172.21.15.141:3300/0 conn(0x7f5cf40cf7d0 0x7f5cf4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.086+0000 7f5cf8a00640 1 -- 172.21.15.141:0/1023206889 >> v1:172.21.15.188:6789/0 conn(0x7f5cf4070ea0 legacy=0x7f5cf40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.086+0000 7f5cf3400640 1 -- 172.21.15.141:0/1023206889 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070415359 0 0) 0x7f5cf40cfbf0 con 0x7f5cf4070ea0 2024-01-16T15:04:40.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.086+0000 7f5cf8a00640 1 -- 172.21.15.141:0/1023206889 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4071970 con 0x7f5cf414b2c0 2024-01-16T15:04:40.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.089+0000 7f5cf8a00640 1 --2- 172.21.15.141:0/1023206889 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf414b2c0 0x7f5cf40cf290 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009d90 tx=0x7f5cec02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d44214d72be013a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.091+0000 7f5cf3400640 1 -- 172.21.15.141:0/1023206889 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec02cc60 con 0x7f5cf414b2c0 2024-01-16T15:04:40.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.091+0000 7f5cf3400640 1 -- 172.21.15.141:0/1023206889 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec02cdc0 con 0x7f5cf414b2c0 2024-01-16T15:04:40.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.091+0000 7f5cf3400640 1 -- 172.21.15.141:0/1023206889 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec034660 con 0x7f5cf414b2c0 2024-01-16T15:04:40.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.092+0000 7f5cfb400640 1 -- 172.21.15.141:0/1023206889 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf414b2c0 msgr2=0x7f5cf40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.092+0000 7f5cfb400640 1 --2- 172.21.15.141:0/1023206889 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf414b2c0 0x7f5cf40cf290 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009d90 tx=0x7f5cec02beb0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.095+0000 7f5cfb400640 1 -- 172.21.15.141:0/1023206889 shutdown_connections 2024-01-16T15:04:40.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.095+0000 7f5cfb400640 1 --2- 172.21.15.141:0/1023206889 >> v2:172.21.15.141:3300/0 conn(0x7f5cf40cf7d0 0x7f5cf4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.095+0000 7f5cfb400640 1 --2- 172.21.15.141:0/1023206889 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf414b2c0 0x7f5cf40cf290 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.095+0000 7f5cfb400640 1 -- 172.21.15.141:0/1023206889 >> 172.21.15.141:0/1023206889 conn(0x7f5cf406a800 msgr2=0x7f5cf406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.096+0000 7f5cfb400640 1 -- 172.21.15.141:0/1023206889 shutdown_connections 2024-01-16T15:04:40.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.098+0000 7f5cfb400640 1 -- 172.21.15.141:0/1023206889 wait complete. 2024-01-16T15:04:40.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.101+0000 7f5cfb400640 1 Processor -- start 2024-01-16T15:04:40.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.102+0000 7f5cfb400640 1 -- start start 2024-01-16T15:04:40.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.104+0000 7f5cfb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cf4070ea0 0x7f5cf41ff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.104+0000 7f5cfb400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf40cf7d0 0x7f5cf4164960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.104+0000 7f5cfb400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5cf414b2c0 0x7f5cf4144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.104+0000 7f5cfb400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf4164ea0 con 0x7f5cf4070ea0 2024-01-16T15:04:40.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.104+0000 7f5cfb400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf4145010 con 0x7f5cf414b2c0 2024-01-16T15:04:40.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.104+0000 7f5cfb400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf4142ef0 con 0x7f5cf40cf7d0 2024-01-16T15:04:40.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.105+0000 7f5cf8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cf4070ea0 0x7f5cf41ff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.105+0000 7f5cf8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cf4070ea0 0x7f5cf41ff6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57258/0 (socket says 172.21.15.141:57258) 2024-01-16T15:04:40.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.105+0000 7f5cf8a00640 1 -- 172.21.15.141:0/3711881165 learned_addr learned my addr 172.21.15.141:0/3711881165 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:40.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.105+0000 7f5cf3e00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf40cf7d0 0x7f5cf4164960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.105+0000 7f5cf9400640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5cf414b2c0 0x7f5cf4144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.106+0000 7f5cf8a00640 1 -- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf40cf7d0 msgr2=0x7f5cf4164960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.106+0000 7f5cf8a00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf40cf7d0 0x7f5cf4164960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.106+0000 7f5cf8a00640 1 -- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5cf414b2c0 msgr2=0x7f5cf4144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.106+0000 7f5cf8a00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5cf414b2c0 0x7f5cf4144ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.106+0000 7f5cf8a00640 1 -- 172.21.15.141:0/3711881165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec009a70 con 0x7f5cf4070ea0 2024-01-16T15:04:40.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.106+0000 7f5cf3e00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf40cf7d0 0x7f5cf4164960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:40.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.106+0000 7f5cf9400640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5cf414b2c0 0x7f5cf4144ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:40.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.107+0000 7f5cf8a00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cf4070ea0 0x7f5cf41ff6c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009d90 tx=0x7f5cec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.109+0000 7f5cf0c00640 1 -- 172.21.15.141:0/3711881165 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec03e610 con 0x7f5cf4070ea0 2024-01-16T15:04:40.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.110+0000 7f5cf0c00640 1 -- 172.21.15.141:0/3711881165 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec03e790 con 0x7f5cf4070ea0 2024-01-16T15:04:40.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.110+0000 7f5cf0c00640 1 -- 172.21.15.141:0/3711881165 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec03eb40 con 0x7f5cf4070ea0 2024-01-16T15:04:40.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.110+0000 7f5cfb400640 1 -- 172.21.15.141:0/3711881165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf4145240 con 0x7f5cf4070ea0 2024-01-16T15:04:40.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.110+0000 7f5cfb400640 1 -- 172.21.15.141:0/3711881165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf4145800 con 0x7f5cf4070ea0 2024-01-16T15:04:40.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.113+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cbc005250 con 0x7f5cf4070ea0 2024-01-16T15:04:40.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.115+0000 7f5cf0c00640 1 -- 172.21.15.141:0/3711881165 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f5cec03b050 con 0x7f5cf4070ea0 2024-01-16T15:04:40.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.116+0000 7f5cf0c00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ccc06bc50 0x7f5ccc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.117+0000 7f5cf0c00640 1 -- 172.21.15.141:0/3711881165 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cec0aa870 con 0x7f5cf4070ea0 2024-01-16T15:04:40.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.117+0000 7f5cf3e00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ccc06bc50 0x7f5ccc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.119+0000 7f5cf3e00640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ccc06bc50 0x7f5ccc06e110 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40adfc0 tx=0x7f5cdc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.126+0000 7f5cf0c00640 1 -- 172.21.15.141:0/3711881165 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5cec076f30 con 0x7f5cf4070ea0 2024-01-16T15:04:40.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.508+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cbc000bc0 con 0x7f5ccc06bc50 2024-01-16T15:04:40.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.545+0000 7f5cf0c00640 1 -- 172.21.15.141:0/3711881165 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5cbc000bc0 con 0x7f5ccc06bc50 2024-01-16T15:04:40.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.549+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ccc06bc50 msgr2=0x7f5ccc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.549+0000 7f5cea000640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ccc06bc50 0x7f5ccc06e110 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40adfc0 tx=0x7f5cdc008040 comp rx=0 tx=0).stop 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cf4070ea0 msgr2=0x7f5cf41ff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cf4070ea0 0x7f5cf41ff6c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009d90 tx=0x7f5cec004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 shutdown_connections 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ccc06bc50 0x7f5ccc06e110 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5cf414b2c0 0x7f5cf4144ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5cf40cf7d0 0x7f5cf4164960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 --2- 172.21.15.141:0/3711881165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cf4070ea0 0x7f5cf41ff6c0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 >> 172.21.15.141:0/3711881165 conn(0x7f5cf406a800 msgr2=0x7f5cf41490e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 shutdown_connections 2024-01-16T15:04:40.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.550+0000 7f5cea000640 1 -- 172.21.15.141:0/3711881165 wait complete. 2024-01-16T15:04:40.555 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66722 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.884+0000 7f4dd7000640 1 Processor -- start 2024-01-16T15:04:40.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.884+0000 7f4dd7000640 1 -- start start 2024-01-16T15:04:40.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.886+0000 7f4dd7000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4dd00b7bf0 0x7f4dd00ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.886+0000 7f4dd7000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00b6750 con 0x7f4dd015d590 2024-01-16T15:04:40.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.886+0000 7f4dd7000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00cf1f0 con 0x7f4dd00d0280 2024-01-16T15:04:40.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.886+0000 7f4dd7000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00cef10 con 0x7f4dd00b7bf0 2024-01-16T15:04:40.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.887+0000 7f4dd5000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4dd00b7bf0 0x7f4dd00ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.887+0000 7f4dcfe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4dd015d590 0x7f4dd00cfb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33300/0 (socket says 172.21.15.141:33300) 2024-01-16T15:04:40.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.887+0000 7f4dcfe00640 1 -- 172.21.15.141:0/331251786 learned_addr learned my addr 172.21.15.141:0/331251786 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:40.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.887+0000 7f4dcf400640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f4dd00d0280 0x7f4dd00ce1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:48420/0 (socket says 172.21.15.141:48420) 2024-01-16T15:04:40.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192724383 0 0) 0x7f4dd00b6750 con 0x7f4dd015d590 2024-01-16T15:04:40.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db0003520 con 0x7f4dd015d590 2024-01-16T15:04:40.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663377245 0 0) 0x7f4dd00cf1f0 con 0x7f4dd00d0280 2024-01-16T15:04:40.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd00b6750 con 0x7f4dd00d0280 2024-01-16T15:04:40.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971507989 0 0) 0x7f4dd00b6750 con 0x7f4dd00d0280 2024-01-16T15:04:40.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 >> v2:172.21.15.188:3300/0 conn(0x7f4dd00b7bf0 msgr2=0x7f4dd00ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 --2- 172.21.15.141:0/331251786 >> v2:172.21.15.188:3300/0 conn(0x7f4dd00b7bf0 0x7f4dd00ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 >> v1:172.21.15.141:6789/0 conn(0x7f4dd015d590 legacy=0x7f4dd00cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00b6100 con 0x7f4dd00d0280 2024-01-16T15:04:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4dc0001670 con 0x7f4dd00d0280 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 >> v1:172.21.15.188:6789/0 conn(0x7f4dd00d0280 legacy=0x7f4dd00ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4db0004b60 0x7f4db0004f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.888+0000 7f4dcea00640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4db0003760 0x7f4db0003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcea00640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0004120 0x7f4db000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00cef10 con 0x7f4db0004120 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db000c000 con 0x7f4db0004b60 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db000c1c0 con 0x7f4db0003760 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcf400640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4db0003760 0x7f4db0003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcfe00640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4db0004b60 0x7f4db0004f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dd5000640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0004120 0x7f4db000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcf400640 1 -- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4db0004b60 msgr2=0x7f4db0004f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcf400640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4db0004b60 0x7f4db0004f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.889+0000 7f4dcf400640 1 -- 172.21.15.141:0/331251786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0004120 msgr2=0x7f4db000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.890+0000 7f4dcf400640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0004120 0x7f4db000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.890+0000 7f4dcf400640 1 -- 172.21.15.141:0/331251786 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00b6100 con 0x7f4db0003760 2024-01-16T15:04:40.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.890+0000 7f4dd5000640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0004120 0x7f4db000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:40.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.890+0000 7f4dd5000640 1 -- 172.21.15.141:0/331251786 reap_dead start 2024-01-16T15:04:40.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.891+0000 7f4dcf400640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4db0003760 0x7f4db0003be0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000c0e0 tx=0x7f4db00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de2a5e6c6694453d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.891+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0007270 con 0x7f4db0003760 2024-01-16T15:04:40.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.891+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc00073f0 con 0x7f4db0003760 2024-01-16T15:04:40.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.891+0000 7f4dcea00640 1 -- 172.21.15.141:0/331251786 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0005810 con 0x7f4db0003760 2024-01-16T15:04:40.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.892+0000 7f4dd7000640 1 -- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4db0003760 msgr2=0x7f4db0003be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.892+0000 7f4dd7000640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4db0003760 0x7f4db0003be0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000c0e0 tx=0x7f4db00057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.893+0000 7f4dd7000640 1 -- 172.21.15.141:0/331251786 shutdown_connections 2024-01-16T15:04:40.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.893+0000 7f4dd7000640 1 --2- 172.21.15.141:0/331251786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4db0003760 0x7f4db0003be0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.893+0000 7f4dd7000640 1 -- 172.21.15.141:0/331251786 >> 172.21.15.141:0/331251786 conn(0x7f4dd009c540 msgr2=0x7f4dd00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.893+0000 7f4dd7000640 1 -- 172.21.15.141:0/331251786 shutdown_connections 2024-01-16T15:04:40.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.893+0000 7f4dd7000640 1 -- 172.21.15.141:0/331251786 wait complete. 2024-01-16T15:04:40.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.895+0000 7f4dd7000640 1 Processor -- start 2024-01-16T15:04:40.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.896+0000 7f4dd7000640 1 -- start start 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.897+0000 7f4dd7000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0003760 0x7f4dd0157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.897+0000 7f4dd7000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd015e6a0 0x7f4dd015eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.897+0000 7f4dd7000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd00c6b70 0x7f4dd00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.897+0000 7f4dd7000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0157d60 con 0x7f4db0003760 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.897+0000 7f4dd7000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c7530 con 0x7f4dd015e6a0 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.897+0000 7f4dd7000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd015fe20 con 0x7f4dd00c6b70 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0003760 0x7f4dd0157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0003760 0x7f4dd0157820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57282/0 (socket says 172.21.15.141:57282) 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 -- 172.21.15.141:0/1953642515 learned_addr learned my addr 172.21.15.141:0/1953642515 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dd5000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd00c6b70 0x7f4dd00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcf400640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd015e6a0 0x7f4dd015eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 -- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd00c6b70 msgr2=0x7f4dd00c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd00c6b70 0x7f4dd00c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 -- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd015e6a0 msgr2=0x7f4dd015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd015e6a0 0x7f4dd015eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 -- 172.21.15.141:0/1953642515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd015f210 con 0x7f4db0003760 2024-01-16T15:04:40.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.898+0000 7f4dcfe00640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0003760 0x7f4dd0157820 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4db8001780 tx=0x7f4db8001c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.899+0000 7f4dc7e00640 1 -- 172.21.15.141:0/1953642515 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8007740 con 0x7f4db0003760 2024-01-16T15:04:40.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.899+0000 7f4dc7e00640 1 -- 172.21.15.141:0/1953642515 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db800f9c0 con 0x7f4db0003760 2024-01-16T15:04:40.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.899+0000 7f4dc7e00640 1 -- 172.21.15.141:0/1953642515 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db800fd50 con 0x7f4db0003760 2024-01-16T15:04:40.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.899+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd015f4f0 con 0x7f4db0003760 2024-01-16T15:04:40.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.900+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd015fab0 con 0x7f4db0003760 2024-01-16T15:04:40.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.901+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd015fc70 con 0x7f4db0003760 2024-01-16T15:04:40.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.901+0000 7f4dc7e00640 1 -- 172.21.15.141:0/1953642515 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f4db801d460 con 0x7f4db0003760 2024-01-16T15:04:40.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.902+0000 7f4dc7e00640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4da806bc80 0x7f4da806e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.902+0000 7f4dc7e00640 1 -- 172.21.15.141:0/1953642515 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4db8088580 con 0x7f4db0003760 2024-01-16T15:04:40.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.902+0000 7f4dcf400640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4da806bc80 0x7f4da806e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.903+0000 7f4dcf400640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4da806bc80 0x7f4da806e140 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0034000 tx=0x7f4dc0009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:40.911+0000 7f4dc7e00640 1 -- 172.21.15.141:0/1953642515 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4db8054d50 con 0x7f4db0003760 2024-01-16T15:04:41.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.278+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66722", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dd015d970 con 0x7f4da806bc80 2024-01-16T15:04:41.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.313+0000 7f4dc7e00640 1 -- 172.21.15.141:0/1953642515 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4dc003a070 con 0x7f4da806bc80 2024-01-16T15:04:41.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.317+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4da806bc80 msgr2=0x7f4da806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.317+0000 7f4dd7000640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4da806bc80 0x7f4da806e140 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0034000 tx=0x7f4dc0009030 comp rx=0 tx=0).stop 2024-01-16T15:04:41.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.317+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0003760 msgr2=0x7f4dd0157820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.317+0000 7f4dd7000640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0003760 0x7f4dd0157820 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4db8001780 tx=0x7f4db8001c40 comp rx=0 tx=0).stop 2024-01-16T15:04:41.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.318+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 shutdown_connections 2024-01-16T15:04:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.318+0000 7f4dd7000640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4da806bc80 0x7f4da806e140 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.318+0000 7f4dd7000640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd00c6b70 0x7f4dd00c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.318+0000 7f4dd7000640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd015e6a0 0x7f4dd015eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.318+0000 7f4dd7000640 1 --2- 172.21.15.141:0/1953642515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4db0003760 0x7f4dd0157820 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.318+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 >> 172.21.15.141:0/1953642515 conn(0x7f4dd009c540 msgr2=0x7f4dd00b8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.318+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 shutdown_connections 2024-01-16T15:04:41.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.319+0000 7f4dd7000640 1 -- 172.21.15.141:0/1953642515 wait complete. 2024-01-16T15:04:41.326 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66721 2024-01-16T15:04:41.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.557+0000 7faf91200640 1 Processor -- start 2024-01-16T15:04:41.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.557+0000 7faf91200640 1 -- start start 2024-01-16T15:04:41.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.558+0000 7faf91200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c071290 0x7faf8c0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.558+0000 7faf91200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c071b90 con 0x7faf8c155540 2024-01-16T15:04:41.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.558+0000 7faf91200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0af1e0 con 0x7faf8c15b470 2024-01-16T15:04:41.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.558+0000 7faf91200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c155920 con 0x7faf8c071290 2024-01-16T15:04:41.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7faf8c155540 0x7faf8c070b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33306/0 (socket says 172.21.15.141:33306) 2024-01-16T15:04:41.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8b400640 1 -- 172.21.15.141:0/2921648742 learned_addr learned my addr 172.21.15.141:0/2921648742 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8aa00640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c071290 0x7faf8c0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537166570 0 0) 0x7faf8c071b90 con 0x7faf8c155540 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c003520 con 0x7faf8c155540 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1269768156 0 0) 0x7faf6c003520 con 0x7faf8c155540 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c071290 msgr2=0x7faf8c0aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c071290 0x7faf8c0aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 >> v1:172.21.15.188:6789/0 conn(0x7faf8c15b470 legacy=0x7faf8c0b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c15b9e0 con 0x7faf8c155540 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7faf7c001670 con 0x7faf8c155540 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 >> v1:172.21.15.141:6789/0 conn(0x7faf8c155540 legacy=0x7faf8c070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf6c0039a0 0x7faf6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf6c0047c0 0x7faf6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c155920 con 0x7faf6c005180 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.559+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c00c080 con 0x7faf6c0047c0 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c00c240 con 0x7faf6c0039a0 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8aa00640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c071290 0x7faf8c0aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8be00640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8b400640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf6c0039a0 0x7faf6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8aa00640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf6c0047c0 0x7faf6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8be00640 1 -- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf6c0039a0 msgr2=0x7faf6c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8be00640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf6c0039a0 0x7faf6c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8be00640 1 -- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf6c0047c0 msgr2=0x7faf6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8be00640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf6c0047c0 0x7faf6c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8be00640 1 -- 172.21.15.141:0/2921648742 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c003520 con 0x7faf6c005180 2024-01-16T15:04:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.560+0000 7faf8b400640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf6c0039a0 0x7faf6c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.561+0000 7faf8be00640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf6c00b980 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7faf84001e50 tx=0x7faf6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ba9492d5f99944 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84003b70 con 0x7faf6c005180 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf84003cd0 con 0x7faf6c005180 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf8a000640 1 -- 172.21.15.141:0/2921648742 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84005310 con 0x7faf6c005180 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf91200640 1 -- 172.21.15.141:0/2921648742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 msgr2=0x7faf6c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf91200640 1 --2- 172.21.15.141:0/2921648742 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf6c00b980 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7faf84001e50 tx=0x7faf6c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf8be00640 1 -- 172.21.15.141:0/2921648742 reap_dead start 2024-01-16T15:04:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf91200640 1 -- 172.21.15.141:0/2921648742 shutdown_connections 2024-01-16T15:04:41.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.562+0000 7faf91200640 1 -- 172.21.15.141:0/2921648742 >> 172.21.15.141:0/2921648742 conn(0x7faf8c0cc1d0 msgr2=0x7faf8c0cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.563+0000 7faf91200640 1 -- 172.21.15.141:0/2921648742 shutdown_connections 2024-01-16T15:04:41.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.563+0000 7faf91200640 1 -- 172.21.15.141:0/2921648742 wait complete. 2024-01-16T15:04:41.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.564+0000 7faf91200640 1 Processor -- start 2024-01-16T15:04:41.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.564+0000 7faf91200640 1 -- start start 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.565+0000 7faf91200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf8c0ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.565+0000 7faf91200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c06d6b0 0x7faf8c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.565+0000 7faf91200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf8c06db30 0x7faf8c146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.565+0000 7faf91200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0adec0 con 0x7faf6c005180 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.565+0000 7faf91200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0ae820 con 0x7faf8c06db30 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.565+0000 7faf91200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0ad240 con 0x7faf8c06d6b0 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf8c0ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf8c0ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57306/0 (socket says 172.21.15.141:57306) 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 -- 172.21.15.141:0/1474971153 learned_addr learned my addr 172.21.15.141:0/1474971153 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8aa00640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c06d6b0 0x7faf8c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8be00640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf8c06db30 0x7faf8c146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 -- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c06d6b0 msgr2=0x7faf8c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c06d6b0 0x7faf8c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 -- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf8c06db30 msgr2=0x7faf8c146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf8c06db30 0x7faf8c146da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.566+0000 7faf8b400640 1 -- 172.21.15.141:0/1474971153 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0ad520 con 0x7faf6c005180 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.567+0000 7faf8b400640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf8c0ae2e0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7faf7c001c40 tx=0x7faf7c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.567+0000 7faf83400640 1 -- 172.21.15.141:0/1474971153 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c00c840 con 0x7faf6c005180 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.567+0000 7faf83400640 1 -- 172.21.15.141:0/1474971153 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7c00c9a0 con 0x7faf6c005180 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.567+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c0ad800 con 0x7faf6c005180 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.567+0000 7faf83400640 1 -- 172.21.15.141:0/1474971153 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c00cd10 con 0x7faf6c005180 2024-01-16T15:04:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.567+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c0ace10 con 0x7faf6c005180 2024-01-16T15:04:41.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.568+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c0acfd0 con 0x7faf6c005180 2024-01-16T15:04:41.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.569+0000 7faf83400640 1 -- 172.21.15.141:0/1474971153 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7faf7c01d460 con 0x7faf6c005180 2024-01-16T15:04:41.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.570+0000 7faf83400640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf6406bc50 0x7faf6406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.570+0000 7faf83400640 1 -- 172.21.15.141:0/1474971153 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf7c088730 con 0x7faf6c005180 2024-01-16T15:04:41.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.570+0000 7faf8aa00640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf6406bc50 0x7faf6406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.571+0000 7faf8aa00640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf6406bc50 0x7faf6406e110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faf8c15e620 tx=0x7faf74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.577+0000 7faf83400640 1 -- 172.21.15.141:0/1474971153 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf7c054df0 con 0x7faf6c005180 2024-01-16T15:04:41.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.782+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66721", "target": ["mon-mgr", ""]}) v1 -- 0x7faf8c15c280 con 0x7faf6406bc50 2024-01-16T15:04:41.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.826+0000 7faf83400640 1 -- 172.21.15.141:0/1474971153 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faf7400edc0 con 0x7faf6406bc50 2024-01-16T15:04:41.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf6406bc50 msgr2=0x7faf6406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf6406bc50 0x7faf6406e110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faf8c15e620 tx=0x7faf74008040 comp rx=0 tx=0).stop 2024-01-16T15:04:41.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 msgr2=0x7faf8c0ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf8c0ae2e0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7faf7c001c40 tx=0x7faf7c00ed50 comp rx=0 tx=0).stop 2024-01-16T15:04:41.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 shutdown_connections 2024-01-16T15:04:41.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7faf6406bc50 0x7faf6406e110 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf8c06db30 0x7faf8c146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7faf8c06d6b0 0x7faf8c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 --2- 172.21.15.141:0/1474971153 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf6c005180 0x7faf8c0ae2e0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 >> 172.21.15.141:0/1474971153 conn(0x7faf8c0cc1d0 msgr2=0x7faf8c0cd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 shutdown_connections 2024-01-16T15:04:41.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:41.829+0000 7faf91200640 1 -- 172.21.15.141:0/1474971153 wait complete. 2024-01-16T15:04:41.828 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66723 2024-01-16T15:04:42.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.041+0000 7f3da4200640 1 Processor -- start 2024-01-16T15:04:42.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.041+0000 7f3da4200640 1 -- start start 2024-01-16T15:04:42.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.043+0000 7f3da4200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c0cf9a0 0x7f3d9c166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.043+0000 7f3da4200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c15c5f0 con 0x7f3d9c13df80 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.043+0000 7f3da4200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c06c470 con 0x7f3d9c13e360 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.043+0000 7f3da4200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c06c750 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.043+0000 7f3da1800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3d9c13df80 0x7f3d9c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33318/0 (socket says 172.21.15.141:33318) 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.043+0000 7f3da1800640 1 -- 172.21.15.141:0/1327281504 learned_addr learned my addr 172.21.15.141:0/1327281504 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.044+0000 7f3da2200640 1 --2- 172.21.15.141:0/1327281504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c0cf9a0 0x7f3d9c166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.044+0000 7f3d9be00640 1 -- 172.21.15.141:0/1327281504 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715699229 0 0) 0x7f3d9c15c5f0 con 0x7f3d9c13df80 2024-01-16T15:04:42.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.044+0000 7f3d9be00640 1 -- 172.21.15.141:0/1327281504 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d7c003520 con 0x7f3d9c13df80 2024-01-16T15:04:42.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.044+0000 7f3d9be00640 1 -- 172.21.15.141:0/1327281504 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2801017160 0 0) 0x7f3d7c003520 con 0x7f3d9c13df80 2024-01-16T15:04:42.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.044+0000 7f3da2200640 1 -- 172.21.15.141:0/1327281504 >> v1:172.21.15.188:6789/0 conn(0x7f3d9c13e360 legacy=0x7f3d9c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.044+0000 7f3da2200640 1 -- 172.21.15.141:0/1327281504 >> v1:172.21.15.141:6789/0 conn(0x7f3d9c13df80 legacy=0x7f3d9c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.044+0000 7f3da2200640 1 -- 172.21.15.141:0/1327281504 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c06ca30 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.045+0000 7f3da2200640 1 --2- 172.21.15.141:0/1327281504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c0cf9a0 0x7f3d9c166030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009dc0 tx=0x7f3d9402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67cec1244e9af321 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.046+0000 7f3d9be00640 1 -- 172.21.15.141:0/1327281504 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c001670 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.046+0000 7f3d9be00640 1 -- 172.21.15.141:0/1327281504 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9402c910 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.046+0000 7f3d9be00640 1 -- 172.21.15.141:0/1327281504 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9402cca0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.047+0000 7f3da4200640 1 -- 172.21.15.141:0/1327281504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c0cf9a0 msgr2=0x7f3d9c166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.047+0000 7f3da4200640 1 --2- 172.21.15.141:0/1327281504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c0cf9a0 0x7f3d9c166030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009dc0 tx=0x7f3d9402bde0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.047+0000 7f3da4200640 1 -- 172.21.15.141:0/1327281504 shutdown_connections 2024-01-16T15:04:42.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.047+0000 7f3da4200640 1 --2- 172.21.15.141:0/1327281504 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c0cf9a0 0x7f3d9c166030 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.047+0000 7f3da4200640 1 -- 172.21.15.141:0/1327281504 >> 172.21.15.141:0/1327281504 conn(0x7f3d9c06a800 msgr2=0x7f3d9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.047+0000 7f3da4200640 1 -- 172.21.15.141:0/1327281504 shutdown_connections 2024-01-16T15:04:42.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.047+0000 7f3da4200640 1 -- 172.21.15.141:0/1327281504 wait complete. 2024-01-16T15:04:42.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.049+0000 7f3da4200640 1 Processor -- start 2024-01-16T15:04:42.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.049+0000 7f3da4200640 1 -- start start 2024-01-16T15:04:42.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da4200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d9c0cf9a0 0x7f3d9c0ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da4200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c13df80 0x7f3d9c0aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da4200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d9c13e360 0x7f3d9c070d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da4200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0702c0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da4200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c070880 con 0x7f3d9c13e360 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da4200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0aded0 con 0x7f3d9c13df80 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da1800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d9c0cf9a0 0x7f3d9c0ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da1800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d9c0cf9a0 0x7f3d9c0ad5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57316/0 (socket says 172.21.15.141:57316) 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da1800640 1 -- 172.21.15.141:0/2494881527 learned_addr learned my addr 172.21.15.141:0/2494881527 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da1800640 1 -- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c13df80 msgr2=0x7f3d9c0aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da1800640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c13df80 0x7f3d9c0aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da1800640 1 -- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d9c13e360 msgr2=0x7f3d9c070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.050+0000 7f3da1800640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d9c13e360 0x7f3d9c070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.051+0000 7f3da1800640 1 -- 172.21.15.141:0/2494881527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d94009a70 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.051+0000 7f3da1800640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d9c0cf9a0 0x7f3d9c0ad5d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c750 tx=0x7f3d8c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.051+0000 7f3d99600640 1 -- 172.21.15.141:0/2494881527 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c005ad0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.051+0000 7f3d99600640 1 -- 172.21.15.141:0/2494881527 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c005c50 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.051+0000 7f3d99600640 1 -- 172.21.15.141:0/2494881527 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c007070 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.051+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c0ae1b0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.051+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c0adb10 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.052+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c0675d0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.053+0000 7f3d99600640 1 -- 172.21.15.141:0/2494881527 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f3d8c0071d0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.054+0000 7f3d99600640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3d7406bc50 0x7f3d7406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.055+0000 7f3d99600640 1 -- 172.21.15.141:0/2494881527 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d8c0883e0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.055+0000 7f3da0e00640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3d7406bc50 0x7f3d7406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.056+0000 7f3da0e00640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3d7406bc50 0x7f3d7406e110 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3d84000f60 tx=0x7f3d840079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.059+0000 7f3d99600640 1 -- 172.21.15.141:0/2494881527 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d8c054aa0 con 0x7f3d9c0cf9a0 2024-01-16T15:04:42.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.260+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66723", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d9c06cff0 con 0x7f3d7406bc50 2024-01-16T15:04:42.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.300+0000 7f3d99600640 1 -- 172.21.15.141:0/2494881527 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3d84005080 con 0x7f3d7406bc50 2024-01-16T15:04:42.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3d7406bc50 msgr2=0x7f3d7406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3d7406bc50 0x7f3d7406e110 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3d84000f60 tx=0x7f3d840079b0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d9c0cf9a0 msgr2=0x7f3d9c0ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d9c0cf9a0 0x7f3d9c0ad5d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c750 tx=0x7f3d8c00cc10 comp rx=0 tx=0).stop 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 shutdown_connections 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3d7406bc50 0x7f3d7406e110 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d9c13e360 0x7f3d9c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3d9c13df80 0x7f3d9c0aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 --2- 172.21.15.141:0/2494881527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d9c0cf9a0 0x7f3d9c0ad5d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 >> 172.21.15.141:0/2494881527 conn(0x7f3d9c06a800 msgr2=0x7f3d9c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 shutdown_connections 2024-01-16T15:04:42.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.303+0000 7f3da4200640 1 -- 172.21.15.141:0/2494881527 wait complete. 2024-01-16T15:04:42.302 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66725 2024-01-16T15:04:42.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.515+0000 7f0150600640 1 Processor -- start 2024-01-16T15:04:42.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.516+0000 7f0150600640 1 -- start start 2024-01-16T15:04:42.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.517+0000 7f0150600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480ce8f0 0x7f01480ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.517+0000 7f0150600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f01480b2a00 0x7f01480b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.517+0000 7f0150600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01480d0de0 con 0x7f0148160f80 2024-01-16T15:04:42.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.517+0000 7f0150600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01480d10c0 con 0x7f01480b2a00 2024-01-16T15:04:42.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.517+0000 7f0150600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01480d13a0 con 0x7f01480ce8f0 2024-01-16T15:04:42.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.517+0000 7f014dc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0148160f80 0x7f0148161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33322/0 (socket says 172.21.15.141:33322) 2024-01-16T15:04:42.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.517+0000 7f014dc00640 1 -- 172.21.15.141:0/4246700508 learned_addr learned my addr 172.21.15.141:0/4246700508 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:42.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.518+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479371679 0 0) 0x7f01480d0de0 con 0x7f0148160f80 2024-01-16T15:04:42.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.518+0000 7f014e600640 1 --2- 172.21.15.141:0/4246700508 >> v2:172.21.15.188:3300/0 conn(0x7f01480b2a00 0x7f01480b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.518+0000 7f014d200640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480ce8f0 0x7f01480ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.518+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0128003520 con 0x7f0148160f80 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.518+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207749595 0 0) 0x7f0128003520 con 0x7f0148160f80 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480ce8f0 msgr2=0x7f01480ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480ce8f0 0x7f01480ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 >> v2:172.21.15.188:3300/0 conn(0x7f01480b2a00 msgr2=0x7f01480b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 --2- 172.21.15.141:0/4246700508 >> v2:172.21.15.188:3300/0 conn(0x7f01480b2a00 0x7f01480b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480d1680 con 0x7f0148160f80 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0138001670 con 0x7f0148160f80 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 >> v1:172.21.15.141:6789/0 conn(0x7f0148160f80 legacy=0x7f0148161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0128003970 0x7f0128003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01280047c0 0x7f0128004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480ce8f0 0x7f0128005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01480d10c0 con 0x7f0128003970 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01480d13a0 con 0x7f01280047c0 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012800b980 con 0x7f01480ce8f0 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f014dc00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0128003970 0x7f0128003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f014dc00640 1 -- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480ce8f0 msgr2=0x7f0128005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f014dc00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480ce8f0 0x7f0128005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f014dc00640 1 -- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01280047c0 msgr2=0x7f0128004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f014dc00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01280047c0 0x7f0128004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.519+0000 7f014dc00640 1 -- 172.21.15.141:0/4246700508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128003520 con 0x7f0128003970 2024-01-16T15:04:42.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.520+0000 7f014dc00640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0128003970 0x7f0128003d90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f013800c110 tx=0x7f01280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce4858d8a66c529 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138005ad0 con 0x7f0128003970 2024-01-16T15:04:42.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138005c50 con 0x7f0128003970 2024-01-16T15:04:42.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0147e00640 1 -- 172.21.15.141:0/4246700508 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013802c810 con 0x7f0128003970 2024-01-16T15:04:42.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0150600640 1 -- 172.21.15.141:0/4246700508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0128003970 msgr2=0x7f0128003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0150600640 1 --2- 172.21.15.141:0/4246700508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0128003970 0x7f0128003d90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f013800c110 tx=0x7f01280057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f014e600640 1 -- 172.21.15.141:0/4246700508 reap_dead start 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0150600640 1 -- 172.21.15.141:0/4246700508 shutdown_connections 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0150600640 1 -- 172.21.15.141:0/4246700508 >> 172.21.15.141:0/4246700508 conn(0x7f014809c550 msgr2=0x7f01480d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0150600640 1 -- 172.21.15.141:0/4246700508 shutdown_connections 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.521+0000 7f0150600640 1 -- 172.21.15.141:0/4246700508 wait complete. 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.523+0000 7f0150600640 1 Processor -- start 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.523+0000 7f0150600640 1 -- start start 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f0150600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0128003970 0x7f01481f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f0150600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01481f7320 0x7f01481f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f0150600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480c6d40 0x7f01480c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f0150600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481f39f0 con 0x7f01481f7320 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f0150600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481f3cd0 con 0x7f0128003970 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f0150600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01481f3fb0 con 0x7f01480c6d40 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f014d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01481f7320 0x7f01481f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f014d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01481f7320 0x7f01481f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57332/0 (socket says 172.21.15.141:57332) 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.524+0000 7f014d200640 1 -- 172.21.15.141:0/2891233079 learned_addr learned my addr 172.21.15.141:0/2891233079 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014dc00640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0128003970 0x7f01481f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014e600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480c6d40 0x7f01480c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014d200640 1 -- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480c6d40 msgr2=0x7f01480c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014d200640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480c6d40 0x7f01480c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014d200640 1 -- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0128003970 msgr2=0x7f01481f04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014d200640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0128003970 0x7f01481f04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014d200640 1 -- 172.21.15.141:0/2891233079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014815ee60 con 0x7f01481f7320 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014e600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480c6d40 0x7f01480c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f014d200640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01481f7320 0x7f01481f3080 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f013000e870 tx=0x7f013000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f0145600640 1 -- 172.21.15.141:0/2891233079 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013000c940 con 0x7f01481f7320 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f0145600640 1 -- 172.21.15.141:0/2891233079 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013000cac0 con 0x7f01481f7320 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f0145600640 1 -- 172.21.15.141:0/2891233079 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013000ce70 con 0x7f01481f7320 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014815f140 con 0x7f01481f7320 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.525+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01481f35c0 con 0x7f01481f7320 2024-01-16T15:04:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.526+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01480675d0 con 0x7f01481f7320 2024-01-16T15:04:42.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.527+0000 7f0145600640 1 -- 172.21.15.141:0/2891233079 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f0130013470 con 0x7f01481f7320 2024-01-16T15:04:42.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.528+0000 7f0145600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f012006bc80 0x7f012006e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.528+0000 7f0145600640 1 -- 172.21.15.141:0/2891233079 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0130088720 con 0x7f01481f7320 2024-01-16T15:04:42.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.529+0000 7f014dc00640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f012006bc80 0x7f012006e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.530+0000 7f014dc00640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f012006bc80 0x7f012006e140 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0138034000 tx=0x7f0138007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.532+0000 7f0145600640 1 -- 172.21.15.141:0/2891233079 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0130054de0 con 0x7f01481f7320 2024-01-16T15:04:42.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.750+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66725", "target": ["mon-mgr", ""]}) v1 -- 0x7f01481f3e40 con 0x7f012006bc80 2024-01-16T15:04:42.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.791+0000 7f0145600640 1 -- 172.21.15.141:0/2891233079 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f01481f3e40 con 0x7f012006bc80 2024-01-16T15:04:42.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f012006bc80 msgr2=0x7f012006e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f012006bc80 0x7f012006e140 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0138034000 tx=0x7f0138007000 comp rx=0 tx=0).stop 2024-01-16T15:04:42.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01481f7320 msgr2=0x7f01481f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01481f7320 0x7f01481f3080 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f013000e870 tx=0x7f013000ed30 comp rx=0 tx=0).stop 2024-01-16T15:04:42.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 shutdown_connections 2024-01-16T15:04:42.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f012006bc80 0x7f012006e140 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f01480c6d40 0x7f01480c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f01481f7320 0x7f01481f3080 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 --2- 172.21.15.141:0/2891233079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0128003970 0x7f01481f04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.794+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 >> 172.21.15.141:0/2891233079 conn(0x7f014809c550 msgr2=0x7f0148161360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.795+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 shutdown_connections 2024-01-16T15:04:42.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:42.795+0000 7f0150600640 1 -- 172.21.15.141:0/2891233079 wait complete. 2024-01-16T15:04:42.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:42.793 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:04:42.842 INFO:teuthology.orchestra.run.smithi141.stdout:3 2024-01-16T15:04:47.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:47.845 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:04:47.909 INFO:teuthology.orchestra.run.smithi141.stdout:2 2024-01-16T15:04:47.910 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:04:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.147+0000 7f42bf600640 1 Processor -- start 2024-01-16T15:04:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.147+0000 7f42bf600640 1 -- start start 2024-01-16T15:04:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.149+0000 7f42bf600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f42b80cf9a0 0x7f42b8166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.149+0000 7f42bf600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b815c5f0 con 0x7f42b813df80 2024-01-16T15:04:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.149+0000 7f42bf600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b806c470 con 0x7f42b813e360 2024-01-16T15:04:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.149+0000 7f42bf600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b806c750 con 0x7f42b80cf9a0 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.149+0000 7f42bcc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f42b813df80 0x7f42b815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43268/0 (socket says 172.21.15.141:43268) 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.149+0000 7f42bcc00640 1 -- 172.21.15.141:0/1107779899 learned_addr learned my addr 172.21.15.141:0/1107779899 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614004017 0 0) 0x7f42b815c5f0 con 0x7f42b813df80 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42bd600640 1 --2- 172.21.15.141:0/1107779899 >> v2:172.21.15.188:3300/0 conn(0x7f42b80cf9a0 0x7f42b8166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4298003520 con 0x7f42b813df80 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517855068 0 0) 0x7f4298003520 con 0x7f42b813df80 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 >> v2:172.21.15.188:3300/0 conn(0x7f42b80cf9a0 msgr2=0x7f42b8166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 --2- 172.21.15.141:0/1107779899 >> v2:172.21.15.188:3300/0 conn(0x7f42b80cf9a0 0x7f42b8166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 >> v1:172.21.15.188:6789/0 conn(0x7f42b813e360 legacy=0x7f42b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b806ca30 con 0x7f42b813df80 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f42a80019f0 con 0x7f42b813df80 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 >> v1:172.21.15.141:6789/0 conn(0x7f42b813df80 legacy=0x7f42b815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4298003970 0x7f4298003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42980047c0 0x7f4298004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4298005180 0x7f429800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b806c750 con 0x7f42980047c0 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429800c0d0 con 0x7f4298005180 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.150+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429800c290 con 0x7f4298003970 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42b7e00640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42980047c0 0x7f4298004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42bd600640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4298005180 0x7f429800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42bcc00640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4298003970 0x7f4298003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42b7e00640 1 -- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4298003970 msgr2=0x7f4298003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42b7e00640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4298003970 0x7f4298003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42b7e00640 1 -- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4298005180 msgr2=0x7f429800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42b7e00640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4298005180 0x7f429800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42b7e00640 1 -- 172.21.15.141:0/1107779899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298003520 con 0x7f42980047c0 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42bd600640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4298005180 0x7f429800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.151+0000 7f42bd600640 1 -- 172.21.15.141:0/1107779899 reap_dead start 2024-01-16T15:04:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.152+0000 7f42b7e00640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42980047c0 0x7f4298004c40 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f42a0001720 tx=0x7f42980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575863fb7ab7f964 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.152+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00052d0 con 0x7f42980047c0 2024-01-16T15:04:48.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.152+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a002e050 con 0x7f42980047c0 2024-01-16T15:04:48.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.152+0000 7f42b7400640 1 -- 172.21.15.141:0/1107779899 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0003830 con 0x7f42980047c0 2024-01-16T15:04:48.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.153+0000 7f42bf600640 1 -- 172.21.15.141:0/1107779899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42980047c0 msgr2=0x7f4298004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.153+0000 7f42bf600640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42980047c0 0x7f4298004c40 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f42a0001720 tx=0x7f42980057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.153+0000 7f42bf600640 1 -- 172.21.15.141:0/1107779899 shutdown_connections 2024-01-16T15:04:48.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.153+0000 7f42bf600640 1 --2- 172.21.15.141:0/1107779899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42980047c0 0x7f4298004c40 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.153+0000 7f42bf600640 1 -- 172.21.15.141:0/1107779899 >> 172.21.15.141:0/1107779899 conn(0x7f42b806a800 msgr2=0x7f42b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.153+0000 7f42bf600640 1 -- 172.21.15.141:0/1107779899 shutdown_connections 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.153+0000 7f42bf600640 1 -- 172.21.15.141:0/1107779899 wait complete. 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.155+0000 7f42bf600640 1 Processor -- start 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.155+0000 7f42bf600640 1 -- start start 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bf600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42980047c0 0x7f42b8150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bf600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b813d650 0x7f42b813cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bf600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b814b210 0x7f42b814b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bf600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80b1ca0 con 0x7f42b814b210 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bf600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b814bbd0 con 0x7f42980047c0 2024-01-16T15:04:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bf600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80b2220 con 0x7f42b813d650 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b814b210 0x7f42b814b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b814b210 0x7f42b814b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37398/0 (socket says 172.21.15.141:37398) 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.156+0000 7f42bd600640 1 -- 172.21.15.141:0/2038011691 learned_addr learned my addr 172.21.15.141:0/2038011691 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bcc00640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42980047c0 0x7f42b8150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42b7e00640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b813d650 0x7f42b813cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bd600640 1 -- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b813d650 msgr2=0x7f42b813cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bd600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b813d650 0x7f42b813cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bd600640 1 -- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42980047c0 msgr2=0x7f42b8150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bd600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42980047c0 0x7f42b8150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bd600640 1 -- 172.21.15.141:0/2038011691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b813dd60 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42b7e00640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b813d650 0x7f42b813cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bd600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b814b210 0x7f42b814b690 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f42b0006fd0 tx=0x7f42b000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0004230 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b00043b0 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0004520 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.157+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b814aa30 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.158+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b80b1eb0 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.158+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b80675d0 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.159+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f42b0003850 con 0x7f42b814b210 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.160+0000 7f42b4c00640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006bc50 0x7f429006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.160+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f42b0087d00 con 0x7f42b814b210 2024-01-16T15:04:48.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.161+0000 7f42bcc00640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006bc50 0x7f429006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.162+0000 7f42bcc00640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006bc50 0x7f429006e110 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f42a8001580 tx=0x7f42a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.165+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42b0054420 con 0x7f42b814b210 2024-01-16T15:04:48.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:48.378+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f42b806cff0 con 0x7f429006bc50 2024-01-16T15:04:48.422 INFO:tasks.ceph.mon.a.smithi141.stderr:2024-01-16T15:04:48.443+0000 7f723b200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:04:48.513 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T15:04:49.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:49.628+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f42b00547f0 con 0x7f42b814b210 2024-01-16T15:04:51.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.475+0000 7f42b4c00640 1 -- 172.21.15.141:0/2038011691 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f42a800d070 con 0x7f429006bc50 2024-01-16T15:04:51.455 INFO:teuthology.orchestra.run.smithi141.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:04:51.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006bc50 msgr2=0x7f429006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006bc50 0x7f429006e110 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f42a8001580 tx=0x7f42a8009040 comp rx=0 tx=0).stop 2024-01-16T15:04:51.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b814b210 msgr2=0x7f42b814b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b814b210 0x7f42b814b690 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f42b0006fd0 tx=0x7f42b000e830 comp rx=0 tx=0).stop 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 shutdown_connections 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f429006bc50 0x7f429006e110 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f42b814b210 0x7f42b814b690 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f42b813d650 0x7f42b813cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 --2- 172.21.15.141:0/2038011691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42980047c0 0x7f42b8150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.478+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 >> 172.21.15.141:0/2038011691 conn(0x7f42b806a800 msgr2=0x7f42b815a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.479+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 shutdown_connections 2024-01-16T15:04:51.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.479+0000 7f42bf600640 1 -- 172.21.15.141:0/2038011691 wait complete. 2024-01-16T15:04:51.479 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:04:51.480 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:51.591 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:51.698 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:04:51.698 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:04:51.698 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:51.755 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:51.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.993+0000 7f5c40c00640 1 Processor -- start 2024-01-16T15:04:51.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.993+0000 7f5c40c00640 1 -- start start 2024-01-16T15:04:51.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.994+0000 7f5c40c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 0x7f5c3c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.994+0000 7f5c40c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5c3c15e1e0 0x7f5c3c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.994+0000 7f5c40c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c0b1ee0 con 0x7f5c3c143ae0 2024-01-16T15:04:51.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.994+0000 7f5c40c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3c0b2780 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.994+0000 7f5c40c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c15e990 con 0x7f5c3c0cf9a0 2024-01-16T15:04:51.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c39e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5c3c143ae0 0x7f5c3c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43284/0 (socket says 172.21.15.141:43284) 2024-01-16T15:04:51.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c39e00640 1 -- 172.21.15.141:0/1882997030 learned_addr learned my addr 172.21.15.141:0/1882997030 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:51.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c3a800640 1 --2- 172.21.15.141:0/1882997030 >> v2:172.21.15.141:3300/0 conn(0x7f5c3c15e1e0 0x7f5c3c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c39400640 1 -- 172.21.15.141:0/1882997030 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701703117 0 0) 0x7f5c3c0b1ee0 con 0x7f5c3c143ae0 2024-01-16T15:04:51.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c39400640 1 -- 172.21.15.141:0/1882997030 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c1c003520 con 0x7f5c3c143ae0 2024-01-16T15:04:51.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c3a800640 1 -- 172.21.15.141:0/1882997030 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 msgr2=0x7f5c3c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c3a800640 1 --2- 172.21.15.141:0/1882997030 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 0x7f5c3c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c3a800640 1 -- 172.21.15.141:0/1882997030 >> v1:172.21.15.141:6789/0 conn(0x7f5c3c143ae0 legacy=0x7f5c3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.995+0000 7f5c3a800640 1 -- 172.21.15.141:0/1882997030 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c0b2130 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.996+0000 7f5c3a800640 1 --2- 172.21.15.141:0/1882997030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c0702c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009d90 tx=0x7f5c3402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac4ec00a1b128d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c39400640 1 -- 172.21.15.141:0/1882997030 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3402cc60 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c39400640 1 -- 172.21.15.141:0/1882997030 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3402cdc0 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c39400640 1 -- 172.21.15.141:0/1882997030 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34034660 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 -- 172.21.15.141:0/1882997030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 msgr2=0x7f5c3c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 --2- 172.21.15.141:0/1882997030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c0702c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009d90 tx=0x7f5c3402beb0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 -- 172.21.15.141:0/1882997030 shutdown_connections 2024-01-16T15:04:51.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 --2- 172.21.15.141:0/1882997030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c0702c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 --2- 172.21.15.141:0/1882997030 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 0x7f5c3c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 -- 172.21.15.141:0/1882997030 >> 172.21.15.141:0/1882997030 conn(0x7f5c3c06a800 msgr2=0x7f5c3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 -- 172.21.15.141:0/1882997030 shutdown_connections 2024-01-16T15:04:51.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.997+0000 7f5c40c00640 1 -- 172.21.15.141:0/1882997030 wait complete. 2024-01-16T15:04:51.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.999+0000 7f5c40c00640 1 Processor -- start 2024-01-16T15:04:51.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:51.999+0000 7f5c40c00640 1 -- start start 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.000+0000 7f5c40c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 0x7f5c3c0c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.000+0000 7f5c40c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c3c143ae0 0x7f5c3c1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.000+0000 7f5c40c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c1416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.000+0000 7f5c40c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c0ccb10 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.000+0000 7f5c40c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c0ccdf0 con 0x7f5c3c143ae0 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.000+0000 7f5c40c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c0cd0d0 con 0x7f5c3c0cf9a0 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c1416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c1416e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37436/0 (socket says 172.21.15.141:37436) 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 -- 172.21.15.141:0/4272176967 learned_addr learned my addr 172.21.15.141:0/4272176967 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c39e00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 0x7f5c3c0c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c33e00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c3c143ae0 0x7f5c3c1411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 -- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 msgr2=0x7f5c3c0c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 0x7f5c3c0c4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 -- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c3c143ae0 msgr2=0x7f5c3c1411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c3c143ae0 0x7f5c3c1411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 -- 172.21.15.141:0/4272176967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34009a70 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.001+0000 7f5c3a800640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c1416e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009d90 tx=0x7f5c34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.002+0000 7f5c32000640 1 -- 172.21.15.141:0/4272176967 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34040650 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.002+0000 7f5c32000640 1 -- 172.21.15.141:0/4272176967 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c340407b0 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.002+0000 7f5c32000640 1 -- 172.21.15.141:0/4272176967 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34040b60 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.002+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c0cd3b0 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.002+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c0c4040 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.003+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c0cd8d0 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.004+0000 7f5c32000640 1 -- 172.21.15.141:0/4272176967 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5c34040cc0 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.005+0000 7f5c32000640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c1406bbf0 0x7f5c1406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.005+0000 7f5c32000640 1 -- 172.21.15.141:0/4272176967 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5c340ab200 con 0x7f5c3c15e1e0 2024-01-16T15:04:51.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.005+0000 7f5c39e00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c1406bbf0 0x7f5c1406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.006+0000 7f5c39e00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c1406bbf0 0x7f5c1406e0b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0011e0 tx=0x7f5c2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.009+0000 7f5c32000640 1 -- 172.21.15.141:0/4272176967 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c3c0cd8d0 con 0x7f5c3c15e1e0 2024-01-16T15:04:52.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.200+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c3c0cd8d0 con 0x7f5c3c15e1e0 2024-01-16T15:04:52.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.202+0000 7f5c32000640 1 -- 172.21.15.141:0/4272176967 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f5c340a4e60 con 0x7f5c3c15e1e0 2024-01-16T15:04:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:52.182 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":64,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:04:51.456530+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/1757492142":"2024-01-17T15:04:48.441581+0000","172.21.15.141:6834/1757492142":"2024-01-17T15:04:48.441581+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c1406bbf0 msgr2=0x7f5c1406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c1406bbf0 0x7f5c1406e0b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0011e0 tx=0x7f5c2c009040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 msgr2=0x7f5c3c1416e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c1416e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009d90 tx=0x7f5c34004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 shutdown_connections 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c1406bbf0 0x7f5c1406e0b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c3c15e1e0 0x7f5c3c1416e0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c3c143ae0 0x7f5c3c1411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 --2- 172.21.15.141:0/4272176967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c3c0cf9a0 0x7f5c3c0c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.204+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 >> 172.21.15.141:0/4272176967 conn(0x7f5c3c06a800 msgr2=0x7f5c3c0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.205+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 shutdown_connections 2024-01-16T15:04:52.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.205+0000 7f5c40c00640 1 -- 172.21.15.141:0/4272176967 wait complete. 2024-01-16T15:04:52.204 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:52.204 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:52.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.427+0000 7f3643a00640 1 Processor -- start 2024-01-16T15:04:52.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.427+0000 7f3643a00640 1 -- start start 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f3643a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f363c0cf9a0 0x7f363c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f3643a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c071690 con 0x7f363c070ea0 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f3643a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c071970 con 0x7f363c14b2c0 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f3643a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363c14b6a0 con 0x7f363c0cf9a0 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f363be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f363c0cf9a0 0x7f363c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f3641a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f363c070ea0 0x7f363c0cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43292/0 (socket says 172.21.15.141:43292) 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f3641a00640 1 -- 172.21.15.141:0/1053934369 learned_addr learned my addr 172.21.15.141:0/1053934369 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:52.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.429+0000 7f363be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f363c0cf9a0 0x7f363c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37450/0 (socket says 172.21.15.141:37450) 2024-01-16T15:04:52.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.430+0000 7f363be00640 1 -- 172.21.15.141:0/1053934369 >> v1:172.21.15.188:6789/0 conn(0x7f363c14b2c0 legacy=0x7f363c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.430+0000 7f363be00640 1 -- 172.21.15.141:0/1053934369 >> v1:172.21.15.141:6789/0 conn(0x7f363c070ea0 legacy=0x7f363c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.430+0000 7f363be00640 1 -- 172.21.15.141:0/1053934369 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c071c50 con 0x7f363c0cf9a0 2024-01-16T15:04:52.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.431+0000 7f363be00640 1 --2- 172.21.15.141:0/1053934369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c070960 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f36240097a0 tx=0x7f362402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf981ee2b9c9e856 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.431+0000 7f363b400640 1 -- 172.21.15.141:0/1053934369 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624031830 con 0x7f363c0cf9a0 2024-01-16T15:04:52.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.431+0000 7f363b400640 1 -- 172.21.15.141:0/1053934369 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624031990 con 0x7f363c0cf9a0 2024-01-16T15:04:52.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.431+0000 7f363b400640 1 -- 172.21.15.141:0/1053934369 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624031d20 con 0x7f363c0cf9a0 2024-01-16T15:04:52.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.432+0000 7f3643a00640 1 -- 172.21.15.141:0/1053934369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 msgr2=0x7f363c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.432+0000 7f3643a00640 1 --2- 172.21.15.141:0/1053934369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c070960 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f36240097a0 tx=0x7f362402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.432+0000 7f3643a00640 1 -- 172.21.15.141:0/1053934369 shutdown_connections 2024-01-16T15:04:52.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.432+0000 7f3643a00640 1 --2- 172.21.15.141:0/1053934369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c070960 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.432+0000 7f3643a00640 1 -- 172.21.15.141:0/1053934369 >> 172.21.15.141:0/1053934369 conn(0x7f363c06a800 msgr2=0x7f363c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.432+0000 7f3643a00640 1 -- 172.21.15.141:0/1053934369 shutdown_connections 2024-01-16T15:04:52.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.432+0000 7f3643a00640 1 -- 172.21.15.141:0/1053934369 wait complete. 2024-01-16T15:04:52.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.434+0000 7f3643a00640 1 Processor -- start 2024-01-16T15:04:52.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.434+0000 7f3643a00640 1 -- start start 2024-01-16T15:04:52.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f3643a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f363c070ea0 0x7f363c145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f3643a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f3643a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f363c14b2c0 0x7f363c1645a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f3643a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c143050 con 0x7f363c0cf9a0 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f3643a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c145c90 con 0x7f363c070ea0 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f3643a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c144410 con 0x7f363c14b2c0 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c160b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37454/0 (socket says 172.21.15.141:37454) 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 -- 172.21.15.141:0/4138269947 learned_addr learned my addr 172.21.15.141:0/4138269947 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f3641000640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f363c070ea0 0x7f363c145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 -- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f363c14b2c0 msgr2=0x7f363c1645a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f363c14b2c0 0x7f363c1645a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 -- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f363c070ea0 msgr2=0x7f363c145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f363c070ea0 0x7f363c145750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.435+0000 7f363be00640 1 -- 172.21.15.141:0/4138269947 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624009400 con 0x7f363c0cf9a0 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.436+0000 7f363be00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c160b50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f36240098c0 tx=0x7f36240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.436+0000 7f3638c00640 1 -- 172.21.15.141:0/4138269947 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362403ba70 con 0x7f363c0cf9a0 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.436+0000 7f3638c00640 1 -- 172.21.15.141:0/4138269947 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362403bbf0 con 0x7f363c0cf9a0 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.436+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c1446b0 con 0x7f363c0cf9a0 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.436+0000 7f3638c00640 1 -- 172.21.15.141:0/4138269947 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36240453f0 con 0x7f363c0cf9a0 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.436+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c144a40 con 0x7f363c0cf9a0 2024-01-16T15:04:52.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.438+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363c0675d0 con 0x7f363c0cf9a0 2024-01-16T15:04:52.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.438+0000 7f3638c00640 1 -- 172.21.15.141:0/4138269947 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f362404e450 con 0x7f363c0cf9a0 2024-01-16T15:04:52.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.439+0000 7f3638c00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361406bbc0 0x7f361406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.439+0000 7f3638c00640 1 -- 172.21.15.141:0/4138269947 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f36240aa750 con 0x7f363c0cf9a0 2024-01-16T15:04:52.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.440+0000 7f3641000640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361406bbc0 0x7f361406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.441+0000 7f3641000640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361406bbc0 0x7f361406e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f362c006fd0 tx=0x7f362c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.443+0000 7f3638c00640 1 -- 172.21.15.141:0/4138269947 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3624077010 con 0x7f363c0cf9a0 2024-01-16T15:04:52.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.634+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f363c0b1e40 con 0x7f363c0cf9a0 2024-01-16T15:04:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.639+0000 7f3638c00640 1 -- 172.21.15.141:0/4138269947 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f36240350a0 con 0x7f363c0cf9a0 2024-01-16T15:04:52.619 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361406bbc0 msgr2=0x7f361406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361406bbc0 0x7f361406e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f362c006fd0 tx=0x7f362c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 msgr2=0x7f363c160b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c160b50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f36240098c0 tx=0x7f36240046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 shutdown_connections 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361406bbc0 0x7f361406e080 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f363c14b2c0 0x7f363c1645a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f363c0cf9a0 0x7f363c160b50 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 --2- 172.21.15.141:0/4138269947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f363c070ea0 0x7f363c145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 >> 172.21.15.141:0/4138269947 conn(0x7f363c06a800 msgr2=0x7f363c0ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 shutdown_connections 2024-01-16T15:04:52.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.642+0000 7f3643a00640 1 -- 172.21.15.141:0/4138269947 wait complete. 2024-01-16T15:04:52.640 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:04:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.878+0000 7f7ff7800640 1 Processor -- start 2024-01-16T15:04:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.878+0000 7f7ff7800640 1 -- start start 2024-01-16T15:04:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7ff7800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff013e360 0x7f7ff00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7ff7800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7ff015f9c0 0x7f7ff00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7ff7800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff015c5f0 con 0x7f7ff013df80 2024-01-16T15:04:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7ff7800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff006c470 con 0x7f7ff015f9c0 2024-01-16T15:04:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7ff7800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff006c750 con 0x7f7ff013e360 2024-01-16T15:04:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7ff4e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7ff013df80 0x7f7ff015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43306/0 (socket says 172.21.15.141:43306) 2024-01-16T15:04:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7ff4e00640 1 -- 172.21.15.141:0/1880970516 learned_addr learned my addr 172.21.15.141:0/1880970516 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:52.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7fe7c00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff013e360 0x7f7ff00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.880+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222318895 0 0) 0x7f7ff015c5f0 con 0x7f7ff013df80 2024-01-16T15:04:52.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd0003520 con 0x7f7ff013df80 2024-01-16T15:04:52.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7ff5800640 1 --2- 172.21.15.141:0/1880970516 >> v2:172.21.15.188:3300/0 conn(0x7f7ff015f9c0 0x7f7ff00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563883324 0 0) 0x7f7fd0003520 con 0x7f7ff013df80 2024-01-16T15:04:52.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff013e360 msgr2=0x7f7ff00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff013e360 0x7f7ff00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 >> v2:172.21.15.188:3300/0 conn(0x7f7ff015f9c0 msgr2=0x7f7ff00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 --2- 172.21.15.141:0/1880970516 >> v2:172.21.15.188:3300/0 conn(0x7f7ff015f9c0 0x7f7ff00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff006ca30 con 0x7f7ff013df80 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7fe0001670 con 0x7f7ff013df80 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 >> v1:172.21.15.141:6789/0 conn(0x7f7ff013df80 legacy=0x7f7ff015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fd0003970 0x7f7fd0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd00047c0 0x7f7fd0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff013e360 0x7f7fd0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff006c470 con 0x7f7fd0003970 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff006c750 con 0x7f7fd00047c0 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd000b9b0 con 0x7f7ff013e360 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.881+0000 7f7ff4e00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fd0003970 0x7f7fd0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.882+0000 7f7fe7c00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd00047c0 0x7f7fd0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.882+0000 7f7ff4e00640 1 -- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff013e360 msgr2=0x7f7fd0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.882+0000 7f7ff4e00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff013e360 0x7f7fd0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.882+0000 7f7ff4e00640 1 -- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd00047c0 msgr2=0x7f7fd0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.882+0000 7f7ff4e00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd00047c0 0x7f7fd0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.882+0000 7f7ff4e00640 1 -- 172.21.15.141:0/1880970516 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd0003520 con 0x7f7fd0003970 2024-01-16T15:04:52.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.883+0000 7f7ff4e00640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fd0003970 0x7f7fd0003d90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000c290 tx=0x7f7fd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed586b735a8eef7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.883+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0005ad0 con 0x7f7fd0003970 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.883+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe0005c30 con 0x7f7fd0003970 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.883+0000 7f7fefe00640 1 -- 172.21.15.141:0/1880970516 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe002c8d0 con 0x7f7fd0003970 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.884+0000 7f7ff7800640 1 -- 172.21.15.141:0/1880970516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fd0003970 msgr2=0x7f7fd0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.884+0000 7f7ff7800640 1 --2- 172.21.15.141:0/1880970516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fd0003970 0x7f7fd0003d90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000c290 tx=0x7f7fd00057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.884+0000 7f7ff5800640 1 -- 172.21.15.141:0/1880970516 reap_dead start 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.884+0000 7f7ff7800640 1 -- 172.21.15.141:0/1880970516 shutdown_connections 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.884+0000 7f7ff7800640 1 -- 172.21.15.141:0/1880970516 >> 172.21.15.141:0/1880970516 conn(0x7f7ff006a800 msgr2=0x7f7ff006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.884+0000 7f7ff7800640 1 -- 172.21.15.141:0/1880970516 shutdown_connections 2024-01-16T15:04:52.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.884+0000 7f7ff7800640 1 -- 172.21.15.141:0/1880970516 wait complete. 2024-01-16T15:04:52.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.886+0000 7f7ff7800640 1 Processor -- start 2024-01-16T15:04:52.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.886+0000 7f7ff7800640 1 -- start start 2024-01-16T15:04:52.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7ff7800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd0003970 0x7f7ff0152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7ff7800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ff013cb70 0x7f7ff013cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7ff7800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff00aff20 0x7f7ff00b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7ff7800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00b08e0 con 0x7f7ff013cb70 2024-01-16T15:04:52.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7ff7800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff0153260 con 0x7f7fd0003970 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7ff7800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff013afe0 con 0x7f7ff00aff20 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7fe7c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ff013cb70 0x7f7ff013cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7fe7c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ff013cb70 0x7f7ff013cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37486/0 (socket says 172.21.15.141:37486) 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7fe7c00640 1 -- 172.21.15.141:0/960908895 learned_addr learned my addr 172.21.15.141:0/960908895 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.887+0000 7f7ff4e00640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd0003970 0x7f7ff0152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fe7c00640 1 -- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff00aff20 msgr2=0x7f7ff00b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fe7c00640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff00aff20 0x7f7ff00b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fe7c00640 1 -- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd0003970 msgr2=0x7f7ff0152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fe7c00640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd0003970 0x7f7ff0152d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fe7c00640 1 -- 172.21.15.141:0/960908895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff013b2c0 con 0x7f7ff013cb70 2024-01-16T15:04:52.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fe7c00640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ff013cb70 0x7f7ff013cff0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8010860 tx=0x7f7fd8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fed600640 1 -- 172.21.15.141:0/960908895 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8009610 con 0x7f7ff013cb70 2024-01-16T15:04:52.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fed600640 1 -- 172.21.15.141:0/960908895 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd800eb60 con 0x7f7ff013cb70 2024-01-16T15:04:52.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7fed600640 1 -- 172.21.15.141:0/960908895 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd801c400 con 0x7f7ff013cb70 2024-01-16T15:04:52.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff013b5a0 con 0x7f7ff013cb70 2024-01-16T15:04:52.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.888+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff013bb60 con 0x7f7ff013cb70 2024-01-16T15:04:52.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.889+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff00675d0 con 0x7f7ff013cb70 2024-01-16T15:04:52.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.890+0000 7f7fed600640 1 -- 172.21.15.141:0/960908895 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f7fd801c560 con 0x7f7ff013cb70 2024-01-16T15:04:52.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.891+0000 7f7fed600640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fc806bbc0 0x7f7fc806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.891+0000 7f7fed600640 1 -- 172.21.15.141:0/960908895 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7fd8088a50 con 0x7f7ff013cb70 2024-01-16T15:04:52.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.892+0000 7f7ff4e00640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fc806bbc0 0x7f7fc806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.893+0000 7f7ff4e00640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fc806bbc0 0x7f7fc806e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0034000 tx=0x7f7fe0007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:52.896+0000 7f7fed600640 1 -- 172.21.15.141:0/960908895 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fd80553c0 con 0x7f7ff013cb70 2024-01-16T15:04:53.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.093+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7ff00cc100 con 0x7f7ff013cb70 2024-01-16T15:04:53.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.094+0000 7f7fed600640 1 -- 172.21.15.141:0/960908895 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v64) v1 ==== 80+0+124 (secure 0 0 0) 0x7f7fd80589e0 con 0x7f7ff013cb70 2024-01-16T15:04:53.074 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/1757492142 2024-01-17T15:04:48.441581+0000 2024-01-16T15:04:53.074 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/1757492142 2024-01-17T15:04:48.441581+0000 2024-01-16T15:04:53.074 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.096+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fc806bbc0 msgr2=0x7f7fc806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.096+0000 7f7ff7800640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fc806bbc0 0x7f7fc806e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0034000 tx=0x7f7fe0007000 comp rx=0 tx=0).stop 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ff013cb70 msgr2=0x7f7ff013cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ff013cb70 0x7f7ff013cff0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8010860 tx=0x7f7fd8010d20 comp rx=0 tx=0).stop 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 shutdown_connections 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7fc806bbc0 0x7f7fc806e080 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ff00aff20 0x7f7ff00b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ff013cb70 0x7f7ff013cff0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 --2- 172.21.15.141:0/960908895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fd0003970 0x7f7ff0152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 >> 172.21.15.141:0/960908895 conn(0x7f7ff006a800 msgr2=0x7f7ff015a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 shutdown_connections 2024-01-16T15:04:53.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.097+0000 7f7ff7800640 1 -- 172.21.15.141:0/960908895 wait complete. 2024-01-16T15:04:53.093 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T15:04:53.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.094 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:53.170 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:04:53.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.171 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:04:53.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[57243]: fuse finished with error 0 and tester_r 0 2024-01-16T15:04:53.363 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:04:53.363 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:53.419 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:53.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.650+0000 7f6262c00640 1 Processor -- start 2024-01-16T15:04:53.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.650+0000 7f6262c00640 1 -- start start 2024-01-16T15:04:53.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.651+0000 7f6262c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f625c0b3f30 0x7f625c0d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.651+0000 7f6262c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c0d0a10 con 0x7f625c0d0100 2024-01-16T15:04:53.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.651+0000 7f6262c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c0cf080 con 0x7f625c160920 2024-01-16T15:04:53.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.651+0000 7f6262c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625c0c6be0 con 0x7f625c0b3f30 2024-01-16T15:04:53.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f6260e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f625c0d0100 0x7f625c0d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59534/0 (socket says 172.21.15.141:59534) 2024-01-16T15:04:53.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f6260e00640 1 -- 172.21.15.141:0/1732390229 learned_addr learned my addr 172.21.15.141:0/1732390229 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:53.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773795759 0 0) 0x7f625c0d0a10 con 0x7f625c0d0100 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f623c003520 con 0x7f625c0d0100 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f6262200640 1 --2- 172.21.15.141:0/1732390229 >> v2:172.21.15.188:3300/0 conn(0x7f625c0b3f30 0x7f625c0d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 824753538 0 0) 0x7f623c003520 con 0x7f625c0d0100 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 >> v2:172.21.15.188:3300/0 conn(0x7f625c0b3f30 msgr2=0x7f625c0d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 --2- 172.21.15.141:0/1732390229 >> v2:172.21.15.188:3300/0 conn(0x7f625c0b3f30 0x7f625c0d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 >> v1:172.21.15.188:6789/0 conn(0x7f625c160920 legacy=0x7f625c0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c0ce7e0 con 0x7f625c0d0100 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6244001670 con 0x7f625c0d0100 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 >> v1:172.21.15.141:6789/0 conn(0x7f625c0d0100 legacy=0x7f625c0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0039c0 0x7f623c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f623c0047c0 0x7f623c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.652+0000 7f625be00640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f623c0051d0 0x7f623c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0c6be0 con 0x7f623c0051d0 2024-01-16T15:04:53.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c00c100 con 0x7f623c0039c0 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623c00c2c0 con 0x7f623c0047c0 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6262200640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f623c0051d0 0x7f623c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6260e00640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f623c0047c0 0x7f623c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6261800640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0039c0 0x7f623c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6262200640 1 -- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f623c0047c0 msgr2=0x7f623c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6262200640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f623c0047c0 0x7f623c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6262200640 1 -- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0039c0 msgr2=0x7f623c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6262200640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0039c0 0x7f623c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6262200640 1 -- 172.21.15.141:0/1732390229 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c003520 con 0x7f623c0051d0 2024-01-16T15:04:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.653+0000 7f6261800640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0039c0 0x7f623c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.654+0000 7f6262200640 1 -- 172.21.15.141:0/1732390229 reap_dead start 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.654+0000 7f6262200640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f623c0051d0 0x7f623c00b980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f625400ba00 tx=0x7f623c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6e684dad4ad0cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.654+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254004020 con 0x7f623c0051d0 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254004180 con 0x7f623c0051d0 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f625be00640 1 -- 172.21.15.141:0/1732390229 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254003610 con 0x7f623c0051d0 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f6262c00640 1 -- 172.21.15.141:0/1732390229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f623c0051d0 msgr2=0x7f623c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f6262c00640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f623c0051d0 0x7f623c00b980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f625400ba00 tx=0x7f623c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f6262c00640 1 -- 172.21.15.141:0/1732390229 shutdown_connections 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f6262c00640 1 --2- 172.21.15.141:0/1732390229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f623c0051d0 0x7f623c00b980 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f6262c00640 1 -- 172.21.15.141:0/1732390229 >> 172.21.15.141:0/1732390229 conn(0x7f625c0d3090 msgr2=0x7f625c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f6262c00640 1 -- 172.21.15.141:0/1732390229 shutdown_connections 2024-01-16T15:04:53.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.655+0000 7f6262c00640 1 -- 172.21.15.141:0/1732390229 wait complete. 2024-01-16T15:04:53.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.657+0000 7f6262c00640 1 Processor -- start 2024-01-16T15:04:53.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.657+0000 7f6262c00640 1 -- start start 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0051d0 0x7f625c0b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f625c0cc8f0 0x7f625c14be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f625c14c750 0x7f625c14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0ccd70 con 0x7f625c14c750 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c1525f0 con 0x7f623c0051d0 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0cbb50 con 0x7f625c0cc8f0 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f625c14c750 0x7f625c14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f625c14c750 0x7f625c14cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33400/0 (socket says 172.21.15.141:33400) 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.658+0000 7f6262200640 1 -- 172.21.15.141:0/1754478181 learned_addr learned my addr 172.21.15.141:0/1754478181 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:53.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6260e00640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f625c0cc8f0 0x7f625c14be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6261800640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0051d0 0x7f625c0b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262200640 1 -- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f625c0cc8f0 msgr2=0x7f625c14be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262200640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f625c0cc8f0 0x7f625c14be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262200640 1 -- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0051d0 msgr2=0x7f625c0b87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262200640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0051d0 0x7f625c0b87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262200640 1 -- 172.21.15.141:0/1754478181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c0b8e00 con 0x7f625c14c750 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6261800640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0051d0 0x7f625c0b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262200640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f625c14c750 0x7f625c14cc00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f625402bf90 tx=0x7f6254004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6259600640 1 -- 172.21.15.141:0/1754478181 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62540042f0 con 0x7f625c14c750 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6259600640 1 -- 172.21.15.141:0/1754478181 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254040070 con 0x7f625c14c750 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6259600640 1 -- 172.21.15.141:0/1754478181 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625403c450 con 0x7f625c14c750 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c0cbe10 con 0x7f625c14c750 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.659+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625c0cc3d0 con 0x7f625c14c750 2024-01-16T15:04:53.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.660+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c067450 con 0x7f625c14c750 2024-01-16T15:04:53.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.661+0000 7f6259600640 1 -- 172.21.15.141:0/1754478181 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6254042030 con 0x7f625c14c750 2024-01-16T15:04:53.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.662+0000 7f6259600640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f623406bbf0 0x7f623406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.663+0000 7f6259600640 1 -- 172.21.15.141:0/1754478181 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f62540aa530 con 0x7f625c14c750 2024-01-16T15:04:53.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.663+0000 7f6261800640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f623406bbf0 0x7f623406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.664+0000 7f6261800640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f623406bbf0 0x7f623406e0b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f624c007fd0 tx=0x7f624c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.667+0000 7f6259600640 1 -- 172.21.15.141:0/1754478181 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6254076d20 con 0x7f625c14c750 2024-01-16T15:04:53.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.859+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625c08c7e0 con 0x7f625c14c750 2024-01-16T15:04:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.861+0000 7f6259600640 1 -- 172.21.15.141:0/1754478181 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f625407a340 con 0x7f625c14c750 2024-01-16T15:04:53.894 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:53.894 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":64,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:04:51.456530+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/1757492142":"2024-01-17T15:04:48.441581+0000","172.21.15.141:6834/1757492142":"2024-01-17T15:04:48.441581+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f623406bbf0 msgr2=0x7f623406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f623406bbf0 0x7f623406e0b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f624c007fd0 tx=0x7f624c009040 comp rx=0 tx=0).stop 2024-01-16T15:04:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f625c14c750 msgr2=0x7f625c14cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f625c14c750 0x7f625c14cc00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f625402bf90 tx=0x7f6254004730 comp rx=0 tx=0).stop 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 shutdown_connections 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f623406bbf0 0x7f623406e0b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f625c14c750 0x7f625c14cc00 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f625c0cc8f0 0x7f625c14be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 --2- 172.21.15.141:0/1754478181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f623c0051d0 0x7f625c0b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.863+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 >> 172.21.15.141:0/1754478181 conn(0x7f625c0d3090 msgr2=0x7f625c0d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.864+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 shutdown_connections 2024-01-16T15:04:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:53.864+0000 7f6262c00640 1 -- 172.21.15.141:0/1754478181 wait complete. 2024-01-16T15:04:53.897 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:53.897 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:54.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.135+0000 7f52e3000640 1 Processor -- start 2024-01-16T15:04:54.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.135+0000 7f52e3000640 1 -- start start 2024-01-16T15:04:54.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52e3000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52dc13df80 0x7f52dc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52e3000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52e3000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc06c470 con 0x7f52dc13e360 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52e3000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52dc06c750 con 0x7f52dc13df80 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52e3000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc06ca30 con 0x7f52dc0cf9a0 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52dbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52dc13df80 0x7f52dc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52d3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f52dc13e360 0x7f52dc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59540/0 (socket says 172.21.15.141:59540) 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52d3e00640 1 -- 172.21.15.141:0/2992223186 learned_addr learned my addr 172.21.15.141:0/2992223186 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.137+0000 7f52e1000640 1 --2- 172.21.15.141:0/2992223186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.138+0000 7f52dbe00640 1 -- 172.21.15.141:0/2992223186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 msgr2=0x7f52dc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.138+0000 7f52dbe00640 1 --2- 172.21.15.141:0/2992223186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.138+0000 7f52dbe00640 1 -- 172.21.15.141:0/2992223186 >> v1:172.21.15.141:6789/0 conn(0x7f52dc13e360 legacy=0x7f52dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.138+0000 7f52dbe00640 1 -- 172.21.15.141:0/2992223186 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc06cd10 con 0x7f52dc13df80 2024-01-16T15:04:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.138+0000 7f52db400640 1 -- 172.21.15.141:0/2992223186 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180477370 0 0) 0x7f52dc06c470 con 0x7f52dc13e360 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.139+0000 7f52dbe00640 1 --2- 172.21.15.141:0/2992223186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc15bee0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0097a0 tx=0x7f52cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a014c65be784da57 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.139+0000 7f52db400640 1 -- 172.21.15.141:0/2992223186 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc031830 con 0x7f52dc13df80 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.139+0000 7f52db400640 1 -- 172.21.15.141:0/2992223186 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc031990 con 0x7f52dc13df80 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.139+0000 7f52db400640 1 -- 172.21.15.141:0/2992223186 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc031d20 con 0x7f52dc13df80 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 -- 172.21.15.141:0/2992223186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 msgr2=0x7f52dc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 --2- 172.21.15.141:0/2992223186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc15bee0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0097a0 tx=0x7f52cc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 -- 172.21.15.141:0/2992223186 shutdown_connections 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 --2- 172.21.15.141:0/2992223186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 --2- 172.21.15.141:0/2992223186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc15bee0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 -- 172.21.15.141:0/2992223186 >> 172.21.15.141:0/2992223186 conn(0x7f52dc06a800 msgr2=0x7f52dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:54.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 -- 172.21.15.141:0/2992223186 shutdown_connections 2024-01-16T15:04:54.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.140+0000 7f52e3000640 1 -- 172.21.15.141:0/2992223186 wait complete. 2024-01-16T15:04:54.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.142+0000 7f52e3000640 1 Processor -- start 2024-01-16T15:04:54.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.142+0000 7f52e3000640 1 -- start start 2024-01-16T15:04:54.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52e3000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc13d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52e3000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc14e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52e3000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52dc13e360 0x7f52dc0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52e3000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc13de10 con 0x7f52dc13df80 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52e3000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc14e710 con 0x7f52dc13e360 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52e3000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc14e990 con 0x7f52dc0cf9a0 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52d3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc14e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52d3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc14e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33420/0 (socket says 172.21.15.141:33420) 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.143+0000 7f52d3e00640 1 -- 172.21.15.141:0/4254005374 learned_addr learned my addr 172.21.15.141:0/4254005374 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52dbe00640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc13d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52e1000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52dc13e360 0x7f52dc0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d3e00640 1 -- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 msgr2=0x7f52dc13d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d3e00640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc13d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d3e00640 1 -- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52dc13e360 msgr2=0x7f52dc0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d3e00640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52dc13e360 0x7f52dc0b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d3e00640 1 -- 172.21.15.141:0/4254005374 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc009400 con 0x7f52dc13df80 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52e1000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52dc13e360 0x7f52dc0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d3e00640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc14e1d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f52c400c4b0 tx=0x7f52c400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d8c00640 1 -- 172.21.15.141:0/4254005374 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c40074e0 con 0x7f52dc13df80 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d8c00640 1 -- 172.21.15.141:0/4254005374 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c4007660 con 0x7f52dc13df80 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52d8c00640 1 -- 172.21.15.141:0/4254005374 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c4004a20 con 0x7f52dc13df80 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc14ec70 con 0x7f52dc13df80 2024-01-16T15:04:54.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.144+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc14adf0 con 0x7f52dc13df80 2024-01-16T15:04:54.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.146+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52dc14f190 con 0x7f52dc13df80 2024-01-16T15:04:54.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.147+0000 7f52d8c00640 1 -- 172.21.15.141:0/4254005374 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f52c4004b80 con 0x7f52dc13df80 2024-01-16T15:04:54.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.147+0000 7f52d8c00640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52b406bbf0 0x7f52b406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.148+0000 7f52d8c00640 1 -- 172.21.15.141:0/4254005374 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f52c4087670 con 0x7f52dc13df80 2024-01-16T15:04:54.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.148+0000 7f52dbe00640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52b406bbf0 0x7f52b406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.149+0000 7f52dbe00640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52b406bbf0 0x7f52b406e0b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f52cc036000 tx=0x7f52cc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.152+0000 7f52d8c00640 1 -- 172.21.15.141:0/4254005374 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52dc14f190 con 0x7f52dc13df80 2024-01-16T15:04:54.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.343+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52dc14ac50 con 0x7f52dc13df80 2024-01-16T15:04:54.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.344+0000 7f52d8c00640 1 -- 172.21.15.141:0/4254005374 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52dc14ac50 con 0x7f52dc13df80 2024-01-16T15:04:54.324 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52b406bbf0 msgr2=0x7f52b406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52b406bbf0 0x7f52b406e0b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f52cc036000 tx=0x7f52cc008030 comp rx=0 tx=0).stop 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 msgr2=0x7f52dc14e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc14e1d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f52c400c4b0 tx=0x7f52c400c970 comp rx=0 tx=0).stop 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 shutdown_connections 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52b406bbf0 0x7f52b406e0b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52dc13e360 0x7f52dc0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52dc13df80 0x7f52dc14e1d0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 --2- 172.21.15.141:0/4254005374 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52dc0cf9a0 0x7f52dc13d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 >> 172.21.15.141:0/4254005374 conn(0x7f52dc06a800 msgr2=0x7f52dc0cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:54.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 shutdown_connections 2024-01-16T15:04:54.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.347+0000 7f52e3000640 1 -- 172.21.15.141:0/4254005374 wait complete. 2024-01-16T15:04:54.346 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:04:54.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.564+0000 7fe4eb200640 1 Processor -- start 2024-01-16T15:04:54.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.565+0000 7fe4eb200640 1 -- start start 2024-01-16T15:04:54.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.566+0000 7fe4eb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 0x7fe4e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.566+0000 7fe4eb200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e40cfbf0 con 0x7fe4e4071070 2024-01-16T15:04:54.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.566+0000 7fe4eb200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e4071690 con 0x7fe4e40cf7d0 2024-01-16T15:04:54.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.566+0000 7fe4eb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e414b6a0 con 0x7fe4e414b2c0 2024-01-16T15:04:54.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.566+0000 7fe4e9200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe4e4071070 0x7fe4e40cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59550/0 (socket says 172.21.15.141:59550) 2024-01-16T15:04:54.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.566+0000 7fe4e9200640 1 -- 172.21.15.141:0/2197147439 learned_addr learned my addr 172.21.15.141:0/2197147439 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:54.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3e00640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 0x7fe4e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816814329 0 0) 0x7fe4e40cfbf0 con 0x7fe4e4071070 2024-01-16T15:04:54.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c4003520 con 0x7fe4e4071070 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3291481656 0 0) 0x7fe4c4003520 con 0x7fe4e4071070 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 msgr2=0x7fe4e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 0x7fe4e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 >> v1:172.21.15.188:6789/0 conn(0x7fe4e40cf7d0 legacy=0x7fe4e4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4071970 con 0x7fe4e4071070 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe4dc0019f0 con 0x7fe4e4071070 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.567+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 >> v1:172.21.15.141:6789/0 conn(0x7fe4e4071070 legacy=0x7fe4e40cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e3400640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 0x7fe4c40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e3400640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4c4003f20 0x7fe4c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e3400640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e414b6a0 con 0x7fe4c4004d00 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c400ba60 con 0x7fe4c4003f20 2024-01-16T15:04:54.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c400bc20 con 0x7fe4e414b2c0 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e9200640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4dbe00640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4c4003f20 0x7fe4c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e3e00640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 0x7fe4c40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e9200640 1 -- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 msgr2=0x7fe4c40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e9200640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e414b2c0 0x7fe4c40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e9200640 1 -- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4c4003f20 msgr2=0x7fe4c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e9200640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4c4003f20 0x7fe4c40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.568+0000 7fe4e9200640 1 -- 172.21.15.141:0/2197147439 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c4003520 con 0x7fe4c4004d00 2024-01-16T15:04:54.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.569+0000 7fe4e9200640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4c4005180 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00a830 tx=0x7fe4c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d389b9cca006d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0072d0 con 0x7fe4c4004d00 2024-01-16T15:04:54.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc02f070 con 0x7fe4c4004d00 2024-01-16T15:04:54.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4e3400640 1 -- 172.21.15.141:0/2197147439 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0076f0 con 0x7fe4c4004d00 2024-01-16T15:04:54.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4eb200640 1 -- 172.21.15.141:0/2197147439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 msgr2=0x7fe4c4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4eb200640 1 --2- 172.21.15.141:0/2197147439 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4c4005180 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00a830 tx=0x7fe4c40057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4e9200640 1 -- 172.21.15.141:0/2197147439 reap_dead start 2024-01-16T15:04:54.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4eb200640 1 -- 172.21.15.141:0/2197147439 shutdown_connections 2024-01-16T15:04:54.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.570+0000 7fe4eb200640 1 -- 172.21.15.141:0/2197147439 >> 172.21.15.141:0/2197147439 conn(0x7fe4e406a800 msgr2=0x7fe4e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:54.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.571+0000 7fe4eb200640 1 -- 172.21.15.141:0/2197147439 shutdown_connections 2024-01-16T15:04:54.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.571+0000 7fe4eb200640 1 -- 172.21.15.141:0/2197147439 wait complete. 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.572+0000 7fe4eb200640 1 Processor -- start 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.572+0000 7fe4eb200640 1 -- start start 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.573+0000 7fe4eb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4e413e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.573+0000 7fe4eb200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4e4151db0 0x7fe4e4142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.573+0000 7fe4eb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e4160e70 0x7fe4e4147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.573+0000 7fe4eb200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e413fbf0 con 0x7fe4c4004d00 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.573+0000 7fe4eb200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e413eec0 con 0x7fe4e4151db0 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.573+0000 7fe4eb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e4142b70 con 0x7fe4e4160e70 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4e413e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4e413e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33442/0 (socket says 172.21.15.141:33442) 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 -- 172.21.15.141:0/620131326 learned_addr learned my addr 172.21.15.141:0/620131326 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:54.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 -- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e4160e70 msgr2=0x7fe4e4147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e4160e70 0x7fe4e4147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 -- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4e4151db0 msgr2=0x7fe4e4142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4dbe00640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4e4151db0 0x7fe4e4142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4e4151db0 0x7fe4e4142630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 -- 172.21.15.141:0/620131326 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4142e40 con 0x7fe4c4004d00 2024-01-16T15:04:54.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.574+0000 7fe4e3e00640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4e413e980 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4011580 tx=0x7fe4d4011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.575+0000 7fe4e0c00640 1 -- 172.21.15.141:0/620131326 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4015870 con 0x7fe4c4004d00 2024-01-16T15:04:54.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.575+0000 7fe4e0c00640 1 -- 172.21.15.141:0/620131326 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d40159d0 con 0x7fe4c4004d00 2024-01-16T15:04:54.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.575+0000 7fe4e0c00640 1 -- 172.21.15.141:0/620131326 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4015d40 con 0x7fe4c4004d00 2024-01-16T15:04:54.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.575+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e4143120 con 0x7fe4c4004d00 2024-01-16T15:04:54.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.575+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e41436e0 con 0x7fe4c4004d00 2024-01-16T15:04:54.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.576+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e40675d0 con 0x7fe4c4004d00 2024-01-16T15:04:54.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.577+0000 7fe4e0c00640 1 -- 172.21.15.141:0/620131326 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe4d400f460 con 0x7fe4c4004d00 2024-01-16T15:04:54.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.578+0000 7fe4e0c00640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe4bc06bbc0 0x7fe4bc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.578+0000 7fe4e0c00640 1 -- 172.21.15.141:0/620131326 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe4d40888f0 con 0x7fe4c4004d00 2024-01-16T15:04:54.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.579+0000 7fe4dbe00640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe4bc06bbc0 0x7fe4bc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.580+0000 7fe4dbe00640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe4bc06bbc0 0x7fe4bc06e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc007fd0 tx=0x7fe4cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.582+0000 7fe4e0c00640 1 -- 172.21.15.141:0/620131326 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4d4055190 con 0x7fe4c4004d00 2024-01-16T15:04:54.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.781+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe4e4067d00 con 0x7fe4c4004d00 2024-01-16T15:04:54.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.782+0000 7fe4e0c00640 1 -- 172.21.15.141:0/620131326 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v64) v1 ==== 80+0+124 (secure 0 0 0) 0x7fe4d40587b0 con 0x7fe4c4004d00 2024-01-16T15:04:54.762 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/1757492142 2024-01-17T15:04:48.441581+0000 2024-01-16T15:04:54.762 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/1757492142 2024-01-17T15:04:48.441581+0000 2024-01-16T15:04:54.762 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:04:54.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe4bc06bbc0 msgr2=0x7fe4bc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe4bc06bbc0 0x7fe4bc06e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc007fd0 tx=0x7fe4cc009040 comp rx=0 tx=0).stop 2024-01-16T15:04:54.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 msgr2=0x7fe4e413e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4e413e980 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4011580 tx=0x7fe4d4011a40 comp rx=0 tx=0).stop 2024-01-16T15:04:54.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 shutdown_connections 2024-01-16T15:04:54.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe4bc06bbc0 0x7fe4bc06e080 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4e4160e70 0x7fe4e4147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4e4151db0 0x7fe4e4142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 --2- 172.21.15.141:0/620131326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4c4004d00 0x7fe4e413e980 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 >> 172.21.15.141:0/620131326 conn(0x7fe4e406a800 msgr2=0x7fe4e406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:54.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 shutdown_connections 2024-01-16T15:04:54.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:54.785+0000 7fe4eb200640 1 -- 172.21.15.141:0/620131326 wait complete. 2024-01-16T15:04:54.783 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T15:04:54.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:54.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:54.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[44195]: fuse finished with error 0 and tester_r 0 2024-01-16T15:04:54.885 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:04:54.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:54.885 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:04:55.067 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:04:55.068 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:55.122 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:04:55.122 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:04:55.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.123 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:04:55.170 INFO:teuthology.orchestra.run.smithi141.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:04:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:04:55.172 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:04:55.172 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:55.225 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:04:55.225 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:04:55.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.226 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:04:55.272 INFO:teuthology.orchestra.run.smithi188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:04:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:04:55.273 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:04:55.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.509+0000 7f6088e00640 1 Processor -- start 2024-01-16T15:04:55.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.509+0000 7f6088e00640 1 -- start start 2024-01-16T15:04:55.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.511+0000 7f6088e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f60840b3ae0 0x7f608415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.511+0000 7f6088e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6084070c40 con 0x7f60840cf290 2024-01-16T15:04:55.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.511+0000 7f6088e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608406c470 con 0x7f60840ae5a0 2024-01-16T15:04:55.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.511+0000 7f6088e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6084071450 con 0x7f60840b3ae0 2024-01-16T15:04:55.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.511+0000 7f6082800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f60840cf290 0x7f60840b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59558/0 (socket says 172.21.15.141:59558) 2024-01-16T15:04:55.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.511+0000 7f6082800640 1 -- 172.21.15.141:0/2223956562 learned_addr learned my addr 172.21.15.141:0/2223956562 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f607be00640 1 --2- 172.21.15.141:0/2223956562 >> v2:172.21.15.188:3300/0 conn(0x7f60840b3ae0 0x7f608415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152162211 0 0) 0x7f6084070c40 con 0x7f60840cf290 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6064003520 con 0x7f60840cf290 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 997939563 0 0) 0x7f6064003520 con 0x7f60840cf290 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 >> v2:172.21.15.188:3300/0 conn(0x7f60840b3ae0 msgr2=0x7f608415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 --2- 172.21.15.141:0/2223956562 >> v2:172.21.15.188:3300/0 conn(0x7f60840b3ae0 0x7f608415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 >> v1:172.21.15.188:6789/0 conn(0x7f60840ae5a0 legacy=0x7f608406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608406c750 con 0x7f60840cf290 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f607c0019f0 con 0x7f60840cf290 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f607be00640 1 --2- 172.21.15.141:0/2223956562 >> v2:172.21.15.188:3300/0 conn(0x7f60840b3ae0 0x7f608415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 >> v1:172.21.15.141:6789/0 conn(0x7f60840cf290 legacy=0x7f60840b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f6064003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60640047c0 0x7f6064004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6064005180 0x7f606400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6084071450 con 0x7f6064003970 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606400c0b0 con 0x7f60640047c0 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.512+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606400c270 con 0x7f6064005180 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f6081e00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f6064003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f607be00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60640047c0 0x7f6064004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f6081e00640 1 -- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6064005180 msgr2=0x7f606400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f6081e00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6064005180 0x7f606400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f6081e00640 1 -- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60640047c0 msgr2=0x7f6064004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f6081e00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60640047c0 0x7f6064004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f6081e00640 1 -- 172.21.15.141:0/2223956562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064003520 con 0x7f6064003970 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f607be00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60640047c0 0x7f6064004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:55.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.513+0000 7f6082800640 1 -- 172.21.15.141:0/2223956562 reap_dead start 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.514+0000 7f6081e00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f6064003d90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6074001750 tx=0x7f60640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94071e586ed2ea5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60740052d0 con 0x7f6064003970 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607402e050 con 0x7f6064003970 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6081400640 1 -- 172.21.15.141:0/2223956562 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074003830 con 0x7f6064003970 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6088e00640 1 -- 172.21.15.141:0/2223956562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 msgr2=0x7f6064003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6088e00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f6064003d90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6074001750 tx=0x7f60640057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6088e00640 1 -- 172.21.15.141:0/2223956562 shutdown_connections 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6088e00640 1 --2- 172.21.15.141:0/2223956562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f6064003d90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6088e00640 1 -- 172.21.15.141:0/2223956562 >> 172.21.15.141:0/2223956562 conn(0x7f608406a800 msgr2=0x7f608406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6088e00640 1 -- 172.21.15.141:0/2223956562 shutdown_connections 2024-01-16T15:04:55.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.515+0000 7f6088e00640 1 -- 172.21.15.141:0/2223956562 wait complete. 2024-01-16T15:04:55.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.518+0000 7f6088e00640 1 Processor -- start 2024-01-16T15:04:55.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.518+0000 7f6088e00640 1 -- start start 2024-01-16T15:04:55.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6088e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f608414fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6088e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60841f2d50 0x7f60841f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6088e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60841595d0 0x7f6084159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6088e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608415bb40 con 0x7f6064003970 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6088e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6084159f90 con 0x7f60841f2d50 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6088e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608415b780 con 0x7f60841595d0 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6081e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f608414fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6081e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f608414fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33466/0 (socket says 172.21.15.141:33466) 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.519+0000 7f6081e00640 1 -- 172.21.15.141:0/435218304 learned_addr learned my addr 172.21.15.141:0/435218304 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f607be00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60841f2d50 0x7f60841f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6082800640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60841595d0 0x7f6084159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6081e00640 1 -- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60841595d0 msgr2=0x7f6084159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6081e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60841595d0 0x7f6084159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6081e00640 1 -- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60841f2d50 msgr2=0x7f60841f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6081e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60841f2d50 0x7f60841f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6081e00640 1 -- 172.21.15.141:0/435218304 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60841f3810 con 0x7f6064003970 2024-01-16T15:04:55.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6081e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f608414fb10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f607402bc00 tx=0x7f6074001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f607a000640 1 -- 172.21.15.141:0/435218304 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60740076c0 con 0x7f6064003970 2024-01-16T15:04:55.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f607a000640 1 -- 172.21.15.141:0/435218304 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6074007a30 con 0x7f6064003970 2024-01-16T15:04:55.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60841f3af0 con 0x7f6064003970 2024-01-16T15:04:55.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.520+0000 7f607a000640 1 -- 172.21.15.141:0/435218304 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074006be0 con 0x7f6064003970 2024-01-16T15:04:55.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.521+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60840ce4f0 con 0x7f6064003970 2024-01-16T15:04:55.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.521+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60841f3d30 con 0x7f6064003970 2024-01-16T15:04:55.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.523+0000 7f607a000640 1 -- 172.21.15.141:0/435218304 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6074006d40 con 0x7f6064003970 2024-01-16T15:04:55.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.523+0000 7f607a000640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f605c06bbc0 0x7f605c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.524+0000 7f607a000640 1 -- 172.21.15.141:0/435218304 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f60740aa110 con 0x7f6064003970 2024-01-16T15:04:55.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.524+0000 7f607be00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f605c06bbc0 0x7f605c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.525+0000 7f607be00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f605c06bbc0 0x7f605c06e080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f608415a2b0 tx=0x7f606c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.528+0000 7f607a000640 1 -- 172.21.15.141:0/435218304 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60740769d0 con 0x7f6064003970 2024-01-16T15:04:55.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.742+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60840675d0 con 0x7f6064003970 2024-01-16T15:04:55.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.743+0000 7f607a000640 1 -- 172.21.15.141:0/435218304 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f6074041e90 con 0x7f6064003970 2024-01-16T15:04:55.722 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:55.723 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/916741426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":916741426},{"type":"v1","addr":"172.21.15.188:6837","nonce":916741426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5332,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3966594812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3966594812},{"type":"v1","addr":"172.21.15.141:6833","nonce":3966594812}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3648732517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3648732517},{"type":"v1","addr":"172.21.15.188:6835","nonce":3648732517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6990,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/456892512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":456892512},{"type":"v1","addr":"172.21.15.141:6835","nonce":456892512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[]} 2024-01-16T15:04:55.723 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 40 2024-01-16T15:04:55.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f605c06bbc0 msgr2=0x7f605c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f605c06bbc0 0x7f605c06e080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f608415a2b0 tx=0x7f606c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:55.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 msgr2=0x7f608414fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f608414fb10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f607402bc00 tx=0x7f6074001120 comp rx=0 tx=0).stop 2024-01-16T15:04:55.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 shutdown_connections 2024-01-16T15:04:55.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f605c06bbc0 0x7f605c06e080 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f60841595d0 0x7f6084159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60841f2d50 0x7f60841f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 --2- 172.21.15.141:0/435218304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6064003970 0x7f608414fb10 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 >> 172.21.15.141:0/435218304 conn(0x7f608406a800 msgr2=0x7f60840b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 shutdown_connections 2024-01-16T15:04:55.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.745+0000 7f6088e00640 1 -- 172.21.15.141:0/435218304 wait complete. 2024-01-16T15:04:55.742 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:04:55.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.971+0000 7fa826600640 1 Processor -- start 2024-01-16T15:04:55.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.971+0000 7fa826600640 1 -- start start 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.973+0000 7fa826600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa820150760 0x7fa820072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.973+0000 7fa826600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa82015d300 0x7fa82015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.973+0000 7fa826600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8200b2bd0 0x7fa8200b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.973+0000 7fa826600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82015dc80 con 0x7fa820150760 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.973+0000 7fa826600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa820071190 con 0x7fa82015d300 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.973+0000 7fa826600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa820071470 con 0x7fa8200b2bd0 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.973+0000 7fa81f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa820150760 0x7fa820072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa820150760 0x7fa820072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33470/0 (socket says 172.21.15.141:33470) 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81f400640 1 -- 172.21.15.141:0/3047659760 learned_addr learned my addr 172.21.15.141:0/3047659760 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81fe00640 1 --2- 172.21.15.141:0/3047659760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8200b2bd0 0x7fa8200b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa816800640 1 --2- 172.21.15.141:0/3047659760 >> v2:172.21.15.188:3300/0 conn(0x7fa82015d300 0x7fa82015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81f400640 1 -- 172.21.15.141:0/3047659760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8200b2bd0 msgr2=0x7fa8200b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81f400640 1 --2- 172.21.15.141:0/3047659760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8200b2bd0 0x7fa8200b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81f400640 1 -- 172.21.15.141:0/3047659760 >> v2:172.21.15.188:3300/0 conn(0x7fa82015d300 msgr2=0x7fa82015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81f400640 1 --2- 172.21.15.141:0/3047659760 >> v2:172.21.15.188:3300/0 conn(0x7fa82015d300 0x7fa82015d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.974+0000 7fa81f400640 1 -- 172.21.15.141:0/3047659760 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa820071750 con 0x7fa820150760 2024-01-16T15:04:55.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.975+0000 7fa81f400640 1 --2- 172.21.15.141:0/3047659760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa820150760 0x7fa820072230 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa8200b1ca0 tx=0x7fa81002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a688f0663a109c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa81ea00640 1 -- 172.21.15.141:0/3047659760 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81002f8e0 con 0x7fa820150760 2024-01-16T15:04:55.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa81ea00640 1 -- 172.21.15.141:0/3047659760 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81002fa40 con 0x7fa820150760 2024-01-16T15:04:55.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa81ea00640 1 -- 172.21.15.141:0/3047659760 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81002fdd0 con 0x7fa820150760 2024-01-16T15:04:55.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa826600640 1 -- 172.21.15.141:0/3047659760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa820150760 msgr2=0x7fa820072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa826600640 1 --2- 172.21.15.141:0/3047659760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa820150760 0x7fa820072230 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa8200b1ca0 tx=0x7fa81002db20 comp rx=0 tx=0).stop 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa826600640 1 -- 172.21.15.141:0/3047659760 shutdown_connections 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa826600640 1 --2- 172.21.15.141:0/3047659760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8200b2bd0 0x7fa8200b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa826600640 1 --2- 172.21.15.141:0/3047659760 >> v2:172.21.15.188:3300/0 conn(0x7fa82015d300 0x7fa82015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa826600640 1 --2- 172.21.15.141:0/3047659760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa820150760 0x7fa820072230 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.976+0000 7fa826600640 1 -- 172.21.15.141:0/3047659760 >> 172.21.15.141:0/3047659760 conn(0x7fa82006a800 msgr2=0x7fa82006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.977+0000 7fa826600640 1 -- 172.21.15.141:0/3047659760 shutdown_connections 2024-01-16T15:04:55.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.977+0000 7fa826600640 1 -- 172.21.15.141:0/3047659760 wait complete. 2024-01-16T15:04:55.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.978+0000 7fa826600640 1 Processor -- start 2024-01-16T15:04:55.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.978+0000 7fa826600640 1 -- start start 2024-01-16T15:04:55.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.979+0000 7fa826600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8200b2bd0 0x7fa820158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.979+0000 7fa826600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa820150760 0x7fa82014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.979+0000 7fa826600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa82015d300 0x7fa8200b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.979+0000 7fa826600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200cce50 con 0x7fa82015d300 2024-01-16T15:04:55.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.979+0000 7fa826600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200cd9d0 con 0x7fa8200b2bd0 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.979+0000 7fa826600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200cd0a0 con 0x7fa820150760 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa82015d300 0x7fa8200b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa82015d300 0x7fa8200b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33480/0 (socket says 172.21.15.141:33480) 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 -- 172.21.15.141:0/2828586289 learned_addr learned my addr 172.21.15.141:0/2828586289 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 -- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa820150760 msgr2=0x7fa82014ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa820150760 0x7fa82014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 -- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8200b2bd0 msgr2=0x7fa820158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8200b2bd0 0x7fa820158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 -- 172.21.15.141:0/2828586289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8100094d0 con 0x7fa82015d300 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.980+0000 7fa81fe00640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa82015d300 0x7fa8200b1ca0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa818007de0 tx=0x7fa81800d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.981+0000 7fa817e00640 1 -- 172.21.15.141:0/2828586289 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818017070 con 0x7fa82015d300 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.981+0000 7fa817e00640 1 -- 172.21.15.141:0/2828586289 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81800dba0 con 0x7fa82015d300 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.981+0000 7fa817e00640 1 -- 172.21.15.141:0/2828586289 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81800f040 con 0x7fa82015d300 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.981+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8200cd380 con 0x7fa82015d300 2024-01-16T15:04:55.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.981+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82014f1d0 con 0x7fa82015d300 2024-01-16T15:04:55.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.982+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8200675d0 con 0x7fa82015d300 2024-01-16T15:04:55.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.983+0000 7fa817e00640 1 -- 172.21.15.141:0/2828586289 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fa818008440 con 0x7fa82015d300 2024-01-16T15:04:55.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.984+0000 7fa817e00640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f806bbc0 0x7fa7f806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.984+0000 7fa817e00640 1 -- 172.21.15.141:0/2828586289 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa818088530 con 0x7fa82015d300 2024-01-16T15:04:55.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.985+0000 7fa81f400640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f806bbc0 0x7fa7f806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.986+0000 7fa81f400640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f806bbc0 0x7fa7f806e080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa810038000 tx=0x7fa810032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:55.989+0000 7fa817e00640 1 -- 172.21.15.141:0/2828586289 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa818054c00 con 0x7fa82015d300 2024-01-16T15:04:56.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.178+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fa8200cdb10 con 0x7fa82015d300 2024-01-16T15:04:56.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.183+0000 7fa817e00640 1 -- 172.21.15.141:0/2828586289 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa818058220 con 0x7fa82015d300 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f806bbc0 msgr2=0x7fa7f806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f806bbc0 0x7fa7f806e080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa810038000 tx=0x7fa810032000 comp rx=0 tx=0).stop 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa82015d300 msgr2=0x7fa8200b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa82015d300 0x7fa8200b1ca0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa818007de0 tx=0x7fa81800d490 comp rx=0 tx=0).stop 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 shutdown_connections 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7f806bbc0 0x7fa7f806e080 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa82015d300 0x7fa8200b1ca0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa820150760 0x7fa82014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 --2- 172.21.15.141:0/2828586289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8200b2bd0 0x7fa820158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 >> 172.21.15.141:0/2828586289 conn(0x7fa82006a800 msgr2=0x7fa82006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 shutdown_connections 2024-01-16T15:04:56.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.186+0000 7fa826600640 1 -- 172.21.15.141:0/2828586289 wait complete. 2024-01-16T15:04:56.185 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:04:56.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.414+0000 7fd57ba00640 1 Processor -- start 2024-01-16T15:04:56.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.414+0000 7fd57ba00640 1 -- start start 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd57ba00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5740af810 0x7fd57406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd57ba00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd5740b3ce0 0x7fd57406a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd57ba00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57415d720 con 0x7fd57415d340 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd57ba00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740b4b00 con 0x7fd5740af810 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd57ba00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740b4dd0 con 0x7fd5740b3ce0 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd573e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5740af810 0x7fd57406a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd573e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5740af810 0x7fd57406a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33484/0 (socket says 172.21.15.141:33484) 2024-01-16T15:04:56.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.416+0000 7fd573e00640 1 -- 172.21.15.141:0/3755012550 learned_addr learned my addr 172.21.15.141:0/3755012550 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:56.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.417+0000 7fd579a00640 1 --2- 172.21.15.141:0/3755012550 >> v2:172.21.15.188:3300/0 conn(0x7fd5740b3ce0 0x7fd57406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.417+0000 7fd573e00640 1 -- 172.21.15.141:0/3755012550 >> v2:172.21.15.188:3300/0 conn(0x7fd5740b3ce0 msgr2=0x7fd57406a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:56.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.417+0000 7fd573e00640 1 --2- 172.21.15.141:0/3755012550 >> v2:172.21.15.188:3300/0 conn(0x7fd5740b3ce0 0x7fd57406a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.417+0000 7fd573e00640 1 -- 172.21.15.141:0/3755012550 >> v1:172.21.15.188:6789/0 conn(0x7fd57415d340 legacy=0x7fd5740b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:56.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.417+0000 7fd573e00640 1 -- 172.21.15.141:0/3755012550 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740b3200 con 0x7fd5740af810 2024-01-16T15:04:56.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.417+0000 7fd579a00640 1 --2- 172.21.15.141:0/3755012550 >> v2:172.21.15.188:3300/0 conn(0x7fd5740b3ce0 0x7fd57406a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:56.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.418+0000 7fd573e00640 1 --2- 172.21.15.141:0/3755012550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd57406a3e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd57406f8f0 tx=0x7fd55c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c997c4b8efa9a06 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.418+0000 7fd573400640 1 -- 172.21.15.141:0/3755012550 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0317e0 con 0x7fd5740af810 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.418+0000 7fd573400640 1 -- 172.21.15.141:0/3755012550 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c031940 con 0x7fd5740af810 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.418+0000 7fd573400640 1 -- 172.21.15.141:0/3755012550 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c031cd0 con 0x7fd5740af810 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 -- 172.21.15.141:0/3755012550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 msgr2=0x7fd57406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 --2- 172.21.15.141:0/3755012550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd57406a3e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd57406f8f0 tx=0x7fd55c02b880 comp rx=0 tx=0).stop 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 -- 172.21.15.141:0/3755012550 shutdown_connections 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 --2- 172.21.15.141:0/3755012550 >> v2:172.21.15.188:3300/0 conn(0x7fd5740b3ce0 0x7fd57406a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 --2- 172.21.15.141:0/3755012550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd57406a3e0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 -- 172.21.15.141:0/3755012550 >> 172.21.15.141:0/3755012550 conn(0x7fd5740b1dd0 msgr2=0x7fd5740b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 -- 172.21.15.141:0/3755012550 shutdown_connections 2024-01-16T15:04:56.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.419+0000 7fd57ba00640 1 -- 172.21.15.141:0/3755012550 wait complete. 2024-01-16T15:04:56.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.421+0000 7fd57ba00640 1 Processor -- start 2024-01-16T15:04:56.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.421+0000 7fd57ba00640 1 -- start start 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd57ba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd5741412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd57ba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5740b3ce0 0x7fd5741589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd57ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd57415d340 0x7fd5741419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd57ba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5741585b0 con 0x7fd5740af810 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd57ba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5741607a0 con 0x7fd5740b3ce0 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd57ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5741f2e70 con 0x7fd57415d340 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd579000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd5741412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd579000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd5741412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33488/0 (socket says 172.21.15.141:33488) 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.422+0000 7fd579000640 1 -- 172.21.15.141:0/2958126032 learned_addr learned my addr 172.21.15.141:0/2958126032 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd579000640 1 -- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd57415d340 msgr2=0x7fd5741419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd579000640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd57415d340 0x7fd5741419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd573e00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5740b3ce0 0x7fd5741589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd579000640 1 -- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5740b3ce0 msgr2=0x7fd5741589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd579000640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5740b3ce0 0x7fd5741589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd579000640 1 -- 172.21.15.141:0/2958126032 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c0094d0 con 0x7fd5740af810 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd573e00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5740b3ce0 0x7fd5741589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd579000640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd5741412a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd56400b3f0 tx=0x7fd56400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd570c00640 1 -- 172.21.15.141:0/2958126032 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564011830 con 0x7fd5740af810 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd570c00640 1 -- 172.21.15.141:0/2958126032 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564011990 con 0x7fd5740af810 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd570c00640 1 -- 172.21.15.141:0/2958126032 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564011d40 con 0x7fd5740af810 2024-01-16T15:04:56.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5741604c0 con 0x7fd5740af810 2024-01-16T15:04:56.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.423+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5741581d0 con 0x7fd5740af810 2024-01-16T15:04:56.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.424+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5740b2af0 con 0x7fd5740af810 2024-01-16T15:04:56.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.425+0000 7fd570c00640 1 -- 172.21.15.141:0/2958126032 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fd56401a400 con 0x7fd5740af810 2024-01-16T15:04:56.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.426+0000 7fd570c00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd54c06bbc0 0x7fd54c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.426+0000 7fd570c00640 1 -- 172.21.15.141:0/2958126032 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd5640882f0 con 0x7fd5740af810 2024-01-16T15:04:56.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.427+0000 7fd573e00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd54c06bbc0 0x7fd54c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.428+0000 7fd573e00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd54c06bbc0 0x7fd54c06e080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd55c036000 tx=0x7fd55c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.430+0000 7fd570c00640 1 -- 172.21.15.141:0/2958126032 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd564054bb0 con 0x7fd5740af810 2024-01-16T15:04:56.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.624+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fd5740675d0 con 0x7fd5740af810 2024-01-16T15:04:56.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.629+0000 7fd570c00640 1 -- 172.21.15.141:0/2958126032 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v24) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd5640581d0 con 0x7fd5740af810 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd54c06bbc0 msgr2=0x7fd54c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd54c06bbc0 0x7fd54c06e080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd55c036000 tx=0x7fd55c03d040 comp rx=0 tx=0).stop 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 msgr2=0x7fd5741412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd5741412a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd56400b3f0 tx=0x7fd56400b8b0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 shutdown_connections 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd54c06bbc0 0x7fd54c06e080 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd57415d340 0x7fd5741419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5740b3ce0 0x7fd5741589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 --2- 172.21.15.141:0/2958126032 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5740af810 0x7fd5741412a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 >> 172.21.15.141:0/2958126032 conn(0x7fd5740b1dd0 msgr2=0x7fd57406ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:56.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 shutdown_connections 2024-01-16T15:04:56.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.632+0000 7fd57ba00640 1 -- 172.21.15.141:0/2958126032 wait complete. 2024-01-16T15:04:56.629 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:04:56.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.855+0000 7f8bc0200640 1 Processor -- start 2024-01-16T15:04:56.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.855+0000 7f8bc0200640 1 -- start start 2024-01-16T15:04:56.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bc0200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8bb8143ae0 0x7f8bb80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bc0200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bc0200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb80cfc30 con 0x7f8bb815e1e0 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bc0200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80b1ee0 con 0x7f8bb8143ae0 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bc0200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80b2780 con 0x7f8bb80cf7d0 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bbd800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8bb8143ae0 0x7f8bb80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bbe200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8bb815e1e0 0x7f8bb815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59574/0 (socket says 172.21.15.141:59574) 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.857+0000 7f8bbe200640 1 -- 172.21.15.141:0/4226737698 learned_addr learned my addr 172.21.15.141:0/4226737698 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.858+0000 7f8bbce00640 1 --2- 172.21.15.141:0/4226737698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.858+0000 7f8bbd800640 1 -- 172.21.15.141:0/4226737698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 msgr2=0x7f8bb815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.858+0000 7f8bbd800640 1 --2- 172.21.15.141:0/4226737698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.858+0000 7f8bbd800640 1 -- 172.21.15.141:0/4226737698 >> v1:172.21.15.141:6789/0 conn(0x7f8bb815e1e0 legacy=0x7f8bb815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.858+0000 7f8bbd800640 1 -- 172.21.15.141:0/4226737698 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb815e990 con 0x7f8bb8143ae0 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.859+0000 7f8bbd800640 1 --2- 172.21.15.141:0/4226737698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb8143ae0 0x7f8bb80cf290 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80097a0 tx=0x7f8ba802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e7d648e3adaf5c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.859+0000 7f8bb7e00640 1 -- 172.21.15.141:0/4226737698 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba802d040 con 0x7f8bb8143ae0 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.859+0000 7f8bb7e00640 1 -- 172.21.15.141:0/4226737698 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba8031810 con 0x7f8bb8143ae0 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.859+0000 7f8bb7e00640 1 -- 172.21.15.141:0/4226737698 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8031ba0 con 0x7f8bb8143ae0 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 -- 172.21.15.141:0/4226737698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb8143ae0 msgr2=0x7f8bb80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 --2- 172.21.15.141:0/4226737698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb8143ae0 0x7f8bb80cf290 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80097a0 tx=0x7f8ba802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 -- 172.21.15.141:0/4226737698 shutdown_connections 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 --2- 172.21.15.141:0/4226737698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 --2- 172.21.15.141:0/4226737698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb8143ae0 0x7f8bb80cf290 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 -- 172.21.15.141:0/4226737698 >> 172.21.15.141:0/4226737698 conn(0x7f8bb806a800 msgr2=0x7f8bb806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:56.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 -- 172.21.15.141:0/4226737698 shutdown_connections 2024-01-16T15:04:56.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.860+0000 7f8bc0200640 1 -- 172.21.15.141:0/4226737698 wait complete. 2024-01-16T15:04:56.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.862+0000 7f8bc0200640 1 Processor -- start 2024-01-16T15:04:56.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.862+0000 7f8bc0200640 1 -- start start 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bc0200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb81f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bc0200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bb8143ae0 0x7f8bb80c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bc0200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb815e1e0 0x7f8bb80c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bc0200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815cee0 con 0x7f8bb815e1e0 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bc0200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80c42c0 con 0x7f8bb8143ae0 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bc0200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80c45a0 con 0x7f8bb80cf7d0 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bbe200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb815e1e0 0x7f8bb80c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bbe200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb815e1e0 0x7f8bb80c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33520/0 (socket says 172.21.15.141:33520) 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bbe200640 1 -- 172.21.15.141:0/1070010923 learned_addr learned my addr 172.21.15.141:0/1070010923 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.863+0000 7f8bbd800640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb81f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bbce00640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bb8143ae0 0x7f8bb80c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bbe200640 1 -- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 msgr2=0x7f8bb81f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bbe200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb81f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bbe200640 1 -- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bb8143ae0 msgr2=0x7f8bb80c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bbe200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bb8143ae0 0x7f8bb80c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bbe200640 1 -- 172.21.15.141:0/1070010923 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba8009400 con 0x7f8bb815e1e0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bbe200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb815e1e0 0x7f8bb80c6da0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000ca80 tx=0x7f8bb000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bb5600640 1 -- 172.21.15.141:0/1070010923 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb000ac30 con 0x7f8bb815e1e0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bb5600640 1 -- 172.21.15.141:0/1070010923 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb000ad90 con 0x7f8bb815e1e0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb80c4880 con 0x7f8bb815e1e0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.864+0000 7f8bb5600640 1 -- 172.21.15.141:0/1070010923 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0012600 con 0x7f8bb815e1e0 2024-01-16T15:04:56.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.865+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb81f2580 con 0x7f8bb815e1e0 2024-01-16T15:04:56.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.866+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb80b2bf0 con 0x7f8bb815e1e0 2024-01-16T15:04:56.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.867+0000 7f8bb5600640 1 -- 172.21.15.141:0/1070010923 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f8bb0007070 con 0x7f8bb815e1e0 2024-01-16T15:04:56.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.868+0000 7f8bb5600640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b9006bbc0 0x7f8b9006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:56.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.868+0000 7f8bb5600640 1 -- 172.21.15.141:0/1070010923 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8bb00881f0 con 0x7f8bb815e1e0 2024-01-16T15:04:56.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.868+0000 7f8bbd800640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b9006bbc0 0x7f8b9006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:56.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.869+0000 7f8bbd800640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b9006bbc0 0x7f8b9006e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8036000 tx=0x7f8ba803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:56.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:56.872+0000 7f8bb5600640 1 -- 172.21.15.141:0/1070010923 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bb80b2bf0 con 0x7f8bb815e1e0 2024-01-16T15:04:57.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.063+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8bb80b2bf0 con 0x7f8bb815e1e0 2024-01-16T15:04:57.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.070+0000 7f8bb5600640 1 -- 172.21.15.141:0/1070010923 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8bb80b2bf0 con 0x7f8bb815e1e0 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b9006bbc0 msgr2=0x7f8b9006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b9006bbc0 0x7f8b9006e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8036000 tx=0x7f8ba803c040 comp rx=0 tx=0).stop 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb815e1e0 msgr2=0x7f8bb80c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb815e1e0 0x7f8bb80c6da0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000ca80 tx=0x7f8bb000cf40 comp rx=0 tx=0).stop 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 shutdown_connections 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b9006bbc0 0x7f8b9006e080 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bb815e1e0 0x7f8bb80c6da0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bb8143ae0 0x7f8bb80c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 --2- 172.21.15.141:0/1070010923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8bb80cf7d0 0x7f8bb81f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 >> 172.21.15.141:0/1070010923 conn(0x7f8bb806a800 msgr2=0x7f8bb80ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 shutdown_connections 2024-01-16T15:04:57.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.073+0000 7f8bc0200640 1 -- 172.21.15.141:0/1070010923 wait complete. 2024-01-16T15:04:57.072 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:04:57.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.301+0000 7f725a200640 1 Processor -- start 2024-01-16T15:04:57.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.301+0000 7f725a200640 1 -- start start 2024-01-16T15:04:57.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f725a200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7254143ae0 0x7f72540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f725a200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f72540cf7d0 0x7f725415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f725a200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72540cfc30 con 0x7f725415e1e0 2024-01-16T15:04:57.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f725a200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72540b1ee0 con 0x7f7254143ae0 2024-01-16T15:04:57.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f725a200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72540b2780 con 0x7f72540cf7d0 2024-01-16T15:04:57.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f7253400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7254143ae0 0x7f72540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f7253e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f725415e1e0 0x7f725415a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59584/0 (socket says 172.21.15.141:59584) 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f7253e00640 1 -- 172.21.15.141:0/3170960253 learned_addr learned my addr 172.21.15.141:0/3170960253 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f7252a00640 1 --2- 172.21.15.141:0/3170960253 >> v2:172.21.15.188:3300/0 conn(0x7f72540cf7d0 0x7f725415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.303+0000 7f7253400640 1 -- 172.21.15.141:0/3170960253 >> v2:172.21.15.188:3300/0 conn(0x7f72540cf7d0 msgr2=0x7f725415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.304+0000 7f7253400640 1 --2- 172.21.15.141:0/3170960253 >> v2:172.21.15.188:3300/0 conn(0x7f72540cf7d0 0x7f725415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.304+0000 7f7253400640 1 -- 172.21.15.141:0/3170960253 >> v1:172.21.15.141:6789/0 conn(0x7f725415e1e0 legacy=0x7f725415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.304+0000 7f7253400640 1 -- 172.21.15.141:0/3170960253 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725415e990 con 0x7f7254143ae0 2024-01-16T15:04:57.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.304+0000 7f7252000640 1 -- 172.21.15.141:0/3170960253 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660076388 0 0) 0x7f72540cfc30 con 0x7f725415e1e0 2024-01-16T15:04:57.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.305+0000 7f7253400640 1 --2- 172.21.15.141:0/3170960253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7254143ae0 0x7f72540cf290 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f72540cb830 tx=0x7f724402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0e0a98897c7145 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.305+0000 7f7252000640 1 -- 172.21.15.141:0/3170960253 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244031830 con 0x7f7254143ae0 2024-01-16T15:04:57.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.305+0000 7f7252000640 1 -- 172.21.15.141:0/3170960253 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7244031990 con 0x7f7254143ae0 2024-01-16T15:04:57.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.305+0000 7f7252000640 1 -- 172.21.15.141:0/3170960253 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244031d20 con 0x7f7254143ae0 2024-01-16T15:04:57.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 -- 172.21.15.141:0/3170960253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7254143ae0 msgr2=0x7f72540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 --2- 172.21.15.141:0/3170960253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7254143ae0 0x7f72540cf290 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f72540cb830 tx=0x7f724402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 -- 172.21.15.141:0/3170960253 shutdown_connections 2024-01-16T15:04:57.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 --2- 172.21.15.141:0/3170960253 >> v2:172.21.15.188:3300/0 conn(0x7f72540cf7d0 0x7f725415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 --2- 172.21.15.141:0/3170960253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7254143ae0 0x7f72540cf290 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 -- 172.21.15.141:0/3170960253 >> 172.21.15.141:0/3170960253 conn(0x7f725406a800 msgr2=0x7f725406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 -- 172.21.15.141:0/3170960253 shutdown_connections 2024-01-16T15:04:57.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.306+0000 7f725a200640 1 -- 172.21.15.141:0/3170960253 wait complete. 2024-01-16T15:04:57.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.308+0000 7f725a200640 1 Processor -- start 2024-01-16T15:04:57.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.308+0000 7f725a200640 1 -- start start 2024-01-16T15:04:57.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f725a200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72540cf7d0 0x7f72540aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f725a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7254143ae0 0x7f72540ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f725a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725415e1e0 0x7f72540cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f725a200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725415b100 con 0x7f72540cf7d0 2024-01-16T15:04:57.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f725a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72540af120 con 0x7f725415e1e0 2024-01-16T15:04:57.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f725a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72540af400 con 0x7f7254143ae0 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f7253400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72540cf7d0 0x7f72540aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f7253400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72540cf7d0 0x7f72540aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33538/0 (socket says 172.21.15.141:33538) 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.309+0000 7f7253400640 1 -- 172.21.15.141:0/744042586 learned_addr learned my addr 172.21.15.141:0/744042586 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7252a00640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7254143ae0 0x7f72540ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253e00640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725415e1e0 0x7f72540cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253400640 1 -- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7254143ae0 msgr2=0x7f72540ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253400640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7254143ae0 0x7f72540ae1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253400640 1 -- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725415e1e0 msgr2=0x7f72540cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253400640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725415e1e0 0x7f72540cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253400640 1 -- 172.21.15.141:0/744042586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7244009400 con 0x7f72540cf7d0 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253e00640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725415e1e0 0x7f72540cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7252a00640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7254143ae0 0x7f72540ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:57.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f7253400640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72540cf7d0 0x7f72540aea20 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7244009890 tx=0x7f72440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f724b400640 1 -- 172.21.15.141:0/744042586 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724403ba70 con 0x7f72540cf7d0 2024-01-16T15:04:57.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f724b400640 1 -- 172.21.15.141:0/744042586 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724403bbf0 con 0x7f72540cf7d0 2024-01-16T15:04:57.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f724b400640 1 -- 172.21.15.141:0/744042586 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244045400 con 0x7f72540cf7d0 2024-01-16T15:04:57.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.310+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72540af6e0 con 0x7f72540cf7d0 2024-01-16T15:04:57.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.311+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72540afca0 con 0x7f72540cf7d0 2024-01-16T15:04:57.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.312+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725415e650 con 0x7f72540cf7d0 2024-01-16T15:04:57.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.312+0000 7f724b400640 1 -- 172.21.15.141:0/744042586 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f724404e440 con 0x7f72540cf7d0 2024-01-16T15:04:57.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.313+0000 7f724b400640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f722c06bbf0 0x7f722c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.313+0000 7f724b400640 1 -- 172.21.15.141:0/744042586 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f72440aa690 con 0x7f72540cf7d0 2024-01-16T15:04:57.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.314+0000 7f7252a00640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f722c06bbf0 0x7f722c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.315+0000 7f7252a00640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f722c06bbf0 0x7f722c06e0b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f72540cbf70 tx=0x7f723c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.318+0000 7f724b400640 1 -- 172.21.15.141:0/744042586 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f725415e650 con 0x7f72540cf7d0 2024-01-16T15:04:57.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.577+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f725415e650 con 0x7f72540cf7d0 2024-01-16T15:04:57.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.664+0000 7f724b400640 1 -- 172.21.15.141:0/744042586 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v378) v1 ==== 154+0+0 (secure 0 0 0) 0x7f725415e650 con 0x7f72540cf7d0 2024-01-16T15:04:57.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f722c06bbf0 msgr2=0x7f722c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f722c06bbf0 0x7f722c06e0b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f72540cbf70 tx=0x7f723c00b040 comp rx=0 tx=0).stop 2024-01-16T15:04:57.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72540cf7d0 msgr2=0x7f72540aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72540cf7d0 0x7f72540aea20 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7244009890 tx=0x7f72440046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 shutdown_connections 2024-01-16T15:04:57.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f722c06bbf0 0x7f722c06e0b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725415e1e0 0x7f72540cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7254143ae0 0x7f72540ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 --2- 172.21.15.141:0/744042586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72540cf7d0 0x7f72540aea20 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 >> 172.21.15.141:0/744042586 conn(0x7f725406a800 msgr2=0x7f725406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 shutdown_connections 2024-01-16T15:04:57.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.668+0000 7f725a200640 1 -- 172.21.15.141:0/744042586 wait complete. 2024-01-16T15:04:57.669 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:04:57.669 INFO:tasks.cephfs_test_runner:Starting test: test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:04:57.669 INFO:tasks.cephfs_test_runner:test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:04:57.670 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:57.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.894+0000 7f5b97c00640 1 Processor -- start 2024-01-16T15:04:57.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.895+0000 7f5b97c00640 1 -- start start 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.896+0000 7f5b97c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5b900cf9a0 0x7f5b9015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.896+0000 7f5b97c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b9015e1e0 0x7f5b900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.896+0000 7f5b97c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900b1ee0 con 0x7f5b90143ae0 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.896+0000 7f5b97c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b900b2780 con 0x7f5b900cf9a0 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.896+0000 7f5b97c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9015e990 con 0x7f5b9015e1e0 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.896+0000 7f5b8fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5b900cf9a0 0x7f5b9015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.897+0000 7f5b8fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5b900cf9a0 0x7f5b9015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33554/0 (socket says 172.21.15.141:33554) 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.897+0000 7f5b8fe00640 1 -- 172.21.15.141:0/3230815159 learned_addr learned my addr 172.21.15.141:0/3230815159 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.897+0000 7f5b95c00640 1 --2- 172.21.15.141:0/3230815159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b9015e1e0 0x7f5b900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.897+0000 7f5b8fe00640 1 -- 172.21.15.141:0/3230815159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b9015e1e0 msgr2=0x7f5b900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.897+0000 7f5b8fe00640 1 --2- 172.21.15.141:0/3230815159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b9015e1e0 0x7f5b900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.897+0000 7f5b8fe00640 1 -- 172.21.15.141:0/3230815159 >> v1:172.21.15.141:6789/0 conn(0x7f5b90143ae0 legacy=0x7f5b900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.897+0000 7f5b8fe00640 1 -- 172.21.15.141:0/3230815159 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900b2130 con 0x7f5b900cf9a0 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.898+0000 7f5b8fe00640 1 --2- 172.21.15.141:0/3230815159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b900cf9a0 0x7f5b9015dca0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009870 tx=0x7f5b7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd8a361923c7ab3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.898+0000 7f5b8f400640 1 -- 172.21.15.141:0/3230815159 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78030840 con 0x7f5b900cf9a0 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.898+0000 7f5b8f400640 1 -- 172.21.15.141:0/3230815159 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b780309a0 con 0x7f5b900cf9a0 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b8f400640 1 -- 172.21.15.141:0/3230815159 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78030d30 con 0x7f5b900cf9a0 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 -- 172.21.15.141:0/3230815159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b900cf9a0 msgr2=0x7f5b9015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 --2- 172.21.15.141:0/3230815159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b900cf9a0 0x7f5b9015dca0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009870 tx=0x7f5b7802b980 comp rx=0 tx=0).stop 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 -- 172.21.15.141:0/3230815159 shutdown_connections 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 --2- 172.21.15.141:0/3230815159 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b9015e1e0 0x7f5b900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 --2- 172.21.15.141:0/3230815159 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b900cf9a0 0x7f5b9015dca0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 -- 172.21.15.141:0/3230815159 >> 172.21.15.141:0/3230815159 conn(0x7f5b9006a800 msgr2=0x7f5b9006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 -- 172.21.15.141:0/3230815159 shutdown_connections 2024-01-16T15:04:57.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.899+0000 7f5b97c00640 1 -- 172.21.15.141:0/3230815159 wait complete. 2024-01-16T15:04:57.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.901+0000 7f5b97c00640 1 Processor -- start 2024-01-16T15:04:57.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.901+0000 7f5b97c00640 1 -- start start 2024-01-16T15:04:57.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b97c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b900cf9a0 0x7f5b90159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b97c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b90143ae0 0x7f5b9015c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b97c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9015e1e0 0x7f5b9015ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b97c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b90159ed0 con 0x7f5b90143ae0 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b97c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b90140c50 con 0x7f5b9015e1e0 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b97c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b90140f30 con 0x7f5b900cf9a0 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b8fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b90143ae0 0x7f5b9015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b8fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b90143ae0 0x7f5b9015c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33566/0 (socket says 172.21.15.141:33566) 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.902+0000 7f5b8fe00640 1 -- 172.21.15.141:0/2461813455 learned_addr learned my addr 172.21.15.141:0/2461813455 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b95200640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b900cf9a0 0x7f5b90159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b95c00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9015e1e0 0x7f5b9015ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b8fe00640 1 -- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b900cf9a0 msgr2=0x7f5b90159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b8fe00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b900cf9a0 0x7f5b90159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b8fe00640 1 -- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9015e1e0 msgr2=0x7f5b9015ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b8fe00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9015e1e0 0x7f5b9015ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b8fe00640 1 -- 172.21.15.141:0/2461813455 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b780094d0 con 0x7f5b90143ae0 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.903+0000 7f5b8fe00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b90143ae0 0x7f5b9015c920 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000c00 tx=0x7f5b78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.904+0000 7f5b8cc00640 1 -- 172.21.15.141:0/2461813455 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78039c90 con 0x7f5b90143ae0 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.904+0000 7f5b8cc00640 1 -- 172.21.15.141:0/2461813455 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78039df0 con 0x7f5b90143ae0 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.904+0000 7f5b8cc00640 1 -- 172.21.15.141:0/2461813455 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78043640 con 0x7f5b90143ae0 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.904+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b90141210 con 0x7f5b90143ae0 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.904+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b901417d0 con 0x7f5b90143ae0 2024-01-16T15:04:57.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.905+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b900675d0 con 0x7f5b90143ae0 2024-01-16T15:04:57.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.906+0000 7f5b8cc00640 1 -- 172.21.15.141:0/2461813455 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5b780437a0 con 0x7f5b90143ae0 2024-01-16T15:04:57.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.907+0000 7f5b8cc00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5b6806bbc0 0x7f5b6806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.907+0000 7f5b8cc00640 1 -- 172.21.15.141:0/2461813455 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5b7803e070 con 0x7f5b90143ae0 2024-01-16T15:04:57.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.908+0000 7f5b95200640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5b6806bbc0 0x7f5b6806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.909+0000 7f5b95200640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5b6806bbc0 0x7f5b6806e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5b900cae90 tx=0x7f5b80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:57.911+0000 7f5b8cc00640 1 -- 172.21.15.141:0/2461813455 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b78077eb0 con 0x7f5b90143ae0 2024-01-16T15:04:58.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.101+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b9006d2d0 con 0x7f5b90143ae0 2024-01-16T15:04:58.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.103+0000 7f5b8cc00640 1 -- 172.21.15.141:0/2461813455 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13036 (secure 0 0 0) 0x7f5b78034090 con 0x7f5b90143ae0 2024-01-16T15:04:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:04:58.083 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":64,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:04:51.456530+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/1757492142":"2024-01-17T15:04:48.441581+0000","172.21.15.141:6834/1757492142":"2024-01-17T15:04:48.441581+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5b6806bbc0 msgr2=0x7f5b6806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5b6806bbc0 0x7f5b6806e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5b900cae90 tx=0x7f5b80009040 comp rx=0 tx=0).stop 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b90143ae0 msgr2=0x7f5b9015c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b90143ae0 0x7f5b9015c920 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000c00 tx=0x7f5b78004730 comp rx=0 tx=0).stop 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 shutdown_connections 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5b6806bbc0 0x7f5b6806e080 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9015e1e0 0x7f5b9015ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b90143ae0 0x7f5b9015c920 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 --2- 172.21.15.141:0/2461813455 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5b900cf9a0 0x7f5b90159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 >> 172.21.15.141:0/2461813455 conn(0x7f5b9006a800 msgr2=0x7f5b9006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 shutdown_connections 2024-01-16T15:04:58.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.106+0000 7f5b97c00640 1 -- 172.21.15.141:0/2461813455 wait complete. 2024-01-16T15:04:58.106 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:58.107 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:58.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.340+0000 7fd599c00640 1 Processor -- start 2024-01-16T15:04:58.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.341+0000 7fd599c00640 1 -- start start 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.342+0000 7fd599c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5940cf9a0 0x7fd5940cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.342+0000 7fd599c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd5940b31d0 0x7fd5940b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.342+0000 7fd599c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940d0320 con 0x7fd59415d4b0 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.342+0000 7fd599c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5940d1c60 con 0x7fd5940cf9a0 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.342+0000 7fd599c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5940b2730 con 0x7fd5940b31d0 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd593400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5940cf9a0 0x7fd5940cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd593e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd59415d4b0 0x7fd5940d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59598/0 (socket says 172.21.15.141:59598) 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd593e00640 1 -- 172.21.15.141:0/2828464050 learned_addr learned my addr 172.21.15.141:0/2828464050 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd599200640 1 --2- 172.21.15.141:0/2828464050 >> v2:172.21.15.188:3300/0 conn(0x7fd5940b31d0 0x7fd5940b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd593400640 1 -- 172.21.15.141:0/2828464050 >> v2:172.21.15.188:3300/0 conn(0x7fd5940b31d0 msgr2=0x7fd5940b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd593400640 1 --2- 172.21.15.141:0/2828464050 >> v2:172.21.15.188:3300/0 conn(0x7fd5940b31d0 0x7fd5940b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd593400640 1 -- 172.21.15.141:0/2828464050 >> v1:172.21.15.141:6789/0 conn(0x7fd59415d4b0 legacy=0x7fd5940d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd593400640 1 -- 172.21.15.141:0/2828464050 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5940d1ec0 con 0x7fd5940cf9a0 2024-01-16T15:04:58.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.343+0000 7fd592a00640 1 -- 172.21.15.141:0/2828464050 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446230604 0 0) 0x7fd5940d0320 con 0x7fd59415d4b0 2024-01-16T15:04:58.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.344+0000 7fd593400640 1 --2- 172.21.15.141:0/2828464050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940cf9a0 0x7fd5940cfde0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd594072a30 tx=0x7fd57c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b6a723d453b9096 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd592a00640 1 -- 172.21.15.141:0/2828464050 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031830 con 0x7fd5940cf9a0 2024-01-16T15:04:58.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd592a00640 1 -- 172.21.15.141:0/2828464050 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c031990 con 0x7fd5940cf9a0 2024-01-16T15:04:58.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd592a00640 1 -- 172.21.15.141:0/2828464050 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031d20 con 0x7fd5940cf9a0 2024-01-16T15:04:58.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd599c00640 1 -- 172.21.15.141:0/2828464050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940cf9a0 msgr2=0x7fd5940cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd599c00640 1 --2- 172.21.15.141:0/2828464050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940cf9a0 0x7fd5940cfde0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd594072a30 tx=0x7fd57c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd599c00640 1 -- 172.21.15.141:0/2828464050 shutdown_connections 2024-01-16T15:04:58.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd599c00640 1 --2- 172.21.15.141:0/2828464050 >> v2:172.21.15.188:3300/0 conn(0x7fd5940b31d0 0x7fd5940b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd599c00640 1 --2- 172.21.15.141:0/2828464050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940cf9a0 0x7fd5940cfde0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.345+0000 7fd599c00640 1 -- 172.21.15.141:0/2828464050 >> 172.21.15.141:0/2828464050 conn(0x7fd5940d3000 msgr2=0x7fd5940d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.346+0000 7fd599c00640 1 -- 172.21.15.141:0/2828464050 shutdown_connections 2024-01-16T15:04:58.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.346+0000 7fd599c00640 1 -- 172.21.15.141:0/2828464050 wait complete. 2024-01-16T15:04:58.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.347+0000 7fd599c00640 1 Processor -- start 2024-01-16T15:04:58.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.347+0000 7fd599c00640 1 -- start start 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.348+0000 7fd599c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940b31d0 0x7fd59415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.348+0000 7fd599c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5940cf9a0 0x7fd5940ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.348+0000 7fd599c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd59415d4b0 0x7fd59409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.348+0000 7fd599c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59415ca00 con 0x7fd5940b31d0 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.348+0000 7fd599c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5940caae0 con 0x7fd5940cf9a0 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.348+0000 7fd599c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5940cadc0 con 0x7fd59415d4b0 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940b31d0 0x7fd59415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940b31d0 0x7fd59415c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33590/0 (socket says 172.21.15.141:33590) 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 -- 172.21.15.141:0/3233380298 learned_addr learned my addr 172.21.15.141:0/3233380298 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593400640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5940cf9a0 0x7fd5940ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 -- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd59415d4b0 msgr2=0x7fd59409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd59415d4b0 0x7fd59409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 -- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5940cf9a0 msgr2=0x7fd5940ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5940cf9a0 0x7fd5940ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 -- 172.21.15.141:0/3233380298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c009400 con 0x7fd5940b31d0 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593400640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5940cf9a0 0x7fd5940ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.349+0000 7fd593e00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940b31d0 0x7fd59415c4c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd58400c7c0 tx=0x7fd58400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.350+0000 7fd58be00640 1 -- 172.21.15.141:0/3233380298 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58400a640 con 0x7fd5940b31d0 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.350+0000 7fd58be00640 1 -- 172.21.15.141:0/3233380298 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58400a7c0 con 0x7fd5940b31d0 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.350+0000 7fd58be00640 1 -- 172.21.15.141:0/3233380298 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58400ab70 con 0x7fd5940b31d0 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.350+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5940cb0a0 con 0x7fd5940b31d0 2024-01-16T15:04:58.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.350+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5940cb660 con 0x7fd5940b31d0 2024-01-16T15:04:58.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.351+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd594067450 con 0x7fd5940b31d0 2024-01-16T15:04:58.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.352+0000 7fd58be00640 1 -- 172.21.15.141:0/3233380298 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fd58400acd0 con 0x7fd5940b31d0 2024-01-16T15:04:58.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.353+0000 7fd58be00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd56c06bbc0 0x7fd56c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.353+0000 7fd58be00640 1 -- 172.21.15.141:0/3233380298 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd584087a90 con 0x7fd5940b31d0 2024-01-16T15:04:58.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.354+0000 7fd593400640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd56c06bbc0 0x7fd56c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.355+0000 7fd593400640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd56c06bbc0 0x7fd56c06e080 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd57c036000 tx=0x7fd57c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.358+0000 7fd58be00640 1 -- 172.21.15.141:0/3233380298 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd584054350 con 0x7fd5940b31d0 2024-01-16T15:04:58.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.547+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5940cb7f0 con 0x7fd5940b31d0 2024-01-16T15:04:58.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.547+0000 7fd58be00640 1 -- 172.21.15.141:0/3233380298 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5840180a0 con 0x7fd5940b31d0 2024-01-16T15:04:58.527 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:04:58.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd56c06bbc0 msgr2=0x7fd56c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd56c06bbc0 0x7fd56c06e080 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd57c036000 tx=0x7fd57c03c040 comp rx=0 tx=0).stop 2024-01-16T15:04:58.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940b31d0 msgr2=0x7fd59415c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940b31d0 0x7fd59415c4c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd58400c7c0 tx=0x7fd58400cc80 comp rx=0 tx=0).stop 2024-01-16T15:04:58.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 shutdown_connections 2024-01-16T15:04:58.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd56c06bbc0 0x7fd56c06e080 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd59415d4b0 0x7fd59409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5940cf9a0 0x7fd5940ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 --2- 172.21.15.141:0/3233380298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5940b31d0 0x7fd59415c4c0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 >> 172.21.15.141:0/3233380298 conn(0x7fd5940d3000 msgr2=0x7fd59406b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 shutdown_connections 2024-01-16T15:04:58.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.550+0000 7fd599c00640 1 -- 172.21.15.141:0/3233380298 wait complete. 2024-01-16T15:04:58.547 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.762+0000 7f69edc00640 1 Processor -- start 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.762+0000 7f69edc00640 1 -- start start 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69edc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f69e81469c0 0x7f69e806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69edc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 0x7f69e80d1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69edc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e80cec60 con 0x7f69e8146da0 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69edc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e80c6a00 con 0x7f69e81469c0 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69edc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e80c6ce0 con 0x7f69e8070910 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69e6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f69e81469c0 0x7f69e806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69e6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f69e81469c0 0x7f69e806b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33596/0 (socket says 172.21.15.141:33596) 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.764+0000 7f69e6a00640 1 -- 172.21.15.141:0/2399011167 learned_addr learned my addr 172.21.15.141:0/2399011167 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:58.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.765+0000 7f69e6000640 1 --2- 172.21.15.141:0/2399011167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 0x7f69e80d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.765+0000 7f69e6a00640 1 -- 172.21.15.141:0/2399011167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 msgr2=0x7f69e80d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.765+0000 7f69e6a00640 1 --2- 172.21.15.141:0/2399011167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 0x7f69e80d1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.765+0000 7f69e6a00640 1 -- 172.21.15.141:0/2399011167 >> v1:172.21.15.188:6789/0 conn(0x7f69e8146da0 legacy=0x7f69e8160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.765+0000 7f69e6a00640 1 -- 172.21.15.141:0/2399011167 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e80c6fc0 con 0x7f69e81469c0 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.766+0000 7f69e6a00640 1 --2- 172.21.15.141:0/2399011167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e806b4f0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009e50 tx=0x7f69d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f9aaa2707f4460 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.766+0000 7f69e5600640 1 -- 172.21.15.141:0/2399011167 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d802c9e0 con 0x7f69e81469c0 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.766+0000 7f69e5600640 1 -- 172.21.15.141:0/2399011167 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d802cb40 con 0x7f69e81469c0 2024-01-16T15:04:58.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.766+0000 7f69e5600640 1 -- 172.21.15.141:0/2399011167 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d80343f0 con 0x7f69e81469c0 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 -- 172.21.15.141:0/2399011167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 msgr2=0x7f69e806b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 --2- 172.21.15.141:0/2399011167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e806b4f0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009e50 tx=0x7f69d802be40 comp rx=0 tx=0).stop 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 -- 172.21.15.141:0/2399011167 shutdown_connections 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 --2- 172.21.15.141:0/2399011167 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 0x7f69e80d1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 --2- 172.21.15.141:0/2399011167 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e806b4f0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 -- 172.21.15.141:0/2399011167 >> 172.21.15.141:0/2399011167 conn(0x7f69e80b2ce0 msgr2=0x7f69e80b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 -- 172.21.15.141:0/2399011167 shutdown_connections 2024-01-16T15:04:58.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.767+0000 7f69edc00640 1 -- 172.21.15.141:0/2399011167 wait complete. 2024-01-16T15:04:58.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.769+0000 7f69edc00640 1 Processor -- start 2024-01-16T15:04:58.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.769+0000 7f69edc00640 1 -- start start 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69edc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 0x7f69e815ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69edc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e80d1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69edc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69e8146da0 0x7f69e80d1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69edc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e815ef90 con 0x7f69e81469c0 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69edc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e80b0200 con 0x7f69e8146da0 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69edc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e815d200 con 0x7f69e8070910 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69e6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e80d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69e6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e80d1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33602/0 (socket says 172.21.15.141:33602) 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69e6000640 1 -- 172.21.15.141:0/2747290907 learned_addr learned my addr 172.21.15.141:0/2747290907 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69e6000640 1 -- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 msgr2=0x7f69e815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69e6000640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 0x7f69e815ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69e7400640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69e8146da0 0x7f69e80d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.770+0000 7f69e6000640 1 -- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69e8146da0 msgr2=0x7f69e80d1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69e6000640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69e8146da0 0x7f69e80d1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69e6000640 1 -- 172.21.15.141:0/2747290907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d8009b30 con 0x7f69e81469c0 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69e7400640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69e8146da0 0x7f69e80d1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69e6000640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e80d1020 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f69d000d830 tx=0x7f69d000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69dea00640 1 -- 172.21.15.141:0/2747290907 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d000b6c0 con 0x7f69e81469c0 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69dea00640 1 -- 172.21.15.141:0/2747290907 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d000b820 con 0x7f69e81469c0 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e80af960 con 0x7f69e81469c0 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69dea00640 1 -- 172.21.15.141:0/2747290907 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d000bbb0 con 0x7f69e81469c0 2024-01-16T15:04:58.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.771+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e80aff20 con 0x7f69e81469c0 2024-01-16T15:04:58.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.772+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e80675d0 con 0x7f69e81469c0 2024-01-16T15:04:58.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.773+0000 7f69dea00640 1 -- 172.21.15.141:0/2747290907 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f69d001e460 con 0x7f69e81469c0 2024-01-16T15:04:58.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.774+0000 7f69dea00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f69c006bbc0 0x7f69c006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.774+0000 7f69dea00640 1 -- 172.21.15.141:0/2747290907 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f69d0088e60 con 0x7f69e81469c0 2024-01-16T15:04:58.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.775+0000 7f69e6a00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f69c006bbc0 0x7f69c006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.776+0000 7f69e6a00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f69c006bbc0 0x7f69c006e080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f69d8035000 tx=0x7f69d802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:58.778+0000 7f69dea00640 1 -- 172.21.15.141:0/2747290907 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69d0055530 con 0x7f69e81469c0 2024-01-16T15:04:59.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.023+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7f69e815f140 con 0x7f69e81469c0 2024-01-16T15:04:59.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.676+0000 7f69dea00640 1 -- 172.21.15.141:0/2747290907 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v380) v1 ==== 169+0+0 (secure 0 0 0) 0x7f69e815f140 con 0x7f69e81469c0 2024-01-16T15:04:59.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f69c006bbc0 msgr2=0x7f69c006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f69c006bbc0 0x7f69c006e080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f69d8035000 tx=0x7f69d802e000 comp rx=0 tx=0).stop 2024-01-16T15:04:59.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 msgr2=0x7f69e80d1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e80d1020 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f69d000d830 tx=0x7f69d000dcf0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 shutdown_connections 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f69c006bbc0 0x7f69c006e080 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69e8146da0 0x7f69e80d1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69e81469c0 0x7f69e80d1020 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 --2- 172.21.15.141:0/2747290907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69e8070910 0x7f69e815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.679+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 >> 172.21.15.141:0/2747290907 conn(0x7f69e80b2ce0 msgr2=0x7f69e80b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.680+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 shutdown_connections 2024-01-16T15:04:59.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.680+0000 7f69edc00640 1 -- 172.21.15.141:0/2747290907 wait complete. 2024-01-16T15:04:59.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:04:59.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.916+0000 7f72e9e00640 1 Processor -- start 2024-01-16T15:04:59.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.916+0000 7f72e9e00640 1 -- start start 2024-01-16T15:04:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e9e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f72e413df80 0x7f72e415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f72e413e360 0x7f72e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e9e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e406c470 con 0x7f72e413e360 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e9e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e406c750 con 0x7f72e413df80 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e9e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e406ca30 con 0x7f72e415f9c0 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72dbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f72e413e360 0x7f72e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72dbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f72e413e360 0x7f72e40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33614/0 (socket says 172.21.15.141:33614) 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72dbe00640 1 -- 172.21.15.141:0/590158050 learned_addr learned my addr 172.21.15.141:0/590158050 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e3400640 1 --2- 172.21.15.141:0/590158050 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72dbe00640 1 -- 172.21.15.141:0/590158050 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 msgr2=0x7f72e40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72e2a00640 1 --2- 172.21.15.141:0/590158050 >> v2:172.21.15.188:3300/0 conn(0x7f72e413df80 0x7f72e415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72dbe00640 1 --2- 172.21.15.141:0/590158050 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.918+0000 7f72dbe00640 1 -- 172.21.15.141:0/590158050 >> v2:172.21.15.188:3300/0 conn(0x7f72e413df80 msgr2=0x7f72e415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.919+0000 7f72dbe00640 1 --2- 172.21.15.141:0/590158050 >> v2:172.21.15.188:3300/0 conn(0x7f72e413df80 0x7f72e415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.919+0000 7f72dbe00640 1 -- 172.21.15.141:0/590158050 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e406cd10 con 0x7f72e413e360 2024-01-16T15:04:59.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.919+0000 7f72e3400640 1 --2- 172.21.15.141:0/590158050 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.920+0000 7f72dbe00640 1 --2- 172.21.15.141:0/590158050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413e360 0x7f72e40cf290 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f72e4072770 tx=0x7f72cc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4123eb6235d679be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.920+0000 7f72e2000640 1 -- 172.21.15.141:0/590158050 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc031860 con 0x7f72e413e360 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.920+0000 7f72e2000640 1 -- 172.21.15.141:0/590158050 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc0319c0 con 0x7f72e413e360 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.920+0000 7f72e2000640 1 -- 172.21.15.141:0/590158050 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc031d50 con 0x7f72e413e360 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.920+0000 7f72e9e00640 1 -- 172.21.15.141:0/590158050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413e360 msgr2=0x7f72e40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.920+0000 7f72e9e00640 1 --2- 172.21.15.141:0/590158050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413e360 0x7f72e40cf290 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f72e4072770 tx=0x7f72cc02b900 comp rx=0 tx=0).stop 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.921+0000 7f72e9e00640 1 -- 172.21.15.141:0/590158050 shutdown_connections 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.921+0000 7f72e9e00640 1 --2- 172.21.15.141:0/590158050 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.921+0000 7f72e9e00640 1 --2- 172.21.15.141:0/590158050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413e360 0x7f72e40cf290 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.921+0000 7f72e9e00640 1 --2- 172.21.15.141:0/590158050 >> v2:172.21.15.188:3300/0 conn(0x7f72e413df80 0x7f72e415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.921+0000 7f72e9e00640 1 -- 172.21.15.141:0/590158050 >> 172.21.15.141:0/590158050 conn(0x7f72e406a800 msgr2=0x7f72e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.921+0000 7f72e9e00640 1 -- 172.21.15.141:0/590158050 shutdown_connections 2024-01-16T15:04:59.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.921+0000 7f72e9e00640 1 -- 172.21.15.141:0/590158050 wait complete. 2024-01-16T15:04:59.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.922+0000 7f72e9e00640 1 Processor -- start 2024-01-16T15:04:59.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.923+0000 7f72e9e00640 1 -- start start 2024-01-16T15:04:59.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.923+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413df80 0x7f72e413c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.923+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72e413e360 0x7f72e41f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.923+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e4159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.923+0000 7f72e9e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41f3930 con 0x7f72e413df80 2024-01-16T15:04:59.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.923+0000 7f72e9e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e413cd70 con 0x7f72e413e360 2024-01-16T15:04:59.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.923+0000 7f72e9e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41f2cb0 con 0x7f72e415f9c0 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413df80 0x7f72e413c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413df80 0x7f72e413c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33620/0 (socket says 172.21.15.141:33620) 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 -- 172.21.15.141:0/10805899 learned_addr learned my addr 172.21.15.141:0/10805899 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 -- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 msgr2=0x7f72e4159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e3400640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e4159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e4159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 -- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72e413e360 msgr2=0x7f72e41f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72e413e360 0x7f72e41f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 -- 172.21.15.141:0/10805899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc0094d0 con 0x7f72e413df80 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.924+0000 7f72e2a00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413df80 0x7f72e413c830 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c7f0 tx=0x7f72d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.925+0000 7f72db400640 1 -- 172.21.15.141:0/10805899 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4004220 con 0x7f72e413df80 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.925+0000 7f72db400640 1 -- 172.21.15.141:0/10805899 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d4004380 con 0x7f72e413df80 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.925+0000 7f72db400640 1 -- 172.21.15.141:0/10805899 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4007860 con 0x7f72e413df80 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.925+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e41f2f90 con 0x7f72e413df80 2024-01-16T15:04:59.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.925+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e41f3550 con 0x7f72e413df80 2024-01-16T15:04:59.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.926+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e41f3a70 con 0x7f72e413df80 2024-01-16T15:04:59.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.927+0000 7f72db400640 1 -- 172.21.15.141:0/10805899 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f72d4007bd0 con 0x7f72e413df80 2024-01-16T15:04:59.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.928+0000 7f72db400640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f72bc06bbc0 0x7f72bc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.928+0000 7f72db400640 1 -- 172.21.15.141:0/10805899 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f72d4088440 con 0x7f72e413df80 2024-01-16T15:04:59.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.928+0000 7f72dbe00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f72bc06bbc0 0x7f72bc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.930+0000 7f72dbe00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f72bc06bbc0 0x7f72bc06e080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f72cc036000 tx=0x7f72cc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:04:59.932+0000 7f72db400640 1 -- 172.21.15.141:0/10805899 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72e41f3a70 con 0x7f72e413df80 2024-01-16T15:05:00.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.123+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f72e41f3a70 con 0x7f72e413df80 2024-01-16T15:05:00.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.128+0000 7f72db400640 1 -- 172.21.15.141:0/10805899 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f72e41f3a70 con 0x7f72e413df80 2024-01-16T15:05:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.130+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f72bc06bbc0 msgr2=0x7f72bc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.130+0000 7f72e9e00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f72bc06bbc0 0x7f72bc06e080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f72cc036000 tx=0x7f72cc030000 comp rx=0 tx=0).stop 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.130+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413df80 msgr2=0x7f72e413c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.130+0000 7f72e9e00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413df80 0x7f72e413c830 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c7f0 tx=0x7f72d400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 shutdown_connections 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f72bc06bbc0 0x7f72bc06e080 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f72e415f9c0 0x7f72e4159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72e413e360 0x7f72e41f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 --2- 172.21.15.141:0/10805899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72e413df80 0x7f72e413c830 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 >> 172.21.15.141:0/10805899 conn(0x7f72e406a800 msgr2=0x7f72e4155400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 shutdown_connections 2024-01-16T15:05:00.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.131+0000 7f72e9e00640 1 -- 172.21.15.141:0/10805899 wait complete. 2024-01-16T15:05:00.130 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:05:00.130 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:05:00.238 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:05:00.356 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:05:00.356 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:05:00.356 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:00.412 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:05:00.413 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:05:00.413 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:00.469 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:05:00.470 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:05:00.470 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:00.527 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:05:00.527 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:05:00.527 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:00.583 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:05:00.583 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:05:00.584 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.821+0000 7fa73d600640 1 Processor -- start 2024-01-16T15:05:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.822+0000 7fa73d600640 1 -- start start 2024-01-16T15:05:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.823+0000 7fa73d600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738143ae0 0x7fa7380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.823+0000 7fa73d600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa73815e3b0 0x7fa7380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.823+0000 7fa73d600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7380cfc30 con 0x7fa7380cf7d0 2024-01-16T15:05:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.823+0000 7fa73d600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7380b1ee0 con 0x7fa73815e3b0 2024-01-16T15:05:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.823+0000 7fa73d600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380b2780 con 0x7fa738143ae0 2024-01-16T15:05:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.823+0000 7fa72de00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa7380cf7d0 0x7fa73815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59608/0 (socket says 172.21.15.141:59608) 2024-01-16T15:05:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa72de00640 1 -- 172.21.15.141:0/2814761006 learned_addr learned my addr 172.21.15.141:0/2814761006 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736a00640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738143ae0 0x7fa7380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262649616 0 0) 0x7fa7380cfc30 con 0x7fa7380cf7d0 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa737400640 1 --2- 172.21.15.141:0/2814761006 >> v2:172.21.15.188:3300/0 conn(0x7fa73815e3b0 0x7fa7380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa718005730 con 0x7fa7380cf7d0 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951193842 0 0) 0x7fa718005730 con 0x7fa7380cf7d0 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738143ae0 msgr2=0x7fa7380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738143ae0 0x7fa7380cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 >> v2:172.21.15.188:3300/0 conn(0x7fa73815e3b0 msgr2=0x7fa7380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 --2- 172.21.15.141:0/2814761006 >> v2:172.21.15.188:3300/0 conn(0x7fa73815e3b0 0x7fa7380702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73815e990 con 0x7fa7380cf7d0 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa720001670 con 0x7fa7380cf7d0 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa737400640 1 --2- 172.21.15.141:0/2814761006 >> v2:172.21.15.188:3300/0 conn(0x7fa73815e3b0 0x7fa7380702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 >> v1:172.21.15.141:6789/0 conn(0x7fa7380cf7d0 legacy=0x7fa73815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.824+0000 7fa736000640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738143ae0 0x7fa718005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa736000640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006100 0x7fa7180069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa736000640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa718006f10 0x7fa718007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380b1ee0 con 0x7fa718006f10 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380b2780 con 0x7fa718006100 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71800dbe0 con 0x7fa738143ae0 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa737400640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa718006f10 0x7fa718007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa737400640 1 -- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738143ae0 msgr2=0x7fa718005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa737400640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738143ae0 0x7fa718005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa72de00640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006100 0x7fa7180069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa737400640 1 -- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006100 msgr2=0x7fa7180069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa737400640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006100 0x7fa7180069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa737400640 1 -- 172.21.15.141:0/2814761006 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa718005730 con 0x7fa718006f10 2024-01-16T15:05:00.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.825+0000 7fa72de00640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006100 0x7fa7180069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.826+0000 7fa737400640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa718006f10 0x7fa718007390 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa7300073e0 tx=0x7fa718007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a024f8bf35fc55ab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73002cdc0 con 0x7fa718006f10 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa730034440 con 0x7fa718006f10 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa736000640 1 -- 172.21.15.141:0/2814761006 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7300347b0 con 0x7fa718006f10 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa73d600640 1 -- 172.21.15.141:0/2814761006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa718006f10 msgr2=0x7fa718007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa73d600640 1 --2- 172.21.15.141:0/2814761006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa718006f10 0x7fa718007390 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa7300073e0 tx=0x7fa718007a00 comp rx=0 tx=0).stop 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa737400640 1 -- 172.21.15.141:0/2814761006 reap_dead start 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa73d600640 1 -- 172.21.15.141:0/2814761006 shutdown_connections 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa73d600640 1 -- 172.21.15.141:0/2814761006 >> 172.21.15.141:0/2814761006 conn(0x7fa73806a800 msgr2=0x7fa73806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa73d600640 1 -- 172.21.15.141:0/2814761006 shutdown_connections 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.827+0000 7fa73d600640 1 -- 172.21.15.141:0/2814761006 wait complete. 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.829+0000 7fa73d600640 1 Processor -- start 2024-01-16T15:05:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.829+0000 7fa73d600640 1 -- start start 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa73d600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006f10 0x7fa738140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa73d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa73813df30 0x7fa73813e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa73d600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738137260 0x7fa7381376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa73d600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73813ed00 con 0x7fa73813df30 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa73d600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73813efe0 con 0x7fa718006f10 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa73d600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73813f2c0 con 0x7fa738137260 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa72de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa73813df30 0x7fa73813e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa72de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa73813df30 0x7fa73813e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33634/0 (socket says 172.21.15.141:33634) 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.830+0000 7fa72de00640 1 -- 172.21.15.141:0/4177961332 learned_addr learned my addr 172.21.15.141:0/4177961332 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72de00640 1 -- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738137260 msgr2=0x7fa7381376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72de00640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738137260 0x7fa7381376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72de00640 1 -- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006f10 msgr2=0x7fa738140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa736a00640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006f10 0x7fa738140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72de00640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006f10 0x7fa738140480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72de00640 1 -- 172.21.15.141:0/4177961332 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73813f5a0 con 0x7fa73813df30 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72de00640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa73813df30 0x7fa73813e3b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa7200018a0 tx=0x7fa72000def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72f400640 1 -- 172.21.15.141:0/4177961332 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7200074e0 con 0x7fa73813df30 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72f400640 1 -- 172.21.15.141:0/4177961332 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa720007640 con 0x7fa73813df30 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73813e8f0 con 0x7fa73813df30 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa72f400640 1 -- 172.21.15.141:0/4177961332 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7200049e0 con 0x7fa73813df30 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.831+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa738140ba0 con 0x7fa73813df30 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.832+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73813f7e0 con 0x7fa73813df30 2024-01-16T15:05:00.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.834+0000 7fa72f400640 1 -- 172.21.15.141:0/4177961332 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fa720004b40 con 0x7fa73813df30 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.834+0000 7fa72f400640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa71006bbc0 0x7fa71006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.835+0000 7fa72f400640 1 -- 172.21.15.141:0/4177961332 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa720088090 con 0x7fa73813df30 2024-01-16T15:05:00.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.835+0000 7fa736a00640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa71006bbc0 0x7fa71006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.836+0000 7fa736a00640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa71006bbc0 0x7fa71006e080 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa728004770 tx=0x7fa728008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:00.839+0000 7fa72f400640 1 -- 172.21.15.141:0/4177961332 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa73813f7e0 con 0x7fa73813df30 2024-01-16T15:05:01.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.064+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa738140ba0 con 0x7fa73813df30 2024-01-16T15:05:01.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.065+0000 7fa72f400640 1 -- 172.21.15.141:0/4177961332 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fa738140ba0 con 0x7fa73813df30 2024-01-16T15:05:01.045 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:01.045 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/916741426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":916741426},{"type":"v1","addr":"172.21.15.188:6837","nonce":916741426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5332,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3966594812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3966594812},{"type":"v1","addr":"172.21.15.141:6833","nonce":3966594812}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3648732517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3648732517},{"type":"v1","addr":"172.21.15.188:6835","nonce":3648732517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6990,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/456892512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":456892512},{"type":"v1","addr":"172.21.15.141:6835","nonce":456892512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[]} 2024-01-16T15:05:01.045 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 40 2024-01-16T15:05:01.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa71006bbc0 msgr2=0x7fa71006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa71006bbc0 0x7fa71006e080 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa728004770 tx=0x7fa728008040 comp rx=0 tx=0).stop 2024-01-16T15:05:01.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa73813df30 msgr2=0x7fa73813e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa73813df30 0x7fa73813e3b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa7200018a0 tx=0x7fa72000def0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 shutdown_connections 2024-01-16T15:05:01.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa71006bbc0 0x7fa71006e080 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa738137260 0x7fa7381376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa73813df30 0x7fa73813e3b0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 --2- 172.21.15.141:0/4177961332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa718006f10 0x7fa738140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 >> 172.21.15.141:0/4177961332 conn(0x7fa73806a800 msgr2=0x7fa73815d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:01.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 shutdown_connections 2024-01-16T15:05:01.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.068+0000 7fa73d600640 1 -- 172.21.15.141:0/4177961332 wait complete. 2024-01-16T15:05:01.067 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:05:01.067 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:05:01.067 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:05:01.067 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:01.067 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:05:01.067 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:05:01.067 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:05:01.068 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:01.068 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:05:01.068 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:05:01.068 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:05:01.068 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:01.068 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:05:01.068 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:05:01.069 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:05:01.069 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:01.087 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:05:01.087 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:05:01.088 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:05:01.088 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:05:01.090 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:05:01.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:05:01.093 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:05:01.093 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:05:01.135 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:05:01.135 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:01.140 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:05:01.140 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:01.141 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:05:01.141 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:01.146 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:05:01.146 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:01.147 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:05:01.291 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T15:05:01.291 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:05:01.303 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T15:05:01.305 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T15:05:01.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.354+0000 7f1925800640 1 Processor -- start 2024-01-16T15:05:01.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.354+0000 7f1925800640 1 -- start start 2024-01-16T15:05:01.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f1925800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f192013e360 0x7f19200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f1925800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f1925800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192015c5f0 con 0x7f192013df80 2024-01-16T15:05:01.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f1925800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192006c470 con 0x7f192013e360 2024-01-16T15:05:01.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f1925800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192006c750 con 0x7f192015f9c0 2024-01-16T15:05:01.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f191f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f191f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:42654/0 (socket says 172.21.15.141:42654) 2024-01-16T15:05:01.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f191f400640 1 -- 172.21.15.141:0/732814165 learned_addr learned my addr 172.21.15.141:0/732814165 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:01.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.356+0000 7f1915e00640 1 --2- 172.21.15.141:0/732814165 >> v2:172.21.15.188:3300/0 conn(0x7f192013e360 0x7f19200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.357+0000 7f191f400640 1 -- 172.21.15.141:0/732814165 >> v2:172.21.15.188:3300/0 conn(0x7f192013e360 msgr2=0x7f19200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:01.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.357+0000 7f191f400640 1 --2- 172.21.15.141:0/732814165 >> v2:172.21.15.188:3300/0 conn(0x7f192013e360 0x7f19200cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.357+0000 7f191f400640 1 -- 172.21.15.141:0/732814165 >> v1:172.21.15.188:6789/0 conn(0x7f192013df80 legacy=0x7f192015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.357+0000 7f191f400640 1 -- 172.21.15.141:0/732814165 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192006ca30 con 0x7f192015f9c0 2024-01-16T15:05:01.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.358+0000 7f191f400640 1 --2- 172.21.15.141:0/732814165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200cf7d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1918009e50 tx=0x7f191802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6392b1d5d9695698 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.358+0000 7f191e000640 1 -- 172.21.15.141:0/732814165 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191802cb00 con 0x7f192015f9c0 2024-01-16T15:05:01.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.358+0000 7f191e000640 1 -- 172.21.15.141:0/732814165 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191802cc60 con 0x7f192015f9c0 2024-01-16T15:05:01.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.358+0000 7f191e000640 1 -- 172.21.15.141:0/732814165 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19180343f0 con 0x7f192015f9c0 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 -- 172.21.15.141:0/732814165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 msgr2=0x7f19200cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 --2- 172.21.15.141:0/732814165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200cf7d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1918009e50 tx=0x7f191802bf60 comp rx=0 tx=0).stop 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 -- 172.21.15.141:0/732814165 shutdown_connections 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 --2- 172.21.15.141:0/732814165 >> v2:172.21.15.188:3300/0 conn(0x7f192013e360 0x7f19200cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 --2- 172.21.15.141:0/732814165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200cf7d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 -- 172.21.15.141:0/732814165 >> 172.21.15.141:0/732814165 conn(0x7f192006a800 msgr2=0x7f192006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 -- 172.21.15.141:0/732814165 shutdown_connections 2024-01-16T15:05:01.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.359+0000 7f1925800640 1 -- 172.21.15.141:0/732814165 wait complete. 2024-01-16T15:05:01.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.361+0000 7f1925800640 1 Processor -- start 2024-01-16T15:05:01.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.361+0000 7f1925800640 1 -- start start 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1925800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f192013df80 0x7f19200a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1925800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f192013e360 0x7f1920153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1925800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1925800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192012f6b0 con 0x7f192013df80 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1925800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192012f990 con 0x7f192013e360 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1925800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192012fc70 con 0x7f192015f9c0 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f191ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f192013df80 0x7f19200a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1915e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f192013e360 0x7f1920153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f191ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f192013df80 0x7f19200a9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33666/0 (socket says 172.21.15.141:33666) 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f191ea00640 1 -- 172.21.15.141:0/498317112 learned_addr learned my addr 172.21.15.141:0/498317112 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f191f400640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1915e00640 1 -- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 msgr2=0x7f19200b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1915e00640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.362+0000 7f1915e00640 1 -- 172.21.15.141:0/498317112 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f192013df80 msgr2=0x7f19200a9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1915e00640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f192013df80 0x7f19200a9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:01.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1915e00640 1 -- 172.21.15.141:0/498317112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1918009b30 con 0x7f192013e360 2024-01-16T15:05:01.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1915e00640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f192013e360 0x7f1920153950 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f190800e820 tx=0x7f190800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1917400640 1 -- 172.21.15.141:0/498317112 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190800c940 con 0x7f192013e360 2024-01-16T15:05:01.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1917400640 1 -- 172.21.15.141:0/498317112 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190800caa0 con 0x7f192013e360 2024-01-16T15:05:01.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192012ff50 con 0x7f192013e360 2024-01-16T15:05:01.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1917400640 1 -- 172.21.15.141:0/498317112 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190800ce30 con 0x7f192013e360 2024-01-16T15:05:01.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.363+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1920153e90 con 0x7f192013e360 2024-01-16T15:05:01.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.364+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19200675d0 con 0x7f192013e360 2024-01-16T15:05:01.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.365+0000 7f1917400640 1 -- 172.21.15.141:0/498317112 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1908013470 con 0x7f192013e360 2024-01-16T15:05:01.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.366+0000 7f1917400640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18f806bbc0 0x7f18f806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:01.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.366+0000 7f1917400640 1 -- 172.21.15.141:0/498317112 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1908088740 con 0x7f192013e360 2024-01-16T15:05:01.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.367+0000 7f191ea00640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18f806bbc0 0x7f18f806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:01.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.368+0000 7f191ea00640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18f806bbc0 0x7f18f806e080 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1910000f60 tx=0x7f1910009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:01.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.372+0000 7f1917400640 1 -- 172.21.15.141:0/498317112 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1908054ec0 con 0x7f192013e360 2024-01-16T15:05:01.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:01.567+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f192006df00 con 0x7f192013e360 2024-01-16T15:05:02.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.730+0000 7f1917400640 1 -- 172.21.15.141:0/498317112 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v69) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1908017080 con 0x7f192013e360 2024-01-16T15:05:02.711 INFO:teuthology.orchestra.run.smithi141.stderr: removed all blocklist entries 2024-01-16T15:05:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18f806bbc0 msgr2=0x7f18f806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18f806bbc0 0x7f18f806e080 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1910000f60 tx=0x7f1910009040 comp rx=0 tx=0).stop 2024-01-16T15:05:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f192013e360 msgr2=0x7f1920153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f192013e360 0x7f1920153950 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f190800e820 tx=0x7f190800ece0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 shutdown_connections 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18f806bbc0 0x7f18f806e080 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f192015f9c0 0x7f19200b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f192013e360 0x7f1920153950 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 --2- 172.21.15.141:0/498317112 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f192013df80 0x7f19200a9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 >> 172.21.15.141:0/498317112 conn(0x7f192006a800 msgr2=0x7f192006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.734+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 shutdown_connections 2024-01-16T15:05:02.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.735+0000 7f1925800640 1 -- 172.21.15.141:0/498317112 wait complete. 2024-01-16T15:05:02.735 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:05:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.960+0000 7f0f0e800640 1 Processor -- start 2024-01-16T15:05:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.960+0000 7f0f0e800640 1 -- start start 2024-01-16T15:05:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.962+0000 7f0f0e800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0f0806d1b0 0x7f0f080cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.962+0000 7f0f0e800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0815aba0 con 0x7f0f0806cd00 2024-01-16T15:05:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.962+0000 7f0f0e800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0806d620 con 0x7f0f08153c10 2024-01-16T15:05:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.962+0000 7f0f0e800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f08159bf0 con 0x7f0f0806d1b0 2024-01-16T15:05:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.962+0000 7f0f0ca00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0f0806cd00 0x7f0f080ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:59630/0 (socket says 172.21.15.141:59630) 2024-01-16T15:05:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.962+0000 7f0f0ca00640 1 -- 172.21.15.141:0/2874030275 learned_addr learned my addr 172.21.15.141:0/2874030275 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f0de00640 1 --2- 172.21.15.141:0/2874030275 >> v2:172.21.15.188:3300/0 conn(0x7f0f0806d1b0 0x7f0f080cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847409735 0 0) 0x7f0f0815aba0 con 0x7f0f0806cd00 2024-01-16T15:05:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee8003520 con 0x7f0f0806cd00 2024-01-16T15:05:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2038884948 0 0) 0x7f0ee8003520 con 0x7f0f0806cd00 2024-01-16T15:05:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 >> v2:172.21.15.188:3300/0 conn(0x7f0f0806d1b0 msgr2=0x7f0f080cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 --2- 172.21.15.141:0/2874030275 >> v2:172.21.15.188:3300/0 conn(0x7f0f0806d1b0 0x7f0f080cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 >> v1:172.21.15.188:6789/0 conn(0x7f0f08153c10 legacy=0x7f0f080b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f08159de0 con 0x7f0f0806cd00 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0ef0001670 con 0x7f0f0806cd00 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 >> v1:172.21.15.141:6789/0 conn(0x7f0f0806cd00 legacy=0x7f0f080ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0ee8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ee80047c0 0x7f0ee8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ee8005180 0x7f0ee800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee800bff0 con 0x7f0ee8003970 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee800c160 con 0x7f0ee80047c0 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee800c320 con 0x7f0ee8005180 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.963+0000 7f0f0d400640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0ee8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0ca00640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ee80047c0 0x7f0ee8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0d400640 1 -- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ee8005180 msgr2=0x7f0ee800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0d400640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ee8005180 0x7f0ee800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0d400640 1 -- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ee80047c0 msgr2=0x7f0ee8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:02.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0d400640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ee80047c0 0x7f0ee8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0d400640 1 -- 172.21.15.141:0/2874030275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8003520 con 0x7f0ee8003970 2024-01-16T15:05:02.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0de00640 1 -- 172.21.15.141:0/2874030275 reap_dead start 2024-01-16T15:05:02.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.964+0000 7f0f0ca00640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ee80047c0 0x7f0ee8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.965+0000 7f0f0d400640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0ee8003d90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8001750 tx=0x7f0ee80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc40fd8facc63b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.965+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80052d0 con 0x7f0ee8003970 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.965+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef802e050 con 0x7f0ee8003970 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.965+0000 7f0f07e00640 1 -- 172.21.15.141:0/2874030275 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8003830 con 0x7f0ee8003970 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.966+0000 7f0f0e800640 1 -- 172.21.15.141:0/2874030275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 msgr2=0x7f0ee8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.966+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0ee8003d90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8001750 tx=0x7f0ee80057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.966+0000 7f0f0e800640 1 -- 172.21.15.141:0/2874030275 shutdown_connections 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.966+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2874030275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0ee8003d90 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.966+0000 7f0f0e800640 1 -- 172.21.15.141:0/2874030275 >> 172.21.15.141:0/2874030275 conn(0x7f0f0806a3e0 msgr2=0x7f0f0806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.966+0000 7f0f0e800640 1 -- 172.21.15.141:0/2874030275 shutdown_connections 2024-01-16T15:05:02.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.966+0000 7f0f0e800640 1 -- 172.21.15.141:0/2874030275 wait complete. 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.968+0000 7f0f0e800640 1 Processor -- start 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.968+0000 7f0f0e800640 1 -- start start 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0f08156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0f080afa20 0x7f0f0814f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f08152950 0x7f0f08152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0e800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08152190 con 0x7f0ee8003970 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0e800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08157330 con 0x7f0f08152950 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f081575d0 con 0x7f0f080afa20 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0f08156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0f08156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33694/0 (socket says 172.21.15.141:33694) 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 -- 172.21.15.141:0/2971262094 learned_addr learned my addr 172.21.15.141:0/2971262094 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0ca00640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0f080afa20 0x7f0f0814f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 -- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0f080afa20 msgr2=0x7f0f0814f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0de00640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f08152950 0x7f0f08152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0f080afa20 0x7f0f0814f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 -- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f08152950 msgr2=0x7f0f08152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f08152950 0x7f0f08152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.969+0000 7f0f0d400640 1 -- 172.21.15.141:0/2971262094 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f081578b0 con 0x7f0ee8003970 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.970+0000 7f0f0d400640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0f08156df0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0ef802bc30 tx=0x7f0ef8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.970+0000 7f0f05600640 1 -- 172.21.15.141:0/2971262094 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80076c0 con 0x7f0ee8003970 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.970+0000 7f0f05600640 1 -- 172.21.15.141:0/2971262094 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8007a30 con 0x7f0ee8003970 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.970+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f08152430 con 0x7f0ee8003970 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.970+0000 7f0f05600640 1 -- 172.21.15.141:0/2971262094 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8006be0 con 0x7f0ee8003970 2024-01-16T15:05:02.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.970+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0814ff20 con 0x7f0ee8003970 2024-01-16T15:05:02.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.971+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f08067450 con 0x7f0ee8003970 2024-01-16T15:05:02.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.973+0000 7f0f05600640 1 -- 172.21.15.141:0/2971262094 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f0ef8006d40 con 0x7f0ee8003970 2024-01-16T15:05:02.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.973+0000 7f0f05600640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ee006bbf0 0x7f0ee006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:02.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.974+0000 7f0f05600640 1 -- 172.21.15.141:0/2971262094 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0ef80a9f90 con 0x7f0ee8003970 2024-01-16T15:05:02.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.974+0000 7f0f0ca00640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ee006bbf0 0x7f0ee006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:02.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.975+0000 7f0f0ca00640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ee006bbf0 0x7f0ee006e0b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000ff0 tx=0x7f0ef0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:02.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:02.978+0000 7f0f05600640 1 -- 172.21.15.141:0/2971262094 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ef80768b0 con 0x7f0ee8003970 2024-01-16T15:05:03.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.236+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0f0806aa60 con 0x7f0ee8003970 2024-01-16T15:05:03.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.238+0000 7f0f05600640 1 -- 172.21.15.141:0/2971262094 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0ef8079ed0 con 0x7f0ee8003970 2024-01-16T15:05:03.217 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:03.217 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "auth_dump": [ 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.a", 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ==", 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.b", 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg==", 2024-01-16T15:05:03.218 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.c", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw==", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.219 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.d", 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg==", 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.0", 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA==", 2024-01-16T15:05:03.220 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.1", 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg==", 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.221 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.2", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g==", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.3", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w==", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.222 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.4", 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ==", 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.5", 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA==", 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.223 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.6", 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ==", 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.7", 2024-01-16T15:05:03.224 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg==", 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.0", 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg==", 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow r", 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.225 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.1", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ==", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.2", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ==", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:05:03.226 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.3", 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA==", 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.admin", 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAEmaZl0VgDMRAABOWZ0senIXHFvij8FQHiAg==", 2024-01-16T15:05:03.227 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.x", 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ==", 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.y", 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA==", 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2024-01-16T15:05:03.229 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:05:03.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ee006bbf0 msgr2=0x7f0ee006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ee006bbf0 0x7f0ee006e0b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000ff0 tx=0x7f0ef0009040 comp rx=0 tx=0).stop 2024-01-16T15:05:03.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 msgr2=0x7f0f08156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0f08156df0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0ef802bc30 tx=0x7f0ef8001120 comp rx=0 tx=0).stop 2024-01-16T15:05:03.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 shutdown_connections 2024-01-16T15:05:03.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ee006bbf0 0x7f0ee006e0b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f08152950 0x7f0f08152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0f080afa20 0x7f0f0814f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 --2- 172.21.15.141:0/2971262094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ee8003970 0x7f0f08156df0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 >> 172.21.15.141:0/2971262094 conn(0x7f0f0806a3e0 msgr2=0x7f0f080aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:03.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.240+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 shutdown_connections 2024-01-16T15:05:03.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.241+0000 7f0f0e800640 1 -- 172.21.15.141:0/2971262094 wait complete. 2024-01-16T15:05:03.237 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:03.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.465+0000 7f40c8a00640 1 Processor -- start 2024-01-16T15:05:03.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.466+0000 7f40c8a00640 1 -- start start 2024-01-16T15:05:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.467+0000 7f40c8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 0x7f40c00cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.467+0000 7f40c8a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c0162100 con 0x7f40c0074520 2024-01-16T15:05:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.467+0000 7f40c8a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c00ccf20 con 0x7f40c00ae750 2024-01-16T15:05:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.467+0000 7f40c8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c00cd200 con 0x7f40c00cc1e0 2024-01-16T15:05:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.467+0000 7f40c5600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f40c0074520 0x7f40c00ac050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36704/0 (socket says 172.21.15.141:36704) 2024-01-16T15:05:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.467+0000 7f40c5600640 1 -- 172.21.15.141:0/2788187493 learned_addr learned my addr 172.21.15.141:0/2788187493 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c6000640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 0x7f40c00cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687892540 0 0) 0x7f40c0162100 con 0x7f40c0074520 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a0003520 con 0x7f40c0074520 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2253262825 0 0) 0x7f40a0003520 con 0x7f40c0074520 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 msgr2=0x7f40c00cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 0x7f40c00cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 >> v1:172.21.15.188:6789/0 conn(0x7f40c00ae750 legacy=0x7f40c0161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c006daa0 con 0x7f40c0074520 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f40a8001670 con 0x7f40c0074520 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 >> v1:172.21.15.141:6789/0 conn(0x7f40c0074520 legacy=0x7f40c00ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40a00039a0 0x7f40a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 0x7f40a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40a0004d00 0x7f40a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c00cd200 con 0x7f40a00039a0 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a000bb40 con 0x7f40a0004d00 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.468+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a000bd00 con 0x7f40c00cc1e0 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6000640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40a00039a0 0x7f40a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c5600640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 0x7f40a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6000640 1 -- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 msgr2=0x7f40a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6000640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40c00cc1e0 0x7f40a00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6000640 1 -- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40a0004d00 msgr2=0x7f40a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6a00640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40a0004d00 0x7f40a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6000640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40a0004d00 0x7f40a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6000640 1 -- 172.21.15.141:0/2788187493 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a0003520 con 0x7f40a00039a0 2024-01-16T15:05:03.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.469+0000 7f40c6a00640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40a0004d00 0x7f40a0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.470+0000 7f40c6000640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40a00039a0 0x7f40a0003dc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f40b0009ac0 tx=0x7f40a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396d8ca0a8a1c7bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b002c6b0 con 0x7f40a00039a0 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b002c810 con 0x7f40a00039a0 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c4c00640 1 -- 172.21.15.141:0/2788187493 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b002cba0 con 0x7f40a00039a0 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c8a00640 1 -- 172.21.15.141:0/2788187493 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40a00039a0 msgr2=0x7f40a0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c8a00640 1 --2- 172.21.15.141:0/2788187493 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40a00039a0 0x7f40a0003dc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f40b0009ac0 tx=0x7f40a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c6a00640 1 -- 172.21.15.141:0/2788187493 reap_dead start 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c8a00640 1 -- 172.21.15.141:0/2788187493 shutdown_connections 2024-01-16T15:05:03.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.471+0000 7f40c8a00640 1 -- 172.21.15.141:0/2788187493 >> 172.21.15.141:0/2788187493 conn(0x7f40c00b0d70 msgr2=0x7f40c00b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:03.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.472+0000 7f40c8a00640 1 -- 172.21.15.141:0/2788187493 shutdown_connections 2024-01-16T15:05:03.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.472+0000 7f40c8a00640 1 -- 172.21.15.141:0/2788187493 wait complete. 2024-01-16T15:05:03.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.473+0000 7f40c8a00640 1 Processor -- start 2024-01-16T15:05:03.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.474+0000 7f40c8a00640 1 -- start start 2024-01-16T15:05:03.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.474+0000 7f40c8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40a00039a0 0x7f40c00c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.474+0000 7f40c8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c0098cd0 0x7f40c00c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.474+0000 7f40c8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40c00aeff0 0x7f40c00c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c8a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c00c53f0 con 0x7f40c00aeff0 2024-01-16T15:05:03.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c00c5dc0 con 0x7f40c0098cd0 2024-01-16T15:05:03.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c0161be0 con 0x7f40a00039a0 2024-01-16T15:05:03.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40c00aeff0 0x7f40c00c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40c00aeff0 0x7f40c00c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46384/0 (socket says 172.21.15.141:46384) 2024-01-16T15:05:03.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 -- 172.21.15.141:0/976664872 learned_addr learned my addr 172.21.15.141:0/976664872 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:03.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c5600640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c0098cd0 0x7f40c00c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6000640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40a00039a0 0x7f40c00c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 -- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40a00039a0 msgr2=0x7f40c00c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40a00039a0 0x7f40c00c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 -- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c0098cd0 msgr2=0x7f40c00c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c0098cd0 0x7f40c00c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6a00640 1 -- 172.21.15.141:0/976664872 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c00c8660 con 0x7f40c00aeff0 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.475+0000 7f40c6000640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40a00039a0 0x7f40c00c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.476+0000 7f40c6a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40c00aeff0 0x7f40c00c78e0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f40b8000f80 tx=0x7f40b800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.476+0000 7f40be000640 1 -- 172.21.15.141:0/976664872 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8006d50 con 0x7f40c00aeff0 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.476+0000 7f40be000640 1 -- 172.21.15.141:0/976664872 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b8003970 con 0x7f40c00aeff0 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.476+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c00c8940 con 0x7f40c00aeff0 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.476+0000 7f40be000640 1 -- 172.21.15.141:0/976664872 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8003d20 con 0x7f40c00aeff0 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.476+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c00c7e20 con 0x7f40c00aeff0 2024-01-16T15:05:03.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.477+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c00675d0 con 0x7f40c00aeff0 2024-01-16T15:05:03.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.478+0000 7f40be000640 1 -- 172.21.15.141:0/976664872 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f40b8005070 con 0x7f40c00aeff0 2024-01-16T15:05:03.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.479+0000 7f40be000640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f409806bbf0 0x7f409806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.479+0000 7f40be000640 1 -- 172.21.15.141:0/976664872 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f40b8088260 con 0x7f40c00aeff0 2024-01-16T15:05:03.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.480+0000 7f40c6000640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f409806bbf0 0x7f409806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.481+0000 7f40c6000640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f409806bbf0 0x7f409806e0b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f40b0036000 tx=0x7f40b0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.483+0000 7f40be000640 1 -- 172.21.15.141:0/976664872 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40b8054b20 con 0x7f40c00aeff0 2024-01-16T15:05:03.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.675+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40c00b51a0 con 0x7f40c00aeff0 2024-01-16T15:05:03.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.677+0000 7f40be000640 1 -- 172.21.15.141:0/976664872 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+12905 (secure 0 0 0) 0x7f40b801ce80 con 0x7f40c00aeff0 2024-01-16T15:05:03.657 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:03.657 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":69,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:05:02.722083+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f409806bbf0 msgr2=0x7f409806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f409806bbf0 0x7f409806e0b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f40b0036000 tx=0x7f40b0030000 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40c00aeff0 msgr2=0x7f40c00c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40c00aeff0 0x7f40c00c78e0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f40b8000f80 tx=0x7f40b800dc00 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 shutdown_connections 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f409806bbf0 0x7f409806e0b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40c00aeff0 0x7f40c00c78e0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c0098cd0 0x7f40c00c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 --2- 172.21.15.141:0/976664872 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f40a00039a0 0x7f40c00c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 >> 172.21.15.141:0/976664872 conn(0x7f40c00b0d70 msgr2=0x7f40c00adfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:03.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 shutdown_connections 2024-01-16T15:05:03.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.680+0000 7f40c8a00640 1 -- 172.21.15.141:0/976664872 wait complete. 2024-01-16T15:05:03.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:05:03.679 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:05:03.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.909+0000 7f8d4be00640 1 Processor -- start 2024-01-16T15:05:03.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.909+0000 7f8d4be00640 1 -- start start 2024-01-16T15:05:03.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4be00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8d4c143810 0x7f8d4c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8d4c0ae170 0x7f8d4c15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4be00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c0ae620 con 0x7f8d4c15d770 2024-01-16T15:05:03.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4be00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4c143bf0 con 0x7f8d4c0ae170 2024-01-16T15:05:03.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4be00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4c06c730 con 0x7f8d4c143810 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4a000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8d4c0ae170 0x7f8d4c15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4a000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8d4c0ae170 0x7f8d4c15e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46392/0 (socket says 172.21.15.141:46392) 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4a000640 1 -- 172.21.15.141:0/664499010 learned_addr learned my addr 172.21.15.141:0/664499010 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.911+0000 7f8d4aa00640 1 --2- 172.21.15.141:0/664499010 >> v2:172.21.15.188:3300/0 conn(0x7f8d4c143810 0x7f8d4c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.912+0000 7f8d4a000640 1 -- 172.21.15.141:0/664499010 >> v2:172.21.15.188:3300/0 conn(0x7f8d4c143810 msgr2=0x7f8d4c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.912+0000 7f8d4a000640 1 --2- 172.21.15.141:0/664499010 >> v2:172.21.15.188:3300/0 conn(0x7f8d4c143810 0x7f8d4c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.912+0000 7f8d4a000640 1 -- 172.21.15.141:0/664499010 >> v1:172.21.15.188:6789/0 conn(0x7f8d4c15d770 legacy=0x7f8d4c15dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:03.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.912+0000 7f8d4a000640 1 -- 172.21.15.141:0/664499010 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c0724d0 con 0x7f8d4c0ae170 2024-01-16T15:05:03.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.913+0000 7f8d4a000640 1 --2- 172.21.15.141:0/664499010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c0ae170 0x7f8d4c15e8a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0cec40 tx=0x7f8d3402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2da1bbd0f412467 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.913+0000 7f8d49600640 1 -- 172.21.15.141:0/664499010 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d340317e0 con 0x7f8d4c0ae170 2024-01-16T15:05:03.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.913+0000 7f8d49600640 1 -- 172.21.15.141:0/664499010 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34031940 con 0x7f8d4c0ae170 2024-01-16T15:05:03.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.913+0000 7f8d49600640 1 -- 172.21.15.141:0/664499010 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34031cd0 con 0x7f8d4c0ae170 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 -- 172.21.15.141:0/664499010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c0ae170 msgr2=0x7f8d4c15e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 --2- 172.21.15.141:0/664499010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c0ae170 0x7f8d4c15e8a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0cec40 tx=0x7f8d3402b880 comp rx=0 tx=0).stop 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 -- 172.21.15.141:0/664499010 shutdown_connections 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 --2- 172.21.15.141:0/664499010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c0ae170 0x7f8d4c15e8a0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 --2- 172.21.15.141:0/664499010 >> v2:172.21.15.188:3300/0 conn(0x7f8d4c143810 0x7f8d4c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 -- 172.21.15.141:0/664499010 >> 172.21.15.141:0/664499010 conn(0x7f8d4c06a7d0 msgr2=0x7f8d4c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 -- 172.21.15.141:0/664499010 shutdown_connections 2024-01-16T15:05:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.914+0000 7f8d4be00640 1 -- 172.21.15.141:0/664499010 wait complete. 2024-01-16T15:05:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.916+0000 7f8d4be00640 1 Processor -- start 2024-01-16T15:05:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.916+0000 7f8d4be00640 1 -- start start 2024-01-16T15:05:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c0ae170 0x7f8d4c0cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c143810 0x7f8d4c15c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c15d770 0x7f8d4c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4be00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0cc0c0 con 0x7f8d4c15d770 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4be00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0cc3a0 con 0x7f8d4c143810 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4be00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0cc680 con 0x7f8d4c0ae170 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c15d770 0x7f8d4c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c15d770 0x7f8d4c0cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46400/0 (socket says 172.21.15.141:46400) 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 -- 172.21.15.141:0/1704512079 learned_addr learned my addr 172.21.15.141:0/1704512079 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 -- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c0ae170 msgr2=0x7f8d4c0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c0ae170 0x7f8d4c0cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 -- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c143810 msgr2=0x7f8d4c15c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c143810 0x7f8d4c15c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.917+0000 7f8d4b400640 1 -- 172.21.15.141:0/1704512079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d340094d0 con 0x7f8d4c15d770 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.918+0000 7f8d4b400640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c15d770 0x7f8d4c0cb5e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400b970 tx=0x7f8d4400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.918+0000 7f8d42a00640 1 -- 172.21.15.141:0/1704512079 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4400cb40 con 0x7f8d4c15d770 2024-01-16T15:05:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.918+0000 7f8d42a00640 1 -- 172.21.15.141:0/1704512079 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4400cca0 con 0x7f8d4c15d770 2024-01-16T15:05:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.918+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c0cc960 con 0x7f8d4c15d770 2024-01-16T15:05:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.918+0000 7f8d42a00640 1 -- 172.21.15.141:0/1704512079 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44014680 con 0x7f8d4c15d770 2024-01-16T15:05:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.918+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c0b06e0 con 0x7f8d4c15d770 2024-01-16T15:05:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.919+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c0cce80 con 0x7f8d4c15d770 2024-01-16T15:05:03.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.921+0000 7f8d42a00640 1 -- 172.21.15.141:0/1704512079 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f8d44003970 con 0x7f8d4c15d770 2024-01-16T15:05:03.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.921+0000 7f8d42a00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406bbc0 0x7f8d2406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:03.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.922+0000 7f8d42a00640 1 -- 172.21.15.141:0/1704512079 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8d44088510 con 0x7f8d4c15d770 2024-01-16T15:05:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.922+0000 7f8d4aa00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406bbc0 0x7f8d2406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:03.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.923+0000 7f8d4aa00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406bbc0 0x7f8d2406e080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0098f0 tx=0x7f8d3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:03.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:03.925+0000 7f8d42a00640 1 -- 172.21.15.141:0/1704512079 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d44054c20 con 0x7f8d4c15d770 2024-01-16T15:05:04.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.116+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d4c0cce80 con 0x7f8d4c15d770 2024-01-16T15:05:04.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.117+0000 7f8d42a00640 1 -- 172.21.15.141:0/1704512079 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d4c0cce80 con 0x7f8d4c15d770 2024-01-16T15:05:04.097 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.119+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406bbc0 msgr2=0x7f8d2406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.119+0000 7f8d4be00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406bbc0 0x7f8d2406e080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0098f0 tx=0x7f8d3c008040 comp rx=0 tx=0).stop 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.119+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c15d770 msgr2=0x7f8d4c0cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.119+0000 7f8d4be00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c15d770 0x7f8d4c0cb5e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400b970 tx=0x7f8d4400be30 comp rx=0 tx=0).stop 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 shutdown_connections 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406bbc0 0x7f8d2406e080 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d4c15d770 0x7f8d4c0cb5e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c143810 0x7f8d4c15c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 --2- 172.21.15.141:0/1704512079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c0ae170 0x7f8d4c0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 >> 172.21.15.141:0/1704512079 conn(0x7f8d4c06a7d0 msgr2=0x7f8d4c06abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 shutdown_connections 2024-01-16T15:05:04.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.120+0000 7f8d4be00640 1 -- 172.21.15.141:0/1704512079 wait complete. 2024-01-16T15:05:04.118 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:05:04.260 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:04.268 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:04.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.343+0000 7f3677200640 1 Processor -- start 2024-01-16T15:05:04.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.343+0000 7f3677200640 1 -- start start 2024-01-16T15:05:04.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3677200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3678160920 0x7f36780b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3677200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36780cf9f0 0x7f36780cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3677200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3678161010 0x7f36781614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3677200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780d03b0 con 0x7f3678161010 2024-01-16T15:05:04.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3677200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780d0a10 con 0x7f3678160920 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3677200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780cf080 con 0x7f36780cf9f0 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3676800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3678161010 0x7f36781614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3676800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3678161010 0x7f36781614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46416/0 (socket says 172.21.15.141:46416) 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3676800640 1 -- 172.21.15.141:0/874243978 learned_addr learned my addr 172.21.15.141:0/874243978 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f3675e00640 1 --2- 172.21.15.141:0/874243978 >> v2:172.21.15.188:3300/0 conn(0x7f3678160920 0x7f36780b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.345+0000 7f366d200640 1 --2- 172.21.15.141:0/874243978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36780cf9f0 0x7f36780cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.346+0000 7f3676800640 1 -- 172.21.15.141:0/874243978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36780cf9f0 msgr2=0x7f36780cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.346+0000 7f3676800640 1 --2- 172.21.15.141:0/874243978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36780cf9f0 0x7f36780cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.346+0000 7f3676800640 1 -- 172.21.15.141:0/874243978 >> v2:172.21.15.188:3300/0 conn(0x7f3678160920 msgr2=0x7f36780b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.346+0000 7f3676800640 1 --2- 172.21.15.141:0/874243978 >> v2:172.21.15.188:3300/0 conn(0x7f3678160920 0x7f36780b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.346+0000 7f3676800640 1 -- 172.21.15.141:0/874243978 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36780c6be0 con 0x7f3678161010 2024-01-16T15:05:04.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.347+0000 7f3676800640 1 --2- 172.21.15.141:0/874243978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3678161010 0x7f36781614a0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f367808c060 tx=0x7f367002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dca7ebbbb7ace4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.347+0000 7f3675400640 1 -- 172.21.15.141:0/874243978 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670031870 con 0x7f3678161010 2024-01-16T15:05:04.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.347+0000 7f3675400640 1 -- 172.21.15.141:0/874243978 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36700319d0 con 0x7f3678161010 2024-01-16T15:05:04.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.347+0000 7f3675400640 1 -- 172.21.15.141:0/874243978 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670031d60 con 0x7f3678161010 2024-01-16T15:05:04.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.347+0000 7f3677200640 1 -- 172.21.15.141:0/874243978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3678161010 msgr2=0x7f36781614a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.347+0000 7f3677200640 1 --2- 172.21.15.141:0/874243978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3678161010 0x7f36781614a0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f367808c060 tx=0x7f367002b910 comp rx=0 tx=0).stop 2024-01-16T15:05:04.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.348+0000 7f3677200640 1 -- 172.21.15.141:0/874243978 shutdown_connections 2024-01-16T15:05:04.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.348+0000 7f3677200640 1 --2- 172.21.15.141:0/874243978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3678161010 0x7f36781614a0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.348+0000 7f3677200640 1 --2- 172.21.15.141:0/874243978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f36780cf9f0 0x7f36780cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.348+0000 7f3677200640 1 --2- 172.21.15.141:0/874243978 >> v2:172.21.15.188:3300/0 conn(0x7f3678160920 0x7f36780b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.348+0000 7f3677200640 1 -- 172.21.15.141:0/874243978 >> 172.21.15.141:0/874243978 conn(0x7f36780d3090 msgr2=0x7f36780d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.348+0000 7f3677200640 1 -- 172.21.15.141:0/874243978 shutdown_connections 2024-01-16T15:05:04.329 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.348+0000 7f3677200640 1 -- 172.21.15.141:0/874243978 wait complete. 2024-01-16T15:05:04.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.350+0000 7f3677200640 1 Processor -- start 2024-01-16T15:05:04.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.350+0000 7f3677200640 1 -- start start 2024-01-16T15:05:04.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.350+0000 7f3677200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36780cf9f0 0x7f36780cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3677200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3678160920 0x7f36780cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3677200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3678161010 0x7f36780cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3677200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36780ccf80 con 0x7f36780cf9f0 2024-01-16T15:05:04.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3677200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36781f3320 con 0x7f3678161010 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3677200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36781f3600 con 0x7f3678160920 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36780cf9f0 0x7f36780cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36780cf9f0 0x7f36780cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46426/0 (socket says 172.21.15.141:46426) 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 -- 172.21.15.141:0/825751038 learned_addr learned my addr 172.21.15.141:0/825751038 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f366d200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3678160920 0x7f36780cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3676800640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3678161010 0x7f36780cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 -- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3678160920 msgr2=0x7f36780cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3678160920 0x7f36780cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 -- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3678161010 msgr2=0x7f36780cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3678161010 0x7f36780cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3675e00640 1 -- 172.21.15.141:0/825751038 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36700094d0 con 0x7f36780cf9f0 2024-01-16T15:05:04.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f366d200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3678160920 0x7f36780cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:04.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.351+0000 7f3676800640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3678161010 0x7f36780cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:04.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.352+0000 7f3675e00640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36780cf9f0 0x7f36780cca40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f366800efc0 tx=0x7f366800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.352+0000 7f366ea00640 1 -- 172.21.15.141:0/825751038 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668014480 con 0x7f36780cf9f0 2024-01-16T15:05:04.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.352+0000 7f366ea00640 1 -- 172.21.15.141:0/825751038 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366800f040 con 0x7f36780cf9f0 2024-01-16T15:05:04.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.352+0000 7f366ea00640 1 -- 172.21.15.141:0/825751038 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668014800 con 0x7f36780cf9f0 2024-01-16T15:05:04.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.352+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36781f38e0 con 0x7f36780cf9f0 2024-01-16T15:05:04.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.352+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367814cda0 con 0x7f36780cf9f0 2024-01-16T15:05:04.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.353+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3678067450 con 0x7f36780cf9f0 2024-01-16T15:05:04.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.355+0000 7f366ea00640 1 -- 172.21.15.141:0/825751038 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f3668014960 con 0x7f36780cf9f0 2024-01-16T15:05:04.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.355+0000 7f366ea00640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f365006bbc0 0x7f365006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.356+0000 7f366ea00640 1 -- 172.21.15.141:0/825751038 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f366808b340 con 0x7f36780cf9f0 2024-01-16T15:05:04.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.356+0000 7f366d200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f365006bbc0 0x7f365006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.357+0000 7f366d200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f365006bbc0 0x7f365006e080 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f367814dae0 tx=0x7f3660008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.360+0000 7f366ea00640 1 -- 172.21.15.141:0/825751038 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36680559f0 con 0x7f36780cf9f0 2024-01-16T15:05:04.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.555+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3678157ea0 con 0x7f36780cf9f0 2024-01-16T15:05:04.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.555+0000 7f366ea00640 1 -- 172.21.15.141:0/825751038 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v69) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3668059010 con 0x7f36780cf9f0 2024-01-16T15:05:04.535 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:04.535 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:05:04.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:04.536 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T15:05:04.536 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T15:05:04.536 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:04.536 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:05:04.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f365006bbc0 msgr2=0x7f365006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f365006bbc0 0x7f365006e080 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f367814dae0 tx=0x7f3660008040 comp rx=0 tx=0).stop 2024-01-16T15:05:04.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36780cf9f0 msgr2=0x7f36780cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36780cf9f0 0x7f36780cca40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f366800efc0 tx=0x7f366800c490 comp rx=0 tx=0).stop 2024-01-16T15:05:04.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 shutdown_connections 2024-01-16T15:05:04.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f365006bbc0 0x7f365006e080 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3678161010 0x7f36780cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3678160920 0x7f36780cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 --2- 172.21.15.141:0/825751038 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f36780cf9f0 0x7f36780cca40 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 >> 172.21.15.141:0/825751038 conn(0x7f36780d3090 msgr2=0x7f36780cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 shutdown_connections 2024-01-16T15:05:04.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.558+0000 7f3677200640 1 -- 172.21.15.141:0/825751038 wait complete. 2024-01-16T15:05:04.557 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:05:04.557 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:05:04.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.792+0000 7f142e600640 1 Processor -- start 2024-01-16T15:05:04.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.793+0000 7f142e600640 1 -- start start 2024-01-16T15:05:04.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.794+0000 7f142e600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14280ae5a0 0x7f142815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.794+0000 7f142e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280cf290 0x7f14280cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.794+0000 7f142e600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1428070c40 con 0x7f14280b3ae0 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.794+0000 7f142e600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1428071450 con 0x7f14280ae5a0 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.794+0000 7f142e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14280cfc60 con 0x7f14280cf290 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1427400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14280ae5a0 0x7f142815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f141e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f14280b3ae0 0x7f142806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36720/0 (socket says 172.21.15.141:36720) 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f141e800640 1 -- 172.21.15.141:0/384084758 learned_addr learned my addr 172.21.15.141:0/384084758 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1427e00640 1 --2- 172.21.15.141:0/384084758 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280cf290 0x7f14280cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1426a00640 1 -- 172.21.15.141:0/384084758 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525292348 0 0) 0x7f1428070c40 con 0x7f14280b3ae0 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1426a00640 1 -- 172.21.15.141:0/384084758 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1408003520 con 0x7f14280b3ae0 2024-01-16T15:05:04.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1427400640 1 -- 172.21.15.141:0/384084758 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280cf290 msgr2=0x7f14280cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1427400640 1 --2- 172.21.15.141:0/384084758 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280cf290 0x7f14280cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1427400640 1 -- 172.21.15.141:0/384084758 >> v1:172.21.15.141:6789/0 conn(0x7f14280b3ae0 legacy=0x7f142806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.795+0000 7f1427400640 1 -- 172.21.15.141:0/384084758 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142806c470 con 0x7f14280ae5a0 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.797+0000 7f1427400640 1 --2- 172.21.15.141:0/384084758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280ae5a0 0x7f142815e430 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f14180097a0 tx=0x7f141802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71ec9a7301ad08a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.798+0000 7f1426a00640 1 -- 172.21.15.141:0/384084758 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418030610 con 0x7f14280ae5a0 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.798+0000 7f1426a00640 1 -- 172.21.15.141:0/384084758 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1418030770 con 0x7f14280ae5a0 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.798+0000 7f1426a00640 1 -- 172.21.15.141:0/384084758 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418030b00 con 0x7f14280ae5a0 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.798+0000 7f142e600640 1 -- 172.21.15.141:0/384084758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280ae5a0 msgr2=0x7f142815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.798+0000 7f142e600640 1 --2- 172.21.15.141:0/384084758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280ae5a0 0x7f142815e430 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f14180097a0 tx=0x7f141802b960 comp rx=0 tx=0).stop 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.799+0000 7f142e600640 1 -- 172.21.15.141:0/384084758 shutdown_connections 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.799+0000 7f142e600640 1 --2- 172.21.15.141:0/384084758 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280cf290 0x7f14280cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.799+0000 7f142e600640 1 --2- 172.21.15.141:0/384084758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280ae5a0 0x7f142815e430 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.799+0000 7f142e600640 1 -- 172.21.15.141:0/384084758 >> 172.21.15.141:0/384084758 conn(0x7f142806a800 msgr2=0x7f142806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.799+0000 7f142e600640 1 -- 172.21.15.141:0/384084758 shutdown_connections 2024-01-16T15:05:04.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.799+0000 7f142e600640 1 -- 172.21.15.141:0/384084758 wait complete. 2024-01-16T15:05:04.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.800+0000 7f142e600640 1 Processor -- start 2024-01-16T15:05:04.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.801+0000 7f142e600640 1 -- start start 2024-01-16T15:05:04.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.801+0000 7f142e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280ae5a0 0x7f1428150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.801+0000 7f142e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14280b3ae0 0x7f14281605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.801+0000 7f142e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280cf290 0x7f142814f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.801+0000 7f142e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14280ce190 con 0x7f14280cf290 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.801+0000 7f142e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1428160ae0 con 0x7f14280b3ae0 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.801+0000 7f142e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142814fa70 con 0x7f14280ae5a0 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280cf290 0x7f142814f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280cf290 0x7f142814f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46446/0 (socket says 172.21.15.141:46446) 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 -- 172.21.15.141:0/3881750056 learned_addr learned my addr 172.21.15.141:0/3881750056 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 -- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280ae5a0 msgr2=0x7f1428150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427400640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280ae5a0 0x7f1428150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280ae5a0 0x7f1428150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 -- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14280b3ae0 msgr2=0x7f14281605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14280b3ae0 0x7f14281605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.802+0000 7f1427e00640 1 -- 172.21.15.141:0/3881750056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1418009400 con 0x7f14280cf290 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.803+0000 7f1427e00640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280cf290 0x7f142814f530 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1420007e90 tx=0x7f142000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.803+0000 7f141fe00640 1 -- 172.21.15.141:0/3881750056 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420014420 con 0x7f14280cf290 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.803+0000 7f141fe00640 1 -- 172.21.15.141:0/3881750056 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14200145a0 con 0x7f14280cf290 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.803+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14281507c0 con 0x7f14280cf290 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.803+0000 7f141fe00640 1 -- 172.21.15.141:0/3881750056 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420014950 con 0x7f14280cf290 2024-01-16T15:05:04.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.803+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1428150d80 con 0x7f14280cf290 2024-01-16T15:05:04.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.804+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14280675d0 con 0x7f14280cf290 2024-01-16T15:05:04.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.805+0000 7f141fe00640 1 -- 172.21.15.141:0/3881750056 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1420014ab0 con 0x7f14280cf290 2024-01-16T15:05:04.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.806+0000 7f141fe00640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f140006bbc0 0x7f140006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.806+0000 7f141fe00640 1 -- 172.21.15.141:0/3881750056 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1420088dc0 con 0x7f14280cf290 2024-01-16T15:05:04.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.806+0000 7f1427400640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f140006bbc0 0x7f140006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.808+0000 7f1427400640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f140006bbc0 0x7f140006e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1418036000 tx=0x7f141803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:04.810+0000 7f141fe00640 1 -- 172.21.15.141:0/3881750056 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1420055680 con 0x7f14280cf290 2024-01-16T15:05:04.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.002+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1428070990 con 0x7f14280cf290 2024-01-16T15:05:05.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.717+0000 7f141fe00640 1 -- 172.21.15.141:0/3881750056 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v70) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1420058ca0 con 0x7f14280cf290 2024-01-16T15:05:05.698 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T15:05:05.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f140006bbc0 msgr2=0x7f140006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f140006bbc0 0x7f140006e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1418036000 tx=0x7f141803b040 comp rx=0 tx=0).stop 2024-01-16T15:05:05.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280cf290 msgr2=0x7f142814f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280cf290 0x7f142814f530 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1420007e90 tx=0x7f142000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 shutdown_connections 2024-01-16T15:05:05.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f140006bbc0 0x7f140006e080 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14280cf290 0x7f142814f530 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14280b3ae0 0x7f14281605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 --2- 172.21.15.141:0/3881750056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14280ae5a0 0x7f1428150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 >> 172.21.15.141:0/3881750056 conn(0x7f142806a800 msgr2=0x7f14280b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 shutdown_connections 2024-01-16T15:05:05.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.722+0000 7f142e600640 1 -- 172.21.15.141:0/3881750056 wait complete. 2024-01-16T15:05:05.729 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:05:05.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.953+0000 7f9ff6200640 1 Processor -- start 2024-01-16T15:05:05.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.953+0000 7f9ff6200640 1 -- start start 2024-01-16T15:05:05.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.954+0000 7f9ff6200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b34e0 0x7f9ff00b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.954+0000 7f9ff6200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9ff00d1d60 0x7f9ff00c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.954+0000 7f9ff6200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff00cec40 con 0x7f9ff00b4140 2024-01-16T15:05:05.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.954+0000 7f9ff6200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff00d0420 con 0x7f9ff00d1d60 2024-01-16T15:05:05.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.955+0000 7f9ff6200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff00d0e70 con 0x7f9ff00b34e0 2024-01-16T15:05:05.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.955+0000 7f9ff4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b34e0 0x7f9ff00b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.955+0000 7f9ff4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b34e0 0x7f9ff00b3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:50446/0 (socket says 172.21.15.141:50446) 2024-01-16T15:05:05.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.955+0000 7f9ff4e00640 1 -- 172.21.15.141:0/3610628157 learned_addr learned my addr 172.21.15.141:0/3610628157 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:05.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.955+0000 7f9fe7e00640 1 --1- 172.21.15.141:0/3610628157 >> v1:172.21.15.188:6789/0 conn(0x7f9ff00b4140 0x7f9ff00d1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:35418/0 (socket says 172.21.15.141:35418) 2024-01-16T15:05:05.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.955+0000 7f9ff5800640 1 --2- 172.21.15.141:0/3610628157 >> v2:172.21.15.188:3300/0 conn(0x7f9ff00d1d60 0x7f9ff00c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.956+0000 7f9ff4e00640 1 -- 172.21.15.141:0/3610628157 >> v2:172.21.15.188:3300/0 conn(0x7f9ff00d1d60 msgr2=0x7f9ff00c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:05.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.956+0000 7f9ff4e00640 1 --2- 172.21.15.141:0/3610628157 >> v2:172.21.15.188:3300/0 conn(0x7f9ff00d1d60 0x7f9ff00c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.956+0000 7f9ff4e00640 1 -- 172.21.15.141:0/3610628157 >> v1:172.21.15.188:6789/0 conn(0x7f9ff00b4140 legacy=0x7f9ff00d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.956+0000 7f9ff4e00640 1 -- 172.21.15.141:0/3610628157 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff00d3d50 con 0x7f9ff00b34e0 2024-01-16T15:05:05.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.958+0000 7f9ff4e00640 1 --2- 172.21.15.141:0/3610628157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b34e0 0x7f9ff00b3c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009870 tx=0x7f9fe002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23534cf2d674fd21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.958+0000 7f9fefe00640 1 -- 172.21.15.141:0/3610628157 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0030840 con 0x7f9ff00b34e0 2024-01-16T15:05:05.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9fefe00640 1 -- 172.21.15.141:0/3610628157 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe00309a0 con 0x7f9ff00b34e0 2024-01-16T15:05:05.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9fefe00640 1 -- 172.21.15.141:0/3610628157 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0030d30 con 0x7f9ff00b34e0 2024-01-16T15:05:05.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 -- 172.21.15.141:0/3610628157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b34e0 msgr2=0x7f9ff00b3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 --2- 172.21.15.141:0/3610628157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b34e0 0x7f9ff00b3c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009870 tx=0x7f9fe002b980 comp rx=0 tx=0).stop 2024-01-16T15:05:05.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 -- 172.21.15.141:0/3610628157 shutdown_connections 2024-01-16T15:05:05.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 --2- 172.21.15.141:0/3610628157 >> v2:172.21.15.188:3300/0 conn(0x7f9ff00d1d60 0x7f9ff00c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 --2- 172.21.15.141:0/3610628157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b34e0 0x7f9ff00b3c00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 -- 172.21.15.141:0/3610628157 >> 172.21.15.141:0/3610628157 conn(0x7f9ff00d3090 msgr2=0x7f9ff00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 -- 172.21.15.141:0/3610628157 shutdown_connections 2024-01-16T15:05:05.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.959+0000 7f9ff6200640 1 -- 172.21.15.141:0/3610628157 wait complete. 2024-01-16T15:05:05.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.961+0000 7f9ff6200640 1 Processor -- start 2024-01-16T15:05:05.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.961+0000 7f9ff6200640 1 -- start start 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.962+0000 7f9ff6200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff00b34e0 0x7f9ff00b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.962+0000 7f9ff6200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b4140 0x7f9ff00afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.962+0000 7f9ff6200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ff00d1d60 0x7f9ff015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.962+0000 7f9ff6200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff0156e30 con 0x7f9ff00b34e0 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.962+0000 7f9ff6200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff015de50 con 0x7f9ff00d1d60 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.962+0000 7f9ff6200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff015e130 con 0x7f9ff00b4140 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.962+0000 7f9ff4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff00b34e0 0x7f9ff00b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff00b34e0 0x7f9ff00b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46460/0 (socket says 172.21.15.141:46460) 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 -- 172.21.15.141:0/1352115950 learned_addr learned my addr 172.21.15.141:0/1352115950 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9fe7e00640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b4140 0x7f9ff00afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff5800640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ff00d1d60 0x7f9ff015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 -- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b4140 msgr2=0x7f9ff00afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b4140 0x7f9ff00afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 -- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ff00d1d60 msgr2=0x7f9ff015b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ff00d1d60 0x7f9ff015b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 -- 172.21.15.141:0/1352115950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe00094d0 con 0x7f9ff00b34e0 2024-01-16T15:05:05.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff5800640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ff00d1d60 0x7f9ff015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:05.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.963+0000 7f9ff4e00640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff00b34e0 0x7f9ff00b1c30 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0000c00 tx=0x7f9fe0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.964+0000 7f9fed600640 1 -- 172.21.15.141:0/1352115950 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0039c90 con 0x7f9ff00b34e0 2024-01-16T15:05:05.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.964+0000 7f9fed600640 1 -- 172.21.15.141:0/1352115950 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe0039df0 con 0x7f9ff00b34e0 2024-01-16T15:05:05.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.964+0000 7f9fed600640 1 -- 172.21.15.141:0/1352115950 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0043640 con 0x7f9ff00b34e0 2024-01-16T15:05:05.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.964+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff015e410 con 0x7f9ff00b34e0 2024-01-16T15:05:05.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.964+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff015e9d0 con 0x7f9ff00b34e0 2024-01-16T15:05:05.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.965+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff0067450 con 0x7f9ff00b34e0 2024-01-16T15:05:05.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.966+0000 7f9fed600640 1 -- 172.21.15.141:0/1352115950 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f9fe00437a0 con 0x7f9ff00b34e0 2024-01-16T15:05:05.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.967+0000 7f9fed600640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fc806bbc0 0x7f9fc806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.967+0000 7f9fed600640 1 -- 172.21.15.141:0/1352115950 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9fe003e070 con 0x7f9ff00b34e0 2024-01-16T15:05:05.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.967+0000 7f9fe7e00640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fc806bbc0 0x7f9fc806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.969+0000 7f9fe7e00640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fc806bbc0 0x7f9fc806e080 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ff01f6330 tx=0x7f9fd8007c80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:05.972+0000 7f9fed600640 1 -- 172.21.15.141:0/1352115950 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fe0077da0 con 0x7f9ff00b34e0 2024-01-16T15:05:06.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.164+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9ff00ce8b0 con 0x7f9ff00b34e0 2024-01-16T15:05:06.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.733+0000 7f9fed600640 1 -- 172.21.15.141:0/1352115950 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v71) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9fe00340d0 con 0x7f9ff00b34e0 2024-01-16T15:05:06.713 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T15:05:06.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fc806bbc0 msgr2=0x7f9fc806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fc806bbc0 0x7f9fc806e080 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ff01f6330 tx=0x7f9fd8007c80 comp rx=0 tx=0).stop 2024-01-16T15:05:06.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff00b34e0 msgr2=0x7f9ff00b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff00b34e0 0x7f9ff00b1c30 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0000c00 tx=0x7f9fe0004730 comp rx=0 tx=0).stop 2024-01-16T15:05:06.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 shutdown_connections 2024-01-16T15:05:06.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fc806bbc0 0x7f9fc806e080 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ff00d1d60 0x7f9ff015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ff00b4140 0x7f9ff00afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 --2- 172.21.15.141:0/1352115950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff00b34e0 0x7f9ff00b1c30 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.737+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 >> 172.21.15.141:0/1352115950 conn(0x7f9ff00d3090 msgr2=0x7f9ff00c7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.738+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 shutdown_connections 2024-01-16T15:05:06.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.738+0000 7f9ff6200640 1 -- 172.21.15.141:0/1352115950 wait complete. 2024-01-16T15:05:06.748 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:05:06.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.952+0000 7f5097600640 1 Processor -- start 2024-01-16T15:05:06.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.952+0000 7f5097600640 1 -- start start 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.953+0000 7f5097600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f509013df80 0x7f509015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.953+0000 7f5097600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f509013e360 0x7f50900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.953+0000 7f5097600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f509006c470 con 0x7f509015f9c0 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.953+0000 7f5097600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509006c750 con 0x7f509013e360 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.953+0000 7f5097600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509006ca30 con 0x7f509013df80 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5087c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f509013e360 0x7f50900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5087c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f509013e360 0x7f50900cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46474/0 (socket says 172.21.15.141:46474) 2024-01-16T15:05:06.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5087c00640 1 -- 172.21.15.141:0/1687180454 learned_addr learned my addr 172.21.15.141:0/1687180454 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:06.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5094c00640 1 --2- 172.21.15.141:0/1687180454 >> v2:172.21.15.188:3300/0 conn(0x7f509013df80 0x7f509015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5087c00640 1 -- 172.21.15.141:0/1687180454 >> v2:172.21.15.188:3300/0 conn(0x7f509013df80 msgr2=0x7f509015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:06.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5087c00640 1 --2- 172.21.15.141:0/1687180454 >> v2:172.21.15.188:3300/0 conn(0x7f509013df80 0x7f509015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5087c00640 1 -- 172.21.15.141:0/1687180454 >> v1:172.21.15.141:6789/0 conn(0x7f509015f9c0 legacy=0x7f50901408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f5087c00640 1 -- 172.21.15.141:0/1687180454 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509006cd10 con 0x7f509013e360 2024-01-16T15:05:06.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.954+0000 7f508fe00640 1 -- 172.21.15.141:0/1687180454 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970090281 0 0) 0x7f509006c470 con 0x7f509015f9c0 2024-01-16T15:05:06.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.955+0000 7f5087c00640 1 --2- 172.21.15.141:0/1687180454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509013e360 0x7f50900cf290 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5090072770 tx=0x7f507802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24b61e0dce3d63d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f508fe00640 1 -- 172.21.15.141:0/1687180454 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078030860 con 0x7f509013e360 2024-01-16T15:05:06.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f508fe00640 1 -- 172.21.15.141:0/1687180454 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50780309c0 con 0x7f509013e360 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f508fe00640 1 -- 172.21.15.141:0/1687180454 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078030d50 con 0x7f509013e360 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 -- 172.21.15.141:0/1687180454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509013e360 msgr2=0x7f50900cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 --2- 172.21.15.141:0/1687180454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509013e360 0x7f50900cf290 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5090072770 tx=0x7f507802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 -- 172.21.15.141:0/1687180454 shutdown_connections 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 --2- 172.21.15.141:0/1687180454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509013e360 0x7f50900cf290 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 --2- 172.21.15.141:0/1687180454 >> v2:172.21.15.188:3300/0 conn(0x7f509013df80 0x7f509015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 -- 172.21.15.141:0/1687180454 >> 172.21.15.141:0/1687180454 conn(0x7f509006a800 msgr2=0x7f509006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 -- 172.21.15.141:0/1687180454 shutdown_connections 2024-01-16T15:05:06.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.956+0000 7f5097600640 1 -- 172.21.15.141:0/1687180454 wait complete. 2024-01-16T15:05:06.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.958+0000 7f5097600640 1 Processor -- start 2024-01-16T15:05:06.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.958+0000 7f5097600640 1 -- start start 2024-01-16T15:05:06.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.959+0000 7f5097600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f509013df80 0x7f50901f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.959+0000 7f5097600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f509013e360 0x7f509014d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.959+0000 7f5097600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509015f9c0 0x7f509015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.959+0000 7f5097600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5090133a80 con 0x7f509015f9c0 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.959+0000 7f5097600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5090133d20 con 0x7f509013df80 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.959+0000 7f5097600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5090134000 con 0x7f509013e360 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509015f9c0 0x7f509015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509015f9c0 0x7f509015d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46480/0 (socket says 172.21.15.141:46480) 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 -- 172.21.15.141:0/454735256 learned_addr learned my addr 172.21.15.141:0/454735256 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5087c00640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f509013e360 0x7f509014d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5094c00640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f509013df80 0x7f50901f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 -- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f509013e360 msgr2=0x7f509014d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f509013e360 0x7f509014d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 -- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f509013df80 msgr2=0x7f50901f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f509013df80 0x7f50901f2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 -- 172.21.15.141:0/454735256 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50780094d0 con 0x7f509015f9c0 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.960+0000 7f5095600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509015f9c0 0x7f509015d8a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f508800c7c0 tx=0x7f508800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.961+0000 7f508d600640 1 -- 172.21.15.141:0/454735256 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088003da0 con 0x7f509015f9c0 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.961+0000 7f508d600640 1 -- 172.21.15.141:0/454735256 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5088005070 con 0x7f509015f9c0 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.961+0000 7f508d600640 1 -- 172.21.15.141:0/454735256 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088005420 con 0x7f509015f9c0 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.961+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50901342e0 con 0x7f509015f9c0 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.961+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50901502e0 con 0x7f509015f9c0 2024-01-16T15:05:06.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.962+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50901344d0 con 0x7f509015f9c0 2024-01-16T15:05:06.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.963+0000 7f508d600640 1 -- 172.21.15.141:0/454735256 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5088005580 con 0x7f509015f9c0 2024-01-16T15:05:06.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.964+0000 7f508d600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f506806bbc0 0x7f506806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.964+0000 7f508d600640 1 -- 172.21.15.141:0/454735256 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f5088087dd0 con 0x7f509015f9c0 2024-01-16T15:05:06.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.964+0000 7f5094c00640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f506806bbc0 0x7f506806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.966+0000 7f5094c00640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f506806bbc0 0x7f506806e080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f508000b7c0 tx=0x7f5080008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:06.969+0000 7f508d600640 1 -- 172.21.15.141:0/454735256 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5088054600 con 0x7f509015f9c0 2024-01-16T15:05:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.192+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f50900675d0 con 0x7f509015f9c0 2024-01-16T15:05:07.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.736+0000 7f508d600640 1 -- 172.21.15.141:0/454735256 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v42) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5088017070 con 0x7f509015f9c0 2024-01-16T15:05:07.716 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 10 and data pool 11 2024-01-16T15:05:07.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.740+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f506806bbc0 msgr2=0x7f506806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.740+0000 7f5097600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f506806bbc0 0x7f506806e080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f508000b7c0 tx=0x7f5080008040 comp rx=0 tx=0).stop 2024-01-16T15:05:07.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.740+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509015f9c0 msgr2=0x7f509015d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.740+0000 7f5097600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509015f9c0 0x7f509015d8a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f508800c7c0 tx=0x7f508800cc80 comp rx=0 tx=0).stop 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.740+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 shutdown_connections 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.741+0000 7f5097600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f506806bbc0 0x7f506806e080 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.741+0000 7f5097600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f509015f9c0 0x7f509015d8a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.741+0000 7f5097600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f509013e360 0x7f509014d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.741+0000 7f5097600640 1 --2- 172.21.15.141:0/454735256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f509013df80 0x7f50901f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.741+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 >> 172.21.15.141:0/454735256 conn(0x7f509006a800 msgr2=0x7f50900afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.741+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 shutdown_connections 2024-01-16T15:05:07.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.741+0000 7f5097600640 1 -- 172.21.15.141:0/454735256 wait complete. 2024-01-16T15:05:07.746 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:07.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.949+0000 7feeb4c00640 1 Processor -- start 2024-01-16T15:05:07.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.950+0000 7feeb4c00640 1 -- start start 2024-01-16T15:05:07.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.951+0000 7feeb4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb0162430 0x7feeb00b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.951+0000 7feeb4c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7feeb0162810 0x7feeb0071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.951+0000 7feeb4c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb00702e0 con 0x7feeb00b3650 2024-01-16T15:05:07.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.951+0000 7feeb4c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb00705c0 con 0x7feeb0162810 2024-01-16T15:05:07.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.951+0000 7feeb4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00708a0 con 0x7feeb0162430 2024-01-16T15:05:07.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.952+0000 7feeafe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7feeb00b3650 0x7feeb01650c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36726/0 (socket says 172.21.15.141:36726) 2024-01-16T15:05:07.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.952+0000 7feeafe00640 1 -- 172.21.15.141:0/2116961378 learned_addr learned my addr 172.21.15.141:0/2116961378 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.952+0000 7feea6800640 1 --2- 172.21.15.141:0/2116961378 >> v2:172.21.15.188:3300/0 conn(0x7feeb0162810 0x7feeb0071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.952+0000 7feeaf400640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb0162430 0x7feeb00b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.952+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214203275 0 0) 0x7feeb00702e0 con 0x7feeb00b3650 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.952+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee90003520 con 0x7feeb00b3650 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61305935 0 0) 0x7fee90003520 con 0x7feeb00b3650 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb0162430 msgr2=0x7feeb00b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb0162430 0x7feeb00b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 >> v2:172.21.15.188:3300/0 conn(0x7feeb0162810 msgr2=0x7feeb0071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 --2- 172.21.15.141:0/2116961378 >> v2:172.21.15.188:3300/0 conn(0x7feeb0162810 0x7feeb0071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb0070b80 con 0x7feeb00b3650 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7feea80019f0 con 0x7feeb00b3650 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 >> v1:172.21.15.141:6789/0 conn(0x7feeb00b3650 legacy=0x7feeb01650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee90003970 0x7fee90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee900047c0 0x7fee90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb0162430 0x7fee90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00705c0 con 0x7fee90003970 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00708a0 con 0x7fee900047c0 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9000b980 con 0x7feeb0162430 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaf400640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee90003970 0x7fee90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feea6800640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee900047c0 0x7fee90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaf400640 1 -- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb0162430 msgr2=0x7fee90005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaf400640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feeb0162430 0x7fee90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaf400640 1 -- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee900047c0 msgr2=0x7fee90004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:07.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.953+0000 7feeaf400640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee900047c0 0x7fee90004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.954+0000 7feeaf400640 1 -- 172.21.15.141:0/2116961378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee90003520 con 0x7fee90003970 2024-01-16T15:05:07.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.954+0000 7feea6800640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee900047c0 0x7fee90004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.955+0000 7feeaf400640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee90003970 0x7fee90003d90 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7feea0009ac0 tx=0x7fee900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c668d9ab050309 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.955+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea002c6d0 con 0x7fee90003970 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.955+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea002c830 con 0x7fee90003970 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.955+0000 7feeaea00640 1 -- 172.21.15.141:0/2116961378 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea002cbc0 con 0x7fee90003970 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.956+0000 7feeb4c00640 1 -- 172.21.15.141:0/2116961378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee90003970 msgr2=0x7fee90003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.956+0000 7feeb4c00640 1 --2- 172.21.15.141:0/2116961378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee90003970 0x7fee90003d90 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7feea0009ac0 tx=0x7fee900057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.956+0000 7feeafe00640 1 -- 172.21.15.141:0/2116961378 reap_dead start 2024-01-16T15:05:07.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.956+0000 7feeb4c00640 1 -- 172.21.15.141:0/2116961378 shutdown_connections 2024-01-16T15:05:07.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.956+0000 7feeb4c00640 1 -- 172.21.15.141:0/2116961378 >> 172.21.15.141:0/2116961378 conn(0x7feeb006a800 msgr2=0x7feeb006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:07.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.956+0000 7feeb4c00640 1 -- 172.21.15.141:0/2116961378 shutdown_connections 2024-01-16T15:05:07.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.956+0000 7feeb4c00640 1 -- 172.21.15.141:0/2116961378 wait complete. 2024-01-16T15:05:07.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.958+0000 7feeb4c00640 1 Processor -- start 2024-01-16T15:05:07.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.958+0000 7feeb4c00640 1 -- start start 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feeb4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee90003970 0x7feeb0162c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feeb4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb01f7750 0x7feeb01f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feeb4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00af820 0x7feeb00afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feeb4c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb01f8f90 con 0x7feeb01f7750 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feeb4c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00b01e0 con 0x7feeb00af820 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feeb4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb0165830 con 0x7fee90003970 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb01f7750 0x7feeb01f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb01f7750 0x7feeb01f6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46496/0 (socket says 172.21.15.141:46496) 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 -- 172.21.15.141:0/3263438594 learned_addr learned my addr 172.21.15.141:0/3263438594 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:07.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 -- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee90003970 msgr2=0x7feeb0162c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee90003970 0x7feeb0162c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 -- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00af820 msgr2=0x7feeb00afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00af820 0x7feeb00afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.959+0000 7feea6800640 1 -- 172.21.15.141:0/3263438594 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb01f8310 con 0x7feeb01f7750 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.960+0000 7feea6800640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb01f7750 0x7feeb01f6550 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fee980109d0 tx=0x7fee98010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.960+0000 7feea7e00640 1 -- 172.21.15.141:0/3263438594 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9800ec20 con 0x7feeb01f7750 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.960+0000 7feea7e00640 1 -- 172.21.15.141:0/3263438594 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9800ed80 con 0x7feeb01f7750 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.960+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb01f85f0 con 0x7feeb01f7750 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.960+0000 7feea7e00640 1 -- 172.21.15.141:0/3263438594 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9801c620 con 0x7feeb01f7750 2024-01-16T15:05:07.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.960+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb01f8bb0 con 0x7feeb01f7750 2024-01-16T15:05:07.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.961+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb01f9200 con 0x7feeb01f7750 2024-01-16T15:05:07.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.962+0000 7feea7e00640 1 -- 172.21.15.141:0/3263438594 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fee9801c780 con 0x7feeb01f7750 2024-01-16T15:05:07.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.963+0000 7feea7e00640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806bbc0 0x7fee8806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.963+0000 7feea7e00640 1 -- 172.21.15.141:0/3263438594 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee98088a30 con 0x7feeb01f7750 2024-01-16T15:05:07.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.964+0000 7feeaf400640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806bbc0 0x7fee8806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.965+0000 7feeaf400640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806bbc0 0x7fee8806e080 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7feea0036000 tx=0x7feea002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:07.968+0000 7feea7e00640 1 -- 172.21.15.141:0/3263438594 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee98055400 con 0x7feeb01f7750 2024-01-16T15:05:08.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.160+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeb0067450 con 0x7feeb01f7750 2024-01-16T15:05:08.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.163+0000 7feea7e00640 1 -- 172.21.15.141:0/3263438594 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16629 (secure 0 0 0) 0x7fee98058a20 con 0x7feeb01f7750 2024-01-16T15:05:08.143 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:08.143 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":72,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:05:07.194354+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:05.052237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:06.215422+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.165+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806bbc0 msgr2=0x7fee8806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806bbc0 0x7fee8806e080 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7feea0036000 tx=0x7feea002e000 comp rx=0 tx=0).stop 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb01f7750 msgr2=0x7feeb01f6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb01f7750 0x7feeb01f6550 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fee980109d0 tx=0x7fee98010e90 comp rx=0 tx=0).stop 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 shutdown_connections 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fee8806bbc0 0x7fee8806e080 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeb00af820 0x7feeb00afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feeb01f7750 0x7feeb01f6550 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 --2- 172.21.15.141:0/3263438594 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fee90003970 0x7feeb0162c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 >> 172.21.15.141:0/3263438594 conn(0x7feeb006a800 msgr2=0x7feeb0071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 shutdown_connections 2024-01-16T15:05:08.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.166+0000 7feeb4c00640 1 -- 172.21.15.141:0/3263438594 wait complete. 2024-01-16T15:05:08.164 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:08.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.393+0000 7f90f7000640 1 Processor -- start 2024-01-16T15:05:08.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.393+0000 7f90f7000640 1 -- start start 2024-01-16T15:05:08.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90f7000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f90f013df80 0x7f90f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90f7000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f90f00cf9a0 0x7f90f00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90f7000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f006c470 con 0x7f90f013e360 2024-01-16T15:05:08.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90f7000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f006c750 con 0x7f90f013df80 2024-01-16T15:05:08.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90f7000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f006ca30 con 0x7f90f00cf9a0 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90efe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f90f013df80 0x7f90f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90efe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f90f013df80 0x7f90f015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46510/0 (socket says 172.21.15.141:46510) 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.395+0000 7f90efe00640 1 -- 172.21.15.141:0/3246170020 learned_addr learned my addr 172.21.15.141:0/3246170020 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.396+0000 7f90efe00640 1 -- 172.21.15.141:0/3246170020 >> v2:172.21.15.188:3300/0 conn(0x7f90f00cf9a0 msgr2=0x7f90f00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.396+0000 7f90f5000640 1 --2- 172.21.15.141:0/3246170020 >> v2:172.21.15.188:3300/0 conn(0x7f90f00cf9a0 0x7f90f00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.396+0000 7f90efe00640 1 --2- 172.21.15.141:0/3246170020 >> v2:172.21.15.188:3300/0 conn(0x7f90f00cf9a0 0x7f90f00cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.396+0000 7f90efe00640 1 -- 172.21.15.141:0/3246170020 >> v1:172.21.15.188:6789/0 conn(0x7f90f013e360 legacy=0x7f90f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.396+0000 7f90efe00640 1 -- 172.21.15.141:0/3246170020 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f006cd10 con 0x7f90f013df80 2024-01-16T15:05:08.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.396+0000 7f90f5000640 1 --2- 172.21.15.141:0/3246170020 >> v2:172.21.15.188:3300/0 conn(0x7f90f00cf9a0 0x7f90f00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:08.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.397+0000 7f90efe00640 1 --2- 172.21.15.141:0/3246170020 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013df80 0x7f90f015bee0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f90f00ccb10 tx=0x7f90e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ad7687d5cd9723 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.397+0000 7f90eea00640 1 -- 172.21.15.141:0/3246170020 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0031870 con 0x7f90f013df80 2024-01-16T15:05:08.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90eea00640 1 -- 172.21.15.141:0/3246170020 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e00319d0 con 0x7f90f013df80 2024-01-16T15:05:08.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90eea00640 1 -- 172.21.15.141:0/3246170020 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0031d60 con 0x7f90f013df80 2024-01-16T15:05:08.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90f7000640 1 -- 172.21.15.141:0/3246170020 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013df80 msgr2=0x7f90f015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90f7000640 1 --2- 172.21.15.141:0/3246170020 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013df80 0x7f90f015bee0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f90f00ccb10 tx=0x7f90e002b910 comp rx=0 tx=0).stop 2024-01-16T15:05:08.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90f7000640 1 -- 172.21.15.141:0/3246170020 shutdown_connections 2024-01-16T15:05:08.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90f7000640 1 --2- 172.21.15.141:0/3246170020 >> v2:172.21.15.188:3300/0 conn(0x7f90f00cf9a0 0x7f90f00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90f7000640 1 --2- 172.21.15.141:0/3246170020 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013df80 0x7f90f015bee0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90f7000640 1 -- 172.21.15.141:0/3246170020 >> 172.21.15.141:0/3246170020 conn(0x7f90f006a800 msgr2=0x7f90f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.398+0000 7f90f7000640 1 -- 172.21.15.141:0/3246170020 shutdown_connections 2024-01-16T15:05:08.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.399+0000 7f90f7000640 1 -- 172.21.15.141:0/3246170020 wait complete. 2024-01-16T15:05:08.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.400+0000 7f90f7000640 1 Processor -- start 2024-01-16T15:05:08.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.400+0000 7f90f7000640 1 -- start start 2024-01-16T15:05:08.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.401+0000 7f90f7000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90f00cf9a0 0x7f90f00a9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.401+0000 7f90f7000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90f013df80 0x7f90f013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.401+0000 7f90f7000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013e360 0x7f90f0152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f7000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0153710 con 0x7f90f013e360 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f7000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0070410 con 0x7f90f013df80 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f7000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00706f0 con 0x7f90f00cf9a0 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013e360 0x7f90f0152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013e360 0x7f90f0152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46520/0 (socket says 172.21.15.141:46520) 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 -- 172.21.15.141:0/273384111 learned_addr learned my addr 172.21.15.141:0/273384111 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90efe00640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90f00cf9a0 0x7f90f00a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90ef400640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90f013df80 0x7f90f013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 -- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90f00cf9a0 msgr2=0x7f90f00a9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90f00cf9a0 0x7f90f00a9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 -- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90f013df80 msgr2=0x7f90f013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90f013df80 0x7f90f013aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90f5000640 1 -- 172.21.15.141:0/273384111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e00094d0 con 0x7f90f013e360 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.402+0000 7f90ef400640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90f013df80 0x7f90f013aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.403+0000 7f90f5000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013e360 0x7f90f0152e60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f90e800cd80 tx=0x7f90e800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.403+0000 7f90e7e00640 1 -- 172.21.15.141:0/273384111 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800f680 con 0x7f90f013e360 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.403+0000 7f90e7e00640 1 -- 172.21.15.141:0/273384111 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e800f800 con 0x7f90f013e360 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.403+0000 7f90e7e00640 1 -- 172.21.15.141:0/273384111 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800fbb0 con 0x7f90f013e360 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.403+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f00709d0 con 0x7f90f013e360 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.403+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f013af90 con 0x7f90f013e360 2024-01-16T15:05:08.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.404+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f0070ef0 con 0x7f90f013e360 2024-01-16T15:05:08.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.405+0000 7f90e7e00640 1 -- 172.21.15.141:0/273384111 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f90e800b040 con 0x7f90f013e360 2024-01-16T15:05:08.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.406+0000 7f90e7e00640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90c806bbc0 0x7f90c806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.406+0000 7f90e7e00640 1 -- 172.21.15.141:0/273384111 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90e8088e80 con 0x7f90f013e360 2024-01-16T15:05:08.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.407+0000 7f90efe00640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90c806bbc0 0x7f90c806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.408+0000 7f90efe00640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90c806bbc0 0x7f90c806e080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90e0036000 tx=0x7f90e003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.412+0000 7f90e7e00640 1 -- 172.21.15.141:0/273384111 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90f0070ef0 con 0x7f90f013e360 2024-01-16T15:05:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.603+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90f0070ef0 con 0x7f90f013e360 2024-01-16T15:05:08.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.605+0000 7f90e7e00640 1 -- 172.21.15.141:0/273384111 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f90f0070ef0 con 0x7f90f013e360 2024-01-16T15:05:08.585 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:08.586 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":72,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:05:07.194354+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:05.052237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:06.215422+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90c806bbc0 msgr2=0x7f90c806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90c806bbc0 0x7f90c806e080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90e0036000 tx=0x7f90e003c040 comp rx=0 tx=0).stop 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013e360 msgr2=0x7f90f0152e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013e360 0x7f90f0152e60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f90e800cd80 tx=0x7f90e800a650 comp rx=0 tx=0).stop 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 shutdown_connections 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90c806bbc0 0x7f90c806e080 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90f013e360 0x7f90f0152e60 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90f013df80 0x7f90f013aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 --2- 172.21.15.141:0/273384111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90f00cf9a0 0x7f90f00a9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 >> 172.21.15.141:0/273384111 conn(0x7f90f006a800 msgr2=0x7f90f006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 shutdown_connections 2024-01-16T15:05:08.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.608+0000 7f90f7000640 1 -- 172.21.15.141:0/273384111 wait complete. 2024-01-16T15:05:08.610 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:05:08.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.844+0000 7f057e000640 1 Processor -- start 2024-01-16T15:05:08.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.844+0000 7f057e000640 1 -- start start 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f057815d4b0 0x7f05780d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 0x7f05780cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057e000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05780d0320 con 0x7f05780cddf0 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057e000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05780b2730 con 0x7f057815d4b0 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05780d1ec0 con 0x7f05780cf9a0 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057cc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f057815d4b0 0x7f05780d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057cc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f057815d4b0 0x7f05780d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46532/0 (socket says 172.21.15.141:46532) 2024-01-16T15:05:08.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.846+0000 7f057cc00640 1 -- 172.21.15.141:0/3426133469 learned_addr learned my addr 172.21.15.141:0/3426133469 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:08.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.847+0000 7f057cc00640 1 -- 172.21.15.141:0/3426133469 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 msgr2=0x7f05780cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.847+0000 7f057cc00640 1 --2- 172.21.15.141:0/3426133469 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 0x7f05780cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.847+0000 7f057cc00640 1 -- 172.21.15.141:0/3426133469 >> v1:172.21.15.188:6789/0 conn(0x7f05780cddf0 legacy=0x7f05780ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.847+0000 7f057cc00640 1 -- 172.21.15.141:0/3426133469 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780b29d0 con 0x7f057815d4b0 2024-01-16T15:05:08.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.848+0000 7f057cc00640 1 --2- 172.21.15.141:0/3426133469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f057815d4b0 0x7f05780d1550 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f05680098a0 tx=0x7f056802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da8327c4475c491 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.848+0000 7f0577400640 1 -- 172.21.15.141:0/3426133469 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568031640 con 0x7f057815d4b0 2024-01-16T15:05:08.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.848+0000 7f0577400640 1 -- 172.21.15.141:0/3426133469 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05680317a0 con 0x7f057815d4b0 2024-01-16T15:05:08.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.848+0000 7f0577400640 1 -- 172.21.15.141:0/3426133469 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568031b30 con 0x7f057815d4b0 2024-01-16T15:05:08.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 -- 172.21.15.141:0/3426133469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f057815d4b0 msgr2=0x7f05780d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 --2- 172.21.15.141:0/3426133469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f057815d4b0 0x7f05780d1550 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f05680098a0 tx=0x7f056802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 -- 172.21.15.141:0/3426133469 shutdown_connections 2024-01-16T15:05:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 --2- 172.21.15.141:0/3426133469 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 0x7f05780cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 --2- 172.21.15.141:0/3426133469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f057815d4b0 0x7f05780d1550 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 -- 172.21.15.141:0/3426133469 >> 172.21.15.141:0/3426133469 conn(0x7f05780d3000 msgr2=0x7f05780d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 -- 172.21.15.141:0/3426133469 shutdown_connections 2024-01-16T15:05:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.849+0000 7f057e000640 1 -- 172.21.15.141:0/3426133469 wait complete. 2024-01-16T15:05:08.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.851+0000 7f057e000640 1 Processor -- start 2024-01-16T15:05:08.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.851+0000 7f057e000640 1 -- start start 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05780cddf0 0x7f057815bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 0x7f05780cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f057815d4b0 0x7f05780a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057e000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057815c2d0 con 0x7f05780cddf0 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057e000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05780ca0d0 con 0x7f057815d4b0 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05780ca320 con 0x7f05780cf9a0 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05780cddf0 0x7f057815bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05780cddf0 0x7f057815bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46542/0 (socket says 172.21.15.141:46542) 2024-01-16T15:05:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 -- 172.21.15.141:0/1464775639 learned_addr learned my addr 172.21.15.141:0/1464775639 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 -- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 msgr2=0x7f05780cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 0x7f05780cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 -- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f057815d4b0 msgr2=0x7f05780a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f057815d4b0 0x7f05780a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.852+0000 7f057cc00640 1 -- 172.21.15.141:0/1464775639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05680094d0 con 0x7f05780cddf0 2024-01-16T15:05:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.853+0000 7f057cc00640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05780cddf0 0x7f057815bd90 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f05680098a0 tx=0x7f05680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.853+0000 7f0574c00640 1 -- 172.21.15.141:0/1464775639 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056803ac60 con 0x7f05780cddf0 2024-01-16T15:05:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.853+0000 7f0574c00640 1 -- 172.21.15.141:0/1464775639 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056803adc0 con 0x7f05780cddf0 2024-01-16T15:05:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.853+0000 7f0574c00640 1 -- 172.21.15.141:0/1464775639 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568044630 con 0x7f05780cddf0 2024-01-16T15:05:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.853+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057815da10 con 0x7f05780cddf0 2024-01-16T15:05:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.853+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05780ca8e0 con 0x7f05780cddf0 2024-01-16T15:05:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.854+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0578067450 con 0x7f05780cddf0 2024-01-16T15:05:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.855+0000 7f0574c00640 1 -- 172.21.15.141:0/1464775639 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f056802f050 con 0x7f05780cddf0 2024-01-16T15:05:08.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.856+0000 7f0574c00640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f055006bbc0 0x7f055006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.856+0000 7f0574c00640 1 -- 172.21.15.141:0/1464775639 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05680aa450 con 0x7f05780cddf0 2024-01-16T15:05:08.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.857+0000 7f0577e00640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f055006bbc0 0x7f055006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.858+0000 7f0577e00640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f055006bbc0 0x7f055006e080 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f056000ab90 tx=0x7f0560009250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:08.861+0000 7f0574c00640 1 -- 172.21.15.141:0/1464775639 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0568076d10 con 0x7f05780cddf0 2024-01-16T15:05:09.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.088+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f05780c9c90 con 0x7f05780cddf0 2024-01-16T15:05:09.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.766+0000 7f0574c00640 1 -- 172.21.15.141:0/1464775639 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0568031e60 con 0x7f05780cddf0 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f055006bbc0 msgr2=0x7f055006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f055006bbc0 0x7f055006e080 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f056000ab90 tx=0x7f0560009250 comp rx=0 tx=0).stop 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05780cddf0 msgr2=0x7f057815bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05780cddf0 0x7f057815bd90 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f05680098a0 tx=0x7f05680046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 shutdown_connections 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f055006bbc0 0x7f055006e080 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f057815d4b0 0x7f05780a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f05780cf9a0 0x7f05780cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 --2- 172.21.15.141:0/1464775639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05780cddf0 0x7f057815bd90 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 >> 172.21.15.141:0/1464775639 conn(0x7f05780d3000 msgr2=0x7f05780b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 shutdown_connections 2024-01-16T15:05:09.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:09.770+0000 7f057e000640 1 -- 172.21.15.141:0/1464775639 wait complete. 2024-01-16T15:05:09.772 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:05:09.772 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:05:09.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.015+0000 7fb4e8a00640 1 Processor -- start 2024-01-16T15:05:09.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.016+0000 7fb4e8a00640 1 -- start start 2024-01-16T15:05:09.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.017+0000 7fb4e8a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb4e00ae5a0 0x7fb4e015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.017+0000 7fb4e8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.017+0000 7fb4e8a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e0070c40 con 0x7fb4e006d120 2024-01-16T15:05:09.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.017+0000 7fb4e8a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e0071450 con 0x7fb4e00ae5a0 2024-01-16T15:05:09.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.017+0000 7fb4e8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e006c470 con 0x7fb4e00b3ae0 2024-01-16T15:05:09.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.018+0000 7fb4e5600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.018+0000 7fb4e5600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e006cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:50536/0 (socket says 172.21.15.141:50536) 2024-01-16T15:05:09.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.018+0000 7fb4e5600640 1 -- 172.21.15.141:0/734164240 learned_addr learned my addr 172.21.15.141:0/734164240 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:09.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.018+0000 7fb4e6000640 1 --2- 172.21.15.141:0/734164240 >> v2:172.21.15.188:3300/0 conn(0x7fb4e00ae5a0 0x7fb4e015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.019+0000 7fb4e5600640 1 -- 172.21.15.141:0/734164240 >> v2:172.21.15.188:3300/0 conn(0x7fb4e00ae5a0 msgr2=0x7fb4e015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.019+0000 7fb4e5600640 1 --2- 172.21.15.141:0/734164240 >> v2:172.21.15.188:3300/0 conn(0x7fb4e00ae5a0 0x7fb4e015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.019+0000 7fb4e5600640 1 -- 172.21.15.141:0/734164240 >> v1:172.21.15.188:6789/0 conn(0x7fb4e006d120 legacy=0x7fb4e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.019+0000 7fb4e5600640 1 -- 172.21.15.141:0/734164240 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e006c750 con 0x7fb4e00b3ae0 2024-01-16T15:05:09.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.020+0000 7fb4e5600640 1 --2- 172.21.15.141:0/734164240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e006cbe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009870 tx=0x7fb4c802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc467198f89324a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e4c00640 1 -- 172.21.15.141:0/734164240 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8030840 con 0x7fb4e00b3ae0 2024-01-16T15:05:09.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e4c00640 1 -- 172.21.15.141:0/734164240 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c80309a0 con 0x7fb4e00b3ae0 2024-01-16T15:05:09.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e4c00640 1 -- 172.21.15.141:0/734164240 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8030d30 con 0x7fb4e00b3ae0 2024-01-16T15:05:10.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 -- 172.21.15.141:0/734164240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 msgr2=0x7fb4e006cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/734164240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e006cbe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009870 tx=0x7fb4c802b980 comp rx=0 tx=0).stop 2024-01-16T15:05:10.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 -- 172.21.15.141:0/734164240 shutdown_connections 2024-01-16T15:05:10.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/734164240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e006cbe0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/734164240 >> v2:172.21.15.188:3300/0 conn(0x7fb4e00ae5a0 0x7fb4e015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 -- 172.21.15.141:0/734164240 >> 172.21.15.141:0/734164240 conn(0x7fb4e006a800 msgr2=0x7fb4e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:10.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 -- 172.21.15.141:0/734164240 shutdown_connections 2024-01-16T15:05:10.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.021+0000 7fb4e8a00640 1 -- 172.21.15.141:0/734164240 wait complete. 2024-01-16T15:05:10.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.023+0000 7fb4e8a00640 1 Processor -- start 2024-01-16T15:05:10.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.023+0000 7fb4e8a00640 1 -- start start 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4e006d120 0x7fb4e01559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4e00ae5a0 0x7fb4e0156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e01567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e8a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00cbef0 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00cbc10 con 0x7fb4e006d120 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00cde00 con 0x7fb4e00b3ae0 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4e00ae5a0 0x7fb4e0156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4e00ae5a0 0x7fb4e0156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46556/0 (socket says 172.21.15.141:46556) 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.024+0000 7fb4e5600640 1 -- 172.21.15.141:0/1236176998 learned_addr learned my addr 172.21.15.141:0/1236176998 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e6a00640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e01567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e6000640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4e006d120 0x7fb4e01559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e5600640 1 -- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 msgr2=0x7fb4e01567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e5600640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e01567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e5600640 1 -- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4e006d120 msgr2=0x7fb4e01559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e5600640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4e006d120 0x7fb4e01559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e5600640 1 -- 172.21.15.141:0/1236176998 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c80094d0 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.025+0000 7fb4e5600640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4e00ae5a0 0x7fb4e0156290 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009870 tx=0x7fb4c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.026+0000 7fb4de000640 1 -- 172.21.15.141:0/1236176998 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8039a90 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.026+0000 7fb4de000640 1 -- 172.21.15.141:0/1236176998 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c8039c10 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.026+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e00ce0e0 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.026+0000 7fb4de000640 1 -- 172.21.15.141:0/1236176998 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80433f0 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.026+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e00cc540 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.027+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e00cdc60 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.028+0000 7fb4de000640 1 -- 172.21.15.141:0/1236176998 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb4c804c3b0 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.029+0000 7fb4de000640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4b806bbc0 0x7fb4b806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.029+0000 7fb4de000640 1 -- 172.21.15.141:0/1236176998 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4c80aa830 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.029+0000 7fb4e6000640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4b806bbc0 0x7fb4b806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:10.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.030+0000 7fb4e6000640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4b806bbc0 0x7fb4b806e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00b1770 tx=0x7fb4d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:10.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.033+0000 7fb4de000640 1 -- 172.21.15.141:0/1236176998 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4e00cdc60 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.259+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb4e00cdc60 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.776+0000 7fb4de000640 1 -- 172.21.15.141:0/1236176998 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb4e00cdc60 con 0x7fb4e00ae5a0 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.781+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4b806bbc0 msgr2=0x7fb4b806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.781+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4b806bbc0 0x7fb4b806e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00b1770 tx=0x7fb4d0008040 comp rx=0 tx=0).stop 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.781+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4e00ae5a0 msgr2=0x7fb4e0156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.781+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4e00ae5a0 0x7fb4e0156290 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009870 tx=0x7fb4c8004730 comp rx=0 tx=0).stop 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.781+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 shutdown_connections 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.782+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4b806bbc0 0x7fb4b806e080 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.782+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4e00b3ae0 0x7fb4e01567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.782+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4e00ae5a0 0x7fb4e0156290 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.782+0000 7fb4e8a00640 1 --2- 172.21.15.141:0/1236176998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4e006d120 0x7fb4e01559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.782+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 >> 172.21.15.141:0/1236176998 conn(0x7fb4e006a800 msgr2=0x7fb4e00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.782+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 shutdown_connections 2024-01-16T15:05:10.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:10.782+0000 7fb4e8a00640 1 -- 172.21.15.141:0/1236176998 wait complete. 2024-01-16T15:05:10.783 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:05:10.783 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.019+0000 7f4a31400640 1 Processor -- start 2024-01-16T15:05:10.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.019+0000 7f4a31400640 1 -- start start 2024-01-16T15:05:10.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a31400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4a2c0cc500 0x7f4a2c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:10.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a31400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c06d160 0x7f4a2c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a31400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c070e50 con 0x7f4a2c143c40 2024-01-16T15:05:11.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a31400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c0723c0 con 0x7f4a2c0cc500 2024-01-16T15:05:11.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a31400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0cced0 con 0x7f4a2c06d160 2024-01-16T15:05:11.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a2be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4a2c143c40 0x7f4a2c071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36742/0 (socket says 172.21.15.141:36742) 2024-01-16T15:05:11.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a2be00640 1 -- 172.21.15.141:0/3908035300 learned_addr learned my addr 172.21.15.141:0/3908035300 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a2b400640 1 --2- 172.21.15.141:0/3908035300 >> v2:172.21.15.188:3300/0 conn(0x7f4a2c0cc500 0x7f4a2c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.021+0000 7f4a30a00640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c06d160 0x7f4a2c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574915846 0 0) 0x7f4a2c070e50 con 0x7f4a2c143c40 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0c003520 con 0x7f4a2c143c40 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 67011106 0 0) 0x7f4a0c003520 con 0x7f4a2c143c40 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c06d160 msgr2=0x7f4a2c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c06d160 0x7f4a2c06d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 >> v2:172.21.15.188:3300/0 conn(0x7f4a2c0cc500 msgr2=0x7f4a2c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 --2- 172.21.15.141:0/3908035300 >> v2:172.21.15.188:3300/0 conn(0x7f4a2c0cc500 0x7f4a2c0cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c144020 con 0x7f4a2c143c40 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4a1c001690 con 0x7f4a2c143c40 2024-01-16T15:05:11.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 >> v1:172.21.15.141:6789/0 conn(0x7f4a2c143c40 legacy=0x7f4a2c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c003970 0x7f4a0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a0c0047c0 0x7f4a0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c06d160 0x7f4a0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0723c0 con 0x7f4a0c0047c0 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0cced0 con 0x7f4a0c003970 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c00b9b0 con 0x7f4a2c06d160 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.022+0000 7f4a2b400640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a0c0047c0 0x7f4a0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.023+0000 7f4a2be00640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c003970 0x7f4a0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.023+0000 7f4a2b400640 1 -- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c06d160 msgr2=0x7f4a0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.023+0000 7f4a2b400640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c06d160 0x7f4a0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.023+0000 7f4a2b400640 1 -- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c003970 msgr2=0x7f4a0c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.023+0000 7f4a2b400640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c003970 0x7f4a0c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.023+0000 7f4a2b400640 1 -- 172.21.15.141:0/3908035300 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c003520 con 0x7f4a0c0047c0 2024-01-16T15:05:11.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.023+0000 7f4a2be00640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c003970 0x7f4a0c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.024+0000 7f4a2b400640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a0c0047c0 0x7f4a0c004c40 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009ac0 tx=0x7f4a0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabb91da55debb93 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.024+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1402c6d0 con 0x7f4a0c0047c0 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.024+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1402c830 con 0x7f4a0c0047c0 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.024+0000 7f4a2aa00640 1 -- 172.21.15.141:0/3908035300 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1402cbc0 con 0x7f4a0c0047c0 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.024+0000 7f4a31400640 1 -- 172.21.15.141:0/3908035300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a0c0047c0 msgr2=0x7f4a0c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.024+0000 7f4a31400640 1 --2- 172.21.15.141:0/3908035300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a0c0047c0 0x7f4a0c004c40 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009ac0 tx=0x7f4a0c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.025+0000 7f4a30a00640 1 -- 172.21.15.141:0/3908035300 reap_dead start 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.025+0000 7f4a31400640 1 -- 172.21.15.141:0/3908035300 shutdown_connections 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.025+0000 7f4a31400640 1 -- 172.21.15.141:0/3908035300 >> 172.21.15.141:0/3908035300 conn(0x7f4a2c06a800 msgr2=0x7f4a2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.025+0000 7f4a31400640 1 -- 172.21.15.141:0/3908035300 shutdown_connections 2024-01-16T15:05:11.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.025+0000 7f4a31400640 1 -- 172.21.15.141:0/3908035300 wait complete. 2024-01-16T15:05:11.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.026+0000 7f4a31400640 1 Processor -- start 2024-01-16T15:05:11.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.027+0000 7f4a31400640 1 -- start start 2024-01-16T15:05:11.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a31400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c0047c0 0x7f4a2c1f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a31400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c0b27b0 0x7f4a2c0b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a31400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a2c0c3d70 0x7f4a2c0c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a31400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0b3170 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a31400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0c4a80 con 0x7f4a0c0047c0 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a31400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0c4d60 con 0x7f4a2c0b27b0 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a2c0c3d70 0x7f4a2c0c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a2c0c3d70 0x7f4a2c0c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46570/0 (socket says 172.21.15.141:46570) 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 -- 172.21.15.141:0/3162022371 learned_addr learned my addr 172.21.15.141:0/3162022371 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 -- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c0b27b0 msgr2=0x7f4a2c0b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c0b27b0 0x7f4a2c0b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 -- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c0047c0 msgr2=0x7f4a2c1f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c0047c0 0x7f4a2c1f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.028+0000 7f4a30a00640 1 -- 172.21.15.141:0/3162022371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0c5040 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.029+0000 7f4a30a00640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a2c0c3d70 0x7f4a2c0c41f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c144ca0 tx=0x7f4a2400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.029+0000 7f4a23e00640 1 -- 172.21.15.141:0/3162022371 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24010040 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.029+0000 7f4a23e00640 1 -- 172.21.15.141:0/3162022371 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2400bae0 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.029+0000 7f4a23e00640 1 -- 172.21.15.141:0/3162022371 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2400be70 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.029+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c0c5320 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.029+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c0c4730 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.030+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c0c5840 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.031+0000 7f4a23e00640 1 -- 172.21.15.141:0/3162022371 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f4a2401e410 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.032+0000 7f4a23e00640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4a0406bbc0 0x7f4a0406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.033+0000 7f4a23e00640 1 -- 172.21.15.141:0/3162022371 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a24088860 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.033+0000 7f4a2be00640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4a0406bbc0 0x7f4a0406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.034+0000 7f4a2be00640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4a0406bbc0 0x7f4a0406e080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c000f60 tx=0x7f4a1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.037+0000 7f4a23e00640 1 -- 172.21.15.141:0/3162022371 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a2c0c5840 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.266+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a2c0c5840 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.267+0000 7f4a23e00640 1 -- 172.21.15.141:0/3162022371 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f4a2c0c5840 con 0x7f4a2c0c3d70 2024-01-16T15:05:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:11.246 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3894915042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3894915042},{"type":"v1","addr":"172.21.15.188:6835","nonce":3894915042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/3263318622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":3263318622},{"type":"v1","addr":"172.21.15.188:6837","nonce":3263318622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3550237844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3550237844},{"type":"v1","addr":"172.21.15.141:6833","nonce":3550237844}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:05:07.194846+0000","modified":"2024-01-16T15:05:10.261152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/3457901158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3457901158},{"type":"v1","addr":"172.21.15.141:6835","nonce":3457901158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:05:11.246 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 46 2024-01-16T15:05:11.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.269+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4a0406bbc0 msgr2=0x7f4a0406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.269+0000 7f4a31400640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4a0406bbc0 0x7f4a0406e080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c000f60 tx=0x7f4a1c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:11.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.269+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a2c0c3d70 msgr2=0x7f4a2c0c41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.269+0000 7f4a31400640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a2c0c3d70 0x7f4a2c0c41f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c144ca0 tx=0x7f4a2400b480 comp rx=0 tx=0).stop 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 shutdown_connections 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4a0406bbc0 0x7f4a0406e080 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a2c0c3d70 0x7f4a2c0c41f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a2c0b27b0 0x7f4a2c0b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 --2- 172.21.15.141:0/3162022371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a0c0047c0 0x7f4a2c1f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 >> 172.21.15.141:0/3162022371 conn(0x7f4a2c06a800 msgr2=0x7f4a2c071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 shutdown_connections 2024-01-16T15:05:11.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.270+0000 7f4a31400640 1 -- 172.21.15.141:0/3162022371 wait complete. 2024-01-16T15:05:11.268 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.494+0000 7f153f800640 1 Processor -- start 2024-01-16T15:05:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.494+0000 7f153f800640 1 -- start start 2024-01-16T15:05:11.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.495+0000 7f153f800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f15380cf9a0 0x7f1538070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.495+0000 7f153f800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1538070ea0 0x7f15380cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.495+0000 7f153f800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1538071690 con 0x7f153814b2c0 2024-01-16T15:05:11.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.495+0000 7f153f800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153814b6a0 con 0x7f1538070ea0 2024-01-16T15:05:11.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.495+0000 7f153f800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1538071970 con 0x7f15380cf9a0 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f153d800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1538070ea0 0x7f15380cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f153d800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1538070ea0 0x7f15380cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46572/0 (socket says 172.21.15.141:46572) 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f153d800640 1 -- 172.21.15.141:0/3624350321 learned_addr learned my addr 172.21.15.141:0/3624350321 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f152fc00640 1 --2- 172.21.15.141:0/3624350321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f15380cf9a0 0x7f1538070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f153d800640 1 -- 172.21.15.141:0/3624350321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f15380cf9a0 msgr2=0x7f1538070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f153d800640 1 --2- 172.21.15.141:0/3624350321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f15380cf9a0 0x7f1538070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f153d800640 1 -- 172.21.15.141:0/3624350321 >> v1:172.21.15.188:6789/0 conn(0x7f153814b2c0 legacy=0x7f15380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.496+0000 7f153d800640 1 -- 172.21.15.141:0/3624350321 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1538071c50 con 0x7f1538070ea0 2024-01-16T15:05:11.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.497+0000 7f153d800640 1 --2- 172.21.15.141:0/3624350321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1538070ea0 0x7f15380cc160 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1530009e50 tx=0x7f153002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809a1574058720b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.498+0000 7f1537e00640 1 -- 172.21.15.141:0/3624350321 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153002ca70 con 0x7f1538070ea0 2024-01-16T15:05:11.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.498+0000 7f1537e00640 1 -- 172.21.15.141:0/3624350321 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153002cbd0 con 0x7f1538070ea0 2024-01-16T15:05:11.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.498+0000 7f1537e00640 1 -- 172.21.15.141:0/3624350321 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15300343f0 con 0x7f1538070ea0 2024-01-16T15:05:11.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.498+0000 7f153f800640 1 -- 172.21.15.141:0/3624350321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1538070ea0 msgr2=0x7f15380cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.498+0000 7f153f800640 1 --2- 172.21.15.141:0/3624350321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1538070ea0 0x7f15380cc160 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1530009e50 tx=0x7f153002bed0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.499+0000 7f153f800640 1 -- 172.21.15.141:0/3624350321 shutdown_connections 2024-01-16T15:05:11.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.499+0000 7f153f800640 1 --2- 172.21.15.141:0/3624350321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1538070ea0 0x7f15380cc160 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.499+0000 7f153f800640 1 --2- 172.21.15.141:0/3624350321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f15380cf9a0 0x7f1538070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.499+0000 7f153f800640 1 -- 172.21.15.141:0/3624350321 >> 172.21.15.141:0/3624350321 conn(0x7f153806a800 msgr2=0x7f153806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.499+0000 7f153f800640 1 -- 172.21.15.141:0/3624350321 shutdown_connections 2024-01-16T15:05:11.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.499+0000 7f153f800640 1 -- 172.21.15.141:0/3624350321 wait complete. 2024-01-16T15:05:11.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.501+0000 7f153f800640 1 Processor -- start 2024-01-16T15:05:11.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.501+0000 7f153f800640 1 -- start start 2024-01-16T15:05:11.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.502+0000 7f153f800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1538070ea0 0x7f153815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.502+0000 7f153f800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15380cf9a0 0x7f1538164600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.502+0000 7f153f800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f153814b2c0 0x7f153813f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.502+0000 7f153f800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15381490f0 con 0x7f15380cf9a0 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.502+0000 7f153f800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153815b790 con 0x7f1538070ea0 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.502+0000 7f153f800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1538148e10 con 0x7f153814b2c0 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15380cf9a0 0x7f1538164600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15380cf9a0 0x7f1538164600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46588/0 (socket says 172.21.15.141:46588) 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 -- 172.21.15.141:0/2200002115 learned_addr learned my addr 172.21.15.141:0/2200002115 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f153d800640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f153814b2c0 0x7f153813f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f153ce00640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1538070ea0 0x7f153815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 -- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f153814b2c0 msgr2=0x7f153813f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f153814b2c0 0x7f153813f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 -- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1538070ea0 msgr2=0x7f153815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1538070ea0 0x7f153815b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 -- 172.21.15.141:0/2200002115 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1530009b30 con 0x7f15380cf9a0 2024-01-16T15:05:11.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.503+0000 7f152fc00640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15380cf9a0 0x7f1538164600 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f152000d800 tx=0x7f152000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.504+0000 7f1535600640 1 -- 172.21.15.141:0/2200002115 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152000b6c0 con 0x7f15380cf9a0 2024-01-16T15:05:11.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.504+0000 7f1535600640 1 -- 172.21.15.141:0/2200002115 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152000b820 con 0x7f15380cf9a0 2024-01-16T15:05:11.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.504+0000 7f1535600640 1 -- 172.21.15.141:0/2200002115 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152000bbb0 con 0x7f15380cf9a0 2024-01-16T15:05:11.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.504+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1538161e00 con 0x7f15380cf9a0 2024-01-16T15:05:11.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.504+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15381623c0 con 0x7f15380cf9a0 2024-01-16T15:05:11.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.505+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15380675d0 con 0x7f15380cf9a0 2024-01-16T15:05:11.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.506+0000 7f1535600640 1 -- 172.21.15.141:0/2200002115 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f152000bd10 con 0x7f15380cf9a0 2024-01-16T15:05:11.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.507+0000 7f1535600640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f151006bbc0 0x7f151006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.508+0000 7f1535600640 1 -- 172.21.15.141:0/2200002115 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1520088c90 con 0x7f15380cf9a0 2024-01-16T15:05:11.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.508+0000 7f153ce00640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f151006bbc0 0x7f151006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.509+0000 7f153ce00640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f151006bbc0 0x7f151006e080 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1528006fd0 tx=0x7f1528008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.512+0000 7f1535600640 1 -- 172.21.15.141:0/2200002115 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1520055360 con 0x7f15380cf9a0 2024-01-16T15:05:11.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.703+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1538070c30 con 0x7f15380cf9a0 2024-01-16T15:05:11.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.706+0000 7f1535600640 1 -- 172.21.15.141:0/2200002115 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16616 (secure 0 0 0) 0x7f1520058980 con 0x7f15380cf9a0 2024-01-16T15:05:11.685 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:11.685 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":73,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:05:08.613132+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:05.052237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:06.215422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f151006bbc0 msgr2=0x7f151006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f151006bbc0 0x7f151006e080 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1528006fd0 tx=0x7f1528008040 comp rx=0 tx=0).stop 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15380cf9a0 msgr2=0x7f1538164600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15380cf9a0 0x7f1538164600 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f152000d800 tx=0x7f152000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 shutdown_connections 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f151006bbc0 0x7f151006e080 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f153814b2c0 0x7f153813f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15380cf9a0 0x7f1538164600 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 --2- 172.21.15.141:0/2200002115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1538070ea0 0x7f153815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 >> 172.21.15.141:0/2200002115 conn(0x7f153806a800 msgr2=0x7f153806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 shutdown_connections 2024-01-16T15:05:11.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.709+0000 7f153f800640 1 -- 172.21.15.141:0/2200002115 wait complete. 2024-01-16T15:05:11.705 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:05:11.705 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:05:11.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.937+0000 7f1a68400640 1 Processor -- start 2024-01-16T15:05:11.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.937+0000 7f1a68400640 1 -- start start 2024-01-16T15:05:11.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a68400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1a60143ae0 0x7f1a600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a68400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a6015e3b0 0x7f1a600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a68400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a600cfc30 con 0x7f1a600cf7d0 2024-01-16T15:05:11.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a68400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a600b1ee0 con 0x7f1a60143ae0 2024-01-16T15:05:11.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a68400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600b2780 con 0x7f1a6015e3b0 2024-01-16T15:05:11.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1a60143ae0 0x7f1a600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1a60143ae0 0x7f1a600cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46592/0 (socket says 172.21.15.141:46592) 2024-01-16T15:05:11.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65a00640 1 -- 172.21.15.141:0/3033668156 learned_addr learned my addr 172.21.15.141:0/3033668156 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:11.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65000640 1 --1- 172.21.15.141:0/3033668156 >> v1:172.21.15.141:6789/0 conn(0x7f1a600cf7d0 0x7f1a6015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36754/0 (socket says 172.21.15.141:36754) 2024-01-16T15:05:11.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65a00640 1 -- 172.21.15.141:0/3033668156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a6015e3b0 msgr2=0x7f1a600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65a00640 1 --2- 172.21.15.141:0/3033668156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a6015e3b0 0x7f1a600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65a00640 1 -- 172.21.15.141:0/3033668156 >> v1:172.21.15.141:6789/0 conn(0x7f1a600cf7d0 legacy=0x7f1a6015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.939+0000 7f1a65a00640 1 -- 172.21.15.141:0/3033668156 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6015e990 con 0x7f1a60143ae0 2024-01-16T15:05:11.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.940+0000 7f1a65a00640 1 --2- 172.21.15.141:0/3033668156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600cf290 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009e50 tx=0x7f1a5002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e89aeb248333eab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a5fe00640 1 -- 172.21.15.141:0/3033668156 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5002cb00 con 0x7f1a60143ae0 2024-01-16T15:05:11.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a5fe00640 1 -- 172.21.15.141:0/3033668156 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5002cc60 con 0x7f1a60143ae0 2024-01-16T15:05:11.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a5fe00640 1 -- 172.21.15.141:0/3033668156 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500343f0 con 0x7f1a60143ae0 2024-01-16T15:05:11.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a68400640 1 -- 172.21.15.141:0/3033668156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 msgr2=0x7f1a600cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a68400640 1 --2- 172.21.15.141:0/3033668156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600cf290 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009e50 tx=0x7f1a5002bf60 comp rx=0 tx=0).stop 2024-01-16T15:05:11.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a68400640 1 -- 172.21.15.141:0/3033668156 shutdown_connections 2024-01-16T15:05:11.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a68400640 1 --2- 172.21.15.141:0/3033668156 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a6015e3b0 0x7f1a600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a68400640 1 --2- 172.21.15.141:0/3033668156 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600cf290 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.941+0000 7f1a68400640 1 -- 172.21.15.141:0/3033668156 >> 172.21.15.141:0/3033668156 conn(0x7f1a6006a800 msgr2=0x7f1a6006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.942+0000 7f1a68400640 1 -- 172.21.15.141:0/3033668156 shutdown_connections 2024-01-16T15:05:11.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.942+0000 7f1a68400640 1 -- 172.21.15.141:0/3033668156 wait complete. 2024-01-16T15:05:11.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.943+0000 7f1a68400640 1 Processor -- start 2024-01-16T15:05:11.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.943+0000 7f1a68400640 1 -- start start 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.944+0000 7f1a68400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a600cf7d0 0x7f1a60159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.944+0000 7f1a68400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.944+0000 7f1a68400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a6015e3b0 0x7f1a600adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.944+0000 7f1a68400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600ad720 con 0x7f1a60143ae0 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.944+0000 7f1a68400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600acaa0 con 0x7f1a6015e3b0 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.944+0000 7f1a68400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600acd80 con 0x7f1a600cf7d0 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46598/0 (socket says 172.21.15.141:46598) 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 -- 172.21.15.141:0/291056574 learned_addr learned my addr 172.21.15.141:0/291056574 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65a00640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a600cf7d0 0x7f1a60159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a66400640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a6015e3b0 0x7f1a600adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 -- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a600cf7d0 msgr2=0x7f1a60159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a600cf7d0 0x7f1a60159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 -- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a6015e3b0 msgr2=0x7f1a600adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a6015e3b0 0x7f1a600adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.945+0000 7f1a65000640 1 -- 172.21.15.141:0/291056574 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a50009b30 con 0x7f1a60143ae0 2024-01-16T15:05:11.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.946+0000 7f1a65000640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600c3f30 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800c4d0 tx=0x7f1a4800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.946+0000 7f1a5d600640 1 -- 172.21.15.141:0/291056574 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48007480 con 0x7f1a60143ae0 2024-01-16T15:05:11.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.946+0000 7f1a5d600640 1 -- 172.21.15.141:0/291056574 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a480075e0 con 0x7f1a60143ae0 2024-01-16T15:05:11.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.946+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a600ad060 con 0x7f1a60143ae0 2024-01-16T15:05:11.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.946+0000 7f1a5d600640 1 -- 172.21.15.141:0/291056574 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a480049e0 con 0x7f1a60143ae0 2024-01-16T15:05:11.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.946+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a60159ed0 con 0x7f1a60143ae0 2024-01-16T15:05:11.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.947+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a600ad860 con 0x7f1a60143ae0 2024-01-16T15:05:11.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.948+0000 7f1a5d600640 1 -- 172.21.15.141:0/291056574 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1a48004b40 con 0x7f1a60143ae0 2024-01-16T15:05:11.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.949+0000 7f1a5d600640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a3806bbc0 0x7f1a3806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.949+0000 7f1a5d600640 1 -- 172.21.15.141:0/291056574 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a480879c0 con 0x7f1a60143ae0 2024-01-16T15:05:11.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.950+0000 7f1a65a00640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a3806bbc0 0x7f1a3806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.951+0000 7f1a65a00640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a3806bbc0 0x7f1a3806e080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1a50035000 tx=0x7f1a50030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:11.954+0000 7f1a5d600640 1 -- 172.21.15.141:0/291056574 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a600ad860 con 0x7f1a60143ae0 2024-01-16T15:05:12.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.139+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1a600b03b0 con 0x7f1a3806bbc0 2024-01-16T15:05:12.127 INFO:tasks.ceph.mds.c.smithi141.stderr:2024-01-16T15:05:12.149+0000 7fd73b400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:05:12.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.167+0000 7f1a5d600640 1 -- 172.21.15.141:0/291056574 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287043 (secure 0 0 0) 0x7f1a600b03b0 con 0x7f1a3806bbc0 2024-01-16T15:05:12.149 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:12.151 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:05:12.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a3806bbc0 msgr2=0x7f1a3806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a3806bbc0 0x7f1a3806e080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1a50035000 tx=0x7f1a50030000 comp rx=0 tx=0).stop 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 msgr2=0x7f1a600c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600c3f30 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800c4d0 tx=0x7f1a4800c990 comp rx=0 tx=0).stop 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 shutdown_connections 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1a3806bbc0 0x7f1a3806e080 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a6015e3b0 0x7f1a600adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1a60143ae0 0x7f1a600c3f30 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 --2- 172.21.15.141:0/291056574 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1a600cf7d0 0x7f1a60159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 >> 172.21.15.141:0/291056574 conn(0x7f1a6006a800 msgr2=0x7f1a6015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 shutdown_connections 2024-01-16T15:05:12.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.172+0000 7f1a68400640 1 -- 172.21.15.141:0/291056574 wait complete. 2024-01-16T15:05:12.170 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-01-16T15:05:10.656274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":406140,"kb_used_data":8136,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754568580,"statfs":{"total":773094113280,"available":772678225920,"internally_reserved":0,"allocated":8331264,"data_stored":4176276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":407424562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.934310"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660742+0000","last_change":"2024-01-16T15:05:07.649890+0000","last_active":"2024-01-16T15:05:08.660742+0000","last_peered":"2024-01-16T15:05:08.660742+0000","last_clean":"2024-01-16T15:05:08.660742+0000","last_became_active":"2024-01-16T15:05:07.649259+0000","last_became_peered":"2024-01-16T15:05:07.649259+0000","last_unstale":"2024-01-16T15:05:08.660742+0000","last_undegraded":"2024-01-16T15:05:08.660742+0000","last_fullsized":"2024-01-16T15:05:08.660742+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:55.884731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.663008+0000","last_change":"2024-01-16T15:05:07.654329+0000","last_active":"2024-01-16T15:05:08.663008+0000","last_peered":"2024-01-16T15:05:08.663008+0000","last_clean":"2024-01-16T15:05:08.663008+0000","last_became_active":"2024-01-16T15:05:07.653331+0000","last_became_peered":"2024-01-16T15:05:07.653331+0000","last_unstale":"2024-01-16T15:05:08.663008+0000","last_undegraded":"2024-01-16T15:05:08.663008+0000","last_fullsized":"2024-01-16T15:05:08.663008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:21.803688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657894+0000","last_change":"2024-01-16T15:05:07.643528+0000","last_active":"2024-01-16T15:05:08.657894+0000","last_peered":"2024-01-16T15:05:08.657894+0000","last_clean":"2024-01-16T15:05:08.657894+0000","last_became_active":"2024-01-16T15:05:07.643139+0000","last_became_peered":"2024-01-16T15:05:07.643139+0000","last_unstale":"2024-01-16T15:05:08.657894+0000","last_undegraded":"2024-01-16T15:05:08.657894+0000","last_fullsized":"2024-01-16T15:05:08.657894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:36.347300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657148+0000","last_change":"2024-01-16T15:05:07.654528+0000","last_active":"2024-01-16T15:05:08.657148+0000","last_peered":"2024-01-16T15:05:08.657148+0000","last_clean":"2024-01-16T15:05:08.657148+0000","last_became_active":"2024-01-16T15:05:07.654179+0000","last_became_peered":"2024-01-16T15:05:07.654179+0000","last_unstale":"2024-01-16T15:05:08.657148+0000","last_undegraded":"2024-01-16T15:05:08.657148+0000","last_fullsized":"2024-01-16T15:05:08.657148+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:56.887399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618911+0000","last_change":"2024-01-16T15:05:07.641683+0000","last_active":"2024-01-16T15:05:08.618911+0000","last_peered":"2024-01-16T15:05:08.618911+0000","last_clean":"2024-01-16T15:05:08.618911+0000","last_became_active":"2024-01-16T15:05:07.641337+0000","last_became_peered":"2024-01-16T15:05:07.641337+0000","last_unstale":"2024-01-16T15:05:08.618911+0000","last_undegraded":"2024-01-16T15:05:08.618911+0000","last_fullsized":"2024-01-16T15:05:08.618911+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:16:27.179259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661509+0000","last_change":"2024-01-16T15:05:07.652652+0000","last_active":"2024-01-16T15:05:08.661509+0000","last_peered":"2024-01-16T15:05:08.661509+0000","last_clean":"2024-01-16T15:05:08.661509+0000","last_became_active":"2024-01-16T15:05:07.652064+0000","last_became_peered":"2024-01-16T15:05:07.652064+0000","last_unstale":"2024-01-16T15:05:08.661509+0000","last_undegraded":"2024-01-16T15:05:08.661509+0000","last_fullsized":"2024-01-16T15:05:08.661509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:24.430819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624644+0000","last_change":"2024-01-16T15:05:07.648445+0000","last_active":"2024-01-16T15:05:08.624644+0000","last_peered":"2024-01-16T15:05:08.624644+0000","last_clean":"2024-01-16T15:05:08.624644+0000","last_became_active":"2024-01-16T15:05:07.648130+0000","last_became_peered":"2024-01-16T15:05:07.648130+0000","last_unstale":"2024-01-16T15:05:08.624644+0000","last_undegraded":"2024-01-16T15:05:08.624644+0000","last_fullsized":"2024-01-16T15:05:08.624644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:58.669334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661092+0000","last_change":"2024-01-16T15:05:07.650119+0000","last_active":"2024-01-16T15:05:08.661092+0000","last_peered":"2024-01-16T15:05:08.661092+0000","last_clean":"2024-01-16T15:05:08.661092+0000","last_became_active":"2024-01-16T15:05:07.648004+0000","last_became_peered":"2024-01-16T15:05:07.648004+0000","last_unstale":"2024-01-16T15:05:08.661092+0000","last_undegraded":"2024-01-16T15:05:08.661092+0000","last_fullsized":"2024-01-16T15:05:08.661092+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:59:58.825941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619455+0000","last_change":"2024-01-16T15:05:07.647170+0000","last_active":"2024-01-16T15:05:08.619455+0000","last_peered":"2024-01-16T15:05:08.619455+0000","last_clean":"2024-01-16T15:05:08.619455+0000","last_became_active":"2024-01-16T15:05:07.646804+0000","last_became_peered":"2024-01-16T15:05:07.646804+0000","last_unstale":"2024-01-16T15:05:08.619455+0000","last_undegraded":"2024-01-16T15:05:08.619455+0000","last_fullsized":"2024-01-16T15:05:08.619455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:33.438472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624235+0000","last_change":"2024-01-16T15:05:07.642257+0000","last_active":"2024-01-16T15:05:08.624235+0000","last_peered":"2024-01-16T15:05:08.624235+0000","last_clean":"2024-01-16T15:05:08.624235+0000","last_became_active":"2024-01-16T15:05:07.641661+0000","last_became_peered":"2024-01-16T15:05:07.641661+0000","last_unstale":"2024-01-16T15:05:08.624235+0000","last_undegraded":"2024-01-16T15:05:08.624235+0000","last_fullsized":"2024-01-16T15:05:08.624235+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:35.167922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660057+0000","last_change":"2024-01-16T15:05:07.645773+0000","last_active":"2024-01-16T15:05:08.660057+0000","last_peered":"2024-01-16T15:05:08.660057+0000","last_clean":"2024-01-16T15:05:08.660057+0000","last_became_active":"2024-01-16T15:05:07.645497+0000","last_became_peered":"2024-01-16T15:05:07.645497+0000","last_unstale":"2024-01-16T15:05:08.660057+0000","last_undegraded":"2024-01-16T15:05:08.660057+0000","last_fullsized":"2024-01-16T15:05:08.660057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:44.091571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657302+0000","last_change":"2024-01-16T15:05:07.651944+0000","last_active":"2024-01-16T15:05:08.657302+0000","last_peered":"2024-01-16T15:05:08.657302+0000","last_clean":"2024-01-16T15:05:08.657302+0000","last_became_active":"2024-01-16T15:05:07.648623+0000","last_became_peered":"2024-01-16T15:05:07.648623+0000","last_unstale":"2024-01-16T15:05:08.657302+0000","last_undegraded":"2024-01-16T15:05:08.657302+0000","last_fullsized":"2024-01-16T15:05:08.657302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:09.921566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658519+0000","last_change":"2024-01-16T15:05:07.648626+0000","last_active":"2024-01-16T15:05:08.658519+0000","last_peered":"2024-01-16T15:05:08.658519+0000","last_clean":"2024-01-16T15:05:08.658519+0000","last_became_active":"2024-01-16T15:05:07.648309+0000","last_became_peered":"2024-01-16T15:05:07.648309+0000","last_unstale":"2024-01-16T15:05:08.658519+0000","last_undegraded":"2024-01-16T15:05:08.658519+0000","last_fullsized":"2024-01-16T15:05:08.658519+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:57.810277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619336+0000","last_change":"2024-01-16T15:05:07.643349+0000","last_active":"2024-01-16T15:05:08.619336+0000","last_peered":"2024-01-16T15:05:08.619336+0000","last_clean":"2024-01-16T15:05:08.619336+0000","last_became_active":"2024-01-16T15:05:07.642551+0000","last_became_peered":"2024-01-16T15:05:07.642551+0000","last_unstale":"2024-01-16T15:05:08.619336+0000","last_undegraded":"2024-01-16T15:05:08.619336+0000","last_fullsized":"2024-01-16T15:05:08.619336+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:10.042586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658073+0000","last_change":"2024-01-16T15:05:07.638815+0000","last_active":"2024-01-16T15:05:08.658073+0000","last_peered":"2024-01-16T15:05:08.658073+0000","last_clean":"2024-01-16T15:05:08.658073+0000","last_became_active":"2024-01-16T15:05:07.638376+0000","last_became_peered":"2024-01-16T15:05:07.638376+0000","last_unstale":"2024-01-16T15:05:08.658073+0000","last_undegraded":"2024-01-16T15:05:08.658073+0000","last_fullsized":"2024-01-16T15:05:08.658073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:45.444669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.623213+0000","last_change":"2024-01-16T15:05:07.652910+0000","last_active":"2024-01-16T15:05:08.623213+0000","last_peered":"2024-01-16T15:05:08.623213+0000","last_clean":"2024-01-16T15:05:08.623213+0000","last_became_active":"2024-01-16T15:05:07.652587+0000","last_became_peered":"2024-01-16T15:05:07.652587+0000","last_unstale":"2024-01-16T15:05:08.623213+0000","last_undegraded":"2024-01-16T15:05:08.623213+0000","last_fullsized":"2024-01-16T15:05:08.623213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:44.196268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619589+0000","last_change":"2024-01-16T15:05:07.641174+0000","last_active":"2024-01-16T15:05:08.619589+0000","last_peered":"2024-01-16T15:05:08.619589+0000","last_clean":"2024-01-16T15:05:08.619589+0000","last_became_active":"2024-01-16T15:05:07.640875+0000","last_became_peered":"2024-01-16T15:05:07.640875+0000","last_unstale":"2024-01-16T15:05:08.619589+0000","last_undegraded":"2024-01-16T15:05:08.619589+0000","last_fullsized":"2024-01-16T15:05:08.619589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:35.988314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658490+0000","last_change":"2024-01-16T15:05:07.653752+0000","last_active":"2024-01-16T15:05:08.658490+0000","last_peered":"2024-01-16T15:05:08.658490+0000","last_clean":"2024-01-16T15:05:08.658490+0000","last_became_active":"2024-01-16T15:05:07.653429+0000","last_became_peered":"2024-01-16T15:05:07.653429+0000","last_unstale":"2024-01-16T15:05:08.658490+0000","last_undegraded":"2024-01-16T15:05:08.658490+0000","last_fullsized":"2024-01-16T15:05:08.658490+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:31.816016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662230+0000","last_change":"2024-01-16T15:05:07.651677+0000","last_active":"2024-01-16T15:05:08.662230+0000","last_peered":"2024-01-16T15:05:08.662230+0000","last_clean":"2024-01-16T15:05:08.662230+0000","last_became_active":"2024-01-16T15:05:07.650806+0000","last_became_peered":"2024-01-16T15:05:07.650806+0000","last_unstale":"2024-01-16T15:05:08.662230+0000","last_undegraded":"2024-01-16T15:05:08.662230+0000","last_fullsized":"2024-01-16T15:05:08.662230+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:49.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658746+0000","last_change":"2024-01-16T15:05:07.640768+0000","last_active":"2024-01-16T15:05:08.658746+0000","last_peered":"2024-01-16T15:05:08.658746+0000","last_clean":"2024-01-16T15:05:08.658746+0000","last_became_active":"2024-01-16T15:05:07.640370+0000","last_became_peered":"2024-01-16T15:05:07.640370+0000","last_unstale":"2024-01-16T15:05:08.658746+0000","last_undegraded":"2024-01-16T15:05:08.658746+0000","last_fullsized":"2024-01-16T15:05:08.658746+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:28.543197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625957+0000","last_change":"2024-01-16T15:05:07.638852+0000","last_active":"2024-01-16T15:05:08.625957+0000","last_peered":"2024-01-16T15:05:08.625957+0000","last_clean":"2024-01-16T15:05:08.625957+0000","last_became_active":"2024-01-16T15:05:07.638503+0000","last_became_peered":"2024-01-16T15:05:07.638503+0000","last_unstale":"2024-01-16T15:05:08.625957+0000","last_undegraded":"2024-01-16T15:05:08.625957+0000","last_fullsized":"2024-01-16T15:05:08.625957+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:43.049772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624936+0000","last_change":"2024-01-16T15:05:07.641347+0000","last_active":"2024-01-16T15:05:08.624936+0000","last_peered":"2024-01-16T15:05:08.624936+0000","last_clean":"2024-01-16T15:05:08.624936+0000","last_became_active":"2024-01-16T15:05:07.641073+0000","last_became_peered":"2024-01-16T15:05:07.641073+0000","last_unstale":"2024-01-16T15:05:08.624936+0000","last_undegraded":"2024-01-16T15:05:08.624936+0000","last_fullsized":"2024-01-16T15:05:08.624936+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:56.704683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659618+0000","last_change":"2024-01-16T15:05:07.650341+0000","last_active":"2024-01-16T15:05:08.659618+0000","last_peered":"2024-01-16T15:05:08.659618+0000","last_clean":"2024-01-16T15:05:08.659618+0000","last_became_active":"2024-01-16T15:05:07.648155+0000","last_became_peered":"2024-01-16T15:05:07.648155+0000","last_unstale":"2024-01-16T15:05:08.659618+0000","last_undegraded":"2024-01-16T15:05:08.659618+0000","last_fullsized":"2024-01-16T15:05:08.659618+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:29.064858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659176+0000","last_change":"2024-01-16T15:05:07.655789+0000","last_active":"2024-01-16T15:05:08.659176+0000","last_peered":"2024-01-16T15:05:08.659176+0000","last_clean":"2024-01-16T15:05:08.659176+0000","last_became_active":"2024-01-16T15:05:07.655398+0000","last_became_peered":"2024-01-16T15:05:07.655398+0000","last_unstale":"2024-01-16T15:05:08.659176+0000","last_undegraded":"2024-01-16T15:05:08.659176+0000","last_fullsized":"2024-01-16T15:05:08.659176+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:55.916665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626638+0000","last_change":"2024-01-16T15:05:07.645589+0000","last_active":"2024-01-16T15:05:08.626638+0000","last_peered":"2024-01-16T15:05:08.626638+0000","last_clean":"2024-01-16T15:05:08.626638+0000","last_became_active":"2024-01-16T15:05:07.645195+0000","last_became_peered":"2024-01-16T15:05:07.645195+0000","last_unstale":"2024-01-16T15:05:08.626638+0000","last_undegraded":"2024-01-16T15:05:08.626638+0000","last_fullsized":"2024-01-16T15:05:08.626638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:30.059276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626139+0000","last_change":"2024-01-16T15:05:07.653869+0000","last_active":"2024-01-16T15:05:08.626139+0000","last_peered":"2024-01-16T15:05:08.626139+0000","last_clean":"2024-01-16T15:05:08.626139+0000","last_became_active":"2024-01-16T15:05:07.652770+0000","last_became_peered":"2024-01-16T15:05:07.652770+0000","last_unstale":"2024-01-16T15:05:08.626139+0000","last_undegraded":"2024-01-16T15:05:08.626139+0000","last_fullsized":"2024-01-16T15:05:08.626139+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:39.011373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657896+0000","last_change":"2024-01-16T15:05:07.646050+0000","last_active":"2024-01-16T15:05:08.657896+0000","last_peered":"2024-01-16T15:05:08.657896+0000","last_clean":"2024-01-16T15:05:08.657896+0000","last_became_active":"2024-01-16T15:05:07.645793+0000","last_became_peered":"2024-01-16T15:05:07.645793+0000","last_unstale":"2024-01-16T15:05:08.657896+0000","last_undegraded":"2024-01-16T15:05:08.657896+0000","last_fullsized":"2024-01-16T15:05:08.657896+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:59.114071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624987+0000","last_change":"2024-01-16T15:05:07.649348+0000","last_active":"2024-01-16T15:05:08.624987+0000","last_peered":"2024-01-16T15:05:08.624987+0000","last_clean":"2024-01-16T15:05:08.624987+0000","last_became_active":"2024-01-16T15:05:07.647907+0000","last_became_peered":"2024-01-16T15:05:07.647907+0000","last_unstale":"2024-01-16T15:05:08.624987+0000","last_undegraded":"2024-01-16T15:05:08.624987+0000","last_fullsized":"2024-01-16T15:05:08.624987+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:32.652965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657399+0000","last_change":"2024-01-16T15:05:07.645281+0000","last_active":"2024-01-16T15:05:08.657399+0000","last_peered":"2024-01-16T15:05:08.657399+0000","last_clean":"2024-01-16T15:05:08.657399+0000","last_became_active":"2024-01-16T15:05:07.644943+0000","last_became_peered":"2024-01-16T15:05:07.644943+0000","last_unstale":"2024-01-16T15:05:08.657399+0000","last_undegraded":"2024-01-16T15:05:08.657399+0000","last_fullsized":"2024-01-16T15:05:08.657399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:54.884845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624445+0000","last_change":"2024-01-16T15:05:07.653231+0000","last_active":"2024-01-16T15:05:08.624445+0000","last_peered":"2024-01-16T15:05:08.624445+0000","last_clean":"2024-01-16T15:05:08.624445+0000","last_became_active":"2024-01-16T15:05:07.652905+0000","last_became_peered":"2024-01-16T15:05:07.652905+0000","last_unstale":"2024-01-16T15:05:08.624445+0000","last_undegraded":"2024-01-16T15:05:08.624445+0000","last_fullsized":"2024-01-16T15:05:08.624445+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:53.194876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625027+0000","last_change":"2024-01-16T15:05:07.640337+0000","last_active":"2024-01-16T15:05:08.625027+0000","last_peered":"2024-01-16T15:05:08.625027+0000","last_clean":"2024-01-16T15:05:08.625027+0000","last_became_active":"2024-01-16T15:05:07.640001+0000","last_became_peered":"2024-01-16T15:05:07.640001+0000","last_unstale":"2024-01-16T15:05:08.625027+0000","last_undegraded":"2024-01-16T15:05:08.625027+0000","last_fullsized":"2024-01-16T15:05:08.625027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:17.307990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661293+0000","last_change":"2024-01-16T15:05:07.648931+0000","last_active":"2024-01-16T15:05:08.661293+0000","last_peered":"2024-01-16T15:05:08.661293+0000","last_clean":"2024-01-16T15:05:08.661293+0000","last_became_active":"2024-01-16T15:05:07.648676+0000","last_became_peered":"2024-01-16T15:05:07.648676+0000","last_unstale":"2024-01-16T15:05:08.661293+0000","last_undegraded":"2024-01-16T15:05:08.661293+0000","last_fullsized":"2024-01-16T15:05:08.661293+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:19.807808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660378+0000","last_change":"2024-01-16T15:05:07.655655+0000","last_active":"2024-01-16T15:05:08.660378+0000","last_peered":"2024-01-16T15:05:08.660378+0000","last_clean":"2024-01-16T15:05:08.660378+0000","last_became_active":"2024-01-16T15:05:07.655308+0000","last_became_peered":"2024-01-16T15:05:07.655308+0000","last_unstale":"2024-01-16T15:05:08.660378+0000","last_undegraded":"2024-01-16T15:05:08.660378+0000","last_fullsized":"2024-01-16T15:05:08.660378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:28.628641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"72'4","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661477+0000","last_change":"2024-01-16T15:05:06.760194+0000","last_active":"2024-01-16T15:05:08.661477+0000","last_peered":"2024-01-16T15:05:08.661477+0000","last_clean":"2024-01-16T15:05:08.661477+0000","last_became_active":"2024-01-16T15:05:06.756007+0000","last_became_peered":"2024-01-16T15:05:06.756007+0000","last_unstale":"2024-01-16T15:05:08.661477+0000","last_undegraded":"2024-01-16T15:05:08.661477+0000","last_fullsized":"2024-01-16T15:05:08.661477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:16.292371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619626+0000","last_change":"2024-01-16T15:05:07.649592+0000","last_active":"2024-01-16T15:05:08.619626+0000","last_peered":"2024-01-16T15:05:08.619626+0000","last_clean":"2024-01-16T15:05:08.619626+0000","last_became_active":"2024-01-16T15:05:07.647227+0000","last_became_peered":"2024-01-16T15:05:07.647227+0000","last_unstale":"2024-01-16T15:05:08.619626+0000","last_undegraded":"2024-01-16T15:05:08.619626+0000","last_fullsized":"2024-01-16T15:05:08.619626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:02.272988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659171+0000","last_change":"2024-01-16T15:05:06.757917+0000","last_active":"2024-01-16T15:05:08.659171+0000","last_peered":"2024-01-16T15:05:08.659171+0000","last_clean":"2024-01-16T15:05:08.659171+0000","last_became_active":"2024-01-16T15:05:06.756979+0000","last_became_peered":"2024-01-16T15:05:06.756979+0000","last_unstale":"2024-01-16T15:05:08.659171+0000","last_undegraded":"2024-01-16T15:05:08.659171+0000","last_fullsized":"2024-01-16T15:05:08.659171+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:14.780391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662377+0000","last_change":"2024-01-16T15:05:07.658327+0000","last_active":"2024-01-16T15:05:08.662377+0000","last_peered":"2024-01-16T15:05:08.662377+0000","last_clean":"2024-01-16T15:05:08.662377+0000","last_became_active":"2024-01-16T15:05:07.658001+0000","last_became_peered":"2024-01-16T15:05:07.658001+0000","last_unstale":"2024-01-16T15:05:08.662377+0000","last_undegraded":"2024-01-16T15:05:08.662377+0000","last_fullsized":"2024-01-16T15:05:08.662377+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:59:09.184565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624377+0000","last_change":"2024-01-16T15:05:06.775062+0000","last_active":"2024-01-16T15:05:08.624377+0000","last_peered":"2024-01-16T15:05:08.624377+0000","last_clean":"2024-01-16T15:05:08.624377+0000","last_became_active":"2024-01-16T15:05:06.774476+0000","last_became_peered":"2024-01-16T15:05:06.774476+0000","last_unstale":"2024-01-16T15:05:08.624377+0000","last_undegraded":"2024-01-16T15:05:08.624377+0000","last_fullsized":"2024-01-16T15:05:08.624377+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:04.884120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660283+0000","last_change":"2024-01-16T15:05:07.649774+0000","last_active":"2024-01-16T15:05:08.660283+0000","last_peered":"2024-01-16T15:05:08.660283+0000","last_clean":"2024-01-16T15:05:08.660283+0000","last_became_active":"2024-01-16T15:05:07.649406+0000","last_became_peered":"2024-01-16T15:05:07.649406+0000","last_unstale":"2024-01-16T15:05:08.660283+0000","last_undegraded":"2024-01-16T15:05:08.660283+0000","last_fullsized":"2024-01-16T15:05:08.660283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:20:50.207173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662699+0000","last_change":"2024-01-16T15:05:06.772357+0000","last_active":"2024-01-16T15:05:08.662699+0000","last_peered":"2024-01-16T15:05:08.662699+0000","last_clean":"2024-01-16T15:05:08.662699+0000","last_became_active":"2024-01-16T15:05:06.772041+0000","last_became_peered":"2024-01-16T15:05:06.772041+0000","last_unstale":"2024-01-16T15:05:08.662699+0000","last_undegraded":"2024-01-16T15:05:08.662699+0000","last_fullsized":"2024-01-16T15:05:08.662699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:46.784398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624520+0000","last_change":"2024-01-16T15:05:07.646253+0000","last_active":"2024-01-16T15:05:08.624520+0000","last_peered":"2024-01-16T15:05:08.624520+0000","last_clean":"2024-01-16T15:05:08.624520+0000","last_became_active":"2024-01-16T15:05:07.645944+0000","last_became_peered":"2024-01-16T15:05:07.645944+0000","last_unstale":"2024-01-16T15:05:08.624520+0000","last_undegraded":"2024-01-16T15:05:08.624520+0000","last_fullsized":"2024-01-16T15:05:08.624520+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:17:23.087309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657698+0000","last_change":"2024-01-16T15:05:06.756923+0000","last_active":"2024-01-16T15:05:08.657698+0000","last_peered":"2024-01-16T15:05:08.657698+0000","last_clean":"2024-01-16T15:05:08.657698+0000","last_became_active":"2024-01-16T15:05:06.756258+0000","last_became_peered":"2024-01-16T15:05:06.756258+0000","last_unstale":"2024-01-16T15:05:08.657698+0000","last_undegraded":"2024-01-16T15:05:08.657698+0000","last_fullsized":"2024-01-16T15:05:08.657698+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:20.678593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625931+0000","last_change":"2024-01-16T15:05:07.645620+0000","last_active":"2024-01-16T15:05:08.625931+0000","last_peered":"2024-01-16T15:05:08.625931+0000","last_clean":"2024-01-16T15:05:08.625931+0000","last_became_active":"2024-01-16T15:05:07.644959+0000","last_became_peered":"2024-01-16T15:05:07.644959+0000","last_unstale":"2024-01-16T15:05:08.625931+0000","last_undegraded":"2024-01-16T15:05:08.625931+0000","last_fullsized":"2024-01-16T15:05:08.625931+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:19:57.069028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661843+0000","last_change":"2024-01-16T15:05:06.774132+0000","last_active":"2024-01-16T15:05:08.661843+0000","last_peered":"2024-01-16T15:05:08.661843+0000","last_clean":"2024-01-16T15:05:08.661843+0000","last_became_active":"2024-01-16T15:05:06.773226+0000","last_became_peered":"2024-01-16T15:05:06.773226+0000","last_unstale":"2024-01-16T15:05:08.661843+0000","last_undegraded":"2024-01-16T15:05:08.661843+0000","last_fullsized":"2024-01-16T15:05:08.661843+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:38.352691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618669+0000","last_change":"2024-01-16T15:05:07.637661+0000","last_active":"2024-01-16T15:05:08.618669+0000","last_peered":"2024-01-16T15:05:08.618669+0000","last_clean":"2024-01-16T15:05:08.618669+0000","last_became_active":"2024-01-16T15:05:07.636002+0000","last_became_peered":"2024-01-16T15:05:07.636002+0000","last_unstale":"2024-01-16T15:05:08.618669+0000","last_undegraded":"2024-01-16T15:05:08.618669+0000","last_fullsized":"2024-01-16T15:05:08.618669+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:10.357784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657610+0000","last_change":"2024-01-16T15:05:06.756400+0000","last_active":"2024-01-16T15:05:08.657610+0000","last_peered":"2024-01-16T15:05:08.657610+0000","last_clean":"2024-01-16T15:05:08.657610+0000","last_became_active":"2024-01-16T15:05:06.756116+0000","last_became_peered":"2024-01-16T15:05:06.756116+0000","last_unstale":"2024-01-16T15:05:08.657610+0000","last_undegraded":"2024-01-16T15:05:08.657610+0000","last_fullsized":"2024-01-16T15:05:08.657610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:02.951844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658245+0000","last_change":"2024-01-16T15:05:07.641802+0000","last_active":"2024-01-16T15:05:08.658245+0000","last_peered":"2024-01-16T15:05:08.658245+0000","last_clean":"2024-01-16T15:05:08.658245+0000","last_became_active":"2024-01-16T15:05:07.641511+0000","last_became_peered":"2024-01-16T15:05:07.641511+0000","last_unstale":"2024-01-16T15:05:08.658245+0000","last_undegraded":"2024-01-16T15:05:08.658245+0000","last_fullsized":"2024-01-16T15:05:08.658245+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:56.551650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618707+0000","last_change":"2024-01-16T15:05:06.755310+0000","last_active":"2024-01-16T15:05:08.618707+0000","last_peered":"2024-01-16T15:05:08.618707+0000","last_clean":"2024-01-16T15:05:08.618707+0000","last_became_active":"2024-01-16T15:05:06.755007+0000","last_became_peered":"2024-01-16T15:05:06.755007+0000","last_unstale":"2024-01-16T15:05:08.618707+0000","last_undegraded":"2024-01-16T15:05:08.618707+0000","last_fullsized":"2024-01-16T15:05:08.618707+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:41.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626342+0000","last_change":"2024-01-16T15:05:07.646074+0000","last_active":"2024-01-16T15:05:08.626342+0000","last_peered":"2024-01-16T15:05:08.626342+0000","last_clean":"2024-01-16T15:05:08.626342+0000","last_became_active":"2024-01-16T15:05:07.645773+0000","last_became_peered":"2024-01-16T15:05:07.645773+0000","last_unstale":"2024-01-16T15:05:08.626342+0000","last_undegraded":"2024-01-16T15:05:08.626342+0000","last_fullsized":"2024-01-16T15:05:08.626342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:10.411121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624798+0000","last_change":"2024-01-16T15:05:06.762652+0000","last_active":"2024-01-16T15:05:08.624798+0000","last_peered":"2024-01-16T15:05:08.624798+0000","last_clean":"2024-01-16T15:05:08.624798+0000","last_became_active":"2024-01-16T15:05:06.762146+0000","last_became_peered":"2024-01-16T15:05:06.762146+0000","last_unstale":"2024-01-16T15:05:08.624798+0000","last_undegraded":"2024-01-16T15:05:08.624798+0000","last_fullsized":"2024-01-16T15:05:08.624798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:16.330927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660806+0000","last_change":"2024-01-16T15:05:07.654491+0000","last_active":"2024-01-16T15:05:08.660806+0000","last_peered":"2024-01-16T15:05:08.660806+0000","last_clean":"2024-01-16T15:05:08.660806+0000","last_became_active":"2024-01-16T15:05:07.654101+0000","last_became_peered":"2024-01-16T15:05:07.654101+0000","last_unstale":"2024-01-16T15:05:08.660806+0000","last_undegraded":"2024-01-16T15:05:08.660806+0000","last_fullsized":"2024-01-16T15:05:08.660806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:31.830075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.663156+0000","last_change":"2024-01-16T15:05:06.773075+0000","last_active":"2024-01-16T15:05:08.663156+0000","last_peered":"2024-01-16T15:05:08.663156+0000","last_clean":"2024-01-16T15:05:08.663156+0000","last_became_active":"2024-01-16T15:05:06.772805+0000","last_became_peered":"2024-01-16T15:05:06.772805+0000","last_unstale":"2024-01-16T15:05:08.663156+0000","last_undegraded":"2024-01-16T15:05:08.663156+0000","last_fullsized":"2024-01-16T15:05:08.663156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:57.728311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662808+0000","last_change":"2024-01-16T15:05:07.656689+0000","last_active":"2024-01-16T15:05:08.662808+0000","last_peered":"2024-01-16T15:05:08.662808+0000","last_clean":"2024-01-16T15:05:08.662808+0000","last_became_active":"2024-01-16T15:05:07.656465+0000","last_became_peered":"2024-01-16T15:05:07.656465+0000","last_unstale":"2024-01-16T15:05:08.662808+0000","last_undegraded":"2024-01-16T15:05:08.662808+0000","last_fullsized":"2024-01-16T15:05:08.662808+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:03.997553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"72'3","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658021+0000","last_change":"2024-01-16T15:05:06.769573+0000","last_active":"2024-01-16T15:05:08.658021+0000","last_peered":"2024-01-16T15:05:08.658021+0000","last_clean":"2024-01-16T15:05:08.658021+0000","last_became_active":"2024-01-16T15:05:06.769264+0000","last_became_peered":"2024-01-16T15:05:06.769264+0000","last_unstale":"2024-01-16T15:05:08.658021+0000","last_undegraded":"2024-01-16T15:05:08.658021+0000","last_fullsized":"2024-01-16T15:05:08.658021+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:36.817166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619404+0000","last_change":"2024-01-16T15:05:07.643330+0000","last_active":"2024-01-16T15:05:08.619404+0000","last_peered":"2024-01-16T15:05:08.619404+0000","last_clean":"2024-01-16T15:05:08.619404+0000","last_became_active":"2024-01-16T15:05:07.641982+0000","last_became_peered":"2024-01-16T15:05:07.641982+0000","last_unstale":"2024-01-16T15:05:08.619404+0000","last_undegraded":"2024-01-16T15:05:08.619404+0000","last_fullsized":"2024-01-16T15:05:08.619404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:40.803835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"72'6","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657571+0000","last_change":"2024-01-16T15:05:06.752220+0000","last_active":"2024-01-16T15:05:08.657571+0000","last_peered":"2024-01-16T15:05:08.657571+0000","last_clean":"2024-01-16T15:05:08.657571+0000","last_became_active":"2024-01-16T15:05:06.751384+0000","last_became_peered":"2024-01-16T15:05:06.751384+0000","last_unstale":"2024-01-16T15:05:08.657571+0000","last_undegraded":"2024-01-16T15:05:08.657571+0000","last_fullsized":"2024-01-16T15:05:08.657571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:28:05.268275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659362+0000","last_change":"2024-01-16T15:05:07.640593+0000","last_active":"2024-01-16T15:05:08.659362+0000","last_peered":"2024-01-16T15:05:08.659362+0000","last_clean":"2024-01-16T15:05:08.659362+0000","last_became_active":"2024-01-16T15:05:07.640107+0000","last_became_peered":"2024-01-16T15:05:07.640107+0000","last_unstale":"2024-01-16T15:05:08.659362+0000","last_undegraded":"2024-01-16T15:05:08.659362+0000","last_fullsized":"2024-01-16T15:05:08.659362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:39.011373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657939+0000","last_change":"2024-01-16T15:05:06.751216+0000","last_active":"2024-01-16T15:05:08.657939+0000","last_peered":"2024-01-16T15:05:08.657939+0000","last_clean":"2024-01-16T15:05:08.657939+0000","last_became_active":"2024-01-16T15:05:06.749321+0000","last_became_peered":"2024-01-16T15:05:06.749321+0000","last_unstale":"2024-01-16T15:05:08.657939+0000","last_undegraded":"2024-01-16T15:05:08.657939+0000","last_fullsized":"2024-01-16T15:05:08.657939+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:34.393559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625552+0000","last_change":"2024-01-16T15:05:07.641958+0000","last_active":"2024-01-16T15:05:08.625552+0000","last_peered":"2024-01-16T15:05:08.625552+0000","last_clean":"2024-01-16T15:05:08.625552+0000","last_became_active":"2024-01-16T15:05:07.641680+0000","last_became_peered":"2024-01-16T15:05:07.641680+0000","last_unstale":"2024-01-16T15:05:08.625552+0000","last_undegraded":"2024-01-16T15:05:08.625552+0000","last_fullsized":"2024-01-16T15:05:08.625552+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:55.916665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661823+0000","last_change":"2024-01-16T15:05:06.772979+0000","last_active":"2024-01-16T15:05:08.661823+0000","last_peered":"2024-01-16T15:05:08.661823+0000","last_clean":"2024-01-16T15:05:08.661823+0000","last_became_active":"2024-01-16T15:05:06.772665+0000","last_became_peered":"2024-01-16T15:05:06.772665+0000","last_unstale":"2024-01-16T15:05:08.661823+0000","last_undegraded":"2024-01-16T15:05:08.661823+0000","last_fullsized":"2024-01-16T15:05:08.661823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:32.194251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657572+0000","last_change":"2024-01-16T15:05:07.654495+0000","last_active":"2024-01-16T15:05:08.657572+0000","last_peered":"2024-01-16T15:05:08.657572+0000","last_clean":"2024-01-16T15:05:08.657572+0000","last_became_active":"2024-01-16T15:05:07.653970+0000","last_became_peered":"2024-01-16T15:05:07.653970+0000","last_unstale":"2024-01-16T15:05:08.657572+0000","last_undegraded":"2024-01-16T15:05:08.657572+0000","last_fullsized":"2024-01-16T15:05:08.657572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:40.099590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624728+0000","last_change":"2024-01-16T15:05:06.768865+0000","last_active":"2024-01-16T15:05:08.624728+0000","last_peered":"2024-01-16T15:05:08.624728+0000","last_clean":"2024-01-16T15:05:08.624728+0000","last_became_active":"2024-01-16T15:05:06.768409+0000","last_became_peered":"2024-01-16T15:05:06.768409+0000","last_unstale":"2024-01-16T15:05:08.624728+0000","last_undegraded":"2024-01-16T15:05:08.624728+0000","last_fullsized":"2024-01-16T15:05:08.624728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:43.180649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618991+0000","last_change":"2024-01-16T15:05:07.639913+0000","last_active":"2024-01-16T15:05:08.618991+0000","last_peered":"2024-01-16T15:05:08.618991+0000","last_clean":"2024-01-16T15:05:08.618991+0000","last_became_active":"2024-01-16T15:05:07.639493+0000","last_became_peered":"2024-01-16T15:05:07.639493+0000","last_unstale":"2024-01-16T15:05:08.618991+0000","last_undegraded":"2024-01-16T15:05:08.618991+0000","last_fullsized":"2024-01-16T15:05:08.618991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:26.138458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"72'8","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.623492+0000","last_change":"2024-01-16T15:05:06.765582+0000","last_active":"2024-01-16T15:05:08.623492+0000","last_peered":"2024-01-16T15:05:08.623492+0000","last_clean":"2024-01-16T15:05:08.623492+0000","last_became_active":"2024-01-16T15:05:06.765282+0000","last_became_peered":"2024-01-16T15:05:06.765282+0000","last_unstale":"2024-01-16T15:05:08.623492+0000","last_undegraded":"2024-01-16T15:05:08.623492+0000","last_fullsized":"2024-01-16T15:05:08.623492+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:48:12.228401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624603+0000","last_change":"2024-01-16T15:05:07.644042+0000","last_active":"2024-01-16T15:05:08.624603+0000","last_peered":"2024-01-16T15:05:08.624603+0000","last_clean":"2024-01-16T15:05:08.624603+0000","last_became_active":"2024-01-16T15:05:07.642001+0000","last_became_peered":"2024-01-16T15:05:07.642001+0000","last_unstale":"2024-01-16T15:05:08.624603+0000","last_undegraded":"2024-01-16T15:05:08.624603+0000","last_fullsized":"2024-01-16T15:05:08.624603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:35.515237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662307+0000","last_change":"2024-01-16T15:05:06.773829+0000","last_active":"2024-01-16T15:05:08.662307+0000","last_peered":"2024-01-16T15:05:08.662307+0000","last_clean":"2024-01-16T15:05:08.662307+0000","last_became_active":"2024-01-16T15:05:06.773552+0000","last_became_peered":"2024-01-16T15:05:06.773552+0000","last_unstale":"2024-01-16T15:05:08.662307+0000","last_undegraded":"2024-01-16T15:05:08.662307+0000","last_fullsized":"2024-01-16T15:05:08.662307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:15:14.671873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661036+0000","last_change":"2024-01-16T15:05:07.650021+0000","last_active":"2024-01-16T15:05:08.661036+0000","last_peered":"2024-01-16T15:05:08.661036+0000","last_clean":"2024-01-16T15:05:08.661036+0000","last_became_active":"2024-01-16T15:05:07.649661+0000","last_became_peered":"2024-01-16T15:05:07.649661+0000","last_unstale":"2024-01-16T15:05:08.661036+0000","last_undegraded":"2024-01-16T15:05:08.661036+0000","last_fullsized":"2024-01-16T15:05:08.661036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:44.885273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624828+0000","last_change":"2024-01-16T15:05:06.773927+0000","last_active":"2024-01-16T15:05:08.624828+0000","last_peered":"2024-01-16T15:05:08.624828+0000","last_clean":"2024-01-16T15:05:08.624828+0000","last_became_active":"2024-01-16T15:05:06.772853+0000","last_became_peered":"2024-01-16T15:05:06.772853+0000","last_unstale":"2024-01-16T15:05:08.624828+0000","last_undegraded":"2024-01-16T15:05:08.624828+0000","last_fullsized":"2024-01-16T15:05:08.624828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:16.379742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657455+0000","last_change":"2024-01-16T15:05:07.648279+0000","last_active":"2024-01-16T15:05:08.657455+0000","last_peered":"2024-01-16T15:05:08.657455+0000","last_clean":"2024-01-16T15:05:08.657455+0000","last_became_active":"2024-01-16T15:05:07.645767+0000","last_became_peered":"2024-01-16T15:05:07.645767+0000","last_unstale":"2024-01-16T15:05:08.657455+0000","last_undegraded":"2024-01-16T15:05:08.657455+0000","last_fullsized":"2024-01-16T15:05:08.657455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:17.346546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662929+0000","last_change":"2024-01-16T15:05:06.772563+0000","last_active":"2024-01-16T15:05:08.662929+0000","last_peered":"2024-01-16T15:05:08.662929+0000","last_clean":"2024-01-16T15:05:08.662929+0000","last_became_active":"2024-01-16T15:05:06.772095+0000","last_became_peered":"2024-01-16T15:05:06.772095+0000","last_unstale":"2024-01-16T15:05:08.662929+0000","last_undegraded":"2024-01-16T15:05:08.662929+0000","last_fullsized":"2024-01-16T15:05:08.662929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:41.505008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619500+0000","last_change":"2024-01-16T15:05:07.636898+0000","last_active":"2024-01-16T15:05:08.619500+0000","last_peered":"2024-01-16T15:05:08.619500+0000","last_clean":"2024-01-16T15:05:08.619500+0000","last_became_active":"2024-01-16T15:05:07.636121+0000","last_became_peered":"2024-01-16T15:05:07.636121+0000","last_unstale":"2024-01-16T15:05:08.619500+0000","last_undegraded":"2024-01-16T15:05:08.619500+0000","last_fullsized":"2024-01-16T15:05:08.619500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:33.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659727+0000","last_change":"2024-01-16T15:05:06.761834+0000","last_active":"2024-01-16T15:05:08.659727+0000","last_peered":"2024-01-16T15:05:08.659727+0000","last_clean":"2024-01-16T15:05:08.659727+0000","last_became_active":"2024-01-16T15:05:06.761548+0000","last_became_peered":"2024-01-16T15:05:06.761548+0000","last_unstale":"2024-01-16T15:05:08.659727+0000","last_undegraded":"2024-01-16T15:05:08.659727+0000","last_fullsized":"2024-01-16T15:05:08.659727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:09.306688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657555+0000","last_change":"2024-01-16T15:05:07.642007+0000","last_active":"2024-01-16T15:05:08.657555+0000","last_peered":"2024-01-16T15:05:08.657555+0000","last_clean":"2024-01-16T15:05:08.657555+0000","last_became_active":"2024-01-16T15:05:07.640865+0000","last_became_peered":"2024-01-16T15:05:07.640865+0000","last_unstale":"2024-01-16T15:05:08.657555+0000","last_undegraded":"2024-01-16T15:05:08.657555+0000","last_fullsized":"2024-01-16T15:05:08.657555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:03.960723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662542+0000","last_change":"2024-01-16T15:05:06.772312+0000","last_active":"2024-01-16T15:05:08.662542+0000","last_peered":"2024-01-16T15:05:08.662542+0000","last_clean":"2024-01-16T15:05:08.662542+0000","last_became_active":"2024-01-16T15:05:06.771785+0000","last_became_peered":"2024-01-16T15:05:06.771785+0000","last_unstale":"2024-01-16T15:05:08.662542+0000","last_undegraded":"2024-01-16T15:05:08.662542+0000","last_fullsized":"2024-01-16T15:05:08.662542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:48.982295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657143+0000","last_change":"2024-01-16T15:05:07.651446+0000","last_active":"2024-01-16T15:05:08.657143+0000","last_peered":"2024-01-16T15:05:08.657143+0000","last_clean":"2024-01-16T15:05:08.657143+0000","last_became_active":"2024-01-16T15:05:07.650911+0000","last_became_peered":"2024-01-16T15:05:07.650911+0000","last_unstale":"2024-01-16T15:05:08.657143+0000","last_undegraded":"2024-01-16T15:05:08.657143+0000","last_fullsized":"2024-01-16T15:05:08.657143+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:54.752672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624948+0000","last_change":"2024-01-16T15:05:06.768685+0000","last_active":"2024-01-16T15:05:08.624948+0000","last_peered":"2024-01-16T15:05:08.624948+0000","last_clean":"2024-01-16T15:05:08.624948+0000","last_became_active":"2024-01-16T15:05:06.768026+0000","last_became_peered":"2024-01-16T15:05:06.768026+0000","last_unstale":"2024-01-16T15:05:08.624948+0000","last_undegraded":"2024-01-16T15:05:08.624948+0000","last_fullsized":"2024-01-16T15:05:08.624948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:34.152302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657537+0000","last_change":"2024-01-16T15:05:07.653100+0000","last_active":"2024-01-16T15:05:08.657537+0000","last_peered":"2024-01-16T15:05:08.657537+0000","last_clean":"2024-01-16T15:05:08.657537+0000","last_became_active":"2024-01-16T15:05:07.652825+0000","last_became_peered":"2024-01-16T15:05:07.652825+0000","last_unstale":"2024-01-16T15:05:08.657537+0000","last_undegraded":"2024-01-16T15:05:08.657537+0000","last_fullsized":"2024-01-16T15:05:08.657537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:20.389514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657014+0000","last_change":"2024-01-16T15:05:06.759707+0000","last_active":"2024-01-16T15:05:08.657014+0000","last_peered":"2024-01-16T15:05:08.657014+0000","last_clean":"2024-01-16T15:05:08.657014+0000","last_became_active":"2024-01-16T15:05:06.759361+0000","last_became_peered":"2024-01-16T15:05:06.759361+0000","last_unstale":"2024-01-16T15:05:08.657014+0000","last_undegraded":"2024-01-16T15:05:08.657014+0000","last_fullsized":"2024-01-16T15:05:08.657014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:31.341354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659111+0000","last_change":"2024-01-16T15:05:07.639656+0000","last_active":"2024-01-16T15:05:08.659111+0000","last_peered":"2024-01-16T15:05:08.659111+0000","last_clean":"2024-01-16T15:05:08.659111+0000","last_became_active":"2024-01-16T15:05:07.639350+0000","last_became_peered":"2024-01-16T15:05:07.639350+0000","last_unstale":"2024-01-16T15:05:08.659111+0000","last_undegraded":"2024-01-16T15:05:08.659111+0000","last_fullsized":"2024-01-16T15:05:08.659111+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:59.016104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662557+0000","last_change":"2024-01-16T15:05:06.773600+0000","last_active":"2024-01-16T15:05:08.662557+0000","last_peered":"2024-01-16T15:05:08.662557+0000","last_clean":"2024-01-16T15:05:08.662557+0000","last_became_active":"2024-01-16T15:05:06.773237+0000","last_became_peered":"2024-01-16T15:05:06.773237+0000","last_unstale":"2024-01-16T15:05:08.662557+0000","last_undegraded":"2024-01-16T15:05:08.662557+0000","last_fullsized":"2024-01-16T15:05:08.662557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:38.629598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625013+0000","last_change":"2024-01-16T15:05:07.643587+0000","last_active":"2024-01-16T15:05:08.625013+0000","last_peered":"2024-01-16T15:05:08.625013+0000","last_clean":"2024-01-16T15:05:08.625013+0000","last_became_active":"2024-01-16T15:05:07.643292+0000","last_became_peered":"2024-01-16T15:05:07.643292+0000","last_unstale":"2024-01-16T15:05:08.625013+0000","last_undegraded":"2024-01-16T15:05:08.625013+0000","last_fullsized":"2024-01-16T15:05:08.625013+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:10.322307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626344+0000","last_change":"2024-01-16T15:05:06.765718+0000","last_active":"2024-01-16T15:05:08.626344+0000","last_peered":"2024-01-16T15:05:08.626344+0000","last_clean":"2024-01-16T15:05:08.626344+0000","last_became_active":"2024-01-16T15:05:06.765434+0000","last_became_peered":"2024-01-16T15:05:06.765434+0000","last_unstale":"2024-01-16T15:05:08.626344+0000","last_undegraded":"2024-01-16T15:05:08.626344+0000","last_fullsized":"2024-01-16T15:05:08.626344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:29.043656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659582+0000","last_change":"2024-01-16T15:05:07.655917+0000","last_active":"2024-01-16T15:05:08.659582+0000","last_peered":"2024-01-16T15:05:08.659582+0000","last_clean":"2024-01-16T15:05:08.659582+0000","last_became_active":"2024-01-16T15:05:07.655367+0000","last_became_peered":"2024-01-16T15:05:07.655367+0000","last_unstale":"2024-01-16T15:05:08.659582+0000","last_undegraded":"2024-01-16T15:05:08.659582+0000","last_fullsized":"2024-01-16T15:05:08.659582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:10.411121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619563+0000","last_change":"2024-01-16T15:05:06.760719+0000","last_active":"2024-01-16T15:05:08.619563+0000","last_peered":"2024-01-16T15:05:08.619563+0000","last_clean":"2024-01-16T15:05:08.619563+0000","last_became_active":"2024-01-16T15:05:06.760358+0000","last_became_peered":"2024-01-16T15:05:06.760358+0000","last_unstale":"2024-01-16T15:05:08.619563+0000","last_undegraded":"2024-01-16T15:05:08.619563+0000","last_fullsized":"2024-01-16T15:05:08.619563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:43:52.397412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660389+0000","last_change":"2024-01-16T15:05:07.642118+0000","last_active":"2024-01-16T15:05:08.660389+0000","last_peered":"2024-01-16T15:05:08.660389+0000","last_clean":"2024-01-16T15:05:08.660389+0000","last_became_active":"2024-01-16T15:05:07.641786+0000","last_became_peered":"2024-01-16T15:05:07.641786+0000","last_unstale":"2024-01-16T15:05:08.660389+0000","last_undegraded":"2024-01-16T15:05:08.660389+0000","last_fullsized":"2024-01-16T15:05:08.660389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:34.258858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658850+0000","last_change":"2024-01-16T15:05:06.762726+0000","last_active":"2024-01-16T15:05:08.658850+0000","last_peered":"2024-01-16T15:05:08.658850+0000","last_clean":"2024-01-16T15:05:08.658850+0000","last_became_active":"2024-01-16T15:05:06.762342+0000","last_became_peered":"2024-01-16T15:05:06.762342+0000","last_unstale":"2024-01-16T15:05:08.658850+0000","last_undegraded":"2024-01-16T15:05:08.658850+0000","last_fullsized":"2024-01-16T15:05:08.658850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:49.831467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619362+0000","last_change":"2024-01-16T15:05:07.647293+0000","last_active":"2024-01-16T15:05:08.619362+0000","last_peered":"2024-01-16T15:05:08.619362+0000","last_clean":"2024-01-16T15:05:08.619362+0000","last_became_active":"2024-01-16T15:05:07.646767+0000","last_became_peered":"2024-01-16T15:05:07.646767+0000","last_unstale":"2024-01-16T15:05:08.619362+0000","last_undegraded":"2024-01-16T15:05:08.619362+0000","last_fullsized":"2024-01-16T15:05:08.619362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:06.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619019+0000","last_change":"2024-01-16T15:05:06.768512+0000","last_active":"2024-01-16T15:05:08.619019+0000","last_peered":"2024-01-16T15:05:08.619019+0000","last_clean":"2024-01-16T15:05:08.619019+0000","last_became_active":"2024-01-16T15:05:06.768051+0000","last_became_peered":"2024-01-16T15:05:06.768051+0000","last_unstale":"2024-01-16T15:05:08.619019+0000","last_undegraded":"2024-01-16T15:05:08.619019+0000","last_fullsized":"2024-01-16T15:05:08.619019+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:17.266108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662819+0000","last_change":"2024-01-16T15:05:07.647077+0000","last_active":"2024-01-16T15:05:08.662819+0000","last_peered":"2024-01-16T15:05:08.662819+0000","last_clean":"2024-01-16T15:05:08.662819+0000","last_became_active":"2024-01-16T15:05:07.646718+0000","last_became_peered":"2024-01-16T15:05:07.646718+0000","last_unstale":"2024-01-16T15:05:08.662819+0000","last_undegraded":"2024-01-16T15:05:08.662819+0000","last_fullsized":"2024-01-16T15:05:08.662819+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:23.042966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657398+0000","last_change":"2024-01-16T15:05:06.757624+0000","last_active":"2024-01-16T15:05:08.657398+0000","last_peered":"2024-01-16T15:05:08.657398+0000","last_clean":"2024-01-16T15:05:08.657398+0000","last_became_active":"2024-01-16T15:05:06.757315+0000","last_became_peered":"2024-01-16T15:05:06.757315+0000","last_unstale":"2024-01-16T15:05:08.657398+0000","last_undegraded":"2024-01-16T15:05:08.657398+0000","last_fullsized":"2024-01-16T15:05:08.657398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:59.925173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657270+0000","last_change":"2024-01-16T15:05:07.649855+0000","last_active":"2024-01-16T15:05:08.657270+0000","last_peered":"2024-01-16T15:05:08.657270+0000","last_clean":"2024-01-16T15:05:08.657270+0000","last_became_active":"2024-01-16T15:05:07.649032+0000","last_became_peered":"2024-01-16T15:05:07.649032+0000","last_unstale":"2024-01-16T15:05:08.657270+0000","last_undegraded":"2024-01-16T15:05:08.657270+0000","last_fullsized":"2024-01-16T15:05:08.657270+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:12.397691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.623382+0000","last_change":"2024-01-16T15:05:06.764342+0000","last_active":"2024-01-16T15:05:08.623382+0000","last_peered":"2024-01-16T15:05:08.623382+0000","last_clean":"2024-01-16T15:05:08.623382+0000","last_became_active":"2024-01-16T15:05:06.762214+0000","last_became_peered":"2024-01-16T15:05:06.762214+0000","last_unstale":"2024-01-16T15:05:08.623382+0000","last_undegraded":"2024-01-16T15:05:08.623382+0000","last_fullsized":"2024-01-16T15:05:08.623382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:08.905947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661398+0000","last_change":"2024-01-16T15:05:07.649800+0000","last_active":"2024-01-16T15:05:08.661398+0000","last_peered":"2024-01-16T15:05:08.661398+0000","last_clean":"2024-01-16T15:05:08.661398+0000","last_became_active":"2024-01-16T15:05:07.649548+0000","last_became_peered":"2024-01-16T15:05:07.649548+0000","last_unstale":"2024-01-16T15:05:08.661398+0000","last_undegraded":"2024-01-16T15:05:08.661398+0000","last_fullsized":"2024-01-16T15:05:08.661398+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:47.475772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661317+0000","last_change":"2024-01-16T15:05:06.772244+0000","last_active":"2024-01-16T15:05:08.661317+0000","last_peered":"2024-01-16T15:05:08.661317+0000","last_clean":"2024-01-16T15:05:08.661317+0000","last_became_active":"2024-01-16T15:05:06.771901+0000","last_became_peered":"2024-01-16T15:05:06.771901+0000","last_unstale":"2024-01-16T15:05:08.661317+0000","last_undegraded":"2024-01-16T15:05:08.661317+0000","last_fullsized":"2024-01-16T15:05:08.661317+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:39.079933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657449+0000","last_change":"2024-01-16T15:05:07.651021+0000","last_active":"2024-01-16T15:05:08.657449+0000","last_peered":"2024-01-16T15:05:08.657449+0000","last_clean":"2024-01-16T15:05:08.657449+0000","last_became_active":"2024-01-16T15:05:07.650732+0000","last_became_peered":"2024-01-16T15:05:07.650732+0000","last_unstale":"2024-01-16T15:05:08.657449+0000","last_undegraded":"2024-01-16T15:05:08.657449+0000","last_fullsized":"2024-01-16T15:05:08.657449+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:02.967841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662128+0000","last_change":"2024-01-16T15:05:06.770836+0000","last_active":"2024-01-16T15:05:08.662128+0000","last_peered":"2024-01-16T15:05:08.662128+0000","last_clean":"2024-01-16T15:05:08.662128+0000","last_became_active":"2024-01-16T15:05:06.770311+0000","last_became_peered":"2024-01-16T15:05:06.770311+0000","last_unstale":"2024-01-16T15:05:08.662128+0000","last_undegraded":"2024-01-16T15:05:08.662128+0000","last_fullsized":"2024-01-16T15:05:08.662128+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:43.218449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":216,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:05:07.628935+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:05:07.628935+0000","last_peered":"2024-01-16T15:05:07.628935+0000","last_clean":"2024-01-16T15:05:07.628935+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:05:07.628935+0000","last_undegraded":"2024-01-16T15:05:07.628935+0000","last_fullsized":"2024-01-16T15:05:07.628935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50684,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321156,"statfs":{"total":96636764160,"available":96584863744,"internally_reserved":0,"allocated":1486848,"data_stored":964824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.6379999999999999,"15min":1.256},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":8.2639999999999993,"5min":15.1,"15min":15.1},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.3240000000000001,"15min":1.2150000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.55400000000000005},"max":{"1min":1.879,"5min":3.5089999999999999,"15min":3.5089999999999999},"last":0.98999999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.611,"15min":1.226},"min":{"1min":0.79200000000000004,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":8.3740000000000006,"5min":14.988,"15min":14.988},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.351,"15min":1.2050000000000001},"min":{"1min":0.70199999999999996,"5min":0.64900000000000002,"15min":0.60599999999999998},"max":{"1min":1.619,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":1.325}]},{"osd":2,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.3049999999999999,"15min":1.212},"min":{"1min":0.748,"5min":0.748,"15min":0.69499999999999995},"max":{"1min":1.6759999999999999,"5min":3.4569999999999999,"15min":3.4569999999999999},"last":1.482},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.6759999999999999,"15min":1.359},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":5.4610000000000003,"5min":20.577000000000002,"15min":20.577000000000002},"last":1.4550000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.9510000000000001,"15min":1.4970000000000001},"min":{"1min":0.73299999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":8.3360000000000003,"5min":15.231,"15min":15.231},"last":1.175},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.6060000000000001,"15min":1.4570000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.73199999999999998},"max":{"1min":4.367,"5min":10.141999999999999,"15min":10.141999999999999},"last":1.542}]},{"osd":4,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.6279999999999999,"15min":1.4419999999999999},"min":{"1min":0.80600000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.605,"5min":20.649999999999999,"15min":20.649999999999999},"last":1.254},{"interface":"front","average":{"1min":1.641,"5min":1.8540000000000001,"15min":1.5780000000000001},"min":{"1min":0.82599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":8.1349999999999998,"5min":15.515000000000001,"15min":15.515000000000001},"last":1.399}]},{"osd":5,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":2.9020000000000001,"5min":1.8129999999999999,"15min":1.625},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.52800000000000002},"max":{"1min":29.567,"5min":29.567,"15min":29.567},"last":1.6599999999999999},{"interface":"front","average":{"1min":2.9620000000000002,"5min":2.0659999999999998,"15min":1.768},"min":{"1min":0.92600000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":27.937000000000001,"5min":27.937000000000001,"15min":27.937000000000001},"last":1.512}]},{"osd":6,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":2.056,"15min":1.7330000000000001},"min":{"1min":0.63800000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.9330000000000001,"5min":41.206000000000003,"15min":41.206000000000003},"last":1.621},{"interface":"front","average":{"1min":1.8720000000000001,"5min":2.3140000000000001,"15min":1.833},"min":{"1min":0.72499999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":8.2219999999999995,"5min":21.908000000000001,"15min":21.908000000000001},"last":1.5740000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542244,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50100,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321740,"statfs":{"total":96636764160,"available":96585461760,"internally_reserved":0,"allocated":892928,"data_stored":373932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.4079999999999999,"15min":1.198},"min":{"1min":0.61299999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":4.6920000000000002,"5min":9.1890000000000001,"15min":9.1890000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.925,"5min":2.1379999999999999,"15min":1.52},"min":{"1min":0.51700000000000002,"5min":0.502,"15min":0.502},"max":{"1min":9.891,"5min":50.472000000000001,"15min":50.472000000000001},"last":1.617}]},{"osd":1,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5580000000000001,"15min":1.2110000000000001},"min":{"1min":0.76200000000000001,"5min":0.66300000000000003,"15min":0.624},"max":{"1min":4.2850000000000001,"5min":10.629,"15min":10.629},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.581,"5min":1.5329999999999999,"15min":1.423},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":4.2590000000000003,"5min":9.3279999999999994,"15min":9.3279999999999994},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":2.1030000000000002,"5min":2.0390000000000001,"15min":1.4830000000000001},"min":{"1min":0.59299999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":9.8140000000000001,"5min":50.643999999999998,"15min":50.643999999999998},"last":1.173},{"interface":"front","average":{"1min":1.9219999999999999,"5min":1.653,"15min":1.405},"min":{"1min":0.91900000000000004,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":6.5860000000000003,"5min":10.843,"15min":10.843},"last":1.6659999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":2.3490000000000002,"5min":1.7929999999999999,"15min":1.4990000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":8.6199999999999992,"5min":15.675000000000001,"15min":15.675000000000001},"last":1.105},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.63900000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":9.9770000000000003,"5min":50.561,"15min":50.561},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.3559999999999999,"15min":1.776},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":14.581,"5min":47.488,"15min":47.488},"last":1.3620000000000001},{"interface":"front","average":{"1min":2.8740000000000001,"5min":2.504,"15min":1.843},"min":{"1min":0.92900000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":11.295999999999999,"5min":50.381,"15min":50.381},"last":1.268}]},{"osd":5,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.74,"15min":1.5609999999999999},"min":{"1min":0.71699999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":4.3769999999999998,"5min":10.654,"15min":10.654},"last":1.4670000000000001},{"interface":"front","average":{"1min":2.0019999999999998,"5min":1.8919999999999999,"15min":1.7370000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":9.2949999999999999,"5min":9.3729999999999993,"15min":9.3729999999999993},"last":1.4410000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":2.702,"15min":2.0099999999999998},"min":{"1min":0.77200000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":18.481000000000002,"5min":52.912999999999997,"15min":52.912999999999997},"last":1.5189999999999999},{"interface":"front","average":{"1min":5.9390000000000001,"5min":2.698,"15min":2.0110000000000001},"min":{"1min":0.76200000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":18.899000000000001,"5min":18.899000000000001,"15min":18.899000000000001},"last":1.5740000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542244,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50100,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321740,"statfs":{"total":96636764160,"available":96585461760,"internally_reserved":0,"allocated":888832,"data_stored":373930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.48,"15min":1.2789999999999999},"min":{"1min":0.75800000000000001,"5min":0.496,"15min":0.496},"max":{"1min":8.298,"5min":13.938000000000001,"15min":13.938000000000001},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.6100000000000001,"15min":1.2529999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":11.895,"5min":16.896000000000001,"15min":16.896000000000001},"last":0.86099999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8109999999999999,"5min":1.6220000000000001,"15min":1.3859999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":11.797000000000001,"5min":13.279999999999999,"15min":13.279999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.8080000000000001,"15min":1.357},"min":{"1min":0.64200000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":4.8860000000000001,"5min":35.328000000000003,"15min":35.328000000000003},"last":1.6890000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.9119999999999999,"15min":1.478},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.9950000000000001,"5min":35.398000000000003,"15min":35.398000000000003},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.744,"5min":1.4410000000000001,"15min":1.3169999999999999},"min":{"1min":0.753,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":8.3510000000000009,"5min":8.3510000000000009,"15min":8.3510000000000009},"last":1.522}]},{"osd":3,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.8769999999999998,"5min":1.978,"15min":1.546},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":24.254000000000001,"5min":24.254000000000001,"15min":24.254000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":2.6869999999999998,"5min":2.2410000000000001,"15min":1.6739999999999999},"min":{"1min":0.78900000000000003,"5min":0.63,"15min":0.63},"max":{"1min":23.367999999999999,"5min":35.268000000000001,"15min":35.268000000000001},"last":1.8620000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.0289999999999999,"15min":1.6419999999999999},"min":{"1min":0.81000000000000005,"5min":0.45400000000000001,"15min":0.39000000000000001},"max":{"1min":14.695,"5min":15.757,"15min":15.757},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.865,"5min":2.1819999999999999,"15min":1.7589999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.45000000000000001},"max":{"1min":11.933999999999999,"5min":26.166,"15min":26.166},"last":1.2609999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":2.1619999999999999,"15min":1.7749999999999999},"min":{"1min":0.90400000000000003,"5min":0.53400000000000003,"15min":0.499},"max":{"1min":6.4349999999999996,"5min":35.427999999999997,"15min":35.427999999999997},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.994,"5min":1.972,"15min":1.7649999999999999},"min":{"1min":0.84199999999999997,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":8.4169999999999998,"5min":13.551,"15min":13.551},"last":2.23}]},{"osd":7,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.1360000000000001,"15min":1.788},"min":{"1min":0.73099999999999998,"5min":0.58099999999999996,"15min":0.41099999999999998},"max":{"1min":13.567,"5min":19.364999999999998,"15min":19.364999999999998},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.7290000000000001,"5min":2.1930000000000001,"15min":1.8700000000000001},"min":{"1min":0.88300000000000001,"5min":0.71699999999999997,"15min":0.42899999999999999},"max":{"1min":5.9039999999999999,"5min":35.363,"15min":35.363},"last":1.9890000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50096,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321744,"statfs":{"total":96636764160,"available":96585465856,"internally_reserved":0,"allocated":888832,"data_stored":373910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.2130000000000001,"5min":1.423,"15min":1.1910000000000001},"min":{"1min":0.78700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":11.647,"5min":11.647,"15min":11.647},"last":0.77200000000000002},{"interface":"front","average":{"1min":2.1930000000000001,"5min":1.605,"15min":1.3460000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":9.8599999999999994,"5min":9.8599999999999994,"15min":9.8599999999999994},"last":1.159}]},{"osd":1,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":1.679,"15min":1.347},"min":{"1min":0.72899999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.3919999999999995,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":1.417},{"interface":"front","average":{"1min":1.8300000000000001,"5min":1.5109999999999999,"15min":1.298},"min":{"1min":0.84199999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":9.1470000000000002,"5min":9.1470000000000002,"15min":9.1470000000000002},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.554,"15min":1.448},"min":{"1min":0.78600000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":12.733000000000001,"5min":12.733000000000001,"15min":15.986000000000001},"last":1.24},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.47,"15min":1.454},"min":{"1min":0.81399999999999995,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":9.2639999999999993,"5min":9.2639999999999993,"15min":9.2639999999999993},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.954,"15min":1.696},"min":{"1min":0.85899999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":7.3849999999999998,"5min":11.481,"15min":11.481},"last":1.3240000000000001},{"interface":"front","average":{"1min":2.2650000000000001,"5min":1.7509999999999999,"15min":1.528},"min":{"1min":0.88,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":7.7990000000000004,"5min":21.706,"15min":21.706},"last":1.4630000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.9079999999999999,"5min":1.4850000000000001,"15min":1.5720000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":9.3149999999999995,"5min":9.3149999999999995,"15min":9.3149999999999995},"last":1.0740000000000001},{"interface":"front","average":{"1min":2.0139999999999998,"5min":1.728,"15min":1.645},"min":{"1min":0.50700000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":9.7739999999999991,"5min":10.831,"15min":10.831},"last":1.5069999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":1.5269999999999999,"15min":1.575},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":7.4240000000000004,"5min":7.4240000000000004,"15min":7.4240000000000004},"last":1.5549999999999999},{"interface":"front","average":{"1min":2.1520000000000001,"5min":1.8560000000000001,"15min":1.752},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":9.8810000000000002,"5min":20.939,"15min":20.939},"last":1.2789999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.952,"5min":1.857,"15min":1.7549999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":7.6929999999999996,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":1.603},{"interface":"front","average":{"1min":2.8700000000000001,"5min":1.917,"15min":1.804},"min":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":20.277999999999999,"5min":20.277999999999999,"15min":20.277999999999999},"last":1.3660000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542244,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54264,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317576,"statfs":{"total":96636764160,"available":96581197824,"internally_reserved":0,"allocated":892928,"data_stored":373952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.044,"15min":0.93799999999999994},"min":{"1min":0.52400000000000002,"5min":0.43099999999999999,"15min":0.33600000000000002},"max":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.198,"15min":0.96599999999999997},"min":{"1min":0.50600000000000001,"5min":0.502,"15min":0.45000000000000001},"max":{"1min":1.9099999999999999,"5min":6.048,"15min":6.048},"last":1.093}]},{"osd":1,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.94899999999999995,"15min":0.875},"min":{"1min":0.437,"5min":0.437,"15min":0.34999999999999998},"max":{"1min":1.49,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.1599999999999999,"15min":1.046},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.5750000000000002,"5min":5.5750000000000002,"15min":5.5750000000000002},"last":0.93899999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1439999999999999,"15min":1.095},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1190000000000002,"5min":3.1190000000000002,"15min":4.008},"last":0.999},{"interface":"front","average":{"1min":1.333,"5min":1.2869999999999999,"15min":1.119},"min":{"1min":0.46400000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.833,"5min":1.542,"15min":1.228},"min":{"1min":0.71799999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":13.292,"5min":13.292,"15min":13.292},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.6499999999999999,"15min":1.365},"min":{"1min":0.82499999999999996,"5min":0.80500000000000005,"15min":0.71699999999999997},"max":{"1min":8.9299999999999997,"5min":11.068,"15min":11.068},"last":1.302}]},{"osd":5,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.5940000000000001,"15min":1.367},"min":{"1min":0.75700000000000001,"5min":0.71399999999999997,"15min":0.67900000000000005},"max":{"1min":4.6189999999999998,"5min":12.791,"15min":12.791},"last":2.3599999999999999},{"interface":"front","average":{"1min":2.0950000000000002,"5min":1.6970000000000001,"15min":1.4710000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":7.0039999999999996,"5min":12.106,"15min":12.106},"last":2.113}]},{"osd":6,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.621,"15min":1.375},"min":{"1min":0.84199999999999997,"5min":0.60899999999999999,"15min":0.58199999999999996},"max":{"1min":2.3460000000000001,"5min":8.1039999999999992,"15min":8.1039999999999992},"last":1.768},{"interface":"front","average":{"1min":1.5,"5min":1.6899999999999999,"15min":1.466},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":4.0309999999999997,"5min":6.891,"15min":6.891},"last":1.409}]},{"osd":7,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.6910000000000001,"15min":1.4870000000000001},"min":{"1min":1.081,"5min":0.79100000000000004,"15min":0.755},"max":{"1min":5.6269999999999998,"5min":5.6269999999999998,"15min":5.6269999999999998},"last":1.268},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.659,"15min":1.4770000000000001},"min":{"1min":0.91100000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":6.4119999999999999,"5min":7.6159999999999997,"15min":18.846},"last":2.3199999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542244,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45944,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325896,"statfs":{"total":96636764160,"available":96589717504,"internally_reserved":0,"allocated":897024,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97099999999999997,"15min":0.91600000000000004},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5129999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.137,"15min":0.98699999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":11.85,"5min":11.85,"15min":11.85},"last":0.61599999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.087,"15min":1.0349999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.3220000000000001,"5min":3.798,"15min":3.798},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.2230000000000001,"15min":1.052},"min":{"1min":0.46000000000000002,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":7.7770000000000001,"5min":7.7770000000000001,"15min":7.7770000000000001},"last":0.68799999999999994}]},{"osd":3,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1200000000000001,"15min":1.002},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.8809999999999998,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.246,"15min":1.147},"min":{"1min":0.499,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":3.399,"5min":8.4610000000000003,"15min":8.4610000000000003},"last":0.73599999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.6200000000000001,"15min":1.4219999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":4.8490000000000002,"5min":11.843,"15min":11.843},"last":1.016},{"interface":"front","average":{"1min":1.9219999999999999,"5min":1.639,"15min":1.444},"min":{"1min":0.65400000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":5.4740000000000002,"5min":11.768000000000001,"15min":11.768000000000001},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.714,"15min":1.4730000000000001},"min":{"1min":0.627,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":9.5169999999999995,"5min":26.501000000000001,"15min":26.501000000000001},"last":1.147},{"interface":"front","average":{"1min":1.5,"5min":1.417,"15min":1.351},"min":{"1min":0.68500000000000005,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.3010000000000002,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":1.179}]},{"osd":6,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.5980000000000001,"15min":1.4430000000000001},"min":{"1min":0.79200000000000004,"5min":0.438,"15min":0.438},"max":{"1min":4.1360000000000001,"5min":15.144,"15min":15.144},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.617,"5min":2.383,"15min":1.716},"min":{"1min":0.73099999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":5.7380000000000004,"5min":66.061999999999998,"15min":66.061999999999998},"last":1.0860000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.712,"15min":1.5449999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":11.243,"5min":11.243,"15min":11.243},"last":1.26},{"interface":"front","average":{"1min":2.1230000000000002,"5min":1.6759999999999999,"15min":1.526},"min":{"1min":0.85799999999999998,"5min":0.5,"15min":0.5},"max":{"1min":11.271000000000001,"5min":11.271000000000001,"15min":11.271000000000001},"last":1.105}]}]},{"osd":1,"up_from":14,"seq":60129542244,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50120,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321720,"statfs":{"total":96636764160,"available":96585441280,"internally_reserved":0,"allocated":905216,"data_stored":376068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.96899999999999997,"15min":0.93999999999999995},"min":{"1min":0.42699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8200000000000001,"5min":1.2689999999999999,"15min":1.093},"min":{"1min":0.59699999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":13.789,"5min":13.789,"15min":13.789},"last":1.363}]},{"osd":2,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":0.96799999999999997},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.2269999999999999,"5min":14.817,"15min":14.817},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.655,"5min":1.1819999999999999,"15min":1.04},"min":{"1min":0.71399999999999997,"5min":0.47199999999999998,"15min":0.41499999999999998},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":1.0549999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.143,"15min":1.0960000000000001},"min":{"1min":0.82099999999999995,"5min":0.437,"15min":0.437},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.4199999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.3580000000000001,"15min":1.125},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":13.722,"5min":20.439,"15min":20.439},"last":1.214}]},{"osd":4,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":2.0819999999999999,"5min":1.6719999999999999,"15min":1.3420000000000001},"min":{"1min":0.68400000000000005,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":16.097999999999999,"5min":16.097999999999999,"15min":16.097999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.635,"15min":1.379},"min":{"1min":0.74199999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.609,"5min":13.789999999999999,"15min":13.789999999999999},"last":1.3999999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.881,"5min":1.631,"15min":1.472},"min":{"1min":0.63600000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":8.7300000000000004,"5min":14.129,"15min":14.129},"last":1.504},{"interface":"front","average":{"1min":2.1360000000000001,"5min":1.978,"15min":1.613},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":8.7449999999999992,"5min":35.747,"15min":35.747},"last":1.4379999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3809999999999998,"5min":1.8009999999999999,"15min":1.5549999999999999},"min":{"1min":0.95299999999999996,"5min":0.629,"15min":0.629},"max":{"1min":16.231000000000002,"5min":16.231000000000002,"15min":16.231000000000002},"last":1.536},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.6499999999999999,"15min":1.494},"min":{"1min":0.70699999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":5.4809999999999999,"5min":13.56,"15min":13.56},"last":1.139}]},{"osd":7,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":3.3500000000000001,"5min":2.0419999999999998,"15min":1.613},"min":{"1min":1.103,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":21.902999999999999,"5min":21.902999999999999,"15min":21.902999999999999},"last":1.1839999999999999},{"interface":"front","average":{"1min":2.5609999999999999,"5min":1.9790000000000001,"15min":1.6160000000000001},"min":{"1min":1.1459999999999999,"5min":0.68400000000000005,"15min":0.64700000000000002},"max":{"1min":16.343,"5min":31.044,"15min":31.044},"last":1.591}]}]},{"osd":0,"up_from":14,"seq":60129542245,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54832,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317008,"statfs":{"total":96636764160,"available":96580616192,"internally_reserved":0,"allocated":1478656,"data_stored":964208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.1799999999999999,"15min":1.044},"min":{"1min":0.26400000000000001,"5min":0.25,"15min":0.25},"max":{"1min":22.619,"5min":22.619,"15min":22.619},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.4419999999999999,"15min":1.1399999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.6499999999999999,"5min":17.876000000000001,"15min":17.876000000000001},"last":1.6499999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.603,"15min":1.2749999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.8490000000000002,"5min":17.760999999999999,"15min":17.760999999999999},"last":1.4950000000000001},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.5509999999999999,"15min":1.349},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":22.672999999999998,"5min":22.672999999999998,"15min":22.672999999999998},"last":1.099}]},{"osd":3,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.504,"15min":1.1000000000000001},"min":{"1min":0.44700000000000001,"5min":0.42599999999999999,"15min":0.39200000000000002},"max":{"1min":3.0600000000000001,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.03},{"interface":"front","average":{"1min":1.323,"5min":1.585,"15min":1.226},"min":{"1min":0.46999999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.7410000000000001,"5min":17.981000000000002,"15min":17.981000000000002},"last":1.2}]},{"osd":4,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":2.5529999999999999,"5min":1.7689999999999999,"15min":1.5760000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":22.780000000000001,"5min":22.780000000000001,"15min":22.780000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.9610000000000001,"15min":1.512},"min":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":3.085,"5min":19.422000000000001,"15min":19.422000000000001},"last":1.1439999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":2.4500000000000002,"15min":1.8140000000000001},"min":{"1min":0.62,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":22.744,"5min":44.930999999999997,"15min":44.930999999999997},"last":1.272},{"interface":"front","average":{"1min":2.1070000000000002,"5min":2.056,"15min":1.548},"min":{"1min":0.69999999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":14.52,"5min":19.315000000000001,"15min":19.315000000000001},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.395,"15min":1.736},"min":{"1min":0.71799999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":3.681,"5min":27.442,"15min":27.442},"last":1.593},{"interface":"front","average":{"1min":2.6880000000000002,"5min":2.4289999999999998,"15min":1.8640000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":22.709,"5min":27.411999999999999,"15min":27.411999999999999},"last":1.5509999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.8819999999999999,"15min":1.613},"min":{"1min":0.71999999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.1669999999999998,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":2.919,"5min":1.9990000000000001,"15min":1.748},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":22.815000000000001,"5min":22.815000000000001,"15min":22.815000000000001},"last":1.6839999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:05:12.175 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:05:12.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.450+0000 7fcc5b200640 1 Processor -- start 2024-01-16T15:05:12.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.450+0000 7fcc5b200640 1 -- start start 2024-01-16T15:05:12.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5b200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcc540cc500 0x7fcc540cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5b200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcc5406d160 0x7fcc5406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5b200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc54070e50 con 0x7fcc54143c40 2024-01-16T15:05:12.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5b200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc540723c0 con 0x7fcc5406d160 2024-01-16T15:05:12.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5b200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc540cced0 con 0x7fcc540cc500 2024-01-16T15:05:12.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcc5406d160 0x7fcc5406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcc5406d160 0x7fcc5406d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46604/0 (socket says 172.21.15.141:46604) 2024-01-16T15:05:12.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.452+0000 7fcc5a800640 1 -- 172.21.15.141:0/3294711093 learned_addr learned my addr 172.21.15.141:0/3294711093 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:12.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.453+0000 7fcc59400640 1 --2- 172.21.15.141:0/3294711093 >> v2:172.21.15.188:3300/0 conn(0x7fcc540cc500 0x7fcc540cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.453+0000 7fcc5a800640 1 -- 172.21.15.141:0/3294711093 >> v2:172.21.15.188:3300/0 conn(0x7fcc540cc500 msgr2=0x7fcc540cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.453+0000 7fcc5a800640 1 --2- 172.21.15.141:0/3294711093 >> v2:172.21.15.188:3300/0 conn(0x7fcc540cc500 0x7fcc540cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.453+0000 7fcc5a800640 1 -- 172.21.15.141:0/3294711093 >> v1:172.21.15.188:6789/0 conn(0x7fcc54143c40 legacy=0x7fcc54071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.453+0000 7fcc5a800640 1 -- 172.21.15.141:0/3294711093 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54144020 con 0x7fcc5406d160 2024-01-16T15:05:12.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.453+0000 7fcc59400640 1 --2- 172.21.15.141:0/3294711093 >> v2:172.21.15.188:3300/0 conn(0x7fcc540cc500 0x7fcc540cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:12.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.454+0000 7fcc5a800640 1 --2- 172.21.15.141:0/3294711093 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc5406d160 0x7fcc5406d560 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcc540b44a0 tx=0x7fcc4c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe5511b50cc4fe2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.454+0000 7fcc58a00640 1 -- 172.21.15.141:0/3294711093 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c02c9f0 con 0x7fcc5406d160 2024-01-16T15:05:12.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.454+0000 7fcc58a00640 1 -- 172.21.15.141:0/3294711093 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c02cb50 con 0x7fcc5406d160 2024-01-16T15:05:12.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.454+0000 7fcc58a00640 1 -- 172.21.15.141:0/3294711093 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0343f0 con 0x7fcc5406d160 2024-01-16T15:05:12.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 -- 172.21.15.141:0/3294711093 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc5406d160 msgr2=0x7fcc5406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 --2- 172.21.15.141:0/3294711093 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc5406d160 0x7fcc5406d560 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcc540b44a0 tx=0x7fcc4c02be50 comp rx=0 tx=0).stop 2024-01-16T15:05:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 -- 172.21.15.141:0/3294711093 shutdown_connections 2024-01-16T15:05:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 --2- 172.21.15.141:0/3294711093 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc5406d160 0x7fcc5406d560 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 --2- 172.21.15.141:0/3294711093 >> v2:172.21.15.188:3300/0 conn(0x7fcc540cc500 0x7fcc540cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 -- 172.21.15.141:0/3294711093 >> 172.21.15.141:0/3294711093 conn(0x7fcc5406a800 msgr2=0x7fcc5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 -- 172.21.15.141:0/3294711093 shutdown_connections 2024-01-16T15:05:12.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.455+0000 7fcc5b200640 1 -- 172.21.15.141:0/3294711093 wait complete. 2024-01-16T15:05:12.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.456+0000 7fcc5b200640 1 Processor -- start 2024-01-16T15:05:12.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.457+0000 7fcc5b200640 1 -- start start 2024-01-16T15:05:12.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc5b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc5406d160 0x7fcc5413aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc5b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc540cc500 0x7fcc54152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc5b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc54143c40 0x7fcc54152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc5b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54156b50 con 0x7fcc540cc500 2024-01-16T15:05:12.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc5b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54153640 con 0x7fcc54143c40 2024-01-16T15:05:12.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc5b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54153920 con 0x7fcc5406d160 2024-01-16T15:05:12.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc540cc500 0x7fcc54152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc540cc500 0x7fcc54152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46612/0 (socket says 172.21.15.141:46612) 2024-01-16T15:05:12.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 -- 172.21.15.141:0/1929624051 learned_addr learned my addr 172.21.15.141:0/1929624051 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 -- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc5406d160 msgr2=0x7fcc5413aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59e00640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc5406d160 0x7fcc5413aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc5406d160 0x7fcc5413aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 -- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc54143c40 msgr2=0x7fcc54152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc54143c40 0x7fcc54152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.458+0000 7fcc59400640 1 -- 172.21.15.141:0/1929624051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c009b30 con 0x7fcc540cc500 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.459+0000 7fcc59400640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc540cc500 0x7fcc54152aa0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c7f0 tx=0x7fcc3c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.459+0000 7fcc52000640 1 -- 172.21.15.141:0/1929624051 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c007a30 con 0x7fcc540cc500 2024-01-16T15:05:12.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.459+0000 7fcc52000640 1 -- 172.21.15.141:0/1929624051 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c007b90 con 0x7fcc540cc500 2024-01-16T15:05:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.459+0000 7fcc52000640 1 -- 172.21.15.141:0/1929624051 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c00ce40 con 0x7fcc540cc500 2024-01-16T15:05:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.459+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5413a6e0 con 0x7fcc540cc500 2024-01-16T15:05:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.459+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5413a300 con 0x7fcc540cc500 2024-01-16T15:05:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.460+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5413ac00 con 0x7fcc540cc500 2024-01-16T15:05:12.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.461+0000 7fcc52000640 1 -- 172.21.15.141:0/1929624051 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fcc3c0113f0 con 0x7fcc540cc500 2024-01-16T15:05:12.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.462+0000 7fcc52000640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc2c06bbc0 0x7fcc2c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.463+0000 7fcc52000640 1 -- 172.21.15.141:0/1929624051 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc3c0885a0 con 0x7fcc540cc500 2024-01-16T15:05:12.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.463+0000 7fcc59e00640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc2c06bbc0 0x7fcc2c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.464+0000 7fcc59e00640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc2c06bbc0 0x7fcc2c06e080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400a750 tx=0x7fcc44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.467+0000 7fcc52000640 1 -- 172.21.15.141:0/1929624051 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc5413ac00 con 0x7fcc540cc500 2024-01-16T15:05:12.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.657+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc5406c960 con 0x7fcc2c06bbc0 2024-01-16T15:05:12.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.680+0000 7fcc52000640 1 -- 172.21.15.141:0/1929624051 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287043 (secure 0 0 0) 0x7fcc4400c520 con 0x7fcc2c06bbc0 2024-01-16T15:05:12.663 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:12.664 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc2c06bbc0 msgr2=0x7fcc2c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc2c06bbc0 0x7fcc2c06e080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400a750 tx=0x7fcc44008040 comp rx=0 tx=0).stop 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc540cc500 msgr2=0x7fcc54152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc540cc500 0x7fcc54152aa0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c7f0 tx=0x7fcc3c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 shutdown_connections 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc2c06bbc0 0x7fcc2c06e080 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc54143c40 0x7fcc54152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc540cc500 0x7fcc54152aa0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 --2- 172.21.15.141:0/1929624051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc5406d160 0x7fcc5413aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 >> 172.21.15.141:0/1929624051 conn(0x7fcc5406a800 msgr2=0x7fcc5406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 shutdown_connections 2024-01-16T15:05:12.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.686+0000 7fcc5b200640 1 -- 172.21.15.141:0/1929624051 wait complete. 2024-01-16T15:05:12.685 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-01-16T15:05:12.657203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":406140,"kb_used_data":8136,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754568580,"statfs":{"total":773094113280,"available":772678225920,"internally_reserved":0,"allocated":8331264,"data_stored":4176276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":407424562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002280"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660742+0000","last_change":"2024-01-16T15:05:07.649890+0000","last_active":"2024-01-16T15:05:08.660742+0000","last_peered":"2024-01-16T15:05:08.660742+0000","last_clean":"2024-01-16T15:05:08.660742+0000","last_became_active":"2024-01-16T15:05:07.649259+0000","last_became_peered":"2024-01-16T15:05:07.649259+0000","last_unstale":"2024-01-16T15:05:08.660742+0000","last_undegraded":"2024-01-16T15:05:08.660742+0000","last_fullsized":"2024-01-16T15:05:08.660742+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:55.884731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.663008+0000","last_change":"2024-01-16T15:05:07.654329+0000","last_active":"2024-01-16T15:05:08.663008+0000","last_peered":"2024-01-16T15:05:08.663008+0000","last_clean":"2024-01-16T15:05:08.663008+0000","last_became_active":"2024-01-16T15:05:07.653331+0000","last_became_peered":"2024-01-16T15:05:07.653331+0000","last_unstale":"2024-01-16T15:05:08.663008+0000","last_undegraded":"2024-01-16T15:05:08.663008+0000","last_fullsized":"2024-01-16T15:05:08.663008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:21.803688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657894+0000","last_change":"2024-01-16T15:05:07.643528+0000","last_active":"2024-01-16T15:05:08.657894+0000","last_peered":"2024-01-16T15:05:08.657894+0000","last_clean":"2024-01-16T15:05:08.657894+0000","last_became_active":"2024-01-16T15:05:07.643139+0000","last_became_peered":"2024-01-16T15:05:07.643139+0000","last_unstale":"2024-01-16T15:05:08.657894+0000","last_undegraded":"2024-01-16T15:05:08.657894+0000","last_fullsized":"2024-01-16T15:05:08.657894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:36.347300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657148+0000","last_change":"2024-01-16T15:05:07.654528+0000","last_active":"2024-01-16T15:05:08.657148+0000","last_peered":"2024-01-16T15:05:08.657148+0000","last_clean":"2024-01-16T15:05:08.657148+0000","last_became_active":"2024-01-16T15:05:07.654179+0000","last_became_peered":"2024-01-16T15:05:07.654179+0000","last_unstale":"2024-01-16T15:05:08.657148+0000","last_undegraded":"2024-01-16T15:05:08.657148+0000","last_fullsized":"2024-01-16T15:05:08.657148+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:56.887399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618911+0000","last_change":"2024-01-16T15:05:07.641683+0000","last_active":"2024-01-16T15:05:08.618911+0000","last_peered":"2024-01-16T15:05:08.618911+0000","last_clean":"2024-01-16T15:05:08.618911+0000","last_became_active":"2024-01-16T15:05:07.641337+0000","last_became_peered":"2024-01-16T15:05:07.641337+0000","last_unstale":"2024-01-16T15:05:08.618911+0000","last_undegraded":"2024-01-16T15:05:08.618911+0000","last_fullsized":"2024-01-16T15:05:08.618911+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:16:27.179259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661509+0000","last_change":"2024-01-16T15:05:07.652652+0000","last_active":"2024-01-16T15:05:08.661509+0000","last_peered":"2024-01-16T15:05:08.661509+0000","last_clean":"2024-01-16T15:05:08.661509+0000","last_became_active":"2024-01-16T15:05:07.652064+0000","last_became_peered":"2024-01-16T15:05:07.652064+0000","last_unstale":"2024-01-16T15:05:08.661509+0000","last_undegraded":"2024-01-16T15:05:08.661509+0000","last_fullsized":"2024-01-16T15:05:08.661509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:24.430819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624644+0000","last_change":"2024-01-16T15:05:07.648445+0000","last_active":"2024-01-16T15:05:08.624644+0000","last_peered":"2024-01-16T15:05:08.624644+0000","last_clean":"2024-01-16T15:05:08.624644+0000","last_became_active":"2024-01-16T15:05:07.648130+0000","last_became_peered":"2024-01-16T15:05:07.648130+0000","last_unstale":"2024-01-16T15:05:08.624644+0000","last_undegraded":"2024-01-16T15:05:08.624644+0000","last_fullsized":"2024-01-16T15:05:08.624644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:58.669334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661092+0000","last_change":"2024-01-16T15:05:07.650119+0000","last_active":"2024-01-16T15:05:08.661092+0000","last_peered":"2024-01-16T15:05:08.661092+0000","last_clean":"2024-01-16T15:05:08.661092+0000","last_became_active":"2024-01-16T15:05:07.648004+0000","last_became_peered":"2024-01-16T15:05:07.648004+0000","last_unstale":"2024-01-16T15:05:08.661092+0000","last_undegraded":"2024-01-16T15:05:08.661092+0000","last_fullsized":"2024-01-16T15:05:08.661092+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:59:58.825941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619455+0000","last_change":"2024-01-16T15:05:07.647170+0000","last_active":"2024-01-16T15:05:08.619455+0000","last_peered":"2024-01-16T15:05:08.619455+0000","last_clean":"2024-01-16T15:05:08.619455+0000","last_became_active":"2024-01-16T15:05:07.646804+0000","last_became_peered":"2024-01-16T15:05:07.646804+0000","last_unstale":"2024-01-16T15:05:08.619455+0000","last_undegraded":"2024-01-16T15:05:08.619455+0000","last_fullsized":"2024-01-16T15:05:08.619455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:33.438472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624235+0000","last_change":"2024-01-16T15:05:07.642257+0000","last_active":"2024-01-16T15:05:08.624235+0000","last_peered":"2024-01-16T15:05:08.624235+0000","last_clean":"2024-01-16T15:05:08.624235+0000","last_became_active":"2024-01-16T15:05:07.641661+0000","last_became_peered":"2024-01-16T15:05:07.641661+0000","last_unstale":"2024-01-16T15:05:08.624235+0000","last_undegraded":"2024-01-16T15:05:08.624235+0000","last_fullsized":"2024-01-16T15:05:08.624235+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:35.167922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660057+0000","last_change":"2024-01-16T15:05:07.645773+0000","last_active":"2024-01-16T15:05:08.660057+0000","last_peered":"2024-01-16T15:05:08.660057+0000","last_clean":"2024-01-16T15:05:08.660057+0000","last_became_active":"2024-01-16T15:05:07.645497+0000","last_became_peered":"2024-01-16T15:05:07.645497+0000","last_unstale":"2024-01-16T15:05:08.660057+0000","last_undegraded":"2024-01-16T15:05:08.660057+0000","last_fullsized":"2024-01-16T15:05:08.660057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:44.091571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657302+0000","last_change":"2024-01-16T15:05:07.651944+0000","last_active":"2024-01-16T15:05:08.657302+0000","last_peered":"2024-01-16T15:05:08.657302+0000","last_clean":"2024-01-16T15:05:08.657302+0000","last_became_active":"2024-01-16T15:05:07.648623+0000","last_became_peered":"2024-01-16T15:05:07.648623+0000","last_unstale":"2024-01-16T15:05:08.657302+0000","last_undegraded":"2024-01-16T15:05:08.657302+0000","last_fullsized":"2024-01-16T15:05:08.657302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:09.921566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658519+0000","last_change":"2024-01-16T15:05:07.648626+0000","last_active":"2024-01-16T15:05:08.658519+0000","last_peered":"2024-01-16T15:05:08.658519+0000","last_clean":"2024-01-16T15:05:08.658519+0000","last_became_active":"2024-01-16T15:05:07.648309+0000","last_became_peered":"2024-01-16T15:05:07.648309+0000","last_unstale":"2024-01-16T15:05:08.658519+0000","last_undegraded":"2024-01-16T15:05:08.658519+0000","last_fullsized":"2024-01-16T15:05:08.658519+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:57.810277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619336+0000","last_change":"2024-01-16T15:05:07.643349+0000","last_active":"2024-01-16T15:05:08.619336+0000","last_peered":"2024-01-16T15:05:08.619336+0000","last_clean":"2024-01-16T15:05:08.619336+0000","last_became_active":"2024-01-16T15:05:07.642551+0000","last_became_peered":"2024-01-16T15:05:07.642551+0000","last_unstale":"2024-01-16T15:05:08.619336+0000","last_undegraded":"2024-01-16T15:05:08.619336+0000","last_fullsized":"2024-01-16T15:05:08.619336+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:10.042586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658073+0000","last_change":"2024-01-16T15:05:07.638815+0000","last_active":"2024-01-16T15:05:08.658073+0000","last_peered":"2024-01-16T15:05:08.658073+0000","last_clean":"2024-01-16T15:05:08.658073+0000","last_became_active":"2024-01-16T15:05:07.638376+0000","last_became_peered":"2024-01-16T15:05:07.638376+0000","last_unstale":"2024-01-16T15:05:08.658073+0000","last_undegraded":"2024-01-16T15:05:08.658073+0000","last_fullsized":"2024-01-16T15:05:08.658073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:45.444669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.623213+0000","last_change":"2024-01-16T15:05:07.652910+0000","last_active":"2024-01-16T15:05:08.623213+0000","last_peered":"2024-01-16T15:05:08.623213+0000","last_clean":"2024-01-16T15:05:08.623213+0000","last_became_active":"2024-01-16T15:05:07.652587+0000","last_became_peered":"2024-01-16T15:05:07.652587+0000","last_unstale":"2024-01-16T15:05:08.623213+0000","last_undegraded":"2024-01-16T15:05:08.623213+0000","last_fullsized":"2024-01-16T15:05:08.623213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:44.196268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619589+0000","last_change":"2024-01-16T15:05:07.641174+0000","last_active":"2024-01-16T15:05:08.619589+0000","last_peered":"2024-01-16T15:05:08.619589+0000","last_clean":"2024-01-16T15:05:08.619589+0000","last_became_active":"2024-01-16T15:05:07.640875+0000","last_became_peered":"2024-01-16T15:05:07.640875+0000","last_unstale":"2024-01-16T15:05:08.619589+0000","last_undegraded":"2024-01-16T15:05:08.619589+0000","last_fullsized":"2024-01-16T15:05:08.619589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:35.988314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658490+0000","last_change":"2024-01-16T15:05:07.653752+0000","last_active":"2024-01-16T15:05:08.658490+0000","last_peered":"2024-01-16T15:05:08.658490+0000","last_clean":"2024-01-16T15:05:08.658490+0000","last_became_active":"2024-01-16T15:05:07.653429+0000","last_became_peered":"2024-01-16T15:05:07.653429+0000","last_unstale":"2024-01-16T15:05:08.658490+0000","last_undegraded":"2024-01-16T15:05:08.658490+0000","last_fullsized":"2024-01-16T15:05:08.658490+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:31.816016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662230+0000","last_change":"2024-01-16T15:05:07.651677+0000","last_active":"2024-01-16T15:05:08.662230+0000","last_peered":"2024-01-16T15:05:08.662230+0000","last_clean":"2024-01-16T15:05:08.662230+0000","last_became_active":"2024-01-16T15:05:07.650806+0000","last_became_peered":"2024-01-16T15:05:07.650806+0000","last_unstale":"2024-01-16T15:05:08.662230+0000","last_undegraded":"2024-01-16T15:05:08.662230+0000","last_fullsized":"2024-01-16T15:05:08.662230+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:49.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658746+0000","last_change":"2024-01-16T15:05:07.640768+0000","last_active":"2024-01-16T15:05:08.658746+0000","last_peered":"2024-01-16T15:05:08.658746+0000","last_clean":"2024-01-16T15:05:08.658746+0000","last_became_active":"2024-01-16T15:05:07.640370+0000","last_became_peered":"2024-01-16T15:05:07.640370+0000","last_unstale":"2024-01-16T15:05:08.658746+0000","last_undegraded":"2024-01-16T15:05:08.658746+0000","last_fullsized":"2024-01-16T15:05:08.658746+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:28.543197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625957+0000","last_change":"2024-01-16T15:05:07.638852+0000","last_active":"2024-01-16T15:05:08.625957+0000","last_peered":"2024-01-16T15:05:08.625957+0000","last_clean":"2024-01-16T15:05:08.625957+0000","last_became_active":"2024-01-16T15:05:07.638503+0000","last_became_peered":"2024-01-16T15:05:07.638503+0000","last_unstale":"2024-01-16T15:05:08.625957+0000","last_undegraded":"2024-01-16T15:05:08.625957+0000","last_fullsized":"2024-01-16T15:05:08.625957+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:43.049772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624936+0000","last_change":"2024-01-16T15:05:07.641347+0000","last_active":"2024-01-16T15:05:08.624936+0000","last_peered":"2024-01-16T15:05:08.624936+0000","last_clean":"2024-01-16T15:05:08.624936+0000","last_became_active":"2024-01-16T15:05:07.641073+0000","last_became_peered":"2024-01-16T15:05:07.641073+0000","last_unstale":"2024-01-16T15:05:08.624936+0000","last_undegraded":"2024-01-16T15:05:08.624936+0000","last_fullsized":"2024-01-16T15:05:08.624936+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:56.704683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659618+0000","last_change":"2024-01-16T15:05:07.650341+0000","last_active":"2024-01-16T15:05:08.659618+0000","last_peered":"2024-01-16T15:05:08.659618+0000","last_clean":"2024-01-16T15:05:08.659618+0000","last_became_active":"2024-01-16T15:05:07.648155+0000","last_became_peered":"2024-01-16T15:05:07.648155+0000","last_unstale":"2024-01-16T15:05:08.659618+0000","last_undegraded":"2024-01-16T15:05:08.659618+0000","last_fullsized":"2024-01-16T15:05:08.659618+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:29.064858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659176+0000","last_change":"2024-01-16T15:05:07.655789+0000","last_active":"2024-01-16T15:05:08.659176+0000","last_peered":"2024-01-16T15:05:08.659176+0000","last_clean":"2024-01-16T15:05:08.659176+0000","last_became_active":"2024-01-16T15:05:07.655398+0000","last_became_peered":"2024-01-16T15:05:07.655398+0000","last_unstale":"2024-01-16T15:05:08.659176+0000","last_undegraded":"2024-01-16T15:05:08.659176+0000","last_fullsized":"2024-01-16T15:05:08.659176+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:55.916665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626638+0000","last_change":"2024-01-16T15:05:07.645589+0000","last_active":"2024-01-16T15:05:08.626638+0000","last_peered":"2024-01-16T15:05:08.626638+0000","last_clean":"2024-01-16T15:05:08.626638+0000","last_became_active":"2024-01-16T15:05:07.645195+0000","last_became_peered":"2024-01-16T15:05:07.645195+0000","last_unstale":"2024-01-16T15:05:08.626638+0000","last_undegraded":"2024-01-16T15:05:08.626638+0000","last_fullsized":"2024-01-16T15:05:08.626638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:30.059276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626139+0000","last_change":"2024-01-16T15:05:07.653869+0000","last_active":"2024-01-16T15:05:08.626139+0000","last_peered":"2024-01-16T15:05:08.626139+0000","last_clean":"2024-01-16T15:05:08.626139+0000","last_became_active":"2024-01-16T15:05:07.652770+0000","last_became_peered":"2024-01-16T15:05:07.652770+0000","last_unstale":"2024-01-16T15:05:08.626139+0000","last_undegraded":"2024-01-16T15:05:08.626139+0000","last_fullsized":"2024-01-16T15:05:08.626139+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:39.011373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657896+0000","last_change":"2024-01-16T15:05:07.646050+0000","last_active":"2024-01-16T15:05:08.657896+0000","last_peered":"2024-01-16T15:05:08.657896+0000","last_clean":"2024-01-16T15:05:08.657896+0000","last_became_active":"2024-01-16T15:05:07.645793+0000","last_became_peered":"2024-01-16T15:05:07.645793+0000","last_unstale":"2024-01-16T15:05:08.657896+0000","last_undegraded":"2024-01-16T15:05:08.657896+0000","last_fullsized":"2024-01-16T15:05:08.657896+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:59.114071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624987+0000","last_change":"2024-01-16T15:05:07.649348+0000","last_active":"2024-01-16T15:05:08.624987+0000","last_peered":"2024-01-16T15:05:08.624987+0000","last_clean":"2024-01-16T15:05:08.624987+0000","last_became_active":"2024-01-16T15:05:07.647907+0000","last_became_peered":"2024-01-16T15:05:07.647907+0000","last_unstale":"2024-01-16T15:05:08.624987+0000","last_undegraded":"2024-01-16T15:05:08.624987+0000","last_fullsized":"2024-01-16T15:05:08.624987+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:35:32.652965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657399+0000","last_change":"2024-01-16T15:05:07.645281+0000","last_active":"2024-01-16T15:05:08.657399+0000","last_peered":"2024-01-16T15:05:08.657399+0000","last_clean":"2024-01-16T15:05:08.657399+0000","last_became_active":"2024-01-16T15:05:07.644943+0000","last_became_peered":"2024-01-16T15:05:07.644943+0000","last_unstale":"2024-01-16T15:05:08.657399+0000","last_undegraded":"2024-01-16T15:05:08.657399+0000","last_fullsized":"2024-01-16T15:05:08.657399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:54.884845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624445+0000","last_change":"2024-01-16T15:05:07.653231+0000","last_active":"2024-01-16T15:05:08.624445+0000","last_peered":"2024-01-16T15:05:08.624445+0000","last_clean":"2024-01-16T15:05:08.624445+0000","last_became_active":"2024-01-16T15:05:07.652905+0000","last_became_peered":"2024-01-16T15:05:07.652905+0000","last_unstale":"2024-01-16T15:05:08.624445+0000","last_undegraded":"2024-01-16T15:05:08.624445+0000","last_fullsized":"2024-01-16T15:05:08.624445+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:53.194876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625027+0000","last_change":"2024-01-16T15:05:07.640337+0000","last_active":"2024-01-16T15:05:08.625027+0000","last_peered":"2024-01-16T15:05:08.625027+0000","last_clean":"2024-01-16T15:05:08.625027+0000","last_became_active":"2024-01-16T15:05:07.640001+0000","last_became_peered":"2024-01-16T15:05:07.640001+0000","last_unstale":"2024-01-16T15:05:08.625027+0000","last_undegraded":"2024-01-16T15:05:08.625027+0000","last_fullsized":"2024-01-16T15:05:08.625027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:17.307990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661293+0000","last_change":"2024-01-16T15:05:07.648931+0000","last_active":"2024-01-16T15:05:08.661293+0000","last_peered":"2024-01-16T15:05:08.661293+0000","last_clean":"2024-01-16T15:05:08.661293+0000","last_became_active":"2024-01-16T15:05:07.648676+0000","last_became_peered":"2024-01-16T15:05:07.648676+0000","last_unstale":"2024-01-16T15:05:08.661293+0000","last_undegraded":"2024-01-16T15:05:08.661293+0000","last_fullsized":"2024-01-16T15:05:08.661293+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:19.807808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660378+0000","last_change":"2024-01-16T15:05:07.655655+0000","last_active":"2024-01-16T15:05:08.660378+0000","last_peered":"2024-01-16T15:05:08.660378+0000","last_clean":"2024-01-16T15:05:08.660378+0000","last_became_active":"2024-01-16T15:05:07.655308+0000","last_became_peered":"2024-01-16T15:05:07.655308+0000","last_unstale":"2024-01-16T15:05:08.660378+0000","last_undegraded":"2024-01-16T15:05:08.660378+0000","last_fullsized":"2024-01-16T15:05:08.660378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:28.628641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"72'4","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661477+0000","last_change":"2024-01-16T15:05:06.760194+0000","last_active":"2024-01-16T15:05:08.661477+0000","last_peered":"2024-01-16T15:05:08.661477+0000","last_clean":"2024-01-16T15:05:08.661477+0000","last_became_active":"2024-01-16T15:05:06.756007+0000","last_became_peered":"2024-01-16T15:05:06.756007+0000","last_unstale":"2024-01-16T15:05:08.661477+0000","last_undegraded":"2024-01-16T15:05:08.661477+0000","last_fullsized":"2024-01-16T15:05:08.661477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:16.292371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619626+0000","last_change":"2024-01-16T15:05:07.649592+0000","last_active":"2024-01-16T15:05:08.619626+0000","last_peered":"2024-01-16T15:05:08.619626+0000","last_clean":"2024-01-16T15:05:08.619626+0000","last_became_active":"2024-01-16T15:05:07.647227+0000","last_became_peered":"2024-01-16T15:05:07.647227+0000","last_unstale":"2024-01-16T15:05:08.619626+0000","last_undegraded":"2024-01-16T15:05:08.619626+0000","last_fullsized":"2024-01-16T15:05:08.619626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:02.272988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659171+0000","last_change":"2024-01-16T15:05:06.757917+0000","last_active":"2024-01-16T15:05:08.659171+0000","last_peered":"2024-01-16T15:05:08.659171+0000","last_clean":"2024-01-16T15:05:08.659171+0000","last_became_active":"2024-01-16T15:05:06.756979+0000","last_became_peered":"2024-01-16T15:05:06.756979+0000","last_unstale":"2024-01-16T15:05:08.659171+0000","last_undegraded":"2024-01-16T15:05:08.659171+0000","last_fullsized":"2024-01-16T15:05:08.659171+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:14.780391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662377+0000","last_change":"2024-01-16T15:05:07.658327+0000","last_active":"2024-01-16T15:05:08.662377+0000","last_peered":"2024-01-16T15:05:08.662377+0000","last_clean":"2024-01-16T15:05:08.662377+0000","last_became_active":"2024-01-16T15:05:07.658001+0000","last_became_peered":"2024-01-16T15:05:07.658001+0000","last_unstale":"2024-01-16T15:05:08.662377+0000","last_undegraded":"2024-01-16T15:05:08.662377+0000","last_fullsized":"2024-01-16T15:05:08.662377+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:59:09.184565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624377+0000","last_change":"2024-01-16T15:05:06.775062+0000","last_active":"2024-01-16T15:05:08.624377+0000","last_peered":"2024-01-16T15:05:08.624377+0000","last_clean":"2024-01-16T15:05:08.624377+0000","last_became_active":"2024-01-16T15:05:06.774476+0000","last_became_peered":"2024-01-16T15:05:06.774476+0000","last_unstale":"2024-01-16T15:05:08.624377+0000","last_undegraded":"2024-01-16T15:05:08.624377+0000","last_fullsized":"2024-01-16T15:05:08.624377+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:04.884120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660283+0000","last_change":"2024-01-16T15:05:07.649774+0000","last_active":"2024-01-16T15:05:08.660283+0000","last_peered":"2024-01-16T15:05:08.660283+0000","last_clean":"2024-01-16T15:05:08.660283+0000","last_became_active":"2024-01-16T15:05:07.649406+0000","last_became_peered":"2024-01-16T15:05:07.649406+0000","last_unstale":"2024-01-16T15:05:08.660283+0000","last_undegraded":"2024-01-16T15:05:08.660283+0000","last_fullsized":"2024-01-16T15:05:08.660283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:20:50.207173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662699+0000","last_change":"2024-01-16T15:05:06.772357+0000","last_active":"2024-01-16T15:05:08.662699+0000","last_peered":"2024-01-16T15:05:08.662699+0000","last_clean":"2024-01-16T15:05:08.662699+0000","last_became_active":"2024-01-16T15:05:06.772041+0000","last_became_peered":"2024-01-16T15:05:06.772041+0000","last_unstale":"2024-01-16T15:05:08.662699+0000","last_undegraded":"2024-01-16T15:05:08.662699+0000","last_fullsized":"2024-01-16T15:05:08.662699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:46.784398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624520+0000","last_change":"2024-01-16T15:05:07.646253+0000","last_active":"2024-01-16T15:05:08.624520+0000","last_peered":"2024-01-16T15:05:08.624520+0000","last_clean":"2024-01-16T15:05:08.624520+0000","last_became_active":"2024-01-16T15:05:07.645944+0000","last_became_peered":"2024-01-16T15:05:07.645944+0000","last_unstale":"2024-01-16T15:05:08.624520+0000","last_undegraded":"2024-01-16T15:05:08.624520+0000","last_fullsized":"2024-01-16T15:05:08.624520+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:17:23.087309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657698+0000","last_change":"2024-01-16T15:05:06.756923+0000","last_active":"2024-01-16T15:05:08.657698+0000","last_peered":"2024-01-16T15:05:08.657698+0000","last_clean":"2024-01-16T15:05:08.657698+0000","last_became_active":"2024-01-16T15:05:06.756258+0000","last_became_peered":"2024-01-16T15:05:06.756258+0000","last_unstale":"2024-01-16T15:05:08.657698+0000","last_undegraded":"2024-01-16T15:05:08.657698+0000","last_fullsized":"2024-01-16T15:05:08.657698+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:20.678593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625931+0000","last_change":"2024-01-16T15:05:07.645620+0000","last_active":"2024-01-16T15:05:08.625931+0000","last_peered":"2024-01-16T15:05:08.625931+0000","last_clean":"2024-01-16T15:05:08.625931+0000","last_became_active":"2024-01-16T15:05:07.644959+0000","last_became_peered":"2024-01-16T15:05:07.644959+0000","last_unstale":"2024-01-16T15:05:08.625931+0000","last_undegraded":"2024-01-16T15:05:08.625931+0000","last_fullsized":"2024-01-16T15:05:08.625931+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:19:57.069028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661843+0000","last_change":"2024-01-16T15:05:06.774132+0000","last_active":"2024-01-16T15:05:08.661843+0000","last_peered":"2024-01-16T15:05:08.661843+0000","last_clean":"2024-01-16T15:05:08.661843+0000","last_became_active":"2024-01-16T15:05:06.773226+0000","last_became_peered":"2024-01-16T15:05:06.773226+0000","last_unstale":"2024-01-16T15:05:08.661843+0000","last_undegraded":"2024-01-16T15:05:08.661843+0000","last_fullsized":"2024-01-16T15:05:08.661843+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:38.352691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618669+0000","last_change":"2024-01-16T15:05:07.637661+0000","last_active":"2024-01-16T15:05:08.618669+0000","last_peered":"2024-01-16T15:05:08.618669+0000","last_clean":"2024-01-16T15:05:08.618669+0000","last_became_active":"2024-01-16T15:05:07.636002+0000","last_became_peered":"2024-01-16T15:05:07.636002+0000","last_unstale":"2024-01-16T15:05:08.618669+0000","last_undegraded":"2024-01-16T15:05:08.618669+0000","last_fullsized":"2024-01-16T15:05:08.618669+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:10.357784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657610+0000","last_change":"2024-01-16T15:05:06.756400+0000","last_active":"2024-01-16T15:05:08.657610+0000","last_peered":"2024-01-16T15:05:08.657610+0000","last_clean":"2024-01-16T15:05:08.657610+0000","last_became_active":"2024-01-16T15:05:06.756116+0000","last_became_peered":"2024-01-16T15:05:06.756116+0000","last_unstale":"2024-01-16T15:05:08.657610+0000","last_undegraded":"2024-01-16T15:05:08.657610+0000","last_fullsized":"2024-01-16T15:05:08.657610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:02.951844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658245+0000","last_change":"2024-01-16T15:05:07.641802+0000","last_active":"2024-01-16T15:05:08.658245+0000","last_peered":"2024-01-16T15:05:08.658245+0000","last_clean":"2024-01-16T15:05:08.658245+0000","last_became_active":"2024-01-16T15:05:07.641511+0000","last_became_peered":"2024-01-16T15:05:07.641511+0000","last_unstale":"2024-01-16T15:05:08.658245+0000","last_undegraded":"2024-01-16T15:05:08.658245+0000","last_fullsized":"2024-01-16T15:05:08.658245+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:56.551650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618707+0000","last_change":"2024-01-16T15:05:06.755310+0000","last_active":"2024-01-16T15:05:08.618707+0000","last_peered":"2024-01-16T15:05:08.618707+0000","last_clean":"2024-01-16T15:05:08.618707+0000","last_became_active":"2024-01-16T15:05:06.755007+0000","last_became_peered":"2024-01-16T15:05:06.755007+0000","last_unstale":"2024-01-16T15:05:08.618707+0000","last_undegraded":"2024-01-16T15:05:08.618707+0000","last_fullsized":"2024-01-16T15:05:08.618707+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:41.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626342+0000","last_change":"2024-01-16T15:05:07.646074+0000","last_active":"2024-01-16T15:05:08.626342+0000","last_peered":"2024-01-16T15:05:08.626342+0000","last_clean":"2024-01-16T15:05:08.626342+0000","last_became_active":"2024-01-16T15:05:07.645773+0000","last_became_peered":"2024-01-16T15:05:07.645773+0000","last_unstale":"2024-01-16T15:05:08.626342+0000","last_undegraded":"2024-01-16T15:05:08.626342+0000","last_fullsized":"2024-01-16T15:05:08.626342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:10.411121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624798+0000","last_change":"2024-01-16T15:05:06.762652+0000","last_active":"2024-01-16T15:05:08.624798+0000","last_peered":"2024-01-16T15:05:08.624798+0000","last_clean":"2024-01-16T15:05:08.624798+0000","last_became_active":"2024-01-16T15:05:06.762146+0000","last_became_peered":"2024-01-16T15:05:06.762146+0000","last_unstale":"2024-01-16T15:05:08.624798+0000","last_undegraded":"2024-01-16T15:05:08.624798+0000","last_fullsized":"2024-01-16T15:05:08.624798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:16.330927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660806+0000","last_change":"2024-01-16T15:05:07.654491+0000","last_active":"2024-01-16T15:05:08.660806+0000","last_peered":"2024-01-16T15:05:08.660806+0000","last_clean":"2024-01-16T15:05:08.660806+0000","last_became_active":"2024-01-16T15:05:07.654101+0000","last_became_peered":"2024-01-16T15:05:07.654101+0000","last_unstale":"2024-01-16T15:05:08.660806+0000","last_undegraded":"2024-01-16T15:05:08.660806+0000","last_fullsized":"2024-01-16T15:05:08.660806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:31.830075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.663156+0000","last_change":"2024-01-16T15:05:06.773075+0000","last_active":"2024-01-16T15:05:08.663156+0000","last_peered":"2024-01-16T15:05:08.663156+0000","last_clean":"2024-01-16T15:05:08.663156+0000","last_became_active":"2024-01-16T15:05:06.772805+0000","last_became_peered":"2024-01-16T15:05:06.772805+0000","last_unstale":"2024-01-16T15:05:08.663156+0000","last_undegraded":"2024-01-16T15:05:08.663156+0000","last_fullsized":"2024-01-16T15:05:08.663156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:57.728311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662808+0000","last_change":"2024-01-16T15:05:07.656689+0000","last_active":"2024-01-16T15:05:08.662808+0000","last_peered":"2024-01-16T15:05:08.662808+0000","last_clean":"2024-01-16T15:05:08.662808+0000","last_became_active":"2024-01-16T15:05:07.656465+0000","last_became_peered":"2024-01-16T15:05:07.656465+0000","last_unstale":"2024-01-16T15:05:08.662808+0000","last_undegraded":"2024-01-16T15:05:08.662808+0000","last_fullsized":"2024-01-16T15:05:08.662808+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:52:03.997553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"72'3","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658021+0000","last_change":"2024-01-16T15:05:06.769573+0000","last_active":"2024-01-16T15:05:08.658021+0000","last_peered":"2024-01-16T15:05:08.658021+0000","last_clean":"2024-01-16T15:05:08.658021+0000","last_became_active":"2024-01-16T15:05:06.769264+0000","last_became_peered":"2024-01-16T15:05:06.769264+0000","last_unstale":"2024-01-16T15:05:08.658021+0000","last_undegraded":"2024-01-16T15:05:08.658021+0000","last_fullsized":"2024-01-16T15:05:08.658021+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:36.817166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619404+0000","last_change":"2024-01-16T15:05:07.643330+0000","last_active":"2024-01-16T15:05:08.619404+0000","last_peered":"2024-01-16T15:05:08.619404+0000","last_clean":"2024-01-16T15:05:08.619404+0000","last_became_active":"2024-01-16T15:05:07.641982+0000","last_became_peered":"2024-01-16T15:05:07.641982+0000","last_unstale":"2024-01-16T15:05:08.619404+0000","last_undegraded":"2024-01-16T15:05:08.619404+0000","last_fullsized":"2024-01-16T15:05:08.619404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:40.803835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"72'6","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657571+0000","last_change":"2024-01-16T15:05:06.752220+0000","last_active":"2024-01-16T15:05:08.657571+0000","last_peered":"2024-01-16T15:05:08.657571+0000","last_clean":"2024-01-16T15:05:08.657571+0000","last_became_active":"2024-01-16T15:05:06.751384+0000","last_became_peered":"2024-01-16T15:05:06.751384+0000","last_unstale":"2024-01-16T15:05:08.657571+0000","last_undegraded":"2024-01-16T15:05:08.657571+0000","last_fullsized":"2024-01-16T15:05:08.657571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:28:05.268275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659362+0000","last_change":"2024-01-16T15:05:07.640593+0000","last_active":"2024-01-16T15:05:08.659362+0000","last_peered":"2024-01-16T15:05:08.659362+0000","last_clean":"2024-01-16T15:05:08.659362+0000","last_became_active":"2024-01-16T15:05:07.640107+0000","last_became_peered":"2024-01-16T15:05:07.640107+0000","last_unstale":"2024-01-16T15:05:08.659362+0000","last_undegraded":"2024-01-16T15:05:08.659362+0000","last_fullsized":"2024-01-16T15:05:08.659362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:39.011373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657939+0000","last_change":"2024-01-16T15:05:06.751216+0000","last_active":"2024-01-16T15:05:08.657939+0000","last_peered":"2024-01-16T15:05:08.657939+0000","last_clean":"2024-01-16T15:05:08.657939+0000","last_became_active":"2024-01-16T15:05:06.749321+0000","last_became_peered":"2024-01-16T15:05:06.749321+0000","last_unstale":"2024-01-16T15:05:08.657939+0000","last_undegraded":"2024-01-16T15:05:08.657939+0000","last_fullsized":"2024-01-16T15:05:08.657939+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:34.393559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625552+0000","last_change":"2024-01-16T15:05:07.641958+0000","last_active":"2024-01-16T15:05:08.625552+0000","last_peered":"2024-01-16T15:05:08.625552+0000","last_clean":"2024-01-16T15:05:08.625552+0000","last_became_active":"2024-01-16T15:05:07.641680+0000","last_became_peered":"2024-01-16T15:05:07.641680+0000","last_unstale":"2024-01-16T15:05:08.625552+0000","last_undegraded":"2024-01-16T15:05:08.625552+0000","last_fullsized":"2024-01-16T15:05:08.625552+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:55.916665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661823+0000","last_change":"2024-01-16T15:05:06.772979+0000","last_active":"2024-01-16T15:05:08.661823+0000","last_peered":"2024-01-16T15:05:08.661823+0000","last_clean":"2024-01-16T15:05:08.661823+0000","last_became_active":"2024-01-16T15:05:06.772665+0000","last_became_peered":"2024-01-16T15:05:06.772665+0000","last_unstale":"2024-01-16T15:05:08.661823+0000","last_undegraded":"2024-01-16T15:05:08.661823+0000","last_fullsized":"2024-01-16T15:05:08.661823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:32.194251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657572+0000","last_change":"2024-01-16T15:05:07.654495+0000","last_active":"2024-01-16T15:05:08.657572+0000","last_peered":"2024-01-16T15:05:08.657572+0000","last_clean":"2024-01-16T15:05:08.657572+0000","last_became_active":"2024-01-16T15:05:07.653970+0000","last_became_peered":"2024-01-16T15:05:07.653970+0000","last_unstale":"2024-01-16T15:05:08.657572+0000","last_undegraded":"2024-01-16T15:05:08.657572+0000","last_fullsized":"2024-01-16T15:05:08.657572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:40.099590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624728+0000","last_change":"2024-01-16T15:05:06.768865+0000","last_active":"2024-01-16T15:05:08.624728+0000","last_peered":"2024-01-16T15:05:08.624728+0000","last_clean":"2024-01-16T15:05:08.624728+0000","last_became_active":"2024-01-16T15:05:06.768409+0000","last_became_peered":"2024-01-16T15:05:06.768409+0000","last_unstale":"2024-01-16T15:05:08.624728+0000","last_undegraded":"2024-01-16T15:05:08.624728+0000","last_fullsized":"2024-01-16T15:05:08.624728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:43.180649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.618991+0000","last_change":"2024-01-16T15:05:07.639913+0000","last_active":"2024-01-16T15:05:08.618991+0000","last_peered":"2024-01-16T15:05:08.618991+0000","last_clean":"2024-01-16T15:05:08.618991+0000","last_became_active":"2024-01-16T15:05:07.639493+0000","last_became_peered":"2024-01-16T15:05:07.639493+0000","last_unstale":"2024-01-16T15:05:08.618991+0000","last_undegraded":"2024-01-16T15:05:08.618991+0000","last_fullsized":"2024-01-16T15:05:08.618991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:26.138458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"72'8","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.623492+0000","last_change":"2024-01-16T15:05:06.765582+0000","last_active":"2024-01-16T15:05:08.623492+0000","last_peered":"2024-01-16T15:05:08.623492+0000","last_clean":"2024-01-16T15:05:08.623492+0000","last_became_active":"2024-01-16T15:05:06.765282+0000","last_became_peered":"2024-01-16T15:05:06.765282+0000","last_unstale":"2024-01-16T15:05:08.623492+0000","last_undegraded":"2024-01-16T15:05:08.623492+0000","last_fullsized":"2024-01-16T15:05:08.623492+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:48:12.228401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624603+0000","last_change":"2024-01-16T15:05:07.644042+0000","last_active":"2024-01-16T15:05:08.624603+0000","last_peered":"2024-01-16T15:05:08.624603+0000","last_clean":"2024-01-16T15:05:08.624603+0000","last_became_active":"2024-01-16T15:05:07.642001+0000","last_became_peered":"2024-01-16T15:05:07.642001+0000","last_unstale":"2024-01-16T15:05:08.624603+0000","last_undegraded":"2024-01-16T15:05:08.624603+0000","last_fullsized":"2024-01-16T15:05:08.624603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:35.515237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662307+0000","last_change":"2024-01-16T15:05:06.773829+0000","last_active":"2024-01-16T15:05:08.662307+0000","last_peered":"2024-01-16T15:05:08.662307+0000","last_clean":"2024-01-16T15:05:08.662307+0000","last_became_active":"2024-01-16T15:05:06.773552+0000","last_became_peered":"2024-01-16T15:05:06.773552+0000","last_unstale":"2024-01-16T15:05:08.662307+0000","last_undegraded":"2024-01-16T15:05:08.662307+0000","last_fullsized":"2024-01-16T15:05:08.662307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:15:14.671873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661036+0000","last_change":"2024-01-16T15:05:07.650021+0000","last_active":"2024-01-16T15:05:08.661036+0000","last_peered":"2024-01-16T15:05:08.661036+0000","last_clean":"2024-01-16T15:05:08.661036+0000","last_became_active":"2024-01-16T15:05:07.649661+0000","last_became_peered":"2024-01-16T15:05:07.649661+0000","last_unstale":"2024-01-16T15:05:08.661036+0000","last_undegraded":"2024-01-16T15:05:08.661036+0000","last_fullsized":"2024-01-16T15:05:08.661036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:44.885273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624828+0000","last_change":"2024-01-16T15:05:06.773927+0000","last_active":"2024-01-16T15:05:08.624828+0000","last_peered":"2024-01-16T15:05:08.624828+0000","last_clean":"2024-01-16T15:05:08.624828+0000","last_became_active":"2024-01-16T15:05:06.772853+0000","last_became_peered":"2024-01-16T15:05:06.772853+0000","last_unstale":"2024-01-16T15:05:08.624828+0000","last_undegraded":"2024-01-16T15:05:08.624828+0000","last_fullsized":"2024-01-16T15:05:08.624828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:16.379742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657455+0000","last_change":"2024-01-16T15:05:07.648279+0000","last_active":"2024-01-16T15:05:08.657455+0000","last_peered":"2024-01-16T15:05:08.657455+0000","last_clean":"2024-01-16T15:05:08.657455+0000","last_became_active":"2024-01-16T15:05:07.645767+0000","last_became_peered":"2024-01-16T15:05:07.645767+0000","last_unstale":"2024-01-16T15:05:08.657455+0000","last_undegraded":"2024-01-16T15:05:08.657455+0000","last_fullsized":"2024-01-16T15:05:08.657455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:17.346546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"72'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662929+0000","last_change":"2024-01-16T15:05:06.772563+0000","last_active":"2024-01-16T15:05:08.662929+0000","last_peered":"2024-01-16T15:05:08.662929+0000","last_clean":"2024-01-16T15:05:08.662929+0000","last_became_active":"2024-01-16T15:05:06.772095+0000","last_became_peered":"2024-01-16T15:05:06.772095+0000","last_unstale":"2024-01-16T15:05:08.662929+0000","last_undegraded":"2024-01-16T15:05:08.662929+0000","last_fullsized":"2024-01-16T15:05:08.662929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:41.505008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619500+0000","last_change":"2024-01-16T15:05:07.636898+0000","last_active":"2024-01-16T15:05:08.619500+0000","last_peered":"2024-01-16T15:05:08.619500+0000","last_clean":"2024-01-16T15:05:08.619500+0000","last_became_active":"2024-01-16T15:05:07.636121+0000","last_became_peered":"2024-01-16T15:05:07.636121+0000","last_unstale":"2024-01-16T15:05:08.619500+0000","last_undegraded":"2024-01-16T15:05:08.619500+0000","last_fullsized":"2024-01-16T15:05:08.619500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:33.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659727+0000","last_change":"2024-01-16T15:05:06.761834+0000","last_active":"2024-01-16T15:05:08.659727+0000","last_peered":"2024-01-16T15:05:08.659727+0000","last_clean":"2024-01-16T15:05:08.659727+0000","last_became_active":"2024-01-16T15:05:06.761548+0000","last_became_peered":"2024-01-16T15:05:06.761548+0000","last_unstale":"2024-01-16T15:05:08.659727+0000","last_undegraded":"2024-01-16T15:05:08.659727+0000","last_fullsized":"2024-01-16T15:05:08.659727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:09.306688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657555+0000","last_change":"2024-01-16T15:05:07.642007+0000","last_active":"2024-01-16T15:05:08.657555+0000","last_peered":"2024-01-16T15:05:08.657555+0000","last_clean":"2024-01-16T15:05:08.657555+0000","last_became_active":"2024-01-16T15:05:07.640865+0000","last_became_peered":"2024-01-16T15:05:07.640865+0000","last_unstale":"2024-01-16T15:05:08.657555+0000","last_undegraded":"2024-01-16T15:05:08.657555+0000","last_fullsized":"2024-01-16T15:05:08.657555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:03.960723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662542+0000","last_change":"2024-01-16T15:05:06.772312+0000","last_active":"2024-01-16T15:05:08.662542+0000","last_peered":"2024-01-16T15:05:08.662542+0000","last_clean":"2024-01-16T15:05:08.662542+0000","last_became_active":"2024-01-16T15:05:06.771785+0000","last_became_peered":"2024-01-16T15:05:06.771785+0000","last_unstale":"2024-01-16T15:05:08.662542+0000","last_undegraded":"2024-01-16T15:05:08.662542+0000","last_fullsized":"2024-01-16T15:05:08.662542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:48.982295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657143+0000","last_change":"2024-01-16T15:05:07.651446+0000","last_active":"2024-01-16T15:05:08.657143+0000","last_peered":"2024-01-16T15:05:08.657143+0000","last_clean":"2024-01-16T15:05:08.657143+0000","last_became_active":"2024-01-16T15:05:07.650911+0000","last_became_peered":"2024-01-16T15:05:07.650911+0000","last_unstale":"2024-01-16T15:05:08.657143+0000","last_undegraded":"2024-01-16T15:05:08.657143+0000","last_fullsized":"2024-01-16T15:05:08.657143+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:54.752672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.624948+0000","last_change":"2024-01-16T15:05:06.768685+0000","last_active":"2024-01-16T15:05:08.624948+0000","last_peered":"2024-01-16T15:05:08.624948+0000","last_clean":"2024-01-16T15:05:08.624948+0000","last_became_active":"2024-01-16T15:05:06.768026+0000","last_became_peered":"2024-01-16T15:05:06.768026+0000","last_unstale":"2024-01-16T15:05:08.624948+0000","last_undegraded":"2024-01-16T15:05:08.624948+0000","last_fullsized":"2024-01-16T15:05:08.624948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:34.152302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657537+0000","last_change":"2024-01-16T15:05:07.653100+0000","last_active":"2024-01-16T15:05:08.657537+0000","last_peered":"2024-01-16T15:05:08.657537+0000","last_clean":"2024-01-16T15:05:08.657537+0000","last_became_active":"2024-01-16T15:05:07.652825+0000","last_became_peered":"2024-01-16T15:05:07.652825+0000","last_unstale":"2024-01-16T15:05:08.657537+0000","last_undegraded":"2024-01-16T15:05:08.657537+0000","last_fullsized":"2024-01-16T15:05:08.657537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:20.389514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657014+0000","last_change":"2024-01-16T15:05:06.759707+0000","last_active":"2024-01-16T15:05:08.657014+0000","last_peered":"2024-01-16T15:05:08.657014+0000","last_clean":"2024-01-16T15:05:08.657014+0000","last_became_active":"2024-01-16T15:05:06.759361+0000","last_became_peered":"2024-01-16T15:05:06.759361+0000","last_unstale":"2024-01-16T15:05:08.657014+0000","last_undegraded":"2024-01-16T15:05:08.657014+0000","last_fullsized":"2024-01-16T15:05:08.657014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:31.341354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659111+0000","last_change":"2024-01-16T15:05:07.639656+0000","last_active":"2024-01-16T15:05:08.659111+0000","last_peered":"2024-01-16T15:05:08.659111+0000","last_clean":"2024-01-16T15:05:08.659111+0000","last_became_active":"2024-01-16T15:05:07.639350+0000","last_became_peered":"2024-01-16T15:05:07.639350+0000","last_unstale":"2024-01-16T15:05:08.659111+0000","last_undegraded":"2024-01-16T15:05:08.659111+0000","last_fullsized":"2024-01-16T15:05:08.659111+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:59.016104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662557+0000","last_change":"2024-01-16T15:05:06.773600+0000","last_active":"2024-01-16T15:05:08.662557+0000","last_peered":"2024-01-16T15:05:08.662557+0000","last_clean":"2024-01-16T15:05:08.662557+0000","last_became_active":"2024-01-16T15:05:06.773237+0000","last_became_peered":"2024-01-16T15:05:06.773237+0000","last_unstale":"2024-01-16T15:05:08.662557+0000","last_undegraded":"2024-01-16T15:05:08.662557+0000","last_fullsized":"2024-01-16T15:05:08.662557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:38.629598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.625013+0000","last_change":"2024-01-16T15:05:07.643587+0000","last_active":"2024-01-16T15:05:08.625013+0000","last_peered":"2024-01-16T15:05:08.625013+0000","last_clean":"2024-01-16T15:05:08.625013+0000","last_became_active":"2024-01-16T15:05:07.643292+0000","last_became_peered":"2024-01-16T15:05:07.643292+0000","last_unstale":"2024-01-16T15:05:08.625013+0000","last_undegraded":"2024-01-16T15:05:08.625013+0000","last_fullsized":"2024-01-16T15:05:08.625013+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:10.322307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.626344+0000","last_change":"2024-01-16T15:05:06.765718+0000","last_active":"2024-01-16T15:05:08.626344+0000","last_peered":"2024-01-16T15:05:08.626344+0000","last_clean":"2024-01-16T15:05:08.626344+0000","last_became_active":"2024-01-16T15:05:06.765434+0000","last_became_peered":"2024-01-16T15:05:06.765434+0000","last_unstale":"2024-01-16T15:05:08.626344+0000","last_undegraded":"2024-01-16T15:05:08.626344+0000","last_fullsized":"2024-01-16T15:05:08.626344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:29.043656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.659582+0000","last_change":"2024-01-16T15:05:07.655917+0000","last_active":"2024-01-16T15:05:08.659582+0000","last_peered":"2024-01-16T15:05:08.659582+0000","last_clean":"2024-01-16T15:05:08.659582+0000","last_became_active":"2024-01-16T15:05:07.655367+0000","last_became_peered":"2024-01-16T15:05:07.655367+0000","last_unstale":"2024-01-16T15:05:08.659582+0000","last_undegraded":"2024-01-16T15:05:08.659582+0000","last_fullsized":"2024-01-16T15:05:08.659582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:10.411121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619563+0000","last_change":"2024-01-16T15:05:06.760719+0000","last_active":"2024-01-16T15:05:08.619563+0000","last_peered":"2024-01-16T15:05:08.619563+0000","last_clean":"2024-01-16T15:05:08.619563+0000","last_became_active":"2024-01-16T15:05:06.760358+0000","last_became_peered":"2024-01-16T15:05:06.760358+0000","last_unstale":"2024-01-16T15:05:08.619563+0000","last_undegraded":"2024-01-16T15:05:08.619563+0000","last_fullsized":"2024-01-16T15:05:08.619563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:43:52.397412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.660389+0000","last_change":"2024-01-16T15:05:07.642118+0000","last_active":"2024-01-16T15:05:08.660389+0000","last_peered":"2024-01-16T15:05:08.660389+0000","last_clean":"2024-01-16T15:05:08.660389+0000","last_became_active":"2024-01-16T15:05:07.641786+0000","last_became_peered":"2024-01-16T15:05:07.641786+0000","last_unstale":"2024-01-16T15:05:08.660389+0000","last_undegraded":"2024-01-16T15:05:08.660389+0000","last_fullsized":"2024-01-16T15:05:08.660389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:34.258858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.658850+0000","last_change":"2024-01-16T15:05:06.762726+0000","last_active":"2024-01-16T15:05:08.658850+0000","last_peered":"2024-01-16T15:05:08.658850+0000","last_clean":"2024-01-16T15:05:08.658850+0000","last_became_active":"2024-01-16T15:05:06.762342+0000","last_became_peered":"2024-01-16T15:05:06.762342+0000","last_unstale":"2024-01-16T15:05:08.658850+0000","last_undegraded":"2024-01-16T15:05:08.658850+0000","last_fullsized":"2024-01-16T15:05:08.658850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:49.831467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619362+0000","last_change":"2024-01-16T15:05:07.647293+0000","last_active":"2024-01-16T15:05:08.619362+0000","last_peered":"2024-01-16T15:05:08.619362+0000","last_clean":"2024-01-16T15:05:08.619362+0000","last_became_active":"2024-01-16T15:05:07.646767+0000","last_became_peered":"2024-01-16T15:05:07.646767+0000","last_unstale":"2024-01-16T15:05:08.619362+0000","last_undegraded":"2024-01-16T15:05:08.619362+0000","last_fullsized":"2024-01-16T15:05:08.619362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:06.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.619019+0000","last_change":"2024-01-16T15:05:06.768512+0000","last_active":"2024-01-16T15:05:08.619019+0000","last_peered":"2024-01-16T15:05:08.619019+0000","last_clean":"2024-01-16T15:05:08.619019+0000","last_became_active":"2024-01-16T15:05:06.768051+0000","last_became_peered":"2024-01-16T15:05:06.768051+0000","last_unstale":"2024-01-16T15:05:08.619019+0000","last_undegraded":"2024-01-16T15:05:08.619019+0000","last_fullsized":"2024-01-16T15:05:08.619019+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:17.266108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662819+0000","last_change":"2024-01-16T15:05:07.647077+0000","last_active":"2024-01-16T15:05:08.662819+0000","last_peered":"2024-01-16T15:05:08.662819+0000","last_clean":"2024-01-16T15:05:08.662819+0000","last_became_active":"2024-01-16T15:05:07.646718+0000","last_became_peered":"2024-01-16T15:05:07.646718+0000","last_unstale":"2024-01-16T15:05:08.662819+0000","last_undegraded":"2024-01-16T15:05:08.662819+0000","last_fullsized":"2024-01-16T15:05:08.662819+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:23.042966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657398+0000","last_change":"2024-01-16T15:05:06.757624+0000","last_active":"2024-01-16T15:05:08.657398+0000","last_peered":"2024-01-16T15:05:08.657398+0000","last_clean":"2024-01-16T15:05:08.657398+0000","last_became_active":"2024-01-16T15:05:06.757315+0000","last_became_peered":"2024-01-16T15:05:06.757315+0000","last_unstale":"2024-01-16T15:05:08.657398+0000","last_undegraded":"2024-01-16T15:05:08.657398+0000","last_fullsized":"2024-01-16T15:05:08.657398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:59.925173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657270+0000","last_change":"2024-01-16T15:05:07.649855+0000","last_active":"2024-01-16T15:05:08.657270+0000","last_peered":"2024-01-16T15:05:08.657270+0000","last_clean":"2024-01-16T15:05:08.657270+0000","last_became_active":"2024-01-16T15:05:07.649032+0000","last_became_peered":"2024-01-16T15:05:07.649032+0000","last_unstale":"2024-01-16T15:05:08.657270+0000","last_undegraded":"2024-01-16T15:05:08.657270+0000","last_fullsized":"2024-01-16T15:05:08.657270+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:12.397691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.623382+0000","last_change":"2024-01-16T15:05:06.764342+0000","last_active":"2024-01-16T15:05:08.623382+0000","last_peered":"2024-01-16T15:05:08.623382+0000","last_clean":"2024-01-16T15:05:08.623382+0000","last_became_active":"2024-01-16T15:05:06.762214+0000","last_became_peered":"2024-01-16T15:05:06.762214+0000","last_unstale":"2024-01-16T15:05:08.623382+0000","last_undegraded":"2024-01-16T15:05:08.623382+0000","last_fullsized":"2024-01-16T15:05:08.623382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:08.905947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661398+0000","last_change":"2024-01-16T15:05:07.649800+0000","last_active":"2024-01-16T15:05:08.661398+0000","last_peered":"2024-01-16T15:05:08.661398+0000","last_clean":"2024-01-16T15:05:08.661398+0000","last_became_active":"2024-01-16T15:05:07.649548+0000","last_became_peered":"2024-01-16T15:05:07.649548+0000","last_unstale":"2024-01-16T15:05:08.661398+0000","last_undegraded":"2024-01-16T15:05:08.661398+0000","last_fullsized":"2024-01-16T15:05:08.661398+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:47.475772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.661317+0000","last_change":"2024-01-16T15:05:06.772244+0000","last_active":"2024-01-16T15:05:08.661317+0000","last_peered":"2024-01-16T15:05:08.661317+0000","last_clean":"2024-01-16T15:05:08.661317+0000","last_became_active":"2024-01-16T15:05:06.771901+0000","last_became_peered":"2024-01-16T15:05:06.771901+0000","last_unstale":"2024-01-16T15:05:08.661317+0000","last_undegraded":"2024-01-16T15:05:08.661317+0000","last_fullsized":"2024-01-16T15:05:08.661317+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:39.079933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.657449+0000","last_change":"2024-01-16T15:05:07.651021+0000","last_active":"2024-01-16T15:05:08.657449+0000","last_peered":"2024-01-16T15:05:08.657449+0000","last_clean":"2024-01-16T15:05:08.657449+0000","last_became_active":"2024-01-16T15:05:07.650732+0000","last_became_peered":"2024-01-16T15:05:07.650732+0000","last_unstale":"2024-01-16T15:05:08.657449+0000","last_undegraded":"2024-01-16T15:05:08.657449+0000","last_fullsized":"2024-01-16T15:05:08.657449+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:06.720112+0000","last_clean_scrub_stamp":"2024-01-16T15:05:06.720112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:02.967841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"72'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-16T15:05:08.662128+0000","last_change":"2024-01-16T15:05:06.770836+0000","last_active":"2024-01-16T15:05:08.662128+0000","last_peered":"2024-01-16T15:05:08.662128+0000","last_clean":"2024-01-16T15:05:08.662128+0000","last_became_active":"2024-01-16T15:05:06.770311+0000","last_became_peered":"2024-01-16T15:05:06.770311+0000","last_unstale":"2024-01-16T15:05:08.662128+0000","last_undegraded":"2024-01-16T15:05:08.662128+0000","last_fullsized":"2024-01-16T15:05:08.662128+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:05.704493+0000","last_clean_scrub_stamp":"2024-01-16T15:05:05.704493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:43.218449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":216,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:05:07.628935+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:05:07.628935+0000","last_peered":"2024-01-16T15:05:07.628935+0000","last_clean":"2024-01-16T15:05:07.628935+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:05:07.628935+0000","last_undegraded":"2024-01-16T15:05:07.628935+0000","last_fullsized":"2024-01-16T15:05:07.628935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50684,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321156,"statfs":{"total":96636764160,"available":96584863744,"internally_reserved":0,"allocated":1486848,"data_stored":964824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.6379999999999999,"15min":1.256},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":8.2639999999999993,"5min":15.1,"15min":15.1},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.3240000000000001,"15min":1.2150000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.55400000000000005},"max":{"1min":1.879,"5min":3.5089999999999999,"15min":3.5089999999999999},"last":0.98999999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.611,"15min":1.226},"min":{"1min":0.79200000000000004,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":8.3740000000000006,"5min":14.988,"15min":14.988},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.351,"15min":1.2050000000000001},"min":{"1min":0.70199999999999996,"5min":0.64900000000000002,"15min":0.60599999999999998},"max":{"1min":1.619,"5min":8.3360000000000003,"15min":8.3360000000000003},"last":1.325}]},{"osd":2,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.3049999999999999,"15min":1.212},"min":{"1min":0.748,"5min":0.748,"15min":0.69499999999999995},"max":{"1min":1.6759999999999999,"5min":3.4569999999999999,"15min":3.4569999999999999},"last":1.482},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.6759999999999999,"15min":1.359},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":5.4610000000000003,"5min":20.577000000000002,"15min":20.577000000000002},"last":1.4550000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.9510000000000001,"15min":1.4970000000000001},"min":{"1min":0.73299999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":8.3360000000000003,"5min":15.231,"15min":15.231},"last":1.175},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.6060000000000001,"15min":1.4570000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.73199999999999998},"max":{"1min":4.367,"5min":10.141999999999999,"15min":10.141999999999999},"last":1.542}]},{"osd":4,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.6279999999999999,"15min":1.4419999999999999},"min":{"1min":0.80600000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.605,"5min":20.649999999999999,"15min":20.649999999999999},"last":1.254},{"interface":"front","average":{"1min":1.641,"5min":1.8540000000000001,"15min":1.5780000000000001},"min":{"1min":0.82599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":8.1349999999999998,"5min":15.515000000000001,"15min":15.515000000000001},"last":1.399}]},{"osd":5,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":2.9020000000000001,"5min":1.8129999999999999,"15min":1.625},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.52800000000000002},"max":{"1min":29.567,"5min":29.567,"15min":29.567},"last":1.6599999999999999},{"interface":"front","average":{"1min":2.9620000000000002,"5min":2.0659999999999998,"15min":1.768},"min":{"1min":0.92600000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":27.937000000000001,"5min":27.937000000000001,"15min":27.937000000000001},"last":1.512}]},{"osd":6,"last update":"Tue Jan 16 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":2.056,"15min":1.7330000000000001},"min":{"1min":0.63800000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.9330000000000001,"5min":41.206000000000003,"15min":41.206000000000003},"last":1.621},{"interface":"front","average":{"1min":1.8720000000000001,"5min":2.3140000000000001,"15min":1.833},"min":{"1min":0.72499999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":8.2219999999999995,"5min":21.908000000000001,"15min":21.908000000000001},"last":1.5740000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542244,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50100,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321740,"statfs":{"total":96636764160,"available":96585461760,"internally_reserved":0,"allocated":892928,"data_stored":373932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.4079999999999999,"15min":1.198},"min":{"1min":0.61299999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":4.6920000000000002,"5min":9.1890000000000001,"15min":9.1890000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.925,"5min":2.1379999999999999,"15min":1.52},"min":{"1min":0.51700000000000002,"5min":0.502,"15min":0.502},"max":{"1min":9.891,"5min":50.472000000000001,"15min":50.472000000000001},"last":1.617}]},{"osd":1,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5580000000000001,"15min":1.2110000000000001},"min":{"1min":0.76200000000000001,"5min":0.66300000000000003,"15min":0.624},"max":{"1min":4.2850000000000001,"5min":10.629,"15min":10.629},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.581,"5min":1.5329999999999999,"15min":1.423},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":4.2590000000000003,"5min":9.3279999999999994,"15min":9.3279999999999994},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":2.1030000000000002,"5min":2.0390000000000001,"15min":1.4830000000000001},"min":{"1min":0.59299999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":9.8140000000000001,"5min":50.643999999999998,"15min":50.643999999999998},"last":1.173},{"interface":"front","average":{"1min":1.9219999999999999,"5min":1.653,"15min":1.405},"min":{"1min":0.91900000000000004,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":6.5860000000000003,"5min":10.843,"15min":10.843},"last":1.6659999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":2.3490000000000002,"5min":1.7929999999999999,"15min":1.4990000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":8.6199999999999992,"5min":15.675000000000001,"15min":15.675000000000001},"last":1.105},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.63900000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":9.9770000000000003,"5min":50.561,"15min":50.561},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.3559999999999999,"15min":1.776},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":14.581,"5min":47.488,"15min":47.488},"last":1.3620000000000001},{"interface":"front","average":{"1min":2.8740000000000001,"5min":2.504,"15min":1.843},"min":{"1min":0.92900000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":11.295999999999999,"5min":50.381,"15min":50.381},"last":1.268}]},{"osd":5,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.74,"15min":1.5609999999999999},"min":{"1min":0.71699999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":4.3769999999999998,"5min":10.654,"15min":10.654},"last":1.4670000000000001},{"interface":"front","average":{"1min":2.0019999999999998,"5min":1.8919999999999999,"15min":1.7370000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":9.2949999999999999,"5min":9.3729999999999993,"15min":9.3729999999999993},"last":1.4410000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:18 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":2.702,"15min":2.0099999999999998},"min":{"1min":0.77200000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":18.481000000000002,"5min":52.912999999999997,"15min":52.912999999999997},"last":1.5189999999999999},{"interface":"front","average":{"1min":5.9390000000000001,"5min":2.698,"15min":2.0110000000000001},"min":{"1min":0.76200000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":18.899000000000001,"5min":18.899000000000001,"15min":18.899000000000001},"last":1.5740000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542244,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50100,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321740,"statfs":{"total":96636764160,"available":96585461760,"internally_reserved":0,"allocated":888832,"data_stored":373930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.48,"15min":1.2789999999999999},"min":{"1min":0.75800000000000001,"5min":0.496,"15min":0.496},"max":{"1min":8.298,"5min":13.938000000000001,"15min":13.938000000000001},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.6100000000000001,"15min":1.2529999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":11.895,"5min":16.896000000000001,"15min":16.896000000000001},"last":0.86099999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8109999999999999,"5min":1.6220000000000001,"15min":1.3859999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":11.797000000000001,"5min":13.279999999999999,"15min":13.279999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.8080000000000001,"15min":1.357},"min":{"1min":0.64200000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":4.8860000000000001,"5min":35.328000000000003,"15min":35.328000000000003},"last":1.6890000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.9119999999999999,"15min":1.478},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.9950000000000001,"5min":35.398000000000003,"15min":35.398000000000003},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.744,"5min":1.4410000000000001,"15min":1.3169999999999999},"min":{"1min":0.753,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":8.3510000000000009,"5min":8.3510000000000009,"15min":8.3510000000000009},"last":1.522}]},{"osd":3,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.8769999999999998,"5min":1.978,"15min":1.546},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":24.254000000000001,"5min":24.254000000000001,"15min":24.254000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":2.6869999999999998,"5min":2.2410000000000001,"15min":1.6739999999999999},"min":{"1min":0.78900000000000003,"5min":0.63,"15min":0.63},"max":{"1min":23.367999999999999,"5min":35.268000000000001,"15min":35.268000000000001},"last":1.8620000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.0289999999999999,"15min":1.6419999999999999},"min":{"1min":0.81000000000000005,"5min":0.45400000000000001,"15min":0.39000000000000001},"max":{"1min":14.695,"5min":15.757,"15min":15.757},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.865,"5min":2.1819999999999999,"15min":1.7589999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.45000000000000001},"max":{"1min":11.933999999999999,"5min":26.166,"15min":26.166},"last":1.2609999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":2.1619999999999999,"15min":1.7749999999999999},"min":{"1min":0.90400000000000003,"5min":0.53400000000000003,"15min":0.499},"max":{"1min":6.4349999999999996,"5min":35.427999999999997,"15min":35.427999999999997},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.994,"5min":1.972,"15min":1.7649999999999999},"min":{"1min":0.84199999999999997,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":8.4169999999999998,"5min":13.551,"15min":13.551},"last":2.23}]},{"osd":7,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.1360000000000001,"15min":1.788},"min":{"1min":0.73099999999999998,"5min":0.58099999999999996,"15min":0.41099999999999998},"max":{"1min":13.567,"5min":19.364999999999998,"15min":19.364999999999998},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.7290000000000001,"5min":2.1930000000000001,"15min":1.8700000000000001},"min":{"1min":0.88300000000000001,"5min":0.71699999999999997,"15min":0.42899999999999999},"max":{"1min":5.9039999999999999,"5min":35.363,"15min":35.363},"last":1.9890000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50096,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321744,"statfs":{"total":96636764160,"available":96585465856,"internally_reserved":0,"allocated":888832,"data_stored":373910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.2130000000000001,"5min":1.423,"15min":1.1910000000000001},"min":{"1min":0.78700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":11.647,"5min":11.647,"15min":11.647},"last":0.77200000000000002},{"interface":"front","average":{"1min":2.1930000000000001,"5min":1.605,"15min":1.3460000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":9.8599999999999994,"5min":9.8599999999999994,"15min":9.8599999999999994},"last":1.159}]},{"osd":1,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":1.679,"15min":1.347},"min":{"1min":0.72899999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.3919999999999995,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":1.417},{"interface":"front","average":{"1min":1.8300000000000001,"5min":1.5109999999999999,"15min":1.298},"min":{"1min":0.84199999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":9.1470000000000002,"5min":9.1470000000000002,"15min":9.1470000000000002},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.554,"15min":1.448},"min":{"1min":0.78600000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":12.733000000000001,"5min":12.733000000000001,"15min":15.986000000000001},"last":1.24},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.47,"15min":1.454},"min":{"1min":0.81399999999999995,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":9.2639999999999993,"5min":9.2639999999999993,"15min":9.2639999999999993},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.954,"15min":1.696},"min":{"1min":0.85899999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":7.3849999999999998,"5min":11.481,"15min":11.481},"last":1.3240000000000001},{"interface":"front","average":{"1min":2.2650000000000001,"5min":1.7509999999999999,"15min":1.528},"min":{"1min":0.88,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":7.7990000000000004,"5min":21.706,"15min":21.706},"last":1.4630000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.9079999999999999,"5min":1.4850000000000001,"15min":1.5720000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":9.3149999999999995,"5min":9.3149999999999995,"15min":9.3149999999999995},"last":1.0740000000000001},{"interface":"front","average":{"1min":2.0139999999999998,"5min":1.728,"15min":1.645},"min":{"1min":0.50700000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":9.7739999999999991,"5min":10.831,"15min":10.831},"last":1.5069999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":1.5269999999999999,"15min":1.575},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":7.4240000000000004,"5min":7.4240000000000004,"15min":7.4240000000000004},"last":1.5549999999999999},{"interface":"front","average":{"1min":2.1520000000000001,"5min":1.8560000000000001,"15min":1.752},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":9.8810000000000002,"5min":20.939,"15min":20.939},"last":1.2789999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:09 2024","interfaces":[{"interface":"back","average":{"1min":1.952,"5min":1.857,"15min":1.7549999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":7.6929999999999996,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":1.603},{"interface":"front","average":{"1min":2.8700000000000001,"5min":1.917,"15min":1.804},"min":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":20.277999999999999,"5min":20.277999999999999,"15min":20.277999999999999},"last":1.3660000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542244,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54264,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317576,"statfs":{"total":96636764160,"available":96581197824,"internally_reserved":0,"allocated":892928,"data_stored":373952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.044,"15min":0.93799999999999994},"min":{"1min":0.52400000000000002,"5min":0.43099999999999999,"15min":0.33600000000000002},"max":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.198,"15min":0.96599999999999997},"min":{"1min":0.50600000000000001,"5min":0.502,"15min":0.45000000000000001},"max":{"1min":1.9099999999999999,"5min":6.048,"15min":6.048},"last":1.093}]},{"osd":1,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.94899999999999995,"15min":0.875},"min":{"1min":0.437,"5min":0.437,"15min":0.34999999999999998},"max":{"1min":1.49,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.1599999999999999,"15min":1.046},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.5750000000000002,"5min":5.5750000000000002,"15min":5.5750000000000002},"last":0.93899999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1439999999999999,"15min":1.095},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1190000000000002,"5min":3.1190000000000002,"15min":4.008},"last":0.999},{"interface":"front","average":{"1min":1.333,"5min":1.2869999999999999,"15min":1.119},"min":{"1min":0.46400000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.833,"5min":1.542,"15min":1.228},"min":{"1min":0.71799999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":13.292,"5min":13.292,"15min":13.292},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.6499999999999999,"15min":1.365},"min":{"1min":0.82499999999999996,"5min":0.80500000000000005,"15min":0.71699999999999997},"max":{"1min":8.9299999999999997,"5min":11.068,"15min":11.068},"last":1.302}]},{"osd":5,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.5940000000000001,"15min":1.367},"min":{"1min":0.75700000000000001,"5min":0.71399999999999997,"15min":0.67900000000000005},"max":{"1min":4.6189999999999998,"5min":12.791,"15min":12.791},"last":2.3599999999999999},{"interface":"front","average":{"1min":2.0950000000000002,"5min":1.6970000000000001,"15min":1.4710000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":7.0039999999999996,"5min":12.106,"15min":12.106},"last":2.113}]},{"osd":6,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.621,"15min":1.375},"min":{"1min":0.84199999999999997,"5min":0.60899999999999999,"15min":0.58199999999999996},"max":{"1min":2.3460000000000001,"5min":8.1039999999999992,"15min":8.1039999999999992},"last":1.768},{"interface":"front","average":{"1min":1.5,"5min":1.6899999999999999,"15min":1.466},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":4.0309999999999997,"5min":6.891,"15min":6.891},"last":1.409}]},{"osd":7,"last update":"Tue Jan 16 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.6910000000000001,"15min":1.4870000000000001},"min":{"1min":1.081,"5min":0.79100000000000004,"15min":0.755},"max":{"1min":5.6269999999999998,"5min":5.6269999999999998,"15min":5.6269999999999998},"last":1.268},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.659,"15min":1.4770000000000001},"min":{"1min":0.91100000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":6.4119999999999999,"5min":7.6159999999999997,"15min":18.846},"last":2.3199999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542244,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45944,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325896,"statfs":{"total":96636764160,"available":96589717504,"internally_reserved":0,"allocated":897024,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97099999999999997,"15min":0.91600000000000004},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5129999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.137,"15min":0.98699999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":11.85,"5min":11.85,"15min":11.85},"last":0.61599999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.087,"15min":1.0349999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.3220000000000001,"5min":3.798,"15min":3.798},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.2230000000000001,"15min":1.052},"min":{"1min":0.46000000000000002,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":7.7770000000000001,"5min":7.7770000000000001,"15min":7.7770000000000001},"last":0.68799999999999994}]},{"osd":3,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1200000000000001,"15min":1.002},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.8809999999999998,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.246,"15min":1.147},"min":{"1min":0.499,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":3.399,"5min":8.4610000000000003,"15min":8.4610000000000003},"last":0.73599999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.6200000000000001,"15min":1.4219999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":4.8490000000000002,"5min":11.843,"15min":11.843},"last":1.016},{"interface":"front","average":{"1min":1.9219999999999999,"5min":1.639,"15min":1.444},"min":{"1min":0.65400000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":5.4740000000000002,"5min":11.768000000000001,"15min":11.768000000000001},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.714,"15min":1.4730000000000001},"min":{"1min":0.627,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":9.5169999999999995,"5min":26.501000000000001,"15min":26.501000000000001},"last":1.147},{"interface":"front","average":{"1min":1.5,"5min":1.417,"15min":1.351},"min":{"1min":0.68500000000000005,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.3010000000000002,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":1.179}]},{"osd":6,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.5980000000000001,"15min":1.4430000000000001},"min":{"1min":0.79200000000000004,"5min":0.438,"15min":0.438},"max":{"1min":4.1360000000000001,"5min":15.144,"15min":15.144},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.617,"5min":2.383,"15min":1.716},"min":{"1min":0.73099999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":5.7380000000000004,"5min":66.061999999999998,"15min":66.061999999999998},"last":1.0860000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.712,"15min":1.5449999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":11.243,"5min":11.243,"15min":11.243},"last":1.26},{"interface":"front","average":{"1min":2.1230000000000002,"5min":1.6759999999999999,"15min":1.526},"min":{"1min":0.85799999999999998,"5min":0.5,"15min":0.5},"max":{"1min":11.271000000000001,"5min":11.271000000000001,"15min":11.271000000000001},"last":1.105}]}]},{"osd":1,"up_from":14,"seq":60129542244,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50120,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321720,"statfs":{"total":96636764160,"available":96585441280,"internally_reserved":0,"allocated":905216,"data_stored":376068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.96899999999999997,"15min":0.93999999999999995},"min":{"1min":0.42699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8200000000000001,"5min":1.2689999999999999,"15min":1.093},"min":{"1min":0.59699999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":13.789,"5min":13.789,"15min":13.789},"last":1.363}]},{"osd":2,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":0.96799999999999997},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.2269999999999999,"5min":14.817,"15min":14.817},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.655,"5min":1.1819999999999999,"15min":1.04},"min":{"1min":0.71399999999999997,"5min":0.47199999999999998,"15min":0.41499999999999998},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":1.0549999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.143,"15min":1.0960000000000001},"min":{"1min":0.82099999999999995,"5min":0.437,"15min":0.437},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.4199999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.3580000000000001,"15min":1.125},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":13.722,"5min":20.439,"15min":20.439},"last":1.214}]},{"osd":4,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":2.0819999999999999,"5min":1.6719999999999999,"15min":1.3420000000000001},"min":{"1min":0.68400000000000005,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":16.097999999999999,"5min":16.097999999999999,"15min":16.097999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.635,"15min":1.379},"min":{"1min":0.74199999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.609,"5min":13.789999999999999,"15min":13.789999999999999},"last":1.3999999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":1.881,"5min":1.631,"15min":1.472},"min":{"1min":0.63600000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":8.7300000000000004,"5min":14.129,"15min":14.129},"last":1.504},{"interface":"front","average":{"1min":2.1360000000000001,"5min":1.978,"15min":1.613},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":8.7449999999999992,"5min":35.747,"15min":35.747},"last":1.4379999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3809999999999998,"5min":1.8009999999999999,"15min":1.5549999999999999},"min":{"1min":0.95299999999999996,"5min":0.629,"15min":0.629},"max":{"1min":16.231000000000002,"5min":16.231000000000002,"15min":16.231000000000002},"last":1.536},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.6499999999999999,"15min":1.494},"min":{"1min":0.70699999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":5.4809999999999999,"5min":13.56,"15min":13.56},"last":1.139}]},{"osd":7,"last update":"Tue Jan 16 15:04:16 2024","interfaces":[{"interface":"back","average":{"1min":3.3500000000000001,"5min":2.0419999999999998,"15min":1.613},"min":{"1min":1.103,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":21.902999999999999,"5min":21.902999999999999,"15min":21.902999999999999},"last":1.1839999999999999},{"interface":"front","average":{"1min":2.5609999999999999,"5min":1.9790000000000001,"15min":1.6160000000000001},"min":{"1min":1.1459999999999999,"5min":0.68400000000000005,"15min":0.64700000000000002},"max":{"1min":16.343,"5min":31.044,"15min":31.044},"last":1.591}]}]},{"osd":0,"up_from":14,"seq":60129542245,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54832,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317008,"statfs":{"total":96636764160,"available":96580616192,"internally_reserved":0,"allocated":1478656,"data_stored":964208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.1799999999999999,"15min":1.044},"min":{"1min":0.26400000000000001,"5min":0.25,"15min":0.25},"max":{"1min":22.619,"5min":22.619,"15min":22.619},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.4419999999999999,"15min":1.1399999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.6499999999999999,"5min":17.876000000000001,"15min":17.876000000000001},"last":1.6499999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.603,"15min":1.2749999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.8490000000000002,"5min":17.760999999999999,"15min":17.760999999999999},"last":1.4950000000000001},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.5509999999999999,"15min":1.349},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":22.672999999999998,"5min":22.672999999999998,"15min":22.672999999999998},"last":1.099}]},{"osd":3,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.504,"15min":1.1000000000000001},"min":{"1min":0.44700000000000001,"5min":0.42599999999999999,"15min":0.39200000000000002},"max":{"1min":3.0600000000000001,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.03},{"interface":"front","average":{"1min":1.323,"5min":1.585,"15min":1.226},"min":{"1min":0.46999999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.7410000000000001,"5min":17.981000000000002,"15min":17.981000000000002},"last":1.2}]},{"osd":4,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":2.5529999999999999,"5min":1.7689999999999999,"15min":1.5760000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":22.780000000000001,"5min":22.780000000000001,"15min":22.780000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.9610000000000001,"15min":1.512},"min":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":3.085,"5min":19.422000000000001,"15min":19.422000000000001},"last":1.1439999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":2.4500000000000002,"15min":1.8140000000000001},"min":{"1min":0.62,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":22.744,"5min":44.930999999999997,"15min":44.930999999999997},"last":1.272},{"interface":"front","average":{"1min":2.1070000000000002,"5min":2.056,"15min":1.548},"min":{"1min":0.69999999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":14.52,"5min":19.315000000000001,"15min":19.315000000000001},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.395,"15min":1.736},"min":{"1min":0.71799999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":3.681,"5min":27.442,"15min":27.442},"last":1.593},{"interface":"front","average":{"1min":2.6880000000000002,"5min":2.4289999999999998,"15min":1.8640000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":22.709,"5min":27.411999999999999,"15min":27.411999999999999},"last":1.5509999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:05:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.8819999999999999,"15min":1.613},"min":{"1min":0.71999999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.1669999999999998,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":2.919,"5min":1.9990000000000001,"15min":1.748},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":22.815000000000001,"5min":22.815000000000001,"15min":22.815000000000001},"last":1.6839999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:05:12.689 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:05:12.690 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:05:12.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.972+0000 7f23b0600640 1 Processor -- start 2024-01-16T15:05:12.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.972+0000 7f23b0600640 1 -- start start 2024-01-16T15:05:12.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.973+0000 7f23b0600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f23a8160fe0 0x7f23a8161470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.973+0000 7f23b0600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a8070bf0 con 0x7f23a80ce490 2024-01-16T15:05:12.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.973+0000 7f23b0600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80702e0 con 0x7f23a80b1f50 2024-01-16T15:05:12.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.973+0000 7f23b0600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a81619b0 con 0x7f23a8160fe0 2024-01-16T15:05:12.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.974+0000 7f23adc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f23a80ce490 0x7f23a8071080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36766/0 (socket says 172.21.15.141:36766) 2024-01-16T15:05:12.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.974+0000 7f23adc00640 1 -- 172.21.15.141:0/2750172091 learned_addr learned my addr 172.21.15.141:0/2750172091 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:12.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.974+0000 7f23ad200640 1 --2- 172.21.15.141:0/2750172091 >> v2:172.21.15.188:3300/0 conn(0x7f23a8160fe0 0x7f23a8161470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.974+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649307917 0 0) 0x7f23a8070bf0 con 0x7f23a80ce490 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.974+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2388003520 con 0x7f23a80ce490 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090321284 0 0) 0x7f2388003520 con 0x7f23a80ce490 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 >> v2:172.21.15.188:3300/0 conn(0x7f23a8160fe0 msgr2=0x7f23a8161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 --2- 172.21.15.141:0/2750172091 >> v2:172.21.15.188:3300/0 conn(0x7f23a8160fe0 0x7f23a8161470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 >> v1:172.21.15.188:6789/0 conn(0x7f23a80b1f50 legacy=0x7f23a80b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a80b2cc0 con 0x7f23a80ce490 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2398001670 con 0x7f23a80ce490 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 >> v1:172.21.15.141:6789/0 conn(0x7f23a80ce490 legacy=0x7f23a8071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2388003970 0x7f2388003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23880047c0 0x7f2388004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388005180 0x7f238800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a81619b0 con 0x7f2388003970 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238800c020 con 0x7f23880047c0 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238800c1e0 con 0x7f2388005180 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.975+0000 7f23adc00640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2388003970 0x7f2388003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23adc00640 1 -- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388005180 msgr2=0x7f238800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23ae600640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388005180 0x7f238800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23ad200640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23880047c0 0x7f2388004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23adc00640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388005180 0x7f238800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23adc00640 1 -- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23880047c0 msgr2=0x7f2388004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23adc00640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23880047c0 0x7f2388004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23adc00640 1 -- 172.21.15.141:0/2750172091 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2388003520 con 0x7f2388003970 2024-01-16T15:05:12.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.976+0000 7f23ae600640 1 -- 172.21.15.141:0/2750172091 reap_dead start 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.977+0000 7f23adc00640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2388003970 0x7f2388003d90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f239800c240 tx=0x7f23880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587ab6afdef6403b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.977+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398005ad0 con 0x7f2388003970 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.977+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2398005c30 con 0x7f2388003970 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.977+0000 7f23a7e00640 1 -- 172.21.15.141:0/2750172091 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239802caf0 con 0x7f2388003970 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.977+0000 7f23b0600640 1 -- 172.21.15.141:0/2750172091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2388003970 msgr2=0x7f2388003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.977+0000 7f23b0600640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2388003970 0x7f2388003d90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f239800c240 tx=0x7f23880057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.978+0000 7f23b0600640 1 -- 172.21.15.141:0/2750172091 shutdown_connections 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.978+0000 7f23b0600640 1 --2- 172.21.15.141:0/2750172091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2388003970 0x7f2388003d90 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.978+0000 7f23b0600640 1 -- 172.21.15.141:0/2750172091 >> 172.21.15.141:0/2750172091 conn(0x7f23a80b0be0 msgr2=0x7f23a80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.978+0000 7f23b0600640 1 -- 172.21.15.141:0/2750172091 shutdown_connections 2024-01-16T15:05:12.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.978+0000 7f23b0600640 1 -- 172.21.15.141:0/2750172091 wait complete. 2024-01-16T15:05:12.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.979+0000 7f23b0600640 1 Processor -- start 2024-01-16T15:05:12.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.980+0000 7f23b0600640 1 -- start start 2024-01-16T15:05:12.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.980+0000 7f23b0600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388003970 0x7f23a8156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.980+0000 7f23b0600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a8143050 0x7f23a81434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23b0600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a81596c0 0x7f23a8159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23b0600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a815a0b0 con 0x7f23a8143050 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23b0600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8142170 con 0x7f23a81596c0 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23b0600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a81454d0 con 0x7f2388003970 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a8143050 0x7f23a81434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a8143050 0x7f23a81434d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46632/0 (socket says 172.21.15.141:46632) 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 -- 172.21.15.141:0/812202603 learned_addr learned my addr 172.21.15.141:0/812202603 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 -- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388003970 msgr2=0x7f23a8156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388003970 0x7f23a8156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 -- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a81596c0 msgr2=0x7f23a8159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a81596c0 0x7f23a8159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 -- 172.21.15.141:0/812202603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a8143b80 con 0x7f23a8143050 2024-01-16T15:05:12.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.981+0000 7f23ad200640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a8143050 0x7f23a81434d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f23900108e0 tx=0x7f2390010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.982+0000 7f23a5600640 1 -- 172.21.15.141:0/812202603 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239000eb00 con 0x7f23a8143050 2024-01-16T15:05:12.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.982+0000 7f23a5600640 1 -- 172.21.15.141:0/812202603 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239000ec60 con 0x7f23a8143050 2024-01-16T15:05:12.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.982+0000 7f23a5600640 1 -- 172.21.15.141:0/812202603 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239001c400 con 0x7f23a8143050 2024-01-16T15:05:12.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.982+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a8143e60 con 0x7f23a8143050 2024-01-16T15:05:12.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.982+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a8144420 con 0x7f23a8143050 2024-01-16T15:05:12.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.983+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a8067670 con 0x7f23a8143050 2024-01-16T15:05:12.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.984+0000 7f23a5600640 1 -- 172.21.15.141:0/812202603 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f239001c560 con 0x7f23a8143050 2024-01-16T15:05:12.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.985+0000 7f23a5600640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006bbc0 0x7f238006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.985+0000 7f23a5600640 1 -- 172.21.15.141:0/812202603 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2390088a00 con 0x7f23a8143050 2024-01-16T15:05:12.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.985+0000 7f23adc00640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006bbc0 0x7f238006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.987+0000 7f23adc00640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006bbc0 0x7f238006e080 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2398034000 tx=0x7f2398039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:12.990+0000 7f23a5600640 1 -- 172.21.15.141:0/812202603 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2390055370 con 0x7f23a8143050 2024-01-16T15:05:13.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.251+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f23a80b18b0 con 0x7f23a8143050 2024-01-16T15:05:13.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.255+0000 7f23a5600640 1 -- 172.21.15.141:0/812202603 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2390058990 con 0x7f23a8143050 2024-01-16T15:05:13.234 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T15:05:13.234 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T15:05:13.234 INFO:teuthology.orchestra.run.smithi141.stdout: caps mds = "allow" 2024-01-16T15:05:13.234 INFO:teuthology.orchestra.run.smithi141.stdout: caps mon = "allow r" 2024-01-16T15:05:13.234 INFO:teuthology.orchestra.run.smithi141.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:05:13.235 INFO:teuthology.orchestra.run.smithi141.stderr:updated caps for client.0 2024-01-16T15:05:13.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006bbc0 msgr2=0x7f238006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006bbc0 0x7f238006e080 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2398034000 tx=0x7f2398039040 comp rx=0 tx=0).stop 2024-01-16T15:05:13.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a8143050 msgr2=0x7f23a81434d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a8143050 0x7f23a81434d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f23900108e0 tx=0x7f2390010da0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 shutdown_connections 2024-01-16T15:05:13.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006bbc0 0x7f238006e080 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a81596c0 0x7f23a8159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a8143050 0x7f23a81434d0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 --2- 172.21.15.141:0/812202603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2388003970 0x7f23a8156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 >> 172.21.15.141:0/812202603 conn(0x7f23a80b0be0 msgr2=0x7f23a80aeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:13.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.258+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 shutdown_connections 2024-01-16T15:05:13.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.259+0000 7f23b0600640 1 -- 172.21.15.141:0/812202603 wait complete. 2024-01-16T15:05:13.257 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:13.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.488+0000 7fbb27600640 1 Processor -- start 2024-01-16T15:05:13.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.489+0000 7fbb27600640 1 -- start start 2024-01-16T15:05:13.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.490+0000 7fbb27600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb2013df80 0x7fbb2015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.490+0000 7fbb27600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2006c470 con 0x7fbb2013e360 2024-01-16T15:05:13.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.490+0000 7fbb27600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2006c750 con 0x7fbb200cf9a0 2024-01-16T15:05:13.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.490+0000 7fbb27600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb2006ca30 con 0x7fbb2013df80 2024-01-16T15:05:13.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.490+0000 7fbb24c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb2013df80 0x7fbb2015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.490+0000 7fbb24c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb2013df80 0x7fbb2015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57880/0 (socket says 172.21.15.141:57880) 2024-01-16T15:05:13.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.490+0000 7fbb24c00640 1 -- 172.21.15.141:0/1846650547 learned_addr learned my addr 172.21.15.141:0/1846650547 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:13.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.491+0000 7fbb24c00640 1 -- 172.21.15.141:0/1846650547 >> v1:172.21.15.188:6789/0 conn(0x7fbb200cf9a0 legacy=0x7fbb200cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:13.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.491+0000 7fbb24c00640 1 -- 172.21.15.141:0/1846650547 >> v1:172.21.15.141:6789/0 conn(0x7fbb2013e360 legacy=0x7fbb200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.491+0000 7fbb24c00640 1 -- 172.21.15.141:0/1846650547 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2006cd10 con 0x7fbb2013df80 2024-01-16T15:05:13.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.492+0000 7fbb24c00640 1 --2- 172.21.15.141:0/1846650547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb2013df80 0x7fbb2015bee0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009870 tx=0x7fbb1002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8796c127d66d628a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:13.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.492+0000 7fbb1f400640 1 -- 172.21.15.141:0/1846650547 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10030850 con 0x7fbb2013df80 2024-01-16T15:05:13.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.492+0000 7fbb1f400640 1 -- 172.21.15.141:0/1846650547 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb100309b0 con 0x7fbb2013df80 2024-01-16T15:05:13.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.492+0000 7fbb1f400640 1 -- 172.21.15.141:0/1846650547 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10030d40 con 0x7fbb2013df80 2024-01-16T15:05:13.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.493+0000 7fbb27600640 1 -- 172.21.15.141:0/1846650547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb2013df80 msgr2=0x7fbb2015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.493+0000 7fbb27600640 1 --2- 172.21.15.141:0/1846650547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb2013df80 0x7fbb2015bee0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009870 tx=0x7fbb1002b990 comp rx=0 tx=0).stop 2024-01-16T15:05:13.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.493+0000 7fbb27600640 1 -- 172.21.15.141:0/1846650547 shutdown_connections 2024-01-16T15:05:13.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.493+0000 7fbb27600640 1 --2- 172.21.15.141:0/1846650547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb2013df80 0x7fbb2015bee0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.493+0000 7fbb27600640 1 -- 172.21.15.141:0/1846650547 >> 172.21.15.141:0/1846650547 conn(0x7fbb2006a800 msgr2=0x7fbb2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:13.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.493+0000 7fbb27600640 1 -- 172.21.15.141:0/1846650547 shutdown_connections 2024-01-16T15:05:13.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.493+0000 7fbb27600640 1 -- 172.21.15.141:0/1846650547 wait complete. 2024-01-16T15:05:13.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.495+0000 7fbb27600640 1 Processor -- start 2024-01-16T15:05:13.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.495+0000 7fbb27600640 1 -- start start 2024-01-16T15:05:13.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb27600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb200cf9a0 0x7fbb2015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb27600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2013df80 0x7fbb2013c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb27600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb2013e360 0x7fbb2013c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb27600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20143020 con 0x7fbb200cf9a0 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb27600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20157000 con 0x7fbb2013df80 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb27600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2015a590 con 0x7fbb2013e360 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb24c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb200cf9a0 0x7fbb2015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb24c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb200cf9a0 0x7fbb2015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57882/0 (socket says 172.21.15.141:57882) 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.496+0000 7fbb24c00640 1 -- 172.21.15.141:0/78079739 learned_addr learned my addr 172.21.15.141:0/78079739 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb1fe00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2013df80 0x7fbb2013c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb24c00640 1 -- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb2013e360 msgr2=0x7fbb2013c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb24c00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb2013e360 0x7fbb2013c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb24c00640 1 -- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2013df80 msgr2=0x7fbb2013c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb24c00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2013df80 0x7fbb2013c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb24c00640 1 -- 172.21.15.141:0/78079739 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb100094d0 con 0x7fbb200cf9a0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb1fe00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2013df80 0x7fbb2013c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb24c00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb200cf9a0 0x7fbb2015a050 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009870 tx=0x7fbb10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb1cc00640 1 -- 172.21.15.141:0/78079739 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10039d60 con 0x7fbb200cf9a0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb1cc00640 1 -- 172.21.15.141:0/78079739 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb10043400 con 0x7fbb200cf9a0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb1cc00640 1 -- 172.21.15.141:0/78079739 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb100437b0 con 0x7fbb200cf9a0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.497+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb20145e50 con 0x7fbb200cf9a0 2024-01-16T15:05:13.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.498+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb20143510 con 0x7fbb200cf9a0 2024-01-16T15:05:13.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.499+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb200675d0 con 0x7fbb200cf9a0 2024-01-16T15:05:13.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.499+0000 7fbb1cc00640 1 -- 172.21.15.141:0/78079739 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fbb1002e070 con 0x7fbb200cf9a0 2024-01-16T15:05:13.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.500+0000 7fbb1cc00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbaf806bbf0 0x7fbaf806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.500+0000 7fbb1cc00640 1 -- 172.21.15.141:0/78079739 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb1003e070 con 0x7fbb200cf9a0 2024-01-16T15:05:13.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.501+0000 7fbb1fe00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbaf806bbf0 0x7fbaf806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.502+0000 7fbb1fe00640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbaf806bbf0 0x7fbaf806e0b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbb201547a0 tx=0x7fbb08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:13.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.504+0000 7fbb1cc00640 1 -- 172.21.15.141:0/78079739 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb100b3020 con 0x7fbb200cf9a0 2024-01-16T15:05:13.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.741+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb2006df00 con 0x7fbb200cf9a0 2024-01-16T15:05:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.743+0000 7fbb1cc00640 1 -- 172.21.15.141:0/78079739 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4927 (secure 0 0 0) 0x7fbb10034070 con 0x7fbb200cf9a0 2024-01-16T15:05:13.722 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:13.722 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3894915042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3894915042},{"type":"v1","addr":"172.21.15.188:6835","nonce":3894915042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/3263318622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":3263318622},{"type":"v1","addr":"172.21.15.188:6837","nonce":3263318622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3550237844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3550237844},{"type":"v1","addr":"172.21.15.141:6833","nonce":3550237844}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:05:07.194846+0000","modified":"2024-01-16T15:05:10.261152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/3457901158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3457901158},{"type":"v1","addr":"172.21.15.141:6835","nonce":3457901158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:05:13.722 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 46 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbaf806bbf0 msgr2=0x7fbaf806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbaf806bbf0 0x7fbaf806e0b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbb201547a0 tx=0x7fbb08009040 comp rx=0 tx=0).stop 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb200cf9a0 msgr2=0x7fbb2015a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb200cf9a0 0x7fbb2015a050 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009870 tx=0x7fbb10004730 comp rx=0 tx=0).stop 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 shutdown_connections 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbaf806bbf0 0x7fbaf806e0b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb2013e360 0x7fbb2013c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2013df80 0x7fbb2013c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 --2- 172.21.15.141:0/78079739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb200cf9a0 0x7fbb2015a050 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 >> 172.21.15.141:0/78079739 conn(0x7fbb2006a800 msgr2=0x7fbb2006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:13.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 shutdown_connections 2024-01-16T15:05:13.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:13.746+0000 7fbb27600640 1 -- 172.21.15.141:0/78079739 wait complete. 2024-01-16T15:05:14.146 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:05:07.194846+0000', 'modified': '2024-01-16T15:05:10.261152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7119': {'gid': 7119, 'name': 'c', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6835/3457901158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6834', 'nonce': 3457901158}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 3457901158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:05:14.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:05:14.147 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:05:14.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.393+0000 7f6f31400640 1 Processor -- start 2024-01-16T15:05:14.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.394+0000 7f6f31400640 1 -- start start 2024-01-16T15:05:14.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.395+0000 7f6f31400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 0x7f6f2c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.395+0000 7f6f31400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c070c40 con 0x7f6f2c0cf290 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.395+0000 7f6f31400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c06c470 con 0x7f6f2c0ae5a0 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.395+0000 7f6f31400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c071450 con 0x7f6f2c0b3ae0 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.396+0000 7f6f2b200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6f2c0cf290 0x7f6f2c0b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:51932/0 (socket says 172.21.15.141:51932) 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.396+0000 7f6f2b200640 1 -- 172.21.15.141:0/546448186 learned_addr learned my addr 172.21.15.141:0/546448186 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.396+0000 7f6f29e00640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 0x7f6f2c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.396+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141017558 0 0) 0x7f6f2c070c40 con 0x7f6f2c0cf290 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.396+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f0c003520 con 0x7f6f2c0cf290 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1119962708 0 0) 0x7f6f0c003520 con 0x7f6f2c0cf290 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 msgr2=0x7f6f2c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 0x7f6f2c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 >> v1:172.21.15.188:6789/0 conn(0x7f6f2c0ae5a0 legacy=0x7f6f2c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c06c750 con 0x7f6f2c0cf290 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6f240019f0 con 0x7f6f2c0cf290 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 >> v1:172.21.15.141:6789/0 conn(0x7f6f2c0cf290 legacy=0x7f6f2c0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0c0039a0 0x7f6f0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 0x7f6f0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c071450 con 0x7f6f0c004d00 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c00baf0 con 0x7f6f0c0039a0 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c00bcb0 con 0x7f6f2c0b3ae0 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f2b200640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f29e00640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 0x7f6f0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.397+0000 7f6f2a800640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0c0039a0 0x7f6f0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.398+0000 7f6f2b200640 1 -- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 msgr2=0x7f6f0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.398+0000 7f6f2b200640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c0b3ae0 0x7f6f0c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.398+0000 7f6f2b200640 1 -- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0c0039a0 msgr2=0x7f6f0c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.398+0000 7f6f2b200640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0c0039a0 0x7f6f0c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.398+0000 7f6f2b200640 1 -- 172.21.15.141:0/546448186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c003520 con 0x7f6f0c004d00 2024-01-16T15:05:14.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f2b200640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f0c005180 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400a700 tx=0x7f6f0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2008eb1bd52f2e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24034630 con 0x7f6f0c004d00 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2402f070 con 0x7f6f0c004d00 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f29400640 1 -- 172.21.15.141:0/546448186 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24034a10 con 0x7f6f0c004d00 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f31400640 1 -- 172.21.15.141:0/546448186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 msgr2=0x7f6f0c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f31400640 1 --2- 172.21.15.141:0/546448186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f0c005180 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400a700 tx=0x7f6f0c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f2b200640 1 -- 172.21.15.141:0/546448186 reap_dead start 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f31400640 1 -- 172.21.15.141:0/546448186 shutdown_connections 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.399+0000 7f6f31400640 1 -- 172.21.15.141:0/546448186 >> 172.21.15.141:0/546448186 conn(0x7f6f2c06a800 msgr2=0x7f6f2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.400+0000 7f6f31400640 1 -- 172.21.15.141:0/546448186 shutdown_connections 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.400+0000 7f6f31400640 1 -- 172.21.15.141:0/546448186 wait complete. 2024-01-16T15:05:14.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.401+0000 7f6f31400640 1 Processor -- start 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.401+0000 7f6f31400640 1 -- start start 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.402+0000 7f6f31400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f2c14b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.402+0000 7f6f31400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f2c0cbf70 0x7f6f2c0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.402+0000 7f6f31400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c15c040 0x7f6f2c15c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.402+0000 7f6f31400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0cc930 con 0x7f6f0c004d00 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.402+0000 7f6f31400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c156db0 con 0x7f6f2c0cbf70 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.402+0000 7f6f31400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c14be80 con 0x7f6f2c15c040 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f2c14b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f2c14b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57902/0 (socket says 172.21.15.141:57902) 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 -- 172.21.15.141:0/2960532736 learned_addr learned my addr 172.21.15.141:0/2960532736 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 -- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c15c040 msgr2=0x7f6f2c15c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c15c040 0x7f6f2c15c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 -- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f2c0cbf70 msgr2=0x7f6f2c0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f2c0cbf70 0x7f6f2c0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 -- 172.21.15.141:0/2960532736 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c15cde0 con 0x7f6f0c004d00 2024-01-16T15:05:14.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.403+0000 7f6f2a800640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f2c14b940 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400e550 tx=0x7f6f1400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.404+0000 7f6f22a00640 1 -- 172.21.15.141:0/2960532736 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f140052e0 con 0x7f6f0c004d00 2024-01-16T15:05:14.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.404+0000 7f6f22a00640 1 -- 172.21.15.141:0/2960532736 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14005440 con 0x7f6f0c004d00 2024-01-16T15:05:14.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.404+0000 7f6f22a00640 1 -- 172.21.15.141:0/2960532736 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14003810 con 0x7f6f0c004d00 2024-01-16T15:05:14.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.404+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2c15d0c0 con 0x7f6f0c004d00 2024-01-16T15:05:14.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.404+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2c156a40 con 0x7f6f0c004d00 2024-01-16T15:05:14.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.406+0000 7f6f22a00640 1 -- 172.21.15.141:0/2960532736 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6f14003970 con 0x7f6f0c004d00 2024-01-16T15:05:14.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.407+0000 7f6f22a00640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6f0406bbf0 0x7f6f0406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.407+0000 7f6f22a00640 1 -- 172.21.15.141:0/2960532736 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f14088390 con 0x7f6f0c004d00 2024-01-16T15:05:14.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.408+0000 7f6f29e00640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6f0406bbf0 0x7f6f0406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.408+0000 7f6f3418f740 1 Processor -- start 2024-01-16T15:05:14.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.408+0000 7f6f3418f740 1 -- start start 2024-01-16T15:05:14.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 0x55f9d8819280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d88180a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6f0406bbf0 0x7f6f0406e0b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0ce8a0 tx=0x7f6f1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 0x55f9d885c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f3418f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8777880 con 0x55f9d8813ba0 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8786ae0 con 0x55f9d8818ea0 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f9d87841d0 con 0x55f9d88185e0 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d88180a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d88180a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57906/0 (socket says 172.21.15.141:57906) 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 -- 172.21.15.141:0/1275048786 learned_addr learned my addr 172.21.15.141:0/1275048786 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 -- 172.21.15.141:0/1275048786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 msgr2=0x55f9d885c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 --2- 172.21.15.141:0/1275048786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 0x55f9d885c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 -- 172.21.15.141:0/1275048786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 msgr2=0x55f9d8819280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 --2- 172.21.15.141:0/1275048786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 0x55f9d8819280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.409+0000 7f6f29e00640 1 -- 172.21.15.141:0/1275048786 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9d8747860 con 0x55f9d8813ba0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.410+0000 7f6f29e00640 1 --2- 172.21.15.141:0/1275048786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d88180a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c011c10 tx=0x7f6f1c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ec90fbda96aa36 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.410+0000 7f6f1b400640 1 -- 172.21.15.141:0/1275048786 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c015a90 con 0x55f9d8813ba0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.410+0000 7f6f1b400640 1 -- 172.21.15.141:0/1275048786 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1c015bf0 con 0x55f9d8813ba0 2024-01-16T15:05:14.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.410+0000 7f6f1b400640 1 -- 172.21.15.141:0/1275048786 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c021440 con 0x55f9d8813ba0 2024-01-16T15:05:14.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 -- 172.21.15.141:0/1275048786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 msgr2=0x55f9d88180a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1275048786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d88180a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c011c10 tx=0x7f6f1c014000 comp rx=0 tx=0).stop 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 -- 172.21.15.141:0/1275048786 shutdown_connections 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1275048786 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 0x55f9d885c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1275048786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d88180a0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1275048786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 0x55f9d8819280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 -- 172.21.15.141:0/1275048786 >> 172.21.15.141:0/1275048786 conn(0x55f9d87dac70 msgr2=0x55f9d885e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 -- 172.21.15.141:0/1275048786 shutdown_connections 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.411+0000 7f6f3418f740 1 -- 172.21.15.141:0/1275048786 wait complete. 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.412+0000 7f6f3418f740 1 -- start start 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.412+0000 7f6f3418f740 1 Processor -- start 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8813ba0 0x55f9d885af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d88185e0 0x55f9d8865b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8818ea0 0x55f9d8866080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f3418f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d877fac0 con 0x55f9d8818ea0 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d871c2f0 con 0x55f9d88185e0 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8739c70 con 0x55f9d8813ba0 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f3418f740 10 client.0 _ll_register_callbacks cb 0x55f9d873e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f2b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8818ea0 0x55f9d8866080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f2b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8818ea0 0x55f9d8866080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57920/0 (socket says 172.21.15.141:57920) 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f2b200640 1 -- 172.21.15.141:0/485941281 learned_addr learned my addr 172.21.15.141:0/485941281 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.413+0000 7f6f2a800640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8813ba0 0x55f9d885af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f29e00640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d88185e0 0x55f9d8865b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f2b200640 1 -- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8813ba0 msgr2=0x55f9d885af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f2b200640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8813ba0 0x55f9d885af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f2b200640 1 -- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d88185e0 msgr2=0x55f9d8865b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f2b200640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d88185e0 0x55f9d8865b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f2b200640 1 -- 172.21.15.141:0/485941281 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1c011860 con 0x55f9d8818ea0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f29e00640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d88185e0 0x55f9d8865b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f2b200640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8818ea0 0x55f9d8866080 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400a700 tx=0x7f6f24001ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2af888dd9a19c0f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f13e00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f13e00640 1 -- 172.21.15.141:0/485941281 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f240019b0 con 0x55f9d8818ea0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f13e00640 1 -- 172.21.15.141:0/485941281 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f24034430 con 0x55f9d8818ea0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9d8778aa0 con 0x55f9d8818ea0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f13e00640 1 -- 172.21.15.141:0/485941281 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f240345f0 con 0x55f9d8818ea0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.414+0000 7f6f3418f740 10 client.7248 fetch_fsmap learned FSMap version 46 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55f9d877e140 con 0x55f9d8818ea0 2024-01-16T15:05:14.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f13e00640 1 -- 172.21.15.141:0/485941281 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f9d8778aa0 con 0x55f9d8818ea0 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f13e00640 1 -- 172.21.15.141:0/485941281 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 46) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6f24034750 con 0x55f9d8818ea0 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f3418f740 10 client.7248 fetch_fsmap finished waiting for FSMap version 46 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f3418f740 10 client.7248 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f3418f740 20 client.7248 populate_metadata read hostname 'smithi141' 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f3418f740 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8841de0 0x55f9d8863790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f3418f740 4 client.7248 mds_command: new command op to 7119 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 --> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9d87787c0 con 0x55f9d8841de0 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.415+0000 7f6f2a800640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8841de0 0x55f9d8863790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.416+0000 7f6f2a800640 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8841de0 0x55f9d8863790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f88983bc36379d0 server_cookie=d4fced2c7e7dbf53 in_seq=0 out_seq=0 2024-01-16T15:05:14.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.416+0000 7f6f13e00640 10 client.7248 ms_handle_connect on v2:172.21.15.141:6834/3457901158 2024-01-16T15:05:14.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.416+0000 7f6f13e00640 1 -- 172.21.15.141:0/485941281 <== mds.0 v2:172.21.15.141:6834/3457901158 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f9d87787c0 con 0x55f9d8841de0 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.416+0000 7f6f13e00640 10 client.7248 handle_command_reply: tid=0 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.416+0000 7f6f3418f740 1 client.7248 shutdown 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8818ea0 msgr2=0x55f9d8866080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8818ea0 0x55f9d8866080 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400a700 tx=0x7f6f24001ea0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 shutdown_connections 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8841de0 0x55f9d8863790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8818ea0 0x55f9d8866080 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d88185e0 0x55f9d8865b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 --2- 172.21.15.141:0/485941281 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8813ba0 0x55f9d885af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f13e00640 0 client.7248 ms_handle_reset on v2:172.21.15.141:6834/3457901158 2024-01-16T15:05:14.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 >> 172.21.15.141:0/485941281 conn(0x55f9d87dac70 msgr2=0x55f9d885bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 shutdown_connections 2024-01-16T15:05:14.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 1 -- 172.21.15.141:0/485941281 wait complete. 2024-01-16T15:05:14.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.417+0000 7f6f3418f740 20 client.7248 trim_cache size 0 max 16384 2024-01-16T15:05:14.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.434+0000 7f6f3418f740 1 Processor -- start 2024-01-16T15:05:14.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.434+0000 7f6f3418f740 1 -- start start 2024-01-16T15:05:14.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d8813380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 0x55f9d88141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 0x55f9d8817a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f3418f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d84e2f20 con 0x55f9d8813ba0 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8847ad0 con 0x55f9d8818ea0 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8812ea0 con 0x55f9d88185e0 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f2a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d8813380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f2a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d8813380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57924/0 (socket says 172.21.15.141:57924) 2024-01-16T15:05:14.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.435+0000 7f6f2a800640 1 -- 172.21.15.141:0/853462141 learned_addr learned my addr 172.21.15.141:0/853462141 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f2a800640 1 -- 172.21.15.141:0/853462141 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 msgr2=0x55f9d88141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f2a800640 1 --2- 172.21.15.141:0/853462141 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 0x55f9d88141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f2a800640 1 -- 172.21.15.141:0/853462141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 msgr2=0x55f9d8817a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f2a800640 1 --2- 172.21.15.141:0/853462141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 0x55f9d8817a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f2a800640 1 -- 172.21.15.141:0/853462141 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9d871bb80 con 0x55f9d8813ba0 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f2a800640 1 --2- 172.21.15.141:0/853462141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d8813380 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6f14054c20 tx=0x7f6f140055b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb229ee94f83de9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f1be00640 1 -- 172.21.15.141:0/853462141 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14085070 con 0x55f9d8813ba0 2024-01-16T15:05:14.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f1be00640 1 -- 172.21.15.141:0/853462141 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14070070 con 0x55f9d8813ba0 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.436+0000 7f6f1be00640 1 -- 172.21.15.141:0/853462141 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14060a20 con 0x55f9d8813ba0 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 -- 172.21.15.141:0/853462141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 msgr2=0x55f9d8813380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 --2- 172.21.15.141:0/853462141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d8813380 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6f14054c20 tx=0x7f6f140055b0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 -- 172.21.15.141:0/853462141 shutdown_connections 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 --2- 172.21.15.141:0/853462141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8818ea0 0x55f9d8817a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 --2- 172.21.15.141:0/853462141 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d88185e0 0x55f9d88141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 --2- 172.21.15.141:0/853462141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d8813ba0 0x55f9d8813380 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 -- 172.21.15.141:0/853462141 >> 172.21.15.141:0/853462141 conn(0x55f9d87dac70 msgr2=0x55f9d885ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 -- 172.21.15.141:0/853462141 shutdown_connections 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 -- 172.21.15.141:0/853462141 wait complete. 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.437+0000 7f6f3418f740 1 -- start start 2024-01-16T15:05:14.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.438+0000 7f6f3418f740 1 Processor -- start 2024-01-16T15:05:14.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8813ba0 0x55f9d88c8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d88185e0 0x55f9d886d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f3418f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8818ea0 0x55f9d886d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f3418f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d88665b0 con 0x55f9d88185e0 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8859450 con 0x55f9d8813ba0 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f3418f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f9d882f2d0 con 0x55f9d8818ea0 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f3418f740 10 client.0 _ll_register_callbacks cb 0x7f6f0c0036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d88185e0 0x55f9d886d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d88185e0 0x55f9d886d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57930/0 (socket says 172.21.15.141:57930) 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 -- 172.21.15.141:0/1117458663 learned_addr learned my addr 172.21.15.141:0/1117458663 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f2b200640 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8818ea0 0x55f9d886d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f2a800640 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8813ba0 0x55f9d88c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 -- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8818ea0 msgr2=0x55f9d886d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8818ea0 0x55f9d886d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 -- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8813ba0 msgr2=0x55f9d88c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8813ba0 0x55f9d88c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.439+0000 7f6f29e00640 1 -- 172.21.15.141:0/1117458663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2c156a40 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f29e00640 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d88185e0 0x55f9d886d220 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c008010 tx=0x7f6f1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f66d984b5318d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f13e00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f13e00640 1 -- 172.21.15.141:0/1117458663 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c015d60 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f13e00640 1 -- 172.21.15.141:0/1117458663 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f1c031070 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f1c015bf0 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f13e00640 1 -- 172.21.15.141:0/1117458663 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f1c0219e0 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f3418f740 10 client.7254 fetch_fsmap learned FSMap version 46 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55f9d8741170 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f13e00640 1 -- 172.21.15.141:0/1117458663 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1c015bf0 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.440+0000 7f6f13e00640 1 -- 172.21.15.141:0/1117458663 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 46) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6f1c036480 con 0x55f9d88185e0 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f3418f740 10 client.7254 fetch_fsmap finished waiting for FSMap version 46 2024-01-16T15:05:14.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f3418f740 10 client.7254 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f3418f740 20 client.7254 populate_metadata read hostname 'smithi141' 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8846ac0 0x55f9d8846ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f3418f740 4 client.7254 mds_command: new command op to 7119 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 --> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f9d84806c0 con 0x55f9d8846ac0 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f2a800640 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8846ac0 0x55f9d8846ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f2a800640 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8846ac0 0x55f9d8846ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc285ffb8d03203 server_cookie=391424124e48a2b2 in_seq=0 out_seq=0 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f13e00640 10 client.7254 ms_handle_connect on v2:172.21.15.141:6834/3457901158 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f13e00640 1 -- 172.21.15.141:0/1117458663 <== mds.0 v2:172.21.15.141:6834/3457901158 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55f9d84806c0 con 0x55f9d8846ac0 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f13e00640 10 client.7254 handle_command_reply: tid=0 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.441+0000 7f6f3418f740 1 client.7254 shutdown 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d88185e0 msgr2=0x55f9d886d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d88185e0 0x55f9d886d220 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c008010 tx=0x7f6f1c004730 comp rx=0 tx=0).stop 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 shutdown_connections 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:6834/3457901158,v1:172.21.15.141:6835/3457901158] conn(0x55f9d8846ac0 0x55f9d8846ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x55f9d8818ea0 0x55f9d886d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55f9d88185e0 0x55f9d886d220 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f13e00640 0 client.7254 ms_handle_reset on v2:172.21.15.141:6834/3457901158 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 --2- 172.21.15.141:0/1117458663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9d8813ba0 0x55f9d88c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 >> 172.21.15.141:0/1117458663 conn(0x55f9d87dac70 msgr2=0x55f9d8813f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 shutdown_connections 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 1 -- 172.21.15.141:0/1117458663 wait complete. 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.442+0000 7f6f3418f740 20 client.7254 trim_cache size 0 max 16384 2024-01-16T15:05:14.425 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 7119, 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 6.702869744, 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 46, 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 73, 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 73, 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 13.123045403000001 2024-01-16T15:05:14.426 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:05:14.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6f0406bbf0 msgr2=0x7f6f0406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6f0406bbf0 0x7f6f0406e0b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0ce8a0 tx=0x7f6f1c008040 comp rx=0 tx=0).stop 2024-01-16T15:05:14.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 msgr2=0x7f6f2c14b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f2c14b940 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400e550 tx=0x7f6f1400ea10 comp rx=0 tx=0).stop 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 shutdown_connections 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6f0406bbf0 0x7f6f0406e0b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f2c15c040 0x7f6f2c15c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f2c0cbf70 0x7f6f2c0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 --2- 172.21.15.141:0/2960532736 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f0c004d00 0x7f6f2c14b940 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 >> 172.21.15.141:0/2960532736 conn(0x7f6f2c06a800 msgr2=0x7f6f2c159ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 shutdown_connections 2024-01-16T15:05:14.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.443+0000 7f6f31400640 1 -- 172.21.15.141:0/2960532736 wait complete. 2024-01-16T15:05:14.439 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:05:14.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.660+0000 7f0e76200640 1 Processor -- start 2024-01-16T15:05:14.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.660+0000 7f0e76200640 1 -- start start 2024-01-16T15:05:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.662+0000 7f0e76200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0e7015c360 0x7f0e700b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.662+0000 7f0e76200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7006ebc0 0x7f0e7006f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.662+0000 7f0e76200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e700af150 con 0x7f0e70152250 2024-01-16T15:05:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.662+0000 7f0e76200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7006f590 con 0x7f0e7015c360 2024-01-16T15:05:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.662+0000 7f0e76200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700ce050 con 0x7f0e7006ebc0 2024-01-16T15:05:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.662+0000 7f0e6f400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0e70152250 0x7f0e7006b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:51942/0 (socket says 172.21.15.141:51942) 2024-01-16T15:05:14.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.662+0000 7f0e6f400640 1 -- 172.21.15.141:0/3109364525 learned_addr learned my addr 172.21.15.141:0/3109364525 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6ea00640 1 --2- 172.21.15.141:0/3109364525 >> v2:172.21.15.188:3300/0 conn(0x7f0e7015c360 0x7f0e700b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6fe00640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7006ebc0 0x7f0e7006f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504157193 0 0) 0x7f0e700af150 con 0x7f0e70152250 2024-01-16T15:05:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e50003520 con 0x7f0e70152250 2024-01-16T15:05:14.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3664775324 0 0) 0x7f0e50003520 con 0x7f0e70152250 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7006ebc0 msgr2=0x7f0e7006f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7006ebc0 0x7f0e7006f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 >> v2:172.21.15.188:3300/0 conn(0x7f0e7015c360 msgr2=0x7f0e700b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 --2- 172.21.15.141:0/3109364525 >> v2:172.21.15.188:3300/0 conn(0x7f0e7015c360 0x7f0e700b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70152630 con 0x7f0e70152250 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0e58001670 con 0x7f0e70152250 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 >> v1:172.21.15.141:6789/0 conn(0x7f0e70152250 legacy=0x7f0e7006b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e50003970 0x7f0e50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e500047c0 0x7f0e50004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7006ebc0 0x7f0e50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7006f590 con 0x7f0e50003970 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700ce050 con 0x7f0e500047c0 2024-01-16T15:05:14.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.663+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5000bab0 con 0x7f0e7006ebc0 2024-01-16T15:05:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.664+0000 7f0e6f400640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e50003970 0x7f0e50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.664+0000 7f0e6ea00640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e500047c0 0x7f0e50004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.664+0000 7f0e6f400640 1 -- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7006ebc0 msgr2=0x7f0e50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.664+0000 7f0e6f400640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7006ebc0 0x7f0e50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.664+0000 7f0e6f400640 1 -- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e500047c0 msgr2=0x7f0e50004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.664+0000 7f0e6f400640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e500047c0 0x7f0e50004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.664+0000 7f0e6f400640 1 -- 172.21.15.141:0/3109364525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50003520 con 0x7f0e50003970 2024-01-16T15:05:14.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.665+0000 7f0e6f400640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e50003970 0x7f0e50003d90 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0e58001eb0 tx=0x7f0e500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44481f7d24f75dc5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.666+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e58005ad0 con 0x7f0e50003970 2024-01-16T15:05:14.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.666+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e58005c30 con 0x7f0e50003970 2024-01-16T15:05:14.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.666+0000 7f0e6e000640 1 -- 172.21.15.141:0/3109364525 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5802c7c0 con 0x7f0e50003970 2024-01-16T15:05:14.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.666+0000 7f0e76200640 1 -- 172.21.15.141:0/3109364525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e50003970 msgr2=0x7f0e50003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.666+0000 7f0e76200640 1 --2- 172.21.15.141:0/3109364525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e50003970 0x7f0e50003d90 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0e58001eb0 tx=0x7f0e500057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.666+0000 7f0e6fe00640 1 -- 172.21.15.141:0/3109364525 reap_dead start 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.666+0000 7f0e76200640 1 -- 172.21.15.141:0/3109364525 shutdown_connections 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.667+0000 7f0e76200640 1 -- 172.21.15.141:0/3109364525 >> 172.21.15.141:0/3109364525 conn(0x7f0e70070b10 msgr2=0x7f0e70070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.667+0000 7f0e76200640 1 -- 172.21.15.141:0/3109364525 shutdown_connections 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.667+0000 7f0e76200640 1 -- 172.21.15.141:0/3109364525 wait complete. 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.669+0000 7f0e76200640 1 Processor -- start 2024-01-16T15:05:14.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.669+0000 7f0e76200640 1 -- start start 2024-01-16T15:05:14.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e76200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e50003970 0x7f0e70154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e76200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7014ff00 0x7f0e7006dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e76200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e7006e4e0 0x7f0e70152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e76200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7006e970 con 0x7f0e7006e4e0 2024-01-16T15:05:14.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e76200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e701e39e0 con 0x7f0e50003970 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e76200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e701e4840 con 0x7f0e7014ff00 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e6fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e7006e4e0 0x7f0e70152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e6fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e7006e4e0 0x7f0e70152850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57942/0 (socket says 172.21.15.141:57942) 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.670+0000 7f0e6fe00640 1 -- 172.21.15.141:0/3055381548 learned_addr learned my addr 172.21.15.141:0/3055381548 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6f400640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e50003970 0x7f0e70154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6ea00640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7014ff00 0x7f0e7006dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6fe00640 1 -- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7014ff00 msgr2=0x7f0e7006dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6fe00640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7014ff00 0x7f0e7006dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6fe00640 1 -- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e50003970 msgr2=0x7f0e70154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6fe00640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e50003970 0x7f0e70154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6fe00640 1 -- 172.21.15.141:0/3055381548 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e701e3c30 con 0x7f0e7006e4e0 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e6fe00640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e7006e4e0 0x7f0e70152850 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0e700724d0 tx=0x7f0e6800d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e67400640 1 -- 172.21.15.141:0/3055381548 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68004030 con 0x7f0e7006e4e0 2024-01-16T15:05:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e67400640 1 -- 172.21.15.141:0/3055381548 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e680041b0 con 0x7f0e7006e4e0 2024-01-16T15:05:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e67400640 1 -- 172.21.15.141:0/3055381548 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e680126b0 con 0x7f0e7006e4e0 2024-01-16T15:05:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.671+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e701e3f10 con 0x7f0e7006e4e0 2024-01-16T15:05:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.672+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e701e44d0 con 0x7f0e7006e4e0 2024-01-16T15:05:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.673+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e701f3140 con 0x7f0e7006e4e0 2024-01-16T15:05:14.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.674+0000 7f0e67400640 1 -- 172.21.15.141:0/3055381548 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f0e68012810 con 0x7f0e7006e4e0 2024-01-16T15:05:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.675+0000 7f0e67400640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0e4806bbc0 0x7f0e4806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.675+0000 7f0e67400640 1 -- 172.21.15.141:0/3055381548 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e68088260 con 0x7f0e7006e4e0 2024-01-16T15:05:14.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.675+0000 7f0e6f400640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0e4806bbc0 0x7f0e4806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.677+0000 7f0e6f400640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0e4806bbc0 0x7f0e4806e080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0e58001eb0 tx=0x7f0e58009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.679+0000 7f0e67400640 1 -- 172.21.15.141:0/3055381548 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e701f3140 con 0x7f0e7006e4e0 2024-01-16T15:05:14.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.911+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0e701f3140 con 0x7f0e7006e4e0 2024-01-16T15:05:14.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.911+0000 7f0e67400640 1 -- 172.21.15.141:0/3055381548 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0e701f3140 con 0x7f0e7006e4e0 2024-01-16T15:05:14.890 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:05:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0e4806bbc0 msgr2=0x7f0e4806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0e4806bbc0 0x7f0e4806e080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0e58001eb0 tx=0x7f0e58009030 comp rx=0 tx=0).stop 2024-01-16T15:05:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e7006e4e0 msgr2=0x7f0e70152850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e7006e4e0 0x7f0e70152850 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0e700724d0 tx=0x7f0e6800d820 comp rx=0 tx=0).stop 2024-01-16T15:05:14.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 shutdown_connections 2024-01-16T15:05:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0e4806bbc0 0x7f0e4806e080 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e7006e4e0 0x7f0e70152850 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e7014ff00 0x7f0e7006dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 --2- 172.21.15.141:0/3055381548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e50003970 0x7f0e70154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.914+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 >> 172.21.15.141:0/3055381548 conn(0x7f0e70070b10 msgr2=0x7f0e700b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.915+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 shutdown_connections 2024-01-16T15:05:14.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:14.915+0000 7f0e76200640 1 -- 172.21.15.141:0/3055381548 wait complete. 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:05:14.911 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:05:14.912 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:14.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.912 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T15:05:14.961 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:05:14.961 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:05:14.961 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:05:14.962 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42148sec preferred_lft 42148sec 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:14.963 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:14.964 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T15:05:14.964 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.964 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:14.964 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:14.964 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::882:e9ff:fe92:c54c/64 scope link 2024-01-16T15:05:14.964 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:14.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:14.965 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:05:15.053 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T15:05:15.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.058 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T15:05:15.113 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T15:05:15.113 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:05:15.113 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:05:15.113 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:05:15.114 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:05:15.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.114 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T15:05:15.114 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T15:05:15.114 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:05:15.114 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:05:15.114 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:05:15.114 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T15:05:15.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.292 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T15:05:15.343 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:05:15.343 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:05:15.344 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:15.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.344 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T15:05:15.437 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:05:15.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:15.437 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:15.491 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:15.491 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:15.491 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:15.544 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:05:15.642 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:15.693 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:05:15.693 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:05:15.693 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:05:15.693 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:05:15.693 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:05:15.693 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:15.694 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:15.695 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:15.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.697 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:05:15.745 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:05:15.745 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:05:15.750 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:05:15.825 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:15.865 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:05:15.865 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:05:15.865 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:05:15.865 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:05:15.865 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:05:15.866 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:05:15.867 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:05:15.868 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:15.868 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:15.868 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:15.868 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:15.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:15.869 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:05:15.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T15:05:15.954+0000 7fa127553480 -1 init, newargv = 0x55c776250990 newargc=15 2024-01-16T15:05:15.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[62201]: starting ceph client 2024-01-16T15:05:15.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[62201]: starting fuse 2024-01-16T15:05:15.954 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T15:05:15.954 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:05:15.954 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:05:15.955 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T15:05:15.956 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:05:15.957 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:05:15.957 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T15:05:15.957 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T15:05:16.155 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.62201.asok 2024-01-16T15:05:16.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:05:16.172 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62201.asok 2024-01-16T15:05:16.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:16.172 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62201.asok status 2024-01-16T15:05:16.410 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "62201", 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T15:05:16.411 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 7266, 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 7266 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 2015631275 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:05:16.412 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 2015631275 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.7266 192.168.0.1:0/2015631275", 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/2015631275", 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 46, 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 73, 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T15:05:16.413 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:05:16.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:16.426 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T15:05:16.426 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:16.482 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T15:05:16.482 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:16.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:16.482 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:16.579 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:05:16.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.807+0000 7f6d5ba00640 1 Processor -- start 2024-01-16T15:05:16.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.807+0000 7f6d5ba00640 1 -- start start 2024-01-16T15:05:16.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d5ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d5ba00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6d540cf7d0 0x7f6d5415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d5ba00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d540cfc30 con 0x7f6d5415e1e0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d5ba00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d540b1ee0 con 0x7f6d540cf7d0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d5ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540b2780 con 0x7f6d54143ae0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d59a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6d5415e1e0 0x7f6d5415a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:51958/0 (socket says 172.21.15.141:51958) 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d59a00640 1 -- 172.21.15.141:0/395812396 learned_addr learned my addr 172.21.15.141:0/395812396 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.809+0000 7f6d4bc00640 1 --2- 172.21.15.141:0/395812396 >> v2:172.21.15.141:3300/0 conn(0x7f6d540cf7d0 0x7f6d5415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d59000640 1 --2- 172.21.15.141:0/395812396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d53e00640 1 -- 172.21.15.141:0/395812396 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520158185 0 0) 0x7f6d540cfc30 con 0x7f6d5415e1e0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d53e00640 1 -- 172.21.15.141:0/395812396 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d34003520 con 0x7f6d5415e1e0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d4bc00640 1 -- 172.21.15.141:0/395812396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 msgr2=0x7f6d540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d4bc00640 1 --2- 172.21.15.141:0/395812396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d4bc00640 1 -- 172.21.15.141:0/395812396 >> v1:172.21.15.141:6789/0 conn(0x7f6d5415e1e0 legacy=0x7f6d5415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d4bc00640 1 -- 172.21.15.141:0/395812396 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5415e990 con 0x7f6d540cf7d0 2024-01-16T15:05:16.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.810+0000 7f6d59000640 1 --2- 172.21.15.141:0/395812396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:16.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.811+0000 7f6d4bc00640 1 --2- 172.21.15.141:0/395812396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d540cf7d0 0x7f6d5415dca0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0097a0 tx=0x7f6d3c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2c375bcf1ddd3c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.811+0000 7f6d53e00640 1 -- 172.21.15.141:0/395812396 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c02d040 con 0x7f6d540cf7d0 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.811+0000 7f6d53e00640 1 -- 172.21.15.141:0/395812396 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3c031810 con 0x7f6d540cf7d0 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.811+0000 7f6d53e00640 1 -- 172.21.15.141:0/395812396 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c031ba0 con 0x7f6d540cf7d0 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 -- 172.21.15.141:0/395812396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d540cf7d0 msgr2=0x7f6d5415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/395812396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d540cf7d0 0x7f6d5415dca0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0097a0 tx=0x7f6d3c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 -- 172.21.15.141:0/395812396 shutdown_connections 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/395812396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d540cf7d0 0x7f6d5415dca0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/395812396 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 -- 172.21.15.141:0/395812396 >> 172.21.15.141:0/395812396 conn(0x7f6d5406a800 msgr2=0x7f6d5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 -- 172.21.15.141:0/395812396 shutdown_connections 2024-01-16T15:05:16.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.812+0000 7f6d5ba00640 1 -- 172.21.15.141:0/395812396 wait complete. 2024-01-16T15:05:16.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.814+0000 7f6d5ba00640 1 Processor -- start 2024-01-16T15:05:16.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.814+0000 7f6d5ba00640 1 -- start start 2024-01-16T15:05:16.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d5ba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d540cf7d0 0x7f6d54141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d5ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d5ba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d5415e1e0 0x7f6d540b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d5ba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540c82f0 con 0x7f6d5415e1e0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d5ba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540c8480 con 0x7f6d540cf7d0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d5ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540c4ca0 con 0x7f6d54143ae0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d5415e1e0 0x7f6d540b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d5415e1e0 0x7f6d540b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57960/0 (socket says 172.21.15.141:57960) 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 -- 172.21.15.141:0/971338773 learned_addr learned my addr 172.21.15.141:0/971338773 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59000640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d540cf7d0 0x7f6d54141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d4bc00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 -- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 msgr2=0x7f6d540b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 -- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d540cf7d0 msgr2=0x7f6d54141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d540cf7d0 0x7f6d54141980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.815+0000 7f6d59a00640 1 -- 172.21.15.141:0/971338773 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c009400 con 0x7f6d5415e1e0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.816+0000 7f6d59a00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d5415e1e0 0x7f6d540b1100 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00c390 tx=0x7f6d4c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.816+0000 7f6d51600640 1 -- 172.21.15.141:0/971338773 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c005ce0 con 0x7f6d5415e1e0 2024-01-16T15:05:16.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.816+0000 7f6d51600640 1 -- 172.21.15.141:0/971338773 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c005e40 con 0x7f6d5415e1e0 2024-01-16T15:05:16.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.816+0000 7f6d51600640 1 -- 172.21.15.141:0/971338773 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c007310 con 0x7f6d5415e1e0 2024-01-16T15:05:16.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.816+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d540c4f80 con 0x7f6d5415e1e0 2024-01-16T15:05:16.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.816+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d540c5540 con 0x7f6d5415e1e0 2024-01-16T15:05:16.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.817+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d540c85f0 con 0x7f6d5415e1e0 2024-01-16T15:05:16.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.818+0000 7f6d51600640 1 -- 172.21.15.141:0/971338773 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6d4c007470 con 0x7f6d5415e1e0 2024-01-16T15:05:16.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.819+0000 7f6d51600640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6d2c06bbf0 0x7f6d2c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.819+0000 7f6d51600640 1 -- 172.21.15.141:0/971338773 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d4c087b30 con 0x7f6d5415e1e0 2024-01-16T15:05:16.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.820+0000 7f6d59000640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6d2c06bbf0 0x7f6d2c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.821+0000 7f6d59000640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6d2c06bbf0 0x7f6d2c06e0b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6d440075b0 tx=0x7f6d44007540 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:16.823+0000 7f6d51600640 1 -- 172.21.15.141:0/971338773 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d4c054360 con 0x7f6d5415e1e0 2024-01-16T15:05:17.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.056+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6d540675d0 con 0x7f6d5415e1e0 2024-01-16T15:05:17.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.057+0000 7f6d51600640 1 -- 172.21.15.141:0/971338773 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6d4c017060 con 0x7f6d5415e1e0 2024-01-16T15:05:17.035 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.059+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6d2c06bbf0 msgr2=0x7f6d2c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.059+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6d2c06bbf0 0x7f6d2c06e0b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6d440075b0 tx=0x7f6d44007540 comp rx=0 tx=0).stop 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.059+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d5415e1e0 msgr2=0x7f6d540b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.059+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d5415e1e0 0x7f6d540b1100 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00c390 tx=0x7f6d4c00c850 comp rx=0 tx=0).stop 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 shutdown_connections 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6d2c06bbf0 0x7f6d2c06e0b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d5415e1e0 0x7f6d540b1100 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d54143ae0 0x7f6d540b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 --2- 172.21.15.141:0/971338773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d540cf7d0 0x7f6d54141980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 >> 172.21.15.141:0/971338773 conn(0x7f6d5406a800 msgr2=0x7f6d5406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 shutdown_connections 2024-01-16T15:05:17.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:17.060+0000 7f6d5ba00640 1 -- 172.21.15.141:0/971338773 wait complete. 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:05:17.057 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:17.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:17.058 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42148sec preferred_lft 42148sec 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T15:05:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::a40a:23ff:fed8:ea96/64 scope link 2024-01-16T15:05:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:17.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:17.118 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:05:17.206 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T15:05:17.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:17.210 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T15:05:17.263 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T15:05:17.263 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:05:17.263 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:05:17.263 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:05:17.263 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:05:17.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:17.264 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T15:05:17.264 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T15:05:17.264 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:05:17.264 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:05:17.264 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:05:17.264 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T15:05:17.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:17.445 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T15:05:17.496 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:05:17.496 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:05:17.497 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:17.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:17.497 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T15:05:17.594 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:05:17.595 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:17.595 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:17.648 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:17.648 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:17.648 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:17.698 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:05:17.790 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:17.844 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:05:17.844 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:05:17.844 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:05:17.844 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:05:17.845 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:05:17.846 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:05:17.847 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:05:17.847 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:05:17.847 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:17.847 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:17.847 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:17.847 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:17.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:17.848 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:05:17.899 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:05:17.899 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:05:17.906 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:05:17.983 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:05:18.030 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:18.031 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:05:18.032 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:18.033 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:18.033 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:18.033 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:18.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:18.034 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:05:18.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T15:05:18.114+0000 7ff4bc642480 -1 init, newargv = 0x5609c9572920 newargc=15 2024-01-16T15:05:18.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[44998]: starting ceph client 2024-01-16T15:05:18.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[44998]: starting fuse 2024-01-16T15:05:18.117 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T15:05:18.117 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:05:18.118 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:05:18.119 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T15:05:18.320 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.44998.asok 2024-01-16T15:05:18.324 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T15:05:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:05:18.337 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44998.asok 2024-01-16T15:05:18.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:18.337 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44998.asok status 2024-01-16T15:05:18.573 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T15:05:18.573 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T15:05:18.573 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:05:18.573 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:05:18.573 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T15:05:18.573 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "44998", 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 5764, 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T15:05:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 5764 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 2516697037 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:05:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 2516697037 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.5764 192.168.0.1:0/2516697037", 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/2516697037", 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 46, 2024-01-16T15:05:18.576 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 73, 2024-01-16T15:05:18.577 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:05:18.577 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T15:05:18.577 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T15:05:18.577 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T15:05:18.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:18.588 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T15:05:18.588 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:18.642 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T15:05:18.643 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:18.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:18.643 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:18.738 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:05:18.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.985+0000 7f45e4000640 1 Processor -- start 2024-01-16T15:05:18.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.985+0000 7f45e4000640 1 -- start start 2024-01-16T15:05:18.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e4000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45dc15a6b0 0x7f45dc15aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e4000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f45dc0cea00 0x7f45dc15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e4000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc070410 con 0x7f45dc15dbf0 2024-01-16T15:05:18.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e4000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45dc15a530 con 0x7f45dc15a6b0 2024-01-16T15:05:18.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e4000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45dc0adb00 con 0x7f45dc0cea00 2024-01-16T15:05:18.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e1600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45dc15a6b0 0x7f45dc15aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e1600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45dc15a6b0 0x7f45dc15aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57976/0 (socket says 172.21.15.141:57976) 2024-01-16T15:05:18.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e1600640 1 -- 172.21.15.141:0/2969616280 learned_addr learned my addr 172.21.15.141:0/2969616280 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:18.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.987+0000 7f45e0c00640 1 --2- 172.21.15.141:0/2969616280 >> v2:172.21.15.188:3300/0 conn(0x7f45dc0cea00 0x7f45dc15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.988+0000 7f45e1600640 1 -- 172.21.15.141:0/2969616280 >> v2:172.21.15.188:3300/0 conn(0x7f45dc0cea00 msgr2=0x7f45dc15afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.988+0000 7f45e1600640 1 --2- 172.21.15.141:0/2969616280 >> v2:172.21.15.188:3300/0 conn(0x7f45dc0cea00 0x7f45dc15afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.988+0000 7f45e1600640 1 -- 172.21.15.141:0/2969616280 >> v1:172.21.15.188:6789/0 conn(0x7f45dc15dbf0 legacy=0x7f45dc15dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.988+0000 7f45e1600640 1 -- 172.21.15.141:0/2969616280 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc0adde0 con 0x7f45dc15a6b0 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.989+0000 7f45e1600640 1 --2- 172.21.15.141:0/2969616280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15a6b0 0x7f45dc15aa90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0b3e40 tx=0x7f45cc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5793e082015b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.989+0000 7f45dbe00640 1 -- 172.21.15.141:0/2969616280 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc02c9f0 con 0x7f45dc15a6b0 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.989+0000 7f45dbe00640 1 -- 172.21.15.141:0/2969616280 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc02cb50 con 0x7f45dc15a6b0 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.989+0000 7f45dbe00640 1 -- 172.21.15.141:0/2969616280 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0343f0 con 0x7f45dc15a6b0 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 -- 172.21.15.141:0/2969616280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15a6b0 msgr2=0x7f45dc15aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 --2- 172.21.15.141:0/2969616280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15a6b0 0x7f45dc15aa90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0b3e40 tx=0x7f45cc02be50 comp rx=0 tx=0).stop 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 -- 172.21.15.141:0/2969616280 shutdown_connections 2024-01-16T15:05:18.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 --2- 172.21.15.141:0/2969616280 >> v2:172.21.15.188:3300/0 conn(0x7f45dc0cea00 0x7f45dc15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 --2- 172.21.15.141:0/2969616280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15a6b0 0x7f45dc15aa90 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 -- 172.21.15.141:0/2969616280 >> 172.21.15.141:0/2969616280 conn(0x7f45dc0b0c00 msgr2=0x7f45dc0b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:18.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 -- 172.21.15.141:0/2969616280 shutdown_connections 2024-01-16T15:05:18.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.990+0000 7f45e4000640 1 -- 172.21.15.141:0/2969616280 wait complete. 2024-01-16T15:05:18.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.991+0000 7f45e4000640 1 Processor -- start 2024-01-16T15:05:18.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.992+0000 7f45e4000640 1 -- start start 2024-01-16T15:05:18.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.992+0000 7f45e4000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45dc15a6b0 0x7f45dc150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.992+0000 7f45e4000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15dbf0 0x7f45dc154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e4000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45dc153cd0 0x7f45dc152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e4000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc151000 con 0x7f45dc15dbf0 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e4000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc15c810 con 0x7f45dc15a6b0 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e4000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc155770 con 0x7f45dc153cd0 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15dbf0 0x7f45dc154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15dbf0 0x7f45dc154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57978/0 (socket says 172.21.15.141:57978) 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 -- 172.21.15.141:0/372445 learned_addr learned my addr 172.21.15.141:0/372445 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e1600640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45dc15a6b0 0x7f45dc150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e2000640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45dc153cd0 0x7f45dc152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 -- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45dc153cd0 msgr2=0x7f45dc152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45dc153cd0 0x7f45dc152810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 -- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45dc15a6b0 msgr2=0x7f45dc150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45dc15a6b0 0x7f45dc150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.993+0000 7f45e0c00640 1 -- 172.21.15.141:0/372445 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc009b30 con 0x7f45dc15dbf0 2024-01-16T15:05:18.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.994+0000 7f45e0c00640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15dbf0 0x7f45dc154d40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f45c400d800 tx=0x7f45c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.994+0000 7f45d9600640 1 -- 172.21.15.141:0/372445 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c400b6c0 con 0x7f45dc15dbf0 2024-01-16T15:05:18.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.994+0000 7f45d9600640 1 -- 172.21.15.141:0/372445 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c400b840 con 0x7f45dc15dbf0 2024-01-16T15:05:18.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.994+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc155a50 con 0x7f45dc15dbf0 2024-01-16T15:05:18.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.994+0000 7f45d9600640 1 -- 172.21.15.141:0/372445 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c400bbf0 con 0x7f45dc15dbf0 2024-01-16T15:05:18.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.994+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc156010 con 0x7f45dc15dbf0 2024-01-16T15:05:18.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.995+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc15c9a0 con 0x7f45dc15dbf0 2024-01-16T15:05:18.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.996+0000 7f45d9600640 1 -- 172.21.15.141:0/372445 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f45c400bd50 con 0x7f45dc15dbf0 2024-01-16T15:05:18.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.997+0000 7f45d9600640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f45b406bbc0 0x7f45b406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.997+0000 7f45d9600640 1 -- 172.21.15.141:0/372445 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45c4088c70 con 0x7f45dc15dbf0 2024-01-16T15:05:18.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.998+0000 7f45e1600640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f45b406bbc0 0x7f45b406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:18.999+0000 7f45e1600640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f45b406bbc0 0x7f45b406e080 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f45cc035000 tx=0x7f45cc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.002+0000 7f45d9600640 1 -- 172.21.15.141:0/372445 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45c4055340 con 0x7f45dc15dbf0 2024-01-16T15:05:19.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.221+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f45dc067640 con 0x7f45dc15dbf0 2024-01-16T15:05:19.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.774+0000 7f45d9600640 1 -- 172.21.15.141:0/372445 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 136+0+0 (secure 0 0 0) 0x7f45c4058960 con 0x7f45dc15dbf0 2024-01-16T15:05:19.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f45b406bbc0 msgr2=0x7f45b406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f45b406bbc0 0x7f45b406e080 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f45cc035000 tx=0x7f45cc030000 comp rx=0 tx=0).stop 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15dbf0 msgr2=0x7f45dc154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15dbf0 0x7f45dc154d40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f45c400d800 tx=0x7f45c400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 shutdown_connections 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f45b406bbc0 0x7f45b406e080 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45dc153cd0 0x7f45dc152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45dc15dbf0 0x7f45dc154d40 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 --2- 172.21.15.141:0/372445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45dc15a6b0 0x7f45dc150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.777+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 >> 172.21.15.141:0/372445 conn(0x7f45dc0b0c00 msgr2=0x7f45dc06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.778+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 shutdown_connections 2024-01-16T15:05:19.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:19.778+0000 7f45e4000640 1 -- 172.21.15.141:0/372445 wait complete. 2024-01-16T15:05:19.779 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:05:19.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.006+0000 7f29f9a00640 1 Processor -- start 2024-01-16T15:05:19.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.006+0000 7f29f9a00640 1 -- start start 2024-01-16T15:05:19.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f9a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f29f413df80 0x7f29f415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f9a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f413e360 0x7f29f40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f9a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f29f415f9c0 0x7f29f40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f9a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f406c470 con 0x7f29f413df80 2024-01-16T15:05:19.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f9a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f406c750 con 0x7f29f415f9c0 2024-01-16T15:05:19.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f9a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f406ca30 con 0x7f29f413e360 2024-01-16T15:05:19.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f2a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f29f413df80 0x7f29f415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f2a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f29f413df80 0x7f29f415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57988/0 (socket says 172.21.15.141:57988) 2024-01-16T15:05:19.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.009+0000 7f29f2a00640 1 -- 172.21.15.141:0/3731566624 learned_addr learned my addr 172.21.15.141:0/3731566624 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:19.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.009+0000 7f29ebe00640 1 --2- 172.21.15.141:0/3731566624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f413e360 0x7f29f40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.008+0000 7f29f3400640 1 --2- 172.21.15.141:0/3731566624 >> v2:172.21.15.188:3300/0 conn(0x7f29f415f9c0 0x7f29f40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.009+0000 7f29f2a00640 1 -- 172.21.15.141:0/3731566624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f413e360 msgr2=0x7f29f40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:19.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.009+0000 7f29f2a00640 1 --2- 172.21.15.141:0/3731566624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f413e360 0x7f29f40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.009+0000 7f29f2a00640 1 -- 172.21.15.141:0/3731566624 >> v2:172.21.15.188:3300/0 conn(0x7f29f415f9c0 msgr2=0x7f29f40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:19.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.009+0000 7f29f2a00640 1 --2- 172.21.15.141:0/3731566624 >> v2:172.21.15.188:3300/0 conn(0x7f29f415f9c0 0x7f29f40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.009+0000 7f29f2a00640 1 -- 172.21.15.141:0/3731566624 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f406cd10 con 0x7f29f413df80 2024-01-16T15:05:19.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.011+0000 7f29f2a00640 1 --2- 172.21.15.141:0/3731566624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f415bee0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f29f406e6d0 tx=0x7f29e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a76ad2620ee1da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.011+0000 7f29f2000640 1 -- 172.21.15.141:0/3731566624 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4031870 con 0x7f29f413df80 2024-01-16T15:05:19.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f2000640 1 -- 172.21.15.141:0/3731566624 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e40319d0 con 0x7f29f413df80 2024-01-16T15:05:19.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f2000640 1 -- 172.21.15.141:0/3731566624 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4031d60 con 0x7f29f413df80 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 -- 172.21.15.141:0/3731566624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 msgr2=0x7f29f415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 --2- 172.21.15.141:0/3731566624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f415bee0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f29f406e6d0 tx=0x7f29e402b910 comp rx=0 tx=0).stop 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 -- 172.21.15.141:0/3731566624 shutdown_connections 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 --2- 172.21.15.141:0/3731566624 >> v2:172.21.15.188:3300/0 conn(0x7f29f415f9c0 0x7f29f40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 --2- 172.21.15.141:0/3731566624 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f413e360 0x7f29f40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 --2- 172.21.15.141:0/3731566624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f415bee0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 -- 172.21.15.141:0/3731566624 >> 172.21.15.141:0/3731566624 conn(0x7f29f406a800 msgr2=0x7f29f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 -- 172.21.15.141:0/3731566624 shutdown_connections 2024-01-16T15:05:19.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.012+0000 7f29f9a00640 1 -- 172.21.15.141:0/3731566624 wait complete. 2024-01-16T15:05:19.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.014+0000 7f29f9a00640 1 Processor -- start 2024-01-16T15:05:19.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.014+0000 7f29f9a00640 1 -- start start 2024-01-16T15:05:19.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.015+0000 7f29f9a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f41f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.015+0000 7f29f9a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29f413e360 0x7f29f40aa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.015+0000 7f29f9a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f415f9c0 0x7f29f40af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.015+0000 7f29f9a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f41f3710 con 0x7f29f413df80 2024-01-16T15:05:19.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.015+0000 7f29f9a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40aa950 con 0x7f29f413e360 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.015+0000 7f29f9a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f41f30c0 con 0x7f29f415f9c0 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.015+0000 7f29f2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f41f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f41f3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58004/0 (socket says 172.21.15.141:58004) 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 -- 172.21.15.141:0/375616004 learned_addr learned my addr 172.21.15.141:0/375616004 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 -- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f415f9c0 msgr2=0x7f29f40af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f415f9c0 0x7f29f40af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 -- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29f413e360 msgr2=0x7f29f40aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29ebe00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29f413e360 0x7f29f40aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29f413e360 0x7f29f40aa000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 -- 172.21.15.141:0/375616004 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40094d0 con 0x7f29f413df80 2024-01-16T15:05:19.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29f2a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f41f3ba0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f29e402b950 tx=0x7f29e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.016+0000 7f29eaa00640 1 -- 172.21.15.141:0/375616004 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e403abe0 con 0x7f29f413df80 2024-01-16T15:05:19.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.017+0000 7f29eaa00640 1 -- 172.21.15.141:0/375616004 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e403ad40 con 0x7f29f413df80 2024-01-16T15:05:19.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.017+0000 7f29eaa00640 1 -- 172.21.15.141:0/375616004 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4044630 con 0x7f29f413df80 2024-01-16T15:05:19.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.017+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f41f33a0 con 0x7f29f413df80 2024-01-16T15:05:19.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.017+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f40aa540 con 0x7f29f413df80 2024-01-16T15:05:19.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.018+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f40675d0 con 0x7f29f413df80 2024-01-16T15:05:19.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.019+0000 7f29eaa00640 1 -- 172.21.15.141:0/375616004 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f29e404d450 con 0x7f29f413df80 2024-01-16T15:05:19.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.019+0000 7f29eaa00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29cc06bbf0 0x7f29cc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.020+0000 7f29eaa00640 1 -- 172.21.15.141:0/375616004 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29e402d040 con 0x7f29f413df80 2024-01-16T15:05:19.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.020+0000 7f29ebe00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29cc06bbf0 0x7f29cc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.021+0000 7f29ebe00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29cc06bbf0 0x7f29cc06e0b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f29f40a9a10 tx=0x7f29dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.024+0000 7f29eaa00640 1 -- 172.21.15.141:0/375616004 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29e40771d0 con 0x7f29f413df80 2024-01-16T15:05:20.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.231+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f29f41408d0 con 0x7f29cc06bbf0 2024-01-16T15:05:20.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.235+0000 7f29eaa00640 1 -- 172.21.15.141:0/375616004 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f29dc010840 con 0x7f29cc06bbf0 2024-01-16T15:05:20.214 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:05:20.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:05:20.214 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:05:20.214 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:05:20.214 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:05:20.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29cc06bbf0 msgr2=0x7f29cc06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29cc06bbf0 0x7f29cc06e0b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f29f40a9a10 tx=0x7f29dc008040 comp rx=0 tx=0).stop 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 msgr2=0x7f29f41f3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f41f3ba0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f29e402b950 tx=0x7f29e40046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 shutdown_connections 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29cc06bbf0 0x7f29cc06e0b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f29f415f9c0 0x7f29f40af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29f413e360 0x7f29f40aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 --2- 172.21.15.141:0/375616004 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29f413df80 0x7f29f41f3ba0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 >> 172.21.15.141:0/375616004 conn(0x7f29f406a800 msgr2=0x7f29f406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 shutdown_connections 2024-01-16T15:05:20.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.237+0000 7f29f9a00640 1 -- 172.21.15.141:0/375616004 wait complete. 2024-01-16T15:05:20.234 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:05:20.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.446+0000 7fefbbc00640 1 Processor -- start 2024-01-16T15:05:20.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.446+0000 7fefbbc00640 1 -- start start 2024-01-16T15:05:20.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.447+0000 7fefbbc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 0x7fefb4166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefbbc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb415c5f0 con 0x7fefb413df80 2024-01-16T15:05:20.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefbbc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb406c470 con 0x7fefb413e360 2024-01-16T15:05:20.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefbbc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb406c750 con 0x7fefb40cf9a0 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefb9200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fefb413df80 0x7fefb415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:51974/0 (socket says 172.21.15.141:51974) 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefb9200640 1 -- 172.21.15.141:0/1220503336 learned_addr learned my addr 172.21.15.141:0/1220503336 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919472191 0 0) 0x7fefb415c5f0 con 0x7fefb413df80 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94003520 con 0x7fefb413df80 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.448+0000 7fefb9c00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 0x7fefb4166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738623896 0 0) 0x7fef94003520 con 0x7fefb413df80 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 msgr2=0x7fefb4166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 0x7fefb4166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 >> v1:172.21.15.188:6789/0 conn(0x7fefb413e360 legacy=0x7fefb40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb406ca30 con 0x7fefb413df80 2024-01-16T15:05:20.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fefa4001670 con 0x7fefb413df80 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 >> v1:172.21.15.141:6789/0 conn(0x7fefb413df80 legacy=0x7fefb415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 0x7fef940039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef94004020 0x7fef940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef94004d00 0x7fef94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb406c750 con 0x7fef94004020 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9400bbc0 con 0x7fef94004d00 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9400bd80 con 0x7fefb40cf9a0 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefabe00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef94004020 0x7fef940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefb9200640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 0x7fef940039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefabe00640 1 -- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 msgr2=0x7fef940039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefabe00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fefb40cf9a0 0x7fef940039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefabe00640 1 -- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef94004d00 msgr2=0x7fef94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefabe00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef94004d00 0x7fef94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.449+0000 7fefabe00640 1 -- 172.21.15.141:0/1220503336 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94003520 con 0x7fef94004020 2024-01-16T15:05:20.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.450+0000 7fefabe00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef94004020 0x7fef940047c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fef9c001720 tx=0x7fef940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e71a9d4b6fc864 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0052d0 con 0x7fef94004020 2024-01-16T15:05:20.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9c02e050 con 0x7fef94004020 2024-01-16T15:05:20.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefb3e00640 1 -- 172.21.15.141:0/1220503336 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c003830 con 0x7fef94004020 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefbbc00640 1 -- 172.21.15.141:0/1220503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef94004020 msgr2=0x7fef940047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefbbc00640 1 --2- 172.21.15.141:0/1220503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef94004020 0x7fef940047c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fef9c001720 tx=0x7fef940057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefb9c00640 1 -- 172.21.15.141:0/1220503336 reap_dead start 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefbbc00640 1 -- 172.21.15.141:0/1220503336 shutdown_connections 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefbbc00640 1 -- 172.21.15.141:0/1220503336 >> 172.21.15.141:0/1220503336 conn(0x7fefb406a800 msgr2=0x7fefb406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefbbc00640 1 -- 172.21.15.141:0/1220503336 shutdown_connections 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.451+0000 7fefbbc00640 1 -- 172.21.15.141:0/1220503336 wait complete. 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.453+0000 7fefbbc00640 1 Processor -- start 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.453+0000 7fefbbc00640 1 -- start start 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefbbc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef94004020 0x7fefb413a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefbbc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb413d510 0x7fefb413afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefbbc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefb414d550 0x7fefb414b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefbbc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40b33a0 con 0x7fefb414d550 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefbbc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40b2770 con 0x7fefb413d510 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefbbc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb413ad70 con 0x7fef94004020 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefb9c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefb414d550 0x7fefb414b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefb9c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefb414d550 0x7fefb414b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58024/0 (socket says 172.21.15.141:58024) 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.454+0000 7fefb9c00640 1 -- 172.21.15.141:0/272159954 learned_addr learned my addr 172.21.15.141:0/272159954 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9200640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef94004020 0x7fefb413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefabe00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb413d510 0x7fefb413afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9c00640 1 -- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef94004020 msgr2=0x7fefb413a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9c00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef94004020 0x7fefb413a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9c00640 1 -- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb413d510 msgr2=0x7fefb413afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9c00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb413d510 0x7fefb413afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9c00640 1 -- 172.21.15.141:0/272159954 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb40b2a00 con 0x7fefb414d550 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9200640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef94004020 0x7fefb413a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb9c00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefb414d550 0x7fefb414b3f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fefb40cfea0 tx=0x7fefac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb1600640 1 -- 172.21.15.141:0/272159954 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac018070 con 0x7fefb414d550 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb1600640 1 -- 172.21.15.141:0/272159954 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac00eb90 con 0x7fefb414d550 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefb1600640 1 -- 172.21.15.141:0/272159954 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac003990 con 0x7fefb414d550 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.455+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb40b2ce0 con 0x7fefb414d550 2024-01-16T15:05:20.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.456+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb413b4e0 con 0x7fefb414d550 2024-01-16T15:05:20.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.457+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb40675d0 con 0x7fefb414d550 2024-01-16T15:05:20.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.458+0000 7fefb1600640 1 -- 172.21.15.141:0/272159954 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fefac014420 con 0x7fefb414d550 2024-01-16T15:05:20.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.458+0000 7fefb1600640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef8c06bbc0 0x7fef8c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.458+0000 7fefb1600640 1 -- 172.21.15.141:0/272159954 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefac088370 con 0x7fefb414d550 2024-01-16T15:05:20.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.459+0000 7fefb9200640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef8c06bbc0 0x7fef8c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.460+0000 7fefb9200640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef8c06bbc0 0x7fef8c06e080 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fefa4007970 tx=0x7fefa4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.462+0000 7fefb1600640 1 -- 172.21.15.141:0/272159954 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefac08ce60 con 0x7fefb414d550 2024-01-16T15:05:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.651+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fefb406df00 con 0x7fefb414d550 2024-01-16T15:05:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.652+0000 7fefb1600640 1 -- 172.21.15.141:0/272159954 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fefac054c30 con 0x7fefb414d550 2024-01-16T15:05:20.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef8c06bbc0 msgr2=0x7fef8c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef8c06bbc0 0x7fef8c06e080 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fefa4007970 tx=0x7fefa4009040 comp rx=0 tx=0).stop 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefb414d550 msgr2=0x7fefb414b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefb414d550 0x7fefb414b3f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fefb40cfea0 tx=0x7fefac00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 shutdown_connections 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef8c06bbc0 0x7fef8c06e080 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fefb414d550 0x7fefb414b3f0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb413d510 0x7fefb413afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 --2- 172.21.15.141:0/272159954 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef94004020 0x7fefb413a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.654+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 >> 172.21.15.141:0/272159954 conn(0x7fefb406a800 msgr2=0x7fefb415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.655+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 shutdown_connections 2024-01-16T15:05:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.655+0000 7fefbbc00640 1 -- 172.21.15.141:0/272159954 wait complete. 2024-01-16T15:05:20.653 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:05:20.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.881+0000 7f8534c00640 1 Processor -- start 2024-01-16T15:05:20.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.882+0000 7f8534c00640 1 -- start start 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.883+0000 7f8534c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8530160920 0x7f85300b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.883+0000 7f8534c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f85300cf9f0 0x7f85300cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.883+0000 7f8534c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85300d0a10 con 0x7f8530161010 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.883+0000 7f8534c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85300d03e0 con 0x7f8530160920 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.883+0000 7f8534c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85300d1f50 con 0x7f85300cf9f0 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.883+0000 7f852f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8530160920 0x7f85300b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.884+0000 7f852f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8530160920 0x7f85300b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58032/0 (socket says 172.21.15.141:58032) 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.884+0000 7f852f400640 1 -- 172.21.15.141:0/2459515638 learned_addr learned my addr 172.21.15.141:0/2459515638 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.884+0000 7f852ea00640 1 --2- 172.21.15.141:0/2459515638 >> v2:172.21.15.188:3300/0 conn(0x7f85300cf9f0 0x7f85300cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.884+0000 7f852f400640 1 -- 172.21.15.141:0/2459515638 >> v2:172.21.15.188:3300/0 conn(0x7f85300cf9f0 msgr2=0x7f85300cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.884+0000 7f852f400640 1 --2- 172.21.15.141:0/2459515638 >> v2:172.21.15.188:3300/0 conn(0x7f85300cf9f0 0x7f85300cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.884+0000 7f852f400640 1 -- 172.21.15.141:0/2459515638 >> v1:172.21.15.141:6789/0 conn(0x7f8530161010 legacy=0x7f8530161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.884+0000 7f852f400640 1 -- 172.21.15.141:0/2459515638 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85300cf080 con 0x7f8530160920 2024-01-16T15:05:20.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.885+0000 7f852f400640 1 --2- 172.21.15.141:0/2459515638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8530160920 0x7f85300b3f30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f853008c1f0 tx=0x7f852002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e542e0257f5374 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.885+0000 7f852e000640 1 -- 172.21.15.141:0/2459515638 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520030860 con 0x7f8530160920 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f852e000640 1 -- 172.21.15.141:0/2459515638 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85200309c0 con 0x7f8530160920 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f852e000640 1 -- 172.21.15.141:0/2459515638 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520030d50 con 0x7f8530160920 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 -- 172.21.15.141:0/2459515638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8530160920 msgr2=0x7f85300b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 --2- 172.21.15.141:0/2459515638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8530160920 0x7f85300b3f30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f853008c1f0 tx=0x7f852002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 -- 172.21.15.141:0/2459515638 shutdown_connections 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 --2- 172.21.15.141:0/2459515638 >> v2:172.21.15.188:3300/0 conn(0x7f85300cf9f0 0x7f85300cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 --2- 172.21.15.141:0/2459515638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8530160920 0x7f85300b3f30 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 -- 172.21.15.141:0/2459515638 >> 172.21.15.141:0/2459515638 conn(0x7f85300d3090 msgr2=0x7f85300d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 -- 172.21.15.141:0/2459515638 shutdown_connections 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.886+0000 7f8534c00640 1 -- 172.21.15.141:0/2459515638 wait complete. 2024-01-16T15:05:20.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.889+0000 7f8534c00640 1 Processor -- start 2024-01-16T15:05:20.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.889+0000 7f8534c00640 1 -- start start 2024-01-16T15:05:20.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f8534c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85300cf9f0 0x7f853014d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f8534c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8530160920 0x7f85301597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f8534c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8530161010 0x7f8530159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f8534c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85300b7800 con 0x7f85300cf9f0 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f8534c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853014d9d0 con 0x7f8530161010 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f8534c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85300b6b80 con 0x7f8530160920 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f852f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85300cf9f0 0x7f853014d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f852f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85300cf9f0 0x7f853014d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58044/0 (socket says 172.21.15.141:58044) 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.890+0000 7f852f400640 1 -- 172.21.15.141:0/2254578939 learned_addr learned my addr 172.21.15.141:0/2254578939 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852ea00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8530160920 0x7f85301597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852f400640 1 -- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8530160920 msgr2=0x7f85301597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852f400640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8530160920 0x7f85301597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852f400640 1 -- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8530161010 msgr2=0x7f8530159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852fe00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8530161010 0x7f8530159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852f400640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8530161010 0x7f8530159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852f400640 1 -- 172.21.15.141:0/2254578939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85200094d0 con 0x7f85300cf9f0 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f852f400640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85300cf9f0 0x7f853014d490 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f852002b9e0 tx=0x7f8520004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f8527400640 1 -- 172.21.15.141:0/2254578939 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520030850 con 0x7f85300cf9f0 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f8527400640 1 -- 172.21.15.141:0/2254578939 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8520043400 con 0x7f85300cf9f0 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85300b6e60 con 0x7f85300cf9f0 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.891+0000 7f8527400640 1 -- 172.21.15.141:0/2254578939 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200437b0 con 0x7f85300cf9f0 2024-01-16T15:05:20.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.892+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85300b7420 con 0x7f85300cf9f0 2024-01-16T15:05:20.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.892+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85301f3060 con 0x7f85300cf9f0 2024-01-16T15:05:20.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.893+0000 7f8527400640 1 -- 172.21.15.141:0/2254578939 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f8520043910 con 0x7f85300cf9f0 2024-01-16T15:05:20.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.894+0000 7f8527400640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850806bbc0 0x7f850806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.894+0000 7f8527400640 1 -- 172.21.15.141:0/2254578939 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f852003e070 con 0x7f85300cf9f0 2024-01-16T15:05:20.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.895+0000 7f852ea00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850806bbc0 0x7f850806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.896+0000 7f852ea00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850806bbc0 0x7f850806e080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f85300cb860 tx=0x7f8518008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:20.899+0000 7f8527400640 1 -- 172.21.15.141:0/2254578939 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85301f3060 con 0x7f85300cf9f0 2024-01-16T15:05:21.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.098+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f85300ceac0 con 0x7f850806bbc0 2024-01-16T15:05:21.088 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:05:21.109+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:21.088 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:05:21.109+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:21.089 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:05:21.109+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:21.089 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:05:21.109+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:21.089 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:05:21.109+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:21.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.171+0000 7f8527400640 1 -- 172.21.15.141:0/2254578939 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f851800c040 con 0x7f850806bbc0 2024-01-16T15:05:21.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850806bbc0 msgr2=0x7f850806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850806bbc0 0x7f850806e080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f85300cb860 tx=0x7f8518008040 comp rx=0 tx=0).stop 2024-01-16T15:05:21.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85300cf9f0 msgr2=0x7f853014d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85300cf9f0 0x7f853014d490 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f852002b9e0 tx=0x7f8520004730 comp rx=0 tx=0).stop 2024-01-16T15:05:21.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 shutdown_connections 2024-01-16T15:05:21.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850806bbc0 0x7f850806e080 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8530161010 0x7f8530159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8530160920 0x7f85301597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 --2- 172.21.15.141:0/2254578939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85300cf9f0 0x7f853014d490 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 >> 172.21.15.141:0/2254578939 conn(0x7f85300d3090 msgr2=0x7f853006c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:21.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 shutdown_connections 2024-01-16T15:05:21.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.175+0000 7f8534c00640 1 -- 172.21.15.141:0/2254578939 wait complete. 2024-01-16T15:05:21.172 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:05:21.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.386+0000 7f215c800640 1 Processor -- start 2024-01-16T15:05:21.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.386+0000 7f215c800640 1 -- start start 2024-01-16T15:05:21.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.387+0000 7f215c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 0x7f215415d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.387+0000 7f215c800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2154150ba0 con 0x7f215415d840 2024-01-16T15:05:21.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.388+0000 7f215c800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2154071190 con 0x7f2154072230 2024-01-16T15:05:21.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.388+0000 7f215c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2154072610 con 0x7f2154150760 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.388+0000 7f215a800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f215415d840 0x7f21540b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:51996/0 (socket says 172.21.15.141:51996) 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.388+0000 7f215a800640 1 -- 172.21.15.141:0/1852882606 learned_addr learned my addr 172.21.15.141:0/1852882606 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.388+0000 7f2159400640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 0x7f215415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.388+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029120509 0 0) 0x7f2154150ba0 con 0x7f215415d840 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.388+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2134005730 con 0x7f215415d840 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3652999978 0 0) 0x7f2134005730 con 0x7f215415d840 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 msgr2=0x7f215415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 0x7f215415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 >> v1:172.21.15.188:6789/0 conn(0x7f2154072230 legacy=0x7f21540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2154071470 con 0x7f215415d840 2024-01-16T15:05:21.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f214c0019f0 con 0x7f215415d840 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 >> v1:172.21.15.141:6789/0 conn(0x7f215415d840 legacy=0x7f21540b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2134005bb0 0x7f2134005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f21340069d0 0x7f2134006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 0x7f2134007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2154072610 con 0x7f21340069d0 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213400dc40 con 0x7f2134005bb0 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213400de00 con 0x7f2154150760 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2159400640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f21340069d0 0x7f2134006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f215a800640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 0x7f2134007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2159400640 1 -- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 msgr2=0x7f2134007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2159400640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2154150760 0x7f2134007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2159400640 1 -- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2134005bb0 msgr2=0x7f2134005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:21.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.389+0000 7f2159e00640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2134005bb0 0x7f2134005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.390+0000 7f2159400640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2134005bb0 0x7f2134005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.390+0000 7f2159400640 1 -- 172.21.15.141:0/1852882606 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2134005730 con 0x7f21340069d0 2024-01-16T15:05:21.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.390+0000 7f2159e00640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2134005bb0 0x7f2134005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:21.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f2159400640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f21340069d0 0x7f2134006e50 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f213c009a90 tx=0x7f2134007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c248a8b15773669 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:21.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c02c660 con 0x7f21340069d0 2024-01-16T15:05:21.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c02c7c0 con 0x7f21340069d0 2024-01-16T15:05:21.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f2158a00640 1 -- 172.21.15.141:0/1852882606 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c02cb50 con 0x7f21340069d0 2024-01-16T15:05:21.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f215c800640 1 -- 172.21.15.141:0/1852882606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f21340069d0 msgr2=0x7f2134006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f215c800640 1 --2- 172.21.15.141:0/1852882606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f21340069d0 0x7f2134006e50 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f213c009a90 tx=0x7f2134007a00 comp rx=0 tx=0).stop 2024-01-16T15:05:21.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f215a800640 1 -- 172.21.15.141:0/1852882606 reap_dead start 2024-01-16T15:05:21.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f215c800640 1 -- 172.21.15.141:0/1852882606 shutdown_connections 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f215c800640 1 -- 172.21.15.141:0/1852882606 >> 172.21.15.141:0/1852882606 conn(0x7f215406a800 msgr2=0x7f215406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f215c800640 1 -- 172.21.15.141:0/1852882606 shutdown_connections 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.391+0000 7f215c800640 1 -- 172.21.15.141:0/1852882606 wait complete. 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.393+0000 7f215c800640 1 Processor -- start 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.393+0000 7f215c800640 1 -- start start 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215c800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21340069d0 0x7f21540ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f21540afe10 0x7f21540ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215c800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2154147010 0x7f21541588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215c800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540cd530 con 0x7f2154147010 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215c800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540ace10 con 0x7f21340069d0 2024-01-16T15:05:21.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21540ad0b0 con 0x7f21540afe10 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2154147010 0x7f21541588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2154147010 0x7f21541588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58074/0 (socket says 172.21.15.141:58074) 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.394+0000 7f215a800640 1 -- 172.21.15.141:0/4166439516 learned_addr learned my addr 172.21.15.141:0/4166439516 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f2159e00640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21340069d0 0x7f21540ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f2159400640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f21540afe10 0x7f21540ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f215a800640 1 -- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f21540afe10 msgr2=0x7f21540ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f215a800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f21540afe10 0x7f21540ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f215a800640 1 -- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21340069d0 msgr2=0x7f21540ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f215a800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21340069d0 0x7f21540ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f215a800640 1 -- 172.21.15.141:0/4166439516 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21540ad390 con 0x7f2154147010 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f2159e00640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21340069d0 0x7f21540ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:21.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f215a800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2154147010 0x7f21541588f0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f21540cb740 tx=0x7f214c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:21.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f2152000640 1 -- 172.21.15.141:0/4166439516 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214c00de70 con 0x7f2154147010 2024-01-16T15:05:21.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f2152000640 1 -- 172.21.15.141:0/4166439516 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214c00bb80 con 0x7f2154147010 2024-01-16T15:05:21.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f2152000640 1 -- 172.21.15.141:0/4166439516 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214c007110 con 0x7f2154147010 2024-01-16T15:05:21.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.395+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21540ad670 con 0x7f2154147010 2024-01-16T15:05:21.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.396+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21541474c0 con 0x7f2154147010 2024-01-16T15:05:21.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.396+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21540cd9c0 con 0x7f2154147010 2024-01-16T15:05:21.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.398+0000 7f2152000640 1 -- 172.21.15.141:0/4166439516 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f214c007270 con 0x7f2154147010 2024-01-16T15:05:21.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.398+0000 7f2152000640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f212c06bbc0 0x7f212c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.399+0000 7f2152000640 1 -- 172.21.15.141:0/4166439516 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f214c088900 con 0x7f2154147010 2024-01-16T15:05:21.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.399+0000 7f2159e00640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f212c06bbc0 0x7f212c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.400+0000 7f2159e00640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f212c06bbc0 0x7f212c06e080 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f21440016f0 tx=0x7f2144009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:21.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.403+0000 7f2152000640 1 -- 172.21.15.141:0/4166439516 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21540cd9c0 con 0x7f2154147010 2024-01-16T15:05:21.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.607+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f21541476b0 con 0x7f212c06bbc0 2024-01-16T15:05:21.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.618+0000 7f2152000640 1 -- 172.21.15.141:0/4166439516 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f21541476b0 con 0x7f212c06bbc0 2024-01-16T15:05:21.597 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934 2024-01-16T15:05:21.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f212c06bbc0 msgr2=0x7f212c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f212c06bbc0 0x7f212c06e080 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f21440016f0 tx=0x7f2144009040 comp rx=0 tx=0).stop 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2154147010 msgr2=0x7f21541588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2154147010 0x7f21541588f0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f21540cb740 tx=0x7f214c00b4a0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 shutdown_connections 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f212c06bbc0 0x7f212c06e080 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2154147010 0x7f21541588f0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f21540afe10 0x7f21540ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 --2- 172.21.15.141:0/4166439516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21340069d0 0x7f21540ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.620+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 >> 172.21.15.141:0/4166439516 conn(0x7f215406a800 msgr2=0x7f2154147940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:21.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.621+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 shutdown_connections 2024-01-16T15:05:21.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:21.621+0000 7f215c800640 1 -- 172.21.15.141:0/4166439516 wait complete. 2024-01-16T15:05:21.617 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934 2024-01-16T15:05:21.617 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:05:21.617 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:21.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:21.668 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:34.404 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:34.413 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:05:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:05:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.9977 s, 76.7 MB/s 2024-01-16T15:05:35.735 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:05:35.736 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:35.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:35.787 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:47.496 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:05:47.496 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:05:47.496 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.6533 s, 92.1 MB/s 2024-01-16T15:05:47.498 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:05:47.498 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:47.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:47.553 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:58.379 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:05:58.379 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:05:58.379 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.7682 s, 99.7 MB/s 2024-01-16T15:05:58.381 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 10 2024-01-16T15:05:58.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.621+0000 7f3647c00640 1 Processor -- start 2024-01-16T15:05:58.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.621+0000 7f3647c00640 1 -- start start 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f3647c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f364013e360 0x7f36400cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f3647c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364015c5f0 con 0x7f364015f9c0 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f3647c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364006c470 con 0x7f364013df80 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f3647c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364006c750 con 0x7f364013e360 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f363fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f364013e360 0x7f36400cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f363fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f364013e360 0x7f36400cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39928/0 (socket says 172.21.15.141:39928) 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f363fe00640 1 -- 172.21.15.141:0/3027855808 learned_addr learned my addr 172.21.15.141:0/3027855808 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:58.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f3645c00640 1 --1- 172.21.15.141:0/3027855808 >> v1:172.21.15.141:6789/0 conn(0x7f364015f9c0 0x7f36400cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54058/0 (socket says 172.21.15.141:54058) 2024-01-16T15:05:58.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f363fe00640 1 -- 172.21.15.141:0/3027855808 >> v1:172.21.15.188:6789/0 conn(0x7f364013df80 legacy=0x7f364015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:58.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f363fe00640 1 -- 172.21.15.141:0/3027855808 >> v1:172.21.15.141:6789/0 conn(0x7f364015f9c0 legacy=0x7f36400cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:58.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.622+0000 7f363fe00640 1 -- 172.21.15.141:0/3027855808 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364006ca30 con 0x7f364013e360 2024-01-16T15:05:58.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f363fe00640 1 --2- 172.21.15.141:0/3027855808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013e360 0x7f36400cf290 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3628009870 tx=0x7f362802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63baacf7da7a5495 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:58.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f363f400640 1 -- 172.21.15.141:0/3027855808 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628030850 con 0x7f364013e360 2024-01-16T15:05:58.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f363f400640 1 -- 172.21.15.141:0/3027855808 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36280309b0 con 0x7f364013e360 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f363f400640 1 -- 172.21.15.141:0/3027855808 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628030d40 con 0x7f364013e360 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f3647c00640 1 -- 172.21.15.141:0/3027855808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013e360 msgr2=0x7f36400cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f3647c00640 1 --2- 172.21.15.141:0/3027855808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013e360 0x7f36400cf290 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3628009870 tx=0x7f362802b990 comp rx=0 tx=0).stop 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f3647c00640 1 -- 172.21.15.141:0/3027855808 shutdown_connections 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f3647c00640 1 --2- 172.21.15.141:0/3027855808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013e360 0x7f36400cf290 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.624+0000 7f3647c00640 1 -- 172.21.15.141:0/3027855808 >> 172.21.15.141:0/3027855808 conn(0x7f364006a800 msgr2=0x7f364006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.625+0000 7f3647c00640 1 -- 172.21.15.141:0/3027855808 shutdown_connections 2024-01-16T15:05:58.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.625+0000 7f3647c00640 1 -- 172.21.15.141:0/3027855808 wait complete. 2024-01-16T15:05:58.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.626+0000 7f3647c00640 1 Processor -- start 2024-01-16T15:05:58.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.626+0000 7f3647c00640 1 -- start start 2024-01-16T15:05:58.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.627+0000 7f3647c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013df80 0x7f364014d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:58.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.627+0000 7f3647c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f364013e360 0x7f3640152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.627+0000 7f3647c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364015f9c0 0x7f3640152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.627+0000 7f3647c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400b3270 con 0x7f364013df80 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.627+0000 7f3647c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364014de70 con 0x7f364015f9c0 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.627+0000 7f3647c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364014ac80 con 0x7f364013e360 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013df80 0x7f364014d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013df80 0x7f364014d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39944/0 (socket says 172.21.15.141:39944) 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 -- 172.21.15.141:0/3834263667 learned_addr learned my addr 172.21.15.141:0/3834263667 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 -- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f364013e360 msgr2=0x7f3640152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:58.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f363fe00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f364013e360 0x7f3640152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645c00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364015f9c0 0x7f3640152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f364013e360 0x7f3640152650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 -- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364015f9c0 msgr2=0x7f3640152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364015f9c0 0x7f3640152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 -- 172.21.15.141:0/3834263667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36280094d0 con 0x7f364013df80 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.628+0000 7f3645200640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013df80 0x7f364014d930 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f363000b3f0 tx=0x7f363000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.629+0000 7f363cc00640 1 -- 172.21.15.141:0/3834263667 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630011830 con 0x7f364013df80 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.629+0000 7f363cc00640 1 -- 172.21.15.141:0/3834263667 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3630011990 con 0x7f364013df80 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.629+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364014af60 con 0x7f364013df80 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.629+0000 7f363cc00640 1 -- 172.21.15.141:0/3834263667 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630011d40 con 0x7f364013df80 2024-01-16T15:05:58.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.629+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3640130a30 con 0x7f364013df80 2024-01-16T15:05:58.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.630+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364014b150 con 0x7f364013df80 2024-01-16T15:05:58.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.631+0000 7f363cc00640 1 -- 172.21.15.141:0/3834263667 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f363001a400 con 0x7f364013df80 2024-01-16T15:05:58.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.632+0000 7f363cc00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361806bc50 0x7f361806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:58.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.632+0000 7f363cc00640 1 -- 172.21.15.141:0/3834263667 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3630088520 con 0x7f364013df80 2024-01-16T15:05:58.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.633+0000 7f363fe00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361806bc50 0x7f361806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:58.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.634+0000 7f363fe00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361806bc50 0x7f361806e110 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3628036000 tx=0x7f362803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:58.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.637+0000 7f363cc00640 1 -- 172.21.15.141:0/3834263667 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f363008e030 con 0x7f364013df80 2024-01-16T15:05:58.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.828+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f36400675d0 con 0x7f364013df80 2024-01-16T15:05:58.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.837+0000 7f363cc00640 1 -- 172.21.15.141:0/3834263667 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3630054be0 con 0x7f364013df80 2024-01-16T15:05:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361806bc50 msgr2=0x7f361806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361806bc50 0x7f361806e110 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3628036000 tx=0x7f362803b040 comp rx=0 tx=0).stop 2024-01-16T15:05:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013df80 msgr2=0x7f364014d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013df80 0x7f364014d930 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f363000b3f0 tx=0x7f363000b8b0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 shutdown_connections 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f361806bc50 0x7f361806e110 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364015f9c0 0x7f3640152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f364013e360 0x7f3640152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 --2- 172.21.15.141:0/3834263667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364013df80 0x7f364014d930 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.840+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 >> 172.21.15.141:0/3834263667 conn(0x7f364006a800 msgr2=0x7f364015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.841+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 shutdown_connections 2024-01-16T15:05:58.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:58.841+0000 7f3647c00640 1 -- 172.21.15.141:0/3834263667 wait complete. 2024-01-16T15:05:58.837 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:05:59.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.064+0000 7fc667a00640 1 Processor -- start 2024-01-16T15:05:59.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.064+0000 7fc667a00640 1 -- start start 2024-01-16T15:05:59.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.066+0000 7fc667a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc66006c6b0 0x7fc6600b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.066+0000 7fc667a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006cea0 con 0x7fc66009a170 2024-01-16T15:05:59.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.066+0000 7fc667a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006cbc0 con 0x7fc6600cd240 2024-01-16T15:05:59.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.066+0000 7fc667a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66006f350 con 0x7fc66006c6b0 2024-01-16T15:05:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.066+0000 7fc665a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc66006c6b0 0x7fc6600b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.066+0000 7fc65fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc66009a170 0x7fc6600cf640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54070/0 (socket says 172.21.15.141:54070) 2024-01-16T15:05:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.066+0000 7fc65fe00640 1 -- 172.21.15.141:0/4006655155 learned_addr learned my addr 172.21.15.141:0/4006655155 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.067+0000 7fc665a00640 1 -- 172.21.15.141:0/4006655155 >> v1:172.21.15.188:6789/0 conn(0x7fc6600cd240 legacy=0x7fc660099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.067+0000 7fc665a00640 1 -- 172.21.15.141:0/4006655155 >> v1:172.21.15.141:6789/0 conn(0x7fc66009a170 legacy=0x7fc6600cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.067+0000 7fc65f400640 1 -- 172.21.15.141:0/4006655155 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821205792 0 0) 0x7fc66006cea0 con 0x7fc66009a170 2024-01-16T15:05:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.067+0000 7fc665a00640 1 -- 172.21.15.141:0/4006655155 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66006f630 con 0x7fc66006c6b0 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.068+0000 7fc665a00640 1 --2- 172.21.15.141:0/4006655155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66006c6b0 0x7fc6600b3820 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc6580097a0 tx=0x7fc65802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff79d7d732d2e68 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.068+0000 7fc65f400640 1 -- 172.21.15.141:0/4006655155 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658031830 con 0x7fc66006c6b0 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.068+0000 7fc65f400640 1 -- 172.21.15.141:0/4006655155 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc658031990 con 0x7fc66006c6b0 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.068+0000 7fc65f400640 1 -- 172.21.15.141:0/4006655155 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658031d20 con 0x7fc66006c6b0 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.069+0000 7fc667a00640 1 -- 172.21.15.141:0/4006655155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66006c6b0 msgr2=0x7fc6600b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.069+0000 7fc667a00640 1 --2- 172.21.15.141:0/4006655155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66006c6b0 0x7fc6600b3820 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc6580097a0 tx=0x7fc65802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.069+0000 7fc667a00640 1 -- 172.21.15.141:0/4006655155 shutdown_connections 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.069+0000 7fc667a00640 1 --2- 172.21.15.141:0/4006655155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66006c6b0 0x7fc6600b3820 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.069+0000 7fc667a00640 1 -- 172.21.15.141:0/4006655155 >> 172.21.15.141:0/4006655155 conn(0x7fc6600b2d00 msgr2=0x7fc6600b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.069+0000 7fc667a00640 1 -- 172.21.15.141:0/4006655155 shutdown_connections 2024-01-16T15:05:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.069+0000 7fc667a00640 1 -- 172.21.15.141:0/4006655155 wait complete. 2024-01-16T15:05:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.071+0000 7fc667a00640 1 Processor -- start 2024-01-16T15:05:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.071+0000 7fc667a00640 1 -- start start 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc667a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc66006c6b0 0x7fc66015b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc667a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66009a170 0x7fc6600c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc667a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6600cd240 0x7fc6601f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc667a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c4d20 con 0x7fc66009a170 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc667a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6601f70f0 con 0x7fc66006c6b0 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc667a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c4f70 con 0x7fc6600cd240 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc65fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66009a170 0x7fc6600c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc65fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66009a170 0x7fc6600c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39962/0 (socket says 172.21.15.141:39962) 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc65fe00640 1 -- 172.21.15.141:0/2161457865 learned_addr learned my addr 172.21.15.141:0/2161457865 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc65fe00640 1 -- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6600cd240 msgr2=0x7fc6601f6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc65fe00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6600cd240 0x7fc6601f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc65fe00640 1 -- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc66006c6b0 msgr2=0x7fc66015b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.072+0000 7fc665000640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc66006c6b0 0x7fc66015b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc65fe00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc66006c6b0 0x7fc66015b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc65fe00640 1 -- 172.21.15.141:0/2161457865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658009400 con 0x7fc66009a170 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc65fe00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66009a170 0x7fc6600c73f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc64800c890 tx=0x7fc64800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc65cc00640 1 -- 172.21.15.141:0/2161457865 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64800a640 con 0x7fc66009a170 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc65cc00640 1 -- 172.21.15.141:0/2161457865 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64800a7c0 con 0x7fc66009a170 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc65cc00640 1 -- 172.21.15.141:0/2161457865 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64800ab70 con 0x7fc66009a170 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6600c5250 con 0x7fc66009a170 2024-01-16T15:05:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.073+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6600c7930 con 0x7fc66009a170 2024-01-16T15:05:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.075+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6600c5770 con 0x7fc66009a170 2024-01-16T15:05:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.075+0000 7fc65cc00640 1 -- 172.21.15.141:0/2161457865 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fc64800acd0 con 0x7fc66009a170 2024-01-16T15:05:59.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.076+0000 7fc65cc00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc63806bc50 0x7fc63806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.076+0000 7fc65cc00640 1 -- 172.21.15.141:0/2161457865 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc648087c40 con 0x7fc66009a170 2024-01-16T15:05:59.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.077+0000 7fc665000640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc63806bc50 0x7fc63806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.078+0000 7fc665000640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc63806bc50 0x7fc63806e110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc650003d90 tx=0x7fc650009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.081+0000 7fc65cc00640 1 -- 172.21.15.141:0/2161457865 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6600c5770 con 0x7fc66009a170 2024-01-16T15:05:59.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.299+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6600cd620 con 0x7fc63806bc50 2024-01-16T15:05:59.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.330+0000 7fc65cc00640 1 -- 172.21.15.141:0/2161457865 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc650003980 con 0x7fc63806bc50 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc63806bc50 msgr2=0x7fc63806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc63806bc50 0x7fc63806e110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc650003d90 tx=0x7fc650009040 comp rx=0 tx=0).stop 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66009a170 msgr2=0x7fc6600c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66009a170 0x7fc6600c73f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc64800c890 tx=0x7fc64800cd50 comp rx=0 tx=0).stop 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 shutdown_connections 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc63806bc50 0x7fc63806e110 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc6600cd240 0x7fc6601f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc66009a170 0x7fc6600c73f0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 --2- 172.21.15.141:0/2161457865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc66006c6b0 0x7fc66015b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.333+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 >> 172.21.15.141:0/2161457865 conn(0x7fc6600b2d00 msgr2=0x7fc660099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:59.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.334+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 shutdown_connections 2024-01-16T15:05:59.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.334+0000 7fc667a00640 1 -- 172.21.15.141:0/2161457865 wait complete. 2024-01-16T15:05:59.330 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:05:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.560+0000 7f4f47800640 1 Processor -- start 2024-01-16T15:05:59.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.560+0000 7f4f47800640 1 -- start start 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.562+0000 7f4f47800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4f4013df80 0x7f4f4015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.562+0000 7f4f47800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4f400cf9a0 0x7f4f400cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.562+0000 7f4f47800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4006c470 con 0x7f4f4013e360 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.562+0000 7f4f47800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4006c750 con 0x7f4f4013df80 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.562+0000 7f4f47800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4006ca30 con 0x7f4f400cf9a0 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.562+0000 7f4f44e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4f4013df80 0x7f4f4015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f3fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4f4013e360 0x7f4f400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54072/0 (socket says 172.21.15.141:54072) 2024-01-16T15:05:59.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f44e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4f4013df80 0x7f4f4015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39974/0 (socket says 172.21.15.141:39974) 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f3fe00640 1 -- 172.21.15.141:0/969307476 learned_addr learned my addr 172.21.15.141:0/969307476 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f45800640 1 --2- 172.21.15.141:0/969307476 >> v2:172.21.15.188:3300/0 conn(0x7f4f400cf9a0 0x7f4f400cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f44e00640 1 -- 172.21.15.141:0/969307476 >> v2:172.21.15.188:3300/0 conn(0x7f4f400cf9a0 msgr2=0x7f4f400cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f44e00640 1 --2- 172.21.15.141:0/969307476 >> v2:172.21.15.188:3300/0 conn(0x7f4f400cf9a0 0x7f4f400cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f44e00640 1 -- 172.21.15.141:0/969307476 >> v1:172.21.15.141:6789/0 conn(0x7f4f4013e360 legacy=0x7f4f400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f44e00640 1 -- 172.21.15.141:0/969307476 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4006cd10 con 0x7f4f4013df80 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f45800640 1 --2- 172.21.15.141:0/969307476 >> v2:172.21.15.188:3300/0 conn(0x7f4f400cf9a0 0x7f4f400cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:59.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.563+0000 7f4f3f400640 1 -- 172.21.15.141:0/969307476 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514494328 0 0) 0x7f4f4006c470 con 0x7f4f4013e360 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.564+0000 7f4f44e00640 1 --2- 172.21.15.141:0/969307476 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f4013df80 0x7f4f4015bee0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f4f400ccb10 tx=0x7f4f3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f25e22343ace08c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.564+0000 7f4f3f400640 1 -- 172.21.15.141:0/969307476 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30031830 con 0x7f4f4013df80 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.564+0000 7f4f3f400640 1 -- 172.21.15.141:0/969307476 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f30031990 con 0x7f4f4013df80 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.564+0000 7f4f3f400640 1 -- 172.21.15.141:0/969307476 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30031d20 con 0x7f4f4013df80 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 -- 172.21.15.141:0/969307476 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f4013df80 msgr2=0x7f4f4015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 --2- 172.21.15.141:0/969307476 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f4013df80 0x7f4f4015bee0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f4f400ccb10 tx=0x7f4f3002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 -- 172.21.15.141:0/969307476 shutdown_connections 2024-01-16T15:05:59.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 --2- 172.21.15.141:0/969307476 >> v2:172.21.15.188:3300/0 conn(0x7f4f400cf9a0 0x7f4f400cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 --2- 172.21.15.141:0/969307476 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f4013df80 0x7f4f4015bee0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 -- 172.21.15.141:0/969307476 >> 172.21.15.141:0/969307476 conn(0x7f4f4006a800 msgr2=0x7f4f4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 -- 172.21.15.141:0/969307476 shutdown_connections 2024-01-16T15:05:59.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.565+0000 7f4f47800640 1 -- 172.21.15.141:0/969307476 wait complete. 2024-01-16T15:05:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.567+0000 7f4f47800640 1 Processor -- start 2024-01-16T15:05:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.567+0000 7f4f47800640 1 -- start start 2024-01-16T15:05:59.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f47800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f400cf9a0 0x7f4f40152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f47800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f4013df80 0x7f4f40153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f47800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4f4013e360 0x7f4f40159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f47800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40152810 con 0x7f4f400cf9a0 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f47800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f400711a0 con 0x7f4f4013df80 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f47800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4014af30 con 0x7f4f4013e360 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f44e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f400cf9a0 0x7f4f40152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f44e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f400cf9a0 0x7f4f40152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39976/0 (socket says 172.21.15.141:39976) 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.568+0000 7f4f44e00640 1 -- 172.21.15.141:0/232278253 learned_addr learned my addr 172.21.15.141:0/232278253 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f44e00640 1 -- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4f4013e360 msgr2=0x7f4f40159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f44e00640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4f4013e360 0x7f4f40159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f44e00640 1 -- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f4013df80 msgr2=0x7f4f40153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f44e00640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f4013df80 0x7f4f40153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f44e00640 1 -- 172.21.15.141:0/232278253 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30009400 con 0x7f4f400cf9a0 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f44e00640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f400cf9a0 0x7f4f40152c30 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009890 tx=0x7f4f300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f3cc00640 1 -- 172.21.15.141:0/232278253 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3003ba70 con 0x7f4f400cf9a0 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f3cc00640 1 -- 172.21.15.141:0/232278253 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3003bbf0 con 0x7f4f400cf9a0 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f3cc00640 1 -- 172.21.15.141:0/232278253 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300453f0 con 0x7f4f400cf9a0 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.569+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f40070be0 con 0x7f4f400cf9a0 2024-01-16T15:05:59.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.570+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4013d900 con 0x7f4f400cf9a0 2024-01-16T15:05:59.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.571+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f40152570 con 0x7f4f400cf9a0 2024-01-16T15:05:59.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.572+0000 7f4f3cc00640 1 -- 172.21.15.141:0/232278253 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f4f30045550 con 0x7f4f400cf9a0 2024-01-16T15:05:59.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.573+0000 7f4f3cc00640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4f1806bc50 0x7f4f1806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:59.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.573+0000 7f4f3cc00640 1 -- 172.21.15.141:0/232278253 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4f300aa980 con 0x7f4f400cf9a0 2024-01-16T15:05:59.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.573+0000 7f4f3fe00640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4f1806bc50 0x7f4f1806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:59.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.574+0000 7f4f3fe00640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4f1806bc50 0x7f4f1806e110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4f28007fd0 tx=0x7f4f28007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:59.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.577+0000 7f4f3cc00640 1 -- 172.21.15.141:0/232278253 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f40152570 con 0x7f4f400cf9a0 2024-01-16T15:05:59.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.797+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f4015f9c0 con 0x7f4f1806bc50 2024-01-16T15:05:59.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.808+0000 7f4f3cc00640 1 -- 172.21.15.141:0/232278253 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f4f280070d0 con 0x7f4f1806bc50 2024-01-16T15:05:59.787 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4f1806bc50 msgr2=0x7f4f1806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4f1806bc50 0x7f4f1806e110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4f28007fd0 tx=0x7f4f28007920 comp rx=0 tx=0).stop 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f400cf9a0 msgr2=0x7f4f40152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f400cf9a0 0x7f4f40152c30 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009890 tx=0x7f4f300046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 shutdown_connections 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4f1806bc50 0x7f4f1806e110 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4f4013e360 0x7f4f40159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f4013df80 0x7f4f40153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 --2- 172.21.15.141:0/232278253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f400cf9a0 0x7f4f40152c30 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 >> 172.21.15.141:0/232278253 conn(0x7f4f4006a800 msgr2=0x7f4f4015d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.811+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 shutdown_connections 2024-01-16T15:05:59.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:05:59.812+0000 7f4f47800640 1 -- 172.21.15.141:0/232278253 wait complete. 2024-01-16T15:05:59.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:59.809 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934) 2024-01-16T15:05:59.868 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934 2024-01-16T15:05:59.869 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:05:59.869 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:05:59.870 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:06:01.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:06:01.871 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934) 2024-01-16T15:06:01.936 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/8188628a-39ee-4039-a413-08e169a4c934 2024-01-16T15:06:01.936 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:06:01.936 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:01.937 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:06:01.937 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59631 2024-01-16T15:06:02.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.181+0000 7f95bf800640 1 Processor -- start 2024-01-16T15:06:02.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.181+0000 7f95bf800640 1 -- start start 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.182+0000 7f95bf800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f95b813df80 0x7f95b815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.182+0000 7f95bf800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f95b80cf9a0 0x7f95b80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.182+0000 7f95bf800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b806c470 con 0x7f95b813e360 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.182+0000 7f95bf800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b806c750 con 0x7f95b813df80 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.182+0000 7f95bf800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b806ca30 con 0x7f95b80cf9a0 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bce00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f95b813df80 0x7f95b815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bce00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f95b813df80 0x7f95b815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39980/0 (socket says 172.21.15.141:39980) 2024-01-16T15:06:02.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bce00640 1 -- 172.21.15.141:0/2121444314 learned_addr learned my addr 172.21.15.141:0/2121444314 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:02.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bd800640 1 --2- 172.21.15.141:0/2121444314 >> v2:172.21.15.188:3300/0 conn(0x7f95b80cf9a0 0x7f95b80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bce00640 1 -- 172.21.15.141:0/2121444314 >> v2:172.21.15.188:3300/0 conn(0x7f95b80cf9a0 msgr2=0x7f95b80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:02.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bce00640 1 --2- 172.21.15.141:0/2121444314 >> v2:172.21.15.188:3300/0 conn(0x7f95b80cf9a0 0x7f95b80cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bce00640 1 -- 172.21.15.141:0/2121444314 >> v1:172.21.15.188:6789/0 conn(0x7f95b813e360 legacy=0x7f95b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:02.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.183+0000 7f95bce00640 1 -- 172.21.15.141:0/2121444314 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b806cd10 con 0x7f95b813df80 2024-01-16T15:06:02.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.184+0000 7f95bce00640 1 --2- 172.21.15.141:0/2121444314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813df80 0x7f95b815bee0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f95b80ccb10 tx=0x7f95a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4206a9047e6b7320 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:02.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95b7e00640 1 -- 172.21.15.141:0/2121444314 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a8031870 con 0x7f95b813df80 2024-01-16T15:06:02.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95b7e00640 1 -- 172.21.15.141:0/2121444314 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a80319d0 con 0x7f95b813df80 2024-01-16T15:06:02.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95b7e00640 1 -- 172.21.15.141:0/2121444314 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a8031d60 con 0x7f95b813df80 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 -- 172.21.15.141:0/2121444314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813df80 msgr2=0x7f95b815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 --2- 172.21.15.141:0/2121444314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813df80 0x7f95b815bee0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f95b80ccb10 tx=0x7f95a802b910 comp rx=0 tx=0).stop 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 -- 172.21.15.141:0/2121444314 shutdown_connections 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 --2- 172.21.15.141:0/2121444314 >> v2:172.21.15.188:3300/0 conn(0x7f95b80cf9a0 0x7f95b80cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 --2- 172.21.15.141:0/2121444314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813df80 0x7f95b815bee0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 -- 172.21.15.141:0/2121444314 >> 172.21.15.141:0/2121444314 conn(0x7f95b806a800 msgr2=0x7f95b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 -- 172.21.15.141:0/2121444314 shutdown_connections 2024-01-16T15:06:02.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.185+0000 7f95bf800640 1 -- 172.21.15.141:0/2121444314 wait complete. 2024-01-16T15:06:02.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.187+0000 7f95bf800640 1 Processor -- start 2024-01-16T15:06:02.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.187+0000 7f95bf800640 1 -- start start 2024-01-16T15:06:02.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.188+0000 7f95bf800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95b80cf9a0 0x7f95b81f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.188+0000 7f95bf800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b813df80 0x7f95b81f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.188+0000 7f95bf800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813e360 0x7f95b813add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.188+0000 7f95bf800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b815ad10 con 0x7f95b813e360 2024-01-16T15:06:02.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.188+0000 7f95bf800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b81f5030 con 0x7f95b813df80 2024-01-16T15:06:02.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.188+0000 7f95bf800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b81f2300 con 0x7f95b80cf9a0 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813e360 0x7f95b813add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813e360 0x7f95b813add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39982/0 (socket says 172.21.15.141:39982) 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 -- 172.21.15.141:0/1857915467 learned_addr learned my addr 172.21.15.141:0/1857915467 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bce00640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95b80cf9a0 0x7f95b81f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 -- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95b80cf9a0 msgr2=0x7f95b81f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95b80cf9a0 0x7f95b81f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 -- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b813df80 msgr2=0x7f95b81f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b813df80 0x7f95b81f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 -- 172.21.15.141:0/1857915467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a80094d0 con 0x7f95b813e360 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.189+0000 7f95bd800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813e360 0x7f95b813add0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f95b000cc30 tx=0x7f95b000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.190+0000 7f95b5600640 1 -- 172.21.15.141:0/1857915467 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b000ce40 con 0x7f95b813e360 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.190+0000 7f95b5600640 1 -- 172.21.15.141:0/1857915467 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b0003990 con 0x7f95b813e360 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.190+0000 7f95b5600640 1 -- 172.21.15.141:0/1857915467 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0003d20 con 0x7f95b813e360 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.190+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b81f25e0 con 0x7f95b813e360 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.190+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b81f2ba0 con 0x7f95b813e360 2024-01-16T15:06:02.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.191+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b80675d0 con 0x7f95b813e360 2024-01-16T15:06:02.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.192+0000 7f95b5600640 1 -- 172.21.15.141:0/1857915467 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f95b0003e80 con 0x7f95b813e360 2024-01-16T15:06:02.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.193+0000 7f95b5600640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f959006bc50 0x7f959006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.193+0000 7f95b5600640 1 -- 172.21.15.141:0/1857915467 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f95b0088a60 con 0x7f95b813e360 2024-01-16T15:06:02.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.194+0000 7f95bce00640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f959006bc50 0x7f959006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.195+0000 7f95bce00640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f959006bc50 0x7f959006e110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f95a8036000 tx=0x7f95a803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:02.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.198+0000 7f95b5600640 1 -- 172.21.15.141:0/1857915467 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95b0055120 con 0x7f95b813e360 2024-01-16T15:06:02.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.423+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59631", "target": ["mon-mgr", ""]}) v1 -- 0x7f95b815f9c0 con 0x7f959006bc50 2024-01-16T15:06:02.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.589+0000 7f95b5600640 1 -- 172.21.15.141:0/1857915467 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f95a8031810 con 0x7f959006bc50 2024-01-16T15:06:02.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f959006bc50 msgr2=0x7f959006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f959006bc50 0x7f959006e110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f95a8036000 tx=0x7f95a803c040 comp rx=0 tx=0).stop 2024-01-16T15:06:02.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813e360 msgr2=0x7f95b813add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813e360 0x7f95b813add0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f95b000cc30 tx=0x7f95b000a5c0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 shutdown_connections 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f959006bc50 0x7f959006e110 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95b813e360 0x7f95b813add0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b813df80 0x7f95b81f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 --2- 172.21.15.141:0/1857915467 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95b80cf9a0 0x7f95b81f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.592+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 >> 172.21.15.141:0/1857915467 conn(0x7f95b806a800 msgr2=0x7f95b806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.593+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 shutdown_connections 2024-01-16T15:06:02.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.593+0000 7f95bf800640 1 -- 172.21.15.141:0/1857915467 wait complete. 2024-01-16T15:06:02.575 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.596+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.575 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.596+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.575 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.596+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.575 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.596+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.575 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.596+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.594 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59632 2024-01-16T15:06:02.605 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.626+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.605 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.626+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.605 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.626+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.605 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.626+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.605 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:02.626+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:02.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.826+0000 7f76f7800640 1 Processor -- start 2024-01-16T15:06:02.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.826+0000 7f76f7800640 1 -- start start 2024-01-16T15:06:02.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76f7800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0157c30 0x7f76f00cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76f7800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f76f01618f0 0x7f76f0070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76f7800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f0071460 con 0x7f76f0071030 2024-01-16T15:06:02.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76f7800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f015ea30 con 0x7f76f01618f0 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76f7800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f006d860 con 0x7f76f0157c30 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76efe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f76f01618f0 0x7f76f0070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76efe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f76f01618f0 0x7f76f0070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39986/0 (socket says 172.21.15.141:39986) 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.828+0000 7f76efe00640 1 -- 172.21.15.141:0/1786088048 learned_addr learned my addr 172.21.15.141:0/1786088048 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.829+0000 7f76f4e00640 1 --2- 172.21.15.141:0/1786088048 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0157c30 0x7f76f00cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.829+0000 7f76efe00640 1 -- 172.21.15.141:0/1786088048 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0157c30 msgr2=0x7f76f00cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.829+0000 7f76efe00640 1 --2- 172.21.15.141:0/1786088048 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0157c30 0x7f76f00cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.829+0000 7f76efe00640 1 -- 172.21.15.141:0/1786088048 >> v1:172.21.15.188:6789/0 conn(0x7f76f0071030 legacy=0x7f76f006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:02.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.829+0000 7f76efe00640 1 -- 172.21.15.141:0/1786088048 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f015deb0 con 0x7f76f01618f0 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.830+0000 7f76efe00640 1 --2- 172.21.15.141:0/1786088048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f01618f0 0x7f76f0070af0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009e50 tx=0x7f76d802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91dd6fe07a44998f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.830+0000 7f76ef400640 1 -- 172.21.15.141:0/1786088048 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d802cb00 con 0x7f76f01618f0 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.830+0000 7f76ef400640 1 -- 172.21.15.141:0/1786088048 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d802cc60 con 0x7f76f01618f0 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.830+0000 7f76ef400640 1 -- 172.21.15.141:0/1786088048 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d80343f0 con 0x7f76f01618f0 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 -- 172.21.15.141:0/1786088048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f01618f0 msgr2=0x7f76f0070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 --2- 172.21.15.141:0/1786088048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f01618f0 0x7f76f0070af0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009e50 tx=0x7f76d802bf60 comp rx=0 tx=0).stop 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 -- 172.21.15.141:0/1786088048 shutdown_connections 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 --2- 172.21.15.141:0/1786088048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f01618f0 0x7f76f0070af0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 --2- 172.21.15.141:0/1786088048 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0157c30 0x7f76f00cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 -- 172.21.15.141:0/1786088048 >> 172.21.15.141:0/1786088048 conn(0x7f76f00cf700 msgr2=0x7f76f00cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:02.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 -- 172.21.15.141:0/1786088048 shutdown_connections 2024-01-16T15:06:02.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.831+0000 7f76f7800640 1 -- 172.21.15.141:0/1786088048 wait complete. 2024-01-16T15:06:02.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.833+0000 7f76f7800640 1 Processor -- start 2024-01-16T15:06:02.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.833+0000 7f76f7800640 1 -- start start 2024-01-16T15:06:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76f7800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0071030 0x7f76f0149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76f7800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0157c30 0x7f76f015eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76f7800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f01618f0 0x7f76f015f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76f7800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0158110 con 0x7f76f0157c30 2024-01-16T15:06:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76f7800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f01583b0 con 0x7f76f0071030 2024-01-16T15:06:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76f7800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0158690 con 0x7f76f01618f0 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76efe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0157c30 0x7f76f015eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76efe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0157c30 0x7f76f015eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39988/0 (socket says 172.21.15.141:39988) 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.834+0000 7f76efe00640 1 -- 172.21.15.141:0/3149343461 learned_addr learned my addr 172.21.15.141:0/3149343461 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76f4e00640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0071030 0x7f76f0149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76f5800640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f01618f0 0x7f76f015f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76efe00640 1 -- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f01618f0 msgr2=0x7f76f015f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76efe00640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f01618f0 0x7f76f015f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76efe00640 1 -- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0071030 msgr2=0x7f76f0149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76efe00640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0071030 0x7f76f0149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76efe00640 1 -- 172.21.15.141:0/3149343461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d8009b30 con 0x7f76f0157c30 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.835+0000 7f76efe00640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0157c30 0x7f76f015eb70 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f76d8000f80 tx=0x7f76d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:02.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.836+0000 7f76ecc00640 1 -- 172.21.15.141:0/3149343461 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8040670 con 0x7f76f0157c30 2024-01-16T15:06:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.836+0000 7f76ecc00640 1 -- 172.21.15.141:0/3149343461 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d8040810 con 0x7f76f0157c30 2024-01-16T15:06:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.836+0000 7f76ecc00640 1 -- 172.21.15.141:0/3149343461 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8040bc0 con 0x7f76f0157c30 2024-01-16T15:06:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.836+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f0158970 con 0x7f76f0157c30 2024-01-16T15:06:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.836+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f01495d0 con 0x7f76f0157c30 2024-01-16T15:06:02.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.837+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f006e9f0 con 0x7f76f0157c30 2024-01-16T15:06:02.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.838+0000 7f76ecc00640 1 -- 172.21.15.141:0/3149343461 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f76d8040d20 con 0x7f76f0157c30 2024-01-16T15:06:02.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.839+0000 7f76ecc00640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76c806bc50 0x7f76c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:02.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.839+0000 7f76ecc00640 1 -- 172.21.15.141:0/3149343461 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76d80ab9c0 con 0x7f76f0157c30 2024-01-16T15:06:02.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.839+0000 7f76f4e00640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76c806bc50 0x7f76c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:02.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.840+0000 7f76f4e00640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76c806bc50 0x7f76c806e110 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006fd0 tx=0x7f76e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:02.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:02.843+0000 7f76ecc00640 1 -- 172.21.15.141:0/3149343461 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76d80781a0 con 0x7f76f0157c30 2024-01-16T15:06:03.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.068+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59632", "target": ["mon-mgr", ""]}) v1 -- 0x7f76f015e750 con 0x7f76c806bc50 2024-01-16T15:06:03.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.216+0000 7f76ecc00640 1 -- 172.21.15.141:0/3149343461 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f76e800f860 con 0x7f76c806bc50 2024-01-16T15:06:03.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76c806bc50 msgr2=0x7f76c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76c806bc50 0x7f76c806e110 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006fd0 tx=0x7f76e8008040 comp rx=0 tx=0).stop 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0157c30 msgr2=0x7f76f015eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0157c30 0x7f76f015eb70 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f76d8000f80 tx=0x7f76d8004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 shutdown_connections 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76c806bc50 0x7f76c806e110 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f01618f0 0x7f76f015f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0157c30 0x7f76f015eb70 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 --2- 172.21.15.141:0/3149343461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0071030 0x7f76f0149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 >> 172.21.15.141:0/3149343461 conn(0x7f76f00cf700 msgr2=0x7f76f00b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 shutdown_connections 2024-01-16T15:06:03.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.221+0000 7f76f7800640 1 -- 172.21.15.141:0/3149343461 wait complete. 2024-01-16T15:06:03.223 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59633 2024-01-16T15:06:03.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.458+0000 7f46ee400640 1 Processor -- start 2024-01-16T15:06:03.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.459+0000 7f46ee400640 1 -- start start 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.460+0000 7f46ee400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46e8160920 0x7f46e80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.460+0000 7f46ee400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e80cfea0 0x7f46e8161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.460+0000 7f46ee400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80d0a10 con 0x7f46e80cf9f0 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.460+0000 7f46ee400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e80d0330 con 0x7f46e8160920 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.460+0000 7f46ee400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80d1f50 con 0x7f46e80cfea0 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46ed000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46e8160920 0x7f46e80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46ed000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46e8160920 0x7f46e80b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39788/0 (socket says 172.21.15.141:39788) 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46e7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f46e80cf9f0 0x7f46e8161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39192/0 (socket says 172.21.15.141:39192) 2024-01-16T15:06:03.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46ed000640 1 -- 172.21.15.141:0/1528579634 learned_addr learned my addr 172.21.15.141:0/1528579634 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:03.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46ed000640 1 -- 172.21.15.141:0/1528579634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e80cfea0 msgr2=0x7f46e8161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:03.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46eda00640 1 --2- 172.21.15.141:0/1528579634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e80cfea0 0x7f46e8161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:03.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46ed000640 1 --2- 172.21.15.141:0/1528579634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e80cfea0 0x7f46e8161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46ed000640 1 -- 172.21.15.141:0/1528579634 >> v1:172.21.15.141:6789/0 conn(0x7f46e80cf9f0 legacy=0x7f46e8161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:03.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46ed000640 1 -- 172.21.15.141:0/1528579634 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e80cf080 con 0x7f46e8160920 2024-01-16T15:06:03.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.461+0000 7f46e7400640 1 -- 172.21.15.141:0/1528579634 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192597434 0 0) 0x7f46e80d0a10 con 0x7f46e80cf9f0 2024-01-16T15:06:03.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.462+0000 7f46ed000640 1 --2- 172.21.15.141:0/1528579634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e8160920 0x7f46e80b3f30 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009870 tx=0x7f46d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de02de1da495038 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46e7400640 1 -- 172.21.15.141:0/1528579634 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8030840 con 0x7f46e8160920 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46e7400640 1 -- 172.21.15.141:0/1528579634 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d80309a0 con 0x7f46e8160920 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46e7400640 1 -- 172.21.15.141:0/1528579634 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8030d30 con 0x7f46e8160920 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 -- 172.21.15.141:0/1528579634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e8160920 msgr2=0x7f46e80b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 --2- 172.21.15.141:0/1528579634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e8160920 0x7f46e80b3f30 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009870 tx=0x7f46d802b980 comp rx=0 tx=0).stop 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 -- 172.21.15.141:0/1528579634 shutdown_connections 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 --2- 172.21.15.141:0/1528579634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e80cfea0 0x7f46e8161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 --2- 172.21.15.141:0/1528579634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e8160920 0x7f46e80b3f30 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 -- 172.21.15.141:0/1528579634 >> 172.21.15.141:0/1528579634 conn(0x7f46e80d3090 msgr2=0x7f46e80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 -- 172.21.15.141:0/1528579634 shutdown_connections 2024-01-16T15:06:03.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.463+0000 7f46ee400640 1 -- 172.21.15.141:0/1528579634 wait complete. 2024-01-16T15:06:03.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.465+0000 7f46ee400640 1 Processor -- start 2024-01-16T15:06:03.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.465+0000 7f46ee400640 1 -- start start 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.466+0000 7f46ee400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46e80cf9f0 0x7f46e80c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.466+0000 7f46ee400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e80cfea0 0x7f46e80b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.466+0000 7f46ee400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e8160920 0x7f46e80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.466+0000 7f46ee400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b6d90 con 0x7f46e80cfea0 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.466+0000 7f46ee400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b5cf0 con 0x7f46e80cf9f0 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.466+0000 7f46ee400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b5fd0 con 0x7f46e8160920 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e80cfea0 0x7f46e80b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e80cfea0 0x7f46e80b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39794/0 (socket says 172.21.15.141:39794) 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 -- 172.21.15.141:0/629819900 learned_addr learned my addr 172.21.15.141:0/629819900 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:03.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46ed000640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46e80cf9f0 0x7f46e80c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 -- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e8160920 msgr2=0x7f46e80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e8160920 0x7f46e80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 -- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46e80cf9f0 msgr2=0x7f46e80c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46e80cf9f0 0x7f46e80c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 -- 172.21.15.141:0/629819900 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80094d0 con 0x7f46e80cfea0 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.467+0000 7f46e7e00640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e80cfea0 0x7f46e80b78f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f46d000c430 tx=0x7f46d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.468+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00074e0 con 0x7f46e80cfea0 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.468+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0007660 con 0x7f46e80cfea0 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.468+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e80b62b0 con 0x7f46e80cfea0 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.468+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0004a20 con 0x7f46e80cfea0 2024-01-16T15:06:03.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.468+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e80b6870 con 0x7f46e80cfea0 2024-01-16T15:06:03.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.469+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e80b6a60 con 0x7f46e80cfea0 2024-01-16T15:06:03.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.470+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f46d0004b80 con 0x7f46e80cfea0 2024-01-16T15:06:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.471+0000 7f46e4c00640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46c006bc50 0x7f46c006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:03.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.471+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f46d00884f0 con 0x7f46e80cfea0 2024-01-16T15:06:03.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.472+0000 7f46ed000640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46c006bc50 0x7f46c006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:03.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.473+0000 7f46ed000640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46c006bc50 0x7f46c006e110 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f46d8036000 tx=0x7f46d803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:03.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.476+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46d0054bb0 con 0x7f46e80cfea0 2024-01-16T15:06:03.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.672+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f46d0054f80 con 0x7f46e80cfea0 2024-01-16T15:06:03.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:03.711+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59633", "target": ["mon-mgr", ""]}) v1 -- 0x7f46e80ceac0 con 0x7f46c006bc50 2024-01-16T15:06:04.551 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:04.559 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:06.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.504+0000 7f46e4c00640 1 -- 172.21.15.141:0/629819900 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46d8030b90 con 0x7f46c006bc50 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46c006bc50 msgr2=0x7f46c006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46c006bc50 0x7f46c006e110 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f46d8036000 tx=0x7f46d803b040 comp rx=0 tx=0).stop 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e80cfea0 msgr2=0x7f46e80b78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e80cfea0 0x7f46e80b78f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f46d000c430 tx=0x7f46d000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 shutdown_connections 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f46c006bc50 0x7f46c006e110 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46e8160920 0x7f46e80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46e80cfea0 0x7f46e80b78f0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 --2- 172.21.15.141:0/629819900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46e80cf9f0 0x7f46e80c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.507+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 >> 172.21.15.141:0/629819900 conn(0x7f46e80d3090 msgr2=0x7f46e806c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:06.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.508+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 shutdown_connections 2024-01-16T15:06:06.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.508+0000 7f46ee400640 1 -- 172.21.15.141:0/629819900 wait complete. 2024-01-16T15:06:06.508 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59634 2024-01-16T15:06:06.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.742+0000 7f61b0200640 1 Processor -- start 2024-01-16T15:06:06.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.743+0000 7f61b0200640 1 -- start start 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61b0200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f61a813e360 0x7f61a80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61b0200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f61a815f9c0 0x7f61a80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61b0200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a815c5f0 con 0x7f61a813df80 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61b0200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a806c470 con 0x7f61a813e360 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61b0200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a806c750 con 0x7f61a815f9c0 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61ace00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f61a813e360 0x7f61a80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61ad800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f61a813df80 0x7f61a815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39208/0 (socket says 172.21.15.141:39208) 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.744+0000 7f61ad800640 1 -- 172.21.15.141:0/1855810865 learned_addr learned my addr 172.21.15.141:0/1855810865 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.745+0000 7f61ae200640 1 --2- 172.21.15.141:0/1855810865 >> v2:172.21.15.188:3300/0 conn(0x7f61a815f9c0 0x7f61a80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.745+0000 7f61ace00640 1 -- 172.21.15.141:0/1855810865 >> v2:172.21.15.188:3300/0 conn(0x7f61a815f9c0 msgr2=0x7f61a80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:06.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.745+0000 7f61ace00640 1 --2- 172.21.15.141:0/1855810865 >> v2:172.21.15.188:3300/0 conn(0x7f61a815f9c0 0x7f61a80cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.745+0000 7f61ace00640 1 -- 172.21.15.141:0/1855810865 >> v1:172.21.15.141:6789/0 conn(0x7f61a813df80 legacy=0x7f61a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.745+0000 7f61ace00640 1 -- 172.21.15.141:0/1855810865 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a806ca30 con 0x7f61a813e360 2024-01-16T15:06:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.745+0000 7f61a7e00640 1 -- 172.21.15.141:0/1855810865 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427405814 0 0) 0x7f61a815c5f0 con 0x7f61a813df80 2024-01-16T15:06:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.746+0000 7f61ace00640 1 --2- 172.21.15.141:0/1855810865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a80cf290 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f61a80ccae0 tx=0x7f619002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76baf826ff09724e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.746+0000 7f61a7e00640 1 -- 172.21.15.141:0/1855810865 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190031830 con 0x7f61a813e360 2024-01-16T15:06:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.746+0000 7f61a7e00640 1 -- 172.21.15.141:0/1855810865 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6190031990 con 0x7f61a813e360 2024-01-16T15:06:06.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.746+0000 7f61a7e00640 1 -- 172.21.15.141:0/1855810865 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190031d20 con 0x7f61a813e360 2024-01-16T15:06:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 -- 172.21.15.141:0/1855810865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 msgr2=0x7f61a80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 --2- 172.21.15.141:0/1855810865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a80cf290 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f61a80ccae0 tx=0x7f619002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 -- 172.21.15.141:0/1855810865 shutdown_connections 2024-01-16T15:06:06.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 --2- 172.21.15.141:0/1855810865 >> v2:172.21.15.188:3300/0 conn(0x7f61a815f9c0 0x7f61a80cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 --2- 172.21.15.141:0/1855810865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a80cf290 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 -- 172.21.15.141:0/1855810865 >> 172.21.15.141:0/1855810865 conn(0x7f61a806a800 msgr2=0x7f61a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:06.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 -- 172.21.15.141:0/1855810865 shutdown_connections 2024-01-16T15:06:06.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.747+0000 7f61b0200640 1 -- 172.21.15.141:0/1855810865 wait complete. 2024-01-16T15:06:06.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.749+0000 7f61b0200640 1 Processor -- start 2024-01-16T15:06:06.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.749+0000 7f61b0200640 1 -- start start 2024-01-16T15:06:06.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61b0200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a813df80 0x7f61a81f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61b0200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a8152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61b0200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61a815f9c0 0x7f61a80a92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61b0200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8153db0 con 0x7f61a813e360 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61b0200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8152f50 con 0x7f61a813df80 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61b0200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80a9830 con 0x7f61a815f9c0 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61ace00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a8152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61ace00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a8152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39812/0 (socket says 172.21.15.141:39812) 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.750+0000 7f61ace00640 1 -- 172.21.15.141:0/3240412596 learned_addr learned my addr 172.21.15.141:0/3240412596 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ad800640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a813df80 0x7f61a81f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ae200640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61a815f9c0 0x7f61a80a92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ace00640 1 -- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61a815f9c0 msgr2=0x7f61a80a92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ace00640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61a815f9c0 0x7f61a80a92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ace00640 1 -- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a813df80 msgr2=0x7f61a81f6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ace00640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a813df80 0x7f61a81f6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ace00640 1 -- 172.21.15.141:0/3240412596 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6190009400 con 0x7f61a813e360 2024-01-16T15:06:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61ace00640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a8152a10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f6190009890 tx=0x7f61900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61a5600640 1 -- 172.21.15.141:0/3240412596 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619003ba70 con 0x7f61a813e360 2024-01-16T15:06:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61a5600640 1 -- 172.21.15.141:0/3240412596 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619003bbf0 con 0x7f61a813e360 2024-01-16T15:06:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a8153130 con 0x7f61a813e360 2024-01-16T15:06:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.751+0000 7f61a5600640 1 -- 172.21.15.141:0/3240412596 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190045400 con 0x7f61a813e360 2024-01-16T15:06:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.752+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a81536f0 con 0x7f61a813e360 2024-01-16T15:06:06.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.753+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a80675d0 con 0x7f61a813e360 2024-01-16T15:06:06.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.754+0000 7f61a5600640 1 -- 172.21.15.141:0/3240412596 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f619004e440 con 0x7f61a813e360 2024-01-16T15:06:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.755+0000 7f61a5600640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f618006bd70 0x7f618006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.755+0000 7f61a5600640 1 -- 172.21.15.141:0/3240412596 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f61900aa790 con 0x7f61a813e360 2024-01-16T15:06:06.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.755+0000 7f61ad800640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f618006bd70 0x7f618006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.756+0000 7f61ad800640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f618006bd70 0x7f618006e230 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6198000f60 tx=0x7f6198009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.759+0000 7f61a5600640 1 -- 172.21.15.141:0/3240412596 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61900770b0 con 0x7f61a813e360 2024-01-16T15:06:06.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:06.989+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7f61a806cff0 con 0x7f618006bd70 2024-01-16T15:06:11.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.526+0000 7f61a5600640 1 -- 172.21.15.141:0/3240412596 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6198001300 con 0x7f618006bd70 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f618006bd70 msgr2=0x7f618006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f618006bd70 0x7f618006e230 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6198000f60 tx=0x7f6198009040 comp rx=0 tx=0).stop 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 msgr2=0x7f61a8152a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a8152a10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f6190009890 tx=0x7f61900046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 shutdown_connections 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f618006bd70 0x7f618006e230 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61a815f9c0 0x7f61a80a92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61a813e360 0x7f61a8152a10 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 --2- 172.21.15.141:0/3240412596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a813df80 0x7f61a81f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.531+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 >> 172.21.15.141:0/3240412596 conn(0x7f61a806a800 msgr2=0x7f61a80705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:11.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.532+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 shutdown_connections 2024-01-16T15:06:11.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.532+0000 7f61b0200640 1 -- 172.21.15.141:0/3240412596 wait complete. 2024-01-16T15:06:11.536 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59635 2024-01-16T15:06:11.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.777+0000 7fa783400640 1 Processor -- start 2024-01-16T15:06:11.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.777+0000 7fa783400640 1 -- start start 2024-01-16T15:06:11.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.779+0000 7fa783400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa77c0cf9a0 0x7fa77c166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:11.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.779+0000 7fa783400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c15c5f0 con 0x7fa77c13e360 2024-01-16T15:06:11.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.779+0000 7fa783400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c06c470 con 0x7fa77c13df80 2024-01-16T15:06:11.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.779+0000 7fa783400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c06c750 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.779+0000 7fa77be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa77c13e360 0x7fa77c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:39224/0 (socket says 172.21.15.141:39224) 2024-01-16T15:06:11.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.780+0000 7fa77be00640 1 -- 172.21.15.141:0/1474763739 learned_addr learned my addr 172.21.15.141:0/1474763739 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:11.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.780+0000 7fa781400640 1 --2- 172.21.15.141:0/1474763739 >> v2:172.21.15.141:3300/0 conn(0x7fa77c0cf9a0 0x7fa77c166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:11.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.780+0000 7fa77b400640 1 -- 172.21.15.141:0/1474763739 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063749336 0 0) 0x7fa77c15c5f0 con 0x7fa77c13e360 2024-01-16T15:06:11.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.780+0000 7fa77b400640 1 -- 172.21.15.141:0/1474763739 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75c003520 con 0x7fa77c13e360 2024-01-16T15:06:11.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.780+0000 7fa781400640 1 -- 172.21.15.141:0/1474763739 >> v1:172.21.15.188:6789/0 conn(0x7fa77c13df80 legacy=0x7fa77c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.780+0000 7fa781400640 1 -- 172.21.15.141:0/1474763739 >> v1:172.21.15.141:6789/0 conn(0x7fa77c13e360 legacy=0x7fa77c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.780+0000 7fa781400640 1 -- 172.21.15.141:0/1474763739 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c06ca30 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.782+0000 7fa781400640 1 --2- 172.21.15.141:0/1474763739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c166030 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa774009d90 tx=0x7fa77402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8283c9eb2aa6196 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:11.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.782+0000 7fa77b400640 1 -- 172.21.15.141:0/1474763739 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77402cd00 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.782+0000 7fa77b400640 1 -- 172.21.15.141:0/1474763739 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77402ce60 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.782+0000 7fa77b400640 1 -- 172.21.15.141:0/1474763739 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774034660 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.783+0000 7fa783400640 1 -- 172.21.15.141:0/1474763739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 msgr2=0x7fa77c166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.783+0000 7fa783400640 1 --2- 172.21.15.141:0/1474763739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c166030 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa774009d90 tx=0x7fa77402bf50 comp rx=0 tx=0).stop 2024-01-16T15:06:11.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.783+0000 7fa783400640 1 -- 172.21.15.141:0/1474763739 shutdown_connections 2024-01-16T15:06:11.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.783+0000 7fa783400640 1 --2- 172.21.15.141:0/1474763739 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c166030 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.783+0000 7fa783400640 1 -- 172.21.15.141:0/1474763739 >> 172.21.15.141:0/1474763739 conn(0x7fa77c06a800 msgr2=0x7fa77c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:11.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.783+0000 7fa783400640 1 -- 172.21.15.141:0/1474763739 shutdown_connections 2024-01-16T15:06:11.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.783+0000 7fa783400640 1 -- 172.21.15.141:0/1474763739 wait complete. 2024-01-16T15:06:11.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.784+0000 7fa783400640 1 Processor -- start 2024-01-16T15:06:11.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.785+0000 7fa783400640 1 -- start start 2024-01-16T15:06:11.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa783400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:11.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa783400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa77c13df80 0x7fa77c15a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:11.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa783400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa77c13e360 0x7fa77c153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa783400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c157e60 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa783400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c158490 con 0x7fa77c13e360 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa783400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c1586e0 con 0x7fa77c13df80 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c132fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39836/0 (socket says 172.21.15.141:39836) 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 -- 172.21.15.141:0/2890035616 learned_addr learned my addr 172.21.15.141:0/2890035616 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 -- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa77c13df80 msgr2=0x7fa77c15a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa77c13df80 0x7fa77c15a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 -- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa77c13e360 msgr2=0x7fa77c153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa77c13e360 0x7fa77c153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.786+0000 7fa780a00640 1 -- 172.21.15.141:0/2890035616 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa774009a70 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.787+0000 7fa780a00640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c132fd0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00c400 tx=0x7fa76c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:11.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.787+0000 7fa778c00640 1 -- 172.21.15.141:0/2890035616 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0052d0 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.787+0000 7fa778c00640 1 -- 172.21.15.141:0/2890035616 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c005430 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.787+0000 7fa778c00640 1 -- 172.21.15.141:0/2890035616 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c003830 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.787+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c1589c0 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.787+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c0710c0 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.788+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c0675d0 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.789+0000 7fa778c00640 1 -- 172.21.15.141:0/2890035616 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fa76c003990 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.790+0000 7fa778c00640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa75406bd70 0x7fa75406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:11.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.790+0000 7fa778c00640 1 -- 172.21.15.141:0/2890035616 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa76c087dc0 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.791+0000 7fa77be00640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa75406bd70 0x7fa75406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:11.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.792+0000 7fa77be00640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa75406bd70 0x7fa75406e230 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa77c13dab0 tx=0x7fa764007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:11.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:11.794+0000 7fa778c00640 1 -- 172.21.15.141:0/2890035616 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa76c0543c0 con 0x7fa77c0cf9a0 2024-01-16T15:06:11.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:12.019+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59635", "target": ["mon-mgr", ""]}) v1 -- 0x7fa77c06e190 con 0x7fa75406bd70 2024-01-16T15:06:16.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.507+0000 7fa778c00640 1 -- 172.21.15.141:0/2890035616 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa77c06e190 con 0x7fa75406bd70 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa75406bd70 msgr2=0x7fa75406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa75406bd70 0x7fa75406e230 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa77c13dab0 tx=0x7fa764007930 comp rx=0 tx=0).stop 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 msgr2=0x7fa77c132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c132fd0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00c400 tx=0x7fa76c00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 shutdown_connections 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa75406bd70 0x7fa75406e230 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa77c13e360 0x7fa77c153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa77c13df80 0x7fa77c15a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 --2- 172.21.15.141:0/2890035616 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa77c0cf9a0 0x7fa77c132fd0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 >> 172.21.15.141:0/2890035616 conn(0x7fa77c06a800 msgr2=0x7fa77c153990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:16.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 shutdown_connections 2024-01-16T15:06:16.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.511+0000 7fa783400640 1 -- 172.21.15.141:0/2890035616 wait complete. 2024-01-16T15:06:16.511 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59636 2024-01-16T15:06:16.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.743+0000 7f855e000640 1 Processor -- start 2024-01-16T15:06:16.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.743+0000 7f855e000640 1 -- start start 2024-01-16T15:06:16.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.744+0000 7f855e000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8558160f80 0x7f8558161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:16.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.744+0000 7f855e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85580ced50 0x7f85580d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:16.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.744+0000 7f855e000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85581618a0 con 0x7f85580ce8f0 2024-01-16T15:06:16.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.744+0000 7f855e000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85580cf1e0 con 0x7f85580ced50 2024-01-16T15:06:16.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.744+0000 7f855e000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85580d0de0 con 0x7f8558160f80 2024-01-16T15:06:16.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.744+0000 7f8557e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85580ced50 0x7f85580d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:16.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.745+0000 7f8557e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f85580ced50 0x7f85580d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48034/0 (socket says 172.21.15.141:48034) 2024-01-16T15:06:16.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.745+0000 7f8557e00640 1 -- 172.21.15.141:0/3483546270 learned_addr learned my addr 172.21.15.141:0/3483546270 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:16.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.745+0000 7f8557e00640 1 -- 172.21.15.141:0/3483546270 >> v2:172.21.15.188:3300/0 conn(0x7f8558160f80 msgr2=0x7f8558161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:16.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.745+0000 7f8557e00640 1 --2- 172.21.15.141:0/3483546270 >> v2:172.21.15.188:3300/0 conn(0x7f8558160f80 0x7f8558161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.745+0000 7f8557e00640 1 -- 172.21.15.141:0/3483546270 >> v1:172.21.15.188:6789/0 conn(0x7f85580ce8f0 legacy=0x7f85580cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:16.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.745+0000 7f8557e00640 1 -- 172.21.15.141:0/3483546270 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580d10c0 con 0x7f85580ced50 2024-01-16T15:06:16.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.746+0000 7f8557e00640 1 --2- 172.21.15.141:0/3483546270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85580ced50 0x7f85580d02f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f855808d490 tx=0x7f855002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9e62bd70535b9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.746+0000 7f8556000640 1 -- 172.21.15.141:0/3483546270 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855002ca80 con 0x7f85580ced50 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f8556000640 1 -- 172.21.15.141:0/3483546270 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855002cbe0 con 0x7f85580ced50 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f8556000640 1 -- 172.21.15.141:0/3483546270 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85500343f0 con 0x7f85580ced50 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 -- 172.21.15.141:0/3483546270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85580ced50 msgr2=0x7f85580d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 --2- 172.21.15.141:0/3483546270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85580ced50 0x7f85580d02f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f855808d490 tx=0x7f855002bee0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 -- 172.21.15.141:0/3483546270 shutdown_connections 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 --2- 172.21.15.141:0/3483546270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85580ced50 0x7f85580d02f0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 --2- 172.21.15.141:0/3483546270 >> v2:172.21.15.188:3300/0 conn(0x7f8558160f80 0x7f8558161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 -- 172.21.15.141:0/3483546270 >> 172.21.15.141:0/3483546270 conn(0x7f855809c550 msgr2=0x7f85580d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 -- 172.21.15.141:0/3483546270 shutdown_connections 2024-01-16T15:06:16.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.747+0000 7f855e000640 1 -- 172.21.15.141:0/3483546270 wait complete. 2024-01-16T15:06:16.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.749+0000 7f855e000640 1 Processor -- start 2024-01-16T15:06:16.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.749+0000 7f855e000640 1 -- start start 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.750+0000 7f855e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85580ce8f0 0x7f85580c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.750+0000 7f855e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85580ced50 0x7f8558149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.750+0000 7f855e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8558160f80 0x7f855814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.750+0000 7f855e000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85580ca760 con 0x7f8558160f80 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.750+0000 7f855e000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85580c8840 con 0x7f85580ced50 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.750+0000 7f855e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85580c8b20 con 0x7f85580ce8f0 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.750+0000 7f8557e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8558160f80 0x7f855814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8558160f80 0x7f855814a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48038/0 (socket says 172.21.15.141:48038) 2024-01-16T15:06:16.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 -- 172.21.15.141:0/2986240047 learned_addr learned my addr 172.21.15.141:0/2986240047 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8556a00640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85580ced50 0x7f8558149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557400640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85580ce8f0 0x7f85580c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 -- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85580ce8f0 msgr2=0x7f85580c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85580ce8f0 0x7f85580c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 -- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85580ced50 msgr2=0x7f8558149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85580ced50 0x7f8558149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 -- 172.21.15.141:0/2986240047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550009b30 con 0x7f8558160f80 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557400640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85580ce8f0 0x7f85580c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.751+0000 7f8557e00640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8558160f80 0x7f855814a880 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8550036040 tx=0x7f8550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.752+0000 7f854f400640 1 -- 172.21.15.141:0/2986240047 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855003e450 con 0x7f8558160f80 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.752+0000 7f854f400640 1 -- 172.21.15.141:0/2986240047 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855003e5d0 con 0x7f8558160f80 2024-01-16T15:06:16.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.752+0000 7f854f400640 1 -- 172.21.15.141:0/2986240047 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855003e980 con 0x7f8558160f80 2024-01-16T15:06:16.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.752+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85580c8e00 con 0x7f8558160f80 2024-01-16T15:06:16.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.752+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85580caa40 con 0x7f8558160f80 2024-01-16T15:06:16.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.753+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85580c9020 con 0x7f8558160f80 2024-01-16T15:06:16.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.754+0000 7f854f400640 1 -- 172.21.15.141:0/2986240047 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f855002cd50 con 0x7f8558160f80 2024-01-16T15:06:16.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.755+0000 7f854f400640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f853006bd70 0x7f853006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:16.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.755+0000 7f854f400640 1 -- 172.21.15.141:0/2986240047 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f85500ab150 con 0x7f8558160f80 2024-01-16T15:06:16.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.755+0000 7f8557400640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f853006bd70 0x7f853006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:16.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.756+0000 7f8557400640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f853006bd70 0x7f853006e230 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8548006fd0 tx=0x7f8548008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:16.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.759+0000 7f854f400640 1 -- 172.21.15.141:0/2986240047 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8550077a70 con 0x7f8558160f80 2024-01-16T15:06:16.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:16.986+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59636", "target": ["mon-mgr", ""]}) v1 -- 0x7f85580d1960 con 0x7f853006bd70 2024-01-16T15:06:17.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.139+0000 7f854f400640 1 -- 172.21.15.141:0/2986240047 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f854800ae20 con 0x7f853006bd70 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.143+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f853006bd70 msgr2=0x7f853006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.143+0000 7f855e000640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f853006bd70 0x7f853006e230 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8548006fd0 tx=0x7f8548008040 comp rx=0 tx=0).stop 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.143+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8558160f80 msgr2=0x7f855814a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.143+0000 7f855e000640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8558160f80 0x7f855814a880 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8550036040 tx=0x7f8550004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 shutdown_connections 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f853006bd70 0x7f853006e230 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8558160f80 0x7f855814a880 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85580ced50 0x7f8558149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 --2- 172.21.15.141:0/2986240047 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f85580ce8f0 0x7f85580c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 >> 172.21.15.141:0/2986240047 conn(0x7f855809c550 msgr2=0x7f85580afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:17.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 shutdown_connections 2024-01-16T15:06:17.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.144+0000 7f855e000640 1 -- 172.21.15.141:0/2986240047 wait complete. 2024-01-16T15:06:17.143 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59637 2024-01-16T15:06:17.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.382+0000 7f389f000640 1 Processor -- start 2024-01-16T15:06:17.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.382+0000 7f389f000640 1 -- start start 2024-01-16T15:06:17.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f389f000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f38980cf9a0 0x7f389815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f389f000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f389815e1e0 0x7f38980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f389f000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980b1ee0 con 0x7f3898143ae0 2024-01-16T15:06:17.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f389f000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38980b2780 con 0x7f389815e1e0 2024-01-16T15:06:17.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f389f000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389815e990 con 0x7f38980cf9a0 2024-01-16T15:06:17.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f389d000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f389815e1e0 0x7f38980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f3897e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3898143ae0 0x7f38980cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35158/0 (socket says 172.21.15.141:35158) 2024-01-16T15:06:17.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.384+0000 7f3897e00640 1 -- 172.21.15.141:0/3004880896 learned_addr learned my addr 172.21.15.141:0/3004880896 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:17.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.385+0000 7f3897400640 1 --2- 172.21.15.141:0/3004880896 >> v2:172.21.15.188:3300/0 conn(0x7f38980cf9a0 0x7f389815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.385+0000 7f389d000640 1 -- 172.21.15.141:0/3004880896 >> v2:172.21.15.188:3300/0 conn(0x7f38980cf9a0 msgr2=0x7f389815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:17.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.385+0000 7f389d000640 1 --2- 172.21.15.141:0/3004880896 >> v2:172.21.15.188:3300/0 conn(0x7f38980cf9a0 0x7f389815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.385+0000 7f389d000640 1 -- 172.21.15.141:0/3004880896 >> v1:172.21.15.141:6789/0 conn(0x7f3898143ae0 legacy=0x7f38980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.385+0000 7f389d000640 1 -- 172.21.15.141:0/3004880896 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980b2130 con 0x7f389815e1e0 2024-01-16T15:06:17.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.385+0000 7f3896a00640 1 -- 172.21.15.141:0/3004880896 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409315725 0 0) 0x7f38980b1ee0 con 0x7f3898143ae0 2024-01-16T15:06:17.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.386+0000 7f389d000640 1 --2- 172.21.15.141:0/3004880896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f389815e1e0 0x7f38980702c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f38980cb830 tx=0x7f389002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f21972c50503af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.386+0000 7f3896a00640 1 -- 172.21.15.141:0/3004880896 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890031830 con 0x7f389815e1e0 2024-01-16T15:06:17.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.386+0000 7f3896a00640 1 -- 172.21.15.141:0/3004880896 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3890031990 con 0x7f389815e1e0 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.386+0000 7f3896a00640 1 -- 172.21.15.141:0/3004880896 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890031d20 con 0x7f389815e1e0 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 -- 172.21.15.141:0/3004880896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f389815e1e0 msgr2=0x7f38980702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 --2- 172.21.15.141:0/3004880896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f389815e1e0 0x7f38980702c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f38980cb830 tx=0x7f389002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 -- 172.21.15.141:0/3004880896 shutdown_connections 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 --2- 172.21.15.141:0/3004880896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f389815e1e0 0x7f38980702c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 --2- 172.21.15.141:0/3004880896 >> v2:172.21.15.188:3300/0 conn(0x7f38980cf9a0 0x7f389815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 -- 172.21.15.141:0/3004880896 >> 172.21.15.141:0/3004880896 conn(0x7f389806a800 msgr2=0x7f389806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 -- 172.21.15.141:0/3004880896 shutdown_connections 2024-01-16T15:06:17.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.387+0000 7f389f000640 1 -- 172.21.15.141:0/3004880896 wait complete. 2024-01-16T15:06:17.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.389+0000 7f389f000640 1 Processor -- start 2024-01-16T15:06:17.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.389+0000 7f389f000640 1 -- start start 2024-01-16T15:06:17.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.390+0000 7f389f000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38980cf9a0 0x7f38980c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.390+0000 7f389f000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3898143ae0 0x7f38981599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.390+0000 7f389f000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f389815e1e0 0x7f38980b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.390+0000 7f389f000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980c4210 con 0x7f3898143ae0 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.390+0000 7f389f000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980c44f0 con 0x7f38980cf9a0 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.390+0000 7f389f000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38980c47d0 con 0x7f389815e1e0 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3898143ae0 0x7f38981599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3898143ae0 0x7f38981599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48048/0 (socket says 172.21.15.141:48048) 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 -- 172.21.15.141:0/3296253310 learned_addr learned my addr 172.21.15.141:0/3296253310 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897e00640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38980cf9a0 0x7f38980c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f389d000640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f389815e1e0 0x7f38980b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 -- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f389815e1e0 msgr2=0x7f38980b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f389815e1e0 0x7f38980b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 -- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38980cf9a0 msgr2=0x7f38980c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38980cf9a0 0x7f38980c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 -- 172.21.15.141:0/3296253310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3890009400 con 0x7f3898143ae0 2024-01-16T15:06:17.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.391+0000 7f3897400640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3898143ae0 0x7f38981599e0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f388000d5f0 tx=0x7f388000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.392+0000 7f388fe00640 1 -- 172.21.15.141:0/3296253310 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38800125f0 con 0x7f3898143ae0 2024-01-16T15:06:17.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.392+0000 7f388fe00640 1 -- 172.21.15.141:0/3296253310 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3880012750 con 0x7f3898143ae0 2024-01-16T15:06:17.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.392+0000 7f388fe00640 1 -- 172.21.15.141:0/3296253310 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880012b00 con 0x7f3898143ae0 2024-01-16T15:06:17.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.392+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38980c4ab0 con 0x7f3898143ae0 2024-01-16T15:06:17.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.392+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3898159f20 con 0x7f3898143ae0 2024-01-16T15:06:17.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.393+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38980b1b10 con 0x7f3898143ae0 2024-01-16T15:06:17.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.394+0000 7f388fe00640 1 -- 172.21.15.141:0/3296253310 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f3880012c60 con 0x7f3898143ae0 2024-01-16T15:06:17.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.395+0000 7f388fe00640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f387006bd70 0x7f387006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.395+0000 7f388fe00640 1 -- 172.21.15.141:0/3296253310 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f38800889d0 con 0x7f3898143ae0 2024-01-16T15:06:17.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.395+0000 7f3897e00640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f387006bd70 0x7f387006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.397+0000 7f3897e00640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f387006bd70 0x7f387006e230 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f38880012e0 tx=0x7f3888007ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.399+0000 7f388fe00640 1 -- 172.21.15.141:0/3296253310 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38800552f0 con 0x7f3898143ae0 2024-01-16T15:06:17.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.624+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59637", "target": ["mon-mgr", ""]}) v1 -- 0x7f389815e670 con 0x7f387006bd70 2024-01-16T15:06:17.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.813+0000 7f388fe00640 1 -- 172.21.15.141:0/3296253310 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38880079d0 con 0x7f387006bd70 2024-01-16T15:06:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f387006bd70 msgr2=0x7f387006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f387006bd70 0x7f387006e230 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f38880012e0 tx=0x7f3888007ed0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3898143ae0 msgr2=0x7f38981599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3898143ae0 0x7f38981599e0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f388000d5f0 tx=0x7f388000dab0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 shutdown_connections 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f387006bd70 0x7f387006e230 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f389815e1e0 0x7f38980b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3898143ae0 0x7f38981599e0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 --2- 172.21.15.141:0/3296253310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38980cf9a0 0x7f38980c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 >> 172.21.15.141:0/3296253310 conn(0x7f389806a800 msgr2=0x7f389815b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 shutdown_connections 2024-01-16T15:06:17.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:17.816+0000 7f389f000640 1 -- 172.21.15.141:0/3296253310 wait complete. 2024-01-16T15:06:17.815 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:06:17.815 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59634 2024-01-16T15:06:18.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.051+0000 7f5ddf400640 1 Processor -- start 2024-01-16T15:06:18.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.051+0000 7f5ddf400640 1 -- start start 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddf400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5dd80cf9a0 0x7f5dd80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddf400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5dd80b31d0 0x7f5dd80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddf400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd80d0320 con 0x7f5dd815d4b0 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddf400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd80d1c60 con 0x7f5dd80cf9a0 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddf400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd80b2730 con 0x7f5dd80b31d0 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddd600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5dd80cf9a0 0x7f5dd80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddd600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5dd80cf9a0 0x7f5dd80cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48052/0 (socket says 172.21.15.141:48052) 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddd600640 1 -- 172.21.15.141:0/2632388904 learned_addr learned my addr 172.21.15.141:0/2632388904 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5dde000640 1 --1- 172.21.15.141:0/2632388904 >> v1:172.21.15.141:6789/0 conn(0x7f5dd815d4b0 0x7f5dd80d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35162/0 (socket says 172.21.15.141:35162) 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddd600640 1 -- 172.21.15.141:0/2632388904 >> v2:172.21.15.188:3300/0 conn(0x7f5dd80b31d0 msgr2=0x7f5dd80b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddd600640 1 --2- 172.21.15.141:0/2632388904 >> v2:172.21.15.188:3300/0 conn(0x7f5dd80b31d0 0x7f5dd80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.053+0000 7f5ddd600640 1 -- 172.21.15.141:0/2632388904 >> v1:172.21.15.141:6789/0 conn(0x7f5dd815d4b0 legacy=0x7f5dd80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.054+0000 7f5ddd600640 1 -- 172.21.15.141:0/2632388904 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd80d1ec0 con 0x7f5dd80cf9a0 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.055+0000 7f5ddd600640 1 --2- 172.21.15.141:0/2632388904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd80cf9a0 0x7f5dd80cfde0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8072a30 tx=0x7f5dc002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed3affbbc403ab6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.055+0000 7f5ddcc00640 1 -- 172.21.15.141:0/2632388904 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0030870 con 0x7f5dd80cf9a0 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.055+0000 7f5ddcc00640 1 -- 172.21.15.141:0/2632388904 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc00309d0 con 0x7f5dd80cf9a0 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.055+0000 7f5ddcc00640 1 -- 172.21.15.141:0/2632388904 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0030d60 con 0x7f5dd80cf9a0 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.055+0000 7f5ddf400640 1 -- 172.21.15.141:0/2632388904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd80cf9a0 msgr2=0x7f5dd80cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.055+0000 7f5ddf400640 1 --2- 172.21.15.141:0/2632388904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd80cf9a0 0x7f5dd80cfde0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8072a30 tx=0x7f5dc002b9b0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.056+0000 7f5ddf400640 1 -- 172.21.15.141:0/2632388904 shutdown_connections 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.056+0000 7f5ddf400640 1 --2- 172.21.15.141:0/2632388904 >> v2:172.21.15.188:3300/0 conn(0x7f5dd80b31d0 0x7f5dd80b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.056+0000 7f5ddf400640 1 --2- 172.21.15.141:0/2632388904 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd80cf9a0 0x7f5dd80cfde0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.056+0000 7f5ddf400640 1 -- 172.21.15.141:0/2632388904 >> 172.21.15.141:0/2632388904 conn(0x7f5dd80d3000 msgr2=0x7f5dd80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:18.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.056+0000 7f5ddf400640 1 -- 172.21.15.141:0/2632388904 shutdown_connections 2024-01-16T15:06:18.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.056+0000 7f5ddf400640 1 -- 172.21.15.141:0/2632388904 wait complete. 2024-01-16T15:06:18.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.057+0000 7f5ddf400640 1 Processor -- start 2024-01-16T15:06:18.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.058+0000 7f5ddf400640 1 -- start start 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.058+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5dd80b31d0 0x7f5dd80ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.058+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5dd80cf9a0 0x7f5dd815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.058+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd815d4b0 0x7f5dd815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.058+0000 7f5ddf400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80cbf20 con 0x7f5dd815d4b0 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.058+0000 7f5ddf400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80cc200 con 0x7f5dd80cf9a0 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.058+0000 7f5ddf400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80cc4e0 con 0x7f5dd80b31d0 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd815d4b0 0x7f5dd815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd815d4b0 0x7f5dd815bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48054/0 (socket says 172.21.15.141:48054) 2024-01-16T15:06:18.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 -- 172.21.15.141:0/796879387 learned_addr learned my addr 172.21.15.141:0/796879387 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:18.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 -- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5dd80b31d0 msgr2=0x7f5dd80ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5dd80b31d0 0x7f5dd80ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddd600640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5dd80cf9a0 0x7f5dd815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 -- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5dd80cf9a0 msgr2=0x7f5dd815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5dd80cf9a0 0x7f5dd815b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.059+0000 7f5ddea00640 1 -- 172.21.15.141:0/796879387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00094d0 con 0x7f5dd815d4b0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.060+0000 7f5ddea00640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd815d4b0 0x7f5dd815bd80 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000b970 tx=0x7f5dd000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.060+0000 7f5dd6000640 1 -- 172.21.15.141:0/796879387 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd000cb40 con 0x7f5dd815d4b0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.060+0000 7f5dd6000640 1 -- 172.21.15.141:0/796879387 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd000ccc0 con 0x7f5dd815d4b0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.060+0000 7f5dd6000640 1 -- 172.21.15.141:0/796879387 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0014680 con 0x7f5dd815d4b0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.060+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd80cc7c0 con 0x7f5dd815d4b0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.060+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd80cbad0 con 0x7f5dd815d4b0 2024-01-16T15:06:18.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.061+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd80cbcf0 con 0x7f5dd815d4b0 2024-01-16T15:06:18.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.063+0000 7f5dd6000640 1 -- 172.21.15.141:0/796879387 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f5dd001d3b0 con 0x7f5dd815d4b0 2024-01-16T15:06:18.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.063+0000 7f5dd6000640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5db006bd70 0x7f5db006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:18.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.064+0000 7f5dd6000640 1 -- 172.21.15.141:0/796879387 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5dd0088500 con 0x7f5dd815d4b0 2024-01-16T15:06:18.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.064+0000 7f5dde000640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5db006bd70 0x7f5db006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:18.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.065+0000 7f5dde000640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5db006bd70 0x7f5db006e230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8001540 tx=0x7f5dc80079c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:18.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.069+0000 7f5dd6000640 1 -- 172.21.15.141:0/796879387 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5dd0054da0 con 0x7f5dd815d4b0 2024-01-16T15:06:18.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.295+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dd80cc370 con 0x7f5db006bd70 2024-01-16T15:06:18.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.312+0000 7f5dd6000640 1 -- 172.21.15.141:0/796879387 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f5dd80cc370 con 0x7f5db006bd70 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:18.291 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:18.292 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:18.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5db006bd70 msgr2=0x7f5db006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5db006bd70 0x7f5db006e230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8001540 tx=0x7f5dc80079c0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd815d4b0 msgr2=0x7f5dd815bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd815d4b0 0x7f5dd815bd80 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000b970 tx=0x7f5dd000be30 comp rx=0 tx=0).stop 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 shutdown_connections 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5db006bd70 0x7f5db006e230 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5dd815d4b0 0x7f5dd815bd80 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5dd80cf9a0 0x7f5dd815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 --2- 172.21.15.141:0/796879387 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5dd80b31d0 0x7f5dd80ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 >> 172.21.15.141:0/796879387 conn(0x7f5dd80d3000 msgr2=0x7f5dd806ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 shutdown_connections 2024-01-16T15:06:18.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:18.315+0000 7f5ddf400640 1 -- 172.21.15.141:0/796879387 wait complete. 2024-01-16T15:06:18.312 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:19.313 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59634 2024-01-16T15:06:19.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.560+0000 7f2b32e00640 1 Processor -- start 2024-01-16T15:06:19.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.560+0000 7f2b32e00640 1 -- start start 2024-01-16T15:06:19.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b32e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:19.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b32e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2b2c0cf290 0x7f2b2c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:19.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b32e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2c070c40 con 0x7f2b2c0ae5a0 2024-01-16T15:06:19.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b32e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2c071450 con 0x7f2b2c0cf290 2024-01-16T15:06:19.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b32e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0cfc60 con 0x7f2b2c0b3ae0 2024-01-16T15:06:19.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b30e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2b2c0cf290 0x7f2b2c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:19.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b2be00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f2b2c0ae5a0 0x7f2b2c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:42388/0 (socket says 172.21.15.141:42388) 2024-01-16T15:06:19.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.562+0000 7f2b2be00640 1 -- 172.21.15.141:0/3603848372 learned_addr learned my addr 172.21.15.141:0/3603848372 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:19.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.563+0000 7f2b23200640 1 --2- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.563+0000 7f2b30e00640 1 -- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0b3ae0 msgr2=0x7f2b2c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.563+0000 7f2b30e00640 1 --2- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.563+0000 7f2b30e00640 1 -- 172.21.15.141:0/3603848372 >> v1:172.21.15.188:6789/0 conn(0x7f2b2c0ae5a0 legacy=0x7f2b2c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.563+0000 7f2b30e00640 1 -- 172.21.15.141:0/3603848372 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c06c470 con 0x7f2b2c0cf290 2024-01-16T15:06:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.564+0000 7f2b23200640 1 --2- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:19.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.563+0000 7f2b2b400640 1 -- 172.21.15.141:0/3603848372 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663358143 0 0) 0x7f2b2c070c40 con 0x7f2b2c0ae5a0 2024-01-16T15:06:19.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.565+0000 7f2b30e00640 1 --2- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c0cf720 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009d90 tx=0x7f2b2402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4c4f54f6a7a12a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:19.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b2b400640 1 -- 172.21.15.141:0/3603848372 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2402cce0 con 0x7f2b2c0cf290 2024-01-16T15:06:19.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b2b400640 1 -- 172.21.15.141:0/3603848372 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2402ce40 con 0x7f2b2c0cf290 2024-01-16T15:06:19.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b2b400640 1 -- 172.21.15.141:0/3603848372 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24034660 con 0x7f2b2c0cf290 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b32e00640 1 -- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0cf290 msgr2=0x7f2b2c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c0cf720 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009d90 tx=0x7f2b2402bf30 comp rx=0 tx=0).stop 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b32e00640 1 -- 172.21.15.141:0/3603848372 shutdown_connections 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c0cf720 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3603848372 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b32e00640 1 -- 172.21.15.141:0/3603848372 >> 172.21.15.141:0/3603848372 conn(0x7f2b2c06a800 msgr2=0x7f2b2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.566+0000 7f2b32e00640 1 -- 172.21.15.141:0/3603848372 shutdown_connections 2024-01-16T15:06:19.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.567+0000 7f2b32e00640 1 -- 172.21.15.141:0/3603848372 wait complete. 2024-01-16T15:06:19.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.568+0000 7f2b32e00640 1 Processor -- start 2024-01-16T15:06:19.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.568+0000 7f2b32e00640 1 -- start start 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.569+0000 7f2b32e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0ae5a0 0x7f2b2c1f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.569+0000 7f2b32e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.569+0000 7f2b32e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c15c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.569+0000 7f2b32e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c157ca0 con 0x7f2b2c0cf290 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.569+0000 7f2b32e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c157f80 con 0x7f2b2c0b3ae0 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.569+0000 7f2b32e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c158260 con 0x7f2b2c0ae5a0 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c15c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c15c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48066/0 (socket says 172.21.15.141:48066) 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 -- 172.21.15.141:0/3626308440 learned_addr learned my addr 172.21.15.141:0/3626308440 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b2be00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0ae5a0 0x7f2b2c1f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b23200640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:19.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 -- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0ae5a0 msgr2=0x7f2b2c1f2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0ae5a0 0x7f2b2c1f2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 -- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0b3ae0 msgr2=0x7f2b2c15c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 -- 172.21.15.141:0/3626308440 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14009400 con 0x7f2b2c0cf290 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.570+0000 7f2b30e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c15c6a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009d90 tx=0x7f2b24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.571+0000 7f2b28c00640 1 -- 172.21.15.141:0/3626308440 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2403e670 con 0x7f2b2c0cf290 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.571+0000 7f2b28c00640 1 -- 172.21.15.141:0/3626308440 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2403e7f0 con 0x7f2b2c0cf290 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.571+0000 7f2b28c00640 1 -- 172.21.15.141:0/3626308440 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2403eba0 con 0x7f2b2c0cf290 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.571+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b24009a70 con 0x7f2b2c0cf290 2024-01-16T15:06:19.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.571+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c156f30 con 0x7f2b2c0cf290 2024-01-16T15:06:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.572+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c0675d0 con 0x7f2b2c0cf290 2024-01-16T15:06:19.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.573+0000 7f2b28c00640 1 -- 172.21.15.141:0/3626308440 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f2b2403b040 con 0x7f2b2c0cf290 2024-01-16T15:06:19.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.574+0000 7f2b28c00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b0406bd70 0x7f2b0406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:19.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.574+0000 7f2b28c00640 1 -- 172.21.15.141:0/3626308440 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b240aaf10 con 0x7f2b2c0cf290 2024-01-16T15:06:19.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.574+0000 7f2b2be00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b0406bd70 0x7f2b0406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:19.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.575+0000 7f2b2be00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b0406bd70 0x7f2b0406e230 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c000f60 tx=0x7f2b1c007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:19.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.578+0000 7f2b28c00640 1 -- 172.21.15.141:0/3626308440 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b240777d0 con 0x7f2b2c0cf290 2024-01-16T15:06:19.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.810+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b2c071170 con 0x7f2b0406bd70 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.824+0000 7f2b28c00640 1 -- 172.21.15.141:0/3626308440 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f2b1c0076a0 con 0x7f2b0406bd70 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:06:19.802 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:19.803 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:19.803 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:19.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b0406bd70 msgr2=0x7f2b0406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:19.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b0406bd70 0x7f2b0406e230 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c000f60 tx=0x7f2b1c007b50 comp rx=0 tx=0).stop 2024-01-16T15:06:19.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b2c0cf290 msgr2=0x7f2b2c15c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:19.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c15c6a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009d90 tx=0x7f2b24004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:19.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 shutdown_connections 2024-01-16T15:06:19.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2b0406bd70 0x7f2b0406e230 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b2c0cf290 0x7f2b2c15c6a0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b2c0b3ae0 0x7f2b2c15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 --2- 172.21.15.141:0/3626308440 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2b2c0ae5a0 0x7f2b2c1f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:19.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 >> 172.21.15.141:0/3626308440 conn(0x7f2b2c06a800 msgr2=0x7f2b2c14fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:19.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 shutdown_connections 2024-01-16T15:06:19.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:19.827+0000 7f2b32e00640 1 -- 172.21.15.141:0/3626308440 wait complete. 2024-01-16T15:06:19.825 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:20.826 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59634 2024-01-16T15:06:21.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.070+0000 7f7a11e00640 1 Processor -- start 2024-01-16T15:06:21.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.071+0000 7f7a11e00640 1 -- start start 2024-01-16T15:06:21.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.072+0000 7f7a11e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7a0c0ccb50 0x7f7a0c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.072+0000 7f7a11e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 0x7f7a0c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.072+0000 7f7a11e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c15f4e0 con 0x7f7a0c0adf20 2024-01-16T15:06:21.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.072+0000 7f7a11e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0c15f970 con 0x7f7a0c0ccb50 2024-01-16T15:06:21.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.072+0000 7f7a11e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c15c3c0 con 0x7f7a0c15bf10 2024-01-16T15:06:21.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.072+0000 7f7a10a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7a0c0adf20 0x7f7a0c0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35172/0 (socket says 172.21.15.141:35172) 2024-01-16T15:06:21.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.072+0000 7f7a10a00640 1 -- 172.21.15.141:0/916536978 learned_addr learned my addr 172.21.15.141:0/916536978 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:21.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a11400640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 0x7f7a0c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566448771 0 0) 0x7f7a0c15f4e0 con 0x7f7a0c0adf20 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0be00640 1 --2- 172.21.15.141:0/916536978 >> v2:172.21.15.188:3300/0 conn(0x7f7a0c0ccb50 0x7f7a0c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ec003520 con 0x7f7a0c0adf20 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136795993 0 0) 0x7f79ec003520 con 0x7f7a0c0adf20 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 msgr2=0x7f7a0c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 0x7f7a0c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 >> v2:172.21.15.188:3300/0 conn(0x7f7a0c0ccb50 msgr2=0x7f7a0c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 --2- 172.21.15.141:0/916536978 >> v2:172.21.15.188:3300/0 conn(0x7f7a0c0ccb50 0x7f7a0c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c071130 con 0x7f7a0c0adf20 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f79fc001670 con 0x7f7a0c0adf20 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 >> v1:172.21.15.141:6789/0 conn(0x7f7a0c0adf20 legacy=0x7f7a0c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79ec003970 0x7f79ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 0x7f79ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79ec004d00 0x7f79ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c15f970 con 0x7f79ec003970 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c15c3c0 con 0x7f79ec004d00 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.073+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ec00b980 con 0x7f7a0c15bf10 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a10a00640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79ec003970 0x7f79ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a11400640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79ec004d00 0x7f79ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a0be00640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 0x7f79ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a10a00640 1 -- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 msgr2=0x7f79ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a10a00640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a0c15bf10 0x7f79ec0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a10a00640 1 -- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79ec004d00 msgr2=0x7f79ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a10a00640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79ec004d00 0x7f79ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a10a00640 1 -- 172.21.15.141:0/916536978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec003520 con 0x7f79ec003970 2024-01-16T15:06:21.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.074+0000 7f7a11400640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79ec004d00 0x7f79ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.075+0000 7f7a10a00640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79ec003970 0x7f79ec003d90 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00c240 tx=0x7f79ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5d2ab28bc86e43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.075+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc005ad0 con 0x7f79ec003970 2024-01-16T15:06:21.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79fc005c30 con 0x7f79ec003970 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a0b400640 1 -- 172.21.15.141:0/916536978 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc02c8d0 con 0x7f79ec003970 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a11e00640 1 -- 172.21.15.141:0/916536978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79ec003970 msgr2=0x7f79ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a11e00640 1 --2- 172.21.15.141:0/916536978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79ec003970 0x7f79ec003d90 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00c240 tx=0x7f79ec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a11400640 1 -- 172.21.15.141:0/916536978 reap_dead start 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a11e00640 1 -- 172.21.15.141:0/916536978 shutdown_connections 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a11e00640 1 -- 172.21.15.141:0/916536978 >> 172.21.15.141:0/916536978 conn(0x7f7a0c06a7d0 msgr2=0x7f7a0c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a11e00640 1 -- 172.21.15.141:0/916536978 shutdown_connections 2024-01-16T15:06:21.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.076+0000 7f7a11e00640 1 -- 172.21.15.141:0/916536978 wait complete. 2024-01-16T15:06:21.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.078+0000 7f7a11e00640 1 Processor -- start 2024-01-16T15:06:21.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.078+0000 7f7a11e00640 1 -- start start 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.079+0000 7f7a11e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79ec003970 0x7f7a0c1f67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.079+0000 7f7a11e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a0c0afcf0 0x7f7a0c1f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.079+0000 7f7a11e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a0c1f2da0 0x7f7a0c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.079+0000 7f7a11e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c1f5de0 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.079+0000 7f7a11e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c1f63a0 con 0x7f7a0c0afcf0 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.079+0000 7f7a11e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c1f6d00 con 0x7f79ec003970 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.079+0000 7f7a11400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a0c1f2da0 0x7f7a0c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a0c1f2da0 0x7f7a0c0c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48080/0 (socket says 172.21.15.141:48080) 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 -- 172.21.15.141:0/2119303453 learned_addr learned my addr 172.21.15.141:0/2119303453 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a0be00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a0c0afcf0 0x7f7a0c1f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a10a00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79ec003970 0x7f7a0c1f67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 -- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79ec003970 msgr2=0x7f7a0c1f67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79ec003970 0x7f7a0c1f67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 -- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a0c0afcf0 msgr2=0x7f7a0c1f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a0c0afcf0 0x7f7a0c1f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 -- 172.21.15.141:0/2119303453 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c1f3240 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a10a00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79ec003970 0x7f7a0c1f67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a11400640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a0c1f2da0 0x7f7a0c0c6460 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0b4270 tx=0x7f7a0400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.080+0000 7f7a08c00640 1 -- 172.21.15.141:0/2119303453 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04017400 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.081+0000 7f7a08c00640 1 -- 172.21.15.141:0/2119303453 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a04011040 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.081+0000 7f7a08c00640 1 -- 172.21.15.141:0/2119303453 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04017780 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.081+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0c1f3520 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.081+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0c0c3e50 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.082+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0c1f64e0 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.083+0000 7f7a08c00640 1 -- 172.21.15.141:0/2119303453 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f7a040203b0 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.084+0000 7f7a08c00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79e406bd70 0x7f79e406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.084+0000 7f7a08c00640 1 -- 172.21.15.141:0/2119303453 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a04088b00 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.084+0000 7f7a10a00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79e406bd70 0x7f79e406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.085+0000 7f7a10a00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79e406bd70 0x7f79e406e230 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f79fc034000 tx=0x7f79fc007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.088+0000 7f7a08c00640 1 -- 172.21.15.141:0/2119303453 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a0c1f64e0 con 0x7f7a0c1f2da0 2024-01-16T15:06:21.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.315+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a0c0c4070 con 0x7f79e406bd70 2024-01-16T15:06:21.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.334+0000 7f7a08c00640 1 -- 172.21.15.141:0/2119303453 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f7a0c0c4070 con 0x7f79e406bd70 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:21.312 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79e406bd70 msgr2=0x7f79e406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79e406bd70 0x7f79e406e230 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f79fc034000 tx=0x7f79fc007000 comp rx=0 tx=0).stop 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a0c1f2da0 msgr2=0x7f7a0c0c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a0c1f2da0 0x7f7a0c0c6460 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0b4270 tx=0x7f7a0400f4a0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 shutdown_connections 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79e406bd70 0x7f79e406e230 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a0c1f2da0 0x7f7a0c0c6460 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a0c0afcf0 0x7f7a0c1f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 --2- 172.21.15.141:0/2119303453 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79ec003970 0x7f7a0c1f67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.336+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 >> 172.21.15.141:0/2119303453 conn(0x7f7a0c06a7d0 msgr2=0x7f7a0c06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:21.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.337+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 shutdown_connections 2024-01-16T15:06:21.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:21.337+0000 7f7a11e00640 1 -- 172.21.15.141:0/2119303453 wait complete. 2024-01-16T15:06:21.332 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:21.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:21.371+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:21.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:21.371+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:21.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:21.371+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:21.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:21.371+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:21.349 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:21.371+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:22.333 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59634 2024-01-16T15:06:22.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.578+0000 7fa59aa00640 1 Processor -- start 2024-01-16T15:06:22.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.578+0000 7fa59aa00640 1 -- start start 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.580+0000 7fa59aa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa5940cf9a0 0x7fa59415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.580+0000 7fa59aa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa59415e1e0 0x7fa5940702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.580+0000 7fa59aa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5940b1ee0 con 0x7fa594143ae0 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.580+0000 7fa59aa00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5940b2780 con 0x7fa5940cf9a0 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.580+0000 7fa59aa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59415e990 con 0x7fa59415e1e0 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.580+0000 7fa598a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa59415e1e0 0x7fa5940702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.580+0000 7fa593400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa5940cf9a0 0x7fa59415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.581+0000 7fa593e00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa594143ae0 0x7fa5940cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:42404/0 (socket says 172.21.15.141:42404) 2024-01-16T15:06:22.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.581+0000 7fa593e00640 1 -- 172.21.15.141:0/3257045713 learned_addr learned my addr 172.21.15.141:0/3257045713 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:22.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.581+0000 7fa593400640 1 -- 172.21.15.141:0/3257045713 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa59415e1e0 msgr2=0x7fa5940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:22.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.581+0000 7fa593400640 1 --2- 172.21.15.141:0/3257045713 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa59415e1e0 0x7fa5940702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:22.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.581+0000 7fa593400640 1 -- 172.21.15.141:0/3257045713 >> v1:172.21.15.188:6789/0 conn(0x7fa594143ae0 legacy=0x7fa5940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:22.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.582+0000 7fa593400640 1 -- 172.21.15.141:0/3257045713 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940b2130 con 0x7fa5940cf9a0 2024-01-16T15:06:22.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.583+0000 7fa593400640 1 --2- 172.21.15.141:0/3257045713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5940cf9a0 0x7fa59415dca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0097a0 tx=0x7fa57c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1db842d6617775a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:22.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.583+0000 7fa592a00640 1 -- 172.21.15.141:0/3257045713 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c031830 con 0x7fa5940cf9a0 2024-01-16T15:06:22.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa592a00640 1 -- 172.21.15.141:0/3257045713 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c031990 con 0x7fa5940cf9a0 2024-01-16T15:06:22.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa592a00640 1 -- 172.21.15.141:0/3257045713 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c031d20 con 0x7fa5940cf9a0 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 -- 172.21.15.141:0/3257045713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5940cf9a0 msgr2=0x7fa59415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 --2- 172.21.15.141:0/3257045713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5940cf9a0 0x7fa59415dca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0097a0 tx=0x7fa57c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 -- 172.21.15.141:0/3257045713 shutdown_connections 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 --2- 172.21.15.141:0/3257045713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5940cf9a0 0x7fa59415dca0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 --2- 172.21.15.141:0/3257045713 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa59415e1e0 0x7fa5940702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 -- 172.21.15.141:0/3257045713 >> 172.21.15.141:0/3257045713 conn(0x7fa59406a800 msgr2=0x7fa59406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 -- 172.21.15.141:0/3257045713 shutdown_connections 2024-01-16T15:06:22.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.584+0000 7fa59aa00640 1 -- 172.21.15.141:0/3257045713 wait complete. 2024-01-16T15:06:22.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.587+0000 7fa59aa00640 1 Processor -- start 2024-01-16T15:06:22.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.587+0000 7fa59aa00640 1 -- start start 2024-01-16T15:06:22.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.588+0000 7fa59aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5940cf9a0 0x7fa594140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:22.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.588+0000 7fa59aa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa594143ae0 0x7fa594140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:22.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.588+0000 7fa59aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59415e1e0 0x7fa594159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.588+0000 7fa59aa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940ae540 con 0x7fa5940cf9a0 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.588+0000 7fa59aa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940cc960 con 0x7fa59415e1e0 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.588+0000 7fa59aa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940ccc40 con 0x7fa594143ae0 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5940cf9a0 0x7fa594140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5940cf9a0 0x7fa594140760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48094/0 (socket says 172.21.15.141:48094) 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 -- 172.21.15.141:0/2284188547 learned_addr learned my addr 172.21.15.141:0/2284188547 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa598a00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59415e1e0 0x7fa594159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593400640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa594143ae0 0x7fa594140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 -- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa594143ae0 msgr2=0x7fa594140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa594143ae0 0x7fa594140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 -- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59415e1e0 msgr2=0x7fa594159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59415e1e0 0x7fa594159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:22.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa593e00640 1 -- 172.21.15.141:0/2284188547 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c009400 con 0x7fa5940cf9a0 2024-01-16T15:06:22.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.589+0000 7fa598a00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59415e1e0 0x7fa594159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:22.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.590+0000 7fa593e00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5940cf9a0 0x7fa594140760 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c4a0 tx=0x7fa58400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:22.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.590+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584005ce0 con 0x7fa5940cf9a0 2024-01-16T15:06:22.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.590+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa584005e40 con 0x7fa5940cf9a0 2024-01-16T15:06:22.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.590+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5940ccf20 con 0x7fa5940cf9a0 2024-01-16T15:06:22.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.590+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584007270 con 0x7fa5940cf9a0 2024-01-16T15:06:22.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.590+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5940cd4e0 con 0x7fa5940cf9a0 2024-01-16T15:06:22.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.591+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59406e0d0 con 0x7fa5940cf9a0 2024-01-16T15:06:22.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.592+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fa584007490 con 0x7fa5940cf9a0 2024-01-16T15:06:22.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.594+0000 7fa58be00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa56c06bd70 0x7fa56c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:22.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.594+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa584087bd0 con 0x7fa5940cf9a0 2024-01-16T15:06:22.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.594+0000 7fa593400640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa56c06bd70 0x7fa56c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:22.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.595+0000 7fa593400640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa56c06bd70 0x7fa56c06e230 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa57c036000 tx=0x7fa57c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:22.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.598+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa584054370 con 0x7fa5940cf9a0 2024-01-16T15:06:22.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:22.826+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7fa59415e670 con 0x7fa56c06bd70 2024-01-16T15:06:23.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:23.681+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 23) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fa584054740 con 0x7fa5940cf9a0 2024-01-16T15:06:26.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.350+0000 7fa58be00640 1 -- 172.21.15.141:0/2284188547 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fa57c03a420 con 0x7fa56c06bd70 2024-01-16T15:06:26.328 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:26.328 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:06:26.328 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:06:26.328 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:06:26.329 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:06:26.329 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:06:26.329 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:06:26.329 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:26.329 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:26.329 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:26.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa56c06bd70 msgr2=0x7fa56c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa56c06bd70 0x7fa56c06e230 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa57c036000 tx=0x7fa57c03c040 comp rx=0 tx=0).stop 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5940cf9a0 msgr2=0x7fa594140760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5940cf9a0 0x7fa594140760 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c4a0 tx=0x7fa58400c960 comp rx=0 tx=0).stop 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 shutdown_connections 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa56c06bd70 0x7fa56c06e230 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59415e1e0 0x7fa594159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa594143ae0 0x7fa594140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 --2- 172.21.15.141:0/2284188547 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5940cf9a0 0x7fa594140760 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 >> 172.21.15.141:0/2284188547 conn(0x7fa59406a800 msgr2=0x7fa59406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 shutdown_connections 2024-01-16T15:06:26.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:26.353+0000 7fa59aa00640 1 -- 172.21.15.141:0/2284188547 wait complete. 2024-01-16T15:06:26.346 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:26.368+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.346 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:26.368+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.346 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:26.368+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.346 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:26.368+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.346 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:06:26.368+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.354 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:06:27.355 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59634 2024-01-16T15:06:27.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.669+0000 7fa370c00640 1 Processor -- start 2024-01-16T15:06:27.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.669+0000 7fa370c00640 1 -- start start 2024-01-16T15:06:27.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa370c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa36c13df80 0x7fa36c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa370c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c0cf9a0 0x7fa36c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa370c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa36c06c470 con 0x7fa36c13e360 2024-01-16T15:06:27.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa370c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36c06c750 con 0x7fa36c13df80 2024-01-16T15:06:27.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa370c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c06ca30 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa363e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa36c13e360 0x7fa36c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34962/0 (socket says 172.21.15.141:34962) 2024-01-16T15:06:27.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa363e00640 1 -- 172.21.15.141:0/2213327041 learned_addr learned my addr 172.21.15.141:0/2213327041 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:27.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.671+0000 7fa36a800640 1 --2- 172.21.15.141:0/2213327041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c0cf9a0 0x7fa36c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.672+0000 7fa369e00640 1 --2- 172.21.15.141:0/2213327041 >> v2:172.21.15.188:3300/0 conn(0x7fa36c13df80 0x7fa36c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.672+0000 7fa36a800640 1 -- 172.21.15.141:0/2213327041 >> v2:172.21.15.188:3300/0 conn(0x7fa36c13df80 msgr2=0x7fa36c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:27.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.672+0000 7fa36a800640 1 --2- 172.21.15.141:0/2213327041 >> v2:172.21.15.188:3300/0 conn(0x7fa36c13df80 0x7fa36c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.672+0000 7fa36a800640 1 -- 172.21.15.141:0/2213327041 >> v1:172.21.15.141:6789/0 conn(0x7fa36c13e360 legacy=0x7fa36c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.673+0000 7fa36a800640 1 -- 172.21.15.141:0/2213327041 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c06cd10 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.673+0000 7fa369e00640 1 --2- 172.21.15.141:0/2213327041 >> v2:172.21.15.188:3300/0 conn(0x7fa36c13df80 0x7fa36c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:27.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.675+0000 7fa36a800640 1 --2- 172.21.15.141:0/2213327041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c0cf9a0 0x7fa36c0cd840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa364009d90 tx=0x7fa36402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a2132010a53bde server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.676+0000 7fa369400640 1 -- 172.21.15.141:0/2213327041 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36402cc50 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.676+0000 7fa369400640 1 -- 172.21.15.141:0/2213327041 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36402cdb0 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.676+0000 7fa369400640 1 -- 172.21.15.141:0/2213327041 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364034660 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.677+0000 7fa370c00640 1 -- 172.21.15.141:0/2213327041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c0cf9a0 msgr2=0x7fa36c0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.677+0000 7fa370c00640 1 --2- 172.21.15.141:0/2213327041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c0cf9a0 0x7fa36c0cd840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa364009d90 tx=0x7fa36402bea0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.677+0000 7fa370c00640 1 -- 172.21.15.141:0/2213327041 shutdown_connections 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.677+0000 7fa370c00640 1 --2- 172.21.15.141:0/2213327041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c0cf9a0 0x7fa36c0cd840 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.677+0000 7fa370c00640 1 --2- 172.21.15.141:0/2213327041 >> v2:172.21.15.188:3300/0 conn(0x7fa36c13df80 0x7fa36c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.677+0000 7fa370c00640 1 -- 172.21.15.141:0/2213327041 >> 172.21.15.141:0/2213327041 conn(0x7fa36c06a800 msgr2=0x7fa36c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.678+0000 7fa370c00640 1 -- 172.21.15.141:0/2213327041 shutdown_connections 2024-01-16T15:06:27.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.678+0000 7fa370c00640 1 -- 172.21.15.141:0/2213327041 wait complete. 2024-01-16T15:06:27.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.680+0000 7fa370c00640 1 Processor -- start 2024-01-16T15:06:27.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.681+0000 7fa370c00640 1 -- start start 2024-01-16T15:06:27.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa370c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa36c0cf9a0 0x7fa36c13d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa370c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36c13df80 0x7fa36c14e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa370c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c13e360 0x7fa36c0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa370c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c14e110 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa370c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c14de30 con 0x7fa36c13df80 2024-01-16T15:06:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa370c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c1f4050 con 0x7fa36c13e360 2024-01-16T15:06:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa369e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa36c0cf9a0 0x7fa36c13d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa36a800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c13e360 0x7fa36c0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa36a800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c13e360 0x7fa36c0b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:59798/0 (socket says 172.21.15.141:59798) 2024-01-16T15:06:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa369e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa36c0cf9a0 0x7fa36c13d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59894/0 (socket says 172.21.15.141:59894) 2024-01-16T15:06:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.682+0000 7fa36a800640 1 -- 172.21.15.141:0/1440893992 learned_addr learned my addr 172.21.15.141:0/1440893992 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.683+0000 7fa369e00640 1 -- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c13e360 msgr2=0x7fa36c0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.683+0000 7fa369e00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c13e360 0x7fa36c0b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.683+0000 7fa369e00640 1 -- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36c13df80 msgr2=0x7fa36c14e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.683+0000 7fa369e00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36c13df80 0x7fa36c14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.683+0000 7fa369e00640 1 -- 172.21.15.141:0/1440893992 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c009400 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.683+0000 7fa369e00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa36c0cf9a0 0x7fa36c13d030 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00e900 tx=0x7fa35c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.684+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c00ca00 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.684+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c00cb60 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.684+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c013470 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.685+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa364009a70 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.685+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c1f4610 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.687+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c0675d0 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.688+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 23) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fa35c0135d0 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.690+0000 7fa362000640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa34406be00 0x7fa34406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.690+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa35c088ae0 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.695+0000 7fa363e00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa34406be00 0x7fa34406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.702+0000 7fa363e00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa34406be00 0x7fa34406e2c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0b3b40 tx=0x7fa354007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.702+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa35c055090 con 0x7fa36c0cf9a0 2024-01-16T15:06:27.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:27.806+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fa35c055460 con 0x7fa36c0cf9a0 2024-01-16T15:06:28.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.057+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7fa36c1f41c0 con 0x7fa34406be00 2024-01-16T15:06:28.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.160+0000 7fa362000640 1 -- 172.21.15.141:0/1440893992 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fa36c1f41c0 con 0x7fa34406be00 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "in-progress", 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_report": { 2024-01-16T15:06:28.139 INFO:teuthology.orchestra.run.smithi141.stdout: "percentage completed": 0.0, 2024-01-16T15:06:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:06:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "regfiles copied": "0/3", 2024-01-16T15:06:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:28.140 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:28.140 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:28.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.164+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa34406be00 msgr2=0x7fa34406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.164+0000 7fa370c00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa34406be00 0x7fa34406e2c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0b3b40 tx=0x7fa354007a40 comp rx=0 tx=0).stop 2024-01-16T15:06:28.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa36c0cf9a0 msgr2=0x7fa36c13d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa36c0cf9a0 0x7fa36c13d030 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00e900 tx=0x7fa35c00edc0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 shutdown_connections 2024-01-16T15:06:28.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa34406be00 0x7fa34406e2c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa36c13e360 0x7fa36c0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36c13df80 0x7fa36c14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 --2- 172.21.15.141:0/1440893992 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa36c0cf9a0 0x7fa36c13d030 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 >> 172.21.15.141:0/1440893992 conn(0x7fa36c06a800 msgr2=0x7fa36c0afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:28.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 shutdown_connections 2024-01-16T15:06:28.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.165+0000 7fa370c00640 1 -- 172.21.15.141:0/1440893992 wait complete. 2024-01-16T15:06:28.164 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:06:28.165 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59637 2024-01-16T15:06:28.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.469+0000 7fd22c000640 1 Processor -- start 2024-01-16T15:06:28.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.469+0000 7fd22c000640 1 -- start start 2024-01-16T15:06:28.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.471+0000 7fd22c000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd224160f80 0x7fd224161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd22c000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2241618a0 con 0x7fd2240ce8f0 2024-01-16T15:06:28.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd22c000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2240d0a00 con 0x7fd2240ced50 2024-01-16T15:06:28.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd22c000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2240cf1e0 con 0x7fd224160f80 2024-01-16T15:06:28.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd229600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd224160f80 0x7fd224161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd229600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd224160f80 0x7fd224161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59902/0 (socket says 172.21.15.141:59902) 2024-01-16T15:06:28.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd229600640 1 -- 172.21.15.141:0/3638115896 learned_addr learned my addr 172.21.15.141:0/3638115896 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:28.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd228c00640 1 --1- 172.21.15.141:0/3638115896 >> v1:172.21.15.141:6789/0 conn(0x7fd2240ce8f0 0x7fd2240cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34966/0 (socket says 172.21.15.141:34966) 2024-01-16T15:06:28.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd229600640 1 -- 172.21.15.141:0/3638115896 >> v1:172.21.15.188:6789/0 conn(0x7fd2240ced50 legacy=0x7fd2240d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:28.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.472+0000 7fd229600640 1 -- 172.21.15.141:0/3638115896 >> v1:172.21.15.141:6789/0 conn(0x7fd2240ce8f0 legacy=0x7fd2240cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.473+0000 7fd229600640 1 -- 172.21.15.141:0/3638115896 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240d0de0 con 0x7fd224160f80 2024-01-16T15:06:28.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.473+0000 7fd223e00640 1 -- 172.21.15.141:0/3638115896 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034586196 0 0) 0x7fd2241618a0 con 0x7fd2240ce8f0 2024-01-16T15:06:28.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.474+0000 7fd229600640 1 --2- 172.21.15.141:0/3638115896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd224161360 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd214009870 tx=0x7fd21402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c39ab70eb682955 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.474+0000 7fd223e00640 1 -- 172.21.15.141:0/3638115896 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214030850 con 0x7fd224160f80 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.474+0000 7fd223e00640 1 -- 172.21.15.141:0/3638115896 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2140309b0 con 0x7fd224160f80 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.474+0000 7fd223e00640 1 -- 172.21.15.141:0/3638115896 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214030d40 con 0x7fd224160f80 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.475+0000 7fd22c000640 1 -- 172.21.15.141:0/3638115896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 msgr2=0x7fd224161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.475+0000 7fd22c000640 1 --2- 172.21.15.141:0/3638115896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd224161360 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd214009870 tx=0x7fd21402b990 comp rx=0 tx=0).stop 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.475+0000 7fd22c000640 1 -- 172.21.15.141:0/3638115896 shutdown_connections 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.475+0000 7fd22c000640 1 --2- 172.21.15.141:0/3638115896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd224161360 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.475+0000 7fd22c000640 1 -- 172.21.15.141:0/3638115896 >> 172.21.15.141:0/3638115896 conn(0x7fd22409c550 msgr2=0x7fd2240d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.475+0000 7fd22c000640 1 -- 172.21.15.141:0/3638115896 shutdown_connections 2024-01-16T15:06:28.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.475+0000 7fd22c000640 1 -- 172.21.15.141:0/3638115896 wait complete. 2024-01-16T15:06:28.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.477+0000 7fd22c000640 1 Processor -- start 2024-01-16T15:06:28.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.477+0000 7fd22c000640 1 -- start start 2024-01-16T15:06:28.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.478+0000 7fd22c000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2240ce8f0 0x7fd224160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.478+0000 7fd22c000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2240ced50 0x7fd2241504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.478+0000 7fd22c000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd2240cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.478+0000 7fd22c000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2240c73a0 con 0x7fd224160f80 2024-01-16T15:06:28.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.478+0000 7fd22c000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2241605c0 con 0x7fd2240ce8f0 2024-01-16T15:06:28.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.478+0000 7fd22c000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2240c9b20 con 0x7fd2240ced50 2024-01-16T15:06:28.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd2240cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd2240cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59908/0 (socket says 172.21.15.141:59908) 2024-01-16T15:06:28.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 -- 172.21.15.141:0/1626784813 learned_addr learned my addr 172.21.15.141:0/1626784813 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:28.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd228c00640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2240ced50 0x7fd2241504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd229600640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2240ce8f0 0x7fd224160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 -- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2240ced50 msgr2=0x7fd2241504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2240ced50 0x7fd2241504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 -- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2240ce8f0 msgr2=0x7fd224160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2240ce8f0 0x7fd224160080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.479+0000 7fd22a000640 1 -- 172.21.15.141:0/1626784813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2140094d0 con 0x7fd224160f80 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.480+0000 7fd22a000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd2240cc4b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00c780 tx=0x7fd21c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.480+0000 7fd221600640 1 -- 172.21.15.141:0/1626784813 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c003d80 con 0x7fd224160f80 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.480+0000 7fd221600640 1 -- 172.21.15.141:0/1626784813 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c00ce90 con 0x7fd224160f80 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.480+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2240c6de0 con 0x7fd224160f80 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.480+0000 7fd221600640 1 -- 172.21.15.141:0/1626784813 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0052b0 con 0x7fd224160f80 2024-01-16T15:06:28.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.480+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2240c9e00 con 0x7fd224160f80 2024-01-16T15:06:28.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.482+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2240675d0 con 0x7fd224160f80 2024-01-16T15:06:28.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.484+0000 7fd221600640 1 -- 172.21.15.141:0/1626784813 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd21c005410 con 0x7fd224160f80 2024-01-16T15:06:28.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.485+0000 7fd221600640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1fc06bee0 0x7fd1fc06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:28.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.485+0000 7fd221600640 1 -- 172.21.15.141:0/1626784813 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd21c088090 con 0x7fd224160f80 2024-01-16T15:06:28.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.487+0000 7fd229600640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1fc06bee0 0x7fd1fc06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:28.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.492+0000 7fd221600640 1 -- 172.21.15.141:0/1626784813 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd21c0546f0 con 0x7fd224160f80 2024-01-16T15:06:28.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.493+0000 7fd229600640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1fc06bee0 0x7fd1fc06e3a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd214036000 tx=0x7fd21403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:28.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.886+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59637", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2240d1680 con 0x7fd1fc06bee0 2024-01-16T15:06:28.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.914+0000 7fd221600640 1 -- 172.21.15.141:0/1626784813 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fd214030ba0 con 0x7fd1fc06bee0 2024-01-16T15:06:28.892 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:28.892 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:06:28.892 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:06:28.892 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:06:28.892 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:06:28.892 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:06:28.893 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:06:28.893 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:28.893 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:28.893 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:28.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1fc06bee0 msgr2=0x7fd1fc06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1fc06bee0 0x7fd1fc06e3a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd214036000 tx=0x7fd21403b040 comp rx=0 tx=0).stop 2024-01-16T15:06:28.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 msgr2=0x7fd2240cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:28.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd2240cc4b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00c780 tx=0x7fd21c00cc40 comp rx=0 tx=0).stop 2024-01-16T15:06:28.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 shutdown_connections 2024-01-16T15:06:28.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd1fc06bee0 0x7fd1fc06e3a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd224160f80 0x7fd2240cc4b0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2240ced50 0x7fd2241504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 --2- 172.21.15.141:0/1626784813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2240ce8f0 0x7fd224160080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:28.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.917+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 >> 172.21.15.141:0/1626784813 conn(0x7fd22409c550 msgr2=0x7fd2240b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:28.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.918+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 shutdown_connections 2024-01-16T15:06:28.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:28.918+0000 7fd22c000640 1 -- 172.21.15.141:0/1626784813 wait complete. 2024-01-16T15:06:28.926 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.207+0000 7f4db4800640 1 Processor -- start 2024-01-16T15:06:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.208+0000 7f4db4800640 1 -- start start 2024-01-16T15:06:29.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.209+0000 7f4db4800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dac1578c0 0x7f4dac0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.209+0000 7f4db4800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac15dce0 con 0x7f4dac15e860 2024-01-16T15:06:29.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.209+0000 7f4db4800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac15dfc0 con 0x7f4dac06d270 2024-01-16T15:06:29.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.209+0000 7f4db4800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac15e2a0 con 0x7f4dac1578c0 2024-01-16T15:06:29.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.210+0000 7f4db1400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4dac15e860 0x7f4dac06cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34974/0 (socket says 172.21.15.141:34974) 2024-01-16T15:06:29.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.210+0000 7f4db1400640 1 -- 172.21.15.141:0/1599516244 learned_addr learned my addr 172.21.15.141:0/1599516244 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:29.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.210+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835419588 0 0) 0x7f4dac15dce0 con 0x7f4dac15e860 2024-01-16T15:06:29.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.210+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c003520 con 0x7f4dac15e860 2024-01-16T15:06:29.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116534932 0 0) 0x7f4d8c003520 con 0x7f4dac15e860 2024-01-16T15:06:29.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dac1578c0 msgr2=0x7f4dac0cee70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:29.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dac1578c0 0x7f4dac0cee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 >> v1:172.21.15.188:6789/0 conn(0x7f4dac06d270 legacy=0x7f4dac0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:29.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac15e580 con 0x7f4dac15e860 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4d94001670 con 0x7f4dac15e860 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 >> v1:172.21.15.141:6789/0 conn(0x7f4dac15e860 legacy=0x7f4dac06cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4d8c0039a0 0x7f4d8c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dac1578c0 0x7f4d8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8c004dd0 0x7f4d8c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac15e2a0 con 0x7f4d8c0039a0 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c00bbe0 con 0x7f4d8c004dd0 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c00bda0 con 0x7f4dac1578c0 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.211+0000 7f4db1e00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4d8c0039a0 0x7f4d8c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.212+0000 7f4db1e00640 1 -- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dac1578c0 msgr2=0x7f4d8c0047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:29.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.212+0000 7f4db1e00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dac1578c0 0x7f4d8c0047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.212+0000 7f4db1e00640 1 -- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8c004dd0 msgr2=0x7f4d8c005250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:29.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.212+0000 7f4db1e00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8c004dd0 0x7f4d8c005250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.212+0000 7f4db1e00640 1 -- 172.21.15.141:0/1599516244 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c003520 con 0x7f4d8c0039a0 2024-01-16T15:06:29.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.213+0000 7f4db1e00640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4d8c0039a0 0x7f4d8c003da0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009860 tx=0x7f4d8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8eab3aacdd2070 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:29.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.214+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c032600 con 0x7f4d8c0039a0 2024-01-16T15:06:29.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.214+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c02e050 con 0x7f4d8c0039a0 2024-01-16T15:06:29.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.214+0000 7f4db0a00640 1 -- 172.21.15.141:0/1599516244 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0329c0 con 0x7f4d8c0039a0 2024-01-16T15:06:29.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.215+0000 7f4db4800640 1 -- 172.21.15.141:0/1599516244 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4d8c0039a0 msgr2=0x7f4d8c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.215+0000 7f4db4800640 1 --2- 172.21.15.141:0/1599516244 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4d8c0039a0 0x7f4d8c003da0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009860 tx=0x7f4d8c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.215+0000 7f4db2800640 1 -- 172.21.15.141:0/1599516244 reap_dead start 2024-01-16T15:06:29.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.216+0000 7f4db4800640 1 -- 172.21.15.141:0/1599516244 shutdown_connections 2024-01-16T15:06:29.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.216+0000 7f4db4800640 1 -- 172.21.15.141:0/1599516244 >> 172.21.15.141:0/1599516244 conn(0x7f4dac0b2c70 msgr2=0x7f4dac0b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:29.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.216+0000 7f4db4800640 1 -- 172.21.15.141:0/1599516244 shutdown_connections 2024-01-16T15:06:29.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.216+0000 7f4db4800640 1 -- 172.21.15.141:0/1599516244 wait complete. 2024-01-16T15:06:29.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.219+0000 7f4db4800640 1 Processor -- start 2024-01-16T15:06:29.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.219+0000 7f4db4800640 1 -- start start 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.220+0000 7f4db4800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4d8c0039a0 0x7f4dac1f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.220+0000 7f4db4800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dac0c4e10 0x7f4dac0c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db4800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dac155ce0 0x7f4dac156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db4800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac1566a0 con 0x7f4dac155ce0 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db4800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac1f4130 con 0x7f4dac0c4e10 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db4800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0c58d0 con 0x7f4d8c0039a0 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dac155ce0 0x7f4dac156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dac155ce0 0x7f4dac156160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59918/0 (socket says 172.21.15.141:59918) 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db2800640 1 -- 172.21.15.141:0/698651319 learned_addr learned my addr 172.21.15.141:0/698651319 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db1e00640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4d8c0039a0 0x7f4dac1f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.221+0000 7f4db2800640 1 -- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4d8c0039a0 msgr2=0x7f4dac1f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4db2800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4d8c0039a0 0x7f4dac1f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4db2800640 1 -- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dac0c4e10 msgr2=0x7f4dac0c5290 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4db2800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dac0c4e10 0x7f4dac0c5290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4db2800640 1 -- 172.21.15.141:0/698651319 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0c5bb0 con 0x7f4dac155ce0 2024-01-16T15:06:29.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4db2800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dac155ce0 0x7f4dac156160 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f4dac072510 tx=0x7f4da400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:29.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4daa000640 1 -- 172.21.15.141:0/698651319 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4004d10 con 0x7f4dac155ce0 2024-01-16T15:06:29.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4daa000640 1 -- 172.21.15.141:0/698651319 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da400ced0 con 0x7f4dac155ce0 2024-01-16T15:06:29.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.222+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac0c5e90 con 0x7f4dac155ce0 2024-01-16T15:06:29.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.223+0000 7f4daa000640 1 -- 172.21.15.141:0/698651319 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da400a880 con 0x7f4dac155ce0 2024-01-16T15:06:29.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.223+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac0c6450 con 0x7f4dac155ce0 2024-01-16T15:06:29.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.224+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac0c6670 con 0x7f4dac155ce0 2024-01-16T15:06:29.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.226+0000 7f4daa000640 1 -- 172.21.15.141:0/698651319 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f4da400a9e0 con 0x7f4dac155ce0 2024-01-16T15:06:29.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.226+0000 7f4daa000640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4d8406bec0 0x7f4d8406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.227+0000 7f4daa000640 1 -- 172.21.15.141:0/698651319 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4da4087c70 con 0x7f4dac155ce0 2024-01-16T15:06:29.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.232+0000 7f4db1e00640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4d8406bec0 0x7f4d8406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.232+0000 7f4daa000640 1 -- 172.21.15.141:0/698651319 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4da4054330 con 0x7f4dac155ce0 2024-01-16T15:06:29.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.238+0000 7f4db1e00640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4d8406bec0 0x7f4d8406e380 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c034000 tx=0x7f4d9c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:29.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.619+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4dac0675d0 con 0x7f4dac155ce0 2024-01-16T15:06:29.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.620+0000 7f4daa000640 1 -- 172.21.15.141:0/698651319 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7f4da4057950 con 0x7f4dac155ce0 2024-01-16T15:06:29.600 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 582, 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:29.604 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 803, 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222388909, 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6872477696, 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766221635584, 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:29.605 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 852, 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 18421, 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 5 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:29.606 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:29.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0, 2024-01-16T15:06:29.610 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:29.611 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:29.611 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:29.611 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:29.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4d8406bec0 msgr2=0x7f4d8406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4d8406bec0 0x7f4d8406e380 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c034000 tx=0x7f4d9c008030 comp rx=0 tx=0).stop 2024-01-16T15:06:29.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dac155ce0 msgr2=0x7f4dac156160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dac155ce0 0x7f4dac156160 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f4dac072510 tx=0x7f4da400cb90 comp rx=0 tx=0).stop 2024-01-16T15:06:29.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 shutdown_connections 2024-01-16T15:06:29.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4d8406bec0 0x7f4d8406e380 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dac155ce0 0x7f4dac156160 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dac0c4e10 0x7f4dac0c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 --2- 172.21.15.141:0/698651319 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4d8c0039a0 0x7f4dac1f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.626+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 >> 172.21.15.141:0/698651319 conn(0x7f4dac0b2c70 msgr2=0x7f4dac15f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:29.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.627+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 shutdown_connections 2024-01-16T15:06:29.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:29.627+0000 7f4db4800640 1 -- 172.21.15.141:0/698651319 wait complete. 2024-01-16T15:06:30.644 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:30.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.885+0000 7fdc81800640 1 Processor -- start 2024-01-16T15:06:30.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.886+0000 7fdc81800640 1 -- start start 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.887+0000 7fdc81800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 0x7fdc7c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.887+0000 7fdc81800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdc7c0cf7d0 0x7fdc7c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.887+0000 7fdc81800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0cfc30 con 0x7fdc7c15e1e0 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.887+0000 7fdc81800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0b1ee0 con 0x7fdc7c0cf7d0 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.887+0000 7fdc81800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0b2780 con 0x7fdc7c143ae0 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdc7c15e1e0 0x7fdc7c15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34982/0 (socket says 172.21.15.141:34982) 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7b400640 1 -- 172.21.15.141:0/150482006 learned_addr learned my addr 172.21.15.141:0/150482006 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:30.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7aa00640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 0x7fdc7c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695632800 0 0) 0x7fdc7c0cfc30 con 0x7fdc7c15e1e0 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc5c003520 con 0x7fdc7c15e1e0 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc71e00640 1 --2- 172.21.15.141:0/150482006 >> v2:172.21.15.188:3300/0 conn(0x7fdc7c0cf7d0 0x7fdc7c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 330927329 0 0) 0x7fdc5c003520 con 0x7fdc7c15e1e0 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 msgr2=0x7fdc7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 0x7fdc7c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 >> v2:172.21.15.188:3300/0 conn(0x7fdc7c0cf7d0 msgr2=0x7fdc7c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 --2- 172.21.15.141:0/150482006 >> v2:172.21.15.188:3300/0 conn(0x7fdc7c0cf7d0 0x7fdc7c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c15e990 con 0x7fdc7c15e1e0 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fdc740019f0 con 0x7fdc7c15e1e0 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.888+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 >> v1:172.21.15.141:6789/0 conn(0x7fdc7c15e1e0 legacy=0x7fdc7c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7a000640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5c003970 0x7fdc5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7a000640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 0x7fdc5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7a000640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5c004d00 0x7fdc5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0b1ee0 con 0x7fdc5c004d00 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0b2780 con 0x7fdc5c003970 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c00ba00 con 0x7fdc7c143ae0 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7b400640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5c004d00 0x7fdc5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc71e00640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 0x7fdc5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7aa00640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5c003970 0x7fdc5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7b400640 1 -- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 msgr2=0x7fdc5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7b400640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc7c143ae0 0x7fdc5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7b400640 1 -- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5c003970 msgr2=0x7fdc5c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7b400640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5c003970 0x7fdc5c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7b400640 1 -- 172.21.15.141:0/150482006 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c003520 con 0x7fdc5c004d00 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.889+0000 7fdc7aa00640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5c003970 0x7fdc5c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.890+0000 7fdc7b400640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5c004d00 0x7fdc5c005180 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400a760 tx=0x7fdc5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7881de034a65470 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74034630 con 0x7fdc5c004d00 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7402f070 con 0x7fdc5c004d00 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc7a000640 1 -- 172.21.15.141:0/150482006 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc740349d0 con 0x7fdc5c004d00 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc81800640 1 -- 172.21.15.141:0/150482006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5c004d00 msgr2=0x7fdc5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc81800640 1 --2- 172.21.15.141:0/150482006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5c004d00 0x7fdc5c005180 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400a760 tx=0x7fdc5c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc7b400640 1 -- 172.21.15.141:0/150482006 reap_dead start 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc81800640 1 -- 172.21.15.141:0/150482006 shutdown_connections 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc81800640 1 -- 172.21.15.141:0/150482006 >> 172.21.15.141:0/150482006 conn(0x7fdc7c06a800 msgr2=0x7fdc7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:30.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc81800640 1 -- 172.21.15.141:0/150482006 shutdown_connections 2024-01-16T15:06:30.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.891+0000 7fdc81800640 1 -- 172.21.15.141:0/150482006 wait complete. 2024-01-16T15:06:30.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.893+0000 7fdc81800640 1 Processor -- start 2024-01-16T15:06:30.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.893+0000 7fdc81800640 1 -- start start 2024-01-16T15:06:30.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc81800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc5c004d00 0x7fdc7c154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc81800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7c15c6d0 0x7fdc7c15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc81800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7c070860 0x7fdc7c070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc81800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c15c510 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc81800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c15cdd0 con 0x7fdc7c070860 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc81800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c071220 con 0x7fdc5c004d00 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc71e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7c15c6d0 0x7fdc7c15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc71e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7c15c6d0 0x7fdc7c15bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59922/0 (socket says 172.21.15.141:59922) 2024-01-16T15:06:30.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.894+0000 7fdc71e00640 1 -- 172.21.15.141:0/3054583930 learned_addr learned my addr 172.21.15.141:0/3054583930 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc7aa00640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc5c004d00 0x7fdc7c154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc7b400640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7c070860 0x7fdc7c070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc71e00640 1 -- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc5c004d00 msgr2=0x7fdc7c154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc71e00640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc5c004d00 0x7fdc7c154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc71e00640 1 -- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7c070860 msgr2=0x7fdc7c070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc71e00640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7c070860 0x7fdc7c070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc71e00640 1 -- 172.21.15.141:0/3054583930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c15cb50 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc7b400640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7c070860 0x7fdc7c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc71e00640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7c15c6d0 0x7fdc7c15bfd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400f630 tx=0x7fdc6400faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc73400640 1 -- 172.21.15.141:0/3054583930 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64014870 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc73400640 1 -- 172.21.15.141:0/3054583930 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc640149d0 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c13dbb0 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.895+0000 7fdc73400640 1 -- 172.21.15.141:0/3054583930 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64014d40 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.896+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c13e170 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.897+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c0675d0 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.897+0000 7fdc73400640 1 -- 172.21.15.141:0/3054583930 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fdc64015410 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.898+0000 7fdc73400640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5406bec0 0x7fdc5406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.898+0000 7fdc73400640 1 -- 172.21.15.141:0/3054583930 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdc640891d0 con 0x7fdc7c15c6d0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.899+0000 7fdc7aa00640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5406bec0 0x7fdc5406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.900+0000 7fdc7aa00640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5406bec0 0x7fdc5406e380 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009aa0 tx=0x7fdc6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:30.903+0000 7fdc73400640 1 -- 172.21.15.141:0/3054583930 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc64055890 con 0x7fdc7c15c6d0 2024-01-16T15:06:31.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.164+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdc7c13a140 con 0x7fdc7c15c6d0 2024-01-16T15:06:31.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.165+0000 7fdc73400640 1 -- 172.21.15.141:0/3054583930 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fdc64058eb0 con 0x7fdc7c15c6d0 2024-01-16T15:06:31.145 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:31.145 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:31.145 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 583, 2024-01-16T15:06:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1293, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5277700097, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10362331136, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762731782144, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 177118974, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 171206812, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 49 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:31.157 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:31.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5406bec0 msgr2=0x7fdc5406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:31.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5406bec0 0x7fdc5406e380 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009aa0 tx=0x7fdc6c008040 comp rx=0 tx=0).stop 2024-01-16T15:06:31.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7c15c6d0 msgr2=0x7fdc7c15bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:31.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7c15c6d0 0x7fdc7c15bfd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400f630 tx=0x7fdc6400faf0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 shutdown_connections 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc5406bec0 0x7fdc5406e380 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc7c070860 0x7fdc7c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc7c15c6d0 0x7fdc7c15bfd0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 --2- 172.21.15.141:0/3054583930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc5c004d00 0x7fdc7c154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 >> 172.21.15.141:0/3054583930 conn(0x7fdc7c06a800 msgr2=0x7fdc7c139360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 shutdown_connections 2024-01-16T15:06:31.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:31.168+0000 7fdc81800640 1 -- 172.21.15.141:0/3054583930 wait complete. 2024-01-16T15:06:32.167 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:32.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.403+0000 7f2596000640 1 Processor -- start 2024-01-16T15:06:32.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.403+0000 7f2596000640 1 -- start start 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.404+0000 7f2596000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f25900b2600 0x7f25900b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.404+0000 7f2596000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900acda0 con 0x7f2590144410 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.404+0000 7f2596000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900b3240 con 0x7f259006aaf0 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.404+0000 7f2596000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25900af740 con 0x7f25900b2600 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.405+0000 7f2594c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2590144410 0x7f259006a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34998/0 (socket says 172.21.15.141:34998) 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.405+0000 7f2594c00640 1 -- 172.21.15.141:0/3365535007 learned_addr learned my addr 172.21.15.141:0/3365535007 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.405+0000 7f258fe00640 1 --2- 172.21.15.141:0/3365535007 >> v2:172.21.15.188:3300/0 conn(0x7f25900b2600 0x7f25900b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.405+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056786450 0 0) 0x7f25900acda0 con 0x7f2590144410 2024-01-16T15:06:32.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.405+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2570003520 con 0x7f2590144410 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1731288564 0 0) 0x7f2570003520 con 0x7f2590144410 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 >> v2:172.21.15.188:3300/0 conn(0x7f25900b2600 msgr2=0x7f25900b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 --2- 172.21.15.141:0/3365535007 >> v2:172.21.15.188:3300/0 conn(0x7f25900b2600 0x7f25900b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 >> v1:172.21.15.188:6789/0 conn(0x7f259006aaf0 legacy=0x7f25900b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25900acff0 con 0x7f2590144410 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2580001670 con 0x7f2590144410 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 >> v1:172.21.15.141:6789/0 conn(0x7f2590144410 legacy=0x7f259006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25700039c0 0x7f2570003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25700047c0 0x7f2570004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2570005180 0x7f257000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25900af740 con 0x7f2570005180 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257000c080 con 0x7f25700047c0 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257000c240 con 0x7f25700039c0 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f2595600640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2570005180 0x7f257000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.406+0000 7f2594c00640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25700039c0 0x7f2570003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.407+0000 7f2595600640 1 -- 172.21.15.141:0/3365535007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25700039c0 msgr2=0x7f2570003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.407+0000 7f2595600640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f25700039c0 0x7f2570003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.407+0000 7f2595600640 1 -- 172.21.15.141:0/3365535007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25700047c0 msgr2=0x7f2570004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.407+0000 7f2595600640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25700047c0 0x7f2570004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.407+0000 7f2595600640 1 -- 172.21.15.141:0/3365535007 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570003520 con 0x7f2570005180 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.407+0000 7f2595600640 1 -- 172.21.15.141:0/3365535007 reap_dead start 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.408+0000 7f2595600640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2570005180 0x7f257000b980 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f259006cfb0 tx=0x7f25700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443fbd255837e84c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.408+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2588003b70 con 0x7f2570005180 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.408+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2588003cd0 con 0x7f2570005180 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.408+0000 7f258f400640 1 -- 172.21.15.141:0/3365535007 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2588005310 con 0x7f2570005180 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.408+0000 7f2596000640 1 -- 172.21.15.141:0/3365535007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2570005180 msgr2=0x7f257000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.408+0000 7f2596000640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2570005180 0x7f257000b980 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f259006cfb0 tx=0x7f25700057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.409+0000 7f2596000640 1 -- 172.21.15.141:0/3365535007 shutdown_connections 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.409+0000 7f2596000640 1 --2- 172.21.15.141:0/3365535007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2570005180 0x7f257000b980 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.409+0000 7f2596000640 1 -- 172.21.15.141:0/3365535007 >> 172.21.15.141:0/3365535007 conn(0x7f25900b19c0 msgr2=0x7f25900b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:32.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.409+0000 7f2596000640 1 -- 172.21.15.141:0/3365535007 shutdown_connections 2024-01-16T15:06:32.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.409+0000 7f2596000640 1 -- 172.21.15.141:0/3365535007 wait complete. 2024-01-16T15:06:32.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.410+0000 7f2596000640 1 Processor -- start 2024-01-16T15:06:32.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.410+0000 7f2596000640 1 -- start start 2024-01-16T15:06:32.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.411+0000 7f2596000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2570005180 0x7f259015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.411+0000 7f2596000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259013e280 0x7f259013fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.411+0000 7f2596000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2590071160 0x7f25900715e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.411+0000 7f2596000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2590071b20 con 0x7f259013e280 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.411+0000 7f2596000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259013f370 con 0x7f2590071160 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.411+0000 7f2596000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259013e7f0 con 0x7f2570005180 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259013e280 0x7f259013fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259013e280 0x7f259013fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59944/0 (socket says 172.21.15.141:59944) 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 -- 172.21.15.141:0/2696454659 learned_addr learned my addr 172.21.15.141:0/2696454659 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f2594c00640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2570005180 0x7f259015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 -- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2570005180 msgr2=0x7f259015aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2570005180 0x7f259015aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 -- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2590071160 msgr2=0x7f25900715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f2595600640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2590071160 0x7f25900715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2590071160 0x7f25900715e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 -- 172.21.15.141:0/2696454659 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259013ead0 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.412+0000 7f258fe00640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259013e280 0x7f259013fa10 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f257800e7d0 tx=0x7f257800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.413+0000 7f258cc00640 1 -- 172.21.15.141:0/2696454659 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257800eee0 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.413+0000 7f258cc00640 1 -- 172.21.15.141:0/2696454659 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257800c940 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.413+0000 7f258cc00640 1 -- 172.21.15.141:0/2696454659 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257800ccd0 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.413+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259013edb0 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.413+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259013ff50 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.414+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2590067b60 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.415+0000 7f258cc00640 1 -- 172.21.15.141:0/2696454659 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2578013470 con 0x7f259013e280 2024-01-16T15:06:32.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.416+0000 7f258cc00640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806be90 0x7f256806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.416+0000 7f258cc00640 1 -- 172.21.15.141:0/2696454659 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f25780888f0 con 0x7f259013e280 2024-01-16T15:06:32.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.416+0000 7f2594c00640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806be90 0x7f256806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.417+0000 7f2594c00640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806be90 0x7f256806e350 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f25800016d0 tx=0x7f2580000f30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:32.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.420+0000 7f258cc00640 1 -- 172.21.15.141:0/2696454659 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2578054f50 con 0x7f259013e280 2024-01-16T15:06:32.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.679+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2590139cb0 con 0x7f259013e280 2024-01-16T15:06:32.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.680+0000 7f258cc00640 1 -- 172.21.15.141:0/2696454659 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f2578021990 con 0x7f259013e280 2024-01-16T15:06:32.659 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:32.659 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:32.660 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 585, 2024-01-16T15:06:32.661 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:32.662 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:32.663 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1293, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5277700097, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10362331136, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762731782144, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 177118974, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 171206812, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 49 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:32.668 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.682+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806be90 msgr2=0x7f256806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.682+0000 7f2596000640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806be90 0x7f256806e350 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f25800016d0 tx=0x7f2580000f30 comp rx=0 tx=0).stop 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259013e280 msgr2=0x7f259013fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259013e280 0x7f259013fa10 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f257800e7d0 tx=0x7f257800ec90 comp rx=0 tx=0).stop 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 shutdown_connections 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f256806be90 0x7f256806e350 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2590071160 0x7f25900715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f259013e280 0x7f259013fa10 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 --2- 172.21.15.141:0/2696454659 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2570005180 0x7f259015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 >> 172.21.15.141:0/2696454659 conn(0x7f25900b19c0 msgr2=0x7f25900733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:32.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 shutdown_connections 2024-01-16T15:06:32.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:32.683+0000 7f2596000640 1 -- 172.21.15.141:0/2696454659 wait complete. 2024-01-16T15:06:33.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:33.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.916+0000 7fb463800640 1 Processor -- start 2024-01-16T15:06:33.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.916+0000 7fb463800640 1 -- start start 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb463800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb45c13df80 0x7fb45c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb463800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb45c13e360 0x7fb45c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb463800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c06c470 con 0x7fb45c15f9c0 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb463800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45c06c750 con 0x7fb45c13df80 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb463800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45c06ca30 con 0x7fb45c13e360 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb460e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb45c13df80 0x7fb45c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb461800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb45c15f9c0 0x7fb45c1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35404/0 (socket says 172.21.15.141:35404) 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.918+0000 7fb461800640 1 -- 172.21.15.141:0/3096256154 learned_addr learned my addr 172.21.15.141:0/3096256154 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:33.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.919+0000 7fb460e00640 1 -- 172.21.15.141:0/3096256154 >> v2:172.21.15.188:3300/0 conn(0x7fb45c13e360 msgr2=0x7fb45c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:33.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.919+0000 7fb453c00640 1 --2- 172.21.15.141:0/3096256154 >> v2:172.21.15.188:3300/0 conn(0x7fb45c13e360 0x7fb45c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.919+0000 7fb460e00640 1 --2- 172.21.15.141:0/3096256154 >> v2:172.21.15.188:3300/0 conn(0x7fb45c13e360 0x7fb45c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.919+0000 7fb460e00640 1 -- 172.21.15.141:0/3096256154 >> v1:172.21.15.141:6789/0 conn(0x7fb45c15f9c0 legacy=0x7fb45c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.919+0000 7fb460e00640 1 -- 172.21.15.141:0/3096256154 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c06cd10 con 0x7fb45c13df80 2024-01-16T15:06:33.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.919+0000 7fb453c00640 1 --2- 172.21.15.141:0/3096256154 >> v2:172.21.15.188:3300/0 conn(0x7fb45c13e360 0x7fb45c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:33.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.919+0000 7fb45be00640 1 -- 172.21.15.141:0/3096256154 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892052244 0 0) 0x7fb45c06c470 con 0x7fb45c15f9c0 2024-01-16T15:06:33.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.920+0000 7fb460e00640 1 --2- 172.21.15.141:0/3096256154 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c13df80 0x7fb45c15bee0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb45c06ded0 tx=0x7fb44c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3001618e9552d33b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:33.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.920+0000 7fb45be00640 1 -- 172.21.15.141:0/3096256154 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c031830 con 0x7fb45c13df80 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.920+0000 7fb45be00640 1 -- 172.21.15.141:0/3096256154 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c031990 con 0x7fb45c13df80 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.920+0000 7fb45be00640 1 -- 172.21.15.141:0/3096256154 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c031d20 con 0x7fb45c13df80 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 -- 172.21.15.141:0/3096256154 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c13df80 msgr2=0x7fb45c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 --2- 172.21.15.141:0/3096256154 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c13df80 0x7fb45c15bee0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb45c06ded0 tx=0x7fb44c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 -- 172.21.15.141:0/3096256154 shutdown_connections 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 --2- 172.21.15.141:0/3096256154 >> v2:172.21.15.188:3300/0 conn(0x7fb45c13e360 0x7fb45c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 --2- 172.21.15.141:0/3096256154 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c13df80 0x7fb45c15bee0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 -- 172.21.15.141:0/3096256154 >> 172.21.15.141:0/3096256154 conn(0x7fb45c06a800 msgr2=0x7fb45c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:33.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 -- 172.21.15.141:0/3096256154 shutdown_connections 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.921+0000 7fb463800640 1 -- 172.21.15.141:0/3096256154 wait complete. 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.923+0000 7fb463800640 1 Processor -- start 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.923+0000 7fb463800640 1 -- start start 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb463800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb45c13df80 0x7fb45c14b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb463800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb45c13e360 0x7fb45c14d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb463800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c15f9c0 0x7fb45c159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb463800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c070510 con 0x7fb45c15f9c0 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb463800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c14bca0 con 0x7fb45c13e360 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb463800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c14d650 con 0x7fb45c13df80 2024-01-16T15:06:33.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c15f9c0 0x7fb45c159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c15f9c0 0x7fb45c159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51160/0 (socket says 172.21.15.141:51160) 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 -- 172.21.15.141:0/3285361958 learned_addr learned my addr 172.21.15.141:0/3285361958 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 -- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb45c13df80 msgr2=0x7fb45c14b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb45c13df80 0x7fb45c14b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 -- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb45c13e360 msgr2=0x7fb45c14d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb45c13e360 0x7fb45c14d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.924+0000 7fb461800640 1 -- 172.21.15.141:0/3285361958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c009400 con 0x7fb45c15f9c0 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.925+0000 7fb461800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c15f9c0 0x7fb45c159ca0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb454001560 tx=0x7fb45400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.925+0000 7fb459600640 1 -- 172.21.15.141:0/3285361958 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45400a640 con 0x7fb45c15f9c0 2024-01-16T15:06:33.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.925+0000 7fb459600640 1 -- 172.21.15.141:0/3285361958 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45400a7a0 con 0x7fb45c15f9c0 2024-01-16T15:06:33.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.925+0000 7fb459600640 1 -- 172.21.15.141:0/3285361958 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45400ab30 con 0x7fb45c15f9c0 2024-01-16T15:06:33.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.925+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c0707e0 con 0x7fb45c15f9c0 2024-01-16T15:06:33.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.925+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c14d8f0 con 0x7fb45c15f9c0 2024-01-16T15:06:33.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.926+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c070f80 con 0x7fb45c15f9c0 2024-01-16T15:06:33.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.927+0000 7fb459600640 1 -- 172.21.15.141:0/3285361958 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb45400ac90 con 0x7fb45c15f9c0 2024-01-16T15:06:33.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.928+0000 7fb459600640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb43406be90 0x7fb43406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:33.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.928+0000 7fb459600640 1 -- 172.21.15.141:0/3285361958 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb454087db0 con 0x7fb45c15f9c0 2024-01-16T15:06:33.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.929+0000 7fb460e00640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb43406be90 0x7fb43406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:33.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.930+0000 7fb460e00640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb43406be90 0x7fb43406e350 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb44c036000 tx=0x7fb44c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:33.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:33.932+0000 7fb459600640 1 -- 172.21.15.141:0/3285361958 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb45c070f80 con 0x7fb45c15f9c0 2024-01-16T15:06:34.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.191+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb45c070f80 con 0x7fb45c15f9c0 2024-01-16T15:06:34.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.192+0000 7fb459600640 1 -- 172.21.15.141:0/3285361958 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fb45c070f80 con 0x7fb45c15f9c0 2024-01-16T15:06:34.172 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:34.172 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:34.172 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:34.172 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:34.172 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:34.172 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:34.172 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:34.173 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 586, 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.174 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:34.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1293, 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5277700097, 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10362331136, 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762731782144, 2024-01-16T15:06:34.176 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 177109856, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 171184165, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 42, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 45 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:34.177 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:34.178 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:34.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:34.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:34.181 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:34.181 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:06:34.181 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:34.181 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:34.181 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:34.181 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:34.181 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:34.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb43406be90 msgr2=0x7fb43406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:34.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb43406be90 0x7fb43406e350 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb44c036000 tx=0x7fb44c030000 comp rx=0 tx=0).stop 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c15f9c0 msgr2=0x7fb45c159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c15f9c0 0x7fb45c159ca0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb454001560 tx=0x7fb45400cc70 comp rx=0 tx=0).stop 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 shutdown_connections 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb43406be90 0x7fb43406e350 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb45c15f9c0 0x7fb45c159ca0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb45c13e360 0x7fb45c14d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 --2- 172.21.15.141:0/3285361958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb45c13df80 0x7fb45c14b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 >> 172.21.15.141:0/3285361958 conn(0x7fb45c06a800 msgr2=0x7fb45c15e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 shutdown_connections 2024-01-16T15:06:34.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:34.195+0000 7fb463800640 1 -- 172.21.15.141:0/3285361958 wait complete. 2024-01-16T15:06:34.730 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:34.741 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:35.195 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:35.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.433+0000 7f818da00640 1 Processor -- start 2024-01-16T15:06:35.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.433+0000 7f818da00640 1 -- start start 2024-01-16T15:06:35.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.434+0000 7f818da00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8188143ae0 0x7f81880cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.434+0000 7f818da00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81880cfc30 con 0x7f818815e3b0 2024-01-16T15:06:35.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.434+0000 7f818da00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81880b1ee0 con 0x7f81880cf7d0 2024-01-16T15:06:35.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.434+0000 7f818da00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81880b2780 con 0x7f8188143ae0 2024-01-16T15:06:35.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.435+0000 7f8186a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8188143ae0 0x7f81880cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.435+0000 7f8187400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f818815e3b0 0x7f81880702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35414/0 (socket says 172.21.15.141:35414) 2024-01-16T15:06:35.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.435+0000 7f8187400640 1 -- 172.21.15.141:0/1715549678 learned_addr learned my addr 172.21.15.141:0/1715549678 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:35.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.435+0000 7f8186a00640 1 -- 172.21.15.141:0/1715549678 >> v1:172.21.15.188:6789/0 conn(0x7f81880cf7d0 legacy=0x7f818815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.435+0000 7f8186a00640 1 -- 172.21.15.141:0/1715549678 >> v1:172.21.15.141:6789/0 conn(0x7f818815e3b0 legacy=0x7f81880702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.435+0000 7f8186a00640 1 -- 172.21.15.141:0/1715549678 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818815e990 con 0x7f8188143ae0 2024-01-16T15:06:35.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.435+0000 7f8186000640 1 -- 172.21.15.141:0/1715549678 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753601892 0 0) 0x7f81880cfc30 con 0x7f818815e3b0 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.436+0000 7f8186a00640 1 --2- 172.21.15.141:0/1715549678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8188143ae0 0x7f81880cf290 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f81780097a0 tx=0x7f817802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4285a6fc9f16c552 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f8186000640 1 -- 172.21.15.141:0/1715549678 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178031830 con 0x7f8188143ae0 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f8186000640 1 -- 172.21.15.141:0/1715549678 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8178031990 con 0x7f8188143ae0 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f8186000640 1 -- 172.21.15.141:0/1715549678 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178031d20 con 0x7f8188143ae0 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f818da00640 1 -- 172.21.15.141:0/1715549678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8188143ae0 msgr2=0x7f81880cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f818da00640 1 --2- 172.21.15.141:0/1715549678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8188143ae0 0x7f81880cf290 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f81780097a0 tx=0x7f817802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f818da00640 1 -- 172.21.15.141:0/1715549678 shutdown_connections 2024-01-16T15:06:35.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f818da00640 1 --2- 172.21.15.141:0/1715549678 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8188143ae0 0x7f81880cf290 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f818da00640 1 -- 172.21.15.141:0/1715549678 >> 172.21.15.141:0/1715549678 conn(0x7f818806a800 msgr2=0x7f818806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f818da00640 1 -- 172.21.15.141:0/1715549678 shutdown_connections 2024-01-16T15:06:35.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.437+0000 7f818da00640 1 -- 172.21.15.141:0/1715549678 wait complete. 2024-01-16T15:06:35.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.439+0000 7f818da00640 1 Processor -- start 2024-01-16T15:06:35.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.439+0000 7f818da00640 1 -- start start 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.440+0000 7f818da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81880cf7d0 0x7f8188159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.440+0000 7f818da00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8188143ae0 0x7f81880cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.440+0000 7f818da00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f818815e3b0 0x7f81880ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.440+0000 7f818da00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8188141070 con 0x7f818815e3b0 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.440+0000 7f818da00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818815a140 con 0x7f81880cf7d0 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.440+0000 7f818da00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818813d000 con 0x7f8188143ae0 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f818815e3b0 0x7f81880ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f818815e3b0 0x7f81880ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51188/0 (socket says 172.21.15.141:51188) 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 -- 172.21.15.141:0/1975598078 learned_addr learned my addr 172.21.15.141:0/1975598078 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8186a00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81880cf7d0 0x7f8188159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f817de00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8188143ae0 0x7f81880cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 -- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8188143ae0 msgr2=0x7f81880cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8188143ae0 0x7f81880cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 -- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81880cf7d0 msgr2=0x7f8188159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81880cf7d0 0x7f8188159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 -- 172.21.15.141:0/1975598078 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8178009400 con 0x7f818815e3b0 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.441+0000 7f8187400640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f818815e3b0 0x7f81880ad920 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f818000c780 tx=0x7f818000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.442+0000 7f817f400640 1 -- 172.21.15.141:0/1975598078 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000ce90 con 0x7f818815e3b0 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.442+0000 7f817f400640 1 -- 172.21.15.141:0/1975598078 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818000a5e0 con 0x7f818815e3b0 2024-01-16T15:06:35.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.442+0000 7f817f400640 1 -- 172.21.15.141:0/1975598078 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000a970 con 0x7f818815e3b0 2024-01-16T15:06:35.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.442+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81881403f0 con 0x7f818815e3b0 2024-01-16T15:06:35.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.442+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81881409b0 con 0x7f818815e3b0 2024-01-16T15:06:35.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.443+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81881411b0 con 0x7f818815e3b0 2024-01-16T15:06:35.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.444+0000 7f817f400640 1 -- 172.21.15.141:0/1975598078 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f818000aad0 con 0x7f818815e3b0 2024-01-16T15:06:35.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.445+0000 7f817f400640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f816006be90 0x7f816006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.445+0000 7f817f400640 1 -- 172.21.15.141:0/1975598078 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8180087ba0 con 0x7f818815e3b0 2024-01-16T15:06:35.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.445+0000 7f8186a00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f816006be90 0x7f816006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.446+0000 7f8186a00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f816006be90 0x7f816006e350 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8178036000 tx=0x7f8178008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.449+0000 7f817f400640 1 -- 172.21.15.141:0/1975598078 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f81881411b0 con 0x7f818815e3b0 2024-01-16T15:06:35.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.717+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f81881411b0 con 0x7f818815e3b0 2024-01-16T15:06:35.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.717+0000 7f817f400640 1 -- 172.21.15.141:0/1975598078 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f81881411b0 con 0x7f818815e3b0 2024-01-16T15:06:35.697 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:35.697 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:35.697 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:35.697 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:35.698 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 588, 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:35.699 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:35.700 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:35.701 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1293, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5277700097, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10362331136, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762731782144, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 177113954, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 171188126, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 42, 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 45 2024-01-16T15:06:35.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:35.703 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:35.704 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:35.705 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:35.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:35.707 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:35.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f816006be90 msgr2=0x7f816006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f816006be90 0x7f816006e350 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8178036000 tx=0x7f8178008030 comp rx=0 tx=0).stop 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f818815e3b0 msgr2=0x7f81880ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f818815e3b0 0x7f81880ad920 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f818000c780 tx=0x7f818000cc40 comp rx=0 tx=0).stop 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 shutdown_connections 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f816006be90 0x7f816006e350 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f818815e3b0 0x7f81880ad920 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8188143ae0 0x7f81880cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 --2- 172.21.15.141:0/1975598078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81880cf7d0 0x7f8188159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.720+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 >> 172.21.15.141:0/1975598078 conn(0x7f818806a800 msgr2=0x7f818815dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.721+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 shutdown_connections 2024-01-16T15:06:35.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:35.721+0000 7f818da00640 1 -- 172.21.15.141:0/1975598078 wait complete. 2024-01-16T15:06:36.720 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:37.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.057+0000 7f180e400640 1 Processor -- start 2024-01-16T15:06:37.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.058+0000 7f180e400640 1 -- start start 2024-01-16T15:06:37.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.060+0000 7f180e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1808143ae0 0x7f18080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.060+0000 7f180e400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f180815e3b0 0x7f18080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.060+0000 7f180e400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080cfc30 con 0x7f18080cf7d0 2024-01-16T15:06:37.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.060+0000 7f180e400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18080b1ee0 con 0x7f180815e3b0 2024-01-16T15:06:37.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.060+0000 7f180e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18080b2780 con 0x7f1808143ae0 2024-01-16T15:06:37.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.061+0000 7f17ffe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f18080cf7d0 0x7f180815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35416/0 (socket says 172.21.15.141:35416) 2024-01-16T15:06:37.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.061+0000 7f17ffe00640 1 -- 172.21.15.141:0/1341525369 learned_addr learned my addr 172.21.15.141:0/1341525369 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:37.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.061+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551833971 0 0) 0x7f18080cfc30 con 0x7f18080cf7d0 2024-01-16T15:06:37.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.061+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e8003520 con 0x7f18080cf7d0 2024-01-16T15:06:37.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1807400640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1808143ae0 0x7f18080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106996611 0 0) 0x7f17e8003520 con 0x7f18080cf7d0 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1808143ae0 msgr2=0x7f18080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1808143ae0 0x7f18080cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 >> v2:172.21.15.188:3300/0 conn(0x7f180815e3b0 msgr2=0x7f18080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 --2- 172.21.15.141:0/1341525369 >> v2:172.21.15.188:3300/0 conn(0x7f180815e3b0 0x7f18080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180815e990 con 0x7f18080cf7d0 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f17f0001670 con 0x7f18080cf7d0 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 >> v1:172.21.15.141:6789/0 conn(0x7f18080cf7d0 legacy=0x7f180815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17e8003970 0x7f17e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1808143ae0 0x7f17e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17e8004d00 0x7f17e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18080b1ee0 con 0x7f17e8003970 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18080b2780 con 0x7f17e8004d00 2024-01-16T15:06:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.062+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e800dd70 con 0x7f1808143ae0 2024-01-16T15:06:37.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.063+0000 7f1807400640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17e8003970 0x7f17e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.063+0000 7f1807400640 1 -- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1808143ae0 msgr2=0x7f17e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.063+0000 7f1807400640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1808143ae0 0x7f17e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.063+0000 7f1807400640 1 -- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17e8004d00 msgr2=0x7f17e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.063+0000 7f1807400640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17e8004d00 0x7f17e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.063+0000 7f1807400640 1 -- 172.21.15.141:0/1341525369 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e8003520 con 0x7f17e8003970 2024-01-16T15:06:37.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.065+0000 7f1807400640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17e8003970 0x7f17e8003d90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f17f800ba60 tx=0x7f17e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf155887287bbedd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.065+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8032680 con 0x7f17e8003970 2024-01-16T15:06:37.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.065+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f80327e0 con 0x7f17e8003970 2024-01-16T15:06:37.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.065+0000 7f1806a00640 1 -- 172.21.15.141:0/1341525369 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8032bb0 con 0x7f17e8003970 2024-01-16T15:06:37.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.066+0000 7f180e400640 1 -- 172.21.15.141:0/1341525369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17e8003970 msgr2=0x7f17e8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.066+0000 7f180e400640 1 --2- 172.21.15.141:0/1341525369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17e8003970 0x7f17e8003d90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f17f800ba60 tx=0x7f17e80057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.066+0000 7f1807e00640 1 -- 172.21.15.141:0/1341525369 reap_dead start 2024-01-16T15:06:37.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.066+0000 7f180e400640 1 -- 172.21.15.141:0/1341525369 shutdown_connections 2024-01-16T15:06:37.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.066+0000 7f180e400640 1 -- 172.21.15.141:0/1341525369 >> 172.21.15.141:0/1341525369 conn(0x7f180806a800 msgr2=0x7f180806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:37.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.066+0000 7f180e400640 1 -- 172.21.15.141:0/1341525369 shutdown_connections 2024-01-16T15:06:37.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.067+0000 7f180e400640 1 -- 172.21.15.141:0/1341525369 wait complete. 2024-01-16T15:06:37.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.070+0000 7f180e400640 1 Processor -- start 2024-01-16T15:06:37.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.070+0000 7f180e400640 1 -- start start 2024-01-16T15:06:37.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.073+0000 7f180e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17e8003970 0x7f180813e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.073+0000 7f180e400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f180812b3c0 0x7f18080a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.073+0000 7f180e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18080a8ae0 0x7f180813f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.073+0000 7f180e400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180813e7f0 con 0x7f18080a8ae0 2024-01-16T15:06:37.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.073+0000 7f180e400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180812b8c0 con 0x7f180812b3c0 2024-01-16T15:06:37.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.073+0000 7f180e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180812bba0 con 0x7f17e8003970 2024-01-16T15:06:37.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.073+0000 7f1807e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18080a8ae0 0x7f180813f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f1807e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18080a8ae0 0x7f180813f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51208/0 (socket says 172.21.15.141:51208) 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f1807e00640 1 -- 172.21.15.141:0/354002364 learned_addr learned my addr 172.21.15.141:0/354002364 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f17ffe00640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f180812b3c0 0x7f18080a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f1807e00640 1 -- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17e8003970 msgr2=0x7f180813e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f1807e00640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17e8003970 0x7f180813e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f1807e00640 1 -- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f180812b3c0 msgr2=0x7f18080a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f1807e00640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f180812b3c0 0x7f18080a85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.074+0000 7f1807e00640 1 -- 172.21.15.141:0/354002364 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180812be80 con 0x7f18080a8ae0 2024-01-16T15:06:37.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.075+0000 7f1807e00640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18080a8ae0 0x7f180813f810 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f18080cc3c0 tx=0x7f180000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.076+0000 7f17ff400640 1 -- 172.21.15.141:0/354002364 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000aa20 con 0x7f18080a8ae0 2024-01-16T15:06:37.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.076+0000 7f17ff400640 1 -- 172.21.15.141:0/354002364 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000ab80 con 0x7f18080a8ae0 2024-01-16T15:06:37.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.076+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180812c160 con 0x7f18080a8ae0 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.076+0000 7f17ff400640 1 -- 172.21.15.141:0/354002364 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000cec0 con 0x7f18080a8ae0 2024-01-16T15:06:37.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.076+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180813fd50 con 0x7f18080a8ae0 2024-01-16T15:06:37.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.081+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1808072610 con 0x7f18080a8ae0 2024-01-16T15:06:37.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.082+0000 7f17ff400640 1 -- 172.21.15.141:0/354002364 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f180001b3b0 con 0x7f18080a8ae0 2024-01-16T15:06:37.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.084+0000 7f17ff400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f17e006bf10 0x7f17e006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.084+0000 7f17ff400640 1 -- 172.21.15.141:0/354002364 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1800088220 con 0x7f18080a8ae0 2024-01-16T15:06:37.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.090+0000 7f17ff400640 1 -- 172.21.15.141:0/354002364 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1800091080 con 0x7f18080a8ae0 2024-01-16T15:06:37.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.104+0000 7f1807400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f17e006bf10 0x7f17e006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.112+0000 7f1807400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f17e006bf10 0x7f17e006e3d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f17f800ba30 tx=0x7f17f8032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.450+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f180812c680 con 0x7f18080a8ae0 2024-01-16T15:06:37.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.451+0000 7f17ff400640 1 -- 172.21.15.141:0/354002364 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f1800017990 con 0x7f18080a8ae0 2024-01-16T15:06:37.430 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:37.431 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 590, 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:37.432 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:37.433 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:37.434 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1293, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5277700097, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 10362331136, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 762731782144, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 177113954, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 171188126, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 42, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 45 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:37.435 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:37.436 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:37.437 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:37.438 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:37.439 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f17e006bf10 msgr2=0x7f17e006e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f17e006bf10 0x7f17e006e3d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f17f800ba30 tx=0x7f17f8032000 comp rx=0 tx=0).stop 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18080a8ae0 msgr2=0x7f180813f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18080a8ae0 0x7f180813f810 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f18080cc3c0 tx=0x7f180000ccd0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 shutdown_connections 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f17e006bf10 0x7f17e006e3d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18080a8ae0 0x7f180813f810 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f180812b3c0 0x7f18080a85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 --2- 172.21.15.141:0/354002364 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17e8003970 0x7f180813e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 >> 172.21.15.141:0/354002364 conn(0x7f180806a800 msgr2=0x7f18080702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:37.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.454+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 shutdown_connections 2024-01-16T15:06:37.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:37.455+0000 7f180e400640 1 -- 172.21.15.141:0/354002364 wait complete. 2024-01-16T15:06:38.463 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:38.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.757+0000 7fa04b200640 1 Processor -- start 2024-01-16T15:06:38.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.757+0000 7fa04b200640 1 -- start start 2024-01-16T15:06:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.760+0000 7fa04b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 0x7fa04c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.760+0000 7fa04b200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa04c0cfe70 0x7fa04c161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.761+0000 7fa04b200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c0d0a10 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.761+0000 7fa04b200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c0d1f50 con 0x7fa04c0cfe70 2024-01-16T15:06:38.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.761+0000 7fa04b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0cf080 con 0x7fa04c160920 2024-01-16T15:06:38.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.761+0000 7fa049400640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa04c0cf9f0 0x7fa04c161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:45612/0 (socket says 172.21.15.141:45612) 2024-01-16T15:06:38.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.761+0000 7fa049400640 1 -- 172.21.15.141:0/4028483361 learned_addr learned my addr 172.21.15.141:0/4028483361 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:38.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.761+0000 7fa04a800640 1 --2- 172.21.15.141:0/4028483361 >> v2:172.21.15.188:3300/0 conn(0x7fa04c0cfe70 0x7fa04c161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.763+0000 7fa04a800640 1 -- 172.21.15.141:0/4028483361 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 msgr2=0x7fa04c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:38.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.763+0000 7fa04a800640 1 --2- 172.21.15.141:0/4028483361 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 0x7fa04c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.763+0000 7fa04a800640 1 -- 172.21.15.141:0/4028483361 >> v1:172.21.15.188:6789/0 conn(0x7fa04c0cf9f0 legacy=0x7fa04c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.763+0000 7fa04a800640 1 -- 172.21.15.141:0/4028483361 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c0c6be0 con 0x7fa04c0cfe70 2024-01-16T15:06:38.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.764+0000 7fa04a800640 1 --2- 172.21.15.141:0/4028483361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c161720 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa044009d90 tx=0x7fa04402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2542dfbbfdd289dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.765+0000 7fa048a00640 1 -- 172.21.15.141:0/4028483361 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04402cc50 con 0x7fa04c0cfe70 2024-01-16T15:06:38.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.765+0000 7fa048a00640 1 -- 172.21.15.141:0/4028483361 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04402cdb0 con 0x7fa04c0cfe70 2024-01-16T15:06:38.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.765+0000 7fa048a00640 1 -- 172.21.15.141:0/4028483361 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044034660 con 0x7fa04c0cfe70 2024-01-16T15:06:38.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 -- 172.21.15.141:0/4028483361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 msgr2=0x7fa04c161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 --2- 172.21.15.141:0/4028483361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c161720 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa044009d90 tx=0x7fa04402bea0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 -- 172.21.15.141:0/4028483361 shutdown_connections 2024-01-16T15:06:38.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 --2- 172.21.15.141:0/4028483361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c161720 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 --2- 172.21.15.141:0/4028483361 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 0x7fa04c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 -- 172.21.15.141:0/4028483361 >> 172.21.15.141:0/4028483361 conn(0x7fa04c0d3090 msgr2=0x7fa04c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:38.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 -- 172.21.15.141:0/4028483361 shutdown_connections 2024-01-16T15:06:38.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.768+0000 7fa04b200640 1 -- 172.21.15.141:0/4028483361 wait complete. 2024-01-16T15:06:38.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.771+0000 7fa04b200640 1 Processor -- start 2024-01-16T15:06:38.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.771+0000 7fa04b200640 1 -- start start 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.772+0000 7fa04b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa04c0cf9f0 0x7fa04c1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.772+0000 7fa04b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.772+0000 7fa04b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 0x7fa04c0b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.772+0000 7fa04b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0ca880 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.772+0000 7fa04b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0ca200 con 0x7fa04c0cfe70 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.772+0000 7fa04b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0c8bb0 con 0x7fa04c160920 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa04c0cf9f0 0x7fa04c1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa04c0cf9f0 0x7fa04c1582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51210/0 (socket says 172.21.15.141:51210) 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049e00640 1 -- 172.21.15.141:0/3557302590 learned_addr learned my addr 172.21.15.141:0/3557302590 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa04a800640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 0x7fa04c0b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049400640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c0b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049e00640 1 -- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 msgr2=0x7fa04c0b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049e00640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 0x7fa04c0b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049e00640 1 -- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 msgr2=0x7fa04c0b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.773+0000 7fa049e00640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c0b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.774+0000 7fa049e00640 1 -- 172.21.15.141:0/3557302590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044009a70 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.774+0000 7fa049400640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:38.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.774+0000 7fa049e00640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa04c0cf9f0 0x7fa04c1582a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b4a0 tx=0x7fa03c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.774+0000 7fa042000640 1 -- 172.21.15.141:0/3557302590 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c010610 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.774+0000 7fa042000640 1 -- 172.21.15.141:0/3557302590 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c010770 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.774+0000 7fa042000640 1 -- 172.21.15.141:0/3557302590 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c010b20 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.775+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c0ca4a0 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.775+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c1587e0 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.777+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c005250 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.777+0000 7fa042000640 1 -- 172.21.15.141:0/3557302590 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fa03c010c80 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.778+0000 7fa042000640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa02406be90 0x7fa02406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.779+0000 7fa042000640 1 -- 172.21.15.141:0/3557302590 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa03c0883b0 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.784+0000 7fa042000640 1 -- 172.21.15.141:0/3557302590 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa03c054a90 con 0x7fa04c0cf9f0 2024-01-16T15:06:38.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.787+0000 7fa049400640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa02406be90 0x7fa02406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:38.789+0000 7fa049400640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa02406be90 0x7fa02406e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0b86e0 tx=0x7fa034009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:39.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.180+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa01c0057d0 con 0x7fa04c0cf9f0 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.181+0000 7fa042000640 1 -- 172.21.15.141:0/3557302590 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fa03c01e990 con 0x7fa04c0cf9f0 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 591, 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1315, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 5369976790, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 11167653888, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 761926459392, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 178812671, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 178822050, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 45 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:39.166 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.183+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa02406be90 msgr2=0x7fa02406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.183+0000 7fa04b200640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa02406be90 0x7fa02406e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0b86e0 tx=0x7fa034009040 comp rx=0 tx=0).stop 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.183+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa04c0cf9f0 msgr2=0x7fa04c1582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.183+0000 7fa04b200640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa04c0cf9f0 0x7fa04c1582a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b4a0 tx=0x7fa03c00b960 comp rx=0 tx=0).stop 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 shutdown_connections 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa02406be90 0x7fa02406e350 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa04c160920 0x7fa04c0b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa04c0cfe70 0x7fa04c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 --2- 172.21.15.141:0/3557302590 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa04c0cf9f0 0x7fa04c1582a0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 >> 172.21.15.141:0/3557302590 conn(0x7fa04c0d3090 msgr2=0x7fa04c0cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:39.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 shutdown_connections 2024-01-16T15:06:39.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:39.184+0000 7fa04b200640 1 -- 172.21.15.141:0/3557302590 wait complete. 2024-01-16T15:06:40.192 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:40.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.472+0000 7f034c400640 1 Processor -- start 2024-01-16T15:06:40.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.472+0000 7f034c400640 1 -- start start 2024-01-16T15:06:40.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.474+0000 7f034c400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f034415d460 0x7f03440ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.474+0000 7f034c400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f03440b30b0 0x7f034406d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.474+0000 7f034c400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034415d840 con 0x7f034406e380 2024-01-16T15:06:40.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.474+0000 7f034c400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0344071490 con 0x7f03440b30b0 2024-01-16T15:06:40.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.474+0000 7f034c400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03440b3790 con 0x7f034415d460 2024-01-16T15:06:40.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.474+0000 7f0349000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f034406e380 0x7f03440b29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35426/0 (socket says 172.21.15.141:35426) 2024-01-16T15:06:40.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.474+0000 7f0349000640 1 -- 172.21.15.141:0/2883858229 learned_addr learned my addr 172.21.15.141:0/2883858229 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:40.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.475+0000 7f0349a00640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f034415d460 0x7f03440ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.475+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442600134 0 0) 0x7f034415d840 con 0x7f034406e380 2024-01-16T15:06:40.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.475+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0324003520 con 0x7f034406e380 2024-01-16T15:06:40.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3283340285 0 0) 0x7f0324003520 con 0x7f034406e380 2024-01-16T15:06:40.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f034415d460 msgr2=0x7f03440ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f034415d460 0x7f03440ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 >> v2:172.21.15.188:3300/0 conn(0x7f03440b30b0 msgr2=0x7f034406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 --2- 172.21.15.141:0/2883858229 >> v2:172.21.15.188:3300/0 conn(0x7f03440b30b0 0x7f034406d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0344072190 con 0x7f034406e380 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f032c001670 con 0x7f034406e380 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 >> v1:172.21.15.141:6789/0 conn(0x7f034406e380 legacy=0x7f03440b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0324003970 0x7f0324003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f034415d460 0x7f03240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0324004d00 0x7f0324005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0344071490 con 0x7f0324004d00 2024-01-16T15:06:40.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032400bb10 con 0x7f0324003970 2024-01-16T15:06:40.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.476+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032400bca0 con 0x7f034415d460 2024-01-16T15:06:40.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.477+0000 7f034a400640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0324004d00 0x7f0324005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.477+0000 7f034a400640 1 -- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f034415d460 msgr2=0x7f03240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.477+0000 7f034a400640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f034415d460 0x7f03240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.477+0000 7f034a400640 1 -- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0324003970 msgr2=0x7f0324003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.477+0000 7f034a400640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0324003970 0x7f0324003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.477+0000 7f034a400640 1 -- 172.21.15.141:0/2883858229 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324003520 con 0x7f0324004d00 2024-01-16T15:06:40.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.485+0000 7f034a400640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0324004d00 0x7f0324005180 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f033c009e10 tx=0x7f03240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e522e1159546cde3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.486+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c02cd10 con 0x7f0324004d00 2024-01-16T15:06:40.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.486+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033c02ce70 con 0x7f0324004d00 2024-01-16T15:06:40.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.487+0000 7f0343e00640 1 -- 172.21.15.141:0/2883858229 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c02cd10 con 0x7f0324004d00 2024-01-16T15:06:40.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.488+0000 7f034c400640 1 -- 172.21.15.141:0/2883858229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0324004d00 msgr2=0x7f0324005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.488+0000 7f034c400640 1 --2- 172.21.15.141:0/2883858229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0324004d00 0x7f0324005180 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f033c009e10 tx=0x7f03240057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.488+0000 7f034a400640 1 -- 172.21.15.141:0/2883858229 reap_dead start 2024-01-16T15:06:40.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.489+0000 7f034c400640 1 -- 172.21.15.141:0/2883858229 shutdown_connections 2024-01-16T15:06:40.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.489+0000 7f034c400640 1 -- 172.21.15.141:0/2883858229 >> 172.21.15.141:0/2883858229 conn(0x7f03440b0be0 msgr2=0x7f03440b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.490+0000 7f034c400640 1 -- 172.21.15.141:0/2883858229 shutdown_connections 2024-01-16T15:06:40.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.490+0000 7f034c400640 1 -- 172.21.15.141:0/2883858229 wait complete. 2024-01-16T15:06:40.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.495+0000 7f034c400640 1 Processor -- start 2024-01-16T15:06:40.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.495+0000 7f034c400640 1 -- start start 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f034c400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f034415c280 0x7f0344155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f034c400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0344150d10 0x7f03440cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f034c400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03440cdfd0 0x7f03441fafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f034c400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03440ce480 con 0x7f034415c280 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f034c400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034415c660 con 0x7f0344150d10 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f034c400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0344153120 con 0x7f03440cdfd0 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f0349a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f034415c280 0x7f0344155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f0349a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f034415c280 0x7f0344155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51228/0 (socket says 172.21.15.141:51228) 2024-01-16T15:06:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.497+0000 7f0349a00640 1 -- 172.21.15.141:0/2049264378 learned_addr learned my addr 172.21.15.141:0/2049264378 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.498+0000 7f0349000640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0344150d10 0x7f03440cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.498+0000 7f0349a00640 1 -- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03440cdfd0 msgr2=0x7f03441fafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.498+0000 7f0349a00640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03440cdfd0 0x7f03441fafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.498+0000 7f0349a00640 1 -- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0344150d10 msgr2=0x7f03440cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.498+0000 7f0349a00640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0344150d10 0x7f03440cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.498+0000 7f0349a00640 1 -- 172.21.15.141:0/2049264378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03441614f0 con 0x7f034415c280 2024-01-16T15:06:40.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.498+0000 7f0349a00640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f034415c280 0x7f0344155380 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f033400d830 tx=0x7f033400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.499+0000 7f0341600640 1 -- 172.21.15.141:0/2049264378 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033400b6c0 con 0x7f034415c280 2024-01-16T15:06:40.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.499+0000 7f0341600640 1 -- 172.21.15.141:0/2049264378 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033400b820 con 0x7f034415c280 2024-01-16T15:06:40.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.499+0000 7f0341600640 1 -- 172.21.15.141:0/2049264378 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033400bbd0 con 0x7f034415c280 2024-01-16T15:06:40.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.499+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03441617d0 con 0x7f034415c280 2024-01-16T15:06:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.500+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0344151410 con 0x7f034415c280 2024-01-16T15:06:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.501+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03441619c0 con 0x7f034415c280 2024-01-16T15:06:40.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.508+0000 7f0341600640 1 -- 172.21.15.141:0/2049264378 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0334014410 con 0x7f034415c280 2024-01-16T15:06:40.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.509+0000 7f0341600640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031c06bf90 0x7f031c06e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.509+0000 7f0341600640 1 -- 172.21.15.141:0/2049264378 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0334088b60 con 0x7f034415c280 2024-01-16T15:06:40.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.515+0000 7f0341600640 1 -- 172.21.15.141:0/2049264378 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03340551c0 con 0x7f034415c280 2024-01-16T15:06:40.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.521+0000 7f0349000640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031c06bf90 0x7f031c06e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.526+0000 7f0349000640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031c06bf90 0x7f031c06e450 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f032c0017a0 tx=0x7f032c0015d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.881+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f03440675d0 con 0x7f034415c280 2024-01-16T15:06:40.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.881+0000 7f0341600640 1 -- 172.21.15.141:0/2049264378 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f0334018990 con 0x7f034415c280 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.861 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 593, 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:40.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:40.863 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1827, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7517550013, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 15018262528, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 758075850752, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 358313830, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 357632757, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 85, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 91 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:40.864 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.865 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:40.866 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.19045713543891907, 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:40.867 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.885+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031c06bf90 msgr2=0x7f031c06e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.885+0000 7f034c400640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031c06bf90 0x7f031c06e450 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f032c0017a0 tx=0x7f032c0015d0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f034415c280 msgr2=0x7f0344155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f034415c280 0x7f0344155380 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f033400d830 tx=0x7f033400dcf0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 shutdown_connections 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f031c06bf90 0x7f031c06e450 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03440cdfd0 0x7f03441fafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0344150d10 0x7f03440cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 --2- 172.21.15.141:0/2049264378 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f034415c280 0x7f0344155380 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.886+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 >> 172.21.15.141:0/2049264378 conn(0x7f03440b0be0 msgr2=0x7f03440acdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.890+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 shutdown_connections 2024-01-16T15:06:40.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:40.893+0000 7f034c400640 1 -- 172.21.15.141:0/2049264378 wait complete. 2024-01-16T15:06:41.901 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:42.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.163+0000 7f93d0400640 1 Processor -- start 2024-01-16T15:06:42.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.164+0000 7f93d0400640 1 -- start start 2024-01-16T15:06:42.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93d0400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f93c8143ae0 0x7f93c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93d0400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f93c80cf7d0 0x7f93c815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93d0400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c80cfc30 con 0x7f93c815e1e0 2024-01-16T15:06:42.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93d0400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c80b1ee0 con 0x7f93c80cf7d0 2024-01-16T15:06:42.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93d0400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c80b2780 con 0x7f93c8143ae0 2024-01-16T15:06:42.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93cd000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f93c80cf7d0 0x7f93c815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93cd000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f93c80cf7d0 0x7f93c815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51240/0 (socket says 172.21.15.141:51240) 2024-01-16T15:06:42.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.165+0000 7f93cd000640 1 -- 172.21.15.141:0/1353414618 learned_addr learned my addr 172.21.15.141:0/1353414618 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:42.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.166+0000 7f93cda00640 1 --2- 172.21.15.141:0/1353414618 >> v2:172.21.15.188:3300/0 conn(0x7f93c8143ae0 0x7f93c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.166+0000 7f93cd000640 1 -- 172.21.15.141:0/1353414618 >> v2:172.21.15.188:3300/0 conn(0x7f93c8143ae0 msgr2=0x7f93c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:42.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.166+0000 7f93cd000640 1 --2- 172.21.15.141:0/1353414618 >> v2:172.21.15.188:3300/0 conn(0x7f93c8143ae0 0x7f93c80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.166+0000 7f93cd000640 1 -- 172.21.15.141:0/1353414618 >> v1:172.21.15.141:6789/0 conn(0x7f93c815e1e0 legacy=0x7f93c815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.166+0000 7f93cd000640 1 -- 172.21.15.141:0/1353414618 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c815e990 con 0x7f93c80cf7d0 2024-01-16T15:06:42.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.167+0000 7f93cd000640 1 --2- 172.21.15.141:0/1353414618 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c815dca0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f93c80cfea0 tx=0x7f93b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b03681c4d817c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.167+0000 7f93c7e00640 1 -- 172.21.15.141:0/1353414618 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0030860 con 0x7f93c80cf7d0 2024-01-16T15:06:42.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.168+0000 7f93c7e00640 1 -- 172.21.15.141:0/1353414618 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b00309c0 con 0x7f93c80cf7d0 2024-01-16T15:06:42.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.168+0000 7f93c7e00640 1 -- 172.21.15.141:0/1353414618 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0030d50 con 0x7f93c80cf7d0 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.168+0000 7f93d0400640 1 -- 172.21.15.141:0/1353414618 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 msgr2=0x7f93c815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.168+0000 7f93d0400640 1 --2- 172.21.15.141:0/1353414618 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c815dca0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f93c80cfea0 tx=0x7f93b002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.169+0000 7f93d0400640 1 -- 172.21.15.141:0/1353414618 shutdown_connections 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.169+0000 7f93d0400640 1 --2- 172.21.15.141:0/1353414618 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c815dca0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.169+0000 7f93d0400640 1 --2- 172.21.15.141:0/1353414618 >> v2:172.21.15.188:3300/0 conn(0x7f93c8143ae0 0x7f93c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.169+0000 7f93d0400640 1 -- 172.21.15.141:0/1353414618 >> 172.21.15.141:0/1353414618 conn(0x7f93c806a800 msgr2=0x7f93c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.169+0000 7f93d0400640 1 -- 172.21.15.141:0/1353414618 shutdown_connections 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.169+0000 7f93d0400640 1 -- 172.21.15.141:0/1353414618 wait complete. 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.172+0000 7f93d0400640 1 Processor -- start 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.173+0000 7f93d0400640 1 -- start start 2024-01-16T15:06:42.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.174+0000 7f93d0400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c813dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.174+0000 7f93d0400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93c8143ae0 0x7f93c813ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.174+0000 7f93d0400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93c815e1e0 0x7f93c813ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.174+0000 7f93d0400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c8140940 con 0x7f93c80cf7d0 2024-01-16T15:06:42.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.174+0000 7f93d0400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c81405d0 con 0x7f93c8143ae0 2024-01-16T15:06:42.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.174+0000 7f93d0400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c8140b90 con 0x7f93c815e1e0 2024-01-16T15:06:42.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.175+0000 7f93cd000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93c8143ae0 0x7f93c813ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.175+0000 7f93cda00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c813dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.176+0000 7f93cd000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93c8143ae0 0x7f93c813ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:46846/0 (socket says 172.21.15.141:46846) 2024-01-16T15:06:42.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.176+0000 7f93cd000640 1 -- 172.21.15.141:0/398786162 learned_addr learned my addr 172.21.15.141:0/398786162 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:42.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.177+0000 7f93cda00640 1 -- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93c815e1e0 msgr2=0x7f93c813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.177+0000 7f93cda00640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93c815e1e0 0x7f93c813ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.177+0000 7f93cda00640 1 -- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93c8143ae0 msgr2=0x7f93c813ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.177+0000 7f93cda00640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93c8143ae0 0x7f93c813ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.177+0000 7f93cda00640 1 -- 172.21.15.141:0/398786162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00094d0 con 0x7f93c80cf7d0 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.177+0000 7f93cda00640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c813dd40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f93b800d620 tx=0x7f93b800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.178+0000 7f93c5600640 1 -- 172.21.15.141:0/398786162 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b800b620 con 0x7f93c80cf7d0 2024-01-16T15:06:42.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.178+0000 7f93c5600640 1 -- 172.21.15.141:0/398786162 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b800b780 con 0x7f93c80cf7d0 2024-01-16T15:06:42.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.178+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c8140e70 con 0x7f93c80cf7d0 2024-01-16T15:06:42.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.178+0000 7f93c5600640 1 -- 172.21.15.141:0/398786162 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b800bb10 con 0x7f93c80cf7d0 2024-01-16T15:06:42.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.180+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c8141430 con 0x7f93c80cf7d0 2024-01-16T15:06:42.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.182+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c80675d0 con 0x7f93c80cf7d0 2024-01-16T15:06:42.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.185+0000 7f93c5600640 1 -- 172.21.15.141:0/398786162 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f93b800bc70 con 0x7f93c80cf7d0 2024-01-16T15:06:42.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.186+0000 7f93c5600640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f93a006bee0 0x7f93a006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.186+0000 7f93c5600640 1 -- 172.21.15.141:0/398786162 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f93b8088c90 con 0x7f93c80cf7d0 2024-01-16T15:06:42.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.188+0000 7f93cd000640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f93a006bee0 0x7f93a006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.190+0000 7f93cd000640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f93a006bee0 0x7f93a006e3a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f93c80adfb0 tx=0x7f93b003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.193+0000 7f93c5600640 1 -- 172.21.15.141:0/398786162 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93b8055330 con 0x7f93c80cf7d0 2024-01-16T15:06:42.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.633+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f93c80ae7e0 con 0x7f93c80cf7d0 2024-01-16T15:06:42.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.634+0000 7f93c5600640 1 -- 172.21.15.141:0/398786162 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f93b8058950 con 0x7f93c80cf7d0 2024-01-16T15:06:42.613 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:42.613 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:42.613 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:42.613 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:42.613 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 595, 2024-01-16T15:06:42.614 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:42.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1827, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7517550013, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 15018262528, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 758075850752, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 358313830, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 357632757, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 85, 2024-01-16T15:06:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 91 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:42.617 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:42.618 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.19045713543891907, 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:42.619 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:42.620 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:42.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f93a006bee0 msgr2=0x7f93a006e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f93a006bee0 0x7f93a006e3a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f93c80adfb0 tx=0x7f93b003b040 comp rx=0 tx=0).stop 2024-01-16T15:06:42.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 msgr2=0x7f93c813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c813dd40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f93b800d620 tx=0x7f93b800dae0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 shutdown_connections 2024-01-16T15:06:42.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f93a006bee0 0x7f93a006e3a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93c815e1e0 0x7f93c813ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93c8143ae0 0x7f93c813ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 --2- 172.21.15.141:0/398786162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93c80cf7d0 0x7f93c813dd40 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.637+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 >> 172.21.15.141:0/398786162 conn(0x7f93c806a800 msgr2=0x7f93c806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:42.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.638+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 shutdown_connections 2024-01-16T15:06:42.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:42.638+0000 7f93d0400640 1 -- 172.21.15.141:0/398786162 wait complete. 2024-01-16T15:06:43.642 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:43.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.880+0000 7fa3d9600640 1 Processor -- start 2024-01-16T15:06:43.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.880+0000 7fa3d9600640 1 -- start start 2024-01-16T15:06:43.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.881+0000 7fa3d9600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa3d415afb0 0x7fa3d413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.881+0000 7fa3d9600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d413ff30 con 0x7fa3d413cfa0 2024-01-16T15:06:43.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.881+0000 7fa3d9600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d40cdc40 con 0x7fa3d413fae0 2024-01-16T15:06:43.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d9600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d40cdec0 con 0x7fa3d415afb0 2024-01-16T15:06:43.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d3400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa3d415afb0 0x7fa3d413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa3d413cfa0 0x7fa3d40cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55796/0 (socket says 172.21.15.141:55796) 2024-01-16T15:06:43.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d3e00640 1 -- 172.21.15.141:0/3633228100 learned_addr learned my addr 172.21.15.141:0/3633228100 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:43.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d3400640 1 -- 172.21.15.141:0/3633228100 >> v1:172.21.15.188:6789/0 conn(0x7fa3d413fae0 legacy=0x7fa3d40b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:43.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d3400640 1 -- 172.21.15.141:0/3633228100 >> v1:172.21.15.141:6789/0 conn(0x7fa3d413cfa0 legacy=0x7fa3d40cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d3400640 1 -- 172.21.15.141:0/3633228100 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d40ce1a0 con 0x7fa3d415afb0 2024-01-16T15:06:43.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.882+0000 7fa3d2a00640 1 -- 172.21.15.141:0/3633228100 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825348304 0 0) 0x7fa3c4001110 con 0x7fa3d413cfa0 2024-01-16T15:06:43.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.883+0000 7fa3d3400640 1 --2- 172.21.15.141:0/3633228100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d415afb0 0x7fa3d413f5a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0097a0 tx=0x7fa3bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3295633512deda server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d2a00640 1 -- 172.21.15.141:0/3633228100 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc02e070 con 0x7fa3d415afb0 2024-01-16T15:06:43.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d2a00640 1 -- 172.21.15.141:0/3633228100 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc0315a0 con 0x7fa3d415afb0 2024-01-16T15:06:43.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d2a00640 1 -- 172.21.15.141:0/3633228100 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc031930 con 0x7fa3d415afb0 2024-01-16T15:06:43.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d9600640 1 -- 172.21.15.141:0/3633228100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d415afb0 msgr2=0x7fa3d413f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d9600640 1 --2- 172.21.15.141:0/3633228100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d415afb0 0x7fa3d413f5a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0097a0 tx=0x7fa3bc02b840 comp rx=0 tx=0).stop 2024-01-16T15:06:43.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d9600640 1 -- 172.21.15.141:0/3633228100 shutdown_connections 2024-01-16T15:06:43.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d9600640 1 --2- 172.21.15.141:0/3633228100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d415afb0 0x7fa3d413f5a0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d9600640 1 -- 172.21.15.141:0/3633228100 >> 172.21.15.141:0/3633228100 conn(0x7fa3d406a800 msgr2=0x7fa3d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:43.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d9600640 1 -- 172.21.15.141:0/3633228100 shutdown_connections 2024-01-16T15:06:43.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.884+0000 7fa3d9600640 1 -- 172.21.15.141:0/3633228100 wait complete. 2024-01-16T15:06:43.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.886+0000 7fa3d9600640 1 Processor -- start 2024-01-16T15:06:43.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.886+0000 7fa3d9600640 1 -- start start 2024-01-16T15:06:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.887+0000 7fa3d9600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3d413cfa0 0x7fa3d413b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.887+0000 7fa3d9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d413fae0 0x7fa3d4152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.887+0000 7fa3d9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3d415afb0 0x7fa3d415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d9600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d413a2e0 con 0x7fa3d413fae0 2024-01-16T15:06:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d9600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d41529a0 con 0x7fa3d415afb0 2024-01-16T15:06:43.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d9600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d413a4c0 con 0x7fa3d413cfa0 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d413fae0 0x7fa3d4152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d413fae0 0x7fa3d4152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57698/0 (socket says 172.21.15.141:57698) 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 -- 172.21.15.141:0/2236141526 learned_addr learned my addr 172.21.15.141:0/2236141526 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 -- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3d413cfa0 msgr2=0x7fa3d413b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3e00640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3d413cfa0 0x7fa3d413b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3d413cfa0 0x7fa3d413b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 -- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3d415afb0 msgr2=0x7fa3d415c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3d415afb0 0x7fa3d415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 -- 172.21.15.141:0/2236141526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc009400 con 0x7fa3d413fae0 2024-01-16T15:06:43.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.888+0000 7fa3d3400640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d413fae0 0x7fa3d4152460 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc034040 tx=0x7fa3bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.889+0000 7fa3cbe00640 1 -- 172.21.15.141:0/2236141526 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc032900 con 0x7fa3d413fae0 2024-01-16T15:06:43.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.889+0000 7fa3cbe00640 1 -- 172.21.15.141:0/2236141526 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc032a60 con 0x7fa3d413fae0 2024-01-16T15:06:43.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.889+0000 7fa3cbe00640 1 -- 172.21.15.141:0/2236141526 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc032e10 con 0x7fa3d413fae0 2024-01-16T15:06:43.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.889+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d413a7a0 con 0x7fa3d413fae0 2024-01-16T15:06:43.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.889+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d413ad60 con 0x7fa3d413fae0 2024-01-16T15:06:43.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.890+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d413af50 con 0x7fa3d413fae0 2024-01-16T15:06:43.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.891+0000 7fa3cbe00640 1 -- 172.21.15.141:0/2236141526 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fa3bc03b3f0 con 0x7fa3d413fae0 2024-01-16T15:06:43.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.892+0000 7fa3cbe00640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3ac06be90 0x7fa3ac06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.892+0000 7fa3cbe00640 1 -- 172.21.15.141:0/2236141526 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa3bc0aa900 con 0x7fa3d413fae0 2024-01-16T15:06:43.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.893+0000 7fa3d3e00640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3ac06be90 0x7fa3ac06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.894+0000 7fa3d3e00640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3ac06be90 0x7fa3ac06e350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4000f60 tx=0x7fa3c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:43.897+0000 7fa3cbe00640 1 -- 172.21.15.141:0/2236141526 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3bc076f60 con 0x7fa3d413fae0 2024-01-16T15:06:44.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.160+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa3d40b26d0 con 0x7fa3d413fae0 2024-01-16T15:06:44.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.161+0000 7fa3cbe00640 1 -- 172.21.15.141:0/2236141526 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fa3bc07a580 con 0x7fa3d413fae0 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:44.141 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 596, 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:44.142 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:44.143 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1827, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7517550013, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 15018262528, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 758075850752, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 181247902, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 186494551, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 47 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:44.144 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:44.145 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:06:44.146 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.19045713543891907, 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:44.147 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3ac06be90 msgr2=0x7fa3ac06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3ac06be90 0x7fa3ac06e350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4000f60 tx=0x7fa3c4009040 comp rx=0 tx=0).stop 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d413fae0 msgr2=0x7fa3d4152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d413fae0 0x7fa3d4152460 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc034040 tx=0x7fa3bc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 shutdown_connections 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa3ac06be90 0x7fa3ac06e350 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3d415afb0 0x7fa3d415c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3d413fae0 0x7fa3d4152460 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.164+0000 7fa3d9600640 1 --2- 172.21.15.141:0/2236141526 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa3d413cfa0 0x7fa3d413b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.165+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 >> 172.21.15.141:0/2236141526 conn(0x7fa3d406a800 msgr2=0x7fa3d40af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:44.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.165+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 shutdown_connections 2024-01-16T15:06:44.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:44.165+0000 7fa3d9600640 1 -- 172.21.15.141:0/2236141526 wait complete. 2024-01-16T15:06:45.164 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:45.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.404+0000 7f6a88400640 1 Processor -- start 2024-01-16T15:06:45.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.404+0000 7f6a88400640 1 -- start start 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a88400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6a800ae5a0 0x7f6a8015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a88400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6a800b3ae0 0x7f6a8006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a88400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a80070c40 con 0x7f6a8006d120 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a88400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a80071450 con 0x7f6a800b3ae0 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a88400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8006c470 con 0x7f6a800ae5a0 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a85000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6a800b3ae0 0x7f6a8006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a86400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6a8006d120 0x7f6a800cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55804/0 (socket says 172.21.15.141:55804) 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.406+0000 7f6a86400640 1 -- 172.21.15.141:0/876216101 learned_addr learned my addr 172.21.15.141:0/876216101 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.407+0000 7f6a85a00640 1 --2- 172.21.15.141:0/876216101 >> v2:172.21.15.188:3300/0 conn(0x7f6a800ae5a0 0x7f6a8015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.386 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.407+0000 7f6a7fe00640 1 -- 172.21.15.141:0/876216101 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449437778 0 0) 0x7f6a80070c40 con 0x7f6a8006d120 2024-01-16T15:06:45.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.407+0000 7f6a85000640 1 -- 172.21.15.141:0/876216101 >> v2:172.21.15.188:3300/0 conn(0x7f6a800ae5a0 msgr2=0x7f6a8015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.407+0000 7f6a85000640 1 --2- 172.21.15.141:0/876216101 >> v2:172.21.15.188:3300/0 conn(0x7f6a800ae5a0 0x7f6a8015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.407+0000 7f6a85000640 1 -- 172.21.15.141:0/876216101 >> v1:172.21.15.141:6789/0 conn(0x7f6a8006d120 legacy=0x7f6a800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.407+0000 7f6a85000640 1 -- 172.21.15.141:0/876216101 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8006c750 con 0x7f6a800b3ae0 2024-01-16T15:06:45.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.408+0000 7f6a85000640 1 --2- 172.21.15.141:0/876216101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a8006cbe0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f6a800cfe20 tx=0x7f6a6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce4a3b819e29260 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.408+0000 7f6a7fe00640 1 -- 172.21.15.141:0/876216101 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68031830 con 0x7f6a800b3ae0 2024-01-16T15:06:45.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.408+0000 7f6a7fe00640 1 -- 172.21.15.141:0/876216101 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a68031990 con 0x7f6a800b3ae0 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.408+0000 7f6a7fe00640 1 -- 172.21.15.141:0/876216101 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68031d20 con 0x7f6a800b3ae0 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 -- 172.21.15.141:0/876216101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 msgr2=0x7f6a8006cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 --2- 172.21.15.141:0/876216101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a8006cbe0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f6a800cfe20 tx=0x7f6a6802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 -- 172.21.15.141:0/876216101 shutdown_connections 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 --2- 172.21.15.141:0/876216101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a8006cbe0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 --2- 172.21.15.141:0/876216101 >> v2:172.21.15.188:3300/0 conn(0x7f6a800ae5a0 0x7f6a8015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 -- 172.21.15.141:0/876216101 >> 172.21.15.141:0/876216101 conn(0x7f6a8006a800 msgr2=0x7f6a8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 -- 172.21.15.141:0/876216101 shutdown_connections 2024-01-16T15:06:45.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.409+0000 7f6a88400640 1 -- 172.21.15.141:0/876216101 wait complete. 2024-01-16T15:06:45.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.411+0000 7f6a88400640 1 Processor -- start 2024-01-16T15:06:45.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.411+0000 7f6a88400640 1 -- start start 2024-01-16T15:06:45.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a88400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a8006d120 0x7f6a800ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a88400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a800ae5a0 0x7f6a801f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a88400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a80160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a88400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a800cc010 con 0x7f6a800b3ae0 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a88400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a801f6a80 con 0x7f6a800ae5a0 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a88400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a800cbd30 con 0x7f6a8006d120 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a86400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a80160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a86400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a80160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57722/0 (socket says 172.21.15.141:57722) 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.412+0000 7f6a86400640 1 -- 172.21.15.141:0/3334989403 learned_addr learned my addr 172.21.15.141:0/3334989403 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a85a00640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a8006d120 0x7f6a800ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a86400640 1 -- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a8006d120 msgr2=0x7f6a800ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a86400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a8006d120 0x7f6a800ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a85000640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a800ae5a0 0x7f6a801f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a86400640 1 -- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a800ae5a0 msgr2=0x7f6a801f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a86400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a800ae5a0 0x7f6a801f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a86400640 1 -- 172.21.15.141:0/3334989403 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a68009400 con 0x7f6a800b3ae0 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a85a00640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a8006d120 0x7f6a800ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a86400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a80160f30 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800c7a0 tx=0x7f6a7800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a7d600640 1 -- 172.21.15.141:0/3334989403 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800a640 con 0x7f6a800b3ae0 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a7d600640 1 -- 172.21.15.141:0/3334989403 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7800a7a0 con 0x7f6a800b3ae0 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a7d600640 1 -- 172.21.15.141:0/3334989403 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800abd0 con 0x7f6a800b3ae0 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a801f6430 con 0x7f6a800b3ae0 2024-01-16T15:06:45.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.413+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a801f2b00 con 0x7f6a800b3ae0 2024-01-16T15:06:45.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.414+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a801f6bf0 con 0x7f6a800b3ae0 2024-01-16T15:06:45.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.415+0000 7f6a7d600640 1 -- 172.21.15.141:0/3334989403 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f6a7800ad30 con 0x7f6a800b3ae0 2024-01-16T15:06:45.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.416+0000 7f6a7d600640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a5806be90 0x7f6a5806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.417+0000 7f6a7d600640 1 -- 172.21.15.141:0/3334989403 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6a78087da0 con 0x7f6a800b3ae0 2024-01-16T15:06:45.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.417+0000 7f6a85a00640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a5806be90 0x7f6a5806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.418+0000 7f6a85a00640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a5806be90 0x7f6a5806e350 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6a70006fd0 tx=0x7f6a70008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.421+0000 7f6a7d600640 1 -- 172.21.15.141:0/3334989403 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a78054400 con 0x7f6a800b3ae0 2024-01-16T15:06:45.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.681+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6a800675d0 con 0x7f6a800b3ae0 2024-01-16T15:06:45.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.682+0000 7f6a7d600640 1 -- 172.21.15.141:0/3334989403 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f6a78018020 con 0x7f6a800b3ae0 2024-01-16T15:06:45.661 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:45.661 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:45.661 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:45.662 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:45.663 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:45.663 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:45.663 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:45.663 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:45.663 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:45.663 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:45.663 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 598, 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:45.664 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:45.665 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 1827, 2024-01-16T15:06:45.666 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 7517550013, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 15018262528, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 758075850752, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 181247902, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 186494551, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 43, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 47 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.667 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:45.668 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.669 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:45.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:45.671 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.19045713543891907, 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:45.672 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:45.673 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:45.673 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:45.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a5806be90 msgr2=0x7f6a5806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a5806be90 0x7f6a5806e350 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6a70006fd0 tx=0x7f6a70008040 comp rx=0 tx=0).stop 2024-01-16T15:06:45.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 msgr2=0x7f6a80160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a80160f30 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800c7a0 tx=0x7f6a7800cc60 comp rx=0 tx=0).stop 2024-01-16T15:06:45.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 shutdown_connections 2024-01-16T15:06:45.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6a5806be90 0x7f6a5806e350 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a800b3ae0 0x7f6a80160f30 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a800ae5a0 0x7f6a801f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 --2- 172.21.15.141:0/3334989403 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6a8006d120 0x7f6a800ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 >> 172.21.15.141:0/3334989403 conn(0x7f6a8006a800 msgr2=0x7f6a8006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 shutdown_connections 2024-01-16T15:06:45.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:45.685+0000 7f6a88400640 1 -- 172.21.15.141:0/3334989403 wait complete. 2024-01-16T15:06:46.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:06:46.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.924+0000 7f7c5ee00640 1 Processor -- start 2024-01-16T15:06:46.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.924+0000 7f7c5ee00640 1 -- start start 2024-01-16T15:06:46.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.925+0000 7f7c5ee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c58143ae0 0x7f7c580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.925+0000 7f7c5ee00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7c580cf7d0 0x7f7c5815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.925+0000 7f7c5ee00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7c5815e1e0 0x7f7c5815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c5ee00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c580cfc30 con 0x7f7c580cf7d0 2024-01-16T15:06:46.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c5ee00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c580b1ee0 con 0x7f7c5815e1e0 2024-01-16T15:06:46.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c5ee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580b2780 con 0x7f7c58143ae0 2024-01-16T15:06:46.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7c580cf7d0 0x7f7c5815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7c580cf7d0 0x7f7c5815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57724/0 (socket says 172.21.15.141:57724) 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 -- 172.21.15.141:0/3733212313 learned_addr learned my addr 172.21.15.141:0/3733212313 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c57e00640 1 --2- 172.21.15.141:0/3733212313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c58143ae0 0x7f7c580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c5ce00640 1 --2- 172.21.15.141:0/3733212313 >> v2:172.21.15.188:3300/0 conn(0x7f7c5815e1e0 0x7f7c5815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 -- 172.21.15.141:0/3733212313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c58143ae0 msgr2=0x7f7c580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 --2- 172.21.15.141:0/3733212313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c58143ae0 0x7f7c580cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 -- 172.21.15.141:0/3733212313 >> v2:172.21.15.188:3300/0 conn(0x7f7c5815e1e0 msgr2=0x7f7c5815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 --2- 172.21.15.141:0/3733212313 >> v2:172.21.15.188:3300/0 conn(0x7f7c5815e1e0 0x7f7c5815a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c4f200640 1 -- 172.21.15.141:0/3733212313 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5815e990 con 0x7f7c580cf7d0 2024-01-16T15:06:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.926+0000 7f7c5ce00640 1 --2- 172.21.15.141:0/3733212313 >> v2:172.21.15.188:3300/0 conn(0x7f7c5815e1e0 0x7f7c5815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:46.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.928+0000 7f7c4f200640 1 --2- 172.21.15.141:0/3733212313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c580cf7d0 0x7f7c5815dca0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7c58072160 tx=0x7f7c4002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275c799632f39308 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.928+0000 7f7c57400640 1 -- 172.21.15.141:0/3733212313 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40031860 con 0x7f7c580cf7d0 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.928+0000 7f7c57400640 1 -- 172.21.15.141:0/3733212313 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c400319c0 con 0x7f7c580cf7d0 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.928+0000 7f7c57400640 1 -- 172.21.15.141:0/3733212313 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40031d50 con 0x7f7c580cf7d0 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.928+0000 7f7c5ee00640 1 -- 172.21.15.141:0/3733212313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c580cf7d0 msgr2=0x7f7c5815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.928+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/3733212313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c580cf7d0 0x7f7c5815dca0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7c58072160 tx=0x7f7c4002b900 comp rx=0 tx=0).stop 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.929+0000 7f7c5ee00640 1 -- 172.21.15.141:0/3733212313 shutdown_connections 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.929+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/3733212313 >> v2:172.21.15.188:3300/0 conn(0x7f7c5815e1e0 0x7f7c5815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.929+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/3733212313 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c580cf7d0 0x7f7c5815dca0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.929+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/3733212313 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c58143ae0 0x7f7c580cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.929+0000 7f7c5ee00640 1 -- 172.21.15.141:0/3733212313 >> 172.21.15.141:0/3733212313 conn(0x7f7c5806a800 msgr2=0x7f7c5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:46.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.929+0000 7f7c5ee00640 1 -- 172.21.15.141:0/3733212313 shutdown_connections 2024-01-16T15:06:46.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.929+0000 7f7c5ee00640 1 -- 172.21.15.141:0/3733212313 wait complete. 2024-01-16T15:06:46.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.930+0000 7f7c5ee00640 1 Processor -- start 2024-01-16T15:06:46.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.931+0000 7f7c5ee00640 1 -- start start 2024-01-16T15:06:46.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.931+0000 7f7c5ee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c580cf7d0 0x7f7c580ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c58143ae0 0x7f7c5813daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ee00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c5815e1e0 0x7f7c58140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ee00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580cdea0 con 0x7f7c5815e1e0 2024-01-16T15:06:46.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580cd600 con 0x7f7c58143ae0 2024-01-16T15:06:46.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580cd8e0 con 0x7f7c580cf7d0 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c5815e1e0 0x7f7c58140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c5815e1e0 0x7f7c58140370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57734/0 (socket says 172.21.15.141:57734) 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 -- 172.21.15.141:0/639076056 learned_addr learned my addr 172.21.15.141:0/639076056 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c4f200640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c58143ae0 0x7f7c5813daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c57e00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c580cf7d0 0x7f7c580ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 -- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c580cf7d0 msgr2=0x7f7c580ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c580cf7d0 0x7f7c580ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 -- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c58143ae0 msgr2=0x7f7c5813daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c58143ae0 0x7f7c5813daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c5ce00640 1 -- 172.21.15.141:0/639076056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c400094d0 con 0x7f7c5815e1e0 2024-01-16T15:06:46.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.932+0000 7f7c4f200640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c58143ae0 0x7f7c5813daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.933+0000 7f7c5ce00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c5815e1e0 0x7f7c58140370 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000cd50 tx=0x7f7c5000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.933+0000 7f7c54c00640 1 -- 172.21.15.141:0/639076056 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5000f680 con 0x7f7c5815e1e0 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.933+0000 7f7c54c00640 1 -- 172.21.15.141:0/639076056 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5000f800 con 0x7f7c5815e1e0 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.933+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c580cdbc0 con 0x7f7c5815e1e0 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.933+0000 7f7c54c00640 1 -- 172.21.15.141:0/639076056 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5000fbb0 con 0x7f7c5815e1e0 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.933+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5813e030 con 0x7f7c5815e1e0 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.934+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c580acc40 con 0x7f7c5815e1e0 2024-01-16T15:06:46.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.935+0000 7f7c54c00640 1 -- 172.21.15.141:0/639076056 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f7c5000b040 con 0x7f7c5815e1e0 2024-01-16T15:06:46.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.936+0000 7f7c54c00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c3006be90 0x7f7c3006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:46.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.936+0000 7f7c54c00640 1 -- 172.21.15.141:0/639076056 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7c50088f60 con 0x7f7c5815e1e0 2024-01-16T15:06:46.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.936+0000 7f7c57e00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c3006be90 0x7f7c3006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:46.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.938+0000 7f7c57e00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c3006be90 0x7f7c3006e350 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7c48006fd0 tx=0x7f7c48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:46.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:46.940+0000 7f7c54c00640 1 -- 172.21.15.141:0/639076056 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c580acc40 con 0x7f7c5815e1e0 2024-01-16T15:06:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.198+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7c580acc40 con 0x7f7c5815e1e0 2024-01-16T15:06:47.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.199+0000 7f7c54c00640 1 -- 172.21.15.141:0/639076056 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2409 (secure 0 0 0) 0x7f7c580acc40 con 0x7f7c5815e1e0 2024-01-16T15:06:47.178 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:06:47.178 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:06:47.179 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 599, 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:06:47.180 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:06:47.181 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 2339, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 9665102607, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 19774939136, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 753319174144, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 359376439, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 365328069, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 85, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 92 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:06:47.182 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:06:47.183 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:06:47.184 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 28.571% (26s)\n [========....................] (remaining: 66s)", 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.28571429848670959, 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (26s)\n [==============..............] (remaining: 26s)", 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.5, 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:06:47.185 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c3006be90 msgr2=0x7f7c3006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c3006be90 0x7f7c3006e350 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7c48006fd0 tx=0x7f7c48008040 comp rx=0 tx=0).stop 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c5815e1e0 msgr2=0x7f7c58140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c5815e1e0 0x7f7c58140370 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000cd50 tx=0x7f7c5000a650 comp rx=0 tx=0).stop 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 shutdown_connections 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7c3006be90 0x7f7c3006e350 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c5815e1e0 0x7f7c58140370 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c58143ae0 0x7f7c5813daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 --2- 172.21.15.141:0/639076056 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7c580cf7d0 0x7f7c580ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 >> 172.21.15.141:0/639076056 conn(0x7f7c5806a800 msgr2=0x7f7c580b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 shutdown_connections 2024-01-16T15:06:47.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:47.202+0000 7f7c5ee00640 1 -- 172.21.15.141:0/639076056 wait complete. 2024-01-16T15:06:48.203 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59638 2024-01-16T15:06:48.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.451+0000 7f9c1a400640 1 Processor -- start 2024-01-16T15:06:48.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.451+0000 7f9c1a400640 1 -- start start 2024-01-16T15:06:48.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c1a400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9c140cf9a0 0x7f9c1415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c1a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 0x7f9c140702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c1a400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140b1ee0 con 0x7f9c14143ae0 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c1a400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140b2780 con 0x7f9c140cf9a0 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c1a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1415e990 con 0x7f9c1415e1e0 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c13400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9c14143ae0 0x7f9c140cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55818/0 (socket says 172.21.15.141:55818) 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c13400640 1 -- 172.21.15.141:0/2774305707 learned_addr learned my addr 172.21.15.141:0/2774305707 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c0be00640 1 --2- 172.21.15.141:0/2774305707 >> v2:172.21.15.188:3300/0 conn(0x7f9c140cf9a0 0x7f9c1415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723906040 0 0) 0x7f9c140b1ee0 con 0x7f9c14143ae0 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c13e00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 0x7f9c140702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.453+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4003520 con 0x7f9c14143ae0 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667563175 0 0) 0x7f9bf4003520 con 0x7f9c14143ae0 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 msgr2=0x7f9c140702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 0x7f9c140702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 >> v2:172.21.15.188:3300/0 conn(0x7f9c140cf9a0 msgr2=0x7f9c1415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 --2- 172.21.15.141:0/2774305707 >> v2:172.21.15.188:3300/0 conn(0x7f9c140cf9a0 0x7f9c1415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140b2130 con 0x7f9c14143ae0 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9c04001670 con 0x7f9c14143ae0 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c13e00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 0x7f9c140702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:48.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 >> v1:172.21.15.141:6789/0 conn(0x7f9c14143ae0 legacy=0x7f9c140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 0x7f9bf4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9bf40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4004d00 0x7f9bf4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140b2780 con 0x7f9bf4003f00 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1415e990 con 0x7f9bf4004d00 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf400ba00 con 0x7f9c1415e1e0 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.454+0000 7f9c0be00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9bf40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.455+0000 7f9c13400640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 0x7f9bf4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.455+0000 7f9c13e00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4004d00 0x7f9bf4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.455+0000 7f9c0be00640 1 -- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 msgr2=0x7f9bf4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:48.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.455+0000 7f9c0be00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c1415e1e0 0x7f9bf4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.455+0000 7f9c0be00640 1 -- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4004d00 msgr2=0x7f9bf4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:48.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.455+0000 7f9c0be00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4004d00 0x7f9bf4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.455+0000 7f9c0be00640 1 -- 172.21.15.141:0/2774305707 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4003520 con 0x7f9bf4003f00 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.456+0000 7f9c0be00640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9bf40047c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009650 tx=0x7f9bf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858680acecbc0d84 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.456+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc02e920 con 0x7f9bf4003f00 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.456+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc02ea80 con 0x7f9bf4003f00 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.456+0000 7f9c12a00640 1 -- 172.21.15.141:0/2774305707 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc02ee10 con 0x7f9bf4003f00 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.457+0000 7f9c1a400640 1 -- 172.21.15.141:0/2774305707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 msgr2=0x7f9bf40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.457+0000 7f9c1a400640 1 --2- 172.21.15.141:0/2774305707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9bf40047c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009650 tx=0x7f9bf40057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.457+0000 7f9c13e00640 1 -- 172.21.15.141:0/2774305707 reap_dead start 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.457+0000 7f9c1a400640 1 -- 172.21.15.141:0/2774305707 shutdown_connections 2024-01-16T15:06:48.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.457+0000 7f9c1a400640 1 -- 172.21.15.141:0/2774305707 >> 172.21.15.141:0/2774305707 conn(0x7f9c1406a800 msgr2=0x7f9c1406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.457+0000 7f9c1a400640 1 -- 172.21.15.141:0/2774305707 shutdown_connections 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.457+0000 7f9c1a400640 1 -- 172.21.15.141:0/2774305707 wait complete. 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.459+0000 7f9c1a400640 1 Processor -- start 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.459+0000 7f9c1a400640 1 -- start start 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c1a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9c14159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c1a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c141f32c0 0x7f9c14134280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c1a400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c141347c0 0x7f9c140adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c1a400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c8730 con 0x7f9bf4003f00 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c1a400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140aca70 con 0x7f9c141347c0 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c1a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140acd50 con 0x7f9c141f32c0 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c13400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9c14159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c13400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9c14159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57746/0 (socket says 172.21.15.141:57746) 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c13400640 1 -- 172.21.15.141:0/1410689039 learned_addr learned my addr 172.21.15.141:0/1410689039 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c0be00640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c141f32c0 0x7f9c14134280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c13400640 1 -- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c141f32c0 msgr2=0x7f9c14134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c13400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c141f32c0 0x7f9c14134280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c13400640 1 -- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c141347c0 msgr2=0x7f9c140adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.460+0000 7f9c13400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c141347c0 0x7f9c140adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.461+0000 7f9c13400640 1 -- 172.21.15.141:0/1410689039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140ad030 con 0x7f9bf4003f00 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.461+0000 7f9c13400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9c14159bb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e7f0 tx=0x7f9c0400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.461+0000 7f9c0b400640 1 -- 172.21.15.141:0/1410689039 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400c870 con 0x7f9bf4003f00 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.461+0000 7f9c0b400640 1 -- 172.21.15.141:0/1410689039 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0400c9d0 con 0x7f9bf4003f00 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.461+0000 7f9c0b400640 1 -- 172.21.15.141:0/1410689039 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400cd80 con 0x7f9bf4003f00 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.461+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c140ad310 con 0x7f9bf4003f00 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.461+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1415a0f0 con 0x7f9bf4003f00 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.462+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c140675d0 con 0x7f9bf4003f00 2024-01-16T15:06:48.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.463+0000 7f9c0b400640 1 -- 172.21.15.141:0/1410689039 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f9c0401d460 con 0x7f9bf4003f00 2024-01-16T15:06:48.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.464+0000 7f9c0b400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9bec06be90 0x7f9bec06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:48.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.464+0000 7f9c0b400640 1 -- 172.21.15.141:0/1410689039 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9c04088940 con 0x7f9bf4003f00 2024-01-16T15:06:48.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.465+0000 7f9c0be00640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9bec06be90 0x7f9bec06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:48.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.466+0000 7f9c0be00640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9bec06be90 0x7f9bec06e350 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc038000 tx=0x7f9bfc032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:48.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.469+0000 7f9c0b400640 1 -- 172.21.15.141:0/1410689039 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c04054ed0 con 0x7f9bf4003f00 2024-01-16T15:06:48.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.698+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59638", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c140b1a80 con 0x7f9bec06be90 2024-01-16T15:06:48.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.884+0000 7f9c0b400640 1 -- 172.21.15.141:0/1410689039 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9c140b1a80 con 0x7f9bec06be90 2024-01-16T15:06:48.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.887+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9bec06be90 msgr2=0x7f9bec06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.887+0000 7f9c1a400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9bec06be90 0x7f9bec06e350 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc038000 tx=0x7f9bfc032000 comp rx=0 tx=0).stop 2024-01-16T15:06:48.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.887+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 msgr2=0x7f9c14159bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.887+0000 7f9c1a400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9c14159bb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e7f0 tx=0x7f9c0400ecb0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 shutdown_connections 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9bec06be90 0x7f9bec06e350 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c141347c0 0x7f9c140adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9c141f32c0 0x7f9c14134280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 --2- 172.21.15.141:0/1410689039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bf4003f00 0x7f9c14159bb0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 >> 172.21.15.141:0/1410689039 conn(0x7f9c1406a800 msgr2=0x7f9c140cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 shutdown_connections 2024-01-16T15:06:48.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:48.888+0000 7f9c1a400640 1 -- 172.21.15.141:0/1410689039 wait complete. 2024-01-16T15:06:48.888 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone59639 2024-01-16T15:06:49.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.121+0000 7f7718800640 1 Processor -- start 2024-01-16T15:06:49.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.121+0000 7f7718800640 1 -- start start 2024-01-16T15:06:49.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.122+0000 7f7718800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 0x7f77100cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.122+0000 7f7718800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f771015f9c0 0x7f77100cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.122+0000 7f7718800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771015c5f0 con 0x7f771013df80 2024-01-16T15:06:49.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.122+0000 7f7718800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771006c470 con 0x7f771015f9c0 2024-01-16T15:06:49.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.122+0000 7f7718800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771006c750 con 0x7f771013e360 2024-01-16T15:06:49.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7715e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f771013df80 0x7f771015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55830/0 (socket says 172.21.15.141:55830) 2024-01-16T15:06:49.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7715e00640 1 -- 172.21.15.141:0/196968127 learned_addr learned my addr 172.21.15.141:0/196968127 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265735883 0 0) 0x7f771015c5f0 con 0x7f771013df80 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f0003520 con 0x7f771013df80 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7716800640 1 --2- 172.21.15.141:0/196968127 >> v2:172.21.15.188:3300/0 conn(0x7f771015f9c0 0x7f77100cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7715400640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 0x7f77100cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520179151 0 0) 0x7f76f0003520 con 0x7f771013df80 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 msgr2=0x7f77100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 0x7f77100cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 >> v2:172.21.15.188:3300/0 conn(0x7f771015f9c0 msgr2=0x7f77100cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 --2- 172.21.15.141:0/196968127 >> v2:172.21.15.188:3300/0 conn(0x7f771015f9c0 0x7f77100cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771006ca30 con 0x7f771013df80 2024-01-16T15:06:49.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7700001670 con 0x7f771013df80 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.123+0000 7f7716800640 1 --2- 172.21.15.141:0/196968127 >> v2:172.21.15.188:3300/0 conn(0x7f771015f9c0 0x7f77100cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 >> v1:172.21.15.141:6789/0 conn(0x7f771013df80 legacy=0x7f771015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7714a00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0003970 0x7f76f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7714a00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 0x7f76f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7714a00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0004d00 0x7f76f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771006c470 con 0x7f76f0003970 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771006c750 con 0x7f76f0004d00 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f000ba60 con 0x7f771013e360 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7715e00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0003970 0x7f76f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7715400640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 0x7f76f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7715e00640 1 -- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 msgr2=0x7f76f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7715e00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f771013e360 0x7f76f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7715e00640 1 -- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0004d00 msgr2=0x7f76f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7716800640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0004d00 0x7f76f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7715e00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0004d00 0x7f76f0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.124+0000 7f7715e00640 1 -- 172.21.15.141:0/196968127 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f0003520 con 0x7f76f0003970 2024-01-16T15:06:49.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.126+0000 7f7715e00640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0003970 0x7f76f0003d90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f770000c240 tx=0x7f76f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c019bb58b4d52f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.126+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770002cae0 con 0x7f76f0003970 2024-01-16T15:06:49.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.126+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770002cc40 con 0x7f76f0003970 2024-01-16T15:06:49.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.126+0000 7f7714a00640 1 -- 172.21.15.141:0/196968127 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77000058d0 con 0x7f76f0003970 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.126+0000 7f7718800640 1 -- 172.21.15.141:0/196968127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0003970 msgr2=0x7f76f0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.126+0000 7f7718800640 1 --2- 172.21.15.141:0/196968127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76f0003970 0x7f76f0003d90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f770000c240 tx=0x7f76f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.126+0000 7f7716800640 1 -- 172.21.15.141:0/196968127 reap_dead start 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.127+0000 7f7718800640 1 -- 172.21.15.141:0/196968127 shutdown_connections 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.127+0000 7f7718800640 1 -- 172.21.15.141:0/196968127 >> 172.21.15.141:0/196968127 conn(0x7f771006a800 msgr2=0x7f771006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.127+0000 7f7718800640 1 -- 172.21.15.141:0/196968127 shutdown_connections 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.127+0000 7f7718800640 1 -- 172.21.15.141:0/196968127 wait complete. 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.128+0000 7f7718800640 1 Processor -- start 2024-01-16T15:06:49.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.129+0000 7f7718800640 1 -- start start 2024-01-16T15:06:49.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.129+0000 7f7718800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0003970 0x7f7710150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.129+0000 7f7718800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771013d7d0 0x7f77101f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.129+0000 7f7718800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771014ea30 0x7f771014eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.129+0000 7f7718800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771014f8d0 con 0x7f771014ea30 2024-01-16T15:06:49.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.129+0000 7f7718800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771014f3f0 con 0x7f771013d7d0 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.129+0000 7f7718800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77101f4090 con 0x7f76f0003970 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771014ea30 0x7f771014eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771014ea30 0x7f771014eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57770/0 (socket says 172.21.15.141:57770) 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 -- 172.21.15.141:0/1615613551 learned_addr learned my addr 172.21.15.141:0/1615613551 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7715e00640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0003970 0x7f7710150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7715400640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771013d7d0 0x7f77101f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 -- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0003970 msgr2=0x7f7710150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0003970 0x7f7710150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 -- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771013d7d0 msgr2=0x7f77101f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771013d7d0 0x7f77101f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 -- 172.21.15.141:0/1615613551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77101f2ff0 con 0x7f771014ea30 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.130+0000 7f7716800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771014ea30 0x7f771014eeb0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f77100cfea0 tx=0x7f7708004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.131+0000 7f770e000640 1 -- 172.21.15.141:0/1615613551 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708007ce0 con 0x7f771014ea30 2024-01-16T15:06:49.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.131+0000 7f770e000640 1 -- 172.21.15.141:0/1615613551 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7708007e80 con 0x7f771014ea30 2024-01-16T15:06:49.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.131+0000 7f770e000640 1 -- 172.21.15.141:0/1615613551 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708014640 con 0x7f771014ea30 2024-01-16T15:06:49.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.131+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77101f32d0 con 0x7f771014ea30 2024-01-16T15:06:49.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.131+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77101f3890 con 0x7f771014ea30 2024-01-16T15:06:49.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.132+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77100675d0 con 0x7f771014ea30 2024-01-16T15:06:49.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.133+0000 7f770e000640 1 -- 172.21.15.141:0/1615613551 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f77080147a0 con 0x7f771014ea30 2024-01-16T15:06:49.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.134+0000 7f770e000640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76e806bec0 0x7f76e806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.134+0000 7f770e000640 1 -- 172.21.15.141:0/1615613551 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f77080880e0 con 0x7f771014ea30 2024-01-16T15:06:49.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.134+0000 7f7715e00640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76e806bec0 0x7f76e806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.135+0000 7f7715e00640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76e806bec0 0x7f76e806e380 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7700034000 tx=0x7f7700007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.138+0000 7f770e000640 1 -- 172.21.15.141:0/1615613551 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7708054a50 con 0x7f771014ea30 2024-01-16T15:06:49.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.363+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone59639", "target": ["mon-mgr", ""]}) v1 -- 0x7f771006cff0 con 0x7f76e806bec0 2024-01-16T15:06:49.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.540+0000 7f770e000640 1 -- 172.21.15.141:0/1615613551 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f770002c8e0 con 0x7f76e806bec0 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.543+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76e806bec0 msgr2=0x7f76e806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.543+0000 7f7718800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76e806bec0 0x7f76e806e380 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7700034000 tx=0x7f7700007000 comp rx=0 tx=0).stop 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.543+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771014ea30 msgr2=0x7f771014eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.543+0000 7f7718800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771014ea30 0x7f771014eeb0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f77100cfea0 tx=0x7f7708004840 comp rx=0 tx=0).stop 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 shutdown_connections 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f76e806bec0 0x7f76e806e380 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f771014ea30 0x7f771014eeb0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f771013d7d0 0x7f77101f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 --2- 172.21.15.141:0/1615613551 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f76f0003970 0x7f7710150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 >> 172.21.15.141:0/1615613551 conn(0x7f771006a800 msgr2=0x7f77101538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:49.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 shutdown_connections 2024-01-16T15:06:49.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.544+0000 7f7718800640 1 -- 172.21.15.141:0/1615613551 wait complete. 2024-01-16T15:06:49.541 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596310 2024-01-16T15:06:49.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.829+0000 7ff483200640 1 Processor -- start 2024-01-16T15:06:49.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.829+0000 7ff483200640 1 -- start start 2024-01-16T15:06:49.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff483200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff47c143ae0 0x7ff47c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff483200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c15e3b0 0x7ff47c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff483200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff47c0cfc30 con 0x7ff47c0cf7d0 2024-01-16T15:06:49.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff483200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47c0b1ee0 con 0x7ff47c143ae0 2024-01-16T15:06:49.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff483200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c0b2780 con 0x7ff47c15e3b0 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff47be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff47c143ae0 0x7ff47c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff47be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff47c143ae0 0x7ff47c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57784/0 (socket says 172.21.15.141:57784) 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.831+0000 7ff47be00640 1 -- 172.21.15.141:0/205888581 learned_addr learned my addr 172.21.15.141:0/205888581 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.832+0000 7ff481200640 1 --2- 172.21.15.141:0/205888581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c15e3b0 0x7ff47c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.832+0000 7ff47be00640 1 -- 172.21.15.141:0/205888581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c15e3b0 msgr2=0x7ff47c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.832+0000 7ff47be00640 1 --2- 172.21.15.141:0/205888581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c15e3b0 0x7ff47c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.832+0000 7ff47be00640 1 -- 172.21.15.141:0/205888581 >> v1:172.21.15.141:6789/0 conn(0x7ff47c0cf7d0 legacy=0x7ff47c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.832+0000 7ff47be00640 1 -- 172.21.15.141:0/205888581 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c15e990 con 0x7ff47c143ae0 2024-01-16T15:06:49.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.832+0000 7ff481200640 1 --2- 172.21.15.141:0/205888581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c15e3b0 0x7ff47c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:49.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.834+0000 7ff47be00640 1 --2- 172.21.15.141:0/205888581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c143ae0 0x7ff47c0cf290 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009870 tx=0x7ff46c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9e6ad956788e61 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.835+0000 7ff47aa00640 1 -- 172.21.15.141:0/205888581 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c030840 con 0x7ff47c143ae0 2024-01-16T15:06:49.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.835+0000 7ff47aa00640 1 -- 172.21.15.141:0/205888581 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c0309a0 con 0x7ff47c143ae0 2024-01-16T15:06:49.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.835+0000 7ff47aa00640 1 -- 172.21.15.141:0/205888581 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c030d30 con 0x7ff47c143ae0 2024-01-16T15:06:49.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 -- 172.21.15.141:0/205888581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c143ae0 msgr2=0x7ff47c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 --2- 172.21.15.141:0/205888581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c143ae0 0x7ff47c0cf290 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009870 tx=0x7ff46c02b980 comp rx=0 tx=0).stop 2024-01-16T15:06:49.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 -- 172.21.15.141:0/205888581 shutdown_connections 2024-01-16T15:06:49.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 --2- 172.21.15.141:0/205888581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c15e3b0 0x7ff47c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 --2- 172.21.15.141:0/205888581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c143ae0 0x7ff47c0cf290 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 -- 172.21.15.141:0/205888581 >> 172.21.15.141:0/205888581 conn(0x7ff47c06a800 msgr2=0x7ff47c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:49.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 -- 172.21.15.141:0/205888581 shutdown_connections 2024-01-16T15:06:49.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.836+0000 7ff483200640 1 -- 172.21.15.141:0/205888581 wait complete. 2024-01-16T15:06:49.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.839+0000 7ff483200640 1 Processor -- start 2024-01-16T15:06:49.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.839+0000 7ff483200640 1 -- start start 2024-01-16T15:06:49.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.841+0000 7ff483200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff47c0cf7d0 0x7ff47c136590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.841+0000 7ff483200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c143ae0 0x7ff47c1344a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.841+0000 7ff483200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c15e3b0 0x7ff47c133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.841+0000 7ff483200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c131d60 con 0x7ff47c15e3b0 2024-01-16T15:06:49.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.841+0000 7ff483200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c1349e0 con 0x7ff47c0cf7d0 2024-01-16T15:06:49.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.841+0000 7ff483200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c131150 con 0x7ff47c143ae0 2024-01-16T15:06:49.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.842+0000 7ff481200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c15e3b0 0x7ff47c133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.842+0000 7ff481200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c15e3b0 0x7ff47c133040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57796/0 (socket says 172.21.15.141:57796) 2024-01-16T15:06:49.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.842+0000 7ff481200640 1 -- 172.21.15.141:0/3923834609 learned_addr learned my addr 172.21.15.141:0/3923834609 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:49.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.843+0000 7ff47be00640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff47c0cf7d0 0x7ff47c136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.843+0000 7ff481200640 1 -- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c143ae0 msgr2=0x7ff47c1344a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.843+0000 7ff481200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c143ae0 0x7ff47c1344a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.843+0000 7ff481200640 1 -- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff47c0cf7d0 msgr2=0x7ff47c136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.843+0000 7ff481200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff47c0cf7d0 0x7ff47c136590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.843+0000 7ff481200640 1 -- 172.21.15.141:0/3923834609 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0094d0 con 0x7ff47c15e3b0 2024-01-16T15:06:49.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.844+0000 7ff481200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c15e3b0 0x7ff47c133040 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff47400cd80 tx=0x7ff47400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.844+0000 7ff473e00640 1 -- 172.21.15.141:0/3923834609 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47400f680 con 0x7ff47c15e3b0 2024-01-16T15:06:49.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.844+0000 7ff473e00640 1 -- 172.21.15.141:0/3923834609 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47400f800 con 0x7ff47c15e3b0 2024-01-16T15:06:49.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.844+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c131430 con 0x7ff47c15e3b0 2024-01-16T15:06:49.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.844+0000 7ff473e00640 1 -- 172.21.15.141:0/3923834609 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47400fbb0 con 0x7ff47c15e3b0 2024-01-16T15:06:49.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.845+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c1319f0 con 0x7ff47c15e3b0 2024-01-16T15:06:49.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.847+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47c131f10 con 0x7ff47c15e3b0 2024-01-16T15:06:49.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.848+0000 7ff473e00640 1 -- 172.21.15.141:0/3923834609 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff47400b040 con 0x7ff47c15e3b0 2024-01-16T15:06:49.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.849+0000 7ff473e00640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45406be90 0x7ff45406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.849+0000 7ff473e00640 1 -- 172.21.15.141:0/3923834609 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff474088f40 con 0x7ff47c15e3b0 2024-01-16T15:06:49.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.851+0000 7ff47be00640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45406be90 0x7ff45406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.854+0000 7ff473e00640 1 -- 172.21.15.141:0/3923834609 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff47c131f10 con 0x7ff47c15e3b0 2024-01-16T15:06:49.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:49.882+0000 7ff47be00640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45406be90 0x7ff45406e350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff46c036000 tx=0x7ff46c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.185+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596310", "target": ["mon-mgr", ""]}) v1 -- 0x7ff47c0b2410 con 0x7ff45406be90 2024-01-16T15:06:50.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.925+0000 7ff473e00640 1 -- 172.21.15.141:0/3923834609 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff46c044470 con 0x7ff45406be90 2024-01-16T15:06:50.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45406be90 msgr2=0x7ff45406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45406be90 0x7ff45406e350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff46c036000 tx=0x7ff46c008030 comp rx=0 tx=0).stop 2024-01-16T15:06:50.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c15e3b0 msgr2=0x7ff47c133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c15e3b0 0x7ff47c133040 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff47400cd80 tx=0x7ff47400a650 comp rx=0 tx=0).stop 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 shutdown_connections 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45406be90 0x7ff45406e350 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff47c15e3b0 0x7ff47c133040 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff47c143ae0 0x7ff47c1344a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 --2- 172.21.15.141:0/3923834609 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff47c0cf7d0 0x7ff47c136590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.930+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 >> 172.21.15.141:0/3923834609 conn(0x7ff47c06a800 msgr2=0x7ff47c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:50.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.931+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 shutdown_connections 2024-01-16T15:06:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:50.931+0000 7ff483200640 1 -- 172.21.15.141:0/3923834609 wait complete. 2024-01-16T15:06:50.940 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596311 2024-01-16T15:06:51.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.269+0000 7face1200640 1 Processor -- start 2024-01-16T15:06:51.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.269+0000 7face1200640 1 -- start start 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7face1200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7facdc143ae0 0x7facdc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7face1200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7facdc0cf7d0 0x7facdc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7face1200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facdc0cfc30 con 0x7facdc15e1e0 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7face1200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc0b1ee0 con 0x7facdc143ae0 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7face1200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc0b2780 con 0x7facdc0cf7d0 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7facd9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7facdc143ae0 0x7facdc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7facd9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7facdc143ae0 0x7facdc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57806/0 (socket says 172.21.15.141:57806) 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7facd9e00640 1 -- 172.21.15.141:0/4179416449 learned_addr learned my addr 172.21.15.141:0/4179416449 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.270+0000 7facda800640 1 --1- 172.21.15.141:0/4179416449 >> v1:172.21.15.141:6789/0 conn(0x7facdc15e1e0 0x7facdc15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55848/0 (socket says 172.21.15.141:55848) 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.271+0000 7facd9400640 1 --2- 172.21.15.141:0/4179416449 >> v2:172.21.15.188:3300/0 conn(0x7facdc0cf7d0 0x7facdc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.271+0000 7facd9e00640 1 -- 172.21.15.141:0/4179416449 >> v2:172.21.15.188:3300/0 conn(0x7facdc0cf7d0 msgr2=0x7facdc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.271+0000 7facd9e00640 1 --2- 172.21.15.141:0/4179416449 >> v2:172.21.15.188:3300/0 conn(0x7facdc0cf7d0 0x7facdc15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.271+0000 7facd9e00640 1 -- 172.21.15.141:0/4179416449 >> v1:172.21.15.141:6789/0 conn(0x7facdc15e1e0 legacy=0x7facdc15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.271+0000 7facd9e00640 1 -- 172.21.15.141:0/4179416449 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc15e990 con 0x7facdc143ae0 2024-01-16T15:06:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.271+0000 7facd8a00640 1 -- 172.21.15.141:0/4179416449 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520756267 0 0) 0x7facdc0cfc30 con 0x7facdc15e1e0 2024-01-16T15:06:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.272+0000 7facd9e00640 1 --2- 172.21.15.141:0/4179416449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc0cf290 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7facdc0cb830 tx=0x7faccc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb0b62c8e513242 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.272+0000 7facd8a00640 1 -- 172.21.15.141:0/4179416449 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc02cb20 con 0x7facdc143ae0 2024-01-16T15:06:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.272+0000 7facd8a00640 1 -- 172.21.15.141:0/4179416449 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc02cc80 con 0x7facdc143ae0 2024-01-16T15:06:51.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.272+0000 7facd8a00640 1 -- 172.21.15.141:0/4179416449 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0343f0 con 0x7facdc143ae0 2024-01-16T15:06:51.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 -- 172.21.15.141:0/4179416449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 msgr2=0x7facdc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 --2- 172.21.15.141:0/4179416449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc0cf290 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7facdc0cb830 tx=0x7faccc02bf80 comp rx=0 tx=0).stop 2024-01-16T15:06:51.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 -- 172.21.15.141:0/4179416449 shutdown_connections 2024-01-16T15:06:51.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 --2- 172.21.15.141:0/4179416449 >> v2:172.21.15.188:3300/0 conn(0x7facdc0cf7d0 0x7facdc15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 --2- 172.21.15.141:0/4179416449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc0cf290 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 -- 172.21.15.141:0/4179416449 >> 172.21.15.141:0/4179416449 conn(0x7facdc06a800 msgr2=0x7facdc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:51.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 -- 172.21.15.141:0/4179416449 shutdown_connections 2024-01-16T15:06:51.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.273+0000 7face1200640 1 -- 172.21.15.141:0/4179416449 wait complete. 2024-01-16T15:06:51.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.276+0000 7face1200640 1 Processor -- start 2024-01-16T15:06:51.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.276+0000 7face1200640 1 -- start start 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7face1200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7facdc0cf7d0 0x7facdc13b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7face1200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc13ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7face1200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facdc15e1e0 0x7facdc13e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7face1200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc13fa70 con 0x7facdc143ae0 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7face1200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc13fd50 con 0x7facdc15e1e0 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7face1200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc140030 con 0x7facdc0cf7d0 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7facd9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc13ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7facd9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc13ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57816/0 (socket says 172.21.15.141:57816) 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.277+0000 7facd9400640 1 -- 172.21.15.141:0/953872514 learned_addr learned my addr 172.21.15.141:0/953872514 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9e00640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7facdc0cf7d0 0x7facdc13b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facda800640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facdc15e1e0 0x7facdc13e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9400640 1 -- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7facdc0cf7d0 msgr2=0x7facdc13b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9400640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7facdc0cf7d0 0x7facdc13b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9400640 1 -- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facdc15e1e0 msgr2=0x7facdc13e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9400640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facdc15e1e0 0x7facdc13e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9400640 1 -- 172.21.15.141:0/953872514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc009b30 con 0x7facdc143ae0 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9e00640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7facdc0cf7d0 0x7facdc13b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facda800640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facdc15e1e0 0x7facdc13e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.278+0000 7facd9400640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc13ca90 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7facc400d5f0 tx=0x7facc400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.279+0000 7facd2000640 1 -- 172.21.15.141:0/953872514 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40125f0 con 0x7facdc143ae0 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.279+0000 7facd2000640 1 -- 172.21.15.141:0/953872514 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc4012750 con 0x7facdc143ae0 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.279+0000 7facd2000640 1 -- 172.21.15.141:0/953872514 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4012b00 con 0x7facdc143ae0 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.279+0000 7face1200640 1 -- 172.21.15.141:0/953872514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc140310 con 0x7facdc143ae0 2024-01-16T15:06:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.279+0000 7face1200640 1 -- 172.21.15.141:0/953872514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc13f420 con 0x7facdc143ae0 2024-01-16T15:06:51.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.281+0000 7face1200640 1 -- 172.21.15.141:0/953872514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc0b2bf0 con 0x7facdc143ae0 2024-01-16T15:06:51.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.281+0000 7facd2000640 1 -- 172.21.15.141:0/953872514 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7facc4012c60 con 0x7facdc143ae0 2024-01-16T15:06:51.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.282+0000 7facd2000640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7facb406be90 0x7facb406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:51.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.282+0000 7facd2000640 1 -- 172.21.15.141:0/953872514 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7facc4088c60 con 0x7facdc143ae0 2024-01-16T15:06:51.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.284+0000 7facd9e00640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7facb406be90 0x7facb406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:51.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.289+0000 7facd9e00640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7facb406be90 0x7facb406e350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7faccc035000 tx=0x7faccc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:51.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.290+0000 7facd2000640 1 -- 172.21.15.141:0/953872514 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facdc0b2bf0 con 0x7facdc143ae0 2024-01-16T15:06:51.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:51.627+0000 7face1200640 1 -- 172.21.15.141:0/953872514 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596311", "target": ["mon-mgr", ""]}) v1 -- 0x7facdc15e670 con 0x7facb406be90 2024-01-16T15:06:52.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.175+0000 7facd2000640 1 -- 172.21.15.141:0/953872514 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faccc040450 con 0x7facb406be90 2024-01-16T15:06:52.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 -- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7facb406be90 msgr2=0x7facb406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7facb406be90 0x7facb406e350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7faccc035000 tx=0x7faccc030000 comp rx=0 tx=0).stop 2024-01-16T15:06:52.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 -- 172.21.15.141:0/953872514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 msgr2=0x7facdc13ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc13ca90 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7facc400d5f0 tx=0x7facc400dab0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 -- 172.21.15.141:0/953872514 shutdown_connections 2024-01-16T15:06:52.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7facb406be90 0x7facb406e350 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facdc15e1e0 0x7facdc13e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7facdc143ae0 0x7facdc13ca90 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 --2- 172.21.15.141:0/953872514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7facdc0cf7d0 0x7facdc13b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.181+0000 7face1200640 1 -- 172.21.15.141:0/953872514 >> 172.21.15.141:0/953872514 conn(0x7facdc06a800 msgr2=0x7facdc15c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:52.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.182+0000 7face1200640 1 -- 172.21.15.141:0/953872514 shutdown_connections 2024-01-16T15:06:52.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.182+0000 7face1200640 1 -- 172.21.15.141:0/953872514 wait complete. 2024-01-16T15:06:52.198 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596312 2024-01-16T15:06:52.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.518+0000 7fe4b0e00640 1 Processor -- start 2024-01-16T15:06:52.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.518+0000 7fe4b0e00640 1 -- start start 2024-01-16T15:06:52.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.520+0000 7fe4b0e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac0cd960 0x7fe4ac0cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4b0e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe4ac162f30 0x7fe4ac165320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4b0e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ac0ce2a0 con 0x7fe4ac15b2a0 2024-01-16T15:06:52.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4b0e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ac0d00a0 con 0x7fe4ac162f30 2024-01-16T15:06:52.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4b0e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac0b4010 con 0x7fe4ac0cd960 2024-01-16T15:06:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4aa800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe4ac162f30 0x7fe4ac165320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4aa800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe4ac162f30 0x7fe4ac165320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57826/0 (socket says 172.21.15.141:57826) 2024-01-16T15:06:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4aa800640 1 -- 172.21.15.141:0/528264620 learned_addr learned my addr 172.21.15.141:0/528264620 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4aa800640 1 -- 172.21.15.141:0/528264620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac0cd960 msgr2=0x7fe4ac0cdd60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:52.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4aa800640 1 --2- 172.21.15.141:0/528264620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac0cd960 0x7fe4ac0cdd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4aa800640 1 -- 172.21.15.141:0/528264620 >> v1:172.21.15.188:6789/0 conn(0x7fe4ac15b2a0 legacy=0x7fe4ac0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:52.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.521+0000 7fe4aa800640 1 -- 172.21.15.141:0/528264620 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0b42f0 con 0x7fe4ac162f30 2024-01-16T15:06:52.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.523+0000 7fe4aa800640 1 --2- 172.21.15.141:0/528264620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac162f30 0x7fe4ac165320 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009e50 tx=0x7fe4a402bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35464b0ddeba95fe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:52.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.523+0000 7fe4a8a00640 1 -- 172.21.15.141:0/528264620 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a402caa0 con 0x7fe4ac162f30 2024-01-16T15:06:52.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.523+0000 7fe4a8a00640 1 -- 172.21.15.141:0/528264620 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a402cc00 con 0x7fe4ac162f30 2024-01-16T15:06:52.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.523+0000 7fe4a8a00640 1 -- 172.21.15.141:0/528264620 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a40343f0 con 0x7fe4ac162f30 2024-01-16T15:06:52.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.524+0000 7fe4b0e00640 1 -- 172.21.15.141:0/528264620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac162f30 msgr2=0x7fe4ac165320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.524+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/528264620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac162f30 0x7fe4ac165320 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009e50 tx=0x7fe4a402bf00 comp rx=0 tx=0).stop 2024-01-16T15:06:52.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.524+0000 7fe4b0e00640 1 -- 172.21.15.141:0/528264620 shutdown_connections 2024-01-16T15:06:52.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.524+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/528264620 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac162f30 0x7fe4ac165320 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.524+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/528264620 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac0cd960 0x7fe4ac0cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.524+0000 7fe4b0e00640 1 -- 172.21.15.141:0/528264620 >> 172.21.15.141:0/528264620 conn(0x7fe4ac0d1020 msgr2=0x7fe4ac0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:52.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.524+0000 7fe4b0e00640 1 -- 172.21.15.141:0/528264620 shutdown_connections 2024-01-16T15:06:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.525+0000 7fe4b0e00640 1 -- 172.21.15.141:0/528264620 wait complete. 2024-01-16T15:06:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.527+0000 7fe4b0e00640 1 Processor -- start 2024-01-16T15:06:52.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.527+0000 7fe4b0e00640 1 -- start start 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4b0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac0cd960 0x7fe4ac06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4b0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4ac15b2a0 0x7fe4ac06efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4b0e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac162f30 0x7fe4ac153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4b0e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac06c160 con 0x7fe4ac0cd960 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4b0e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac06b590 con 0x7fe4ac15b2a0 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4b0e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac06b830 con 0x7fe4ac162f30 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4a9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac0cd960 0x7fe4ac06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4a9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac0cd960 0x7fe4ac06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57836/0 (socket says 172.21.15.141:57836) 2024-01-16T15:06:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4a9e00640 1 -- 172.21.15.141:0/1414664512 learned_addr learned my addr 172.21.15.141:0/1414664512 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4aa800640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac162f30 0x7fe4ac153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.528+0000 7fe4a9400640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4ac15b2a0 0x7fe4ac06efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a9e00640 1 -- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac162f30 msgr2=0x7fe4ac153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a9e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac162f30 0x7fe4ac153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a9e00640 1 -- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4ac15b2a0 msgr2=0x7fe4ac06efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a9e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4ac15b2a0 0x7fe4ac06efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a9e00640 1 -- 172.21.15.141:0/1414664512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a4009b30 con 0x7fe4ac0cd960 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a9400640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4ac15b2a0 0x7fe4ac06efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a9e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac0cd960 0x7fe4ac06c5f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00c410 tx=0x7fe49c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a2000640 1 -- 172.21.15.141:0/1414664512 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c0035f0 con 0x7fe4ac0cd960 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a2000640 1 -- 172.21.15.141:0/1414664512 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c003750 con 0x7fe4ac0cd960 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4a2000640 1 -- 172.21.15.141:0/1414664512 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c003b40 con 0x7fe4ac0cd960 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.529+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac06bb10 con 0x7fe4ac0cd960 2024-01-16T15:06:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.530+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac06f510 con 0x7fe4ac0cd960 2024-01-16T15:06:52.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.531+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac0b6d60 con 0x7fe4ac0cd960 2024-01-16T15:06:52.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.531+0000 7fe4a2000640 1 -- 172.21.15.141:0/1414664512 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe49c003ca0 con 0x7fe4ac0cd960 2024-01-16T15:06:52.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.532+0000 7fe4a2000640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe48406be90 0x7fe48406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.532+0000 7fe4a2000640 1 -- 172.21.15.141:0/1414664512 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe49c087b30 con 0x7fe4ac0cd960 2024-01-16T15:06:52.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.539+0000 7fe4a2000640 1 -- 172.21.15.141:0/1414664512 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe49c054190 con 0x7fe4ac0cd960 2024-01-16T15:06:52.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.551+0000 7fe4a9400640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe48406be90 0x7fe48406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.563+0000 7fe4a9400640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe48406be90 0x7fe48406e350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe494006fd0 tx=0x7fe494008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:52.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.806+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596312", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4ac0b48b0 con 0x7fe48406be90 2024-01-16T15:06:52.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.964+0000 7fe4a2000640 1 -- 172.21.15.141:0/1414664512 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe494009140 con 0x7fe48406be90 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe48406be90 msgr2=0x7fe48406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe48406be90 0x7fe48406e350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe494006fd0 tx=0x7fe494008040 comp rx=0 tx=0).stop 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac0cd960 msgr2=0x7fe4ac06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac0cd960 0x7fe4ac06c5f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00c410 tx=0x7fe49c00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 shutdown_connections 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe48406be90 0x7fe48406e350 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe4ac162f30 0x7fe4ac153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4ac15b2a0 0x7fe4ac06efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 --2- 172.21.15.141:0/1414664512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4ac0cd960 0x7fe4ac06c5f0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.967+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 >> 172.21.15.141:0/1414664512 conn(0x7fe4ac0d1020 msgr2=0x7fe4ac0b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.968+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 shutdown_connections 2024-01-16T15:06:52.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:52.968+0000 7fe4b0e00640 1 -- 172.21.15.141:0/1414664512 wait complete. 2024-01-16T15:06:52.966 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596313 2024-01-16T15:06:53.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.199+0000 7f1fcf800640 1 Processor -- start 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.199+0000 7f1fcf800640 1 -- start start 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.201+0000 7f1fcf800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc813df80 0x7f1fc815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.201+0000 7f1fcf800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc806c470 con 0x7f1fc80cf9a0 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.201+0000 7f1fcf800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc806c750 con 0x7f1fc813e360 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.201+0000 7f1fcf800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc806ca30 con 0x7f1fc813df80 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.201+0000 7f1fcd800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1fc80cf9a0 0x7f1fc80cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53030/0 (socket says 172.21.15.141:53030) 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.201+0000 7f1fcd800640 1 -- 172.21.15.141:0/3610115297 learned_addr learned my addr 172.21.15.141:0/3610115297 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963944818 0 0) 0x7f1fc806c470 con 0x7f1fc80cf9a0 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa8003520 con 0x7f1fc80cf9a0 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fcce00640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc813df80 0x7f1fc815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209890913 0 0) 0x7f1fa8003520 con 0x7f1fc80cf9a0 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc813df80 msgr2=0x7f1fc815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc813df80 0x7f1fc815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 >> v1:172.21.15.188:6789/0 conn(0x7f1fc813e360 legacy=0x7f1fc80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc806cd10 con 0x7f1fc80cf9a0 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1fc00019f0 con 0x7f1fc80cf9a0 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 >> v1:172.21.15.141:6789/0 conn(0x7f1fc80cf9a0 legacy=0x7f1fc80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fa8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa80047c0 0x7f1fa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.202+0000 7f1fc7400640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc813df80 0x7f1fa800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc806ca30 con 0x7f1fa80039a0 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80054e0 con 0x7f1fa80047c0 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa800bec0 con 0x7f1fc813df80 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fcce00640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fa8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fcce00640 1 -- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc813df80 msgr2=0x7f1fa800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fcce00640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc813df80 0x7f1fa800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fcce00640 1 -- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa80047c0 msgr2=0x7f1fa8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:53.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fcce00640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa80047c0 0x7f1fa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.203+0000 7f1fcce00640 1 -- 172.21.15.141:0/3610115297 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa8003520 con 0x7f1fa80039a0 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.205+0000 7f1fcce00640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fa8003da0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800a9a0 tx=0x7f1fa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68235bd63275045 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.205+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8030640 con 0x7f1fa80039a0 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.205+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb80307a0 con 0x7f1fa80039a0 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.205+0000 7f1fc7400640 1 -- 172.21.15.141:0/3610115297 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8030b50 con 0x7f1fa80039a0 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.206+0000 7f1fcf800640 1 -- 172.21.15.141:0/3610115297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 msgr2=0x7f1fa8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.206+0000 7f1fcf800640 1 --2- 172.21.15.141:0/3610115297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fa8003da0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800a9a0 tx=0x7f1fa80057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.206+0000 7f1fcd800640 1 -- 172.21.15.141:0/3610115297 reap_dead start 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.206+0000 7f1fcf800640 1 -- 172.21.15.141:0/3610115297 shutdown_connections 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.206+0000 7f1fcf800640 1 -- 172.21.15.141:0/3610115297 >> 172.21.15.141:0/3610115297 conn(0x7f1fc806a800 msgr2=0x7f1fc806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:53.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.206+0000 7f1fcf800640 1 -- 172.21.15.141:0/3610115297 shutdown_connections 2024-01-16T15:06:53.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.206+0000 7f1fcf800640 1 -- 172.21.15.141:0/3610115297 wait complete. 2024-01-16T15:06:53.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.208+0000 7f1fcf800640 1 Processor -- start 2024-01-16T15:06:53.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.208+0000 7f1fcf800640 1 -- start start 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.209+0000 7f1fcf800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fc81f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.209+0000 7f1fcf800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fc8071820 0x7f1fc814b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.209+0000 7f1fcf800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc8071c80 0x7f1fc814da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.209+0000 7f1fcf800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc81f6ea0 con 0x7f1fa80039a0 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.209+0000 7f1fcf800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc81f3920 con 0x7f1fc8071820 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.209+0000 7f1fcf800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc814d580 con 0x7f1fc8071c80 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fc81f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fc81f6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50054/0 (socket says 172.21.15.141:50054) 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 -- 172.21.15.141:0/2423396176 learned_addr learned my addr 172.21.15.141:0/2423396176 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 -- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc8071c80 msgr2=0x7f1fc814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc8071c80 0x7f1fc814da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 -- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fc8071820 msgr2=0x7f1fc814b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fc8071820 0x7f1fc814b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 -- 172.21.15.141:0/2423396176 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc81f3080 con 0x7f1fa80039a0 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.210+0000 7f1fcce00640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fc81f6960 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800a120 tx=0x7f1fb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.211+0000 7f1fc4c00640 1 -- 172.21.15.141:0/2423396176 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb803e040 con 0x7f1fa80039a0 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.211+0000 7f1fc4c00640 1 -- 172.21.15.141:0/2423396176 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb8039960 con 0x7f1fa80039a0 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.211+0000 7f1fc4c00640 1 -- 172.21.15.141:0/2423396176 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8039cd0 con 0x7f1fa80039a0 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.211+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc81f3360 con 0x7f1fa80039a0 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.211+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc814b870 con 0x7f1fa80039a0 2024-01-16T15:06:53.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.212+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc80675d0 con 0x7f1fa80039a0 2024-01-16T15:06:53.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.213+0000 7f1fc4c00640 1 -- 172.21.15.141:0/2423396176 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1fb802e070 con 0x7f1fa80039a0 2024-01-16T15:06:53.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.214+0000 7f1fc4c00640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1fa006bec0 0x7f1fa006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.214+0000 7f1fc4c00640 1 -- 172.21.15.141:0/2423396176 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1fb80aa590 con 0x7f1fa80039a0 2024-01-16T15:06:53.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.214+0000 7f1fc7e00640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1fa006bec0 0x7f1fa006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.216+0000 7f1fc7e00640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1fa006bec0 0x7f1fa006e380 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0003cb0 tx=0x7f1fb00079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.218+0000 7f1fc4c00640 1 -- 172.21.15.141:0/2423396176 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fb807a850 con 0x7f1fa80039a0 2024-01-16T15:06:53.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.573+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596313", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fc81f6ea0 con 0x7f1fa006bec0 2024-01-16T15:06:53.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:53.994+0000 7f1fc4c00640 1 -- 172.21.15.141:0/2423396176 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1fc81f6ea0 con 0x7f1fa006bec0 2024-01-16T15:06:53.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.003+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1fa006bec0 msgr2=0x7f1fa006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.003+0000 7f1fcf800640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1fa006bec0 0x7f1fa006e380 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0003cb0 tx=0x7f1fb00079b0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.003+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 msgr2=0x7f1fc81f6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.003+0000 7f1fcf800640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fc81f6960 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800a120 tx=0x7f1fb80046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.005+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 shutdown_connections 2024-01-16T15:06:53.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.005+0000 7f1fcf800640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1fa006bec0 0x7f1fa006e380 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.005+0000 7f1fcf800640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fc8071c80 0x7f1fc814da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.005+0000 7f1fcf800640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fc8071820 0x7f1fc814b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.005+0000 7f1fcf800640 1 --2- 172.21.15.141:0/2423396176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fa80039a0 0x7f1fc81f6960 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.005+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 >> 172.21.15.141:0/2423396176 conn(0x7f1fc806a800 msgr2=0x7f1fc80cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:53.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.008+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 shutdown_connections 2024-01-16T15:06:53.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.011+0000 7f1fcf800640 1 -- 172.21.15.141:0/2423396176 wait complete. 2024-01-16T15:06:54.021 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596314 2024-01-16T15:06:54.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.318+0000 7f3a09c00640 1 Processor -- start 2024-01-16T15:06:54.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.319+0000 7f3a09c00640 1 -- start start 2024-01-16T15:06:54.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f3a09c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3a040ae5a0 0x7f3a0415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f3a09c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3a040cf290 0x7f3a040cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f3a09c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a04070c40 con 0x7f3a040b3ae0 2024-01-16T15:06:54.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f3a09c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a04071450 con 0x7f3a040ae5a0 2024-01-16T15:06:54.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f3a09c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a040cfc60 con 0x7f3a040cf290 2024-01-16T15:06:54.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f39fbe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3a040b3ae0 0x7f3a0406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53046/0 (socket says 172.21.15.141:53046) 2024-01-16T15:06:54.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f39fbe00640 1 -- 172.21.15.141:0/3802554370 learned_addr learned my addr 172.21.15.141:0/3802554370 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:54.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.320+0000 7f3a02a00640 1 --2- 172.21.15.141:0/3802554370 >> v2:172.21.15.141:3300/0 conn(0x7f3a040ae5a0 0x7f3a0415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.321+0000 7f3a03400640 1 --2- 172.21.15.141:0/3802554370 >> v2:172.21.15.188:3300/0 conn(0x7f3a040cf290 0x7f3a040cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.321+0000 7f3a02a00640 1 -- 172.21.15.141:0/3802554370 >> v2:172.21.15.188:3300/0 conn(0x7f3a040cf290 msgr2=0x7f3a040cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:54.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.321+0000 7f3a02a00640 1 --2- 172.21.15.141:0/3802554370 >> v2:172.21.15.188:3300/0 conn(0x7f3a040cf290 0x7f3a040cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.321+0000 7f3a02a00640 1 -- 172.21.15.141:0/3802554370 >> v1:172.21.15.141:6789/0 conn(0x7f3a040b3ae0 legacy=0x7f3a0406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.321+0000 7f3a02a00640 1 -- 172.21.15.141:0/3802554370 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0406c470 con 0x7f3a040ae5a0 2024-01-16T15:06:54.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.322+0000 7f3a02a00640 1 --2- 172.21.15.141:0/3802554370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040ae5a0 0x7f3a0415e430 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f3a040b3f70 tx=0x7f39f402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109ae5d41064099 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:54.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.323+0000 7f3a02000640 1 -- 172.21.15.141:0/3802554370 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4033640 con 0x7f3a040ae5a0 2024-01-16T15:06:54.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.323+0000 7f3a02000640 1 -- 172.21.15.141:0/3802554370 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f40337a0 con 0x7f3a040ae5a0 2024-01-16T15:06:54.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.323+0000 7f3a02000640 1 -- 172.21.15.141:0/3802554370 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4033b30 con 0x7f3a040ae5a0 2024-01-16T15:06:54.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.324+0000 7f3a09c00640 1 -- 172.21.15.141:0/3802554370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040ae5a0 msgr2=0x7f3a0415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.324+0000 7f3a09c00640 1 --2- 172.21.15.141:0/3802554370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040ae5a0 0x7f3a0415e430 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f3a040b3f70 tx=0x7f39f402b7a0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.325+0000 7f3a09c00640 1 -- 172.21.15.141:0/3802554370 shutdown_connections 2024-01-16T15:06:54.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.325+0000 7f3a09c00640 1 --2- 172.21.15.141:0/3802554370 >> v2:172.21.15.188:3300/0 conn(0x7f3a040cf290 0x7f3a040cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.325+0000 7f3a09c00640 1 --2- 172.21.15.141:0/3802554370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040ae5a0 0x7f3a0415e430 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.325+0000 7f3a09c00640 1 -- 172.21.15.141:0/3802554370 >> 172.21.15.141:0/3802554370 conn(0x7f3a0406a800 msgr2=0x7f3a0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:54.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.329+0000 7f3a09c00640 1 -- 172.21.15.141:0/3802554370 shutdown_connections 2024-01-16T15:06:54.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.329+0000 7f3a09c00640 1 -- 172.21.15.141:0/3802554370 wait complete. 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.332+0000 7f3a09c00640 1 Processor -- start 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.332+0000 7f3a09c00640 1 -- start start 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.333+0000 7f3a09c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a040ae5a0 0x7f3a04157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.333+0000 7f3a09c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040b3ae0 0x7f3a0414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.333+0000 7f3a09c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a040cf290 0x7f3a041fa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.333+0000 7f3a09c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04158350 con 0x7f3a040b3ae0 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.333+0000 7f3a09c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04150590 con 0x7f3a040ae5a0 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.333+0000 7f3a09c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04159100 con 0x7f3a040cf290 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.334+0000 7f39fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040b3ae0 0x7f3a0414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.334+0000 7f39fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040b3ae0 0x7f3a0414d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50070/0 (socket says 172.21.15.141:50070) 2024-01-16T15:06:54.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.334+0000 7f39fbe00640 1 -- 172.21.15.141:0/4049209917 learned_addr learned my addr 172.21.15.141:0/4049209917 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.335+0000 7f39fbe00640 1 -- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a040cf290 msgr2=0x7f3a041fa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.335+0000 7f39fbe00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a040cf290 0x7f3a041fa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.335+0000 7f39fbe00640 1 -- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a040ae5a0 msgr2=0x7f3a04157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.335+0000 7f39fbe00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a040ae5a0 0x7f3a04157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.335+0000 7f39fbe00640 1 -- 172.21.15.141:0/4049209917 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4009400 con 0x7f3a040b3ae0 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.335+0000 7f39fbe00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040b3ae0 0x7f3a0414d390 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00c780 tx=0x7f39ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.335+0000 7f39faa00640 1 -- 172.21.15.141:0/4049209917 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0072d0 con 0x7f3a040b3ae0 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.336+0000 7f39faa00640 1 -- 172.21.15.141:0/4049209917 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec007430 con 0x7f3a040b3ae0 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.336+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a04150830 con 0x7f3a040b3ae0 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.336+0000 7f39faa00640 1 -- 172.21.15.141:0/4049209917 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0047e0 con 0x7f3a040b3ae0 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.336+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a04150df0 con 0x7f3a040b3ae0 2024-01-16T15:06:54.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.338+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d4005250 con 0x7f3a040b3ae0 2024-01-16T15:06:54.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.345+0000 7f39faa00640 1 -- 172.21.15.141:0/4049209917 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f39ec004940 con 0x7f3a040b3ae0 2024-01-16T15:06:54.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.351+0000 7f39faa00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39dc06bee0 0x7f39dc06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.354+0000 7f39faa00640 1 -- 172.21.15.141:0/4049209917 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f39ec088410 con 0x7f3a040b3ae0 2024-01-16T15:06:54.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.354+0000 7f39faa00640 1 -- 172.21.15.141:0/4049209917 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39ec08b240 con 0x7f3a040b3ae0 2024-01-16T15:06:54.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.355+0000 7f3a02a00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39dc06bee0 0x7f39dc06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.401 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.424+0000 7f3a02a00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39dc06bee0 0x7f39dc06e3a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f39f4035000 tx=0x7f39f402d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:54.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.680+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596314", "target": ["mon-mgr", ""]}) v1 -- 0x7f39d4000c80 con 0x7f39dc06bee0 2024-01-16T15:06:54.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.959+0000 7f39faa00640 1 -- 172.21.15.141:0/4049209917 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f39d4000c80 con 0x7f39dc06bee0 2024-01-16T15:06:54.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39dc06bee0 msgr2=0x7f39dc06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39dc06bee0 0x7f39dc06e3a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f39f4035000 tx=0x7f39f402d000 comp rx=0 tx=0).stop 2024-01-16T15:06:54.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040b3ae0 msgr2=0x7f3a0414d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040b3ae0 0x7f3a0414d390 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00c780 tx=0x7f39ec00cc40 comp rx=0 tx=0).stop 2024-01-16T15:06:54.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 shutdown_connections 2024-01-16T15:06:54.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39dc06bee0 0x7f39dc06e3a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a040cf290 0x7f3a041fa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a040b3ae0 0x7f3a0414d390 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 --2- 172.21.15.141:0/4049209917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a040ae5a0 0x7f3a04157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.963+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 >> 172.21.15.141:0/4049209917 conn(0x7f3a0406a800 msgr2=0x7f3a04159720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:54.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.966+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 shutdown_connections 2024-01-16T15:06:54.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:54.966+0000 7f3a09c00640 1 -- 172.21.15.141:0/4049209917 wait complete. 2024-01-16T15:06:54.978 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596315 2024-01-16T15:06:55.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.262+0000 7f0e06c00640 1 Processor -- start 2024-01-16T15:06:55.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.262+0000 7f0e06c00640 1 -- start start 2024-01-16T15:06:55.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.263+0000 7f0e06c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e000ce8f0 0x7f0e000ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.263+0000 7f0e06c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e000d0de0 con 0x7f0e000b2a00 2024-01-16T15:06:55.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.263+0000 7f0e06c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e000d10c0 con 0x7f0e00160f80 2024-01-16T15:06:55.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.263+0000 7f0e06c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e000d13a0 con 0x7f0e000ce8f0 2024-01-16T15:06:55.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.264+0000 7f0e04c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0e000b2a00 0x7f0e000b2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53054/0 (socket says 172.21.15.141:53054) 2024-01-16T15:06:55.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.264+0000 7f0e04c00640 1 -- 172.21.15.141:0/211567349 learned_addr learned my addr 172.21.15.141:0/211567349 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:06:55.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.264+0000 7f0dff400640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e000ce8f0 0x7f0e000ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.264+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512020307 0 0) 0x7f0e000d0de0 con 0x7f0e000b2a00 2024-01-16T15:06:55.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de0003520 con 0x7f0e000b2a00 2024-01-16T15:06:55.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2882496520 0 0) 0x7f0de0003520 con 0x7f0e000b2a00 2024-01-16T15:06:55.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e000ce8f0 msgr2=0x7f0e000ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:55.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e000ce8f0 0x7f0e000ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 >> v1:172.21.15.188:6789/0 conn(0x7f0e00160f80 legacy=0x7f0e00161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e000d1680 con 0x7f0e000b2a00 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0df80019f0 con 0x7f0e000b2a00 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 >> v1:172.21.15.141:6789/0 conn(0x7f0e000b2a00 legacy=0x7f0e000b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e000ce8f0 0x7f0de00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0de0003fd0 0x7f0de00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0de0004d00 0x7f0de0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e000d13a0 con 0x7f0de0003fd0 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de000bb40 con 0x7f0de0004d00 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de000bd00 con 0x7f0e000ce8f0 2024-01-16T15:06:55.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.265+0000 7f0dff400640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0de0003fd0 0x7f0de00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.266+0000 7f0dff400640 1 -- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e000ce8f0 msgr2=0x7f0de00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:55.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.266+0000 7f0dff400640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e000ce8f0 0x7f0de00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.266+0000 7f0dff400640 1 -- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0de0004d00 msgr2=0x7f0de0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:55.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.266+0000 7f0dff400640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0de0004d00 0x7f0de0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.266+0000 7f0dff400640 1 -- 172.21.15.141:0/211567349 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de0003520 con 0x7f0de0003fd0 2024-01-16T15:06:55.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.267+0000 7f0dff400640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0de0003fd0 0x7f0de00047c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f0de800ba30 tx=0x7f0de00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e13fa6a4aff330 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:55.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.267+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8032680 con 0x7f0de0003fd0 2024-01-16T15:06:55.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.267+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80327e0 con 0x7f0de0003fd0 2024-01-16T15:06:55.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.267+0000 7f0dfea00640 1 -- 172.21.15.141:0/211567349 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8032b70 con 0x7f0de0003fd0 2024-01-16T15:06:55.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.268+0000 7f0e06c00640 1 -- 172.21.15.141:0/211567349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0de0003fd0 msgr2=0x7f0de00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.268+0000 7f0e06c00640 1 --2- 172.21.15.141:0/211567349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0de0003fd0 0x7f0de00047c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f0de800ba30 tx=0x7f0de00057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.268+0000 7f0e04c00640 1 -- 172.21.15.141:0/211567349 reap_dead start 2024-01-16T15:06:55.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.268+0000 7f0e06c00640 1 -- 172.21.15.141:0/211567349 shutdown_connections 2024-01-16T15:06:55.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.268+0000 7f0e06c00640 1 -- 172.21.15.141:0/211567349 >> 172.21.15.141:0/211567349 conn(0x7f0e0009c550 msgr2=0x7f0e000d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:55.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.270+0000 7f0e06c00640 1 -- 172.21.15.141:0/211567349 shutdown_connections 2024-01-16T15:06:55.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.270+0000 7f0e06c00640 1 -- 172.21.15.141:0/211567349 wait complete. 2024-01-16T15:06:55.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.272+0000 7f0e06c00640 1 Processor -- start 2024-01-16T15:06:55.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.272+0000 7f0e06c00640 1 -- start start 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0e06c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e00154f30 0x7f0e00153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0e06c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e00153590 0x7f0e0015fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0e06c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e001602f0 0x7f0e000ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0e06c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015e400 con 0x7f0e00154f30 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0e06c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015e0d0 con 0x7f0e001602f0 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0e06c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e000caf20 con 0x7f0e00153590 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0dffe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e00154f30 0x7f0e00153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0dffe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e00154f30 0x7f0e00153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50090/0 (socket says 172.21.15.141:50090) 2024-01-16T15:06:55.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.273+0000 7f0dffe00640 1 -- 172.21.15.141:0/2424653850 learned_addr learned my addr 172.21.15.141:0/2424653850 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:55.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.274+0000 7f0dffe00640 1 -- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e00153590 msgr2=0x7f0e0015fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.274+0000 7f0dffe00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e00153590 0x7f0e0015fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.274+0000 7f0dffe00640 1 -- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e001602f0 msgr2=0x7f0e000ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.274+0000 7f0dffe00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e001602f0 0x7f0e000ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.274+0000 7f0dffe00640 1 -- 172.21.15.141:0/2424653850 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e000cb200 con 0x7f0e00154f30 2024-01-16T15:06:55.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.274+0000 7f0dffe00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e00154f30 0x7f0e00153050 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c500 tx=0x7f0df000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:55.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.275+0000 7f0df7e00640 1 -- 172.21.15.141:0/2424653850 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df00052d0 con 0x7f0e00154f30 2024-01-16T15:06:55.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.275+0000 7f0df7e00640 1 -- 172.21.15.141:0/2424653850 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df0005450 con 0x7f0e00154f30 2024-01-16T15:06:55.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.275+0000 7f0df7e00640 1 -- 172.21.15.141:0/2424653850 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0012870 con 0x7f0e00154f30 2024-01-16T15:06:55.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.275+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e000cb4e0 con 0x7f0e00154f30 2024-01-16T15:06:55.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.276+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e00155780 con 0x7f0e00154f30 2024-01-16T15:06:55.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.278+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e000675d0 con 0x7f0e00154f30 2024-01-16T15:06:55.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.283+0000 7f0df7e00640 1 -- 172.21.15.141:0/2424653850 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0df00129d0 con 0x7f0e00154f30 2024-01-16T15:06:55.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.285+0000 7f0df7e00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0dd806be90 0x7f0dd806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:55.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.285+0000 7f0df7e00640 1 -- 172.21.15.141:0/2424653850 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0df0088f80 con 0x7f0e00154f30 2024-01-16T15:06:55.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.287+0000 7f0dff400640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0dd806be90 0x7f0dd806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:55.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.292+0000 7f0df7e00640 1 -- 172.21.15.141:0/2424653850 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0df00555e0 con 0x7f0e00154f30 2024-01-16T15:06:55.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.300+0000 7f0dff400640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0dd806be90 0x7f0dd806e350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0de800ba00 tx=0x7f0de8032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:55.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.607+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596315", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e000d1c40 con 0x7f0dd806be90 2024-01-16T15:06:55.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.948+0000 7f0df7e00640 1 -- 172.21.15.141:0/2424653850 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0de8032a50 con 0x7f0dd806be90 2024-01-16T15:06:55.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.952+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0dd806be90 msgr2=0x7f0dd806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.952+0000 7f0e06c00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0dd806be90 0x7f0dd806e350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0de800ba00 tx=0x7f0de8032000 comp rx=0 tx=0).stop 2024-01-16T15:06:55.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.952+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e00154f30 msgr2=0x7f0e00153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:55.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.952+0000 7f0e06c00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e00154f30 0x7f0e00153050 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c500 tx=0x7f0df000c9c0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 shutdown_connections 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0dd806be90 0x7f0dd806e350 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e001602f0 0x7f0e000ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0e00153590 0x7f0e0015fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 --2- 172.21.15.141:0/2424653850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e00154f30 0x7f0e00153050 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 >> 172.21.15.141:0/2424653850 conn(0x7f0e0009c550 msgr2=0x7f0e0006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 shutdown_connections 2024-01-16T15:06:55.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:55.953+0000 7f0e06c00640 1 -- 172.21.15.141:0/2424653850 wait complete. 2024-01-16T15:06:55.964 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596316 2024-01-16T15:06:56.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.271+0000 7f2e9ec00640 1 Processor -- start 2024-01-16T15:06:56.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.272+0000 7f2e9ec00640 1 -- start start 2024-01-16T15:06:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.273+0000 7f2e9ec00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2e98143ae0 0x7f2e980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.273+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e9815e3b0 0x7f2e980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.273+0000 7f2e9ec00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980cfc30 con 0x7f2e980cf7d0 2024-01-16T15:06:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.273+0000 7f2e9ec00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e980b1ee0 con 0x7f2e98143ae0 2024-01-16T15:06:56.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.273+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b2780 con 0x7f2e9815e3b0 2024-01-16T15:06:56.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e9cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e9815e3b0 0x7f2e980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e9cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e9815e3b0 0x7f2e980702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:58232/0 (socket says 172.21.15.141:58232) 2024-01-16T15:06:56.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e9cc00640 1 -- 172.21.15.141:0/3835157226 learned_addr learned my addr 172.21.15.141:0/3835157226 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:56.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e97400640 1 --1- 172.21.15.141:0/3835157226 >> v1:172.21.15.188:6789/0 conn(0x7f2e980cf7d0 0x7f2e9815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:38636/0 (socket says 172.21.15.141:38636) 2024-01-16T15:06:56.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e97e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2e98143ae0 0x7f2e980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e9cc00640 1 -- 172.21.15.141:0/3835157226 >> v2:172.21.15.188:3300/0 conn(0x7f2e98143ae0 msgr2=0x7f2e980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e9cc00640 1 --2- 172.21.15.141:0/3835157226 >> v2:172.21.15.188:3300/0 conn(0x7f2e98143ae0 0x7f2e980cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e9cc00640 1 -- 172.21.15.141:0/3835157226 >> v1:172.21.15.188:6789/0 conn(0x7f2e980cf7d0 legacy=0x7f2e9815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.274+0000 7f2e9cc00640 1 -- 172.21.15.141:0/3835157226 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9815e990 con 0x7f2e9815e3b0 2024-01-16T15:06:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.275+0000 7f2e9cc00640 1 --2- 172.21.15.141:0/3835157226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e9815e3b0 0x7f2e980702c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009870 tx=0x7f2e9002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6256c347530374a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e96a00640 1 -- 172.21.15.141:0/3835157226 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e90030840 con 0x7f2e9815e3b0 2024-01-16T15:06:56.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e96a00640 1 -- 172.21.15.141:0/3835157226 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e900309a0 con 0x7f2e9815e3b0 2024-01-16T15:06:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e96a00640 1 -- 172.21.15.141:0/3835157226 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e90030d30 con 0x7f2e9815e3b0 2024-01-16T15:06:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e9ec00640 1 -- 172.21.15.141:0/3835157226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e9815e3b0 msgr2=0x7f2e980702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/3835157226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e9815e3b0 0x7f2e980702c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009870 tx=0x7f2e9002b980 comp rx=0 tx=0).stop 2024-01-16T15:06:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e9ec00640 1 -- 172.21.15.141:0/3835157226 shutdown_connections 2024-01-16T15:06:56.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/3835157226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e9815e3b0 0x7f2e980702c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/3835157226 >> v2:172.21.15.188:3300/0 conn(0x7f2e98143ae0 0x7f2e980cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.276+0000 7f2e9ec00640 1 -- 172.21.15.141:0/3835157226 >> 172.21.15.141:0/3835157226 conn(0x7f2e9806a800 msgr2=0x7f2e9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:56.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.277+0000 7f2e9ec00640 1 -- 172.21.15.141:0/3835157226 shutdown_connections 2024-01-16T15:06:56.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.277+0000 7f2e9ec00640 1 -- 172.21.15.141:0/3835157226 wait complete. 2024-01-16T15:06:56.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.278+0000 7f2e9ec00640 1 Processor -- start 2024-01-16T15:06:56.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.278+0000 7f2e9ec00640 1 -- start start 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.279+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e980cf7d0 0x7f2e980cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.279+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e98143ae0 0x7f2e98158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.279+0000 7f2e9ec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e9815e3b0 0x7f2e980cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.279+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980cce60 con 0x7f2e98143ae0 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.279+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980cd140 con 0x7f2e9815e3b0 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.279+0000 7f2e9ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980cd420 con 0x7f2e980cf7d0 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e980cf7d0 0x7f2e980cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e980cf7d0 0x7f2e980cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:58242/0 (socket says 172.21.15.141:58242) 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 -- 172.21.15.141:0/2905708582 learned_addr learned my addr 172.21.15.141:0/2905708582 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97400640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e98143ae0 0x7f2e98158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e9cc00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e9815e3b0 0x7f2e980cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 -- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e9815e3b0 msgr2=0x7f2e980cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e9815e3b0 0x7f2e980cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 -- 172.21.15.141:0/2905708582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e98143ae0 msgr2=0x7f2e98158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e98143ae0 0x7f2e98158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97e00640 1 -- 172.21.15.141:0/2905708582 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900094d0 con 0x7f2e980cf7d0 2024-01-16T15:06:56.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.280+0000 7f2e97400640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e98143ae0 0x7f2e98158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.281+0000 7f2e97e00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e980cf7d0 0x7f2e980cc610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800efc0 tx=0x7f2e8800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.281+0000 7f2e8fe00640 1 -- 172.21.15.141:0/2905708582 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88014480 con 0x7f2e980cf7d0 2024-01-16T15:06:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.281+0000 7f2e8fe00640 1 -- 172.21.15.141:0/2905708582 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8800f040 con 0x7f2e980cf7d0 2024-01-16T15:06:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.281+0000 7f2e8fe00640 1 -- 172.21.15.141:0/2905708582 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88014800 con 0x7f2e980cf7d0 2024-01-16T15:06:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.281+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e980cd700 con 0x7f2e980cf7d0 2024-01-16T15:06:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.281+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e980ccb50 con 0x7f2e980cf7d0 2024-01-16T15:06:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.283+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980b1ab0 con 0x7f2e980cf7d0 2024-01-16T15:06:56.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.284+0000 7f2e8fe00640 1 -- 172.21.15.141:0/2905708582 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2e88014960 con 0x7f2e980cf7d0 2024-01-16T15:06:56.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.285+0000 7f2e8fe00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e7006bee0 0x7f2e7006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.286+0000 7f2e8fe00640 1 -- 172.21.15.141:0/2905708582 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2e88089010 con 0x7f2e980cf7d0 2024-01-16T15:06:56.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.286+0000 7f2e97400640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e7006bee0 0x7f2e7006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.286+0000 7f2e97400640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e7006bee0 0x7f2e7006e3a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2e980cbe10 tx=0x7f2e80000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.291+0000 7f2e8fe00640 1 -- 172.21.15.141:0/2905708582 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e88055980 con 0x7f2e980cf7d0 2024-01-16T15:06:56.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:06:56.526+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596316", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e980b4340 con 0x7f2e7006bee0 2024-01-16T15:07:02.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.590+0000 7f2e8fe00640 1 -- 172.21.15.141:0/2905708582 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2e980b4340 con 0x7f2e7006bee0 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.594+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e7006bee0 msgr2=0x7f2e7006e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.594+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e7006bee0 0x7f2e7006e3a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2e980cbe10 tx=0x7f2e80000ed0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.594+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e980cf7d0 msgr2=0x7f2e980cc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.594+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e980cf7d0 0x7f2e980cc610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800efc0 tx=0x7f2e8800c490 comp rx=0 tx=0).stop 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 shutdown_connections 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2e7006bee0 0x7f2e7006e3a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e9815e3b0 0x7f2e980cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2e98143ae0 0x7f2e98158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 --2- 172.21.15.141:0/2905708582 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2e980cf7d0 0x7f2e980cc610 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 >> 172.21.15.141:0/2905708582 conn(0x7f2e9806a800 msgr2=0x7f2e9815ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:02.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 shutdown_connections 2024-01-16T15:07:02.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.595+0000 7f2e9ec00640 1 -- 172.21.15.141:0/2905708582 wait complete. 2024-01-16T15:07:02.596 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596317 2024-01-16T15:07:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.832+0000 7f0d15800640 1 Processor -- start 2024-01-16T15:07:02.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.832+0000 7f0d15800640 1 -- start start 2024-01-16T15:07:02.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.834+0000 7f0d15800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0d10157c30 0x7f0d100cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.834+0000 7f0d15800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0d101618f0 0x7f0d10070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.834+0000 7f0d15800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d10071450 con 0x7f0d10071030 2024-01-16T15:07:02.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.834+0000 7f0d15800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1015ea30 con 0x7f0d101618f0 2024-01-16T15:07:02.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.834+0000 7f0d15800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1006d860 con 0x7f0d10157c30 2024-01-16T15:07:02.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.834+0000 7f0d07e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0d101618f0 0x7f0d10070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.835+0000 7f0d07e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0d101618f0 0x7f0d10070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50096/0 (socket says 172.21.15.141:50096) 2024-01-16T15:07:02.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.835+0000 7f0d07e00640 1 -- 172.21.15.141:0/210496824 learned_addr learned my addr 172.21.15.141:0/210496824 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:02.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.835+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/210496824 >> v2:172.21.15.188:3300/0 conn(0x7f0d10157c30 0x7f0d100cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.835+0000 7f0d07e00640 1 -- 172.21.15.141:0/210496824 >> v2:172.21.15.188:3300/0 conn(0x7f0d10157c30 msgr2=0x7f0d100cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.835+0000 7f0d07e00640 1 --2- 172.21.15.141:0/210496824 >> v2:172.21.15.188:3300/0 conn(0x7f0d10157c30 0x7f0d100cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.835+0000 7f0d07e00640 1 -- 172.21.15.141:0/210496824 >> v1:172.21.15.188:6789/0 conn(0x7f0d10071030 legacy=0x7f0d1006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.835+0000 7f0d07e00640 1 -- 172.21.15.141:0/210496824 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1015deb0 con 0x7f0d101618f0 2024-01-16T15:07:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.837+0000 7f0d07e00640 1 --2- 172.21.15.141:0/210496824 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d101618f0 0x7f0d10070af0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0d100723f0 tx=0x7f0cf802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeead3d8ba4bc6b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.837+0000 7f0d0e000640 1 -- 172.21.15.141:0/210496824 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8030860 con 0x7f0d101618f0 2024-01-16T15:07:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.837+0000 7f0d0e000640 1 -- 172.21.15.141:0/210496824 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf80309c0 con 0x7f0d101618f0 2024-01-16T15:07:02.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.837+0000 7f0d0e000640 1 -- 172.21.15.141:0/210496824 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8030d50 con 0x7f0d101618f0 2024-01-16T15:07:02.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.838+0000 7f0d15800640 1 -- 172.21.15.141:0/210496824 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d101618f0 msgr2=0x7f0d10070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.838+0000 7f0d15800640 1 --2- 172.21.15.141:0/210496824 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d101618f0 0x7f0d10070af0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0d100723f0 tx=0x7f0cf802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.839+0000 7f0d15800640 1 -- 172.21.15.141:0/210496824 shutdown_connections 2024-01-16T15:07:02.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.839+0000 7f0d15800640 1 --2- 172.21.15.141:0/210496824 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d101618f0 0x7f0d10070af0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.839+0000 7f0d15800640 1 --2- 172.21.15.141:0/210496824 >> v2:172.21.15.188:3300/0 conn(0x7f0d10157c30 0x7f0d100cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.839+0000 7f0d15800640 1 -- 172.21.15.141:0/210496824 >> 172.21.15.141:0/210496824 conn(0x7f0d100cf700 msgr2=0x7f0d100cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:02.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.839+0000 7f0d15800640 1 -- 172.21.15.141:0/210496824 shutdown_connections 2024-01-16T15:07:02.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.839+0000 7f0d15800640 1 -- 172.21.15.141:0/210496824 wait complete. 2024-01-16T15:07:02.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.841+0000 7f0d15800640 1 Processor -- start 2024-01-16T15:07:02.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.841+0000 7f0d15800640 1 -- start start 2024-01-16T15:07:02.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.842+0000 7f0d15800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d10071030 0x7f0d1015cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.842+0000 7f0d15800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d10157c30 0x7f0d1014b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.842+0000 7f0d15800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d101618f0 0x7f0d100ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.843+0000 7f0d15800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1014b7b0 con 0x7f0d10071030 2024-01-16T15:07:02.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.843+0000 7f0d15800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d100ae870 con 0x7f0d10157c30 2024-01-16T15:07:02.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.843+0000 7f0d15800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d100aead0 con 0x7f0d101618f0 2024-01-16T15:07:02.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.844+0000 7f0d0f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d101618f0 0x7f0d100ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.844+0000 7f0d0f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d101618f0 0x7f0d100ae330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:58256/0 (socket says 172.21.15.141:58256) 2024-01-16T15:07:02.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.844+0000 7f0d0f400640 1 -- 172.21.15.141:0/446446094 learned_addr learned my addr 172.21.15.141:0/446446094 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:02.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.844+0000 7f0d07e00640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d10157c30 0x7f0d1014b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.845+0000 7f0d0f400640 1 -- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d10157c30 msgr2=0x7f0d1014b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.845+0000 7f0d0f400640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d10157c30 0x7f0d1014b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.845+0000 7f0d0f400640 1 -- 172.21.15.141:0/446446094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d10071030 msgr2=0x7f0d1015cc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:07:02.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.848+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d10071030 0x7f0d1015cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.848+0000 7f0d0f400640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d10071030 0x7f0d1015cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.848+0000 7f0d0f400640 1 -- 172.21.15.141:0/446446094 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80094d0 con 0x7f0d101618f0 2024-01-16T15:07:02.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.849+0000 7f0d0f400640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d101618f0 0x7f0d100ae330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800c6b0 tx=0x7f0d0800cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.850+0000 7f0d06a00640 1 -- 172.21.15.141:0/446446094 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08003b70 con 0x7f0d101618f0 2024-01-16T15:07:02.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.850+0000 7f0d06a00640 1 -- 172.21.15.141:0/446446094 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d08003cf0 con 0x7f0d101618f0 2024-01-16T15:07:02.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.850+0000 7f0d06a00640 1 -- 172.21.15.141:0/446446094 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08003ea0 con 0x7f0d101618f0 2024-01-16T15:07:02.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.851+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d100aedb0 con 0x7f0d101618f0 2024-01-16T15:07:02.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.851+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d101581f0 con 0x7f0d101618f0 2024-01-16T15:07:02.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.853+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d100af2d0 con 0x7f0d101618f0 2024-01-16T15:07:02.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.857+0000 7f0d06a00640 1 -- 172.21.15.141:0/446446094 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0d08005310 con 0x7f0d101618f0 2024-01-16T15:07:02.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.859+0000 7f0d06a00640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce806bf10 0x7f0ce806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.859+0000 7f0d06a00640 1 -- 172.21.15.141:0/446446094 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0d08088dc0 con 0x7f0d101618f0 2024-01-16T15:07:02.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.870+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce806bf10 0x7f0ce806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.873+0000 7f0d0ea00640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce806bf10 0x7f0ce806e3d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0d00006fd0 tx=0x7f0d00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:02.875+0000 7f0d06a00640 1 -- 172.21.15.141:0/446446094 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d08055460 con 0x7f0d101618f0 2024-01-16T15:07:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.219+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596317", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d1015e750 con 0x7f0ce806bf10 2024-01-16T15:07:03.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.857+0000 7f0d06a00640 1 -- 172.21.15.141:0/446446094 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0d00009520 con 0x7f0ce806bf10 2024-01-16T15:07:03.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce806bf10 msgr2=0x7f0ce806e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce806bf10 0x7f0ce806e3d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0d00006fd0 tx=0x7f0d00008040 comp rx=0 tx=0).stop 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d101618f0 msgr2=0x7f0d100ae330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d101618f0 0x7f0d100ae330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800c6b0 tx=0x7f0d0800cb70 comp rx=0 tx=0).stop 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 shutdown_connections 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ce806bf10 0x7f0ce806e3d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d101618f0 0x7f0d100ae330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d10157c30 0x7f0d1014b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 --2- 172.21.15.141:0/446446094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d10071030 0x7f0d1015cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.860+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 >> 172.21.15.141:0/446446094 conn(0x7f0d100cf700 msgr2=0x7f0d100b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:03.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.862+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 shutdown_connections 2024-01-16T15:07:03.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:03.862+0000 7f0d15800640 1 -- 172.21.15.141:0/446446094 wait complete. 2024-01-16T15:07:03.873 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596318 2024-01-16T15:07:04.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.184+0000 7ff9fee00640 1 Processor -- start 2024-01-16T15:07:04.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.185+0000 7ff9fee00640 1 -- start start 2024-01-16T15:07:04.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.187+0000 7ff9fee00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff9f80cf9a0 0x7ff9f8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.187+0000 7ff9fee00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f8071690 con 0x7ff9f814b2c0 2024-01-16T15:07:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.187+0000 7ff9fee00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f8071970 con 0x7ff9f8070ea0 2024-01-16T15:07:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.187+0000 7ff9fee00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f814b6a0 con 0x7ff9f80cf9a0 2024-01-16T15:07:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.187+0000 7ff9f7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff9f80cf9a0 0x7ff9f8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.188+0000 7ff9f7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff9f80cf9a0 0x7ff9f8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33214/0 (socket says 172.21.15.141:33214) 2024-01-16T15:07:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.188+0000 7ff9f7400640 1 -- 172.21.15.141:0/2476098046 learned_addr learned my addr 172.21.15.141:0/2476098046 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:04.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.188+0000 7ff9f7400640 1 -- 172.21.15.141:0/2476098046 >> v1:172.21.15.188:6789/0 conn(0x7ff9f8070ea0 legacy=0x7ff9f80cc160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:07:04.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.188+0000 7ff9f7400640 1 -- 172.21.15.141:0/2476098046 >> v1:172.21.15.141:6789/0 conn(0x7ff9f814b2c0 legacy=0x7ff9f80cf290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:07:04.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.189+0000 7ff9f7400640 1 -- 172.21.15.141:0/2476098046 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f8071c50 con 0x7ff9f80cf9a0 2024-01-16T15:07:04.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.191+0000 7ff9f7400640 1 --2- 172.21.15.141:0/2476098046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f80cf9a0 0x7ff9f8070960 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009870 tx=0x7ff9e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c75c7d0df60462 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.193+0000 7ff9f6a00640 1 -- 172.21.15.141:0/2476098046 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0031870 con 0x7ff9f80cf9a0 2024-01-16T15:07:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.193+0000 7ff9f6a00640 1 -- 172.21.15.141:0/2476098046 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e00319d0 con 0x7ff9f80cf9a0 2024-01-16T15:07:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.193+0000 7ff9f6a00640 1 -- 172.21.15.141:0/2476098046 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0031d60 con 0x7ff9f80cf9a0 2024-01-16T15:07:04.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.194+0000 7ff9fee00640 1 -- 172.21.15.141:0/2476098046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f80cf9a0 msgr2=0x7ff9f8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.194+0000 7ff9fee00640 1 --2- 172.21.15.141:0/2476098046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f80cf9a0 0x7ff9f8070960 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009870 tx=0x7ff9e002b910 comp rx=0 tx=0).stop 2024-01-16T15:07:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.194+0000 7ff9fee00640 1 -- 172.21.15.141:0/2476098046 shutdown_connections 2024-01-16T15:07:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.194+0000 7ff9fee00640 1 --2- 172.21.15.141:0/2476098046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f80cf9a0 0x7ff9f8070960 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.194+0000 7ff9fee00640 1 -- 172.21.15.141:0/2476098046 >> 172.21.15.141:0/2476098046 conn(0x7ff9f806a800 msgr2=0x7ff9f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.195+0000 7ff9fee00640 1 -- 172.21.15.141:0/2476098046 shutdown_connections 2024-01-16T15:07:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.195+0000 7ff9fee00640 1 -- 172.21.15.141:0/2476098046 wait complete. 2024-01-16T15:07:04.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.196+0000 7ff9fee00640 1 Processor -- start 2024-01-16T15:07:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.197+0000 7ff9fee00640 1 -- start start 2024-01-16T15:07:04.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9fee00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9f8070ea0 0x7ff9f8162850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9fee00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f814b2c0 0x7ff9f8145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9fee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9f8157e40 0x7ff9f8163ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9fee00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8145c70 con 0x7ff9f814b2c0 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9fee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f81451f0 con 0x7ff9f8157e40 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9fee00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f81582f0 con 0x7ff9f8070ea0 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9f7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f814b2c0 0x7ff9f8145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9f7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f814b2c0 0x7ff9f8145730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33222/0 (socket says 172.21.15.141:33222) 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.198+0000 7ff9f7400640 1 -- 172.21.15.141:0/1218710380 learned_addr learned my addr 172.21.15.141:0/1218710380 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9f7400640 1 -- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9f8070ea0 msgr2=0x7ff9f8162850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9f7400640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9f8070ea0 0x7ff9f8162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9f7400640 1 -- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9f8157e40 msgr2=0x7ff9f8163ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9f7400640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9f8157e40 0x7ff9f8163ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9f7400640 1 -- 172.21.15.141:0/1218710380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e00094d0 con 0x7ff9f814b2c0 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9f7400640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f814b2c0 0x7ff9f8145730 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0000c00 tx=0x7ff9e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9efe00640 1 -- 172.21.15.141:0/1218710380 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0004230 con 0x7ff9f814b2c0 2024-01-16T15:07:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9efe00640 1 -- 172.21.15.141:0/1218710380 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e0004390 con 0x7ff9f814b2c0 2024-01-16T15:07:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9efe00640 1 -- 172.21.15.141:0/1218710380 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e003ab40 con 0x7ff9f814b2c0 2024-01-16T15:07:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.199+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f8162ef0 con 0x7ff9f814b2c0 2024-01-16T15:07:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.200+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f81634b0 con 0x7ff9f814b2c0 2024-01-16T15:07:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.201+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f80675b0 con 0x7ff9f814b2c0 2024-01-16T15:07:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.202+0000 7ff9efe00640 1 -- 172.21.15.141:0/1218710380 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff9e0042070 con 0x7ff9f814b2c0 2024-01-16T15:07:04.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.203+0000 7ff9efe00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9d006bec0 0x7ff9d006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.203+0000 7ff9efe00640 1 -- 172.21.15.141:0/1218710380 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff9e002d040 con 0x7ff9f814b2c0 2024-01-16T15:07:04.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.213+0000 7ff9efe00640 1 -- 172.21.15.141:0/1218710380 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9e0076f80 con 0x7ff9f814b2c0 2024-01-16T15:07:04.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.222+0000 7ff9f7e00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9d006bec0 0x7ff9d006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.268+0000 7ff9f7e00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9d006bec0 0x7ff9d006e380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8009650 tx=0x7ff9e8006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:04.571+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596318", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9f81461b0 con 0x7ff9d006bec0 2024-01-16T15:07:04.889 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:04.896 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:05.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.354+0000 7ff9efe00640 1 -- 172.21.15.141:0/1218710380 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff9f81461b0 con 0x7ff9d006bec0 2024-01-16T15:07:05.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9d006bec0 msgr2=0x7ff9d006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9d006bec0 0x7ff9d006e380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8009650 tx=0x7ff9e8006ba0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f814b2c0 msgr2=0x7ff9f8145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f814b2c0 0x7ff9f8145730 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0000c00 tx=0x7ff9e00046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 shutdown_connections 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9d006bec0 0x7ff9d006e380 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9f8157e40 0x7ff9f8163ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9f814b2c0 0x7ff9f8145730 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 --2- 172.21.15.141:0/1218710380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9f8070ea0 0x7ff9f8162850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 >> 172.21.15.141:0/1218710380 conn(0x7ff9f806a800 msgr2=0x7ff9f80b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:05.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 shutdown_connections 2024-01-16T15:07:05.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.357+0000 7ff9fee00640 1 -- 172.21.15.141:0/1218710380 wait complete. 2024-01-16T15:07:05.373 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596319 2024-01-16T15:07:05.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.663+0000 7fe067800640 1 Processor -- start 2024-01-16T15:07:05.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.663+0000 7fe067800640 1 -- start start 2024-01-16T15:07:05.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.665+0000 7fe067800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600ce8f0 0x7fe0600ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.665+0000 7fe067800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe0600b2a00 0x7fe0600b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.666+0000 7fe067800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0600d0de0 con 0x7fe060160f80 2024-01-16T15:07:05.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.666+0000 7fe067800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0600d10c0 con 0x7fe0600b2a00 2024-01-16T15:07:05.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.666+0000 7fe067800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600d13a0 con 0x7fe0600ce8f0 2024-01-16T15:07:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.666+0000 7fe064e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe060160f80 0x7fe060161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38124/0 (socket says 172.21.15.141:38124) 2024-01-16T15:07:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.666+0000 7fe064e00640 1 -- 172.21.15.141:0/1223726933 learned_addr learned my addr 172.21.15.141:0/1223726933 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.667+0000 7fe057c00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600ce8f0 0x7fe0600ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.667+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981428407 0 0) 0x7fe0600d0de0 con 0x7fe060160f80 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.667+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe040003520 con 0x7fe060160f80 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3658782717 0 0) 0x7fe040003520 con 0x7fe060160f80 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600ce8f0 msgr2=0x7fe0600ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600ce8f0 0x7fe0600ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 >> v2:172.21.15.188:3300/0 conn(0x7fe0600b2a00 msgr2=0x7fe0600b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 --2- 172.21.15.141:0/1223726933 >> v2:172.21.15.188:3300/0 conn(0x7fe0600b2a00 0x7fe0600b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600d1680 con 0x7fe060160f80 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe0500019f0 con 0x7fe060160f80 2024-01-16T15:07:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 >> v1:172.21.15.141:6789/0 conn(0x7fe060160f80 legacy=0x7fe060161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600ce8f0 0x7fe040003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe040003ef0 0x7fe0400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe040004d00 0x7fe040005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600d10c0 con 0x7fe040003ef0 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600d13a0 con 0x7fe040004d00 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04000bab0 con 0x7fe0600ce8f0 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.668+0000 7fe057c00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe040003ef0 0x7fe0400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.669+0000 7fe057c00640 1 -- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600ce8f0 msgr2=0x7fe040003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.669+0000 7fe057c00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600ce8f0 0x7fe040003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.669+0000 7fe057c00640 1 -- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe040004d00 msgr2=0x7fe040005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.669+0000 7fe057c00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe040004d00 0x7fe040005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.669+0000 7fe057c00640 1 -- 172.21.15.141:0/1223726933 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe040003520 con 0x7fe040003ef0 2024-01-16T15:07:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.671+0000 7fe057c00640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe040003ef0 0x7fe0400047c0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fe048009ac0 tx=0x7fe0400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0376c28ac8a7b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.671+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04802c710 con 0x7fe040003ef0 2024-01-16T15:07:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.671+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04802c870 con 0x7fe040003ef0 2024-01-16T15:07:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.671+0000 7fe05fe00640 1 -- 172.21.15.141:0/1223726933 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04802cc00 con 0x7fe040003ef0 2024-01-16T15:07:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.672+0000 7fe067800640 1 -- 172.21.15.141:0/1223726933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe040003ef0 msgr2=0x7fe0400047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.672+0000 7fe067800640 1 --2- 172.21.15.141:0/1223726933 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe040003ef0 0x7fe0400047c0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fe048009ac0 tx=0x7fe0400057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.672+0000 7fe065800640 1 -- 172.21.15.141:0/1223726933 reap_dead start 2024-01-16T15:07:05.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.673+0000 7fe067800640 1 -- 172.21.15.141:0/1223726933 shutdown_connections 2024-01-16T15:07:05.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.673+0000 7fe067800640 1 -- 172.21.15.141:0/1223726933 >> 172.21.15.141:0/1223726933 conn(0x7fe06009c550 msgr2=0x7fe0600d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:05.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.674+0000 7fe067800640 1 -- 172.21.15.141:0/1223726933 shutdown_connections 2024-01-16T15:07:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.674+0000 7fe067800640 1 -- 172.21.15.141:0/1223726933 wait complete. 2024-01-16T15:07:05.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.679+0000 7fe067800640 1 Processor -- start 2024-01-16T15:07:05.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.680+0000 7fe067800640 1 -- start start 2024-01-16T15:07:05.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.681+0000 7fe067800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe040003ef0 0x7fe060152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.681+0000 7fe067800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe06006c350 0x7fe06006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.681+0000 7fe067800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600b0e90 0x7fe0600b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.681+0000 7fe067800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600c7a60 con 0x7fe06006c350 2024-01-16T15:07:05.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.681+0000 7fe067800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600b1850 con 0x7fe040003ef0 2024-01-16T15:07:05.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.681+0000 7fe067800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe060152ab0 con 0x7fe0600b0e90 2024-01-16T15:07:05.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.681+0000 7fe057c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe06006c350 0x7fe06006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:05.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.682+0000 7fe057c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe06006c350 0x7fe06006c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33248/0 (socket says 172.21.15.141:33248) 2024-01-16T15:07:05.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.682+0000 7fe057c00640 1 -- 172.21.15.141:0/4245562706 learned_addr learned my addr 172.21.15.141:0/4245562706 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:05.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.682+0000 7fe065800640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600b0e90 0x7fe0600b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:05.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.683+0000 7fe057c00640 1 -- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600b0e90 msgr2=0x7fe0600b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:05.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.683+0000 7fe057c00640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600b0e90 0x7fe0600b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.683+0000 7fe057c00640 1 -- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe040003ef0 msgr2=0x7fe060152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:05.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.683+0000 7fe057c00640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe040003ef0 0x7fe060152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:05.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.683+0000 7fe057c00640 1 -- 172.21.15.141:0/4245562706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600c6de0 con 0x7fe06006c350 2024-01-16T15:07:05.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.684+0000 7fe057c00640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe06006c350 0x7fe06006c7d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fe04802c2b0 tx=0x7fe048004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:05.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.685+0000 7fe05d600640 1 -- 172.21.15.141:0/4245562706 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048003810 con 0x7fe06006c350 2024-01-16T15:07:05.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.685+0000 7fe05d600640 1 -- 172.21.15.141:0/4245562706 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04802de90 con 0x7fe06006c350 2024-01-16T15:07:05.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.685+0000 7fe05d600640 1 -- 172.21.15.141:0/4245562706 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048003b90 con 0x7fe06006c350 2024-01-16T15:07:05.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.685+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0600c70c0 con 0x7fe06006c350 2024-01-16T15:07:05.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.685+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0600c7680 con 0x7fe06006c350 2024-01-16T15:07:05.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.687+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0600c7810 con 0x7fe06006c350 2024-01-16T15:07:05.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.688+0000 7fe05d600640 1 -- 172.21.15.141:0/4245562706 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe048003cf0 con 0x7fe06006c350 2024-01-16T15:07:05.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.689+0000 7fe05d600640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe03806be90 0x7fe03806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:05.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.689+0000 7fe05d600640 1 -- 172.21.15.141:0/4245562706 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe0480aa7f0 con 0x7fe06006c350 2024-01-16T15:07:05.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.691+0000 7fe064e00640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe03806be90 0x7fe03806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:05.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.695+0000 7fe05d600640 1 -- 172.21.15.141:0/4245562706 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe048077080 con 0x7fe06006c350 2024-01-16T15:07:05.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.727+0000 7fe064e00640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe03806be90 0x7fe03806e350 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe050007fd0 tx=0x7fe0500018c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:05.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:05.991+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596319", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0600d1c40 con 0x7fe03806be90 2024-01-16T15:07:07.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.583+0000 7fe05d600640 1 -- 172.21.15.141:0/4245562706 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe050002070 con 0x7fe03806be90 2024-01-16T15:07:07.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe03806be90 msgr2=0x7fe03806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe03806be90 0x7fe03806e350 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe050007fd0 tx=0x7fe0500018c0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe06006c350 msgr2=0x7fe06006c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe06006c350 0x7fe06006c7d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fe04802c2b0 tx=0x7fe048004730 comp rx=0 tx=0).stop 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 shutdown_connections 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe03806be90 0x7fe03806e350 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe0600b0e90 0x7fe0600b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe06006c350 0x7fe06006c7d0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 --2- 172.21.15.141:0/4245562706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe040003ef0 0x7fe060152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 >> 172.21.15.141:0/4245562706 conn(0x7fe06009c550 msgr2=0x7fe060161360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 shutdown_connections 2024-01-16T15:07:07.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.586+0000 7fe067800640 1 -- 172.21.15.141:0/4245562706 wait complete. 2024-01-16T15:07:07.585 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone596320 2024-01-16T15:07:07.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.820+0000 7f2cc4a00640 1 Processor -- start 2024-01-16T15:07:07.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.821+0000 7f2cc4a00640 1 -- start start 2024-01-16T15:07:07.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.822+0000 7f2cc4a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2cc0071410 0x7f2cc0146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.822+0000 7f2cc4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc006c7c0 0x7f2cc00717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.822+0000 7f2cc4a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc006cc20 con 0x7f2cc0071d30 2024-01-16T15:07:07.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.822+0000 7f2cc4a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc006cec0 con 0x7f2cc0071410 2024-01-16T15:07:07.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.822+0000 7f2cc4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc006d1a0 con 0x7f2cc006c7c0 2024-01-16T15:07:07.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbfe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2cc0071d30 0x7f2cc00b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38128/0 (socket says 172.21.15.141:38128) 2024-01-16T15:07:07.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbfe00640 1 -- 172.21.15.141:0/1024963498 learned_addr learned my addr 172.21.15.141:0/1024963498 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1024963498 >> v2:172.21.15.188:3300/0 conn(0x7f2cc0071410 0x7f2cc0146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cb6800640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc006c7c0 0x7f2cc00717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273219693 0 0) 0x7f2cc006cc20 con 0x7f2cc0071d30 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0003520 con 0x7f2cc0071d30 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1348455081 0 0) 0x7f2ca0003520 con 0x7f2cc0071d30 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc006c7c0 msgr2=0x7f2cc00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc006c7c0 0x7f2cc00717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 >> v2:172.21.15.188:3300/0 conn(0x7f2cc0071410 msgr2=0x7f2cc0146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 --2- 172.21.15.141:0/1024963498 >> v2:172.21.15.188:3300/0 conn(0x7f2cc0071410 0x7f2cc0146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc006d480 con 0x7f2cc0071d30 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2cb80019f0 con 0x7f2cc0071d30 2024-01-16T15:07:07.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.823+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 >> v1:172.21.15.141:6789/0 conn(0x7f2cc0071d30 legacy=0x7f2cc00b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbea00640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2ca0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbea00640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc006c7c0 0x7f2ca00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbea00640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ca0004d00 0x7f2ca0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc006cec0 con 0x7f2ca0003970 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc006d1a0 con 0x7f2ca0004d00 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca000b980 con 0x7f2cc006c7c0 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2ca0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbf400640 1 -- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc006c7c0 msgr2=0x7f2ca00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc006c7c0 0x7f2ca00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbf400640 1 -- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ca0004d00 msgr2=0x7f2ca0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ca0004d00 0x7f2ca0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.824+0000 7f2cbf400640 1 -- 172.21.15.141:0/1024963498 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0003520 con 0x7f2ca0003970 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.825+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2ca0003d90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009af0 tx=0x7f2ca00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d767ee66730cca92 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb002c710 con 0x7f2ca0003970 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb002c870 con 0x7f2ca0003970 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cbea00640 1 -- 172.21.15.141:0/1024963498 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb002cc00 con 0x7f2ca0003970 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1024963498 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 msgr2=0x7f2ca0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cc4a00640 1 --2- 172.21.15.141:0/1024963498 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2ca0003d90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009af0 tx=0x7f2ca00057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cbfe00640 1 -- 172.21.15.141:0/1024963498 reap_dead start 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1024963498 shutdown_connections 2024-01-16T15:07:07.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.826+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1024963498 >> 172.21.15.141:0/1024963498 conn(0x7f2cc006a800 msgr2=0x7f2cc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:07.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.827+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1024963498 shutdown_connections 2024-01-16T15:07:07.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.827+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1024963498 wait complete. 2024-01-16T15:07:07.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.828+0000 7f2cc4a00640 1 Processor -- start 2024-01-16T15:07:07.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.829+0000 7f2cc4a00640 1 -- start start 2024-01-16T15:07:07.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.829+0000 7f2cc4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2cc00c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cc4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cc00cc7d0 0x7f2cc013e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cc4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc00ce050 0x7f2cc00ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cc4a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00ce9f0 con 0x7f2ca0003970 2024-01-16T15:07:07.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cc4a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00cc340 con 0x7f2cc00cc7d0 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cc4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00c65c0 con 0x7f2cc00ce050 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2cc00c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2cc00c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33274/0 (socket says 172.21.15.141:33274) 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 -- 172.21.15.141:0/1215397974 learned_addr learned my addr 172.21.15.141:0/1215397974 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 -- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc00ce050 msgr2=0x7f2cc00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc00ce050 0x7f2cc00ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 -- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cc00cc7d0 msgr2=0x7f2cc013e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cc00cc7d0 0x7f2cc013e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.830+0000 7f2cbf400640 1 -- 172.21.15.141:0/1215397974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc00cb730 con 0x7f2ca0003970 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.831+0000 7f2cbf400640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2cc00c6080 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0037040 tx=0x7f2cb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.831+0000 7f2cb7e00640 1 -- 172.21.15.141:0/1215397974 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0003810 con 0x7f2ca0003970 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.831+0000 7f2cb7e00640 1 -- 172.21.15.141:0/1215397974 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb002de90 con 0x7f2ca0003970 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.831+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc00cba10 con 0x7f2ca0003970 2024-01-16T15:07:07.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.831+0000 7f2cb7e00640 1 -- 172.21.15.141:0/1215397974 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0003b70 con 0x7f2ca0003970 2024-01-16T15:07:07.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.831+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc00cbfd0 con 0x7f2ca0003970 2024-01-16T15:07:07.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.832+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc0067450 con 0x7f2ca0003970 2024-01-16T15:07:07.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.833+0000 7f2cb7e00640 1 -- 172.21.15.141:0/1215397974 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2cb0003cd0 con 0x7f2ca0003970 2024-01-16T15:07:07.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.834+0000 7f2cb7e00640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2c9806be90 0x7f2c9806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.834+0000 7f2cb7e00640 1 -- 172.21.15.141:0/1215397974 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2cb00aa7b0 con 0x7f2ca0003970 2024-01-16T15:07:07.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.835+0000 7f2cb6800640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2c9806be90 0x7f2c9806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.836+0000 7f2cb6800640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2c9806be90 0x7f2c9806e350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006a90 tx=0x7f2ca8006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:07.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:07.839+0000 7f2cb7e00640 1 -- 172.21.15.141:0/1215397974 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cb0076e30 con 0x7f2ca0003970 2024-01-16T15:07:08.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.068+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone596320", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc0072150 con 0x7f2c9806be90 2024-01-16T15:07:08.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.236+0000 7f2cb7e00640 1 -- 172.21.15.141:0/1215397974 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2ca80096c0 con 0x7f2c9806be90 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.239+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2c9806be90 msgr2=0x7f2c9806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.239+0000 7f2cc4a00640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2c9806be90 0x7f2c9806e350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006a90 tx=0x7f2ca8006a20 comp rx=0 tx=0).stop 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.239+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 msgr2=0x7f2cc00c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.239+0000 7f2cc4a00640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2cc00c6080 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0037040 tx=0x7f2cb0004730 comp rx=0 tx=0).stop 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 shutdown_connections 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2c9806be90 0x7f2c9806e350 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cc00ce050 0x7f2cc00ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cc00cc7d0 0x7f2cc013e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 --2- 172.21.15.141:0/1215397974 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ca0003970 0x7f2cc00c6080 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 >> 172.21.15.141:0/1215397974 conn(0x7f2cc006a800 msgr2=0x7f2cc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 shutdown_connections 2024-01-16T15:07:08.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.240+0000 7f2cc4a00640 1 -- 172.21.15.141:0/1215397974 wait complete. 2024-01-16T15:07:08.238 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:07:08.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.460+0000 7f7ddd600640 1 Processor -- start 2024-01-16T15:07:08.438 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.460+0000 7f7ddd600640 1 -- start start 2024-01-16T15:07:08.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7ddd600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7dd813df80 0x7f7dd815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7ddd600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 0x7f7dd80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7ddd600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd806c470 con 0x7f7dd815f9c0 2024-01-16T15:07:08.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7ddd600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd806c750 con 0x7f7dd813df80 2024-01-16T15:07:08.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7ddd600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd806ca30 con 0x7f7dd813e360 2024-01-16T15:07:08.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7dd6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7dd813df80 0x7f7dd815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7dd6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7dd813df80 0x7f7dd815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33288/0 (socket says 172.21.15.141:33288) 2024-01-16T15:07:08.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.461+0000 7f7dd6a00640 1 -- 172.21.15.141:0/276898896 learned_addr learned my addr 172.21.15.141:0/276898896 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:08.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.462+0000 7f7dd6a00640 1 -- 172.21.15.141:0/276898896 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 msgr2=0x7f7dd80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:08.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.462+0000 7f7dd6a00640 1 --2- 172.21.15.141:0/276898896 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 0x7f7dd80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.462+0000 7f7dd6a00640 1 -- 172.21.15.141:0/276898896 >> v1:172.21.15.188:6789/0 conn(0x7f7dd815f9c0 legacy=0x7f7dd81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:08.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.462+0000 7f7dd6a00640 1 -- 172.21.15.141:0/276898896 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd806cd10 con 0x7f7dd813df80 2024-01-16T15:07:08.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.463+0000 7f7dd6a00640 1 --2- 172.21.15.141:0/276898896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd813df80 0x7f7dd815bee0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009870 tx=0x7f7dc802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321de1e3c73d389 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.463+0000 7f7dd5600640 1 -- 172.21.15.141:0/276898896 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8031640 con 0x7f7dd813df80 2024-01-16T15:07:08.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.463+0000 7f7dd5600640 1 -- 172.21.15.141:0/276898896 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc80317a0 con 0x7f7dd813df80 2024-01-16T15:07:08.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.463+0000 7f7dd5600640 1 -- 172.21.15.141:0/276898896 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8031b30 con 0x7f7dd813df80 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 -- 172.21.15.141:0/276898896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd813df80 msgr2=0x7f7dd815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 --2- 172.21.15.141:0/276898896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd813df80 0x7f7dd815bee0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009870 tx=0x7f7dc802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 -- 172.21.15.141:0/276898896 shutdown_connections 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 --2- 172.21.15.141:0/276898896 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 0x7f7dd80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 --2- 172.21.15.141:0/276898896 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd813df80 0x7f7dd815bee0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 -- 172.21.15.141:0/276898896 >> 172.21.15.141:0/276898896 conn(0x7f7dd806a800 msgr2=0x7f7dd806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 -- 172.21.15.141:0/276898896 shutdown_connections 2024-01-16T15:07:08.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.464+0000 7f7ddd600640 1 -- 172.21.15.141:0/276898896 wait complete. 2024-01-16T15:07:08.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.466+0000 7f7ddd600640 1 Processor -- start 2024-01-16T15:07:08.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.466+0000 7f7ddd600640 1 -- start start 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7ddd600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7dd813df80 0x7f7dd8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7ddd600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 0x7f7dd8071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7ddd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd815f9c0 0x7f7dd80b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7ddd600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd813af10 con 0x7f7dd815f9c0 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7ddd600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd813d3a0 con 0x7f7dd813df80 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7ddd600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd813b120 con 0x7f7dd813e360 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7dd7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd815f9c0 0x7f7dd80b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7dd7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd815f9c0 0x7f7dd80b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33296/0 (socket says 172.21.15.141:33296) 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7dd7400640 1 -- 172.21.15.141:0/2859658628 learned_addr learned my addr 172.21.15.141:0/2859658628 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7dd6000640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 0x7f7dd8071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.467+0000 7f7dd6a00640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7dd813df80 0x7f7dd8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dd7400640 1 -- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 msgr2=0x7f7dd8071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dd7400640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 0x7f7dd8071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dd7400640 1 -- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7dd813df80 msgr2=0x7f7dd8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dd7400640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7dd813df80 0x7f7dd8070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dd7400640 1 -- 172.21.15.141:0/2859658628 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc80094d0 con 0x7f7dd815f9c0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dd6a00640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7dd813df80 0x7f7dd8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dd7400640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd815f9c0 0x7f7dd80b2cf0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000c780 tx=0x7f7dd000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dcea00640 1 -- 172.21.15.141:0/2859658628 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0003d80 con 0x7f7dd815f9c0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dcea00640 1 -- 172.21.15.141:0/2859658628 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd000ce90 con 0x7f7dd815f9c0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd813b400 con 0x7f7dd815f9c0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7dcea00640 1 -- 172.21.15.141:0/2859658628 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00052b0 con 0x7f7dd815f9c0 2024-01-16T15:07:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.468+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd8153510 con 0x7f7dd815f9c0 2024-01-16T15:07:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.469+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd813b920 con 0x7f7dd815f9c0 2024-01-16T15:07:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.470+0000 7f7dcea00640 1 -- 172.21.15.141:0/2859658628 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f7dd0005410 con 0x7f7dd815f9c0 2024-01-16T15:07:08.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.472+0000 7f7dcea00640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7db006be90 0x7f7db006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.472+0000 7f7dcea00640 1 -- 172.21.15.141:0/2859658628 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7dd0088090 con 0x7f7dd815f9c0 2024-01-16T15:07:08.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.472+0000 7f7dd6a00640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7db006be90 0x7f7db006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.473+0000 7f7dd6a00640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7db006be90 0x7f7db006e350 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8036000 tx=0x7f7dc803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.476+0000 7f7dcea00640 1 -- 172.21.15.141:0/2859658628 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7dd813b920 con 0x7f7dd815f9c0 2024-01-16T15:07:08.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.737+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7dd813b920 con 0x7f7dd815f9c0 2024-01-16T15:07:08.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.738+0000 7f7dcea00640 1 -- 172.21.15.141:0/2859658628 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2394 (secure 0 0 0) 0x7f7dd813b920 con 0x7f7dd815f9c0 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:07:08.717 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 621, 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 75, 2024-01-16T15:07:08.718 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:07:08.719 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 3701, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 15328172336, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 30777425920, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 742316687360, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 466072593, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 464732826, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 112, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 126 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 47, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 5, 2024-01-16T15:07:08.720 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 5, 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 7119 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:07:08.721 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 13, 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:06:12.684592+0000", 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": { 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "0417ea54-fe71-4030-bf3c-fd8052c20229": { 2024-01-16T15:07:08.722 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "18 ongoing+pending cloning jobs; Avg progress = 22.222% (49s)\n [======......................] (remaining: 2m)", 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 0.2222222238779068, 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: "b5e58f12-4ecf-4f65-b662-f396825cc91a": { 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: "message": "2 ongoing cloning jobs; Avg progress = 100.0% (49s)\n [============================] ", 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: "progress": 1, 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: "add_to_ceph_s": true 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:07:08.723 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.741+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7db006be90 msgr2=0x7f7db006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.741+0000 7f7ddd600640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7db006be90 0x7f7db006e350 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8036000 tx=0x7f7dc803c040 comp rx=0 tx=0).stop 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.741+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd815f9c0 msgr2=0x7f7dd80b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.741+0000 7f7ddd600640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd815f9c0 0x7f7dd80b2cf0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000c780 tx=0x7f7dd000cc40 comp rx=0 tx=0).stop 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 shutdown_connections 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7db006be90 0x7f7db006e350 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7dd815f9c0 0x7f7dd80b2cf0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7dd813e360 0x7f7dd8071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 --2- 172.21.15.141:0/2859658628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7dd813df80 0x7f7dd8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 >> 172.21.15.141:0/2859658628 conn(0x7f7dd806a800 msgr2=0x7f7dd8070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:08.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 shutdown_connections 2024-01-16T15:07:08.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.742+0000 7f7ddd600640 1 -- 172.21.15.141:0/2859658628 wait complete. 2024-01-16T15:07:08.739 INFO:tasks.cephfs.test_volumes:new_onpen_p is less than onpen_p. 2024-01-16T15:07:08.739 INFO:tasks.cephfs.test_volumes:new_onpen_p = 0.2222222238779068; onpen_p = 0.2857142984867096 2024-01-16T15:07:08.739 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59631 2024-01-16T15:07:08.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.976+0000 7f339d400640 1 Processor -- start 2024-01-16T15:07:08.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.976+0000 7f339d400640 1 -- start start 2024-01-16T15:07:08.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.977+0000 7f339d400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f339813cf10 0x7f339813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.977+0000 7f339d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 0x7f33980b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.977+0000 7f339d400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33980ce440 con 0x7f339813fae0 2024-01-16T15:07:08.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.977+0000 7f339d400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339813ff90 con 0x7f339813cf10 2024-01-16T15:07:08.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.977+0000 7f339d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33980ce260 con 0x7f33980cddf0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f338f200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f339813fae0 0x7f33980cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38132/0 (socket says 172.21.15.141:38132) 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f338f200640 1 -- 172.21.15.141:0/3533717250 learned_addr learned my addr 172.21.15.141:0/3533717250 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f3397e00640 1 --2- 172.21.15.141:0/3533717250 >> v2:172.21.15.188:3300/0 conn(0x7f339813cf10 0x7f339813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f339ca00640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 0x7f33980b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304173944 0 0) 0x7f33980ce440 con 0x7f339813fae0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3378003520 con 0x7f339813fae0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692860027 0 0) 0x7f3378003520 con 0x7f339813fae0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.978+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 msgr2=0x7f33980b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 0x7f33980b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 >> v2:172.21.15.188:3300/0 conn(0x7f339813cf10 msgr2=0x7f339813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 --2- 172.21.15.141:0/3533717250 >> v2:172.21.15.188:3300/0 conn(0x7f339813cf10 0x7f339813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33980ce720 con 0x7f339813fae0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3380001670 con 0x7f339813fae0 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397e00640 1 --2- 172.21.15.141:0/3533717250 >> v2:172.21.15.188:3300/0 conn(0x7f339813cf10 0x7f339813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:08.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f339ca00640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 0x7f33980b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 >> v1:172.21.15.141:6789/0 conn(0x7f339813fae0 legacy=0x7f33980cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3378003970 0x7f3378003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33780047c0 0x7f3378004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 0x7f3378005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339813ff90 con 0x7f33780047c0 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33980ce260 con 0x7f3378003970 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337800ba00 con 0x7f33980cddf0 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f338f200640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33780047c0 0x7f3378004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f3397e00640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3378003970 0x7f3378003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f339ca00640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 0x7f3378005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.979+0000 7f338f200640 1 -- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 msgr2=0x7f3378005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.980+0000 7f338f200640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33980cddf0 0x7f3378005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.980+0000 7f338f200640 1 -- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3378003970 msgr2=0x7f3378003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.980+0000 7f338f200640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3378003970 0x7f3378003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.980+0000 7f338f200640 1 -- 172.21.15.141:0/3533717250 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3378003520 con 0x7f33780047c0 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.981+0000 7f338f200640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33780047c0 0x7f3378004c40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f338000c240 tx=0x7f33780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63d37e92151c02f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.981+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338002cb00 con 0x7f33780047c0 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.981+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338002cc60 con 0x7f33780047c0 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.981+0000 7f3397400640 1 -- 172.21.15.141:0/3533717250 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380005af0 con 0x7f33780047c0 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.981+0000 7f339d400640 1 -- 172.21.15.141:0/3533717250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33780047c0 msgr2=0x7f3378004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.981+0000 7f339d400640 1 --2- 172.21.15.141:0/3533717250 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33780047c0 0x7f3378004c40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f338000c240 tx=0x7f33780057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.982+0000 7f339ca00640 1 -- 172.21.15.141:0/3533717250 reap_dead start 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.982+0000 7f339d400640 1 -- 172.21.15.141:0/3533717250 shutdown_connections 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.982+0000 7f339d400640 1 -- 172.21.15.141:0/3533717250 >> 172.21.15.141:0/3533717250 conn(0x7f339806a7d0 msgr2=0x7f339806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.982+0000 7f339d400640 1 -- 172.21.15.141:0/3533717250 shutdown_connections 2024-01-16T15:07:08.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.982+0000 7f339d400640 1 -- 172.21.15.141:0/3533717250 wait complete. 2024-01-16T15:07:08.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.983+0000 7f339d400640 1 Processor -- start 2024-01-16T15:07:08.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.984+0000 7f339d400640 1 -- start start 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f339d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33780047c0 0x7f33981f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f339d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f339813a0b0 0x7f33981f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f339d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339813f010 0x7f339815a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f339d400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33981f31a0 con 0x7f339813a0b0 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f339d400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33980ad8a0 con 0x7f339813f010 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f339d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33980ae390 con 0x7f33780047c0 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f339813a0b0 0x7f33981f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f339813a0b0 0x7f33981f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33326/0 (socket says 172.21.15.141:33326) 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 -- 172.21.15.141:0/27258581 learned_addr learned my addr 172.21.15.141:0/27258581 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f3397e00640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33780047c0 0x7f33981f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f339ca00640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339813f010 0x7f339815a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 -- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33780047c0 msgr2=0x7f33981f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33780047c0 0x7f33981f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 -- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339813f010 msgr2=0x7f339815a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339813f010 0x7f339815a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.985+0000 7f338f200640 1 -- 172.21.15.141:0/27258581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33980adaf0 con 0x7f339813a0b0 2024-01-16T15:07:08.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.986+0000 7f338f200640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f339813a0b0 0x7f33981f3520 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f338000c360 tx=0x7f3380001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.986+0000 7f3394c00640 1 -- 172.21.15.141:0/27258581 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338003c040 con 0x7f339813a0b0 2024-01-16T15:07:08.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.986+0000 7f3394c00640 1 -- 172.21.15.141:0/27258581 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3380005c80 con 0x7f339813a0b0 2024-01-16T15:07:08.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.986+0000 7f3394c00640 1 -- 172.21.15.141:0/27258581 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380007070 con 0x7f339813a0b0 2024-01-16T15:07:08.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.986+0000 7f339d400640 1 -- 172.21.15.141:0/27258581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33980addd0 con 0x7f339813a0b0 2024-01-16T15:07:08.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.986+0000 7f339d400640 1 -- 172.21.15.141:0/27258581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33981f3a60 con 0x7f339813a0b0 2024-01-16T15:07:08.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.988+0000 7f339d400640 1 -- 172.21.15.141:0/27258581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33981f3c20 con 0x7f339813a0b0 2024-01-16T15:07:08.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.988+0000 7f3394c00640 1 -- 172.21.15.141:0/27258581 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3380032070 con 0x7f339813a0b0 2024-01-16T15:07:08.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.989+0000 7f3394c00640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f337006be80 0x7f337006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.990+0000 7f3394c00640 1 -- 172.21.15.141:0/27258581 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f33800aa2e0 con 0x7f339813a0b0 2024-01-16T15:07:08.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.990+0000 7f3397e00640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f337006be80 0x7f337006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.991+0000 7f3397e00640 1 --2- 172.21.15.141:0/27258581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f337006be80 0x7f337006e340 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3388006fd0 tx=0x7f3388008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:08.994+0000 7f3394c00640 1 -- 172.21.15.141:0/27258581 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33800769a0 con 0x7f339813a0b0 2024-01-16T15:07:09.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:09.220+0000 7f339d400640 1 -- 172.21.15.141:0/27258581 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59631", "target": ["mon-mgr", ""]}) v1 -- 0x7f33980b2070 con 0x7f337006be80 2024-01-16T15:07:09.220 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:09.242+0000 7f0f2bc00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:09.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:09.243+0000 7f3394c00640 1 -- 172.21.15.141:0/27258581 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f338800c040 con 0x7f337006be80 2024-01-16T15:07:09.221 INFO:teuthology.orchestra.run.smithi141.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:11.242 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59632 2024-01-16T15:07:11.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.483+0000 7f5127800640 1 Processor -- start 2024-01-16T15:07:11.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.483+0000 7f5127800640 1 -- start start 2024-01-16T15:07:11.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.484+0000 7f5127800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 0x7f51200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.484+0000 7f5127800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f512015e3b0 0x7f51200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.484+0000 7f5127800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51200cfc30 con 0x7f51200cf7d0 2024-01-16T15:07:11.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.485+0000 7f5127800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51200b1ee0 con 0x7f512015e3b0 2024-01-16T15:07:11.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.485+0000 7f5127800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51200b2780 con 0x7f5120143ae0 2024-01-16T15:07:11.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.485+0000 7f5124e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 0x7f51200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.485+0000 7f5117e00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f51200cf7d0 0x7f512015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:48224/0 (socket says 172.21.15.141:48224) 2024-01-16T15:07:11.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.485+0000 7f5117e00640 1 -- 172.21.15.141:0/3780259488 learned_addr learned my addr 172.21.15.141:0/3780259488 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:11.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.485+0000 7f5125800640 1 --2- 172.21.15.141:0/3780259488 >> v2:172.21.15.188:3300/0 conn(0x7f512015e3b0 0x7f51200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.486+0000 7f511fe00640 1 -- 172.21.15.141:0/3780259488 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981192147 0 0) 0x7f51200cfc30 con 0x7f51200cf7d0 2024-01-16T15:07:11.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.486+0000 7f511fe00640 1 -- 172.21.15.141:0/3780259488 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5100005730 con 0x7f51200cf7d0 2024-01-16T15:07:11.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.486+0000 7f5124e00640 1 -- 172.21.15.141:0/3780259488 >> v2:172.21.15.188:3300/0 conn(0x7f512015e3b0 msgr2=0x7f51200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:11.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.486+0000 7f5124e00640 1 --2- 172.21.15.141:0/3780259488 >> v2:172.21.15.188:3300/0 conn(0x7f512015e3b0 0x7f51200702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.486+0000 7f5124e00640 1 -- 172.21.15.141:0/3780259488 >> v1:172.21.15.188:6789/0 conn(0x7f51200cf7d0 legacy=0x7f512015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.486+0000 7f5124e00640 1 -- 172.21.15.141:0/3780259488 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512015e990 con 0x7f5120143ae0 2024-01-16T15:07:11.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.486+0000 7f5125800640 1 --2- 172.21.15.141:0/3780259488 >> v2:172.21.15.188:3300/0 conn(0x7f512015e3b0 0x7f51200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:11.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.488+0000 7f5124e00640 1 --2- 172.21.15.141:0/3780259488 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 0x7f51200cf290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f511002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c9983fdf313920 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.488+0000 7f511fe00640 1 -- 172.21.15.141:0/3780259488 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511002d040 con 0x7f5120143ae0 2024-01-16T15:07:11.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.488+0000 7f511fe00640 1 -- 172.21.15.141:0/3780259488 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5110031810 con 0x7f5120143ae0 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.488+0000 7f511fe00640 1 -- 172.21.15.141:0/3780259488 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110031ba0 con 0x7f5120143ae0 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 -- 172.21.15.141:0/3780259488 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 msgr2=0x7f51200cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 --2- 172.21.15.141:0/3780259488 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 0x7f51200cf290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f511002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 -- 172.21.15.141:0/3780259488 shutdown_connections 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 --2- 172.21.15.141:0/3780259488 >> v2:172.21.15.188:3300/0 conn(0x7f512015e3b0 0x7f51200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 --2- 172.21.15.141:0/3780259488 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 0x7f51200cf290 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 -- 172.21.15.141:0/3780259488 >> 172.21.15.141:0/3780259488 conn(0x7f512006a800 msgr2=0x7f512006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 -- 172.21.15.141:0/3780259488 shutdown_connections 2024-01-16T15:07:11.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.489+0000 7f5127800640 1 -- 172.21.15.141:0/3780259488 wait complete. 2024-01-16T15:07:11.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.491+0000 7f5127800640 1 Processor -- start 2024-01-16T15:07:11.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.491+0000 7f5127800640 1 -- start start 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5127800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51200cf7d0 0x7f51200ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5127800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 0x7f512012e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5127800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f512015e3b0 0x7f51200aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5127800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51200adab0 con 0x7f51200cf7d0 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5127800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51201311f0 con 0x7f512015e3b0 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5127800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51200ad3f0 con 0x7f5120143ae0 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51200cf7d0 0x7f51200ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51200cf7d0 0x7f51200ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33342/0 (socket says 172.21.15.141:33342) 2024-01-16T15:07:11.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 -- 172.21.15.141:0/2129943683 learned_addr learned my addr 172.21.15.141:0/2129943683 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 -- 172.21.15.141:0/2129943683 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 msgr2=0x7f512012e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 --2- 172.21.15.141:0/2129943683 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5120143ae0 0x7f512012e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 -- 172.21.15.141:0/2129943683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f512015e3b0 msgr2=0x7f51200aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 --2- 172.21.15.141:0/2129943683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f512015e3b0 0x7f51200aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.492+0000 7f5124e00640 1 -- 172.21.15.141:0/2129943683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5118009a70 con 0x7f51200cf7d0 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.493+0000 7f5124e00640 1 --2- 172.21.15.141:0/2129943683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51200cf7d0 0x7f51200ab7e0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f51100098c0 tx=0x7f51100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.493+0000 7f511d600640 1 -- 172.21.15.141:0/2129943683 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511003ac60 con 0x7f51200cf7d0 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.493+0000 7f511d600640 1 -- 172.21.15.141:0/2129943683 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511003ade0 con 0x7f51200cf7d0 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.493+0000 7f5127800640 1 -- 172.21.15.141:0/2129943683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5110009400 con 0x7f51200cf7d0 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.493+0000 7f511d600640 1 -- 172.21.15.141:0/2129943683 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110044640 con 0x7f51200cf7d0 2024-01-16T15:07:11.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.493+0000 7f5127800640 1 -- 172.21.15.141:0/2129943683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51200ad0a0 con 0x7f51200cf7d0 2024-01-16T15:07:11.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.494+0000 7f5127800640 1 -- 172.21.15.141:0/2129943683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51200adbf0 con 0x7f51200cf7d0 2024-01-16T15:07:11.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.495+0000 7f511d600640 1 -- 172.21.15.141:0/2129943683 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f511004d440 con 0x7f51200cf7d0 2024-01-16T15:07:11.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.496+0000 7f511d600640 1 --2- 172.21.15.141:0/2129943683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f50f806be90 0x7f50f806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.496+0000 7f511d600640 1 -- 172.21.15.141:0/2129943683 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51100aa9a0 con 0x7f51200cf7d0 2024-01-16T15:07:11.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.497+0000 7f5117e00640 1 --2- 172.21.15.141:0/2129943683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f50f806be90 0x7f50f806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.498+0000 7f5117e00640 1 --2- 172.21.15.141:0/2129943683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f50f806be90 0x7f50f806e350 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f51080011e0 tx=0x7f5108007910 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.500+0000 7f511d600640 1 -- 172.21.15.141:0/2129943683 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51200adbf0 con 0x7f51200cf7d0 2024-01-16T15:07:11.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.725+0000 7f5127800640 1 -- 172.21.15.141:0/2129943683 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59632", "target": ["mon-mgr", ""]}) v1 -- 0x7f51200b2410 con 0x7f50f806be90 2024-01-16T15:07:11.722 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:11.745+0000 7f0f2bc00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:11.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:11.745+0000 7f511d600640 1 -- 172.21.15.141:0/2129943683 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f5108005870 con 0x7f50f806be90 2024-01-16T15:07:11.723 INFO:teuthology.orchestra.run.smithi141.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:13.739 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59633 2024-01-16T15:07:13.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.985+0000 7f5783200640 1 Processor -- start 2024-01-16T15:07:13.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.985+0000 7f5783200640 1 -- start start 2024-01-16T15:07:13.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5783200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f578413cf10 0x7f578413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5783200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 0x7f57840cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5783200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57840ce440 con 0x7f57840cdc20 2024-01-16T15:07:13.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5783200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578413ff60 con 0x7f578413cf10 2024-01-16T15:07:13.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5783200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57840ce720 con 0x7f578413fae0 2024-01-16T15:07:13.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5782800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f57840cdc20 0x7f578415d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44992/0 (socket says 172.21.15.141:44992) 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5782800640 1 -- 172.21.15.141:0/2249779765 learned_addr learned my addr 172.21.15.141:0/2249779765 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5779200640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 0x7f57840cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5781e00640 1 --2- 172.21.15.141:0/2249779765 >> v2:172.21.15.188:3300/0 conn(0x7f578413cf10 0x7f578413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.987+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468659584 0 0) 0x7f57840ce440 con 0x7f57840cdc20 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5764003520 con 0x7f57840cdc20 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417278556 0 0) 0x7f5764003520 con 0x7f57840cdc20 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 msgr2=0x7f57840cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 0x7f57840cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 >> v2:172.21.15.188:3300/0 conn(0x7f578413cf10 msgr2=0x7f578413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 --2- 172.21.15.141:0/2249779765 >> v2:172.21.15.188:3300/0 conn(0x7f578413cf10 0x7f578413f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57840cea00 con 0x7f57840cdc20 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f577c0019f0 con 0x7f57840cdc20 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 >> v1:172.21.15.141:6789/0 conn(0x7f57840cdc20 legacy=0x7f578415d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5764003970 0x7f5764003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 0x7f57640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5764004d00 0x7f5764005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578413ff60 con 0x7f5764003970 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840ce720 con 0x7f5764004d00 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576400b980 con 0x7f578413fae0 2024-01-16T15:07:13.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.988+0000 7f5781e00640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5764003970 0x7f5764003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.989+0000 7f5779200640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 0x7f57640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.989+0000 7f5781e00640 1 -- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 msgr2=0x7f57640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.989+0000 7f5781e00640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 0x7f57640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.989+0000 7f5781e00640 1 -- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5764004d00 msgr2=0x7f5764005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.989+0000 7f5781e00640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5764004d00 0x7f5764005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.989+0000 7f5781e00640 1 -- 172.21.15.141:0/2249779765 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764003520 con 0x7f5764003970 2024-01-16T15:07:13.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.989+0000 7f5779200640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578413fae0 0x7f57640047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:13.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.990+0000 7f5781e00640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5764003970 0x7f5764003d90 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5774009af0 tx=0x7f57640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f34247bb3cbf8e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.990+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577402c710 con 0x7f5764003970 2024-01-16T15:07:13.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.990+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577402c870 con 0x7f5764003970 2024-01-16T15:07:13.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.990+0000 7f5781400640 1 -- 172.21.15.141:0/2249779765 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577402cc00 con 0x7f5764003970 2024-01-16T15:07:13.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.991+0000 7f5783200640 1 -- 172.21.15.141:0/2249779765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5764003970 msgr2=0x7f5764003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.991+0000 7f5783200640 1 --2- 172.21.15.141:0/2249779765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5764003970 0x7f5764003d90 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5774009af0 tx=0x7f57640057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.991+0000 7f5782800640 1 -- 172.21.15.141:0/2249779765 reap_dead start 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.991+0000 7f5783200640 1 -- 172.21.15.141:0/2249779765 shutdown_connections 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.991+0000 7f5783200640 1 -- 172.21.15.141:0/2249779765 >> 172.21.15.141:0/2249779765 conn(0x7f578406a7d0 msgr2=0x7f578406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.991+0000 7f5783200640 1 -- 172.21.15.141:0/2249779765 shutdown_connections 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.991+0000 7f5783200640 1 -- 172.21.15.141:0/2249779765 wait complete. 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.993+0000 7f5783200640 1 Processor -- start 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.993+0000 7f5783200640 1 -- start start 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5783200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5764003970 0x7f5784152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5783200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f578415cdc0 0x7f57841559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5783200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578415d240 0x7f5784155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5783200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578413ef40 con 0x7f578415cdc0 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5783200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578413e6a0 con 0x7f5764003970 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5783200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578413e980 con 0x7f578415d240 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f578415cdc0 0x7f57841559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f578415cdc0 0x7f57841559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58420/0 (socket says 172.21.15.141:58420) 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 -- 172.21.15.141:0/4152965322 learned_addr learned my addr 172.21.15.141:0/4152965322 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 -- 172.21.15.141:0/4152965322 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578415d240 msgr2=0x7f5784155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 --2- 172.21.15.141:0/4152965322 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f578415d240 0x7f5784155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 -- 172.21.15.141:0/4152965322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5764003970 msgr2=0x7f5784152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 --2- 172.21.15.141:0/4152965322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5764003970 0x7f5784152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.994+0000 7f5779200640 1 -- 172.21.15.141:0/4152965322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578413ec60 con 0x7f578415cdc0 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.995+0000 7f5779200640 1 --2- 172.21.15.141:0/4152965322 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f578415cdc0 0x7f57841559c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f576c0108b0 tx=0x7f576c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.995+0000 7f577aa00640 1 -- 172.21.15.141:0/4152965322 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c00eb60 con 0x7f578415cdc0 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.995+0000 7f577aa00640 1 -- 172.21.15.141:0/4152965322 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c00ecc0 con 0x7f578415cdc0 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.995+0000 7f5783200640 1 -- 172.21.15.141:0/4152965322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5784070290 con 0x7f578415cdc0 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.995+0000 7f577aa00640 1 -- 172.21.15.141:0/4152965322 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c01c400 con 0x7f578415cdc0 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.995+0000 7f5783200640 1 -- 172.21.15.141:0/4152965322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578413e2c0 con 0x7f578415cdc0 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.996+0000 7f5783200640 1 -- 172.21.15.141:0/4152965322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5784067450 con 0x7f578415cdc0 2024-01-16T15:07:13.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.997+0000 7f577aa00640 1 -- 172.21.15.141:0/4152965322 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f576c01c560 con 0x7f578415cdc0 2024-01-16T15:07:13.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.998+0000 7f577aa00640 1 --2- 172.21.15.141:0/4152965322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f575c06bec0 0x7f575c06e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.998+0000 7f577aa00640 1 -- 172.21.15.141:0/4152965322 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f576c088dd0 con 0x7f578415cdc0 2024-01-16T15:07:13.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:13.999+0000 7f5781e00640 1 --2- 172.21.15.141:0/4152965322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f575c06bec0 0x7f575c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:14.000+0000 7f5781e00640 1 --2- 172.21.15.141:0/4152965322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f575c06bec0 0x7f575c06e380 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5774036000 tx=0x7f577402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:14.003+0000 7f577aa00640 1 -- 172.21.15.141:0/4152965322 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f576c055540 con 0x7f578415cdc0 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:14.238+0000 7f5783200640 1 -- 172.21.15.141:0/4152965322 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59633", "target": ["mon-mgr", ""]}) v1 -- 0x7f578413eaf0 con 0x7f575c06bec0 2024-01-16T15:07:16.327 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:16.349+0000 7f0f2bc00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:16.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:16.349+0000 7f577aa00640 1 -- 172.21.15.141:0/4152965322 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f578413eaf0 con 0x7f575c06bec0 2024-01-16T15:07:16.327 INFO:teuthology.orchestra.run.smithi141.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:18.345 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59634 2024-01-16T15:07:18.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.590+0000 7fac80a00640 1 Processor -- start 2024-01-16T15:07:18.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.590+0000 7fac80a00640 1 -- start start 2024-01-16T15:07:18.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.592+0000 7fac80a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fac7c0adf20 0x7fac7c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.592+0000 7fac80a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0ccb50 0x7fac7c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.592+0000 7fac80a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c15f4e0 con 0x7fac7c15bf10 2024-01-16T15:07:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.592+0000 7fac80a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7c15f970 con 0x7fac7c0adf20 2024-01-16T15:07:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.592+0000 7fac80a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c071130 con 0x7fac7c0ccb50 2024-01-16T15:07:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.593+0000 7fac7a800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fac7c0adf20 0x7fac7c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.593+0000 7fac73e00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fac7c15bf10 0x7fac7c0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:50884/0 (socket says 172.21.15.141:50884) 2024-01-16T15:07:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.593+0000 7fac73e00640 1 -- 172.21.15.141:0/2236961186 learned_addr learned my addr 172.21.15.141:0/2236961186 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.593+0000 7fac7b200640 1 --2- 172.21.15.141:0/2236961186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0ccb50 0x7fac7c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.593+0000 7fac7b200640 1 -- 172.21.15.141:0/2236961186 >> v2:172.21.15.188:3300/0 conn(0x7fac7c0adf20 msgr2=0x7fac7c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.593+0000 7fac7b200640 1 --2- 172.21.15.141:0/2236961186 >> v2:172.21.15.188:3300/0 conn(0x7fac7c0adf20 0x7fac7c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.593+0000 7fac7b200640 1 -- 172.21.15.141:0/2236961186 >> v1:172.21.15.188:6789/0 conn(0x7fac7c15bf10 legacy=0x7fac7c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.594+0000 7fac7b200640 1 -- 172.21.15.141:0/2236961186 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c15c390 con 0x7fac7c0ccb50 2024-01-16T15:07:18.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.594+0000 7fac7a800640 1 --2- 172.21.15.141:0/2236961186 >> v2:172.21.15.188:3300/0 conn(0x7fac7c0adf20 0x7fac7c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:18.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.595+0000 7fac7b200640 1 --2- 172.21.15.141:0/2236961186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0ccb50 0x7fac7c143d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fac74009d90 tx=0x7fac7402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=320006a4f7691148 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.595+0000 7fac79e00640 1 -- 172.21.15.141:0/2236961186 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7402cbc0 con 0x7fac7c0ccb50 2024-01-16T15:07:18.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.595+0000 7fac79e00640 1 -- 172.21.15.141:0/2236961186 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7402cd20 con 0x7fac7c0ccb50 2024-01-16T15:07:18.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.595+0000 7fac79e00640 1 -- 172.21.15.141:0/2236961186 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74034660 con 0x7fac7c0ccb50 2024-01-16T15:07:18.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 -- 172.21.15.141:0/2236961186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0ccb50 msgr2=0x7fac7c143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 --2- 172.21.15.141:0/2236961186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0ccb50 0x7fac7c143d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fac74009d90 tx=0x7fac7402be10 comp rx=0 tx=0).stop 2024-01-16T15:07:18.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 -- 172.21.15.141:0/2236961186 shutdown_connections 2024-01-16T15:07:18.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 --2- 172.21.15.141:0/2236961186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0ccb50 0x7fac7c143d30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 --2- 172.21.15.141:0/2236961186 >> v2:172.21.15.188:3300/0 conn(0x7fac7c0adf20 0x7fac7c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 -- 172.21.15.141:0/2236961186 >> 172.21.15.141:0/2236961186 conn(0x7fac7c06a7d0 msgr2=0x7fac7c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:18.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 -- 172.21.15.141:0/2236961186 shutdown_connections 2024-01-16T15:07:18.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.596+0000 7fac80a00640 1 -- 172.21.15.141:0/2236961186 wait complete. 2024-01-16T15:07:18.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.598+0000 7fac80a00640 1 Processor -- start 2024-01-16T15:07:18.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.598+0000 7fac80a00640 1 -- start start 2024-01-16T15:07:18.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac80a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0adf20 0x7fac7c140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac80a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac7c0ccb50 0x7fac7c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac80a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac7c15bf10 0x7fac7c1e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac80a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0b2410 con 0x7fac7c15bf10 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac80a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c1e7240 con 0x7fac7c0ccb50 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac80a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0b2d70 con 0x7fac7c0adf20 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac7b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac7c15bf10 0x7fac7c1e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac7b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac7c15bf10 0x7fac7c1e62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58434/0 (socket says 172.21.15.141:58434) 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.599+0000 7fac7b200640 1 -- 172.21.15.141:0/229435312 learned_addr learned my addr 172.21.15.141:0/229435312 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac73e00640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac7c0ccb50 0x7fac7c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac7a800640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0adf20 0x7fac7c140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac7b200640 1 -- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0adf20 msgr2=0x7fac7c140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac7b200640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac7c0adf20 0x7fac7c140970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac7b200640 1 -- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac7c0ccb50 msgr2=0x7fac7c0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac7b200640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac7c0ccb50 0x7fac7c0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac7b200640 1 -- 172.21.15.141:0/229435312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c009400 con 0x7fac7c15bf10 2024-01-16T15:07:18.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.600+0000 7fac7b200640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac7c15bf10 0x7fac7c1e62a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fac74009d90 tx=0x7fac74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.601+0000 7fac72a00640 1 -- 172.21.15.141:0/229435312 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74040650 con 0x7fac7c15bf10 2024-01-16T15:07:18.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.601+0000 7fac72a00640 1 -- 172.21.15.141:0/229435312 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7402ce90 con 0x7fac7c15bf10 2024-01-16T15:07:18.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.601+0000 7fac72a00640 1 -- 172.21.15.141:0/229435312 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac740409f0 con 0x7fac7c15bf10 2024-01-16T15:07:18.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.601+0000 7fac80a00640 1 -- 172.21.15.141:0/229435312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac74009a70 con 0x7fac7c15bf10 2024-01-16T15:07:18.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.601+0000 7fac80a00640 1 -- 172.21.15.141:0/229435312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7c1e6f60 con 0x7fac7c15bf10 2024-01-16T15:07:18.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.602+0000 7fac80a00640 1 -- 172.21.15.141:0/229435312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7c0b2550 con 0x7fac7c15bf10 2024-01-16T15:07:18.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.603+0000 7fac72a00640 1 -- 172.21.15.141:0/229435312 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fac74040b50 con 0x7fac7c15bf10 2024-01-16T15:07:18.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.604+0000 7fac72a00640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac5406be90 0x7fac5406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.604+0000 7fac72a00640 1 -- 172.21.15.141:0/229435312 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fac740ab1c0 con 0x7fac7c15bf10 2024-01-16T15:07:18.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.604+0000 7fac7a800640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac5406be90 0x7fac5406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.605+0000 7fac7a800640 1 --2- 172.21.15.141:0/229435312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac5406be90 0x7fac5406e350 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fac7c1f5fc0 tx=0x7fac6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.608+0000 7fac72a00640 1 -- 172.21.15.141:0/229435312 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac7c0b2550 con 0x7fac7c15bf10 2024-01-16T15:07:18.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:18.838+0000 7fac80a00640 1 -- 172.21.15.141:0/229435312 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7fac7c15f170 con 0x7fac5406be90 2024-01-16T15:07:18.829 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:18.852+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:18.829 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:18.852+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:18.829 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:18.852+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:18.829 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:18.852+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:18.829 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:18.852+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:21.325 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:07:21.348+0000 7f0f2bc00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:21.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:21.347+0000 7fac72a00640 1 -- 172.21.15.141:0/229435312 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7fac6c0098d0 con 0x7fac5406be90 2024-01-16T15:07:21.326 INFO:teuthology.orchestra.run.smithi141.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:07:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:07:23.346 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59635 2024-01-16T15:07:23.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.629+0000 7fca32600640 1 Processor -- start 2024-01-16T15:07:23.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.629+0000 7fca32600640 1 -- start start 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.630+0000 7fca32600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c150760 0x7fca2c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca32600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fca2c0b32e0 0x7fca2c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca32600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c15dc80 con 0x7fca2c15d300 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca32600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c0b3770 con 0x7fca2c0b32e0 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca32600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c071190 con 0x7fca2c150760 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca2aa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fca2c15d300 0x7fca2c0b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36390/0 (socket says 172.21.15.141:36390) 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca2aa00640 1 -- 172.21.15.141:0/834733347 learned_addr learned my addr 172.21.15.141:0/834733347 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:23.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca2be00640 1 --2- 172.21.15.141:0/834733347 >> v2:172.21.15.141:3300/0 conn(0x7fca2c0b32e0 0x7fca2c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.631+0000 7fca2a000640 1 -- 172.21.15.141:0/834733347 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572892673 0 0) 0x7fca2c15dc80 con 0x7fca2c15d300 2024-01-16T15:07:23.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.632+0000 7fca2a000640 1 -- 172.21.15.141:0/834733347 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c003440 con 0x7fca2c15d300 2024-01-16T15:07:23.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.632+0000 7fca2be00640 1 -- 172.21.15.141:0/834733347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c150760 msgr2=0x7fca2c072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:23.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.632+0000 7fca2be00640 1 --2- 172.21.15.141:0/834733347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c150760 0x7fca2c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.632+0000 7fca2be00640 1 -- 172.21.15.141:0/834733347 >> v1:172.21.15.141:6789/0 conn(0x7fca2c15d300 legacy=0x7fca2c0b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.632+0000 7fca2be00640 1 -- 172.21.15.141:0/834733347 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c071470 con 0x7fca2c0b32e0 2024-01-16T15:07:23.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.633+0000 7fca2be00640 1 --2- 172.21.15.141:0/834733347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c0b32e0 0x7fca2c15d740 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fca24009d90 tx=0x7fca2402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918dfe22bc19bff7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.633+0000 7fca2a000640 1 -- 172.21.15.141:0/834733347 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2402cc50 con 0x7fca2c0b32e0 2024-01-16T15:07:23.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.633+0000 7fca2a000640 1 -- 172.21.15.141:0/834733347 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2402cdb0 con 0x7fca2c0b32e0 2024-01-16T15:07:23.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.633+0000 7fca2a000640 1 -- 172.21.15.141:0/834733347 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24034660 con 0x7fca2c0b32e0 2024-01-16T15:07:23.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.634+0000 7fca32600640 1 -- 172.21.15.141:0/834733347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c0b32e0 msgr2=0x7fca2c15d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.634+0000 7fca32600640 1 --2- 172.21.15.141:0/834733347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c0b32e0 0x7fca2c15d740 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fca24009d90 tx=0x7fca2402bea0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.634+0000 7fca32600640 1 -- 172.21.15.141:0/834733347 shutdown_connections 2024-01-16T15:07:23.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.634+0000 7fca32600640 1 --2- 172.21.15.141:0/834733347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c0b32e0 0x7fca2c15d740 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.634+0000 7fca32600640 1 --2- 172.21.15.141:0/834733347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c150760 0x7fca2c072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.634+0000 7fca32600640 1 -- 172.21.15.141:0/834733347 >> 172.21.15.141:0/834733347 conn(0x7fca2c06a800 msgr2=0x7fca2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:23.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.635+0000 7fca32600640 1 -- 172.21.15.141:0/834733347 shutdown_connections 2024-01-16T15:07:23.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.635+0000 7fca32600640 1 -- 172.21.15.141:0/834733347 wait complete. 2024-01-16T15:07:23.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.637+0000 7fca32600640 1 Processor -- start 2024-01-16T15:07:23.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.637+0000 7fca32600640 1 -- start start 2024-01-16T15:07:23.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.639+0000 7fca32600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c0b32e0 0x7fca2c13f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.639+0000 7fca32600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c150760 0x7fca2c14cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.639+0000 7fca32600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c15d300 0x7fca2c1e4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.639+0000 7fca32600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0adaa0 con 0x7fca2c150760 2024-01-16T15:07:23.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.639+0000 7fca32600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1497f0 con 0x7fca2c15d300 2024-01-16T15:07:23.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.639+0000 7fca32600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0ace20 con 0x7fca2c0b32e0 2024-01-16T15:07:23.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.639+0000 7fca2aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c150760 0x7fca2c14cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c150760 0x7fca2c14cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33966/0 (socket says 172.21.15.141:33966) 2024-01-16T15:07:23.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2aa00640 1 -- 172.21.15.141:0/2208469959 learned_addr learned my addr 172.21.15.141:0/2208469959 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:23.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2be00640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c15d300 0x7fca2c1e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2b400640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c0b32e0 0x7fca2c13f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2aa00640 1 -- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c0b32e0 msgr2=0x7fca2c13f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2aa00640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c0b32e0 0x7fca2c13f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2aa00640 1 -- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c15d300 msgr2=0x7fca2c1e4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2aa00640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c15d300 0x7fca2c1e4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.640+0000 7fca2aa00640 1 -- 172.21.15.141:0/2208469959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca24009a70 con 0x7fca2c150760 2024-01-16T15:07:23.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.641+0000 7fca2aa00640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c150760 0x7fca2c14cd10 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fca1400c3c0 tx=0x7fca1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.641+0000 7fca23400640 1 -- 172.21.15.141:0/2208469959 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14005ce0 con 0x7fca2c150760 2024-01-16T15:07:23.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.641+0000 7fca23400640 1 -- 172.21.15.141:0/2208469959 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca14005e40 con 0x7fca2c150760 2024-01-16T15:07:23.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.641+0000 7fca23400640 1 -- 172.21.15.141:0/2208469959 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca140072f0 con 0x7fca2c150760 2024-01-16T15:07:23.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.642+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c0ad100 con 0x7fca2c150760 2024-01-16T15:07:23.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.642+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c0ad6c0 con 0x7fca2c150760 2024-01-16T15:07:23.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.644+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c0adbe0 con 0x7fca2c150760 2024-01-16T15:07:23.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.646+0000 7fca23400640 1 -- 172.21.15.141:0/2208469959 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fca14007450 con 0x7fca2c150760 2024-01-16T15:07:23.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.647+0000 7fca23400640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0406bf20 0x7fca0406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.647+0000 7fca23400640 1 -- 172.21.15.141:0/2208469959 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fca14087d40 con 0x7fca2c150760 2024-01-16T15:07:23.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.648+0000 7fca2b400640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0406bf20 0x7fca0406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.650+0000 7fca2b400640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0406bf20 0x7fca0406e3e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009890 tx=0x7fca1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:23.655+0000 7fca23400640 1 -- 172.21.15.141:0/2208469959 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca2c0adbe0 con 0x7fca2c150760 2024-01-16T15:07:24.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.043+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59635", "target": ["mon-mgr", ""]}) v1 -- 0x7fca2c071d10 con 0x7fca0406bf20 2024-01-16T15:07:24.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.542+0000 7fca23400640 1 -- 172.21.15.141:0/2208469959 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fca1c009e70 con 0x7fca0406bf20 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.544+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0406bf20 msgr2=0x7fca0406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.544+0000 7fca32600640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0406bf20 0x7fca0406e3e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009890 tx=0x7fca1c008040 comp rx=0 tx=0).stop 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.544+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c150760 msgr2=0x7fca2c14cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c150760 0x7fca2c14cd10 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fca1400c3c0 tx=0x7fca1400c880 comp rx=0 tx=0).stop 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 shutdown_connections 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0406bf20 0x7fca0406e3e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c15d300 0x7fca2c1e4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca2c150760 0x7fca2c14cd10 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 --2- 172.21.15.141:0/2208469959 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca2c0b32e0 0x7fca2c13f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 >> 172.21.15.141:0/2208469959 conn(0x7fca2c06a800 msgr2=0x7fca2c14ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 shutdown_connections 2024-01-16T15:07:24.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:24.545+0000 7fca32600640 1 -- 172.21.15.141:0/2208469959 wait complete. 2024-01-16T15:07:26.550 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59636 2024-01-16T15:07:26.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.786+0000 7f52d5c00640 1 Processor -- start 2024-01-16T15:07:26.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.786+0000 7f52d5c00640 1 -- start start 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52d5c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52d013fcb0 0x7f52d00cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52d5c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f52d00cdc20 0x7f52d00b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52d5c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00ce440 con 0x7f52d013cf10 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52d5c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00ce720 con 0x7f52d013fcb0 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52d5c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00cea00 con 0x7f52d00cdc20 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52cf400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52d013fcb0 0x7f52d00cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52cf400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52d013fcb0 0x7f52d00cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33976/0 (socket says 172.21.15.141:33976) 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.788+0000 7f52cf400640 1 -- 172.21.15.141:0/2836701444 learned_addr learned my addr 172.21.15.141:0/2836701444 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.789+0000 7f52d5200640 1 --2- 172.21.15.141:0/2836701444 >> v2:172.21.15.188:3300/0 conn(0x7f52d00cdc20 0x7f52d00b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.789+0000 7f52cf400640 1 -- 172.21.15.141:0/2836701444 >> v2:172.21.15.188:3300/0 conn(0x7f52d00cdc20 msgr2=0x7f52d00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:26.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.789+0000 7f52cf400640 1 --2- 172.21.15.141:0/2836701444 >> v2:172.21.15.188:3300/0 conn(0x7f52d00cdc20 0x7f52d00b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.789+0000 7f52cf400640 1 -- 172.21.15.141:0/2836701444 >> v1:172.21.15.188:6789/0 conn(0x7f52d013cf10 legacy=0x7f52d013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:26.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.789+0000 7f52cf400640 1 -- 172.21.15.141:0/2836701444 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00b1d90 con 0x7f52d013fcb0 2024-01-16T15:07:26.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.789+0000 7f52d5200640 1 --2- 172.21.15.141:0/2836701444 >> v2:172.21.15.188:3300/0 conn(0x7f52d00cdc20 0x7f52d00b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.790+0000 7f52cf400640 1 --2- 172.21.15.141:0/2836701444 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d013fcb0 0x7f52d00cd6e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f52d0154a30 tx=0x7f52b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44524b31dc455d7c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.790+0000 7f52cea00640 1 -- 172.21.15.141:0/2836701444 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8031870 con 0x7f52d013fcb0 2024-01-16T15:07:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.790+0000 7f52cea00640 1 -- 172.21.15.141:0/2836701444 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b80319d0 con 0x7f52d013fcb0 2024-01-16T15:07:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.790+0000 7f52cea00640 1 -- 172.21.15.141:0/2836701444 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8031d60 con 0x7f52d013fcb0 2024-01-16T15:07:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 -- 172.21.15.141:0/2836701444 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d013fcb0 msgr2=0x7f52d00cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2836701444 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d013fcb0 0x7f52d00cd6e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f52d0154a30 tx=0x7f52b802b910 comp rx=0 tx=0).stop 2024-01-16T15:07:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 -- 172.21.15.141:0/2836701444 shutdown_connections 2024-01-16T15:07:26.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2836701444 >> v2:172.21.15.188:3300/0 conn(0x7f52d00cdc20 0x7f52d00b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2836701444 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d013fcb0 0x7f52d00cd6e0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 -- 172.21.15.141:0/2836701444 >> 172.21.15.141:0/2836701444 conn(0x7f52d006a7d0 msgr2=0x7f52d006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:26.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 -- 172.21.15.141:0/2836701444 shutdown_connections 2024-01-16T15:07:26.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.791+0000 7f52d5c00640 1 -- 172.21.15.141:0/2836701444 wait complete. 2024-01-16T15:07:26.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.793+0000 7f52d5c00640 1 Processor -- start 2024-01-16T15:07:26.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.793+0000 7f52d5c00640 1 -- start start 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52d5c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cdc20 0x7f52d0136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52d5c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d013cf10 0x7f52d00b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52d5c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d013fcb0 0x7f52d014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52d5c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0137970 con 0x7f52d00cdc20 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52d5c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0137ea0 con 0x7f52d013fcb0 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52d5c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0158e90 con 0x7f52d013cf10 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52cfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cdc20 0x7f52d0136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52cfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cdc20 0x7f52d0136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33980/0 (socket says 172.21.15.141:33980) 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.794+0000 7f52cfe00640 1 -- 172.21.15.141:0/2797824533 learned_addr learned my addr 172.21.15.141:0/2797824533 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52d5200640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d013fcb0 0x7f52d014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52cf400640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d013cf10 0x7f52d00b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52cfe00640 1 -- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d013cf10 msgr2=0x7f52d00b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52cfe00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d013cf10 0x7f52d00b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52cfe00640 1 -- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d013fcb0 msgr2=0x7f52d014fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52cfe00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d013fcb0 0x7f52d014fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52cfe00640 1 -- 172.21.15.141:0/2797824533 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80094d0 con 0x7f52d00cdc20 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52cfe00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cdc20 0x7f52d0136f80 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f52c000c460 tx=0x7f52c000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52c7e00640 1 -- 172.21.15.141:0/2797824533 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0012830 con 0x7f52d00cdc20 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d0150320 con 0x7f52d00cdc20 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52c7e00640 1 -- 172.21.15.141:0/2797824533 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c00129d0 con 0x7f52d00cdc20 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52c7e00640 1 -- 172.21.15.141:0/2797824533 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0012d80 con 0x7f52d00cdc20 2024-01-16T15:07:26.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.795+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d0158b20 con 0x7f52d00cdc20 2024-01-16T15:07:26.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.797+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d0067450 con 0x7f52d00cdc20 2024-01-16T15:07:26.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.798+0000 7f52c7e00640 1 -- 172.21.15.141:0/2797824533 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f52c001a3b0 con 0x7f52d00cdc20 2024-01-16T15:07:26.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.799+0000 7f52c7e00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806bf20 0x7f52a806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.799+0000 7f52c7e00640 1 -- 172.21.15.141:0/2797824533 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f52c0088270 con 0x7f52d00cdc20 2024-01-16T15:07:26.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.799+0000 7f52cf400640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806bf20 0x7f52a806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.800+0000 7f52cf400640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806bf20 0x7f52a806e3e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f52b8036000 tx=0x7f52b803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:26.803+0000 7f52c7e00640 1 -- 172.21.15.141:0/2797824533 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52c0054b30 con 0x7f52d00cdc20 2024-01-16T15:07:27.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:27.039+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59636", "target": ["mon-mgr", ""]}) v1 -- 0x7f52d00b2ad0 con 0x7f52a806bf20 2024-01-16T15:07:28.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.135+0000 7f52c7e00640 1 -- 172.21.15.141:0/2797824533 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f52b8031cc0 con 0x7f52a806bf20 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.138+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806bf20 msgr2=0x7f52a806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.138+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806bf20 0x7f52a806e3e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f52b8036000 tx=0x7f52b803c040 comp rx=0 tx=0).stop 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.138+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cdc20 msgr2=0x7f52d0136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.138+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cdc20 0x7f52d0136f80 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f52c000c460 tx=0x7f52c000c920 comp rx=0 tx=0).stop 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 shutdown_connections 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806bf20 0x7f52a806e3e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d013fcb0 0x7f52d014fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d013cf10 0x7f52d00b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 --2- 172.21.15.141:0/2797824533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cdc20 0x7f52d0136f80 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 >> 172.21.15.141:0/2797824533 conn(0x7f52d006a7d0 msgr2=0x7f52d006abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 shutdown_connections 2024-01-16T15:07:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:28.139+0000 7f52d5c00640 1 -- 172.21.15.141:0/2797824533 wait complete. 2024-01-16T15:07:30.138 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59637 2024-01-16T15:07:30.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.378+0000 7efd82200640 1 Processor -- start 2024-01-16T15:07:30.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.378+0000 7efd82200640 1 -- start start 2024-01-16T15:07:30.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.379+0000 7efd82200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efd7c143ae0 0x7efd7c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.379+0000 7efd82200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c0cf7d0 0x7efd7c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.379+0000 7efd82200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c0cfc30 con 0x7efd7c15e1e0 2024-01-16T15:07:30.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd82200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c0b1ee0 con 0x7efd7c143ae0 2024-01-16T15:07:30.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd82200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0b2780 con 0x7efd7c0cf7d0 2024-01-16T15:07:30.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efd7c143ae0 0x7efd7c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efd7c143ae0 0x7efd7c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33984/0 (socket says 172.21.15.141:33984) 2024-01-16T15:07:30.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7b400640 1 -- 172.21.15.141:0/881276063 learned_addr learned my addr 172.21.15.141:0/881276063 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:30.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd73e00640 1 --2- 172.21.15.141:0/881276063 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c0cf7d0 0x7efd7c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7b400640 1 -- 172.21.15.141:0/881276063 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c0cf7d0 msgr2=0x7efd7c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7b400640 1 --2- 172.21.15.141:0/881276063 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c0cf7d0 0x7efd7c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7b400640 1 -- 172.21.15.141:0/881276063 >> v1:172.21.15.141:6789/0 conn(0x7efd7c15e1e0 legacy=0x7efd7c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7b400640 1 -- 172.21.15.141:0/881276063 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c15e990 con 0x7efd7c143ae0 2024-01-16T15:07:30.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd73e00640 1 --2- 172.21.15.141:0/881276063 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c0cf7d0 0x7efd7c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:30.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.380+0000 7efd7aa00640 1 -- 172.21.15.141:0/881276063 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866227498 0 0) 0x7efd7c0cfc30 con 0x7efd7c15e1e0 2024-01-16T15:07:30.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.382+0000 7efd7b400640 1 --2- 172.21.15.141:0/881276063 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c0cf290 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009870 tx=0x7efd6c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1254b525252ea026 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.382+0000 7efd7aa00640 1 -- 172.21.15.141:0/881276063 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c030840 con 0x7efd7c143ae0 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.382+0000 7efd7aa00640 1 -- 172.21.15.141:0/881276063 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c0309a0 con 0x7efd7c143ae0 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.382+0000 7efd7aa00640 1 -- 172.21.15.141:0/881276063 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c030d30 con 0x7efd7c143ae0 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 -- 172.21.15.141:0/881276063 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 msgr2=0x7efd7c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 --2- 172.21.15.141:0/881276063 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c0cf290 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009870 tx=0x7efd6c02b980 comp rx=0 tx=0).stop 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 -- 172.21.15.141:0/881276063 shutdown_connections 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 --2- 172.21.15.141:0/881276063 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c0cf7d0 0x7efd7c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 --2- 172.21.15.141:0/881276063 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c0cf290 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 -- 172.21.15.141:0/881276063 >> 172.21.15.141:0/881276063 conn(0x7efd7c06a800 msgr2=0x7efd7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 -- 172.21.15.141:0/881276063 shutdown_connections 2024-01-16T15:07:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.383+0000 7efd82200640 1 -- 172.21.15.141:0/881276063 wait complete. 2024-01-16T15:07:30.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.385+0000 7efd82200640 1 Processor -- start 2024-01-16T15:07:30.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.385+0000 7efd82200640 1 -- start start 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd82200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd7c0cf7d0 0x7efd7c13f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd82200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c13c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd82200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c15e1e0 0x7efd7c15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd82200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c159cf0 con 0x7efd7c143ae0 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd82200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0ae0c0 con 0x7efd7c0cf7d0 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd82200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0ae3a0 con 0x7efd7c15e1e0 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd73e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c13c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd73e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c13c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33998/0 (socket says 172.21.15.141:33998) 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd73e00640 1 -- 172.21.15.141:0/2559382982 learned_addr learned my addr 172.21.15.141:0/2559382982 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd73e00640 1 -- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c15e1e0 msgr2=0x7efd7c15c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd73e00640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c15e1e0 0x7efd7c15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd7b400640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd7c0cf7d0 0x7efd7c13f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.386+0000 7efd73e00640 1 -- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd7c0cf7d0 msgr2=0x7efd7c13f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd73e00640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd7c0cf7d0 0x7efd7c13f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd73e00640 1 -- 172.21.15.141:0/2559382982 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c0094d0 con 0x7efd7c143ae0 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd7b400640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd7c0cf7d0 0x7efd7c13f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd73e00640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c13c8e0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c7f0 tx=0x7efd6400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd73400640 1 -- 172.21.15.141:0/2559382982 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64007a30 con 0x7efd7c143ae0 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd73400640 1 -- 172.21.15.141:0/2559382982 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd64007b90 con 0x7efd7c143ae0 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd73400640 1 -- 172.21.15.141:0/2559382982 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6400ce40 con 0x7efd7c143ae0 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c0ae680 con 0x7efd7c143ae0 2024-01-16T15:07:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.387+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c0ccb80 con 0x7efd7c143ae0 2024-01-16T15:07:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.388+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c0b2bf0 con 0x7efd7c143ae0 2024-01-16T15:07:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.389+0000 7efd73400640 1 -- 172.21.15.141:0/2559382982 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7efd640113f0 con 0x7efd7c143ae0 2024-01-16T15:07:30.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.390+0000 7efd73400640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd5406bf20 0x7efd5406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.390+0000 7efd73400640 1 -- 172.21.15.141:0/2559382982 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efd64088a30 con 0x7efd7c143ae0 2024-01-16T15:07:30.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.391+0000 7efd7b400640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd5406bf20 0x7efd5406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.392+0000 7efd7b400640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd5406bf20 0x7efd5406e3e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7efd6c036000 tx=0x7efd6c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.395+0000 7efd73400640 1 -- 172.21.15.141:0/2559382982 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd7c0b2bf0 con 0x7efd7c143ae0 2024-01-16T15:07:30.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:30.620+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59637", "target": ["mon-mgr", ""]}) v1 -- 0x7efd7c13d000 con 0x7efd5406bf20 2024-01-16T15:07:35.085 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:35.093 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:35.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.563+0000 7efd73400640 1 -- 172.21.15.141:0/2559382982 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efd7c13d000 con 0x7efd5406bf20 2024-01-16T15:07:35.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd5406bf20 msgr2=0x7efd5406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd5406bf20 0x7efd5406e3e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7efd6c036000 tx=0x7efd6c03b040 comp rx=0 tx=0).stop 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 msgr2=0x7efd7c13c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c13c8e0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c7f0 tx=0x7efd6400cbf0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 shutdown_connections 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efd5406bf20 0x7efd5406e3e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7efd7c15e1e0 0x7efd7c15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd7c143ae0 0x7efd7c13c8e0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 --2- 172.21.15.141:0/2559382982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd7c0cf7d0 0x7efd7c13f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 >> 172.21.15.141:0/2559382982 conn(0x7efd7c06a800 msgr2=0x7efd7c15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.566+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 shutdown_connections 2024-01-16T15:07:35.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:35.567+0000 7efd82200640 1 -- 172.21.15.141:0/2559382982 wait complete. 2024-01-16T15:07:37.569 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59638 2024-01-16T15:07:37.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.813+0000 7fb643e00640 1 Processor -- start 2024-01-16T15:07:37.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.814+0000 7fb643e00640 1 -- start start 2024-01-16T15:07:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.815+0000 7fb643e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb6440adf20 0x7fb644070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.815+0000 7fb643e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 0x7fb6440cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.815+0000 7fb643e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64415f4e0 con 0x7fb6440cc980 2024-01-16T15:07:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.815+0000 7fb643e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64415f970 con 0x7fb6440adf20 2024-01-16T15:07:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.815+0000 7fb643e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440cce10 con 0x7fb64415bf10 2024-01-16T15:07:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb643400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb6440cc980 0x7fb644143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42838/0 (socket says 172.21.15.141:42838) 2024-01-16T15:07:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb643400640 1 -- 172.21.15.141:0/26637321 learned_addr learned my addr 172.21.15.141:0/26637321 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb642000640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 0x7fb6440cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb642a00640 1 --2- 172.21.15.141:0/26637321 >> v2:172.21.15.188:3300/0 conn(0x7fb6440adf20 0x7fb644070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134851652 0 0) 0x7fb64415f4e0 con 0x7fb6440cc980 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb624003520 con 0x7fb6440cc980 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 545024871 0 0) 0x7fb624003520 con 0x7fb6440cc980 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 msgr2=0x7fb6440cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 0x7fb6440cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 >> v2:172.21.15.188:3300/0 conn(0x7fb6440adf20 msgr2=0x7fb644070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 --2- 172.21.15.141:0/26637321 >> v2:172.21.15.188:3300/0 conn(0x7fb6440adf20 0x7fb644070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb644071130 con 0x7fb6440cc980 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.816+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb63c0019f0 con 0x7fb6440cc980 2024-01-16T15:07:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 >> v1:172.21.15.141:6789/0 conn(0x7fb6440cc980 legacy=0x7fb644143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb641600640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb624003970 0x7fb624003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb641600640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 0x7fb6240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb641600640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb624004d00 0x7fb624005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64415f970 con 0x7fb624003970 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440cce10 con 0x7fb624004d00 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62400b980 con 0x7fb64415bf10 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb642a00640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb624003970 0x7fb624003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb642000640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 0x7fb6240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb643400640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb624004d00 0x7fb624005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb642a00640 1 -- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 msgr2=0x7fb6240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb642a00640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb64415bf10 0x7fb6240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb642a00640 1 -- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb624004d00 msgr2=0x7fb624005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb642a00640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb624004d00 0x7fb624005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.817+0000 7fb642a00640 1 -- 172.21.15.141:0/26637321 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb624003520 con 0x7fb624003970 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb642a00640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb624003970 0x7fb624003d90 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb634009ac0 tx=0x7fb6240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e25da81d5f6815 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63402c710 con 0x7fb624003970 2024-01-16T15:07:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63402c870 con 0x7fb624003970 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb641600640 1 -- 172.21.15.141:0/26637321 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63402cc00 con 0x7fb624003970 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb643e00640 1 -- 172.21.15.141:0/26637321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb624003970 msgr2=0x7fb624003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb643e00640 1 --2- 172.21.15.141:0/26637321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb624003970 0x7fb624003d90 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb634009ac0 tx=0x7fb6240057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb643400640 1 -- 172.21.15.141:0/26637321 reap_dead start 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb643e00640 1 -- 172.21.15.141:0/26637321 shutdown_connections 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.819+0000 7fb643e00640 1 -- 172.21.15.141:0/26637321 >> 172.21.15.141:0/26637321 conn(0x7fb64406a7d0 msgr2=0x7fb64406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.820+0000 7fb643e00640 1 -- 172.21.15.141:0/26637321 shutdown_connections 2024-01-16T15:07:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.820+0000 7fb643e00640 1 -- 172.21.15.141:0/26637321 wait complete. 2024-01-16T15:07:37.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.821+0000 7fb643e00640 1 Processor -- start 2024-01-16T15:07:37.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.822+0000 7fb643e00640 1 -- start start 2024-01-16T15:07:37.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.822+0000 7fb643e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb624003970 0x7fb6440b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb643e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6440b1e90 0x7fb644141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb643e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6440c5d80 0x7fb6440c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb643e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb644142080 con 0x7fb6440b1e90 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb643e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440c6740 con 0x7fb6440c5d80 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb643e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440c7d90 con 0x7fb624003970 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6440b1e90 0x7fb644141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6440b1e90 0x7fb644141b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33968/0 (socket says 172.21.15.141:33968) 2024-01-16T15:07:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 -- 172.21.15.141:0/1537962572 learned_addr learned my addr 172.21.15.141:0/1537962572 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:37.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 -- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb624003970 msgr2=0x7fb6440b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb624003970 0x7fb6440b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 -- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6440c5d80 msgr2=0x7fb6440c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6440c5d80 0x7fb6440c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 -- 172.21.15.141:0/1537962572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440b3390 con 0x7fb6440b1e90 2024-01-16T15:07:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.823+0000 7fb642000640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6440b1e90 0x7fb644141b40 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb62c011630 tx=0x7fb62c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.824+0000 7fb63aa00640 1 -- 172.21.15.141:0/1537962572 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00f8c0 con 0x7fb6440b1e90 2024-01-16T15:07:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.824+0000 7fb63aa00640 1 -- 172.21.15.141:0/1537962572 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c00fa20 con 0x7fb6440b1e90 2024-01-16T15:07:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.824+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6440b3670 con 0x7fb6440b1e90 2024-01-16T15:07:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.824+0000 7fb63aa00640 1 -- 172.21.15.141:0/1537962572 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00fd90 con 0x7fb6440b1e90 2024-01-16T15:07:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.824+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6440b3c30 con 0x7fb6440b1e90 2024-01-16T15:07:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.825+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6440b3df0 con 0x7fb6440b1e90 2024-01-16T15:07:37.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.826+0000 7fb63aa00640 1 -- 172.21.15.141:0/1537962572 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb62c015460 con 0x7fb6440b1e90 2024-01-16T15:07:37.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.827+0000 7fb63aa00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb61c06bf20 0x7fb61c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.827+0000 7fb63aa00640 1 -- 172.21.15.141:0/1537962572 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb62c0890b0 con 0x7fb6440b1e90 2024-01-16T15:07:37.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.828+0000 7fb642a00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb61c06bf20 0x7fb61c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.829+0000 7fb642a00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb61c06bf20 0x7fb61c06e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb634036000 tx=0x7fb63402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:37.831+0000 7fb63aa00640 1 -- 172.21.15.141:0/1537962572 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb62c0557a0 con 0x7fb6440b1e90 2024-01-16T15:07:38.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:38.059+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59638", "target": ["mon-mgr", ""]}) v1 -- 0x7fb64415ee90 con 0x7fb61c06bf20 2024-01-16T15:07:40.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.039+0000 7fb63aa00640 1 -- 172.21.15.141:0/1537962572 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb63402c9e0 con 0x7fb61c06bf20 2024-01-16T15:07:40.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb61c06bf20 msgr2=0x7fb61c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb61c06bf20 0x7fb61c06e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb634036000 tx=0x7fb63402e000 comp rx=0 tx=0).stop 2024-01-16T15:07:40.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6440b1e90 msgr2=0x7fb644141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6440b1e90 0x7fb644141b40 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb62c011630 tx=0x7fb62c011af0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 shutdown_connections 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb61c06bf20 0x7fb61c06e3e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6440c5d80 0x7fb6440c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6440b1e90 0x7fb644141b40 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 --2- 172.21.15.141:0/1537962572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb624003970 0x7fb6440b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 >> 172.21.15.141:0/1537962572 conn(0x7fb64406a7d0 msgr2=0x7fb64406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 shutdown_connections 2024-01-16T15:07:40.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:40.042+0000 7fb643e00640 1 -- 172.21.15.141:0/1537962572 wait complete. 2024-01-16T15:07:42.041 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone59639 2024-01-16T15:07:42.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.280+0000 7fbb59e00640 1 Processor -- start 2024-01-16T15:07:42.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.280+0000 7fbb59e00640 1 -- start start 2024-01-16T15:07:42.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb59e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb5413df80 0x7fbb5415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb59e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5413e360 0x7fbb540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb59e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5406c470 con 0x7fbb5415f9c0 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb59e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5406c750 con 0x7fbb5413df80 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb59e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5406ca30 con 0x7fbb5413e360 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb52a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb5413df80 0x7fbb5415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb52a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb5413df80 0x7fbb5415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33978/0 (socket says 172.21.15.141:33978) 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.282+0000 7fbb52a00640 1 -- 172.21.15.141:0/901740075 learned_addr learned my addr 172.21.15.141:0/901740075 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.283+0000 7fbb52000640 1 --2- 172.21.15.141:0/901740075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5413e360 0x7fbb540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.283+0000 7fbb52a00640 1 -- 172.21.15.141:0/901740075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5413e360 msgr2=0x7fbb540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.283+0000 7fbb52a00640 1 --2- 172.21.15.141:0/901740075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5413e360 0x7fbb540cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.283+0000 7fbb52a00640 1 -- 172.21.15.141:0/901740075 >> v1:172.21.15.188:6789/0 conn(0x7fbb5415f9c0 legacy=0x7fbb541408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.283+0000 7fbb52a00640 1 -- 172.21.15.141:0/901740075 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5406cd10 con 0x7fbb5413df80 2024-01-16T15:07:42.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.283+0000 7fbb52000640 1 --2- 172.21.15.141:0/901740075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5413e360 0x7fbb540cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:42.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.284+0000 7fbb52a00640 1 --2- 172.21.15.141:0/901740075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5415bee0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d130e8127ee7f3ea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.284+0000 7fbb51600640 1 -- 172.21.15.141:0/901740075 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44031640 con 0x7fbb5413df80 2024-01-16T15:07:42.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.284+0000 7fbb51600640 1 -- 172.21.15.141:0/901740075 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb440317a0 con 0x7fbb5413df80 2024-01-16T15:07:42.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.284+0000 7fbb51600640 1 -- 172.21.15.141:0/901740075 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44031b30 con 0x7fbb5413df80 2024-01-16T15:07:42.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 -- 172.21.15.141:0/901740075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 msgr2=0x7fbb5415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 --2- 172.21.15.141:0/901740075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5415bee0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb4402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 -- 172.21.15.141:0/901740075 shutdown_connections 2024-01-16T15:07:42.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 --2- 172.21.15.141:0/901740075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5413e360 0x7fbb540cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 --2- 172.21.15.141:0/901740075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5415bee0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 -- 172.21.15.141:0/901740075 >> 172.21.15.141:0/901740075 conn(0x7fbb5406a800 msgr2=0x7fbb5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:42.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 -- 172.21.15.141:0/901740075 shutdown_connections 2024-01-16T15:07:42.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.285+0000 7fbb59e00640 1 -- 172.21.15.141:0/901740075 wait complete. 2024-01-16T15:07:42.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.287+0000 7fbb59e00640 1 Processor -- start 2024-01-16T15:07:42.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.287+0000 7fbb59e00640 1 -- start start 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb59e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5413d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb59e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb5413e360 0x7fbb5413d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb59e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5415f9c0 0x7fbb54159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb59e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb54139380 con 0x7fbb5413df80 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb59e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5414ab20 con 0x7fbb5413e360 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb59e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5414b980 con 0x7fbb5415f9c0 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb52a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5413d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb52a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5413d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33986/0 (socket says 172.21.15.141:33986) 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.288+0000 7fbb52a00640 1 -- 172.21.15.141:0/704533376 learned_addr learned my addr 172.21.15.141:0/704533376 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb52000640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb5413e360 0x7fbb5413d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb53400640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5415f9c0 0x7fbb54159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb52a00640 1 -- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5415f9c0 msgr2=0x7fbb54159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb52a00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5415f9c0 0x7fbb54159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb52a00640 1 -- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb5413e360 msgr2=0x7fbb5413d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb52a00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb5413e360 0x7fbb5413d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb52a00640 1 -- 172.21.15.141:0/704533376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb440094d0 con 0x7fbb5413df80 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb53400640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5415f9c0 0x7fbb54159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb52a00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5413d480 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb4b400640 1 -- 172.21.15.141:0/704533376 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4403ac60 con 0x7fbb5413df80 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb4b400640 1 -- 172.21.15.141:0/704533376 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4403ade0 con 0x7fbb5413df80 2024-01-16T15:07:42.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5414ad70 con 0x7fbb5413df80 2024-01-16T15:07:42.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.289+0000 7fbb4b400640 1 -- 172.21.15.141:0/704533376 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44044630 con 0x7fbb5413df80 2024-01-16T15:07:42.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.290+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5414b330 con 0x7fbb5413df80 2024-01-16T15:07:42.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.291+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb540675d0 con 0x7fbb5413df80 2024-01-16T15:07:42.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.292+0000 7fbb4b400640 1 -- 172.21.15.141:0/704533376 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fbb4402f050 con 0x7fbb5413df80 2024-01-16T15:07:42.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.292+0000 7fbb4b400640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb2c06bf50 0x7fbb2c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.293+0000 7fbb4b400640 1 -- 172.21.15.141:0/704533376 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbb440aa910 con 0x7fbb5413df80 2024-01-16T15:07:42.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.293+0000 7fbb52000640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb2c06bf50 0x7fbb2c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.294+0000 7fbb52000640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb2c06bf50 0x7fbb2c06e410 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbb54150ab0 tx=0x7fbb3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.297+0000 7fbb4b400640 1 -- 172.21.15.141:0/704533376 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb44076f50 con 0x7fbb5413df80 2024-01-16T15:07:42.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:42.530+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone59639", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb540ce850 con 0x7fbb2c06bf50 2024-01-16T15:07:45.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.954+0000 7fbb4b400640 1 -- 172.21.15.141:0/704533376 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbb3c00a660 con 0x7fbb2c06bf50 2024-01-16T15:07:45.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.958+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb2c06bf50 msgr2=0x7fbb2c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:45.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.958+0000 7fbb59e00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb2c06bf50 0x7fbb2c06e410 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbb54150ab0 tx=0x7fbb3c008040 comp rx=0 tx=0).stop 2024-01-16T15:07:45.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.958+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 msgr2=0x7fbb5413d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:45.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.958+0000 7fbb59e00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5413d480 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb440046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 shutdown_connections 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb2c06bf50 0x7fbb2c06e410 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb5415f9c0 0x7fbb54159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb5413e360 0x7fbb5413d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 --2- 172.21.15.141:0/704533376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb5413df80 0x7fbb5413d480 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 >> 172.21.15.141:0/704533376 conn(0x7fbb5406a800 msgr2=0x7fbb5415d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 shutdown_connections 2024-01-16T15:07:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:45.959+0000 7fbb59e00640 1 -- 172.21.15.141:0/704533376 wait complete. 2024-01-16T15:07:47.959 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596310 2024-01-16T15:07:48.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.205+0000 7f9b0a800640 1 Processor -- start 2024-01-16T15:07:48.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.205+0000 7f9b0a800640 1 -- start start 2024-01-16T15:07:48.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9b0a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9b040cf9a0 0x7f9b0415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:48.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9b0a800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b040b1ee0 con 0x7f9b0415e1e0 2024-01-16T15:07:48.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9b0a800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b040b2780 con 0x7f9b04143ae0 2024-01-16T15:07:48.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9b0a800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0415e990 con 0x7f9b040cf9a0 2024-01-16T15:07:48.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9afbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9b040cf9a0 0x7f9b0415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:48.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9b03e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9b0415e1e0 0x7f9b040702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36888/0 (socket says 172.21.15.141:36888) 2024-01-16T15:07:48.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9b03e00640 1 -- 172.21.15.141:0/3008386921 learned_addr learned my addr 172.21.15.141:0/3008386921 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:48.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9afbe00640 1 -- 172.21.15.141:0/3008386921 >> v1:172.21.15.188:6789/0 conn(0x7f9b04143ae0 legacy=0x7f9b040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:48.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9b02a00640 1 -- 172.21.15.141:0/3008386921 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148235921 0 0) 0x7f9b040b1ee0 con 0x7f9b0415e1e0 2024-01-16T15:07:48.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9afbe00640 1 -- 172.21.15.141:0/3008386921 >> v1:172.21.15.141:6789/0 conn(0x7f9b0415e1e0 legacy=0x7f9b040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:48.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.207+0000 7f9afbe00640 1 -- 172.21.15.141:0/3008386921 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040b2130 con 0x7f9b040cf9a0 2024-01-16T15:07:48.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.209+0000 7f9afbe00640 1 --2- 172.21.15.141:0/3008386921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b040cf9a0 0x7f9b0415dca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0097a0 tx=0x7f9aec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e74da80f99989f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:48.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.209+0000 7f9b02a00640 1 -- 172.21.15.141:0/3008386921 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec031830 con 0x7f9b040cf9a0 2024-01-16T15:07:48.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.209+0000 7f9b02a00640 1 -- 172.21.15.141:0/3008386921 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec031990 con 0x7f9b040cf9a0 2024-01-16T15:07:48.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.209+0000 7f9b02a00640 1 -- 172.21.15.141:0/3008386921 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec031d20 con 0x7f9b040cf9a0 2024-01-16T15:07:48.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.209+0000 7f9b0a800640 1 -- 172.21.15.141:0/3008386921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b040cf9a0 msgr2=0x7f9b0415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:48.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.209+0000 7f9b0a800640 1 --2- 172.21.15.141:0/3008386921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b040cf9a0 0x7f9b0415dca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0097a0 tx=0x7f9aec02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.210+0000 7f9b0a800640 1 -- 172.21.15.141:0/3008386921 shutdown_connections 2024-01-16T15:07:48.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.210+0000 7f9b0a800640 1 --2- 172.21.15.141:0/3008386921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b040cf9a0 0x7f9b0415dca0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.210+0000 7f9b0a800640 1 -- 172.21.15.141:0/3008386921 >> 172.21.15.141:0/3008386921 conn(0x7f9b0406a800 msgr2=0x7f9b0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:48.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.210+0000 7f9b0a800640 1 -- 172.21.15.141:0/3008386921 shutdown_connections 2024-01-16T15:07:48.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.210+0000 7f9b0a800640 1 -- 172.21.15.141:0/3008386921 wait complete. 2024-01-16T15:07:48.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.211+0000 7f9b0a800640 1 Processor -- start 2024-01-16T15:07:48.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.212+0000 7f9b0a800640 1 -- start start 2024-01-16T15:07:48.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b0a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b040cf9a0 0x7f9b040cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:48.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b0a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b04143ae0 0x7f9b040ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:48.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b0a800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b0415e1e0 0x7f9b040ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:48.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b0a800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040cd890 con 0x7f9b04143ae0 2024-01-16T15:07:48.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b0a800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040ccc80 con 0x7f9b040cf9a0 2024-01-16T15:07:48.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b0a800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040ccf60 con 0x7f9b0415e1e0 2024-01-16T15:07:48.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b04143ae0 0x7f9b040ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b04143ae0 0x7f9b040ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43812/0 (socket says 172.21.15.141:43812) 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 -- 172.21.15.141:0/1518502192 learned_addr learned my addr 172.21.15.141:0/1518502192 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b03400640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b040cf9a0 0x7f9b040cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 -- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b0415e1e0 msgr2=0x7f9b040ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9b03e00640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b0415e1e0 0x7f9b040ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b0415e1e0 0x7f9b040ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 -- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b040cf9a0 msgr2=0x7f9b040cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b040cf9a0 0x7f9b040cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.213+0000 7f9afbe00640 1 -- 172.21.15.141:0/1518502192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec009400 con 0x7f9b04143ae0 2024-01-16T15:07:48.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.214+0000 7f9afbe00640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b04143ae0 0x7f9b040ce120 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098c0 tx=0x7f9aec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:48.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.214+0000 7f9afb400640 1 -- 172.21.15.141:0/1518502192 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec03bb70 con 0x7f9b04143ae0 2024-01-16T15:07:48.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.214+0000 7f9afb400640 1 -- 172.21.15.141:0/1518502192 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec03bcf0 con 0x7f9b04143ae0 2024-01-16T15:07:48.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.214+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b040cd240 con 0x7f9b04143ae0 2024-01-16T15:07:48.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.214+0000 7f9afb400640 1 -- 172.21.15.141:0/1518502192 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec045630 con 0x7f9b04143ae0 2024-01-16T15:07:48.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.214+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b040affe0 con 0x7f9b04143ae0 2024-01-16T15:07:48.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.216+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b040cda00 con 0x7f9b04143ae0 2024-01-16T15:07:48.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.216+0000 7f9afb400640 1 -- 172.21.15.141:0/1518502192 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9aec04e450 con 0x7f9b04143ae0 2024-01-16T15:07:48.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.217+0000 7f9afb400640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9adc06bf20 0x7f9adc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:48.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.217+0000 7f9afb400640 1 -- 172.21.15.141:0/1518502192 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9aec0aacc0 con 0x7f9b04143ae0 2024-01-16T15:07:48.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.218+0000 7f9b03400640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9adc06bf20 0x7f9adc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:48.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.219+0000 7f9b03400640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9adc06bf20 0x7f9adc06e3e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9af4004ea0 tx=0x7f9af4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:48.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.222+0000 7f9afb400640 1 -- 172.21.15.141:0/1518502192 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9aec077480 con 0x7f9b04143ae0 2024-01-16T15:07:48.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:48.454+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596310", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b04066fe0 con 0x7f9adc06bf20 2024-01-16T15:07:49.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.576+0000 7f9afb400640 1 -- 172.21.15.141:0/1518502192 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9b04066fe0 con 0x7f9adc06bf20 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9adc06bf20 msgr2=0x7f9adc06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9adc06bf20 0x7f9adc06e3e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9af4004ea0 tx=0x7f9af4009040 comp rx=0 tx=0).stop 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b04143ae0 msgr2=0x7f9b040ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b04143ae0 0x7f9b040ce120 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098c0 tx=0x7f9aec0046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 shutdown_connections 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9adc06bf20 0x7f9adc06e3e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9b0415e1e0 0x7f9b040ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9b04143ae0 0x7f9b040ce120 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 --2- 172.21.15.141:0/1518502192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b040cf9a0 0x7f9b040cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 >> 172.21.15.141:0/1518502192 conn(0x7f9b0406a800 msgr2=0x7f9b0415b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:49.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 shutdown_connections 2024-01-16T15:07:49.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:49.579+0000 7f9b0a800640 1 -- 172.21.15.141:0/1518502192 wait complete. 2024-01-16T15:07:51.578 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596311 2024-01-16T15:07:51.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.824+0000 7fa9a9800640 1 Processor -- start 2024-01-16T15:07:51.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.824+0000 7fa9a9800640 1 -- start start 2024-01-16T15:07:51.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.826+0000 7fa9a9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 0x7fa9a415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.826+0000 7fa9a9800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40b1ee0 con 0x7fa9a4143ae0 2024-01-16T15:07:51.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.826+0000 7fa9a9800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40b2780 con 0x7fa9a415e1e0 2024-01-16T15:07:51.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.826+0000 7fa9a9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a415e990 con 0x7fa9a40cf9a0 2024-01-16T15:07:51.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.826+0000 7fa9a2a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa9a4143ae0 0x7fa9a40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36892/0 (socket says 172.21.15.141:36892) 2024-01-16T15:07:51.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.826+0000 7fa9a2a00640 1 -- 172.21.15.141:0/1368869464 learned_addr learned my addr 172.21.15.141:0/1368869464 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa99be00640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 0x7fa9a415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624759211 0 0) 0x7fa9a40b1ee0 con 0x7fa9a4143ae0 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa984003520 con 0x7fa9a4143ae0 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3716315213 0 0) 0x7fa984003520 con 0x7fa9a4143ae0 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 msgr2=0x7fa9a415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 0x7fa9a415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 >> v1:172.21.15.188:6789/0 conn(0x7fa9a415e1e0 legacy=0x7fa9a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a40b2130 con 0x7fa9a4143ae0 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa994001670 con 0x7fa9a4143ae0 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 >> v1:172.21.15.141:6789/0 conn(0x7fa9a4143ae0 legacy=0x7fa9a40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9840039a0 0x7fa984003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 0x7fa9840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa984004d00 0x7fa984005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a415e990 con 0x7fa9840039a0 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98400ba30 con 0x7fa984004d00 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.827+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98400bbf0 con 0x7fa9a40cf9a0 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.828+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9840039a0 0x7fa984003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.828+0000 7fa99be00640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 0x7fa9840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.828+0000 7fa9a2a00640 1 -- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 msgr2=0x7fa9840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.828+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a40cf9a0 0x7fa9840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.828+0000 7fa9a2a00640 1 -- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa984004d00 msgr2=0x7fa984005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.828+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa984004d00 0x7fa984005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.828+0000 7fa9a2a00640 1 -- 172.21.15.141:0/1368869464 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa984003520 con 0x7fa9840039a0 2024-01-16T15:07:51.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.829+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9840039a0 0x7fa984003dc0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa994007fd0 tx=0x7fa9840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1731ab8b1f4e60a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.830+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994005ad0 con 0x7fa9840039a0 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.830+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994005c30 con 0x7fa9840039a0 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.830+0000 7fa9a2000640 1 -- 172.21.15.141:0/1368869464 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99402c870 con 0x7fa9840039a0 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.830+0000 7fa9a9800640 1 -- 172.21.15.141:0/1368869464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9840039a0 msgr2=0x7fa984003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.830+0000 7fa9a9800640 1 --2- 172.21.15.141:0/1368869464 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9840039a0 0x7fa984003dc0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa994007fd0 tx=0x7fa9840057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.830+0000 7fa9a3400640 1 -- 172.21.15.141:0/1368869464 reap_dead start 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.831+0000 7fa9a9800640 1 -- 172.21.15.141:0/1368869464 shutdown_connections 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.831+0000 7fa9a9800640 1 -- 172.21.15.141:0/1368869464 >> 172.21.15.141:0/1368869464 conn(0x7fa9a406a800 msgr2=0x7fa9a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.831+0000 7fa9a9800640 1 -- 172.21.15.141:0/1368869464 shutdown_connections 2024-01-16T15:07:51.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.831+0000 7fa9a9800640 1 -- 172.21.15.141:0/1368869464 wait complete. 2024-01-16T15:07:51.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.832+0000 7fa9a9800640 1 Processor -- start 2024-01-16T15:07:51.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.833+0000 7fa9a9800640 1 -- start start 2024-01-16T15:07:51.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.833+0000 7fa9a9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9840039a0 0x7fa9a413f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a415c1a0 0x7fa9a415c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a9800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9a4139380 0x7fa9a4139800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a9800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a415d220 con 0x7fa9a4139380 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a9800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a4141770 con 0x7fa9840039a0 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a4141160 con 0x7fa9a415c1a0 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9a4139380 0x7fa9a4139800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9a4139380 0x7fa9a4139800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43822/0 (socket says 172.21.15.141:43822) 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 -- 172.21.15.141:0/3887348986 learned_addr learned my addr 172.21.15.141:0/3887348986 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9840039a0 0x7fa9a413f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa99be00640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a415c1a0 0x7fa9a415c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 -- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a415c1a0 msgr2=0x7fa9a415c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a415c1a0 0x7fa9a415c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 -- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9840039a0 msgr2=0x7fa9a413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9840039a0 0x7fa9a413f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a3400640 1 -- 172.21.15.141:0/3887348986 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4139d40 con 0x7fa9a4139380 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.834+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9840039a0 0x7fa9a413f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:51.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.835+0000 7fa9a3400640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9a4139380 0x7fa9a4139800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa984003970 tx=0x7fa99c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.835+0000 7fa99aa00640 1 -- 172.21.15.141:0/3887348986 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c005d40 con 0x7fa9a4139380 2024-01-16T15:07:51.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.835+0000 7fa99aa00640 1 -- 172.21.15.141:0/3887348986 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c00ced0 con 0x7fa9a4139380 2024-01-16T15:07:51.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.835+0000 7fa99aa00640 1 -- 172.21.15.141:0/3887348986 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c00a850 con 0x7fa9a4139380 2024-01-16T15:07:51.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.835+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a415cc60 con 0x7fa9a4139380 2024-01-16T15:07:51.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.835+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a4141400 con 0x7fa9a4139380 2024-01-16T15:07:51.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.836+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a40ce970 con 0x7fa9a4139380 2024-01-16T15:07:51.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.837+0000 7fa99aa00640 1 -- 172.21.15.141:0/3887348986 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fa99c00a9b0 con 0x7fa9a4139380 2024-01-16T15:07:51.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.839+0000 7fa99aa00640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa97c06bf20 0x7fa97c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.839+0000 7fa99aa00640 1 -- 172.21.15.141:0/3887348986 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa99c0879d0 con 0x7fa9a4139380 2024-01-16T15:07:51.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.839+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa97c06bf20 0x7fa97c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.841+0000 7fa9a2a00640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa97c06bf20 0x7fa97c06e3e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa994034000 tx=0x7fa994009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:51.842+0000 7fa99aa00640 1 -- 172.21.15.141:0/3887348986 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa99c054010 con 0x7fa9a4139380 2024-01-16T15:07:52.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:07:52.070+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596311", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9a40b2bf0 con 0x7fa97c06bf20 2024-01-16T15:08:01.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.446+0000 7fa99aa00640 1 -- 172.21.15.141:0/3887348986 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa9940014b0 con 0x7fa97c06bf20 2024-01-16T15:08:01.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.449+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa97c06bf20 msgr2=0x7fa97c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.449+0000 7fa9a9800640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa97c06bf20 0x7fa97c06e3e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa994034000 tx=0x7fa994009030 comp rx=0 tx=0).stop 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.449+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9a4139380 msgr2=0x7fa9a4139800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.449+0000 7fa9a9800640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9a4139380 0x7fa9a4139800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa984003970 tx=0x7fa99c00cb90 comp rx=0 tx=0).stop 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 shutdown_connections 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa97c06bf20 0x7fa97c06e3e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9a4139380 0x7fa9a4139800 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa9a415c1a0 0x7fa9a415c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 --2- 172.21.15.141:0/3887348986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9840039a0 0x7fa9a413f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 >> 172.21.15.141:0/3887348986 conn(0x7fa9a406a800 msgr2=0x7fa9a40cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 shutdown_connections 2024-01-16T15:08:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:01.450+0000 7fa9a9800640 1 -- 172.21.15.141:0/3887348986 wait complete. 2024-01-16T15:08:03.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596312 2024-01-16T15:08:03.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.699+0000 7fcc89c00640 1 Processor -- start 2024-01-16T15:08:03.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.699+0000 7fcc89c00640 1 -- start start 2024-01-16T15:08:03.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.700+0000 7fcc89c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcc8413df80 0x7fcc8415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.700+0000 7fcc89c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413e360 0x7fcc840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.701+0000 7fcc89c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8406c470 con 0x7fcc8415f9c0 2024-01-16T15:08:03.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.701+0000 7fcc89c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8406c750 con 0x7fcc8413df80 2024-01-16T15:08:03.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.701+0000 7fcc89c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8406ca30 con 0x7fcc8413e360 2024-01-16T15:08:03.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.701+0000 7fcc79e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413e360 0x7fcc840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.701+0000 7fcc79e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413e360 0x7fcc840cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:44614/0 (socket says 172.21.15.141:44614) 2024-01-16T15:08:03.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.701+0000 7fcc79e00640 1 -- 172.21.15.141:0/4252088534 learned_addr learned my addr 172.21.15.141:0/4252088534 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:03.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.701+0000 7fcc82a00640 1 --2- 172.21.15.141:0/4252088534 >> v2:172.21.15.188:3300/0 conn(0x7fcc8413df80 0x7fcc8415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.702+0000 7fcc82a00640 1 -- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413e360 msgr2=0x7fcc840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:03.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.702+0000 7fcc82a00640 1 --2- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413e360 0x7fcc840cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.702+0000 7fcc82a00640 1 -- 172.21.15.141:0/4252088534 >> v1:172.21.15.188:6789/0 conn(0x7fcc8415f9c0 legacy=0x7fcc841408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.702+0000 7fcc82a00640 1 -- 172.21.15.141:0/4252088534 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8406cd10 con 0x7fcc8413df80 2024-01-16T15:08:03.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.702+0000 7fcc79e00640 1 --2- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413e360 0x7fcc840cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:08:03.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.703+0000 7fcc82a00640 1 --2- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8413df80 0x7fcc8415bee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc740097a0 tx=0x7fcc7402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=221beb72a58e4ce2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:03.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.704+0000 7fcc82000640 1 -- 172.21.15.141:0/4252088534 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7402d040 con 0x7fcc8413df80 2024-01-16T15:08:03.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.704+0000 7fcc82000640 1 -- 172.21.15.141:0/4252088534 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc74031810 con 0x7fcc8413df80 2024-01-16T15:08:03.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.704+0000 7fcc82000640 1 -- 172.21.15.141:0/4252088534 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74031ba0 con 0x7fcc8413df80 2024-01-16T15:08:03.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 -- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8413df80 msgr2=0x7fcc8415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 --2- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8413df80 0x7fcc8415bee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc740097a0 tx=0x7fcc7402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 -- 172.21.15.141:0/4252088534 shutdown_connections 2024-01-16T15:08:03.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 --2- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413e360 0x7fcc840cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 --2- 172.21.15.141:0/4252088534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8413df80 0x7fcc8415bee0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 -- 172.21.15.141:0/4252088534 >> 172.21.15.141:0/4252088534 conn(0x7fcc8406a800 msgr2=0x7fcc8406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:03.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 -- 172.21.15.141:0/4252088534 shutdown_connections 2024-01-16T15:08:03.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.705+0000 7fcc89c00640 1 -- 172.21.15.141:0/4252088534 wait complete. 2024-01-16T15:08:03.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.707+0000 7fcc89c00640 1 Processor -- start 2024-01-16T15:08:03.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.707+0000 7fcc89c00640 1 -- start start 2024-01-16T15:08:03.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc89c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413df80 0x7fcc84071de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc89c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc8413e360 0x7fcc8415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc89c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8415f9c0 0x7fcc840b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc89c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc84072320 con 0x7fcc8413e360 2024-01-16T15:08:03.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc89c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8414b620 con 0x7fcc8415f9c0 2024-01-16T15:08:03.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc89c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8414b900 con 0x7fcc8413df80 2024-01-16T15:08:03.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc79e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc8413e360 0x7fcc8415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc79e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc8413e360 0x7fcc8415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55148/0 (socket says 172.21.15.141:55148) 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc79e00640 1 -- 172.21.15.141:0/3501903654 learned_addr learned my addr 172.21.15.141:0/3501903654 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc82a00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413df80 0x7fcc84071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.708+0000 7fcc83400640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8415f9c0 0x7fcc840b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc79e00640 1 -- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413df80 msgr2=0x7fcc84071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc79e00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413df80 0x7fcc84071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc79e00640 1 -- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8415f9c0 msgr2=0x7fcc840b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc79e00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8415f9c0 0x7fcc840b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc79e00640 1 -- 172.21.15.141:0/3501903654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0094d0 con 0x7fcc8413e360 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc83400640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8415f9c0 0x7fcc840b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:03.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc79e00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc8413e360 0x7fcc8415a030 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00e9d0 tx=0x7fcc6c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:03.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc7b400640 1 -- 172.21.15.141:0/3501903654 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c00ca60 con 0x7fcc8413e360 2024-01-16T15:08:03.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc7b400640 1 -- 172.21.15.141:0/3501903654 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c00cbc0 con 0x7fcc8413e360 2024-01-16T15:08:03.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc7b400640 1 -- 172.21.15.141:0/3501903654 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c013470 con 0x7fcc8413e360 2024-01-16T15:08:03.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc74009400 con 0x7fcc8413e360 2024-01-16T15:08:03.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.709+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8414bec0 con 0x7fcc8413e360 2024-01-16T15:08:03.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.711+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc840675d0 con 0x7fcc8413e360 2024-01-16T15:08:03.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.711+0000 7fcc7b400640 1 -- 172.21.15.141:0/3501903654 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fcc6c0135d0 con 0x7fcc8413e360 2024-01-16T15:08:03.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.712+0000 7fcc7b400640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc5c06bf20 0x7fcc5c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.712+0000 7fcc7b400640 1 -- 172.21.15.141:0/3501903654 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcc6c088bb0 con 0x7fcc8413e360 2024-01-16T15:08:03.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.713+0000 7fcc82a00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc5c06bf20 0x7fcc5c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.714+0000 7fcc82a00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc5c06bf20 0x7fcc5c06e3e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcc74036000 tx=0x7fcc7403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:03.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.717+0000 7fcc7b400640 1 -- 172.21.15.141:0/3501903654 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc6c0551b0 con 0x7fcc8413e360 2024-01-16T15:08:03.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:03.946+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596312", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc8414b790 con 0x7fcc5c06bf20 2024-01-16T15:08:05.222 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:05.230 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:09.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.116+0000 7fcc7b400640 1 -- 172.21.15.141:0/3501903654 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcc8414b790 con 0x7fcc5c06bf20 2024-01-16T15:08:09.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc5c06bf20 msgr2=0x7fcc5c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc5c06bf20 0x7fcc5c06e3e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcc74036000 tx=0x7fcc7403c040 comp rx=0 tx=0).stop 2024-01-16T15:08:09.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc8413e360 msgr2=0x7fcc8415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc8413e360 0x7fcc8415a030 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00e9d0 tx=0x7fcc6c00ee90 comp rx=0 tx=0).stop 2024-01-16T15:08:09.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 shutdown_connections 2024-01-16T15:08:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcc5c06bf20 0x7fcc5c06e3e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8415f9c0 0x7fcc840b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc8413e360 0x7fcc8415a030 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.119+0000 7fcc89c00640 1 --2- 172.21.15.141:0/3501903654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcc8413df80 0x7fcc84071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.120+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 >> 172.21.15.141:0/3501903654 conn(0x7fcc8406a800 msgr2=0x7fcc840b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.120+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 shutdown_connections 2024-01-16T15:08:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:09.120+0000 7fcc89c00640 1 -- 172.21.15.141:0/3501903654 wait complete. 2024-01-16T15:08:11.126 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596313 2024-01-16T15:08:11.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.436+0000 7f0de3600640 1 Processor -- start 2024-01-16T15:08:11.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.436+0000 7f0de3600640 1 -- start start 2024-01-16T15:08:11.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de3600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 0x7f0ddc0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de3600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0ddc0aede0 0x7f0ddc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de3600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc071de0 con 0x7f0ddc06a3e0 2024-01-16T15:08:11.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de3600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ddc0af270 con 0x7f0ddc0aede0 2024-01-16T15:08:11.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de3600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc072030 con 0x7f0ddc0d0b60 2024-01-16T15:08:11.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de0c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0ddc06a3e0 0x7f0ddc06a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:37508/0 (socket says 172.21.15.141:37508) 2024-01-16T15:08:11.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de0c00640 1 -- 172.21.15.141:0/2990720173 learned_addr learned my addr 172.21.15.141:0/2990720173 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:08:11.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.438+0000 7f0de1600640 1 --2- 172.21.15.141:0/2990720173 >> v2:172.21.15.141:3300/0 conn(0x7f0ddc0aede0 0x7f0ddc06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.439+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 0x7f0ddc0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.439+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034868081 0 0) 0x7f0ddc071de0 con 0x7f0ddc06a3e0 2024-01-16T15:08:11.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.439+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc003520 con 0x7f0ddc06a3e0 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.439+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3038893842 0 0) 0x7f0dbc003520 con 0x7f0ddc06a3e0 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 msgr2=0x7f0ddc0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 0x7f0ddc0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 >> v2:172.21.15.141:3300/0 conn(0x7f0ddc0aede0 msgr2=0x7f0ddc06f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 --2- 172.21.15.141:0/2990720173 >> v2:172.21.15.141:3300/0 conn(0x7f0ddc0aede0 0x7f0ddc06f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc072310 con 0x7f0ddc06a3e0 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0dcc001670 con 0x7f0ddc06a3e0 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 >> v1:172.21.15.141:6789/0 conn(0x7f0ddc06a3e0 legacy=0x7f0ddc06a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dbc003970 0x7f0dbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0dbc004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 0x7f0dbc004f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0af270 con 0x7f0dbc004360 2024-01-16T15:08:11.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc072030 con 0x7f0dbc003970 2024-01-16T15:08:11.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc00b980 con 0x7f0ddc0d0b60 2024-01-16T15:08:11.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.440+0000 7f0de1600640 1 --2- 172.21.15.141:0/2990720173 >> v2:172.21.15.141:3300/0 conn(0x7f0ddc0aede0 0x7f0ddc06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:08:11.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0dbc004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0de1600640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 0x7f0dbc004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0de0c00640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dbc003970 0x7f0dbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0ddbe00640 1 -- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 msgr2=0x7f0dbc004f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 0x7f0dbc004f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0ddbe00640 1 -- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dbc003970 msgr2=0x7f0dbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dbc003970 0x7f0dbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0ddbe00640 1 -- 172.21.15.141:0/2990720173 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc072310 con 0x7f0dbc004360 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0de1600640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0d0b60 0x7f0dbc004f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:11.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.441+0000 7f0de0c00640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dbc003970 0x7f0dbc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:11.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.443+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0dbc004840 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009a90 tx=0x7f0dbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7874120a36f2ded server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.445+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc402c6b0 con 0x7f0dbc004360 2024-01-16T15:08:11.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.445+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc402c810 con 0x7f0dbc004360 2024-01-16T15:08:11.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.445+0000 7f0ddb400640 1 -- 172.21.15.141:0/2990720173 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc402cba0 con 0x7f0dbc004360 2024-01-16T15:08:11.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.446+0000 7f0de3600640 1 -- 172.21.15.141:0/2990720173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 msgr2=0x7f0dbc004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.446+0000 7f0de3600640 1 --2- 172.21.15.141:0/2990720173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0dbc004840 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009a90 tx=0x7f0dbc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.447+0000 7f0de1600640 1 -- 172.21.15.141:0/2990720173 reap_dead start 2024-01-16T15:08:11.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.447+0000 7f0de3600640 1 -- 172.21.15.141:0/2990720173 shutdown_connections 2024-01-16T15:08:11.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.447+0000 7f0de3600640 1 -- 172.21.15.141:0/2990720173 >> 172.21.15.141:0/2990720173 conn(0x7f0ddc0cfef0 msgr2=0x7f0ddc0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.447+0000 7f0de3600640 1 -- 172.21.15.141:0/2990720173 shutdown_connections 2024-01-16T15:08:11.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.447+0000 7f0de3600640 1 -- 172.21.15.141:0/2990720173 wait complete. 2024-01-16T15:08:11.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.450+0000 7f0de3600640 1 Processor -- start 2024-01-16T15:08:11.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.450+0000 7f0de3600640 1 -- start start 2024-01-16T15:08:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.451+0000 7f0de3600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0ddc0cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.451+0000 7f0de3600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0cf6e0 0x7f0ddc13f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.451+0000 7f0de3600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ddc13fc90 0x7f0ddc140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.451+0000 7f0de3600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc13d430 con 0x7f0dbc004360 2024-01-16T15:08:11.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.451+0000 7f0de3600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0cd3f0 con 0x7f0ddc13fc90 2024-01-16T15:08:11.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.451+0000 7f0de3600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0cd6d0 con 0x7f0ddc0cf6e0 2024-01-16T15:08:11.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.452+0000 7f0de0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0ddc0cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.452+0000 7f0de0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0ddc0cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55182/0 (socket says 172.21.15.141:55182) 2024-01-16T15:08:11.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.452+0000 7f0de0c00640 1 -- 172.21.15.141:0/3967662125 learned_addr learned my addr 172.21.15.141:0/3967662125 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:11.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.452+0000 7f0de1600640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ddc13fc90 0x7f0ddc140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.452+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0cf6e0 0x7f0ddc13f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0de0c00640 1 -- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0cf6e0 msgr2=0x7f0ddc13f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0de0c00640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0cf6e0 0x7f0ddc13f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0de0c00640 1 -- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ddc13fc90 msgr2=0x7f0ddc140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0de0c00640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ddc13fc90 0x7f0ddc140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0de0c00640 1 -- 172.21.15.141:0/3967662125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc0cd9b0 con 0x7f0dbc004360 2024-01-16T15:08:11.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0de0c00640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0ddc0cf1a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0017e0 tx=0x7f0dcc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0dd8c00640 1 -- 172.21.15.141:0/3967662125 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00c870 con 0x7f0dbc004360 2024-01-16T15:08:11.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0dd8c00640 1 -- 172.21.15.141:0/3967662125 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc00c9d0 con 0x7f0dbc004360 2024-01-16T15:08:11.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.453+0000 7f0dd8c00640 1 -- 172.21.15.141:0/3967662125 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00cd60 con 0x7f0dbc004360 2024-01-16T15:08:11.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.454+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc0cdc90 con 0x7f0dbc004360 2024-01-16T15:08:11.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.454+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc159fe0 con 0x7f0dbc004360 2024-01-16T15:08:11.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.455+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ddc0ce2e0 con 0x7f0dbc004360 2024-01-16T15:08:11.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.457+0000 7f0dd8c00640 1 -- 172.21.15.141:0/3967662125 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0dcc027430 con 0x7f0dbc004360 2024-01-16T15:08:11.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.458+0000 7f0dd8c00640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0db406bf20 0x7f0db406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.459+0000 7f0dd8c00640 1 -- 172.21.15.141:0/3967662125 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0dcc088f80 con 0x7f0dbc004360 2024-01-16T15:08:11.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.459+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0db406bf20 0x7f0db406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.462+0000 7f0ddbe00640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0db406bf20 0x7f0db406e3e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4036000 tx=0x7f0dc4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.462+0000 7f0dd8c00640 1 -- 172.21.15.141:0/3967662125 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0dcc055880 con 0x7f0dbc004360 2024-01-16T15:08:11.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:11.797+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596313", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ddc06eb60 con 0x7f0db406bf20 2024-01-16T15:08:19.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.484+0000 7f0dd8c00640 1 -- 172.21.15.141:0/3967662125 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0dc40040c0 con 0x7f0db406bf20 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0db406bf20 msgr2=0x7f0db406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0db406bf20 0x7f0db406e3e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4036000 tx=0x7f0dc4030000 comp rx=0 tx=0).stop 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 msgr2=0x7f0ddc0cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0ddc0cf1a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0017e0 tx=0x7f0dcc00ecd0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 shutdown_connections 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0db406bf20 0x7f0db406e3e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ddc13fc90 0x7f0ddc140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ddc0cf6e0 0x7f0ddc13f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 --2- 172.21.15.141:0/3967662125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0dbc004360 0x7f0ddc0cf1a0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 >> 172.21.15.141:0/3967662125 conn(0x7f0ddc0cfef0 msgr2=0x7f0ddc0adb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:19.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 shutdown_connections 2024-01-16T15:08:19.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:19.488+0000 7f0de3600640 1 -- 172.21.15.141:0/3967662125 wait complete. 2024-01-16T15:08:21.490 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596314 2024-01-16T15:08:21.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.737+0000 7f8d1a400640 1 Processor -- start 2024-01-16T15:08:21.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.738+0000 7f8d1a400640 1 -- start start 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d1a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d1a400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8d140ced60 0x7f8d140d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d1a400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d140b3400 con 0x7f8d140d1d20 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d1a400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d140d1390 con 0x7f8d140ced60 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d1a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140d21b0 con 0x7f8d1415d610 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d0a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8d140ced60 0x7f8d140d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d0a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8d140ced60 0x7f8d140d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39794/0 (socket says 172.21.15.141:39794) 2024-01-16T15:08:21.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.739+0000 7f8d0a800640 1 -- 172.21.15.141:0/1244916443 learned_addr learned my addr 172.21.15.141:0/1244916443 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.740+0000 7f8d13400640 1 --2- 172.21.15.141:0/1244916443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.740+0000 7f8d0a800640 1 -- 172.21.15.141:0/1244916443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 msgr2=0x7f8d140ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.740+0000 7f8d0a800640 1 --2- 172.21.15.141:0/1244916443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.740+0000 7f8d0a800640 1 -- 172.21.15.141:0/1244916443 >> v1:172.21.15.188:6789/0 conn(0x7f8d140d1d20 legacy=0x7f8d140cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.740+0000 7f8d0a800640 1 -- 172.21.15.141:0/1244916443 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140b27f0 con 0x7f8d140ced60 2024-01-16T15:08:21.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.740+0000 7f8d13400640 1 --2- 172.21.15.141:0/1244916443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.741+0000 7f8d0a800640 1 --2- 172.21.15.141:0/1244916443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140ced60 0x7f8d140d17e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009870 tx=0x7f8cfc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4b75b21f0e3414 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.741+0000 7f8d12a00640 1 -- 172.21.15.141:0/1244916443 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc031640 con 0x7f8d140ced60 2024-01-16T15:08:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.741+0000 7f8d12a00640 1 -- 172.21.15.141:0/1244916443 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc0317a0 con 0x7f8d140ced60 2024-01-16T15:08:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.741+0000 7f8d12a00640 1 -- 172.21.15.141:0/1244916443 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc031b30 con 0x7f8d140ced60 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 -- 172.21.15.141:0/1244916443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140ced60 msgr2=0x7f8d140d17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1244916443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140ced60 0x7f8d140d17e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009870 tx=0x7f8cfc02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 -- 172.21.15.141:0/1244916443 shutdown_connections 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1244916443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140ced60 0x7f8d140d17e0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1244916443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 -- 172.21.15.141:0/1244916443 >> 172.21.15.141:0/1244916443 conn(0x7f8d1409c540 msgr2=0x7f8d140d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 -- 172.21.15.141:0/1244916443 shutdown_connections 2024-01-16T15:08:21.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.742+0000 7f8d1a400640 1 -- 172.21.15.141:0/1244916443 wait complete. 2024-01-16T15:08:21.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.744+0000 7f8d1a400640 1 Processor -- start 2024-01-16T15:08:21.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.744+0000 7f8d1a400640 1 -- start start 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d1a400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d140ced60 0x7f8d140a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d1a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140d1d20 0x7f8d140b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d1a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140b6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d1a400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140cb470 con 0x7f8d140d1d20 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d1a400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140a09c0 con 0x7f8d140ced60 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d1a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d141605c0 con 0x7f8d1415d610 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d0a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140d1d20 0x7f8d140b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d0a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140d1d20 0x7f8d140b6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39800/0 (socket says 172.21.15.141:39800) 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.745+0000 7f8d0a800640 1 -- 172.21.15.141:0/1637558110 learned_addr learned my addr 172.21.15.141:0/1637558110 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:21.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0a800640 1 -- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 msgr2=0x7f8d140b6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0a800640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0a800640 1 -- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d140ced60 msgr2=0x7f8d140a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0a800640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d140ced60 0x7f8d140a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:21.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0a800640 1 -- 172.21.15.141:0/1637558110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc0094d0 con 0x7f8d140d1d20 2024-01-16T15:08:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0a800640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140d1d20 0x7f8d140b6050 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009870 tx=0x7f8cfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0be00640 1 -- 172.21.15.141:0/1637558110 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc03ac60 con 0x7f8d140d1d20 2024-01-16T15:08:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0be00640 1 -- 172.21.15.141:0/1637558110 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc03adc0 con 0x7f8d140d1d20 2024-01-16T15:08:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d0be00640 1 -- 172.21.15.141:0/1637558110 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc044630 con 0x7f8d140d1d20 2024-01-16T15:08:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.746+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d140cab40 con 0x7f8d140d1d20 2024-01-16T15:08:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.747+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d140cb100 con 0x7f8d140d1d20 2024-01-16T15:08:21.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.748+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d140675d0 con 0x7f8d140d1d20 2024-01-16T15:08:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.749+0000 7f8d0be00640 1 -- 172.21.15.141:0/1637558110 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8cfc02f050 con 0x7f8d140d1d20 2024-01-16T15:08:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.749+0000 7f8d0be00640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8cec06bf50 0x7f8cec06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.750+0000 7f8d0be00640 1 -- 172.21.15.141:0/1637558110 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8cfc0aa7d0 con 0x7f8d140d1d20 2024-01-16T15:08:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.750+0000 7f8d13400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8cec06bf50 0x7f8cec06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.751+0000 7f8d13400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8cec06bf50 0x7f8cec06e410 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006fd0 tx=0x7f8d04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.754+0000 7f8d0be00640 1 -- 172.21.15.141:0/1637558110 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cfc076e70 con 0x7f8d140d1d20 2024-01-16T15:08:21.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:21.983+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596314", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d140735c0 con 0x7f8cec06bf50 2024-01-16T15:08:23.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.467+0000 7f8d0be00640 1 -- 172.21.15.141:0/1637558110 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8d140735c0 con 0x7f8cec06bf50 2024-01-16T15:08:23.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8cec06bf50 msgr2=0x7f8cec06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:23.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8cec06bf50 0x7f8cec06e410 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006fd0 tx=0x7f8d04008040 comp rx=0 tx=0).stop 2024-01-16T15:08:23.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140d1d20 msgr2=0x7f8d140b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:23.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140d1d20 0x7f8d140b6050 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009870 tx=0x7f8cfc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:23.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 shutdown_connections 2024-01-16T15:08:23.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8cec06bf50 0x7f8cec06e410 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:23.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d1415d610 0x7f8d140b6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:23.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d140d1d20 0x7f8d140b6050 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:23.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 --2- 172.21.15.141:0/1637558110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d140ced60 0x7f8d140a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:23.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 >> 172.21.15.141:0/1637558110 conn(0x7f8d1409c540 msgr2=0x7f8d140d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:23.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 shutdown_connections 2024-01-16T15:08:23.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:23.472+0000 7f8d1a400640 1 -- 172.21.15.141:0/1637558110 wait complete. 2024-01-16T15:08:25.485 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596315 2024-01-16T15:08:25.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.727+0000 7f2ac7400640 1 Processor -- start 2024-01-16T15:08:25.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.728+0000 7f2ac7400640 1 -- start start 2024-01-16T15:08:25.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.729+0000 7f2ac7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ac0071e40 0x7f2ac00ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.729+0000 7f2ac7400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2ac015d770 0x7f2ac015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.729+0000 7f2ac7400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac006c730 con 0x7f2ac0143810 2024-01-16T15:08:25.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.729+0000 7f2ac7400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac0143bf0 con 0x7f2ac0071e40 2024-01-16T15:08:25.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.729+0000 7f2ac7400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac00722f0 con 0x7f2ac015d770 2024-01-16T15:08:25.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac5600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ac0071e40 0x7f2ac00ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac5600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ac0071e40 0x7f2ac00ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60858/0 (socket says 172.21.15.141:60858) 2024-01-16T15:08:25.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac5600640 1 -- 172.21.15.141:0/3471450035 learned_addr learned my addr 172.21.15.141:0/3471450035 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:25.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac6a00640 1 --2- 172.21.15.141:0/3471450035 >> v2:172.21.15.188:3300/0 conn(0x7f2ac015d770 0x7f2ac015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac5600640 1 -- 172.21.15.141:0/3471450035 >> v2:172.21.15.188:3300/0 conn(0x7f2ac015d770 msgr2=0x7f2ac015dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:25.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac5600640 1 --2- 172.21.15.141:0/3471450035 >> v2:172.21.15.188:3300/0 conn(0x7f2ac015d770 0x7f2ac015dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac5600640 1 -- 172.21.15.141:0/3471450035 >> v1:172.21.15.188:6789/0 conn(0x7f2ac0143810 legacy=0x7f2ac015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac5600640 1 -- 172.21.15.141:0/3471450035 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac00724d0 con 0x7f2ac0071e40 2024-01-16T15:08:25.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.730+0000 7f2ac6a00640 1 --2- 172.21.15.141:0/3471450035 >> v2:172.21.15.188:3300/0 conn(0x7f2ac015d770 0x7f2ac015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:25.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.732+0000 7f2ac5600640 1 --2- 172.21.15.141:0/3471450035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0071e40 0x7f2ac00ae170 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00b4440 tx=0x7f2aa802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba8057a996308c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:25.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.732+0000 7f2ac4c00640 1 -- 172.21.15.141:0/3471450035 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa80317e0 con 0x7f2ac0071e40 2024-01-16T15:08:25.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.732+0000 7f2ac4c00640 1 -- 172.21.15.141:0/3471450035 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa8031940 con 0x7f2ac0071e40 2024-01-16T15:08:25.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.732+0000 7f2ac4c00640 1 -- 172.21.15.141:0/3471450035 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8031cd0 con 0x7f2ac0071e40 2024-01-16T15:08:25.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 -- 172.21.15.141:0/3471450035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0071e40 msgr2=0x7f2ac00ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 --2- 172.21.15.141:0/3471450035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0071e40 0x7f2ac00ae170 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00b4440 tx=0x7f2aa802b880 comp rx=0 tx=0).stop 2024-01-16T15:08:25.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 -- 172.21.15.141:0/3471450035 shutdown_connections 2024-01-16T15:08:25.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 --2- 172.21.15.141:0/3471450035 >> v2:172.21.15.188:3300/0 conn(0x7f2ac015d770 0x7f2ac015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 --2- 172.21.15.141:0/3471450035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0071e40 0x7f2ac00ae170 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 -- 172.21.15.141:0/3471450035 >> 172.21.15.141:0/3471450035 conn(0x7f2ac006a7d0 msgr2=0x7f2ac006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:25.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 -- 172.21.15.141:0/3471450035 shutdown_connections 2024-01-16T15:08:25.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.733+0000 7f2ac7400640 1 -- 172.21.15.141:0/3471450035 wait complete. 2024-01-16T15:08:25.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.735+0000 7f2ac7400640 1 Processor -- start 2024-01-16T15:08:25.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.735+0000 7f2ac7400640 1 -- start start 2024-01-16T15:08:25.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac0071e40 0x7f2ac00cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0143810 0x7f2ac00b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac015d770 0x7f2ac00b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac7400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac00ce340 con 0x7f2ac0143810 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac7400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac00cd730 con 0x7f2ac015d770 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac7400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac00cda10 con 0x7f2ac0071e40 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0143810 0x7f2ac00b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0143810 0x7f2ac00b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60866/0 (socket says 172.21.15.141:60866) 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac5600640 1 -- 172.21.15.141:0/1961336354 learned_addr learned my addr 172.21.15.141:0/1961336354 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac6000640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac0071e40 0x7f2ac00cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac5600640 1 -- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac0071e40 msgr2=0x7f2ac00cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac5600640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac0071e40 0x7f2ac00cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac5600640 1 -- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac015d770 msgr2=0x7f2ac00b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.736+0000 7f2ac6a00640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac015d770 0x7f2ac00b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2ac5600640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac015d770 0x7f2ac00b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2ac5600640 1 -- 172.21.15.141:0/1961336354 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa80094d0 con 0x7f2ac0143810 2024-01-16T15:08:25.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2ac5600640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0143810 0x7f2ac00b2720 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f2aa80090b0 tx=0x7f2aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:25.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa80329c0 con 0x7f2ac0143810 2024-01-16T15:08:25.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa8032b20 con 0x7f2ac0143810 2024-01-16T15:08:25.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac00cdcf0 con 0x7f2ac0143810 2024-01-16T15:08:25.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa80453f0 con 0x7f2ac0143810 2024-01-16T15:08:25.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.737+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac00cd3e0 con 0x7f2ac0143810 2024-01-16T15:08:25.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.738+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac0067450 con 0x7f2ac0143810 2024-01-16T15:08:25.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.739+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f2aa804e450 con 0x7f2ac0143810 2024-01-16T15:08:25.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.740+0000 7f2abe000640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2a9806bf20 0x7f2a9806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.740+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2aa80aaa10 con 0x7f2ac0143810 2024-01-16T15:08:25.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.741+0000 7f2ac6000640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2a9806bf20 0x7f2a9806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.742+0000 7f2ac6000640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2a9806bf20 0x7f2a9806e3e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0001640 tx=0x7f2ab0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:25.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.745+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2aa8077170 con 0x7f2ac0143810 2024-01-16T15:08:25.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.785+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2aa8077540 con 0x7f2ac0143810 2024-01-16T15:08:25.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:25.979+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596315", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ac006cfb0 con 0x7f2a9806bf20 2024-01-16T15:08:35.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.034+0000 7f2abe000640 1 -- 172.21.15.141:0/1961336354 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2ab00012c0 con 0x7f2a9806bf20 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2a9806bf20 msgr2=0x7f2a9806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2a9806bf20 0x7f2a9806e3e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0001640 tx=0x7f2ab0009040 comp rx=0 tx=0).stop 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0143810 msgr2=0x7f2ac00b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0143810 0x7f2ac00b2720 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f2aa80090b0 tx=0x7f2aa80046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 shutdown_connections 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2a9806bf20 0x7f2a9806e3e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac015d770 0x7f2ac00b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac0143810 0x7f2ac00b2720 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 --2- 172.21.15.141:0/1961336354 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac0071e40 0x7f2ac00cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 >> 172.21.15.141:0/1961336354 conn(0x7f2ac006a7d0 msgr2=0x7f2ac006abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:35.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.038+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 shutdown_connections 2024-01-16T15:08:35.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:35.039+0000 7f2ac7400640 1 -- 172.21.15.141:0/1961336354 wait complete. 2024-01-16T15:08:35.366 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:35.374 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:37.040 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596316 2024-01-16T15:08:37.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.283+0000 7f617c600640 1 Processor -- start 2024-01-16T15:08:37.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.283+0000 7f617c600640 1 -- start start 2024-01-16T15:08:37.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f617c600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413e360 0x7f61740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f617c600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f617415f9c0 0x7f61740cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f617c600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617415c5f0 con 0x7f617413df80 2024-01-16T15:08:37.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f617c600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617406c470 con 0x7f617415f9c0 2024-01-16T15:08:37.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f617c600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617406c750 con 0x7f617413e360 2024-01-16T15:08:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f617a600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f617415f9c0 0x7f61740cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f6179c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f617413df80 0x7f617415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50568/0 (socket says 172.21.15.141:50568) 2024-01-16T15:08:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f6179c00640 1 -- 172.21.15.141:0/1971369939 learned_addr learned my addr 172.21.15.141:0/1971369939 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:08:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.285+0000 7f6179200640 1 --2- 172.21.15.141:0/1971369939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413e360 0x7f61740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.286+0000 7f617a600640 1 -- 172.21.15.141:0/1971369939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413e360 msgr2=0x7f61740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.286+0000 7f617a600640 1 --2- 172.21.15.141:0/1971369939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413e360 0x7f61740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.286+0000 7f617a600640 1 -- 172.21.15.141:0/1971369939 >> v1:172.21.15.141:6789/0 conn(0x7f617413df80 legacy=0x7f617415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:37.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.286+0000 7f6173e00640 1 -- 172.21.15.141:0/1971369939 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594696444 0 0) 0x7f617415c5f0 con 0x7f617413df80 2024-01-16T15:08:37.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.286+0000 7f617a600640 1 -- 172.21.15.141:0/1971369939 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617406ca30 con 0x7f617415f9c0 2024-01-16T15:08:37.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.287+0000 7f617a600640 1 --2- 172.21.15.141:0/1971369939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617415f9c0 0x7f61740cf7d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f616c009d90 tx=0x7f616c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1129fbeb72471fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:37.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.287+0000 7f6173e00640 1 -- 172.21.15.141:0/1971369939 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c02cc50 con 0x7f617415f9c0 2024-01-16T15:08:37.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.287+0000 7f6173e00640 1 -- 172.21.15.141:0/1971369939 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c02cdb0 con 0x7f617415f9c0 2024-01-16T15:08:37.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.287+0000 7f6173e00640 1 -- 172.21.15.141:0/1971369939 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c034660 con 0x7f617415f9c0 2024-01-16T15:08:37.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 -- 172.21.15.141:0/1971369939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617415f9c0 msgr2=0x7f61740cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:37.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 --2- 172.21.15.141:0/1971369939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617415f9c0 0x7f61740cf7d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f616c009d90 tx=0x7f616c02bea0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 -- 172.21.15.141:0/1971369939 shutdown_connections 2024-01-16T15:08:37.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 --2- 172.21.15.141:0/1971369939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617415f9c0 0x7f61740cf7d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 --2- 172.21.15.141:0/1971369939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413e360 0x7f61740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 -- 172.21.15.141:0/1971369939 >> 172.21.15.141:0/1971369939 conn(0x7f617406a800 msgr2=0x7f617406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:37.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 -- 172.21.15.141:0/1971369939 shutdown_connections 2024-01-16T15:08:37.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.288+0000 7f617c600640 1 -- 172.21.15.141:0/1971369939 wait complete. 2024-01-16T15:08:37.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.290+0000 7f617c600640 1 Processor -- start 2024-01-16T15:08:37.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.290+0000 7f617c600640 1 -- start start 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f617c600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413df80 0x7f61740af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f617c600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617413e360 0x7f61740afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f617c600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617415f9c0 0x7f61741f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f617c600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741f74e0 con 0x7f617413e360 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f617c600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6174153c50 con 0x7f617415f9c0 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f617c600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61741f7200 con 0x7f617413df80 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f6179200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617413e360 0x7f61740afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f6179200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617413e360 0x7f61740afe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42858/0 (socket says 172.21.15.141:42858) 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.291+0000 7f6179200640 1 -- 172.21.15.141:0/2888845454 learned_addr learned my addr 172.21.15.141:0/2888845454 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6179c00640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413df80 0x7f61740af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f617a600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617415f9c0 0x7f61741f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6179200640 1 -- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413df80 msgr2=0x7f61740af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6179200640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413df80 0x7f61740af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6179200640 1 -- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617415f9c0 msgr2=0x7f61741f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6179200640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617415f9c0 0x7f61741f6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6179200640 1 -- 172.21.15.141:0/2888845454 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c009a70 con 0x7f617413e360 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f617a600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617415f9c0 0x7f61741f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6179200640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617413e360 0x7f61740afe60 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f615c00e9b0 tx=0x7f615c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6171600640 1 -- 172.21.15.141:0/2888845454 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c009670 con 0x7f617413e360 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6171600640 1 -- 172.21.15.141:0/2888845454 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c00c9a0 con 0x7f617413e360 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61740a8660 con 0x7f617413e360 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.292+0000 7f6171600640 1 -- 172.21.15.141:0/2888845454 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c00cd50 con 0x7f617413e360 2024-01-16T15:08:37.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.293+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61740a8c20 con 0x7f617413e360 2024-01-16T15:08:37.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.294+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61740a84c0 con 0x7f617413e360 2024-01-16T15:08:37.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.295+0000 7f6171600640 1 -- 172.21.15.141:0/2888845454 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f615c013470 con 0x7f617413e360 2024-01-16T15:08:37.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.295+0000 7f6171600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f614c06be90 0x7f614c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:37.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.296+0000 7f6171600640 1 -- 172.21.15.141:0/2888845454 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f615c088a60 con 0x7f617413e360 2024-01-16T15:08:37.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.296+0000 7f6179c00640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f614c06be90 0x7f614c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:37.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.297+0000 7f6179c00640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f614c06be90 0x7f614c06e350 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6164000f60 tx=0x7f6164007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:37.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.300+0000 7f6171600640 1 -- 172.21.15.141:0/2888845454 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61740a84c0 con 0x7f617413e360 2024-01-16T15:08:37.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:37.529+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596316", "target": ["mon-mgr", ""]}) v1 -- 0x7f617406cff0 con 0x7f614c06be90 2024-01-16T15:08:42.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.765+0000 7f6171600640 1 -- 172.21.15.141:0/2888845454 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6164007270 con 0x7f614c06be90 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f614c06be90 msgr2=0x7f614c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f614c06be90 0x7f614c06e350 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6164000f60 tx=0x7f6164007c00 comp rx=0 tx=0).stop 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617413e360 msgr2=0x7f61740afe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617413e360 0x7f61740afe60 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f615c00e9b0 tx=0x7f615c00ee70 comp rx=0 tx=0).stop 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 shutdown_connections 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f614c06be90 0x7f614c06e350 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617415f9c0 0x7f61741f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:42.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f617413e360 0x7f61740afe60 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:42.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 --2- 172.21.15.141:0/2888845454 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f617413df80 0x7f61740af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:42.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 >> 172.21.15.141:0/2888845454 conn(0x7f617406a800 msgr2=0x7f617406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:42.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 shutdown_connections 2024-01-16T15:08:42.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:42.768+0000 7f617c600640 1 -- 172.21.15.141:0/2888845454 wait complete. 2024-01-16T15:08:44.767 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596317 2024-01-16T15:08:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.085+0000 7ff58f400640 1 Processor -- start 2024-01-16T15:08:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.086+0000 7ff58f400640 1 -- start start 2024-01-16T15:08:45.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.087+0000 7ff58f400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff588157c30 0x7ff5880cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.087+0000 7ff58f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5881618f0 0x7ff588070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.087+0000 7ff58f400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff588071460 con 0x7ff588071030 2024-01-16T15:08:45.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.087+0000 7ff58f400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58815ea30 con 0x7ff588157c30 2024-01-16T15:08:45.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.087+0000 7ff58f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58806d860 con 0x7ff5881618f0 2024-01-16T15:08:45.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.087+0000 7ff58d400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff588071030 0x7ff58806dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57016/0 (socket says 172.21.15.141:57016) 2024-01-16T15:08:45.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.088+0000 7ff58d400640 1 -- 172.21.15.141:0/2088881380 learned_addr learned my addr 172.21.15.141:0/2088881380 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.088+0000 7ff587e00640 1 --2- 172.21.15.141:0/2088881380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5881618f0 0x7ff588070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.088+0000 7ff58ca00640 1 --2- 172.21.15.141:0/2088881380 >> v2:172.21.15.188:3300/0 conn(0x7ff588157c30 0x7ff5880cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.088+0000 7ff587400640 1 -- 172.21.15.141:0/2088881380 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280007174 0 0) 0x7ff588071460 con 0x7ff588071030 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.088+0000 7ff587400640 1 -- 172.21.15.141:0/2088881380 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff568003520 con 0x7ff588071030 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.089+0000 7ff587e00640 1 -- 172.21.15.141:0/2088881380 >> v2:172.21.15.188:3300/0 conn(0x7ff588157c30 msgr2=0x7ff5880cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.089+0000 7ff587e00640 1 --2- 172.21.15.141:0/2088881380 >> v2:172.21.15.188:3300/0 conn(0x7ff588157c30 0x7ff5880cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.089+0000 7ff587e00640 1 -- 172.21.15.141:0/2088881380 >> v1:172.21.15.141:6789/0 conn(0x7ff588071030 legacy=0x7ff58806dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.089+0000 7ff587e00640 1 -- 172.21.15.141:0/2088881380 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58815deb0 con 0x7ff5881618f0 2024-01-16T15:08:45.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.091+0000 7ff587e00640 1 --2- 172.21.15.141:0/2088881380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5881618f0 0x7ff588070af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff5700097a0 tx=0x7ff57002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=989f342695b72eb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:45.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.092+0000 7ff587400640 1 -- 172.21.15.141:0/2088881380 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57002d040 con 0x7ff5881618f0 2024-01-16T15:08:45.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.092+0000 7ff587400640 1 -- 172.21.15.141:0/2088881380 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff570031810 con 0x7ff5881618f0 2024-01-16T15:08:45.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.092+0000 7ff587400640 1 -- 172.21.15.141:0/2088881380 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570031ba0 con 0x7ff5881618f0 2024-01-16T15:08:45.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 -- 172.21.15.141:0/2088881380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5881618f0 msgr2=0x7ff588070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 --2- 172.21.15.141:0/2088881380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5881618f0 0x7ff588070af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff5700097a0 tx=0x7ff57002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 -- 172.21.15.141:0/2088881380 shutdown_connections 2024-01-16T15:08:45.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 --2- 172.21.15.141:0/2088881380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5881618f0 0x7ff588070af0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 --2- 172.21.15.141:0/2088881380 >> v2:172.21.15.188:3300/0 conn(0x7ff588157c30 0x7ff5880cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 -- 172.21.15.141:0/2088881380 >> 172.21.15.141:0/2088881380 conn(0x7ff5880cf700 msgr2=0x7ff5880cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:45.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 -- 172.21.15.141:0/2088881380 shutdown_connections 2024-01-16T15:08:45.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.093+0000 7ff58f400640 1 -- 172.21.15.141:0/2088881380 wait complete. 2024-01-16T15:08:45.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.098+0000 7ff58f400640 1 Processor -- start 2024-01-16T15:08:45.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.099+0000 7ff58f400640 1 -- start start 2024-01-16T15:08:45.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.100+0000 7ff58f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff588071030 0x7ff5880ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.100+0000 7ff58f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff588157c30 0x7ff5880ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.100+0000 7ff58f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5881618f0 0x7ff588158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.100+0000 7ff58f400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5881585b0 con 0x7ff5881618f0 2024-01-16T15:08:45.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.100+0000 7ff58f400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff588143a10 con 0x7ff588157c30 2024-01-16T15:08:45.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.100+0000 7ff58f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff588143cf0 con 0x7ff588071030 2024-01-16T15:08:45.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5881618f0 0x7ff588158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5881618f0 0x7ff588158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34972/0 (socket says 172.21.15.141:34972) 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 -- 172.21.15.141:0/1301487664 learned_addr learned my addr 172.21.15.141:0/1301487664 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58ca00640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff588071030 0x7ff5880ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff587e00640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff588157c30 0x7ff5880ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 -- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff588071030 msgr2=0x7ff5880ae420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff588071030 0x7ff5880ae420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 -- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff588157c30 msgr2=0x7ff5880ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff588157c30 0x7ff5880ad470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.101+0000 7ff58d400640 1 -- 172.21.15.141:0/1301487664 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff570009400 con 0x7ff5881618f0 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.102+0000 7ff58d400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5881618f0 0x7ff588158070 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff58000cae0 tx=0x7ff58000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.102+0000 7ff584c00640 1 -- 172.21.15.141:0/1301487664 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58000ac40 con 0x7ff5881618f0 2024-01-16T15:08:45.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.102+0000 7ff584c00640 1 -- 172.21.15.141:0/1301487664 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58000ada0 con 0x7ff5881618f0 2024-01-16T15:08:45.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.102+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff588143fd0 con 0x7ff5881618f0 2024-01-16T15:08:45.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.102+0000 7ff584c00640 1 -- 172.21.15.141:0/1301487664 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff580012600 con 0x7ff5881618f0 2024-01-16T15:08:45.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.102+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff588144590 con 0x7ff5881618f0 2024-01-16T15:08:45.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.104+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff588144780 con 0x7ff5881618f0 2024-01-16T15:08:45.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.104+0000 7ff584c00640 1 -- 172.21.15.141:0/1301487664 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff580007070 con 0x7ff5881618f0 2024-01-16T15:08:45.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.105+0000 7ff584c00640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff56006be90 0x7ff56006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.105+0000 7ff584c00640 1 -- 172.21.15.141:0/1301487664 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff5800885c0 con 0x7ff5881618f0 2024-01-16T15:08:45.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.106+0000 7ff58ca00640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff56006be90 0x7ff56006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.107+0000 7ff58ca00640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff56006be90 0x7ff56006e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff578006fd0 tx=0x7ff578008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:45.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.112+0000 7ff584c00640 1 -- 172.21.15.141:0/1301487664 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff580054d30 con 0x7ff5881618f0 2024-01-16T15:08:45.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:45.449+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596317", "target": ["mon-mgr", ""]}) v1 -- 0x7ff58815e470 con 0x7ff56006be90 2024-01-16T15:08:52.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.001+0000 7ff584c00640 1 -- 172.21.15.141:0/1301487664 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff57800b200 con 0x7ff56006be90 2024-01-16T15:08:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff56006be90 msgr2=0x7ff56006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff56006be90 0x7ff56006e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff578006fd0 tx=0x7ff578008040 comp rx=0 tx=0).stop 2024-01-16T15:08:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5881618f0 msgr2=0x7ff588158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5881618f0 0x7ff588158070 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff58000cae0 tx=0x7ff58000a060 comp rx=0 tx=0).stop 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 shutdown_connections 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff56006be90 0x7ff56006e350 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5881618f0 0x7ff588158070 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff588157c30 0x7ff5880ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 --2- 172.21.15.141:0/1301487664 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff588071030 0x7ff5880ae420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 >> 172.21.15.141:0/1301487664 conn(0x7ff5880cf700 msgr2=0x7ff58806e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 shutdown_connections 2024-01-16T15:08:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:53.005+0000 7ff58f400640 1 -- 172.21.15.141:0/1301487664 wait complete. 2024-01-16T15:08:55.005 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596318 2024-01-16T15:08:55.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.247+0000 7f1793600640 1 Processor -- start 2024-01-16T15:08:55.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.248+0000 7f1793600640 1 -- start start 2024-01-16T15:08:55.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.249+0000 7f1793600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 0x7f178c166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.249+0000 7f1793600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c15c5f0 con 0x7f178c13e360 2024-01-16T15:08:55.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.249+0000 7f1793600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c06c470 con 0x7f178c13df80 2024-01-16T15:08:55.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.249+0000 7f1793600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c06c750 con 0x7f178c0cf9a0 2024-01-16T15:08:55.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f1783e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f178c13e360 0x7f178c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35356/0 (socket says 172.21.15.141:35356) 2024-01-16T15:08:55.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f1783e00640 1 -- 172.21.15.141:0/1234704026 learned_addr learned my addr 172.21.15.141:0/1234704026 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:08:55.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f1791600640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 0x7f178c166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823555752 0 0) 0x7f178c15c5f0 con 0x7f178c13e360 2024-01-16T15:08:55.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c003520 con 0x7f178c13e360 2024-01-16T15:08:55.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 823480421 0 0) 0x7f176c003520 con 0x7f178c13e360 2024-01-16T15:08:55.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 msgr2=0x7f178c166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f178be00640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 0x7f178c166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.250+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 >> v1:172.21.15.188:6789/0 conn(0x7f178c13df80 legacy=0x7f178c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c06ca30 con 0x7f178c13e360 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1774001670 con 0x7f178c13e360 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 >> v1:172.21.15.141:6789/0 conn(0x7f178c13e360 legacy=0x7f178c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 0x7f176c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c003f70 0x7f176c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f176c004d00 0x7f176c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c06c750 con 0x7f176c004d00 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c00bac0 con 0x7f176c003f70 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c00bc80 con 0x7f178c0cf9a0 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f1791600640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f176c004d00 0x7f176c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f1790c00640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 0x7f176c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f1791600640 1 -- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 msgr2=0x7f176c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:55.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f1791600640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c0cf9a0 0x7f176c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f1783e00640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c003f70 0x7f176c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f1791600640 1 -- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c003f70 msgr2=0x7f176c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:55.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.251+0000 7f1791600640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c003f70 0x7f176c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.252+0000 7f1791600640 1 -- 172.21.15.141:0/1234704026 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c003520 con 0x7f176c004d00 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.253+0000 7f1791600640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f176c004d00 0x7f176c005180 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f1784009f90 tx=0x7f176c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8ced435c28e146 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.253+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178402cbf0 con 0x7f176c004d00 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.253+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178400f040 con 0x7f176c004d00 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.253+0000 7f178be00640 1 -- 172.21.15.141:0/1234704026 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784034440 con 0x7f176c004d00 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.253+0000 7f1793600640 1 -- 172.21.15.141:0/1234704026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f176c004d00 msgr2=0x7f176c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.253+0000 7f1793600640 1 --2- 172.21.15.141:0/1234704026 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f176c004d00 0x7f176c005180 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f1784009f90 tx=0x7f176c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.253+0000 7f1791600640 1 -- 172.21.15.141:0/1234704026 reap_dead start 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.254+0000 7f1793600640 1 -- 172.21.15.141:0/1234704026 shutdown_connections 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.254+0000 7f1793600640 1 -- 172.21.15.141:0/1234704026 >> 172.21.15.141:0/1234704026 conn(0x7f178c06a800 msgr2=0x7f178c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.254+0000 7f1793600640 1 -- 172.21.15.141:0/1234704026 shutdown_connections 2024-01-16T15:08:55.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.254+0000 7f1793600640 1 -- 172.21.15.141:0/1234704026 wait complete. 2024-01-16T15:08:55.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.255+0000 7f1793600640 1 Processor -- start 2024-01-16T15:08:55.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.256+0000 7f1793600640 1 -- start start 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.256+0000 7f1793600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c004d00 0x7f178c1f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.256+0000 7f1793600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f178c13d7b0 0x7f178c152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1793600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c13d1f0 0x7f178c1530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1793600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c14bae0 con 0x7f178c13d7b0 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1793600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c14b560 con 0x7f176c004d00 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1793600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c14b800 con 0x7f178c13d1f0 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f178c13d7b0 0x7f178c152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f178c13d7b0 0x7f178c152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57162/0 (socket says 172.21.15.141:57162) 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 -- 172.21.15.141:0/1907327733 learned_addr learned my addr 172.21.15.141:0/1907327733 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 -- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c13d1f0 msgr2=0x7f178c1530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1791600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c13d1f0 0x7f178c1530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1790c00640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c004d00 0x7f178c1f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c13d1f0 0x7f178c1530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 -- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c004d00 msgr2=0x7f178c1f0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c004d00 0x7f178c1f0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.257+0000 7f1783e00640 1 -- 172.21.15.141:0/1907327733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c14e1d0 con 0x7f178c13d7b0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.258+0000 7f1783e00640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f178c13d7b0 0x7f178c152ba0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f177400d810 tx=0x7f177400dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.258+0000 7f1789600640 1 -- 172.21.15.141:0/1907327733 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17740074e0 con 0x7f178c13d7b0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.258+0000 7f1789600640 1 -- 172.21.15.141:0/1907327733 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1774007640 con 0x7f178c13d7b0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.258+0000 7f1789600640 1 -- 172.21.15.141:0/1907327733 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1774004a00 con 0x7f178c13d7b0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.258+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c14c990 con 0x7f178c13d7b0 2024-01-16T15:08:55.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.258+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c1f4700 con 0x7f178c13d7b0 2024-01-16T15:08:55.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.259+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c14e030 con 0x7f178c13d7b0 2024-01-16T15:08:55.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.260+0000 7f1789600640 1 -- 172.21.15.141:0/1907327733 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1774004b60 con 0x7f178c13d7b0 2024-01-16T15:08:55.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.261+0000 7f1789600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f176406bec0 0x7f176406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.261+0000 7f1789600640 1 -- 172.21.15.141:0/1907327733 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1774087f30 con 0x7f178c13d7b0 2024-01-16T15:08:55.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.262+0000 7f1790c00640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f176406bec0 0x7f176406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.262+0000 7f1790c00640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f176406bec0 0x7f176406e380 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f177c007fd0 tx=0x7f177c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:55.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.266+0000 7f1789600640 1 -- 172.21.15.141:0/1907327733 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f178c14e030 con 0x7f178c13d7b0 2024-01-16T15:08:55.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:55.493+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596318", "target": ["mon-mgr", ""]}) v1 -- 0x7f178c06cff0 con 0x7f176406bec0 2024-01-16T15:08:57.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.038+0000 7f1789600640 1 -- 172.21.15.141:0/1907327733 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f177c00d050 con 0x7f176406bec0 2024-01-16T15:08:57.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.047+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f176406bec0 msgr2=0x7f176406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f176406bec0 0x7f176406e380 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f177c007fd0 tx=0x7f177c009040 comp rx=0 tx=0).stop 2024-01-16T15:08:57.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f178c13d7b0 msgr2=0x7f178c152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f178c13d7b0 0x7f178c152ba0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f177400d810 tx=0x7f177400dcd0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 shutdown_connections 2024-01-16T15:08:57.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f176406bec0 0x7f176406e380 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f178c13d1f0 0x7f178c1530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f178c13d7b0 0x7f178c152ba0 secure :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f177400d810 tx=0x7f177400dcd0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 --2- 172.21.15.141:0/1907327733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176c004d00 0x7f178c1f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.048+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 >> 172.21.15.141:0/1907327733 conn(0x7f178c06a800 msgr2=0x7f178c0b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:57.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.049+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 shutdown_connections 2024-01-16T15:08:57.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:57.050+0000 7f1793600640 1 -- 172.21.15.141:0/1907327733 wait complete. 2024-01-16T15:08:59.058 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596319 2024-01-16T15:08:59.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.301+0000 7fba0f600640 1 Processor -- start 2024-01-16T15:08:59.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.301+0000 7fba0f600640 1 -- start start 2024-01-16T15:08:59.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.303+0000 7fba0f600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fba080cf9a0 0x7fba08166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.303+0000 7fba0f600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0815c5f0 con 0x7fba0813df80 2024-01-16T15:08:59.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.303+0000 7fba0f600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0806c470 con 0x7fba0813e360 2024-01-16T15:08:59.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.303+0000 7fba0f600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0806c750 con 0x7fba080cf9a0 2024-01-16T15:08:59.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.304+0000 7fba0cc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fba0813df80 0x7fba0815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:35368/0 (socket says 172.21.15.141:35368) 2024-01-16T15:08:59.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.304+0000 7fba0cc00640 1 -- 172.21.15.141:0/2949473155 learned_addr learned my addr 172.21.15.141:0/2949473155 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:08:59.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.304+0000 7fba0d600640 1 --2- 172.21.15.141:0/2949473155 >> v2:172.21.15.188:3300/0 conn(0x7fba080cf9a0 0x7fba08166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.304+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258278283 0 0) 0x7fba0815c5f0 con 0x7fba0813df80 2024-01-16T15:08:59.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.304+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e8003520 con 0x7fba0813df80 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044377876 0 0) 0x7fb9e8003520 con 0x7fba0813df80 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 >> v2:172.21.15.188:3300/0 conn(0x7fba080cf9a0 msgr2=0x7fba08166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 --2- 172.21.15.141:0/2949473155 >> v2:172.21.15.188:3300/0 conn(0x7fba080cf9a0 0x7fba08166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 >> v1:172.21.15.188:6789/0 conn(0x7fba0813e360 legacy=0x7fba080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0806ca30 con 0x7fba0813df80 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb9f8001670 con 0x7fba0813df80 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 >> v1:172.21.15.141:6789/0 conn(0x7fba0813df80 legacy=0x7fba0815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e8003970 0x7fb9e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 0x7fb9e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8005180 0x7fb9e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0806c750 con 0x7fb9e8003970 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e800c020 con 0x7fb9e80047c0 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e800c1e0 con 0x7fb9e8005180 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba0cc00640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e8003970 0x7fb9e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba07e00640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 0x7fb9e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba0cc00640 1 -- 172.21.15.141:0/2949473155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8005180 msgr2=0x7fb9e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba0cc00640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8005180 0x7fb9e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba0cc00640 1 -- 172.21.15.141:0/2949473155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 msgr2=0x7fb9e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.305+0000 7fba0cc00640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9e80047c0 0x7fb9e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.306+0000 7fba0cc00640 1 -- 172.21.15.141:0/2949473155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8003520 con 0x7fb9e8003970 2024-01-16T15:08:59.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.306+0000 7fba0d600640 1 -- 172.21.15.141:0/2949473155 reap_dead start 2024-01-16T15:08:59.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba0cc00640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e8003970 0x7fb9e8003d90 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c240 tx=0x7fb9e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9473f7b04181c63 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8005ad0 con 0x7fb9e8003970 2024-01-16T15:08:59.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f8005c30 con 0x7fb9e8003970 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba07400640 1 -- 172.21.15.141:0/2949473155 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f802cab0 con 0x7fb9e8003970 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba0f600640 1 -- 172.21.15.141:0/2949473155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e8003970 msgr2=0x7fb9e8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba0f600640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e8003970 0x7fb9e8003d90 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c240 tx=0x7fb9e80057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba0f600640 1 -- 172.21.15.141:0/2949473155 shutdown_connections 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba0f600640 1 --2- 172.21.15.141:0/2949473155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9e8003970 0x7fb9e8003d90 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.307+0000 7fba0f600640 1 -- 172.21.15.141:0/2949473155 >> 172.21.15.141:0/2949473155 conn(0x7fba0806a800 msgr2=0x7fba0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.308+0000 7fba0f600640 1 -- 172.21.15.141:0/2949473155 shutdown_connections 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.308+0000 7fba0f600640 1 -- 172.21.15.141:0/2949473155 wait complete. 2024-01-16T15:08:59.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.309+0000 7fba0f600640 1 Processor -- start 2024-01-16T15:08:59.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.309+0000 7fba0f600640 1 -- start start 2024-01-16T15:08:59.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.310+0000 7fba0f600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8003970 0x7fba0814e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.310+0000 7fba0f600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba080b2f60 0x7fba0813cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.310+0000 7fba0f600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba080b33e0 0x7fba0813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.310+0000 7fba0f600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0814ebc0 con 0x7fba080b33e0 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.310+0000 7fba0f600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0814eea0 con 0x7fba080b2f60 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.310+0000 7fba0f600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0814bda0 con 0x7fb9e8003970 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba080b33e0 0x7fba0813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba080b33e0 0x7fba0813d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57174/0 (socket says 172.21.15.141:57174) 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 -- 172.21.15.141:0/2267193511 learned_addr learned my addr 172.21.15.141:0/2267193511 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba07e00640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba080b2f60 0x7fba0813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0cc00640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8003970 0x7fba0814e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 -- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8003970 msgr2=0x7fba0814e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8003970 0x7fba0814e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 -- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba080b2f60 msgr2=0x7fba0813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba080b2f60 0x7fba0813cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 -- 172.21.15.141:0/2267193511 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0814c080 con 0x7fba080b33e0 2024-01-16T15:08:59.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.311+0000 7fba0d600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba080b33e0 0x7fba0813d4b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fba080ccae0 tx=0x7fba0000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.312+0000 7fba04c00640 1 -- 172.21.15.141:0/2267193511 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00010040 con 0x7fba080b33e0 2024-01-16T15:08:59.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.312+0000 7fba04c00640 1 -- 172.21.15.141:0/2267193511 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0000bb70 con 0x7fba080b33e0 2024-01-16T15:08:59.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.312+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0814f180 con 0x7fba080b33e0 2024-01-16T15:08:59.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.312+0000 7fba04c00640 1 -- 172.21.15.141:0/2267193511 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00014400 con 0x7fba080b33e0 2024-01-16T15:08:59.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.312+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0815ae50 con 0x7fba080b33e0 2024-01-16T15:08:59.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.313+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0814c2c0 con 0x7fba080b33e0 2024-01-16T15:08:59.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.314+0000 7fba04c00640 1 -- 172.21.15.141:0/2267193511 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fba00014560 con 0x7fba080b33e0 2024-01-16T15:08:59.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.315+0000 7fba04c00640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006be90 0x7fb9e006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:59.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.315+0000 7fba04c00640 1 -- 172.21.15.141:0/2267193511 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba00088810 con 0x7fba080b33e0 2024-01-16T15:08:59.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.315+0000 7fba0cc00640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006be90 0x7fb9e006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:59.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.317+0000 7fba0cc00640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006be90 0x7fb9e006e350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8034000 tx=0x7fb9f8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:59.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.319+0000 7fba04c00640 1 -- 172.21.15.141:0/2267193511 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba0814c2c0 con 0x7fba080b33e0 2024-01-16T15:08:59.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:08:59.545+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596319", "target": ["mon-mgr", ""]}) v1 -- 0x7fba0815b070 con 0x7fb9e006be90 2024-01-16T15:09:05.514 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:05.521 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:08.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.465+0000 7fba04c00640 1 -- 172.21.15.141:0/2267193511 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fba0815b070 con 0x7fb9e006be90 2024-01-16T15:09:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006be90 msgr2=0x7fb9e006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006be90 0x7fb9e006e350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8034000 tx=0x7fb9f8009030 comp rx=0 tx=0).stop 2024-01-16T15:09:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba080b33e0 msgr2=0x7fba0813d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba080b33e0 0x7fba0813d4b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fba080ccae0 tx=0x7fba0000ef30 comp rx=0 tx=0).stop 2024-01-16T15:09:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 shutdown_connections 2024-01-16T15:09:08.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb9e006be90 0x7fb9e006e350 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba080b33e0 0x7fba0813d4b0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba080b2f60 0x7fba0813cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 --2- 172.21.15.141:0/2267193511 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb9e8003970 0x7fba0814e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 >> 172.21.15.141:0/2267193511 conn(0x7fba0806a800 msgr2=0x7fba08153a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 shutdown_connections 2024-01-16T15:09:08.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:08.469+0000 7fba0f600640 1 -- 172.21.15.141:0/2267193511 wait complete. 2024-01-16T15:09:10.471 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone596320 2024-01-16T15:09:10.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.717+0000 7ff217200640 1 Processor -- start 2024-01-16T15:09:10.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.717+0000 7ff217200640 1 -- start start 2024-01-16T15:09:10.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff217200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff210157c30 0x7ff2100cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff217200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2101618f0 0x7ff210070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff217200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff210071460 con 0x7ff210071030 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff217200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21015ea30 con 0x7ff210157c30 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff217200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21006d860 con 0x7ff2101618f0 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff210157c30 0x7ff2100cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff210157c30 0x7ff2100cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42272/0 (socket says 172.21.15.141:42272) 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20fe00640 1 -- 172.21.15.141:0/3415444837 learned_addr learned my addr 172.21.15.141:0/3415444837 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20f400640 1 --2- 172.21.15.141:0/3415444837 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2101618f0 0x7ff210070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20fe00640 1 -- 172.21.15.141:0/3415444837 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2101618f0 msgr2=0x7ff210070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20fe00640 1 --2- 172.21.15.141:0/3415444837 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2101618f0 0x7ff210070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20fe00640 1 -- 172.21.15.141:0/3415444837 >> v1:172.21.15.188:6789/0 conn(0x7ff210071030 legacy=0x7ff21006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.719+0000 7ff20fe00640 1 -- 172.21.15.141:0/3415444837 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21015deb0 con 0x7ff210157c30 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.721+0000 7ff20fe00640 1 --2- 172.21.15.141:0/3415444837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2100cd600 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff2100723f0 tx=0x7ff20002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a4d7e464241759 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.721+0000 7ff20ea00640 1 -- 172.21.15.141:0/3415444837 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200031830 con 0x7ff210157c30 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.721+0000 7ff20ea00640 1 -- 172.21.15.141:0/3415444837 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200031990 con 0x7ff210157c30 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.721+0000 7ff20ea00640 1 -- 172.21.15.141:0/3415444837 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200031d20 con 0x7ff210157c30 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.721+0000 7ff217200640 1 -- 172.21.15.141:0/3415444837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 msgr2=0x7ff2100cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.721+0000 7ff217200640 1 --2- 172.21.15.141:0/3415444837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2100cd600 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff2100723f0 tx=0x7ff20002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.722+0000 7ff217200640 1 -- 172.21.15.141:0/3415444837 shutdown_connections 2024-01-16T15:09:10.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.722+0000 7ff217200640 1 --2- 172.21.15.141:0/3415444837 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff2101618f0 0x7ff210070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.722+0000 7ff217200640 1 --2- 172.21.15.141:0/3415444837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2100cd600 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.722+0000 7ff217200640 1 -- 172.21.15.141:0/3415444837 >> 172.21.15.141:0/3415444837 conn(0x7ff2100cf700 msgr2=0x7ff2100cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.722+0000 7ff217200640 1 -- 172.21.15.141:0/3415444837 shutdown_connections 2024-01-16T15:09:10.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.722+0000 7ff217200640 1 -- 172.21.15.141:0/3415444837 wait complete. 2024-01-16T15:09:10.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.723+0000 7ff217200640 1 Processor -- start 2024-01-16T15:09:10.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.724+0000 7ff217200640 1 -- start start 2024-01-16T15:09:10.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff217200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff210071030 0x7ff210158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff217200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2101e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff217200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2101618f0 0x7ff2101e8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff217200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100ae930 con 0x7ff210157c30 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff217200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100addb0 con 0x7ff2101618f0 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff217200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100ae090 con 0x7ff210071030 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff20f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2101e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff20f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2101e7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42288/0 (socket says 172.21.15.141:42288) 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff20f400640 1 -- 172.21.15.141:0/3511250141 learned_addr learned my addr 172.21.15.141:0/3511250141 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff215200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2101618f0 0x7ff2101e8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff20f400640 1 -- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff210071030 msgr2=0x7ff210158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff20f400640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff210071030 0x7ff210158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff20f400640 1 -- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2101618f0 msgr2=0x7ff2101e8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.725+0000 7ff20f400640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2101618f0 0x7ff2101e8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff20f400640 1 -- 172.21.15.141:0/3511250141 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2000094d0 con 0x7ff210157c30 2024-01-16T15:09:10.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff215200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2101618f0 0x7ff2101e8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:10.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff20f400640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2101e7e40 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800c7f0 tx=0x7ff1f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff207e00640 1 -- 172.21.15.141:0/3511250141 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8004220 con 0x7ff210157c30 2024-01-16T15:09:10.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff207e00640 1 -- 172.21.15.141:0/3511250141 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f80043a0 con 0x7ff210157c30 2024-01-16T15:09:10.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff207e00640 1 -- 172.21.15.141:0/3511250141 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8007860 con 0x7ff210157c30 2024-01-16T15:09:10.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2100ae370 con 0x7ff210157c30 2024-01-16T15:09:10.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.726+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2101585b0 con 0x7ff210157c30 2024-01-16T15:09:10.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.727+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2100675d0 con 0x7ff210157c30 2024-01-16T15:09:10.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.728+0000 7ff207e00640 1 -- 172.21.15.141:0/3511250141 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff1f8007bd0 con 0x7ff210157c30 2024-01-16T15:09:10.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.730+0000 7ff207e00640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1e806be90 0x7ff1e806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.730+0000 7ff207e00640 1 -- 172.21.15.141:0/3511250141 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff1f8088530 con 0x7ff210157c30 2024-01-16T15:09:10.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.730+0000 7ff20fe00640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1e806be90 0x7ff1e806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.731+0000 7ff20fe00640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1e806be90 0x7ff1e806e350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff200036000 tx=0x7ff200030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.735+0000 7ff207e00640 1 -- 172.21.15.141:0/3511250141 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1f8054df0 con 0x7ff210157c30 2024-01-16T15:09:10.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:10.962+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone596320", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2100ae200 con 0x7ff1e806be90 2024-01-16T15:09:16.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.492+0000 7ff207e00640 1 -- 172.21.15.141:0/3511250141 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff2100ae200 con 0x7ff1e806be90 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1e806be90 msgr2=0x7ff1e806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1e806be90 0x7ff1e806e350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff200036000 tx=0x7ff200030000 comp rx=0 tx=0).stop 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 msgr2=0x7ff2101e7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2101e7e40 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800c7f0 tx=0x7ff1f800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 shutdown_connections 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1e806be90 0x7ff1e806e350 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2101618f0 0x7ff2101e8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff210157c30 0x7ff2101e7e40 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 --2- 172.21.15.141:0/3511250141 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff210071030 0x7ff210158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 >> 172.21.15.141:0/3511250141 conn(0x7ff2100cf700 msgr2=0x7ff210156f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:16.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.495+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 shutdown_connections 2024-01-16T15:09:16.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:16.497+0000 7ff217200640 1 -- 172.21.15.141:0/3511250141 wait complete. 2024-01-16T15:09:18.495 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:18.496 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59631 2024-01-16T15:09:18.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.740+0000 7f3ab8200640 1 Processor -- start 2024-01-16T15:09:18.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.740+0000 7f3ab8200640 1 -- start start 2024-01-16T15:09:18.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab8200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3ab0157c30 0x7f3ab00cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab8200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3ab01618f0 0x7f3ab0070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab8200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab8200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab015ea30 con 0x7f3ab0157c30 2024-01-16T15:09:18.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab8200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab006d860 con 0x7f3ab01618f0 2024-01-16T15:09:18.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab8200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab0071450 con 0x7f3ab0071030 2024-01-16T15:09:18.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab5800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3ab0157c30 0x7f3ab00cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab5800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3ab0157c30 0x7f3ab00cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58814/0 (socket says 172.21.15.141:58814) 2024-01-16T15:09:18.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab5800640 1 -- 172.21.15.141:0/4109391825 learned_addr learned my addr 172.21.15.141:0/4109391825 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:18.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.742+0000 7f3ab4e00640 1 --2- 172.21.15.141:0/4109391825 >> v2:172.21.15.188:3300/0 conn(0x7f3ab01618f0 0x7f3ab0070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.743+0000 7f3ab6200640 1 --2- 172.21.15.141:0/4109391825 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.743+0000 7f3ab5800640 1 -- 172.21.15.141:0/4109391825 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 msgr2=0x7f3ab006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:18.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.743+0000 7f3ab5800640 1 --2- 172.21.15.141:0/4109391825 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab006dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.743+0000 7f3ab5800640 1 -- 172.21.15.141:0/4109391825 >> v2:172.21.15.188:3300/0 conn(0x7f3ab01618f0 msgr2=0x7f3ab0070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:18.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.743+0000 7f3ab5800640 1 --2- 172.21.15.141:0/4109391825 >> v2:172.21.15.188:3300/0 conn(0x7f3ab01618f0 0x7f3ab0070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.743+0000 7f3ab5800640 1 -- 172.21.15.141:0/4109391825 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab015deb0 con 0x7f3ab0157c30 2024-01-16T15:09:18.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.744+0000 7f3ab5800640 1 --2- 172.21.15.141:0/4109391825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00cd600 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0066670 tx=0x7f3aa002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8d93ff67ac4d30 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:18.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.744+0000 7f3aafe00640 1 -- 172.21.15.141:0/4109391825 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0031870 con 0x7f3ab0157c30 2024-01-16T15:09:18.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3aafe00640 1 -- 172.21.15.141:0/4109391825 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa00319d0 con 0x7f3ab0157c30 2024-01-16T15:09:18.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3aafe00640 1 -- 172.21.15.141:0/4109391825 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0031d60 con 0x7f3ab0157c30 2024-01-16T15:09:18.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 -- 172.21.15.141:0/4109391825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 msgr2=0x7f3ab00cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 --2- 172.21.15.141:0/4109391825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00cd600 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0066670 tx=0x7f3aa002b910 comp rx=0 tx=0).stop 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 -- 172.21.15.141:0/4109391825 shutdown_connections 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 --2- 172.21.15.141:0/4109391825 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab006dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 --2- 172.21.15.141:0/4109391825 >> v2:172.21.15.188:3300/0 conn(0x7f3ab01618f0 0x7f3ab0070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 --2- 172.21.15.141:0/4109391825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00cd600 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 -- 172.21.15.141:0/4109391825 >> 172.21.15.141:0/4109391825 conn(0x7f3ab00cf700 msgr2=0x7f3ab00cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 -- 172.21.15.141:0/4109391825 shutdown_connections 2024-01-16T15:09:18.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.745+0000 7f3ab8200640 1 -- 172.21.15.141:0/4109391825 wait complete. 2024-01-16T15:09:18.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.747+0000 7f3ab8200640 1 Processor -- start 2024-01-16T15:09:18.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.748+0000 7f3ab8200640 1 -- start start 2024-01-16T15:09:18.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab8200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab01f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab8200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab8200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ab01618f0 0x7f3ab0148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab8200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab01f8050 con 0x7f3ab0157c30 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab8200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab0146fc0 con 0x7f3ab01618f0 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab8200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab01472a0 con 0x7f3ab0071030 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58826/0 (socket says 172.21.15.141:58826) 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.749+0000 7f3ab4e00640 1 -- 172.21.15.141:0/3624833500 learned_addr learned my addr 172.21.15.141:0/3624833500 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab5800640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab01f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab4e00640 1 -- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 msgr2=0x7f3ab01f7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab4e00640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab01f7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab6200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ab01618f0 0x7f3ab0148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab4e00640 1 -- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ab01618f0 msgr2=0x7f3ab0148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab4e00640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ab01618f0 0x7f3ab0148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab4e00640 1 -- 172.21.15.141:0/3624833500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa00094d0 con 0x7f3ab0157c30 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab6200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ab01618f0 0x7f3ab0148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab4e00640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00afb10 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d800 tx=0x7f3a9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3aad600640 1 -- 172.21.15.141:0/3624833500 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800b6c0 con 0x7f3ab0157c30 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3aad600640 1 -- 172.21.15.141:0/3624833500 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9800b820 con 0x7f3ab0157c30 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3aad600640 1 -- 172.21.15.141:0/3624833500 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800bbb0 con 0x7f3ab0157c30 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.750+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab0147580 con 0x7f3ab0157c30 2024-01-16T15:09:18.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.751+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab0147b40 con 0x7f3ab0157c30 2024-01-16T15:09:18.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.752+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab00675d0 con 0x7f3ab0157c30 2024-01-16T15:09:18.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.752+0000 7f3aad600640 1 -- 172.21.15.141:0/3624833500 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3a9800bd10 con 0x7f3ab0157c30 2024-01-16T15:09:18.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.753+0000 7f3aad600640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a8806be90 0x7f3a8806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.753+0000 7f3aad600640 1 -- 172.21.15.141:0/3624833500 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3a98088d50 con 0x7f3ab0157c30 2024-01-16T15:09:18.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.754+0000 7f3ab5800640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a8806be90 0x7f3a8806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.755+0000 7f3ab5800640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a8806be90 0x7f3a8806e350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0036000 tx=0x7f3aa003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:18.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.761+0000 7f3aad600640 1 -- 172.21.15.141:0/3624833500 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a98055410 con 0x7f3ab0157c30 2024-01-16T15:09:18.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:18.990+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59631", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ab015e750 con 0x7f3a8806be90 2024-01-16T15:09:18.979 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:19.002+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:18.979 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:19.002+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:18.979 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:19.002+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:18.979 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:19.002+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:18.979 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:19.002+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:19.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.034+0000 7f3aad600640 1 -- 172.21.15.141:0/3624833500 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f3aa0031bc0 con 0x7f3a8806be90 2024-01-16T15:09:19.011 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:09:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:19.012 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:19.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.037+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a8806be90 msgr2=0x7f3a8806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.037+0000 7f3ab8200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a8806be90 0x7f3a8806e350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0036000 tx=0x7f3aa003c040 comp rx=0 tx=0).stop 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.037+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 msgr2=0x7f3ab00afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.037+0000 7f3ab8200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00afb10 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d800 tx=0x7f3a9800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 shutdown_connections 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a8806be90 0x7f3a8806e350 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ab01618f0 0x7f3ab0148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3ab0157c30 0x7f3ab00afb10 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 --2- 172.21.15.141:0/3624833500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3ab0071030 0x7f3ab01f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 >> 172.21.15.141:0/3624833500 conn(0x7f3ab00cf700 msgr2=0x7f3ab0157190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 shutdown_connections 2024-01-16T15:09:19.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.038+0000 7f3ab8200640 1 -- 172.21.15.141:0/3624833500 wait complete. 2024-01-16T15:09:19.034 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:19.034 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59632 2024-01-16T15:09:19.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.269+0000 7fb796200640 1 Processor -- start 2024-01-16T15:09:19.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.269+0000 7fb796200640 1 -- start start 2024-01-16T15:09:19.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb796200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 0x7fb79015a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb796200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7900b3fb0 con 0x7fb79006d6d0 2024-01-16T15:09:19.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb796200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7900b1e30 con 0x7fb7901505b0 2024-01-16T15:09:19.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb796200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79015abd0 con 0x7fb7900b3b20 2024-01-16T15:09:19.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb78fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb79006d6d0 0x7fb7900ccc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43940/0 (socket says 172.21.15.141:43940) 2024-01-16T15:09:19.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb78fe00640 1 -- 172.21.15.141:0/3921144202 learned_addr learned my addr 172.21.15.141:0/3921144202 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:19.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb795800640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 0x7fb79015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.271+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232646509 0 0) 0x7fb7900b3fb0 con 0x7fb79006d6d0 2024-01-16T15:09:19.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb770003520 con 0x7fb79006d6d0 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354439523 0 0) 0x7fb770003520 con 0x7fb79006d6d0 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 msgr2=0x7fb79015a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 0x7fb79015a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 >> v1:172.21.15.188:6789/0 conn(0x7fb7901505b0 legacy=0x7fb79006cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7900b20a0 con 0x7fb79006d6d0 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb778001670 con 0x7fb79006d6d0 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 >> v1:172.21.15.141:6789/0 conn(0x7fb79006d6d0 legacy=0x7fb7900ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700039a0 0x7fb770003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7700047c0 0x7fb770004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 0x7fb7700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79015abd0 con 0x7fb7700047c0 2024-01-16T15:09:19.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77000bae0 con 0x7fb7700039a0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77000bca0 con 0x7fb7900b3b20 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.272+0000 7fb78fe00640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7700047c0 0x7fb770004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.273+0000 7fb794e00640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700039a0 0x7fb770003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.273+0000 7fb795800640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 0x7fb7700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.273+0000 7fb78fe00640 1 -- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 msgr2=0x7fb7700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.273+0000 7fb78fe00640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7900b3b20 0x7fb7700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.273+0000 7fb78fe00640 1 -- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700039a0 msgr2=0x7fb770003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.273+0000 7fb78fe00640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700039a0 0x7fb770003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.273+0000 7fb78fe00640 1 -- 172.21.15.141:0/3921144202 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770003520 con 0x7fb7700047c0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.274+0000 7fb78fe00640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7700047c0 0x7fb770004c40 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb77800c210 tx=0x7fb7700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c7e266dea48cfb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.274+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778005ad0 con 0x7fb7700047c0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.274+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb778005c30 con 0x7fb7700047c0 2024-01-16T15:09:19.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb78f400640 1 -- 172.21.15.141:0/3921144202 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77802c890 con 0x7fb7700047c0 2024-01-16T15:09:19.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb796200640 1 -- 172.21.15.141:0/3921144202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7700047c0 msgr2=0x7fb770004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb796200640 1 --2- 172.21.15.141:0/3921144202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7700047c0 0x7fb770004c40 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb77800c210 tx=0x7fb7700057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb795800640 1 -- 172.21.15.141:0/3921144202 reap_dead start 2024-01-16T15:09:19.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb796200640 1 -- 172.21.15.141:0/3921144202 shutdown_connections 2024-01-16T15:09:19.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb796200640 1 -- 172.21.15.141:0/3921144202 >> 172.21.15.141:0/3921144202 conn(0x7fb79006a7d0 msgr2=0x7fb79006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb796200640 1 -- 172.21.15.141:0/3921144202 shutdown_connections 2024-01-16T15:09:19.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.275+0000 7fb796200640 1 -- 172.21.15.141:0/3921144202 wait complete. 2024-01-16T15:09:19.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.277+0000 7fb796200640 1 Processor -- start 2024-01-16T15:09:19.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.277+0000 7fb796200640 1 -- start start 2024-01-16T15:09:19.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb796200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700047c0 0x7fb790150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb796200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb79013e6d0 0x7fb79013d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb796200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb79013e180 0x7fb7901e51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb796200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7901511d0 con 0x7fb79013e180 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb796200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790070840 con 0x7fb7700047c0 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb796200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb790070b20 con 0x7fb79013e6d0 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb795800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb79013e180 0x7fb7901e51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb795800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb79013e180 0x7fb7901e51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58854/0 (socket says 172.21.15.141:58854) 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.278+0000 7fb795800640 1 -- 172.21.15.141:0/1795638204 learned_addr learned my addr 172.21.15.141:0/1795638204 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb78fe00640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb79013e6d0 0x7fb79013d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb794e00640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700047c0 0x7fb790150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb795800640 1 -- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb79013e6d0 msgr2=0x7fb79013d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb795800640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb79013e6d0 0x7fb79013d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb795800640 1 -- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700047c0 msgr2=0x7fb790150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb795800640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700047c0 0x7fb790150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb795800640 1 -- 172.21.15.141:0/1795638204 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb790070e00 con 0x7fb79013e180 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb795800640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb79013e180 0x7fb7901e51a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb79006e4c0 tx=0x7fb788008e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb78cc00640 1 -- 172.21.15.141:0/1795638204 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78800f050 con 0x7fb79013e180 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb78cc00640 1 -- 172.21.15.141:0/1795638204 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb788010040 con 0x7fb79013e180 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb78cc00640 1 -- 172.21.15.141:0/1795638204 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788007e70 con 0x7fb79013e180 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.279+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7900710e0 con 0x7fb79013e180 2024-01-16T15:09:19.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.280+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7900aeb40 con 0x7fb79013e180 2024-01-16T15:09:19.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.281+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7900712d0 con 0x7fb79013e180 2024-01-16T15:09:19.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.281+0000 7fb78cc00640 1 -- 172.21.15.141:0/1795638204 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb78801f460 con 0x7fb79013e180 2024-01-16T15:09:19.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.282+0000 7fb78cc00640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb76806bec0 0x7fb76806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.283+0000 7fb78cc00640 1 -- 172.21.15.141:0/1795638204 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb788088ab0 con 0x7fb79013e180 2024-01-16T15:09:19.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.283+0000 7fb794e00640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb76806bec0 0x7fb76806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.284+0000 7fb794e00640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb76806bec0 0x7fb76806e380 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb780007fd0 tx=0x7fb780009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.288+0000 7fb78cc00640 1 -- 172.21.15.141:0/1795638204 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb788055370 con 0x7fb79013e180 2024-01-16T15:09:19.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.519+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59632", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7900b41e0 con 0x7fb76806bec0 2024-01-16T15:09:19.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.543+0000 7fb78cc00640 1 -- 172.21.15.141:0/1795638204 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fb78000d050 con 0x7fb76806bec0 2024-01-16T15:09:19.520 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:19.520 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:19.520 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:09:19.520 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:19.521 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:19.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb76806bec0 msgr2=0x7fb76806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb76806bec0 0x7fb76806e380 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb780007fd0 tx=0x7fb780009040 comp rx=0 tx=0).stop 2024-01-16T15:09:19.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb79013e180 msgr2=0x7fb7901e51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb79013e180 0x7fb7901e51a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb79006e4c0 tx=0x7fb788008e70 comp rx=0 tx=0).stop 2024-01-16T15:09:19.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 shutdown_connections 2024-01-16T15:09:19.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb76806bec0 0x7fb76806e380 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb79013e180 0x7fb7901e51a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb79013e6d0 0x7fb79013d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 --2- 172.21.15.141:0/1795638204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7700047c0 0x7fb790150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 >> 172.21.15.141:0/1795638204 conn(0x7fb79006a7d0 msgr2=0x7fb79014f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 shutdown_connections 2024-01-16T15:09:19.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.546+0000 7fb796200640 1 -- 172.21.15.141:0/1795638204 wait complete. 2024-01-16T15:09:19.542 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:19.543 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59633 2024-01-16T15:09:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.780+0000 7f86b6e00640 1 Processor -- start 2024-01-16T15:09:19.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.780+0000 7f86b6e00640 1 -- start start 2024-01-16T15:09:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.782+0000 7f86b6e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b0143ae0 0x7f86b00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.782+0000 7f86b6e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f86b00cf7d0 0x7f86b015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.782+0000 7f86b6e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b00cfc30 con 0x7f86b015e1e0 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.782+0000 7f86b6e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b00b1ee0 con 0x7f86b00cf7d0 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.782+0000 7f86b6e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b00b2780 con 0x7f86b0143ae0 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.782+0000 7f86af400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f86b00cf7d0 0x7f86b015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.783+0000 7f86af400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f86b00cf7d0 0x7f86b015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58860/0 (socket says 172.21.15.141:58860) 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.783+0000 7f86af400640 1 -- 172.21.15.141:0/1139116199 learned_addr learned my addr 172.21.15.141:0/1139116199 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.783+0000 7f86afe00640 1 --2- 172.21.15.141:0/1139116199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b0143ae0 0x7f86b00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.783+0000 7f86af400640 1 -- 172.21.15.141:0/1139116199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b0143ae0 msgr2=0x7f86b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.783+0000 7f86af400640 1 --2- 172.21.15.141:0/1139116199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b0143ae0 0x7f86b00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.783+0000 7f86af400640 1 -- 172.21.15.141:0/1139116199 >> v1:172.21.15.188:6789/0 conn(0x7f86b015e1e0 legacy=0x7f86b015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.783+0000 7f86af400640 1 -- 172.21.15.141:0/1139116199 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b015e990 con 0x7f86b00cf7d0 2024-01-16T15:09:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.785+0000 7f86af400640 1 --2- 172.21.15.141:0/1139116199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b00cf7d0 0x7f86b015dca0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f869800bb10 tx=0x7f869802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9981986c3077fdde server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.785+0000 7f86aea00640 1 -- 172.21.15.141:0/1139116199 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869802ee70 con 0x7f86b00cf7d0 2024-01-16T15:09:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.785+0000 7f86aea00640 1 -- 172.21.15.141:0/1139116199 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8698036440 con 0x7f86b00cf7d0 2024-01-16T15:09:19.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.785+0000 7f86aea00640 1 -- 172.21.15.141:0/1139116199 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86980367d0 con 0x7f86b00cf7d0 2024-01-16T15:09:19.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 -- 172.21.15.141:0/1139116199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b00cf7d0 msgr2=0x7f86b015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 --2- 172.21.15.141:0/1139116199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b00cf7d0 0x7f86b015dca0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f869800bb10 tx=0x7f869802e0c0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 -- 172.21.15.141:0/1139116199 shutdown_connections 2024-01-16T15:09:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 --2- 172.21.15.141:0/1139116199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b00cf7d0 0x7f86b015dca0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 --2- 172.21.15.141:0/1139116199 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b0143ae0 0x7f86b00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 -- 172.21.15.141:0/1139116199 >> 172.21.15.141:0/1139116199 conn(0x7f86b006a800 msgr2=0x7f86b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 -- 172.21.15.141:0/1139116199 shutdown_connections 2024-01-16T15:09:19.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.786+0000 7f86b6e00640 1 -- 172.21.15.141:0/1139116199 wait complete. 2024-01-16T15:09:19.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.788+0000 7f86b6e00640 1 Processor -- start 2024-01-16T15:09:19.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.788+0000 7f86b6e00640 1 -- start start 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b6e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b00cf7d0 0x7f86b01f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b6e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b0143ae0 0x7f86b0141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b6e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b015e1e0 0x7f86b0141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b6e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b015d190 con 0x7f86b015e1e0 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b6e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b01f3cd0 con 0x7f86b0143ae0 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b6e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b0140650 con 0x7f86b00cf7d0 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b015e1e0 0x7f86b0141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b015e1e0 0x7f86b0141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58870/0 (socket says 172.21.15.141:58870) 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.789+0000 7f86b4e00640 1 -- 172.21.15.141:0/14867813 learned_addr learned my addr 172.21.15.141:0/14867813 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86af400640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b0143ae0 0x7f86b0141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b4e00640 1 -- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b00cf7d0 msgr2=0x7f86b01f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b4e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b00cf7d0 0x7f86b01f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b4e00640 1 -- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b0143ae0 msgr2=0x7f86b0141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b4e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b0143ae0 0x7f86b0141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b4e00640 1 -- 172.21.15.141:0/14867813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869800b770 con 0x7f86b015e1e0 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b4e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b015e1e0 0x7f86b0141560 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f86a800b970 tx=0x7f86a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86a7e00640 1 -- 172.21.15.141:0/14867813 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a800cae0 con 0x7f86b015e1e0 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86a7e00640 1 -- 172.21.15.141:0/14867813 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a800cc40 con 0x7f86b015e1e0 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86a7e00640 1 -- 172.21.15.141:0/14867813 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a8014680 con 0x7f86b015e1e0 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b013fa40 con 0x7f86b015e1e0 2024-01-16T15:09:19.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.790+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b0140000 con 0x7f86b015e1e0 2024-01-16T15:09:19.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.792+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b00675d0 con 0x7f86b015e1e0 2024-01-16T15:09:19.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.792+0000 7f86a7e00640 1 -- 172.21.15.141:0/14867813 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f86a801d3b0 con 0x7f86b015e1e0 2024-01-16T15:09:19.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.794+0000 7f86a7e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f868806be90 0x7f868806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.794+0000 7f86a7e00640 1 -- 172.21.15.141:0/14867813 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f86a8088560 con 0x7f86b015e1e0 2024-01-16T15:09:19.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.794+0000 7f86afe00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f868806be90 0x7f868806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.795+0000 7f86afe00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f868806be90 0x7f868806e350 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f86a00098f0 tx=0x7f86a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:19.798+0000 7f86a7e00640 1 -- 172.21.15.141:0/14867813 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86a8054c20 con 0x7f86b015e1e0 2024-01-16T15:09:19.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.008+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59633", "target": ["mon-mgr", ""]}) v1 -- 0x7f86b01407c0 con 0x7f868806be90 2024-01-16T15:09:20.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.032+0000 7f86a7e00640 1 -- 172.21.15.141:0/14867813 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f86b01407c0 con 0x7f868806be90 2024-01-16T15:09:20.009 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:20.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.034+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f868806be90 msgr2=0x7f868806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.034+0000 7f86b6e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f868806be90 0x7f868806e350 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f86a00098f0 tx=0x7f86a0008040 comp rx=0 tx=0).stop 2024-01-16T15:09:20.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.034+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b015e1e0 msgr2=0x7f86b0141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.034+0000 7f86b6e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b015e1e0 0x7f86b0141560 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f86a800b970 tx=0x7f86a800be30 comp rx=0 tx=0).stop 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.034+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 shutdown_connections 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.034+0000 7f86b6e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f868806be90 0x7f868806e350 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.035+0000 7f86b6e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b015e1e0 0x7f86b0141560 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.035+0000 7f86b6e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b0143ae0 0x7f86b0141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.035+0000 7f86b6e00640 1 --2- 172.21.15.141:0/14867813 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86b00cf7d0 0x7f86b01f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.035+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 >> 172.21.15.141:0/14867813 conn(0x7f86b006a800 msgr2=0x7f86b006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.035+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 shutdown_connections 2024-01-16T15:09:20.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.035+0000 7f86b6e00640 1 -- 172.21.15.141:0/14867813 wait complete. 2024-01-16T15:09:20.031 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:20.031 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59634 2024-01-16T15:09:20.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.261+0000 7fd563a00640 1 Processor -- start 2024-01-16T15:09:20.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.261+0000 7fd563a00640 1 -- start start 2024-01-16T15:09:20.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.262+0000 7fd563a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd55c13df80 0x7fd55c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd563a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd55c13e360 0x7fd55c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd563a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c15f9c0 0x7fd55c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd563a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55c06c470 con 0x7fd55c13e360 2024-01-16T15:09:20.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd563a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55c06c750 con 0x7fd55c13df80 2024-01-16T15:09:20.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd563a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c06ca30 con 0x7fd55c15f9c0 2024-01-16T15:09:20.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd55be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd55c13e360 0x7fd55c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd55be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd55c13e360 0x7fd55c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58882/0 (socket says 172.21.15.141:58882) 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd55be00640 1 -- 172.21.15.141:0/3300004402 learned_addr learned my addr 172.21.15.141:0/3300004402 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd561a00640 1 --2- 172.21.15.141:0/3300004402 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c15f9c0 0x7fd55c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.263+0000 7fd561000640 1 --2- 172.21.15.141:0/3300004402 >> v2:172.21.15.188:3300/0 conn(0x7fd55c13df80 0x7fd55c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.264+0000 7fd55be00640 1 -- 172.21.15.141:0/3300004402 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c15f9c0 msgr2=0x7fd55c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.264+0000 7fd55be00640 1 --2- 172.21.15.141:0/3300004402 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c15f9c0 0x7fd55c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.264+0000 7fd55be00640 1 -- 172.21.15.141:0/3300004402 >> v2:172.21.15.188:3300/0 conn(0x7fd55c13df80 msgr2=0x7fd55c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.264+0000 7fd55be00640 1 --2- 172.21.15.141:0/3300004402 >> v2:172.21.15.188:3300/0 conn(0x7fd55c13df80 0x7fd55c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.264+0000 7fd55be00640 1 -- 172.21.15.141:0/3300004402 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c06cd10 con 0x7fd55c13e360 2024-01-16T15:09:20.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.265+0000 7fd55be00640 1 --2- 172.21.15.141:0/3300004402 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c13e360 0x7fd55c0cf290 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd55c072770 tx=0x7fd54402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1613924665aa321 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.265+0000 7fd55b400640 1 -- 172.21.15.141:0/3300004402 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544031870 con 0x7fd55c13e360 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.265+0000 7fd55b400640 1 -- 172.21.15.141:0/3300004402 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5440319d0 con 0x7fd55c13e360 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.265+0000 7fd55b400640 1 -- 172.21.15.141:0/3300004402 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544031d60 con 0x7fd55c13e360 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 -- 172.21.15.141:0/3300004402 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c13e360 msgr2=0x7fd55c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 --2- 172.21.15.141:0/3300004402 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c13e360 0x7fd55c0cf290 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd55c072770 tx=0x7fd54402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 -- 172.21.15.141:0/3300004402 shutdown_connections 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 --2- 172.21.15.141:0/3300004402 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c15f9c0 0x7fd55c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 --2- 172.21.15.141:0/3300004402 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c13e360 0x7fd55c0cf290 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 --2- 172.21.15.141:0/3300004402 >> v2:172.21.15.188:3300/0 conn(0x7fd55c13df80 0x7fd55c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 -- 172.21.15.141:0/3300004402 >> 172.21.15.141:0/3300004402 conn(0x7fd55c06a800 msgr2=0x7fd55c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 -- 172.21.15.141:0/3300004402 shutdown_connections 2024-01-16T15:09:20.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.266+0000 7fd563a00640 1 -- 172.21.15.141:0/3300004402 wait complete. 2024-01-16T15:09:20.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.267+0000 7fd563a00640 1 Processor -- start 2024-01-16T15:09:20.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.267+0000 7fd563a00640 1 -- start start 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.268+0000 7fd563a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd55c13df80 0x7fd55c1f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.268+0000 7fd563a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c13e360 0x7fd55c15d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.268+0000 7fd563a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c15f9c0 0x7fd55c0aab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.268+0000 7fd563a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c1f3220 con 0x7fd55c15f9c0 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.268+0000 7fd563a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c13d330 con 0x7fd55c13df80 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.268+0000 7fd563a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0ab050 con 0x7fd55c13e360 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c15f9c0 0x7fd55c0aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c15f9c0 0x7fd55c0aab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58886/0 (socket says 172.21.15.141:58886) 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 -- 172.21.15.141:0/2376361186 learned_addr learned my addr 172.21.15.141:0/2376361186 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561000640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd55c13df80 0x7fd55c1f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 -- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c13e360 msgr2=0x7fd55c15d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c13e360 0x7fd55c15d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 -- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd55c13df80 msgr2=0x7fd55c1f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd55c13df80 0x7fd55c1f6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 -- 172.21.15.141:0/2376361186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5440094d0 con 0x7fd55c15f9c0 2024-01-16T15:09:20.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.269+0000 7fd561a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c15f9c0 0x7fd55c0aab10 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd554007e30 tx=0x7fd55400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.270+0000 7fd558c00640 1 -- 172.21.15.141:0/2376361186 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554014620 con 0x7fd55c15f9c0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.270+0000 7fd558c00640 1 -- 172.21.15.141:0/2376361186 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55400ee90 con 0x7fd55c15f9c0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.270+0000 7fd558c00640 1 -- 172.21.15.141:0/2376361186 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5540149c0 con 0x7fd55c15f9c0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.270+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c1f74f0 con 0x7fd55c15f9c0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.270+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c1f6f30 con 0x7fd55c15f9c0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.271+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c1f3360 con 0x7fd55c15f9c0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.272+0000 7fd558c00640 1 -- 172.21.15.141:0/2376361186 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd554014b20 con 0x7fd55c15f9c0 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.272+0000 7fd558c00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd53406be90 0x7fd53406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.273+0000 7fd558c00640 1 -- 172.21.15.141:0/2376361186 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd554088f00 con 0x7fd55c15f9c0 2024-01-16T15:09:20.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.273+0000 7fd561000640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd53406be90 0x7fd53406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.274+0000 7fd561000640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd53406be90 0x7fd53406e350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd54c006fd0 tx=0x7fd54c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.277+0000 7fd558c00640 1 -- 172.21.15.141:0/2376361186 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd55c1f3360 con 0x7fd55c15f9c0 2024-01-16T15:09:20.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.501+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7fd55c0cfaf0 con 0x7fd53406be90 2024-01-16T15:09:20.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.527+0000 7fd558c00640 1 -- 172.21.15.141:0/2376361186 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fd55c0cfaf0 con 0x7fd53406be90 2024-01-16T15:09:20.505 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:20.505 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:20.505 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "complete" 2024-01-16T15:09:20.505 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:20.505 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd53406be90 msgr2=0x7fd53406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd53406be90 0x7fd53406e350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd54c006fd0 tx=0x7fd54c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c15f9c0 msgr2=0x7fd55c0aab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c15f9c0 0x7fd55c0aab10 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd554007e30 tx=0x7fd55400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 shutdown_connections 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd53406be90 0x7fd53406e350 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd55c15f9c0 0x7fd55c0aab10 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd55c13e360 0x7fd55c15d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 --2- 172.21.15.141:0/2376361186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd55c13df80 0x7fd55c1f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 >> 172.21.15.141:0/2376361186 conn(0x7fd55c06a800 msgr2=0x7fd55c0af230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 shutdown_connections 2024-01-16T15:09:20.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.530+0000 7fd563a00640 1 -- 172.21.15.141:0/2376361186 wait complete. 2024-01-16T15:09:20.526 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:20.526 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59635 2024-01-16T15:09:20.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.763+0000 7f4b48800640 1 Processor -- start 2024-01-16T15:09:20.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.763+0000 7f4b48800640 1 -- start start 2024-01-16T15:09:20.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b48800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4b400cf9a0 0x7f4b4015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b48800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 0x7f4b400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b48800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400b1ee0 con 0x7f4b40143ae0 2024-01-16T15:09:20.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b48800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b400b2780 con 0x7f4b400cf9a0 2024-01-16T15:09:20.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b48800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4015e990 con 0x7f4b4015e1e0 2024-01-16T15:09:20.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b45e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4b40143ae0 0x7f4b400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43954/0 (socket says 172.21.15.141:43954) 2024-01-16T15:09:20.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b45e00640 1 -- 172.21.15.141:0/1523212447 learned_addr learned my addr 172.21.15.141:0/1523212447 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:20.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b45400640 1 --2- 172.21.15.141:0/1523212447 >> v2:172.21.15.188:3300/0 conn(0x7f4b400cf9a0 0x7f4b4015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.765+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437097404 0 0) 0x7f4b400b1ee0 con 0x7f4b40143ae0 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b46800640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 0x7f4b400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b20003520 con 0x7f4b40143ae0 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705243811 0 0) 0x7f4b20003520 con 0x7f4b40143ae0 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 msgr2=0x7f4b400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 0x7f4b400702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 >> v2:172.21.15.188:3300/0 conn(0x7f4b400cf9a0 msgr2=0x7f4b4015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 --2- 172.21.15.141:0/1523212447 >> v2:172.21.15.188:3300/0 conn(0x7f4b400cf9a0 0x7f4b4015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b400b2130 con 0x7f4b40143ae0 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4b30001670 con 0x7f4b40143ae0 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 >> v1:172.21.15.141:6789/0 conn(0x7f4b40143ae0 legacy=0x7f4b400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b200039a0 0x7f4b20003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 0x7f4b200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b20004d00 0x7f4b20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b400b2780 con 0x7f4b20004d00 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4015e990 con 0x7f4b200039a0 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2000ba00 con 0x7f4b4015e1e0 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.766+0000 7f4b46800640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b20004d00 0x7f4b20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b45e00640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b200039a0 0x7f4b20003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b45400640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 0x7f4b200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b46800640 1 -- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 msgr2=0x7f4b200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b46800640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4015e1e0 0x7f4b200047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b46800640 1 -- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b200039a0 msgr2=0x7f4b20003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b46800640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b200039a0 0x7f4b20003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b46800640 1 -- 172.21.15.141:0/1523212447 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b20003520 con 0x7f4b20004d00 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.767+0000 7f4b45e00640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b200039a0 0x7f4b20003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:20.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.768+0000 7f4b46800640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b20004d00 0x7f4b20005180 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f4b38007360 tx=0x7f4b200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c090bb1ab997fb2c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.768+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3802ccd0 con 0x7f4b20004d00 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.768+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3802ce30 con 0x7f4b20004d00 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.768+0000 7f4b44a00640 1 -- 172.21.15.141:0/1523212447 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38034660 con 0x7f4b20004d00 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.769+0000 7f4b48800640 1 -- 172.21.15.141:0/1523212447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b20004d00 msgr2=0x7f4b20005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.769+0000 7f4b48800640 1 --2- 172.21.15.141:0/1523212447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b20004d00 0x7f4b20005180 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f4b38007360 tx=0x7f4b200057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.769+0000 7f4b46800640 1 -- 172.21.15.141:0/1523212447 reap_dead start 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.769+0000 7f4b48800640 1 -- 172.21.15.141:0/1523212447 shutdown_connections 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.769+0000 7f4b48800640 1 -- 172.21.15.141:0/1523212447 >> 172.21.15.141:0/1523212447 conn(0x7f4b4006a800 msgr2=0x7f4b4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.769+0000 7f4b48800640 1 -- 172.21.15.141:0/1523212447 shutdown_connections 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.769+0000 7f4b48800640 1 -- 172.21.15.141:0/1523212447 wait complete. 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.771+0000 7f4b48800640 1 Processor -- start 2024-01-16T15:09:20.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.771+0000 7f4b48800640 1 -- start start 2024-01-16T15:09:20.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b48800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b20004d00 0x7f4b40158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b48800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b40137d00 0x7f4b40136f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b48800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4013a730 0x7f4b4013abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b48800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4013b100 con 0x7f4b40137d00 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b48800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b40158aa0 con 0x7f4b20004d00 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b48800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b40158d80 con 0x7f4b4013a730 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b45400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b40137d00 0x7f4b40136f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b45400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b40137d00 0x7f4b40136f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58906/0 (socket says 172.21.15.141:58906) 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b45400640 1 -- 172.21.15.141:0/2739564929 learned_addr learned my addr 172.21.15.141:0/2739564929 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.772+0000 7f4b45e00640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b20004d00 0x7f4b40158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b45400640 1 -- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4013a730 msgr2=0x7f4b4013abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b45400640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4013a730 0x7f4b4013abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b45400640 1 -- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b20004d00 msgr2=0x7f4b40158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b45400640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b20004d00 0x7f4b40158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b45400640 1 -- 172.21.15.141:0/2739564929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40159060 con 0x7f4b40137d00 2024-01-16T15:09:20.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b45e00640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b20004d00 0x7f4b40158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b45400640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b40137d00 0x7f4b40136f60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f4b280108e0 tx=0x7f4b28010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b3e000640 1 -- 172.21.15.141:0/2739564929 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800eae0 con 0x7f4b40137d00 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b3e000640 1 -- 172.21.15.141:0/2739564929 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2800ec40 con 0x7f4b40137d00 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b40159340 con 0x7f4b40137d00 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b3e000640 1 -- 172.21.15.141:0/2739564929 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2801c400 con 0x7f4b40137d00 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.773+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b40070c80 con 0x7f4b40137d00 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.775+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b40070ea0 con 0x7f4b40137d00 2024-01-16T15:09:20.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.775+0000 7f4b3e000640 1 -- 172.21.15.141:0/2739564929 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f4b2801c560 con 0x7f4b40137d00 2024-01-16T15:09:20.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.777+0000 7f4b3e000640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b1806bf10 0x7f4b1806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.777+0000 7f4b3e000640 1 -- 172.21.15.141:0/2739564929 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4b28088e40 con 0x7f4b40137d00 2024-01-16T15:09:20.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.778+0000 7f4b45e00640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b1806bf10 0x7f4b1806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.779+0000 7f4b45e00640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b1806bf10 0x7f4b1806e3d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4b4015d7d0 tx=0x7f4b30001560 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:20.781+0000 7f4b3e000640 1 -- 172.21.15.141:0/2739564929 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b28055550 con 0x7f4b40137d00 2024-01-16T15:09:20.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.011+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59635", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b40158ef0 con 0x7f4b1806bf10 2024-01-16T15:09:21.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.033+0000 7f4b3e000640 1 -- 172.21.15.141:0/2739564929 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f4b40158ef0 con 0x7f4b1806bf10 2024-01-16T15:09:21.010 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:21.010 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:21.010 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:21.011 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:21.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b1806bf10 msgr2=0x7f4b1806e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b1806bf10 0x7f4b1806e3d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4b4015d7d0 tx=0x7f4b30001560 comp rx=0 tx=0).stop 2024-01-16T15:09:21.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b40137d00 msgr2=0x7f4b40136f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b40137d00 0x7f4b40136f60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f4b280108e0 tx=0x7f4b28010da0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 shutdown_connections 2024-01-16T15:09:21.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b1806bf10 0x7f4b1806e3d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b4013a730 0x7f4b4013abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b40137d00 0x7f4b40136f60 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 --2- 172.21.15.141:0/2739564929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b20004d00 0x7f4b40158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 >> 172.21.15.141:0/2739564929 conn(0x7f4b4006a800 msgr2=0x7f4b400cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 shutdown_connections 2024-01-16T15:09:21.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.036+0000 7f4b48800640 1 -- 172.21.15.141:0/2739564929 wait complete. 2024-01-16T15:09:21.030 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:21.030 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59636 2024-01-16T15:09:21.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.250+0000 7fcbd1800640 1 Processor -- start 2024-01-16T15:09:21.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.250+0000 7fcbd1800640 1 -- start start 2024-01-16T15:09:21.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.251+0000 7fcbd1800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcbcc13e360 0x7fcbcc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.251+0000 7fcbd1800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcbcc15f9c0 0x7fcbcc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.251+0000 7fcbd1800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc15c5f0 con 0x7fcbcc13df80 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.251+0000 7fcbd1800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbcc06c470 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.251+0000 7fcbd1800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbcc06c750 con 0x7fcbcc13e360 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbcaa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fcbcc13df80 0x7fcbcc15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43966/0 (socket says 172.21.15.141:43966) 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbcaa00640 1 -- 172.21.15.141:0/40883472 learned_addr learned my addr 172.21.15.141:0/40883472 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbcb400640 1 --2- 172.21.15.141:0/40883472 >> v2:172.21.15.141:3300/0 conn(0x7fcbcc15f9c0 0x7fcbcc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbc1e00640 1 --2- 172.21.15.141:0/40883472 >> v2:172.21.15.188:3300/0 conn(0x7fcbcc13e360 0x7fcbcc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbca000640 1 -- 172.21.15.141:0/40883472 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628811956 0 0) 0x7fcbcc15c5f0 con 0x7fcbcc13df80 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbcb400640 1 -- 172.21.15.141:0/40883472 >> v2:172.21.15.188:3300/0 conn(0x7fcbcc13e360 msgr2=0x7fcbcc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbcb400640 1 --2- 172.21.15.141:0/40883472 >> v2:172.21.15.188:3300/0 conn(0x7fcbcc13e360 0x7fcbcc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbcb400640 1 -- 172.21.15.141:0/40883472 >> v1:172.21.15.141:6789/0 conn(0x7fcbcc13df80 legacy=0x7fcbcc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.252+0000 7fcbcb400640 1 -- 172.21.15.141:0/40883472 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc06ca30 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.254+0000 7fcbcb400640 1 --2- 172.21.15.141:0/40883472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc0cf7d0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc06ded0 tx=0x7fcbc402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5ec61550c61acc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.254+0000 7fcbca000640 1 -- 172.21.15.141:0/40883472 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc402cc70 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.254+0000 7fcbca000640 1 -- 172.21.15.141:0/40883472 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc402cdd0 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.254+0000 7fcbca000640 1 -- 172.21.15.141:0/40883472 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4034660 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.254+0000 7fcbd1800640 1 -- 172.21.15.141:0/40883472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 msgr2=0x7fcbcc0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.254+0000 7fcbd1800640 1 --2- 172.21.15.141:0/40883472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc0cf7d0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc06ded0 tx=0x7fcbc402bec0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.255+0000 7fcbd1800640 1 -- 172.21.15.141:0/40883472 shutdown_connections 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.255+0000 7fcbd1800640 1 --2- 172.21.15.141:0/40883472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc0cf7d0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.255+0000 7fcbd1800640 1 --2- 172.21.15.141:0/40883472 >> v2:172.21.15.188:3300/0 conn(0x7fcbcc13e360 0x7fcbcc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.255+0000 7fcbd1800640 1 -- 172.21.15.141:0/40883472 >> 172.21.15.141:0/40883472 conn(0x7fcbcc06a800 msgr2=0x7fcbcc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.255+0000 7fcbd1800640 1 -- 172.21.15.141:0/40883472 shutdown_connections 2024-01-16T15:09:21.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.255+0000 7fcbd1800640 1 -- 172.21.15.141:0/40883472 wait complete. 2024-01-16T15:09:21.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.256+0000 7fcbd1800640 1 Processor -- start 2024-01-16T15:09:21.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.257+0000 7fcbd1800640 1 -- start start 2024-01-16T15:09:21.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbd1800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbcc13df80 0x7fcbcc153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbd1800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcbcc13e360 0x7fcbcc15d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbd1800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc15dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbd1800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc153940 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbd1800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc13c630 con 0x7fcbcc13df80 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbd1800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc13c910 con 0x7fcbcc13e360 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc15dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc15dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58922/0 (socket says 172.21.15.141:58922) 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 -- 172.21.15.141:0/2743209145 learned_addr learned my addr 172.21.15.141:0/2743209145 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 -- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcbcc13e360 msgr2=0x7fcbcc15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcbcc13e360 0x7fcbcc15d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcaa00640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbcc13df80 0x7fcbcc153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 -- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbcc13df80 msgr2=0x7fcbcc153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbcc13df80 0x7fcbcc153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.258+0000 7fcbcb400640 1 -- 172.21.15.141:0/2743209145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc4009a70 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.259+0000 7fcbcb400640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc15dde0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009e80 tx=0x7fcbc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.259+0000 7fcbc3400640 1 -- 172.21.15.141:0/2743209145 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4040650 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.259+0000 7fcbc3400640 1 -- 172.21.15.141:0/2743209145 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc40407d0 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.259+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbcc13cbf0 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.259+0000 7fcbc3400640 1 -- 172.21.15.141:0/2743209145 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4040b80 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.259+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc13d1b0 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.260+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc0675d0 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.261+0000 7fcbc3400640 1 -- 172.21.15.141:0/2743209145 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fcbc4040ce0 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.262+0000 7fcbc3400640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcba406be90 0x7fcba406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.262+0000 7fcbc3400640 1 -- 172.21.15.141:0/2743209145 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcbc40ab2f0 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.263+0000 7fcbcaa00640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcba406be90 0x7fcba406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.264+0000 7fcbcaa00640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcba406be90 0x7fcba406e350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc000f60 tx=0x7fcbbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.266+0000 7fcbc3400640 1 -- 172.21.15.141:0/2743209145 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbc4077c10 con 0x7fcbcc15f9c0 2024-01-16T15:09:21.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.494+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59636", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbcc1559b0 con 0x7fcba406be90 2024-01-16T15:09:21.494 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:21.494 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:21.494 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:21.494 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:21.494 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:21.494 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:21.494 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:21.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.517+0000 7fcbc3400640 1 -- 172.21.15.141:0/2743209145 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fcbcc1559b0 con 0x7fcba406be90 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.519+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcba406be90 msgr2=0x7fcba406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.519+0000 7fcbd1800640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcba406be90 0x7fcba406e350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc000f60 tx=0x7fcbbc009040 comp rx=0 tx=0).stop 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.519+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 msgr2=0x7fcbcc15dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.519+0000 7fcbd1800640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc15dde0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009e80 tx=0x7fcbc4004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 shutdown_connections 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcba406be90 0x7fcba406e350 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcbcc15f9c0 0x7fcbcc15dde0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcbcc13e360 0x7fcbcc15d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 --2- 172.21.15.141:0/2743209145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbcc13df80 0x7fcbcc153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 >> 172.21.15.141:0/2743209145 conn(0x7fcbcc06a800 msgr2=0x7fcbcc0af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 shutdown_connections 2024-01-16T15:09:21.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.520+0000 7fcbd1800640 1 -- 172.21.15.141:0/2743209145 wait complete. 2024-01-16T15:09:21.515 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:21.515 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59637 2024-01-16T15:09:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.750+0000 7f316bc00640 1 Processor -- start 2024-01-16T15:09:21.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.750+0000 7f316bc00640 1 -- start start 2024-01-16T15:09:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.751+0000 7f316bc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3164143ae0 0x7f31640cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.751+0000 7f316bc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31640cf7d0 0x7f316415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.751+0000 7f316bc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31640cfc30 con 0x7f316415e1e0 2024-01-16T15:09:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.751+0000 7f316bc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31640b1ee0 con 0x7f3164143ae0 2024-01-16T15:09:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.751+0000 7f316bc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31640b2780 con 0x7f31640cf7d0 2024-01-16T15:09:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.752+0000 7f3169200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3164143ae0 0x7f31640cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.752+0000 7f315be00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31640cf7d0 0x7f316415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.752+0000 7f3169200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3164143ae0 0x7f31640cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:39976/0 (socket says 172.21.15.141:39976) 2024-01-16T15:09:21.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.752+0000 7f3169200640 1 -- 172.21.15.141:0/3468467088 learned_addr learned my addr 172.21.15.141:0/3468467088 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:21.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.753+0000 7f3169200640 1 -- 172.21.15.141:0/3468467088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31640cf7d0 msgr2=0x7f316415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:21.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.753+0000 7f3169200640 1 --2- 172.21.15.141:0/3468467088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31640cf7d0 0x7f316415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.753+0000 7f3169200640 1 -- 172.21.15.141:0/3468467088 >> v1:172.21.15.188:6789/0 conn(0x7f316415e1e0 legacy=0x7f316415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.753+0000 7f3169200640 1 -- 172.21.15.141:0/3468467088 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316415e990 con 0x7f3164143ae0 2024-01-16T15:09:21.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.754+0000 7f3169200640 1 --2- 172.21.15.141:0/3468467088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3164143ae0 0x7f31640cf290 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315402db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5f32e62891c07f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.755+0000 7f3163e00640 1 -- 172.21.15.141:0/3468467088 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315402e940 con 0x7f3164143ae0 2024-01-16T15:09:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.755+0000 7f3163e00640 1 -- 172.21.15.141:0/3468467088 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315402eaa0 con 0x7f3164143ae0 2024-01-16T15:09:21.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.755+0000 7f3163e00640 1 -- 172.21.15.141:0/3468467088 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315402ee30 con 0x7f3164143ae0 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 -- 172.21.15.141:0/3468467088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3164143ae0 msgr2=0x7f31640cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 --2- 172.21.15.141:0/3468467088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3164143ae0 0x7f31640cf290 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315402db90 comp rx=0 tx=0).stop 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 -- 172.21.15.141:0/3468467088 shutdown_connections 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 --2- 172.21.15.141:0/3468467088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31640cf7d0 0x7f316415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 --2- 172.21.15.141:0/3468467088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3164143ae0 0x7f31640cf290 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 -- 172.21.15.141:0/3468467088 >> 172.21.15.141:0/3468467088 conn(0x7f316406a800 msgr2=0x7f316406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 -- 172.21.15.141:0/3468467088 shutdown_connections 2024-01-16T15:09:21.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.756+0000 7f316bc00640 1 -- 172.21.15.141:0/3468467088 wait complete. 2024-01-16T15:09:21.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.758+0000 7f316bc00640 1 Processor -- start 2024-01-16T15:09:21.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.758+0000 7f316bc00640 1 -- start start 2024-01-16T15:09:21.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f316bc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31640cf7d0 0x7f31640cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f316bc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3164143ae0 0x7f3164158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f316bc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f316415e1e0 0x7f3164159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f316bc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316413c2e0 con 0x7f31640cf7d0 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f316bc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316413c5c0 con 0x7f316415e1e0 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f316bc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316413c8a0 con 0x7f3164143ae0 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f3169200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31640cf7d0 0x7f31640cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f3169200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31640cf7d0 0x7f31640cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58938/0 (socket says 172.21.15.141:58938) 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f3169200640 1 -- 172.21.15.141:0/2860851539 learned_addr learned my addr 172.21.15.141:0/2860851539 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f315be00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3164143ae0 0x7f3164158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f3169200640 1 -- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3164143ae0 msgr2=0x7f3164158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f3169200640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3164143ae0 0x7f3164158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f3169200640 1 -- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f316415e1e0 msgr2=0x7f3164159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.759+0000 7f3169c00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f316415e1e0 0x7f3164159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f3169200640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f316415e1e0 0x7f3164159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f3169200640 1 -- 172.21.15.141:0/2860851539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540094d0 con 0x7f31640cf7d0 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f3169c00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f316415e1e0 0x7f3164159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f3169200640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31640cf7d0 0x7f31640cd170 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f3154000c00 tx=0x7f31540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154040600 con 0x7f31640cf7d0 2024-01-16T15:09:21.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3154040780 con 0x7f31640cf7d0 2024-01-16T15:09:21.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316413cb80 con 0x7f31640cf7d0 2024-01-16T15:09:21.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154040b30 con 0x7f31640cf7d0 2024-01-16T15:09:21.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.760+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316413d380 con 0x7f31640cf7d0 2024-01-16T15:09:21.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.762+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316413d0a0 con 0x7f31640cf7d0 2024-01-16T15:09:21.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.762+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3154040c90 con 0x7f31640cf7d0 2024-01-16T15:09:21.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.763+0000 7f3161600640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f313c06be90 0x7f313c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:21.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.763+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f31540ab020 con 0x7f31640cf7d0 2024-01-16T15:09:21.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.764+0000 7f315be00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f313c06be90 0x7f313c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:21.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.765+0000 7f315be00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f313c06be90 0x7f313c06e350 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f31640af550 tx=0x7f314c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.768+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f316413d0a0 con 0x7f31640cf7d0 2024-01-16T15:09:21.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.816+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f3154077a50 con 0x7f31640cf7d0 2024-01-16T15:09:21.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:21.999+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59637", "target": ["mon-mgr", ""]}) v1 -- 0x7f31640670a0 con 0x7f313c06be90 2024-01-16T15:09:22.003 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:22.003 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:22.003 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:22.003 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:22.004 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:22.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.025+0000 7f3161600640 1 -- 172.21.15.141:0/2860851539 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f31640670a0 con 0x7f313c06be90 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f313c06be90 msgr2=0x7f313c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f313c06be90 0x7f313c06e350 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f31640af550 tx=0x7f314c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31640cf7d0 msgr2=0x7f31640cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31640cf7d0 0x7f31640cd170 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f3154000c00 tx=0x7f31540046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 shutdown_connections 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f313c06be90 0x7f313c06e350 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f316415e1e0 0x7f3164159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3164143ae0 0x7f3164158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 --2- 172.21.15.141:0/2860851539 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31640cf7d0 0x7f31640cd170 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 >> 172.21.15.141:0/2860851539 conn(0x7f316406a800 msgr2=0x7f31640afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 shutdown_connections 2024-01-16T15:09:22.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.028+0000 7f316bc00640 1 -- 172.21.15.141:0/2860851539 wait complete. 2024-01-16T15:09:22.024 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:22.024 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59638 2024-01-16T15:09:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.262+0000 7fdf8f200640 1 Processor -- start 2024-01-16T15:09:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.262+0000 7fdf8f200640 1 -- start start 2024-01-16T15:09:22.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.263+0000 7fdf8f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdf900adf20 0x7fdf90070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.263+0000 7fdf8f200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9015f4e0 con 0x7fdf9015bf10 2024-01-16T15:09:22.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.263+0000 7fdf8f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf90071130 con 0x7fdf900ccb50 2024-01-16T15:09:22.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.263+0000 7fdf8f200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9015f970 con 0x7fdf900adf20 2024-01-16T15:09:22.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.264+0000 7fdf87e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdf9015bf10 0x7fdf900cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43970/0 (socket says 172.21.15.141:43970) 2024-01-16T15:09:22.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.264+0000 7fdf87e00640 1 -- 172.21.15.141:0/933556319 learned_addr learned my addr 172.21.15.141:0/933556319 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:22.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.264+0000 7fdf8de00640 1 --2- 172.21.15.141:0/933556319 >> v2:172.21.15.141:3300/0 conn(0x7fdf900adf20 0x7fdf90070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.264+0000 7fdf8d400640 1 -- 172.21.15.141:0/933556319 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816216676 0 0) 0x7fdf9015f4e0 con 0x7fdf9015bf10 2024-01-16T15:09:22.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.264+0000 7fdf8de00640 1 -- 172.21.15.141:0/933556319 >> v1:172.21.15.188:6789/0 conn(0x7fdf900ccb50 legacy=0x7fdf90143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:22.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.264+0000 7fdf8de00640 1 -- 172.21.15.141:0/933556319 >> v1:172.21.15.141:6789/0 conn(0x7fdf9015bf10 legacy=0x7fdf900cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.264+0000 7fdf8de00640 1 -- 172.21.15.141:0/933556319 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9015c390 con 0x7fdf900adf20 2024-01-16T15:09:22.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.266+0000 7fdf8de00640 1 --2- 172.21.15.141:0/933556319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf90070a30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fdf800097a0 tx=0x7fdf8002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20db25d623b616dc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.266+0000 7fdf8d400640 1 -- 172.21.15.141:0/933556319 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8002e070 con 0x7fdf900adf20 2024-01-16T15:09:22.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.266+0000 7fdf8d400640 1 -- 172.21.15.141:0/933556319 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf800315a0 con 0x7fdf900adf20 2024-01-16T15:09:22.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.266+0000 7fdf8d400640 1 -- 172.21.15.141:0/933556319 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80031930 con 0x7fdf900adf20 2024-01-16T15:09:22.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.267+0000 7fdf8f200640 1 -- 172.21.15.141:0/933556319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 msgr2=0x7fdf90070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.267+0000 7fdf8f200640 1 --2- 172.21.15.141:0/933556319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf90070a30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fdf800097a0 tx=0x7fdf8002b840 comp rx=0 tx=0).stop 2024-01-16T15:09:22.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.267+0000 7fdf8f200640 1 -- 172.21.15.141:0/933556319 shutdown_connections 2024-01-16T15:09:22.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.267+0000 7fdf8f200640 1 --2- 172.21.15.141:0/933556319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf90070a30 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.267+0000 7fdf8f200640 1 -- 172.21.15.141:0/933556319 >> 172.21.15.141:0/933556319 conn(0x7fdf9006a7d0 msgr2=0x7fdf9006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.267+0000 7fdf8f200640 1 -- 172.21.15.141:0/933556319 shutdown_connections 2024-01-16T15:09:22.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.267+0000 7fdf8f200640 1 -- 172.21.15.141:0/933556319 wait complete. 2024-01-16T15:09:22.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.269+0000 7fdf8f200640 1 Processor -- start 2024-01-16T15:09:22.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.269+0000 7fdf8f200640 1 -- start start 2024-01-16T15:09:22.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf900ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdf900ccb50 0x7fdf90132440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf9015bf10 0x7fdf90132980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9013e7b0 con 0x7fdf900adf20 2024-01-16T15:09:22.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf90133000 con 0x7fdf9015bf10 2024-01-16T15:09:22.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf901332e0 con 0x7fdf900ccb50 2024-01-16T15:09:22.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf900ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf900ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58948/0 (socket says 172.21.15.141:58948) 2024-01-16T15:09:22.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.270+0000 7fdf8de00640 1 -- 172.21.15.141:0/3033645077 learned_addr learned my addr 172.21.15.141:0/3033645077 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:22.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf87e00640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdf900ccb50 0x7fdf90132440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf8e800640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf9015bf10 0x7fdf90132980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf8de00640 1 -- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdf900ccb50 msgr2=0x7fdf90132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf8de00640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdf900ccb50 0x7fdf90132440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf8de00640 1 -- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf9015bf10 msgr2=0x7fdf90132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf8de00640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf9015bf10 0x7fdf90132980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf8de00640 1 -- 172.21.15.141:0/3033645077 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80009400 con 0x7fdf900adf20 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf8de00640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf900ca810 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fdf80000c00 tx=0x7fdf800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf86000640 1 -- 172.21.15.141:0/3033645077 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80032900 con 0x7fdf900adf20 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf86000640 1 -- 172.21.15.141:0/3033645077 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80032a80 con 0x7fdf900adf20 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.271+0000 7fdf86000640 1 -- 172.21.15.141:0/3033645077 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80032e30 con 0x7fdf900adf20 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.272+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf90130400 con 0x7fdf900adf20 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.272+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf901309c0 con 0x7fdf900adf20 2024-01-16T15:09:22.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.273+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf90067450 con 0x7fdf900adf20 2024-01-16T15:09:22.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.274+0000 7fdf86000640 1 -- 172.21.15.141:0/3033645077 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fdf8003b3f0 con 0x7fdf900adf20 2024-01-16T15:09:22.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.275+0000 7fdf86000640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdf6806bf20 0x7fdf6806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.275+0000 7fdf86000640 1 -- 172.21.15.141:0/3033645077 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf800aa6d0 con 0x7fdf900adf20 2024-01-16T15:09:22.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.275+0000 7fdf87e00640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdf6806bf20 0x7fdf6806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.277+0000 7fdf87e00640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdf6806bf20 0x7fdf6806e3e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdf900cdc00 tx=0x7fdf78007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.255 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.279+0000 7fdf86000640 1 -- 172.21.15.141:0/3033645077 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf80076f90 con 0x7fdf900adf20 2024-01-16T15:09:22.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.504+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59638", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf9015f170 con 0x7fdf6806bf20 2024-01-16T15:09:22.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.525+0000 7fdf86000640 1 -- 172.21.15.141:0/3033645077 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fdf78007270 con 0x7fdf6806bf20 2024-01-16T15:09:22.502 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:22.502 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:22.502 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:22.503 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:22.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdf6806bf20 msgr2=0x7fdf6806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdf6806bf20 0x7fdf6806e3e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdf900cdc00 tx=0x7fdf78007c00 comp rx=0 tx=0).stop 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 msgr2=0x7fdf900ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf900ca810 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fdf80000c00 tx=0x7fdf800046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 shutdown_connections 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdf6806bf20 0x7fdf6806e3e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf9015bf10 0x7fdf90132980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdf900ccb50 0x7fdf90132440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 --2- 172.21.15.141:0/3033645077 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdf900adf20 0x7fdf900ca810 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 >> 172.21.15.141:0/3033645077 conn(0x7fdf9006a7d0 msgr2=0x7fdf9015c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 shutdown_connections 2024-01-16T15:09:22.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.528+0000 7fdf8f200640 1 -- 172.21.15.141:0/3033645077 wait complete. 2024-01-16T15:09:22.524 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:22.524 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone59639 2024-01-16T15:09:22.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.757+0000 7f82a1600640 1 Processor -- start 2024-01-16T15:09:22.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.758+0000 7f82a1600640 1 -- start start 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.759+0000 7f82a1600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f829c13df80 0x7f829c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.759+0000 7f82a1600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f829c0cf9a0 0x7f829c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.759+0000 7f82a1600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c06c470 con 0x7f829c13e360 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.759+0000 7f82a1600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c06c750 con 0x7f829c0cf9a0 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.759+0000 7f82a1600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c06ca30 con 0x7f829c13df80 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.759+0000 7f829b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f829c0cf9a0 0x7f829c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.760+0000 7f829b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f829c0cf9a0 0x7f829c0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58956/0 (socket says 172.21.15.141:58956) 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.760+0000 7f829b400640 1 -- 172.21.15.141:0/2322589002 learned_addr learned my addr 172.21.15.141:0/2322589002 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.760+0000 7f829aa00640 1 --2- 172.21.15.141:0/2322589002 >> v2:172.21.15.188:3300/0 conn(0x7f829c13df80 0x7f829c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.760+0000 7f829b400640 1 -- 172.21.15.141:0/2322589002 >> v2:172.21.15.188:3300/0 conn(0x7f829c13df80 msgr2=0x7f829c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.760+0000 7f829b400640 1 --2- 172.21.15.141:0/2322589002 >> v2:172.21.15.188:3300/0 conn(0x7f829c13df80 0x7f829c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.760+0000 7f829b400640 1 -- 172.21.15.141:0/2322589002 >> v1:172.21.15.188:6789/0 conn(0x7f829c13e360 legacy=0x7f829c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:22.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.760+0000 7f829b400640 1 -- 172.21.15.141:0/2322589002 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c06cd10 con 0x7f829c0cf9a0 2024-01-16T15:09:22.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.761+0000 7f829b400640 1 --2- 172.21.15.141:0/2322589002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c0cd840 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f829c072770 tx=0x7f829402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716f020e18257ea5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f8299600640 1 -- 172.21.15.141:0/2322589002 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294031870 con 0x7f829c0cf9a0 2024-01-16T15:09:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f8299600640 1 -- 172.21.15.141:0/2322589002 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82940319d0 con 0x7f829c0cf9a0 2024-01-16T15:09:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f8299600640 1 -- 172.21.15.141:0/2322589002 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294031d60 con 0x7f829c0cf9a0 2024-01-16T15:09:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 -- 172.21.15.141:0/2322589002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 msgr2=0x7f829c0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 --2- 172.21.15.141:0/2322589002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c0cd840 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f829c072770 tx=0x7f829402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 -- 172.21.15.141:0/2322589002 shutdown_connections 2024-01-16T15:09:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 --2- 172.21.15.141:0/2322589002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c0cd840 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 --2- 172.21.15.141:0/2322589002 >> v2:172.21.15.188:3300/0 conn(0x7f829c13df80 0x7f829c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 -- 172.21.15.141:0/2322589002 >> 172.21.15.141:0/2322589002 conn(0x7f829c06a800 msgr2=0x7f829c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 -- 172.21.15.141:0/2322589002 shutdown_connections 2024-01-16T15:09:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.762+0000 7f82a1600640 1 -- 172.21.15.141:0/2322589002 wait complete. 2024-01-16T15:09:22.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.764+0000 7f82a1600640 1 Processor -- start 2024-01-16T15:09:22.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.764+0000 7f82a1600640 1 -- start start 2024-01-16T15:09:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.765+0000 7f82a1600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c14c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.765+0000 7f82a1600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f829c13df80 0x7f829c1f7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.765+0000 7f82a1600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c13e360 0x7f829c13d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.765+0000 7f82a1600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c071cc0 con 0x7f829c0cf9a0 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f82a1600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c1f8410 con 0x7f829c13e360 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f82a1600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0707c0 con 0x7f829c13df80 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c14c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c14c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58964/0 (socket says 172.21.15.141:58964) 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 -- 172.21.15.141:0/1822721508 learned_addr learned my addr 172.21.15.141:0/1822721508 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829b400640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c13e360 0x7f829c13d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 -- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f829c13df80 msgr2=0x7f829c1f7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f829c13df80 0x7f829c1f7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 -- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c13e360 msgr2=0x7f829c13d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c13e360 0x7f829c13d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.766+0000 7f829aa00640 1 -- 172.21.15.141:0/1822721508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82940094d0 con 0x7f829c0cf9a0 2024-01-16T15:09:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.767+0000 7f829aa00640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c14c5e0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f828c007ef0 tx=0x7f828c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.767+0000 7f8292a00640 1 -- 172.21.15.141:0/1822721508 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c014620 con 0x7f829c0cf9a0 2024-01-16T15:09:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.767+0000 7f8292a00640 1 -- 172.21.15.141:0/1822721508 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c014780 con 0x7f829c0cf9a0 2024-01-16T15:09:22.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.767+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c070aa0 con 0x7f829c0cf9a0 2024-01-16T15:09:22.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.767+0000 7f8292a00640 1 -- 172.21.15.141:0/1822721508 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c014b30 con 0x7f829c0cf9a0 2024-01-16T15:09:22.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.767+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c071060 con 0x7f829c0cf9a0 2024-01-16T15:09:22.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.768+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c0675d0 con 0x7f829c0cf9a0 2024-01-16T15:09:22.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.769+0000 7f8292a00640 1 -- 172.21.15.141:0/1822721508 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f828c014c90 con 0x7f829c0cf9a0 2024-01-16T15:09:22.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.770+0000 7f8292a00640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f827406bf20 0x7f827406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.770+0000 7f8292a00640 1 -- 172.21.15.141:0/1822721508 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f828c088fa0 con 0x7f829c0cf9a0 2024-01-16T15:09:22.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.771+0000 7f829a000640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f827406bf20 0x7f827406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.772+0000 7f829a000640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f827406bf20 0x7f827406e3e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f829c1552d0 tx=0x7f82840079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:22.775+0000 7f8292a00640 1 -- 172.21.15.141:0/1822721508 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f828c0558a0 con 0x7f829c0cf9a0 2024-01-16T15:09:22.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.004+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone59639", "target": ["mon-mgr", ""]}) v1 -- 0x7f829c15f9c0 con 0x7f827406bf20 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.028+0000 7f8292a00640 1 -- 172.21.15.141:0/1822721508 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f8284003950 con 0x7f827406bf20 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:23.007 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:23.008 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:23.008 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:23.008 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:23.008 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:23.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f827406bf20 msgr2=0x7f827406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f827406bf20 0x7f827406e3e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f829c1552d0 tx=0x7f82840079b0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 msgr2=0x7f829c14c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c14c5e0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f828c007ef0 tx=0x7f828c00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 shutdown_connections 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f827406bf20 0x7f827406e3e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c13e360 0x7f829c13d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f829c13df80 0x7f829c1f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 --2- 172.21.15.141:0/1822721508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f829c0cf9a0 0x7f829c14c5e0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 >> 172.21.15.141:0/1822721508 conn(0x7f829c06a800 msgr2=0x7f829c1540f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 shutdown_connections 2024-01-16T15:09:23.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.031+0000 7f82a1600640 1 -- 172.21.15.141:0/1822721508 wait complete. 2024-01-16T15:09:23.028 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:23.028 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596310 2024-01-16T15:09:23.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.258+0000 7f1ebfc00640 1 Processor -- start 2024-01-16T15:09:23.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.258+0000 7f1ebfc00640 1 -- start start 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.260+0000 7f1ebfc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb806b690 0x7f1eb815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.260+0000 7f1ebfc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1eb815e270 0x7f1eb8070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.260+0000 7f1ebfc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8154490 con 0x7f1eb81540b0 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.260+0000 7f1ebfc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb815d550 con 0x7f1eb815e270 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.260+0000 7f1ebfc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb815a7b0 con 0x7f1eb806b690 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1ebdc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1eb815e270 0x7f1eb8070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1ebdc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1eb815e270 0x7f1eb8070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34110/0 (socket says 172.21.15.141:34110) 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4168382214 learned_addr learned my addr 172.21.15.141:0/4168382214 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:23.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1eb7e00640 1 --2- 172.21.15.141:0/4168382214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb806b690 0x7f1eb815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4168382214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb806b690 msgr2=0x7f1eb815dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1ebdc00640 1 --2- 172.21.15.141:0/4168382214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb806b690 0x7f1eb815dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4168382214 >> v1:172.21.15.188:6789/0 conn(0x7f1eb81540b0 legacy=0x7f1eb8159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.261+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4168382214 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb815aa90 con 0x7f1eb815e270 2024-01-16T15:09:23.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.262+0000 7f1ebdc00640 1 --2- 172.21.15.141:0/4168382214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815e270 0x7f1eb8070af0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009e50 tx=0x7f1eb002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3787d4fe8f428953 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.262+0000 7f1eb7400640 1 -- 172.21.15.141:0/4168382214 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb002c9e0 con 0x7f1eb815e270 2024-01-16T15:09:23.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1eb7400640 1 -- 172.21.15.141:0/4168382214 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb002cb40 con 0x7f1eb815e270 2024-01-16T15:09:23.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1eb7400640 1 -- 172.21.15.141:0/4168382214 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb00343f0 con 0x7f1eb815e270 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4168382214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815e270 msgr2=0x7f1eb8070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4168382214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815e270 0x7f1eb8070af0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009e50 tx=0x7f1eb002be40 comp rx=0 tx=0).stop 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4168382214 shutdown_connections 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4168382214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815e270 0x7f1eb8070af0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4168382214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb806b690 0x7f1eb815dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4168382214 >> 172.21.15.141:0/4168382214 conn(0x7f1eb80cf700 msgr2=0x7f1eb80cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4168382214 shutdown_connections 2024-01-16T15:09:23.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.263+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4168382214 wait complete. 2024-01-16T15:09:23.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.266+0000 7f1ebfc00640 1 Processor -- start 2024-01-16T15:09:23.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.266+0000 7f1ebfc00640 1 -- start start 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebfc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb806b690 0x7f1eb814ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebfc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb81540b0 0x7f1eb8150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebfc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815b090 0x7f1eb80b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebfc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80ce4c0 con 0x7f1eb815b090 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebfc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb815b520 con 0x7f1eb806b690 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebfc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb815b800 con 0x7f1eb81540b0 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebdc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815b090 0x7f1eb80b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebdc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815b090 0x7f1eb80b22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34116/0 (socket says 172.21.15.141:34116) 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.267+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4083291776 learned_addr learned my addr 172.21.15.141:0/4083291776 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb81540b0 msgr2=0x7f1eb8150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebdc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb81540b0 0x7f1eb8150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb806b690 msgr2=0x7f1eb814ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebd200640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb806b690 0x7f1eb814ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebdc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb806b690 0x7f1eb814ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebdc00640 1 -- 172.21.15.141:0/4083291776 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb0009b30 con 0x7f1eb815b090 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebd200640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb806b690 0x7f1eb814ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebdc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815b090 0x7f1eb80b22b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0036040 tx=0x7f1eb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1eb4c00640 1 -- 172.21.15.141:0/4083291776 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0009510 con 0x7f1eb815b090 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1eb4c00640 1 -- 172.21.15.141:0/4083291776 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb003e450 con 0x7f1eb815b090 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb815bae0 con 0x7f1eb815b090 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.268+0000 7f1eb4c00640 1 -- 172.21.15.141:0/4083291776 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb003e7e0 con 0x7f1eb815b090 2024-01-16T15:09:23.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.269+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb815c0a0 con 0x7f1eb815b090 2024-01-16T15:09:23.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.270+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb80675d0 con 0x7f1eb815b090 2024-01-16T15:09:23.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.270+0000 7f1eb4c00640 1 -- 172.21.15.141:0/4083291776 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f1eb003e940 con 0x7f1eb815b090 2024-01-16T15:09:23.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.271+0000 7f1eb4c00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e9006bf20 0x7f1e9006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.271+0000 7f1eb4c00640 1 -- 172.21.15.141:0/4083291776 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1eb00ab5e0 con 0x7f1eb815b090 2024-01-16T15:09:23.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.272+0000 7f1ebd200640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e9006bf20 0x7f1e9006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.273+0000 7f1ebd200640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e9006bf20 0x7f1e9006e3e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8003eb0 tx=0x7f1ea8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.276+0000 7f1eb4c00640 1 -- 172.21.15.141:0/4083291776 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1eb0077c80 con 0x7f1eb815b090 2024-01-16T15:09:23.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.499+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596310", "target": ["mon-mgr", ""]}) v1 -- 0x7f1eb8073640 con 0x7f1e9006bf20 2024-01-16T15:09:23.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.520+0000 7f1eb4c00640 1 -- 172.21.15.141:0/4083291776 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f1eb8073640 con 0x7f1e9006bf20 2024-01-16T15:09:23.497 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:23.497 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:23.498 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:23.499 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:23.499 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e9006bf20 msgr2=0x7f1e9006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e9006bf20 0x7f1e9006e3e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8003eb0 tx=0x7f1ea8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815b090 msgr2=0x7f1eb80b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815b090 0x7f1eb80b22b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0036040 tx=0x7f1eb0004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 shutdown_connections 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e9006bf20 0x7f1e9006e3e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1eb815b090 0x7f1eb80b22b0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1eb81540b0 0x7f1eb8150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 --2- 172.21.15.141:0/4083291776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eb806b690 0x7f1eb814ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 >> 172.21.15.141:0/4083291776 conn(0x7f1eb80cf700 msgr2=0x7f1eb81e3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 shutdown_connections 2024-01-16T15:09:23.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.523+0000 7f1ebfc00640 1 -- 172.21.15.141:0/4083291776 wait complete. 2024-01-16T15:09:23.517 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:23.517 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596311 2024-01-16T15:09:23.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.750+0000 7f6f1e000640 1 Processor -- start 2024-01-16T15:09:23.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.751+0000 7f6f1e000640 1 -- start start 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.752+0000 7f6f1e000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6f18143ae0 0x7f6f180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.752+0000 7f6f1e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f180cf7d0 0x7f6f1815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.752+0000 7f6f1e000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f180cfc30 con 0x7f6f1815e1e0 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.752+0000 7f6f1e000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f180b1ee0 con 0x7f6f18143ae0 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.752+0000 7f6f1e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f180b2780 con 0x7f6f180cf7d0 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f16a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f180cf7d0 0x7f6f1815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f17400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6f18143ae0 0x7f6f180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f17400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6f18143ae0 0x7f6f180cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:60558/0 (socket says 172.21.15.141:60558) 2024-01-16T15:09:23.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f16a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f180cf7d0 0x7f6f1815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:49028/0 (socket says 172.21.15.141:49028) 2024-01-16T15:09:23.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f17400640 1 -- 172.21.15.141:0/2264791292 learned_addr learned my addr 172.21.15.141:0/2264791292 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:23.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f16a00640 1 -- 172.21.15.141:0/2264791292 >> v2:172.21.15.188:3300/0 conn(0x7f6f18143ae0 msgr2=0x7f6f180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f16a00640 1 --2- 172.21.15.141:0/2264791292 >> v2:172.21.15.188:3300/0 conn(0x7f6f18143ae0 0x7f6f180cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f16a00640 1 -- 172.21.15.141:0/2264791292 >> v1:172.21.15.188:6789/0 conn(0x7f6f1815e1e0 legacy=0x7f6f1815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.753+0000 7f6f16a00640 1 -- 172.21.15.141:0/2264791292 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1815e990 con 0x7f6f180cf7d0 2024-01-16T15:09:23.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.755+0000 7f6f16a00640 1 --2- 172.21.15.141:0/2264791292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f180cf7d0 0x7f6f1815dca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6f000097a0 tx=0x7f6f0002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4c6c567587b7afb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.755+0000 7f6f16000640 1 -- 172.21.15.141:0/2264791292 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00031830 con 0x7f6f180cf7d0 2024-01-16T15:09:23.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.755+0000 7f6f16000640 1 -- 172.21.15.141:0/2264791292 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00031990 con 0x7f6f180cf7d0 2024-01-16T15:09:23.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.755+0000 7f6f16000640 1 -- 172.21.15.141:0/2264791292 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00031d20 con 0x7f6f180cf7d0 2024-01-16T15:09:23.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 -- 172.21.15.141:0/2264791292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f180cf7d0 msgr2=0x7f6f1815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2264791292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f180cf7d0 0x7f6f1815dca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6f000097a0 tx=0x7f6f0002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 -- 172.21.15.141:0/2264791292 shutdown_connections 2024-01-16T15:09:23.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2264791292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f180cf7d0 0x7f6f1815dca0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2264791292 >> v2:172.21.15.188:3300/0 conn(0x7f6f18143ae0 0x7f6f180cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 -- 172.21.15.141:0/2264791292 >> 172.21.15.141:0/2264791292 conn(0x7f6f1806a800 msgr2=0x7f6f1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 -- 172.21.15.141:0/2264791292 shutdown_connections 2024-01-16T15:09:23.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.756+0000 7f6f1e000640 1 -- 172.21.15.141:0/2264791292 wait complete. 2024-01-16T15:09:23.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.758+0000 7f6f1e000640 1 Processor -- start 2024-01-16T15:09:23.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.758+0000 7f6f1e000640 1 -- start start 2024-01-16T15:09:23.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f1e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f180cf7d0 0x7f6f180c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f1e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f18143ae0 0x7f6f180c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f1e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f1815e1e0 0x7f6f180aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f1e000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1813ff60 con 0x7f6f180cf7d0 2024-01-16T15:09:23.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f1e000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18140300 con 0x7f6f18143ae0 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f1e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18140e80 con 0x7f6f1815e1e0 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f180cf7d0 0x7f6f180c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f180cf7d0 0x7f6f180c4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34128/0 (socket says 172.21.15.141:34128) 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 -- 172.21.15.141:0/2926018730 learned_addr learned my addr 172.21.15.141:0/2926018730 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 -- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f1815e1e0 msgr2=0x7f6f180aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f1815e1e0 0x7f6f180aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 -- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f18143ae0 msgr2=0x7f6f180c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f18143ae0 0x7f6f180c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.759+0000 7f6f17400640 1 -- 172.21.15.141:0/2926018730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00009400 con 0x7f6f180cf7d0 2024-01-16T15:09:23.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.760+0000 7f6f17400640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f180cf7d0 0x7f6f180c4b40 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800eb60 tx=0x7f6f0800d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.760+0000 7f6f0f400640 1 -- 172.21.15.141:0/2926018730 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08019680 con 0x7f6f180cf7d0 2024-01-16T15:09:23.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.760+0000 7f6f0f400640 1 -- 172.21.15.141:0/2926018730 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f080197e0 con 0x7f6f180cf7d0 2024-01-16T15:09:23.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.760+0000 7f6f0f400640 1 -- 172.21.15.141:0/2926018730 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08019b90 con 0x7f6f180cf7d0 2024-01-16T15:09:23.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.760+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f181f2ae0 con 0x7f6f180cf7d0 2024-01-16T15:09:23.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.760+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f18140830 con 0x7f6f180cf7d0 2024-01-16T15:09:23.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.761+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1815e620 con 0x7f6f180cf7d0 2024-01-16T15:09:23.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.762+0000 7f6f0f400640 1 -- 172.21.15.141:0/2926018730 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6f08019cf0 con 0x7f6f180cf7d0 2024-01-16T15:09:23.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.763+0000 7f6f0f400640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ef006bf50 0x7f6ef006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.764+0000 7f6f0f400640 1 -- 172.21.15.141:0/2926018730 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6f08014070 con 0x7f6f180cf7d0 2024-01-16T15:09:23.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.764+0000 7f6f16a00640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ef006bf50 0x7f6ef006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.765+0000 7f6f16a00640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ef006bf50 0x7f6ef006e410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f00030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.768+0000 7f6f0f400640 1 -- 172.21.15.141:0/2926018730 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f1815e620 con 0x7f6f180cf7d0 2024-01-16T15:09:23.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:23.979+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596311", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f180b2410 con 0x7f6ef006bf50 2024-01-16T15:09:23.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.004+0000 7f6f0f400640 1 -- 172.21.15.141:0/2926018730 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f6f00031b00 con 0x7f6ef006bf50 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:23.982 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:23.983 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:23.983 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:23.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ef006bf50 msgr2=0x7f6ef006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ef006bf50 0x7f6ef006e410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f00030000 comp rx=0 tx=0).stop 2024-01-16T15:09:23.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f180cf7d0 msgr2=0x7f6f180c4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f180cf7d0 0x7f6f180c4b40 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800eb60 tx=0x7f6f0800d720 comp rx=0 tx=0).stop 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 shutdown_connections 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6ef006bf50 0x7f6ef006e410 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6f1815e1e0 0x7f6f180aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f18143ae0 0x7f6f180c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 --2- 172.21.15.141:0/2926018730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f180cf7d0 0x7f6f180c4b40 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 >> 172.21.15.141:0/2926018730 conn(0x7f6f1806a800 msgr2=0x7f6f180b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 shutdown_connections 2024-01-16T15:09:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.007+0000 7f6f1e000640 1 -- 172.21.15.141:0/2926018730 wait complete. 2024-01-16T15:09:24.003 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:24.003 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596312 2024-01-16T15:09:24.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.234+0000 7ff567200640 1 Processor -- start 2024-01-16T15:09:24.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.235+0000 7ff567200640 1 -- start start 2024-01-16T15:09:24.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.236+0000 7ff567200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff5600cf9a0 0x7ff56015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.236+0000 7ff567200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff56015e1e0 0x7ff5600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.236+0000 7ff567200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600b1ee0 con 0x7ff560143ae0 2024-01-16T15:09:24.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.236+0000 7ff567200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5600b2780 con 0x7ff5600cf9a0 2024-01-16T15:09:24.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.236+0000 7ff567200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56015e990 con 0x7ff56015e1e0 2024-01-16T15:09:24.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff557e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff5600cf9a0 0x7ff56015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff557e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff5600cf9a0 0x7ff56015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34132/0 (socket says 172.21.15.141:34132) 2024-01-16T15:09:24.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff557e00640 1 -- 172.21.15.141:0/2909906314 learned_addr learned my addr 172.21.15.141:0/2909906314 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:24.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff565200640 1 --2- 172.21.15.141:0/2909906314 >> v2:172.21.15.188:3300/0 conn(0x7ff56015e1e0 0x7ff5600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff557e00640 1 -- 172.21.15.141:0/2909906314 >> v2:172.21.15.188:3300/0 conn(0x7ff56015e1e0 msgr2=0x7ff5600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff557e00640 1 --2- 172.21.15.141:0/2909906314 >> v2:172.21.15.188:3300/0 conn(0x7ff56015e1e0 0x7ff5600702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff557e00640 1 -- 172.21.15.141:0/2909906314 >> v1:172.21.15.188:6789/0 conn(0x7ff560143ae0 legacy=0x7ff5600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff557e00640 1 -- 172.21.15.141:0/2909906314 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600b2130 con 0x7ff5600cf9a0 2024-01-16T15:09:24.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.237+0000 7ff565200640 1 --2- 172.21.15.141:0/2909906314 >> v2:172.21.15.188:3300/0 conn(0x7ff56015e1e0 0x7ff5600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:24.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.238+0000 7ff557e00640 1 --2- 172.21.15.141:0/2909906314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5600cf9a0 0x7ff56015dca0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ff5600ae9d0 tx=0x7ff54802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b080f1b680f9047e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.239+0000 7ff55f400640 1 -- 172.21.15.141:0/2909906314 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548031870 con 0x7ff5600cf9a0 2024-01-16T15:09:24.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.239+0000 7ff55f400640 1 -- 172.21.15.141:0/2909906314 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5480319d0 con 0x7ff5600cf9a0 2024-01-16T15:09:24.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.239+0000 7ff55f400640 1 -- 172.21.15.141:0/2909906314 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548031d60 con 0x7ff5600cf9a0 2024-01-16T15:09:24.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.239+0000 7ff567200640 1 -- 172.21.15.141:0/2909906314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5600cf9a0 msgr2=0x7ff56015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.239+0000 7ff567200640 1 --2- 172.21.15.141:0/2909906314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5600cf9a0 0x7ff56015dca0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ff5600ae9d0 tx=0x7ff54802b910 comp rx=0 tx=0).stop 2024-01-16T15:09:24.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.240+0000 7ff567200640 1 -- 172.21.15.141:0/2909906314 shutdown_connections 2024-01-16T15:09:24.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.240+0000 7ff567200640 1 --2- 172.21.15.141:0/2909906314 >> v2:172.21.15.188:3300/0 conn(0x7ff56015e1e0 0x7ff5600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.240+0000 7ff567200640 1 --2- 172.21.15.141:0/2909906314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5600cf9a0 0x7ff56015dca0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.240+0000 7ff567200640 1 -- 172.21.15.141:0/2909906314 >> 172.21.15.141:0/2909906314 conn(0x7ff56006a800 msgr2=0x7ff56006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.240+0000 7ff567200640 1 -- 172.21.15.141:0/2909906314 shutdown_connections 2024-01-16T15:09:24.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.240+0000 7ff567200640 1 -- 172.21.15.141:0/2909906314 wait complete. 2024-01-16T15:09:24.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.241+0000 7ff567200640 1 Processor -- start 2024-01-16T15:09:24.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.242+0000 7ff567200640 1 -- start start 2024-01-16T15:09:24.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff567200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5600cf9a0 0x7ff5601597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff567200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff560143ae0 0x7ff5601397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff567200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff56015e1e0 0x7ff560141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff567200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56013b310 con 0x7ff56015e1e0 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff567200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56013a7e0 con 0x7ff5600cf9a0 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff567200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56013a220 con 0x7ff560143ae0 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff56015e1e0 0x7ff560141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff56015e1e0 0x7ff560141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34144/0 (socket says 172.21.15.141:34144) 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 -- 172.21.15.141:0/2939598069 learned_addr learned my addr 172.21.15.141:0/2939598069 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff55fe00640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5600cf9a0 0x7ff5601597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff557e00640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff560143ae0 0x7ff5601397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 -- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff560143ae0 msgr2=0x7ff5601397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff560143ae0 0x7ff5601397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 -- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5600cf9a0 msgr2=0x7ff5601597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5600cf9a0 0x7ff5601597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff565200640 1 -- 172.21.15.141:0/2939598069 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480094d0 con 0x7ff56015e1e0 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.243+0000 7ff557e00640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff560143ae0 0x7ff5601397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:24.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.244+0000 7ff565200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff56015e1e0 0x7ff560141320 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff55800cef0 tx=0x7ff55800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.244+0000 7ff55cc00640 1 -- 172.21.15.141:0/2939598069 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55800d840 con 0x7ff56015e1e0 2024-01-16T15:09:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.244+0000 7ff55cc00640 1 -- 172.21.15.141:0/2939598069 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55800d9c0 con 0x7ff56015e1e0 2024-01-16T15:09:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.244+0000 7ff55cc00640 1 -- 172.21.15.141:0/2939598069 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55800dd70 con 0x7ff56015e1e0 2024-01-16T15:09:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.244+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56013a500 con 0x7ff56015e1e0 2024-01-16T15:09:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.244+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff560159cf0 con 0x7ff56015e1e0 2024-01-16T15:09:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.245+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56013b450 con 0x7ff56015e1e0 2024-01-16T15:09:24.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.246+0000 7ff55cc00640 1 -- 172.21.15.141:0/2939598069 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff55801e3b0 con 0x7ff56015e1e0 2024-01-16T15:09:24.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.247+0000 7ff55cc00640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff53806bf20 0x7ff53806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.247+0000 7ff55cc00640 1 -- 172.21.15.141:0/2939598069 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff558089340 con 0x7ff56015e1e0 2024-01-16T15:09:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.248+0000 7ff55fe00640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff53806bf20 0x7ff53806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.249+0000 7ff55fe00640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff53806bf20 0x7ff53806e3e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff550004ea0 tx=0x7ff550009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.251+0000 7ff55cc00640 1 -- 172.21.15.141:0/2939598069 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff56013b450 con 0x7ff56015e1e0 2024-01-16T15:09:24.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.477+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596312", "target": ["mon-mgr", ""]}) v1 -- 0x7ff56015e670 con 0x7ff53806bf20 2024-01-16T15:09:24.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.501+0000 7ff55cc00640 1 -- 172.21.15.141:0/2939598069 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7ff550007db0 con 0x7ff53806bf20 2024-01-16T15:09:24.478 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:24.479 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:24.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.503+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff53806bf20 msgr2=0x7ff53806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.503+0000 7ff567200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff53806bf20 0x7ff53806e3e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff550004ea0 tx=0x7ff550009040 comp rx=0 tx=0).stop 2024-01-16T15:09:24.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.503+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff56015e1e0 msgr2=0x7ff560141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.503+0000 7ff567200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff56015e1e0 0x7ff560141320 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff55800cef0 tx=0x7ff55800a710 comp rx=0 tx=0).stop 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 shutdown_connections 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff53806bf20 0x7ff53806e3e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff56015e1e0 0x7ff560141320 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff560143ae0 0x7ff5601397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 --2- 172.21.15.141:0/2939598069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5600cf9a0 0x7ff5601597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 >> 172.21.15.141:0/2939598069 conn(0x7ff56006a800 msgr2=0x7ff56015da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 shutdown_connections 2024-01-16T15:09:24.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.504+0000 7ff567200640 1 -- 172.21.15.141:0/2939598069 wait complete. 2024-01-16T15:09:24.498 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:24.498 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596313 2024-01-16T15:09:24.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.723+0000 7f14af600640 1 Processor -- start 2024-01-16T15:09:24.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.724+0000 7f14af600640 1 -- start start 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.725+0000 7f14af600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f14a813e360 0x7f14a80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.725+0000 7f14af600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14a815f9c0 0x7f14a80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.725+0000 7f14af600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a815c5f0 con 0x7f14a813df80 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.725+0000 7f14af600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a806c470 con 0x7f14a815f9c0 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.725+0000 7f14af600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a806c750 con 0x7f14a813e360 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.725+0000 7f14ad600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14a815f9c0 0x7f14a80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.726+0000 7f14ad600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14a815f9c0 0x7f14a80cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34154/0 (socket says 172.21.15.141:34154) 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.726+0000 7f14ad600640 1 -- 172.21.15.141:0/2500290198 learned_addr learned my addr 172.21.15.141:0/2500290198 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:24.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.726+0000 7f149fc00640 1 --2- 172.21.15.141:0/2500290198 >> v2:172.21.15.188:3300/0 conn(0x7f14a813e360 0x7f14a80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.726+0000 7f14ad600640 1 -- 172.21.15.141:0/2500290198 >> v2:172.21.15.188:3300/0 conn(0x7f14a813e360 msgr2=0x7f14a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.726+0000 7f14ad600640 1 --2- 172.21.15.141:0/2500290198 >> v2:172.21.15.188:3300/0 conn(0x7f14a813e360 0x7f14a80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.726+0000 7f14ad600640 1 -- 172.21.15.141:0/2500290198 >> v1:172.21.15.188:6789/0 conn(0x7f14a813df80 legacy=0x7f14a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.726+0000 7f14ad600640 1 -- 172.21.15.141:0/2500290198 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a806ca30 con 0x7f14a815f9c0 2024-01-16T15:09:24.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.727+0000 7f14ad600640 1 --2- 172.21.15.141:0/2500290198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a815f9c0 0x7f14a80cf7d0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f14a80ccae0 tx=0x7f14a002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5965842449f84259 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.727+0000 7f14a7e00640 1 -- 172.21.15.141:0/2500290198 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a002ca50 con 0x7f14a815f9c0 2024-01-16T15:09:24.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.727+0000 7f14a7e00640 1 -- 172.21.15.141:0/2500290198 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a002cbb0 con 0x7f14a815f9c0 2024-01-16T15:09:24.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.727+0000 7f14a7e00640 1 -- 172.21.15.141:0/2500290198 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a00343f0 con 0x7f14a815f9c0 2024-01-16T15:09:24.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 -- 172.21.15.141:0/2500290198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a815f9c0 msgr2=0x7f14a80cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 --2- 172.21.15.141:0/2500290198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a815f9c0 0x7f14a80cf7d0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f14a80ccae0 tx=0x7f14a002beb0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 -- 172.21.15.141:0/2500290198 shutdown_connections 2024-01-16T15:09:24.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 --2- 172.21.15.141:0/2500290198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a815f9c0 0x7f14a80cf7d0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 --2- 172.21.15.141:0/2500290198 >> v2:172.21.15.188:3300/0 conn(0x7f14a813e360 0x7f14a80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 -- 172.21.15.141:0/2500290198 >> 172.21.15.141:0/2500290198 conn(0x7f14a806a800 msgr2=0x7f14a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 -- 172.21.15.141:0/2500290198 shutdown_connections 2024-01-16T15:09:24.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.728+0000 7f14af600640 1 -- 172.21.15.141:0/2500290198 wait complete. 2024-01-16T15:09:24.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.730+0000 7f14af600640 1 Processor -- start 2024-01-16T15:09:24.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.730+0000 7f14af600640 1 -- start start 2024-01-16T15:09:24.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14af600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a813df80 0x7f14a813d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14af600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14a813e360 0x7f14a80b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14af600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14a815f9c0 0x7f14a80b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14af600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80aaaa0 con 0x7f14a813df80 2024-01-16T15:09:24.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14af600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80a9a40 con 0x7f14a815f9c0 2024-01-16T15:09:24.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14af600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80a9ce0 con 0x7f14a813e360 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14acc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a813df80 0x7f14a813d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14acc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a813df80 0x7f14a813d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34166/0 (socket says 172.21.15.141:34166) 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14acc00640 1 -- 172.21.15.141:0/2854467226 learned_addr learned my addr 172.21.15.141:0/2854467226 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f149fc00640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14a813e360 0x7f14a80b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14acc00640 1 -- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14a813e360 msgr2=0x7f14a80b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14acc00640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14a813e360 0x7f14a80b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14acc00640 1 -- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14a815f9c0 msgr2=0x7f14a80b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.731+0000 7f14acc00640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14a815f9c0 0x7f14a80b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.732+0000 7f14acc00640 1 -- 172.21.15.141:0/2854467226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a0009b30 con 0x7f14a813df80 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.732+0000 7f14acc00640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a813df80 0x7f14a813d940 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f149800b3f0 tx=0x7f149800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.732+0000 7f14a5600640 1 -- 172.21.15.141:0/2854467226 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498011830 con 0x7f14a813df80 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.732+0000 7f14a5600640 1 -- 172.21.15.141:0/2854467226 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1498011990 con 0x7f14a813df80 2024-01-16T15:09:24.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.732+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a80a9fc0 con 0x7f14a813df80 2024-01-16T15:09:24.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.732+0000 7f14a5600640 1 -- 172.21.15.141:0/2854467226 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498011d40 con 0x7f14a813df80 2024-01-16T15:09:24.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.732+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a80aa580 con 0x7f14a813df80 2024-01-16T15:09:24.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.733+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a8073c30 con 0x7f14a813df80 2024-01-16T15:09:24.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.734+0000 7f14a5600640 1 -- 172.21.15.141:0/2854467226 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f149801a400 con 0x7f14a813df80 2024-01-16T15:09:24.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.735+0000 7f14a5600640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f148006bf20 0x7f148006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.735+0000 7f14a5600640 1 -- 172.21.15.141:0/2854467226 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1498088630 con 0x7f14a813df80 2024-01-16T15:09:24.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.736+0000 7f149fc00640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f148006bf20 0x7f148006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.737+0000 7f149fc00640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f148006bf20 0x7f148006e3e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1490006fd0 tx=0x7f1490008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.739+0000 7f14a5600640 1 -- 172.21.15.141:0/2854467226 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14a8073c30 con 0x7f14a813df80 2024-01-16T15:09:24.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.969+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596313", "target": ["mon-mgr", ""]}) v1 -- 0x7f14a8073c30 con 0x7f148006bf20 2024-01-16T15:09:24.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.989+0000 7f14a5600640 1 -- 172.21.15.141:0/2854467226 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f14a8073c30 con 0x7f148006bf20 2024-01-16T15:09:24.966 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:24.966 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:24.967 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:24.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f148006bf20 msgr2=0x7f148006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f148006bf20 0x7f148006e3e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1490006fd0 tx=0x7f1490008040 comp rx=0 tx=0).stop 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a813df80 msgr2=0x7f14a813d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a813df80 0x7f14a813d940 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f149800b3f0 tx=0x7f149800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 shutdown_connections 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f148006bf20 0x7f148006e3e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14a815f9c0 0x7f14a80b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14a813e360 0x7f14a80b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 --2- 172.21.15.141:0/2854467226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14a813df80 0x7f14a813d940 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 >> 172.21.15.141:0/2854467226 conn(0x7f14a806a800 msgr2=0x7f14a815a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 shutdown_connections 2024-01-16T15:09:24.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:24.992+0000 7f14af600640 1 -- 172.21.15.141:0/2854467226 wait complete. 2024-01-16T15:09:24.988 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:24.988 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596314 2024-01-16T15:09:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.212+0000 7f9f6e200640 1 Processor -- start 2024-01-16T15:09:25.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.213+0000 7f9f6e200640 1 -- start start 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.214+0000 7f9f6e200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9f680cf9a0 0x7f9f68166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.214+0000 7f9f6e200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6815c5f0 con 0x7f9f6813df80 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f6e200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6806c470 con 0x7f9f6813e360 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f6e200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6806c750 con 0x7f9f680cf9a0 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f67400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9f6813df80 0x7f9f6815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55554/0 (socket says 172.21.15.141:55554) 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f67400640 1 -- 172.21.15.141:0/2589780390 learned_addr learned my addr 172.21.15.141:0/2589780390 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806698575 0 0) 0x7f9f6815c5f0 con 0x7f9f6813df80 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f48003520 con 0x7f9f6813df80 2024-01-16T15:09:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f67e00640 1 --2- 172.21.15.141:0/2589780390 >> v2:172.21.15.188:3300/0 conn(0x7f9f680cf9a0 0x7f9f68166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274084569 0 0) 0x7f9f48003520 con 0x7f9f6813df80 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 >> v2:172.21.15.188:3300/0 conn(0x7f9f680cf9a0 msgr2=0x7f9f68166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f66a00640 1 --2- 172.21.15.141:0/2589780390 >> v2:172.21.15.188:3300/0 conn(0x7f9f680cf9a0 0x7f9f68166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.215+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 >> v1:172.21.15.188:6789/0 conn(0x7f9f6813e360 legacy=0x7f9f680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6806ca30 con 0x7f9f6813df80 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9f58001670 con 0x7f9f6813df80 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 >> v1:172.21.15.141:6789/0 conn(0x7f9f6813df80 legacy=0x7f9f6815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f48003970 0x7f9f48003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f480047c0 0x7f9f48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f48005180 0x7f9f4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6806c750 con 0x7f9f48003970 2024-01-16T15:09:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4800c0b0 con 0x7f9f48005180 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4800c270 con 0x7f9f480047c0 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f67400640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f48003970 0x7f9f48003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f67400640 1 -- 172.21.15.141:0/2589780390 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f480047c0 msgr2=0x7f9f48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f67400640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f480047c0 0x7f9f48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f67400640 1 -- 172.21.15.141:0/2589780390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f48005180 msgr2=0x7f9f4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f67400640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f48005180 0x7f9f4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f67400640 1 -- 172.21.15.141:0/2589780390 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48003520 con 0x7f9f48003970 2024-01-16T15:09:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.216+0000 7f9f67e00640 1 -- 172.21.15.141:0/2589780390 reap_dead start 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.217+0000 7f9f67400640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f48003970 0x7f9f48003d70 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9f58001eb0 tx=0x7f9f480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebbc398176118f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58005ad0 con 0x7f9f48003970 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58005c50 con 0x7f9f48003970 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f66a00640 1 -- 172.21.15.141:0/2589780390 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5802c8b0 con 0x7f9f48003970 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f6e200640 1 -- 172.21.15.141:0/2589780390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f48003970 msgr2=0x7f9f48003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f6e200640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f48003970 0x7f9f48003d70 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9f58001eb0 tx=0x7f9f480057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f6e200640 1 -- 172.21.15.141:0/2589780390 shutdown_connections 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f6e200640 1 --2- 172.21.15.141:0/2589780390 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f48003970 0x7f9f48003d70 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f6e200640 1 -- 172.21.15.141:0/2589780390 >> 172.21.15.141:0/2589780390 conn(0x7f9f6806a800 msgr2=0x7f9f6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f6e200640 1 -- 172.21.15.141:0/2589780390 shutdown_connections 2024-01-16T15:09:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.218+0000 7f9f6e200640 1 -- 172.21.15.141:0/2589780390 wait complete. 2024-01-16T15:09:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.220+0000 7f9f6e200640 1 Processor -- start 2024-01-16T15:09:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.220+0000 7f9f6e200640 1 -- start start 2024-01-16T15:09:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.221+0000 7f9f6e200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f48003970 0x7f9f6813d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.221+0000 7f9f6e200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f6815a140 0x7f9f6815a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.221+0000 7f9f6e200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f68070ae0 0x7f9f68070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.221+0000 7f9f6e200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f681e4ec0 con 0x7f9f6815a140 2024-01-16T15:09:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.221+0000 7f9f6e200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6815ab00 con 0x7f9f68070ae0 2024-01-16T15:09:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.221+0000 7f9f6e200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680714d0 con 0x7f9f48003970 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.221+0000 7f9f5e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f6815a140 0x7f9f6815a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f6815a140 0x7f9f6815a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34174/0 (socket says 172.21.15.141:34174) 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 -- 172.21.15.141:0/1527057494 learned_addr learned my addr 172.21.15.141:0/1527057494 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 -- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f48003970 msgr2=0x7f9f6813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f48003970 0x7f9f6813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 -- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f68070ae0 msgr2=0x7f9f68070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f68070ae0 0x7f9f68070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 -- 172.21.15.141:0/1527057494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f68071710 con 0x7f9f6815a140 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5e800640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f6815a140 0x7f9f6815a5c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000c3c0 tx=0x7f9f5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5fe00640 1 -- 172.21.15.141:0/1527057494 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50011790 con 0x7f9f6815a140 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5fe00640 1 -- 172.21.15.141:0/1527057494 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f500118f0 con 0x7f9f6815a140 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f680719f0 con 0x7f9f6815a140 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.222+0000 7f9f5fe00640 1 -- 172.21.15.141:0/1527057494 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50011ce0 con 0x7f9f6815a140 2024-01-16T15:09:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.223+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f68071fb0 con 0x7f9f6815a140 2024-01-16T15:09:25.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.224+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f680675d0 con 0x7f9f6815a140 2024-01-16T15:09:25.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.224+0000 7f9f5fe00640 1 -- 172.21.15.141:0/1527057494 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9f5001c460 con 0x7f9f6815a140 2024-01-16T15:09:25.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.225+0000 7f9f5fe00640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f4006bf20 0x7f9f4006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.225+0000 7f9f5fe00640 1 -- 172.21.15.141:0/1527057494 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9f500884a0 con 0x7f9f6815a140 2024-01-16T15:09:25.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.226+0000 7f9f67400640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f4006bf20 0x7f9f4006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.227+0000 7f9f67400640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f4006bf20 0x7f9f4006e3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9f58034000 tx=0x7f9f58039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.229+0000 7f9f5fe00640 1 -- 172.21.15.141:0/1527057494 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f50054a50 con 0x7f9f6815a140 2024-01-16T15:09:25.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.455+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596314", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f6806cff0 con 0x7f9f4006bf20 2024-01-16T15:09:25.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.477+0000 7f9f5fe00640 1 -- 172.21.15.141:0/1527057494 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f9f58001430 con 0x7f9f4006bf20 2024-01-16T15:09:25.454 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:25.454 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:25.454 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:25.454 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:25.455 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:25.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f4006bf20 msgr2=0x7f9f4006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f4006bf20 0x7f9f4006e3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9f58034000 tx=0x7f9f58039040 comp rx=0 tx=0).stop 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f6815a140 msgr2=0x7f9f6815a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f6815a140 0x7f9f6815a5c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000c3c0 tx=0x7f9f5000c880 comp rx=0 tx=0).stop 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 shutdown_connections 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f4006bf20 0x7f9f4006e3e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f68070ae0 0x7f9f68070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f6815a140 0x7f9f6815a5c0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 --2- 172.21.15.141:0/1527057494 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f48003970 0x7f9f6813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 >> 172.21.15.141:0/1527057494 conn(0x7f9f6806a800 msgr2=0x7f9f6815cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 shutdown_connections 2024-01-16T15:09:25.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.480+0000 7f9f6e200640 1 -- 172.21.15.141:0/1527057494 wait complete. 2024-01-16T15:09:25.476 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:25.476 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596315 2024-01-16T15:09:25.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.712+0000 7f7354000640 1 Processor -- start 2024-01-16T15:09:25.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.712+0000 7f7354000640 1 -- start start 2024-01-16T15:09:25.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7354000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f734c13df80 0x7f734c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7354000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f734c0cf9a0 0x7f734c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7354000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c06c470 con 0x7f734c13e360 2024-01-16T15:09:25.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7354000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734c06c750 con 0x7f734c13df80 2024-01-16T15:09:25.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7354000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734c06ca30 con 0x7f734c0cf9a0 2024-01-16T15:09:25.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7350c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f734c13e360 0x7f734c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55562/0 (socket says 172.21.15.141:55562) 2024-01-16T15:09:25.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7351600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f734c13df80 0x7f734c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.714+0000 7f7350c00640 1 -- 172.21.15.141:0/3684517524 learned_addr learned my addr 172.21.15.141:0/3684517524 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:25.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.715+0000 7f7352000640 1 --2- 172.21.15.141:0/3684517524 >> v2:172.21.15.188:3300/0 conn(0x7f734c0cf9a0 0x7f734c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.715+0000 7f7351600640 1 -- 172.21.15.141:0/3684517524 >> v2:172.21.15.188:3300/0 conn(0x7f734c0cf9a0 msgr2=0x7f734c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.715+0000 7f7351600640 1 --2- 172.21.15.141:0/3684517524 >> v2:172.21.15.188:3300/0 conn(0x7f734c0cf9a0 0x7f734c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.715+0000 7f7351600640 1 -- 172.21.15.141:0/3684517524 >> v1:172.21.15.141:6789/0 conn(0x7f734c13e360 legacy=0x7f734c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.715+0000 7f734be00640 1 -- 172.21.15.141:0/3684517524 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207611922 0 0) 0x7f734c06c470 con 0x7f734c13e360 2024-01-16T15:09:25.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.715+0000 7f7351600640 1 -- 172.21.15.141:0/3684517524 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c06cd10 con 0x7f734c13df80 2024-01-16T15:09:25.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.716+0000 7f7351600640 1 --2- 172.21.15.141:0/3684517524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13df80 0x7f734c15bee0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f734c0ccb10 tx=0x7f733c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa38b69359d590b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.716+0000 7f734be00640 1 -- 172.21.15.141:0/3684517524 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c02cc80 con 0x7f734c13df80 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.716+0000 7f734be00640 1 -- 172.21.15.141:0/3684517524 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c02cde0 con 0x7f734c13df80 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.716+0000 7f734be00640 1 -- 172.21.15.141:0/3684517524 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c034660 con 0x7f734c13df80 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 -- 172.21.15.141:0/3684517524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13df80 msgr2=0x7f734c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 --2- 172.21.15.141:0/3684517524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13df80 0x7f734c15bee0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f734c0ccb10 tx=0x7f733c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 -- 172.21.15.141:0/3684517524 shutdown_connections 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 --2- 172.21.15.141:0/3684517524 >> v2:172.21.15.188:3300/0 conn(0x7f734c0cf9a0 0x7f734c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 --2- 172.21.15.141:0/3684517524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13df80 0x7f734c15bee0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 -- 172.21.15.141:0/3684517524 >> 172.21.15.141:0/3684517524 conn(0x7f734c06a800 msgr2=0x7f734c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 -- 172.21.15.141:0/3684517524 shutdown_connections 2024-01-16T15:09:25.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.717+0000 7f7354000640 1 -- 172.21.15.141:0/3684517524 wait complete. 2024-01-16T15:09:25.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.719+0000 7f7354000640 1 Processor -- start 2024-01-16T15:09:25.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.719+0000 7f7354000640 1 -- start start 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7354000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f734c0cf9a0 0x7f734c0b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7354000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f734c13df80 0x7f734c153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7354000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13e360 0x7f734c0adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7354000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0b3b80 con 0x7f734c13e360 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7354000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0b2cf0 con 0x7f734c13df80 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7354000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c13ced0 con 0x7f734c0cf9a0 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7352000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13e360 0x7f734c0adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7352000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13e360 0x7f734c0adb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34188/0 (socket says 172.21.15.141:34188) 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7352000640 1 -- 172.21.15.141:0/2741889003 learned_addr learned my addr 172.21.15.141:0/2741889003 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.720+0000 7f7350c00640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f734c13df80 0x7f734c153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7351600640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f734c0cf9a0 0x7f734c0b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7352000640 1 -- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f734c0cf9a0 msgr2=0x7f734c0b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7352000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f734c0cf9a0 0x7f734c0b3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7352000640 1 -- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f734c13df80 msgr2=0x7f734c153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7352000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f734c13df80 0x7f734c153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7352000640 1 -- 172.21.15.141:0/2741889003 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c009a70 con 0x7f734c13e360 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7350c00640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f734c13df80 0x7f734c153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7352000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13e360 0x7f734c0adb40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f734400d5a0 tx=0x7f734400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7349600640 1 -- 172.21.15.141:0/2741889003 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73440125f0 con 0x7f734c13e360 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7349600640 1 -- 172.21.15.141:0/2741889003 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7344012770 con 0x7f734c13e360 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7349600640 1 -- 172.21.15.141:0/2741889003 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344012b20 con 0x7f734c13e360 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c0b2f00 con 0x7f734c13e360 2024-01-16T15:09:25.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.721+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c0b34c0 con 0x7f734c13e360 2024-01-16T15:09:25.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.723+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734c0b3cc0 con 0x7f734c13e360 2024-01-16T15:09:25.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.723+0000 7f7349600640 1 -- 172.21.15.141:0/2741889003 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f7344012c80 con 0x7f734c13e360 2024-01-16T15:09:25.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.724+0000 7f7349600640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f732406bf20 0x7f732406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.724+0000 7f7349600640 1 -- 172.21.15.141:0/2741889003 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7344088ab0 con 0x7f734c13e360 2024-01-16T15:09:25.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.725+0000 7f7351600640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f732406bf20 0x7f732406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.726+0000 7f7351600640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f732406bf20 0x7f732406e3e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f733c036000 tx=0x7f733c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.729+0000 7f7349600640 1 -- 172.21.15.141:0/2741889003 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f734c0b3cc0 con 0x7f734c13e360 2024-01-16T15:09:25.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.955+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596315", "target": ["mon-mgr", ""]}) v1 -- 0x7f734c15f9c0 con 0x7f732406bf20 2024-01-16T15:09:25.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.977+0000 7f7349600640 1 -- 172.21.15.141:0/2741889003 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f733c0344c0 con 0x7f732406bf20 2024-01-16T15:09:25.954 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:25.954 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:25.955 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:25.956 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:25.956 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:25.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.979+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f732406bf20 msgr2=0x7f732406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.979+0000 7f7354000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f732406bf20 0x7f732406e3e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f733c036000 tx=0x7f733c030000 comp rx=0 tx=0).stop 2024-01-16T15:09:25.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.979+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13e360 msgr2=0x7f734c0adb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.979+0000 7f7354000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13e360 0x7f734c0adb40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f734400d5a0 tx=0x7f734400da60 comp rx=0 tx=0).stop 2024-01-16T15:09:25.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 shutdown_connections 2024-01-16T15:09:25.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f732406bf20 0x7f732406e3e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f734c13e360 0x7f734c0adb40 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f734c13df80 0x7f734c153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 --2- 172.21.15.141:0/2741889003 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f734c0cf9a0 0x7f734c0b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 >> 172.21.15.141:0/2741889003 conn(0x7f734c06a800 msgr2=0x7f734c0afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 shutdown_connections 2024-01-16T15:09:25.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:25.980+0000 7f7354000640 1 -- 172.21.15.141:0/2741889003 wait complete. 2024-01-16T15:09:25.974 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:25.974 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596316 2024-01-16T15:09:26.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.209+0000 7f6053200640 1 Processor -- start 2024-01-16T15:09:26.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.209+0000 7f6053200640 1 -- start start 2024-01-16T15:09:26.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.210+0000 7f6053200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f604c0adf20 0x7f604c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.210+0000 7f6053200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 0x7f604c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6053200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c15f4e0 con 0x7f604c0cc980 2024-01-16T15:09:26.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6053200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604c15f970 con 0x7f604c0adf20 2024-01-16T15:09:26.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6053200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0cce10 con 0x7f604c15bf10 2024-01-16T15:09:26.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f604c0adf20 0x7f604c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f604c0adf20 0x7f604c070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34190/0 (socket says 172.21.15.141:34190) 2024-01-16T15:09:26.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051e00640 1 -- 172.21.15.141:0/2889260236 learned_addr learned my addr 172.21.15.141:0/2889260236 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:26.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051400640 1 --2- 172.21.15.141:0/2889260236 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 0x7f604c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051e00640 1 -- 172.21.15.141:0/2889260236 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 msgr2=0x7f604c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:26.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051e00640 1 --2- 172.21.15.141:0/2889260236 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 0x7f604c0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051e00640 1 -- 172.21.15.141:0/2889260236 >> v1:172.21.15.188:6789/0 conn(0x7f604c0cc980 legacy=0x7f604c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:26.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.211+0000 7f6051e00640 1 -- 172.21.15.141:0/2889260236 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c071130 con 0x7f604c0adf20 2024-01-16T15:09:26.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.213+0000 7f6051e00640 1 --2- 172.21.15.141:0/2889260236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c070a30 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717a8431ff118919 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.213+0000 7f6050a00640 1 -- 172.21.15.141:0/2889260236 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c0317c0 con 0x7f604c0adf20 2024-01-16T15:09:26.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.213+0000 7f6050a00640 1 -- 172.21.15.141:0/2889260236 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c031920 con 0x7f604c0adf20 2024-01-16T15:09:26.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.213+0000 7f6050a00640 1 -- 172.21.15.141:0/2889260236 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c031cb0 con 0x7f604c0adf20 2024-01-16T15:09:26.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.213+0000 7f6053200640 1 -- 172.21.15.141:0/2889260236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 msgr2=0x7f604c070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.214+0000 7f6053200640 1 --2- 172.21.15.141:0/2889260236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c070a30 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c02b860 comp rx=0 tx=0).stop 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.214+0000 7f6053200640 1 -- 172.21.15.141:0/2889260236 shutdown_connections 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.214+0000 7f6053200640 1 --2- 172.21.15.141:0/2889260236 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 0x7f604c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.214+0000 7f6053200640 1 --2- 172.21.15.141:0/2889260236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c070a30 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.214+0000 7f6053200640 1 -- 172.21.15.141:0/2889260236 >> 172.21.15.141:0/2889260236 conn(0x7f604c06a7d0 msgr2=0x7f604c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.214+0000 7f6053200640 1 -- 172.21.15.141:0/2889260236 shutdown_connections 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.214+0000 7f6053200640 1 -- 172.21.15.141:0/2889260236 wait complete. 2024-01-16T15:09:26.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.215+0000 7f6053200640 1 Processor -- start 2024-01-16T15:09:26.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.216+0000 7f6053200640 1 -- start start 2024-01-16T15:09:26.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.216+0000 7f6053200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c138440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.216+0000 7f6053200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f604c0cc980 0x7f604c1430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.216+0000 7f6053200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 0x7f604c143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.216+0000 7f6053200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c138980 con 0x7f604c0adf20 2024-01-16T15:09:26.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.216+0000 7f6053200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c4af0 con 0x7f604c0cc980 2024-01-16T15:09:26.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.216+0000 7f6053200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c4dd0 con 0x7f604c15bf10 2024-01-16T15:09:26.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c138440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c138440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34192/0 (socket says 172.21.15.141:34192) 2024-01-16T15:09:26.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 -- 172.21.15.141:0/2135556110 learned_addr learned my addr 172.21.15.141:0/2135556110 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 -- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 msgr2=0x7f604c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 0x7f604c143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 -- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f604c0cc980 msgr2=0x7f604c1430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051400640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f604c0cc980 0x7f604c1430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f604c0cc980 0x7f604c1430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 -- 172.21.15.141:0/2135556110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0094d0 con 0x7f604c0adf20 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.217+0000 7f6051e00640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c138440 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.218+0000 7f604a000640 1 -- 172.21.15.141:0/2135556110 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c032860 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.218+0000 7f604a000640 1 -- 172.21.15.141:0/2135556110 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c0329c0 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.218+0000 7f604a000640 1 -- 172.21.15.141:0/2135556110 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c032d70 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.218+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c0c50b0 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.218+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c0c5670 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.219+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c067450 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.220+0000 7f604a000640 1 -- 172.21.15.141:0/2135556110 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f603c04d3f0 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.221+0000 7f604a000640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f602406bf20 0x7f602406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.221+0000 7f604a000640 1 -- 172.21.15.141:0/2135556110 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f603c0aa900 con 0x7f604c0adf20 2024-01-16T15:09:26.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.221+0000 7f6051400640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f602406bf20 0x7f602406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.222+0000 7f6051400640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f602406bf20 0x7f602406e3e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f604c15bee0 tx=0x7f6034008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.225+0000 7f604a000640 1 -- 172.21.15.141:0/2135556110 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f603c076df0 con 0x7f604c0adf20 2024-01-16T15:09:26.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.458+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596316", "target": ["mon-mgr", ""]}) v1 -- 0x7f604c15ee90 con 0x7f602406bf20 2024-01-16T15:09:26.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.480+0000 7f604a000640 1 -- 172.21.15.141:0/2135556110 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f6034009520 con 0x7f602406bf20 2024-01-16T15:09:26.457 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:26.458 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:26.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f602406bf20 msgr2=0x7f602406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f602406bf20 0x7f602406e3e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f604c15bee0 tx=0x7f6034008040 comp rx=0 tx=0).stop 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 msgr2=0x7f604c138440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c138440 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 shutdown_connections 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f602406bf20 0x7f602406e3e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f604c15bf10 0x7f604c143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f604c0cc980 0x7f604c1430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 --2- 172.21.15.141:0/2135556110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f604c0adf20 0x7f604c138440 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 >> 172.21.15.141:0/2135556110 conn(0x7f604c06a7d0 msgr2=0x7f604c143e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 shutdown_connections 2024-01-16T15:09:26.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.483+0000 7f6053200640 1 -- 172.21.15.141:0/2135556110 wait complete. 2024-01-16T15:09:26.479 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:26.480 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596317 2024-01-16T15:09:26.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.707+0000 7f13b2000640 1 Processor -- start 2024-01-16T15:09:26.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.707+0000 7f13b2000640 1 -- start start 2024-01-16T15:09:26.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b2000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06c730 0x7f13ac0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b2000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f13ac070160 0x7f13ac071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b2000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f13ac06cb10 0x7f13ac06cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b2000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac0708d0 con 0x7f13ac06cb10 2024-01-16T15:09:26.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b2000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac070bb0 con 0x7f13ac070160 2024-01-16T15:09:26.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b2000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac070e90 con 0x7f13ac06c730 2024-01-16T15:09:26.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b1600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f13ac06cb10 0x7f13ac06cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b1600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f13ac06cb10 0x7f13ac06cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34194/0 (socket says 172.21.15.141:34194) 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b1600640 1 -- 172.21.15.141:0/3901726385 learned_addr learned my addr 172.21.15.141:0/3901726385 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13abe00640 1 --2- 172.21.15.141:0/3901726385 >> v2:172.21.15.188:3300/0 conn(0x7f13ac070160 0x7f13ac071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b0c00640 1 --2- 172.21.15.141:0/3901726385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06c730 0x7f13ac0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.709+0000 7f13b1600640 1 -- 172.21.15.141:0/3901726385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06c730 msgr2=0x7f13ac0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.710+0000 7f13b1600640 1 --2- 172.21.15.141:0/3901726385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06c730 0x7f13ac0b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.710+0000 7f13b1600640 1 -- 172.21.15.141:0/3901726385 >> v2:172.21.15.188:3300/0 conn(0x7f13ac070160 msgr2=0x7f13ac071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.710+0000 7f13b1600640 1 --2- 172.21.15.141:0/3901726385 >> v2:172.21.15.188:3300/0 conn(0x7f13ac070160 0x7f13ac071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.710+0000 7f13b1600640 1 -- 172.21.15.141:0/3901726385 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac071170 con 0x7f13ac06cb10 2024-01-16T15:09:26.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.710+0000 7f13b0c00640 1 --2- 172.21.15.141:0/3901726385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06c730 0x7f13ac0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:26.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.710+0000 7f13abe00640 1 --2- 172.21.15.141:0/3901726385 >> v2:172.21.15.188:3300/0 conn(0x7f13ac070160 0x7f13ac071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:26.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.711+0000 7f13b1600640 1 --2- 172.21.15.141:0/3901726385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06cb10 0x7f13ac06cf80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f13ac06e350 tx=0x7f13a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18f671bb53aaa5d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.711+0000 7f13ab400640 1 -- 172.21.15.141:0/3901726385 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a402ca80 con 0x7f13ac06cb10 2024-01-16T15:09:26.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.711+0000 7f13ab400640 1 -- 172.21.15.141:0/3901726385 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a402cbe0 con 0x7f13ac06cb10 2024-01-16T15:09:26.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.711+0000 7f13ab400640 1 -- 172.21.15.141:0/3901726385 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a40343f0 con 0x7f13ac06cb10 2024-01-16T15:09:26.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 -- 172.21.15.141:0/3901726385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06cb10 msgr2=0x7f13ac06cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 --2- 172.21.15.141:0/3901726385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06cb10 0x7f13ac06cf80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f13ac06e350 tx=0x7f13a402bee0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 -- 172.21.15.141:0/3901726385 shutdown_connections 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 --2- 172.21.15.141:0/3901726385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06cb10 0x7f13ac06cf80 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 --2- 172.21.15.141:0/3901726385 >> v2:172.21.15.188:3300/0 conn(0x7f13ac070160 0x7f13ac071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 --2- 172.21.15.141:0/3901726385 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06c730 0x7f13ac0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 -- 172.21.15.141:0/3901726385 >> 172.21.15.141:0/3901726385 conn(0x7f13ac06a7d0 msgr2=0x7f13ac06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 -- 172.21.15.141:0/3901726385 shutdown_connections 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.712+0000 7f13b2000640 1 -- 172.21.15.141:0/3901726385 wait complete. 2024-01-16T15:09:26.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.713+0000 7f13b2000640 1 Processor -- start 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.714+0000 7f13b2000640 1 -- start start 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.714+0000 7f13b2000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06c730 0x7f13ac147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.714+0000 7f13b2000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06cb10 0x7f13ac1f9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b2000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13ac070160 0x7f13ac140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b2000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac1483d0 con 0x7f13ac06c730 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b2000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac1f9e70 con 0x7f13ac070160 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b2000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac146df0 con 0x7f13ac06cb10 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06c730 0x7f13ac147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06c730 0x7f13ac147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34208/0 (socket says 172.21.15.141:34208) 2024-01-16T15:09:26.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 -- 172.21.15.141:0/1596818626 learned_addr learned my addr 172.21.15.141:0/1596818626 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 -- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06cb10 msgr2=0x7f13ac1f9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06cb10 0x7f13ac1f9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 -- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13ac070160 msgr2=0x7f13ac140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13ac070160 0x7f13ac140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 -- 172.21.15.141:0/1596818626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a4009b30 con 0x7f13ac06c730 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.715+0000 7f13b0c00640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06c730 0x7f13ac147e90 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c7f0 tx=0x7f139c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.716+0000 7f13a8c00640 1 -- 172.21.15.141:0/1596818626 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c0077a0 con 0x7f13ac06c730 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.716+0000 7f13a8c00640 1 -- 172.21.15.141:0/1596818626 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c007900 con 0x7f13ac06c730 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.716+0000 7f13a8c00640 1 -- 172.21.15.141:0/1596818626 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c007cb0 con 0x7f13ac06c730 2024-01-16T15:09:26.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.716+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac1470d0 con 0x7f13ac06c730 2024-01-16T15:09:26.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.716+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac147690 con 0x7f13ac06c730 2024-01-16T15:09:26.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.717+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac067450 con 0x7f13ac06c730 2024-01-16T15:09:26.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.718+0000 7f13a8c00640 1 -- 172.21.15.141:0/1596818626 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f139c0113f0 con 0x7f13ac06c730 2024-01-16T15:09:26.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.719+0000 7f13a8c00640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f138406bf20 0x7f138406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.719+0000 7f13a8c00640 1 -- 172.21.15.141:0/1596818626 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f139c088b00 con 0x7f13ac06c730 2024-01-16T15:09:26.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.720+0000 7f13abe00640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f138406bf20 0x7f138406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.721+0000 7f13abe00640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f138406bf20 0x7f138406e3e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1394006fd0 tx=0x7f1394008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.724+0000 7f13a8c00640 1 -- 172.21.15.141:0/1596818626 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f139c0550c0 con 0x7f13ac06c730 2024-01-16T15:09:26.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.955+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596317", "target": ["mon-mgr", ""]}) v1 -- 0x7f13ac0705c0 con 0x7f138406bf20 2024-01-16T15:09:26.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.978+0000 7f13a8c00640 1 -- 172.21.15.141:0/1596818626 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f139400ae40 con 0x7f138406bf20 2024-01-16T15:09:26.955 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:26.955 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:26.955 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:26.955 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:26.955 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:26.955 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:26.956 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:26.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f138406bf20 msgr2=0x7f138406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f138406bf20 0x7f138406e3e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1394006fd0 tx=0x7f1394008040 comp rx=0 tx=0).stop 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06c730 msgr2=0x7f13ac147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06c730 0x7f13ac147e90 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c7f0 tx=0x7f139c00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 shutdown_connections 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f138406bf20 0x7f138406e3e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13ac070160 0x7f13ac140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13ac06cb10 0x7f13ac1f9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 --2- 172.21.15.141:0/1596818626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13ac06c730 0x7f13ac147e90 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 >> 172.21.15.141:0/1596818626 conn(0x7f13ac06a7d0 msgr2=0x7f13ac0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 shutdown_connections 2024-01-16T15:09:26.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:26.981+0000 7f13b2000640 1 -- 172.21.15.141:0/1596818626 wait complete. 2024-01-16T15:09:26.978 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:26.978 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596318 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.219+0000 7f686f200640 1 Processor -- start 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.219+0000 7f686f200640 1 -- start start 2024-01-16T15:09:27.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.221+0000 7f686f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 0x7f6870071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.221+0000 7f686f200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687006c730 con 0x7f68700ae170 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.221+0000 7f686f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68700ae5f0 con 0x7f687015d770 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.221+0000 7f686f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6870143bf0 con 0x7f6870143810 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.221+0000 7f686d400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f68700ae170 0x7f687015e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55566/0 (socket says 172.21.15.141:55566) 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.221+0000 7f686d400640 1 -- 172.21.15.141:0/1537319646 learned_addr learned my addr 172.21.15.141:0/1537319646 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.221+0000 7f686de00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 0x7f6870071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172281790 0 0) 0x7f687006c730 con 0x7f68700ae170 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6850003520 con 0x7f68700ae170 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3750220573 0 0) 0x7f6850003520 con 0x7f68700ae170 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 msgr2=0x7f6870071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 0x7f6870071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 >> v1:172.21.15.188:6789/0 conn(0x7f687015d770 legacy=0x7f687015dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700724d0 con 0x7f68700ae170 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6858001670 con 0x7f68700ae170 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 >> v1:172.21.15.141:6789/0 conn(0x7f68700ae170 legacy=0x7f687015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68500039a0 0x7f6850003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 0x7f68500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6850004d00 0x7f6850005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6870143bf0 con 0x7f68500039a0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685000bb90 con 0x7f6850004d00 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685000bd50 con 0x7f6870143810 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.222+0000 7f686de00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68500039a0 0x7f6850003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.223+0000 7f686d400640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 0x7f68500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.223+0000 7f686de00640 1 -- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 msgr2=0x7f68500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.223+0000 7f686de00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6870143810 0x7f68500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.223+0000 7f686de00640 1 -- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6850004d00 msgr2=0x7f6850005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.223+0000 7f686de00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6850004d00 0x7f6850005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.223+0000 7f686de00640 1 -- 172.21.15.141:0/1537319646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6850003520 con 0x7f68500039a0 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.224+0000 7f686de00640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68500039a0 0x7f6850003dc0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6860009fc0 tx=0x7f68500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598f3cee2f9eff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.224+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686002cc20 con 0x7f68500039a0 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.224+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686000f040 con 0x7f68500039a0 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.224+0000 7f686ca00640 1 -- 172.21.15.141:0/1537319646 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860034440 con 0x7f68500039a0 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.225+0000 7f686f200640 1 -- 172.21.15.141:0/1537319646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68500039a0 msgr2=0x7f6850003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.225+0000 7f686f200640 1 --2- 172.21.15.141:0/1537319646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68500039a0 0x7f6850003dc0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6860009fc0 tx=0x7f68500057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.225+0000 7f686e800640 1 -- 172.21.15.141:0/1537319646 reap_dead start 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.225+0000 7f686f200640 1 -- 172.21.15.141:0/1537319646 shutdown_connections 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.225+0000 7f686f200640 1 -- 172.21.15.141:0/1537319646 >> 172.21.15.141:0/1537319646 conn(0x7f687006a7d0 msgr2=0x7f687006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.225+0000 7f686f200640 1 -- 172.21.15.141:0/1537319646 shutdown_connections 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.225+0000 7f686f200640 1 -- 172.21.15.141:0/1537319646 wait complete. 2024-01-16T15:09:27.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.227+0000 7f686f200640 1 Processor -- start 2024-01-16T15:09:27.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.227+0000 7f686f200640 1 -- start start 2024-01-16T15:09:27.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68500039a0 0x7f68700cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f687015c880 0x7f68700c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68700c5160 0x7f68700c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700cb5e0 con 0x7f687015c880 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700cb8c0 con 0x7f68700c5160 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700cbba0 con 0x7f68500039a0 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f687015c880 0x7f68700c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f687015c880 0x7f68700c4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34222/0 (socket says 172.21.15.141:34222) 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.228+0000 7f686d400640 1 -- 172.21.15.141:0/3504986497 learned_addr learned my addr 172.21.15.141:0/3504986497 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686de00640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68500039a0 0x7f68700cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686e800640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68700c5160 0x7f68700c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686d400640 1 -- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68500039a0 msgr2=0x7f68700cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686d400640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68500039a0 0x7f68700cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686d400640 1 -- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68700c5160 msgr2=0x7f68700c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686d400640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68700c5160 0x7f68700c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686d400640 1 -- 172.21.15.141:0/3504986497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700cbe80 con 0x7f687015c880 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686e800640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68700c5160 0x7f68700c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686d400640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f687015c880 0x7f68700c4c20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f685800e780 tx=0x7f685800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f6866000640 1 -- 172.21.15.141:0/3504986497 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685800ee30 con 0x7f687015c880 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f6866000640 1 -- 172.21.15.141:0/3504986497 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685800c870 con 0x7f687015c880 2024-01-16T15:09:27.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f6866000640 1 -- 172.21.15.141:0/3504986497 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685800cc20 con 0x7f687015c880 2024-01-16T15:09:27.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.229+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68700cc160 con 0x7f687015c880 2024-01-16T15:09:27.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.230+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68700ccea0 con 0x7f687015c880 2024-01-16T15:09:27.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.231+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68700cc680 con 0x7f687015c880 2024-01-16T15:09:27.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.232+0000 7f6866000640 1 -- 172.21.15.141:0/3504986497 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f685801d460 con 0x7f687015c880 2024-01-16T15:09:27.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.232+0000 7f6866000640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f684806bf20 0x7f684806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.233+0000 7f6866000640 1 -- 172.21.15.141:0/3504986497 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f68580885f0 con 0x7f687015c880 2024-01-16T15:09:27.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.233+0000 7f686de00640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f684806bf20 0x7f684806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.234+0000 7f686de00640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f684806bf20 0x7f684806e3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6860036000 tx=0x7f686002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.237+0000 7f6866000640 1 -- 172.21.15.141:0/3504986497 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68700cc680 con 0x7f687015c880 2024-01-16T15:09:27.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.465+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596318", "target": ["mon-mgr", ""]}) v1 -- 0x7f68700cbd10 con 0x7f684806bf20 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.488+0000 7f6866000640 1 -- 172.21.15.141:0/3504986497 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f68700cbd10 con 0x7f684806bf20 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:27.465 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:27.466 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:27.466 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:27.466 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:27.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f684806bf20 msgr2=0x7f684806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f684806bf20 0x7f684806e3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6860036000 tx=0x7f686002e000 comp rx=0 tx=0).stop 2024-01-16T15:09:27.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f687015c880 msgr2=0x7f68700c4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f687015c880 0x7f68700c4c20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f685800e780 tx=0x7f685800ec40 comp rx=0 tx=0).stop 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 shutdown_connections 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f684806bf20 0x7f684806e3e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68700c5160 0x7f68700c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f687015c880 0x7f68700c4c20 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 --2- 172.21.15.141:0/3504986497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68500039a0 0x7f68700cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 >> 172.21.15.141:0/3504986497 conn(0x7f687006a7d0 msgr2=0x7f68701602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 shutdown_connections 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.490+0000 7f686f200640 1 -- 172.21.15.141:0/3504986497 wait complete. 2024-01-16T15:09:27.487 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:27.487 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596319 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.720+0000 7f7e7ca00640 1 Processor -- start 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.720+0000 7f7e7ca00640 1 -- start start 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.722+0000 7f7e7ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 0x7f7e78070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.722+0000 7f7e7ca00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7e7815bf10 0x7f7e78143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.722+0000 7f7e7ca00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7815f4e0 con 0x7f7e780adf20 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.722+0000 7f7e7ca00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7815f970 con 0x7f7e7815bf10 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.722+0000 7f7e7ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815c3c0 con 0x7f7e780ccb50 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.722+0000 7f7e76800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7e780adf20 0x7f7e780cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55570/0 (socket says 172.21.15.141:55570) 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.722+0000 7f7e76800640 1 -- 172.21.15.141:0/2625215438 learned_addr learned my addr 172.21.15.141:0/2625215438 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:27.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75e00640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 0x7f7e78070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e77200640 1 --2- 172.21.15.141:0/2625215438 >> v2:172.21.15.188:3300/0 conn(0x7f7e7815bf10 0x7f7e78143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711863986 0 0) 0x7f7e7815f4e0 con 0x7f7e780adf20 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e58003520 con 0x7f7e780adf20 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 472146808 0 0) 0x7f7e58003520 con 0x7f7e780adf20 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 msgr2=0x7f7e78070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 0x7f7e78070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 >> v2:172.21.15.188:3300/0 conn(0x7f7e7815bf10 msgr2=0x7f7e78143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 --2- 172.21.15.141:0/2625215438 >> v2:172.21.15.188:3300/0 conn(0x7f7e7815bf10 0x7f7e78143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e78071130 con 0x7f7e780adf20 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7e68001670 con 0x7f7e780adf20 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 >> v1:172.21.15.141:6789/0 conn(0x7f7e780adf20 legacy=0x7f7e780cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.723+0000 7f7e75400640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e75400640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580047c0 0x7f7e58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e75400640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 0x7f7e58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815f970 con 0x7f7e58003970 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815c3c0 con 0x7f7e580047c0 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5800b980 con 0x7f7e780ccb50 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e76800640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e75e00640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580047c0 0x7f7e58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e76800640 1 -- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 msgr2=0x7f7e58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e77200640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 0x7f7e58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e76800640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e780ccb50 0x7f7e58005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e76800640 1 -- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580047c0 msgr2=0x7f7e58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e76800640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580047c0 0x7f7e58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e76800640 1 -- 172.21.15.141:0/2625215438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58003520 con 0x7f7e58003970 2024-01-16T15:09:27.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.724+0000 7f7e75e00640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e580047c0 0x7f7e58004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:27.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.725+0000 7f7e76800640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800c270 tx=0x7f7e580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1083c1b8323b529 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68005ad0 con 0x7f7e58003970 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68005c50 con 0x7f7e58003970 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e75400640 1 -- 172.21.15.141:0/2625215438 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6802c8b0 con 0x7f7e58003970 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e7ca00640 1 -- 172.21.15.141:0/2625215438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 msgr2=0x7f7e58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e7ca00640 1 --2- 172.21.15.141:0/2625215438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800c270 tx=0x7f7e580057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e77200640 1 -- 172.21.15.141:0/2625215438 reap_dead start 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e7ca00640 1 -- 172.21.15.141:0/2625215438 shutdown_connections 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e7ca00640 1 -- 172.21.15.141:0/2625215438 >> 172.21.15.141:0/2625215438 conn(0x7f7e7806a7d0 msgr2=0x7f7e7806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e7ca00640 1 -- 172.21.15.141:0/2625215438 shutdown_connections 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.726+0000 7f7e7ca00640 1 -- 172.21.15.141:0/2625215438 wait complete. 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.728+0000 7f7e7ca00640 1 Processor -- start 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.728+0000 7f7e7ca00640 1 -- start start 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.729+0000 7f7e7ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e7815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.729+0000 7f7e7ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e7813fde0 0x7f7e78140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.729+0000 7f7e7ca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e78071b30 0x7f7e78071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.729+0000 7f7e7ca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e780724f0 con 0x7f7e58003970 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.729+0000 7f7e7ca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815aae0 con 0x7f7e78071b30 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.729+0000 7f7e7ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78140910 con 0x7f7e7813fde0 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.729+0000 7f7e76800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e7815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e7815a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34240/0 (socket says 172.21.15.141:34240) 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 -- 172.21.15.141:0/655652646 learned_addr learned my addr 172.21.15.141:0/655652646 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e77200640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e78071b30 0x7f7e78071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 -- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e7813fde0 msgr2=0x7f7e78140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e75e00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e7813fde0 0x7f7e78140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e7813fde0 0x7f7e78140260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 -- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e78071b30 msgr2=0x7f7e78071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e78071b30 0x7f7e78071fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 -- 172.21.15.141:0/655652646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e781f4e00 con 0x7f7e58003970 2024-01-16T15:09:27.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e76800640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e7815a5a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f7e6802c450 tx=0x7f7e7815f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e6ea00640 1 -- 172.21.15.141:0/655652646 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6803c040 con 0x7f7e58003970 2024-01-16T15:09:27.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.730+0000 7f7e6ea00640 1 -- 172.21.15.141:0/655652646 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6802ccc0 con 0x7f7e58003970 2024-01-16T15:09:27.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.731+0000 7f7e6ea00640 1 -- 172.21.15.141:0/655652646 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68007290 con 0x7f7e58003970 2024-01-16T15:09:27.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.731+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e781f4040 con 0x7f7e58003970 2024-01-16T15:09:27.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.731+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e781f4600 con 0x7f7e58003970 2024-01-16T15:09:27.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.732+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e78067450 con 0x7f7e58003970 2024-01-16T15:09:27.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.733+0000 7f7e6ea00640 1 -- 172.21.15.141:0/655652646 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f7e68032070 con 0x7f7e58003970 2024-01-16T15:09:27.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.733+0000 7f7e6ea00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5006bf20 0x7f7e5006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.734+0000 7f7e6ea00640 1 -- 172.21.15.141:0/655652646 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e680a9570 con 0x7f7e58003970 2024-01-16T15:09:27.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.734+0000 7f7e75e00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5006bf20 0x7f7e5006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.735+0000 7f7e75e00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5006bf20 0x7f7e5006e3e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7e780ad8f0 tx=0x7f7e60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.738+0000 7f7e6ea00640 1 -- 172.21.15.141:0/655652646 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e68075bb0 con 0x7f7e58003970 2024-01-16T15:09:27.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.965+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596319", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e7815f170 con 0x7f7e5006bf20 2024-01-16T15:09:27.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.990+0000 7f7e6ea00640 1 -- 172.21.15.141:0/655652646 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f7e60009650 con 0x7f7e5006bf20 2024-01-16T15:09:27.967 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:27.967 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:27.967 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:27.968 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:27.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5006bf20 msgr2=0x7f7e5006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.970 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5006bf20 0x7f7e5006e3e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7e780ad8f0 tx=0x7f7e60008040 comp rx=0 tx=0).stop 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 msgr2=0x7f7e7815a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e7815a5a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f7e6802c450 tx=0x7f7e7815f450 comp rx=0 tx=0).stop 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 shutdown_connections 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7e5006bf20 0x7f7e5006e3e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e78071b30 0x7f7e78071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7e7813fde0 0x7f7e78140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 --2- 172.21.15.141:0/655652646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e58003970 0x7f7e7815a5a0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 >> 172.21.15.141:0/655652646 conn(0x7f7e7806a7d0 msgr2=0x7f7e7815ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 shutdown_connections 2024-01-16T15:09:27.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:27.993+0000 7f7e7ca00640 1 -- 172.21.15.141:0/655652646 wait complete. 2024-01-16T15:09:27.990 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:09:27.990 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone596320 2024-01-16T15:09:28.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.226+0000 7f9339c00640 1 Processor -- start 2024-01-16T15:09:28.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.226+0000 7f9339c00640 1 -- start start 2024-01-16T15:09:28.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.228+0000 7f9339c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93340cf9a0 0x7f933415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.228+0000 7f9339c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f933415e1e0 0x7f93340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.228+0000 7f9339c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340b1ee0 con 0x7f9334143ae0 2024-01-16T15:09:28.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.228+0000 7f9339c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93340b2780 con 0x7f933415e1e0 2024-01-16T15:09:28.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.228+0000 7f9339c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933415e990 con 0x7f93340cf9a0 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9332000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93340cf9a0 0x7f933415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9333400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f933415e1e0 0x7f93340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9332000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93340cf9a0 0x7f933415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:49190/0 (socket says 172.21.15.141:49190) 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9332000640 1 -- 172.21.15.141:0/1913384917 learned_addr learned my addr 172.21.15.141:0/1913384917 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9332000640 1 -- 172.21.15.141:0/1913384917 >> v2:172.21.15.188:3300/0 conn(0x7f933415e1e0 msgr2=0x7f93340702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9332000640 1 --2- 172.21.15.141:0/1913384917 >> v2:172.21.15.188:3300/0 conn(0x7f933415e1e0 0x7f93340702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9332000640 1 -- 172.21.15.141:0/1913384917 >> v1:172.21.15.188:6789/0 conn(0x7f9334143ae0 legacy=0x7f93340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.229+0000 7f9332000640 1 -- 172.21.15.141:0/1913384917 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340b2130 con 0x7f93340cf9a0 2024-01-16T15:09:28.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.231+0000 7f9332000640 1 --2- 172.21.15.141:0/1913384917 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93340cf9a0 0x7f933415dca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f931c00bb10 tx=0x7f931c02e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1c255aed408151d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.231+0000 7f9331600640 1 -- 172.21.15.141:0/1913384917 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c036440 con 0x7f93340cf9a0 2024-01-16T15:09:28.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.231+0000 7f9331600640 1 -- 172.21.15.141:0/1913384917 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c0365a0 con 0x7f93340cf9a0 2024-01-16T15:09:28.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.231+0000 7f9331600640 1 -- 172.21.15.141:0/1913384917 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c036930 con 0x7f93340cf9a0 2024-01-16T15:09:28.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 -- 172.21.15.141:0/1913384917 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93340cf9a0 msgr2=0x7f933415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 --2- 172.21.15.141:0/1913384917 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93340cf9a0 0x7f933415dca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f931c00bb10 tx=0x7f931c02e150 comp rx=0 tx=0).stop 2024-01-16T15:09:28.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 -- 172.21.15.141:0/1913384917 shutdown_connections 2024-01-16T15:09:28.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 --2- 172.21.15.141:0/1913384917 >> v2:172.21.15.188:3300/0 conn(0x7f933415e1e0 0x7f93340702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 --2- 172.21.15.141:0/1913384917 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f93340cf9a0 0x7f933415dca0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 -- 172.21.15.141:0/1913384917 >> 172.21.15.141:0/1913384917 conn(0x7f933406a800 msgr2=0x7f933406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 -- 172.21.15.141:0/1913384917 shutdown_connections 2024-01-16T15:09:28.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.232+0000 7f9339c00640 1 -- 172.21.15.141:0/1913384917 wait complete. 2024-01-16T15:09:28.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.235+0000 7f9339c00640 1 Processor -- start 2024-01-16T15:09:28.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.235+0000 7f9339c00640 1 -- start start 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9339c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93340cf9a0 0x7f933413f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9339c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9334143ae0 0x7f933415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9339c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f933415e1e0 0x7f93340aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9339c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9334140540 con 0x7f93340cf9a0 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9339c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933413f5f0 con 0x7f9334143ae0 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9339c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933413f8c0 con 0x7f933415e1e0 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9332a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93340cf9a0 0x7f933413f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9332a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93340cf9a0 0x7f933413f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34252/0 (socket says 172.21.15.141:34252) 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.236+0000 7f9332a00640 1 -- 172.21.15.141:0/4015864702 learned_addr learned my addr 172.21.15.141:0/4015864702 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:28.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332000640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9334143ae0 0x7f933415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9333400640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f933415e1e0 0x7f93340aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332a00640 1 -- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f933415e1e0 msgr2=0x7f93340aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332a00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f933415e1e0 0x7f93340aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332a00640 1 -- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9334143ae0 msgr2=0x7f933415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332a00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9334143ae0 0x7f933415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332a00640 1 -- 172.21.15.141:0/4015864702 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c00b770 con 0x7f93340cf9a0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332000640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9334143ae0 0x7f933415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9332a00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93340cf9a0 0x7f933413f0b0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f932400c870 tx=0x7f932400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f932aa00640 1 -- 172.21.15.141:0/4015864702 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324003b70 con 0x7f93340cf9a0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f932aa00640 1 -- 172.21.15.141:0/4015864702 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324003cf0 con 0x7f93340cf9a0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f932aa00640 1 -- 172.21.15.141:0/4015864702 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932400aa90 con 0x7f93340cf9a0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.237+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933413fba0 con 0x7f93340cf9a0 2024-01-16T15:09:28.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.238+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9334140160 con 0x7f93340cf9a0 2024-01-16T15:09:28.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.239+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93340675d0 con 0x7f93340cf9a0 2024-01-16T15:09:28.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.240+0000 7f932aa00640 1 -- 172.21.15.141:0/4015864702 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f932400abf0 con 0x7f93340cf9a0 2024-01-16T15:09:28.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.241+0000 7f932aa00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f930c06bf20 0x7f930c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.241+0000 7f932aa00640 1 -- 172.21.15.141:0/4015864702 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f93240884b0 con 0x7f93340cf9a0 2024-01-16T15:09:28.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.241+0000 7f9332000640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f930c06bf20 0x7f930c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.242+0000 7f9332000640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f930c06bf20 0x7f930c06e3e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f931c038000 tx=0x7f931c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.245+0000 7f932aa00640 1 -- 172.21.15.141:0/4015864702 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93240bde90 con 0x7f93340cf9a0 2024-01-16T15:09:28.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.473+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone596320", "target": ["mon-mgr", ""]}) v1 -- 0x7f933415e640 con 0x7f930c06bf20 2024-01-16T15:09:28.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.494+0000 7f932aa00640 1 -- 172.21.15.141:0/4015864702 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f931c036920 con 0x7f930c06bf20 2024-01-16T15:09:28.472 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:28.472 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:09:28.472 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:09:28.472 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:09:28.472 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:09:28.472 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:28.473 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f930c06bf20 msgr2=0x7f930c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f930c06bf20 0x7f930c06e3e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f931c038000 tx=0x7f931c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93340cf9a0 msgr2=0x7f933413f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93340cf9a0 0x7f933413f0b0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f932400c870 tx=0x7f932400cd30 comp rx=0 tx=0).stop 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 shutdown_connections 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f930c06bf20 0x7f930c06e3e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f933415e1e0 0x7f93340aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9334143ae0 0x7f933415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 --2- 172.21.15.141:0/4015864702 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f93340cf9a0 0x7f933413f0b0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 >> 172.21.15.141:0/4015864702 conn(0x7f933406a800 msgr2=0x7f933406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.497+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 shutdown_connections 2024-01-16T15:09:28.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.498+0000 7f9339c00640 1 -- 172.21.15.141:0/4015864702 wait complete. 2024-01-16T15:09:28.494 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:09:28.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.727+0000 7fde59400640 1 Processor -- start 2024-01-16T15:09:28.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.727+0000 7fde59400640 1 -- start start 2024-01-16T15:09:28.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.728+0000 7fde59400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde540ce8f0 0x7fde540ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.728+0000 7fde59400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde540b2a00 0x7fde540b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.728+0000 7fde59400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde540d0de0 con 0x7fde54160f80 2024-01-16T15:09:28.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.728+0000 7fde59400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde540d10c0 con 0x7fde540ce8f0 2024-01-16T15:09:28.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.728+0000 7fde59400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde540d13a0 con 0x7fde540b2a00 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde51e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde540ce8f0 0x7fde540ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde52800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fde54160f80 0x7fde54161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55572/0 (socket says 172.21.15.141:55572) 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde52800640 1 -- 172.21.15.141:0/1943058221 learned_addr learned my addr 172.21.15.141:0/1943058221 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde53200640 1 --2- 172.21.15.141:0/1943058221 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde540b2a00 0x7fde540b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde51e00640 1 -- 172.21.15.141:0/1943058221 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde540b2a00 msgr2=0x7fde540b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde51e00640 1 --2- 172.21.15.141:0/1943058221 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde540b2a00 0x7fde540b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde51e00640 1 -- 172.21.15.141:0/1943058221 >> v1:172.21.15.141:6789/0 conn(0x7fde54160f80 legacy=0x7fde54161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde51400640 1 -- 172.21.15.141:0/1943058221 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604128765 0 0) 0x7fde540d0de0 con 0x7fde54160f80 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde51e00640 1 -- 172.21.15.141:0/1943058221 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540d1680 con 0x7fde540ce8f0 2024-01-16T15:09:28.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.729+0000 7fde53200640 1 --2- 172.21.15.141:0/1943058221 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde540b2a00 0x7fde540b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:28.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.730+0000 7fde51e00640 1 --2- 172.21.15.141:0/1943058221 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde540ced30 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0097a0 tx=0x7fde3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b3ed00824d1ff4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde51400640 1 -- 172.21.15.141:0/1943058221 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c031830 con 0x7fde540ce8f0 2024-01-16T15:09:28.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde51400640 1 -- 172.21.15.141:0/1943058221 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c031990 con 0x7fde540ce8f0 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde51400640 1 -- 172.21.15.141:0/1943058221 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c031d20 con 0x7fde540ce8f0 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 -- 172.21.15.141:0/1943058221 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 msgr2=0x7fde540ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 --2- 172.21.15.141:0/1943058221 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde540ced30 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0097a0 tx=0x7fde3c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 -- 172.21.15.141:0/1943058221 shutdown_connections 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 --2- 172.21.15.141:0/1943058221 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde540b2a00 0x7fde540b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 --2- 172.21.15.141:0/1943058221 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde540ced30 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 -- 172.21.15.141:0/1943058221 >> 172.21.15.141:0/1943058221 conn(0x7fde5409c550 msgr2=0x7fde540d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 -- 172.21.15.141:0/1943058221 shutdown_connections 2024-01-16T15:09:28.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.731+0000 7fde59400640 1 -- 172.21.15.141:0/1943058221 wait complete. 2024-01-16T15:09:28.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.733+0000 7fde59400640 1 Processor -- start 2024-01-16T15:09:28.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.733+0000 7fde59400640 1 -- start start 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.734+0000 7fde59400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde540b2a00 0x7fde541587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.734+0000 7fde59400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde54158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.734+0000 7fde59400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde54160f80 0x7fde5415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.734+0000 7fde59400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5415df10 con 0x7fde540ce8f0 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.734+0000 7fde59400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5415e1f0 con 0x7fde540b2a00 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.734+0000 7fde59400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5415e4d0 con 0x7fde54160f80 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.734+0000 7fde51e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde54158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde54158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34270/0 (socket says 172.21.15.141:34270) 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 -- 172.21.15.141:0/873881 learned_addr learned my addr 172.21.15.141:0/873881 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde52800640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde540b2a00 0x7fde541587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde53200640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde54160f80 0x7fde5415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 -- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde54160f80 msgr2=0x7fde5415d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde54160f80 0x7fde5415d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 -- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde540b2a00 msgr2=0x7fde541587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde540b2a00 0x7fde541587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 -- 172.21.15.141:0/873881 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c009400 con 0x7fde540ce8f0 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde53200640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde54160f80 0x7fde5415d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:28.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.735+0000 7fde51e00640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde54158200 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0098c0 tx=0x7fde3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.736+0000 7fde4aa00640 1 -- 172.21.15.141:0/873881 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c03b9f0 con 0x7fde540ce8f0 2024-01-16T15:09:28.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.736+0000 7fde4aa00640 1 -- 172.21.15.141:0/873881 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c03bb70 con 0x7fde540ce8f0 2024-01-16T15:09:28.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.736+0000 7fde4aa00640 1 -- 172.21.15.141:0/873881 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c0453f0 con 0x7fde540ce8f0 2024-01-16T15:09:28.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.736+0000 7fde59400640 1 -- 172.21.15.141:0/873881 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5415e7b0 con 0x7fde540ce8f0 2024-01-16T15:09:28.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.736+0000 7fde59400640 1 -- 172.21.15.141:0/873881 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5415dc00 con 0x7fde540ce8f0 2024-01-16T15:09:28.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.737+0000 7fde59400640 1 -- 172.21.15.141:0/873881 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5415ecd0 con 0x7fde540ce8f0 2024-01-16T15:09:28.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.738+0000 7fde4aa00640 1 -- 172.21.15.141:0/873881 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fde3c04e450 con 0x7fde540ce8f0 2024-01-16T15:09:28.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.738+0000 7fde4aa00640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde2c06bf70 0x7fde2c06e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.739+0000 7fde4aa00640 1 -- 172.21.15.141:0/873881 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fde3c0aab00 con 0x7fde540ce8f0 2024-01-16T15:09:28.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.739+0000 7fde52800640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde2c06bf70 0x7fde2c06e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.740+0000 7fde52800640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde2c06bf70 0x7fde2c06e430 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fde440012e0 tx=0x7fde44005eb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.743+0000 7fde4aa00640 1 -- 172.21.15.141:0/873881 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde5415ecd0 con 0x7fde540ce8f0 2024-01-16T15:09:28.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.960+0000 7fde59400640 1 -- 172.21.15.141:0/873881 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fde5415e360 con 0x7fde2c06bf70 2024-01-16T15:09:28.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.963+0000 7fde4aa00640 1 -- 172.21.15.141:0/873881 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fde5415e360 con 0x7fde2c06bf70 2024-01-16T15:09:28.940 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:09:28.940 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:28.940 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:09:28.940 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:28.941 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:09:28.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 -- 172.21.15.141:0/873881 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde2c06bf70 msgr2=0x7fde2c06e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde2c06bf70 0x7fde2c06e430 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fde440012e0 tx=0x7fde44005eb0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 -- 172.21.15.141:0/873881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 msgr2=0x7fde54158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde54158200 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0098c0 tx=0x7fde3c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 -- 172.21.15.141:0/873881 shutdown_connections 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fde2c06bf70 0x7fde2c06e430 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fde54160f80 0x7fde5415d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde540ce8f0 0x7fde54158200 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 --2- 172.21.15.141:0/873881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde540b2a00 0x7fde541587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 -- 172.21.15.141:0/873881 >> 172.21.15.141:0/873881 conn(0x7fde5409c550 msgr2=0x7fde540b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 -- 172.21.15.141:0/873881 shutdown_connections 2024-01-16T15:09:28.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:28.966+0000 7fde59400640 1 -- 172.21.15.141:0/873881 wait complete. 2024-01-16T15:09:28.963 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:09:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.197+0000 7f59c9000640 1 Processor -- start 2024-01-16T15:09:29.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.197+0000 7f59c9000640 1 -- start start 2024-01-16T15:09:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c9000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59c414b2c0 0x7f59c40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c9000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f59c40cf7d0 0x7f59c4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c9000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c40cfbf0 con 0x7f59c4070ea0 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c9000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c414b6a0 con 0x7f59c414b2c0 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c9000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c4071690 con 0x7f59c40cf7d0 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c1e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59c414b2c0 0x7f59c40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c1e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59c414b2c0 0x7f59c40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34284/0 (socket says 172.21.15.141:34284) 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c1e00640 1 -- 172.21.15.141:0/4041956732 learned_addr learned my addr 172.21.15.141:0/4041956732 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59b9200640 1 --2- 172.21.15.141:0/4041956732 >> v2:172.21.15.188:3300/0 conn(0x7f59c40cf7d0 0x7f59c4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c1e00640 1 -- 172.21.15.141:0/4041956732 >> v2:172.21.15.188:3300/0 conn(0x7f59c40cf7d0 msgr2=0x7f59c4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.199+0000 7f59c1e00640 1 --2- 172.21.15.141:0/4041956732 >> v2:172.21.15.188:3300/0 conn(0x7f59c40cf7d0 0x7f59c4070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.200+0000 7f59c1e00640 1 -- 172.21.15.141:0/4041956732 >> v1:172.21.15.188:6789/0 conn(0x7f59c4070ea0 legacy=0x7f59c40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.200+0000 7f59c1e00640 1 -- 172.21.15.141:0/4041956732 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c4071970 con 0x7f59c414b2c0 2024-01-16T15:09:29.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.201+0000 7f59c1e00640 1 --2- 172.21.15.141:0/4041956732 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c40cf290 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f59c40cfea0 tx=0x7f59b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb550b37ab55b389 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.201+0000 7f59c1400640 1 -- 172.21.15.141:0/4041956732 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4031870 con 0x7f59c414b2c0 2024-01-16T15:09:29.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.201+0000 7f59c1400640 1 -- 172.21.15.141:0/4041956732 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b40319d0 con 0x7f59c414b2c0 2024-01-16T15:09:29.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.201+0000 7f59c1400640 1 -- 172.21.15.141:0/4041956732 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4031d60 con 0x7f59c414b2c0 2024-01-16T15:09:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 -- 172.21.15.141:0/4041956732 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 msgr2=0x7f59c40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 --2- 172.21.15.141:0/4041956732 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c40cf290 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f59c40cfea0 tx=0x7f59b402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 -- 172.21.15.141:0/4041956732 shutdown_connections 2024-01-16T15:09:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 --2- 172.21.15.141:0/4041956732 >> v2:172.21.15.188:3300/0 conn(0x7f59c40cf7d0 0x7f59c4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 --2- 172.21.15.141:0/4041956732 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c40cf290 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 -- 172.21.15.141:0/4041956732 >> 172.21.15.141:0/4041956732 conn(0x7f59c406a800 msgr2=0x7f59c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 -- 172.21.15.141:0/4041956732 shutdown_connections 2024-01-16T15:09:29.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.202+0000 7f59c9000640 1 -- 172.21.15.141:0/4041956732 wait complete. 2024-01-16T15:09:29.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.204+0000 7f59c9000640 1 Processor -- start 2024-01-16T15:09:29.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.204+0000 7f59c9000640 1 -- start start 2024-01-16T15:09:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c9000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59c4070ea0 0x7f59c4152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c9000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59c40cf7d0 0x7f59c414fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c9000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c41f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c9000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4164e20 con 0x7f59c414b2c0 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c9000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4146a00 con 0x7f59c40cf7d0 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c9000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4146110 con 0x7f59c4070ea0 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c41f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c41f6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34292/0 (socket says 172.21.15.141:34292) 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59c2800640 1 -- 172.21.15.141:0/2679225603 learned_addr learned my addr 172.21.15.141:0/2679225603 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.206+0000 7f59b9200640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59c40cf7d0 0x7f59c414fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c1e00640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59c4070ea0 0x7f59c4152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c2800640 1 -- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59c4070ea0 msgr2=0x7f59c4152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c2800640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59c4070ea0 0x7f59c4152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c2800640 1 -- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59c40cf7d0 msgr2=0x7f59c414fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c2800640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59c40cf7d0 0x7f59c414fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c2800640 1 -- 172.21.15.141:0/2679225603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40094d0 con 0x7f59c414b2c0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c2800640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c41f6b40 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00c780 tx=0x7f59bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59baa00640 1 -- 172.21.15.141:0/2679225603 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59bc003d80 con 0x7f59c414b2c0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59baa00640 1 -- 172.21.15.141:0/2679225603 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59bc00ce90 con 0x7f59c414b2c0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c4164b40 con 0x7f59c414b2c0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59baa00640 1 -- 172.21.15.141:0/2679225603 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59bc0052b0 con 0x7f59c414b2c0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.207+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c4146690 con 0x7f59c414b2c0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.208+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c4146bb0 con 0x7f59c414b2c0 2024-01-16T15:09:29.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.209+0000 7f59baa00640 1 -- 172.21.15.141:0/2679225603 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f59bc005410 con 0x7f59c414b2c0 2024-01-16T15:09:29.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.210+0000 7f59baa00640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f599c06bf50 0x7f599c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.210+0000 7f59baa00640 1 -- 172.21.15.141:0/2679225603 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f59bc0880a0 con 0x7f59c414b2c0 2024-01-16T15:09:29.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.211+0000 7f59c1e00640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f599c06bf50 0x7f599c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.212+0000 7f59c1e00640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f599c06bf50 0x7f599c06e410 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f59b4036000 tx=0x7f59b403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.214+0000 7f59baa00640 1 -- 172.21.15.141:0/2679225603 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59c4146bb0 con 0x7f59c414b2c0 2024-01-16T15:09:29.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.433+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59c40cbc40 con 0x7f599c06bf50 2024-01-16T15:09:29.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.478+0000 7f59baa00640 1 -- 172.21.15.141:0/2679225603 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1029 (secure 0 0 0) 0x7f59b4031d60 con 0x7f599c06bf50 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59636" 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596316" 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596318" 2024-01-16T15:09:29.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59635" 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59632" 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59634" 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59631" 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.456 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "sv1" 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59638" 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596311" 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596312" 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596310" 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596314" 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596317" 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59639" 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596315" 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59633" 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596319" 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596313" 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone59637" 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone596320" 2024-01-16T15:09:29.459 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:29.460 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f599c06bf50 msgr2=0x7f599c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f599c06bf50 0x7f599c06e410 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f59b4036000 tx=0x7f59b403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 msgr2=0x7f59c41f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c41f6b40 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00c780 tx=0x7f59bc00cc40 comp rx=0 tx=0).stop 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 shutdown_connections 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f599c06bf50 0x7f599c06e410 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59c414b2c0 0x7f59c41f6b40 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59c40cf7d0 0x7f59c414fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 --2- 172.21.15.141:0/2679225603 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59c4070ea0 0x7f59c4152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 >> 172.21.15.141:0/2679225603 conn(0x7f59c406a800 msgr2=0x7f59c40ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 shutdown_connections 2024-01-16T15:09:29.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.481+0000 7f59c9000640 1 -- 172.21.15.141:0/2679225603 wait complete. 2024-01-16T15:09:29.477 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59636 2024-01-16T15:09:29.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.685+0000 7fdaa3400640 1 Processor -- start 2024-01-16T15:09:29.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.686+0000 7fdaa3400640 1 -- start start 2024-01-16T15:09:29.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.687+0000 7fdaa3400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fda9c143ae0 0x7fda9c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.687+0000 7fdaa3400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c15e3b0 0x7fda9c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.687+0000 7fdaa3400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c0cfc30 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.687+0000 7fdaa3400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9c0b1ee0 con 0x7fda9c143ae0 2024-01-16T15:09:29.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.687+0000 7fdaa3400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c0b2780 con 0x7fda9c15e3b0 2024-01-16T15:09:29.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.688+0000 7fdaa0a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fda9c143ae0 0x7fda9c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.688+0000 7fdaa1400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c15e3b0 0x7fda9c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.688+0000 7fdaa0a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fda9c143ae0 0x7fda9c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:60778/0 (socket says 172.21.15.141:60778) 2024-01-16T15:09:29.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.688+0000 7fdaa0a00640 1 -- 172.21.15.141:0/1740357102 learned_addr learned my addr 172.21.15.141:0/1740357102 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:29.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.689+0000 7fdaa1400640 1 -- 172.21.15.141:0/1740357102 >> v2:172.21.15.188:3300/0 conn(0x7fda9c143ae0 msgr2=0x7fda9c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.689+0000 7fdaa1400640 1 --2- 172.21.15.141:0/1740357102 >> v2:172.21.15.188:3300/0 conn(0x7fda9c143ae0 0x7fda9c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.689+0000 7fdaa1400640 1 -- 172.21.15.141:0/1740357102 >> v1:172.21.15.188:6789/0 conn(0x7fda9c0cf7d0 legacy=0x7fda9c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.689+0000 7fdaa1400640 1 -- 172.21.15.141:0/1740357102 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c15e990 con 0x7fda9c15e3b0 2024-01-16T15:09:29.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.689+0000 7fdaa0a00640 1 --2- 172.21.15.141:0/1740357102 >> v2:172.21.15.188:3300/0 conn(0x7fda9c143ae0 0x7fda9c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:29.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.690+0000 7fdaa1400640 1 --2- 172.21.15.141:0/1740357102 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c15e3b0 0x7fda9c0702c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fda940097a0 tx=0x7fda9402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32a5e59b0873670e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.690+0000 7fda9be00640 1 -- 172.21.15.141:0/1740357102 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9402d040 con 0x7fda9c15e3b0 2024-01-16T15:09:29.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.690+0000 7fda9be00640 1 -- 172.21.15.141:0/1740357102 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda94031810 con 0x7fda9c15e3b0 2024-01-16T15:09:29.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.690+0000 7fda9be00640 1 -- 172.21.15.141:0/1740357102 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94031ba0 con 0x7fda9c15e3b0 2024-01-16T15:09:29.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 -- 172.21.15.141:0/1740357102 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c15e3b0 msgr2=0x7fda9c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1740357102 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c15e3b0 0x7fda9c0702c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fda940097a0 tx=0x7fda9402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 -- 172.21.15.141:0/1740357102 shutdown_connections 2024-01-16T15:09:29.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1740357102 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c15e3b0 0x7fda9c0702c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1740357102 >> v2:172.21.15.188:3300/0 conn(0x7fda9c143ae0 0x7fda9c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 -- 172.21.15.141:0/1740357102 >> 172.21.15.141:0/1740357102 conn(0x7fda9c06a800 msgr2=0x7fda9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 -- 172.21.15.141:0/1740357102 shutdown_connections 2024-01-16T15:09:29.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.691+0000 7fdaa3400640 1 -- 172.21.15.141:0/1740357102 wait complete. 2024-01-16T15:09:29.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.693+0000 7fdaa3400640 1 Processor -- start 2024-01-16T15:09:29.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.693+0000 7fdaa3400640 1 -- start start 2024-01-16T15:09:29.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.694+0000 7fdaa3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda9c0cf7d0 0x7fda9c1f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa3400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c143ae0 0x7fda9c159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda9c15e3b0 0x7fda9c0af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa3400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c1f7700 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa3400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c159ed0 con 0x7fda9c15e3b0 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa3400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c140430 con 0x7fda9c143ae0 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda9c0cf7d0 0x7fda9c1f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda9c0cf7d0 0x7fda9c1f7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34294/0 (socket says 172.21.15.141:34294) 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 -- 172.21.15.141:0/1125943600 learned_addr learned my addr 172.21.15.141:0/1125943600 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fda93c00640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c143ae0 0x7fda9c159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa1400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda9c15e3b0 0x7fda9c0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 -- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c143ae0 msgr2=0x7fda9c159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c143ae0 0x7fda9c159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 -- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda9c15e3b0 msgr2=0x7fda9c0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda9c15e3b0 0x7fda9c0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa0a00640 1 -- 172.21.15.141:0/1125943600 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c0094d0 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fda93c00640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c143ae0 0x7fda9c159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.695+0000 7fdaa1400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda9c15e3b0 0x7fda9c0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.696+0000 7fdaa0a00640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda9c0cf7d0 0x7fda9c1f7120 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00e9d0 tx=0x7fda8c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.696+0000 7fda99600640 1 -- 172.21.15.141:0/1125943600 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c00ca60 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.696+0000 7fda99600640 1 -- 172.21.15.141:0/1125943600 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c00cbc0 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.696+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda94009400 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.696+0000 7fda99600640 1 -- 172.21.15.141:0/1125943600 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c013470 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.696+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9c1409f0 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.697+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9c0675d0 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.698+0000 7fda99600640 1 -- 172.21.15.141:0/1125943600 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fda8c0135d0 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.700+0000 7fda99600640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7406bf20 0x7fda7406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.700+0000 7fda99600640 1 -- 172.21.15.141:0/1125943600 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fda8c088b40 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.700+0000 7fda93c00640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7406bf20 0x7fda7406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.701+0000 7fda93c00640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7406bf20 0x7fda7406e3e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fda9c142cd0 tx=0x7fda84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.704+0000 7fda99600640 1 -- 172.21.15.141:0/1125943600 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda8c0551e0 con 0x7fda9c0cf7d0 2024-01-16T15:09:29.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.908+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59636", "target": ["mon-mgr", ""]}) v1 -- 0x7fda9c0b2410 con 0x7fda7406bf20 2024-01-16T15:09:29.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.952+0000 7fda99600640 1 -- 172.21.15.141:0/1125943600 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fda84001a50 con 0x7fda7406bf20 2024-01-16T15:09:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7406bf20 msgr2=0x7fda7406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7406bf20 0x7fda7406e3e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fda9c142cd0 tx=0x7fda84009040 comp rx=0 tx=0).stop 2024-01-16T15:09:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda9c0cf7d0 msgr2=0x7fda9c1f7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda9c0cf7d0 0x7fda9c1f7120 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00e9d0 tx=0x7fda8c00ee90 comp rx=0 tx=0).stop 2024-01-16T15:09:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 shutdown_connections 2024-01-16T15:09:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda7406bf20 0x7fda7406e3e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda9c15e3b0 0x7fda9c0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda9c143ae0 0x7fda9c159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 --2- 172.21.15.141:0/1125943600 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda9c0cf7d0 0x7fda9c1f7120 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 >> 172.21.15.141:0/1125943600 conn(0x7fda9c06a800 msgr2=0x7fda9c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 shutdown_connections 2024-01-16T15:09:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:29.955+0000 7fdaa3400640 1 -- 172.21.15.141:0/1125943600 wait complete. 2024-01-16T15:09:29.939 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.961+0000 7f0f2da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.939 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.962+0000 7f0f2da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.939 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.962+0000 7f0f2da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.939 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.962+0000 7f0f2da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.939 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.962+0000 7f0f2da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.949 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596316 2024-01-16T15:09:29.968 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.990+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.968 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.990+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.968 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.990+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.968 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.990+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:29.968 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:09:29.990+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:30.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.171+0000 7f5c94800640 1 Processor -- start 2024-01-16T15:09:30.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.171+0000 7f5c94800640 1 -- start start 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.172+0000 7f5c94800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5c8c13df80 0x7f5c8c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c94800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c13e360 0x7f5c8c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c94800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5c8c15f9c0 0x7f5c8c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c94800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c06c470 con 0x7f5c8c13df80 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c94800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c06c750 con 0x7f5c8c15f9c0 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c94800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c06ca30 con 0x7f5c8c13e360 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c91e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5c8c13df80 0x7f5c8c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c91e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5c8c13df80 0x7f5c8c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34310/0 (socket says 172.21.15.141:34310) 2024-01-16T15:09:30.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c91e00640 1 -- 172.21.15.141:0/2656673977 learned_addr learned my addr 172.21.15.141:0/2656673977 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:30.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c92800640 1 --2- 172.21.15.141:0/2656673977 >> v2:172.21.15.188:3300/0 conn(0x7f5c8c15f9c0 0x7f5c8c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.173+0000 7f5c91400640 1 --2- 172.21.15.141:0/2656673977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c13e360 0x7f5c8c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.174+0000 7f5c91e00640 1 -- 172.21.15.141:0/2656673977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c13e360 msgr2=0x7f5c8c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:30.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.174+0000 7f5c91e00640 1 --2- 172.21.15.141:0/2656673977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c13e360 0x7f5c8c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.174+0000 7f5c91e00640 1 -- 172.21.15.141:0/2656673977 >> v2:172.21.15.188:3300/0 conn(0x7f5c8c15f9c0 msgr2=0x7f5c8c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:30.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.174+0000 7f5c91e00640 1 --2- 172.21.15.141:0/2656673977 >> v2:172.21.15.188:3300/0 conn(0x7f5c8c15f9c0 0x7f5c8c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.174+0000 7f5c91e00640 1 -- 172.21.15.141:0/2656673977 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c06cd10 con 0x7f5c8c13df80 2024-01-16T15:09:30.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.175+0000 7f5c91e00640 1 --2- 172.21.15.141:0/2656673977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13df80 0x7f5c8c15bee0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06e6d0 tx=0x7f5c7c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3257c99179866f7f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.176+0000 7f5c90a00640 1 -- 172.21.15.141:0/2656673977 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c031860 con 0x7f5c8c13df80 2024-01-16T15:09:30.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.176+0000 7f5c90a00640 1 -- 172.21.15.141:0/2656673977 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c0319c0 con 0x7f5c8c13df80 2024-01-16T15:09:30.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.176+0000 7f5c90a00640 1 -- 172.21.15.141:0/2656673977 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c031d50 con 0x7f5c8c13df80 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 -- 172.21.15.141:0/2656673977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13df80 msgr2=0x7f5c8c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 --2- 172.21.15.141:0/2656673977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13df80 0x7f5c8c15bee0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06e6d0 tx=0x7f5c7c02b900 comp rx=0 tx=0).stop 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 -- 172.21.15.141:0/2656673977 shutdown_connections 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 --2- 172.21.15.141:0/2656673977 >> v2:172.21.15.188:3300/0 conn(0x7f5c8c15f9c0 0x7f5c8c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 --2- 172.21.15.141:0/2656673977 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c13e360 0x7f5c8c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 --2- 172.21.15.141:0/2656673977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13df80 0x7f5c8c15bee0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 -- 172.21.15.141:0/2656673977 >> 172.21.15.141:0/2656673977 conn(0x7f5c8c06a800 msgr2=0x7f5c8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 -- 172.21.15.141:0/2656673977 shutdown_connections 2024-01-16T15:09:30.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.177+0000 7f5c94800640 1 -- 172.21.15.141:0/2656673977 wait complete. 2024-01-16T15:09:30.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.178+0000 7f5c94800640 1 Processor -- start 2024-01-16T15:09:30.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.179+0000 7f5c94800640 1 -- start start 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.179+0000 7f5c94800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c8c13df80 0x7f5c8c0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c94800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13e360 0x7f5c8c153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c94800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c15f9c0 0x7f5c8c154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c94800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c15a2c0 con 0x7f5c8c13e360 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c94800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c1f1400 con 0x7f5c8c13df80 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c94800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c1f16e0 con 0x7f5c8c15f9c0 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13e360 0x7f5c8c153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13e360 0x7f5c8c153d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34324/0 (socket says 172.21.15.141:34324) 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 -- 172.21.15.141:0/100044292 learned_addr learned my addr 172.21.15.141:0/100044292 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 -- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c15f9c0 msgr2=0x7f5c8c154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c15f9c0 0x7f5c8c154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 -- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c8c13df80 msgr2=0x7f5c8c0b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c8c13df80 0x7f5c8c0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 -- 172.21.15.141:0/100044292 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c0094d0 con 0x7f5c8c13e360 2024-01-16T15:09:30.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.180+0000 7f5c91400640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13e360 0x7f5c8c153d20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400d800 tx=0x7f5c7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.181+0000 7f5c8a000640 1 -- 172.21.15.141:0/100044292 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400b6c0 con 0x7f5c8c13e360 2024-01-16T15:09:30.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.181+0000 7f5c8a000640 1 -- 172.21.15.141:0/100044292 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7400b820 con 0x7f5c8c13e360 2024-01-16T15:09:30.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.181+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c1f1c80 con 0x7f5c8c13e360 2024-01-16T15:09:30.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.181+0000 7f5c8a000640 1 -- 172.21.15.141:0/100044292 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400bbb0 con 0x7f5c8c13e360 2024-01-16T15:09:30.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.181+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c154dc0 con 0x7f5c8c13e360 2024-01-16T15:09:30.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.182+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c1f1900 con 0x7f5c8c13e360 2024-01-16T15:09:30.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.183+0000 7f5c8a000640 1 -- 172.21.15.141:0/100044292 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f5c7400bd10 con 0x7f5c8c13e360 2024-01-16T15:09:30.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.184+0000 7f5c8a000640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c6406bf20 0x7f5c6406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.184+0000 7f5c8a000640 1 -- 172.21.15.141:0/100044292 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5c74088b40 con 0x7f5c8c13e360 2024-01-16T15:09:30.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.184+0000 7f5c91e00640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c6406bf20 0x7f5c6406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.186+0000 7f5c91e00640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c6406bf20 0x7f5c6406e3e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c036000 tx=0x7f5c7c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.188+0000 7f5c8a000640 1 -- 172.21.15.141:0/100044292 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c74055400 con 0x7f5c8c13e360 2024-01-16T15:09:30.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.408+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596316", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c8c1408d0 con 0x7f5c6406bf20 2024-01-16T15:09:30.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.459+0000 7f5c8a000640 1 -- 172.21.15.141:0/100044292 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5c7c03a420 con 0x7f5c6406bf20 2024-01-16T15:09:30.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c6406bf20 msgr2=0x7f5c6406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c6406bf20 0x7f5c6406e3e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c036000 tx=0x7f5c7c030000 comp rx=0 tx=0).stop 2024-01-16T15:09:30.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13e360 msgr2=0x7f5c8c153d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13e360 0x7f5c8c153d20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400d800 tx=0x7f5c7400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 shutdown_connections 2024-01-16T15:09:30.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c6406bf20 0x7f5c6406e3e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.439 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c8c15f9c0 0x7f5c8c154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c8c13e360 0x7f5c8c153d20 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 --2- 172.21.15.141:0/100044292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c8c13df80 0x7f5c8c0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 >> 172.21.15.141:0/100044292 conn(0x7f5c8c06a800 msgr2=0x7f5c8c15d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 shutdown_connections 2024-01-16T15:09:30.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.462+0000 7f5c94800640 1 -- 172.21.15.141:0/100044292 wait complete. 2024-01-16T15:09:30.457 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596318 2024-01-16T15:09:30.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.687+0000 7fea41000640 1 Processor -- start 2024-01-16T15:09:30.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.687+0000 7fea41000640 1 -- start start 2024-01-16T15:09:30.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.689+0000 7fea41000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 0x7fea3c164d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.689+0000 7fea41000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c0cdc40 con 0x7fea3c13cfa0 2024-01-16T15:09:30.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.689+0000 7fea41000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c0cdec0 con 0x7fea3c15afb0 2024-01-16T15:09:30.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.689+0000 7fea41000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c0ce1a0 con 0x7fea3c13fcb0 2024-01-16T15:09:30.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.689+0000 7fea3b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fea3c13cfa0 0x7fea3c0cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55586/0 (socket says 172.21.15.141:55586) 2024-01-16T15:09:30.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.689+0000 7fea3b400640 1 -- 172.21.15.141:0/1816578708 learned_addr learned my addr 172.21.15.141:0/1816578708 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755596702 0 0) 0x7fea3c0cdc40 con 0x7fea3c13cfa0 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3be00640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 0x7fea3c164d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c003520 con 0x7fea3c13cfa0 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610171042 0 0) 0x7fea1c003520 con 0x7fea3c13cfa0 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 msgr2=0x7fea3c164d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 0x7fea3c164d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 >> v1:172.21.15.188:6789/0 conn(0x7fea3c15afb0 legacy=0x7fea3c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c0ce480 con 0x7fea3c13cfa0 2024-01-16T15:09:30.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fea2c001670 con 0x7fea3c13cfa0 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 >> v1:172.21.15.141:6789/0 conn(0x7fea3c13cfa0 legacy=0x7fea3c0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 0x7fea1c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c003fd0 0x7fea1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c0ce1a0 con 0x7fea1c004d00 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c00ddd0 con 0x7fea1c003fd0 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c00df90 con 0x7fea3c13fcb0 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.690+0000 7fea3be00640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.691+0000 7fea3aa00640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c003fd0 0x7fea1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.691+0000 7fea3b400640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 0x7fea1c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.691+0000 7fea3be00640 1 -- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 msgr2=0x7fea1c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.691+0000 7fea3be00640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c13fcb0 0x7fea1c0039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.691+0000 7fea3be00640 1 -- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c003fd0 msgr2=0x7fea1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.691+0000 7fea3be00640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c003fd0 0x7fea1c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.691+0000 7fea3be00640 1 -- 172.21.15.141:0/1816578708 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c003520 con 0x7fea1c004d00 2024-01-16T15:09:30.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.692+0000 7fea3be00640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fea34007360 tx=0x7fea1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7bc0d30b71f615b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.692+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3402ccd0 con 0x7fea1c004d00 2024-01-16T15:09:30.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.692+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3402ce30 con 0x7fea1c004d00 2024-01-16T15:09:30.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.692+0000 7fea3a000640 1 -- 172.21.15.141:0/1816578708 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34034660 con 0x7fea1c004d00 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.693+0000 7fea41000640 1 -- 172.21.15.141:0/1816578708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea1c004d00 msgr2=0x7fea1c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.693+0000 7fea41000640 1 --2- 172.21.15.141:0/1816578708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fea34007360 tx=0x7fea1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.693+0000 7fea3be00640 1 -- 172.21.15.141:0/1816578708 reap_dead start 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.693+0000 7fea41000640 1 -- 172.21.15.141:0/1816578708 shutdown_connections 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.693+0000 7fea41000640 1 -- 172.21.15.141:0/1816578708 >> 172.21.15.141:0/1816578708 conn(0x7fea3c06a800 msgr2=0x7fea3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.693+0000 7fea41000640 1 -- 172.21.15.141:0/1816578708 shutdown_connections 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.693+0000 7fea41000640 1 -- 172.21.15.141:0/1816578708 wait complete. 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.695+0000 7fea41000640 1 Processor -- start 2024-01-16T15:09:30.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.695+0000 7fea41000640 1 -- start start 2024-01-16T15:09:30.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea41000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c004d00 0x7fea3c156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea41000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c1f7240 0x7fea3c1f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea41000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea3c159720 0x7fea3c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea41000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c154ed0 con 0x7fea3c159720 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea41000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c1f3960 con 0x7fea1c004d00 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea41000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c1f3c40 con 0x7fea3c1f7240 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea3c159720 0x7fea3c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea3c159720 0x7fea3c156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34348/0 (socket says 172.21.15.141:34348) 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 -- 172.21.15.141:0/1324776975 learned_addr learned my addr 172.21.15.141:0/1324776975 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3aa00640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c1f7240 0x7fea3c1f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3b400640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c004d00 0x7fea3c156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 -- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c1f7240 msgr2=0x7fea3c1f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c1f7240 0x7fea3c1f2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 -- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c004d00 msgr2=0x7fea3c156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c004d00 0x7fea3c156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3be00640 1 -- 172.21.15.141:0/1324776975 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c1f3f20 con 0x7fea3c159720 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.696+0000 7fea3b400640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c004d00 0x7fea3c156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.697+0000 7fea3be00640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea3c159720 0x7fea3c156ef0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fea34007360 tx=0x7fea34004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.697+0000 7fea33400640 1 -- 172.21.15.141:0/1324776975 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34045070 con 0x7fea3c159720 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.697+0000 7fea33400640 1 -- 172.21.15.141:0/1324776975 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea34034e20 con 0x7fea3c159720 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.697+0000 7fea33400640 1 -- 172.21.15.141:0/1324776975 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3403d620 con 0x7fea3c159720 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.697+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c154b60 con 0x7fea3c159720 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.697+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c1f3500 con 0x7fea3c159720 2024-01-16T15:09:30.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.698+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c1f36f0 con 0x7fea3c159720 2024-01-16T15:09:30.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.699+0000 7fea33400640 1 -- 172.21.15.141:0/1324776975 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fea3403b040 con 0x7fea3c159720 2024-01-16T15:09:30.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.700+0000 7fea33400640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1406bf50 0x7fea1406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:30.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.700+0000 7fea33400640 1 -- 172.21.15.141:0/1324776975 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea340a9860 con 0x7fea3c159720 2024-01-16T15:09:30.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.701+0000 7fea3b400640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1406bf50 0x7fea1406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:30.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.702+0000 7fea3b400640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1406bf50 0x7fea1406e410 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0017a0 tx=0x7fea2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:30.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.705+0000 7fea33400640 1 -- 172.21.15.141:0/1324776975 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea34075e60 con 0x7fea3c159720 2024-01-16T15:09:30.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.926+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596318", "target": ["mon-mgr", ""]}) v1 -- 0x7fea3c0cea40 con 0x7fea1406bf50 2024-01-16T15:09:30.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.968+0000 7fea33400640 1 -- 172.21.15.141:0/1324776975 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fea2c00d040 con 0x7fea1406bf50 2024-01-16T15:09:30.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.971+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1406bf50 msgr2=0x7fea1406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.971+0000 7fea41000640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1406bf50 0x7fea1406e410 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0017a0 tx=0x7fea2c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:30.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea3c159720 msgr2=0x7fea3c156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea3c159720 0x7fea3c156ef0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fea34007360 tx=0x7fea34004ab0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 shutdown_connections 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea1406bf50 0x7fea1406e410 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea3c159720 0x7fea3c156ef0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fea3c1f7240 0x7fea3c1f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 --2- 172.21.15.141:0/1324776975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea1c004d00 0x7fea3c156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 >> 172.21.15.141:0/1324776975 conn(0x7fea3c06a800 msgr2=0x7fea3c153730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 shutdown_connections 2024-01-16T15:09:30.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:30.972+0000 7fea41000640 1 -- 172.21.15.141:0/1324776975 wait complete. 2024-01-16T15:09:30.972 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59635 2024-01-16T15:09:31.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.198+0000 7fe213200640 1 Processor -- start 2024-01-16T15:09:31.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.198+0000 7fe213200640 1 -- start start 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe213200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe20c143ae0 0x7fe20c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe213200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe20c15e3b0 0x7fe20c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe213200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0cfc30 con 0x7fe20c0cf7d0 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe213200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c0b1ee0 con 0x7fe20c143ae0 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe213200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c0b2780 con 0x7fe20c15e3b0 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe20be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe20c143ae0 0x7fe20c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe20be00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe20c143ae0 0x7fe20c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34356/0 (socket says 172.21.15.141:34356) 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.200+0000 7fe20be00640 1 -- 172.21.15.141:0/2613972118 learned_addr learned my addr 172.21.15.141:0/2613972118 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.201+0000 7fe211200640 1 --2- 172.21.15.141:0/2613972118 >> v2:172.21.15.188:3300/0 conn(0x7fe20c15e3b0 0x7fe20c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.201+0000 7fe20be00640 1 -- 172.21.15.141:0/2613972118 >> v2:172.21.15.188:3300/0 conn(0x7fe20c15e3b0 msgr2=0x7fe20c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.201+0000 7fe20be00640 1 --2- 172.21.15.141:0/2613972118 >> v2:172.21.15.188:3300/0 conn(0x7fe20c15e3b0 0x7fe20c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.201+0000 7fe20be00640 1 -- 172.21.15.141:0/2613972118 >> v1:172.21.15.188:6789/0 conn(0x7fe20c0cf7d0 legacy=0x7fe20c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.201+0000 7fe20be00640 1 -- 172.21.15.141:0/2613972118 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c15e990 con 0x7fe20c143ae0 2024-01-16T15:09:31.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.202+0000 7fe20be00640 1 --2- 172.21.15.141:0/2613972118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c0cf290 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fe20c06e4f0 tx=0x7fe1f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ab76aaeb4b7ffd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.202+0000 7fe20aa00640 1 -- 172.21.15.141:0/2613972118 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4031870 con 0x7fe20c143ae0 2024-01-16T15:09:31.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe20aa00640 1 -- 172.21.15.141:0/2613972118 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f40319d0 con 0x7fe20c143ae0 2024-01-16T15:09:31.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe20aa00640 1 -- 172.21.15.141:0/2613972118 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4031d60 con 0x7fe20c143ae0 2024-01-16T15:09:31.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 -- 172.21.15.141:0/2613972118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 msgr2=0x7fe20c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 --2- 172.21.15.141:0/2613972118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c0cf290 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fe20c06e4f0 tx=0x7fe1f402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:31.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 -- 172.21.15.141:0/2613972118 shutdown_connections 2024-01-16T15:09:31.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 --2- 172.21.15.141:0/2613972118 >> v2:172.21.15.188:3300/0 conn(0x7fe20c15e3b0 0x7fe20c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 --2- 172.21.15.141:0/2613972118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c0cf290 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 -- 172.21.15.141:0/2613972118 >> 172.21.15.141:0/2613972118 conn(0x7fe20c06a800 msgr2=0x7fe20c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:31.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 -- 172.21.15.141:0/2613972118 shutdown_connections 2024-01-16T15:09:31.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.203+0000 7fe213200640 1 -- 172.21.15.141:0/2613972118 wait complete. 2024-01-16T15:09:31.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.205+0000 7fe213200640 1 Processor -- start 2024-01-16T15:09:31.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.205+0000 7fe213200640 1 -- start start 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.206+0000 7fe213200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe20c0cf7d0 0x7fe20c141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.206+0000 7fe213200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c06c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.206+0000 7fe213200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe20c15e3b0 0x7fe20c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.206+0000 7fe213200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c1349a0 con 0x7fe20c143ae0 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.206+0000 7fe213200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c142420 con 0x7fe20c0cf7d0 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.206+0000 7fe213200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c131b10 con 0x7fe20c15e3b0 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.206+0000 7fe20b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c06c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c06c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34366/0 (socket says 172.21.15.141:34366) 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 -- 172.21.15.141:0/3976600495 learned_addr learned my addr 172.21.15.141:0/3976600495 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20be00640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe20c0cf7d0 0x7fe20c141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 -- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe20c15e3b0 msgr2=0x7fe20c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe211200640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe20c15e3b0 0x7fe20c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe20c15e3b0 0x7fe20c06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 -- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe20c0cf7d0 msgr2=0x7fe20c141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe20c0cf7d0 0x7fe20c141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 -- 172.21.15.141:0/3976600495 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f40094d0 con 0x7fe20c143ae0 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20be00640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe20c0cf7d0 0x7fe20c141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:31.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.207+0000 7fe20b400640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c06c6a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00c430 tx=0x7fe1fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.208+0000 7fe203e00640 1 -- 172.21.15.141:0/3976600495 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc011790 con 0x7fe20c143ae0 2024-01-16T15:09:31.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.208+0000 7fe203e00640 1 -- 172.21.15.141:0/3976600495 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc0118f0 con 0x7fe20c143ae0 2024-01-16T15:09:31.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.208+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20c131df0 con 0x7fe20c143ae0 2024-01-16T15:09:31.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.208+0000 7fe203e00640 1 -- 172.21.15.141:0/3976600495 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc011ca0 con 0x7fe20c143ae0 2024-01-16T15:09:31.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.208+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c1323b0 con 0x7fe20c143ae0 2024-01-16T15:09:31.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.209+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20c0675d0 con 0x7fe20c143ae0 2024-01-16T15:09:31.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.210+0000 7fe203e00640 1 -- 172.21.15.141:0/3976600495 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe1fc01c460 con 0x7fe20c143ae0 2024-01-16T15:09:31.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.211+0000 7fe203e00640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe1e406bf20 0x7fe1e406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.211+0000 7fe203e00640 1 -- 172.21.15.141:0/3976600495 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe1fc088530 con 0x7fe20c143ae0 2024-01-16T15:09:31.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.211+0000 7fe20be00640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe1e406bf20 0x7fe1e406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.213+0000 7fe20be00640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe1e406bf20 0x7fe1e406e3e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4036000 tx=0x7fe1f403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.216+0000 7fe203e00640 1 -- 172.21.15.141:0/3976600495 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1fc054e50 con 0x7fe20c143ae0 2024-01-16T15:09:31.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.433+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59635", "target": ["mon-mgr", ""]}) v1 -- 0x7fe20c0b2410 con 0x7fe1e406bf20 2024-01-16T15:09:31.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.472+0000 7fe203e00640 1 -- 172.21.15.141:0/3976600495 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe1f403a420 con 0x7fe1e406bf20 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe1e406bf20 msgr2=0x7fe1e406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe1e406bf20 0x7fe1e406e3e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4036000 tx=0x7fe1f403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 msgr2=0x7fe20c06c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c06c6a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00c430 tx=0x7fe1fc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 shutdown_connections 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe1e406bf20 0x7fe1e406e3e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe20c15e3b0 0x7fe20c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe20c143ae0 0x7fe20c06c6a0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 --2- 172.21.15.141:0/3976600495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe20c0cf7d0 0x7fe20c141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 >> 172.21.15.141:0/3976600495 conn(0x7fe20c06a800 msgr2=0x7fe20c15a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:31.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 shutdown_connections 2024-01-16T15:09:31.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.475+0000 7fe213200640 1 -- 172.21.15.141:0/3976600495 wait complete. 2024-01-16T15:09:31.475 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59632 2024-01-16T15:09:31.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.700+0000 7f5f72400640 1 Processor -- start 2024-01-16T15:09:31.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.700+0000 7f5f72400640 1 -- start start 2024-01-16T15:09:31.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.701+0000 7f5f72400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5f6c0cf9a0 0x7f5f6c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.701+0000 7f5f72400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c15e1e0 0x7f5f6c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.701+0000 7f5f72400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c0b1ee0 con 0x7f5f6c143ae0 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.701+0000 7f5f72400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6c0b2780 con 0x7f5f6c0cf9a0 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.701+0000 7f5f72400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c15e990 con 0x7f5f6c15e1e0 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5f6c143ae0 0x7f5f6c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55588/0 (socket says 172.21.15.141:55588) 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6b400640 1 -- 172.21.15.141:0/3967665698 learned_addr learned my addr 172.21.15.141:0/3967665698 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968652533 0 0) 0x7f5f6c0b1ee0 con 0x7f5f6c143ae0 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c003520 con 0x7f5f6c143ae0 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6be00640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c15e1e0 0x7f5f6c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f62800640 1 --2- 172.21.15.141:0/3967665698 >> v2:172.21.15.188:3300/0 conn(0x7f5f6c0cf9a0 0x7f5f6c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057292894 0 0) 0x7f5f4c003520 con 0x7f5f6c143ae0 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c15e1e0 msgr2=0x7f5f6c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c15e1e0 0x7f5f6c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 >> v2:172.21.15.188:3300/0 conn(0x7f5f6c0cf9a0 msgr2=0x7f5f6c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 --2- 172.21.15.141:0/3967665698 >> v2:172.21.15.188:3300/0 conn(0x7f5f6c0cf9a0 0x7f5f6c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.702+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c0b2130 con 0x7f5f6c143ae0 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f5f5c001670 con 0x7f5f6c143ae0 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 >> v1:172.21.15.141:6789/0 conn(0x7f5f6c143ae0 legacy=0x7f5f6c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f4c003970 0x7f5f4c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c15e1e0 0x7f5f4c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f4c004d00 0x7f5f4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c0b2780 con 0x7f5f4c003970 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c15e990 con 0x7f5f4c004d00 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c00dcf0 con 0x7f5f6c15e1e0 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6b400640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f4c003970 0x7f5f4c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6b400640 1 -- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c15e1e0 msgr2=0x7f5f4c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6b400640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c15e1e0 0x7f5f4c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6b400640 1 -- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f4c004d00 msgr2=0x7f5f4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6b400640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f4c004d00 0x7f5f4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.703+0000 7f5f6b400640 1 -- 172.21.15.141:0/3967665698 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c003520 con 0x7f5f4c003970 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.704+0000 7f5f6b400640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f4c003970 0x7f5f4c003d90 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00c190 tx=0x7f5f4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8010791172f2307 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c005ad0 con 0x7f5f4c003970 2024-01-16T15:09:31.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c005c50 con 0x7f5f4c003970 2024-01-16T15:09:31.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f6aa00640 1 -- 172.21.15.141:0/3967665698 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c02c810 con 0x7f5f4c003970 2024-01-16T15:09:31.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f72400640 1 -- 172.21.15.141:0/3967665698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f4c003970 msgr2=0x7f5f4c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f72400640 1 --2- 172.21.15.141:0/3967665698 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f4c003970 0x7f5f4c003d90 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00c190 tx=0x7f5f4c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f6be00640 1 -- 172.21.15.141:0/3967665698 reap_dead start 2024-01-16T15:09:31.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f72400640 1 -- 172.21.15.141:0/3967665698 shutdown_connections 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f72400640 1 -- 172.21.15.141:0/3967665698 >> 172.21.15.141:0/3967665698 conn(0x7f5f6c06a800 msgr2=0x7f5f6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f72400640 1 -- 172.21.15.141:0/3967665698 shutdown_connections 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.705+0000 7f5f72400640 1 -- 172.21.15.141:0/3967665698 wait complete. 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.707+0000 7f5f72400640 1 Processor -- start 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.707+0000 7f5f72400640 1 -- start start 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f72400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f4c003970 0x7f5f6c06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f72400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f6c1f2c90 0x7f5f6c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f72400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c136cc0 0x7f5f6c139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f72400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c136550 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f72400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1f34e0 con 0x7f5f4c003970 2024-01-16T15:09:31.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f72400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1f37c0 con 0x7f5f6c136cc0 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f62800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f6c1f2c90 0x7f5f6c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f62800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f6c1f2c90 0x7f5f6c159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34384/0 (socket says 172.21.15.141:34384) 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.708+0000 7f5f62800640 1 -- 172.21.15.141:0/21015817 learned_addr learned my addr 172.21.15.141:0/21015817 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f62800640 1 -- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c136cc0 msgr2=0x7f5f6c139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f62800640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c136cc0 0x7f5f6c139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f62800640 1 -- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f4c003970 msgr2=0x7f5f6c06cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f62800640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f4c003970 0x7f5f6c06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f62800640 1 -- 172.21.15.141:0/21015817 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c1f3aa0 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f62800640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f6c1f2c90 0x7f5f6c159a90 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400d800 tx=0x7f5f5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400b6c0 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5400b820 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400bbd0 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c1367a0 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.709+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c1f3110 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.710+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c0675d0 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.711+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 27) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f5f5400bd30 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.712+0000 7f5f63e00640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5f4406bf20 0x7f5f4406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:31.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.712+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f54088b00 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.713+0000 7f5f6b400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5f4406bf20 0x7f5f4406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:31.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.714+0000 7f5f6b400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5f4406bf20 0x7f5f4406e3e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c034000 tx=0x7f5f5c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:31.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.716+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f540553c0 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.818+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f5f54055790 con 0x7f5f6c1f2c90 2024-01-16T15:09:31.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.936+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59632", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f6c15e670 con 0x7f5f4406bf20 2024-01-16T15:09:31.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.975+0000 7f5f63e00640 1 -- 172.21.15.141:0/21015817 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5f5c001530 con 0x7f5f4406bf20 2024-01-16T15:09:31.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.977+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5f4406bf20 msgr2=0x7f5f4406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.977+0000 7f5f72400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5f4406bf20 0x7f5f4406e3e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c034000 tx=0x7f5f5c02e000 comp rx=0 tx=0).stop 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.977+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f6c1f2c90 msgr2=0x7f5f6c159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.977+0000 7f5f72400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f6c1f2c90 0x7f5f6c159a90 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400d800 tx=0x7f5f5400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 shutdown_connections 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5f4406bf20 0x7f5f4406e3e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5f6c136cc0 0x7f5f6c139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f6c1f2c90 0x7f5f6c159a90 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 --2- 172.21.15.141:0/21015817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f4c003970 0x7f5f6c06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 >> 172.21.15.141:0/21015817 conn(0x7f5f6c06a800 msgr2=0x7f5f6c0aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 shutdown_connections 2024-01-16T15:09:31.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:31.978+0000 7f5f72400640 1 -- 172.21.15.141:0/21015817 wait complete. 2024-01-16T15:09:31.976 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59634 2024-01-16T15:09:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.220+0000 7f6007a00640 1 Processor -- start 2024-01-16T15:09:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.220+0000 7f6007a00640 1 -- start start 2024-01-16T15:09:32.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.221+0000 7f6007a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f6007a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f600015e3b0 0x7f60000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f6007a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60000cfc30 con 0x7f60000cf7d0 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f6007a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60000b1ee0 con 0x7f600015e3b0 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f6007a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60000b2780 con 0x7f6000143ae0 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f5fffe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f60000cf7d0 0x7f600015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55602/0 (socket says 172.21.15.141:55602) 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f5fffe00640 1 -- 172.21.15.141:0/35264870 learned_addr learned my addr 172.21.15.141:0/35264870 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f6005000640 1 --2- 172.21.15.141:0/35264870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.222+0000 7f6005a00640 1 --2- 172.21.15.141:0/35264870 >> v2:172.21.15.188:3300/0 conn(0x7f600015e3b0 0x7f60000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.223+0000 7f5fff400640 1 -- 172.21.15.141:0/35264870 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724829356 0 0) 0x7f60000cfc30 con 0x7f60000cf7d0 2024-01-16T15:09:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.223+0000 7f5fff400640 1 -- 172.21.15.141:0/35264870 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe0003520 con 0x7f60000cf7d0 2024-01-16T15:09:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.223+0000 7f6005000640 1 -- 172.21.15.141:0/35264870 >> v2:172.21.15.188:3300/0 conn(0x7f600015e3b0 msgr2=0x7f60000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.223+0000 7f6005000640 1 --2- 172.21.15.141:0/35264870 >> v2:172.21.15.188:3300/0 conn(0x7f600015e3b0 0x7f60000702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.223+0000 7f6005000640 1 -- 172.21.15.141:0/35264870 >> v1:172.21.15.141:6789/0 conn(0x7f60000cf7d0 legacy=0x7f600015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.223+0000 7f6005000640 1 -- 172.21.15.141:0/35264870 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600015e990 con 0x7f6000143ae0 2024-01-16T15:09:32.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.223+0000 7f6005a00640 1 --2- 172.21.15.141:0/35264870 >> v2:172.21.15.188:3300/0 conn(0x7f600015e3b0 0x7f60000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:32.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.225+0000 7f6005000640 1 --2- 172.21.15.141:0/35264870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000cf290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00097a0 tx=0x7f5ff002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f98c34cce18ffc5e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.225+0000 7f5fff400640 1 -- 172.21.15.141:0/35264870 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff002d040 con 0x7f6000143ae0 2024-01-16T15:09:32.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.226+0000 7f5fff400640 1 -- 172.21.15.141:0/35264870 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff0031810 con 0x7f6000143ae0 2024-01-16T15:09:32.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.226+0000 7f5fff400640 1 -- 172.21.15.141:0/35264870 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0031b80 con 0x7f6000143ae0 2024-01-16T15:09:32.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.227+0000 7f6007a00640 1 -- 172.21.15.141:0/35264870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 msgr2=0x7f60000cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.227+0000 7f6007a00640 1 --2- 172.21.15.141:0/35264870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000cf290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00097a0 tx=0x7f5ff002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.228+0000 7f6007a00640 1 -- 172.21.15.141:0/35264870 shutdown_connections 2024-01-16T15:09:32.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.228+0000 7f6007a00640 1 --2- 172.21.15.141:0/35264870 >> v2:172.21.15.188:3300/0 conn(0x7f600015e3b0 0x7f60000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.228+0000 7f6007a00640 1 --2- 172.21.15.141:0/35264870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000cf290 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.228+0000 7f6007a00640 1 -- 172.21.15.141:0/35264870 >> 172.21.15.141:0/35264870 conn(0x7f600006a800 msgr2=0x7f600006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.228+0000 7f6007a00640 1 -- 172.21.15.141:0/35264870 shutdown_connections 2024-01-16T15:09:32.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.228+0000 7f6007a00640 1 -- 172.21.15.141:0/35264870 wait complete. 2024-01-16T15:09:32.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.231+0000 7f6007a00640 1 Processor -- start 2024-01-16T15:09:32.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.231+0000 7f6007a00640 1 -- start start 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6007a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60000cf7d0 0x7f600013d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6007a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6007a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f600015e3b0 0x7f600013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6007a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60000ae7c0 con 0x7f60000cf7d0 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6007a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600013d8f0 con 0x7f600015e3b0 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6007a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600013dbd0 con 0x7f6000143ae0 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6005000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60000cf7d0 0x7f600013d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f5fffe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6005000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60000cf7d0 0x7f600013d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34390/0 (socket says 172.21.15.141:34390) 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6005000640 1 -- 172.21.15.141:0/1889874634 learned_addr learned my addr 172.21.15.141:0/1889874634 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.233+0000 7f6005a00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f600015e3b0 0x7f600013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.234+0000 7f6005000640 1 -- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 msgr2=0x7f60000ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.234+0000 7f6005000640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.234+0000 7f6005000640 1 -- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f600015e3b0 msgr2=0x7f600013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.234+0000 7f6005000640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f600015e3b0 0x7f600013c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.234+0000 7f6005000640 1 -- 172.21.15.141:0/1889874634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff800bc80 con 0x7f60000cf7d0 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.236+0000 7f6005000640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60000cf7d0 0x7f600013d310 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0000c00 tx=0x7f5ff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.236+0000 7f5ffcc00640 1 -- 172.21.15.141:0/1889874634 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0003820 con 0x7f60000cf7d0 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.236+0000 7f5ffcc00640 1 -- 172.21.15.141:0/1889874634 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff003f070 con 0x7f60000cf7d0 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.236+0000 7f5ffcc00640 1 -- 172.21.15.141:0/1889874634 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0003be0 con 0x7f60000cf7d0 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.236+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff0009400 con 0x7f60000cf7d0 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.236+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600013e190 con 0x7f60000cf7d0 2024-01-16T15:09:32.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.238+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60000675d0 con 0x7f60000cf7d0 2024-01-16T15:09:32.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.241+0000 7f5ffcc00640 1 -- 172.21.15.141:0/1889874634 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f5ff0003d40 con 0x7f60000cf7d0 2024-01-16T15:09:32.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.243+0000 7f5ffcc00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5fd806c110 0x7f5fd806e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.243+0000 7f5ffcc00640 1 -- 172.21.15.141:0/1889874634 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5ff0041070 con 0x7f60000cf7d0 2024-01-16T15:09:32.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.244+0000 7f5fffe00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5fd806c110 0x7f5fd806e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.245+0000 7f5fffe00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5fd806c110 0x7f5fd806e5d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6000140160 tx=0x7f5fe8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.248+0000 7f5ffcc00640 1 -- 172.21.15.141:0/1889874634 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ff00773e0 con 0x7f60000cf7d0 2024-01-16T15:09:32.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.508+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59634", "target": ["mon-mgr", ""]}) v1 -- 0x7f60000b2410 con 0x7f5fd806c110 2024-01-16T15:09:32.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.560+0000 7f5ffcc00640 1 -- 172.21.15.141:0/1889874634 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5fe8001ce0 con 0x7f5fd806c110 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5fd806c110 msgr2=0x7f5fd806e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5fd806c110 0x7f5fd806e5d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6000140160 tx=0x7f5fe8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60000cf7d0 msgr2=0x7f600013d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60000cf7d0 0x7f600013d310 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0000c00 tx=0x7f5ff00046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 shutdown_connections 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5fd806c110 0x7f5fd806e5d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f600015e3b0 0x7f600013c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6000143ae0 0x7f60000ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 --2- 172.21.15.141:0/1889874634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60000cf7d0 0x7f600013d310 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.563+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 >> 172.21.15.141:0/1889874634 conn(0x7f600006a800 msgr2=0x7f600015c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.564+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 shutdown_connections 2024-01-16T15:09:32.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.564+0000 7f6007a00640 1 -- 172.21.15.141:0/1889874634 wait complete. 2024-01-16T15:09:32.561 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59631 2024-01-16T15:09:32.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.788+0000 7fe34d200640 1 Processor -- start 2024-01-16T15:09:32.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.789+0000 7fe34d200640 1 -- start start 2024-01-16T15:09:32.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe34d200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe348159590 0x7fe348070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe34d200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe34806d050 0x7fe34806d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe34d200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe348071790 con 0x7fe3480b3c20 2024-01-16T15:09:32.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe34d200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34806d990 con 0x7fe348159590 2024-01-16T15:09:32.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe34d200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe348071100 con 0x7fe34806d050 2024-01-16T15:09:32.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe346a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe3480b3c20 0x7fe3480ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55618/0 (socket says 172.21.15.141:55618) 2024-01-16T15:09:32.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe347400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe348159590 0x7fe348070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.790+0000 7fe346a00640 1 -- 172.21.15.141:0/3516632114 learned_addr learned my addr 172.21.15.141:0/3516632114 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:32.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.791+0000 7fe347e00640 1 --2- 172.21.15.141:0/3516632114 >> v2:172.21.15.188:3300/0 conn(0x7fe34806d050 0x7fe34806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.791+0000 7fe347400640 1 -- 172.21.15.141:0/3516632114 >> v2:172.21.15.188:3300/0 conn(0x7fe34806d050 msgr2=0x7fe34806d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:32.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.791+0000 7fe347400640 1 --2- 172.21.15.141:0/3516632114 >> v2:172.21.15.188:3300/0 conn(0x7fe34806d050 0x7fe34806d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.791+0000 7fe347400640 1 -- 172.21.15.141:0/3516632114 >> v1:172.21.15.141:6789/0 conn(0x7fe3480b3c20 legacy=0x7fe3480ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.791+0000 7fe347400640 1 -- 172.21.15.141:0/3516632114 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe348070860 con 0x7fe348159590 2024-01-16T15:09:32.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.791+0000 7fe346000640 1 -- 172.21.15.141:0/3516632114 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097355672 0 0) 0x7fe348071790 con 0x7fe3480b3c20 2024-01-16T15:09:32.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.792+0000 7fe347400640 1 --2- 172.21.15.141:0/3516632114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348070160 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe3480b3770 tx=0x7fe33802cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e1caeefe20b406 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.792+0000 7fe346000640 1 -- 172.21.15.141:0/3516632114 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33802dad0 con 0x7fe348159590 2024-01-16T15:09:32.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.792+0000 7fe346000640 1 -- 172.21.15.141:0/3516632114 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33802dc30 con 0x7fe348159590 2024-01-16T15:09:32.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.792+0000 7fe346000640 1 -- 172.21.15.141:0/3516632114 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338035440 con 0x7fe348159590 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 -- 172.21.15.141:0/3516632114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 msgr2=0x7fe348070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 --2- 172.21.15.141:0/3516632114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348070160 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe3480b3770 tx=0x7fe33802cd20 comp rx=0 tx=0).stop 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 -- 172.21.15.141:0/3516632114 shutdown_connections 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 --2- 172.21.15.141:0/3516632114 >> v2:172.21.15.188:3300/0 conn(0x7fe34806d050 0x7fe34806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 --2- 172.21.15.141:0/3516632114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348070160 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 -- 172.21.15.141:0/3516632114 >> 172.21.15.141:0/3516632114 conn(0x7fe34806a800 msgr2=0x7fe34806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 -- 172.21.15.141:0/3516632114 shutdown_connections 2024-01-16T15:09:32.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.793+0000 7fe34d200640 1 -- 172.21.15.141:0/3516632114 wait complete. 2024-01-16T15:09:32.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.795+0000 7fe34d200640 1 Processor -- start 2024-01-16T15:09:32.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.795+0000 7fe34d200640 1 -- start start 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe34d200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe34806d050 0x7fe3480a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe34d200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3480b3c20 0x7fe34814ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe34d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe34d200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34814ef70 con 0x7fe348159590 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe34d200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3481581a0 con 0x7fe3480b3c20 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe34d200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480aa150 con 0x7fe34806d050 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe347e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe347e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348157c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34402/0 (socket says 172.21.15.141:34402) 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.796+0000 7fe347e00640 1 -- 172.21.15.141:0/587454572 learned_addr learned my addr 172.21.15.141:0/587454572 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347400640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe34806d050 0x7fe3480a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe346a00640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3480b3c20 0x7fe34814ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347e00640 1 -- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe34806d050 msgr2=0x7fe3480a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347e00640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe34806d050 0x7fe3480a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347e00640 1 -- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3480b3c20 msgr2=0x7fe34814ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347e00640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3480b3c20 0x7fe34814ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347e00640 1 -- 172.21.15.141:0/587454572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33800a5a0 con 0x7fe348159590 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe346a00640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3480b3c20 0x7fe34814ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347400640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe34806d050 0x7fe3480a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe347e00640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348157c60 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fe340007e60 tx=0x7fe34000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe33f400640 1 -- 172.21.15.141:0/587454572 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340014420 con 0x7fe348159590 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe33f400640 1 -- 172.21.15.141:0/587454572 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400145a0 con 0x7fe348159590 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe33f400640 1 -- 172.21.15.141:0/587454572 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340014950 con 0x7fe348159590 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.797+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3480a95d0 con 0x7fe348159590 2024-01-16T15:09:32.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.798+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3480a9b90 con 0x7fe348159590 2024-01-16T15:09:32.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.799+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe348067450 con 0x7fe348159590 2024-01-16T15:09:32.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.800+0000 7fe33f400640 1 -- 172.21.15.141:0/587454572 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fe340014ab0 con 0x7fe348159590 2024-01-16T15:09:32.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.800+0000 7fe33f400640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe32006c070 0x7fe32006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.801+0000 7fe33f400640 1 -- 172.21.15.141:0/587454572 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe3400891d0 con 0x7fe348159590 2024-01-16T15:09:32.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.801+0000 7fe347400640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe32006c070 0x7fe32006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.802+0000 7fe347400640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe32006c070 0x7fe32006e530 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe338037000 tx=0x7fe338031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:32.805+0000 7fe33f400640 1 -- 172.21.15.141:0/587454572 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3400557e0 con 0x7fe348159590 2024-01-16T15:09:33.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.023+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59631", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3480b1f30 con 0x7fe32006c070 2024-01-16T15:09:33.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.066+0000 7fe33f400640 1 -- 172.21.15.141:0/587454572 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe338035540 con 0x7fe32006c070 2024-01-16T15:09:33.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.069+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe32006c070 msgr2=0x7fe32006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.069+0000 7fe34d200640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe32006c070 0x7fe32006e530 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe338037000 tx=0x7fe338031000 comp rx=0 tx=0).stop 2024-01-16T15:09:33.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.069+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 msgr2=0x7fe348157c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.069+0000 7fe34d200640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348157c60 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fe340007e60 tx=0x7fe34000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 shutdown_connections 2024-01-16T15:09:33.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe32006c070 0x7fe32006e530 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe348159590 0x7fe348157c60 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3480b3c20 0x7fe34814ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 --2- 172.21.15.141:0/587454572 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe34806d050 0x7fe3480a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 >> 172.21.15.141:0/587454572 conn(0x7fe34806a800 msgr2=0x7fe348071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 shutdown_connections 2024-01-16T15:09:33.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.070+0000 7fe34d200640 1 -- 172.21.15.141:0/587454572 wait complete. 2024-01-16T15:09:33.066 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:09:33.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.355+0000 7ffac0000640 1 Processor -- start 2024-01-16T15:09:33.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.355+0000 7ffac0000640 1 -- start start 2024-01-16T15:09:33.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.357+0000 7ffac0000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ffab8160f80 0x7ffab8161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.357+0000 7ffac0000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab80ced50 0x7ffab80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.357+0000 7ffac0000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab81618a0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.357+0000 7ffac0000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab80cf1e0 con 0x7ffab8160f80 2024-01-16T15:09:33.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.357+0000 7ffac0000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab80d0de0 con 0x7ffab80ced50 2024-01-16T15:09:33.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.358+0000 7ffabcc00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7ffab80ce8f0 0x7ffab80cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:55084/0 (socket says 172.21.15.141:55084) 2024-01-16T15:09:33.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.358+0000 7ffabd600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ffab8160f80 0x7ffab8161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.358+0000 7ffabd600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ffab8160f80 0x7ffab8161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:38250/0 (socket says 172.21.15.141:38250) 2024-01-16T15:09:33.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.358+0000 7ffabd600640 1 -- 172.21.15.141:0/279238687 learned_addr learned my addr 172.21.15.141:0/279238687 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:33.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.358+0000 7ffabe000640 1 --2- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab80ced50 0x7ffab80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.359+0000 7ffabd600640 1 -- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab80ced50 msgr2=0x7ffab80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:33.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.359+0000 7ffabd600640 1 --2- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab80ced50 0x7ffab80d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.360+0000 7ffabd600640 1 -- 172.21.15.141:0/279238687 >> v1:172.21.15.188:6789/0 conn(0x7ffab80ce8f0 legacy=0x7ffab80cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.360+0000 7ffabd600640 1 -- 172.21.15.141:0/279238687 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab80d10c0 con 0x7ffab8160f80 2024-01-16T15:09:33.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.363+0000 7ffabd600640 1 --2- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab8160f80 0x7ffab8161360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009870 tx=0x7ffaa802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=426cd3dbe3dfdaae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.364+0000 7ffab7e00640 1 -- 172.21.15.141:0/279238687 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8030840 con 0x7ffab8160f80 2024-01-16T15:09:33.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.364+0000 7ffab7e00640 1 -- 172.21.15.141:0/279238687 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa80309a0 con 0x7ffab8160f80 2024-01-16T15:09:33.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.364+0000 7ffab7e00640 1 -- 172.21.15.141:0/279238687 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8030d30 con 0x7ffab8160f80 2024-01-16T15:09:33.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 -- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab8160f80 msgr2=0x7ffab8161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 --2- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab8160f80 0x7ffab8161360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009870 tx=0x7ffaa802b980 comp rx=0 tx=0).stop 2024-01-16T15:09:33.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 -- 172.21.15.141:0/279238687 shutdown_connections 2024-01-16T15:09:33.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 --2- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab80ced50 0x7ffab80d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 --2- 172.21.15.141:0/279238687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab8160f80 0x7ffab8161360 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 -- 172.21.15.141:0/279238687 >> 172.21.15.141:0/279238687 conn(0x7ffab809c550 msgr2=0x7ffab80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 -- 172.21.15.141:0/279238687 shutdown_connections 2024-01-16T15:09:33.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.366+0000 7ffac0000640 1 -- 172.21.15.141:0/279238687 wait complete. 2024-01-16T15:09:33.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.371+0000 7ffac0000640 1 Processor -- start 2024-01-16T15:09:33.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.372+0000 7ffac0000640 1 -- start start 2024-01-16T15:09:33.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.374+0000 7ffac0000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab80ce8f0 0x7ffab814c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.374+0000 7ffac0000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffab80ced50 0x7ffab80b7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.374+0000 7ffac0000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab8160f80 0x7ffab80b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffac0000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab815d6c0 con 0x7ffab80ced50 2024-01-16T15:09:33.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffac0000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab8150290 con 0x7ffab80ce8f0 2024-01-16T15:09:33.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffac0000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab815d910 con 0x7ffab8160f80 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffabd600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab80ce8f0 0x7ffab814c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffabd600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab80ce8f0 0x7ffab814c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:38264/0 (socket says 172.21.15.141:38264) 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffabd600640 1 -- 172.21.15.141:0/2704766696 learned_addr learned my addr 172.21.15.141:0/2704766696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffabe000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab8160f80 0x7ffab80b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.375+0000 7ffabcc00640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffab80ced50 0x7ffab80b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabd600640 1 -- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab8160f80 msgr2=0x7ffab80b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabd600640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab8160f80 0x7ffab80b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabd600640 1 -- 172.21.15.141:0/2704766696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffab80ced50 msgr2=0x7ffab80b7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabd600640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffab80ced50 0x7ffab80b7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabd600640 1 -- 172.21.15.141:0/2704766696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa80094d0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabe000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab8160f80 0x7ffab80b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabcc00640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffab80ced50 0x7ffab80b7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.376+0000 7ffabd600640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab80ce8f0 0x7ffab814c9f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8000c00 tx=0x7ffaa8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.377+0000 7ffab5600640 1 -- 172.21.15.141:0/2704766696 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8039a90 con 0x7ffab80ce8f0 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.377+0000 7ffab5600640 1 -- 172.21.15.141:0/2704766696 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8039c10 con 0x7ffab80ce8f0 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.377+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab815dbf0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.377+0000 7ffab5600640 1 -- 172.21.15.141:0/2704766696 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8043410 con 0x7ffab80ce8f0 2024-01-16T15:09:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.377+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab815e1b0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.379+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab81690e0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.381+0000 7ffab5600640 1 -- 172.21.15.141:0/2704766696 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7ffaa804c3b0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.383+0000 7ffab5600640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9006c110 0x7ffa9006e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.383+0000 7ffab5600640 1 -- 172.21.15.141:0/2704766696 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffaa80aacf0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.383+0000 7ffabcc00640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9006c110 0x7ffa9006e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.385+0000 7ffabcc00640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9006c110 0x7ffa9006e5d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0007fd0 tx=0x7ffaa0007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.394+0000 7ffab5600640 1 -- 172.21.15.141:0/2704766696 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffaa80773b0 con 0x7ffab80ce8f0 2024-01-16T15:09:33.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.623+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffab80d1960 con 0x7ffa9006c110 2024-01-16T15:09:33.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.662+0000 7ffab5600640 1 -- 172.21.15.141:0/2704766696 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ffaa00075d0 con 0x7ffa9006c110 2024-01-16T15:09:33.640 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:33.640 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:09:33.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:33.640 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1" 2024-01-16T15:09:33.640 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:33.640 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:09:33.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.664+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9006c110 msgr2=0x7ffa9006e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.664+0000 7ffac0000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9006c110 0x7ffa9006e5d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0007fd0 tx=0x7ffaa0007d40 comp rx=0 tx=0).stop 2024-01-16T15:09:33.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab80ce8f0 msgr2=0x7ffab814c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab80ce8f0 0x7ffab814c9f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8000c00 tx=0x7ffaa8004730 comp rx=0 tx=0).stop 2024-01-16T15:09:33.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 shutdown_connections 2024-01-16T15:09:33.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9006c110 0x7ffa9006e5d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffab8160f80 0x7ffab80b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffab80ced50 0x7ffab80b7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 --2- 172.21.15.141:0/2704766696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffab80ce8f0 0x7ffab814c9f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 >> 172.21.15.141:0/2704766696 conn(0x7ffab809c550 msgr2=0x7ffab80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 shutdown_connections 2024-01-16T15:09:33.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.665+0000 7ffac0000640 1 -- 172.21.15.141:0/2704766696 wait complete. 2024-01-16T15:09:33.660 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:09:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.898+0000 7fae45400640 1 Processor -- start 2024-01-16T15:09:33.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.899+0000 7fae45400640 1 -- start start 2024-01-16T15:09:33.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae45400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b3c10 0x7fae400713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae45400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fae400ce420 0x7fae400ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae45400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae400cdf90 con 0x7fae400b4280 2024-01-16T15:09:33.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae45400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae40072000 con 0x7fae400ce420 2024-01-16T15:09:33.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae45400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae40071d20 con 0x7fae400b3c10 2024-01-16T15:09:33.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae3e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fae400b4280 0x7fae400b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58166/0 (socket says 172.21.15.141:58166) 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae3e800640 1 -- 172.21.15.141:0/1784881469 learned_addr learned my addr 172.21.15.141:0/1784881469 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.900+0000 7fae3f200640 1 --2- 172.21.15.141:0/1784881469 >> v2:172.21.15.141:3300/0 conn(0x7fae400ce420 0x7fae400ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.901+0000 7fae3de00640 1 --2- 172.21.15.141:0/1784881469 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b3c10 0x7fae400713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.901+0000 7fae3d400640 1 -- 172.21.15.141:0/1784881469 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447954580 0 0) 0x7fae400cdf90 con 0x7fae400b4280 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.901+0000 7fae3d400640 1 -- 172.21.15.141:0/1784881469 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae20003520 con 0x7fae400b4280 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.901+0000 7fae3f200640 1 -- 172.21.15.141:0/1784881469 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b3c10 msgr2=0x7fae400713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.901+0000 7fae3f200640 1 --2- 172.21.15.141:0/1784881469 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b3c10 0x7fae400713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.901+0000 7fae3f200640 1 -- 172.21.15.141:0/1784881469 >> v1:172.21.15.141:6789/0 conn(0x7fae400b4280 legacy=0x7fae400b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.901+0000 7fae3f200640 1 -- 172.21.15.141:0/1784881469 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae400cd940 con 0x7fae400ce420 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.903+0000 7fae3f200640 1 --2- 172.21.15.141:0/1784881469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae400ce8d0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fae38009dc0 tx=0x7fae3802be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed941da4f7ffc687 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.903+0000 7fae3d400640 1 -- 172.21.15.141:0/1784881469 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3802cc30 con 0x7fae400ce420 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.903+0000 7fae3d400640 1 -- 172.21.15.141:0/1784881469 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3802cd90 con 0x7fae400ce420 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.903+0000 7fae3d400640 1 -- 172.21.15.141:0/1784881469 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38034640 con 0x7fae400ce420 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.903+0000 7fae45400640 1 -- 172.21.15.141:0/1784881469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 msgr2=0x7fae400ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.903+0000 7fae45400640 1 --2- 172.21.15.141:0/1784881469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae400ce8d0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fae38009dc0 tx=0x7fae3802be80 comp rx=0 tx=0).stop 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.904+0000 7fae45400640 1 -- 172.21.15.141:0/1784881469 shutdown_connections 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.904+0000 7fae45400640 1 --2- 172.21.15.141:0/1784881469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae400ce8d0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.904+0000 7fae45400640 1 --2- 172.21.15.141:0/1784881469 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b3c10 0x7fae400713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.904+0000 7fae45400640 1 -- 172.21.15.141:0/1784881469 >> 172.21.15.141:0/1784881469 conn(0x7fae400cfef0 msgr2=0x7fae400d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.904+0000 7fae45400640 1 -- 172.21.15.141:0/1784881469 shutdown_connections 2024-01-16T15:09:33.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.904+0000 7fae45400640 1 -- 172.21.15.141:0/1784881469 wait complete. 2024-01-16T15:09:33.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.905+0000 7fae45400640 1 Processor -- start 2024-01-16T15:09:33.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.906+0000 7fae45400640 1 -- start start 2024-01-16T15:09:33.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.906+0000 7fae45400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae400b3c10 0x7fae40155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.906+0000 7fae45400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b4280 0x7fae401583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.906+0000 7fae45400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae40153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.906+0000 7fae45400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40158dd0 con 0x7fae400ce420 2024-01-16T15:09:33.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.906+0000 7fae45400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae401590b0 con 0x7fae400b3c10 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.906+0000 7fae45400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae40159390 con 0x7fae400b4280 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae40153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae40153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42310/0 (socket says 172.21.15.141:42310) 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 -- 172.21.15.141:0/278334022 learned_addr learned my addr 172.21.15.141:0/278334022 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3de00640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b4280 0x7fae401583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3e800640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae400b3c10 0x7fae40155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 -- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b4280 msgr2=0x7fae401583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b4280 0x7fae401583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 -- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae400b3c10 msgr2=0x7fae40155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae400b3c10 0x7fae40155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3f200640 1 -- 172.21.15.141:0/278334022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009a70 con 0x7fae400ce420 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.907+0000 7fae3de00640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b4280 0x7fae401583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:33.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.908+0000 7fae3f200640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae40153f10 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fae3802c490 tx=0x7fae38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.908+0000 7fae36a00640 1 -- 172.21.15.141:0/278334022 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3803e610 con 0x7fae400ce420 2024-01-16T15:09:33.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.908+0000 7fae36a00640 1 -- 172.21.15.141:0/278334022 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3803e770 con 0x7fae400ce420 2024-01-16T15:09:33.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.908+0000 7fae36a00640 1 -- 172.21.15.141:0/278334022 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3803eb60 con 0x7fae400ce420 2024-01-16T15:09:33.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.908+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae40159670 con 0x7fae400ce420 2024-01-16T15:09:33.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.908+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae401558b0 con 0x7fae400ce420 2024-01-16T15:09:33.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.909+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae40159b90 con 0x7fae400ce420 2024-01-16T15:09:33.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.910+0000 7fae36a00640 1 -- 172.21.15.141:0/278334022 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fae3803b050 con 0x7fae400ce420 2024-01-16T15:09:33.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.911+0000 7fae36a00640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae1806c070 0x7fae1806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:33.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.911+0000 7fae36a00640 1 -- 172.21.15.141:0/278334022 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fae380aaa60 con 0x7fae400ce420 2024-01-16T15:09:33.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.912+0000 7fae3e800640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae1806c070 0x7fae1806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:33.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.913+0000 7fae3e800640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae1806c070 0x7fae1806e530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fae30000f60 tx=0x7fae30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:33.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:33.915+0000 7fae36a00640 1 -- 172.21.15.141:0/278334022 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae40159b90 con 0x7fae400ce420 2024-01-16T15:09:34.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.142+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae40071910 con 0x7fae1806c070 2024-01-16T15:09:34.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.226+0000 7fae36a00640 1 -- 172.21.15.141:0/278334022 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fae300058d0 con 0x7fae1806c070 2024-01-16T15:09:34.204 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:34.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.230+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae1806c070 msgr2=0x7fae1806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.230+0000 7fae45400640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae1806c070 0x7fae1806e530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fae30000f60 tx=0x7fae30009040 comp rx=0 tx=0).stop 2024-01-16T15:09:34.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.230+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 msgr2=0x7fae40153f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.230+0000 7fae45400640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae40153f10 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fae3802c490 tx=0x7fae38004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 shutdown_connections 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fae1806c070 0x7fae1806e530 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae400ce420 0x7fae40153f10 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fae400b4280 0x7fae401583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 --2- 172.21.15.141:0/278334022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae400b3c10 0x7fae40155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 >> 172.21.15.141:0/278334022 conn(0x7fae400cfef0 msgr2=0x7fae4006e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 shutdown_connections 2024-01-16T15:09:34.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.231+0000 7fae45400640 1 -- 172.21.15.141:0/278334022 wait complete. 2024-01-16T15:09:34.236 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:09:34.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.539+0000 7f4dda800640 1 Processor -- start 2024-01-16T15:09:34.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.539+0000 7f4dda800640 1 -- start start 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.541+0000 7f4dda800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.541+0000 7f4dda800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4dd406c6b0 0x7f4dd4099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.541+0000 7f4dda800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd406cea0 con 0x7f4dd4099fa0 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.541+0000 7f4dda800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd409a430 con 0x7f4dd406c6b0 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.541+0000 7f4dda800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd406cbc0 con 0x7f4dd40cd240 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.542+0000 7f4dd3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4dd4099fa0 0x7f4dd406b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58180/0 (socket says 172.21.15.141:58180) 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.542+0000 7f4dd3e00640 1 -- 172.21.15.141:0/531075443 learned_addr learned my addr 172.21.15.141:0/531075443 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.542+0000 7f4dd3400640 1 --2- 172.21.15.141:0/531075443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.542+0000 7f4dd2a00640 1 --2- 172.21.15.141:0/531075443 >> v2:172.21.15.188:3300/0 conn(0x7f4dd406c6b0 0x7f4dd4099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.542+0000 7f4dd3400640 1 -- 172.21.15.141:0/531075443 >> v2:172.21.15.188:3300/0 conn(0x7f4dd406c6b0 msgr2=0x7f4dd4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:34.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.542+0000 7f4dd3400640 1 --2- 172.21.15.141:0/531075443 >> v2:172.21.15.188:3300/0 conn(0x7f4dd406c6b0 0x7f4dd4099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.543+0000 7f4dd3400640 1 -- 172.21.15.141:0/531075443 >> v1:172.21.15.141:6789/0 conn(0x7f4dd4099fa0 legacy=0x7f4dd406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.543+0000 7f4dd2000640 1 -- 172.21.15.141:0/531075443 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797120207 0 0) 0x7f4dd406cea0 con 0x7f4dd4099fa0 2024-01-16T15:09:34.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.543+0000 7f4dd3400640 1 -- 172.21.15.141:0/531075443 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd406f350 con 0x7f4dd40cd240 2024-01-16T15:09:34.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.546+0000 7f4dd3400640 1 --2- 172.21.15.141:0/531075443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b3820 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40097a0 tx=0x7f4dc402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24780318f44758ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.547+0000 7f4dd2000640 1 -- 172.21.15.141:0/531075443 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc402d040 con 0x7f4dd40cd240 2024-01-16T15:09:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.548+0000 7f4dd2000640 1 -- 172.21.15.141:0/531075443 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4031810 con 0x7f4dd40cd240 2024-01-16T15:09:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.548+0000 7f4dd2000640 1 -- 172.21.15.141:0/531075443 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4031ba0 con 0x7f4dd40cd240 2024-01-16T15:09:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.549+0000 7f4dda800640 1 -- 172.21.15.141:0/531075443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 msgr2=0x7f4dd40b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.549+0000 7f4dda800640 1 --2- 172.21.15.141:0/531075443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b3820 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40097a0 tx=0x7f4dc402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.551+0000 7f4dda800640 1 -- 172.21.15.141:0/531075443 shutdown_connections 2024-01-16T15:09:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.551+0000 7f4dda800640 1 --2- 172.21.15.141:0/531075443 >> v2:172.21.15.188:3300/0 conn(0x7f4dd406c6b0 0x7f4dd4099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.551+0000 7f4dda800640 1 --2- 172.21.15.141:0/531075443 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b3820 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.551+0000 7f4dda800640 1 -- 172.21.15.141:0/531075443 >> 172.21.15.141:0/531075443 conn(0x7f4dd40b2d00 msgr2=0x7f4dd40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.551+0000 7f4dda800640 1 -- 172.21.15.141:0/531075443 shutdown_connections 2024-01-16T15:09:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.551+0000 7f4dda800640 1 -- 172.21.15.141:0/531075443 wait complete. 2024-01-16T15:09:34.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.554+0000 7f4dda800640 1 Processor -- start 2024-01-16T15:09:34.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.554+0000 7f4dda800640 1 -- start start 2024-01-16T15:09:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dda800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dd406c6b0 0x7f4dd41555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dda800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd4099fa0 0x7f4dd40b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dda800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dda800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b6690 con 0x7f4dd406c6b0 2024-01-16T15:09:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dda800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b6ab0 con 0x7f4dd4099fa0 2024-01-16T15:09:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dda800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b5a10 con 0x7f4dd40cd240 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dd3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dd406c6b0 0x7f4dd41555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dd3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dd406c6b0 0x7f4dd41555c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42316/0 (socket says 172.21.15.141:42316) 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dd3400640 1 -- 172.21.15.141:0/1421272929 learned_addr learned my addr 172.21.15.141:0/1421272929 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.556+0000 7f4dd3e00640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.557+0000 7f4dd2a00640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd4099fa0 0x7f4dd40b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.557+0000 7f4dd3400640 1 -- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 msgr2=0x7f4dd40b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.557+0000 7f4dd3400640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.557+0000 7f4dd3400640 1 -- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd4099fa0 msgr2=0x7f4dd40b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.557+0000 7f4dd3400640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd4099fa0 0x7f4dd40b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.557+0000 7f4dd3400640 1 -- 172.21.15.141:0/1421272929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc4009400 con 0x7f4dd406c6b0 2024-01-16T15:09:34.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.558+0000 7f4dd3400640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dd406c6b0 0x7f4dd41555c0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4dc402bec0 tx=0x7f4dc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.558+0000 7f4dcb400640 1 -- 172.21.15.141:0/1421272929 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc40423f0 con 0x7f4dd406c6b0 2024-01-16T15:09:34.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.558+0000 7f4dcb400640 1 -- 172.21.15.141:0/1421272929 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4042570 con 0x7f4dd406c6b0 2024-01-16T15:09:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.558+0000 7f4dcb400640 1 -- 172.21.15.141:0/1421272929 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4042920 con 0x7f4dd406c6b0 2024-01-16T15:09:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.558+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd40b5cf0 con 0x7f4dd406c6b0 2024-01-16T15:09:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.558+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd40b62b0 con 0x7f4dd406c6b0 2024-01-16T15:09:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.560+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd40b6900 con 0x7f4dd406c6b0 2024-01-16T15:09:34.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.563+0000 7f4dcb400640 1 -- 172.21.15.141:0/1421272929 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f4dc4042a80 con 0x7f4dd406c6b0 2024-01-16T15:09:34.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.564+0000 7f4dcb400640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4dac06c0c0 0x7f4dac06e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:34.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.564+0000 7f4dcb400640 1 -- 172.21.15.141:0/1421272929 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4dc40aa6f0 con 0x7f4dd406c6b0 2024-01-16T15:09:34.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.567+0000 7f4dd2a00640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4dac06c0c0 0x7f4dac06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:34.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.568+0000 7f4dd2a00640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4dac06c0c0 0x7f4dac06e580 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4099ee0 tx=0x7f4dbc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:34.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.573+0000 7f4dcb400640 1 -- 172.21.15.141:0/1421272929 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4dc4076df0 con 0x7f4dd406c6b0 2024-01-16T15:09:34.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.818+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dd40cd620 con 0x7f4dac06c0c0 2024-01-16T15:09:34.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.865+0000 7f4dcb400640 1 -- 172.21.15.141:0/1421272929 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4dbc004020 con 0x7f4dac06c0c0 2024-01-16T15:09:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.867+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4dac06c0c0 msgr2=0x7f4dac06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.867+0000 7f4dda800640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4dac06c0c0 0x7f4dac06e580 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4099ee0 tx=0x7f4dbc009190 comp rx=0 tx=0).stop 2024-01-16T15:09:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.867+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dd406c6b0 msgr2=0x7f4dd41555c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dd406c6b0 0x7f4dd41555c0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4dc402bec0 tx=0x7f4dc40046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 shutdown_connections 2024-01-16T15:09:34.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4dac06c0c0 0x7f4dac06e580 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4dd40cd240 0x7f4dd40b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dd4099fa0 0x7f4dd40b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 --2- 172.21.15.141:0/1421272929 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4dd406c6b0 0x7f4dd41555c0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:34.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 >> 172.21.15.141:0/1421272929 conn(0x7f4dd40b2d00 msgr2=0x7f4dd40b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:34.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 shutdown_connections 2024-01-16T15:09:34.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:34.868+0000 7f4dda800640 1 -- 172.21.15.141:0/1421272929 wait complete. 2024-01-16T15:09:34.864 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59638 2024-01-16T15:09:35.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.070+0000 7f3083800640 1 Processor -- start 2024-01-16T15:09:35.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.070+0000 7f3083800640 1 -- start start 2024-01-16T15:09:35.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f3083800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f307c15e3b0 0x7f307c164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f3083800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0b1ee0 con 0x7f307c0cf9a0 2024-01-16T15:09:35.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f3083800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0b2780 con 0x7f307c143ae0 2024-01-16T15:09:35.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f3083800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c15e990 con 0x7f307c15e3b0 2024-01-16T15:09:35.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f3073e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f307c0cf9a0 0x7f307c15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58188/0 (socket says 172.21.15.141:58188) 2024-01-16T15:09:35.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f3073e00640 1 -- 172.21.15.141:0/3546697459 learned_addr learned my addr 172.21.15.141:0/3546697459 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424672385 0 0) 0x7f307c0b1ee0 con 0x7f307c0cf9a0 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.072+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f305c003520 con 0x7f307c0cf9a0 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f3081800640 1 --2- 172.21.15.141:0/3546697459 >> v2:172.21.15.188:3300/0 conn(0x7f307c15e3b0 0x7f307c164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 353579658 0 0) 0x7f305c003520 con 0x7f307c0cf9a0 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 >> v2:172.21.15.188:3300/0 conn(0x7f307c15e3b0 msgr2=0x7f307c164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 --2- 172.21.15.141:0/3546697459 >> v2:172.21.15.188:3300/0 conn(0x7f307c15e3b0 0x7f307c164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 >> v1:172.21.15.188:6789/0 conn(0x7f307c143ae0 legacy=0x7f307c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c0b2130 con 0x7f307c0cf9a0 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3064001670 con 0x7f307c0cf9a0 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 >> v1:172.21.15.141:6789/0 conn(0x7f307c0cf9a0 legacy=0x7f307c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0039c0 0x7f305c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f305c0051d0 0x7f305c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c15e990 con 0x7f305c0047c0 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c00c100 con 0x7f305c0051d0 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c00c2c0 con 0x7f305c0039c0 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.073+0000 7f3073e00640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3081800640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f305c0051d0 0x7f305c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3073e00640 1 -- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0039c0 msgr2=0x7f305c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3073e00640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0039c0 0x7f305c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3073e00640 1 -- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f305c0051d0 msgr2=0x7f305c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3073e00640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f305c0051d0 0x7f305c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3073e00640 1 -- 172.21.15.141:0/3546697459 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c003520 con 0x7f305c0047c0 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3081800640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f305c0051d0 0x7f305c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.074+0000 7f3081800640 1 -- 172.21.15.141:0/3546697459 reap_dead start 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.075+0000 7f3073e00640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f306400c160 tx=0x7f305c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e320fa2b878fcaf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.075+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064005ad0 con 0x7f305c0047c0 2024-01-16T15:09:35.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.075+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064005c30 con 0x7f305c0047c0 2024-01-16T15:09:35.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.075+0000 7f307be00640 1 -- 172.21.15.141:0/3546697459 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306402c7e0 con 0x7f305c0047c0 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.075+0000 7f3083800640 1 -- 172.21.15.141:0/3546697459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f305c0047c0 msgr2=0x7f305c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.075+0000 7f3083800640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f306400c160 tx=0x7f305c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.076+0000 7f3083800640 1 -- 172.21.15.141:0/3546697459 shutdown_connections 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.076+0000 7f3083800640 1 --2- 172.21.15.141:0/3546697459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f305c0047c0 0x7f305c004c40 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.076+0000 7f3083800640 1 -- 172.21.15.141:0/3546697459 >> 172.21.15.141:0/3546697459 conn(0x7f307c06a800 msgr2=0x7f307c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.076+0000 7f3083800640 1 -- 172.21.15.141:0/3546697459 shutdown_connections 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.076+0000 7f3083800640 1 -- 172.21.15.141:0/3546697459 wait complete. 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.077+0000 7f3083800640 1 Processor -- start 2024-01-16T15:09:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.078+0000 7f3083800640 1 -- start start 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.078+0000 7f3083800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0047c0 0x7f307c13dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.078+0000 7f3083800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f307c15d080 0x7f307c159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.078+0000 7f3083800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f307c166cb0 0x7f307c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.078+0000 7f3083800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c15a000 con 0x7f307c15d080 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.078+0000 7f3083800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c142bf0 con 0x7f307c166cb0 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.078+0000 7f3083800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c166820 con 0x7f305c0047c0 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f307c15d080 0x7f307c159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f307c15d080 0x7f307c159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42346/0 (socket says 172.21.15.141:42346) 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 -- 172.21.15.141:0/1556474689 learned_addr learned my addr 172.21.15.141:0/1556474689 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3080e00640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0047c0 0x7f307c13dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 -- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0047c0 msgr2=0x7f307c13dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0047c0 0x7f307c13dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 -- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f307c166cb0 msgr2=0x7f307c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f307c166cb0 0x7f307c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 -- 172.21.15.141:0/1556474689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c142e40 con 0x7f307c15d080 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.079+0000 7f3073e00640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f307c15d080 0x7f307c159ac0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f306400c160 tx=0x7f306402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.080+0000 7f3079600640 1 -- 172.21.15.141:0/1556474689 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306403c040 con 0x7f307c15d080 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.080+0000 7f3079600640 1 -- 172.21.15.141:0/1556474689 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306402ccd0 con 0x7f307c15d080 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.080+0000 7f3079600640 1 -- 172.21.15.141:0/1556474689 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30640072b0 con 0x7f307c15d080 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.080+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c143120 con 0x7f307c15d080 2024-01-16T15:09:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.080+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c167130 con 0x7f307c15d080 2024-01-16T15:09:35.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.081+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c0675d0 con 0x7f307c15d080 2024-01-16T15:09:35.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.082+0000 7f3079600640 1 -- 172.21.15.141:0/1556474689 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f3064032070 con 0x7f307c15d080 2024-01-16T15:09:35.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.083+0000 7f3079600640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f305406c070 0x7f305406e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.083+0000 7f3079600640 1 -- 172.21.15.141:0/1556474689 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f30640a9dd0 con 0x7f307c15d080 2024-01-16T15:09:35.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.083+0000 7f3080e00640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f305406c070 0x7f305406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.085+0000 7f3080e00640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f305406c070 0x7f305406e530 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f307c0affd0 tx=0x7f306c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.087+0000 7f3079600640 1 -- 172.21.15.141:0/1556474689 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30640764f0 con 0x7f307c15d080 2024-01-16T15:09:35.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.395+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59638", "target": ["mon-mgr", ""]}) v1 -- 0x7f307c15dca0 con 0x7f305406c070 2024-01-16T15:09:35.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.429+0000 7f3079600640 1 -- 172.21.15.141:0/1556474689 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f306c00d4a0 con 0x7f305406c070 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f305406c070 msgr2=0x7f305406e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f305406c070 0x7f305406e530 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f307c0affd0 tx=0x7f306c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f307c15d080 msgr2=0x7f307c159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f307c15d080 0x7f307c159ac0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f306400c160 tx=0x7f306402e070 comp rx=0 tx=0).stop 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 shutdown_connections 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f305406c070 0x7f305406e530 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f307c166cb0 0x7f307c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f307c15d080 0x7f307c159ac0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 --2- 172.21.15.141:0/1556474689 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f305c0047c0 0x7f307c13dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 >> 172.21.15.141:0/1556474689 conn(0x7f307c06a800 msgr2=0x7f307c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 shutdown_connections 2024-01-16T15:09:35.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.433+0000 7f3083800640 1 -- 172.21.15.141:0/1556474689 wait complete. 2024-01-16T15:09:35.438 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596311 2024-01-16T15:09:35.689 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:35.696 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:35.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.752+0000 7f1c9b200640 1 Processor -- start 2024-01-16T15:09:35.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.753+0000 7f1c9b200640 1 -- start start 2024-01-16T15:09:35.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.755+0000 7f1c9b200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1c9414b0a0 0x7f1c940b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.755+0000 7f1c9b200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1c94165d90 0x7f1c940b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.756+0000 7f1c9b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c940b4090 0x7f1c94161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.756+0000 7f1c9b200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c940b4520 con 0x7f1c9414b0a0 2024-01-16T15:09:35.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.756+0000 7f1c9b200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c940b3160 con 0x7f1c94165d90 2024-01-16T15:09:35.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.756+0000 7f1c9b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940b25e0 con 0x7f1c940b4090 2024-01-16T15:09:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1c9414b0a0 0x7f1c940b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1c94165d90 0x7f1c940b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1c94165d90 0x7f1c940b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:38322/0 (socket says 172.21.15.141:38322) 2024-01-16T15:09:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 -- 172.21.15.141:0/49487136 learned_addr learned my addr 172.21.15.141:0/49487136 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c99200640 1 --2- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c940b4090 0x7f1c94161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 -- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c940b4090 msgr2=0x7f1c94161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 --2- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c940b4090 0x7f1c94161bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 -- 172.21.15.141:0/49487136 >> v2:172.21.15.141:3300/0 conn(0x7f1c9414b0a0 msgr2=0x7f1c940b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 --2- 172.21.15.141:0/49487136 >> v2:172.21.15.141:3300/0 conn(0x7f1c9414b0a0 0x7f1c940b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.757+0000 7f1c93400640 1 -- 172.21.15.141:0/49487136 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c940b28c0 con 0x7f1c94165d90 2024-01-16T15:09:35.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.761+0000 7f1c93400640 1 --2- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c94165d90 0x7f1c940b3b50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1c9406fdd0 tx=0x7f1c7c02b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c95f8d761b189f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.762+0000 7f1c92a00640 1 -- 172.21.15.141:0/49487136 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c031880 con 0x7f1c94165d90 2024-01-16T15:09:35.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.762+0000 7f1c92a00640 1 -- 172.21.15.141:0/49487136 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c0319e0 con 0x7f1c94165d90 2024-01-16T15:09:35.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.762+0000 7f1c92a00640 1 -- 172.21.15.141:0/49487136 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c031d70 con 0x7f1c94165d90 2024-01-16T15:09:35.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 -- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c94165d90 msgr2=0x7f1c940b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 --2- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c94165d90 0x7f1c940b3b50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1c9406fdd0 tx=0x7f1c7c02b920 comp rx=0 tx=0).stop 2024-01-16T15:09:35.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 -- 172.21.15.141:0/49487136 shutdown_connections 2024-01-16T15:09:35.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 --2- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c940b4090 0x7f1c94161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 --2- 172.21.15.141:0/49487136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c94165d90 0x7f1c940b3b50 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 --2- 172.21.15.141:0/49487136 >> v2:172.21.15.141:3300/0 conn(0x7f1c9414b0a0 0x7f1c940b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 -- 172.21.15.141:0/49487136 >> 172.21.15.141:0/49487136 conn(0x7f1c9406c340 msgr2=0x7f1c9406c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.763+0000 7f1c9b200640 1 -- 172.21.15.141:0/49487136 shutdown_connections 2024-01-16T15:09:35.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.764+0000 7f1c9b200640 1 -- 172.21.15.141:0/49487136 wait complete. 2024-01-16T15:09:35.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.767+0000 7f1c9b200640 1 Processor -- start 2024-01-16T15:09:35.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.767+0000 7f1c9b200640 1 -- start start 2024-01-16T15:09:35.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.769+0000 7f1c9b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c940b4090 0x7f1c940c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.769+0000 7f1c9b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c9414b0a0 0x7f1c940a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.769+0000 7f1c9b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c94165d90 0x7f1c941fa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c9b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c94139540 con 0x7f1c940b4090 2024-01-16T15:09:35.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c9b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940ab010 con 0x7f1c9414b0a0 2024-01-16T15:09:35.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c9b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940aa770 con 0x7f1c94165d90 2024-01-16T15:09:35.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c99200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c94165d90 0x7f1c941fa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c99200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c94165d90 0x7f1c941fa720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:55692/0 (socket says 172.21.15.141:55692) 2024-01-16T15:09:35.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c99200640 1 -- 172.21.15.141:0/3931138017 learned_addr learned my addr 172.21.15.141:0/3931138017 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c93400640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c9414b0a0 0x7f1c940a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.770+0000 7f1c93e00640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c940b4090 0x7f1c940c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.771+0000 7f1c99200640 1 -- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c9414b0a0 msgr2=0x7f1c940a9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.771+0000 7f1c99200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c9414b0a0 0x7f1c940a9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.771+0000 7f1c99200640 1 -- 172.21.15.141:0/3931138017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c940b4090 msgr2=0x7f1c940c4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.771+0000 7f1c99200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c940b4090 0x7f1c940c4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.771+0000 7f1c99200640 1 -- 172.21.15.141:0/3931138017 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c0094d0 con 0x7f1c94165d90 2024-01-16T15:09:35.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.771+0000 7f1c93e00640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c940b4090 0x7f1c940c4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:35.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.771+0000 7f1c99200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c94165d90 0x7f1c941fa720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c007d10 tx=0x7f1c8c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.772+0000 7f1c93400640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c9414b0a0 0x7f1c940a9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:35.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.772+0000 7f1c8be00640 1 -- 172.21.15.141:0/3931138017 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c014420 con 0x7f1c94165d90 2024-01-16T15:09:35.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.772+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c940aaa50 con 0x7f1c94165d90 2024-01-16T15:09:35.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.772+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c940aa460 con 0x7f1c94165d90 2024-01-16T15:09:35.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.772+0000 7f1c8be00640 1 -- 172.21.15.141:0/3931138017 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8c014580 con 0x7f1c94165d90 2024-01-16T15:09:35.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.772+0000 7f1c8be00640 1 -- 172.21.15.141:0/3931138017 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c014420 con 0x7f1c94165d90 2024-01-16T15:09:35.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.774+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c64005250 con 0x7f1c94165d90 2024-01-16T15:09:35.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.778+0000 7f1c8be00640 1 -- 172.21.15.141:0/3931138017 <== mon.2 v2:172.21.15.188:3301/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f1c8c0146f0 con 0x7f1c94165d90 2024-01-16T15:09:35.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.779+0000 7f1c8be00640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c6c06c0d0 0x7f1c6c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.780+0000 7f1c8be00640 1 -- 172.21.15.141:0/3931138017 <== mon.2 v2:172.21.15.188:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1c8c0892e0 con 0x7f1c94165d90 2024-01-16T15:09:35.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.780+0000 7f1c93e00640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c6c06c0d0 0x7f1c6c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.782+0000 7f1c93e00640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c6c06c0d0 0x7f1c6c06e590 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1c84006fd0 tx=0x7f1c84008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:35.788+0000 7f1c8be00640 1 -- 172.21.15.141:0/3931138017 <== mon.2 v2:172.21.15.188:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c8c055930 con 0x7f1c94165d90 2024-01-16T15:09:36.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.163+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596311", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c64000bc0 con 0x7f1c6c06c0d0 2024-01-16T15:09:36.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.178+0000 7f1c8be00640 1 -- 172.21.15.141:0/3931138017 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1c64000bc0 con 0x7f1c6c06c0d0 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c6c06c0d0 msgr2=0x7f1c6c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c6c06c0d0 0x7f1c6c06e590 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1c84006fd0 tx=0x7f1c84008040 comp rx=0 tx=0).stop 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c94165d90 msgr2=0x7f1c941fa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c94165d90 0x7f1c941fa720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c007d10 tx=0x7f1c8c00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 shutdown_connections 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1c6c06c0d0 0x7f1c6c06e590 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1c94165d90 0x7f1c941fa720 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c9414b0a0 0x7f1c940a9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 --2- 172.21.15.141:0/3931138017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1c940b4090 0x7f1c940c4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 >> 172.21.15.141:0/3931138017 conn(0x7f1c9406c340 msgr2=0x7f1c941624a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 shutdown_connections 2024-01-16T15:09:36.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.181+0000 7f1c9b200640 1 -- 172.21.15.141:0/3931138017 wait complete. 2024-01-16T15:09:36.176 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596312 2024-01-16T15:09:36.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.380+0000 7f5189000640 1 Processor -- start 2024-01-16T15:09:36.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.380+0000 7f5189000640 1 -- start start 2024-01-16T15:09:36.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.381+0000 7f5189000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51840717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.381+0000 7f5189000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518415e680 con 0x7f51840ad880 2024-01-16T15:09:36.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.381+0000 7f5189000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5184071cf0 con 0x7f51840ac8d0 2024-01-16T15:09:36.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.381+0000 7f5189000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518415dd50 con 0x7f518406bb00 2024-01-16T15:09:36.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5181e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f51840ad880 0x7f518406b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58194/0 (socket says 172.21.15.141:58194) 2024-01-16T15:09:36.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5181e00640 1 -- 172.21.15.141:0/1978540342 learned_addr learned my addr 172.21.15.141:0/1978540342 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:36.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136826861 0 0) 0x7f518415e680 con 0x7f51840ad880 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5164003520 con 0x7f51840ad880 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5181400640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51840717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1945137560 0 0) 0x7f5164003520 con 0x7f51840ad880 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 msgr2=0x7f51840717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51840717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 >> v1:172.21.15.188:6789/0 conn(0x7f51840ac8d0 legacy=0x7f51840b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518415e030 con 0x7f51840ad880 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f517c001670 con 0x7f51840ad880 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5181400640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51840717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 >> v1:172.21.15.141:6789/0 conn(0x7f51840ad880 legacy=0x7f518406b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.382+0000 7f5180a00640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51640039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5180a00640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164003f90 0x7f51640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5180a00640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5164004d00 0x7f5164005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518415dd50 con 0x7f5164004d00 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516400bae0 con 0x7f5164003f90 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516400bca0 con 0x7f518406bb00 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5182800640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5164004d00 0x7f5164005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5181400640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164003f90 0x7f51640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5181e00640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51640039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5182800640 1 -- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 msgr2=0x7f51640039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5182800640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51640039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5182800640 1 -- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164003f90 msgr2=0x7f51640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5182800640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164003f90 0x7f51640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5182800640 1 -- 172.21.15.141:0/1978540342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5164003520 con 0x7f5164004d00 2024-01-16T15:09:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.383+0000 7f5181e00640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518406bb00 0x7f51640039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:36.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.384+0000 7f5182800640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5164004d00 0x7f5164005180 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f5184071e80 tx=0x7f51640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ba78f961d11508 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174003b70 con 0x7f5164004d00 2024-01-16T15:09:36.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174003cd0 con 0x7f5164004d00 2024-01-16T15:09:36.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5180a00640 1 -- 172.21.15.141:0/1978540342 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174005310 con 0x7f5164004d00 2024-01-16T15:09:36.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5189000640 1 -- 172.21.15.141:0/1978540342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5164004d00 msgr2=0x7f5164005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5189000640 1 --2- 172.21.15.141:0/1978540342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5164004d00 0x7f5164005180 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f5184071e80 tx=0x7f51640057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5182800640 1 -- 172.21.15.141:0/1978540342 reap_dead start 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5189000640 1 -- 172.21.15.141:0/1978540342 shutdown_connections 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5189000640 1 -- 172.21.15.141:0/1978540342 >> 172.21.15.141:0/1978540342 conn(0x7f5184070a80 msgr2=0x7f5184070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5189000640 1 -- 172.21.15.141:0/1978540342 shutdown_connections 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.385+0000 7f5189000640 1 -- 172.21.15.141:0/1978540342 wait complete. 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.387+0000 7f5189000640 1 Processor -- start 2024-01-16T15:09:36.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.387+0000 7f5189000640 1 -- start start 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5189000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164004d00 0x7f51840ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5189000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518414d4b0 0x7f51841403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5189000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51841408e0 0x7f518413d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5189000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5184141340 con 0x7f51841408e0 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5189000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51840cc620 con 0x7f5164004d00 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5189000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51840ce410 con 0x7f518414d4b0 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5182800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51841408e0 0x7f518413d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5182800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51841408e0 0x7f518413d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42368/0 (socket says 172.21.15.141:42368) 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.388+0000 7f5182800640 1 -- 172.21.15.141:0/2703040674 learned_addr learned my addr 172.21.15.141:0/2703040674 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5181e00640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164004d00 0x7f51840ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5181400640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518414d4b0 0x7f51841403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5182800640 1 -- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518414d4b0 msgr2=0x7f51841403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5182800640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518414d4b0 0x7f51841403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5182800640 1 -- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164004d00 msgr2=0x7f51840ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5182800640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164004d00 0x7f51840ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5182800640 1 -- 172.21.15.141:0/2703040674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5184140d80 con 0x7f51841408e0 2024-01-16T15:09:36.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5182800640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51841408e0 0x7f518413d890 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f5174001f10 tx=0x7f51740014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f517a000640 1 -- 172.21.15.141:0/2703040674 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517403e070 con 0x7f51841408e0 2024-01-16T15:09:36.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f517a000640 1 -- 172.21.15.141:0/2703040674 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174007dd0 con 0x7f51841408e0 2024-01-16T15:09:36.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f517a000640 1 -- 172.21.15.141:0/2703040674 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174005790 con 0x7f51841408e0 2024-01-16T15:09:36.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.389+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5184141060 con 0x7f51841408e0 2024-01-16T15:09:36.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.390+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51840ccff0 con 0x7f51841408e0 2024-01-16T15:09:36.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.391+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51840741c0 con 0x7f51841408e0 2024-01-16T15:09:36.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.391+0000 7f517a000640 1 -- 172.21.15.141:0/2703040674 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f5174007040 con 0x7f51841408e0 2024-01-16T15:09:36.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.392+0000 7f517a000640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f515c06c040 0x7f515c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.392+0000 7f517a000640 1 -- 172.21.15.141:0/2703040674 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51740a9b10 con 0x7f51841408e0 2024-01-16T15:09:36.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.393+0000 7f5181e00640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f515c06c040 0x7f515c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.394+0000 7f5181e00640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f515c06c040 0x7f515c06e500 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f517c001200 tx=0x7f517c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.397+0000 7f517a000640 1 -- 172.21.15.141:0/2703040674 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5174076120 con 0x7f51841408e0 2024-01-16T15:09:36.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.597+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596312", "target": ["mon-mgr", ""]}) v1 -- 0x7f51840675d0 con 0x7f515c06c040 2024-01-16T15:09:36.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.634+0000 7f517a000640 1 -- 172.21.15.141:0/2703040674 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f51840675d0 con 0x7f515c06c040 2024-01-16T15:09:36.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f515c06c040 msgr2=0x7f515c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f515c06c040 0x7f515c06e500 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f517c001200 tx=0x7f517c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:36.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51841408e0 msgr2=0x7f518413d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51841408e0 0x7f518413d890 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f5174001f10 tx=0x7f51740014a0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 shutdown_connections 2024-01-16T15:09:36.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f515c06c040 0x7f515c06e500 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f51841408e0 0x7f518413d890 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f518414d4b0 0x7f51841403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 --2- 172.21.15.141:0/2703040674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5164004d00 0x7f51840ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 >> 172.21.15.141:0/2703040674 conn(0x7f5184070a80 msgr2=0x7f51840b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 shutdown_connections 2024-01-16T15:09:36.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.637+0000 7f5189000640 1 -- 172.21.15.141:0/2703040674 wait complete. 2024-01-16T15:09:36.632 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596310 2024-01-16T15:09:36.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.842+0000 7f683c800640 1 Processor -- start 2024-01-16T15:09:36.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.843+0000 7f683c800640 1 -- start start 2024-01-16T15:09:36.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.844+0000 7f683c800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f68340cfcf0 0x7f6834160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.844+0000 7f683c800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340d4160 con 0x7f68340d3910 2024-01-16T15:09:36.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.844+0000 7f683c800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340b2420 con 0x7f68340d00d0 2024-01-16T15:09:36.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.844+0000 7f683c800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68340ceff0 con 0x7f68340cfcf0 2024-01-16T15:09:36.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.844+0000 7f6839e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f68340cfcf0 0x7f6834160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.844+0000 7f6839400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f68340d3910 0x7f68340b2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58198/0 (socket says 172.21.15.141:58198) 2024-01-16T15:09:36.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.844+0000 7f6839400640 1 -- 172.21.15.141:0/2091100182 learned_addr learned my addr 172.21.15.141:0/2091100182 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:36.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.845+0000 7f6839e00640 1 -- 172.21.15.141:0/2091100182 >> v1:172.21.15.188:6789/0 conn(0x7f68340d00d0 legacy=0x7f68340b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:36.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.845+0000 7f6839e00640 1 -- 172.21.15.141:0/2091100182 >> v1:172.21.15.141:6789/0 conn(0x7f68340d3910 legacy=0x7f68340b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.845+0000 7f6839e00640 1 -- 172.21.15.141:0/2091100182 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68340b2140 con 0x7f68340cfcf0 2024-01-16T15:09:36.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.846+0000 7f6839e00640 1 --2- 172.21.15.141:0/2091100182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340cfcf0 0x7f6834160700 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f6824009d90 tx=0x7f682402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfc807380b9f4f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.846+0000 7f6838a00640 1 -- 172.21.15.141:0/2091100182 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682402cbd0 con 0x7f68340cfcf0 2024-01-16T15:09:36.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.846+0000 7f6838a00640 1 -- 172.21.15.141:0/2091100182 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682402cd30 con 0x7f68340cfcf0 2024-01-16T15:09:36.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.846+0000 7f6838a00640 1 -- 172.21.15.141:0/2091100182 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682402cea0 con 0x7f68340cfcf0 2024-01-16T15:09:36.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.847+0000 7f683c800640 1 -- 172.21.15.141:0/2091100182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340cfcf0 msgr2=0x7f6834160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.847+0000 7f683c800640 1 --2- 172.21.15.141:0/2091100182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340cfcf0 0x7f6834160700 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f6824009d90 tx=0x7f682402be20 comp rx=0 tx=0).stop 2024-01-16T15:09:36.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.847+0000 7f683c800640 1 -- 172.21.15.141:0/2091100182 shutdown_connections 2024-01-16T15:09:36.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.847+0000 7f683c800640 1 --2- 172.21.15.141:0/2091100182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340cfcf0 0x7f6834160700 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.847+0000 7f683c800640 1 -- 172.21.15.141:0/2091100182 >> 172.21.15.141:0/2091100182 conn(0x7f683409c530 msgr2=0x7f68340d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.847+0000 7f683c800640 1 -- 172.21.15.141:0/2091100182 shutdown_connections 2024-01-16T15:09:36.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.847+0000 7f683c800640 1 -- 172.21.15.141:0/2091100182 wait complete. 2024-01-16T15:09:36.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.849+0000 7f683c800640 1 Processor -- start 2024-01-16T15:09:36.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.849+0000 7f683c800640 1 -- start start 2024-01-16T15:09:36.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f683c800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68340cfcf0 0x7f68340b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f683c800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340d00d0 0x7f68340c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f683c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68340d3910 0x7f68340c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f683c800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68340b6840 con 0x7f68340d00d0 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f683c800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68340cd260 con 0x7f68340cfcf0 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f683c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68340ccf80 con 0x7f68340d3910 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f6839400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340d00d0 0x7f68340c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f6839400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340d00d0 0x7f68340c9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42378/0 (socket says 172.21.15.141:42378) 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.850+0000 7f6839400640 1 -- 172.21.15.141:0/442702470 learned_addr learned my addr 172.21.15.141:0/442702470 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6839400640 1 -- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68340d3910 msgr2=0x7f68340c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6839400640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68340d3910 0x7f68340c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6839400640 1 -- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68340cfcf0 msgr2=0x7f68340b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6839400640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68340cfcf0 0x7f68340b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6839400640 1 -- 172.21.15.141:0/442702470 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824009a70 con 0x7f68340d00d0 2024-01-16T15:09:36.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6839400640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340d00d0 0x7f68340c9290 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c4a0 tx=0x7f681c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6832000640 1 -- 172.21.15.141:0/442702470 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c005ce0 con 0x7f68340d00d0 2024-01-16T15:09:36.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6832000640 1 -- 172.21.15.141:0/442702470 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c005e40 con 0x7f68340d00d0 2024-01-16T15:09:36.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f6832000640 1 -- 172.21.15.141:0/442702470 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c007270 con 0x7f68340d00d0 2024-01-16T15:09:36.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68340b6180 con 0x7f68340d00d0 2024-01-16T15:09:36.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.851+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68340c8200 con 0x7f68340d00d0 2024-01-16T15:09:36.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.852+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68340b6980 con 0x7f68340d00d0 2024-01-16T15:09:36.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.853+0000 7f6832000640 1 -- 172.21.15.141:0/442702470 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f681c007490 con 0x7f68340d00d0 2024-01-16T15:09:36.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.854+0000 7f6832000640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f680c06c040 0x7f680c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.854+0000 7f6832000640 1 -- 172.21.15.141:0/442702470 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f681c087d30 con 0x7f68340d00d0 2024-01-16T15:09:36.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.855+0000 7f6839e00640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f680c06c040 0x7f680c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.856+0000 7f6839e00640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f680c06c040 0x7f680c06e500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6824036000 tx=0x7f682402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:36.858+0000 7f6832000640 1 -- 172.21.15.141:0/442702470 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68340b6980 con 0x7f68340d00d0 2024-01-16T15:09:37.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.065+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596310", "target": ["mon-mgr", ""]}) v1 -- 0x7f68340cec10 con 0x7f680c06c040 2024-01-16T15:09:37.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.114+0000 7f6832000640 1 -- 172.21.15.141:0/442702470 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f68240344c0 con 0x7f680c06c040 2024-01-16T15:09:37.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f680c06c040 msgr2=0x7f680c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f680c06c040 0x7f680c06e500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6824036000 tx=0x7f682402e000 comp rx=0 tx=0).stop 2024-01-16T15:09:37.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340d00d0 msgr2=0x7f68340c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340d00d0 0x7f68340c9290 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c4a0 tx=0x7f681c00c960 comp rx=0 tx=0).stop 2024-01-16T15:09:37.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 shutdown_connections 2024-01-16T15:09:37.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f680c06c040 0x7f680c06e500 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68340d3910 0x7f68340c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68340d00d0 0x7f68340c9290 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 --2- 172.21.15.141:0/442702470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68340cfcf0 0x7f68340b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 >> 172.21.15.141:0/442702470 conn(0x7f683409c530 msgr2=0x7f6834158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:37.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 shutdown_connections 2024-01-16T15:09:37.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.117+0000 7f683c800640 1 -- 172.21.15.141:0/442702470 wait complete. 2024-01-16T15:09:37.115 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596314 2024-01-16T15:09:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.398+0000 7f811a600640 1 Processor -- start 2024-01-16T15:09:37.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.399+0000 7f811a600640 1 -- start start 2024-01-16T15:09:37.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f811a600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f811413df80 0x7f811415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f811a600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f811413e360 0x7f81140cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f811a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 0x7f81140cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f811a600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811406c470 con 0x7f811413df80 2024-01-16T15:09:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f811a600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811406c750 con 0x7f811413e360 2024-01-16T15:09:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f811a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811406ca30 con 0x7f811415f9c0 2024-01-16T15:09:37.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f8113400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f811413df80 0x7f811415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f8113400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f811413df80 0x7f811415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42390/0 (socket says 172.21.15.141:42390) 2024-01-16T15:09:37.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.401+0000 7f8113400640 1 -- 172.21.15.141:0/1281390134 learned_addr learned my addr 172.21.15.141:0/1281390134 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:37.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.402+0000 7f8113400640 1 -- 172.21.15.141:0/1281390134 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 msgr2=0x7f81140cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:37.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.402+0000 7f8113400640 1 --2- 172.21.15.141:0/1281390134 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 0x7f81140cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.402+0000 7f8113400640 1 -- 172.21.15.141:0/1281390134 >> v2:172.21.15.188:3300/0 conn(0x7f811413e360 msgr2=0x7f81140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:37.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.402+0000 7f8113400640 1 --2- 172.21.15.141:0/1281390134 >> v2:172.21.15.188:3300/0 conn(0x7f811413e360 0x7f81140cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.402+0000 7f8113400640 1 -- 172.21.15.141:0/1281390134 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811406cd10 con 0x7f811413df80 2024-01-16T15:09:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.403+0000 7f8113400640 1 --2- 172.21.15.141:0/1281390134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811415bee0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f811406ded0 tx=0x7f810402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aac57ee46f6b71 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.403+0000 7f8112a00640 1 -- 172.21.15.141:0/1281390134 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104031860 con 0x7f811413df80 2024-01-16T15:09:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.403+0000 7f8112a00640 1 -- 172.21.15.141:0/1281390134 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81040319c0 con 0x7f811413df80 2024-01-16T15:09:37.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.403+0000 7f8112a00640 1 -- 172.21.15.141:0/1281390134 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104031d50 con 0x7f811413df80 2024-01-16T15:09:37.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.404+0000 7f811a600640 1 -- 172.21.15.141:0/1281390134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 msgr2=0x7f811415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.404+0000 7f811a600640 1 --2- 172.21.15.141:0/1281390134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811415bee0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f811406ded0 tx=0x7f810402b900 comp rx=0 tx=0).stop 2024-01-16T15:09:37.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.404+0000 7f811a600640 1 -- 172.21.15.141:0/1281390134 shutdown_connections 2024-01-16T15:09:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.404+0000 7f811a600640 1 --2- 172.21.15.141:0/1281390134 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 0x7f81140cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.404+0000 7f811a600640 1 --2- 172.21.15.141:0/1281390134 >> v2:172.21.15.188:3300/0 conn(0x7f811413e360 0x7f81140cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.404+0000 7f811a600640 1 --2- 172.21.15.141:0/1281390134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811415bee0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.404+0000 7f811a600640 1 -- 172.21.15.141:0/1281390134 >> 172.21.15.141:0/1281390134 conn(0x7f811406a800 msgr2=0x7f811406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.405+0000 7f811a600640 1 -- 172.21.15.141:0/1281390134 shutdown_connections 2024-01-16T15:09:37.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.405+0000 7f811a600640 1 -- 172.21.15.141:0/1281390134 wait complete. 2024-01-16T15:09:37.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.407+0000 7f811a600640 1 Processor -- start 2024-01-16T15:09:37.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.407+0000 7f811a600640 1 -- start start 2024-01-16T15:09:37.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.408+0000 7f811a600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811413f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.408+0000 7f811a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f811413e360 0x7f811414f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.408+0000 7f811a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 0x7f8114153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.408+0000 7f811a600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8114151910 con 0x7f811413df80 2024-01-16T15:09:37.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.408+0000 7f811a600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811414f6a0 con 0x7f811413e360 2024-01-16T15:09:37.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.408+0000 7f811a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8114151070 con 0x7f811415f9c0 2024-01-16T15:09:37.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.409+0000 7f810be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f811413e360 0x7f811414f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.409+0000 7f810be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f811413e360 0x7f811414f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:38386/0 (socket says 172.21.15.141:38386) 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.409+0000 7f810be00640 1 -- 172.21.15.141:0/3274858607 learned_addr learned my addr 172.21.15.141:0/3274858607 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.410+0000 7f8113400640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811413f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.410+0000 7f8113e00640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 0x7f8114153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.411+0000 7f810be00640 1 -- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 msgr2=0x7f8114153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.411+0000 7f810be00640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 0x7f8114153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.411+0000 7f810be00640 1 -- 172.21.15.141:0/3274858607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 msgr2=0x7f811413f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.411+0000 7f810be00640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811413f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.411+0000 7f810be00640 1 -- 172.21.15.141:0/3274858607 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81040094d0 con 0x7f811413e360 2024-01-16T15:09:37.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.411+0000 7f8113400640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811413f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:37.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.412+0000 7f810be00640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f811413e360 0x7f811414f160 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00b450 tx=0x7f80fc00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.412+0000 7f810b400640 1 -- 172.21.15.141:0/3274858607 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc011890 con 0x7f811413e360 2024-01-16T15:09:37.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.413+0000 7f810b400640 1 -- 172.21.15.141:0/3274858607 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc0119f0 con 0x7f811413e360 2024-01-16T15:09:37.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.413+0000 7f810b400640 1 -- 172.21.15.141:0/3274858607 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc011da0 con 0x7f811413e360 2024-01-16T15:09:37.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.414+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811414f8d0 con 0x7f811413e360 2024-01-16T15:09:37.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.414+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81141515a0 con 0x7f811413e360 2024-01-16T15:09:37.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.416+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8114151ac0 con 0x7f811413e360 2024-01-16T15:09:37.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.418+0000 7f810b400640 1 -- 172.21.15.141:0/3274858607 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f80fc01a400 con 0x7f811413e360 2024-01-16T15:09:37.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.419+0000 7f810b400640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ec06c070 0x7f80ec06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.420+0000 7f810b400640 1 -- 172.21.15.141:0/3274858607 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f80fc088be0 con 0x7f811413e360 2024-01-16T15:09:37.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.420+0000 7f8113400640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ec06c070 0x7f80ec06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.422+0000 7f8113400640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ec06c070 0x7f80ec06e530 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8104036000 tx=0x7f810403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.427+0000 7f810b400640 1 -- 172.21.15.141:0/3274858607 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8114151ac0 con 0x7f811413e360 2024-01-16T15:09:37.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.744+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596314", "target": ["mon-mgr", ""]}) v1 -- 0x7f81141408d0 con 0x7f80ec06c070 2024-01-16T15:09:37.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.791+0000 7f810b400640 1 -- 172.21.15.141:0/3274858607 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f810403a4a0 con 0x7f80ec06c070 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ec06c070 msgr2=0x7f80ec06e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ec06c070 0x7f80ec06e530 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8104036000 tx=0x7f810403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f811413e360 msgr2=0x7f811414f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f811413e360 0x7f811414f160 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00b450 tx=0x7f80fc00b910 comp rx=0 tx=0).stop 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 shutdown_connections 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80ec06c070 0x7f80ec06e530 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f811415f9c0 0x7f8114153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f811413e360 0x7f811414f160 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 --2- 172.21.15.141:0/3274858607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f811413df80 0x7f811413f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 >> 172.21.15.141:0/3274858607 conn(0x7f811406a800 msgr2=0x7f8114140090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:37.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 shutdown_connections 2024-01-16T15:09:37.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:37.794+0000 7f811a600640 1 -- 172.21.15.141:0/3274858607 wait complete. 2024-01-16T15:09:37.790 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596317 2024-01-16T15:09:37.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.004+0000 7f0a32a00640 1 Processor -- start 2024-01-16T15:09:37.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.004+0000 7f0a32a00640 1 -- start start 2024-01-16T15:09:37.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0a2c15dca0 0x7f0a2c0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0a2c15acc0 0x7f0a2c06cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c0ce370 con 0x7f0a2c0aeac0 2024-01-16T15:09:37.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2c15e920 con 0x7f0a2c15acc0 2024-01-16T15:09:37.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2c0cddb0 con 0x7f0a2c15dca0 2024-01-16T15:09:37.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0a2c15acc0 0x7f0a2c06cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0a2c15acc0 0x7f0a2c06cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42406/0 (socket says 172.21.15.141:42406) 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.006+0000 7f0a32000640 1 -- 172.21.15.141:0/230140139 learned_addr learned my addr 172.21.15.141:0/230140139 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.007+0000 7f0a31600640 1 --2- 172.21.15.141:0/230140139 >> v2:172.21.15.188:3300/0 conn(0x7f0a2c15dca0 0x7f0a2c0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.007+0000 7f0a32000640 1 -- 172.21.15.141:0/230140139 >> v2:172.21.15.188:3300/0 conn(0x7f0a2c15dca0 msgr2=0x7f0a2c0cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.007+0000 7f0a32000640 1 --2- 172.21.15.141:0/230140139 >> v2:172.21.15.188:3300/0 conn(0x7f0a2c15dca0 0x7f0a2c0cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.007+0000 7f0a32000640 1 -- 172.21.15.141:0/230140139 >> v1:172.21.15.188:6789/0 conn(0x7f0a2c0aeac0 legacy=0x7f0a2c15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.007+0000 7f0a32000640 1 -- 172.21.15.141:0/230140139 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c0ce090 con 0x7f0a2c15acc0 2024-01-16T15:09:37.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.007+0000 7f0a31600640 1 --2- 172.21.15.141:0/230140139 >> v2:172.21.15.188:3300/0 conn(0x7f0a2c15dca0 0x7f0a2c0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:37.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.008+0000 7f0a32000640 1 --2- 172.21.15.141:0/230140139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c06cf20 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c071d50 tx=0x7f0a2402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b010503f3fa6a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a2be00640 1 -- 172.21.15.141:0/230140139 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2402c960 con 0x7f0a2c15acc0 2024-01-16T15:09:37.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a2be00640 1 -- 172.21.15.141:0/230140139 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2402cac0 con 0x7f0a2c15acc0 2024-01-16T15:09:37.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a2be00640 1 -- 172.21.15.141:0/230140139 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2402ce50 con 0x7f0a2c15acc0 2024-01-16T15:09:37.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 -- 172.21.15.141:0/230140139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 msgr2=0x7f0a2c06cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 --2- 172.21.15.141:0/230140139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c06cf20 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c071d50 tx=0x7f0a2402bdc0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 -- 172.21.15.141:0/230140139 shutdown_connections 2024-01-16T15:09:37.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 --2- 172.21.15.141:0/230140139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c06cf20 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 --2- 172.21.15.141:0/230140139 >> v2:172.21.15.188:3300/0 conn(0x7f0a2c15dca0 0x7f0a2c0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 -- 172.21.15.141:0/230140139 >> 172.21.15.141:0/230140139 conn(0x7f0a2c0cc1d0 msgr2=0x7f0a2c0cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:37.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 -- 172.21.15.141:0/230140139 shutdown_connections 2024-01-16T15:09:37.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.009+0000 7f0a32a00640 1 -- 172.21.15.141:0/230140139 wait complete. 2024-01-16T15:09:37.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.011+0000 7f0a32a00640 1 Processor -- start 2024-01-16T15:09:37.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.011+0000 7f0a32a00640 1 -- start start 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a32a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0aeac0 0x7f0a2c149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a32a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a32a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c15dca0 0x7f0a2c15c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a32a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c158ef0 con 0x7f0a2c15acc0 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a32a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0af2b0 con 0x7f0a2c15dca0 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a32a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c1497a0 con 0x7f0a2c0aeac0 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a30c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a30c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42412/0 (socket says 172.21.15.141:42412) 2024-01-16T15:09:37.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.012+0000 7f0a30c00640 1 -- 172.21.15.141:0/645598497 learned_addr learned my addr 172.21.15.141:0/645598497 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:37.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a32000640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c15dca0 0x7f0a2c15c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a31600640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0aeac0 0x7f0a2c149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a30c00640 1 -- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0aeac0 msgr2=0x7f0a2c149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a30c00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0aeac0 0x7f0a2c149260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a30c00640 1 -- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c15dca0 msgr2=0x7f0a2c15c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a30c00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c15dca0 0x7f0a2c15c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a30c00640 1 -- 172.21.15.141:0/645598497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a24009b30 con 0x7f0a2c15acc0 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a32000640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c15dca0 0x7f0a2c15c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a30c00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c157ed0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400c3f0 tx=0x7f0a1400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a29600640 1 -- 172.21.15.141:0/645598497 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14011790 con 0x7f0a2c15acc0 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a29600640 1 -- 172.21.15.141:0/645598497 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a14011910 con 0x7f0a2c15acc0 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a29600640 1 -- 172.21.15.141:0/645598497 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14011cc0 con 0x7f0a2c15acc0 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2c158930 con 0x7f0a2c15acc0 2024-01-16T15:09:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.013+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2c0aca00 con 0x7f0a2c15acc0 2024-01-16T15:09:37.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.014+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2c067b60 con 0x7f0a2c15acc0 2024-01-16T15:09:37.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.015+0000 7f0a29600640 1 -- 172.21.15.141:0/645598497 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f0a14012420 con 0x7f0a2c15acc0 2024-01-16T15:09:37.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.016+0000 7f0a29600640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a0406c040 0x7f0a0406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:37.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.016+0000 7f0a29600640 1 -- 172.21.15.141:0/645598497 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f0a14088530 con 0x7f0a2c15acc0 2024-01-16T15:09:37.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.017+0000 7f0a31600640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a0406c040 0x7f0a0406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:37.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.018+0000 7f0a31600640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a0406c040 0x7f0a0406e500 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00b840 tx=0x7f0a1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.021+0000 7f0a29600640 1 -- 172.21.15.141:0/645598497 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a14054ba0 con 0x7f0a2c15acc0 2024-01-16T15:09:38.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.256+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596317", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a2c06d460 con 0x7f0a0406c040 2024-01-16T15:09:38.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.286+0000 7f0a29600640 1 -- 172.21.15.141:0/645598497 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0a1c00c040 con 0x7f0a0406c040 2024-01-16T15:09:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.289+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a0406c040 msgr2=0x7f0a0406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.289+0000 7f0a32a00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a0406c040 0x7f0a0406e500 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00b840 tx=0x7f0a1c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.289+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 msgr2=0x7f0a2c157ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.289+0000 7f0a32a00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c157ed0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400c3f0 tx=0x7f0a1400c8b0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 shutdown_connections 2024-01-16T15:09:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a0406c040 0x7f0a0406e500 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c15dca0 0x7f0a2c15c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c15acc0 0x7f0a2c157ed0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 --2- 172.21.15.141:0/645598497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0aeac0 0x7f0a2c149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 >> 172.21.15.141:0/645598497 conn(0x7f0a2c0cc1d0 msgr2=0x7f0a2c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 shutdown_connections 2024-01-16T15:09:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.290+0000 7f0a32a00640 1 -- 172.21.15.141:0/645598497 wait complete. 2024-01-16T15:09:38.298 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59639 2024-01-16T15:09:38.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.510+0000 7f8b5fe00640 1 Processor -- start 2024-01-16T15:09:38.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.510+0000 7f8b5fe00640 1 -- start start 2024-01-16T15:09:38.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.511+0000 7f8b5fe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8b5813e360 0x7f8b580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b5fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5815c5f0 con 0x7f8b5813df80 2024-01-16T15:09:38.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b5fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5806c470 con 0x7f8b5815f9c0 2024-01-16T15:09:38.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b5fe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5806c750 con 0x7f8b5813e360 2024-01-16T15:09:38.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b5d400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8b5813df80 0x7f8b5815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58202/0 (socket says 172.21.15.141:58202) 2024-01-16T15:09:38.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b5d400640 1 -- 172.21.15.141:0/507137878 learned_addr learned my addr 172.21.15.141:0/507137878 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069709669 0 0) 0x7f8b5815c5f0 con 0x7f8b5813df80 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b38003520 con 0x7f8b5813df80 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727781676 0 0) 0x7f8b38003520 con 0x7f8b5813df80 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/507137878 >> v2:172.21.15.188:3300/0 conn(0x7f8b5813e360 0x7f8b580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.512+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 >> v2:172.21.15.188:3300/0 conn(0x7f8b5813e360 msgr2=0x7f8b580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 --2- 172.21.15.141:0/507137878 >> v2:172.21.15.188:3300/0 conn(0x7f8b5813e360 0x7f8b580cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 >> v1:172.21.15.188:6789/0 conn(0x7f8b5815f9c0 legacy=0x7f8b580cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5806ca30 con 0x7f8b5813df80 2024-01-16T15:09:38.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8b48001690 con 0x7f8b5813df80 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 >> v1:172.21.15.141:6789/0 conn(0x7f8b5813df80 legacy=0x7f8b5815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b38003970 0x7f8b38003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b380047c0 0x7f8b38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380052a0 0x7f8b3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5806c750 con 0x7f8b380047c0 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3800c1a0 con 0x7f8b38003970 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3800c360 con 0x7f8b380052a0 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b380047c0 0x7f8b38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5d400640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b38003970 0x7f8b38003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5ca00640 1 -- 172.21.15.141:0/507137878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380052a0 msgr2=0x7f8b3800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380052a0 0x7f8b3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5ca00640 1 -- 172.21.15.141:0/507137878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b38003970 msgr2=0x7f8b38003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b38003970 0x7f8b38003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5ca00640 1 -- 172.21.15.141:0/507137878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b38003520 con 0x7f8b380047c0 2024-01-16T15:09:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.513+0000 7f8b5de00640 1 -- 172.21.15.141:0/507137878 reap_dead start 2024-01-16T15:09:38.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.514+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b380047c0 0x7f8b38004c40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8b580cc790 tx=0x7f8b380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f9fb48cfdac063 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40004020 con 0x7f8b380047c0 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40004180 con 0x7f8b380047c0 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b57e00640 1 -- 172.21.15.141:0/507137878 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40003610 con 0x7f8b380047c0 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b5fe00640 1 -- 172.21.15.141:0/507137878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b380047c0 msgr2=0x7f8b38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b380047c0 0x7f8b38004c40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8b580cc790 tx=0x7f8b380057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b5fe00640 1 -- 172.21.15.141:0/507137878 shutdown_connections 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/507137878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b380047c0 0x7f8b38004c40 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b5fe00640 1 -- 172.21.15.141:0/507137878 >> 172.21.15.141:0/507137878 conn(0x7f8b5806a800 msgr2=0x7f8b5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b5fe00640 1 -- 172.21.15.141:0/507137878 shutdown_connections 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.515+0000 7f8b5fe00640 1 -- 172.21.15.141:0/507137878 wait complete. 2024-01-16T15:09:38.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.517+0000 7f8b5fe00640 1 Processor -- start 2024-01-16T15:09:38.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.517+0000 7f8b5fe00640 1 -- start start 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.518+0000 7f8b5fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380047c0 0x7f8b5813a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.518+0000 7f8b5fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b580b2cf0 0x7f8b580b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.518+0000 7f8b5fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b581e5310 0x7f8b581e5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.518+0000 7f8b5fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580b36b0 con 0x7f8b580b2cf0 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.518+0000 7f8b5fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58152700 con 0x7f8b581e5310 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.518+0000 7f8b5fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b581e5cd0 con 0x7f8b380047c0 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b580b2cf0 0x7f8b580b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b580b2cf0 0x7f8b580b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42442/0 (socket says 172.21.15.141:42442) 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 -- 172.21.15.141:0/1987429211 learned_addr learned my addr 172.21.15.141:0/1987429211 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5d400640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380047c0 0x7f8b5813a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5de00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b581e5310 0x7f8b581e5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 -- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380047c0 msgr2=0x7f8b5813a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380047c0 0x7f8b5813a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 -- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b581e5310 msgr2=0x7f8b581e5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b581e5310 0x7f8b581e5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 -- 172.21.15.141:0/1987429211 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58152950 con 0x7f8b580b2cf0 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.519+0000 7f8b5ca00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b580b2cf0 0x7f8b580b3170 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000a120 tx=0x7f8b40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.520+0000 7f8b55600640 1 -- 172.21.15.141:0/1987429211 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40004490 con 0x7f8b580b2cf0 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.520+0000 7f8b55600640 1 -- 172.21.15.141:0/1987429211 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40031660 con 0x7f8b580b2cf0 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.520+0000 7f8b55600640 1 -- 172.21.15.141:0/1987429211 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b400319d0 con 0x7f8b580b2cf0 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.520+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b58152c30 con 0x7f8b580b2cf0 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.520+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b581531f0 con 0x7f8b580b2cf0 2024-01-16T15:09:38.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.521+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b580675d0 con 0x7f8b580b2cf0 2024-01-16T15:09:38.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.522+0000 7f8b55600640 1 -- 172.21.15.141:0/1987429211 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f8b40031b30 con 0x7f8b580b2cf0 2024-01-16T15:09:38.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.523+0000 7f8b55600640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b3006c040 0x7f8b3006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.523+0000 7f8b55600640 1 -- 172.21.15.141:0/1987429211 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8b400ab030 con 0x7f8b580b2cf0 2024-01-16T15:09:38.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.524+0000 7f8b5d400640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b3006c040 0x7f8b3006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.525+0000 7f8b5d400640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b3006c040 0x7f8b3006e500 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000ff0 tx=0x7f8b48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.527+0000 7f8b55600640 1 -- 172.21.15.141:0/1987429211 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b40077760 con 0x7f8b580b2cf0 2024-01-16T15:09:38.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.746+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59639", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b5806cff0 con 0x7f8b3006c040 2024-01-16T15:09:38.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.792+0000 7f8b55600640 1 -- 172.21.15.141:0/1987429211 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8b4800d070 con 0x7f8b3006c040 2024-01-16T15:09:38.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b3006c040 msgr2=0x7f8b3006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b3006c040 0x7f8b3006e500 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000ff0 tx=0x7f8b48009040 comp rx=0 tx=0).stop 2024-01-16T15:09:38.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b580b2cf0 msgr2=0x7f8b580b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b580b2cf0 0x7f8b580b3170 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000a120 tx=0x7f8b40004730 comp rx=0 tx=0).stop 2024-01-16T15:09:38.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 shutdown_connections 2024-01-16T15:09:38.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8b3006c040 0x7f8b3006e500 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b581e5310 0x7f8b581e5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b580b2cf0 0x7f8b580b3170 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 --2- 172.21.15.141:0/1987429211 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8b380047c0 0x7f8b5813a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 >> 172.21.15.141:0/1987429211 conn(0x7f8b5806a800 msgr2=0x7f8b5806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 shutdown_connections 2024-01-16T15:09:38.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:38.795+0000 7f8b5fe00640 1 -- 172.21.15.141:0/1987429211 wait complete. 2024-01-16T15:09:38.792 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596315 2024-01-16T15:09:38.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.005+0000 7fbe79a00640 1 Processor -- start 2024-01-16T15:09:38.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.005+0000 7fbe79a00640 1 -- start start 2024-01-16T15:09:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.006+0000 7fbe79a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fbe74143ae0 0x7fbe740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe79a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe7415e3b0 0x7fbe740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe79a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe740cfc30 con 0x7fbe740cf7d0 2024-01-16T15:09:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe79a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe740b1ee0 con 0x7fbe74143ae0 2024-01-16T15:09:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe79a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740b2780 con 0x7fbe7415e3b0 2024-01-16T15:09:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe6be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbe740cf7d0 0x7fbe7415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58204/0 (socket says 172.21.15.141:58204) 2024-01-16T15:09:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe6be00640 1 -- 172.21.15.141:0/2851330219 learned_addr learned my addr 172.21.15.141:0/2851330219 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128838740 0 0) 0x7fbe740cfc30 con 0x7fbe740cf7d0 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe54003520 con 0x7fbe740cf7d0 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72a00640 1 --2- 172.21.15.141:0/2851330219 >> v2:172.21.15.188:3300/0 conn(0x7fbe74143ae0 0x7fbe740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe73400640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe7415e3b0 0x7fbe740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1848087121 0 0) 0x7fbe54003520 con 0x7fbe740cf7d0 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe7415e3b0 msgr2=0x7fbe740702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72000640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe7415e3b0 0x7fbe740702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 >> v2:172.21.15.188:3300/0 conn(0x7fbe74143ae0 msgr2=0x7fbe740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.007+0000 7fbe72000640 1 --2- 172.21.15.141:0/2851330219 >> v2:172.21.15.188:3300/0 conn(0x7fbe74143ae0 0x7fbe740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7415e990 con 0x7fbe740cf7d0 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fbe5c001670 con 0x7fbe740cf7d0 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 >> v1:172.21.15.141:6789/0 conn(0x7fbe740cf7d0 legacy=0x7fbe7415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe54003970 0x7fbe54003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe540047c0 0x7fbe54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe54005180 0x7fbe5400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740b1ee0 con 0x7fbe54005180 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740b2780 con 0x7fbe540047c0 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5400e280 con 0x7fbe54003970 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe73400640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe7415e3b0 0x7fbe740702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe73400640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe54005180 0x7fbe5400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe73400640 1 -- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe54003970 msgr2=0x7fbe54003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe73400640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe54003970 0x7fbe54003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe73400640 1 -- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe540047c0 msgr2=0x7fbe54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe73400640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe540047c0 0x7fbe54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.008+0000 7fbe73400640 1 -- 172.21.15.141:0/2851330219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54003520 con 0x7fbe54005180 2024-01-16T15:09:38.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.009+0000 7fbe73400640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe54005180 0x7fbe5400db90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c007e20 tx=0x7fbe540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9a78e4d08586c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c033600 con 0x7fbe54005180 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c02f050 con 0x7fbe54005180 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe72000640 1 -- 172.21.15.141:0/2851330219 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c033a00 con 0x7fbe54005180 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe79a00640 1 -- 172.21.15.141:0/2851330219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe54005180 msgr2=0x7fbe5400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe79a00640 1 --2- 172.21.15.141:0/2851330219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe54005180 0x7fbe5400db90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c007e20 tx=0x7fbe540057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe73400640 1 -- 172.21.15.141:0/2851330219 reap_dead start 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe79a00640 1 -- 172.21.15.141:0/2851330219 shutdown_connections 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe79a00640 1 -- 172.21.15.141:0/2851330219 >> 172.21.15.141:0/2851330219 conn(0x7fbe7406a800 msgr2=0x7fbe7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe79a00640 1 -- 172.21.15.141:0/2851330219 shutdown_connections 2024-01-16T15:09:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.010+0000 7fbe79a00640 1 -- 172.21.15.141:0/2851330219 wait complete. 2024-01-16T15:09:38.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.012+0000 7fbe79a00640 1 Processor -- start 2024-01-16T15:09:38.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.012+0000 7fbe79a00640 1 -- start start 2024-01-16T15:09:38.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.013+0000 7fbe79a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe54005180 0x7fbe741371f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.013+0000 7fbe79a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe74137730 0x7fbe741f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.013+0000 7fbe79a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe740cb2e0 0x7fbe741f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.013+0000 7fbe79a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe74140d50 con 0x7fbe740cb2e0 2024-01-16T15:09:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.013+0000 7fbe79a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740c41a0 con 0x7fbe54005180 2024-01-16T15:09:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.013+0000 7fbe79a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7413fdf0 con 0x7fbe74137730 2024-01-16T15:09:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe740cb2e0 0x7fbe741f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe740cb2e0 0x7fbe741f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42462/0 (socket says 172.21.15.141:42462) 2024-01-16T15:09:38.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 -- 172.21.15.141:0/4007259999 learned_addr learned my addr 172.21.15.141:0/4007259999 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:38.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 -- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe74137730 msgr2=0x7fbe741f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe74137730 0x7fbe741f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 -- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe54005180 msgr2=0x7fbe741371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe72a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe54005180 0x7fbe741371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe54005180 0x7fbe741371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 -- 172.21.15.141:0/4007259999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe741400d0 con 0x7fbe740cb2e0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe73400640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe740cb2e0 0x7fbe741f6750 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c02cb10 tx=0x7fbe6c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.014+0000 7fbe6aa00640 1 -- 172.21.15.141:0/4007259999 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c003990 con 0x7fbe740cb2e0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.015+0000 7fbe6aa00640 1 -- 172.21.15.141:0/4007259999 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c003af0 con 0x7fbe740cb2e0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.015+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe741403b0 con 0x7fbe740cb2e0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.015+0000 7fbe6aa00640 1 -- 172.21.15.141:0/4007259999 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c003e60 con 0x7fbe740cb2e0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.015+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe74140970 con 0x7fbe740cb2e0 2024-01-16T15:09:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.016+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe74140e90 con 0x7fbe740cb2e0 2024-01-16T15:09:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.017+0000 7fbe6aa00640 1 -- 172.21.15.141:0/4007259999 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fbe6c003fc0 con 0x7fbe740cb2e0 2024-01-16T15:09:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.017+0000 7fbe6aa00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbe4c06c070 0x7fbe4c06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.017+0000 7fbe6aa00640 1 -- 172.21.15.141:0/4007259999 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fbe6c0ab470 con 0x7fbe740cb2e0 2024-01-16T15:09:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.018+0000 7fbe72a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbe4c06c070 0x7fbe4c06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.019+0000 7fbe72a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbe4c06c070 0x7fbe4c06e530 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbe640047b0 tx=0x7fbe64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.022+0000 7fbe6aa00640 1 -- 172.21.15.141:0/4007259999 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe74140e90 con 0x7fbe740cb2e0 2024-01-16T15:09:39.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.261+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596315", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe740b2410 con 0x7fbe4c06c070 2024-01-16T15:09:39.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.290+0000 7fbe6aa00640 1 -- 172.21.15.141:0/4007259999 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe64004030 con 0x7fbe4c06c070 2024-01-16T15:09:39.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.294+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbe4c06c070 msgr2=0x7fbe4c06e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.294+0000 7fbe79a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbe4c06c070 0x7fbe4c06e530 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbe640047b0 tx=0x7fbe64008040 comp rx=0 tx=0).stop 2024-01-16T15:09:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.294+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe740cb2e0 msgr2=0x7fbe741f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.294+0000 7fbe79a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe740cb2e0 0x7fbe741f6750 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c02cb10 tx=0x7fbe6c004ab0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.295+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 shutdown_connections 2024-01-16T15:09:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.295+0000 7fbe79a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbe4c06c070 0x7fbe4c06e530 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.295+0000 7fbe79a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe740cb2e0 0x7fbe741f6750 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.295+0000 7fbe79a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbe74137730 0x7fbe741f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.295+0000 7fbe79a00640 1 --2- 172.21.15.141:0/4007259999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe54005180 0x7fbe741371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.295+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 >> 172.21.15.141:0/4007259999 conn(0x7fbe7406a800 msgr2=0x7fbe740ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:39.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.296+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 shutdown_connections 2024-01-16T15:09:39.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.296+0000 7fbe79a00640 1 -- 172.21.15.141:0/4007259999 wait complete. 2024-01-16T15:09:39.300 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59633 2024-01-16T15:09:39.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.568+0000 7f68e0000640 1 Processor -- start 2024-01-16T15:09:39.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.569+0000 7f68e0000640 1 -- start start 2024-01-16T15:09:39.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68e0000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f68d813df80 0x7f68d815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68e0000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d813e360 0x7f68d80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68e0000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d806c470 con 0x7f68d815f9c0 2024-01-16T15:09:39.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68e0000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d806c750 con 0x7f68d813df80 2024-01-16T15:09:39.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68e0000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d806ca30 con 0x7f68d813e360 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68dcc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d813e360 0x7f68d80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68dcc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d813e360 0x7f68d80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:55810/0 (socket says 172.21.15.141:55810) 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68dcc00640 1 -- 172.21.15.141:0/3696935942 learned_addr learned my addr 172.21.15.141:0/3696935942 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68dd600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f68d813df80 0x7f68d815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.570+0000 7f68de000640 1 --1- 172.21.15.141:0/3696935942 >> v1:172.21.15.188:6789/0 conn(0x7f68d815f9c0 0x7f68d81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:55126/0 (socket says 172.21.15.141:55126) 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.571+0000 7f68dcc00640 1 -- 172.21.15.141:0/3696935942 >> v2:172.21.15.188:3300/0 conn(0x7f68d813df80 msgr2=0x7f68d815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.571+0000 7f68dcc00640 1 --2- 172.21.15.141:0/3696935942 >> v2:172.21.15.188:3300/0 conn(0x7f68d813df80 0x7f68d815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.571+0000 7f68dcc00640 1 -- 172.21.15.141:0/3696935942 >> v1:172.21.15.188:6789/0 conn(0x7f68d815f9c0 legacy=0x7f68d81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.571+0000 7f68dcc00640 1 -- 172.21.15.141:0/3696935942 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d806cd10 con 0x7f68d813e360 2024-01-16T15:09:39.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.572+0000 7f68dcc00640 1 --2- 172.21.15.141:0/3696935942 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d813e360 0x7f68d80cf290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009870 tx=0x7f68c002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=328b3f685db384da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.572+0000 7f68d7e00640 1 -- 172.21.15.141:0/3696935942 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0030840 con 0x7f68d813e360 2024-01-16T15:09:39.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.572+0000 7f68d7e00640 1 -- 172.21.15.141:0/3696935942 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c00309a0 con 0x7f68d813e360 2024-01-16T15:09:39.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.572+0000 7f68d7e00640 1 -- 172.21.15.141:0/3696935942 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0030d30 con 0x7f68d813e360 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 -- 172.21.15.141:0/3696935942 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d813e360 msgr2=0x7f68d80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 --2- 172.21.15.141:0/3696935942 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d813e360 0x7f68d80cf290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009870 tx=0x7f68c002b980 comp rx=0 tx=0).stop 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 -- 172.21.15.141:0/3696935942 shutdown_connections 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 --2- 172.21.15.141:0/3696935942 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d813e360 0x7f68d80cf290 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 --2- 172.21.15.141:0/3696935942 >> v2:172.21.15.188:3300/0 conn(0x7f68d813df80 0x7f68d815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 -- 172.21.15.141:0/3696935942 >> 172.21.15.141:0/3696935942 conn(0x7f68d806a800 msgr2=0x7f68d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 -- 172.21.15.141:0/3696935942 shutdown_connections 2024-01-16T15:09:39.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.573+0000 7f68e0000640 1 -- 172.21.15.141:0/3696935942 wait complete. 2024-01-16T15:09:39.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.575+0000 7f68e0000640 1 Processor -- start 2024-01-16T15:09:39.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.575+0000 7f68e0000640 1 -- start start 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68e0000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d813df80 0x7f68d814fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68e0000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68d813e360 0x7f68d8152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68e0000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d815f9c0 0x7f68d81531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68e0000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d814f710 con 0x7f68d813e360 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68e0000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d814eb00 con 0x7f68d813df80 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68e0000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d814ede0 con 0x7f68d815f9c0 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68dcc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68d813e360 0x7f68d8152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68dcc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68d813e360 0x7f68d8152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42474/0 (socket says 172.21.15.141:42474) 2024-01-16T15:09:39.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68dcc00640 1 -- 172.21.15.141:0/3969958207 learned_addr learned my addr 172.21.15.141:0/3969958207 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68dd600640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d813df80 0x7f68d814fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.576+0000 7f68de000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d815f9c0 0x7f68d81531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68dcc00640 1 -- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d815f9c0 msgr2=0x7f68d81531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68dcc00640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d815f9c0 0x7f68d81531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68dcc00640 1 -- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d813df80 msgr2=0x7f68d814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68dcc00640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d813df80 0x7f68d814fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68dcc00640 1 -- 172.21.15.141:0/3969958207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c00094d0 con 0x7f68d813e360 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68dd600640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d813df80 0x7f68d814fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:39.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68de000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d815f9c0 0x7f68d81531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68dcc00640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68d813e360 0x7f68d8152c60 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f68c0000c00 tx=0x7f68c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68d5600640 1 -- 172.21.15.141:0/3969958207 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0039c90 con 0x7f68d813e360 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68d5600640 1 -- 172.21.15.141:0/3969958207 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0039df0 con 0x7f68d813e360 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68d5600640 1 -- 172.21.15.141:0/3969958207 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0043640 con 0x7f68d813e360 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d814f0c0 con 0x7f68d813e360 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.577+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d814e2a0 con 0x7f68d813e360 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.578+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d814f880 con 0x7f68d813e360 2024-01-16T15:09:39.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.579+0000 7f68d5600640 1 -- 172.21.15.141:0/3969958207 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f68c00437a0 con 0x7f68d813e360 2024-01-16T15:09:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.580+0000 7f68d5600640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68b006c040 0x7f68b006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.580+0000 7f68d5600640 1 -- 172.21.15.141:0/3969958207 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f68c003e070 con 0x7f68d813e360 2024-01-16T15:09:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.581+0000 7f68dd600640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68b006c040 0x7f68b006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:39.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.581+0000 7f68dd600640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68b006c040 0x7f68b006e500 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009a20 tx=0x7f68c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:39.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.584+0000 7f68d5600640 1 -- 172.21.15.141:0/3969958207 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68c0077f20 con 0x7f68d813e360 2024-01-16T15:09:39.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.794+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59633", "target": ["mon-mgr", ""]}) v1 -- 0x7f68d80ce850 con 0x7f68b006c040 2024-01-16T15:09:39.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.831+0000 7f68d5600640 1 -- 172.21.15.141:0/3969958207 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f68c8004030 con 0x7f68b006c040 2024-01-16T15:09:39.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68b006c040 msgr2=0x7f68b006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68b006c040 0x7f68b006e500 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009a20 tx=0x7f68c8008040 comp rx=0 tx=0).stop 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68d813e360 msgr2=0x7f68d8152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68d813e360 0x7f68d8152c60 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f68c0000c00 tx=0x7f68c0004730 comp rx=0 tx=0).stop 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 shutdown_connections 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f68b006c040 0x7f68b006e500 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f68d815f9c0 0x7f68d81531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68d813e360 0x7f68d8152c60 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.834+0000 7f68e0000640 1 --2- 172.21.15.141:0/3969958207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d813df80 0x7f68d814fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.835+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 >> 172.21.15.141:0/3969958207 conn(0x7f68d806a800 msgr2=0x7f68d815d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.835+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 shutdown_connections 2024-01-16T15:09:39.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:39.835+0000 7f68e0000640 1 -- 172.21.15.141:0/3969958207 wait complete. 2024-01-16T15:09:39.829 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596319 2024-01-16T15:09:40.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.043+0000 7f2d01c00640 1 Processor -- start 2024-01-16T15:09:40.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.043+0000 7f2d01c00640 1 -- start start 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2d01c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2cfc143ae0 0x7f2cfc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2d01c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2cfc15e3b0 0x7f2cfc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2d01c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc0cfc30 con 0x7f2cfc0cf7d0 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2d01c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0b1ee0 con 0x7f2cfc143ae0 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2d01c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0b2780 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2cfaa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2cfc143ae0 0x7f2cfc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2cfaa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2cfc143ae0 0x7f2cfc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42490/0 (socket says 172.21.15.141:42490) 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2cfaa00640 1 -- 172.21.15.141:0/1984906505 learned_addr learned my addr 172.21.15.141:0/1984906505 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:40.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.045+0000 7f2cf3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2cfc0cf7d0 0x7f2cfc15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58210/0 (socket says 172.21.15.141:58210) 2024-01-16T15:09:40.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.046+0000 7f2cfaa00640 1 -- 172.21.15.141:0/1984906505 >> v2:172.21.15.188:3300/0 conn(0x7f2cfc15e3b0 msgr2=0x7f2cfc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.046+0000 7f2cfb400640 1 --2- 172.21.15.141:0/1984906505 >> v2:172.21.15.188:3300/0 conn(0x7f2cfc15e3b0 0x7f2cfc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.046+0000 7f2cfaa00640 1 --2- 172.21.15.141:0/1984906505 >> v2:172.21.15.188:3300/0 conn(0x7f2cfc15e3b0 0x7f2cfc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.046+0000 7f2cfaa00640 1 -- 172.21.15.141:0/1984906505 >> v1:172.21.15.141:6789/0 conn(0x7f2cfc0cf7d0 legacy=0x7f2cfc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.046+0000 7f2cfaa00640 1 -- 172.21.15.141:0/1984906505 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc15e990 con 0x7f2cfc143ae0 2024-01-16T15:09:40.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.046+0000 7f2cfa000640 1 -- 172.21.15.141:0/1984906505 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458101283 0 0) 0x7f2cfc0cfc30 con 0x7f2cfc0cf7d0 2024-01-16T15:09:40.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.046+0000 7f2cfb400640 1 --2- 172.21.15.141:0/1984906505 >> v2:172.21.15.188:3300/0 conn(0x7f2cfc15e3b0 0x7f2cfc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:40.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.047+0000 7f2cfaa00640 1 --2- 172.21.15.141:0/1984906505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc143ae0 0x7f2cfc0cf290 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc06e4f0 tx=0x7f2cec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617426d39b731fd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.047+0000 7f2cfa000640 1 -- 172.21.15.141:0/1984906505 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec030860 con 0x7f2cfc143ae0 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.047+0000 7f2cfa000640 1 -- 172.21.15.141:0/1984906505 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec0309c0 con 0x7f2cfc143ae0 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.047+0000 7f2cfa000640 1 -- 172.21.15.141:0/1984906505 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec030d50 con 0x7f2cfc143ae0 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 -- 172.21.15.141:0/1984906505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc143ae0 msgr2=0x7f2cfc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 --2- 172.21.15.141:0/1984906505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc143ae0 0x7f2cfc0cf290 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc06e4f0 tx=0x7f2cec02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 -- 172.21.15.141:0/1984906505 shutdown_connections 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 --2- 172.21.15.141:0/1984906505 >> v2:172.21.15.188:3300/0 conn(0x7f2cfc15e3b0 0x7f2cfc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 --2- 172.21.15.141:0/1984906505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc143ae0 0x7f2cfc0cf290 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 -- 172.21.15.141:0/1984906505 >> 172.21.15.141:0/1984906505 conn(0x7f2cfc06a800 msgr2=0x7f2cfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 -- 172.21.15.141:0/1984906505 shutdown_connections 2024-01-16T15:09:40.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.048+0000 7f2d01c00640 1 -- 172.21.15.141:0/1984906505 wait complete. 2024-01-16T15:09:40.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.050+0000 7f2d01c00640 1 Processor -- start 2024-01-16T15:09:40.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.050+0000 7f2d01c00640 1 -- start start 2024-01-16T15:09:40.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2d01c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cfc0cf7d0 0x7f2cfc1f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2d01c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cfc143ae0 0x7f2cfc13e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2d01c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc15e3b0 0x7f2cfc0ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2d01c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0ad510 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2d01c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0ac8d0 con 0x7f2cfc0cf7d0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2d01c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc13e550 con 0x7f2cfc143ae0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2cfb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc15e3b0 0x7f2cfc0ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2cfb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc15e3b0 0x7f2cfc0ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42502/0 (socket says 172.21.15.141:42502) 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.051+0000 7f2cfb400640 1 -- 172.21.15.141:0/890503803 learned_addr learned my addr 172.21.15.141:0/890503803 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cfaa00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cfc0cf7d0 0x7f2cfc1f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cf3e00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cfc143ae0 0x7f2cfc13e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cfb400640 1 -- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cfc143ae0 msgr2=0x7f2cfc13e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cfb400640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cfc143ae0 0x7f2cfc13e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cfb400640 1 -- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cfc0cf7d0 msgr2=0x7f2cfc1f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cfb400640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cfc0cf7d0 0x7f2cfc1f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cfb400640 1 -- 172.21.15.141:0/890503803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0094d0 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cfb400640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc15e3b0 0x7f2cfc0ce090 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400cd50 tx=0x7f2cf400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cf2a00640 1 -- 172.21.15.141:0/890503803 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf400f680 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cf2a00640 1 -- 172.21.15.141:0/890503803 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf400f800 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc0acb70 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.052+0000 7f2cf2a00640 1 -- 172.21.15.141:0/890503803 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf400fbb0 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.053+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc0ad130 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.053+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc0ad650 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.054+0000 7f2cf2a00640 1 -- 172.21.15.141:0/890503803 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f2cf400b040 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.055+0000 7f2cf2a00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd406c040 0x7f2cd406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.055+0000 7f2cf2a00640 1 -- 172.21.15.141:0/890503803 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f2cf40892c0 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.056+0000 7f2cfaa00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd406c040 0x7f2cd406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.057+0000 7f2cfaa00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd406c040 0x7f2cd406e500 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2cec036000 tx=0x7f2cec008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.059+0000 7f2cf2a00640 1 -- 172.21.15.141:0/890503803 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cfc0ad650 con 0x7f2cfc15e3b0 2024-01-16T15:09:40.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.348+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596319", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cfc0b2410 con 0x7f2cd406c040 2024-01-16T15:09:40.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.369+0000 7f2cf2a00640 1 -- 172.21.15.141:0/890503803 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2cec044470 con 0x7f2cd406c040 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.371+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd406c040 msgr2=0x7f2cd406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.371+0000 7f2d01c00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd406c040 0x7f2cd406e500 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2cec036000 tx=0x7f2cec008030 comp rx=0 tx=0).stop 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.371+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc15e3b0 msgr2=0x7f2cfc0ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.371+0000 7f2d01c00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc15e3b0 0x7f2cfc0ce090 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400cd50 tx=0x7f2cf400a650 comp rx=0 tx=0).stop 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.371+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 shutdown_connections 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.371+0000 7f2d01c00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2cd406c040 0x7f2cd406e500 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.371+0000 7f2d01c00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cfc15e3b0 0x7f2cfc0ce090 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.372+0000 7f2d01c00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2cfc143ae0 0x7f2cfc13e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.372+0000 7f2d01c00640 1 --2- 172.21.15.141:0/890503803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2cfc0cf7d0 0x7f2cfc1f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.372+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 >> 172.21.15.141:0/890503803 conn(0x7f2cfc06a800 msgr2=0x7f2cfc15b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.372+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 shutdown_connections 2024-01-16T15:09:40.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.372+0000 7f2d01c00640 1 -- 172.21.15.141:0/890503803 wait complete. 2024-01-16T15:09:40.370 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596313 2024-01-16T15:09:40.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.569+0000 7fe0a1600640 1 Processor -- start 2024-01-16T15:09:40.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.569+0000 7fe0a1600640 1 -- start start 2024-01-16T15:09:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.571+0000 7fe0a1600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c0cd240 0x7fe09c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.571+0000 7fe0a1600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe09c09a170 0x7fe09c0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.571+0000 7fe0a1600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c06cea0 con 0x7fe09c06c6b0 2024-01-16T15:09:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.571+0000 7fe0a1600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09c06cbc0 con 0x7fe09c09a170 2024-01-16T15:09:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.571+0000 7fe0a1600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c06f350 con 0x7fe09c0cd240 2024-01-16T15:09:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.571+0000 7fe091e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe09c06c6b0 0x7fe09c099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58212/0 (socket says 172.21.15.141:58212) 2024-01-16T15:09:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.571+0000 7fe091e00640 1 -- 172.21.15.141:0/3532439293 learned_addr learned my addr 172.21.15.141:0/3532439293 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c0cd240 0x7fe09c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294778154 0 0) 0x7fe09c06cea0 con 0x7fe09c06c6b0 2024-01-16T15:09:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe07c003520 con 0x7fe09c06c6b0 2024-01-16T15:09:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09b400640 1 --2- 172.21.15.141:0/3532439293 >> v2:172.21.15.188:3300/0 conn(0x7fe09c09a170 0x7fe09c0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588526615 0 0) 0x7fe07c003520 con 0x7fe09c06c6b0 2024-01-16T15:09:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c0cd240 msgr2=0x7fe09c0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c0cd240 0x7fe09c0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 >> v2:172.21.15.188:3300/0 conn(0x7fe09c09a170 msgr2=0x7fe09c0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 --2- 172.21.15.141:0/3532439293 >> v2:172.21.15.188:3300/0 conn(0x7fe09c09a170 0x7fe09c0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c06f630 con 0x7fe09c06c6b0 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe084001670 con 0x7fe09c06c6b0 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 >> v1:172.21.15.141:6789/0 conn(0x7fe09c06c6b0 legacy=0x7fe09c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe07c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c0cd240 0x7fe07c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe07c004d00 0x7fe07c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c06cbc0 con 0x7fe07c003970 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c06f350 con 0x7fe07c004d00 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c00b980 con 0x7fe09c0cd240 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.572+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe07c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.573+0000 7fe09aa00640 1 -- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c0cd240 msgr2=0x7fe07c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.573+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c0cd240 0x7fe07c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.573+0000 7fe09aa00640 1 -- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe07c004d00 msgr2=0x7fe07c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.573+0000 7fe09b400640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe07c004d00 0x7fe07c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.573+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe07c004d00 0x7fe07c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.573+0000 7fe09aa00640 1 -- 172.21.15.141:0/3532439293 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c003520 con 0x7fe07c003970 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.574+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe07c003d90 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009ac0 tx=0x7fe07c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10147b94a1290892 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.574+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c02c710 con 0x7fe07c003970 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.574+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c02c870 con 0x7fe07c003970 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.574+0000 7fe09a000640 1 -- 172.21.15.141:0/3532439293 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c02cc00 con 0x7fe07c003970 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.575+0000 7fe0a1600640 1 -- 172.21.15.141:0/3532439293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 msgr2=0x7fe07c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.575+0000 7fe0a1600640 1 --2- 172.21.15.141:0/3532439293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe07c003d90 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009ac0 tx=0x7fe07c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.575+0000 7fe09b400640 1 -- 172.21.15.141:0/3532439293 reap_dead start 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.575+0000 7fe0a1600640 1 -- 172.21.15.141:0/3532439293 shutdown_connections 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.575+0000 7fe0a1600640 1 -- 172.21.15.141:0/3532439293 >> 172.21.15.141:0/3532439293 conn(0x7fe09c0b2d00 msgr2=0x7fe09c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.575+0000 7fe0a1600640 1 -- 172.21.15.141:0/3532439293 shutdown_connections 2024-01-16T15:09:40.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.575+0000 7fe0a1600640 1 -- 172.21.15.141:0/3532439293 wait complete. 2024-01-16T15:09:40.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.577+0000 7fe0a1600640 1 Processor -- start 2024-01-16T15:09:40.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.577+0000 7fe0a1600640 1 -- start start 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe0a1600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe09c0b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe0a1600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c155200 0x7fe09c0c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe0a1600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe09c0c7de0 0x7fe09c1f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe0a1600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0cb2c0 con 0x7fe07c003970 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe0a1600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0cad00 con 0x7fe09c0c7de0 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe0a1600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0cafe0 con 0x7fe09c155200 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe09c0b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe09c0b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42522/0 (socket says 172.21.15.141:42522) 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 -- 172.21.15.141:0/3983248051 learned_addr learned my addr 172.21.15.141:0/3983248051 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 -- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c155200 msgr2=0x7fe09c0c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c155200 0x7fe09c0c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 -- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe09c0c7de0 msgr2=0x7fe09c1f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe09c0c7de0 0x7fe09c1f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.578+0000 7fe09aa00640 1 -- 172.21.15.141:0/3983248051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c0b5d20 con 0x7fe07c003970 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.579+0000 7fe09aa00640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe09c0b6800 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe08c02c2b0 tx=0x7fe08c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.579+0000 7fe093400640 1 -- 172.21.15.141:0/3983248051 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c003810 con 0x7fe07c003970 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.579+0000 7fe093400640 1 -- 172.21.15.141:0/3983248051 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c02de90 con 0x7fe07c003970 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.579+0000 7fe093400640 1 -- 172.21.15.141:0/3983248051 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c003b90 con 0x7fe07c003970 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.579+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09c0b6000 con 0x7fe07c003970 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.579+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09c1f2450 con 0x7fe07c003970 2024-01-16T15:09:40.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.580+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09c0675d0 con 0x7fe07c003970 2024-01-16T15:09:40.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.581+0000 7fe093400640 1 -- 172.21.15.141:0/3983248051 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fe08c003cf0 con 0x7fe07c003970 2024-01-16T15:09:40.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.582+0000 7fe093400640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe07406c040 0x7fe07406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.582+0000 7fe093400640 1 -- 172.21.15.141:0/3983248051 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fe08c0aa880 con 0x7fe07c003970 2024-01-16T15:09:40.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.583+0000 7fe091e00640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe07406c040 0x7fe07406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.584+0000 7fe091e00640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe07406c040 0x7fe07406e500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe09c15f9f0 tx=0x7fe084001500 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.586+0000 7fe093400640 1 -- 172.21.15.141:0/3983248051 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe08c076e80 con 0x7fe07c003970 2024-01-16T15:09:40.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.805+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596313", "target": ["mon-mgr", ""]}) v1 -- 0x7fe09c0cb2c0 con 0x7fe07406c040 2024-01-16T15:09:40.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.848+0000 7fe093400640 1 -- 172.21.15.141:0/3983248051 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe09c0cb2c0 con 0x7fe07406c040 2024-01-16T15:09:40.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe07406c040 msgr2=0x7fe07406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe07406c040 0x7fe07406e500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe09c15f9f0 tx=0x7fe084001500 comp rx=0 tx=0).stop 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 msgr2=0x7fe09c0b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe09c0b6800 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe08c02c2b0 tx=0x7fe08c004730 comp rx=0 tx=0).stop 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 shutdown_connections 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe07406c040 0x7fe07406e500 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe09c0c7de0 0x7fe09c1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe09c155200 0x7fe09c0c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 --2- 172.21.15.141:0/3983248051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe07c003970 0x7fe09c0b6800 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 >> 172.21.15.141:0/3983248051 conn(0x7fe09c0b2d00 msgr2=0x7fe09c1607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 shutdown_connections 2024-01-16T15:09:40.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:40.851+0000 7fe0a1600640 1 -- 172.21.15.141:0/3983248051 wait complete. 2024-01-16T15:09:40.847 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone59637 2024-01-16T15:09:41.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.061+0000 7f1504e00640 1 Processor -- start 2024-01-16T15:09:41.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.061+0000 7f1504e00640 1 -- start start 2024-01-16T15:09:41.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.063+0000 7f1504e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f15000b3c10 0x7f15000713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.063+0000 7f1504e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15000cdf90 con 0x7f15000ce420 2024-01-16T15:09:41.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.063+0000 7f1504e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1500072000 con 0x7f15000b4280 2024-01-16T15:09:41.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.063+0000 7f1504e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1500071d20 con 0x7f15000b3c10 2024-01-16T15:09:41.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.063+0000 7f14fe800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f15000ce420 0x7f15000ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58224/0 (socket says 172.21.15.141:58224) 2024-01-16T15:09:41.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.063+0000 7f14fe800640 1 -- 172.21.15.141:0/1447158627 learned_addr learned my addr 172.21.15.141:0/1447158627 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:41.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762159688 0 0) 0x7f15000cdf90 con 0x7f15000ce420 2024-01-16T15:09:41.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fd400640 1 --2- 172.21.15.141:0/1447158627 >> v2:172.21.15.188:3300/0 conn(0x7f15000b3c10 0x7f15000713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e0005730 con 0x7f15000ce420 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1507481846 0 0) 0x7f14e0005730 con 0x7f15000ce420 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 >> v2:172.21.15.188:3300/0 conn(0x7f15000b3c10 msgr2=0x7f15000713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 --2- 172.21.15.141:0/1447158627 >> v2:172.21.15.188:3300/0 conn(0x7f15000b3c10 0x7f15000713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 >> v1:172.21.15.188:6789/0 conn(0x7f15000b4280 legacy=0x7f15000b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000cd940 con 0x7f15000ce420 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f14f8001670 con 0x7f15000ce420 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 >> v1:172.21.15.141:6789/0 conn(0x7f15000ce420 legacy=0x7f15000ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f14e0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14e00069d0 0x7f14e0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14e0007390 0x7f14e000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1500071d20 con 0x7f14e0005b80 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e000e2e0 con 0x7f14e00069d0 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e000e4a0 con 0x7f14e0007390 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.064+0000 7f14fde00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f14e0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.065+0000 7f14fd400640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14e00069d0 0x7f14e0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.065+0000 7f14fde00640 1 -- 172.21.15.141:0/1447158627 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14e0007390 msgr2=0x7f14e000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.065+0000 7f14fde00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14e0007390 0x7f14e000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.065+0000 7f14fde00640 1 -- 172.21.15.141:0/1447158627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14e00069d0 msgr2=0x7f14e0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.065+0000 7f14fde00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14e00069d0 0x7f14e0006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.065+0000 7f14fde00640 1 -- 172.21.15.141:0/1447158627 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e0005730 con 0x7f14e0005b80 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.065+0000 7f14fe800640 1 -- 172.21.15.141:0/1447158627 reap_dead start 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.066+0000 7f14fde00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f14e0005fa0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f14f0001750 tx=0x7f14e0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66294adca99f0d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.066+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f00052d0 con 0x7f14e0005b80 2024-01-16T15:09:41.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.066+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f002e050 con 0x7f14e0005b80 2024-01-16T15:09:41.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.066+0000 7f14fca00640 1 -- 172.21.15.141:0/1447158627 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0003830 con 0x7f14e0005b80 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.067+0000 7f1504e00640 1 -- 172.21.15.141:0/1447158627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 msgr2=0x7f14e0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.067+0000 7f1504e00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f14e0005fa0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f14f0001750 tx=0x7f14e0007a00 comp rx=0 tx=0).stop 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.067+0000 7f1504e00640 1 -- 172.21.15.141:0/1447158627 shutdown_connections 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.067+0000 7f1504e00640 1 --2- 172.21.15.141:0/1447158627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f14e0005fa0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.067+0000 7f1504e00640 1 -- 172.21.15.141:0/1447158627 >> 172.21.15.141:0/1447158627 conn(0x7f15000cfef0 msgr2=0x7f15000d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.067+0000 7f1504e00640 1 -- 172.21.15.141:0/1447158627 shutdown_connections 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.067+0000 7f1504e00640 1 -- 172.21.15.141:0/1447158627 wait complete. 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.069+0000 7f1504e00640 1 Processor -- start 2024-01-16T15:09:41.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.069+0000 7f1504e00640 1 -- start start 2024-01-16T15:09:41.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f1504e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f15000cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f1504e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1500159970 0x7f150015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f1504e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150015ab10 0x7f150015cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f1504e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150015bbc0 con 0x7f14e0005b80 2024-01-16T15:09:41.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f1504e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150015afb0 con 0x7f1500159970 2024-01-16T15:09:41.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f1504e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150015b290 con 0x7f150015ab10 2024-01-16T15:09:41.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f15000cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f15000cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42534/0 (socket says 172.21.15.141:42534) 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 -- 172.21.15.141:0/3227728990 learned_addr learned my addr 172.21.15.141:0/3227728990 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 -- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150015ab10 msgr2=0x7f150015cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fd400640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1500159970 0x7f150015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150015ab10 0x7f150015cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 -- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1500159970 msgr2=0x7f150015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1500159970 0x7f150015a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.070+0000 7f14fde00640 1 -- 172.21.15.141:0/3227728990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150015b570 con 0x7f14e0005b80 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.071+0000 7f14fde00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f15000cc040 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f14f002bc00 tx=0x7f14f0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.071+0000 7f14f6000640 1 -- 172.21.15.141:0/3227728990 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0003e80 con 0x7f14e0005b80 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.071+0000 7f14f6000640 1 -- 172.21.15.141:0/3227728990 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f0007a30 con 0x7f14e0005b80 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.071+0000 7f14f6000640 1 -- 172.21.15.141:0/3227728990 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f003d670 con 0x7f14e0005b80 2024-01-16T15:09:41.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.071+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150015b850 con 0x7f14e0005b80 2024-01-16T15:09:41.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.071+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15000cc580 con 0x7f14e0005b80 2024-01-16T15:09:41.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.072+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15000cc7a0 con 0x7f14e0005b80 2024-01-16T15:09:41.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.073+0000 7f14f6000640 1 -- 172.21.15.141:0/3227728990 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f14f003d7d0 con 0x7f14e0005b80 2024-01-16T15:09:41.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.074+0000 7f14f6000640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14d806c070 0x7f14d806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.074+0000 7f14f6000640 1 -- 172.21.15.141:0/3227728990 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f14f00aa510 con 0x7f14e0005b80 2024-01-16T15:09:41.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.075+0000 7f14fd400640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14d806c070 0x7f14d806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.076+0000 7f14fd400640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14d806c070 0x7f14d806e530 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f15000cd1b0 tx=0x7f14e800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.079+0000 7f14f6000640 1 -- 172.21.15.141:0/3227728990 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15000cc7a0 con 0x7f14e0005b80 2024-01-16T15:09:41.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.369+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone59637", "target": ["mon-mgr", ""]}) v1 -- 0x7f15000b4660 con 0x7f14d806c070 2024-01-16T15:09:41.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.385+0000 7f14f6000640 1 -- 172.21.15.141:0/3227728990 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f14e800f040 con 0x7f14d806c070 2024-01-16T15:09:41.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14d806c070 msgr2=0x7f14d806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14d806c070 0x7f14d806e530 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f15000cd1b0 tx=0x7f14e800b040 comp rx=0 tx=0).stop 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 msgr2=0x7f15000cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f15000cc040 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f14f002bc00 tx=0x7f14f0001120 comp rx=0 tx=0).stop 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 shutdown_connections 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14d806c070 0x7f14d806e530 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150015ab10 0x7f150015cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1500159970 0x7f150015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 --2- 172.21.15.141:0/3227728990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14e0005b80 0x7f15000cc040 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 >> 172.21.15.141:0/3227728990 conn(0x7f15000cfef0 msgr2=0x7f1500141340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 shutdown_connections 2024-01-16T15:09:41.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.388+0000 7f1504e00640 1 -- 172.21.15.141:0/3227728990 wait complete. 2024-01-16T15:09:41.383 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone596320 2024-01-16T15:09:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.590+0000 7f6ad8a00640 1 Processor -- start 2024-01-16T15:09:41.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.591+0000 7f6ad8a00640 1 -- start start 2024-01-16T15:09:41.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.592+0000 7f6ad8a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6ad00cd240 0x7f6ad00b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.592+0000 7f6ad8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad006c6b0 0x7f6ad0099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.592+0000 7f6ad8a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad006cea0 con 0x7f6ad0099fa0 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.592+0000 7f6ad8a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad009a430 con 0x7f6ad00cd240 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.592+0000 7f6ad8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad006cbc0 con 0x7f6ad006c6b0 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.592+0000 7f6ad6a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6ad0099fa0 0x7f6ad006b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58230/0 (socket says 172.21.15.141:58230) 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.592+0000 7f6ad6a00640 1 -- 172.21.15.141:0/4275063338 learned_addr learned my addr 172.21.15.141:0/4275063338 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad5600640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad006c6b0 0x7f6ad0099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4275063338 >> v2:172.21.15.188:3300/0 conn(0x7f6ad00cd240 0x7f6ad00b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645996519 0 0) 0x7f6ad006cea0 con 0x7f6ad0099fa0 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab0003520 con 0x7f6ad0099fa0 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3744033643 0 0) 0x7f6ab0003520 con 0x7f6ad0099fa0 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad006c6b0 msgr2=0x7f6ad0099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad006c6b0 0x7f6ad0099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 >> v2:172.21.15.188:3300/0 conn(0x7f6ad00cd240 msgr2=0x7f6ad00b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 --2- 172.21.15.141:0/4275063338 >> v2:172.21.15.188:3300/0 conn(0x7f6ad00cd240 0x7f6ad00b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad006f350 con 0x7f6ad0099fa0 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6ac80019f0 con 0x7f6ad0099fa0 2024-01-16T15:09:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 >> v1:172.21.15.141:6789/0 conn(0x7f6ad0099fa0 legacy=0x7f6ad006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ab0003970 0x7f6ab0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.593+0000 7f6ad4c00640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab00047c0 0x7f6ab0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad4c00640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad006c6b0 0x7f6ab0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad009a430 con 0x7f6ab0003970 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad006cbc0 con 0x7f6ab00047c0 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab000b980 con 0x7f6ad006c6b0 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ab0003970 0x7f6ab0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad5600640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab00047c0 0x7f6ab0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad6000640 1 -- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad006c6b0 msgr2=0x7f6ab0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad006c6b0 0x7f6ab0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad6000640 1 -- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab00047c0 msgr2=0x7f6ab0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:41.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab00047c0 0x7f6ab0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.594+0000 7f6ad6000640 1 -- 172.21.15.141:0/4275063338 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0003520 con 0x7f6ab0003970 2024-01-16T15:09:41.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.595+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ab0003970 0x7f6ab0003d90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009ac0 tx=0x7f6ab00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4993759fba0cab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.595+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac002c710 con 0x7f6ab0003970 2024-01-16T15:09:41.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.595+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac002c870 con 0x7f6ab0003970 2024-01-16T15:09:41.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.595+0000 7f6ad4c00640 1 -- 172.21.15.141:0/4275063338 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac002cc00 con 0x7f6ab0003970 2024-01-16T15:09:41.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.596+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4275063338 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ab0003970 msgr2=0x7f6ab0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.596+0000 7f6ad8a00640 1 --2- 172.21.15.141:0/4275063338 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ab0003970 0x7f6ab0003d90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009ac0 tx=0x7f6ab00057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.596+0000 7f6ad6a00640 1 -- 172.21.15.141:0/4275063338 reap_dead start 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.596+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4275063338 shutdown_connections 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.596+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4275063338 >> 172.21.15.141:0/4275063338 conn(0x7f6ad00b2d00 msgr2=0x7f6ad00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.596+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4275063338 shutdown_connections 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.596+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4275063338 wait complete. 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.598+0000 7f6ad8a00640 1 Processor -- start 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.598+0000 7f6ad8a00640 1 -- start start 2024-01-16T15:09:41.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad8a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad00c7830 0x7f6ad00c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad01f2f80 0x7f6ad0152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ad00c5d80 0x7f6ad0152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad8a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad01f3460 con 0x7f6ad00c5d80 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad01f3740 con 0x7f6ad01f2f80 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad8a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad01f3a20 con 0x7f6ad00c7830 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ad00c5d80 0x7f6ad0152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ad00c5d80 0x7f6ad0152640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42564/0 (socket says 172.21.15.141:42564) 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.599+0000 7f6ad6a00640 1 -- 172.21.15.141:0/4013263956 learned_addr learned my addr 172.21.15.141:0/4013263956 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad00c7830 0x7f6ad00c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad5600640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad01f2f80 0x7f6ad0152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad6a00640 1 -- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad00c7830 msgr2=0x7f6ad00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad6a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad00c7830 0x7f6ad00c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad6a00640 1 -- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad01f2f80 msgr2=0x7f6ad0152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad6a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad01f2f80 0x7f6ad0152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad6a00640 1 -- 172.21.15.141:0/4013263956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad01f3d00 con 0x7f6ad00c5d80 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad6a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ad00c5d80 0x7f6ad0152640 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f6ad00709e0 tx=0x7f6ac800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ace000640 1 -- 172.21.15.141:0/4013263956 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac800aa20 con 0x7f6ad00c5d80 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ace000640 1 -- 172.21.15.141:0/4013263956 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac800ab80 con 0x7f6ad00c5d80 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ace000640 1 -- 172.21.15.141:0/4013263956 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8007030 con 0x7f6ad00c5d80 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.600+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad0152b80 con 0x7f6ad00c5d80 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.601+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad00c7df0 con 0x7f6ad00c5d80 2024-01-16T15:09:41.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.602+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad00675d0 con 0x7f6ad00c5d80 2024-01-16T15:09:41.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.603+0000 7f6ace000640 1 -- 172.21.15.141:0/4013263956 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f6ac8007190 con 0x7f6ad00c5d80 2024-01-16T15:09:41.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.604+0000 7f6ace000640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6aa806c040 0x7f6aa806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.604+0000 7f6ace000640 1 -- 172.21.15.141:0/4013263956 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6ac8087e70 con 0x7f6ad00c5d80 2024-01-16T15:09:41.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.604+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6aa806c040 0x7f6aa806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.605+0000 7f6ad6000640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6aa806c040 0x7f6aa806e500 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0036000 tx=0x7f6ac0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.608+0000 7f6ace000640 1 -- 172.21.15.141:0/4013263956 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ac8054480 con 0x7f6ad00c5d80 2024-01-16T15:09:41.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.824+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone596320", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ad00cd620 con 0x7f6aa806c040 2024-01-16T15:09:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.866+0000 7f6ace000640 1 -- 172.21.15.141:0/4013263956 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6ac002c9e0 con 0x7f6aa806c040 2024-01-16T15:09:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.868+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6aa806c040 msgr2=0x7f6aa806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.868+0000 7f6ad8a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6aa806c040 0x7f6aa806e500 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0036000 tx=0x7f6ac0030000 comp rx=0 tx=0).stop 2024-01-16T15:09:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.868+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ad00c5d80 msgr2=0x7f6ad0152640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.868+0000 7f6ad8a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ad00c5d80 0x7f6ad0152640 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f6ad00709e0 tx=0x7f6ac800cc30 comp rx=0 tx=0).stop 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 shutdown_connections 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6aa806c040 0x7f6aa806e500 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ad00c5d80 0x7f6ad0152640 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad01f2f80 0x7f6ad0152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 --2- 172.21.15.141:0/4013263956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ad00c7830 0x7f6ad00c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 >> 172.21.15.141:0/4013263956 conn(0x7f6ad00b2d00 msgr2=0x7f6ad006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 shutdown_connections 2024-01-16T15:09:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:41.869+0000 7f6ad8a00640 1 -- 172.21.15.141:0/4013263956 wait complete. 2024-01-16T15:09:41.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:41.865 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:09:41.923 INFO:teuthology.orchestra.run.smithi141.stdout:2 2024-01-16T15:09:41.924 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:09:42.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.140+0000 7fbc97c00640 1 Processor -- start 2024-01-16T15:09:42.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.140+0000 7fbc97c00640 1 -- start start 2024-01-16T15:09:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc97c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbc9014b2c0 0x7fbc900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc97c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc90071070 0x7fbc900cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc97c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc900cfbf0 con 0x7fbc900cf7d0 2024-01-16T15:09:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc97c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9014b6a0 con 0x7fbc9014b2c0 2024-01-16T15:09:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc97c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc90071690 con 0x7fbc90071070 2024-01-16T15:09:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc95200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbc9014b2c0 0x7fbc900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc95200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbc9014b2c0 0x7fbc900cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42580/0 (socket says 172.21.15.141:42580) 2024-01-16T15:09:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.142+0000 7fbc95200640 1 -- 172.21.15.141:0/3539047203 learned_addr learned my addr 172.21.15.141:0/3539047203 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.143+0000 7fbc95c00640 1 --2- 172.21.15.141:0/3539047203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc90071070 0x7fbc900cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.143+0000 7fbc95200640 1 -- 172.21.15.141:0/3539047203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc90071070 msgr2=0x7fbc900cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.143+0000 7fbc95200640 1 --2- 172.21.15.141:0/3539047203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc90071070 0x7fbc900cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.143+0000 7fbc95200640 1 -- 172.21.15.141:0/3539047203 >> v1:172.21.15.188:6789/0 conn(0x7fbc900cf7d0 legacy=0x7fbc90070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.143+0000 7fbc95200640 1 -- 172.21.15.141:0/3539047203 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90071970 con 0x7fbc9014b2c0 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.144+0000 7fbc95200640 1 --2- 172.21.15.141:0/3539047203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc9014b2c0 0x7fbc900cf290 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009870 tx=0x7fbc8002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3712fde5236a4efc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.144+0000 7fbc8f400640 1 -- 172.21.15.141:0/3539047203 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80030840 con 0x7fbc9014b2c0 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.144+0000 7fbc8f400640 1 -- 172.21.15.141:0/3539047203 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc800309a0 con 0x7fbc9014b2c0 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.144+0000 7fbc8f400640 1 -- 172.21.15.141:0/3539047203 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80030d30 con 0x7fbc9014b2c0 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 -- 172.21.15.141:0/3539047203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc9014b2c0 msgr2=0x7fbc900cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3539047203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc9014b2c0 0x7fbc900cf290 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009870 tx=0x7fbc8002b980 comp rx=0 tx=0).stop 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 -- 172.21.15.141:0/3539047203 shutdown_connections 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3539047203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc90071070 0x7fbc900cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3539047203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc9014b2c0 0x7fbc900cf290 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 -- 172.21.15.141:0/3539047203 >> 172.21.15.141:0/3539047203 conn(0x7fbc9006a800 msgr2=0x7fbc9006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:42.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 -- 172.21.15.141:0/3539047203 shutdown_connections 2024-01-16T15:09:42.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.145+0000 7fbc97c00640 1 -- 172.21.15.141:0/3539047203 wait complete. 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.147+0000 7fbc97c00640 1 Processor -- start 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.147+0000 7fbc97c00640 1 -- start start 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc97c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc90071070 0x7fbc900aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc97c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc900cf7d0 0x7fbc9015db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc97c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc9014b2c0 0x7fbc90164530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc97c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9015e090 con 0x7fbc900cf7d0 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc97c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc90163ff0 con 0x7fbc90071070 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc97c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900a9220 con 0x7fbc9014b2c0 2024-01-16T15:09:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc8fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc900cf7d0 0x7fbc9015db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc8fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc900cf7d0 0x7fbc9015db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42588/0 (socket says 172.21.15.141:42588) 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.148+0000 7fbc8fe00640 1 -- 172.21.15.141:0/3907196601 learned_addr learned my addr 172.21.15.141:0/3907196601 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc95200640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc90071070 0x7fbc900aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc95c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc9014b2c0 0x7fbc90164530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8fe00640 1 -- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc9014b2c0 msgr2=0x7fbc90164530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8fe00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc9014b2c0 0x7fbc90164530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8fe00640 1 -- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc90071070 msgr2=0x7fbc900aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8fe00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc90071070 0x7fbc900aa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8fe00640 1 -- 172.21.15.141:0/3907196601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800094d0 con 0x7fbc900cf7d0 2024-01-16T15:09:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc95c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc9014b2c0 0x7fbc90164530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8fe00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc900cf7d0 0x7fbc9015db50 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800c450 tx=0x7fbc7800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc780052d0 con 0x7fbc900cf7d0 2024-01-16T15:09:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc78005430 con 0x7fbc900cf7d0 2024-01-16T15:09:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc900a9500 con 0x7fbc900cf7d0 2024-01-16T15:09:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.149+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78012870 con 0x7fbc900cf7d0 2024-01-16T15:09:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.150+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc900a9ac0 con 0x7fbc900cf7d0 2024-01-16T15:09:42.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.150+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc900675d0 con 0x7fbc900cf7d0 2024-01-16T15:09:42.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.151+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 28) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fbc780129d0 con 0x7fbc900cf7d0 2024-01-16T15:09:42.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.152+0000 7fbc8cc00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc6806c040 0x7fbc6806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:42.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.152+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fbc78088420 con 0x7fbc900cf7d0 2024-01-16T15:09:42.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.153+0000 7fbc95200640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc6806c040 0x7fbc6806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:42.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.154+0000 7fbc95200640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc6806c040 0x7fbc6806e500 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbc80036000 tx=0x7fbc8003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:42.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.157+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc78054a30 con 0x7fbc900cf7d0 2024-01-16T15:09:42.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:42.520+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc900cbc40 con 0x7fbc6806c040 2024-01-16T15:09:42.536 INFO:tasks.ceph.mon.a.smithi141.stderr:2024-01-16T15:09:42.558+0000 7f723b200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:09:42.628 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:09:43.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:43.817+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fbc78087fc0 con 0x7fbc900cf7d0 2024-01-16T15:09:45.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.602+0000 7fbc8cc00640 1 -- 172.21.15.141:0/3907196601 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fbc80030c80 con 0x7fbc6806c040 2024-01-16T15:09:45.580 INFO:teuthology.orchestra.run.smithi141.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:09:45.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.607+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc6806c040 msgr2=0x7fbc6806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.607+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc6806c040 0x7fbc6806e500 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbc80036000 tx=0x7fbc8003b040 comp rx=0 tx=0).stop 2024-01-16T15:09:45.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.607+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc900cf7d0 msgr2=0x7fbc9015db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.607+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc900cf7d0 0x7fbc9015db50 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800c450 tx=0x7fbc7800c910 comp rx=0 tx=0).stop 2024-01-16T15:09:45.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 shutdown_connections 2024-01-16T15:09:45.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbc6806c040 0x7fbc6806e500 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbc9014b2c0 0x7fbc90164530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbc900cf7d0 0x7fbc9015db50 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 --2- 172.21.15.141:0/3907196601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc90071070 0x7fbc900aa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 >> 172.21.15.141:0/3907196601 conn(0x7fbc9006a800 msgr2=0x7fbc900b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:45.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 shutdown_connections 2024-01-16T15:09:45.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:45.608+0000 7fbc97c00640 1 -- 172.21.15.141:0/3907196601 wait complete. 2024-01-16T15:09:45.616 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:09:45.616 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:45.771 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:45.925 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:09:45.925 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:09:45.925 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:45.968 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:46.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.201+0000 7f8237400640 1 Processor -- start 2024-01-16T15:09:46.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.202+0000 7f8237400640 1 -- start start 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8237400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f823013cfa0 0x7f823015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8237400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f823013fcb0 0x7f82300b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8237400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82300cdec0 con 0x7f82300cd530 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8237400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82300ce1a0 con 0x7f823013fcb0 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8237400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82300ce480 con 0x7f823013cfa0 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8235600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f82300cd530 0x7f823013f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33490/0 (socket says 172.21.15.141:33490) 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8235600640 1 -- 172.21.15.141:0/1825137967 learned_addr learned my addr 172.21.15.141:0/1825137967 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:46.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.203+0000 7f8236a00640 1 --2- 172.21.15.141:0/1825137967 >> v2:172.21.15.141:3300/0 conn(0x7f823013fcb0 0x7f82300b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.204+0000 7f8236000640 1 --2- 172.21.15.141:0/1825137967 >> v2:172.21.15.188:3300/0 conn(0x7f823013cfa0 0x7f823015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.204+0000 7f8234c00640 1 -- 172.21.15.141:0/1825137967 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102481106 0 0) 0x7f82300cdec0 con 0x7f82300cd530 2024-01-16T15:09:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.204+0000 7f8234c00640 1 -- 172.21.15.141:0/1825137967 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8210003520 con 0x7f82300cd530 2024-01-16T15:09:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.204+0000 7f8236a00640 1 -- 172.21.15.141:0/1825137967 >> v2:172.21.15.188:3300/0 conn(0x7f823013cfa0 msgr2=0x7f823015afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.204+0000 7f8236a00640 1 --2- 172.21.15.141:0/1825137967 >> v2:172.21.15.188:3300/0 conn(0x7f823013cfa0 0x7f823015afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.204+0000 7f8236a00640 1 -- 172.21.15.141:0/1825137967 >> v1:172.21.15.141:6789/0 conn(0x7f82300cd530 legacy=0x7f823013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.204+0000 7f8236a00640 1 -- 172.21.15.141:0/1825137967 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82300ce760 con 0x7f823013fcb0 2024-01-16T15:09:46.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.205+0000 7f8236a00640 1 --2- 172.21.15.141:0/1825137967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f823013fcb0 0x7f82300b2040 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f823006cf00 tx=0x7f822802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d11efbaf4de594d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.205+0000 7f8234c00640 1 -- 172.21.15.141:0/1825137967 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822802cbe0 con 0x7f823013fcb0 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.205+0000 7f8234c00640 1 -- 172.21.15.141:0/1825137967 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822802cd40 con 0x7f823013fcb0 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.205+0000 7f8234c00640 1 -- 172.21.15.141:0/1825137967 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228034660 con 0x7f823013fcb0 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 -- 172.21.15.141:0/1825137967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f823013fcb0 msgr2=0x7f82300b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 --2- 172.21.15.141:0/1825137967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f823013fcb0 0x7f82300b2040 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f823006cf00 tx=0x7f822802be30 comp rx=0 tx=0).stop 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 -- 172.21.15.141:0/1825137967 shutdown_connections 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 --2- 172.21.15.141:0/1825137967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f823013fcb0 0x7f82300b2040 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 --2- 172.21.15.141:0/1825137967 >> v2:172.21.15.188:3300/0 conn(0x7f823013cfa0 0x7f823015afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 -- 172.21.15.141:0/1825137967 >> 172.21.15.141:0/1825137967 conn(0x7f823006a800 msgr2=0x7f823006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 -- 172.21.15.141:0/1825137967 shutdown_connections 2024-01-16T15:09:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.206+0000 7f8237400640 1 -- 172.21.15.141:0/1825137967 wait complete. 2024-01-16T15:09:46.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.207+0000 7f8237400640 1 Processor -- start 2024-01-16T15:09:46.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.207+0000 7f8237400640 1 -- start start 2024-01-16T15:09:46.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.208+0000 7f8237400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82300cd530 0x7f8230157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.208+0000 7f8237400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f823013cfa0 0x7f82301518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.208+0000 7f8237400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f823013fcb0 0x7f823013e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.208+0000 7f8237400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230166ea0 con 0x7f82300cd530 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.208+0000 7f8237400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230157950 con 0x7f823013fcb0 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.208+0000 7f8237400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8230166290 con 0x7f823013cfa0 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82300cd530 0x7f8230157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82300cd530 0x7f8230157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39378/0 (socket says 172.21.15.141:39378) 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 -- 172.21.15.141:0/1680521348 learned_addr learned my addr 172.21.15.141:0/1680521348 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 -- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f823013cfa0 msgr2=0x7f82301518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f823013cfa0 0x7f82301518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 -- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f823013fcb0 msgr2=0x7f823013e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f823013fcb0 0x7f823013e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 -- 172.21.15.141:0/1680521348 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228009a70 con 0x7f82300cd530 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8236000640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82300cd530 0x7f8230157410 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f822000d800 tx=0x7f822000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f822e000640 1 -- 172.21.15.141:0/1680521348 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000b6c0 con 0x7f82300cd530 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f822e000640 1 -- 172.21.15.141:0/1680521348 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822000b820 con 0x7f82300cd530 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f822e000640 1 -- 172.21.15.141:0/1680521348 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000bbb0 con 0x7f82300cd530 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.209+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8230166570 con 0x7f82300cd530 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.210+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8230166b30 con 0x7f82300cd530 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.211+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8230067450 con 0x7f82300cd530 2024-01-16T15:09:46.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.212+0000 7f822e000640 1 -- 172.21.15.141:0/1680521348 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f822000bd10 con 0x7f82300cd530 2024-01-16T15:09:46.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.212+0000 7f822e000640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f820806bfb0 0x7f820806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.212+0000 7f822e000640 1 -- 172.21.15.141:0/1680521348 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8220088e40 con 0x7f82300cd530 2024-01-16T15:09:46.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.213+0000 7f8235600640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f820806bfb0 0x7f820806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.214+0000 7f8235600640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f820806bfb0 0x7f820806e470 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f82301f65d0 tx=0x7f8218007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.217+0000 7f822e000640 1 -- 172.21.15.141:0/1680521348 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8220055440 con 0x7f82300cd530 2024-01-16T15:09:46.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.392+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82300cbff0 con 0x7f82300cd530 2024-01-16T15:09:46.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.394+0000 7f822e000640 1 -- 172.21.15.141:0/1680521348 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+13037 (secure 0 0 0) 0x7f8220058a60 con 0x7f82300cd530 2024-01-16T15:09:46.371 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:46.372 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":80,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:09:45.570741+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/3457901158":"2024-01-17T15:09:42.558294+0000","172.21.15.141:6834/3457901158":"2024-01-17T15:09:42.558294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:46.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.396+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f820806bfb0 msgr2=0x7f820806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.396+0000 7f8237400640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f820806bfb0 0x7f820806e470 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f82301f65d0 tx=0x7f8218007930 comp rx=0 tx=0).stop 2024-01-16T15:09:46.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.396+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82300cd530 msgr2=0x7f8230157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.396+0000 7f8237400640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82300cd530 0x7f8230157410 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f822000d800 tx=0x7f822000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 shutdown_connections 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f820806bfb0 0x7f820806e470 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f823013fcb0 0x7f823013e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f823013cfa0 0x7f82301518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 --2- 172.21.15.141:0/1680521348 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82300cd530 0x7f8230157410 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 >> 172.21.15.141:0/1680521348 conn(0x7f823006a800 msgr2=0x7f82300b2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 shutdown_connections 2024-01-16T15:09:46.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.397+0000 7f8237400640 1 -- 172.21.15.141:0/1680521348 wait complete. 2024-01-16T15:09:46.391 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:46.392 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:46.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.620+0000 7f31e4a00640 1 Processor -- start 2024-01-16T15:09:46.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.621+0000 7f31e4a00640 1 -- start start 2024-01-16T15:09:46.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.622+0000 7f31e4a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f31dc13e360 0x7f31dc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.622+0000 7f31e4a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f31dc15f9c0 0x7f31dc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.622+0000 7f31e4a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc15c5f0 con 0x7f31dc13df80 2024-01-16T15:09:46.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.622+0000 7f31e4a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc06c470 con 0x7f31dc15f9c0 2024-01-16T15:09:46.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.622+0000 7f31e4a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc06c750 con 0x7f31dc13e360 2024-01-16T15:09:46.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.622+0000 7f31e2000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f31dc13df80 0x7f31dc15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33498/0 (socket says 172.21.15.141:33498) 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.622+0000 7f31e2000640 1 -- 172.21.15.141:0/752530720 learned_addr learned my addr 172.21.15.141:0/752530720 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e2a00640 1 --2- 172.21.15.141:0/752530720 >> v2:172.21.15.141:3300/0 conn(0x7f31dc15f9c0 0x7f31dc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e1600640 1 --2- 172.21.15.141:0/752530720 >> v2:172.21.15.188:3300/0 conn(0x7f31dc13e360 0x7f31dc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e0c00640 1 -- 172.21.15.141:0/752530720 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223332232 0 0) 0x7f31dc15c5f0 con 0x7f31dc13df80 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e0c00640 1 -- 172.21.15.141:0/752530720 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc003520 con 0x7f31dc13df80 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e2a00640 1 -- 172.21.15.141:0/752530720 >> v2:172.21.15.188:3300/0 conn(0x7f31dc13e360 msgr2=0x7f31dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e2a00640 1 --2- 172.21.15.141:0/752530720 >> v2:172.21.15.188:3300/0 conn(0x7f31dc13e360 0x7f31dc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e2a00640 1 -- 172.21.15.141:0/752530720 >> v1:172.21.15.141:6789/0 conn(0x7f31dc13df80 legacy=0x7f31dc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.623+0000 7f31e2a00640 1 -- 172.21.15.141:0/752530720 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc06ca30 con 0x7f31dc15f9c0 2024-01-16T15:09:46.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.624+0000 7f31e2a00640 1 --2- 172.21.15.141:0/752530720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc15f9c0 0x7f31dc0cf7d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f31dc06ded0 tx=0x7f31d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddf7282ba33f96a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e0c00640 1 -- 172.21.15.141:0/752530720 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d402cc80 con 0x7f31dc15f9c0 2024-01-16T15:09:46.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e0c00640 1 -- 172.21.15.141:0/752530720 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d402cde0 con 0x7f31dc15f9c0 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e0c00640 1 -- 172.21.15.141:0/752530720 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4034660 con 0x7f31dc15f9c0 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e4a00640 1 -- 172.21.15.141:0/752530720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc15f9c0 msgr2=0x7f31dc0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e4a00640 1 --2- 172.21.15.141:0/752530720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc15f9c0 0x7f31dc0cf7d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f31dc06ded0 tx=0x7f31d402bed0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e4a00640 1 -- 172.21.15.141:0/752530720 shutdown_connections 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e4a00640 1 --2- 172.21.15.141:0/752530720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc15f9c0 0x7f31dc0cf7d0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e4a00640 1 --2- 172.21.15.141:0/752530720 >> v2:172.21.15.188:3300/0 conn(0x7f31dc13e360 0x7f31dc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e4a00640 1 -- 172.21.15.141:0/752530720 >> 172.21.15.141:0/752530720 conn(0x7f31dc06a800 msgr2=0x7f31dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.625+0000 7f31e4a00640 1 -- 172.21.15.141:0/752530720 shutdown_connections 2024-01-16T15:09:46.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.626+0000 7f31e4a00640 1 -- 172.21.15.141:0/752530720 wait complete. 2024-01-16T15:09:46.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.627+0000 7f31e4a00640 1 Processor -- start 2024-01-16T15:09:46.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.627+0000 7f31e4a00640 1 -- start start 2024-01-16T15:09:46.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.628+0000 7f31e4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31dc13df80 0x7f31dc135770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.628+0000 7f31e4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc13e360 0x7f31dc0a8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.628+0000 7f31e4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31dc15f9c0 0x7f31dc1f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.628+0000 7f31e4a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc1535f0 con 0x7f31dc13e360 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.628+0000 7f31e4a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc135cb0 con 0x7f31dc13df80 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.628+0000 7f31e4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc1529e0 con 0x7f31dc15f9c0 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.628+0000 7f31e1600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc13e360 0x7f31dc0a8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc13e360 0x7f31dc0a8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39386/0 (socket says 172.21.15.141:39386) 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 -- 172.21.15.141:0/2940407045 learned_addr learned my addr 172.21.15.141:0/2940407045 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e2000640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31dc13df80 0x7f31dc135770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 -- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31dc15f9c0 msgr2=0x7f31dc1f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e2a00640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31dc15f9c0 0x7f31dc1f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31dc15f9c0 0x7f31dc1f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 -- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31dc13df80 msgr2=0x7f31dc135770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31dc13df80 0x7f31dc135770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 -- 172.21.15.141:0/2940407045 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4009a70 con 0x7f31dc13e360 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e2000640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31dc13df80 0x7f31dc135770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e1600640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc13e360 0x7f31dc0a8dd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f31c400d7d0 tx=0x7f31c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31da000640 1 -- 172.21.15.141:0/2940407045 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4018070 con 0x7f31dc13e360 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31da000640 1 -- 172.21.15.141:0/2940407045 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c4004020 con 0x7f31dc13e360 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc152cc0 con 0x7f31dc13e360 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.629+0000 7f31da000640 1 -- 172.21.15.141:0/2940407045 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40041b0 con 0x7f31dc13e360 2024-01-16T15:09:46.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.630+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc153280 con 0x7f31dc13e360 2024-01-16T15:09:46.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.631+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc0675d0 con 0x7f31dc13e360 2024-01-16T15:09:46.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.632+0000 7f31da000640 1 -- 172.21.15.141:0/2940407045 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f31c4004310 con 0x7f31dc13e360 2024-01-16T15:09:46.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.632+0000 7f31da000640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31b406bfb0 0x7f31b406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.632+0000 7f31da000640 1 -- 172.21.15.141:0/2940407045 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f31c40886d0 con 0x7f31dc13e360 2024-01-16T15:09:46.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.633+0000 7f31e2000640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31b406bfb0 0x7f31b406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.634+0000 7f31e2000640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31b406bfb0 0x7f31b406e470 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f31cc005880 tx=0x7f31cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.637+0000 7f31da000640 1 -- 172.21.15.141:0/2940407045 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31c4054d60 con 0x7f31dc13e360 2024-01-16T15:09:46.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.811+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31dc0b42a0 con 0x7f31dc13e360 2024-01-16T15:09:46.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.812+0000 7f31da000640 1 -- 172.21.15.141:0/2940407045 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31c4058380 con 0x7f31dc13e360 2024-01-16T15:09:46.789 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:09:46.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31b406bfb0 msgr2=0x7f31b406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31b406bfb0 0x7f31b406e470 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f31cc005880 tx=0x7f31cc009040 comp rx=0 tx=0).stop 2024-01-16T15:09:46.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc13e360 msgr2=0x7f31dc0a8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc13e360 0x7f31dc0a8dd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f31c400d7d0 tx=0x7f31c400dc90 comp rx=0 tx=0).stop 2024-01-16T15:09:46.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 shutdown_connections 2024-01-16T15:09:46.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f31b406bfb0 0x7f31b406e470 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31dc15f9c0 0x7f31dc1f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31dc13e360 0x7f31dc0a8dd0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 --2- 172.21.15.141:0/2940407045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31dc13df80 0x7f31dc135770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 >> 172.21.15.141:0/2940407045 conn(0x7f31dc06a800 msgr2=0x7f31dc0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 shutdown_connections 2024-01-16T15:09:46.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:46.815+0000 7f31e4a00640 1 -- 172.21.15.141:0/2940407045 wait complete. 2024-01-16T15:09:46.810 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:09:47.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.023+0000 7fb89a600640 1 Processor -- start 2024-01-16T15:09:47.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.023+0000 7fb89a600640 1 -- start start 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.024+0000 7fb89a600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb894143c40 0x7fb894071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.024+0000 7fb89a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940717d0 0x7fb8940cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.024+0000 7fb89a600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb8940cca40 0x7fb89415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.024+0000 7fb89a600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8940723c0 con 0x7fb894143c40 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb89a600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb894070e50 con 0x7fb8940cca40 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb89a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940cce70 con 0x7fb8940717d0 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb894143c40 0x7fb894071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb894143c40 0x7fb894071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39398/0 (socket says 172.21.15.141:39398) 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 -- 172.21.15.141:0/2340985270 learned_addr learned my addr 172.21.15.141:0/2340985270 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:47.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 -- 172.21.15.141:0/2340985270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940717d0 msgr2=0x7fb8940cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 --2- 172.21.15.141:0/2340985270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940717d0 0x7fb8940cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 -- 172.21.15.141:0/2340985270 >> v2:172.21.15.188:3300/0 conn(0x7fb8940cca40 msgr2=0x7fb89415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 --2- 172.21.15.141:0/2340985270 >> v2:172.21.15.188:3300/0 conn(0x7fb8940cca40 0x7fb89415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.025+0000 7fb899200640 1 -- 172.21.15.141:0/2340985270 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894144020 con 0x7fb894143c40 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.026+0000 7fb899200640 1 --2- 172.21.15.141:0/2340985270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb894143c40 0x7fb894071290 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb8940cc3f0 tx=0x7fb87c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb939a9dc0cd5fdd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb893e00640 1 -- 172.21.15.141:0/2340985270 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0317d0 con 0x7fb894143c40 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb893e00640 1 -- 172.21.15.141:0/2340985270 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c031930 con 0x7fb894143c40 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb893e00640 1 -- 172.21.15.141:0/2340985270 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c031cc0 con 0x7fb894143c40 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 -- 172.21.15.141:0/2340985270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb894143c40 msgr2=0x7fb894071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 --2- 172.21.15.141:0/2340985270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb894143c40 0x7fb894071290 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb8940cc3f0 tx=0x7fb87c02b870 comp rx=0 tx=0).stop 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 -- 172.21.15.141:0/2340985270 shutdown_connections 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 --2- 172.21.15.141:0/2340985270 >> v2:172.21.15.188:3300/0 conn(0x7fb8940cca40 0x7fb89415eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 --2- 172.21.15.141:0/2340985270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940717d0 0x7fb8940cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 --2- 172.21.15.141:0/2340985270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb894143c40 0x7fb894071290 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 -- 172.21.15.141:0/2340985270 >> 172.21.15.141:0/2340985270 conn(0x7fb89406a800 msgr2=0x7fb89406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 -- 172.21.15.141:0/2340985270 shutdown_connections 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.027+0000 7fb89a600640 1 -- 172.21.15.141:0/2340985270 wait complete. 2024-01-16T15:09:47.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.029+0000 7fb89a600640 1 Processor -- start 2024-01-16T15:09:47.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.029+0000 7fb89a600640 1 -- start start 2024-01-16T15:09:47.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb89a600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8940717d0 0x7fb894142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb89a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940cca40 0x7fb894158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb89a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb894143c40 0x7fb8941589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb89a600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89413a700 con 0x7fb8940717d0 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb89a600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940a9170 con 0x7fb894143c40 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb89a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89413a9a0 con 0x7fb8940cca40 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb899200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8940717d0 0x7fb894142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb899200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8940717d0 0x7fb894142180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39414/0 (socket says 172.21.15.141:39414) 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.030+0000 7fb899200640 1 -- 172.21.15.141:0/3990232978 learned_addr learned my addr 172.21.15.141:0/3990232978 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb88be00640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940cca40 0x7fb894158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb899200640 1 -- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940cca40 msgr2=0x7fb894158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb899200640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940cca40 0x7fb894158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb899200640 1 -- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb894143c40 msgr2=0x7fb8941589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb899c00640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb894143c40 0x7fb8941589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb899200640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb894143c40 0x7fb8941589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb899200640 1 -- 172.21.15.141:0/3990232978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0094d0 con 0x7fb8940717d0 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb899200640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8940717d0 0x7fb894142180 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fb87c02b8b0 tx=0x7fb87c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb891600640 1 -- 172.21.15.141:0/3990232978 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c032930 con 0x7fb8940717d0 2024-01-16T15:09:47.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb891600640 1 -- 172.21.15.141:0/3990232978 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c032ab0 con 0x7fb8940717d0 2024-01-16T15:09:47.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb891600640 1 -- 172.21.15.141:0/3990232978 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c032e60 con 0x7fb8940717d0 2024-01-16T15:09:47.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.031+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89413ac80 con 0x7fb8940717d0 2024-01-16T15:09:47.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.032+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8940a9450 con 0x7fb8940717d0 2024-01-16T15:09:47.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.033+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb894067450 con 0x7fb8940717d0 2024-01-16T15:09:47.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.033+0000 7fb891600640 1 -- 172.21.15.141:0/3990232978 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fb87c030070 con 0x7fb8940717d0 2024-01-16T15:09:47.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.034+0000 7fb891600640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86c06bfb0 0x7fb86c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.034+0000 7fb891600640 1 -- 172.21.15.141:0/3990232978 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb87c0aa440 con 0x7fb8940717d0 2024-01-16T15:09:47.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.035+0000 7fb88be00640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86c06bfb0 0x7fb86c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.036+0000 7fb88be00640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86c06bfb0 0x7fb86c06e470 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb89413a630 tx=0x7fb884008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.038+0000 7fb891600640 1 -- 172.21.15.141:0/3990232978 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb87c076a70 con 0x7fb8940717d0 2024-01-16T15:09:47.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.243+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb89413ae40 con 0x7fb8940717d0 2024-01-16T15:09:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.244+0000 7fb891600640 1 -- 172.21.15.141:0/3990232978 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v80) v1 ==== 80+0+124 (secure 0 0 0) 0x7fb87c07a090 con 0x7fb8940717d0 2024-01-16T15:09:47.221 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/3457901158 2024-01-17T15:09:42.558294+0000 2024-01-16T15:09:47.221 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/3457901158 2024-01-17T15:09:42.558294+0000 2024-01-16T15:09:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:09:47.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86c06bfb0 msgr2=0x7fb86c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86c06bfb0 0x7fb86c06e470 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb89413a630 tx=0x7fb884008040 comp rx=0 tx=0).stop 2024-01-16T15:09:47.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8940717d0 msgr2=0x7fb894142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8940717d0 0x7fb894142180 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fb87c02b8b0 tx=0x7fb87c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 shutdown_connections 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86c06bfb0 0x7fb86c06e470 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb894143c40 0x7fb8941589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb8940cca40 0x7fb894158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 --2- 172.21.15.141:0/3990232978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8940717d0 0x7fb894142180 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.247+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 >> 172.21.15.141:0/3990232978 conn(0x7fb89406a800 msgr2=0x7fb89415a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.248+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 shutdown_connections 2024-01-16T15:09:47.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.248+0000 7fb89a600640 1 -- 172.21.15.141:0/3990232978 wait complete. 2024-01-16T15:09:47.243 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T15:09:47.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:47.244 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:09:47.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[62201]: fuse finished with error 0 and tester_r 0 2024-01-16T15:09:47.337 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:09:47.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:47.337 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:09:47.583 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:09:47.583 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:47.640 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:47.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.879+0000 7f188ce00640 1 Processor -- start 2024-01-16T15:09:47.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.879+0000 7f188ce00640 1 -- start start 2024-01-16T15:09:47.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.881+0000 7f188ce00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813df80 0x7f188815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.881+0000 7f188ce00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f18880cf9a0 0x7f18880cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.881+0000 7f188ce00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f188806c470 con 0x7f188813e360 2024-01-16T15:09:47.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.881+0000 7f188ce00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f188806c750 con 0x7f18880cf9a0 2024-01-16T15:09:47.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.881+0000 7f188ce00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188806ca30 con 0x7f188813df80 2024-01-16T15:09:47.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.881+0000 7f1886800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f18880cf9a0 0x7f18880cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.881+0000 7f1885e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813df80 0x7f188815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f1885e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813df80 0x7f188815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:44094/0 (socket says 172.21.15.141:44094) 2024-01-16T15:09:47.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f1886800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f18880cf9a0 0x7f18880cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:54728/0 (socket says 172.21.15.141:54728) 2024-01-16T15:09:47.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f1885e00640 1 -- 172.21.15.141:0/37483521 learned_addr learned my addr 172.21.15.141:0/37483521 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:47.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f187fe00640 1 --1- 172.21.15.141:0/37483521 >> v1:172.21.15.188:6789/0 conn(0x7f188813e360 0x7f18880cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:49744/0 (socket says 172.21.15.141:49744) 2024-01-16T15:09:47.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f1885e00640 1 -- 172.21.15.141:0/37483521 >> v2:172.21.15.188:3300/0 conn(0x7f18880cf9a0 msgr2=0x7f18880cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f1885e00640 1 --2- 172.21.15.141:0/37483521 >> v2:172.21.15.188:3300/0 conn(0x7f18880cf9a0 0x7f18880cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f1885e00640 1 -- 172.21.15.141:0/37483521 >> v1:172.21.15.188:6789/0 conn(0x7f188813e360 legacy=0x7f18880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.882+0000 7f1885e00640 1 -- 172.21.15.141:0/37483521 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188806cd10 con 0x7f188813df80 2024-01-16T15:09:47.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.884+0000 7f1885e00640 1 --2- 172.21.15.141:0/37483521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813df80 0x7f188815bee0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1878009870 tx=0x7f187802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51c6055277f81bd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.884+0000 7f1885400640 1 -- 172.21.15.141:0/37483521 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1878030840 con 0x7f188813df80 2024-01-16T15:09:47.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.884+0000 7f1885400640 1 -- 172.21.15.141:0/37483521 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18780309a0 con 0x7f188813df80 2024-01-16T15:09:47.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.884+0000 7f1885400640 1 -- 172.21.15.141:0/37483521 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1878030d30 con 0x7f188813df80 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 -- 172.21.15.141:0/37483521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813df80 msgr2=0x7f188815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 --2- 172.21.15.141:0/37483521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813df80 0x7f188815bee0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1878009870 tx=0x7f187802b980 comp rx=0 tx=0).stop 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 -- 172.21.15.141:0/37483521 shutdown_connections 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 --2- 172.21.15.141:0/37483521 >> v2:172.21.15.188:3300/0 conn(0x7f18880cf9a0 0x7f18880cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 --2- 172.21.15.141:0/37483521 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813df80 0x7f188815bee0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 -- 172.21.15.141:0/37483521 >> 172.21.15.141:0/37483521 conn(0x7f188806a800 msgr2=0x7f188806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 -- 172.21.15.141:0/37483521 shutdown_connections 2024-01-16T15:09:47.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.885+0000 7f188ce00640 1 -- 172.21.15.141:0/37483521 wait complete. 2024-01-16T15:09:47.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.887+0000 7f188ce00640 1 Processor -- start 2024-01-16T15:09:47.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.887+0000 7f188ce00640 1 -- start start 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.888+0000 7f188ce00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18880cf9a0 0x7f18881501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.888+0000 7f188ce00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f188813df80 0x7f18880afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.888+0000 7f188ce00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813e360 0x7f18880b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.888+0000 7f188ce00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1888158690 con 0x7f188813df80 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.888+0000 7f188ce00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18881506f0 con 0x7f18880cf9a0 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.888+0000 7f188ce00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188813dd20 con 0x7f188813e360 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f188813df80 0x7f18880afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f188813df80 0x7f18880afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39424/0 (socket says 172.21.15.141:39424) 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 -- 172.21.15.141:0/2543012958 learned_addr learned my addr 172.21.15.141:0/2543012958 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f1886800640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813e360 0x7f18880b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f1885e00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18880cf9a0 0x7f18881501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 -- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813e360 msgr2=0x7f18880b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813e360 0x7f18880b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 -- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18880cf9a0 msgr2=0x7f18881501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18880cf9a0 0x7f18881501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f187fe00640 1 -- 172.21.15.141:0/2543012958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780094d0 con 0x7f188813df80 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.889+0000 7f1886800640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813e360 0x7f18880b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.890+0000 7f187fe00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f188813df80 0x7f18880afb80 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f187000c4d0 tx=0x7f187000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.890+0000 7f187e000640 1 -- 172.21.15.141:0/2543012958 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1870007480 con 0x7f188813df80 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.890+0000 7f187e000640 1 -- 172.21.15.141:0/2543012958 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18700075e0 con 0x7f188813df80 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.890+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188812e030 con 0x7f188813df80 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.890+0000 7f187e000640 1 -- 172.21.15.141:0/2543012958 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18700049e0 con 0x7f188813df80 2024-01-16T15:09:47.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.890+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188812b740 con 0x7f188813df80 2024-01-16T15:09:47.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.891+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18880675d0 con 0x7f188813df80 2024-01-16T15:09:47.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.892+0000 7f187e000640 1 -- 172.21.15.141:0/2543012958 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1870004b40 con 0x7f188813df80 2024-01-16T15:09:47.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.893+0000 7f187e000640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f186006bfb0 0x7f186006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.893+0000 7f187e000640 1 -- 172.21.15.141:0/2543012958 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1870087690 con 0x7f188813df80 2024-01-16T15:09:47.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.894+0000 7f1885e00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f186006bfb0 0x7f186006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.895+0000 7f1885e00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f186006bfb0 0x7f186006e470 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1878036000 tx=0x7f187803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:47.898+0000 7f187e000640 1 -- 172.21.15.141:0/2543012958 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1870054000 con 0x7f188813df80 2024-01-16T15:09:48.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.089+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f188806df00 con 0x7f188813df80 2024-01-16T15:09:48.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.091+0000 7f187e000640 1 -- 172.21.15.141:0/2543012958 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+13037 (secure 0 0 0) 0x7f1870018070 con 0x7f188813df80 2024-01-16T15:09:48.068 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:48.069 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":80,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:09:45.570741+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/3457901158":"2024-01-17T15:09:42.558294+0000","172.21.15.141:6834/3457901158":"2024-01-17T15:09:42.558294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:48.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.093+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f186006bfb0 msgr2=0x7f186006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.093+0000 7f188ce00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f186006bfb0 0x7f186006e470 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1878036000 tx=0x7f187803b040 comp rx=0 tx=0).stop 2024-01-16T15:09:48.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.093+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f188813df80 msgr2=0x7f18880afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.093+0000 7f188ce00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f188813df80 0x7f18880afb80 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f187000c4d0 tx=0x7f187000c990 comp rx=0 tx=0).stop 2024-01-16T15:09:48.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 shutdown_connections 2024-01-16T15:09:48.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f186006bfb0 0x7f186006e470 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f188813e360 0x7f18880b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f188813df80 0x7f18880afb80 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 --2- 172.21.15.141:0/2543012958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18880cf9a0 0x7f18881501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 >> 172.21.15.141:0/2543012958 conn(0x7f188806a800 msgr2=0x7f188815d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 shutdown_connections 2024-01-16T15:09:48.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.094+0000 7f188ce00640 1 -- 172.21.15.141:0/2543012958 wait complete. 2024-01-16T15:09:48.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:48.090 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:48.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.327+0000 7fd6c3e00640 1 Processor -- start 2024-01-16T15:09:48.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.327+0000 7fd6c3e00640 1 -- start start 2024-01-16T15:09:48.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.329+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0b3c10 0x7fd6bc0713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.329+0000 7fd6c3e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd6bc0ce420 0x7fd6bc0ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.329+0000 7fd6c3e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc0cdf90 con 0x7fd6bc0b4280 2024-01-16T15:09:48.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.329+0000 7fd6c3e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6bc072000 con 0x7fd6bc0ce420 2024-01-16T15:09:48.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.329+0000 7fd6c3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc071d20 con 0x7fd6bc0b3c10 2024-01-16T15:09:48.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.330+0000 7fd6c0a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0b3c10 0x7fd6bc0713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.330+0000 7fd6c0a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0b3c10 0x7fd6bc0713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:44104/0 (socket says 172.21.15.141:44104) 2024-01-16T15:09:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.330+0000 7fd6c0a00640 1 -- 172.21.15.141:0/1516549275 learned_addr learned my addr 172.21.15.141:0/1516549275 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.330+0000 7fd6c1e00640 1 --2- 172.21.15.141:0/1516549275 >> v2:172.21.15.188:3300/0 conn(0x7fd6bc0ce420 0x7fd6bc0ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.331+0000 7fd6c0a00640 1 -- 172.21.15.141:0/1516549275 >> v2:172.21.15.188:3300/0 conn(0x7fd6bc0ce420 msgr2=0x7fd6bc0ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.331+0000 7fd6c0a00640 1 --2- 172.21.15.141:0/1516549275 >> v2:172.21.15.188:3300/0 conn(0x7fd6bc0ce420 0x7fd6bc0ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.331+0000 7fd6c0a00640 1 -- 172.21.15.141:0/1516549275 >> v1:172.21.15.188:6789/0 conn(0x7fd6bc0b4280 legacy=0x7fd6bc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.331+0000 7fd6c0a00640 1 -- 172.21.15.141:0/1516549275 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc0cd940 con 0x7fd6bc0b3c10 2024-01-16T15:09:48.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.332+0000 7fd6c0a00640 1 --2- 172.21.15.141:0/1516549275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0b3c10 0x7fd6bc0713d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400d180 tx=0x7fd6a402e160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0b7c1109cec02ba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.332+0000 7fd6bbe00640 1 -- 172.21.15.141:0/1516549275 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400b0d0 con 0x7fd6bc0b3c10 2024-01-16T15:09:48.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6bbe00640 1 -- 172.21.15.141:0/1516549275 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a400b230 con 0x7fd6bc0b3c10 2024-01-16T15:09:48.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6bbe00640 1 -- 172.21.15.141:0/1516549275 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4009600 con 0x7fd6bc0b3c10 2024-01-16T15:09:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 -- 172.21.15.141:0/1516549275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0b3c10 msgr2=0x7fd6bc0713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/1516549275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0b3c10 0x7fd6bc0713d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400d180 tx=0x7fd6a402e160 comp rx=0 tx=0).stop 2024-01-16T15:09:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 -- 172.21.15.141:0/1516549275 shutdown_connections 2024-01-16T15:09:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/1516549275 >> v2:172.21.15.188:3300/0 conn(0x7fd6bc0ce420 0x7fd6bc0ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/1516549275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0b3c10 0x7fd6bc0713d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 -- 172.21.15.141:0/1516549275 >> 172.21.15.141:0/1516549275 conn(0x7fd6bc0cfef0 msgr2=0x7fd6bc0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 -- 172.21.15.141:0/1516549275 shutdown_connections 2024-01-16T15:09:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.333+0000 7fd6c3e00640 1 -- 172.21.15.141:0/1516549275 wait complete. 2024-01-16T15:09:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.335+0000 7fd6c3e00640 1 Processor -- start 2024-01-16T15:09:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.335+0000 7fd6c3e00640 1 -- start start 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.336+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6bc0b3c10 0x7fd6bc136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.336+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6bc0b4280 0x7fd6bc15b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.336+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0ce420 0x7fd6bc149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.336+0000 7fd6c3e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc13fe70 con 0x7fd6bc0b4280 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.336+0000 7fd6c3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc15bd80 con 0x7fd6bc0b3c10 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.336+0000 7fd6c3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1397c0 con 0x7fd6bc0ce420 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6bc0b4280 0x7fd6bc15b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6bc0b4280 0x7fd6bc15b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39430/0 (socket says 172.21.15.141:39430) 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 -- 172.21.15.141:0/3877096458 learned_addr learned my addr 172.21.15.141:0/3877096458 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c1e00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0ce420 0x7fd6bc149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c1400640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6bc0b3c10 0x7fd6bc136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 -- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0ce420 msgr2=0x7fd6bc149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0ce420 0x7fd6bc149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 -- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6bc0b3c10 msgr2=0x7fd6bc136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6bc0b3c10 0x7fd6bc136430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 -- 172.21.15.141:0/3877096458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a400bd40 con 0x7fd6bc0b4280 2024-01-16T15:09:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.337+0000 7fd6c0a00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6bc0b4280 0x7fd6bc15b440 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4038040 tx=0x7fd6a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.338+0000 7fd6b9600640 1 -- 172.21.15.141:0/3877096458 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4009e50 con 0x7fd6bc0b4280 2024-01-16T15:09:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.338+0000 7fd6b9600640 1 -- 172.21.15.141:0/3877096458 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a4040660 con 0x7fd6bc0b4280 2024-01-16T15:09:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.338+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc139240 con 0x7fd6bc0b4280 2024-01-16T15:09:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.338+0000 7fd6b9600640 1 -- 172.21.15.141:0/3877096458 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4040a10 con 0x7fd6bc0b4280 2024-01-16T15:09:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.338+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc1394e0 con 0x7fd6bc0b4280 2024-01-16T15:09:48.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.339+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc0675d0 con 0x7fd6bc0b4280 2024-01-16T15:09:48.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.340+0000 7fd6b9600640 1 -- 172.21.15.141:0/3877096458 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fd6a4040b70 con 0x7fd6bc0b4280 2024-01-16T15:09:48.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.341+0000 7fd6b9600640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd69406bfb0 0x7fd69406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.341+0000 7fd6b9600640 1 -- 172.21.15.141:0/3877096458 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd6a40ad900 con 0x7fd6bc0b4280 2024-01-16T15:09:48.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.341+0000 7fd6c1400640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd69406bfb0 0x7fd69406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.321 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.342+0000 7fd6c1400640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd69406bfb0 0x7fd69406e470 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac000f60 tx=0x7fd6ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.345+0000 7fd6b9600640 1 -- 172.21.15.141:0/3877096458 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6a4079f50 con 0x7fd6bc0b4280 2024-01-16T15:09:48.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.532+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6bc143570 con 0x7fd6bc0b4280 2024-01-16T15:09:48.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.533+0000 7fd6b9600640 1 -- 172.21.15.141:0/3877096458 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6a4045020 con 0x7fd6bc0b4280 2024-01-16T15:09:48.511 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.535+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd69406bfb0 msgr2=0x7fd69406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.535+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd69406bfb0 0x7fd69406e470 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac000f60 tx=0x7fd6ac009040 comp rx=0 tx=0).stop 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.535+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6bc0b4280 msgr2=0x7fd6bc15b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.535+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6bc0b4280 0x7fd6bc15b440 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4038040 tx=0x7fd6a4004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 shutdown_connections 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd69406bfb0 0x7fd69406e470 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd6bc0ce420 0x7fd6bc149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6bc0b4280 0x7fd6bc15b440 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 --2- 172.21.15.141:0/3877096458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6bc0b3c10 0x7fd6bc136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 >> 172.21.15.141:0/3877096458 conn(0x7fd6bc0cfef0 msgr2=0x7fd6bc142a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 shutdown_connections 2024-01-16T15:09:48.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.536+0000 7fd6c3e00640 1 -- 172.21.15.141:0/3877096458 wait complete. 2024-01-16T15:09:48.533 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:09:48.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.768+0000 7f78bb600640 1 Processor -- start 2024-01-16T15:09:48.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.768+0000 7f78bb600640 1 -- start start 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78bb600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f78b40b3ae0 0x7f78b415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78bb600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f78b40cf290 0x7f78b40cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78bb600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b4070c40 con 0x7f78b40ae5a0 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78bb600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b4071450 con 0x7f78b40b3ae0 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78bb600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b40cfc60 con 0x7f78b40cf290 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78b3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f78b40b3ae0 0x7f78b415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78b3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f78b40b3ae0 0x7f78b415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39436/0 (socket says 172.21.15.141:39436) 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.770+0000 7f78b3e00640 1 -- 172.21.15.141:0/1447674046 learned_addr learned my addr 172.21.15.141:0/1447674046 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.771+0000 7f78b9600640 1 --2- 172.21.15.141:0/1447674046 >> v2:172.21.15.188:3300/0 conn(0x7f78b40cf290 0x7f78b40cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.771+0000 7f78b3e00640 1 -- 172.21.15.141:0/1447674046 >> v2:172.21.15.188:3300/0 conn(0x7f78b40cf290 msgr2=0x7f78b40cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.771+0000 7f78b3e00640 1 --2- 172.21.15.141:0/1447674046 >> v2:172.21.15.188:3300/0 conn(0x7f78b40cf290 0x7f78b40cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.771+0000 7f78b3e00640 1 -- 172.21.15.141:0/1447674046 >> v1:172.21.15.188:6789/0 conn(0x7f78b40ae5a0 legacy=0x7f78b406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.771+0000 7f78b3e00640 1 -- 172.21.15.141:0/1447674046 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b406c470 con 0x7f78b40b3ae0 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.772+0000 7f78b3e00640 1 --2- 172.21.15.141:0/1447674046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40b3ae0 0x7f78b415e430 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f78b40b3f70 tx=0x7f789c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44a2aa631469fd1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.772+0000 7f78b3400640 1 -- 172.21.15.141:0/1447674046 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c02f850 con 0x7f78b40b3ae0 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.772+0000 7f78b3400640 1 -- 172.21.15.141:0/1447674046 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789c02f9b0 con 0x7f78b40b3ae0 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.772+0000 7f78b3400640 1 -- 172.21.15.141:0/1447674046 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c02fd40 con 0x7f78b40b3ae0 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 -- 172.21.15.141:0/1447674046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40b3ae0 msgr2=0x7f78b415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 --2- 172.21.15.141:0/1447674046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40b3ae0 0x7f78b415e430 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f78b40b3f70 tx=0x7f789c02ba30 comp rx=0 tx=0).stop 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 -- 172.21.15.141:0/1447674046 shutdown_connections 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 --2- 172.21.15.141:0/1447674046 >> v2:172.21.15.188:3300/0 conn(0x7f78b40cf290 0x7f78b40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 --2- 172.21.15.141:0/1447674046 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40b3ae0 0x7f78b415e430 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 -- 172.21.15.141:0/1447674046 >> 172.21.15.141:0/1447674046 conn(0x7f78b406a800 msgr2=0x7f78b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 -- 172.21.15.141:0/1447674046 shutdown_connections 2024-01-16T15:09:48.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.773+0000 7f78bb600640 1 -- 172.21.15.141:0/1447674046 wait complete. 2024-01-16T15:09:48.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.775+0000 7f78bb600640 1 Processor -- start 2024-01-16T15:09:48.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.775+0000 7f78bb600640 1 -- start start 2024-01-16T15:09:48.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78bb600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b40ae5a0 0x7f78b40ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78bb600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f78b40b3ae0 0x7f78b40ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78bb600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40cf290 0x7f78b414fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78bb600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40ceb60 con 0x7f78b40cf290 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78bb600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b41f64f0 con 0x7f78b40ae5a0 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78bb600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b41f67d0 con 0x7f78b40b3ae0 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78b9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40cf290 0x7f78b414fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78b9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40cf290 0x7f78b414fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39438/0 (socket says 172.21.15.141:39438) 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.776+0000 7f78b9600640 1 -- 172.21.15.141:0/3065716347 learned_addr learned my addr 172.21.15.141:0/3065716347 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b8c00640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b40ae5a0 0x7f78b40ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b3e00640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f78b40b3ae0 0x7f78b40ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b9600640 1 -- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f78b40b3ae0 msgr2=0x7f78b40ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b9600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f78b40b3ae0 0x7f78b40ce620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b9600640 1 -- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b40ae5a0 msgr2=0x7f78b40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b9600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b40ae5a0 0x7f78b40ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b9600640 1 -- 172.21.15.141:0/3065716347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c0094d0 con 0x7f78b40cf290 2024-01-16T15:09:48.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b8c00640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b40ae5a0 0x7f78b40ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:48.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b9600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40cf290 0x7f78b414fa10 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00e7f0 tx=0x7f78ac00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b0c00640 1 -- 172.21.15.141:0/3065716347 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac009510 con 0x7f78b40cf290 2024-01-16T15:09:48.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b0c00640 1 -- 172.21.15.141:0/3065716347 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac00ee40 con 0x7f78b40cf290 2024-01-16T15:09:48.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78b0c00640 1 -- 172.21.15.141:0/3065716347 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00cb80 con 0x7f78b40cf290 2024-01-16T15:09:48.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.777+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b41f6ab0 con 0x7f78b40cf290 2024-01-16T15:09:48.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.778+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b41f7070 con 0x7f78b40cf290 2024-01-16T15:09:48.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.779+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b40675d0 con 0x7f78b40cf290 2024-01-16T15:09:48.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.780+0000 7f78b0c00640 1 -- 172.21.15.141:0/3065716347 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f78ac00cce0 con 0x7f78b40cf290 2024-01-16T15:09:48.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.780+0000 7f78b0c00640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f788c06bfb0 0x7f788c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.781+0000 7f78b0c00640 1 -- 172.21.15.141:0/3065716347 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f78ac0886d0 con 0x7f78b40cf290 2024-01-16T15:09:48.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.781+0000 7f78b8c00640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f788c06bfb0 0x7f788c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.782+0000 7f78b8c00640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f788c06bfb0 0x7f788c06e470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f78a40012e0 tx=0x7f78a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.785+0000 7f78b0c00640 1 -- 172.21.15.141:0/3065716347 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78ac054f90 con 0x7f78b40cf290 2024-01-16T15:09:48.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.989+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f78b40ceca0 con 0x7f78b40cf290 2024-01-16T15:09:48.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.990+0000 7f78b0c00640 1 -- 172.21.15.141:0/3065716347 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v80) v1 ==== 80+0+124 (secure 0 0 0) 0x7f78ac021080 con 0x7f78b40cf290 2024-01-16T15:09:48.968 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6835/3457901158 2024-01-17T15:09:42.558294+0000 2024-01-16T15:09:48.968 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/3457901158 2024-01-17T15:09:42.558294+0000 2024-01-16T15:09:48.969 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f788c06bfb0 msgr2=0x7f788c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f788c06bfb0 0x7f788c06e470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f78a40012e0 tx=0x7f78a4009040 comp rx=0 tx=0).stop 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40cf290 msgr2=0x7f78b414fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40cf290 0x7f78b414fa10 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00e7f0 tx=0x7f78ac00ebf0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 shutdown_connections 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f788c06bfb0 0x7f788c06e470 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78b40cf290 0x7f78b414fa10 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f78b40b3ae0 0x7f78b40ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 --2- 172.21.15.141:0/3065716347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b40ae5a0 0x7f78b40ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 >> 172.21.15.141:0/3065716347 conn(0x7f78b406a800 msgr2=0x7f78b415a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 shutdown_connections 2024-01-16T15:09:48.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:48.993+0000 7f78bb600640 1 -- 172.21.15.141:0/3065716347 wait complete. 2024-01-16T15:09:48.990 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T15:09:48.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:48.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:09:49.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[44998]: fuse finished with error 0 and tester_r 0 2024-01-16T15:09:49.091 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:09:49.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:49.091 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:09:49.266 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:09:49.266 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:49.322 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:09:49.322 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:09:49.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:49.322 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:09:49.372 INFO:teuthology.orchestra.run.smithi141.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:09:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:09:49.373 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:09:49.373 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:49.429 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:09:49.429 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:09:49.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:49.429 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:09:49.480 INFO:teuthology.orchestra.run.smithi188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:09:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:09:49.482 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:09:49.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.716+0000 7ff6f8800640 1 Processor -- start 2024-01-16T15:09:49.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.717+0000 7ff6f8800640 1 -- start start 2024-01-16T15:09:49.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.718+0000 7ff6f8800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 0x7ff6f0070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.718+0000 7ff6f8800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0071690 con 0x7ff6f014b2c0 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.718+0000 7ff6f8800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0071970 con 0x7ff6f0070ea0 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.718+0000 7ff6f8800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f014b6a0 con 0x7ff6f00cf9a0 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.718+0000 7ff6f5e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff6f014b2c0 0x7ff6f00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33512/0 (socket says 172.21.15.141:33512) 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.718+0000 7ff6f5e00640 1 -- 172.21.15.141:0/233640284 learned_addr learned my addr 172.21.15.141:0/233640284 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f5400640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 0x7ff6f0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831890388 0 0) 0x7ff6f0071690 con 0x7ff6f014b2c0 2024-01-16T15:09:49.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d0003520 con 0x7ff6f014b2c0 2024-01-16T15:09:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3423806068 0 0) 0x7ff6d0003520 con 0x7ff6f014b2c0 2024-01-16T15:09:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 msgr2=0x7ff6f0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 0x7ff6f0070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 >> v1:172.21.15.188:6789/0 conn(0x7ff6f0070ea0 legacy=0x7ff6f00cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0071c50 con 0x7ff6f014b2c0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ff6e0001670 con 0x7ff6f014b2c0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 >> v1:172.21.15.141:6789/0 conn(0x7ff6f014b2c0 legacy=0x7ff6f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.719+0000 7ff6f4a00640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d00039a0 0x7ff6d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f4a00640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f4a00640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 0x7ff6d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f014b6a0 con 0x7ff6d00047c0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d000bab0 con 0x7ff6d00039a0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d000bc70 con 0x7ff6f00cf9a0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f5400640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f5e00640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d00039a0 0x7ff6d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f6800640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 0x7ff6d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f5400640 1 -- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 msgr2=0x7ff6d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f5400640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 0x7ff6d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f5400640 1 -- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d00039a0 msgr2=0x7ff6d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:49.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f5400640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d00039a0 0x7ff6d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f5400640 1 -- 172.21.15.141:0/233640284 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0003520 con 0x7ff6d00047c0 2024-01-16T15:09:49.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.720+0000 7ff6f6800640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00cf9a0 0x7ff6d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:49.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.722+0000 7ff6f5400640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6d0004c40 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800ba30 tx=0x7ff6d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb880e6c14f4e2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.722+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8032630 con 0x7ff6d00047c0 2024-01-16T15:09:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.722+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d8032790 con 0x7ff6d00047c0 2024-01-16T15:09:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.722+0000 7ff6f4a00640 1 -- 172.21.15.141:0/233640284 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8032b20 con 0x7ff6d00047c0 2024-01-16T15:09:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.722+0000 7ff6f8800640 1 -- 172.21.15.141:0/233640284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 msgr2=0x7ff6d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.722+0000 7ff6f8800640 1 --2- 172.21.15.141:0/233640284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6d0004c40 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800ba30 tx=0x7ff6d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.722+0000 7ff6f6800640 1 -- 172.21.15.141:0/233640284 reap_dead start 2024-01-16T15:09:49.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.723+0000 7ff6f8800640 1 -- 172.21.15.141:0/233640284 shutdown_connections 2024-01-16T15:09:49.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.723+0000 7ff6f8800640 1 -- 172.21.15.141:0/233640284 >> 172.21.15.141:0/233640284 conn(0x7ff6f006a800 msgr2=0x7ff6f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:49.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.723+0000 7ff6f8800640 1 -- 172.21.15.141:0/233640284 shutdown_connections 2024-01-16T15:09:49.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.723+0000 7ff6f8800640 1 -- 172.21.15.141:0/233640284 wait complete. 2024-01-16T15:09:49.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.724+0000 7ff6f8800640 1 Processor -- start 2024-01-16T15:09:49.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.725+0000 7ff6f8800640 1 -- start start 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.725+0000 7ff6f8800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6f00ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.725+0000 7ff6f8800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00ab4a0 0x7ff6f00a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f8800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6f01fab10 0x7ff6f01faf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f8800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f01fb4d0 con 0x7ff6d00047c0 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f8800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f01f8930 con 0x7ff6f01fab10 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f8800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f01f8c10 con 0x7ff6f00ab4a0 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6f00ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6f00ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39460/0 (socket says 172.21.15.141:39460) 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 -- 172.21.15.141:0/2821694189 learned_addr learned my addr 172.21.15.141:0/2821694189 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 -- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00ab4a0 msgr2=0x7ff6f00a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00ab4a0 0x7ff6f00a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f6800640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6f01fab10 0x7ff6f01faf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 -- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6f01fab10 msgr2=0x7ff6f01faf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6f01fab10 0x7ff6f01faf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.726+0000 7ff6f5e00640 1 -- 172.21.15.141:0/2821694189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f01f8ef0 con 0x7ff6d00047c0 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.727+0000 7ff6f5e00640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6f00ade40 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000e7f0 tx=0x7ff6e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.727+0000 7ff6ee000640 1 -- 172.21.15.141:0/2821694189 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e000c8d0 con 0x7ff6d00047c0 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.727+0000 7ff6ee000640 1 -- 172.21.15.141:0/2821694189 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e000ca30 con 0x7ff6d00047c0 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.727+0000 7ff6ee000640 1 -- 172.21.15.141:0/2821694189 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e000cde0 con 0x7ff6d00047c0 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.727+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f01f91d0 con 0x7ff6d00047c0 2024-01-16T15:09:49.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.727+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f01f8630 con 0x7ff6d00047c0 2024-01-16T15:09:49.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.728+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f01f96f0 con 0x7ff6d00047c0 2024-01-16T15:09:49.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.729+0000 7ff6ee000640 1 -- 172.21.15.141:0/2821694189 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ff6e001d460 con 0x7ff6d00047c0 2024-01-16T15:09:49.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.730+0000 7ff6ee000640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6c806bfe0 0x7ff6c806e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.731+0000 7ff6ee000640 1 -- 172.21.15.141:0/2821694189 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff6e0088890 con 0x7ff6d00047c0 2024-01-16T15:09:49.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.731+0000 7ff6f5400640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6c806bfe0 0x7ff6c806e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.732+0000 7ff6f5400640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6c806bfe0 0x7ff6c806e4a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800ba00 tx=0x7ff6d8032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.735+0000 7ff6ee000640 1 -- 172.21.15.141:0/2821694189 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6f01f96f0 con 0x7ff6d00047c0 2024-01-16T15:09:49.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.983+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f01f96f0 con 0x7ff6d00047c0 2024-01-16T15:09:49.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.984+0000 7ff6ee000640 1 -- 172.21.15.141:0/2821694189 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3711 (secure 0 0 0) 0x7ff6f01f96f0 con 0x7ff6d00047c0 2024-01-16T15:09:49.962 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:49.962 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3894915042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3894915042},{"type":"v1","addr":"172.21.15.188:6835","nonce":3894915042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/3263318622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":3263318622},{"type":"v1","addr":"172.21.15.188:6837","nonce":3263318622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3550237844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3550237844},{"type":"v1","addr":"172.21.15.141:6833","nonce":3550237844}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3870765964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3870765964},{"type":"v1","addr":"172.21.15.141:6835","nonce":3870765964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-01-16T15:09:49.963 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 49 2024-01-16T15:09:49.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6c806bfe0 msgr2=0x7ff6c806e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6c806bfe0 0x7ff6c806e4a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800ba00 tx=0x7ff6d8032000 comp rx=0 tx=0).stop 2024-01-16T15:09:49.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 msgr2=0x7ff6f00ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6f00ade40 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000e7f0 tx=0x7ff6e000ecb0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 shutdown_connections 2024-01-16T15:09:49.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff6c806bfe0 0x7ff6c806e4a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6f01fab10 0x7ff6f01faf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff6f00ab4a0 0x7ff6f00a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 --2- 172.21.15.141:0/2821694189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6d00047c0 0x7ff6f00ade40 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 >> 172.21.15.141:0/2821694189 conn(0x7ff6f006a800 msgr2=0x7ff6f0149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:49.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 shutdown_connections 2024-01-16T15:09:49.966 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:49.987+0000 7ff6f8800640 1 -- 172.21.15.141:0/2821694189 wait complete. 2024-01-16T15:09:49.985 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:09:50.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.216+0000 7ff34a200640 1 Processor -- start 2024-01-16T15:09:50.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.216+0000 7ff34a200640 1 -- start start 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.217+0000 7ff34a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff344143ae0 0x7ff3440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.217+0000 7ff34a200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff3440cf7d0 0x7ff34415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.217+0000 7ff34a200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff34415e1e0 0x7ff34415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.217+0000 7ff34a200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3440cfc30 con 0x7ff34415e1e0 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.217+0000 7ff34a200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3440b1ee0 con 0x7ff3440cf7d0 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.217+0000 7ff34a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440b2780 con 0x7ff344143ae0 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff34415e1e0 0x7ff34415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff34415e1e0 0x7ff34415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39462/0 (socket says 172.21.15.141:39462) 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 -- 172.21.15.141:0/2363856178 learned_addr learned my addr 172.21.15.141:0/2363856178 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff33a800640 1 --2- 172.21.15.141:0/2363856178 >> v2:172.21.15.188:3300/0 conn(0x7ff3440cf7d0 0x7ff34415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343400640 1 --2- 172.21.15.141:0/2363856178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff344143ae0 0x7ff3440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 -- 172.21.15.141:0/2363856178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff344143ae0 msgr2=0x7ff3440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 --2- 172.21.15.141:0/2363856178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff344143ae0 0x7ff3440cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 -- 172.21.15.141:0/2363856178 >> v2:172.21.15.188:3300/0 conn(0x7ff3440cf7d0 msgr2=0x7ff34415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 --2- 172.21.15.141:0/2363856178 >> v2:172.21.15.188:3300/0 conn(0x7ff3440cf7d0 0x7ff34415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.218+0000 7ff343e00640 1 -- 172.21.15.141:0/2363856178 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34415e990 con 0x7ff34415e1e0 2024-01-16T15:09:50.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.219+0000 7ff343e00640 1 --2- 172.21.15.141:0/2363856178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff34415a670 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff34406e350 tx=0x7ff33c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444a83795ed2389f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff342a00640 1 -- 172.21.15.141:0/2363856178 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c02ca70 con 0x7ff34415e1e0 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff342a00640 1 -- 172.21.15.141:0/2363856178 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c02cbd0 con 0x7ff34415e1e0 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff342a00640 1 -- 172.21.15.141:0/2363856178 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0343f0 con 0x7ff34415e1e0 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff34a200640 1 -- 172.21.15.141:0/2363856178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 msgr2=0x7ff34415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff34a200640 1 --2- 172.21.15.141:0/2363856178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff34415a670 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff34406e350 tx=0x7ff33c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff34a200640 1 -- 172.21.15.141:0/2363856178 shutdown_connections 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff34a200640 1 --2- 172.21.15.141:0/2363856178 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff34415a670 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff34a200640 1 --2- 172.21.15.141:0/2363856178 >> v2:172.21.15.188:3300/0 conn(0x7ff3440cf7d0 0x7ff34415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff34a200640 1 --2- 172.21.15.141:0/2363856178 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff344143ae0 0x7ff3440cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.220+0000 7ff34a200640 1 -- 172.21.15.141:0/2363856178 >> 172.21.15.141:0/2363856178 conn(0x7ff34406a800 msgr2=0x7ff34406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.221+0000 7ff34a200640 1 -- 172.21.15.141:0/2363856178 shutdown_connections 2024-01-16T15:09:50.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.221+0000 7ff34a200640 1 -- 172.21.15.141:0/2363856178 wait complete. 2024-01-16T15:09:50.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.222+0000 7ff34a200640 1 Processor -- start 2024-01-16T15:09:50.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.223+0000 7ff34a200640 1 -- start start 2024-01-16T15:09:50.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.223+0000 7ff34a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3440cf7d0 0x7ff3440b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.223+0000 7ff34a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff344143ae0 0x7ff3440c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.223+0000 7ff34a200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff344159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.223+0000 7ff34a200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440c6210 con 0x7ff34415e1e0 2024-01-16T15:09:50.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.223+0000 7ff34a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440c5590 con 0x7ff344143ae0 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.223+0000 7ff34a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440c5870 con 0x7ff3440cf7d0 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff344159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff344159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39476/0 (socket says 172.21.15.141:39476) 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 -- 172.21.15.141:0/1291153898 learned_addr learned my addr 172.21.15.141:0/1291153898 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 -- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3440cf7d0 msgr2=0x7ff3440b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff33a800640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff344143ae0 0x7ff3440c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343400640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3440cf7d0 0x7ff3440b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3440cf7d0 0x7ff3440b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 -- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff344143ae0 msgr2=0x7ff3440c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff344143ae0 0x7ff3440c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff343e00640 1 -- 172.21.15.141:0/1291153898 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c009b30 con 0x7ff34415e1e0 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.224+0000 7ff33a800640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff344143ae0 0x7ff3440c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:50.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.225+0000 7ff343e00640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff344159c90 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff33c036040 tx=0x7ff33c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.225+0000 7ff33be00640 1 -- 172.21.15.141:0/1291153898 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c040670 con 0x7ff34415e1e0 2024-01-16T15:09:50.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.225+0000 7ff33be00640 1 -- 172.21.15.141:0/1291153898 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c0407f0 con 0x7ff34415e1e0 2024-01-16T15:09:50.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.225+0000 7ff33be00640 1 -- 172.21.15.141:0/1291153898 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c040ba0 con 0x7ff34415e1e0 2024-01-16T15:09:50.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.225+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3440c5b50 con 0x7ff34415e1e0 2024-01-16T15:09:50.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.225+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3440c6630 con 0x7ff34415e1e0 2024-01-16T15:09:50.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.226+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3440c6350 con 0x7ff34415e1e0 2024-01-16T15:09:50.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.227+0000 7ff33be00640 1 -- 172.21.15.141:0/1291153898 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ff33c040d00 con 0x7ff34415e1e0 2024-01-16T15:09:50.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.228+0000 7ff33be00640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff31c06bfb0 0x7ff31c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.228+0000 7ff33be00640 1 -- 172.21.15.141:0/1291153898 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff33c0ab8f0 con 0x7ff34415e1e0 2024-01-16T15:09:50.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.229+0000 7ff343400640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff31c06bfb0 0x7ff31c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.230+0000 7ff343400640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff31c06bfb0 0x7ff31c06e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff334006fd0 tx=0x7ff334008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.234+0000 7ff33be00640 1 -- 172.21.15.141:0/1291153898 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3440c6350 con 0x7ff34415e1e0 2024-01-16T15:09:50.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.425+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7ff3440c6350 con 0x7ff34415e1e0 2024-01-16T15:09:50.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.433+0000 7ff33be00640 1 -- 172.21.15.141:0/1291153898 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v28) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff3440c6350 con 0x7ff34415e1e0 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff31c06bfb0 msgr2=0x7ff31c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff31c06bfb0 0x7ff31c06e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff334006fd0 tx=0x7ff334008040 comp rx=0 tx=0).stop 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 msgr2=0x7ff344159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff344159c90 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff33c036040 tx=0x7ff33c004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 shutdown_connections 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff31c06bfb0 0x7ff31c06e470 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff34415e1e0 0x7ff344159c90 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.435+0000 7ff34a200640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff344143ae0 0x7ff3440c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.436+0000 7ff34a200640 1 --2- 172.21.15.141:0/1291153898 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3440cf7d0 0x7ff3440b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.436+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 >> 172.21.15.141:0/1291153898 conn(0x7ff34406a800 msgr2=0x7ff344142d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.436+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 shutdown_connections 2024-01-16T15:09:50.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.436+0000 7ff34a200640 1 -- 172.21.15.141:0/1291153898 wait complete. 2024-01-16T15:09:50.434 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:09:50.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.658+0000 7f4fdfe00640 1 Processor -- start 2024-01-16T15:09:50.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.658+0000 7f4fdfe00640 1 -- start start 2024-01-16T15:09:50.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.659+0000 7f4fdfe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4fe015d4b0 0x7f4fe00d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.659+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00b41e0 0x7f4fe00cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.659+0000 7f4fdfe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe00d0320 con 0x7f4fe00cf9a0 2024-01-16T15:09:50.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.659+0000 7f4fdfe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe00b2730 con 0x7f4fe015d4b0 2024-01-16T15:09:50.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.659+0000 7f4fdfe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00d1ec0 con 0x7f4fe00b41e0 2024-01-16T15:09:50.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.660+0000 7f4fdf400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00b41e0 0x7f4fe00cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.660+0000 7f4fdf400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00b41e0 0x7f4fe00cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:44182/0 (socket says 172.21.15.141:44182) 2024-01-16T15:09:50.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.660+0000 7f4fdf400640 1 -- 172.21.15.141:0/139999976 learned_addr learned my addr 172.21.15.141:0/139999976 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:50.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.660+0000 7f4fdea00640 1 --2- 172.21.15.141:0/139999976 >> v2:172.21.15.188:3300/0 conn(0x7f4fe015d4b0 0x7f4fe00d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.661+0000 7f4fdf400640 1 -- 172.21.15.141:0/139999976 >> v2:172.21.15.188:3300/0 conn(0x7f4fe015d4b0 msgr2=0x7f4fe00d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:50.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.661+0000 7f4fdf400640 1 --2- 172.21.15.141:0/139999976 >> v2:172.21.15.188:3300/0 conn(0x7f4fe015d4b0 0x7f4fe00d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.661+0000 7f4fdf400640 1 -- 172.21.15.141:0/139999976 >> v1:172.21.15.188:6789/0 conn(0x7f4fe00cf9a0 legacy=0x7f4fe00cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.661+0000 7f4fdf400640 1 -- 172.21.15.141:0/139999976 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe00b29d0 con 0x7f4fe00b41e0 2024-01-16T15:09:50.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.662+0000 7f4fdf400640 1 --2- 172.21.15.141:0/139999976 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00b41e0 0x7f4fe00cfde0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009870 tx=0x7f4fd002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5bb0f8281320c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.663+0000 7f4fdd600640 1 -- 172.21.15.141:0/139999976 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0030840 con 0x7f4fe00b41e0 2024-01-16T15:09:50.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.663+0000 7f4fdd600640 1 -- 172.21.15.141:0/139999976 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd00309a0 con 0x7f4fe00b41e0 2024-01-16T15:09:50.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.663+0000 7f4fdd600640 1 -- 172.21.15.141:0/139999976 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0030d30 con 0x7f4fe00b41e0 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.663+0000 7f4fdfe00640 1 -- 172.21.15.141:0/139999976 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00b41e0 msgr2=0x7f4fe00cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.663+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/139999976 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00b41e0 0x7f4fe00cfde0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009870 tx=0x7f4fd002b980 comp rx=0 tx=0).stop 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.664+0000 7f4fdfe00640 1 -- 172.21.15.141:0/139999976 shutdown_connections 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.664+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/139999976 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00b41e0 0x7f4fe00cfde0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.664+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/139999976 >> v2:172.21.15.188:3300/0 conn(0x7f4fe015d4b0 0x7f4fe00d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.664+0000 7f4fdfe00640 1 -- 172.21.15.141:0/139999976 >> 172.21.15.141:0/139999976 conn(0x7f4fe00d3000 msgr2=0x7f4fe00d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.664+0000 7f4fdfe00640 1 -- 172.21.15.141:0/139999976 shutdown_connections 2024-01-16T15:09:50.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.664+0000 7f4fdfe00640 1 -- 172.21.15.141:0/139999976 wait complete. 2024-01-16T15:09:50.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.665+0000 7f4fdfe00640 1 Processor -- start 2024-01-16T15:09:50.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.665+0000 7f4fdfe00640 1 -- start start 2024-01-16T15:09:50.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.666+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe00b41e0 0x7f4fe00a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.666+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00cf9a0 0x7f4fe00b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.666+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe015d4b0 0x7f4fe0154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.666+0000 7f4fdfe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b7010 con 0x7f4fe015d4b0 2024-01-16T15:09:50.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.666+0000 7f4fdfe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b6160 con 0x7f4fe00b41e0 2024-01-16T15:09:50.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.666+0000 7f4fdfe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b6390 con 0x7f4fe00cf9a0 2024-01-16T15:09:50.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe015d4b0 0x7f4fe0154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe015d4b0 0x7f4fe0154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39480/0 (socket says 172.21.15.141:39480) 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 -- 172.21.15.141:0/370073752 learned_addr learned my addr 172.21.15.141:0/370073752 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdea00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe00b41e0 0x7f4fe00a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fde000640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00cf9a0 0x7f4fe00b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 -- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00cf9a0 msgr2=0x7f4fe00b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00cf9a0 0x7f4fe00b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 -- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe00b41e0 msgr2=0x7f4fe00a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe00b41e0 0x7f4fe00a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fdf400640 1 -- 172.21.15.141:0/370073752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd00094d0 con 0x7f4fe015d4b0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.667+0000 7f4fde000640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00cf9a0 0x7f4fe00b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.668+0000 7f4fdf400640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe015d4b0 0x7f4fe0154440 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f4fd002b9c0 tx=0x7f4fd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.668+0000 7f4fd6a00640 1 -- 172.21.15.141:0/370073752 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0039c90 con 0x7f4fe015d4b0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.668+0000 7f4fd6a00640 1 -- 172.21.15.141:0/370073752 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd0039df0 con 0x7f4fe015d4b0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.668+0000 7f4fd6a00640 1 -- 172.21.15.141:0/370073752 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00446c0 con 0x7f4fe015d4b0 2024-01-16T15:09:50.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.668+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe015da10 con 0x7f4fe015d4b0 2024-01-16T15:09:50.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.668+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe00b6950 con 0x7f4fe015d4b0 2024-01-16T15:09:50.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.669+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe00b7150 con 0x7f4fe015d4b0 2024-01-16T15:09:50.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.670+0000 7f4fd6a00640 1 -- 172.21.15.141:0/370073752 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f4fd004d3b0 con 0x7f4fe015d4b0 2024-01-16T15:09:50.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.671+0000 7f4fd6a00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fb806bfb0 0x7f4fb806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:50.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.671+0000 7f4fd6a00640 1 -- 172.21.15.141:0/370073752 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4fd003e070 con 0x7f4fe015d4b0 2024-01-16T15:09:50.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.672+0000 7f4fdea00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fb806bfb0 0x7f4fb806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.673+0000 7f4fdea00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fb806bfb0 0x7f4fb806e470 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b8c20 tx=0x7f4fd800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.675+0000 7f4fd6a00640 1 -- 172.21.15.141:0/370073752 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fd00779a0 con 0x7f4fe015d4b0 2024-01-16T15:09:50.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.869+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4fe0067450 con 0x7f4fe015d4b0 2024-01-16T15:09:50.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.875+0000 7f4fd6a00640 1 -- 172.21.15.141:0/370073752 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4fd00490a0 con 0x7f4fe015d4b0 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.878+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fb806bfb0 msgr2=0x7f4fb806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.878+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fb806bfb0 0x7f4fb806e470 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b8c20 tx=0x7f4fd800b040 comp rx=0 tx=0).stop 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.878+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe015d4b0 msgr2=0x7f4fe0154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.878+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe015d4b0 0x7f4fe0154440 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f4fd002b9c0 tx=0x7f4fd0004730 comp rx=0 tx=0).stop 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 shutdown_connections 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4fb806bfb0 0x7f4fb806e470 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4fe015d4b0 0x7f4fe0154440 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4fe00cf9a0 0x7f4fe00b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 --2- 172.21.15.141:0/370073752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fe00b41e0 0x7f4fe00a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 >> 172.21.15.141:0/370073752 conn(0x7f4fe00d3000 msgr2=0x7f4fe00afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 shutdown_connections 2024-01-16T15:09:50.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:50.879+0000 7f4fdfe00640 1 -- 172.21.15.141:0/370073752 wait complete. 2024-01-16T15:09:50.876 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:09:51.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.101+0000 7ff4eac00640 1 Processor -- start 2024-01-16T15:09:51.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.101+0000 7ff4eac00640 1 -- start start 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.103+0000 7ff4eac00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff4e414b2c0 0x7ff4e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.103+0000 7ff4eac00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff4e4071070 0x7ff4e40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.103+0000 7ff4eac00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e40cfbf0 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.103+0000 7ff4eac00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e414b6a0 con 0x7ff4e414b2c0 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.103+0000 7ff4eac00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e4071690 con 0x7ff4e4071070 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.103+0000 7ff4e3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff4e414b2c0 0x7ff4e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.104+0000 7ff4e3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff4e414b2c0 0x7ff4e40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39486/0 (socket says 172.21.15.141:39486) 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.104+0000 7ff4e3e00640 1 -- 172.21.15.141:0/2550739100 learned_addr learned my addr 172.21.15.141:0/2550739100 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.104+0000 7ff4e8c00640 1 --2- 172.21.15.141:0/2550739100 >> v2:172.21.15.188:3300/0 conn(0x7ff4e4071070 0x7ff4e40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.104+0000 7ff4e3e00640 1 -- 172.21.15.141:0/2550739100 >> v2:172.21.15.188:3300/0 conn(0x7ff4e4071070 msgr2=0x7ff4e40cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.104+0000 7ff4e3e00640 1 --2- 172.21.15.141:0/2550739100 >> v2:172.21.15.188:3300/0 conn(0x7ff4e4071070 0x7ff4e40cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.104+0000 7ff4e3e00640 1 -- 172.21.15.141:0/2550739100 >> v1:172.21.15.188:6789/0 conn(0x7ff4e40cf7d0 legacy=0x7ff4e4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:51.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.104+0000 7ff4e3e00640 1 -- 172.21.15.141:0/2550739100 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4071970 con 0x7ff4e414b2c0 2024-01-16T15:09:51.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.106+0000 7ff4e3e00640 1 --2- 172.21.15.141:0/2550739100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e414b2c0 0x7ff4e40cf290 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4165330 tx=0x7ff4d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c41381f6a94b304 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:51.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.106+0000 7ff4e2a00640 1 -- 172.21.15.141:0/2550739100 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4031870 con 0x7ff4e414b2c0 2024-01-16T15:09:51.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.106+0000 7ff4e2a00640 1 -- 172.21.15.141:0/2550739100 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d40319d0 con 0x7ff4e414b2c0 2024-01-16T15:09:51.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.106+0000 7ff4e2a00640 1 -- 172.21.15.141:0/2550739100 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4031d60 con 0x7ff4e414b2c0 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 -- 172.21.15.141:0/2550739100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e414b2c0 msgr2=0x7ff4e40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2550739100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e414b2c0 0x7ff4e40cf290 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4165330 tx=0x7ff4d402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 -- 172.21.15.141:0/2550739100 shutdown_connections 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2550739100 >> v2:172.21.15.188:3300/0 conn(0x7ff4e4071070 0x7ff4e40cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2550739100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e414b2c0 0x7ff4e40cf290 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 -- 172.21.15.141:0/2550739100 >> 172.21.15.141:0/2550739100 conn(0x7ff4e406a800 msgr2=0x7ff4e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 -- 172.21.15.141:0/2550739100 shutdown_connections 2024-01-16T15:09:51.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.107+0000 7ff4eac00640 1 -- 172.21.15.141:0/2550739100 wait complete. 2024-01-16T15:09:51.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.109+0000 7ff4eac00640 1 Processor -- start 2024-01-16T15:09:51.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.109+0000 7ff4eac00640 1 -- start start 2024-01-16T15:09:51.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4eac00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4e4071070 0x7ff4e4141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4eac00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e40cf7d0 0x7ff4e4161e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4eac00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e414b2c0 0x7ff4e415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4eac00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4164dc0 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4eac00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4164ae0 con 0x7ff4e414b2c0 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4eac00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4159c10 con 0x7ff4e4071070 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4e3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e40cf7d0 0x7ff4e4161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4e3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e40cf7d0 0x7ff4e4161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39488/0 (socket says 172.21.15.141:39488) 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.110+0000 7ff4e3400640 1 -- 172.21.15.141:0/2628848801 learned_addr learned my addr 172.21.15.141:0/2628848801 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e8c00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e414b2c0 0x7ff4e415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e3e00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4e4071070 0x7ff4e4141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e3400640 1 -- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4e4071070 msgr2=0x7ff4e4141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e3400640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4e4071070 0x7ff4e4141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e3400640 1 -- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e414b2c0 msgr2=0x7ff4e415a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e3400640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e414b2c0 0x7ff4e415a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e3400640 1 -- 172.21.15.141:0/2628848801 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40094d0 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4e3400640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e40cf7d0 0x7ff4e4161e70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00c430 tx=0x7ff4cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4dbe00640 1 -- 172.21.15.141:0/2628848801 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc011790 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4dbe00640 1 -- 172.21.15.141:0/2628848801 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc0118f0 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4dbe00640 1 -- 172.21.15.141:0/2628848801 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc011ce0 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e4159ef0 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.111+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e4142500 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.113+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e415a410 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.114+0000 7ff4dbe00640 1 -- 172.21.15.141:0/2628848801 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ff4cc01c460 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.114+0000 7ff4dbe00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4bc06bfb0 0x7ff4bc06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.115+0000 7ff4dbe00640 1 -- 172.21.15.141:0/2628848801 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff4cc0885c0 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.115+0000 7ff4e3e00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4bc06bfb0 0x7ff4bc06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.116+0000 7ff4e3e00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4bc06bfb0 0x7ff4bc06e470 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4036000 tx=0x7ff4d403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:51.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.119+0000 7ff4dbe00640 1 -- 172.21.15.141:0/2628848801 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4e415a410 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.379+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7ff4e415a410 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.808+0000 7ff4dbe00640 1 -- 172.21.15.141:0/2628848801 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v576) v1 ==== 166+0+0 (secure 0 0 0) 0x7ff4e415a410 con 0x7ff4e40cf7d0 2024-01-16T15:09:51.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4bc06bfb0 msgr2=0x7ff4bc06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4bc06bfb0 0x7ff4bc06e470 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4036000 tx=0x7ff4d403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e40cf7d0 msgr2=0x7ff4e4161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e40cf7d0 0x7ff4e4161e70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00c430 tx=0x7ff4cc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 shutdown_connections 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff4bc06bfb0 0x7ff4bc06e470 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e414b2c0 0x7ff4e415a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e40cf7d0 0x7ff4e4161e70 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 --2- 172.21.15.141:0/2628848801 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff4e4071070 0x7ff4e4141fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.811+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 >> 172.21.15.141:0/2628848801 conn(0x7ff4e406a800 msgr2=0x7ff4e406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.812+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 shutdown_connections 2024-01-16T15:09:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:51.812+0000 7ff4eac00640 1 -- 172.21.15.141:0/2628848801 wait complete. 2024-01-16T15:09:51.811 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:09:51.811 INFO:tasks.cephfs_test_runner:Starting test: test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:09:51.812 INFO:tasks.cephfs_test_runner:test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:09:51.812 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:52.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.039+0000 7f1766400640 1 Processor -- start 2024-01-16T15:09:52.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.039+0000 7f1766400640 1 -- start start 2024-01-16T15:09:52.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f1766400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1760143ae0 0x7f17600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f1766400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f17600cf7d0 0x7f176015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f1766400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f176015e1e0 0x7f176015a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f1766400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17600cfc30 con 0x7f17600cf7d0 2024-01-16T15:09:52.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f1766400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17600b1ee0 con 0x7f1760143ae0 2024-01-16T15:09:52.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f1766400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17600b2780 con 0x7f176015e1e0 2024-01-16T15:09:52.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f17600cf7d0 0x7f176015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f17600cf7d0 0x7f176015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39492/0 (socket says 172.21.15.141:39492) 2024-01-16T15:09:52.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175ea00640 1 -- 172.21.15.141:0/3935237188 learned_addr learned my addr 172.21.15.141:0/3935237188 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:52.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175fe00640 1 --2- 172.21.15.141:0/3935237188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f176015e1e0 0x7f176015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175ea00640 1 -- 172.21.15.141:0/3935237188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f176015e1e0 msgr2=0x7f176015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175ea00640 1 --2- 172.21.15.141:0/3935237188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f176015e1e0 0x7f176015a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175f400640 1 --2- 172.21.15.141:0/3935237188 >> v2:172.21.15.188:3300/0 conn(0x7f1760143ae0 0x7f17600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.041+0000 7f175ea00640 1 -- 172.21.15.141:0/3935237188 >> v2:172.21.15.188:3300/0 conn(0x7f1760143ae0 msgr2=0x7f17600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.042+0000 7f175ea00640 1 --2- 172.21.15.141:0/3935237188 >> v2:172.21.15.188:3300/0 conn(0x7f1760143ae0 0x7f17600cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.042+0000 7f175ea00640 1 -- 172.21.15.141:0/3935237188 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176015e990 con 0x7f17600cf7d0 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.042+0000 7f175fe00640 1 --2- 172.21.15.141:0/3935237188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f176015e1e0 0x7f176015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:52.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.043+0000 7f175ea00640 1 --2- 172.21.15.141:0/3935237188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17600cf7d0 0x7f176015dca0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f17600cfea0 tx=0x7f174802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668d3379e62a00b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.043+0000 7f175e000640 1 -- 172.21.15.141:0/3935237188 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748031860 con 0x7f17600cf7d0 2024-01-16T15:09:52.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.043+0000 7f175e000640 1 -- 172.21.15.141:0/3935237188 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17480319c0 con 0x7f17600cf7d0 2024-01-16T15:09:52.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.043+0000 7f175e000640 1 -- 172.21.15.141:0/3935237188 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748031d50 con 0x7f17600cf7d0 2024-01-16T15:09:52.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 -- 172.21.15.141:0/3935237188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17600cf7d0 msgr2=0x7f176015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 --2- 172.21.15.141:0/3935237188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17600cf7d0 0x7f176015dca0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f17600cfea0 tx=0x7f174802b900 comp rx=0 tx=0).stop 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 -- 172.21.15.141:0/3935237188 shutdown_connections 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 --2- 172.21.15.141:0/3935237188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f176015e1e0 0x7f176015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 --2- 172.21.15.141:0/3935237188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17600cf7d0 0x7f176015dca0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 --2- 172.21.15.141:0/3935237188 >> v2:172.21.15.188:3300/0 conn(0x7f1760143ae0 0x7f17600cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 -- 172.21.15.141:0/3935237188 >> 172.21.15.141:0/3935237188 conn(0x7f176006a800 msgr2=0x7f176006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 -- 172.21.15.141:0/3935237188 shutdown_connections 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.044+0000 7f1766400640 1 -- 172.21.15.141:0/3935237188 wait complete. 2024-01-16T15:09:52.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.046+0000 7f1766400640 1 Processor -- start 2024-01-16T15:09:52.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.046+0000 7f1766400640 1 -- start start 2024-01-16T15:09:52.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f1766400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17600cf7d0 0x7f1760138430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f1766400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1760143ae0 0x7f17600b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f1766400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176015e1e0 0x7f17600b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f1766400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1760138970 con 0x7f1760143ae0 2024-01-16T15:09:52.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f1766400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17601f73f0 con 0x7f176015e1e0 2024-01-16T15:09:52.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f1766400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17601f7920 con 0x7f17600cf7d0 2024-01-16T15:09:52.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f175ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1760143ae0 0x7f17600b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f175ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1760143ae0 0x7f17600b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39498/0 (socket says 172.21.15.141:39498) 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f175ea00640 1 -- 172.21.15.141:0/184255347 learned_addr learned my addr 172.21.15.141:0/184255347 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f175f400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17600cf7d0 0x7f1760138430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f175fe00640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176015e1e0 0x7f17600b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f175ea00640 1 -- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17600cf7d0 msgr2=0x7f1760138430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.047+0000 7f175ea00640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17600cf7d0 0x7f1760138430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f175ea00640 1 -- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176015e1e0 msgr2=0x7f17600b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f175ea00640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176015e1e0 0x7f17600b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f175ea00640 1 -- 172.21.15.141:0/184255347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17480094d0 con 0x7f1760143ae0 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f175f400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17600cf7d0 0x7f1760138430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f175ea00640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1760143ae0 0x7f17600b0bc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1748000c00 tx=0x7f17480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f1757400640 1 -- 172.21.15.141:0/184255347 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748031810 con 0x7f1760143ae0 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f1757400640 1 -- 172.21.15.141:0/184255347 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1748044400 con 0x7f1760143ae0 2024-01-16T15:09:52.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f1757400640 1 -- 172.21.15.141:0/184255347 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17480447b0 con 0x7f1760143ae0 2024-01-16T15:09:52.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17600710b0 con 0x7f1760143ae0 2024-01-16T15:09:52.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.048+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17601f7640 con 0x7f1760143ae0 2024-01-16T15:09:52.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.050+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17600675d0 con 0x7f1760143ae0 2024-01-16T15:09:52.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.050+0000 7f1757400640 1 -- 172.21.15.141:0/184255347 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f174804d440 con 0x7f1760143ae0 2024-01-16T15:09:52.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.051+0000 7f1757400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f173806c000 0x7f173806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.051+0000 7f1757400640 1 -- 172.21.15.141:0/184255347 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f17480aade0 con 0x7f1760143ae0 2024-01-16T15:09:52.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.052+0000 7f175f400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f173806c000 0x7f173806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.053+0000 7f175f400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f173806c000 0x7f173806e4c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1750006fd0 tx=0x7f1750008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.056+0000 7f1757400640 1 -- 172.21.15.141:0/184255347 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17480776e0 con 0x7f1760143ae0 2024-01-16T15:09:52.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.249+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f176015c120 con 0x7f1760143ae0 2024-01-16T15:09:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.250+0000 7f1757400640 1 -- 172.21.15.141:0/184255347 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+13037 (secure 0 0 0) 0x7f174807ad00 con 0x7f1760143ae0 2024-01-16T15:09:52.229 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:52.229 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":80,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:09:45.570741+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6835/3457901158":"2024-01-17T15:09:42.558294+0000","172.21.15.141:6834/3457901158":"2024-01-17T15:09:42.558294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:52.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f173806c000 msgr2=0x7f173806e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f173806c000 0x7f173806e4c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1750006fd0 tx=0x7f1750008040 comp rx=0 tx=0).stop 2024-01-16T15:09:52.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1760143ae0 msgr2=0x7f17600b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1760143ae0 0x7f17600b0bc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1748000c00 tx=0x7f17480046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 shutdown_connections 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f173806c000 0x7f173806e4c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f176015e1e0 0x7f17600b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1760143ae0 0x7f17600b0bc0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 --2- 172.21.15.141:0/184255347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f17600cf7d0 0x7f1760138430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 >> 172.21.15.141:0/184255347 conn(0x7f176006a800 msgr2=0x7f176006ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 shutdown_connections 2024-01-16T15:09:52.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.253+0000 7f1766400640 1 -- 172.21.15.141:0/184255347 wait complete. 2024-01-16T15:09:52.250 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:52.250 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:52.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.481+0000 7fc19ca00640 1 Processor -- start 2024-01-16T15:09:52.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.481+0000 7fc19ca00640 1 -- start start 2024-01-16T15:09:52.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.482+0000 7fc19ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19413df80 0x7fc19415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.482+0000 7fc19ca00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc19413e360 0x7fc1940cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc19ca00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc19415f9c0 0x7fc1940cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc19ca00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19406c470 con 0x7fc19413e360 2024-01-16T15:09:52.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc19ca00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19406c750 con 0x7fc19415f9c0 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc19ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19406ca30 con 0x7fc19413df80 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc19413e360 0x7fc1940cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc19413e360 0x7fc1940cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39500/0 (socket says 172.21.15.141:39500) 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 -- 172.21.15.141:0/1138244139 learned_addr learned my addr 172.21.15.141:0/1138244139 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc19a000640 1 --2- 172.21.15.141:0/1138244139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19413df80 0x7fc19415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 -- 172.21.15.141:0/1138244139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19413df80 msgr2=0x7fc19415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 --2- 172.21.15.141:0/1138244139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19413df80 0x7fc19415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 -- 172.21.15.141:0/1138244139 >> v2:172.21.15.188:3300/0 conn(0x7fc19415f9c0 msgr2=0x7fc1940cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 --2- 172.21.15.141:0/1138244139 >> v2:172.21.15.188:3300/0 conn(0x7fc19415f9c0 0x7fc1940cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.483+0000 7fc199600640 1 -- 172.21.15.141:0/1138244139 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19406cd10 con 0x7fc19413e360 2024-01-16T15:09:52.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc199600640 1 --2- 172.21.15.141:0/1138244139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc1940cf290 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc19406e6d0 tx=0x7fc17c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c53e39a1dc6953 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc198c00640 1 -- 172.21.15.141:0/1138244139 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c031860 con 0x7fc19413e360 2024-01-16T15:09:52.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc198c00640 1 -- 172.21.15.141:0/1138244139 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c0319c0 con 0x7fc19413e360 2024-01-16T15:09:52.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc198c00640 1 -- 172.21.15.141:0/1138244139 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c031d50 con 0x7fc19413e360 2024-01-16T15:09:52.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc19ca00640 1 -- 172.21.15.141:0/1138244139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 msgr2=0x7fc1940cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc19ca00640 1 --2- 172.21.15.141:0/1138244139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc1940cf290 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc19406e6d0 tx=0x7fc17c02b900 comp rx=0 tx=0).stop 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc19ca00640 1 -- 172.21.15.141:0/1138244139 shutdown_connections 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc19ca00640 1 --2- 172.21.15.141:0/1138244139 >> v2:172.21.15.188:3300/0 conn(0x7fc19415f9c0 0x7fc1940cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc19ca00640 1 --2- 172.21.15.141:0/1138244139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc1940cf290 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc19ca00640 1 --2- 172.21.15.141:0/1138244139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19413df80 0x7fc19415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.485+0000 7fc19ca00640 1 -- 172.21.15.141:0/1138244139 >> 172.21.15.141:0/1138244139 conn(0x7fc19406a800 msgr2=0x7fc19406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.486+0000 7fc19ca00640 1 -- 172.21.15.141:0/1138244139 shutdown_connections 2024-01-16T15:09:52.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.486+0000 7fc19ca00640 1 -- 172.21.15.141:0/1138244139 wait complete. 2024-01-16T15:09:52.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.487+0000 7fc19ca00640 1 Processor -- start 2024-01-16T15:09:52.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.487+0000 7fc19ca00640 1 -- start start 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.488+0000 7fc19ca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19413df80 0x7fc194154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.488+0000 7fc19ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc19415a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.488+0000 7fc19ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19415f9c0 0x7fc194139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.488+0000 7fc19ca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19415aa10 con 0x7fc19413e360 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.488+0000 7fc19ca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19415acf0 con 0x7fc19413df80 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.488+0000 7fc19ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19415afd0 con 0x7fc19415f9c0 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc19415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc19415a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39508/0 (socket says 172.21.15.141:39508) 2024-01-16T15:09:52.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 -- 172.21.15.141:0/3977292735 learned_addr learned my addr 172.21.15.141:0/3977292735 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc19a000640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19413df80 0x7fc194154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc19aa00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19415f9c0 0x7fc194139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 -- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19415f9c0 msgr2=0x7fc194139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19415f9c0 0x7fc194139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 -- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19413df80 msgr2=0x7fc194154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19413df80 0x7fc194154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 -- 172.21.15.141:0/3977292735 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c0094d0 con 0x7fc19413e360 2024-01-16T15:09:52.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc19aa00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19415f9c0 0x7fc194139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:52.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.489+0000 7fc199600640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc19415a410 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fc17c000c00 tx=0x7fc17c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.490+0000 7fc192000640 1 -- 172.21.15.141:0/3977292735 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c031830 con 0x7fc19413e360 2024-01-16T15:09:52.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.490+0000 7fc192000640 1 -- 172.21.15.141:0/3977292735 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c0443f0 con 0x7fc19413e360 2024-01-16T15:09:52.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.490+0000 7fc192000640 1 -- 172.21.15.141:0/3977292735 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c0447a0 con 0x7fc19413e360 2024-01-16T15:09:52.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.490+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc194155e80 con 0x7fc19413e360 2024-01-16T15:09:52.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.490+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19413a480 con 0x7fc19413e360 2024-01-16T15:09:52.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.491+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1941563a0 con 0x7fc19413e360 2024-01-16T15:09:52.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.492+0000 7fc192000640 1 -- 172.21.15.141:0/3977292735 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc17c04d450 con 0x7fc19413e360 2024-01-16T15:09:52.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.493+0000 7fc192000640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc16c06bfb0 0x7fc16c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.493+0000 7fc192000640 1 -- 172.21.15.141:0/3977292735 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc17c0aafd0 con 0x7fc19413e360 2024-01-16T15:09:52.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.494+0000 7fc19a000640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc16c06bfb0 0x7fc16c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.495+0000 7fc19a000640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc16c06bfb0 0x7fc16c06e470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc1840070b0 tx=0x7fc184007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.497+0000 7fc192000640 1 -- 172.21.15.141:0/3977292735 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1941563a0 con 0x7fc19413e360 2024-01-16T15:09:52.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.687+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1941563a0 con 0x7fc19413e360 2024-01-16T15:09:52.665 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.688+0000 7fc192000640 1 -- 172.21.15.141:0/3977292735 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1941563a0 con 0x7fc19413e360 2024-01-16T15:09:52.666 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc16c06bfb0 msgr2=0x7fc16c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc16c06bfb0 0x7fc16c06e470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc1840070b0 tx=0x7fc184007040 comp rx=0 tx=0).stop 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 msgr2=0x7fc19415a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc19415a410 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fc17c000c00 tx=0x7fc17c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 shutdown_connections 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc16c06bfb0 0x7fc16c06e470 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc19415f9c0 0x7fc194139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc19413e360 0x7fc19415a410 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.690+0000 7fc19ca00640 1 --2- 172.21.15.141:0/3977292735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc19413df80 0x7fc194154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.691+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 >> 172.21.15.141:0/3977292735 conn(0x7fc19406a800 msgr2=0x7fc1940cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.691+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 shutdown_connections 2024-01-16T15:09:52.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.691+0000 7fc19ca00640 1 -- 172.21.15.141:0/3977292735 wait complete. 2024-01-16T15:09:52.687 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:09:52.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.917+0000 7f5657e00640 1 Processor -- start 2024-01-16T15:09:52.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.918+0000 7f5657e00640 1 -- start start 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.919+0000 7f5657e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f56580cc990 0x7f565806d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.919+0000 7f5657e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5658070e50 con 0x7f5658143c40 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.919+0000 7f5657e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5658144020 con 0x7f56580cc500 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.919+0000 7f5657e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56580723c0 con 0x7f56580cc990 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.919+0000 7f5656a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5658143c40 0x7f5658071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33528/0 (socket says 172.21.15.141:33528) 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.920+0000 7f5656a00640 1 -- 172.21.15.141:0/1525355448 learned_addr learned my addr 172.21.15.141:0/1525355448 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.920+0000 7f5657400640 1 --2- 172.21.15.141:0/1525355448 >> v2:172.21.15.141:3300/0 conn(0x7f56580cc990 0x7f565806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.920+0000 7f5656000640 1 -- 172.21.15.141:0/1525355448 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587728243 0 0) 0x7f5658070e50 con 0x7f5658143c40 2024-01-16T15:09:52.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.920+0000 7f5656000640 1 -- 172.21.15.141:0/1525355448 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5638003520 con 0x7f5658143c40 2024-01-16T15:09:52.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.920+0000 7f5657400640 1 -- 172.21.15.141:0/1525355448 >> v1:172.21.15.188:6789/0 conn(0x7f56580cc500 legacy=0x7f565806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:52.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.920+0000 7f5657400640 1 -- 172.21.15.141:0/1525355448 >> v1:172.21.15.141:6789/0 conn(0x7f5658143c40 legacy=0x7f5658071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.920+0000 7f5657400640 1 -- 172.21.15.141:0/1525355448 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5658071e00 con 0x7f56580cc990 2024-01-16T15:09:52.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.921+0000 7f5657400640 1 --2- 172.21.15.141:0/1525355448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56580cc990 0x7f565806d870 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f5650009d90 tx=0x7f565002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e8f4ecf4afd280 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.921+0000 7f5656000640 1 -- 172.21.15.141:0/1525355448 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565002cbd0 con 0x7f56580cc990 2024-01-16T15:09:52.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5656000640 1 -- 172.21.15.141:0/1525355448 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565002cd30 con 0x7f56580cc990 2024-01-16T15:09:52.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5656000640 1 -- 172.21.15.141:0/1525355448 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565002cea0 con 0x7f56580cc990 2024-01-16T15:09:52.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5657e00640 1 -- 172.21.15.141:0/1525355448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56580cc990 msgr2=0x7f565806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5657e00640 1 --2- 172.21.15.141:0/1525355448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56580cc990 0x7f565806d870 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f5650009d90 tx=0x7f565002be20 comp rx=0 tx=0).stop 2024-01-16T15:09:52.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5657e00640 1 -- 172.21.15.141:0/1525355448 shutdown_connections 2024-01-16T15:09:52.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5657e00640 1 --2- 172.21.15.141:0/1525355448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56580cc990 0x7f565806d870 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5657e00640 1 -- 172.21.15.141:0/1525355448 >> 172.21.15.141:0/1525355448 conn(0x7f565806a800 msgr2=0x7f565806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:52.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5657e00640 1 -- 172.21.15.141:0/1525355448 shutdown_connections 2024-01-16T15:09:52.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.922+0000 7f5657e00640 1 -- 172.21.15.141:0/1525355448 wait complete. 2024-01-16T15:09:52.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.924+0000 7f5657e00640 1 Processor -- start 2024-01-16T15:09:52.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.924+0000 7f5657e00640 1 -- start start 2024-01-16T15:09:52.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56580cc500 0x7f56581e8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56580cc990 0x7f565806d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5658143c40 0x7f565806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56581e27c0 con 0x7f5658143c40 2024-01-16T15:09:52.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565813d820 con 0x7f56580cc500 2024-01-16T15:09:52.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565813db00 con 0x7f56580cc990 2024-01-16T15:09:52.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5658143c40 0x7f565806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5658143c40 0x7f565806d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39530/0 (socket says 172.21.15.141:39530) 2024-01-16T15:09:52.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.925+0000 7f5657400640 1 -- 172.21.15.141:0/1945530683 learned_addr learned my addr 172.21.15.141:0/1945530683 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f5657400640 1 -- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56580cc990 msgr2=0x7f565806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f5657400640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56580cc990 0x7f565806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f5657400640 1 -- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56580cc500 msgr2=0x7f56581e8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f5657400640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56580cc500 0x7f56581e8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f5657400640 1 -- 172.21.15.141:0/1945530683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650009a70 con 0x7f5658143c40 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f5657400640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5658143c40 0x7f565806d6a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5650009d90 tx=0x7f5650004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f564f400640 1 -- 172.21.15.141:0/1945530683 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565002cea0 con 0x7f5658143c40 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f564f400640 1 -- 172.21.15.141:0/1945530683 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650040650 con 0x7f5658143c40 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565813dde0 con 0x7f5658143c40 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.926+0000 7f564f400640 1 -- 172.21.15.141:0/1945530683 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650040a00 con 0x7f5658143c40 2024-01-16T15:09:52.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.927+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56581e8e90 con 0x7f5658143c40 2024-01-16T15:09:52.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.927+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56581e2930 con 0x7f5658143c40 2024-01-16T15:09:52.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.928+0000 7f564f400640 1 -- 172.21.15.141:0/1945530683 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f5650040b60 con 0x7f5658143c40 2024-01-16T15:09:52.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.929+0000 7f564f400640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f563006bfb0 0x7f563006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:52.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.929+0000 7f564f400640 1 -- 172.21.15.141:0/1945530683 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f56500ab530 con 0x7f5658143c40 2024-01-16T15:09:52.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.930+0000 7f5656a00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f563006bfb0 0x7f563006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:52.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.931+0000 7f5656a00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f563006bfb0 0x7f563006e470 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f56581f6470 tx=0x7f5648007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:52.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:52.934+0000 7f564f400640 1 -- 172.21.15.141:0/1945530683 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5650077c20 con 0x7f5658143c40 2024-01-16T15:09:53.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.194+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f5658067450 con 0x7f5658143c40 2024-01-16T15:09:53.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.822+0000 7f564f400640 1 -- 172.21.15.141:0/1945530683 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v578) v1 ==== 156+0+0 (secure 0 0 0) 0x7f5650009450 con 0x7f5658143c40 2024-01-16T15:09:53.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f563006bfb0 msgr2=0x7f563006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f563006bfb0 0x7f563006e470 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f56581f6470 tx=0x7f5648007930 comp rx=0 tx=0).stop 2024-01-16T15:09:53.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5658143c40 msgr2=0x7f565806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5658143c40 0x7f565806d6a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5650009d90 tx=0x7f5650004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:53.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 shutdown_connections 2024-01-16T15:09:53.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f563006bfb0 0x7f563006e470 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5658143c40 0x7f565806d6a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56580cc990 0x7f565806d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 --2- 172.21.15.141:0/1945530683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56580cc500 0x7f56581e8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 >> 172.21.15.141:0/1945530683 conn(0x7f565806a800 msgr2=0x7f565806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:53.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 shutdown_connections 2024-01-16T15:09:53.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:53.826+0000 7f5657e00640 1 -- 172.21.15.141:0/1945530683 wait complete. 2024-01-16T15:09:53.825 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:09:54.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.057+0000 7fd4f6000640 1 Processor -- start 2024-01-16T15:09:54.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.057+0000 7fd4f6000640 1 -- start start 2024-01-16T15:09:54.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.059+0000 7fd4f6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f013cda0 0x7fd4f015da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.059+0000 7fd4f6000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd4f015dfa0 0x7fd4f00cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.059+0000 7fd4f6000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f0071b50 con 0x7fd4f00cdef0 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.059+0000 7fd4f6000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f00700f0 con 0x7fd4f015dfa0 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.059+0000 7fd4f6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f015e3e0 con 0x7fd4f013cda0 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4efe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd4f015dfa0 0x7fd4f00cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4efe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd4f015dfa0 0x7fd4f00cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:34114/0 (socket says 172.21.15.141:34114) 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4efe00640 1 -- 172.21.15.141:0/2691972908 learned_addr learned my addr 172.21.15.141:0/2691972908 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:54.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4f4c00640 1 --2- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f013cda0 0x7fd4f015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4efe00640 1 -- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f013cda0 msgr2=0x7fd4f015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4efe00640 1 --2- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f013cda0 0x7fd4f015da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4efe00640 1 -- 172.21.15.141:0/2691972908 >> v1:172.21.15.188:6789/0 conn(0x7fd4f00cdef0 legacy=0x7fd4f015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.060+0000 7fd4efe00640 1 -- 172.21.15.141:0/2691972908 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f00ce720 con 0x7fd4f015dfa0 2024-01-16T15:09:54.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.062+0000 7fd4efe00640 1 --2- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f015dfa0 0x7fd4f00cd9b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800bb10 tx=0x7fd4d802e0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d160d17c1132691a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.062+0000 7fd4ef400640 1 -- 172.21.15.141:0/2691972908 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d802ee70 con 0x7fd4f015dfa0 2024-01-16T15:09:54.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.062+0000 7fd4ef400640 1 -- 172.21.15.141:0/2691972908 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8036440 con 0x7fd4f015dfa0 2024-01-16T15:09:54.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.062+0000 7fd4ef400640 1 -- 172.21.15.141:0/2691972908 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80367d0 con 0x7fd4f015dfa0 2024-01-16T15:09:54.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 -- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f015dfa0 msgr2=0x7fd4f00cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 --2- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f015dfa0 0x7fd4f00cd9b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800bb10 tx=0x7fd4d802e0c0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 -- 172.21.15.141:0/2691972908 shutdown_connections 2024-01-16T15:09:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 --2- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f015dfa0 0x7fd4f00cd9b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 --2- 172.21.15.141:0/2691972908 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f013cda0 0x7fd4f015da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 -- 172.21.15.141:0/2691972908 >> 172.21.15.141:0/2691972908 conn(0x7fd4f006a3e0 msgr2=0x7fd4f006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 -- 172.21.15.141:0/2691972908 shutdown_connections 2024-01-16T15:09:54.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.063+0000 7fd4f6000640 1 -- 172.21.15.141:0/2691972908 wait complete. 2024-01-16T15:09:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.066+0000 7fd4f6000640 1 Processor -- start 2024-01-16T15:09:54.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.066+0000 7fd4f6000640 1 -- start start 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4f00cdef0 0x7fd4f0138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f013cda0 0x7fd4f0153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f015dfa0 0x7fd4f00b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f6000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0151ad0 con 0x7fd4f00cdef0 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f6000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0151db0 con 0x7fd4f013cda0 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0152090 con 0x7fd4f015dfa0 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4f00cdef0 0x7fd4f0138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4f00cdef0 0x7fd4f0138960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51046/0 (socket says 172.21.15.141:51046) 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.067+0000 7fd4f4c00640 1 -- 172.21.15.141:0/1003345711 learned_addr learned my addr 172.21.15.141:0/1003345711 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4efe00640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f013cda0 0x7fd4f0153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f5600640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f015dfa0 0x7fd4f00b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f4c00640 1 -- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f015dfa0 msgr2=0x7fd4f00b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f4c00640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f015dfa0 0x7fd4f00b2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f4c00640 1 -- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f013cda0 msgr2=0x7fd4f0153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f4c00640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f013cda0 0x7fd4f0153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f4c00640 1 -- 172.21.15.141:0/1003345711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d800b770 con 0x7fd4f00cdef0 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f5600640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f015dfa0 0x7fd4f00b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4f4c00640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4f00cdef0 0x7fd4f0138960 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000e9b0 tx=0x7fd4e000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.068+0000 7fd4ecc00640 1 -- 172.21.15.141:0/1003345711 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0009670 con 0x7fd4f00cdef0 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.069+0000 7fd4ecc00640 1 -- 172.21.15.141:0/1003345711 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e000c9a0 con 0x7fd4f00cdef0 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.069+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f0152370 con 0x7fd4f00cdef0 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.069+0000 7fd4ecc00640 1 -- 172.21.15.141:0/1003345711 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e000cd50 con 0x7fd4f00cdef0 2024-01-16T15:09:54.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.069+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f0138ea0 con 0x7fd4f00cdef0 2024-01-16T15:09:54.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.070+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f0067450 con 0x7fd4f00cdef0 2024-01-16T15:09:54.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.071+0000 7fd4ecc00640 1 -- 172.21.15.141:0/1003345711 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fd4e0013470 con 0x7fd4f00cdef0 2024-01-16T15:09:54.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.072+0000 7fd4ecc00640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4c806bfb0 0x7fd4c806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.072+0000 7fd4ecc00640 1 -- 172.21.15.141:0/1003345711 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd4e0088900 con 0x7fd4f00cdef0 2024-01-16T15:09:54.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.073+0000 7fd4efe00640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4c806bfb0 0x7fd4c806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.074+0000 7fd4efe00640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4c806bfb0 0x7fd4c806e470 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8038000 tx=0x7fd4d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.077+0000 7fd4ecc00640 1 -- 172.21.15.141:0/1003345711 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4e0055120 con 0x7fd4f00cdef0 2024-01-16T15:09:54.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.268+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd4f0072060 con 0x7fd4f00cdef0 2024-01-16T15:09:54.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.276+0000 7fd4ecc00640 1 -- 172.21.15.141:0/1003345711 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd4e0058740 con 0x7fd4f00cdef0 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4c806bfb0 msgr2=0x7fd4c806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4c806bfb0 0x7fd4c806e470 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8038000 tx=0x7fd4d8008040 comp rx=0 tx=0).stop 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4f00cdef0 msgr2=0x7fd4f0138960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4f00cdef0 0x7fd4f0138960 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000e9b0 tx=0x7fd4e000ee70 comp rx=0 tx=0).stop 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 shutdown_connections 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4c806bfb0 0x7fd4c806e470 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd4f015dfa0 0x7fd4f00b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f013cda0 0x7fd4f0153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 --2- 172.21.15.141:0/1003345711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4f00cdef0 0x7fd4f0138960 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 >> 172.21.15.141:0/1003345711 conn(0x7fd4f006a3e0 msgr2=0x7fd4f0071200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:54.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 shutdown_connections 2024-01-16T15:09:54.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.279+0000 7fd4f6000640 1 -- 172.21.15.141:0/1003345711 wait complete. 2024-01-16T15:09:54.275 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:09:54.275 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:54.382 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:09:54.503 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:09:54.503 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:09:54.503 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:54.563 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:09:54.563 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:09:54.563 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:54.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:09:54.619 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:09:54.619 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:54.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:09:54.678 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:09:54.678 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:09:54.735 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:09:54.735 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:09:54.735 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:09:54.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.980+0000 7f3316e00640 1 Processor -- start 2024-01-16T15:09:54.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.980+0000 7f3316e00640 1 -- start start 2024-01-16T15:09:54.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.981+0000 7f3316e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 0x7f3310166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.981+0000 7f3316e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331015c5f0 con 0x7f331013e360 2024-01-16T15:09:54.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.981+0000 7f3316e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331006c470 con 0x7f331013df80 2024-01-16T15:09:54.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.981+0000 7f3316e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331006c750 con 0x7f33100cf9a0 2024-01-16T15:09:54.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.982+0000 7f330f400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f331013e360 0x7f33100cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56138/0 (socket says 172.21.15.141:56138) 2024-01-16T15:09:54.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.982+0000 7f330f400640 1 -- 172.21.15.141:0/574330692 learned_addr learned my addr 172.21.15.141:0/574330692 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:54.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.982+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269150648 0 0) 0x7f331015c5f0 con 0x7f331013e360 2024-01-16T15:09:54.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.982+0000 7f3314e00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 0x7f3310166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.982+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f0003520 con 0x7f331013e360 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248478570 0 0) 0x7f32f0003520 con 0x7f331013e360 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 msgr2=0x7f3310166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 0x7f3310166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 >> v1:172.21.15.188:6789/0 conn(0x7f331013df80 legacy=0x7f331015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331006ca30 con 0x7f331013e360 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f32f8001670 con 0x7f331013e360 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 >> v1:172.21.15.141:6789/0 conn(0x7f331013e360 legacy=0x7f33100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 0x7f32f00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32f0004020 0x7f32f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32f0004d00 0x7f32f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331006c750 con 0x7f32f0004020 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f000bbc0 con 0x7f32f0004d00 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f000bd80 con 0x7f33100cf9a0 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330f400640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32f0004020 0x7f32f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330fe00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 0x7f32f00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f330f400640 1 -- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 msgr2=0x7f32f00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.983+0000 7f3314e00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32f0004d00 0x7f32f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.984+0000 7f330f400640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f33100cf9a0 0x7f32f00039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.984+0000 7f330f400640 1 -- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32f0004d00 msgr2=0x7f32f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.984+0000 7f330f400640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32f0004d00 0x7f32f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.984+0000 7f330f400640 1 -- 172.21.15.141:0/574330692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f0003520 con 0x7f32f0004020 2024-01-16T15:09:54.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.984+0000 7f3314e00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32f0004d00 0x7f32f0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.985+0000 7f330f400640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32f0004020 0x7f32f00047c0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f33100cc790 tx=0x7f32f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912d358c197ac0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.985+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8005ad0 con 0x7f32f0004020 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.985+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f8005c30 con 0x7f32f0004020 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.985+0000 7f330ea00640 1 -- 172.21.15.141:0/574330692 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f802ca00 con 0x7f32f0004020 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.985+0000 7f3316e00640 1 -- 172.21.15.141:0/574330692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32f0004020 msgr2=0x7f32f00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.985+0000 7f3316e00640 1 --2- 172.21.15.141:0/574330692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32f0004020 0x7f32f00047c0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f33100cc790 tx=0x7f32f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.986+0000 7f3314e00640 1 -- 172.21.15.141:0/574330692 reap_dead start 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.986+0000 7f3316e00640 1 -- 172.21.15.141:0/574330692 shutdown_connections 2024-01-16T15:09:54.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.986+0000 7f3316e00640 1 -- 172.21.15.141:0/574330692 >> 172.21.15.141:0/574330692 conn(0x7f331006a800 msgr2=0x7f331006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:54.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.986+0000 7f3316e00640 1 -- 172.21.15.141:0/574330692 shutdown_connections 2024-01-16T15:09:54.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.986+0000 7f3316e00640 1 -- 172.21.15.141:0/574330692 wait complete. 2024-01-16T15:09:54.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.987+0000 7f3316e00640 1 Processor -- start 2024-01-16T15:09:54.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.988+0000 7f3316e00640 1 -- start start 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.988+0000 7f3316e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32f0004020 0x7f33101f0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.988+0000 7f3316e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33101f0ea0 0x7f33100b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3316e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33100b37d0 0x7f331014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3316e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33101f12e0 con 0x7f33100b37d0 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3316e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33100b3c50 con 0x7f33101f0ea0 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3316e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331014f500 con 0x7f32f0004020 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33100b37d0 0x7f331014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33100b37d0 0x7f331014df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51060/0 (socket says 172.21.15.141:51060) 2024-01-16T15:09:54.967 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 -- 172.21.15.141:0/3907456183 learned_addr learned my addr 172.21.15.141:0/3907456183 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:54.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f330fe00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32f0004020 0x7f33101f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 -- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32f0004020 msgr2=0x7f33101f0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32f0004020 0x7f33101f0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 -- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33101f0ea0 msgr2=0x7f33100b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33101f0ea0 0x7f33100b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.989+0000 7f3314e00640 1 -- 172.21.15.141:0/3907456183 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331014ecd0 con 0x7f33100b37d0 2024-01-16T15:09:54.968 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.990+0000 7f3314e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33100b37d0 0x7f331014df80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f33100cfea0 tx=0x7f3308007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.990+0000 7f3307e00640 1 -- 172.21.15.141:0/3907456183 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308003ce0 con 0x7f33100b37d0 2024-01-16T15:09:54.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.990+0000 7f3307e00640 1 -- 172.21.15.141:0/3907456183 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308003e40 con 0x7f33100b37d0 2024-01-16T15:09:54.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.990+0000 7f3307e00640 1 -- 172.21.15.141:0/3907456183 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308014640 con 0x7f33100b37d0 2024-01-16T15:09:54.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.990+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331014efb0 con 0x7f33100b37d0 2024-01-16T15:09:54.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.990+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331014fa30 con 0x7f33100b37d0 2024-01-16T15:09:54.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.991+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33100675d0 con 0x7f33100b37d0 2024-01-16T15:09:54.969 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.992+0000 7f3307e00640 1 -- 172.21.15.141:0/3907456183 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f330801d3b0 con 0x7f33100b37d0 2024-01-16T15:09:54.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.993+0000 7f3307e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f32e806bfb0 0x7f32e806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.993+0000 7f3307e00640 1 -- 172.21.15.141:0/3907456183 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f33080886c0 con 0x7f33100b37d0 2024-01-16T15:09:54.971 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.994+0000 7f330fe00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f32e806bfb0 0x7f32e806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.995+0000 7f330fe00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f32e806bfb0 0x7f32e806e470 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3300001550 tx=0x7f3300009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:54.998+0000 7f3307e00640 1 -- 172.21.15.141:0/3907456183 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3308054ef0 con 0x7f33100b37d0 2024-01-16T15:09:55.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.223+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f331006df00 con 0x7f33100b37d0 2024-01-16T15:09:55.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.224+0000 7f3307e00640 1 -- 172.21.15.141:0/3907456183 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f3308019070 con 0x7f33100b37d0 2024-01-16T15:09:55.202 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:55.202 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3894915042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3894915042},{"type":"v1","addr":"172.21.15.188:6835","nonce":3894915042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/3263318622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":3263318622},{"type":"v1","addr":"172.21.15.188:6837","nonce":3263318622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3550237844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3550237844},{"type":"v1","addr":"172.21.15.141:6833","nonce":3550237844}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3870765964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3870765964},{"type":"v1","addr":"172.21.15.141:6835","nonce":3870765964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 49 2024-01-16T15:09:55.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f32e806bfb0 msgr2=0x7f32e806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f32e806bfb0 0x7f32e806e470 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3300001550 tx=0x7f3300009040 comp rx=0 tx=0).stop 2024-01-16T15:09:55.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33100b37d0 msgr2=0x7f331014df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33100b37d0 0x7f331014df80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f33100cfea0 tx=0x7f3308007700 comp rx=0 tx=0).stop 2024-01-16T15:09:55.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 shutdown_connections 2024-01-16T15:09:55.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f32e806bfb0 0x7f32e806e470 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f33100b37d0 0x7f331014df80 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33101f0ea0 0x7f33100b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 --2- 172.21.15.141:0/3907456183 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32f0004020 0x7f33101f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 >> 172.21.15.141:0/3907456183 conn(0x7f331006a800 msgr2=0x7f331015cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:55.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 shutdown_connections 2024-01-16T15:09:55.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.227+0000 7f3316e00640 1 -- 172.21.15.141:0/3907456183 wait complete. 2024-01-16T15:09:55.224 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:09:55.224 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:09:55.225 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:09:55.225 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:55.225 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:09:55.225 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:09:55.225 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:09:55.225 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:55.225 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:09:55.225 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:09:55.226 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:09:55.226 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:55.226 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:09:55.226 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:09:55.226 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:09:55.226 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:09:55.244 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:09:55.244 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:09:55.248 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:09:55.248 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:09:55.268 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:09:55.268 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:09:55.269 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:09:55.270 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:09:55.295 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:09:55.296 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:55.302 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:09:55.302 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:55.316 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:09:55.316 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:55.322 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:09:55.323 DEBUG:teuthology.parallel:result is None 2024-01-16T15:09:55.323 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:09:55.454 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T15:09:55.454 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:09:55.472 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T15:09:55.479 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T15:09:55.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.529+0000 7f3aa4e00640 1 Processor -- start 2024-01-16T15:09:55.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.529+0000 7f3aa4e00640 1 -- start start 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.530+0000 7f3aa4e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3aa013df80 0x7f3aa015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.530+0000 7f3aa4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa00cf9a0 0x7f3aa00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.530+0000 7f3aa4e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa006c470 con 0x7f3aa013e360 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3aa4e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa006c750 con 0x7f3aa013df80 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3aa4e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa006ca30 con 0x7f3aa00cf9a0 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a9de00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3aa013df80 0x7f3aa015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a9e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa00cf9a0 0x7f3aa00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a95200640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f3aa013e360 0x7f3aa00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:52686/0 (socket says 172.21.15.141:52686) 2024-01-16T15:09:55.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a95200640 1 -- 172.21.15.141:0/1634605015 learned_addr learned my addr 172.21.15.141:0/1634605015 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:55.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a9e800640 1 -- 172.21.15.141:0/1634605015 >> v2:172.21.15.188:3300/0 conn(0x7f3aa013df80 msgr2=0x7f3aa015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:55.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a9e800640 1 --2- 172.21.15.141:0/1634605015 >> v2:172.21.15.188:3300/0 conn(0x7f3aa013df80 0x7f3aa015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a9e800640 1 -- 172.21.15.141:0/1634605015 >> v1:172.21.15.188:6789/0 conn(0x7f3aa013e360 legacy=0x7f3aa00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.531+0000 7f3a9e800640 1 -- 172.21.15.141:0/1634605015 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa006cd10 con 0x7f3aa00cf9a0 2024-01-16T15:09:55.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.532+0000 7f3a9e800640 1 --2- 172.21.15.141:0/1634605015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa00cf9a0 0x7f3aa00cd840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a980097a0 tx=0x7f3a9802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a320de99796b9964 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:55.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3a9d400640 1 -- 172.21.15.141:0/1634605015 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98031830 con 0x7f3aa00cf9a0 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3a9d400640 1 -- 172.21.15.141:0/1634605015 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a98031990 con 0x7f3aa00cf9a0 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3a9d400640 1 -- 172.21.15.141:0/1634605015 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98031d20 con 0x7f3aa00cf9a0 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1634605015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa00cf9a0 msgr2=0x7f3aa00cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1634605015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa00cf9a0 0x7f3aa00cd840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a980097a0 tx=0x7f3a9802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1634605015 shutdown_connections 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1634605015 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa00cf9a0 0x7f3aa00cd840 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1634605015 >> v2:172.21.15.188:3300/0 conn(0x7f3aa013df80 0x7f3aa015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.533+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1634605015 >> 172.21.15.141:0/1634605015 conn(0x7f3aa006a800 msgr2=0x7f3aa006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.534+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1634605015 shutdown_connections 2024-01-16T15:09:55.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.534+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1634605015 wait complete. 2024-01-16T15:09:55.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.535+0000 7f3aa4e00640 1 Processor -- start 2024-01-16T15:09:55.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.535+0000 7f3aa4e00640 1 -- start start 2024-01-16T15:09:55.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.536+0000 7f3aa4e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3aa00cf9a0 0x7f3aa01f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.536+0000 7f3aa4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3aa013df80 0x7f3aa00b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.536+0000 7f3aa4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa013e360 0x7f3aa00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.536+0000 7f3aa4e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00702c0 con 0x7f3aa013df80 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.536+0000 7f3aa4e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00ad320 con 0x7f3aa00cf9a0 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.536+0000 7f3aa4e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00ad600 con 0x7f3aa013e360 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3aa013df80 0x7f3aa00b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3aa013df80 0x7f3aa00b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51094/0 (socket says 172.21.15.141:51094) 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 -- 172.21.15.141:0/1570787373 learned_addr learned my addr 172.21.15.141:0/1570787373 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a9de00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3aa00cf9a0 0x7f3aa01f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a9e800640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa013e360 0x7f3aa00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 -- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa013e360 msgr2=0x7f3aa00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa013e360 0x7f3aa00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 -- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3aa00cf9a0 msgr2=0x7f3aa01f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3aa00cf9a0 0x7f3aa01f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 -- 172.21.15.141:0/1570787373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a98009400 con 0x7f3aa013df80 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a9e800640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa013e360 0x7f3aa00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a9de00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3aa00cf9a0 0x7f3aa01f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a95200640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3aa013df80 0x7f3aa00b0270 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800c500 tx=0x7f3a8800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.537+0000 7f3a96a00640 1 -- 172.21.15.141:0/1570787373 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88005ce0 con 0x7f3aa013df80 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.538+0000 7f3a96a00640 1 -- 172.21.15.141:0/1570787373 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a88005e40 con 0x7f3aa013df80 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.538+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa0070510 con 0x7f3aa013df80 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.538+0000 7f3a96a00640 1 -- 172.21.15.141:0/1570787373 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a880072f0 con 0x7f3aa013df80 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.538+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa013da30 con 0x7f3aa013df80 2024-01-16T15:09:55.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.539+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa0070a30 con 0x7f3aa013df80 2024-01-16T15:09:55.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.540+0000 7f3a96a00640 1 -- 172.21.15.141:0/1570787373 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f3a88007510 con 0x7f3aa013df80 2024-01-16T15:09:55.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.541+0000 7f3a96a00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7806bfb0 0x7f3a7806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.541+0000 7f3a96a00640 1 -- 172.21.15.141:0/1570787373 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3a88087ae0 con 0x7f3aa013df80 2024-01-16T15:09:55.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.541+0000 7f3a9de00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7806bfb0 0x7f3a7806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.542+0000 7f3a9de00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7806bfb0 0x7f3a7806e470 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3a900074c0 tx=0x7f3a9000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:55.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.545+0000 7f3a96a00640 1 -- 172.21.15.141:0/1570787373 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3aa0070a30 con 0x7f3aa013df80 2024-01-16T15:09:55.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:55.746+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3aa0070a30 con 0x7f3aa013df80 2024-01-16T15:09:56.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.860+0000 7f3a96a00640 1 -- 172.21.15.141:0/1570787373 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3aa0070a30 con 0x7f3aa013df80 2024-01-16T15:09:56.838 INFO:teuthology.orchestra.run.smithi141.stderr: removed all blocklist entries 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.863+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7806bfb0 msgr2=0x7f3a7806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.863+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7806bfb0 0x7f3a7806e470 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3a900074c0 tx=0x7f3a9000b040 comp rx=0 tx=0).stop 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.863+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3aa013df80 msgr2=0x7f3aa00b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.863+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3aa013df80 0x7f3aa00b0270 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800c500 tx=0x7f3a8800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 shutdown_connections 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7806bfb0 0x7f3a7806e470 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3aa013e360 0x7f3aa00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3aa013df80 0x7f3aa00b0270 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 --2- 172.21.15.141:0/1570787373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3aa00cf9a0 0x7f3aa01f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 >> 172.21.15.141:0/1570787373 conn(0x7f3aa006a800 msgr2=0x7f3aa006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 shutdown_connections 2024-01-16T15:09:56.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:56.864+0000 7f3aa4e00640 1 -- 172.21.15.141:0/1570787373 wait complete. 2024-01-16T15:09:56.863 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:09:57.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.086+0000 7f0c4fa00640 1 Processor -- start 2024-01-16T15:09:57.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.086+0000 7f0c4fa00640 1 -- start start 2024-01-16T15:09:57.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4fa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0c4813df80 0x7f0c4815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4fa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c480cf9a0 0x7f0c480cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4fa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4806c470 con 0x7f0c4813e360 2024-01-16T15:09:57.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4fa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4806c750 con 0x7f0c480cf9a0 2024-01-16T15:09:57.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4fa00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4806ca30 con 0x7f0c4813df80 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4da00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c480cf9a0 0x7f0c480cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4da00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c480cf9a0 0x7f0c480cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51106/0 (socket says 172.21.15.141:51106) 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4da00640 1 -- 172.21.15.141:0/84730091 learned_addr learned my addr 172.21.15.141:0/84730091 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4d000640 1 --2- 172.21.15.141:0/84730091 >> v2:172.21.15.188:3300/0 conn(0x7f0c4813df80 0x7f0c4815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4da00640 1 -- 172.21.15.141:0/84730091 >> v2:172.21.15.188:3300/0 conn(0x7f0c4813df80 msgr2=0x7f0c4815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4da00640 1 --2- 172.21.15.141:0/84730091 >> v2:172.21.15.188:3300/0 conn(0x7f0c4813df80 0x7f0c4815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4da00640 1 -- 172.21.15.141:0/84730091 >> v1:172.21.15.188:6789/0 conn(0x7f0c4813e360 legacy=0x7f0c480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.088+0000 7f0c4da00640 1 -- 172.21.15.141:0/84730091 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4806cd10 con 0x7f0c480cf9a0 2024-01-16T15:09:57.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.089+0000 7f0c4da00640 1 --2- 172.21.15.141:0/84730091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c480cf9a0 0x7f0c480cd840 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f0c48072770 tx=0x7f0c4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f4784e3a3e72a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.090+0000 7f0c47400640 1 -- 172.21.15.141:0/84730091 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4002ca80 con 0x7f0c480cf9a0 2024-01-16T15:09:57.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.090+0000 7f0c47400640 1 -- 172.21.15.141:0/84730091 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4002cbe0 con 0x7f0c480cf9a0 2024-01-16T15:09:57.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.090+0000 7f0c47400640 1 -- 172.21.15.141:0/84730091 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c400343f0 con 0x7f0c480cf9a0 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 -- 172.21.15.141:0/84730091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c480cf9a0 msgr2=0x7f0c480cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/84730091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c480cf9a0 0x7f0c480cd840 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f0c48072770 tx=0x7f0c4002bee0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 -- 172.21.15.141:0/84730091 shutdown_connections 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/84730091 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c480cf9a0 0x7f0c480cd840 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/84730091 >> v2:172.21.15.188:3300/0 conn(0x7f0c4813df80 0x7f0c4815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 -- 172.21.15.141:0/84730091 >> 172.21.15.141:0/84730091 conn(0x7f0c4806a800 msgr2=0x7f0c4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 -- 172.21.15.141:0/84730091 shutdown_connections 2024-01-16T15:09:57.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.091+0000 7f0c4fa00640 1 -- 172.21.15.141:0/84730091 wait complete. 2024-01-16T15:09:57.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.093+0000 7f0c4fa00640 1 Processor -- start 2024-01-16T15:09:57.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.093+0000 7f0c4fa00640 1 -- start start 2024-01-16T15:09:57.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4fa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c480cf9a0 0x7f0c481540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c4813df80 0x7f0c4815dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4fa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c4813e360 0x7f0c480ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4fa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c48154600 con 0x7f0c4813e360 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4fa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c480a94f0 con 0x7f0c480cf9a0 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c480ad1a0 con 0x7f0c4813df80 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4da00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c4813e360 0x7f0c480ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4da00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c4813e360 0x7f0c480ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51108/0 (socket says 172.21.15.141:51108) 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.094+0000 7f0c4da00640 1 -- 172.21.15.141:0/281739072 learned_addr learned my addr 172.21.15.141:0/281739072 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4d000640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c480cf9a0 0x7f0c481540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c47e00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c4813df80 0x7f0c4815dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4da00640 1 -- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c4813df80 msgr2=0x7f0c4815dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4da00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c4813df80 0x7f0c4815dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4da00640 1 -- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c480cf9a0 msgr2=0x7f0c481540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4da00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c480cf9a0 0x7f0c481540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4da00640 1 -- 172.21.15.141:0/281739072 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40009b30 con 0x7f0c4813e360 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c47e00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c4813df80 0x7f0c4815dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4da00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c4813e360 0x7f0c480ac630 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f0c40036040 tx=0x7f0c40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c44c00640 1 -- 172.21.15.141:0/281739072 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4003e450 con 0x7f0c4813e360 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.095+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c48153300 con 0x7f0c4813e360 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.096+0000 7f0c44c00640 1 -- 172.21.15.141:0/281739072 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4003e5d0 con 0x7f0c4813e360 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.096+0000 7f0c44c00640 1 -- 172.21.15.141:0/281739072 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4003e980 con 0x7f0c4813e360 2024-01-16T15:09:57.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.096+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c481538c0 con 0x7f0c4813e360 2024-01-16T15:09:57.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.097+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c480675d0 con 0x7f0c4813e360 2024-01-16T15:09:57.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.097+0000 7f0c44c00640 1 -- 172.21.15.141:0/281739072 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f0c4002cd50 con 0x7f0c4813e360 2024-01-16T15:09:57.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.098+0000 7f0c44c00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c2006bfb0 0x7f0c2006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.098+0000 7f0c44c00640 1 -- 172.21.15.141:0/281739072 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0c400ab460 con 0x7f0c4813e360 2024-01-16T15:09:57.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.099+0000 7f0c4d000640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c2006bfb0 0x7f0c2006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.100+0000 7f0c4d000640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c2006bfb0 0x7f0c2006e470 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0c38006fd0 tx=0x7f0c38008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.103+0000 7f0c44c00640 1 -- 172.21.15.141:0/281739072 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c40077af0 con 0x7f0c4813e360 2024-01-16T15:09:57.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.361+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0c4806df00 con 0x7f0c4813e360 2024-01-16T15:09:57.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.362+0000 7f0c44c00640 1 -- 172.21.15.141:0/281739072 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0c40043020 con 0x7f0c4813e360 2024-01-16T15:09:57.340 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:57.340 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:09:57.340 INFO:teuthology.orchestra.run.smithi141.stdout: "auth_dump": [ 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.a", 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ==", 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.341 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.b", 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg==", 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.c", 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw==", 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.d", 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg==", 2024-01-16T15:09:57.343 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.0", 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA==", 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.344 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.1", 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg==", 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.2", 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g==", 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.3", 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w==", 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.346 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.4", 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ==", 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.347 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.5", 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA==", 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.348 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.6", 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ==", 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.7", 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg==", 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.0", 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg==", 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow r", 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.1", 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ==", 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.2", 2024-01-16T15:09:57.351 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ==", 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.3", 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA==", 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:09:57.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.admin", 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAEmaZl0VgDMRAABOWZ0senIXHFvij8FQHiAg==", 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:09:57.353 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.x", 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ==", 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:57.354 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.y", 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA==", 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2024-01-16T15:09:57.355 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:09:57.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c2006bfb0 msgr2=0x7f0c2006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c2006bfb0 0x7f0c2006e470 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0c38006fd0 tx=0x7f0c38008040 comp rx=0 tx=0).stop 2024-01-16T15:09:57.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c4813e360 msgr2=0x7f0c480ac630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c4813e360 0x7f0c480ac630 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f0c40036040 tx=0x7f0c40004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:57.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 shutdown_connections 2024-01-16T15:09:57.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c2006bfb0 0x7f0c2006e470 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c4813e360 0x7f0c480ac630 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c4813df80 0x7f0c4815dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 --2- 172.21.15.141:0/281739072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c480cf9a0 0x7f0c481540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 >> 172.21.15.141:0/281739072 conn(0x7f0c4806a800 msgr2=0x7f0c4815aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 shutdown_connections 2024-01-16T15:09:57.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.365+0000 7f0c4fa00640 1 -- 172.21.15.141:0/281739072 wait complete. 2024-01-16T15:09:57.361 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:09:57.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.591+0000 7fefa2e00640 1 Processor -- start 2024-01-16T15:09:57.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.591+0000 7fefa2e00640 1 -- start start 2024-01-16T15:09:57.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fefa2e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fef9c13df80 0x7fef9c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fefa2e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fef9c13e360 0x7fef9c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fefa2e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c06c470 con 0x7fef9c15f9c0 2024-01-16T15:09:57.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fefa2e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c06c750 con 0x7fef9c13e360 2024-01-16T15:09:57.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fefa2e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c06ca30 con 0x7fef9c13df80 2024-01-16T15:09:57.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef93200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fef9c13e360 0x7fef9c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef93200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fef9c13e360 0x7fef9c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51112/0 (socket says 172.21.15.141:51112) 2024-01-16T15:09:57.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef93200640 1 -- 172.21.15.141:0/2482764239 learned_addr learned my addr 172.21.15.141:0/2482764239 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:57.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef9be00640 1 --2- 172.21.15.141:0/2482764239 >> v2:172.21.15.188:3300/0 conn(0x7fef9c13df80 0x7fef9c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef93200640 1 -- 172.21.15.141:0/2482764239 >> v2:172.21.15.188:3300/0 conn(0x7fef9c13df80 msgr2=0x7fef9c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef93200640 1 --2- 172.21.15.141:0/2482764239 >> v2:172.21.15.188:3300/0 conn(0x7fef9c13df80 0x7fef9c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef93200640 1 -- 172.21.15.141:0/2482764239 >> v1:172.21.15.188:6789/0 conn(0x7fef9c15f9c0 legacy=0x7fef9c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.593+0000 7fef93200640 1 -- 172.21.15.141:0/2482764239 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c06cd10 con 0x7fef9c13e360 2024-01-16T15:09:57.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.595+0000 7fef93200640 1 --2- 172.21.15.141:0/2482764239 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c13e360 0x7fef9c0cf290 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fef9c072770 tx=0x7fef8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7930f573167a6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.595+0000 7fef9b400640 1 -- 172.21.15.141:0/2482764239 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84031870 con 0x7fef9c13e360 2024-01-16T15:09:57.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.595+0000 7fef9b400640 1 -- 172.21.15.141:0/2482764239 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef840319d0 con 0x7fef9c13e360 2024-01-16T15:09:57.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.595+0000 7fef9b400640 1 -- 172.21.15.141:0/2482764239 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84031d60 con 0x7fef9c13e360 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 -- 172.21.15.141:0/2482764239 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c13e360 msgr2=0x7fef9c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 --2- 172.21.15.141:0/2482764239 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c13e360 0x7fef9c0cf290 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fef9c072770 tx=0x7fef8402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 -- 172.21.15.141:0/2482764239 shutdown_connections 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 --2- 172.21.15.141:0/2482764239 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c13e360 0x7fef9c0cf290 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 --2- 172.21.15.141:0/2482764239 >> v2:172.21.15.188:3300/0 conn(0x7fef9c13df80 0x7fef9c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 -- 172.21.15.141:0/2482764239 >> 172.21.15.141:0/2482764239 conn(0x7fef9c06a800 msgr2=0x7fef9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 -- 172.21.15.141:0/2482764239 shutdown_connections 2024-01-16T15:09:57.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.596+0000 7fefa2e00640 1 -- 172.21.15.141:0/2482764239 wait complete. 2024-01-16T15:09:57.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.598+0000 7fefa2e00640 1 Processor -- start 2024-01-16T15:09:57.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.598+0000 7fefa2e00640 1 -- start start 2024-01-16T15:09:57.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa2e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c13df80 0x7fef9c13d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa2e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef9c13e360 0x7fef9c150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa2e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c15f9c0 0x7fef9c150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa2e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c15d580 con 0x7fef9c15f9c0 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa2e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c15d860 con 0x7fef9c13df80 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa2e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c15db40 con 0x7fef9c13e360 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c15f9c0 0x7fef9c150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c15f9c0 0x7fef9c150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51120/0 (socket says 172.21.15.141:51120) 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fefa0e00640 1 -- 172.21.15.141:0/928234637 learned_addr learned my addr 172.21.15.141:0/928234637 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.599+0000 7fef9be00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c13df80 0x7fef9c13d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa0e00640 1 -- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef9c13e360 msgr2=0x7fef9c150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa0e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef9c13e360 0x7fef9c150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa0e00640 1 -- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c13df80 msgr2=0x7fef9c13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa0e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c13df80 0x7fef9c13d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa0e00640 1 -- 172.21.15.141:0/928234637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef840094d0 con 0x7fef9c15f9c0 2024-01-16T15:09:57.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa0e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c15f9c0 0x7fef9c150c80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fef9400c780 tx=0x7fef9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fef98c00640 1 -- 172.21.15.141:0/928234637 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94003d80 con 0x7fef9c15f9c0 2024-01-16T15:09:57.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fef98c00640 1 -- 172.21.15.141:0/928234637 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9400ce30 con 0x7fef9c15f9c0 2024-01-16T15:09:57.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fef98c00640 1 -- 172.21.15.141:0/928234637 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94005290 con 0x7fef9c15f9c0 2024-01-16T15:09:57.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c15de20 con 0x7fef9c15f9c0 2024-01-16T15:09:57.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.600+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c13da60 con 0x7fef9c15f9c0 2024-01-16T15:09:57.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.601+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c12f7a0 con 0x7fef9c15f9c0 2024-01-16T15:09:57.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.602+0000 7fef98c00640 1 -- 172.21.15.141:0/928234637 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fef94006b10 con 0x7fef9c15f9c0 2024-01-16T15:09:57.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.603+0000 7fef98c00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef7406bfb0 0x7fef7406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.603+0000 7fef98c00640 1 -- 172.21.15.141:0/928234637 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fef94087920 con 0x7fef9c15f9c0 2024-01-16T15:09:57.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.604+0000 7fef9be00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef7406bfb0 0x7fef7406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.605+0000 7fef9be00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef7406bfb0 0x7fef7406e470 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fef8c006fd0 tx=0x7fef8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.608+0000 7fef98c00640 1 -- 172.21.15.141:0/928234637 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef9c12f7a0 con 0x7fef9c15f9c0 2024-01-16T15:09:57.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.802+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef9c12f7a0 con 0x7fef9c15f9c0 2024-01-16T15:09:57.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.803+0000 7fef98c00640 1 -- 172.21.15.141:0/928234637 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+12906 (secure 0 0 0) 0x7fef9c12f7a0 con 0x7fef9c15f9c0 2024-01-16T15:09:57.781 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:57.781 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":85,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:09:56.857683+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:09:57.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.806+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef7406bfb0 msgr2=0x7fef7406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.806+0000 7fefa2e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef7406bfb0 0x7fef7406e470 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fef8c006fd0 tx=0x7fef8c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:57.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.806+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c15f9c0 msgr2=0x7fef9c150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.806+0000 7fefa2e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c15f9c0 0x7fef9c150c80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fef9400c780 tx=0x7fef9400cc40 comp rx=0 tx=0).stop 2024-01-16T15:09:57.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.806+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 shutdown_connections 2024-01-16T15:09:57.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.807+0000 7fefa2e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fef7406bfb0 0x7fef7406e470 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.807+0000 7fefa2e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef9c15f9c0 0x7fef9c150c80 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.807+0000 7fefa2e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fef9c13e360 0x7fef9c150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.807+0000 7fefa2e00640 1 --2- 172.21.15.141:0/928234637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c13df80 0x7fef9c13d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.807+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 >> 172.21.15.141:0/928234637 conn(0x7fef9c06a800 msgr2=0x7fef9c0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.807+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 shutdown_connections 2024-01-16T15:09:57.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:57.807+0000 7fefa2e00640 1 -- 172.21.15.141:0/928234637 wait complete. 2024-01-16T15:09:57.804 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:09:57.804 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:09:58.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.035+0000 7f38dbe00640 1 Processor -- start 2024-01-16T15:09:58.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.035+0000 7f38dbe00640 1 -- start start 2024-01-16T15:09:58.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38dbe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f38d413df80 0x7f38d415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38dbe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d406c470 con 0x7f38d413e360 2024-01-16T15:09:58.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38dbe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d406c750 con 0x7f38d40cf9a0 2024-01-16T15:09:58.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38dbe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d406ca30 con 0x7f38d413df80 2024-01-16T15:09:58.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38d8a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f38d413e360 0x7f38d40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56156/0 (socket says 172.21.15.141:56156) 2024-01-16T15:09:58.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38d8a00640 1 -- 172.21.15.141:0/2812129500 learned_addr learned my addr 172.21.15.141:0/2812129500 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:09:58.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38d9400640 1 --2- 172.21.15.141:0/2812129500 >> v2:172.21.15.188:3300/0 conn(0x7f38d413df80 0x7f38d415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101329386 0 0) 0x7f38d406c470 con 0x7f38d413e360 2024-01-16T15:09:58.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.037+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4003520 con 0x7f38d413e360 2024-01-16T15:09:58.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3703130282 0 0) 0x7f38b4003520 con 0x7f38d413e360 2024-01-16T15:09:58.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 >> v2:172.21.15.188:3300/0 conn(0x7f38d413df80 msgr2=0x7f38d415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 --2- 172.21.15.141:0/2812129500 >> v2:172.21.15.188:3300/0 conn(0x7f38d413df80 0x7f38d415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 >> v1:172.21.15.188:6789/0 conn(0x7f38d40cf9a0 legacy=0x7f38d40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d406cd10 con 0x7f38d413e360 2024-01-16T15:09:58.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f38bc001670 con 0x7f38d413e360 2024-01-16T15:09:58.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 >> v1:172.21.15.141:6789/0 conn(0x7f38d413e360 legacy=0x7f38d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b40039c0 0x7f38b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b40047c0 0x7f38b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38b400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d406ca30 con 0x7f38b4005180 2024-01-16T15:09:58.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b400c080 con 0x7f38b40047c0 2024-01-16T15:09:58.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b400c240 con 0x7f38b40039c0 2024-01-16T15:09:58.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.038+0000 7f38d9e00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38b400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d8a00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b40047c0 0x7f38b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d9400640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b40039c0 0x7f38b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d9e00640 1 -- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b40039c0 msgr2=0x7f38b4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d9e00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b40039c0 0x7f38b4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d9e00640 1 -- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b40047c0 msgr2=0x7f38b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d9e00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b40047c0 0x7f38b4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d9e00640 1 -- 172.21.15.141:0/2812129500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4003520 con 0x7f38b4005180 2024-01-16T15:09:58.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.039+0000 7f38d9400640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b40039c0 0x7f38b4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:58.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.040+0000 7f38d9e00640 1 -- 172.21.15.141:0/2812129500 reap_dead start 2024-01-16T15:09:58.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.040+0000 7f38d9e00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38b400b980 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f38cc001e20 tx=0x7f38b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e15994d9fd8854 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.040+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc003b70 con 0x7f38b4005180 2024-01-16T15:09:58.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.040+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc003cd0 con 0x7f38b4005180 2024-01-16T15:09:58.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.040+0000 7f38d3e00640 1 -- 172.21.15.141:0/2812129500 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc005310 con 0x7f38b4005180 2024-01-16T15:09:58.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.041+0000 7f38dbe00640 1 -- 172.21.15.141:0/2812129500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 msgr2=0x7f38b400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.041+0000 7f38dbe00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38b400b980 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f38cc001e20 tx=0x7f38b40057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.041+0000 7f38dbe00640 1 -- 172.21.15.141:0/2812129500 shutdown_connections 2024-01-16T15:09:58.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.041+0000 7f38dbe00640 1 --2- 172.21.15.141:0/2812129500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38b400b980 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.041+0000 7f38dbe00640 1 -- 172.21.15.141:0/2812129500 >> 172.21.15.141:0/2812129500 conn(0x7f38d406a800 msgr2=0x7f38d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.041+0000 7f38dbe00640 1 -- 172.21.15.141:0/2812129500 shutdown_connections 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.041+0000 7f38dbe00640 1 -- 172.21.15.141:0/2812129500 wait complete. 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.043+0000 7f38dbe00640 1 Processor -- start 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.043+0000 7f38dbe00640 1 -- start start 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38dbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38d41f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38dbe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38d4147db0 0x7f38d415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38dbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38d4148230 0x7f38d415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38dbe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d41f0490 con 0x7f38b4005180 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38dbe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d415c850 con 0x7f38d4148230 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38dbe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d415caa0 con 0x7f38d4147db0 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38d41f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38d41f69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51140/0 (socket says 172.21.15.141:51140) 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 -- 172.21.15.141:0/1315162887 learned_addr learned my addr 172.21.15.141:0/1315162887 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 -- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38d4147db0 msgr2=0x7f38d415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38d4147db0 0x7f38d415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 -- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38d4148230 msgr2=0x7f38d415dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38d4148230 0x7f38d415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.044+0000 7f38d9400640 1 -- 172.21.15.141:0/1315162887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d415cd80 con 0x7f38b4005180 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.045+0000 7f38d9400640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38d41f69c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f38c40108b0 tx=0x7f38c4010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.045+0000 7f38d1600640 1 -- 172.21.15.141:0/1315162887 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c400eb60 con 0x7f38b4005180 2024-01-16T15:09:58.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.045+0000 7f38d1600640 1 -- 172.21.15.141:0/1315162887 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c400ecc0 con 0x7f38b4005180 2024-01-16T15:09:58.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.045+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d415d060 con 0x7f38b4005180 2024-01-16T15:09:58.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.045+0000 7f38d1600640 1 -- 172.21.15.141:0/1315162887 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c401c400 con 0x7f38b4005180 2024-01-16T15:09:58.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.045+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d41f6f00 con 0x7f38b4005180 2024-01-16T15:09:58.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.046+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d415ea40 con 0x7f38b4005180 2024-01-16T15:09:58.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.048+0000 7f38d1600640 1 -- 172.21.15.141:0/1315162887 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f38c401c560 con 0x7f38b4005180 2024-01-16T15:09:58.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.048+0000 7f38d1600640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f38ac06bfb0 0x7f38ac06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.049+0000 7f38d1600640 1 -- 172.21.15.141:0/1315162887 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f38c4088c10 con 0x7f38b4005180 2024-01-16T15:09:58.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.049+0000 7f38d8a00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f38ac06bfb0 0x7f38ac06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.051+0000 7f38d8a00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f38ac06bfb0 0x7f38ac06e470 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f38bc001050 tx=0x7f38bc000f60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.054+0000 7f38d1600640 1 -- 172.21.15.141:0/1315162887 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38c4055580 con 0x7f38b4005180 2024-01-16T15:09:58.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.250+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38d40675d0 con 0x7f38b4005180 2024-01-16T15:09:58.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.251+0000 7f38d1600640 1 -- 172.21.15.141:0/1315162887 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38c4058ba0 con 0x7f38b4005180 2024-01-16T15:09:58.229 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.253+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f38ac06bfb0 msgr2=0x7f38ac06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.253+0000 7f38dbe00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f38ac06bfb0 0x7f38ac06e470 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f38bc001050 tx=0x7f38bc000f60 comp rx=0 tx=0).stop 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.253+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 msgr2=0x7f38d41f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.253+0000 7f38dbe00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38d41f69c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f38c40108b0 tx=0x7f38c4010d70 comp rx=0 tx=0).stop 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 shutdown_connections 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f38ac06bfb0 0x7f38ac06e470 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38d4148230 0x7f38d415dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38d4147db0 0x7f38d415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 --2- 172.21.15.141:0/1315162887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b4005180 0x7f38d41f69c0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 >> 172.21.15.141:0/1315162887 conn(0x7f38d406a800 msgr2=0x7f38d40af590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 shutdown_connections 2024-01-16T15:09:58.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.254+0000 7f38dbe00640 1 -- 172.21.15.141:0/1315162887 wait complete. 2024-01-16T15:09:58.252 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:09:58.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.481+0000 7f2de2a00640 1 Processor -- start 2024-01-16T15:09:58.459 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.481+0000 7f2de2a00640 1 -- start start 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.483+0000 7f2de2a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ddc13e360 0x7f2ddc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.483+0000 7f2de2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.483+0000 7f2de2a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc15c5f0 con 0x7f2ddc13df80 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.483+0000 7f2de2a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc06c470 con 0x7f2ddc13e360 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.483+0000 7f2de2a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc06c750 con 0x7f2ddc15f9c0 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.483+0000 7f2dd3200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ddc13e360 0x7f2ddc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2dd3200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ddc13e360 0x7f2ddc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51150/0 (socket says 172.21.15.141:51150) 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2dd3200640 1 -- 172.21.15.141:0/1081214213 learned_addr learned my addr 172.21.15.141:0/1081214213 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2de0a00640 1 --2- 172.21.15.141:0/1081214213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2dd3200640 1 -- 172.21.15.141:0/1081214213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 msgr2=0x7f2ddc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2dd3200640 1 --2- 172.21.15.141:0/1081214213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2dd3200640 1 -- 172.21.15.141:0/1081214213 >> v1:172.21.15.188:6789/0 conn(0x7f2ddc13df80 legacy=0x7f2ddc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2dd3200640 1 -- 172.21.15.141:0/1081214213 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc06ca30 con 0x7f2ddc13e360 2024-01-16T15:09:58.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.484+0000 7f2de0a00640 1 --2- 172.21.15.141:0/1081214213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:58.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.485+0000 7f2dd3200640 1 --2- 172.21.15.141:0/1081214213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc0cf290 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009870 tx=0x7f2dc402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30d53f653d4d1bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.486+0000 7f2ddb400640 1 -- 172.21.15.141:0/1081214213 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4030840 con 0x7f2ddc13e360 2024-01-16T15:09:58.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.486+0000 7f2ddb400640 1 -- 172.21.15.141:0/1081214213 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc40309a0 con 0x7f2ddc13e360 2024-01-16T15:09:58.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.486+0000 7f2ddb400640 1 -- 172.21.15.141:0/1081214213 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4030d30 con 0x7f2ddc13e360 2024-01-16T15:09:58.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 -- 172.21.15.141:0/1081214213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 msgr2=0x7f2ddc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 --2- 172.21.15.141:0/1081214213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc0cf290 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009870 tx=0x7f2dc402b980 comp rx=0 tx=0).stop 2024-01-16T15:09:58.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 -- 172.21.15.141:0/1081214213 shutdown_connections 2024-01-16T15:09:58.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 --2- 172.21.15.141:0/1081214213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc0cf290 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 --2- 172.21.15.141:0/1081214213 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 -- 172.21.15.141:0/1081214213 >> 172.21.15.141:0/1081214213 conn(0x7f2ddc06a800 msgr2=0x7f2ddc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 -- 172.21.15.141:0/1081214213 shutdown_connections 2024-01-16T15:09:58.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.487+0000 7f2de2a00640 1 -- 172.21.15.141:0/1081214213 wait complete. 2024-01-16T15:09:58.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.489+0000 7f2de2a00640 1 Processor -- start 2024-01-16T15:09:58.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.489+0000 7f2de2a00640 1 -- start start 2024-01-16T15:09:58.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2de2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13df80 0x7f2ddc1f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2de2a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13e360 0x7f2ddc0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2de2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2de2a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc1f2050 con 0x7f2ddc13df80 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2de2a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc14c300 con 0x7f2ddc13e360 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2de2a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc1f1980 con 0x7f2ddc15f9c0 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2ddbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13df80 0x7f2ddc1f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2ddbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13df80 0x7f2ddc1f0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51158/0 (socket says 172.21.15.141:51158) 2024-01-16T15:09:58.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2046219274 learned_addr learned my addr 172.21.15.141:0/2046219274 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:58.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2dd3200640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13e360 0x7f2ddc0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2de0a00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 msgr2=0x7f2ddc0b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.490+0000 7f2ddbe00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13e360 msgr2=0x7f2ddc0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2ddbe00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13e360 0x7f2ddc0b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2046219274 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc40094d0 con 0x7f2ddc13df80 2024-01-16T15:09:58.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2ddbe00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13df80 0x7f2ddc1f0eb0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00c710 tx=0x7f2dcc00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2dd8c00640 1 -- 172.21.15.141:0/2046219274 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc003b90 con 0x7f2ddc13df80 2024-01-16T15:09:58.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2dd8c00640 1 -- 172.21.15.141:0/2046219274 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc003cf0 con 0x7f2ddc13df80 2024-01-16T15:09:58.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2dd8c00640 1 -- 172.21.15.141:0/2046219274 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc00aa90 con 0x7f2ddc13df80 2024-01-16T15:09:58.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc1f1c60 con 0x7f2ddc13df80 2024-01-16T15:09:58.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.491+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc14c830 con 0x7f2ddc13df80 2024-01-16T15:09:58.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.492+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc1f1e50 con 0x7f2ddc13df80 2024-01-16T15:09:58.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.493+0000 7f2dd8c00640 1 -- 172.21.15.141:0/2046219274 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f2dcc00abf0 con 0x7f2ddc13df80 2024-01-16T15:09:58.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.494+0000 7f2dd8c00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.494+0000 7f2dd8c00640 1 -- 172.21.15.141:0/2046219274 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2dcc088440 con 0x7f2ddc13df80 2024-01-16T15:09:58.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.495+0000 7f2dd3200640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.496+0000 7f2dd3200640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4036000 tx=0x7f2dc4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.499+0000 7f2dd8c00640 1 -- 172.21.15.141:0/2046219274 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dcc054d00 con 0x7f2ddc13df80 2024-01-16T15:09:58.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.694+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2ddc0cd420 con 0x7f2ddc13df80 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.694+0000 7f2dd8c00640 1 -- 172.21.15.141:0/2046219274 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2dcc058320 con 0x7f2ddc13df80 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T15:09:58.673 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:09:58.673 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:09:58.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 msgr2=0x7f2db406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4036000 tx=0x7f2dc4008030 comp rx=0 tx=0).stop 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13df80 msgr2=0x7f2ddc1f0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13df80 0x7f2ddc1f0eb0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00c710 tx=0x7f2dcc00cbd0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 shutdown_connections 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc15f9c0 0x7f2ddc0b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13e360 0x7f2ddc0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 --2- 172.21.15.141:0/2046219274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13df80 0x7f2ddc1f0eb0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 >> 172.21.15.141:0/2046219274 conn(0x7f2ddc06a800 msgr2=0x7f2ddc15d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.697+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 shutdown_connections 2024-01-16T15:09:58.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.698+0000 7f2de2a00640 1 -- 172.21.15.141:0/2046219274 wait complete. 2024-01-16T15:09:58.695 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:09:58.695 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:09:58.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.927+0000 7fb2fe000640 1 Processor -- start 2024-01-16T15:09:58.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.927+0000 7fb2fe000640 1 -- start start 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.928+0000 7fb2fe000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2f8150760 0x7fb2f8072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.928+0000 7fb2fe000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 0x7fb2f815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2fe000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f815dc80 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2fe000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f80b3770 con 0x7fb2f8150760 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2fe000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8071190 con 0x7fb2f815d300 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2f7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2f8150760 0x7fb2f8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2f7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2f8150760 0x7fb2f8072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51160/0 (socket says 172.21.15.141:51160) 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2f7400640 1 -- 172.21.15.141:0/228097778 learned_addr learned my addr 172.21.15.141:0/228097778 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2f7400640 1 -- 172.21.15.141:0/228097778 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 msgr2=0x7fb2f815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2f7400640 1 --2- 172.21.15.141:0/228097778 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 0x7fb2f815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2f7400640 1 -- 172.21.15.141:0/228097778 >> v1:172.21.15.188:6789/0 conn(0x7fb2f80b2bd0 legacy=0x7fb2f80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.929+0000 7fb2f7400640 1 -- 172.21.15.141:0/228097778 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8071470 con 0x7fb2f8150760 2024-01-16T15:09:58.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.930+0000 7fb2f7400640 1 --2- 172.21.15.141:0/228097778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f8150760 0x7fb2f8072230 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009870 tx=0x7fb2e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26032bab9c440c5d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.930+0000 7fb2f6a00640 1 -- 172.21.15.141:0/228097778 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8031640 con 0x7fb2f8150760 2024-01-16T15:09:58.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2f6a00640 1 -- 172.21.15.141:0/228097778 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e80317a0 con 0x7fb2f8150760 2024-01-16T15:09:58.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2f6a00640 1 -- 172.21.15.141:0/228097778 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8031b30 con 0x7fb2f8150760 2024-01-16T15:09:58.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 -- 172.21.15.141:0/228097778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f8150760 msgr2=0x7fb2f8072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 --2- 172.21.15.141:0/228097778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f8150760 0x7fb2f8072230 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009870 tx=0x7fb2e802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 -- 172.21.15.141:0/228097778 shutdown_connections 2024-01-16T15:09:58.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 --2- 172.21.15.141:0/228097778 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 0x7fb2f815d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 --2- 172.21.15.141:0/228097778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f8150760 0x7fb2f8072230 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 -- 172.21.15.141:0/228097778 >> 172.21.15.141:0/228097778 conn(0x7fb2f806a800 msgr2=0x7fb2f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 -- 172.21.15.141:0/228097778 shutdown_connections 2024-01-16T15:09:58.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.931+0000 7fb2fe000640 1 -- 172.21.15.141:0/228097778 wait complete. 2024-01-16T15:09:58.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.933+0000 7fb2fe000640 1 Processor -- start 2024-01-16T15:09:58.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.933+0000 7fb2fe000640 1 -- start start 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2fe000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f80b2bd0 0x7fb2f80adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2fe000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f8150760 0x7fb2f8153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2fe000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 0x7fb2f814a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2fe000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8159260 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2fe000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8153a90 con 0x7fb2f8150760 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2fe000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8140d90 con 0x7fb2f815d300 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2f7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f80b2bd0 0x7fb2f80adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2f7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f80b2bd0 0x7fb2f80adae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51164/0 (socket says 172.21.15.141:51164) 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.934+0000 7fb2f7400640 1 -- 172.21.15.141:0/3278027534 learned_addr learned my addr 172.21.15.141:0/3278027534 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2f7e00640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 0x7fb2f814a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2f7400640 1 -- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 msgr2=0x7fb2f814a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2f7400640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 0x7fb2f814a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2f7400640 1 -- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f8150760 msgr2=0x7fb2f8153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2ee800640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f8150760 0x7fb2f8153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2f7400640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f8150760 0x7fb2f8153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2f7400640 1 -- 172.21.15.141:0/3278027534 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80094d0 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2ee800640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f8150760 0x7fb2f8153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2f7400640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f80b2bd0 0x7fb2f80adae0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009870 tx=0x7fb2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2efe00640 1 -- 172.21.15.141:0/3278027534 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e803ac60 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2efe00640 1 -- 172.21.15.141:0/3278027534 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e803adc0 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.935+0000 7fb2efe00640 1 -- 172.21.15.141:0/3278027534 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8044630 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.936+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f80ae170 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.936+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f8158c10 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.937+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f80675d0 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.938+0000 7fb2efe00640 1 -- 172.21.15.141:0/3278027534 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fb2e802f050 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.939+0000 7fb2efe00640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2d006bfe0 0x7fb2d006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.939+0000 7fb2efe00640 1 -- 172.21.15.141:0/3278027534 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb2e80aa7f0 con 0x7fb2f80b2bd0 2024-01-16T15:09:58.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.940+0000 7fb2ee800640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2d006bfe0 0x7fb2d006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.941+0000 7fb2ee800640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2d006bfe0 0x7fb2d006e4a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb2f806cda0 tx=0x7fb2e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:58.944+0000 7fb2efe00640 1 -- 172.21.15.141:0/3278027534 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2e8076e80 con 0x7fb2f80b2bd0 2024-01-16T15:09:59.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.136+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb2f8141240 con 0x7fb2f80b2bd0 2024-01-16T15:09:59.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.886+0000 7fb2efe00640 1 -- 172.21.15.141:0/3278027534 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb2e807a4a0 con 0x7fb2f80b2bd0 2024-01-16T15:09:59.864 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T15:09:59.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2d006bfe0 msgr2=0x7fb2d006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2d006bfe0 0x7fb2d006e4a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb2f806cda0 tx=0x7fb2e0008040 comp rx=0 tx=0).stop 2024-01-16T15:09:59.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f80b2bd0 msgr2=0x7fb2f80adae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f80b2bd0 0x7fb2f80adae0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009870 tx=0x7fb2e80046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 shutdown_connections 2024-01-16T15:09:59.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb2d006bfe0 0x7fb2d006e4a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2f815d300 0x7fb2f814a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f8150760 0x7fb2f8153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 --2- 172.21.15.141:0/3278027534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2f80b2bd0 0x7fb2f80adae0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 >> 172.21.15.141:0/3278027534 conn(0x7fb2f806a800 msgr2=0x7fb2f806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:59.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.890+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 shutdown_connections 2024-01-16T15:09:59.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:09:59.891+0000 7fb2fe000640 1 -- 172.21.15.141:0/3278027534 wait complete. 2024-01-16T15:09:59.893 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:10:00.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.121+0000 7fc965e00640 1 Processor -- start 2024-01-16T15:10:00.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.121+0000 7fc965e00640 1 -- start start 2024-01-16T15:10:00.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.123+0000 7fc965e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc960160920 0x7fc9600b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.123+0000 7fc965e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9600d0a10 con 0x7fc9600cfea0 2024-01-16T15:10:00.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.123+0000 7fc965e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc960161e30 con 0x7fc9600cf9f0 2024-01-16T15:10:00.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.123+0000 7fc965e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9600d1f50 con 0x7fc960160920 2024-01-16T15:10:00.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.123+0000 7fc964a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc960160920 0x7fc9600b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.123+0000 7fc965400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc9600cfea0 0x7fc960161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56160/0 (socket says 172.21.15.141:56160) 2024-01-16T15:10:00.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.123+0000 7fc965400640 1 -- 172.21.15.141:0/2128018337 learned_addr learned my addr 172.21.15.141:0/2128018337 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:00.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.124+0000 7fc964a00640 1 -- 172.21.15.141:0/2128018337 >> v1:172.21.15.188:6789/0 conn(0x7fc9600cf9f0 legacy=0x7fc960161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:00.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.124+0000 7fc964a00640 1 -- 172.21.15.141:0/2128018337 >> v1:172.21.15.141:6789/0 conn(0x7fc9600cfea0 legacy=0x7fc960161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.124+0000 7fc964a00640 1 -- 172.21.15.141:0/2128018337 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600cf080 con 0x7fc960160920 2024-01-16T15:10:00.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.124+0000 7fc95fe00640 1 -- 172.21.15.141:0/2128018337 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632291785 0 0) 0x7fc9600d0a10 con 0x7fc9600cfea0 2024-01-16T15:10:00.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.125+0000 7fc964a00640 1 --2- 172.21.15.141:0/2128018337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc960160920 0x7fc9600b3f30 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fc9500097a0 tx=0x7fc95002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f011f4a69565b404 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:00.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc95fe00640 1 -- 172.21.15.141:0/2128018337 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950031830 con 0x7fc960160920 2024-01-16T15:10:00.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc95fe00640 1 -- 172.21.15.141:0/2128018337 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc950031990 con 0x7fc960160920 2024-01-16T15:10:00.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc95fe00640 1 -- 172.21.15.141:0/2128018337 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950031d20 con 0x7fc960160920 2024-01-16T15:10:00.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc965e00640 1 -- 172.21.15.141:0/2128018337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc960160920 msgr2=0x7fc9600b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc965e00640 1 --2- 172.21.15.141:0/2128018337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc960160920 0x7fc9600b3f30 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fc9500097a0 tx=0x7fc95002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc965e00640 1 -- 172.21.15.141:0/2128018337 shutdown_connections 2024-01-16T15:10:00.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc965e00640 1 --2- 172.21.15.141:0/2128018337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc960160920 0x7fc9600b3f30 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.126+0000 7fc965e00640 1 -- 172.21.15.141:0/2128018337 >> 172.21.15.141:0/2128018337 conn(0x7fc9600d3090 msgr2=0x7fc9600d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:00.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.127+0000 7fc965e00640 1 -- 172.21.15.141:0/2128018337 shutdown_connections 2024-01-16T15:10:00.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.127+0000 7fc965e00640 1 -- 172.21.15.141:0/2128018337 wait complete. 2024-01-16T15:10:00.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.128+0000 7fc965e00640 1 Processor -- start 2024-01-16T15:10:00.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.128+0000 7fc965e00640 1 -- start start 2024-01-16T15:10:00.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.129+0000 7fc965e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9600cf9f0 0x7fc960151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.129+0000 7fc965e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9600cfea0 0x7fc9600ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.129+0000 7fc965e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc960160920 0x7fc9600b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.129+0000 7fc965e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600b6c60 con 0x7fc9600cf9f0 2024-01-16T15:10:00.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.129+0000 7fc965e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600b7ae0 con 0x7fc960160920 2024-01-16T15:10:00.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.129+0000 7fc965e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600b6980 con 0x7fc9600cfea0 2024-01-16T15:10:00.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9600cf9f0 0x7fc960151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9600cf9f0 0x7fc960151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51182/0 (socket says 172.21.15.141:51182) 2024-01-16T15:10:00.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 -- 172.21.15.141:0/3456956001 learned_addr learned my addr 172.21.15.141:0/3456956001 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:00.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc957e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9600cfea0 0x7fc9600ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc965400640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc960160920 0x7fc9600b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 -- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9600cfea0 msgr2=0x7fc9600ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9600cfea0 0x7fc9600ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 -- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc960160920 msgr2=0x7fc9600b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc960160920 0x7fc9600b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.130+0000 7fc964a00640 1 -- 172.21.15.141:0/3456956001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc950009400 con 0x7fc9600cf9f0 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.131+0000 7fc964a00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9600cf9f0 0x7fc960151710 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fc950000c00 tx=0x7fc9500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.131+0000 7fc95d600640 1 -- 172.21.15.141:0/3456956001 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950004230 con 0x7fc9600cf9f0 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.131+0000 7fc95d600640 1 -- 172.21.15.141:0/3456956001 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc950004390 con 0x7fc9600cf9f0 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.131+0000 7fc95d600640 1 -- 172.21.15.141:0/3456956001 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95003a940 con 0x7fc9600cf9f0 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.131+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9600b7490 con 0x7fc9600cf9f0 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.131+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9600ca6a0 con 0x7fc9600cf9f0 2024-01-16T15:10:00.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.132+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9600b72f0 con 0x7fc9600cf9f0 2024-01-16T15:10:00.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.134+0000 7fc95d600640 1 -- 172.21.15.141:0/3456956001 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc95003aaa0 con 0x7fc9600cf9f0 2024-01-16T15:10:00.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.135+0000 7fc95d600640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc93806bfb0 0x7fc93806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.135+0000 7fc95d600640 1 -- 172.21.15.141:0/3456956001 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc9500aaf50 con 0x7fc9600cf9f0 2024-01-16T15:10:00.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.136+0000 7fc957e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc93806bfb0 0x7fc93806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.137+0000 7fc957e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc93806bfb0 0x7fc93806e470 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc9600b4bb0 tx=0x7fc948009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:00.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.140+0000 7fc95d600640 1 -- 172.21.15.141:0/3456956001 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc950077810 con 0x7fc9600cf9f0 2024-01-16T15:10:00.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.340+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc9600b72f0 con 0x7fc9600cf9f0 2024-01-16T15:10:00.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.893+0000 7fc95d600640 1 -- 172.21.15.141:0/3456956001 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc9600b72f0 con 0x7fc9600cf9f0 2024-01-16T15:10:00.871 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.896+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc93806bfb0 msgr2=0x7fc93806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.896+0000 7fc965e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc93806bfb0 0x7fc93806e470 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc9600b4bb0 tx=0x7fc948009060 comp rx=0 tx=0).stop 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9600cf9f0 msgr2=0x7fc960151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9600cf9f0 0x7fc960151710 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fc950000c00 tx=0x7fc9500046c0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 shutdown_connections 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc93806bfb0 0x7fc93806e470 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc960160920 0x7fc9600b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9600cfea0 0x7fc9600ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 --2- 172.21.15.141:0/3456956001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9600cf9f0 0x7fc960151710 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 >> 172.21.15.141:0/3456956001 conn(0x7fc9600d3090 msgr2=0x7fc96006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:00.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 shutdown_connections 2024-01-16T15:10:00.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:00.897+0000 7fc965e00640 1 -- 172.21.15.141:0/3456956001 wait complete. 2024-01-16T15:10:00.907 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:10:01.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.114+0000 7f1e95600640 1 Processor -- start 2024-01-16T15:10:01.092 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.114+0000 7f1e95600640 1 -- start start 2024-01-16T15:10:01.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e95600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1e9014b2c0 0x7f1e900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.094 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e95600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1e90071070 0x7f1e900cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e95600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e900cfbf0 con 0x7f1e900cf7d0 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e95600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9014b6a0 con 0x7f1e9014b2c0 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e95600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e90071690 con 0x7f1e90071070 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e8ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1e9014b2c0 0x7f1e900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e8ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1e9014b2c0 0x7f1e900cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51190/0 (socket says 172.21.15.141:51190) 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e8ea00640 1 -- 172.21.15.141:0/2648009785 learned_addr learned my addr 172.21.15.141:0/2648009785 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.116+0000 7f1e8e000640 1 --1- 172.21.15.141:0/2648009785 >> v1:172.21.15.141:6789/0 conn(0x7f1e900cf7d0 0x7f1e90070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56166/0 (socket says 172.21.15.141:56166) 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.117+0000 7f1e8f400640 1 --2- 172.21.15.141:0/2648009785 >> v2:172.21.15.188:3300/0 conn(0x7f1e90071070 0x7f1e900cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.117+0000 7f1e8ea00640 1 -- 172.21.15.141:0/2648009785 >> v2:172.21.15.188:3300/0 conn(0x7f1e90071070 msgr2=0x7f1e900cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.117+0000 7f1e8ea00640 1 --2- 172.21.15.141:0/2648009785 >> v2:172.21.15.188:3300/0 conn(0x7f1e90071070 0x7f1e900cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.117+0000 7f1e8ea00640 1 -- 172.21.15.141:0/2648009785 >> v1:172.21.15.141:6789/0 conn(0x7f1e900cf7d0 legacy=0x7f1e90070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.117+0000 7f1e8ea00640 1 -- 172.21.15.141:0/2648009785 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e90071970 con 0x7f1e9014b2c0 2024-01-16T15:10:01.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.117+0000 7f1e8d600640 1 -- 172.21.15.141:0/2648009785 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097158274 0 0) 0x7f1e900cfbf0 con 0x7f1e900cf7d0 2024-01-16T15:10:01.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.118+0000 7f1e8ea00640 1 --2- 172.21.15.141:0/2648009785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e9014b2c0 0x7f1e900cf290 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f1e90165330 tx=0x7f1e8002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a4206a14094c1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:01.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e8d600640 1 -- 172.21.15.141:0/2648009785 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8002cb20 con 0x7f1e9014b2c0 2024-01-16T15:10:01.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e8d600640 1 -- 172.21.15.141:0/2648009785 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8002cc80 con 0x7f1e9014b2c0 2024-01-16T15:10:01.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e8d600640 1 -- 172.21.15.141:0/2648009785 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800343f0 con 0x7f1e9014b2c0 2024-01-16T15:10:01.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 -- 172.21.15.141:0/2648009785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e9014b2c0 msgr2=0x7f1e900cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 --2- 172.21.15.141:0/2648009785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e9014b2c0 0x7f1e900cf290 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f1e90165330 tx=0x7f1e8002bf80 comp rx=0 tx=0).stop 2024-01-16T15:10:01.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 -- 172.21.15.141:0/2648009785 shutdown_connections 2024-01-16T15:10:01.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 --2- 172.21.15.141:0/2648009785 >> v2:172.21.15.188:3300/0 conn(0x7f1e90071070 0x7f1e900cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 --2- 172.21.15.141:0/2648009785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e9014b2c0 0x7f1e900cf290 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 -- 172.21.15.141:0/2648009785 >> 172.21.15.141:0/2648009785 conn(0x7f1e9006a800 msgr2=0x7f1e9006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:01.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 -- 172.21.15.141:0/2648009785 shutdown_connections 2024-01-16T15:10:01.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.119+0000 7f1e95600640 1 -- 172.21.15.141:0/2648009785 wait complete. 2024-01-16T15:10:01.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.121+0000 7f1e95600640 1 Processor -- start 2024-01-16T15:10:01.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.121+0000 7f1e95600640 1 -- start start 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.122+0000 7f1e95600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e90071070 0x7f1e90142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.122+0000 7f1e95600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e900cf7d0 0x7f1e901437d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.122+0000 7f1e95600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e9014b2c0 0x7f1e90144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.122+0000 7f1e95600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90144770 con 0x7f1e900cf7d0 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.122+0000 7f1e95600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90142b50 con 0x7f1e90071070 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.122+0000 7f1e95600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90146960 con 0x7f1e9014b2c0 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e900cf7d0 0x7f1e901437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e900cf7d0 0x7f1e901437d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51198/0 (socket says 172.21.15.141:51198) 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 -- 172.21.15.141:0/1462093866 learned_addr learned my addr 172.21.15.141:0/1462093866 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8ea00640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e90071070 0x7f1e90142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8f400640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e9014b2c0 0x7f1e90144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 -- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e9014b2c0 msgr2=0x7f1e90144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e9014b2c0 0x7f1e90144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 -- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e90071070 msgr2=0x7f1e90142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e90071070 0x7f1e90142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.123+0000 7f1e8e000640 1 -- 172.21.15.141:0/1462093866 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e80009b30 con 0x7f1e900cf7d0 2024-01-16T15:10:01.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.124+0000 7f1e8e000640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e900cf7d0 0x7f1e901437d0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800c4d0 tx=0x7f1e7800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:01.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.124+0000 7f1e86a00640 1 -- 172.21.15.141:0/1462093866 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e780074e0 con 0x7f1e900cf7d0 2024-01-16T15:10:01.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.124+0000 7f1e86a00640 1 -- 172.21.15.141:0/1462093866 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e78007660 con 0x7f1e900cf7d0 2024-01-16T15:10:01.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.124+0000 7f1e86a00640 1 -- 172.21.15.141:0/1462093866 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e78004a20 con 0x7f1e900cf7d0 2024-01-16T15:10:01.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.124+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e901441b0 con 0x7f1e900cf7d0 2024-01-16T15:10:01.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.124+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9015baa0 con 0x7f1e900cf7d0 2024-01-16T15:10:01.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.125+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e900675d0 con 0x7f1e900cf7d0 2024-01-16T15:10:01.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.126+0000 7f1e86a00640 1 -- 172.21.15.141:0/1462093866 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1e78004b80 con 0x7f1e900cf7d0 2024-01-16T15:10:01.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.127+0000 7f1e86a00640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e6806c000 0x7f1e6806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.127+0000 7f1e86a00640 1 -- 172.21.15.141:0/1462093866 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1e78087cb0 con 0x7f1e900cf7d0 2024-01-16T15:10:01.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.128+0000 7f1e8ea00640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e6806c000 0x7f1e6806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.129+0000 7f1e8ea00640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e6806c000 0x7f1e6806e4c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1e80035000 tx=0x7f1e80030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:01.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.132+0000 7f1e86a00640 1 -- 172.21.15.141:0/1462093866 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e78054570 con 0x7f1e900cf7d0 2024-01-16T15:10:01.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.360+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1e90067b60 con 0x7f1e900cf7d0 2024-01-16T15:10:01.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.914+0000 7f1e86a00640 1 -- 172.21.15.141:0/1462093866 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v51) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1e78018070 con 0x7f1e900cf7d0 2024-01-16T15:10:01.893 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 12 and data pool 13 2024-01-16T15:10:01.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e6806c000 msgr2=0x7f1e6806e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e6806c000 0x7f1e6806e4c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1e80035000 tx=0x7f1e80030000 comp rx=0 tx=0).stop 2024-01-16T15:10:01.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e900cf7d0 msgr2=0x7f1e901437d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e900cf7d0 0x7f1e901437d0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800c4d0 tx=0x7f1e7800c990 comp rx=0 tx=0).stop 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 shutdown_connections 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1e6806c000 0x7f1e6806e4c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1e9014b2c0 0x7f1e90144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1e900cf7d0 0x7f1e901437d0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 --2- 172.21.15.141:0/1462093866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e90071070 0x7f1e90142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 >> 172.21.15.141:0/1462093866 conn(0x7f1e9006a800 msgr2=0x7f1e9006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 shutdown_connections 2024-01-16T15:10:01.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:01.918+0000 7f1e95600640 1 -- 172.21.15.141:0/1462093866 wait complete. 2024-01-16T15:10:01.933 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:10:02.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.134+0000 7f594ba00640 1 Processor -- start 2024-01-16T15:10:02.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.134+0000 7f594ba00640 1 -- start start 2024-01-16T15:10:02.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f594ba00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f594409a170 0x7f59440b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f594ba00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594406cea0 con 0x7f59440cd240 2024-01-16T15:10:02.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f594ba00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594406cbc0 con 0x7f594406c6b0 2024-01-16T15:10:02.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f594ba00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594406f350 con 0x7f594409a170 2024-01-16T15:10:02.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f5949000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f59440cd240 0x7f5944099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56174/0 (socket says 172.21.15.141:56174) 2024-01-16T15:10:02.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f5949000640 1 -- 172.21.15.141:0/1199768041 learned_addr learned my addr 172.21.15.141:0/1199768041 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f5943e00640 1 --2- 172.21.15.141:0/1199768041 >> v2:172.21.15.188:3300/0 conn(0x7f594409a170 0x7f59440b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.136+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37691587 0 0) 0x7f594406cea0 con 0x7f59440cd240 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5924003520 con 0x7f59440cd240 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2282843045 0 0) 0x7f5924003520 con 0x7f59440cd240 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 >> v2:172.21.15.188:3300/0 conn(0x7f594409a170 msgr2=0x7f59440b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 --2- 172.21.15.141:0/1199768041 >> v2:172.21.15.188:3300/0 conn(0x7f594409a170 0x7f59440b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 >> v1:172.21.15.188:6789/0 conn(0x7f594406c6b0 legacy=0x7f59440cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594406f630 con 0x7f59440cd240 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f5934001670 con 0x7f59440cd240 2024-01-16T15:10:02.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 >> v1:172.21.15.141:6789/0 conn(0x7f59440cd240 legacy=0x7f5944099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5924003970 0x7f5924003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59240047c0 0x7f5924004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f592400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594406f350 con 0x7f5924005180 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592400c020 con 0x7f5924003970 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592400c1e0 con 0x7f59240047c0 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.137+0000 7f5949a00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f592400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5949000640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5924003970 0x7f5924003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5943e00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59240047c0 0x7f5924004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5949a00640 1 -- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59240047c0 msgr2=0x7f5924004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5949a00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59240047c0 0x7f5924004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5949a00640 1 -- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5924003970 msgr2=0x7f5924003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5949a00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5924003970 0x7f5924003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5949a00640 1 -- 172.21.15.141:0/1199768041 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924003520 con 0x7f5924005180 2024-01-16T15:10:02.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.138+0000 7f5943e00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59240047c0 0x7f5924004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.139+0000 7f5949a00640 1 -- 172.21.15.141:0/1199768041 reap_dead start 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.139+0000 7f5949a00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f592400b980 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f593c001e50 tx=0x7f59240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388f1ef09634c055 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.139+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c003b70 con 0x7f5924005180 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.139+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c003d10 con 0x7f5924005180 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.139+0000 7f5943400640 1 -- 172.21.15.141:0/1199768041 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c005310 con 0x7f5924005180 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.140+0000 7f594ba00640 1 -- 172.21.15.141:0/1199768041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 msgr2=0x7f592400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.140+0000 7f594ba00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f592400b980 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f593c001e50 tx=0x7f59240057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.140+0000 7f594ba00640 1 -- 172.21.15.141:0/1199768041 shutdown_connections 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.140+0000 7f594ba00640 1 --2- 172.21.15.141:0/1199768041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f592400b980 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.140+0000 7f594ba00640 1 -- 172.21.15.141:0/1199768041 >> 172.21.15.141:0/1199768041 conn(0x7f59440b2d00 msgr2=0x7f59440b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.140+0000 7f594ba00640 1 -- 172.21.15.141:0/1199768041 shutdown_connections 2024-01-16T15:10:02.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.140+0000 7f594ba00640 1 -- 172.21.15.141:0/1199768041 wait complete. 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.142+0000 7f594ba00640 1 Processor -- start 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.142+0000 7f594ba00640 1 -- start start 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f594ba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f59440cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f594ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59441f2e00 0x7f59440c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f594ba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59440cfaa0 0x7f59440cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f594ba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440b4060 con 0x7f5924005180 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f594ba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440d0460 con 0x7f59440cfaa0 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f594ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59441f3630 con 0x7f59441f2e00 2024-01-16T15:10:02.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f59440cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f59440cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51214/0 (socket says 172.21.15.141:51214) 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 -- 172.21.15.141:0/861317527 learned_addr learned my addr 172.21.15.141:0/861317527 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949a00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59440cfaa0 0x7f59440cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5943e00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59441f2e00 0x7f59440c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 -- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59441f2e00 msgr2=0x7f59440c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59441f2e00 0x7f59440c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 -- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59440cfaa0 msgr2=0x7f59440cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59440cfaa0 0x7f59440cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5949000640 1 -- 172.21.15.141:0/861317527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59441f3910 con 0x7f5924005180 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.143+0000 7f5943e00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59441f2e00 0x7f59440c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:02.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.144+0000 7f5949000640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f59440cb1d0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f593400e6c0 tx=0x7f593400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.144+0000 7f5940c00640 1 -- 172.21.15.141:0/861317527 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400c6c0 con 0x7f5924005180 2024-01-16T15:10:02.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.144+0000 7f5940c00640 1 -- 172.21.15.141:0/861317527 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593400c840 con 0x7f5924005180 2024-01-16T15:10:02.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.144+0000 7f5940c00640 1 -- 172.21.15.141:0/861317527 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400cbf0 con 0x7f5924005180 2024-01-16T15:10:02.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.144+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59441f3bf0 con 0x7f5924005180 2024-01-16T15:10:02.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.144+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59440c95f0 con 0x7f5924005180 2024-01-16T15:10:02.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.145+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59440b44b0 con 0x7f5924005180 2024-01-16T15:10:02.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.146+0000 7f5940c00640 1 -- 172.21.15.141:0/861317527 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f593400cd50 con 0x7f5924005180 2024-01-16T15:10:02.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.147+0000 7f5940c00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591c06bfe0 0x7f591c06e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.147+0000 7f5940c00640 1 -- 172.21.15.141:0/861317527 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59340887b0 con 0x7f5924005180 2024-01-16T15:10:02.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.148+0000 7f5943e00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591c06bfe0 0x7f591c06e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.149+0000 7f5943e00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591c06bfe0 0x7f591c06e4a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f59441f74b0 tx=0x7f592c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.151+0000 7f5940c00640 1 -- 172.21.15.141:0/861317527 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59440b44b0 con 0x7f5924005180 2024-01-16T15:10:02.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.330+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59440b44b0 con 0x7f5924005180 2024-01-16T15:10:02.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.332+0000 7f5940c00640 1 -- 172.21.15.141:0/861317527 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f59440b44b0 con 0x7f5924005180 2024-01-16T15:10:02.310 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:02.310 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":88,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:10:01.362570+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:09:59.187465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:10:00.391379+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.334+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591c06bfe0 msgr2=0x7f591c06e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.334+0000 7f594ba00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591c06bfe0 0x7f591c06e4a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f59441f74b0 tx=0x7f592c008040 comp rx=0 tx=0).stop 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.334+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 msgr2=0x7f59440cb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.334+0000 7f594ba00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f59440cb1d0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f593400e6c0 tx=0x7f593400eb80 comp rx=0 tx=0).stop 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 shutdown_connections 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f591c06bfe0 0x7f591c06e4a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59440cfaa0 0x7f59440cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59441f2e00 0x7f59440c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 --2- 172.21.15.141:0/861317527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5924005180 0x7f59440cb1d0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 >> 172.21.15.141:0/861317527 conn(0x7f59440b2d00 msgr2=0x7f59440b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:02.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 shutdown_connections 2024-01-16T15:10:02.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.335+0000 7f594ba00640 1 -- 172.21.15.141:0/861317527 wait complete. 2024-01-16T15:10:02.330 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:10:02.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.559+0000 7f2de4600640 1 Processor -- start 2024-01-16T15:10:02.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.559+0000 7f2de4600640 1 -- start start 2024-01-16T15:10:02.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de4600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2ddc13df80 0x7f2ddc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de4600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ddc0cf9a0 0x7f2ddc0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de4600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc06c470 con 0x7f2ddc13e360 2024-01-16T15:10:02.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de4600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc06c750 con 0x7f2ddc0cf9a0 2024-01-16T15:10:02.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de4600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc06ca30 con 0x7f2ddc13df80 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de2600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2ddc0cf9a0 0x7f2ddc0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de1200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2ddc13e360 0x7f2ddc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:56180/0 (socket says 172.21.15.141:56180) 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.561+0000 7f2de1200640 1 -- 172.21.15.141:0/2613840045 learned_addr learned my addr 172.21.15.141:0/2613840045 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.562+0000 7f2de1c00640 1 --2- 172.21.15.141:0/2613840045 >> v2:172.21.15.188:3300/0 conn(0x7f2ddc13df80 0x7f2ddc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.562+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2613840045 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616753165 0 0) 0x7f2ddc06c470 con 0x7f2ddc13e360 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.562+0000 7f2de2600640 1 -- 172.21.15.141:0/2613840045 >> v2:172.21.15.188:3300/0 conn(0x7f2ddc13df80 msgr2=0x7f2ddc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.562+0000 7f2de2600640 1 --2- 172.21.15.141:0/2613840045 >> v2:172.21.15.188:3300/0 conn(0x7f2ddc13df80 0x7f2ddc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.562+0000 7f2de2600640 1 -- 172.21.15.141:0/2613840045 >> v1:172.21.15.141:6789/0 conn(0x7f2ddc13e360 legacy=0x7f2ddc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.562+0000 7f2de2600640 1 -- 172.21.15.141:0/2613840045 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc06cd10 con 0x7f2ddc0cf9a0 2024-01-16T15:10:02.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.562+0000 7f2de1c00640 1 --2- 172.21.15.141:0/2613840045 >> v2:172.21.15.188:3300/0 conn(0x7f2ddc13df80 0x7f2ddc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:02.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.563+0000 7f2de2600640 1 --2- 172.21.15.141:0/2613840045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc0cf9a0 0x7f2ddc0cd840 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc072770 tx=0x7f2dd402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbc77dfb5d30753 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.563+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2613840045 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd402cc70 con 0x7f2ddc0cf9a0 2024-01-16T15:10:02.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.563+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2613840045 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd402cdd0 con 0x7f2ddc0cf9a0 2024-01-16T15:10:02.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.563+0000 7f2ddbe00640 1 -- 172.21.15.141:0/2613840045 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4034660 con 0x7f2ddc0cf9a0 2024-01-16T15:10:02.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 -- 172.21.15.141:0/2613840045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc0cf9a0 msgr2=0x7f2ddc0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 --2- 172.21.15.141:0/2613840045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc0cf9a0 0x7f2ddc0cd840 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc072770 tx=0x7f2dd402bec0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 -- 172.21.15.141:0/2613840045 shutdown_connections 2024-01-16T15:10:02.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 --2- 172.21.15.141:0/2613840045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc0cf9a0 0x7f2ddc0cd840 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 --2- 172.21.15.141:0/2613840045 >> v2:172.21.15.188:3300/0 conn(0x7f2ddc13df80 0x7f2ddc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 -- 172.21.15.141:0/2613840045 >> 172.21.15.141:0/2613840045 conn(0x7f2ddc06a800 msgr2=0x7f2ddc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:02.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 -- 172.21.15.141:0/2613840045 shutdown_connections 2024-01-16T15:10:02.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.564+0000 7f2de4600640 1 -- 172.21.15.141:0/2613840045 wait complete. 2024-01-16T15:10:02.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.566+0000 7f2de4600640 1 Processor -- start 2024-01-16T15:10:02.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.566+0000 7f2de4600640 1 -- start start 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de4600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc0cf9a0 0x7f2ddc136ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de4600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13df80 0x7f2ddc14dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de4600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de4600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc1370e0 con 0x7f2ddc13e360 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de4600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc1503a0 con 0x7f2ddc13df80 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de4600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc150680 con 0x7f2ddc0cf9a0 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de2600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de2600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51240/0 (socket says 172.21.15.141:51240) 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.567+0000 7f2de2600640 1 -- 172.21.15.141:0/1696577887 learned_addr learned my addr 172.21.15.141:0/1696577887 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de1c00640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc0cf9a0 0x7f2ddc136ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de1200640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13df80 0x7f2ddc14dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de2600640 1 -- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc0cf9a0 msgr2=0x7f2ddc136ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de2600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc0cf9a0 0x7f2ddc136ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de2600640 1 -- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13df80 msgr2=0x7f2ddc14dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de2600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13df80 0x7f2ddc14dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de2600640 1 -- 172.21.15.141:0/1696577887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4009a70 con 0x7f2ddc13e360 2024-01-16T15:10:02.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2de2600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc151440 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4037040 tx=0x7f2dd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2dd9600640 1 -- 172.21.15.141:0/1696577887 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4040650 con 0x7f2ddc13e360 2024-01-16T15:10:02.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2dd9600640 1 -- 172.21.15.141:0/1696577887 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd40407b0 con 0x7f2ddc13e360 2024-01-16T15:10:02.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.568+0000 7f2dd9600640 1 -- 172.21.15.141:0/1696577887 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4040b20 con 0x7f2ddc13e360 2024-01-16T15:10:02.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.569+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc150960 con 0x7f2ddc13e360 2024-01-16T15:10:02.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.569+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc150f20 con 0x7f2ddc13e360 2024-01-16T15:10:02.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.570+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc0675d0 con 0x7f2ddc13e360 2024-01-16T15:10:02.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.571+0000 7f2dd9600640 1 -- 172.21.15.141:0/1696577887 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f2dd4040c80 con 0x7f2ddc13e360 2024-01-16T15:10:02.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.572+0000 7f2dd9600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.572+0000 7f2dd9600640 1 -- 172.21.15.141:0/1696577887 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dd40ab760 con 0x7f2ddc13e360 2024-01-16T15:10:02.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.572+0000 7f2de1c00640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.573+0000 7f2de1c00640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00a970 tx=0x7f2dcc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.576+0000 7f2dd9600640 1 -- 172.21.15.141:0/1696577887 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dd4077d90 con 0x7f2ddc13e360 2024-01-16T15:10:02.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.768+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ddc06df00 con 0x7f2ddc13e360 2024-01-16T15:10:02.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.770+0000 7f2dd9600640 1 -- 172.21.15.141:0/1696577887 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f2dd407b3b0 con 0x7f2ddc13e360 2024-01-16T15:10:02.748 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:02.748 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":88,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:10:01.362570+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:09:59.187465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:10:00.391379+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 msgr2=0x7f2db406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00a970 tx=0x7f2dcc008040 comp rx=0 tx=0).stop 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 msgr2=0x7f2ddc151440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc151440 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4037040 tx=0x7f2dd4004a40 comp rx=0 tx=0).stop 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 shutdown_connections 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2db406bfb0 0x7f2db406e470 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ddc13e360 0x7f2ddc151440 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ddc13df80 0x7f2ddc14dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 --2- 172.21.15.141:0/1696577887 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ddc0cf9a0 0x7f2ddc136ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 >> 172.21.15.141:0/1696577887 conn(0x7f2ddc06a800 msgr2=0x7f2ddc0aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:02.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 shutdown_connections 2024-01-16T15:10:02.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:02.773+0000 7f2de4600640 1 -- 172.21.15.141:0/1696577887 wait complete. 2024-01-16T15:10:02.772 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:10:02.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.002+0000 7fa93ea00640 1 Processor -- start 2024-01-16T15:10:02.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.003+0000 7fa93ea00640 1 -- start start 2024-01-16T15:10:02.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ea00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa93813e360 0x7fa9380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa93815f9c0 0x7fa9380cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ea00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93815c5f0 con 0x7fa93813df80 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ea00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93806c470 con 0x7fa93815f9c0 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ea00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93806c750 con 0x7fa93813e360 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ca00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa93815f9c0 0x7fa9380cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ca00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa93815f9c0 0x7fa9380cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46096/0 (socket says 172.21.15.141:46096) 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.004+0000 7fa93ca00640 1 -- 172.21.15.141:0/3224290409 learned_addr learned my addr 172.21.15.141:0/3224290409 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.005+0000 7fa92f200640 1 --2- 172.21.15.141:0/3224290409 >> v2:172.21.15.188:3300/0 conn(0x7fa93813e360 0x7fa9380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.005+0000 7fa93ca00640 1 -- 172.21.15.141:0/3224290409 >> v2:172.21.15.188:3300/0 conn(0x7fa93813e360 msgr2=0x7fa9380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.005+0000 7fa93ca00640 1 --2- 172.21.15.141:0/3224290409 >> v2:172.21.15.188:3300/0 conn(0x7fa93813e360 0x7fa9380cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.005+0000 7fa93ca00640 1 -- 172.21.15.141:0/3224290409 >> v1:172.21.15.188:6789/0 conn(0x7fa93813df80 legacy=0x7fa93815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:02.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.005+0000 7fa93ca00640 1 -- 172.21.15.141:0/3224290409 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93806ca30 con 0x7fa93815f9c0 2024-01-16T15:10:02.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.006+0000 7fa93ca00640 1 --2- 172.21.15.141:0/3224290409 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93815f9c0 0x7fa9380cf7d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa9380ccae0 tx=0x7fa93002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fc9b26966d4692 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.006+0000 7fa937400640 1 -- 172.21.15.141:0/3224290409 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93002ca80 con 0x7fa93815f9c0 2024-01-16T15:10:02.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.006+0000 7fa937400640 1 -- 172.21.15.141:0/3224290409 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93002cbe0 con 0x7fa93815f9c0 2024-01-16T15:10:02.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.006+0000 7fa937400640 1 -- 172.21.15.141:0/3224290409 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9300343f0 con 0x7fa93815f9c0 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 -- 172.21.15.141:0/3224290409 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93815f9c0 msgr2=0x7fa9380cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 --2- 172.21.15.141:0/3224290409 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93815f9c0 0x7fa9380cf7d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa9380ccae0 tx=0x7fa93002bee0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 -- 172.21.15.141:0/3224290409 shutdown_connections 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 --2- 172.21.15.141:0/3224290409 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93815f9c0 0x7fa9380cf7d0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 --2- 172.21.15.141:0/3224290409 >> v2:172.21.15.188:3300/0 conn(0x7fa93813e360 0x7fa9380cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 -- 172.21.15.141:0/3224290409 >> 172.21.15.141:0/3224290409 conn(0x7fa93806a800 msgr2=0x7fa93806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 -- 172.21.15.141:0/3224290409 shutdown_connections 2024-01-16T15:10:02.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.007+0000 7fa93ea00640 1 -- 172.21.15.141:0/3224290409 wait complete. 2024-01-16T15:10:02.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.009+0000 7fa93ea00640 1 Processor -- start 2024-01-16T15:10:02.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.009+0000 7fa93ea00640 1 -- start start 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa93ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93813df80 0x7fa9380b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa93ea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa93813e360 0x7fa93814aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa93ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93815f9c0 0x7fa93814b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa93ea00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93814c440 con 0x7fa93813df80 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa93ea00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93814b830 con 0x7fa93815f9c0 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa93ea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93814bb10 con 0x7fa93813e360 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa937e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93813df80 0x7fa9380b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa937e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93813df80 0x7fa9380b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46102/0 (socket says 172.21.15.141:46102) 2024-01-16T15:10:02.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa937e00640 1 -- 172.21.15.141:0/2835492685 learned_addr learned my addr 172.21.15.141:0/2835492685 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:02.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa937e00640 1 -- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa93813e360 msgr2=0x7fa93814aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa937e00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa93813e360 0x7fa93814aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa937e00640 1 -- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93815f9c0 msgr2=0x7fa93814b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa93ca00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93815f9c0 0x7fa93814b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.010+0000 7fa937e00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93815f9c0 0x7fa93814b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.011+0000 7fa937e00640 1 -- 172.21.15.141:0/2835492685 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930009b30 con 0x7fa93813df80 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.011+0000 7fa937e00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93813df80 0x7fa9380b3500 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa92800b3e0 tx=0x7fa92800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.011+0000 7fa934c00640 1 -- 172.21.15.141:0/2835492685 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928011830 con 0x7fa93813df80 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.011+0000 7fa934c00640 1 -- 172.21.15.141:0/2835492685 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9280119b0 con 0x7fa93813df80 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.011+0000 7fa934c00640 1 -- 172.21.15.141:0/2835492685 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928011d60 con 0x7fa93813df80 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.011+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93814bdf0 con 0x7fa93813df80 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.011+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93814b540 con 0x7fa93813df80 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.012+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa938155600 con 0x7fa93813df80 2024-01-16T15:10:02.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.013+0000 7fa934c00640 1 -- 172.21.15.141:0/2835492685 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fa92801a400 con 0x7fa93813df80 2024-01-16T15:10:02.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.014+0000 7fa934c00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa91006bfb0 0x7fa91006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.014+0000 7fa934c00640 1 -- 172.21.15.141:0/2835492685 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9280887d0 con 0x7fa93813df80 2024-01-16T15:10:02.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.014+0000 7fa92f200640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa91006bfb0 0x7fa91006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.016+0000 7fa92f200640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa91006bfb0 0x7fa91006e470 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa9381f44d0 tx=0x7fa920008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.018+0000 7fa934c00640 1 -- 172.21.15.141:0/2835492685 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa928055050 con 0x7fa93813df80 2024-01-16T15:10:03.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.229+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa9380675d0 con 0x7fa93813df80 2024-01-16T15:10:03.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.952+0000 7fa934c00640 1 -- 172.21.15.141:0/2835492685 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v53) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa9280150b0 con 0x7fa93813df80 2024-01-16T15:10:03.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa91006bfb0 msgr2=0x7fa91006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa91006bfb0 0x7fa91006e470 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa9381f44d0 tx=0x7fa920008040 comp rx=0 tx=0).stop 2024-01-16T15:10:03.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93813df80 msgr2=0x7fa9380b3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93813df80 0x7fa9380b3500 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa92800b3e0 tx=0x7fa92800b8a0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 shutdown_connections 2024-01-16T15:10:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa91006bfb0 0x7fa91006e470 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93815f9c0 0x7fa93814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa93813e360 0x7fa93814aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 --2- 172.21.15.141:0/2835492685 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93813df80 0x7fa9380b3500 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 >> 172.21.15.141:0/2835492685 conn(0x7fa93806a800 msgr2=0x7fa93815cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 shutdown_connections 2024-01-16T15:10:03.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:03.956+0000 7fa93ea00640 1 -- 172.21.15.141:0/2835492685 wait complete. 2024-01-16T15:10:03.953 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:10:03.954 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:10:04.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.190+0000 7f3934000640 1 Processor -- start 2024-01-16T15:10:04.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.190+0000 7f3934000640 1 -- start start 2024-01-16T15:10:04.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.191+0000 7f3934000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f392c0c9870 0x7f392c0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:04.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.191+0000 7f3934000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0d0560 0x7f392c161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:04.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.191+0000 7f3934000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c0cf4c0 con 0x7f392c0d2260 2024-01-16T15:10:04.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.191+0000 7f3934000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c0d2ef0 con 0x7f392c0c9870 2024-01-16T15:10:04.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.191+0000 7f3934000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0c9c50 con 0x7f392c0d0560 2024-01-16T15:10:04.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3931600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f392c0c9870 0x7f392c0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:04.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3931600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f392c0c9870 0x7f392c0ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46108/0 (socket says 172.21.15.141:46108) 2024-01-16T15:10:04.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3931600640 1 -- 172.21.15.141:0/2562904201 learned_addr learned my addr 172.21.15.141:0/2562904201 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3930c00640 1 --2- 172.21.15.141:0/2562904201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0d0560 0x7f392c161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3931600640 1 -- 172.21.15.141:0/2562904201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0d0560 msgr2=0x7f392c161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3931600640 1 --2- 172.21.15.141:0/2562904201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0d0560 0x7f392c161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3931600640 1 -- 172.21.15.141:0/2562904201 >> v1:172.21.15.141:6789/0 conn(0x7f392c0d2260 legacy=0x7f392c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3931600640 1 -- 172.21.15.141:0/2562904201 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c0cf7a0 con 0x7f392c0c9870 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f392be00640 1 -- 172.21.15.141:0/2562904201 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644563691 0 0) 0x7f392c0cf4c0 con 0x7f392c0d2260 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.192+0000 7f3930c00640 1 --2- 172.21.15.141:0/2562904201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0d0560 0x7f392c161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.193+0000 7f3931600640 1 --2- 172.21.15.141:0/2562904201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0c9870 0x7f392c0ceb50 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f391c009e50 tx=0x7f391c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a489883370766f5a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:04.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.193+0000 7f392be00640 1 -- 172.21.15.141:0/2562904201 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c02cb00 con 0x7f392c0c9870 2024-01-16T15:10:04.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.193+0000 7f392be00640 1 -- 172.21.15.141:0/2562904201 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c02cc60 con 0x7f392c0c9870 2024-01-16T15:10:04.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.193+0000 7f392be00640 1 -- 172.21.15.141:0/2562904201 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c0343f0 con 0x7f392c0c9870 2024-01-16T15:10:04.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 -- 172.21.15.141:0/2562904201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0c9870 msgr2=0x7f392c0ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:04.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 --2- 172.21.15.141:0/2562904201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0c9870 0x7f392c0ceb50 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f391c009e50 tx=0x7f391c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 -- 172.21.15.141:0/2562904201 shutdown_connections 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 --2- 172.21.15.141:0/2562904201 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0d0560 0x7f392c161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 --2- 172.21.15.141:0/2562904201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0c9870 0x7f392c0ceb50 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 -- 172.21.15.141:0/2562904201 >> 172.21.15.141:0/2562904201 conn(0x7f392c09c560 msgr2=0x7f392c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 -- 172.21.15.141:0/2562904201 shutdown_connections 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.194+0000 7f3934000640 1 -- 172.21.15.141:0/2562904201 wait complete. 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.196+0000 7f3934000640 1 Processor -- start 2024-01-16T15:10:04.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.196+0000 7f3934000640 1 -- start start 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3934000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0c9870 0x7f392c1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3934000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0d0560 0x7f392c0cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3934000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f392c0d2260 0x7f392c0b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3934000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c155de0 con 0x7f392c0d0560 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3934000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0ca3c0 con 0x7f392c0d2260 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3934000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0ca6a0 con 0x7f392c0c9870 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3930c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0d0560 0x7f392c0cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3930c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0d0560 0x7f392c0cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46114/0 (socket says 172.21.15.141:46114) 2024-01-16T15:10:04.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.197+0000 7f3930c00640 1 -- 172.21.15.141:0/3283906039 learned_addr learned my addr 172.21.15.141:0/3283906039 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:04.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3931600640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0c9870 0x7f392c1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:04.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3930c00640 1 -- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0c9870 msgr2=0x7f392c1558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3930c00640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0c9870 0x7f392c1558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3930c00640 1 -- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f392c0d2260 msgr2=0x7f392c0b6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3930c00640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f392c0d2260 0x7f392c0b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3930c00640 1 -- 172.21.15.141:0/3283906039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c009b30 con 0x7f392c0d0560 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3930c00640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0d0560 0x7f392c0cc8e0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f391400c7f0 tx=0x7f391400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3929600640 1 -- 172.21.15.141:0/3283906039 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914007820 con 0x7f392c0d0560 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3929600640 1 -- 172.21.15.141:0/3283906039 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3914007980 con 0x7f392c0d0560 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c0ca980 con 0x7f392c0d0560 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3929600640 1 -- 172.21.15.141:0/3283906039 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914007d30 con 0x7f392c0d0560 2024-01-16T15:10:04.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.198+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c0caf40 con 0x7f392c0d0560 2024-01-16T15:10:04.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.199+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c0675d0 con 0x7f392c0d0560 2024-01-16T15:10:04.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.200+0000 7f3929600640 1 -- 172.21.15.141:0/3283906039 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f39140113f0 con 0x7f392c0d0560 2024-01-16T15:10:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.201+0000 7f3929600640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f390406bfb0 0x7f390406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.201+0000 7f3929600640 1 -- 172.21.15.141:0/3283906039 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3914088990 con 0x7f392c0d0560 2024-01-16T15:10:04.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.202+0000 7f3931600640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f390406bfb0 0x7f390406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:04.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.203+0000 7f3931600640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f390406bfb0 0x7f390406e470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f391c035000 tx=0x7f391c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:04.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.206+0000 7f3929600640 1 -- 172.21.15.141:0/3283906039 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3914055250 con 0x7f392c0d0560 2024-01-16T15:10:04.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.434+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f392c0d0a20 con 0x7f392c0d0560 2024-01-16T15:10:04.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.963+0000 7f3929600640 1 -- 172.21.15.141:0/3283906039 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v54) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3914058870 con 0x7f392c0d0560 2024-01-16T15:10:04.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f390406bfb0 msgr2=0x7f390406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:04.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f390406bfb0 0x7f390406e470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f391c035000 tx=0x7f391c030000 comp rx=0 tx=0).stop 2024-01-16T15:10:04.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0d0560 msgr2=0x7f392c0cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0d0560 0x7f392c0cc8e0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f391400c7f0 tx=0x7f391400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 shutdown_connections 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f390406bfb0 0x7f390406e470 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f392c0d2260 0x7f392c0b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f392c0d0560 0x7f392c0cc8e0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 --2- 172.21.15.141:0/3283906039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f392c0c9870 0x7f392c1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.967+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 >> 172.21.15.141:0/3283906039 conn(0x7f392c09c560 msgr2=0x7f392c0c6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.968+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 shutdown_connections 2024-01-16T15:10:04.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:04.968+0000 7f3934000640 1 -- 172.21.15.141:0/3283906039 wait complete. 2024-01-16T15:10:04.968 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:10:04.969 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:10:05.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.196+0000 7f8039800640 1 Processor -- start 2024-01-16T15:10:05.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.196+0000 7f8039800640 1 -- start start 2024-01-16T15:10:05.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8039800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80340cf9a0 0x7f803415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8039800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80340b1ee0 con 0x7f8034143ae0 2024-01-16T15:10:05.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8039800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80340b2780 con 0x7f803415e1e0 2024-01-16T15:10:05.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8039800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803415e990 con 0x7f80340cf9a0 2024-01-16T15:10:05.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8032000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80340cf9a0 0x7f803415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8032a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8034143ae0 0x7f80340cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:45258/0 (socket says 172.21.15.141:45258) 2024-01-16T15:10:05.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8032a00640 1 -- 172.21.15.141:0/1651530340 learned_addr learned my addr 172.21.15.141:0/1651530340 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:05.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8032000640 1 -- 172.21.15.141:0/1651530340 >> v1:172.21.15.188:6789/0 conn(0x7f803415e1e0 legacy=0x7f80340702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:05.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8032000640 1 -- 172.21.15.141:0/1651530340 >> v1:172.21.15.141:6789/0 conn(0x7f8034143ae0 legacy=0x7f80340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8032000640 1 -- 172.21.15.141:0/1651530340 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80340b2130 con 0x7f80340cf9a0 2024-01-16T15:10:05.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.198+0000 7f8031600640 1 -- 172.21.15.141:0/1651530340 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503278313 0 0) 0x7f80340b1ee0 con 0x7f8034143ae0 2024-01-16T15:10:05.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.199+0000 7f8032000640 1 --2- 172.21.15.141:0/1651530340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80340cf9a0 0x7f803415dca0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f80240097a0 tx=0x7f802402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6645d4994c3629e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:05.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.200+0000 7f8031600640 1 -- 172.21.15.141:0/1651530340 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024031830 con 0x7f80340cf9a0 2024-01-16T15:10:05.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.200+0000 7f8031600640 1 -- 172.21.15.141:0/1651530340 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8024031990 con 0x7f80340cf9a0 2024-01-16T15:10:05.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.200+0000 7f8031600640 1 -- 172.21.15.141:0/1651530340 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024031d20 con 0x7f80340cf9a0 2024-01-16T15:10:05.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.200+0000 7f8039800640 1 -- 172.21.15.141:0/1651530340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80340cf9a0 msgr2=0x7f803415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.200+0000 7f8039800640 1 --2- 172.21.15.141:0/1651530340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80340cf9a0 0x7f803415dca0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f80240097a0 tx=0x7f802402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.200+0000 7f8039800640 1 -- 172.21.15.141:0/1651530340 shutdown_connections 2024-01-16T15:10:05.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.200+0000 7f8039800640 1 --2- 172.21.15.141:0/1651530340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80340cf9a0 0x7f803415dca0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.201+0000 7f8039800640 1 -- 172.21.15.141:0/1651530340 >> 172.21.15.141:0/1651530340 conn(0x7f803406a800 msgr2=0x7f803406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:05.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.201+0000 7f8039800640 1 -- 172.21.15.141:0/1651530340 shutdown_connections 2024-01-16T15:10:05.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.201+0000 7f8039800640 1 -- 172.21.15.141:0/1651530340 wait complete. 2024-01-16T15:10:05.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.202+0000 7f8039800640 1 Processor -- start 2024-01-16T15:10:05.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.202+0000 7f8039800640 1 -- start start 2024-01-16T15:10:05.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.203+0000 7f8039800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340cf9a0 0x7f8034138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.203+0000 7f8039800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8034143ae0 0x7f80340adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.203+0000 7f8039800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803415e1e0 0x7f80340b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.203+0000 7f8039800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8034138c50 con 0x7f803415e1e0 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.203+0000 7f8039800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80340ac8d0 con 0x7f80340cf9a0 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.203+0000 7f8039800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80340acbb0 con 0x7f8034143ae0 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803415e1e0 0x7f80340b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803415e1e0 0x7f80340b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46142/0 (socket says 172.21.15.141:46142) 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 -- 172.21.15.141:0/867061139 learned_addr learned my addr 172.21.15.141:0/867061139 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:05.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8032a00640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340cf9a0 0x7f8034138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8032000640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8034143ae0 0x7f80340adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 -- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8034143ae0 msgr2=0x7f80340adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8034143ae0 0x7f80340adc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 -- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340cf9a0 msgr2=0x7f8034138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340cf9a0 0x7f8034138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.204+0000 7f8033400640 1 -- 172.21.15.141:0/867061139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8024009400 con 0x7f803415e1e0 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.205+0000 7f8033400640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803415e1e0 0x7f80340b0bc0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b970 tx=0x7f802c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.205+0000 7f802aa00640 1 -- 172.21.15.141:0/867061139 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00cb40 con 0x7f803415e1e0 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.205+0000 7f802aa00640 1 -- 172.21.15.141:0/867061139 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c00cca0 con 0x7f803415e1e0 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.205+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80340ace90 con 0x7f803415e1e0 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.205+0000 7f802aa00640 1 -- 172.21.15.141:0/867061139 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c014680 con 0x7f803415e1e0 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.205+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80340ad450 con 0x7f803415e1e0 2024-01-16T15:10:05.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.206+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80340675d0 con 0x7f803415e1e0 2024-01-16T15:10:05.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.207+0000 7f802aa00640 1 -- 172.21.15.141:0/867061139 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f802c01d3b0 con 0x7f803415e1e0 2024-01-16T15:10:05.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.208+0000 7f802aa00640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06bfb0 0x7f800c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.208+0000 7f802aa00640 1 -- 172.21.15.141:0/867061139 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f802c0883e0 con 0x7f803415e1e0 2024-01-16T15:10:05.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.209+0000 7f8032a00640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06bfb0 0x7f800c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.210+0000 7f8032a00640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06bfb0 0x7f800c06e470 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f801c000f60 tx=0x7f801c007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:05.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.212+0000 7f802aa00640 1 -- 172.21.15.141:0/867061139 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f802c054d00 con 0x7f803415e1e0 2024-01-16T15:10:05.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.437+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80340707f0 con 0x7f803415e1e0 2024-01-16T15:10:05.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.438+0000 7f802aa00640 1 -- 172.21.15.141:0/867061139 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f802c058320 con 0x7f803415e1e0 2024-01-16T15:10:05.416 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:05.416 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/1211961850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1211961850},{"type":"v1","addr":"172.21.15.141:6835","nonce":1211961850}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3677420872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3677420872},{"type":"v1","addr":"172.21.15.188:6835","nonce":3677420872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/4249270856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":4249270856},{"type":"v1","addr":"172.21.15.188:6837","nonce":4249270856}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:10:01.362921+0000","modified":"2024-01-16T15:10:04.436791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8148":{"gid":8148,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6834/3399417259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3399417259},{"type":"v1","addr":"172.21.15.141:6834","nonce":3399417259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:10:05.416 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 54 2024-01-16T15:10:05.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06bfb0 msgr2=0x7f800c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06bfb0 0x7f800c06e470 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f801c000f60 tx=0x7f801c007b50 comp rx=0 tx=0).stop 2024-01-16T15:10:05.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803415e1e0 msgr2=0x7f80340b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803415e1e0 0x7f80340b0bc0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b970 tx=0x7f802c00be30 comp rx=0 tx=0).stop 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 shutdown_connections 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f800c06bfb0 0x7f800c06e470 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f803415e1e0 0x7f80340b0bc0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8034143ae0 0x7f80340adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 --2- 172.21.15.141:0/867061139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340cf9a0 0x7f8034138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 >> 172.21.15.141:0/867061139 conn(0x7f803406a800 msgr2=0x7f80340ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 shutdown_connections 2024-01-16T15:10:05.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.440+0000 7f8039800640 1 -- 172.21.15.141:0/867061139 wait complete. 2024-01-16T15:10:05.437 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:10:05.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.668+0000 7f984cc00640 1 Processor -- start 2024-01-16T15:10:05.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.668+0000 7f984cc00640 1 -- start start 2024-01-16T15:10:05.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f984cc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f984813e360 0x7f98480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f984cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984815f9c0 0x7f98480cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f984cc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984815c5f0 con 0x7f984813df80 2024-01-16T15:10:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f984cc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984806c470 con 0x7f984813e360 2024-01-16T15:10:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f984cc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984806c750 con 0x7f984815f9c0 2024-01-16T15:10:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f9845e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f984813df80 0x7f984815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:45270/0 (socket says 172.21.15.141:45270) 2024-01-16T15:10:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f9845e00640 1 -- 172.21.15.141:0/1371813741 learned_addr learned my addr 172.21.15.141:0/1371813741 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:05.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f983fe00640 1 --2- 172.21.15.141:0/1371813741 >> v2:172.21.15.141:3300/0 conn(0x7f984813e360 0x7f98480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f9846800640 1 --2- 172.21.15.141:0/1371813741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984815f9c0 0x7f98480cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f9845400640 1 -- 172.21.15.141:0/1371813741 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938506647 0 0) 0x7f984815c5f0 con 0x7f984813df80 2024-01-16T15:10:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f9845400640 1 -- 172.21.15.141:0/1371813741 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9828003520 con 0x7f984813df80 2024-01-16T15:10:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f983fe00640 1 -- 172.21.15.141:0/1371813741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984815f9c0 msgr2=0x7f98480cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:05.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f983fe00640 1 --2- 172.21.15.141:0/1371813741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984815f9c0 0x7f98480cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f983fe00640 1 -- 172.21.15.141:0/1371813741 >> v1:172.21.15.141:6789/0 conn(0x7f984813df80 legacy=0x7f984815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.670+0000 7f983fe00640 1 -- 172.21.15.141:0/1371813741 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984806ca30 con 0x7f984813e360 2024-01-16T15:10:05.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.671+0000 7f983fe00640 1 --2- 172.21.15.141:0/1371813741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813e360 0x7f98480cf290 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f98300097a0 tx=0x7f983002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06aa741526e43b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.671+0000 7f9845400640 1 -- 172.21.15.141:0/1371813741 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830031830 con 0x7f984813e360 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.671+0000 7f9845400640 1 -- 172.21.15.141:0/1371813741 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9830031990 con 0x7f984813e360 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.671+0000 7f9845400640 1 -- 172.21.15.141:0/1371813741 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830031d20 con 0x7f984813e360 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 -- 172.21.15.141:0/1371813741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813e360 msgr2=0x7f98480cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 --2- 172.21.15.141:0/1371813741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813e360 0x7f98480cf290 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f98300097a0 tx=0x7f983002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 -- 172.21.15.141:0/1371813741 shutdown_connections 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 --2- 172.21.15.141:0/1371813741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984815f9c0 0x7f98480cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 --2- 172.21.15.141:0/1371813741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813e360 0x7f98480cf290 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 -- 172.21.15.141:0/1371813741 >> 172.21.15.141:0/1371813741 conn(0x7f984806a800 msgr2=0x7f984806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 -- 172.21.15.141:0/1371813741 shutdown_connections 2024-01-16T15:10:05.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.672+0000 7f984cc00640 1 -- 172.21.15.141:0/1371813741 wait complete. 2024-01-16T15:10:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.674+0000 7f984cc00640 1 Processor -- start 2024-01-16T15:10:05.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.674+0000 7f984cc00640 1 -- start start 2024-01-16T15:10:05.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f984cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813df80 0x7f984814d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f984cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984813e360 0x7f984814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f984cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984815f9c0 0x7f984814b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f984cc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984814b0e0 con 0x7f984813df80 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f984cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9848135db0 con 0x7f984815f9c0 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f984cc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984815ae40 con 0x7f984813e360 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813df80 0x7f984814d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813df80 0x7f984814d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46156/0 (socket says 172.21.15.141:46156) 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 -- 172.21.15.141:0/534010939 learned_addr learned my addr 172.21.15.141:0/534010939 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 -- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984813e360 msgr2=0x7f984814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984813e360 0x7f984814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 -- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984815f9c0 msgr2=0x7f984814b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984815f9c0 0x7f984814b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.675+0000 7f9845e00640 1 -- 172.21.15.141:0/534010939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830009400 con 0x7f984813df80 2024-01-16T15:10:05.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.676+0000 7f9845e00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813df80 0x7f984814d7e0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f983800c430 tx=0x7f983800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:05.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.676+0000 7f983e000640 1 -- 172.21.15.141:0/534010939 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838005ce0 con 0x7f984813df80 2024-01-16T15:10:05.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.676+0000 7f983e000640 1 -- 172.21.15.141:0/534010939 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9838005e40 con 0x7f984813df80 2024-01-16T15:10:05.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.676+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9848070b70 con 0x7f984813df80 2024-01-16T15:10:05.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.676+0000 7f983e000640 1 -- 172.21.15.141:0/534010939 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98380072d0 con 0x7f984813df80 2024-01-16T15:10:05.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.676+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9848071130 con 0x7f984813df80 2024-01-16T15:10:05.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.677+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98480675d0 con 0x7f984813df80 2024-01-16T15:10:05.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.678+0000 7f983e000640 1 -- 172.21.15.141:0/534010939 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f98380074f0 con 0x7f984813df80 2024-01-16T15:10:05.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.679+0000 7f983e000640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f982006bfb0 0x7f982006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:05.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.679+0000 7f983e000640 1 -- 172.21.15.141:0/534010939 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9838087ab0 con 0x7f984813df80 2024-01-16T15:10:05.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.679+0000 7f983fe00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f982006bfb0 0x7f982006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:05.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.680+0000 7f983fe00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f982006bfb0 0x7f982006e470 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9830036000 tx=0x7f9830030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:05.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.683+0000 7f983e000640 1 -- 172.21.15.141:0/534010939 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9838054480 con 0x7f984813df80 2024-01-16T15:10:05.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.874+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f984814dd20 con 0x7f984813df80 2024-01-16T15:10:05.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.877+0000 7f983e000640 1 -- 172.21.15.141:0/534010939 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16608 (secure 0 0 0) 0x7f9838057aa0 con 0x7f984813df80 2024-01-16T15:10:05.854 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:05.854 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":89,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:10:02.932379+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:09:59.187465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:10:00.391379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:10:05.856 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f982006bfb0 msgr2=0x7f982006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f982006bfb0 0x7f982006e470 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9830036000 tx=0x7f9830030000 comp rx=0 tx=0).stop 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813df80 msgr2=0x7f984814d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813df80 0x7f984814d7e0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f983800c430 tx=0x7f983800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 shutdown_connections 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f982006bfb0 0x7f982006e470 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984815f9c0 0x7f984814b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f984813e360 0x7f984814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 --2- 172.21.15.141:0/534010939 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f984813df80 0x7f984814d7e0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:05.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.879+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 >> 172.21.15.141:0/534010939 conn(0x7f984806a800 msgr2=0x7f984815cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:05.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.880+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 shutdown_connections 2024-01-16T15:10:05.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:05.880+0000 7f984cc00640 1 -- 172.21.15.141:0/534010939 wait complete. 2024-01-16T15:10:05.864 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:05.876 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:10:05.876 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:10:06.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.147+0000 7f00ac000640 1 Processor -- start 2024-01-16T15:10:06.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.147+0000 7f00ac000640 1 -- start start 2024-01-16T15:10:06.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00ac000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f00a4143ae0 0x7f00a40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00ac000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 0x7f00a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00ac000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a40cfc30 con 0x7f00a40cf7d0 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00ac000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a40b1ee0 con 0x7f00a4143ae0 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00ac000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40b2780 con 0x7f00a415e3b0 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00a9600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f00a4143ae0 0x7f00a40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00a9600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f00a4143ae0 0x7f00a40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46170/0 (socket says 172.21.15.141:46170) 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.149+0000 7f00a9600640 1 -- 172.21.15.141:0/2286341382 learned_addr learned my addr 172.21.15.141:0/2286341382 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.150+0000 7f00aa000640 1 --2- 172.21.15.141:0/2286341382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 0x7f00a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.150+0000 7f00a9600640 1 -- 172.21.15.141:0/2286341382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 msgr2=0x7f00a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.150+0000 7f00a9600640 1 --2- 172.21.15.141:0/2286341382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 0x7f00a40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.150+0000 7f00a9600640 1 -- 172.21.15.141:0/2286341382 >> v1:172.21.15.188:6789/0 conn(0x7f00a40cf7d0 legacy=0x7f00a415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:06.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.150+0000 7f00a9600640 1 -- 172.21.15.141:0/2286341382 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a415e990 con 0x7f00a4143ae0 2024-01-16T15:10:06.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.151+0000 7f00a9600640 1 --2- 172.21.15.141:0/2286341382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a4143ae0 0x7f00a40cf290 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f009400bb10 tx=0x7f009402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337de13ccca60ac0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:06.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.151+0000 7f00a3e00640 1 -- 172.21.15.141:0/2286341382 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009402ee70 con 0x7f00a4143ae0 2024-01-16T15:10:06.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.151+0000 7f00a3e00640 1 -- 172.21.15.141:0/2286341382 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094036440 con 0x7f00a4143ae0 2024-01-16T15:10:06.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.151+0000 7f00a3e00640 1 -- 172.21.15.141:0/2286341382 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940367d0 con 0x7f00a4143ae0 2024-01-16T15:10:06.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 -- 172.21.15.141:0/2286341382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a4143ae0 msgr2=0x7f00a40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 --2- 172.21.15.141:0/2286341382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a4143ae0 0x7f00a40cf290 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f009400bb10 tx=0x7f009402e0c0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 -- 172.21.15.141:0/2286341382 shutdown_connections 2024-01-16T15:10:06.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 --2- 172.21.15.141:0/2286341382 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 0x7f00a40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 --2- 172.21.15.141:0/2286341382 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a4143ae0 0x7f00a40cf290 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 -- 172.21.15.141:0/2286341382 >> 172.21.15.141:0/2286341382 conn(0x7f00a406a800 msgr2=0x7f00a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:06.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 -- 172.21.15.141:0/2286341382 shutdown_connections 2024-01-16T15:10:06.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.152+0000 7f00ac000640 1 -- 172.21.15.141:0/2286341382 wait complete. 2024-01-16T15:10:06.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.154+0000 7f00ac000640 1 Processor -- start 2024-01-16T15:10:06.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.154+0000 7f00ac000640 1 -- start start 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00ac000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a40cf7d0 0x7f00a4159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00ac000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00a4143ae0 0x7f00a41e4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00ac000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 0x7f00a40adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00ac000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40ace40 con 0x7f00a40cf7d0 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00ac000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40ad120 con 0x7f00a4143ae0 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00ac000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40ad400 con 0x7f00a415e3b0 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00a9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a40cf7d0 0x7f00a4159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00a9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a40cf7d0 0x7f00a4159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46176/0 (socket says 172.21.15.141:46176) 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.155+0000 7f00a9600640 1 -- 172.21.15.141:0/2705545309 learned_addr learned my addr 172.21.15.141:0/2705545309 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a9600640 1 -- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 msgr2=0x7f00a40adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a9600640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 0x7f00a40adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a9600640 1 -- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00a4143ae0 msgr2=0x7f00a41e4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a9600640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00a4143ae0 0x7f00a41e4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a9600640 1 -- 172.21.15.141:0/2705545309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009400b770 con 0x7f00a40cf7d0 2024-01-16T15:10:06.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a9600640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a40cf7d0 0x7f00a4159a90 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f009400b2c0 tx=0x7f00940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:06.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a1600640 1 -- 172.21.15.141:0/2705545309 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940348d0 con 0x7f00a40cf7d0 2024-01-16T15:10:06.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a1600640 1 -- 172.21.15.141:0/2705545309 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094034a50 con 0x7f00a40cf7d0 2024-01-16T15:10:06.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00a1600640 1 -- 172.21.15.141:0/2705545309 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094034e00 con 0x7f00a40cf7d0 2024-01-16T15:10:06.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.156+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a40ad6e0 con 0x7f00a40cf7d0 2024-01-16T15:10:06.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.157+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a4142450 con 0x7f00a40cf7d0 2024-01-16T15:10:06.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.158+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a40adc00 con 0x7f00a40cf7d0 2024-01-16T15:10:06.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.159+0000 7f00a1600640 1 -- 172.21.15.141:0/2705545309 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f00940403e0 con 0x7f00a40cf7d0 2024-01-16T15:10:06.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.160+0000 7f00a1600640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f007c06bfb0 0x7f007c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.160+0000 7f00a1600640 1 -- 172.21.15.141:0/2705545309 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00940aabb0 con 0x7f00a40cf7d0 2024-01-16T15:10:06.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.160+0000 7f00a8c00640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f007c06bfb0 0x7f007c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.161+0000 7f00a8c00640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f007c06bfb0 0x7f007c06e470 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f00a4141c20 tx=0x7f008c0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:06.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.165+0000 7f00a1600640 1 -- 172.21.15.141:0/2705545309 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00a40adc00 con 0x7f00a40cf7d0 2024-01-16T15:10:06.270 INFO:tasks.ceph.mds.c.smithi141.stderr:2024-01-16T15:10:06.294+0000 7fba30400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:10:06.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.350+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00a40b2410 con 0x7f007c06bfb0 2024-01-16T15:10:06.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.378+0000 7f00a1600640 1 -- 172.21.15.141:0/2705545309 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278098 (secure 0 0 0) 0x7f008c014400 con 0x7f007c06bfb0 2024-01-16T15:10:06.359 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:06.360 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:10:06.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.383+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f007c06bfb0 msgr2=0x7f007c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.383+0000 7f00ac000640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f007c06bfb0 0x7f007c06e470 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f00a4141c20 tx=0x7f008c0079b0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.383+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a40cf7d0 msgr2=0x7f00a4159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.383+0000 7f00ac000640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a40cf7d0 0x7f00a4159a90 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f009400b2c0 tx=0x7f00940046c0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 shutdown_connections 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f007c06bfb0 0x7f007c06e470 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00a415e3b0 0x7f00a40adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00a4143ae0 0x7f00a41e4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 --2- 172.21.15.141:0/2705545309 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00a40cf7d0 0x7f00a4159a90 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 >> 172.21.15.141:0/2705545309 conn(0x7f00a406a800 msgr2=0x7f00a415b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 shutdown_connections 2024-01-16T15:10:06.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.384+0000 7f00ac000640 1 -- 172.21.15.141:0/2705545309 wait complete. 2024-01-16T15:10:06.381 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-01-16T15:10:04.785191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591980,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":186,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":533356,"kb_used_data":9456,"kb_used_omap":12,"kb_used_meta":523763,"kb_avail":754441364,"statfs":{"total":773094113280,"available":772547956736,"internally_reserved":0,"allocated":9682944,"data_stored":4793887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":536333874},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.893727"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949242+0000","last_change":"2024-01-16T15:10:01.954843+0000","last_active":"2024-01-16T15:10:02.949242+0000","last_peered":"2024-01-16T15:10:02.949242+0000","last_clean":"2024-01-16T15:10:02.949242+0000","last_became_active":"2024-01-16T15:10:01.954556+0000","last_became_peered":"2024-01-16T15:10:01.954556+0000","last_unstale":"2024-01-16T15:10:02.949242+0000","last_undegraded":"2024-01-16T15:10:02.949242+0000","last_fullsized":"2024-01-16T15:10:02.949242+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:22.743543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950553+0000","last_change":"2024-01-16T15:10:01.948258+0000","last_active":"2024-01-16T15:10:02.950553+0000","last_peered":"2024-01-16T15:10:02.950553+0000","last_clean":"2024-01-16T15:10:02.950553+0000","last_became_active":"2024-01-16T15:10:01.947983+0000","last_became_peered":"2024-01-16T15:10:01.947983+0000","last_unstale":"2024-01-16T15:10:02.950553+0000","last_undegraded":"2024-01-16T15:10:02.950553+0000","last_fullsized":"2024-01-16T15:10:02.950553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:48.782361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948844+0000","last_change":"2024-01-16T15:10:01.946350+0000","last_active":"2024-01-16T15:10:02.948844+0000","last_peered":"2024-01-16T15:10:02.948844+0000","last_clean":"2024-01-16T15:10:02.948844+0000","last_became_active":"2024-01-16T15:10:01.946095+0000","last_became_peered":"2024-01-16T15:10:01.946095+0000","last_unstale":"2024-01-16T15:10:02.948844+0000","last_undegraded":"2024-01-16T15:10:02.948844+0000","last_fullsized":"2024-01-16T15:10:02.948844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.681946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949603+0000","last_change":"2024-01-16T15:10:01.956517+0000","last_active":"2024-01-16T15:10:02.949603+0000","last_peered":"2024-01-16T15:10:02.949603+0000","last_clean":"2024-01-16T15:10:02.949603+0000","last_became_active":"2024-01-16T15:10:01.954117+0000","last_became_peered":"2024-01-16T15:10:01.954117+0000","last_unstale":"2024-01-16T15:10:02.949603+0000","last_undegraded":"2024-01-16T15:10:02.949603+0000","last_fullsized":"2024-01-16T15:10:02.949603+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:32:00.871458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950520+0000","last_change":"2024-01-16T15:10:01.961525+0000","last_active":"2024-01-16T15:10:02.950520+0000","last_peered":"2024-01-16T15:10:02.950520+0000","last_clean":"2024-01-16T15:10:02.950520+0000","last_became_active":"2024-01-16T15:10:01.961148+0000","last_became_peered":"2024-01-16T15:10:01.961148+0000","last_unstale":"2024-01-16T15:10:02.950520+0000","last_undegraded":"2024-01-16T15:10:02.950520+0000","last_fullsized":"2024-01-16T15:10:02.950520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:13:16.862717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949525+0000","last_change":"2024-01-16T15:10:01.959882+0000","last_active":"2024-01-16T15:10:02.949525+0000","last_peered":"2024-01-16T15:10:02.949525+0000","last_clean":"2024-01-16T15:10:02.949525+0000","last_became_active":"2024-01-16T15:10:01.959588+0000","last_became_peered":"2024-01-16T15:10:01.959588+0000","last_unstale":"2024-01-16T15:10:02.949525+0000","last_undegraded":"2024-01-16T15:10:02.949525+0000","last_fullsized":"2024-01-16T15:10:02.949525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:05.704870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949363+0000","last_change":"2024-01-16T15:10:01.954713+0000","last_active":"2024-01-16T15:10:02.949363+0000","last_peered":"2024-01-16T15:10:02.949363+0000","last_clean":"2024-01-16T15:10:02.949363+0000","last_became_active":"2024-01-16T15:10:01.954427+0000","last_became_peered":"2024-01-16T15:10:01.954427+0000","last_unstale":"2024-01-16T15:10:02.949363+0000","last_undegraded":"2024-01-16T15:10:02.949363+0000","last_fullsized":"2024-01-16T15:10:02.949363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:42.665836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998616+0000","last_change":"2024-01-16T15:10:01.935256+0000","last_active":"2024-01-16T15:10:02.998616+0000","last_peered":"2024-01-16T15:10:02.998616+0000","last_clean":"2024-01-16T15:10:02.998616+0000","last_became_active":"2024-01-16T15:10:01.932457+0000","last_became_peered":"2024-01-16T15:10:01.932457+0000","last_unstale":"2024-01-16T15:10:02.998616+0000","last_undegraded":"2024-01-16T15:10:02.998616+0000","last_fullsized":"2024-01-16T15:10:02.998616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:52:44.520711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949557+0000","last_change":"2024-01-16T15:10:01.937972+0000","last_active":"2024-01-16T15:10:02.949557+0000","last_peered":"2024-01-16T15:10:02.949557+0000","last_clean":"2024-01-16T15:10:02.949557+0000","last_became_active":"2024-01-16T15:10:01.937629+0000","last_became_peered":"2024-01-16T15:10:01.937629+0000","last_unstale":"2024-01-16T15:10:02.949557+0000","last_undegraded":"2024-01-16T15:10:02.949557+0000","last_fullsized":"2024-01-16T15:10:02.949557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:23.435918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949005+0000","last_change":"2024-01-16T15:10:01.946491+0000","last_active":"2024-01-16T15:10:02.949005+0000","last_peered":"2024-01-16T15:10:02.949005+0000","last_clean":"2024-01-16T15:10:02.949005+0000","last_became_active":"2024-01-16T15:10:01.946236+0000","last_became_peered":"2024-01-16T15:10:01.946236+0000","last_unstale":"2024-01-16T15:10:02.949005+0000","last_undegraded":"2024-01-16T15:10:02.949005+0000","last_fullsized":"2024-01-16T15:10:02.949005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:41.287672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948562+0000","last_change":"2024-01-16T15:10:01.940418+0000","last_active":"2024-01-16T15:10:02.948562+0000","last_peered":"2024-01-16T15:10:02.948562+0000","last_clean":"2024-01-16T15:10:02.948562+0000","last_became_active":"2024-01-16T15:10:01.938464+0000","last_became_peered":"2024-01-16T15:10:01.938464+0000","last_unstale":"2024-01-16T15:10:02.948562+0000","last_undegraded":"2024-01-16T15:10:02.948562+0000","last_fullsized":"2024-01-16T15:10:02.948562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:21:40.890614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.901176+0000","last_change":"2024-01-16T15:09:59.901176+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.901176+0000","last_undegraded":"2024-01-16T15:09:59.901176+0000","last_fullsized":"2024-01-16T15:09:59.901176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998686+0000","last_change":"2024-01-16T15:10:00.922840+0000","last_active":"2024-01-16T15:10:02.998686+0000","last_peered":"2024-01-16T15:10:02.998686+0000","last_clean":"2024-01-16T15:10:02.998686+0000","last_became_active":"2024-01-16T15:10:00.922548+0000","last_became_peered":"2024-01-16T15:10:00.922548+0000","last_unstale":"2024-01-16T15:10:02.998686+0000","last_undegraded":"2024-01-16T15:10:02.998686+0000","last_fullsized":"2024-01-16T15:10:02.998686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:59.219101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998433+0000","last_change":"2024-01-16T15:10:00.924755+0000","last_active":"2024-01-16T15:10:02.998433+0000","last_peered":"2024-01-16T15:10:02.998433+0000","last_clean":"2024-01-16T15:10:02.998433+0000","last_became_active":"2024-01-16T15:10:00.924265+0000","last_became_peered":"2024-01-16T15:10:00.924265+0000","last_unstale":"2024-01-16T15:10:02.998433+0000","last_undegraded":"2024-01-16T15:10:02.998433+0000","last_fullsized":"2024-01-16T15:10:02.998433+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:10:30.167886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998221+0000","last_change":"2024-01-16T15:10:00.920218+0000","last_active":"2024-01-16T15:10:02.998221+0000","last_peered":"2024-01-16T15:10:02.998221+0000","last_clean":"2024-01-16T15:10:02.998221+0000","last_became_active":"2024-01-16T15:10:00.919605+0000","last_became_peered":"2024-01-16T15:10:00.919605+0000","last_unstale":"2024-01-16T15:10:02.998221+0000","last_undegraded":"2024-01-16T15:10:02.998221+0000","last_fullsized":"2024-01-16T15:10:02.998221+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:17:16.029879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.899808+0000","last_change":"2024-01-16T15:09:59.899808+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.899808+0000","last_undegraded":"2024-01-16T15:09:59.899808+0000","last_fullsized":"2024-01-16T15:09:59.899808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.900553+0000","last_change":"2024-01-16T15:09:59.900553+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.900553+0000","last_undegraded":"2024-01-16T15:09:59.900553+0000","last_fullsized":"2024-01-16T15:09:59.900553+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950840+0000","last_change":"2024-01-16T15:10:01.947485+0000","last_active":"2024-01-16T15:10:02.950840+0000","last_peered":"2024-01-16T15:10:02.950840+0000","last_clean":"2024-01-16T15:10:02.950840+0000","last_became_active":"2024-01-16T15:10:01.947158+0000","last_became_peered":"2024-01-16T15:10:01.947158+0000","last_unstale":"2024-01-16T15:10:02.950840+0000","last_undegraded":"2024-01-16T15:10:02.950840+0000","last_fullsized":"2024-01-16T15:10:02.950840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:41.946530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.900809+0000","last_change":"2024-01-16T15:09:59.900809+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.900809+0000","last_undegraded":"2024-01-16T15:09:59.900809+0000","last_fullsized":"2024-01-16T15:09:59.900809+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998338+0000","last_change":"2024-01-16T15:10:01.958890+0000","last_active":"2024-01-16T15:10:02.998338+0000","last_peered":"2024-01-16T15:10:02.998338+0000","last_clean":"2024-01-16T15:10:02.998338+0000","last_became_active":"2024-01-16T15:10:01.958565+0000","last_became_peered":"2024-01-16T15:10:01.958565+0000","last_unstale":"2024-01-16T15:10:02.998338+0000","last_undegraded":"2024-01-16T15:10:02.998338+0000","last_fullsized":"2024-01-16T15:10:02.998338+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:28.537570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.903289+0000","last_change":"2024-01-16T15:09:59.903289+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.903289+0000","last_undegraded":"2024-01-16T15:09:59.903289+0000","last_fullsized":"2024-01-16T15:09:59.903289+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949971+0000","last_change":"2024-01-16T15:10:00.927885+0000","last_active":"2024-01-16T15:10:02.949971+0000","last_peered":"2024-01-16T15:10:02.949971+0000","last_clean":"2024-01-16T15:10:02.949971+0000","last_became_active":"2024-01-16T15:10:00.927560+0000","last_became_peered":"2024-01-16T15:10:00.927560+0000","last_unstale":"2024-01-16T15:10:02.949971+0000","last_undegraded":"2024-01-16T15:10:02.949971+0000","last_fullsized":"2024-01-16T15:10:02.949971+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:44.854412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998339+0000","last_change":"2024-01-16T15:10:01.943627+0000","last_active":"2024-01-16T15:10:02.998339+0000","last_peered":"2024-01-16T15:10:02.998339+0000","last_clean":"2024-01-16T15:10:02.998339+0000","last_became_active":"2024-01-16T15:10:01.943360+0000","last_became_peered":"2024-01-16T15:10:01.943360+0000","last_unstale":"2024-01-16T15:10:02.998339+0000","last_undegraded":"2024-01-16T15:10:02.998339+0000","last_fullsized":"2024-01-16T15:10:02.998339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:01.056572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.899795+0000","last_change":"2024-01-16T15:09:59.899795+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.899795+0000","last_undegraded":"2024-01-16T15:09:59.899795+0000","last_fullsized":"2024-01-16T15:09:59.899795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949596+0000","last_change":"2024-01-16T15:10:00.919867+0000","last_active":"2024-01-16T15:10:02.949596+0000","last_peered":"2024-01-16T15:10:02.949596+0000","last_clean":"2024-01-16T15:10:02.949596+0000","last_became_active":"2024-01-16T15:10:00.919523+0000","last_became_peered":"2024-01-16T15:10:00.919523+0000","last_unstale":"2024-01-16T15:10:02.949596+0000","last_undegraded":"2024-01-16T15:10:02.949596+0000","last_fullsized":"2024-01-16T15:10:02.949596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:44.225208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998038+0000","last_change":"2024-01-16T15:10:00.922876+0000","last_active":"2024-01-16T15:10:02.998038+0000","last_peered":"2024-01-16T15:10:02.998038+0000","last_clean":"2024-01-16T15:10:02.998038+0000","last_became_active":"2024-01-16T15:10:00.922592+0000","last_became_peered":"2024-01-16T15:10:00.922592+0000","last_unstale":"2024-01-16T15:10:02.998038+0000","last_undegraded":"2024-01-16T15:10:02.998038+0000","last_fullsized":"2024-01-16T15:10:02.998038+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:06.653597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.899889+0000","last_change":"2024-01-16T15:09:59.899889+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.899889+0000","last_undegraded":"2024-01-16T15:09:59.899889+0000","last_fullsized":"2024-01-16T15:09:59.899889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998691+0000","last_change":"2024-01-16T15:10:00.921808+0000","last_active":"2024-01-16T15:10:02.998691+0000","last_peered":"2024-01-16T15:10:02.998691+0000","last_clean":"2024-01-16T15:10:02.998691+0000","last_became_active":"2024-01-16T15:10:00.921529+0000","last_became_peered":"2024-01-16T15:10:00.921529+0000","last_unstale":"2024-01-16T15:10:02.998691+0000","last_undegraded":"2024-01-16T15:10:02.998691+0000","last_fullsized":"2024-01-16T15:10:02.998691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:16.867223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998465+0000","last_change":"2024-01-16T15:10:00.918587+0000","last_active":"2024-01-16T15:10:02.998465+0000","last_peered":"2024-01-16T15:10:02.998465+0000","last_clean":"2024-01-16T15:10:02.998465+0000","last_became_active":"2024-01-16T15:10:00.918309+0000","last_became_peered":"2024-01-16T15:10:00.918309+0000","last_unstale":"2024-01-16T15:10:02.998465+0000","last_undegraded":"2024-01-16T15:10:02.998465+0000","last_fullsized":"2024-01-16T15:10:02.998465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:34.228418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949882+0000","last_change":"2024-01-16T15:10:01.952227+0000","last_active":"2024-01-16T15:10:02.949882+0000","last_peered":"2024-01-16T15:10:02.949882+0000","last_clean":"2024-01-16T15:10:02.949882+0000","last_became_active":"2024-01-16T15:10:01.951964+0000","last_became_peered":"2024-01-16T15:10:01.951964+0000","last_unstale":"2024-01-16T15:10:02.949882+0000","last_undegraded":"2024-01-16T15:10:02.949882+0000","last_fullsized":"2024-01-16T15:10:02.949882+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:00.914138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949504+0000","last_change":"2024-01-16T15:10:01.952242+0000","last_active":"2024-01-16T15:10:02.949504+0000","last_peered":"2024-01-16T15:10:02.949504+0000","last_clean":"2024-01-16T15:10:02.949504+0000","last_became_active":"2024-01-16T15:10:01.951960+0000","last_became_peered":"2024-01-16T15:10:01.951960+0000","last_unstale":"2024-01-16T15:10:02.949504+0000","last_undegraded":"2024-01-16T15:10:02.949504+0000","last_fullsized":"2024-01-16T15:10:02.949504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:20.817578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950095+0000","last_change":"2024-01-16T15:10:00.924186+0000","last_active":"2024-01-16T15:10:02.950095+0000","last_peered":"2024-01-16T15:10:02.950095+0000","last_clean":"2024-01-16T15:10:02.950095+0000","last_became_active":"2024-01-16T15:10:00.921679+0000","last_became_peered":"2024-01-16T15:10:00.921679+0000","last_unstale":"2024-01-16T15:10:02.950095+0000","last_undegraded":"2024-01-16T15:10:02.950095+0000","last_fullsized":"2024-01-16T15:10:02.950095+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:33.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948945+0000","last_change":"2024-01-16T15:10:01.945338+0000","last_active":"2024-01-16T15:10:02.948945+0000","last_peered":"2024-01-16T15:10:02.948945+0000","last_clean":"2024-01-16T15:10:02.948945+0000","last_became_active":"2024-01-16T15:10:01.943420+0000","last_became_peered":"2024-01-16T15:10:01.943420+0000","last_unstale":"2024-01-16T15:10:02.948945+0000","last_undegraded":"2024-01-16T15:10:02.948945+0000","last_fullsized":"2024-01-16T15:10:02.948945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:58:55.577512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949102+0000","last_change":"2024-01-16T15:10:00.931391+0000","last_active":"2024-01-16T15:10:02.949102+0000","last_peered":"2024-01-16T15:10:02.949102+0000","last_clean":"2024-01-16T15:10:02.949102+0000","last_became_active":"2024-01-16T15:10:00.931033+0000","last_became_peered":"2024-01-16T15:10:00.931033+0000","last_unstale":"2024-01-16T15:10:02.949102+0000","last_undegraded":"2024-01-16T15:10:02.949102+0000","last_fullsized":"2024-01-16T15:10:02.949102+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:17:21.907268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950378+0000","last_change":"2024-01-16T15:10:01.960465+0000","last_active":"2024-01-16T15:10:02.950378+0000","last_peered":"2024-01-16T15:10:02.950378+0000","last_clean":"2024-01-16T15:10:02.950378+0000","last_became_active":"2024-01-16T15:10:01.960151+0000","last_became_peered":"2024-01-16T15:10:01.960151+0000","last_unstale":"2024-01-16T15:10:02.950378+0000","last_undegraded":"2024-01-16T15:10:02.950378+0000","last_fullsized":"2024-01-16T15:10:02.950378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:18.268867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.899945+0000","last_change":"2024-01-16T15:09:59.899945+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.899945+0000","last_undegraded":"2024-01-16T15:09:59.899945+0000","last_fullsized":"2024-01-16T15:09:59.899945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.899947+0000","last_change":"2024-01-16T15:09:59.899947+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.899947+0000","last_undegraded":"2024-01-16T15:09:59.899947+0000","last_fullsized":"2024-01-16T15:09:59.899947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.900951+0000","last_change":"2024-01-16T15:09:59.900951+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.900951+0000","last_undegraded":"2024-01-16T15:09:59.900951+0000","last_fullsized":"2024-01-16T15:09:59.900951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.899910+0000","last_change":"2024-01-16T15:09:59.899910+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.899910+0000","last_undegraded":"2024-01-16T15:09:59.899910+0000","last_fullsized":"2024-01-16T15:09:59.899910+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950870+0000","last_change":"2024-01-16T15:10:01.951549+0000","last_active":"2024-01-16T15:10:02.950870+0000","last_peered":"2024-01-16T15:10:02.950870+0000","last_clean":"2024-01-16T15:10:02.950870+0000","last_became_active":"2024-01-16T15:10:01.951269+0000","last_became_peered":"2024-01-16T15:10:01.951269+0000","last_unstale":"2024-01-16T15:10:02.950870+0000","last_undegraded":"2024-01-16T15:10:02.950870+0000","last_fullsized":"2024-01-16T15:10:02.950870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:36.818022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998526+0000","last_change":"2024-01-16T15:10:00.916165+0000","last_active":"2024-01-16T15:10:02.998526+0000","last_peered":"2024-01-16T15:10:02.998526+0000","last_clean":"2024-01-16T15:10:02.998526+0000","last_became_active":"2024-01-16T15:10:00.915439+0000","last_became_peered":"2024-01-16T15:10:00.915439+0000","last_unstale":"2024-01-16T15:10:02.998526+0000","last_undegraded":"2024-01-16T15:10:02.998526+0000","last_fullsized":"2024-01-16T15:10:02.998526+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:35.968089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998389+0000","last_change":"2024-01-16T15:10:01.946503+0000","last_active":"2024-01-16T15:10:02.998389+0000","last_peered":"2024-01-16T15:10:02.998389+0000","last_clean":"2024-01-16T15:10:02.998389+0000","last_became_active":"2024-01-16T15:10:01.946233+0000","last_became_peered":"2024-01-16T15:10:01.946233+0000","last_unstale":"2024-01-16T15:10:02.998389+0000","last_undegraded":"2024-01-16T15:10:02.998389+0000","last_fullsized":"2024-01-16T15:10:02.998389+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:14.532732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.899570+0000","last_change":"2024-01-16T15:09:59.899570+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.899570+0000","last_undegraded":"2024-01-16T15:09:59.899570+0000","last_fullsized":"2024-01-16T15:09:59.899570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:09:59.873546+0000","last_change":"2024-01-16T15:09:59.873546+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.873546+0000","last_undegraded":"2024-01-16T15:09:59.873546+0000","last_fullsized":"2024-01-16T15:09:59.873546+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:10:00.887744+0000","last_change":"2024-01-16T15:10:00.887744+0000","last_active":"2024-01-16T15:10:00.887744+0000","last_peered":"2024-01-16T15:10:00.887744+0000","last_clean":"2024-01-16T15:10:00.887744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:10:00.887744+0000","last_undegraded":"2024-01-16T15:10:00.887744+0000","last_fullsized":"2024-01-16T15:10:00.887744+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-01-16T15:09:59.902685+0000","last_change":"2024-01-16T15:09:59.902685+0000","last_active":"2024-01-16T15:09:59.873546+0000","last_peered":"2024-01-16T15:09:59.873546+0000","last_clean":"2024-01-16T15:09:59.873546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:09:59.902685+0000","last_undegraded":"2024-01-16T15:09:59.902685+0000","last_fullsized":"2024-01-16T15:09:59.902685+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":229,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:09:56.867627+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:09:56.867627+0000","last_peered":"2024-01-16T15:09:56.867627+0000","last_clean":"2024-01-16T15:09:56.867627+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:09:56.867627+0000","last_undegraded":"2024-01-16T15:09:56.867627+0000","last_fullsized":"2024-01-16T15:09:56.867627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":40,"acting":40,"num_store_stats":3},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":48,"acting":48,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68816,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303024,"statfs":{"total":96636764160,"available":96566296576,"internally_reserved":0,"allocated":1634304,"data_stored":1032293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.6439999999999999,"15min":1.452},"min":{"1min":0.67900000000000005,"5min":0.37,"15min":0.37},"max":{"1min":4.0069999999999997,"5min":19.603000000000002,"15min":19.603000000000002},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.597,"15min":1.365},"min":{"1min":0.77600000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.7889999999999997,"5min":24.015000000000001,"15min":24.015000000000001},"last":1.052}]},{"osd":1,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.5980000000000001,"15min":1.4179999999999999},"min":{"1min":0.59699999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.6070000000000002,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":2.016,"5min":1.841,"15min":1.4470000000000001},"min":{"1min":0.65900000000000003,"5min":0.443,"15min":0.443},"max":{"1min":14.592000000000001,"5min":24.07,"15min":24.07},"last":1.169}]},{"osd":2,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.617,"15min":1.367},"min":{"1min":0.73199999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":4.9000000000000004,"5min":24.108000000000001,"15min":24.108000000000001},"last":1.365},{"interface":"front","average":{"1min":2.3239999999999998,"5min":1.698,"15min":1.52},"min":{"1min":0.67600000000000005,"5min":0.37,"15min":0.37},"max":{"1min":24.806000000000001,"5min":24.806000000000001,"15min":24.806000000000001},"last":1.389}]},{"osd":3,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":2.5800000000000001,"5min":2.004,"15min":1.7370000000000001},"min":{"1min":0.91300000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":23.741,"5min":23.741,"15min":23.741},"last":1.3300000000000001},{"interface":"front","average":{"1min":2.7149999999999999,"5min":2.0049999999999999,"15min":1.6679999999999999},"min":{"1min":0.86299999999999999,"5min":0.626,"15min":0.626},"max":{"1min":23.777000000000001,"5min":24.143000000000001,"15min":24.143000000000001},"last":1.4490000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":2.2320000000000002,"5min":2.0750000000000002,"15min":1.6830000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":22.355,"5min":25.637,"15min":25.637},"last":1.284},{"interface":"front","average":{"1min":2.3809999999999998,"5min":2.2799999999999998,"15min":1.855},"min":{"1min":0.63600000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":22.271999999999998,"5min":25.594000000000001,"15min":25.594000000000001},"last":1.1220000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":1.9059999999999999,"15min":1.75},"min":{"1min":0.52100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":7.8300000000000001,"5min":24.178999999999998,"15min":29.567},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.704,"5min":1.8300000000000001,"15min":1.8300000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":7.8540000000000001,"5min":24.538,"15min":27.937000000000001},"last":1.5980000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":2.1579999999999999,"15min":1.9239999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.6389999999999998,"5min":36.826000000000001,"15min":41.206000000000003},"last":1.51},{"interface":"front","average":{"1min":1.5349999999999999,"5min":2.5960000000000001,"15min":2.1589999999999998},"min":{"1min":0.71799999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.6339999999999999,"5min":39.932000000000002,"15min":39.932000000000002},"last":1.4830000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68232,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303608,"statfs":{"total":96636764160,"available":96566894592,"internally_reserved":0,"allocated":1040384,"data_stored":441925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.712,"15min":1.405},"min":{"1min":0.64600000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4179999999999999,"5min":20.739000000000001,"15min":20.739000000000001},"last":1.113},{"interface":"front","average":{"1min":1.456,"5min":1.4590000000000001,"15min":1.5960000000000001},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":8.0630000000000006,"5min":15.401,"15min":50.472000000000001},"last":1.036}]},{"osd":1,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.323,"15min":1.3120000000000001},"min":{"1min":0.78500000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.794,"5min":9.7769999999999992,"15min":10.629},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.546,"15min":1.4870000000000001},"min":{"1min":0.66100000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":7.2919999999999998,"5min":9.6880000000000006,"15min":9.6880000000000006},"last":1.673}]},{"osd":2,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.53,"15min":1.607},"min":{"1min":0.63700000000000001,"5min":0.50800000000000001,"15min":0.46899999999999997},"max":{"1min":6.3040000000000003,"5min":15.576000000000001,"15min":50.643999999999998},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.706,"5min":1.4870000000000001,"15min":1.4710000000000001},"min":{"1min":0.73399999999999999,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":8.8919999999999995,"5min":8.8919999999999995,"15min":10.843},"last":1.6220000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.5800000000000001,"15min":1.575},"min":{"1min":0.78900000000000003,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.8640000000000001,"5min":9.7460000000000004,"15min":15.675000000000001},"last":1.403},{"interface":"front","average":{"1min":2.0009999999999999,"5min":1.6870000000000001,"15min":1.7330000000000001},"min":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.52600000000000002},"max":{"1min":9.0960000000000001,"5min":15.504,"15min":50.561},"last":1.7609999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.6200000000000001,"15min":1.8160000000000001},"min":{"1min":0.49299999999999999,"5min":0.48599999999999999,"15min":0.46300000000000002},"max":{"1min":1.9399999999999999,"5min":7.391,"15min":47.488},"last":1.569},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.714,"15min":1.9059999999999999},"min":{"1min":0.52900000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":5.3360000000000003,"5min":15.638,"15min":50.381},"last":1.518}]},{"osd":5,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.351,"15min":1.518},"min":{"1min":0.45600000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.9179999999999999,"5min":6.718,"15min":10.654},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.71,"5min":1.899,"15min":1.8109999999999999},"min":{"1min":0.59499999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.9640000000000004,"5min":18.699000000000002,"15min":18.699000000000002},"last":1.8220000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.8660000000000001,"15min":2.073},"min":{"1min":0.65800000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":5.6609999999999996,"5min":15.702,"15min":52.912999999999997},"last":2.0859999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.5089999999999999,"15min":2.2869999999999999},"min":{"1min":0.60699999999999998,"5min":0.495,"15min":0.495},"max":{"1min":16.175999999999998,"5min":26.413,"15min":26.413},"last":1.8700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542303,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58628,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313212,"statfs":{"total":96636764160,"available":96576729088,"internally_reserved":0,"allocated":1105920,"data_stored":468726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.391,"15min":1.345},"min":{"1min":0.56000000000000005,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.677,"5min":10.56,"15min":13.938000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.4119999999999999,"15min":1.3640000000000001},"min":{"1min":0.73699999999999999,"5min":0.56699999999999995,"15min":0.42299999999999999},"max":{"1min":3.605,"5min":14.379,"15min":16.896000000000001},"last":0.82999999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.482,"15min":1.45},"min":{"1min":0.85299999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6899999999999999,"5min":15.500999999999999,"15min":15.500999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.292,"15min":1.4079999999999999},"min":{"1min":0.64200000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5840000000000001,"5min":6.7720000000000002,"15min":35.328000000000003},"last":1.1519999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.347,"15min":1.506},"min":{"1min":0.68500000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.5980000000000001,"5min":6.9020000000000001,"15min":35.398000000000003},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.5269999999999999,"15min":1.403},"min":{"1min":0.91000000000000003,"5min":0.47999999999999998,"15min":0.38900000000000001},"max":{"1min":1.7609999999999999,"5min":15.4,"15min":15.4},"last":1.048}]},{"osd":3,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.909,"15min":1.7390000000000001},"min":{"1min":1.123,"5min":0.63200000000000001,"15min":0.50900000000000001},"max":{"1min":16.931999999999999,"5min":16.931999999999999,"15min":24.254000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":2.399,"5min":1.788,"15min":1.804},"min":{"1min":0.72299999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":19.111999999999998,"5min":19.111999999999998,"15min":35.268000000000001},"last":1.3080000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.5589999999999999,"15min":1.669},"min":{"1min":0.89600000000000002,"5min":0.42199999999999999,"15min":0.39000000000000001},"max":{"1min":2.0030000000000001,"5min":19.059000000000001,"15min":19.059000000000001},"last":1.333},{"interface":"front","average":{"1min":1.524,"5min":1.667,"15min":1.7909999999999999},"min":{"1min":1.002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.8839999999999999,"5min":28.760000000000002,"15min":28.760000000000002},"last":1.272}]},{"osd":6,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.821,"15min":1.847},"min":{"1min":0.78900000000000003,"5min":0.54100000000000004,"15min":0.499},"max":{"1min":2.0209999999999999,"5min":16.271000000000001,"15min":35.427999999999997},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.97,"15min":1.859},"min":{"1min":0.749,"5min":0.53500000000000003,"15min":0.48199999999999998},"max":{"1min":1.9119999999999999,"5min":16.149999999999999,"15min":16.149999999999999},"last":1.468}]},{"osd":7,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.181,"15min":1.97},"min":{"1min":1.1339999999999999,"5min":0.54300000000000004,"15min":0.41099999999999998},"max":{"1min":12.385999999999999,"5min":29.672999999999998,"15min":29.672999999999998},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.627,"15min":1.8340000000000001},"min":{"1min":0.85699999999999998,"5min":0.45900000000000002,"15min":0.42899999999999999},"max":{"1min":2.1200000000000001,"5min":14.02,"15min":35.363},"last":1.5069999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64068,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307772,"statfs":{"total":96636764160,"available":96571158528,"internally_reserved":0,"allocated":1040384,"data_stored":441925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.6579999999999999,"15min":1.3859999999999999},"min":{"1min":0.72999999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.645,"5min":15.942,"15min":15.942},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.4319999999999999,"15min":1.417},"min":{"1min":0.89900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5569999999999999,"5min":16.219999999999999,"15min":16.219999999999999},"last":1.7849999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.5029999999999999,"15min":1.458},"min":{"1min":1.038,"5min":0.61499999999999999,"15min":0.54100000000000004},"max":{"1min":1.488,"5min":16.280000000000001,"15min":16.280000000000001},"last":1.278},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.7050000000000001,"15min":1.4650000000000001},"min":{"1min":0.85499999999999998,"5min":0.60699999999999998,"15min":0.60499999999999998},"max":{"1min":1.6830000000000001,"5min":15.888,"15min":15.888},"last":1.484}]},{"osd":2,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.5780000000000001,"15min":1.53},"min":{"1min":0.83199999999999996,"5min":0.55100000000000005,"15min":0.50800000000000001},"max":{"1min":1.4730000000000001,"5min":13.837,"15min":15.986000000000001},"last":1.379},{"interface":"front","average":{"1min":1.339,"5min":1.9279999999999999,"15min":1.6060000000000001},"min":{"1min":1.0960000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5600000000000001,"5min":17.175000000000001,"15min":17.175000000000001},"last":1.968}]},{"osd":3,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.859,"15min":1.7869999999999999},"min":{"1min":0.878,"5min":0.60299999999999998,"15min":0.57599999999999996},"max":{"1min":1.6899999999999999,"5min":16.469000000000001,"15min":16.469000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.6339999999999999,"15min":1.593},"min":{"1min":0.92000000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6259999999999999,"5min":15.089,"15min":21.706},"last":1.5900000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.8600000000000001,"15min":1.643},"min":{"1min":0.872,"5min":0.67200000000000004,"15min":0.59599999999999997},"max":{"1min":1.6319999999999999,"5min":18.347999999999999,"15min":18.347999999999999},"last":1.841},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.6819999999999999,"15min":1.667},"min":{"1min":0.89100000000000001,"5min":0.53900000000000003,"15min":0.45800000000000002},"max":{"1min":1.665,"5min":22.937000000000001,"15min":22.937000000000001},"last":1.5549999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.7110000000000001,"15min":1.6000000000000001},"min":{"1min":0.91800000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.643,"5min":18.033000000000001,"15min":18.033000000000001},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.474,"5min":2.1680000000000001,"15min":1.8979999999999999},"min":{"1min":1,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.7390000000000001,"5min":17.794,"15min":20.939},"last":1.6990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.7669999999999999,"15min":1.77},"min":{"1min":1.208,"5min":0.58299999999999996,"15min":0.44500000000000001},"max":{"1min":1.7769999999999999,"5min":16.324999999999999,"15min":16.324999999999999},"last":2.0110000000000001},{"interface":"front","average":{"1min":1.46,"5min":2.0190000000000001,"15min":1.881},"min":{"1min":1.032,"5min":0.57699999999999996,"15min":0.52200000000000002},"max":{"1min":1.7150000000000001,"5min":12.335000000000001,"15min":20.277999999999999},"last":2.0499999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72392,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299448,"statfs":{"total":96636764160,"available":96562634752,"internally_reserved":0,"allocated":1040384,"data_stored":441925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.997,"15min":0.97499999999999998},"min":{"1min":0.42699999999999999,"5min":0.35199999999999998,"15min":0.33600000000000002},"max":{"1min":1.5429999999999999,"5min":4.3700000000000001,"15min":5.3570000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.167,"15min":1.0780000000000001},"min":{"1min":0.52700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":12.292999999999999,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.69199999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.94499999999999995,"15min":0.91400000000000003},"min":{"1min":0.50900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.8999999999999999,"5min":4.0979999999999999,"15min":4.0979999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.2989999999999999,"15min":1.1479999999999999},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.875,"5min":13.443,"15min":13.443},"last":1.0549999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1759999999999999,"15min":1.129},"min":{"1min":0.55900000000000005,"5min":0.45300000000000001,"15min":0.376},"max":{"1min":1.7110000000000001,"5min":3.806,"15min":4.008},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.173,"5min":1.2749999999999999,"15min":1.2050000000000001},"min":{"1min":0.46300000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6020000000000001,"5min":5.4539999999999997,"15min":5.5110000000000001},"last":1.3799999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.9319999999999999,"5min":1.4079999999999999,"15min":1.3400000000000001},"min":{"1min":0.77700000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":12.744,"5min":12.744,"15min":13.292},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.431,"15min":1.4339999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.3780000000000001,"5min":6.4279999999999999,"15min":11.068},"last":1.137}]},{"osd":5,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.724,"15min":1.5269999999999999},"min":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.6659999999999999,"5min":17.096,"15min":17.096},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.605,"15min":1.5509999999999999},"min":{"1min":0.78500000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.137,"5min":9.3629999999999995,"15min":12.106},"last":1.5329999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":2.294,"15min":1.7210000000000001},"min":{"1min":0.85799999999999998,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":12.675000000000001,"5min":43.609999999999999,"15min":43.609999999999999},"last":1.095},{"interface":"front","average":{"1min":1.462,"5min":1.9390000000000001,"15min":1.6659999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.4649999999999999,"5min":38.784999999999997,"15min":38.784999999999997},"last":1.4350000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":2.0070000000000001,"15min":1.6919999999999999},"min":{"1min":0.84599999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4489999999999998,"5min":25.071000000000002,"15min":25.071000000000002},"last":1.474},{"interface":"front","average":{"1min":2.448,"5min":2.081,"15min":1.7350000000000001},"min":{"1min":0.71999999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":12.836,"5min":13.365,"15min":18.846},"last":1.1899999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542303,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68288,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303552,"statfs":{"total":96636764160,"available":96566837248,"internally_reserved":0,"allocated":1093632,"data_stored":467674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.1359999999999999,"15min":0.998},"min":{"1min":0.53600000000000003,"5min":0.224,"15min":0.224},"max":{"1min":1.502,"5min":11.804,"15min":11.804},"last":0.311},{"interface":"front","average":{"1min":1.048,"5min":1.2450000000000001,"15min":1.1020000000000001},"min":{"1min":0.57999999999999996,"5min":0.435,"15min":0.35799999999999998},"max":{"1min":1.649,"5min":13.066000000000001,"15min":13.066000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.353,"15min":1.147},"min":{"1min":0.60199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.4850000000000001,"5min":14.199999999999999,"15min":14.199999999999999},"last":0.434},{"interface":"front","average":{"1min":1.133,"5min":1.345,"15min":1.1799999999999999},"min":{"1min":0.495,"5min":0.42999999999999999,"15min":0.372},"max":{"1min":1.5820000000000001,"5min":13.15,"15min":13.15},"last":0.70599999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.329,"15min":1.1319999999999999},"min":{"1min":0.54700000000000004,"5min":0.38200000000000001,"15min":0.31900000000000001},"max":{"1min":1.617,"5min":14.965,"15min":14.965},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.357,"15min":1.2330000000000001},"min":{"1min":0.56799999999999995,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.7270000000000001,"5min":11.717000000000001,"15min":11.717000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.9710000000000001,"15min":1.6399999999999999},"min":{"1min":0.85899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.173,"5min":19.957999999999998,"15min":19.957999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.385,"5min":1.77,"15min":1.585},"min":{"1min":0.91800000000000004,"5min":0.78400000000000003,"15min":0.55700000000000005},"max":{"1min":1.8520000000000001,"5min":15.151,"15min":15.151},"last":0.95499999999999996}]},{"osd":5,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.611,"15min":1.573},"min":{"1min":1.071,"5min":0.61099999999999999,"15min":0.60899999999999999},"max":{"1min":2.0409999999999999,"5min":15.077,"15min":26.501000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.593,"15min":1.4430000000000001},"min":{"1min":0.97499999999999998,"5min":0.78000000000000003,"15min":0.47999999999999998},"max":{"1min":2,"5min":13.760999999999999,"15min":13.760999999999999},"last":0.97799999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5069999999999999,"15min":1.49},"min":{"1min":0.93700000000000006,"5min":0.77500000000000002,"15min":0.438},"max":{"1min":2.1400000000000001,"5min":6.3940000000000001,"15min":15.144},"last":1.036},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.718,"15min":1.8280000000000001},"min":{"1min":0.95899999999999996,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":2.3079999999999998,"5min":14.327,"15min":66.061999999999998},"last":0.81100000000000005}]},{"osd":7,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.885,"15min":1.6839999999999999},"min":{"1min":1.012,"5min":0.81200000000000006,"15min":0.61099999999999999},"max":{"1min":1.952,"5min":16.667000000000002,"15min":16.667000000000002},"last":0.871},{"interface":"front","average":{"1min":1.494,"5min":1.708,"15min":1.6120000000000001},"min":{"1min":0.89800000000000002,"5min":0.71499999999999997,"15min":0.5},"max":{"1min":2.0699999999999998,"5min":8.8109999999999999,"15min":11.271000000000001},"last":1.1599999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68228,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303612,"statfs":{"total":96636764160,"available":96566898688,"internally_reserved":0,"allocated":1040384,"data_stored":441925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.349,"15min":1.0800000000000001},"min":{"1min":0.755,"5min":0.32600000000000001,"15min":0.248},"max":{"1min":1.637,"5min":21.260000000000002,"15min":21.260000000000002},"last":1.117},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.478,"15min":1.2490000000000001},"min":{"1min":0.63200000000000001,"5min":0.44900000000000001,"15min":0.35199999999999998},"max":{"1min":2.3559999999999999,"5min":32.787999999999997,"15min":32.787999999999997},"last":0.77500000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.014,"15min":1.0009999999999999},"min":{"1min":0.439,"5min":0.38,"15min":0.28100000000000003},"max":{"1min":1.4610000000000001,"5min":7.3970000000000002,"15min":14.817},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2909999999999999,"15min":1.1499999999999999},"min":{"1min":0.74099999999999999,"5min":0.42599999999999999,"15min":0.41499999999999998},"max":{"1min":2.9660000000000002,"5min":8.8450000000000006,"15min":8.8450000000000006},"last":0.98999999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3919999999999999,"15min":1.2030000000000001},"min":{"1min":0.745,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":1.7709999999999999,"5min":9.3149999999999995,"15min":9.3149999999999995},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.5069999999999999,"15min":1.2929999999999999},"min":{"1min":0.47099999999999997,"5min":0.41499999999999998,"15min":0.40400000000000003},"max":{"1min":2.2040000000000002,"5min":41.133000000000003,"15min":41.133000000000003},"last":0.504}]},{"osd":4,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.724,"15min":1.528},"min":{"1min":0.80900000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.2480000000000002,"5min":19.684000000000001,"15min":19.684000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4019999999999999,"15min":1.4279999999999999},"min":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.53900000000000003},"max":{"1min":2.052,"5min":10.785,"15min":13.789999999999999},"last":1.5600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.698,"15min":1.569},"min":{"1min":1.004,"5min":0.622,"15min":0.43099999999999999},"max":{"1min":5.3680000000000003,"5min":15.778,"15min":15.778},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.643,"5min":1.901,"15min":1.7689999999999999},"min":{"1min":1.0189999999999999,"5min":0.623,"15min":0.623},"max":{"1min":5.4610000000000003,"5min":15.723000000000001,"15min":35.747},"last":1.282}]},{"osd":6,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.734,"15min":1.6539999999999999},"min":{"1min":0.93200000000000005,"5min":0.64400000000000002,"15min":0.629},"max":{"1min":4.6260000000000003,"5min":15.586,"15min":16.231000000000002},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.5920000000000001,"15min":1.55},"min":{"1min":0.64400000000000002,"5min":0.63200000000000001,"15min":0.56599999999999995},"max":{"1min":2.1659999999999999,"5min":11.433999999999999,"15min":13.56},"last":1.238}]},{"osd":7,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.754,"5min":1.964,"15min":1.8},"min":{"1min":1.0549999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":6.7990000000000004,"5min":18.907,"15min":21.902999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.98,"5min":1.9930000000000001,"15min":1.8009999999999999},"min":{"1min":1.1910000000000001,"5min":0.70299999999999996,"15min":0.64700000000000002},"max":{"1min":5.1260000000000003,"5min":15.452999999999999,"15min":31.044},"last":1.153}]}]},{"osd":0,"up_from":14,"seq":60129542304,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64704,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307136,"statfs":{"total":96636764160,"available":96570507264,"internally_reserved":0,"allocated":1687552,"data_stored":1057494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.121,"15min":1.026},"min":{"1min":0.54400000000000004,"5min":0.26400000000000001,"15min":0.25},"max":{"1min":2.7320000000000002,"5min":22.619,"15min":22.619},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.0840000000000001,"15min":1.1899999999999999},"min":{"1min":0.438,"5min":0.31,"15min":0.31},"max":{"1min":9.4499999999999993,"5min":9.4499999999999993,"15min":17.876000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0819999999999999,"5min":1.353,"15min":1.365},"min":{"1min":1.093,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":17.760999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.351,"5min":1.403,"15min":1.3320000000000001},"min":{"1min":0.40200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.7930000000000001,"5min":22.672999999999998,"15min":22.672999999999998},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.016,"15min":1.1359999999999999},"min":{"1min":0.60499999999999998,"5min":0.40000000000000002,"15min":0.39200000000000002},"max":{"1min":1.4730000000000001,"5min":3.0600000000000001,"15min":16.382999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":2.2839999999999998,"5min":1.4510000000000001,"15min":1.347},"min":{"1min":0.56799999999999995,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":9.6029999999999998,"5min":9.6029999999999998,"15min":17.981000000000002},"last":0.60399999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.591,"15min":1.5549999999999999},"min":{"1min":0.63400000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.6399999999999999,"5min":22.780000000000001,"15min":22.780000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.284,"15min":1.5169999999999999},"min":{"1min":0.86699999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.881,"5min":3.085,"15min":19.422000000000001},"last":1.151}]},{"osd":5,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.7629999999999999,"15min":1.861},"min":{"1min":0.70499999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.7490000000000001,"5min":22.744,"15min":44.930999999999997},"last":1.369},{"interface":"front","average":{"1min":2.0640000000000001,"5min":1.972,"15min":1.7230000000000001},"min":{"1min":0.81200000000000006,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":14.561999999999999,"5min":21.797999999999998,"15min":21.797999999999998},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":1.8879999999999999,"15min":1.8919999999999999},"min":{"1min":0.76600000000000001,"5min":0.622,"15min":0.622},"max":{"1min":14.958,"5min":14.958,"15min":27.442},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.968,"15min":1.927},"min":{"1min":0.84099999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.081,"5min":22.709,"15min":27.411999999999999},"last":1.1299999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.548,"5min":1.7390000000000001,"15min":1.724},"min":{"1min":1.319,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":15.005000000000001,"5min":15.005000000000001,"15min":18.152999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.911,"15min":1.764},"min":{"1min":0.70699999999999996,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9199999999999999,"5min":22.815000000000001,"15min":22.815000000000001},"last":0.88}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:10:06.386 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:10:06.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.659+0000 7f7bf8e00640 1 Processor -- start 2024-01-16T15:10:06.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.659+0000 7f7bf8e00640 1 -- start start 2024-01-16T15:10:06.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.660+0000 7f7bf8e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7bf4143ae0 0x7f7bf40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.660+0000 7f7bf8e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.660+0000 7f7bf8e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf40cfc30 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.660+0000 7f7bf8e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf40b1ee0 con 0x7f7bf4143ae0 2024-01-16T15:10:06.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.660+0000 7f7bf8e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40b2780 con 0x7f7bf415e3b0 2024-01-16T15:10:06.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.661+0000 7f7bf1e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7bf4143ae0 0x7f7bf40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.661+0000 7f7bf2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.661+0000 7f7bf1e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7bf4143ae0 0x7f7bf40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:60026/0 (socket says 172.21.15.141:60026) 2024-01-16T15:10:06.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.661+0000 7f7bf1e00640 1 -- 172.21.15.141:0/1482228079 learned_addr learned my addr 172.21.15.141:0/1482228079 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:06.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.662+0000 7f7bf1e00640 1 -- 172.21.15.141:0/1482228079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 msgr2=0x7f7bf40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:06.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.662+0000 7f7bf1e00640 1 --2- 172.21.15.141:0/1482228079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.662+0000 7f7bf1e00640 1 -- 172.21.15.141:0/1482228079 >> v1:172.21.15.188:6789/0 conn(0x7f7bf40cf7d0 legacy=0x7f7bf415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.662+0000 7f7bf1e00640 1 -- 172.21.15.141:0/1482228079 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf415e990 con 0x7f7bf4143ae0 2024-01-16T15:10:06.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.663+0000 7f7bf1e00640 1 --2- 172.21.15.141:0/1482228079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 0x7f7bf40cf290 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bfff4d9f249a992 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:06.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf0a00640 1 -- 172.21.15.141:0/1482228079 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be402e940 con 0x7f7bf4143ae0 2024-01-16T15:10:06.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf0a00640 1 -- 172.21.15.141:0/1482228079 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be402eaa0 con 0x7f7bf4143ae0 2024-01-16T15:10:06.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf0a00640 1 -- 172.21.15.141:0/1482228079 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be402ee30 con 0x7f7bf4143ae0 2024-01-16T15:10:06.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 -- 172.21.15.141:0/1482228079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 msgr2=0x7f7bf40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/1482228079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 0x7f7bf40cf290 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402db90 comp rx=0 tx=0).stop 2024-01-16T15:10:06.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 -- 172.21.15.141:0/1482228079 shutdown_connections 2024-01-16T15:10:06.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/1482228079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/1482228079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 0x7f7bf40cf290 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 -- 172.21.15.141:0/1482228079 >> 172.21.15.141:0/1482228079 conn(0x7f7bf406a800 msgr2=0x7f7bf406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:06.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 -- 172.21.15.141:0/1482228079 shutdown_connections 2024-01-16T15:10:06.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.664+0000 7f7bf8e00640 1 -- 172.21.15.141:0/1482228079 wait complete. 2024-01-16T15:10:06.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.666+0000 7f7bf8e00640 1 Processor -- start 2024-01-16T15:10:06.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.666+0000 7f7bf8e00640 1 -- start start 2024-01-16T15:10:06.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.667+0000 7f7bf8e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40cf7d0 0x7f7bf413f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.667+0000 7f7bf8e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 0x7f7bf40acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.667+0000 7f7bf8e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf4140980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.667+0000 7f7bf8e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40ae200 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.667+0000 7f7bf8e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40ad5f0 con 0x7f7bf4143ae0 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.667+0000 7f7bf8e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40ad8d0 con 0x7f7bf415e3b0 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40cf7d0 0x7f7bf413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40cf7d0 0x7f7bf413f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46192/0 (socket says 172.21.15.141:46192) 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 -- 172.21.15.141:0/2955905963 learned_addr learned my addr 172.21.15.141:0/2955905963 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1400640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 0x7f7bf40acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf2800640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf4140980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 -- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 msgr2=0x7f7bf4140980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf4140980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 -- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 msgr2=0x7f7bf40acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 0x7f7bf40acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 -- 172.21.15.141:0/2955905963 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be40094d0 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.668+0000 7f7bf1e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40cf7d0 0x7f7bf413f100 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.669+0000 7f7bea000640 1 -- 172.21.15.141:0/2955905963 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4040600 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.669+0000 7f7bea000640 1 -- 172.21.15.141:0/2955905963 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be4040760 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.669+0000 7f7bea000640 1 -- 172.21.15.141:0/2955905963 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4040b10 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.669+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf40adbb0 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.669+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf413f640 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.670+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf413f7d0 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.671+0000 7f7bea000640 1 -- 172.21.15.141:0/2955905963 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7be4040c70 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.672+0000 7f7bea000640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06bfe0 0x7f7bcc06e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:06.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.672+0000 7f7bea000640 1 -- 172.21.15.141:0/2955905963 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7be40ab190 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.673+0000 7f7bf1400640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06bfe0 0x7f7bcc06e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:06.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.674+0000 7f7bf1400640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06bfe0 0x7f7bcc06e4a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc000f60 tx=0x7f7bdc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:06.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.676+0000 7f7bea000640 1 -- 172.21.15.141:0/2955905963 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bf413f7d0 con 0x7f7bf40cf7d0 2024-01-16T15:10:06.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.865+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7bf40b2410 con 0x7f7bcc06bfe0 2024-01-16T15:10:06.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.891+0000 7f7bea000640 1 -- 172.21.15.141:0/2955905963 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286742 (secure 0 0 0) 0x7f7bdc001c70 con 0x7f7bcc06bfe0 2024-01-16T15:10:06.872 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:06.874 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06bfe0 msgr2=0x7f7bcc06e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06bfe0 0x7f7bcc06e4a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc000f60 tx=0x7f7bdc009040 comp rx=0 tx=0).stop 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40cf7d0 msgr2=0x7f7bf413f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40cf7d0 0x7f7bf413f100 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be40046c0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 shutdown_connections 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7bcc06bfe0 0x7f7bcc06e4a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7bf415e3b0 0x7f7bf4140980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bf4143ae0 0x7f7bf40acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 --2- 172.21.15.141:0/2955905963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7bf40cf7d0 0x7f7bf413f100 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 >> 172.21.15.141:0/2955905963 conn(0x7f7bf406a800 msgr2=0x7f7bf406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 shutdown_connections 2024-01-16T15:10:06.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:06.898+0000 7f7bf8e00640 1 -- 172.21.15.141:0/2955905963 wait complete. 2024-01-16T15:10:06.895 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-01-16T15:10:06.785927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":533648,"kb_used_data":9736,"kb_used_omap":12,"kb_used_meta":523763,"kb_avail":754441072,"statfs":{"total":773094113280,"available":772547657728,"internally_reserved":0,"allocated":9969664,"data_stored":4922292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":536333874},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002031"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028151+0000","last_change":"2024-01-16T15:10:01.954220+0000","last_active":"2024-01-16T15:10:03.028151+0000","last_peered":"2024-01-16T15:10:03.028151+0000","last_clean":"2024-01-16T15:10:03.028151+0000","last_became_active":"2024-01-16T15:10:01.952310+0000","last_became_peered":"2024-01-16T15:10:01.952310+0000","last_unstale":"2024-01-16T15:10:03.028151+0000","last_undegraded":"2024-01-16T15:10:03.028151+0000","last_fullsized":"2024-01-16T15:10:03.028151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:02.446788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.025448+0000","last_change":"2024-01-16T15:10:01.961018+0000","last_active":"2024-01-16T15:10:03.025448+0000","last_peered":"2024-01-16T15:10:03.025448+0000","last_clean":"2024-01-16T15:10:03.025448+0000","last_became_active":"2024-01-16T15:10:01.960721+0000","last_became_peered":"2024-01-16T15:10:01.960721+0000","last_unstale":"2024-01-16T15:10:03.025448+0000","last_undegraded":"2024-01-16T15:10:03.025448+0000","last_fullsized":"2024-01-16T15:10:03.025448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:51.222715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027385+0000","last_change":"2024-01-16T15:10:01.949623+0000","last_active":"2024-01-16T15:10:03.027385+0000","last_peered":"2024-01-16T15:10:03.027385+0000","last_clean":"2024-01-16T15:10:03.027385+0000","last_became_active":"2024-01-16T15:10:01.949332+0000","last_became_peered":"2024-01-16T15:10:01.949332+0000","last_unstale":"2024-01-16T15:10:03.027385+0000","last_undegraded":"2024-01-16T15:10:03.027385+0000","last_fullsized":"2024-01-16T15:10:03.027385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:25:38.026601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949242+0000","last_change":"2024-01-16T15:10:01.954843+0000","last_active":"2024-01-16T15:10:02.949242+0000","last_peered":"2024-01-16T15:10:02.949242+0000","last_clean":"2024-01-16T15:10:02.949242+0000","last_became_active":"2024-01-16T15:10:01.954556+0000","last_became_peered":"2024-01-16T15:10:01.954556+0000","last_unstale":"2024-01-16T15:10:02.949242+0000","last_undegraded":"2024-01-16T15:10:02.949242+0000","last_fullsized":"2024-01-16T15:10:02.949242+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:23:22.743543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950724+0000","last_change":"2024-01-16T15:10:01.957493+0000","last_active":"2024-01-16T15:10:02.950724+0000","last_peered":"2024-01-16T15:10:02.950724+0000","last_clean":"2024-01-16T15:10:02.950724+0000","last_became_active":"2024-01-16T15:10:01.954212+0000","last_became_peered":"2024-01-16T15:10:01.954212+0000","last_unstale":"2024-01-16T15:10:02.950724+0000","last_undegraded":"2024-01-16T15:10:02.950724+0000","last_fullsized":"2024-01-16T15:10:02.950724+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:20.817578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950553+0000","last_change":"2024-01-16T15:10:01.948258+0000","last_active":"2024-01-16T15:10:02.950553+0000","last_peered":"2024-01-16T15:10:02.950553+0000","last_clean":"2024-01-16T15:10:02.950553+0000","last_became_active":"2024-01-16T15:10:01.947983+0000","last_became_peered":"2024-01-16T15:10:01.947983+0000","last_unstale":"2024-01-16T15:10:02.950553+0000","last_undegraded":"2024-01-16T15:10:02.950553+0000","last_fullsized":"2024-01-16T15:10:02.950553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:48.782361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948844+0000","last_change":"2024-01-16T15:10:01.946350+0000","last_active":"2024-01-16T15:10:02.948844+0000","last_peered":"2024-01-16T15:10:02.948844+0000","last_clean":"2024-01-16T15:10:02.948844+0000","last_became_active":"2024-01-16T15:10:01.946095+0000","last_became_peered":"2024-01-16T15:10:01.946095+0000","last_unstale":"2024-01-16T15:10:02.948844+0000","last_undegraded":"2024-01-16T15:10:02.948844+0000","last_fullsized":"2024-01-16T15:10:02.948844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:14.681946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949603+0000","last_change":"2024-01-16T15:10:01.956517+0000","last_active":"2024-01-16T15:10:02.949603+0000","last_peered":"2024-01-16T15:10:02.949603+0000","last_clean":"2024-01-16T15:10:02.949603+0000","last_became_active":"2024-01-16T15:10:01.954117+0000","last_became_peered":"2024-01-16T15:10:01.954117+0000","last_unstale":"2024-01-16T15:10:02.949603+0000","last_undegraded":"2024-01-16T15:10:02.949603+0000","last_fullsized":"2024-01-16T15:10:02.949603+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:32:00.871458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950520+0000","last_change":"2024-01-16T15:10:01.961525+0000","last_active":"2024-01-16T15:10:02.950520+0000","last_peered":"2024-01-16T15:10:02.950520+0000","last_clean":"2024-01-16T15:10:02.950520+0000","last_became_active":"2024-01-16T15:10:01.961148+0000","last_became_peered":"2024-01-16T15:10:01.961148+0000","last_unstale":"2024-01-16T15:10:02.950520+0000","last_undegraded":"2024-01-16T15:10:02.950520+0000","last_fullsized":"2024-01-16T15:10:02.950520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:13:16.862717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028074+0000","last_change":"2024-01-16T15:10:01.951626+0000","last_active":"2024-01-16T15:10:03.028074+0000","last_peered":"2024-01-16T15:10:03.028074+0000","last_clean":"2024-01-16T15:10:03.028074+0000","last_became_active":"2024-01-16T15:10:01.950187+0000","last_became_peered":"2024-01-16T15:10:01.950187+0000","last_unstale":"2024-01-16T15:10:03.028074+0000","last_undegraded":"2024-01-16T15:10:03.028074+0000","last_fullsized":"2024-01-16T15:10:03.028074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:22.665983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.025467+0000","last_change":"2024-01-16T15:10:01.950980+0000","last_active":"2024-01-16T15:10:03.025467+0000","last_peered":"2024-01-16T15:10:03.025467+0000","last_clean":"2024-01-16T15:10:03.025467+0000","last_became_active":"2024-01-16T15:10:01.950685+0000","last_became_peered":"2024-01-16T15:10:01.950685+0000","last_unstale":"2024-01-16T15:10:03.025467+0000","last_undegraded":"2024-01-16T15:10:03.025467+0000","last_fullsized":"2024-01-16T15:10:03.025467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:00.372980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027343+0000","last_change":"2024-01-16T15:10:01.951225+0000","last_active":"2024-01-16T15:10:03.027343+0000","last_peered":"2024-01-16T15:10:03.027343+0000","last_clean":"2024-01-16T15:10:03.027343+0000","last_became_active":"2024-01-16T15:10:01.949442+0000","last_became_peered":"2024-01-16T15:10:01.949442+0000","last_unstale":"2024-01-16T15:10:03.027343+0000","last_undegraded":"2024-01-16T15:10:03.027343+0000","last_fullsized":"2024-01-16T15:10:03.027343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:21.572489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.026823+0000","last_change":"2024-01-16T15:10:01.942609+0000","last_active":"2024-01-16T15:10:03.026823+0000","last_peered":"2024-01-16T15:10:03.026823+0000","last_clean":"2024-01-16T15:10:03.026823+0000","last_became_active":"2024-01-16T15:10:01.942275+0000","last_became_peered":"2024-01-16T15:10:01.942275+0000","last_unstale":"2024-01-16T15:10:03.026823+0000","last_undegraded":"2024-01-16T15:10:03.026823+0000","last_fullsized":"2024-01-16T15:10:03.026823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:01.197727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949525+0000","last_change":"2024-01-16T15:10:01.959882+0000","last_active":"2024-01-16T15:10:02.949525+0000","last_peered":"2024-01-16T15:10:02.949525+0000","last_clean":"2024-01-16T15:10:02.949525+0000","last_became_active":"2024-01-16T15:10:01.959588+0000","last_became_peered":"2024-01-16T15:10:01.959588+0000","last_unstale":"2024-01-16T15:10:02.949525+0000","last_undegraded":"2024-01-16T15:10:02.949525+0000","last_fullsized":"2024-01-16T15:10:02.949525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:05.704870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024525+0000","last_change":"2024-01-16T15:10:01.937904+0000","last_active":"2024-01-16T15:10:03.024525+0000","last_peered":"2024-01-16T15:10:03.024525+0000","last_clean":"2024-01-16T15:10:03.024525+0000","last_became_active":"2024-01-16T15:10:01.937639+0000","last_became_peered":"2024-01-16T15:10:01.937639+0000","last_unstale":"2024-01-16T15:10:03.024525+0000","last_undegraded":"2024-01-16T15:10:03.024525+0000","last_fullsized":"2024-01-16T15:10:03.024525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:52.596830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948806+0000","last_change":"2024-01-16T15:10:01.950652+0000","last_active":"2024-01-16T15:10:02.948806+0000","last_peered":"2024-01-16T15:10:02.948806+0000","last_clean":"2024-01-16T15:10:02.948806+0000","last_became_active":"2024-01-16T15:10:01.950138+0000","last_became_peered":"2024-01-16T15:10:01.950138+0000","last_unstale":"2024-01-16T15:10:02.948806+0000","last_undegraded":"2024-01-16T15:10:02.948806+0000","last_fullsized":"2024-01-16T15:10:02.948806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:45.868610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951452+0000","last_change":"2024-01-16T15:10:01.947112+0000","last_active":"2024-01-16T15:10:02.951452+0000","last_peered":"2024-01-16T15:10:02.951452+0000","last_clean":"2024-01-16T15:10:02.951452+0000","last_became_active":"2024-01-16T15:10:01.946833+0000","last_became_peered":"2024-01-16T15:10:01.946833+0000","last_unstale":"2024-01-16T15:10:02.951452+0000","last_undegraded":"2024-01-16T15:10:02.951452+0000","last_fullsized":"2024-01-16T15:10:02.951452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:13:16.862717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949363+0000","last_change":"2024-01-16T15:10:01.954713+0000","last_active":"2024-01-16T15:10:02.949363+0000","last_peered":"2024-01-16T15:10:02.949363+0000","last_clean":"2024-01-16T15:10:02.949363+0000","last_became_active":"2024-01-16T15:10:01.954427+0000","last_became_peered":"2024-01-16T15:10:01.954427+0000","last_unstale":"2024-01-16T15:10:02.949363+0000","last_undegraded":"2024-01-16T15:10:02.949363+0000","last_fullsized":"2024-01-16T15:10:02.949363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:42.665836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998616+0000","last_change":"2024-01-16T15:10:01.935256+0000","last_active":"2024-01-16T15:10:02.998616+0000","last_peered":"2024-01-16T15:10:02.998616+0000","last_clean":"2024-01-16T15:10:02.998616+0000","last_became_active":"2024-01-16T15:10:01.932457+0000","last_became_peered":"2024-01-16T15:10:01.932457+0000","last_unstale":"2024-01-16T15:10:02.998616+0000","last_undegraded":"2024-01-16T15:10:02.998616+0000","last_fullsized":"2024-01-16T15:10:02.998616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:52:44.520711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949557+0000","last_change":"2024-01-16T15:10:01.937972+0000","last_active":"2024-01-16T15:10:02.949557+0000","last_peered":"2024-01-16T15:10:02.949557+0000","last_clean":"2024-01-16T15:10:02.949557+0000","last_became_active":"2024-01-16T15:10:01.937629+0000","last_became_peered":"2024-01-16T15:10:01.937629+0000","last_unstale":"2024-01-16T15:10:02.949557+0000","last_undegraded":"2024-01-16T15:10:02.949557+0000","last_fullsized":"2024-01-16T15:10:02.949557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:23.435918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949005+0000","last_change":"2024-01-16T15:10:01.946491+0000","last_active":"2024-01-16T15:10:02.949005+0000","last_peered":"2024-01-16T15:10:02.949005+0000","last_clean":"2024-01-16T15:10:02.949005+0000","last_became_active":"2024-01-16T15:10:01.946236+0000","last_became_peered":"2024-01-16T15:10:01.946236+0000","last_unstale":"2024-01-16T15:10:02.949005+0000","last_undegraded":"2024-01-16T15:10:02.949005+0000","last_fullsized":"2024-01-16T15:10:02.949005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:41.287672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024502+0000","last_change":"2024-01-16T15:10:01.961238+0000","last_active":"2024-01-16T15:10:03.024502+0000","last_peered":"2024-01-16T15:10:03.024502+0000","last_clean":"2024-01-16T15:10:03.024502+0000","last_became_active":"2024-01-16T15:10:01.960245+0000","last_became_peered":"2024-01-16T15:10:01.960245+0000","last_unstale":"2024-01-16T15:10:03.024502+0000","last_undegraded":"2024-01-16T15:10:03.024502+0000","last_fullsized":"2024-01-16T15:10:03.024502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:04:32.835751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027200+0000","last_change":"2024-01-16T15:10:01.964921+0000","last_active":"2024-01-16T15:10:03.027200+0000","last_peered":"2024-01-16T15:10:03.027200+0000","last_clean":"2024-01-16T15:10:03.027200+0000","last_became_active":"2024-01-16T15:10:01.964540+0000","last_became_peered":"2024-01-16T15:10:01.964540+0000","last_unstale":"2024-01-16T15:10:03.027200+0000","last_undegraded":"2024-01-16T15:10:03.027200+0000","last_fullsized":"2024-01-16T15:10:03.027200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:50.150508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948562+0000","last_change":"2024-01-16T15:10:01.940418+0000","last_active":"2024-01-16T15:10:02.948562+0000","last_peered":"2024-01-16T15:10:02.948562+0000","last_clean":"2024-01-16T15:10:02.948562+0000","last_became_active":"2024-01-16T15:10:01.938464+0000","last_became_peered":"2024-01-16T15:10:01.938464+0000","last_unstale":"2024-01-16T15:10:02.948562+0000","last_undegraded":"2024-01-16T15:10:02.948562+0000","last_fullsized":"2024-01-16T15:10:02.948562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:21:40.890614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.025475+0000","last_change":"2024-01-16T15:10:01.962811+0000","last_active":"2024-01-16T15:10:03.025475+0000","last_peered":"2024-01-16T15:10:03.025475+0000","last_clean":"2024-01-16T15:10:03.025475+0000","last_became_active":"2024-01-16T15:10:01.962525+0000","last_became_peered":"2024-01-16T15:10:01.962525+0000","last_unstale":"2024-01-16T15:10:03.025475+0000","last_undegraded":"2024-01-16T15:10:03.025475+0000","last_fullsized":"2024-01-16T15:10:03.025475+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:09.303571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028045+0000","last_change":"2024-01-16T15:10:01.954910+0000","last_active":"2024-01-16T15:10:03.028045+0000","last_peered":"2024-01-16T15:10:03.028045+0000","last_clean":"2024-01-16T15:10:03.028045+0000","last_became_active":"2024-01-16T15:10:01.954635+0000","last_became_peered":"2024-01-16T15:10:01.954635+0000","last_unstale":"2024-01-16T15:10:03.028045+0000","last_undegraded":"2024-01-16T15:10:03.028045+0000","last_fullsized":"2024-01-16T15:10:03.028045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:54.681736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951285+0000","last_change":"2024-01-16T15:10:01.946001+0000","last_active":"2024-01-16T15:10:02.951285+0000","last_peered":"2024-01-16T15:10:02.951285+0000","last_clean":"2024-01-16T15:10:02.951285+0000","last_became_active":"2024-01-16T15:10:01.945580+0000","last_became_peered":"2024-01-16T15:10:01.945580+0000","last_unstale":"2024-01-16T15:10:02.951285+0000","last_undegraded":"2024-01-16T15:10:02.951285+0000","last_fullsized":"2024-01-16T15:10:02.951285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:18.268867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027510+0000","last_change":"2024-01-16T15:10:01.960309+0000","last_active":"2024-01-16T15:10:03.027510+0000","last_peered":"2024-01-16T15:10:03.027510+0000","last_clean":"2024-01-16T15:10:03.027510+0000","last_became_active":"2024-01-16T15:10:01.959551+0000","last_became_peered":"2024-01-16T15:10:01.959551+0000","last_unstale":"2024-01-16T15:10:03.027510+0000","last_undegraded":"2024-01-16T15:10:03.027510+0000","last_fullsized":"2024-01-16T15:10:03.027510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:33.792440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024489+0000","last_change":"2024-01-16T15:10:01.945898+0000","last_active":"2024-01-16T15:10:03.024489+0000","last_peered":"2024-01-16T15:10:03.024489+0000","last_clean":"2024-01-16T15:10:03.024489+0000","last_became_active":"2024-01-16T15:10:01.945585+0000","last_became_peered":"2024-01-16T15:10:01.945585+0000","last_unstale":"2024-01-16T15:10:03.024489+0000","last_undegraded":"2024-01-16T15:10:03.024489+0000","last_fullsized":"2024-01-16T15:10:03.024489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:24.722977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024836+0000","last_change":"2024-01-16T15:10:01.953614+0000","last_active":"2024-01-16T15:10:03.024836+0000","last_peered":"2024-01-16T15:10:03.024836+0000","last_clean":"2024-01-16T15:10:03.024836+0000","last_became_active":"2024-01-16T15:10:01.953339+0000","last_became_peered":"2024-01-16T15:10:01.953339+0000","last_unstale":"2024-01-16T15:10:03.024836+0000","last_undegraded":"2024-01-16T15:10:03.024836+0000","last_fullsized":"2024-01-16T15:10:03.024836+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:19:38.517180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949008+0000","last_change":"2024-01-16T15:10:01.940219+0000","last_active":"2024-01-16T15:10:02.949008+0000","last_peered":"2024-01-16T15:10:02.949008+0000","last_clean":"2024-01-16T15:10:02.949008+0000","last_became_active":"2024-01-16T15:10:01.939669+0000","last_became_peered":"2024-01-16T15:10:01.939669+0000","last_unstale":"2024-01-16T15:10:02.949008+0000","last_undegraded":"2024-01-16T15:10:02.949008+0000","last_fullsized":"2024-01-16T15:10:02.949008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:45.239405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.026972+0000","last_change":"2024-01-16T15:10:01.956793+0000","last_active":"2024-01-16T15:10:03.026972+0000","last_peered":"2024-01-16T15:10:03.026972+0000","last_clean":"2024-01-16T15:10:03.026972+0000","last_became_active":"2024-01-16T15:10:01.956466+0000","last_became_peered":"2024-01-16T15:10:01.956466+0000","last_unstale":"2024-01-16T15:10:03.026972+0000","last_undegraded":"2024-01-16T15:10:03.026972+0000","last_fullsized":"2024-01-16T15:10:03.026972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:11.518344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951948+0000","last_change":"2024-01-16T15:10:01.954624+0000","last_active":"2024-01-16T15:10:02.951948+0000","last_peered":"2024-01-16T15:10:02.951948+0000","last_clean":"2024-01-16T15:10:02.951948+0000","last_became_active":"2024-01-16T15:10:01.954317+0000","last_became_peered":"2024-01-16T15:10:01.954317+0000","last_unstale":"2024-01-16T15:10:02.951948+0000","last_undegraded":"2024-01-16T15:10:02.951948+0000","last_fullsized":"2024-01-16T15:10:02.951948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:36.818022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.965695+0000","last_change":"2024-01-16T15:10:00.927347+0000","last_active":"2024-01-16T15:10:01.965695+0000","last_peered":"2024-01-16T15:10:01.965695+0000","last_clean":"2024-01-16T15:10:01.965695+0000","last_became_active":"2024-01-16T15:10:00.927026+0000","last_became_peered":"2024-01-16T15:10:00.927026+0000","last_unstale":"2024-01-16T15:10:01.965695+0000","last_undegraded":"2024-01-16T15:10:01.965695+0000","last_fullsized":"2024-01-16T15:10:01.965695+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:04:55.753436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028264+0000","last_change":"2024-01-16T15:10:01.961552+0000","last_active":"2024-01-16T15:10:03.028264+0000","last_peered":"2024-01-16T15:10:03.028264+0000","last_clean":"2024-01-16T15:10:03.028264+0000","last_became_active":"2024-01-16T15:10:01.961249+0000","last_became_peered":"2024-01-16T15:10:01.961249+0000","last_unstale":"2024-01-16T15:10:03.028264+0000","last_undegraded":"2024-01-16T15:10:03.028264+0000","last_fullsized":"2024-01-16T15:10:03.028264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:07.487475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.952102+0000","last_change":"2024-01-16T15:10:00.932604+0000","last_active":"2024-01-16T15:10:02.952102+0000","last_peered":"2024-01-16T15:10:02.952102+0000","last_clean":"2024-01-16T15:10:02.952102+0000","last_became_active":"2024-01-16T15:10:00.932209+0000","last_became_peered":"2024-01-16T15:10:00.932209+0000","last_unstale":"2024-01-16T15:10:02.952102+0000","last_undegraded":"2024-01-16T15:10:02.952102+0000","last_fullsized":"2024-01-16T15:10:02.952102+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:50.641157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027711+0000","last_change":"2024-01-16T15:10:01.960642+0000","last_active":"2024-01-16T15:10:03.027711+0000","last_peered":"2024-01-16T15:10:03.027711+0000","last_clean":"2024-01-16T15:10:03.027711+0000","last_became_active":"2024-01-16T15:10:01.959928+0000","last_became_peered":"2024-01-16T15:10:01.959928+0000","last_unstale":"2024-01-16T15:10:03.027711+0000","last_undegraded":"2024-01-16T15:10:03.027711+0000","last_fullsized":"2024-01-16T15:10:03.027711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:34.121771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998686+0000","last_change":"2024-01-16T15:10:00.922840+0000","last_active":"2024-01-16T15:10:02.998686+0000","last_peered":"2024-01-16T15:10:02.998686+0000","last_clean":"2024-01-16T15:10:02.998686+0000","last_became_active":"2024-01-16T15:10:00.922548+0000","last_became_peered":"2024-01-16T15:10:00.922548+0000","last_unstale":"2024-01-16T15:10:02.998686+0000","last_undegraded":"2024-01-16T15:10:02.998686+0000","last_fullsized":"2024-01-16T15:10:02.998686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:15:59.219101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027714+0000","last_change":"2024-01-16T15:10:01.948460+0000","last_active":"2024-01-16T15:10:03.027714+0000","last_peered":"2024-01-16T15:10:03.027714+0000","last_clean":"2024-01-16T15:10:03.027714+0000","last_became_active":"2024-01-16T15:10:01.948176+0000","last_became_peered":"2024-01-16T15:10:01.948176+0000","last_unstale":"2024-01-16T15:10:03.027714+0000","last_undegraded":"2024-01-16T15:10:03.027714+0000","last_fullsized":"2024-01-16T15:10:03.027714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:59.680766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998433+0000","last_change":"2024-01-16T15:10:00.924755+0000","last_active":"2024-01-16T15:10:02.998433+0000","last_peered":"2024-01-16T15:10:02.998433+0000","last_clean":"2024-01-16T15:10:02.998433+0000","last_became_active":"2024-01-16T15:10:00.924265+0000","last_became_peered":"2024-01-16T15:10:00.924265+0000","last_unstale":"2024-01-16T15:10:02.998433+0000","last_undegraded":"2024-01-16T15:10:02.998433+0000","last_fullsized":"2024-01-16T15:10:02.998433+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:10:30.167886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024712+0000","last_change":"2024-01-16T15:10:01.950726+0000","last_active":"2024-01-16T15:10:03.024712+0000","last_peered":"2024-01-16T15:10:03.024712+0000","last_clean":"2024-01-16T15:10:03.024712+0000","last_became_active":"2024-01-16T15:10:01.949487+0000","last_became_peered":"2024-01-16T15:10:01.949487+0000","last_unstale":"2024-01-16T15:10:03.024712+0000","last_undegraded":"2024-01-16T15:10:03.024712+0000","last_fullsized":"2024-01-16T15:10:03.024712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:10.822807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951400+0000","last_change":"2024-01-16T15:10:00.930044+0000","last_active":"2024-01-16T15:10:02.951400+0000","last_peered":"2024-01-16T15:10:02.951400+0000","last_clean":"2024-01-16T15:10:02.951400+0000","last_became_active":"2024-01-16T15:10:00.929061+0000","last_became_peered":"2024-01-16T15:10:00.929061+0000","last_unstale":"2024-01-16T15:10:02.951400+0000","last_undegraded":"2024-01-16T15:10:02.951400+0000","last_fullsized":"2024-01-16T15:10:02.951400+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:59.358782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951014+0000","last_change":"2024-01-16T15:10:01.945853+0000","last_active":"2024-01-16T15:10:02.951014+0000","last_peered":"2024-01-16T15:10:02.951014+0000","last_clean":"2024-01-16T15:10:02.951014+0000","last_became_active":"2024-01-16T15:10:01.945544+0000","last_became_peered":"2024-01-16T15:10:01.945544+0000","last_unstale":"2024-01-16T15:10:02.951014+0000","last_undegraded":"2024-01-16T15:10:02.951014+0000","last_fullsized":"2024-01-16T15:10:02.951014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:32:00.871458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949901+0000","last_change":"2024-01-16T15:10:00.925134+0000","last_active":"2024-01-16T15:10:02.949901+0000","last_peered":"2024-01-16T15:10:02.949901+0000","last_clean":"2024-01-16T15:10:02.949901+0000","last_became_active":"2024-01-16T15:10:00.924856+0000","last_became_peered":"2024-01-16T15:10:00.924856+0000","last_unstale":"2024-01-16T15:10:02.949901+0000","last_undegraded":"2024-01-16T15:10:02.949901+0000","last_fullsized":"2024-01-16T15:10:02.949901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:55.025664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.026254+0000","last_change":"2024-01-16T15:10:01.934746+0000","last_active":"2024-01-16T15:10:03.026254+0000","last_peered":"2024-01-16T15:10:03.026254+0000","last_clean":"2024-01-16T15:10:03.026254+0000","last_became_active":"2024-01-16T15:10:01.934279+0000","last_became_peered":"2024-01-16T15:10:01.934279+0000","last_unstale":"2024-01-16T15:10:03.026254+0000","last_undegraded":"2024-01-16T15:10:03.026254+0000","last_fullsized":"2024-01-16T15:10:03.026254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:12.798770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998221+0000","last_change":"2024-01-16T15:10:00.920218+0000","last_active":"2024-01-16T15:10:02.998221+0000","last_peered":"2024-01-16T15:10:02.998221+0000","last_clean":"2024-01-16T15:10:02.998221+0000","last_became_active":"2024-01-16T15:10:00.919605+0000","last_became_peered":"2024-01-16T15:10:00.919605+0000","last_unstale":"2024-01-16T15:10:02.998221+0000","last_undegraded":"2024-01-16T15:10:02.998221+0000","last_fullsized":"2024-01-16T15:10:02.998221+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:17:16.029879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024640+0000","last_change":"2024-01-16T15:10:01.938992+0000","last_active":"2024-01-16T15:10:03.024640+0000","last_peered":"2024-01-16T15:10:03.024640+0000","last_clean":"2024-01-16T15:10:03.024640+0000","last_became_active":"2024-01-16T15:10:01.938709+0000","last_became_peered":"2024-01-16T15:10:01.938709+0000","last_unstale":"2024-01-16T15:10:03.024640+0000","last_undegraded":"2024-01-16T15:10:03.024640+0000","last_fullsized":"2024-01-16T15:10:03.024640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:34.996890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.942102+0000","last_change":"2024-01-16T15:10:00.931031+0000","last_active":"2024-01-16T15:10:01.942102+0000","last_peered":"2024-01-16T15:10:01.942102+0000","last_clean":"2024-01-16T15:10:01.942102+0000","last_became_active":"2024-01-16T15:10:00.930761+0000","last_became_peered":"2024-01-16T15:10:00.930761+0000","last_unstale":"2024-01-16T15:10:01.942102+0000","last_undegraded":"2024-01-16T15:10:01.942102+0000","last_fullsized":"2024-01-16T15:10:01.942102+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:16.528701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.025194+0000","last_change":"2024-01-16T15:10:01.955285+0000","last_active":"2024-01-16T15:10:03.025194+0000","last_peered":"2024-01-16T15:10:03.025194+0000","last_clean":"2024-01-16T15:10:03.025194+0000","last_became_active":"2024-01-16T15:10:01.954954+0000","last_became_peered":"2024-01-16T15:10:01.954954+0000","last_unstale":"2024-01-16T15:10:03.025194+0000","last_undegraded":"2024-01-16T15:10:03.025194+0000","last_fullsized":"2024-01-16T15:10:03.025194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:02.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.947446+0000","last_change":"2024-01-16T15:10:00.931837+0000","last_active":"2024-01-16T15:10:01.947446+0000","last_peered":"2024-01-16T15:10:01.947446+0000","last_clean":"2024-01-16T15:10:01.947446+0000","last_became_active":"2024-01-16T15:10:00.931534+0000","last_became_peered":"2024-01-16T15:10:00.931534+0000","last_unstale":"2024-01-16T15:10:01.947446+0000","last_undegraded":"2024-01-16T15:10:01.947446+0000","last_fullsized":"2024-01-16T15:10:01.947446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:09:29.742770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950840+0000","last_change":"2024-01-16T15:10:01.947485+0000","last_active":"2024-01-16T15:10:02.950840+0000","last_peered":"2024-01-16T15:10:02.950840+0000","last_clean":"2024-01-16T15:10:02.950840+0000","last_became_active":"2024-01-16T15:10:01.947158+0000","last_became_peered":"2024-01-16T15:10:01.947158+0000","last_unstale":"2024-01-16T15:10:02.950840+0000","last_undegraded":"2024-01-16T15:10:02.950840+0000","last_fullsized":"2024-01-16T15:10:02.950840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:41.946530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.941623+0000","last_change":"2024-01-16T15:10:00.927277+0000","last_active":"2024-01-16T15:10:01.941623+0000","last_peered":"2024-01-16T15:10:01.941623+0000","last_clean":"2024-01-16T15:10:01.941623+0000","last_became_active":"2024-01-16T15:10:00.926773+0000","last_became_peered":"2024-01-16T15:10:00.926773+0000","last_unstale":"2024-01-16T15:10:01.941623+0000","last_undegraded":"2024-01-16T15:10:01.941623+0000","last_fullsized":"2024-01-16T15:10:01.941623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:43.337705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998338+0000","last_change":"2024-01-16T15:10:01.958890+0000","last_active":"2024-01-16T15:10:02.998338+0000","last_peered":"2024-01-16T15:10:02.998338+0000","last_clean":"2024-01-16T15:10:02.998338+0000","last_became_active":"2024-01-16T15:10:01.958565+0000","last_became_peered":"2024-01-16T15:10:01.958565+0000","last_unstale":"2024-01-16T15:10:02.998338+0000","last_undegraded":"2024-01-16T15:10:02.998338+0000","last_fullsized":"2024-01-16T15:10:02.998338+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:28.537570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.964793+0000","last_change":"2024-01-16T15:10:00.928782+0000","last_active":"2024-01-16T15:10:01.964793+0000","last_peered":"2024-01-16T15:10:01.964793+0000","last_clean":"2024-01-16T15:10:01.964793+0000","last_became_active":"2024-01-16T15:10:00.928506+0000","last_became_peered":"2024-01-16T15:10:00.928506+0000","last_unstale":"2024-01-16T15:10:01.964793+0000","last_undegraded":"2024-01-16T15:10:01.964793+0000","last_fullsized":"2024-01-16T15:10:01.964793+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:36.622514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951605+0000","last_change":"2024-01-16T15:10:01.943877+0000","last_active":"2024-01-16T15:10:02.951605+0000","last_peered":"2024-01-16T15:10:02.951605+0000","last_clean":"2024-01-16T15:10:02.951605+0000","last_became_active":"2024-01-16T15:10:01.943572+0000","last_became_peered":"2024-01-16T15:10:01.943572+0000","last_unstale":"2024-01-16T15:10:02.951605+0000","last_undegraded":"2024-01-16T15:10:02.951605+0000","last_fullsized":"2024-01-16T15:10:02.951605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:48.782361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949971+0000","last_change":"2024-01-16T15:10:00.927885+0000","last_active":"2024-01-16T15:10:02.949971+0000","last_peered":"2024-01-16T15:10:02.949971+0000","last_clean":"2024-01-16T15:10:02.949971+0000","last_became_active":"2024-01-16T15:10:00.927560+0000","last_became_peered":"2024-01-16T15:10:00.927560+0000","last_unstale":"2024-01-16T15:10:02.949971+0000","last_undegraded":"2024-01-16T15:10:02.949971+0000","last_fullsized":"2024-01-16T15:10:02.949971+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:44.854412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998339+0000","last_change":"2024-01-16T15:10:01.943627+0000","last_active":"2024-01-16T15:10:02.998339+0000","last_peered":"2024-01-16T15:10:02.998339+0000","last_clean":"2024-01-16T15:10:02.998339+0000","last_became_active":"2024-01-16T15:10:01.943360+0000","last_became_peered":"2024-01-16T15:10:01.943360+0000","last_unstale":"2024-01-16T15:10:02.998339+0000","last_undegraded":"2024-01-16T15:10:02.998339+0000","last_fullsized":"2024-01-16T15:10:02.998339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:01.056572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.964032+0000","last_change":"2024-01-16T15:10:00.925398+0000","last_active":"2024-01-16T15:10:01.964032+0000","last_peered":"2024-01-16T15:10:01.964032+0000","last_clean":"2024-01-16T15:10:01.964032+0000","last_became_active":"2024-01-16T15:10:00.925139+0000","last_became_peered":"2024-01-16T15:10:00.925139+0000","last_unstale":"2024-01-16T15:10:01.964032+0000","last_undegraded":"2024-01-16T15:10:01.964032+0000","last_fullsized":"2024-01-16T15:10:01.964032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:57:58.356584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027484+0000","last_change":"2024-01-16T15:10:01.939061+0000","last_active":"2024-01-16T15:10:03.027484+0000","last_peered":"2024-01-16T15:10:03.027484+0000","last_clean":"2024-01-16T15:10:03.027484+0000","last_became_active":"2024-01-16T15:10:01.935630+0000","last_became_peered":"2024-01-16T15:10:01.935630+0000","last_unstale":"2024-01-16T15:10:03.027484+0000","last_undegraded":"2024-01-16T15:10:03.027484+0000","last_fullsized":"2024-01-16T15:10:03.027484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:32.383755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024782+0000","last_change":"2024-01-16T15:10:00.933437+0000","last_active":"2024-01-16T15:10:03.024782+0000","last_peered":"2024-01-16T15:10:03.024782+0000","last_clean":"2024-01-16T15:10:03.024782+0000","last_became_active":"2024-01-16T15:10:00.932597+0000","last_became_peered":"2024-01-16T15:10:00.932597+0000","last_unstale":"2024-01-16T15:10:03.024782+0000","last_undegraded":"2024-01-16T15:10:03.024782+0000","last_fullsized":"2024-01-16T15:10:03.024782+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:01.708282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024407+0000","last_change":"2024-01-16T15:10:01.949270+0000","last_active":"2024-01-16T15:10:03.024407+0000","last_peered":"2024-01-16T15:10:03.024407+0000","last_clean":"2024-01-16T15:10:03.024407+0000","last_became_active":"2024-01-16T15:10:01.948972+0000","last_became_peered":"2024-01-16T15:10:01.948972+0000","last_unstale":"2024-01-16T15:10:03.024407+0000","last_undegraded":"2024-01-16T15:10:03.024407+0000","last_fullsized":"2024-01-16T15:10:03.024407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:20:30.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949596+0000","last_change":"2024-01-16T15:10:00.919867+0000","last_active":"2024-01-16T15:10:02.949596+0000","last_peered":"2024-01-16T15:10:02.949596+0000","last_clean":"2024-01-16T15:10:02.949596+0000","last_became_active":"2024-01-16T15:10:00.919523+0000","last_became_peered":"2024-01-16T15:10:00.919523+0000","last_unstale":"2024-01-16T15:10:02.949596+0000","last_undegraded":"2024-01-16T15:10:02.949596+0000","last_fullsized":"2024-01-16T15:10:02.949596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:44.225208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024326+0000","last_change":"2024-01-16T15:10:01.963425+0000","last_active":"2024-01-16T15:10:03.024326+0000","last_peered":"2024-01-16T15:10:03.024326+0000","last_clean":"2024-01-16T15:10:03.024326+0000","last_became_active":"2024-01-16T15:10:01.963164+0000","last_became_peered":"2024-01-16T15:10:01.963164+0000","last_unstale":"2024-01-16T15:10:03.024326+0000","last_undegraded":"2024-01-16T15:10:03.024326+0000","last_fullsized":"2024-01-16T15:10:03.024326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:34:17.413492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998038+0000","last_change":"2024-01-16T15:10:00.922876+0000","last_active":"2024-01-16T15:10:02.998038+0000","last_peered":"2024-01-16T15:10:02.998038+0000","last_clean":"2024-01-16T15:10:02.998038+0000","last_became_active":"2024-01-16T15:10:00.922592+0000","last_became_peered":"2024-01-16T15:10:00.922592+0000","last_unstale":"2024-01-16T15:10:02.998038+0000","last_undegraded":"2024-01-16T15:10:02.998038+0000","last_fullsized":"2024-01-16T15:10:02.998038+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:06.653597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028287+0000","last_change":"2024-01-16T15:10:01.945033+0000","last_active":"2024-01-16T15:10:03.028287+0000","last_peered":"2024-01-16T15:10:03.028287+0000","last_clean":"2024-01-16T15:10:03.028287+0000","last_became_active":"2024-01-16T15:10:01.944753+0000","last_became_peered":"2024-01-16T15:10:01.944753+0000","last_unstale":"2024-01-16T15:10:03.028287+0000","last_undegraded":"2024-01-16T15:10:03.028287+0000","last_fullsized":"2024-01-16T15:10:03.028287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:34.185246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.959829+0000","last_change":"2024-01-16T15:10:00.927199+0000","last_active":"2024-01-16T15:10:01.959829+0000","last_peered":"2024-01-16T15:10:01.959829+0000","last_clean":"2024-01-16T15:10:01.959829+0000","last_became_active":"2024-01-16T15:10:00.926913+0000","last_became_peered":"2024-01-16T15:10:00.926913+0000","last_unstale":"2024-01-16T15:10:01.959829+0000","last_undegraded":"2024-01-16T15:10:01.959829+0000","last_fullsized":"2024-01-16T15:10:01.959829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:11:56.156114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028511+0000","last_change":"2024-01-16T15:10:01.959342+0000","last_active":"2024-01-16T15:10:03.028511+0000","last_peered":"2024-01-16T15:10:03.028511+0000","last_clean":"2024-01-16T15:10:03.028511+0000","last_became_active":"2024-01-16T15:10:01.958344+0000","last_became_peered":"2024-01-16T15:10:01.958344+0000","last_unstale":"2024-01-16T15:10:03.028511+0000","last_undegraded":"2024-01-16T15:10:03.028511+0000","last_fullsized":"2024-01-16T15:10:03.028511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:16:00.233298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998691+0000","last_change":"2024-01-16T15:10:00.921808+0000","last_active":"2024-01-16T15:10:02.998691+0000","last_peered":"2024-01-16T15:10:02.998691+0000","last_clean":"2024-01-16T15:10:02.998691+0000","last_became_active":"2024-01-16T15:10:00.921529+0000","last_became_peered":"2024-01-16T15:10:00.921529+0000","last_unstale":"2024-01-16T15:10:02.998691+0000","last_undegraded":"2024-01-16T15:10:02.998691+0000","last_fullsized":"2024-01-16T15:10:02.998691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:16.867223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951623+0000","last_change":"2024-01-16T15:10:01.958493+0000","last_active":"2024-01-16T15:10:02.951623+0000","last_peered":"2024-01-16T15:10:02.951623+0000","last_clean":"2024-01-16T15:10:02.951623+0000","last_became_active":"2024-01-16T15:10:01.956765+0000","last_became_peered":"2024-01-16T15:10:01.956765+0000","last_unstale":"2024-01-16T15:10:02.951623+0000","last_undegraded":"2024-01-16T15:10:02.951623+0000","last_fullsized":"2024-01-16T15:10:02.951623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:41.946530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998465+0000","last_change":"2024-01-16T15:10:00.918587+0000","last_active":"2024-01-16T15:10:02.998465+0000","last_peered":"2024-01-16T15:10:02.998465+0000","last_clean":"2024-01-16T15:10:02.998465+0000","last_became_active":"2024-01-16T15:10:00.918309+0000","last_became_peered":"2024-01-16T15:10:00.918309+0000","last_unstale":"2024-01-16T15:10:02.998465+0000","last_undegraded":"2024-01-16T15:10:02.998465+0000","last_fullsized":"2024-01-16T15:10:02.998465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:34.228418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949882+0000","last_change":"2024-01-16T15:10:01.952227+0000","last_active":"2024-01-16T15:10:02.949882+0000","last_peered":"2024-01-16T15:10:02.949882+0000","last_clean":"2024-01-16T15:10:02.949882+0000","last_became_active":"2024-01-16T15:10:01.951964+0000","last_became_peered":"2024-01-16T15:10:01.951964+0000","last_unstale":"2024-01-16T15:10:02.949882+0000","last_undegraded":"2024-01-16T15:10:02.949882+0000","last_fullsized":"2024-01-16T15:10:02.949882+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:00.914138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024929+0000","last_change":"2024-01-16T15:10:00.926102+0000","last_active":"2024-01-16T15:10:03.024929+0000","last_peered":"2024-01-16T15:10:03.024929+0000","last_clean":"2024-01-16T15:10:03.024929+0000","last_became_active":"2024-01-16T15:10:00.924736+0000","last_became_peered":"2024-01-16T15:10:00.924736+0000","last_unstale":"2024-01-16T15:10:03.024929+0000","last_undegraded":"2024-01-16T15:10:03.024929+0000","last_fullsized":"2024-01-16T15:10:03.024929+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:44.802601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949504+0000","last_change":"2024-01-16T15:10:01.952242+0000","last_active":"2024-01-16T15:10:02.949504+0000","last_peered":"2024-01-16T15:10:02.949504+0000","last_clean":"2024-01-16T15:10:02.949504+0000","last_became_active":"2024-01-16T15:10:01.951960+0000","last_became_peered":"2024-01-16T15:10:01.951960+0000","last_unstale":"2024-01-16T15:10:02.949504+0000","last_undegraded":"2024-01-16T15:10:02.949504+0000","last_fullsized":"2024-01-16T15:10:02.949504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:20.817578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950095+0000","last_change":"2024-01-16T15:10:00.924186+0000","last_active":"2024-01-16T15:10:02.950095+0000","last_peered":"2024-01-16T15:10:02.950095+0000","last_clean":"2024-01-16T15:10:02.950095+0000","last_became_active":"2024-01-16T15:10:00.921679+0000","last_became_peered":"2024-01-16T15:10:00.921679+0000","last_unstale":"2024-01-16T15:10:02.950095+0000","last_undegraded":"2024-01-16T15:10:02.950095+0000","last_fullsized":"2024-01-16T15:10:02.950095+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:33.976859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948945+0000","last_change":"2024-01-16T15:10:01.945338+0000","last_active":"2024-01-16T15:10:02.948945+0000","last_peered":"2024-01-16T15:10:02.948945+0000","last_clean":"2024-01-16T15:10:02.948945+0000","last_became_active":"2024-01-16T15:10:01.943420+0000","last_became_peered":"2024-01-16T15:10:01.943420+0000","last_unstale":"2024-01-16T15:10:02.948945+0000","last_undegraded":"2024-01-16T15:10:02.948945+0000","last_fullsized":"2024-01-16T15:10:02.948945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:58:55.577512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.949102+0000","last_change":"2024-01-16T15:10:00.931391+0000","last_active":"2024-01-16T15:10:02.949102+0000","last_peered":"2024-01-16T15:10:02.949102+0000","last_clean":"2024-01-16T15:10:02.949102+0000","last_became_active":"2024-01-16T15:10:00.931033+0000","last_became_peered":"2024-01-16T15:10:00.931033+0000","last_unstale":"2024-01-16T15:10:02.949102+0000","last_undegraded":"2024-01-16T15:10:02.949102+0000","last_fullsized":"2024-01-16T15:10:02.949102+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:17:21.907268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950378+0000","last_change":"2024-01-16T15:10:01.960465+0000","last_active":"2024-01-16T15:10:02.950378+0000","last_peered":"2024-01-16T15:10:02.950378+0000","last_clean":"2024-01-16T15:10:02.950378+0000","last_became_active":"2024-01-16T15:10:01.960151+0000","last_became_peered":"2024-01-16T15:10:01.960151+0000","last_unstale":"2024-01-16T15:10:02.950378+0000","last_undegraded":"2024-01-16T15:10:02.950378+0000","last_fullsized":"2024-01-16T15:10:02.950378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:18.268867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.024368+0000","last_change":"2024-01-16T15:10:00.922555+0000","last_active":"2024-01-16T15:10:03.024368+0000","last_peered":"2024-01-16T15:10:03.024368+0000","last_clean":"2024-01-16T15:10:03.024368+0000","last_became_active":"2024-01-16T15:10:00.922281+0000","last_became_peered":"2024-01-16T15:10:00.922281+0000","last_unstale":"2024-01-16T15:10:03.024368+0000","last_undegraded":"2024-01-16T15:10:03.024368+0000","last_fullsized":"2024-01-16T15:10:03.024368+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:45.551949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.952161+0000","last_change":"2024-01-16T15:10:01.946100+0000","last_active":"2024-01-16T15:10:02.952161+0000","last_peered":"2024-01-16T15:10:02.952161+0000","last_clean":"2024-01-16T15:10:02.952161+0000","last_became_active":"2024-01-16T15:10:01.945011+0000","last_became_peered":"2024-01-16T15:10:01.945011+0000","last_unstale":"2024-01-16T15:10:02.952161+0000","last_undegraded":"2024-01-16T15:10:02.952161+0000","last_fullsized":"2024-01-16T15:10:02.952161+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:27.172011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.951994+0000","last_change":"2024-01-16T15:10:00.930451+0000","last_active":"2024-01-16T15:10:01.951994+0000","last_peered":"2024-01-16T15:10:01.951994+0000","last_clean":"2024-01-16T15:10:01.951994+0000","last_became_active":"2024-01-16T15:10:00.930176+0000","last_became_peered":"2024-01-16T15:10:00.930176+0000","last_unstale":"2024-01-16T15:10:01.951994+0000","last_undegraded":"2024-01-16T15:10:01.951994+0000","last_fullsized":"2024-01-16T15:10:01.951994+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:45.099047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028623+0000","last_change":"2024-01-16T15:10:01.954649+0000","last_active":"2024-01-16T15:10:03.028623+0000","last_peered":"2024-01-16T15:10:03.028623+0000","last_clean":"2024-01-16T15:10:03.028623+0000","last_became_active":"2024-01-16T15:10:01.952569+0000","last_became_peered":"2024-01-16T15:10:01.952569+0000","last_unstale":"2024-01-16T15:10:03.028623+0000","last_undegraded":"2024-01-16T15:10:03.028623+0000","last_fullsized":"2024-01-16T15:10:03.028623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:16.206515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.938760+0000","last_change":"2024-01-16T15:10:00.926840+0000","last_active":"2024-01-16T15:10:01.938760+0000","last_peered":"2024-01-16T15:10:01.938760+0000","last_clean":"2024-01-16T15:10:01.938760+0000","last_became_active":"2024-01-16T15:10:00.926543+0000","last_became_peered":"2024-01-16T15:10:00.926543+0000","last_unstale":"2024-01-16T15:10:01.938760+0000","last_undegraded":"2024-01-16T15:10:01.938760+0000","last_fullsized":"2024-01-16T15:10:01.938760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:24.253271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.028203+0000","last_change":"2024-01-16T15:10:01.960815+0000","last_active":"2024-01-16T15:10:03.028203+0000","last_peered":"2024-01-16T15:10:03.028203+0000","last_clean":"2024-01-16T15:10:03.028203+0000","last_became_active":"2024-01-16T15:10:01.959675+0000","last_became_peered":"2024-01-16T15:10:01.959675+0000","last_unstale":"2024-01-16T15:10:03.028203+0000","last_undegraded":"2024-01-16T15:10:03.028203+0000","last_fullsized":"2024-01-16T15:10:03.028203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:39.644673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.937681+0000","last_change":"2024-01-16T15:10:00.927135+0000","last_active":"2024-01-16T15:10:01.937681+0000","last_peered":"2024-01-16T15:10:01.937681+0000","last_clean":"2024-01-16T15:10:01.937681+0000","last_became_active":"2024-01-16T15:10:00.926835+0000","last_became_peered":"2024-01-16T15:10:00.926835+0000","last_unstale":"2024-01-16T15:10:01.937681+0000","last_undegraded":"2024-01-16T15:10:01.937681+0000","last_fullsized":"2024-01-16T15:10:01.937681+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:41.675007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.951971+0000","last_change":"2024-01-16T15:10:01.943260+0000","last_active":"2024-01-16T15:10:02.951971+0000","last_peered":"2024-01-16T15:10:02.951971+0000","last_clean":"2024-01-16T15:10:02.951971+0000","last_became_active":"2024-01-16T15:10:01.940795+0000","last_became_peered":"2024-01-16T15:10:01.940795+0000","last_unstale":"2024-01-16T15:10:02.951971+0000","last_undegraded":"2024-01-16T15:10:02.951971+0000","last_fullsized":"2024-01-16T15:10:02.951971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:12.798770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.943033+0000","last_change":"2024-01-16T15:10:00.928183+0000","last_active":"2024-01-16T15:10:01.943033+0000","last_peered":"2024-01-16T15:10:01.943033+0000","last_clean":"2024-01-16T15:10:01.943033+0000","last_became_active":"2024-01-16T15:10:00.927668+0000","last_became_peered":"2024-01-16T15:10:00.927668+0000","last_unstale":"2024-01-16T15:10:01.943033+0000","last_undegraded":"2024-01-16T15:10:01.943033+0000","last_fullsized":"2024-01-16T15:10:01.943033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:29.861520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950870+0000","last_change":"2024-01-16T15:10:01.951549+0000","last_active":"2024-01-16T15:10:02.950870+0000","last_peered":"2024-01-16T15:10:02.950870+0000","last_clean":"2024-01-16T15:10:02.950870+0000","last_became_active":"2024-01-16T15:10:01.951269+0000","last_became_peered":"2024-01-16T15:10:01.951269+0000","last_unstale":"2024-01-16T15:10:02.950870+0000","last_undegraded":"2024-01-16T15:10:02.950870+0000","last_fullsized":"2024-01-16T15:10:02.950870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:36.818022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998526+0000","last_change":"2024-01-16T15:10:00.916165+0000","last_active":"2024-01-16T15:10:02.998526+0000","last_peered":"2024-01-16T15:10:02.998526+0000","last_clean":"2024-01-16T15:10:02.998526+0000","last_became_active":"2024-01-16T15:10:00.915439+0000","last_became_peered":"2024-01-16T15:10:00.915439+0000","last_unstale":"2024-01-16T15:10:02.998526+0000","last_undegraded":"2024-01-16T15:10:02.998526+0000","last_fullsized":"2024-01-16T15:10:02.998526+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:35.968089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.998389+0000","last_change":"2024-01-16T15:10:01.946503+0000","last_active":"2024-01-16T15:10:02.998389+0000","last_peered":"2024-01-16T15:10:02.998389+0000","last_clean":"2024-01-16T15:10:02.998389+0000","last_became_active":"2024-01-16T15:10:01.946233+0000","last_became_peered":"2024-01-16T15:10:01.946233+0000","last_unstale":"2024-01-16T15:10:02.998389+0000","last_undegraded":"2024-01-16T15:10:02.998389+0000","last_fullsized":"2024-01-16T15:10:02.998389+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:14.532732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.950125+0000","last_change":"2024-01-16T15:10:00.925120+0000","last_active":"2024-01-16T15:10:02.950125+0000","last_peered":"2024-01-16T15:10:02.950125+0000","last_clean":"2024-01-16T15:10:02.950125+0000","last_became_active":"2024-01-16T15:10:00.924806+0000","last_became_peered":"2024-01-16T15:10:00.924806+0000","last_unstale":"2024-01-16T15:10:02.950125+0000","last_undegraded":"2024-01-16T15:10:02.950125+0000","last_fullsized":"2024-01-16T15:10:02.950125+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:08.289373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.026471+0000","last_change":"2024-01-16T15:10:01.953330+0000","last_active":"2024-01-16T15:10:03.026471+0000","last_peered":"2024-01-16T15:10:03.026471+0000","last_clean":"2024-01-16T15:10:03.026471+0000","last_became_active":"2024-01-16T15:10:01.953034+0000","last_became_peered":"2024-01-16T15:10:01.953034+0000","last_unstale":"2024-01-16T15:10:03.026471+0000","last_undegraded":"2024-01-16T15:10:03.026471+0000","last_fullsized":"2024-01-16T15:10:03.026471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:27.172011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.946448+0000","last_change":"2024-01-16T15:10:00.927996+0000","last_active":"2024-01-16T15:10:01.946448+0000","last_peered":"2024-01-16T15:10:01.946448+0000","last_clean":"2024-01-16T15:10:01.946448+0000","last_became_active":"2024-01-16T15:10:00.927654+0000","last_became_peered":"2024-01-16T15:10:00.927654+0000","last_unstale":"2024-01-16T15:10:01.946448+0000","last_undegraded":"2024-01-16T15:10:01.946448+0000","last_fullsized":"2024-01-16T15:10:01.946448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:19.363753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.027717+0000","last_change":"2024-01-16T15:10:01.957550+0000","last_active":"2024-01-16T15:10:03.027717+0000","last_peered":"2024-01-16T15:10:03.027717+0000","last_clean":"2024-01-16T15:10:03.027717+0000","last_became_active":"2024-01-16T15:10:01.957225+0000","last_became_peered":"2024-01-16T15:10:01.957225+0000","last_unstale":"2024-01-16T15:10:03.027717+0000","last_undegraded":"2024-01-16T15:10:03.027717+0000","last_fullsized":"2024-01-16T15:10:03.027717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:43.359511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:02.948847+0000","last_change":"2024-01-16T15:10:00.923928+0000","last_active":"2024-01-16T15:10:02.948847+0000","last_peered":"2024-01-16T15:10:02.948847+0000","last_clean":"2024-01-16T15:10:02.948847+0000","last_became_active":"2024-01-16T15:10:00.923627+0000","last_became_peered":"2024-01-16T15:10:00.923627+0000","last_unstale":"2024-01-16T15:10:02.948847+0000","last_undegraded":"2024-01-16T15:10:02.948847+0000","last_fullsized":"2024-01-16T15:10:02.948847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:26.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-16T15:10:03.025178+0000","last_change":"2024-01-16T15:10:01.956357+0000","last_active":"2024-01-16T15:10:03.025178+0000","last_peered":"2024-01-16T15:10:03.025178+0000","last_clean":"2024-01-16T15:10:03.025178+0000","last_became_active":"2024-01-16T15:10:01.956088+0000","last_became_peered":"2024-01-16T15:10:01.956088+0000","last_unstale":"2024-01-16T15:10:03.025178+0000","last_undegraded":"2024-01-16T15:10:03.025178+0000","last_fullsized":"2024-01-16T15:10:03.025178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:10:00.887744+0000","last_clean_scrub_stamp":"2024-01-16T15:10:00.887744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:27.525499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.959426+0000","last_change":"2024-01-16T15:10:00.929509+0000","last_active":"2024-01-16T15:10:01.959426+0000","last_peered":"2024-01-16T15:10:01.959426+0000","last_clean":"2024-01-16T15:10:01.959426+0000","last_became_active":"2024-01-16T15:10:00.929186+0000","last_became_peered":"2024-01-16T15:10:00.929186+0000","last_unstale":"2024-01-16T15:10:01.959426+0000","last_undegraded":"2024-01-16T15:10:01.959426+0000","last_fullsized":"2024-01-16T15:10:01.959426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:09:59.873546+0000","last_clean_scrub_stamp":"2024-01-16T15:09:59.873546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:19.803381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":232,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-16T15:10:01.928347+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:10:01.928347+0000","last_peered":"2024-01-16T15:10:01.928347+0000","last_clean":"2024-01-16T15:10:01.928347+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:10:01.928347+0000","last_undegraded":"2024-01-16T15:10:01.928347+0000","last_fullsized":"2024-01-16T15:10:01.928347+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542304,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68868,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302972,"statfs":{"total":96636764160,"available":96566243328,"internally_reserved":0,"allocated":1687552,"data_stored":1057530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.6439999999999999,"15min":1.452},"min":{"1min":0.67900000000000005,"5min":0.37,"15min":0.37},"max":{"1min":4.0069999999999997,"5min":19.603000000000002,"15min":19.603000000000002},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.597,"15min":1.365},"min":{"1min":0.77600000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.7889999999999997,"5min":24.015000000000001,"15min":24.015000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.5980000000000001,"15min":1.4179999999999999},"min":{"1min":0.59699999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.6070000000000002,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.149},{"interface":"front","average":{"1min":2.016,"5min":1.841,"15min":1.4470000000000001},"min":{"1min":0.65900000000000003,"5min":0.443,"15min":0.443},"max":{"1min":14.592000000000001,"5min":24.07,"15min":24.07},"last":0.89700000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.617,"15min":1.367},"min":{"1min":0.73199999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":4.9000000000000004,"5min":24.108000000000001,"15min":24.108000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.3239999999999998,"5min":1.698,"15min":1.52},"min":{"1min":0.67600000000000005,"5min":0.37,"15min":0.37},"max":{"1min":24.806000000000001,"5min":24.806000000000001,"15min":24.806000000000001},"last":1.3600000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":2.5800000000000001,"5min":2.004,"15min":1.7370000000000001},"min":{"1min":0.91300000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":23.741,"5min":23.741,"15min":23.741},"last":0.67000000000000004},{"interface":"front","average":{"1min":2.7149999999999999,"5min":2.0049999999999999,"15min":1.6679999999999999},"min":{"1min":0.86299999999999999,"5min":0.626,"15min":0.626},"max":{"1min":23.777000000000001,"5min":24.143000000000001,"15min":24.143000000000001},"last":1.0960000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":2.2320000000000002,"5min":2.0750000000000002,"15min":1.6830000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":22.355,"5min":25.637,"15min":25.637},"last":1.5309999999999999},{"interface":"front","average":{"1min":2.3809999999999998,"5min":2.2799999999999998,"15min":1.855},"min":{"1min":0.63600000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":22.271999999999998,"5min":25.594000000000001,"15min":25.594000000000001},"last":1.4770000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":1.9059999999999999,"15min":1.75},"min":{"1min":0.52100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":7.8300000000000001,"5min":24.178999999999998,"15min":29.567},"last":1.306},{"interface":"front","average":{"1min":1.704,"5min":1.8300000000000001,"15min":1.8300000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":7.8540000000000001,"5min":24.538,"15min":27.937000000000001},"last":1.6899999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":2.1579999999999999,"15min":1.9239999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.6389999999999998,"5min":36.826000000000001,"15min":41.206000000000003},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":2.5960000000000001,"15min":2.1589999999999998},"min":{"1min":0.71799999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.6339999999999999,"5min":39.932000000000002,"15min":39.932000000000002},"last":1.5980000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542303,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68292,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303548,"statfs":{"total":96636764160,"available":96566833152,"internally_reserved":0,"allocated":1101824,"data_stored":467786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.712,"15min":1.405},"min":{"1min":0.64600000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4179999999999999,"5min":20.739000000000001,"15min":20.739000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.456,"5min":1.4590000000000001,"15min":1.5960000000000001},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":8.0630000000000006,"5min":15.401,"15min":50.472000000000001},"last":1.3129999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.323,"15min":1.3120000000000001},"min":{"1min":0.78500000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.794,"5min":9.7769999999999992,"15min":10.629},"last":1.155},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.546,"15min":1.4870000000000001},"min":{"1min":0.66100000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":7.2919999999999998,"5min":9.6880000000000006,"15min":9.6880000000000006},"last":0.93400000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.53,"15min":1.607},"min":{"1min":0.63700000000000001,"5min":0.50800000000000001,"15min":0.46899999999999997},"max":{"1min":6.3040000000000003,"5min":15.576000000000001,"15min":50.643999999999998},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.706,"5min":1.4870000000000001,"15min":1.4710000000000001},"min":{"1min":0.73399999999999999,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":8.8919999999999995,"5min":8.8919999999999995,"15min":10.843},"last":0.98899999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.5800000000000001,"15min":1.575},"min":{"1min":0.78900000000000003,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.8640000000000001,"5min":9.7460000000000004,"15min":15.675000000000001},"last":1.224},{"interface":"front","average":{"1min":2.0009999999999999,"5min":1.6870000000000001,"15min":1.7330000000000001},"min":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.52600000000000002},"max":{"1min":9.0960000000000001,"5min":15.504,"15min":50.561},"last":1.4390000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.6200000000000001,"15min":1.8160000000000001},"min":{"1min":0.49299999999999999,"5min":0.48599999999999999,"15min":0.46300000000000002},"max":{"1min":1.9399999999999999,"5min":7.391,"15min":47.488},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.714,"15min":1.9059999999999999},"min":{"1min":0.52900000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":5.3360000000000003,"5min":15.638,"15min":50.381},"last":1.536}]},{"osd":5,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.351,"15min":1.518},"min":{"1min":0.45600000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.9179999999999999,"5min":6.718,"15min":10.654},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.71,"5min":1.899,"15min":1.8109999999999999},"min":{"1min":0.59499999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.9640000000000004,"5min":18.699000000000002,"15min":18.699000000000002},"last":1.4850000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:09:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.8660000000000001,"15min":2.073},"min":{"1min":0.65800000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":5.6609999999999996,"5min":15.702,"15min":52.912999999999997},"last":1.635},{"interface":"front","average":{"1min":2.274,"5min":2.5089999999999999,"15min":2.2869999999999999},"min":{"1min":0.60699999999999998,"5min":0.495,"15min":0.495},"max":{"1min":16.175999999999998,"5min":26.413,"15min":26.413},"last":1.411}]}]},{"osd":5,"up_from":14,"seq":60129542303,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58628,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313212,"statfs":{"total":96636764160,"available":96576729088,"internally_reserved":0,"allocated":1105920,"data_stored":468726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.391,"15min":1.345},"min":{"1min":0.56000000000000005,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.677,"5min":10.56,"15min":13.938000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.4119999999999999,"15min":1.3640000000000001},"min":{"1min":0.73699999999999999,"5min":0.56699999999999995,"15min":0.42299999999999999},"max":{"1min":3.605,"5min":14.379,"15min":16.896000000000001},"last":0.82999999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.482,"15min":1.45},"min":{"1min":0.85299999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6899999999999999,"5min":15.500999999999999,"15min":15.500999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.292,"15min":1.4079999999999999},"min":{"1min":0.64200000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5840000000000001,"5min":6.7720000000000002,"15min":35.328000000000003},"last":1.1519999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.347,"15min":1.506},"min":{"1min":0.68500000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.5980000000000001,"5min":6.9020000000000001,"15min":35.398000000000003},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.5269999999999999,"15min":1.403},"min":{"1min":0.91000000000000003,"5min":0.47999999999999998,"15min":0.38900000000000001},"max":{"1min":1.7609999999999999,"5min":15.4,"15min":15.4},"last":1.048}]},{"osd":3,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.909,"15min":1.7390000000000001},"min":{"1min":1.123,"5min":0.63200000000000001,"15min":0.50900000000000001},"max":{"1min":16.931999999999999,"5min":16.931999999999999,"15min":24.254000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":2.399,"5min":1.788,"15min":1.804},"min":{"1min":0.72299999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":19.111999999999998,"5min":19.111999999999998,"15min":35.268000000000001},"last":1.3080000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.5589999999999999,"15min":1.669},"min":{"1min":0.89600000000000002,"5min":0.42199999999999999,"15min":0.39000000000000001},"max":{"1min":2.0030000000000001,"5min":19.059000000000001,"15min":19.059000000000001},"last":1.333},{"interface":"front","average":{"1min":1.524,"5min":1.667,"15min":1.7909999999999999},"min":{"1min":1.002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.8839999999999999,"5min":28.760000000000002,"15min":28.760000000000002},"last":1.272}]},{"osd":6,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.821,"15min":1.847},"min":{"1min":0.78900000000000003,"5min":0.54100000000000004,"15min":0.499},"max":{"1min":2.0209999999999999,"5min":16.271000000000001,"15min":35.427999999999997},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.97,"15min":1.859},"min":{"1min":0.749,"5min":0.53500000000000003,"15min":0.48199999999999998},"max":{"1min":1.9119999999999999,"5min":16.149999999999999,"15min":16.149999999999999},"last":1.468}]},{"osd":7,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.181,"15min":1.97},"min":{"1min":1.1339999999999999,"5min":0.54300000000000004,"15min":0.41099999999999998},"max":{"1min":12.385999999999999,"5min":29.672999999999998,"15min":29.672999999999998},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.627,"15min":1.8340000000000001},"min":{"1min":0.85699999999999998,"5min":0.45900000000000002,"15min":0.42899999999999999},"max":{"1min":2.1200000000000001,"5min":14.02,"15min":35.363},"last":1.5069999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542304,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64116,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307724,"statfs":{"total":96636764160,"available":96571109376,"internally_reserved":0,"allocated":1089536,"data_stored":467092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.6579999999999999,"15min":1.3859999999999999},"min":{"1min":0.72999999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.645,"5min":15.942,"15min":15.942},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.4319999999999999,"15min":1.417},"min":{"1min":0.89900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5569999999999999,"5min":16.219999999999999,"15min":16.219999999999999},"last":1.851}]},{"osd":1,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.5029999999999999,"15min":1.458},"min":{"1min":1.038,"5min":0.61499999999999999,"15min":0.54100000000000004},"max":{"1min":1.488,"5min":16.280000000000001,"15min":16.280000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.7050000000000001,"15min":1.4650000000000001},"min":{"1min":0.85499999999999998,"5min":0.60699999999999998,"15min":0.60499999999999998},"max":{"1min":1.6830000000000001,"5min":15.888,"15min":15.888},"last":1.597}]},{"osd":2,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.5780000000000001,"15min":1.53},"min":{"1min":0.83199999999999996,"5min":0.55100000000000005,"15min":0.50800000000000001},"max":{"1min":1.4730000000000001,"5min":13.837,"15min":15.986000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.339,"5min":1.9279999999999999,"15min":1.6060000000000001},"min":{"1min":1.0960000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5600000000000001,"5min":17.175000000000001,"15min":17.175000000000001},"last":1.2450000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.859,"15min":1.7869999999999999},"min":{"1min":0.878,"5min":0.60299999999999998,"15min":0.57599999999999996},"max":{"1min":1.6899999999999999,"5min":16.469000000000001,"15min":16.469000000000001},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.6339999999999999,"15min":1.593},"min":{"1min":0.92000000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6259999999999999,"5min":15.089,"15min":21.706},"last":0.95999999999999996}]},{"osd":5,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.8600000000000001,"15min":1.643},"min":{"1min":0.872,"5min":0.67200000000000004,"15min":0.59599999999999997},"max":{"1min":1.6319999999999999,"5min":18.347999999999999,"15min":18.347999999999999},"last":1.768},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.6819999999999999,"15min":1.667},"min":{"1min":0.89100000000000001,"5min":0.53900000000000003,"15min":0.45800000000000002},"max":{"1min":1.665,"5min":22.937000000000001,"15min":22.937000000000001},"last":2.157}]},{"osd":6,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.7110000000000001,"15min":1.6000000000000001},"min":{"1min":0.91800000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.643,"5min":18.033000000000001,"15min":18.033000000000001},"last":1.109},{"interface":"front","average":{"1min":1.474,"5min":2.1680000000000001,"15min":1.8979999999999999},"min":{"1min":1,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.7390000000000001,"5min":17.794,"15min":20.939},"last":1.831}]},{"osd":7,"last update":"Tue Jan 16 15:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.7669999999999999,"15min":1.77},"min":{"1min":1.208,"5min":0.58299999999999996,"15min":0.44500000000000001},"max":{"1min":1.7769999999999999,"5min":16.324999999999999,"15min":16.324999999999999},"last":2.298},{"interface":"front","average":{"1min":1.46,"5min":2.0190000000000001,"15min":1.881},"min":{"1min":1.032,"5min":0.57699999999999996,"15min":0.52200000000000002},"max":{"1min":1.7150000000000001,"5min":12.335000000000001,"15min":20.277999999999999},"last":1.482}]}]},{"osd":3,"up_from":14,"seq":60129542303,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72456,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299384,"statfs":{"total":96636764160,"available":96562569216,"internally_reserved":0,"allocated":1101824,"data_stored":468704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.997,"15min":0.97499999999999998},"min":{"1min":0.42699999999999999,"5min":0.35199999999999998,"15min":0.33600000000000002},"max":{"1min":1.5429999999999999,"5min":4.3700000000000001,"15min":5.3570000000000002},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.167,"15min":1.0780000000000001},"min":{"1min":0.52700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":12.292999999999999,"5min":12.292999999999999,"15min":12.292999999999999},"last":1.0760000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.94499999999999995,"15min":0.91400000000000003},"min":{"1min":0.50900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.8999999999999999,"5min":4.0979999999999999,"15min":4.0979999999999999},"last":0.875},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.2989999999999999,"15min":1.1479999999999999},"min":{"1min":0.39000000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.875,"5min":13.443,"15min":13.443},"last":1.1180000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1759999999999999,"15min":1.129},"min":{"1min":0.55900000000000005,"5min":0.45300000000000001,"15min":0.376},"max":{"1min":1.7110000000000001,"5min":3.806,"15min":4.008},"last":1.339},{"interface":"front","average":{"1min":1.173,"5min":1.2749999999999999,"15min":1.2050000000000001},"min":{"1min":0.46300000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6020000000000001,"5min":5.4539999999999997,"15min":5.5110000000000001},"last":1.173}]},{"osd":4,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.9319999999999999,"5min":1.4079999999999999,"15min":1.3400000000000001},"min":{"1min":0.77700000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":12.744,"5min":12.744,"15min":13.292},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.431,"15min":1.4339999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.3780000000000001,"5min":6.4279999999999999,"15min":11.068},"last":1.579}]},{"osd":5,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.724,"15min":1.5269999999999999},"min":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.6659999999999999,"5min":17.096,"15min":17.096},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.605,"15min":1.5509999999999999},"min":{"1min":0.78500000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.137,"5min":9.3629999999999995,"15min":12.106},"last":1.3979999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":2.294,"15min":1.7210000000000001},"min":{"1min":0.85799999999999998,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":12.675000000000001,"5min":43.609999999999999,"15min":43.609999999999999},"last":1.548},{"interface":"front","average":{"1min":1.462,"5min":1.9390000000000001,"15min":1.6659999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.4649999999999999,"5min":38.784999999999997,"15min":38.784999999999997},"last":1.661}]},{"osd":7,"last update":"Tue Jan 16 15:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":2.0070000000000001,"15min":1.6919999999999999},"min":{"1min":0.84599999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4489999999999998,"5min":25.071000000000002,"15min":25.071000000000002},"last":1.5129999999999999},{"interface":"front","average":{"1min":2.448,"5min":2.081,"15min":1.7350000000000001},"min":{"1min":0.71999999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":12.836,"5min":13.365,"15min":18.846},"last":1.4199999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542303,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68288,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303552,"statfs":{"total":96636764160,"available":96566837248,"internally_reserved":0,"allocated":1093632,"data_stored":467674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.1359999999999999,"15min":0.998},"min":{"1min":0.53600000000000003,"5min":0.224,"15min":0.224},"max":{"1min":1.502,"5min":11.804,"15min":11.804},"last":0.311},{"interface":"front","average":{"1min":1.048,"5min":1.2450000000000001,"15min":1.1020000000000001},"min":{"1min":0.57999999999999996,"5min":0.435,"15min":0.35799999999999998},"max":{"1min":1.649,"5min":13.066000000000001,"15min":13.066000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.353,"15min":1.147},"min":{"1min":0.60199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.4850000000000001,"5min":14.199999999999999,"15min":14.199999999999999},"last":0.434},{"interface":"front","average":{"1min":1.133,"5min":1.345,"15min":1.1799999999999999},"min":{"1min":0.495,"5min":0.42999999999999999,"15min":0.372},"max":{"1min":1.5820000000000001,"5min":13.15,"15min":13.15},"last":0.70599999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.329,"15min":1.1319999999999999},"min":{"1min":0.54700000000000004,"5min":0.38200000000000001,"15min":0.31900000000000001},"max":{"1min":1.617,"5min":14.965,"15min":14.965},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.357,"15min":1.2330000000000001},"min":{"1min":0.56799999999999995,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.7270000000000001,"5min":11.717000000000001,"15min":11.717000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.9710000000000001,"15min":1.6399999999999999},"min":{"1min":0.85899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.173,"5min":19.957999999999998,"15min":19.957999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.385,"5min":1.77,"15min":1.585},"min":{"1min":0.91800000000000004,"5min":0.78400000000000003,"15min":0.55700000000000005},"max":{"1min":1.8520000000000001,"5min":15.151,"15min":15.151},"last":0.95499999999999996}]},{"osd":5,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.611,"15min":1.573},"min":{"1min":1.071,"5min":0.61099999999999999,"15min":0.60899999999999999},"max":{"1min":2.0409999999999999,"5min":15.077,"15min":26.501000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.593,"15min":1.4430000000000001},"min":{"1min":0.97499999999999998,"5min":0.78000000000000003,"15min":0.47999999999999998},"max":{"1min":2,"5min":13.760999999999999,"15min":13.760999999999999},"last":0.97799999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5069999999999999,"15min":1.49},"min":{"1min":0.93700000000000006,"5min":0.77500000000000002,"15min":0.438},"max":{"1min":2.1400000000000001,"5min":6.3940000000000001,"15min":15.144},"last":1.036},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.718,"15min":1.8280000000000001},"min":{"1min":0.95899999999999996,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":2.3079999999999998,"5min":14.327,"15min":66.061999999999998},"last":0.81100000000000005}]},{"osd":7,"last update":"Tue Jan 16 15:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.885,"15min":1.6839999999999999},"min":{"1min":1.012,"5min":0.81200000000000006,"15min":0.61099999999999999},"max":{"1min":1.952,"5min":16.667000000000002,"15min":16.667000000000002},"last":0.871},{"interface":"front","average":{"1min":1.494,"5min":1.708,"15min":1.6120000000000001},"min":{"1min":0.89800000000000002,"5min":0.71499999999999997,"15min":0.5},"max":{"1min":2.0699999999999998,"5min":8.8109999999999999,"15min":11.271000000000001},"last":1.1599999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542303,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68296,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303544,"statfs":{"total":96636764160,"available":96566829056,"internally_reserved":0,"allocated":1101824,"data_stored":467286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.349,"15min":1.0800000000000001},"min":{"1min":0.755,"5min":0.32600000000000001,"15min":0.248},"max":{"1min":1.637,"5min":21.260000000000002,"15min":21.260000000000002},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.478,"15min":1.2490000000000001},"min":{"1min":0.63200000000000001,"5min":0.44900000000000001,"15min":0.35199999999999998},"max":{"1min":2.3559999999999999,"5min":32.787999999999997,"15min":32.787999999999997},"last":1.4279999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.014,"15min":1.0009999999999999},"min":{"1min":0.439,"5min":0.38,"15min":0.28100000000000003},"max":{"1min":1.4610000000000001,"5min":7.3970000000000002,"15min":14.817},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2909999999999999,"15min":1.1499999999999999},"min":{"1min":0.74099999999999999,"5min":0.42599999999999999,"15min":0.41499999999999998},"max":{"1min":2.9660000000000002,"5min":8.8450000000000006,"15min":8.8450000000000006},"last":0.82199999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3919999999999999,"15min":1.2030000000000001},"min":{"1min":0.745,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":1.7709999999999999,"5min":9.3149999999999995,"15min":9.3149999999999995},"last":1.252},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.5069999999999999,"15min":1.2929999999999999},"min":{"1min":0.47099999999999997,"5min":0.41499999999999998,"15min":0.40400000000000003},"max":{"1min":2.2040000000000002,"5min":41.133000000000003,"15min":41.133000000000003},"last":1.167}]},{"osd":4,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.724,"15min":1.528},"min":{"1min":0.80900000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.2480000000000002,"5min":19.684000000000001,"15min":19.684000000000001},"last":1.292},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4019999999999999,"15min":1.4279999999999999},"min":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.53900000000000003},"max":{"1min":2.052,"5min":10.785,"15min":13.789999999999999},"last":1.2170000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.698,"15min":1.569},"min":{"1min":1.004,"5min":0.622,"15min":0.43099999999999999},"max":{"1min":5.3680000000000003,"5min":15.778,"15min":15.778},"last":1.373},{"interface":"front","average":{"1min":1.643,"5min":1.901,"15min":1.7689999999999999},"min":{"1min":1.0189999999999999,"5min":0.623,"15min":0.623},"max":{"1min":5.4610000000000003,"5min":15.723000000000001,"15min":35.747},"last":1.3260000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.734,"15min":1.6539999999999999},"min":{"1min":0.93200000000000005,"5min":0.64400000000000002,"15min":0.629},"max":{"1min":4.6260000000000003,"5min":15.586,"15min":16.231000000000002},"last":1.502},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.5920000000000001,"15min":1.55},"min":{"1min":0.64400000000000002,"5min":0.63200000000000001,"15min":0.56599999999999995},"max":{"1min":2.1659999999999999,"5min":11.433999999999999,"15min":13.56},"last":1.607}]},{"osd":7,"last update":"Tue Jan 16 15:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.754,"5min":1.964,"15min":1.8},"min":{"1min":1.0549999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":6.7990000000000004,"5min":18.907,"15min":21.902999999999999},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.98,"5min":1.9930000000000001,"15min":1.8009999999999999},"min":{"1min":1.1910000000000001,"5min":0.70299999999999996,"15min":0.64700000000000002},"max":{"1min":5.1260000000000003,"5min":15.452999999999999,"15min":31.044},"last":1.573}]}]},{"osd":0,"up_from":14,"seq":60129542304,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64704,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307136,"statfs":{"total":96636764160,"available":96570507264,"internally_reserved":0,"allocated":1687552,"data_stored":1057494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.121,"15min":1.026},"min":{"1min":0.54400000000000004,"5min":0.26400000000000001,"15min":0.25},"max":{"1min":2.7320000000000002,"5min":22.619,"15min":22.619},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.0840000000000001,"15min":1.1899999999999999},"min":{"1min":0.438,"5min":0.31,"15min":0.31},"max":{"1min":9.4499999999999993,"5min":9.4499999999999993,"15min":17.876000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0819999999999999,"5min":1.353,"15min":1.365},"min":{"1min":1.093,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":17.760999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.351,"5min":1.403,"15min":1.3320000000000001},"min":{"1min":0.40200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.7930000000000001,"5min":22.672999999999998,"15min":22.672999999999998},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.016,"15min":1.1359999999999999},"min":{"1min":0.60499999999999998,"5min":0.40000000000000002,"15min":0.39200000000000002},"max":{"1min":1.4730000000000001,"5min":3.0600000000000001,"15min":16.382999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":2.2839999999999998,"5min":1.4510000000000001,"15min":1.347},"min":{"1min":0.56799999999999995,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":9.6029999999999998,"5min":9.6029999999999998,"15min":17.981000000000002},"last":0.60399999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.591,"15min":1.5549999999999999},"min":{"1min":0.63400000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.6399999999999999,"5min":22.780000000000001,"15min":22.780000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.284,"15min":1.5169999999999999},"min":{"1min":0.86699999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.881,"5min":3.085,"15min":19.422000000000001},"last":1.151}]},{"osd":5,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.7629999999999999,"15min":1.861},"min":{"1min":0.70499999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.7490000000000001,"5min":22.744,"15min":44.930999999999997},"last":1.369},{"interface":"front","average":{"1min":2.0640000000000001,"5min":1.972,"15min":1.7230000000000001},"min":{"1min":0.81200000000000006,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":14.561999999999999,"5min":21.797999999999998,"15min":21.797999999999998},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":1.8879999999999999,"15min":1.8919999999999999},"min":{"1min":0.76600000000000001,"5min":0.622,"15min":0.622},"max":{"1min":14.958,"5min":14.958,"15min":27.442},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.968,"15min":1.927},"min":{"1min":0.84099999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.081,"5min":22.709,"15min":27.411999999999999},"last":1.1299999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.548,"5min":1.7390000000000001,"15min":1.724},"min":{"1min":1.319,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":15.005000000000001,"5min":15.005000000000001,"15min":18.152999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.911,"15min":1.764},"min":{"1min":0.70699999999999996,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9199999999999999,"5min":22.815000000000001,"15min":22.815000000000001},"last":0.88}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:10:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:10:06.900 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:10:07.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.184+0000 7f52bbe00640 1 Processor -- start 2024-01-16T15:10:07.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.185+0000 7f52bbe00640 1 -- start start 2024-01-16T15:10:07.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.186+0000 7f52bbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52bc0b1f30 0x7f52bc0b0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.186+0000 7f52bbe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc0d0050 con 0x7f52bc0d1d00 2024-01-16T15:10:07.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.186+0000 7f52bbe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc0d02a0 con 0x7f52bc15d190 2024-01-16T15:10:07.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.186+0000 7f52bbe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52bc0d0580 con 0x7f52bc0b1f30 2024-01-16T15:10:07.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.186+0000 7f52ba000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f52bc0d1d00 0x7f52bc0b0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:45278/0 (socket says 172.21.15.141:45278) 2024-01-16T15:10:07.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.186+0000 7f52ba000640 1 -- 172.21.15.141:0/2945805523 learned_addr learned my addr 172.21.15.141:0/2945805523 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:07.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.186+0000 7f52bb400640 1 --2- 172.21.15.141:0/2945805523 >> v2:172.21.15.141:3300/0 conn(0x7f52bc0b1f30 0x7f52bc0b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.187+0000 7f52b9600640 1 -- 172.21.15.141:0/2945805523 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928665681 0 0) 0x7f52bc0d0050 con 0x7f52bc0d1d00 2024-01-16T15:10:07.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.187+0000 7f52b9600640 1 -- 172.21.15.141:0/2945805523 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f529c003520 con 0x7f52bc0d1d00 2024-01-16T15:10:07.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.187+0000 7f52bb400640 1 -- 172.21.15.141:0/2945805523 >> v1:172.21.15.188:6789/0 conn(0x7f52bc15d190 legacy=0x7f52bc0d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.187+0000 7f52bb400640 1 -- 172.21.15.141:0/2945805523 >> v1:172.21.15.141:6789/0 conn(0x7f52bc0d1d00 legacy=0x7f52bc0b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.187+0000 7f52bb400640 1 -- 172.21.15.141:0/2945805523 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc0d0860 con 0x7f52bc0b1f30 2024-01-16T15:10:07.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.189+0000 7f52bb400640 1 --2- 172.21.15.141:0/2945805523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc0b0890 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f52b40097d0 tx=0x7f52b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5613ea7912e89bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:07.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.189+0000 7f52b9600640 1 -- 172.21.15.141:0/2945805523 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4030610 con 0x7f52bc0b1f30 2024-01-16T15:10:07.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.189+0000 7f52b9600640 1 -- 172.21.15.141:0/2945805523 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b4030770 con 0x7f52bc0b1f30 2024-01-16T15:10:07.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.189+0000 7f52b9600640 1 -- 172.21.15.141:0/2945805523 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4030b00 con 0x7f52bc0b1f30 2024-01-16T15:10:07.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.190+0000 7f52bbe00640 1 -- 172.21.15.141:0/2945805523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 msgr2=0x7f52bc0b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.190+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2945805523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc0b0890 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f52b40097d0 tx=0x7f52b402b960 comp rx=0 tx=0).stop 2024-01-16T15:10:07.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.190+0000 7f52bbe00640 1 -- 172.21.15.141:0/2945805523 shutdown_connections 2024-01-16T15:10:07.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.190+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2945805523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc0b0890 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.190+0000 7f52bbe00640 1 -- 172.21.15.141:0/2945805523 >> 172.21.15.141:0/2945805523 conn(0x7f52bc0d3010 msgr2=0x7f52bc0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:07.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.190+0000 7f52bbe00640 1 -- 172.21.15.141:0/2945805523 shutdown_connections 2024-01-16T15:10:07.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.190+0000 7f52bbe00640 1 -- 172.21.15.141:0/2945805523 wait complete. 2024-01-16T15:10:07.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.192+0000 7f52bbe00640 1 Processor -- start 2024-01-16T15:10:07.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.192+0000 7f52bbe00640 1 -- start start 2024-01-16T15:10:07.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52bbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc15b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52bbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52bc0d1d00 0x7f52bc15f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52bbe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52bc15d190 0x7f52bc152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52bbe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15f570 con 0x7f52bc0b1f30 2024-01-16T15:10:07.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52bbe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15df90 con 0x7f52bc0d1d00 2024-01-16T15:10:07.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52bbe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15e270 con 0x7f52bc15d190 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc15b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc15b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46208/0 (socket says 172.21.15.141:46208) 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 -- 172.21.15.141:0/2112005059 learned_addr learned my addr 172.21.15.141:0/2112005059 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 -- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52bc15d190 msgr2=0x7f52bc152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52bc15d190 0x7f52bc152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 -- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52bc0d1d00 msgr2=0x7f52bc15f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52bc0d1d00 0x7f52bc15f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.193+0000 7f52baa00640 1 -- 172.21.15.141:0/2112005059 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b4009400 con 0x7f52bc0b1f30 2024-01-16T15:10:07.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.194+0000 7f52baa00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc15b860 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c400 tx=0x7f52ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:07.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.194+0000 7f52b2a00640 1 -- 172.21.15.141:0/2112005059 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac0052d0 con 0x7f52bc0b1f30 2024-01-16T15:10:07.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.194+0000 7f52b2a00640 1 -- 172.21.15.141:0/2112005059 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac005430 con 0x7f52bc0b1f30 2024-01-16T15:10:07.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.194+0000 7f52b2a00640 1 -- 172.21.15.141:0/2112005059 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac003830 con 0x7f52bc0b1f30 2024-01-16T15:10:07.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.194+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc15e550 con 0x7f52bc0b1f30 2024-01-16T15:10:07.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.194+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc15eb10 con 0x7f52bc0b1f30 2024-01-16T15:10:07.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.195+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc15ed20 con 0x7f52bc0b1f30 2024-01-16T15:10:07.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.196+0000 7f52b2a00640 1 -- 172.21.15.141:0/2112005059 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f52ac003990 con 0x7f52bc0b1f30 2024-01-16T15:10:07.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.197+0000 7f52b2a00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f529406bfb0 0x7f529406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.197+0000 7f52b2a00640 1 -- 172.21.15.141:0/2112005059 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52ac087c00 con 0x7f52bc0b1f30 2024-01-16T15:10:07.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.198+0000 7f52ba000640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f529406bfb0 0x7f529406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.199+0000 7f52ba000640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f529406bfb0 0x7f529406e470 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0b3c70 tx=0x7f52a4001690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:07.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.201+0000 7f52b2a00640 1 -- 172.21.15.141:0/2112005059 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52ac054430 con 0x7f52bc0b1f30 2024-01-16T15:10:07.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.457+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f52bc067450 con 0x7f52bc0b1f30 2024-01-16T15:10:07.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.463+0000 7f52b2a00640 1 -- 172.21.15.141:0/2112005059 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f52ac017020 con 0x7f52bc0b1f30 2024-01-16T15:10:07.440 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T15:10:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T15:10:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: caps mds = "allow" 2024-01-16T15:10:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: caps mon = "allow r" 2024-01-16T15:10:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:10:07.441 INFO:teuthology.orchestra.run.smithi141.stderr:updated caps for client.0 2024-01-16T15:10:07.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f529406bfb0 msgr2=0x7f529406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f529406bfb0 0x7f529406e470 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0b3c70 tx=0x7f52a4001690 comp rx=0 tx=0).stop 2024-01-16T15:10:07.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 msgr2=0x7f52bc15b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc15b860 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c400 tx=0x7f52ac00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 shutdown_connections 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f529406bfb0 0x7f529406e470 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52bc15d190 0x7f52bc152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52bc0d1d00 0x7f52bc15f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 --2- 172.21.15.141:0/2112005059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52bc0b1f30 0x7f52bc15b860 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 >> 172.21.15.141:0/2112005059 conn(0x7f52bc0d3010 msgr2=0x7f52bc0b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 shutdown_connections 2024-01-16T15:10:07.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.466+0000 7f52bbe00640 1 -- 172.21.15.141:0/2112005059 wait complete. 2024-01-16T15:10:07.462 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:10:07.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.691+0000 7f8741a00640 1 Processor -- start 2024-01-16T15:10:07.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.691+0000 7f8741a00640 1 -- start start 2024-01-16T15:10:07.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.693+0000 7f8741a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f873c13df80 0x7f873c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.693+0000 7f8741a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0cf9a0 0x7f873c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.693+0000 7f8741a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c06c470 con 0x7f873c13e360 2024-01-16T15:10:07.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.693+0000 7f8741a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873c06c750 con 0x7f873c13df80 2024-01-16T15:10:07.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.693+0000 7f8741a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c06ca30 con 0x7f873c0cf9a0 2024-01-16T15:10:07.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f8731e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f873c13e360 0x7f873c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:45280/0 (socket says 172.21.15.141:45280) 2024-01-16T15:10:07.670 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f8731e00640 1 -- 172.21.15.141:0/4044611829 learned_addr learned my addr 172.21.15.141:0/4044611829 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:07.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873b400640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0cf9a0 0x7f873c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873aa00640 1 --2- 172.21.15.141:0/4044611829 >> v2:172.21.15.188:3300/0 conn(0x7f873c13df80 0x7f873c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641879929 0 0) 0x7f873c06c470 con 0x7f873c13e360 2024-01-16T15:10:07.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871c003520 con 0x7f873c13e360 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4031316613 0 0) 0x7f871c003520 con 0x7f873c13e360 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0cf9a0 msgr2=0x7f873c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0cf9a0 0x7f873c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 >> v2:172.21.15.188:3300/0 conn(0x7f873c13df80 msgr2=0x7f873c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 --2- 172.21.15.141:0/4044611829 >> v2:172.21.15.188:3300/0 conn(0x7f873c13df80 0x7f873c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.694+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c06cd10 con 0x7f873c13e360 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8724001670 con 0x7f873c13e360 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 >> v1:172.21.15.141:6789/0 conn(0x7f873c13e360 legacy=0x7f873c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871c003970 0x7f871c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c0047c0 0x7f871c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0cf9a0 0x7f871c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c06c750 con 0x7f871c003970 2024-01-16T15:10:07.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c06ca30 con 0x7f871c0047c0 2024-01-16T15:10:07.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c00b980 con 0x7f873c0cf9a0 2024-01-16T15:10:07.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873aa00640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871c003970 0x7f871c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873aa00640 1 -- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0cf9a0 msgr2=0x7f871c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:07.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873aa00640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0cf9a0 0x7f871c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873aa00640 1 -- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c0047c0 msgr2=0x7f871c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:07.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873aa00640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c0047c0 0x7f871c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.695+0000 7f873aa00640 1 -- 172.21.15.141:0/4044611829 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c003520 con 0x7f871c003970 2024-01-16T15:10:07.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.696+0000 7f873aa00640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871c003970 0x7f871c003d90 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f872c009ac0 tx=0x7f871c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a793e7052140be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:07.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c02c710 con 0x7f871c003970 2024-01-16T15:10:07.675 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c02c870 con 0x7f871c003970 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f873a000640 1 -- 172.21.15.141:0/4044611829 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c02cc00 con 0x7f871c003970 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f8741a00640 1 -- 172.21.15.141:0/4044611829 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871c003970 msgr2=0x7f871c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f8741a00640 1 --2- 172.21.15.141:0/4044611829 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871c003970 0x7f871c003d90 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f872c009ac0 tx=0x7f871c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f873b400640 1 -- 172.21.15.141:0/4044611829 reap_dead start 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f8741a00640 1 -- 172.21.15.141:0/4044611829 shutdown_connections 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.697+0000 7f8741a00640 1 -- 172.21.15.141:0/4044611829 >> 172.21.15.141:0/4044611829 conn(0x7f873c06a800 msgr2=0x7f873c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.698+0000 7f8741a00640 1 -- 172.21.15.141:0/4044611829 shutdown_connections 2024-01-16T15:10:07.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.698+0000 7f8741a00640 1 -- 172.21.15.141:0/4044611829 wait complete. 2024-01-16T15:10:07.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.699+0000 7f8741a00640 1 Processor -- start 2024-01-16T15:10:07.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.699+0000 7f8741a00640 1 -- start start 2024-01-16T15:10:07.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.700+0000 7f8741a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c003970 0x7f873c13d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.700+0000 7f8741a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f873c0b2f60 0x7f873c0ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.700+0000 7f8741a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0abcf0 0x7f873c152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.700+0000 7f8741a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c13d200 con 0x7f873c0b2f60 2024-01-16T15:10:07.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.700+0000 7f8741a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c13db60 con 0x7f871c003970 2024-01-16T15:10:07.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.700+0000 7f8741a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0ac170 con 0x7f873c0abcf0 2024-01-16T15:10:07.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f873c0b2f60 0x7f873c0ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f873c0b2f60 0x7f873c0ab7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46228/0 (socket says 172.21.15.141:46228) 2024-01-16T15:10:07.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 -- 172.21.15.141:0/392917243 learned_addr learned my addr 172.21.15.141:0/392917243 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:07.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 -- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0abcf0 msgr2=0x7f873c152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0abcf0 0x7f873c152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.679 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f873aa00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c003970 0x7f873c13d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 -- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c003970 msgr2=0x7f873c13d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c003970 0x7f873c13d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 -- 172.21.15.141:0/392917243 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c13c860 con 0x7f873c0b2f60 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.701+0000 7f8731e00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f873c0b2f60 0x7f873c0ab7b0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f872400c730 tx=0x7f872400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.702+0000 7f8733400640 1 -- 172.21.15.141:0/392917243 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87240074e0 con 0x7f873c0b2f60 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.702+0000 7f8733400640 1 -- 172.21.15.141:0/392917243 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8724007660 con 0x7f873c0b2f60 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.702+0000 7f8733400640 1 -- 172.21.15.141:0/392917243 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724004a20 con 0x7f873c0b2f60 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.702+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c13cb40 con 0x7f873c0b2f60 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.702+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c0b3ed0 con 0x7f873c0b2f60 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.703+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c13d340 con 0x7f873c0b2f60 2024-01-16T15:10:07.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.704+0000 7f8733400640 1 -- 172.21.15.141:0/392917243 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f8724004b80 con 0x7f873c0b2f60 2024-01-16T15:10:07.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.705+0000 7f8733400640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f871406bfb0 0x7f871406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:07.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.705+0000 7f8733400640 1 -- 172.21.15.141:0/392917243 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8724087970 con 0x7f873c0b2f60 2024-01-16T15:10:07.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.705+0000 7f873aa00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f871406bfb0 0x7f871406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:07.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.706+0000 7f873aa00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f871406bfb0 0x7f871406e470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f872c036000 tx=0x7f872c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:07.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.709+0000 7f8733400640 1 -- 172.21.15.141:0/392917243 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8724054270 con 0x7f873c0b2f60 2024-01-16T15:10:07.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.938+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f873c0675d0 con 0x7f873c0b2f60 2024-01-16T15:10:07.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.940+0000 7f8733400640 1 -- 172.21.15.141:0/392917243 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f8724018090 con 0x7f873c0b2f60 2024-01-16T15:10:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:07.917 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/1211961850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1211961850},{"type":"v1","addr":"172.21.15.141:6835","nonce":1211961850}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3677420872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3677420872},{"type":"v1","addr":"172.21.15.188:6835","nonce":3677420872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/4249270856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":4249270856},{"type":"v1","addr":"172.21.15.188:6837","nonce":4249270856}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:10:01.362921+0000","modified":"2024-01-16T15:10:04.436791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8148":{"gid":8148,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6834/3399417259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3399417259},{"type":"v1","addr":"172.21.15.141:6834","nonce":3399417259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:10:07.918 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 54 2024-01-16T15:10:07.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.942+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f871406bfb0 msgr2=0x7f871406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.942+0000 7f8741a00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f871406bfb0 0x7f871406e470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f872c036000 tx=0x7f872c030000 comp rx=0 tx=0).stop 2024-01-16T15:10:07.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f873c0b2f60 msgr2=0x7f873c0ab7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:07.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f873c0b2f60 0x7f873c0ab7b0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f872400c730 tx=0x7f872400cbf0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 shutdown_connections 2024-01-16T15:10:07.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f871406bfb0 0x7f871406e470 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f873c0abcf0 0x7f873c152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f873c0b2f60 0x7f873c0ab7b0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 --2- 172.21.15.141:0/392917243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f871c003970 0x7f873c13d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:07.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 >> 172.21.15.141:0/392917243 conn(0x7f873c06a800 msgr2=0x7f873c0aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:07.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 shutdown_connections 2024-01-16T15:10:07.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:07.943+0000 7f8741a00640 1 -- 172.21.15.141:0/392917243 wait complete. 2024-01-16T15:10:08.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:10:01.362921+0000', 'modified': '2024-01-16T15:10:04.436791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8148}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8148': {'gid': 8148, 'name': 'c', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6834/3399417259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6832', 'nonce': 3399417259}, {'type': 'v1', 'addr': '172.21.15.141:6834', 'nonce': 3399417259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:10:08.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:10:08.342 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:10:08.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.580+0000 7f404a400640 1 Processor -- start 2024-01-16T15:10:08.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.581+0000 7f404a400640 1 -- start start 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.582+0000 7f404a400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f40440cf9a0 0x7f404415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.582+0000 7f404a400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f404415e1e0 0x7f40440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.582+0000 7f404a400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40440b1ee0 con 0x7f4044143ae0 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.582+0000 7f404a400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40440b2780 con 0x7f404415e1e0 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.582+0000 7f404a400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404415e990 con 0x7f40440cf9a0 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.582+0000 7f4043e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f404415e1e0 0x7f40440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.583+0000 7f4043e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f404415e1e0 0x7f40440702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46244/0 (socket says 172.21.15.141:46244) 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.583+0000 7f4043e00640 1 -- 172.21.15.141:0/2782185358 learned_addr learned my addr 172.21.15.141:0/2782185358 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.583+0000 7f403a800640 1 --2- 172.21.15.141:0/2782185358 >> v2:172.21.15.188:3300/0 conn(0x7f40440cf9a0 0x7f404415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.583+0000 7f4043e00640 1 -- 172.21.15.141:0/2782185358 >> v2:172.21.15.188:3300/0 conn(0x7f40440cf9a0 msgr2=0x7f404415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.583+0000 7f4043e00640 1 --2- 172.21.15.141:0/2782185358 >> v2:172.21.15.188:3300/0 conn(0x7f40440cf9a0 0x7f404415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.583+0000 7f4043e00640 1 -- 172.21.15.141:0/2782185358 >> v1:172.21.15.188:6789/0 conn(0x7f4044143ae0 legacy=0x7f40440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.583+0000 7f4043e00640 1 -- 172.21.15.141:0/2782185358 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40440b2130 con 0x7f404415e1e0 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.584+0000 7f4043e00640 1 --2- 172.21.15.141:0/2782185358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f40440702c0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f40440cb830 tx=0x7f403c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932d3a602ff46242 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.584+0000 7f4042a00640 1 -- 172.21.15.141:0/2782185358 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c02ca80 con 0x7f404415e1e0 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f4042a00640 1 -- 172.21.15.141:0/2782185358 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c02cbe0 con 0x7f404415e1e0 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f4042a00640 1 -- 172.21.15.141:0/2782185358 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c0343f0 con 0x7f404415e1e0 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 -- 172.21.15.141:0/2782185358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 msgr2=0x7f40440702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 --2- 172.21.15.141:0/2782185358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f40440702c0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f40440cb830 tx=0x7f403c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 -- 172.21.15.141:0/2782185358 shutdown_connections 2024-01-16T15:10:08.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 --2- 172.21.15.141:0/2782185358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f40440702c0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 --2- 172.21.15.141:0/2782185358 >> v2:172.21.15.188:3300/0 conn(0x7f40440cf9a0 0x7f404415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 -- 172.21.15.141:0/2782185358 >> 172.21.15.141:0/2782185358 conn(0x7f404406a800 msgr2=0x7f404406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 -- 172.21.15.141:0/2782185358 shutdown_connections 2024-01-16T15:10:08.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.585+0000 7f404a400640 1 -- 172.21.15.141:0/2782185358 wait complete. 2024-01-16T15:10:08.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.587+0000 7f404a400640 1 Processor -- start 2024-01-16T15:10:08.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.587+0000 7f404a400640 1 -- start start 2024-01-16T15:10:08.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.588+0000 7f404a400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40440cf9a0 0x7f4044159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.588+0000 7f404a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4044143ae0 0x7f404413f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.588+0000 7f404a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f404413fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.588+0000 7f404a400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4044159eb0 con 0x7f404415e1e0 2024-01-16T15:10:08.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.588+0000 7f404a400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40441401b0 con 0x7f40440cf9a0 2024-01-16T15:10:08.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.588+0000 7f404a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4044139a00 con 0x7f4044143ae0 2024-01-16T15:10:08.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.588+0000 7f4043e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f404413fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f404413fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46248/0 (socket says 172.21.15.141:46248) 2024-01-16T15:10:08.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 -- 172.21.15.141:0/1130081759 learned_addr learned my addr 172.21.15.141:0/1130081759 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40440cf9a0 0x7f4044159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f403a800640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4044143ae0 0x7f404413f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 -- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4044143ae0 msgr2=0x7f404413f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4044143ae0 0x7f404413f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 -- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40440cf9a0 msgr2=0x7f4044159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40440cf9a0 0x7f4044159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 -- 172.21.15.141:0/1130081759 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c009b30 con 0x7f404415e1e0 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.589+0000 7f4043e00640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f404413fb30 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f403c036040 tx=0x7f403c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.590+0000 7f403be00640 1 -- 172.21.15.141:0/1130081759 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c009510 con 0x7f404415e1e0 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.590+0000 7f403be00640 1 -- 172.21.15.141:0/1130081759 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c03e450 con 0x7f404415e1e0 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.590+0000 7f403be00640 1 -- 172.21.15.141:0/1130081759 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c03e7e0 con 0x7f404415e1e0 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.590+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4044139ce0 con 0x7f404415e1e0 2024-01-16T15:10:08.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.590+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404413a2a0 con 0x7f404415e1e0 2024-01-16T15:10:08.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.593+0000 7f403be00640 1 -- 172.21.15.141:0/1130081759 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f403c02cd50 con 0x7f404415e1e0 2024-01-16T15:10:08.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.593+0000 7f403be00640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f401c06bfb0 0x7f401c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.594+0000 7f403be00640 1 -- 172.21.15.141:0/1130081759 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f403c0ab3f0 con 0x7f404415e1e0 2024-01-16T15:10:08.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.594+0000 7f4043400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f401c06bfb0 0x7f401c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.594+0000 7f404d1e2740 1 Processor -- start 2024-01-16T15:10:08.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.594+0000 7f404d1e2740 1 -- start start 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.595+0000 7f4043400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f401c06bfb0 0x7f401c06e470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f404413cd90 tx=0x7f4034009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.595+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 0x56358f1c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.595+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f198ba0 0x56358f19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.595+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f19efe0 0x56358f1e1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.595+0000 7f404d1e2740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358f0fc880 con 0x56358f19efe0 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.595+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358f10bae0 con 0x56358f198ba0 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.595+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358f1091d0 con 0x56358f1c8b80 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f19efe0 0x56358f1e1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f19efe0 0x56358f1e1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46258/0 (socket says 172.21.15.141:46258) 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 -- 172.21.15.141:0/2856632182 learned_addr learned my addr 172.21.15.141:0/2856632182 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043400640 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 0x56358f1c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f403a800640 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f198ba0 0x56358f19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 -- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 msgr2=0x56358f1c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 0x56358f1c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 -- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f198ba0 msgr2=0x56358f19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f198ba0 0x56358f19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.596+0000 7f4043e00640 1 -- 172.21.15.141:0/2856632182 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358f0cc860 con 0x56358f19efe0 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f4043e00640 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f19efe0 0x56358f1e1560 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f403c07b440 tx=0x7f403c07b0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6040aa2ce9f9613 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f4033e00640 1 -- 172.21.15.141:0/2856632182 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c07f890 con 0x56358f19efe0 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f4033e00640 1 -- 172.21.15.141:0/2856632182 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c083a40 con 0x56358f19efe0 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f4033e00640 1 -- 172.21.15.141:0/2856632182 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c07ac30 con 0x56358f19efe0 2024-01-16T15:10:08.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f404d1e2740 1 -- 172.21.15.141:0/2856632182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f19efe0 msgr2=0x56358f1e1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f404d1e2740 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f19efe0 0x56358f1e1560 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f403c07b440 tx=0x7f403c07b0a0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f404d1e2740 1 -- 172.21.15.141:0/2856632182 shutdown_connections 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f404d1e2740 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f19efe0 0x56358f1e1560 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f404d1e2740 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f198ba0 0x56358f19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f404d1e2740 1 --2- 172.21.15.141:0/2856632182 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 0x56358f1c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.597+0000 7f404d1e2740 1 -- 172.21.15.141:0/2856632182 >> 172.21.15.141:0/2856632182 conn(0x56358f15fc70 msgr2=0x56358f1e3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.598+0000 7f404d1e2740 1 -- 172.21.15.141:0/2856632182 shutdown_connections 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.598+0000 7f404d1e2740 1 -- 172.21.15.141:0/2856632182 wait complete. 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.598+0000 7f404d1e2740 1 -- start start 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.598+0000 7f404d1e2740 1 Processor -- start 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.599+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.599+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c8b80 0x56358f1c6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.599+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c6eb0 0x56358f1e6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.599+0000 7f404d1e2740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358f104ac0 con 0x56358f198ba0 2024-01-16T15:10:08.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.599+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358f0a12f0 con 0x56358f1c8b80 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.599+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358f0bec70 con 0x56358f1c6eb0 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f404d1e2740 10 client.0 _ll_register_callbacks cb 0x56358f0c33b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1c5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46264/0 (socket says 172.21.15.141:46264) 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 -- 172.21.15.141:0/323516926 learned_addr learned my addr 172.21.15.141:0/323516926 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 -- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c6eb0 msgr2=0x56358f1e6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043e00640 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c6eb0 0x56358f1e6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f403a800640 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c8b80 0x56358f1c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c6eb0 0x56358f1e6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 -- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c8b80 msgr2=0x56358f1c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c8b80 0x56358f1c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 -- 172.21.15.141:0/323516926 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404413a2a0 con 0x56358f198ba0 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4043400640 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1c5600 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f403400ed90 tx=0x7f40340036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c21c43ea9508b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.600+0000 7f4030c00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f4030c00640 1 -- 172.21.15.141:0/323516926 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034022480 con 0x56358f198ba0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f4030c00640 1 -- 172.21.15.141:0/323516926 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40340225e0 con 0x56358f198ba0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56358f0fdaa0 con 0x56358f198ba0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f4030c00640 1 -- 172.21.15.141:0/323516926 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40340227a0 con 0x56358f198ba0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 10 client.8298 fetch_fsmap learned FSMap version 54 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x56358f103140 con 0x56358f198ba0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f4030c00640 1 -- 172.21.15.141:0/323516926 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56358f0fdaa0 con 0x56358f198ba0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f4030c00640 1 -- 172.21.15.141:0/323516926 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 54) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4034022900 con 0x56358f198ba0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 10 client.8298 fetch_fsmap finished waiting for FSMap version 54 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 10 client.8298 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 20 client.8298 populate_metadata read hostname 'smithi141' 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f2032e0 0x56358f2056d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 4 client.8298 mds_command: new command op to 8148 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.601+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 --> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56358f0fd7c0 con 0x56358f2032e0 2024-01-16T15:10:08.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.602+0000 7f403a800640 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f2032e0 0x56358f2056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.602+0000 7f403a800640 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f2032e0 0x56358f2056d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77deb9e59fbb542 server_cookie=3b3d421f788bfbab in_seq=0 out_seq=0 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.602+0000 7f4030c00640 10 client.8298 ms_handle_connect on v2:172.21.15.141:6832/3399417259 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f4030c00640 1 -- 172.21.15.141:0/323516926 <== mds.0 v2:172.21.15.141:6832/3399417259 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56358f0fd7c0 con 0x56358f2032e0 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f4030c00640 10 client.8298 handle_command_reply: tid=0 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 client.8298 shutdown 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 msgr2=0x56358f1c5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1c5600 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f403400ed90 tx=0x7f40340036f0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 shutdown_connections 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f2032e0 0x56358f2056d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c6eb0 0x56358f1e6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c8b80 0x56358f1c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f4030c00640 0 client.8298 ms_handle_reset on v2:172.21.15.141:6832/3399417259 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 --2- 172.21.15.141:0/323516926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1c5600 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 >> 172.21.15.141:0/323516926 conn(0x56358f15fc70 msgr2=0x56358f19da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 shutdown_connections 2024-01-16T15:10:08.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.603+0000 7f404d1e2740 1 -- 172.21.15.141:0/323516926 wait complete. 2024-01-16T15:10:08.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.604+0000 7f404d1e2740 20 client.8298 trim_cache size 0 max 16384 2024-01-16T15:10:08.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.620+0000 7f404d1e2740 1 Processor -- start 2024-01-16T15:10:08.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.621+0000 7f404d1e2740 1 -- start start 2024-01-16T15:10:08.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.621+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f198ba0 0x56358f207930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.621+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 0x56358f1e3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.621+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f1c8b80 0x56358f1e6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.621+0000 7f404d1e2740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358ee67f20 con 0x56358f1c8b80 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.621+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358f1e81f0 con 0x56358f1c6eb0 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.621+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358f20a310 con 0x56358f198ba0 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f1c8b80 0x56358f1e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f1c8b80 0x56358f1e6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46278/0 (socket says 172.21.15.141:46278) 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 -- 172.21.15.141:0/1180522240 learned_addr learned my addr 172.21.15.141:0/1180522240 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 -- 172.21.15.141:0/1180522240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f198ba0 msgr2=0x56358f207930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 --2- 172.21.15.141:0/1180522240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f198ba0 0x56358f207930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 -- 172.21.15.141:0/1180522240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 msgr2=0x56358f1e3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 --2- 172.21.15.141:0/1180522240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 0x56358f1e3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 -- 172.21.15.141:0/1180522240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358f0a0b80 con 0x56358f1c8b80 2024-01-16T15:10:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.622+0000 7f4043e00640 1 --2- 172.21.15.141:0/1180522240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f1c8b80 0x56358f1e6750 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f403c07b440 tx=0x7f403c090000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18372d0f3c294a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f4038a00640 1 -- 172.21.15.141:0/1180522240 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c07f890 con 0x56358f1c8b80 2024-01-16T15:10:08.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f4038a00640 1 -- 172.21.15.141:0/1180522240 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c079ae0 con 0x56358f1c8b80 2024-01-16T15:10:08.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f4038a00640 1 -- 172.21.15.141:0/1180522240 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c07ac30 con 0x56358f1c8b80 2024-01-16T15:10:08.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 -- 172.21.15.141:0/1180522240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f1c8b80 msgr2=0x56358f1e6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1180522240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f1c8b80 0x56358f1e6750 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f403c07b440 tx=0x7f403c090000 comp rx=0 tx=0).stop 2024-01-16T15:10:08.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 -- 172.21.15.141:0/1180522240 shutdown_connections 2024-01-16T15:10:08.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1180522240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f1c8b80 0x56358f1e6750 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1180522240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 0x56358f1e3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1180522240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f198ba0 0x56358f207930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 -- 172.21.15.141:0/1180522240 >> 172.21.15.141:0/1180522240 conn(0x56358f15fc70 msgr2=0x56358f1c5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 -- 172.21.15.141:0/1180522240 shutdown_connections 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.623+0000 7f404d1e2740 1 -- 172.21.15.141:0/1180522240 wait complete. 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.624+0000 7f404d1e2740 1 -- start start 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.624+0000 7f404d1e2740 1 Processor -- start 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 0x56358f1ef720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f404d1e2740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 0x56358f19c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f404d1e2740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358f1c5a20 con 0x56358f198ba0 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358f1ab7a0 con 0x56358f1c6eb0 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f404d1e2740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358f19a640 con 0x56358f1c8b80 2024-01-16T15:10:08.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f404d1e2740 10 client.0 _ll_register_callbacks cb 0x7f4034007e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f4043400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f4043400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1aa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46282/0 (socket says 172.21.15.141:46282) 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.625+0000 7f4043400640 1 -- 172.21.15.141:0/1250805562 learned_addr learned my addr 172.21.15.141:0/1250805562 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4043400640 1 -- 172.21.15.141:0/1250805562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 msgr2=0x56358f19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4043400640 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 0x56358f19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4043400640 1 -- 172.21.15.141:0/1250805562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 msgr2=0x56358f1ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4043400640 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 0x56358f1ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4043400640 1 -- 172.21.15.141:0/1250805562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56358f0cc860 con 0x56358f198ba0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4043400640 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1aa9f0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f4034022730 tx=0x7f4034003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8bbbffa1996b78 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4030c00640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4030c00640 1 -- 172.21.15.141:0/1250805562 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034015c90 con 0x56358f198ba0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f4030c00640 1 -- 172.21.15.141:0/1250805562 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4034005460 con 0x56358f198ba0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.626+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f403c083a40 con 0x56358f198ba0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f4030c00640 1 -- 172.21.15.141:0/1250805562 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4034032470 con 0x56358f198ba0 2024-01-16T15:10:08.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 10 client.8304 fetch_fsmap learned FSMap version 54 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x56358f0c6170 con 0x56358f198ba0 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f4030c00640 1 -- 172.21.15.141:0/1250805562 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c083a40 con 0x56358f198ba0 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f4030c00640 1 -- 172.21.15.141:0/1250805562 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 54) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f40340325d0 con 0x56358f198ba0 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 10 client.8304 fetch_fsmap finished waiting for FSMap version 54 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 10 client.8304 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 20 client.8304 populate_metadata read hostname 'smithi141' 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f19cc40 0x56358f1f9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 4 client.8304 mds_command: new command op to 8148 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 --> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56358ee056c0 con 0x56358f19cc40 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.627+0000 7f403a800640 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f19cc40 0x56358f1f9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f4030c00640 10 client.8304 ms_handle_connect on v2:172.21.15.141:6832/3399417259 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f403a800640 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f19cc40 0x56358f1f9c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d9b6f3cf8dc383 server_cookie=4928dd810f2a5bee in_seq=0 out_seq=0 2024-01-16T15:10:08.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f4030c00640 1 -- 172.21.15.141:0/1250805562 <== mds.0 v2:172.21.15.141:6832/3399417259 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56358ee056c0 con 0x56358f19cc40 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f4030c00640 10 client.8304 handle_command_reply: tid=0 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 client.8304 shutdown 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 msgr2=0x56358f1aa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1aa9f0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f4034022730 tx=0x7f4034003760 comp rx=0 tx=0).stop 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 shutdown_connections 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:6832/3399417259,v1:172.21.15.141:6834/3399417259] conn(0x56358f19cc40 0x56358f1f9c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x56358f1c8b80 0x56358f19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56358f1c6eb0 0x56358f1ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 --2- 172.21.15.141:0/1250805562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56358f198ba0 0x56358f1aa9f0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f4030c00640 0 client.8304 ms_handle_reset on v2:172.21.15.141:6832/3399417259 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 >> 172.21.15.141:0/1250805562 conn(0x56358f15fc70 msgr2=0x56358f1e6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 shutdown_connections 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.628+0000 7f404d1e2740 1 -- 172.21.15.141:0/1250805562 wait complete. 2024-01-16T15:10:08.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404d1e2740 20 client.8304 trim_cache size 0 max 16384 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 8148, 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 6.7108062720000001, 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 54, 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 89, 2024-01-16T15:10:08.610 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 89, 2024-01-16T15:10:08.611 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 13.147260586 2024-01-16T15:10:08.611 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:10:08.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f401c06bfb0 msgr2=0x7f401c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f401c06bfb0 0x7f401c06e470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f404413cd90 tx=0x7f4034009040 comp rx=0 tx=0).stop 2024-01-16T15:10:08.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 msgr2=0x7f404413fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f404413fb30 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f403c036040 tx=0x7f403c004a40 comp rx=0 tx=0).stop 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 shutdown_connections 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f401c06bfb0 0x7f401c06e470 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f404415e1e0 0x7f404413fb30 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4044143ae0 0x7f404413f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 --2- 172.21.15.141:0/1130081759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40440cf9a0 0x7f4044159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 >> 172.21.15.141:0/1130081759 conn(0x7f404406a800 msgr2=0x7f404415c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 shutdown_connections 2024-01-16T15:10:08.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.629+0000 7f404a400640 1 -- 172.21.15.141:0/1130081759 wait complete. 2024-01-16T15:10:08.623 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:10:08.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.850+0000 7f3a20c00640 1 Processor -- start 2024-01-16T15:10:08.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.851+0000 7f3a20c00640 1 -- start start 2024-01-16T15:10:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.852+0000 7f3a20c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.852+0000 7f3a20c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3a1c0cf7d0 0x7f3a1c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.852+0000 7f3a20c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c0cfc30 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.852+0000 7f3a20c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b1ee0 con 0x7f3a1c0cf7d0 2024-01-16T15:10:08.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.852+0000 7f3a20c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b2780 con 0x7f3a1c143ae0 2024-01-16T15:10:08.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.853+0000 7f3a11200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3a1c0cf7d0 0x7f3a1c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.853+0000 7f3a19e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.853+0000 7f3a19e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:52128/0 (socket says 172.21.15.141:52128) 2024-01-16T15:10:08.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.853+0000 7f3a19e00640 1 -- 172.21.15.141:0/3062557827 learned_addr learned my addr 172.21.15.141:0/3062557827 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.853+0000 7f3a11200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3a1c0cf7d0 0x7f3a1c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:60136/0 (socket says 172.21.15.141:60136) 2024-01-16T15:10:08.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.853+0000 7f3a11200640 1 -- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 msgr2=0x7f3a1c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:08.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.854+0000 7f3a11200640 1 --2- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.854+0000 7f3a11200640 1 -- 172.21.15.141:0/3062557827 >> v1:172.21.15.188:6789/0 conn(0x7f3a1c15e1e0 legacy=0x7f3a1c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.854+0000 7f3a11200640 1 -- 172.21.15.141:0/3062557827 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c15e990 con 0x7f3a1c0cf7d0 2024-01-16T15:10:08.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.854+0000 7f3a19e00640 1 --2- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:10:08.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.855+0000 7f3a11200640 1 --2- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 0x7f3a1c15dca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a040097a0 tx=0x7f3a0402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dd26421480e766d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.855+0000 7f3a19400640 1 -- 172.21.15.141:0/3062557827 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0402d040 con 0x7f3a1c0cf7d0 2024-01-16T15:10:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.855+0000 7f3a19400640 1 -- 172.21.15.141:0/3062557827 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04031810 con 0x7f3a1c0cf7d0 2024-01-16T15:10:08.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.855+0000 7f3a19400640 1 -- 172.21.15.141:0/3062557827 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04031ba0 con 0x7f3a1c0cf7d0 2024-01-16T15:10:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 -- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 msgr2=0x7f3a1c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 --2- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 0x7f3a1c15dca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a040097a0 tx=0x7f3a0402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 -- 172.21.15.141:0/3062557827 shutdown_connections 2024-01-16T15:10:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 --2- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 0x7f3a1c15dca0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 --2- 172.21.15.141:0/3062557827 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 -- 172.21.15.141:0/3062557827 >> 172.21.15.141:0/3062557827 conn(0x7f3a1c06a800 msgr2=0x7f3a1c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 -- 172.21.15.141:0/3062557827 shutdown_connections 2024-01-16T15:10:08.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.856+0000 7f3a20c00640 1 -- 172.21.15.141:0/3062557827 wait complete. 2024-01-16T15:10:08.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.859+0000 7f3a20c00640 1 Processor -- start 2024-01-16T15:10:08.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.859+0000 7f3a20c00640 1 -- start start 2024-01-16T15:10:08.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a20c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 0x7f3a1c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a20c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a20c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a1c15e1e0 0x7f3a1c0c4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a20c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b0770 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a20c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0c4a30 con 0x7f3a1c0cf7d0 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a20c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0c4d10 con 0x7f3a1c143ae0 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a1a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a1c15e1e0 0x7f3a1c0c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a1a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a1c15e1e0 0x7f3a1c0c4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46288/0 (socket says 172.21.15.141:46288) 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.860+0000 7f3a1a800640 1 -- 172.21.15.141:0/2887162637 learned_addr learned my addr 172.21.15.141:0/2887162637 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a11200640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a19e00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 0x7f3a1c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a1a800640 1 -- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 msgr2=0x7f3a1c0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a1a800640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a1a800640 1 -- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 msgr2=0x7f3a1c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a1a800640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 0x7f3a1c0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a1a800640 1 -- 172.21.15.141:0/2887162637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c00b770 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a1a800640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a1c15e1e0 0x7f3a1c0c4390 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400c6b0 tx=0x7f3a1400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.861+0000 7f3a12a00640 1 -- 172.21.15.141:0/2887162637 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a14003b70 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.862+0000 7f3a12a00640 1 -- 172.21.15.141:0/2887162637 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a14003cf0 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.862+0000 7f3a12a00640 1 -- 172.21.15.141:0/2887162637 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a14003ea0 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.862+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a04009400 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.862+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1c0c52d0 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.863+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1c0675d0 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.864+0000 7f3a12a00640 1 -- 172.21.15.141:0/2887162637 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f3a14005310 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.865+0000 7f3a12a00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39f406bfb0 0x7f39f406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:08.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.865+0000 7f3a12a00640 1 -- 172.21.15.141:0/2887162637 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a140883c0 con 0x7f3a1c15e1e0 2024-01-16T15:10:08.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.865+0000 7f3a19e00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39f406bfb0 0x7f39f406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:08.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.866+0000 7f3a19e00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39f406bfb0 0x7f39f406e470 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00bef0 tx=0x7f3a0c00b450 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:08.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:08.870+0000 7f3a12a00640 1 -- 172.21.15.141:0/2887162637 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a14054c80 con 0x7f3a1c15e1e0 2024-01-16T15:10:09.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.102+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3a1c0cea00 con 0x7f3a1c15e1e0 2024-01-16T15:10:09.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.103+0000 7f3a12a00640 1 -- 172.21.15.141:0/2887162637 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3a140582a0 con 0x7f3a1c15e1e0 2024-01-16T15:10:09.080 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39f406bfb0 msgr2=0x7f39f406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39f406bfb0 0x7f39f406e470 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00bef0 tx=0x7f3a0c00b450 comp rx=0 tx=0).stop 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a1c15e1e0 msgr2=0x7f3a1c0c4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a1c15e1e0 0x7f3a1c0c4390 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400c6b0 tx=0x7f3a1400cb70 comp rx=0 tx=0).stop 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 shutdown_connections 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f39f406bfb0 0x7f39f406e470 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a1c15e1e0 0x7f3a1c0c4390 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a1c143ae0 0x7f3a1c0c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:09.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 --2- 172.21.15.141:0/2887162637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a1c0cf7d0 0x7f3a1c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:09.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 >> 172.21.15.141:0/2887162637 conn(0x7f3a1c06a800 msgr2=0x7f3a1c15afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:09.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 shutdown_connections 2024-01-16T15:10:09.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:09.106+0000 7f3a20c00640 1 -- 172.21.15.141:0/2887162637 wait complete. 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:10:09.102 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:10:09.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:09.103 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T15:10:09.152 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:09.153 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 41854sec preferred_lft 41854sec 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T15:10:09.154 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:09.155 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:10:09.155 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:10:09.155 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::882:e9ff:fe92:c54c/64 scope link 2024-01-16T15:10:09.155 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:09.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:09.156 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:10:09.246 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T15:10:09.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:09.250 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T15:10:09.301 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T15:10:09.301 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:10:09.301 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:10:09.301 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:10:09.301 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:10:09.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:09.301 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T15:10:09.301 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T15:10:09.301 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:10:09.302 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:10:09.302 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:10:09.302 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T15:10:09.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:09.486 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T15:10:09.538 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:10:09.538 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:10:09.539 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:10:09.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:09.539 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T15:10:09.626 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:10:09.626 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:10:09.626 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:10:09.679 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:10:09.679 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:10:09.679 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:10:09.729 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:10:09.821 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:10:09.877 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:10:09.877 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:10:09.877 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:10:09.877 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:10:09.878 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:10:09.879 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:10:09.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:09.880 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:10:09.934 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:10:09.934 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:10:09.939 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:10:10.013 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:10:10.053 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:10:10.053 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:10:10.054 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:10:10.055 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:10:10.056 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:10:10.056 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:10:10.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:10.057 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:10:10.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T15:10:10.138+0000 7fedc860f480 -1 init, newargv = 0x556f0a2748b0 newargc=15 2024-01-16T15:10:10.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[69456]: starting ceph client 2024-01-16T15:10:10.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[69456]: starting fuse 2024-01-16T15:10:10.140 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T15:10:10.140 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:10:10.140 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:10:10.140 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T15:10:10.141 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T15:10:10.142 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T15:10:10.335 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.69456.asok 2024-01-16T15:10:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:10:10.352 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.69456.asok 2024-01-16T15:10:10.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:10.352 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.69456.asok status 2024-01-16T15:10:10.601 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "69456", 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 8316, 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T15:10:10.602 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 8316 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 2174908515 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 2174908515 2024-01-16T15:10:10.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.8316 192.168.0.1:0/2174908515", 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/2174908515", 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 54, 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 89, 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T15:10:10.604 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:10:10.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:10.618 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T15:10:10.618 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:10:10.673 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T15:10:10.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:10:10.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:10.674 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:10:10.771 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:10:10.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.005+0000 7fbcef400640 1 Processor -- start 2024-01-16T15:10:10.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.006+0000 7fbcef400640 1 -- start start 2024-01-16T15:10:10.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.007+0000 7fbcef400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.007+0000 7fbcef400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbce813e360 0x7fbce80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.007+0000 7fbcef400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce806c470 con 0x7fbce815f9c0 2024-01-16T15:10:10.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.007+0000 7fbcef400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce806c750 con 0x7fbce813e360 2024-01-16T15:10:10.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.007+0000 7fbcef400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce806ca30 con 0x7fbce813df80 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbced400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbce815f9c0 0x7fbce81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:45282/0 (socket says 172.21.15.141:45282) 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbced400640 1 -- 172.21.15.141:0/1930279672 learned_addr learned my addr 172.21.15.141:0/1930279672 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbcdfc00640 1 --2- 172.21.15.141:0/1930279672 >> v2:172.21.15.141:3300/0 conn(0x7fbce813e360 0x7fbce80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbceca00640 1 --2- 172.21.15.141:0/1930279672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbcdfc00640 1 -- 172.21.15.141:0/1930279672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 msgr2=0x7fbce815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbcdfc00640 1 --2- 172.21.15.141:0/1930279672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbcdfc00640 1 -- 172.21.15.141:0/1930279672 >> v1:172.21.15.141:6789/0 conn(0x7fbce815f9c0 legacy=0x7fbce81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.008+0000 7fbcdfc00640 1 -- 172.21.15.141:0/1930279672 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce806cd10 con 0x7fbce813e360 2024-01-16T15:10:10.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.010+0000 7fbcdfc00640 1 --2- 172.21.15.141:0/1930279672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce813e360 0x7fbce80cf290 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00097a0 tx=0x7fbcd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd32eea7c0dcca6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:10.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.010+0000 7fbce7e00640 1 -- 172.21.15.141:0/1930279672 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0031830 con 0x7fbce813e360 2024-01-16T15:10:10.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.010+0000 7fbce7e00640 1 -- 172.21.15.141:0/1930279672 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd0031990 con 0x7fbce813e360 2024-01-16T15:10:10.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.010+0000 7fbce7e00640 1 -- 172.21.15.141:0/1930279672 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0031d20 con 0x7fbce813e360 2024-01-16T15:10:10.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 -- 172.21.15.141:0/1930279672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce813e360 msgr2=0x7fbce80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 --2- 172.21.15.141:0/1930279672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce813e360 0x7fbce80cf290 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00097a0 tx=0x7fbcd002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 -- 172.21.15.141:0/1930279672 shutdown_connections 2024-01-16T15:10:10.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 --2- 172.21.15.141:0/1930279672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce813e360 0x7fbce80cf290 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 --2- 172.21.15.141:0/1930279672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 -- 172.21.15.141:0/1930279672 >> 172.21.15.141:0/1930279672 conn(0x7fbce806a800 msgr2=0x7fbce806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:10.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 -- 172.21.15.141:0/1930279672 shutdown_connections 2024-01-16T15:10:10.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.011+0000 7fbcef400640 1 -- 172.21.15.141:0/1930279672 wait complete. 2024-01-16T15:10:10.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.013+0000 7fbcef400640 1 Processor -- start 2024-01-16T15:10:10.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.013+0000 7fbcef400640 1 -- start start 2024-01-16T15:10:10.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbcef400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce81f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbcef400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbce813e360 0x7fbce815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbcef400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce815f9c0 0x7fbce815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbcef400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81e7ce0 con 0x7fbce815f9c0 2024-01-16T15:10:10.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbcef400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81e7fc0 con 0x7fbce813e360 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbcef400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81e82a0 con 0x7fbce813df80 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbced400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce815f9c0 0x7fbce815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbced400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce815f9c0 0x7fbce815dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46304/0 (socket says 172.21.15.141:46304) 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.014+0000 7fbced400640 1 -- 172.21.15.141:0/427018124 learned_addr learned my addr 172.21.15.141:0/427018124 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbceca00640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce81f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbced400640 1 -- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 msgr2=0x7fbce81f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbced400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce81f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbced400640 1 -- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbce813e360 msgr2=0x7fbce815d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbced400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbce813e360 0x7fbce815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbced400640 1 -- 172.21.15.141:0/427018124 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd0009400 con 0x7fbce815f9c0 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbceca00640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce81f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbced400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce815f9c0 0x7fbce815dde0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fbce000cbc0 tx=0x7fbce000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbce5600640 1 -- 172.21.15.141:0/427018124 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce000ad40 con 0x7fbce815f9c0 2024-01-16T15:10:10.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbce5600640 1 -- 172.21.15.141:0/427018124 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce0012400 con 0x7fbce815f9c0 2024-01-16T15:10:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbce5600640 1 -- 172.21.15.141:0/427018124 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce000aea0 con 0x7fbce815f9c0 2024-01-16T15:10:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.015+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce81e8580 con 0x7fbce815f9c0 2024-01-16T15:10:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.016+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce81f2d70 con 0x7fbce815f9c0 2024-01-16T15:10:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.017+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce81f2fc0 con 0x7fbce815f9c0 2024-01-16T15:10:10.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.018+0000 7fbce5600640 1 -- 172.21.15.141:0/427018124 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fbce0007070 con 0x7fbce815f9c0 2024-01-16T15:10:10.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.018+0000 7fbce5600640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbcc006c000 0x7fbcc006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.019+0000 7fbce5600640 1 -- 172.21.15.141:0/427018124 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbce0088490 con 0x7fbce815f9c0 2024-01-16T15:10:10.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.019+0000 7fbceca00640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbcc006c000 0x7fbcc006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.020+0000 7fbceca00640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbcc006c000 0x7fbcc006e4c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbce81e62a0 tx=0x7fbcd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:11.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.024+0000 7fbce5600640 1 -- 172.21.15.141:0/427018124 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbce00bde90 con 0x7fbce815f9c0 2024-01-16T15:10:11.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.252+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbce81e8770 con 0x7fbce815f9c0 2024-01-16T15:10:11.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.253+0000 7fbce5600640 1 -- 172.21.15.141:0/427018124 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbce0017070 con 0x7fbce815f9c0 2024-01-16T15:10:11.230 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:10:11.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.255+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbcc006c000 msgr2=0x7fbcc006e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:11.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.255+0000 7fbcef400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbcc006c000 0x7fbcc006e4c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbce81e62a0 tx=0x7fbcd8008040 comp rx=0 tx=0).stop 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.255+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce815f9c0 msgr2=0x7fbce815dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.255+0000 7fbcef400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce815f9c0 0x7fbce815dde0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fbce000cbc0 tx=0x7fbce000a430 comp rx=0 tx=0).stop 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 shutdown_connections 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbcc006c000 0x7fbcc006e4c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce815f9c0 0x7fbce815dde0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbce813e360 0x7fbce815d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 --2- 172.21.15.141:0/427018124 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbce813df80 0x7fbce81f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 >> 172.21.15.141:0/427018124 conn(0x7fbce806a800 msgr2=0x7fbce80affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 shutdown_connections 2024-01-16T15:10:11.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:11.256+0000 7fbcef400640 1 -- 172.21.15.141:0/427018124 wait complete. 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:10:11.251 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:10:11.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:11.251 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T15:10:11.308 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:10:11.308 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:10:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 41854sec preferred_lft 41854sec 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:10:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:10:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::a40a:23ff:fed8:ea96/64 scope link 2024-01-16T15:10:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:10:11.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:11.311 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:10:11.397 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T15:10:11.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:11.401 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T15:10:11.455 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T15:10:11.455 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:10:11.455 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:10:11.455 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:10:11.455 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:10:11.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:11.456 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T15:10:11.457 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T15:10:11.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:10:11.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:10:11.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:10:11.457 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T15:10:11.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:11.640 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T15:10:11.690 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:10:11.690 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:10:11.690 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:10:11.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:11.690 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T15:10:11.778 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:10:11.778 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:10:11.778 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:10:11.830 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:10:11.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:10:11.831 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:10:11.880 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:10:11.980 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:10:12.038 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:10:12.038 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:10:12.038 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:10:12.038 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:10:12.038 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:10:12.038 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:10:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:10:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:10:12.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:12.041 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:10:12.093 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:10:12.093 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:10:12.100 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:10:12.177 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:10:12.218 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:10:12.218 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:10:12.218 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:10:12.218 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:10:12.218 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:10:12.219 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:10:12.220 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:10:12.221 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:10:12.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:12.221 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:10:12.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T15:10:12.311+0000 7fa54cc35480 -1 init, newargv = 0x561e26656980 newargc=15 2024-01-16T15:10:12.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[45914]: starting ceph client 2024-01-16T15:10:12.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[45914]: starting fuse 2024-01-16T15:10:12.306 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T15:10:12.306 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:10:12.307 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T15:10:12.308 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T15:10:12.498 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.45914.asok 2024-01-16T15:10:12.503 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T15:10:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:10:12.516 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45914.asok 2024-01-16T15:10:12.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:12.516 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45914.asok status 2024-01-16T15:10:12.749 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T15:10:12.749 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T15:10:12.749 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:10:12.749 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:10:12.749 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T15:10:12.749 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "45914", 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 6328, 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 6328 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:10:12.750 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 1037103888 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 1037103888 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.6328 192.168.0.1:0/1037103888", 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/1037103888", 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 54, 2024-01-16T15:10:12.752 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 89, 2024-01-16T15:10:12.752 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:10:12.752 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T15:10:12.752 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T15:10:12.752 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T15:10:12.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:12.765 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T15:10:12.765 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:10:12.817 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T15:10:12.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:10:12.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:12.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:10:12.907 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:10:13.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.152+0000 7ff547c00640 1 Processor -- start 2024-01-16T15:10:13.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.152+0000 7ff547c00640 1 -- start start 2024-01-16T15:10:13.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.153+0000 7ff547c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 0x7ff54015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.153+0000 7ff547c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54006c470 con 0x7ff54013e360 2024-01-16T15:10:13.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.153+0000 7ff547c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54006c750 con 0x7ff5400cf9a0 2024-01-16T15:10:13.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.154+0000 7ff547c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54006ca30 con 0x7ff54013df80 2024-01-16T15:10:13.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.154+0000 7ff537c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff54013e360 0x7ff5400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44732/0 (socket says 172.21.15.141:44732) 2024-01-16T15:10:13.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.154+0000 7ff537c00640 1 -- 172.21.15.141:0/1359074814 learned_addr learned my addr 172.21.15.141:0/1359074814 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.154+0000 7ff545200640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 0x7ff54015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.154+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041503164 0 0) 0x7ff54006c470 con 0x7ff54013e360 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.154+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff520003520 con 0x7ff54013e360 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 868440962 0 0) 0x7ff520003520 con 0x7ff54013e360 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 msgr2=0x7ff54015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 0x7ff54015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 >> v1:172.21.15.188:6789/0 conn(0x7ff5400cf9a0 legacy=0x7ff5400cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54006cd10 con 0x7ff54013e360 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ff528001670 con 0x7ff54013e360 2024-01-16T15:10:13.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 >> v1:172.21.15.141:6789/0 conn(0x7ff54013e360 legacy=0x7ff5400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 0x7ff5200039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520003f20 0x7ff5200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff520005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54006ca30 con 0x7ff520004d00 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52000ba60 con 0x7ff520003f20 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52000bc20 con 0x7ff54013df80 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.155+0000 7ff545c00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff520005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff545200640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 0x7ff5200039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff537c00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520003f20 0x7ff5200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff545c00640 1 -- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 msgr2=0x7ff5200039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff545c00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013df80 0x7ff5200039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff545c00640 1 -- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520003f20 msgr2=0x7ff5200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff545c00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520003f20 0x7ff5200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff545c00640 1 -- 172.21.15.141:0/1359074814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff520003520 con 0x7ff520004d00 2024-01-16T15:10:13.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.156+0000 7ff537c00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520003f20 0x7ff5200047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.157+0000 7ff545c00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff520005180 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff538001e50 tx=0x7ff5200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798bc836a9ddc9d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.157+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538003b70 con 0x7ff520004d00 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.157+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff538003cd0 con 0x7ff520004d00 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.157+0000 7ff53fe00640 1 -- 172.21.15.141:0/1359074814 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538005310 con 0x7ff520004d00 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.158+0000 7ff547c00640 1 -- 172.21.15.141:0/1359074814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 msgr2=0x7ff520005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.158+0000 7ff547c00640 1 --2- 172.21.15.141:0/1359074814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff520005180 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff538001e50 tx=0x7ff5200057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.158+0000 7ff545c00640 1 -- 172.21.15.141:0/1359074814 reap_dead start 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.158+0000 7ff547c00640 1 -- 172.21.15.141:0/1359074814 shutdown_connections 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.158+0000 7ff547c00640 1 -- 172.21.15.141:0/1359074814 >> 172.21.15.141:0/1359074814 conn(0x7ff54006a800 msgr2=0x7ff54006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.158+0000 7ff547c00640 1 -- 172.21.15.141:0/1359074814 shutdown_connections 2024-01-16T15:10:13.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.158+0000 7ff547c00640 1 -- 172.21.15.141:0/1359074814 wait complete. 2024-01-16T15:10:13.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.160+0000 7ff547c00640 1 Processor -- start 2024-01-16T15:10:13.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.160+0000 7ff547c00640 1 -- start start 2024-01-16T15:10:13.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff547c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff540152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff547c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5401508f0 0x7ff5400afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff547c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013a610 0x7ff54013aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff547c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54013afd0 con 0x7ff520004d00 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff547c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5401f0950 con 0x7ff5401508f0 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff547c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5401f0c10 con 0x7ff54013a610 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff540152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff540152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49234/0 (socket says 172.21.15.141:49234) 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 -- 172.21.15.141:0/827951643 learned_addr learned my addr 172.21.15.141:0/827951643 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 -- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013a610 msgr2=0x7ff54013aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013a610 0x7ff54013aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff537c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5401508f0 0x7ff5400afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 -- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5401508f0 msgr2=0x7ff5400afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5401508f0 0x7ff5400afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.161+0000 7ff545200640 1 -- 172.21.15.141:0/827951643 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5401f0ef0 con 0x7ff520004d00 2024-01-16T15:10:13.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.162+0000 7ff545200640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff540152840 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff530011580 tx=0x7ff530011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:13.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.162+0000 7ff53d600640 1 -- 172.21.15.141:0/827951643 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff530015870 con 0x7ff520004d00 2024-01-16T15:10:13.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.162+0000 7ff53d600640 1 -- 172.21.15.141:0/827951643 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5300159d0 con 0x7ff520004d00 2024-01-16T15:10:13.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.162+0000 7ff53d600640 1 -- 172.21.15.141:0/827951643 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff530015d40 con 0x7ff520004d00 2024-01-16T15:10:13.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.162+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5401f11d0 con 0x7ff520004d00 2024-01-16T15:10:13.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.162+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff540152d80 con 0x7ff520004d00 2024-01-16T15:10:13.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.163+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5400675d0 con 0x7ff520004d00 2024-01-16T15:10:13.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.164+0000 7ff53d600640 1 -- 172.21.15.141:0/827951643 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ff53000f460 con 0x7ff520004d00 2024-01-16T15:10:13.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.165+0000 7ff53d600640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff51806bfb0 0x7ff51806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.165+0000 7ff53d600640 1 -- 172.21.15.141:0/827951643 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff530088a50 con 0x7ff520004d00 2024-01-16T15:10:13.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.166+0000 7ff537c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff51806bfb0 0x7ff51806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.167+0000 7ff537c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff51806bfb0 0x7ff51806e470 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff528001eb0 tx=0x7ff528009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:13.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.170+0000 7ff53d600640 1 -- 172.21.15.141:0/827951643 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5300552f0 con 0x7ff520004d00 2024-01-16T15:10:13.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:13.397+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff5400b1c20 con 0x7ff520004d00 2024-01-16T15:10:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.016+0000 7ff53d600640 1 -- 172.21.15.141:0/827951643 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff530058910 con 0x7ff520004d00 2024-01-16T15:10:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff51806bfb0 msgr2=0x7ff51806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff51806bfb0 0x7ff51806e470 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff528001eb0 tx=0x7ff528009040 comp rx=0 tx=0).stop 2024-01-16T15:10:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 msgr2=0x7ff540152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff540152840 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff530011580 tx=0x7ff530011a40 comp rx=0 tx=0).stop 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 shutdown_connections 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff51806bfb0 0x7ff51806e470 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff54013a610 0x7ff54013aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5401508f0 0x7ff5400afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 --2- 172.21.15.141:0/827951643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff520004d00 0x7ff540152840 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 >> 172.21.15.141:0/827951643 conn(0x7ff54006a800 msgr2=0x7ff540159840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.020+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 shutdown_connections 2024-01-16T15:10:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.021+0000 7ff547c00640 1 -- 172.21.15.141:0/827951643 wait complete. 2024-01-16T15:10:14.021 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:10:14.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.250+0000 7fc90f200640 1 Processor -- start 2024-01-16T15:10:14.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.250+0000 7fc90f200640 1 -- start start 2024-01-16T15:10:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.251+0000 7fc90f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 0x7fc90815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc90f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc9080cf290 0x7fc9080cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc90f200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc908070c40 con 0x7fc9080b3ae0 2024-01-16T15:10:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc90f200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc908071450 con 0x7fc9080cf290 2024-01-16T15:10:14.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc90f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9080cfc60 con 0x7fc9080ae5a0 2024-01-16T15:10:14.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc90d200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc9080cf290 0x7fc9080cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc90d200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc9080cf290 0x7fc9080cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49240/0 (socket says 172.21.15.141:49240) 2024-01-16T15:10:14.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc90d200640 1 -- 172.21.15.141:0/1794629473 learned_addr learned my addr 172.21.15.141:0/1794629473 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:14.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.252+0000 7fc907e00640 1 --2- 172.21.15.141:0/1794629473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 0x7fc90815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.253+0000 7fc90d200640 1 -- 172.21.15.141:0/1794629473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 msgr2=0x7fc90815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.253+0000 7fc90d200640 1 --2- 172.21.15.141:0/1794629473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 0x7fc90815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.253+0000 7fc90d200640 1 -- 172.21.15.141:0/1794629473 >> v1:172.21.15.188:6789/0 conn(0x7fc9080b3ae0 legacy=0x7fc90806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.253+0000 7fc90d200640 1 -- 172.21.15.141:0/1794629473 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90806c470 con 0x7fc9080cf290 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.254+0000 7fc90d200640 1 --2- 172.21.15.141:0/1794629473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9080cf720 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fc900009e50 tx=0x7fc90002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12f7da2a507cf1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.254+0000 7fc907400640 1 -- 172.21.15.141:0/1794629473 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90002cb90 con 0x7fc9080cf290 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.254+0000 7fc907400640 1 -- 172.21.15.141:0/1794629473 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90002ccf0 con 0x7fc9080cf290 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.254+0000 7fc907400640 1 -- 172.21.15.141:0/1794629473 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900034610 con 0x7fc9080cf290 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 -- 172.21.15.141:0/1794629473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 msgr2=0x7fc9080cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 --2- 172.21.15.141:0/1794629473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9080cf720 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fc900009e50 tx=0x7fc90002bff0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 -- 172.21.15.141:0/1794629473 shutdown_connections 2024-01-16T15:10:14.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 --2- 172.21.15.141:0/1794629473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9080cf720 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 --2- 172.21.15.141:0/1794629473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 0x7fc90815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 -- 172.21.15.141:0/1794629473 >> 172.21.15.141:0/1794629473 conn(0x7fc90806a800 msgr2=0x7fc90806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:14.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 -- 172.21.15.141:0/1794629473 shutdown_connections 2024-01-16T15:10:14.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.255+0000 7fc90f200640 1 -- 172.21.15.141:0/1794629473 wait complete. 2024-01-16T15:10:14.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.257+0000 7fc90f200640 1 Processor -- start 2024-01-16T15:10:14.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.257+0000 7fc90f200640 1 -- start start 2024-01-16T15:10:14.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 0x7fc9081f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9080b3ae0 0x7fc90814df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9081593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc908150460 con 0x7fc9080cf290 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9081f2d30 con 0x7fc9080b3ae0 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc908158dc0 con 0x7fc9080ae5a0 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9081593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9081593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49244/0 (socket says 172.21.15.141:49244) 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 -- 172.21.15.141:0/1884697008 learned_addr learned my addr 172.21.15.141:0/1884697008 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 -- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 msgr2=0x7fc9081f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 0x7fc9081f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 -- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9080b3ae0 msgr2=0x7fc90814df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9080b3ae0 0x7fc90814df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.258+0000 7fc90d200640 1 -- 172.21.15.141:0/1884697008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900009b30 con 0x7fc9080cf290 2024-01-16T15:10:14.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.259+0000 7fc90d200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9081593b0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc900009e50 tx=0x7fc900004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.259+0000 7fc904c00640 1 -- 172.21.15.141:0/1884697008 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900009510 con 0x7fc9080cf290 2024-01-16T15:10:14.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.259+0000 7fc904c00640 1 -- 172.21.15.141:0/1884697008 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90003e450 con 0x7fc9080cf290 2024-01-16T15:10:14.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.259+0000 7fc904c00640 1 -- 172.21.15.141:0/1884697008 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90003e800 con 0x7fc9080cf290 2024-01-16T15:10:14.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.259+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90814e480 con 0x7fc9080cf290 2024-01-16T15:10:14.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.259+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc908160220 con 0x7fc9080cf290 2024-01-16T15:10:14.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.260+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9081505d0 con 0x7fc9080cf290 2024-01-16T15:10:14.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.261+0000 7fc904c00640 1 -- 172.21.15.141:0/1884697008 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc90003e960 con 0x7fc9080cf290 2024-01-16T15:10:14.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.262+0000 7fc904c00640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e006bfb0 0x7fc8e006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.262+0000 7fc904c00640 1 -- 172.21.15.141:0/1884697008 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9000ab4d0 con 0x7fc9080cf290 2024-01-16T15:10:14.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.262+0000 7fc907e00640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e006bfb0 0x7fc8e006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.263+0000 7fc907e00640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e006bfb0 0x7fc8e006e470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8006a90 tx=0x7fc8f8006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.267+0000 7fc904c00640 1 -- 172.21.15.141:0/1884697008 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc900077bc0 con 0x7fc9080cf290 2024-01-16T15:10:14.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.480+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc90806c750 con 0x7fc8e006bfb0 2024-01-16T15:10:14.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.484+0000 7fc904c00640 1 -- 172.21.15.141:0/1884697008 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc8f80096c0 con 0x7fc8e006bfb0 2024-01-16T15:10:14.460 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:10:14.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:10:14.461 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:10:14.461 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:10:14.461 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:10:14.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e006bfb0 msgr2=0x7fc8e006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e006bfb0 0x7fc8e006e470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8006a90 tx=0x7fc8f8006a20 comp rx=0 tx=0).stop 2024-01-16T15:10:14.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 msgr2=0x7fc9081593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9081593b0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc900009e50 tx=0x7fc900004a40 comp rx=0 tx=0).stop 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 shutdown_connections 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc8e006bfb0 0x7fc8e006e470 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9080cf290 0x7fc9081593b0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9080b3ae0 0x7fc90814df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 --2- 172.21.15.141:0/1884697008 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc9080ae5a0 0x7fc9081f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.486+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 >> 172.21.15.141:0/1884697008 conn(0x7fc90806a800 msgr2=0x7fc90815aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.487+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 shutdown_connections 2024-01-16T15:10:14.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.487+0000 7fc90f200640 1 -- 172.21.15.141:0/1884697008 wait complete. 2024-01-16T15:10:14.482 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:10:14.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.718+0000 7f55cec00640 1 Processor -- start 2024-01-16T15:10:14.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.718+0000 7f55cec00640 1 -- start start 2024-01-16T15:10:14.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.720+0000 7f55cec00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f55c80cf9a0 0x7f55c815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.720+0000 7f55cec00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c80b1ee0 con 0x7f55c8143ae0 2024-01-16T15:10:14.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.720+0000 7f55cec00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c80b2780 con 0x7f55c815e1e0 2024-01-16T15:10:14.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.720+0000 7f55cec00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c815e990 con 0x7f55c80cf9a0 2024-01-16T15:10:14.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.720+0000 7f55c7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f55c8143ae0 0x7f55c80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44744/0 (socket says 172.21.15.141:44744) 2024-01-16T15:10:14.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.720+0000 7f55c7e00640 1 -- 172.21.15.141:0/3635465791 learned_addr learned my addr 172.21.15.141:0/3635465791 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:14.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711892833 0 0) 0x7f55c80b1ee0 con 0x7f55c8143ae0 2024-01-16T15:10:14.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c7400640 1 --2- 172.21.15.141:0/3635465791 >> v2:172.21.15.188:3300/0 conn(0x7f55c80cf9a0 0x7f55c815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a8003520 con 0x7f55c8143ae0 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393704750 0 0) 0x7f55a8003520 con 0x7f55c8143ae0 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 >> v2:172.21.15.188:3300/0 conn(0x7f55c80cf9a0 msgr2=0x7f55c815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 --2- 172.21.15.141:0/3635465791 >> v2:172.21.15.188:3300/0 conn(0x7f55c80cf9a0 0x7f55c815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 >> v1:172.21.15.188:6789/0 conn(0x7f55c815e1e0 legacy=0x7f55c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c80b2130 con 0x7f55c8143ae0 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f55b8001670 con 0x7f55c8143ae0 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 >> v1:172.21.15.141:6789/0 conn(0x7f55c8143ae0 legacy=0x7f55c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55a8003970 0x7f55a8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55a80047c0 0x7f55a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8005180 0x7f55a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c815e990 con 0x7f55a8003970 2024-01-16T15:10:14.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a800c0b0 con 0x7f55a80047c0 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.721+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a800c270 con 0x7f55a8005180 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55c7e00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55a8003970 0x7f55a8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55c7400640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55a80047c0 0x7f55a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55ccc00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8005180 0x7f55a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55c7e00640 1 -- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8005180 msgr2=0x7f55a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55c7e00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8005180 0x7f55a800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55c7e00640 1 -- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55a80047c0 msgr2=0x7f55a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55c7e00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55a80047c0 0x7f55a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55c7e00640 1 -- 172.21.15.141:0/3635465791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8003520 con 0x7f55a8003970 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55ccc00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8005180 0x7f55a800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:14.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.722+0000 7f55ccc00640 1 -- 172.21.15.141:0/3635465791 reap_dead start 2024-01-16T15:10:14.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.723+0000 7f55c7e00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55a8003970 0x7f55a8003d70 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f55b8001eb0 tx=0x7f55a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dfd4fddab19cbb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8005ad0 con 0x7f55a8003970 2024-01-16T15:10:14.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b8005c50 con 0x7f55a8003970 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55c6a00640 1 -- 172.21.15.141:0/3635465791 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b802c890 con 0x7f55a8003970 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55cec00640 1 -- 172.21.15.141:0/3635465791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55a8003970 msgr2=0x7f55a8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55cec00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55a8003970 0x7f55a8003d70 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f55b8001eb0 tx=0x7f55a80057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55cec00640 1 -- 172.21.15.141:0/3635465791 shutdown_connections 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55cec00640 1 --2- 172.21.15.141:0/3635465791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55a8003970 0x7f55a8003d70 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55cec00640 1 -- 172.21.15.141:0/3635465791 >> 172.21.15.141:0/3635465791 conn(0x7f55c806a800 msgr2=0x7f55c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55cec00640 1 -- 172.21.15.141:0/3635465791 shutdown_connections 2024-01-16T15:10:14.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.724+0000 7f55cec00640 1 -- 172.21.15.141:0/3635465791 wait complete. 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.726+0000 7f55cec00640 1 Processor -- start 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.726+0000 7f55cec00640 1 -- start start 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.727+0000 7f55cec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8003970 0x7f55c80af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.727+0000 7f55cec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55c80af930 0x7f55c80cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.727+0000 7f55cec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55c80cdc70 0x7f55c80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.727+0000 7f55cec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c815d4a0 con 0x7f55c80cdc70 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.727+0000 7f55cec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c80afdb0 con 0x7f55c80af930 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.727+0000 7f55cec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c815c400 con 0x7f55a8003970 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.727+0000 7f55ccc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55c80cdc70 0x7f55c80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55c80cdc70 0x7f55c80ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49264/0 (socket says 172.21.15.141:49264) 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 -- 172.21.15.141:0/399449164 learned_addr learned my addr 172.21.15.141:0/399449164 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55c7e00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8003970 0x7f55c80af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55c7400640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55c80af930 0x7f55c80cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 -- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8003970 msgr2=0x7f55c80af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8003970 0x7f55c80af3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 -- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55c80af930 msgr2=0x7f55c80cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55c80af930 0x7f55c80cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 -- 172.21.15.141:0/399449164 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c815c6e0 con 0x7f55c80cdc70 2024-01-16T15:10:14.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55ccc00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55c80cdc70 0x7f55c80ce0f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f55c8071f10 tx=0x7f55c000d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.728+0000 7f55bfe00640 1 -- 172.21.15.141:0/399449164 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0005230 con 0x7f55c80cdc70 2024-01-16T15:10:14.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.729+0000 7f55bfe00640 1 -- 172.21.15.141:0/399449164 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c00053b0 con 0x7f55c80cdc70 2024-01-16T15:10:14.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.729+0000 7f55bfe00640 1 -- 172.21.15.141:0/399449164 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0007260 con 0x7f55c80cdc70 2024-01-16T15:10:14.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.729+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c815c9c0 con 0x7f55c80cdc70 2024-01-16T15:10:14.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.729+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c815cf80 con 0x7f55c80cdc70 2024-01-16T15:10:14.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.730+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c80675d0 con 0x7f55c80cdc70 2024-01-16T15:10:14.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.731+0000 7f55bfe00640 1 -- 172.21.15.141:0/399449164 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f55c00073c0 con 0x7f55c80cdc70 2024-01-16T15:10:14.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.732+0000 7f55bfe00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55a006c030 0x7f55a006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.732+0000 7f55bfe00640 1 -- 172.21.15.141:0/399449164 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55c00878d0 con 0x7f55c80cdc70 2024-01-16T15:10:14.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.732+0000 7f55c7e00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55a006c030 0x7f55a006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.733+0000 7f55c7e00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55a006c030 0x7f55a006e4f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f55b8034000 tx=0x7f55b8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.736+0000 7f55bfe00640 1 -- 172.21.15.141:0/399449164 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55c0053f40 con 0x7f55c80cdc70 2024-01-16T15:10:14.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.925+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f55c8072330 con 0x7f55c80cdc70 2024-01-16T15:10:14.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.926+0000 7f55bfe00640 1 -- 172.21.15.141:0/399449164 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f55c0057560 con 0x7f55c80cdc70 2024-01-16T15:10:14.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55a006c030 msgr2=0x7f55a006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55a006c030 0x7f55a006e4f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f55b8034000 tx=0x7f55b8009030 comp rx=0 tx=0).stop 2024-01-16T15:10:14.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55c80cdc70 msgr2=0x7f55c80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55c80cdc70 0x7f55c80ce0f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f55c8071f10 tx=0x7f55c000d7d0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 shutdown_connections 2024-01-16T15:10:14.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55a006c030 0x7f55a006e4f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55c80cdc70 0x7f55c80ce0f0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55c80af930 0x7f55c80cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 --2- 172.21.15.141:0/399449164 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55a8003970 0x7f55c80af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 >> 172.21.15.141:0/399449164 conn(0x7f55c806a800 msgr2=0x7f55c80cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:14.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 shutdown_connections 2024-01-16T15:10:14.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:14.929+0000 7f55cec00640 1 -- 172.21.15.141:0/399449164 wait complete. 2024-01-16T15:10:14.923 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:10:15.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.161+0000 7f8539c00640 1 Processor -- start 2024-01-16T15:10:15.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.162+0000 7f8539c00640 1 -- start start 2024-01-16T15:10:15.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.163+0000 7f8539c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f853415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.163+0000 7f8539c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f853413e360 0x7f85340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.163+0000 7f8539c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853406c470 con 0x7f853415f9c0 2024-01-16T15:10:15.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.163+0000 7f8539c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853406c750 con 0x7f853413e360 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.163+0000 7f8539c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853406ca30 con 0x7f853413df80 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8532000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f853413e360 0x7f85340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8533400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f853415f9c0 0x7f85341408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44746/0 (socket says 172.21.15.141:44746) 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8533400640 1 -- 172.21.15.141:0/2780807989 learned_addr learned my addr 172.21.15.141:0/2780807989 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8532a00640 1 --2- 172.21.15.141:0/2780807989 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f853415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8532000640 1 -- 172.21.15.141:0/2780807989 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 msgr2=0x7f853415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8532000640 1 --2- 172.21.15.141:0/2780807989 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f853415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8532000640 1 -- 172.21.15.141:0/2780807989 >> v1:172.21.15.141:6789/0 conn(0x7f853415f9c0 legacy=0x7f85341408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8532000640 1 -- 172.21.15.141:0/2780807989 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853406cd10 con 0x7f853413e360 2024-01-16T15:10:15.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.164+0000 7f8531600640 1 -- 172.21.15.141:0/2780807989 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129871088 0 0) 0x7f853406c470 con 0x7f853415f9c0 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.165+0000 7f8532000640 1 --2- 172.21.15.141:0/2780807989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853413e360 0x7f85340cf290 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f8524009d90 tx=0x7f852402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418ce3888b29e8cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8531600640 1 -- 172.21.15.141:0/2780807989 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852402cc50 con 0x7f853413e360 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8531600640 1 -- 172.21.15.141:0/2780807989 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852402cdb0 con 0x7f853413e360 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8531600640 1 -- 172.21.15.141:0/2780807989 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524034660 con 0x7f853413e360 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 -- 172.21.15.141:0/2780807989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853413e360 msgr2=0x7f85340cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 --2- 172.21.15.141:0/2780807989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853413e360 0x7f85340cf290 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f8524009d90 tx=0x7f852402bea0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 -- 172.21.15.141:0/2780807989 shutdown_connections 2024-01-16T15:10:15.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 --2- 172.21.15.141:0/2780807989 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853413e360 0x7f85340cf290 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 --2- 172.21.15.141:0/2780807989 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f853415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 -- 172.21.15.141:0/2780807989 >> 172.21.15.141:0/2780807989 conn(0x7f853406a800 msgr2=0x7f853406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:15.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 -- 172.21.15.141:0/2780807989 shutdown_connections 2024-01-16T15:10:15.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.166+0000 7f8539c00640 1 -- 172.21.15.141:0/2780807989 wait complete. 2024-01-16T15:10:15.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.168+0000 7f8539c00640 1 Processor -- start 2024-01-16T15:10:15.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.168+0000 7f8539c00640 1 -- start start 2024-01-16T15:10:15.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.169+0000 7f8539c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f8534150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.169+0000 7f8539c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f853413e360 0x7f853415cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.169+0000 7f8539c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853415f9c0 0x7f853413d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.169+0000 7f8539c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85341f4280 con 0x7f853415f9c0 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.169+0000 7f8539c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534151000 con 0x7f853413e360 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.169+0000 7f8539c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85341f4460 con 0x7f853413df80 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853415f9c0 0x7f853413d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853415f9c0 0x7f853413d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49280/0 (socket says 172.21.15.141:49280) 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 -- 172.21.15.141:0/2628630473 learned_addr learned my addr 172.21.15.141:0/2628630473 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8532a00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f8534150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8532000640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f853413e360 0x7f853415cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 -- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 msgr2=0x7f8534150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f8534150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 -- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f853413e360 msgr2=0x7f853415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f853413e360 0x7f853415cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 -- 172.21.15.141:0/2628630473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524009a70 con 0x7f853415f9c0 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8532000640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f853413e360 0x7f853415cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.170+0000 7f8533400640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853415f9c0 0x7f853413d2c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f852c00c450 tx=0x7f852c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.171+0000 7f852aa00640 1 -- 172.21.15.141:0/2628630473 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0074e0 con 0x7f853415f9c0 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.171+0000 7f852aa00640 1 -- 172.21.15.141:0/2628630473 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c007660 con 0x7f853415f9c0 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.171+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85341f4740 con 0x7f853415f9c0 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.171+0000 7f852aa00640 1 -- 172.21.15.141:0/2628630473 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c004a20 con 0x7f853415f9c0 2024-01-16T15:10:15.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.171+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8534150cb0 con 0x7f853415f9c0 2024-01-16T15:10:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.172+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85341f4c60 con 0x7f853415f9c0 2024-01-16T15:10:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.173+0000 7f852aa00640 1 -- 172.21.15.141:0/2628630473 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f852c004b80 con 0x7f853415f9c0 2024-01-16T15:10:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.174+0000 7f852aa00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850c06c000 0x7f850c06e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.174+0000 7f852aa00640 1 -- 172.21.15.141:0/2628630473 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f852c087b80 con 0x7f853415f9c0 2024-01-16T15:10:15.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.174+0000 7f8532a00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850c06c000 0x7f850c06e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.176+0000 7f8532a00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850c06c000 0x7f850c06e4c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f851c006fd0 tx=0x7f851c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:15.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.178+0000 7f852aa00640 1 -- 172.21.15.141:0/2628630473 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85341f4c60 con 0x7f853415f9c0 2024-01-16T15:10:15.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.368+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f85341f4c60 con 0x7f853415f9c0 2024-01-16T15:10:15.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.373+0000 7f852aa00640 1 -- 172.21.15.141:0/2628630473 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v31)=0 v31) v1 ==== 131+0+0 (secure 0 0 0) 0x7f85341f4c60 con 0x7f853415f9c0 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850c06c000 msgr2=0x7f850c06e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850c06c000 0x7f850c06e4c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f851c006fd0 tx=0x7f851c008040 comp rx=0 tx=0).stop 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853415f9c0 msgr2=0x7f853413d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853415f9c0 0x7f853413d2c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f852c00c450 tx=0x7f852c00c910 comp rx=0 tx=0).stop 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 shutdown_connections 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f850c06c000 0x7f850c06e4c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f853415f9c0 0x7f853413d2c0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f853413e360 0x7f853415cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 --2- 172.21.15.141:0/2628630473 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f853413df80 0x7f8534150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.376+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 >> 172.21.15.141:0/2628630473 conn(0x7f853406a800 msgr2=0x7f853415e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.377+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 shutdown_connections 2024-01-16T15:10:15.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.377+0000 7f8539c00640 1 -- 172.21.15.141:0/2628630473 wait complete. 2024-01-16T15:10:15.372 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:10:15.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.595+0000 7f7ce3600640 1 Processor -- start 2024-01-16T15:10:15.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.595+0000 7f7ce3600640 1 -- start start 2024-01-16T15:10:15.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.596+0000 7f7ce3600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7cdc0cd960 0x7f7cdc0cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.596+0000 7f7ce3600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc165320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.596+0000 7f7ce3600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cdc0ce2a0 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.596+0000 7f7ce3600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cdc0d00a0 con 0x7f7cdc0cd960 2024-01-16T15:10:15.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.596+0000 7f7ce3600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc0b4010 con 0x7f7cdc162f30 2024-01-16T15:10:15.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7ce0c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7cdc15b2a0 0x7f7cdc0cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44756/0 (socket says 172.21.15.141:44756) 2024-01-16T15:10:15.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7ce0c00640 1 -- 172.21.15.141:0/2524580006 learned_addr learned my addr 172.21.15.141:0/2524580006 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7cdbe00640 1 --2- 172.21.15.141:0/2524580006 >> v2:172.21.15.141:3300/0 conn(0x7f7cdc0cd960 0x7f7cdc0cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7ce1600640 1 --2- 172.21.15.141:0/2524580006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc165320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7cdb400640 1 -- 172.21.15.141:0/2524580006 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704469741 0 0) 0x7f7cdc0ce2a0 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7cdb400640 1 -- 172.21.15.141:0/2524580006 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc003520 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7cdbe00640 1 -- 172.21.15.141:0/2524580006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 msgr2=0x7f7cdc165320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7cdbe00640 1 --2- 172.21.15.141:0/2524580006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc165320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7cdbe00640 1 -- 172.21.15.141:0/2524580006 >> v1:172.21.15.141:6789/0 conn(0x7f7cdc15b2a0 legacy=0x7f7cdc0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.597+0000 7f7cdbe00640 1 -- 172.21.15.141:0/2524580006 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0b42f0 con 0x7f7cdc0cd960 2024-01-16T15:10:15.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.598+0000 7f7cdbe00640 1 --2- 172.21.15.141:0/2524580006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc0cd960 0x7f7cdc0cdd60 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009d90 tx=0x7f7ccc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268357b9d7dbf2ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:15.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7cdb400640 1 -- 172.21.15.141:0/2524580006 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc02cc50 con 0x7f7cdc0cd960 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7cdb400640 1 -- 172.21.15.141:0/2524580006 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc02cdb0 con 0x7f7cdc0cd960 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7cdb400640 1 -- 172.21.15.141:0/2524580006 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc034640 con 0x7f7cdc0cd960 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 -- 172.21.15.141:0/2524580006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc0cd960 msgr2=0x7f7cdc0cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2524580006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc0cd960 0x7f7cdc0cdd60 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009d90 tx=0x7f7ccc02bea0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 -- 172.21.15.141:0/2524580006 shutdown_connections 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2524580006 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc165320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2524580006 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc0cd960 0x7f7cdc0cdd60 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 -- 172.21.15.141:0/2524580006 >> 172.21.15.141:0/2524580006 conn(0x7f7cdc0d1020 msgr2=0x7f7cdc0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 -- 172.21.15.141:0/2524580006 shutdown_connections 2024-01-16T15:10:15.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.599+0000 7f7ce3600640 1 -- 172.21.15.141:0/2524580006 wait complete. 2024-01-16T15:10:15.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.601+0000 7f7ce3600640 1 Processor -- start 2024-01-16T15:10:15.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.601+0000 7f7ce3600640 1 -- start start 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7ce3600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cdc0cd960 0x7f7cdc1550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7ce3600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc15b2a0 0x7f7cdc09daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7ce3600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc09bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7ce3600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc15e170 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7ce3600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc155a00 con 0x7f7cdc0cd960 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7ce3600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc09ca00 con 0x7f7cdc162f30 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc15b2a0 0x7f7cdc09daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc15b2a0 0x7f7cdc09daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49308/0 (socket says 172.21.15.141:49308) 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.602+0000 7f7cdbe00640 1 -- 172.21.15.141:0/2807055302 learned_addr learned my addr 172.21.15.141:0/2807055302 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7ce0c00640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cdc0cd960 0x7f7cdc1550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7ce1600640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc09bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7cdbe00640 1 -- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 msgr2=0x7f7cdc09bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7cdbe00640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc09bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7cdbe00640 1 -- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cdc0cd960 msgr2=0x7f7cdc1550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7cdbe00640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cdc0cd960 0x7f7cdc1550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7cdbe00640 1 -- 172.21.15.141:0/2807055302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc009a70 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.603+0000 7f7cdbe00640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc15b2a0 0x7f7cdc09daa0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009eb0 tx=0x7f7ccc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:15.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.604+0000 7f7cd8c00640 1 -- 172.21.15.141:0/2807055302 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc03e610 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.604+0000 7f7cd8c00640 1 -- 172.21.15.141:0/2807055302 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc03e770 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.604+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cdc09cce0 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.604+0000 7f7cd8c00640 1 -- 172.21.15.141:0/2807055302 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc03eb20 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.604+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc09d2a0 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.605+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cdc0675d0 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.606+0000 7f7cd8c00640 1 -- 172.21.15.141:0/2807055302 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7ccc03b050 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.607+0000 7f7cd8c00640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cb406bfb0 0x7f7cb406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:15.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.607+0000 7f7cd8c00640 1 -- 172.21.15.141:0/2807055302 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ccc0aaf70 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.608+0000 7f7ce0c00640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cb406bfb0 0x7f7cb406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:15.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.609+0000 7f7ce0c00640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cb406bfb0 0x7f7cb406e470 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4005880 tx=0x7f7cc4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:15.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.611+0000 7f7cd8c00640 1 -- 172.21.15.141:0/2807055302 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ccc077830 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.803+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f7cdc0cdd40 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.809+0000 7f7cd8c00640 1 -- 172.21.15.141:0/2807055302 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v32)=0 v32) v1 ==== 137+0+0 (secure 0 0 0) 0x7f7ccc0390d0 con 0x7f7cdc15b2a0 2024-01-16T15:10:15.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cb406bfb0 msgr2=0x7f7cb406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cb406bfb0 0x7f7cb406e470 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4005880 tx=0x7f7cc4009040 comp rx=0 tx=0).stop 2024-01-16T15:10:15.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc15b2a0 msgr2=0x7f7cdc09daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:15.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc15b2a0 0x7f7cdc09daa0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009eb0 tx=0x7f7ccc004a40 comp rx=0 tx=0).stop 2024-01-16T15:10:15.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 shutdown_connections 2024-01-16T15:10:15.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cb406bfb0 0x7f7cb406e470 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7cdc162f30 0x7f7cdc09bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7cdc15b2a0 0x7f7cdc09daa0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 --2- 172.21.15.141:0/2807055302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cdc0cd960 0x7f7cdc1550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:15.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 >> 172.21.15.141:0/2807055302 conn(0x7f7cdc0d1020 msgr2=0x7f7cdc0b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:15.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 shutdown_connections 2024-01-16T15:10:15.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:15.812+0000 7f7ce3600640 1 -- 172.21.15.141:0/2807055302 wait complete. 2024-01-16T15:10:15.808 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:10:16.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.044+0000 7f5363400640 1 Processor -- start 2024-01-16T15:10:16.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.045+0000 7f5363400640 1 -- start start 2024-01-16T15:10:16.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.046+0000 7f5363400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f535c13df80 0x7f535c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.046+0000 7f5363400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c06c470 con 0x7f535c13e360 2024-01-16T15:10:16.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5363400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c06c750 con 0x7f535c0cf9a0 2024-01-16T15:10:16.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5363400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535c06ca30 con 0x7f535c13df80 2024-01-16T15:10:16.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5360a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f535c13df80 0x7f535c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5353c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f535c13e360 0x7f535c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44770/0 (socket says 172.21.15.141:44770) 2024-01-16T15:10:16.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5353c00640 1 -- 172.21.15.141:0/3314931285 learned_addr learned my addr 172.21.15.141:0/3314931285 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:16.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5360a00640 1 -- 172.21.15.141:0/3314931285 >> v1:172.21.15.188:6789/0 conn(0x7f535c0cf9a0 legacy=0x7f535c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:16.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5360a00640 1 -- 172.21.15.141:0/3314931285 >> v1:172.21.15.141:6789/0 conn(0x7f535c13e360 legacy=0x7f535c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.047+0000 7f5360a00640 1 -- 172.21.15.141:0/3314931285 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c06cd10 con 0x7f535c13df80 2024-01-16T15:10:16.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.049+0000 7f5360a00640 1 --2- 172.21.15.141:0/3314931285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c13df80 0x7f535c15bee0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f534c0097a0 tx=0x7f534c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e36bf39ffe38b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.049+0000 7f535be00640 1 -- 172.21.15.141:0/3314931285 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c031830 con 0x7f535c13df80 2024-01-16T15:10:16.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.049+0000 7f535be00640 1 -- 172.21.15.141:0/3314931285 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534c031990 con 0x7f535c13df80 2024-01-16T15:10:16.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.049+0000 7f535be00640 1 -- 172.21.15.141:0/3314931285 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c031d20 con 0x7f535c13df80 2024-01-16T15:10:16.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.049+0000 7f5363400640 1 -- 172.21.15.141:0/3314931285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c13df80 msgr2=0x7f535c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.049+0000 7f5363400640 1 --2- 172.21.15.141:0/3314931285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c13df80 0x7f535c15bee0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f534c0097a0 tx=0x7f534c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.050+0000 7f5363400640 1 -- 172.21.15.141:0/3314931285 shutdown_connections 2024-01-16T15:10:16.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.050+0000 7f5363400640 1 --2- 172.21.15.141:0/3314931285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c13df80 0x7f535c15bee0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.050+0000 7f5363400640 1 -- 172.21.15.141:0/3314931285 >> 172.21.15.141:0/3314931285 conn(0x7f535c06a800 msgr2=0x7f535c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:16.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.050+0000 7f5363400640 1 -- 172.21.15.141:0/3314931285 shutdown_connections 2024-01-16T15:10:16.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.050+0000 7f5363400640 1 -- 172.21.15.141:0/3314931285 wait complete. 2024-01-16T15:10:16.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.051+0000 7f5363400640 1 Processor -- start 2024-01-16T15:10:16.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.052+0000 7f5363400640 1 -- start start 2024-01-16T15:10:16.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5363400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c0cf9a0 0x7f535c0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5363400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f535c13df80 0x7f535c1f17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5363400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f535c13e360 0x7f535c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5363400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c1f1d20 con 0x7f535c0cf9a0 2024-01-16T15:10:16.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5363400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c1ee770 con 0x7f535c13df80 2024-01-16T15:10:16.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5363400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535c157800 con 0x7f535c13e360 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c0cf9a0 0x7f535c0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c0cf9a0 0x7f535c0a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49314/0 (socket says 172.21.15.141:49314) 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 -- 172.21.15.141:0/2412029118 learned_addr learned my addr 172.21.15.141:0/2412029118 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 -- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f535c13e360 msgr2=0x7f535c159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f535c13e360 0x7f535c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 -- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f535c13df80 msgr2=0x7f535c1f17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f535c13df80 0x7f535c1f17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.053+0000 7f5360a00640 1 -- 172.21.15.141:0/2412029118 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c009400 con 0x7f535c0cf9a0 2024-01-16T15:10:16.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.054+0000 7f5360a00640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c0cf9a0 0x7f535c0a84c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f534c0098c0 tx=0x7f534c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.054+0000 7f5359600640 1 -- 172.21.15.141:0/2412029118 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c03ba70 con 0x7f535c0cf9a0 2024-01-16T15:10:16.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.054+0000 7f5359600640 1 -- 172.21.15.141:0/2412029118 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534c03bbd0 con 0x7f535c0cf9a0 2024-01-16T15:10:16.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.054+0000 7f5359600640 1 -- 172.21.15.141:0/2412029118 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c0453f0 con 0x7f535c0cf9a0 2024-01-16T15:10:16.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.054+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535c0a8b40 con 0x7f535c0cf9a0 2024-01-16T15:10:16.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.054+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535c159660 con 0x7f535c0cf9a0 2024-01-16T15:10:16.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.055+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535c0675d0 con 0x7f535c0cf9a0 2024-01-16T15:10:16.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.056+0000 7f5359600640 1 -- 172.21.15.141:0/2412029118 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f534c04e450 con 0x7f535c0cf9a0 2024-01-16T15:10:16.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.057+0000 7f5359600640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f533406c030 0x7f533406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.057+0000 7f5359600640 1 -- 172.21.15.141:0/2412029118 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f534c0aa830 con 0x7f535c0cf9a0 2024-01-16T15:10:16.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.058+0000 7f5353c00640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f533406c030 0x7f533406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.059+0000 7f5353c00640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f533406c030 0x7f533406e4f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f535c15ad90 tx=0x7f5344007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.061+0000 7f5359600640 1 -- 172.21.15.141:0/2412029118 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f534c0770f0 con 0x7f535c0cf9a0 2024-01-16T15:10:16.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.280+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f535c15f9c0 con 0x7f533406c030 2024-01-16T15:10:16.269 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:10:16.293+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:16.269 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:10:16.293+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:16.269 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:10:16.293+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:16.269 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:10:16.293+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:16.269 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:10:16.293+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:16.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.368+0000 7f5359600640 1 -- 172.21.15.141:0/2412029118 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f53440012b0 con 0x7f533406c030 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f533406c030 msgr2=0x7f533406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f533406c030 0x7f533406e4f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f535c15ad90 tx=0x7f5344007a40 comp rx=0 tx=0).stop 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c0cf9a0 msgr2=0x7f535c0a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c0cf9a0 0x7f535c0a84c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f534c0098c0 tx=0x7f534c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 shutdown_connections 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f533406c030 0x7f533406e4f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f535c13e360 0x7f535c159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f535c13df80 0x7f535c1f17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 --2- 172.21.15.141:0/2412029118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f535c0cf9a0 0x7f535c0a84c0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 >> 172.21.15.141:0/2412029118 conn(0x7f535c06a800 msgr2=0x7f535c15d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:16.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 shutdown_connections 2024-01-16T15:10:16.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.371+0000 7f5363400640 1 -- 172.21.15.141:0/2412029118 wait complete. 2024-01-16T15:10:16.370 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:10:16.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.601+0000 7fb3eb200640 1 Processor -- start 2024-01-16T15:10:16.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.601+0000 7fb3eb200640 1 -- start start 2024-01-16T15:10:16.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.602+0000 7fb3eb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 0x7fb3ec0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3eb200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb3ec0cdc20 0x7fb3ec0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3eb200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec0ce440 con 0x7fb3ec13cf10 2024-01-16T15:10:16.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3eb200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ec0ce720 con 0x7fb3ec0cdc20 2024-01-16T15:10:16.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3eb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec0cea00 con 0x7fb3ec13fcb0 2024-01-16T15:10:16.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3e9e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb3ec13cf10 0x7fb3ec13f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44786/0 (socket says 172.21.15.141:44786) 2024-01-16T15:10:16.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3e9e00640 1 -- 172.21.15.141:0/3504332384 learned_addr learned my addr 172.21.15.141:0/3504332384 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914110645 0 0) 0x7fb3ec0ce440 con 0x7fb3ec13cf10 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3cc003520 con 0x7fb3ec13cf10 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3e9400640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 0x7fb3ec0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.603+0000 7fb3ea800640 1 --2- 172.21.15.141:0/3504332384 >> v2:172.21.15.188:3300/0 conn(0x7fb3ec0cdc20 0x7fb3ec0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351932555 0 0) 0x7fb3cc003520 con 0x7fb3ec13cf10 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 msgr2=0x7fb3ec0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 0x7fb3ec0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 >> v2:172.21.15.188:3300/0 conn(0x7fb3ec0cdc20 msgr2=0x7fb3ec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:16.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 --2- 172.21.15.141:0/3504332384 >> v2:172.21.15.188:3300/0 conn(0x7fb3ec0cdc20 0x7fb3ec0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec0b1d90 con 0x7fb3ec13cf10 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb3dc001670 con 0x7fb3ec13cf10 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 >> v1:172.21.15.141:6789/0 conn(0x7fb3ec13cf10 legacy=0x7fb3ec13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 0x7fb3cc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3cc004d00 0x7fb3cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec0ce720 con 0x7fb3cc003f40 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec0cea00 con 0x7fb3cc004d00 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc00ba30 con 0x7fb3ec13fcb0 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e9400640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.604+0000 7fb3e9e00640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 0x7fb3cc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.605+0000 7fb3e9400640 1 -- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 msgr2=0x7fb3cc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.605+0000 7fb3ea800640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3cc004d00 0x7fb3cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.605+0000 7fb3e9400640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec13fcb0 0x7fb3cc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.605+0000 7fb3e9400640 1 -- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3cc004d00 msgr2=0x7fb3cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:16.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.605+0000 7fb3e9400640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3cc004d00 0x7fb3cc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.605+0000 7fb3e9400640 1 -- 172.21.15.141:0/3504332384 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc003520 con 0x7fb3cc003f40 2024-01-16T15:10:16.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.606+0000 7fb3e9400640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3cc0047c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009650 tx=0x7fb3cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5833af12f188b590 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.606+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d402e8c0 con 0x7fb3cc003f40 2024-01-16T15:10:16.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.606+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d402ea20 con 0x7fb3cc003f40 2024-01-16T15:10:16.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.606+0000 7fb3e8a00640 1 -- 172.21.15.141:0/3504332384 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d402edb0 con 0x7fb3cc003f40 2024-01-16T15:10:16.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.606+0000 7fb3eb200640 1 -- 172.21.15.141:0/3504332384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 msgr2=0x7fb3cc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.606+0000 7fb3eb200640 1 --2- 172.21.15.141:0/3504332384 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3cc0047c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009650 tx=0x7fb3cc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.607+0000 7fb3ea800640 1 -- 172.21.15.141:0/3504332384 reap_dead start 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.607+0000 7fb3eb200640 1 -- 172.21.15.141:0/3504332384 shutdown_connections 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.607+0000 7fb3eb200640 1 -- 172.21.15.141:0/3504332384 >> 172.21.15.141:0/3504332384 conn(0x7fb3ec06a7d0 msgr2=0x7fb3ec06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.607+0000 7fb3eb200640 1 -- 172.21.15.141:0/3504332384 shutdown_connections 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.607+0000 7fb3eb200640 1 -- 172.21.15.141:0/3504332384 wait complete. 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.609+0000 7fb3eb200640 1 Processor -- start 2024-01-16T15:10:16.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.609+0000 7fb3eb200640 1 -- start start 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3eb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3ec070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3eb200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3ec152cb0 0x7fb3ec13b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3eb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec15c820 0x7fb3ec156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3eb200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec0708a0 con 0x7fb3cc003f40 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3eb200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec13bf80 con 0x7fb3ec152cb0 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3eb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec070290 con 0x7fb3ec15c820 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3ec070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3ec070d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49328/0 (socket says 172.21.15.141:49328) 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 -- 172.21.15.141:0/2246195593 learned_addr learned my addr 172.21.15.141:0/2246195593 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 -- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec15c820 msgr2=0x7fb3ec156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec15c820 0x7fb3ec156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 -- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3ec152cb0 msgr2=0x7fb3ec13b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3ec152cb0 0x7fb3ec13b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.610+0000 7fb3e9e00640 1 -- 172.21.15.141:0/2246195593 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec13bca0 con 0x7fb3cc003f40 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.611+0000 7fb3e9e00640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3ec070d30 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc001ae0 tx=0x7fb3dc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.611+0000 7fb3e2000640 1 -- 172.21.15.141:0/2246195593 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc00c850 con 0x7fb3cc003f40 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.611+0000 7fb3e2000640 1 -- 172.21.15.141:0/2246195593 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc00c9b0 con 0x7fb3cc003f40 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.611+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec070530 con 0x7fb3cc003f40 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.611+0000 7fb3e2000640 1 -- 172.21.15.141:0/2246195593 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc00cd40 con 0x7fb3cc003f40 2024-01-16T15:10:16.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.611+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec13b900 con 0x7fb3cc003f40 2024-01-16T15:10:16.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.612+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec067450 con 0x7fb3cc003f40 2024-01-16T15:10:16.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.613+0000 7fb3e2000640 1 -- 172.21.15.141:0/2246195593 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fb3dc01d460 con 0x7fb3cc003f40 2024-01-16T15:10:16.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.614+0000 7fb3e2000640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb3c406bfb0 0x7fb3c406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.614+0000 7fb3e2000640 1 -- 172.21.15.141:0/2246195593 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3dc088900 con 0x7fb3cc003f40 2024-01-16T15:10:16.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.615+0000 7fb3e9400640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb3c406bfb0 0x7fb3c406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.616+0000 7fb3e9400640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb3c406bfb0 0x7fb3c406e470 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4038000 tx=0x7fb3d4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.619+0000 7fb3e2000640 1 -- 172.21.15.141:0/2246195593 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3dc054f70 con 0x7fb3cc003f40 2024-01-16T15:10:16.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.837+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3ec0ce0b0 con 0x7fb3c406bfb0 2024-01-16T15:10:16.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.849+0000 7fb3e2000640 1 -- 172.21.15.141:0/2246195593 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fb3d402eb90 con 0x7fb3c406bfb0 2024-01-16T15:10:16.826 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb3c406bfb0 msgr2=0x7fb3c406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb3c406bfb0 0x7fb3c406e470 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4038000 tx=0x7fb3d4030000 comp rx=0 tx=0).stop 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 msgr2=0x7fb3ec070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3ec070d30 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc001ae0 tx=0x7fb3dc00ecd0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 shutdown_connections 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb3c406bfb0 0x7fb3c406e470 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3ec15c820 0x7fb3ec156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3ec152cb0 0x7fb3ec13b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 --2- 172.21.15.141:0/2246195593 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3cc003f40 0x7fb3ec070d30 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 >> 172.21.15.141:0/2246195593 conn(0x7fb3ec06a7d0 msgr2=0x7fb3ec0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 shutdown_connections 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:16.852+0000 7fb3eb200640 1 -- 172.21.15.141:0/2246195593 wait complete. 2024-01-16T15:10:16.849 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:10:17.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.092+0000 7ffac8600640 1 Processor -- start 2024-01-16T15:10:17.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.092+0000 7ffac8600640 1 -- start start 2024-01-16T15:10:17.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.094+0000 7ffac8600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ffac00cf9a0 0x7ffac015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:17.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.094+0000 7ffac8600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0070c40 con 0x7ffac00ae5a0 2024-01-16T15:10:17.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.094+0000 7ffac8600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac006c470 con 0x7ffac00b3ae0 2024-01-16T15:10:17.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.094+0000 7ffac8600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac0071450 con 0x7ffac00cf9a0 2024-01-16T15:10:17.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.094+0000 7ffac5c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ffac00ae5a0 0x7ffac006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44800/0 (socket says 172.21.15.141:44800) 2024-01-16T15:10:17.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.094+0000 7ffac5c00640 1 -- 172.21.15.141:0/2112141073 learned_addr learned my addr 172.21.15.141:0/2112141073 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:17.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.094+0000 7ffac6600640 1 --2- 172.21.15.141:0/2112141073 >> v2:172.21.15.141:3300/0 conn(0x7ffac00cf9a0 0x7ffac015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:17.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.095+0000 7ffabfe00640 1 -- 172.21.15.141:0/2112141073 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273510920 0 0) 0x7ffac0070c40 con 0x7ffac00ae5a0 2024-01-16T15:10:17.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.095+0000 7ffabfe00640 1 -- 172.21.15.141:0/2112141073 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa0003520 con 0x7ffac00ae5a0 2024-01-16T15:10:17.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.095+0000 7ffac6600640 1 -- 172.21.15.141:0/2112141073 >> v1:172.21.15.188:6789/0 conn(0x7ffac00b3ae0 legacy=0x7ffac00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:17.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.095+0000 7ffac6600640 1 -- 172.21.15.141:0/2112141073 >> v1:172.21.15.141:6789/0 conn(0x7ffac00ae5a0 legacy=0x7ffac006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:17.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.095+0000 7ffac6600640 1 -- 172.21.15.141:0/2112141073 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac006c750 con 0x7ffac00cf9a0 2024-01-16T15:10:17.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.096+0000 7ffac6600640 1 --2- 172.21.15.141:0/2112141073 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00cf9a0 0x7ffac015e430 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256ed42088969c97 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:17.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.096+0000 7ffabfe00640 1 -- 172.21.15.141:0/2112141073 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab802f810 con 0x7ffac00cf9a0 2024-01-16T15:10:17.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.096+0000 7ffabfe00640 1 -- 172.21.15.141:0/2112141073 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab802f970 con 0x7ffac00cf9a0 2024-01-16T15:10:17.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.096+0000 7ffabfe00640 1 -- 172.21.15.141:0/2112141073 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab802fd00 con 0x7ffac00cf9a0 2024-01-16T15:10:17.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.097+0000 7ffac8600640 1 -- 172.21.15.141:0/2112141073 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00cf9a0 msgr2=0x7ffac015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:17.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.097+0000 7ffac8600640 1 --2- 172.21.15.141:0/2112141073 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00cf9a0 0x7ffac015e430 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b9f0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.097+0000 7ffac8600640 1 -- 172.21.15.141:0/2112141073 shutdown_connections 2024-01-16T15:10:17.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.097+0000 7ffac8600640 1 --2- 172.21.15.141:0/2112141073 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00cf9a0 0x7ffac015e430 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.097+0000 7ffac8600640 1 -- 172.21.15.141:0/2112141073 >> 172.21.15.141:0/2112141073 conn(0x7ffac006a800 msgr2=0x7ffac006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:17.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.097+0000 7ffac8600640 1 -- 172.21.15.141:0/2112141073 shutdown_connections 2024-01-16T15:10:17.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.097+0000 7ffac8600640 1 -- 172.21.15.141:0/2112141073 wait complete. 2024-01-16T15:10:17.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.099+0000 7ffac8600640 1 Processor -- start 2024-01-16T15:10:17.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.099+0000 7ffac8600640 1 -- start start 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac8600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac00ae5a0 0x7ffac015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac8600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00b3ae0 0x7ffac00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac8600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac00cf9a0 0x7ffac00ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac8600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac0156290 con 0x7ffac00b3ae0 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac8600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00cec90 con 0x7ffac00ae5a0 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac8600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00ce6d0 con 0x7ffac00cf9a0 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00b3ae0 0x7ffac00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac5200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00b3ae0 0x7ffac00cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49348/0 (socket says 172.21.15.141:49348) 2024-01-16T15:10:17.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac5200640 1 -- 172.21.15.141:0/3546458356 learned_addr learned my addr 172.21.15.141:0/3546458356 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:17.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.100+0000 7ffac5c00640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac00ae5a0 0x7ffac015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:17.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac6600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac00cf9a0 0x7ffac00ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac5200640 1 -- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac00cf9a0 msgr2=0x7ffac00ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac5200640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac00cf9a0 0x7ffac00ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac5200640 1 -- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac00ae5a0 msgr2=0x7ffac015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac5200640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac00ae5a0 0x7ffac015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac5200640 1 -- 172.21.15.141:0/3546458356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8009400 con 0x7ffac00b3ae0 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac5c00640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac00ae5a0 0x7ffac015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac5200640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00b3ae0 0x7ffac00cdba0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c870 tx=0x7ffaa800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffabd600640 1 -- 172.21.15.141:0/3546458356 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8003b70 con 0x7ffac00b3ae0 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffabd600640 1 -- 172.21.15.141:0/3546458356 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8003cf0 con 0x7ffac00b3ae0 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffabd600640 1 -- 172.21.15.141:0/3546458356 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa800aa90 con 0x7ffac00b3ae0 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac00ce9b0 con 0x7ffac00b3ae0 2024-01-16T15:10:17.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.101+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac0156830 con 0x7ffac00b3ae0 2024-01-16T15:10:17.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.102+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac0156a50 con 0x7ffac00b3ae0 2024-01-16T15:10:17.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.103+0000 7ffabd600640 1 -- 172.21.15.141:0/3546458356 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ffaa800abf0 con 0x7ffac00b3ae0 2024-01-16T15:10:17.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.105+0000 7ffabd600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9806bfb0 0x7ffa9806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:17.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.105+0000 7ffabd600640 1 -- 172.21.15.141:0/3546458356 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffaa8088520 con 0x7ffac00b3ae0 2024-01-16T15:10:17.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.105+0000 7ffac5c00640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9806bfb0 0x7ffa9806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:17.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.106+0000 7ffac5c00640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9806bfb0 0x7ffa9806e470 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ffab00011e0 tx=0x7ffab0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:17.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.109+0000 7ffabd600640 1 -- 172.21.15.141:0/3546458356 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffaa8054de0 con 0x7ffac00b3ae0 2024-01-16T15:10:17.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.326+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffac0070550 con 0x7ffa9806bfb0 2024-01-16T15:10:17.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.338+0000 7ffabd600640 1 -- 172.21.15.141:0/3546458356 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7ffab00013f0 con 0x7ffa9806bfb0 2024-01-16T15:10:17.315 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9806bfb0 msgr2=0x7ffa9806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9806bfb0 0x7ffa9806e470 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ffab00011e0 tx=0x7ffab0009040 comp rx=0 tx=0).stop 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00b3ae0 msgr2=0x7ffac00cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00b3ae0 0x7ffac00cdba0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c870 tx=0x7ffaa800cd30 comp rx=0 tx=0).stop 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 shutdown_connections 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa9806bfb0 0x7ffa9806e470 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffac00cf9a0 0x7ffac00ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffac00b3ae0 0x7ffac00cdba0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 --2- 172.21.15.141:0/3546458356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffac00ae5a0 0x7ffac015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.340+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 >> 172.21.15.141:0/3546458356 conn(0x7ffac006a800 msgr2=0x7ffac00b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.341+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 shutdown_connections 2024-01-16T15:10:17.318 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:17.341+0000 7ffac8600640 1 -- 172.21.15.141:0/3546458356 wait complete. 2024-01-16T15:10:17.337 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac 2024-01-16T15:10:17.338 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:10:17.338 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:10:17.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:17.387 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:10:31.896 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:10:31.896 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:10:31.896 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.3883 s, 74.6 MB/s 2024-01-16T15:10:31.898 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:10:31.898 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:10:31.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:31.951 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:10:36.021 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:36.033 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:43.834 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:10:43.835 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:10:43.835 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.8321 s, 90.7 MB/s 2024-01-16T15:10:43.836 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:10:43.836 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:10:43.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:43.888 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:10:55.396 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:10:55.396 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:10:55.396 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.4551 s, 93.7 MB/s 2024-01-16T15:10:55.397 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:10:55.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.631+0000 7f645f800640 1 Processor -- start 2024-01-16T15:10:55.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.631+0000 7f645f800640 1 -- start start 2024-01-16T15:10:55.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.633+0000 7f645f800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64580b3ae0 0x7f645815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:55.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.633+0000 7f645f800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6458070c40 con 0x7f64580cf290 2024-01-16T15:10:55.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.633+0000 7f645f800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645806c470 con 0x7f64580ae5a0 2024-01-16T15:10:55.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.633+0000 7f645f800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6458071450 con 0x7f64580b3ae0 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f645d800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f64580cf290 0x7f64580b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58644/0 (socket says 172.21.15.141:58644) 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f6457e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64580b3ae0 0x7f645815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f645d800640 1 -- 172.21.15.141:0/3369060134 learned_addr learned my addr 172.21.15.141:0/3369060134 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f6457400640 1 -- 172.21.15.141:0/3369060134 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197363268 0 0) 0x7f6458070c40 con 0x7f64580cf290 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f6457400640 1 -- 172.21.15.141:0/3369060134 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438003520 con 0x7f64580cf290 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f6457e00640 1 -- 172.21.15.141:0/3369060134 >> v1:172.21.15.188:6789/0 conn(0x7f64580ae5a0 legacy=0x7f645806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f6457e00640 1 -- 172.21.15.141:0/3369060134 >> v1:172.21.15.141:6789/0 conn(0x7f64580cf290 legacy=0x7f64580b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:55.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.634+0000 7f6457e00640 1 -- 172.21.15.141:0/3369060134 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645806c750 con 0x7f64580b3ae0 2024-01-16T15:10:55.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.635+0000 7f6457e00640 1 --2- 172.21.15.141:0/3369060134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580b3ae0 0x7f645815e430 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f64400097a0 tx=0x7f644002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb5b1afb075f132 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:55.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.636+0000 7f6457400640 1 -- 172.21.15.141:0/3369060134 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644002f810 con 0x7f64580b3ae0 2024-01-16T15:10:55.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.636+0000 7f6457400640 1 -- 172.21.15.141:0/3369060134 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644002f970 con 0x7f64580b3ae0 2024-01-16T15:10:55.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.636+0000 7f6457400640 1 -- 172.21.15.141:0/3369060134 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644002fd00 con 0x7f64580b3ae0 2024-01-16T15:10:55.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.636+0000 7f645f800640 1 -- 172.21.15.141:0/3369060134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580b3ae0 msgr2=0x7f645815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:55.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.636+0000 7f645f800640 1 --2- 172.21.15.141:0/3369060134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580b3ae0 0x7f645815e430 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f64400097a0 tx=0x7f644002b9f0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.637+0000 7f645f800640 1 -- 172.21.15.141:0/3369060134 shutdown_connections 2024-01-16T15:10:55.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.637+0000 7f645f800640 1 --2- 172.21.15.141:0/3369060134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580b3ae0 0x7f645815e430 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.637+0000 7f645f800640 1 -- 172.21.15.141:0/3369060134 >> 172.21.15.141:0/3369060134 conn(0x7f645806a800 msgr2=0x7f645806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:55.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.637+0000 7f645f800640 1 -- 172.21.15.141:0/3369060134 shutdown_connections 2024-01-16T15:10:55.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.637+0000 7f645f800640 1 -- 172.21.15.141:0/3369060134 wait complete. 2024-01-16T15:10:55.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.638+0000 7f645f800640 1 Processor -- start 2024-01-16T15:10:55.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.638+0000 7f645f800640 1 -- start start 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.639+0000 7f645f800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64580ae5a0 0x7f64580b0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.639+0000 7f645f800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64580b3ae0 0x7f64581594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.639+0000 7f645f800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580cf290 0x7f64581599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.639+0000 7f645f800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64581527c0 con 0x7f64580cf290 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.639+0000 7f645f800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6458150330 con 0x7f64580b3ae0 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.639+0000 7f645f800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64580b09a0 con 0x7f64580ae5a0 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580cf290 0x7f64581599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:55.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580cf290 0x7f64581599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53034/0 (socket says 172.21.15.141:53034) 2024-01-16T15:10:55.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 -- 172.21.15.141:0/49586549 learned_addr learned my addr 172.21.15.141:0/49586549 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f6457e00640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64580b3ae0 0x7f64581594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 -- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64580ae5a0 msgr2=0x7f64580b0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64580ae5a0 0x7f64580b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 -- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64580b3ae0 msgr2=0x7f64581594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64580b3ae0 0x7f64581594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.640+0000 7f645d800640 1 -- 172.21.15.141:0/49586549 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6440009400 con 0x7f64580cf290 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.641+0000 7f645d800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580cf290 0x7f64581599e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f645000cad0 tx=0x7f645000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.641+0000 7f6454c00640 1 -- 172.21.15.141:0/49586549 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000ac30 con 0x7f64580cf290 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.641+0000 7f6454c00640 1 -- 172.21.15.141:0/49586549 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645000ad90 con 0x7f64580cf290 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.641+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64581f4020 con 0x7f64580cf290 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.641+0000 7f6454c00640 1 -- 172.21.15.141:0/49586549 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450012620 con 0x7f64580cf290 2024-01-16T15:10:55.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.641+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6458150860 con 0x7f64580cf290 2024-01-16T15:10:55.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.642+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64580675d0 con 0x7f64580cf290 2024-01-16T15:10:55.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.643+0000 7f6454c00640 1 -- 172.21.15.141:0/49586549 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 30) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f6450007070 con 0x7f64580cf290 2024-01-16T15:10:55.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.644+0000 7f6454c00640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f643006c070 0x7f643006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:55.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.644+0000 7f6454c00640 1 -- 172.21.15.141:0/49586549 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6450088850 con 0x7f64580cf290 2024-01-16T15:10:55.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.645+0000 7f645ce00640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f643006c070 0x7f643006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:55.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.646+0000 7f645ce00640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f643006c070 0x7f643006e530 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6448000f60 tx=0x7f64480079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:55.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.649+0000 7f6454c00640 1 -- 172.21.15.141:0/49586549 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6450054ea0 con 0x7f64580cf290 2024-01-16T15:10:55.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.869+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6458070550 con 0x7f643006c070 2024-01-16T15:10:55.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.902+0000 7f6454c00640 1 -- 172.21.15.141:0/49586549 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6448005080 con 0x7f643006c070 2024-01-16T15:10:55.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f643006c070 msgr2=0x7f643006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:55.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f643006c070 0x7f643006e530 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6448000f60 tx=0x7f64480079b0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580cf290 msgr2=0x7f64581599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:55.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580cf290 0x7f64581599e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f645000cad0 tx=0x7f645000cf90 comp rx=0 tx=0).stop 2024-01-16T15:10:55.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 shutdown_connections 2024-01-16T15:10:55.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f643006c070 0x7f643006e530 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64580cf290 0x7f64581599e0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64580b3ae0 0x7f64581594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 --2- 172.21.15.141:0/49586549 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64580ae5a0 0x7f64580b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:55.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.905+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 >> 172.21.15.141:0/49586549 conn(0x7f645806a800 msgr2=0x7f64580b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:55.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.906+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 shutdown_connections 2024-01-16T15:10:55.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:55.906+0000 7f645f800640 1 -- 172.21.15.141:0/49586549 wait complete. 2024-01-16T15:10:55.902 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:10:56.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.136+0000 7f3e6e600640 1 Processor -- start 2024-01-16T15:10:56.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.137+0000 7f3e6e600640 1 -- start start 2024-01-16T15:10:56.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.138+0000 7f3e6e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 0x7f3e680d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.138+0000 7f3e6e600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3e680b41e0 0x7f3e680cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.138+0000 7f3e6e600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e680d0320 con 0x7f3e680cf9a0 2024-01-16T15:10:56.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.138+0000 7f3e6e600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e680b2730 con 0x7f3e680b41e0 2024-01-16T15:10:56.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.138+0000 7f3e6e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680d1ec0 con 0x7f3e6815d4b0 2024-01-16T15:10:56.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.138+0000 7f3e67e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3e680cf9a0 0x7f3e680cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58648/0 (socket says 172.21.15.141:58648) 2024-01-16T15:10:56.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.138+0000 7f3e67e00640 1 -- 172.21.15.141:0/197756690 learned_addr learned my addr 172.21.15.141:0/197756690 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:10:56.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/197756690 >> v2:172.21.15.188:3300/0 conn(0x7f3e680b41e0 0x7f3e680cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e6d200640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 0x7f3e680d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259606478 0 0) 0x7f3e680d0320 con 0x7f3e680cf9a0 2024-01-16T15:10:56.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e48003520 con 0x7f3e680cf9a0 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797375398 0 0) 0x7f3e48003520 con 0x7f3e680cf9a0 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 msgr2=0x7f3e680d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 0x7f3e680d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 >> v2:172.21.15.188:3300/0 conn(0x7f3e680b41e0 msgr2=0x7f3e680cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 --2- 172.21.15.141:0/197756690 >> v2:172.21.15.188:3300/0 conn(0x7f3e680b41e0 0x7f3e680cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e680b29d0 con 0x7f3e680cf9a0 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3e50001670 con 0x7f3e680cf9a0 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 >> v1:172.21.15.141:6789/0 conn(0x7f3e680cf9a0 legacy=0x7f3e680cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e48003970 0x7f3e48003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.139+0000 7f3e67400640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 0x7f3e480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e67400640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e48004d00 0x7f3e48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680b2730 con 0x7f3e48004d00 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680d1ec0 con 0x7f3e48003970 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4800b980 con 0x7f3e6815d4b0 2024-01-16T15:10:56.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e48004d00 0x7f3e48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6d200640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e48003970 0x7f3e48003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e67e00640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 0x7f3e480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6dc00640 1 -- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 msgr2=0x7f3e480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6815d4b0 0x7f3e480047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6dc00640 1 -- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e48003970 msgr2=0x7f3e48003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e48003970 0x7f3e48003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6dc00640 1 -- 172.21.15.141:0/197756690 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48003520 con 0x7f3e48004d00 2024-01-16T15:10:56.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.140+0000 7f3e6d200640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e48003970 0x7f3e48003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:10:56.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.141+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e48004d00 0x7f3e48005180 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009fc0 tx=0x7f3e480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6adef4b4fbe9d7e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:56.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6002cbf0 con 0x7f3e48004d00 2024-01-16T15:10:56.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6000f040 con 0x7f3e48004d00 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e67400640 1 -- 172.21.15.141:0/197756690 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60034440 con 0x7f3e48004d00 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e6e600640 1 -- 172.21.15.141:0/197756690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e48004d00 msgr2=0x7f3e48005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e6e600640 1 --2- 172.21.15.141:0/197756690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e48004d00 0x7f3e48005180 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009fc0 tx=0x7f3e480057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e6dc00640 1 -- 172.21.15.141:0/197756690 reap_dead start 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e6e600640 1 -- 172.21.15.141:0/197756690 shutdown_connections 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e6e600640 1 -- 172.21.15.141:0/197756690 >> 172.21.15.141:0/197756690 conn(0x7f3e680d3000 msgr2=0x7f3e680d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.142+0000 7f3e6e600640 1 -- 172.21.15.141:0/197756690 shutdown_connections 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.143+0000 7f3e6e600640 1 -- 172.21.15.141:0/197756690 wait complete. 2024-01-16T15:10:56.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.144+0000 7f3e6e600640 1 Processor -- start 2024-01-16T15:10:56.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.144+0000 7f3e6e600640 1 -- start start 2024-01-16T15:10:56.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.145+0000 7f3e6e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e48004d00 0x7f3e6809cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.145+0000 7f3e6e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e681e6c30 0x7f3e681e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.145+0000 7f3e6e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e681e5b00 0x7f3e681f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.145+0000 7f3e6e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680cd440 con 0x7f3e681e5b00 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.145+0000 7f3e6e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680ccef0 con 0x7f3e681e6c30 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.145+0000 7f3e6e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681e7070 con 0x7f3e48004d00 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e681e5b00 0x7f3e681f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e681e5b00 0x7f3e681f2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53062/0 (socket says 172.21.15.141:53062) 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 -- 172.21.15.141:0/3668048066 learned_addr learned my addr 172.21.15.141:0/3668048066 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e67e00640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e681e6c30 0x7f3e681e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6d200640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e48004d00 0x7f3e6809cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 -- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e48004d00 msgr2=0x7f3e6809cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e48004d00 0x7f3e6809cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 -- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e681e6c30 msgr2=0x7f3e681e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e681e6c30 0x7f3e681e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 -- 172.21.15.141:0/3668048066 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6815da10 con 0x7f3e681e5b00 2024-01-16T15:10:56.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.146+0000 7f3e6dc00640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e681e5b00 0x7f3e681f2230 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000e230 tx=0x7f3e60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:56.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.147+0000 7f3e64c00640 1 -- 172.21.15.141:0/3668048066 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60034bc0 con 0x7f3e681e5b00 2024-01-16T15:10:56.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.147+0000 7f3e64c00640 1 -- 172.21.15.141:0/3668048066 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e60034d40 con 0x7f3e681e5b00 2024-01-16T15:10:56.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.147+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e680cd160 con 0x7f3e681e5b00 2024-01-16T15:10:56.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.147+0000 7f3e64c00640 1 -- 172.21.15.141:0/3668048066 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6003e680 con 0x7f3e681e5b00 2024-01-16T15:10:56.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.147+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e681e6430 con 0x7f3e681e5b00 2024-01-16T15:10:56.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.148+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e681e5f80 con 0x7f3e681e5b00 2024-01-16T15:10:56.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.149+0000 7f3e64c00640 1 -- 172.21.15.141:0/3668048066 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 30) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f3e6003e7e0 con 0x7f3e681e5b00 2024-01-16T15:10:56.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.150+0000 7f3e64c00640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4006c070 0x7f3e4006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:56.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.150+0000 7f3e64c00640 1 -- 172.21.15.141:0/3668048066 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3e600aaad0 con 0x7f3e681e5b00 2024-01-16T15:10:56.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.151+0000 7f3e6d200640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4006c070 0x7f3e4006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:56.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.152+0000 7f3e6d200640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4006c070 0x7f3e4006e530 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009990 tx=0x7f3e58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:56.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.154+0000 7f3e64c00640 1 -- 172.21.15.141:0/3668048066 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e600770e0 con 0x7f3e681e5b00 2024-01-16T15:10:56.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.374+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e680d08e0 con 0x7f3e4006c070 2024-01-16T15:10:56.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.385+0000 7f3e64c00640 1 -- 172.21.15.141:0/3668048066 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f3e680d08e0 con 0x7f3e4006c070 2024-01-16T15:10:56.361 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac 2024-01-16T15:10:56.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.387+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4006c070 msgr2=0x7f3e4006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:56.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.387+0000 7f3e6e600640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4006c070 0x7f3e4006e530 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009990 tx=0x7f3e58008040 comp rx=0 tx=0).stop 2024-01-16T15:10:56.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.387+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e681e5b00 msgr2=0x7f3e681f2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:56.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.387+0000 7f3e6e600640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e681e5b00 0x7f3e681f2230 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000e230 tx=0x7f3e60009750 comp rx=0 tx=0).stop 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 shutdown_connections 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4006c070 0x7f3e4006e530 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e681e5b00 0x7f3e681f2230 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e681e6c30 0x7f3e681e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 --2- 172.21.15.141:0/3668048066 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e48004d00 0x7f3e6809cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 >> 172.21.15.141:0/3668048066 conn(0x7f3e680d3000 msgr2=0x7f3e680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 shutdown_connections 2024-01-16T15:10:56.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:10:56.388+0000 7f3e6e600640 1 -- 172.21.15.141:0/3668048066 wait complete. 2024-01-16T15:10:56.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:56.383 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac) 2024-01-16T15:10:56.440 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac 2024-01-16T15:10:56.441 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:10:56.441 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:56.442 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:10:58.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:10:58.442 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac) 2024-01-16T15:10:58.510 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac 2024-01-16T15:10:58.510 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:10:58.510 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:10:58.512 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:11:00.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:00.512 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac) 2024-01-16T15:11:00.577 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/1dbce001-cd46-4367-bfd8-96e09caeaeac 2024-01-16T15:11:00.578 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:11:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:00.579 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:11:00.579 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T15:11:00.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.820+0000 7f342f000640 1 Processor -- start 2024-01-16T15:11:00.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.820+0000 7f342f000640 1 -- start start 2024-01-16T15:11:00.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.822+0000 7f342f000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 0x7f342806d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.822+0000 7f342f000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3428070e50 con 0x7f3428143c40 2024-01-16T15:11:00.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.822+0000 7f342f000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3428144020 con 0x7f34280cc500 2024-01-16T15:11:00.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.822+0000 7f342f000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34280723c0 con 0x7f34280cc990 2024-01-16T15:11:00.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.822+0000 7f342dc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3428143c40 0x7f3428071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58650/0 (socket says 172.21.15.141:58650) 2024-01-16T15:11:00.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.822+0000 7f342dc00640 1 -- 172.21.15.141:0/3366613993 learned_addr learned my addr 172.21.15.141:0/3366613993 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f342e600640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 0x7f342806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725423521 0 0) 0x7f3428070e50 con 0x7f3428143c40 2024-01-16T15:11:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3408003520 con 0x7f3428143c40 2024-01-16T15:11:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25472239 0 0) 0x7f3408003520 con 0x7f3428143c40 2024-01-16T15:11:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 msgr2=0x7f342806d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 0x7f342806d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 >> v1:172.21.15.188:6789/0 conn(0x7f34280cc500 legacy=0x7f342806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428071e00 con 0x7f3428143c40 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3418001690 con 0x7f3428143c40 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 >> v1:172.21.15.141:6789/0 conn(0x7f3428143c40 legacy=0x7f3428071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 0x7f34080039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3408003f20 0x7f34080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408004d00 0x7f3408005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34280723c0 con 0x7f3408003f20 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340800bb10 con 0x7f3408004d00 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.823+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340800bcd0 con 0x7f34280cc990 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342d200640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3408003f20 0x7f34080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342dc00640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 0x7f34080039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342e600640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408004d00 0x7f3408005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342d200640 1 -- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 msgr2=0x7f34080039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342d200640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 0x7f34080039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342d200640 1 -- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408004d00 msgr2=0x7f3408005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342d200640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408004d00 0x7f3408005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342d200640 1 -- 172.21.15.141:0/3366613993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3408003520 con 0x7f3408003f20 2024-01-16T15:11:00.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.824+0000 7f342dc00640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f34280cc990 0x7f34080039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.825+0000 7f342d200640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3408003f20 0x7f34080047c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f3410001750 tx=0x7f34080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4fca418190af43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.825+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34100052d0 con 0x7f3408003f20 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.825+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341002e050 con 0x7f3408003f20 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.825+0000 7f3427e00640 1 -- 172.21.15.141:0/3366613993 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34100035f0 con 0x7f3408003f20 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.826+0000 7f342f000640 1 -- 172.21.15.141:0/3366613993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3408003f20 msgr2=0x7f34080047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.826+0000 7f342f000640 1 --2- 172.21.15.141:0/3366613993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3408003f20 0x7f34080047c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f3410001750 tx=0x7f34080057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.826+0000 7f342e600640 1 -- 172.21.15.141:0/3366613993 reap_dead start 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.826+0000 7f342f000640 1 -- 172.21.15.141:0/3366613993 shutdown_connections 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.826+0000 7f342f000640 1 -- 172.21.15.141:0/3366613993 >> 172.21.15.141:0/3366613993 conn(0x7f342806a800 msgr2=0x7f342806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:00.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.826+0000 7f342f000640 1 -- 172.21.15.141:0/3366613993 shutdown_connections 2024-01-16T15:11:00.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.826+0000 7f342f000640 1 -- 172.21.15.141:0/3366613993 wait complete. 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.828+0000 7f342f000640 1 Processor -- start 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.828+0000 7f342f000640 1 -- start start 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342f000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408003f20 0x7f342815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342f000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f342813ece0 0x7f342813c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342f000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f342813df70 0x7f342814af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342f000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342813db50 con 0x7f342813ece0 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342f000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342814dec0 con 0x7f3408003f20 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342f000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342813f160 con 0x7f342813df70 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f342813ece0 0x7f342813c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f342813ece0 0x7f342813c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53084/0 (socket says 172.21.15.141:53084) 2024-01-16T15:11:00.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.829+0000 7f342d200640 1 -- 172.21.15.141:0/53731520 learned_addr learned my addr 172.21.15.141:0/53731520 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342e600640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f342813df70 0x7f342814af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342dc00640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408003f20 0x7f342815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342d200640 1 -- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f342813df70 msgr2=0x7f342814af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342d200640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f342813df70 0x7f342814af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342d200640 1 -- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408003f20 msgr2=0x7f342815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342d200640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408003f20 0x7f342815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342d200640 1 -- 172.21.15.141:0/53731520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342813f3e0 con 0x7f342813ece0 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342dc00640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408003f20 0x7f342815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.830+0000 7f342d200640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f342813ece0 0x7f342813c610 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3410008fd0 tx=0x7f3410001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.831+0000 7f3425600640 1 -- 172.21.15.141:0/53731520 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410007a30 con 0x7f342813ece0 2024-01-16T15:11:00.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.831+0000 7f3425600640 1 -- 172.21.15.141:0/53731520 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3410007640 con 0x7f342813ece0 2024-01-16T15:11:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.831+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342813d1b0 con 0x7f342813ece0 2024-01-16T15:11:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.831+0000 7f3425600640 1 -- 172.21.15.141:0/53731520 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410006c00 con 0x7f342813ece0 2024-01-16T15:11:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.831+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342813d770 con 0x7f342813ece0 2024-01-16T15:11:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.832+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3428067450 con 0x7f342813ece0 2024-01-16T15:11:00.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.833+0000 7f3425600640 1 -- 172.21.15.141:0/53731520 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 30) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f34100035f0 con 0x7f342813ece0 2024-01-16T15:11:00.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.834+0000 7f3425600640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f340006c040 0x7f340006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.834+0000 7f3425600640 1 -- 172.21.15.141:0/53731520 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f34100aa580 con 0x7f342813ece0 2024-01-16T15:11:00.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.834+0000 7f342dc00640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f340006c040 0x7f340006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.836+0000 7f342dc00640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f340006c040 0x7f340006e500 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f34280cbd00 tx=0x7f3418009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:00.838+0000 7f3425600640 1 -- 172.21.15.141:0/53731520 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3410076bf0 con 0x7f342813ece0 2024-01-16T15:11:01.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.064+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f342806cc40 con 0x7f340006c040 2024-01-16T15:11:01.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.222+0000 7f3425600640 1 -- 172.21.15.141:0/53731520 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f341800d040 con 0x7f340006c040 2024-01-16T15:11:01.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f340006c040 msgr2=0x7f340006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f340006c040 0x7f340006e500 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f34280cbd00 tx=0x7f3418009040 comp rx=0 tx=0).stop 2024-01-16T15:11:01.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f342813ece0 msgr2=0x7f342813c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f342813ece0 0x7f342813c610 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3410008fd0 tx=0x7f3410001120 comp rx=0 tx=0).stop 2024-01-16T15:11:01.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 shutdown_connections 2024-01-16T15:11:01.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f340006c040 0x7f340006e500 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f342813df70 0x7f342814af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f342813ece0 0x7f342813c610 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 --2- 172.21.15.141:0/53731520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3408003f20 0x7f342815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 >> 172.21.15.141:0/53731520 conn(0x7f342806a800 msgr2=0x7f34280b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:01.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 shutdown_connections 2024-01-16T15:11:01.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.226+0000 7f342f000640 1 -- 172.21.15.141:0/53731520 wait complete. 2024-01-16T15:11:01.208 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.232+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.208 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.232+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.208 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.232+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.209 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.232+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.209 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.232+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.222 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:11:01.222 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:11:01.234 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.258+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.234 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.258+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.234 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.258+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.234 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.258+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.234 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:01.258+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.444+0000 7f52d5e00640 1 Processor -- start 2024-01-16T15:11:01.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.444+0000 7f52d5e00640 1 -- start start 2024-01-16T15:11:01.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.445+0000 7f52d5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52d00cf9a0 0x7f52d015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.445+0000 7f52d5e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.445+0000 7f52d5e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00b1ee0 con 0x7f52d0143ae0 2024-01-16T15:11:01.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.445+0000 7f52d5e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00b2780 con 0x7f52d00cf9a0 2024-01-16T15:11:01.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52d5e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d015e990 con 0x7f52d015e1e0 2024-01-16T15:11:01.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52ce000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52d00cf9a0 0x7f52d015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52ce000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f52d00cf9a0 0x7f52d015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53088/0 (socket says 172.21.15.141:53088) 2024-01-16T15:11:01.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52ce000640 1 -- 172.21.15.141:0/3116514388 learned_addr learned my addr 172.21.15.141:0/3116514388 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:01.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52ce000640 1 -- 172.21.15.141:0/3116514388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 msgr2=0x7f52d00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:01.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52cf400640 1 --2- 172.21.15.141:0/3116514388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52ce000640 1 --2- 172.21.15.141:0/3116514388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52ce000640 1 -- 172.21.15.141:0/3116514388 >> v1:172.21.15.188:6789/0 conn(0x7f52d0143ae0 legacy=0x7f52d00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:01.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52ce000640 1 -- 172.21.15.141:0/3116514388 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00b2130 con 0x7f52d00cf9a0 2024-01-16T15:11:01.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.446+0000 7f52cf400640 1 --2- 172.21.15.141:0/3116514388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:01.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.447+0000 7f52ce000640 1 --2- 172.21.15.141:0/3116514388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d015dca0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009e50 tx=0x7f52c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925c1a2470029db9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.448+0000 7f52cd600640 1 -- 172.21.15.141:0/3116514388 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c002ca80 con 0x7f52d00cf9a0 2024-01-16T15:11:01.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.448+0000 7f52cd600640 1 -- 172.21.15.141:0/3116514388 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c002cbe0 con 0x7f52d00cf9a0 2024-01-16T15:11:01.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.448+0000 7f52cd600640 1 -- 172.21.15.141:0/3116514388 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00343f0 con 0x7f52d00cf9a0 2024-01-16T15:11:01.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.448+0000 7f52d5e00640 1 -- 172.21.15.141:0/3116514388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 msgr2=0x7f52d015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.448+0000 7f52d5e00640 1 --2- 172.21.15.141:0/3116514388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d015dca0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009e50 tx=0x7f52c002bee0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.449+0000 7f52d5e00640 1 -- 172.21.15.141:0/3116514388 shutdown_connections 2024-01-16T15:11:01.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.449+0000 7f52d5e00640 1 --2- 172.21.15.141:0/3116514388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d015dca0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.449+0000 7f52d5e00640 1 --2- 172.21.15.141:0/3116514388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.449+0000 7f52d5e00640 1 -- 172.21.15.141:0/3116514388 >> 172.21.15.141:0/3116514388 conn(0x7f52d006a800 msgr2=0x7f52d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:01.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.449+0000 7f52d5e00640 1 -- 172.21.15.141:0/3116514388 shutdown_connections 2024-01-16T15:11:01.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.449+0000 7f52d5e00640 1 -- 172.21.15.141:0/3116514388 wait complete. 2024-01-16T15:11:01.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.450+0000 7f52d5e00640 1 Processor -- start 2024-01-16T15:11:01.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.450+0000 7f52d5e00640 1 -- start start 2024-01-16T15:11:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.451+0000 7f52d5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d0159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.451+0000 7f52d5e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d0143ae0 0x7f52d00cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.451+0000 7f52d5e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d013c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.451+0000 7f52d5e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ccb50 con 0x7f52d00cf9a0 2024-01-16T15:11:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.451+0000 7f52d5e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ccfc0 con 0x7f52d0143ae0 2024-01-16T15:11:01.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.451+0000 7f52d5e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00cd2a0 con 0x7f52d015e1e0 2024-01-16T15:11:01.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d0159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d0159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53096/0 (socket says 172.21.15.141:53096) 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 -- 172.21.15.141:0/2631428158 learned_addr learned my addr 172.21.15.141:0/2631428158 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52ce000640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d0143ae0 0x7f52d00cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cf400640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d013c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 -- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 msgr2=0x7f52d013c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d013c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 -- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d0143ae0 msgr2=0x7f52d00cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d0143ae0 0x7f52d00cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.452+0000 7f52cea00640 1 -- 172.21.15.141:0/2631428158 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c0009b30 con 0x7f52d00cf9a0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.453+0000 7f52cea00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d0159a90 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f52b800c430 tx=0x7f52b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.453+0000 7f52c7400640 1 -- 172.21.15.141:0/2631428158 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8011790 con 0x7f52d00cf9a0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.453+0000 7f52c7400640 1 -- 172.21.15.141:0/2631428158 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b80118f0 con 0x7f52d00cf9a0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.453+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d0141bd0 con 0x7f52d00cf9a0 2024-01-16T15:11:01.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.453+0000 7f52c7400640 1 -- 172.21.15.141:0/2631428158 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8011ce0 con 0x7f52d00cf9a0 2024-01-16T15:11:01.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.453+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d00cd580 con 0x7f52d00cf9a0 2024-01-16T15:11:01.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.454+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d00719e0 con 0x7f52d00cf9a0 2024-01-16T15:11:01.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.455+0000 7f52c7400640 1 -- 172.21.15.141:0/2631428158 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 30) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f52b801c460 con 0x7f52d00cf9a0 2024-01-16T15:11:01.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.456+0000 7f52c7400640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806c040 0x7f52a806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.456+0000 7f52c7400640 1 -- 172.21.15.141:0/2631428158 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f52b8088850 con 0x7f52d00cf9a0 2024-01-16T15:11:01.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.456+0000 7f52ce000640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806c040 0x7f52a806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.458+0000 7f52ce000640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806c040 0x7f52a806e500 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f52c0035000 tx=0x7f52c0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.436 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.460+0000 7f52c7400640 1 -- 172.21.15.141:0/2631428158 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52b8054ec0 con 0x7f52d00cf9a0 2024-01-16T15:11:01.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.687+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f52d015e670 con 0x7f52a806c040 2024-01-16T15:11:01.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.703+0000 7f52c7400640 1 -- 172.21.15.141:0/2631428158 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f52c00440c0 con 0x7f52a806c040 2024-01-16T15:11:01.680 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:11:01.680 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:11:01.680 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:11:01.680 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:11:01.681 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:11:01.681 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:11:01.681 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:11:01.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:01.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:01.681 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:11:01.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806c040 msgr2=0x7f52a806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806c040 0x7f52a806e500 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f52c0035000 tx=0x7f52c0030000 comp rx=0 tx=0).stop 2024-01-16T15:11:01.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 msgr2=0x7f52d0159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d0159a90 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f52b800c430 tx=0x7f52b800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 shutdown_connections 2024-01-16T15:11:01.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f52a806c040 0x7f52a806e500 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f52d015e1e0 0x7f52d013c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52d0143ae0 0x7f52d00cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 --2- 172.21.15.141:0/2631428158 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52d00cf9a0 0x7f52d0159a90 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.706+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 >> 172.21.15.141:0/2631428158 conn(0x7f52d006a800 msgr2=0x7f52d006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:01.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.707+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 shutdown_connections 2024-01-16T15:11:01.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.707+0000 7f52d5e00640 1 -- 172.21.15.141:0/2631428158 wait complete. 2024-01-16T15:11:01.701 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-01-16T15:11:01.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.937+0000 7f0d6d000640 1 Processor -- start 2024-01-16T15:11:01.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.937+0000 7f0d6d000640 1 -- start start 2024-01-16T15:11:01.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d6d000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0d6813df80 0x7f0d6815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d6d000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0d6813e360 0x7f0d680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d6d000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6806c470 con 0x7f0d6815f9c0 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d6d000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6806c750 con 0x7f0d6813df80 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d6d000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6806ca30 con 0x7f0d6813e360 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d65e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0d6813df80 0x7f0d6815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d65e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0d6813df80 0x7f0d6815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53108/0 (socket says 172.21.15.141:53108) 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d65e00640 1 -- 172.21.15.141:0/1060057771 learned_addr learned my addr 172.21.15.141:0/1060057771 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d65e00640 1 -- 172.21.15.141:0/1060057771 >> v2:172.21.15.188:3300/0 conn(0x7f0d6813e360 msgr2=0x7f0d680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d65e00640 1 --2- 172.21.15.141:0/1060057771 >> v2:172.21.15.188:3300/0 conn(0x7f0d6813e360 0x7f0d680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d65e00640 1 -- 172.21.15.141:0/1060057771 >> v1:172.21.15.188:6789/0 conn(0x7f0d6815f9c0 legacy=0x7f0d681408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:01.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.939+0000 7f0d65e00640 1 -- 172.21.15.141:0/1060057771 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6806cd10 con 0x7f0d6813df80 2024-01-16T15:11:01.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.941+0000 7f0d65e00640 1 --2- 172.21.15.141:0/1060057771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813df80 0x7f0d6815bee0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0d6806ded0 tx=0x7f0d5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7276c8945efbb615 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.941+0000 7f0d65400640 1 -- 172.21.15.141:0/1060057771 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58031870 con 0x7f0d6813df80 2024-01-16T15:11:01.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.941+0000 7f0d65400640 1 -- 172.21.15.141:0/1060057771 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d580319d0 con 0x7f0d6813df80 2024-01-16T15:11:01.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.941+0000 7f0d65400640 1 -- 172.21.15.141:0/1060057771 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58031d60 con 0x7f0d6813df80 2024-01-16T15:11:01.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 -- 172.21.15.141:0/1060057771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813df80 msgr2=0x7f0d6815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 --2- 172.21.15.141:0/1060057771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813df80 0x7f0d6815bee0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0d6806ded0 tx=0x7f0d5802b910 comp rx=0 tx=0).stop 2024-01-16T15:11:01.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 -- 172.21.15.141:0/1060057771 shutdown_connections 2024-01-16T15:11:01.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 --2- 172.21.15.141:0/1060057771 >> v2:172.21.15.188:3300/0 conn(0x7f0d6813e360 0x7f0d680cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 --2- 172.21.15.141:0/1060057771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813df80 0x7f0d6815bee0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 -- 172.21.15.141:0/1060057771 >> 172.21.15.141:0/1060057771 conn(0x7f0d6806a800 msgr2=0x7f0d6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:01.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 -- 172.21.15.141:0/1060057771 shutdown_connections 2024-01-16T15:11:01.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.942+0000 7f0d6d000640 1 -- 172.21.15.141:0/1060057771 wait complete. 2024-01-16T15:11:01.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.943+0000 7f0d6d000640 1 Processor -- start 2024-01-16T15:11:01.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.944+0000 7f0d6d000640 1 -- start start 2024-01-16T15:11:01.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.944+0000 7f0d6d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d6813df80 0x7f0d681f0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.944+0000 7f0d6d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813e360 0x7f0d6814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d6d000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d6815f9c0 0x7f0d6814f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d6d000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6814f040 con 0x7f0d6813e360 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d6d000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6814e430 con 0x7f0d6813df80 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d6d000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6814e710 con 0x7f0d6815f9c0 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813e360 0x7f0d6814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813e360 0x7f0d6814f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:53114/0 (socket says 172.21.15.141:53114) 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 -- 172.21.15.141:0/2245732173 learned_addr learned my addr 172.21.15.141:0/2245732173 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 -- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d6815f9c0 msgr2=0x7f0d6814f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d6815f9c0 0x7f0d6814f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 -- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d6813df80 msgr2=0x7f0d681f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d6813df80 0x7f0d681f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:01.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.945+0000 7f0d5fe00640 1 -- 172.21.15.141:0/2245732173 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d580094d0 con 0x7f0d6813e360 2024-01-16T15:11:01.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.946+0000 7f0d5fe00640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813e360 0x7f0d6814f1a0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000b450 tx=0x7f0d5000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.946+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50011890 con 0x7f0d6813e360 2024-01-16T15:11:01.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.946+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d500119f0 con 0x7f0d6813e360 2024-01-16T15:11:01.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.946+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50011da0 con 0x7f0d6813e360 2024-01-16T15:11:01.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.946+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6814e9f0 con 0x7f0d6813e360 2024-01-16T15:11:01.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.946+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d681f0ec0 con 0x7f0d6813e360 2024-01-16T15:11:01.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.947+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d680675d0 con 0x7f0d6813e360 2024-01-16T15:11:01.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.948+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 30) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f0d5001a400 con 0x7f0d6813e360 2024-01-16T15:11:01.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.949+0000 7f0d5e000640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0d4006c070 0x7f0d4006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:01.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.949+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0d50088cc0 con 0x7f0d6813e360 2024-01-16T15:11:01.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.950+0000 7f0d65e00640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0d4006c070 0x7f0d4006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:01.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.951+0000 7f0d65e00640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0d4006c070 0x7f0d4006e530 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0d58036000 tx=0x7f0d58030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:01.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:01.953+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d500552d0 con 0x7f0d6813e360 2024-01-16T15:11:02.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:02.186+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d6814e880 con 0x7f0d4006c070 2024-01-16T15:11:03.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:03.948+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f0d50088860 con 0x7f0d6813e360 2024-01-16T15:11:06.184 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:06.192 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:15.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.652+0000 7f0d5e000640 1 -- 172.21.15.141:0/2245732173 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0d6814e880 con 0x7f0d4006c070 2024-01-16T15:11:15.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0d4006c070 msgr2=0x7f0d4006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0d4006c070 0x7f0d4006e530 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0d58036000 tx=0x7f0d58030000 comp rx=0 tx=0).stop 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813e360 msgr2=0x7f0d6814f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813e360 0x7f0d6814f1a0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000b450 tx=0x7f0d5000b910 comp rx=0 tx=0).stop 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 shutdown_connections 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0d4006c070 0x7f0d4006e530 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0d6815f9c0 0x7f0d6814f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0d6813e360 0x7f0d6814f1a0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 --2- 172.21.15.141:0/2245732173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d6813df80 0x7f0d681f0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.655+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 >> 172.21.15.141:0/2245732173 conn(0x7f0d6806a800 msgr2=0x7f0d6815df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.656+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 shutdown_connections 2024-01-16T15:11:15.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:15.656+0000 7f0d6d000640 1 -- 172.21.15.141:0/2245732173 wait complete. 2024-01-16T15:11:17.655 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:11:17.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.908+0000 7fa6d7200640 1 Processor -- start 2024-01-16T15:11:17.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.908+0000 7fa6d7200640 1 -- start start 2024-01-16T15:11:17.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.910+0000 7fa6d7200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa6d80adf20 0x7fa6d8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.910+0000 7fa6d7200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ccb50 0x7fa6d8143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.910+0000 7fa6d7200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d815f4e0 con 0x7fa6d815bf10 2024-01-16T15:11:17.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.910+0000 7fa6d7200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d815f970 con 0x7fa6d80adf20 2024-01-16T15:11:17.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.910+0000 7fa6d7200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8071130 con 0x7fa6d80ccb50 2024-01-16T15:11:17.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.910+0000 7fa6cfe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa6d815bf10 0x7fa6d80cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38924/0 (socket says 172.21.15.141:38924) 2024-01-16T15:11:17.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.910+0000 7fa6cfe00640 1 -- 172.21.15.141:0/2622626641 learned_addr learned my addr 172.21.15.141:0/2622626641 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.911+0000 7fa6d6800640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ccb50 0x7fa6d8143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.911+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/2622626641 >> v2:172.21.15.188:3300/0 conn(0x7fa6d80adf20 0x7fa6d8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.911+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707873180 0 0) 0x7fa6d815f4e0 con 0x7fa6d815bf10 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.911+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b8003520 con 0x7fa6d815bf10 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.911+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3883394343 0 0) 0x7fa6b8003520 con 0x7fa6d815bf10 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ccb50 msgr2=0x7fa6d8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ccb50 0x7fa6d8143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 >> v2:172.21.15.188:3300/0 conn(0x7fa6d80adf20 msgr2=0x7fa6d8070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 --2- 172.21.15.141:0/2622626641 >> v2:172.21.15.188:3300/0 conn(0x7fa6d80adf20 0x7fa6d8070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d815c390 con 0x7fa6d815bf10 2024-01-16T15:11:17.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa6c0001670 con 0x7fa6d815bf10 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 >> v1:172.21.15.141:6789/0 conn(0x7fa6d815bf10 legacy=0x7fa6d80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ccb50 0x7fa6b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8004d00 0x7fa6b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d815f970 con 0x7fa6b8003970 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8071130 con 0x7fa6b8004d00 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b800ba30 con 0x7fa6d80ccb50 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5e00640 1 -- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ccb50 msgr2=0x7fa6b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ccb50 0x7fa6b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5e00640 1 -- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8004d00 msgr2=0x7fa6b8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:17.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8004d00 0x7fa6b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.912+0000 7fa6d5e00640 1 -- 172.21.15.141:0/2622626641 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8003520 con 0x7fa6b8003970 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.913+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6b8003d90 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800ba30 tx=0x7fa6b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a6ed9699d5cbd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8032680 con 0x7fa6b8003970 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c80327e0 con 0x7fa6b8003970 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d5400640 1 -- 172.21.15.141:0/2622626641 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8032b70 con 0x7fa6b8003970 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d7200640 1 -- 172.21.15.141:0/2622626641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 msgr2=0x7fa6b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d7200640 1 --2- 172.21.15.141:0/2622626641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6b8003d90 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800ba30 tx=0x7fa6b80057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d6800640 1 -- 172.21.15.141:0/2622626641 reap_dead start 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d7200640 1 -- 172.21.15.141:0/2622626641 shutdown_connections 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d7200640 1 -- 172.21.15.141:0/2622626641 >> 172.21.15.141:0/2622626641 conn(0x7fa6d806a7d0 msgr2=0x7fa6d806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.914+0000 7fa6d7200640 1 -- 172.21.15.141:0/2622626641 shutdown_connections 2024-01-16T15:11:17.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.915+0000 7fa6d7200640 1 -- 172.21.15.141:0/2622626641 wait complete. 2024-01-16T15:11:17.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.916+0000 7fa6d7200640 1 Processor -- start 2024-01-16T15:11:17.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.916+0000 7fa6d7200640 1 -- start start 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.917+0000 7fa6d7200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6d8140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.917+0000 7fa6d7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6d813f000 0x7fa6d815a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.917+0000 7fa6d7200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ac8d0 0x7fa6d80acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.917+0000 7fa6d7200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80ad2a0 con 0x7fa6b8003970 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.917+0000 7fa6d7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d813f490 con 0x7fa6d813f000 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.917+0000 7fa6d7200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d813f770 con 0x7fa6d80ac8d0 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6d8140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6d8140be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49820/0 (socket says 172.21.15.141:49820) 2024-01-16T15:11:17.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 -- 172.21.15.141:0/1103869990 learned_addr learned my addr 172.21.15.141:0/1103869990 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 -- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ac8d0 msgr2=0x7fa6d80acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ac8d0 0x7fa6d80acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 -- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6d813f000 msgr2=0x7fa6d815a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6d813f000 0x7fa6d815a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 -- 172.21.15.141:0/1103869990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d813fa50 con 0x7fa6b8003970 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.918+0000 7fa6d5e00640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6d8140be0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8035040 tx=0x7fa6c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.919+0000 7fa6ce000640 1 -- 172.21.15.141:0/1103869990 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8004220 con 0x7fa6b8003970 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.919+0000 7fa6ce000640 1 -- 172.21.15.141:0/1103869990 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8004380 con 0x7fa6b8003970 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.919+0000 7fa6ce000640 1 -- 172.21.15.141:0/1103869990 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8003850 con 0x7fa6b8003970 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.919+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d813fd30 con 0x7fa6b8003970 2024-01-16T15:11:17.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.919+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d8141120 con 0x7fa6b8003970 2024-01-16T15:11:17.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.920+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d8067450 con 0x7fa6b8003970 2024-01-16T15:11:17.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.921+0000 7fa6ce000640 1 -- 172.21.15.141:0/1103869990 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fa6c8043070 con 0x7fa6b8003970 2024-01-16T15:11:17.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.922+0000 7fa6ce000640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa6b006c160 0x7fa6b006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:17.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.922+0000 7fa6ce000640 1 -- 172.21.15.141:0/1103869990 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa6c80aab90 con 0x7fa6b8003970 2024-01-16T15:11:17.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.923+0000 7fa6cfe00640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa6b006c160 0x7fa6b006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:17.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.924+0000 7fa6cfe00640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa6b006c160 0x7fa6b006e620 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa6d81405f0 tx=0x7fa6c0001430 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:17.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:17.926+0000 7fa6ce000640 1 -- 172.21.15.141:0/1103869990 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6c8077570 con 0x7fa6b8003970 2024-01-16T15:11:18.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:18.155+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6d815f170 con 0x7fa6b006c160 2024-01-16T15:11:20.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.509+0000 7fa6ce000640 1 -- 172.21.15.141:0/1103869990 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fa6c0007e80 con 0x7fa6b006c160 2024-01-16T15:11:20.485 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:20.486 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:11:20.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa6b006c160 msgr2=0x7fa6b006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa6b006c160 0x7fa6b006e620 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa6d81405f0 tx=0x7fa6c0001430 comp rx=0 tx=0).stop 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 msgr2=0x7fa6d8140be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6d8140be0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8035040 tx=0x7fa6c8004730 comp rx=0 tx=0).stop 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 shutdown_connections 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa6b006c160 0x7fa6b006e620 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6d80ac8d0 0x7fa6d80acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6d813f000 0x7fa6d815a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 --2- 172.21.15.141:0/1103869990 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b8003970 0x7fa6d8140be0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 >> 172.21.15.141:0/1103869990 conn(0x7fa6d806a7d0 msgr2=0x7fa6d806c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 shutdown_connections 2024-01-16T15:11:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.512+0000 7fa6d7200640 1 -- 172.21.15.141:0/1103869990 wait complete. 2024-01-16T15:11:20.510 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:11:20.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.741+0000 7f4952400640 1 Processor -- start 2024-01-16T15:11:20.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.741+0000 7f4952400640 1 -- start start 2024-01-16T15:11:20.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.743+0000 7f4952400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f494c1607a0 0x7f494c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.743+0000 7f4952400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 0x7f494c1617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.743+0000 7f4952400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c0b2b90 con 0x7f494c0b3eb0 2024-01-16T15:11:20.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.743+0000 7f4952400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c0d21d0 con 0x7f494c1607a0 2024-01-16T15:11:20.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.743+0000 7f4952400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0b2e30 con 0x7f494c160b80 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.743+0000 7f494be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f494c0b3eb0 0x7f494c1610c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38930/0 (socket says 172.21.15.141:38930) 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.743+0000 7f494be00640 1 -- 172.21.15.141:0/2598366157 learned_addr learned my addr 172.21.15.141:0/2598366157 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f4951000640 1 --2- 172.21.15.141:0/2598366157 >> v2:172.21.15.188:3300/0 conn(0x7f494c1607a0 0x7f494c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f4951a00640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 0x7f494c1617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668558051 0 0) 0x7f494c0b2b90 con 0x7f494c0b3eb0 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492c003520 con 0x7f494c0b3eb0 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045864737 0 0) 0x7f492c003520 con 0x7f494c0b3eb0 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 msgr2=0x7f494c1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 0x7f494c1617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 >> v2:172.21.15.188:3300/0 conn(0x7f494c1607a0 msgr2=0x7f494c0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 --2- 172.21.15.141:0/2598366157 >> v2:172.21.15.188:3300/0 conn(0x7f494c1607a0 0x7f494c0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c0b3110 con 0x7f494c0b3eb0 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f493c0019f0 con 0x7f494c0b3eb0 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 >> v1:172.21.15.141:6789/0 conn(0x7f494c0b3eb0 legacy=0x7f494c1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f492c003970 0x7f492c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.744+0000 7f494b400640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f492c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494b400640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 0x7f492c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0d21d0 con 0x7f492c0047c0 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0b2e30 con 0x7f492c003970 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c00b980 con 0x7f494c160b80 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494be00640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f492c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f4951000640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f492c003970 0x7f492c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494be00640 1 -- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 msgr2=0x7f492c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f4951a00640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 0x7f492c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494be00640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c160b80 0x7f492c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494be00640 1 -- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f492c003970 msgr2=0x7f492c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:20.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494be00640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f492c003970 0x7f492c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.745+0000 7f494be00640 1 -- 172.21.15.141:0/2598366157 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c003520 con 0x7f492c0047c0 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.746+0000 7f494be00640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f492c004c40 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f494c0b42e0 tx=0x7f492c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3405602049bf238e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c034630 con 0x7f492c0047c0 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c02f070 con 0x7f492c0047c0 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f494b400640 1 -- 172.21.15.141:0/2598366157 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c0349d0 con 0x7f492c0047c0 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f4952400640 1 -- 172.21.15.141:0/2598366157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 msgr2=0x7f492c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f4952400640 1 --2- 172.21.15.141:0/2598366157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f492c004c40 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f494c0b42e0 tx=0x7f492c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f4951a00640 1 -- 172.21.15.141:0/2598366157 reap_dead start 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f4952400640 1 -- 172.21.15.141:0/2598366157 shutdown_connections 2024-01-16T15:11:20.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f4952400640 1 -- 172.21.15.141:0/2598366157 >> 172.21.15.141:0/2598366157 conn(0x7f494c09c590 msgr2=0x7f494c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:20.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f4952400640 1 -- 172.21.15.141:0/2598366157 shutdown_connections 2024-01-16T15:11:20.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.747+0000 7f4952400640 1 -- 172.21.15.141:0/2598366157 wait complete. 2024-01-16T15:11:20.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.749+0000 7f4952400640 1 Processor -- start 2024-01-16T15:11:20.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.749+0000 7f4952400640 1 -- start start 2024-01-16T15:11:20.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.750+0000 7f4952400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f494c1f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.750+0000 7f4952400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c1e8110 0x7f494c0c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.750+0000 7f4952400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f494c0c8480 0x7f494c1e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.750+0000 7f4952400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0c99b0 con 0x7f492c0047c0 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.750+0000 7f4952400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0c9520 con 0x7f494c0c8480 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.750+0000 7f4952400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0c8910 con 0x7f494c1e8110 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f494c1f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f494c1f2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49840/0 (socket says 172.21.15.141:49840) 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 -- 172.21.15.141:0/4142488984 learned_addr learned my addr 172.21.15.141:0/4142488984 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951a00640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f494c0c8480 0x7f494c1e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 -- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c1e8110 msgr2=0x7f494c0c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f494be00640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c1e8110 0x7f494c0c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c1e8110 0x7f494c0c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 -- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f494c0c8480 msgr2=0x7f494c1e8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f494c0c8480 0x7f494c1e8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 -- 172.21.15.141:0/4142488984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c0c8bf0 con 0x7f492c0047c0 2024-01-16T15:11:20.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951a00640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f494c0c8480 0x7f494c1e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:20.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.751+0000 7f4951000640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f494c1f2cf0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f4934011630 tx=0x7f4934011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.752+0000 7f4948c00640 1 -- 172.21.15.141:0/4142488984 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400f920 con 0x7f492c0047c0 2024-01-16T15:11:20.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.752+0000 7f4948c00640 1 -- 172.21.15.141:0/4142488984 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493400fa80 con 0x7f492c0047c0 2024-01-16T15:11:20.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.752+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c0c8ed0 con 0x7f492c0047c0 2024-01-16T15:11:20.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.752+0000 7f4948c00640 1 -- 172.21.15.141:0/4142488984 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400fdf0 con 0x7f492c0047c0 2024-01-16T15:11:20.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.752+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c1f3230 con 0x7f492c0047c0 2024-01-16T15:11:20.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.753+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c067450 con 0x7f492c0047c0 2024-01-16T15:11:20.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.754+0000 7f4948c00640 1 -- 172.21.15.141:0/4142488984 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4934015460 con 0x7f492c0047c0 2024-01-16T15:11:20.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.755+0000 7f4948c00640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492406c160 0x7f492406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.755+0000 7f4948c00640 1 -- 172.21.15.141:0/4142488984 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4934088f20 con 0x7f492c0047c0 2024-01-16T15:11:20.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.756+0000 7f494be00640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492406c160 0x7f492406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.757+0000 7f494be00640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492406c160 0x7f492406e620 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f493c034000 tx=0x7f493c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.735 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:20.759+0000 7f4948c00640 1 -- 172.21.15.141:0/4142488984 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4934055840 con 0x7f492c0047c0 2024-01-16T15:11:21.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.024+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f494c08d340 con 0x7f492c0047c0 2024-01-16T15:11:21.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.025+0000 7f4948c00640 1 -- 172.21.15.141:0/4142488984 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1852 (secure 0 0 0) 0x7f4934058e60 con 0x7f492c0047c0 2024-01-16T15:11:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:21.001 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:11:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 873, 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 91, 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:11:21.003 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:11:21.004 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 797, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222156106, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 6993993728, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766100119552, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 426, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 7591, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 2 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 55, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 6, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:11:21.005 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 6, 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "c", 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 8148 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:11:21.006 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 15, 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:11:06.812634+0000", 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:21.007 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": {} 2024-01-16T15:11:21.008 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:11:21.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492406c160 msgr2=0x7f492406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492406c160 0x7f492406e620 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f493c034000 tx=0x7f493c009030 comp rx=0 tx=0).stop 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 msgr2=0x7f494c1f2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f494c1f2cf0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f4934011630 tx=0x7f4934011af0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 shutdown_connections 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f492406c160 0x7f492406e620 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f494c0c8480 0x7f494c1e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f494c1e8110 0x7f494c0c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 --2- 172.21.15.141:0/4142488984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f492c0047c0 0x7f494c1f2cf0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 >> 172.21.15.141:0/4142488984 conn(0x7f494c09c590 msgr2=0x7f494c06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 shutdown_connections 2024-01-16T15:11:21.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.028+0000 7f4952400640 1 -- 172.21.15.141:0/4142488984 wait complete. 2024-01-16T15:11:21.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:21.024 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/ss1clone1/1dbce001-cd46-4367-bfd8-96e09caeaeac) 2024-01-16T15:11:21.078 INFO:teuthology.orchestra.run.smithi141.stderr:ls: cannot access 'volumes/_nogroup/ss1clone1/1dbce001-cd46-4367-bfd8-96e09caeaeac': No such file or directory 2024-01-16T15:11:21.079 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:11:21.080 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:11:21.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.303+0000 7fe788600640 1 Processor -- start 2024-01-16T15:11:21.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.303+0000 7fe788600640 1 -- start start 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.304+0000 7fe788600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe78015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe788600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe7800cf9a0 0x7fe7800cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe788600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78006c470 con 0x7fe78013e360 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe788600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe78006c750 con 0x7fe7800cf9a0 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe788600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78006ca30 con 0x7fe78013df80 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe786600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe7800cf9a0 0x7fe7800cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe786600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe7800cf9a0 0x7fe7800cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49848/0 (socket says 172.21.15.141:49848) 2024-01-16T15:11:21.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe786600640 1 -- 172.21.15.141:0/2252549144 learned_addr learned my addr 172.21.15.141:0/2252549144 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe785c00640 1 --2- 172.21.15.141:0/2252549144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe78015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe786600640 1 -- 172.21.15.141:0/2252549144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 msgr2=0x7fe78015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe786600640 1 --2- 172.21.15.141:0/2252549144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe78015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe786600640 1 -- 172.21.15.141:0/2252549144 >> v1:172.21.15.188:6789/0 conn(0x7fe78013e360 legacy=0x7fe7800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.305+0000 7fe786600640 1 -- 172.21.15.141:0/2252549144 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78006cd10 con 0x7fe7800cf9a0 2024-01-16T15:11:21.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.306+0000 7fe785c00640 1 --2- 172.21.15.141:0/2252549144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe78015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.307+0000 7fe786600640 1 --2- 172.21.15.141:0/2252549144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800cd840 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fe778009e50 tx=0x7fe77802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4d9314c0b25ce2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.307+0000 7fe77fe00640 1 -- 172.21.15.141:0/2252549144 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77802ca80 con 0x7fe7800cf9a0 2024-01-16T15:11:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.307+0000 7fe77fe00640 1 -- 172.21.15.141:0/2252549144 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77802cbe0 con 0x7fe7800cf9a0 2024-01-16T15:11:21.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.307+0000 7fe77fe00640 1 -- 172.21.15.141:0/2252549144 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7780343f0 con 0x7fe7800cf9a0 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 -- 172.21.15.141:0/2252549144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 msgr2=0x7fe7800cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 --2- 172.21.15.141:0/2252549144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800cd840 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fe778009e50 tx=0x7fe77802bee0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 -- 172.21.15.141:0/2252549144 shutdown_connections 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 --2- 172.21.15.141:0/2252549144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800cd840 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 --2- 172.21.15.141:0/2252549144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe78015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 -- 172.21.15.141:0/2252549144 >> 172.21.15.141:0/2252549144 conn(0x7fe78006a800 msgr2=0x7fe78006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 -- 172.21.15.141:0/2252549144 shutdown_connections 2024-01-16T15:11:21.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.308+0000 7fe788600640 1 -- 172.21.15.141:0/2252549144 wait complete. 2024-01-16T15:11:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.310+0000 7fe788600640 1 Processor -- start 2024-01-16T15:11:21.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.310+0000 7fe788600640 1 -- start start 2024-01-16T15:11:21.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe788600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800ad740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe788600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe7800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe788600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78013e360 0x7fe780071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe788600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe780070ba0 con 0x7fe7800cf9a0 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe788600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe780070e80 con 0x7fe78013e360 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe788600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe780071160 con 0x7fe78013df80 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe785c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe785c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800ad740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49856/0 (socket says 172.21.15.141:49856) 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe785c00640 1 -- 172.21.15.141:0/2579247379 learned_addr learned my addr 172.21.15.141:0/2579247379 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe786600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78013e360 0x7fe780071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.311+0000 7fe785200640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe7800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe785c00640 1 -- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 msgr2=0x7fe7800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe785c00640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe7800702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe785c00640 1 -- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78013e360 msgr2=0x7fe780071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe785c00640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78013e360 0x7fe780071c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe785c00640 1 -- 172.21.15.141:0/2579247379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe778009b30 con 0x7fe7800cf9a0 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe785200640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe7800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe785c00640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800ad740 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe77000c7f0 tx=0x7fe77000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe77d600640 1 -- 172.21.15.141:0/2579247379 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770007a30 con 0x7fe7800cf9a0 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe77d600640 1 -- 172.21.15.141:0/2579247379 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe770007b90 con 0x7fe7800cf9a0 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe780071440 con 0x7fe7800cf9a0 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe77d600640 1 -- 172.21.15.141:0/2579247379 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77000ce40 con 0x7fe7800cf9a0 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.312+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7800adc80 con 0x7fe7800cf9a0 2024-01-16T15:11:21.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.313+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe780071960 con 0x7fe7800cf9a0 2024-01-16T15:11:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.314+0000 7fe77d600640 1 -- 172.21.15.141:0/2579247379 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fe7700113f0 con 0x7fe7800cf9a0 2024-01-16T15:11:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.315+0000 7fe77d600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe75806c190 0x7fe75806e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.315+0000 7fe77d600640 1 -- 172.21.15.141:0/2579247379 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe770088a70 con 0x7fe7800cf9a0 2024-01-16T15:11:21.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.316+0000 7fe785200640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe75806c190 0x7fe75806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.317+0000 7fe785200640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe75806c190 0x7fe75806e650 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe780133130 tx=0x7fe768009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.320+0000 7fe77d600640 1 -- 172.21.15.141:0/2579247379 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe780071960 con 0x7fe7800cf9a0 2024-01-16T15:11:21.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.534+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe78015e2c0 con 0x7fe75806c190 2024-01-16T15:11:21.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.538+0000 7fe77d600640 1 -- 172.21.15.141:0/2579247379 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe78015e2c0 con 0x7fe75806c190 2024-01-16T15:11:21.514 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:11:21.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:21.515 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:11:21.515 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:21.515 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:11:21.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe75806c190 msgr2=0x7fe75806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe75806c190 0x7fe75806e650 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe780133130 tx=0x7fe768009040 comp rx=0 tx=0).stop 2024-01-16T15:11:21.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 msgr2=0x7fe7800ad740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800ad740 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe77000c7f0 tx=0x7fe77000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 shutdown_connections 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe75806c190 0x7fe75806e650 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78013e360 0x7fe780071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe78013df80 0x7fe7800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 --2- 172.21.15.141:0/2579247379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7800cf9a0 0x7fe7800ad740 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 >> 172.21.15.141:0/2579247379 conn(0x7fe78006a800 msgr2=0x7fe7800ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 shutdown_connections 2024-01-16T15:11:21.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.541+0000 7fe788600640 1 -- 172.21.15.141:0/2579247379 wait complete. 2024-01-16T15:11:21.537 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:11:21.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.772+0000 7f6cff200640 1 Processor -- start 2024-01-16T15:11:21.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.773+0000 7f6cff200640 1 -- start start 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.774+0000 7f6cff200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6d0015d4b0 0x7f6d000d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.774+0000 7f6cff200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d000cf9a0 0x7f6d000cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.774+0000 7f6cff200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d000d0320 con 0x7f6d000cddf0 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.774+0000 7f6cff200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d000b2730 con 0x7f6d0015d4b0 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.774+0000 7f6cff200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d000d1ec0 con 0x7f6d000cf9a0 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cfe800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6d000cddf0 0x7f6d000ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38946/0 (socket says 172.21.15.141:38946) 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cfe800640 1 -- 172.21.15.141:0/2518534122 learned_addr learned my addr 172.21.15.141:0/2518534122 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cfde00640 1 --2- 172.21.15.141:0/2518534122 >> v2:172.21.15.188:3300/0 conn(0x7f6d0015d4b0 0x7f6d000d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cf5200640 1 --2- 172.21.15.141:0/2518534122 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d000cf9a0 0x7f6d000cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cfd400640 1 -- 172.21.15.141:0/2518534122 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230981117 0 0) 0x7f6d000d0320 con 0x7f6d000cddf0 2024-01-16T15:11:21.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cfd400640 1 -- 172.21.15.141:0/2518534122 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce0003520 con 0x7f6d000cddf0 2024-01-16T15:11:21.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cfd400640 1 -- 172.21.15.141:0/2518534122 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885394010 0 0) 0x7f6ce0003520 con 0x7f6d000cddf0 2024-01-16T15:11:21.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cf5200640 1 -- 172.21.15.141:0/2518534122 >> v2:172.21.15.188:3300/0 conn(0x7f6d0015d4b0 msgr2=0x7f6d000d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:21.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cf5200640 1 --2- 172.21.15.141:0/2518534122 >> v2:172.21.15.188:3300/0 conn(0x7f6d0015d4b0 0x7f6d000d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cf5200640 1 -- 172.21.15.141:0/2518534122 >> v1:172.21.15.141:6789/0 conn(0x7f6d000cddf0 legacy=0x7f6d000ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.775+0000 7f6cf5200640 1 -- 172.21.15.141:0/2518534122 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d000b29d0 con 0x7f6d000cf9a0 2024-01-16T15:11:21.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.776+0000 7f6cfde00640 1 --2- 172.21.15.141:0/2518534122 >> v2:172.21.15.188:3300/0 conn(0x7f6d0015d4b0 0x7f6d000d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:11:21.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.777+0000 7f6cf5200640 1 --2- 172.21.15.141:0/2518534122 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d000cf9a0 0x7f6d000cfde0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80097d0 tx=0x7f6ce802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f34c53e4630c3b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.777+0000 7f6cfd400640 1 -- 172.21.15.141:0/2518534122 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80019f0 con 0x7f6d000cf9a0 2024-01-16T15:11:21.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.777+0000 7f6cfd400640 1 -- 172.21.15.141:0/2518534122 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce802e050 con 0x7f6d000cf9a0 2024-01-16T15:11:21.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.777+0000 7f6cfd400640 1 -- 172.21.15.141:0/2518534122 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8030710 con 0x7f6d000cf9a0 2024-01-16T15:11:21.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 -- 172.21.15.141:0/2518534122 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d000cf9a0 msgr2=0x7f6d000cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 --2- 172.21.15.141:0/2518534122 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d000cf9a0 0x7f6d000cfde0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80097d0 tx=0x7f6ce802b770 comp rx=0 tx=0).stop 2024-01-16T15:11:21.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 -- 172.21.15.141:0/2518534122 shutdown_connections 2024-01-16T15:11:21.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 --2- 172.21.15.141:0/2518534122 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d000cf9a0 0x7f6d000cfde0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 --2- 172.21.15.141:0/2518534122 >> v2:172.21.15.188:3300/0 conn(0x7f6d0015d4b0 0x7f6d000d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 -- 172.21.15.141:0/2518534122 >> 172.21.15.141:0/2518534122 conn(0x7f6d000d3000 msgr2=0x7f6d000d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:21.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 -- 172.21.15.141:0/2518534122 shutdown_connections 2024-01-16T15:11:21.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.778+0000 7f6cff200640 1 -- 172.21.15.141:0/2518534122 wait complete. 2024-01-16T15:11:21.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.779+0000 7f6cff200640 1 Processor -- start 2024-01-16T15:11:21.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.780+0000 7f6cff200640 1 -- start start 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cff200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d000cddf0 0x7f6d00160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cff200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d000cf9a0 0x7f6d0006c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cff200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d0015d4b0 0x7f6d0015a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cff200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d000b6be0 con 0x7f6d000cddf0 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cff200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001f6650 con 0x7f6d000cf9a0 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cff200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001f6930 con 0x7f6d0015d4b0 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d000cddf0 0x7f6d00160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d000cddf0 0x7f6d00160820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49862/0 (socket says 172.21.15.141:49862) 2024-01-16T15:11:21.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 -- 172.21.15.141:0/813714397 learned_addr learned my addr 172.21.15.141:0/813714397 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cf5200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d000cf9a0 0x7f6d0006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 -- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d0015d4b0 msgr2=0x7f6d0015a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfe800640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d0015d4b0 0x7f6d0015a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d0015d4b0 0x7f6d0015a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 -- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d000cf9a0 msgr2=0x7f6d0006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d000cf9a0 0x7f6d0006c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfde00640 1 -- 172.21.15.141:0/813714397 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf0009400 con 0x7f6d000cddf0 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cf5200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d000cf9a0 0x7f6d0006c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.781+0000 7f6cfe800640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d0015d4b0 0x7f6d0015a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.782+0000 7f6cfde00640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d000cddf0 0x7f6d00160820 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e960 tx=0x7f6cf000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.782+0000 7f6cf6a00640 1 -- 172.21.15.141:0/813714397 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000ca00 con 0x7f6d000cddf0 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.782+0000 7f6cf6a00640 1 -- 172.21.15.141:0/813714397 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf000cb60 con 0x7f6d000cddf0 2024-01-16T15:11:21.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.782+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce8009400 con 0x7f6d000cddf0 2024-01-16T15:11:21.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.782+0000 7f6cf6a00640 1 -- 172.21.15.141:0/813714397 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0013470 con 0x7f6d000cddf0 2024-01-16T15:11:21.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.782+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0015da10 con 0x7f6d000cddf0 2024-01-16T15:11:21.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.783+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d000b6e50 con 0x7f6d000cddf0 2024-01-16T15:11:21.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.784+0000 7f6cf6a00640 1 -- 172.21.15.141:0/813714397 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f6cf00135d0 con 0x7f6d000cddf0 2024-01-16T15:11:21.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.785+0000 7f6cf6a00640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6cd806c160 0x7f6cd806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:21.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.785+0000 7f6cf6a00640 1 -- 172.21.15.141:0/813714397 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6cf0088920 con 0x7f6d000cddf0 2024-01-16T15:11:21.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.786+0000 7f6cf5200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6cd806c160 0x7f6cd806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:21.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.787+0000 7f6cf5200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6cd806c160 0x7f6cd806e620 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8034000 tx=0x7f6ce8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:21.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:21.789+0000 7f6cf6a00640 1 -- 172.21.15.141:0/813714397 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6cf00551e0 con 0x7f6d000cddf0 2024-01-16T15:11:21.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.011+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d0015d890 con 0x7f6cd806c160 2024-01-16T15:11:22.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.024+0000 7f6cf6a00640 1 -- 172.21.15.141:0/813714397 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f6ce8004440 con 0x7f6cd806c160 2024-01-16T15:11:22.000 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:22.000 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:11:22.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:22.001 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "sv1" 2024-01-16T15:11:22.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:22.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:22.001 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1clone1" 2024-01-16T15:11:22.001 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:22.001 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:11:22.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.026+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6cd806c160 msgr2=0x7f6cd806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.026+0000 7f6cff200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6cd806c160 0x7f6cd806e620 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8034000 tx=0x7f6ce8008030 comp rx=0 tx=0).stop 2024-01-16T15:11:22.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.026+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d000cddf0 msgr2=0x7f6d00160820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.026+0000 7f6cff200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d000cddf0 0x7f6d00160820 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e960 tx=0x7f6cf000ee20 comp rx=0 tx=0).stop 2024-01-16T15:11:22.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 shutdown_connections 2024-01-16T15:11:22.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6cd806c160 0x7f6cd806e620 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6d0015d4b0 0x7f6d0015a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d000cf9a0 0x7f6d0006c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 --2- 172.21.15.141:0/813714397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d000cddf0 0x7f6d00160820 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 >> 172.21.15.141:0/813714397 conn(0x7f6d000d3000 msgr2=0x7f6d000cfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 shutdown_connections 2024-01-16T15:11:22.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.027+0000 7f6cff200640 1 -- 172.21.15.141:0/813714397 wait complete. 2024-01-16T15:11:22.022 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:11:22.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.252+0000 7f61c9800640 1 Processor -- start 2024-01-16T15:11:22.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.252+0000 7f61c9800640 1 -- start start 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c9800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f61c40cc500 0x7f61c40cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 0x7f61c406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c9800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c4070e50 con 0x7f61c4143c40 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c9800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c40723c0 con 0x7f61c40cc500 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40cced0 con 0x7f61c406d160 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f61c4143c40 0x7f61c4071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38954/0 (socket says 172.21.15.141:38954) 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c3e00640 1 -- 172.21.15.141:0/1857624911 learned_addr learned my addr 172.21.15.141:0/1857624911 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:22.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61bb200640 1 --2- 172.21.15.141:0/1857624911 >> v2:172.21.15.141:3300/0 conn(0x7f61c40cc500 0x7f61c40cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c3400640 1 -- 172.21.15.141:0/1857624911 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402555412 0 0) 0x7f61c4070e50 con 0x7f61c4143c40 2024-01-16T15:11:22.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61bb200640 1 -- 172.21.15.141:0/1857624911 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 msgr2=0x7f61c406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.254+0000 7f61c8e00640 1 --2- 172.21.15.141:0/1857624911 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 0x7f61c406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.255+0000 7f61bb200640 1 --2- 172.21.15.141:0/1857624911 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 0x7f61c406d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.255+0000 7f61bb200640 1 -- 172.21.15.141:0/1857624911 >> v1:172.21.15.141:6789/0 conn(0x7f61c4143c40 legacy=0x7f61c4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.255+0000 7f61bb200640 1 -- 172.21.15.141:0/1857624911 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4144020 con 0x7f61c40cc500 2024-01-16T15:11:22.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61bb200640 1 --2- 172.21.15.141:0/1857624911 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c40cc990 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0097a0 tx=0x7f61ac02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe77d3b4e026753 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c3400640 1 -- 172.21.15.141:0/1857624911 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac031780 con 0x7f61c40cc500 2024-01-16T15:11:22.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c3400640 1 -- 172.21.15.141:0/1857624911 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac0318e0 con 0x7f61c40cc500 2024-01-16T15:11:22.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c3400640 1 -- 172.21.15.141:0/1857624911 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac031c70 con 0x7f61c40cc500 2024-01-16T15:11:22.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c9800640 1 -- 172.21.15.141:0/1857624911 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 msgr2=0x7f61c40cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c9800640 1 --2- 172.21.15.141:0/1857624911 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c40cc990 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0097a0 tx=0x7f61ac02b820 comp rx=0 tx=0).stop 2024-01-16T15:11:22.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c9800640 1 -- 172.21.15.141:0/1857624911 shutdown_connections 2024-01-16T15:11:22.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c9800640 1 --2- 172.21.15.141:0/1857624911 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 0x7f61c406d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.256+0000 7f61c9800640 1 --2- 172.21.15.141:0/1857624911 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c40cc990 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.257+0000 7f61c9800640 1 -- 172.21.15.141:0/1857624911 >> 172.21.15.141:0/1857624911 conn(0x7f61c406a800 msgr2=0x7f61c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.257+0000 7f61c9800640 1 -- 172.21.15.141:0/1857624911 shutdown_connections 2024-01-16T15:11:22.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.257+0000 7f61c9800640 1 -- 172.21.15.141:0/1857624911 wait complete. 2024-01-16T15:11:22.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.258+0000 7f61c9800640 1 Processor -- start 2024-01-16T15:11:22.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.258+0000 7f61c9800640 1 -- start start 2024-01-16T15:11:22.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.259+0000 7f61c9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 0x7f61c4157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.259+0000 7f61c9800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c41f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.259+0000 7f61c9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61c4143c40 0x7f61c413b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.259+0000 7f61c9800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c413cca0 con 0x7f61c40cc500 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.259+0000 7f61c9800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c41582d0 con 0x7f61c4143c40 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.259+0000 7f61c9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c413c0d0 con 0x7f61c406d160 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c41f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c41f2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49870/0 (socket says 172.21.15.141:49870) 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 -- 172.21.15.141:0/1750629562 learned_addr learned my addr 172.21.15.141:0/1750629562 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61c8e00640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61c4143c40 0x7f61c413b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 -- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 msgr2=0x7f61c4157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 0x7f61c4157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 -- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61c4143c40 msgr2=0x7f61c413b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61c4143c40 0x7f61c413b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 -- 172.21.15.141:0/1750629562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac009400 con 0x7f61c40cc500 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61c8e00640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61c4143c40 0x7f61c413b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:22.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.260+0000 7f61bb200640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c41f2ef0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f61ac034040 tx=0x7f61ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.261+0000 7f61c0c00640 1 -- 172.21.15.141:0/1750629562 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac032960 con 0x7f61c40cc500 2024-01-16T15:11:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.261+0000 7f61c0c00640 1 -- 172.21.15.141:0/1750629562 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac032ac0 con 0x7f61c40cc500 2024-01-16T15:11:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.261+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c41f3430 con 0x7f61c40cc500 2024-01-16T15:11:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.261+0000 7f61c0c00640 1 -- 172.21.15.141:0/1750629562 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac032e70 con 0x7f61c40cc500 2024-01-16T15:11:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.261+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c413c650 con 0x7f61c40cc500 2024-01-16T15:11:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.262+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c40cd970 con 0x7f61c40cc500 2024-01-16T15:11:22.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.263+0000 7f61c0c00640 1 -- 172.21.15.141:0/1750629562 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f61ac03b3f0 con 0x7f61c40cc500 2024-01-16T15:11:22.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.264+0000 7f61c0c00640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f619c06c160 0x7f619c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.264+0000 7f61c0c00640 1 -- 172.21.15.141:0/1750629562 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f61ac0aab60 con 0x7f61c40cc500 2024-01-16T15:11:22.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.265+0000 7f61c3e00640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f619c06c160 0x7f619c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.266+0000 7f61c3e00640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f619c06c160 0x7f619c06e620 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f61b4005c00 tx=0x7f61b40015e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.268+0000 7f61c0c00640 1 -- 172.21.15.141:0/1750629562 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61ac077420 con 0x7f61c40cc500 2024-01-16T15:11:22.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.489+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61c406c960 con 0x7f619c06c160 2024-01-16T15:11:22.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.519+0000 7f61c0c00640 1 -- 172.21.15.141:0/1750629562 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f61b400cbc0 con 0x7f619c06c160 2024-01-16T15:11:22.495 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:22.495 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:11:22.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:22.496 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1" 2024-01-16T15:11:22.496 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:22.496 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:11:22.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.521+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f619c06c160 msgr2=0x7f619c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.521+0000 7f61c9800640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f619c06c160 0x7f619c06e620 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f61b4005c00 tx=0x7f61b40015e0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 msgr2=0x7f61c41f2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c41f2ef0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f61ac034040 tx=0x7f61ac0046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 shutdown_connections 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f619c06c160 0x7f619c06e620 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61c4143c40 0x7f61c413b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61c40cc500 0x7f61c41f2ef0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 --2- 172.21.15.141:0/1750629562 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f61c406d160 0x7f61c4157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 >> 172.21.15.141:0/1750629562 conn(0x7f61c406a800 msgr2=0x7f61c413f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 shutdown_connections 2024-01-16T15:11:22.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.522+0000 7f61c9800640 1 -- 172.21.15.141:0/1750629562 wait complete. 2024-01-16T15:11:22.517 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:11:22.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.757+0000 7f3652000640 1 Processor -- start 2024-01-16T15:11:22.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.758+0000 7f3652000640 1 -- start start 2024-01-16T15:11:22.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.759+0000 7f3652000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f364c0cf9a0 0x7f364c166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.759+0000 7f3652000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c15c5f0 con 0x7f364c13e360 2024-01-16T15:11:22.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.759+0000 7f3652000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c06c470 con 0x7f364c13df80 2024-01-16T15:11:22.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.759+0000 7f3652000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364c06c750 con 0x7f364c0cf9a0 2024-01-16T15:11:22.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364aa00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f364c13e360 0x7f364c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38964/0 (socket says 172.21.15.141:38964) 2024-01-16T15:11:22.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364aa00640 1 -- 172.21.15.141:0/1047369445 learned_addr learned my addr 172.21.15.141:0/1047369445 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:22.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364be00640 1 --2- 172.21.15.141:0/1047369445 >> v2:172.21.15.188:3300/0 conn(0x7f364c0cf9a0 0x7f364c166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748033897 0 0) 0x7f364c15c5f0 con 0x7f364c13e360 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f362c003520 con 0x7f364c13e360 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199322626 0 0) 0x7f362c003520 con 0x7f364c13e360 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 >> v2:172.21.15.188:3300/0 conn(0x7f364c0cf9a0 msgr2=0x7f364c166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364a000640 1 --2- 172.21.15.141:0/1047369445 >> v2:172.21.15.188:3300/0 conn(0x7f364c0cf9a0 0x7f364c166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.760+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 >> v1:172.21.15.188:6789/0 conn(0x7f364c13df80 legacy=0x7f364c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c06ca30 con 0x7f364c13e360 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3634001670 con 0x7f364c13e360 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 >> v1:172.21.15.141:6789/0 conn(0x7f364c13e360 legacy=0x7f364c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f362c003970 0x7f362c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f362c0047c0 0x7f362c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f362c005180 0x7f362c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c06c750 con 0x7f362c005180 2024-01-16T15:11:22.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c00c020 con 0x7f362c003970 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362c00c1e0 con 0x7f362c0047c0 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364be00640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f362c005180 0x7f362c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364b400640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f362c003970 0x7f362c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364be00640 1 -- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f362c0047c0 msgr2=0x7f362c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364be00640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f362c0047c0 0x7f362c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364be00640 1 -- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f362c003970 msgr2=0x7f362c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364be00640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f362c003970 0x7f362c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364be00640 1 -- 172.21.15.141:0/1047369445 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c003520 con 0x7f362c005180 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.761+0000 7f364b400640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f362c003970 0x7f362c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.762+0000 7f364be00640 1 -- 172.21.15.141:0/1047369445 reap_dead start 2024-01-16T15:11:22.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f364be00640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f362c005180 0x7f362c00b980 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3644009a90 tx=0x7f362c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff58cd4440dd23 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644004020 con 0x7f362c005180 2024-01-16T15:11:22.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3644004180 con 0x7f362c005180 2024-01-16T15:11:22.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f364a000640 1 -- 172.21.15.141:0/1047369445 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364402c6b0 con 0x7f362c005180 2024-01-16T15:11:22.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f3652000640 1 -- 172.21.15.141:0/1047369445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f362c005180 msgr2=0x7f362c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f3652000640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f362c005180 0x7f362c00b980 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3644009a90 tx=0x7f362c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f3652000640 1 -- 172.21.15.141:0/1047369445 shutdown_connections 2024-01-16T15:11:22.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f3652000640 1 --2- 172.21.15.141:0/1047369445 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f362c005180 0x7f362c00b980 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.763+0000 7f3652000640 1 -- 172.21.15.141:0/1047369445 >> 172.21.15.141:0/1047369445 conn(0x7f364c06a800 msgr2=0x7f364c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:22.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.764+0000 7f3652000640 1 -- 172.21.15.141:0/1047369445 shutdown_connections 2024-01-16T15:11:22.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.764+0000 7f3652000640 1 -- 172.21.15.141:0/1047369445 wait complete. 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.765+0000 7f3652000640 1 Processor -- start 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.766+0000 7f3652000640 1 -- start start 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.766+0000 7f3652000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f362c005180 0x7f364c13d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f3652000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364c14d810 0x7f364c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f3652000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364c0b2cf0 0x7f364c0b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f3652000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0b36e0 con 0x7f364c14d810 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f3652000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0710d0 con 0x7f364c0b2cf0 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f3652000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0713b0 con 0x7f362c005180 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364c14d810 0x7f364c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.743 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364c14d810 0x7f364c0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49890/0 (socket says 172.21.15.141:49890) 2024-01-16T15:11:22.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 -- 172.21.15.141:0/838388165 learned_addr learned my addr 172.21.15.141:0/838388165 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 -- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f362c005180 msgr2=0x7f364c13d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f362c005180 0x7f364c13d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 -- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364c0b2cf0 msgr2=0x7f364c0b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364c0b2cf0 0x7f364c0b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 -- 172.21.15.141:0/838388165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c071690 con 0x7f364c14d810 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.767+0000 7f364aa00640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364c14d810 0x7f364c0707d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f363400c6e0 tx=0x7f363400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.768+0000 7f3643400640 1 -- 172.21.15.141:0/838388165 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36340072d0 con 0x7f364c14d810 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.768+0000 7f3643400640 1 -- 172.21.15.141:0/838388165 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3634007430 con 0x7f364c14d810 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.768+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c071970 con 0x7f364c14d810 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.768+0000 7f3643400640 1 -- 172.21.15.141:0/838388165 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36340047e0 con 0x7f364c14d810 2024-01-16T15:11:22.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.768+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c13dc70 con 0x7f364c14d810 2024-01-16T15:11:22.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.769+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c071e90 con 0x7f364c14d810 2024-01-16T15:11:22.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.770+0000 7f3643400640 1 -- 172.21.15.141:0/838388165 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3634004940 con 0x7f364c14d810 2024-01-16T15:11:22.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.771+0000 7f3643400640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f362406c190 0x7f362406e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:22.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.771+0000 7f3643400640 1 -- 172.21.15.141:0/838388165 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f36340880b0 con 0x7f364c14d810 2024-01-16T15:11:22.747 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.772+0000 7f364b400640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f362406c190 0x7f362406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:22.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.773+0000 7f364b400640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f362406c190 0x7f362406e650 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f363c0012e0 tx=0x7f363c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:22.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:22.775+0000 7f3643400640 1 -- 172.21.15.141:0/838388165 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f364c071e90 con 0x7f364c14d810 2024-01-16T15:11:22.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.001+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f364c06cff0 con 0x7f362406c190 2024-01-16T15:11:23.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.068+0000 7f3643400640 1 -- 172.21.15.141:0/838388165 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f363c00d9a0 con 0x7f362406c190 2024-01-16T15:11:23.045 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:23.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.071+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f362406c190 msgr2=0x7f362406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.071+0000 7f3652000640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f362406c190 0x7f362406e650 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f363c0012e0 tx=0x7f363c009040 comp rx=0 tx=0).stop 2024-01-16T15:11:23.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.071+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364c14d810 msgr2=0x7f364c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.071+0000 7f3652000640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364c14d810 0x7f364c0707d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f363400c6e0 tx=0x7f363400cba0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 shutdown_connections 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f362406c190 0x7f362406e650 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f364c0b2cf0 0x7f364c0b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f364c14d810 0x7f364c0707d0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 --2- 172.21.15.141:0/838388165 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f362c005180 0x7f364c13d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 >> 172.21.15.141:0/838388165 conn(0x7f364c06a800 msgr2=0x7f364c153950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 shutdown_connections 2024-01-16T15:11:23.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.072+0000 7f3652000640 1 -- 172.21.15.141:0/838388165 wait complete. 2024-01-16T15:11:23.067 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:11:23.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.306+0000 7fdb13600640 1 Processor -- start 2024-01-16T15:11:23.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.306+0000 7fdb13600640 1 -- start start 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb13600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdb0c143ae0 0x7fdb0c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb13600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdb0c0cf7d0 0x7fdb0c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb13600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c15e1e0 0x7fdb0c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb13600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0c0cfc30 con 0x7fdb0c143ae0 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb13600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0c0b1ee0 con 0x7fdb0c0cf7d0 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb13600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0b2780 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb10c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdb0c143ae0 0x7fdb0c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb10c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdb0c143ae0 0x7fdb0c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42310/0 (socket says 172.21.15.141:42310) 2024-01-16T15:11:23.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.308+0000 7fdb10c00640 1 -- 172.21.15.141:0/3743748644 learned_addr learned my addr 172.21.15.141:0/3743748644 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb10c00640 1 -- 172.21.15.141:0/3743748644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c15e1e0 msgr2=0x7fdb0c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb11600640 1 --2- 172.21.15.141:0/3743748644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c15e1e0 0x7fdb0c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb0be00640 1 --2- 172.21.15.141:0/3743748644 >> v2:172.21.15.188:3300/0 conn(0x7fdb0c0cf7d0 0x7fdb0c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb10c00640 1 --2- 172.21.15.141:0/3743748644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c15e1e0 0x7fdb0c15a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb10c00640 1 -- 172.21.15.141:0/3743748644 >> v2:172.21.15.188:3300/0 conn(0x7fdb0c0cf7d0 msgr2=0x7fdb0c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb10c00640 1 --2- 172.21.15.141:0/3743748644 >> v2:172.21.15.188:3300/0 conn(0x7fdb0c0cf7d0 0x7fdb0c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb10c00640 1 -- 172.21.15.141:0/3743748644 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c15e990 con 0x7fdb0c143ae0 2024-01-16T15:11:23.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.309+0000 7fdb11600640 1 --2- 172.21.15.141:0/3743748644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c15e1e0 0x7fdb0c15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.310+0000 7fdb10c00640 1 --2- 172.21.15.141:0/3743748644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c0cf290 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0cb830 tx=0x7fdafc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f7e01af358121b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.310+0000 7fdb0b400640 1 -- 172.21.15.141:0/3743748644 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc02ca70 con 0x7fdb0c143ae0 2024-01-16T15:11:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.310+0000 7fdb0b400640 1 -- 172.21.15.141:0/3743748644 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc02cbd0 con 0x7fdb0c143ae0 2024-01-16T15:11:23.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.310+0000 7fdb0b400640 1 -- 172.21.15.141:0/3743748644 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0343f0 con 0x7fdb0c143ae0 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 -- 172.21.15.141:0/3743748644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c143ae0 msgr2=0x7fdb0c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 --2- 172.21.15.141:0/3743748644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c0cf290 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0cb830 tx=0x7fdafc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 -- 172.21.15.141:0/3743748644 shutdown_connections 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 --2- 172.21.15.141:0/3743748644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c15e1e0 0x7fdb0c15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 --2- 172.21.15.141:0/3743748644 >> v2:172.21.15.188:3300/0 conn(0x7fdb0c0cf7d0 0x7fdb0c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 --2- 172.21.15.141:0/3743748644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c0cf290 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 -- 172.21.15.141:0/3743748644 >> 172.21.15.141:0/3743748644 conn(0x7fdb0c06a800 msgr2=0x7fdb0c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 -- 172.21.15.141:0/3743748644 shutdown_connections 2024-01-16T15:11:23.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.311+0000 7fdb13600640 1 -- 172.21.15.141:0/3743748644 wait complete. 2024-01-16T15:11:23.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.313+0000 7fdb13600640 1 Processor -- start 2024-01-16T15:11:23.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.313+0000 7fdb13600640 1 -- start start 2024-01-16T15:11:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb13600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c0cf7d0 0x7fdb0c1594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb13600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c1f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb13600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c15e1e0 0x7fdb0c1f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb13600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c1f1fa0 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb13600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c1f2280 con 0x7fdb0c143ae0 2024-01-16T15:11:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb13600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c1f2560 con 0x7fdb0c0cf7d0 2024-01-16T15:11:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb11600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c15e1e0 0x7fdb0c1f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb11600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c15e1e0 0x7fdb0c1f4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42322/0 (socket says 172.21.15.141:42322) 2024-01-16T15:11:23.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.314+0000 7fdb11600640 1 -- 172.21.15.141:0/2217832888 learned_addr learned my addr 172.21.15.141:0/2217832888 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb0be00640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c1f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb10c00640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c0cf7d0 0x7fdb0c1594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb11600640 1 -- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c0cf7d0 msgr2=0x7fdb0c1594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb11600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c0cf7d0 0x7fdb0c1594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb11600640 1 -- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb0c143ae0 msgr2=0x7fdb0c1f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb11600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c1f4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb11600640 1 -- 172.21.15.141:0/2217832888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc009b30 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb0be00640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c1f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb11600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c15e1e0 0x7fdb0c1f4890 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c7c0 tx=0x7fdb0400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb08c00640 1 -- 172.21.15.141:0/2217832888 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04004230 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb08c00640 1 -- 172.21.15.141:0/2217832888 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb04004390 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c1f2840 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb08c00640 1 -- 172.21.15.141:0/2217832888 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04007860 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.315+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c1ee910 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.317+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c0b2bf0 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.317+0000 7fdb08c00640 1 -- 172.21.15.141:0/2217832888 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fdb040079c0 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.318+0000 7fdb08c00640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdae406c160 0x7fdae406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.318+0000 7fdb08c00640 1 -- 172.21.15.141:0/2217832888 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdb04088be0 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.319+0000 7fdb10c00640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdae406c160 0x7fdae406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.320+0000 7fdb10c00640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdae406c160 0x7fdae406e620 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdafc035000 tx=0x7fdafc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.323+0000 7fdb08c00640 1 -- 172.21.15.141:0/2217832888 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb0c0b2bf0 con 0x7fdb0c15e1e0 2024-01-16T15:11:23.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.539+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb0c15e670 con 0x7fdae406c160 2024-01-16T15:11:23.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.583+0000 7fdb08c00640 1 -- 172.21.15.141:0/2217832888 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdafc040450 con 0x7fdae406c160 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.586+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdae406c160 msgr2=0x7fdae406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.586+0000 7fdb13600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdae406c160 0x7fdae406e620 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdafc035000 tx=0x7fdafc030000 comp rx=0 tx=0).stop 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.586+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c15e1e0 msgr2=0x7fdb0c1f4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.586+0000 7fdb13600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c15e1e0 0x7fdb0c1f4890 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c7c0 tx=0x7fdb0400cc80 comp rx=0 tx=0).stop 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 shutdown_connections 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdae406c160 0x7fdae406e620 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdb0c15e1e0 0x7fdb0c1f4890 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb0c143ae0 0x7fdb0c1f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 --2- 172.21.15.141:0/2217832888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdb0c0cf7d0 0x7fdb0c1594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 >> 172.21.15.141:0/2217832888 conn(0x7fdb0c06a800 msgr2=0x7fdb0c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:23.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 shutdown_connections 2024-01-16T15:11:23.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.587+0000 7fdb13600640 1 -- 172.21.15.141:0/2217832888 wait complete. 2024-01-16T15:11:23.569 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.592+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.569 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.592+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.569 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.592+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.569 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.592+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.569 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.592+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.584 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T15:11:23.598 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.621+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.598 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.621+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.598 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.621+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.598 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.621+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.598 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:23.621+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:23.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.811+0000 7f45c2c00640 1 Processor -- start 2024-01-16T15:11:23.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.811+0000 7f45c2c00640 1 -- start start 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45c2c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45bc0cd240 0x7f45bc0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45c2c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f45bc06c6b0 0x7f45bc099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45c2c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc099fa0 0x7f45bc06b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45c2c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc09a430 con 0x7f45bc0cd240 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45c2c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc06cea0 con 0x7f45bc06c6b0 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45c2c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc06cbc0 con 0x7f45bc099fa0 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45bbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45bc0cd240 0x7f45bc0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45bbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f45bc0cd240 0x7f45bc0b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42334/0 (socket says 172.21.15.141:42334) 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.813+0000 7f45bbe00640 1 -- 172.21.15.141:0/3545598627 learned_addr learned my addr 172.21.15.141:0/3545598627 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.814+0000 7f45bb400640 1 --2- 172.21.15.141:0/3545598627 >> v2:172.21.15.188:3300/0 conn(0x7f45bc06c6b0 0x7f45bc099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.814+0000 7f45c0c00640 1 --2- 172.21.15.141:0/3545598627 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc099fa0 0x7f45bc06b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.814+0000 7f45bbe00640 1 -- 172.21.15.141:0/3545598627 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc099fa0 msgr2=0x7f45bc06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.814+0000 7f45bbe00640 1 --2- 172.21.15.141:0/3545598627 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc099fa0 0x7f45bc06b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.814+0000 7f45bbe00640 1 -- 172.21.15.141:0/3545598627 >> v2:172.21.15.188:3300/0 conn(0x7f45bc06c6b0 msgr2=0x7f45bc099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.814+0000 7f45bbe00640 1 --2- 172.21.15.141:0/3545598627 >> v2:172.21.15.188:3300/0 conn(0x7f45bc06c6b0 0x7f45bc099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.814+0000 7f45bbe00640 1 -- 172.21.15.141:0/3545598627 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc06f350 con 0x7f45bc0cd240 2024-01-16T15:11:23.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.815+0000 7f45bbe00640 1 --2- 172.21.15.141:0/3545598627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc0b3820 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f45bc071800 tx=0x7f45ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c050e0787a41a071 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.815+0000 7f45baa00640 1 -- 172.21.15.141:0/3545598627 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac031870 con 0x7f45bc0cd240 2024-01-16T15:11:23.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.815+0000 7f45baa00640 1 -- 172.21.15.141:0/3545598627 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac0319d0 con 0x7f45bc0cd240 2024-01-16T15:11:23.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.815+0000 7f45baa00640 1 -- 172.21.15.141:0/3545598627 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac031d60 con 0x7f45bc0cd240 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 -- 172.21.15.141:0/3545598627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 msgr2=0x7f45bc0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 --2- 172.21.15.141:0/3545598627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc0b3820 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f45bc071800 tx=0x7f45ac02b910 comp rx=0 tx=0).stop 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 -- 172.21.15.141:0/3545598627 shutdown_connections 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 --2- 172.21.15.141:0/3545598627 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc099fa0 0x7f45bc06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 --2- 172.21.15.141:0/3545598627 >> v2:172.21.15.188:3300/0 conn(0x7f45bc06c6b0 0x7f45bc099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 --2- 172.21.15.141:0/3545598627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc0b3820 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 -- 172.21.15.141:0/3545598627 >> 172.21.15.141:0/3545598627 conn(0x7f45bc0b2d00 msgr2=0x7f45bc0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 -- 172.21.15.141:0/3545598627 shutdown_connections 2024-01-16T15:11:23.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.816+0000 7f45c2c00640 1 -- 172.21.15.141:0/3545598627 wait complete. 2024-01-16T15:11:23.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.818+0000 7f45c2c00640 1 Processor -- start 2024-01-16T15:11:23.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.818+0000 7f45c2c00640 1 -- start start 2024-01-16T15:11:23.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c2c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc06c6b0 0x7f45bc0b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c2c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45bc099fa0 0x7f45bc15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c2c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc15f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c2c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0c4d00 con 0x7f45bc0cd240 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c2c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0c4f50 con 0x7f45bc099fa0 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c2c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0c5230 con 0x7f45bc06c6b0 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc15f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c0c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc15f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42336/0 (socket says 172.21.15.141:42336) 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45c0c00640 1 -- 172.21.15.141:0/43870734 learned_addr learned my addr 172.21.15.141:0/43870734 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45bb400640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45bc099fa0 0x7f45bc15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.819+0000 7f45bbe00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc06c6b0 0x7f45bc0b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c0c00640 1 -- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc06c6b0 msgr2=0x7f45bc0b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c0c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc06c6b0 0x7f45bc0b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c0c00640 1 -- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45bc099fa0 msgr2=0x7f45bc15eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:23.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c0c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45bc099fa0 0x7f45bc15eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c0c00640 1 -- 172.21.15.141:0/43870734 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0094d0 con 0x7f45bc0cd240 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45bbe00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc06c6b0 0x7f45bc0b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c0c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc15f2e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f45b400cc30 tx=0x7f45b400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45b3e00640 1 -- 172.21.15.141:0/43870734 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b400ce40 con 0x7f45bc0cd240 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45b3e00640 1 -- 172.21.15.141:0/43870734 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b4003990 con 0x7f45bc0cd240 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45b3e00640 1 -- 172.21.15.141:0/43870734 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b4003d20 con 0x7f45bc0cd240 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc0c5510 con 0x7f45bc0cd240 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.820+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc0c5ad0 con 0x7f45bc0cd240 2024-01-16T15:11:23.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.821+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc0c5cf0 con 0x7f45bc0cd240 2024-01-16T15:11:23.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.823+0000 7f45b3e00640 1 -- 172.21.15.141:0/43870734 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f45b4003e80 con 0x7f45bc0cd240 2024-01-16T15:11:23.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.824+0000 7f45b3e00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459406c160 0x7f459406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:23.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.824+0000 7f45b3e00640 1 -- 172.21.15.141:0/43870734 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f45b4088df0 con 0x7f45bc0cd240 2024-01-16T15:11:23.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.825+0000 7f45bbe00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459406c160 0x7f459406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:23.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.826+0000 7f45bbe00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459406c160 0x7f459406e620 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f45ac036000 tx=0x7f45ac03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:23.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:23.831+0000 7f45b3e00640 1 -- 172.21.15.141:0/43870734 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45b40556b0 con 0x7f45bc0cd240 2024-01-16T15:11:24.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.051+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f45bc0cd620 con 0x7f459406c160 2024-01-16T15:11:24.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.088+0000 7f45b3e00640 1 -- 172.21.15.141:0/43870734 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f45ac031c40 con 0x7f459406c160 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459406c160 msgr2=0x7f459406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459406c160 0x7f459406e620 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f45ac036000 tx=0x7f45ac03c040 comp rx=0 tx=0).stop 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 msgr2=0x7f45bc15f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc15f2e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f45b400cc30 tx=0x7f45b400a5c0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 shutdown_connections 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f459406c160 0x7f459406e620 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45bc0cd240 0x7f45bc15f2e0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45bc099fa0 0x7f45bc15eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 --2- 172.21.15.141:0/43870734 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f45bc06c6b0 0x7f45bc0b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 >> 172.21.15.141:0/43870734 conn(0x7f45bc0b2d00 msgr2=0x7f45bc1607a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.092+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 shutdown_connections 2024-01-16T15:11:24.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.093+0000 7f45c2c00640 1 -- 172.21.15.141:0/43870734 wait complete. 2024-01-16T15:11:24.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:24.088 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:11:24.151 INFO:teuthology.orchestra.run.smithi141.stdout:2 2024-01-16T15:11:24.152 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:11:24.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.359+0000 7f4643400640 1 Processor -- start 2024-01-16T15:11:24.335 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.359+0000 7f4643400640 1 -- start start 2024-01-16T15:11:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.360+0000 7f4643400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 0x7f463c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.360+0000 7f4643400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f463c0cf9a0 0x7f463c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.360+0000 7f4643400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463c06c470 con 0x7f463c13e360 2024-01-16T15:11:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.360+0000 7f4643400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463c06c750 con 0x7f463c0cf9a0 2024-01-16T15:11:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.360+0000 7f4643400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c06ca30 con 0x7f463c13df80 2024-01-16T15:11:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.361+0000 7f463be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f463c13e360 0x7f463c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34458/0 (socket says 172.21.15.141:34458) 2024-01-16T15:11:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.361+0000 7f463be00640 1 -- 172.21.15.141:0/3705452147 learned_addr learned my addr 172.21.15.141:0/3705452147 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:24.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.361+0000 7f4640a00640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 0x7f463c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.361+0000 7f4641400640 1 --2- 172.21.15.141:0/3705452147 >> v2:172.21.15.188:3300/0 conn(0x7f463c0cf9a0 0x7f463c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.361+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102900653 0 0) 0x7f463c06c470 con 0x7f463c13e360 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.361+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461c003520 con 0x7f463c13e360 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1037634889 0 0) 0x7f461c003520 con 0x7f463c13e360 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 msgr2=0x7f463c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 0x7f463c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 >> v2:172.21.15.188:3300/0 conn(0x7f463c0cf9a0 msgr2=0x7f463c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 --2- 172.21.15.141:0/3705452147 >> v2:172.21.15.188:3300/0 conn(0x7f463c0cf9a0 0x7f463c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c06cd10 con 0x7f463c13e360 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4624001670 con 0x7f463c13e360 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 >> v1:172.21.15.141:6789/0 conn(0x7f463c13e360 legacy=0x7f463c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 0x7f461c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f461c003ef0 0x7f461c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f461c004d00 0x7f461c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c06c750 con 0x7f461c003ef0 2024-01-16T15:11:24.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c06ca30 con 0x7f461c004d00 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c00b980 con 0x7f463c13df80 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f463be00640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f461c003ef0 0x7f461c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.362+0000 7f4640a00640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 0x7f461c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.363+0000 7f463be00640 1 -- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 msgr2=0x7f461c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.363+0000 7f463be00640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f463c13df80 0x7f461c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.363+0000 7f4641400640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f461c004d00 0x7f461c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.363+0000 7f463be00640 1 -- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f461c004d00 msgr2=0x7f461c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.363+0000 7f463be00640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f461c004d00 0x7f461c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.363+0000 7f463be00640 1 -- 172.21.15.141:0/3705452147 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c003520 con 0x7f461c003ef0 2024-01-16T15:11:24.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.364+0000 7f463be00640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f461c003ef0 0x7f461c0047c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f462400c240 tx=0x7f461c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8318768c69be3a7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:24.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.364+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462402cae0 con 0x7f461c003ef0 2024-01-16T15:11:24.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.364+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462402cc40 con 0x7f461c003ef0 2024-01-16T15:11:24.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.364+0000 7f463b400640 1 -- 172.21.15.141:0/3705452147 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624005af0 con 0x7f461c003ef0 2024-01-16T15:11:24.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.365+0000 7f4643400640 1 -- 172.21.15.141:0/3705452147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f461c003ef0 msgr2=0x7f461c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.365+0000 7f4643400640 1 --2- 172.21.15.141:0/3705452147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f461c003ef0 0x7f461c0047c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f462400c240 tx=0x7f461c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.342 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.365+0000 7f4641400640 1 -- 172.21.15.141:0/3705452147 reap_dead start 2024-01-16T15:11:24.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.365+0000 7f4643400640 1 -- 172.21.15.141:0/3705452147 shutdown_connections 2024-01-16T15:11:24.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.365+0000 7f4643400640 1 -- 172.21.15.141:0/3705452147 >> 172.21.15.141:0/3705452147 conn(0x7f463c06a800 msgr2=0x7f463c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:24.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.365+0000 7f4643400640 1 -- 172.21.15.141:0/3705452147 shutdown_connections 2024-01-16T15:11:24.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.365+0000 7f4643400640 1 -- 172.21.15.141:0/3705452147 wait complete. 2024-01-16T15:11:24.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.367+0000 7f4643400640 1 Processor -- start 2024-01-16T15:11:24.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.367+0000 7f4643400640 1 -- start start 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f4643400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f461c003ef0 0x7f463c1f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f4643400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f463c1569e0 0x7f463c0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f4643400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f463c151850 0x7f463c151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f4643400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1f1a30 con 0x7f463c1569e0 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f4643400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1f1190 con 0x7f463c151850 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f4643400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1f1470 con 0x7f461c003ef0 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f463be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f463c1569e0 0x7f463c0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f463be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f463c1569e0 0x7f463c0afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42350/0 (socket says 172.21.15.141:42350) 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.368+0000 7f463be00640 1 -- 172.21.15.141:0/2363421892 learned_addr learned my addr 172.21.15.141:0/2363421892 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f4640a00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f461c003ef0 0x7f463c1f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f4641400640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f463c151850 0x7f463c151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f463be00640 1 -- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f461c003ef0 msgr2=0x7f463c1f0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f463be00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f461c003ef0 0x7f463c1f0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f463be00640 1 -- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f463c151850 msgr2=0x7f463c151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f463be00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f463c151850 0x7f463c151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f463be00640 1 -- 172.21.15.141:0/2363421892 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c1f1750 con 0x7f463c1569e0 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f4640a00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f461c003ef0 0x7f463c1f0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f463be00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f463c1569e0 0x7f463c0afdf0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f462402c4e0 tx=0x7f4624001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624039070 con 0x7f463c1569e0 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4624004bf0 con 0x7f463c1569e0 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462403c040 con 0x7f463c1569e0 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.369+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c1507e0 con 0x7f463c1569e0 2024-01-16T15:11:24.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.370+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c0b0330 con 0x7f463c1569e0 2024-01-16T15:11:24.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.370+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463c0b0520 con 0x7f463c1569e0 2024-01-16T15:11:24.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.372+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 31) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4624032070 con 0x7f463c1569e0 2024-01-16T15:11:24.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.372+0000 7f4638c00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f461406c190 0x7f461406e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:24.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.373+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f46240a9750 con 0x7f463c1569e0 2024-01-16T15:11:24.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.373+0000 7f4640a00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f461406c190 0x7f461406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:24.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.374+0000 7f4640a00640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f461406c190 0x7f461406e650 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f462c006ce0 tx=0x7f462c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:24.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.377+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4624079aa0 con 0x7f463c1569e0 2024-01-16T15:11:24.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:24.591+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f463c157020 con 0x7f461406c190 2024-01-16T15:11:24.611 INFO:tasks.ceph.mon.a.smithi141.stderr:2024-01-16T15:11:24.632+0000 7f723b200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:11:24.700 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:11:25.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:25.667+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f46240763e0 con 0x7f463c1569e0 2024-01-16T15:11:27.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.684+0000 7f4638c00640 1 -- 172.21.15.141:0/2363421892 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f463c157020 con 0x7f461406c190 2024-01-16T15:11:27.663 INFO:teuthology.orchestra.run.smithi141.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:11:27.667 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f461406c190 msgr2=0x7f461406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f461406c190 0x7f461406e650 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f462c006ce0 tx=0x7f462c008040 comp rx=0 tx=0).stop 2024-01-16T15:11:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f463c1569e0 msgr2=0x7f463c0afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f463c1569e0 0x7f463c0afdf0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f462402c4e0 tx=0x7f4624001a00 comp rx=0 tx=0).stop 2024-01-16T15:11:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 shutdown_connections 2024-01-16T15:11:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f461406c190 0x7f461406e650 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f463c151850 0x7f463c151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f463c1569e0 0x7f463c0afdf0 secure :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f462402c4e0 tx=0x7f4624001a00 comp rx=0 tx=0).stop 2024-01-16T15:11:27.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 --2- 172.21.15.141:0/2363421892 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f461c003ef0 0x7f463c1f0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:27.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.690+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 >> 172.21.15.141:0/2363421892 conn(0x7f463c06a800 msgr2=0x7f463c0cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:27.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.691+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 shutdown_connections 2024-01-16T15:11:27.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:27.691+0000 7f4643400640 1 -- 172.21.15.141:0/2363421892 wait complete. 2024-01-16T15:11:27.698 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:11:27.698 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:27.804 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:27.914 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:11:27.914 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:11:27.914 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:27.971 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:28.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.205+0000 7f35aa000640 1 Processor -- start 2024-01-16T15:11:28.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.205+0000 7f35aa000640 1 -- start start 2024-01-16T15:11:28.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.207+0000 7f35aa000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f35a40b3ae0 0x7f35a415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.207+0000 7f35aa000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a4070c40 con 0x7f35a40cf290 2024-01-16T15:11:28.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.207+0000 7f35aa000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a406c470 con 0x7f35a40ae5a0 2024-01-16T15:11:28.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.207+0000 7f35aa000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a4071450 con 0x7f35a40b3ae0 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.207+0000 7f35a3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f35a40cf290 0x7f35a40b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34488/0 (socket says 172.21.15.141:34488) 2024-01-16T15:11:28.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.207+0000 7f35a3e00640 1 -- 172.21.15.141:0/346722667 learned_addr learned my addr 172.21.15.141:0/346722667 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f359be00640 1 --2- 172.21.15.141:0/346722667 >> v2:172.21.15.188:3300/0 conn(0x7f35a40b3ae0 0x7f35a415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291170538 0 0) 0x7f35a4070c40 con 0x7f35a40cf290 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584003520 con 0x7f35a40cf290 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 482289614 0 0) 0x7f3584003520 con 0x7f35a40cf290 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 >> v2:172.21.15.188:3300/0 conn(0x7f35a40b3ae0 msgr2=0x7f35a415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 --2- 172.21.15.141:0/346722667 >> v2:172.21.15.188:3300/0 conn(0x7f35a40b3ae0 0x7f35a415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 >> v1:172.21.15.188:6789/0 conn(0x7f35a40ae5a0 legacy=0x7f35a406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a406c750 con 0x7f35a40cf290 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f359c0019f0 con 0x7f35a40cf290 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 >> v1:172.21.15.141:6789/0 conn(0x7f35a40cf290 legacy=0x7f35a40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f3584003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35840047c0 0x7f3584004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3584005180 0x7f358400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a4071450 con 0x7f35840039c0 2024-01-16T15:11:28.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358400c060 con 0x7f35840047c0 2024-01-16T15:11:28.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.208+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358400c220 con 0x7f3584005180 2024-01-16T15:11:28.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f35a3400640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f3584003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f359be00640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35840047c0 0x7f3584004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f35a3400640 1 -- 172.21.15.141:0/346722667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3584005180 msgr2=0x7f358400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f35a3400640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3584005180 0x7f358400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f35a3400640 1 -- 172.21.15.141:0/346722667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35840047c0 msgr2=0x7f3584004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f35a3400640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35840047c0 0x7f3584004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f35a3400640 1 -- 172.21.15.141:0/346722667 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584003520 con 0x7f35840039c0 2024-01-16T15:11:28.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.209+0000 7f35a3e00640 1 -- 172.21.15.141:0/346722667 reap_dead start 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.211+0000 7f35a3400640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f3584003de0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f3594001750 tx=0x7f35840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b14135af9ddfad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.211+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35940052d0 con 0x7f35840039c0 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.211+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359402e050 con 0x7f35840039c0 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.211+0000 7f35a2a00640 1 -- 172.21.15.141:0/346722667 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594003830 con 0x7f35840039c0 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.211+0000 7f35aa000640 1 -- 172.21.15.141:0/346722667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 msgr2=0x7f3584003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.211+0000 7f35aa000640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f3584003de0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f3594001750 tx=0x7f35840057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.212+0000 7f35aa000640 1 -- 172.21.15.141:0/346722667 shutdown_connections 2024-01-16T15:11:28.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.212+0000 7f35aa000640 1 --2- 172.21.15.141:0/346722667 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f3584003de0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.212+0000 7f35aa000640 1 -- 172.21.15.141:0/346722667 >> 172.21.15.141:0/346722667 conn(0x7f35a406a800 msgr2=0x7f35a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.212+0000 7f35aa000640 1 -- 172.21.15.141:0/346722667 shutdown_connections 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.212+0000 7f35aa000640 1 -- 172.21.15.141:0/346722667 wait complete. 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.213+0000 7f35aa000640 1 Processor -- start 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.214+0000 7f35aa000640 1 -- start start 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.214+0000 7f35aa000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f35a414fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.214+0000 7f35aa000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35a41f2d50 0x7f35a41f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.214+0000 7f35aa000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f35a41595d0 0x7f35a4159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.214+0000 7f35aa000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a415bb40 con 0x7f35840039c0 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.214+0000 7f35aa000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a4159f90 con 0x7f35a41f2d50 2024-01-16T15:11:28.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.214+0000 7f35aa000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a415b780 con 0x7f35a41595d0 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f35a414fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f35a414fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42388/0 (socket says 172.21.15.141:42388) 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 -- 172.21.15.141:0/1621888956 learned_addr learned my addr 172.21.15.141:0/1621888956 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3e00640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f35a41595d0 0x7f35a4159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f359be00640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35a41f2d50 0x7f35a41f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 -- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f35a41595d0 msgr2=0x7f35a4159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f35a41595d0 0x7f35a4159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 -- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35a41f2d50 msgr2=0x7f35a41f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35a41f2d50 0x7f35a41f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.215+0000 7f35a3400640 1 -- 172.21.15.141:0/1621888956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a41f3810 con 0x7f35840039c0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.216+0000 7f35a3400640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f35a414fb10 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f3594000c00 tx=0x7f3594001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.216+0000 7f359b400640 1 -- 172.21.15.141:0/1621888956 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35940076c0 con 0x7f35840039c0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.216+0000 7f359b400640 1 -- 172.21.15.141:0/1621888956 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594007a30 con 0x7f35840039c0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.216+0000 7f359b400640 1 -- 172.21.15.141:0/1621888956 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594006be0 con 0x7f35840039c0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.216+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a41f3af0 con 0x7f35840039c0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.216+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a40ce4f0 con 0x7f35840039c0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.217+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a41f3ce0 con 0x7f35840039c0 2024-01-16T15:11:28.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.218+0000 7f359b400640 1 -- 172.21.15.141:0/1621888956 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f3594006d40 con 0x7f35840039c0 2024-01-16T15:11:28.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.219+0000 7f359b400640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f357c06c1f0 0x7f357c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.219+0000 7f359b400640 1 -- 172.21.15.141:0/1621888956 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f35940aa380 con 0x7f35840039c0 2024-01-16T15:11:28.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.220+0000 7f359be00640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f357c06c1f0 0x7f357c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.221+0000 7f359be00640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f357c06c1f0 0x7f357c06e6b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f35a415a2b0 tx=0x7f358c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.224+0000 7f359b400640 1 -- 172.21.15.141:0/1621888956 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35940dfe90 con 0x7f35840039c0 2024-01-16T15:11:28.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.415+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35a40675d0 con 0x7f35840039c0 2024-01-16T15:11:28.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.418+0000 7f359b400640 1 -- 172.21.15.141:0/1621888956 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13037 (secure 0 0 0) 0x7f3594076c00 con 0x7f35840039c0 2024-01-16T15:11:28.395 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:28.395 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":94,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:11:27.660596+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6834/3399417259":"2024-01-17T15:11:24.632792+0000","172.21.15.141:6832/3399417259":"2024-01-17T15:11:24.632792+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:28.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f357c06c1f0 msgr2=0x7f357c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f357c06c1f0 0x7f357c06e6b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f35a415a2b0 tx=0x7f358c008040 comp rx=0 tx=0).stop 2024-01-16T15:11:28.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 msgr2=0x7f35a414fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f35a414fb10 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f3594000c00 tx=0x7f3594001120 comp rx=0 tx=0).stop 2024-01-16T15:11:28.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 shutdown_connections 2024-01-16T15:11:28.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f357c06c1f0 0x7f357c06e6b0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f35a41595d0 0x7f35a4159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35a41f2d50 0x7f35a41f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 --2- 172.21.15.141:0/1621888956 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35840039c0 0x7f35a414fb10 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 >> 172.21.15.141:0/1621888956 conn(0x7f35a406a800 msgr2=0x7f35a40b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 shutdown_connections 2024-01-16T15:11:28.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.420+0000 7f35aa000640 1 -- 172.21.15.141:0/1621888956 wait complete. 2024-01-16T15:11:28.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:28.418 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:28.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.651+0000 7f59f4c00640 1 Processor -- start 2024-01-16T15:11:28.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.651+0000 7f59f4c00640 1 -- start start 2024-01-16T15:11:28.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59f4c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59f0159c90 0x7f59f006e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59f4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f00b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59f4c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f00cda70 con 0x7f59f00cd500 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59f4c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f00cdd50 con 0x7f59f0159c90 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59f4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00ce030 con 0x7f59f015a070 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59ef400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59f0159c90 0x7f59f006e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59ef400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59f0159c90 0x7f59f006e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42394/0 (socket says 172.21.15.141:42394) 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59ef400640 1 -- 172.21.15.141:0/782497054 learned_addr learned my addr 172.21.15.141:0/782497054 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.653+0000 7f59e7e00640 1 --2- 172.21.15.141:0/782497054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f00b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.654+0000 7f59ef400640 1 -- 172.21.15.141:0/782497054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 msgr2=0x7f59f00b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.654+0000 7f59ef400640 1 --2- 172.21.15.141:0/782497054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f00b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.654+0000 7f59ef400640 1 -- 172.21.15.141:0/782497054 >> v1:172.21.15.188:6789/0 conn(0x7f59f00cd500 legacy=0x7f59f0071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.654+0000 7f59ef400640 1 -- 172.21.15.141:0/782497054 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f00ce310 con 0x7f59f0159c90 2024-01-16T15:11:28.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.656+0000 7f59ef400640 1 --2- 172.21.15.141:0/782497054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f0159c90 0x7f59f006e550 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009870 tx=0x7f59e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ce282dea28eb75 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.656+0000 7f59eea00640 1 -- 172.21.15.141:0/782497054 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0030840 con 0x7f59f0159c90 2024-01-16T15:11:28.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59eea00640 1 -- 172.21.15.141:0/782497054 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e00309a0 con 0x7f59f0159c90 2024-01-16T15:11:28.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59eea00640 1 -- 172.21.15.141:0/782497054 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0030d30 con 0x7f59f0159c90 2024-01-16T15:11:28.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 -- 172.21.15.141:0/782497054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f0159c90 msgr2=0x7f59f006e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 --2- 172.21.15.141:0/782497054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f0159c90 0x7f59f006e550 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009870 tx=0x7f59e002b980 comp rx=0 tx=0).stop 2024-01-16T15:11:28.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 -- 172.21.15.141:0/782497054 shutdown_connections 2024-01-16T15:11:28.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 --2- 172.21.15.141:0/782497054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f00b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 --2- 172.21.15.141:0/782497054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f0159c90 0x7f59f006e550 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 -- 172.21.15.141:0/782497054 >> 172.21.15.141:0/782497054 conn(0x7f59f006b4f0 msgr2=0x7f59f006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 -- 172.21.15.141:0/782497054 shutdown_connections 2024-01-16T15:11:28.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.657+0000 7f59f4c00640 1 -- 172.21.15.141:0/782497054 wait complete. 2024-01-16T15:11:28.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.659+0000 7f59f4c00640 1 Processor -- start 2024-01-16T15:11:28.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.659+0000 7f59f4c00640 1 -- start start 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.660+0000 7f59f4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f00cd500 0x7f59f00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.660+0000 7f59f4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59f0159c90 0x7f59f00b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.660+0000 7f59f4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f01f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.660+0000 7f59f4c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00b2af0 con 0x7f59f00cd500 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.660+0000 7f59f4c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f015b980 con 0x7f59f0159c90 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.660+0000 7f59f4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f015bc60 con 0x7f59f015a070 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.660+0000 7f59ef400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f00cd500 0x7f59f00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f00cd500 0x7f59f00b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42396/0 (socket says 172.21.15.141:42396) 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 -- 172.21.15.141:0/3825667234 learned_addr learned my addr 172.21.15.141:0/3825667234 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59e7e00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59f0159c90 0x7f59f00b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59efe00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f01f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 -- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 msgr2=0x7f59f01f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f01f2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 -- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59f0159c90 msgr2=0x7f59f00b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59f0159c90 0x7f59f00b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 -- 172.21.15.141:0/3825667234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e00094d0 con 0x7f59f00cd500 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59efe00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f01f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59ef400640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f00cd500 0x7f59f00b2480 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f59e0000c00 tx=0x7f59e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.661+0000 7f59e7400640 1 -- 172.21.15.141:0/3825667234 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0039c90 con 0x7f59f00cd500 2024-01-16T15:11:28.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.662+0000 7f59e7400640 1 -- 172.21.15.141:0/3825667234 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0039df0 con 0x7f59f00cd500 2024-01-16T15:11:28.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.662+0000 7f59e7400640 1 -- 172.21.15.141:0/3825667234 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0043640 con 0x7f59f00cd500 2024-01-16T15:11:28.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.662+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f015bf40 con 0x7f59f00cd500 2024-01-16T15:11:28.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.662+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f014fb70 con 0x7f59f00cd500 2024-01-16T15:11:28.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.663+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f00d07c0 con 0x7f59f00cd500 2024-01-16T15:11:28.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.664+0000 7f59e7400640 1 -- 172.21.15.141:0/3825667234 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f59e00437a0 con 0x7f59f00cd500 2024-01-16T15:11:28.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.664+0000 7f59e7400640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f59c806c1f0 0x7f59c806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.665+0000 7f59e7400640 1 -- 172.21.15.141:0/3825667234 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f59e003e070 con 0x7f59f00cd500 2024-01-16T15:11:28.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.665+0000 7f59e7e00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f59c806c1f0 0x7f59c806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.666+0000 7f59e7e00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f59c806c1f0 0x7f59c806e6b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f59f00af330 tx=0x7f59d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.669+0000 7f59e7400640 1 -- 172.21.15.141:0/3825667234 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59e0077fc0 con 0x7f59f00cd500 2024-01-16T15:11:28.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.927+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59f0067450 con 0x7f59f00cd500 2024-01-16T15:11:28.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.929+0000 7f59e7400640 1 -- 172.21.15.141:0/3825667234 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59e0034020 con 0x7f59f00cd500 2024-01-16T15:11:28.907 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:11:28.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f59c806c1f0 msgr2=0x7f59c806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f59c806c1f0 0x7f59c806e6b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f59f00af330 tx=0x7f59d8008040 comp rx=0 tx=0).stop 2024-01-16T15:11:28.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f00cd500 msgr2=0x7f59f00b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f00cd500 0x7f59f00b2480 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f59e0000c00 tx=0x7f59e0004730 comp rx=0 tx=0).stop 2024-01-16T15:11:28.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 shutdown_connections 2024-01-16T15:11:28.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f59c806c1f0 0x7f59c806e6b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f59f015a070 0x7f59f01f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59f0159c90 0x7f59f00b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.933+0000 7f59f4c00640 1 --2- 172.21.15.141:0/3825667234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59f00cd500 0x7f59f00b2480 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.934+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 >> 172.21.15.141:0/3825667234 conn(0x7f59f006b4f0 msgr2=0x7f59f0071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.934+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 shutdown_connections 2024-01-16T15:11:28.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:28.934+0000 7f59f4c00640 1 -- 172.21.15.141:0/3825667234 wait complete. 2024-01-16T15:11:28.942 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:11:29.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.145+0000 7fe55ae00640 1 Processor -- start 2024-01-16T15:11:29.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.145+0000 7fe55ae00640 1 -- start start 2024-01-16T15:11:29.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.146+0000 7fe55ae00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe554143ae0 0x7fe5540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.146+0000 7fe55ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5540cf7d0 0x7fe55415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.146+0000 7fe55ae00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5540cfc30 con 0x7fe55415e1e0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.146+0000 7fe55ae00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5540b1ee0 con 0x7fe554143ae0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.146+0000 7fe55ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b2780 con 0x7fe5540cf7d0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe558e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe55415e1e0 0x7fe55415a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34490/0 (socket says 172.21.15.141:34490) 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe558e00640 1 -- 172.21.15.141:0/109178019 learned_addr learned my addr 172.21.15.141:0/109178019 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe553e00640 1 --2- 172.21.15.141:0/109178019 >> v2:172.21.15.188:3300/0 conn(0x7fe554143ae0 0x7fe5540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe54b200640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5540cf7d0 0x7fe55415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987761349 0 0) 0x7fe5540cfc30 con 0x7fe55415e1e0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534003520 con 0x7fe55415e1e0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2432867462 0 0) 0x7fe534003520 con 0x7fe55415e1e0 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5540cf7d0 msgr2=0x7fe55415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.147+0000 7fe553400640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5540cf7d0 0x7fe55415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 >> v2:172.21.15.188:3300/0 conn(0x7fe554143ae0 msgr2=0x7fe5540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 --2- 172.21.15.141:0/109178019 >> v2:172.21.15.188:3300/0 conn(0x7fe554143ae0 0x7fe5540cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55415e990 con 0x7fe55415e1e0 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe54c0019f0 con 0x7fe55415e1e0 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 >> v1:172.21.15.141:6789/0 conn(0x7fe55415e1e0 legacy=0x7fe55415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5540cf7d0 0x7fe534003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe534003f40 0x7fe5340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe534004d00 0x7fe534005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b1ee0 con 0x7fe534003f40 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b2780 con 0x7fe534004d00 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53400b980 con 0x7fe5540cf7d0 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe54b200640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe534003f40 0x7fe5340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe54b200640 1 -- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5540cf7d0 msgr2=0x7fe534003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe54b200640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe5540cf7d0 0x7fe534003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe54b200640 1 -- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe534004d00 msgr2=0x7fe534005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:29.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe54b200640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe534004d00 0x7fe534005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.148+0000 7fe54b200640 1 -- 172.21.15.141:0/109178019 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534003520 con 0x7fe534003f40 2024-01-16T15:11:29.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.149+0000 7fe54b200640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe534003f40 0x7fe5340047c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009af0 tx=0x7fe5340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f83708a5eaf6e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:29.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c02c710 con 0x7fe534003f40 2024-01-16T15:11:29.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c02c870 con 0x7fe534003f40 2024-01-16T15:11:29.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe553400640 1 -- 172.21.15.141:0/109178019 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c02cc00 con 0x7fe534003f40 2024-01-16T15:11:29.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe55ae00640 1 -- 172.21.15.141:0/109178019 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe534003f40 msgr2=0x7fe5340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe55ae00640 1 --2- 172.21.15.141:0/109178019 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe534003f40 0x7fe5340047c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009af0 tx=0x7fe5340057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe558e00640 1 -- 172.21.15.141:0/109178019 reap_dead start 2024-01-16T15:11:29.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe55ae00640 1 -- 172.21.15.141:0/109178019 shutdown_connections 2024-01-16T15:11:29.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe55ae00640 1 -- 172.21.15.141:0/109178019 >> 172.21.15.141:0/109178019 conn(0x7fe55406a800 msgr2=0x7fe55406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe55ae00640 1 -- 172.21.15.141:0/109178019 shutdown_connections 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.150+0000 7fe55ae00640 1 -- 172.21.15.141:0/109178019 wait complete. 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.152+0000 7fe55ae00640 1 Processor -- start 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.152+0000 7fe55ae00640 1 -- start start 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe55ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe534003f40 0x7fe5541f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe55ae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55415cfa0 0x7fe5541f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe55ae00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe55406c470 0x7fe55406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe55ae00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540c46f0 con 0x7fe55406c470 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe55ae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55406ce60 con 0x7fe55415cfa0 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe55ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe554140130 con 0x7fe534003f40 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe558e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe55406c470 0x7fe55406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe558e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe55406c470 0x7fe55406c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42408/0 (socket says 172.21.15.141:42408) 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.153+0000 7fe558e00640 1 -- 172.21.15.141:0/1480643870 learned_addr learned my addr 172.21.15.141:0/1480643870 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe553e00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe534003f40 0x7fe5541f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe54b200640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55415cfa0 0x7fe5541f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe558e00640 1 -- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe534003f40 msgr2=0x7fe5541f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe558e00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe534003f40 0x7fe5541f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe558e00640 1 -- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55415cfa0 msgr2=0x7fe5541f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe558e00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55415cfa0 0x7fe5541f21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe558e00640 1 -- 172.21.15.141:0/1480643870 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5541f39f0 con 0x7fe55406c470 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe54b200640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55415cfa0 0x7fe5541f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe558e00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe55406c470 0x7fe55406c920 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fe5540cc3c0 tx=0x7fe54c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe550c00640 1 -- 172.21.15.141:0/1480643870 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c00a610 con 0x7fe55406c470 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe550c00640 1 -- 172.21.15.141:0/1480643870 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c00a770 con 0x7fe55406c470 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55415d510 con 0x7fe55406c470 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.154+0000 7fe550c00640 1 -- 172.21.15.141:0/1480643870 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c00ab40 con 0x7fe55406c470 2024-01-16T15:11:29.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.155+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5540c4130 con 0x7fe55406c470 2024-01-16T15:11:29.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.156+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55415a5a0 con 0x7fe55406c470 2024-01-16T15:11:29.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.157+0000 7fe550c00640 1 -- 172.21.15.141:0/1480643870 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fe54c00aca0 con 0x7fe55406c470 2024-01-16T15:11:29.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.157+0000 7fe550c00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe52c06c1f0 0x7fe52c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.157+0000 7fe550c00640 1 -- 172.21.15.141:0/1480643870 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe54c088370 con 0x7fe55406c470 2024-01-16T15:11:29.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.158+0000 7fe553e00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe52c06c1f0 0x7fe52c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.159+0000 7fe553e00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe52c06c1f0 0x7fe52c06e6b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe544004770 tx=0x7fe544008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:29.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.162+0000 7fe550c00640 1 -- 172.21.15.141:0/1480643870 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe54c054c30 con 0x7fe55406c470 2024-01-16T15:11:29.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.351+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe5540675d0 con 0x7fe55406c470 2024-01-16T15:11:29.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.351+0000 7fe550c00640 1 -- 172.21.15.141:0/1480643870 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v94) v1 ==== 80+0+124 (secure 0 0 0) 0x7fe54c058250 con 0x7fe55406c470 2024-01-16T15:11:29.329 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/3399417259 2024-01-17T15:11:24.632792+0000 2024-01-16T15:11:29.329 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6832/3399417259 2024-01-17T15:11:24.632792+0000 2024-01-16T15:11:29.329 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:11:29.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.353+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe52c06c1f0 msgr2=0x7fe52c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.353+0000 7fe55ae00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe52c06c1f0 0x7fe52c06e6b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe544004770 tx=0x7fe544008040 comp rx=0 tx=0).stop 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.353+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe55406c470 msgr2=0x7fe55406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.353+0000 7fe55ae00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe55406c470 0x7fe55406c920 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fe5540cc3c0 tx=0x7fe54c00ccd0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 shutdown_connections 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe52c06c1f0 0x7fe52c06e6b0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe55406c470 0x7fe55406c920 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55415cfa0 0x7fe5541f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 --2- 172.21.15.141:0/1480643870 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe534003f40 0x7fe5541f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 >> 172.21.15.141:0/1480643870 conn(0x7fe55406a800 msgr2=0x7fe55406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 shutdown_connections 2024-01-16T15:11:29.332 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.354+0000 7fe55ae00640 1 -- 172.21.15.141:0/1480643870 wait complete. 2024-01-16T15:11:29.348 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T15:11:29.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:29.349 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:29.440 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:11:29.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:29.441 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:11:29.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[69456]: fuse finished with error 0 and tester_r 0 2024-01-16T15:11:29.660 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:11:29.661 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:29.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:29.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.948+0000 7fc575200640 1 Processor -- start 2024-01-16T15:11:29.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.949+0000 7fc575200640 1 -- start start 2024-01-16T15:11:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.950+0000 7fc575200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 0x7fc570071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.950+0000 7fc575200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc570070e50 con 0x7fc5700ccc10 2024-01-16T15:11:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.950+0000 7fc575200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc570144020 con 0x7fc5700717d0 2024-01-16T15:11:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.950+0000 7fc575200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700723c0 con 0x7fc570143c40 2024-01-16T15:11:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.951+0000 7fc56fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc5700ccc10 0x7fc57006d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34496/0 (socket says 172.21.15.141:34496) 2024-01-16T15:11:29.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.951+0000 7fc56fe00640 1 -- 172.21.15.141:0/1706938203 learned_addr learned my addr 172.21.15.141:0/1706938203 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.951+0000 7fc56f400640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 0x7fc570071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.951+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985239057 0 0) 0x7fc570070e50 con 0x7fc5700ccc10 2024-01-16T15:11:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.951+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc550003520 con 0x7fc5700ccc10 2024-01-16T15:11:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.951+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 738521753 0 0) 0x7fc550003520 con 0x7fc5700ccc10 2024-01-16T15:11:29.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 msgr2=0x7fc570071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 0x7fc570071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 >> v1:172.21.15.188:6789/0 conn(0x7fc5700717d0 legacy=0x7fc5700cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc570071e00 con 0x7fc5700ccc10 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fc568001670 con 0x7fc5700ccc10 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 >> v1:172.21.15.141:6789/0 conn(0x7fc5700ccc10 legacy=0x7fc57006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5500039a0 0x7fc550003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5500047c0 0x7fc550004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 0x7fc55000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700723c0 con 0x7fc5500047c0 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500054e0 con 0x7fc5500039a0 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55000bec0 con 0x7fc570143c40 2024-01-16T15:11:29.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc566800640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5500047c0 0x7fc550004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56f400640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5500039a0 0x7fc550003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc56fe00640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 0x7fc55000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc566800640 1 -- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 msgr2=0x7fc55000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.952+0000 7fc566800640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc570143c40 0x7fc55000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.953+0000 7fc566800640 1 -- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5500039a0 msgr2=0x7fc550003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.953+0000 7fc566800640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5500039a0 0x7fc550003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.953+0000 7fc566800640 1 -- 172.21.15.141:0/1706938203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc550003520 con 0x7fc5500047c0 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc566800640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5500047c0 0x7fc550004c40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fc558001750 tx=0x7fc5500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2316ea04de9e676 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580052d0 con 0x7fc5500047c0 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55802e050 con 0x7fc5500047c0 2024-01-16T15:11:29.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc56ea00640 1 -- 172.21.15.141:0/1706938203 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580035f0 con 0x7fc5500047c0 2024-01-16T15:11:29.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc575200640 1 -- 172.21.15.141:0/1706938203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5500047c0 msgr2=0x7fc550004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc575200640 1 --2- 172.21.15.141:0/1706938203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5500047c0 0x7fc550004c40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fc558001750 tx=0x7fc5500057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc56fe00640 1 -- 172.21.15.141:0/1706938203 reap_dead start 2024-01-16T15:11:29.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc575200640 1 -- 172.21.15.141:0/1706938203 shutdown_connections 2024-01-16T15:11:29.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.954+0000 7fc575200640 1 -- 172.21.15.141:0/1706938203 >> 172.21.15.141:0/1706938203 conn(0x7fc57006a800 msgr2=0x7fc57006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.955+0000 7fc575200640 1 -- 172.21.15.141:0/1706938203 shutdown_connections 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.955+0000 7fc575200640 1 -- 172.21.15.141:0/1706938203 wait complete. 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.956+0000 7fc575200640 1 Processor -- start 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.956+0000 7fc575200640 1 -- start start 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.957+0000 7fc575200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5500047c0 0x7fc5701e0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.957+0000 7fc575200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5701e28e0 0x7fc5700adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.957+0000 7fc575200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc57015a330 0x7fc57015a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.957+0000 7fc575200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701e1b80 con 0x7fc57015a330 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.957+0000 7fc575200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701e15c0 con 0x7fc5701e28e0 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.957+0000 7fc575200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701e18a0 con 0x7fc5500047c0 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc57015a330 0x7fc57015a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc57015a330 0x7fc57015a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42420/0 (socket says 172.21.15.141:42420) 2024-01-16T15:11:29.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 -- 172.21.15.141:0/108871768 learned_addr learned my addr 172.21.15.141:0/108871768 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56f400640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5500047c0 0x7fc5701e0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc566800640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5701e28e0 0x7fc5700adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 -- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5500047c0 msgr2=0x7fc5701e0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5500047c0 0x7fc5701e0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 -- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5701e28e0 msgr2=0x7fc5700adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5701e28e0 0x7fc5700adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 -- 172.21.15.141:0/108871768 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700aad40 con 0x7fc57015a330 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc566800640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5701e28e0 0x7fc5700adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:29.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.958+0000 7fc56fe00640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc57015a330 0x7fc57015a790 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc568001200 tx=0x7fc56800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.959+0000 7fc567e00640 1 -- 172.21.15.141:0/108871768 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568007230 con 0x7fc57015a330 2024-01-16T15:11:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.959+0000 7fc567e00640 1 -- 172.21.15.141:0/108871768 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568007390 con 0x7fc57015a330 2024-01-16T15:11:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.959+0000 7fc567e00640 1 -- 172.21.15.141:0/108871768 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5680047e0 con 0x7fc57015a330 2024-01-16T15:11:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.959+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5700ab020 con 0x7fc57015a330 2024-01-16T15:11:29.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.959+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57015acd0 con 0x7fc57015a330 2024-01-16T15:11:29.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.960+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5700ab210 con 0x7fc57015a330 2024-01-16T15:11:29.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.961+0000 7fc567e00640 1 -- 172.21.15.141:0/108871768 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fc568004940 con 0x7fc57015a330 2024-01-16T15:11:29.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.962+0000 7fc567e00640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54806c220 0x7fc54806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:29.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.962+0000 7fc567e00640 1 -- 172.21.15.141:0/108871768 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc5680881a0 con 0x7fc57015a330 2024-01-16T15:11:29.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.962+0000 7fc56f400640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54806c220 0x7fc54806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:29.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.964+0000 7fc56f400640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54806c220 0x7fc54806e6e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc560004470 tx=0x7fc560003c70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:29.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:29.966+0000 7fc567e00640 1 -- 172.21.15.141:0/108871768 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc568054a60 con 0x7fc57015a330 2024-01-16T15:11:30.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.158+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc570067450 con 0x7fc57015a330 2024-01-16T15:11:30.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.160+0000 7fc567e00640 1 -- 172.21.15.141:0/108871768 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13037 (secure 0 0 0) 0x7fc568058080 con 0x7fc57015a330 2024-01-16T15:11:30.137 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:30.137 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":94,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:11:27.660596+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6834/3399417259":"2024-01-17T15:11:24.632792+0000","172.21.15.141:6832/3399417259":"2024-01-17T15:11:24.632792+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:30.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54806c220 msgr2=0x7fc54806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54806c220 0x7fc54806e6e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc560004470 tx=0x7fc560003c70 comp rx=0 tx=0).stop 2024-01-16T15:11:30.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc57015a330 msgr2=0x7fc57015a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc57015a330 0x7fc57015a790 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc568001200 tx=0x7fc56800dc00 comp rx=0 tx=0).stop 2024-01-16T15:11:30.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 shutdown_connections 2024-01-16T15:11:30.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54806c220 0x7fc54806e6e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc57015a330 0x7fc57015a790 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5701e28e0 0x7fc5700adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 --2- 172.21.15.141:0/108871768 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc5500047c0 0x7fc5701e0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.162+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 >> 172.21.15.141:0/108871768 conn(0x7fc57006a800 msgr2=0x7fc5701402b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:30.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.163+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 shutdown_connections 2024-01-16T15:11:30.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.163+0000 7fc575200640 1 -- 172.21.15.141:0/108871768 wait complete. 2024-01-16T15:11:30.157 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:30.158 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:30.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.385+0000 7fc2a2e00640 1 Processor -- start 2024-01-16T15:11:30.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.385+0000 7fc2a2e00640 1 -- start start 2024-01-16T15:11:30.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.387+0000 7fc2a2e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc29c15d4b0 0x7fc29c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.387+0000 7fc2a2e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0b2730 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.387+0000 7fc2a2e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0d1ec0 con 0x7fc29c0b41e0 2024-01-16T15:11:30.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.387+0000 7fc2a2e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c0b29d0 con 0x7fc29c15d4b0 2024-01-16T15:11:30.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.387+0000 7fc2a1a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc29c15d4b0 0x7fc29c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.387+0000 7fc2a1a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc29c15d4b0 0x7fc29c0d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42424/0 (socket says 172.21.15.141:42424) 2024-01-16T15:11:30.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.387+0000 7fc2a1a00640 1 -- 172.21.15.141:0/194381131 learned_addr learned my addr 172.21.15.141:0/194381131 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.388+0000 7fc2a1a00640 1 -- 172.21.15.141:0/194381131 >> v1:172.21.15.188:6789/0 conn(0x7fc29c0b41e0 legacy=0x7fc29c0ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.388+0000 7fc2a1a00640 1 -- 172.21.15.141:0/194381131 >> v1:172.21.15.141:6789/0 conn(0x7fc29c0cf9a0 legacy=0x7fc29c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.388+0000 7fc2a1a00640 1 -- 172.21.15.141:0/194381131 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c0b2cb0 con 0x7fc29c15d4b0 2024-01-16T15:11:30.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.389+0000 7fc2a1a00640 1 --2- 172.21.15.141:0/194381131 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c15d4b0 0x7fc29c0d1550 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009e50 tx=0x7fc28c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca741d2ff08b118 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.389+0000 7fc29be00640 1 -- 172.21.15.141:0/194381131 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c02cb10 con 0x7fc29c15d4b0 2024-01-16T15:11:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.389+0000 7fc29be00640 1 -- 172.21.15.141:0/194381131 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c02cc70 con 0x7fc29c15d4b0 2024-01-16T15:11:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.389+0000 7fc29be00640 1 -- 172.21.15.141:0/194381131 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0343f0 con 0x7fc29c15d4b0 2024-01-16T15:11:30.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.390+0000 7fc2a2e00640 1 -- 172.21.15.141:0/194381131 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c15d4b0 msgr2=0x7fc29c0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.390+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/194381131 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c15d4b0 0x7fc29c0d1550 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009e50 tx=0x7fc28c02bf70 comp rx=0 tx=0).stop 2024-01-16T15:11:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.390+0000 7fc2a2e00640 1 -- 172.21.15.141:0/194381131 shutdown_connections 2024-01-16T15:11:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.390+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/194381131 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c15d4b0 0x7fc29c0d1550 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.390+0000 7fc2a2e00640 1 -- 172.21.15.141:0/194381131 >> 172.21.15.141:0/194381131 conn(0x7fc29c0d3000 msgr2=0x7fc29c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.390+0000 7fc2a2e00640 1 -- 172.21.15.141:0/194381131 shutdown_connections 2024-01-16T15:11:30.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.390+0000 7fc2a2e00640 1 -- 172.21.15.141:0/194381131 wait complete. 2024-01-16T15:11:30.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.392+0000 7fc2a2e00640 1 Processor -- start 2024-01-16T15:11:30.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.392+0000 7fc2a2e00640 1 -- start start 2024-01-16T15:11:30.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a2e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc29c0b41e0 0x7fc29c1f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a2e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c0cf9a0 0x7fc29c09f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a2e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc29c15d4b0 0x7fc29c09f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a2e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c0cc0d0 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a2e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c149f80 con 0x7fc29c15d4b0 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a2e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c149650 con 0x7fc29c0b41e0 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a1000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c0cf9a0 0x7fc29c09f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a1000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c0cf9a0 0x7fc29c09f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42436/0 (socket says 172.21.15.141:42436) 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.393+0000 7fc2a1000640 1 -- 172.21.15.141:0/633552294 learned_addr learned my addr 172.21.15.141:0/633552294 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a1a00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc29c0b41e0 0x7fc29c1f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a2400640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc29c15d4b0 0x7fc29c09f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a1000640 1 -- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc29c0b41e0 msgr2=0x7fc29c1f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a1000640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc29c0b41e0 0x7fc29c1f2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a1000640 1 -- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc29c15d4b0 msgr2=0x7fc29c09f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a1000640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc29c15d4b0 0x7fc29c09f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a1000640 1 -- 172.21.15.141:0/633552294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c009b30 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a1000640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c0cf9a0 0x7fc29c09f4a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc28400c480 tx=0x7fc28400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc299600640 1 -- 172.21.15.141:0/633552294 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840052d0 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc299600640 1 -- 172.21.15.141:0/633552294 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284005430 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc299600640 1 -- 172.21.15.141:0/633552294 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284012870 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c15da10 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.394+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c149c10 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.395+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c14a110 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.397+0000 7fc299600640 1 -- 172.21.15.141:0/633552294 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fc2840129d0 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.397+0000 7fc299600640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc27406c1f0 0x7fc27406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.398+0000 7fc299600640 1 -- 172.21.15.141:0/633552294 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc284088630 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.398+0000 7fc2a1a00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc27406c1f0 0x7fc27406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.399+0000 7fc2a1a00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc27406c1f0 0x7fc27406e6b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc28c035000 tx=0x7fc28c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.402+0000 7fc299600640 1 -- 172.21.15.141:0/633552294 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2840bde90 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.594+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc29c067450 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.595+0000 7fc299600640 1 -- 172.21.15.141:0/633552294 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc284054b40 con 0x7fc29c0cf9a0 2024-01-16T15:11:30.572 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:11:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc27406c1f0 msgr2=0x7fc27406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc27406c1f0 0x7fc27406e6b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc28c035000 tx=0x7fc28c030000 comp rx=0 tx=0).stop 2024-01-16T15:11:30.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c0cf9a0 msgr2=0x7fc29c09f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c0cf9a0 0x7fc29c09f4a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc28400c480 tx=0x7fc28400c940 comp rx=0 tx=0).stop 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 shutdown_connections 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc27406c1f0 0x7fc27406e6b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc29c15d4b0 0x7fc29c09f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc29c0cf9a0 0x7fc29c09f4a0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 --2- 172.21.15.141:0/633552294 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc29c0b41e0 0x7fc29c1f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 >> 172.21.15.141:0/633552294 conn(0x7fc29c0d3000 msgr2=0x7fc29c0cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 shutdown_connections 2024-01-16T15:11:30.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.598+0000 7fc2a2e00640 1 -- 172.21.15.141:0/633552294 wait complete. 2024-01-16T15:11:30.596 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:11:30.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.828+0000 7fac6ca00640 1 Processor -- start 2024-01-16T15:11:30.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.829+0000 7fac6ca00640 1 -- start start 2024-01-16T15:11:30.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.830+0000 7fac6ca00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fac680cc500 0x7fac680cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.830+0000 7fac6ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 0x7fac6806d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.830+0000 7fac6ca00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac68070e50 con 0x7fac68143c40 2024-01-16T15:11:30.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.830+0000 7fac6ca00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac680723c0 con 0x7fac680cc500 2024-01-16T15:11:30.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.830+0000 7fac6ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac680cced0 con 0x7fac6806d160 2024-01-16T15:11:30.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.830+0000 7fac67400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fac68143c40 0x7fac68071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34514/0 (socket says 172.21.15.141:34514) 2024-01-16T15:11:30.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.830+0000 7fac67400640 1 -- 172.21.15.141:0/2177560436 learned_addr learned my addr 172.21.15.141:0/2177560436 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac67e00640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 0x7fac6806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514796966 0 0) 0x7fac68070e50 con 0x7fac68143c40 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48003520 con 0x7fac68143c40 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac5e800640 1 --2- 172.21.15.141:0/2177560436 >> v2:172.21.15.188:3300/0 conn(0x7fac680cc500 0x7fac680cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2088893687 0 0) 0x7fac48003520 con 0x7fac68143c40 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 msgr2=0x7fac6806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 0x7fac6806d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 >> v2:172.21.15.188:3300/0 conn(0x7fac680cc500 msgr2=0x7fac680cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 --2- 172.21.15.141:0/2177560436 >> v2:172.21.15.188:3300/0 conn(0x7fac680cc500 0x7fac680cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac68144020 con 0x7fac68143c40 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fac58001690 con 0x7fac68143c40 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 >> v1:172.21.15.141:6789/0 conn(0x7fac68143c40 legacy=0x7fac68071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.831+0000 7fac66a00640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac48003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac66a00640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 0x7fac480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac66a00640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac48004d00 0x7fac48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac680723c0 con 0x7fac48003970 2024-01-16T15:11:30.812 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac680cced0 con 0x7fac48004d00 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4800bb10 con 0x7fac6806d160 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac67400640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac48003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac5e800640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 0x7fac480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac67400640 1 -- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 msgr2=0x7fac480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac67400640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac6806d160 0x7fac480047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac67400640 1 -- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac48004d00 msgr2=0x7fac48005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac67400640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac48004d00 0x7fac48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.832+0000 7fac67400640 1 -- 172.21.15.141:0/2177560436 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48003520 con 0x7fac48003970 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.833+0000 7fac67400640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac48003d90 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c210 tx=0x7fac480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cadffcdd45fdcc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58005ad0 con 0x7fac48003970 2024-01-16T15:11:30.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58005c30 con 0x7fac48003970 2024-01-16T15:11:30.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac66a00640 1 -- 172.21.15.141:0/2177560436 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5802c8b0 con 0x7fac48003970 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac6ca00640 1 -- 172.21.15.141:0/2177560436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 msgr2=0x7fac48003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac6ca00640 1 --2- 172.21.15.141:0/2177560436 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac48003d90 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c210 tx=0x7fac480057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac67e00640 1 -- 172.21.15.141:0/2177560436 reap_dead start 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac6ca00640 1 -- 172.21.15.141:0/2177560436 shutdown_connections 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac6ca00640 1 -- 172.21.15.141:0/2177560436 >> 172.21.15.141:0/2177560436 conn(0x7fac6806a800 msgr2=0x7fac6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac6ca00640 1 -- 172.21.15.141:0/2177560436 shutdown_connections 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.834+0000 7fac6ca00640 1 -- 172.21.15.141:0/2177560436 wait complete. 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.836+0000 7fac6ca00640 1 Processor -- start 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.836+0000 7fac6ca00640 1 -- start start 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac6ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac680b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac6ca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac68142810 0x7fac68142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac6ca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac681e6ea0 0x7fac681e7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac6ca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac681f3320 con 0x7fac48003970 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac6ca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac681f3600 con 0x7fac681e6ea0 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac6ca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac681f38e0 con 0x7fac68142810 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac67400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac680b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac67400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac680b35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42452/0 (socket says 172.21.15.141:42452) 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.837+0000 7fac67400640 1 -- 172.21.15.141:0/664227645 learned_addr learned my addr 172.21.15.141:0/664227645 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac5e800640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac68142810 0x7fac68142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67e00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac681e6ea0 0x7fac681e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67400640 1 -- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac68142810 msgr2=0x7fac68142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67400640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac68142810 0x7fac68142c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67400640 1 -- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac681e6ea0 msgr2=0x7fac681e7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67400640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac681e6ea0 0x7fac681e7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67400640 1 -- 172.21.15.141:0/664227645 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac681f3bc0 con 0x7fac48003970 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67e00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac681e6ea0 0x7fac681e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac67400640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac680b35c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fac58001950 tx=0x7fac58001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.838+0000 7fac5fe00640 1 -- 172.21.15.141:0/664227645 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5802dd50 con 0x7fac48003970 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.839+0000 7fac5fe00640 1 -- 172.21.15.141:0/664227645 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58007070 con 0x7fac48003970 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.839+0000 7fac5fe00640 1 -- 172.21.15.141:0/664227645 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58007420 con 0x7fac48003970 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.839+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac681f3ea0 con 0x7fac48003970 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.839+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac681e7860 con 0x7fac48003970 2024-01-16T15:11:30.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.840+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac68067450 con 0x7fac48003970 2024-01-16T15:11:30.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.841+0000 7fac5fe00640 1 -- 172.21.15.141:0/664227645 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fac58032070 con 0x7fac48003970 2024-01-16T15:11:30.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.841+0000 7fac5fe00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac4006c1f0 0x7fac4006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.842+0000 7fac5fe00640 1 -- 172.21.15.141:0/664227645 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fac580a99a0 con 0x7fac48003970 2024-01-16T15:11:30.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.842+0000 7fac5e800640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac4006c1f0 0x7fac4006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.843+0000 7fac5e800640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac4006c1f0 0x7fac4006e6b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fac680cccc0 tx=0x7fac50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:30.846+0000 7fac5fe00640 1 -- 172.21.15.141:0/664227645 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac58075fd0 con 0x7fac48003970 2024-01-16T15:11:31.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.048+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fac680ceb30 con 0x7fac48003970 2024-01-16T15:11:31.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.049+0000 7fac5fe00640 1 -- 172.21.15.141:0/664227645 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v94) v1 ==== 80+0+124 (secure 0 0 0) 0x7fac58045080 con 0x7fac48003970 2024-01-16T15:11:31.026 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6834/3399417259 2024-01-17T15:11:24.632792+0000 2024-01-16T15:11:31.026 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141:6832/3399417259 2024-01-17T15:11:24.632792+0000 2024-01-16T15:11:31.027 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:11:31.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.051+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac4006c1f0 msgr2=0x7fac4006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:31.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.051+0000 7fac6ca00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac4006c1f0 0x7fac4006e6b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fac680cccc0 tx=0x7fac50008040 comp rx=0 tx=0).stop 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.051+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 msgr2=0x7fac680b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.051+0000 7fac6ca00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac680b35c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fac58001950 tx=0x7fac58001a20 comp rx=0 tx=0).stop 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 shutdown_connections 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fac4006c1f0 0x7fac4006e6b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac681e6ea0 0x7fac681e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fac68142810 0x7fac68142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 --2- 172.21.15.141:0/664227645 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac48003970 0x7fac680b35c0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 >> 172.21.15.141:0/664227645 conn(0x7fac6806a800 msgr2=0x7fac68071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 shutdown_connections 2024-01-16T15:11:31.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.052+0000 7fac6ca00640 1 -- 172.21.15.141:0/664227645 wait complete. 2024-01-16T15:11:31.047 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T15:11:31.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.047 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:31.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[45914]: fuse finished with error 0 and tester_r 0 2024-01-16T15:11:31.146 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:11:31.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.146 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:11:31.330 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:11:31.330 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:31.387 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:11:31.388 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:11:31.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.388 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:11:31.442 INFO:teuthology.orchestra.run.smithi141.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:11:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:11:31.444 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:11:31.444 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:31.501 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:11:31.501 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:11:31.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:31.501 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:11:31.556 INFO:teuthology.orchestra.run.smithi188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:11:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:11:31.558 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:31.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.796+0000 7f9147400640 1 Processor -- start 2024-01-16T15:11:31.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.796+0000 7f9147400640 1 -- start start 2024-01-16T15:11:31.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9147400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f914015d380 0x7f9140070410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:31.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9147400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91400b38c0 0x7f9140070950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:31.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9147400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914006ded0 con 0x7f91400b1f00 2024-01-16T15:11:31.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9147400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91400b2e20 con 0x7f914015d380 2024-01-16T15:11:31.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9147400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914006c470 con 0x7f91400b38c0 2024-01-16T15:11:31.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9144a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f914015d380 0x7f9140070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:31.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9144a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f914015d380 0x7f9140070410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42464/0 (socket says 172.21.15.141:42464) 2024-01-16T15:11:31.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9144a00640 1 -- 172.21.15.141:0/1058933088 learned_addr learned my addr 172.21.15.141:0/1058933088 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:31.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9144a00640 1 -- 172.21.15.141:0/1058933088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91400b38c0 msgr2=0x7f9140070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:31.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.798+0000 7f9144a00640 1 --2- 172.21.15.141:0/1058933088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91400b38c0 0x7f9140070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.799+0000 7f9144a00640 1 -- 172.21.15.141:0/1058933088 >> v1:172.21.15.188:6789/0 conn(0x7f91400b1f00 legacy=0x7f9140160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:31.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.799+0000 7f9144a00640 1 -- 172.21.15.141:0/1058933088 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91400b4240 con 0x7f914015d380 2024-01-16T15:11:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.800+0000 7f9144a00640 1 --2- 172.21.15.141:0/1058933088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f914015d380 0x7f9140070410 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9130009870 tx=0x7f913002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d968639dc32a152b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.800+0000 7f913f400640 1 -- 172.21.15.141:0/1058933088 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130031640 con 0x7f914015d380 2024-01-16T15:11:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.800+0000 7f913f400640 1 -- 172.21.15.141:0/1058933088 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91300317a0 con 0x7f914015d380 2024-01-16T15:11:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.800+0000 7f913f400640 1 -- 172.21.15.141:0/1058933088 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130031b30 con 0x7f914015d380 2024-01-16T15:11:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 -- 172.21.15.141:0/1058933088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f914015d380 msgr2=0x7f9140070410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 --2- 172.21.15.141:0/1058933088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f914015d380 0x7f9140070410 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9130009870 tx=0x7f913002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 -- 172.21.15.141:0/1058933088 shutdown_connections 2024-01-16T15:11:31.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 --2- 172.21.15.141:0/1058933088 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91400b38c0 0x7f9140070950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 --2- 172.21.15.141:0/1058933088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f914015d380 0x7f9140070410 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 -- 172.21.15.141:0/1058933088 >> 172.21.15.141:0/1058933088 conn(0x7f91400b0c00 msgr2=0x7f91400b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:31.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 -- 172.21.15.141:0/1058933088 shutdown_connections 2024-01-16T15:11:31.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.801+0000 7f9147400640 1 -- 172.21.15.141:0/1058933088 wait complete. 2024-01-16T15:11:31.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.802+0000 7f9147400640 1 Processor -- start 2024-01-16T15:11:31.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.803+0000 7f9147400640 1 -- start start 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.803+0000 7f9147400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91400b1f00 0x7f914015e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.803+0000 7f9147400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91400b38c0 0x7f9140155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f9147400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f914015d380 0x7f9140155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f9147400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914015ebf0 con 0x7f91400b38c0 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f9147400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914015eed0 con 0x7f91400b1f00 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f9147400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914015f1b0 con 0x7f914015d380 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91400b38c0 0x7f9140155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91400b38c0 0x7f9140155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42470/0 (socket says 172.21.15.141:42470) 2024-01-16T15:11:31.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 -- 172.21.15.141:0/2791290162 learned_addr learned my addr 172.21.15.141:0/2791290162 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 -- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f914015d380 msgr2=0x7f9140155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f914015d380 0x7f9140155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 -- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91400b1f00 msgr2=0x7f914015e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91400b1f00 0x7f914015e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.804+0000 7f913fe00640 1 -- 172.21.15.141:0/2791290162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91300094d0 con 0x7f91400b38c0 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.805+0000 7f913fe00640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91400b38c0 0x7f9140155810 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f912800b410 tx=0x7f912800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.805+0000 7f913cc00640 1 -- 172.21.15.141:0/2791290162 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128011830 con 0x7f91400b38c0 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.805+0000 7f913cc00640 1 -- 172.21.15.141:0/2791290162 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91280119b0 con 0x7f91400b38c0 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.805+0000 7f913cc00640 1 -- 172.21.15.141:0/2791290162 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128011d60 con 0x7f91400b38c0 2024-01-16T15:11:31.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.805+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914015f490 con 0x7f91400b38c0 2024-01-16T15:11:31.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.805+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914009bd40 con 0x7f91400b38c0 2024-01-16T15:11:31.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.806+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914015f9b0 con 0x7f91400b38c0 2024-01-16T15:11:31.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.807+0000 7f913cc00640 1 -- 172.21.15.141:0/2791290162 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f912801a400 con 0x7f91400b38c0 2024-01-16T15:11:31.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.808+0000 7f913cc00640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f911806c1f0 0x7f911806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:31.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.808+0000 7f913cc00640 1 -- 172.21.15.141:0/2791290162 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9128088800 con 0x7f91400b38c0 2024-01-16T15:11:31.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.809+0000 7f9144a00640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f911806c1f0 0x7f911806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:31.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.810+0000 7f9144a00640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f911806c1f0 0x7f911806e6b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9130036000 tx=0x7f9130030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:31.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:31.812+0000 7f913cc00640 1 -- 172.21.15.141:0/2791290162 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f914015f9b0 con 0x7f91400b38c0 2024-01-16T15:11:32.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.038+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f914015f9b0 con 0x7f91400b38c0 2024-01-16T15:11:32.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.038+0000 7f913cc00640 1 -- 172.21.15.141:0/2791290162 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f9128015e90 con 0x7f91400b38c0 2024-01-16T15:11:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:32.016 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":57,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/1211961850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1211961850},{"type":"v1","addr":"172.21.15.141:6835","nonce":1211961850}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3677420872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3677420872},{"type":"v1","addr":"172.21.15.188:6835","nonce":3677420872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/4249270856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":4249270856},{"type":"v1","addr":"172.21.15.188:6837","nonce":4249270856}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/804049895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":804049895},{"type":"v1","addr":"172.21.15.141:6834","nonce":804049895}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[]} 2024-01-16T15:11:32.017 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 57 2024-01-16T15:11:32.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f911806c1f0 msgr2=0x7f911806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f911806c1f0 0x7f911806e6b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9130036000 tx=0x7f9130030000 comp rx=0 tx=0).stop 2024-01-16T15:11:32.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91400b38c0 msgr2=0x7f9140155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91400b38c0 0x7f9140155810 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f912800b410 tx=0x7f912800b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 shutdown_connections 2024-01-16T15:11:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f911806c1f0 0x7f911806e6b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f914015d380 0x7f9140155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91400b38c0 0x7f9140155810 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 --2- 172.21.15.141:0/2791290162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91400b1f00 0x7f914015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 >> 172.21.15.141:0/2791290162 conn(0x7f91400b0c00 msgr2=0x7f91400ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 shutdown_connections 2024-01-16T15:11:32.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.041+0000 7f9147400640 1 -- 172.21.15.141:0/2791290162 wait complete. 2024-01-16T15:11:32.040 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:11:32.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.276+0000 7f3e34400640 1 Processor -- start 2024-01-16T15:11:32.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.277+0000 7f3e34400640 1 -- start start 2024-01-16T15:11:32.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.278+0000 7f3e34400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e2c13e360 0x7f3e2c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.278+0000 7f3e34400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c15c5f0 con 0x7f3e2c15f9c0 2024-01-16T15:11:32.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.278+0000 7f3e34400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c06c470 con 0x7f3e2c13df80 2024-01-16T15:11:32.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.278+0000 7f3e34400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2c06c750 con 0x7f3e2c13e360 2024-01-16T15:11:32.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.278+0000 7f3e31000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e2c13e360 0x7f3e2c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.279+0000 7f3e32400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3e2c15f9c0 0x7f3e2c0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34522/0 (socket says 172.21.15.141:34522) 2024-01-16T15:11:32.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.279+0000 7f3e32400640 1 -- 172.21.15.141:0/1809810565 learned_addr learned my addr 172.21.15.141:0/1809810565 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:32.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.279+0000 7f3e31000640 1 -- 172.21.15.141:0/1809810565 >> v1:172.21.15.188:6789/0 conn(0x7f3e2c13df80 legacy=0x7f3e2c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:32.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.279+0000 7f3e31000640 1 -- 172.21.15.141:0/1809810565 >> v1:172.21.15.141:6789/0 conn(0x7f3e2c15f9c0 legacy=0x7f3e2c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.279+0000 7f3e31000640 1 -- 172.21.15.141:0/1809810565 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c06ca30 con 0x7f3e2c13e360 2024-01-16T15:11:32.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.279+0000 7f3e2be00640 1 -- 172.21.15.141:0/1809810565 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102979421 0 0) 0x7f3e2c15c5f0 con 0x7f3e2c15f9c0 2024-01-16T15:11:32.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.280+0000 7f3e31000640 1 --2- 172.21.15.141:0/1809810565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c0cf290 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3e140097a0 tx=0x7f3e1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baafc4cb6885d202 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:32.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.280+0000 7f3e2be00640 1 -- 172.21.15.141:0/1809810565 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14031830 con 0x7f3e2c13e360 2024-01-16T15:11:32.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.280+0000 7f3e2be00640 1 -- 172.21.15.141:0/1809810565 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e14031990 con 0x7f3e2c13e360 2024-01-16T15:11:32.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.280+0000 7f3e2be00640 1 -- 172.21.15.141:0/1809810565 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14031d20 con 0x7f3e2c13e360 2024-01-16T15:11:32.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.281+0000 7f3e34400640 1 -- 172.21.15.141:0/1809810565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 msgr2=0x7f3e2c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.281+0000 7f3e34400640 1 --2- 172.21.15.141:0/1809810565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c0cf290 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3e140097a0 tx=0x7f3e1402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.281+0000 7f3e34400640 1 -- 172.21.15.141:0/1809810565 shutdown_connections 2024-01-16T15:11:32.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.281+0000 7f3e34400640 1 --2- 172.21.15.141:0/1809810565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c0cf290 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.281+0000 7f3e34400640 1 -- 172.21.15.141:0/1809810565 >> 172.21.15.141:0/1809810565 conn(0x7f3e2c06a800 msgr2=0x7f3e2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:32.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.282+0000 7f3e34400640 1 -- 172.21.15.141:0/1809810565 shutdown_connections 2024-01-16T15:11:32.259 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.282+0000 7f3e34400640 1 -- 172.21.15.141:0/1809810565 wait complete. 2024-01-16T15:11:32.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.283+0000 7f3e34400640 1 Processor -- start 2024-01-16T15:11:32.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.283+0000 7f3e34400640 1 -- start start 2024-01-16T15:11:32.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.284+0000 7f3e34400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e2c13df80 0x7f3e2c154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.284+0000 7f3e34400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c1f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.284+0000 7f3e34400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e2c15f9c0 0x7f3e2c15d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.284+0000 7f3e34400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c1547a0 con 0x7f3e2c13e360 2024-01-16T15:11:32.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.284+0000 7f3e34400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c1f49f0 con 0x7f3e2c13df80 2024-01-16T15:11:32.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.284+0000 7f3e34400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c1f4cd0 con 0x7f3e2c15f9c0 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c1f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c1f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42482/0 (socket says 172.21.15.141:42482) 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 -- 172.21.15.141:0/3843057013 learned_addr learned my addr 172.21.15.141:0/3843057013 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 -- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e2c15f9c0 msgr2=0x7f3e2c15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e2c15f9c0 0x7f3e2c15d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 -- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e2c13df80 msgr2=0x7f3e2c154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e2c13df80 0x7f3e2c154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.285+0000 7f3e31000640 1 -- 172.21.15.141:0/3843057013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14009400 con 0x7f3e2c13e360 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.286+0000 7f3e31000640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c1f5a90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3e140098c0 tx=0x7f3e140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.286+0000 7f3e29600640 1 -- 172.21.15.141:0/3843057013 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1403ba70 con 0x7f3e2c13e360 2024-01-16T15:11:32.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.286+0000 7f3e29600640 1 -- 172.21.15.141:0/3843057013 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1403bbf0 con 0x7f3e2c13e360 2024-01-16T15:11:32.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.286+0000 7f3e29600640 1 -- 172.21.15.141:0/3843057013 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e140453f0 con 0x7f3e2c13e360 2024-01-16T15:11:32.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.286+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c1f4fb0 con 0x7f3e2c13e360 2024-01-16T15:11:32.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.286+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c1f5570 con 0x7f3e2c13e360 2024-01-16T15:11:32.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.287+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c1f31a0 con 0x7f3e2c13e360 2024-01-16T15:11:32.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.288+0000 7f3e29600640 1 -- 172.21.15.141:0/3843057013 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f3e1404e450 con 0x7f3e2c13e360 2024-01-16T15:11:32.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.289+0000 7f3e29600640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e0406c1f0 0x7f3e0406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.289+0000 7f3e29600640 1 -- 172.21.15.141:0/3843057013 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3e140aa960 con 0x7f3e2c13e360 2024-01-16T15:11:32.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.290+0000 7f3e31a00640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e0406c1f0 0x7f3e0406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.291+0000 7f3e31a00640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e0406c1f0 0x7f3e0406e6b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00ab90 tx=0x7f3e1c009230 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:32.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.294+0000 7f3e29600640 1 -- 172.21.15.141:0/3843057013 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e14077220 con 0x7f3e2c13e360 2024-01-16T15:11:32.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.483+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f3e2c0675d0 con 0x7f3e2c13e360 2024-01-16T15:11:32.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.488+0000 7f3e29600640 1 -- 172.21.15.141:0/3843057013 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v33) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3e1407a840 con 0x7f3e2c13e360 2024-01-16T15:11:32.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.491+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e0406c1f0 msgr2=0x7f3e0406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.491+0000 7f3e34400640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e0406c1f0 0x7f3e0406e6b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00ab90 tx=0x7f3e1c009230 comp rx=0 tx=0).stop 2024-01-16T15:11:32.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.491+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 msgr2=0x7f3e2c1f5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.491+0000 7f3e34400640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c1f5a90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3e140098c0 tx=0x7f3e140046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 shutdown_connections 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e0406c1f0 0x7f3e0406e6b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e2c15f9c0 0x7f3e2c15d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e2c13e360 0x7f3e2c1f5a90 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 --2- 172.21.15.141:0/3843057013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e2c13df80 0x7f3e2c154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 >> 172.21.15.141:0/3843057013 conn(0x7f3e2c06a800 msgr2=0x7f3e2c0b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 shutdown_connections 2024-01-16T15:11:32.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.492+0000 7f3e34400640 1 -- 172.21.15.141:0/3843057013 wait complete. 2024-01-16T15:11:32.489 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:11:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.721+0000 7f2f87e00640 1 Processor -- start 2024-01-16T15:11:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.721+0000 7f2f87e00640 1 -- start start 2024-01-16T15:11:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.723+0000 7f2f87e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 0x7f2f88070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.723+0000 7f2f87e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2f8815bf10 0x7f2f88143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.723+0000 7f2f87e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8815f4e0 con 0x7f2f880adf20 2024-01-16T15:11:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.723+0000 7f2f87e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8815f970 con 0x7f2f8815bf10 2024-01-16T15:11:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.723+0000 7f2f87e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8815c3c0 con 0x7f2f880ccb50 2024-01-16T15:11:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.723+0000 7f2f86a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2f880adf20 0x7f2f880cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:34530/0 (socket says 172.21.15.141:34530) 2024-01-16T15:11:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.723+0000 7f2f86a00640 1 -- 172.21.15.141:0/2489968259 learned_addr learned my addr 172.21.15.141:0/2489968259 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:32.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489303116 0 0) 0x7f2f8815f4e0 con 0x7f2f880adf20 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f87400640 1 --2- 172.21.15.141:0/2489968259 >> v2:172.21.15.188:3300/0 conn(0x7f2f8815bf10 0x7f2f88143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f68003520 con 0x7f2f880adf20 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f7de00640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 0x7f2f88070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168314653 0 0) 0x7f2f68003520 con 0x7f2f880adf20 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 msgr2=0x7f2f88070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 0x7f2f88070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 >> v2:172.21.15.188:3300/0 conn(0x7f2f8815bf10 msgr2=0x7f2f88143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 --2- 172.21.15.141:0/2489968259 >> v2:172.21.15.188:3300/0 conn(0x7f2f8815bf10 0x7f2f88143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f88071130 con 0x7f2f880adf20 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2f78001670 con 0x7f2f880adf20 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 >> v1:172.21.15.141:6789/0 conn(0x7f2f880adf20 legacy=0x7f2f880cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 0x7f2f68003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f68003fa0 0x7f2f680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f68004d00 0x7f2f68005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8815f970 con 0x7f2f68004d00 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8815c3c0 con 0x7f2f68003fa0 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.724+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6800bb40 con 0x7f2f880ccb50 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f87400640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f68004d00 0x7f2f68005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f7de00640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f68003fa0 0x7f2f680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f86a00640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 0x7f2f68003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f87400640 1 -- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 msgr2=0x7f2f68003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f87400640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 0x7f2f68003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f87400640 1 -- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f68003fa0 msgr2=0x7f2f680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f87400640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f68003fa0 0x7f2f680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f87400640 1 -- 172.21.15.141:0/2489968259 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f68003520 con 0x7f2f68004d00 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.725+0000 7f2f86a00640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f880ccb50 0x7f2f68003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.726+0000 7f2f87400640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f68004d00 0x7f2f68005180 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f2f80007360 tx=0x7f2f680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea70e6c58af25bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:32.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.726+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8002ccf0 con 0x7f2f68004d00 2024-01-16T15:11:32.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.726+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8002ce50 con 0x7f2f68004d00 2024-01-16T15:11:32.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.726+0000 7f2f86000640 1 -- 172.21.15.141:0/2489968259 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80034660 con 0x7f2f68004d00 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.727+0000 7f2f87e00640 1 -- 172.21.15.141:0/2489968259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f68004d00 msgr2=0x7f2f68005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.727+0000 7f2f87e00640 1 --2- 172.21.15.141:0/2489968259 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f68004d00 0x7f2f68005180 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f2f80007360 tx=0x7f2f680057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.727+0000 7f2f87400640 1 -- 172.21.15.141:0/2489968259 reap_dead start 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.727+0000 7f2f87e00640 1 -- 172.21.15.141:0/2489968259 shutdown_connections 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.727+0000 7f2f87e00640 1 -- 172.21.15.141:0/2489968259 >> 172.21.15.141:0/2489968259 conn(0x7f2f8806a7d0 msgr2=0x7f2f8806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.727+0000 7f2f87e00640 1 -- 172.21.15.141:0/2489968259 shutdown_connections 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.727+0000 7f2f87e00640 1 -- 172.21.15.141:0/2489968259 wait complete. 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.729+0000 7f2f87e00640 1 Processor -- start 2024-01-16T15:11:32.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.729+0000 7f2f87e00640 1 -- start start 2024-01-16T15:11:32.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f87e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68004d00 0x7f2f880b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f87e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f8813c760 0x7f2f881525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f87e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f880b36a0 0x7f2f880b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f87e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f880b3210 con 0x7f2f8813c760 2024-01-16T15:11:32.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f87e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f88152b20 con 0x7f2f880b36a0 2024-01-16T15:11:32.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f87e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8815aad0 con 0x7f2f68004d00 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f7de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f8813c760 0x7f2f881525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f7de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f8813c760 0x7f2f881525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42508/0 (socket says 172.21.15.141:42508) 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.730+0000 7f2f7de00640 1 -- 172.21.15.141:0/2331931661 learned_addr learned my addr 172.21.15.141:0/2331931661 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f86a00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68004d00 0x7f2f880b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f87400640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f880b36a0 0x7f2f880b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7de00640 1 -- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68004d00 msgr2=0x7f2f880b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7de00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68004d00 0x7f2f880b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7de00640 1 -- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f880b36a0 msgr2=0x7f2f880b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7de00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f880b36a0 0x7f2f880b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7de00640 1 -- 172.21.15.141:0/2331931661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f880b2600 con 0x7f2f8813c760 2024-01-16T15:11:32.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7de00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f8813c760 0x7f2f881525e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000f550 tx=0x7f2f7000fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:32.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7f400640 1 -- 172.21.15.141:0/2331931661 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70014870 con 0x7f2f8813c760 2024-01-16T15:11:32.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7f400640 1 -- 172.21.15.141:0/2331931661 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f700149d0 con 0x7f2f8813c760 2024-01-16T15:11:32.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f880b28e0 con 0x7f2f8813c760 2024-01-16T15:11:32.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.731+0000 7f2f7f400640 1 -- 172.21.15.141:0/2331931661 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70014d80 con 0x7f2f8813c760 2024-01-16T15:11:32.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.732+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f880b2ea0 con 0x7f2f8813c760 2024-01-16T15:11:32.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.733+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f880b33c0 con 0x7f2f8813c760 2024-01-16T15:11:32.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.734+0000 7f2f7f400640 1 -- 172.21.15.141:0/2331931661 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f2f70015410 con 0x7f2f8813c760 2024-01-16T15:11:32.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.735+0000 7f2f7f400640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f6006c220 0x7f2f6006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:32.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.735+0000 7f2f7f400640 1 -- 172.21.15.141:0/2331931661 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2f70088fe0 con 0x7f2f8813c760 2024-01-16T15:11:32.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.735+0000 7f2f86a00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f6006c220 0x7f2f6006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:32.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.736+0000 7f2f86a00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f6006c220 0x7f2f6006e6e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2f78005cd0 tx=0x7f2f78009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:32.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.739+0000 7f2f7f400640 1 -- 172.21.15.141:0/2331931661 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f880b33c0 con 0x7f2f8813c760 2024-01-16T15:11:32.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.926+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f2f88067450 con 0x7f2f8813c760 2024-01-16T15:11:32.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.932+0000 7f2f7f400640 1 -- 172.21.15.141:0/2331931661 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v34) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2f70055900 con 0x7f2f8813c760 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f6006c220 msgr2=0x7f2f6006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f6006c220 0x7f2f6006e6e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2f78005cd0 tx=0x7f2f78009040 comp rx=0 tx=0).stop 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f8813c760 msgr2=0x7f2f881525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f8813c760 0x7f2f881525e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000f550 tx=0x7f2f7000fa10 comp rx=0 tx=0).stop 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 shutdown_connections 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f6006c220 0x7f2f6006e6e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f880b36a0 0x7f2f880b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f8813c760 0x7f2f881525e0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 --2- 172.21.15.141:0/2331931661 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68004d00 0x7f2f880b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 >> 172.21.15.141:0/2331931661 conn(0x7f2f8806a7d0 msgr2=0x7f2f8815ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:32.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 shutdown_connections 2024-01-16T15:11:32.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:32.935+0000 7f2f87e00640 1 -- 172.21.15.141:0/2331931661 wait complete. 2024-01-16T15:11:32.929 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:11:33.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.159+0000 7f150a600640 1 Processor -- start 2024-01-16T15:11:33.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.160+0000 7f150a600640 1 -- start start 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.161+0000 7f150a600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f150413df80 0x7f150415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.161+0000 7f150a600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f150413e360 0x7f15040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.161+0000 7f150a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150415f9c0 0x7f15040cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.161+0000 7f150a600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150406c470 con 0x7f150413df80 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.161+0000 7f150a600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150406c750 con 0x7f150413e360 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.161+0000 7f150a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150406ca30 con 0x7f150415f9c0 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f150413df80 0x7f150415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f150413df80 0x7f150415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57100/0 (socket says 172.21.15.141:57100) 2024-01-16T15:11:33.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 -- 172.21.15.141:0/102988104 learned_addr learned my addr 172.21.15.141:0/102988104 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:33.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 -- 172.21.15.141:0/102988104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150415f9c0 msgr2=0x7f15040cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 --2- 172.21.15.141:0/102988104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150415f9c0 0x7f15040cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 -- 172.21.15.141:0/102988104 >> v2:172.21.15.188:3300/0 conn(0x7f150413e360 msgr2=0x7f15040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 --2- 172.21.15.141:0/102988104 >> v2:172.21.15.188:3300/0 conn(0x7f150413e360 0x7f15040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.162+0000 7f1503400640 1 -- 172.21.15.141:0/102988104 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150406cd10 con 0x7f150413df80 2024-01-16T15:11:33.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.163+0000 7f1503400640 1 --2- 172.21.15.141:0/102988104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150415bee0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f150406ded0 tx=0x7f14f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac4d7d8b75b2f4f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.163+0000 7f1502a00640 1 -- 172.21.15.141:0/102988104 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4031870 con 0x7f150413df80 2024-01-16T15:11:33.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f1502a00640 1 -- 172.21.15.141:0/102988104 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f40319d0 con 0x7f150413df80 2024-01-16T15:11:33.141 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f1502a00640 1 -- 172.21.15.141:0/102988104 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4031d60 con 0x7f150413df80 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 -- 172.21.15.141:0/102988104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 msgr2=0x7f150415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 --2- 172.21.15.141:0/102988104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150415bee0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f150406ded0 tx=0x7f14f402b910 comp rx=0 tx=0).stop 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 -- 172.21.15.141:0/102988104 shutdown_connections 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 --2- 172.21.15.141:0/102988104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150415f9c0 0x7f15040cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 --2- 172.21.15.141:0/102988104 >> v2:172.21.15.188:3300/0 conn(0x7f150413e360 0x7f15040cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 --2- 172.21.15.141:0/102988104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150415bee0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 -- 172.21.15.141:0/102988104 >> 172.21.15.141:0/102988104 conn(0x7f150406a800 msgr2=0x7f150406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 -- 172.21.15.141:0/102988104 shutdown_connections 2024-01-16T15:11:33.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.164+0000 7f150a600640 1 -- 172.21.15.141:0/102988104 wait complete. 2024-01-16T15:11:33.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.166+0000 7f150a600640 1 Processor -- start 2024-01-16T15:11:33.143 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.166+0000 7f150a600640 1 -- start start 2024-01-16T15:11:33.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.167+0000 7f150a600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150413c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.167+0000 7f150a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150413e360 0x7f15040a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.167+0000 7f150a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f150415f9c0 0x7f1504071570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.167+0000 7f150a600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15040aa0d0 con 0x7f150413df80 2024-01-16T15:11:33.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.167+0000 7f150a600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15040aa320 con 0x7f150415f9c0 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.167+0000 7f150a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15040aa600 con 0x7f150413e360 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150413c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150413c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57110/0 (socket says 172.21.15.141:57110) 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 -- 172.21.15.141:0/1990075420 learned_addr learned my addr 172.21.15.141:0/1990075420 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f14fa800640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150413e360 0x7f15040a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503e00640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f150415f9c0 0x7f1504071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 -- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150413e360 msgr2=0x7f15040a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150413e360 0x7f15040a93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 -- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f150415f9c0 msgr2=0x7f1504071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f150415f9c0 0x7f1504071570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 -- 172.21.15.141:0/1990075420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40094d0 con 0x7f150413df80 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f14fa800640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150413e360 0x7f15040a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:33.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.168+0000 7f1503400640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150413c7c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f14f4000c00 tx=0x7f14f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.169+0000 7f14fbe00640 1 -- 172.21.15.141:0/1990075420 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f403ad10 con 0x7f150413df80 2024-01-16T15:11:33.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.169+0000 7f14fbe00640 1 -- 172.21.15.141:0/1990075420 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f403ae70 con 0x7f150413df80 2024-01-16T15:11:33.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.169+0000 7f14fbe00640 1 -- 172.21.15.141:0/1990075420 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4044610 con 0x7f150413df80 2024-01-16T15:11:33.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.169+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15040aa8e0 con 0x7f150413df80 2024-01-16T15:11:33.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.169+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15040aaea0 con 0x7f150413df80 2024-01-16T15:11:33.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.170+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15040ab090 con 0x7f150413df80 2024-01-16T15:11:33.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.171+0000 7f14fbe00640 1 -- 172.21.15.141:0/1990075420 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f14f404d450 con 0x7f150413df80 2024-01-16T15:11:33.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.172+0000 7f14fbe00640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14dc06c1f0 0x7f14dc06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.172+0000 7f14fbe00640 1 -- 172.21.15.141:0/1990075420 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f14f402d040 con 0x7f150413df80 2024-01-16T15:11:33.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.173+0000 7f14fa800640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14dc06c1f0 0x7f14dc06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.174+0000 7f14fa800640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14dc06c1f0 0x7f14dc06e6b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f14ec006fd0 tx=0x7f14ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.176+0000 7f14fbe00640 1 -- 172.21.15.141:0/1990075420 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14f4077660 con 0x7f150413df80 2024-01-16T15:11:33.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.368+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f15040675d0 con 0x7f150413df80 2024-01-16T15:11:33.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.375+0000 7f14fbe00640 1 -- 172.21.15.141:0/1990075420 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f14f407ac80 con 0x7f150413df80 2024-01-16T15:11:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14dc06c1f0 msgr2=0x7f14dc06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14dc06c1f0 0x7f14dc06e6b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f14ec006fd0 tx=0x7f14ec008040 comp rx=0 tx=0).stop 2024-01-16T15:11:33.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 msgr2=0x7f150413c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150413c7c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f14f4000c00 tx=0x7f14f40046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 shutdown_connections 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14dc06c1f0 0x7f14dc06e6b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f150415f9c0 0x7f1504071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f150413e360 0x7f15040a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 --2- 172.21.15.141:0/1990075420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f150413df80 0x7f150413c7c0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 >> 172.21.15.141:0/1990075420 conn(0x7f150406a800 msgr2=0x7f150406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 shutdown_connections 2024-01-16T15:11:33.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.378+0000 7f150a600640 1 -- 172.21.15.141:0/1990075420 wait complete. 2024-01-16T15:11:33.374 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:11:33.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.596+0000 7fe926400640 1 Processor -- start 2024-01-16T15:11:33.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.597+0000 7fe926400640 1 -- start start 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe926400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe9200d3910 0x7fe9200d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe926400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe920160e10 0x7fe9200d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe926400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9200d4160 con 0x7fe9200cfcf0 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe926400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9200ceff0 con 0x7fe920160e10 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe926400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9200cf410 con 0x7fe9200d3910 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe91fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe920160e10 0x7fe9200d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe91f400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe9200cfcf0 0x7fe920160700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52576/0 (socket says 172.21.15.141:52576) 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.598+0000 7fe91f400640 1 -- 172.21.15.141:0/3907996524 learned_addr learned my addr 172.21.15.141:0/3907996524 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.599+0000 7fe916800640 1 --2- 172.21.15.141:0/3907996524 >> v2:172.21.15.188:3300/0 conn(0x7fe9200d3910 0x7fe9200d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.599+0000 7fe91fe00640 1 -- 172.21.15.141:0/3907996524 >> v2:172.21.15.188:3300/0 conn(0x7fe9200d3910 msgr2=0x7fe9200d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.599+0000 7fe91fe00640 1 --2- 172.21.15.141:0/3907996524 >> v2:172.21.15.188:3300/0 conn(0x7fe9200d3910 0x7fe9200d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.599+0000 7fe91fe00640 1 -- 172.21.15.141:0/3907996524 >> v1:172.21.15.141:6789/0 conn(0x7fe9200cfcf0 legacy=0x7fe920160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.599+0000 7fe91fe00640 1 -- 172.21.15.141:0/3907996524 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200b2420 con 0x7fe920160e10 2024-01-16T15:11:33.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.599+0000 7fe916800640 1 --2- 172.21.15.141:0/3907996524 >> v2:172.21.15.188:3300/0 conn(0x7fe9200d3910 0x7fe9200d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:33.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.600+0000 7fe91fe00640 1 --2- 172.21.15.141:0/3907996524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe920160e10 0x7fe9200d1640 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe92008b640 tx=0x7fe91802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7428fe488e24eb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.600+0000 7fe91ea00640 1 -- 172.21.15.141:0/3907996524 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91802cbd0 con 0x7fe920160e10 2024-01-16T15:11:33.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.600+0000 7fe91ea00640 1 -- 172.21.15.141:0/3907996524 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91802cd30 con 0x7fe920160e10 2024-01-16T15:11:33.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.600+0000 7fe91ea00640 1 -- 172.21.15.141:0/3907996524 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91802cea0 con 0x7fe920160e10 2024-01-16T15:11:33.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 -- 172.21.15.141:0/3907996524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe920160e10 msgr2=0x7fe9200d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 --2- 172.21.15.141:0/3907996524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe920160e10 0x7fe9200d1640 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe92008b640 tx=0x7fe91802be20 comp rx=0 tx=0).stop 2024-01-16T15:11:33.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 -- 172.21.15.141:0/3907996524 shutdown_connections 2024-01-16T15:11:33.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 --2- 172.21.15.141:0/3907996524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe920160e10 0x7fe9200d1640 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 --2- 172.21.15.141:0/3907996524 >> v2:172.21.15.188:3300/0 conn(0x7fe9200d3910 0x7fe9200d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 -- 172.21.15.141:0/3907996524 >> 172.21.15.141:0/3907996524 conn(0x7fe92009c530 msgr2=0x7fe9200d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:33.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 -- 172.21.15.141:0/3907996524 shutdown_connections 2024-01-16T15:11:33.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.601+0000 7fe926400640 1 -- 172.21.15.141:0/3907996524 wait complete. 2024-01-16T15:11:33.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.603+0000 7fe926400640 1 Processor -- start 2024-01-16T15:11:33.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.603+0000 7fe926400640 1 -- start start 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe926400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9200cfcf0 0x7fe9200cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe926400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9200d3910 0x7fe9200b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe926400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe920160e10 0x7fe920150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe926400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92015da40 con 0x7fe9200d3910 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe926400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9200caaf0 con 0x7fe920160e10 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe926400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92015d480 con 0x7fe9200cfcf0 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9200d3910 0x7fe9200b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9200d3910 0x7fe9200b7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57126/0 (socket says 172.21.15.141:57126) 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 -- 172.21.15.141:0/735365743 learned_addr learned my addr 172.21.15.141:0/735365743 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 -- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9200cfcf0 msgr2=0x7fe9200cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9200cfcf0 0x7fe9200cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 -- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe920160e10 msgr2=0x7fe920150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe920160e10 0x7fe920150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.604+0000 7fe916800640 1 -- 172.21.15.141:0/735365743 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918009a70 con 0x7fe9200d3910 2024-01-16T15:11:33.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.605+0000 7fe916800640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9200d3910 0x7fe9200b7e40 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c7f0 tx=0x7fe90800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.605+0000 7fe917e00640 1 -- 172.21.15.141:0/735365743 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908007a30 con 0x7fe9200d3910 2024-01-16T15:11:33.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.605+0000 7fe917e00640 1 -- 172.21.15.141:0/735365743 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe908007b90 con 0x7fe9200d3910 2024-01-16T15:11:33.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.605+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92015d760 con 0x7fe9200d3910 2024-01-16T15:11:33.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.605+0000 7fe917e00640 1 -- 172.21.15.141:0/735365743 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90800ce40 con 0x7fe9200d3910 2024-01-16T15:11:33.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.605+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9200cd1f0 con 0x7fe9200d3910 2024-01-16T15:11:33.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.606+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9200ca570 con 0x7fe9200d3910 2024-01-16T15:11:33.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.607+0000 7fe917e00640 1 -- 172.21.15.141:0/735365743 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fe9080113f0 con 0x7fe9200d3910 2024-01-16T15:11:33.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.608+0000 7fe917e00640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8f806c1f0 0x7fe8f806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:33.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.608+0000 7fe917e00640 1 -- 172.21.15.141:0/735365743 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe908088880 con 0x7fe9200d3910 2024-01-16T15:11:33.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.609+0000 7fe91f400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8f806c1f0 0x7fe8f806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:33.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.610+0000 7fe91f400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8f806c1f0 0x7fe8f806e6b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe910001550 tx=0x7fe910007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:33.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.612+0000 7fe917e00640 1 -- 172.21.15.141:0/735365743 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9200ca570 con 0x7fe9200d3910 2024-01-16T15:11:33.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:33.854+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7fe9200cac30 con 0x7fe9200d3910 2024-01-16T15:11:34.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.840+0000 7fe917e00640 1 -- 172.21.15.141:0/735365743 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v650) v1 ==== 153+0+0 (secure 0 0 0) 0x7fe9200cac30 con 0x7fe9200d3910 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8f806c1f0 msgr2=0x7fe8f806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8f806c1f0 0x7fe8f806e6b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe910001550 tx=0x7fe910007a40 comp rx=0 tx=0).stop 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9200d3910 msgr2=0x7fe9200b7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9200d3910 0x7fe9200b7e40 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c7f0 tx=0x7fe90800cbf0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 shutdown_connections 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8f806c1f0 0x7fe8f806e6b0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe920160e10 0x7fe920150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9200d3910 0x7fe9200b7e40 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 --2- 172.21.15.141:0/735365743 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe9200cfcf0 0x7fe9200cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 >> 172.21.15.141:0/735365743 conn(0x7fe92009c530 msgr2=0x7fe9200d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 shutdown_connections 2024-01-16T15:11:34.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:34.843+0000 7fe926400640 1 -- 172.21.15.141:0/735365743 wait complete. 2024-01-16T15:11:34.843 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:11:34.843 INFO:tasks.cephfs_test_runner:Starting test: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:11:34.843 INFO:tasks.cephfs_test_runner:test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:11:34.844 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.075+0000 7fb21aa00640 1 Processor -- start 2024-01-16T15:11:35.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.076+0000 7fb21aa00640 1 -- start start 2024-01-16T15:11:35.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.077+0000 7fb21aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2140cd240 0x7fb2140b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.077+0000 7fb21aa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21406cea0 con 0x7fb21409a170 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.077+0000 7fb21aa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21406cbc0 con 0x7fb21406c6b0 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.077+0000 7fb21aa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21406f350 con 0x7fb2140cd240 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.077+0000 7fb213e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2140cd240 0x7fb2140b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.078+0000 7fb213e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2140cd240 0x7fb2140b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57140/0 (socket says 172.21.15.141:57140) 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.078+0000 7fb213e00640 1 -- 172.21.15.141:0/2013894112 learned_addr learned my addr 172.21.15.141:0/2013894112 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.077+0000 7fb218a00640 1 --1- 172.21.15.141:0/2013894112 >> v1:172.21.15.141:6789/0 conn(0x7fb21409a170 0x7fb2140cf640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52578/0 (socket says 172.21.15.141:52578) 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.078+0000 7fb213e00640 1 -- 172.21.15.141:0/2013894112 >> v1:172.21.15.188:6789/0 conn(0x7fb21406c6b0 legacy=0x7fb214099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.078+0000 7fb213e00640 1 -- 172.21.15.141:0/2013894112 >> v1:172.21.15.141:6789/0 conn(0x7fb21409a170 legacy=0x7fb2140cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.078+0000 7fb213e00640 1 -- 172.21.15.141:0/2013894112 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21406f630 con 0x7fb2140cd240 2024-01-16T15:11:35.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.079+0000 7fb213e00640 1 --2- 172.21.15.141:0/2013894112 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2140cd240 0x7fb2140b3820 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb204009870 tx=0x7fb20402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea6b0c928a9ab9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.079+0000 7fb213400640 1 -- 172.21.15.141:0/2013894112 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204030850 con 0x7fb2140cd240 2024-01-16T15:11:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.079+0000 7fb213400640 1 -- 172.21.15.141:0/2013894112 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2040309b0 con 0x7fb2140cd240 2024-01-16T15:11:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.079+0000 7fb213400640 1 -- 172.21.15.141:0/2013894112 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204030d40 con 0x7fb2140cd240 2024-01-16T15:11:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.080+0000 7fb21aa00640 1 -- 172.21.15.141:0/2013894112 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2140cd240 msgr2=0x7fb2140b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.080+0000 7fb21aa00640 1 --2- 172.21.15.141:0/2013894112 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2140cd240 0x7fb2140b3820 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb204009870 tx=0x7fb20402b990 comp rx=0 tx=0).stop 2024-01-16T15:11:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.080+0000 7fb21aa00640 1 -- 172.21.15.141:0/2013894112 shutdown_connections 2024-01-16T15:11:35.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.080+0000 7fb21aa00640 1 --2- 172.21.15.141:0/2013894112 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2140cd240 0x7fb2140b3820 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.080+0000 7fb21aa00640 1 -- 172.21.15.141:0/2013894112 >> 172.21.15.141:0/2013894112 conn(0x7fb2140b2d00 msgr2=0x7fb2140b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.080+0000 7fb21aa00640 1 -- 172.21.15.141:0/2013894112 shutdown_connections 2024-01-16T15:11:35.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.080+0000 7fb21aa00640 1 -- 172.21.15.141:0/2013894112 wait complete. 2024-01-16T15:11:35.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.082+0000 7fb21aa00640 1 Processor -- start 2024-01-16T15:11:35.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.082+0000 7fb21aa00640 1 -- start start 2024-01-16T15:11:35.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb21aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb21406c6b0 0x7fb2140c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb21aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb21409a170 0x7fb2140b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb21aa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2140cd240 0x7fb2140b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb21aa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140b67c0 con 0x7fb21406c6b0 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb21aa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141f3f70 con 0x7fb21409a170 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb21aa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140b4190 con 0x7fb2140cd240 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb213e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb21406c6b0 0x7fb2140c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb213e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb21406c6b0 0x7fb2140c71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57144/0 (socket says 172.21.15.141:57144) 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.083+0000 7fb213e00640 1 -- 172.21.15.141:0/65198688 learned_addr learned my addr 172.21.15.141:0/65198688 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb20b200640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb21409a170 0x7fb2140b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb213e00640 1 -- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2140cd240 msgr2=0x7fb2140b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb213e00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2140cd240 0x7fb2140b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb213e00640 1 -- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb21409a170 msgr2=0x7fb2140b5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb213e00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb21409a170 0x7fb2140b5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb213e00640 1 -- 172.21.15.141:0/65198688 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2040094d0 con 0x7fb21406c6b0 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb20b200640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb21409a170 0x7fb2140b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb213e00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb21406c6b0 0x7fb2140c71c0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fb204009870 tx=0x7fb204004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb210c00640 1 -- 172.21.15.141:0/65198688 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204030800 con 0x7fb21406c6b0 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb210c00640 1 -- 172.21.15.141:0/65198688 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb204039d90 con 0x7fb21406c6b0 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb210c00640 1 -- 172.21.15.141:0/65198688 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204043630 con 0x7fb21406c6b0 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2140b4470 con 0x7fb21406c6b0 2024-01-16T15:11:35.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.084+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2140b4a30 con 0x7fb21406c6b0 2024-01-16T15:11:35.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.086+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2140675d0 con 0x7fb21406c6b0 2024-01-16T15:11:35.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.086+0000 7fb210c00640 1 -- 172.21.15.141:0/65198688 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fb20402e070 con 0x7fb21406c6b0 2024-01-16T15:11:35.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.087+0000 7fb210c00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1ec06c1f0 0x7fb1ec06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.087+0000 7fb210c00640 1 -- 172.21.15.141:0/65198688 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb2040aa860 con 0x7fb21406c6b0 2024-01-16T15:11:35.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.088+0000 7fb20b200640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1ec06c1f0 0x7fb1ec06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.089+0000 7fb20b200640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1ec06c1f0 0x7fb1ec06e6b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb214160d10 tx=0x7fb1fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.092+0000 7fb210c00640 1 -- 172.21.15.141:0/65198688 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb204077120 con 0x7fb21406c6b0 2024-01-16T15:11:35.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.283+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2140739e0 con 0x7fb21406c6b0 2024-01-16T15:11:35.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.285+0000 7fb210c00640 1 -- 172.21.15.141:0/65198688 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13037 (secure 0 0 0) 0x7fb20407a740 con 0x7fb21406c6b0 2024-01-16T15:11:35.262 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:35.263 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":94,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:11:27.660596+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.141:6834/3399417259":"2024-01-17T15:11:24.632792+0000","172.21.15.141:6832/3399417259":"2024-01-17T15:11:24.632792+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1ec06c1f0 msgr2=0x7fb1ec06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1ec06c1f0 0x7fb1ec06e6b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb214160d10 tx=0x7fb1fc009040 comp rx=0 tx=0).stop 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb21406c6b0 msgr2=0x7fb2140c71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb21406c6b0 0x7fb2140c71c0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fb204009870 tx=0x7fb204004730 comp rx=0 tx=0).stop 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 shutdown_connections 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb1ec06c1f0 0x7fb1ec06e6b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb2140cd240 0x7fb2140b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb21409a170 0x7fb2140b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 --2- 172.21.15.141:0/65198688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb21406c6b0 0x7fb2140c71c0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.287+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 >> 172.21.15.141:0/65198688 conn(0x7fb2140b2d00 msgr2=0x7fb214099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.288+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 shutdown_connections 2024-01-16T15:11:35.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.288+0000 7fb21aa00640 1 -- 172.21.15.141:0/65198688 wait complete. 2024-01-16T15:11:35.283 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:35.284 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:35.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.518+0000 7f6ff9400640 1 Processor -- start 2024-01-16T15:11:35.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.519+0000 7f6ff9400640 1 -- start start 2024-01-16T15:11:35.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.520+0000 7f6ff9400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6ff4143ae0 0x7f6ff40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.520+0000 7f6ff9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff40cf7d0 0x7f6ff415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.520+0000 7f6ff9400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40cfc30 con 0x7f6ff415e1e0 2024-01-16T15:11:35.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.520+0000 7f6ff9400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff40b1ee0 con 0x7f6ff4143ae0 2024-01-16T15:11:35.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.520+0000 7f6ff9400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40b2780 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.521+0000 7f6ff2800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6ff4143ae0 0x7f6ff40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.521+0000 7f6ff2800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6ff4143ae0 0x7f6ff40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:58386/0 (socket says 172.21.15.141:58386) 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.521+0000 7f6ff2800640 1 -- 172.21.15.141:0/978902947 learned_addr learned my addr 172.21.15.141:0/978902947 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.521+0000 7f6febe00640 1 --2- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff40cf7d0 0x7f6ff415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.522+0000 7f6ff2800640 1 -- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff40cf7d0 msgr2=0x7f6ff415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.522+0000 7f6ff2800640 1 --2- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff40cf7d0 0x7f6ff415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.522+0000 7f6ff2800640 1 -- 172.21.15.141:0/978902947 >> v1:172.21.15.188:6789/0 conn(0x7f6ff415e1e0 legacy=0x7f6ff415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.522+0000 7f6ff2800640 1 -- 172.21.15.141:0/978902947 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff415e990 con 0x7f6ff4143ae0 2024-01-16T15:11:35.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.523+0000 7f6ff2800640 1 --2- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff4143ae0 0x7f6ff40cf290 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009870 tx=0x7f6fe402db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f168a03a52569b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.523+0000 7f6ff1e00640 1 -- 172.21.15.141:0/978902947 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe402e940 con 0x7f6ff4143ae0 2024-01-16T15:11:35.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff1e00640 1 -- 172.21.15.141:0/978902947 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe402eaa0 con 0x7f6ff4143ae0 2024-01-16T15:11:35.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff1e00640 1 -- 172.21.15.141:0/978902947 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe402ee30 con 0x7f6ff4143ae0 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 -- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff4143ae0 msgr2=0x7f6ff40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 --2- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff4143ae0 0x7f6ff40cf290 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009870 tx=0x7f6fe402db90 comp rx=0 tx=0).stop 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 -- 172.21.15.141:0/978902947 shutdown_connections 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 --2- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff40cf7d0 0x7f6ff415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 --2- 172.21.15.141:0/978902947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff4143ae0 0x7f6ff40cf290 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 -- 172.21.15.141:0/978902947 >> 172.21.15.141:0/978902947 conn(0x7f6ff406a800 msgr2=0x7f6ff406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 -- 172.21.15.141:0/978902947 shutdown_connections 2024-01-16T15:11:35.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.524+0000 7f6ff9400640 1 -- 172.21.15.141:0/978902947 wait complete. 2024-01-16T15:11:35.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.526+0000 7f6ff9400640 1 Processor -- start 2024-01-16T15:11:35.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.526+0000 7f6ff9400640 1 -- start start 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.527+0000 7f6ff9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ff40cf7d0 0x7f6ff40cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.527+0000 7f6ff9400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff4143ae0 0x7f6ff4141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.527+0000 7f6ff9400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff415e1e0 0x7f6ff40c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.527+0000 7f6ff9400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40c8560 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.527+0000 7f6ff9400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41422a0 con 0x7f6ff415e1e0 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.527+0000 7f6ff9400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40c55c0 con 0x7f6ff4143ae0 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ff40cf7d0 0x7f6ff40cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ff40cf7d0 0x7f6ff40cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57152/0 (socket says 172.21.15.141:57152) 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 -- 172.21.15.141:0/4101687788 learned_addr learned my addr 172.21.15.141:0/4101687788 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 -- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff4143ae0 msgr2=0x7f6ff4141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff4143ae0 0x7f6ff4141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 -- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff415e1e0 msgr2=0x7f6ff40c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff415e1e0 0x7f6ff40c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 -- 172.21.15.141:0/4101687788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40094d0 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.528+0000 7f6ff2800640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ff40cf7d0 0x7f6ff40cd020 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000c00 tx=0x7f6fe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.529+0000 7f6feaa00640 1 -- 172.21.15.141:0/4101687788 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4040600 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.529+0000 7f6feaa00640 1 -- 172.21.15.141:0/4101687788 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4040760 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.529+0000 7f6feaa00640 1 -- 172.21.15.141:0/4101687788 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4040b10 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.529+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff40cd720 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.529+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff40cdce0 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.530+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff40c8330 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.531+0000 7f6feaa00640 1 -- 172.21.15.141:0/4101687788 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f6fe4040c70 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.532+0000 7f6feaa00640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6fcc06c1f0 0x7f6fcc06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.532+0000 7f6feaa00640 1 -- 172.21.15.141:0/4101687788 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6fe40ab010 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.532+0000 7f6febe00640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6fcc06c1f0 0x7f6fcc06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.534+0000 7f6febe00640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6fcc06c1f0 0x7f6fcc06e6b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc006a90 tx=0x7f6fdc006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.536+0000 7f6feaa00640 1 -- 172.21.15.141:0/4101687788 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fe4077a50 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.726+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ff40675d0 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.727+0000 7f6feaa00640 1 -- 172.21.15.141:0/4101687788 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fe403b0a0 con 0x7f6ff40cf7d0 2024-01-16T15:11:35.705 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6fcc06c1f0 msgr2=0x7f6fcc06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6fcc06c1f0 0x7f6fcc06e6b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc006a90 tx=0x7f6fdc006a20 comp rx=0 tx=0).stop 2024-01-16T15:11:35.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ff40cf7d0 msgr2=0x7f6ff40cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ff40cf7d0 0x7f6ff40cd020 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000c00 tx=0x7f6fe40046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 shutdown_connections 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6fcc06c1f0 0x7f6fcc06e6b0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ff415e1e0 0x7f6ff40c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6ff4143ae0 0x7f6ff4141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 --2- 172.21.15.141:0/4101687788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ff40cf7d0 0x7f6ff40cd020 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.730+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 >> 172.21.15.141:0/4101687788 conn(0x7f6ff406a800 msgr2=0x7f6ff415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.731+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 shutdown_connections 2024-01-16T15:11:35.710 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.731+0000 7f6ff9400640 1 -- 172.21.15.141:0/4101687788 wait complete. 2024-01-16T15:11:35.726 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:11:35.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.960+0000 7f32bd800640 1 Processor -- start 2024-01-16T15:11:35.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.960+0000 7f32bd800640 1 -- start start 2024-01-16T15:11:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.961+0000 7f32bd800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f32b80cc990 0x7f32b806d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.961+0000 7f32bd800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b8070e50 con 0x7f32b80cc500 2024-01-16T15:11:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.961+0000 7f32bd800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b8144020 con 0x7f32b8143c40 2024-01-16T15:11:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.961+0000 7f32bd800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b80723c0 con 0x7f32b80cc990 2024-01-16T15:11:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.962+0000 7f32b7400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f32b80cc500 0x7f32b806d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52586/0 (socket says 172.21.15.141:52586) 2024-01-16T15:11:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.962+0000 7f32b7400640 1 -- 172.21.15.141:0/687003036 learned_addr learned my addr 172.21.15.141:0/687003036 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:35.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.962+0000 7f32bce00640 1 --2- 172.21.15.141:0/687003036 >> v2:172.21.15.141:3300/0 conn(0x7f32b80cc990 0x7f32b806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.962+0000 7f32b6a00640 1 -- 172.21.15.141:0/687003036 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558461301 0 0) 0x7f32b8070e50 con 0x7f32b80cc500 2024-01-16T15:11:35.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.962+0000 7f32bce00640 1 -- 172.21.15.141:0/687003036 >> v1:172.21.15.188:6789/0 conn(0x7f32b8143c40 legacy=0x7f32b8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:35.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.962+0000 7f32bce00640 1 -- 172.21.15.141:0/687003036 >> v1:172.21.15.141:6789/0 conn(0x7f32b80cc500 legacy=0x7f32b806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.962+0000 7f32bce00640 1 -- 172.21.15.141:0/687003036 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b8071e00 con 0x7f32b80cc990 2024-01-16T15:11:35.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.963+0000 7f32bce00640 1 --2- 172.21.15.141:0/687003036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b806d870 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009d90 tx=0x7f32b002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a918814b6b41bcf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32b6a00640 1 -- 172.21.15.141:0/687003036 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b002cbd0 con 0x7f32b80cc990 2024-01-16T15:11:35.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32b6a00640 1 -- 172.21.15.141:0/687003036 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b002cd30 con 0x7f32b80cc990 2024-01-16T15:11:35.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32b6a00640 1 -- 172.21.15.141:0/687003036 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b002cea0 con 0x7f32b80cc990 2024-01-16T15:11:35.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32bd800640 1 -- 172.21.15.141:0/687003036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 msgr2=0x7f32b806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32bd800640 1 --2- 172.21.15.141:0/687003036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b806d870 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009d90 tx=0x7f32b002be20 comp rx=0 tx=0).stop 2024-01-16T15:11:35.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32bd800640 1 -- 172.21.15.141:0/687003036 shutdown_connections 2024-01-16T15:11:35.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32bd800640 1 --2- 172.21.15.141:0/687003036 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b806d870 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.964+0000 7f32bd800640 1 -- 172.21.15.141:0/687003036 >> 172.21.15.141:0/687003036 conn(0x7f32b806a800 msgr2=0x7f32b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:35.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.965+0000 7f32bd800640 1 -- 172.21.15.141:0/687003036 shutdown_connections 2024-01-16T15:11:35.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.965+0000 7f32bd800640 1 -- 172.21.15.141:0/687003036 wait complete. 2024-01-16T15:11:35.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.966+0000 7f32bd800640 1 Processor -- start 2024-01-16T15:11:35.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.966+0000 7f32bd800640 1 -- start start 2024-01-16T15:11:35.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.967+0000 7f32bd800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32b80cc500 0x7f32b813bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.967+0000 7f32bd800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b80ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.967+0000 7f32bd800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32b8143c40 0x7f32b80a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.967+0000 7f32bd800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b813c270 con 0x7f32b80cc990 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.967+0000 7f32bd800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b813c550 con 0x7f32b8143c40 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.967+0000 7f32bd800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b813c830 con 0x7f32b80cc500 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b80ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b80ac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57172/0 (socket says 172.21.15.141:57172) 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 -- 172.21.15.141:0/3838136322 learned_addr learned my addr 172.21.15.141:0/3838136322 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7e00640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32b80cc500 0x7f32b813bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 -- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32b80cc500 msgr2=0x7f32b813bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32b80cc500 0x7f32b813bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 -- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32b8143c40 msgr2=0x7f32b80a92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32bce00640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32b8143c40 0x7f32b80a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32b8143c40 0x7f32b80a92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.968+0000 7f32b7400640 1 -- 172.21.15.141:0/3838136322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b0009a70 con 0x7f32b80cc990 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.969+0000 7f32b7400640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b80ac0e0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c4d0 tx=0x7f32a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.969+0000 7f32afe00640 1 -- 172.21.15.141:0/3838136322 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a00074e0 con 0x7f32b80cc990 2024-01-16T15:11:35.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.969+0000 7f32afe00640 1 -- 172.21.15.141:0/3838136322 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a0007660 con 0x7f32b80cc990 2024-01-16T15:11:35.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.969+0000 7f32afe00640 1 -- 172.21.15.141:0/3838136322 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0004a20 con 0x7f32b80cc990 2024-01-16T15:11:35.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.969+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b813cb10 con 0x7f32b80cc990 2024-01-16T15:11:35.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.969+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b80ac620 con 0x7f32b80cc990 2024-01-16T15:11:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.970+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b8067d10 con 0x7f32b80cc990 2024-01-16T15:11:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.971+0000 7f32afe00640 1 -- 172.21.15.141:0/3838136322 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f32a0004b80 con 0x7f32b80cc990 2024-01-16T15:11:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.972+0000 7f32afe00640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f329006c1f0 0x7f329006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.972+0000 7f32afe00640 1 -- 172.21.15.141:0/3838136322 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f32a0087b70 con 0x7f32b80cc990 2024-01-16T15:11:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.972+0000 7f32b7e00640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f329006c1f0 0x7f329006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:35.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.974+0000 7f32b7e00640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f329006c1f0 0x7f329006e6b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f32a800a730 tx=0x7f32a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:35.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:35.976+0000 7f32afe00640 1 -- 172.21.15.141:0/3838136322 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32a0054140 con 0x7f32b80cc990 2024-01-16T15:11:36.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.239+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f32b8067450 con 0x7f32b80cc990 2024-01-16T15:11:36.319 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:36.329 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:36.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.852+0000 7f32afe00640 1 -- 172.21.15.141:0/3838136322 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v652) v1 ==== 158+0+0 (secure 0 0 0) 0x7f32a00180e0 con 0x7f32b80cc990 2024-01-16T15:11:36.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f329006c1f0 msgr2=0x7f329006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f329006c1f0 0x7f329006e6b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f32a800a730 tx=0x7f32a8008040 comp rx=0 tx=0).stop 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 msgr2=0x7f32b80ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b80ac0e0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c4d0 tx=0x7f32a000c990 comp rx=0 tx=0).stop 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 shutdown_connections 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f329006c1f0 0x7f329006e6b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32b8143c40 0x7f32b80a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f32b80cc990 0x7f32b80ac0e0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 --2- 172.21.15.141:0/3838136322 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f32b80cc500 0x7f32b813bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 >> 172.21.15.141:0/3838136322 conn(0x7f32b806a800 msgr2=0x7f32b80aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 shutdown_connections 2024-01-16T15:11:36.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:36.856+0000 7f32bd800640 1 -- 172.21.15.141:0/3838136322 wait complete. 2024-01-16T15:11:36.856 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:11:37.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.093+0000 7f842ea00640 1 Processor -- start 2024-01-16T15:11:37.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.093+0000 7f842ea00640 1 -- start start 2024-01-16T15:11:37.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f842ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f842814b2c0 0x7f84280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:37.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f842ea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84280cf7d0 0x7f8428070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:37.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f842ea00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8428070ea0 0x7f84280b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:37.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f842ea00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842814b6a0 con 0x7f842814b2c0 2024-01-16T15:11:37.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f842ea00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84280cfbf0 con 0x7f8428070ea0 2024-01-16T15:11:37.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f842ea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8428071690 con 0x7f84280cf7d0 2024-01-16T15:11:37.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f8427e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f842814b2c0 0x7f84280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:37.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f8427e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f842814b2c0 0x7f84280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57186/0 (socket says 172.21.15.141:57186) 2024-01-16T15:11:37.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f8427e00640 1 -- 172.21.15.141:0/1635950417 learned_addr learned my addr 172.21.15.141:0/1635950417 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:37.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f842ca00640 1 --2- 172.21.15.141:0/1635950417 >> v2:172.21.15.188:3300/0 conn(0x7f8428070ea0 0x7f84280b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f8427e00640 1 -- 172.21.15.141:0/1635950417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84280cf7d0 msgr2=0x7f8428070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.095+0000 7f841f200640 1 --2- 172.21.15.141:0/1635950417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84280cf7d0 0x7f8428070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.096+0000 7f8427e00640 1 --2- 172.21.15.141:0/1635950417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84280cf7d0 0x7f8428070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.096+0000 7f8427e00640 1 -- 172.21.15.141:0/1635950417 >> v2:172.21.15.188:3300/0 conn(0x7f8428070ea0 msgr2=0x7f84280b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.096+0000 7f8427e00640 1 --2- 172.21.15.141:0/1635950417 >> v2:172.21.15.188:3300/0 conn(0x7f8428070ea0 0x7f84280b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.096+0000 7f8427e00640 1 -- 172.21.15.141:0/1635950417 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8428071970 con 0x7f842814b2c0 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.096+0000 7f842ca00640 1 --2- 172.21.15.141:0/1635950417 >> v2:172.21.15.188:3300/0 conn(0x7f8428070ea0 0x7f84280b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:37.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.097+0000 7f8427e00640 1 --2- 172.21.15.141:0/1635950417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842814b2c0 0x7f84280cf290 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f842806e2a0 tx=0x7f841802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5a418a4ba289f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:37.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.097+0000 7f8427400640 1 -- 172.21.15.141:0/1635950417 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418031870 con 0x7f842814b2c0 2024-01-16T15:11:37.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.097+0000 7f8427400640 1 -- 172.21.15.141:0/1635950417 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84180319d0 con 0x7f842814b2c0 2024-01-16T15:11:37.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.097+0000 7f8427400640 1 -- 172.21.15.141:0/1635950417 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418031d60 con 0x7f842814b2c0 2024-01-16T15:11:37.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.097+0000 7f842ea00640 1 -- 172.21.15.141:0/1635950417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842814b2c0 msgr2=0x7f84280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:37.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.097+0000 7f842ea00640 1 --2- 172.21.15.141:0/1635950417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842814b2c0 0x7f84280cf290 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f842806e2a0 tx=0x7f841802b910 comp rx=0 tx=0).stop 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.098+0000 7f842ea00640 1 -- 172.21.15.141:0/1635950417 shutdown_connections 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.098+0000 7f842ea00640 1 --2- 172.21.15.141:0/1635950417 >> v2:172.21.15.188:3300/0 conn(0x7f8428070ea0 0x7f84280b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.098+0000 7f842ea00640 1 --2- 172.21.15.141:0/1635950417 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f84280cf7d0 0x7f8428070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.098+0000 7f842ea00640 1 --2- 172.21.15.141:0/1635950417 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f842814b2c0 0x7f84280cf290 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.098+0000 7f842ea00640 1 -- 172.21.15.141:0/1635950417 >> 172.21.15.141:0/1635950417 conn(0x7f842806a800 msgr2=0x7f842806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.098+0000 7f842ea00640 1 -- 172.21.15.141:0/1635950417 shutdown_connections 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.098+0000 7f842ea00640 1 -- 172.21.15.141:0/1635950417 wait complete. 2024-01-16T15:11:37.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.100+0000 7f842ea00640 1 Processor -- start 2024-01-16T15:11:37.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.100+0000 7f842ea00640 1 -- start start 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f842ea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8428070ea0 0x7f8428147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f842ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84280cf7d0 0x7f8428145bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f842ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842814b2c0 0x7f84280aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f842ea00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84281563c0 con 0x7f84280cf7d0 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f842ea00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8428146130 con 0x7f842814b2c0 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f842ea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8428155d70 con 0x7f8428070ea0 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84280cf7d0 0x7f8428145bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84280cf7d0 0x7f8428145bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57200/0 (socket says 172.21.15.141:57200) 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 -- 172.21.15.141:0/3528630696 learned_addr learned my addr 172.21.15.141:0/3528630696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f8427e00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8428070ea0 0x7f8428147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 -- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8428070ea0 msgr2=0x7f8428147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8428070ea0 0x7f8428147d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 -- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842814b2c0 msgr2=0x7f84280aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:37.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f842ca00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842814b2c0 0x7f84280aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:37.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842814b2c0 0x7f84280aff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.101+0000 7f841f200640 1 -- 172.21.15.141:0/3528630696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84180094d0 con 0x7f84280cf7d0 2024-01-16T15:11:37.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.102+0000 7f841f200640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84280cf7d0 0x7f8428145bf0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f841000c820 tx=0x7f841000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:37.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.102+0000 7f8424c00640 1 -- 172.21.15.141:0/3528630696 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410004220 con 0x7f84280cf7d0 2024-01-16T15:11:37.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.102+0000 7f8424c00640 1 -- 172.21.15.141:0/3528630696 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84100043a0 con 0x7f84280cf7d0 2024-01-16T15:11:37.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.102+0000 7f8424c00640 1 -- 172.21.15.141:0/3528630696 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410007a70 con 0x7f84280cf7d0 2024-01-16T15:11:37.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.102+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8428146360 con 0x7f84280cf7d0 2024-01-16T15:11:37.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.102+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8428146640 con 0x7f84280cf7d0 2024-01-16T15:11:37.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.103+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8428156500 con 0x7f84280cf7d0 2024-01-16T15:11:37.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.104+0000 7f8424c00640 1 -- 172.21.15.141:0/3528630696 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f8410007bd0 con 0x7f84280cf7d0 2024-01-16T15:11:37.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.105+0000 7f8424c00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f840006c1f0 0x7f840006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:37.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.105+0000 7f8424c00640 1 -- 172.21.15.141:0/3528630696 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f84100886a0 con 0x7f84280cf7d0 2024-01-16T15:11:37.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.106+0000 7f8427e00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f840006c1f0 0x7f840006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:37.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.107+0000 7f8427e00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f840006c1f0 0x7f840006e6b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8418036000 tx=0x7f841803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:37.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.110+0000 7f8424c00640 1 -- 172.21.15.141:0/3528630696 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8428156500 con 0x7f84280cf7d0 2024-01-16T15:11:37.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.298+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8428156500 con 0x7f84280cf7d0 2024-01-16T15:11:37.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.304+0000 7f8424c00640 1 -- 172.21.15.141:0/3528630696 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8428156500 con 0x7f84280cf7d0 2024-01-16T15:11:37.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f840006c1f0 msgr2=0x7f840006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:37.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f840006c1f0 0x7f840006e6b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8418036000 tx=0x7f841803c040 comp rx=0 tx=0).stop 2024-01-16T15:11:37.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84280cf7d0 msgr2=0x7f8428145bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:37.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84280cf7d0 0x7f8428145bf0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f841000c820 tx=0x7f841000cce0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 shutdown_connections 2024-01-16T15:11:37.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f840006c1f0 0x7f840006e6b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842814b2c0 0x7f84280aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84280cf7d0 0x7f8428145bf0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 --2- 172.21.15.141:0/3528630696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8428070ea0 0x7f8428147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:37.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 >> 172.21.15.141:0/3528630696 conn(0x7f842806a800 msgr2=0x7f842806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:37.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 shutdown_connections 2024-01-16T15:11:37.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:37.307+0000 7f842ea00640 1 -- 172.21.15.141:0/3528630696 wait complete. 2024-01-16T15:11:37.301 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:11:37.302 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:37.417 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:11:37.570 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:11:37.570 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:11:37.570 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:37.627 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:11:37.627 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:11:37.627 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:37.684 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:11:37.684 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:11:37.684 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:37.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:11:37.738 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:11:37.738 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:37.792 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:11:37.793 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:11:37.793 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:38.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.031+0000 7fa6ba800640 1 Processor -- start 2024-01-16T15:11:38.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.031+0000 7fa6ba800640 1 -- start start 2024-01-16T15:11:38.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.032+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40cd240 0x7fa6b40b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.032+0000 7fa6ba800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b406cea0 con 0x7fa6b406c6b0 2024-01-16T15:11:38.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.032+0000 7fa6ba800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b406cbc0 con 0x7fa6b409a170 2024-01-16T15:11:38.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.032+0000 7fa6ba800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b406f350 con 0x7fa6b40cd240 2024-01-16T15:11:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.033+0000 7fa6abe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa6b406c6b0 0x7fa6b4099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52590/0 (socket says 172.21.15.141:52590) 2024-01-16T15:11:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.033+0000 7fa6abe00640 1 -- 172.21.15.141:0/2103681902 learned_addr learned my addr 172.21.15.141:0/2103681902 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.033+0000 7fa6b3400640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40cd240 0x7fa6b40b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307066320 0 0) 0x7fa6b406cea0 con 0x7fa6b406c6b0 2024-01-16T15:11:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa694003520 con 0x7fa6b406c6b0 2024-01-16T15:11:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777878811 0 0) 0x7fa694003520 con 0x7fa6b406c6b0 2024-01-16T15:11:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40cd240 msgr2=0x7fa6b40b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40cd240 0x7fa6b40b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 >> v1:172.21.15.188:6789/0 conn(0x7fa6b409a170 legacy=0x7fa6b40cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b406f630 con 0x7fa6b406c6b0 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa69c001670 con 0x7fa6b406c6b0 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 >> v1:172.21.15.141:6789/0 conn(0x7fa6b406c6b0 legacy=0x7fa6b4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6940039a0 0x7fa694003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940047c0 0x7fa694004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40cd240 0x7fa694005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b406f350 con 0x7fa6940039a0 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69400ba60 con 0x7fa6940047c0 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69400bc20 con 0x7fa6b40cd240 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.034+0000 7fa6b3400640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6940039a0 0x7fa694003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.035+0000 7fa6b3400640 1 -- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40cd240 msgr2=0x7fa694005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:38.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.035+0000 7fa6b3400640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40cd240 0x7fa694005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.035+0000 7fa6b3400640 1 -- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940047c0 msgr2=0x7fa694004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:38.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.035+0000 7fa6b3400640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940047c0 0x7fa694004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.035+0000 7fa6b3400640 1 -- 172.21.15.141:0/2103681902 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa694003520 con 0x7fa6940039a0 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.036+0000 7fa6b3400640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6940039a0 0x7fa694003dc0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009a90 tx=0x7fa6940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a424455c81fd17cc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.036+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a402c6b0 con 0x7fa6940039a0 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.036+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a402c810 con 0x7fa6940039a0 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.036+0000 7fa6b2a00640 1 -- 172.21.15.141:0/2103681902 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a402cba0 con 0x7fa6940039a0 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.037+0000 7fa6ba800640 1 -- 172.21.15.141:0/2103681902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6940039a0 msgr2=0x7fa694003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.037+0000 7fa6ba800640 1 --2- 172.21.15.141:0/2103681902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6940039a0 0x7fa694003dc0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009a90 tx=0x7fa6940057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.037+0000 7fa6b3e00640 1 -- 172.21.15.141:0/2103681902 reap_dead start 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.037+0000 7fa6ba800640 1 -- 172.21.15.141:0/2103681902 shutdown_connections 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.037+0000 7fa6ba800640 1 -- 172.21.15.141:0/2103681902 >> 172.21.15.141:0/2103681902 conn(0x7fa6b40b2d00 msgr2=0x7fa6b40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.037+0000 7fa6ba800640 1 -- 172.21.15.141:0/2103681902 shutdown_connections 2024-01-16T15:11:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.037+0000 7fa6ba800640 1 -- 172.21.15.141:0/2103681902 wait complete. 2024-01-16T15:11:38.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.039+0000 7fa6ba800640 1 Processor -- start 2024-01-16T15:11:38.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.039+0000 7fa6ba800640 1 -- start start 2024-01-16T15:11:38.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940039a0 0x7fa6b41f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40b4b90 0x7fa6b40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b415e8a0 0x7fa6b415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6ba800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b40cae30 con 0x7fa6b415e8a0 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6ba800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b40c56b0 con 0x7fa6940039a0 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6ba800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b40c5990 con 0x7fa6b40b4b90 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6b3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b415e8a0 0x7fa6b415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6b3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b415e8a0 0x7fa6b415ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57218/0 (socket says 172.21.15.141:57218) 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.040+0000 7fa6b3e00640 1 -- 172.21.15.141:0/96447298 learned_addr learned my addr 172.21.15.141:0/96447298 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3400640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940039a0 0x7fa6b41f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6abe00640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40b4b90 0x7fa6b40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3e00640 1 -- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40b4b90 msgr2=0x7fa6b40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3e00640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40b4b90 0x7fa6b40b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3e00640 1 -- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940039a0 msgr2=0x7fa6b41f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3e00640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940039a0 0x7fa6b41f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3e00640 1 -- 172.21.15.141:0/96447298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b40c5c70 con 0x7fa6b415e8a0 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3400640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940039a0 0x7fa6b41f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6b3e00640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b415e8a0 0x7fa6b415ed30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4070a10 tx=0x7fa6ac00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6ab400640 1 -- 172.21.15.141:0/96447298 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac005d40 con 0x7fa6b415e8a0 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6ab400640 1 -- 172.21.15.141:0/96447298 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac00ced0 con 0x7fa6b415e8a0 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6ab400640 1 -- 172.21.15.141:0/96447298 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac00a850 con 0x7fa6b415e8a0 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.041+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b40c5f50 con 0x7fa6b415e8a0 2024-01-16T15:11:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.042+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b415f270 con 0x7fa6b415e8a0 2024-01-16T15:11:38.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.043+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b40c6140 con 0x7fa6b415e8a0 2024-01-16T15:11:38.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.044+0000 7fa6ab400640 1 -- 172.21.15.141:0/96447298 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fa6ac00a9b0 con 0x7fa6b415e8a0 2024-01-16T15:11:38.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.044+0000 7fa6ab400640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa68c06c1f0 0x7fa68c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.045+0000 7fa6ab400640 1 -- 172.21.15.141:0/96447298 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa6ac087b40 con 0x7fa6b415e8a0 2024-01-16T15:11:38.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.045+0000 7fa6b3400640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa68c06c1f0 0x7fa68c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.046+0000 7fa6b3400640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa68c06c1f0 0x7fa68c06e6b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4036000 tx=0x7fa6a4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.049+0000 7fa6ab400640 1 -- 172.21.15.141:0/96447298 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6ac054110 con 0x7fa6b415e8a0 2024-01-16T15:11:38.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.274+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6b40675d0 con 0x7fa6b415e8a0 2024-01-16T15:11:38.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.275+0000 7fa6ab400640 1 -- 172.21.15.141:0/96447298 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+3708 (secure 0 0 0) 0x7fa6ac017e90 con 0x7fa6b415e8a0 2024-01-16T15:11:38.253 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:38.253 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":57,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/1211961850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1211961850},{"type":"v1","addr":"172.21.15.141:6835","nonce":1211961850}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3677420872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3677420872},{"type":"v1","addr":"172.21.15.188:6835","nonce":3677420872}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/4249270856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":4249270856},{"type":"v1","addr":"172.21.15.188:6837","nonce":4249270856}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/804049895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":804049895},{"type":"v1","addr":"172.21.15.141:6834","nonce":804049895}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[]} 2024-01-16T15:11:38.253 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 57 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.278+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa68c06c1f0 msgr2=0x7fa68c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.278+0000 7fa6ba800640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa68c06c1f0 0x7fa68c06e6b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4036000 tx=0x7fa6a4030000 comp rx=0 tx=0).stop 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.278+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b415e8a0 msgr2=0x7fa6b415ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.278+0000 7fa6ba800640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b415e8a0 0x7fa6b415ed30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4070a10 tx=0x7fa6ac00cb90 comp rx=0 tx=0).stop 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 shutdown_connections 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa68c06c1f0 0x7fa68c06e6b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6b415e8a0 0x7fa6b415ed30 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6b40b4b90 0x7fa6b40b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 --2- 172.21.15.141:0/96447298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6940039a0 0x7fa6b41f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 >> 172.21.15.141:0/96447298 conn(0x7fa6b40b2d00 msgr2=0x7fa6b406b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:38.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 shutdown_connections 2024-01-16T15:11:38.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.279+0000 7fa6ba800640 1 -- 172.21.15.141:0/96447298 wait complete. 2024-01-16T15:11:38.276 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:11:38.276 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:11:38.276 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:11:38.276 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:38.276 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:11:38.276 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:11:38.276 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:11:38.277 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:38.277 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:11:38.277 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:11:38.277 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:11:38.277 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:38.277 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:11:38.277 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:11:38.277 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:11:38.278 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:11:38.294 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:11:38.294 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:11:38.296 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:11:38.296 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:11:38.297 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:11:38.298 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:11:38.300 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:11:38.300 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:11:38.344 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:11:38.345 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:38.347 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:11:38.348 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:38.351 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:11:38.351 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:38.354 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:11:38.354 DEBUG:teuthology.parallel:result is None 2024-01-16T15:11:38.354 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:11:38.500 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:11:38.500 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T15:11:38.505 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T15:11:38.505 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T15:11:38.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.557+0000 7f2ad0200640 1 Processor -- start 2024-01-16T15:11:38.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.557+0000 7f2ad0200640 1 -- start start 2024-01-16T15:11:38.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ad0200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80ae5a0 0x7f2ac815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ad0200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2ac80cf290 0x7f2ac80cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ad0200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac8070c40 con 0x7f2ac80b3ae0 2024-01-16T15:11:38.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ad0200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac8071450 con 0x7f2ac80cf290 2024-01-16T15:11:38.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ad0200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80cfc60 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2acce00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2ac80b3ae0 0x7f2ac806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52626/0 (socket says 172.21.15.141:52626) 2024-01-16T15:11:38.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2acce00640 1 -- 172.21.15.141:0/2444891516 learned_addr learned my addr 172.21.15.141:0/2444891516 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:38.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ace200640 1 --2- 172.21.15.141:0/2444891516 >> v2:172.21.15.188:3300/0 conn(0x7f2ac80cf290 0x7f2ac80cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2acd800640 1 --2- 172.21.15.141:0/2444891516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80ae5a0 0x7f2ac815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ac7e00640 1 -- 172.21.15.141:0/2444891516 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565950124 0 0) 0x7f2ac8070c40 con 0x7f2ac80b3ae0 2024-01-16T15:11:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.559+0000 7f2ac7e00640 1 -- 172.21.15.141:0/2444891516 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa8003520 con 0x7f2ac80b3ae0 2024-01-16T15:11:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.560+0000 7f2acd800640 1 -- 172.21.15.141:0/2444891516 >> v2:172.21.15.188:3300/0 conn(0x7f2ac80cf290 msgr2=0x7f2ac80cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.560+0000 7f2acd800640 1 --2- 172.21.15.141:0/2444891516 >> v2:172.21.15.188:3300/0 conn(0x7f2ac80cf290 0x7f2ac80cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.560+0000 7f2acd800640 1 -- 172.21.15.141:0/2444891516 >> v1:172.21.15.141:6789/0 conn(0x7f2ac80b3ae0 legacy=0x7f2ac806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.560+0000 7f2acd800640 1 -- 172.21.15.141:0/2444891516 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac806c470 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.561+0000 7f2acd800640 1 --2- 172.21.15.141:0/2444891516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80ae5a0 0x7f2ac815e430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80097a0 tx=0x7f2ab802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc7b64013fdacd86 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.561+0000 7f2ac7e00640 1 -- 172.21.15.141:0/2444891516 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8030610 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.561+0000 7f2ac7e00640 1 -- 172.21.15.141:0/2444891516 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8030770 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.561+0000 7f2ac7e00640 1 -- 172.21.15.141:0/2444891516 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8030b00 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 -- 172.21.15.141:0/2444891516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80ae5a0 msgr2=0x7f2ac815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 --2- 172.21.15.141:0/2444891516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80ae5a0 0x7f2ac815e430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80097a0 tx=0x7f2ab802b960 comp rx=0 tx=0).stop 2024-01-16T15:11:38.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 -- 172.21.15.141:0/2444891516 shutdown_connections 2024-01-16T15:11:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 --2- 172.21.15.141:0/2444891516 >> v2:172.21.15.188:3300/0 conn(0x7f2ac80cf290 0x7f2ac80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 --2- 172.21.15.141:0/2444891516 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80ae5a0 0x7f2ac815e430 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 -- 172.21.15.141:0/2444891516 >> 172.21.15.141:0/2444891516 conn(0x7f2ac806a800 msgr2=0x7f2ac806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 -- 172.21.15.141:0/2444891516 shutdown_connections 2024-01-16T15:11:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.562+0000 7f2ad0200640 1 -- 172.21.15.141:0/2444891516 wait complete. 2024-01-16T15:11:38.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.563+0000 7f2ad0200640 1 Processor -- start 2024-01-16T15:11:38.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.564+0000 7f2ad0200640 1 -- start start 2024-01-16T15:11:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.564+0000 7f2ad0200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac80ae5a0 0x7f2ac8153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.564+0000 7f2ad0200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80b3ae0 0x7f2ac815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.564+0000 7f2ad0200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac80cf290 0x7f2ac81f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.564+0000 7f2ad0200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81552d0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.564+0000 7f2ad0200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81f66a0 con 0x7f2ac80cf290 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.564+0000 7f2ad0200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81546c0 con 0x7f2ac80b3ae0 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac80ae5a0 0x7f2ac8153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac80ae5a0 0x7f2ac8153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57262/0 (socket says 172.21.15.141:57262) 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 -- 172.21.15.141:0/3198961280 learned_addr learned my addr 172.21.15.141:0/3198961280 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acce00640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80b3ae0 0x7f2ac815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2ace200640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac80cf290 0x7f2ac81f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 -- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80b3ae0 msgr2=0x7f2ac815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80b3ae0 0x7f2ac815ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 -- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac80cf290 msgr2=0x7f2ac81f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac80cf290 0x7f2ac81f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.565+0000 7f2acd800640 1 -- 172.21.15.141:0/3198961280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab8009400 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.566+0000 7f2acd800640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac80ae5a0 0x7f2ac8153fc0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f2ab802bf70 tx=0x7f2ab8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.566+0000 7f2ac5600640 1 -- 172.21.15.141:0/3198961280 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab803e040 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.566+0000 7f2ac5600640 1 -- 172.21.15.141:0/3198961280 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8039a20 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.566+0000 7f2ac5600640 1 -- 172.21.15.141:0/3198961280 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8039dd0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.566+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac81549a0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.566+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac8154f60 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.567+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80afcd0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.568+0000 7f2ac5600640 1 -- 172.21.15.141:0/3198961280 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f2ab804d3b0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.569+0000 7f2ac5600640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2aa006c1f0 0x7f2aa006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:38.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.569+0000 7f2ac5600640 1 -- 172.21.15.141:0/3198961280 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2ab80aab40 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.570+0000 7f2acce00640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2aa006c1f0 0x7f2aa006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:38.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.570+0000 7f2acce00640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2aa006c1f0 0x7f2aa006e6b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80b0270 tx=0x7f2ab0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:38.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.573+0000 7f2ac5600640 1 -- 172.21.15.141:0/3198961280 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ab80774b0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.776+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2ac80675d0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.878+0000 7f2ac5600640 1 -- 172.21.15.141:0/3198961280 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v96) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2ab807aad0 con 0x7f2ac80ae5a0 2024-01-16T15:11:38.855 INFO:teuthology.orchestra.run.smithi141.stderr: removed all blocklist entries 2024-01-16T15:11:38.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2aa006c1f0 msgr2=0x7f2aa006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2aa006c1f0 0x7f2aa006e6b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80b0270 tx=0x7f2ab0007d30 comp rx=0 tx=0).stop 2024-01-16T15:11:38.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac80ae5a0 msgr2=0x7f2ac8153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac80ae5a0 0x7f2ac8153fc0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f2ab802bf70 tx=0x7f2ab8004730 comp rx=0 tx=0).stop 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 shutdown_connections 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2aa006c1f0 0x7f2aa006e6b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac80cf290 0x7f2ac81f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2ac80b3ae0 0x7f2ac815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 --2- 172.21.15.141:0/3198961280 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2ac80ae5a0 0x7f2ac8153fc0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 >> 172.21.15.141:0/3198961280 conn(0x7f2ac806a800 msgr2=0x7f2ac815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 shutdown_connections 2024-01-16T15:11:38.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:38.882+0000 7f2ad0200640 1 -- 172.21.15.141:0/3198961280 wait complete. 2024-01-16T15:11:38.880 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:11:39.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.092+0000 7f9aa3000640 1 Processor -- start 2024-01-16T15:11:39.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.092+0000 7f9aa3000640 1 -- start start 2024-01-16T15:11:39.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.093+0000 7f9aa3000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a9c150760 0x7f9a9c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9aa3000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f9a9c15d300 0x7f9a9c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9aa3000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c15dc80 con 0x7f9a9c0b2bd0 2024-01-16T15:11:39.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9aa3000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c0b3770 con 0x7f9a9c15d300 2024-01-16T15:11:39.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9aa3000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c071190 con 0x7f9a9c150760 2024-01-16T15:11:39.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9aa1000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9a9c0b2bd0 0x7f9a9c0b3060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52630/0 (socket says 172.21.15.141:52630) 2024-01-16T15:11:39.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9aa1000640 1 -- 172.21.15.141:0/1788871136 learned_addr learned my addr 172.21.15.141:0/1788871136 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57706444 0 0) 0x7f9a9c15dc80 con 0x7f9a9c0b2bd0 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9a9be00640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a9c150760 0x7f9a9c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c003520 con 0x7f9a9c0b2bd0 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.094+0000 7f9a93200640 1 --2- 172.21.15.141:0/1788871136 >> v2:172.21.15.188:3300/0 conn(0x7f9a9c15d300 0x7f9a9c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997818266 0 0) 0x7f9a7c003520 con 0x7f9a9c0b2bd0 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a9c150760 msgr2=0x7f9a9c072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a9c150760 0x7f9a9c072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 >> v2:172.21.15.188:3300/0 conn(0x7f9a9c15d300 msgr2=0x7f9a9c15d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 --2- 172.21.15.141:0/1788871136 >> v2:172.21.15.188:3300/0 conn(0x7f9a9c15d300 0x7f9a9c15d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c071470 con 0x7f9a9c0b2bd0 2024-01-16T15:11:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9a940019f0 con 0x7f9a9c0b2bd0 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 >> v1:172.21.15.141:6789/0 conn(0x7f9a9c0b2bd0 legacy=0x7f9a9c0b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a7c003970 0x7f9a7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a7c0047c0 0x7f9a7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a9c150760 0x7f9a7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0b3770 con 0x7f9a7c003970 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c071190 con 0x7f9a7c0047c0 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c00db90 con 0x7f9a9c150760 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9be00640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a7c003970 0x7f9a7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9be00640 1 -- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a9c150760 msgr2=0x7f9a7c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9be00640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a9c150760 0x7f9a7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.095+0000 7f9a9be00640 1 -- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a7c0047c0 msgr2=0x7f9a7c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.096+0000 7f9a9be00640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a7c0047c0 0x7f9a7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.096+0000 7f9a9be00640 1 -- 172.21.15.141:0/1788871136 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c003520 con 0x7f9a7c003970 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9a9be00640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a7c003970 0x7f9a7c003d90 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009ac0 tx=0x7f9a7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6b24b79ec3260b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:39.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c02c710 con 0x7f9a7c003970 2024-01-16T15:11:39.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c02c870 con 0x7f9a7c003970 2024-01-16T15:11:39.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9a9b400640 1 -- 172.21.15.141:0/1788871136 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c02cc00 con 0x7f9a7c003970 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9aa3000640 1 -- 172.21.15.141:0/1788871136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a7c003970 msgr2=0x7f9a7c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9aa3000640 1 --2- 172.21.15.141:0/1788871136 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a7c003970 0x7f9a7c003d90 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009ac0 tx=0x7f9a7c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9aa1000640 1 -- 172.21.15.141:0/1788871136 reap_dead start 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9aa3000640 1 -- 172.21.15.141:0/1788871136 shutdown_connections 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9aa3000640 1 -- 172.21.15.141:0/1788871136 >> 172.21.15.141:0/1788871136 conn(0x7f9a9c06a800 msgr2=0x7f9a9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.097+0000 7f9aa3000640 1 -- 172.21.15.141:0/1788871136 shutdown_connections 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.098+0000 7f9aa3000640 1 -- 172.21.15.141:0/1788871136 wait complete. 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.099+0000 7f9aa3000640 1 Processor -- start 2024-01-16T15:11:39.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.099+0000 7f9aa3000640 1 -- start start 2024-01-16T15:11:39.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.100+0000 7f9aa3000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c003970 0x7f9a9c14dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.100+0000 7f9aa3000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a9c14e960 0x7f9a9c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.100+0000 7f9aa3000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a9c14e410 0x7f9a9c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.100+0000 7f9aa3000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c15a3d0 con 0x7f9a9c14e960 2024-01-16T15:11:39.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.100+0000 7f9aa3000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0cbe40 con 0x7f9a9c14e410 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.100+0000 7f9aa3000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0ccca0 con 0x7f9a7c003970 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.100+0000 7f9a93200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a9c14e960 0x7f9a9c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a9c14e960 0x7f9a9c06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57278/0 (socket says 172.21.15.141:57278) 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 -- 172.21.15.141:0/2801711305 learned_addr learned my addr 172.21.15.141:0/2801711305 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a9be00640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c003970 0x7f9a9c14dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9aa1000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a9c14e410 0x7f9a9c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 -- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c003970 msgr2=0x7f9a9c14dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c003970 0x7f9a9c14dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 -- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a9c14e410 msgr2=0x7f9a9c06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a9c14e410 0x7f9a9c06c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 -- 172.21.15.141:0/2801711305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c0cc090 con 0x7f9a9c14e960 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9aa1000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a9c14e410 0x7f9a9c06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.101+0000 7f9a93200640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a9c14e960 0x7f9a9c06c470 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400d830 tx=0x7f9a8400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:39.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.102+0000 7f9a98c00640 1 -- 172.21.15.141:0/2801711305 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8400b660 con 0x7f9a9c14e960 2024-01-16T15:11:39.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.102+0000 7f9a98c00640 1 -- 172.21.15.141:0/2801711305 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8400b7c0 con 0x7f9a9c14e960 2024-01-16T15:11:39.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.102+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c0cc370 con 0x7f9a9c14e960 2024-01-16T15:11:39.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.102+0000 7f9a98c00640 1 -- 172.21.15.141:0/2801711305 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8400bb70 con 0x7f9a9c14e960 2024-01-16T15:11:39.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.102+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c0cc930 con 0x7f9a9c14e960 2024-01-16T15:11:39.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.103+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c0675d0 con 0x7f9a9c14e960 2024-01-16T15:11:39.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.104+0000 7f9a98c00640 1 -- 172.21.15.141:0/2801711305 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f9a8400bcd0 con 0x7f9a9c14e960 2024-01-16T15:11:39.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.104+0000 7f9a98c00640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a7406c1f0 0x7f9a7406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.105+0000 7f9a98c00640 1 -- 172.21.15.141:0/2801711305 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9a84088d90 con 0x7f9a9c14e960 2024-01-16T15:11:39.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.105+0000 7f9a9be00640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a7406c1f0 0x7f9a7406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.106+0000 7f9a9be00640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a7406c1f0 0x7f9a7406e6b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c036000 tx=0x7f9a8c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:39.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.109+0000 7f9a98c00640 1 -- 172.21.15.141:0/2801711305 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a84055360 con 0x7f9a9c14e960 2024-01-16T15:11:39.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.368+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9a9c0ccde0 con 0x7f9a9c14e960 2024-01-16T15:11:39.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.369+0000 7f9a98c00640 1 -- 172.21.15.141:0/2801711305 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9a84058980 con 0x7f9a9c14e960 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "auth_dump": [ 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.a", 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ==", 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:11:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.b", 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg==", 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.c", 2024-01-16T15:11:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw==", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.d", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg==", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:11:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.0", 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA==", 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.1", 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg==", 2024-01-16T15:11:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.2", 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g==", 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.3", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w==", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.4", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ==", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.5", 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA==", 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.6", 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ==", 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.7", 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg==", 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.0", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg==", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow r", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.1", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ==", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:11:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.2", 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ==", 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.3", 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA==", 2024-01-16T15:11:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:11:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:11:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.admin", 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAEmaZl0VgDMRAABOWZ0senIXHFvij8FQHiAg==", 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:11:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.x", 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ==", 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.y", 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA==", 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2024-01-16T15:11:39.361 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:11:39.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a7406c1f0 msgr2=0x7f9a7406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a7406c1f0 0x7f9a7406e6b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c036000 tx=0x7f9a8c030000 comp rx=0 tx=0).stop 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a9c14e960 msgr2=0x7f9a9c06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a9c14e960 0x7f9a9c06c470 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400d830 tx=0x7f9a8400dcf0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 shutdown_connections 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a7406c1f0 0x7f9a7406e6b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a9c14e410 0x7f9a9c06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a9c14e960 0x7f9a9c06c470 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 --2- 172.21.15.141:0/2801711305 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9a7c003970 0x7f9a9c14dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 >> 172.21.15.141:0/2801711305 conn(0x7f9a9c06a800 msgr2=0x7f9a9c0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.372+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 shutdown_connections 2024-01-16T15:11:39.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.373+0000 7f9aa3000640 1 -- 172.21.15.141:0/2801711305 wait complete. 2024-01-16T15:11:39.369 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:39.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.603+0000 7f0a52c00640 1 Processor -- start 2024-01-16T15:11:39.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.603+0000 7f0a52c00640 1 -- start start 2024-01-16T15:11:39.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.604+0000 7f0a52c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 0x7f0a4c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.604+0000 7f0a52c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0a4c15e1e0 0x7f0a4c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.604+0000 7f0a52c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4c0b1ee0 con 0x7f0a4c143ae0 2024-01-16T15:11:39.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a52c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4c0b2780 con 0x7f0a4c15e1e0 2024-01-16T15:11:39.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a52c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c15e990 con 0x7f0a4c0cf9a0 2024-01-16T15:11:39.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a4be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0a4c143ae0 0x7f0a4c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52634/0 (socket says 172.21.15.141:52634) 2024-01-16T15:11:39.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a4be00640 1 -- 172.21.15.141:0/3822239509 learned_addr learned my addr 172.21.15.141:0/3822239509 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303171627 0 0) 0x7f0a4c0b1ee0 con 0x7f0a4c143ae0 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a43200640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 0x7f0a4c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2c005730 con 0x7f0a4c143ae0 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.605+0000 7f0a50c00640 1 --2- 172.21.15.141:0/3822239509 >> v2:172.21.15.188:3300/0 conn(0x7f0a4c15e1e0 0x7f0a4c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1367148980 0 0) 0x7f0a2c005730 con 0x7f0a4c143ae0 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 msgr2=0x7f0a4c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 0x7f0a4c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 >> v2:172.21.15.188:3300/0 conn(0x7f0a4c15e1e0 msgr2=0x7f0a4c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 --2- 172.21.15.141:0/3822239509 >> v2:172.21.15.188:3300/0 conn(0x7f0a4c15e1e0 0x7f0a4c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c0b2130 con 0x7f0a4c143ae0 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0a3c0019f0 con 0x7f0a4c143ae0 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 >> v1:172.21.15.141:6789/0 conn(0x7f0a4c143ae0 legacy=0x7f0a4c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 0x7f0a2c005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c0061b0 0x7f0a2c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c006f10 0x7f0a2c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0b2780 con 0x7f0a2c0061b0 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c15e990 con 0x7f0a2c006f10 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c00dc90 con 0x7f0a4c0cf9a0 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a43200640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c0061b0 0x7f0a2c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.606+0000 7f0a4be00640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 0x7f0a2c005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.607+0000 7f0a43200640 1 -- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 msgr2=0x7f0a2c005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.607+0000 7f0a50c00640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c006f10 0x7f0a2c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.607+0000 7f0a43200640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a4c0cf9a0 0x7f0a2c005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.607+0000 7f0a43200640 1 -- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c006f10 msgr2=0x7f0a2c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.607+0000 7f0a43200640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c006f10 0x7f0a2c007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.607+0000 7f0a43200640 1 -- 172.21.15.141:0/3822239509 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c005730 con 0x7f0a2c0061b0 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.607+0000 7f0a50c00640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a2c006f10 0x7f0a2c007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.608+0000 7f0a43200640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c0061b0 0x7f0a2c0069d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400ba30 tx=0x7f0a2c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2950acbe39f4def server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.608+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34032640 con 0x7f0a2c0061b0 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.608+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a340327a0 con 0x7f0a2c0061b0 2024-01-16T15:11:39.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.608+0000 7f0a4b400640 1 -- 172.21.15.141:0/3822239509 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34032b30 con 0x7f0a2c0061b0 2024-01-16T15:11:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.609+0000 7f0a52c00640 1 -- 172.21.15.141:0/3822239509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c0061b0 msgr2=0x7f0a2c0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.609+0000 7f0a52c00640 1 --2- 172.21.15.141:0/3822239509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a2c0061b0 0x7f0a2c0069d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400ba30 tx=0x7f0a2c007a00 comp rx=0 tx=0).stop 2024-01-16T15:11:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.609+0000 7f0a50c00640 1 -- 172.21.15.141:0/3822239509 reap_dead start 2024-01-16T15:11:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.609+0000 7f0a52c00640 1 -- 172.21.15.141:0/3822239509 shutdown_connections 2024-01-16T15:11:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.609+0000 7f0a52c00640 1 -- 172.21.15.141:0/3822239509 >> 172.21.15.141:0/3822239509 conn(0x7f0a4c06a800 msgr2=0x7f0a4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.609+0000 7f0a52c00640 1 -- 172.21.15.141:0/3822239509 shutdown_connections 2024-01-16T15:11:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.609+0000 7f0a52c00640 1 -- 172.21.15.141:0/3822239509 wait complete. 2024-01-16T15:11:39.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.611+0000 7f0a52c00640 1 Processor -- start 2024-01-16T15:11:39.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.611+0000 7f0a52c00640 1 -- start start 2024-01-16T15:11:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a52c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0061b0 0x7f0a4c1f6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a52c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a4c1f35f0 0x7f0a4c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a52c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a4c0cd6f0 0x7f0a4c13d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a52c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c1f3a70 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a52c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c15aa20 con 0x7f0a4c0cd6f0 2024-01-16T15:11:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a52c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c15aca0 con 0x7f0a2c0061b0 2024-01-16T15:11:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a4c1f35f0 0x7f0a4c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a4c1f35f0 0x7f0a4c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57292/0 (socket says 172.21.15.141:57292) 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 -- 172.21.15.141:0/3738710388 learned_addr learned my addr 172.21.15.141:0/3738710388 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 -- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0061b0 msgr2=0x7f0a4c1f6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0061b0 0x7f0a4c1f6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 -- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a4c0cd6f0 msgr2=0x7f0a4c13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a4c0cd6f0 0x7f0a4c13d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.612+0000 7f0a43200640 1 -- 172.21.15.141:0/3738710388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c1f6170 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.613+0000 7f0a43200640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a4c1f35f0 0x7f0a4c15a4e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0a34035040 tx=0x7f0a34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.613+0000 7f0a48c00640 1 -- 172.21.15.141:0/3738710388 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34004220 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.613+0000 7f0a48c00640 1 -- 172.21.15.141:0/3738710388 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a34004380 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.613+0000 7f0a48c00640 1 -- 172.21.15.141:0/3738710388 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3403ba90 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.613+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4c1f6450 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.613+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4c15af80 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.614+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4c0675d0 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.615+0000 7f0a48c00640 1 -- 172.21.15.141:0/3738710388 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f0a3403bbf0 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.616+0000 7f0a48c00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a2406c1f0 0x7f0a2406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:39.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.616+0000 7f0a48c00640 1 -- 172.21.15.141:0/3738710388 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0a340aaf10 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.617+0000 7f0a4be00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a2406c1f0 0x7f0a2406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:39.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.618+0000 7f0a4be00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a2406c1f0 0x7f0a2406e6b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c001330 tx=0x7f0a3c001830 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:39.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.621+0000 7f0a48c00640 1 -- 172.21.15.141:0/3738710388 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a340777d0 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.814+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a4c0675d0 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.816+0000 7f0a48c00640 1 -- 172.21.15.141:0/3738710388 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+12906 (secure 0 0 0) 0x7f0a34006c30 con 0x7f0a4c1f35f0 2024-01-16T15:11:39.793 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:39.793 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":98,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:11:38.891500+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:39.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a2406c1f0 msgr2=0x7f0a2406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a2406c1f0 0x7f0a2406e6b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c001330 tx=0x7f0a3c001830 comp rx=0 tx=0).stop 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a4c1f35f0 msgr2=0x7f0a4c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a4c1f35f0 0x7f0a4c15a4e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0a34035040 tx=0x7f0a34004730 comp rx=0 tx=0).stop 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 shutdown_connections 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a2406c1f0 0x7f0a2406e6b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a4c0cd6f0 0x7f0a4c13d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a4c1f35f0 0x7f0a4c15a4e0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 --2- 172.21.15.141:0/3738710388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0a2c0061b0 0x7f0a4c1f6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:39.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.818+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 >> 172.21.15.141:0/3738710388 conn(0x7f0a4c06a800 msgr2=0x7f0a4c070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:39.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.819+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 shutdown_connections 2024-01-16T15:11:39.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:39.819+0000 7f0a52c00640 1 -- 172.21.15.141:0/3738710388 wait complete. 2024-01-16T15:11:39.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:11:39.816 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:11:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.053+0000 7f14d0e00640 1 Processor -- start 2024-01-16T15:11:40.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.053+0000 7f14d0e00640 1 -- start start 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.054+0000 7f14d0e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14cc13df80 0x7f14cc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.054+0000 7f14d0e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc06c470 con 0x7f14cc13e360 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.054+0000 7f14d0e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc06c750 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.054+0000 7f14d0e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14cc06ca30 con 0x7f14cc13df80 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14cc13df80 0x7f14cc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f14cc13df80 0x7f14cc15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57296/0 (socket says 172.21.15.141:57296) 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c9e00640 1 -- 172.21.15.141:0/4119721568 learned_addr learned my addr 172.21.15.141:0/4119721568 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:40.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c3e00640 1 --1- 172.21.15.141:0/4119721568 >> v1:172.21.15.141:6789/0 conn(0x7f14cc13e360 0x7f14cc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52638/0 (socket says 172.21.15.141:52638) 2024-01-16T15:11:40.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c9e00640 1 -- 172.21.15.141:0/4119721568 >> v1:172.21.15.188:6789/0 conn(0x7f14cc0cf9a0 legacy=0x7f14cc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:40.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c9e00640 1 -- 172.21.15.141:0/4119721568 >> v1:172.21.15.141:6789/0 conn(0x7f14cc13e360 legacy=0x7f14cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c9e00640 1 -- 172.21.15.141:0/4119721568 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc06cd10 con 0x7f14cc13df80 2024-01-16T15:11:40.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.055+0000 7f14c9400640 1 -- 172.21.15.141:0/4119721568 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246756449 0 0) 0x7f14cc06c470 con 0x7f14cc13e360 2024-01-16T15:11:40.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.056+0000 7f14c9e00640 1 --2- 172.21.15.141:0/4119721568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc13df80 0x7f14cc15bee0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009870 tx=0x7f14bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a24ac7612dafb0c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14c9400640 1 -- 172.21.15.141:0/4119721568 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc030860 con 0x7f14cc13df80 2024-01-16T15:11:40.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14c9400640 1 -- 172.21.15.141:0/4119721568 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc0309c0 con 0x7f14cc13df80 2024-01-16T15:11:40.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14c9400640 1 -- 172.21.15.141:0/4119721568 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc030d50 con 0x7f14cc13df80 2024-01-16T15:11:40.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14d0e00640 1 -- 172.21.15.141:0/4119721568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc13df80 msgr2=0x7f14cc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14d0e00640 1 --2- 172.21.15.141:0/4119721568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc13df80 0x7f14cc15bee0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009870 tx=0x7f14bc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14d0e00640 1 -- 172.21.15.141:0/4119721568 shutdown_connections 2024-01-16T15:11:40.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14d0e00640 1 --2- 172.21.15.141:0/4119721568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc13df80 0x7f14cc15bee0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14d0e00640 1 -- 172.21.15.141:0/4119721568 >> 172.21.15.141:0/4119721568 conn(0x7f14cc06a800 msgr2=0x7f14cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:40.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14d0e00640 1 -- 172.21.15.141:0/4119721568 shutdown_connections 2024-01-16T15:11:40.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.057+0000 7f14d0e00640 1 -- 172.21.15.141:0/4119721568 wait complete. 2024-01-16T15:11:40.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.059+0000 7f14d0e00640 1 Processor -- start 2024-01-16T15:11:40.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.059+0000 7f14d0e00640 1 -- start start 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14d0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc0cf9a0 0x7f14cc153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14d0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc13df80 0x7f14cc158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14d0e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14cc13e360 0x7f14cc157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14d0e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc1562a0 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14d0e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc156580 con 0x7f14cc13df80 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14d0e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc156860 con 0x7f14cc13e360 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14c9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc0cf9a0 0x7f14cc153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14c9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc0cf9a0 0x7f14cc153170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57300/0 (socket says 172.21.15.141:57300) 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.060+0000 7f14c9e00640 1 -- 172.21.15.141:0/1021453879 learned_addr learned my addr 172.21.15.141:0/1021453879 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:40.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c3e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc13df80 0x7f14cc158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14ca800640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14cc13e360 0x7f14cc157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c9e00640 1 -- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14cc13e360 msgr2=0x7f14cc157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c9e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14cc13e360 0x7f14cc157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c9e00640 1 -- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc13df80 msgr2=0x7f14cc158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c9e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc13df80 0x7f14cc158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c9e00640 1 -- 172.21.15.141:0/1021453879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc0094d0 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c9e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc0cf9a0 0x7f14cc153170 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009870 tx=0x7f14bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.061+0000 7f14c2000640 1 -- 172.21.15.141:0/1021453879 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc039d60 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.062+0000 7f14c2000640 1 -- 172.21.15.141:0/1021453879 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc043400 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.062+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc156b40 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.062+0000 7f14c2000640 1 -- 172.21.15.141:0/1021453879 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0437b0 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.062+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc1536b0 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.063+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc157060 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.064+0000 7f14c2000640 1 -- 172.21.15.141:0/1021453879 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f14bc043910 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.065+0000 7f14c2000640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14a406c1f0 0x7f14a406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.065+0000 7f14c2000640 1 -- 172.21.15.141:0/1021453879 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f14bc03e070 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.066+0000 7f14c3e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14a406c1f0 0x7f14a406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.067+0000 7f14c3e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14a406c1f0 0x7f14a406e6b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f14cc159270 tx=0x7f14b4007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.070+0000 7f14c2000640 1 -- 172.21.15.141:0/1021453879 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14cc157060 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.262+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14cc157060 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.263+0000 7f14c2000640 1 -- 172.21.15.141:0/1021453879 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14cc157060 con 0x7f14cc0cf9a0 2024-01-16T15:11:40.243 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14a406c1f0 msgr2=0x7f14a406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14a406c1f0 0x7f14a406e6b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f14cc159270 tx=0x7f14b4007b50 comp rx=0 tx=0).stop 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc0cf9a0 msgr2=0x7f14cc153170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc0cf9a0 0x7f14cc153170 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009870 tx=0x7f14bc004730 comp rx=0 tx=0).stop 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 shutdown_connections 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f14a406c1f0 0x7f14a406e6b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f14cc13e360 0x7f14cc157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc13df80 0x7f14cc158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 --2- 172.21.15.141:0/1021453879 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14cc0cf9a0 0x7f14cc153170 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 >> 172.21.15.141:0/1021453879 conn(0x7f14cc06a800 msgr2=0x7f14cc153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:40.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 shutdown_connections 2024-01-16T15:11:40.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.266+0000 7f14d0e00640 1 -- 172.21.15.141:0/1021453879 wait complete. 2024-01-16T15:11:40.262 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:11:40.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.492+0000 7fdfda600640 1 Processor -- start 2024-01-16T15:11:40.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.493+0000 7fdfda600640 1 -- start start 2024-01-16T15:11:40.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.494+0000 7fdfda600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdfd415d240 0x7fdfd40b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.494+0000 7fdfda600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd406d7d0 0x7fdfd40aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.494+0000 7fdfda600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd40b1d20 con 0x7fdfd40b27c0 2024-01-16T15:11:40.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.494+0000 7fdfda600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd40b38d0 con 0x7fdfd415d240 2024-01-16T15:11:40.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.494+0000 7fdfda600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40b1fc0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.495+0000 7fdfd2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd406d7d0 0x7fdfd40aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.495+0000 7fdfd2a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd406d7d0 0x7fdfd40aedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:60240/0 (socket says 172.21.15.141:60240) 2024-01-16T15:11:40.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.495+0000 7fdfd2a00640 1 -- 172.21.15.141:0/441678230 learned_addr learned my addr 172.21.15.141:0/441678230 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:40.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.495+0000 7fdfd3400640 1 --2- 172.21.15.141:0/441678230 >> v2:172.21.15.188:3300/0 conn(0x7fdfd415d240 0x7fdfd40b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.496+0000 7fdfd2a00640 1 -- 172.21.15.141:0/441678230 >> v2:172.21.15.188:3300/0 conn(0x7fdfd415d240 msgr2=0x7fdfd40b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:40.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.496+0000 7fdfd2a00640 1 --2- 172.21.15.141:0/441678230 >> v2:172.21.15.188:3300/0 conn(0x7fdfd415d240 0x7fdfd40b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.496+0000 7fdfd2a00640 1 -- 172.21.15.141:0/441678230 >> v1:172.21.15.188:6789/0 conn(0x7fdfd40b27c0 legacy=0x7fdfd4071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.496+0000 7fdfd2a00640 1 -- 172.21.15.141:0/441678230 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd40b22a0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.497+0000 7fdfd2a00640 1 --2- 172.21.15.141:0/441678230 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd406d7d0 0x7fdfd40aedb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009e50 tx=0x7fdfc402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852e124c12b0821f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfd2000640 1 -- 172.21.15.141:0/441678230 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc402ca70 con 0x7fdfd406d7d0 2024-01-16T15:11:40.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfd2000640 1 -- 172.21.15.141:0/441678230 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc402cbd0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfd2000640 1 -- 172.21.15.141:0/441678230 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40343f0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfda600640 1 -- 172.21.15.141:0/441678230 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd406d7d0 msgr2=0x7fdfd40aedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfda600640 1 --2- 172.21.15.141:0/441678230 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd406d7d0 0x7fdfd40aedb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009e50 tx=0x7fdfc402bed0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfda600640 1 -- 172.21.15.141:0/441678230 shutdown_connections 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfda600640 1 --2- 172.21.15.141:0/441678230 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd406d7d0 0x7fdfd40aedb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfda600640 1 --2- 172.21.15.141:0/441678230 >> v2:172.21.15.188:3300/0 conn(0x7fdfd415d240 0x7fdfd40b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfda600640 1 -- 172.21.15.141:0/441678230 >> 172.21.15.141:0/441678230 conn(0x7fdfd406b4f0 msgr2=0x7fdfd406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.498+0000 7fdfda600640 1 -- 172.21.15.141:0/441678230 shutdown_connections 2024-01-16T15:11:40.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.499+0000 7fdfda600640 1 -- 172.21.15.141:0/441678230 wait complete. 2024-01-16T15:11:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.500+0000 7fdfda600640 1 Processor -- start 2024-01-16T15:11:40.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.500+0000 7fdfda600640 1 -- start start 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.501+0000 7fdfda600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdfd406d7d0 0x7fdfd40ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.501+0000 7fdfda600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd40b27c0 0x7fdfd40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.501+0000 7fdfda600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdfd415d240 0x7fdfd414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.501+0000 7fdfda600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd41594e0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.501+0000 7fdfda600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40ad120 con 0x7fdfd415d240 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.501+0000 7fdfda600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40ad400 con 0x7fdfd40b27c0 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdfd406d7d0 0x7fdfd40ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdfd406d7d0 0x7fdfd40ae510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57316/0 (socket says 172.21.15.141:57316) 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 -- 172.21.15.141:0/1042032282 learned_addr learned my addr 172.21.15.141:0/1042032282 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd2a00640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd40b27c0 0x7fdfd40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3e00640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdfd415d240 0x7fdfd414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 -- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd40b27c0 msgr2=0x7fdfd40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd40b27c0 0x7fdfd40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 -- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdfd415d240 msgr2=0x7fdfd414b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdfd415d240 0x7fdfd414b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 -- 172.21.15.141:0/1042032282 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4009b30 con 0x7fdfd406d7d0 2024-01-16T15:11:40.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.502+0000 7fdfd3400640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdfd406d7d0 0x7fdfd40ae510 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00e9b0 tx=0x7fdfbc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.503+0000 7fdfcb400640 1 -- 172.21.15.141:0/1042032282 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc009670 con 0x7fdfd406d7d0 2024-01-16T15:11:40.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.503+0000 7fdfcb400640 1 -- 172.21.15.141:0/1042032282 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc00c9a0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.503+0000 7fdfcb400640 1 -- 172.21.15.141:0/1042032282 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc00cd50 con 0x7fdfd406d7d0 2024-01-16T15:11:40.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.503+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd40ad6e0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.503+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd40ace10 con 0x7fdfd406d7d0 2024-01-16T15:11:40.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.504+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd40675d0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.505+0000 7fdfcb400640 1 -- 172.21.15.141:0/1042032282 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fdfbc013470 con 0x7fdfd406d7d0 2024-01-16T15:11:40.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.506+0000 7fdfcb400640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdfac06c1f0 0x7fdfac06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.506+0000 7fdfcb400640 1 -- 172.21.15.141:0/1042032282 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdfbc0888d0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.506+0000 7fdfd2a00640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdfac06c1f0 0x7fdfac06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.508+0000 7fdfd2a00640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdfac06c1f0 0x7fdfac06e6b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4035000 tx=0x7fdfc4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.510+0000 7fdfcb400640 1 -- 172.21.15.141:0/1042032282 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfbc0551f0 con 0x7fdfd406d7d0 2024-01-16T15:11:40.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.703+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdfd406d570 con 0x7fdfd406d7d0 2024-01-16T15:11:40.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.703+0000 7fdfcb400640 1 -- 172.21.15.141:0/1042032282 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v98) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdfbc058810 con 0x7fdfd406d7d0 2024-01-16T15:11:40.681 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:40.681 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:11:40.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:40.681 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T15:11:40.681 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T15:11:40.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:40.681 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:11:40.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdfac06c1f0 msgr2=0x7fdfac06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdfac06c1f0 0x7fdfac06e6b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4035000 tx=0x7fdfc4030000 comp rx=0 tx=0).stop 2024-01-16T15:11:40.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdfd406d7d0 msgr2=0x7fdfd40ae510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdfd406d7d0 0x7fdfd40ae510 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00e9b0 tx=0x7fdfbc00ee70 comp rx=0 tx=0).stop 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 shutdown_connections 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdfac06c1f0 0x7fdfac06e6b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdfd415d240 0x7fdfd414b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdfd40b27c0 0x7fdfd40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 --2- 172.21.15.141:0/1042032282 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdfd406d7d0 0x7fdfd40ae510 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 >> 172.21.15.141:0/1042032282 conn(0x7fdfd406b4f0 msgr2=0x7fdfd415c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 shutdown_connections 2024-01-16T15:11:40.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.706+0000 7fdfda600640 1 -- 172.21.15.141:0/1042032282 wait complete. 2024-01-16T15:11:40.702 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:11:40.703 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:11:40.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.940+0000 7fe87b400640 1 Processor -- start 2024-01-16T15:11:40.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.940+0000 7fe87b400640 1 -- start start 2024-01-16T15:11:40.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe87b400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe8740cfcf0 0x7fe874160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe87b400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d3910 0x7fe874160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe87b400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe8740d00d0 0x7fe8740b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe87b400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8740ceff0 con 0x7fe8740d00d0 2024-01-16T15:11:40.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe87b400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8740cf410 con 0x7fe8740cfcf0 2024-01-16T15:11:40.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe87b400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740d4160 con 0x7fe8740d3910 2024-01-16T15:11:40.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe879400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe8740d00d0 0x7fe8740b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe879400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe8740d00d0 0x7fe8740b2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57320/0 (socket says 172.21.15.141:57320) 2024-01-16T15:11:40.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.942+0000 7fe879400640 1 -- 172.21.15.141:0/146472219 learned_addr learned my addr 172.21.15.141:0/146472219 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe873e00640 1 --2- 172.21.15.141:0/146472219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d3910 0x7fe874160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe878a00640 1 --2- 172.21.15.141:0/146472219 >> v2:172.21.15.188:3300/0 conn(0x7fe8740cfcf0 0x7fe874160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe879400640 1 -- 172.21.15.141:0/146472219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d3910 msgr2=0x7fe874160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe879400640 1 --2- 172.21.15.141:0/146472219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d3910 0x7fe874160c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe879400640 1 -- 172.21.15.141:0/146472219 >> v2:172.21.15.188:3300/0 conn(0x7fe8740cfcf0 msgr2=0x7fe874160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe879400640 1 --2- 172.21.15.141:0/146472219 >> v2:172.21.15.188:3300/0 conn(0x7fe8740cfcf0 0x7fe874160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe879400640 1 -- 172.21.15.141:0/146472219 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740b2420 con 0x7fe8740d00d0 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.943+0000 7fe878a00640 1 --2- 172.21.15.141:0/146472219 >> v2:172.21.15.188:3300/0 conn(0x7fe8740cfcf0 0x7fe874160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:40.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.944+0000 7fe879400640 1 --2- 172.21.15.141:0/146472219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d00d0 0x7fe8740b2a50 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe8740b4cc0 tx=0x7fe86402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a482e01287d01e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.944+0000 7fe873400640 1 -- 172.21.15.141:0/146472219 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640317c0 con 0x7fe8740d00d0 2024-01-16T15:11:40.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.944+0000 7fe873400640 1 -- 172.21.15.141:0/146472219 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe864031920 con 0x7fe8740d00d0 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.944+0000 7fe873400640 1 -- 172.21.15.141:0/146472219 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864031cb0 con 0x7fe8740d00d0 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 -- 172.21.15.141:0/146472219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d00d0 msgr2=0x7fe8740b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 --2- 172.21.15.141:0/146472219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d00d0 0x7fe8740b2a50 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe8740b4cc0 tx=0x7fe86402b860 comp rx=0 tx=0).stop 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 -- 172.21.15.141:0/146472219 shutdown_connections 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 --2- 172.21.15.141:0/146472219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d00d0 0x7fe8740b2a50 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 --2- 172.21.15.141:0/146472219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d3910 0x7fe874160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 --2- 172.21.15.141:0/146472219 >> v2:172.21.15.188:3300/0 conn(0x7fe8740cfcf0 0x7fe874160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 -- 172.21.15.141:0/146472219 >> 172.21.15.141:0/146472219 conn(0x7fe87409c530 msgr2=0x7fe8740d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 -- 172.21.15.141:0/146472219 shutdown_connections 2024-01-16T15:11:40.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.945+0000 7fe87b400640 1 -- 172.21.15.141:0/146472219 wait complete. 2024-01-16T15:11:40.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.947+0000 7fe87b400640 1 Processor -- start 2024-01-16T15:11:40.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.947+0000 7fe87b400640 1 -- start start 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe87b400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8740cfcf0 0x7fe8740c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe87b400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d00d0 0x7fe8740c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe87b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d3910 0x7fe8741f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe87b400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740b7800 con 0x7fe8740d3910 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe87b400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740c9b60 con 0x7fe8740cfcf0 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe87b400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740c9e40 con 0x7fe8740d00d0 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe879400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d3910 0x7fe8741f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe879400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d3910 0x7fe8741f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57336/0 (socket says 172.21.15.141:57336) 2024-01-16T15:11:40.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.948+0000 7fe879400640 1 -- 172.21.15.141:0/1543838514 learned_addr learned my addr 172.21.15.141:0/1543838514 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:40.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe873e00640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d00d0 0x7fe8740c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe878a00640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8740cfcf0 0x7fe8740c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe879400640 1 -- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d00d0 msgr2=0x7fe8740c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe879400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d00d0 0x7fe8740c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe879400640 1 -- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8740cfcf0 msgr2=0x7fe8740c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe879400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8740cfcf0 0x7fe8740c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe879400640 1 -- 172.21.15.141:0/1543838514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8640094d0 con 0x7fe8740d3910 2024-01-16T15:11:40.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe879400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d3910 0x7fe8741f6c40 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fe8640090b0 tx=0x7fe8640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.949+0000 7fe870c00640 1 -- 172.21.15.141:0/1543838514 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864032920 con 0x7fe8740d3910 2024-01-16T15:11:40.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.950+0000 7fe870c00640 1 -- 172.21.15.141:0/1543838514 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe864032ac0 con 0x7fe8740d3910 2024-01-16T15:11:40.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.950+0000 7fe870c00640 1 -- 172.21.15.141:0/1543838514 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864032e70 con 0x7fe8740d3910 2024-01-16T15:11:40.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.950+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8740b7140 con 0x7fe8740d3910 2024-01-16T15:11:40.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.950+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8740c74a0 con 0x7fe8740d3910 2024-01-16T15:11:40.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.951+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8740675d0 con 0x7fe8740d3910 2024-01-16T15:11:40.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.952+0000 7fe870c00640 1 -- 172.21.15.141:0/1543838514 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fe86403b3f0 con 0x7fe8740d3910 2024-01-16T15:11:40.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.952+0000 7fe870c00640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe84c06c1f0 0x7fe84c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.953+0000 7fe870c00640 1 -- 172.21.15.141:0/1543838514 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe8640aab90 con 0x7fe8740d3910 2024-01-16T15:11:40.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.953+0000 7fe878a00640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe84c06c1f0 0x7fe84c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.954+0000 7fe878a00640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe84c06c1f0 0x7fe84c06e6b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe874066810 tx=0x7fe86c007550 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:40.957+0000 7fe870c00640 1 -- 172.21.15.141:0/1543838514 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe864077450 con 0x7fe8740d3910 2024-01-16T15:11:41.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.150+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe8740b4560 con 0x7fe8740d3910 2024-01-16T15:11:41.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.909+0000 7fe870c00640 1 -- 172.21.15.141:0/1543838514 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v99) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe86407aa70 con 0x7fe8740d3910 2024-01-16T15:11:41.887 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T15:11:41.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.912+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe84c06c1f0 msgr2=0x7fe84c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.912+0000 7fe87b400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe84c06c1f0 0x7fe84c06e6b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe874066810 tx=0x7fe86c007550 comp rx=0 tx=0).stop 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d3910 msgr2=0x7fe8741f6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d3910 0x7fe8741f6c40 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fe8640090b0 tx=0x7fe8640046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 shutdown_connections 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe84c06c1f0 0x7fe84c06e6b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8740d3910 0x7fe8741f6c40 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8740d00d0 0x7fe8740c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 --2- 172.21.15.141:0/1543838514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8740cfcf0 0x7fe8740c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 >> 172.21.15.141:0/1543838514 conn(0x7fe87409c530 msgr2=0x7fe87406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 shutdown_connections 2024-01-16T15:11:41.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:41.913+0000 7fe87b400640 1 -- 172.21.15.141:0/1543838514 wait complete. 2024-01-16T15:11:41.916 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:11:42.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.140+0000 7f64d0600640 1 Processor -- start 2024-01-16T15:11:42.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.141+0000 7f64d0600640 1 -- start start 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.142+0000 7f64d0600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c806a7a0 0x7f64c806ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.142+0000 7f64d0600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64c8070b10 0x7f64c8070f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.142+0000 7f64d0600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f64c8160f70 0x7f64c8161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.142+0000 7f64d0600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c80714b0 con 0x7f64c8070b10 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.142+0000 7f64d0600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c80ced80 con 0x7f64c8160f70 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.142+0000 7f64d0600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c80cf060 con 0x7f64c806a7a0 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64c8070b10 0x7f64c8070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f64c8070b10 0x7f64c8070f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57340/0 (socket says 172.21.15.141:57340) 2024-01-16T15:11:42.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 -- 172.21.15.141:0/2078548079 learned_addr learned my addr 172.21.15.141:0/2078548079 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64ce600640 1 --2- 172.21.15.141:0/2078548079 >> v2:172.21.15.188:3300/0 conn(0x7f64c8160f70 0x7f64c8161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cdc00640 1 --2- 172.21.15.141:0/2078548079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c806a7a0 0x7f64c806ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 -- 172.21.15.141:0/2078548079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c806a7a0 msgr2=0x7f64c806ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:42.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 --2- 172.21.15.141:0/2078548079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c806a7a0 0x7f64c806ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 -- 172.21.15.141:0/2078548079 >> v2:172.21.15.188:3300/0 conn(0x7f64c8160f70 msgr2=0x7f64c8161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:42.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 --2- 172.21.15.141:0/2078548079 >> v2:172.21.15.188:3300/0 conn(0x7f64c8160f70 0x7f64c8161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.143+0000 7f64cd200640 1 -- 172.21.15.141:0/2078548079 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c80cd720 con 0x7f64c8070b10 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.144+0000 7f64cd200640 1 --2- 172.21.15.141:0/2078548079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c8070b10 0x7f64c8070f70 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f64c806b940 tx=0x7f64b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f002b43ec3081f8a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.145+0000 7f64c7e00640 1 -- 172.21.15.141:0/2078548079 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b002ca80 con 0x7f64c8070b10 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.145+0000 7f64c7e00640 1 -- 172.21.15.141:0/2078548079 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b002cbe0 con 0x7f64c8070b10 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.145+0000 7f64c7e00640 1 -- 172.21.15.141:0/2078548079 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b00343f0 con 0x7f64c8070b10 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 -- 172.21.15.141:0/2078548079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c8070b10 msgr2=0x7f64c8070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 --2- 172.21.15.141:0/2078548079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c8070b10 0x7f64c8070f70 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f64c806b940 tx=0x7f64b002bee0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 -- 172.21.15.141:0/2078548079 shutdown_connections 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 --2- 172.21.15.141:0/2078548079 >> v2:172.21.15.188:3300/0 conn(0x7f64c8160f70 0x7f64c8161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 --2- 172.21.15.141:0/2078548079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c8070b10 0x7f64c8070f70 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 --2- 172.21.15.141:0/2078548079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c806a7a0 0x7f64c806ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 -- 172.21.15.141:0/2078548079 >> 172.21.15.141:0/2078548079 conn(0x7f64c80b1a60 msgr2=0x7f64c80cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:42.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 -- 172.21.15.141:0/2078548079 shutdown_connections 2024-01-16T15:11:42.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.146+0000 7f64d0600640 1 -- 172.21.15.141:0/2078548079 wait complete. 2024-01-16T15:11:42.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.148+0000 7f64d0600640 1 Processor -- start 2024-01-16T15:11:42.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.148+0000 7f64d0600640 1 -- start start 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64d0600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c806a7a0 0x7f64c80b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64d0600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64c8070b10 0x7f64c8155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64d0600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c8160f70 0x7f64c815cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64d0600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c80b2450 con 0x7f64c806a7a0 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64d0600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c815d2e0 con 0x7f64c8070b10 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64d0600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c815d560 con 0x7f64c8160f70 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c806a7a0 0x7f64c80b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c806a7a0 0x7f64c80b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57350/0 (socket says 172.21.15.141:57350) 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 -- 172.21.15.141:0/2951950694 learned_addr learned my addr 172.21.15.141:0/2951950694 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 -- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c8160f70 msgr2=0x7f64c815cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c8160f70 0x7f64c815cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 -- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64c8070b10 msgr2=0x7f64c8155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64c8070b10 0x7f64c8155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.149+0000 7f64cdc00640 1 -- 172.21.15.141:0/2951950694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0009b30 con 0x7f64c806a7a0 2024-01-16T15:11:42.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.150+0000 7f64cdc00640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c806a7a0 0x7f64c80b4580 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f64b800d5f0 tx=0x7f64b800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.150+0000 7f64c5600640 1 -- 172.21.15.141:0/2951950694 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b80125f0 con 0x7f64c806a7a0 2024-01-16T15:11:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.150+0000 7f64c5600640 1 -- 172.21.15.141:0/2951950694 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b8012750 con 0x7f64c806a7a0 2024-01-16T15:11:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.150+0000 7f64c5600640 1 -- 172.21.15.141:0/2951950694 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8012b00 con 0x7f64c806a7a0 2024-01-16T15:11:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.150+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c815d840 con 0x7f64c806a7a0 2024-01-16T15:11:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.150+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c80b2980 con 0x7f64c806a7a0 2024-01-16T15:11:42.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.151+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c8067d70 con 0x7f64c806a7a0 2024-01-16T15:11:42.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.153+0000 7f64c5600640 1 -- 172.21.15.141:0/2951950694 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f64b8012c60 con 0x7f64c806a7a0 2024-01-16T15:11:42.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.154+0000 7f64c5600640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64a006c1f0 0x7f64a006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:42.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.154+0000 7f64c5600640 1 -- 172.21.15.141:0/2951950694 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f64b8089990 con 0x7f64c806a7a0 2024-01-16T15:11:42.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.155+0000 7f64cd200640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64a006c1f0 0x7f64a006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:42.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.156+0000 7f64cd200640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64a006c1f0 0x7f64a006e6b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f64b0035000 tx=0x7f64b002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:42.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.158+0000 7f64c5600640 1 -- 172.21.15.141:0/2951950694 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64b8055fa0 con 0x7f64c806a7a0 2024-01-16T15:11:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.349+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f64c8067670 con 0x7f64c806a7a0 2024-01-16T15:11:42.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.915+0000 7f64c5600640 1 -- 172.21.15.141:0/2951950694 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v100) v1 ==== 170+0+0 (secure 0 0 0) 0x7f64b80595c0 con 0x7f64c806a7a0 2024-01-16T15:11:42.892 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T15:11:42.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64a006c1f0 msgr2=0x7f64a006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:42.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64a006c1f0 0x7f64a006e6b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f64b0035000 tx=0x7f64b002e000 comp rx=0 tx=0).stop 2024-01-16T15:11:42.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c806a7a0 msgr2=0x7f64c80b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:42.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c806a7a0 0x7f64c80b4580 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f64b800d5f0 tx=0x7f64b800dab0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 shutdown_connections 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f64a006c1f0 0x7f64a006e6b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f64c8160f70 0x7f64c815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64c8070b10 0x7f64c8155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 --2- 172.21.15.141:0/2951950694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64c806a7a0 0x7f64c80b4580 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 >> 172.21.15.141:0/2951950694 conn(0x7f64c80b1a60 msgr2=0x7f64c80cdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 shutdown_connections 2024-01-16T15:11:42.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:42.920+0000 7f64d0600640 1 -- 172.21.15.141:0/2951950694 wait complete. 2024-01-16T15:11:42.930 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:11:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.136+0000 7f1b63800640 1 Processor -- start 2024-01-16T15:11:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.136+0000 7f1b63800640 1 -- start start 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.137+0000 7f1b63800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c143ae0 0x7f1b5c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.137+0000 7f1b63800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1b5c15e3b0 0x7f1b5c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.137+0000 7f1b63800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5c0cfc30 con 0x7f1b5c0cf7d0 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b63800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5c0b1ee0 con 0x7f1b5c15e3b0 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b63800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c0b2780 con 0x7f1b5c143ae0 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b61800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1b5c15e3b0 0x7f1b5c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b61800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1b5c15e3b0 0x7f1b5c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38812/0 (socket says 172.21.15.141:38812) 2024-01-16T15:11:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b61800640 1 -- 172.21.15.141:0/4245248480 learned_addr learned my addr 172.21.15.141:0/4245248480 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:43.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b60e00640 1 --2- 172.21.15.141:0/4245248480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c143ae0 0x7f1b5c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.115 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b61800640 1 -- 172.21.15.141:0/4245248480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c143ae0 msgr2=0x7f1b5c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:43.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b61800640 1 --2- 172.21.15.141:0/4245248480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c143ae0 0x7f1b5c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b61800640 1 -- 172.21.15.141:0/4245248480 >> v1:172.21.15.141:6789/0 conn(0x7f1b5c0cf7d0 legacy=0x7f1b5c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b61800640 1 -- 172.21.15.141:0/4245248480 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c15e990 con 0x7f1b5c15e3b0 2024-01-16T15:11:43.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.138+0000 7f1b5be00640 1 -- 172.21.15.141:0/4245248480 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580584854 0 0) 0x7f1b5c0cfc30 con 0x7f1b5c0cf7d0 2024-01-16T15:11:43.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b61800640 1 --2- 172.21.15.141:0/4245248480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c15e3b0 0x7f1b5c0702c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009e50 tx=0x7f1b5402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b7110512a23e83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:43.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b5be00640 1 -- 172.21.15.141:0/4245248480 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5402cb00 con 0x7f1b5c15e3b0 2024-01-16T15:11:43.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b5be00640 1 -- 172.21.15.141:0/4245248480 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5402cc60 con 0x7f1b5c15e3b0 2024-01-16T15:11:43.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b5be00640 1 -- 172.21.15.141:0/4245248480 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b540343f0 con 0x7f1b5c15e3b0 2024-01-16T15:11:43.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b63800640 1 -- 172.21.15.141:0/4245248480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c15e3b0 msgr2=0x7f1b5c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b63800640 1 --2- 172.21.15.141:0/4245248480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c15e3b0 0x7f1b5c0702c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009e50 tx=0x7f1b5402bf60 comp rx=0 tx=0).stop 2024-01-16T15:11:43.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b63800640 1 -- 172.21.15.141:0/4245248480 shutdown_connections 2024-01-16T15:11:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b63800640 1 --2- 172.21.15.141:0/4245248480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c15e3b0 0x7f1b5c0702c0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b63800640 1 --2- 172.21.15.141:0/4245248480 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c143ae0 0x7f1b5c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.140+0000 7f1b63800640 1 -- 172.21.15.141:0/4245248480 >> 172.21.15.141:0/4245248480 conn(0x7f1b5c06a800 msgr2=0x7f1b5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.141+0000 7f1b63800640 1 -- 172.21.15.141:0/4245248480 shutdown_connections 2024-01-16T15:11:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.141+0000 7f1b63800640 1 -- 172.21.15.141:0/4245248480 wait complete. 2024-01-16T15:11:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.142+0000 7f1b63800640 1 Processor -- start 2024-01-16T15:11:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.142+0000 7f1b63800640 1 -- start start 2024-01-16T15:11:43.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.143+0000 7f1b63800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c0cf7d0 0x7f1b5c13c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.143+0000 7f1b63800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c143ae0 0x7f1b5c14db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.143+0000 7f1b63800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b5c15e3b0 0x7f1b5c14e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.143+0000 7f1b63800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c14d760 con 0x7f1b5c143ae0 2024-01-16T15:11:43.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.143+0000 7f1b63800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c14f490 con 0x7f1b5c15e3b0 2024-01-16T15:11:43.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.143+0000 7f1b63800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c13c710 con 0x7f1b5c0cf7d0 2024-01-16T15:11:43.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c143ae0 0x7f1b5c14db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c143ae0 0x7f1b5c14db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38820/0 (socket says 172.21.15.141:38820) 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 -- 172.21.15.141:0/465395203 learned_addr learned my addr 172.21.15.141:0/465395203 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b60e00640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c0cf7d0 0x7f1b5c13c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b61800640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b5c15e3b0 0x7f1b5c14e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 -- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c0cf7d0 msgr2=0x7f1b5c13c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c0cf7d0 0x7f1b5c13c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 -- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b5c15e3b0 msgr2=0x7f1b5c14e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b5c15e3b0 0x7f1b5c14e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 -- 172.21.15.141:0/465395203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b54009b30 con 0x7f1b5c143ae0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.144+0000 7f1b53c00640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c143ae0 0x7f1b5c14db80 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c4e0 tx=0x7f1b4400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.145+0000 7f1b59600640 1 -- 172.21.15.141:0/465395203 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44005480 con 0x7f1b5c143ae0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.145+0000 7f1b59600640 1 -- 172.21.15.141:0/465395203 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b440035f0 con 0x7f1b5c143ae0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.145+0000 7f1b59600640 1 -- 172.21.15.141:0/465395203 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440039c0 con 0x7f1b5c143ae0 2024-01-16T15:11:43.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.145+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5c14cae0 con 0x7f1b5c143ae0 2024-01-16T15:11:43.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.145+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5c14d0a0 con 0x7f1b5c143ae0 2024-01-16T15:11:43.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.146+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5c0675d0 con 0x7f1b5c143ae0 2024-01-16T15:11:43.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.147+0000 7f1b59600640 1 -- 172.21.15.141:0/465395203 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f1b44003b20 con 0x7f1b5c143ae0 2024-01-16T15:11:43.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.148+0000 7f1b59600640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1b3406c1f0 0x7f1b3406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.148+0000 7f1b59600640 1 -- 172.21.15.141:0/465395203 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1b44087bd0 con 0x7f1b5c143ae0 2024-01-16T15:11:43.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.149+0000 7f1b60e00640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1b3406c1f0 0x7f1b3406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.150+0000 7f1b60e00640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1b3406c1f0 0x7f1b3406e6b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006fd0 tx=0x7f1b4c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:43.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.153+0000 7f1b59600640 1 -- 172.21.15.141:0/465395203 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b44054540 con 0x7f1b5c143ae0 2024-01-16T15:11:43.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.381+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1b5c067da0 con 0x7f1b5c143ae0 2024-01-16T15:11:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.968+0000 7f1b59600640 1 -- 172.21.15.141:0/465395203 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v60) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1b44017020 con 0x7f1b5c143ae0 2024-01-16T15:11:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 14 and data pool 15 2024-01-16T15:11:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.974+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1b3406c1f0 msgr2=0x7f1b3406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.974+0000 7f1b63800640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1b3406c1f0 0x7f1b3406e6b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006fd0 tx=0x7f1b4c008040 comp rx=0 tx=0).stop 2024-01-16T15:11:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c143ae0 msgr2=0x7f1b5c14db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c143ae0 0x7f1b5c14db80 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c4e0 tx=0x7f1b4400c9a0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 shutdown_connections 2024-01-16T15:11:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1b3406c1f0 0x7f1b3406e6b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b5c15e3b0 0x7f1b5c14e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1b5c143ae0 0x7f1b5c14db80 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 --2- 172.21.15.141:0/465395203 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1b5c0cf7d0 0x7f1b5c13c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.975+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 >> 172.21.15.141:0/465395203 conn(0x7f1b5c06a800 msgr2=0x7f1b5c0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.976+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 shutdown_connections 2024-01-16T15:11:43.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:43.976+0000 7f1b63800640 1 -- 172.21.15.141:0/465395203 wait complete. 2024-01-16T15:11:43.983 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:44.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.185+0000 7f1700a00640 1 Processor -- start 2024-01-16T15:11:44.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.185+0000 7f1700a00640 1 -- start start 2024-01-16T15:11:44.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.186+0000 7f1700a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f16fc071410 0x7f16fc146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.186+0000 7f1700a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 0x7f16fc0717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.186+0000 7f1700a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc06cc20 con 0x7f16fc071d30 2024-01-16T15:11:44.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.186+0000 7f1700a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16fc06cec0 con 0x7f16fc071410 2024-01-16T15:11:44.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.186+0000 7f1700a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc06d1a0 con 0x7f16fc06c7c0 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.186+0000 7f16fbe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f16fc071d30 0x7f16fc0b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36632/0 (socket says 172.21.15.141:36632) 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fbe00640 1 -- 172.21.15.141:0/3713160941 learned_addr learned my addr 172.21.15.141:0/3713160941 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16faa00640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 0x7f16fc0717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fb400640 1 --2- 172.21.15.141:0/3713160941 >> v2:172.21.15.188:3300/0 conn(0x7f16fc071410 0x7f16fc146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652772779 0 0) 0x7f16fc06cc20 con 0x7f16fc071d30 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc003520 con 0x7f16fc071d30 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229536826 0 0) 0x7f16dc003520 con 0x7f16fc071d30 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 msgr2=0x7f16fc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 0x7f16fc0717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 >> v2:172.21.15.188:3300/0 conn(0x7f16fc071410 msgr2=0x7f16fc146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 --2- 172.21.15.141:0/3713160941 >> v2:172.21.15.188:3300/0 conn(0x7f16fc071410 0x7f16fc146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc06d480 con 0x7f16fc071d30 2024-01-16T15:11:44.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.187+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f16f40019f0 con 0x7f16fc071d30 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 >> v1:172.21.15.141:6789/0 conn(0x7f16fc071d30 legacy=0x7f16fc0b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fa000640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 0x7f16dc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fa000640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16dc003ef0 0x7f16dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fa000640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16dc004d00 0x7f16dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc06cec0 con 0x7f16dc004d00 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc06d1a0 con 0x7f16dc003ef0 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc00b980 con 0x7f16fc06c7c0 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fbe00640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16dc004d00 0x7f16dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fb400640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 0x7f16dc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fbe00640 1 -- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 msgr2=0x7f16dc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fbe00640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16fc06c7c0 0x7f16dc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fbe00640 1 -- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16dc003ef0 msgr2=0x7f16dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fbe00640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16dc003ef0 0x7f16dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.188+0000 7f16fbe00640 1 -- 172.21.15.141:0/3713160941 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc003520 con 0x7f16dc004d00 2024-01-16T15:11:44.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.189+0000 7f16fbe00640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16dc004d00 0x7f16dc005180 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f16f400a830 tx=0x7f16dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569b0c2460098f7c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:44.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4034630 con 0x7f16dc004d00 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f402f070 con 0x7f16dc004d00 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f16fa000640 1 -- 172.21.15.141:0/3713160941 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4034a50 con 0x7f16dc004d00 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f1700a00640 1 -- 172.21.15.141:0/3713160941 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16dc004d00 msgr2=0x7f16dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f1700a00640 1 --2- 172.21.15.141:0/3713160941 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16dc004d00 0x7f16dc005180 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f16f400a830 tx=0x7f16dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f16fbe00640 1 -- 172.21.15.141:0/3713160941 reap_dead start 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f1700a00640 1 -- 172.21.15.141:0/3713160941 shutdown_connections 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f1700a00640 1 -- 172.21.15.141:0/3713160941 >> 172.21.15.141:0/3713160941 conn(0x7f16fc06a800 msgr2=0x7f16fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f1700a00640 1 -- 172.21.15.141:0/3713160941 shutdown_connections 2024-01-16T15:11:44.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.190+0000 7f1700a00640 1 -- 172.21.15.141:0/3713160941 wait complete. 2024-01-16T15:11:44.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.192+0000 7f1700a00640 1 Processor -- start 2024-01-16T15:11:44.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.192+0000 7f1700a00640 1 -- start start 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.193+0000 7f1700a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16dc004d00 0x7f16fc15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.193+0000 7f1700a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16fc143bc0 0x7f16fc144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.193+0000 7f1700a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16fc0c82b0 0x7f16fc0c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.193+0000 7f1700a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1310f0 con 0x7f16fc0c82b0 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.193+0000 7f1700a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc144560 con 0x7f16fc143bc0 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.193+0000 7f1700a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc130500 con 0x7f16dc004d00 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.193+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16fc0c82b0 0x7f16fc0c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16fc0c82b0 0x7f16fc0c8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38840/0 (socket says 172.21.15.141:38840) 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 -- 172.21.15.141:0/3977650669 learned_addr learned my addr 172.21.15.141:0/3977650669 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:44.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fb400640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16dc004d00 0x7f16fc15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16faa00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16fc143bc0 0x7f16fc144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 -- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16dc004d00 msgr2=0x7f16fc15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16dc004d00 0x7f16fc15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 -- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16fc143bc0 msgr2=0x7f16fc144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16fc143bc0 0x7f16fc144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 -- 172.21.15.141:0/3977650669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc0c8c70 con 0x7f16fc0c82b0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fb400640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16dc004d00 0x7f16fc15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.194+0000 7f16fbe00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16fc0c82b0 0x7f16fc0c8730 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f16f402cbe0 tx=0x7f16fc06cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.195+0000 7f16f3400640 1 -- 172.21.15.141:0/3977650669 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4034630 con 0x7f16fc0c82b0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.195+0000 7f16f3400640 1 -- 172.21.15.141:0/3977650669 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4034790 con 0x7f16fc0c82b0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.195+0000 7f16f3400640 1 -- 172.21.15.141:0/3977650669 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4034b00 con 0x7f16fc0c82b0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.195+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc130750 con 0x7f16fc0c82b0 2024-01-16T15:11:44.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.195+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc130d10 con 0x7f16fc0c82b0 2024-01-16T15:11:44.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.196+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc067450 con 0x7f16fc0c82b0 2024-01-16T15:11:44.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.197+0000 7f16f3400640 1 -- 172.21.15.141:0/3977650669 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f16f4031030 con 0x7f16fc0c82b0 2024-01-16T15:11:44.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.198+0000 7f16f3400640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f16d406c1f0 0x7f16d406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.198+0000 7f16f3400640 1 -- 172.21.15.141:0/3977650669 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16f40aa900 con 0x7f16fc0c82b0 2024-01-16T15:11:44.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.198+0000 7f16fb400640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f16d406c1f0 0x7f16d406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.200+0000 7f16fb400640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f16d406c1f0 0x7f16d406e6b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009b50 tx=0x7f16ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:44.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.202+0000 7f16f3400640 1 -- 172.21.15.141:0/3977650669 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16f407aac0 con 0x7f16fc0c82b0 2024-01-16T15:11:44.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.391+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16fc067450 con 0x7f16fc0c82b0 2024-01-16T15:11:44.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.395+0000 7f16f3400640 1 -- 172.21.15.141:0/3977650669 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f16f4001fc0 con 0x7f16fc0c82b0 2024-01-16T15:11:44.371 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:44.371 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":101,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:11:43.383230+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:11:41.205278+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:11:42.403706+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:44.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f16d406c1f0 msgr2=0x7f16d406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f16d406c1f0 0x7f16d406e6b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009b50 tx=0x7f16ec008040 comp rx=0 tx=0).stop 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16fc0c82b0 msgr2=0x7f16fc0c8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16fc0c82b0 0x7f16fc0c8730 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f16f402cbe0 tx=0x7f16fc06cdb0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 shutdown_connections 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f16d406c1f0 0x7f16d406e6b0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16fc0c82b0 0x7f16fc0c8730 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16fc143bc0 0x7f16fc144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 --2- 172.21.15.141:0/3977650669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f16dc004d00 0x7f16fc15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 >> 172.21.15.141:0/3977650669 conn(0x7f16fc06a800 msgr2=0x7f16fc0b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 shutdown_connections 2024-01-16T15:11:44.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.397+0000 7f1700a00640 1 -- 172.21.15.141:0/3977650669 wait complete. 2024-01-16T15:11:44.393 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:44.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.629+0000 7f0bce000640 1 Processor -- start 2024-01-16T15:11:44.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.629+0000 7f0bce000640 1 -- start start 2024-01-16T15:11:44.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.630+0000 7f0bce000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 0x7f0bc80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.630+0000 7f0bce000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc815c5f0 con 0x7f0bc815f9c0 2024-01-16T15:11:44.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.631+0000 7f0bce000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc806c470 con 0x7f0bc813df80 2024-01-16T15:11:44.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.631+0000 7f0bce000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc806c750 con 0x7f0bc813e360 2024-01-16T15:11:44.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.631+0000 7f0bc7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0bc815f9c0 0x7f0bc80cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36636/0 (socket says 172.21.15.141:36636) 2024-01-16T15:11:44.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.631+0000 7f0bc7e00640 1 -- 172.21.15.141:0/2570609991 learned_addr learned my addr 172.21.15.141:0/2570609991 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:44.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.631+0000 7f0bbfe00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 0x7f0bc80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.631+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207982491 0 0) 0x7f0bc815c5f0 con 0x7f0bc815f9c0 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.631+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba8003520 con 0x7f0bc815f9c0 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562645995 0 0) 0x7f0ba8003520 con 0x7f0bc815f9c0 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 msgr2=0x7f0bc80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 0x7f0bc80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 >> v1:172.21.15.188:6789/0 conn(0x7f0bc813df80 legacy=0x7f0bc815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc806ca30 con 0x7f0bc815f9c0 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f0bc00019f0 con 0x7f0bc815f9c0 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 >> v1:172.21.15.141:6789/0 conn(0x7f0bc815f9c0 legacy=0x7f0bc80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba80039a0 0x7f0ba8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 0x7f0ba80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ba8004d00 0x7f0ba8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc806c750 con 0x7f0ba8004d00 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba800ba60 con 0x7f0ba80039a0 2024-01-16T15:11:44.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba800bc20 con 0x7f0bc813e360 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.632+0000 7f0bc7e00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ba8004d00 0x7f0ba8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bc7400640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba80039a0 0x7f0ba8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bbfe00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 0x7f0ba80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bc7e00640 1 -- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 msgr2=0x7f0ba80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bc7e00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 0x7f0ba80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bc7e00640 1 -- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba80039a0 msgr2=0x7f0ba8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bc7e00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba80039a0 0x7f0ba8003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bc7e00640 1 -- 172.21.15.141:0/2570609991 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8003520 con 0x7f0ba8004d00 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.633+0000 7f0bbfe00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc813e360 0x7f0ba80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:44.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.634+0000 7f0bc7e00640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ba8004d00 0x7f0ba8005180 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000a800 tx=0x7f0ba80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33adab3bced3eba7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.634+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc00072d0 con 0x7f0ba8004d00 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.634+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc002f070 con 0x7f0ba8004d00 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.634+0000 7f0bc6a00640 1 -- 172.21.15.141:0/2570609991 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc00076b0 con 0x7f0ba8004d00 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.634+0000 7f0bce000640 1 -- 172.21.15.141:0/2570609991 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ba8004d00 msgr2=0x7f0ba8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.634+0000 7f0bce000640 1 --2- 172.21.15.141:0/2570609991 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ba8004d00 0x7f0ba8005180 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000a800 tx=0x7f0ba80057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.634+0000 7f0bc7e00640 1 -- 172.21.15.141:0/2570609991 reap_dead start 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.635+0000 7f0bce000640 1 -- 172.21.15.141:0/2570609991 shutdown_connections 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.635+0000 7f0bce000640 1 -- 172.21.15.141:0/2570609991 >> 172.21.15.141:0/2570609991 conn(0x7f0bc806a800 msgr2=0x7f0bc806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.635+0000 7f0bce000640 1 -- 172.21.15.141:0/2570609991 shutdown_connections 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.635+0000 7f0bce000640 1 -- 172.21.15.141:0/2570609991 wait complete. 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.636+0000 7f0bce000640 1 Processor -- start 2024-01-16T15:11:44.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.637+0000 7f0bce000640 1 -- start start 2024-01-16T15:11:44.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.637+0000 7f0bce000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba8004d00 0x7f0bc81503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.637+0000 7f0bce000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bc8153890 0x7f0bc8151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bce000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc81521e0 0x7f0bc80b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bce000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc81527f0 con 0x7f0bc8153890 2024-01-16T15:11:44.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bce000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc8152ad0 con 0x7f0ba8004d00 2024-01-16T15:11:44.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bce000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc8152db0 con 0x7f0bc81521e0 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bc8153890 0x7f0bc8151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bc8153890 0x7f0bc8151ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38872/0 (socket says 172.21.15.141:38872) 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 -- 172.21.15.141:0/1951744714 learned_addr learned my addr 172.21.15.141:0/1951744714 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 -- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc81521e0 msgr2=0x7f0bc80b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc81521e0 0x7f0bc80b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 -- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba8004d00 msgr2=0x7f0bc81503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba8004d00 0x7f0bc81503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 -- 172.21.15.141:0/1951744714 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8153090 con 0x7f0bc8153890 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.638+0000 7f0bbfe00640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bc8153890 0x7f0bc8151ca0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00108b0 tx=0x7f0bb0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.639+0000 7f0bbf400640 1 -- 172.21.15.141:0/1951744714 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000eae0 con 0x7f0bc8153890 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.639+0000 7f0bbf400640 1 -- 172.21.15.141:0/1951744714 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb000ec40 con 0x7f0bc8153890 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.639+0000 7f0bbf400640 1 -- 172.21.15.141:0/1951744714 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb001c400 con 0x7f0bc8153890 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.639+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc8153370 con 0x7f0bc8153890 2024-01-16T15:11:44.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.639+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc80b3230 con 0x7f0bc8153890 2024-01-16T15:11:44.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.640+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc80675d0 con 0x7f0bc8153890 2024-01-16T15:11:44.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.641+0000 7f0bbf400640 1 -- 172.21.15.141:0/1951744714 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f0bb001c560 con 0x7f0bc8153890 2024-01-16T15:11:44.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.642+0000 7f0bbf400640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba006c1f0 0x7f0ba006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:44.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.642+0000 7f0bbf400640 1 -- 172.21.15.141:0/1951744714 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bb0088ef0 con 0x7f0bc8153890 2024-01-16T15:11:44.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.643+0000 7f0bc7400640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba006c1f0 0x7f0ba006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:44.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.644+0000 7f0bc7400640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba006c1f0 0x7f0ba006e6b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8001640 tx=0x7f0bb8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:44.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.646+0000 7f0bbf400640 1 -- 172.21.15.141:0/1951744714 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bb00555d0 con 0x7f0bc8153890 2024-01-16T15:11:44.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.837+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bc806df00 con 0x7f0bc8153890 2024-01-16T15:11:44.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.840+0000 7f0bbf400640 1 -- 172.21.15.141:0/1951744714 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f0bb0058bf0 con 0x7f0bc8153890 2024-01-16T15:11:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:44.816 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":101,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:11:43.383230+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:11:41.205278+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:11:42.403706+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:44.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba006c1f0 msgr2=0x7f0ba006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba006c1f0 0x7f0ba006e6b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8001640 tx=0x7f0bb8009040 comp rx=0 tx=0).stop 2024-01-16T15:11:44.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bc8153890 msgr2=0x7f0bc8151ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:44.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bc8153890 0x7f0bc8151ca0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00108b0 tx=0x7f0bb0010d70 comp rx=0 tx=0).stop 2024-01-16T15:11:44.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 shutdown_connections 2024-01-16T15:11:44.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0ba006c1f0 0x7f0ba006e6b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0bc81521e0 0x7f0bc80b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0bc8153890 0x7f0bc8151ca0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 --2- 172.21.15.141:0/1951744714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ba8004d00 0x7f0bc81503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:44.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 >> 172.21.15.141:0/1951744714 conn(0x7f0bc806a800 msgr2=0x7f0bc815a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:44.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 shutdown_connections 2024-01-16T15:11:44.820 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:44.843+0000 7f0bce000640 1 -- 172.21.15.141:0/1951744714 wait complete. 2024-01-16T15:11:44.839 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:11:45.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.076+0000 7fc3d5000640 1 Processor -- start 2024-01-16T15:11:45.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.077+0000 7fc3d5000640 1 -- start start 2024-01-16T15:11:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.078+0000 7fc3d5000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc3d013cf10 0x7fc3d013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:45.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.078+0000 7fc3d5000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013fae0 0x7fc3d00cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.078+0000 7fc3d5000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc3d00cdc20 0x7fc3d015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.078+0000 7fc3d5000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d013ff60 con 0x7fc3d00cdc20 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.078+0000 7fc3d5000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00ce440 con 0x7fc3d013cf10 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.078+0000 7fc3d5000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d00ce720 con 0x7fc3d013fae0 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc3d00cdc20 0x7fc3d015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc3d00cdc20 0x7fc3d015d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38878/0 (socket says 172.21.15.141:38878) 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 -- 172.21.15.141:0/4254833458 learned_addr learned my addr 172.21.15.141:0/4254833458 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cf400640 1 --2- 172.21.15.141:0/4254833458 >> v2:172.21.15.188:3300/0 conn(0x7fc3d013cf10 0x7fc3d013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3c7e00640 1 --2- 172.21.15.141:0/4254833458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013fae0 0x7fc3d00cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 -- 172.21.15.141:0/4254833458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013fae0 msgr2=0x7fc3d00cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 --2- 172.21.15.141:0/4254833458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013fae0 0x7fc3d00cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 -- 172.21.15.141:0/4254833458 >> v2:172.21.15.188:3300/0 conn(0x7fc3d013cf10 msgr2=0x7fc3d013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 --2- 172.21.15.141:0/4254833458 >> v2:172.21.15.188:3300/0 conn(0x7fc3d013cf10 0x7fc3d013f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.079+0000 7fc3cfe00640 1 -- 172.21.15.141:0/4254833458 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00cea00 con 0x7fc3d00cdc20 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.080+0000 7fc3cfe00640 1 --2- 172.21.15.141:0/4254833458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d015d9b0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc3d006e390 tx=0x7fc3c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ff4ac961137f8b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3cea00640 1 -- 172.21.15.141:0/4254833458 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c802ca80 con 0x7fc3d00cdc20 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3cea00640 1 -- 172.21.15.141:0/4254833458 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c802cbe0 con 0x7fc3d00cdc20 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3cea00640 1 -- 172.21.15.141:0/4254833458 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c80343f0 con 0x7fc3d00cdc20 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 -- 172.21.15.141:0/4254833458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 msgr2=0x7fc3d015d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 --2- 172.21.15.141:0/4254833458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d015d9b0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc3d006e390 tx=0x7fc3c802bee0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 -- 172.21.15.141:0/4254833458 shutdown_connections 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 --2- 172.21.15.141:0/4254833458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d015d9b0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 --2- 172.21.15.141:0/4254833458 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013fae0 0x7fc3d00cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 --2- 172.21.15.141:0/4254833458 >> v2:172.21.15.188:3300/0 conn(0x7fc3d013cf10 0x7fc3d013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 -- 172.21.15.141:0/4254833458 >> 172.21.15.141:0/4254833458 conn(0x7fc3d006a7d0 msgr2=0x7fc3d006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:45.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.081+0000 7fc3d5000640 1 -- 172.21.15.141:0/4254833458 shutdown_connections 2024-01-16T15:11:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.082+0000 7fc3d5000640 1 -- 172.21.15.141:0/4254833458 wait complete. 2024-01-16T15:11:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.083+0000 7fc3d5000640 1 Processor -- start 2024-01-16T15:11:45.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.083+0000 7fc3d5000640 1 -- start start 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.084+0000 7fc3d5000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d013ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.084+0000 7fc3d5000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013cf10 0x7fc3d01365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.084+0000 7fc3d5000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d013fae0 0x7fc3d013d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.084+0000 7fc3d5000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d0136b10 con 0x7fc3d00cdc20 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.084+0000 7fc3d5000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d013db20 con 0x7fc3d013fae0 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.084+0000 7fc3d5000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d013de00 con 0x7fc3d013cf10 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d013ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d013ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38894/0 (socket says 172.21.15.141:38894) 2024-01-16T15:11:45.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 -- 172.21.15.141:0/1923879254 learned_addr learned my addr 172.21.15.141:0/1923879254 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:45.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 -- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013cf10 msgr2=0x7fc3d01365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:45.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013cf10 0x7fc3d01365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 -- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d013fae0 msgr2=0x7fc3d013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:45.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d013fae0 0x7fc3d013d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 -- 172.21.15.141:0/1923879254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8009b30 con 0x7fc3d00cdc20 2024-01-16T15:11:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3cf400640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d013ef40 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000c7f0 tx=0x7fc3c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.085+0000 7fc3c7400640 1 -- 172.21.15.141:0/1923879254 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0004220 con 0x7fc3d00cdc20 2024-01-16T15:11:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.086+0000 7fc3c7400640 1 -- 172.21.15.141:0/1923879254 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0004380 con 0x7fc3d00cdc20 2024-01-16T15:11:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.086+0000 7fc3c7400640 1 -- 172.21.15.141:0/1923879254 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0007860 con 0x7fc3d00cdc20 2024-01-16T15:11:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.086+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d013e0e0 con 0x7fc3d00cdc20 2024-01-16T15:11:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.086+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d013d830 con 0x7fc3d00cdc20 2024-01-16T15:11:45.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.087+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d013e300 con 0x7fc3d00cdc20 2024-01-16T15:11:45.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.088+0000 7fc3c7400640 1 -- 172.21.15.141:0/1923879254 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fc3c0007bd0 con 0x7fc3d00cdc20 2024-01-16T15:11:45.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.089+0000 7fc3c7400640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc3a806c1f0 0x7fc3a806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:45.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.089+0000 7fc3c7400640 1 -- 172.21.15.141:0/1923879254 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3c0088910 con 0x7fc3d00cdc20 2024-01-16T15:11:45.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.090+0000 7fc3c7e00640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc3a806c1f0 0x7fc3a806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:45.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.091+0000 7fc3c7e00640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc3a806c1f0 0x7fc3a806e6b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01e6030 tx=0x7fc3b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:45.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.094+0000 7fc3c7400640 1 -- 172.21.15.141:0/1923879254 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3c0054eb0 con 0x7fc3d00cdc20 2024-01-16T15:11:45.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.322+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc3d01590b0 con 0x7fc3d00cdc20 2024-01-16T15:11:45.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.983+0000 7fc3c7400640 1 -- 172.21.15.141:0/1923879254 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v63) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc3c001e070 con 0x7fc3d00cdc20 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.986+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc3a806c1f0 msgr2=0x7fc3a806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.986+0000 7fc3d5000640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc3a806c1f0 0x7fc3a806e6b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01e6030 tx=0x7fc3b8008040 comp rx=0 tx=0).stop 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.986+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 msgr2=0x7fc3d013ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.986+0000 7fc3d5000640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d013ef40 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000c7f0 tx=0x7fc3c000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 shutdown_connections 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc3a806c1f0 0x7fc3a806e6b0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d013fae0 0x7fc3d013d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3d013cf10 0x7fc3d01365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 --2- 172.21.15.141:0/1923879254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d00cdc20 0x7fc3d013ef40 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 >> 172.21.15.141:0/1923879254 conn(0x7fc3d006a7d0 msgr2=0x7fc3d0153930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 shutdown_connections 2024-01-16T15:11:45.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:45.987+0000 7fc3d5000640 1 -- 172.21.15.141:0/1923879254 wait complete. 2024-01-16T15:11:45.984 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:11:45.984 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:11:46.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.224+0000 7f5056600640 1 Processor -- start 2024-01-16T15:11:46.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.224+0000 7f5056600640 1 -- start start 2024-01-16T15:11:46.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f5056600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f5050070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:46.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f5056600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5050071030 0x7f505006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:46.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f5056600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5050071460 con 0x7f5050157c30 2024-01-16T15:11:46.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f5056600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505015ea30 con 0x7f5050071030 2024-01-16T15:11:46.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f5056600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505006d860 con 0x7f50500cd600 2024-01-16T15:11:46.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f504fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5050071030 0x7f505006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:46.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f504fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5050071030 0x7f505006dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38910/0 (socket says 172.21.15.141:38910) 2024-01-16T15:11:46.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f504fe00640 1 -- 172.21.15.141:0/3670504655 learned_addr learned my addr 172.21.15.141:0/3670504655 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:46.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.226+0000 7f5047e00640 1 --2- 172.21.15.141:0/3670504655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f5050070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:46.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.227+0000 7f504fe00640 1 -- 172.21.15.141:0/3670504655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 msgr2=0x7f5050070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:46.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.227+0000 7f504fe00640 1 --2- 172.21.15.141:0/3670504655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f5050070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.227+0000 7f504fe00640 1 -- 172.21.15.141:0/3670504655 >> v1:172.21.15.188:6789/0 conn(0x7f5050157c30 legacy=0x7f50501618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:46.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.227+0000 7f504fe00640 1 -- 172.21.15.141:0/3670504655 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505015deb0 con 0x7f5050071030 2024-01-16T15:11:46.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.227+0000 7f5047e00640 1 --2- 172.21.15.141:0/3670504655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f5050070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:46.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.228+0000 7f504fe00640 1 --2- 172.21.15.141:0/3670504655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f505006dcb0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f505006b630 tx=0x7f504802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823fea7ddb7a51ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:46.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.228+0000 7f504ea00640 1 -- 172.21.15.141:0/3670504655 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504802ca30 con 0x7f5050071030 2024-01-16T15:11:46.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.228+0000 7f504ea00640 1 -- 172.21.15.141:0/3670504655 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504802cb90 con 0x7f5050071030 2024-01-16T15:11:46.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.228+0000 7f504ea00640 1 -- 172.21.15.141:0/3670504655 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50480343f0 con 0x7f5050071030 2024-01-16T15:11:46.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 -- 172.21.15.141:0/3670504655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 msgr2=0x7f505006dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:46.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 --2- 172.21.15.141:0/3670504655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f505006dcb0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f505006b630 tx=0x7f504802be90 comp rx=0 tx=0).stop 2024-01-16T15:11:46.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 -- 172.21.15.141:0/3670504655 shutdown_connections 2024-01-16T15:11:46.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 --2- 172.21.15.141:0/3670504655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f505006dcb0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 --2- 172.21.15.141:0/3670504655 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f5050070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 -- 172.21.15.141:0/3670504655 >> 172.21.15.141:0/3670504655 conn(0x7f50500cf700 msgr2=0x7f50500cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:46.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 -- 172.21.15.141:0/3670504655 shutdown_connections 2024-01-16T15:11:46.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.229+0000 7f5056600640 1 -- 172.21.15.141:0/3670504655 wait complete. 2024-01-16T15:11:46.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.231+0000 7f5056600640 1 Processor -- start 2024-01-16T15:11:46.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.231+0000 7f5056600640 1 -- start start 2024-01-16T15:11:46.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f5056600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f5050140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:46.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f5056600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f505013fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:46.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f5056600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5050157c30 0x7f50501401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:46.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f5056600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5050154f50 con 0x7f5050071030 2024-01-16T15:11:46.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f5056600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5050140710 con 0x7f5050157c30 2024-01-16T15:11:46.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f5056600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5050153ff0 con 0x7f50500cd600 2024-01-16T15:11:46.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f504f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f5050140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:46.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f504f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f5050140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38918/0 (socket says 172.21.15.141:38918) 2024-01-16T15:11:46.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.232+0000 7f504f400640 1 -- 172.21.15.141:0/2561579296 learned_addr learned my addr 172.21.15.141:0/2561579296 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f504f400640 1 -- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 msgr2=0x7f505013fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f504fe00640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5050157c30 0x7f50501401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f504f400640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f505013fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f504f400640 1 -- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5050157c30 msgr2=0x7f50501401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f504f400640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5050157c30 0x7f50501401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f504f400640 1 -- 172.21.15.141:0/2561579296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048009b30 con 0x7f5050071030 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f504f400640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f5050140a60 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f504000c430 tx=0x7f504000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f5047400640 1 -- 172.21.15.141:0/2561579296 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040011790 con 0x7f5050071030 2024-01-16T15:11:46.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f5047400640 1 -- 172.21.15.141:0/2561579296 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50400118f0 con 0x7f5050071030 2024-01-16T15:11:46.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f5047400640 1 -- 172.21.15.141:0/2561579296 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040011ce0 con 0x7f5050071030 2024-01-16T15:11:46.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50501542d0 con 0x7f5050071030 2024-01-16T15:11:46.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.233+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5050154890 con 0x7f5050071030 2024-01-16T15:11:46.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.235+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50500675d0 con 0x7f5050071030 2024-01-16T15:11:46.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.235+0000 7f5047400640 1 -- 172.21.15.141:0/2561579296 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f504001c460 con 0x7f5050071030 2024-01-16T15:11:46.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.236+0000 7f5047400640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f502806c1f0 0x7f502806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:46.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.236+0000 7f5047400640 1 -- 172.21.15.141:0/2561579296 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5040088930 con 0x7f5050071030 2024-01-16T15:11:46.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.237+0000 7f5047e00640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f502806c1f0 0x7f502806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:46.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.238+0000 7f5047e00640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f502806c1f0 0x7f502806e6b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5038006fd0 tx=0x7f5038008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:46.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.240+0000 7f5047400640 1 -- 172.21.15.141:0/2561579296 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5040054f60 con 0x7f5050071030 2024-01-16T15:11:46.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.471+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5050160dc0 con 0x7f5050071030 2024-01-16T15:11:46.973 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:46.997+0000 7f5047400640 1 -- 172.21.15.141:0/2561579296 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5040058580 con 0x7f5050071030 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f502806c1f0 msgr2=0x7f502806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f502806c1f0 0x7f502806e6b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5038006fd0 tx=0x7f5038008040 comp rx=0 tx=0).stop 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 msgr2=0x7f5050140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f5050140a60 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f504000c430 tx=0x7f504000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 shutdown_connections 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f502806c1f0 0x7f502806e6b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5050157c30 0x7f50501401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f50500cd600 0x7f505013fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 --2- 172.21.15.141:0/2561579296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5050071030 0x7f5050140a60 secure :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f504000c430 tx=0x7f504000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 >> 172.21.15.141:0/2561579296 conn(0x7f50500cf700 msgr2=0x7f505006a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:46.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 shutdown_connections 2024-01-16T15:11:46.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.001+0000 7f5056600640 1 -- 172.21.15.141:0/2561579296 wait complete. 2024-01-16T15:11:47.000 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:11:47.000 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:47.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.228+0000 7fa961e00640 1 Processor -- start 2024-01-16T15:11:47.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.228+0000 7fa961e00640 1 -- start start 2024-01-16T15:11:47.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa961e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa95c0ae5a0 0x7fa95c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa961e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 0x7fa95c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa961e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c070c40 con 0x7fa95c0b3ae0 2024-01-16T15:11:47.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa961e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c071450 con 0x7fa95c0ae5a0 2024-01-16T15:11:47.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa961e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0cfc60 con 0x7fa95c0cf290 2024-01-16T15:11:47.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa95a000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa95c0b3ae0 0x7fa95c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36640/0 (socket says 172.21.15.141:36640) 2024-01-16T15:11:47.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa95a000640 1 -- 172.21.15.141:0/3970746579 learned_addr learned my addr 172.21.15.141:0/3970746579 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:47.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.230+0000 7fa95b400640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 0x7fa95c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa95aa00640 1 --2- 172.21.15.141:0/3970746579 >> v2:172.21.15.188:3300/0 conn(0x7fa95c0ae5a0 0x7fa95c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386348947 0 0) 0x7fa95c070c40 con 0x7fa95c0b3ae0 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa93c003520 con 0x7fa95c0b3ae0 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 891107913 0 0) 0x7fa93c003520 con 0x7fa95c0b3ae0 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 msgr2=0x7fa95c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 0x7fa95c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 >> v2:172.21.15.188:3300/0 conn(0x7fa95c0ae5a0 msgr2=0x7fa95c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 --2- 172.21.15.141:0/3970746579 >> v2:172.21.15.188:3300/0 conn(0x7fa95c0ae5a0 0x7fa95c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c06c470 con 0x7fa95c0b3ae0 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa944001670 con 0x7fa95c0b3ae0 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 >> v1:172.21.15.141:6789/0 conn(0x7fa95c0b3ae0 legacy=0x7fa95c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c003970 0x7fa93c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 0x7fa93c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93c004d00 0x7fa93c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c071450 con 0x7fa93c004d00 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0cfc60 con 0x7fa93c003970 2024-01-16T15:11:47.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.231+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c00ba00 con 0x7fa95c0cf290 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95b400640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93c004d00 0x7fa93c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95a000640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 0x7fa93c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95b400640 1 -- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 msgr2=0x7fa93c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95b400640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c0cf290 0x7fa93c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95b400640 1 -- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c003970 msgr2=0x7fa93c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95aa00640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c003970 0x7fa93c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95b400640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c003970 0x7fa93c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.232+0000 7fa95b400640 1 -- 172.21.15.141:0/3970746579 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c003520 con 0x7fa93c004d00 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.233+0000 7fa95b400640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93c004d00 0x7fa93c005180 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fa954009fc0 tx=0x7fa93c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01a17f621d1871e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.233+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95402cc20 con 0x7fa93c004d00 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.233+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95400f040 con 0x7fa93c004d00 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.233+0000 7fa959600640 1 -- 172.21.15.141:0/3970746579 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954034440 con 0x7fa93c004d00 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.234+0000 7fa961e00640 1 -- 172.21.15.141:0/3970746579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93c004d00 msgr2=0x7fa93c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.234+0000 7fa961e00640 1 --2- 172.21.15.141:0/3970746579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa93c004d00 0x7fa93c005180 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fa954009fc0 tx=0x7fa93c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.234+0000 7fa95b400640 1 -- 172.21.15.141:0/3970746579 reap_dead start 2024-01-16T15:11:47.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.234+0000 7fa961e00640 1 -- 172.21.15.141:0/3970746579 shutdown_connections 2024-01-16T15:11:47.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.234+0000 7fa961e00640 1 -- 172.21.15.141:0/3970746579 >> 172.21.15.141:0/3970746579 conn(0x7fa95c06a800 msgr2=0x7fa95c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:47.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.234+0000 7fa961e00640 1 -- 172.21.15.141:0/3970746579 shutdown_connections 2024-01-16T15:11:47.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.234+0000 7fa961e00640 1 -- 172.21.15.141:0/3970746579 wait complete. 2024-01-16T15:11:47.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.236+0000 7fa961e00640 1 Processor -- start 2024-01-16T15:11:47.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.236+0000 7fa961e00640 1 -- start start 2024-01-16T15:11:47.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa961e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c1471f0 0x7fa95c15b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa961e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa95c1475d0 0x7fa95c1f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa961e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa95c1f6460 0x7fa95c1f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa961e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c1f6e50 con 0x7fa95c1f6460 2024-01-16T15:11:47.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa961e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c15c840 con 0x7fa95c1475d0 2024-01-16T15:11:47.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa961e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c1f3e50 con 0x7fa95c1471f0 2024-01-16T15:11:47.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa95b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa95c1f6460 0x7fa95c1f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa95b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa95c1f6460 0x7fa95c1f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38936/0 (socket says 172.21.15.141:38936) 2024-01-16T15:11:47.214 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.237+0000 7fa95b400640 1 -- 172.21.15.141:0/2486141111 learned_addr learned my addr 172.21.15.141:0/2486141111 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa95b400640 1 -- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c1471f0 msgr2=0x7fa95c15b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa95b400640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c1471f0 0x7fa95c15b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa95b400640 1 -- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa95c1475d0 msgr2=0x7fa95c1f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa95b400640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa95c1475d0 0x7fa95c1f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa95b400640 1 -- 172.21.15.141:0/2486141111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c15bc30 con 0x7fa95c1f6460 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa95b400640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa95c1f6460 0x7fa95c1f6910 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fa954002230 tx=0x7fa954009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa953400640 1 -- 172.21.15.141:0/2486141111 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954034ad0 con 0x7fa95c1f6460 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa953400640 1 -- 172.21.15.141:0/2486141111 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa954034c30 con 0x7fa95c1f6460 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c15bf10 con 0x7fa95c1f6460 2024-01-16T15:11:47.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa953400640 1 -- 172.21.15.141:0/2486141111 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954003990 con 0x7fa95c1f6460 2024-01-16T15:11:47.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.238+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c15c4d0 con 0x7fa95c1f6460 2024-01-16T15:11:47.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.240+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c0675d0 con 0x7fa95c1f6460 2024-01-16T15:11:47.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.240+0000 7fa953400640 1 -- 172.21.15.141:0/2486141111 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fa954003af0 con 0x7fa95c1f6460 2024-01-16T15:11:47.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.241+0000 7fa953400640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa93406c220 0x7fa93406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.241+0000 7fa953400640 1 -- 172.21.15.141:0/2486141111 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9540a9e00 con 0x7fa95c1f6460 2024-01-16T15:11:47.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.242+0000 7fa95aa00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa93406c220 0x7fa93406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.243+0000 7fa95aa00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa93406c220 0x7fa93406e6e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0099d0 tx=0x7fa94c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:47.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.245+0000 7fa953400640 1 -- 172.21.15.141:0/2486141111 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa954076770 con 0x7fa95c1f6460 2024-01-16T15:11:47.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.472+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa95c15c9b0 con 0x7fa95c1f6460 2024-01-16T15:11:47.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.473+0000 7fa953400640 1 -- 172.21.15.141:0/2486141111 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fa954045090 con 0x7fa95c1f6460 2024-01-16T15:11:47.450 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:47.450 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3880199275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":3880199275},{"type":"v1","addr":"172.21.15.141:6835","nonce":3880199275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2564906195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2564906195},{"type":"v1","addr":"172.21.15.141:6834","nonce":2564906195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6466,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/924822672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":924822672},{"type":"v1","addr":"172.21.15.188:6837","nonce":924822672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:11:43.384065+0000","modified":"2024-01-16T15:11:46.473187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6836/1516772217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1516772217},{"type":"v1","addr":"172.21.15.188:6836","nonce":1516772217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:11:47.450 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 64 2024-01-16T15:11:47.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa93406c220 msgr2=0x7fa93406e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa93406c220 0x7fa93406e6e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0099d0 tx=0x7fa94c008040 comp rx=0 tx=0).stop 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa95c1f6460 msgr2=0x7fa95c1f6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa95c1f6460 0x7fa95c1f6910 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fa954002230 tx=0x7fa954009750 comp rx=0 tx=0).stop 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 shutdown_connections 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa93406c220 0x7fa93406e6e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa95c1f6460 0x7fa95c1f6910 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa95c1475d0 0x7fa95c1f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 --2- 172.21.15.141:0/2486141111 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa95c1471f0 0x7fa95c15b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 >> 172.21.15.141:0/2486141111 conn(0x7fa95c06a800 msgr2=0x7fa95c161860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 shutdown_connections 2024-01-16T15:11:47.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.476+0000 7fa961e00640 1 -- 172.21.15.141:0/2486141111 wait complete. 2024-01-16T15:11:47.470 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:11:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.700+0000 7f9488a00640 1 Processor -- start 2024-01-16T15:11:47.676 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.700+0000 7f9488a00640 1 -- start start 2024-01-16T15:11:47.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.702+0000 7f9488a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 0x7f9480070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.702+0000 7f9488a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9480071690 con 0x7f9480070ea0 2024-01-16T15:11:47.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.702+0000 7f9488a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9480071970 con 0x7f948014b2c0 2024-01-16T15:11:47.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.702+0000 7f9488a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948014b6a0 con 0x7f94800cf9a0 2024-01-16T15:11:47.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.702+0000 7f9486a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9480070ea0 0x7f94800cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36654/0 (socket says 172.21.15.141:36654) 2024-01-16T15:11:47.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.702+0000 7f9486a00640 1 -- 172.21.15.141:0/2657318996 learned_addr learned my addr 172.21.15.141:0/2657318996 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:47.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.702+0000 7f9485600640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 0x7f9480070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305165739 0 0) 0x7f9480071690 con 0x7f9480070ea0 2024-01-16T15:11:47.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9460003520 con 0x7f9480070ea0 2024-01-16T15:11:47.680 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664835916 0 0) 0x7f9460003520 con 0x7f9480070ea0 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 msgr2=0x7f9480070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 0x7f9480070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 >> v1:172.21.15.188:6789/0 conn(0x7f948014b2c0 legacy=0x7f94800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9480071c50 con 0x7f9480070ea0 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f94780019f0 con 0x7f9480070ea0 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 >> v1:172.21.15.141:6789/0 conn(0x7f9480070ea0 legacy=0x7f94800cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94600039a0 0x7f9460003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 0x7f94600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9460004d00 0x7f9460005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948014b6a0 con 0x7f94600039a0 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946000ba30 con 0x7f9460004d00 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.703+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946000bbf0 con 0x7f94800cf9a0 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9486000640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94600039a0 0x7f9460003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9485600640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 0x7f94600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9486a00640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9460004d00 0x7f9460005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9486000640 1 -- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 msgr2=0x7f94600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:47.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9486000640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94800cf9a0 0x7f94600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9486000640 1 -- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9460004d00 msgr2=0x7f9460005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:47.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9486000640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9460004d00 0x7f9460005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.704+0000 7f9486000640 1 -- 172.21.15.141:0/2657318996 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9460003520 con 0x7f94600039a0 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.705+0000 7f9486000640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94600039a0 0x7f9460003dc0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f9470001720 tx=0x7f94600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c410482103de7d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.705+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94700052d0 con 0x7f94600039a0 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947002e050 con 0x7f94600039a0 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9484c00640 1 -- 172.21.15.141:0/2657318996 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94700035f0 con 0x7f94600039a0 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9488a00640 1 -- 172.21.15.141:0/2657318996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94600039a0 msgr2=0x7f9460003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9488a00640 1 --2- 172.21.15.141:0/2657318996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94600039a0 0x7f9460003dc0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f9470001720 tx=0x7f94600057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9486a00640 1 -- 172.21.15.141:0/2657318996 reap_dead start 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9488a00640 1 -- 172.21.15.141:0/2657318996 shutdown_connections 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9488a00640 1 -- 172.21.15.141:0/2657318996 >> 172.21.15.141:0/2657318996 conn(0x7f948006a800 msgr2=0x7f948006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:47.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9488a00640 1 -- 172.21.15.141:0/2657318996 shutdown_connections 2024-01-16T15:11:47.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.706+0000 7f9488a00640 1 -- 172.21.15.141:0/2657318996 wait complete. 2024-01-16T15:11:47.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.708+0000 7f9488a00640 1 Processor -- start 2024-01-16T15:11:47.684 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.708+0000 7f9488a00640 1 -- start start 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.709+0000 7f9488a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94600039a0 0x7f948015c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.709+0000 7f9488a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f948013f420 0x7f948015d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.709+0000 7f9488a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94800af320 0x7f948015dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.709+0000 7f9488a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948015ced0 con 0x7f94800af320 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.709+0000 7f9488a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948015cbf0 con 0x7f948013f420 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.709+0000 7f9488a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9480162ee0 con 0x7f94600039a0 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.709+0000 7f9486a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94800af320 0x7f948015dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94800af320 0x7f948015dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38954/0 (socket says 172.21.15.141:38954) 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 -- 172.21.15.141:0/3894423764 learned_addr learned my addr 172.21.15.141:0/3894423764 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486000640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94600039a0 0x7f948015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9485600640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f948013f420 0x7f948015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 -- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94600039a0 msgr2=0x7f948015c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94600039a0 0x7f948015c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 -- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f948013f420 msgr2=0x7f948015d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f948013f420 0x7f948015d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 -- 172.21.15.141:0/3894423764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94801631c0 con 0x7f94800af320 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.710+0000 7f9486a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94800af320 0x7f948015dbb0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f9478001370 tx=0x7f947800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.711+0000 7f947e000640 1 -- 172.21.15.141:0/3894423764 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947800bb80 con 0x7f94800af320 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.711+0000 7f947e000640 1 -- 172.21.15.141:0/3894423764 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947800bd00 con 0x7f94800af320 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.711+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94801634a0 con 0x7f94800af320 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.711+0000 7f947e000640 1 -- 172.21.15.141:0/3894423764 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478007350 con 0x7f94800af320 2024-01-16T15:11:47.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.711+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94801f75c0 con 0x7f94800af320 2024-01-16T15:11:47.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.712+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9480163690 con 0x7f94800af320 2024-01-16T15:11:47.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.713+0000 7f947e000640 1 -- 172.21.15.141:0/3894423764 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f94780074b0 con 0x7f94800af320 2024-01-16T15:11:47.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.714+0000 7f947e000640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f945806c220 0x7f945806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:47.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.714+0000 7f947e000640 1 -- 172.21.15.141:0/3894423764 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9478088960 con 0x7f94800af320 2024-01-16T15:11:47.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.714+0000 7f9486000640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f945806c220 0x7f945806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:47.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.715+0000 7f9486000640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f945806c220 0x7f945806e6e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9470033000 tx=0x7f9470009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:47.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.718+0000 7f947e000640 1 -- 172.21.15.141:0/3894423764 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9478055280 con 0x7f94800af320 2024-01-16T15:11:47.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.910+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94800675d0 con 0x7f94800af320 2024-01-16T15:11:47.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.913+0000 7f947e000640 1 -- 172.21.15.141:0/3894423764 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16567 (secure 0 0 0) 0x7f94780588a0 con 0x7f94800af320 2024-01-16T15:11:47.890 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:47.890 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":102,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:11:44.940256+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:11:41.205278+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:11:42.403706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:11:47.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f945806c220 msgr2=0x7f945806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f945806c220 0x7f945806e6e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9470033000 tx=0x7f9470009030 comp rx=0 tx=0).stop 2024-01-16T15:11:47.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94800af320 msgr2=0x7f948015dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:47.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94800af320 0x7f948015dbb0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f9478001370 tx=0x7f947800b4a0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 shutdown_connections 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f945806c220 0x7f945806e6e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f94800af320 0x7f948015dbb0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f948013f420 0x7f948015d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 --2- 172.21.15.141:0/3894423764 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f94600039a0 0x7f948015c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.916+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 >> 172.21.15.141:0/3894423764 conn(0x7f948006a800 msgr2=0x7f9480149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.917+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 shutdown_connections 2024-01-16T15:11:47.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:47.917+0000 7f9488a00640 1 -- 172.21.15.141:0/3894423764 wait complete. 2024-01-16T15:11:47.911 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:11:47.911 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:11:48.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.143+0000 7f8d53000640 1 Processor -- start 2024-01-16T15:11:48.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.143+0000 7f8d53000640 1 -- start start 2024-01-16T15:11:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.144+0000 7f8d53000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 0x7f8d4c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d53000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c06c470 con 0x7f8d4c13e360 2024-01-16T15:11:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d53000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c06c750 con 0x7f8d4c0cf9a0 2024-01-16T15:11:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d53000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c06ca30 con 0x7f8d4c13df80 2024-01-16T15:11:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d43e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8d4c13e360 0x7f8d4c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36656/0 (socket says 172.21.15.141:36656) 2024-01-16T15:11:48.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d43e00640 1 -- 172.21.15.141:0/202589351 learned_addr learned my addr 172.21.15.141:0/202589351 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328160562 0 0) 0x7f8d4c06c470 con 0x7f8d4c13e360 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2c003520 con 0x7f8d4c13e360 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.145+0000 7f8d4be00640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 0x7f8d4c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2649915501 0 0) 0x7f8d2c003520 con 0x7f8d4c13e360 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 msgr2=0x7f8d4c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 0x7f8d4c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 >> v1:172.21.15.188:6789/0 conn(0x7f8d4c0cf9a0 legacy=0x7f8d4c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c06cd10 con 0x7f8d4c13e360 2024-01-16T15:11:48.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8d34001670 con 0x7f8d4c13e360 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 >> v1:172.21.15.141:6789/0 conn(0x7f8d4c13e360 legacy=0x7f8d4c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 0x7f8d2c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d2c004d00 0x7f8d2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c06ca30 con 0x7f8d2c003fd0 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c00bb40 con 0x7f8d2c004d00 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c00bd00 con 0x7f8d4c13df80 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.146+0000 7f8d43e00640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d4be00640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 0x7f8d2c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d43e00640 1 -- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 msgr2=0x7f8d2c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d43e00640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 0x7f8d2c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d43e00640 1 -- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d2c004d00 msgr2=0x7f8d2c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d51000640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d2c004d00 0x7f8d2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d43e00640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d2c004d00 0x7f8d2c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d43e00640 1 -- 172.21.15.141:0/202589351 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c003520 con 0x7f8d2c003fd0 2024-01-16T15:11:48.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.147+0000 7f8d4be00640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c13df80 0x7f8d2c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.148+0000 7f8d43e00640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d2c0047c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c240 tx=0x7f8d2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ebfe842775712a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.148+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34005ad0 con 0x7f8d2c003fd0 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.148+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34005c30 con 0x7f8d2c003fd0 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.148+0000 7f8d4b400640 1 -- 172.21.15.141:0/202589351 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3402caf0 con 0x7f8d2c003fd0 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.148+0000 7f8d53000640 1 -- 172.21.15.141:0/202589351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 msgr2=0x7f8d2c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.148+0000 7f8d53000640 1 --2- 172.21.15.141:0/202589351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d2c0047c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c240 tx=0x7f8d2c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.149+0000 7f8d51000640 1 -- 172.21.15.141:0/202589351 reap_dead start 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.149+0000 7f8d53000640 1 -- 172.21.15.141:0/202589351 shutdown_connections 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.149+0000 7f8d53000640 1 -- 172.21.15.141:0/202589351 >> 172.21.15.141:0/202589351 conn(0x7f8d4c06a800 msgr2=0x7f8d4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.149+0000 7f8d53000640 1 -- 172.21.15.141:0/202589351 shutdown_connections 2024-01-16T15:11:48.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.149+0000 7f8d53000640 1 -- 172.21.15.141:0/202589351 wait complete. 2024-01-16T15:11:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.150+0000 7f8d53000640 1 Processor -- start 2024-01-16T15:11:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.151+0000 7f8d53000640 1 -- start start 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d53000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d4c1f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d53000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c1f3ae0 0x7f8d4c1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d53000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c1587f0 0x7f8d4c1f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d53000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c155e40 con 0x7f8d2c003fd0 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d53000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c158c70 con 0x7f8d4c1587f0 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d53000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c155250 con 0x7f8d4c1f3ae0 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d4c1f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d4c1f1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38976/0 (socket says 172.21.15.141:38976) 2024-01-16T15:11:48.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 -- 172.21.15.141:0/552253696 learned_addr learned my addr 172.21.15.141:0/552253696 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:48.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d43e00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c1f3ae0 0x7f8d4c1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 -- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c1f3ae0 msgr2=0x7f8d4c1582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c1f3ae0 0x7f8d4c1582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 -- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c1587f0 msgr2=0x7f8d4c1f0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d51000640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c1587f0 0x7f8d4c1f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c1587f0 0x7f8d4c1f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d4be00640 1 -- 172.21.15.141:0/552253696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c1554a0 con 0x7f8d2c003fd0 2024-01-16T15:11:48.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.152+0000 7f8d43e00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c1f3ae0 0x7f8d4c1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.153+0000 7f8d4be00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d4c1f1e00 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00e810 tx=0x7f8d3c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.153+0000 7f8d48c00640 1 -- 172.21.15.141:0/552253696 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c00c6d0 con 0x7f8d2c003fd0 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.153+0000 7f8d48c00640 1 -- 172.21.15.141:0/552253696 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c00c830 con 0x7f8d2c003fd0 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.153+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c155780 con 0x7f8d2c003fd0 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.153+0000 7f8d48c00640 1 -- 172.21.15.141:0/552253696 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c00cba0 con 0x7f8d2c003fd0 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.153+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c1f3f60 con 0x7f8d2c003fd0 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.154+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c1560b0 con 0x7f8d2c003fd0 2024-01-16T15:11:48.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.155+0000 7f8d48c00640 1 -- 172.21.15.141:0/552253696 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f8d3c00cd00 con 0x7f8d2c003fd0 2024-01-16T15:11:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.156+0000 7f8d48c00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406c1f0 0x7f8d2406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.156+0000 7f8d48c00640 1 -- 172.21.15.141:0/552253696 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d3c088440 con 0x7f8d2c003fd0 2024-01-16T15:11:48.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.157+0000 7f8d43e00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406c1f0 0x7f8d2406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.158+0000 7f8d43e00640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406c1f0 0x7f8d2406e6b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8d34034000 tx=0x7f8d34009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.160+0000 7f8d48c00640 1 -- 172.21.15.141:0/552253696 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d3c054d60 con 0x7f8d2c003fd0 2024-01-16T15:11:48.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.347+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d4c15bee0 con 0x7f8d2406c1f0 2024-01-16T15:11:48.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.369+0000 7f8d48c00640 1 -- 172.21.15.141:0/552253696 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286885 (secure 0 0 0) 0x7f8d340014b0 con 0x7f8d2406c1f0 2024-01-16T15:11:48.351 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:48.361 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.375+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406c1f0 msgr2=0x7f8d2406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406c1f0 0x7f8d2406e6b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8d34034000 tx=0x7f8d34009030 comp rx=0 tx=0).stop 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 msgr2=0x7f8d4c1f1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d4c1f1e00 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00e810 tx=0x7f8d3c00ecd0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 shutdown_connections 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8d2406c1f0 0x7f8d2406e6b0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d4c1587f0 0x7f8d4c1f0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8d4c1f3ae0 0x7f8d4c1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 --2- 172.21.15.141:0/552253696 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8d2c003fd0 0x7f8d4c1f1e00 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 >> 172.21.15.141:0/552253696 conn(0x7f8d4c06a800 msgr2=0x7f8d4c0cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 shutdown_connections 2024-01-16T15:11:48.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.376+0000 7f8d53000640 1 -- 172.21.15.141:0/552253696 wait complete. 2024-01-16T15:11:48.372 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2024-01-16T15:11:46.826527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":556952,"kb_used_data":10920,"kb_used_omap":12,"kb_used_meta":545907,"kb_avail":754417768,"statfs":{"total":773094113280,"available":772523794432,"internally_reserved":0,"allocated":11182080,"data_stored":5459742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":559009330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.911621"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951872+0000","last_change":"2024-01-16T15:11:43.980470+0000","last_active":"2024-01-16T15:11:44.951872+0000","last_peered":"2024-01-16T15:11:44.951872+0000","last_clean":"2024-01-16T15:11:44.951872+0000","last_became_active":"2024-01-16T15:11:43.980118+0000","last_became_peered":"2024-01-16T15:11:43.980118+0000","last_unstale":"2024-01-16T15:11:44.951872+0000","last_undegraded":"2024-01-16T15:11:44.951872+0000","last_fullsized":"2024-01-16T15:11:44.951872+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:18.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985054+0000","last_change":"2024-01-16T15:11:43.982509+0000","last_active":"2024-01-16T15:11:43.985054+0000","last_peered":"2024-01-16T15:11:43.985054+0000","last_clean":"2024-01-16T15:11:43.985054+0000","last_became_active":"2024-01-16T15:11:43.981847+0000","last_became_peered":"2024-01-16T15:11:43.981847+0000","last_unstale":"2024-01-16T15:11:43.985054+0000","last_undegraded":"2024-01-16T15:11:43.985054+0000","last_fullsized":"2024-01-16T15:11:43.985054+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:20.786959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951761+0000","last_change":"2024-01-16T15:11:43.969123+0000","last_active":"2024-01-16T15:11:44.951761+0000","last_peered":"2024-01-16T15:11:44.951761+0000","last_clean":"2024-01-16T15:11:44.951761+0000","last_became_active":"2024-01-16T15:11:43.968784+0000","last_became_peered":"2024-01-16T15:11:43.968784+0000","last_unstale":"2024-01-16T15:11:44.951761+0000","last_undegraded":"2024-01-16T15:11:44.951761+0000","last_fullsized":"2024-01-16T15:11:44.951761+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:19.083239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981974+0000","last_change":"2024-01-16T15:11:43.975345+0000","last_active":"2024-01-16T15:11:43.981974+0000","last_peered":"2024-01-16T15:11:43.981974+0000","last_clean":"2024-01-16T15:11:43.981974+0000","last_became_active":"2024-01-16T15:11:43.974874+0000","last_became_peered":"2024-01-16T15:11:43.974874+0000","last_unstale":"2024-01-16T15:11:43.981974+0000","last_undegraded":"2024-01-16T15:11:43.981974+0000","last_fullsized":"2024-01-16T15:11:43.981974+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:56.775943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125616+0000","last_change":"2024-01-16T15:11:43.979769+0000","last_active":"2024-01-16T15:11:45.125616+0000","last_peered":"2024-01-16T15:11:45.125616+0000","last_clean":"2024-01-16T15:11:45.125616+0000","last_became_active":"2024-01-16T15:11:43.979465+0000","last_became_peered":"2024-01-16T15:11:43.979465+0000","last_unstale":"2024-01-16T15:11:45.125616+0000","last_undegraded":"2024-01-16T15:11:45.125616+0000","last_fullsized":"2024-01-16T15:11:45.125616+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:17.534495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.950131+0000","last_change":"2024-01-16T15:11:43.982175+0000","last_active":"2024-01-16T15:11:44.950131+0000","last_peered":"2024-01-16T15:11:44.950131+0000","last_clean":"2024-01-16T15:11:44.950131+0000","last_became_active":"2024-01-16T15:11:43.981884+0000","last_became_peered":"2024-01-16T15:11:43.981884+0000","last_unstale":"2024-01-16T15:11:44.950131+0000","last_undegraded":"2024-01-16T15:11:44.950131+0000","last_fullsized":"2024-01-16T15:11:44.950131+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:20.853365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.948819+0000","last_change":"2024-01-16T15:11:43.980074+0000","last_active":"2024-01-16T15:11:44.948819+0000","last_peered":"2024-01-16T15:11:44.948819+0000","last_clean":"2024-01-16T15:11:44.948819+0000","last_became_active":"2024-01-16T15:11:43.979768+0000","last_became_peered":"2024-01-16T15:11:43.979768+0000","last_unstale":"2024-01-16T15:11:44.948819+0000","last_undegraded":"2024-01-16T15:11:44.948819+0000","last_fullsized":"2024-01-16T15:11:44.948819+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:21.612480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.989023+0000","last_change":"2024-01-16T15:11:43.987503+0000","last_active":"2024-01-16T15:11:43.989023+0000","last_peered":"2024-01-16T15:11:43.989023+0000","last_clean":"2024-01-16T15:11:43.989023+0000","last_became_active":"2024-01-16T15:11:43.986145+0000","last_became_peered":"2024-01-16T15:11:43.986145+0000","last_unstale":"2024-01-16T15:11:43.989023+0000","last_undegraded":"2024-01-16T15:11:43.989023+0000","last_fullsized":"2024-01-16T15:11:43.989023+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:17.534495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.986016+0000","last_change":"2024-01-16T15:11:43.980734+0000","last_active":"2024-01-16T15:11:43.986016+0000","last_peered":"2024-01-16T15:11:43.986016+0000","last_clean":"2024-01-16T15:11:43.986016+0000","last_became_active":"2024-01-16T15:11:43.980467+0000","last_became_peered":"2024-01-16T15:11:43.980467+0000","last_unstale":"2024-01-16T15:11:43.986016+0000","last_undegraded":"2024-01-16T15:11:43.986016+0000","last_fullsized":"2024-01-16T15:11:43.986016+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:08:52.281754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125573+0000","last_change":"2024-01-16T15:11:43.971024+0000","last_active":"2024-01-16T15:11:45.125573+0000","last_peered":"2024-01-16T15:11:45.125573+0000","last_clean":"2024-01-16T15:11:45.125573+0000","last_became_active":"2024-01-16T15:11:43.969962+0000","last_became_peered":"2024-01-16T15:11:43.969962+0000","last_unstale":"2024-01-16T15:11:45.125573+0000","last_undegraded":"2024-01-16T15:11:45.125573+0000","last_fullsized":"2024-01-16T15:11:45.125573+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:32.538245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.948815+0000","last_change":"2024-01-16T15:11:43.981999+0000","last_active":"2024-01-16T15:11:44.948815+0000","last_peered":"2024-01-16T15:11:44.948815+0000","last_clean":"2024-01-16T15:11:44.948815+0000","last_became_active":"2024-01-16T15:11:43.978354+0000","last_became_peered":"2024-01-16T15:11:43.978354+0000","last_unstale":"2024-01-16T15:11:44.948815+0000","last_undegraded":"2024-01-16T15:11:44.948815+0000","last_fullsized":"2024-01-16T15:11:44.948815+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:14.500746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.966582+0000","last_change":"2024-01-16T15:11:43.962264+0000","last_active":"2024-01-16T15:11:43.966582+0000","last_peered":"2024-01-16T15:11:43.966582+0000","last_clean":"2024-01-16T15:11:43.966582+0000","last_became_active":"2024-01-16T15:11:43.961051+0000","last_became_peered":"2024-01-16T15:11:43.961051+0000","last_unstale":"2024-01-16T15:11:43.966582+0000","last_undegraded":"2024-01-16T15:11:43.966582+0000","last_fullsized":"2024-01-16T15:11:43.966582+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:27.542322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981952+0000","last_change":"2024-01-16T15:11:43.975880+0000","last_active":"2024-01-16T15:11:43.981952+0000","last_peered":"2024-01-16T15:11:43.981952+0000","last_clean":"2024-01-16T15:11:43.981952+0000","last_became_active":"2024-01-16T15:11:43.975323+0000","last_became_peered":"2024-01-16T15:11:43.975323+0000","last_unstale":"2024-01-16T15:11:43.981952+0000","last_undegraded":"2024-01-16T15:11:43.981952+0000","last_fullsized":"2024-01-16T15:11:43.981952+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:12.797328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983508+0000","last_change":"2024-01-16T15:11:43.978831+0000","last_active":"2024-01-16T15:11:43.983508+0000","last_peered":"2024-01-16T15:11:43.983508+0000","last_clean":"2024-01-16T15:11:43.983508+0000","last_became_active":"2024-01-16T15:11:43.978521+0000","last_became_peered":"2024-01-16T15:11:43.978521+0000","last_unstale":"2024-01-16T15:11:43.983508+0000","last_undegraded":"2024-01-16T15:11:43.983508+0000","last_fullsized":"2024-01-16T15:11:43.983508+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:30.805142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981984+0000","last_change":"2024-01-16T15:11:43.975962+0000","last_active":"2024-01-16T15:11:43.981984+0000","last_peered":"2024-01-16T15:11:43.981984+0000","last_clean":"2024-01-16T15:11:43.981984+0000","last_became_active":"2024-01-16T15:11:43.975628+0000","last_became_peered":"2024-01-16T15:11:43.975628+0000","last_unstale":"2024-01-16T15:11:43.981984+0000","last_undegraded":"2024-01-16T15:11:43.981984+0000","last_fullsized":"2024-01-16T15:11:43.981984+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:04.795502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.947929+0000","last_change":"2024-01-16T15:11:43.981094+0000","last_active":"2024-01-16T15:11:44.947929+0000","last_peered":"2024-01-16T15:11:44.947929+0000","last_clean":"2024-01-16T15:11:44.947929+0000","last_became_active":"2024-01-16T15:11:43.980737+0000","last_became_peered":"2024-01-16T15:11:43.980737+0000","last_unstale":"2024-01-16T15:11:44.947929+0000","last_undegraded":"2024-01-16T15:11:44.947929+0000","last_fullsized":"2024-01-16T15:11:44.947929+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:00.946690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.988681+0000","last_change":"2024-01-16T15:11:43.986808+0000","last_active":"2024-01-16T15:11:43.988681+0000","last_peered":"2024-01-16T15:11:43.988681+0000","last_clean":"2024-01-16T15:11:43.988681+0000","last_became_active":"2024-01-16T15:11:43.986464+0000","last_became_peered":"2024-01-16T15:11:43.986464+0000","last_unstale":"2024-01-16T15:11:43.988681+0000","last_undegraded":"2024-01-16T15:11:43.988681+0000","last_fullsized":"2024-01-16T15:11:43.988681+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:36.098695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983183+0000","last_change":"2024-01-16T15:11:43.977991+0000","last_active":"2024-01-16T15:11:43.983183+0000","last_peered":"2024-01-16T15:11:43.983183+0000","last_clean":"2024-01-16T15:11:43.983183+0000","last_became_active":"2024-01-16T15:11:43.977319+0000","last_became_peered":"2024-01-16T15:11:43.977319+0000","last_unstale":"2024-01-16T15:11:43.983183+0000","last_undegraded":"2024-01-16T15:11:43.983183+0000","last_fullsized":"2024-01-16T15:11:43.983183+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:24:00.099420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.965709+0000","last_change":"2024-01-16T15:11:43.961989+0000","last_active":"2024-01-16T15:11:43.965709+0000","last_peered":"2024-01-16T15:11:43.965709+0000","last_clean":"2024-01-16T15:11:43.965709+0000","last_became_active":"2024-01-16T15:11:43.961535+0000","last_became_peered":"2024-01-16T15:11:43.961535+0000","last_unstale":"2024-01-16T15:11:43.965709+0000","last_undegraded":"2024-01-16T15:11:43.965709+0000","last_fullsized":"2024-01-16T15:11:43.965709+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:43.220508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.977141+0000","last_change":"2024-01-16T15:11:43.975833+0000","last_active":"2024-01-16T15:11:43.977141+0000","last_peered":"2024-01-16T15:11:43.977141+0000","last_clean":"2024-01-16T15:11:43.977141+0000","last_became_active":"2024-01-16T15:11:43.975453+0000","last_became_peered":"2024-01-16T15:11:43.975453+0000","last_unstale":"2024-01-16T15:11:43.977141+0000","last_undegraded":"2024-01-16T15:11:43.977141+0000","last_fullsized":"2024-01-16T15:11:43.977141+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:54.244819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981689+0000","last_change":"2024-01-16T15:11:43.977924+0000","last_active":"2024-01-16T15:11:43.981689+0000","last_peered":"2024-01-16T15:11:43.981689+0000","last_clean":"2024-01-16T15:11:43.981689+0000","last_became_active":"2024-01-16T15:11:43.977277+0000","last_became_peered":"2024-01-16T15:11:43.977277+0000","last_unstale":"2024-01-16T15:11:43.981689+0000","last_undegraded":"2024-01-16T15:11:43.981689+0000","last_fullsized":"2024-01-16T15:11:43.981689+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:58.229296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.984884+0000","last_change":"2024-01-16T15:11:43.982155+0000","last_active":"2024-01-16T15:11:43.984884+0000","last_peered":"2024-01-16T15:11:43.984884+0000","last_clean":"2024-01-16T15:11:43.984884+0000","last_became_active":"2024-01-16T15:11:43.981413+0000","last_became_peered":"2024-01-16T15:11:43.981413+0000","last_unstale":"2024-01-16T15:11:43.984884+0000","last_undegraded":"2024-01-16T15:11:43.984884+0000","last_fullsized":"2024-01-16T15:11:43.984884+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:27.023432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981386+0000","last_change":"2024-01-16T15:11:43.977635+0000","last_active":"2024-01-16T15:11:43.981386+0000","last_peered":"2024-01-16T15:11:43.981386+0000","last_clean":"2024-01-16T15:11:43.981386+0000","last_became_active":"2024-01-16T15:11:43.976856+0000","last_became_peered":"2024-01-16T15:11:43.976856+0000","last_unstale":"2024-01-16T15:11:43.981386+0000","last_undegraded":"2024-01-16T15:11:43.981386+0000","last_fullsized":"2024-01-16T15:11:43.981386+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:32.538245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.948511+0000","last_change":"2024-01-16T15:11:43.985323+0000","last_active":"2024-01-16T15:11:44.948511+0000","last_peered":"2024-01-16T15:11:44.948511+0000","last_clean":"2024-01-16T15:11:44.948511+0000","last_became_active":"2024-01-16T15:11:43.985000+0000","last_became_peered":"2024-01-16T15:11:43.985000+0000","last_unstale":"2024-01-16T15:11:44.948511+0000","last_undegraded":"2024-01-16T15:11:44.948511+0000","last_fullsized":"2024-01-16T15:11:44.948511+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:01.008552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125911+0000","last_change":"2024-01-16T15:11:43.980713+0000","last_active":"2024-01-16T15:11:45.125911+0000","last_peered":"2024-01-16T15:11:45.125911+0000","last_clean":"2024-01-16T15:11:45.125911+0000","last_became_active":"2024-01-16T15:11:43.980427+0000","last_became_peered":"2024-01-16T15:11:43.980427+0000","last_unstale":"2024-01-16T15:11:45.125911+0000","last_undegraded":"2024-01-16T15:11:45.125911+0000","last_fullsized":"2024-01-16T15:11:45.125911+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:09.570949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985089+0000","last_change":"2024-01-16T15:11:43.982697+0000","last_active":"2024-01-16T15:11:43.985089+0000","last_peered":"2024-01-16T15:11:43.985089+0000","last_clean":"2024-01-16T15:11:43.985089+0000","last_became_active":"2024-01-16T15:11:43.982272+0000","last_became_peered":"2024-01-16T15:11:43.982272+0000","last_unstale":"2024-01-16T15:11:43.985089+0000","last_undegraded":"2024-01-16T15:11:43.985089+0000","last_fullsized":"2024-01-16T15:11:43.985089+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:34.263979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949272+0000","last_change":"2024-01-16T15:11:43.988167+0000","last_active":"2024-01-16T15:11:44.949272+0000","last_peered":"2024-01-16T15:11:44.949272+0000","last_clean":"2024-01-16T15:11:44.949272+0000","last_became_active":"2024-01-16T15:11:43.987882+0000","last_became_peered":"2024-01-16T15:11:43.987882+0000","last_unstale":"2024-01-16T15:11:44.949272+0000","last_undegraded":"2024-01-16T15:11:44.949272+0000","last_fullsized":"2024-01-16T15:11:44.949272+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:22.759489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126016+0000","last_change":"2024-01-16T15:11:43.984030+0000","last_active":"2024-01-16T15:11:45.126016+0000","last_peered":"2024-01-16T15:11:45.126016+0000","last_clean":"2024-01-16T15:11:45.126016+0000","last_became_active":"2024-01-16T15:11:43.982544+0000","last_became_peered":"2024-01-16T15:11:43.982544+0000","last_unstale":"2024-01-16T15:11:45.126016+0000","last_undegraded":"2024-01-16T15:11:45.126016+0000","last_fullsized":"2024-01-16T15:11:45.126016+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:27.023432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983484+0000","last_change":"2024-01-16T15:11:43.980067+0000","last_active":"2024-01-16T15:11:43.983484+0000","last_peered":"2024-01-16T15:11:43.983484+0000","last_clean":"2024-01-16T15:11:43.983484+0000","last_became_active":"2024-01-16T15:11:43.979800+0000","last_became_peered":"2024-01-16T15:11:43.979800+0000","last_unstale":"2024-01-16T15:11:43.983484+0000","last_undegraded":"2024-01-16T15:11:43.983484+0000","last_fullsized":"2024-01-16T15:11:43.983484+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:26.954494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.972584+0000","last_change":"2024-01-16T15:11:43.964860+0000","last_active":"2024-01-16T15:11:43.972584+0000","last_peered":"2024-01-16T15:11:43.972584+0000","last_clean":"2024-01-16T15:11:43.972584+0000","last_became_active":"2024-01-16T15:11:43.964057+0000","last_became_peered":"2024-01-16T15:11:43.964057+0000","last_unstale":"2024-01-16T15:11:43.972584+0000","last_undegraded":"2024-01-16T15:11:43.972584+0000","last_fullsized":"2024-01-16T15:11:43.972584+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:23.969311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985559+0000","last_change":"2024-01-16T15:11:43.984275+0000","last_active":"2024-01-16T15:11:43.985559+0000","last_peered":"2024-01-16T15:11:43.985559+0000","last_clean":"2024-01-16T15:11:43.985559+0000","last_became_active":"2024-01-16T15:11:43.984017+0000","last_became_peered":"2024-01-16T15:11:43.984017+0000","last_unstale":"2024-01-16T15:11:43.985559+0000","last_undegraded":"2024-01-16T15:11:43.985559+0000","last_fullsized":"2024-01-16T15:11:43.985559+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:31.754450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.950944+0000","last_change":"2024-01-16T15:11:43.970636+0000","last_active":"2024-01-16T15:11:44.950944+0000","last_peered":"2024-01-16T15:11:44.950944+0000","last_clean":"2024-01-16T15:11:44.950944+0000","last_became_active":"2024-01-16T15:11:43.970299+0000","last_became_peered":"2024-01-16T15:11:43.970299+0000","last_unstale":"2024-01-16T15:11:44.950944+0000","last_undegraded":"2024-01-16T15:11:44.950944+0000","last_fullsized":"2024-01-16T15:11:44.950944+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:54.244819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.984771+0000","last_change":"2024-01-16T15:11:43.981592+0000","last_active":"2024-01-16T15:11:43.984771+0000","last_peered":"2024-01-16T15:11:43.984771+0000","last_clean":"2024-01-16T15:11:43.984771+0000","last_became_active":"2024-01-16T15:11:43.980749+0000","last_became_peered":"2024-01-16T15:11:43.980749+0000","last_unstale":"2024-01-16T15:11:43.984771+0000","last_undegraded":"2024-01-16T15:11:43.984771+0000","last_fullsized":"2024-01-16T15:11:43.984771+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:14.406536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"101'4","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.997333+0000","last_change":"2024-01-16T15:11:42.950909+0000","last_active":"2024-01-16T15:11:43.997333+0000","last_peered":"2024-01-16T15:11:43.997333+0000","last_clean":"2024-01-16T15:11:43.997333+0000","last_became_active":"2024-01-16T15:11:42.950605+0000","last_became_peered":"2024-01-16T15:11:42.950605+0000","last_unstale":"2024-01-16T15:11:43.997333+0000","last_undegraded":"2024-01-16T15:11:43.997333+0000","last_fullsized":"2024-01-16T15:11:43.997333+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:37:37.087146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.987220+0000","last_change":"2024-01-16T15:11:43.984315+0000","last_active":"2024-01-16T15:11:43.987220+0000","last_peered":"2024-01-16T15:11:43.987220+0000","last_clean":"2024-01-16T15:11:43.987220+0000","last_became_active":"2024-01-16T15:11:43.983962+0000","last_became_peered":"2024-01-16T15:11:43.983962+0000","last_unstale":"2024-01-16T15:11:43.987220+0000","last_undegraded":"2024-01-16T15:11:43.987220+0000","last_fullsized":"2024-01-16T15:11:43.987220+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:06:53.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949010+0000","last_change":"2024-01-16T15:11:42.948218+0000","last_active":"2024-01-16T15:11:44.949010+0000","last_peered":"2024-01-16T15:11:44.949010+0000","last_clean":"2024-01-16T15:11:44.949010+0000","last_became_active":"2024-01-16T15:11:42.947906+0000","last_became_peered":"2024-01-16T15:11:42.947906+0000","last_unstale":"2024-01-16T15:11:44.949010+0000","last_undegraded":"2024-01-16T15:11:44.949010+0000","last_fullsized":"2024-01-16T15:11:44.949010+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:14.172800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951554+0000","last_change":"2024-01-16T15:11:43.958604+0000","last_active":"2024-01-16T15:11:44.951554+0000","last_peered":"2024-01-16T15:11:44.951554+0000","last_clean":"2024-01-16T15:11:44.951554+0000","last_became_active":"2024-01-16T15:11:43.957374+0000","last_became_peered":"2024-01-16T15:11:43.957374+0000","last_unstale":"2024-01-16T15:11:44.951554+0000","last_undegraded":"2024-01-16T15:11:44.951554+0000","last_fullsized":"2024-01-16T15:11:44.951554+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:54.894378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.994019+0000","last_change":"2024-01-16T15:11:42.945607+0000","last_active":"2024-01-16T15:11:43.994019+0000","last_peered":"2024-01-16T15:11:43.994019+0000","last_clean":"2024-01-16T15:11:43.994019+0000","last_became_active":"2024-01-16T15:11:42.944951+0000","last_became_peered":"2024-01-16T15:11:42.944951+0000","last_unstale":"2024-01-16T15:11:43.994019+0000","last_undegraded":"2024-01-16T15:11:43.994019+0000","last_fullsized":"2024-01-16T15:11:43.994019+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:47.379076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.975983+0000","last_change":"2024-01-16T15:11:43.969634+0000","last_active":"2024-01-16T15:11:43.975983+0000","last_peered":"2024-01-16T15:11:43.975983+0000","last_clean":"2024-01-16T15:11:43.975983+0000","last_became_active":"2024-01-16T15:11:43.969231+0000","last_became_peered":"2024-01-16T15:11:43.969231+0000","last_unstale":"2024-01-16T15:11:43.975983+0000","last_undegraded":"2024-01-16T15:11:43.975983+0000","last_fullsized":"2024-01-16T15:11:43.975983+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:06.349978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.999015+0000","last_change":"2024-01-16T15:11:42.941334+0000","last_active":"2024-01-16T15:11:43.999015+0000","last_peered":"2024-01-16T15:11:43.999015+0000","last_clean":"2024-01-16T15:11:43.999015+0000","last_became_active":"2024-01-16T15:11:42.939561+0000","last_became_peered":"2024-01-16T15:11:42.939561+0000","last_unstale":"2024-01-16T15:11:43.999015+0000","last_undegraded":"2024-01-16T15:11:43.999015+0000","last_fullsized":"2024-01-16T15:11:43.999015+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:34.842850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.990794+0000","last_change":"2024-01-16T15:11:43.986599+0000","last_active":"2024-01-16T15:11:43.990794+0000","last_peered":"2024-01-16T15:11:43.990794+0000","last_clean":"2024-01-16T15:11:43.990794+0000","last_became_active":"2024-01-16T15:11:43.985316+0000","last_became_peered":"2024-01-16T15:11:43.985316+0000","last_unstale":"2024-01-16T15:11:43.990794+0000","last_undegraded":"2024-01-16T15:11:43.990794+0000","last_fullsized":"2024-01-16T15:11:43.990794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:23:01.100385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951475+0000","last_change":"2024-01-16T15:11:42.947866+0000","last_active":"2024-01-16T15:11:44.951475+0000","last_peered":"2024-01-16T15:11:44.951475+0000","last_clean":"2024-01-16T15:11:44.951475+0000","last_became_active":"2024-01-16T15:11:42.946072+0000","last_became_peered":"2024-01-16T15:11:42.946072+0000","last_unstale":"2024-01-16T15:11:44.951475+0000","last_undegraded":"2024-01-16T15:11:44.951475+0000","last_fullsized":"2024-01-16T15:11:44.951475+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:27.716130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949519+0000","last_change":"2024-01-16T15:11:43.991198+0000","last_active":"2024-01-16T15:11:44.949519+0000","last_peered":"2024-01-16T15:11:44.949519+0000","last_clean":"2024-01-16T15:11:44.949519+0000","last_became_active":"2024-01-16T15:11:43.987988+0000","last_became_peered":"2024-01-16T15:11:43.987988+0000","last_unstale":"2024-01-16T15:11:44.949519+0000","last_undegraded":"2024-01-16T15:11:44.949519+0000","last_fullsized":"2024-01-16T15:11:44.949519+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:02.663504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.993681+0000","last_change":"2024-01-16T15:11:42.951606+0000","last_active":"2024-01-16T15:11:43.993681+0000","last_peered":"2024-01-16T15:11:43.993681+0000","last_clean":"2024-01-16T15:11:43.993681+0000","last_became_active":"2024-01-16T15:11:42.950303+0000","last_became_peered":"2024-01-16T15:11:42.950303+0000","last_unstale":"2024-01-16T15:11:43.993681+0000","last_undegraded":"2024-01-16T15:11:43.993681+0000","last_fullsized":"2024-01-16T15:11:43.993681+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:08.489605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.982203+0000","last_change":"2024-01-16T15:11:43.976432+0000","last_active":"2024-01-16T15:11:43.982203+0000","last_peered":"2024-01-16T15:11:43.982203+0000","last_clean":"2024-01-16T15:11:43.982203+0000","last_became_active":"2024-01-16T15:11:43.976102+0000","last_became_peered":"2024-01-16T15:11:43.976102+0000","last_unstale":"2024-01-16T15:11:43.982203+0000","last_undegraded":"2024-01-16T15:11:43.982203+0000","last_fullsized":"2024-01-16T15:11:43.982203+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:06.232276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125631+0000","last_change":"2024-01-16T15:11:42.952992+0000","last_active":"2024-01-16T15:11:45.125631+0000","last_peered":"2024-01-16T15:11:45.125631+0000","last_clean":"2024-01-16T15:11:45.125631+0000","last_became_active":"2024-01-16T15:11:42.952619+0000","last_became_peered":"2024-01-16T15:11:42.952619+0000","last_unstale":"2024-01-16T15:11:45.125631+0000","last_undegraded":"2024-01-16T15:11:45.125631+0000","last_fullsized":"2024-01-16T15:11:45.125631+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:48.460524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.982016+0000","last_change":"2024-01-16T15:11:43.977711+0000","last_active":"2024-01-16T15:11:43.982016+0000","last_peered":"2024-01-16T15:11:43.982016+0000","last_clean":"2024-01-16T15:11:43.982016+0000","last_became_active":"2024-01-16T15:11:43.976821+0000","last_became_peered":"2024-01-16T15:11:43.976821+0000","last_unstale":"2024-01-16T15:11:43.982016+0000","last_undegraded":"2024-01-16T15:11:43.982016+0000","last_fullsized":"2024-01-16T15:11:43.982016+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:54.894378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.950883+0000","last_change":"2024-01-16T15:11:42.942480+0000","last_active":"2024-01-16T15:11:44.950883+0000","last_peered":"2024-01-16T15:11:44.950883+0000","last_clean":"2024-01-16T15:11:44.950883+0000","last_became_active":"2024-01-16T15:11:42.942226+0000","last_became_peered":"2024-01-16T15:11:42.942226+0000","last_unstale":"2024-01-16T15:11:44.950883+0000","last_undegraded":"2024-01-16T15:11:44.950883+0000","last_fullsized":"2024-01-16T15:11:44.950883+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:55.449507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985837+0000","last_change":"2024-01-16T15:11:43.984671+0000","last_active":"2024-01-16T15:11:43.985837+0000","last_peered":"2024-01-16T15:11:43.985837+0000","last_clean":"2024-01-16T15:11:43.985837+0000","last_became_active":"2024-01-16T15:11:43.984362+0000","last_became_peered":"2024-01-16T15:11:43.984362+0000","last_unstale":"2024-01-16T15:11:43.985837+0000","last_undegraded":"2024-01-16T15:11:43.985837+0000","last_fullsized":"2024-01-16T15:11:43.985837+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:31.854959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951991+0000","last_change":"2024-01-16T15:11:42.940557+0000","last_active":"2024-01-16T15:11:44.951991+0000","last_peered":"2024-01-16T15:11:44.951991+0000","last_clean":"2024-01-16T15:11:44.951991+0000","last_became_active":"2024-01-16T15:11:42.939209+0000","last_became_peered":"2024-01-16T15:11:42.939209+0000","last_unstale":"2024-01-16T15:11:44.951991+0000","last_undegraded":"2024-01-16T15:11:44.951991+0000","last_fullsized":"2024-01-16T15:11:44.951991+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:03.956372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.987460+0000","last_change":"2024-01-16T15:11:43.983616+0000","last_active":"2024-01-16T15:11:43.987460+0000","last_peered":"2024-01-16T15:11:43.987460+0000","last_clean":"2024-01-16T15:11:43.987460+0000","last_became_active":"2024-01-16T15:11:43.983269+0000","last_became_peered":"2024-01-16T15:11:43.983269+0000","last_unstale":"2024-01-16T15:11:43.987460+0000","last_undegraded":"2024-01-16T15:11:43.987460+0000","last_fullsized":"2024-01-16T15:11:43.987460+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:18.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.964996+0000","last_change":"2024-01-16T15:11:42.950112+0000","last_active":"2024-01-16T15:11:43.964996+0000","last_peered":"2024-01-16T15:11:43.964996+0000","last_clean":"2024-01-16T15:11:43.964996+0000","last_became_active":"2024-01-16T15:11:42.949829+0000","last_became_peered":"2024-01-16T15:11:42.949829+0000","last_unstale":"2024-01-16T15:11:43.964996+0000","last_undegraded":"2024-01-16T15:11:43.964996+0000","last_fullsized":"2024-01-16T15:11:43.964996+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:30.474487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126055+0000","last_change":"2024-01-16T15:11:43.978831+0000","last_active":"2024-01-16T15:11:45.126055+0000","last_peered":"2024-01-16T15:11:45.126055+0000","last_clean":"2024-01-16T15:11:45.126055+0000","last_became_active":"2024-01-16T15:11:43.978406+0000","last_became_peered":"2024-01-16T15:11:43.978406+0000","last_unstale":"2024-01-16T15:11:45.126055+0000","last_undegraded":"2024-01-16T15:11:45.126055+0000","last_fullsized":"2024-01-16T15:11:45.126055+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:09.504678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"101'3","reported_seq":15,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949902+0000","last_change":"2024-01-16T15:11:42.948216+0000","last_active":"2024-01-16T15:11:44.949902+0000","last_peered":"2024-01-16T15:11:44.949902+0000","last_clean":"2024-01-16T15:11:44.949902+0000","last_became_active":"2024-01-16T15:11:42.947911+0000","last_became_peered":"2024-01-16T15:11:42.947911+0000","last_unstale":"2024-01-16T15:11:44.949902+0000","last_undegraded":"2024-01-16T15:11:44.949902+0000","last_fullsized":"2024-01-16T15:11:44.949902+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:10.212029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949694+0000","last_change":"2024-01-16T15:11:43.986773+0000","last_active":"2024-01-16T15:11:44.949694+0000","last_peered":"2024-01-16T15:11:44.949694+0000","last_clean":"2024-01-16T15:11:44.949694+0000","last_became_active":"2024-01-16T15:11:43.986399+0000","last_became_peered":"2024-01-16T15:11:43.986399+0000","last_unstale":"2024-01-16T15:11:44.949694+0000","last_undegraded":"2024-01-16T15:11:44.949694+0000","last_fullsized":"2024-01-16T15:11:44.949694+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:44.820410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"101'6","reported_seq":17,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.000737+0000","last_change":"2024-01-16T15:11:42.935808+0000","last_active":"2024-01-16T15:11:44.000737+0000","last_peered":"2024-01-16T15:11:44.000737+0000","last_clean":"2024-01-16T15:11:44.000737+0000","last_became_active":"2024-01-16T15:11:42.935305+0000","last_became_peered":"2024-01-16T15:11:42.935305+0000","last_unstale":"2024-01-16T15:11:44.000737+0000","last_undegraded":"2024-01-16T15:11:44.000737+0000","last_fullsized":"2024-01-16T15:11:44.000737+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:23.659985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985313+0000","last_change":"2024-01-16T15:11:43.982783+0000","last_active":"2024-01-16T15:11:43.985313+0000","last_peered":"2024-01-16T15:11:43.985313+0000","last_clean":"2024-01-16T15:11:43.985313+0000","last_became_active":"2024-01-16T15:11:43.982487+0000","last_became_peered":"2024-01-16T15:11:43.982487+0000","last_unstale":"2024-01-16T15:11:43.985313+0000","last_undegraded":"2024-01-16T15:11:43.985313+0000","last_fullsized":"2024-01-16T15:11:43.985313+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:22.996185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.998760+0000","last_change":"2024-01-16T15:11:42.943355+0000","last_active":"2024-01-16T15:11:43.998760+0000","last_peered":"2024-01-16T15:11:43.998760+0000","last_clean":"2024-01-16T15:11:43.998760+0000","last_became_active":"2024-01-16T15:11:42.941670+0000","last_became_peered":"2024-01-16T15:11:42.941670+0000","last_unstale":"2024-01-16T15:11:43.998760+0000","last_undegraded":"2024-01-16T15:11:43.998760+0000","last_fullsized":"2024-01-16T15:11:43.998760+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:15.192954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951499+0000","last_change":"2024-01-16T15:11:43.966711+0000","last_active":"2024-01-16T15:11:44.951499+0000","last_peered":"2024-01-16T15:11:44.951499+0000","last_clean":"2024-01-16T15:11:44.951499+0000","last_became_active":"2024-01-16T15:11:43.966408+0000","last_became_peered":"2024-01-16T15:11:43.966408+0000","last_unstale":"2024-01-16T15:11:44.951499+0000","last_undegraded":"2024-01-16T15:11:44.951499+0000","last_fullsized":"2024-01-16T15:11:44.951499+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:03.764468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.951480+0000","last_change":"2024-01-16T15:11:42.948490+0000","last_active":"2024-01-16T15:11:43.951480+0000","last_peered":"2024-01-16T15:11:43.951480+0000","last_clean":"2024-01-16T15:11:43.951480+0000","last_became_active":"2024-01-16T15:11:42.948248+0000","last_became_peered":"2024-01-16T15:11:42.948248+0000","last_unstale":"2024-01-16T15:11:43.951480+0000","last_undegraded":"2024-01-16T15:11:43.951480+0000","last_fullsized":"2024-01-16T15:11:43.951480+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:08.179719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.976376+0000","last_change":"2024-01-16T15:11:43.968616+0000","last_active":"2024-01-16T15:11:43.976376+0000","last_peered":"2024-01-16T15:11:43.976376+0000","last_clean":"2024-01-16T15:11:43.976376+0000","last_became_active":"2024-01-16T15:11:43.968234+0000","last_became_peered":"2024-01-16T15:11:43.968234+0000","last_unstale":"2024-01-16T15:11:43.976376+0000","last_undegraded":"2024-01-16T15:11:43.976376+0000","last_fullsized":"2024-01-16T15:11:43.976376+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:23:01.100385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.948607+0000","last_change":"2024-01-16T15:11:42.947081+0000","last_active":"2024-01-16T15:11:43.948607+0000","last_peered":"2024-01-16T15:11:43.948607+0000","last_clean":"2024-01-16T15:11:43.948607+0000","last_became_active":"2024-01-16T15:11:42.946702+0000","last_became_peered":"2024-01-16T15:11:42.946702+0000","last_unstale":"2024-01-16T15:11:43.948607+0000","last_undegraded":"2024-01-16T15:11:43.948607+0000","last_fullsized":"2024-01-16T15:11:43.948607+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:18.068166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.964786+0000","last_change":"2024-01-16T15:11:43.961346+0000","last_active":"2024-01-16T15:11:43.964786+0000","last_peered":"2024-01-16T15:11:43.964786+0000","last_clean":"2024-01-16T15:11:43.964786+0000","last_became_active":"2024-01-16T15:11:43.960179+0000","last_became_peered":"2024-01-16T15:11:43.960179+0000","last_unstale":"2024-01-16T15:11:43.964786+0000","last_undegraded":"2024-01-16T15:11:43.964786+0000","last_fullsized":"2024-01-16T15:11:43.964786+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:33:42.894239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"101'8","reported_seq":20,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951028+0000","last_change":"2024-01-16T15:11:42.946994+0000","last_active":"2024-01-16T15:11:44.951028+0000","last_peered":"2024-01-16T15:11:44.951028+0000","last_clean":"2024-01-16T15:11:44.951028+0000","last_became_active":"2024-01-16T15:11:42.945711+0000","last_became_peered":"2024-01-16T15:11:42.945711+0000","last_unstale":"2024-01-16T15:11:44.951028+0000","last_undegraded":"2024-01-16T15:11:44.951028+0000","last_fullsized":"2024-01-16T15:11:44.951028+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:11.035449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125998+0000","last_change":"2024-01-16T15:11:43.968879+0000","last_active":"2024-01-16T15:11:45.125998+0000","last_peered":"2024-01-16T15:11:45.125998+0000","last_clean":"2024-01-16T15:11:45.125998+0000","last_became_active":"2024-01-16T15:11:43.968459+0000","last_became_peered":"2024-01-16T15:11:43.968459+0000","last_unstale":"2024-01-16T15:11:45.125998+0000","last_undegraded":"2024-01-16T15:11:45.125998+0000","last_fullsized":"2024-01-16T15:11:45.125998+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:07:37.262045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.996637+0000","last_change":"2024-01-16T15:11:42.947197+0000","last_active":"2024-01-16T15:11:43.996637+0000","last_peered":"2024-01-16T15:11:43.996637+0000","last_clean":"2024-01-16T15:11:43.996637+0000","last_became_active":"2024-01-16T15:11:42.946907+0000","last_became_peered":"2024-01-16T15:11:42.946907+0000","last_unstale":"2024-01-16T15:11:43.996637+0000","last_undegraded":"2024-01-16T15:11:43.996637+0000","last_fullsized":"2024-01-16T15:11:43.996637+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:58.422117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983490+0000","last_change":"2024-01-16T15:11:43.981694+0000","last_active":"2024-01-16T15:11:43.983490+0000","last_peered":"2024-01-16T15:11:43.983490+0000","last_clean":"2024-01-16T15:11:43.983490+0000","last_became_active":"2024-01-16T15:11:43.981400+0000","last_became_peered":"2024-01-16T15:11:43.981400+0000","last_unstale":"2024-01-16T15:11:43.983490+0000","last_undegraded":"2024-01-16T15:11:43.983490+0000","last_fullsized":"2024-01-16T15:11:43.983490+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:04.971445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125904+0000","last_change":"2024-01-16T15:11:42.946199+0000","last_active":"2024-01-16T15:11:45.125904+0000","last_peered":"2024-01-16T15:11:45.125904+0000","last_clean":"2024-01-16T15:11:45.125904+0000","last_became_active":"2024-01-16T15:11:42.945878+0000","last_became_peered":"2024-01-16T15:11:42.945878+0000","last_unstale":"2024-01-16T15:11:45.125904+0000","last_undegraded":"2024-01-16T15:11:45.125904+0000","last_fullsized":"2024-01-16T15:11:45.125904+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:33.893645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949657+0000","last_change":"2024-01-16T15:11:43.982943+0000","last_active":"2024-01-16T15:11:44.949657+0000","last_peered":"2024-01-16T15:11:44.949657+0000","last_clean":"2024-01-16T15:11:44.949657+0000","last_became_active":"2024-01-16T15:11:43.981974+0000","last_became_peered":"2024-01-16T15:11:43.981974+0000","last_unstale":"2024-01-16T15:11:44.949657+0000","last_undegraded":"2024-01-16T15:11:44.949657+0000","last_fullsized":"2024-01-16T15:11:44.949657+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:54.339048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.967165+0000","last_change":"2024-01-16T15:11:42.939715+0000","last_active":"2024-01-16T15:11:43.967165+0000","last_peered":"2024-01-16T15:11:43.967165+0000","last_clean":"2024-01-16T15:11:43.967165+0000","last_became_active":"2024-01-16T15:11:42.939273+0000","last_became_peered":"2024-01-16T15:11:42.939273+0000","last_unstale":"2024-01-16T15:11:43.967165+0000","last_undegraded":"2024-01-16T15:11:43.967165+0000","last_fullsized":"2024-01-16T15:11:43.967165+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:47.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981368+0000","last_change":"2024-01-16T15:11:43.978138+0000","last_active":"2024-01-16T15:11:43.981368+0000","last_peered":"2024-01-16T15:11:43.981368+0000","last_clean":"2024-01-16T15:11:43.981368+0000","last_became_active":"2024-01-16T15:11:43.975628+0000","last_became_peered":"2024-01-16T15:11:43.975628+0000","last_unstale":"2024-01-16T15:11:43.981368+0000","last_undegraded":"2024-01-16T15:11:43.981368+0000","last_fullsized":"2024-01-16T15:11:43.981368+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:56:18.840803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.994112+0000","last_change":"2024-01-16T15:11:42.945945+0000","last_active":"2024-01-16T15:11:43.994112+0000","last_peered":"2024-01-16T15:11:43.994112+0000","last_clean":"2024-01-16T15:11:43.994112+0000","last_became_active":"2024-01-16T15:11:42.945668+0000","last_became_peered":"2024-01-16T15:11:42.945668+0000","last_unstale":"2024-01-16T15:11:43.994112+0000","last_undegraded":"2024-01-16T15:11:43.994112+0000","last_fullsized":"2024-01-16T15:11:43.994112+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:11.480993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981708+0000","last_change":"2024-01-16T15:11:43.978141+0000","last_active":"2024-01-16T15:11:43.981708+0000","last_peered":"2024-01-16T15:11:43.981708+0000","last_clean":"2024-01-16T15:11:43.981708+0000","last_became_active":"2024-01-16T15:11:43.977635+0000","last_became_peered":"2024-01-16T15:11:43.977635+0000","last_unstale":"2024-01-16T15:11:43.981708+0000","last_undegraded":"2024-01-16T15:11:43.981708+0000","last_fullsized":"2024-01-16T15:11:43.981708+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:41.703547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.978046+0000","last_change":"2024-01-16T15:11:42.943533+0000","last_active":"2024-01-16T15:11:43.978046+0000","last_peered":"2024-01-16T15:11:43.978046+0000","last_clean":"2024-01-16T15:11:43.978046+0000","last_became_active":"2024-01-16T15:11:42.941668+0000","last_became_peered":"2024-01-16T15:11:42.941668+0000","last_unstale":"2024-01-16T15:11:43.978046+0000","last_undegraded":"2024-01-16T15:11:43.978046+0000","last_fullsized":"2024-01-16T15:11:43.978046+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:02.749395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.973202+0000","last_change":"2024-01-16T15:11:43.967549+0000","last_active":"2024-01-16T15:11:43.973202+0000","last_peered":"2024-01-16T15:11:43.973202+0000","last_clean":"2024-01-16T15:11:43.973202+0000","last_became_active":"2024-01-16T15:11:43.966248+0000","last_became_peered":"2024-01-16T15:11:43.966248+0000","last_unstale":"2024-01-16T15:11:43.973202+0000","last_undegraded":"2024-01-16T15:11:43.973202+0000","last_fullsized":"2024-01-16T15:11:43.973202+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:11:00.291648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.977684+0000","last_change":"2024-01-16T15:11:42.937123+0000","last_active":"2024-01-16T15:11:43.977684+0000","last_peered":"2024-01-16T15:11:43.977684+0000","last_clean":"2024-01-16T15:11:43.977684+0000","last_became_active":"2024-01-16T15:11:42.936685+0000","last_became_peered":"2024-01-16T15:11:42.936685+0000","last_unstale":"2024-01-16T15:11:43.977684+0000","last_undegraded":"2024-01-16T15:11:43.977684+0000","last_fullsized":"2024-01-16T15:11:43.977684+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:53.229746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125234+0000","last_change":"2024-01-16T15:11:43.974486+0000","last_active":"2024-01-16T15:11:45.125234+0000","last_peered":"2024-01-16T15:11:45.125234+0000","last_clean":"2024-01-16T15:11:45.125234+0000","last_became_active":"2024-01-16T15:11:43.974165+0000","last_became_peered":"2024-01-16T15:11:43.974165+0000","last_unstale":"2024-01-16T15:11:45.125234+0000","last_undegraded":"2024-01-16T15:11:45.125234+0000","last_fullsized":"2024-01-16T15:11:45.125234+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:09.604583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.965299+0000","last_change":"2024-01-16T15:11:42.951323+0000","last_active":"2024-01-16T15:11:43.965299+0000","last_peered":"2024-01-16T15:11:43.965299+0000","last_clean":"2024-01-16T15:11:43.965299+0000","last_became_active":"2024-01-16T15:11:42.950416+0000","last_became_peered":"2024-01-16T15:11:42.950416+0000","last_unstale":"2024-01-16T15:11:43.965299+0000","last_undegraded":"2024-01-16T15:11:43.965299+0000","last_fullsized":"2024-01-16T15:11:43.965299+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:07:36.246971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985074+0000","last_change":"2024-01-16T15:11:43.982550+0000","last_active":"2024-01-16T15:11:43.985074+0000","last_peered":"2024-01-16T15:11:43.985074+0000","last_clean":"2024-01-16T15:11:43.985074+0000","last_became_active":"2024-01-16T15:11:43.982165+0000","last_became_peered":"2024-01-16T15:11:43.982165+0000","last_unstale":"2024-01-16T15:11:43.985074+0000","last_undegraded":"2024-01-16T15:11:43.985074+0000","last_fullsized":"2024-01-16T15:11:43.985074+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:39.732748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.999373+0000","last_change":"2024-01-16T15:11:42.940245+0000","last_active":"2024-01-16T15:11:43.999373+0000","last_peered":"2024-01-16T15:11:43.999373+0000","last_clean":"2024-01-16T15:11:43.999373+0000","last_became_active":"2024-01-16T15:11:42.939956+0000","last_became_peered":"2024-01-16T15:11:42.939956+0000","last_unstale":"2024-01-16T15:11:43.999373+0000","last_undegraded":"2024-01-16T15:11:43.999373+0000","last_fullsized":"2024-01-16T15:11:43.999373+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:53.879305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951878+0000","last_change":"2024-01-16T15:11:43.969368+0000","last_active":"2024-01-16T15:11:44.951878+0000","last_peered":"2024-01-16T15:11:44.951878+0000","last_clean":"2024-01-16T15:11:44.951878+0000","last_became_active":"2024-01-16T15:11:43.969050+0000","last_became_peered":"2024-01-16T15:11:43.969050+0000","last_unstale":"2024-01-16T15:11:44.951878+0000","last_undegraded":"2024-01-16T15:11:44.951878+0000","last_fullsized":"2024-01-16T15:11:44.951878+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:24.675058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.974899+0000","last_change":"2024-01-16T15:11:42.946563+0000","last_active":"2024-01-16T15:11:43.974899+0000","last_peered":"2024-01-16T15:11:43.974899+0000","last_clean":"2024-01-16T15:11:43.974899+0000","last_became_active":"2024-01-16T15:11:42.946280+0000","last_became_peered":"2024-01-16T15:11:42.946280+0000","last_unstale":"2024-01-16T15:11:43.974899+0000","last_undegraded":"2024-01-16T15:11:43.974899+0000","last_fullsized":"2024-01-16T15:11:43.974899+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:40:47.802703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951690+0000","last_change":"2024-01-16T15:11:43.975625+0000","last_active":"2024-01-16T15:11:44.951690+0000","last_peered":"2024-01-16T15:11:44.951690+0000","last_clean":"2024-01-16T15:11:44.951690+0000","last_became_active":"2024-01-16T15:11:43.973975+0000","last_became_peered":"2024-01-16T15:11:43.973975+0000","last_unstale":"2024-01-16T15:11:44.951690+0000","last_undegraded":"2024-01-16T15:11:44.951690+0000","last_fullsized":"2024-01-16T15:11:44.951690+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:27.542322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.964374+0000","last_change":"2024-01-16T15:11:42.944690+0000","last_active":"2024-01-16T15:11:43.964374+0000","last_peered":"2024-01-16T15:11:43.964374+0000","last_clean":"2024-01-16T15:11:43.964374+0000","last_became_active":"2024-01-16T15:11:42.942599+0000","last_became_peered":"2024-01-16T15:11:42.942599+0000","last_unstale":"2024-01-16T15:11:43.964374+0000","last_undegraded":"2024-01-16T15:11:43.964374+0000","last_fullsized":"2024-01-16T15:11:43.964374+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:08.555876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125371+0000","last_change":"2024-01-16T15:11:43.984063+0000","last_active":"2024-01-16T15:11:45.125371+0000","last_peered":"2024-01-16T15:11:45.125371+0000","last_clean":"2024-01-16T15:11:45.125371+0000","last_became_active":"2024-01-16T15:11:43.983805+0000","last_became_peered":"2024-01-16T15:11:43.983805+0000","last_unstale":"2024-01-16T15:11:45.125371+0000","last_undegraded":"2024-01-16T15:11:45.125371+0000","last_fullsized":"2024-01-16T15:11:45.125371+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:48.394149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126150+0000","last_change":"2024-01-16T15:11:42.949900+0000","last_active":"2024-01-16T15:11:45.126150+0000","last_peered":"2024-01-16T15:11:45.126150+0000","last_clean":"2024-01-16T15:11:45.126150+0000","last_became_active":"2024-01-16T15:11:42.949210+0000","last_became_peered":"2024-01-16T15:11:42.949210+0000","last_unstale":"2024-01-16T15:11:45.126150+0000","last_undegraded":"2024-01-16T15:11:45.126150+0000","last_fullsized":"2024-01-16T15:11:45.126150+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:38.536768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.972610+0000","last_change":"2024-01-16T15:11:43.965937+0000","last_active":"2024-01-16T15:11:43.972610+0000","last_peered":"2024-01-16T15:11:43.972610+0000","last_clean":"2024-01-16T15:11:43.972610+0000","last_became_active":"2024-01-16T15:11:43.964521+0000","last_became_peered":"2024-01-16T15:11:43.964521+0000","last_unstale":"2024-01-16T15:11:43.972610+0000","last_undegraded":"2024-01-16T15:11:43.972610+0000","last_fullsized":"2024-01-16T15:11:43.972610+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:20.049382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.973902+0000","last_change":"2024-01-16T15:11:42.949691+0000","last_active":"2024-01-16T15:11:43.973902+0000","last_peered":"2024-01-16T15:11:43.973902+0000","last_clean":"2024-01-16T15:11:43.973902+0000","last_became_active":"2024-01-16T15:11:42.946557+0000","last_became_peered":"2024-01-16T15:11:42.946557+0000","last_unstale":"2024-01-16T15:11:43.973902+0000","last_undegraded":"2024-01-16T15:11:43.973902+0000","last_fullsized":"2024-01-16T15:11:43.973902+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:00.048547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.977035+0000","last_change":"2024-01-16T15:11:43.974388+0000","last_active":"2024-01-16T15:11:43.977035+0000","last_peered":"2024-01-16T15:11:43.977035+0000","last_clean":"2024-01-16T15:11:43.977035+0000","last_became_active":"2024-01-16T15:11:43.972973+0000","last_became_peered":"2024-01-16T15:11:43.972973+0000","last_unstale":"2024-01-16T15:11:43.977035+0000","last_undegraded":"2024-01-16T15:11:43.977035+0000","last_fullsized":"2024-01-16T15:11:43.977035+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:03.764468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125043+0000","last_change":"2024-01-16T15:11:42.948209+0000","last_active":"2024-01-16T15:11:45.125043+0000","last_peered":"2024-01-16T15:11:45.125043+0000","last_clean":"2024-01-16T15:11:45.125043+0000","last_became_active":"2024-01-16T15:11:42.947919+0000","last_became_peered":"2024-01-16T15:11:42.947919+0000","last_unstale":"2024-01-16T15:11:45.125043+0000","last_undegraded":"2024-01-16T15:11:45.125043+0000","last_fullsized":"2024-01-16T15:11:45.125043+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:50.653737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.965210+0000","last_change":"2024-01-16T15:11:43.962876+0000","last_active":"2024-01-16T15:11:43.965210+0000","last_peered":"2024-01-16T15:11:43.965210+0000","last_clean":"2024-01-16T15:11:43.965210+0000","last_became_active":"2024-01-16T15:11:43.962156+0000","last_became_peered":"2024-01-16T15:11:43.962156+0000","last_unstale":"2024-01-16T15:11:43.965210+0000","last_undegraded":"2024-01-16T15:11:43.965210+0000","last_fullsized":"2024-01-16T15:11:43.965210+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:14:58.885498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.963709+0000","last_change":"2024-01-16T15:11:42.942997+0000","last_active":"2024-01-16T15:11:43.963709+0000","last_peered":"2024-01-16T15:11:43.963709+0000","last_clean":"2024-01-16T15:11:43.963709+0000","last_became_active":"2024-01-16T15:11:42.941797+0000","last_became_peered":"2024-01-16T15:11:42.941797+0000","last_unstale":"2024-01-16T15:11:43.963709+0000","last_undegraded":"2024-01-16T15:11:43.963709+0000","last_fullsized":"2024-01-16T15:11:43.963709+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:57.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126130+0000","last_change":"2024-01-16T15:11:43.985660+0000","last_active":"2024-01-16T15:11:45.126130+0000","last_peered":"2024-01-16T15:11:45.126130+0000","last_clean":"2024-01-16T15:11:45.126130+0000","last_became_active":"2024-01-16T15:11:43.984537+0000","last_became_peered":"2024-01-16T15:11:43.984537+0000","last_unstale":"2024-01-16T15:11:45.126130+0000","last_undegraded":"2024-01-16T15:11:45.126130+0000","last_fullsized":"2024-01-16T15:11:45.126130+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:48.585576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.951058+0000","last_change":"2024-01-16T15:11:42.948507+0000","last_active":"2024-01-16T15:11:43.951058+0000","last_peered":"2024-01-16T15:11:43.951058+0000","last_clean":"2024-01-16T15:11:43.951058+0000","last_became_active":"2024-01-16T15:11:42.948148+0000","last_became_peered":"2024-01-16T15:11:42.948148+0000","last_unstale":"2024-01-16T15:11:43.951058+0000","last_undegraded":"2024-01-16T15:11:43.951058+0000","last_fullsized":"2024-01-16T15:11:43.951058+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:30:53.806477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.984508+0000","last_change":"2024-01-16T15:11:43.979245+0000","last_active":"2024-01-16T15:11:43.984508+0000","last_peered":"2024-01-16T15:11:43.984508+0000","last_clean":"2024-01-16T15:11:43.984508+0000","last_became_active":"2024-01-16T15:11:43.978948+0000","last_became_peered":"2024-01-16T15:11:43.978948+0000","last_unstale":"2024-01-16T15:11:43.984508+0000","last_undegraded":"2024-01-16T15:11:43.984508+0000","last_fullsized":"2024-01-16T15:11:43.984508+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:36.704517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949799+0000","last_change":"2024-01-16T15:11:42.946730+0000","last_active":"2024-01-16T15:11:44.949799+0000","last_peered":"2024-01-16T15:11:44.949799+0000","last_clean":"2024-01-16T15:11:44.949799+0000","last_became_active":"2024-01-16T15:11:42.946446+0000","last_became_peered":"2024-01-16T15:11:42.946446+0000","last_unstale":"2024-01-16T15:11:44.949799+0000","last_undegraded":"2024-01-16T15:11:44.949799+0000","last_fullsized":"2024-01-16T15:11:44.949799+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:27.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":245,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.958912+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:11:43.958912+0000","last_peered":"2024-01-16T15:11:43.958912+0000","last_clean":"2024-01-16T15:11:43.958912+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:11:43.958912+0000","last_undegraded":"2024-01-16T15:11:43.958912+0000","last_fullsized":"2024-01-16T15:11:43.958912+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542324,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69020,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302820,"statfs":{"total":96636764160,"available":96566087680,"internally_reserved":0,"allocated":1847296,"data_stored":1126460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7230000000000001,"15min":1.504},"min":{"1min":0.57799999999999996,"5min":0.502,"15min":0.37},"max":{"1min":10.679,"5min":19.603000000000002,"15min":19.603000000000002},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7050000000000001,"15min":1.409},"min":{"1min":0.63600000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":10.728,"5min":24.015000000000001,"15min":24.015000000000001},"last":1.6839999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.552,"15min":1.4390000000000001},"min":{"1min":0.65600000000000003,"5min":0.5,"15min":0.40999999999999998},"max":{"1min":3.0139999999999998,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.6499999999999999,"15min":1.446},"min":{"1min":0.77800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.575,"5min":24.07,"15min":24.07},"last":1.198}]},{"osd":2,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.579,"15min":1.375},"min":{"1min":0.81499999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6319999999999999,"5min":24.108000000000001,"15min":24.108000000000001},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.6799999999999999,"15min":1.5069999999999999},"min":{"1min":0.86399999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.7210000000000001,"5min":24.806000000000001,"15min":24.806000000000001},"last":1.1220000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.96,"15min":1.7370000000000001},"min":{"1min":0.80200000000000005,"5min":0.67000000000000004,"15min":0.52600000000000002},"max":{"1min":2.427,"5min":23.741,"15min":23.741},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.365,"5min":2.012,"15min":1.6719999999999999},"min":{"1min":1.143,"5min":0.626,"15min":0.626},"max":{"1min":1.635,"5min":24.143000000000001,"15min":24.143000000000001},"last":1.5920000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":2.04,"15min":1.6679999999999999},"min":{"1min":0.79300000000000004,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":1.627,"5min":25.637,"15min":25.637},"last":1.722},{"interface":"front","average":{"1min":1.6759999999999999,"5min":2.2789999999999999,"15min":1.8680000000000001},"min":{"1min":0.97999999999999998,"5min":0.48099999999999998,"15min":0.44400000000000001},"max":{"1min":3.7879999999999998,"5min":25.594000000000001,"15min":25.594000000000001},"last":1.7390000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.897,"15min":1.736},"min":{"1min":1.1279999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.756,"5min":24.178999999999998,"15min":29.567},"last":1.494},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.8120000000000001,"15min":1.802},"min":{"1min":1.0680000000000001,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":1.845,"5min":24.538,"15min":27.937000000000001},"last":1.5409999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":2.1640000000000001,"15min":1.905},"min":{"1min":1.2350000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.758,"5min":36.826000000000001,"15min":41.206000000000003},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.633,"5min":2.5430000000000001,"15min":2.1509999999999998},"min":{"1min":1.089,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.847,"5min":39.932000000000002,"15min":39.932000000000002},"last":1.641}]}]},{"osd":6,"up_from":14,"seq":60129542323,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68436,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303404,"statfs":{"total":96636764160,"available":96566685696,"internally_reserved":0,"allocated":1253376,"data_stored":536632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.468,"15min":1.4219999999999999},"min":{"1min":0.64700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.919,"5min":12.467000000000001,"15min":20.739000000000001},"last":1.335},{"interface":"front","average":{"1min":1.28,"5min":1.55,"15min":1.613},"min":{"1min":0.70899999999999996,"5min":0.50600000000000001,"15min":0.498},"max":{"1min":1.798,"5min":15.401,"15min":50.472000000000001},"last":1.24}]},{"osd":1,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.48,"15min":1.385},"min":{"1min":0.73199999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":10.518000000000001,"5min":10.518000000000001,"15min":10.629},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.323,"5min":1.583,"15min":1.488},"min":{"1min":0.874,"5min":0.61799999999999999,"15min":0.53400000000000003},"max":{"1min":2.5590000000000002,"5min":9.6880000000000006,"15min":9.6880000000000006},"last":0.78300000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.569,"15min":1.633},"min":{"1min":0.65800000000000003,"5min":0.50800000000000001,"15min":0.46899999999999997},"max":{"1min":1.49,"5min":15.576000000000001,"15min":50.643999999999998},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.8560000000000001,"5min":1.6910000000000001,"15min":1.54},"min":{"1min":0.89100000000000001,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":10.728,"5min":10.728,"15min":10.843},"last":1.2769999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.609,"15min":1.595},"min":{"1min":1.0880000000000001,"5min":0.621,"15min":0.54900000000000004},"max":{"1min":3.0259999999999998,"5min":9.7460000000000004,"15min":15.675000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.796,"15min":1.7589999999999999},"min":{"1min":0.94899999999999995,"5min":0.55500000000000005,"15min":0.52600000000000002},"max":{"1min":2.609,"5min":15.504,"15min":50.561},"last":1.589}]},{"osd":4,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.6339999999999999,"15min":1.8240000000000001},"min":{"1min":0.97499999999999998,"5min":0.49299999999999999,"15min":0.46300000000000002},"max":{"1min":3.1320000000000001,"5min":7.391,"15min":47.488},"last":1.663},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.738,"15min":1.9219999999999999},"min":{"1min":0.99299999999999999,"5min":0.41299999999999998,"15min":0.40400000000000003},"max":{"1min":2.2200000000000002,"5min":15.638,"15min":50.381},"last":1.708}]},{"osd":5,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":1.631,"15min":1.5800000000000001},"min":{"1min":0.94099999999999995,"5min":0.443,"15min":0.443},"max":{"1min":10.388999999999999,"5min":10.388999999999999,"15min":10.654},"last":1.401},{"interface":"front","average":{"1min":1.712,"5min":1.97,"15min":1.8240000000000001},"min":{"1min":1.163,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.6640000000000001,"5min":18.699000000000002,"15min":18.699000000000002},"last":1.462}]},{"osd":7,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.9430000000000001,"15min":2.0800000000000001},"min":{"1min":0.71199999999999997,"5min":0.46600000000000003,"15min":0.44900000000000001},"max":{"1min":2.7109999999999999,"5min":15.702,"15min":52.912999999999997},"last":1.8360000000000001},{"interface":"front","average":{"1min":2.056,"5min":2.2650000000000001,"15min":2.3359999999999999},"min":{"1min":1.0409999999999999,"5min":0.495,"15min":0.495},"max":{"1min":10.659000000000001,"5min":16.690000000000001,"15min":26.413},"last":1.5169999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542323,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":1232896,"data_stored":529205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.298,"15min":1.357},"min":{"1min":0.78000000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.222,"5min":5.8369999999999997,"15min":13.938000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.585,"5min":1.351,"15min":1.3959999999999999},"min":{"1min":0.77300000000000002,"5min":0.56699999999999995,"15min":0.42299999999999999},"max":{"1min":5.4039999999999999,"5min":6.0010000000000003,"15min":16.896000000000001},"last":0.75700000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.4550000000000001,"15min":1.484},"min":{"1min":0.85499999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":8.1820000000000004,"5min":8.1820000000000004,"15min":15.500999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.23,"15min":1.4199999999999999},"min":{"1min":0.73399999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.2970000000000002,"5min":6.7720000000000002,"15min":35.328000000000003},"last":1.159}]},{"osd":2,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.355,"15min":1.5209999999999999},"min":{"1min":0.75900000000000001,"5min":0.52000000000000002,"15min":0.46200000000000002},"max":{"1min":3.0110000000000001,"5min":6.9020000000000001,"15min":35.398000000000003},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3799999999999999,"15min":1.419},"min":{"1min":0.81999999999999995,"5min":0.47999999999999998,"15min":0.38900000000000001},"max":{"1min":2.5550000000000002,"5min":8.0619999999999994,"15min":15.4},"last":1.254}]},{"osd":3,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.8200000000000001,"15min":1.7889999999999999},"min":{"1min":0.98399999999999999,"5min":0.63200000000000001,"15min":0.50900000000000001},"max":{"1min":8.0809999999999995,"5min":16.931999999999999,"15min":24.254000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.6850000000000001,"15min":1.8169999999999999},"min":{"1min":0.89600000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"max":{"1min":3.1619999999999999,"5min":19.111999999999998,"15min":35.268000000000001},"last":1.468}]},{"osd":4,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.3759999999999999,"15min":1.6599999999999999},"min":{"1min":0.83199999999999996,"5min":0.42199999999999999,"15min":0.39000000000000001},"max":{"1min":3.2599999999999998,"5min":7.2569999999999997,"15min":19.059000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5409999999999999,"15min":1.8120000000000001},"min":{"1min":1.1279999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":28.760000000000002},"last":1.5509999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.8620000000000001,"15min":1.897},"min":{"1min":1.0569999999999999,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":3.3220000000000001,"5min":17.419,"15min":35.427999999999997},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.643,"5min":1.903,"15min":1.889},"min":{"1min":1.173,"5min":0.53500000000000003,"15min":0.48199999999999998},"max":{"1min":3.391,"5min":16.262,"15min":16.262},"last":1.502}]},{"osd":7,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":2.0590000000000002,"5min":1.9730000000000001,"15min":2.0089999999999999},"min":{"1min":1.117,"5min":0.60899999999999999,"15min":0.41099999999999998},"max":{"1min":8.2509999999999994,"5min":12.385999999999999,"15min":29.672999999999998},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6579999999999999,"15min":1.827},"min":{"1min":1.216,"5min":0.47599999999999998,"15min":0.42899999999999999},"max":{"1min":3.2229999999999999,"5min":14.02,"15min":35.363},"last":1.363}]}]},{"osd":4,"up_from":14,"seq":60129542324,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64292,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307548,"statfs":{"total":96636764160,"available":96570929152,"internally_reserved":0,"allocated":1265664,"data_stored":537716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.639,"15min":1.3999999999999999},"min":{"1min":0.69699999999999995,"5min":0.59399999999999997,"15min":0.52800000000000002},"max":{"1min":1.623,"5min":15.942,"15min":15.942},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.3500000000000001,"15min":1.425},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.45400000000000001},"max":{"1min":1.661,"5min":16.219999999999999,"15min":16.219999999999999},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.468,"15min":1.474},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.54100000000000004},"max":{"1min":1.9490000000000001,"5min":16.280000000000001,"15min":16.280000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.7,"15min":1.48},"min":{"1min":0.81999999999999995,"5min":0.68899999999999995,"15min":0.60499999999999998},"max":{"1min":3.411,"5min":15.888,"15min":15.888},"last":1.034}]},{"osd":2,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.25,"15min":1.5049999999999999},"min":{"1min":0.76300000000000001,"5min":0.70999999999999996,"15min":0.50800000000000001},"max":{"1min":1.4790000000000001,"5min":5.2859999999999996,"15min":15.986000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.897,"15min":1.5920000000000001},"min":{"1min":0.878,"5min":0.64900000000000002,"15min":0.64800000000000002},"max":{"1min":1.9239999999999999,"5min":17.175000000000001,"15min":17.175000000000001},"last":1.319}]},{"osd":3,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.5820000000000001,"15min":1.7569999999999999},"min":{"1min":0.83699999999999997,"5min":0.71499999999999997,"15min":0.57599999999999996},"max":{"1min":1.643,"5min":16.469000000000001,"15min":16.469000000000001},"last":1.379},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.415,"15min":1.5820000000000001},"min":{"1min":0.86399999999999999,"5min":0.71699999999999997,"15min":0.55600000000000005},"max":{"1min":1.6020000000000001,"5min":5.2469999999999999,"15min":21.706},"last":1.248}]},{"osd":5,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.825,"15min":1.609},"min":{"1min":0.83799999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.9279999999999999,"5min":18.347999999999999,"15min":18.347999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.5960000000000001,"15min":1.643},"min":{"1min":0.86399999999999999,"5min":0.58299999999999996,"15min":0.45800000000000002},"max":{"1min":1.736,"5min":22.937000000000001,"15min":22.937000000000001},"last":1.611}]},{"osd":6,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.6619999999999999,"15min":1.5720000000000001},"min":{"1min":0.877,"5min":0.60699999999999998,"15min":0.501},"max":{"1min":1.7110000000000001,"5min":18.033000000000001,"15min":18.033000000000001},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.482,"5min":2.0950000000000002,"15min":1.8759999999999999},"min":{"1min":1.3140000000000001,"5min":0.52400000000000002,"15min":0.51800000000000002},"max":{"1min":2.133,"5min":17.794,"15min":20.939},"last":1.5660000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.708,"15min":1.7509999999999999},"min":{"1min":0.97699999999999998,"5min":0.53500000000000003,"15min":0.44500000000000001},"max":{"1min":1.873,"5min":16.324999999999999,"15min":16.324999999999999},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.764,"15min":1.851},"min":{"1min":1.22,"5min":0.55200000000000005,"15min":0.52200000000000002},"max":{"1min":1.893,"5min":12.335000000000001,"15min":20.277999999999999},"last":1.655}]}]},{"osd":3,"up_from":14,"seq":60129542323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72608,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299232,"statfs":{"total":96636764160,"available":96562413568,"internally_reserved":0,"allocated":1261568,"data_stored":537614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.95099999999999996,"15min":0.98099999999999998},"min":{"1min":0.61599999999999999,"5min":0.34300000000000003,"15min":0.33600000000000002},"max":{"1min":1.599,"5min":4.3700000000000001,"15min":5.3570000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.137,"15min":1.099},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.5859999999999999,"5min":12.292999999999999,"15min":12.292999999999999},"last":1.3200000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.97099999999999997,"15min":0.92900000000000005},"min":{"1min":0.65400000000000003,"5min":0.33100000000000002,"15min":0.309},"max":{"1min":1.8380000000000001,"5min":4.0979999999999999,"15min":4.0979999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.119,"15min":1.1499999999999999},"min":{"1min":0.624,"5min":0.315,"15min":0.315},"max":{"1min":1.861,"5min":5.0359999999999996,"15min":13.443},"last":1.117}]},{"osd":2,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.1890000000000001,"15min":1.1399999999999999},"min":{"1min":0.69999999999999996,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":4.4850000000000003},"last":1.071},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.236,"15min":1.2210000000000001},"min":{"1min":0.746,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.129,"5min":5.4539999999999997,"15min":5.5110000000000001},"last":1.242}]},{"osd":4,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3839999999999999,"15min":1.353},"min":{"1min":0.71999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4,"5min":12.744,"15min":13.292},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.4330000000000001,"15min":1.4510000000000001},"min":{"1min":0.94399999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.4590000000000001,"5min":6.4279999999999999,"15min":11.068},"last":1.5249999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.5669999999999999,"15min":1.5449999999999999},"min":{"1min":1.0900000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.29,"5min":17.096,"15min":17.096},"last":1.292},{"interface":"front","average":{"1min":1.492,"5min":1.5149999999999999,"15min":1.552},"min":{"1min":1.1439999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.2639999999999998,"5min":9.3629999999999995,"15min":12.106},"last":1.3740000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.1909999999999998,"15min":1.736},"min":{"1min":0.80100000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":3.8239999999999998,"5min":43.609999999999999,"15min":43.609999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.903,"15min":1.6779999999999999},"min":{"1min":1.056,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.7799999999999998,"5min":38.784999999999997,"15min":38.784999999999997},"last":1.448}]},{"osd":7,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.8999999999999999,"15min":1.712},"min":{"1min":0.95999999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":7.8899999999999997,"5min":25.071000000000002,"15min":25.071000000000002},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.806,"15min":1.7549999999999999},"min":{"1min":0.93400000000000005,"5min":0.435,"15min":0.435},"max":{"1min":4.2880000000000003,"5min":13.365,"15min":18.846},"last":1.4219999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68420,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303420,"statfs":{"total":96636764160,"available":96566702080,"internally_reserved":0,"allocated":1232896,"data_stored":529205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.89600000000000002,"15min":1.002},"min":{"1min":0.5,"5min":0.311,"15min":0.224},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":11.804},"last":0.5},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.054,"15min":1.1200000000000001},"min":{"1min":0.443,"5min":0.435,"15min":0.35799999999999998},"max":{"1min":2.8450000000000002,"5min":2.8839999999999999,"15min":13.066000000000001},"last":0.443}]},{"osd":1,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.107,"15min":1.1519999999999999},"min":{"1min":0.52000000000000002,"5min":0.434,"15min":0.29899999999999999},"max":{"1min":2.2599999999999998,"5min":4.1440000000000001,"15min":14.199999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.143,"15min":1.196},"min":{"1min":0.52800000000000002,"5min":0.45400000000000001,"15min":0.372},"max":{"1min":2.9900000000000002,"5min":4.056,"15min":13.15},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.98599999999999999,"15min":1.1359999999999999},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.31900000000000001},"max":{"1min":2.605,"5min":2.605,"15min":14.965},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.1850000000000001,"15min":1.2509999999999999},"min":{"1min":0.57599999999999996,"5min":0.47999999999999998,"15min":0.39300000000000002},"max":{"1min":2.3370000000000002,"5min":2.7890000000000001,"15min":11.717000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.5600000000000001,"15min":1.6539999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.46300000000000002},"max":{"1min":2.3010000000000002,"5min":11.558,"15min":19.957999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.552,"15min":1.5920000000000001},"min":{"1min":0.95899999999999996,"5min":0.71099999999999997,"15min":0.55700000000000005},"max":{"1min":3.419,"5min":11.58,"15min":15.151},"last":0.95899999999999996}]},{"osd":5,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.397,"15min":1.5780000000000001},"min":{"1min":0.79800000000000004,"5min":0.61099999999999999,"15min":0.60899999999999999},"max":{"1min":3.6160000000000001,"5min":3.6160000000000001,"15min":26.501000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.5429999999999999,"15min":1.4430000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.47999999999999998},"max":{"1min":3.1949999999999998,"5min":13.760999999999999,"15min":13.760999999999999},"last":1.0109999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.454,"15min":1.488},"min":{"1min":0.89200000000000002,"5min":0.86599999999999999,"15min":0.438},"max":{"1min":3.1200000000000001,"5min":4.4269999999999996,"15min":15.144},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.522,"5min":1.5229999999999999,"15min":1.8440000000000001},"min":{"1min":0.92800000000000005,"5min":0.71099999999999997,"15min":0.48199999999999998},"max":{"1min":3.2469999999999999,"5min":5.1310000000000002,"15min":66.061999999999998},"last":0.995}]},{"osd":7,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.5960000000000001,"15min":1.6899999999999999},"min":{"1min":0.874,"5min":0.871,"15min":0.61099999999999999},"max":{"1min":3.7290000000000001,"5min":7.5700000000000003,"15min":16.667000000000002},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.5720000000000001,"15min":1.6160000000000001},"min":{"1min":0.93400000000000005,"5min":0.89800000000000002,"15min":0.5},"max":{"1min":3.1600000000000001,"5min":7.5960000000000001,"15min":11.271000000000001},"last":1.0589999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542323,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68428,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303412,"statfs":{"total":96636764160,"available":96566693888,"internally_reserved":0,"allocated":1245184,"data_stored":535980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.0609999999999999,"15min":1.0900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.248},"max":{"1min":1.8089999999999999,"5min":8.7219999999999995,"15min":21.260000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1459999999999999,"15min":1.272},"min":{"1min":0.59599999999999997,"5min":0.47199999999999998,"15min":0.35199999999999998},"max":{"1min":1.712,"5min":5.75,"15min":32.787999999999997},"last":0.61399999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.999,"15min":1.0049999999999999},"min":{"1min":0.52900000000000003,"5min":0.34899999999999998,"15min":0.28100000000000003},"max":{"1min":2.226,"5min":7.3970000000000002,"15min":14.817},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2250000000000001,"15min":1.169},"min":{"1min":0.80300000000000005,"5min":0.42599999999999999,"15min":0.41499999999999998},"max":{"1min":1.7769999999999999,"5min":8.8040000000000003,"15min":8.8450000000000006},"last":0.94699999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.343,"15min":1.234},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":1.907,"5min":9.2330000000000005,"15min":9.3149999999999995},"last":1.113},{"interface":"front","average":{"1min":1.159,"5min":1.163,"15min":1.323},"min":{"1min":0.747,"5min":0.45500000000000002,"15min":0.40400000000000003},"max":{"1min":1.871,"5min":8.6910000000000007,"15min":41.133000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.55,"15min":1.5529999999999999},"min":{"1min":0.78800000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.0790000000000002,"5min":9.9700000000000006,"15min":19.684000000000001},"last":1.05},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4179999999999999,"15min":1.448},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.53900000000000003},"max":{"1min":2,"5min":10.785,"15min":13.789999999999999},"last":1.0760000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.54,"15min":1.5649999999999999},"min":{"1min":0.77000000000000002,"5min":0.622,"15min":0.43099999999999999},"max":{"1min":1.9059999999999999,"5min":7.617,"15min":15.778},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.494,"5min":1.6839999999999999,"15min":1.774},"min":{"1min":0.84099999999999997,"5min":0.82399999999999995,"15min":0.623},"max":{"1min":2.0390000000000001,"5min":9.4890000000000008,"15min":35.747},"last":1.4219999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.679,"15min":1.663},"min":{"1min":0.82799999999999996,"5min":0.64400000000000002,"15min":0.629},"max":{"1min":2.1920000000000002,"5min":15.586,"15min":16.231000000000002},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.6359999999999999,"15min":1.577},"min":{"1min":0.69299999999999995,"5min":0.63200000000000001,"15min":0.56599999999999995},"max":{"1min":4.1349999999999998,"5min":11.433999999999999,"15min":13.56},"last":1.264}]},{"osd":7,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.8460000000000001,"15min":1.8080000000000001},"min":{"1min":0.72599999999999998,"5min":0.66900000000000004,"15min":0.58699999999999997},"max":{"1min":2.1909999999999998,"5min":18.907,"15min":21.902999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.659,"15min":1.8120000000000001},"min":{"1min":0.86099999999999999,"5min":0.70299999999999996,"15min":0.64700000000000002},"max":{"1min":2.1259999999999999,"5min":5.1260000000000003,"15min":31.044},"last":0.83199999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69012,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302828,"statfs":{"total":96636764160,"available":96566095872,"internally_reserved":0,"allocated":1843200,"data_stored":1126930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.873,"15min":1.004},"min":{"1min":0.63500000000000001,"5min":0.29499999999999998,"15min":0.25},"max":{"1min":1.4179999999999999,"5min":2.7320000000000002,"15min":22.619},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.052,"15min":1.1719999999999999},"min":{"1min":0.56599999999999995,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.403,"5min":9.4499999999999993,"15min":17.876000000000001},"last":1.093}]},{"osd":2,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.3620000000000001,"15min":1.3759999999999999},"min":{"1min":0.628,"5min":0.45100000000000001,"15min":0.42999999999999999},"max":{"1min":5.3860000000000001,"5min":9.5579999999999998,"15min":17.760999999999999},"last":1.036},{"interface":"front","average":{"1min":1.383,"5min":1.181,"15min":1.325},"min":{"1min":0.69799999999999995,"5min":0.34999999999999998,"15min":0.33100000000000002},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":22.672999999999998},"last":1.4370000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.0409999999999999,"15min":1.1659999999999999},"min":{"1min":0.71799999999999997,"5min":0.40000000000000002,"15min":0.39200000000000002},"max":{"1min":6.7850000000000001,"5min":6.7850000000000001,"15min":16.382999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.143,"5min":1.331,"15min":1.349},"min":{"1min":0.85099999999999998,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":1.621,"5min":9.6029999999999998,"15min":17.981000000000002},"last":0.90900000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.339,"15min":1.538},"min":{"1min":1.0860000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.2330000000000001,"5min":5.109,"15min":22.780000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.262,"15min":1.5109999999999999},"min":{"1min":0.80700000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.181,"5min":3.028,"15min":19.422000000000001},"last":1.2110000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.534,"15min":1.847},"min":{"1min":1.071,"5min":0.441,"15min":0.441},"max":{"1min":2.1139999999999999,"5min":7.9530000000000003,"15min":44.930999999999997},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.798,"15min":1.73},"min":{"1min":0.94299999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":3.859,"5min":21.797999999999998,"15min":21.797999999999998},"last":1.2889999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.829,"15min":1.887},"min":{"1min":1.1539999999999999,"5min":0.622,"15min":0.622},"max":{"1min":2.3780000000000001,"5min":14.958,"15min":27.442},"last":1.264},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.6699999999999999,"15min":1.903},"min":{"1min":1.0409999999999999,"5min":0.57399999999999995,"15min":0.50700000000000001},"max":{"1min":2.2999999999999998,"5min":9.8759999999999994,"15min":27.411999999999999},"last":1.532}]},{"osd":7,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.7230000000000001,"15min":1.7270000000000001},"min":{"1min":1.1739999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2669999999999999,"5min":15.005000000000001,"15min":18.152999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.843,"5min":1.645,"15min":1.7589999999999999},"min":{"1min":1.1339999999999999,"5min":0.63700000000000001,"15min":0.60099999999999998},"max":{"1min":5.2359999999999998,"5min":9.6259999999999994,"15min":22.815000000000001},"last":1.327}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:11:48.377 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:11:48.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.645+0000 7f2f40e00640 1 Processor -- start 2024-01-16T15:11:48.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.645+0000 7f2f40e00640 1 -- start start 2024-01-16T15:11:48.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.646+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c143c40 0x7f2f3c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.646+0000 7f2f40e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2f3c0717d0 0x7f2f3c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.646+0000 7f2f40e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3c070e50 con 0x7f2f3c0cca40 2024-01-16T15:11:48.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.646+0000 7f2f40e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3c0723c0 con 0x7f2f3c0717d0 2024-01-16T15:11:48.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.646+0000 7f2f40e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0cce70 con 0x7f2f3c143c40 2024-01-16T15:11:48.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.647+0000 7f2f3b400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c143c40 0x7f2f3c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.647+0000 7f2f3b400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c143c40 0x7f2f3c071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:38732/0 (socket says 172.21.15.141:38732) 2024-01-16T15:11:48.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.647+0000 7f2f3b400640 1 -- 172.21.15.141:0/2087756399 learned_addr learned my addr 172.21.15.141:0/2087756399 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:48.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.647+0000 7f2f3aa00640 1 --2- 172.21.15.141:0/2087756399 >> v2:172.21.15.188:3300/0 conn(0x7f2f3c0717d0 0x7f2f3c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.648+0000 7f2f3b400640 1 -- 172.21.15.141:0/2087756399 >> v2:172.21.15.188:3300/0 conn(0x7f2f3c0717d0 msgr2=0x7f2f3c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:48.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.648+0000 7f2f3b400640 1 --2- 172.21.15.141:0/2087756399 >> v2:172.21.15.188:3300/0 conn(0x7f2f3c0717d0 0x7f2f3c0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.648+0000 7f2f3b400640 1 -- 172.21.15.141:0/2087756399 >> v1:172.21.15.188:6789/0 conn(0x7f2f3c0cca40 legacy=0x7f2f3c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.648+0000 7f2f3b400640 1 -- 172.21.15.141:0/2087756399 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c06e170 con 0x7f2f3c143c40 2024-01-16T15:11:48.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.649+0000 7f2f3b400640 1 --2- 172.21.15.141:0/2087756399 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c143c40 0x7f2f3c071290 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009870 tx=0x7f2f2c02b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a78f902ce8f518 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.650+0000 7f2f3a000640 1 -- 172.21.15.141:0/2087756399 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c031860 con 0x7f2f3c143c40 2024-01-16T15:11:48.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.650+0000 7f2f3a000640 1 -- 172.21.15.141:0/2087756399 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c0319c0 con 0x7f2f3c143c40 2024-01-16T15:11:48.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.650+0000 7f2f3a000640 1 -- 172.21.15.141:0/2087756399 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c031d50 con 0x7f2f3c143c40 2024-01-16T15:11:48.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 -- 172.21.15.141:0/2087756399 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c143c40 msgr2=0x7f2f3c071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2087756399 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c143c40 0x7f2f3c071290 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009870 tx=0x7f2f2c02b900 comp rx=0 tx=0).stop 2024-01-16T15:11:48.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 -- 172.21.15.141:0/2087756399 shutdown_connections 2024-01-16T15:11:48.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2087756399 >> v2:172.21.15.188:3300/0 conn(0x7f2f3c0717d0 0x7f2f3c0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2087756399 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c143c40 0x7f2f3c071290 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 -- 172.21.15.141:0/2087756399 >> 172.21.15.141:0/2087756399 conn(0x7f2f3c06a800 msgr2=0x7f2f3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:48.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 -- 172.21.15.141:0/2087756399 shutdown_connections 2024-01-16T15:11:48.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.651+0000 7f2f40e00640 1 -- 172.21.15.141:0/2087756399 wait complete. 2024-01-16T15:11:48.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.653+0000 7f2f40e00640 1 Processor -- start 2024-01-16T15:11:48.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.653+0000 7f2f40e00640 1 -- start start 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c0717d0 0x7f2f3c158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f3c0cca40 0x7f2f3c13b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c143c40 0x7f2f3c0ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f40e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c1f42a0 con 0x7f2f3c0cca40 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f40e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c159130 con 0x7f2f3c143c40 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f40e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c13c220 con 0x7f2f3c0717d0 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f3aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f3c0cca40 0x7f2f3c13b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f3aa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f3c0cca40 0x7f2f3c13b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38990/0 (socket says 172.21.15.141:38990) 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f3aa00640 1 -- 172.21.15.141:0/2618810791 learned_addr learned my addr 172.21.15.141:0/2618810791 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f3b400640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c0717d0 0x7f2f3c158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f3aa00640 1 -- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c0717d0 msgr2=0x7f2f3c158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f3aa00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c0717d0 0x7f2f3c158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.654+0000 7f2f3be00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c143c40 0x7f2f3c0ab3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f3aa00640 1 -- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c143c40 msgr2=0x7f2f3c0ab3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f3aa00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c143c40 0x7f2f3c0ab3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f3aa00640 1 -- 172.21.15.141:0/2618810791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c0094d0 con 0x7f2f3c0cca40 2024-01-16T15:11:48.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f3be00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c143c40 0x7f2f3c0ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:48.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f3aa00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f3c0cca40 0x7f2f3c13b470 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400e9b0 tx=0x7f2f2400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f33400640 1 -- 172.21.15.141:0/2618810791 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24009670 con 0x7f2f3c0cca40 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f33400640 1 -- 172.21.15.141:0/2618810791 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2400c9a0 con 0x7f2f3c0cca40 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f33400640 1 -- 172.21.15.141:0/2618810791 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400cd50 con 0x7f2f3c0cca40 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c13c500 con 0x7f2f3c0cca40 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.655+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c13cac0 con 0x7f2f3c0cca40 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.656+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c067450 con 0x7f2f3c0cca40 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.657+0000 7f2f33400640 1 -- 172.21.15.141:0/2618810791 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f2f24013470 con 0x7f2f3c0cca40 2024-01-16T15:11:48.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.658+0000 7f2f33400640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f1406c1f0 0x7f2f1406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.658+0000 7f2f33400640 1 -- 172.21.15.141:0/2618810791 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f240888d0 con 0x7f2f3c0cca40 2024-01-16T15:11:48.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.659+0000 7f2f3b400640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f1406c1f0 0x7f2f1406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.660+0000 7f2f3b400640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f1406c1f0 0x7f2f1406e6b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c036000 tx=0x7f2f2c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.662+0000 7f2f33400640 1 -- 172.21.15.141:0/2618810791 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f240551f0 con 0x7f2f3c0cca40 2024-01-16T15:11:48.813 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.836+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f3c0720e0 con 0x7f2f1406c1f0 2024-01-16T15:11:48.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.860+0000 7f2f33400640 1 -- 172.21.15.141:0/2618810791 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286885 (secure 0 0 0) 0x7f2f2c044400 con 0x7f2f1406c1f0 2024-01-16T15:11:48.838 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:48.842 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:11:48.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.866+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f1406c1f0 msgr2=0x7f2f1406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.866+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f1406c1f0 0x7f2f1406e6b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c036000 tx=0x7f2f2c03c040 comp rx=0 tx=0).stop 2024-01-16T15:11:48.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.866+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f3c0cca40 msgr2=0x7f2f3c13b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.866+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f3c0cca40 0x7f2f3c13b470 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400e9b0 tx=0x7f2f2400ee70 comp rx=0 tx=0).stop 2024-01-16T15:11:48.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 shutdown_connections 2024-01-16T15:11:48.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f1406c1f0 0x7f2f1406e6b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c143c40 0x7f2f3c0ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f3c0cca40 0x7f2f3c13b470 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 --2- 172.21.15.141:0/2618810791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f3c0717d0 0x7f2f3c158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 >> 172.21.15.141:0/2618810791 conn(0x7f2f3c06a800 msgr2=0x7f2f3c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:48.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 shutdown_connections 2024-01-16T15:11:48.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:48.867+0000 7f2f40e00640 1 -- 172.21.15.141:0/2618810791 wait complete. 2024-01-16T15:11:48.861 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2024-01-16T15:11:48.827290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":556952,"kb_used_data":10920,"kb_used_omap":12,"kb_used_meta":545907,"kb_avail":754417768,"statfs":{"total":773094113280,"available":772523794432,"internally_reserved":0,"allocated":11182080,"data_stored":5459742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":559009330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002155"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951872+0000","last_change":"2024-01-16T15:11:43.980470+0000","last_active":"2024-01-16T15:11:44.951872+0000","last_peered":"2024-01-16T15:11:44.951872+0000","last_clean":"2024-01-16T15:11:44.951872+0000","last_became_active":"2024-01-16T15:11:43.980118+0000","last_became_peered":"2024-01-16T15:11:43.980118+0000","last_unstale":"2024-01-16T15:11:44.951872+0000","last_undegraded":"2024-01-16T15:11:44.951872+0000","last_fullsized":"2024-01-16T15:11:44.951872+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:18.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985054+0000","last_change":"2024-01-16T15:11:43.982509+0000","last_active":"2024-01-16T15:11:43.985054+0000","last_peered":"2024-01-16T15:11:43.985054+0000","last_clean":"2024-01-16T15:11:43.985054+0000","last_became_active":"2024-01-16T15:11:43.981847+0000","last_became_peered":"2024-01-16T15:11:43.981847+0000","last_unstale":"2024-01-16T15:11:43.985054+0000","last_undegraded":"2024-01-16T15:11:43.985054+0000","last_fullsized":"2024-01-16T15:11:43.985054+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:20.786959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951761+0000","last_change":"2024-01-16T15:11:43.969123+0000","last_active":"2024-01-16T15:11:44.951761+0000","last_peered":"2024-01-16T15:11:44.951761+0000","last_clean":"2024-01-16T15:11:44.951761+0000","last_became_active":"2024-01-16T15:11:43.968784+0000","last_became_peered":"2024-01-16T15:11:43.968784+0000","last_unstale":"2024-01-16T15:11:44.951761+0000","last_undegraded":"2024-01-16T15:11:44.951761+0000","last_fullsized":"2024-01-16T15:11:44.951761+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:19.083239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981974+0000","last_change":"2024-01-16T15:11:43.975345+0000","last_active":"2024-01-16T15:11:43.981974+0000","last_peered":"2024-01-16T15:11:43.981974+0000","last_clean":"2024-01-16T15:11:43.981974+0000","last_became_active":"2024-01-16T15:11:43.974874+0000","last_became_peered":"2024-01-16T15:11:43.974874+0000","last_unstale":"2024-01-16T15:11:43.981974+0000","last_undegraded":"2024-01-16T15:11:43.981974+0000","last_fullsized":"2024-01-16T15:11:43.981974+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:56.775943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125616+0000","last_change":"2024-01-16T15:11:43.979769+0000","last_active":"2024-01-16T15:11:45.125616+0000","last_peered":"2024-01-16T15:11:45.125616+0000","last_clean":"2024-01-16T15:11:45.125616+0000","last_became_active":"2024-01-16T15:11:43.979465+0000","last_became_peered":"2024-01-16T15:11:43.979465+0000","last_unstale":"2024-01-16T15:11:45.125616+0000","last_undegraded":"2024-01-16T15:11:45.125616+0000","last_fullsized":"2024-01-16T15:11:45.125616+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:17.534495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.950131+0000","last_change":"2024-01-16T15:11:43.982175+0000","last_active":"2024-01-16T15:11:44.950131+0000","last_peered":"2024-01-16T15:11:44.950131+0000","last_clean":"2024-01-16T15:11:44.950131+0000","last_became_active":"2024-01-16T15:11:43.981884+0000","last_became_peered":"2024-01-16T15:11:43.981884+0000","last_unstale":"2024-01-16T15:11:44.950131+0000","last_undegraded":"2024-01-16T15:11:44.950131+0000","last_fullsized":"2024-01-16T15:11:44.950131+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:20.853365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.948819+0000","last_change":"2024-01-16T15:11:43.980074+0000","last_active":"2024-01-16T15:11:44.948819+0000","last_peered":"2024-01-16T15:11:44.948819+0000","last_clean":"2024-01-16T15:11:44.948819+0000","last_became_active":"2024-01-16T15:11:43.979768+0000","last_became_peered":"2024-01-16T15:11:43.979768+0000","last_unstale":"2024-01-16T15:11:44.948819+0000","last_undegraded":"2024-01-16T15:11:44.948819+0000","last_fullsized":"2024-01-16T15:11:44.948819+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:21.612480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.989023+0000","last_change":"2024-01-16T15:11:43.987503+0000","last_active":"2024-01-16T15:11:43.989023+0000","last_peered":"2024-01-16T15:11:43.989023+0000","last_clean":"2024-01-16T15:11:43.989023+0000","last_became_active":"2024-01-16T15:11:43.986145+0000","last_became_peered":"2024-01-16T15:11:43.986145+0000","last_unstale":"2024-01-16T15:11:43.989023+0000","last_undegraded":"2024-01-16T15:11:43.989023+0000","last_fullsized":"2024-01-16T15:11:43.989023+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:17.534495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.986016+0000","last_change":"2024-01-16T15:11:43.980734+0000","last_active":"2024-01-16T15:11:43.986016+0000","last_peered":"2024-01-16T15:11:43.986016+0000","last_clean":"2024-01-16T15:11:43.986016+0000","last_became_active":"2024-01-16T15:11:43.980467+0000","last_became_peered":"2024-01-16T15:11:43.980467+0000","last_unstale":"2024-01-16T15:11:43.986016+0000","last_undegraded":"2024-01-16T15:11:43.986016+0000","last_fullsized":"2024-01-16T15:11:43.986016+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:08:52.281754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125573+0000","last_change":"2024-01-16T15:11:43.971024+0000","last_active":"2024-01-16T15:11:45.125573+0000","last_peered":"2024-01-16T15:11:45.125573+0000","last_clean":"2024-01-16T15:11:45.125573+0000","last_became_active":"2024-01-16T15:11:43.969962+0000","last_became_peered":"2024-01-16T15:11:43.969962+0000","last_unstale":"2024-01-16T15:11:45.125573+0000","last_undegraded":"2024-01-16T15:11:45.125573+0000","last_fullsized":"2024-01-16T15:11:45.125573+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:32.538245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.948815+0000","last_change":"2024-01-16T15:11:43.981999+0000","last_active":"2024-01-16T15:11:44.948815+0000","last_peered":"2024-01-16T15:11:44.948815+0000","last_clean":"2024-01-16T15:11:44.948815+0000","last_became_active":"2024-01-16T15:11:43.978354+0000","last_became_peered":"2024-01-16T15:11:43.978354+0000","last_unstale":"2024-01-16T15:11:44.948815+0000","last_undegraded":"2024-01-16T15:11:44.948815+0000","last_fullsized":"2024-01-16T15:11:44.948815+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:14.500746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.966582+0000","last_change":"2024-01-16T15:11:43.962264+0000","last_active":"2024-01-16T15:11:43.966582+0000","last_peered":"2024-01-16T15:11:43.966582+0000","last_clean":"2024-01-16T15:11:43.966582+0000","last_became_active":"2024-01-16T15:11:43.961051+0000","last_became_peered":"2024-01-16T15:11:43.961051+0000","last_unstale":"2024-01-16T15:11:43.966582+0000","last_undegraded":"2024-01-16T15:11:43.966582+0000","last_fullsized":"2024-01-16T15:11:43.966582+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:27.542322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981952+0000","last_change":"2024-01-16T15:11:43.975880+0000","last_active":"2024-01-16T15:11:43.981952+0000","last_peered":"2024-01-16T15:11:43.981952+0000","last_clean":"2024-01-16T15:11:43.981952+0000","last_became_active":"2024-01-16T15:11:43.975323+0000","last_became_peered":"2024-01-16T15:11:43.975323+0000","last_unstale":"2024-01-16T15:11:43.981952+0000","last_undegraded":"2024-01-16T15:11:43.981952+0000","last_fullsized":"2024-01-16T15:11:43.981952+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:12.797328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983508+0000","last_change":"2024-01-16T15:11:43.978831+0000","last_active":"2024-01-16T15:11:43.983508+0000","last_peered":"2024-01-16T15:11:43.983508+0000","last_clean":"2024-01-16T15:11:43.983508+0000","last_became_active":"2024-01-16T15:11:43.978521+0000","last_became_peered":"2024-01-16T15:11:43.978521+0000","last_unstale":"2024-01-16T15:11:43.983508+0000","last_undegraded":"2024-01-16T15:11:43.983508+0000","last_fullsized":"2024-01-16T15:11:43.983508+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:30.805142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981984+0000","last_change":"2024-01-16T15:11:43.975962+0000","last_active":"2024-01-16T15:11:43.981984+0000","last_peered":"2024-01-16T15:11:43.981984+0000","last_clean":"2024-01-16T15:11:43.981984+0000","last_became_active":"2024-01-16T15:11:43.975628+0000","last_became_peered":"2024-01-16T15:11:43.975628+0000","last_unstale":"2024-01-16T15:11:43.981984+0000","last_undegraded":"2024-01-16T15:11:43.981984+0000","last_fullsized":"2024-01-16T15:11:43.981984+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:04.795502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.947929+0000","last_change":"2024-01-16T15:11:43.981094+0000","last_active":"2024-01-16T15:11:44.947929+0000","last_peered":"2024-01-16T15:11:44.947929+0000","last_clean":"2024-01-16T15:11:44.947929+0000","last_became_active":"2024-01-16T15:11:43.980737+0000","last_became_peered":"2024-01-16T15:11:43.980737+0000","last_unstale":"2024-01-16T15:11:44.947929+0000","last_undegraded":"2024-01-16T15:11:44.947929+0000","last_fullsized":"2024-01-16T15:11:44.947929+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:00.946690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.988681+0000","last_change":"2024-01-16T15:11:43.986808+0000","last_active":"2024-01-16T15:11:43.988681+0000","last_peered":"2024-01-16T15:11:43.988681+0000","last_clean":"2024-01-16T15:11:43.988681+0000","last_became_active":"2024-01-16T15:11:43.986464+0000","last_became_peered":"2024-01-16T15:11:43.986464+0000","last_unstale":"2024-01-16T15:11:43.988681+0000","last_undegraded":"2024-01-16T15:11:43.988681+0000","last_fullsized":"2024-01-16T15:11:43.988681+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:36.098695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983183+0000","last_change":"2024-01-16T15:11:43.977991+0000","last_active":"2024-01-16T15:11:43.983183+0000","last_peered":"2024-01-16T15:11:43.983183+0000","last_clean":"2024-01-16T15:11:43.983183+0000","last_became_active":"2024-01-16T15:11:43.977319+0000","last_became_peered":"2024-01-16T15:11:43.977319+0000","last_unstale":"2024-01-16T15:11:43.983183+0000","last_undegraded":"2024-01-16T15:11:43.983183+0000","last_fullsized":"2024-01-16T15:11:43.983183+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:24:00.099420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.965709+0000","last_change":"2024-01-16T15:11:43.961989+0000","last_active":"2024-01-16T15:11:43.965709+0000","last_peered":"2024-01-16T15:11:43.965709+0000","last_clean":"2024-01-16T15:11:43.965709+0000","last_became_active":"2024-01-16T15:11:43.961535+0000","last_became_peered":"2024-01-16T15:11:43.961535+0000","last_unstale":"2024-01-16T15:11:43.965709+0000","last_undegraded":"2024-01-16T15:11:43.965709+0000","last_fullsized":"2024-01-16T15:11:43.965709+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:43.220508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.977141+0000","last_change":"2024-01-16T15:11:43.975833+0000","last_active":"2024-01-16T15:11:43.977141+0000","last_peered":"2024-01-16T15:11:43.977141+0000","last_clean":"2024-01-16T15:11:43.977141+0000","last_became_active":"2024-01-16T15:11:43.975453+0000","last_became_peered":"2024-01-16T15:11:43.975453+0000","last_unstale":"2024-01-16T15:11:43.977141+0000","last_undegraded":"2024-01-16T15:11:43.977141+0000","last_fullsized":"2024-01-16T15:11:43.977141+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:54.244819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981689+0000","last_change":"2024-01-16T15:11:43.977924+0000","last_active":"2024-01-16T15:11:43.981689+0000","last_peered":"2024-01-16T15:11:43.981689+0000","last_clean":"2024-01-16T15:11:43.981689+0000","last_became_active":"2024-01-16T15:11:43.977277+0000","last_became_peered":"2024-01-16T15:11:43.977277+0000","last_unstale":"2024-01-16T15:11:43.981689+0000","last_undegraded":"2024-01-16T15:11:43.981689+0000","last_fullsized":"2024-01-16T15:11:43.981689+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:53:58.229296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.984884+0000","last_change":"2024-01-16T15:11:43.982155+0000","last_active":"2024-01-16T15:11:43.984884+0000","last_peered":"2024-01-16T15:11:43.984884+0000","last_clean":"2024-01-16T15:11:43.984884+0000","last_became_active":"2024-01-16T15:11:43.981413+0000","last_became_peered":"2024-01-16T15:11:43.981413+0000","last_unstale":"2024-01-16T15:11:43.984884+0000","last_undegraded":"2024-01-16T15:11:43.984884+0000","last_fullsized":"2024-01-16T15:11:43.984884+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:27.023432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981386+0000","last_change":"2024-01-16T15:11:43.977635+0000","last_active":"2024-01-16T15:11:43.981386+0000","last_peered":"2024-01-16T15:11:43.981386+0000","last_clean":"2024-01-16T15:11:43.981386+0000","last_became_active":"2024-01-16T15:11:43.976856+0000","last_became_peered":"2024-01-16T15:11:43.976856+0000","last_unstale":"2024-01-16T15:11:43.981386+0000","last_undegraded":"2024-01-16T15:11:43.981386+0000","last_fullsized":"2024-01-16T15:11:43.981386+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:32.538245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.948511+0000","last_change":"2024-01-16T15:11:43.985323+0000","last_active":"2024-01-16T15:11:44.948511+0000","last_peered":"2024-01-16T15:11:44.948511+0000","last_clean":"2024-01-16T15:11:44.948511+0000","last_became_active":"2024-01-16T15:11:43.985000+0000","last_became_peered":"2024-01-16T15:11:43.985000+0000","last_unstale":"2024-01-16T15:11:44.948511+0000","last_undegraded":"2024-01-16T15:11:44.948511+0000","last_fullsized":"2024-01-16T15:11:44.948511+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:01.008552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125911+0000","last_change":"2024-01-16T15:11:43.980713+0000","last_active":"2024-01-16T15:11:45.125911+0000","last_peered":"2024-01-16T15:11:45.125911+0000","last_clean":"2024-01-16T15:11:45.125911+0000","last_became_active":"2024-01-16T15:11:43.980427+0000","last_became_peered":"2024-01-16T15:11:43.980427+0000","last_unstale":"2024-01-16T15:11:45.125911+0000","last_undegraded":"2024-01-16T15:11:45.125911+0000","last_fullsized":"2024-01-16T15:11:45.125911+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:09.570949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985089+0000","last_change":"2024-01-16T15:11:43.982697+0000","last_active":"2024-01-16T15:11:43.985089+0000","last_peered":"2024-01-16T15:11:43.985089+0000","last_clean":"2024-01-16T15:11:43.985089+0000","last_became_active":"2024-01-16T15:11:43.982272+0000","last_became_peered":"2024-01-16T15:11:43.982272+0000","last_unstale":"2024-01-16T15:11:43.985089+0000","last_undegraded":"2024-01-16T15:11:43.985089+0000","last_fullsized":"2024-01-16T15:11:43.985089+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:34.263979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949272+0000","last_change":"2024-01-16T15:11:43.988167+0000","last_active":"2024-01-16T15:11:44.949272+0000","last_peered":"2024-01-16T15:11:44.949272+0000","last_clean":"2024-01-16T15:11:44.949272+0000","last_became_active":"2024-01-16T15:11:43.987882+0000","last_became_peered":"2024-01-16T15:11:43.987882+0000","last_unstale":"2024-01-16T15:11:44.949272+0000","last_undegraded":"2024-01-16T15:11:44.949272+0000","last_fullsized":"2024-01-16T15:11:44.949272+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:22.759489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126016+0000","last_change":"2024-01-16T15:11:43.984030+0000","last_active":"2024-01-16T15:11:45.126016+0000","last_peered":"2024-01-16T15:11:45.126016+0000","last_clean":"2024-01-16T15:11:45.126016+0000","last_became_active":"2024-01-16T15:11:43.982544+0000","last_became_peered":"2024-01-16T15:11:43.982544+0000","last_unstale":"2024-01-16T15:11:45.126016+0000","last_undegraded":"2024-01-16T15:11:45.126016+0000","last_fullsized":"2024-01-16T15:11:45.126016+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:27.023432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983484+0000","last_change":"2024-01-16T15:11:43.980067+0000","last_active":"2024-01-16T15:11:43.983484+0000","last_peered":"2024-01-16T15:11:43.983484+0000","last_clean":"2024-01-16T15:11:43.983484+0000","last_became_active":"2024-01-16T15:11:43.979800+0000","last_became_peered":"2024-01-16T15:11:43.979800+0000","last_unstale":"2024-01-16T15:11:43.983484+0000","last_undegraded":"2024-01-16T15:11:43.983484+0000","last_fullsized":"2024-01-16T15:11:43.983484+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:26.954494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.972584+0000","last_change":"2024-01-16T15:11:43.964860+0000","last_active":"2024-01-16T15:11:43.972584+0000","last_peered":"2024-01-16T15:11:43.972584+0000","last_clean":"2024-01-16T15:11:43.972584+0000","last_became_active":"2024-01-16T15:11:43.964057+0000","last_became_peered":"2024-01-16T15:11:43.964057+0000","last_unstale":"2024-01-16T15:11:43.972584+0000","last_undegraded":"2024-01-16T15:11:43.972584+0000","last_fullsized":"2024-01-16T15:11:43.972584+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:23.969311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985559+0000","last_change":"2024-01-16T15:11:43.984275+0000","last_active":"2024-01-16T15:11:43.985559+0000","last_peered":"2024-01-16T15:11:43.985559+0000","last_clean":"2024-01-16T15:11:43.985559+0000","last_became_active":"2024-01-16T15:11:43.984017+0000","last_became_peered":"2024-01-16T15:11:43.984017+0000","last_unstale":"2024-01-16T15:11:43.985559+0000","last_undegraded":"2024-01-16T15:11:43.985559+0000","last_fullsized":"2024-01-16T15:11:43.985559+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:31.754450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.950944+0000","last_change":"2024-01-16T15:11:43.970636+0000","last_active":"2024-01-16T15:11:44.950944+0000","last_peered":"2024-01-16T15:11:44.950944+0000","last_clean":"2024-01-16T15:11:44.950944+0000","last_became_active":"2024-01-16T15:11:43.970299+0000","last_became_peered":"2024-01-16T15:11:43.970299+0000","last_unstale":"2024-01-16T15:11:44.950944+0000","last_undegraded":"2024-01-16T15:11:44.950944+0000","last_fullsized":"2024-01-16T15:11:44.950944+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:54.244819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.984771+0000","last_change":"2024-01-16T15:11:43.981592+0000","last_active":"2024-01-16T15:11:43.984771+0000","last_peered":"2024-01-16T15:11:43.984771+0000","last_clean":"2024-01-16T15:11:43.984771+0000","last_became_active":"2024-01-16T15:11:43.980749+0000","last_became_peered":"2024-01-16T15:11:43.980749+0000","last_unstale":"2024-01-16T15:11:43.984771+0000","last_undegraded":"2024-01-16T15:11:43.984771+0000","last_fullsized":"2024-01-16T15:11:43.984771+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:14.406536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"101'4","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.997333+0000","last_change":"2024-01-16T15:11:42.950909+0000","last_active":"2024-01-16T15:11:43.997333+0000","last_peered":"2024-01-16T15:11:43.997333+0000","last_clean":"2024-01-16T15:11:43.997333+0000","last_became_active":"2024-01-16T15:11:42.950605+0000","last_became_peered":"2024-01-16T15:11:42.950605+0000","last_unstale":"2024-01-16T15:11:43.997333+0000","last_undegraded":"2024-01-16T15:11:43.997333+0000","last_fullsized":"2024-01-16T15:11:43.997333+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:37:37.087146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.987220+0000","last_change":"2024-01-16T15:11:43.984315+0000","last_active":"2024-01-16T15:11:43.987220+0000","last_peered":"2024-01-16T15:11:43.987220+0000","last_clean":"2024-01-16T15:11:43.987220+0000","last_became_active":"2024-01-16T15:11:43.983962+0000","last_became_peered":"2024-01-16T15:11:43.983962+0000","last_unstale":"2024-01-16T15:11:43.987220+0000","last_undegraded":"2024-01-16T15:11:43.987220+0000","last_fullsized":"2024-01-16T15:11:43.987220+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:06:53.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949010+0000","last_change":"2024-01-16T15:11:42.948218+0000","last_active":"2024-01-16T15:11:44.949010+0000","last_peered":"2024-01-16T15:11:44.949010+0000","last_clean":"2024-01-16T15:11:44.949010+0000","last_became_active":"2024-01-16T15:11:42.947906+0000","last_became_peered":"2024-01-16T15:11:42.947906+0000","last_unstale":"2024-01-16T15:11:44.949010+0000","last_undegraded":"2024-01-16T15:11:44.949010+0000","last_fullsized":"2024-01-16T15:11:44.949010+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:14.172800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951554+0000","last_change":"2024-01-16T15:11:43.958604+0000","last_active":"2024-01-16T15:11:44.951554+0000","last_peered":"2024-01-16T15:11:44.951554+0000","last_clean":"2024-01-16T15:11:44.951554+0000","last_became_active":"2024-01-16T15:11:43.957374+0000","last_became_peered":"2024-01-16T15:11:43.957374+0000","last_unstale":"2024-01-16T15:11:44.951554+0000","last_undegraded":"2024-01-16T15:11:44.951554+0000","last_fullsized":"2024-01-16T15:11:44.951554+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:54.894378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.994019+0000","last_change":"2024-01-16T15:11:42.945607+0000","last_active":"2024-01-16T15:11:43.994019+0000","last_peered":"2024-01-16T15:11:43.994019+0000","last_clean":"2024-01-16T15:11:43.994019+0000","last_became_active":"2024-01-16T15:11:42.944951+0000","last_became_peered":"2024-01-16T15:11:42.944951+0000","last_unstale":"2024-01-16T15:11:43.994019+0000","last_undegraded":"2024-01-16T15:11:43.994019+0000","last_fullsized":"2024-01-16T15:11:43.994019+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:47.379076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.975983+0000","last_change":"2024-01-16T15:11:43.969634+0000","last_active":"2024-01-16T15:11:43.975983+0000","last_peered":"2024-01-16T15:11:43.975983+0000","last_clean":"2024-01-16T15:11:43.975983+0000","last_became_active":"2024-01-16T15:11:43.969231+0000","last_became_peered":"2024-01-16T15:11:43.969231+0000","last_unstale":"2024-01-16T15:11:43.975983+0000","last_undegraded":"2024-01-16T15:11:43.975983+0000","last_fullsized":"2024-01-16T15:11:43.975983+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:06.349978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.999015+0000","last_change":"2024-01-16T15:11:42.941334+0000","last_active":"2024-01-16T15:11:43.999015+0000","last_peered":"2024-01-16T15:11:43.999015+0000","last_clean":"2024-01-16T15:11:43.999015+0000","last_became_active":"2024-01-16T15:11:42.939561+0000","last_became_peered":"2024-01-16T15:11:42.939561+0000","last_unstale":"2024-01-16T15:11:43.999015+0000","last_undegraded":"2024-01-16T15:11:43.999015+0000","last_fullsized":"2024-01-16T15:11:43.999015+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:34.842850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.990794+0000","last_change":"2024-01-16T15:11:43.986599+0000","last_active":"2024-01-16T15:11:43.990794+0000","last_peered":"2024-01-16T15:11:43.990794+0000","last_clean":"2024-01-16T15:11:43.990794+0000","last_became_active":"2024-01-16T15:11:43.985316+0000","last_became_peered":"2024-01-16T15:11:43.985316+0000","last_unstale":"2024-01-16T15:11:43.990794+0000","last_undegraded":"2024-01-16T15:11:43.990794+0000","last_fullsized":"2024-01-16T15:11:43.990794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:23:01.100385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951475+0000","last_change":"2024-01-16T15:11:42.947866+0000","last_active":"2024-01-16T15:11:44.951475+0000","last_peered":"2024-01-16T15:11:44.951475+0000","last_clean":"2024-01-16T15:11:44.951475+0000","last_became_active":"2024-01-16T15:11:42.946072+0000","last_became_peered":"2024-01-16T15:11:42.946072+0000","last_unstale":"2024-01-16T15:11:44.951475+0000","last_undegraded":"2024-01-16T15:11:44.951475+0000","last_fullsized":"2024-01-16T15:11:44.951475+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:27.716130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949519+0000","last_change":"2024-01-16T15:11:43.991198+0000","last_active":"2024-01-16T15:11:44.949519+0000","last_peered":"2024-01-16T15:11:44.949519+0000","last_clean":"2024-01-16T15:11:44.949519+0000","last_became_active":"2024-01-16T15:11:43.987988+0000","last_became_peered":"2024-01-16T15:11:43.987988+0000","last_unstale":"2024-01-16T15:11:44.949519+0000","last_undegraded":"2024-01-16T15:11:44.949519+0000","last_fullsized":"2024-01-16T15:11:44.949519+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:02.663504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.993681+0000","last_change":"2024-01-16T15:11:42.951606+0000","last_active":"2024-01-16T15:11:43.993681+0000","last_peered":"2024-01-16T15:11:43.993681+0000","last_clean":"2024-01-16T15:11:43.993681+0000","last_became_active":"2024-01-16T15:11:42.950303+0000","last_became_peered":"2024-01-16T15:11:42.950303+0000","last_unstale":"2024-01-16T15:11:43.993681+0000","last_undegraded":"2024-01-16T15:11:43.993681+0000","last_fullsized":"2024-01-16T15:11:43.993681+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:08.489605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.982203+0000","last_change":"2024-01-16T15:11:43.976432+0000","last_active":"2024-01-16T15:11:43.982203+0000","last_peered":"2024-01-16T15:11:43.982203+0000","last_clean":"2024-01-16T15:11:43.982203+0000","last_became_active":"2024-01-16T15:11:43.976102+0000","last_became_peered":"2024-01-16T15:11:43.976102+0000","last_unstale":"2024-01-16T15:11:43.982203+0000","last_undegraded":"2024-01-16T15:11:43.982203+0000","last_fullsized":"2024-01-16T15:11:43.982203+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:06.232276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125631+0000","last_change":"2024-01-16T15:11:42.952992+0000","last_active":"2024-01-16T15:11:45.125631+0000","last_peered":"2024-01-16T15:11:45.125631+0000","last_clean":"2024-01-16T15:11:45.125631+0000","last_became_active":"2024-01-16T15:11:42.952619+0000","last_became_peered":"2024-01-16T15:11:42.952619+0000","last_unstale":"2024-01-16T15:11:45.125631+0000","last_undegraded":"2024-01-16T15:11:45.125631+0000","last_fullsized":"2024-01-16T15:11:45.125631+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:48.460524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.982016+0000","last_change":"2024-01-16T15:11:43.977711+0000","last_active":"2024-01-16T15:11:43.982016+0000","last_peered":"2024-01-16T15:11:43.982016+0000","last_clean":"2024-01-16T15:11:43.982016+0000","last_became_active":"2024-01-16T15:11:43.976821+0000","last_became_peered":"2024-01-16T15:11:43.976821+0000","last_unstale":"2024-01-16T15:11:43.982016+0000","last_undegraded":"2024-01-16T15:11:43.982016+0000","last_fullsized":"2024-01-16T15:11:43.982016+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:54.894378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.950883+0000","last_change":"2024-01-16T15:11:42.942480+0000","last_active":"2024-01-16T15:11:44.950883+0000","last_peered":"2024-01-16T15:11:44.950883+0000","last_clean":"2024-01-16T15:11:44.950883+0000","last_became_active":"2024-01-16T15:11:42.942226+0000","last_became_peered":"2024-01-16T15:11:42.942226+0000","last_unstale":"2024-01-16T15:11:44.950883+0000","last_undegraded":"2024-01-16T15:11:44.950883+0000","last_fullsized":"2024-01-16T15:11:44.950883+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:55.449507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985837+0000","last_change":"2024-01-16T15:11:43.984671+0000","last_active":"2024-01-16T15:11:43.985837+0000","last_peered":"2024-01-16T15:11:43.985837+0000","last_clean":"2024-01-16T15:11:43.985837+0000","last_became_active":"2024-01-16T15:11:43.984362+0000","last_became_peered":"2024-01-16T15:11:43.984362+0000","last_unstale":"2024-01-16T15:11:43.985837+0000","last_undegraded":"2024-01-16T15:11:43.985837+0000","last_fullsized":"2024-01-16T15:11:43.985837+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:31.854959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951991+0000","last_change":"2024-01-16T15:11:42.940557+0000","last_active":"2024-01-16T15:11:44.951991+0000","last_peered":"2024-01-16T15:11:44.951991+0000","last_clean":"2024-01-16T15:11:44.951991+0000","last_became_active":"2024-01-16T15:11:42.939209+0000","last_became_peered":"2024-01-16T15:11:42.939209+0000","last_unstale":"2024-01-16T15:11:44.951991+0000","last_undegraded":"2024-01-16T15:11:44.951991+0000","last_fullsized":"2024-01-16T15:11:44.951991+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:03.956372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.987460+0000","last_change":"2024-01-16T15:11:43.983616+0000","last_active":"2024-01-16T15:11:43.987460+0000","last_peered":"2024-01-16T15:11:43.987460+0000","last_clean":"2024-01-16T15:11:43.987460+0000","last_became_active":"2024-01-16T15:11:43.983269+0000","last_became_peered":"2024-01-16T15:11:43.983269+0000","last_unstale":"2024-01-16T15:11:43.987460+0000","last_undegraded":"2024-01-16T15:11:43.987460+0000","last_fullsized":"2024-01-16T15:11:43.987460+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:18.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.964996+0000","last_change":"2024-01-16T15:11:42.950112+0000","last_active":"2024-01-16T15:11:43.964996+0000","last_peered":"2024-01-16T15:11:43.964996+0000","last_clean":"2024-01-16T15:11:43.964996+0000","last_became_active":"2024-01-16T15:11:42.949829+0000","last_became_peered":"2024-01-16T15:11:42.949829+0000","last_unstale":"2024-01-16T15:11:43.964996+0000","last_undegraded":"2024-01-16T15:11:43.964996+0000","last_fullsized":"2024-01-16T15:11:43.964996+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:30.474487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126055+0000","last_change":"2024-01-16T15:11:43.978831+0000","last_active":"2024-01-16T15:11:45.126055+0000","last_peered":"2024-01-16T15:11:45.126055+0000","last_clean":"2024-01-16T15:11:45.126055+0000","last_became_active":"2024-01-16T15:11:43.978406+0000","last_became_peered":"2024-01-16T15:11:43.978406+0000","last_unstale":"2024-01-16T15:11:45.126055+0000","last_undegraded":"2024-01-16T15:11:45.126055+0000","last_fullsized":"2024-01-16T15:11:45.126055+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:09.504678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"101'3","reported_seq":15,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949902+0000","last_change":"2024-01-16T15:11:42.948216+0000","last_active":"2024-01-16T15:11:44.949902+0000","last_peered":"2024-01-16T15:11:44.949902+0000","last_clean":"2024-01-16T15:11:44.949902+0000","last_became_active":"2024-01-16T15:11:42.947911+0000","last_became_peered":"2024-01-16T15:11:42.947911+0000","last_unstale":"2024-01-16T15:11:44.949902+0000","last_undegraded":"2024-01-16T15:11:44.949902+0000","last_fullsized":"2024-01-16T15:11:44.949902+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:10.212029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949694+0000","last_change":"2024-01-16T15:11:43.986773+0000","last_active":"2024-01-16T15:11:44.949694+0000","last_peered":"2024-01-16T15:11:44.949694+0000","last_clean":"2024-01-16T15:11:44.949694+0000","last_became_active":"2024-01-16T15:11:43.986399+0000","last_became_peered":"2024-01-16T15:11:43.986399+0000","last_unstale":"2024-01-16T15:11:44.949694+0000","last_undegraded":"2024-01-16T15:11:44.949694+0000","last_fullsized":"2024-01-16T15:11:44.949694+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:44.820410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"101'6","reported_seq":17,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.000737+0000","last_change":"2024-01-16T15:11:42.935808+0000","last_active":"2024-01-16T15:11:44.000737+0000","last_peered":"2024-01-16T15:11:44.000737+0000","last_clean":"2024-01-16T15:11:44.000737+0000","last_became_active":"2024-01-16T15:11:42.935305+0000","last_became_peered":"2024-01-16T15:11:42.935305+0000","last_unstale":"2024-01-16T15:11:44.000737+0000","last_undegraded":"2024-01-16T15:11:44.000737+0000","last_fullsized":"2024-01-16T15:11:44.000737+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:23.659985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985313+0000","last_change":"2024-01-16T15:11:43.982783+0000","last_active":"2024-01-16T15:11:43.985313+0000","last_peered":"2024-01-16T15:11:43.985313+0000","last_clean":"2024-01-16T15:11:43.985313+0000","last_became_active":"2024-01-16T15:11:43.982487+0000","last_became_peered":"2024-01-16T15:11:43.982487+0000","last_unstale":"2024-01-16T15:11:43.985313+0000","last_undegraded":"2024-01-16T15:11:43.985313+0000","last_fullsized":"2024-01-16T15:11:43.985313+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:22.996185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.998760+0000","last_change":"2024-01-16T15:11:42.943355+0000","last_active":"2024-01-16T15:11:43.998760+0000","last_peered":"2024-01-16T15:11:43.998760+0000","last_clean":"2024-01-16T15:11:43.998760+0000","last_became_active":"2024-01-16T15:11:42.941670+0000","last_became_peered":"2024-01-16T15:11:42.941670+0000","last_unstale":"2024-01-16T15:11:43.998760+0000","last_undegraded":"2024-01-16T15:11:43.998760+0000","last_fullsized":"2024-01-16T15:11:43.998760+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:15.192954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951499+0000","last_change":"2024-01-16T15:11:43.966711+0000","last_active":"2024-01-16T15:11:44.951499+0000","last_peered":"2024-01-16T15:11:44.951499+0000","last_clean":"2024-01-16T15:11:44.951499+0000","last_became_active":"2024-01-16T15:11:43.966408+0000","last_became_peered":"2024-01-16T15:11:43.966408+0000","last_unstale":"2024-01-16T15:11:44.951499+0000","last_undegraded":"2024-01-16T15:11:44.951499+0000","last_fullsized":"2024-01-16T15:11:44.951499+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:03.764468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.951480+0000","last_change":"2024-01-16T15:11:42.948490+0000","last_active":"2024-01-16T15:11:43.951480+0000","last_peered":"2024-01-16T15:11:43.951480+0000","last_clean":"2024-01-16T15:11:43.951480+0000","last_became_active":"2024-01-16T15:11:42.948248+0000","last_became_peered":"2024-01-16T15:11:42.948248+0000","last_unstale":"2024-01-16T15:11:43.951480+0000","last_undegraded":"2024-01-16T15:11:43.951480+0000","last_fullsized":"2024-01-16T15:11:43.951480+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:08.179719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.976376+0000","last_change":"2024-01-16T15:11:43.968616+0000","last_active":"2024-01-16T15:11:43.976376+0000","last_peered":"2024-01-16T15:11:43.976376+0000","last_clean":"2024-01-16T15:11:43.976376+0000","last_became_active":"2024-01-16T15:11:43.968234+0000","last_became_peered":"2024-01-16T15:11:43.968234+0000","last_unstale":"2024-01-16T15:11:43.976376+0000","last_undegraded":"2024-01-16T15:11:43.976376+0000","last_fullsized":"2024-01-16T15:11:43.976376+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:23:01.100385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.948607+0000","last_change":"2024-01-16T15:11:42.947081+0000","last_active":"2024-01-16T15:11:43.948607+0000","last_peered":"2024-01-16T15:11:43.948607+0000","last_clean":"2024-01-16T15:11:43.948607+0000","last_became_active":"2024-01-16T15:11:42.946702+0000","last_became_peered":"2024-01-16T15:11:42.946702+0000","last_unstale":"2024-01-16T15:11:43.948607+0000","last_undegraded":"2024-01-16T15:11:43.948607+0000","last_fullsized":"2024-01-16T15:11:43.948607+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:18.068166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.964786+0000","last_change":"2024-01-16T15:11:43.961346+0000","last_active":"2024-01-16T15:11:43.964786+0000","last_peered":"2024-01-16T15:11:43.964786+0000","last_clean":"2024-01-16T15:11:43.964786+0000","last_became_active":"2024-01-16T15:11:43.960179+0000","last_became_peered":"2024-01-16T15:11:43.960179+0000","last_unstale":"2024-01-16T15:11:43.964786+0000","last_undegraded":"2024-01-16T15:11:43.964786+0000","last_fullsized":"2024-01-16T15:11:43.964786+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:33:42.894239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"101'8","reported_seq":20,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951028+0000","last_change":"2024-01-16T15:11:42.946994+0000","last_active":"2024-01-16T15:11:44.951028+0000","last_peered":"2024-01-16T15:11:44.951028+0000","last_clean":"2024-01-16T15:11:44.951028+0000","last_became_active":"2024-01-16T15:11:42.945711+0000","last_became_peered":"2024-01-16T15:11:42.945711+0000","last_unstale":"2024-01-16T15:11:44.951028+0000","last_undegraded":"2024-01-16T15:11:44.951028+0000","last_fullsized":"2024-01-16T15:11:44.951028+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:11.035449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125998+0000","last_change":"2024-01-16T15:11:43.968879+0000","last_active":"2024-01-16T15:11:45.125998+0000","last_peered":"2024-01-16T15:11:45.125998+0000","last_clean":"2024-01-16T15:11:45.125998+0000","last_became_active":"2024-01-16T15:11:43.968459+0000","last_became_peered":"2024-01-16T15:11:43.968459+0000","last_unstale":"2024-01-16T15:11:45.125998+0000","last_undegraded":"2024-01-16T15:11:45.125998+0000","last_fullsized":"2024-01-16T15:11:45.125998+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:07:37.262045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.996637+0000","last_change":"2024-01-16T15:11:42.947197+0000","last_active":"2024-01-16T15:11:43.996637+0000","last_peered":"2024-01-16T15:11:43.996637+0000","last_clean":"2024-01-16T15:11:43.996637+0000","last_became_active":"2024-01-16T15:11:42.946907+0000","last_became_peered":"2024-01-16T15:11:42.946907+0000","last_unstale":"2024-01-16T15:11:43.996637+0000","last_undegraded":"2024-01-16T15:11:43.996637+0000","last_fullsized":"2024-01-16T15:11:43.996637+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:58.422117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.983490+0000","last_change":"2024-01-16T15:11:43.981694+0000","last_active":"2024-01-16T15:11:43.983490+0000","last_peered":"2024-01-16T15:11:43.983490+0000","last_clean":"2024-01-16T15:11:43.983490+0000","last_became_active":"2024-01-16T15:11:43.981400+0000","last_became_peered":"2024-01-16T15:11:43.981400+0000","last_unstale":"2024-01-16T15:11:43.983490+0000","last_undegraded":"2024-01-16T15:11:43.983490+0000","last_fullsized":"2024-01-16T15:11:43.983490+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:04.971445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125904+0000","last_change":"2024-01-16T15:11:42.946199+0000","last_active":"2024-01-16T15:11:45.125904+0000","last_peered":"2024-01-16T15:11:45.125904+0000","last_clean":"2024-01-16T15:11:45.125904+0000","last_became_active":"2024-01-16T15:11:42.945878+0000","last_became_peered":"2024-01-16T15:11:42.945878+0000","last_unstale":"2024-01-16T15:11:45.125904+0000","last_undegraded":"2024-01-16T15:11:45.125904+0000","last_fullsized":"2024-01-16T15:11:45.125904+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:33.893645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949657+0000","last_change":"2024-01-16T15:11:43.982943+0000","last_active":"2024-01-16T15:11:44.949657+0000","last_peered":"2024-01-16T15:11:44.949657+0000","last_clean":"2024-01-16T15:11:44.949657+0000","last_became_active":"2024-01-16T15:11:43.981974+0000","last_became_peered":"2024-01-16T15:11:43.981974+0000","last_unstale":"2024-01-16T15:11:44.949657+0000","last_undegraded":"2024-01-16T15:11:44.949657+0000","last_fullsized":"2024-01-16T15:11:44.949657+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:54.339048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.967165+0000","last_change":"2024-01-16T15:11:42.939715+0000","last_active":"2024-01-16T15:11:43.967165+0000","last_peered":"2024-01-16T15:11:43.967165+0000","last_clean":"2024-01-16T15:11:43.967165+0000","last_became_active":"2024-01-16T15:11:42.939273+0000","last_became_peered":"2024-01-16T15:11:42.939273+0000","last_unstale":"2024-01-16T15:11:43.967165+0000","last_undegraded":"2024-01-16T15:11:43.967165+0000","last_fullsized":"2024-01-16T15:11:43.967165+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:47.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981368+0000","last_change":"2024-01-16T15:11:43.978138+0000","last_active":"2024-01-16T15:11:43.981368+0000","last_peered":"2024-01-16T15:11:43.981368+0000","last_clean":"2024-01-16T15:11:43.981368+0000","last_became_active":"2024-01-16T15:11:43.975628+0000","last_became_peered":"2024-01-16T15:11:43.975628+0000","last_unstale":"2024-01-16T15:11:43.981368+0000","last_undegraded":"2024-01-16T15:11:43.981368+0000","last_fullsized":"2024-01-16T15:11:43.981368+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:56:18.840803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.994112+0000","last_change":"2024-01-16T15:11:42.945945+0000","last_active":"2024-01-16T15:11:43.994112+0000","last_peered":"2024-01-16T15:11:43.994112+0000","last_clean":"2024-01-16T15:11:43.994112+0000","last_became_active":"2024-01-16T15:11:42.945668+0000","last_became_peered":"2024-01-16T15:11:42.945668+0000","last_unstale":"2024-01-16T15:11:43.994112+0000","last_undegraded":"2024-01-16T15:11:43.994112+0000","last_fullsized":"2024-01-16T15:11:43.994112+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:11.480993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.981708+0000","last_change":"2024-01-16T15:11:43.978141+0000","last_active":"2024-01-16T15:11:43.981708+0000","last_peered":"2024-01-16T15:11:43.981708+0000","last_clean":"2024-01-16T15:11:43.981708+0000","last_became_active":"2024-01-16T15:11:43.977635+0000","last_became_peered":"2024-01-16T15:11:43.977635+0000","last_unstale":"2024-01-16T15:11:43.981708+0000","last_undegraded":"2024-01-16T15:11:43.981708+0000","last_fullsized":"2024-01-16T15:11:43.981708+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:41.703547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.978046+0000","last_change":"2024-01-16T15:11:42.943533+0000","last_active":"2024-01-16T15:11:43.978046+0000","last_peered":"2024-01-16T15:11:43.978046+0000","last_clean":"2024-01-16T15:11:43.978046+0000","last_became_active":"2024-01-16T15:11:42.941668+0000","last_became_peered":"2024-01-16T15:11:42.941668+0000","last_unstale":"2024-01-16T15:11:43.978046+0000","last_undegraded":"2024-01-16T15:11:43.978046+0000","last_fullsized":"2024-01-16T15:11:43.978046+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:02.749395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.973202+0000","last_change":"2024-01-16T15:11:43.967549+0000","last_active":"2024-01-16T15:11:43.973202+0000","last_peered":"2024-01-16T15:11:43.973202+0000","last_clean":"2024-01-16T15:11:43.973202+0000","last_became_active":"2024-01-16T15:11:43.966248+0000","last_became_peered":"2024-01-16T15:11:43.966248+0000","last_unstale":"2024-01-16T15:11:43.973202+0000","last_undegraded":"2024-01-16T15:11:43.973202+0000","last_fullsized":"2024-01-16T15:11:43.973202+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:11:00.291648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.977684+0000","last_change":"2024-01-16T15:11:42.937123+0000","last_active":"2024-01-16T15:11:43.977684+0000","last_peered":"2024-01-16T15:11:43.977684+0000","last_clean":"2024-01-16T15:11:43.977684+0000","last_became_active":"2024-01-16T15:11:42.936685+0000","last_became_peered":"2024-01-16T15:11:42.936685+0000","last_unstale":"2024-01-16T15:11:43.977684+0000","last_undegraded":"2024-01-16T15:11:43.977684+0000","last_fullsized":"2024-01-16T15:11:43.977684+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:53.229746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125234+0000","last_change":"2024-01-16T15:11:43.974486+0000","last_active":"2024-01-16T15:11:45.125234+0000","last_peered":"2024-01-16T15:11:45.125234+0000","last_clean":"2024-01-16T15:11:45.125234+0000","last_became_active":"2024-01-16T15:11:43.974165+0000","last_became_peered":"2024-01-16T15:11:43.974165+0000","last_unstale":"2024-01-16T15:11:45.125234+0000","last_undegraded":"2024-01-16T15:11:45.125234+0000","last_fullsized":"2024-01-16T15:11:45.125234+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:09.604583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.965299+0000","last_change":"2024-01-16T15:11:42.951323+0000","last_active":"2024-01-16T15:11:43.965299+0000","last_peered":"2024-01-16T15:11:43.965299+0000","last_clean":"2024-01-16T15:11:43.965299+0000","last_became_active":"2024-01-16T15:11:42.950416+0000","last_became_peered":"2024-01-16T15:11:42.950416+0000","last_unstale":"2024-01-16T15:11:43.965299+0000","last_undegraded":"2024-01-16T15:11:43.965299+0000","last_fullsized":"2024-01-16T15:11:43.965299+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:07:36.246971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.985074+0000","last_change":"2024-01-16T15:11:43.982550+0000","last_active":"2024-01-16T15:11:43.985074+0000","last_peered":"2024-01-16T15:11:43.985074+0000","last_clean":"2024-01-16T15:11:43.985074+0000","last_became_active":"2024-01-16T15:11:43.982165+0000","last_became_peered":"2024-01-16T15:11:43.982165+0000","last_unstale":"2024-01-16T15:11:43.985074+0000","last_undegraded":"2024-01-16T15:11:43.985074+0000","last_fullsized":"2024-01-16T15:11:43.985074+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:39.732748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.999373+0000","last_change":"2024-01-16T15:11:42.940245+0000","last_active":"2024-01-16T15:11:43.999373+0000","last_peered":"2024-01-16T15:11:43.999373+0000","last_clean":"2024-01-16T15:11:43.999373+0000","last_became_active":"2024-01-16T15:11:42.939956+0000","last_became_peered":"2024-01-16T15:11:42.939956+0000","last_unstale":"2024-01-16T15:11:43.999373+0000","last_undegraded":"2024-01-16T15:11:43.999373+0000","last_fullsized":"2024-01-16T15:11:43.999373+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:53.879305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951878+0000","last_change":"2024-01-16T15:11:43.969368+0000","last_active":"2024-01-16T15:11:44.951878+0000","last_peered":"2024-01-16T15:11:44.951878+0000","last_clean":"2024-01-16T15:11:44.951878+0000","last_became_active":"2024-01-16T15:11:43.969050+0000","last_became_peered":"2024-01-16T15:11:43.969050+0000","last_unstale":"2024-01-16T15:11:44.951878+0000","last_undegraded":"2024-01-16T15:11:44.951878+0000","last_fullsized":"2024-01-16T15:11:44.951878+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:24.675058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.974899+0000","last_change":"2024-01-16T15:11:42.946563+0000","last_active":"2024-01-16T15:11:43.974899+0000","last_peered":"2024-01-16T15:11:43.974899+0000","last_clean":"2024-01-16T15:11:43.974899+0000","last_became_active":"2024-01-16T15:11:42.946280+0000","last_became_peered":"2024-01-16T15:11:42.946280+0000","last_unstale":"2024-01-16T15:11:43.974899+0000","last_undegraded":"2024-01-16T15:11:43.974899+0000","last_fullsized":"2024-01-16T15:11:43.974899+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:40:47.802703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.951690+0000","last_change":"2024-01-16T15:11:43.975625+0000","last_active":"2024-01-16T15:11:44.951690+0000","last_peered":"2024-01-16T15:11:44.951690+0000","last_clean":"2024-01-16T15:11:44.951690+0000","last_became_active":"2024-01-16T15:11:43.973975+0000","last_became_peered":"2024-01-16T15:11:43.973975+0000","last_unstale":"2024-01-16T15:11:44.951690+0000","last_undegraded":"2024-01-16T15:11:44.951690+0000","last_fullsized":"2024-01-16T15:11:44.951690+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:27.542322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.964374+0000","last_change":"2024-01-16T15:11:42.944690+0000","last_active":"2024-01-16T15:11:43.964374+0000","last_peered":"2024-01-16T15:11:43.964374+0000","last_clean":"2024-01-16T15:11:43.964374+0000","last_became_active":"2024-01-16T15:11:42.942599+0000","last_became_peered":"2024-01-16T15:11:42.942599+0000","last_unstale":"2024-01-16T15:11:43.964374+0000","last_undegraded":"2024-01-16T15:11:43.964374+0000","last_fullsized":"2024-01-16T15:11:43.964374+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:08.555876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125371+0000","last_change":"2024-01-16T15:11:43.984063+0000","last_active":"2024-01-16T15:11:45.125371+0000","last_peered":"2024-01-16T15:11:45.125371+0000","last_clean":"2024-01-16T15:11:45.125371+0000","last_became_active":"2024-01-16T15:11:43.983805+0000","last_became_peered":"2024-01-16T15:11:43.983805+0000","last_unstale":"2024-01-16T15:11:45.125371+0000","last_undegraded":"2024-01-16T15:11:45.125371+0000","last_fullsized":"2024-01-16T15:11:45.125371+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:48.394149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126150+0000","last_change":"2024-01-16T15:11:42.949900+0000","last_active":"2024-01-16T15:11:45.126150+0000","last_peered":"2024-01-16T15:11:45.126150+0000","last_clean":"2024-01-16T15:11:45.126150+0000","last_became_active":"2024-01-16T15:11:42.949210+0000","last_became_peered":"2024-01-16T15:11:42.949210+0000","last_unstale":"2024-01-16T15:11:45.126150+0000","last_undegraded":"2024-01-16T15:11:45.126150+0000","last_fullsized":"2024-01-16T15:11:45.126150+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:38.536768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.972610+0000","last_change":"2024-01-16T15:11:43.965937+0000","last_active":"2024-01-16T15:11:43.972610+0000","last_peered":"2024-01-16T15:11:43.972610+0000","last_clean":"2024-01-16T15:11:43.972610+0000","last_became_active":"2024-01-16T15:11:43.964521+0000","last_became_peered":"2024-01-16T15:11:43.964521+0000","last_unstale":"2024-01-16T15:11:43.972610+0000","last_undegraded":"2024-01-16T15:11:43.972610+0000","last_fullsized":"2024-01-16T15:11:43.972610+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:20.049382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.973902+0000","last_change":"2024-01-16T15:11:42.949691+0000","last_active":"2024-01-16T15:11:43.973902+0000","last_peered":"2024-01-16T15:11:43.973902+0000","last_clean":"2024-01-16T15:11:43.973902+0000","last_became_active":"2024-01-16T15:11:42.946557+0000","last_became_peered":"2024-01-16T15:11:42.946557+0000","last_unstale":"2024-01-16T15:11:43.973902+0000","last_undegraded":"2024-01-16T15:11:43.973902+0000","last_fullsized":"2024-01-16T15:11:43.973902+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:00.048547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.977035+0000","last_change":"2024-01-16T15:11:43.974388+0000","last_active":"2024-01-16T15:11:43.977035+0000","last_peered":"2024-01-16T15:11:43.977035+0000","last_clean":"2024-01-16T15:11:43.977035+0000","last_became_active":"2024-01-16T15:11:43.972973+0000","last_became_peered":"2024-01-16T15:11:43.972973+0000","last_unstale":"2024-01-16T15:11:43.977035+0000","last_undegraded":"2024-01-16T15:11:43.977035+0000","last_fullsized":"2024-01-16T15:11:43.977035+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:03.764468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.125043+0000","last_change":"2024-01-16T15:11:42.948209+0000","last_active":"2024-01-16T15:11:45.125043+0000","last_peered":"2024-01-16T15:11:45.125043+0000","last_clean":"2024-01-16T15:11:45.125043+0000","last_became_active":"2024-01-16T15:11:42.947919+0000","last_became_peered":"2024-01-16T15:11:42.947919+0000","last_unstale":"2024-01-16T15:11:45.125043+0000","last_undegraded":"2024-01-16T15:11:45.125043+0000","last_fullsized":"2024-01-16T15:11:45.125043+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:50.653737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.965210+0000","last_change":"2024-01-16T15:11:43.962876+0000","last_active":"2024-01-16T15:11:43.965210+0000","last_peered":"2024-01-16T15:11:43.965210+0000","last_clean":"2024-01-16T15:11:43.965210+0000","last_became_active":"2024-01-16T15:11:43.962156+0000","last_became_peered":"2024-01-16T15:11:43.962156+0000","last_unstale":"2024-01-16T15:11:43.965210+0000","last_undegraded":"2024-01-16T15:11:43.965210+0000","last_fullsized":"2024-01-16T15:11:43.965210+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:14:58.885498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.963709+0000","last_change":"2024-01-16T15:11:42.942997+0000","last_active":"2024-01-16T15:11:43.963709+0000","last_peered":"2024-01-16T15:11:43.963709+0000","last_clean":"2024-01-16T15:11:43.963709+0000","last_became_active":"2024-01-16T15:11:42.941797+0000","last_became_peered":"2024-01-16T15:11:42.941797+0000","last_unstale":"2024-01-16T15:11:43.963709+0000","last_undegraded":"2024-01-16T15:11:43.963709+0000","last_fullsized":"2024-01-16T15:11:43.963709+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:57.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:45.126130+0000","last_change":"2024-01-16T15:11:43.985660+0000","last_active":"2024-01-16T15:11:45.126130+0000","last_peered":"2024-01-16T15:11:45.126130+0000","last_clean":"2024-01-16T15:11:45.126130+0000","last_became_active":"2024-01-16T15:11:43.984537+0000","last_became_peered":"2024-01-16T15:11:43.984537+0000","last_unstale":"2024-01-16T15:11:45.126130+0000","last_undegraded":"2024-01-16T15:11:45.126130+0000","last_fullsized":"2024-01-16T15:11:45.126130+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:48.585576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.951058+0000","last_change":"2024-01-16T15:11:42.948507+0000","last_active":"2024-01-16T15:11:43.951058+0000","last_peered":"2024-01-16T15:11:43.951058+0000","last_clean":"2024-01-16T15:11:43.951058+0000","last_became_active":"2024-01-16T15:11:42.948148+0000","last_became_peered":"2024-01-16T15:11:42.948148+0000","last_unstale":"2024-01-16T15:11:43.951058+0000","last_undegraded":"2024-01-16T15:11:43.951058+0000","last_fullsized":"2024-01-16T15:11:43.951058+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:30:53.806477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.984508+0000","last_change":"2024-01-16T15:11:43.979245+0000","last_active":"2024-01-16T15:11:43.984508+0000","last_peered":"2024-01-16T15:11:43.984508+0000","last_clean":"2024-01-16T15:11:43.984508+0000","last_became_active":"2024-01-16T15:11:43.978948+0000","last_became_peered":"2024-01-16T15:11:43.978948+0000","last_unstale":"2024-01-16T15:11:43.984508+0000","last_undegraded":"2024-01-16T15:11:43.984508+0000","last_fullsized":"2024-01-16T15:11:43.984508+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:42.910524+0000","last_clean_scrub_stamp":"2024-01-16T15:11:42.910524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:36.704517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"101'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-16T15:11:44.949799+0000","last_change":"2024-01-16T15:11:42.946730+0000","last_active":"2024-01-16T15:11:44.949799+0000","last_peered":"2024-01-16T15:11:44.949799+0000","last_clean":"2024-01-16T15:11:44.949799+0000","last_became_active":"2024-01-16T15:11:42.946446+0000","last_became_peered":"2024-01-16T15:11:42.946446+0000","last_unstale":"2024-01-16T15:11:44.949799+0000","last_undegraded":"2024-01-16T15:11:44.949799+0000","last_fullsized":"2024-01-16T15:11:44.949799+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:11:41.895451+0000","last_clean_scrub_stamp":"2024-01-16T15:11:41.895451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:27.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":245,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:11:43.958912+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:11:43.958912+0000","last_peered":"2024-01-16T15:11:43.958912+0000","last_clean":"2024-01-16T15:11:43.958912+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:11:43.958912+0000","last_undegraded":"2024-01-16T15:11:43.958912+0000","last_fullsized":"2024-01-16T15:11:43.958912+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542324,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69020,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302820,"statfs":{"total":96636764160,"available":96566087680,"internally_reserved":0,"allocated":1847296,"data_stored":1126460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7230000000000001,"15min":1.504},"min":{"1min":0.57799999999999996,"5min":0.502,"15min":0.37},"max":{"1min":10.679,"5min":19.603000000000002,"15min":19.603000000000002},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7050000000000001,"15min":1.409},"min":{"1min":0.63600000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":10.728,"5min":24.015000000000001,"15min":24.015000000000001},"last":1.6839999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.552,"15min":1.4390000000000001},"min":{"1min":0.65600000000000003,"5min":0.5,"15min":0.40999999999999998},"max":{"1min":3.0139999999999998,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.6499999999999999,"15min":1.446},"min":{"1min":0.77800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.575,"5min":24.07,"15min":24.07},"last":1.198}]},{"osd":2,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.579,"15min":1.375},"min":{"1min":0.81499999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6319999999999999,"5min":24.108000000000001,"15min":24.108000000000001},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.6799999999999999,"15min":1.5069999999999999},"min":{"1min":0.86399999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.7210000000000001,"5min":24.806000000000001,"15min":24.806000000000001},"last":1.1220000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.96,"15min":1.7370000000000001},"min":{"1min":0.80200000000000005,"5min":0.67000000000000004,"15min":0.52600000000000002},"max":{"1min":2.427,"5min":23.741,"15min":23.741},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.365,"5min":2.012,"15min":1.6719999999999999},"min":{"1min":1.143,"5min":0.626,"15min":0.626},"max":{"1min":1.635,"5min":24.143000000000001,"15min":24.143000000000001},"last":1.5920000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":2.04,"15min":1.6679999999999999},"min":{"1min":0.79300000000000004,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":1.627,"5min":25.637,"15min":25.637},"last":1.722},{"interface":"front","average":{"1min":1.6759999999999999,"5min":2.2789999999999999,"15min":1.8680000000000001},"min":{"1min":0.97999999999999998,"5min":0.48099999999999998,"15min":0.44400000000000001},"max":{"1min":3.7879999999999998,"5min":25.594000000000001,"15min":25.594000000000001},"last":1.7390000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.897,"15min":1.736},"min":{"1min":1.1279999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.756,"5min":24.178999999999998,"15min":29.567},"last":1.494},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.8120000000000001,"15min":1.802},"min":{"1min":1.0680000000000001,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":1.845,"5min":24.538,"15min":27.937000000000001},"last":1.5409999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":2.1640000000000001,"15min":1.905},"min":{"1min":1.2350000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.758,"5min":36.826000000000001,"15min":41.206000000000003},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.633,"5min":2.5430000000000001,"15min":2.1509999999999998},"min":{"1min":1.089,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.847,"5min":39.932000000000002,"15min":39.932000000000002},"last":1.641}]}]},{"osd":6,"up_from":14,"seq":60129542323,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68436,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303404,"statfs":{"total":96636764160,"available":96566685696,"internally_reserved":0,"allocated":1253376,"data_stored":536632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.468,"15min":1.4219999999999999},"min":{"1min":0.64700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.919,"5min":12.467000000000001,"15min":20.739000000000001},"last":1.335},{"interface":"front","average":{"1min":1.28,"5min":1.55,"15min":1.613},"min":{"1min":0.70899999999999996,"5min":0.50600000000000001,"15min":0.498},"max":{"1min":1.798,"5min":15.401,"15min":50.472000000000001},"last":1.24}]},{"osd":1,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.48,"15min":1.385},"min":{"1min":0.73199999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":10.518000000000001,"5min":10.518000000000001,"15min":10.629},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.323,"5min":1.583,"15min":1.488},"min":{"1min":0.874,"5min":0.61799999999999999,"15min":0.53400000000000003},"max":{"1min":2.5590000000000002,"5min":9.6880000000000006,"15min":9.6880000000000006},"last":0.78300000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.569,"15min":1.633},"min":{"1min":0.65800000000000003,"5min":0.50800000000000001,"15min":0.46899999999999997},"max":{"1min":1.49,"5min":15.576000000000001,"15min":50.643999999999998},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.8560000000000001,"5min":1.6910000000000001,"15min":1.54},"min":{"1min":0.89100000000000001,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":10.728,"5min":10.728,"15min":10.843},"last":1.2769999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.609,"15min":1.595},"min":{"1min":1.0880000000000001,"5min":0.621,"15min":0.54900000000000004},"max":{"1min":3.0259999999999998,"5min":9.7460000000000004,"15min":15.675000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.796,"15min":1.7589999999999999},"min":{"1min":0.94899999999999995,"5min":0.55500000000000005,"15min":0.52600000000000002},"max":{"1min":2.609,"5min":15.504,"15min":50.561},"last":1.589}]},{"osd":4,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.6339999999999999,"15min":1.8240000000000001},"min":{"1min":0.97499999999999998,"5min":0.49299999999999999,"15min":0.46300000000000002},"max":{"1min":3.1320000000000001,"5min":7.391,"15min":47.488},"last":1.663},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.738,"15min":1.9219999999999999},"min":{"1min":0.99299999999999999,"5min":0.41299999999999998,"15min":0.40400000000000003},"max":{"1min":2.2200000000000002,"5min":15.638,"15min":50.381},"last":1.708}]},{"osd":5,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":1.631,"15min":1.5800000000000001},"min":{"1min":0.94099999999999995,"5min":0.443,"15min":0.443},"max":{"1min":10.388999999999999,"5min":10.388999999999999,"15min":10.654},"last":1.401},{"interface":"front","average":{"1min":1.712,"5min":1.97,"15min":1.8240000000000001},"min":{"1min":1.163,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.6640000000000001,"5min":18.699000000000002,"15min":18.699000000000002},"last":1.462}]},{"osd":7,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.9430000000000001,"15min":2.0800000000000001},"min":{"1min":0.71199999999999997,"5min":0.46600000000000003,"15min":0.44900000000000001},"max":{"1min":2.7109999999999999,"5min":15.702,"15min":52.912999999999997},"last":1.8360000000000001},{"interface":"front","average":{"1min":2.056,"5min":2.2650000000000001,"15min":2.3359999999999999},"min":{"1min":1.0409999999999999,"5min":0.495,"15min":0.495},"max":{"1min":10.659000000000001,"5min":16.690000000000001,"15min":26.413},"last":1.5169999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542323,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":1232896,"data_stored":529205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.298,"15min":1.357},"min":{"1min":0.78000000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.222,"5min":5.8369999999999997,"15min":13.938000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.585,"5min":1.351,"15min":1.3959999999999999},"min":{"1min":0.77300000000000002,"5min":0.56699999999999995,"15min":0.42299999999999999},"max":{"1min":5.4039999999999999,"5min":6.0010000000000003,"15min":16.896000000000001},"last":0.75700000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.4550000000000001,"15min":1.484},"min":{"1min":0.85499999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":8.1820000000000004,"5min":8.1820000000000004,"15min":15.500999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.23,"15min":1.4199999999999999},"min":{"1min":0.73399999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.2970000000000002,"5min":6.7720000000000002,"15min":35.328000000000003},"last":1.159}]},{"osd":2,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.355,"15min":1.5209999999999999},"min":{"1min":0.75900000000000001,"5min":0.52000000000000002,"15min":0.46200000000000002},"max":{"1min":3.0110000000000001,"5min":6.9020000000000001,"15min":35.398000000000003},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3799999999999999,"15min":1.419},"min":{"1min":0.81999999999999995,"5min":0.47999999999999998,"15min":0.38900000000000001},"max":{"1min":2.5550000000000002,"5min":8.0619999999999994,"15min":15.4},"last":1.254}]},{"osd":3,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.8200000000000001,"15min":1.7889999999999999},"min":{"1min":0.98399999999999999,"5min":0.63200000000000001,"15min":0.50900000000000001},"max":{"1min":8.0809999999999995,"5min":16.931999999999999,"15min":24.254000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.6850000000000001,"15min":1.8169999999999999},"min":{"1min":0.89600000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"max":{"1min":3.1619999999999999,"5min":19.111999999999998,"15min":35.268000000000001},"last":1.468}]},{"osd":4,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.3759999999999999,"15min":1.6599999999999999},"min":{"1min":0.83199999999999996,"5min":0.42199999999999999,"15min":0.39000000000000001},"max":{"1min":3.2599999999999998,"5min":7.2569999999999997,"15min":19.059000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5409999999999999,"15min":1.8120000000000001},"min":{"1min":1.1279999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":28.760000000000002},"last":1.5509999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.8620000000000001,"15min":1.897},"min":{"1min":1.0569999999999999,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":3.3220000000000001,"5min":17.419,"15min":35.427999999999997},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.643,"5min":1.903,"15min":1.889},"min":{"1min":1.173,"5min":0.53500000000000003,"15min":0.48199999999999998},"max":{"1min":3.391,"5min":16.262,"15min":16.262},"last":1.502}]},{"osd":7,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":2.0590000000000002,"5min":1.9730000000000001,"15min":2.0089999999999999},"min":{"1min":1.117,"5min":0.60899999999999999,"15min":0.41099999999999998},"max":{"1min":8.2509999999999994,"5min":12.385999999999999,"15min":29.672999999999998},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6579999999999999,"15min":1.827},"min":{"1min":1.216,"5min":0.47599999999999998,"15min":0.42899999999999999},"max":{"1min":3.2229999999999999,"5min":14.02,"15min":35.363},"last":1.363}]}]},{"osd":4,"up_from":14,"seq":60129542324,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64292,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307548,"statfs":{"total":96636764160,"available":96570929152,"internally_reserved":0,"allocated":1265664,"data_stored":537716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.639,"15min":1.3999999999999999},"min":{"1min":0.69699999999999995,"5min":0.59399999999999997,"15min":0.52800000000000002},"max":{"1min":1.623,"5min":15.942,"15min":15.942},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.3500000000000001,"15min":1.425},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.45400000000000001},"max":{"1min":1.661,"5min":16.219999999999999,"15min":16.219999999999999},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.468,"15min":1.474},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.54100000000000004},"max":{"1min":1.9490000000000001,"5min":16.280000000000001,"15min":16.280000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.7,"15min":1.48},"min":{"1min":0.81999999999999995,"5min":0.68899999999999995,"15min":0.60499999999999998},"max":{"1min":3.411,"5min":15.888,"15min":15.888},"last":1.034}]},{"osd":2,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.25,"15min":1.5049999999999999},"min":{"1min":0.76300000000000001,"5min":0.70999999999999996,"15min":0.50800000000000001},"max":{"1min":1.4790000000000001,"5min":5.2859999999999996,"15min":15.986000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.897,"15min":1.5920000000000001},"min":{"1min":0.878,"5min":0.64900000000000002,"15min":0.64800000000000002},"max":{"1min":1.9239999999999999,"5min":17.175000000000001,"15min":17.175000000000001},"last":1.319}]},{"osd":3,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.5820000000000001,"15min":1.7569999999999999},"min":{"1min":0.83699999999999997,"5min":0.71499999999999997,"15min":0.57599999999999996},"max":{"1min":1.643,"5min":16.469000000000001,"15min":16.469000000000001},"last":1.379},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.415,"15min":1.5820000000000001},"min":{"1min":0.86399999999999999,"5min":0.71699999999999997,"15min":0.55600000000000005},"max":{"1min":1.6020000000000001,"5min":5.2469999999999999,"15min":21.706},"last":1.248}]},{"osd":5,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.825,"15min":1.609},"min":{"1min":0.83799999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.9279999999999999,"5min":18.347999999999999,"15min":18.347999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.5960000000000001,"15min":1.643},"min":{"1min":0.86399999999999999,"5min":0.58299999999999996,"15min":0.45800000000000002},"max":{"1min":1.736,"5min":22.937000000000001,"15min":22.937000000000001},"last":1.611}]},{"osd":6,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.6619999999999999,"15min":1.5720000000000001},"min":{"1min":0.877,"5min":0.60699999999999998,"15min":0.501},"max":{"1min":1.7110000000000001,"5min":18.033000000000001,"15min":18.033000000000001},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.482,"5min":2.0950000000000002,"15min":1.8759999999999999},"min":{"1min":1.3140000000000001,"5min":0.52400000000000002,"15min":0.51800000000000002},"max":{"1min":2.133,"5min":17.794,"15min":20.939},"last":1.5660000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:11:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.708,"15min":1.7509999999999999},"min":{"1min":0.97699999999999998,"5min":0.53500000000000003,"15min":0.44500000000000001},"max":{"1min":1.873,"5min":16.324999999999999,"15min":16.324999999999999},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.764,"15min":1.851},"min":{"1min":1.22,"5min":0.55200000000000005,"15min":0.52200000000000002},"max":{"1min":1.893,"5min":12.335000000000001,"15min":20.277999999999999},"last":1.655}]}]},{"osd":3,"up_from":14,"seq":60129542323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72608,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299232,"statfs":{"total":96636764160,"available":96562413568,"internally_reserved":0,"allocated":1261568,"data_stored":537614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.95099999999999996,"15min":0.98099999999999998},"min":{"1min":0.61599999999999999,"5min":0.34300000000000003,"15min":0.33600000000000002},"max":{"1min":1.599,"5min":4.3700000000000001,"15min":5.3570000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.137,"15min":1.099},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.5859999999999999,"5min":12.292999999999999,"15min":12.292999999999999},"last":1.3200000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.97099999999999997,"15min":0.92900000000000005},"min":{"1min":0.65400000000000003,"5min":0.33100000000000002,"15min":0.309},"max":{"1min":1.8380000000000001,"5min":4.0979999999999999,"15min":4.0979999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.119,"15min":1.1499999999999999},"min":{"1min":0.624,"5min":0.315,"15min":0.315},"max":{"1min":1.861,"5min":5.0359999999999996,"15min":13.443},"last":1.117}]},{"osd":2,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.1890000000000001,"15min":1.1399999999999999},"min":{"1min":0.69999999999999996,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":4.4850000000000003},"last":1.071},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.236,"15min":1.2210000000000001},"min":{"1min":0.746,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.129,"5min":5.4539999999999997,"15min":5.5110000000000001},"last":1.242}]},{"osd":4,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3839999999999999,"15min":1.353},"min":{"1min":0.71999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4,"5min":12.744,"15min":13.292},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.4330000000000001,"15min":1.4510000000000001},"min":{"1min":0.94399999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.4590000000000001,"5min":6.4279999999999999,"15min":11.068},"last":1.5249999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.5669999999999999,"15min":1.5449999999999999},"min":{"1min":1.0900000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.29,"5min":17.096,"15min":17.096},"last":1.292},{"interface":"front","average":{"1min":1.492,"5min":1.5149999999999999,"15min":1.552},"min":{"1min":1.1439999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.2639999999999998,"5min":9.3629999999999995,"15min":12.106},"last":1.3740000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.1909999999999998,"15min":1.736},"min":{"1min":0.80100000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":3.8239999999999998,"5min":43.609999999999999,"15min":43.609999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.903,"15min":1.6779999999999999},"min":{"1min":1.056,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.7799999999999998,"5min":38.784999999999997,"15min":38.784999999999997},"last":1.448}]},{"osd":7,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.8999999999999999,"15min":1.712},"min":{"1min":0.95999999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":7.8899999999999997,"5min":25.071000000000002,"15min":25.071000000000002},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.806,"15min":1.7549999999999999},"min":{"1min":0.93400000000000005,"5min":0.435,"15min":0.435},"max":{"1min":4.2880000000000003,"5min":13.365,"15min":18.846},"last":1.4219999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68420,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303420,"statfs":{"total":96636764160,"available":96566702080,"internally_reserved":0,"allocated":1232896,"data_stored":529205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.89600000000000002,"15min":1.002},"min":{"1min":0.5,"5min":0.311,"15min":0.224},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":11.804},"last":0.5},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.054,"15min":1.1200000000000001},"min":{"1min":0.443,"5min":0.435,"15min":0.35799999999999998},"max":{"1min":2.8450000000000002,"5min":2.8839999999999999,"15min":13.066000000000001},"last":0.443}]},{"osd":1,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.107,"15min":1.1519999999999999},"min":{"1min":0.52000000000000002,"5min":0.434,"15min":0.29899999999999999},"max":{"1min":2.2599999999999998,"5min":4.1440000000000001,"15min":14.199999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.143,"15min":1.196},"min":{"1min":0.52800000000000002,"5min":0.45400000000000001,"15min":0.372},"max":{"1min":2.9900000000000002,"5min":4.056,"15min":13.15},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.98599999999999999,"15min":1.1359999999999999},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.31900000000000001},"max":{"1min":2.605,"5min":2.605,"15min":14.965},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.1850000000000001,"15min":1.2509999999999999},"min":{"1min":0.57599999999999996,"5min":0.47999999999999998,"15min":0.39300000000000002},"max":{"1min":2.3370000000000002,"5min":2.7890000000000001,"15min":11.717000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.5600000000000001,"15min":1.6539999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.46300000000000002},"max":{"1min":2.3010000000000002,"5min":11.558,"15min":19.957999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.552,"15min":1.5920000000000001},"min":{"1min":0.95899999999999996,"5min":0.71099999999999997,"15min":0.55700000000000005},"max":{"1min":3.419,"5min":11.58,"15min":15.151},"last":0.95899999999999996}]},{"osd":5,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.397,"15min":1.5780000000000001},"min":{"1min":0.79800000000000004,"5min":0.61099999999999999,"15min":0.60899999999999999},"max":{"1min":3.6160000000000001,"5min":3.6160000000000001,"15min":26.501000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.5429999999999999,"15min":1.4430000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.47999999999999998},"max":{"1min":3.1949999999999998,"5min":13.760999999999999,"15min":13.760999999999999},"last":1.0109999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.454,"15min":1.488},"min":{"1min":0.89200000000000002,"5min":0.86599999999999999,"15min":0.438},"max":{"1min":3.1200000000000001,"5min":4.4269999999999996,"15min":15.144},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.522,"5min":1.5229999999999999,"15min":1.8440000000000001},"min":{"1min":0.92800000000000005,"5min":0.71099999999999997,"15min":0.48199999999999998},"max":{"1min":3.2469999999999999,"5min":5.1310000000000002,"15min":66.061999999999998},"last":0.995}]},{"osd":7,"last update":"Tue Jan 16 15:11:40 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.5960000000000001,"15min":1.6899999999999999},"min":{"1min":0.874,"5min":0.871,"15min":0.61099999999999999},"max":{"1min":3.7290000000000001,"5min":7.5700000000000003,"15min":16.667000000000002},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.5720000000000001,"15min":1.6160000000000001},"min":{"1min":0.93400000000000005,"5min":0.89800000000000002,"15min":0.5},"max":{"1min":3.1600000000000001,"5min":7.5960000000000001,"15min":11.271000000000001},"last":1.0589999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542323,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68428,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303412,"statfs":{"total":96636764160,"available":96566693888,"internally_reserved":0,"allocated":1245184,"data_stored":535980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.0609999999999999,"15min":1.0900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.248},"max":{"1min":1.8089999999999999,"5min":8.7219999999999995,"15min":21.260000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1459999999999999,"15min":1.272},"min":{"1min":0.59599999999999997,"5min":0.47199999999999998,"15min":0.35199999999999998},"max":{"1min":1.712,"5min":5.75,"15min":32.787999999999997},"last":0.61399999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.999,"15min":1.0049999999999999},"min":{"1min":0.52900000000000003,"5min":0.34899999999999998,"15min":0.28100000000000003},"max":{"1min":2.226,"5min":7.3970000000000002,"15min":14.817},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2250000000000001,"15min":1.169},"min":{"1min":0.80300000000000005,"5min":0.42599999999999999,"15min":0.41499999999999998},"max":{"1min":1.7769999999999999,"5min":8.8040000000000003,"15min":8.8450000000000006},"last":0.94699999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.343,"15min":1.234},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":1.907,"5min":9.2330000000000005,"15min":9.3149999999999995},"last":1.113},{"interface":"front","average":{"1min":1.159,"5min":1.163,"15min":1.323},"min":{"1min":0.747,"5min":0.45500000000000002,"15min":0.40400000000000003},"max":{"1min":1.871,"5min":8.6910000000000007,"15min":41.133000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.55,"15min":1.5529999999999999},"min":{"1min":0.78800000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.0790000000000002,"5min":9.9700000000000006,"15min":19.684000000000001},"last":1.05},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4179999999999999,"15min":1.448},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.53900000000000003},"max":{"1min":2,"5min":10.785,"15min":13.789999999999999},"last":1.0760000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.54,"15min":1.5649999999999999},"min":{"1min":0.77000000000000002,"5min":0.622,"15min":0.43099999999999999},"max":{"1min":1.9059999999999999,"5min":7.617,"15min":15.778},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.494,"5min":1.6839999999999999,"15min":1.774},"min":{"1min":0.84099999999999997,"5min":0.82399999999999995,"15min":0.623},"max":{"1min":2.0390000000000001,"5min":9.4890000000000008,"15min":35.747},"last":1.4219999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.679,"15min":1.663},"min":{"1min":0.82799999999999996,"5min":0.64400000000000002,"15min":0.629},"max":{"1min":2.1920000000000002,"5min":15.586,"15min":16.231000000000002},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.6359999999999999,"15min":1.577},"min":{"1min":0.69299999999999995,"5min":0.63200000000000001,"15min":0.56599999999999995},"max":{"1min":4.1349999999999998,"5min":11.433999999999999,"15min":13.56},"last":1.264}]},{"osd":7,"last update":"Tue Jan 16 15:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.8460000000000001,"15min":1.8080000000000001},"min":{"1min":0.72599999999999998,"5min":0.66900000000000004,"15min":0.58699999999999997},"max":{"1min":2.1909999999999998,"5min":18.907,"15min":21.902999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.659,"15min":1.8120000000000001},"min":{"1min":0.86099999999999999,"5min":0.70299999999999996,"15min":0.64700000000000002},"max":{"1min":2.1259999999999999,"5min":5.1260000000000003,"15min":31.044},"last":0.83199999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69012,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302828,"statfs":{"total":96636764160,"available":96566095872,"internally_reserved":0,"allocated":1843200,"data_stored":1126930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.873,"15min":1.004},"min":{"1min":0.63500000000000001,"5min":0.29499999999999998,"15min":0.25},"max":{"1min":1.4179999999999999,"5min":2.7320000000000002,"15min":22.619},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.052,"15min":1.1719999999999999},"min":{"1min":0.56599999999999995,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.403,"5min":9.4499999999999993,"15min":17.876000000000001},"last":1.093}]},{"osd":2,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.3620000000000001,"15min":1.3759999999999999},"min":{"1min":0.628,"5min":0.45100000000000001,"15min":0.42999999999999999},"max":{"1min":5.3860000000000001,"5min":9.5579999999999998,"15min":17.760999999999999},"last":1.036},{"interface":"front","average":{"1min":1.383,"5min":1.181,"15min":1.325},"min":{"1min":0.69799999999999995,"5min":0.34999999999999998,"15min":0.33100000000000002},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":22.672999999999998},"last":1.4370000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.0409999999999999,"15min":1.1659999999999999},"min":{"1min":0.71799999999999997,"5min":0.40000000000000002,"15min":0.39200000000000002},"max":{"1min":6.7850000000000001,"5min":6.7850000000000001,"15min":16.382999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.143,"5min":1.331,"15min":1.349},"min":{"1min":0.85099999999999998,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":1.621,"5min":9.6029999999999998,"15min":17.981000000000002},"last":0.90900000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.339,"15min":1.538},"min":{"1min":1.0860000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.2330000000000001,"5min":5.109,"15min":22.780000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.262,"15min":1.5109999999999999},"min":{"1min":0.80700000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.181,"5min":3.028,"15min":19.422000000000001},"last":1.2110000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.534,"15min":1.847},"min":{"1min":1.071,"5min":0.441,"15min":0.441},"max":{"1min":2.1139999999999999,"5min":7.9530000000000003,"15min":44.930999999999997},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.798,"15min":1.73},"min":{"1min":0.94299999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":3.859,"5min":21.797999999999998,"15min":21.797999999999998},"last":1.2889999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.829,"15min":1.887},"min":{"1min":1.1539999999999999,"5min":0.622,"15min":0.622},"max":{"1min":2.3780000000000001,"5min":14.958,"15min":27.442},"last":1.264},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.6699999999999999,"15min":1.903},"min":{"1min":1.0409999999999999,"5min":0.57399999999999995,"15min":0.50700000000000001},"max":{"1min":2.2999999999999998,"5min":9.8759999999999994,"15min":27.411999999999999},"last":1.532}]},{"osd":7,"last update":"Tue Jan 16 15:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.7230000000000001,"15min":1.7270000000000001},"min":{"1min":1.1739999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2669999999999999,"5min":15.005000000000001,"15min":18.152999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.843,"5min":1.645,"15min":1.7589999999999999},"min":{"1min":1.1339999999999999,"5min":0.63700000000000001,"15min":0.60099999999999998},"max":{"1min":5.2359999999999998,"5min":9.6259999999999994,"15min":22.815000000000001},"last":1.327}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:11:48.866 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:11:48.866 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:11:49.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.145+0000 7fda8d400640 1 Processor -- start 2024-01-16T15:11:49.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.145+0000 7fda8d400640 1 -- start start 2024-01-16T15:11:49.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.147+0000 7fda8d400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fda88141360 0x7fda880cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.147+0000 7fda8d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880cdcc0 0x7fda8806c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.147+0000 7fda8d400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8813e440 con 0x7fda8815ed20 2024-01-16T15:11:49.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.147+0000 7fda8d400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda880ce1d0 con 0x7fda88141360 2024-01-16T15:11:49.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.147+0000 7fda8d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda880ce4b0 con 0x7fda880cdcc0 2024-01-16T15:11:49.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.147+0000 7fda86800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fda8815ed20 0x7fda8815f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36662/0 (socket says 172.21.15.141:36662) 2024-01-16T15:11:49.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.147+0000 7fda86800640 1 -- 172.21.15.141:0/2205214189 learned_addr learned my addr 172.21.15.141:0/2205214189 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:49.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda87200640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880cdcc0 0x7fda8806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.125 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda7dc00640 1 --2- 172.21.15.141:0/2205214189 >> v2:172.21.15.188:3300/0 conn(0x7fda88141360 0x7fda880cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091568954 0 0) 0x7fda8813e440 con 0x7fda8815ed20 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda68003520 con 0x7fda8815ed20 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 962811742 0 0) 0x7fda68003520 con 0x7fda8815ed20 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880cdcc0 msgr2=0x7fda8806c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880cdcc0 0x7fda8806c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 >> v2:172.21.15.188:3300/0 conn(0x7fda88141360 msgr2=0x7fda880cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 --2- 172.21.15.141:0/2205214189 >> v2:172.21.15.188:3300/0 conn(0x7fda88141360 0x7fda880cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda880ce790 con 0x7fda8815ed20 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fda780019f0 con 0x7fda8815ed20 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 >> v1:172.21.15.141:6789/0 conn(0x7fda8815ed20 legacy=0x7fda8815f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880cdcc0 0x7fda68003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda68003f40 0x7fda680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda68005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.148+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880ce1d0 con 0x7fda68004d00 2024-01-16T15:11:49.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880ce4b0 con 0x7fda68003f40 2024-01-16T15:11:49.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6800ba30 con 0x7fda880cdcc0 2024-01-16T15:11:49.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda87200640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda68005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda87200640 1 -- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880cdcc0 msgr2=0x7fda68003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:49.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda87200640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880cdcc0 0x7fda68003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda87200640 1 -- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda68003f40 msgr2=0x7fda680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:49.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda87200640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda68003f40 0x7fda680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.149+0000 7fda87200640 1 -- 172.21.15.141:0/2205214189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda68003520 con 0x7fda68004d00 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.150+0000 7fda87200640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda68005180 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fda80009ac0 tx=0x7fda680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b861953abb951e60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.150+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8002c710 con 0x7fda68004d00 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.150+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8002c870 con 0x7fda68004d00 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.150+0000 7fda85e00640 1 -- 172.21.15.141:0/2205214189 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8002cc00 con 0x7fda68004d00 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.151+0000 7fda8d400640 1 -- 172.21.15.141:0/2205214189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 msgr2=0x7fda68005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.151+0000 7fda8d400640 1 --2- 172.21.15.141:0/2205214189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda68005180 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fda80009ac0 tx=0x7fda680057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.151+0000 7fda87200640 1 -- 172.21.15.141:0/2205214189 reap_dead start 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.151+0000 7fda8d400640 1 -- 172.21.15.141:0/2205214189 shutdown_connections 2024-01-16T15:11:49.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.151+0000 7fda8d400640 1 -- 172.21.15.141:0/2205214189 >> 172.21.15.141:0/2205214189 conn(0x7fda880b0be0 msgr2=0x7fda880b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:49.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.151+0000 7fda8d400640 1 -- 172.21.15.141:0/2205214189 shutdown_connections 2024-01-16T15:11:49.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.151+0000 7fda8d400640 1 -- 172.21.15.141:0/2205214189 wait complete. 2024-01-16T15:11:49.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.153+0000 7fda8d400640 1 Processor -- start 2024-01-16T15:11:49.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.153+0000 7fda8d400640 1 -- start start 2024-01-16T15:11:49.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda8d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda8815d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda8d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda880ad170 0x7fda880b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda8d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880b3b00 0x7fda880cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda8d400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8815d9b0 con 0x7fda68004d00 2024-01-16T15:11:49.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda8d400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880b3fa0 con 0x7fda880ad170 2024-01-16T15:11:49.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda8d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8815c6b0 con 0x7fda880b3b00 2024-01-16T15:11:49.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda86800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda8815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda86800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda8815d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39010/0 (socket says 172.21.15.141:39010) 2024-01-16T15:11:49.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.154+0000 7fda86800640 1 -- 172.21.15.141:0/1106399888 learned_addr learned my addr 172.21.15.141:0/1106399888 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda87200640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880b3b00 0x7fda880cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda86800640 1 -- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880b3b00 msgr2=0x7fda880cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda86800640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880b3b00 0x7fda880cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda86800640 1 -- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda880ad170 msgr2=0x7fda880b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda86800640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda880ad170 0x7fda880b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda86800640 1 -- 172.21.15.141:0/1106399888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8815c990 con 0x7fda68004d00 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda86800640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda8815d470 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fda7800ca70 tx=0x7fda7800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda7f400640 1 -- 172.21.15.141:0/1106399888 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7800a7c0 con 0x7fda68004d00 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda7f400640 1 -- 172.21.15.141:0/1106399888 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7800a920 con 0x7fda68004d00 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda7f400640 1 -- 172.21.15.141:0/1106399888 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7800acb0 con 0x7fda68004d00 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8815cc70 con 0x7fda68004d00 2024-01-16T15:11:49.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.155+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda88151690 con 0x7fda68004d00 2024-01-16T15:11:49.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.156+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda88067670 con 0x7fda68004d00 2024-01-16T15:11:49.133 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.157+0000 7fda7f400640 1 -- 172.21.15.141:0/1106399888 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fda780070d0 con 0x7fda68004d00 2024-01-16T15:11:49.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.159+0000 7fda7f400640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda6006c220 0x7fda6006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.159+0000 7fda7f400640 1 -- 172.21.15.141:0/1106399888 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda78088150 con 0x7fda68004d00 2024-01-16T15:11:49.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.159+0000 7fda7dc00640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda6006c220 0x7fda6006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.161+0000 7fda7dc00640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda6006c220 0x7fda6006e6e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fda70006a90 tx=0x7fda70006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:49.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.163+0000 7fda7f400640 1 -- 172.21.15.141:0/1106399888 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda78054720 con 0x7fda68004d00 2024-01-16T15:11:49.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.422+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fda880ad830 con 0x7fda68004d00 2024-01-16T15:11:49.403 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.427+0000 7fda7f400640 1 -- 172.21.15.141:0/1106399888 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7fda78018020 con 0x7fda68004d00 2024-01-16T15:11:49.404 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T15:11:49.404 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T15:11:49.404 INFO:teuthology.orchestra.run.smithi141.stdout: caps mds = "allow" 2024-01-16T15:11:49.404 INFO:teuthology.orchestra.run.smithi141.stdout: caps mon = "allow r" 2024-01-16T15:11:49.404 INFO:teuthology.orchestra.run.smithi141.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:11:49.404 INFO:teuthology.orchestra.run.smithi141.stderr:updated caps for client.0 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda6006c220 msgr2=0x7fda6006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda6006c220 0x7fda6006e6e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fda70006a90 tx=0x7fda70006a20 comp rx=0 tx=0).stop 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 msgr2=0x7fda8815d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda8815d470 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fda7800ca70 tx=0x7fda7800cf30 comp rx=0 tx=0).stop 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 shutdown_connections 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fda6006c220 0x7fda6006e6e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fda880b3b00 0x7fda880cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda880ad170 0x7fda880b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 --2- 172.21.15.141:0/1106399888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda68004d00 0x7fda8815d470 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.430+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 >> 172.21.15.141:0/1106399888 conn(0x7fda880b0be0 msgr2=0x7fda8815aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:49.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.431+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 shutdown_connections 2024-01-16T15:11:49.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.431+0000 7fda8d400640 1 -- 172.21.15.141:0/1106399888 wait complete. 2024-01-16T15:11:49.425 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:11:49.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.663+0000 7fe021600640 1 Processor -- start 2024-01-16T15:11:49.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.663+0000 7fe021600640 1 -- start start 2024-01-16T15:11:49.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.664+0000 7fe021600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe01c143c40 0x7fe01c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.664+0000 7fe021600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c070e50 con 0x7fe01c0717d0 2024-01-16T15:11:49.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.664+0000 7fe021600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c144020 con 0x7fe01c0ccc10 2024-01-16T15:11:49.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.664+0000 7fe021600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01c0723c0 con 0x7fe01c143c40 2024-01-16T15:11:49.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.665+0000 7fe013200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe01c0717d0 0x7fe01c0cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36678/0 (socket says 172.21.15.141:36678) 2024-01-16T15:11:49.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.665+0000 7fe013200640 1 -- 172.21.15.141:0/802584706 learned_addr learned my addr 172.21.15.141:0/802584706 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.665+0000 7fe01be00640 1 --2- 172.21.15.141:0/802584706 >> v2:172.21.15.188:3300/0 conn(0x7fe01c143c40 0x7fe01c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.665+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659308810 0 0) 0x7fe01c070e50 con 0x7fe01c0717d0 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.665+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc003520 con 0x7fe01c0717d0 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.665+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877170971 0 0) 0x7fdffc003520 con 0x7fe01c0717d0 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 >> v2:172.21.15.188:3300/0 conn(0x7fe01c143c40 msgr2=0x7fe01c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 --2- 172.21.15.141:0/802584706 >> v2:172.21.15.188:3300/0 conn(0x7fe01c143c40 0x7fe01c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 >> v1:172.21.15.188:6789/0 conn(0x7fe01c0ccc10 legacy=0x7fe01c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c071e00 con 0x7fe01c0717d0 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe004001670 con 0x7fe01c0717d0 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 >> v1:172.21.15.141:6789/0 conn(0x7fe01c0717d0 legacy=0x7fe01c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdffc0039c0 0x7fdffc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fdffc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdffc0051d0 0x7fdffc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0723c0 con 0x7fdffc0047c0 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc00c120 con 0x7fdffc0051d0 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc00c2e0 con 0x7fdffc0039c0 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe013200640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fdffc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe01be00640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdffc0039c0 0x7fdffc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe020c00640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdffc0051d0 0x7fdffc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe013200640 1 -- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdffc0039c0 msgr2=0x7fdffc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe013200640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdffc0039c0 0x7fdffc003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe013200640 1 -- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdffc0051d0 msgr2=0x7fdffc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe013200640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdffc0051d0 0x7fdffc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.666+0000 7fe013200640 1 -- 172.21.15.141:0/802584706 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc003520 con 0x7fdffc0047c0 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.667+0000 7fe020c00640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdffc0051d0 0x7fdffc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:49.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.667+0000 7fe020c00640 1 -- 172.21.15.141:0/802584706 reap_dead start 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.667+0000 7fe013200640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fdffc004c40 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe004001eb0 tx=0x7fdffc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36373028e999a1c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004005ad0 con 0x7fdffc0047c0 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe004005c30 con 0x7fdffc0047c0 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe01b400640 1 -- 172.21.15.141:0/802584706 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00402ca00 con 0x7fdffc0047c0 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe021600640 1 -- 172.21.15.141:0/802584706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 msgr2=0x7fdffc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe021600640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fdffc004c40 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe004001eb0 tx=0x7fdffc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe021600640 1 -- 172.21.15.141:0/802584706 shutdown_connections 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe021600640 1 --2- 172.21.15.141:0/802584706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fdffc004c40 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe021600640 1 -- 172.21.15.141:0/802584706 >> 172.21.15.141:0/802584706 conn(0x7fe01c06a800 msgr2=0x7fe01c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.668+0000 7fe021600640 1 -- 172.21.15.141:0/802584706 shutdown_connections 2024-01-16T15:11:49.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.669+0000 7fe021600640 1 -- 172.21.15.141:0/802584706 wait complete. 2024-01-16T15:11:49.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.670+0000 7fe021600640 1 Processor -- start 2024-01-16T15:11:49.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.670+0000 7fe021600640 1 -- start start 2024-01-16T15:11:49.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.671+0000 7fe021600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fe01c15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.671+0000 7fe021600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe01c0caf70 0x7fe01c13ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.671+0000 7fe021600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe01c0cb4c0 0x7fe01c13b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.671+0000 7fe021600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c138e90 con 0x7fdffc0047c0 2024-01-16T15:11:49.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.671+0000 7fe021600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c15c6c0 con 0x7fe01c0caf70 2024-01-16T15:11:49.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.671+0000 7fe021600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c137d90 con 0x7fe01c0cb4c0 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fe01c15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fe01c15a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39038/0 (socket says 172.21.15.141:39038) 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 -- 172.21.15.141:0/2742778284 learned_addr learned my addr 172.21.15.141:0/2742778284 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe013200640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe01c0caf70 0x7fe01c13ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 -- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe01c0cb4c0 msgr2=0x7fe01c13b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe01c0cb4c0 0x7fe01c13b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 -- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe01c0caf70 msgr2=0x7fe01c13ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe01c0caf70 0x7fe01c13ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 -- 172.21.15.141:0/2742778284 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c1377d0 con 0x7fdffc0047c0 2024-01-16T15:11:49.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.672+0000 7fe01be00640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fe01c15a8f0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe00c010790 tx=0x7fe00c010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:49.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.673+0000 7fe018c00640 1 -- 172.21.15.141:0/2742778284 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c009630 con 0x7fdffc0047c0 2024-01-16T15:11:49.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.673+0000 7fe018c00640 1 -- 172.21.15.141:0/2742778284 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c00eb60 con 0x7fdffc0047c0 2024-01-16T15:11:49.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.673+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c137ab0 con 0x7fdffc0047c0 2024-01-16T15:11:49.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.673+0000 7fe018c00640 1 -- 172.21.15.141:0/2742778284 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c010ea0 con 0x7fdffc0047c0 2024-01-16T15:11:49.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.673+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c136450 con 0x7fdffc0047c0 2024-01-16T15:11:49.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.674+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c139100 con 0x7fdffc0047c0 2024-01-16T15:11:49.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.675+0000 7fe018c00640 1 -- 172.21.15.141:0/2742778284 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fe00c01c400 con 0x7fdffc0047c0 2024-01-16T15:11:49.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.676+0000 7fe018c00640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdff406c1f0 0x7fdff406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:49.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.676+0000 7fe018c00640 1 -- 172.21.15.141:0/2742778284 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe00c088eb0 con 0x7fdffc0047c0 2024-01-16T15:11:49.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.676+0000 7fe013200640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdff406c1f0 0x7fdff406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:49.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.678+0000 7fe013200640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdff406c1f0 0x7fdff406e6b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe004034000 tx=0x7fe004009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:49.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.680+0000 7fe018c00640 1 -- 172.21.15.141:0/2742778284 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe00c055590 con 0x7fdffc0047c0 2024-01-16T15:11:49.871 INFO:tasks.ceph.mds.d.smithi188.stderr:2024-01-16T15:11:49.895+0000 7ff7f6e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:11:49.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.907+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe01c066ea0 con 0x7fdffc0047c0 2024-01-16T15:11:49.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.909+0000 7fe018c00640 1 -- 172.21.15.141:0/2742778284 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fe00c058bb0 con 0x7fdffc0047c0 2024-01-16T15:11:49.885 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:11:49.885 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3880199275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":3880199275},{"type":"v1","addr":"172.21.15.141:6835","nonce":3880199275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2564906195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2564906195},{"type":"v1","addr":"172.21.15.141:6834","nonce":2564906195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6466,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/924822672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":924822672},{"type":"v1","addr":"172.21.15.188:6837","nonce":924822672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:11:43.384065+0000","modified":"2024-01-16T15:11:46.473187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"d","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6836/1516772217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1516772217},{"type":"v1","addr":"172.21.15.188:6836","nonce":1516772217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:11:49.885 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 64 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.911+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdff406c1f0 msgr2=0x7fdff406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.911+0000 7fe021600640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdff406c1f0 0x7fdff406e6b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe004034000 tx=0x7fe004009030 comp rx=0 tx=0).stop 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.911+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 msgr2=0x7fe01c15a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.911+0000 7fe021600640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fe01c15a8f0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe00c010790 tx=0x7fe00c010c50 comp rx=0 tx=0).stop 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 shutdown_connections 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdff406c1f0 0x7fdff406e6b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe01c0cb4c0 0x7fe01c13b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe01c0caf70 0x7fe01c13ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 --2- 172.21.15.141:0/2742778284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdffc0047c0 0x7fe01c15a8f0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:49.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 >> 172.21.15.141:0/2742778284 conn(0x7fe01c06a800 msgr2=0x7fe01c0cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:49.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 shutdown_connections 2024-01-16T15:11:49.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:49.912+0000 7fe021600640 1 -- 172.21.15.141:0/2742778284 wait complete. 2024-01-16T15:11:50.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:11:43.384065+0000', 'modified': '2024-01-16T15:11:46.473187+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8640': {'gid': 8640, 'name': 'd', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.188:6836/1516772217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 1516772217}, {'type': 'v1', 'addr': '172.21.15.188:6836', 'nonce': 1516772217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:11:50.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:11:50.310 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-16T15:11:50.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.549+0000 7f9178800640 1 Processor -- start 2024-01-16T15:11:50.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.549+0000 7f9178800640 1 -- start start 2024-01-16T15:11:50.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.551+0000 7f9178800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 0x7f917015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.551+0000 7f9178800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f91700b3ae0 0x7f917006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.551+0000 7f9178800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9170070c40 con 0x7f917006d120 2024-01-16T15:11:50.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.551+0000 7f9178800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9170071450 con 0x7f91700b3ae0 2024-01-16T15:11:50.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.551+0000 7f9178800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917006c470 con 0x7f91700ae5a0 2024-01-16T15:11:50.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.551+0000 7f9176800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f917006d120 0x7f91700cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36680/0 (socket says 172.21.15.141:36680) 2024-01-16T15:11:50.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.551+0000 7f9176800640 1 -- 172.21.15.141:0/28566875 learned_addr learned my addr 172.21.15.141:0/28566875 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:50.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9175e00640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 0x7f917015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9175400640 1 --2- 172.21.15.141:0/28566875 >> v2:172.21.15.188:3300/0 conn(0x7f91700b3ae0 0x7f917006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023424491 0 0) 0x7f9170070c40 con 0x7f917006d120 2024-01-16T15:11:50.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9150003520 con 0x7f917006d120 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711886578 0 0) 0x7f9150003520 con 0x7f917006d120 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 msgr2=0x7f917015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 0x7f917015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 >> v2:172.21.15.188:3300/0 conn(0x7f91700b3ae0 msgr2=0x7f917006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 --2- 172.21.15.141:0/28566875 >> v2:172.21.15.188:3300/0 conn(0x7f91700b3ae0 0x7f917006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917006c750 con 0x7f917006d120 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f91680019f0 con 0x7f917006d120 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 >> v1:172.21.15.141:6789/0 conn(0x7f917006d120 legacy=0x7f91700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150003970 0x7f9150003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 0x7f91500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9150004d00 0x7f9150005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9170071450 con 0x7f9150004d00 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917006c470 con 0x7f9150003970 2024-01-16T15:11:50.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.552+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915000b980 con 0x7f91700ae5a0 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9176800640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9150004d00 0x7f9150005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9175400640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 0x7f91500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9175e00640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150003970 0x7f9150003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9176800640 1 -- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 msgr2=0x7f91500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9176800640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91700ae5a0 0x7f91500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9176800640 1 -- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150003970 msgr2=0x7f9150003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9176800640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150003970 0x7f9150003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9176800640 1 -- 172.21.15.141:0/28566875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150003520 con 0x7f9150004d00 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.553+0000 7f9175e00640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150003970 0x7f9150003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.554+0000 7f9176800640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9150004d00 0x7f9150005180 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f916800a870 tx=0x7f91500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92023621565ab57 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.554+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168034630 con 0x7f9150004d00 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.554+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916802f070 con 0x7f9150004d00 2024-01-16T15:11:50.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.554+0000 7f9174a00640 1 -- 172.21.15.141:0/28566875 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680349f0 con 0x7f9150004d00 2024-01-16T15:11:50.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.555+0000 7f9178800640 1 -- 172.21.15.141:0/28566875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9150004d00 msgr2=0x7f9150005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.555+0000 7f9178800640 1 --2- 172.21.15.141:0/28566875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9150004d00 0x7f9150005180 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f916800a870 tx=0x7f91500057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.555+0000 7f9176800640 1 -- 172.21.15.141:0/28566875 reap_dead start 2024-01-16T15:11:50.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.555+0000 7f9178800640 1 -- 172.21.15.141:0/28566875 shutdown_connections 2024-01-16T15:11:50.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.555+0000 7f9178800640 1 -- 172.21.15.141:0/28566875 >> 172.21.15.141:0/28566875 conn(0x7f917006a800 msgr2=0x7f917006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:50.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.555+0000 7f9178800640 1 -- 172.21.15.141:0/28566875 shutdown_connections 2024-01-16T15:11:50.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.555+0000 7f9178800640 1 -- 172.21.15.141:0/28566875 wait complete. 2024-01-16T15:11:50.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.557+0000 7f9178800640 1 Processor -- start 2024-01-16T15:11:50.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.557+0000 7f9178800640 1 -- start start 2024-01-16T15:11:50.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9178800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150004d00 0x7f9170144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9178800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91700cb8a0 0x7f91700cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9178800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91701473b0 0x7f9170147830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9178800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701f6ae0 con 0x7f91700cb8a0 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9178800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9170144e80 con 0x7f9150004d00 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9178800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91700cc260 con 0x7f91701473b0 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9175400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91700cb8a0 0x7f91700cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9175400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91700cb8a0 0x7f91700cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39056/0 (socket says 172.21.15.141:39056) 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9175400640 1 -- 172.21.15.141:0/4269266944 learned_addr learned my addr 172.21.15.141:0/4269266944 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9175400640 1 -- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91701473b0 msgr2=0x7f9170147830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9175400640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91701473b0 0x7f9170147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.558+0000 7f9175400640 1 -- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150004d00 msgr2=0x7f9170144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f9175400640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150004d00 0x7f9170144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f9175400640 1 -- 172.21.15.141:0/4269266944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91701480e0 con 0x7f91700cb8a0 2024-01-16T15:11:50.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f9175400640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91700cb8a0 0x7f91700cbd20 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f91580116d0 tx=0x7f9158011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f916e000640 1 -- 172.21.15.141:0/4269266944 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915800f940 con 0x7f91700cb8a0 2024-01-16T15:11:50.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f916e000640 1 -- 172.21.15.141:0/4269266944 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915800faa0 con 0x7f91700cb8a0 2024-01-16T15:11:50.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91701f6f00 con 0x7f91700cb8a0 2024-01-16T15:11:50.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f916e000640 1 -- 172.21.15.141:0/4269266944 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915800fe10 con 0x7f91700cb8a0 2024-01-16T15:11:50.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.559+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91701f6420 con 0x7f91700cb8a0 2024-01-16T15:11:50.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.561+0000 7f916e000640 1 -- 172.21.15.141:0/4269266944 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f9158015460 con 0x7f91700cb8a0 2024-01-16T15:11:50.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.562+0000 7f916e000640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f914806c220 0x7f914806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.562+0000 7f916e000640 1 -- 172.21.15.141:0/4269266944 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91580891f0 con 0x7f91700cb8a0 2024-01-16T15:11:50.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.563+0000 7f9175e00640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f914806c220 0x7f914806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.563+0000 7f917b45f740 1 Processor -- start 2024-01-16T15:11:50.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.563+0000 7f917b45f740 1 -- start start 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f914806c220 0x7f914806e6e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9160006fd0 tx=0x7f9160009350 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f917b45f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x5646584cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x564658512560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f917b45f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56465842d880 con 0x5646584ceea0 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56465843cae0 con 0x5646584ce5e0 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x56465843a1d0 con 0x5646584c9ba0 2024-01-16T15:11:50.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x5646584cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x5646584cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39070/0 (socket says 172.21.15.141:39070) 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 -- 172.21.15.141:0/3321667843 learned_addr learned my addr 172.21.15.141:0/3321667843 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 -- 172.21.15.141:0/3321667843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 msgr2=0x5646584ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 --2- 172.21.15.141:0/3321667843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 -- 172.21.15.141:0/3321667843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 msgr2=0x564658512560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 --2- 172.21.15.141:0/3321667843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x564658512560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.564+0000 7f9175e00640 1 -- 172.21.15.141:0/3321667843 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646583fd860 con 0x5646584ceea0 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.565+0000 7f9175e00640 1 --2- 172.21.15.141:0/3321667843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x5646584cf280 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f916000e920 tx=0x7f9160011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5ad1cdff9fdffe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.565+0000 7f9166a00640 1 -- 172.21.15.141:0/3321667843 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160015640 con 0x5646584ceea0 2024-01-16T15:11:50.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.565+0000 7f9166a00640 1 -- 172.21.15.141:0/3321667843 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91600157a0 con 0x5646584ceea0 2024-01-16T15:11:50.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.565+0000 7f9166a00640 1 -- 172.21.15.141:0/3321667843 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160015b30 con 0x5646584ceea0 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 -- 172.21.15.141:0/3321667843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 msgr2=0x5646584cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 --2- 172.21.15.141:0/3321667843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x5646584cf280 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f916000e920 tx=0x7f9160011000 comp rx=0 tx=0).stop 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 -- 172.21.15.141:0/3321667843 shutdown_connections 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 --2- 172.21.15.141:0/3321667843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x564658512560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 --2- 172.21.15.141:0/3321667843 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 --2- 172.21.15.141:0/3321667843 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x5646584cf280 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 -- 172.21.15.141:0/3321667843 >> 172.21.15.141:0/3321667843 conn(0x564658490c70 msgr2=0x564658514990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 -- 172.21.15.141:0/3321667843 shutdown_connections 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 -- 172.21.15.141:0/3321667843 wait complete. 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.566+0000 7f917b45f740 1 -- start start 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.567+0000 7f917b45f740 1 Processor -- start 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f917b45f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x564658519340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f917b45f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x564658435ac0 con 0x5646584ceea0 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646583d22f0 con 0x5646584ce5e0 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646583efc70 con 0x5646584c9ba0 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f917b45f740 10 client.0 _ll_register_callbacks cb 0x5646583f43b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x564658519340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x564658519340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39086/0 (socket says 172.21.15.141:39086) 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 -- 172.21.15.141:0/860746275 learned_addr learned my addr 172.21.15.141:0/860746275 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9175400640 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9175e00640 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 -- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 msgr2=0x5646584f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584f81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 -- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 msgr2=0x5646584f8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.568+0000 7f9176800640 1 -- 172.21.15.141:0/860746275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916000e600 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f9176800640 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x564658519340 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f916800a870 tx=0x7f9168002180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8c0ac920019f40 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f915f400640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f915f400640 1 -- 172.21.15.141:0/860746275 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168034430 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f915f400640 1 -- 172.21.15.141:0/860746275 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9168034590 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56465842eaa0 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f915f400640 1 -- 172.21.15.141:0/860746275 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9168034750 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f917b45f740 10 client.8763 fetch_fsmap learned FSMap version 64 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x564658434140 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f915f400640 1 -- 172.21.15.141:0/860746275 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56465842eaa0 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.569+0000 7f915f400640 1 -- 172.21.15.141:0/860746275 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f91680349d0 con 0x5646584ceea0 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.570+0000 7f917b45f740 10 client.8763 fetch_fsmap finished waiting for FSMap version 64 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.570+0000 7f917b45f740 10 client.8763 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.570+0000 7f917b45f740 20 client.8763 populate_metadata read hostname 'smithi141' 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.570+0000 7f917b45f740 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x5646584f4550 0x564658531c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.570+0000 7f917b45f740 4 client.8763 mds_command: new command op to 8640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:11:50.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.570+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 --> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56465842e7c0 con 0x5646584f4550 2024-01-16T15:11:50.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.570+0000 7f9175e00640 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x5646584f4550 0x564658531c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.571+0000 7f9175e00640 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x5646584f4550 0x564658531c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace8373fcf68770d server_cookie=e420822c94062280 in_seq=0 out_seq=0 2024-01-16T15:11:50.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.571+0000 7f915f400640 10 client.8763 ms_handle_connect on v2:172.21.15.188:6834/1516772217 2024-01-16T15:11:50.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f915f400640 1 -- 172.21.15.141:0/860746275 <== mds.0 v2:172.21.15.188:6834/1516772217 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56465842e7c0 con 0x5646584f4550 2024-01-16T15:11:50.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f915f400640 10 client.8763 handle_command_reply: tid=0 2024-01-16T15:11:50.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 client.8763 shutdown 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 msgr2=0x564658519340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x564658519340 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f916800a870 tx=0x7f9168002180 comp rx=0 tx=0).stop 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 shutdown_connections 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x5646584f4550 0x564658531c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ceea0 0x564658519340 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584f8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 --2- 172.21.15.141:0/860746275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 >> 172.21.15.141:0/860746275 conn(0x564658490c70 msgr2=0x564658517ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f915f400640 0 client.8763 ms_handle_reset on v2:172.21.15.188:6834/1516772217 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 shutdown_connections 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 1 -- 172.21.15.141:0/860746275 wait complete. 2024-01-16T15:11:50.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.574+0000 7f917b45f740 20 client.8763 trim_cache size 0 max 16384 2024-01-16T15:11:50.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.591+0000 7f917b45f740 1 Processor -- start 2024-01-16T15:11:50.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.591+0000 7f917b45f740 1 -- start start 2024-01-16T15:11:50.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f917b45f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584c9ba0 0x5646584f9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584ceea0 0x5646584c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f917b45f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x564658198f20 con 0x5646584c9ba0 2024-01-16T15:11:50.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56465843e2f0 con 0x5646584ce5e0 2024-01-16T15:11:50.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x564658536a00 con 0x5646584ceea0 2024-01-16T15:11:50.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f9175e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584c9ba0 0x5646584f9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f9175e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584c9ba0 0x5646584f9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39098/0 (socket says 172.21.15.141:39098) 2024-01-16T15:11:50.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.592+0000 7f9175e00640 1 -- 172.21.15.141:0/4018103326 learned_addr learned my addr 172.21.15.141:0/4018103326 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175400640 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9176800640 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584ceea0 0x5646584c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175e00640 1 -- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584ceea0 msgr2=0x5646584c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175e00640 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584ceea0 0x5646584c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175e00640 1 -- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 msgr2=0x5646584c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175e00640 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175e00640 1 -- 172.21.15.141:0/4018103326 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646583d1b80 con 0x5646584c9ba0 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175400640 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9176800640 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584ceea0 0x5646584c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9175e00640 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584c9ba0 0x5646584f9790 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f9160023040 tx=0x7f916001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ef239ff65ced77 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9167400640 1 -- 172.21.15.141:0/4018103326 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160030ea0 con 0x5646584c9ba0 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9167400640 1 -- 172.21.15.141:0/4018103326 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9160015b10 con 0x5646584c9ba0 2024-01-16T15:11:50.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.593+0000 7f9167400640 1 -- 172.21.15.141:0/4018103326 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160015ea0 con 0x5646584c9ba0 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 -- 172.21.15.141:0/4018103326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584c9ba0 msgr2=0x5646584f9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584c9ba0 0x5646584f9790 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f9160023040 tx=0x7f916001d000 comp rx=0 tx=0).stop 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 -- 172.21.15.141:0/4018103326 shutdown_connections 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584ceea0 0x5646584c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ce5e0 0x5646584c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 --2- 172.21.15.141:0/4018103326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584c9ba0 0x5646584f9790 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 -- 172.21.15.141:0/4018103326 >> 172.21.15.141:0/4018103326 conn(0x564658490c70 msgr2=0x5646584c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 -- 172.21.15.141:0/4018103326 shutdown_connections 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.594+0000 7f917b45f740 1 -- 172.21.15.141:0/4018103326 wait complete. 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.595+0000 7f917b45f740 1 -- start start 2024-01-16T15:11:50.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.595+0000 7f917b45f740 1 Processor -- start 2024-01-16T15:11:50.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f917b45f740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ce5e0 0x56465857e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f917b45f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ceea0 0x56465857ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f917b45f740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646584f8300 con 0x5646584ce5e0 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56465851aa20 con 0x5646584ceea0 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f917b45f740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646584c94c0 con 0x5646584c9ba0 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f917b45f740 10 client.0 _ll_register_callbacks cb 0x7f91500036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ce5e0 0x56465857e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ce5e0 0x56465857e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39110/0 (socket says 172.21.15.141:39110) 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175400640 1 -- 172.21.15.141:0/4173647554 learned_addr learned my addr 172.21.15.141:0/4173647554 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:50.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9176800640 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ceea0 0x56465857ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175e00640 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175400640 1 -- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 msgr2=0x5646584c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175400640 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175400640 1 -- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ceea0 msgr2=0x56465857ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.596+0000 7f9175400640 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ceea0 0x56465857ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f9175400640 1 -- 172.21.15.141:0/4173647554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646583fd860 con 0x5646584ce5e0 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f9175400640 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ce5e0 0x56465857e540 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f915801c000 tx=0x7f9158059360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9259905222c00f5d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f915f400640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f915f400640 1 -- 172.21.15.141:0/4173647554 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915805d5d0 con 0x5646584ce5e0 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f915f400640 1 -- 172.21.15.141:0/4173647554 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9158061780 con 0x5646584ce5e0 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91600157a0 con 0x5646584ce5e0 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f915f400640 1 -- 172.21.15.141:0/4173647554 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9158061500 con 0x5646584ce5e0 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f917b45f740 10 client.8769 fetch_fsmap learned FSMap version 64 2024-01-16T15:11:50.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x5646583f7170 con 0x5646584ce5e0 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f915f400640 1 -- 172.21.15.141:0/4173647554 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91600157a0 con 0x5646584ce5e0 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f915f400640 1 -- 172.21.15.141:0/4173647554 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f915801b020 con 0x5646584ce5e0 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f917b45f740 10 client.8769 fetch_fsmap finished waiting for FSMap version 64 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f917b45f740 10 client.8769 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.597+0000 7f917b45f740 20 client.8769 populate_metadata read hostname 'smithi141' 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.598+0000 7f917b45f740 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x56465852e0f0 0x56465852e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.598+0000 7f917b45f740 4 client.8769 mds_command: new command op to 8640 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.598+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 --> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5646581366c0 con 0x56465852e0f0 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.598+0000 7f9175e00640 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x56465852e0f0 0x56465852e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.599+0000 7f9175e00640 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x56465852e0f0 0x56465852e4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d89b92b63942c7 server_cookie=855b04201f6de595 in_seq=0 out_seq=0 2024-01-16T15:11:50.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.599+0000 7f915f400640 10 client.8769 ms_handle_connect on v2:172.21.15.188:6834/1516772217 2024-01-16T15:11:50.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f915f400640 1 -- 172.21.15.141:0/4173647554 <== mds.0 v2:172.21.15.188:6834/1516772217 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5646581366c0 con 0x56465852e0f0 2024-01-16T15:11:50.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f915f400640 10 client.8769 handle_command_reply: tid=0 2024-01-16T15:11:50.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 client.8769 shutdown 2024-01-16T15:11:50.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ce5e0 msgr2=0x56465857e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ce5e0 0x56465857e540 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f915801c000 tx=0x7f9158059360 comp rx=0 tx=0).stop 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 shutdown_connections 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:6834/1516772217,v1:172.21.15.188:6836/1516772217] conn(0x56465852e0f0 0x56465852e4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5646584ceea0 0x56465857ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646584ce5e0 0x56465857e540 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 --2- 172.21.15.141:0/4173647554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5646584c9ba0 0x5646584c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 >> 172.21.15.141:0/4173647554 conn(0x564658490c70 msgr2=0x5646584cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f915f400640 0 client.8769 ms_handle_reset on v2:172.21.15.188:6834/1516772217 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 shutdown_connections 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 1 -- 172.21.15.141:0/4173647554 wait complete. 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.600+0000 7f917b45f740 20 client.8769 trim_cache size 0 max 16384 2024-01-16T15:11:50.582 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 8640, 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 6.6297307590000001, 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 64, 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 102, 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 102, 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 12.066851939999999 2024-01-16T15:11:50.583 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:11:50.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f914806c220 msgr2=0x7f914806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f914806c220 0x7f914806e6e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9160006fd0 tx=0x7f9160009350 comp rx=0 tx=0).stop 2024-01-16T15:11:50.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91700cb8a0 msgr2=0x7f91700cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91700cb8a0 0x7f91700cbd20 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f91580116d0 tx=0x7f9158011b90 comp rx=0 tx=0).stop 2024-01-16T15:11:50.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 shutdown_connections 2024-01-16T15:11:50.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f914806c220 0x7f914806e6e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91701473b0 0x7f9170147830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91700cb8a0 0x7f91700cbd20 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 --2- 172.21.15.141:0/4269266944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9150004d00 0x7f9170144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 >> 172.21.15.141:0/4269266944 conn(0x7f917006a800 msgr2=0x7f91700cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:50.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 shutdown_connections 2024-01-16T15:11:50.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.601+0000 7f9178800640 1 -- 172.21.15.141:0/4269266944 wait complete. 2024-01-16T15:11:50.593 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:11:50.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.804+0000 7feabcc00640 1 Processor -- start 2024-01-16T15:11:50.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.804+0000 7feabcc00640 1 -- start start 2024-01-16T15:11:50.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.805+0000 7feabcc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7feab813cfa0 0x7feab815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.805+0000 7feabcc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab80cd530 0x7feab813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.805+0000 7feabcc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80cdec0 con 0x7feab80cd9b0 2024-01-16T15:11:50.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.805+0000 7feabcc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab80ce1a0 con 0x7feab813cfa0 2024-01-16T15:11:50.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.805+0000 7feabcc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab80ce480 con 0x7feab80cd530 2024-01-16T15:11:50.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.806+0000 7feab7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7feab80cd9b0 0x7feab815d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:36688/0 (socket says 172.21.15.141:36688) 2024-01-16T15:11:50.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.806+0000 7feab7e00640 1 -- 172.21.15.141:0/293889973 learned_addr learned my addr 172.21.15.141:0/293889973 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.806+0000 7feaae800640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab80cd530 0x7feab813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.806+0000 7feab7400640 1 --2- 172.21.15.141:0/293889973 >> v2:172.21.15.188:3300/0 conn(0x7feab813cfa0 0x7feab815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.806+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94029336 0 0) 0x7feab80cdec0 con 0x7feab80cd9b0 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.806+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea98005730 con 0x7feab80cd9b0 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950359365 0 0) 0x7fea98005730 con 0x7feab80cd9b0 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab80cd530 msgr2=0x7feab813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab80cd530 0x7feab813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 >> v2:172.21.15.188:3300/0 conn(0x7feab813cfa0 msgr2=0x7feab815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 --2- 172.21.15.141:0/293889973 >> v2:172.21.15.188:3300/0 conn(0x7feab813cfa0 0x7feab815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80ce760 con 0x7feab80cd9b0 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7feab00019f0 con 0x7feab80cd9b0 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 >> v1:172.21.15.141:6789/0 conn(0x7feab80cd9b0 legacy=0x7feab815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7fea98005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea980069d0 0x7fea98006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab80cd530 0x7fea98007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80ce1a0 con 0x7fea98005b80 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80ce480 con 0x7fea980069d0 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9800dbc0 con 0x7feab80cd530 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab7400640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7fea98005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab7400640 1 -- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab80cd530 msgr2=0x7fea98007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab7400640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab80cd530 0x7fea98007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab7400640 1 -- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea980069d0 msgr2=0x7fea98006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab7400640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea980069d0 0x7fea98006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.785 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.807+0000 7feab7400640 1 -- 172.21.15.141:0/293889973 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98005730 con 0x7fea98005b80 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.808+0000 7feab7400640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7fea98005fa0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009ac0 tx=0x7fea98007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c71df45de2db7d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa802c6d0 con 0x7fea98005b80 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa802c830 con 0x7fea98005b80 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feab6a00640 1 -- 172.21.15.141:0/293889973 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa802cbc0 con 0x7fea98005b80 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feabcc00640 1 -- 172.21.15.141:0/293889973 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 msgr2=0x7fea98005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feabcc00640 1 --2- 172.21.15.141:0/293889973 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7fea98005fa0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009ac0 tx=0x7fea98007a00 comp rx=0 tx=0).stop 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feab7e00640 1 -- 172.21.15.141:0/293889973 reap_dead start 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feabcc00640 1 -- 172.21.15.141:0/293889973 shutdown_connections 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feabcc00640 1 -- 172.21.15.141:0/293889973 >> 172.21.15.141:0/293889973 conn(0x7feab806a800 msgr2=0x7feab806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feabcc00640 1 -- 172.21.15.141:0/293889973 shutdown_connections 2024-01-16T15:11:50.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.809+0000 7feabcc00640 1 -- 172.21.15.141:0/293889973 wait complete. 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.811+0000 7feabcc00640 1 Processor -- start 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.811+0000 7feabcc00640 1 -- start start 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.812+0000 7feabcc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7feab80c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.812+0000 7feabcc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab80c3d70 0x7feab81ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.812+0000 7feabcc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab806cf00 0x7feab806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.812+0000 7feabcc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab806d8a0 con 0x7fea98005b80 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.812+0000 7feabcc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80c4550 con 0x7feab80c3d70 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.812+0000 7feabcc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab80c4830 con 0x7feab806cf00 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7feab80c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7feab80c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39132/0 (socket says 172.21.15.141:39132) 2024-01-16T15:11:50.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 -- 172.21.15.141:0/1201744058 learned_addr learned my addr 172.21.15.141:0/1201744058 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feaae800640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab80c3d70 0x7feab81ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 -- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab806cf00 msgr2=0x7feab806d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7e00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab806cf00 0x7feab806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab806cf00 0x7feab806d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 -- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab80c3d70 msgr2=0x7feab81ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab80c3d70 0x7feab81ee8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 -- 172.21.15.141:0/1201744058 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80c4b10 con 0x7fea98005b80 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feaae800640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab80c3d70 0x7feab81ee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.813+0000 7feab7400640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7feab80c68f0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7feaa802c2d0 tx=0x7feaa8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.814+0000 7feaafe00640 1 -- 172.21.15.141:0/1201744058 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8003810 con 0x7fea98005b80 2024-01-16T15:11:50.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.814+0000 7feaafe00640 1 -- 172.21.15.141:0/1201744058 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8003970 con 0x7fea98005b80 2024-01-16T15:11:50.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.814+0000 7feaafe00640 1 -- 172.21.15.141:0/1201744058 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8003ce0 con 0x7fea98005b80 2024-01-16T15:11:50.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.814+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab80c4df0 con 0x7fea98005b80 2024-01-16T15:11:50.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.814+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab80c41f0 con 0x7fea98005b80 2024-01-16T15:11:50.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.815+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab8067450 con 0x7fea98005b80 2024-01-16T15:11:50.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.816+0000 7feaafe00640 1 -- 172.21.15.141:0/1201744058 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7feaa8040660 con 0x7fea98005b80 2024-01-16T15:11:50.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.817+0000 7feaafe00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea9006c220 0x7fea9006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:50.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.817+0000 7feaafe00640 1 -- 172.21.15.141:0/1201744058 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feaa80aa690 con 0x7fea98005b80 2024-01-16T15:11:50.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.817+0000 7feaae800640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea9006c220 0x7fea9006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:50.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.819+0000 7feaae800640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea9006c220 0x7fea9006e6e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7feab80c5750 tx=0x7feaa0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:50.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:50.821+0000 7feaafe00640 1 -- 172.21.15.141:0/1201744058 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feaa8077070 con 0x7fea98005b80 2024-01-16T15:11:51.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.056+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feab80cbff0 con 0x7fea98005b80 2024-01-16T15:11:51.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.057+0000 7feaafe00640 1 -- 172.21.15.141:0/1201744058 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7feaa80044e0 con 0x7fea98005b80 2024-01-16T15:11:51.033 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:11:51.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea9006c220 msgr2=0x7fea9006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:51.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea9006c220 0x7fea9006e6e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7feab80c5750 tx=0x7feaa0008040 comp rx=0 tx=0).stop 2024-01-16T15:11:51.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 msgr2=0x7feab80c68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:51.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7feab80c68f0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7feaa802c2d0 tx=0x7feaa8004730 comp rx=0 tx=0).stop 2024-01-16T15:11:51.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 shutdown_connections 2024-01-16T15:11:51.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fea9006c220 0x7fea9006e6e0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feab806cf00 0x7feab806d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab80c3d70 0x7feab81ee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 --2- 172.21.15.141:0/1201744058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fea98005b80 0x7feab80c68f0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 >> 172.21.15.141:0/1201744058 conn(0x7feab806a800 msgr2=0x7feab8158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:51.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 shutdown_connections 2024-01-16T15:11:51.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:51.061+0000 7feabcc00640 1 -- 172.21.15.141:0/1201744058 wait complete. 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:11:51.054 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:11:51.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:51.055 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T15:11:51.099 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:11:51.099 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:11:51.099 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:11:51.099 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:51.099 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T15:11:51.099 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 41752sec preferred_lft 41752sec 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:11:51.100 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::882:e9ff:fe92:c54c/64 scope link 2024-01-16T15:11:51.101 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:51.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:51.102 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:11:51.233 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T15:11:51.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:51.237 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T15:11:51.285 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T15:11:51.285 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:11:51.286 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:11:51.286 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:11:51.286 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:11:51.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:51.287 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T15:11:51.287 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T15:11:51.287 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:11:51.287 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:11:51.287 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:11:51.287 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T15:11:51.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:51.469 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T15:11:51.518 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:11:51.518 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:11:51.519 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:11:51.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:51.519 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T15:11:51.611 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:11:51.611 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:51.611 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:51.660 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:11:51.660 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:51.660 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:51.709 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:11:51.803 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:51.861 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:11:51.861 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:11:51.861 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:11:51.861 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:11:51.861 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:11:51.861 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:11:51.861 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:51.862 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:51.863 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:51.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:51.864 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:11:51.916 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:11:51.916 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:11:51.921 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:11:52.004 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:52.048 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:11:52.048 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:11:52.048 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:11:52.048 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:11:52.048 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:52.049 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:52.050 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:11:52.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:52.051 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:11:52.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T15:11:52.139+0000 7f78384cb480 -1 init, newargv = 0x55d2005199f0 newargc=15 2024-01-16T15:11:52.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[72831]: starting ceph client 2024-01-16T15:11:52.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[72831]: starting fuse 2024-01-16T15:11:52.137 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T15:11:52.138 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:11:52.138 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T15:11:52.139 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T15:11:52.332 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.72831.asok 2024-01-16T15:11:52.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:11:52.350 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72831.asok 2024-01-16T15:11:52.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:52.350 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72831.asok status 2024-01-16T15:11:52.584 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "72831", 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 8784, 2024-01-16T15:11:52.585 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 8784 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 3165930079 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 3165930079 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.8784 192.168.0.1:0/3165930079", 2024-01-16T15:11:52.586 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/3165930079", 2024-01-16T15:11:52.587 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T15:11:52.587 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 64, 2024-01-16T15:11:52.587 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 102, 2024-01-16T15:11:52.587 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:11:52.587 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T15:11:52.587 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T15:11:52.587 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:11:52.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:52.600 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T15:11:52.600 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:52.655 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T15:11:52.655 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:52.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:52.655 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:11:52.751 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:11:52.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.996+0000 7f2837000640 1 Processor -- start 2024-01-16T15:11:52.972 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.996+0000 7f2837000640 1 -- start start 2024-01-16T15:11:52.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.997+0000 7f2837000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f283013e360 0x7f28300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:52.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.997+0000 7f2837000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f283015f9c0 0x7f28300cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:52.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.997+0000 7f2837000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283015c5f0 con 0x7f283013df80 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.997+0000 7f2837000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283006c470 con 0x7f283013e360 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.997+0000 7f2837000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283006c750 con 0x7f283015f9c0 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2827200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f283013e360 0x7f28300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2827200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f283013e360 0x7f28300cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39136/0 (socket says 172.21.15.141:39136) 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2827200640 1 -- 172.21.15.141:0/2386411764 learned_addr learned my addr 172.21.15.141:0/2386411764 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2835000640 1 --2- 172.21.15.141:0/2386411764 >> v2:172.21.15.188:3300/0 conn(0x7f283015f9c0 0x7f28300cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2827200640 1 -- 172.21.15.141:0/2386411764 >> v2:172.21.15.188:3300/0 conn(0x7f283015f9c0 msgr2=0x7f28300cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2827200640 1 --2- 172.21.15.141:0/2386411764 >> v2:172.21.15.188:3300/0 conn(0x7f283015f9c0 0x7f28300cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2827200640 1 -- 172.21.15.141:0/2386411764 >> v1:172.21.15.188:6789/0 conn(0x7f283013df80 legacy=0x7f283015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:52.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.998+0000 7f2827200640 1 -- 172.21.15.141:0/2386411764 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283006ca30 con 0x7f283013e360 2024-01-16T15:11:52.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:52.999+0000 7f2827200640 1 --2- 172.21.15.141:0/2386411764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013e360 0x7f28300cf290 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f283006ded0 tx=0x7f281802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2896a2fcd361e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f282f400640 1 -- 172.21.15.141:0/2386411764 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818031640 con 0x7f283013e360 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f282f400640 1 -- 172.21.15.141:0/2386411764 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28180317a0 con 0x7f283013e360 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f282f400640 1 -- 172.21.15.141:0/2386411764 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818031b30 con 0x7f283013e360 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 -- 172.21.15.141:0/2386411764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013e360 msgr2=0x7f28300cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 --2- 172.21.15.141:0/2386411764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013e360 0x7f28300cf290 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f283006ded0 tx=0x7f281802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 -- 172.21.15.141:0/2386411764 shutdown_connections 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 --2- 172.21.15.141:0/2386411764 >> v2:172.21.15.188:3300/0 conn(0x7f283015f9c0 0x7f28300cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 --2- 172.21.15.141:0/2386411764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013e360 0x7f28300cf290 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 -- 172.21.15.141:0/2386411764 >> 172.21.15.141:0/2386411764 conn(0x7f283006a800 msgr2=0x7f283006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 -- 172.21.15.141:0/2386411764 shutdown_connections 2024-01-16T15:11:52.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.000+0000 7f2837000640 1 -- 172.21.15.141:0/2386411764 wait complete. 2024-01-16T15:11:52.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.002+0000 7f2837000640 1 Processor -- start 2024-01-16T15:11:52.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.002+0000 7f2837000640 1 -- start start 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.003+0000 7f2837000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013df80 0x7f283013d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.003+0000 7f2837000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f283013e360 0x7f2830152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.003+0000 7f2837000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f283015f9c0 0x7f28300cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.003+0000 7f2837000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28300a8910 con 0x7f283013df80 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.003+0000 7f2837000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28300a8bf0 con 0x7f283013e360 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.003+0000 7f2837000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28300a8ed0 con 0x7f283015f9c0 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.003+0000 7f282fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013df80 0x7f283013d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013df80 0x7f283013d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39144/0 (socket says 172.21.15.141:39144) 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 -- 172.21.15.141:0/1107197587 learned_addr learned my addr 172.21.15.141:0/1107197587 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f2827200640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f283013e360 0x7f2830152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:52.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 -- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f283015f9c0 msgr2=0x7f28300cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f2835000640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f283015f9c0 0x7f28300cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f283015f9c0 0x7f28300cdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 -- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f283013e360 msgr2=0x7f2830152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f283013e360 0x7f2830152d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 -- 172.21.15.141:0/1107197587 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28180094d0 con 0x7f283013df80 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f2827200640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f283013e360 0x7f2830152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.004+0000 7f282fe00640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013df80 0x7f283013d630 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f282000b3f0 tx=0x7f282000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.005+0000 7f282cc00640 1 -- 172.21.15.141:0/1107197587 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820011830 con 0x7f283013df80 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.005+0000 7f282cc00640 1 -- 172.21.15.141:0/1107197587 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820011990 con 0x7f283013df80 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.005+0000 7f282cc00640 1 -- 172.21.15.141:0/1107197587 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820011d40 con 0x7f283013df80 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.005+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2830150de0 con 0x7f283013df80 2024-01-16T15:11:52.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.005+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28300a84c0 con 0x7f283013df80 2024-01-16T15:11:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.006+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2830150fd0 con 0x7f283013df80 2024-01-16T15:11:52.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.007+0000 7f282cc00640 1 -- 172.21.15.141:0/1107197587 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f282001a400 con 0x7f283013df80 2024-01-16T15:11:52.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.008+0000 7f282cc00640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f280806c1f0 0x7f280806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:52.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.008+0000 7f282cc00640 1 -- 172.21.15.141:0/1107197587 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28200885e0 con 0x7f283013df80 2024-01-16T15:11:52.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.008+0000 7f2827200640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f280806c1f0 0x7f280806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:52.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.010+0000 7f2827200640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f280806c1f0 0x7f280806e6b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2818036000 tx=0x7f281803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:52.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.013+0000 7f282cc00640 1 -- 172.21.15.141:0/1107197587 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2820054de0 con 0x7f283013df80 2024-01-16T15:11:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.245+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f28300675d0 con 0x7f283013df80 2024-01-16T15:11:53.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.246+0000 7f282cc00640 1 -- 172.21.15.141:0/1107197587 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2820015070 con 0x7f283013df80 2024-01-16T15:11:53.222 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f280806c1f0 msgr2=0x7f280806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f280806c1f0 0x7f280806e6b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2818036000 tx=0x7f281803c040 comp rx=0 tx=0).stop 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013df80 msgr2=0x7f283013d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013df80 0x7f283013d630 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f282000b3f0 tx=0x7f282000b8b0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 shutdown_connections 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f280806c1f0 0x7f280806e6b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f283015f9c0 0x7f28300cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f283013e360 0x7f2830152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 --2- 172.21.15.141:0/1107197587 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f283013df80 0x7f283013d630 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 >> 172.21.15.141:0/1107197587 conn(0x7f283006a800 msgr2=0x7f283006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 shutdown_connections 2024-01-16T15:11:53.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:53.249+0000 7f2837000640 1 -- 172.21.15.141:0/1107197587 wait complete. 2024-01-16T15:11:53.244 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:11:53.244 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:11:53.244 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T15:11:53.244 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:11:53.244 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:53.245 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:11:53.245 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:11:53.245 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:11:53.245 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:11:53.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:53.245 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T15:11:53.304 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:11:53.304 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:11:53.304 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:11:53.304 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:53.305 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 41752sec preferred_lft 41752sec 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:11:53.306 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:53.307 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::a40a:23ff:fed8:ea96/64 scope link 2024-01-16T15:11:53.307 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:11:53.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:53.308 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:11:53.399 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T15:11:53.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:53.404 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T15:11:53.455 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T15:11:53.455 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:11:53.456 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:11:53.456 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:11:53.456 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:11:53.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:53.456 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T15:11:53.456 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T15:11:53.456 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:11:53.456 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:11:53.456 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:11:53.456 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T15:11:53.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:53.633 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T15:11:53.679 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:11:53.679 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:11:53.680 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:11:53.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:53.680 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T15:11:53.819 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:11:53.819 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:53.819 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:53.873 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:11:53.873 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:11:53.873 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:53.921 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:11:54.013 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:54.068 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:11:54.068 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:11:54.068 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:11:54.069 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:11:54.070 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:54.071 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:54.071 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:54.071 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:54.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:54.072 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:11:54.123 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:11:54.123 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:11:54.130 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:11:54.208 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:11:54.248 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:11:54.248 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:11:54.249 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:11:54.250 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:54.251 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:11:54.251 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:54.251 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:11:54.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:54.252 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:11:54.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T15:11:54.331+0000 7f5102c45480 -1 init, newargv = 0x563f58a5f790 newargc=15 2024-01-16T15:11:54.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[46661]: starting ceph client 2024-01-16T15:11:54.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[46661]: starting fuse 2024-01-16T15:11:54.334 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T15:11:54.334 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T15:11:54.335 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T15:11:54.336 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T15:11:54.536 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.46661.asok 2024-01-16T15:11:54.540 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T15:11:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:11:54.553 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.46661.asok 2024-01-16T15:11:54.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:54.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.46661.asok status 2024-01-16T15:11:54.788 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "46661", 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 5981, 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T15:11:54.789 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 5981 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 570312823 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 570312823 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.5981 192.168.0.1:0/570312823", 2024-01-16T15:11:54.790 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/570312823", 2024-01-16T15:11:54.791 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T15:11:54.791 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 64, 2024-01-16T15:11:54.791 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 102, 2024-01-16T15:11:54.791 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:11:54.791 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T15:11:54.791 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T15:11:54.791 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T15:11:54.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:54.803 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T15:11:54.803 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:54.859 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T15:11:54.859 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:54.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:54.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:11:54.956 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:11:55.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.192+0000 7f9f83000640 1 Processor -- start 2024-01-16T15:11:55.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.192+0000 7f9f83000640 1 -- start start 2024-01-16T15:11:55.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f83000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071f00 0x7f9f7c146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:55.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f83000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9f7c06c7c0 0x7f9f7c166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:55.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f83000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7c0b2e10 con 0x7f9f7c071410 2024-01-16T15:11:55.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f83000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7c06cec0 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f83000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c06d1a0 con 0x7f9f7c071f00 2024-01-16T15:11:55.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f82600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9f7c06c7c0 0x7f9f7c166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:55.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f82600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9f7c06c7c0 0x7f9f7c166cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51340/0 (socket says 172.21.15.141:51340) 2024-01-16T15:11:55.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f82600640 1 -- 172.21.15.141:0/307233007 learned_addr learned my addr 172.21.15.141:0/307233007 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:55.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.194+0000 7f9f81200640 1 --2- 172.21.15.141:0/307233007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071f00 0x7f9f7c146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:55.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.195+0000 7f9f82600640 1 -- 172.21.15.141:0/307233007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071f00 msgr2=0x7f9f7c146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:55.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.195+0000 7f9f82600640 1 --2- 172.21.15.141:0/307233007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071f00 0x7f9f7c146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:55.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.195+0000 7f9f82600640 1 -- 172.21.15.141:0/307233007 >> v1:172.21.15.188:6789/0 conn(0x7f9f7c071410 legacy=0x7f9f7c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:55.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.195+0000 7f9f82600640 1 -- 172.21.15.141:0/307233007 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c06d480 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.196+0000 7f9f82600640 1 --2- 172.21.15.141:0/307233007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c166cf0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009e50 tx=0x7f9f7402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a5792966d8ba71 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:55.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f7be00640 1 -- 172.21.15.141:0/307233007 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7402cb00 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f7be00640 1 -- 172.21.15.141:0/307233007 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7402cc60 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f7be00640 1 -- 172.21.15.141:0/307233007 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f740343f0 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 -- 172.21.15.141:0/307233007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 msgr2=0x7f9f7c166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:55.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 --2- 172.21.15.141:0/307233007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c166cf0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009e50 tx=0x7f9f7402bf60 comp rx=0 tx=0).stop 2024-01-16T15:11:55.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 -- 172.21.15.141:0/307233007 shutdown_connections 2024-01-16T15:11:55.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 --2- 172.21.15.141:0/307233007 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c166cf0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:55.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 --2- 172.21.15.141:0/307233007 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071f00 0x7f9f7c146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:55.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 -- 172.21.15.141:0/307233007 >> 172.21.15.141:0/307233007 conn(0x7f9f7c06a800 msgr2=0x7f9f7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:55.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 -- 172.21.15.141:0/307233007 shutdown_connections 2024-01-16T15:11:55.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.197+0000 7f9f83000640 1 -- 172.21.15.141:0/307233007 wait complete. 2024-01-16T15:11:55.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.199+0000 7f9f83000640 1 Processor -- start 2024-01-16T15:11:55.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.200+0000 7f9f83000640 1 -- start start 2024-01-16T15:11:55.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f83000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c12fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:55.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f83000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071410 0x7f9f7c15c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f83000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f7c071f00 0x7f9f7c15db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f83000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c130180 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f83000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c12d540 con 0x7f9f7c071f00 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f83000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c12d820 con 0x7f9f7c071410 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f81c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c12fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f81c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c12fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51354/0 (socket says 172.21.15.141:51354) 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.201+0000 7f9f81c00640 1 -- 172.21.15.141:0/1475601895 learned_addr learned my addr 172.21.15.141:0/1475601895 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f82600640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f7c071f00 0x7f9f7c15db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81200640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071410 0x7f9f7c15c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81c00640 1 -- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071410 msgr2=0x7f9f7c15c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81c00640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071410 0x7f9f7c15c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81c00640 1 -- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f7c071f00 msgr2=0x7f9f7c15db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81c00640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f7c071f00 0x7f9f7c15db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:55.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81c00640 1 -- 172.21.15.141:0/1475601895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f74009b30 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81200640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071410 0x7f9f7c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f82600640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f7c071f00 0x7f9f7c15db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f81c00640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c12fc40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c450 tx=0x7f9f6c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f79600640 1 -- 172.21.15.141:0/1475601895 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0052d0 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f79600640 1 -- 172.21.15.141:0/1475601895 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c005430 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f79600640 1 -- 172.21.15.141:0/1475601895 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c012870 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7c12db00 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.202+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7c12e0c0 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.203+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7c12e2e0 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.204+0000 7f9f79600640 1 -- 172.21.15.141:0/1475601895 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f9f6c0129d0 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.205+0000 7f9f79600640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f5406c1f0 0x7f9f5406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:55.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.205+0000 7f9f79600640 1 -- 172.21.15.141:0/1475601895 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f6c088610 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.206+0000 7f9f81200640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f5406c1f0 0x7f9f5406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:55.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.207+0000 7f9f81200640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f5406c1f0 0x7f9f5406e6b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9f64006fd0 tx=0x7f9f64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:55.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.209+0000 7f9f79600640 1 -- 172.21.15.141:0/1475601895 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f6c054ae0 con 0x7f9f7c06c7c0 2024-01-16T15:11:55.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:55.442+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9f7c067450 con 0x7f9f7c06c7c0 2024-01-16T15:11:56.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.038+0000 7f9f79600640 1 -- 172.21.15.141:0/1475601895 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v65) v1 ==== 136+0+0 (secure 0 0 0) 0x7f9f6c058100 con 0x7f9f7c06c7c0 2024-01-16T15:11:56.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.042+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f5406c1f0 msgr2=0x7f9f5406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f5406c1f0 0x7f9f5406e6b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9f64006fd0 tx=0x7f9f64008040 comp rx=0 tx=0).stop 2024-01-16T15:11:56.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 msgr2=0x7f9f7c12fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c12fc40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c450 tx=0x7f9f6c00c910 comp rx=0 tx=0).stop 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 shutdown_connections 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9f5406c1f0 0x7f9f5406e6b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f7c071f00 0x7f9f7c15db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9f7c071410 0x7f9f7c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 --2- 172.21.15.141:0/1475601895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f7c06c7c0 0x7f9f7c12fc40 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 >> 172.21.15.141:0/1475601895 conn(0x7f9f7c06a800 msgr2=0x7f9f7c0b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 shutdown_connections 2024-01-16T15:11:56.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.043+0000 7f9f83000640 1 -- 172.21.15.141:0/1475601895 wait complete. 2024-01-16T15:11:56.044 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:11:56.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.284+0000 7f5e06400640 1 Processor -- start 2024-01-16T15:11:56.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.284+0000 7f5e06400640 1 -- start start 2024-01-16T15:11:56.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.286+0000 7f5e06400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5e000ae5a0 0x7f5e0015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.286+0000 7f5e06400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 0x7f5e0006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.286+0000 7f5e06400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e00070c40 con 0x7f5e0006d120 2024-01-16T15:11:56.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.286+0000 7f5e06400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e00071450 con 0x7f5e000ae5a0 2024-01-16T15:11:56.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.286+0000 7f5e06400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0006c470 con 0x7f5e000b3ae0 2024-01-16T15:11:56.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.286+0000 7f5dffe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5e0006d120 0x7f5e000cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:46750/0 (socket says 172.21.15.141:46750) 2024-01-16T15:11:56.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.286+0000 7f5dffe00640 1 -- 172.21.15.141:0/509861707 learned_addr learned my addr 172.21.15.141:0/509861707 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:56.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5df6800640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 0x7f5e0006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dff400640 1 --2- 172.21.15.141:0/509861707 >> v2:172.21.15.188:3300/0 conn(0x7f5e000ae5a0 0x7f5e0015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456402078 0 0) 0x7f5e00070c40 con 0x7f5e0006d120 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de0003520 con 0x7f5e0006d120 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4284156910 0 0) 0x7f5de0003520 con 0x7f5e0006d120 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 msgr2=0x7f5e0006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 0x7f5e0006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 >> v2:172.21.15.188:3300/0 conn(0x7f5e000ae5a0 msgr2=0x7f5e0015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 --2- 172.21.15.141:0/509861707 >> v2:172.21.15.188:3300/0 conn(0x7f5e000ae5a0 0x7f5e0015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0006c750 con 0x7f5e0006d120 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f5df80019f0 con 0x7f5e0006d120 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 >> v1:172.21.15.141:6789/0 conn(0x7f5e0006d120 legacy=0x7f5e000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 0x7f5de0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5de0003ef0 0x7f5de00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0004d00 0x7f5de0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00071450 con 0x7f5de0003ef0 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0006c470 con 0x7f5de0004d00 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.287+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de000b980 con 0x7f5e000b3ae0 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5df6800640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5de0003ef0 0x7f5de00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5dff400640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 0x7f5de0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5dffe00640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0004d00 0x7f5de0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5df6800640 1 -- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 msgr2=0x7f5de0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5df6800640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5e000b3ae0 0x7f5de0003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5df6800640 1 -- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0004d00 msgr2=0x7f5de0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5df6800640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0004d00 0x7f5de0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5df6800640 1 -- 172.21.15.141:0/509861707 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0003520 con 0x7f5de0003ef0 2024-01-16T15:11:56.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.288+0000 7f5dffe00640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0004d00 0x7f5de0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:56.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.289+0000 7f5df6800640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5de0003ef0 0x7f5de00047c0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009ac0 tx=0x7f5de00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6790aa56f7330d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:56.267 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de802c6b0 con 0x7f5de0003ef0 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de802c810 con 0x7f5de0003ef0 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5dfea00640 1 -- 172.21.15.141:0/509861707 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de802cba0 con 0x7f5de0003ef0 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5e06400640 1 -- 172.21.15.141:0/509861707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5de0003ef0 msgr2=0x7f5de00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5e06400640 1 --2- 172.21.15.141:0/509861707 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5de0003ef0 0x7f5de00047c0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009ac0 tx=0x7f5de00057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5dffe00640 1 -- 172.21.15.141:0/509861707 reap_dead start 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5e06400640 1 -- 172.21.15.141:0/509861707 shutdown_connections 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5e06400640 1 -- 172.21.15.141:0/509861707 >> 172.21.15.141:0/509861707 conn(0x7f5e0006a800 msgr2=0x7f5e0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5e06400640 1 -- 172.21.15.141:0/509861707 shutdown_connections 2024-01-16T15:11:56.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.290+0000 7f5e06400640 1 -- 172.21.15.141:0/509861707 wait complete. 2024-01-16T15:11:56.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.292+0000 7f5e06400640 1 Processor -- start 2024-01-16T15:11:56.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.292+0000 7f5e06400640 1 -- start start 2024-01-16T15:11:56.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5e06400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5de0003ef0 0x7f5e000ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5e06400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e000ce960 0x7f5e000cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5e06400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e001f2bd0 0x7f5e00160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5e06400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e000cac20 con 0x7f5e000ce960 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5e06400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e000cc090 con 0x7f5e001f2bd0 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5e06400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e000cbde0 con 0x7f5de0003ef0 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5df6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e000ce960 0x7f5e000cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5df6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e000ce960 0x7f5e000cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51368/0 (socket says 172.21.15.141:51368) 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.293+0000 7f5df6800640 1 -- 172.21.15.141:0/1796566371 learned_addr learned my addr 172.21.15.141:0/1796566371 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5dff400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5de0003ef0 0x7f5e000ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5dffe00640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e001f2bd0 0x7f5e00160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5df6800640 1 -- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5de0003ef0 msgr2=0x7f5e000ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5df6800640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5de0003ef0 0x7f5e000ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5df6800640 1 -- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e001f2bd0 msgr2=0x7f5e00160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5df6800640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e001f2bd0 0x7f5e00160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5df6800640 1 -- 172.21.15.141:0/1796566371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00156ed0 con 0x7f5e000ce960 2024-01-16T15:11:56.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.294+0000 7f5df6800640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e000ce960 0x7f5e000cb8a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f5de8037040 tx=0x7f5de8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:56.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.295+0000 7f5df7e00640 1 -- 172.21.15.141:0/1796566371 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8003810 con 0x7f5e000ce960 2024-01-16T15:11:56.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.295+0000 7f5df7e00640 1 -- 172.21.15.141:0/1796566371 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de8003970 con 0x7f5e000ce960 2024-01-16T15:11:56.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.295+0000 7f5df7e00640 1 -- 172.21.15.141:0/1796566371 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8003d20 con 0x7f5e000ce960 2024-01-16T15:11:56.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.295+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e001571b0 con 0x7f5e000ce960 2024-01-16T15:11:56.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.295+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e00157770 con 0x7f5e000ce960 2024-01-16T15:11:56.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.296+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e00157900 con 0x7f5e000ce960 2024-01-16T15:11:56.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.297+0000 7f5df7e00640 1 -- 172.21.15.141:0/1796566371 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f5de8034440 con 0x7f5e000ce960 2024-01-16T15:11:56.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.298+0000 7f5df7e00640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806c220 0x7f5dd806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.298+0000 7f5df7e00640 1 -- 172.21.15.141:0/1796566371 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5de80aa690 con 0x7f5e000ce960 2024-01-16T15:11:56.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.299+0000 7f5dff400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806c220 0x7f5dd806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.300+0000 7f5dff400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806c220 0x7f5dd806e6e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009a90 tx=0x7f5df0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:56.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.302+0000 7f5df7e00640 1 -- 172.21.15.141:0/1796566371 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5de8077120 con 0x7f5e000ce960 2024-01-16T15:11:56.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.510+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e000cac20 con 0x7f5dd806c220 2024-01-16T15:11:56.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.514+0000 7f5df7e00640 1 -- 172.21.15.141:0/1796566371 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5df00097c0 con 0x7f5dd806c220 2024-01-16T15:11:56.490 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:11:56.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:11:56.490 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:11:56.490 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:11:56.491 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.516+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806c220 msgr2=0x7f5dd806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.516+0000 7f5e06400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806c220 0x7f5dd806e6e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009a90 tx=0x7f5df0008040 comp rx=0 tx=0).stop 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.516+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e000ce960 msgr2=0x7f5e000cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.516+0000 7f5e06400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e000ce960 0x7f5e000cb8a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f5de8037040 tx=0x7f5de8004730 comp rx=0 tx=0).stop 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.516+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 shutdown_connections 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.516+0000 7f5e06400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5dd806c220 0x7f5dd806e6e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.517+0000 7f5e06400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e001f2bd0 0x7f5e00160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.517+0000 7f5e06400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e000ce960 0x7f5e000cb8a0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.517+0000 7f5e06400640 1 --2- 172.21.15.141:0/1796566371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5de0003ef0 0x7f5e000ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.517+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 >> 172.21.15.141:0/1796566371 conn(0x7f5e0006a800 msgr2=0x7f5e000cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:56.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.517+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 shutdown_connections 2024-01-16T15:11:56.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.517+0000 7f5e06400640 1 -- 172.21.15.141:0/1796566371 wait complete. 2024-01-16T15:11:56.510 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:11:56.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.739+0000 7f91f7400640 1 Processor -- start 2024-01-16T15:11:56.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.739+0000 7f91f7400640 1 -- start start 2024-01-16T15:11:56.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f91f013cda0 0x7f91f015da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f7400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f91f00ce0c0 0x7f91f0070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f7400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00ce4f0 con 0x7f91f015dfa0 2024-01-16T15:11:56.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f7400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00700f0 con 0x7f91f013cda0 2024-01-16T15:11:56.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f7400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f015e3e0 con 0x7f91f00ce0c0 2024-01-16T15:11:56.717 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f6000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f91f013cda0 0x7f91f015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f6000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f91f013cda0 0x7f91f015da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51378/0 (socket says 172.21.15.141:51378) 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f6000640 1 -- 172.21.15.141:0/2939791351 learned_addr learned my addr 172.21.15.141:0/2939791351 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.740+0000 7f91f5600640 1 --1- 172.21.15.141:0/2939791351 >> v1:172.21.15.141:6789/0 conn(0x7f91f015dfa0 0x7f91f00cd9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:46762/0 (socket says 172.21.15.141:46762) 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.741+0000 7f91f6a00640 1 --2- 172.21.15.141:0/2939791351 >> v2:172.21.15.188:3300/0 conn(0x7f91f00ce0c0 0x7f91f0070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.741+0000 7f91f6000640 1 -- 172.21.15.141:0/2939791351 >> v2:172.21.15.188:3300/0 conn(0x7f91f00ce0c0 msgr2=0x7f91f0070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.741+0000 7f91f6000640 1 --2- 172.21.15.141:0/2939791351 >> v2:172.21.15.188:3300/0 conn(0x7f91f00ce0c0 0x7f91f0070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.741+0000 7f91f6000640 1 -- 172.21.15.141:0/2939791351 >> v1:172.21.15.141:6789/0 conn(0x7f91f015dfa0 legacy=0x7f91f00cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.741+0000 7f91f6000640 1 -- 172.21.15.141:0/2939791351 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00ce720 con 0x7f91f013cda0 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.741+0000 7f91f4c00640 1 -- 172.21.15.141:0/2939791351 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540044158 0 0) 0x7f91f00ce4f0 con 0x7f91f015dfa0 2024-01-16T15:11:56.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.741+0000 7f91f6a00640 1 --2- 172.21.15.141:0/2939791351 >> v2:172.21.15.188:3300/0 conn(0x7f91f00ce0c0 0x7f91f0070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:56.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.742+0000 7f91f6000640 1 --2- 172.21.15.141:0/2939791351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f013cda0 0x7f91f015da60 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f91f00aea90 tx=0x7f91e002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c7a741b84d8468 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:56.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.742+0000 7f91f4c00640 1 -- 172.21.15.141:0/2939791351 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0031890 con 0x7f91f013cda0 2024-01-16T15:11:56.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.742+0000 7f91f4c00640 1 -- 172.21.15.141:0/2939791351 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e00319f0 con 0x7f91f013cda0 2024-01-16T15:11:56.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.742+0000 7f91f4c00640 1 -- 172.21.15.141:0/2939791351 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0031d80 con 0x7f91f013cda0 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 -- 172.21.15.141:0/2939791351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f013cda0 msgr2=0x7f91f015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 --2- 172.21.15.141:0/2939791351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f013cda0 0x7f91f015da60 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f91f00aea90 tx=0x7f91e002b930 comp rx=0 tx=0).stop 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 -- 172.21.15.141:0/2939791351 shutdown_connections 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 --2- 172.21.15.141:0/2939791351 >> v2:172.21.15.188:3300/0 conn(0x7f91f00ce0c0 0x7f91f0070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 --2- 172.21.15.141:0/2939791351 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f013cda0 0x7f91f015da60 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 -- 172.21.15.141:0/2939791351 >> 172.21.15.141:0/2939791351 conn(0x7f91f006a3e0 msgr2=0x7f91f006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 -- 172.21.15.141:0/2939791351 shutdown_connections 2024-01-16T15:11:56.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.743+0000 7f91f7400640 1 -- 172.21.15.141:0/2939791351 wait complete. 2024-01-16T15:11:56.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.745+0000 7f91f7400640 1 Processor -- start 2024-01-16T15:11:56.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.745+0000 7f91f7400640 1 -- start start 2024-01-16T15:11:56.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f00ce0c0 0x7f91f0153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f013cda0 0x7f91f00b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f7400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f015dfa0 0x7f91f00b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f7400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00ad7e0 con 0x7f91f00ce0c0 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f7400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00adac0 con 0x7f91f013cda0 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f7400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00adda0 con 0x7f91f015dfa0 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f00ce0c0 0x7f91f0153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f00ce0c0 0x7f91f0153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51392/0 (socket says 172.21.15.141:51392) 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.746+0000 7f91f6000640 1 -- 172.21.15.141:0/1389139481 learned_addr learned my addr 172.21.15.141:0/1389139481 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6a00640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f015dfa0 0x7f91f00b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f5600640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f013cda0 0x7f91f00b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6000640 1 -- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f015dfa0 msgr2=0x7f91f00b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6000640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f015dfa0 0x7f91f00b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6000640 1 -- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f013cda0 msgr2=0x7f91f00b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6000640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f013cda0 0x7f91f00b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6000640 1 -- 172.21.15.141:0/1389139481 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00094d0 con 0x7f91f00ce0c0 2024-01-16T15:11:56.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6a00640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f015dfa0 0x7f91f00b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f5600640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f013cda0 0x7f91f00b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f6000640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f00ce0c0 0x7f91f0153d10 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f91e002b970 tx=0x7f91e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91ee000640 1 -- 172.21.15.141:0/1389139481 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0031860 con 0x7f91f00ce0c0 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91ee000640 1 -- 172.21.15.141:0/1389139481 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e00443f0 con 0x7f91f00ce0c0 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91ee000640 1 -- 172.21.15.141:0/1389139481 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00447a0 con 0x7f91f00ce0c0 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.747+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f00ae080 con 0x7f91f00ce0c0 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.748+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f00ad2e0 con 0x7f91f00ce0c0 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.748+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f00ad530 con 0x7f91f00ce0c0 2024-01-16T15:11:56.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.749+0000 7f91ee000640 1 -- 172.21.15.141:0/1389139481 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f91e004d450 con 0x7f91f00ce0c0 2024-01-16T15:11:56.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.750+0000 7f91ee000640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91c806c1f0 0x7f91c806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:56.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.750+0000 7f91ee000640 1 -- 172.21.15.141:0/1389139481 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91e00aaf70 con 0x7f91f00ce0c0 2024-01-16T15:11:56.727 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.751+0000 7f91f5600640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91c806c1f0 0x7f91c806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:56.728 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.752+0000 7f91f5600640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91c806c1f0 0x7f91c806e6b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f91f00b4560 tx=0x7f91d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:56.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.755+0000 7f91ee000640 1 -- 172.21.15.141:0/1389139481 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91e0077870 con 0x7f91f00ce0c0 2024-01-16T15:11:56.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.944+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f91f0067450 con 0x7f91f00ce0c0 2024-01-16T15:11:56.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.945+0000 7f91ee000640 1 -- 172.21.15.141:0/1389139481 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f91e0035020 con 0x7f91f00ce0c0 2024-01-16T15:11:56.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.947+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91c806c1f0 msgr2=0x7f91c806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91c806c1f0 0x7f91c806e6b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f91f00b4560 tx=0x7f91d8009040 comp rx=0 tx=0).stop 2024-01-16T15:11:56.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f00ce0c0 msgr2=0x7f91f0153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:56.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f00ce0c0 0x7f91f0153d10 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f91e002b970 tx=0x7f91e00046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 shutdown_connections 2024-01-16T15:11:56.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91c806c1f0 0x7f91c806e6b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f015dfa0 0x7f91f00b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f013cda0 0x7f91f00b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 --2- 172.21.15.141:0/1389139481 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f00ce0c0 0x7f91f0153d10 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:56.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 >> 172.21.15.141:0/1389139481 conn(0x7f91f006a3e0 msgr2=0x7f91f0150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:56.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 shutdown_connections 2024-01-16T15:11:56.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:56.948+0000 7f91f7400640 1 -- 172.21.15.141:0/1389139481 wait complete. 2024-01-16T15:11:56.941 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:11:57.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.168+0000 7f8e36800640 1 Processor -- start 2024-01-16T15:11:57.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.169+0000 7f8e36800640 1 -- start start 2024-01-16T15:11:57.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e36800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8e3015d4b0 0x7f8e300d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e36800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8e300cf9a0 0x7f8e300cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e36800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e300d0320 con 0x7f8e300cddf0 2024-01-16T15:11:57.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e36800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e300b2730 con 0x7f8e300cf9a0 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e36800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e300d1ec0 con 0x7f8e3015d4b0 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e34a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8e300cf9a0 0x7f8e300cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e35e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8e300cddf0 0x7f8e300ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:46766/0 (socket says 172.21.15.141:46766) 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e34a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8e300cf9a0 0x7f8e300cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51408/0 (socket says 172.21.15.141:51408) 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.170+0000 7f8e35e00640 1 -- 172.21.15.141:0/2762188955 learned_addr learned my addr 172.21.15.141:0/2762188955 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.171+0000 7f8e34a00640 1 -- 172.21.15.141:0/2762188955 >> v2:172.21.15.188:3300/0 conn(0x7f8e3015d4b0 msgr2=0x7f8e300d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.171+0000 7f8e34a00640 1 --2- 172.21.15.141:0/2762188955 >> v2:172.21.15.188:3300/0 conn(0x7f8e3015d4b0 0x7f8e300d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.171+0000 7f8e34a00640 1 -- 172.21.15.141:0/2762188955 >> v1:172.21.15.141:6789/0 conn(0x7f8e300cddf0 legacy=0x7f8e300ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.171+0000 7f8e34a00640 1 -- 172.21.15.141:0/2762188955 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e300b29d0 con 0x7f8e300cf9a0 2024-01-16T15:11:57.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.172+0000 7f8e34a00640 1 --2- 172.21.15.141:0/2762188955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e300cf9a0 0x7f8e300cfde0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f8e3008b5b0 tx=0x7f8e1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b268e999c1e018b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:57.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.172+0000 7f8e2fe00640 1 -- 172.21.15.141:0/2762188955 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18031830 con 0x7f8e300cf9a0 2024-01-16T15:11:57.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.172+0000 7f8e2fe00640 1 -- 172.21.15.141:0/2762188955 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e18031990 con 0x7f8e300cf9a0 2024-01-16T15:11:57.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.172+0000 7f8e2fe00640 1 -- 172.21.15.141:0/2762188955 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18031d20 con 0x7f8e300cf9a0 2024-01-16T15:11:57.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.172+0000 7f8e36800640 1 -- 172.21.15.141:0/2762188955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e300cf9a0 msgr2=0x7f8e300cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.172+0000 7f8e36800640 1 --2- 172.21.15.141:0/2762188955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e300cf9a0 0x7f8e300cfde0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f8e3008b5b0 tx=0x7f8e1802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.173+0000 7f8e36800640 1 -- 172.21.15.141:0/2762188955 shutdown_connections 2024-01-16T15:11:57.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.173+0000 7f8e36800640 1 --2- 172.21.15.141:0/2762188955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e300cf9a0 0x7f8e300cfde0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.173+0000 7f8e36800640 1 --2- 172.21.15.141:0/2762188955 >> v2:172.21.15.188:3300/0 conn(0x7f8e3015d4b0 0x7f8e300d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.173+0000 7f8e36800640 1 -- 172.21.15.141:0/2762188955 >> 172.21.15.141:0/2762188955 conn(0x7f8e300d3000 msgr2=0x7f8e300d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:57.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.173+0000 7f8e36800640 1 -- 172.21.15.141:0/2762188955 shutdown_connections 2024-01-16T15:11:57.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.173+0000 7f8e36800640 1 -- 172.21.15.141:0/2762188955 wait complete. 2024-01-16T15:11:57.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.174+0000 7f8e36800640 1 Processor -- start 2024-01-16T15:11:57.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.174+0000 7f8e36800640 1 -- start start 2024-01-16T15:11:57.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.175+0000 7f8e36800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e300cddf0 0x7f8e301f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.175+0000 7f8e36800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e300cf9a0 0x7f8e301e9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.175+0000 7f8e36800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3015d4b0 0x7f8e3006bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.175+0000 7f8e36800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3009cce0 con 0x7f8e3015d4b0 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.175+0000 7f8e36800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e301e96b0 con 0x7f8e300cddf0 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.175+0000 7f8e36800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3009cfa0 con 0x7f8e300cf9a0 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3015d4b0 0x7f8e3006bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3015d4b0 0x7f8e3006bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51414/0 (socket says 172.21.15.141:51414) 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 -- 172.21.15.141:0/2660889591 learned_addr learned my addr 172.21.15.141:0/2660889591 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e34a00640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e300cf9a0 0x7f8e301e9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 -- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e300cf9a0 msgr2=0x7f8e301e9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35400640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e300cddf0 0x7f8e301f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e300cf9a0 0x7f8e301e9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 -- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e300cddf0 msgr2=0x7f8e301f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e300cddf0 0x7f8e301f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 -- 172.21.15.141:0/2660889591 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18009400 con 0x7f8e3015d4b0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35400640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e300cddf0 0x7f8e301f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e35e00640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3015d4b0 0x7f8e3006bae0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800cbc0 tx=0x7f8e2800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.176+0000 7f8e2d600640 1 -- 172.21.15.141:0/2660889591 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800ad40 con 0x7f8e3015d4b0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.177+0000 7f8e2d600640 1 -- 172.21.15.141:0/2660889591 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e28012400 con 0x7f8e3015d4b0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.177+0000 7f8e2d600640 1 -- 172.21.15.141:0/2660889591 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800aea0 con 0x7f8e3015d4b0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.177+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3015da10 con 0x7f8e3015d4b0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.177+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3009d560 con 0x7f8e3015d4b0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.178+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e300b38e0 con 0x7f8e3015d4b0 2024-01-16T15:11:57.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.179+0000 7f8e2d600640 1 -- 172.21.15.141:0/2660889591 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f8e28007070 con 0x7f8e3015d4b0 2024-01-16T15:11:57.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.179+0000 7f8e2d600640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e0806c1f0 0x7f8e0806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.180+0000 7f8e2d600640 1 -- 172.21.15.141:0/2660889591 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e28088560 con 0x7f8e3015d4b0 2024-01-16T15:11:57.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.180+0000 7f8e35400640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e0806c1f0 0x7f8e0806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.181+0000 7f8e35400640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e0806c1f0 0x7f8e0806e6b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000a870 tx=0x7f8e20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:57.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.184+0000 7f8e2d600640 1 -- 172.21.15.141:0/2660889591 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e28054f30 con 0x7f8e3015d4b0 2024-01-16T15:11:57.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.376+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f8e3015d060 con 0x7f8e3015d4b0 2024-01-16T15:11:57.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.382+0000 7f8e2d600640 1 -- 172.21.15.141:0/2660889591 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v37)=0 v37) v1 ==== 131+0+0 (secure 0 0 0) 0x7f8e280170c0 con 0x7f8e3015d4b0 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.385+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e0806c1f0 msgr2=0x7f8e0806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.385+0000 7f8e36800640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e0806c1f0 0x7f8e0806e6b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000a870 tx=0x7f8e20008040 comp rx=0 tx=0).stop 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.385+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3015d4b0 msgr2=0x7f8e3006bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.385+0000 7f8e36800640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3015d4b0 0x7f8e3006bae0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800cbc0 tx=0x7f8e2800a430 comp rx=0 tx=0).stop 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.385+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 shutdown_connections 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.385+0000 7f8e36800640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8e0806c1f0 0x7f8e0806e6b0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.386+0000 7f8e36800640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e3015d4b0 0x7f8e3006bae0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.386+0000 7f8e36800640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8e300cf9a0 0x7f8e301e9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.386+0000 7f8e36800640 1 --2- 172.21.15.141:0/2660889591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e300cddf0 0x7f8e301f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.386+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 >> 172.21.15.141:0/2660889591 conn(0x7f8e300d3000 msgr2=0x7f8e300b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:57.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.386+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 shutdown_connections 2024-01-16T15:11:57.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.386+0000 7f8e36800640 1 -- 172.21.15.141:0/2660889591 wait complete. 2024-01-16T15:11:57.380 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:11:57.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.618+0000 7f96a3400640 1 Processor -- start 2024-01-16T15:11:57.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.618+0000 7f96a3400640 1 -- start start 2024-01-16T15:11:57.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.620+0000 7f96a3400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f969c143ae0 0x7f969c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.620+0000 7f96a3400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0cfc30 con 0x7f969c0cf7d0 2024-01-16T15:11:57.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.620+0000 7f96a3400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0b1ee0 con 0x7f969c15e3b0 2024-01-16T15:11:57.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.620+0000 7f96a3400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c0b2780 con 0x7f969c143ae0 2024-01-16T15:11:57.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.620+0000 7f96a0a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f969c143ae0 0x7f969c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.620+0000 7f969be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f969c0cf7d0 0x7f969c15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:46772/0 (socket says 172.21.15.141:46772) 2024-01-16T15:11:57.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.620+0000 7f969be00640 1 -- 172.21.15.141:0/3064582268 learned_addr learned my addr 172.21.15.141:0/3064582268 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:11:57.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.621+0000 7f96a0a00640 1 -- 172.21.15.141:0/3064582268 >> v1:172.21.15.188:6789/0 conn(0x7f969c15e3b0 legacy=0x7f969c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:57.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.621+0000 7f96a0a00640 1 -- 172.21.15.141:0/3064582268 >> v1:172.21.15.141:6789/0 conn(0x7f969c0cf7d0 legacy=0x7f969c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.621+0000 7f96a0a00640 1 -- 172.21.15.141:0/3064582268 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c15e990 con 0x7f969c143ae0 2024-01-16T15:11:57.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.621+0000 7f969b400640 1 -- 172.21.15.141:0/3064582268 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206961298 0 0) 0x7f969c0cfc30 con 0x7f969c0cf7d0 2024-01-16T15:11:57.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.622+0000 7f96a0a00640 1 --2- 172.21.15.141:0/3064582268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c143ae0 0x7f969c0cf290 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f968c0097a0 tx=0x7f968c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d19788252549a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:57.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.622+0000 7f969b400640 1 -- 172.21.15.141:0/3064582268 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c031830 con 0x7f969c143ae0 2024-01-16T15:11:57.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.622+0000 7f969b400640 1 -- 172.21.15.141:0/3064582268 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c031990 con 0x7f969c143ae0 2024-01-16T15:11:57.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.622+0000 7f969b400640 1 -- 172.21.15.141:0/3064582268 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c031d20 con 0x7f969c143ae0 2024-01-16T15:11:57.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.623+0000 7f96a3400640 1 -- 172.21.15.141:0/3064582268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c143ae0 msgr2=0x7f969c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.623+0000 7f96a3400640 1 --2- 172.21.15.141:0/3064582268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c143ae0 0x7f969c0cf290 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f968c0097a0 tx=0x7f968c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.623+0000 7f96a3400640 1 -- 172.21.15.141:0/3064582268 shutdown_connections 2024-01-16T15:11:57.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.623+0000 7f96a3400640 1 --2- 172.21.15.141:0/3064582268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c143ae0 0x7f969c0cf290 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.623+0000 7f96a3400640 1 -- 172.21.15.141:0/3064582268 >> 172.21.15.141:0/3064582268 conn(0x7f969c06a800 msgr2=0x7f969c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:57.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.623+0000 7f96a3400640 1 -- 172.21.15.141:0/3064582268 shutdown_connections 2024-01-16T15:11:57.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.623+0000 7f96a3400640 1 -- 172.21.15.141:0/3064582268 wait complete. 2024-01-16T15:11:57.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.625+0000 7f96a3400640 1 Processor -- start 2024-01-16T15:11:57.601 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.625+0000 7f96a3400640 1 -- start start 2024-01-16T15:11:57.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f969c0cf7d0 0x7f969c159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a3400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f969c143ae0 0x7f969c137240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c15e3b0 0x7f969c0cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a3400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0ce8f0 con 0x7f969c15e3b0 2024-01-16T15:11:57.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a3400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c15a0d0 con 0x7f969c0cf7d0 2024-01-16T15:11:57.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a3400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0ce610 con 0x7f969c143ae0 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a1400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c15e3b0 0x7f969c0cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a1400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c15e3b0 0x7f969c0cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51434/0 (socket says 172.21.15.141:51434) 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a1400640 1 -- 172.21.15.141:0/3637122461 learned_addr learned my addr 172.21.15.141:0/3637122461 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a0a00640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f969c0cf7d0 0x7f969c159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f969be00640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f969c143ae0 0x7f969c137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a1400640 1 -- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f969c143ae0 msgr2=0x7f969c137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a1400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f969c143ae0 0x7f969c137240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a1400640 1 -- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f969c0cf7d0 msgr2=0x7f969c159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.626+0000 7f96a1400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f969c0cf7d0 0x7f969c159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f96a1400640 1 -- 172.21.15.141:0/3637122461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c009400 con 0x7f969c15e3b0 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f96a0a00640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f969c0cf7d0 0x7f969c159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f96a1400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c15e3b0 0x7f969c0cdb70 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f969400b970 tx=0x7f969400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:57.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f9698c00640 1 -- 172.21.15.141:0/3637122461 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969400cb40 con 0x7f969c15e3b0 2024-01-16T15:11:57.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f9698c00640 1 -- 172.21.15.141:0/3637122461 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969400cca0 con 0x7f969c15e3b0 2024-01-16T15:11:57.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f9698c00640 1 -- 172.21.15.141:0/3637122461 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694014680 con 0x7f969c15e3b0 2024-01-16T15:11:57.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c0c3ff0 con 0x7f969c15e3b0 2024-01-16T15:11:57.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.627+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c0c45b0 con 0x7f969c15e3b0 2024-01-16T15:11:57.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.629+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c0c3e50 con 0x7f969c15e3b0 2024-01-16T15:11:57.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.629+0000 7f9698c00640 1 -- 172.21.15.141:0/3637122461 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f969401d3b0 con 0x7f969c15e3b0 2024-01-16T15:11:57.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.630+0000 7f9698c00640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f967406c1f0 0x7f967406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:57.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.630+0000 7f9698c00640 1 -- 172.21.15.141:0/3637122461 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9694088620 con 0x7f969c15e3b0 2024-01-16T15:11:57.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.631+0000 7f96a0a00640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f967406c1f0 0x7f967406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:57.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.632+0000 7f96a0a00640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f967406c1f0 0x7f967406e6b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f968c036000 tx=0x7f968c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:57.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.635+0000 7f9698c00640 1 -- 172.21.15.141:0/3637122461 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f969c0c3e50 con 0x7f969c15e3b0 2024-01-16T15:11:57.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.829+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f969c0c3e50 con 0x7f969c15e3b0 2024-01-16T15:11:57.811 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.835+0000 7f9698c00640 1 -- 172.21.15.141:0/3637122461 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v38)=0 v38) v1 ==== 137+0+0 (secure 0 0 0) 0x7f969c0c3e50 con 0x7f969c15e3b0 2024-01-16T15:11:57.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.837+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f967406c1f0 msgr2=0x7f967406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.814 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.837+0000 7f96a3400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f967406c1f0 0x7f967406e6b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f968c036000 tx=0x7f968c008030 comp rx=0 tx=0).stop 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.837+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c15e3b0 msgr2=0x7f969c0cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.837+0000 7f96a3400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c15e3b0 0x7f969c0cdb70 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f969400b970 tx=0x7f969400be30 comp rx=0 tx=0).stop 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 shutdown_connections 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f967406c1f0 0x7f967406e6b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f969c15e3b0 0x7f969c0cdb70 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f969c143ae0 0x7f969c137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 --2- 172.21.15.141:0/3637122461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f969c0cf7d0 0x7f969c159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 >> 172.21.15.141:0/3637122461 conn(0x7f969c06a800 msgr2=0x7f969c0b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 shutdown_connections 2024-01-16T15:11:57.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:57.838+0000 7f96a3400640 1 -- 172.21.15.141:0/3637122461 wait complete. 2024-01-16T15:11:57.835 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:11:58.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.064+0000 7f7edbe00640 1 Processor -- start 2024-01-16T15:11:58.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.065+0000 7f7edbe00640 1 -- start start 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.066+0000 7f7edbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7ed40cf9a0 0x7f7ed415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.066+0000 7f7edbe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40b1ee0 con 0x7f7ed4143ae0 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.066+0000 7f7edbe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40b2780 con 0x7f7ed415e1e0 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.066+0000 7f7edbe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed415e990 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.066+0000 7f7ed8a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7ed40cf9a0 0x7f7ed415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.067+0000 7f7ed8a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7ed40cf9a0 0x7f7ed415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51450/0 (socket says 172.21.15.141:51450) 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.067+0000 7f7ed8a00640 1 -- 172.21.15.141:0/1999903139 learned_addr learned my addr 172.21.15.141:0/1999903139 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:58.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.067+0000 7f7ed9400640 1 --1- 172.21.15.141:0/1999903139 >> v1:172.21.15.141:6789/0 conn(0x7f7ed4143ae0 0x7f7ed40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:46778/0 (socket says 172.21.15.141:46778) 2024-01-16T15:11:58.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.067+0000 7f7ed8a00640 1 -- 172.21.15.141:0/1999903139 >> v1:172.21.15.188:6789/0 conn(0x7f7ed415e1e0 legacy=0x7f7ed40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:58.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.067+0000 7f7ed8a00640 1 -- 172.21.15.141:0/1999903139 >> v1:172.21.15.141:6789/0 conn(0x7f7ed4143ae0 legacy=0x7f7ed40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.067+0000 7f7ed8a00640 1 -- 172.21.15.141:0/1999903139 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed40b2130 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.067+0000 7f7ed3e00640 1 -- 172.21.15.141:0/1999903139 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062445916 0 0) 0x7f7ed40b1ee0 con 0x7f7ed4143ae0 2024-01-16T15:11:58.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.068+0000 7f7ed8a00640 1 --2- 172.21.15.141:0/1999903139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed415dca0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009870 tx=0x7f7ebc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd83619ea2f1b69c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:58.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.068+0000 7f7ed3e00640 1 -- 172.21.15.141:0/1999903139 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc030860 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.068+0000 7f7ed3e00640 1 -- 172.21.15.141:0/1999903139 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc0309c0 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.068+0000 7f7ed3e00640 1 -- 172.21.15.141:0/1999903139 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc030d50 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.069+0000 7f7edbe00640 1 -- 172.21.15.141:0/1999903139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 msgr2=0x7f7ed415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.069+0000 7f7edbe00640 1 --2- 172.21.15.141:0/1999903139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed415dca0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009870 tx=0x7f7ebc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.069+0000 7f7edbe00640 1 -- 172.21.15.141:0/1999903139 shutdown_connections 2024-01-16T15:11:58.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.069+0000 7f7edbe00640 1 --2- 172.21.15.141:0/1999903139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed415dca0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.069+0000 7f7edbe00640 1 -- 172.21.15.141:0/1999903139 >> 172.21.15.141:0/1999903139 conn(0x7f7ed406a800 msgr2=0x7f7ed406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:58.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.069+0000 7f7edbe00640 1 -- 172.21.15.141:0/1999903139 shutdown_connections 2024-01-16T15:11:58.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.069+0000 7f7edbe00640 1 -- 172.21.15.141:0/1999903139 wait complete. 2024-01-16T15:11:58.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.071+0000 7f7edbe00640 1 Processor -- start 2024-01-16T15:11:58.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.071+0000 7f7edbe00640 1 -- start start 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7edbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed40af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7edbe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ed4143ae0 0x7f7ed413e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7edbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ed415e1e0 0x7f7ed41f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7edbe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed40af0e0 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7edbe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed40afa40 con 0x7f7ed415e1e0 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7edbe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed41f6d70 con 0x7f7ed4143ae0 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7ed9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed40af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7ed9400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed40af500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51466/0 (socket says 172.21.15.141:51466) 2024-01-16T15:11:58.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.072+0000 7f7ed9400640 1 -- 172.21.15.141:0/315465575 learned_addr learned my addr 172.21.15.141:0/315465575 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:58.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed9400640 1 -- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ed4143ae0 msgr2=0x7f7ed413e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed9400640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ed4143ae0 0x7f7ed413e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed9400640 1 -- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ed415e1e0 msgr2=0x7f7ed41f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed9400640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ed415e1e0 0x7f7ed41f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed9400640 1 -- 172.21.15.141:0/315465575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0094d0 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed9400640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed40af500 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c890 tx=0x7f7ec400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:58.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed1600640 1 -- 172.21.15.141:0/315465575 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec400a640 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed1600640 1 -- 172.21.15.141:0/315465575 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec400a7c0 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.073+0000 7f7ed1600640 1 -- 172.21.15.141:0/315465575 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec400ab70 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.074+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed41f7050 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.074+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed40aed00 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.075+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed40675d0 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.076+0000 7f7ed1600640 1 -- 172.21.15.141:0/315465575 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f7ec400acd0 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.076+0000 7f7ed1600640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7eac06c220 0x7f7eac06e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.077+0000 7f7ed1600640 1 -- 172.21.15.141:0/315465575 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ec4087ec0 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.077+0000 7f7ed8a00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7eac06c220 0x7f7eac06e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.078+0000 7f7ed8a00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7eac06c220 0x7f7eac06e6e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc036000 tx=0x7f7ebc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:58.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.081+0000 7f7ed1600640 1 -- 172.21.15.141:0/315465575 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ec4054490 con 0x7f7ed40cf9a0 2024-01-16T15:11:58.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.297+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ed415e670 con 0x7f7eac06c220 2024-01-16T15:11:58.283 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:58.308+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:58.283 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:58.308+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:58.283 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:58.308+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:58.284 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:58.308+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:58.284 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:11:58.308+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:11:58.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.367+0000 7f7ed1600640 1 -- 172.21.15.141:0/315465575 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7ebc030bb0 con 0x7f7eac06c220 2024-01-16T15:11:58.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7eac06c220 msgr2=0x7f7eac06e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7eac06c220 0x7f7eac06e6e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc036000 tx=0x7f7ebc03b040 comp rx=0 tx=0).stop 2024-01-16T15:11:58.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 msgr2=0x7f7ed40af500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed40af500 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c890 tx=0x7f7ec400cd50 comp rx=0 tx=0).stop 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 shutdown_connections 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7eac06c220 0x7f7eac06e6e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ed415e1e0 0x7f7ed41f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ed4143ae0 0x7f7ed413e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 --2- 172.21.15.141:0/315465575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed40cf9a0 0x7f7ed40af500 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 >> 172.21.15.141:0/315465575 conn(0x7f7ed406a800 msgr2=0x7f7ed406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 shutdown_connections 2024-01-16T15:11:58.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.370+0000 7f7edbe00640 1 -- 172.21.15.141:0/315465575 wait complete. 2024-01-16T15:11:58.367 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:11:58.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.606+0000 7f3e74200640 1 Processor -- start 2024-01-16T15:11:58.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.606+0000 7f3e74200640 1 -- start start 2024-01-16T15:11:58.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.608+0000 7f3e74200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e6c0ae5a0 0x7f3e6c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.608+0000 7f3e74200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c070c40 con 0x7f3e6c0cf290 2024-01-16T15:11:58.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.608+0000 7f3e74200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c06c470 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.608+0000 7f3e74200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6c071450 con 0x7f3e6c0ae5a0 2024-01-16T15:11:58.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.608+0000 7f3e71800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e6c0ae5a0 0x7f3e6c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.608+0000 7f3e71800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e6c0ae5a0 0x7f3e6c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51472/0 (socket says 172.21.15.141:51472) 2024-01-16T15:11:58.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.608+0000 7f3e71800640 1 -- 172.21.15.141:0/154505608 learned_addr learned my addr 172.21.15.141:0/154505608 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:58.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.609+0000 7f3e71800640 1 -- 172.21.15.141:0/154505608 >> v1:172.21.15.188:6789/0 conn(0x7f3e6c0b3ae0 legacy=0x7f3e6c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:58.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.609+0000 7f3e71800640 1 -- 172.21.15.141:0/154505608 >> v1:172.21.15.141:6789/0 conn(0x7f3e6c0cf290 legacy=0x7f3e6c0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.585 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.609+0000 7f3e71800640 1 -- 172.21.15.141:0/154505608 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c06c750 con 0x7f3e6c0ae5a0 2024-01-16T15:11:58.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.610+0000 7f3e71800640 1 --2- 172.21.15.141:0/154505608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0ae5a0 0x7f3e6c15e430 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009870 tx=0x7f3e5c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce9fb131721013b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:58.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.610+0000 7f3e6be00640 1 -- 172.21.15.141:0/154505608 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c02f850 con 0x7f3e6c0ae5a0 2024-01-16T15:11:58.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.610+0000 7f3e6be00640 1 -- 172.21.15.141:0/154505608 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c02f9b0 con 0x7f3e6c0ae5a0 2024-01-16T15:11:58.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.610+0000 7f3e6be00640 1 -- 172.21.15.141:0/154505608 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c02fd40 con 0x7f3e6c0ae5a0 2024-01-16T15:11:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.611+0000 7f3e74200640 1 -- 172.21.15.141:0/154505608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0ae5a0 msgr2=0x7f3e6c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.611+0000 7f3e74200640 1 --2- 172.21.15.141:0/154505608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0ae5a0 0x7f3e6c15e430 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009870 tx=0x7f3e5c02ba30 comp rx=0 tx=0).stop 2024-01-16T15:11:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.611+0000 7f3e74200640 1 -- 172.21.15.141:0/154505608 shutdown_connections 2024-01-16T15:11:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.611+0000 7f3e74200640 1 --2- 172.21.15.141:0/154505608 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0ae5a0 0x7f3e6c15e430 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.611+0000 7f3e74200640 1 -- 172.21.15.141:0/154505608 >> 172.21.15.141:0/154505608 conn(0x7f3e6c06a800 msgr2=0x7f3e6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.611+0000 7f3e74200640 1 -- 172.21.15.141:0/154505608 shutdown_connections 2024-01-16T15:11:58.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.611+0000 7f3e74200640 1 -- 172.21.15.141:0/154505608 wait complete. 2024-01-16T15:11:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.613+0000 7f3e74200640 1 Processor -- start 2024-01-16T15:11:58.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.613+0000 7f3e74200640 1 -- start start 2024-01-16T15:11:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e74200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6c0ae5a0 0x7f3e6c157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e74200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0b3ae0 0x7f3e6c0aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e74200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e6c0cf290 0x7f3e6c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e74200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c0aef90 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e74200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c0b0810 con 0x7f3e6c0cf290 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e74200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c0af230 con 0x7f3e6c0ae5a0 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e70e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0b3ae0 0x7f3e6c0aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e70e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0b3ae0 0x7f3e6c0aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51482/0 (socket says 172.21.15.141:51482) 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e70e00640 1 -- 172.21.15.141:0/435013325 learned_addr learned my addr 172.21.15.141:0/435013325 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.614+0000 7f3e71800640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6c0ae5a0 0x7f3e6c157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e70e00640 1 -- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6c0ae5a0 msgr2=0x7f3e6c157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e70e00640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6c0ae5a0 0x7f3e6c157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e70e00640 1 -- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e6c0cf290 msgr2=0x7f3e6c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e72200640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e6c0cf290 0x7f3e6c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e70e00640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e6c0cf290 0x7f3e6c0b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e70e00640 1 -- 172.21.15.141:0/435013325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0094d0 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e70e00640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0b3ae0 0x7f3e6c0aea50 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400c430 tx=0x7f3e5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:58.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e69600640 1 -- 172.21.15.141:0/435013325 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54011790 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e69600640 1 -- 172.21.15.141:0/435013325 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e540118f0 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c0af510 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e69600640 1 -- 172.21.15.141:0/435013325 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54011ce0 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.615+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c0afad0 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.617+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c0675d0 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.618+0000 7f3e69600640 1 -- 172.21.15.141:0/435013325 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f3e5401c460 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.619+0000 7f3e69600640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4406c1f0 0x7f3e4406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:58.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.619+0000 7f3e69600640 1 -- 172.21.15.141:0/435013325 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e540889a0 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.619+0000 7f3e71800640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4406c1f0 0x7f3e4406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:58.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.620+0000 7f3e71800640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4406c1f0 0x7f3e4406e6b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c036000 tx=0x7f3e5c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:58.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.623+0000 7f3e69600640 1 -- 172.21.15.141:0/435013325 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e54054f70 con 0x7f3e6c0b3ae0 2024-01-16T15:11:58.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.827+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e6c070550 con 0x7f3e4406c1f0 2024-01-16T15:11:58.815 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.839+0000 7f3e69600640 1 -- 172.21.15.141:0/435013325 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f3e5c02fcf0 con 0x7f3e4406c1f0 2024-01-16T15:11:58.815 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.841+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4406c1f0 msgr2=0x7f3e4406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.841+0000 7f3e74200640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4406c1f0 0x7f3e4406e6b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c036000 tx=0x7f3e5c008030 comp rx=0 tx=0).stop 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.841+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0b3ae0 msgr2=0x7f3e6c0aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.841+0000 7f3e74200640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0b3ae0 0x7f3e6c0aea50 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400c430 tx=0x7f3e5400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 shutdown_connections 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3e4406c1f0 0x7f3e4406e6b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e6c0cf290 0x7f3e6c0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e6c0b3ae0 0x7f3e6c0aea50 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 --2- 172.21.15.141:0/435013325 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3e6c0ae5a0 0x7f3e6c157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 >> 172.21.15.141:0/435013325 conn(0x7f3e6c06a800 msgr2=0x7f3e6c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 shutdown_connections 2024-01-16T15:11:58.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:58.842+0000 7f3e74200640 1 -- 172.21.15.141:0/435013325 wait complete. 2024-01-16T15:11:58.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:11:59.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.067+0000 7f46ba400640 1 Processor -- start 2024-01-16T15:11:59.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.068+0000 7f46ba400640 1 -- start start 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.069+0000 7f46ba400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46b40ae5a0 0x7f46b415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.069+0000 7f46ba400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f46b40cf290 0x7f46b40cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.069+0000 7f46ba400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b4070c40 con 0x7f46b40b3ae0 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.069+0000 7f46ba400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b4071450 con 0x7f46b40ae5a0 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.069+0000 7f46ba400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b40cfc60 con 0x7f46b40cf290 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.070+0000 7f46b3400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46b40ae5a0 0x7f46b415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.070+0000 7f46b3400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46b40ae5a0 0x7f46b415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51488/0 (socket says 172.21.15.141:51488) 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.070+0000 7f46b3400640 1 -- 172.21.15.141:0/3275995289 learned_addr learned my addr 172.21.15.141:0/3275995289 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:59.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.070+0000 7f46b3400640 1 -- 172.21.15.141:0/3275995289 >> v2:172.21.15.188:3300/0 conn(0x7f46b40cf290 msgr2=0x7f46b40cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:59.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.070+0000 7f46b3400640 1 --2- 172.21.15.141:0/3275995289 >> v2:172.21.15.188:3300/0 conn(0x7f46b40cf290 0x7f46b40cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.070+0000 7f46b3400640 1 -- 172.21.15.141:0/3275995289 >> v1:172.21.15.188:6789/0 conn(0x7f46b40b3ae0 legacy=0x7f46b406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:59.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.070+0000 7f46b3400640 1 -- 172.21.15.141:0/3275995289 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b406c470 con 0x7f46b40ae5a0 2024-01-16T15:11:59.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.071+0000 7f46b3400640 1 --2- 172.21.15.141:0/3275995289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40ae5a0 0x7f46b415e430 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f46b40b3f70 tx=0x7f46a402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e224126f53dfdb2f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.071+0000 7f46b2a00640 1 -- 172.21.15.141:0/3275995289 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a4030860 con 0x7f46b40ae5a0 2024-01-16T15:11:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46b2a00640 1 -- 172.21.15.141:0/3275995289 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a40309c0 con 0x7f46b40ae5a0 2024-01-16T15:11:59.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46b2a00640 1 -- 172.21.15.141:0/3275995289 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a4030d50 con 0x7f46b40ae5a0 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 -- 172.21.15.141:0/3275995289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40ae5a0 msgr2=0x7f46b415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 --2- 172.21.15.141:0/3275995289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40ae5a0 0x7f46b415e430 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f46b40b3f70 tx=0x7f46a402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 -- 172.21.15.141:0/3275995289 shutdown_connections 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 --2- 172.21.15.141:0/3275995289 >> v2:172.21.15.188:3300/0 conn(0x7f46b40cf290 0x7f46b40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 --2- 172.21.15.141:0/3275995289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40ae5a0 0x7f46b415e430 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 -- 172.21.15.141:0/3275995289 >> 172.21.15.141:0/3275995289 conn(0x7f46b406a800 msgr2=0x7f46b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 -- 172.21.15.141:0/3275995289 shutdown_connections 2024-01-16T15:11:59.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.072+0000 7f46ba400640 1 -- 172.21.15.141:0/3275995289 wait complete. 2024-01-16T15:11:59.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.074+0000 7f46ba400640 1 Processor -- start 2024-01-16T15:11:59.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.074+0000 7f46ba400640 1 -- start start 2024-01-16T15:11:59.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46ba400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46b40ae5a0 0x7f46b413e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46ba400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b40b3ae0 0x7f46b413ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46ba400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40cf290 0x7f46b41447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46ba400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b4158d20 con 0x7f46b40cf290 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46ba400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b41f2b30 con 0x7f46b40b3ae0 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46ba400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40cdf50 con 0x7f46b40ae5a0 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46b3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40cf290 0x7f46b41447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46b3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40cf290 0x7f46b41447b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:51498/0 (socket says 172.21.15.141:51498) 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.075+0000 7f46b3e00640 1 -- 172.21.15.141:0/3409001496 learned_addr learned my addr 172.21.15.141:0/3409001496 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46b3400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46b40ae5a0 0x7f46b413e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46aa800640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b40b3ae0 0x7f46b413ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46b3e00640 1 -- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46b40ae5a0 msgr2=0x7f46b413e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46b3e00640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46b40ae5a0 0x7f46b413e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46b3e00640 1 -- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b40b3ae0 msgr2=0x7f46b413ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:59.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46b3e00640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b40b3ae0 0x7f46b413ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46b3e00640 1 -- 172.21.15.141:0/3409001496 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40094d0 con 0x7f46b40cf290 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46aa800640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b40b3ae0 0x7f46b413ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46b3e00640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40cf290 0x7f46b41447b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00b970 tx=0x7f46ac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46abe00640 1 -- 172.21.15.141:0/3409001496 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00cb40 con 0x7f46b40cf290 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46abe00640 1 -- 172.21.15.141:0/3409001496 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac00ccc0 con 0x7f46b40cf290 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.076+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b40ce230 con 0x7f46b40cf290 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.077+0000 7f46abe00640 1 -- 172.21.15.141:0/3409001496 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac014680 con 0x7f46b40cf290 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.077+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b40ce7f0 con 0x7f46b40cf290 2024-01-16T15:11:59.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.078+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b40675d0 con 0x7f46b40cf290 2024-01-16T15:11:59.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.079+0000 7f46abe00640 1 -- 172.21.15.141:0/3409001496 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 32) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f46ac01d3b0 con 0x7f46b40cf290 2024-01-16T15:11:59.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.079+0000 7f46abe00640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f468c06c1f0 0x7f468c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:59.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.080+0000 7f46abe00640 1 -- 172.21.15.141:0/3409001496 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46ac088650 con 0x7f46b40cf290 2024-01-16T15:11:59.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.080+0000 7f46b3400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f468c06c1f0 0x7f468c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:59.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.081+0000 7f46b3400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f468c06c1f0 0x7f468c06e6b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f46a4036000 tx=0x7f46a4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:59.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.084+0000 7f46abe00640 1 -- 172.21.15.141:0/3409001496 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46ac054f50 con 0x7f46b40cf290 2024-01-16T15:11:59.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.304+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f46b4071170 con 0x7f468c06c1f0 2024-01-16T15:11:59.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.315+0000 7f46abe00640 1 -- 172.21.15.141:0/3409001496 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f46a4044470 con 0x7f468c06c1f0 2024-01-16T15:11:59.292 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597 2024-01-16T15:11:59.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.317+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f468c06c1f0 msgr2=0x7f468c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.317+0000 7f46ba400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f468c06c1f0 0x7f468c06e6b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f46a4036000 tx=0x7f46a4008030 comp rx=0 tx=0).stop 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.317+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40cf290 msgr2=0x7f46b41447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.317+0000 7f46ba400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40cf290 0x7f46b41447b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00b970 tx=0x7f46ac00be30 comp rx=0 tx=0).stop 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 shutdown_connections 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f468c06c1f0 0x7f468c06e6b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46b40cf290 0x7f46b41447b0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b40b3ae0 0x7f46b413ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 --2- 172.21.15.141:0/3409001496 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f46b40ae5a0 0x7f46b413e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 >> 172.21.15.141:0/3409001496 conn(0x7f46b406a800 msgr2=0x7f46b4160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 shutdown_connections 2024-01-16T15:11:59.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:11:59.318+0000 7f46ba400640 1 -- 172.21.15.141:0/3409001496 wait complete. 2024-01-16T15:11:59.312 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597 2024-01-16T15:11:59.313 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:11:59.313 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:11:59.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:11:59.364 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:12:06.497 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:06.506 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:12:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:12:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.5731 s, 73.7 MB/s 2024-01-16T15:12:14.061 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:12:14.061 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:14.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:14.116 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:12:25.462 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:12:25.463 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:12:25.463 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.2851 s, 95.1 MB/s 2024-01-16T15:12:25.464 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:12:25.464 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:25.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:25.514 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:12:36.648 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:36.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:37.186 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:12:37.187 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:12:37.187 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.5682 s, 92.8 MB/s 2024-01-16T15:12:37.189 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:12:37.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.428+0000 7f091a800640 1 Processor -- start 2024-01-16T15:12:37.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.428+0000 7f091a800640 1 -- start start 2024-01-16T15:12:37.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.429+0000 7f091a800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f09140c6cd0 0x7f09140cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.429+0000 7f091a800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09140d0870 con 0x7f09140d00c0 2024-01-16T15:12:37.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.429+0000 7f091a800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09140cb650 con 0x7f091415b330 2024-01-16T15:12:37.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.429+0000 7f091a800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09140cb930 con 0x7f09140c6cd0 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.430+0000 7f0919e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f09140d00c0 0x7f09140cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47934/0 (socket says 172.21.15.141:47934) 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.430+0000 7f0919e00640 1 -- 172.21.15.141:0/864115550 learned_addr learned my addr 172.21.15.141:0/864115550 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.430+0000 7f0918a00640 1 --2- 172.21.15.141:0/864115550 >> v2:172.21.15.188:3300/0 conn(0x7f09140c6cd0 0x7f09140cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.430+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227183130 0 0) 0x7f09140d0870 con 0x7f09140d00c0 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.430+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f4003520 con 0x7f09140d00c0 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2388000218 0 0) 0x7f08f4003520 con 0x7f09140d00c0 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 >> v2:172.21.15.188:3300/0 conn(0x7f09140c6cd0 msgr2=0x7f09140cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 --2- 172.21.15.141:0/864115550 >> v2:172.21.15.188:3300/0 conn(0x7f09140c6cd0 0x7f09140cfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 >> v1:172.21.15.188:6789/0 conn(0x7f091415b330 legacy=0x7f09140c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09140cbc10 con 0x7f09140d00c0 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f090c0019f0 con 0x7f09140d00c0 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 >> v1:172.21.15.141:6789/0 conn(0x7f09140d00c0 legacy=0x7f09140cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4003970 0x7f08f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08f40047c0 0x7f08f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08f4005180 0x7f08f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09140cb930 con 0x7f08f4005180 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f400c020 con 0x7f08f40047c0 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f400c1e0 con 0x7f08f4003970 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.431+0000 7f0919e00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08f4005180 0x7f08f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0918a00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08f40047c0 0x7f08f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0919400640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4003970 0x7f08f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0919e00640 1 -- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4003970 msgr2=0x7f08f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0919e00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4003970 0x7f08f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0919e00640 1 -- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08f40047c0 msgr2=0x7f08f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0919e00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08f40047c0 0x7f08f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0919e00640 1 -- 172.21.15.141:0/864115550 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4003520 con 0x7f08f4005180 2024-01-16T15:12:37.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0919400640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4003970 0x7f08f4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:37.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.432+0000 7f0918a00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08f40047c0 0x7f08f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.433+0000 7f0919e00640 1 -- 172.21.15.141:0/864115550 reap_dead start 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.433+0000 7f0919e00640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08f4005180 0x7f08f400b980 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f090c00a830 tx=0x7f08f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121dd2547f405e29 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.433+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c0072d0 con 0x7f08f4005180 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.433+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c02f070 con 0x7f08f4005180 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.433+0000 7f0913e00640 1 -- 172.21.15.141:0/864115550 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c007690 con 0x7f08f4005180 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.434+0000 7f091a800640 1 -- 172.21.15.141:0/864115550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08f4005180 msgr2=0x7f08f400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.434+0000 7f091a800640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08f4005180 0x7f08f400b980 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f090c00a830 tx=0x7f08f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.434+0000 7f091a800640 1 -- 172.21.15.141:0/864115550 shutdown_connections 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.434+0000 7f091a800640 1 --2- 172.21.15.141:0/864115550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08f4005180 0x7f08f400b980 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.434+0000 7f091a800640 1 -- 172.21.15.141:0/864115550 >> 172.21.15.141:0/864115550 conn(0x7f09140d0e80 msgr2=0x7f09140d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.434+0000 7f091a800640 1 -- 172.21.15.141:0/864115550 shutdown_connections 2024-01-16T15:12:37.412 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.434+0000 7f091a800640 1 -- 172.21.15.141:0/864115550 wait complete. 2024-01-16T15:12:37.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.436+0000 7f091a800640 1 Processor -- start 2024-01-16T15:12:37.413 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.436+0000 7f091a800640 1 -- start start 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.437+0000 7f091a800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4005180 0x7f0914151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.437+0000 7f091a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09140caa40 0x7f0914150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.437+0000 7f091a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09140d1260 0x7f09140d16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.437+0000 7f091a800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09140d1c00 con 0x7f09140caa40 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.437+0000 7f091a800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09140b3bc0 con 0x7f09140d1260 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.437+0000 7f091a800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09140caec0 con 0x7f08f4005180 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09140caa40 0x7f0914150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09140caa40 0x7f0914150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60364/0 (socket says 172.21.15.141:60364) 2024-01-16T15:12:37.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 -- 172.21.15.141:0/2657314553 learned_addr learned my addr 172.21.15.141:0/2657314553 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0919400640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4005180 0x7f0914151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0919e00640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09140d1260 0x7f09140d16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 -- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4005180 msgr2=0x7f0914151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4005180 0x7f0914151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 -- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09140d1260 msgr2=0x7f09140d16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09140d1260 0x7f09140d16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.438+0000 7f0918a00640 1 -- 172.21.15.141:0/2657314553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0914151a60 con 0x7f09140caa40 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.439+0000 7f0918a00640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09140caa40 0x7f0914150b20 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f08fc011540 tx=0x7f08fc011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:37.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.439+0000 7f0911600640 1 -- 172.21.15.141:0/2657314553 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc015870 con 0x7f09140caa40 2024-01-16T15:12:37.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.439+0000 7f0911600640 1 -- 172.21.15.141:0/2657314553 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08fc0159d0 con 0x7f09140caa40 2024-01-16T15:12:37.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.439+0000 7f0911600640 1 -- 172.21.15.141:0/2657314553 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc015d40 con 0x7f09140caa40 2024-01-16T15:12:37.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.439+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09140b3e10 con 0x7f09140caa40 2024-01-16T15:12:37.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.439+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09140b43d0 con 0x7f09140caa40 2024-01-16T15:12:37.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.440+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091415ae90 con 0x7f09140caa40 2024-01-16T15:12:37.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.441+0000 7f0911600640 1 -- 172.21.15.141:0/2657314553 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 33) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f08fc00f460 con 0x7f09140caa40 2024-01-16T15:12:37.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.442+0000 7f0911600640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f08ec06c300 0x7f08ec06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.442+0000 7f0911600640 1 -- 172.21.15.141:0/2657314553 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08fc089230 con 0x7f09140caa40 2024-01-16T15:12:37.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.443+0000 7f0919400640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f08ec06c300 0x7f08ec06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.444+0000 7f0919400640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f08ec06c300 0x7f08ec06e7c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0904000c00 tx=0x7f0904009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:37.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.447+0000 7f0911600640 1 -- 172.21.15.141:0/2657314553 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08fc0557e0 con 0x7f09140caa40 2024-01-16T15:12:37.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.670+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0914067b60 con 0x7f08ec06c300 2024-01-16T15:12:37.681 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.705+0000 7f0911600640 1 -- 172.21.15.141:0/2657314553 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0914067b60 con 0x7f08ec06c300 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.707+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f08ec06c300 msgr2=0x7f08ec06e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.707+0000 7f091a800640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f08ec06c300 0x7f08ec06e7c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0904000c00 tx=0x7f0904009040 comp rx=0 tx=0).stop 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09140caa40 msgr2=0x7f0914150b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09140caa40 0x7f0914150b20 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f08fc011540 tx=0x7f08fc011a00 comp rx=0 tx=0).stop 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 shutdown_connections 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f08ec06c300 0x7f08ec06e7c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09140d1260 0x7f09140d16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f09140caa40 0x7f0914150b20 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 --2- 172.21.15.141:0/2657314553 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f08f4005180 0x7f0914151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 >> 172.21.15.141:0/2657314553 conn(0x7f09140d0e80 msgr2=0x7f09140cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 shutdown_connections 2024-01-16T15:12:37.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.708+0000 7f091a800640 1 -- 172.21.15.141:0/2657314553 wait complete. 2024-01-16T15:12:37.703 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:12:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.939+0000 7fdbda200640 1 Processor -- start 2024-01-16T15:12:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.939+0000 7fdbda200640 1 -- start start 2024-01-16T15:12:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.941+0000 7fdbda200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdbd4070a30 0x7fdbd415bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.941+0000 7fdbda200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd415f4e0 con 0x7fdbd40ccb50 2024-01-16T15:12:37.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.941+0000 7fdbda200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd4070ea0 con 0x7fdbd40adf20 2024-01-16T15:12:37.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.941+0000 7fdbda200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd415c450 con 0x7fdbd4070a30 2024-01-16T15:12:37.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.941+0000 7fdbd3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdbd40ccb50 0x7fdbd4143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47940/0 (socket says 172.21.15.141:47940) 2024-01-16T15:12:37.918 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.941+0000 7fdbd3e00640 1 -- 172.21.15.141:0/1988931864 learned_addr learned my addr 172.21.15.141:0/1988931864 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.941+0000 7fdbd9800640 1 --2- 172.21.15.141:0/1988931864 >> v2:172.21.15.188:3300/0 conn(0x7fdbd4070a30 0x7fdbd415bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442599422 0 0) 0x7fdbd415f4e0 con 0x7fdbd40ccb50 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb4003520 con 0x7fdbd40ccb50 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3568892457 0 0) 0x7fdbb4003520 con 0x7fdbd40ccb50 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 >> v2:172.21.15.188:3300/0 conn(0x7fdbd4070a30 msgr2=0x7fdbd415bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 --2- 172.21.15.141:0/1988931864 >> v2:172.21.15.188:3300/0 conn(0x7fdbd4070a30 0x7fdbd415bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 >> v1:172.21.15.188:6789/0 conn(0x7fdbd40adf20 legacy=0x7fdbd40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd4071130 con 0x7fdbd40ccb50 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fdbc4001670 con 0x7fdbd40ccb50 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 >> v1:172.21.15.141:6789/0 conn(0x7fdbd40ccb50 legacy=0x7fdbd4143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb4003970 0x7fdbb4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbb40047c0 0x7fdbb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbb4005180 0x7fdbb400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd415c450 con 0x7fdbb40047c0 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb400c120 con 0x7fdbb4003970 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb400c2e0 con 0x7fdbb4005180 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.942+0000 7fdbd3e00640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbb40047c0 0x7fdbb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd9800640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbb4005180 0x7fdbb400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd8e00640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb4003970 0x7fdbb4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd3e00640 1 -- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbb4005180 msgr2=0x7fdbb400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd3e00640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbb4005180 0x7fdbb400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd3e00640 1 -- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb4003970 msgr2=0x7fdbb4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd3e00640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb4003970 0x7fdbb4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd3e00640 1 -- 172.21.15.141:0/1988931864 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb4003520 con 0x7fdbb40047c0 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd9800640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbb4005180 0x7fdbb400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:37.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd8e00640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb4003970 0x7fdbb4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:37.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.943+0000 7fdbd9800640 1 -- 172.21.15.141:0/1988931864 reap_dead start 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.944+0000 7fdbd3e00640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbb40047c0 0x7fdbb4004c40 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4007fd0 tx=0x7fdbb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9555588156f83ea8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.944+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4005ad0 con 0x7fdbb40047c0 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.944+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc4005c30 con 0x7fdbb40047c0 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.944+0000 7fdbd3400640 1 -- 172.21.15.141:0/1988931864 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc402c9b0 con 0x7fdbb40047c0 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.945+0000 7fdbda200640 1 -- 172.21.15.141:0/1988931864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbb40047c0 msgr2=0x7fdbb4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.945+0000 7fdbda200640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbb40047c0 0x7fdbb4004c40 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4007fd0 tx=0x7fdbb40057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.945+0000 7fdbda200640 1 -- 172.21.15.141:0/1988931864 shutdown_connections 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.945+0000 7fdbda200640 1 --2- 172.21.15.141:0/1988931864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbb40047c0 0x7fdbb4004c40 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.945+0000 7fdbda200640 1 -- 172.21.15.141:0/1988931864 >> 172.21.15.141:0/1988931864 conn(0x7fdbd406a7d0 msgr2=0x7fdbd406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.945+0000 7fdbda200640 1 -- 172.21.15.141:0/1988931864 shutdown_connections 2024-01-16T15:12:37.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.945+0000 7fdbda200640 1 -- 172.21.15.141:0/1988931864 wait complete. 2024-01-16T15:12:37.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.947+0000 7fdbda200640 1 Processor -- start 2024-01-16T15:12:37.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.947+0000 7fdbda200640 1 -- start start 2024-01-16T15:12:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbda200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb40047c0 0x7fdbd40ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbda200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbd415acd0 0x7fdbd413a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbda200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbd413a690 0x7fdbd413b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbda200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd413b1d0 con 0x7fdbd413a690 2024-01-16T15:12:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbda200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd413ac10 con 0x7fdbb40047c0 2024-01-16T15:12:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbda200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd413aef0 con 0x7fdbd415acd0 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbd9800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbd413a690 0x7fdbd413b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbd9800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbd413a690 0x7fdbd413b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60388/0 (socket says 172.21.15.141:60388) 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.948+0000 7fdbd9800640 1 -- 172.21.15.141:0/833344089 learned_addr learned my addr 172.21.15.141:0/833344089 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd8e00640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb40047c0 0x7fdbd40ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd3e00640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbd415acd0 0x7fdbd413a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd9800640 1 -- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbd415acd0 msgr2=0x7fdbd413a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd9800640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbd415acd0 0x7fdbd413a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd9800640 1 -- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb40047c0 msgr2=0x7fdbd40ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd9800640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb40047c0 0x7fdbd40ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd9800640 1 -- 172.21.15.141:0/833344089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd4141b80 con 0x7fdbd413a690 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd3e00640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbd415acd0 0x7fdbd413a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd9800640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbd413a690 0x7fdbd413b5c0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40ae7d0 tx=0x7fdbcc00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd0c00640 1 -- 172.21.15.141:0/833344089 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc016620 con 0x7fdbd413a690 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd0c00640 1 -- 172.21.15.141:0/833344089 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc016780 con 0x7fdbd413a690 2024-01-16T15:12:37.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbd0c00640 1 -- 172.21.15.141:0/833344089 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc016b10 con 0x7fdbd413a690 2024-01-16T15:12:37.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.949+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd4141e60 con 0x7fdbd413a690 2024-01-16T15:12:37.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.950+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd40aee40 con 0x7fdbd413a690 2024-01-16T15:12:37.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.951+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd4067450 con 0x7fdbd413a690 2024-01-16T15:12:37.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.952+0000 7fdbd0c00640 1 -- 172.21.15.141:0/833344089 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 33) v1 ==== 90967+0+0 (secure 0 0 0) 0x7fdbcc016c70 con 0x7fdbd413a690 2024-01-16T15:12:37.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.953+0000 7fdbd0c00640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbac06c280 0x7fdbac06e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:37.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.953+0000 7fdbd0c00640 1 -- 172.21.15.141:0/833344089 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbcc089650 con 0x7fdbd413a690 2024-01-16T15:12:37.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.953+0000 7fdbd8e00640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbac06c280 0x7fdbac06e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:37.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.954+0000 7fdbd8e00640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbac06c280 0x7fdbac06e740 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc000c00 tx=0x7fdbbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:37.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:37.957+0000 7fdbd0c00640 1 -- 172.21.15.141:0/833344089 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbcc055f70 con 0x7fdbd413a690 2024-01-16T15:12:38.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.176+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbd415ee90 con 0x7fdbac06c280 2024-01-16T15:12:38.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.191+0000 7fdbd0c00640 1 -- 172.21.15.141:0/833344089 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fdbbc00d050 con 0x7fdbac06c280 2024-01-16T15:12:38.167 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597 2024-01-16T15:12:38.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbac06c280 msgr2=0x7fdbac06e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:38.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbac06c280 0x7fdbac06e740 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc000c00 tx=0x7fdbbc009040 comp rx=0 tx=0).stop 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbd413a690 msgr2=0x7fdbd413b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbd413a690 0x7fdbd413b5c0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40ae7d0 tx=0x7fdbcc00f520 comp rx=0 tx=0).stop 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 shutdown_connections 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbac06c280 0x7fdbac06e740 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbd413a690 0x7fdbd413b5c0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbd415acd0 0x7fdbd413a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.193+0000 7fdbda200640 1 --2- 172.21.15.141:0/833344089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbb40047c0 0x7fdbd40ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.194+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 >> 172.21.15.141:0/833344089 conn(0x7fdbd406a7d0 msgr2=0x7fdbd40acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.194+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 shutdown_connections 2024-01-16T15:12:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:38.194+0000 7fdbda200640 1 -- 172.21.15.141:0/833344089 wait complete. 2024-01-16T15:12:38.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:38.189 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597) 2024-01-16T15:12:38.249 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597 2024-01-16T15:12:38.249 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:12:38.250 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:12:38.250 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:12:40.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:40.251 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597) 2024-01-16T15:12:40.317 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/d6eabf46-18bc-4093-8305-1e4d9e7f6597 2024-01-16T15:12:40.317 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:12:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:12:40.318 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:12:40.319 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51671 2024-01-16T15:12:40.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.561+0000 7fb981a00640 1 Processor -- start 2024-01-16T15:12:40.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.561+0000 7fb981a00640 1 -- start start 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb981a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb97c0ad760 0x7fb97c15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb981a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb97c15b930 0x7fb97c0cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb981a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97c15ea30 con 0x7fb97c0cd4a0 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb981a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97c0b4160 con 0x7fb97c0ad760 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb981a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97c0adc40 con 0x7fb97c15b930 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb973200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb97c0ad760 0x7fb97c15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb973200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb97c0ad760 0x7fb97c15b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60394/0 (socket says 172.21.15.141:60394) 2024-01-16T15:12:40.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb973200640 1 -- 172.21.15.141:0/528809783 learned_addr learned my addr 172.21.15.141:0/528809783 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:40.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb973200640 1 -- 172.21.15.141:0/528809783 >> v2:172.21.15.188:3300/0 conn(0x7fb97c15b930 msgr2=0x7fb97c0cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:40.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb973200640 1 --2- 172.21.15.141:0/528809783 >> v2:172.21.15.188:3300/0 conn(0x7fb97c15b930 0x7fb97c0cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb973200640 1 -- 172.21.15.141:0/528809783 >> v1:172.21.15.188:6789/0 conn(0x7fb97c0cd4a0 legacy=0x7fb97c0b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:40.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.563+0000 7fb973200640 1 -- 172.21.15.141:0/528809783 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c0adf20 con 0x7fb97c0ad760 2024-01-16T15:12:40.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.564+0000 7fb973200640 1 --2- 172.21.15.141:0/528809783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c15b3f0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0acfc0 tx=0x7fb96402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8db76439d18cac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb97b400640 1 -- 172.21.15.141:0/528809783 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640317e0 con 0x7fb97c0ad760 2024-01-16T15:12:40.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb97b400640 1 -- 172.21.15.141:0/528809783 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb964031940 con 0x7fb97c0ad760 2024-01-16T15:12:40.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb97b400640 1 -- 172.21.15.141:0/528809783 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964031cd0 con 0x7fb97c0ad760 2024-01-16T15:12:40.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 -- 172.21.15.141:0/528809783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 msgr2=0x7fb97c15b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 --2- 172.21.15.141:0/528809783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c15b3f0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0acfc0 tx=0x7fb96402b880 comp rx=0 tx=0).stop 2024-01-16T15:12:40.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 -- 172.21.15.141:0/528809783 shutdown_connections 2024-01-16T15:12:40.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 --2- 172.21.15.141:0/528809783 >> v2:172.21.15.188:3300/0 conn(0x7fb97c15b930 0x7fb97c0cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 --2- 172.21.15.141:0/528809783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c15b3f0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 -- 172.21.15.141:0/528809783 >> 172.21.15.141:0/528809783 conn(0x7fb97c06a800 msgr2=0x7fb97c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:40.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 -- 172.21.15.141:0/528809783 shutdown_connections 2024-01-16T15:12:40.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.565+0000 7fb981a00640 1 -- 172.21.15.141:0/528809783 wait complete. 2024-01-16T15:12:40.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.567+0000 7fb981a00640 1 Processor -- start 2024-01-16T15:12:40.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.567+0000 7fb981a00640 1 -- start start 2024-01-16T15:12:40.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.568+0000 7fb981a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.568+0000 7fb981a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb97c0cd4a0 0x7fb97c159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.568+0000 7fb981a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb97c15b930 0x7fb97c1595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.568+0000 7fb981a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c1f3eb0 con 0x7fb97c0ad760 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.568+0000 7fb981a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c1f3a20 con 0x7fb97c15b930 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.568+0000 7fb981a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c1f2e90 con 0x7fb97c0cd4a0 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.568+0000 7fb97be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c071a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60402/0 (socket says 172.21.15.141:60402) 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 -- 172.21.15.141:0/1948943177 learned_addr learned my addr 172.21.15.141:0/1948943177 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb973200640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb97c0cd4a0 0x7fb97c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 -- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb97c0cd4a0 msgr2=0x7fb97c159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb97c0cd4a0 0x7fb97c159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb981000640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb97c15b930 0x7fb97c1595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 -- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb97c15b930 msgr2=0x7fb97c1595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb97c15b930 0x7fb97c1595a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 -- 172.21.15.141:0/1948943177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9640094d0 con 0x7fb97c0ad760 2024-01-16T15:12:40.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.569+0000 7fb97be00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c071a40 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00c7a0 tx=0x7fb96c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.570+0000 7fb978c00640 1 -- 172.21.15.141:0/1948943177 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c003d80 con 0x7fb97c0ad760 2024-01-16T15:12:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.570+0000 7fb978c00640 1 -- 172.21.15.141:0/1948943177 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c005070 con 0x7fb97c0ad760 2024-01-16T15:12:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.570+0000 7fb978c00640 1 -- 172.21.15.141:0/1948943177 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c005420 con 0x7fb97c0ad760 2024-01-16T15:12:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.570+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97c1f30f0 con 0x7fb97c0ad760 2024-01-16T15:12:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.570+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97c1f36b0 con 0x7fb97c0ad760 2024-01-16T15:12:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.571+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97c067450 con 0x7fb97c0ad760 2024-01-16T15:12:40.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.572+0000 7fb978c00640 1 -- 172.21.15.141:0/1948943177 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 33) v1 ==== 90967+0+0 (secure 0 0 0) 0x7fb96c005580 con 0x7fb97c0ad760 2024-01-16T15:12:40.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.573+0000 7fb978c00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb95406c280 0x7fb95406e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.573+0000 7fb978c00640 1 -- 172.21.15.141:0/1948943177 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb96c087ef0 con 0x7fb97c0ad760 2024-01-16T15:12:40.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.574+0000 7fb973200640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb95406c280 0x7fb95406e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.575+0000 7fb973200640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb95406c280 0x7fb95406e740 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb964036000 tx=0x7fb96403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.578+0000 7fb978c00640 1 -- 172.21.15.141:0/1948943177 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb96c0544a0 con 0x7fb97c0ad760 2024-01-16T15:12:40.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.809+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51671", "target": ["mon-mgr", ""]}) v1 -- 0x7fb97c0ae4e0 con 0x7fb95406c280 2024-01-16T15:12:40.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.973+0000 7fb978c00640 1 -- 172.21.15.141:0/1948943177 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb964031bb0 con 0x7fb95406c280 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.976+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb95406c280 msgr2=0x7fb95406e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.976+0000 7fb981a00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb95406c280 0x7fb95406e740 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb964036000 tx=0x7fb96403d040 comp rx=0 tx=0).stop 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.976+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 msgr2=0x7fb97c071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.976+0000 7fb981a00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c071a40 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00c7a0 tx=0x7fb96c00cc60 comp rx=0 tx=0).stop 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 shutdown_connections 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb95406c280 0x7fb95406e740 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb97c15b930 0x7fb97c1595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb97c0cd4a0 0x7fb97c159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 --2- 172.21.15.141:0/1948943177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb97c0ad760 0x7fb97c071a40 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 >> 172.21.15.141:0/1948943177 conn(0x7fb97c06a800 msgr2=0x7fb97c15a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 shutdown_connections 2024-01-16T15:12:40.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:40.977+0000 7fb981a00640 1 -- 172.21.15.141:0/1948943177 wait complete. 2024-01-16T15:12:40.959 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:40.983+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.959 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:40.983+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.959 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:40.983+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.959 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:40.983+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.959 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:40.983+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.973 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51672 2024-01-16T15:12:40.981 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:41.005+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.981 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:41.005+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.981 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:41.005+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.981 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:41.005+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:40.982 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:41.005+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:41.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.216+0000 7fb099a00640 1 Processor -- start 2024-01-16T15:12:41.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.216+0000 7fb099a00640 1 -- start start 2024-01-16T15:12:41.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.218+0000 7fb099a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb09406a3e0 0x7fb09406a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.218+0000 7fb099a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0941472c0 con 0x7fb0940cca60 2024-01-16T15:12:41.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.218+0000 7fb099a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940cd790 con 0x7fb094146ee0 2024-01-16T15:12:41.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.218+0000 7fb099a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09406add0 con 0x7fb09406a3e0 2024-01-16T15:12:41.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.218+0000 7fb093400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb0940cca60 0x7fb0940cce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47944/0 (socket says 172.21.15.141:47944) 2024-01-16T15:12:41.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.218+0000 7fb093400640 1 -- 172.21.15.141:0/2526174605 learned_addr learned my addr 172.21.15.141:0/2526174605 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:12:41.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb089e00640 1 --2- 172.21.15.141:0/2526174605 >> v2:172.21.15.188:3300/0 conn(0x7fb09406a3e0 0x7fb09406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401263561 0 0) 0x7fb0941472c0 con 0x7fb0940cca60 2024-01-16T15:12:41.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb074003520 con 0x7fb0940cca60 2024-01-16T15:12:41.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866677563 0 0) 0x7fb074003520 con 0x7fb0940cca60 2024-01-16T15:12:41.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 >> v2:172.21.15.188:3300/0 conn(0x7fb09406a3e0 msgr2=0x7fb09406a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 --2- 172.21.15.141:0/2526174605 >> v2:172.21.15.188:3300/0 conn(0x7fb09406a3e0 0x7fb09406a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 >> v1:172.21.15.188:6789/0 conn(0x7fb094146ee0 legacy=0x7fb094071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094071210 con 0x7fb0940cca60 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb08c001670 con 0x7fb0940cca60 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 >> v1:172.21.15.141:6789/0 conn(0x7fb0940cca60 legacy=0x7fb0940cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740039c0 0x7fb074003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.219+0000 7fb092000640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0740047c0 0x7fb074004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb092000640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb074005180 0x7fb07400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09406add0 con 0x7fb0740047c0 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07400c080 con 0x7fb0740039c0 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07400c240 con 0x7fb074005180 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb089e00640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0740047c0 0x7fb074004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb089e00640 1 -- 172.21.15.141:0/2526174605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb074005180 msgr2=0x7fb07400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb089e00640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb074005180 0x7fb07400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb089e00640 1 -- 172.21.15.141:0/2526174605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740039c0 msgr2=0x7fb074003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb089e00640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740039c0 0x7fb074003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb089e00640 1 -- 172.21.15.141:0/2526174605 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074003520 con 0x7fb0740047c0 2024-01-16T15:12:41.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.220+0000 7fb093400640 1 -- 172.21.15.141:0/2526174605 reap_dead start 2024-01-16T15:12:41.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.221+0000 7fb089e00640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0740047c0 0x7fb074004c40 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009ac0 tx=0x7fb0740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90f4efc73db6937 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.221+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c02c6b0 con 0x7fb0740047c0 2024-01-16T15:12:41.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c02c810 con 0x7fb0740047c0 2024-01-16T15:12:41.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb092000640 1 -- 172.21.15.141:0/2526174605 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c02cba0 con 0x7fb0740047c0 2024-01-16T15:12:41.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb099a00640 1 -- 172.21.15.141:0/2526174605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0740047c0 msgr2=0x7fb074004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb099a00640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0740047c0 0x7fb074004c40 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009ac0 tx=0x7fb0740057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb099a00640 1 -- 172.21.15.141:0/2526174605 shutdown_connections 2024-01-16T15:12:41.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb099a00640 1 --2- 172.21.15.141:0/2526174605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0740047c0 0x7fb074004c40 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb099a00640 1 -- 172.21.15.141:0/2526174605 >> 172.21.15.141:0/2526174605 conn(0x7fb0940cff90 msgr2=0x7fb0940d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:41.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb099a00640 1 -- 172.21.15.141:0/2526174605 shutdown_connections 2024-01-16T15:12:41.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.222+0000 7fb099a00640 1 -- 172.21.15.141:0/2526174605 wait complete. 2024-01-16T15:12:41.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.224+0000 7fb099a00640 1 Processor -- start 2024-01-16T15:12:41.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.224+0000 7fb099a00640 1 -- start start 2024-01-16T15:12:41.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.225+0000 7fb099a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740047c0 0x7fb094146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.225+0000 7fb099a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb09413af80 0x7fb094140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.225+0000 7fb099a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb09413b400 0x7fb094140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.225+0000 7fb099a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941438a0 con 0x7fb09413b400 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.225+0000 7fb099a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094145b70 con 0x7fb0740047c0 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.225+0000 7fb099a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb094143af0 con 0x7fb09413af80 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb09413b400 0x7fb094140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb09413b400 0x7fb094140970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60430/0 (socket says 172.21.15.141:60430) 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 -- 172.21.15.141:0/3765529438 learned_addr learned my addr 172.21.15.141:0/3765529438 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb089e00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb09413af80 0x7fb094140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb092a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740047c0 0x7fb094146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 -- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb09413af80 msgr2=0x7fb094140430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb09413af80 0x7fb094140430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 -- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740047c0 msgr2=0x7fb094146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740047c0 0x7fb094146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 -- 172.21.15.141:0/3765529438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094143dd0 con 0x7fb09413b400 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.226+0000 7fb093400640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb09413b400 0x7fb094140970 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fb0940ce300 tx=0x7fb08c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.227+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c0019f0 con 0x7fb09413b400 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.227+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08c0074e0 con 0x7fb09413b400 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.227+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c004a20 con 0x7fb09413b400 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.227+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0941440b0 con 0x7fb09413b400 2024-01-16T15:12:41.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.227+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094146540 con 0x7fb09413b400 2024-01-16T15:12:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.228+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094145ce0 con 0x7fb09413b400 2024-01-16T15:12:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.229+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 33) v1 ==== 90967+0+0 (secure 0 0 0) 0x7fb08c004b80 con 0x7fb09413b400 2024-01-16T15:12:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.230+0000 7fb08b400640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb06c06c280 0x7fb06c06e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.230+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb08c087cf0 con 0x7fb09413b400 2024-01-16T15:12:41.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.231+0000 7fb092a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb06c06c280 0x7fb06c06e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.232+0000 7fb092a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb06c06c280 0x7fb06c06e740 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb084007fd0 tx=0x7fb084009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.234+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb08c0542a0 con 0x7fb09413b400 2024-01-16T15:12:41.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.460+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51672", "target": ["mon-mgr", ""]}) v1 -- 0x7fb094071ab0 con 0x7fb06c06c280 2024-01-16T15:12:41.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.558+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 34) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fb08c054670 con 0x7fb09413b400 2024-01-16T15:12:41.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.623+0000 7fb08b400640 1 -- 172.21.15.141:0/3765529438 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb084001ab0 con 0x7fb06c06c280 2024-01-16T15:12:41.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb06c06c280 msgr2=0x7fb06c06e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb06c06c280 0x7fb06c06e740 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb084007fd0 tx=0x7fb084009040 comp rx=0 tx=0).stop 2024-01-16T15:12:41.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb09413b400 msgr2=0x7fb094140970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb09413b400 0x7fb094140970 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fb0940ce300 tx=0x7fb08c00c940 comp rx=0 tx=0).stop 2024-01-16T15:12:41.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 shutdown_connections 2024-01-16T15:12:41.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb06c06c280 0x7fb06c06e740 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb09413b400 0x7fb094140970 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb09413af80 0x7fb094140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 --2- 172.21.15.141:0/3765529438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0740047c0 0x7fb094146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 >> 172.21.15.141:0/3765529438 conn(0x7fb0940cff90 msgr2=0x7fb0940ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:41.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 shutdown_connections 2024-01-16T15:12:41.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.626+0000 7fb099a00640 1 -- 172.21.15.141:0/3765529438 wait complete. 2024-01-16T15:12:41.626 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51673 2024-01-16T15:12:41.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.868+0000 7fca36c00640 1 Processor -- start 2024-01-16T15:12:41.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.868+0000 7fca36c00640 1 -- start start 2024-01-16T15:12:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.869+0000 7fca36c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fca30150760 0x7fca30072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.869+0000 7fca36c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca300b3710 con 0x7fca300b32e0 2024-01-16T15:12:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca36c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca30071190 con 0x7fca3015d300 2024-01-16T15:12:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca36c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca30071470 con 0x7fca30150760 2024-01-16T15:12:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca34c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fca300b32e0 0x7fca300ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47948/0 (socket says 172.21.15.141:47948) 2024-01-16T15:12:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca34c00640 1 -- 172.21.15.141:0/557958105 learned_addr learned my addr 172.21.15.141:0/557958105 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430099952 0 0) 0x7fca300b3710 con 0x7fca300b32e0 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca2fe00640 1 --2- 172.21.15.141:0/557958105 >> v2:172.21.15.188:3300/0 conn(0x7fca30150760 0x7fca30072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca10003440 con 0x7fca300b32e0 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.870+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2649382332 0 0) 0x7fca10003440 con 0x7fca300b32e0 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 >> v2:172.21.15.188:3300/0 conn(0x7fca30150760 msgr2=0x7fca30072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 --2- 172.21.15.141:0/557958105 >> v2:172.21.15.188:3300/0 conn(0x7fca30150760 0x7fca30072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 >> v1:172.21.15.188:6789/0 conn(0x7fca3015d300 legacy=0x7fca300b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30071750 con 0x7fca300b32e0 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fca28001670 con 0x7fca300b32e0 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 >> v1:172.21.15.141:6789/0 conn(0x7fca300b32e0 legacy=0x7fca300ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca10003a60 0x7fca10003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca10008850 0x7fca1000ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca1000b480 0x7fca1000d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca30071470 con 0x7fca1000b480 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1000e220 con 0x7fca10003a60 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1000e3e0 con 0x7fca10008850 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca34c00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca1000b480 0x7fca1000d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca2fe00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca10003a60 0x7fca10003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.871+0000 7fca34c00640 1 -- 172.21.15.141:0/557958105 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca10008850 msgr2=0x7fca1000ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.872+0000 7fca34c00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca10008850 0x7fca1000ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.872+0000 7fca34c00640 1 -- 172.21.15.141:0/557958105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca10003a60 msgr2=0x7fca10003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.872+0000 7fca34c00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca10003a60 0x7fca10003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.872+0000 7fca34c00640 1 -- 172.21.15.141:0/557958105 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10003440 con 0x7fca1000b480 2024-01-16T15:12:41.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.872+0000 7fca34c00640 1 -- 172.21.15.141:0/557958105 reap_dead start 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca34c00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca1000b480 0x7fca1000d940 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c190 tx=0x7fca100036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683b27d8e67f42e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca28005ad0 con 0x7fca1000b480 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca28005c70 con 0x7fca1000b480 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca2ea00640 1 -- 172.21.15.141:0/557958105 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2802c8a0 con 0x7fca1000b480 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca36c00640 1 -- 172.21.15.141:0/557958105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca1000b480 msgr2=0x7fca1000d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca36c00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca1000b480 0x7fca1000d940 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c190 tx=0x7fca100036f0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca36c00640 1 -- 172.21.15.141:0/557958105 shutdown_connections 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.873+0000 7fca36c00640 1 --2- 172.21.15.141:0/557958105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca1000b480 0x7fca1000d940 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.874+0000 7fca36c00640 1 -- 172.21.15.141:0/557958105 >> 172.21.15.141:0/557958105 conn(0x7fca3006a800 msgr2=0x7fca3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.874+0000 7fca36c00640 1 -- 172.21.15.141:0/557958105 shutdown_connections 2024-01-16T15:12:41.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.874+0000 7fca36c00640 1 -- 172.21.15.141:0/557958105 wait complete. 2024-01-16T15:12:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.875+0000 7fca36c00640 1 Processor -- start 2024-01-16T15:12:41.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.876+0000 7fca36c00640 1 -- start start 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca36c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca1000b480 0x7fca3014a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca36c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca300ce0e0 0x7fca3014cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca36c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca3014d0c0 0x7fca301e34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca36c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca300cd040 con 0x7fca300ce0e0 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca36c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca300cd320 con 0x7fca1000b480 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca36c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca300cd600 con 0x7fca3014d0c0 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca300ce0e0 0x7fca3014cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca300ce0e0 0x7fca3014cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60446/0 (socket says 172.21.15.141:60446) 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 -- 172.21.15.141:0/1355533035 learned_addr learned my addr 172.21.15.141:0/1355533035 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 -- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca3014d0c0 msgr2=0x7fca301e34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca3014d0c0 0x7fca301e34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2fe00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca1000b480 0x7fca3014a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 -- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca1000b480 msgr2=0x7fca3014a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca1000b480 0x7fca3014a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.877+0000 7fca2f400640 1 -- 172.21.15.141:0/1355533035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca300cd8e0 con 0x7fca300ce0e0 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.878+0000 7fca2f400640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca300ce0e0 0x7fca3014cb80 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fca1800c7a0 tx=0x7fca1800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.878+0000 7fca27e00640 1 -- 172.21.15.141:0/1355533035 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18003b70 con 0x7fca300ce0e0 2024-01-16T15:12:41.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.878+0000 7fca27e00640 1 -- 172.21.15.141:0/1355533035 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca18003cf0 con 0x7fca300ce0e0 2024-01-16T15:12:41.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.878+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca300cdbc0 con 0x7fca300ce0e0 2024-01-16T15:12:41.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.878+0000 7fca27e00640 1 -- 172.21.15.141:0/1355533035 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1800aa90 con 0x7fca300ce0e0 2024-01-16T15:12:41.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.878+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca300ccbf0 con 0x7fca300ce0e0 2024-01-16T15:12:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.879+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca300cce10 con 0x7fca300ce0e0 2024-01-16T15:12:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.880+0000 7fca27e00640 1 -- 172.21.15.141:0/1355533035 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 34) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fca1800abf0 con 0x7fca300ce0e0 2024-01-16T15:12:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.881+0000 7fca27e00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0806c340 0x7fca0806e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.881+0000 7fca27e00640 1 -- 172.21.15.141:0/1355533035 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca18088860 con 0x7fca300ce0e0 2024-01-16T15:12:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.881+0000 7fca2fe00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0806c340 0x7fca0806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.882+0000 7fca2fe00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0806c340 0x7fca0806e800 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fca200098d0 tx=0x7fca20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:41.885+0000 7fca27e00640 1 -- 172.21.15.141:0/1355533035 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca180bee90 con 0x7fca300ce0e0 2024-01-16T15:12:42.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:42.109+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51673", "target": ["mon-mgr", ""]}) v1 -- 0x7fca300cd490 con 0x7fca0806c340 2024-01-16T15:12:43.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.725+0000 7fca27e00640 1 -- 172.21.15.141:0/1355533035 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fca300cd490 con 0x7fca0806c340 2024-01-16T15:12:43.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0806c340 msgr2=0x7fca0806e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:43.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0806c340 0x7fca0806e800 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fca200098d0 tx=0x7fca20008040 comp rx=0 tx=0).stop 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca300ce0e0 msgr2=0x7fca3014cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca300ce0e0 0x7fca3014cb80 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fca1800c7a0 tx=0x7fca1800cc60 comp rx=0 tx=0).stop 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 shutdown_connections 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fca0806c340 0x7fca0806e800 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fca3014d0c0 0x7fca301e34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca300ce0e0 0x7fca3014cb80 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 --2- 172.21.15.141:0/1355533035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca1000b480 0x7fca3014a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 >> 172.21.15.141:0/1355533035 conn(0x7fca3006a800 msgr2=0x7fca3014e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 shutdown_connections 2024-01-16T15:12:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.729+0000 7fca36c00640 1 -- 172.21.15.141:0/1355533035 wait complete. 2024-01-16T15:12:43.728 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:12:43.728 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51673 2024-01-16T15:12:43.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.963+0000 7f0085400640 1 Processor -- start 2024-01-16T15:12:43.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.964+0000 7f0085400640 1 -- start start 2024-01-16T15:12:43.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.965+0000 7f0085400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0080143ae0 0x7f00800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:43.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.965+0000 7f0085400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00800cf7d0 0x7f008015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:43.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.965+0000 7f0085400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800cfc30 con 0x7f008015e1e0 2024-01-16T15:12:43.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.965+0000 7f0085400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00800b1ee0 con 0x7f0080143ae0 2024-01-16T15:12:43.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.965+0000 7f0085400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00800b2780 con 0x7f00800cf7d0 2024-01-16T15:12:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.966+0000 7f007e800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0080143ae0 0x7f00800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.966+0000 7f007e800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0080143ae0 0x7f00800cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:38732/0 (socket says 172.21.15.141:38732) 2024-01-16T15:12:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.966+0000 7f007e800640 1 -- 172.21.15.141:0/3293401419 learned_addr learned my addr 172.21.15.141:0/3293401419 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.966+0000 7f0075c00640 1 --2- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00800cf7d0 0x7f008015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.967+0000 7f007e800640 1 -- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00800cf7d0 msgr2=0x7f008015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.967+0000 7f007e800640 1 --2- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00800cf7d0 0x7f008015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.967+0000 7f007e800640 1 -- 172.21.15.141:0/3293401419 >> v1:172.21.15.188:6789/0 conn(0x7f008015e1e0 legacy=0x7f008015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.967+0000 7f007e800640 1 -- 172.21.15.141:0/3293401419 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008015e990 con 0x7f0080143ae0 2024-01-16T15:12:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.968+0000 7f007e800640 1 --2- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 0x7f00800cf290 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0070009870 tx=0x7f007002db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=959a9aab5bd8dd13 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f007de00640 1 -- 172.21.15.141:0/3293401419 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007002e940 con 0x7f0080143ae0 2024-01-16T15:12:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f007de00640 1 -- 172.21.15.141:0/3293401419 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007002eaa0 con 0x7f0080143ae0 2024-01-16T15:12:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f007de00640 1 -- 172.21.15.141:0/3293401419 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007002ee30 con 0x7f0080143ae0 2024-01-16T15:12:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 -- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 msgr2=0x7f00800cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 --2- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 0x7f00800cf290 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0070009870 tx=0x7f007002db90 comp rx=0 tx=0).stop 2024-01-16T15:12:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 -- 172.21.15.141:0/3293401419 shutdown_connections 2024-01-16T15:12:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 --2- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f00800cf7d0 0x7f008015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 --2- 172.21.15.141:0/3293401419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 0x7f00800cf290 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 -- 172.21.15.141:0/3293401419 >> 172.21.15.141:0/3293401419 conn(0x7f008006a800 msgr2=0x7f008006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 -- 172.21.15.141:0/3293401419 shutdown_connections 2024-01-16T15:12:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.969+0000 7f0085400640 1 -- 172.21.15.141:0/3293401419 wait complete. 2024-01-16T15:12:43.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.971+0000 7f0085400640 1 Processor -- start 2024-01-16T15:12:43.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.971+0000 7f0085400640 1 -- start start 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.972+0000 7f0085400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00800cf7d0 0x7f008013e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.972+0000 7f0085400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 0x7f00801f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.972+0000 7f0085400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f008015e1e0 0x7f00800ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.972+0000 7f0085400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00800ad580 con 0x7f00800cf7d0 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.972+0000 7f0085400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00800ac970 con 0x7f0080143ae0 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.972+0000 7f0085400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00800acc50 con 0x7f008015e1e0 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.972+0000 7f007e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00800cf7d0 0x7f008013e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00800cf7d0 0x7f008013e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59890/0 (socket says 172.21.15.141:59890) 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 -- 172.21.15.141:0/4267038834 learned_addr learned my addr 172.21.15.141:0/4267038834 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f0075c00640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 0x7f00801f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007f200640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f008015e1e0 0x7f00800ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 -- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f008015e1e0 msgr2=0x7f00800ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f008015e1e0 0x7f00800ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 -- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 msgr2=0x7f00801f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 0x7f00801f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 -- 172.21.15.141:0/4267038834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700094d0 con 0x7f00800cf7d0 2024-01-16T15:12:43.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.973+0000 7f007e800640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00800cf7d0 0x7f008013e950 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f0070000c00 tx=0x7f00700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.974+0000 7f0077400640 1 -- 172.21.15.141:0/4267038834 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070040600 con 0x7f00800cf7d0 2024-01-16T15:12:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.974+0000 7f0077400640 1 -- 172.21.15.141:0/4267038834 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0070040760 con 0x7f00800cf7d0 2024-01-16T15:12:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.974+0000 7f0077400640 1 -- 172.21.15.141:0/4267038834 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070040b10 con 0x7f00800cf7d0 2024-01-16T15:12:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.974+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00800acf30 con 0x7f00800cf7d0 2024-01-16T15:12:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.974+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008013ee90 con 0x7f00800cf7d0 2024-01-16T15:12:43.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.975+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00800675d0 con 0x7f00800cf7d0 2024-01-16T15:12:43.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.976+0000 7f0077400640 1 -- 172.21.15.141:0/4267038834 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 35) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f0070040c70 con 0x7f00800cf7d0 2024-01-16T15:12:43.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.978+0000 7f0077400640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f005806c3a0 0x7f005806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:43.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.978+0000 7f0077400640 1 -- 172.21.15.141:0/4267038834 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00700aba90 con 0x7f00800cf7d0 2024-01-16T15:12:43.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.978+0000 7f0075c00640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f005806c3a0 0x7f005806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:43.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.979+0000 7f0075c00640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f005806c3a0 0x7f005806e860 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f00801f6070 tx=0x7f0068008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:43.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:43.982+0000 7f0077400640 1 -- 172.21.15.141:0/4267038834 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0070078350 con 0x7f00800cf7d0 2024-01-16T15:12:44.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:44.209+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51673", "target": ["mon-mgr", ""]}) v1 -- 0x7f008015e640 con 0x7f005806c3a0 2024-01-16T15:12:48.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.576+0000 7f0077400640 1 -- 172.21.15.141:0/4267038834 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f0068009800 con 0x7f005806c3a0 2024-01-16T15:12:48.553 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:12:48.553 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:12:48.553 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:12:48.554 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:12:48.554 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:12:48.554 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:12:48.554 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:12:48.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:12:48.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:12:48.554 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:12:48.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f005806c3a0 msgr2=0x7f005806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f005806c3a0 0x7f005806e860 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f00801f6070 tx=0x7f0068008040 comp rx=0 tx=0).stop 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00800cf7d0 msgr2=0x7f008013e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00800cf7d0 0x7f008013e950 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f0070000c00 tx=0x7f00700046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 shutdown_connections 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f005806c3a0 0x7f005806e860 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f008015e1e0 0x7f00800ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0080143ae0 0x7f00801f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 --2- 172.21.15.141:0/4267038834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f00800cf7d0 0x7f008013e950 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 >> 172.21.15.141:0/4267038834 conn(0x7f008006a800 msgr2=0x7f00800b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 shutdown_connections 2024-01-16T15:12:48.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.579+0000 7f0085400640 1 -- 172.21.15.141:0/4267038834 wait complete. 2024-01-16T15:12:48.577 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51671 2024-01-16T15:12:48.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.813+0000 7f13dc800640 1 Processor -- start 2024-01-16T15:12:48.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.813+0000 7f13dc800640 1 -- start start 2024-01-16T15:12:48.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.815+0000 7f13dc800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f13d40cf9a0 0x7f13d4166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.815+0000 7f13dc800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d415c5f0 con 0x7f13d413e360 2024-01-16T15:12:48.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.815+0000 7f13dc800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d406c470 con 0x7f13d413df80 2024-01-16T15:12:48.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.815+0000 7f13dc800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d406c750 con 0x7f13d40cf9a0 2024-01-16T15:12:48.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.815+0000 7f13da800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f13d40cf9a0 0x7f13d4166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.816+0000 7f13d9400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f13d413e360 0x7f13d40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54382/0 (socket says 172.21.15.141:54382) 2024-01-16T15:12:48.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.816+0000 7f13d9400640 1 -- 172.21.15.141:0/4145993572 learned_addr learned my addr 172.21.15.141:0/4145993572 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:12:48.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.816+0000 7f13da800640 1 -- 172.21.15.141:0/4145993572 >> v1:172.21.15.188:6789/0 conn(0x7f13d413df80 legacy=0x7f13d415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.816+0000 7f13da800640 1 -- 172.21.15.141:0/4145993572 >> v1:172.21.15.141:6789/0 conn(0x7f13d413e360 legacy=0x7f13d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.816+0000 7f13da800640 1 -- 172.21.15.141:0/4145993572 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d406ca30 con 0x7f13d40cf9a0 2024-01-16T15:12:48.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.817+0000 7f13da800640 1 --2- 172.21.15.141:0/4145993572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d4166030 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0097a0 tx=0x7f13cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a4077f4f4333f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.817+0000 7f13d8a00640 1 -- 172.21.15.141:0/4145993572 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc030610 con 0x7f13d40cf9a0 2024-01-16T15:12:48.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.817+0000 7f13d8a00640 1 -- 172.21.15.141:0/4145993572 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc030770 con 0x7f13d40cf9a0 2024-01-16T15:12:48.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.817+0000 7f13d8a00640 1 -- 172.21.15.141:0/4145993572 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc030b00 con 0x7f13d40cf9a0 2024-01-16T15:12:48.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.818+0000 7f13dc800640 1 -- 172.21.15.141:0/4145993572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 msgr2=0x7f13d4166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.818+0000 7f13dc800640 1 --2- 172.21.15.141:0/4145993572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d4166030 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0097a0 tx=0x7f13cc02b960 comp rx=0 tx=0).stop 2024-01-16T15:12:48.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.818+0000 7f13dc800640 1 -- 172.21.15.141:0/4145993572 shutdown_connections 2024-01-16T15:12:48.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.818+0000 7f13dc800640 1 --2- 172.21.15.141:0/4145993572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d4166030 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.818+0000 7f13dc800640 1 -- 172.21.15.141:0/4145993572 >> 172.21.15.141:0/4145993572 conn(0x7f13d406a800 msgr2=0x7f13d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:48.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.818+0000 7f13dc800640 1 -- 172.21.15.141:0/4145993572 shutdown_connections 2024-01-16T15:12:48.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.818+0000 7f13dc800640 1 -- 172.21.15.141:0/4145993572 wait complete. 2024-01-16T15:12:48.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.820+0000 7f13dc800640 1 Processor -- start 2024-01-16T15:12:48.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.820+0000 7f13dc800640 1 -- start start 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13dc800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d40a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13dc800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d413df80 0x7f13d40a9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13dc800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13d413e360 0x7f13d40a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13dc800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d4159700 con 0x7f13d40cf9a0 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13dc800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40a8560 con 0x7f13d413df80 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13dc800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40a8840 con 0x7f13d413e360 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13d9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d40a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13d9e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d40a9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59906/0 (socket says 172.21.15.141:59906) 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.821+0000 7f13d9e00640 1 -- 172.21.15.141:0/1647496642 learned_addr learned my addr 172.21.15.141:0/1647496642 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:48.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d9400640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d413df80 0x7f13d40a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13da800640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13d413e360 0x7f13d40a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d9e00640 1 -- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13d413e360 msgr2=0x7f13d40a9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d9e00640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13d413e360 0x7f13d40a9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d9e00640 1 -- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d413df80 msgr2=0x7f13d40a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d9e00640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d413df80 0x7f13d40a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d9e00640 1 -- 172.21.15.141:0/1647496642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc009400 con 0x7f13d40cf9a0 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d9e00640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d40a9320 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00c400 tx=0x7f13bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d2000640 1 -- 172.21.15.141:0/1647496642 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0052d0 con 0x7f13d40cf9a0 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d2000640 1 -- 172.21.15.141:0/1647496642 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13bc005430 con 0x7f13d40cf9a0 2024-01-16T15:12:48.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13d2000640 1 -- 172.21.15.141:0/1647496642 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc003830 con 0x7f13d40cf9a0 2024-01-16T15:12:48.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.822+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d40a8b20 con 0x7f13d40cf9a0 2024-01-16T15:12:48.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.823+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d415a4f0 con 0x7f13d40cf9a0 2024-01-16T15:12:48.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.824+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d40a9040 con 0x7f13d40cf9a0 2024-01-16T15:12:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.824+0000 7f13d2000640 1 -- 172.21.15.141:0/1647496642 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 35) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f13bc003990 con 0x7f13d40cf9a0 2024-01-16T15:12:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.825+0000 7f13d2000640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f13ac06c3a0 0x7f13ac06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.825+0000 7f13d2000640 1 -- 172.21.15.141:0/1647496642 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13bc087fd0 con 0x7f13d40cf9a0 2024-01-16T15:12:48.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.826+0000 7f13d9400640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f13ac06c3a0 0x7f13ac06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.827+0000 7f13d9400640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f13ac06c3a0 0x7f13ac06e860 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f13d4154ab0 tx=0x7f13c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:48.831+0000 7f13d2000640 1 -- 172.21.15.141:0/1647496642 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13d40a9040 con 0x7f13d40cf9a0 2024-01-16T15:12:49.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:49.060+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51671", "target": ["mon-mgr", ""]}) v1 -- 0x7f13d406cff0 con 0x7f13ac06c3a0 2024-01-16T15:12:53.555 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:53.578+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:53.556 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:53.578+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:53.556 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:53.578+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:53.556 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:53.578+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:53.556 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:12:53.578+0000 7f0f32a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:53.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.767+0000 7f13d2000640 1 -- 172.21.15.141:0/1647496642 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f13c4001d40 con 0x7f13ac06c3a0 2024-01-16T15:12:53.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f13ac06c3a0 msgr2=0x7f13ac06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f13ac06c3a0 0x7f13ac06e860 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f13d4154ab0 tx=0x7f13c4009040 comp rx=0 tx=0).stop 2024-01-16T15:12:53.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 msgr2=0x7f13d40a9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d40a9320 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00c400 tx=0x7f13bc00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 shutdown_connections 2024-01-16T15:12:53.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f13ac06c3a0 0x7f13ac06e860 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f13d413e360 0x7f13d40a9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d413df80 0x7f13d40a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 --2- 172.21.15.141:0/1647496642 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f13d40cf9a0 0x7f13d40a9320 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 >> 172.21.15.141:0/1647496642 conn(0x7f13d406a800 msgr2=0x7f13d415d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 shutdown_connections 2024-01-16T15:12:53.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:53.771+0000 7f13dc800640 1 -- 172.21.15.141:0/1647496642 wait complete. 2024-01-16T15:12:55.769 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51672 2024-01-16T15:12:55.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.018+0000 7f8aa6600640 1 Processor -- start 2024-01-16T15:12:55.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.019+0000 7f8aa6600640 1 -- start start 2024-01-16T15:12:55.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.020+0000 7f8aa6600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 0x7f8aa015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.020+0000 7f8aa6600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8aa013e360 0x7f8aa00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.020+0000 7f8aa6600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa006c470 con 0x7f8aa015f9c0 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.020+0000 7f8aa6600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa006c750 con 0x7f8aa013e360 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.020+0000 7f8aa6600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa006ca30 con 0x7f8aa013df80 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.020+0000 7f8a96800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8aa013e360 0x7f8aa00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.020+0000 7f8a9fe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f8aa015f9c0 0x7f8aa01408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:49790/0 (socket says 172.21.15.141:49790) 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a96800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8aa013e360 0x7f8aa00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50120/0 (socket says 172.21.15.141:50120) 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a9fe00640 1 -- 172.21.15.141:0/1086098878 learned_addr learned my addr 172.21.15.141:0/1086098878 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:12:55.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a9f400640 1 --2- 172.21.15.141:0/1086098878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 0x7f8aa015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a96800640 1 -- 172.21.15.141:0/1086098878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 msgr2=0x7f8aa015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:56.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a96800640 1 --2- 172.21.15.141:0/1086098878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 0x7f8aa015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a96800640 1 -- 172.21.15.141:0/1086098878 >> v1:172.21.15.141:6789/0 conn(0x7f8aa015f9c0 legacy=0x7f8aa01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a9ea00640 1 -- 172.21.15.141:0/1086098878 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288631092 0 0) 0x7f8aa006c470 con 0x7f8aa015f9c0 2024-01-16T15:12:56.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.021+0000 7f8a96800640 1 -- 172.21.15.141:0/1086098878 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa006cd10 con 0x7f8aa013e360 2024-01-16T15:12:56.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.022+0000 7f8a96800640 1 --2- 172.21.15.141:0/1086098878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa013e360 0x7f8aa00cf290 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f8a880097a0 tx=0x7f8a8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771d5f40763d4c92 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.022+0000 7f8a9ea00640 1 -- 172.21.15.141:0/1086098878 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88031830 con 0x7f8aa013e360 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.022+0000 7f8a9ea00640 1 -- 172.21.15.141:0/1086098878 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a88031990 con 0x7f8aa013e360 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.022+0000 7f8a9ea00640 1 -- 172.21.15.141:0/1086098878 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88031d20 con 0x7f8aa013e360 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 -- 172.21.15.141:0/1086098878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa013e360 msgr2=0x7f8aa00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 --2- 172.21.15.141:0/1086098878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa013e360 0x7f8aa00cf290 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f8a880097a0 tx=0x7f8a8802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 -- 172.21.15.141:0/1086098878 shutdown_connections 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 --2- 172.21.15.141:0/1086098878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa013e360 0x7f8aa00cf290 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 --2- 172.21.15.141:0/1086098878 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 0x7f8aa015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 -- 172.21.15.141:0/1086098878 >> 172.21.15.141:0/1086098878 conn(0x7f8aa006a800 msgr2=0x7f8aa006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:56.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 -- 172.21.15.141:0/1086098878 shutdown_connections 2024-01-16T15:12:56.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.023+0000 7f8aa6600640 1 -- 172.21.15.141:0/1086098878 wait complete. 2024-01-16T15:12:56.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.025+0000 7f8aa6600640 1 Processor -- start 2024-01-16T15:12:56.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.025+0000 7f8aa6600640 1 -- start start 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8aa6600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 0x7f8aa01322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8aa6600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8aa013e360 0x7f8aa014f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8aa6600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa015f9c0 0x7f8aa0152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8aa6600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa014fef0 con 0x7f8aa015f9c0 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8aa6600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa0130ae0 con 0x7f8aa013e360 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8aa6600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa0152ba0 con 0x7f8aa013df80 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8a9fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa015f9c0 0x7f8aa0152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8a9fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa015f9c0 0x7f8aa0152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50134/0 (socket says 172.21.15.141:50134) 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.026+0000 7f8a9fe00640 1 -- 172.21.15.141:0/243142730 learned_addr learned my addr 172.21.15.141:0/243142730 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a96800640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8aa013e360 0x7f8aa014f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a9fe00640 1 -- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 msgr2=0x7f8aa01322d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a9fe00640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 0x7f8aa01322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a9fe00640 1 -- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8aa013e360 msgr2=0x7f8aa014f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a9fe00640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8aa013e360 0x7f8aa014f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a9fe00640 1 -- 172.21.15.141:0/243142730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a88009400 con 0x7f8aa015f9c0 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a9fe00640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa015f9c0 0x7f8aa0152330 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c7e0 tx=0x7f8a9800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800a640 con 0x7f8aa015f9c0 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9800a7c0 con 0x7f8aa015f9c0 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa0152e80 con 0x7f8aa015f9c0 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800ab70 con 0x7f8aa015f9c0 2024-01-16T15:12:56.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.027+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa0153440 con 0x7f8aa015f9c0 2024-01-16T15:12:56.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.029+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa00675d0 con 0x7f8aa015f9c0 2024-01-16T15:12:56.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.030+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 36) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f8a9800acd0 con 0x7f8aa015f9c0 2024-01-16T15:12:56.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.030+0000 7f8a97e00640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a7806c460 0x7f8a7806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.031+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a98088310 con 0x7f8aa015f9c0 2024-01-16T15:12:56.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.031+0000 7f8a9f400640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a7806c460 0x7f8a7806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.032+0000 7f8a9f400640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a7806c460 0x7f8a7806e920 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8a90006fd0 tx=0x7f8a90008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.035+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a98054880 con 0x7f8aa015f9c0 2024-01-16T15:12:56.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:12:56.266+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51672", "target": ["mon-mgr", ""]}) v1 -- 0x7f8aa00705e0 con 0x7f8a7806c460 2024-01-16T15:13:06.789 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:06.798 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:36.927 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:36.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:55.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:13:55.849+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 37) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f8a98054c50 con 0x7f8aa015f9c0 2024-01-16T15:14:07.082 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:07.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:37.221 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:37.231 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:53.997+0000 7f8a97e00640 1 -- 172.21.15.141:0/243142730 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8aa00705e0 con 0x7f8a7806c460 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a7806c460 msgr2=0x7f8a7806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a7806c460 0x7f8a7806e920 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8a90006fd0 tx=0x7f8a90008040 comp rx=0 tx=0).stop 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa015f9c0 msgr2=0x7f8aa0152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa015f9c0 0x7f8aa0152330 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c7e0 tx=0x7f8a9800cca0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 shutdown_connections 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8a7806c460 0x7f8a7806e920 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa015f9c0 0x7f8aa0152330 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8aa013e360 0x7f8aa014f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 --2- 172.21.15.141:0/243142730 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8aa013df80 0x7f8aa01322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.001+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 >> 172.21.15.141:0/243142730 conn(0x7f8aa006a800 msgr2=0x7f8aa00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.002+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 shutdown_connections 2024-01-16T15:14:53.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:54.002+0000 7f8aa6600640 1 -- 172.21.15.141:0/243142730 wait complete. 2024-01-16T15:14:56.006 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51673 2024-01-16T15:14:56.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.242+0000 7fd987600640 1 Processor -- start 2024-01-16T15:14:56.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.243+0000 7fd987600640 1 -- start start 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.244+0000 7fd987600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd98006a3e0 0x7fd9800ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.244+0000 7fd987600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9800aede0 0x7fd98006f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.244+0000 7fd987600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd980071de0 con 0x7fd9800d0b60 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.244+0000 7fd987600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9800af270 con 0x7fd9800aede0 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.244+0000 7fd987600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd980072030 con 0x7fd98006a3e0 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd985600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9800aede0 0x7fd98006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd985600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9800aede0 0x7fd98006f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49834/0 (socket says 172.21.15.141:49834) 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd985600640 1 -- 172.21.15.141:0/1874049921 learned_addr learned my addr 172.21.15.141:0/1874049921 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd984c00640 1 --2- 172.21.15.141:0/1874049921 >> v2:172.21.15.188:3300/0 conn(0x7fd98006a3e0 0x7fd9800ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd985600640 1 -- 172.21.15.141:0/1874049921 >> v2:172.21.15.188:3300/0 conn(0x7fd98006a3e0 msgr2=0x7fd9800ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd985600640 1 --2- 172.21.15.141:0/1874049921 >> v2:172.21.15.188:3300/0 conn(0x7fd98006a3e0 0x7fd9800ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd985600640 1 -- 172.21.15.141:0/1874049921 >> v1:172.21.15.188:6789/0 conn(0x7fd9800d0b60 legacy=0x7fd98006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.245+0000 7fd985600640 1 -- 172.21.15.141:0/1874049921 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd980072310 con 0x7fd9800aede0 2024-01-16T15:14:56.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.247+0000 7fd985600640 1 --2- 172.21.15.141:0/1874049921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800aede0 0x7fd98006f080 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fd980073fe0 tx=0x7fd97802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f448e8a9ec2b6e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.247+0000 7fd97f400640 1 -- 172.21.15.141:0/1874049921 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97802ca80 con 0x7fd9800aede0 2024-01-16T15:14:56.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.247+0000 7fd97f400640 1 -- 172.21.15.141:0/1874049921 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97802cbe0 con 0x7fd9800aede0 2024-01-16T15:14:56.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.247+0000 7fd97f400640 1 -- 172.21.15.141:0/1874049921 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9780343f0 con 0x7fd9800aede0 2024-01-16T15:14:56.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.247+0000 7fd987600640 1 -- 172.21.15.141:0/1874049921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800aede0 msgr2=0x7fd98006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.247+0000 7fd987600640 1 --2- 172.21.15.141:0/1874049921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800aede0 0x7fd98006f080 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fd980073fe0 tx=0x7fd97802bee0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.248+0000 7fd987600640 1 -- 172.21.15.141:0/1874049921 shutdown_connections 2024-01-16T15:14:56.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.248+0000 7fd987600640 1 --2- 172.21.15.141:0/1874049921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800aede0 0x7fd98006f080 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.248+0000 7fd987600640 1 --2- 172.21.15.141:0/1874049921 >> v2:172.21.15.188:3300/0 conn(0x7fd98006a3e0 0x7fd9800ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.248+0000 7fd987600640 1 -- 172.21.15.141:0/1874049921 >> 172.21.15.141:0/1874049921 conn(0x7fd9800cfef0 msgr2=0x7fd9800d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.248+0000 7fd987600640 1 -- 172.21.15.141:0/1874049921 shutdown_connections 2024-01-16T15:14:56.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.248+0000 7fd987600640 1 -- 172.21.15.141:0/1874049921 wait complete. 2024-01-16T15:14:56.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.249+0000 7fd987600640 1 Processor -- start 2024-01-16T15:14:56.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.250+0000 7fd987600640 1 -- start start 2024-01-16T15:14:56.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.250+0000 7fd987600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd98006a3e0 0x7fd98014eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd987600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9800aede0 0x7fd980154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd987600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800d0b60 0x7fd98015c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd987600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98015b440 con 0x7fd9800d0b60 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd987600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98015b720 con 0x7fd9800aede0 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd987600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98015ba00 con 0x7fd98006a3e0 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800d0b60 0x7fd98015c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800d0b60 0x7fd98015c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49842/0 (socket says 172.21.15.141:49842) 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 -- 172.21.15.141:0/2186063948 learned_addr learned my addr 172.21.15.141:0/2186063948 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd984c00640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd98006a3e0 0x7fd98014eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd97fe00640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9800aede0 0x7fd980154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 -- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd98006a3e0 msgr2=0x7fd98014eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd98006a3e0 0x7fd98014eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 -- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9800aede0 msgr2=0x7fd980154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9800aede0 0x7fd980154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd985600640 1 -- 172.21.15.141:0/2186063948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd978009b30 con 0x7fd9800d0b60 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.251+0000 7fd97fe00640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9800aede0 0x7fd980154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.252+0000 7fd985600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800d0b60 0x7fd98015c4e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd978036040 tx=0x7fd978004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.252+0000 7fd97cc00640 1 -- 172.21.15.141:0/2186063948 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978009510 con 0x7fd9800d0b60 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.252+0000 7fd97cc00640 1 -- 172.21.15.141:0/2186063948 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97803e450 con 0x7fd9800d0b60 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.252+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98015bce0 con 0x7fd9800d0b60 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.252+0000 7fd97cc00640 1 -- 172.21.15.141:0/2186063948 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97803e7e0 con 0x7fd9800d0b60 2024-01-16T15:14:56.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.252+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98015b0d0 con 0x7fd9800d0b60 2024-01-16T15:14:56.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.253+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98015c200 con 0x7fd9800d0b60 2024-01-16T15:14:56.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.254+0000 7fd97cc00640 1 -- 172.21.15.141:0/2186063948 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 37) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fd97802cd50 con 0x7fd9800d0b60 2024-01-16T15:14:56.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.255+0000 7fd97cc00640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd95806c3a0 0x7fd95806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.255+0000 7fd97cc00640 1 -- 172.21.15.141:0/2186063948 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9780ab2d0 con 0x7fd9800d0b60 2024-01-16T15:14:56.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.256+0000 7fd984c00640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd95806c3a0 0x7fd95806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.257+0000 7fd984c00640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd95806c3a0 0x7fd95806e860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd970006fd0 tx=0x7fd970008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.260+0000 7fd97cc00640 1 -- 172.21.15.141:0/2186063948 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd98015c200 con 0x7fd9800d0b60 2024-01-16T15:14:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:56.486+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51673", "target": ["mon-mgr", ""]}) v1 -- 0x7fd98015bb70 con 0x7fd95806c3a0 2024-01-16T15:14:58.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.002+0000 7fd97cc00640 1 -- 172.21.15.141:0/2186063948 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd98015bb70 con 0x7fd95806c3a0 2024-01-16T15:14:58.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd95806c3a0 msgr2=0x7fd95806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd95806c3a0 0x7fd95806e860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd970006fd0 tx=0x7fd970008040 comp rx=0 tx=0).stop 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800d0b60 msgr2=0x7fd98015c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800d0b60 0x7fd98015c4e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd978036040 tx=0x7fd978004a40 comp rx=0 tx=0).stop 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 shutdown_connections 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd95806c3a0 0x7fd95806e860 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9800d0b60 0x7fd98015c4e0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9800aede0 0x7fd980154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.005+0000 7fd987600640 1 --2- 172.21.15.141:0/2186063948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd98006a3e0 0x7fd98014eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.006+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 >> 172.21.15.141:0/2186063948 conn(0x7fd9800cfef0 msgr2=0x7fd98006d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.006+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 shutdown_connections 2024-01-16T15:14:58.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:14:59.006+0000 7fd987600640 1 -- 172.21.15.141:0/2186063948 wait complete. 2024-01-16T15:15:01.013 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51671 2024-01-16T15:15:01.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.252+0000 7fa025400640 1 Processor -- start 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.253+0000 7fa025400640 1 -- start start 2024-01-16T15:15:01.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.254+0000 7fa025400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa02013df80 0x7fa02015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.254+0000 7fa025400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02006c470 con 0x7fa02013e360 2024-01-16T15:15:01.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.254+0000 7fa025400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02006c750 con 0x7fa0200cf9a0 2024-01-16T15:15:01.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.254+0000 7fa025400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02006ca30 con 0x7fa02013df80 2024-01-16T15:15:01.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.254+0000 7fa015c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa02013e360 0x7fa0200cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58634/0 (socket says 172.21.15.141:58634) 2024-01-16T15:15:01.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.254+0000 7fa015c00640 1 -- 172.21.15.141:0/1444272808 learned_addr learned my addr 172.21.15.141:0/1444272808 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:01.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01e800640 1 --2- 172.21.15.141:0/1444272808 >> v2:172.21.15.188:3300/0 conn(0x7fa02013df80 0x7fa02015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026987158 0 0) 0x7fa02006c470 con 0x7fa02013e360 2024-01-16T15:15:01.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa000003520 con 0x7fa02013e360 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3890773731 0 0) 0x7fa000003520 con 0x7fa02013e360 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 >> v2:172.21.15.188:3300/0 conn(0x7fa02013df80 msgr2=0x7fa02015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 --2- 172.21.15.141:0/1444272808 >> v2:172.21.15.188:3300/0 conn(0x7fa02013df80 0x7fa02015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 >> v1:172.21.15.188:6789/0 conn(0x7fa0200cf9a0 legacy=0x7fa0200cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02006cd10 con 0x7fa02013e360 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa008001670 con 0x7fa02013e360 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 >> v1:172.21.15.141:6789/0 conn(0x7fa02013e360 legacy=0x7fa0200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000003970 0x7fa000003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0000047c0 0x7fa000004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa000005180 0x7fa00000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02006ca30 con 0x7fa000005180 2024-01-16T15:15:01.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000c020 con 0x7fa0000047c0 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.255+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000c1e0 con 0x7fa000003970 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01f200640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa000005180 0x7fa00000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01e800640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000003970 0x7fa000003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa015c00640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0000047c0 0x7fa000004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01f200640 1 -- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000003970 msgr2=0x7fa000003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01f200640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000003970 0x7fa000003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01f200640 1 -- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0000047c0 msgr2=0x7fa000004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01f200640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0000047c0 0x7fa000004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01f200640 1 -- 172.21.15.141:0/1444272808 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000003520 con 0x7fa000005180 2024-01-16T15:15:01.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.256+0000 7fa01e800640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000003970 0x7fa000003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.257+0000 7fa01f200640 1 -- 172.21.15.141:0/1444272808 reap_dead start 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.257+0000 7fa01f200640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa000005180 0x7fa00000b980 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fa018001e20 tx=0x7fa0000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b764c88995daf6e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa018003b70 con 0x7fa000005180 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa018003cd0 con 0x7fa000005180 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa01de00640 1 -- 172.21.15.141:0/1444272808 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa018005310 con 0x7fa000005180 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa025400640 1 -- 172.21.15.141:0/1444272808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa000005180 msgr2=0x7fa00000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa025400640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa000005180 0x7fa00000b980 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fa018001e20 tx=0x7fa0000057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa025400640 1 -- 172.21.15.141:0/1444272808 shutdown_connections 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa025400640 1 --2- 172.21.15.141:0/1444272808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa000005180 0x7fa00000b980 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa025400640 1 -- 172.21.15.141:0/1444272808 >> 172.21.15.141:0/1444272808 conn(0x7fa02006a800 msgr2=0x7fa02006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.258+0000 7fa025400640 1 -- 172.21.15.141:0/1444272808 shutdown_connections 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.259+0000 7fa025400640 1 -- 172.21.15.141:0/1444272808 wait complete. 2024-01-16T15:15:01.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.260+0000 7fa025400640 1 Processor -- start 2024-01-16T15:15:01.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.260+0000 7fa025400640 1 -- start start 2024-01-16T15:15:01.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.261+0000 7fa025400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000005180 0x7fa020156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.261+0000 7fa025400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa020157c00 0x7fa0200ad380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.261+0000 7fa025400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0200ad8c0 0x7fa02014a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.261+0000 7fa025400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0200add70 con 0x7fa0200ad8c0 2024-01-16T15:15:01.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.261+0000 7fa025400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0201484a0 con 0x7fa020157c00 2024-01-16T15:15:01.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.261+0000 7fa025400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa020148780 con 0x7fa000005180 2024-01-16T15:15:01.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0200ad8c0 0x7fa02014a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0200ad8c0 0x7fa02014a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49846/0 (socket says 172.21.15.141:49846) 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 -- 172.21.15.141:0/1917130447 learned_addr learned my addr 172.21.15.141:0/1917130447 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01e800640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000005180 0x7fa020156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa015c00640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa020157c00 0x7fa0200ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 -- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000005180 msgr2=0x7fa020156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000005180 0x7fa020156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 -- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa020157c00 msgr2=0x7fa0200ad380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa020157c00 0x7fa0200ad380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 -- 172.21.15.141:0/1917130447 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa020148a60 con 0x7fa0200ad8c0 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.262+0000 7fa01f200640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0200ad8c0 0x7fa02014a630 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fa018001f40 tx=0x7fa0180014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.263+0000 7fa017400640 1 -- 172.21.15.141:0/1917130447 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa018006d50 con 0x7fa0200ad8c0 2024-01-16T15:15:01.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.263+0000 7fa017400640 1 -- 172.21.15.141:0/1917130447 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa018007030 con 0x7fa0200ad8c0 2024-01-16T15:15:01.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.263+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa020148d40 con 0x7fa0200ad8c0 2024-01-16T15:15:01.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.263+0000 7fa017400640 1 -- 172.21.15.141:0/1917130447 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0180073c0 con 0x7fa0200ad8c0 2024-01-16T15:15:01.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.263+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02014ab70 con 0x7fa0200ad8c0 2024-01-16T15:15:01.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.264+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0200add70 con 0x7fa0200ad8c0 2024-01-16T15:15:01.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.265+0000 7fa017400640 1 -- 172.21.15.141:0/1917130447 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 37) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fa018007520 con 0x7fa0200ad8c0 2024-01-16T15:15:01.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.266+0000 7fa017400640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9ff806c3a0 0x7f9ff806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.266+0000 7fa017400640 1 -- 172.21.15.141:0/1917130447 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0180aa650 con 0x7fa0200ad8c0 2024-01-16T15:15:01.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.266+0000 7fa01e800640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9ff806c3a0 0x7f9ff806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.268+0000 7fa01e800640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9ff806c3a0 0x7f9ff806e860 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa0201f6450 tx=0x7fa010008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.271+0000 7fa017400640 1 -- 172.21.15.141:0/1917130447 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0200add70 con 0x7fa0200ad8c0 2024-01-16T15:15:01.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.505+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51671", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0200cf290 con 0x7f9ff806c3a0 2024-01-16T15:15:01.501 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:01.517+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:01.501 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:01.517+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:01.501 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:01.517+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:01.501 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:01.517+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:01.501 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:01.517+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.540+0000 7fa017400640 1 -- 172.21.15.141:0/1917130447 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fa010004030 con 0x7f9ff806c3a0 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:15:01.523 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:15:01.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:01.524 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:15:01.524 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:15:01.524 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:15:01.524 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:01.524 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:01.524 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:15:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9ff806c3a0 msgr2=0x7f9ff806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9ff806c3a0 0x7f9ff806e860 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa0201f6450 tx=0x7fa010008040 comp rx=0 tx=0).stop 2024-01-16T15:15:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0200ad8c0 msgr2=0x7fa02014a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0200ad8c0 0x7fa02014a630 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fa018001f40 tx=0x7fa0180014a0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 shutdown_connections 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9ff806c3a0 0x7f9ff806e860 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0200ad8c0 0x7fa02014a630 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa020157c00 0x7fa0200ad380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 --2- 172.21.15.141:0/1917130447 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa000005180 0x7fa020156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 >> 172.21.15.141:0/1917130447 conn(0x7fa02006a800 msgr2=0x7fa0200af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 shutdown_connections 2024-01-16T15:15:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.543+0000 7fa025400640 1 -- 172.21.15.141:0/1917130447 wait complete. 2024-01-16T15:15:01.547 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51672 2024-01-16T15:15:01.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.778+0000 7ff0d3e00640 1 Processor -- start 2024-01-16T15:15:01.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.778+0000 7ff0d3e00640 1 -- start start 2024-01-16T15:15:01.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.779+0000 7ff0d3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff0d40ccb50 0x7ff0d4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.779+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d415bf10 0x7ff0d4143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.779+0000 7ff0d3e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d415f4e0 con 0x7ff0d40adf20 2024-01-16T15:15:01.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.779+0000 7ff0d3e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d415f970 con 0x7ff0d40ccb50 2024-01-16T15:15:01.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.779+0000 7ff0d3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d415c3c0 con 0x7ff0d415bf10 2024-01-16T15:15:01.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0c9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff0d40ccb50 0x7ff0d4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0c9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff0d40ccb50 0x7ff0d4070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49856/0 (socket says 172.21.15.141:49856) 2024-01-16T15:15:01.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0c9e00640 1 -- 172.21.15.141:0/2650274853 learned_addr learned my addr 172.21.15.141:0/2650274853 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:01.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0d3400640 1 --2- 172.21.15.141:0/2650274853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d415bf10 0x7ff0d4143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0c9e00640 1 -- 172.21.15.141:0/2650274853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d415bf10 msgr2=0x7ff0d4143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0c9e00640 1 --2- 172.21.15.141:0/2650274853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d415bf10 0x7ff0d4143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0c9e00640 1 -- 172.21.15.141:0/2650274853 >> v1:172.21.15.188:6789/0 conn(0x7ff0d40adf20 legacy=0x7ff0d40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0c9e00640 1 -- 172.21.15.141:0/2650274853 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d4071130 con 0x7ff0d40ccb50 2024-01-16T15:15:01.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.780+0000 7ff0d3400640 1 --2- 172.21.15.141:0/2650274853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d415bf10 0x7ff0d4143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:01.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.781+0000 7ff0c9e00640 1 --2- 172.21.15.141:0/2650274853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d40ccb50 0x7ff0d4070a30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009870 tx=0x7ff0bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fdfb417d00820a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.782+0000 7ff0d2000640 1 -- 172.21.15.141:0/2650274853 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc031640 con 0x7ff0d40ccb50 2024-01-16T15:15:01.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.782+0000 7ff0d2000640 1 -- 172.21.15.141:0/2650274853 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc0317a0 con 0x7ff0d40ccb50 2024-01-16T15:15:01.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.782+0000 7ff0d2000640 1 -- 172.21.15.141:0/2650274853 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc031b30 con 0x7ff0d40ccb50 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.782+0000 7ff0d3e00640 1 -- 172.21.15.141:0/2650274853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d40ccb50 msgr2=0x7ff0d4070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.782+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/2650274853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d40ccb50 0x7ff0d4070a30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009870 tx=0x7ff0bc02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.783+0000 7ff0d3e00640 1 -- 172.21.15.141:0/2650274853 shutdown_connections 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.783+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/2650274853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d40ccb50 0x7ff0d4070a30 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.783+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/2650274853 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d415bf10 0x7ff0d4143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.783+0000 7ff0d3e00640 1 -- 172.21.15.141:0/2650274853 >> 172.21.15.141:0/2650274853 conn(0x7ff0d406a7d0 msgr2=0x7ff0d406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.783+0000 7ff0d3e00640 1 -- 172.21.15.141:0/2650274853 shutdown_connections 2024-01-16T15:15:01.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.783+0000 7ff0d3e00640 1 -- 172.21.15.141:0/2650274853 wait complete. 2024-01-16T15:15:01.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.784+0000 7ff0d3e00640 1 Processor -- start 2024-01-16T15:15:01.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.785+0000 7ff0d3e00640 1 -- start start 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.785+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d40adf20 0x7ff0d40c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.785+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d40ccb50 0x7ff0d40c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.785+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d415bf10 0x7ff0d4141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.785+0000 7ff0d3e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40c6780 con 0x7ff0d415bf10 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.785+0000 7ff0d3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d406cfd0 con 0x7ff0d40adf20 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.785+0000 7ff0d3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40c3d00 con 0x7ff0d40ccb50 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d415bf10 0x7ff0d4141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d415bf10 0x7ff0d4141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49870/0 (socket says 172.21.15.141:49870) 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 -- 172.21.15.141:0/4117203055 learned_addr learned my addr 172.21.15.141:0/4117203055 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d2a00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d40adf20 0x7ff0d40c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0c9e00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d40ccb50 0x7ff0d40c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 -- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d40ccb50 msgr2=0x7ff0d40c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d40ccb50 0x7ff0d40c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 -- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d40adf20 msgr2=0x7ff0d40c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d40adf20 0x7ff0d40c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.786+0000 7ff0d3400640 1 -- 172.21.15.141:0/4117203055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc0094d0 con 0x7ff0d415bf10 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.787+0000 7ff0d3400640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d415bf10 0x7ff0d4141660 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00cd50 tx=0x7ff0cc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.787+0000 7ff0cb400640 1 -- 172.21.15.141:0/4117203055 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00f680 con 0x7ff0d415bf10 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.787+0000 7ff0cb400640 1 -- 172.21.15.141:0/4117203055 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc00f800 con 0x7ff0d415bf10 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.787+0000 7ff0cb400640 1 -- 172.21.15.141:0/4117203055 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00fbb0 con 0x7ff0d415bf10 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.787+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d41f61b0 con 0x7ff0d415bf10 2024-01-16T15:15:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.787+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d41f6770 con 0x7ff0d415bf10 2024-01-16T15:15:01.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.788+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d40c3ec0 con 0x7ff0d415bf10 2024-01-16T15:15:01.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.789+0000 7ff0cb400640 1 -- 172.21.15.141:0/4117203055 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 37) v1 ==== 91069+0+0 (secure 0 0 0) 0x7ff0cc00b040 con 0x7ff0d415bf10 2024-01-16T15:15:01.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.790+0000 7ff0cb400640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0ac06c3a0 0x7ff0ac06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.790+0000 7ff0cb400640 1 -- 172.21.15.141:0/4117203055 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0cc0894b0 con 0x7ff0d415bf10 2024-01-16T15:15:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.790+0000 7ff0d2a00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0ac06c3a0 0x7ff0ac06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.792+0000 7ff0d2a00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0ac06c3a0 0x7ff0ac06e860 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4000f60 tx=0x7ff0c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:01.794+0000 7ff0cb400640 1 -- 172.21.15.141:0/4117203055 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0cc055a30 con 0x7ff0d415bf10 2024-01-16T15:15:02.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.020+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51672", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0d415f170 con 0x7ff0ac06c3a0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.045+0000 7ff0cb400640 1 -- 172.21.15.141:0/4117203055 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7ff0c4001360 con 0x7ff0ac06c3a0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:02.029 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:15:02.029 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:15:02.029 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:15:02.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:02.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:02.029 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.047+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0ac06c3a0 msgr2=0x7ff0ac06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.047+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0ac06c3a0 0x7ff0ac06e860 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4000f60 tx=0x7ff0c4009040 comp rx=0 tx=0).stop 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.047+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d415bf10 msgr2=0x7ff0d4141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.047+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d415bf10 0x7ff0d4141660 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00cd50 tx=0x7ff0cc00a650 comp rx=0 tx=0).stop 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.047+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 shutdown_connections 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.048+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff0ac06c3a0 0x7ff0ac06e860 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.048+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0d415bf10 0x7ff0d4141660 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.048+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff0d40ccb50 0x7ff0d40c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.048+0000 7ff0d3e00640 1 --2- 172.21.15.141:0/4117203055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d40adf20 0x7ff0d40c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.048+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 >> 172.21.15.141:0/4117203055 conn(0x7ff0d406a7d0 msgr2=0x7ff0d40b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.048+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 shutdown_connections 2024-01-16T15:15:02.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.048+0000 7ff0d3e00640 1 -- 172.21.15.141:0/4117203055 wait complete. 2024-01-16T15:15:02.049 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51673 2024-01-16T15:15:02.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.275+0000 7fc712200640 1 Processor -- start 2024-01-16T15:15:02.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.276+0000 7fc712200640 1 -- start start 2024-01-16T15:15:02.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc712200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0cc1e0 0x7fc70c0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc712200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc70c074520 0x7fc70c0ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.260 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc712200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70c162100 con 0x7fc70c0ae750 2024-01-16T15:15:02.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc712200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c0ccf20 con 0x7fc70c074520 2024-01-16T15:15:02.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc712200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c0cd200 con 0x7fc70c0cc1e0 2024-01-16T15:15:02.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc70be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc70c0ae750 0x7fc70c0ac590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58646/0 (socket says 172.21.15.141:58646) 2024-01-16T15:15:02.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc70be00640 1 -- 172.21.15.141:0/2475359433 learned_addr learned my addr 172.21.15.141:0/2475359433 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:02.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.277+0000 7fc70aa00640 1 --2- 172.21.15.141:0/2475359433 >> v2:172.21.15.141:3300/0 conn(0x7fc70c074520 0x7fc70c0ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.261 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.278+0000 7fc70b400640 1 --2- 172.21.15.141:0/2475359433 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0cc1e0 0x7fc70c0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.278+0000 7fc70a000640 1 -- 172.21.15.141:0/2475359433 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833786343 0 0) 0x7fc70c162100 con 0x7fc70c0ae750 2024-01-16T15:15:02.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.278+0000 7fc70a000640 1 -- 172.21.15.141:0/2475359433 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ec003520 con 0x7fc70c0ae750 2024-01-16T15:15:02.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.278+0000 7fc70aa00640 1 -- 172.21.15.141:0/2475359433 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0cc1e0 msgr2=0x7fc70c0cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.278+0000 7fc70aa00640 1 --2- 172.21.15.141:0/2475359433 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0cc1e0 0x7fc70c0cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.278+0000 7fc70aa00640 1 -- 172.21.15.141:0/2475359433 >> v1:172.21.15.141:6789/0 conn(0x7fc70c0ae750 legacy=0x7fc70c0ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.262 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.278+0000 7fc70aa00640 1 -- 172.21.15.141:0/2475359433 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c06daa0 con 0x7fc70c074520 2024-01-16T15:15:02.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.279+0000 7fc70aa00640 1 --2- 172.21.15.141:0/2475359433 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0ac050 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40097a0 tx=0x7fc6f402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b493adabc4a267b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.280+0000 7fc70a000640 1 -- 172.21.15.141:0/2475359433 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f40316f0 con 0x7fc70c074520 2024-01-16T15:15:02.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.280+0000 7fc70a000640 1 -- 172.21.15.141:0/2475359433 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4031850 con 0x7fc70c074520 2024-01-16T15:15:02.263 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.280+0000 7fc70a000640 1 -- 172.21.15.141:0/2475359433 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4031be0 con 0x7fc70c074520 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.280+0000 7fc712200640 1 -- 172.21.15.141:0/2475359433 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 msgr2=0x7fc70c0ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.280+0000 7fc712200640 1 --2- 172.21.15.141:0/2475359433 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0ac050 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40097a0 tx=0x7fc6f402b790 comp rx=0 tx=0).stop 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.281+0000 7fc712200640 1 -- 172.21.15.141:0/2475359433 shutdown_connections 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.281+0000 7fc712200640 1 --2- 172.21.15.141:0/2475359433 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0ac050 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.281+0000 7fc712200640 1 --2- 172.21.15.141:0/2475359433 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0cc1e0 0x7fc70c0cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.281+0000 7fc712200640 1 -- 172.21.15.141:0/2475359433 >> 172.21.15.141:0/2475359433 conn(0x7fc70c0b0d70 msgr2=0x7fc70c0b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.281+0000 7fc712200640 1 -- 172.21.15.141:0/2475359433 shutdown_connections 2024-01-16T15:15:02.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.281+0000 7fc712200640 1 -- 172.21.15.141:0/2475359433 wait complete. 2024-01-16T15:15:02.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.282+0000 7fc712200640 1 Processor -- start 2024-01-16T15:15:02.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.283+0000 7fc712200640 1 -- start start 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc712200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc712200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0ae750 0x7fc70c15c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc712200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc70c0cc1e0 0x7fc70c0c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc712200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c15c650 con 0x7fc70c074520 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc712200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c0c34d0 con 0x7fc70c0cc1e0 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc712200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c0c37b0 con 0x7fc70c0ae750 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:49884/0 (socket says 172.21.15.141:49884) 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 -- 172.21.15.141:0/1567781240 learned_addr learned my addr 172.21.15.141:0/1567781240 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 -- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0ae750 msgr2=0x7fc70c15c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0ae750 0x7fc70c15c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 -- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc70c0cc1e0 msgr2=0x7fc70c0c2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc70c0cc1e0 0x7fc70c0c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.284+0000 7fc70b400640 1 -- 172.21.15.141:0/1567781240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4009400 con 0x7fc70c074520 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.285+0000 7fc70b400640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0c62f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00e870 tx=0x7fc6fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.285+0000 7fc703400640 1 -- 172.21.15.141:0/1567781240 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00c940 con 0x7fc70c074520 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.285+0000 7fc703400640 1 -- 172.21.15.141:0/1567781240 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc00caa0 con 0x7fc70c074520 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.285+0000 7fc703400640 1 -- 172.21.15.141:0/1567781240 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00ce50 con 0x7fc70c074520 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.285+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70c15eb80 con 0x7fc70c074520 2024-01-16T15:15:02.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.285+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70c0c31c0 con 0x7fc70c074520 2024-01-16T15:15:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.286+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70c0c39d0 con 0x7fc70c074520 2024-01-16T15:15:02.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.287+0000 7fc703400640 1 -- 172.21.15.141:0/1567781240 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 37) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fc6fc013470 con 0x7fc70c074520 2024-01-16T15:15:02.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.288+0000 7fc703400640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6e406c3a0 0x7fc6e406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.288+0000 7fc703400640 1 -- 172.21.15.141:0/1567781240 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6fc088bc0 con 0x7fc70c074520 2024-01-16T15:15:02.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.289+0000 7fc70aa00640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6e406c3a0 0x7fc6e406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.290+0000 7fc70aa00640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6e406c3a0 0x7fc6e406e860 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4036000 tx=0x7fc6f4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.292+0000 7fc703400640 1 -- 172.21.15.141:0/1567781240 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6fc055140 con 0x7fc70c074520 2024-01-16T15:15:02.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:02.521+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51673", "target": ["mon-mgr", ""]}) v1 -- 0x7fc70c0cc5c0 con 0x7fc6e406c3a0 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.566+0000 7fc703400640 1 -- 172.21.15.141:0/1567781240 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fc6f4031a40 con 0x7fc6e406c3a0 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "canceled", 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:15:03.550 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:15:03.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:03.551 INFO:teuthology.orchestra.run.smithi141.stdout: "failure": { 2024-01-16T15:15:03.551 INFO:teuthology.orchestra.run.smithi141.stdout: "errno": "4", 2024-01-16T15:15:03.551 INFO:teuthology.orchestra.run.smithi141.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:15:03.551 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:03.551 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:03.551 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:15:03.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6e406c3a0 msgr2=0x7fc6e406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6e406c3a0 0x7fc6e406e860 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4036000 tx=0x7fc6f4030000 comp rx=0 tx=0).stop 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 msgr2=0x7fc70c0c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0c62f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00e870 tx=0x7fc6fc00ed30 comp rx=0 tx=0).stop 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 shutdown_connections 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6e406c3a0 0x7fc6e406e860 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc70c0cc1e0 0x7fc70c0c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc70c0ae750 0x7fc70c15c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 --2- 172.21.15.141:0/1567781240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc70c074520 0x7fc70c0c62f0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 >> 172.21.15.141:0/1567781240 conn(0x7fc70c0b0d70 msgr2=0x7fc70c0ca1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 shutdown_connections 2024-01-16T15:15:03.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.570+0000 7fc712200640 1 -- 172.21.15.141:0/1567781240 wait complete. 2024-01-16T15:15:03.573 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:15:03.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.797+0000 7fb4d2800640 1 Processor -- start 2024-01-16T15:15:03.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.797+0000 7fb4d2800640 1 -- start start 2024-01-16T15:15:03.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.799+0000 7fb4d2800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb4cc15e3b0 0x7fb4cc164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.799+0000 7fb4d2800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0b1ee0 con 0x7fb4cc143ae0 2024-01-16T15:15:03.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.800+0000 7fb4d2800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0b2780 con 0x7fb4cc0cf9a0 2024-01-16T15:15:03.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.800+0000 7fb4d2800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc15e990 con 0x7fb4cc15e3b0 2024-01-16T15:15:03.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.800+0000 7fb4cb400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb4cc143ae0 0x7fb4cc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52548/0 (socket says 172.21.15.141:52548) 2024-01-16T15:15:03.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.800+0000 7fb4cb400640 1 -- 172.21.15.141:0/1686587692 learned_addr learned my addr 172.21.15.141:0/1686587692 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.800+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196670183 0 0) 0x7fb4cc0b1ee0 con 0x7fb4cc143ae0 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.800+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ac005730 con 0x7fb4cc143ae0 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.800+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/1686587692 >> v2:172.21.15.188:3300/0 conn(0x7fb4cc15e3b0 0x7fb4cc164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4004131425 0 0) 0x7fb4ac005730 con 0x7fb4cc143ae0 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 >> v2:172.21.15.188:3300/0 conn(0x7fb4cc15e3b0 msgr2=0x7fb4cc164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 --2- 172.21.15.141:0/1686587692 >> v2:172.21.15.188:3300/0 conn(0x7fb4cc15e3b0 0x7fb4cc164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 >> v1:172.21.15.188:6789/0 conn(0x7fb4cc0cf9a0 legacy=0x7fb4cc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0b2130 con 0x7fb4cc143ae0 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb4b4001670 con 0x7fb4cc143ae0 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 >> v1:172.21.15.141:6789/0 conn(0x7fb4cc143ae0 legacy=0x7fb4cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac005b80 0x7fb4ac005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.786 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4ac0069d0 0x7fb4ac006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4ac007390 0x7fb4ac00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc15e990 con 0x7fb4ac007390 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac00e2e0 con 0x7fb4ac005b80 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac00e4a0 con 0x7fb4ac0069d0 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4ac007390 0x7fb4ac00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4cb400640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac005b80 0x7fb4ac005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4cbe00640 1 -- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4ac0069d0 msgr2=0x7fb4ac006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4c2800640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4ac0069d0 0x7fb4ac006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.801+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4ac0069d0 0x7fb4ac006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.802+0000 7fb4cbe00640 1 -- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac005b80 msgr2=0x7fb4ac005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.802+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac005b80 0x7fb4ac005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.802+0000 7fb4cbe00640 1 -- 172.21.15.141:0/1686587692 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac005730 con 0x7fb4ac007390 2024-01-16T15:15:03.787 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.802+0000 7fb4cb400640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac005b80 0x7fb4ac005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:03.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.802+0000 7fb4cbe00640 1 -- 172.21.15.141:0/1686587692 reap_dead start 2024-01-16T15:15:03.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.803+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4ac007390 0x7fb4ac00db90 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4007360 tx=0x7fb4ac007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81c2de963a73029 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.803+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c402ccf0 con 0x7fb4ac007390 2024-01-16T15:15:03.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.803+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c402ce50 con 0x7fb4ac007390 2024-01-16T15:15:03.788 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.803+0000 7fb4caa00640 1 -- 172.21.15.141:0/1686587692 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4003bd0 con 0x7fb4ac007390 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.803+0000 7fb4d2800640 1 -- 172.21.15.141:0/1686587692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4ac007390 msgr2=0x7fb4ac00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.803+0000 7fb4d2800640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4ac007390 0x7fb4ac00db90 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4007360 tx=0x7fb4ac007a00 comp rx=0 tx=0).stop 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.804+0000 7fb4d2800640 1 -- 172.21.15.141:0/1686587692 shutdown_connections 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.804+0000 7fb4d2800640 1 --2- 172.21.15.141:0/1686587692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4ac007390 0x7fb4ac00db90 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.804+0000 7fb4d2800640 1 -- 172.21.15.141:0/1686587692 >> 172.21.15.141:0/1686587692 conn(0x7fb4cc06a800 msgr2=0x7fb4cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.804+0000 7fb4d2800640 1 -- 172.21.15.141:0/1686587692 shutdown_connections 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.804+0000 7fb4d2800640 1 -- 172.21.15.141:0/1686587692 wait complete. 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.805+0000 7fb4d2800640 1 Processor -- start 2024-01-16T15:15:03.789 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.806+0000 7fb4d2800640 1 -- start start 2024-01-16T15:15:03.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.806+0000 7fb4d2800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac007390 0x7fb4cc137660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.806+0000 7fb4d2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4cc0702c0 0x7fb4cc070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.806+0000 7fb4d2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4cc0ac8d0 0x7fb4cc0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.806+0000 7fb4d2800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0ce0c0 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4d2800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc070c80 con 0x7fb4ac007390 2024-01-16T15:15:03.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4d2800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0ae0f0 con 0x7fb4cc0702c0 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4cc0ac8d0 0x7fb4cc0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4cc0ac8d0 0x7fb4cc0acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43082/0 (socket says 172.21.15.141:43082) 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 -- 172.21.15.141:0/987341959 learned_addr learned my addr 172.21.15.141:0/987341959 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 -- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4cc0702c0 msgr2=0x7fb4cc070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4cc0702c0 0x7fb4cc070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 -- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac007390 msgr2=0x7fb4cc137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac007390 0x7fb4cc137660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 -- 172.21.15.141:0/987341959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0ad290 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.807+0000 7fb4cbe00640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4cc0ac8d0 0x7fb4cc0acd50 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402c760 tx=0x7fb4c4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.808+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4039070 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.808+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4003e10 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.808+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40044b0 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.808+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc0ae580 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.808+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc0ad7c0 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.809+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc0675d0 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.810+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 37) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fb4c403d070 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.811+0000 7fb4c3e00640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4a406c3a0 0x7fb4a406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.811+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4c40a9cb0 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.812+0000 7fb4cb400640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4a406c3a0 0x7fb4a406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.813+0000 7fb4cb400640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4a406c3a0 0x7fb4a406e860 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40017a0 tx=0x7fb4b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.816+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4c4076230 con 0x7fb4cc0ac8d0 2024-01-16T15:15:03.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:03.947+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fb4c4076600 con 0x7fb4cc0ac8d0 2024-01-16T15:15:04.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.083+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb4cc0ae280 con 0x7fb4cc0ac8d0 2024-01-16T15:15:04.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.084+0000 7fb4c3e00640 1 -- 172.21.15.141:0/987341959 <== mon.0 v2:172.21.15.141:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1855 (secure 0 0 0) 0x7fb4c4044750 con 0x7fb4cc0ac8d0 2024-01-16T15:15:04.067 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:04.067 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "health": { 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "HEALTH_OK", 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "checks": {}, 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "mutes": [] 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "election_epoch": 4, 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum": [ 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: 0, 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: 1, 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: 2 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_names": [ 2024-01-16T15:15:04.068 INFO:teuthology.orchestra.run.smithi141.stdout: "a", 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "b", 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "c" 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "quorum_age": 1096, 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "monmap": { 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 1, 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "min_mon_release_name": "reef", 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "num_mons": 3 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap": { 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 102, 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "num_osds": 8, 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "num_up_osds": 8, 2024-01-16T15:15:04.069 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_up_since": 1705417019, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "num_in_osds": 8, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_in_since": 1705417011, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "num_remapped_pgs": 0 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap": { 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "pgs_by_state": [ 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "state_name": "active+clean", 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "count": 97 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pgs": 97, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "num_pools": 3, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "num_objects": 802, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "data_bytes": 3222400956, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_used": 7009820672, 2024-01-16T15:15:04.070 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_avail": 766084292608, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "bytes_total": 773094113280, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "read_bytes_sec": 597, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "write_bytes_sec": 10576, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "read_op_per_sec": 0, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "write_op_per_sec": 3 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "fsmap": { 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 65, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 7, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "up": 1, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "in": 1, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "by_rank": [ 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "filesystem_id": 7, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "rank": 0, 2024-01-16T15:15:04.071 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "d", 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "status": "up:active", 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "gid": 8640 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "up:standby": 3 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "mgrmap": { 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "available": true, 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "num_standbys": 1, 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "modules": [ 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "iostat", 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "nfs", 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "restful" 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2024-01-16T15:15:04.072 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout: "servicemap": { 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout: "epoch": 18, 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout: "modified": "2024-01-16T15:12:54.856312+0000", 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout: "services": {} 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout: "progress_events": {} 2024-01-16T15:15:04.073 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:15:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4a406c3a0 msgr2=0x7fb4a406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4a406c3a0 0x7fb4a406e860 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40017a0 tx=0x7fb4b4009040 comp rx=0 tx=0).stop 2024-01-16T15:15:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4cc0ac8d0 msgr2=0x7fb4cc0acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4cc0ac8d0 0x7fb4cc0acd50 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402c760 tx=0x7fb4c4004ab0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 shutdown_connections 2024-01-16T15:15:04.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb4a406c3a0 0x7fb4a406e860 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4cc0ac8d0 0x7fb4cc0acd50 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb4cc0702c0 0x7fb4cc070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 --2- 172.21.15.141:0/987341959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac007390 0x7fb4cc137660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 >> 172.21.15.141:0/987341959 conn(0x7fb4cc06a800 msgr2=0x7fb4cc06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 shutdown_connections 2024-01-16T15:15:04.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.087+0000 7fb4d2800640 1 -- 172.21.15.141:0/987341959 wait complete. 2024-01-16T15:15:04.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:04.089 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/subvol_clone51671/d6eabf46-18bc-4093-8305-1e4d9e7f6597) 2024-01-16T15:15:04.148 INFO:teuthology.orchestra.run.smithi141.stderr:ls: cannot access 'volumes/_nogroup/subvol_clone51671/d6eabf46-18bc-4093-8305-1e4d9e7f6597': No such file or directory 2024-01-16T15:15:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:15:04.150 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:15:04.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.376+0000 7f80edc00640 1 Processor -- start 2024-01-16T15:15:04.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.377+0000 7f80edc00640 1 -- start start 2024-01-16T15:15:04.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.378+0000 7f80edc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e8143c40 0x7f80e8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.378+0000 7f80edc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e8070e50 con 0x7f80e80ccc10 2024-01-16T15:15:04.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.378+0000 7f80edc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e8144020 con 0x7f80e80717d0 2024-01-16T15:15:04.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.378+0000 7f80edc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e80723c0 con 0x7f80e8143c40 2024-01-16T15:15:04.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80ed200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f80e80ccc10 0x7f80e806d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52562/0 (socket says 172.21.15.141:52562) 2024-01-16T15:15:04.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80ed200640 1 -- 172.21.15.141:0/4260652299 learned_addr learned my addr 172.21.15.141:0/4260652299 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7e00640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e8143c40 0x7f80e8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070629828 0 0) 0x7f80e8070e50 con 0x7f80e80ccc10 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c8003520 con 0x7f80e80ccc10 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1923017524 0 0) 0x7f80c8003520 con 0x7f80e80ccc10 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e8143c40 msgr2=0x7f80e8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e8143c40 0x7f80e8071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 >> v1:172.21.15.188:6789/0 conn(0x7f80e80717d0 legacy=0x7f80e80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e8071e00 con 0x7f80e80ccc10 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f80e00019f0 con 0x7f80e80ccc10 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.379+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 >> v1:172.21.15.141:6789/0 conn(0x7f80e80ccc10 legacy=0x7f80e806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7400640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7400640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c80047c0 0x7f80c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7400640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e8143c40 0x7f80c8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e80723c0 con 0x7f80c80039a0 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c800bb10 con 0x7f80c80047c0 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c800bcd0 con 0x7f80e8143c40 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7e00640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7e00640 1 -- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e8143c40 msgr2=0x7f80c8005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7e00640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e8143c40 0x7f80c8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7e00640 1 -- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c80047c0 msgr2=0x7f80c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7e00640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c80047c0 0x7f80c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.380+0000 7f80e7e00640 1 -- 172.21.15.141:0/4260652299 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c8003520 con 0x7f80c80039a0 2024-01-16T15:15:04.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.381+0000 7f80e7e00640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80c8003dc0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009ac0 tx=0x7f80c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab1869caf36c19c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d802c710 con 0x7f80c80039a0 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d802c870 con 0x7f80c80039a0 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80e7400640 1 -- 172.21.15.141:0/4260652299 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d802cc00 con 0x7f80c80039a0 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80edc00640 1 -- 172.21.15.141:0/4260652299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 msgr2=0x7f80c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80edc00640 1 --2- 172.21.15.141:0/4260652299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80c8003dc0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009ac0 tx=0x7f80c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80ed200640 1 -- 172.21.15.141:0/4260652299 reap_dead start 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80edc00640 1 -- 172.21.15.141:0/4260652299 shutdown_connections 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80edc00640 1 -- 172.21.15.141:0/4260652299 >> 172.21.15.141:0/4260652299 conn(0x7f80e806a800 msgr2=0x7f80e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80edc00640 1 -- 172.21.15.141:0/4260652299 shutdown_connections 2024-01-16T15:15:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.382+0000 7f80edc00640 1 -- 172.21.15.141:0/4260652299 wait complete. 2024-01-16T15:15:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.384+0000 7f80edc00640 1 Processor -- start 2024-01-16T15:15:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.384+0000 7f80edc00640 1 -- start start 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.385+0000 7f80edc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80e813a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.385+0000 7f80edc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e80b2fe0 0x7f80e81f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.385+0000 7f80edc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e80b3460 0x7f80e815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.385+0000 7f80edc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81425f0 con 0x7f80c80039a0 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.385+0000 7f80edc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81413a0 con 0x7f80e80b2fe0 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.385+0000 7f80edc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81419e0 con 0x7f80e80b3460 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80e813a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80e813a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43098/0 (socket says 172.21.15.141:43098) 2024-01-16T15:15:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 -- 172.21.15.141:0/3989309065 learned_addr learned my addr 172.21.15.141:0/3989309065 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80df200640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e80b2fe0 0x7f80e81f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80ed200640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e80b3460 0x7f80e815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 -- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e80b3460 msgr2=0x7f80e815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e80b3460 0x7f80e815a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 -- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e80b2fe0 msgr2=0x7f80e81f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e80b2fe0 0x7f80e81f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80e7e00640 1 -- 172.21.15.141:0/3989309065 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e8141cc0 con 0x7f80c80039a0 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80df200640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e80b2fe0 0x7f80e81f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.386+0000 7f80ed200640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e80b3460 0x7f80e815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.387+0000 7f80e7e00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80e813a0a0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f80d8037040 tx=0x7f80d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.387+0000 7f80e4c00640 1 -- 172.21.15.141:0/3989309065 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8003a20 con 0x7f80c80039a0 2024-01-16T15:15:04.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.387+0000 7f80e4c00640 1 -- 172.21.15.141:0/3989309065 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d8003b80 con 0x7f80c80039a0 2024-01-16T15:15:04.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.387+0000 7f80e4c00640 1 -- 172.21.15.141:0/3989309065 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d802de30 con 0x7f80c80039a0 2024-01-16T15:15:04.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.387+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e8141fa0 con 0x7f80c80039a0 2024-01-16T15:15:04.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.387+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e813a5e0 con 0x7f80c80039a0 2024-01-16T15:15:04.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.388+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e8067450 con 0x7f80c80039a0 2024-01-16T15:15:04.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.389+0000 7f80e4c00640 1 -- 172.21.15.141:0/3989309065 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f80d8040450 con 0x7f80c80039a0 2024-01-16T15:15:04.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.390+0000 7f80e4c00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80c006c430 0x7f80c006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.390+0000 7f80e4c00640 1 -- 172.21.15.141:0/3989309065 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80d80aaab0 con 0x7f80c80039a0 2024-01-16T15:15:04.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.391+0000 7f80df200640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80c006c430 0x7f80c006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.392+0000 7f80df200640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80c006c430 0x7f80c006e8f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f80e813c720 tx=0x7f80d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.395+0000 7f80e4c00640 1 -- 172.21.15.141:0/3989309065 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80d8077020 con 0x7f80c80039a0 2024-01-16T15:15:04.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.604+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f80e806cc40 con 0x7f80c006c430 2024-01-16T15:15:04.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.608+0000 7f80e4c00640 1 -- 172.21.15.141:0/3989309065 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f80d00013b0 con 0x7f80c006c430 2024-01-16T15:15:04.591 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:15:04.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:04.592 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:15:04.592 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:04.592 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:15:04.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80c006c430 msgr2=0x7f80c006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80c006c430 0x7f80c006e8f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f80e813c720 tx=0x7f80d0009040 comp rx=0 tx=0).stop 2024-01-16T15:15:04.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 msgr2=0x7f80e813a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80e813a0a0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f80d8037040 tx=0x7f80d8004730 comp rx=0 tx=0).stop 2024-01-16T15:15:04.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 shutdown_connections 2024-01-16T15:15:04.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80c006c430 0x7f80c006e8f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80e80b3460 0x7f80e815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e80b2fe0 0x7f80e81f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 --2- 172.21.15.141:0/3989309065 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80c80039a0 0x7f80e813a0a0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 >> 172.21.15.141:0/3989309065 conn(0x7f80e806a800 msgr2=0x7f80e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 shutdown_connections 2024-01-16T15:15:04.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.611+0000 7f80edc00640 1 -- 172.21.15.141:0/3989309065 wait complete. 2024-01-16T15:15:04.611 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:15:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.840+0000 7fc164e00640 1 Processor -- start 2024-01-16T15:15:04.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.840+0000 7fc164e00640 1 -- start start 2024-01-16T15:15:04.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.841+0000 7fc164e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc16006c610 0x7fc1600cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.841+0000 7fc164e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006b700 0x7fc16015caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.841+0000 7fc164e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1600ce7b0 con 0x7fc1600cde20 2024-01-16T15:15:04.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.842+0000 7fc164e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1600b0f80 con 0x7fc16006c610 2024-01-16T15:15:04.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.842+0000 7fc164e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1600ce200 con 0x7fc16006b700 2024-01-16T15:15:04.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.842+0000 7fc15f400640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fc1600cde20 0x7fc16006bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:45026/0 (socket says 172.21.15.141:45026) 2024-01-16T15:15:04.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.842+0000 7fc15f400640 1 -- 172.21.15.141:0/2344940368 learned_addr learned my addr 172.21.15.141:0/2344940368 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:04.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.842+0000 7fc15fe00640 1 --2- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006b700 0x7fc16015caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.842+0000 7fc157e00640 1 --2- 172.21.15.141:0/2344940368 >> v2:172.21.15.188:3300/0 conn(0x7fc16006c610 0x7fc1600cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.843+0000 7fc157e00640 1 -- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006b700 msgr2=0x7fc16015caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.843+0000 7fc157e00640 1 --2- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006b700 0x7fc16015caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.843+0000 7fc157e00640 1 -- 172.21.15.141:0/2344940368 >> v1:172.21.15.188:6789/0 conn(0x7fc1600cde20 legacy=0x7fc16006bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.843+0000 7fc157e00640 1 -- 172.21.15.141:0/2344940368 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1600d0350 con 0x7fc16006c610 2024-01-16T15:15:04.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.844+0000 7fc157e00640 1 --2- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006c610 0x7fc1600cfc70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc1480097a0 tx=0x7fc14802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e91d7ac9786506e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc15ea00640 1 -- 172.21.15.141:0/2344940368 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480316f0 con 0x7fc16006c610 2024-01-16T15:15:04.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc15ea00640 1 -- 172.21.15.141:0/2344940368 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc148031850 con 0x7fc16006c610 2024-01-16T15:15:04.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc15ea00640 1 -- 172.21.15.141:0/2344940368 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc148031be0 con 0x7fc16006c610 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc164e00640 1 -- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006c610 msgr2=0x7fc1600cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc164e00640 1 --2- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006c610 0x7fc1600cfc70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc1480097a0 tx=0x7fc14802b790 comp rx=0 tx=0).stop 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc164e00640 1 -- 172.21.15.141:0/2344940368 shutdown_connections 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc164e00640 1 --2- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006c610 0x7fc1600cfc70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc164e00640 1 --2- 172.21.15.141:0/2344940368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006b700 0x7fc16015caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc164e00640 1 -- 172.21.15.141:0/2344940368 >> 172.21.15.141:0/2344940368 conn(0x7fc1600d0e30 msgr2=0x7fc1600d1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.845+0000 7fc164e00640 1 -- 172.21.15.141:0/2344940368 shutdown_connections 2024-01-16T15:15:04.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.846+0000 7fc164e00640 1 -- 172.21.15.141:0/2344940368 wait complete. 2024-01-16T15:15:04.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.847+0000 7fc164e00640 1 Processor -- start 2024-01-16T15:15:04.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.847+0000 7fc164e00640 1 -- start start 2024-01-16T15:15:04.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.848+0000 7fc164e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006b700 0x7fc16009ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.848+0000 7fc164e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006c610 0x7fc1601e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.848+0000 7fc164e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1600cde20 0x7fc16015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.848+0000 7fc164e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16009e1c0 con 0x7fc1600cde20 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.848+0000 7fc164e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16015b9b0 con 0x7fc16006b700 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.848+0000 7fc164e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16009d5b0 con 0x7fc16006c610 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1600cde20 0x7fc16015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1600cde20 0x7fc16015b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43112/0 (socket says 172.21.15.141:43112) 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 -- 172.21.15.141:0/4250130500 learned_addr learned my addr 172.21.15.141:0/4250130500 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15f400640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006b700 0x7fc16009ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc157e00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006c610 0x7fc1601e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 -- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006c610 msgr2=0x7fc1601e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006c610 0x7fc1601e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 -- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006b700 msgr2=0x7fc16009ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006b700 0x7fc16009ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.849+0000 7fc15fe00640 1 -- 172.21.15.141:0/4250130500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009400 con 0x7fc1600cde20 2024-01-16T15:15:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.850+0000 7fc15fe00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1600cde20 0x7fc16015b470 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fc15800efc0 tx=0x7fc15800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.850+0000 7fc157400640 1 -- 172.21.15.141:0/4250130500 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158014480 con 0x7fc1600cde20 2024-01-16T15:15:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.850+0000 7fc157400640 1 -- 172.21.15.141:0/4250130500 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15800f040 con 0x7fc1600cde20 2024-01-16T15:15:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.850+0000 7fc157400640 1 -- 172.21.15.141:0/4250130500 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158014820 con 0x7fc1600cde20 2024-01-16T15:15:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.850+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16009d890 con 0x7fc1600cde20 2024-01-16T15:15:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.850+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16009de50 con 0x7fc1600cde20 2024-01-16T15:15:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.851+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16009e330 con 0x7fc1600cde20 2024-01-16T15:15:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.852+0000 7fc157400640 1 -- 172.21.15.141:0/4250130500 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc158014980 con 0x7fc1600cde20 2024-01-16T15:15:04.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.853+0000 7fc157400640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc13806c430 0x7fc13806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.853+0000 7fc157400640 1 -- 172.21.15.141:0/4250130500 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc158089620 con 0x7fc1600cde20 2024-01-16T15:15:04.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.854+0000 7fc15f400640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc13806c430 0x7fc13806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.855+0000 7fc15f400640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc13806c430 0x7fc13806e8f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc150005880 tx=0x7fc150009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:04.858+0000 7fc157400640 1 -- 172.21.15.141:0/4250130500 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc158055b90 con 0x7fc1600cde20 2024-01-16T15:15:05.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.070+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1600b0c10 con 0x7fc13806c430 2024-01-16T15:15:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.088+0000 7fc157400640 1 -- 172.21.15.141:0/4250130500 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+185 (secure 0 0 0) 0x7fc1500014e0 con 0x7fc13806c430 2024-01-16T15:15:05.071 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:05.071 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:15:05.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:05.071 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "sv1" 2024-01-16T15:15:05.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone51672" 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone51671" 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone51673" 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:05.072 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:15:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc13806c430 msgr2=0x7fc13806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc13806c430 0x7fc13806e8f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc150005880 tx=0x7fc150009040 comp rx=0 tx=0).stop 2024-01-16T15:15:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1600cde20 msgr2=0x7fc16015b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1600cde20 0x7fc16015b470 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fc15800efc0 tx=0x7fc15800c490 comp rx=0 tx=0).stop 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 shutdown_connections 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc13806c430 0x7fc13806e8f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1600cde20 0x7fc16015b470 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc16006c610 0x7fc1601e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 --2- 172.21.15.141:0/4250130500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc16006b700 0x7fc16009ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 >> 172.21.15.141:0/4250130500 conn(0x7fc1600d0e30 msgr2=0x7fc16006e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 shutdown_connections 2024-01-16T15:15:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.090+0000 7fc164e00640 1 -- 172.21.15.141:0/4250130500 wait complete. 2024-01-16T15:15:05.092 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:15:05.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.310+0000 7fa8aca00640 1 Processor -- start 2024-01-16T15:15:05.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.310+0000 7fa8aca00640 1 -- start start 2024-01-16T15:15:05.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8aca00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa8a8143c40 0x7fa8a8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8aca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 0x7fa8a806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8aca00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a8070e50 con 0x7fa8a80717d0 2024-01-16T15:15:05.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8aca00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a80723c0 con 0x7fa8a8143c40 2024-01-16T15:15:05.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8aca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8144020 con 0x7fa8a80ccc10 2024-01-16T15:15:05.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8a5e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa8a80717d0 0x7fa8a80cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52566/0 (socket says 172.21.15.141:52566) 2024-01-16T15:15:05.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8a5e00640 1 -- 172.21.15.141:0/4048005623 learned_addr learned my addr 172.21.15.141:0/4048005623 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937038081 0 0) 0x7fa8a8070e50 con 0x7fa8a80717d0 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa888003520 con 0x7fa8a80717d0 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8a6800640 1 --2- 172.21.15.141:0/4048005623 >> v2:172.21.15.188:3300/0 conn(0x7fa8a8143c40 0x7fa8a8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.312+0000 7fa8a7200640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 0x7fa8a806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 723640294 0 0) 0x7fa888003520 con 0x7fa8a80717d0 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 msgr2=0x7fa8a806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 0x7fa8a806d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 >> v2:172.21.15.188:3300/0 conn(0x7fa8a8143c40 msgr2=0x7fa8a8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 --2- 172.21.15.141:0/4048005623 >> v2:172.21.15.188:3300/0 conn(0x7fa8a8143c40 0x7fa8a8071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a8071e00 con 0x7fa8a80717d0 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa890001670 con 0x7fa8a80717d0 2024-01-16T15:15:05.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a6800640 1 --2- 172.21.15.141:0/4048005623 >> v2:172.21.15.188:3300/0 conn(0x7fa8a8143c40 0x7fa8a8071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 >> v1:172.21.15.141:6789/0 conn(0x7fa8a80717d0 legacy=0x7fa8a80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 0x7fa888003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa888003ff0 0x7fa8880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888004d00 0x7fa888005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80723c0 con 0x7fa888003ff0 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8144020 con 0x7fa888004d00 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88800bb90 con 0x7fa8a80ccc10 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5e00640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa888003ff0 0x7fa8880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a6800640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 0x7fa888003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a7200640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888004d00 0x7fa888005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5e00640 1 -- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 msgr2=0x7fa888003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5e00640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80ccc10 0x7fa888003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5e00640 1 -- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888004d00 msgr2=0x7fa888005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:05.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.313+0000 7fa8a5e00640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888004d00 0x7fa888005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.314+0000 7fa8a5e00640 1 -- 172.21.15.141:0/4048005623 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888003520 con 0x7fa888003ff0 2024-01-16T15:15:05.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.314+0000 7fa8a7200640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888004d00 0x7fa888005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:05.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8a5e00640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa888003ff0 0x7fa8880047c0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fa89000c110 tx=0x7fa8880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6198ffb08112e6e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:05.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890005ad0 con 0x7fa888003ff0 2024-01-16T15:15:05.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa890005c50 con 0x7fa888003ff0 2024-01-16T15:15:05.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8a5400640 1 -- 172.21.15.141:0/4048005623 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89002c830 con 0x7fa888003ff0 2024-01-16T15:15:05.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8aca00640 1 -- 172.21.15.141:0/4048005623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa888003ff0 msgr2=0x7fa8880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8aca00640 1 --2- 172.21.15.141:0/4048005623 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa888003ff0 0x7fa8880047c0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fa89000c110 tx=0x7fa8880057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8a7200640 1 -- 172.21.15.141:0/4048005623 reap_dead start 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8aca00640 1 -- 172.21.15.141:0/4048005623 shutdown_connections 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8aca00640 1 -- 172.21.15.141:0/4048005623 >> 172.21.15.141:0/4048005623 conn(0x7fa8a806a800 msgr2=0x7fa8a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8aca00640 1 -- 172.21.15.141:0/4048005623 shutdown_connections 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.315+0000 7fa8aca00640 1 -- 172.21.15.141:0/4048005623 wait complete. 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.317+0000 7fa8aca00640 1 Processor -- start 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.317+0000 7fa8aca00640 1 -- start start 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8aca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888003ff0 0x7fa8a813c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8aca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8a8140f00 0x7fa8a8141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8aca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80b2c00 0x7fa8a80b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8aca00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80b35f0 con 0x7fa8a8140f00 2024-01-16T15:15:05.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8aca00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a81419a0 con 0x7fa888003ff0 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8aca00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8141c80 con 0x7fa8a80b2c00 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8a8140f00 0x7fa8a8141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8a8140f00 0x7fa8a8141380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43132/0 (socket says 172.21.15.141:43132) 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.318+0000 7fa8a5e00640 1 -- 172.21.15.141:0/1460592483 learned_addr learned my addr 172.21.15.141:0/1460592483 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a6800640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888003ff0 0x7fa8a813c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a7200640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80b2c00 0x7fa8a80b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a5e00640 1 -- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80b2c00 msgr2=0x7fa8a80b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a5e00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80b2c00 0x7fa8a80b30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a5e00640 1 -- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888003ff0 msgr2=0x7fa8a813c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a5e00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888003ff0 0x7fa8a813c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a5e00640 1 -- 172.21.15.141:0/1460592483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a8141f60 con 0x7fa8a8140f00 2024-01-16T15:15:05.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a6800640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888003ff0 0x7fa8a813c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:05.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a7200640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80b2c00 0x7fa8a80b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:05.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa8a5e00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8a8140f00 0x7fa8a8141380 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fa89002c3d0 tx=0x7fa890001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:05.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa89ea00640 1 -- 172.21.15.141:0/1460592483 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89003c040 con 0x7fa8a8140f00 2024-01-16T15:15:05.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.319+0000 7fa89ea00640 1 -- 172.21.15.141:0/1460592483 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89002dcd0 con 0x7fa8a8140f00 2024-01-16T15:15:05.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.320+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a8142240 con 0x7fa8a8140f00 2024-01-16T15:15:05.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.320+0000 7fa89ea00640 1 -- 172.21.15.141:0/1460592483 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900072b0 con 0x7fa8a8140f00 2024-01-16T15:15:05.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.320+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a80b3be0 con 0x7fa8a8140f00 2024-01-16T15:15:05.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.321+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a8142480 con 0x7fa8a8140f00 2024-01-16T15:15:05.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.322+0000 7fa89ea00640 1 -- 172.21.15.141:0/1460592483 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fa890032070 con 0x7fa8a8140f00 2024-01-16T15:15:05.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.322+0000 7fa89ea00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa88006c460 0x7fa88006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:05.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.322+0000 7fa89ea00640 1 -- 172.21.15.141:0/1460592483 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8900a9dc0 con 0x7fa8a8140f00 2024-01-16T15:15:05.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.323+0000 7fa8a6800640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa88006c460 0x7fa88006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:05.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.324+0000 7fa8a6800640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa88006c460 0x7fa88006e920 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa8a81f6310 tx=0x7fa898008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:05.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.326+0000 7fa89ea00640 1 -- 172.21.15.141:0/1460592483 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8a8142480 con 0x7fa8a8140f00 2024-01-16T15:15:05.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:05.533+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8a806cc40 con 0x7fa88006c460 2024-01-16T15:15:07.366 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:07.374 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:08.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.567+0000 7fa89ea00640 1 -- 172.21.15.141:0/1460592483 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fa89800c040 con 0x7fa88006c460 2024-01-16T15:15:08.551 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:08.551 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:15:08.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:08.551 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1" 2024-01-16T15:15:08.551 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:08.551 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:15:08.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.570+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa88006c460 msgr2=0x7fa88006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.570+0000 7fa8aca00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa88006c460 0x7fa88006e920 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa8a81f6310 tx=0x7fa898008040 comp rx=0 tx=0).stop 2024-01-16T15:15:08.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.570+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8a8140f00 msgr2=0x7fa8a8141380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.570+0000 7fa8aca00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8a8140f00 0x7fa8a8141380 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fa89002c3d0 tx=0x7fa890001930 comp rx=0 tx=0).stop 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 shutdown_connections 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa88006c460 0x7fa88006e920 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa8a80b2c00 0x7fa8a80b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa8a8140f00 0x7fa8a8141380 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 --2- 172.21.15.141:0/1460592483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888003ff0 0x7fa8a813c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 >> 172.21.15.141:0/1460592483 conn(0x7fa8a806a800 msgr2=0x7fa8a806d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 shutdown_connections 2024-01-16T15:15:08.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.571+0000 7fa8aca00640 1 -- 172.21.15.141:0/1460592483 wait complete. 2024-01-16T15:15:08.574 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:15:08.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.807+0000 7f001dc00640 1 Processor -- start 2024-01-16T15:15:08.791 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.808+0000 7f001dc00640 1 -- start start 2024-01-16T15:15:08.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.809+0000 7f001dc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f001813df80 0x7f001815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.809+0000 7f001dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f00180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.809+0000 7f001dc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001806c470 con 0x7f001815f9c0 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.809+0000 7f001dc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001806c750 con 0x7f001813df80 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.809+0000 7f001dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001806ca30 con 0x7f001813e360 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f001813df80 0x7f001815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f001813df80 0x7f001815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43136/0 (socket says 172.21.15.141:43136) 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016a00640 1 -- 172.21.15.141:0/1696969923 learned_addr learned my addr 172.21.15.141:0/1696969923 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016000640 1 --2- 172.21.15.141:0/1696969923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f00180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016a00640 1 -- 172.21.15.141:0/1696969923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 msgr2=0x7f00180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016a00640 1 --2- 172.21.15.141:0/1696969923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f00180cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016a00640 1 -- 172.21.15.141:0/1696969923 >> v1:172.21.15.188:6789/0 conn(0x7f001815f9c0 legacy=0x7f00181408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.794 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.810+0000 7f0016a00640 1 -- 172.21.15.141:0/1696969923 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001806cd10 con 0x7f001813df80 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.811+0000 7f0016a00640 1 --2- 172.21.15.141:0/1696969923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815bee0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f0008009870 tx=0x7f000802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e8797ff36101d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f0015600640 1 -- 172.21.15.141:0/1696969923 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008031640 con 0x7f001813df80 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f0015600640 1 -- 172.21.15.141:0/1696969923 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00080317a0 con 0x7f001813df80 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f0015600640 1 -- 172.21.15.141:0/1696969923 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008031b30 con 0x7f001813df80 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f001dc00640 1 -- 172.21.15.141:0/1696969923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 msgr2=0x7f001815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f001dc00640 1 --2- 172.21.15.141:0/1696969923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815bee0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f0008009870 tx=0x7f000802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f001dc00640 1 -- 172.21.15.141:0/1696969923 shutdown_connections 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f001dc00640 1 --2- 172.21.15.141:0/1696969923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f00180cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f001dc00640 1 --2- 172.21.15.141:0/1696969923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815bee0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f001dc00640 1 -- 172.21.15.141:0/1696969923 >> 172.21.15.141:0/1696969923 conn(0x7f001806a800 msgr2=0x7f001806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.812+0000 7f001dc00640 1 -- 172.21.15.141:0/1696969923 shutdown_connections 2024-01-16T15:15:08.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.813+0000 7f001dc00640 1 -- 172.21.15.141:0/1696969923 wait complete. 2024-01-16T15:15:08.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.815+0000 7f001dc00640 1 Processor -- start 2024-01-16T15:15:08.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.815+0000 7f001dc00640 1 -- start start 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f001dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f001dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f0018135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f001dc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f001815f9c0 0x7f0018135660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f001dc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018135ba0 con 0x7f001813df80 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f001dc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00181528a0 con 0x7f001815f9c0 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f001dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0018152b80 con 0x7f001813e360 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f0016a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f0016a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43152/0 (socket says 172.21.15.141:43152) 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.816+0000 7f0016a00640 1 -- 172.21.15.141:0/2543512573 learned_addr learned my addr 172.21.15.141:0/2543512573 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016000640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f0018135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0017400640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f001815f9c0 0x7f0018135660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016a00640 1 -- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 msgr2=0x7f0018135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016a00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f0018135120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016a00640 1 -- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f001815f9c0 msgr2=0x7f0018135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016a00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f001815f9c0 0x7f0018135660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016a00640 1 -- 172.21.15.141:0/2543512573 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00080094d0 con 0x7f001813df80 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016000640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f0018135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f0016a00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815db10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f0008009870 tx=0x7f00080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.817+0000 7f000ea00640 1 -- 172.21.15.141:0/2543512573 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000803ac60 con 0x7f001813df80 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.818+0000 7f000ea00640 1 -- 172.21.15.141:0/2543512573 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000803ade0 con 0x7f001813df80 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.818+0000 7f000ea00640 1 -- 172.21.15.141:0/2543512573 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008044630 con 0x7f001813df80 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.818+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0018152e60 con 0x7f001813df80 2024-01-16T15:15:08.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.818+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0018153420 con 0x7f001813df80 2024-01-16T15:15:08.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.819+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0018153610 con 0x7f001813df80 2024-01-16T15:15:08.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.820+0000 7f000ea00640 1 -- 172.21.15.141:0/2543512573 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f000802f050 con 0x7f001813df80 2024-01-16T15:15:08.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.821+0000 7f000ea00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efff006c430 0x7efff006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.821+0000 7f000ea00640 1 -- 172.21.15.141:0/2543512573 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00080aac40 con 0x7f001813df80 2024-01-16T15:15:08.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.822+0000 7f0016000640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efff006c430 0x7efff006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.823+0000 7f0016000640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efff006c430 0x7efff006e8f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f001815bc50 tx=0x7f0000008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:08.826+0000 7f000ea00640 1 -- 172.21.15.141:0/2543512573 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0008077180 con 0x7f001813df80 2024-01-16T15:15:09.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.049+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f001815ba80 con 0x7efff006c430 2024-01-16T15:15:09.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.118+0000 7f000ea00640 1 -- 172.21.15.141:0/2543512573 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f001815ba80 con 0x7efff006c430 2024-01-16T15:15:09.102 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efff006c430 msgr2=0x7efff006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efff006c430 0x7efff006e8f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f001815bc50 tx=0x7f0000008040 comp rx=0 tx=0).stop 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 msgr2=0x7f001815db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815db10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f0008009870 tx=0x7f00080046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 shutdown_connections 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7efff006c430 0x7efff006e8f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f001815f9c0 0x7f0018135660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f001813e360 0x7f0018135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 --2- 172.21.15.141:0/2543512573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f001813df80 0x7f001815db10 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 >> 172.21.15.141:0/2543512573 conn(0x7f001806a800 msgr2=0x7f00180b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 shutdown_connections 2024-01-16T15:15:09.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.122+0000 7f001dc00640 1 -- 172.21.15.141:0/2543512573 wait complete. 2024-01-16T15:15:09.126 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:15:09.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.358+0000 7fd7c4c00640 1 Processor -- start 2024-01-16T15:15:09.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.358+0000 7fd7c4c00640 1 -- start start 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7c4c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd7c014b2c0 0x7fd7c00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7c4c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd7c0071070 0x7fd7c00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7c4c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c00cfbf0 con 0x7fd7c00cf7d0 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7c4c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c014b6a0 con 0x7fd7c0071070 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7c4c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c0071690 con 0x7fd7c014b2c0 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7be800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd7c0071070 0x7fd7c00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7be800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd7c0071070 0x7fd7c00cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43166/0 (socket says 172.21.15.141:43166) 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.360+0000 7fd7be800640 1 -- 172.21.15.141:0/635257747 learned_addr learned my addr 172.21.15.141:0/635257747 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.361+0000 7fd7bde00640 1 --2- 172.21.15.141:0/635257747 >> v2:172.21.15.188:3300/0 conn(0x7fd7c014b2c0 0x7fd7c00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.361+0000 7fd7be800640 1 -- 172.21.15.141:0/635257747 >> v2:172.21.15.188:3300/0 conn(0x7fd7c014b2c0 msgr2=0x7fd7c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.361+0000 7fd7be800640 1 --2- 172.21.15.141:0/635257747 >> v2:172.21.15.188:3300/0 conn(0x7fd7c014b2c0 0x7fd7c00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.361+0000 7fd7be800640 1 -- 172.21.15.141:0/635257747 >> v1:172.21.15.188:6789/0 conn(0x7fd7c00cf7d0 legacy=0x7fd7c0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.361+0000 7fd7be800640 1 -- 172.21.15.141:0/635257747 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0071970 con 0x7fd7c0071070 2024-01-16T15:15:09.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.362+0000 7fd7be800640 1 --2- 172.21.15.141:0/635257747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c00cc160 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fd7c006e650 tx=0x7fd7b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b9affa14bd8e25 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.362+0000 7fd7bd400640 1 -- 172.21.15.141:0/635257747 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b802ca80 con 0x7fd7c0071070 2024-01-16T15:15:09.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.362+0000 7fd7bd400640 1 -- 172.21.15.141:0/635257747 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b802cbe0 con 0x7fd7c0071070 2024-01-16T15:15:09.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.362+0000 7fd7bd400640 1 -- 172.21.15.141:0/635257747 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b80343f0 con 0x7fd7c0071070 2024-01-16T15:15:09.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 -- 172.21.15.141:0/635257747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 msgr2=0x7fd7c00cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.346 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/635257747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c00cc160 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fd7c006e650 tx=0x7fd7b802bee0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 -- 172.21.15.141:0/635257747 shutdown_connections 2024-01-16T15:15:09.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/635257747 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c00cc160 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/635257747 >> v2:172.21.15.188:3300/0 conn(0x7fd7c014b2c0 0x7fd7c00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 -- 172.21.15.141:0/635257747 >> 172.21.15.141:0/635257747 conn(0x7fd7c006a800 msgr2=0x7fd7c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 -- 172.21.15.141:0/635257747 shutdown_connections 2024-01-16T15:15:09.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.363+0000 7fd7c4c00640 1 -- 172.21.15.141:0/635257747 wait complete. 2024-01-16T15:15:09.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.365+0000 7fd7c4c00640 1 Processor -- start 2024-01-16T15:15:09.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.365+0000 7fd7c4c00640 1 -- start start 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7c4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c01fb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7c4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c00cf7d0 0x7fd7c015d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7c4c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7c014b2c0 0x7fd7c0141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7c4c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c01fb9a0 con 0x7fd7c0071070 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7c4c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c015dd50 con 0x7fd7c00cf7d0 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7c4c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c01449a0 con 0x7fd7c014b2c0 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7bde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c01fb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7bde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c01fb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43178/0 (socket says 172.21.15.141:43178) 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.366+0000 7fd7bde00640 1 -- 172.21.15.141:0/140002984 learned_addr learned my addr 172.21.15.141:0/140002984 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7b7e00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c00cf7d0 0x7fd7c015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7be800640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7c014b2c0 0x7fd7c0141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7bde00640 1 -- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7c014b2c0 msgr2=0x7fd7c0141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7bde00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7c014b2c0 0x7fd7c0141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7bde00640 1 -- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c00cf7d0 msgr2=0x7fd7c015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7bde00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c00cf7d0 0x7fd7c015d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7bde00640 1 -- 172.21.15.141:0/140002984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b8009b30 con 0x7fd7c0071070 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7b7e00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c00cf7d0 0x7fd7c015d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7bde00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c01fb460 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000d5f0 tx=0x7fd7b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7b6000640 1 -- 172.21.15.141:0/140002984 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00125f0 con 0x7fd7c0071070 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7b6000640 1 -- 172.21.15.141:0/140002984 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b0012750 con 0x7fd7c0071070 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7b6000640 1 -- 172.21.15.141:0/140002984 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0012b00 con 0x7fd7c0071070 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c0144c80 con 0x7fd7c0071070 2024-01-16T15:15:09.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.367+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c0145240 con 0x7fd7c0071070 2024-01-16T15:15:09.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.369+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c00675d0 con 0x7fd7c0071070 2024-01-16T15:15:09.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.370+0000 7fd7b6000640 1 -- 172.21.15.141:0/140002984 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd7b0012c60 con 0x7fd7c0071070 2024-01-16T15:15:09.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.370+0000 7fd7b6000640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd79806c430 0x7fd79806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.371+0000 7fd7b6000640 1 -- 172.21.15.141:0/140002984 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7b0088f90 con 0x7fd7c0071070 2024-01-16T15:15:09.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.371+0000 7fd7b7e00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd79806c430 0x7fd79806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.372+0000 7fd7b7e00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd79806c430 0x7fd79806e8f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd7c014a750 tx=0x7fd7a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.375+0000 7fd7b6000640 1 -- 172.21.15.141:0/140002984 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7b0055500 con 0x7fd7c0071070 2024-01-16T15:15:09.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.593+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7c00b0540 con 0x7fd79806c430 2024-01-16T15:15:09.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.641+0000 7fd7b6000640 1 -- 172.21.15.141:0/140002984 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd7c00b0540 con 0x7fd79806c430 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd79806c430 msgr2=0x7fd79806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd79806c430 0x7fd79806e8f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd7c014a750 tx=0x7fd7a8009040 comp rx=0 tx=0).stop 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 msgr2=0x7fd7c01fb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c01fb460 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000d5f0 tx=0x7fd7b000dab0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 shutdown_connections 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd79806c430 0x7fd79806e8f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7c014b2c0 0x7fd7c0141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c00cf7d0 0x7fd7c015d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 --2- 172.21.15.141:0/140002984 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7c0071070 0x7fd7c01fb460 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 >> 172.21.15.141:0/140002984 conn(0x7fd7c006a800 msgr2=0x7fd7c00aeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 shutdown_connections 2024-01-16T15:15:09.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.643+0000 7fd7c4c00640 1 -- 172.21.15.141:0/140002984 wait complete. 2024-01-16T15:15:09.633 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.649+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.633 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.649+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.633 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.649+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.633 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.649+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.633 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.649+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51672 2024-01-16T15:15:09.655 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.672+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.656 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.672+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.656 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.672+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.656 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.672+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.656 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:09.672+0000 7f0f2e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:09.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.865+0000 7f1286e00640 1 Processor -- start 2024-01-16T15:15:09.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.865+0000 7f1286e00640 1 -- start start 2024-01-16T15:15:09.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f1286e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 0x7f1280160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f1286e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f12800d3910 0x7f1280160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f1286e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800d4160 con 0x7f12800d00d0 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f1286e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12800ceff0 con 0x7f12800d3910 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f1286e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12800cf410 con 0x7f12800cfcf0 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f1284e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f12800d00d0 0x7f12800b2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52574/0 (socket says 172.21.15.141:52574) 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f1284e00640 1 -- 172.21.15.141:0/901137070 learned_addr learned my addr 172.21.15.141:0/901137070 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f127f400640 1 --2- 172.21.15.141:0/901137070 >> v2:172.21.15.188:3300/0 conn(0x7f12800d3910 0x7f1280160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f127fe00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 0x7f1280160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.867+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808657508 0 0) 0x7f12800d4160 con 0x7f12800d00d0 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1260003520 con 0x7f12800d00d0 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903450912 0 0) 0x7f1260003520 con 0x7f12800d00d0 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 msgr2=0x7f1280160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 0x7f1280160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 >> v2:172.21.15.188:3300/0 conn(0x7f12800d3910 msgr2=0x7f1280160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 --2- 172.21.15.141:0/901137070 >> v2:172.21.15.188:3300/0 conn(0x7f12800d3910 0x7f1280160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800b2420 con 0x7f12800d00d0 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f12780019f0 con 0x7f12800d00d0 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 >> v1:172.21.15.141:6789/0 conn(0x7f12800d00d0 legacy=0x7f12800b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12600039a0 0x7f1260003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 0x7f12600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1260004d00 0x7f1260005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800ceff0 con 0x7f12600039a0 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800cf410 con 0x7f1260004d00 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126000b9b0 con 0x7f12800cfcf0 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.868+0000 7f127fe00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12600039a0 0x7f1260003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.869+0000 7f1284e00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1260004d00 0x7f1260005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.869+0000 7f127f400640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 0x7f12600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.869+0000 7f127fe00640 1 -- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 msgr2=0x7f12600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.869+0000 7f127fe00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12800cfcf0 0x7f12600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.869+0000 7f127fe00640 1 -- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1260004d00 msgr2=0x7f1260005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.869+0000 7f127fe00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1260004d00 0x7f1260005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.869+0000 7f127fe00640 1 -- 172.21.15.141:0/901137070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1260003520 con 0x7f12600039a0 2024-01-16T15:15:09.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.870+0000 7f127fe00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12600039a0 0x7f1260003dc0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f1270009ac0 tx=0x7f12600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343b2c8960e5f51f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.870+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127002c6b0 con 0x7f12600039a0 2024-01-16T15:15:09.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.870+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127002c810 con 0x7f12600039a0 2024-01-16T15:15:09.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.870+0000 7f127ea00640 1 -- 172.21.15.141:0/901137070 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127002cba0 con 0x7f12600039a0 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.871+0000 7f1286e00640 1 -- 172.21.15.141:0/901137070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12600039a0 msgr2=0x7f1260003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.871+0000 7f1286e00640 1 --2- 172.21.15.141:0/901137070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12600039a0 0x7f1260003dc0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f1270009ac0 tx=0x7f12600057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.871+0000 7f1284e00640 1 -- 172.21.15.141:0/901137070 reap_dead start 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.871+0000 7f1286e00640 1 -- 172.21.15.141:0/901137070 shutdown_connections 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.871+0000 7f1286e00640 1 -- 172.21.15.141:0/901137070 >> 172.21.15.141:0/901137070 conn(0x7f128009c530 msgr2=0x7f12800d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.871+0000 7f1286e00640 1 -- 172.21.15.141:0/901137070 shutdown_connections 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.871+0000 7f1286e00640 1 -- 172.21.15.141:0/901137070 wait complete. 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.873+0000 7f1286e00640 1 Processor -- start 2024-01-16T15:15:09.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.873+0000 7f1286e00640 1 -- start start 2024-01-16T15:15:09.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.874+0000 7f1286e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12600039a0 0x7f128015ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.874+0000 7f1286e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12800c6ca0 0x7f12800c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.874+0000 7f1286e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f128014b180 0x7f128014b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.874+0000 7f1286e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280151c50 con 0x7f128014b180 2024-01-16T15:15:09.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.874+0000 7f1286e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12801f6690 con 0x7f12800c6ca0 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.874+0000 7f1286e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12801f6f30 con 0x7f12600039a0 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f128014b180 0x7f128014b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f128014b180 0x7f128014b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43190/0 (socket says 172.21.15.141:43190) 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 -- 172.21.15.141:0/2832080634 learned_addr learned my addr 172.21.15.141:0/2832080634 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f127fe00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12600039a0 0x7f128015ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f127f400640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12800c6ca0 0x7f12800c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 -- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12600039a0 msgr2=0x7f128015ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12600039a0 0x7f128015ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 -- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12800c6ca0 msgr2=0x7f12800c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12800c6ca0 0x7f12800c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.875+0000 7f1284e00640 1 -- 172.21.15.141:0/2832080634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800c7660 con 0x7f128014b180 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.876+0000 7f1284e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f128014b180 0x7f128014b600 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f12800cf6b0 tx=0x7f127800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.876+0000 7f1277e00640 1 -- 172.21.15.141:0/2832080634 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278007ae0 con 0x7f128014b180 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.876+0000 7f1277e00640 1 -- 172.21.15.141:0/2832080634 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127800eda0 con 0x7f128014b180 2024-01-16T15:15:09.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.876+0000 7f1277e00640 1 -- 172.21.15.141:0/2832080634 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278014690 con 0x7f128014b180 2024-01-16T15:15:09.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.876+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128014bb40 con 0x7f128014b180 2024-01-16T15:15:09.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.876+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12801f68e0 con 0x7f128014b180 2024-01-16T15:15:09.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.877+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12801f70c0 con 0x7f128014b180 2024-01-16T15:15:09.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.879+0000 7f1277e00640 1 -- 172.21.15.141:0/2832080634 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f12780147f0 con 0x7f128014b180 2024-01-16T15:15:09.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.880+0000 7f1277e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f125806c460 0x7f125806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.880+0000 7f1277e00640 1 -- 172.21.15.141:0/2832080634 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1278088830 con 0x7f128014b180 2024-01-16T15:15:09.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.881+0000 7f127fe00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f125806c460 0x7f125806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.882+0000 7f127fe00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f125806c460 0x7f125806e920 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1270036000 tx=0x7f1270030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:09.885+0000 7f1277e00640 1 -- 172.21.15.141:0/2832080634 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1278054da0 con 0x7f128014b180 2024-01-16T15:15:10.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.106+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51672", "target": ["mon-mgr", ""]}) v1 -- 0x7f12800cec10 con 0x7f125806c460 2024-01-16T15:15:10.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.148+0000 7f1277e00640 1 -- 172.21.15.141:0/2832080634 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f12700040c0 con 0x7f125806c460 2024-01-16T15:15:10.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f125806c460 msgr2=0x7f125806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f125806c460 0x7f125806e920 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1270036000 tx=0x7f1270030000 comp rx=0 tx=0).stop 2024-01-16T15:15:10.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f128014b180 msgr2=0x7f128014b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f128014b180 0x7f128014b600 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f12800cf6b0 tx=0x7f127800e530 comp rx=0 tx=0).stop 2024-01-16T15:15:10.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 shutdown_connections 2024-01-16T15:15:10.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f125806c460 0x7f125806e920 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f128014b180 0x7f128014b600 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12800c6ca0 0x7f12800c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 --2- 172.21.15.141:0/2832080634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12600039a0 0x7f128015ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 >> 172.21.15.141:0/2832080634 conn(0x7f128009c530 msgr2=0x7f128015ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 shutdown_connections 2024-01-16T15:15:10.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.151+0000 7f1286e00640 1 -- 172.21.15.141:0/2832080634 wait complete. 2024-01-16T15:15:10.154 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51671 2024-01-16T15:15:10.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.366+0000 7f7565000640 1 Processor -- start 2024-01-16T15:15:10.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.366+0000 7f7565000640 1 -- start start 2024-01-16T15:15:10.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f7565000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f75600cd960 0x7f756015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f7565000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f756006f550 0x7f756006f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f7565000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756006eb00 con 0x7f756006a3e0 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f7565000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756006f0c0 con 0x7f756006f550 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f7565000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756006ab80 con 0x7f75600cd960 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f755e800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f756006f550 0x7f756006f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f755d400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f756006a3e0 0x7f75600cdd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52578/0 (socket says 172.21.15.141:52578) 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.368+0000 7f755d400640 1 -- 172.21.15.141:0/3549614994 learned_addr learned my addr 172.21.15.141:0/3549614994 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.369+0000 7f755de00640 1 --2- 172.21.15.141:0/3549614994 >> v2:172.21.15.188:3300/0 conn(0x7f75600cd960 0x7f756015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.369+0000 7f755e800640 1 -- 172.21.15.141:0/3549614994 >> v2:172.21.15.188:3300/0 conn(0x7f75600cd960 msgr2=0x7f756015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.369+0000 7f755e800640 1 --2- 172.21.15.141:0/3549614994 >> v2:172.21.15.188:3300/0 conn(0x7f75600cd960 0x7f756015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.369+0000 7f755e800640 1 -- 172.21.15.141:0/3549614994 >> v1:172.21.15.141:6789/0 conn(0x7f756006a3e0 legacy=0x7f75600cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.353 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.369+0000 7f755e800640 1 -- 172.21.15.141:0/3549614994 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756006ae60 con 0x7f756006f550 2024-01-16T15:15:10.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.369+0000 7f755de00640 1 --2- 172.21.15.141:0/3549614994 >> v2:172.21.15.188:3300/0 conn(0x7f75600cd960 0x7f756015c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:10.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.369+0000 7f755ca00640 1 -- 172.21.15.141:0/3549614994 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758992641 0 0) 0x7f756006eb00 con 0x7f756006a3e0 2024-01-16T15:15:10.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.370+0000 7f755e800640 1 --2- 172.21.15.141:0/3549614994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f756006f9e0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f75600b5500 tx=0x7f755802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bde8385ebbcef4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.370+0000 7f755ca00640 1 -- 172.21.15.141:0/3549614994 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755802cbf0 con 0x7f756006f550 2024-01-16T15:15:10.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.370+0000 7f755ca00640 1 -- 172.21.15.141:0/3549614994 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755802cd50 con 0x7f756006f550 2024-01-16T15:15:10.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.370+0000 7f755ca00640 1 -- 172.21.15.141:0/3549614994 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558034660 con 0x7f756006f550 2024-01-16T15:15:10.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 -- 172.21.15.141:0/3549614994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 msgr2=0x7f756006f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 --2- 172.21.15.141:0/3549614994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f756006f9e0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f75600b5500 tx=0x7f755802be40 comp rx=0 tx=0).stop 2024-01-16T15:15:10.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 -- 172.21.15.141:0/3549614994 shutdown_connections 2024-01-16T15:15:10.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 --2- 172.21.15.141:0/3549614994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f756006f9e0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 --2- 172.21.15.141:0/3549614994 >> v2:172.21.15.188:3300/0 conn(0x7f75600cd960 0x7f756015c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 -- 172.21.15.141:0/3549614994 >> 172.21.15.141:0/3549614994 conn(0x7f75600b1dd0 msgr2=0x7f75600b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 -- 172.21.15.141:0/3549614994 shutdown_connections 2024-01-16T15:15:10.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.371+0000 7f7565000640 1 -- 172.21.15.141:0/3549614994 wait complete. 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.372+0000 7f7565000640 1 Processor -- start 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.373+0000 7f7565000640 1 -- start start 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.373+0000 7f7565000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f756006a3e0 0x7f75600ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.373+0000 7f7565000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f75600ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.373+0000 7f7565000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75600cd960 0x7f7560133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.373+0000 7f7565000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600acec0 con 0x7f756006f550 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.373+0000 7f7565000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600ac8d0 con 0x7f75600cd960 2024-01-16T15:15:10.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.373+0000 7f7565000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7560134830 con 0x7f756006a3e0 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f75600ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f75600ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43212/0 (socket says 172.21.15.141:43212) 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 -- 172.21.15.141:0/1976659233 learned_addr learned my addr 172.21.15.141:0/1976659233 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755e800640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75600cd960 0x7f7560133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 -- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f756006a3e0 msgr2=0x7f75600ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f756006a3e0 0x7f75600ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 -- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75600cd960 msgr2=0x7f7560133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75600cd960 0x7f7560133d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 -- 172.21.15.141:0/1976659233 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558009a70 con 0x7f756006f550 2024-01-16T15:15:10.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.374+0000 7f755d400640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f75600ad2e0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f754800c430 tx=0x7f754800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.375+0000 7f7556000640 1 -- 172.21.15.141:0/1976659233 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75480074e0 con 0x7f756006f550 2024-01-16T15:15:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.375+0000 7f7556000640 1 -- 172.21.15.141:0/1976659233 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7548007660 con 0x7f756006f550 2024-01-16T15:15:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.375+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7560134b10 con 0x7f756006f550 2024-01-16T15:15:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.375+0000 7f7556000640 1 -- 172.21.15.141:0/1976659233 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548004a20 con 0x7f756006f550 2024-01-16T15:15:10.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.375+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756013efb0 con 0x7f756006f550 2024-01-16T15:15:10.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.376+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560134ca0 con 0x7f756006f550 2024-01-16T15:15:10.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.377+0000 7f7556000640 1 -- 172.21.15.141:0/1976659233 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f7548004b80 con 0x7f756006f550 2024-01-16T15:15:10.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.377+0000 7f7556000640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f753806c460 0x7f753806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.378+0000 7f7556000640 1 -- 172.21.15.141:0/1976659233 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7548088940 con 0x7f756006f550 2024-01-16T15:15:10.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.378+0000 7f755de00640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f753806c460 0x7f753806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.379+0000 7f755de00640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f753806c460 0x7f753806e920 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7550006fd0 tx=0x7f7550008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.382+0000 7f7556000640 1 -- 172.21.15.141:0/1976659233 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7548054eb0 con 0x7f756006f550 2024-01-16T15:15:10.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.598+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51671", "target": ["mon-mgr", ""]}) v1 -- 0x7f756006a860 con 0x7f753806c460 2024-01-16T15:15:10.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.655+0000 7f7556000640 1 -- 172.21.15.141:0/1976659233 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f755000ae40 con 0x7f753806c460 2024-01-16T15:15:10.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f753806c460 msgr2=0x7f753806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f753806c460 0x7f753806e920 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7550006fd0 tx=0x7f7550008040 comp rx=0 tx=0).stop 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 msgr2=0x7f75600ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f75600ad2e0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f754800c430 tx=0x7f754800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 shutdown_connections 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f753806c460 0x7f753806e920 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75600cd960 0x7f7560133d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f756006f550 0x7f75600ad2e0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 --2- 172.21.15.141:0/1976659233 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f756006a3e0 0x7f75600ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 >> 172.21.15.141:0/1976659233 conn(0x7f75600b1dd0 msgr2=0x7f7560140fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 shutdown_connections 2024-01-16T15:15:10.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.658+0000 7f7565000640 1 -- 172.21.15.141:0/1976659233 wait complete. 2024-01-16T15:15:10.659 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51673 2024-01-16T15:15:10.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.884+0000 7fc75a400640 1 Processor -- start 2024-01-16T15:15:10.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.884+0000 7fc75a400640 1 -- start start 2024-01-16T15:15:10.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc75a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc754143ae0 0x7fc7540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc75a400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc7540cf7d0 0x7fc75415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc75a400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc75415e1e0 0x7fc75415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc75a400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7540cfc30 con 0x7fc75415e1e0 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc75a400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7540b1ee0 con 0x7fc7540cf7d0 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc75a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540b2780 con 0x7fc754143ae0 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc753e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc75415e1e0 0x7fc75415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc753e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc75415e1e0 0x7fc75415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43214/0 (socket says 172.21.15.141:43214) 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.886+0000 7fc753e00640 1 -- 172.21.15.141:0/2533896204 learned_addr learned my addr 172.21.15.141:0/2533896204 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.887+0000 7fc753400640 1 --2- 172.21.15.141:0/2533896204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc754143ae0 0x7fc7540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.887+0000 7fc752a00640 1 --2- 172.21.15.141:0/2533896204 >> v2:172.21.15.188:3300/0 conn(0x7fc7540cf7d0 0x7fc75415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.887+0000 7fc753e00640 1 -- 172.21.15.141:0/2533896204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc754143ae0 msgr2=0x7fc7540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.887+0000 7fc753e00640 1 --2- 172.21.15.141:0/2533896204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc754143ae0 0x7fc7540cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.887+0000 7fc753e00640 1 -- 172.21.15.141:0/2533896204 >> v2:172.21.15.188:3300/0 conn(0x7fc7540cf7d0 msgr2=0x7fc75415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.887+0000 7fc753e00640 1 --2- 172.21.15.141:0/2533896204 >> v2:172.21.15.188:3300/0 conn(0x7fc7540cf7d0 0x7fc75415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.887+0000 7fc753e00640 1 -- 172.21.15.141:0/2533896204 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75415e990 con 0x7fc75415e1e0 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.888+0000 7fc753e00640 1 --2- 172.21.15.141:0/2533896204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc75415e1e0 0x7fc75415a670 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fc75406e350 tx=0x7fc74c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2a7b73ff75d637 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.888+0000 7fc752000640 1 -- 172.21.15.141:0/2533896204 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c02ee90 con 0x7fc75415e1e0 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.888+0000 7fc752000640 1 -- 172.21.15.141:0/2533896204 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c036440 con 0x7fc75415e1e0 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc752000640 1 -- 172.21.15.141:0/2533896204 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0367d0 con 0x7fc75415e1e0 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 -- 172.21.15.141:0/2533896204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc75415e1e0 msgr2=0x7fc75415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 --2- 172.21.15.141:0/2533896204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc75415e1e0 0x7fc75415a670 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fc75406e350 tx=0x7fc74c02e0e0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 -- 172.21.15.141:0/2533896204 shutdown_connections 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 --2- 172.21.15.141:0/2533896204 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc75415e1e0 0x7fc75415a670 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 --2- 172.21.15.141:0/2533896204 >> v2:172.21.15.188:3300/0 conn(0x7fc7540cf7d0 0x7fc75415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 --2- 172.21.15.141:0/2533896204 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc754143ae0 0x7fc7540cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 -- 172.21.15.141:0/2533896204 >> 172.21.15.141:0/2533896204 conn(0x7fc75406a800 msgr2=0x7fc75406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 -- 172.21.15.141:0/2533896204 shutdown_connections 2024-01-16T15:15:10.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.889+0000 7fc75a400640 1 -- 172.21.15.141:0/2533896204 wait complete. 2024-01-16T15:15:10.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.891+0000 7fc75a400640 1 Processor -- start 2024-01-16T15:15:10.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.891+0000 7fc75a400640 1 -- start start 2024-01-16T15:15:10.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.892+0000 7fc75a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7540cf7d0 0x7fc7540c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.892+0000 7fc75a400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc754143ae0 0x7fc754141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.892+0000 7fc75a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc75415e1e0 0x7fc7541f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.892+0000 7fc75a400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7541f66b0 con 0x7fc7540cf7d0 2024-01-16T15:15:10.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.892+0000 7fc75a400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7541f6990 con 0x7fc754143ae0 2024-01-16T15:15:10.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.892+0000 7fc75a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540c5fe0 con 0x7fc75415e1e0 2024-01-16T15:15:10.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.892+0000 7fc753400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7540cf7d0 0x7fc7540c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7540cf7d0 0x7fc7540c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:43228/0 (socket says 172.21.15.141:43228) 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 -- 172.21.15.141:0/3456955515 learned_addr learned my addr 172.21.15.141:0/3456955515 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 -- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc75415e1e0 msgr2=0x7fc7541f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc75415e1e0 0x7fc7541f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 -- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc754143ae0 msgr2=0x7fc754141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc754143ae0 0x7fc754141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 -- 172.21.15.141:0/3456955515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c00b770 con 0x7fc7540cf7d0 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc753400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7540cf7d0 0x7fc7540c6ac0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00d800 tx=0x7fc73c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00b6c0 con 0x7fc7540cf7d0 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c00b820 con 0x7fc7540cf7d0 2024-01-16T15:15:10.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.893+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7540c62c0 con 0x7fc7540cf7d0 2024-01-16T15:15:10.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.894+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00bbb0 con 0x7fc7540cf7d0 2024-01-16T15:15:10.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.894+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7540c7000 con 0x7fc7540cf7d0 2024-01-16T15:15:10.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.895+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7540675d0 con 0x7fc7540cf7d0 2024-01-16T15:15:10.879 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.896+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 38) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc73c00bd10 con 0x7fc7540cf7d0 2024-01-16T15:15:10.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.897+0000 7fc74b400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc72c06c430 0x7fc72c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.897+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc73c089060 con 0x7fc7540cf7d0 2024-01-16T15:15:10.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.897+0000 7fc752a00640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc72c06c430 0x7fc72c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.882 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.899+0000 7fc752a00640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc72c06c430 0x7fc72c06e8f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc744009540 tx=0x7fc744009450 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:10.901+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc73c0555d0 con 0x7fc7540cf7d0 2024-01-16T15:15:11.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:11.107+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51673", "target": ["mon-mgr", ""]}) v1 -- 0x7fc75415e640 con 0x7fc72c06c430 2024-01-16T15:15:11.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:11.731+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 39) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fc73c0559a0 con 0x7fc7540cf7d0 2024-01-16T15:15:18.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.564+0000 7fc74b400640 1 -- 172.21.15.141:0/3456955515 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc744009670 con 0x7fc72c06c430 2024-01-16T15:15:18.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc72c06c430 msgr2=0x7fc72c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc72c06c430 0x7fc72c06e8f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc744009540 tx=0x7fc744009450 comp rx=0 tx=0).stop 2024-01-16T15:15:18.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7540cf7d0 msgr2=0x7fc7540c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7540cf7d0 0x7fc7540c6ac0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00d800 tx=0x7fc73c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 shutdown_connections 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc72c06c430 0x7fc72c06e8f0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc75415e1e0 0x7fc7541f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc754143ae0 0x7fc754141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 --2- 172.21.15.141:0/3456955515 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7540cf7d0 0x7fc7540c6ac0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 >> 172.21.15.141:0/3456955515 conn(0x7fc75406a800 msgr2=0x7fc75406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.568+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 shutdown_connections 2024-01-16T15:15:18.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.569+0000 7fc75a400640 1 -- 172.21.15.141:0/3456955515 wait complete. 2024-01-16T15:15:18.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:18.574 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:15:18.638 INFO:teuthology.orchestra.run.smithi141.stdout:2 2024-01-16T15:15:18.639 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:15:18.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.864+0000 7ff1d3c00640 1 Processor -- start 2024-01-16T15:15:18.848 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.864+0000 7ff1d3c00640 1 -- start start 2024-01-16T15:15:18.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d3c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc143ae0 0x7ff1cc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d3c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff1cc15e3b0 0x7ff1cc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d3c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1cc0cfc30 con 0x7ff1cc0cf7d0 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d3c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc0b1ee0 con 0x7ff1cc15e3b0 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d3c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0b2780 con 0x7ff1cc143ae0 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d1c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff1cc15e3b0 0x7ff1cc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d1c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff1cc15e3b0 0x7ff1cc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44160/0 (socket says 172.21.15.141:44160) 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.866+0000 7ff1d1c00640 1 -- 172.21.15.141:0/537381394 learned_addr learned my addr 172.21.15.141:0/537381394 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.867+0000 7ff1d1200640 1 --2- 172.21.15.141:0/537381394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc143ae0 0x7ff1cc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.867+0000 7ff1d1c00640 1 -- 172.21.15.141:0/537381394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc143ae0 msgr2=0x7ff1cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.867+0000 7ff1d1c00640 1 --2- 172.21.15.141:0/537381394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc143ae0 0x7ff1cc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.867+0000 7ff1d1c00640 1 -- 172.21.15.141:0/537381394 >> v1:172.21.15.188:6789/0 conn(0x7ff1cc0cf7d0 legacy=0x7ff1cc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.867+0000 7ff1d1c00640 1 -- 172.21.15.141:0/537381394 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc15e990 con 0x7ff1cc15e3b0 2024-01-16T15:15:18.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.868+0000 7ff1d1c00640 1 --2- 172.21.15.141:0/537381394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc15e3b0 0x7ff1cc0702c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009870 tx=0x7ff1c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d510043248ca7923 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1cbe00640 1 -- 172.21.15.141:0/537381394 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4031640 con 0x7ff1cc15e3b0 2024-01-16T15:15:18.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1cbe00640 1 -- 172.21.15.141:0/537381394 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c40317a0 con 0x7ff1cc15e3b0 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1cbe00640 1 -- 172.21.15.141:0/537381394 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4031b30 con 0x7ff1cc15e3b0 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1d3c00640 1 -- 172.21.15.141:0/537381394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc15e3b0 msgr2=0x7ff1cc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/537381394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc15e3b0 0x7ff1cc0702c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009870 tx=0x7ff1c402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1d3c00640 1 -- 172.21.15.141:0/537381394 shutdown_connections 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/537381394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc15e3b0 0x7ff1cc0702c0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/537381394 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc143ae0 0x7ff1cc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.869+0000 7ff1d3c00640 1 -- 172.21.15.141:0/537381394 >> 172.21.15.141:0/537381394 conn(0x7ff1cc06a800 msgr2=0x7ff1cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.870+0000 7ff1d3c00640 1 -- 172.21.15.141:0/537381394 shutdown_connections 2024-01-16T15:15:18.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.870+0000 7ff1d3c00640 1 -- 172.21.15.141:0/537381394 wait complete. 2024-01-16T15:15:18.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.872+0000 7ff1d3c00640 1 Processor -- start 2024-01-16T15:15:18.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.872+0000 7ff1d3c00640 1 -- start start 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1d3c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1cc0cf7d0 0x7ff1cc13a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1d3c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc143ae0 0x7ff1cc13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1d3c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc15e3b0 0x7ff1cc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1d3c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc13c8c0 con 0x7ff1cc143ae0 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1d3c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc13c020 con 0x7ff1cc0cf7d0 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1d3c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc13c300 con 0x7ff1cc15e3b0 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1c3c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc143ae0 0x7ff1cc13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1c3c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc143ae0 0x7ff1cc13ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44174/0 (socket says 172.21.15.141:44174) 2024-01-16T15:15:18.857 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.873+0000 7ff1c3c00640 1 -- 172.21.15.141:0/1811993993 learned_addr learned my addr 172.21.15.141:0/1811993993 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1d1200640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1cc0cf7d0 0x7ff1cc13a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1d1c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc15e3b0 0x7ff1cc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c3c00640 1 -- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc15e3b0 msgr2=0x7ff1cc13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc15e3b0 0x7ff1cc13f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c3c00640 1 -- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1cc0cf7d0 msgr2=0x7ff1cc13a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1cc0cf7d0 0x7ff1cc13a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c3c00640 1 -- 172.21.15.141:0/1811993993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40094d0 con 0x7ff1cc143ae0 2024-01-16T15:15:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc143ae0 0x7ff1cc13ef90 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7ff1b400c7c0 tx=0x7ff1b400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b400aa50 con 0x7ff1cc143ae0 2024-01-16T15:15:18.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b400abb0 con 0x7ff1cc143ae0 2024-01-16T15:15:18.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4012420 con 0x7ff1cc143ae0 2024-01-16T15:15:18.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.874+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc13c5e0 con 0x7ff1cc143ae0 2024-01-16T15:15:18.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.875+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc13ace0 con 0x7ff1cc143ae0 2024-01-16T15:15:18.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.876+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc0aea20 con 0x7ff1cc143ae0 2024-01-16T15:15:18.861 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.877+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 39) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff1b4012580 con 0x7ff1cc143ae0 2024-01-16T15:15:18.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.878+0000 7ff1c9600640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1a406c550 0x7ff1a406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.878+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff1b4088aa0 con 0x7ff1cc143ae0 2024-01-16T15:15:18.862 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.879+0000 7ff1d1200640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1a406c550 0x7ff1a406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.880+0000 7ff1d1200640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1a406c550 0x7ff1a406ea10 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc006fd0 tx=0x7ff1bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:18.883+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1cc0aea20 con 0x7ff1cc143ae0 2024-01-16T15:15:19.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:19.102+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1cc0b2410 con 0x7ff1a406c550 2024-01-16T15:15:19.106 INFO:tasks.ceph.mon.a.smithi141.stderr:2024-01-16T15:15:19.121+0000 7f723b200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:15:19.218 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T15:15:21.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:21.783+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff1b4055410 con 0x7ff1cc143ae0 2024-01-16T15:15:22.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.147+0000 7ff1c9600640 1 -- 172.21.15.141:0/1811993993 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7ff1bc00b200 con 0x7ff1a406c550 2024-01-16T15:15:22.134 INFO:teuthology.orchestra.run.smithi141.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:15:22.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1a406c550 msgr2=0x7ff1a406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1a406c550 0x7ff1a406ea10 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc006fd0 tx=0x7ff1bc008040 comp rx=0 tx=0).stop 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc143ae0 msgr2=0x7ff1cc13ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc143ae0 0x7ff1cc13ef90 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7ff1b400c7c0 tx=0x7ff1b400cc80 comp rx=0 tx=0).stop 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 shutdown_connections 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1a406c550 0x7ff1a406ea10 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1cc15e3b0 0x7ff1cc13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1cc143ae0 0x7ff1cc13ef90 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 --2- 172.21.15.141:0/1811993993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1cc0cf7d0 0x7ff1cc13a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.154+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 >> 172.21.15.141:0/1811993993 conn(0x7ff1cc06a800 msgr2=0x7ff1cc15a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.155+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 shutdown_connections 2024-01-16T15:15:22.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.155+0000 7ff1d3c00640 1 -- 172.21.15.141:0/1811993993 wait complete. 2024-01-16T15:15:22.166 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:15:22.166 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:22.262 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:22.377 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:15:22.377 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:15:22.377 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:22.435 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:22.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.661+0000 7fec28400640 1 Processor -- start 2024-01-16T15:15:22.646 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.661+0000 7fec28400640 1 -- start start 2024-01-16T15:15:22.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.662+0000 7fec28400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 0x7fec2015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec28400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200b1ee0 con 0x7fec2015e1e0 2024-01-16T15:15:22.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec28400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200b2780 con 0x7fec20143ae0 2024-01-16T15:15:22.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec28400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015e990 con 0x7fec200cf9a0 2024-01-16T15:15:22.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec26400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fec2015e1e0 0x7fec200702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52546/0 (socket says 172.21.15.141:52546) 2024-01-16T15:15:22.648 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec26400640 1 -- 172.21.15.141:0/2187593368 learned_addr learned my addr 172.21.15.141:0/2187593368 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:22.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec25000640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 0x7fec2015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145705549 0 0) 0x7fec200b1ee0 con 0x7fec2015e1e0 2024-01-16T15:15:22.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.663+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec00005730 con 0x7fec2015e1e0 2024-01-16T15:15:22.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1843016184 0 0) 0x7fec00005730 con 0x7fec2015e1e0 2024-01-16T15:15:22.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 msgr2=0x7fec2015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 0x7fec2015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 >> v1:172.21.15.188:6789/0 conn(0x7fec20143ae0 legacy=0x7fec200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec200b2130 con 0x7fec2015e1e0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fec18001670 con 0x7fec2015e1e0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 >> v1:172.21.15.141:6789/0 conn(0x7fec2015e1e0 legacy=0x7fec200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec00005bb0 0x7fec00005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000069d0 0x7fec00006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 0x7fec00007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015e990 con 0x7fec000069d0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0000dc40 con 0x7fec00005bb0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0000de00 con 0x7fec200cf9a0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.664+0000 7fec25000640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000069d0 0x7fec00006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec26400640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 0x7fec00007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec25a00640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec00005bb0 0x7fec00005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec25000640 1 -- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 msgr2=0x7fec00007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec25000640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 0x7fec00007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec25000640 1 -- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec00005bb0 msgr2=0x7fec00005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec25000640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec00005bb0 0x7fec00005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec25000640 1 -- 172.21.15.141:0/2187593368 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec00005730 con 0x7fec000069d0 2024-01-16T15:15:22.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec26400640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec200cf9a0 0x7fec00007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:22.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.665+0000 7fec25a00640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec00005bb0 0x7fec00005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.666+0000 7fec25000640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000069d0 0x7fec00006e50 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fec08009f90 tx=0x7fec00007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609394bf6fc86784 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.666+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0802cbd0 con 0x7fec000069d0 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.666+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0800f040 con 0x7fec000069d0 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.666+0000 7fec1fe00640 1 -- 172.21.15.141:0/2187593368 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08034440 con 0x7fec000069d0 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.666+0000 7fec28400640 1 -- 172.21.15.141:0/2187593368 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000069d0 msgr2=0x7fec00006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.666+0000 7fec28400640 1 --2- 172.21.15.141:0/2187593368 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000069d0 0x7fec00006e50 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fec08009f90 tx=0x7fec00007a00 comp rx=0 tx=0).stop 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.667+0000 7fec26400640 1 -- 172.21.15.141:0/2187593368 reap_dead start 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.667+0000 7fec28400640 1 -- 172.21.15.141:0/2187593368 shutdown_connections 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.667+0000 7fec28400640 1 -- 172.21.15.141:0/2187593368 >> 172.21.15.141:0/2187593368 conn(0x7fec2006a800 msgr2=0x7fec2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.667+0000 7fec28400640 1 -- 172.21.15.141:0/2187593368 shutdown_connections 2024-01-16T15:15:22.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.667+0000 7fec28400640 1 -- 172.21.15.141:0/2187593368 wait complete. 2024-01-16T15:15:22.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.669+0000 7fec28400640 1 Processor -- start 2024-01-16T15:15:22.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.669+0000 7fec28400640 1 -- start start 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec28400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000069d0 0x7fec20159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec28400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec200af3b0 0x7fec200c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec28400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2015b710 0x7fec2015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec28400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec20132960 con 0x7fec200af3b0 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec28400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015c100 con 0x7fec2015b710 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec28400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec20131a90 con 0x7fec000069d0 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec200af3b0 0x7fec200c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec200af3b0 0x7fec200c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:44200/0 (socket says 172.21.15.141:44200) 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 -- 172.21.15.141:0/842977365 learned_addr learned my addr 172.21.15.141:0/842977365 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25a00640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000069d0 0x7fec20159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 -- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000069d0 msgr2=0x7fec20159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000069d0 0x7fec20159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec26400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2015b710 0x7fec2015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 -- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2015b710 msgr2=0x7fec2015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2015b710 0x7fec2015bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec25000640 1 -- 172.21.15.141:0/842977365 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec20131ce0 con 0x7fec200af3b0 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.670+0000 7fec26400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2015b710 0x7fec2015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.671+0000 7fec25000640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec200af3b0 0x7fec200c43e0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fec08037040 tx=0x7fec08009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.671+0000 7fec1d600640 1 -- 172.21.15.141:0/842977365 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08034d60 con 0x7fec200af3b0 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.671+0000 7fec1d600640 1 -- 172.21.15.141:0/842977365 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0803e400 con 0x7fec200af3b0 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.671+0000 7fec1d600640 1 -- 172.21.15.141:0/842977365 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0803e7b0 con 0x7fec200af3b0 2024-01-16T15:15:22.658 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.671+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec20131fc0 con 0x7fec200af3b0 2024-01-16T15:15:22.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.671+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec20132580 con 0x7fec200af3b0 2024-01-16T15:15:22.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.672+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec200675d0 con 0x7fec200af3b0 2024-01-16T15:15:22.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.673+0000 7fec1d600640 1 -- 172.21.15.141:0/842977365 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fec0803e910 con 0x7fec200af3b0 2024-01-16T15:15:22.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.674+0000 7fec1d600640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febf806c4f0 0x7febf806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.674+0000 7fec1d600640 1 -- 172.21.15.141:0/842977365 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fec080aafa0 con 0x7fec200af3b0 2024-01-16T15:15:22.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.675+0000 7fec25a00640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febf806c4f0 0x7febf806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.676+0000 7fec25a00640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febf806c4f0 0x7febf806e9b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fec10001630 tx=0x7fec10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.679+0000 7fec1d600640 1 -- 172.21.15.141:0/842977365 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec08077a30 con 0x7fec200af3b0 2024-01-16T15:15:22.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.885+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec2006adb0 con 0x7fec200af3b0 2024-01-16T15:15:22.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.886+0000 7fec1d600640 1 -- 172.21.15.141:0/842977365 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+13046 (secure 0 0 0) 0x7fec0804c090 con 0x7fec200af3b0 2024-01-16T15:15:22.871 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:22.872 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":107,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:22.133123+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6836/1516772217":"2024-01-17T15:15:19.121421+0000","172.21.15.188:6834/1516772217":"2024-01-17T15:15:19.121421+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febf806c4f0 msgr2=0x7febf806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febf806c4f0 0x7febf806e9b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fec10001630 tx=0x7fec10009040 comp rx=0 tx=0).stop 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec200af3b0 msgr2=0x7fec200c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec200af3b0 0x7fec200c43e0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fec08037040 tx=0x7fec08009750 comp rx=0 tx=0).stop 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 shutdown_connections 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febf806c4f0 0x7febf806e9b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2015b710 0x7fec2015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec200af3b0 0x7fec200c43e0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 --2- 172.21.15.141:0/842977365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000069d0 0x7fec20159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 >> 172.21.15.141:0/842977365 conn(0x7fec2006a800 msgr2=0x7fec2006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 shutdown_connections 2024-01-16T15:15:22.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:22.889+0000 7fec28400640 1 -- 172.21.15.141:0/842977365 wait complete. 2024-01-16T15:15:22.893 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:22.893 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:23.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.101+0000 7fd40e400640 1 Processor -- start 2024-01-16T15:15:23.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.101+0000 7fd40e400640 1 -- start start 2024-01-16T15:15:23.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd40e400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd40813e360 0x7fd4080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd40e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 0x7fd4080cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd40e400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40815c5f0 con 0x7fd40813df80 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd40e400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40806c470 con 0x7fd40813e360 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd40e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40806c750 con 0x7fd40815f9c0 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd3ffe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd40813e360 0x7fd4080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd3ffe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd40813e360 0x7fd4080cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32772/0 (socket says 172.21.15.141:32772) 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd3ffe00640 1 -- 172.21.15.141:0/1320926279 learned_addr learned my addr 172.21.15.141:0/1320926279 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.103+0000 7fd407e00640 1 --2- 172.21.15.141:0/1320926279 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 0x7fd4080cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.104+0000 7fd3ffe00640 1 -- 172.21.15.141:0/1320926279 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 msgr2=0x7fd4080cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.104+0000 7fd3ffe00640 1 --2- 172.21.15.141:0/1320926279 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 0x7fd4080cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.104+0000 7fd3ffe00640 1 -- 172.21.15.141:0/1320926279 >> v1:172.21.15.188:6789/0 conn(0x7fd40813df80 legacy=0x7fd40815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.104+0000 7fd3ffe00640 1 -- 172.21.15.141:0/1320926279 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40806ca30 con 0x7fd40813e360 2024-01-16T15:15:23.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.105+0000 7fd3ffe00640 1 --2- 172.21.15.141:0/1320926279 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813e360 0x7fd4080cf290 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fd40806ded0 tx=0x7fd3f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e03b65a73645c48 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.105+0000 7fd406a00640 1 -- 172.21.15.141:0/1320926279 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0031830 con 0x7fd40813e360 2024-01-16T15:15:23.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.105+0000 7fd406a00640 1 -- 172.21.15.141:0/1320926279 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f0031990 con 0x7fd40813e360 2024-01-16T15:15:23.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.105+0000 7fd406a00640 1 -- 172.21.15.141:0/1320926279 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0031d20 con 0x7fd40813e360 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 -- 172.21.15.141:0/1320926279 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813e360 msgr2=0x7fd4080cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 --2- 172.21.15.141:0/1320926279 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813e360 0x7fd4080cf290 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fd40806ded0 tx=0x7fd3f002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 -- 172.21.15.141:0/1320926279 shutdown_connections 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 --2- 172.21.15.141:0/1320926279 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813e360 0x7fd4080cf290 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 --2- 172.21.15.141:0/1320926279 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 0x7fd4080cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 -- 172.21.15.141:0/1320926279 >> 172.21.15.141:0/1320926279 conn(0x7fd40806a800 msgr2=0x7fd40806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 -- 172.21.15.141:0/1320926279 shutdown_connections 2024-01-16T15:15:23.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.106+0000 7fd40e400640 1 -- 172.21.15.141:0/1320926279 wait complete. 2024-01-16T15:15:23.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.108+0000 7fd40e400640 1 Processor -- start 2024-01-16T15:15:23.093 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.109+0000 7fd40e400640 1 -- start start 2024-01-16T15:15:23.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd40e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813df80 0x7fd40814c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd40e400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd40813e360 0x7fd4081f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd40e400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 0x7fd40814b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd40e400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40814d4a0 con 0x7fd40813df80 2024-01-16T15:15:23.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd40e400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081f4440 con 0x7fd40813e360 2024-01-16T15:15:23.095 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd40e400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40814b940 con 0x7fd40815f9c0 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813df80 0x7fd40814c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813df80 0x7fd40814c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32774/0 (socket says 172.21.15.141:32774) 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 -- 172.21.15.141:0/2423372399 learned_addr learned my addr 172.21.15.141:0/2423372399 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 -- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 msgr2=0x7fd40814b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 0x7fd40814b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 -- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd40813e360 msgr2=0x7fd4081f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd40813e360 0x7fd4081f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.110+0000 7fd407400640 1 -- 172.21.15.141:0/2423372399 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f00094d0 con 0x7fd40813df80 2024-01-16T15:15:23.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.111+0000 7fd407400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813df80 0x7fd40814c200 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800b3f0 tx=0x7fd3f800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.111+0000 7fd3ff400640 1 -- 172.21.15.141:0/2423372399 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8011830 con 0x7fd40813df80 2024-01-16T15:15:23.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.111+0000 7fd3ff400640 1 -- 172.21.15.141:0/2423372399 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f8011990 con 0x7fd40813df80 2024-01-16T15:15:23.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.111+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40814c890 con 0x7fd40813df80 2024-01-16T15:15:23.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.111+0000 7fd3ff400640 1 -- 172.21.15.141:0/2423372399 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8011d40 con 0x7fd40813df80 2024-01-16T15:15:23.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.111+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40814ce50 con 0x7fd40813df80 2024-01-16T15:15:23.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.112+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4080675d0 con 0x7fd40813df80 2024-01-16T15:15:23.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.113+0000 7fd3ff400640 1 -- 172.21.15.141:0/2423372399 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fd3f801a400 con 0x7fd40813df80 2024-01-16T15:15:23.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.114+0000 7fd3ff400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006c4c0 0x7fd3e006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.114+0000 7fd3ff400640 1 -- 172.21.15.141:0/2423372399 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd3f80889c0 con 0x7fd40813df80 2024-01-16T15:15:23.099 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.115+0000 7fd3ffe00640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006c4c0 0x7fd3e006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.116+0000 7fd3ffe00640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006c4c0 0x7fd3e006e980 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0036000 tx=0x7fd3f0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.119+0000 7fd3ff400640 1 -- 172.21.15.141:0/2423372399 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3f80bee90 con 0x7fd40813df80 2024-01-16T15:15:23.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.309+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd40806df00 con 0x7fd40813df80 2024-01-16T15:15:23.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.310+0000 7fd3ff400640 1 -- 172.21.15.141:0/2423372399 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd3f8015020 con 0x7fd40813df80 2024-01-16T15:15:23.295 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006c4c0 msgr2=0x7fd3e006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006c4c0 0x7fd3e006e980 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0036000 tx=0x7fd3f0030000 comp rx=0 tx=0).stop 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813df80 msgr2=0x7fd40814c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813df80 0x7fd40814c200 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800b3f0 tx=0x7fd3f800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 shutdown_connections 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd3e006c4c0 0x7fd3e006e980 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd40815f9c0 0x7fd40814b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd40813e360 0x7fd4081f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 --2- 172.21.15.141:0/2423372399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd40813df80 0x7fd40814c200 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.312+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 >> 172.21.15.141:0/2423372399 conn(0x7fd40806a800 msgr2=0x7fd40815de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:23.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.313+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 shutdown_connections 2024-01-16T15:15:23.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.313+0000 7fd40e400640 1 -- 172.21.15.141:0/2423372399 wait complete. 2024-01-16T15:15:23.315 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:15:23.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.540+0000 7ff379200640 1 Processor -- start 2024-01-16T15:15:23.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.541+0000 7ff379200640 1 -- start start 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff379200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff3740cf9a0 0x7ff37415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff379200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff37415e1e0 0x7ff3740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff379200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3740b1ee0 con 0x7ff374143ae0 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff379200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740b2780 con 0x7ff37415e1e0 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff379200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff37415e990 con 0x7ff3740cf9a0 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff371e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff374143ae0 0x7ff3740cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60982/0 (socket says 172.21.15.141:60982) 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff371e00640 1 -- 172.21.15.141:0/2222534469 learned_addr learned my addr 172.21.15.141:0/2222534469 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.542+0000 7ff372800640 1 --2- 172.21.15.141:0/2222534469 >> v2:172.21.15.141:3300/0 conn(0x7ff37415e1e0 0x7ff3740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.543+0000 7ff369200640 1 --2- 172.21.15.141:0/2222534469 >> v2:172.21.15.188:3300/0 conn(0x7ff3740cf9a0 0x7ff37415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.543+0000 7ff372800640 1 -- 172.21.15.141:0/2222534469 >> v2:172.21.15.188:3300/0 conn(0x7ff3740cf9a0 msgr2=0x7ff37415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.543+0000 7ff372800640 1 --2- 172.21.15.141:0/2222534469 >> v2:172.21.15.188:3300/0 conn(0x7ff3740cf9a0 0x7ff37415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.543+0000 7ff372800640 1 -- 172.21.15.141:0/2222534469 >> v1:172.21.15.141:6789/0 conn(0x7ff374143ae0 legacy=0x7ff3740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.543+0000 7ff371400640 1 -- 172.21.15.141:0/2222534469 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094705352 0 0) 0x7ff3740b1ee0 con 0x7ff374143ae0 2024-01-16T15:15:23.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.543+0000 7ff372800640 1 -- 172.21.15.141:0/2222534469 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740b2130 con 0x7ff37415e1e0 2024-01-16T15:15:23.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.543+0000 7ff369200640 1 --2- 172.21.15.141:0/2222534469 >> v2:172.21.15.188:3300/0 conn(0x7ff3740cf9a0 0x7ff37415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:23.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.544+0000 7ff372800640 1 --2- 172.21.15.141:0/2222534469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff37415e1e0 0x7ff3740702c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7ff3740cb830 tx=0x7ff36c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1573e8510362f212 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.544+0000 7ff371400640 1 -- 172.21.15.141:0/2222534469 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c02cc60 con 0x7ff37415e1e0 2024-01-16T15:15:23.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.544+0000 7ff371400640 1 -- 172.21.15.141:0/2222534469 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c02cdc0 con 0x7ff37415e1e0 2024-01-16T15:15:23.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.544+0000 7ff371400640 1 -- 172.21.15.141:0/2222534469 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c034660 con 0x7ff37415e1e0 2024-01-16T15:15:23.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 -- 172.21.15.141:0/2222534469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff37415e1e0 msgr2=0x7ff3740702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 --2- 172.21.15.141:0/2222534469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff37415e1e0 0x7ff3740702c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7ff3740cb830 tx=0x7ff36c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 -- 172.21.15.141:0/2222534469 shutdown_connections 2024-01-16T15:15:23.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 --2- 172.21.15.141:0/2222534469 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff37415e1e0 0x7ff3740702c0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 --2- 172.21.15.141:0/2222534469 >> v2:172.21.15.188:3300/0 conn(0x7ff3740cf9a0 0x7ff37415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 -- 172.21.15.141:0/2222534469 >> 172.21.15.141:0/2222534469 conn(0x7ff37406a800 msgr2=0x7ff37406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:23.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 -- 172.21.15.141:0/2222534469 shutdown_connections 2024-01-16T15:15:23.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.545+0000 7ff379200640 1 -- 172.21.15.141:0/2222534469 wait complete. 2024-01-16T15:15:23.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.547+0000 7ff379200640 1 Processor -- start 2024-01-16T15:15:23.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.547+0000 7ff379200640 1 -- start start 2024-01-16T15:15:23.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff379200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3740cf9a0 0x7ff37415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff379200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff374143ae0 0x7ff374159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff379200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff37415e1e0 0x7ff374159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff379200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff374139f40 con 0x7ff374143ae0 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff379200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37413b280 con 0x7ff37415e1e0 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff379200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37413a1e0 con 0x7ff3740cf9a0 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff374143ae0 0x7ff374159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff374143ae0 0x7ff374159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32802/0 (socket says 172.21.15.141:32802) 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 -- 172.21.15.141:0/2730517144 learned_addr learned my addr 172.21.15.141:0/2730517144 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff372800640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff37415e1e0 0x7ff374159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 -- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3740cf9a0 msgr2=0x7ff37415a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff371e00640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3740cf9a0 0x7ff37415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3740cf9a0 0x7ff37415a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 -- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff37415e1e0 msgr2=0x7ff374159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff37415e1e0 0x7ff374159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff369200640 1 -- 172.21.15.141:0/2730517144 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c009a70 con 0x7ff374143ae0 2024-01-16T15:15:23.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.549+0000 7ff371e00640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3740cf9a0 0x7ff37415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:23.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.550+0000 7ff369200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff374143ae0 0x7ff374159700 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00c7f0 tx=0x7ff35c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.550+0000 7ff36aa00640 1 -- 172.21.15.141:0/2730517144 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c007a30 con 0x7ff374143ae0 2024-01-16T15:15:23.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.550+0000 7ff36aa00640 1 -- 172.21.15.141:0/2730517144 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c007b90 con 0x7ff374143ae0 2024-01-16T15:15:23.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.550+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37413a4c0 con 0x7ff374143ae0 2024-01-16T15:15:23.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.550+0000 7ff36aa00640 1 -- 172.21.15.141:0/2730517144 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c00ce40 con 0x7ff374143ae0 2024-01-16T15:15:23.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.550+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37413aa80 con 0x7ff374143ae0 2024-01-16T15:15:23.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.551+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3740675d0 con 0x7ff374143ae0 2024-01-16T15:15:23.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.552+0000 7ff36aa00640 1 -- 172.21.15.141:0/2730517144 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff35c0113f0 con 0x7ff374143ae0 2024-01-16T15:15:23.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.553+0000 7ff36aa00640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff34c06c4c0 0x7ff34c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.553+0000 7ff36aa00640 1 -- 172.21.15.141:0/2730517144 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff35c088ce0 con 0x7ff374143ae0 2024-01-16T15:15:23.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.554+0000 7ff371e00640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff34c06c4c0 0x7ff34c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.555+0000 7ff371e00640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff34c06c4c0 0x7ff34c06e980 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff364000f60 tx=0x7ff364009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.557+0000 7ff36aa00640 1 -- 172.21.15.141:0/2730517144 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff35c055230 con 0x7ff374143ae0 2024-01-16T15:15:23.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.757+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff37413f2a0 con 0x7ff374143ae0 2024-01-16T15:15:23.742 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.757+0000 7ff36aa00640 1 -- 172.21.15.141:0/2730517144 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v107) v1 ==== 80+0+124 (secure 0 0 0) 0x7ff35c01f080 con 0x7ff374143ae0 2024-01-16T15:15:23.743 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.188:6836/1516772217 2024-01-17T15:15:19.121421+0000 2024-01-16T15:15:23.743 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.188:6834/1516772217 2024-01-17T15:15:19.121421+0000 2024-01-16T15:15:23.743 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:15:23.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff34c06c4c0 msgr2=0x7ff34c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff34c06c4c0 0x7ff34c06e980 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff364000f60 tx=0x7ff364009040 comp rx=0 tx=0).stop 2024-01-16T15:15:23.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff374143ae0 msgr2=0x7ff374159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff374143ae0 0x7ff374159700 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00c7f0 tx=0x7ff35c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 shutdown_connections 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff34c06c4c0 0x7ff34c06e980 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff37415e1e0 0x7ff374159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff374143ae0 0x7ff374159700 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 --2- 172.21.15.141:0/2730517144 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff3740cf9a0 0x7ff37415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 >> 172.21.15.141:0/2730517144 conn(0x7ff37406a800 msgr2=0x7ff37406cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 shutdown_connections 2024-01-16T15:15:23.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:23.760+0000 7ff379200640 1 -- 172.21.15.141:0/2730517144 wait complete. 2024-01-16T15:15:23.763 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T15:15:23.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:23.764 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:23.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[72831]: fuse finished with error 0 and tester_r 0 2024-01-16T15:15:23.857 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:15:23.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:23.857 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:15:24.055 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:15:24.055 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:24.111 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:24.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.340+0000 7f3f6d800640 1 Processor -- start 2024-01-16T15:15:24.325 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.340+0000 7f3f6d800640 1 -- start start 2024-01-16T15:15:24.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f6d800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3f680cf9a0 0x7f3f6815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f6d800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f680b1ee0 con 0x7f3f6815e1e0 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f6d800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f680b2780 con 0x7f3f68143ae0 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f6d800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6815e990 con 0x7f3f680cf9a0 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f5fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3f680cf9a0 0x7f3f6815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f67400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3f6815e1e0 0x7f3f680702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60990/0 (socket says 172.21.15.141:60990) 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f67400640 1 -- 172.21.15.141:0/3882366897 learned_addr learned my addr 172.21.15.141:0/3882366897 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f5fe00640 1 -- 172.21.15.141:0/3882366897 >> v1:172.21.15.188:6789/0 conn(0x7f3f68143ae0 legacy=0x7f3f680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f5fe00640 1 -- 172.21.15.141:0/3882366897 >> v1:172.21.15.141:6789/0 conn(0x7f3f6815e1e0 legacy=0x7f3f680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.343+0000 7f3f5fe00640 1 -- 172.21.15.141:0/3882366897 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f680b2130 con 0x7f3f680cf9a0 2024-01-16T15:15:24.328 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.342+0000 7f3f66000640 1 -- 172.21.15.141:0/3882366897 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422376642 0 0) 0x7f3f680b1ee0 con 0x7f3f6815e1e0 2024-01-16T15:15:24.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f5fe00640 1 --2- 172.21.15.141:0/3882366897 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f680cf9a0 0x7f3f6815dca0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f3f500097a0 tx=0x7f3f5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8994794ebf03a8b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f66000640 1 -- 172.21.15.141:0/3882366897 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50031830 con 0x7f3f680cf9a0 2024-01-16T15:15:24.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f66000640 1 -- 172.21.15.141:0/3882366897 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f50031990 con 0x7f3f680cf9a0 2024-01-16T15:15:24.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f66000640 1 -- 172.21.15.141:0/3882366897 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50031d20 con 0x7f3f680cf9a0 2024-01-16T15:15:24.330 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f6d800640 1 -- 172.21.15.141:0/3882366897 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f680cf9a0 msgr2=0x7f3f6815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3882366897 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f680cf9a0 0x7f3f6815dca0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f3f500097a0 tx=0x7f3f5002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f6d800640 1 -- 172.21.15.141:0/3882366897 shutdown_connections 2024-01-16T15:15:24.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.344+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3882366897 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f680cf9a0 0x7f3f6815dca0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.345+0000 7f3f6d800640 1 -- 172.21.15.141:0/3882366897 >> 172.21.15.141:0/3882366897 conn(0x7f3f6806a800 msgr2=0x7f3f6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:24.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.345+0000 7f3f6d800640 1 -- 172.21.15.141:0/3882366897 shutdown_connections 2024-01-16T15:15:24.331 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.345+0000 7f3f6d800640 1 -- 172.21.15.141:0/3882366897 wait complete. 2024-01-16T15:15:24.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.346+0000 7f3f6d800640 1 Processor -- start 2024-01-16T15:15:24.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.346+0000 7f3f6d800640 1 -- start start 2024-01-16T15:15:24.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.347+0000 7f3f6d800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3f680cf9a0 0x7f3f68141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.347+0000 7f3f6d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f68143ae0 0x7f3f680c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.347+0000 7f3f6d800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f6815e1e0 0x7f3f6813a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.347+0000 7f3f6d800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f681419d0 con 0x7f3f68143ae0 2024-01-16T15:15:24.333 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.347+0000 7f3f6d800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6813ace0 con 0x7f3f6815e1e0 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.347+0000 7f3f6d800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680c4550 con 0x7f3f680cf9a0 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f68143ae0 0x7f3f680c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f68143ae0 0x7f3f680c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32832/0 (socket says 172.21.15.141:32832) 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 -- 172.21.15.141:0/3860701275 learned_addr learned my addr 172.21.15.141:0/3860701275 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f67400640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f6815e1e0 0x7f3f6813a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 -- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3f680cf9a0 msgr2=0x7f3f68141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3f680cf9a0 0x7f3f68141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 -- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f6815e1e0 msgr2=0x7f3f6813a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f6815e1e0 0x7f3f6813a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 -- 172.21.15.141:0/3860701275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50009400 con 0x7f3f68143ae0 2024-01-16T15:15:24.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.348+0000 7f3f5fe00640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f68143ae0 0x7f3f680c3e50 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f3f500098c0 tx=0x7f3f500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.349+0000 7f3f5ea00640 1 -- 172.21.15.141:0/3860701275 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5003ba70 con 0x7f3f68143ae0 2024-01-16T15:15:24.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.349+0000 7f3f5ea00640 1 -- 172.21.15.141:0/3860701275 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5003bbf0 con 0x7f3f68143ae0 2024-01-16T15:15:24.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.349+0000 7f3f5ea00640 1 -- 172.21.15.141:0/3860701275 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f500453f0 con 0x7f3f68143ae0 2024-01-16T15:15:24.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.349+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f680c4830 con 0x7f3f68143ae0 2024-01-16T15:15:24.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.349+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f680c4df0 con 0x7f3f68143ae0 2024-01-16T15:15:24.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.350+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f680675d0 con 0x7f3f68143ae0 2024-01-16T15:15:24.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.351+0000 7f3f5ea00640 1 -- 172.21.15.141:0/3860701275 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f3f5004e450 con 0x7f3f68143ae0 2024-01-16T15:15:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.352+0000 7f3f5ea00640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3f4006c4f0 0x7f3f4006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.352+0000 7f3f5ea00640 1 -- 172.21.15.141:0/3860701275 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3f500aadb0 con 0x7f3f68143ae0 2024-01-16T15:15:24.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.353+0000 7f3f66a00640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3f4006c4f0 0x7f3f4006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.354+0000 7f3f66a00640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3f4006c4f0 0x7f3f4006e9b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3f58003cc0 tx=0x7f3f580079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.356+0000 7f3f5ea00640 1 -- 172.21.15.141:0/3860701275 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f50077420 con 0x7f3f68143ae0 2024-01-16T15:15:24.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.547+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f68067c00 con 0x7f3f68143ae0 2024-01-16T15:15:24.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.549+0000 7f3f5ea00640 1 -- 172.21.15.141:0/3860701275 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+13046 (secure 0 0 0) 0x7f3f50035020 con 0x7f3f68143ae0 2024-01-16T15:15:24.534 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:24.534 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":107,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:22.133123+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6836/1516772217":"2024-01-17T15:15:19.121421+0000","172.21.15.188:6834/1516772217":"2024-01-17T15:15:19.121421+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.551+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3f4006c4f0 msgr2=0x7f3f4006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.551+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3f4006c4f0 0x7f3f4006e9b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3f58003cc0 tx=0x7f3f580079b0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.551+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f68143ae0 msgr2=0x7f3f680c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.551+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f68143ae0 0x7f3f680c3e50 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f3f500098c0 tx=0x7f3f500046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 shutdown_connections 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3f4006c4f0 0x7f3f4006e9b0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f6815e1e0 0x7f3f6813a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f68143ae0 0x7f3f680c3e50 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 --2- 172.21.15.141:0/3860701275 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3f680cf9a0 0x7f3f68141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 >> 172.21.15.141:0/3860701275 conn(0x7f3f6806a800 msgr2=0x7f3f6806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:24.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 shutdown_connections 2024-01-16T15:15:24.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.552+0000 7f3f6d800640 1 -- 172.21.15.141:0/3860701275 wait complete. 2024-01-16T15:15:24.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:24.557 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:24.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.782+0000 7f262f000640 1 Processor -- start 2024-01-16T15:15:24.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.782+0000 7f262f000640 1 -- start start 2024-01-16T15:15:24.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.784+0000 7f262f000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f262815d380 0x7f2628070410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.784+0000 7f262f000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f26280b38c0 0x7f2628070950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.784+0000 7f262f000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f26280b1f00 0x7f2628160f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.784+0000 7f262f000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26280b2e20 con 0x7f26280b38c0 2024-01-16T15:15:24.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.784+0000 7f262f000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262806c470 con 0x7f262815d380 2024-01-16T15:15:24.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.784+0000 7f262f000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26280b4240 con 0x7f26280b1f00 2024-01-16T15:15:24.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.784+0000 7f261f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f26280b38c0 0x7f2628070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f261f200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f26280b38c0 0x7f2628070950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32848/0 (socket says 172.21.15.141:32848) 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f261f200640 1 -- 172.21.15.141:0/3028821064 learned_addr learned my addr 172.21.15.141:0/3028821064 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f2627e00640 1 --2- 172.21.15.141:0/3028821064 >> v2:172.21.15.188:3300/0 conn(0x7f262815d380 0x7f2628070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f262d000640 1 --2- 172.21.15.141:0/3028821064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f26280b1f00 0x7f2628160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f261f200640 1 -- 172.21.15.141:0/3028821064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f26280b1f00 msgr2=0x7f2628160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f261f200640 1 --2- 172.21.15.141:0/3028821064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f26280b1f00 0x7f2628160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f261f200640 1 -- 172.21.15.141:0/3028821064 >> v2:172.21.15.188:3300/0 conn(0x7f262815d380 msgr2=0x7f2628070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f261f200640 1 --2- 172.21.15.141:0/3028821064 >> v2:172.21.15.188:3300/0 conn(0x7f262815d380 0x7f2628070410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.785+0000 7f261f200640 1 -- 172.21.15.141:0/3028821064 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280b3070 con 0x7f26280b38c0 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.786+0000 7f261f200640 1 --2- 172.21.15.141:0/3028821064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f2628070950 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f262806e740 tx=0x7f261002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa9509c4b3413c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f2627400640 1 -- 172.21.15.141:0/3028821064 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610031870 con 0x7f26280b38c0 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f2627400640 1 -- 172.21.15.141:0/3028821064 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26100319d0 con 0x7f26280b38c0 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f2627400640 1 -- 172.21.15.141:0/3028821064 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610031d60 con 0x7f26280b38c0 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 -- 172.21.15.141:0/3028821064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 msgr2=0x7f2628070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 --2- 172.21.15.141:0/3028821064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f2628070950 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f262806e740 tx=0x7f261002b910 comp rx=0 tx=0).stop 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 -- 172.21.15.141:0/3028821064 shutdown_connections 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 --2- 172.21.15.141:0/3028821064 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f26280b1f00 0x7f2628160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 --2- 172.21.15.141:0/3028821064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f2628070950 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 --2- 172.21.15.141:0/3028821064 >> v2:172.21.15.188:3300/0 conn(0x7f262815d380 0x7f2628070410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 -- 172.21.15.141:0/3028821064 >> 172.21.15.141:0/3028821064 conn(0x7f26280b0c00 msgr2=0x7f26280b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:24.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 -- 172.21.15.141:0/3028821064 shutdown_connections 2024-01-16T15:15:24.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.787+0000 7f262f000640 1 -- 172.21.15.141:0/3028821064 wait complete. 2024-01-16T15:15:24.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.789+0000 7f262f000640 1 Processor -- start 2024-01-16T15:15:24.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.789+0000 7f262f000640 1 -- start start 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.790+0000 7f262f000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26280b1f00 0x7f262815c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.790+0000 7f262f000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f26281f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.790+0000 7f262f000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f262815d380 0x7f26280c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.790+0000 7f262f000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2628149b90 con 0x7f26280b38c0 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.790+0000 7f262f000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262814a000 con 0x7f26280b1f00 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.790+0000 7f262f000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26280c8840 con 0x7f262815d380 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f26281f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f26281f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32852/0 (socket says 172.21.15.141:32852) 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 -- 172.21.15.141:0/3437570550 learned_addr learned my addr 172.21.15.141:0/3437570550 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f2627e00640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26280b1f00 0x7f262815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f262d000640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f262815d380 0x7f26280c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 -- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f262815d380 msgr2=0x7f26280c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f262815d380 0x7f26280c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 -- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26280b1f00 msgr2=0x7f262815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26280b1f00 0x7f262815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 -- 172.21.15.141:0/3437570550 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26100094d0 con 0x7f26280b38c0 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.791+0000 7f261f200640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f26281f2c90 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f2610000c00 tx=0x7f26100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.792+0000 7f2624c00640 1 -- 172.21.15.141:0/3437570550 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261003abe0 con 0x7f26280b38c0 2024-01-16T15:15:24.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.792+0000 7f2624c00640 1 -- 172.21.15.141:0/3437570550 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261003ad60 con 0x7f26280b38c0 2024-01-16T15:15:24.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.792+0000 7f2624c00640 1 -- 172.21.15.141:0/3437570550 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610044630 con 0x7f26280b38c0 2024-01-16T15:15:24.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.792+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26280c8b20 con 0x7f26280b38c0 2024-01-16T15:15:24.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.792+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26280c90e0 con 0x7f26280b38c0 2024-01-16T15:15:24.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.793+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628067640 con 0x7f26280b38c0 2024-01-16T15:15:24.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.794+0000 7f2624c00640 1 -- 172.21.15.141:0/3437570550 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2610044790 con 0x7f26280b38c0 2024-01-16T15:15:24.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.795+0000 7f2624c00640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f260006c4c0 0x7f260006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.795+0000 7f2624c00640 1 -- 172.21.15.141:0/3437570550 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f261002d040 con 0x7f26280b38c0 2024-01-16T15:15:24.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.796+0000 7f2627e00640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f260006c4c0 0x7f260006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.797+0000 7f2627e00640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f260006c4c0 0x7f260006e980 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2618009a30 tx=0x7f2618008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.799+0000 7f2624c00640 1 -- 172.21.15.141:0/3437570550 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26100776b0 con 0x7f26280b38c0 2024-01-16T15:15:24.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.989+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26280726d0 con 0x7f26280b38c0 2024-01-16T15:15:24.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.990+0000 7f2624c00640 1 -- 172.21.15.141:0/3437570550 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2610035030 con 0x7f26280b38c0 2024-01-16T15:15:24.975 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f260006c4c0 msgr2=0x7f260006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f260006c4c0 0x7f260006e980 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2618009a30 tx=0x7f2618008040 comp rx=0 tx=0).stop 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 msgr2=0x7f26281f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f26281f2c90 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f2610000c00 tx=0x7f26100046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 shutdown_connections 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f260006c4c0 0x7f260006e980 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f262815d380 0x7f26280c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26280b38c0 0x7f26281f2c90 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 --2- 172.21.15.141:0/3437570550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26280b1f00 0x7f262815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 >> 172.21.15.141:0/3437570550 conn(0x7f26280b0c00 msgr2=0x7f26280ac7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:24.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 shutdown_connections 2024-01-16T15:15:24.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:24.993+0000 7f262f000640 1 -- 172.21.15.141:0/3437570550 wait complete. 2024-01-16T15:15:24.996 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:15:25.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.209+0000 7f23ad200640 1 Processor -- start 2024-01-16T15:15:25.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.209+0000 7f23ad200640 1 -- start start 2024-01-16T15:15:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.211+0000 7f23ad200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f23a813df80 0x7f23a815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.211+0000 7f23ad200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.211+0000 7f23ad200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a806c470 con 0x7f23a813e360 2024-01-16T15:15:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.211+0000 7f23ad200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a806c750 con 0x7f23a813df80 2024-01-16T15:15:25.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.211+0000 7f23ad200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a806ca30 con 0x7f23a80cf9a0 2024-01-16T15:15:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a6800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a5e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f23a813df80 0x7f23a815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a6800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a80cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:45406/0 (socket says 172.21.15.141:45406) 2024-01-16T15:15:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a6800640 1 -- 172.21.15.141:0/4178883851 learned_addr learned my addr 172.21.15.141:0/4178883851 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a6800640 1 -- 172.21.15.141:0/4178883851 >> v2:172.21.15.188:3300/0 conn(0x7f23a813df80 msgr2=0x7f23a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a6800640 1 --2- 172.21.15.141:0/4178883851 >> v2:172.21.15.188:3300/0 conn(0x7f23a813df80 0x7f23a815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a6800640 1 -- 172.21.15.141:0/4178883851 >> v1:172.21.15.188:6789/0 conn(0x7f23a813e360 legacy=0x7f23a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.212+0000 7f23a6800640 1 -- 172.21.15.141:0/4178883851 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a806cd10 con 0x7f23a80cf9a0 2024-01-16T15:15:25.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.214+0000 7f23a6800640 1 --2- 172.21.15.141:0/4178883851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a80cd840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009e50 tx=0x7f23a002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f24d6a5da82c24 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.214+0000 7f23a5400640 1 -- 172.21.15.141:0/4178883851 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a002cb00 con 0x7f23a80cf9a0 2024-01-16T15:15:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.214+0000 7f23a5400640 1 -- 172.21.15.141:0/4178883851 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a002cc60 con 0x7f23a80cf9a0 2024-01-16T15:15:25.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.214+0000 7f23a5400640 1 -- 172.21.15.141:0/4178883851 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00343f0 con 0x7f23a80cf9a0 2024-01-16T15:15:25.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 -- 172.21.15.141:0/4178883851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 msgr2=0x7f23a80cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 --2- 172.21.15.141:0/4178883851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a80cd840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009e50 tx=0x7f23a002bf60 comp rx=0 tx=0).stop 2024-01-16T15:15:25.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 -- 172.21.15.141:0/4178883851 shutdown_connections 2024-01-16T15:15:25.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 --2- 172.21.15.141:0/4178883851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a80cd840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 --2- 172.21.15.141:0/4178883851 >> v2:172.21.15.188:3300/0 conn(0x7f23a813df80 0x7f23a815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 -- 172.21.15.141:0/4178883851 >> 172.21.15.141:0/4178883851 conn(0x7f23a806a800 msgr2=0x7f23a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 -- 172.21.15.141:0/4178883851 shutdown_connections 2024-01-16T15:15:25.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.215+0000 7f23ad200640 1 -- 172.21.15.141:0/4178883851 wait complete. 2024-01-16T15:15:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.217+0000 7f23ad200640 1 Processor -- start 2024-01-16T15:15:25.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.217+0000 7f23ad200640 1 -- start start 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23ad200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a814ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23ad200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a813df80 0x7f23a80711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23ad200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a813e360 0x7f23a81f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23ad200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80716f0 con 0x7f23a813e360 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23ad200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a81f6140 con 0x7f23a813df80 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23ad200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a814c0f0 con 0x7f23a80cf9a0 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23a6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a813e360 0x7f23a81f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23a6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a813e360 0x7f23a81f5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32858/0 (socket says 172.21.15.141:32858) 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.218+0000 7f23a6800640 1 -- 172.21.15.141:0/513938347 learned_addr learned my addr 172.21.15.141:0/513938347 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:25.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f239fe00640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a813df80 0x7f23a80711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f23a5e00640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a814ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f23a6800640 1 -- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 msgr2=0x7f23a814ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f23a6800640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a814ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f23a6800640 1 -- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a813df80 msgr2=0x7f23a80711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f23a6800640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a813df80 0x7f23a80711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f23a6800640 1 -- 172.21.15.141:0/513938347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0009b30 con 0x7f23a813e360 2024-01-16T15:15:25.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f23a6800640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a813e360 0x7f23a81f5c00 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f23a0036040 tx=0x7f23a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:25.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.219+0000 7f239e000640 1 -- 172.21.15.141:0/513938347 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0009510 con 0x7f23a813e360 2024-01-16T15:15:25.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.220+0000 7f239e000640 1 -- 172.21.15.141:0/513938347 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a003e450 con 0x7f23a813e360 2024-01-16T15:15:25.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.220+0000 7f239e000640 1 -- 172.21.15.141:0/513938347 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a003e820 con 0x7f23a813e360 2024-01-16T15:15:25.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.220+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a814c3d0 con 0x7f23a813e360 2024-01-16T15:15:25.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.220+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a814c990 con 0x7f23a813e360 2024-01-16T15:15:25.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.221+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a81f2d70 con 0x7f23a813e360 2024-01-16T15:15:25.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.222+0000 7f239e000640 1 -- 172.21.15.141:0/513938347 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f23a003e980 con 0x7f23a813e360 2024-01-16T15:15:25.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.223+0000 7f239e000640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006c4c0 0x7f238006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.223+0000 7f239e000640 1 -- 172.21.15.141:0/513938347 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f23a00ab820 con 0x7f23a813e360 2024-01-16T15:15:25.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.223+0000 7f23a5e00640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006c4c0 0x7f238006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.225+0000 7f23a5e00640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006c4c0 0x7f238006e980 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2398009ab0 tx=0x7f2398008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:25.212 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.227+0000 7f239e000640 1 -- 172.21.15.141:0/513938347 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23a0077d70 con 0x7f23a813e360 2024-01-16T15:15:25.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.431+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f23a80675d0 con 0x7f23a813e360 2024-01-16T15:15:25.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.432+0000 7f239e000640 1 -- 172.21.15.141:0/513938347 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v107) v1 ==== 80+0+124 (secure 0 0 0) 0x7f23a0043080 con 0x7f23a813e360 2024-01-16T15:15:25.417 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.188:6836/1516772217 2024-01-17T15:15:19.121421+0000 2024-01-16T15:15:25.417 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.188:6834/1516772217 2024-01-17T15:15:19.121421+0000 2024-01-16T15:15:25.417 INFO:teuthology.orchestra.run.smithi141.stderr:listed 2 entries 2024-01-16T15:15:25.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.434+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006c4c0 msgr2=0x7f238006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.434+0000 7f23ad200640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006c4c0 0x7f238006e980 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2398009ab0 tx=0x7f2398008040 comp rx=0 tx=0).stop 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.434+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a813e360 msgr2=0x7f23a81f5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.434+0000 7f23ad200640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a813e360 0x7f23a81f5c00 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f23a0036040 tx=0x7f23a0004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 shutdown_connections 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f238006c4c0 0x7f238006e980 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a813e360 0x7f23a81f5c00 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23a813df80 0x7f23a80711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 --2- 172.21.15.141:0/513938347 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f23a80cf9a0 0x7f23a814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 >> 172.21.15.141:0/513938347 conn(0x7f23a806a800 msgr2=0x7f23a806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 shutdown_connections 2024-01-16T15:15:25.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:25.435+0000 7f23ad200640 1 -- 172.21.15.141:0/513938347 wait complete. 2024-01-16T15:15:25.440 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T15:15:25.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:25.440 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:25.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[46661]: fuse finished with error 0 and tester_r 0 2024-01-16T15:15:25.540 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:15:25.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:25.541 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:15:25.725 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:15:25.725 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:25.782 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:25.782 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:15:25.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:25.782 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:15:25.834 INFO:teuthology.orchestra.run.smithi141.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:15:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:15:25.836 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:15:25.836 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:25.892 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:25.893 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:15:25.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:25.893 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:15:25.947 INFO:teuthology.orchestra.run.smithi188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:15:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:15:25.948 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:26.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.179+0000 7fdba8e00640 1 Processor -- start 2024-01-16T15:15:26.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.179+0000 7fdba8e00640 1 -- start start 2024-01-16T15:15:26.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.181+0000 7fdba8e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdba4143c40 0x7fdba4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.181+0000 7fdba8e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.181+0000 7fdba8e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4070e50 con 0x7fdba40717d0 2024-01-16T15:15:26.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.181+0000 7fdba8e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba40723c0 con 0x7fdba4143c40 2024-01-16T15:15:26.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.181+0000 7fdba8e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4144020 con 0x7fdba40ccc10 2024-01-16T15:15:26.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.181+0000 7fdb9a800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdba40717d0 0x7fdba40cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:32774/0 (socket says 172.21.15.141:32774) 2024-01-16T15:15:26.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.181+0000 7fdb9a800640 1 -- 172.21.15.141:0/871181906 learned_addr learned my addr 172.21.15.141:0/871181906 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:26.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.182+0000 7fdba3400640 1 --2- 172.21.15.141:0/871181906 >> v2:172.21.15.188:3300/0 conn(0x7fdba4143c40 0x7fdba4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.182+0000 7fdba3e00640 1 --2- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.182+0000 7fdba2a00640 1 -- 172.21.15.141:0/871181906 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461611576 0 0) 0x7fdba4070e50 con 0x7fdba40717d0 2024-01-16T15:15:26.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.182+0000 7fdba2a00640 1 -- 172.21.15.141:0/871181906 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb84003520 con 0x7fdba40717d0 2024-01-16T15:15:26.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.183+0000 7fdba3400640 1 -- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 msgr2=0x7fdba406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.183+0000 7fdba3400640 1 --2- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba406d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.183+0000 7fdba3400640 1 -- 172.21.15.141:0/871181906 >> v1:172.21.15.141:6789/0 conn(0x7fdba40717d0 legacy=0x7fdba40cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.183+0000 7fdba3400640 1 -- 172.21.15.141:0/871181906 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4071e00 con 0x7fdba4143c40 2024-01-16T15:15:26.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.184+0000 7fdba3400640 1 --2- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba4143c40 0x7fdba4071290 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009d90 tx=0x7fdb9402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77bcb6e92e281f10 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.187+0000 7fdba2a00640 1 -- 172.21.15.141:0/871181906 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9402cbc0 con 0x7fdba4143c40 2024-01-16T15:15:26.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.187+0000 7fdba2a00640 1 -- 172.21.15.141:0/871181906 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9402cd20 con 0x7fdba4143c40 2024-01-16T15:15:26.172 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.187+0000 7fdba2a00640 1 -- 172.21.15.141:0/871181906 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94034660 con 0x7fdba4143c40 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 -- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba4143c40 msgr2=0x7fdba4071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 --2- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba4143c40 0x7fdba4071290 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009d90 tx=0x7fdb9402be10 comp rx=0 tx=0).stop 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 -- 172.21.15.141:0/871181906 shutdown_connections 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 --2- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba406d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 --2- 172.21.15.141:0/871181906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba4143c40 0x7fdba4071290 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 -- 172.21.15.141:0/871181906 >> 172.21.15.141:0/871181906 conn(0x7fdba406a800 msgr2=0x7fdba406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 -- 172.21.15.141:0/871181906 shutdown_connections 2024-01-16T15:15:26.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.188+0000 7fdba8e00640 1 -- 172.21.15.141:0/871181906 wait complete. 2024-01-16T15:15:26.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.190+0000 7fdba8e00640 1 Processor -- start 2024-01-16T15:15:26.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.190+0000 7fdba8e00640 1 -- start start 2024-01-16T15:15:26.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba8e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40717d0 0x7fdba41e4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba8e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba8e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba4143c40 0x7fdba4141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba8e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba413aad0 con 0x7fdba4143c40 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba8e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba413adb0 con 0x7fdba40717d0 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba8e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba413b090 con 0x7fdba40ccc10 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba4143c40 0x7fdba4141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba4143c40 0x7fdba4141fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32866/0 (socket says 172.21.15.141:32866) 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.191+0000 7fdba3e00640 1 -- 172.21.15.141:0/515355746 learned_addr learned my addr 172.21.15.141:0/515355746 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdb9a800640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3400640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40717d0 0x7fdba41e4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3e00640 1 -- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 msgr2=0x7fdba415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3e00640 1 -- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40717d0 msgr2=0x7fdba41e4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40717d0 0x7fdba41e4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3e00640 1 -- 172.21.15.141:0/515355746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb94009a70 con 0x7fdba4143c40 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3400640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40717d0 0x7fdba41e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdba3e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba4143c40 0x7fdba4141fe0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00e930 tx=0x7fdb9c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdb9be00640 1 -- 172.21.15.141:0/515355746 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c009690 con 0x7fdba4143c40 2024-01-16T15:15:26.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.192+0000 7fdb9be00640 1 -- 172.21.15.141:0/515355746 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c00c980 con 0x7fdba4143c40 2024-01-16T15:15:26.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.193+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba413b370 con 0x7fdba4143c40 2024-01-16T15:15:26.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.193+0000 7fdb9be00640 1 -- 172.21.15.141:0/515355746 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c00cd30 con 0x7fdba4143c40 2024-01-16T15:15:26.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.193+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba413b890 con 0x7fdba4143c40 2024-01-16T15:15:26.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.194+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba413b590 con 0x7fdba4143c40 2024-01-16T15:15:26.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.195+0000 7fdb9be00640 1 -- 172.21.15.141:0/515355746 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fdb9c013470 con 0x7fdba4143c40 2024-01-16T15:15:26.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.196+0000 7fdb9be00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c4c0 0x7fdb7c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.196+0000 7fdb9be00640 1 -- 172.21.15.141:0/515355746 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdb9c088d60 con 0x7fdba4143c40 2024-01-16T15:15:26.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.196+0000 7fdba3400640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c4c0 0x7fdb7c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.198+0000 7fdba3400640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c4c0 0x7fdb7c06e980 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdb94036000 tx=0x7fdb94030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.200+0000 7fdb9be00640 1 -- 172.21.15.141:0/515355746 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb9c055370 con 0x7fdba4143c40 2024-01-16T15:15:26.414 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.430+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba413ac10 con 0x7fdba4143c40 2024-01-16T15:15:26.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.431+0000 7fdb9be00640 1 -- 172.21.15.141:0/515355746 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+3708 (secure 0 0 0) 0x7fdb9c021e90 con 0x7fdba4143c40 2024-01-16T15:15:26.416 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:26.416 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3880199275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":3880199275},{"type":"v1","addr":"172.21.15.141:6835","nonce":3880199275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2564906195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2564906195},{"type":"v1","addr":"172.21.15.141:6834","nonce":2564906195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6466,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/924822672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":924822672},{"type":"v1","addr":"172.21.15.188:6837","nonce":924822672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6694,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/1117717164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1117717164},{"type":"v1","addr":"172.21.15.188:6836","nonce":1117717164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[]} 2024-01-16T15:15:26.416 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 67 2024-01-16T15:15:26.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c4c0 msgr2=0x7fdb7c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c4c0 0x7fdb7c06e980 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdb94036000 tx=0x7fdb94030000 comp rx=0 tx=0).stop 2024-01-16T15:15:26.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba4143c40 msgr2=0x7fdba4141fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba4143c40 0x7fdba4141fe0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00e930 tx=0x7fdb9c00edf0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 shutdown_connections 2024-01-16T15:15:26.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c4c0 0x7fdb7c06e980 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba4143c40 0x7fdba4141fe0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40ccc10 0x7fdba415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 --2- 172.21.15.141:0/515355746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40717d0 0x7fdba41e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 >> 172.21.15.141:0/515355746 conn(0x7fdba406a800 msgr2=0x7fdba40cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 shutdown_connections 2024-01-16T15:15:26.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.433+0000 7fdba8e00640 1 -- 172.21.15.141:0/515355746 wait complete. 2024-01-16T15:15:26.437 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:15:26.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.664+0000 7fb37d600640 1 Processor -- start 2024-01-16T15:15:26.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.664+0000 7fb37d600640 1 -- start start 2024-01-16T15:15:26.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb37d600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb37815d580 0x7fb3780cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb37d600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb37815d960 0x7fb3780cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb37d600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780ce760 con 0x7fb3780b20d0 2024-01-16T15:15:26.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb37d600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3780d0a00 con 0x7fb37815d580 2024-01-16T15:15:26.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb37d600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780cf250 con 0x7fb37815d960 2024-01-16T15:15:26.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb37cc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb3780b20d0 0x7fb3780d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:32786/0 (socket says 172.21.15.141:32786) 2024-01-16T15:15:26.651 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb37cc00640 1 -- 172.21.15.141:0/3701410650 learned_addr learned my addr 172.21.15.141:0/3701410650 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:26.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.666+0000 7fb377e00640 1 --2- 172.21.15.141:0/3701410650 >> v2:172.21.15.188:3300/0 conn(0x7fb37815d580 0x7fb3780cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb377400640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb37815d960 0x7fb3780cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357302402 0 0) 0x7fb3780ce760 con 0x7fb3780b20d0 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb358003520 con 0x7fb3780b20d0 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892312616 0 0) 0x7fb358003520 con 0x7fb3780b20d0 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb37815d960 msgr2=0x7fb3780cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb37815d960 0x7fb3780cfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 >> v2:172.21.15.188:3300/0 conn(0x7fb37815d580 msgr2=0x7fb3780cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 --2- 172.21.15.141:0/3701410650 >> v2:172.21.15.188:3300/0 conn(0x7fb37815d580 0x7fb3780cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3780ce9b0 con 0x7fb3780b20d0 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fb3700019f0 con 0x7fb3780b20d0 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 >> v1:172.21.15.141:6789/0 conn(0x7fb3780b20d0 legacy=0x7fb3780d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb358003970 0x7fb358003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb358004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb37815d960 0x7fb358005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780d0a00 con 0x7fb3580047c0 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780cf250 con 0x7fb358003970 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.667+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35800b980 con 0x7fb37815d960 2024-01-16T15:15:26.654 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.668+0000 7fb377400640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb358004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.668+0000 7fb377e00640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb358003970 0x7fb358003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.668+0000 7fb377400640 1 -- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb37815d960 msgr2=0x7fb358005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.668+0000 7fb377400640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb37815d960 0x7fb358005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.668+0000 7fb377400640 1 -- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb358003970 msgr2=0x7fb358003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.668+0000 7fb377400640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb358003970 0x7fb358003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.668+0000 7fb377400640 1 -- 172.21.15.141:0/3701410650 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358003520 con 0x7fb3580047c0 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.669+0000 7fb377400640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb358004c40 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb360009af0 tx=0x7fb3580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9058f0353ee013 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.669+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36002c710 con 0x7fb3580047c0 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.669+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36002c870 con 0x7fb3580047c0 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.669+0000 7fb376a00640 1 -- 172.21.15.141:0/3701410650 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36002cc00 con 0x7fb3580047c0 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.670+0000 7fb37d600640 1 -- 172.21.15.141:0/3701410650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 msgr2=0x7fb358004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.670+0000 7fb37d600640 1 --2- 172.21.15.141:0/3701410650 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb358004c40 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb360009af0 tx=0x7fb3580057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.670+0000 7fb37cc00640 1 -- 172.21.15.141:0/3701410650 reap_dead start 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.670+0000 7fb37d600640 1 -- 172.21.15.141:0/3701410650 shutdown_connections 2024-01-16T15:15:26.656 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.670+0000 7fb37d600640 1 -- 172.21.15.141:0/3701410650 >> 172.21.15.141:0/3701410650 conn(0x7fb3780d3040 msgr2=0x7fb3780d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.670+0000 7fb37d600640 1 -- 172.21.15.141:0/3701410650 shutdown_connections 2024-01-16T15:15:26.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.670+0000 7fb37d600640 1 -- 172.21.15.141:0/3701410650 wait complete. 2024-01-16T15:15:26.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.672+0000 7fb37d600640 1 Processor -- start 2024-01-16T15:15:26.657 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.672+0000 7fb37d600640 1 -- start start 2024-01-16T15:15:26.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.673+0000 7fb37d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb378159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.673+0000 7fb37d600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb378159610 0x7fb37806b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.673+0000 7fb37d600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3781f6850 0x7fb3781f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.673+0000 7fb37d600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780ce7d0 con 0x7fb3580047c0 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.673+0000 7fb37d600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780b2e60 con 0x7fb378159610 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.673+0000 7fb37d600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780b3280 con 0x7fb3781f6850 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb378159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb378159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32890/0 (socket says 172.21.15.141:32890) 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 -- 172.21.15.141:0/2719452380 learned_addr learned my addr 172.21.15.141:0/2719452380 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 -- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3781f6850 msgr2=0x7fb3781f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3781f6850 0x7fb3781f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 -- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb378159610 msgr2=0x7fb37806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb378159610 0x7fb37806b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 -- 172.21.15.141:0/2719452380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3780ce4f0 con 0x7fb3580047c0 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.674+0000 7fb377e00640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb378159050 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb3680116d0 tx=0x7fb368011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.675+0000 7fb36fe00640 1 -- 172.21.15.141:0/2719452380 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36800f9c0 con 0x7fb3580047c0 2024-01-16T15:15:26.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.675+0000 7fb36fe00640 1 -- 172.21.15.141:0/2719452380 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36800fb20 con 0x7fb3580047c0 2024-01-16T15:15:26.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.675+0000 7fb36fe00640 1 -- 172.21.15.141:0/2719452380 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36800fe90 con 0x7fb3580047c0 2024-01-16T15:15:26.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.675+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3780b27a0 con 0x7fb3580047c0 2024-01-16T15:15:26.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.675+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3780b7c50 con 0x7fb3580047c0 2024-01-16T15:15:26.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.676+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3780b2600 con 0x7fb3580047c0 2024-01-16T15:15:26.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.677+0000 7fb36fe00640 1 -- 172.21.15.141:0/2719452380 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fb368015460 con 0x7fb3580047c0 2024-01-16T15:15:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.678+0000 7fb36fe00640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb35006c4c0 0x7fb35006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.678+0000 7fb36fe00640 1 -- 172.21.15.141:0/2719452380 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb368088cf0 con 0x7fb3580047c0 2024-01-16T15:15:26.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.679+0000 7fb377400640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb35006c4c0 0x7fb35006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.664 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.680+0000 7fb377400640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb35006c4c0 0x7fb35006e980 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb360036000 tx=0x7fb36002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.683+0000 7fb36fe00640 1 -- 172.21.15.141:0/2719452380 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3780b2600 con 0x7fb3580047c0 2024-01-16T15:15:26.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.875+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fb3780b2600 con 0x7fb3580047c0 2024-01-16T15:15:26.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.882+0000 7fb36fe00640 1 -- 172.21.15.141:0/2719452380 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v39) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb3780b2600 con 0x7fb3580047c0 2024-01-16T15:15:26.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb35006c4c0 msgr2=0x7fb35006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb35006c4c0 0x7fb35006e980 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb360036000 tx=0x7fb36002e000 comp rx=0 tx=0).stop 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 msgr2=0x7fb378159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb378159050 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb3680116d0 tx=0x7fb368011b90 comp rx=0 tx=0).stop 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 shutdown_connections 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb35006c4c0 0x7fb35006e980 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb3781f6850 0x7fb3781f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb378159610 0x7fb37806b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 --2- 172.21.15.141:0/2719452380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb3580047c0 0x7fb378159050 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 >> 172.21.15.141:0/2719452380 conn(0x7fb3780d3040 msgr2=0x7fb3780cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.885+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 shutdown_connections 2024-01-16T15:15:26.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:26.886+0000 7fb37d600640 1 -- 172.21.15.141:0/2719452380 wait complete. 2024-01-16T15:15:26.890 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:15:27.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.111+0000 7f796fe00640 1 Processor -- start 2024-01-16T15:15:27.096 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.111+0000 7f796fe00640 1 -- start start 2024-01-16T15:15:27.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.112+0000 7f796fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 0x7f796815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.112+0000 7f796fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79680b1ee0 con 0x7f7968143ae0 2024-01-16T15:15:27.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.112+0000 7f796fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79680b2780 con 0x7f796815e1e0 2024-01-16T15:15:27.097 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.112+0000 7f796fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796815e990 con 0x7f79680cf9a0 2024-01-16T15:15:27.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.113+0000 7f796d400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7968143ae0 0x7f79680cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:32790/0 (socket says 172.21.15.141:32790) 2024-01-16T15:15:27.098 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.113+0000 7f796d400640 1 -- 172.21.15.141:0/1454314198 learned_addr learned my addr 172.21.15.141:0/1454314198 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.113+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074268111 0 0) 0x7f79680b1ee0 con 0x7f7968143ae0 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.113+0000 7f796ca00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 0x7f796815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.113+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7948005730 con 0x7f7968143ae0 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 495848872 0 0) 0x7f7948005730 con 0x7f7968143ae0 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 msgr2=0x7f796815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 0x7f796815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 >> v1:172.21.15.188:6789/0 conn(0x7f796815e1e0 legacy=0x7f79680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680b2130 con 0x7f7968143ae0 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f79580019f0 con 0x7f7968143ae0 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 >> v1:172.21.15.141:6789/0 conn(0x7f7968143ae0 legacy=0x7f79680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7948005bb0 0x7f7948005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79480069d0 0x7f7948006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 0x7f79480073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796815e990 con 0x7f79480069d0 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794800dcc0 con 0x7f7948005bb0 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794800de80 con 0x7f79680cf9a0 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f796ca00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79480069d0 0x7f7948006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f796d400640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7948005bb0 0x7f7948005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f796de00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 0x7f79480073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.114+0000 7f796ca00640 1 -- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 msgr2=0x7f79480073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.115+0000 7f796ca00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 0x7f79480073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.115+0000 7f796ca00640 1 -- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7948005bb0 msgr2=0x7f7948005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.115+0000 7f796ca00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7948005bb0 0x7f7948005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.115+0000 7f796ca00640 1 -- 172.21.15.141:0/1454314198 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7948005730 con 0x7f79480069d0 2024-01-16T15:15:27.101 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.115+0000 7f796de00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680cf9a0 0x7f79480073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:27.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.116+0000 7f796ca00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79480069d0 0x7f7948006e50 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f795000ba30 tx=0x7f7948007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a687ad51915469 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.102 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.116+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950032630 con 0x7f79480069d0 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.116+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7950032790 con 0x7f79480069d0 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.116+0000 7f7967e00640 1 -- 172.21.15.141:0/1454314198 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950032b20 con 0x7f79480069d0 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.116+0000 7f796fe00640 1 -- 172.21.15.141:0/1454314198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79480069d0 msgr2=0x7f7948006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.116+0000 7f796fe00640 1 --2- 172.21.15.141:0/1454314198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79480069d0 0x7f7948006e50 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f795000ba30 tx=0x7f7948007a00 comp rx=0 tx=0).stop 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.116+0000 7f796de00640 1 -- 172.21.15.141:0/1454314198 reap_dead start 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.117+0000 7f796fe00640 1 -- 172.21.15.141:0/1454314198 shutdown_connections 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.117+0000 7f796fe00640 1 -- 172.21.15.141:0/1454314198 >> 172.21.15.141:0/1454314198 conn(0x7f796806a800 msgr2=0x7f796806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.117+0000 7f796fe00640 1 -- 172.21.15.141:0/1454314198 shutdown_connections 2024-01-16T15:15:27.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.117+0000 7f796fe00640 1 -- 172.21.15.141:0/1454314198 wait complete. 2024-01-16T15:15:27.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.118+0000 7f796fe00640 1 Processor -- start 2024-01-16T15:15:27.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.119+0000 7f796fe00640 1 -- start start 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79480069d0 0x7f79680adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79680b1440 0x7f7968159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680a9150 0x7f79680abb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7968140c30 con 0x7f79680b1440 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680b0fb0 con 0x7f79480069d0 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680b06e0 con 0x7f79680a9150 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79680b1440 0x7f7968159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79680b1440 0x7f7968159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32904/0 (socket says 172.21.15.141:32904) 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 -- 172.21.15.141:0/80479022 learned_addr learned my addr 172.21.15.141:0/80479022 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796d400640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79480069d0 0x7f79680adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796de00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680a9150 0x7f79680abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 -- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680a9150 msgr2=0x7f79680abb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680a9150 0x7f79680abb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 -- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79480069d0 msgr2=0x7f79680adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79480069d0 0x7f79680adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796ca00640 1 -- 172.21.15.141:0/80479022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680ae160 con 0x7f79680b1440 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.120+0000 7f796d400640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79480069d0 0x7f79680adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.121+0000 7f796ca00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79680b1440 0x7f7968159a00 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f795000bb50 tx=0x7f7950004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.121+0000 7f7965600640 1 -- 172.21.15.141:0/80479022 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795003ba70 con 0x7f79680b1440 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.121+0000 7f7965600640 1 -- 172.21.15.141:0/80479022 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795003bbd0 con 0x7f79680b1440 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.121+0000 7f7965600640 1 -- 172.21.15.141:0/80479022 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950030400 con 0x7f79680b1440 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.121+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79680b0960 con 0x7f79680b1440 2024-01-16T15:15:27.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.121+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79680a95d0 con 0x7f79680b1440 2024-01-16T15:15:27.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.122+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79680b1160 con 0x7f79680b1440 2024-01-16T15:15:27.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.123+0000 7f7965600640 1 -- 172.21.15.141:0/80479022 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7950030560 con 0x7f79680b1440 2024-01-16T15:15:27.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.124+0000 7f7965600640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f794006c4f0 0x7f794006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.124+0000 7f7965600640 1 -- 172.21.15.141:0/80479022 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f79500ab2c0 con 0x7f79680b1440 2024-01-16T15:15:27.109 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.125+0000 7f796d400640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f794006c4f0 0x7f794006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.126+0000 7f796d400640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f794006c4f0 0x7f794006e9b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f79580013d0 tx=0x7f79580020b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.129+0000 7f7965600640 1 -- 172.21.15.141:0/80479022 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79680b1160 con 0x7f79680b1440 2024-01-16T15:15:27.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.319+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f79680b1160 con 0x7f79680b1440 2024-01-16T15:15:27.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.326+0000 7f7965600640 1 -- 172.21.15.141:0/80479022 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v40) v1 ==== 120+0+0 (secure 0 0 0) 0x7f79680b1160 con 0x7f79680b1440 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.329+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f794006c4f0 msgr2=0x7f794006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.329+0000 7f796fe00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f794006c4f0 0x7f794006e9b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f79580013d0 tx=0x7f79580020b0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.329+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79680b1440 msgr2=0x7f7968159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.329+0000 7f796fe00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79680b1440 0x7f7968159a00 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f795000bb50 tx=0x7f7950004730 comp rx=0 tx=0).stop 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 shutdown_connections 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f794006c4f0 0x7f794006e9b0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f79680a9150 0x7f79680abb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f79680b1440 0x7f7968159a00 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 --2- 172.21.15.141:0/80479022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79480069d0 0x7f79680adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 >> 172.21.15.141:0/80479022 conn(0x7f796806a800 msgr2=0x7f796806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:27.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 shutdown_connections 2024-01-16T15:15:27.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.330+0000 7f796fe00640 1 -- 172.21.15.141:0/80479022 wait complete. 2024-01-16T15:15:27.334 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:15:27.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.563+0000 7fa2d9200640 1 Processor -- start 2024-01-16T15:15:27.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.564+0000 7fa2d9200640 1 -- start start 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.565+0000 7fa2d9200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa2d415e8d0 0x7fa2d40cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d9200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa2d4071200 0x7fa2d40b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d9200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d40b36f0 con 0x7fa2d406a3e0 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d9200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d40aec70 con 0x7fa2d4071200 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d9200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d40aef50 con 0x7fa2d415e8d0 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa2d4071200 0x7fa2d40b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d1400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa2d406a3e0 0x7fa2d4070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:32796/0 (socket says 172.21.15.141:32796) 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d1400640 1 -- 172.21.15.141:0/3726019043 learned_addr learned my addr 172.21.15.141:0/3726019043 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa2d4071200 0x7fa2d40b3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32908/0 (socket says 172.21.15.141:32908) 2024-01-16T15:15:27.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d1e00640 1 --2- 172.21.15.141:0/3726019043 >> v2:172.21.15.188:3300/0 conn(0x7fa2d415e8d0 0x7fa2d40cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d2800640 1 -- 172.21.15.141:0/3726019043 >> v2:172.21.15.188:3300/0 conn(0x7fa2d415e8d0 msgr2=0x7fa2d40cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:27.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d2800640 1 --2- 172.21.15.141:0/3726019043 >> v2:172.21.15.188:3300/0 conn(0x7fa2d415e8d0 0x7fa2d40cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.566+0000 7fa2d2800640 1 -- 172.21.15.141:0/3726019043 >> v1:172.21.15.141:6789/0 conn(0x7fa2d406a3e0 legacy=0x7fa2d4070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.567+0000 7fa2d2800640 1 -- 172.21.15.141:0/3726019043 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d40b3920 con 0x7fa2d4071200 2024-01-16T15:15:27.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.567+0000 7fa2d0a00640 1 -- 172.21.15.141:0/3726019043 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955954617 0 0) 0x7fa2d40b36f0 con 0x7fa2d406a3e0 2024-01-16T15:15:27.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.568+0000 7fa2d2800640 1 --2- 172.21.15.141:0/3726019043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d4071200 0x7fa2d40b3e40 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4070500 tx=0x7fa2cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6042e99610e0f258 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.568+0000 7fa2d0a00640 1 -- 172.21.15.141:0/3726019043 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc031830 con 0x7fa2d4071200 2024-01-16T15:15:27.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.568+0000 7fa2d0a00640 1 -- 172.21.15.141:0/3726019043 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc031990 con 0x7fa2d4071200 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.568+0000 7fa2d0a00640 1 -- 172.21.15.141:0/3726019043 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc031d20 con 0x7fa2d4071200 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.568+0000 7fa2d9200640 1 -- 172.21.15.141:0/3726019043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d4071200 msgr2=0x7fa2d40b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.568+0000 7fa2d9200640 1 --2- 172.21.15.141:0/3726019043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d4071200 0x7fa2d40b3e40 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4070500 tx=0x7fa2cc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.569+0000 7fa2d9200640 1 -- 172.21.15.141:0/3726019043 shutdown_connections 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.569+0000 7fa2d9200640 1 --2- 172.21.15.141:0/3726019043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d4071200 0x7fa2d40b3e40 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.569+0000 7fa2d9200640 1 --2- 172.21.15.141:0/3726019043 >> v2:172.21.15.188:3300/0 conn(0x7fa2d415e8d0 0x7fa2d40cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.569+0000 7fa2d9200640 1 -- 172.21.15.141:0/3726019043 >> 172.21.15.141:0/3726019043 conn(0x7fa2d40cf700 msgr2=0x7fa2d40cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.569+0000 7fa2d9200640 1 -- 172.21.15.141:0/3726019043 shutdown_connections 2024-01-16T15:15:27.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.569+0000 7fa2d9200640 1 -- 172.21.15.141:0/3726019043 wait complete. 2024-01-16T15:15:27.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.570+0000 7fa2d9200640 1 Processor -- start 2024-01-16T15:15:27.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.571+0000 7fa2d9200640 1 -- start start 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.571+0000 7fa2d9200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d406a3e0 0x7fa2d41523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.571+0000 7fa2d9200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4071200 0x7fa2d4155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d9200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa2d415e8d0 0x7fa2d40a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d9200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d4155ee0 con 0x7fa2d406a3e0 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d9200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d40acf60 con 0x7fa2d4071200 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d9200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d40ad240 con 0x7fa2d415e8d0 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d406a3e0 0x7fa2d41523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d406a3e0 0x7fa2d41523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32914/0 (socket says 172.21.15.141:32914) 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 -- 172.21.15.141:0/2484132918 learned_addr learned my addr 172.21.15.141:0/2484132918 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:27.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 -- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa2d415e8d0 msgr2=0x7fa2d40a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa2d415e8d0 0x7fa2d40a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 -- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4071200 msgr2=0x7fa2d4155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4071200 0x7fa2d4155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.572+0000 7fa2d1e00640 1 -- 172.21.15.141:0/2484132918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc009400 con 0x7fa2d406a3e0 2024-01-16T15:15:27.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.573+0000 7fa2d1e00640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d406a3e0 0x7fa2d41523e0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400d5f0 tx=0x7fa2c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.573+0000 7fa2ca000640 1 -- 172.21.15.141:0/2484132918 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c40125f0 con 0x7fa2d406a3e0 2024-01-16T15:15:27.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.573+0000 7fa2ca000640 1 -- 172.21.15.141:0/2484132918 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c4012750 con 0x7fa2d406a3e0 2024-01-16T15:15:27.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.573+0000 7fa2ca000640 1 -- 172.21.15.141:0/2484132918 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4012ae0 con 0x7fa2d406a3e0 2024-01-16T15:15:27.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.573+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d40ad520 con 0x7fa2d406a3e0 2024-01-16T15:15:27.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.573+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d4155bd0 con 0x7fa2d406a3e0 2024-01-16T15:15:27.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.574+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d40b3260 con 0x7fa2d406a3e0 2024-01-16T15:15:27.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.575+0000 7fa2ca000640 1 -- 172.21.15.141:0/2484132918 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa2c4012c40 con 0x7fa2d406a3e0 2024-01-16T15:15:27.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.576+0000 7fa2ca000640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2ac06c4c0 0x7fa2ac06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.576+0000 7fa2ca000640 1 -- 172.21.15.141:0/2484132918 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa2c4088c90 con 0x7fa2d406a3e0 2024-01-16T15:15:27.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.576+0000 7fa2d1400640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2ac06c4c0 0x7fa2ac06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.578+0000 7fa2d1400640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2ac06c4c0 0x7fa2ac06e980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc001560 tx=0x7fa2bc007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.580+0000 7fa2ca000640 1 -- 172.21.15.141:0/2484132918 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2c4055550 con 0x7fa2d406a3e0 2024-01-16T15:15:27.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.770+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa2d4073970 con 0x7fa2d406a3e0 2024-01-16T15:15:27.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.777+0000 7fa2ca000640 1 -- 172.21.15.141:0/2484132918 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa2c4058b70 con 0x7fa2d406a3e0 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.780+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2ac06c4c0 msgr2=0x7fa2ac06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.780+0000 7fa2d9200640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2ac06c4c0 0x7fa2ac06e980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc001560 tx=0x7fa2bc007ca0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.780+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d406a3e0 msgr2=0x7fa2d41523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.780+0000 7fa2d9200640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d406a3e0 0x7fa2d41523e0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400d5f0 tx=0x7fa2c400dab0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 shutdown_connections 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa2ac06c4c0 0x7fa2ac06e980 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa2d415e8d0 0x7fa2d40a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4071200 0x7fa2d4155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 --2- 172.21.15.141:0/2484132918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa2d406a3e0 0x7fa2d41523e0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 >> 172.21.15.141:0/2484132918 conn(0x7fa2d40cf700 msgr2=0x7fa2d406b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 shutdown_connections 2024-01-16T15:15:27.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:27.781+0000 7fa2d9200640 1 -- 172.21.15.141:0/2484132918 wait complete. 2024-01-16T15:15:27.785 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:15:27.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.014+0000 7f5afc000640 1 Processor -- start 2024-01-16T15:15:27.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.014+0000 7f5afc000640 1 -- start start 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.015+0000 7f5afc000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5af413df80 0x7f5af415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.015+0000 7f5afc000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af413e360 0x7f5af40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.015+0000 7f5afc000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af406c470 con 0x7f5af415f9c0 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.015+0000 7f5afc000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af406c750 con 0x7f5af413df80 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5afc000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af406ca30 con 0x7f5af413e360 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5af9600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5af413df80 0x7f5af415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5af9600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5af413df80 0x7f5af415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32926/0 (socket says 172.21.15.141:32926) 2024-01-16T15:15:28.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5af9600640 1 -- 172.21.15.141:0/856129362 learned_addr learned my addr 172.21.15.141:0/856129362 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:28.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5af9600640 1 -- 172.21.15.141:0/856129362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af413e360 msgr2=0x7f5af40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:28.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5af9600640 1 --2- 172.21.15.141:0/856129362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af413e360 0x7f5af40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5af9600640 1 -- 172.21.15.141:0/856129362 >> v1:172.21.15.188:6789/0 conn(0x7f5af415f9c0 legacy=0x7f5af41408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:28.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.016+0000 7f5af9600640 1 -- 172.21.15.141:0/856129362 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af406cd10 con 0x7f5af413df80 2024-01-16T15:15:28.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.017+0000 7f5af9600640 1 --2- 172.21.15.141:0/856129362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af415bee0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009e50 tx=0x7f5ae402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aabeb540f205c11 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5af3e00640 1 -- 172.21.15.141:0/856129362 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae402ca70 con 0x7f5af413df80 2024-01-16T15:15:28.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5af3e00640 1 -- 172.21.15.141:0/856129362 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae402cbd0 con 0x7f5af413df80 2024-01-16T15:15:28.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5af3e00640 1 -- 172.21.15.141:0/856129362 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40343f0 con 0x7f5af413df80 2024-01-16T15:15:28.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 -- 172.21.15.141:0/856129362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 msgr2=0x7f5af415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 --2- 172.21.15.141:0/856129362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af415bee0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009e50 tx=0x7f5ae402bed0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 -- 172.21.15.141:0/856129362 shutdown_connections 2024-01-16T15:15:28.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 --2- 172.21.15.141:0/856129362 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af413e360 0x7f5af40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 --2- 172.21.15.141:0/856129362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af415bee0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 -- 172.21.15.141:0/856129362 >> 172.21.15.141:0/856129362 conn(0x7f5af406a800 msgr2=0x7f5af406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:28.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 -- 172.21.15.141:0/856129362 shutdown_connections 2024-01-16T15:15:28.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.018+0000 7f5afc000640 1 -- 172.21.15.141:0/856129362 wait complete. 2024-01-16T15:15:28.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.020+0000 7f5afc000640 1 Processor -- start 2024-01-16T15:15:28.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.020+0000 7f5afc000640 1 -- start start 2024-01-16T15:15:28.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.021+0000 7f5afc000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.021+0000 7f5afc000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5af413e360 0x7f5af40b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.021+0000 7f5afc000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af415f9c0 0x7f5af4150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.021+0000 7f5afc000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41f1cb0 con 0x7f5af413df80 2024-01-16T15:15:28.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.021+0000 7f5afc000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41f1e90 con 0x7f5af413e360 2024-01-16T15:15:28.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.021+0000 7f5afc000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41f2170 con 0x7f5af415f9c0 2024-01-16T15:15:28.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af413d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32932/0 (socket says 172.21.15.141:32932) 2024-01-16T15:15:28.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 -- 172.21.15.141:0/172997857 learned_addr learned my addr 172.21.15.141:0/172997857 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:28.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 -- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af415f9c0 msgr2=0x7f5af4150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5afa000640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af415f9c0 0x7f5af4150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af8c00640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5af413e360 0x7f5af40b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af415f9c0 0x7f5af4150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 -- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5af413e360 msgr2=0x7f5af40b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5af413e360 0x7f5af40b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 -- 172.21.15.141:0/172997857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4009b30 con 0x7f5af413df80 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.022+0000 7f5af9600640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af413d940 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009e50 tx=0x7f5ae4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.023+0000 7f5af1600640 1 -- 172.21.15.141:0/172997857 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae403e450 con 0x7f5af413df80 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.023+0000 7f5af1600640 1 -- 172.21.15.141:0/172997857 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae403e5b0 con 0x7f5af413df80 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.023+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af41f2450 con 0x7f5af413df80 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.023+0000 7f5af1600640 1 -- 172.21.15.141:0/172997857 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae403e960 con 0x7f5af413df80 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.023+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af41f2a10 con 0x7f5af413df80 2024-01-16T15:15:28.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.024+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af40675d0 con 0x7f5af413df80 2024-01-16T15:15:28.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.025+0000 7f5af1600640 1 -- 172.21.15.141:0/172997857 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f5ae403eac0 con 0x7f5af413df80 2024-01-16T15:15:28.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.026+0000 7f5af1600640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5acc06c4c0 0x7f5acc06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.026+0000 7f5af1600640 1 -- 172.21.15.141:0/172997857 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5ae40aba80 con 0x7f5af413df80 2024-01-16T15:15:28.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.027+0000 7f5af8c00640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5acc06c4c0 0x7f5acc06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.028+0000 7f5af8c00640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5acc06c4c0 0x7f5acc06e980 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5af40b3ce0 tx=0x7f5adc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.031+0000 7f5af1600640 1 -- 172.21.15.141:0/172997857 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ae40780f0 con 0x7f5af413df80 2024-01-16T15:15:28.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:28.284+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f5af406df00 con 0x7f5af413df80 2024-01-16T15:15:29.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.195+0000 7f5af1600640 1 -- 172.21.15.141:0/172997857 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v790) v1 ==== 155+0+0 (secure 0 0 0) 0x7f5ae407b710 con 0x7f5af413df80 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5acc06c4c0 msgr2=0x7f5acc06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5acc06c4c0 0x7f5acc06e980 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5af40b3ce0 tx=0x7f5adc008040 comp rx=0 tx=0).stop 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 msgr2=0x7f5af413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af413d940 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009e50 tx=0x7f5ae4004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 shutdown_connections 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5acc06c4c0 0x7f5acc06e980 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5af415f9c0 0x7f5af4150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5af413e360 0x7f5af40b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 --2- 172.21.15.141:0/172997857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5af413df80 0x7f5af413d940 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.198+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 >> 172.21.15.141:0/172997857 conn(0x7f5af406a800 msgr2=0x7f5af40aef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.199+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 shutdown_connections 2024-01-16T15:15:29.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.199+0000 7f5afc000640 1 -- 172.21.15.141:0/172997857 wait complete. 2024-01-16T15:15:29.207 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:15:29.207 INFO:tasks.cephfs_test_runner:Starting test: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:15:29.207 INFO:tasks.cephfs_test_runner:test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:15:29.207 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:29.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.432+0000 7f6954a00640 1 Processor -- start 2024-01-16T15:15:29.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.433+0000 7f6954a00640 1 -- start start 2024-01-16T15:15:29.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.434+0000 7f6954a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6950143c40 0x7f6950071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.434+0000 7f6954a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500ccc10 0x7f695006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.434+0000 7f6954a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6950070e50 con 0x7f69500717d0 2024-01-16T15:15:29.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.434+0000 7f6954a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69500723c0 con 0x7f6950143c40 2024-01-16T15:15:29.419 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.434+0000 7f6954a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6950144020 con 0x7f69500ccc10 2024-01-16T15:15:29.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.435+0000 7f694fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500ccc10 0x7f695006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.435+0000 7f694f400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6950143c40 0x7f6950071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.435+0000 7f6947e00640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f69500717d0 0x7f69500cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:50698/0 (socket says 172.21.15.141:50698) 2024-01-16T15:15:29.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.435+0000 7f694f400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f6950143c40 0x7f6950071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:38312/0 (socket says 172.21.15.141:38312) 2024-01-16T15:15:29.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.435+0000 7f6947e00640 1 -- 172.21.15.141:0/1931159247 learned_addr learned my addr 172.21.15.141:0/1931159247 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:29.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.436+0000 7f694fe00640 1 -- 172.21.15.141:0/1931159247 >> v2:172.21.15.188:3300/0 conn(0x7f6950143c40 msgr2=0x7f6950071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:29.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.436+0000 7f694fe00640 1 --2- 172.21.15.141:0/1931159247 >> v2:172.21.15.188:3300/0 conn(0x7f6950143c40 0x7f6950071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.436+0000 7f694fe00640 1 -- 172.21.15.141:0/1931159247 >> v1:172.21.15.188:6789/0 conn(0x7f69500717d0 legacy=0x7f69500cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.436+0000 7f694fe00640 1 -- 172.21.15.141:0/1931159247 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950071e00 con 0x7f69500ccc10 2024-01-16T15:15:29.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.436+0000 7f694ea00640 1 -- 172.21.15.141:0/1931159247 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693846591 0 0) 0x7f6950070e50 con 0x7f69500717d0 2024-01-16T15:15:29.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.437+0000 7f694fe00640 1 --2- 172.21.15.141:0/1931159247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500ccc10 0x7f695006d160 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6948009d90 tx=0x7f694802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04d6456f0b9f61f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:29.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f694ea00640 1 -- 172.21.15.141:0/1931159247 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694802cbc0 con 0x7f69500ccc10 2024-01-16T15:15:29.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f694ea00640 1 -- 172.21.15.141:0/1931159247 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694802cd20 con 0x7f69500ccc10 2024-01-16T15:15:29.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f694ea00640 1 -- 172.21.15.141:0/1931159247 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948034660 con 0x7f69500ccc10 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f6954a00640 1 -- 172.21.15.141:0/1931159247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500ccc10 msgr2=0x7f695006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f6954a00640 1 --2- 172.21.15.141:0/1931159247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500ccc10 0x7f695006d160 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6948009d90 tx=0x7f694802be10 comp rx=0 tx=0).stop 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f6954a00640 1 -- 172.21.15.141:0/1931159247 shutdown_connections 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f6954a00640 1 --2- 172.21.15.141:0/1931159247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500ccc10 0x7f695006d160 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f6954a00640 1 --2- 172.21.15.141:0/1931159247 >> v2:172.21.15.188:3300/0 conn(0x7f6950143c40 0x7f6950071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f6954a00640 1 -- 172.21.15.141:0/1931159247 >> 172.21.15.141:0/1931159247 conn(0x7f695006a800 msgr2=0x7f695006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.438+0000 7f6954a00640 1 -- 172.21.15.141:0/1931159247 shutdown_connections 2024-01-16T15:15:29.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.439+0000 7f6954a00640 1 -- 172.21.15.141:0/1931159247 wait complete. 2024-01-16T15:15:29.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.441+0000 7f6954a00640 1 Processor -- start 2024-01-16T15:15:29.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.441+0000 7f6954a00640 1 -- start start 2024-01-16T15:15:29.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f6954a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500717d0 0x7f6950137630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f6954a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69500ccc10 0x7f69500a8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f6954a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6950143c40 0x7f69500aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f6954a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695014a540 con 0x7f6950143c40 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f6954a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695014a820 con 0x7f69500ccc10 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f6954a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695014ab00 con 0x7f69500717d0 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f694fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6950143c40 0x7f69500aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f694fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6950143c40 0x7f69500aeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32936/0 (socket says 172.21.15.141:32936) 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.442+0000 7f694fe00640 1 -- 172.21.15.141:0/3758344421 learned_addr learned my addr 172.21.15.141:0/3758344421 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f6947e00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69500ccc10 0x7f69500a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f694f400640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500717d0 0x7f6950137630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f694fe00640 1 -- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500717d0 msgr2=0x7f6950137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f694fe00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500717d0 0x7f6950137630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f694fe00640 1 -- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69500ccc10 msgr2=0x7f69500a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f694fe00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69500ccc10 0x7f69500a8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f694fe00640 1 -- 172.21.15.141:0/3758344421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948009a70 con 0x7f6950143c40 2024-01-16T15:15:29.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f694fe00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6950143c40 0x7f69500aeb30 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f6948009d90 tx=0x7f6948004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f6947400640 1 -- 172.21.15.141:0/3758344421 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948003b90 con 0x7f6950143c40 2024-01-16T15:15:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f6947400640 1 -- 172.21.15.141:0/3758344421 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694802ce90 con 0x7f6950143c40 2024-01-16T15:15:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695014ade0 con 0x7f6950143c40 2024-01-16T15:15:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f6947400640 1 -- 172.21.15.141:0/3758344421 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948009450 con 0x7f6950143c40 2024-01-16T15:15:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.443+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6950137b70 con 0x7f6950143c40 2024-01-16T15:15:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.445+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6950067450 con 0x7f6950143c40 2024-01-16T15:15:29.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.445+0000 7f6947400640 1 -- 172.21.15.141:0/3758344421 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f694803b050 con 0x7f6950143c40 2024-01-16T15:15:29.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.446+0000 7f6947400640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f692806c4c0 0x7f692806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.446+0000 7f6947400640 1 -- 172.21.15.141:0/3758344421 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f69480aab20 con 0x7f6950143c40 2024-01-16T15:15:29.432 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.447+0000 7f694f400640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f692806c4c0 0x7f692806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.448+0000 7f694f400640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f692806c4c0 0x7f692806e980 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f695014a0b0 tx=0x7f6940008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:29.435 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.451+0000 7f6947400640 1 -- 172.21.15.141:0/3758344421 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6948077440 con 0x7f6950143c40 2024-01-16T15:15:29.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.644+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69500ceb30 con 0x7f6950143c40 2024-01-16T15:15:29.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.646+0000 7f6947400640 1 -- 172.21.15.141:0/3758344421 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+13046 (secure 0 0 0) 0x7f694807aa60 con 0x7f6950143c40 2024-01-16T15:15:29.631 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:29.631 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":107,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:22.133123+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6836/1516772217":"2024-01-17T15:15:19.121421+0000","172.21.15.188:6834/1516772217":"2024-01-17T15:15:19.121421+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:29.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.648+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f692806c4c0 msgr2=0x7f692806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.648+0000 7f6954a00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f692806c4c0 0x7f692806e980 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f695014a0b0 tx=0x7f6940008040 comp rx=0 tx=0).stop 2024-01-16T15:15:29.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.648+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6950143c40 msgr2=0x7f69500aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.648+0000 7f6954a00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6950143c40 0x7f69500aeb30 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f6948009d90 tx=0x7f6948004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:29.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 shutdown_connections 2024-01-16T15:15:29.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f692806c4c0 0x7f692806e980 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6950143c40 0x7f69500aeb30 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69500ccc10 0x7f69500a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 --2- 172.21.15.141:0/3758344421 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f69500717d0 0x7f6950137630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 >> 172.21.15.141:0/3758344421 conn(0x7f695006a800 msgr2=0x7f69500cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:29.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 shutdown_connections 2024-01-16T15:15:29.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.649+0000 7f6954a00640 1 -- 172.21.15.141:0/3758344421 wait complete. 2024-01-16T15:15:29.654 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:29.655 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:29.863 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.878+0000 7fc71fe00640 1 Processor -- start 2024-01-16T15:15:29.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.879+0000 7fc71fe00640 1 -- start start 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.880+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013cf10 0x7fc72013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.880+0000 7fc71fe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc7200cddf0 0x7fc7200b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.880+0000 7fc71fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7200ce440 con 0x7fc72013fae0 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.880+0000 7fc71fe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72013ff60 con 0x7fc7200cddf0 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.880+0000 7fc71fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200ce720 con 0x7fc72013cf10 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc7200cddf0 0x7fc7200b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc7200cddf0 0x7fc7200b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32946/0 (socket says 172.21.15.141:32946) 2024-01-16T15:15:29.866 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71f400640 1 -- 172.21.15.141:0/3810525495 learned_addr learned my addr 172.21.15.141:0/3810525495 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:29.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71f400640 1 -- 172.21.15.141:0/3810525495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013cf10 msgr2=0x7fc72013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:29.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71ea00640 1 --2- 172.21.15.141:0/3810525495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013cf10 0x7fc72013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71f400640 1 --2- 172.21.15.141:0/3810525495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013cf10 0x7fc72013f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71f400640 1 -- 172.21.15.141:0/3810525495 >> v1:172.21.15.188:6789/0 conn(0x7fc72013fae0 legacy=0x7fc7200cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:29.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.881+0000 7fc71f400640 1 -- 172.21.15.141:0/3810525495 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200cea00 con 0x7fc7200cddf0 2024-01-16T15:15:29.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.882+0000 7fc71f400640 1 --2- 172.21.15.141:0/3810525495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200b2590 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e50 tx=0x7fc71802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a433b6d5ed84cfd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:29.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71e000640 1 -- 172.21.15.141:0/3810525495 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71802ca70 con 0x7fc7200cddf0 2024-01-16T15:15:29.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71e000640 1 -- 172.21.15.141:0/3810525495 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71802cbd0 con 0x7fc7200cddf0 2024-01-16T15:15:29.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71e000640 1 -- 172.21.15.141:0/3810525495 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7180343f0 con 0x7fc7200cddf0 2024-01-16T15:15:29.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 -- 172.21.15.141:0/3810525495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 msgr2=0x7fc7200b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 --2- 172.21.15.141:0/3810525495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200b2590 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e50 tx=0x7fc71802bed0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 -- 172.21.15.141:0/3810525495 shutdown_connections 2024-01-16T15:15:29.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 --2- 172.21.15.141:0/3810525495 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200b2590 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 --2- 172.21.15.141:0/3810525495 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013cf10 0x7fc72013f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 -- 172.21.15.141:0/3810525495 >> 172.21.15.141:0/3810525495 conn(0x7fc72006a7d0 msgr2=0x7fc72006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:29.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 -- 172.21.15.141:0/3810525495 shutdown_connections 2024-01-16T15:15:29.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.883+0000 7fc71fe00640 1 -- 172.21.15.141:0/3810525495 wait complete. 2024-01-16T15:15:29.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.885+0000 7fc71fe00640 1 Processor -- start 2024-01-16T15:15:29.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.885+0000 7fc71fe00640 1 -- start start 2024-01-16T15:15:29.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc72013cf10 0x7fc7201f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013fae0 0x7fc720071e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200706b0 con 0x7fc7200cddf0 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc720071210 con 0x7fc72013cf10 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200723d0 con 0x7fc72013fae0 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32962/0 (socket says 172.21.15.141:32962) 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.886+0000 7fc71ea00640 1 -- 172.21.15.141:0/771896219 learned_addr learned my addr 172.21.15.141:0/771896219 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc715e00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc72013cf10 0x7fc7201f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71f400640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013fae0 0x7fc720071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71ea00640 1 -- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013fae0 msgr2=0x7fc720071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71ea00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013fae0 0x7fc720071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.873 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71ea00640 1 -- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc72013cf10 msgr2=0x7fc7201f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71ea00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc72013cf10 0x7fc7201f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71ea00640 1 -- 172.21.15.141:0/771896219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc718009b30 con 0x7fc7200cddf0 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc715e00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc72013cf10 0x7fc7201f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71f400640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013fae0 0x7fc720071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71ea00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200716a0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c820 tx=0x7fc71000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc717e00640 1 -- 172.21.15.141:0/771896219 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710004220 con 0x7fc7200cddf0 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc717e00640 1 -- 172.21.15.141:0/771896219 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7100043a0 con 0x7fc7200cddf0 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc717e00640 1 -- 172.21.15.141:0/771896219 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710007a10 con 0x7fc7200cddf0 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.887+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc720072610 con 0x7fc7200cddf0 2024-01-16T15:15:29.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.888+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc720070bc0 con 0x7fc7200cddf0 2024-01-16T15:15:29.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.889+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc720067450 con 0x7fc7200cddf0 2024-01-16T15:15:29.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.890+0000 7fc717e00640 1 -- 172.21.15.141:0/771896219 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fc710007b70 con 0x7fc7200cddf0 2024-01-16T15:15:29.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.891+0000 7fc717e00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6f806c4c0 0x7fc6f806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:29.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.891+0000 7fc717e00640 1 -- 172.21.15.141:0/771896219 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc710088850 con 0x7fc7200cddf0 2024-01-16T15:15:29.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.892+0000 7fc715e00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6f806c4c0 0x7fc6f806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:29.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.893+0000 7fc715e00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6f806c4c0 0x7fc6f806e980 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc720157cd0 tx=0x7fc708008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:29.881 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:29.896+0000 7fc717e00640 1 -- 172.21.15.141:0/771896219 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7100bee90 con 0x7fc7200cddf0 2024-01-16T15:15:30.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.071+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7200cbf10 con 0x7fc7200cddf0 2024-01-16T15:15:30.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.072+0000 7fc717e00640 1 -- 172.21.15.141:0/771896219 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc71001e070 con 0x7fc7200cddf0 2024-01-16T15:15:30.057 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:15:30.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6f806c4c0 msgr2=0x7fc6f806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:30.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6f806c4c0 0x7fc6f806e980 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc720157cd0 tx=0x7fc708008040 comp rx=0 tx=0).stop 2024-01-16T15:15:30.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 msgr2=0x7fc7200716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200716a0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c820 tx=0x7fc71000cce0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 shutdown_connections 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc6f806c4c0 0x7fc6f806e980 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc72013fae0 0x7fc720071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc72013cf10 0x7fc7201f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 --2- 172.21.15.141:0/771896219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7200cddf0 0x7fc7200716a0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 >> 172.21.15.141:0/771896219 conn(0x7fc72006a7d0 msgr2=0x7fc720158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 shutdown_connections 2024-01-16T15:15:30.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.074+0000 7fc71fe00640 1 -- 172.21.15.141:0/771896219 wait complete. 2024-01-16T15:15:30.076 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled' 2024-01-16T15:15:30.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.295+0000 7ff5ff000640 1 Processor -- start 2024-01-16T15:15:30.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.296+0000 7ff5ff000640 1 -- start start 2024-01-16T15:15:30.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.297+0000 7ff5ff000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff5f813e360 0x7ff5f80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.297+0000 7ff5ff000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f815c5f0 con 0x7ff5f813df80 2024-01-16T15:15:30.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.297+0000 7ff5ff000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f806c470 con 0x7ff5f815f9c0 2024-01-16T15:15:30.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.297+0000 7ff5ff000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f806c750 con 0x7ff5f813e360 2024-01-16T15:15:30.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.298+0000 7ff5f7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff5f813df80 0x7ff5f815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:32802/0 (socket says 172.21.15.141:32802) 2024-01-16T15:15:30.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.298+0000 7ff5f7e00640 1 -- 172.21.15.141:0/3386047189 learned_addr learned my addr 172.21.15.141:0/3386047189 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.298+0000 7ff5ef200640 1 --2- 172.21.15.141:0/3386047189 >> v2:172.21.15.188:3300/0 conn(0x7ff5f813e360 0x7ff5f80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.298+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664632257 0 0) 0x7ff5f815c5f0 con 0x7ff5f813df80 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.298+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5d8003520 con 0x7ff5f813df80 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.298+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2618977807 0 0) 0x7ff5d8003520 con 0x7ff5f813df80 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 >> v2:172.21.15.188:3300/0 conn(0x7ff5f813e360 msgr2=0x7ff5f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 --2- 172.21.15.141:0/3386047189 >> v2:172.21.15.188:3300/0 conn(0x7ff5f813e360 0x7ff5f80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 >> v1:172.21.15.188:6789/0 conn(0x7ff5f815f9c0 legacy=0x7ff5f80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f806ca30 con 0x7ff5f813df80 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ff5e8001690 con 0x7ff5f813df80 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 >> v1:172.21.15.141:6789/0 conn(0x7ff5f813df80 legacy=0x7ff5f815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:30.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d80039c0 0x7ff5d8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5d80047c0 0x7ff5d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d8005180 0x7ff5d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f806c750 con 0x7ff5d8005180 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d800c080 con 0x7ff5d80047c0 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d800c240 con 0x7ff5d80039c0 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5fd000640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d8005180 0x7ff5d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5f7e00640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d80039c0 0x7ff5d8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5fd000640 1 -- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d80039c0 msgr2=0x7ff5d8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5ef200640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5d80047c0 0x7ff5d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5fd000640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d80039c0 0x7ff5d8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5fd000640 1 -- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5d80047c0 msgr2=0x7ff5d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:30.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5fd000640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5d80047c0 0x7ff5d8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.299+0000 7ff5fd000640 1 -- 172.21.15.141:0/3386047189 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d8003520 con 0x7ff5d8005180 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.300+0000 7ff5fd000640 1 -- 172.21.15.141:0/3386047189 reap_dead start 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.300+0000 7ff5fd000640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d8005180 0x7ff5d800b980 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0001e20 tx=0x7ff5d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a141622e9226df8a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0003b70 con 0x7ff5d8005180 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f0003cd0 con 0x7ff5d8005180 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5f7400640 1 -- 172.21.15.141:0/3386047189 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00052f0 con 0x7ff5d8005180 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5ff000640 1 -- 172.21.15.141:0/3386047189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d8005180 msgr2=0x7ff5d800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5ff000640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d8005180 0x7ff5d800b980 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0001e20 tx=0x7ff5d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5ff000640 1 -- 172.21.15.141:0/3386047189 shutdown_connections 2024-01-16T15:15:30.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5ff000640 1 --2- 172.21.15.141:0/3386047189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d8005180 0x7ff5d800b980 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5ff000640 1 -- 172.21.15.141:0/3386047189 >> 172.21.15.141:0/3386047189 conn(0x7ff5f806a800 msgr2=0x7ff5f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:30.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5ff000640 1 -- 172.21.15.141:0/3386047189 shutdown_connections 2024-01-16T15:15:30.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.301+0000 7ff5ff000640 1 -- 172.21.15.141:0/3386047189 wait complete. 2024-01-16T15:15:30.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.303+0000 7ff5ff000640 1 Processor -- start 2024-01-16T15:15:30.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.303+0000 7ff5ff000640 1 -- start start 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.304+0000 7ff5ff000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d8005180 0x7ff5f81524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.304+0000 7ff5ff000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f81f0c10 0x7ff5f80a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.304+0000 7ff5ff000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5f80a8a00 0x7ff5f815c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.304+0000 7ff5ff000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80a92a0 con 0x7ff5f80a8a00 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.304+0000 7ff5ff000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80a8fc0 con 0x7ff5f81f0c10 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.304+0000 7ff5ff000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8158740 con 0x7ff5d8005180 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.304+0000 7ff5fd000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5f80a8a00 0x7ff5f815c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5f80a8a00 0x7ff5f815c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32982/0 (socket says 172.21.15.141:32982) 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 -- 172.21.15.141:0/2029620303 learned_addr learned my addr 172.21.15.141:0/2029620303 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5f7e00640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d8005180 0x7ff5f81524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5ef200640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f81f0c10 0x7ff5f80a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 -- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d8005180 msgr2=0x7ff5f81524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:30.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d8005180 0x7ff5f81524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 -- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f81f0c10 msgr2=0x7ff5f80a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f81f0c10 0x7ff5f80a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 -- 172.21.15.141:0/2029620303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8158a20 con 0x7ff5f80a8a00 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5f7e00640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d8005180 0x7ff5f81524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5fd000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5f80a8a00 0x7ff5f815c850 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0001f40 tx=0x7ff5f00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.305+0000 7ff5f4c00640 1 -- 172.21.15.141:0/2029620303 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0006d50 con 0x7ff5f80a8a00 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.306+0000 7ff5f4c00640 1 -- 172.21.15.141:0/2029620303 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f0007020 con 0x7ff5f80a8a00 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.306+0000 7ff5f4c00640 1 -- 172.21.15.141:0/2029620303 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0007390 con 0x7ff5f80a8a00 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.306+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f8158d00 con 0x7ff5f80a8a00 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.306+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f8152a30 con 0x7ff5f80a8a00 2024-01-16T15:15:30.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.307+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f8158580 con 0x7ff5f80a8a00 2024-01-16T15:15:30.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.308+0000 7ff5f4c00640 1 -- 172.21.15.141:0/2029620303 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff5f003f030 con 0x7ff5f80a8a00 2024-01-16T15:15:30.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.309+0000 7ff5f4c00640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff5d006c4c0 0x7ff5d006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:30.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.309+0000 7ff5f4c00640 1 -- 172.21.15.141:0/2029620303 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff5f0003e40 con 0x7ff5f80a8a00 2024-01-16T15:15:30.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.310+0000 7ff5f7e00640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff5d006c4c0 0x7ff5d006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:30.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.311+0000 7ff5f7e00640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff5d006c4c0 0x7ff5d006e980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8001010 tx=0x7ff5e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:30.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.312+0000 7ff5f4c00640 1 -- 172.21.15.141:0/2029620303 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5f00774e0 con 0x7ff5f80a8a00 2024-01-16T15:15:30.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:30.574+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]} v 0) v1 -- 0x7ff5f80a92a0 con 0x7ff5f80a8a00 2024-01-16T15:15:31.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.210+0000 7ff5f4c00640 1 -- 172.21.15.141:0/2029620303 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]}]=0 v792) v1 ==== 158+0+0 (secure 0 0 0) 0x7ff5f0047030 con 0x7ff5f80a8a00 2024-01-16T15:15:31.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff5d006c4c0 msgr2=0x7ff5d006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:31.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff5d006c4c0 0x7ff5d006e980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8001010 tx=0x7ff5e8009040 comp rx=0 tx=0).stop 2024-01-16T15:15:31.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5f80a8a00 msgr2=0x7ff5f815c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:31.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5f80a8a00 0x7ff5f815c850 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0001f40 tx=0x7ff5f00014a0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 shutdown_connections 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff5d006c4c0 0x7ff5d006e980 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5f80a8a00 0x7ff5f815c850 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f81f0c10 0x7ff5f80a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 --2- 172.21.15.141:0/2029620303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff5d8005180 0x7ff5f81524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 >> 172.21.15.141:0/2029620303 conn(0x7ff5f806a800 msgr2=0x7ff5f815e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 shutdown_connections 2024-01-16T15:15:31.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.214+0000 7ff5ff000640 1 -- 172.21.15.141:0/2029620303 wait complete. 2024-01-16T15:15:31.223 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:15:31.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.452+0000 7f2af5a00640 1 Processor -- start 2024-01-16T15:15:31.437 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.452+0000 7f2af5a00640 1 -- start start 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2af5a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2af00cd240 0x7f2af00b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2af5a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2af006c6b0 0x7f2af0099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2af5a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af0099fa0 0x7f2af006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2af5a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af009a430 con 0x7f2af006c6b0 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2af5a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af006cea0 con 0x7f2af00cd240 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2af5a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af006cbc0 con 0x7f2af0099fa0 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2ae5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2af006c6b0 0x7f2af0099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2ae5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2af006c6b0 0x7f2af0099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32998/0 (socket says 172.21.15.141:32998) 2024-01-16T15:15:31.440 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.454+0000 7f2ae5e00640 1 -- 172.21.15.141:0/430875565 learned_addr learned my addr 172.21.15.141:0/430875565 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:31.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.455+0000 7f2aef400640 1 --2- 172.21.15.141:0/430875565 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af0099fa0 0x7f2af006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:31.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.455+0000 7f2aeea00640 1 --2- 172.21.15.141:0/430875565 >> v2:172.21.15.188:3300/0 conn(0x7f2af00cd240 0x7f2af00b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:31.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.455+0000 7f2ae5e00640 1 -- 172.21.15.141:0/430875565 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af0099fa0 msgr2=0x7f2af006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:31.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.455+0000 7f2ae5e00640 1 --2- 172.21.15.141:0/430875565 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af0099fa0 0x7f2af006b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.455+0000 7f2ae5e00640 1 -- 172.21.15.141:0/430875565 >> v2:172.21.15.188:3300/0 conn(0x7f2af00cd240 msgr2=0x7f2af00b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:31.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.455+0000 7f2ae5e00640 1 --2- 172.21.15.141:0/430875565 >> v2:172.21.15.188:3300/0 conn(0x7f2af00cd240 0x7f2af00b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.441 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.455+0000 7f2ae5e00640 1 -- 172.21.15.141:0/430875565 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af006f350 con 0x7f2af006c6b0 2024-01-16T15:15:31.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.456+0000 7f2ae5e00640 1 --2- 172.21.15.141:0/430875565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af006c6b0 0x7f2af0099a60 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f2af0070a10 tx=0x7f2ad802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d35b2476be6b47 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:31.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.456+0000 7f2aee000640 1 -- 172.21.15.141:0/430875565 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8031860 con 0x7f2af006c6b0 2024-01-16T15:15:31.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.456+0000 7f2aee000640 1 -- 172.21.15.141:0/430875565 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad80319c0 con 0x7f2af006c6b0 2024-01-16T15:15:31.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.456+0000 7f2aee000640 1 -- 172.21.15.141:0/430875565 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8031d50 con 0x7f2af006c6b0 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 -- 172.21.15.141:0/430875565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af006c6b0 msgr2=0x7f2af0099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 --2- 172.21.15.141:0/430875565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af006c6b0 0x7f2af0099a60 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f2af0070a10 tx=0x7f2ad802b900 comp rx=0 tx=0).stop 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 -- 172.21.15.141:0/430875565 shutdown_connections 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 --2- 172.21.15.141:0/430875565 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af0099fa0 0x7f2af006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 --2- 172.21.15.141:0/430875565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af006c6b0 0x7f2af0099a60 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 --2- 172.21.15.141:0/430875565 >> v2:172.21.15.188:3300/0 conn(0x7f2af00cd240 0x7f2af00b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 -- 172.21.15.141:0/430875565 >> 172.21.15.141:0/430875565 conn(0x7f2af00b2d00 msgr2=0x7f2af00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 -- 172.21.15.141:0/430875565 shutdown_connections 2024-01-16T15:15:31.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.457+0000 7f2af5a00640 1 -- 172.21.15.141:0/430875565 wait complete. 2024-01-16T15:15:31.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.459+0000 7f2af5a00640 1 Processor -- start 2024-01-16T15:15:31.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.459+0000 7f2af5a00640 1 -- start start 2024-01-16T15:15:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2af5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af006c6b0 0x7f2af0156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2af5a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af0099fa0 0x7f2af01f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2af5a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af00cd240 0x7f2af00af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2af5a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af00b4060 con 0x7f2af0099fa0 2024-01-16T15:15:31.446 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2af5a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af01565c0 con 0x7f2af006c6b0 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2af5a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af01f70b0 con 0x7f2af00cd240 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2ae5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af0099fa0 0x7f2af01f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2ae5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af0099fa0 0x7f2af01f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33010/0 (socket says 172.21.15.141:33010) 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2ae5e00640 1 -- 172.21.15.141:0/777789931 learned_addr learned my addr 172.21.15.141:0/777789931 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2aeea00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af006c6b0 0x7f2af0156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2ae5e00640 1 -- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af00cd240 msgr2=0x7f2af00af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.460+0000 7f2ae5e00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af00cd240 0x7f2af00af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2ae5e00640 1 -- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af006c6b0 msgr2=0x7f2af0156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2ae5e00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af006c6b0 0x7f2af0156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2ae5e00640 1 -- 172.21.15.141:0/777789931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad80094d0 con 0x7f2af0099fa0 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2ae5e00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af0099fa0 0x7f2af01f6b70 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f2ad802b940 tx=0x7f2ad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2ae7400640 1 -- 172.21.15.141:0/777789931 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad803ac60 con 0x7f2af0099fa0 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2ae7400640 1 -- 172.21.15.141:0/777789931 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad803adc0 con 0x7f2af0099fa0 2024-01-16T15:15:31.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2ae7400640 1 -- 172.21.15.141:0/777789931 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8044610 con 0x7f2af0099fa0 2024-01-16T15:15:31.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.461+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af01f7350 con 0x7f2af0099fa0 2024-01-16T15:15:31.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.462+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af01f7910 con 0x7f2af0099fa0 2024-01-16T15:15:31.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.463+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af00675d0 con 0x7f2af0099fa0 2024-01-16T15:15:31.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.463+0000 7f2ae7400640 1 -- 172.21.15.141:0/777789931 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2ad8044770 con 0x7f2af0099fa0 2024-01-16T15:15:31.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.464+0000 7f2ae7400640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2ac806c4c0 0x7f2ac806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:31.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.464+0000 7f2ae7400640 1 -- 172.21.15.141:0/777789931 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2ad80ab100 con 0x7f2af0099fa0 2024-01-16T15:15:31.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.465+0000 7f2aeea00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2ac806c4c0 0x7f2ac806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:31.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.466+0000 7f2aeea00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2ac806c4c0 0x7f2ac806e980 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0006fd0 tx=0x7f2ae0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:31.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.469+0000 7f2ae7400640 1 -- 172.21.15.141:0/777789931 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ad8077650 con 0x7f2af0099fa0 2024-01-16T15:15:31.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.659+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2af00b09b0 con 0x7f2af0099fa0 2024-01-16T15:15:31.650 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.664+0000 7f2ae7400640 1 -- 172.21.15.141:0/777789931 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2ad807ac70 con 0x7f2af0099fa0 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2ac806c4c0 msgr2=0x7f2ac806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2ac806c4c0 0x7f2ac806e980 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0006fd0 tx=0x7f2ae0008040 comp rx=0 tx=0).stop 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af0099fa0 msgr2=0x7f2af01f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af0099fa0 0x7f2af01f6b70 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f2ad802b940 tx=0x7f2ad80046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 shutdown_connections 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2ac806c4c0 0x7f2ac806e980 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2af00cd240 0x7f2af00af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2af0099fa0 0x7f2af01f6b70 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 --2- 172.21.15.141:0/777789931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2af006c6b0 0x7f2af0156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.667+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 >> 172.21.15.141:0/777789931 conn(0x7f2af00b2d00 msgr2=0x7f2af00b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.668+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 shutdown_connections 2024-01-16T15:15:31.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:31.668+0000 7f2af5a00640 1 -- 172.21.15.141:0/777789931 wait complete. 2024-01-16T15:15:31.673 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:15:31.673 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:31.782 DEBUG:teuthology.orchestra.run.smithi188:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:31.901 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:15:31.901 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:15:31.901 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:31.958 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:31.958 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:15:31.958 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:32.014 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:32.015 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:15:32.015 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:32.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:32.070 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:15:32.070 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:32.128 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:32.128 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:15:32.128 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:32.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.359+0000 7f0abba00640 1 Processor -- start 2024-01-16T15:15:32.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.360+0000 7f0abba00640 1 -- start start 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.361+0000 7f0abba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab4143ae0 0x7f0ab40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.361+0000 7f0abba00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0ab415e3b0 0x7f0ab40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.361+0000 7f0abba00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab40cfc30 con 0x7f0ab40cf7d0 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.361+0000 7f0abba00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab40b1ee0 con 0x7f0ab415e3b0 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.361+0000 7f0abba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40b2780 con 0x7f0ab4143ae0 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0ab415e3b0 0x7f0ab40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0ab415e3b0 0x7f0ab40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33022/0 (socket says 172.21.15.141:33022) 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9a00640 1 -- 172.21.15.141:0/923513761 learned_addr learned my addr 172.21.15.141:0/923513761 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9000640 1 --2- 172.21.15.141:0/923513761 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab4143ae0 0x7f0ab40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9a00640 1 -- 172.21.15.141:0/923513761 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab4143ae0 msgr2=0x7f0ab40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9a00640 1 --2- 172.21.15.141:0/923513761 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab4143ae0 0x7f0ab40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9a00640 1 -- 172.21.15.141:0/923513761 >> v1:172.21.15.141:6789/0 conn(0x7f0ab40cf7d0 legacy=0x7f0ab415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab3e00640 1 -- 172.21.15.141:0/923513761 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882707953 0 0) 0x7f0ab40cfc30 con 0x7f0ab40cf7d0 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.362+0000 7f0ab9a00640 1 -- 172.21.15.141:0/923513761 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab415e990 con 0x7f0ab415e3b0 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.363+0000 7f0ab9a00640 1 --2- 172.21.15.141:0/923513761 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab415e3b0 0x7f0ab40702c0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009870 tx=0x7f0aac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc4b056ad1af011 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0ab3e00640 1 -- 172.21.15.141:0/923513761 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac030840 con 0x7f0ab415e3b0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0ab3e00640 1 -- 172.21.15.141:0/923513761 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac0309a0 con 0x7f0ab415e3b0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0ab3e00640 1 -- 172.21.15.141:0/923513761 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac030d30 con 0x7f0ab415e3b0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0abba00640 1 -- 172.21.15.141:0/923513761 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab415e3b0 msgr2=0x7f0ab40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0abba00640 1 --2- 172.21.15.141:0/923513761 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab415e3b0 0x7f0ab40702c0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009870 tx=0x7f0aac02b980 comp rx=0 tx=0).stop 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0abba00640 1 -- 172.21.15.141:0/923513761 shutdown_connections 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0abba00640 1 --2- 172.21.15.141:0/923513761 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab415e3b0 0x7f0ab40702c0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0abba00640 1 --2- 172.21.15.141:0/923513761 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab4143ae0 0x7f0ab40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0abba00640 1 -- 172.21.15.141:0/923513761 >> 172.21.15.141:0/923513761 conn(0x7f0ab406a800 msgr2=0x7f0ab406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.364+0000 7f0abba00640 1 -- 172.21.15.141:0/923513761 shutdown_connections 2024-01-16T15:15:32.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.365+0000 7f0abba00640 1 -- 172.21.15.141:0/923513761 wait complete. 2024-01-16T15:15:32.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.367+0000 7f0abba00640 1 Processor -- start 2024-01-16T15:15:32.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.367+0000 7f0abba00640 1 -- start start 2024-01-16T15:15:32.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0abba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab40cf7d0 0x7f0ab40ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0abba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab4143ae0 0x7f0ab41598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0abba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ab415e3b0 0x7f0ab40afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0abba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40ae5b0 con 0x7f0ab4143ae0 2024-01-16T15:15:32.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0abba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab41362d0 con 0x7f0ab415e3b0 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0abba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab4133950 con 0x7f0ab40cf7d0 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0aabc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab4143ae0 0x7f0ab41598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0aabc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab4143ae0 0x7f0ab41598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33028/0 (socket says 172.21.15.141:33028) 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.368+0000 7f0aabc00640 1 -- 172.21.15.141:0/2954719945 learned_addr learned my addr 172.21.15.141:0/2954719945 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0ab9000640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab40cf7d0 0x7f0ab40ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0aabc00640 1 -- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab40cf7d0 msgr2=0x7f0ab40ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0aabc00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab40cf7d0 0x7f0ab40ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0ab9a00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ab415e3b0 0x7f0ab40afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0aabc00640 1 -- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ab415e3b0 msgr2=0x7f0ab40afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0aabc00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ab415e3b0 0x7f0ab40afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0aabc00640 1 -- 172.21.15.141:0/2954719945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0094d0 con 0x7f0ab4143ae0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0aabc00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab4143ae0 0x7f0ab41598e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c890 tx=0x7f0a9c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0ab1600640 1 -- 172.21.15.141:0/2954719945 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c003da0 con 0x7f0ab4143ae0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0ab1600640 1 -- 172.21.15.141:0/2954719945 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c005070 con 0x7f0ab4143ae0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab4133c30 con 0x7f0ab4143ae0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0ab1600640 1 -- 172.21.15.141:0/2954719945 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c005420 con 0x7f0ab4143ae0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.369+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab41341f0 con 0x7f0ab4143ae0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.371+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab40675d0 con 0x7f0ab4143ae0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.372+0000 7f0ab1600640 1 -- 172.21.15.141:0/2954719945 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f0a9c005580 con 0x7f0ab4143ae0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.373+0000 7f0ab1600640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a8c06c4c0 0x7f0a8c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.373+0000 7f0ab1600640 1 -- 172.21.15.141:0/2954719945 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0a9c088540 con 0x7f0ab4143ae0 2024-01-16T15:15:32.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.374+0000 7f0ab9000640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a8c06c4c0 0x7f0a8c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.375+0000 7f0ab9000640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a8c06c4c0 0x7f0a8c06e980 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4006fd0 tx=0x7f0aa4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.378+0000 7f0ab1600640 1 -- 172.21.15.141:0/2954719945 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a9c054a60 con 0x7f0ab4143ae0 2024-01-16T15:15:32.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.606+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ab40704d0 con 0x7f0ab4143ae0 2024-01-16T15:15:32.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.607+0000 7f0ab1600640 1 -- 172.21.15.141:0/2954719945 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f0a9c017020 con 0x7f0ab4143ae0 2024-01-16T15:15:32.592 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:32.593 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3880199275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":3880199275},{"type":"v1","addr":"172.21.15.141:6835","nonce":3880199275}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2564906195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2564906195},{"type":"v1","addr":"172.21.15.141:6834","nonce":2564906195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":6466,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/924822672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6835","nonce":924822672},{"type":"v1","addr":"172.21.15.188:6837","nonce":924822672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6694,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6836/1117717164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1117717164},{"type":"v1","addr":"172.21.15.188:6836","nonce":1117717164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[]} 2024-01-16T15:15:32.593 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 67 2024-01-16T15:15:32.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.609+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a8c06c4c0 msgr2=0x7f0a8c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.609+0000 7f0abba00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a8c06c4c0 0x7f0a8c06e980 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4006fd0 tx=0x7f0aa4008040 comp rx=0 tx=0).stop 2024-01-16T15:15:32.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.609+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab4143ae0 msgr2=0x7f0ab41598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.609+0000 7f0abba00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab4143ae0 0x7f0ab41598e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c890 tx=0x7f0a9c00cd50 comp rx=0 tx=0).stop 2024-01-16T15:15:32.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 shutdown_connections 2024-01-16T15:15:32.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0a8c06c4c0 0x7f0a8c06e980 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ab415e3b0 0x7f0ab40afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ab4143ae0 0x7f0ab41598e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 --2- 172.21.15.141:0/2954719945 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0ab40cf7d0 0x7f0ab40ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 >> 172.21.15.141:0/2954719945 conn(0x7f0ab406a800 msgr2=0x7f0ab40708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:32.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 shutdown_connections 2024-01-16T15:15:32.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.610+0000 7f0abba00640 1 -- 172.21.15.141:0/2954719945 wait complete. 2024-01-16T15:15:32.616 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:15:32.616 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:15:32.616 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:15:32.616 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:32.616 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:15:32.617 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:15:32.617 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:15:32.617 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:32.617 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:15:32.617 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:15:32.618 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:15:32.618 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:32.618 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:15:32.618 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:15:32.618 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:15:32.618 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:32.641 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:15:32.641 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:15:32.645 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:15:32.645 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:15:32.646 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:15:32.646 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:15:32.647 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:15:32.648 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:15:32.692 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:15:32.692 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:32.695 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:15:32.695 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:32.698 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:15:32.698 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:32.702 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:15:32.702 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:32.702 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:15:32.845 INFO:tasks.ceph.mds.b.smithi188.stdout:starting mds.b at 2024-01-16T15:15:32.849 INFO:tasks.ceph.mds.a.smithi141.stdout:starting mds.a at 2024-01-16T15:15:32.849 INFO:tasks.ceph.mds.c.smithi141.stdout:starting mds.c at 2024-01-16T15:15:32.851 INFO:tasks.ceph.mds.d.smithi188.stdout:starting mds.d at 2024-01-16T15:15:32.883 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.898+0000 7f6320600640 1 Processor -- start 2024-01-16T15:15:32.884 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.899+0000 7f6320600640 1 -- start start 2024-01-16T15:15:32.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.900+0000 7f6320600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f631813e360 0x7f63180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.900+0000 7f6320600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631815c5f0 con 0x7f631815f9c0 2024-01-16T15:15:32.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.900+0000 7f6320600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631806c470 con 0x7f631813df80 2024-01-16T15:15:32.885 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.900+0000 7f6320600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f631806c750 con 0x7f631813e360 2024-01-16T15:15:32.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.900+0000 7f631e600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f631815f9c0 0x7f63180cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:32838/0 (socket says 172.21.15.141:32838) 2024-01-16T15:15:32.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.900+0000 7f631e600640 1 -- 172.21.15.141:0/3113488829 learned_addr learned my addr 172.21.15.141:0/3113488829 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:32.886 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.900+0000 7f631d200640 1 --2- 172.21.15.141:0/3113488829 >> v2:172.21.15.188:3300/0 conn(0x7f631813e360 0x7f63180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.901+0000 7f6317e00640 1 -- 172.21.15.141:0/3113488829 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932990814 0 0) 0x7f631806c470 con 0x7f631813df80 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.901+0000 7f6317e00640 1 -- 172.21.15.141:0/3113488829 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f8003520 con 0x7f631813df80 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.901+0000 7f631d200640 1 -- 172.21.15.141:0/3113488829 >> v1:172.21.15.188:6789/0 conn(0x7f631813df80 legacy=0x7f631815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.901+0000 7f631d200640 1 -- 172.21.15.141:0/3113488829 >> v1:172.21.15.141:6789/0 conn(0x7f631815f9c0 legacy=0x7f63180cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.901+0000 7f631d200640 1 -- 172.21.15.141:0/3113488829 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631806ca30 con 0x7f631813e360 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.902+0000 7f631d200640 1 --2- 172.21.15.141:0/3113488829 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63180cf290 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f63000097a0 tx=0x7f630002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f1a67c620bd04d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.902+0000 7f6317e00640 1 -- 172.21.15.141:0/3113488829 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300030610 con 0x7f631813e360 2024-01-16T15:15:32.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.902+0000 7f6317e00640 1 -- 172.21.15.141:0/3113488829 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6300030770 con 0x7f631813e360 2024-01-16T15:15:32.888 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.902+0000 7f6317e00640 1 -- 172.21.15.141:0/3113488829 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300030b00 con 0x7f631813e360 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.903+0000 7f6320600640 1 -- 172.21.15.141:0/3113488829 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 msgr2=0x7f63180cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.903+0000 7f6320600640 1 --2- 172.21.15.141:0/3113488829 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63180cf290 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f63000097a0 tx=0x7f630002b960 comp rx=0 tx=0).stop 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.903+0000 7f6320600640 1 -- 172.21.15.141:0/3113488829 shutdown_connections 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.903+0000 7f6320600640 1 --2- 172.21.15.141:0/3113488829 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63180cf290 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.903+0000 7f6320600640 1 -- 172.21.15.141:0/3113488829 >> 172.21.15.141:0/3113488829 conn(0x7f631806a800 msgr2=0x7f631806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.903+0000 7f6320600640 1 -- 172.21.15.141:0/3113488829 shutdown_connections 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.903+0000 7f6320600640 1 -- 172.21.15.141:0/3113488829 wait complete. 2024-01-16T15:15:32.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.904+0000 7f6320600640 1 Processor -- start 2024-01-16T15:15:32.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.905+0000 7f6320600640 1 -- start start 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.905+0000 7f6320600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f631813df80 0x7f631815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.905+0000 7f6320600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63181565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.905+0000 7f6320600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f631815f9c0 0x7f6318156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.905+0000 7f6320600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631814c280 con 0x7f631813df80 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.905+0000 7f6320600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631814b820 con 0x7f631813e360 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.905+0000 7f6320600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6318155f10 con 0x7f631815f9c0 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f631813df80 0x7f631815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f631813df80 0x7f631815a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33070/0 (socket says 172.21.15.141:33070) 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 -- 172.21.15.141:0/3645346104 learned_addr learned my addr 172.21.15.141:0/3645346104 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631e600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f631815f9c0 0x7f6318156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631d200640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63181565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 -- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f631815f9c0 msgr2=0x7f6318156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f631815f9c0 0x7f6318156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 -- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 msgr2=0x7f63181565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63181565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631dc00640 1 -- 172.21.15.141:0/3645346104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300009400 con 0x7f631813df80 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.906+0000 7f631d200640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63181565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.907+0000 7f631dc00640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f631813df80 0x7f631815a5e0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f630800c7a0 tx=0x7f630800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.907+0000 7f6315600640 1 -- 172.21.15.141:0/3645346104 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308003b70 con 0x7f631813df80 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.907+0000 7f6315600640 1 -- 172.21.15.141:0/3645346104 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6308003cd0 con 0x7f631813df80 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.907+0000 7f6315600640 1 -- 172.21.15.141:0/3645346104 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630800aa90 con 0x7f631813df80 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.907+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63181561f0 con 0x7f631813df80 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.907+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631814bf10 con 0x7f631813df80 2024-01-16T15:15:32.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.908+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63180b09c0 con 0x7f631813df80 2024-01-16T15:15:32.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.909+0000 7f6315600640 1 -- 172.21.15.141:0/3645346104 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f630800abf0 con 0x7f631813df80 2024-01-16T15:15:32.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.911+0000 7f6315600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f62f006c4f0 0x7f62f006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.911+0000 7f6315600640 1 -- 172.21.15.141:0/3645346104 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6308088970 con 0x7f631813df80 2024-01-16T15:15:32.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.911+0000 7f631d200640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f62f006c4f0 0x7f62f006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.912+0000 7f631d200640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f62f006c4f0 0x7f62f006e9b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6300036000 tx=0x7f630003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:32.915+0000 7f6315600640 1 -- 172.21.15.141:0/3645346104 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6308054f20 con 0x7f631813df80 2024-01-16T15:15:33.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.132+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f63180675d0 con 0x7f631813df80 2024-01-16T15:15:33.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.972+0000 7f6315600640 1 -- 172.21.15.141:0/3645346104 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v112) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6308058540 con 0x7f631813df80 2024-01-16T15:15:33.958 INFO:teuthology.orchestra.run.smithi141.stderr: removed all blocklist entries 2024-01-16T15:15:33.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.977+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f62f006c4f0 msgr2=0x7f62f006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.977+0000 7f6320600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f62f006c4f0 0x7f62f006e9b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6300036000 tx=0x7f630003b040 comp rx=0 tx=0).stop 2024-01-16T15:15:33.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.977+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f631813df80 msgr2=0x7f631815a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.977+0000 7f6320600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f631813df80 0x7f631815a5e0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f630800c7a0 tx=0x7f630800cc60 comp rx=0 tx=0).stop 2024-01-16T15:15:33.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 shutdown_connections 2024-01-16T15:15:33.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f62f006c4f0 0x7f62f006e9b0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f631815f9c0 0x7f6318156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f631813e360 0x7f63181565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 --2- 172.21.15.141:0/3645346104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f631813df80 0x7f631815a5e0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 >> 172.21.15.141:0/3645346104 conn(0x7f631806a800 msgr2=0x7f63180aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:33.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 shutdown_connections 2024-01-16T15:15:33.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:33.978+0000 7f6320600640 1 -- 172.21.15.141:0/3645346104 wait complete. 2024-01-16T15:15:33.985 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:15:34.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.206+0000 7fd11b200640 1 Processor -- start 2024-01-16T15:15:34.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.207+0000 7fd11b200640 1 -- start start 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.208+0000 7fd11b200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd11c06c470 0x7fd11c0cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.208+0000 7fd11b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.208+0000 7fd11b200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11c06ca10 con 0x7fd11c1447f0 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.208+0000 7fd11b200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11c06d690 con 0x7fd11c06c470 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.208+0000 7fd11b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c0cebb0 con 0x7fd11c06dab0 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd119400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd11c06c470 0x7fd11c0cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd119400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd11c06c470 0x7fd11c0cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57600/0 (socket says 172.21.15.141:57600) 2024-01-16T15:15:34.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd119400640 1 -- 172.21.15.141:0/2757973212 learned_addr learned my addr 172.21.15.141:0/2757973212 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:34.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd11a800640 1 --2- 172.21.15.141:0/2757973212 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd119400640 1 -- 172.21.15.141:0/2757973212 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 msgr2=0x7fd11c0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd119400640 1 --2- 172.21.15.141:0/2757973212 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd119400640 1 -- 172.21.15.141:0/2757973212 >> v1:172.21.15.188:6789/0 conn(0x7fd11c1447f0 legacy=0x7fd11c15c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.209+0000 7fd119400640 1 -- 172.21.15.141:0/2757973212 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c06ccf0 con 0x7fd11c06c470 2024-01-16T15:15:34.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.210+0000 7fd119400640 1 --2- 172.21.15.141:0/2757973212 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c0cc5e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd104009870 tx=0x7fd10402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f337e330676d4923 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd118a00640 1 -- 172.21.15.141:0/2757973212 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1040317c0 con 0x7fd11c06c470 2024-01-16T15:15:34.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd118a00640 1 -- 172.21.15.141:0/2757973212 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd104031920 con 0x7fd11c06c470 2024-01-16T15:15:34.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd118a00640 1 -- 172.21.15.141:0/2757973212 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104031cb0 con 0x7fd11c06c470 2024-01-16T15:15:34.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 -- 172.21.15.141:0/2757973212 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 msgr2=0x7fd11c0cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 --2- 172.21.15.141:0/2757973212 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c0cc5e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd104009870 tx=0x7fd10402b860 comp rx=0 tx=0).stop 2024-01-16T15:15:34.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 -- 172.21.15.141:0/2757973212 shutdown_connections 2024-01-16T15:15:34.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 --2- 172.21.15.141:0/2757973212 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c0cc5e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 --2- 172.21.15.141:0/2757973212 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 -- 172.21.15.141:0/2757973212 >> 172.21.15.141:0/2757973212 conn(0x7fd11c06a3e0 msgr2=0x7fd11c06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 -- 172.21.15.141:0/2757973212 shutdown_connections 2024-01-16T15:15:34.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.211+0000 7fd11b200640 1 -- 172.21.15.141:0/2757973212 wait complete. 2024-01-16T15:15:34.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.213+0000 7fd11b200640 1 Processor -- start 2024-01-16T15:15:34.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.213+0000 7fd11b200640 1 -- start start 2024-01-16T15:15:34.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.214+0000 7fd11b200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.214+0000 7fd11b200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ad810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.214+0000 7fd11b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd11c1447f0 0x7fd11c0add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.214+0000 7fd11b200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c150670 con 0x7fd11c06c470 2024-01-16T15:15:34.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.214+0000 7fd11b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c0ae290 con 0x7fd11c1447f0 2024-01-16T15:15:34.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.214+0000 7fd11b200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c159fe0 con 0x7fd11c06dab0 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57606/0 (socket says 172.21.15.141:57606) 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 -- 172.21.15.141:0/1986204944 learned_addr learned my addr 172.21.15.141:0/1986204944 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd11a800640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd11c1447f0 0x7fd11c0add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119400640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 -- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 msgr2=0x7fd11c0ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 -- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd11c1447f0 msgr2=0x7fd11c0add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd11c1447f0 0x7fd11c0add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 -- 172.21.15.141:0/1986204944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1040094d0 con 0x7fd11c06c470 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.215+0000 7fd119e00640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c150130 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00c460 tx=0x7fd10c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.216+0000 7fd112000640 1 -- 172.21.15.141:0/1986204944 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c0052d0 con 0x7fd11c06c470 2024-01-16T15:15:34.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.216+0000 7fd112000640 1 -- 172.21.15.141:0/1986204944 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10c005430 con 0x7fd11c06c470 2024-01-16T15:15:34.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.216+0000 7fd112000640 1 -- 172.21.15.141:0/1986204944 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c003830 con 0x7fd11c06c470 2024-01-16T15:15:34.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.216+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c139170 con 0x7fd11c06c470 2024-01-16T15:15:34.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.216+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c139730 con 0x7fd11c06c470 2024-01-16T15:15:34.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.217+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c067450 con 0x7fd11c06c470 2024-01-16T15:15:34.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.218+0000 7fd112000640 1 -- 172.21.15.141:0/1986204944 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fd10c003990 con 0x7fd11c06c470 2024-01-16T15:15:34.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.219+0000 7fd112000640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd0f406c4c0 0x7fd0f406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.219+0000 7fd112000640 1 -- 172.21.15.141:0/1986204944 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd10c088050 con 0x7fd11c06c470 2024-01-16T15:15:34.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.219+0000 7fd119400640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd0f406c4c0 0x7fd0f406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.206 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.221+0000 7fd119400640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd0f406c4c0 0x7fd0f406e980 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd104036000 tx=0x7fd10403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.211 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.223+0000 7fd112000640 1 -- 172.21.15.141:0/1986204944 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd10c0545a0 con 0x7fd11c06c470 2024-01-16T15:15:34.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.484+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd11c06a8f0 con 0x7fd11c06c470 2024-01-16T15:15:34.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.485+0000 7fd112000640 1 -- 172.21.15.141:0/1986204944 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd10c017020 con 0x7fd11c06c470 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "auth_dump": [ 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.a", 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlrqKAORAA6I5jqBtvLkeXdXCW87SoIQ==", 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.b", 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlIl+gHRAAWd1XzYBJIBUwBw9ATQQcxg==", 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.c", 2024-01-16T15:15:34.472 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZllfk7DhAAZXnkeNkrbLKSTBV2Vct5nw==", 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mds.d", 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZlyHpqLRAAIt6/qSYVNvhlPSFKBN/UDg==", 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.473 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.0", 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQALmaZlgHoNIRAAh/pyUpE2Jnq6aA75tDuURA==", 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.474 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.1", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAOmaZl9t9yORAAJEx2cWkEEsra1M+8zX2ICg==", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.2", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQASmaZlhs6cFRAAYXihjDCTi1qVHBDJQV8D9g==", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.475 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.3", 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAVmaZl7HxfHxAAqCSDY9ZfbqjRe0xVVo+u/w==", 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.4", 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAcmaZlxCtUKhAAfFB3Ljq60lKKI5qTmNUWZQ==", 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.5", 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAgmaZl379aCBAArnHaRqvCEKkTchch9i6JSA==", 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.477 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.6", 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAjmaZl5ViDIBAAPPFSqwNF9Kg0XBSx11zkHQ==", 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "osd.7", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAmmaZlapVSKxAA3hgGavNnv0MfCMXyDRfKqg==", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile osd", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.0", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg==", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow r", 2024-01-16T15:15:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.1", 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl/OVTFRAAWQIWMSrf7glGlU3AEW34uQ==", 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.2", 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl3XkBChAASeUWdYICS5liN0tfkJy0OQ==", 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.3", 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAHmaZl5F3+HxAAzN0CXYPTv1fk1Kuv0E79OA==", 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.481 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow r", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow rw", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow rwx" 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "client.admin", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAEmaZl0VgDMRAABOWZ0senIXHFvij8FQHiAg==", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "mgr": "allow *", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow *", 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.482 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.x", 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZlgSmrLhAAg8DmNmx2eqVce3jBVYK/yQ==", 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "entity": "mgr.y", 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "key": "AQAFmaZl4gb6IhAARhmzSO3UyRjU56E0NAatpA==", 2024-01-16T15:15:34.483 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": { 2024-01-16T15:15:34.484 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": "allow *", 2024-01-16T15:15:34.484 INFO:teuthology.orchestra.run.smithi141.stdout: "mon": "allow profile mgr", 2024-01-16T15:15:34.484 INFO:teuthology.orchestra.run.smithi141.stdout: "osd": "allow *" 2024-01-16T15:15:34.484 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.484 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:34.484 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2024-01-16T15:15:34.484 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd0f406c4c0 msgr2=0x7fd0f406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd0f406c4c0 0x7fd0f406e980 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd104036000 tx=0x7fd10403d040 comp rx=0 tx=0).stop 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 msgr2=0x7fd11c150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c150130 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00c460 tx=0x7fd10c00c920 comp rx=0 tx=0).stop 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 shutdown_connections 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd0f406c4c0 0x7fd0f406e980 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd11c1447f0 0x7fd11c0add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd11c06dab0 0x7fd11c0ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 --2- 172.21.15.141:0/1986204944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd11c06c470 0x7fd11c150130 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 >> 172.21.15.141:0/1986204944 conn(0x7fd11c06a3e0 msgr2=0x7fd11c1425b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 shutdown_connections 2024-01-16T15:15:34.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.488+0000 7fd11b200640 1 -- 172.21.15.141:0/1986204944 wait complete. 2024-01-16T15:15:34.491 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:34.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.717+0000 7fb892a00640 1 Processor -- start 2024-01-16T15:15:34.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.718+0000 7fb892a00640 1 -- start start 2024-01-16T15:15:34.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb892a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb88c0cf9a0 0x7fb88c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb892a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb88c15e1e0 0x7fb88c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb892a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88c0b1ee0 con 0x7fb88c143ae0 2024-01-16T15:15:34.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb892a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb88c0b2780 con 0x7fb88c0cf9a0 2024-01-16T15:15:34.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb892a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb88c15e990 con 0x7fb88c15e1e0 2024-01-16T15:15:34.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb88be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb88c143ae0 0x7fb88c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54466/0 (socket says 172.21.15.141:54466) 2024-01-16T15:15:34.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb88be00640 1 -- 172.21.15.141:0/318933604 learned_addr learned my addr 172.21.15.141:0/318933604 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:34.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.719+0000 7fb883200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb88c0cf9a0 0x7fb88c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb88b400640 1 -- 172.21.15.141:0/318933604 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873523021 0 0) 0x7fb88c0b1ee0 con 0x7fb88c143ae0 2024-01-16T15:15:34.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb890a00640 1 --2- 172.21.15.141:0/318933604 >> v2:172.21.15.188:3300/0 conn(0x7fb88c15e1e0 0x7fb88c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb88b400640 1 -- 172.21.15.141:0/318933604 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c003520 con 0x7fb88c143ae0 2024-01-16T15:15:34.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb883200640 1 -- 172.21.15.141:0/318933604 >> v2:172.21.15.188:3300/0 conn(0x7fb88c15e1e0 msgr2=0x7fb88c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb883200640 1 --2- 172.21.15.141:0/318933604 >> v2:172.21.15.188:3300/0 conn(0x7fb88c15e1e0 0x7fb88c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb883200640 1 -- 172.21.15.141:0/318933604 >> v1:172.21.15.141:6789/0 conn(0x7fb88c143ae0 legacy=0x7fb88c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb883200640 1 -- 172.21.15.141:0/318933604 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c0b2130 con 0x7fb88c0cf9a0 2024-01-16T15:15:34.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.720+0000 7fb890a00640 1 --2- 172.21.15.141:0/318933604 >> v2:172.21.15.188:3300/0 conn(0x7fb88c15e1e0 0x7fb88c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.721+0000 7fb883200640 1 --2- 172.21.15.141:0/318933604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c0cf9a0 0x7fb88c15dca0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0ae9d0 tx=0x7fb87402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13149c87100ea93f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.721+0000 7fb88b400640 1 -- 172.21.15.141:0/318933604 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874031830 con 0x7fb88c0cf9a0 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.721+0000 7fb88b400640 1 -- 172.21.15.141:0/318933604 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874031990 con 0x7fb88c0cf9a0 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.721+0000 7fb88b400640 1 -- 172.21.15.141:0/318933604 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874031d20 con 0x7fb88c0cf9a0 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 -- 172.21.15.141:0/318933604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c0cf9a0 msgr2=0x7fb88c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 --2- 172.21.15.141:0/318933604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c0cf9a0 0x7fb88c15dca0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0ae9d0 tx=0x7fb87402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 -- 172.21.15.141:0/318933604 shutdown_connections 2024-01-16T15:15:34.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 --2- 172.21.15.141:0/318933604 >> v2:172.21.15.188:3300/0 conn(0x7fb88c15e1e0 0x7fb88c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 --2- 172.21.15.141:0/318933604 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c0cf9a0 0x7fb88c15dca0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 -- 172.21.15.141:0/318933604 >> 172.21.15.141:0/318933604 conn(0x7fb88c06a800 msgr2=0x7fb88c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 -- 172.21.15.141:0/318933604 shutdown_connections 2024-01-16T15:15:34.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.722+0000 7fb892a00640 1 -- 172.21.15.141:0/318933604 wait complete. 2024-01-16T15:15:34.708 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.723+0000 7fb892a00640 1 Processor -- start 2024-01-16T15:15:34.709 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.724+0000 7fb892a00640 1 -- start start 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.724+0000 7fb892a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb88c0cf9a0 0x7fb88c0ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb892a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c143ae0 0x7fb88c1f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb892a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb88c15e1e0 0x7fb88c0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb892a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c140130 con 0x7fb88c143ae0 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb892a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c140380 con 0x7fb88c15e1e0 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb892a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c140660 con 0x7fb88c0cf9a0 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c143ae0 0x7fb88c1f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c143ae0 0x7fb88c1f28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57618/0 (socket says 172.21.15.141:57618) 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 -- 172.21.15.141:0/1785023909 learned_addr learned my addr 172.21.15.141:0/1785023909 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:34.711 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 -- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb88c0cf9a0 msgr2=0x7fb88c0ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb88c0cf9a0 0x7fb88c0ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 -- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb88c15e1e0 msgr2=0x7fb88c0ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb88c15e1e0 0x7fb88c0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.725+0000 7fb883200640 1 -- 172.21.15.141:0/1785023909 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb874009400 con 0x7fb88c143ae0 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.726+0000 7fb883200640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c143ae0 0x7fb88c1f28f0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fb874000c00 tx=0x7fb8740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.726+0000 7fb888c00640 1 -- 172.21.15.141:0/1785023909 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87403a8f0 con 0x7fb88c143ae0 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.726+0000 7fb888c00640 1 -- 172.21.15.141:0/1785023909 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87403aa70 con 0x7fb88c143ae0 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.726+0000 7fb888c00640 1 -- 172.21.15.141:0/1785023909 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87403ae20 con 0x7fb88c143ae0 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.726+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c140940 con 0x7fb88c143ae0 2024-01-16T15:15:34.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.726+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c140f00 con 0x7fb88c143ae0 2024-01-16T15:15:34.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.727+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88c0675d0 con 0x7fb88c143ae0 2024-01-16T15:15:34.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.728+0000 7fb888c00640 1 -- 172.21.15.141:0/1785023909 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fb87404d450 con 0x7fb88c143ae0 2024-01-16T15:15:34.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.729+0000 7fb888c00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86406c4c0 0x7fb86406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.729+0000 7fb888c00640 1 -- 172.21.15.141:0/1785023909 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb8740aabc0 con 0x7fb88c143ae0 2024-01-16T15:15:34.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.729+0000 7fb88be00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86406c4c0 0x7fb86406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.716 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.731+0000 7fb88be00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86406c4c0 0x7fb86406e980 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb87c001610 tx=0x7fb87c007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.733+0000 7fb888c00640 1 -- 172.21.15.141:0/1785023909 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb874077110 con 0x7fb88c143ae0 2024-01-16T15:15:34.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.927+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb88c1410c0 con 0x7fb88c143ae0 2024-01-16T15:15:34.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.928+0000 7fb888c00640 1 -- 172.21.15.141:0/1785023909 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+12915 (secure 0 0 0) 0x7fb87407a730 con 0x7fb88c143ae0 2024-01-16T15:15:34.913 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:34.914 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":112,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:33.970498+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86406c4c0 msgr2=0x7fb86406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86406c4c0 0x7fb86406e980 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb87c001610 tx=0x7fb87c007930 comp rx=0 tx=0).stop 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c143ae0 msgr2=0x7fb88c1f28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c143ae0 0x7fb88c1f28f0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fb874000c00 tx=0x7fb8740046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 shutdown_connections 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb86406c4c0 0x7fb86406e980 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb88c15e1e0 0x7fb88c0ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb88c143ae0 0x7fb88c1f28f0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 --2- 172.21.15.141:0/1785023909 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb88c0cf9a0 0x7fb88c0ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 >> 172.21.15.141:0/1785023909 conn(0x7fb88c06a800 msgr2=0x7fb88c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 shutdown_connections 2024-01-16T15:15:34.917 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:34.931+0000 7fb892a00640 1 -- 172.21.15.141:0/1785023909 wait complete. 2024-01-16T15:15:34.935 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:34.935 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:35.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.164+0000 7fcce0e00640 1 Processor -- start 2024-01-16T15:15:35.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.164+0000 7fcce0e00640 1 -- start start 2024-01-16T15:15:35.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.166+0000 7fcce0e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 0x7fccdc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.166+0000 7fcce0e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fccdc0ccb50 0x7fccdc143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.166+0000 7fcce0e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc15f4e0 con 0x7fccdc15bf10 2024-01-16T15:15:35.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.166+0000 7fcce0e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccdc15f970 con 0x7fccdc0ccb50 2024-01-16T15:15:35.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.166+0000 7fcce0e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc071130 con 0x7fccdc0adf20 2024-01-16T15:15:35.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.166+0000 7fccd3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fccdc15bf10 0x7fccdc0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54478/0 (socket says 172.21.15.141:54478) 2024-01-16T15:15:35.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.166+0000 7fccd3e00640 1 -- 172.21.15.141:0/3369204119 learned_addr learned my addr 172.21.15.141:0/3369204119 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdbe00640 1 --2- 172.21.15.141:0/3369204119 >> v2:172.21.15.188:3300/0 conn(0x7fccdc0ccb50 0x7fccdc143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdb400640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 0x7fccdc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739583622 0 0) 0x7fccdc15f4e0 con 0x7fccdc15bf10 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccbc003520 con 0x7fccdc15bf10 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 5448032 0 0) 0x7fccbc003520 con 0x7fccdc15bf10 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 msgr2=0x7fccdc070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 0x7fccdc070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 >> v2:172.21.15.188:3300/0 conn(0x7fccdc0ccb50 msgr2=0x7fccdc143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 --2- 172.21.15.141:0/3369204119 >> v2:172.21.15.188:3300/0 conn(0x7fccdc0ccb50 0x7fccdc143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc15c390 con 0x7fccdc15bf10 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fccc4001670 con 0x7fccdc15bf10 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdbe00640 1 --2- 172.21.15.141:0/3369204119 >> v2:172.21.15.188:3300/0 conn(0x7fccdc0ccb50 0x7fccdc143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:35.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 >> v1:172.21.15.141:6789/0 conn(0x7fccdc15bf10 legacy=0x7fccdc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccbc003970 0x7fccbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 0x7fccbc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc15f970 con 0x7fccbc0047c0 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc071130 con 0x7fccbc003970 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.167+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc00b9d0 con 0x7fccdc0adf20 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccd3e00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccdb400640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccbc003970 0x7fccbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccdbe00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 0x7fccbc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccd3e00640 1 -- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 msgr2=0x7fccbc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccd3e00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0adf20 0x7fccbc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccd3e00640 1 -- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccbc003970 msgr2=0x7fccbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccd3e00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccbc003970 0x7fccbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.168+0000 7fccd3e00640 1 -- 172.21.15.141:0/3369204119 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc003520 con 0x7fccbc0047c0 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.169+0000 7fccd3e00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccbc004c40 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c1c0 tx=0x7fccbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757612665f8d85dc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4005ad0 con 0x7fccbc0047c0 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc4005c50 con 0x7fccbc0047c0 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fccdaa00640 1 -- 172.21.15.141:0/3369204119 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc402c810 con 0x7fccbc0047c0 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fcce0e00640 1 -- 172.21.15.141:0/3369204119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 msgr2=0x7fccbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fcce0e00640 1 --2- 172.21.15.141:0/3369204119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccbc004c40 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c1c0 tx=0x7fccbc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fccdbe00640 1 -- 172.21.15.141:0/3369204119 reap_dead start 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fcce0e00640 1 -- 172.21.15.141:0/3369204119 shutdown_connections 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.170+0000 7fcce0e00640 1 -- 172.21.15.141:0/3369204119 >> 172.21.15.141:0/3369204119 conn(0x7fccdc06a7d0 msgr2=0x7fccdc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.171+0000 7fcce0e00640 1 -- 172.21.15.141:0/3369204119 shutdown_connections 2024-01-16T15:15:35.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.171+0000 7fcce0e00640 1 -- 172.21.15.141:0/3369204119 wait complete. 2024-01-16T15:15:35.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.172+0000 7fcce0e00640 1 Processor -- start 2024-01-16T15:15:35.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.172+0000 7fcce0e00640 1 -- start start 2024-01-16T15:15:35.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.173+0000 7fcce0e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccdc15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.173+0000 7fcce0e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0cb3b0 0x7fccdc0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.173+0000 7fcce0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccdc0b3c90 0x7fccdc15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.173+0000 7fcce0e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc15af50 con 0x7fccbc0047c0 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.173+0000 7fcce0e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0b26b0 con 0x7fccdc0b3c90 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.173+0000 7fcce0e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0b2990 con 0x7fccdc0cb3b0 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccdc15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccdc15aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57632/0 (socket says 172.21.15.141:57632) 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 -- 172.21.15.141:0/3351637902 learned_addr learned my addr 172.21.15.141:0/3351637902 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdbe00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccdc0b3c90 0x7fccdc15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccd3e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0cb3b0 0x7fccdc0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 -- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0cb3b0 msgr2=0x7fccdc0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0cb3b0 0x7fccdc0b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 -- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccdc0b3c90 msgr2=0x7fccdc15c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccdc0b3c90 0x7fccdc15c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 -- 172.21.15.141:0/3351637902 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc0b2c70 con 0x7fccbc0047c0 2024-01-16T15:15:35.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.174+0000 7fccdb400640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccdc15aa10 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0118e0 tx=0x7fcccc011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:35.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.175+0000 7fccd3400640 1 -- 172.21.15.141:0/3351637902 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc00faf0 con 0x7fccbc0047c0 2024-01-16T15:15:35.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.175+0000 7fccd3400640 1 -- 172.21.15.141:0/3351637902 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc00fc50 con 0x7fccbc0047c0 2024-01-16T15:15:35.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.175+0000 7fccd3400640 1 -- 172.21.15.141:0/3351637902 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc014440 con 0x7fccbc0047c0 2024-01-16T15:15:35.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.175+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc0b2f50 con 0x7fccbc0047c0 2024-01-16T15:15:35.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.175+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc0b2360 con 0x7fccbc0047c0 2024-01-16T15:15:35.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.176+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc067450 con 0x7fccbc0047c0 2024-01-16T15:15:35.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.177+0000 7fccd3400640 1 -- 172.21.15.141:0/3351637902 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fcccc0145a0 con 0x7fccbc0047c0 2024-01-16T15:15:35.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.178+0000 7fccd3400640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fccb406c4c0 0x7fccb406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.178+0000 7fccd3400640 1 -- 172.21.15.141:0/3351637902 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcccc0893e0 con 0x7fccbc0047c0 2024-01-16T15:15:35.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.179+0000 7fccd3e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fccb406c4c0 0x7fccb406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.165 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.180+0000 7fccd3e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fccb406c4c0 0x7fccb406e980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fccc4034000 tx=0x7fccc4007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:35.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.183+0000 7fccd3400640 1 -- 172.21.15.141:0/3351637902 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcccc055930 con 0x7fccbc0047c0 2024-01-16T15:15:35.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.373+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fccdc1449e0 con 0x7fccbc0047c0 2024-01-16T15:15:35.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.374+0000 7fccd3400640 1 -- 172.21.15.141:0/3351637902 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcccc0220a0 con 0x7fccbc0047c0 2024-01-16T15:15:35.359 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:15:35.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fccb406c4c0 msgr2=0x7fccb406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fccb406c4c0 0x7fccb406e980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fccc4034000 tx=0x7fccc4007000 comp rx=0 tx=0).stop 2024-01-16T15:15:35.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 msgr2=0x7fccdc15aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccdc15aa10 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0118e0 tx=0x7fcccc011da0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 shutdown_connections 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fccb406c4c0 0x7fccb406e980 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccdc0b3c90 0x7fccdc15c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fccdc0cb3b0 0x7fccdc0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 --2- 172.21.15.141:0/3351637902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fccbc0047c0 0x7fccdc15aa10 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.376+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 >> 172.21.15.141:0/3351637902 conn(0x7fccdc06a7d0 msgr2=0x7fccdc0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.377+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 shutdown_connections 2024-01-16T15:15:35.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.377+0000 7fcce0e00640 1 -- 172.21.15.141:0/3351637902 wait complete. 2024-01-16T15:15:35.381 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:15:35.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.606+0000 7f7881400640 1 Processor -- start 2024-01-16T15:15:35.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.607+0000 7f7881400640 1 -- start start 2024-01-16T15:15:35.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.608+0000 7f7881400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f787c072230 0x7f787c0b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.608+0000 7f7881400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c071190 con 0x7f787c15da10 2024-01-16T15:15:35.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.608+0000 7f7881400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c071470 con 0x7f787c150760 2024-01-16T15:15:35.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.608+0000 7f7881400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787c071750 con 0x7f787c072230 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.609+0000 7f7879e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f787c15da10 0x7f787c0b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54486/0 (socket says 172.21.15.141:54486) 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.609+0000 7f7879e00640 1 -- 172.21.15.141:0/2607812524 learned_addr learned my addr 172.21.15.141:0/2607812524 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.609+0000 7f787b200640 1 --2- 172.21.15.141:0/2607812524 >> v2:172.21.15.188:3300/0 conn(0x7f787c072230 0x7f787c0b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.609+0000 7f7879400640 1 -- 172.21.15.141:0/2607812524 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366500842 0 0) 0x7f787c071190 con 0x7f787c15da10 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.609+0000 7f7879400640 1 -- 172.21.15.141:0/2607812524 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785c003520 con 0x7f787c15da10 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.610+0000 7f7879400640 1 -- 172.21.15.141:0/2607812524 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3639292223 0 0) 0x7f785c003520 con 0x7f787c15da10 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.610+0000 7f787b200640 1 -- 172.21.15.141:0/2607812524 >> v1:172.21.15.188:6789/0 conn(0x7f787c150760 legacy=0x7f787c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.610+0000 7f787b200640 1 -- 172.21.15.141:0/2607812524 >> v1:172.21.15.141:6789/0 conn(0x7f787c15da10 legacy=0x7f787c0b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.610+0000 7f787b200640 1 -- 172.21.15.141:0/2607812524 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c071a30 con 0x7f787c072230 2024-01-16T15:15:35.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.611+0000 7f787b200640 1 --2- 172.21.15.141:0/2607812524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c072230 0x7f787c0b32e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f787400aea0 tx=0x7f787402cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3e3b98ee7ca43fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:35.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.612+0000 7f7879400640 1 -- 172.21.15.141:0/2607812524 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864001670 con 0x7f787c072230 2024-01-16T15:15:35.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.612+0000 7f7879400640 1 -- 172.21.15.141:0/2607812524 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787400a2e0 con 0x7f787c072230 2024-01-16T15:15:35.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.612+0000 7f7879400640 1 -- 172.21.15.141:0/2607812524 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787402da50 con 0x7f787c072230 2024-01-16T15:15:35.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.612+0000 7f7881400640 1 -- 172.21.15.141:0/2607812524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c072230 msgr2=0x7f787c0b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.612+0000 7f7881400640 1 --2- 172.21.15.141:0/2607812524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c072230 0x7f787c0b32e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f787400aea0 tx=0x7f787402cf10 comp rx=0 tx=0).stop 2024-01-16T15:15:35.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.613+0000 7f7881400640 1 -- 172.21.15.141:0/2607812524 shutdown_connections 2024-01-16T15:15:35.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.613+0000 7f7881400640 1 --2- 172.21.15.141:0/2607812524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c072230 0x7f787c0b32e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.613+0000 7f7881400640 1 -- 172.21.15.141:0/2607812524 >> 172.21.15.141:0/2607812524 conn(0x7f787c06a800 msgr2=0x7f787c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:35.598 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.613+0000 7f7881400640 1 -- 172.21.15.141:0/2607812524 shutdown_connections 2024-01-16T15:15:35.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.613+0000 7f7881400640 1 -- 172.21.15.141:0/2607812524 wait complete. 2024-01-16T15:15:35.599 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.614+0000 7f7881400640 1 Processor -- start 2024-01-16T15:15:35.600 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.615+0000 7f7881400640 1 -- start start 2024-01-16T15:15:35.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.615+0000 7f7881400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f787c072230 0x7f787c0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.615+0000 7f7881400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c150760 0x7f787c1588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.615+0000 7f7881400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f787c15da10 0x7f787c158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.615+0000 7f7881400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c0cd950 con 0x7f787c072230 2024-01-16T15:15:35.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.615+0000 7f7881400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c0cd060 con 0x7f787c150760 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.615+0000 7f7881400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787c0af420 con 0x7f787c15da10 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f787c072230 0x7f787c0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f787c072230 0x7f787c0cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57640/0 (socket says 172.21.15.141:57640) 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 -- 172.21.15.141:0/2944126174 learned_addr learned my addr 172.21.15.141:0/2944126174 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787b200640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f787c15da10 0x7f787c158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f7879e00640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c150760 0x7f787c1588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 -- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f787c15da10 msgr2=0x7f787c158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f787c15da10 0x7f787c158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 -- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c150760 msgr2=0x7f787c1588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c150760 0x7f787c1588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f787a800640 1 -- 172.21.15.141:0/2944126174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787400ab80 con 0x7f787c072230 2024-01-16T15:15:35.603 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.616+0000 7f7879e00640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c150760 0x7f787c1588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:35.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.617+0000 7f787a800640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f787c072230 0x7f787c0cdde0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f786c00c450 tx=0x7f786c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:35.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.617+0000 7f7872a00640 1 -- 172.21.15.141:0/2944126174 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c0052d0 con 0x7f787c072230 2024-01-16T15:15:35.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.617+0000 7f7872a00640 1 -- 172.21.15.141:0/2944126174 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c005430 con 0x7f787c072230 2024-01-16T15:15:35.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.617+0000 7f7872a00640 1 -- 172.21.15.141:0/2944126174 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c012870 con 0x7f787c072230 2024-01-16T15:15:35.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.617+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787c0cd300 con 0x7f787c072230 2024-01-16T15:15:35.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.617+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c0ce320 con 0x7f787c072230 2024-01-16T15:15:35.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.618+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787c0ce4e0 con 0x7f787c072230 2024-01-16T15:15:35.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.619+0000 7f7872a00640 1 -- 172.21.15.141:0/2944126174 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f786c0129d0 con 0x7f787c072230 2024-01-16T15:15:35.605 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.620+0000 7f7872a00640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f785406c4c0 0x7f785406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:35.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.621+0000 7f7872a00640 1 -- 172.21.15.141:0/2944126174 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f786c088620 con 0x7f787c072230 2024-01-16T15:15:35.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.621+0000 7f7879e00640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f785406c4c0 0x7f785406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:35.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.622+0000 7f7879e00640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f785406c4c0 0x7f785406e980 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f787c14d6a0 tx=0x7f7864009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:35.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.625+0000 7f7872a00640 1 -- 172.21.15.141:0/2944126174 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f786c054bd0 con 0x7f787c072230 2024-01-16T15:15:35.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.816+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f787c0675d0 con 0x7f787c072230 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.817+0000 7f7872a00640 1 -- 172.21.15.141:0/2944126174 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v112) v1 ==== 84+0+65 (secure 0 0 0) 0x7f786c0581f0 con 0x7f787c072230 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stdout: "poolnum": 1, 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stdout: "poolname": ".mgr" 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:35.803 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:15:35.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f785406c4c0 msgr2=0x7f785406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f785406c4c0 0x7f785406e980 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f787c14d6a0 tx=0x7f7864009040 comp rx=0 tx=0).stop 2024-01-16T15:15:35.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f787c072230 msgr2=0x7f787c0cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:35.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f787c072230 0x7f787c0cdde0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f786c00c450 tx=0x7f786c00c910 comp rx=0 tx=0).stop 2024-01-16T15:15:35.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 shutdown_connections 2024-01-16T15:15:35.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f785406c4c0 0x7f785406e980 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f787c15da10 0x7f787c158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f787c150760 0x7f787c1588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 --2- 172.21.15.141:0/2944126174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f787c072230 0x7f787c0cdde0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:35.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.820+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 >> 172.21.15.141:0/2944126174 conn(0x7f787c06a800 msgr2=0x7f787c0b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:35.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.821+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 shutdown_connections 2024-01-16T15:15:35.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:35.821+0000 7f7881400640 1 -- 172.21.15.141:0/2944126174 wait complete. 2024-01-16T15:15:35.825 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:15:35.825 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:15:36.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.046+0000 7fdbe0a00640 1 Processor -- start 2024-01-16T15:15:36.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.047+0000 7fdbe0a00640 1 -- start start 2024-01-16T15:15:36.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.048+0000 7fdbe0a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdbdc0d0100 0x7fdbdc0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.048+0000 7fdbe0a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0f40 0x7fdbdc0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.048+0000 7fdbe0a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc0d0a10 con 0x7fdbdc160920 2024-01-16T15:15:36.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.048+0000 7fdbe0a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc0cf080 con 0x7fdbdc0d0100 2024-01-16T15:15:36.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.048+0000 7fdbe0a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0c6be0 con 0x7fdbdc0d0f40 2024-01-16T15:15:36.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.049+0000 7fdbdbe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0f40 0x7fdbdc0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.049+0000 7fdbdaa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fdbdc0d0100 0x7fdbdc0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.049+0000 7fdbdb400640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fdbdc160920 0x7fdbdc0cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:53324/0 (socket says 172.21.15.141:53324) 2024-01-16T15:15:36.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.049+0000 7fdbdb400640 1 -- 172.21.15.141:0/1824177686 learned_addr learned my addr 172.21.15.141:0/1824177686 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:36.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.050+0000 7fdbdbe00640 1 -- 172.21.15.141:0/1824177686 >> v2:172.21.15.188:3300/0 conn(0x7fdbdc0d0100 msgr2=0x7fdbdc0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:36.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.050+0000 7fdbdbe00640 1 --2- 172.21.15.141:0/1824177686 >> v2:172.21.15.188:3300/0 conn(0x7fdbdc0d0100 0x7fdbdc0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.050+0000 7fdbdbe00640 1 -- 172.21.15.141:0/1824177686 >> v1:172.21.15.188:6789/0 conn(0x7fdbdc160920 legacy=0x7fdbdc0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.050+0000 7fdbdbe00640 1 -- 172.21.15.141:0/1824177686 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc0ce7e0 con 0x7fdbdc0d0f40 2024-01-16T15:15:36.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.050+0000 7fdbdaa00640 1 --2- 172.21.15.141:0/1824177686 >> v2:172.21.15.188:3300/0 conn(0x7fdbdc0d0100 0x7fdbdc0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:36.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.051+0000 7fdbdbe00640 1 --2- 172.21.15.141:0/1824177686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0f40 0x7fdbdc0d13d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40097a0 tx=0x7fdbd402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b838b8b546ed0f1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:36.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbda000640 1 -- 172.21.15.141:0/1824177686 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd402d040 con 0x7fdbdc0d0f40 2024-01-16T15:15:36.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbda000640 1 -- 172.21.15.141:0/1824177686 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd4031810 con 0x7fdbdc0d0f40 2024-01-16T15:15:36.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbda000640 1 -- 172.21.15.141:0/1824177686 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4031ba0 con 0x7fdbdc0d0f40 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1824177686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0f40 msgr2=0x7fdbdc0d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1824177686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0f40 0x7fdbdc0d13d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40097a0 tx=0x7fdbd402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1824177686 shutdown_connections 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1824177686 >> v2:172.21.15.188:3300/0 conn(0x7fdbdc0d0100 0x7fdbdc0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1824177686 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0f40 0x7fdbdc0d13d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1824177686 >> 172.21.15.141:0/1824177686 conn(0x7fdbdc0d3090 msgr2=0x7fdbdc0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1824177686 shutdown_connections 2024-01-16T15:15:36.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.052+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1824177686 wait complete. 2024-01-16T15:15:36.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.054+0000 7fdbe0a00640 1 Processor -- start 2024-01-16T15:15:36.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.054+0000 7fdbe0a00640 1 -- start start 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.055+0000 7fdbe0a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0100 0x7fdbdc0c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.055+0000 7fdbe0a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbdc0d0f40 0x7fdbdc0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.055+0000 7fdbe0a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbdc160920 0x7fdbdc14a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.055+0000 7fdbe0a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0ca8c0 con 0x7fdbdc160920 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.055+0000 7fdbe0a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc1f4030 con 0x7fdbdc0d0f40 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.055+0000 7fdbe0a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc1f3d50 con 0x7fdbdc0d0100 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.055+0000 7fdbdbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbdc160920 0x7fdbdc14a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbdc160920 0x7fdbdc14a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57646/0 (socket says 172.21.15.141:57646) 2024-01-16T15:15:36.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 -- 172.21.15.141:0/1941763718 learned_addr learned my addr 172.21.15.141:0/1941763718 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:36.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdb400640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0100 0x7fdbdc0c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdaa00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbdc0d0f40 0x7fdbdc0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 -- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0100 msgr2=0x7fdbdc0c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0100 0x7fdbdc0c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 -- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbdc0d0f40 msgr2=0x7fdbdc0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbdc0d0f40 0x7fdbdc0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 -- 172.21.15.141:0/1941763718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc4009a70 con 0x7fdbdc160920 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.056+0000 7fdbdbe00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbdc160920 0x7fdbdc14a000 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40098c0 tx=0x7fdbd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.057+0000 7fdbd3400640 1 -- 172.21.15.141:0/1941763718 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd403ace0 con 0x7fdbdc160920 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.057+0000 7fdbd3400640 1 -- 172.21.15.141:0/1941763718 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd403ae60 con 0x7fdbdc160920 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.057+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd4009400 con 0x7fdbdc160920 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.057+0000 7fdbd3400640 1 -- 172.21.15.141:0/1941763718 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4042630 con 0x7fdbdc160920 2024-01-16T15:15:36.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.057+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc0ca550 con 0x7fdbdc160920 2024-01-16T15:15:36.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.058+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc0caa70 con 0x7fdbdc160920 2024-01-16T15:15:36.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.059+0000 7fdbd3400640 1 -- 172.21.15.141:0/1941763718 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fdbd404b3b0 con 0x7fdbdc160920 2024-01-16T15:15:36.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.060+0000 7fdbd3400640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbb406c4c0 0x7fdbb406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.060+0000 7fdbd3400640 1 -- 172.21.15.141:0/1941763718 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdbd40aaae0 con 0x7fdbdc160920 2024-01-16T15:15:36.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.061+0000 7fdbdb400640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbb406c4c0 0x7fdbb406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.062+0000 7fdbdb400640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbb406c4c0 0x7fdbb406e980 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc005880 tx=0x7fdbcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:36.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.065+0000 7fdbd3400640 1 -- 172.21.15.141:0/1941763718 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbdc0caa70 con 0x7fdbdc160920 2024-01-16T15:15:36.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:36.258+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdbdc0caa70 con 0x7fdbdc160920 2024-01-16T15:15:37.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.253+0000 7fdbd3400640 1 -- 172.21.15.141:0/1941763718 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v113) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdbdc0caa70 con 0x7fdbdc160920 2024-01-16T15:15:37.239 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_metadata' created 2024-01-16T15:15:37.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbb406c4c0 msgr2=0x7fdbb406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbb406c4c0 0x7fdbb406e980 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc005880 tx=0x7fdbcc009040 comp rx=0 tx=0).stop 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbdc160920 msgr2=0x7fdbdc14a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbdc160920 0x7fdbdc14a000 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40098c0 tx=0x7fdbd40046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 shutdown_connections 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdbb406c4c0 0x7fdbb406e980 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdbdc160920 0x7fdbdc14a000 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbdc0d0f40 0x7fdbdc0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 --2- 172.21.15.141:0/1941763718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdbdc0d0100 0x7fdbdc0c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 >> 172.21.15.141:0/1941763718 conn(0x7fdbdc0d3090 msgr2=0x7fdbdc0d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 shutdown_connections 2024-01-16T15:15:37.244 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.258+0000 7fdbe0a00640 1 -- 172.21.15.141:0/1941763718 wait complete. 2024-01-16T15:15:37.275 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:15:37.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.498+0000 7eff93800640 1 Processor -- start 2024-01-16T15:15:37.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.498+0000 7eff93800640 1 -- start start 2024-01-16T15:15:37.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff93800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c15ed20 0x7eff8c141360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:37.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff93800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7eff8c15f100 0x7eff8c0cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:37.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff93800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7eff8c0cdcc0 0x7eff8c15b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:37.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff93800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff8c13e440 con 0x7eff8c15f100 2024-01-16T15:15:37.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff93800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff8c0ce1d0 con 0x7eff8c0cdcc0 2024-01-16T15:15:37.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff93800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c0ce4b0 con 0x7eff8c15ed20 2024-01-16T15:15:37.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7eff8c15f100 0x7eff8c0cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:37.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7eff8c15f100 0x7eff8c0cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57652/0 (socket says 172.21.15.141:57652) 2024-01-16T15:15:37.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 -- 172.21.15.141:0/2044482110 learned_addr learned my addr 172.21.15.141:0/2044482110 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:37.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff91800640 1 --2- 172.21.15.141:0/2044482110 >> v2:172.21.15.188:3300/0 conn(0x7eff8c0cdcc0 0x7eff8c15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:37.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff90e00640 1 --2- 172.21.15.141:0/2044482110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c15ed20 0x7eff8c141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:37.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 -- 172.21.15.141:0/2044482110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c15ed20 msgr2=0x7eff8c141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:37.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 --2- 172.21.15.141:0/2044482110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c15ed20 0x7eff8c141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 -- 172.21.15.141:0/2044482110 >> v2:172.21.15.188:3300/0 conn(0x7eff8c0cdcc0 msgr2=0x7eff8c15b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:37.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 --2- 172.21.15.141:0/2044482110 >> v2:172.21.15.188:3300/0 conn(0x7eff8c0cdcc0 0x7eff8c15b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff83c00640 1 -- 172.21.15.141:0/2044482110 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8c0ce790 con 0x7eff8c15f100 2024-01-16T15:15:37.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff90e00640 1 --2- 172.21.15.141:0/2044482110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c15ed20 0x7eff8c141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:37.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.500+0000 7eff91800640 1 --2- 172.21.15.141:0/2044482110 >> v2:172.21.15.188:3300/0 conn(0x7eff8c0cdcc0 0x7eff8c15b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:37.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.501+0000 7eff83c00640 1 --2- 172.21.15.141:0/2044482110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0cd780 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7eff8c073dd0 tx=0x7eff7402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccdbda25b83ba24 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:37.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.502+0000 7eff8be00640 1 -- 172.21.15.141:0/2044482110 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff740317d0 con 0x7eff8c15f100 2024-01-16T15:15:37.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.502+0000 7eff8be00640 1 -- 172.21.15.141:0/2044482110 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff74031930 con 0x7eff8c15f100 2024-01-16T15:15:37.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.502+0000 7eff8be00640 1 -- 172.21.15.141:0/2044482110 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74031cc0 con 0x7eff8c15f100 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.502+0000 7eff93800640 1 -- 172.21.15.141:0/2044482110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 msgr2=0x7eff8c0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.502+0000 7eff93800640 1 --2- 172.21.15.141:0/2044482110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0cd780 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7eff8c073dd0 tx=0x7eff7402b870 comp rx=0 tx=0).stop 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.503+0000 7eff93800640 1 -- 172.21.15.141:0/2044482110 shutdown_connections 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.503+0000 7eff93800640 1 --2- 172.21.15.141:0/2044482110 >> v2:172.21.15.188:3300/0 conn(0x7eff8c0cdcc0 0x7eff8c15b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.503+0000 7eff93800640 1 --2- 172.21.15.141:0/2044482110 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0cd780 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.503+0000 7eff93800640 1 --2- 172.21.15.141:0/2044482110 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c15ed20 0x7eff8c141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.503+0000 7eff93800640 1 -- 172.21.15.141:0/2044482110 >> 172.21.15.141:0/2044482110 conn(0x7eff8c0b0be0 msgr2=0x7eff8c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.503+0000 7eff93800640 1 -- 172.21.15.141:0/2044482110 shutdown_connections 2024-01-16T15:15:37.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.503+0000 7eff93800640 1 -- 172.21.15.141:0/2044482110 wait complete. 2024-01-16T15:15:37.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.505+0000 7eff93800640 1 Processor -- start 2024-01-16T15:15:37.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.505+0000 7eff93800640 1 -- start start 2024-01-16T15:15:37.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff93800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c0cdcc0 0x7eff8c150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:37.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff93800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff8c15ed20 0x7eff8c0b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:37.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff93800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:37.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff93800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c0b40b0 con 0x7eff8c15f100 2024-01-16T15:15:37.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff93800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c0b1ea0 con 0x7eff8c15ed20 2024-01-16T15:15:37.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff93800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c0b20b0 con 0x7eff8c0cdcc0 2024-01-16T15:15:37.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:37.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0b1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57654/0 (socket says 172.21.15.141:57654) 2024-01-16T15:15:37.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 -- 172.21.15.141:0/4064065865 learned_addr learned my addr 172.21.15.141:0/4064065865 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:37.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff83c00640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff8c15ed20 0x7eff8c0b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:37.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff90e00640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c0cdcc0 0x7eff8c150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:37.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 -- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c0cdcc0 msgr2=0x7eff8c150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:37.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c0cdcc0 0x7eff8c150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 -- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff8c15ed20 msgr2=0x7eff8c0b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:37.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff8c15ed20 0x7eff8c0b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff91800640 1 -- 172.21.15.141:0/4064065865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff740094d0 con 0x7eff8c15f100 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff83c00640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff8c15ed20 0x7eff8c0b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.506+0000 7eff90e00640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c0cdcc0 0x7eff8c150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.507+0000 7eff91800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0b1960 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7eff8400cef0 tx=0x7eff8400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.507+0000 7eff89600640 1 -- 172.21.15.141:0/4064065865 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400d840 con 0x7eff8c15f100 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.507+0000 7eff89600640 1 -- 172.21.15.141:0/4064065865 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8400d9a0 con 0x7eff8c15f100 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.507+0000 7eff89600640 1 -- 172.21.15.141:0/4064065865 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400dd50 con 0x7eff8c15f100 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.507+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8c15f6c0 con 0x7eff8c15f100 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.507+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8c0b2670 con 0x7eff8c15f100 2024-01-16T15:15:37.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.508+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8c067670 con 0x7eff8c15f100 2024-01-16T15:15:37.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.510+0000 7eff89600640 1 -- 172.21.15.141:0/4064065865 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7eff8401e3b0 con 0x7eff8c15f100 2024-01-16T15:15:37.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.511+0000 7eff89600640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff6406c4c0 0x7eff6406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:37.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.511+0000 7eff89600640 1 -- 172.21.15.141:0/4064065865 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5227+0+0 (secure 0 0 0) 0x7eff840896a0 con 0x7eff8c15f100 2024-01-16T15:15:37.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.512+0000 7eff90e00640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff6406c4c0 0x7eff6406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:37.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.513+0000 7eff90e00640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff6406c4c0 0x7eff6406e980 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7eff7c006fd0 tx=0x7eff7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:37.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.515+0000 7eff89600640 1 -- 172.21.15.141:0/4064065865 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff84055fa0 con 0x7eff8c15f100 2024-01-16T15:15:37.511 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:37.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:37.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:37.708+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7eff8c072550 con 0x7eff8c15f100 2024-01-16T15:15:38.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.260+0000 7eff89600640 1 -- 172.21.15.141:0/4064065865 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v114) v1 ==== 170+0+0 (secure 0 0 0) 0x7eff8401a070 con 0x7eff8c15f100 2024-01-16T15:15:38.245 INFO:teuthology.orchestra.run.smithi141.stderr:pool 'cephfs_data' created 2024-01-16T15:15:38.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff6406c4c0 msgr2=0x7eff6406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:38.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff6406c4c0 0x7eff6406e980 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7eff7c006fd0 tx=0x7eff7c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:38.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 msgr2=0x7eff8c0b1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:38.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0b1960 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7eff8400cef0 tx=0x7eff8400a710 comp rx=0 tx=0).stop 2024-01-16T15:15:38.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 shutdown_connections 2024-01-16T15:15:38.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7eff6406c4c0 0x7eff6406e980 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7eff8c15f100 0x7eff8c0b1960 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff8c15ed20 0x7eff8c0b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 --2- 172.21.15.141:0/4064065865 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7eff8c0cdcc0 0x7eff8c150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.264+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 >> 172.21.15.141:0/4064065865 conn(0x7eff8c0b0be0 msgr2=0x7eff8c151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:38.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.265+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 shutdown_connections 2024-01-16T15:15:38.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.266+0000 7eff93800640 1 -- 172.21.15.141:0/4064065865 wait complete. 2024-01-16T15:15:38.286 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:15:38.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.484+0000 7f89a2600640 1 Processor -- start 2024-01-16T15:15:38.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.484+0000 7f89a2600640 1 -- start start 2024-01-16T15:15:38.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.485+0000 7f89a2600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f899c0ae5a0 0x7f899c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f89a2600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899c070c40 con 0x7f899c0cf290 2024-01-16T15:15:38.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f89a2600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899c06c470 con 0x7f899c0b3ae0 2024-01-16T15:15:38.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f89a2600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f899c071450 con 0x7f899c0ae5a0 2024-01-16T15:15:38.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f899be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f899c0cf290 0x7f899c0b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54492/0 (socket says 172.21.15.141:54492) 2024-01-16T15:15:38.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f899be00640 1 -- 172.21.15.141:0/2380137055 learned_addr learned my addr 172.21.15.141:0/2380137055 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057392023 0 0) 0x7f899c070c40 con 0x7f899c0cf290 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f897c003520 con 0x7f899c0cf290 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.486+0000 7f899b400640 1 --2- 172.21.15.141:0/2380137055 >> v2:172.21.15.188:3300/0 conn(0x7f899c0ae5a0 0x7f899c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2858989430 0 0) 0x7f897c003520 con 0x7f899c0cf290 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 >> v2:172.21.15.188:3300/0 conn(0x7f899c0ae5a0 msgr2=0x7f899c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 --2- 172.21.15.141:0/2380137055 >> v2:172.21.15.188:3300/0 conn(0x7f899c0ae5a0 0x7f899c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 >> v1:172.21.15.188:6789/0 conn(0x7f899c0b3ae0 legacy=0x7f899c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c06c750 con 0x7f899c0cf290 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f89940019f0 con 0x7f899c0cf290 2024-01-16T15:15:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 >> v1:172.21.15.141:6789/0 conn(0x7f899c0cf290 legacy=0x7f899c0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f897c003970 0x7f897c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c0047c0 0x7f897c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f897c005180 0x7f897c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c071450 con 0x7f897c005180 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c00c0b0 con 0x7f897c0047c0 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897c00c270 con 0x7f897c003970 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.487+0000 7f899be00640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f897c005180 0x7f897c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.488+0000 7f8992800640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c0047c0 0x7f897c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.488+0000 7f899b400640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f897c003970 0x7f897c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.488+0000 7f899be00640 1 -- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f897c003970 msgr2=0x7f897c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.488+0000 7f899be00640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f897c003970 0x7f897c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.488+0000 7f899be00640 1 -- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c0047c0 msgr2=0x7f897c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.488+0000 7f899be00640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c0047c0 0x7f897c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.488+0000 7f899be00640 1 -- 172.21.15.141:0/2380137055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c003520 con 0x7f897c005180 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.489+0000 7f899be00640 1 -- 172.21.15.141:0/2380137055 reap_dead start 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.489+0000 7f899be00640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f897c005180 0x7f897c00b980 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f899400a800 tx=0x7f897c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8eb81c837808c74 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.489+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994034630 con 0x7f897c005180 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.489+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899402f070 con 0x7f897c005180 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.489+0000 7f899aa00640 1 -- 172.21.15.141:0/2380137055 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994034a10 con 0x7f897c005180 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.490+0000 7f89a2600640 1 -- 172.21.15.141:0/2380137055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f897c005180 msgr2=0x7f897c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.490+0000 7f89a2600640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f897c005180 0x7f897c00b980 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f899400a800 tx=0x7f897c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.490+0000 7f89a2600640 1 -- 172.21.15.141:0/2380137055 shutdown_connections 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.490+0000 7f89a2600640 1 --2- 172.21.15.141:0/2380137055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f897c005180 0x7f897c00b980 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.490+0000 7f89a2600640 1 -- 172.21.15.141:0/2380137055 >> 172.21.15.141:0/2380137055 conn(0x7f899c06a800 msgr2=0x7f899c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:38.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.490+0000 7f89a2600640 1 -- 172.21.15.141:0/2380137055 shutdown_connections 2024-01-16T15:15:38.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.490+0000 7f89a2600640 1 -- 172.21.15.141:0/2380137055 wait complete. 2024-01-16T15:15:38.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.492+0000 7f89a2600640 1 Processor -- start 2024-01-16T15:15:38.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.492+0000 7f89a2600640 1 -- start start 2024-01-16T15:15:38.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f89a2600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c005180 0x7f899c151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f89a2600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f899c15b940 0x7f899c15cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f89a2600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f899c14fa60 0x7f899c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f89a2600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c0cec70 con 0x7f899c14fa60 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f89a2600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c14fee0 con 0x7f897c005180 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f89a2600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899c15bf00 con 0x7f899c15b940 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f899c14fa60 0x7f899c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f899c14fa60 0x7f899c0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57680/0 (socket says 172.21.15.141:57680) 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 -- 172.21.15.141:0/2147384828 learned_addr learned my addr 172.21.15.141:0/2147384828 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899b400640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c005180 0x7f899c151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 -- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f899c15b940 msgr2=0x7f899c15cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f8992800640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f899c15b940 0x7f899c15cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f899c15b940 0x7f899c15cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 -- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c005180 msgr2=0x7f899c151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c005180 0x7f899c151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899be00640 1 -- 172.21.15.141:0/2147384828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c15c1e0 con 0x7f899c14fa60 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.493+0000 7f899b400640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c005180 0x7f899c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.494+0000 7f899be00640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f899c14fa60 0x7f899c0b29c0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f8994002230 tx=0x7f899c06c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.494+0000 7f8993e00640 1 -- 172.21.15.141:0/2147384828 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994042070 con 0x7f899c14fa60 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.494+0000 7f8993e00640 1 -- 172.21.15.141:0/2147384828 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899402ce90 con 0x7f899c14fa60 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.494+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c15c4c0 con 0x7f899c14fa60 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.494+0000 7f8993e00640 1 -- 172.21.15.141:0/2147384828 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994001970 con 0x7f899c14fa60 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.494+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899c15ca80 con 0x7f899c14fa60 2024-01-16T15:15:38.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.495+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899c0675d0 con 0x7f899c14fa60 2024-01-16T15:15:38.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.496+0000 7f8993e00640 1 -- 172.21.15.141:0/2147384828 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f8994031030 con 0x7f899c14fa60 2024-01-16T15:15:38.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.497+0000 7f8993e00640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f897406c4c0 0x7f897406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:38.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.497+0000 7f8993e00640 1 -- 172.21.15.141:0/2147384828 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f89940a9ea0 con 0x7f899c14fa60 2024-01-16T15:15:38.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.498+0000 7f899b400640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f897406c4c0 0x7f897406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:38.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.499+0000 7f899b400640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f897406c4c0 0x7f897406e980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f898c006fd0 tx=0x7f898c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:38.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.502+0000 7f8993e00640 1 -- 172.21.15.141:0/2147384828 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8994076570 con 0x7f899c14fa60 2024-01-16T15:15:38.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:38.719+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f899c0675d0 con 0x7f899c14fa60 2024-01-16T15:15:39.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.307+0000 7f8993e00640 1 -- 172.21.15.141:0/2147384828 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v69) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8994079b90 con 0x7f899c14fa60 2024-01-16T15:15:39.296 INFO:teuthology.orchestra.run.smithi141.stderr:new fs with metadata pool 16 and data pool 17 2024-01-16T15:15:39.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.314+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f897406c4c0 msgr2=0x7f897406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.314+0000 7f89a2600640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f897406c4c0 0x7f897406e980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f898c006fd0 tx=0x7f898c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:39.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.314+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f899c14fa60 msgr2=0x7f899c0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.314+0000 7f89a2600640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f899c14fa60 0x7f899c0b29c0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f8994002230 tx=0x7f899c06c6c0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 shutdown_connections 2024-01-16T15:15:39.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f897406c4c0 0x7f897406e980 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f899c14fa60 0x7f899c0b29c0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f899c15b940 0x7f899c15cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 --2- 172.21.15.141:0/2147384828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f897c005180 0x7f899c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 >> 172.21.15.141:0/2147384828 conn(0x7f899c06a800 msgr2=0x7f899c1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 shutdown_connections 2024-01-16T15:15:39.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.315+0000 7f89a2600640 1 -- 172.21.15.141:0/2147384828 wait complete. 2024-01-16T15:15:39.337 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:39.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.528+0000 7fc862400640 1 Processor -- start 2024-01-16T15:15:39.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.528+0000 7fc862400640 1 -- start start 2024-01-16T15:15:39.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.530+0000 7fc862400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c0cc500 0x7fc85c0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.530+0000 7fc862400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c070e50 con 0x7fc85c143c40 2024-01-16T15:15:39.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.530+0000 7fc862400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c0ccea0 con 0x7fc85c06d160 2024-01-16T15:15:39.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.530+0000 7fc862400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0723c0 con 0x7fc85c0cc500 2024-01-16T15:15:39.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.530+0000 7fc861000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc85c143c40 0x7fc85c071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54502/0 (socket says 172.21.15.141:54502) 2024-01-16T15:15:39.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.530+0000 7fc861000640 1 -- 172.21.15.141:0/1430664184 learned_addr learned my addr 172.21.15.141:0/1430664184 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc853c00640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c0cc500 0x7fc85c0cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284623395 0 0) 0x7fc85c070e50 con 0x7fc85c143c40 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc83c003520 con 0x7fc85c143c40 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3912113993 0 0) 0x7fc83c003520 con 0x7fc85c143c40 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c0cc500 msgr2=0x7fc85c0cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c0cc500 0x7fc85c0cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 >> v1:172.21.15.188:6789/0 conn(0x7fc85c06d160 legacy=0x7fc85c06d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c144020 con 0x7fc85c143c40 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fc84c001670 con 0x7fc85c143c40 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 >> v1:172.21.15.141:6789/0 conn(0x7fc85c143c40 legacy=0x7fc85c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc83c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c0047c0 0x7fc83c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c0cc500 0x7fc83c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0723c0 con 0x7fc83c0039a0 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c00bae0 con 0x7fc83c0047c0 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c00bca0 con 0x7fc85c0cc500 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.531+0000 7fc861000640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc83c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.532+0000 7fc853c00640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c0047c0 0x7fc83c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.532+0000 7fc861000640 1 -- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c0cc500 msgr2=0x7fc83c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.532+0000 7fc861000640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c0cc500 0x7fc83c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.532+0000 7fc861000640 1 -- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c0047c0 msgr2=0x7fc83c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.532+0000 7fc861000640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c0047c0 0x7fc83c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.532+0000 7fc861000640 1 -- 172.21.15.141:0/1430664184 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c003520 con 0x7fc83c0039a0 2024-01-16T15:15:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.532+0000 7fc853c00640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c0047c0 0x7fc83c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:39.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.533+0000 7fc861000640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc83c003dc0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c240 tx=0x7fc83c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984ee870e7c36b70 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.533+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c005ad0 con 0x7fc83c0039a0 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.533+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c005c30 con 0x7fc83c0039a0 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.533+0000 7fc85be00640 1 -- 172.21.15.141:0/1430664184 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c02c8b0 con 0x7fc83c0039a0 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.534+0000 7fc862400640 1 -- 172.21.15.141:0/1430664184 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 msgr2=0x7fc83c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.534+0000 7fc862400640 1 --2- 172.21.15.141:0/1430664184 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc83c003dc0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c240 tx=0x7fc83c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.534+0000 7fc861a00640 1 -- 172.21.15.141:0/1430664184 reap_dead start 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.534+0000 7fc862400640 1 -- 172.21.15.141:0/1430664184 shutdown_connections 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.534+0000 7fc862400640 1 -- 172.21.15.141:0/1430664184 >> 172.21.15.141:0/1430664184 conn(0x7fc85c06a800 msgr2=0x7fc85c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.534+0000 7fc862400640 1 -- 172.21.15.141:0/1430664184 shutdown_connections 2024-01-16T15:15:39.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.534+0000 7fc862400640 1 -- 172.21.15.141:0/1430664184 wait complete. 2024-01-16T15:15:39.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.536+0000 7fc862400640 1 Processor -- start 2024-01-16T15:15:39.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.536+0000 7fc862400640 1 -- start start 2024-01-16T15:15:39.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc862400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc85c1f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc862400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c1569d0 0x7fc85c142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc862400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc85c139050 0x7fc85c1394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc862400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c139ed0 con 0x7fc83c0039a0 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc862400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c13a1b0 con 0x7fc85c139050 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc862400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c13a490 con 0x7fc85c1569d0 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc85c1f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc85c1f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57690/0 (socket says 172.21.15.141:57690) 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 -- 172.21.15.141:0/2954752039 learned_addr learned my addr 172.21.15.141:0/2954752039 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 -- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c1569d0 msgr2=0x7fc85c142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c1569d0 0x7fc85c142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 -- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc85c139050 msgr2=0x7fc85c1394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc85c139050 0x7fc85c1394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.537+0000 7fc861000640 1 -- 172.21.15.141:0/2954752039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c13a770 con 0x7fc83c0039a0 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.538+0000 7fc861000640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc85c1f2750 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fc84c001930 tx=0x7fc84c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.538+0000 7fc859600640 1 -- 172.21.15.141:0/2954752039 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c03c040 con 0x7fc83c0039a0 2024-01-16T15:15:39.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.538+0000 7fc859600640 1 -- 172.21.15.141:0/2954752039 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c02cce0 con 0x7fc83c0039a0 2024-01-16T15:15:39.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.538+0000 7fc859600640 1 -- 172.21.15.141:0/2954752039 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0072b0 con 0x7fc83c0039a0 2024-01-16T15:15:39.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.538+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c1f2c90 con 0x7fc83c0039a0 2024-01-16T15:15:39.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.538+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c139a10 con 0x7fc83c0039a0 2024-01-16T15:15:39.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.539+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85c13a900 con 0x7fc83c0039a0 2024-01-16T15:15:39.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.540+0000 7fc859600640 1 -- 172.21.15.141:0/2954752039 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fc84c032070 con 0x7fc83c0039a0 2024-01-16T15:15:39.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.541+0000 7fc859600640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc83406c4c0 0x7fc83406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.541+0000 7fc859600640 1 -- 172.21.15.141:0/2954752039 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc84c0a97b0 con 0x7fc83c0039a0 2024-01-16T15:15:39.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.542+0000 7fc853c00640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc83406c4c0 0x7fc83406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.543+0000 7fc853c00640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc83406c4c0 0x7fc83406e980 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc844006d40 tx=0x7fc844008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.546+0000 7fc859600640 1 -- 172.21.15.141:0/2954752039 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc84c075d00 con 0x7fc83c0039a0 2024-01-16T15:15:39.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.729+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc85c13a010 con 0x7fc83c0039a0 2024-01-16T15:15:39.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.731+0000 7fc859600640 1 -- 172.21.15.141:0/2954752039 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16588 (secure 0 0 0) 0x7fc85c13a010 con 0x7fc83c0039a0 2024-01-16T15:15:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:39.716 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":115,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:38.720930+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:39.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc83406c4c0 msgr2=0x7fc83406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.718 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc83406c4c0 0x7fc83406e980 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc844006d40 tx=0x7fc844008040 comp rx=0 tx=0).stop 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 msgr2=0x7fc85c1f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc85c1f2750 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fc84c001930 tx=0x7fc84c001a00 comp rx=0 tx=0).stop 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 shutdown_connections 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc83406c4c0 0x7fc83406e980 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc85c139050 0x7fc85c1394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc85c1569d0 0x7fc85c142770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 --2- 172.21.15.141:0/2954752039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc83c0039a0 0x7fc85c1f2750 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 >> 172.21.15.141:0/2954752039 conn(0x7fc85c06a800 msgr2=0x7fc85c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 shutdown_connections 2024-01-16T15:15:39.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.734+0000 7fc862400640 1 -- 172.21.15.141:0/2954752039 wait complete. 2024-01-16T15:15:39.737 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:39.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.955+0000 7fa08c200640 1 Processor -- start 2024-01-16T15:15:39.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.956+0000 7fa08c200640 1 -- start start 2024-01-16T15:15:39.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.957+0000 7fa08c200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 0x7fa0840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.957+0000 7fa08c200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08415c5f0 con 0x7fa08413df80 2024-01-16T15:15:39.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.957+0000 7fa08c200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08406c470 con 0x7fa08415f9c0 2024-01-16T15:15:39.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.957+0000 7fa08c200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08406c750 con 0x7fa08413e360 2024-01-16T15:15:39.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.957+0000 7fa089800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa08413df80 0x7fa08415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54506/0 (socket says 172.21.15.141:54506) 2024-01-16T15:15:39.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.957+0000 7fa089800640 1 -- 172.21.15.141:0/4125867671 learned_addr learned my addr 172.21.15.141:0/4125867671 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126180783 0 0) 0x7fa08415c5f0 con 0x7fa08413df80 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa064003520 con 0x7fa08413df80 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa088e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 0x7fa0840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1138031168 0 0) 0x7fa064003520 con 0x7fa08413df80 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 msgr2=0x7fa0840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 0x7fa0840cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 >> v1:172.21.15.188:6789/0 conn(0x7fa08415f9c0 legacy=0x7fa0840cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08406ca30 con 0x7fa08413df80 2024-01-16T15:15:39.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa074001670 con 0x7fa08413df80 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 >> v1:172.21.15.141:6789/0 conn(0x7fa08413df80 legacy=0x7fa08415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 0x7fa0640039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa064004d00 0x7fa064005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08406c750 con 0x7fa064003fd0 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06400bb40 con 0x7fa064004d00 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.958+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06400bd00 con 0x7fa08413e360 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa088e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa089800640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 0x7fa0640039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa08a200640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa064004d00 0x7fa064005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa088e00640 1 -- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 msgr2=0x7fa0640039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa088e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 0x7fa0640039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa088e00640 1 -- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa064004d00 msgr2=0x7fa064005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa088e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa064004d00 0x7fa064005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa088e00640 1 -- 172.21.15.141:0/4125867671 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064003520 con 0x7fa064003fd0 2024-01-16T15:15:39.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.959+0000 7fa089800640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413e360 0x7fa0640039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:39.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.961+0000 7fa088e00640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0640047c0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00ba30 tx=0x7fa0640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c45ab2072cfb02 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.961+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c032630 con 0x7fa064003fd0 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.961+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c032790 con 0x7fa064003fd0 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.961+0000 7fa083e00640 1 -- 172.21.15.141:0/4125867671 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c032b20 con 0x7fa064003fd0 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.961+0000 7fa08c200640 1 -- 172.21.15.141:0/4125867671 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 msgr2=0x7fa0640047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.961+0000 7fa08c200640 1 --2- 172.21.15.141:0/4125867671 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0640047c0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00ba30 tx=0x7fa0640057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.961+0000 7fa08a200640 1 -- 172.21.15.141:0/4125867671 reap_dead start 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.962+0000 7fa08c200640 1 -- 172.21.15.141:0/4125867671 shutdown_connections 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.962+0000 7fa08c200640 1 -- 172.21.15.141:0/4125867671 >> 172.21.15.141:0/4125867671 conn(0x7fa08406a800 msgr2=0x7fa08406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.962+0000 7fa08c200640 1 -- 172.21.15.141:0/4125867671 shutdown_connections 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.962+0000 7fa08c200640 1 -- 172.21.15.141:0/4125867671 wait complete. 2024-01-16T15:15:39.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.963+0000 7fa08c200640 1 Processor -- start 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.964+0000 7fa08c200640 1 -- start start 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.964+0000 7fa08c200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0840b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.964+0000 7fa08c200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413d1f0 0x7fa0841f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.964+0000 7fa08c200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa084159980 0x7fa084159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa08c200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08415a340 con 0x7fa064003fd0 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa08c200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0841f2c00 con 0x7fa084159980 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa08c200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0841f4430 con 0x7fa08413d1f0 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0840b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0840b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57714/0 (socket says 172.21.15.141:57714) 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 -- 172.21.15.141:0/3941027150 learned_addr learned my addr 172.21.15.141:0/3941027150 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 -- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413d1f0 msgr2=0x7fa0841f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413d1f0 0x7fa0841f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 -- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa084159980 msgr2=0x7fa084159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa084159980 0x7fa084159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.965+0000 7fa089800640 1 -- 172.21.15.141:0/3941027150 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0841f2e50 con 0x7fa064003fd0 2024-01-16T15:15:39.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.966+0000 7fa089800640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0840b3d40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa07400e870 tx=0x7fa07400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.966+0000 7fa081600640 1 -- 172.21.15.141:0/3941027150 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07400c8d0 con 0x7fa064003fd0 2024-01-16T15:15:39.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.966+0000 7fa081600640 1 -- 172.21.15.141:0/3941027150 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07400ca30 con 0x7fa064003fd0 2024-01-16T15:15:39.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.966+0000 7fa081600640 1 -- 172.21.15.141:0/3941027150 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07400cda0 con 0x7fa064003fd0 2024-01-16T15:15:39.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.966+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0841f3130 con 0x7fa064003fd0 2024-01-16T15:15:39.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.966+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0841f36f0 con 0x7fa064003fd0 2024-01-16T15:15:39.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.967+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0840675d0 con 0x7fa064003fd0 2024-01-16T15:15:39.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.968+0000 7fa081600640 1 -- 172.21.15.141:0/3941027150 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa07401d460 con 0x7fa064003fd0 2024-01-16T15:15:39.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.969+0000 7fa081600640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa05c06c4c0 0x7fa05c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.969+0000 7fa081600640 1 -- 172.21.15.141:0/3941027150 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa074088bf0 con 0x7fa064003fd0 2024-01-16T15:15:39.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.969+0000 7fa088e00640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa05c06c4c0 0x7fa05c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.970+0000 7fa088e00640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa05c06c4c0 0x7fa05c06e980 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00ba00 tx=0x7fa06c032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:39.973+0000 7fa081600640 1 -- 172.21.15.141:0/3941027150 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0740551a0 con 0x7fa064003fd0 2024-01-16T15:15:40.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.150+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa08406df00 con 0x7fa064003fd0 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.151+0000 7fa081600640 1 -- 172.21.15.141:0/3941027150 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16588 (secure 0 0 0) 0x7fa0740587c0 con 0x7fa064003fd0 2024-01-16T15:15:40.136 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:40.136 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":115,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:38.720930+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa05c06c4c0 msgr2=0x7fa05c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa05c06c4c0 0x7fa05c06e980 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00ba00 tx=0x7fa06c032000 comp rx=0 tx=0).stop 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 msgr2=0x7fa0840b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0840b3d40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa07400e870 tx=0x7fa07400ed30 comp rx=0 tx=0).stop 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 shutdown_connections 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa05c06c4c0 0x7fa05c06e980 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa084159980 0x7fa084159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa08413d1f0 0x7fa0841f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 --2- 172.21.15.141:0/3941027150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa064003fd0 0x7fa0840b3d40 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 >> 172.21.15.141:0/3941027150 conn(0x7fa08406a800 msgr2=0x7fa08415c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 shutdown_connections 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.154+0000 7fa08c200640 1 -- 172.21.15.141:0/3941027150 wait complete. 2024-01-16T15:15:40.155 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:15:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.369+0000 7f435ae00640 1 Processor -- start 2024-01-16T15:15:40.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.370+0000 7f435ae00640 1 -- start start 2024-01-16T15:15:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.371+0000 7f435ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 0x7f435415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.371+0000 7f435ae00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435406c470 con 0x7f435413e360 2024-01-16T15:15:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.371+0000 7f435ae00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435406c750 con 0x7f43540cf9a0 2024-01-16T15:15:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.371+0000 7f435ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435406ca30 con 0x7f435413df80 2024-01-16T15:15:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f434b200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f435413e360 0x7f43540cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54520/0 (socket says 172.21.15.141:54520) 2024-01-16T15:15:40.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f434b200640 1 -- 172.21.15.141:0/2670717648 learned_addr learned my addr 172.21.15.141:0/2670717648 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353e00640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 0x7f435415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855004030 0 0) 0x7f435406c470 con 0x7f435413e360 2024-01-16T15:15:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4334003520 con 0x7f435413e360 2024-01-16T15:15:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011265474 0 0) 0x7f4334003520 con 0x7f435413e360 2024-01-16T15:15:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 msgr2=0x7f435415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 0x7f435415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 >> v1:172.21.15.188:6789/0 conn(0x7f43540cf9a0 legacy=0x7f43540cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435406cd10 con 0x7f435413e360 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f433c001670 con 0x7f435413e360 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 >> v1:172.21.15.141:6789/0 conn(0x7f435413e360 legacy=0x7f43540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43340039a0 0x7f4334003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340047c0 0x7f4334004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 0x7f4334005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435406ca30 con 0x7f43340039a0 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433400ba60 con 0x7f43340047c0 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.372+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433400bc20 con 0x7f435413df80 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f4353e00640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43340039a0 0x7f4334003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f434b200640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340047c0 0x7f4334004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f4353e00640 1 -- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 msgr2=0x7f4334005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f4358e00640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 0x7f4334005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.359 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f4353e00640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f435413df80 0x7f4334005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f4353e00640 1 -- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340047c0 msgr2=0x7f4334004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f4353e00640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340047c0 0x7f4334004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.360 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.373+0000 7f4353e00640 1 -- 172.21.15.141:0/2670717648 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4334003520 con 0x7f43340039a0 2024-01-16T15:15:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.374+0000 7f4353e00640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43340039a0 0x7f4334003dc0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4344009a90 tx=0x7f43340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a77e81f02735ff3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.374+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434402c680 con 0x7f43340039a0 2024-01-16T15:15:40.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.374+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434402c7e0 con 0x7f43340039a0 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.374+0000 7f4353400640 1 -- 172.21.15.141:0/2670717648 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434402cb70 con 0x7f43340039a0 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.375+0000 7f435ae00640 1 -- 172.21.15.141:0/2670717648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43340039a0 msgr2=0x7f4334003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.375+0000 7f435ae00640 1 --2- 172.21.15.141:0/2670717648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43340039a0 0x7f4334003dc0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4344009a90 tx=0x7f43340057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.375+0000 7f4358e00640 1 -- 172.21.15.141:0/2670717648 reap_dead start 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.375+0000 7f435ae00640 1 -- 172.21.15.141:0/2670717648 shutdown_connections 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.375+0000 7f435ae00640 1 -- 172.21.15.141:0/2670717648 >> 172.21.15.141:0/2670717648 conn(0x7f435406a800 msgr2=0x7f435406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.375+0000 7f435ae00640 1 -- 172.21.15.141:0/2670717648 shutdown_connections 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.375+0000 7f435ae00640 1 -- 172.21.15.141:0/2670717648 wait complete. 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.377+0000 7f435ae00640 1 Processor -- start 2024-01-16T15:15:40.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.377+0000 7f435ae00640 1 -- start start 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.378+0000 7f435ae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340039a0 0x7f435414f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.378+0000 7f435ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4354152e60 0x7f4354154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.378+0000 7f435ae00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4354156700 0x7f4354157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.378+0000 7f435ae00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43541562e0 con 0x7f4354156700 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.378+0000 7f435ae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4354155660 con 0x7f43340039a0 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.378+0000 7f435ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4354155940 con 0x7f4354152e60 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4354156700 0x7f4354157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4354156700 0x7f4354157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57726/0 (socket says 172.21.15.141:57726) 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 -- 172.21.15.141:0/3510243441 learned_addr learned my addr 172.21.15.141:0/3510243441 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f434b200640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4354152e60 0x7f4354154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4353e00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340039a0 0x7f435414f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 -- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4354152e60 msgr2=0x7f4354154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4354152e60 0x7f4354154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 -- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340039a0 msgr2=0x7f435414f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340039a0 0x7f435414f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 -- 172.21.15.141:0/3510243441 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4354155c20 con 0x7f4354156700 2024-01-16T15:15:40.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4353e00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340039a0 0x7f435414f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:40.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f434b200640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4354152e60 0x7f4354154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:40.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.379+0000 7f4358e00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4354156700 0x7f4354157ba0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f43540cfea0 tx=0x7f434c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.380+0000 7f4350c00640 1 -- 172.21.15.141:0/3510243441 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c003b70 con 0x7f4354156700 2024-01-16T15:15:40.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.380+0000 7f4350c00640 1 -- 172.21.15.141:0/3510243441 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c003cd0 con 0x7f4354156700 2024-01-16T15:15:40.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.380+0000 7f4350c00640 1 -- 172.21.15.141:0/3510243441 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c0052b0 con 0x7f4354156700 2024-01-16T15:15:40.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.380+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4354155f00 con 0x7f4354156700 2024-01-16T15:15:40.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.380+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4354155230 con 0x7f4354156700 2024-01-16T15:15:40.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.381+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4354156420 con 0x7f4354156700 2024-01-16T15:15:40.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.382+0000 7f4350c00640 1 -- 172.21.15.141:0/3510243441 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f434c005410 con 0x7f4354156700 2024-01-16T15:15:40.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.383+0000 7f4350c00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f432c06c4c0 0x7f432c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.383+0000 7f4350c00640 1 -- 172.21.15.141:0/3510243441 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f434c088590 con 0x7f4354156700 2024-01-16T15:15:40.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.384+0000 7f4353e00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f432c06c4c0 0x7f432c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.385+0000 7f4353e00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f432c06c4c0 0x7f432c06e980 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4344036000 tx=0x7f4344030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.388+0000 7f4350c00640 1 -- 172.21.15.141:0/3510243441 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f434c0bee90 con 0x7f4354156700 2024-01-16T15:15:40.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:40.620+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f435415beb0 con 0x7f4354156700 2024-01-16T15:15:41.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.334+0000 7f4350c00640 1 -- 172.21.15.141:0/3510243441 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f434c018030 con 0x7f4354156700 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f432c06c4c0 msgr2=0x7f432c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f432c06c4c0 0x7f432c06e980 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4344036000 tx=0x7f4344030000 comp rx=0 tx=0).stop 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4354156700 msgr2=0x7f4354157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4354156700 0x7f4354157ba0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f43540cfea0 tx=0x7f434c00dc00 comp rx=0 tx=0).stop 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 shutdown_connections 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f432c06c4c0 0x7f432c06e980 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4354156700 0x7f4354157ba0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4354152e60 0x7f4354154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 --2- 172.21.15.141:0/3510243441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43340039a0 0x7f435414f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 >> 172.21.15.141:0/3510243441 conn(0x7f435406a800 msgr2=0x7f4354159740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:41.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.338+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 shutdown_connections 2024-01-16T15:15:41.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.339+0000 7f435ae00640 1 -- 172.21.15.141:0/3510243441 wait complete. 2024-01-16T15:15:41.344 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:15:41.344 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:15:41.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.571+0000 7f519a200640 1 Processor -- start 2024-01-16T15:15:41.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.571+0000 7f519a200640 1 -- start start 2024-01-16T15:15:41.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.572+0000 7f519a200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f51940adf20 0x7f5194070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f519a200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519415f4e0 con 0x7f51940ccb50 2024-01-16T15:15:41.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f519a200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5194071130 con 0x7f519415bf10 2024-01-16T15:15:41.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f519a200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519415f970 con 0x7f51940adf20 2024-01-16T15:15:41.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f5199800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f51940ccb50 0x7f5194143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54530/0 (socket says 172.21.15.141:54530) 2024-01-16T15:15:41.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f5199800640 1 -- 172.21.15.141:0/919580477 learned_addr learned my addr 172.21.15.141:0/919580477 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f5198e00640 1 --2- 172.21.15.141:0/919580477 >> v2:172.21.15.188:3300/0 conn(0x7f51940adf20 0x7f5194070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332591134 0 0) 0x7f519415f4e0 con 0x7f51940ccb50 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.573+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5174003520 con 0x7f51940ccb50 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289773387 0 0) 0x7f5174003520 con 0x7f51940ccb50 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 >> v2:172.21.15.188:3300/0 conn(0x7f51940adf20 msgr2=0x7f5194070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 --2- 172.21.15.141:0/919580477 >> v2:172.21.15.188:3300/0 conn(0x7f51940adf20 0x7f5194070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 >> v1:172.21.15.188:6789/0 conn(0x7f519415bf10 legacy=0x7f51940cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519415c390 con 0x7f51940ccb50 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f518c0019f0 con 0x7f51940ccb50 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 >> v1:172.21.15.141:6789/0 conn(0x7f51940ccb50 legacy=0x7f5194143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5174003970 0x7f5174003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51740047c0 0x7f5174004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174005180 0x7f517400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519415f970 con 0x7f5174003970 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517400c120 con 0x7f5174005180 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517400c2e0 con 0x7f51740047c0 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.574+0000 7f5198e00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5174003970 0x7f5174003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f518bc00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51740047c0 0x7f5174004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5198e00640 1 -- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51740047c0 msgr2=0x7f5174004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5198e00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f51740047c0 0x7f5174004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5199800640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174005180 0x7f517400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5198e00640 1 -- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174005180 msgr2=0x7f517400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5198e00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174005180 0x7f517400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5198e00640 1 -- 172.21.15.141:0/919580477 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174003520 con 0x7f5174003970 2024-01-16T15:15:41.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5199800640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174005180 0x7f517400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:41.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.575+0000 7f5199800640 1 -- 172.21.15.141:0/919580477 reap_dead start 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.576+0000 7f5198e00640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5174003970 0x7f5174003d90 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5184009ac0 tx=0x7f51740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c58db3329db161e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.576+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518402c6b0 con 0x7f5174003970 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.576+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518402c810 con 0x7f5174003970 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.576+0000 7f5193e00640 1 -- 172.21.15.141:0/919580477 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518402cba0 con 0x7f5174003970 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.577+0000 7f519a200640 1 -- 172.21.15.141:0/919580477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5174003970 msgr2=0x7f5174003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.577+0000 7f519a200640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5174003970 0x7f5174003d90 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5184009ac0 tx=0x7f51740057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.577+0000 7f519a200640 1 -- 172.21.15.141:0/919580477 shutdown_connections 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.577+0000 7f519a200640 1 --2- 172.21.15.141:0/919580477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5174003970 0x7f5174003d90 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.577+0000 7f519a200640 1 -- 172.21.15.141:0/919580477 >> 172.21.15.141:0/919580477 conn(0x7f519406a7d0 msgr2=0x7f519406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:41.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.577+0000 7f519a200640 1 -- 172.21.15.141:0/919580477 shutdown_connections 2024-01-16T15:15:41.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.577+0000 7f519a200640 1 -- 172.21.15.141:0/919580477 wait complete. 2024-01-16T15:15:41.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.579+0000 7f519a200640 1 Processor -- start 2024-01-16T15:15:41.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.579+0000 7f519a200640 1 -- start start 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f519a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174003970 0x7f519413abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f519a200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5194141770 0x7f519413c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f519a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5194141bf0 0x7f519413cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f519a200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519413c3e0 con 0x7f5194141770 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f519a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5194142980 con 0x7f5174003970 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f519a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51941420e0 con 0x7f5194141bf0 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5194141770 0x7f519413c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5194141770 0x7f519413c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57738/0 (socket says 172.21.15.141:57738) 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 -- 172.21.15.141:0/732359312 learned_addr learned my addr 172.21.15.141:0/732359312 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 -- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5194141bf0 msgr2=0x7f519413cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f5199800640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5194141bf0 0x7f519413cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f5198e00640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174003970 0x7f519413abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5194141bf0 0x7f519413cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 -- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174003970 msgr2=0x7f519413abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174003970 0x7f519413abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.580+0000 7f518bc00640 1 -- 172.21.15.141:0/732359312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51941423c0 con 0x7f5194141770 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.581+0000 7f518bc00640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5194141770 0x7f519413c830 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f517c001eb0 tx=0x7f517c00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.581+0000 7f5191600640 1 -- 172.21.15.141:0/732359312 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c005480 con 0x7f5194141770 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.581+0000 7f5191600640 1 -- 172.21.15.141:0/732359312 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c003610 con 0x7f5194141770 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.581+0000 7f5191600640 1 -- 172.21.15.141:0/732359312 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c003980 con 0x7f5194141770 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.581+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51941426a0 con 0x7f5194141770 2024-01-16T15:15:41.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.582+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519415a7b0 con 0x7f5194141770 2024-01-16T15:15:41.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.583+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519415a9d0 con 0x7f5194141770 2024-01-16T15:15:41.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.583+0000 7f5191600640 1 -- 172.21.15.141:0/732359312 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f517c003ae0 con 0x7f5194141770 2024-01-16T15:15:41.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.584+0000 7f5191600640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f516c06c4c0 0x7f516c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.584+0000 7f5191600640 1 -- 172.21.15.141:0/732359312 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f517c088a00 con 0x7f5194141770 2024-01-16T15:15:41.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.585+0000 7f5198e00640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f516c06c4c0 0x7f516c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.586+0000 7f5198e00640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f516c06c4c0 0x7f516c06e980 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5184036000 tx=0x7f5184030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.573 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.588+0000 7f5191600640 1 -- 172.21.15.141:0/732359312 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f519415a9d0 con 0x7f5194141770 2024-01-16T15:15:41.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:41.816+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f519415a9d0 con 0x7f5194141770 2024-01-16T15:15:42.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.337+0000 7f5191600640 1 -- 172.21.15.141:0/732359312 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f519415a9d0 con 0x7f5194141770 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f516c06c4c0 msgr2=0x7f516c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f516c06c4c0 0x7f516c06e980 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5184036000 tx=0x7f5184030000 comp rx=0 tx=0).stop 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5194141770 msgr2=0x7f519413c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5194141770 0x7f519413c830 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f517c001eb0 tx=0x7f517c00d9d0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 shutdown_connections 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f516c06c4c0 0x7f516c06e980 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5194141bf0 0x7f519413cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.326 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5194141770 0x7f519413c830 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 --2- 172.21.15.141:0/732359312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5174003970 0x7f519413abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 >> 172.21.15.141:0/732359312 conn(0x7f519406a7d0 msgr2=0x7f51940acd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:42.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 shutdown_connections 2024-01-16T15:15:42.327 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.341+0000 7f519a200640 1 -- 172.21.15.141:0/732359312 wait complete. 2024-01-16T15:15:42.346 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:15:42.346 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:42.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.569+0000 7feba5c00640 1 Processor -- start 2024-01-16T15:15:42.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.569+0000 7feba5c00640 1 -- start start 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.570+0000 7feba5c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7feba00cf9a0 0x7feba015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.570+0000 7feba5c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feba015e1e0 0x7feba00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.570+0000 7feba5c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba00b1ee0 con 0x7feba0143ae0 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.570+0000 7feba5c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba00b2780 con 0x7feba015e1e0 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.570+0000 7feba5c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba015e990 con 0x7feba00cf9a0 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9f400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feba015e1e0 0x7feba00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7feba0143ae0 0x7feba00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54534/0 (socket says 172.21.15.141:54534) 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9ea00640 1 -- 172.21.15.141:0/1430969462 learned_addr learned my addr 172.21.15.141:0/1430969462 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb95e00640 1 --2- 172.21.15.141:0/1430969462 >> v2:172.21.15.188:3300/0 conn(0x7feba00cf9a0 0x7feba015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9e000640 1 -- 172.21.15.141:0/1430969462 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404880773 0 0) 0x7feba00b1ee0 con 0x7feba0143ae0 2024-01-16T15:15:42.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9f400640 1 -- 172.21.15.141:0/1430969462 >> v2:172.21.15.188:3300/0 conn(0x7feba00cf9a0 msgr2=0x7feba015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:42.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9f400640 1 --2- 172.21.15.141:0/1430969462 >> v2:172.21.15.188:3300/0 conn(0x7feba00cf9a0 0x7feba015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9f400640 1 -- 172.21.15.141:0/1430969462 >> v1:172.21.15.141:6789/0 conn(0x7feba0143ae0 legacy=0x7feba00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb9f400640 1 -- 172.21.15.141:0/1430969462 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba00b2130 con 0x7feba015e1e0 2024-01-16T15:15:42.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.571+0000 7feb95e00640 1 --2- 172.21.15.141:0/1430969462 >> v2:172.21.15.188:3300/0 conn(0x7feba00cf9a0 0x7feba015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:42.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.572+0000 7feb9f400640 1 --2- 172.21.15.141:0/1430969462 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba00702c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7feba00cb830 tx=0x7feb9802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b43d9bc1a15d7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feb9e000640 1 -- 172.21.15.141:0/1430969462 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9802cc70 con 0x7feba015e1e0 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feb9e000640 1 -- 172.21.15.141:0/1430969462 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9802cdd0 con 0x7feba015e1e0 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feb9e000640 1 -- 172.21.15.141:0/1430969462 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98034660 con 0x7feba015e1e0 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 -- 172.21.15.141:0/1430969462 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 msgr2=0x7feba00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 --2- 172.21.15.141:0/1430969462 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba00702c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7feba00cb830 tx=0x7feb9802bec0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 -- 172.21.15.141:0/1430969462 shutdown_connections 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 --2- 172.21.15.141:0/1430969462 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba00702c0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 --2- 172.21.15.141:0/1430969462 >> v2:172.21.15.188:3300/0 conn(0x7feba00cf9a0 0x7feba015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 -- 172.21.15.141:0/1430969462 >> 172.21.15.141:0/1430969462 conn(0x7feba006a800 msgr2=0x7feba006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 -- 172.21.15.141:0/1430969462 shutdown_connections 2024-01-16T15:15:42.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.573+0000 7feba5c00640 1 -- 172.21.15.141:0/1430969462 wait complete. 2024-01-16T15:15:42.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.575+0000 7feba5c00640 1 Processor -- start 2024-01-16T15:15:42.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.575+0000 7feba5c00640 1 -- start start 2024-01-16T15:15:42.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feba5c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feba00cf9a0 0x7feba01ee170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feba5c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feba0143ae0 0x7feba0070a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feba5c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba006c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feba5c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0130ee0 con 0x7feba015e1e0 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feba5c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba01343d0 con 0x7feba0143ae0 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feba5c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba0130010 con 0x7feba00cf9a0 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feb9f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba006c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feb9f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba006c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57762/0 (socket says 172.21.15.141:57762) 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.576+0000 7feb9f400640 1 -- 172.21.15.141:0/4025571306 learned_addr learned my addr 172.21.15.141:0/4025571306 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb95e00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feba0143ae0 0x7feba0070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb9ea00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feba00cf9a0 0x7feba01ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb9f400640 1 -- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feba00cf9a0 msgr2=0x7feba01ee170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb9f400640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feba00cf9a0 0x7feba01ee170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb9f400640 1 -- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feba0143ae0 msgr2=0x7feba0070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb9f400640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feba0143ae0 0x7feba0070a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb9f400640 1 -- 172.21.15.141:0/4025571306 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98009a70 con 0x7feba015e1e0 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.577+0000 7feb9f400640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba006c890 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7feb9802c4d0 tx=0x7feb98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.578+0000 7feb97400640 1 -- 172.21.15.141:0/4025571306 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9803e610 con 0x7feba015e1e0 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.578+0000 7feb97400640 1 -- 172.21.15.141:0/4025571306 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9803e790 con 0x7feba015e1e0 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.578+0000 7feb97400640 1 -- 172.21.15.141:0/4025571306 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9803eb40 con 0x7feba015e1e0 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.578+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba0130260 con 0x7feba015e1e0 2024-01-16T15:15:42.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.578+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba0130820 con 0x7feba015e1e0 2024-01-16T15:15:42.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.579+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba00675d0 con 0x7feba015e1e0 2024-01-16T15:15:42.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.580+0000 7feb97400640 1 -- 172.21.15.141:0/4025571306 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7feb9803b050 con 0x7feba015e1e0 2024-01-16T15:15:42.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.581+0000 7feb97400640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb7806c4c0 0x7feb7806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.581+0000 7feb97400640 1 -- 172.21.15.141:0/4025571306 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb980ab130 con 0x7feba015e1e0 2024-01-16T15:15:42.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.581+0000 7feb9ea00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb7806c4c0 0x7feb7806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.582+0000 7feb9ea00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb7806c4c0 0x7feb7806e980 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feb90000f60 tx=0x7feb90007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.585+0000 7feb97400640 1 -- 172.21.15.141:0/4025571306 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb98077a50 con 0x7feba015e1e0 2024-01-16T15:15:42.798 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.814+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feba006aec0 con 0x7feba015e1e0 2024-01-16T15:15:42.799 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.815+0000 7feb97400640 1 -- 172.21.15.141:0/4025571306 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4924 (secure 0 0 0) 0x7feb9804e020 con 0x7feba015e1e0 2024-01-16T15:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3065530834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3065530834},{"type":"v1","addr":"172.21.15.188:6835","nonce":3065530834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6140,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/1935621003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1935621003},{"type":"v1","addr":"172.21.15.141:6834","nonce":1935621003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6146,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/935439310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":935439310},{"type":"v1","addr":"172.21.15.188:6837","nonce":935439310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:38.721379+0000","modified":"2024-01-16T15:15:41.817485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1556719184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":1556719184},{"type":"v1","addr":"172.21.15.141:6835","nonce":1556719184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:15:42.800 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 73 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb7806c4c0 msgr2=0x7feb7806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb7806c4c0 0x7feb7806e980 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feb90000f60 tx=0x7feb90007b50 comp rx=0 tx=0).stop 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 msgr2=0x7feba006c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba006c890 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7feb9802c4d0 tx=0x7feb98004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 shutdown_connections 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7feb7806c4c0 0x7feb7806e980 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feba015e1e0 0x7feba006c890 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feba0143ae0 0x7feba0070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 --2- 172.21.15.141:0/4025571306 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7feba00cf9a0 0x7feba01ee170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 >> 172.21.15.141:0/4025571306 conn(0x7feba006a800 msgr2=0x7feba015ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:42.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 shutdown_connections 2024-01-16T15:15:42.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:42.818+0000 7feba5c00640 1 -- 172.21.15.141:0/4025571306 wait complete. 2024-01-16T15:15:42.822 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:43.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.037+0000 7f38b4a00640 1 Processor -- start 2024-01-16T15:15:43.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.038+0000 7f38b4a00640 1 -- start start 2024-01-16T15:15:43.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.039+0000 7f38b4a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f38b00d0100 0x7f38b00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.039+0000 7f38b4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0f40 0x7f38b00d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.039+0000 7f38b4a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b00d0a10 con 0x7f38b0160920 2024-01-16T15:15:43.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.039+0000 7f38b4a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b00cf080 con 0x7f38b00d0100 2024-01-16T15:15:43.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.039+0000 7f38b4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b00c6be0 con 0x7f38b00d0f40 2024-01-16T15:15:43.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.040+0000 7f38af200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0f40 0x7f38b00d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.040+0000 7f38ae800640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f38b0160920 0x7f38b00cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:50106/0 (socket says 172.21.15.141:50106) 2024-01-16T15:15:43.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.040+0000 7f38ae800640 1 -- 172.21.15.141:0/3208659271 learned_addr learned my addr 172.21.15.141:0/3208659271 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:43.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.040+0000 7f38ade00640 1 --2- 172.21.15.141:0/3208659271 >> v2:172.21.15.188:3300/0 conn(0x7f38b00d0100 0x7f38b00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.040+0000 7f38ad400640 1 -- 172.21.15.141:0/3208659271 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822913570 0 0) 0x7f38b00d0a10 con 0x7f38b0160920 2024-01-16T15:15:43.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.040+0000 7f38ad400640 1 -- 172.21.15.141:0/3208659271 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3890003520 con 0x7f38b0160920 2024-01-16T15:15:43.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.041+0000 7f38af200640 1 -- 172.21.15.141:0/3208659271 >> v2:172.21.15.188:3300/0 conn(0x7f38b00d0100 msgr2=0x7f38b00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:43.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.041+0000 7f38af200640 1 --2- 172.21.15.141:0/3208659271 >> v2:172.21.15.188:3300/0 conn(0x7f38b00d0100 0x7f38b00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.041+0000 7f38af200640 1 -- 172.21.15.141:0/3208659271 >> v1:172.21.15.188:6789/0 conn(0x7f38b0160920 legacy=0x7f38b00cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.041+0000 7f38af200640 1 -- 172.21.15.141:0/3208659271 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b00ce7e0 con 0x7f38b00d0f40 2024-01-16T15:15:43.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.042+0000 7f38af200640 1 --2- 172.21.15.141:0/3208659271 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0f40 0x7f38b00d13d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f38a80097a0 tx=0x7f38a802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=effbc02f322162e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.042+0000 7f38ad400640 1 -- 172.21.15.141:0/3208659271 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8031830 con 0x7f38b00d0f40 2024-01-16T15:15:43.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38ad400640 1 -- 172.21.15.141:0/3208659271 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a8031990 con 0x7f38b00d0f40 2024-01-16T15:15:43.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38ad400640 1 -- 172.21.15.141:0/3208659271 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8031d20 con 0x7f38b00d0f40 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 -- 172.21.15.141:0/3208659271 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0f40 msgr2=0x7f38b00d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 --2- 172.21.15.141:0/3208659271 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0f40 0x7f38b00d13d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f38a80097a0 tx=0x7f38a802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 -- 172.21.15.141:0/3208659271 shutdown_connections 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 --2- 172.21.15.141:0/3208659271 >> v2:172.21.15.188:3300/0 conn(0x7f38b00d0100 0x7f38b00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 --2- 172.21.15.141:0/3208659271 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0f40 0x7f38b00d13d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 -- 172.21.15.141:0/3208659271 >> 172.21.15.141:0/3208659271 conn(0x7f38b00d3090 msgr2=0x7f38b00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 -- 172.21.15.141:0/3208659271 shutdown_connections 2024-01-16T15:15:43.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.043+0000 7f38b4a00640 1 -- 172.21.15.141:0/3208659271 wait complete. 2024-01-16T15:15:43.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.045+0000 7f38b4a00640 1 Processor -- start 2024-01-16T15:15:43.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.045+0000 7f38b4a00640 1 -- start start 2024-01-16T15:15:43.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38b4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0100 0x7f38b01f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38b4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b00d0f40 0x7f38b015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38b4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b0160920 0x7f38b015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38b4a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b01600f0 con 0x7f38b0160920 2024-01-16T15:15:43.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38b4a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b015f7c0 con 0x7f38b00d0f40 2024-01-16T15:15:43.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38b4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b015faa0 con 0x7f38b00d0100 2024-01-16T15:15:43.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38af200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b0160920 0x7f38b015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38af200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b0160920 0x7f38b015ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55350/0 (socket says 172.21.15.141:55350) 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.046+0000 7f38af200640 1 -- 172.21.15.141:0/2648261497 learned_addr learned my addr 172.21.15.141:0/2648261497 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38ae800640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0100 0x7f38b01f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38ade00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b00d0f40 0x7f38b015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38af200640 1 -- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0100 msgr2=0x7f38b01f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38af200640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0100 0x7f38b01f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38af200640 1 -- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b00d0f40 msgr2=0x7f38b015e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38af200640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b00d0f40 0x7f38b015e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38af200640 1 -- 172.21.15.141:0/2648261497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8009400 con 0x7f38b0160920 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38ade00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b00d0f40 0x7f38b015e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38af200640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b0160920 0x7f38b015ee80 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f38a80097a0 tx=0x7f38a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38a6a00640 1 -- 172.21.15.141:0/2648261497 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a803a630 con 0x7f38b0160920 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38a6a00640 1 -- 172.21.15.141:0/2648261497 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a803a790 con 0x7f38b0160920 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.047+0000 7f38a6a00640 1 -- 172.21.15.141:0/2648261497 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a803ab40 con 0x7f38b0160920 2024-01-16T15:15:43.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.048+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b015fd80 con 0x7f38b0160920 2024-01-16T15:15:43.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.048+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b01f2a50 con 0x7f38b0160920 2024-01-16T15:15:43.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.049+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b00b6700 con 0x7f38b0160920 2024-01-16T15:15:43.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.050+0000 7f38a6a00640 1 -- 172.21.15.141:0/2648261497 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f38a803aca0 con 0x7f38b0160920 2024-01-16T15:15:43.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.050+0000 7f38a6a00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f388806c4f0 0x7f388806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.051+0000 7f38a6a00640 1 -- 172.21.15.141:0/2648261497 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38a80aa3a0 con 0x7f38b0160920 2024-01-16T15:15:43.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.051+0000 7f38ae800640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f388806c4f0 0x7f388806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.052+0000 7f38ae800640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f388806c4f0 0x7f388806e9b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f38980011e0 tx=0x7f3898009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.055+0000 7f38a6a00640 1 -- 172.21.15.141:0/2648261497 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38b00b6700 con 0x7f38b0160920 2024-01-16T15:15:43.222 INFO:tasks.ceph.mds.a.smithi141.stderr:2024-01-16T15:15:43.237+0000 7f744f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:15:43.229 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.245+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38b00b6700 con 0x7f38b0160920 2024-01-16T15:15:43.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.247+0000 7f38a6a00640 1 -- 172.21.15.141:0/2648261497 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16575 (secure 0 0 0) 0x7f38b00b6700 con 0x7f38b0160920 2024-01-16T15:15:43.232 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:43.232 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":116,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:40.306846+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:43.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.250+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f388806c4f0 msgr2=0x7f388806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.250+0000 7f38b4a00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f388806c4f0 0x7f388806e9b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f38980011e0 tx=0x7f3898009040 comp rx=0 tx=0).stop 2024-01-16T15:15:43.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.250+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b0160920 msgr2=0x7f38b015ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.250+0000 7f38b4a00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b0160920 0x7f38b015ee80 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f38a80097a0 tx=0x7f38a80046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.250+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 shutdown_connections 2024-01-16T15:15:43.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.250+0000 7f38b4a00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f388806c4f0 0x7f388806e9b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.251+0000 7f38b4a00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f38b0160920 0x7f38b015ee80 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.251+0000 7f38b4a00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38b00d0f40 0x7f38b015e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.251+0000 7f38b4a00640 1 --2- 172.21.15.141:0/2648261497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f38b00d0100 0x7f38b01f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.251+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 >> 172.21.15.141:0/2648261497 conn(0x7f38b00d3090 msgr2=0x7f38b006aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:43.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.251+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 shutdown_connections 2024-01-16T15:15:43.236 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.251+0000 7f38b4a00640 1 -- 172.21.15.141:0/2648261497 wait complete. 2024-01-16T15:15:43.255 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:15:43.255 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:43.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.481+0000 7fecc7c00640 1 Processor -- start 2024-01-16T15:15:43.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.481+0000 7fecc7c00640 1 -- start start 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.482+0000 7fecc7c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc00cf9a0 0x7fecc0070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.482+0000 7fecc7c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fecc0070ea0 0x7fecc00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.482+0000 7fecc7c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc0071690 con 0x7fecc014b2c0 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.482+0000 7fecc7c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc014b6a0 con 0x7fecc0070ea0 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.482+0000 7fecc7c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0071970 con 0x7fecc00cf9a0 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecc5200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fecc014b2c0 0x7fecc00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57456/0 (socket says 172.21.15.141:57456) 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecc5200640 1 -- 172.21.15.141:0/4728132 learned_addr learned my addr 172.21.15.141:0/4728132 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:43.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecc5c00640 1 --2- 172.21.15.141:0/4728132 >> v2:172.21.15.141:3300/0 conn(0x7fecc0070ea0 0x7fecc00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecbfe00640 1 --2- 172.21.15.141:0/4728132 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc00cf9a0 0x7fecc0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecbf400640 1 -- 172.21.15.141:0/4728132 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958711801 0 0) 0x7fecc0071690 con 0x7fecc014b2c0 2024-01-16T15:15:43.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecbf400640 1 -- 172.21.15.141:0/4728132 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca0003520 con 0x7fecc014b2c0 2024-01-16T15:15:43.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecc5c00640 1 -- 172.21.15.141:0/4728132 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc00cf9a0 msgr2=0x7fecc0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:43.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.483+0000 7fecc5c00640 1 --2- 172.21.15.141:0/4728132 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc00cf9a0 0x7fecc0070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.484+0000 7fecc5c00640 1 -- 172.21.15.141:0/4728132 >> v1:172.21.15.141:6789/0 conn(0x7fecc014b2c0 legacy=0x7fecc00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.484+0000 7fecc5c00640 1 -- 172.21.15.141:0/4728132 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0071c50 con 0x7fecc0070ea0 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.485+0000 7fecc5c00640 1 --2- 172.21.15.141:0/4728132 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc0070ea0 0x7fecc00cc160 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009d90 tx=0x7fecb802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa50838c6eaff77 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.485+0000 7fecbf400640 1 -- 172.21.15.141:0/4728132 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb802cc50 con 0x7fecc0070ea0 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.485+0000 7fecbf400640 1 -- 172.21.15.141:0/4728132 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb802cdb0 con 0x7fecc0070ea0 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.485+0000 7fecbf400640 1 -- 172.21.15.141:0/4728132 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8034660 con 0x7fecc0070ea0 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.485+0000 7fecc7c00640 1 -- 172.21.15.141:0/4728132 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc0070ea0 msgr2=0x7fecc00cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.485+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4728132 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc0070ea0 0x7fecc00cc160 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009d90 tx=0x7fecb802bea0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.486+0000 7fecc7c00640 1 -- 172.21.15.141:0/4728132 shutdown_connections 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.486+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4728132 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc0070ea0 0x7fecc00cc160 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.486+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4728132 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc00cf9a0 0x7fecc0070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.486+0000 7fecc7c00640 1 -- 172.21.15.141:0/4728132 >> 172.21.15.141:0/4728132 conn(0x7fecc006a800 msgr2=0x7fecc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.486+0000 7fecc7c00640 1 -- 172.21.15.141:0/4728132 shutdown_connections 2024-01-16T15:15:43.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.486+0000 7fecc7c00640 1 -- 172.21.15.141:0/4728132 wait complete. 2024-01-16T15:15:43.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.487+0000 7fecc7c00640 1 Processor -- start 2024-01-16T15:15:43.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.488+0000 7fecc7c00640 1 -- start start 2024-01-16T15:15:43.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.488+0000 7fecc7c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecc0070ea0 0x7fecc0160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.488+0000 7fecc7c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc00cf9a0 0x7fecc013f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.488+0000 7fecc7c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc014b2c0 0x7fecc0164390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.488+0000 7fecc7c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0140170 con 0x7fecc00cf9a0 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.488+0000 7fecc7c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0140450 con 0x7fecc0070ea0 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.488+0000 7fecc7c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0140730 con 0x7fecc014b2c0 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc00cf9a0 0x7fecc013f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc00cf9a0 0x7fecc013f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55378/0 (socket says 172.21.15.141:55378) 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 -- 172.21.15.141:0/4134365663 learned_addr learned my addr 172.21.15.141:0/4134365663 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecc5200640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecc0070ea0 0x7fecc0160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 -- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc014b2c0 msgr2=0x7fecc0164390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecc5c00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc014b2c0 0x7fecc0164390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc014b2c0 0x7fecc0164390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 -- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecc0070ea0 msgr2=0x7fecc0160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecc0070ea0 0x7fecc0160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 -- 172.21.15.141:0/4134365663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb8009a70 con 0x7fecc00cf9a0 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.489+0000 7fecbfe00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc00cf9a0 0x7fecc013f840 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7feca800d800 tx=0x7feca800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.490+0000 7fecbcc00640 1 -- 172.21.15.141:0/4134365663 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800b6c0 con 0x7fecc00cf9a0 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.490+0000 7fecbcc00640 1 -- 172.21.15.141:0/4134365663 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca800b820 con 0x7fecc00cf9a0 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.490+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc0140a10 con 0x7fecc00cf9a0 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.490+0000 7fecbcc00640 1 -- 172.21.15.141:0/4134365663 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800bbb0 con 0x7fecc00cf9a0 2024-01-16T15:15:43.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.490+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc0160da0 con 0x7fecc00cf9a0 2024-01-16T15:15:43.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.491+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc00675d0 con 0x7fecc00cf9a0 2024-01-16T15:15:43.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.492+0000 7fecbcc00640 1 -- 172.21.15.141:0/4134365663 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7feca800bd10 con 0x7fecc00cf9a0 2024-01-16T15:15:43.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.493+0000 7fecbcc00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec9806c4c0 0x7fec9806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.493+0000 7fecbcc00640 1 -- 172.21.15.141:0/4134365663 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feca8088d20 con 0x7fecc00cf9a0 2024-01-16T15:15:43.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.493+0000 7fecc5200640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec9806c4c0 0x7fec9806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.494+0000 7fecc5200640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec9806c4c0 0x7fec9806e980 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fecb00011e0 tx=0x7fecb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.497+0000 7fecbcc00640 1 -- 172.21.15.141:0/4134365663 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feca8055640 con 0x7fecc00cf9a0 2024-01-16T15:15:43.671 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.686+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fecc01405c0 con 0x7fec9806c4c0 2024-01-16T15:15:43.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.713+0000 7fecbcc00640 1 -- 172.21.15.141:0/4134365663 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286996 (secure 0 0 0) 0x7fecc01405c0 con 0x7fec9806c4c0 2024-01-16T15:15:43.703 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:43.705 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:15:43.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec9806c4c0 msgr2=0x7fec9806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec9806c4c0 0x7fec9806e980 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fecb00011e0 tx=0x7fecb0009040 comp rx=0 tx=0).stop 2024-01-16T15:15:43.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc00cf9a0 msgr2=0x7fecc013f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc00cf9a0 0x7fecc013f840 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7feca800d800 tx=0x7feca800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 shutdown_connections 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fec9806c4c0 0x7fec9806e980 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fecc014b2c0 0x7fecc0164390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fecc00cf9a0 0x7fecc013f840 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 --2- 172.21.15.141:0/4134365663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecc0070ea0 0x7fecc0160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 >> 172.21.15.141:0/4134365663 conn(0x7fecc006a800 msgr2=0x7fecc00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.720+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 shutdown_connections 2024-01-16T15:15:43.707 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.721+0000 7fecc7c00640 1 -- 172.21.15.141:0/4134365663 wait complete. 2024-01-16T15:15:43.725 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2024-01-16T15:15:42.921873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":566676,"kb_used_data":12316,"kb_used_omap":12,"kb_used_meta":554227,"kb_avail":754408044,"statfs":{"total":773094113280,"available":772513837056,"internally_reserved":0,"allocated":12611584,"data_stored":6099367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":567529010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.663917"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332086+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:15:39.332086+0000","last_peered":"2024-01-16T15:15:39.332086+0000","last_clean":"2024-01-16T15:15:39.332086+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:15:39.332086+0000","last_undegraded":"2024-01-16T15:15:39.332086+0000","last_fullsized":"2024-01-16T15:15:39.332086+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:01.053271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331443+0000","last_change":"2024-01-16T15:15:39.327644+0000","last_active":"2024-01-16T15:15:39.331443+0000","last_peered":"2024-01-16T15:15:39.331443+0000","last_clean":"2024-01-16T15:15:39.331443+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:15:39.331443+0000","last_undegraded":"2024-01-16T15:15:39.331443+0000","last_fullsized":"2024-01-16T15:15:39.331443+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:38.284829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330731+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:15:39.330731+0000","last_peered":"2024-01-16T15:15:39.330731+0000","last_clean":"2024-01-16T15:15:39.330731+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:15:39.330731+0000","last_undegraded":"2024-01-16T15:15:39.330731+0000","last_fullsized":"2024-01-16T15:15:39.330731+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:56.026622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330388+0000","last_change":"2024-01-16T15:15:39.320416+0000","last_active":"2024-01-16T15:15:39.330388+0000","last_peered":"2024-01-16T15:15:39.330388+0000","last_clean":"2024-01-16T15:15:39.330388+0000","last_became_active":"2024-01-16T15:15:39.317971+0000","last_became_peered":"2024-01-16T15:15:39.317971+0000","last_unstale":"2024-01-16T15:15:39.330388+0000","last_undegraded":"2024-01-16T15:15:39.330388+0000","last_fullsized":"2024-01-16T15:15:39.330388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:39:41.607736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326309+0000","last_change":"2024-01-16T15:15:39.321081+0000","last_active":"2024-01-16T15:15:39.326309+0000","last_peered":"2024-01-16T15:15:39.326309+0000","last_clean":"2024-01-16T15:15:39.326309+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:15:39.326309+0000","last_undegraded":"2024-01-16T15:15:39.326309+0000","last_fullsized":"2024-01-16T15:15:39.326309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:42.051788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331857+0000","last_change":"2024-01-16T15:15:39.327813+0000","last_active":"2024-01-16T15:15:39.331857+0000","last_peered":"2024-01-16T15:15:39.331857+0000","last_clean":"2024-01-16T15:15:39.331857+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:15:39.331857+0000","last_undegraded":"2024-01-16T15:15:39.331857+0000","last_fullsized":"2024-01-16T15:15:39.331857+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:52.806951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326306+0000","last_change":"2024-01-16T15:15:39.324707+0000","last_active":"2024-01-16T15:15:39.326306+0000","last_peered":"2024-01-16T15:15:39.326306+0000","last_clean":"2024-01-16T15:15:39.326306+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:15:39.326306+0000","last_undegraded":"2024-01-16T15:15:39.326306+0000","last_fullsized":"2024-01-16T15:15:39.326306+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:32.958658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331955+0000","last_change":"2024-01-16T15:15:39.328547+0000","last_active":"2024-01-16T15:15:39.331955+0000","last_peered":"2024-01-16T15:15:39.331955+0000","last_clean":"2024-01-16T15:15:39.331955+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:15:39.331955+0000","last_undegraded":"2024-01-16T15:15:39.331955+0000","last_fullsized":"2024-01-16T15:15:39.331955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:35.351694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330368+0000","last_change":"2024-01-16T15:15:39.320236+0000","last_active":"2024-01-16T15:15:39.330368+0000","last_peered":"2024-01-16T15:15:39.330368+0000","last_clean":"2024-01-16T15:15:39.330368+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:15:39.330368+0000","last_undegraded":"2024-01-16T15:15:39.330368+0000","last_fullsized":"2024-01-16T15:15:39.330368+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:49.706005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330878+0000","last_change":"2024-01-16T15:15:39.326528+0000","last_active":"2024-01-16T15:15:39.330878+0000","last_peered":"2024-01-16T15:15:39.330878+0000","last_clean":"2024-01-16T15:15:39.330878+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:15:39.330878+0000","last_undegraded":"2024-01-16T15:15:39.330878+0000","last_fullsized":"2024-01-16T15:15:39.330878+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:28.634370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.327773+0000","last_change":"2024-01-16T15:15:39.325379+0000","last_active":"2024-01-16T15:15:39.327773+0000","last_peered":"2024-01-16T15:15:39.327773+0000","last_clean":"2024-01-16T15:15:39.327773+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:15:39.327773+0000","last_undegraded":"2024-01-16T15:15:39.327773+0000","last_fullsized":"2024-01-16T15:15:39.327773+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:36.770903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330299+0000","last_change":"2024-01-16T15:15:39.317978+0000","last_active":"2024-01-16T15:15:39.330299+0000","last_peered":"2024-01-16T15:15:39.330299+0000","last_clean":"2024-01-16T15:15:39.330299+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:15:39.330299+0000","last_undegraded":"2024-01-16T15:15:39.330299+0000","last_fullsized":"2024-01-16T15:15:39.330299+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:55.882282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332865+0000","last_change":"2024-01-16T15:15:39.329570+0000","last_active":"2024-01-16T15:15:39.332865+0000","last_peered":"2024-01-16T15:15:39.332865+0000","last_clean":"2024-01-16T15:15:39.332865+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:15:39.332865+0000","last_undegraded":"2024-01-16T15:15:39.332865+0000","last_fullsized":"2024-01-16T15:15:39.332865+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:39.747172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328318+0000","last_change":"2024-01-16T15:15:39.325613+0000","last_active":"2024-01-16T15:15:39.328318+0000","last_peered":"2024-01-16T15:15:39.328318+0000","last_clean":"2024-01-16T15:15:39.328318+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:15:39.328318+0000","last_undegraded":"2024-01-16T15:15:39.328318+0000","last_fullsized":"2024-01-16T15:15:39.328318+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:58.758941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.336943+0000","last_change":"2024-01-16T15:15:39.333121+0000","last_active":"2024-01-16T15:15:39.336943+0000","last_peered":"2024-01-16T15:15:39.336943+0000","last_clean":"2024-01-16T15:15:39.336943+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:15:39.336943+0000","last_undegraded":"2024-01-16T15:15:39.336943+0000","last_fullsized":"2024-01-16T15:15:39.336943+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:15.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.315001+0000","last_change":"2024-01-16T15:15:39.313940+0000","last_active":"2024-01-16T15:15:39.315001+0000","last_peered":"2024-01-16T15:15:39.315001+0000","last_clean":"2024-01-16T15:15:39.315001+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:15:39.315001+0000","last_undegraded":"2024-01-16T15:15:39.315001+0000","last_fullsized":"2024-01-16T15:15:39.315001+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:56.164840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331698+0000","last_change":"2024-01-16T15:15:39.329528+0000","last_active":"2024-01-16T15:15:39.331698+0000","last_peered":"2024-01-16T15:15:39.331698+0000","last_clean":"2024-01-16T15:15:39.331698+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:15:39.331698+0000","last_undegraded":"2024-01-16T15:15:39.331698+0000","last_fullsized":"2024-01-16T15:15:39.331698+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333188+0000","last_change":"2024-01-16T15:15:39.330676+0000","last_active":"2024-01-16T15:15:39.333188+0000","last_peered":"2024-01-16T15:15:39.333188+0000","last_clean":"2024-01-16T15:15:39.333188+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:15:39.333188+0000","last_undegraded":"2024-01-16T15:15:39.333188+0000","last_fullsized":"2024-01-16T15:15:39.333188+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:49.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332871+0000","last_change":"2024-01-16T15:15:39.330437+0000","last_active":"2024-01-16T15:15:39.332871+0000","last_peered":"2024-01-16T15:15:39.332871+0000","last_clean":"2024-01-16T15:15:39.332871+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:15:39.332871+0000","last_undegraded":"2024-01-16T15:15:39.332871+0000","last_fullsized":"2024-01-16T15:15:39.332871+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:41.474484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.321853+0000","last_change":"2024-01-16T15:15:39.318689+0000","last_active":"2024-01-16T15:15:39.321853+0000","last_peered":"2024-01-16T15:15:39.321853+0000","last_clean":"2024-01-16T15:15:39.321853+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:15:39.321853+0000","last_undegraded":"2024-01-16T15:15:39.321853+0000","last_fullsized":"2024-01-16T15:15:39.321853+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:39:32.420035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.323628+0000","last_change":"2024-01-16T15:15:39.321055+0000","last_active":"2024-01-16T15:15:39.323628+0000","last_peered":"2024-01-16T15:15:39.323628+0000","last_clean":"2024-01-16T15:15:39.323628+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:15:39.323628+0000","last_undegraded":"2024-01-16T15:15:39.323628+0000","last_fullsized":"2024-01-16T15:15:39.323628+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.327999+0000","last_change":"2024-01-16T15:15:39.323290+0000","last_active":"2024-01-16T15:15:39.327999+0000","last_peered":"2024-01-16T15:15:39.327999+0000","last_clean":"2024-01-16T15:15:39.327999+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:15:39.327999+0000","last_undegraded":"2024-01-16T15:15:39.327999+0000","last_fullsized":"2024-01-16T15:15:39.327999+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:28.634370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.319883+0000","last_change":"2024-01-16T15:15:39.317974+0000","last_active":"2024-01-16T15:15:39.319883+0000","last_peered":"2024-01-16T15:15:39.319883+0000","last_clean":"2024-01-16T15:15:39.319883+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:15:39.319883+0000","last_undegraded":"2024-01-16T15:15:39.319883+0000","last_fullsized":"2024-01-16T15:15:39.319883+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:15.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328548+0000","last_change":"2024-01-16T15:15:39.324363+0000","last_active":"2024-01-16T15:15:39.328548+0000","last_peered":"2024-01-16T15:15:39.328548+0000","last_clean":"2024-01-16T15:15:39.328548+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:15:39.328548+0000","last_undegraded":"2024-01-16T15:15:39.328548+0000","last_fullsized":"2024-01-16T15:15:39.328548+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:18.050547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331679+0000","last_change":"2024-01-16T15:15:39.328896+0000","last_active":"2024-01-16T15:15:39.331679+0000","last_peered":"2024-01-16T15:15:39.331679+0000","last_clean":"2024-01-16T15:15:39.331679+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:15:39.331679+0000","last_undegraded":"2024-01-16T15:15:39.331679+0000","last_fullsized":"2024-01-16T15:15:39.331679+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:19.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326057+0000","last_change":"2024-01-16T15:15:39.317589+0000","last_active":"2024-01-16T15:15:39.326057+0000","last_peered":"2024-01-16T15:15:39.326057+0000","last_clean":"2024-01-16T15:15:39.326057+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:15:39.326057+0000","last_undegraded":"2024-01-16T15:15:39.326057+0000","last_fullsized":"2024-01-16T15:15:39.326057+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:06.562007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337670+0000","last_change":"2024-01-16T15:15:39.335455+0000","last_active":"2024-01-16T15:15:39.337670+0000","last_peered":"2024-01-16T15:15:39.337670+0000","last_clean":"2024-01-16T15:15:39.337670+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:15:39.337670+0000","last_undegraded":"2024-01-16T15:15:39.337670+0000","last_fullsized":"2024-01-16T15:15:39.337670+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.325678+0000","last_change":"2024-01-16T15:15:39.317292+0000","last_active":"2024-01-16T15:15:39.325678+0000","last_peered":"2024-01-16T15:15:39.325678+0000","last_clean":"2024-01-16T15:15:39.325678+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:15:39.325678+0000","last_undegraded":"2024-01-16T15:15:39.325678+0000","last_fullsized":"2024-01-16T15:15:39.325678+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:55.882282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.322773+0000","last_change":"2024-01-16T15:15:39.320340+0000","last_active":"2024-01-16T15:15:39.322773+0000","last_peered":"2024-01-16T15:15:39.322773+0000","last_clean":"2024-01-16T15:15:39.322773+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:15:39.322773+0000","last_undegraded":"2024-01-16T15:15:39.322773+0000","last_fullsized":"2024-01-16T15:15:39.322773+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:35:57.563013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.318994+0000","last_change":"2024-01-16T15:15:39.317544+0000","last_active":"2024-01-16T15:15:39.318994+0000","last_peered":"2024-01-16T15:15:39.318994+0000","last_clean":"2024-01-16T15:15:39.318994+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:15:39.318994+0000","last_undegraded":"2024-01-16T15:15:39.318994+0000","last_fullsized":"2024-01-16T15:15:39.318994+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:08.504738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.314764+0000","last_change":"2024-01-16T15:15:39.313788+0000","last_active":"2024-01-16T15:15:39.314764+0000","last_peered":"2024-01-16T15:15:39.314764+0000","last_clean":"2024-01-16T15:15:39.314764+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:15:39.314764+0000","last_undegraded":"2024-01-16T15:15:39.314764+0000","last_fullsized":"2024-01-16T15:15:39.314764+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:40.960949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326025+0000","last_change":"2024-01-16T15:15:39.321572+0000","last_active":"2024-01-16T15:15:39.326025+0000","last_peered":"2024-01-16T15:15:39.326025+0000","last_clean":"2024-01-16T15:15:39.326025+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:15:39.326025+0000","last_undegraded":"2024-01-16T15:15:39.326025+0000","last_fullsized":"2024-01-16T15:15:39.326025+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:20.421089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332430+0000","last_change":"2024-01-16T15:15:39.329833+0000","last_active":"2024-01-16T15:15:39.332430+0000","last_peered":"2024-01-16T15:15:39.332430+0000","last_clean":"2024-01-16T15:15:39.332430+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:15:39.332430+0000","last_undegraded":"2024-01-16T15:15:39.332430+0000","last_fullsized":"2024-01-16T15:15:39.332430+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:36.827801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'4","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.342972+0000","last_change":"2024-01-16T15:15:38.295673+0000","last_active":"2024-01-16T15:15:39.342972+0000","last_peered":"2024-01-16T15:15:39.342972+0000","last_clean":"2024-01-16T15:15:39.342972+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:15:39.342972+0000","last_undegraded":"2024-01-16T15:15:39.342972+0000","last_fullsized":"2024-01-16T15:15:39.342972+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333684+0000","last_change":"2024-01-16T15:15:39.329620+0000","last_active":"2024-01-16T15:15:39.333684+0000","last_peered":"2024-01-16T15:15:39.333684+0000","last_clean":"2024-01-16T15:15:39.333684+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:15:39.333684+0000","last_undegraded":"2024-01-16T15:15:39.333684+0000","last_fullsized":"2024-01-16T15:15:39.333684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:49:22.746780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.341114+0000","last_change":"2024-01-16T15:15:38.292769+0000","last_active":"2024-01-16T15:15:39.341114+0000","last_peered":"2024-01-16T15:15:39.341114+0000","last_clean":"2024-01-16T15:15:39.341114+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:15:39.341114+0000","last_undegraded":"2024-01-16T15:15:39.341114+0000","last_fullsized":"2024-01-16T15:15:39.341114+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:45.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.336878+0000","last_change":"2024-01-16T15:15:39.332870+0000","last_active":"2024-01-16T15:15:39.336878+0000","last_peered":"2024-01-16T15:15:39.336878+0000","last_clean":"2024-01-16T15:15:39.336878+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:15:39.336878+0000","last_undegraded":"2024-01-16T15:15:39.336878+0000","last_fullsized":"2024-01-16T15:15:39.336878+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:42.051788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.350874+0000","last_change":"2024-01-16T15:15:38.288495+0000","last_active":"2024-01-16T15:15:39.350874+0000","last_peered":"2024-01-16T15:15:39.350874+0000","last_clean":"2024-01-16T15:15:39.350874+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:15:39.350874+0000","last_undegraded":"2024-01-16T15:15:39.350874+0000","last_fullsized":"2024-01-16T15:15:39.350874+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:07.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332179+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:15:39.332179+0000","last_peered":"2024-01-16T15:15:39.332179+0000","last_clean":"2024-01-16T15:15:39.332179+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:15:39.332179+0000","last_undegraded":"2024-01-16T15:15:39.332179+0000","last_fullsized":"2024-01-16T15:15:39.332179+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:21.913468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.347018+0000","last_change":"2024-01-16T15:15:38.288324+0000","last_active":"2024-01-16T15:15:39.347018+0000","last_peered":"2024-01-16T15:15:39.347018+0000","last_clean":"2024-01-16T15:15:39.347018+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:15:39.347018+0000","last_undegraded":"2024-01-16T15:15:39.347018+0000","last_fullsized":"2024-01-16T15:15:39.347018+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:31.849193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330784+0000","last_change":"2024-01-16T15:15:39.321037+0000","last_active":"2024-01-16T15:15:39.330784+0000","last_peered":"2024-01-16T15:15:39.330784+0000","last_clean":"2024-01-16T15:15:39.330784+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:15:39.330784+0000","last_undegraded":"2024-01-16T15:15:39.330784+0000","last_fullsized":"2024-01-16T15:15:39.330784+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:04:44.476846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.343928+0000","last_change":"2024-01-16T15:15:38.288226+0000","last_active":"2024-01-16T15:15:39.343928+0000","last_peered":"2024-01-16T15:15:39.343928+0000","last_clean":"2024-01-16T15:15:39.343928+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:15:39.343928+0000","last_undegraded":"2024-01-16T15:15:39.343928+0000","last_fullsized":"2024-01-16T15:15:39.343928+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:41.040743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332266+0000","last_change":"2024-01-16T15:15:39.330201+0000","last_active":"2024-01-16T15:15:39.332266+0000","last_peered":"2024-01-16T15:15:39.332266+0000","last_clean":"2024-01-16T15:15:39.332266+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:15:39.332266+0000","last_undegraded":"2024-01-16T15:15:39.332266+0000","last_fullsized":"2024-01-16T15:15:39.332266+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:40.033000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.342620+0000","last_change":"2024-01-16T15:15:38.292721+0000","last_active":"2024-01-16T15:15:39.342620+0000","last_peered":"2024-01-16T15:15:39.342620+0000","last_clean":"2024-01-16T15:15:39.342620+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:15:39.342620+0000","last_undegraded":"2024-01-16T15:15:39.342620+0000","last_fullsized":"2024-01-16T15:15:39.342620+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:32.177991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.329937+0000","last_change":"2024-01-16T15:15:39.322497+0000","last_active":"2024-01-16T15:15:39.329937+0000","last_peered":"2024-01-16T15:15:39.329937+0000","last_clean":"2024-01-16T15:15:39.329937+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:15:39.329937+0000","last_undegraded":"2024-01-16T15:15:39.329937+0000","last_fullsized":"2024-01-16T15:15:39.329937+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:57.002611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.294502+0000","last_change":"2024-01-16T15:15:38.276895+0000","last_active":"2024-01-16T15:15:39.294502+0000","last_peered":"2024-01-16T15:15:39.294502+0000","last_clean":"2024-01-16T15:15:39.294502+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:15:39.294502+0000","last_undegraded":"2024-01-16T15:15:39.294502+0000","last_fullsized":"2024-01-16T15:15:39.294502+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326325+0000","last_change":"2024-01-16T15:15:39.325224+0000","last_active":"2024-01-16T15:15:39.326325+0000","last_peered":"2024-01-16T15:15:39.326325+0000","last_clean":"2024-01-16T15:15:39.326325+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:15:39.326325+0000","last_undegraded":"2024-01-16T15:15:39.326325+0000","last_fullsized":"2024-01-16T15:15:39.326325+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:32.863366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301149+0000","last_change":"2024-01-16T15:15:38.297269+0000","last_active":"2024-01-16T15:15:39.301149+0000","last_peered":"2024-01-16T15:15:39.301149+0000","last_clean":"2024-01-16T15:15:39.301149+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:15:39.301149+0000","last_undegraded":"2024-01-16T15:15:39.301149+0000","last_fullsized":"2024-01-16T15:15:39.301149+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:02.166771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.322612+0000","last_change":"2024-01-16T15:15:39.320446+0000","last_active":"2024-01-16T15:15:39.322612+0000","last_peered":"2024-01-16T15:15:39.322612+0000","last_clean":"2024-01-16T15:15:39.322612+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:15:39.322612+0000","last_undegraded":"2024-01-16T15:15:39.322612+0000","last_fullsized":"2024-01-16T15:15:39.322612+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:36.827801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.292107+0000","last_change":"2024-01-16T15:15:38.287453+0000","last_active":"2024-01-16T15:15:39.292107+0000","last_peered":"2024-01-16T15:15:39.292107+0000","last_clean":"2024-01-16T15:15:39.292107+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:15:39.292107+0000","last_undegraded":"2024-01-16T15:15:39.292107+0000","last_fullsized":"2024-01-16T15:15:39.292107+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.334533+0000","last_change":"2024-01-16T15:15:39.333810+0000","last_active":"2024-01-16T15:15:39.334533+0000","last_peered":"2024-01-16T15:15:39.334533+0000","last_clean":"2024-01-16T15:15:39.334533+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:15:39.334533+0000","last_undegraded":"2024-01-16T15:15:39.334533+0000","last_fullsized":"2024-01-16T15:15:39.334533+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:18.350475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.318263+0000","last_change":"2024-01-16T15:15:38.293583+0000","last_active":"2024-01-16T15:15:39.318263+0000","last_peered":"2024-01-16T15:15:39.318263+0000","last_clean":"2024-01-16T15:15:39.318263+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:15:39.318263+0000","last_undegraded":"2024-01-16T15:15:39.318263+0000","last_fullsized":"2024-01-16T15:15:39.318263+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:48.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.325695+0000","last_change":"2024-01-16T15:15:39.319368+0000","last_active":"2024-01-16T15:15:39.325695+0000","last_peered":"2024-01-16T15:15:39.325695+0000","last_clean":"2024-01-16T15:15:39.325695+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:15:39.325695+0000","last_undegraded":"2024-01-16T15:15:39.325695+0000","last_fullsized":"2024-01-16T15:15:39.325695+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:29.630325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'3","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.350395+0000","last_change":"2024-01-16T15:15:38.295750+0000","last_active":"2024-01-16T15:15:39.350395+0000","last_peered":"2024-01-16T15:15:39.350395+0000","last_clean":"2024-01-16T15:15:39.350395+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:15:39.350395+0000","last_undegraded":"2024-01-16T15:15:39.350395+0000","last_fullsized":"2024-01-16T15:15:39.350395+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:17.036375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.317322+0000","last_change":"2024-01-16T15:15:39.312858+0000","last_active":"2024-01-16T15:15:39.317322+0000","last_peered":"2024-01-16T15:15:39.317322+0000","last_clean":"2024-01-16T15:15:39.317322+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:15:39.317322+0000","last_undegraded":"2024-01-16T15:15:39.317322+0000","last_fullsized":"2024-01-16T15:15:39.317322+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:42.051788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'6","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.352515+0000","last_change":"2024-01-16T15:15:38.296732+0000","last_active":"2024-01-16T15:15:39.352515+0000","last_peered":"2024-01-16T15:15:39.352515+0000","last_clean":"2024-01-16T15:15:39.352515+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:15:39.352515+0000","last_undegraded":"2024-01-16T15:15:39.352515+0000","last_fullsized":"2024-01-16T15:15:39.352515+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.316975+0000","last_change":"2024-01-16T15:15:39.312723+0000","last_active":"2024-01-16T15:15:39.316975+0000","last_peered":"2024-01-16T15:15:39.316975+0000","last_clean":"2024-01-16T15:15:39.316975+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:15:39.316975+0000","last_undegraded":"2024-01-16T15:15:39.316975+0000","last_fullsized":"2024-01-16T15:15:39.316975+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:39:41.607736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.346109+0000","last_change":"2024-01-16T15:15:38.280236+0000","last_active":"2024-01-16T15:15:39.346109+0000","last_peered":"2024-01-16T15:15:39.346109+0000","last_clean":"2024-01-16T15:15:39.346109+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:15:39.346109+0000","last_undegraded":"2024-01-16T15:15:39.346109+0000","last_fullsized":"2024-01-16T15:15:39.346109+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:14.715313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328110+0000","last_change":"2024-01-16T15:15:39.324127+0000","last_active":"2024-01-16T15:15:39.328110+0000","last_peered":"2024-01-16T15:15:39.328110+0000","last_clean":"2024-01-16T15:15:39.328110+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:15:39.328110+0000","last_undegraded":"2024-01-16T15:15:39.328110+0000","last_fullsized":"2024-01-16T15:15:39.328110+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:06.562007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.317399+0000","last_change":"2024-01-16T15:15:38.295857+0000","last_active":"2024-01-16T15:15:39.317399+0000","last_peered":"2024-01-16T15:15:39.317399+0000","last_clean":"2024-01-16T15:15:39.317399+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:15:39.317399+0000","last_undegraded":"2024-01-16T15:15:39.317399+0000","last_fullsized":"2024-01-16T15:15:39.317399+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:31.063560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326208+0000","last_change":"2024-01-16T15:15:39.319495+0000","last_active":"2024-01-16T15:15:39.326208+0000","last_peered":"2024-01-16T15:15:39.326208+0000","last_clean":"2024-01-16T15:15:39.326208+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:15:39.326208+0000","last_undegraded":"2024-01-16T15:15:39.326208+0000","last_fullsized":"2024-01-16T15:15:39.326208+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:02.542534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'8","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.351392+0000","last_change":"2024-01-16T15:15:38.293904+0000","last_active":"2024-01-16T15:15:39.351392+0000","last_peered":"2024-01-16T15:15:39.351392+0000","last_clean":"2024-01-16T15:15:39.351392+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:15:39.351392+0000","last_undegraded":"2024-01-16T15:15:39.351392+0000","last_fullsized":"2024-01-16T15:15:39.351392+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.317862+0000","last_change":"2024-01-16T15:15:39.314200+0000","last_active":"2024-01-16T15:15:39.317862+0000","last_peered":"2024-01-16T15:15:39.317862+0000","last_clean":"2024-01-16T15:15:39.317862+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:15:39.317862+0000","last_undegraded":"2024-01-16T15:15:39.317862+0000","last_fullsized":"2024-01-16T15:15:39.317862+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:36.827801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.345442+0000","last_change":"2024-01-16T15:15:38.296521+0000","last_active":"2024-01-16T15:15:39.345442+0000","last_peered":"2024-01-16T15:15:39.345442+0000","last_clean":"2024-01-16T15:15:39.345442+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:15:39.345442+0000","last_undegraded":"2024-01-16T15:15:39.345442+0000","last_fullsized":"2024-01-16T15:15:39.345442+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:24.030528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337557+0000","last_change":"2024-01-16T15:15:39.334617+0000","last_active":"2024-01-16T15:15:39.337557+0000","last_peered":"2024-01-16T15:15:39.337557+0000","last_clean":"2024-01-16T15:15:39.337557+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:15:39.337557+0000","last_undegraded":"2024-01-16T15:15:39.337557+0000","last_fullsized":"2024-01-16T15:15:39.337557+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.347180+0000","last_change":"2024-01-16T15:15:38.289440+0000","last_active":"2024-01-16T15:15:39.347180+0000","last_peered":"2024-01-16T15:15:39.347180+0000","last_clean":"2024-01-16T15:15:39.347180+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:15:39.347180+0000","last_undegraded":"2024-01-16T15:15:39.347180+0000","last_fullsized":"2024-01-16T15:15:39.347180+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333320+0000","last_change":"2024-01-16T15:15:39.332704+0000","last_active":"2024-01-16T15:15:39.333320+0000","last_peered":"2024-01-16T15:15:39.333320+0000","last_clean":"2024-01-16T15:15:39.333320+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:15:39.333320+0000","last_undegraded":"2024-01-16T15:15:39.333320+0000","last_fullsized":"2024-01-16T15:15:39.333320+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:52.162066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.344109+0000","last_change":"2024-01-16T15:15:38.291309+0000","last_active":"2024-01-16T15:15:39.344109+0000","last_peered":"2024-01-16T15:15:39.344109+0000","last_clean":"2024-01-16T15:15:39.344109+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:15:39.344109+0000","last_undegraded":"2024-01-16T15:15:39.344109+0000","last_fullsized":"2024-01-16T15:15:39.344109+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:35.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330751+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:15:39.330751+0000","last_peered":"2024-01-16T15:15:39.330751+0000","last_clean":"2024-01-16T15:15:39.330751+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:15:39.330751+0000","last_undegraded":"2024-01-16T15:15:39.330751+0000","last_fullsized":"2024-01-16T15:15:39.330751+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:45.950432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301594+0000","last_change":"2024-01-16T15:15:38.298667+0000","last_active":"2024-01-16T15:15:39.301594+0000","last_peered":"2024-01-16T15:15:39.301594+0000","last_clean":"2024-01-16T15:15:39.301594+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:15:39.301594+0000","last_undegraded":"2024-01-16T15:15:39.301594+0000","last_fullsized":"2024-01-16T15:15:39.301594+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328339+0000","last_change":"2024-01-16T15:15:39.326224+0000","last_active":"2024-01-16T15:15:39.328339+0000","last_peered":"2024-01-16T15:15:39.328339+0000","last_clean":"2024-01-16T15:15:39.328339+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:15:39.328339+0000","last_undegraded":"2024-01-16T15:15:39.328339+0000","last_fullsized":"2024-01-16T15:15:39.328339+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:02.559083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.347887+0000","last_change":"2024-01-16T15:15:38.278400+0000","last_active":"2024-01-16T15:15:39.347887+0000","last_peered":"2024-01-16T15:15:39.347887+0000","last_clean":"2024-01-16T15:15:39.347887+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:15:39.347887+0000","last_undegraded":"2024-01-16T15:15:39.347887+0000","last_fullsized":"2024-01-16T15:15:39.347887+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:37.301739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330809+0000","last_change":"2024-01-16T15:15:39.325408+0000","last_active":"2024-01-16T15:15:39.330809+0000","last_peered":"2024-01-16T15:15:39.330809+0000","last_clean":"2024-01-16T15:15:39.330809+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:15:39.330809+0000","last_undegraded":"2024-01-16T15:15:39.330809+0000","last_fullsized":"2024-01-16T15:15:39.330809+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:10.305933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.292084+0000","last_change":"2024-01-16T15:15:38.287926+0000","last_active":"2024-01-16T15:15:39.292084+0000","last_peered":"2024-01-16T15:15:39.292084+0000","last_clean":"2024-01-16T15:15:39.292084+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:15:39.292084+0000","last_undegraded":"2024-01-16T15:15:39.292084+0000","last_fullsized":"2024-01-16T15:15:39.292084+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:31.944486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332823+0000","last_change":"2024-01-16T15:15:39.330480+0000","last_active":"2024-01-16T15:15:39.332823+0000","last_peered":"2024-01-16T15:15:39.332823+0000","last_clean":"2024-01-16T15:15:39.332823+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:15:39.332823+0000","last_undegraded":"2024-01-16T15:15:39.332823+0000","last_fullsized":"2024-01-16T15:15:39.332823+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:42.153458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.323111+0000","last_change":"2024-01-16T15:15:38.276702+0000","last_active":"2024-01-16T15:15:39.323111+0000","last_peered":"2024-01-16T15:15:39.323111+0000","last_clean":"2024-01-16T15:15:39.323111+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:15:39.323111+0000","last_undegraded":"2024-01-16T15:15:39.323111+0000","last_fullsized":"2024-01-16T15:15:39.323111+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.325460+0000","last_change":"2024-01-16T15:15:39.316268+0000","last_active":"2024-01-16T15:15:39.325460+0000","last_peered":"2024-01-16T15:15:39.325460+0000","last_clean":"2024-01-16T15:15:39.325460+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:15:39.325460+0000","last_undegraded":"2024-01-16T15:15:39.325460+0000","last_fullsized":"2024-01-16T15:15:39.325460+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:04:44.476846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301502+0000","last_change":"2024-01-16T15:15:38.297590+0000","last_active":"2024-01-16T15:15:39.301502+0000","last_peered":"2024-01-16T15:15:39.301502+0000","last_clean":"2024-01-16T15:15:39.301502+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:15:39.301502+0000","last_undegraded":"2024-01-16T15:15:39.301502+0000","last_fullsized":"2024-01-16T15:15:39.301502+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:18.917382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.334063+0000","last_change":"2024-01-16T15:15:39.333001+0000","last_active":"2024-01-16T15:15:39.334063+0000","last_peered":"2024-01-16T15:15:39.334063+0000","last_clean":"2024-01-16T15:15:39.334063+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:15:39.334063+0000","last_undegraded":"2024-01-16T15:15:39.334063+0000","last_fullsized":"2024-01-16T15:15:39.334063+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:41.770690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.345982+0000","last_change":"2024-01-16T15:15:38.291784+0000","last_active":"2024-01-16T15:15:39.345982+0000","last_peered":"2024-01-16T15:15:39.345982+0000","last_clean":"2024-01-16T15:15:39.345982+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:15:39.345982+0000","last_undegraded":"2024-01-16T15:15:39.345982+0000","last_fullsized":"2024-01-16T15:15:39.345982+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330348+0000","last_change":"2024-01-16T15:15:39.320032+0000","last_active":"2024-01-16T15:15:39.330348+0000","last_peered":"2024-01-16T15:15:39.330348+0000","last_clean":"2024-01-16T15:15:39.330348+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:15:39.330348+0000","last_undegraded":"2024-01-16T15:15:39.330348+0000","last_fullsized":"2024-01-16T15:15:39.330348+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:33.192164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.324013+0000","last_change":"2024-01-16T15:15:38.295797+0000","last_active":"2024-01-16T15:15:39.324013+0000","last_peered":"2024-01-16T15:15:39.324013+0000","last_clean":"2024-01-16T15:15:39.324013+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:15:39.324013+0000","last_undegraded":"2024-01-16T15:15:39.324013+0000","last_fullsized":"2024-01-16T15:15:39.324013+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:49.218031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337264+0000","last_change":"2024-01-16T15:15:39.335229+0000","last_active":"2024-01-16T15:15:39.337264+0000","last_peered":"2024-01-16T15:15:39.337264+0000","last_clean":"2024-01-16T15:15:39.337264+0000","last_became_active":"2024-01-16T15:15:39.333516+0000","last_became_peered":"2024-01-16T15:15:39.333516+0000","last_unstale":"2024-01-16T15:15:39.337264+0000","last_undegraded":"2024-01-16T15:15:39.337264+0000","last_fullsized":"2024-01-16T15:15:39.337264+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:02.542534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.303295+0000","last_change":"2024-01-16T15:15:38.297697+0000","last_active":"2024-01-16T15:15:39.303295+0000","last_peered":"2024-01-16T15:15:39.303295+0000","last_clean":"2024-01-16T15:15:39.303295+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:15:39.303295+0000","last_undegraded":"2024-01-16T15:15:39.303295+0000","last_fullsized":"2024-01-16T15:15:39.303295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:28.616152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333744+0000","last_change":"2024-01-16T15:15:39.331210+0000","last_active":"2024-01-16T15:15:39.333744+0000","last_peered":"2024-01-16T15:15:39.333744+0000","last_clean":"2024-01-16T15:15:39.333744+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:15:39.333744+0000","last_undegraded":"2024-01-16T15:15:39.333744+0000","last_fullsized":"2024-01-16T15:15:39.333744+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:24.608493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.318562+0000","last_change":"2024-01-16T15:15:38.287424+0000","last_active":"2024-01-16T15:15:39.318562+0000","last_peered":"2024-01-16T15:15:39.318562+0000","last_clean":"2024-01-16T15:15:39.318562+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:15:39.318562+0000","last_undegraded":"2024-01-16T15:15:39.318562+0000","last_fullsized":"2024-01-16T15:15:39.318562+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:00.039099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.316776+0000","last_change":"2024-01-16T15:15:39.313583+0000","last_active":"2024-01-16T15:15:39.316776+0000","last_peered":"2024-01-16T15:15:39.316776+0000","last_clean":"2024-01-16T15:15:39.316776+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:15:39.316776+0000","last_undegraded":"2024-01-16T15:15:39.316776+0000","last_fullsized":"2024-01-16T15:15:39.316776+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:38.315912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301157+0000","last_change":"2024-01-16T15:15:38.290534+0000","last_active":"2024-01-16T15:15:39.301157+0000","last_peered":"2024-01-16T15:15:39.301157+0000","last_clean":"2024-01-16T15:15:39.301157+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:15:39.301157+0000","last_undegraded":"2024-01-16T15:15:39.301157+0000","last_fullsized":"2024-01-16T15:15:39.301157+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.314637+0000","last_change":"2024-01-16T15:15:39.305386+0000","last_active":"2024-01-16T15:15:39.314637+0000","last_peered":"2024-01-16T15:15:39.314637+0000","last_clean":"2024-01-16T15:15:39.314637+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:15:39.314637+0000","last_undegraded":"2024-01-16T15:15:39.314637+0000","last_fullsized":"2024-01-16T15:15:39.314637+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.295093+0000","last_change":"2024-01-16T15:15:38.287042+0000","last_active":"2024-01-16T15:15:39.295093+0000","last_peered":"2024-01-16T15:15:39.295093+0000","last_clean":"2024-01-16T15:15:39.295093+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:15:39.295093+0000","last_undegraded":"2024-01-16T15:15:39.295093+0000","last_fullsized":"2024-01-16T15:15:39.295093+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:50.656301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332800+0000","last_change":"2024-01-16T15:15:39.330274+0000","last_active":"2024-01-16T15:15:39.332800+0000","last_peered":"2024-01-16T15:15:39.332800+0000","last_clean":"2024-01-16T15:15:39.332800+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:15:39.332800+0000","last_undegraded":"2024-01-16T15:15:39.332800+0000","last_fullsized":"2024-01-16T15:15:39.332800+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:11.903533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.291887+0000","last_change":"2024-01-16T15:15:38.281868+0000","last_active":"2024-01-16T15:15:39.291887+0000","last_peered":"2024-01-16T15:15:39.291887+0000","last_clean":"2024-01-16T15:15:39.291887+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:15:39.291887+0000","last_undegraded":"2024-01-16T15:15:39.291887+0000","last_fullsized":"2024-01-16T15:15:39.291887+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337252+0000","last_change":"2024-01-16T15:15:39.333842+0000","last_active":"2024-01-16T15:15:39.337252+0000","last_peered":"2024-01-16T15:15:39.337252+0000","last_clean":"2024-01-16T15:15:39.337252+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:15:39.337252+0000","last_undegraded":"2024-01-16T15:15:39.337252+0000","last_fullsized":"2024-01-16T15:15:39.337252+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:47.137989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301875+0000","last_change":"2024-01-16T15:15:38.297094+0000","last_active":"2024-01-16T15:15:39.301875+0000","last_peered":"2024-01-16T15:15:39.301875+0000","last_clean":"2024-01-16T15:15:39.301875+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:15:39.301875+0000","last_undegraded":"2024-01-16T15:15:39.301875+0000","last_fullsized":"2024-01-16T15:15:39.301875+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:48.520471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328755+0000","last_change":"2024-01-16T15:15:39.324548+0000","last_active":"2024-01-16T15:15:39.328755+0000","last_peered":"2024-01-16T15:15:39.328755+0000","last_clean":"2024-01-16T15:15:39.328755+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:15:39.328755+0000","last_undegraded":"2024-01-16T15:15:39.328755+0000","last_fullsized":"2024-01-16T15:15:39.328755+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:49.534643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.304750+0000","last_change":"2024-01-16T15:15:38.299858+0000","last_active":"2024-01-16T15:15:39.304750+0000","last_peered":"2024-01-16T15:15:39.304750+0000","last_clean":"2024-01-16T15:15:39.304750+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:15:39.304750+0000","last_undegraded":"2024-01-16T15:15:39.304750+0000","last_fullsized":"2024-01-16T15:15:39.304750+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":257,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-16T15:15:38.267237+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:15:38.267237+0000","last_peered":"2024-01-16T15:15:38.267237+0000","last_clean":"2024-01-16T15:15:38.267237+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:15:38.267237+0000","last_undegraded":"2024-01-16T15:15:38.267237+0000","last_fullsized":"2024-01-16T15:15:38.267237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73332,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298508,"statfs":{"total":96636764160,"available":96561672192,"internally_reserved":0,"allocated":2007040,"data_stored":1200437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.3400000000000001,"15min":1.538},"min":{"1min":0.68200000000000005,"5min":0.57799999999999996,"15min":0.37},"max":{"1min":1.3700000000000001,"5min":13.566000000000001,"15min":19.603000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.3899999999999999},"min":{"1min":0.73499999999999999,"5min":0.63600000000000001,"15min":0.49299999999999999},"max":{"1min":1.4710000000000001,"5min":10.728,"15min":24.015000000000001},"last":1.399}]},{"osd":1,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.248,"15min":1.48},"min":{"1min":0.68799999999999994,"5min":0.65600000000000003,"15min":0.40999999999999998},"max":{"1min":1.6060000000000001,"5min":13.375,"15min":19.527000000000001},"last":1.042},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1080000000000001,"15min":1.3939999999999999},"min":{"1min":0.70999999999999996,"5min":0.69499999999999995,"15min":0.443},"max":{"1min":1.5,"5min":1.867,"15min":24.07},"last":1.25}]},{"osd":2,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.2230000000000001,"15min":1.3660000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.46700000000000003},"max":{"1min":1.484,"5min":1.7909999999999999,"15min":24.108000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.1419999999999999,"15min":1.472},"min":{"1min":0.76900000000000002,"5min":0.76400000000000001,"15min":0.37},"max":{"1min":1.383,"5min":1.7210000000000001,"15min":24.806000000000001},"last":0.96999999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.3640000000000001,"15min":1.754},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.52600000000000002},"max":{"1min":1.52,"5min":13.657999999999999,"15min":23.741},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.375,"15min":1.647},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.626},"max":{"1min":1.748,"5min":1.806,"15min":24.143000000000001},"last":1.484}]},{"osd":4,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2430000000000001,"15min":1.641},"min":{"1min":0.79400000000000004,"5min":0.71999999999999997,"15min":0.45800000000000002},"max":{"1min":1.6659999999999999,"5min":1.776,"15min":25.637},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.55,"15min":1.8919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.44400000000000001},"max":{"1min":1.6319999999999999,"5min":13.782999999999999,"15min":25.594000000000001},"last":1.5600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.419,"15min":1.708},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.47999999999999998},"max":{"1min":1.8089999999999999,"5min":1.9670000000000001,"15min":29.567},"last":1.54},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4490000000000001,"15min":1.764},"min":{"1min":0.873,"5min":0.83999999999999997,"15min":0.55700000000000005},"max":{"1min":1.6970000000000001,"5min":1.9970000000000001,"15min":27.937000000000001},"last":1.514}]},{"osd":6,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.425,"15min":1.8640000000000001},"min":{"1min":0.67900000000000005,"5min":0.65800000000000003,"15min":0.50800000000000001},"max":{"1min":1.6890000000000001,"5min":2.0419999999999998,"15min":41.206000000000003},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.6399999999999999,"15min":2.1749999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.41699999999999998},"max":{"1min":1.758,"5min":13.725,"15min":39.932000000000002},"last":1.1399999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68608,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303232,"statfs":{"total":96636764160,"available":96566509568,"internally_reserved":0,"allocated":1429504,"data_stored":616914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.212,"15min":1.444},"min":{"1min":0.90900000000000003,"5min":0.628,"15min":0.52700000000000002},"max":{"1min":1.6759999999999999,"5min":3.9289999999999998,"15min":20.739000000000001},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.268,"15min":1.6220000000000001},"min":{"1min":0.84299999999999997,"5min":0.51400000000000001,"15min":0.498},"max":{"1min":1.538,"5min":4.6070000000000002,"15min":50.472000000000001},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.379,"15min":1.4199999999999999},"min":{"1min":0.89800000000000002,"5min":0.497,"15min":0.497},"max":{"1min":1.5589999999999999,"5min":10.518000000000001,"15min":10.629},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.272,"15min":1.45},"min":{"1min":0.871,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.9810000000000001,"5min":4.0339999999999998,"15min":9.6880000000000006},"last":1.444}]},{"osd":2,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.288,"15min":1.619},"min":{"1min":0.98499999999999999,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":2.1960000000000002,"5min":4.7140000000000004,"15min":50.643999999999998},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.6299999999999999,"15min":1.5900000000000001},"min":{"1min":0.90200000000000002,"5min":0.66000000000000003,"15min":0.59399999999999997},"max":{"1min":1.738,"5min":15.282,"15min":15.282},"last":1.476}]},{"osd":3,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.419,"15min":1.597},"min":{"1min":0.97499999999999998,"5min":0.64100000000000001,"15min":0.54900000000000004},"max":{"1min":1.8160000000000001,"5min":4.1109999999999998,"15min":15.675000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.458,"15min":1.7609999999999999},"min":{"1min":1.075,"5min":0.60099999999999998,"15min":0.52600000000000002},"max":{"1min":1.657,"5min":4.8330000000000002,"15min":50.561},"last":1.587}]},{"osd":4,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.4510000000000001,"15min":1.8089999999999999},"min":{"1min":0.92800000000000005,"5min":0.63200000000000001,"15min":0.46300000000000002},"max":{"1min":1.6779999999999999,"5min":5.4669999999999996,"15min":47.488},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.46,"15min":1.893},"min":{"1min":0.92400000000000004,"5min":0.41199999999999998,"15min":0.40400000000000003},"max":{"1min":1.6859999999999999,"5min":5.2350000000000003,"15min":50.381},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.5629999999999999,"15min":1.5509999999999999},"min":{"1min":1.022,"5min":0.436,"15min":0.436},"max":{"1min":1.633,"5min":10.388999999999999,"15min":10.654},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.613,"15min":1.8009999999999999},"min":{"1min":1.173,"5min":0.5,"15min":0.42799999999999999},"max":{"1min":2.1579999999999999,"5min":4.5449999999999999,"15min":18.699000000000002},"last":1.5089999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.667,"15min":2.0779999999999998},"min":{"1min":1.139,"5min":0.60599999999999998,"15min":0.44900000000000001},"max":{"1min":2.0449999999999999,"5min":5.9420000000000002,"15min":52.912999999999997},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.7070000000000001,"15min":2.3050000000000002},"min":{"1min":1.028,"5min":0.58499999999999996,"15min":0.495},"max":{"1min":2.3260000000000001,"5min":10.659000000000001,"15min":26.413},"last":1.377}]}]},{"osd":5,"up_from":14,"seq":60129542370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64460,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307380,"statfs":{"total":96636764160,"available":96570757120,"internally_reserved":0,"allocated":1433600,"data_stored":617448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1839999999999999,"15min":1.3600000000000001},"min":{"1min":0.80400000000000005,"5min":0.77700000000000002,"15min":0.38900000000000001},"max":{"1min":1.4610000000000001,"5min":2.222,"15min":13.938000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.248,"15min":1.4159999999999999},"min":{"1min":0.875,"5min":0.71999999999999997,"15min":0.42299999999999999},"max":{"1min":1.5569999999999999,"5min":5.4039999999999999,"15min":16.896000000000001},"last":1.093}]},{"osd":1,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.3240000000000001,"15min":1.49},"min":{"1min":0.80200000000000005,"5min":0.76000000000000001,"15min":0.50700000000000001},"max":{"1min":1.643,"5min":8.1820000000000004,"15min":15.500999999999999},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.179,"15min":1.369},"min":{"1min":0.73599999999999999,"5min":0.73099999999999998,"15min":0.46300000000000002},"max":{"1min":1.4810000000000001,"5min":2.2970000000000002,"15min":35.328000000000003},"last":1.4079999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2490000000000001,"15min":1.4630000000000001},"min":{"1min":0.96699999999999997,"5min":0.75900000000000001,"15min":0.46200000000000002},"max":{"1min":1.6799999999999999,"5min":3.0110000000000001,"15min":35.398000000000003},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.276,"15min":1.4219999999999999},"min":{"1min":1.056,"5min":0.75,"15min":0.38900000000000001},"max":{"1min":1.5329999999999999,"5min":2.5550000000000002,"15min":15.4},"last":1.2150000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.468,"15min":1.79},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.50900000000000001},"max":{"1min":1.6319999999999999,"5min":8.0809999999999995,"15min":24.254000000000001},"last":1.649},{"interface":"front","average":{"1min":1.401,"5min":1.4350000000000001,"15min":1.776},"min":{"1min":0.98199999999999998,"5min":0.89600000000000002,"15min":0.57399999999999995},"max":{"1min":1.6200000000000001,"5min":3.1619999999999999,"15min":35.268000000000001},"last":1.468}]},{"osd":4,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.2969999999999999,"15min":1.639},"min":{"1min":0.89200000000000002,"5min":0.74299999999999999,"15min":0.42199999999999999},"max":{"1min":1.6259999999999999,"5min":3.2599999999999998,"15min":19.059000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4590000000000001,"15min":1.7829999999999999},"min":{"1min":0.91700000000000004,"5min":0.57399999999999995,"15min":0.42799999999999999},"max":{"1min":1.7609999999999999,"5min":8.3550000000000004,"15min":28.760000000000002},"last":1.526}]},{"osd":6,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.468,"15min":1.837},"min":{"1min":1.252,"5min":0.73899999999999999,"15min":0.53400000000000003},"max":{"1min":1.8700000000000001,"5min":3.3220000000000001,"15min":35.427999999999997},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.5569999999999999,"15min":1.8819999999999999},"min":{"1min":1.3819999999999999,"5min":0.89400000000000002,"15min":0.53100000000000003},"max":{"1min":1.7809999999999999,"5min":3.391,"15min":16.262},"last":1.4950000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.6259999999999999,"15min":1.996},"min":{"1min":1.228,"5min":0.67000000000000004,"15min":0.54300000000000004},"max":{"1min":1.9450000000000001,"5min":8.2509999999999994,"15min":29.672999999999998},"last":1.742},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.5580000000000001,"15min":1.76},"min":{"1min":1.202,"5min":0.71999999999999997,"15min":0.45900000000000002},"max":{"1min":1.9950000000000001,"5min":3.2229999999999999,"15min":35.363},"last":1.1559999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542371,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68596,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303244,"statfs":{"total":96636764160,"available":96566521856,"internally_reserved":0,"allocated":1413120,"data_stored":610069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.26,"15min":1.4390000000000001},"min":{"1min":0.81200000000000006,"5min":0.63300000000000001,"15min":0.52800000000000002},"max":{"1min":1.8460000000000001,"5min":3.3159999999999998,"15min":15.942},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.238,"15min":1.4059999999999999},"min":{"1min":0.746,"5min":0.68999999999999995,"15min":0.45400000000000001},"max":{"1min":1.5469999999999999,"5min":1.9350000000000001,"15min":16.219999999999999},"last":1.1519999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3280000000000001,"15min":1.486},"min":{"1min":0.93000000000000005,"5min":0.76300000000000001,"15min":0.54100000000000004},"max":{"1min":1.901,"5min":1.9590000000000001,"15min":16.280000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.2030000000000001,"15min":1.4650000000000001},"min":{"1min":0.85699999999999998,"5min":0.77900000000000003,"15min":0.60699999999999998},"max":{"1min":1.6120000000000001,"5min":3.411,"15min":15.888},"last":0.91600000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.288,"15min":1.4650000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":13.731999999999999,"15min":13.837},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.3680000000000001,"15min":1.593},"min":{"1min":1.091,"5min":0.84999999999999998,"15min":0.64800000000000002},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":17.175000000000001},"last":1.411}]},{"osd":3,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.335,"15min":1.6990000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.57599999999999996},"max":{"1min":1.6919999999999999,"5min":2.077,"15min":16.469000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.371,"5min":1.458,"15min":1.5429999999999999},"min":{"1min":1.0229999999999999,"5min":0.86399999999999999,"15min":0.55600000000000005},"max":{"1min":1.643,"5min":13.914,"15min":15.089},"last":1.496}]},{"osd":5,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.393,"15min":1.5940000000000001},"min":{"1min":0.95199999999999996,"5min":0.83799999999999997,"15min":0.47299999999999998},"max":{"1min":1.6779999999999999,"5min":10.211,"15min":18.347999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.4319999999999999,"15min":1.6040000000000001},"min":{"1min":0.91300000000000003,"5min":0.86399999999999999,"15min":0.45800000000000002},"max":{"1min":1.768,"5min":2.149,"15min":22.937000000000001},"last":1.456}]},{"osd":6,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.51,"15min":1.589},"min":{"1min":0.998,"5min":0.877,"15min":0.501},"max":{"1min":1.661,"5min":13.973000000000001,"15min":18.033000000000001},"last":1.647},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5449999999999999,"15min":1.8640000000000001},"min":{"1min":1.3089999999999999,"5min":1.099,"15min":0.51800000000000002},"max":{"1min":1.752,"5min":3.0979999999999999,"15min":20.939},"last":1.5820000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.5489999999999999,"15min":1.71},"min":{"1min":1.097,"5min":0.97699999999999998,"15min":0.44500000000000001},"max":{"1min":1.8899999999999999,"5min":2.3199999999999998,"15min":16.324999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.6699999999999999,"15min":1.873},"min":{"1min":1.395,"5min":1.048,"15min":0.52200000000000002},"max":{"1min":2.2040000000000002,"5min":13.852,"15min":20.277999999999999},"last":1.7170000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76944,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294896,"statfs":{"total":96636764160,"available":96557973504,"internally_reserved":0,"allocated":1441792,"data_stored":618526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.95999999999999996,"15min":0.98099999999999998},"min":{"1min":0.503,"5min":0.502,"15min":0.34300000000000003},"max":{"1min":1.2869999999999999,"5min":2.9129999999999998,"15min":5.3570000000000002},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1080000000000001,"15min":1.127},"min":{"1min":0.55200000000000005,"5min":0.42399999999999999,"15min":0.29699999999999999},"max":{"1min":1.4930000000000001,"5min":3.5859999999999999,"15min":12.292999999999999},"last":0.83099999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.96799999999999997,"15min":0.95499999999999996},"min":{"1min":0.65600000000000003,"5min":0.52200000000000002,"15min":0.309},"max":{"1min":1.2470000000000001,"5min":1.8380000000000001,"15min":4.0979999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1950000000000001,"15min":1.2010000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.315},"max":{"1min":1.5169999999999999,"5min":10.638,"15min":13.443},"last":0.75900000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.145,"15min":1.151},"min":{"1min":0.70599999999999996,"5min":0.48199999999999998,"15min":0.35999999999999999},"max":{"1min":1.5720000000000001,"5min":4.4850000000000003,"15min":4.4850000000000003},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.131,"5min":1.228,"15min":1.2370000000000001},"min":{"1min":0.625,"5min":0.55000000000000004,"15min":0.35299999999999998},"max":{"1min":1.617,"5min":2.129,"15min":5.5110000000000001},"last":0.95999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.264,"15min":1.3859999999999999},"min":{"1min":0.79400000000000004,"5min":0.69899999999999995,"15min":0.39300000000000002},"max":{"1min":1.6899999999999999,"5min":4,"15min":13.292},"last":1.661},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.3600000000000001,"15min":1.4750000000000001},"min":{"1min":0.75,"5min":0.69499999999999995,"15min":0.52100000000000002},"max":{"1min":1.625,"5min":3.1150000000000002,"15min":11.068},"last":1.488}]},{"osd":5,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.4219999999999999,"15min":1.5780000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.48499999999999999},"max":{"1min":1.649,"5min":3.0019999999999998,"15min":17.096},"last":1.302},{"interface":"front","average":{"1min":1.339,"5min":1.419,"15min":1.5549999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.42099999999999999},"max":{"1min":1.8420000000000001,"5min":2.2639999999999998,"15min":12.106},"last":1.3560000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.3959999999999999,"15min":1.754},"min":{"1min":0.749,"5min":0.749,"15min":0.41499999999999998},"max":{"1min":1.8340000000000001,"5min":3.8239999999999998,"15min":43.609999999999999},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.4670000000000001,"15min":1.6919999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.54900000000000004},"max":{"1min":1.5860000000000001,"5min":2.7799999999999998,"15min":38.784999999999997},"last":1.5880000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.5620000000000001,"15min":1.732},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.53700000000000003},"max":{"1min":1.9570000000000001,"5min":7.8899999999999997,"15min":25.071000000000002},"last":1.044},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4830000000000001,"15min":1.718},"min":{"1min":0.95599999999999996,"5min":0.87,"15min":0.435},"max":{"1min":1.9370000000000001,"5min":4.2880000000000003,"15min":13.365},"last":1.397}]}]},{"osd":2,"up_from":14,"seq":60129542370,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":1413120,"data_stored":610069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.91400000000000003,"15min":1.0069999999999999},"min":{"1min":0.69999999999999996,"5min":0.311,"15min":0.224},"max":{"1min":1.5009999999999999,"5min":2.1579999999999999,"15min":11.804},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.181,"5min":1.0840000000000001,"15min":1.155},"min":{"1min":0.70199999999999996,"5min":0.443,"15min":0.35799999999999998},"max":{"1min":1.756,"5min":3.52,"15min":13.066000000000001},"last":0.318}]},{"osd":1,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0509999999999999,"15min":1.1639999999999999},"min":{"1min":0.82399999999999995,"5min":0.434,"15min":0.29899999999999999},"max":{"1min":1.448,"5min":2.2599999999999998,"15min":14.199999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.131,"5min":1.1060000000000001,"15min":1.224},"min":{"1min":0.78500000000000003,"5min":0.47799999999999998,"15min":0.42999999999999999},"max":{"1min":1.762,"5min":2.9900000000000002,"15min":13.15},"last":0.48699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.073,"15min":1.1739999999999999},"min":{"1min":0.78800000000000003,"5min":0.45700000000000002,"15min":0.31900000000000001},"max":{"1min":1.274,"5min":6.1260000000000003,"15min":14.965},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.268,"5min":1.254,"15min":1.286},"min":{"1min":0.93400000000000005,"5min":0.47999999999999998,"15min":0.39300000000000002},"max":{"1min":1.5720000000000001,"5min":6.2510000000000003,"15min":11.717000000000001},"last":0.58199999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.4330000000000001,"15min":1.675},"min":{"1min":1.1339999999999999,"5min":0.68400000000000005,"15min":0.46300000000000002},"max":{"1min":1.9490000000000001,"5min":2.3010000000000002,"15min":19.957999999999998},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3979999999999999,"15min":1.6020000000000001},"min":{"1min":1.0680000000000001,"5min":0.71099999999999997,"15min":0.55700000000000005},"max":{"1min":1.6879999999999999,"5min":3.419,"15min":15.151},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.367,"15min":1.5640000000000001},"min":{"1min":1.177,"5min":0.76200000000000001,"15min":0.60899999999999999},"max":{"1min":1.8340000000000001,"5min":3.6160000000000001,"15min":26.501000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3580000000000001,"15min":1.456},"min":{"1min":1.0209999999999999,"5min":0.81699999999999995,"15min":0.47999999999999998},"max":{"1min":1.577,"5min":6.3499999999999996,"15min":13.760999999999999},"last":0.84499999999999997}]},{"osd":6,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.4019999999999999,"15min":1.502},"min":{"1min":1.153,"5min":0.86599999999999999,"15min":0.438},"max":{"1min":1.794,"5min":6.4729999999999999,"15min":15.144},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.532,"5min":1.464,"15min":1.855},"min":{"1min":1.0960000000000001,"5min":0.71099999999999997,"15min":0.48199999999999998},"max":{"1min":2.024,"5min":3.2469999999999999,"15min":66.061999999999998},"last":1.012}]},{"osd":7,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5029999999999999,"15min":1.7},"min":{"1min":1.147,"5min":0.871,"15min":0.61099999999999999},"max":{"1min":2.1469999999999998,"5min":3.7290000000000001,"15min":16.667000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.522,"15min":1.635},"min":{"1min":1.1799999999999999,"5min":0.90100000000000002,"15min":0.5},"max":{"1min":1.9319999999999999,"5min":6.4249999999999998,"15min":11.271000000000001},"last":0.99099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542370,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72772,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299068,"statfs":{"total":96636764160,"available":96562245632,"internally_reserved":0,"allocated":1433600,"data_stored":616968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.96299999999999997,"15min":1.111},"min":{"1min":0.495,"5min":0.33400000000000002,"15min":0.248},"max":{"1min":1.4670000000000001,"5min":1.8089999999999999,"15min":21.260000000000002},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.123,"15min":1.306},"min":{"1min":0.74099999999999999,"5min":0.59599999999999997,"15min":0.35199999999999998},"max":{"1min":1.4970000000000001,"5min":2.407,"15min":32.787999999999997},"last":0.96199999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.93100000000000005,"15min":1.004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.28100000000000003},"max":{"1min":1.2490000000000001,"5min":2.226,"15min":14.817},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.159,"15min":1.224},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.42599999999999999},"max":{"1min":1.512,"5min":2.3420000000000001,"15min":8.8450000000000006},"last":0.90000000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.2210000000000001,"15min":1.278},"min":{"1min":0.628,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":1.645,"5min":1.907,"15min":9.3149999999999995},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.1100000000000001,"15min":1.3440000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.40400000000000003},"max":{"1min":1.496,"5min":2.508,"15min":41.133000000000003},"last":1.135}]},{"osd":4,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.3520000000000001,"15min":1.5860000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.59999999999999998},"max":{"1min":1.7689999999999999,"5min":3.044,"15min":19.684000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.4610000000000001},"min":{"1min":0.64000000000000001,"5min":0.57899999999999996,"15min":0.53900000000000003},"max":{"1min":1.6890000000000001,"5min":3.3100000000000001,"15min":13.789999999999999},"last":1.2250000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3360000000000001,"15min":1.5600000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.43099999999999999},"max":{"1min":1.7470000000000001,"5min":1.9059999999999999,"15min":15.778},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.397,"5min":1.4570000000000001,"15min":1.788},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.623},"max":{"1min":1.861,"5min":2.226,"15min":35.747},"last":1.171}]},{"osd":6,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.4379999999999999,"15min":1.671},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.64400000000000002},"max":{"1min":2.0150000000000001,"5min":2.9809999999999999,"15min":16.231000000000002},"last":1.198},{"interface":"front","average":{"1min":1.383,"5min":1.431,"15min":1.579},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56599999999999995},"max":{"1min":1.847,"5min":4.1349999999999998,"15min":13.56},"last":1.3500000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.48,"15min":1.825},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58699999999999997},"max":{"1min":2.0089999999999999,"5min":3.2690000000000001,"15min":21.902999999999999},"last":1.306},{"interface":"front","average":{"1min":1.429,"5min":1.5169999999999999,"15min":1.8400000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.68700000000000006},"max":{"1min":1.931,"5min":3.4079999999999999,"15min":31.044},"last":1.264}]}]},{"osd":0,"up_from":14,"seq":60129542371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69204,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302636,"statfs":{"total":96636764160,"available":96565899264,"internally_reserved":0,"allocated":2039808,"data_stored":1208936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.85699999999999998,"15min":0.96799999999999997},"min":{"1min":0.59699999999999998,"5min":0.35099999999999998,"15min":0.25},"max":{"1min":1.4119999999999999,"5min":1.6100000000000001,"15min":22.619},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0680000000000001,"15min":1.194},"min":{"1min":0.65900000000000003,"5min":0.40100000000000002,"15min":0.29399999999999998},"max":{"1min":1.2549999999999999,"5min":8.2119999999999997,"15min":17.876000000000001},"last":1.1839999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.306,"15min":1.4139999999999999},"min":{"1min":0.90800000000000003,"5min":0.54100000000000004,"15min":0.42999999999999999},"max":{"1min":1.5760000000000001,"5min":6.1189999999999998,"15min":17.760999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.119,"5min":1.135,"15min":1.274},"min":{"1min":0.79700000000000004,"5min":0.46400000000000002,"15min":0.33100000000000002},"max":{"1min":1.5820000000000001,"5min":5.3159999999999998,"15min":22.672999999999998},"last":0.61699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.1579999999999999,"15min":1.2130000000000001},"min":{"1min":0.72999999999999998,"5min":0.51600000000000001,"15min":0.40000000000000002},"max":{"1min":1.401,"5min":10.83,"15min":16.382999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2270000000000001,"15min":1.3959999999999999},"min":{"1min":0.80200000000000005,"5min":0.49099999999999999,"15min":0.40999999999999998},"max":{"1min":1.4770000000000001,"5min":7.9720000000000004,"15min":17.981000000000002},"last":1.2270000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3169999999999999,"15min":1.466},"min":{"1min":0.90000000000000002,"5min":0.75600000000000001,"15min":0.51800000000000002},"max":{"1min":1.7190000000000001,"5min":2.2330000000000001,"15min":22.780000000000001},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.365,"15min":1.53},"min":{"1min":0.82799999999999996,"5min":0.80700000000000005,"15min":0.56299999999999994},"max":{"1min":1.627,"5min":10.986000000000001,"15min":19.422000000000001},"last":1.536}]},{"osd":5,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3899999999999999,"15min":1.7889999999999999},"min":{"1min":1.0780000000000001,"5min":0.80600000000000005,"15min":0.441},"max":{"1min":1.885,"5min":2.1819999999999999,"15min":44.930999999999997},"last":1.421},{"interface":"front","average":{"1min":1.296,"5min":1.4319999999999999,"15min":1.758},"min":{"1min":0.93100000000000005,"5min":0.67500000000000004,"15min":0.55400000000000005},"max":{"1min":1.754,"5min":10.928000000000001,"15min":21.797999999999998},"last":1.3600000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.5029999999999999,"15min":1.911},"min":{"1min":1.2130000000000001,"5min":0.91400000000000003,"15min":0.622},"max":{"1min":1.841,"5min":8.0399999999999991,"15min":27.442},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4690000000000001,"15min":1.859},"min":{"1min":1.131,"5min":0.85899999999999999,"15min":0.50700000000000001},"max":{"1min":1.966,"5min":2.2999999999999998,"15min":27.411999999999999},"last":1.655}]},{"osd":7,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.518,"15min":1.7090000000000001},"min":{"1min":1.1659999999999999,"5min":0.90800000000000003,"15min":0.59099999999999997},"max":{"1min":1.7,"5min":8.0960000000000001,"15min":18.152999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.655,"15min":1.7450000000000001},"min":{"1min":1.1799999999999999,"5min":0.84199999999999997,"15min":0.60099999999999998},"max":{"1min":1.796,"5min":13.718,"15min":22.815000000000001},"last":1.7450000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:43.760 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:43.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.994+0000 7f3bf7e00640 1 Processor -- start 2024-01-16T15:15:43.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.994+0000 7f3bf7e00640 1 -- start start 2024-01-16T15:15:43.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3bf80cc500 0x7f3bf80cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3bf806d160 0x7f3bf806d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf8070e50 con 0x7f3bf8143c40 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80723c0 con 0x7f3bf806d160 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80cced0 con 0x7f3bf80cc500 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3bf806d160 0x7f3bf806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3bf806d160 0x7f3bf806d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55384/0 (socket says 172.21.15.141:55384) 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3bf7400640 1 -- 172.21.15.141:0/3115230166 learned_addr learned my addr 172.21.15.141:0/3115230166 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.996+0000 7f3befe00640 1 --2- 172.21.15.141:0/3115230166 >> v2:172.21.15.188:3300/0 conn(0x7f3bf80cc500 0x7f3bf80cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.997+0000 7f3bf7400640 1 -- 172.21.15.141:0/3115230166 >> v2:172.21.15.188:3300/0 conn(0x7f3bf80cc500 msgr2=0x7f3bf80cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.997+0000 7f3bf7400640 1 --2- 172.21.15.141:0/3115230166 >> v2:172.21.15.188:3300/0 conn(0x7f3bf80cc500 0x7f3bf80cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.997+0000 7f3bf7400640 1 -- 172.21.15.141:0/3115230166 >> v1:172.21.15.188:6789/0 conn(0x7f3bf8143c40 legacy=0x7f3bf8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:43.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.997+0000 7f3bf7400640 1 -- 172.21.15.141:0/3115230166 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8144020 con 0x7f3bf806d160 2024-01-16T15:15:43.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.998+0000 7f3bf7400640 1 --2- 172.21.15.141:0/3115230166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf806d560 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b44a0 tx=0x7f3bf002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403d8fdf926c3d99 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.998+0000 7f3bf6000640 1 -- 172.21.15.141:0/3115230166 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf002c9f0 con 0x7f3bf806d160 2024-01-16T15:15:43.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.998+0000 7f3bf6000640 1 -- 172.21.15.141:0/3115230166 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf002cb50 con 0x7f3bf806d160 2024-01-16T15:15:43.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.998+0000 7f3bf6000640 1 -- 172.21.15.141:0/3115230166 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00343f0 con 0x7f3bf806d160 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 -- 172.21.15.141:0/3115230166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 msgr2=0x7f3bf806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/3115230166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf806d560 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b44a0 tx=0x7f3bf002be50 comp rx=0 tx=0).stop 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 -- 172.21.15.141:0/3115230166 shutdown_connections 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/3115230166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf806d560 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/3115230166 >> v2:172.21.15.188:3300/0 conn(0x7f3bf80cc500 0x7f3bf80cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 -- 172.21.15.141:0/3115230166 >> 172.21.15.141:0/3115230166 conn(0x7f3bf806a800 msgr2=0x7f3bf806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 -- 172.21.15.141:0/3115230166 shutdown_connections 2024-01-16T15:15:43.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:43.999+0000 7f3bf7e00640 1 -- 172.21.15.141:0/3115230166 wait complete. 2024-01-16T15:15:43.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.001+0000 7f3bf7e00640 1 Processor -- start 2024-01-16T15:15:43.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.001+0000 7f3bf7e00640 1 -- start start 2024-01-16T15:15:43.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.002+0000 7f3bf7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf813d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.002+0000 7f3bf7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bf80cc500 0x7f3bf80acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.002+0000 7f3bf7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bf8143c40 0x7f3bf80ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.002+0000 7f3bf7e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf813dce0 con 0x7f3bf806d160 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.002+0000 7f3bf7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80ad620 con 0x7f3bf8143c40 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.002+0000 7f3bf7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80ad900 con 0x7f3bf80cc500 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf813d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55388/0 (socket says 172.21.15.141:55388) 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 -- 172.21.15.141:0/2726650450 learned_addr learned my addr 172.21.15.141:0/2726650450 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf7400640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bf8143c40 0x7f3bf80ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3befe00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bf80cc500 0x7f3bf80acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 -- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bf80cc500 msgr2=0x7f3bf80acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bf80cc500 0x7f3bf80acad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 -- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bf8143c40 msgr2=0x7f3bf80ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:43.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bf8143c40 0x7f3bf80ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 -- 172.21.15.141:0/2726650450 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0009b30 con 0x7f3bf806d160 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf7400640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bf8143c40 0x7f3bf80ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.003+0000 7f3bf6a00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf813d7a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f3be800b3f0 tx=0x7f3be800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.004+0000 7f3beea00640 1 -- 172.21.15.141:0/2726650450 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8011830 con 0x7f3bf806d160 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.004+0000 7f3beea00640 1 -- 172.21.15.141:0/2726650450 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be8011990 con 0x7f3bf806d160 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.004+0000 7f3beea00640 1 -- 172.21.15.141:0/2726650450 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8011d40 con 0x7f3bf806d160 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.004+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf80adbe0 con 0x7f3bf806d160 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.004+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf80ae1a0 con 0x7f3bf806d160 2024-01-16T15:15:43.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.005+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf80ae390 con 0x7f3bf806d160 2024-01-16T15:15:43.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.006+0000 7f3beea00640 1 -- 172.21.15.141:0/2726650450 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f3be801a400 con 0x7f3bf806d160 2024-01-16T15:15:43.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.007+0000 7f3beea00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3bd006c4c0 0x7f3bd006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:43.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.007+0000 7f3beea00640 1 -- 172.21.15.141:0/2726650450 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3be8088910 con 0x7f3bf806d160 2024-01-16T15:15:43.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.007+0000 7f3befe00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3bd006c4c0 0x7f3bd006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:43.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.009+0000 7f3befe00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3bd006c4c0 0x7f3bd006e980 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8070260 tx=0x7f3be0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:43.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.011+0000 7f3beea00640 1 -- 172.21.15.141:0/2726650450 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3be8054ec0 con 0x7f3bf806d160 2024-01-16T15:15:44.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.194+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3bf806c960 con 0x7f3bd006c4c0 2024-01-16T15:15:44.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.216+0000 7f3beea00640 1 -- 172.21.15.141:0/2726650450 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286996 (secure 0 0 0) 0x7f3be000b200 con 0x7f3bd006c4c0 2024-01-16T15:15:44.205 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:44.206 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:15:44.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.222+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3bd006c4c0 msgr2=0x7f3bd006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.222+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3bd006c4c0 0x7f3bd006e980 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8070260 tx=0x7f3be0008040 comp rx=0 tx=0).stop 2024-01-16T15:15:44.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.222+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 msgr2=0x7f3bf813d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.222+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf813d7a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f3be800b3f0 tx=0x7f3be800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 shutdown_connections 2024-01-16T15:15:44.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3bd006c4c0 0x7f3bd006e980 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bf8143c40 0x7f3bf80ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3bf80cc500 0x7f3bf80acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 --2- 172.21.15.141:0/2726650450 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806d160 0x7f3bf813d7a0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 >> 172.21.15.141:0/2726650450 conn(0x7f3bf806a800 msgr2=0x7f3bf8071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:44.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 shutdown_connections 2024-01-16T15:15:44.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.223+0000 7f3bf7e00640 1 -- 172.21.15.141:0/2726650450 wait complete. 2024-01-16T15:15:44.226 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2024-01-16T15:15:42.921873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":566676,"kb_used_data":12316,"kb_used_omap":12,"kb_used_meta":554227,"kb_avail":754408044,"statfs":{"total":773094113280,"available":772513837056,"internally_reserved":0,"allocated":12611584,"data_stored":6099367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":567529010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.663917"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332086+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:15:39.332086+0000","last_peered":"2024-01-16T15:15:39.332086+0000","last_clean":"2024-01-16T15:15:39.332086+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:15:39.332086+0000","last_undegraded":"2024-01-16T15:15:39.332086+0000","last_fullsized":"2024-01-16T15:15:39.332086+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:01.053271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331443+0000","last_change":"2024-01-16T15:15:39.327644+0000","last_active":"2024-01-16T15:15:39.331443+0000","last_peered":"2024-01-16T15:15:39.331443+0000","last_clean":"2024-01-16T15:15:39.331443+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:15:39.331443+0000","last_undegraded":"2024-01-16T15:15:39.331443+0000","last_fullsized":"2024-01-16T15:15:39.331443+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:38.284829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330731+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:15:39.330731+0000","last_peered":"2024-01-16T15:15:39.330731+0000","last_clean":"2024-01-16T15:15:39.330731+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:15:39.330731+0000","last_undegraded":"2024-01-16T15:15:39.330731+0000","last_fullsized":"2024-01-16T15:15:39.330731+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:56.026622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330388+0000","last_change":"2024-01-16T15:15:39.320416+0000","last_active":"2024-01-16T15:15:39.330388+0000","last_peered":"2024-01-16T15:15:39.330388+0000","last_clean":"2024-01-16T15:15:39.330388+0000","last_became_active":"2024-01-16T15:15:39.317971+0000","last_became_peered":"2024-01-16T15:15:39.317971+0000","last_unstale":"2024-01-16T15:15:39.330388+0000","last_undegraded":"2024-01-16T15:15:39.330388+0000","last_fullsized":"2024-01-16T15:15:39.330388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:39:41.607736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326309+0000","last_change":"2024-01-16T15:15:39.321081+0000","last_active":"2024-01-16T15:15:39.326309+0000","last_peered":"2024-01-16T15:15:39.326309+0000","last_clean":"2024-01-16T15:15:39.326309+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:15:39.326309+0000","last_undegraded":"2024-01-16T15:15:39.326309+0000","last_fullsized":"2024-01-16T15:15:39.326309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:42.051788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331857+0000","last_change":"2024-01-16T15:15:39.327813+0000","last_active":"2024-01-16T15:15:39.331857+0000","last_peered":"2024-01-16T15:15:39.331857+0000","last_clean":"2024-01-16T15:15:39.331857+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:15:39.331857+0000","last_undegraded":"2024-01-16T15:15:39.331857+0000","last_fullsized":"2024-01-16T15:15:39.331857+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:52.806951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326306+0000","last_change":"2024-01-16T15:15:39.324707+0000","last_active":"2024-01-16T15:15:39.326306+0000","last_peered":"2024-01-16T15:15:39.326306+0000","last_clean":"2024-01-16T15:15:39.326306+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:15:39.326306+0000","last_undegraded":"2024-01-16T15:15:39.326306+0000","last_fullsized":"2024-01-16T15:15:39.326306+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:32.958658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331955+0000","last_change":"2024-01-16T15:15:39.328547+0000","last_active":"2024-01-16T15:15:39.331955+0000","last_peered":"2024-01-16T15:15:39.331955+0000","last_clean":"2024-01-16T15:15:39.331955+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:15:39.331955+0000","last_undegraded":"2024-01-16T15:15:39.331955+0000","last_fullsized":"2024-01-16T15:15:39.331955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:35.351694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330368+0000","last_change":"2024-01-16T15:15:39.320236+0000","last_active":"2024-01-16T15:15:39.330368+0000","last_peered":"2024-01-16T15:15:39.330368+0000","last_clean":"2024-01-16T15:15:39.330368+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:15:39.330368+0000","last_undegraded":"2024-01-16T15:15:39.330368+0000","last_fullsized":"2024-01-16T15:15:39.330368+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:49.706005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330878+0000","last_change":"2024-01-16T15:15:39.326528+0000","last_active":"2024-01-16T15:15:39.330878+0000","last_peered":"2024-01-16T15:15:39.330878+0000","last_clean":"2024-01-16T15:15:39.330878+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:15:39.330878+0000","last_undegraded":"2024-01-16T15:15:39.330878+0000","last_fullsized":"2024-01-16T15:15:39.330878+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:28.634370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.327773+0000","last_change":"2024-01-16T15:15:39.325379+0000","last_active":"2024-01-16T15:15:39.327773+0000","last_peered":"2024-01-16T15:15:39.327773+0000","last_clean":"2024-01-16T15:15:39.327773+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:15:39.327773+0000","last_undegraded":"2024-01-16T15:15:39.327773+0000","last_fullsized":"2024-01-16T15:15:39.327773+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:36.770903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330299+0000","last_change":"2024-01-16T15:15:39.317978+0000","last_active":"2024-01-16T15:15:39.330299+0000","last_peered":"2024-01-16T15:15:39.330299+0000","last_clean":"2024-01-16T15:15:39.330299+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:15:39.330299+0000","last_undegraded":"2024-01-16T15:15:39.330299+0000","last_fullsized":"2024-01-16T15:15:39.330299+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:55.882282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332865+0000","last_change":"2024-01-16T15:15:39.329570+0000","last_active":"2024-01-16T15:15:39.332865+0000","last_peered":"2024-01-16T15:15:39.332865+0000","last_clean":"2024-01-16T15:15:39.332865+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:15:39.332865+0000","last_undegraded":"2024-01-16T15:15:39.332865+0000","last_fullsized":"2024-01-16T15:15:39.332865+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:39.747172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328318+0000","last_change":"2024-01-16T15:15:39.325613+0000","last_active":"2024-01-16T15:15:39.328318+0000","last_peered":"2024-01-16T15:15:39.328318+0000","last_clean":"2024-01-16T15:15:39.328318+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:15:39.328318+0000","last_undegraded":"2024-01-16T15:15:39.328318+0000","last_fullsized":"2024-01-16T15:15:39.328318+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:58.758941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.336943+0000","last_change":"2024-01-16T15:15:39.333121+0000","last_active":"2024-01-16T15:15:39.336943+0000","last_peered":"2024-01-16T15:15:39.336943+0000","last_clean":"2024-01-16T15:15:39.336943+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:15:39.336943+0000","last_undegraded":"2024-01-16T15:15:39.336943+0000","last_fullsized":"2024-01-16T15:15:39.336943+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:15.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.315001+0000","last_change":"2024-01-16T15:15:39.313940+0000","last_active":"2024-01-16T15:15:39.315001+0000","last_peered":"2024-01-16T15:15:39.315001+0000","last_clean":"2024-01-16T15:15:39.315001+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:15:39.315001+0000","last_undegraded":"2024-01-16T15:15:39.315001+0000","last_fullsized":"2024-01-16T15:15:39.315001+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:56.164840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331698+0000","last_change":"2024-01-16T15:15:39.329528+0000","last_active":"2024-01-16T15:15:39.331698+0000","last_peered":"2024-01-16T15:15:39.331698+0000","last_clean":"2024-01-16T15:15:39.331698+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:15:39.331698+0000","last_undegraded":"2024-01-16T15:15:39.331698+0000","last_fullsized":"2024-01-16T15:15:39.331698+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333188+0000","last_change":"2024-01-16T15:15:39.330676+0000","last_active":"2024-01-16T15:15:39.333188+0000","last_peered":"2024-01-16T15:15:39.333188+0000","last_clean":"2024-01-16T15:15:39.333188+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:15:39.333188+0000","last_undegraded":"2024-01-16T15:15:39.333188+0000","last_fullsized":"2024-01-16T15:15:39.333188+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:49.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332871+0000","last_change":"2024-01-16T15:15:39.330437+0000","last_active":"2024-01-16T15:15:39.332871+0000","last_peered":"2024-01-16T15:15:39.332871+0000","last_clean":"2024-01-16T15:15:39.332871+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:15:39.332871+0000","last_undegraded":"2024-01-16T15:15:39.332871+0000","last_fullsized":"2024-01-16T15:15:39.332871+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:41.474484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.321853+0000","last_change":"2024-01-16T15:15:39.318689+0000","last_active":"2024-01-16T15:15:39.321853+0000","last_peered":"2024-01-16T15:15:39.321853+0000","last_clean":"2024-01-16T15:15:39.321853+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:15:39.321853+0000","last_undegraded":"2024-01-16T15:15:39.321853+0000","last_fullsized":"2024-01-16T15:15:39.321853+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:39:32.420035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.323628+0000","last_change":"2024-01-16T15:15:39.321055+0000","last_active":"2024-01-16T15:15:39.323628+0000","last_peered":"2024-01-16T15:15:39.323628+0000","last_clean":"2024-01-16T15:15:39.323628+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:15:39.323628+0000","last_undegraded":"2024-01-16T15:15:39.323628+0000","last_fullsized":"2024-01-16T15:15:39.323628+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.327999+0000","last_change":"2024-01-16T15:15:39.323290+0000","last_active":"2024-01-16T15:15:39.327999+0000","last_peered":"2024-01-16T15:15:39.327999+0000","last_clean":"2024-01-16T15:15:39.327999+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:15:39.327999+0000","last_undegraded":"2024-01-16T15:15:39.327999+0000","last_fullsized":"2024-01-16T15:15:39.327999+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:28.634370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.319883+0000","last_change":"2024-01-16T15:15:39.317974+0000","last_active":"2024-01-16T15:15:39.319883+0000","last_peered":"2024-01-16T15:15:39.319883+0000","last_clean":"2024-01-16T15:15:39.319883+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:15:39.319883+0000","last_undegraded":"2024-01-16T15:15:39.319883+0000","last_fullsized":"2024-01-16T15:15:39.319883+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:15.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328548+0000","last_change":"2024-01-16T15:15:39.324363+0000","last_active":"2024-01-16T15:15:39.328548+0000","last_peered":"2024-01-16T15:15:39.328548+0000","last_clean":"2024-01-16T15:15:39.328548+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:15:39.328548+0000","last_undegraded":"2024-01-16T15:15:39.328548+0000","last_fullsized":"2024-01-16T15:15:39.328548+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:18.050547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.331679+0000","last_change":"2024-01-16T15:15:39.328896+0000","last_active":"2024-01-16T15:15:39.331679+0000","last_peered":"2024-01-16T15:15:39.331679+0000","last_clean":"2024-01-16T15:15:39.331679+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:15:39.331679+0000","last_undegraded":"2024-01-16T15:15:39.331679+0000","last_fullsized":"2024-01-16T15:15:39.331679+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:19.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326057+0000","last_change":"2024-01-16T15:15:39.317589+0000","last_active":"2024-01-16T15:15:39.326057+0000","last_peered":"2024-01-16T15:15:39.326057+0000","last_clean":"2024-01-16T15:15:39.326057+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:15:39.326057+0000","last_undegraded":"2024-01-16T15:15:39.326057+0000","last_fullsized":"2024-01-16T15:15:39.326057+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:06.562007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337670+0000","last_change":"2024-01-16T15:15:39.335455+0000","last_active":"2024-01-16T15:15:39.337670+0000","last_peered":"2024-01-16T15:15:39.337670+0000","last_clean":"2024-01-16T15:15:39.337670+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:15:39.337670+0000","last_undegraded":"2024-01-16T15:15:39.337670+0000","last_fullsized":"2024-01-16T15:15:39.337670+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.325678+0000","last_change":"2024-01-16T15:15:39.317292+0000","last_active":"2024-01-16T15:15:39.325678+0000","last_peered":"2024-01-16T15:15:39.325678+0000","last_clean":"2024-01-16T15:15:39.325678+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:15:39.325678+0000","last_undegraded":"2024-01-16T15:15:39.325678+0000","last_fullsized":"2024-01-16T15:15:39.325678+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:55.882282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.322773+0000","last_change":"2024-01-16T15:15:39.320340+0000","last_active":"2024-01-16T15:15:39.322773+0000","last_peered":"2024-01-16T15:15:39.322773+0000","last_clean":"2024-01-16T15:15:39.322773+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:15:39.322773+0000","last_undegraded":"2024-01-16T15:15:39.322773+0000","last_fullsized":"2024-01-16T15:15:39.322773+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:35:57.563013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.318994+0000","last_change":"2024-01-16T15:15:39.317544+0000","last_active":"2024-01-16T15:15:39.318994+0000","last_peered":"2024-01-16T15:15:39.318994+0000","last_clean":"2024-01-16T15:15:39.318994+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:15:39.318994+0000","last_undegraded":"2024-01-16T15:15:39.318994+0000","last_fullsized":"2024-01-16T15:15:39.318994+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:08.504738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.314764+0000","last_change":"2024-01-16T15:15:39.313788+0000","last_active":"2024-01-16T15:15:39.314764+0000","last_peered":"2024-01-16T15:15:39.314764+0000","last_clean":"2024-01-16T15:15:39.314764+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:15:39.314764+0000","last_undegraded":"2024-01-16T15:15:39.314764+0000","last_fullsized":"2024-01-16T15:15:39.314764+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:40.960949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326025+0000","last_change":"2024-01-16T15:15:39.321572+0000","last_active":"2024-01-16T15:15:39.326025+0000","last_peered":"2024-01-16T15:15:39.326025+0000","last_clean":"2024-01-16T15:15:39.326025+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:15:39.326025+0000","last_undegraded":"2024-01-16T15:15:39.326025+0000","last_fullsized":"2024-01-16T15:15:39.326025+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:20.421089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332430+0000","last_change":"2024-01-16T15:15:39.329833+0000","last_active":"2024-01-16T15:15:39.332430+0000","last_peered":"2024-01-16T15:15:39.332430+0000","last_clean":"2024-01-16T15:15:39.332430+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:15:39.332430+0000","last_undegraded":"2024-01-16T15:15:39.332430+0000","last_fullsized":"2024-01-16T15:15:39.332430+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:36.827801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'4","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.342972+0000","last_change":"2024-01-16T15:15:38.295673+0000","last_active":"2024-01-16T15:15:39.342972+0000","last_peered":"2024-01-16T15:15:39.342972+0000","last_clean":"2024-01-16T15:15:39.342972+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:15:39.342972+0000","last_undegraded":"2024-01-16T15:15:39.342972+0000","last_fullsized":"2024-01-16T15:15:39.342972+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333684+0000","last_change":"2024-01-16T15:15:39.329620+0000","last_active":"2024-01-16T15:15:39.333684+0000","last_peered":"2024-01-16T15:15:39.333684+0000","last_clean":"2024-01-16T15:15:39.333684+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:15:39.333684+0000","last_undegraded":"2024-01-16T15:15:39.333684+0000","last_fullsized":"2024-01-16T15:15:39.333684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:49:22.746780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.341114+0000","last_change":"2024-01-16T15:15:38.292769+0000","last_active":"2024-01-16T15:15:39.341114+0000","last_peered":"2024-01-16T15:15:39.341114+0000","last_clean":"2024-01-16T15:15:39.341114+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:15:39.341114+0000","last_undegraded":"2024-01-16T15:15:39.341114+0000","last_fullsized":"2024-01-16T15:15:39.341114+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:45.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.336878+0000","last_change":"2024-01-16T15:15:39.332870+0000","last_active":"2024-01-16T15:15:39.336878+0000","last_peered":"2024-01-16T15:15:39.336878+0000","last_clean":"2024-01-16T15:15:39.336878+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:15:39.336878+0000","last_undegraded":"2024-01-16T15:15:39.336878+0000","last_fullsized":"2024-01-16T15:15:39.336878+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:42.051788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.350874+0000","last_change":"2024-01-16T15:15:38.288495+0000","last_active":"2024-01-16T15:15:39.350874+0000","last_peered":"2024-01-16T15:15:39.350874+0000","last_clean":"2024-01-16T15:15:39.350874+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:15:39.350874+0000","last_undegraded":"2024-01-16T15:15:39.350874+0000","last_fullsized":"2024-01-16T15:15:39.350874+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:07.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332179+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:15:39.332179+0000","last_peered":"2024-01-16T15:15:39.332179+0000","last_clean":"2024-01-16T15:15:39.332179+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:15:39.332179+0000","last_undegraded":"2024-01-16T15:15:39.332179+0000","last_fullsized":"2024-01-16T15:15:39.332179+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:21.913468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.347018+0000","last_change":"2024-01-16T15:15:38.288324+0000","last_active":"2024-01-16T15:15:39.347018+0000","last_peered":"2024-01-16T15:15:39.347018+0000","last_clean":"2024-01-16T15:15:39.347018+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:15:39.347018+0000","last_undegraded":"2024-01-16T15:15:39.347018+0000","last_fullsized":"2024-01-16T15:15:39.347018+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:31.849193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330784+0000","last_change":"2024-01-16T15:15:39.321037+0000","last_active":"2024-01-16T15:15:39.330784+0000","last_peered":"2024-01-16T15:15:39.330784+0000","last_clean":"2024-01-16T15:15:39.330784+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:15:39.330784+0000","last_undegraded":"2024-01-16T15:15:39.330784+0000","last_fullsized":"2024-01-16T15:15:39.330784+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:04:44.476846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.343928+0000","last_change":"2024-01-16T15:15:38.288226+0000","last_active":"2024-01-16T15:15:39.343928+0000","last_peered":"2024-01-16T15:15:39.343928+0000","last_clean":"2024-01-16T15:15:39.343928+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:15:39.343928+0000","last_undegraded":"2024-01-16T15:15:39.343928+0000","last_fullsized":"2024-01-16T15:15:39.343928+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:41.040743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332266+0000","last_change":"2024-01-16T15:15:39.330201+0000","last_active":"2024-01-16T15:15:39.332266+0000","last_peered":"2024-01-16T15:15:39.332266+0000","last_clean":"2024-01-16T15:15:39.332266+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:15:39.332266+0000","last_undegraded":"2024-01-16T15:15:39.332266+0000","last_fullsized":"2024-01-16T15:15:39.332266+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:40.033000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.342620+0000","last_change":"2024-01-16T15:15:38.292721+0000","last_active":"2024-01-16T15:15:39.342620+0000","last_peered":"2024-01-16T15:15:39.342620+0000","last_clean":"2024-01-16T15:15:39.342620+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:15:39.342620+0000","last_undegraded":"2024-01-16T15:15:39.342620+0000","last_fullsized":"2024-01-16T15:15:39.342620+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:32.177991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.329937+0000","last_change":"2024-01-16T15:15:39.322497+0000","last_active":"2024-01-16T15:15:39.329937+0000","last_peered":"2024-01-16T15:15:39.329937+0000","last_clean":"2024-01-16T15:15:39.329937+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:15:39.329937+0000","last_undegraded":"2024-01-16T15:15:39.329937+0000","last_fullsized":"2024-01-16T15:15:39.329937+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:57.002611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.294502+0000","last_change":"2024-01-16T15:15:38.276895+0000","last_active":"2024-01-16T15:15:39.294502+0000","last_peered":"2024-01-16T15:15:39.294502+0000","last_clean":"2024-01-16T15:15:39.294502+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:15:39.294502+0000","last_undegraded":"2024-01-16T15:15:39.294502+0000","last_fullsized":"2024-01-16T15:15:39.294502+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326325+0000","last_change":"2024-01-16T15:15:39.325224+0000","last_active":"2024-01-16T15:15:39.326325+0000","last_peered":"2024-01-16T15:15:39.326325+0000","last_clean":"2024-01-16T15:15:39.326325+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:15:39.326325+0000","last_undegraded":"2024-01-16T15:15:39.326325+0000","last_fullsized":"2024-01-16T15:15:39.326325+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:32.863366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301149+0000","last_change":"2024-01-16T15:15:38.297269+0000","last_active":"2024-01-16T15:15:39.301149+0000","last_peered":"2024-01-16T15:15:39.301149+0000","last_clean":"2024-01-16T15:15:39.301149+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:15:39.301149+0000","last_undegraded":"2024-01-16T15:15:39.301149+0000","last_fullsized":"2024-01-16T15:15:39.301149+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:02.166771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.322612+0000","last_change":"2024-01-16T15:15:39.320446+0000","last_active":"2024-01-16T15:15:39.322612+0000","last_peered":"2024-01-16T15:15:39.322612+0000","last_clean":"2024-01-16T15:15:39.322612+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:15:39.322612+0000","last_undegraded":"2024-01-16T15:15:39.322612+0000","last_fullsized":"2024-01-16T15:15:39.322612+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:36.827801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.292107+0000","last_change":"2024-01-16T15:15:38.287453+0000","last_active":"2024-01-16T15:15:39.292107+0000","last_peered":"2024-01-16T15:15:39.292107+0000","last_clean":"2024-01-16T15:15:39.292107+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:15:39.292107+0000","last_undegraded":"2024-01-16T15:15:39.292107+0000","last_fullsized":"2024-01-16T15:15:39.292107+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.334533+0000","last_change":"2024-01-16T15:15:39.333810+0000","last_active":"2024-01-16T15:15:39.334533+0000","last_peered":"2024-01-16T15:15:39.334533+0000","last_clean":"2024-01-16T15:15:39.334533+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:15:39.334533+0000","last_undegraded":"2024-01-16T15:15:39.334533+0000","last_fullsized":"2024-01-16T15:15:39.334533+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:01:18.350475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.318263+0000","last_change":"2024-01-16T15:15:38.293583+0000","last_active":"2024-01-16T15:15:39.318263+0000","last_peered":"2024-01-16T15:15:39.318263+0000","last_clean":"2024-01-16T15:15:39.318263+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:15:39.318263+0000","last_undegraded":"2024-01-16T15:15:39.318263+0000","last_fullsized":"2024-01-16T15:15:39.318263+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:48.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.325695+0000","last_change":"2024-01-16T15:15:39.319368+0000","last_active":"2024-01-16T15:15:39.325695+0000","last_peered":"2024-01-16T15:15:39.325695+0000","last_clean":"2024-01-16T15:15:39.325695+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:15:39.325695+0000","last_undegraded":"2024-01-16T15:15:39.325695+0000","last_fullsized":"2024-01-16T15:15:39.325695+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:29.630325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'3","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.350395+0000","last_change":"2024-01-16T15:15:38.295750+0000","last_active":"2024-01-16T15:15:39.350395+0000","last_peered":"2024-01-16T15:15:39.350395+0000","last_clean":"2024-01-16T15:15:39.350395+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:15:39.350395+0000","last_undegraded":"2024-01-16T15:15:39.350395+0000","last_fullsized":"2024-01-16T15:15:39.350395+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:17.036375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.317322+0000","last_change":"2024-01-16T15:15:39.312858+0000","last_active":"2024-01-16T15:15:39.317322+0000","last_peered":"2024-01-16T15:15:39.317322+0000","last_clean":"2024-01-16T15:15:39.317322+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:15:39.317322+0000","last_undegraded":"2024-01-16T15:15:39.317322+0000","last_fullsized":"2024-01-16T15:15:39.317322+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:42.051788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'6","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.352515+0000","last_change":"2024-01-16T15:15:38.296732+0000","last_active":"2024-01-16T15:15:39.352515+0000","last_peered":"2024-01-16T15:15:39.352515+0000","last_clean":"2024-01-16T15:15:39.352515+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:15:39.352515+0000","last_undegraded":"2024-01-16T15:15:39.352515+0000","last_fullsized":"2024-01-16T15:15:39.352515+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.316975+0000","last_change":"2024-01-16T15:15:39.312723+0000","last_active":"2024-01-16T15:15:39.316975+0000","last_peered":"2024-01-16T15:15:39.316975+0000","last_clean":"2024-01-16T15:15:39.316975+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:15:39.316975+0000","last_undegraded":"2024-01-16T15:15:39.316975+0000","last_fullsized":"2024-01-16T15:15:39.316975+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:39:41.607736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.346109+0000","last_change":"2024-01-16T15:15:38.280236+0000","last_active":"2024-01-16T15:15:39.346109+0000","last_peered":"2024-01-16T15:15:39.346109+0000","last_clean":"2024-01-16T15:15:39.346109+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:15:39.346109+0000","last_undegraded":"2024-01-16T15:15:39.346109+0000","last_fullsized":"2024-01-16T15:15:39.346109+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:14.715313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328110+0000","last_change":"2024-01-16T15:15:39.324127+0000","last_active":"2024-01-16T15:15:39.328110+0000","last_peered":"2024-01-16T15:15:39.328110+0000","last_clean":"2024-01-16T15:15:39.328110+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:15:39.328110+0000","last_undegraded":"2024-01-16T15:15:39.328110+0000","last_fullsized":"2024-01-16T15:15:39.328110+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:06.562007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.317399+0000","last_change":"2024-01-16T15:15:38.295857+0000","last_active":"2024-01-16T15:15:39.317399+0000","last_peered":"2024-01-16T15:15:39.317399+0000","last_clean":"2024-01-16T15:15:39.317399+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:15:39.317399+0000","last_undegraded":"2024-01-16T15:15:39.317399+0000","last_fullsized":"2024-01-16T15:15:39.317399+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:31.063560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.326208+0000","last_change":"2024-01-16T15:15:39.319495+0000","last_active":"2024-01-16T15:15:39.326208+0000","last_peered":"2024-01-16T15:15:39.326208+0000","last_clean":"2024-01-16T15:15:39.326208+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:15:39.326208+0000","last_undegraded":"2024-01-16T15:15:39.326208+0000","last_fullsized":"2024-01-16T15:15:39.326208+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:02.542534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'8","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.351392+0000","last_change":"2024-01-16T15:15:38.293904+0000","last_active":"2024-01-16T15:15:39.351392+0000","last_peered":"2024-01-16T15:15:39.351392+0000","last_clean":"2024-01-16T15:15:39.351392+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:15:39.351392+0000","last_undegraded":"2024-01-16T15:15:39.351392+0000","last_fullsized":"2024-01-16T15:15:39.351392+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.317862+0000","last_change":"2024-01-16T15:15:39.314200+0000","last_active":"2024-01-16T15:15:39.317862+0000","last_peered":"2024-01-16T15:15:39.317862+0000","last_clean":"2024-01-16T15:15:39.317862+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:15:39.317862+0000","last_undegraded":"2024-01-16T15:15:39.317862+0000","last_fullsized":"2024-01-16T15:15:39.317862+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:36.827801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.345442+0000","last_change":"2024-01-16T15:15:38.296521+0000","last_active":"2024-01-16T15:15:39.345442+0000","last_peered":"2024-01-16T15:15:39.345442+0000","last_clean":"2024-01-16T15:15:39.345442+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:15:39.345442+0000","last_undegraded":"2024-01-16T15:15:39.345442+0000","last_fullsized":"2024-01-16T15:15:39.345442+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:24.030528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337557+0000","last_change":"2024-01-16T15:15:39.334617+0000","last_active":"2024-01-16T15:15:39.337557+0000","last_peered":"2024-01-16T15:15:39.337557+0000","last_clean":"2024-01-16T15:15:39.337557+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:15:39.337557+0000","last_undegraded":"2024-01-16T15:15:39.337557+0000","last_fullsized":"2024-01-16T15:15:39.337557+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.347180+0000","last_change":"2024-01-16T15:15:38.289440+0000","last_active":"2024-01-16T15:15:39.347180+0000","last_peered":"2024-01-16T15:15:39.347180+0000","last_clean":"2024-01-16T15:15:39.347180+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:15:39.347180+0000","last_undegraded":"2024-01-16T15:15:39.347180+0000","last_fullsized":"2024-01-16T15:15:39.347180+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333320+0000","last_change":"2024-01-16T15:15:39.332704+0000","last_active":"2024-01-16T15:15:39.333320+0000","last_peered":"2024-01-16T15:15:39.333320+0000","last_clean":"2024-01-16T15:15:39.333320+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:15:39.333320+0000","last_undegraded":"2024-01-16T15:15:39.333320+0000","last_fullsized":"2024-01-16T15:15:39.333320+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:52.162066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.344109+0000","last_change":"2024-01-16T15:15:38.291309+0000","last_active":"2024-01-16T15:15:39.344109+0000","last_peered":"2024-01-16T15:15:39.344109+0000","last_clean":"2024-01-16T15:15:39.344109+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:15:39.344109+0000","last_undegraded":"2024-01-16T15:15:39.344109+0000","last_fullsized":"2024-01-16T15:15:39.344109+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:35.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330751+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:15:39.330751+0000","last_peered":"2024-01-16T15:15:39.330751+0000","last_clean":"2024-01-16T15:15:39.330751+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:15:39.330751+0000","last_undegraded":"2024-01-16T15:15:39.330751+0000","last_fullsized":"2024-01-16T15:15:39.330751+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:45.950432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301594+0000","last_change":"2024-01-16T15:15:38.298667+0000","last_active":"2024-01-16T15:15:39.301594+0000","last_peered":"2024-01-16T15:15:39.301594+0000","last_clean":"2024-01-16T15:15:39.301594+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:15:39.301594+0000","last_undegraded":"2024-01-16T15:15:39.301594+0000","last_fullsized":"2024-01-16T15:15:39.301594+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328339+0000","last_change":"2024-01-16T15:15:39.326224+0000","last_active":"2024-01-16T15:15:39.328339+0000","last_peered":"2024-01-16T15:15:39.328339+0000","last_clean":"2024-01-16T15:15:39.328339+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:15:39.328339+0000","last_undegraded":"2024-01-16T15:15:39.328339+0000","last_fullsized":"2024-01-16T15:15:39.328339+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:02.559083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.347887+0000","last_change":"2024-01-16T15:15:38.278400+0000","last_active":"2024-01-16T15:15:39.347887+0000","last_peered":"2024-01-16T15:15:39.347887+0000","last_clean":"2024-01-16T15:15:39.347887+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:15:39.347887+0000","last_undegraded":"2024-01-16T15:15:39.347887+0000","last_fullsized":"2024-01-16T15:15:39.347887+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:37.301739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330809+0000","last_change":"2024-01-16T15:15:39.325408+0000","last_active":"2024-01-16T15:15:39.330809+0000","last_peered":"2024-01-16T15:15:39.330809+0000","last_clean":"2024-01-16T15:15:39.330809+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:15:39.330809+0000","last_undegraded":"2024-01-16T15:15:39.330809+0000","last_fullsized":"2024-01-16T15:15:39.330809+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:10.305933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.292084+0000","last_change":"2024-01-16T15:15:38.287926+0000","last_active":"2024-01-16T15:15:39.292084+0000","last_peered":"2024-01-16T15:15:39.292084+0000","last_clean":"2024-01-16T15:15:39.292084+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:15:39.292084+0000","last_undegraded":"2024-01-16T15:15:39.292084+0000","last_fullsized":"2024-01-16T15:15:39.292084+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:31.944486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332823+0000","last_change":"2024-01-16T15:15:39.330480+0000","last_active":"2024-01-16T15:15:39.332823+0000","last_peered":"2024-01-16T15:15:39.332823+0000","last_clean":"2024-01-16T15:15:39.332823+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:15:39.332823+0000","last_undegraded":"2024-01-16T15:15:39.332823+0000","last_fullsized":"2024-01-16T15:15:39.332823+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:42.153458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.323111+0000","last_change":"2024-01-16T15:15:38.276702+0000","last_active":"2024-01-16T15:15:39.323111+0000","last_peered":"2024-01-16T15:15:39.323111+0000","last_clean":"2024-01-16T15:15:39.323111+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:15:39.323111+0000","last_undegraded":"2024-01-16T15:15:39.323111+0000","last_fullsized":"2024-01-16T15:15:39.323111+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.325460+0000","last_change":"2024-01-16T15:15:39.316268+0000","last_active":"2024-01-16T15:15:39.325460+0000","last_peered":"2024-01-16T15:15:39.325460+0000","last_clean":"2024-01-16T15:15:39.325460+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:15:39.325460+0000","last_undegraded":"2024-01-16T15:15:39.325460+0000","last_fullsized":"2024-01-16T15:15:39.325460+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:04:44.476846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301502+0000","last_change":"2024-01-16T15:15:38.297590+0000","last_active":"2024-01-16T15:15:39.301502+0000","last_peered":"2024-01-16T15:15:39.301502+0000","last_clean":"2024-01-16T15:15:39.301502+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:15:39.301502+0000","last_undegraded":"2024-01-16T15:15:39.301502+0000","last_fullsized":"2024-01-16T15:15:39.301502+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:18.917382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.334063+0000","last_change":"2024-01-16T15:15:39.333001+0000","last_active":"2024-01-16T15:15:39.334063+0000","last_peered":"2024-01-16T15:15:39.334063+0000","last_clean":"2024-01-16T15:15:39.334063+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:15:39.334063+0000","last_undegraded":"2024-01-16T15:15:39.334063+0000","last_fullsized":"2024-01-16T15:15:39.334063+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:41.770690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.345982+0000","last_change":"2024-01-16T15:15:38.291784+0000","last_active":"2024-01-16T15:15:39.345982+0000","last_peered":"2024-01-16T15:15:39.345982+0000","last_clean":"2024-01-16T15:15:39.345982+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:15:39.345982+0000","last_undegraded":"2024-01-16T15:15:39.345982+0000","last_fullsized":"2024-01-16T15:15:39.345982+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.330348+0000","last_change":"2024-01-16T15:15:39.320032+0000","last_active":"2024-01-16T15:15:39.330348+0000","last_peered":"2024-01-16T15:15:39.330348+0000","last_clean":"2024-01-16T15:15:39.330348+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:15:39.330348+0000","last_undegraded":"2024-01-16T15:15:39.330348+0000","last_fullsized":"2024-01-16T15:15:39.330348+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:33.192164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.324013+0000","last_change":"2024-01-16T15:15:38.295797+0000","last_active":"2024-01-16T15:15:39.324013+0000","last_peered":"2024-01-16T15:15:39.324013+0000","last_clean":"2024-01-16T15:15:39.324013+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:15:39.324013+0000","last_undegraded":"2024-01-16T15:15:39.324013+0000","last_fullsized":"2024-01-16T15:15:39.324013+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:49.218031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337264+0000","last_change":"2024-01-16T15:15:39.335229+0000","last_active":"2024-01-16T15:15:39.337264+0000","last_peered":"2024-01-16T15:15:39.337264+0000","last_clean":"2024-01-16T15:15:39.337264+0000","last_became_active":"2024-01-16T15:15:39.333516+0000","last_became_peered":"2024-01-16T15:15:39.333516+0000","last_unstale":"2024-01-16T15:15:39.337264+0000","last_undegraded":"2024-01-16T15:15:39.337264+0000","last_fullsized":"2024-01-16T15:15:39.337264+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:32:02.542534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.303295+0000","last_change":"2024-01-16T15:15:38.297697+0000","last_active":"2024-01-16T15:15:39.303295+0000","last_peered":"2024-01-16T15:15:39.303295+0000","last_clean":"2024-01-16T15:15:39.303295+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:15:39.303295+0000","last_undegraded":"2024-01-16T15:15:39.303295+0000","last_fullsized":"2024-01-16T15:15:39.303295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:28.616152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.333744+0000","last_change":"2024-01-16T15:15:39.331210+0000","last_active":"2024-01-16T15:15:39.333744+0000","last_peered":"2024-01-16T15:15:39.333744+0000","last_clean":"2024-01-16T15:15:39.333744+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:15:39.333744+0000","last_undegraded":"2024-01-16T15:15:39.333744+0000","last_fullsized":"2024-01-16T15:15:39.333744+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:24.608493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.318562+0000","last_change":"2024-01-16T15:15:38.287424+0000","last_active":"2024-01-16T15:15:39.318562+0000","last_peered":"2024-01-16T15:15:39.318562+0000","last_clean":"2024-01-16T15:15:39.318562+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:15:39.318562+0000","last_undegraded":"2024-01-16T15:15:39.318562+0000","last_fullsized":"2024-01-16T15:15:39.318562+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:00.039099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.316776+0000","last_change":"2024-01-16T15:15:39.313583+0000","last_active":"2024-01-16T15:15:39.316776+0000","last_peered":"2024-01-16T15:15:39.316776+0000","last_clean":"2024-01-16T15:15:39.316776+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:15:39.316776+0000","last_undegraded":"2024-01-16T15:15:39.316776+0000","last_fullsized":"2024-01-16T15:15:39.316776+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:38.315912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301157+0000","last_change":"2024-01-16T15:15:38.290534+0000","last_active":"2024-01-16T15:15:39.301157+0000","last_peered":"2024-01-16T15:15:39.301157+0000","last_clean":"2024-01-16T15:15:39.301157+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:15:39.301157+0000","last_undegraded":"2024-01-16T15:15:39.301157+0000","last_fullsized":"2024-01-16T15:15:39.301157+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.314637+0000","last_change":"2024-01-16T15:15:39.305386+0000","last_active":"2024-01-16T15:15:39.314637+0000","last_peered":"2024-01-16T15:15:39.314637+0000","last_clean":"2024-01-16T15:15:39.314637+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:15:39.314637+0000","last_undegraded":"2024-01-16T15:15:39.314637+0000","last_fullsized":"2024-01-16T15:15:39.314637+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.295093+0000","last_change":"2024-01-16T15:15:38.287042+0000","last_active":"2024-01-16T15:15:39.295093+0000","last_peered":"2024-01-16T15:15:39.295093+0000","last_clean":"2024-01-16T15:15:39.295093+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:15:39.295093+0000","last_undegraded":"2024-01-16T15:15:39.295093+0000","last_fullsized":"2024-01-16T15:15:39.295093+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:50.656301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.332800+0000","last_change":"2024-01-16T15:15:39.330274+0000","last_active":"2024-01-16T15:15:39.332800+0000","last_peered":"2024-01-16T15:15:39.332800+0000","last_clean":"2024-01-16T15:15:39.332800+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:15:39.332800+0000","last_undegraded":"2024-01-16T15:15:39.332800+0000","last_fullsized":"2024-01-16T15:15:39.332800+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:11.903533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.291887+0000","last_change":"2024-01-16T15:15:38.281868+0000","last_active":"2024-01-16T15:15:39.291887+0000","last_peered":"2024-01-16T15:15:39.291887+0000","last_clean":"2024-01-16T15:15:39.291887+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:15:39.291887+0000","last_undegraded":"2024-01-16T15:15:39.291887+0000","last_fullsized":"2024-01-16T15:15:39.291887+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.337252+0000","last_change":"2024-01-16T15:15:39.333842+0000","last_active":"2024-01-16T15:15:39.337252+0000","last_peered":"2024-01-16T15:15:39.337252+0000","last_clean":"2024-01-16T15:15:39.337252+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:15:39.337252+0000","last_undegraded":"2024-01-16T15:15:39.337252+0000","last_fullsized":"2024-01-16T15:15:39.337252+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:47.137989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.301875+0000","last_change":"2024-01-16T15:15:38.297094+0000","last_active":"2024-01-16T15:15:39.301875+0000","last_peered":"2024-01-16T15:15:39.301875+0000","last_clean":"2024-01-16T15:15:39.301875+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:15:39.301875+0000","last_undegraded":"2024-01-16T15:15:39.301875+0000","last_fullsized":"2024-01-16T15:15:39.301875+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:48.520471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.328755+0000","last_change":"2024-01-16T15:15:39.324548+0000","last_active":"2024-01-16T15:15:39.328755+0000","last_peered":"2024-01-16T15:15:39.328755+0000","last_clean":"2024-01-16T15:15:39.328755+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:15:39.328755+0000","last_undegraded":"2024-01-16T15:15:39.328755+0000","last_fullsized":"2024-01-16T15:15:39.328755+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:49.534643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:15:39.304750+0000","last_change":"2024-01-16T15:15:38.299858+0000","last_active":"2024-01-16T15:15:39.304750+0000","last_peered":"2024-01-16T15:15:39.304750+0000","last_clean":"2024-01-16T15:15:39.304750+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:15:39.304750+0000","last_undegraded":"2024-01-16T15:15:39.304750+0000","last_fullsized":"2024-01-16T15:15:39.304750+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":257,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-16T15:15:38.267237+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:15:38.267237+0000","last_peered":"2024-01-16T15:15:38.267237+0000","last_clean":"2024-01-16T15:15:38.267237+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:15:38.267237+0000","last_undegraded":"2024-01-16T15:15:38.267237+0000","last_fullsized":"2024-01-16T15:15:38.267237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73332,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298508,"statfs":{"total":96636764160,"available":96561672192,"internally_reserved":0,"allocated":2007040,"data_stored":1200437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.3400000000000001,"15min":1.538},"min":{"1min":0.68200000000000005,"5min":0.57799999999999996,"15min":0.37},"max":{"1min":1.3700000000000001,"5min":13.566000000000001,"15min":19.603000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.3899999999999999},"min":{"1min":0.73499999999999999,"5min":0.63600000000000001,"15min":0.49299999999999999},"max":{"1min":1.4710000000000001,"5min":10.728,"15min":24.015000000000001},"last":1.399}]},{"osd":1,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.248,"15min":1.48},"min":{"1min":0.68799999999999994,"5min":0.65600000000000003,"15min":0.40999999999999998},"max":{"1min":1.6060000000000001,"5min":13.375,"15min":19.527000000000001},"last":1.042},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1080000000000001,"15min":1.3939999999999999},"min":{"1min":0.70999999999999996,"5min":0.69499999999999995,"15min":0.443},"max":{"1min":1.5,"5min":1.867,"15min":24.07},"last":1.25}]},{"osd":2,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.2230000000000001,"15min":1.3660000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.46700000000000003},"max":{"1min":1.484,"5min":1.7909999999999999,"15min":24.108000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.1419999999999999,"15min":1.472},"min":{"1min":0.76900000000000002,"5min":0.76400000000000001,"15min":0.37},"max":{"1min":1.383,"5min":1.7210000000000001,"15min":24.806000000000001},"last":0.96999999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.3640000000000001,"15min":1.754},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.52600000000000002},"max":{"1min":1.52,"5min":13.657999999999999,"15min":23.741},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.375,"15min":1.647},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.626},"max":{"1min":1.748,"5min":1.806,"15min":24.143000000000001},"last":1.484}]},{"osd":4,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2430000000000001,"15min":1.641},"min":{"1min":0.79400000000000004,"5min":0.71999999999999997,"15min":0.45800000000000002},"max":{"1min":1.6659999999999999,"5min":1.776,"15min":25.637},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.55,"15min":1.8919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.44400000000000001},"max":{"1min":1.6319999999999999,"5min":13.782999999999999,"15min":25.594000000000001},"last":1.5600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.419,"15min":1.708},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.47999999999999998},"max":{"1min":1.8089999999999999,"5min":1.9670000000000001,"15min":29.567},"last":1.54},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4490000000000001,"15min":1.764},"min":{"1min":0.873,"5min":0.83999999999999997,"15min":0.55700000000000005},"max":{"1min":1.6970000000000001,"5min":1.9970000000000001,"15min":27.937000000000001},"last":1.514}]},{"osd":6,"last update":"Tue Jan 16 15:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.425,"15min":1.8640000000000001},"min":{"1min":0.67900000000000005,"5min":0.65800000000000003,"15min":0.50800000000000001},"max":{"1min":1.6890000000000001,"5min":2.0419999999999998,"15min":41.206000000000003},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.6399999999999999,"15min":2.1749999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.41699999999999998},"max":{"1min":1.758,"5min":13.725,"15min":39.932000000000002},"last":1.1399999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68608,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303232,"statfs":{"total":96636764160,"available":96566509568,"internally_reserved":0,"allocated":1429504,"data_stored":616914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.212,"15min":1.444},"min":{"1min":0.90900000000000003,"5min":0.628,"15min":0.52700000000000002},"max":{"1min":1.6759999999999999,"5min":3.9289999999999998,"15min":20.739000000000001},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.268,"15min":1.6220000000000001},"min":{"1min":0.84299999999999997,"5min":0.51400000000000001,"15min":0.498},"max":{"1min":1.538,"5min":4.6070000000000002,"15min":50.472000000000001},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.379,"15min":1.4199999999999999},"min":{"1min":0.89800000000000002,"5min":0.497,"15min":0.497},"max":{"1min":1.5589999999999999,"5min":10.518000000000001,"15min":10.629},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.272,"15min":1.45},"min":{"1min":0.871,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.9810000000000001,"5min":4.0339999999999998,"15min":9.6880000000000006},"last":1.444}]},{"osd":2,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.288,"15min":1.619},"min":{"1min":0.98499999999999999,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":2.1960000000000002,"5min":4.7140000000000004,"15min":50.643999999999998},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.6299999999999999,"15min":1.5900000000000001},"min":{"1min":0.90200000000000002,"5min":0.66000000000000003,"15min":0.59399999999999997},"max":{"1min":1.738,"5min":15.282,"15min":15.282},"last":1.476}]},{"osd":3,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.419,"15min":1.597},"min":{"1min":0.97499999999999998,"5min":0.64100000000000001,"15min":0.54900000000000004},"max":{"1min":1.8160000000000001,"5min":4.1109999999999998,"15min":15.675000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.458,"15min":1.7609999999999999},"min":{"1min":1.075,"5min":0.60099999999999998,"15min":0.52600000000000002},"max":{"1min":1.657,"5min":4.8330000000000002,"15min":50.561},"last":1.587}]},{"osd":4,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.4510000000000001,"15min":1.8089999999999999},"min":{"1min":0.92800000000000005,"5min":0.63200000000000001,"15min":0.46300000000000002},"max":{"1min":1.6779999999999999,"5min":5.4669999999999996,"15min":47.488},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.46,"15min":1.893},"min":{"1min":0.92400000000000004,"5min":0.41199999999999998,"15min":0.40400000000000003},"max":{"1min":1.6859999999999999,"5min":5.2350000000000003,"15min":50.381},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.5629999999999999,"15min":1.5509999999999999},"min":{"1min":1.022,"5min":0.436,"15min":0.436},"max":{"1min":1.633,"5min":10.388999999999999,"15min":10.654},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.613,"15min":1.8009999999999999},"min":{"1min":1.173,"5min":0.5,"15min":0.42799999999999999},"max":{"1min":2.1579999999999999,"5min":4.5449999999999999,"15min":18.699000000000002},"last":1.5089999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.667,"15min":2.0779999999999998},"min":{"1min":1.139,"5min":0.60599999999999998,"15min":0.44900000000000001},"max":{"1min":2.0449999999999999,"5min":5.9420000000000002,"15min":52.912999999999997},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.7070000000000001,"15min":2.3050000000000002},"min":{"1min":1.028,"5min":0.58499999999999996,"15min":0.495},"max":{"1min":2.3260000000000001,"5min":10.659000000000001,"15min":26.413},"last":1.377}]}]},{"osd":5,"up_from":14,"seq":60129542370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64460,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307380,"statfs":{"total":96636764160,"available":96570757120,"internally_reserved":0,"allocated":1433600,"data_stored":617448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1839999999999999,"15min":1.3600000000000001},"min":{"1min":0.80400000000000005,"5min":0.77700000000000002,"15min":0.38900000000000001},"max":{"1min":1.4610000000000001,"5min":2.222,"15min":13.938000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.248,"15min":1.4159999999999999},"min":{"1min":0.875,"5min":0.71999999999999997,"15min":0.42299999999999999},"max":{"1min":1.5569999999999999,"5min":5.4039999999999999,"15min":16.896000000000001},"last":1.093}]},{"osd":1,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.3240000000000001,"15min":1.49},"min":{"1min":0.80200000000000005,"5min":0.76000000000000001,"15min":0.50700000000000001},"max":{"1min":1.643,"5min":8.1820000000000004,"15min":15.500999999999999},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.179,"15min":1.369},"min":{"1min":0.73599999999999999,"5min":0.73099999999999998,"15min":0.46300000000000002},"max":{"1min":1.4810000000000001,"5min":2.2970000000000002,"15min":35.328000000000003},"last":1.4079999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2490000000000001,"15min":1.4630000000000001},"min":{"1min":0.96699999999999997,"5min":0.75900000000000001,"15min":0.46200000000000002},"max":{"1min":1.6799999999999999,"5min":3.0110000000000001,"15min":35.398000000000003},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.276,"15min":1.4219999999999999},"min":{"1min":1.056,"5min":0.75,"15min":0.38900000000000001},"max":{"1min":1.5329999999999999,"5min":2.5550000000000002,"15min":15.4},"last":1.2150000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.468,"15min":1.79},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.50900000000000001},"max":{"1min":1.6319999999999999,"5min":8.0809999999999995,"15min":24.254000000000001},"last":1.649},{"interface":"front","average":{"1min":1.401,"5min":1.4350000000000001,"15min":1.776},"min":{"1min":0.98199999999999998,"5min":0.89600000000000002,"15min":0.57399999999999995},"max":{"1min":1.6200000000000001,"5min":3.1619999999999999,"15min":35.268000000000001},"last":1.468}]},{"osd":4,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.2969999999999999,"15min":1.639},"min":{"1min":0.89200000000000002,"5min":0.74299999999999999,"15min":0.42199999999999999},"max":{"1min":1.6259999999999999,"5min":3.2599999999999998,"15min":19.059000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4590000000000001,"15min":1.7829999999999999},"min":{"1min":0.91700000000000004,"5min":0.57399999999999995,"15min":0.42799999999999999},"max":{"1min":1.7609999999999999,"5min":8.3550000000000004,"15min":28.760000000000002},"last":1.526}]},{"osd":6,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.468,"15min":1.837},"min":{"1min":1.252,"5min":0.73899999999999999,"15min":0.53400000000000003},"max":{"1min":1.8700000000000001,"5min":3.3220000000000001,"15min":35.427999999999997},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.5569999999999999,"15min":1.8819999999999999},"min":{"1min":1.3819999999999999,"5min":0.89400000000000002,"15min":0.53100000000000003},"max":{"1min":1.7809999999999999,"5min":3.391,"15min":16.262},"last":1.4950000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.6259999999999999,"15min":1.996},"min":{"1min":1.228,"5min":0.67000000000000004,"15min":0.54300000000000004},"max":{"1min":1.9450000000000001,"5min":8.2509999999999994,"15min":29.672999999999998},"last":1.742},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.5580000000000001,"15min":1.76},"min":{"1min":1.202,"5min":0.71999999999999997,"15min":0.45900000000000002},"max":{"1min":1.9950000000000001,"5min":3.2229999999999999,"15min":35.363},"last":1.1559999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542371,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68596,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303244,"statfs":{"total":96636764160,"available":96566521856,"internally_reserved":0,"allocated":1413120,"data_stored":610069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.26,"15min":1.4390000000000001},"min":{"1min":0.81200000000000006,"5min":0.63300000000000001,"15min":0.52800000000000002},"max":{"1min":1.8460000000000001,"5min":3.3159999999999998,"15min":15.942},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.238,"15min":1.4059999999999999},"min":{"1min":0.746,"5min":0.68999999999999995,"15min":0.45400000000000001},"max":{"1min":1.5469999999999999,"5min":1.9350000000000001,"15min":16.219999999999999},"last":1.1519999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3280000000000001,"15min":1.486},"min":{"1min":0.93000000000000005,"5min":0.76300000000000001,"15min":0.54100000000000004},"max":{"1min":1.901,"5min":1.9590000000000001,"15min":16.280000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.2030000000000001,"15min":1.4650000000000001},"min":{"1min":0.85699999999999998,"5min":0.77900000000000003,"15min":0.60699999999999998},"max":{"1min":1.6120000000000001,"5min":3.411,"15min":15.888},"last":0.91600000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.288,"15min":1.4650000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":13.731999999999999,"15min":13.837},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.3680000000000001,"15min":1.593},"min":{"1min":1.091,"5min":0.84999999999999998,"15min":0.64800000000000002},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":17.175000000000001},"last":1.411}]},{"osd":3,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.335,"15min":1.6990000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.57599999999999996},"max":{"1min":1.6919999999999999,"5min":2.077,"15min":16.469000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.371,"5min":1.458,"15min":1.5429999999999999},"min":{"1min":1.0229999999999999,"5min":0.86399999999999999,"15min":0.55600000000000005},"max":{"1min":1.643,"5min":13.914,"15min":15.089},"last":1.496}]},{"osd":5,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.393,"15min":1.5940000000000001},"min":{"1min":0.95199999999999996,"5min":0.83799999999999997,"15min":0.47299999999999998},"max":{"1min":1.6779999999999999,"5min":10.211,"15min":18.347999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.4319999999999999,"15min":1.6040000000000001},"min":{"1min":0.91300000000000003,"5min":0.86399999999999999,"15min":0.45800000000000002},"max":{"1min":1.768,"5min":2.149,"15min":22.937000000000001},"last":1.456}]},{"osd":6,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.51,"15min":1.589},"min":{"1min":0.998,"5min":0.877,"15min":0.501},"max":{"1min":1.661,"5min":13.973000000000001,"15min":18.033000000000001},"last":1.647},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5449999999999999,"15min":1.8640000000000001},"min":{"1min":1.3089999999999999,"5min":1.099,"15min":0.51800000000000002},"max":{"1min":1.752,"5min":3.0979999999999999,"15min":20.939},"last":1.5820000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.5489999999999999,"15min":1.71},"min":{"1min":1.097,"5min":0.97699999999999998,"15min":0.44500000000000001},"max":{"1min":1.8899999999999999,"5min":2.3199999999999998,"15min":16.324999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.6699999999999999,"15min":1.873},"min":{"1min":1.395,"5min":1.048,"15min":0.52200000000000002},"max":{"1min":2.2040000000000002,"5min":13.852,"15min":20.277999999999999},"last":1.7170000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76944,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294896,"statfs":{"total":96636764160,"available":96557973504,"internally_reserved":0,"allocated":1441792,"data_stored":618526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.95999999999999996,"15min":0.98099999999999998},"min":{"1min":0.503,"5min":0.502,"15min":0.34300000000000003},"max":{"1min":1.2869999999999999,"5min":2.9129999999999998,"15min":5.3570000000000002},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1080000000000001,"15min":1.127},"min":{"1min":0.55200000000000005,"5min":0.42399999999999999,"15min":0.29699999999999999},"max":{"1min":1.4930000000000001,"5min":3.5859999999999999,"15min":12.292999999999999},"last":0.83099999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.96799999999999997,"15min":0.95499999999999996},"min":{"1min":0.65600000000000003,"5min":0.52200000000000002,"15min":0.309},"max":{"1min":1.2470000000000001,"5min":1.8380000000000001,"15min":4.0979999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1950000000000001,"15min":1.2010000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.315},"max":{"1min":1.5169999999999999,"5min":10.638,"15min":13.443},"last":0.75900000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.145,"15min":1.151},"min":{"1min":0.70599999999999996,"5min":0.48199999999999998,"15min":0.35999999999999999},"max":{"1min":1.5720000000000001,"5min":4.4850000000000003,"15min":4.4850000000000003},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.131,"5min":1.228,"15min":1.2370000000000001},"min":{"1min":0.625,"5min":0.55000000000000004,"15min":0.35299999999999998},"max":{"1min":1.617,"5min":2.129,"15min":5.5110000000000001},"last":0.95999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.264,"15min":1.3859999999999999},"min":{"1min":0.79400000000000004,"5min":0.69899999999999995,"15min":0.39300000000000002},"max":{"1min":1.6899999999999999,"5min":4,"15min":13.292},"last":1.661},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.3600000000000001,"15min":1.4750000000000001},"min":{"1min":0.75,"5min":0.69499999999999995,"15min":0.52100000000000002},"max":{"1min":1.625,"5min":3.1150000000000002,"15min":11.068},"last":1.488}]},{"osd":5,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.4219999999999999,"15min":1.5780000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.48499999999999999},"max":{"1min":1.649,"5min":3.0019999999999998,"15min":17.096},"last":1.302},{"interface":"front","average":{"1min":1.339,"5min":1.419,"15min":1.5549999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.42099999999999999},"max":{"1min":1.8420000000000001,"5min":2.2639999999999998,"15min":12.106},"last":1.3560000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.3959999999999999,"15min":1.754},"min":{"1min":0.749,"5min":0.749,"15min":0.41499999999999998},"max":{"1min":1.8340000000000001,"5min":3.8239999999999998,"15min":43.609999999999999},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.4670000000000001,"15min":1.6919999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.54900000000000004},"max":{"1min":1.5860000000000001,"5min":2.7799999999999998,"15min":38.784999999999997},"last":1.5880000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.5620000000000001,"15min":1.732},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.53700000000000003},"max":{"1min":1.9570000000000001,"5min":7.8899999999999997,"15min":25.071000000000002},"last":1.044},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4830000000000001,"15min":1.718},"min":{"1min":0.95599999999999996,"5min":0.87,"15min":0.435},"max":{"1min":1.9370000000000001,"5min":4.2880000000000003,"15min":13.365},"last":1.397}]}]},{"osd":2,"up_from":14,"seq":60129542370,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":1413120,"data_stored":610069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.91400000000000003,"15min":1.0069999999999999},"min":{"1min":0.69999999999999996,"5min":0.311,"15min":0.224},"max":{"1min":1.5009999999999999,"5min":2.1579999999999999,"15min":11.804},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.181,"5min":1.0840000000000001,"15min":1.155},"min":{"1min":0.70199999999999996,"5min":0.443,"15min":0.35799999999999998},"max":{"1min":1.756,"5min":3.52,"15min":13.066000000000001},"last":0.318}]},{"osd":1,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0509999999999999,"15min":1.1639999999999999},"min":{"1min":0.82399999999999995,"5min":0.434,"15min":0.29899999999999999},"max":{"1min":1.448,"5min":2.2599999999999998,"15min":14.199999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.131,"5min":1.1060000000000001,"15min":1.224},"min":{"1min":0.78500000000000003,"5min":0.47799999999999998,"15min":0.42999999999999999},"max":{"1min":1.762,"5min":2.9900000000000002,"15min":13.15},"last":0.48699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.073,"15min":1.1739999999999999},"min":{"1min":0.78800000000000003,"5min":0.45700000000000002,"15min":0.31900000000000001},"max":{"1min":1.274,"5min":6.1260000000000003,"15min":14.965},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.268,"5min":1.254,"15min":1.286},"min":{"1min":0.93400000000000005,"5min":0.47999999999999998,"15min":0.39300000000000002},"max":{"1min":1.5720000000000001,"5min":6.2510000000000003,"15min":11.717000000000001},"last":0.58199999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.4330000000000001,"15min":1.675},"min":{"1min":1.1339999999999999,"5min":0.68400000000000005,"15min":0.46300000000000002},"max":{"1min":1.9490000000000001,"5min":2.3010000000000002,"15min":19.957999999999998},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3979999999999999,"15min":1.6020000000000001},"min":{"1min":1.0680000000000001,"5min":0.71099999999999997,"15min":0.55700000000000005},"max":{"1min":1.6879999999999999,"5min":3.419,"15min":15.151},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.367,"15min":1.5640000000000001},"min":{"1min":1.177,"5min":0.76200000000000001,"15min":0.60899999999999999},"max":{"1min":1.8340000000000001,"5min":3.6160000000000001,"15min":26.501000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3580000000000001,"15min":1.456},"min":{"1min":1.0209999999999999,"5min":0.81699999999999995,"15min":0.47999999999999998},"max":{"1min":1.577,"5min":6.3499999999999996,"15min":13.760999999999999},"last":0.84499999999999997}]},{"osd":6,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.4019999999999999,"15min":1.502},"min":{"1min":1.153,"5min":0.86599999999999999,"15min":0.438},"max":{"1min":1.794,"5min":6.4729999999999999,"15min":15.144},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.532,"5min":1.464,"15min":1.855},"min":{"1min":1.0960000000000001,"5min":0.71099999999999997,"15min":0.48199999999999998},"max":{"1min":2.024,"5min":3.2469999999999999,"15min":66.061999999999998},"last":1.012}]},{"osd":7,"last update":"Tue Jan 16 15:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5029999999999999,"15min":1.7},"min":{"1min":1.147,"5min":0.871,"15min":0.61099999999999999},"max":{"1min":2.1469999999999998,"5min":3.7290000000000001,"15min":16.667000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.522,"15min":1.635},"min":{"1min":1.1799999999999999,"5min":0.90100000000000002,"15min":0.5},"max":{"1min":1.9319999999999999,"5min":6.4249999999999998,"15min":11.271000000000001},"last":0.99099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542370,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72772,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299068,"statfs":{"total":96636764160,"available":96562245632,"internally_reserved":0,"allocated":1433600,"data_stored":616968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.96299999999999997,"15min":1.111},"min":{"1min":0.495,"5min":0.33400000000000002,"15min":0.248},"max":{"1min":1.4670000000000001,"5min":1.8089999999999999,"15min":21.260000000000002},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.123,"15min":1.306},"min":{"1min":0.74099999999999999,"5min":0.59599999999999997,"15min":0.35199999999999998},"max":{"1min":1.4970000000000001,"5min":2.407,"15min":32.787999999999997},"last":0.96199999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.93100000000000005,"15min":1.004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.28100000000000003},"max":{"1min":1.2490000000000001,"5min":2.226,"15min":14.817},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.159,"15min":1.224},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.42599999999999999},"max":{"1min":1.512,"5min":2.3420000000000001,"15min":8.8450000000000006},"last":0.90000000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.2210000000000001,"15min":1.278},"min":{"1min":0.628,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":1.645,"5min":1.907,"15min":9.3149999999999995},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.1100000000000001,"15min":1.3440000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.40400000000000003},"max":{"1min":1.496,"5min":2.508,"15min":41.133000000000003},"last":1.135}]},{"osd":4,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.3520000000000001,"15min":1.5860000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.59999999999999998},"max":{"1min":1.7689999999999999,"5min":3.044,"15min":19.684000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.4610000000000001},"min":{"1min":0.64000000000000001,"5min":0.57899999999999996,"15min":0.53900000000000003},"max":{"1min":1.6890000000000001,"5min":3.3100000000000001,"15min":13.789999999999999},"last":1.2250000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3360000000000001,"15min":1.5600000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.43099999999999999},"max":{"1min":1.7470000000000001,"5min":1.9059999999999999,"15min":15.778},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.397,"5min":1.4570000000000001,"15min":1.788},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.623},"max":{"1min":1.861,"5min":2.226,"15min":35.747},"last":1.171}]},{"osd":6,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.4379999999999999,"15min":1.671},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.64400000000000002},"max":{"1min":2.0150000000000001,"5min":2.9809999999999999,"15min":16.231000000000002},"last":1.198},{"interface":"front","average":{"1min":1.383,"5min":1.431,"15min":1.579},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56599999999999995},"max":{"1min":1.847,"5min":4.1349999999999998,"15min":13.56},"last":1.3500000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:32 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.48,"15min":1.825},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58699999999999997},"max":{"1min":2.0089999999999999,"5min":3.2690000000000001,"15min":21.902999999999999},"last":1.306},{"interface":"front","average":{"1min":1.429,"5min":1.5169999999999999,"15min":1.8400000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.68700000000000006},"max":{"1min":1.931,"5min":3.4079999999999999,"15min":31.044},"last":1.264}]}]},{"osd":0,"up_from":14,"seq":60129542371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69204,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302636,"statfs":{"total":96636764160,"available":96565899264,"internally_reserved":0,"allocated":2039808,"data_stored":1208936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.85699999999999998,"15min":0.96799999999999997},"min":{"1min":0.59699999999999998,"5min":0.35099999999999998,"15min":0.25},"max":{"1min":1.4119999999999999,"5min":1.6100000000000001,"15min":22.619},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0680000000000001,"15min":1.194},"min":{"1min":0.65900000000000003,"5min":0.40100000000000002,"15min":0.29399999999999998},"max":{"1min":1.2549999999999999,"5min":8.2119999999999997,"15min":17.876000000000001},"last":1.1839999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.306,"15min":1.4139999999999999},"min":{"1min":0.90800000000000003,"5min":0.54100000000000004,"15min":0.42999999999999999},"max":{"1min":1.5760000000000001,"5min":6.1189999999999998,"15min":17.760999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.119,"5min":1.135,"15min":1.274},"min":{"1min":0.79700000000000004,"5min":0.46400000000000002,"15min":0.33100000000000002},"max":{"1min":1.5820000000000001,"5min":5.3159999999999998,"15min":22.672999999999998},"last":0.61699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.1579999999999999,"15min":1.2130000000000001},"min":{"1min":0.72999999999999998,"5min":0.51600000000000001,"15min":0.40000000000000002},"max":{"1min":1.401,"5min":10.83,"15min":16.382999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2270000000000001,"15min":1.3959999999999999},"min":{"1min":0.80200000000000005,"5min":0.49099999999999999,"15min":0.40999999999999998},"max":{"1min":1.4770000000000001,"5min":7.9720000000000004,"15min":17.981000000000002},"last":1.2270000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3169999999999999,"15min":1.466},"min":{"1min":0.90000000000000002,"5min":0.75600000000000001,"15min":0.51800000000000002},"max":{"1min":1.7190000000000001,"5min":2.2330000000000001,"15min":22.780000000000001},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.365,"15min":1.53},"min":{"1min":0.82799999999999996,"5min":0.80700000000000005,"15min":0.56299999999999994},"max":{"1min":1.627,"5min":10.986000000000001,"15min":19.422000000000001},"last":1.536}]},{"osd":5,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3899999999999999,"15min":1.7889999999999999},"min":{"1min":1.0780000000000001,"5min":0.80600000000000005,"15min":0.441},"max":{"1min":1.885,"5min":2.1819999999999999,"15min":44.930999999999997},"last":1.421},{"interface":"front","average":{"1min":1.296,"5min":1.4319999999999999,"15min":1.758},"min":{"1min":0.93100000000000005,"5min":0.67500000000000004,"15min":0.55400000000000005},"max":{"1min":1.754,"5min":10.928000000000001,"15min":21.797999999999998},"last":1.3600000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.5029999999999999,"15min":1.911},"min":{"1min":1.2130000000000001,"5min":0.91400000000000003,"15min":0.622},"max":{"1min":1.841,"5min":8.0399999999999991,"15min":27.442},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4690000000000001,"15min":1.859},"min":{"1min":1.131,"5min":0.85899999999999999,"15min":0.50700000000000001},"max":{"1min":1.966,"5min":2.2999999999999998,"15min":27.411999999999999},"last":1.655}]},{"osd":7,"last update":"Tue Jan 16 15:15:30 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.518,"15min":1.7090000000000001},"min":{"1min":1.1659999999999999,"5min":0.90800000000000003,"15min":0.59099999999999997},"max":{"1min":1.7,"5min":8.0960000000000001,"15min":18.152999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.655,"15min":1.7450000000000001},"min":{"1min":1.1799999999999999,"5min":0.84199999999999997,"15min":0.60099999999999998},"max":{"1min":1.796,"5min":13.718,"15min":22.815000000000001},"last":1.7450000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:44.230 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:15:44.230 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:15:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.507+0000 7f2f70000640 1 Processor -- start 2024-01-16T15:15:44.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.508+0000 7f2f70000640 1 -- start start 2024-01-16T15:15:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.509+0000 7f2f70000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2f68141360 0x7f2f680cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.509+0000 7f2f70000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6813e440 con 0x7f2f6815ed20 2024-01-16T15:15:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.509+0000 7f2f70000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f680ce1d0 con 0x7f2f680cdcc0 2024-01-16T15:15:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.509+0000 7f2f70000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f680ce4b0 con 0x7f2f68141360 2024-01-16T15:15:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f6d600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2f6815ed20 0x7f2f6815f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57462/0 (socket says 172.21.15.141:57462) 2024-01-16T15:15:44.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f6d600640 1 -- 172.21.15.141:0/2518727231 learned_addr learned my addr 172.21.15.141:0/2518727231 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:44.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679304566 0 0) 0x7f2f6813e440 con 0x7f2f6815ed20 2024-01-16T15:15:44.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f48003520 con 0x7f2f6815ed20 2024-01-16T15:15:44.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f6cc00640 1 --2- 172.21.15.141:0/2518727231 >> v2:172.21.15.188:3300/0 conn(0x7f2f68141360 0x7f2f680cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:44.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1495979835 0 0) 0x7f2f48003520 con 0x7f2f6815ed20 2024-01-16T15:15:44.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 >> v2:172.21.15.188:3300/0 conn(0x7f2f68141360 msgr2=0x7f2f680cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 --2- 172.21.15.141:0/2518727231 >> v2:172.21.15.188:3300/0 conn(0x7f2f68141360 0x7f2f680cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 >> v1:172.21.15.188:6789/0 conn(0x7f2f680cdcc0 legacy=0x7f2f6806c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f680ce790 con 0x7f2f6815ed20 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2f58001670 con 0x7f2f6815ed20 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 >> v1:172.21.15.141:6789/0 conn(0x7f2f6815ed20 legacy=0x7f2f6815f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f48003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.510+0000 7f2f67e00640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f480047c0 0x7f2f48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f67e00640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f480052a0 0x7f2f4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f680ce4b0 con 0x7f2f48003970 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4800c170 con 0x7f2f480052a0 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4800c330 con 0x7f2f480047c0 2024-01-16T15:15:44.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6d600640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f48003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6d600640 1 -- 172.21.15.141:0/2518727231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f480047c0 msgr2=0x7f2f48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6cc00640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f480047c0 0x7f2f48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6d600640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f480047c0 0x7f2f48004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6d600640 1 -- 172.21.15.141:0/2518727231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f480052a0 msgr2=0x7f2f4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6d600640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f480052a0 0x7f2f4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6d600640 1 -- 172.21.15.141:0/2518727231 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48003520 con 0x7f2f48003970 2024-01-16T15:15:44.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.511+0000 7f2f6e000640 1 -- 172.21.15.141:0/2518727231 reap_dead start 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.512+0000 7f2f6d600640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f48003d90 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c210 tx=0x7f2f480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546a6d145c96629d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.512+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005ad0 con 0x7f2f48003970 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f58005c30 con 0x7f2f48003970 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f67e00640 1 -- 172.21.15.141:0/2518727231 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5802cad0 con 0x7f2f48003970 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f70000640 1 -- 172.21.15.141:0/2518727231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 msgr2=0x7f2f48003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f70000640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f48003d90 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c210 tx=0x7f2f480057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f70000640 1 -- 172.21.15.141:0/2518727231 shutdown_connections 2024-01-16T15:15:44.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f70000640 1 --2- 172.21.15.141:0/2518727231 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f48003d90 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f70000640 1 -- 172.21.15.141:0/2518727231 >> 172.21.15.141:0/2518727231 conn(0x7f2f680b0be0 msgr2=0x7f2f680b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:44.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f70000640 1 -- 172.21.15.141:0/2518727231 shutdown_connections 2024-01-16T15:15:44.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.513+0000 7f2f70000640 1 -- 172.21.15.141:0/2518727231 wait complete. 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.515+0000 7f2f70000640 1 Processor -- start 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.515+0000 7f2f70000640 1 -- start start 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.516+0000 7f2f70000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f680a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.516+0000 7f2f70000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68138f10 0x7f2f68139370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.516+0000 7f2f70000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f6813a9e0 0x7f2f6813ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.516+0000 7f2f70000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6813b3a0 con 0x7f2f48003970 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.516+0000 7f2f70000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68139c40 con 0x7f2f6813a9e0 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.516+0000 7f2f70000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6813c190 con 0x7f2f68138f10 2024-01-16T15:15:44.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f680a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:44.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f680a9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55410/0 (socket says 172.21.15.141:55410) 2024-01-16T15:15:44.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 -- 172.21.15.141:0/2984891114 learned_addr learned my addr 172.21.15.141:0/2984891114 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6cc00640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68138f10 0x7f2f68139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 -- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68138f10 msgr2=0x7f2f68139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68138f10 0x7f2f68139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 -- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f6813a9e0 msgr2=0x7f2f6813ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f6813a9e0 0x7f2f6813ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 -- 172.21.15.141:0/2984891114 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6813b580 con 0x7f2f48003970 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6cc00640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68138f10 0x7f2f68139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.517+0000 7f2f6d600640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f680a9670 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c210 tx=0x7f2f5802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.518+0000 7f2f65600640 1 -- 172.21.15.141:0/2984891114 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5803c040 con 0x7f2f48003970 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.518+0000 7f2f65600640 1 -- 172.21.15.141:0/2984891114 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5802cde0 con 0x7f2f48003970 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.518+0000 7f2f65600640 1 -- 172.21.15.141:0/2984891114 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f580072b0 con 0x7f2f48003970 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.518+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6813b860 con 0x7f2f48003970 2024-01-16T15:15:44.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.518+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6813be20 con 0x7f2f48003970 2024-01-16T15:15:44.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.519+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f68067670 con 0x7f2f48003970 2024-01-16T15:15:44.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.520+0000 7f2f65600640 1 -- 172.21.15.141:0/2984891114 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2f58032070 con 0x7f2f48003970 2024-01-16T15:15:44.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.521+0000 7f2f65600640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f4006c4f0 0x7f2f4006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:44.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.521+0000 7f2f65600640 1 -- 172.21.15.141:0/2984891114 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f580a9e10 con 0x7f2f48003970 2024-01-16T15:15:44.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.521+0000 7f2f6cc00640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f4006c4f0 0x7f2f4006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:44.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.522+0000 7f2f6cc00640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f4006c4f0 0x7f2f4006e9b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806e1e0 tx=0x7f2f50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:44.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.525+0000 7f2f65600640 1 -- 172.21.15.141:0/2984891114 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f58076850 con 0x7f2f48003970 2024-01-16T15:15:44.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.784+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2f6813bfe0 con 0x7f2f48003970 2024-01-16T15:15:44.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.790+0000 7f2f65600640 1 -- 172.21.15.141:0/2984891114 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2f58079e70 con 0x7f2f48003970 2024-01-16T15:15:44.775 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-16T15:15:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQAGmaZls6HyOhAAzJkT3y9e+MWgwUbOZU+UQg== 2024-01-16T15:15:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: caps mds = "allow" 2024-01-16T15:15:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: caps mon = "allow r" 2024-01-16T15:15:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:15:44.775 INFO:teuthology.orchestra.run.smithi141.stderr:updated caps for client.0 2024-01-16T15:15:44.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.792+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f4006c4f0 msgr2=0x7f2f4006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.792+0000 7f2f70000640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f4006c4f0 0x7f2f4006e9b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806e1e0 tx=0x7f2f50008040 comp rx=0 tx=0).stop 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.792+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 msgr2=0x7f2f680a9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.792+0000 7f2f70000640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f680a9670 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c210 tx=0x7f2f5802e070 comp rx=0 tx=0).stop 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 shutdown_connections 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2f4006c4f0 0x7f2f4006e9b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f6813a9e0 0x7f2f6813ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2f68138f10 0x7f2f68139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 --2- 172.21.15.141:0/2984891114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f48003970 0x7f2f680a9670 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 >> 172.21.15.141:0/2984891114 conn(0x7f2f680b0be0 msgr2=0x7f2f68140750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 shutdown_connections 2024-01-16T15:15:44.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:44.793+0000 7f2f70000640 1 -- 172.21.15.141:0/2984891114 wait complete. 2024-01-16T15:15:44.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:44.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.014+0000 7ff9e6800640 1 Processor -- start 2024-01-16T15:15:44.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.014+0000 7ff9e6800640 1 -- start start 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e6800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff9e0071f00 0x7ff9e0146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e6800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e006c7c0 0x7ff9e0166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e6800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e00b2e10 con 0x7ff9e0071410 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e6800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e006cc10 con 0x7ff9e0071f00 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e6800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e006cec0 con 0x7ff9e006c7c0 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e4a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff9e0071f00 0x7ff9e0146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e4a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff9e0071f00 0x7ff9e0146770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55412/0 (socket says 172.21.15.141:55412) 2024-01-16T15:15:45.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e4a00640 1 -- 172.21.15.141:0/1386396948 learned_addr learned my addr 172.21.15.141:0/1386396948 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:45.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e5e00640 1 --2- 172.21.15.141:0/1386396948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e006c7c0 0x7ff9e0166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e4a00640 1 -- 172.21.15.141:0/1386396948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e006c7c0 msgr2=0x7ff9e0166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e4a00640 1 --2- 172.21.15.141:0/1386396948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e006c7c0 0x7ff9e0166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e4a00640 1 -- 172.21.15.141:0/1386396948 >> v1:172.21.15.188:6789/0 conn(0x7ff9e0071410 legacy=0x7ff9e00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.016+0000 7ff9e4a00640 1 -- 172.21.15.141:0/1386396948 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e006d1a0 con 0x7ff9e0071f00 2024-01-16T15:15:45.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.018+0000 7ff9e4a00640 1 --2- 172.21.15.141:0/1386396948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e0071f00 0x7ff9e0146770 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009870 tx=0x7ff9c802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c2439a60c20a9a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.018+0000 7ff9dfe00640 1 -- 172.21.15.141:0/1386396948 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80317c0 con 0x7ff9e0071f00 2024-01-16T15:15:45.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.018+0000 7ff9dfe00640 1 -- 172.21.15.141:0/1386396948 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8031920 con 0x7ff9e0071f00 2024-01-16T15:15:45.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.018+0000 7ff9dfe00640 1 -- 172.21.15.141:0/1386396948 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8031cb0 con 0x7ff9e0071f00 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 -- 172.21.15.141:0/1386396948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e0071f00 msgr2=0x7ff9e0146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 --2- 172.21.15.141:0/1386396948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e0071f00 0x7ff9e0146770 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009870 tx=0x7ff9c802b860 comp rx=0 tx=0).stop 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 -- 172.21.15.141:0/1386396948 shutdown_connections 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 --2- 172.21.15.141:0/1386396948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e0071f00 0x7ff9e0146770 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 --2- 172.21.15.141:0/1386396948 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e006c7c0 0x7ff9e0166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 -- 172.21.15.141:0/1386396948 >> 172.21.15.141:0/1386396948 conn(0x7ff9e006a800 msgr2=0x7ff9e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 -- 172.21.15.141:0/1386396948 shutdown_connections 2024-01-16T15:15:45.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.019+0000 7ff9e6800640 1 -- 172.21.15.141:0/1386396948 wait complete. 2024-01-16T15:15:45.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.021+0000 7ff9e6800640 1 Processor -- start 2024-01-16T15:15:45.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.021+0000 7ff9e6800640 1 -- start start 2024-01-16T15:15:45.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.022+0000 7ff9e6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e006c7c0 0x7ff9e015f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.022+0000 7ff9e6800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e0071410 0x7ff9e01f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.022+0000 7ff9e6800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e0071f00 0x7ff9e015fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.022+0000 7ff9e6800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e01f2830 con 0x7ff9e006c7c0 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.022+0000 7ff9e6800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0160a20 con 0x7ff9e0071f00 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.022+0000 7ff9e6800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e015e620 con 0x7ff9e0071410 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e006c7c0 0x7ff9e015f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e006c7c0 0x7ff9e015f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55418/0 (socket says 172.21.15.141:55418) 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 -- 172.21.15.141:0/584313527 learned_addr learned my addr 172.21.15.141:0/584313527 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 -- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e0071410 msgr2=0x7ff9e01f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e4a00640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e0071410 0x7ff9e01f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5e00640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e0071f00 0x7ff9e015fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e0071410 0x7ff9e01f22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 -- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e0071f00 msgr2=0x7ff9e015fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e0071f00 0x7ff9e015fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.023+0000 7ff9e5400640 1 -- 172.21.15.141:0/584313527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c80094d0 con 0x7ff9e006c7c0 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.024+0000 7ff9e5400640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e006c7c0 0x7ff9e015f0c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000c430 tx=0x7ff9d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.024+0000 7ff9dd600640 1 -- 172.21.15.141:0/584313527 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0011790 con 0x7ff9e006c7c0 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.024+0000 7ff9dd600640 1 -- 172.21.15.141:0/584313527 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d00118f0 con 0x7ff9e006c7c0 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.024+0000 7ff9dd600640 1 -- 172.21.15.141:0/584313527 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0011ce0 con 0x7ff9e006c7c0 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.024+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e0160740 con 0x7ff9e006c7c0 2024-01-16T15:15:45.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.024+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e015eba0 con 0x7ff9e006c7c0 2024-01-16T15:15:45.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.025+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e006aef0 con 0x7ff9e006c7c0 2024-01-16T15:15:45.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.026+0000 7ff9dd600640 1 -- 172.21.15.141:0/584313527 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff9d001c460 con 0x7ff9e006c7c0 2024-01-16T15:15:45.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.027+0000 7ff9dd600640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9b806c4c0 0x7ff9b806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.027+0000 7ff9dd600640 1 -- 172.21.15.141:0/584313527 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9d0088ab0 con 0x7ff9e006c7c0 2024-01-16T15:15:45.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.028+0000 7ff9e4a00640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9b806c4c0 0x7ff9b806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.029+0000 7ff9e4a00640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9b806c4c0 0x7ff9b806e980 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8036000 tx=0x7ff9c803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.032+0000 7ff9dd600640 1 -- 172.21.15.141:0/584313527 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9d0055060 con 0x7ff9e006c7c0 2024-01-16T15:15:45.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.267+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9e0067450 con 0x7ff9e006c7c0 2024-01-16T15:15:45.252 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.268+0000 7ff9dd600640 1 -- 172.21.15.141:0/584313527 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4924 (secure 0 0 0) 0x7ff9d0058680 con 0x7ff9e006c7c0 2024-01-16T15:15:45.253 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:15:45.253 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3065530834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3065530834},{"type":"v1","addr":"172.21.15.188:6835","nonce":3065530834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6140,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/1935621003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1935621003},{"type":"v1","addr":"172.21.15.141:6834","nonce":1935621003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6146,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/935439310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":935439310},{"type":"v1","addr":"172.21.15.188:6837","nonce":935439310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:38.721379+0000","modified":"2024-01-16T15:15:41.817485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1556719184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":1556719184},{"type":"v1","addr":"172.21.15.141:6835","nonce":1556719184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:15:45.253 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 73 2024-01-16T15:15:45.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9b806c4c0 msgr2=0x7ff9b806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9b806c4c0 0x7ff9b806e980 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8036000 tx=0x7ff9c803d040 comp rx=0 tx=0).stop 2024-01-16T15:15:45.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e006c7c0 msgr2=0x7ff9e015f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e006c7c0 0x7ff9e015f0c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000c430 tx=0x7ff9d000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 shutdown_connections 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff9b806c4c0 0x7ff9b806e980 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e0071f00 0x7ff9e015fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff9e0071410 0x7ff9e01f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 --2- 172.21.15.141:0/584313527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9e006c7c0 0x7ff9e015f0c0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.271+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 >> 172.21.15.141:0/584313527 conn(0x7ff9e006a800 msgr2=0x7ff9e0145510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.272+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 shutdown_connections 2024-01-16T15:15:45.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.272+0000 7ff9e6800640 1 -- 172.21.15.141:0/584313527 wait complete. 2024-01-16T15:15:45.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:15:38.721379+0000', 'modified': '2024-01-16T15:15:41.817485+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9168}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9168': {'gid': 9168, 'name': 'a', 'rank': 0, 'incarnation': 69, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6835/1556719184', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6832', 'nonce': 1556719184}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 1556719184}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:15:45.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:15:45.679 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T15:15:45.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.913+0000 7fa7e7600640 1 Processor -- start 2024-01-16T15:15:45.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.913+0000 7fa7e7600640 1 -- start start 2024-01-16T15:15:45.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7e7600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa7e00b3ae0 0x7fa7e015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7e7600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00cf290 0x7fa7e00cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7e7600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e0070c40 con 0x7fa7e00ae5a0 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7e7600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e0071450 con 0x7fa7e00b3ae0 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7e7600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00cfc60 con 0x7fa7e00cf290 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7dfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa7e00b3ae0 0x7fa7e015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7e4c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa7e00ae5a0 0x7fa7e006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57474/0 (socket says 172.21.15.141:57474) 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.915+0000 7fa7e4c00640 1 -- 172.21.15.141:0/3916926601 learned_addr learned my addr 172.21.15.141:0/3916926601 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3916926601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00cf290 0x7fa7e00cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7df400640 1 -- 172.21.15.141:0/3916926601 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960178737 0 0) 0x7fa7e0070c40 con 0x7fa7e00ae5a0 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7df400640 1 -- 172.21.15.141:0/3916926601 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c0003520 con 0x7fa7e00ae5a0 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7dfe00640 1 -- 172.21.15.141:0/3916926601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00cf290 msgr2=0x7fa7e00cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7dfe00640 1 --2- 172.21.15.141:0/3916926601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00cf290 0x7fa7e00cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7dfe00640 1 -- 172.21.15.141:0/3916926601 >> v1:172.21.15.141:6789/0 conn(0x7fa7e00ae5a0 legacy=0x7fa7e006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7dfe00640 1 -- 172.21.15.141:0/3916926601 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e006c470 con 0x7fa7e00b3ae0 2024-01-16T15:15:45.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.916+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3916926601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00cf290 0x7fa7e00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.917+0000 7fa7dfe00640 1 --2- 172.21.15.141:0/3916926601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00b3ae0 0x7fa7e015e430 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80097a0 tx=0x7fa7c802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1bb7745ae55a43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.917+0000 7fa7df400640 1 -- 172.21.15.141:0/3916926601 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8030800 con 0x7fa7e00b3ae0 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.917+0000 7fa7df400640 1 -- 172.21.15.141:0/3916926601 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c8030960 con 0x7fa7e00b3ae0 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.917+0000 7fa7df400640 1 -- 172.21.15.141:0/3916926601 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8030cf0 con 0x7fa7e00b3ae0 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 -- 172.21.15.141:0/3916926601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00b3ae0 msgr2=0x7fa7e015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 --2- 172.21.15.141:0/3916926601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00b3ae0 0x7fa7e015e430 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80097a0 tx=0x7fa7c802b940 comp rx=0 tx=0).stop 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 -- 172.21.15.141:0/3916926601 shutdown_connections 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 --2- 172.21.15.141:0/3916926601 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00cf290 0x7fa7e00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 --2- 172.21.15.141:0/3916926601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00b3ae0 0x7fa7e015e430 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 -- 172.21.15.141:0/3916926601 >> 172.21.15.141:0/3916926601 conn(0x7fa7e006a800 msgr2=0x7fa7e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 -- 172.21.15.141:0/3916926601 shutdown_connections 2024-01-16T15:15:45.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.918+0000 7fa7e7600640 1 -- 172.21.15.141:0/3916926601 wait complete. 2024-01-16T15:15:45.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.920+0000 7fa7e7600640 1 Processor -- start 2024-01-16T15:15:45.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.920+0000 7fa7e7600640 1 -- start start 2024-01-16T15:15:45.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e7600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00ae5a0 0x7fa7e0150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e7600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e00b3ae0 0x7fa7e01f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e7600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00cf290 0x7fa7e01f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e7600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e01f6390 con 0x7fa7e00cf290 2024-01-16T15:15:45.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e7600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e01f6670 con 0x7fa7e00b3ae0 2024-01-16T15:15:45.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e7600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e01f2c30 con 0x7fa7e00ae5a0 2024-01-16T15:15:45.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00cf290 0x7fa7e01f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00cf290 0x7fa7e01f6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55432/0 (socket says 172.21.15.141:55432) 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.921+0000 7fa7e5600640 1 -- 172.21.15.141:0/206341820 learned_addr learned my addr 172.21.15.141:0/206341820 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00ae5a0 0x7fa7e0150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7dfe00640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e00b3ae0 0x7fa7e01f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e5600640 1 -- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00ae5a0 msgr2=0x7fa7e0150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e5600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00ae5a0 0x7fa7e0150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e5600640 1 -- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e00b3ae0 msgr2=0x7fa7e01f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e5600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e00b3ae0 0x7fa7e01f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e5600640 1 -- 172.21.15.141:0/206341820 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c8009400 con 0x7fa7e00cf290 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00ae5a0 0x7fa7e0150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e5600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00cf290 0x7fa7e01f6b00 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007df0 tx=0x7fa7d800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7dcc00640 1 -- 172.21.15.141:0/206341820 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8003990 con 0x7fa7e00cf290 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7dcc00640 1 -- 172.21.15.141:0/206341820 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d800f040 con 0x7fa7e00cf290 2024-01-16T15:15:45.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7dcc00640 1 -- 172.21.15.141:0/206341820 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8003d30 con 0x7fa7e00cf290 2024-01-16T15:15:45.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.922+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e0153550 con 0x7fa7e00cf290 2024-01-16T15:15:45.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.923+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e0150fe0 con 0x7fa7e00cf290 2024-01-16T15:15:45.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.925+0000 7fa7dcc00640 1 -- 172.21.15.141:0/206341820 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa7d8004030 con 0x7fa7e00cf290 2024-01-16T15:15:45.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.925+0000 7fa7dcc00640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7b806c4c0 0x7fa7b806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.910 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.926+0000 7fa7dcc00640 1 -- 172.21.15.141:0/206341820 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7d8088e60 con 0x7fa7e00cf290 2024-01-16T15:15:45.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.926+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7b806c4c0 0x7fa7b806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.926+0000 7fa7ea401740 1 Processor -- start 2024-01-16T15:15:45.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.926+0000 7fa7ea401740 1 -- start start 2024-01-16T15:15:45.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.927+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4ba4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.927+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4ba30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.927+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 0x5583d4be7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.927+0000 7fa7ea401740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d4b02880 con 0x5583d4ba3ea0 2024-01-16T15:15:45.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.927+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d4b11ae0 con 0x5583d4b9eba0 2024-01-16T15:15:45.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.927+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583d4b0f1d0 con 0x5583d4ba35e0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4ba4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4ba4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55436/0 (socket says 172.21.15.141:55436) 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 -- 172.21.15.141:0/931417718 learned_addr learned my addr 172.21.15.141:0/931417718 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7b806c4c0 0x7fa7b806e980 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00011e0 tx=0x7fa7d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 -- 172.21.15.141:0/931417718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 msgr2=0x5583d4be7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 0x5583d4be7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 -- 172.21.15.141:0/931417718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 msgr2=0x5583d4ba30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7dfe00640 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4ba30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4ba30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 -- 172.21.15.141:0/931417718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583d4ad2860 con 0x5583d4ba3ea0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.928+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4ba4280 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000d900 tx=0x7fa7d0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7979f50e87fc7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7d5600640 1 -- 172.21.15.141:0/931417718 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0013e00 con 0x5583d4ba3ea0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7d5600640 1 -- 172.21.15.141:0/931417718 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d000de80 con 0x5583d4ba3ea0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7d5600640 1 -- 172.21.15.141:0/931417718 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0005ad0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7ea401740 1 -- 172.21.15.141:0/931417718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 msgr2=0x5583d4ba4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7ea401740 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4ba4280 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000d900 tx=0x7fa7d0010000 comp rx=0 tx=0).stop 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7ea401740 1 -- 172.21.15.141:0/931417718 shutdown_connections 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7ea401740 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 0x5583d4be7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7ea401740 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4ba30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7ea401740 1 --2- 172.21.15.141:0/931417718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4ba4280 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.929+0000 7fa7ea401740 1 -- 172.21.15.141:0/931417718 >> 172.21.15.141:0/931417718 conn(0x5583d4b65c70 msgr2=0x5583d4be9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.930+0000 7fa7ea401740 1 -- 172.21.15.141:0/931417718 shutdown_connections 2024-01-16T15:15:45.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.930+0000 7fa7ea401740 1 -- 172.21.15.141:0/931417718 wait complete. 2024-01-16T15:15:45.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.930+0000 7fa7ea401740 1 -- start start 2024-01-16T15:15:45.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.930+0000 7fa7ea401740 1 Processor -- start 2024-01-16T15:15:45.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.931+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 0x5583d4bc8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.931+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4bcd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.920 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.931+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4be5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.931+0000 7fa7ea401740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d4b0aac0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.931+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d4aa72f0 con 0x5583d4ba35e0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.931+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583d4ac4c70 con 0x5583d4b9eba0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7ea401740 10 client.0 _ll_register_callbacks cb 0x5583d4ac93b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4be5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4be5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55444/0 (socket says 172.21.15.141:55444) 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 -- 172.21.15.141:0/4215166043 learned_addr learned my addr 172.21.15.141:0/4215166043 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 -- 172.21.15.141:0/4215166043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 msgr2=0x5583d4bc8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 0x5583d4bc8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7dfe00640 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4bcd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 -- 172.21.15.141:0/4215166043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 msgr2=0x5583d4bcd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4bcd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 -- 172.21.15.141:0/4215166043 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c0003520 con 0x5583d4ba3ea0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.932+0000 7fa7e5600640 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4be5e80 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8058d70 tx=0x7fa7d80598b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705dc97cf938f0a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.921 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ce000640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ce000640 1 -- 172.21.15.141:0/4215166043 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d805f2d0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ce000640 1 -- 172.21.15.141:0/4215166043 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7d80592d0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583d4b037c0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ce000640 1 -- 172.21.15.141:0/4215166043 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7d8059490 con 0x5583d4ba3ea0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 10 client.9297 fetch_fsmap learned FSMap version 73 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x5583d4b09140 con 0x5583d4ba3ea0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ce000640 1 -- 172.21.15.141:0/4215166043 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5583d4b037c0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ce000640 1 -- 172.21.15.141:0/4215166043 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa7d8012070 con 0x5583d4ba3ea0 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 10 client.9297 fetch_fsmap finished waiting for FSMap version 73 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 10 client.9297 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 20 client.9297 populate_metadata read hostname 'smithi141' 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bccd40 0x5583d4c06db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 4 client.9297 mds_command: new command op to 9168 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:15:45.922 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.933+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 --> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583d480b6c0 con 0x5583d4bccd40 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.934+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bccd40 0x5583d4c06db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.934+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bccd40 0x5583d4c06db0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1574f7d68ea2bf2a server_cookie=3e8741e049cfafd5 in_seq=0 out_seq=0 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.934+0000 7fa7ce000640 10 client.9297 ms_handle_connect on v2:172.21.15.141:6832/1556719184 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ce000640 1 -- 172.21.15.141:0/4215166043 <== mds.0 v2:172.21.15.141:6832/1556719184 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5583d480b6c0 con 0x5583d4bccd40 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ce000640 10 client.9297 handle_command_reply: tid=0 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 client.9297 shutdown 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 msgr2=0x5583d4be5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4be5e80 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8058d70 tx=0x7fa7d80598b0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 shutdown_connections 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bccd40 0x5583d4c06db0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4be5e80 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4bcd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 --2- 172.21.15.141:0/4215166043 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 0x5583d4bc8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 >> 172.21.15.141:0/4215166043 conn(0x5583d4b65c70 msgr2=0x5583d4becad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.935+0000 7fa7ce000640 0 client.9297 ms_handle_reset on v2:172.21.15.141:6832/1556719184 2024-01-16T15:15:45.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.936+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 shutdown_connections 2024-01-16T15:15:45.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.936+0000 7fa7ea401740 1 -- 172.21.15.141:0/4215166043 wait complete. 2024-01-16T15:15:45.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.936+0000 7fa7ea401740 20 client.9297 trim_cache size 0 max 16384 2024-01-16T15:15:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.953+0000 7fa7ea401740 1 Processor -- start 2024-01-16T15:15:45.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.953+0000 7fa7ea401740 1 -- start start 2024-01-16T15:15:45.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4beff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 0x5583d4b9db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4bcd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7ea401740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d486df20 con 0x5583d4ba3ea0 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d4bf9460 con 0x5583d4b9eba0 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583d4ba2790 con 0x5583d4ba35e0 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4bcd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4bcd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55450/0 (socket says 172.21.15.141:55450) 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7e5600640 1 -- 172.21.15.141:0/3730911448 learned_addr learned my addr 172.21.15.141:0/3730911448 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.954+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4beff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7dfe00640 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 0x5583d4b9db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7e5600640 1 -- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 msgr2=0x5583d4b9db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 0x5583d4b9db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7e5600640 1 -- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 msgr2=0x5583d4beff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4beff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7e5600640 1 -- 172.21.15.141:0/3730911448 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583d4aa6b80 con 0x5583d4ba3ea0 2024-01-16T15:15:45.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4beff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:45.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4bcd4f0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8055570 tx=0x7fa7d8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2978fbf1e5e657cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7d6000640 1 -- 172.21.15.141:0/3730911448 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8088cb0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7d6000640 1 -- 172.21.15.141:0/3730911448 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d80592d0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.955+0000 7fa7d6000640 1 -- 172.21.15.141:0/3730911448 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d805f0d0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 -- 172.21.15.141:0/3730911448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 msgr2=0x5583d4bcd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4bcd4f0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8055570 tx=0x7fa7d8004ab0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 -- 172.21.15.141:0/3730911448 shutdown_connections 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4bcd4f0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4ba35e0 0x5583d4b9db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3730911448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4b9eba0 0x5583d4beff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 -- 172.21.15.141:0/3730911448 >> 172.21.15.141:0/3730911448 conn(0x5583d4b65c70 msgr2=0x5583d4be5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 -- 172.21.15.141:0/3730911448 shutdown_connections 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.956+0000 7fa7ea401740 1 -- 172.21.15.141:0/3730911448 wait complete. 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.957+0000 7fa7ea401740 1 -- start start 2024-01-16T15:15:45.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.957+0000 7fa7ea401740 1 Processor -- start 2024-01-16T15:15:45.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 0x5583d4b9f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4c5f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7ea401740 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4c5fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7ea401740 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d4c09280 con 0x5583d4ba3ea0 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583d4c603f0 con 0x5583d4ba35e0 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7ea401740 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583d4be8cd0 con 0x5583d4b9eba0 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7ea401740 10 client.0 _ll_register_callbacks cb 0x7fa7d8009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4c5fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4c5fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55466/0 (socket says 172.21.15.141:55466) 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7e5600640 1 -- 172.21.15.141:0/3179053175 learned_addr learned my addr 172.21.15.141:0/3179053175 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7e5600640 1 -- 172.21.15.141:0/3179053175 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 msgr2=0x5583d4b9f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7dfe00640 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4c5f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 0x5583d4b9f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7e5600640 1 -- 172.21.15.141:0/3179053175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 msgr2=0x5583d4c5f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.958+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4c5f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7e5600640 1 -- 172.21.15.141:0/3179053175 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583d4b09140 con 0x5583d4ba3ea0 2024-01-16T15:15:45.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7e5600640 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4c5fb30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8061530 tx=0x7fa7d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d8737cabade1b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ce000640 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ce000640 1 -- 172.21.15.141:0/3179053175 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80610e0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ce000640 1 -- 172.21.15.141:0/3179053175 <== mon.0 v2:172.21.15.141:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7d80779a0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583d4b03aa0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ce000640 1 -- 172.21.15.141:0/3179053175 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7d805f2e0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ea401740 10 client.9303 fetch_fsmap learned FSMap version 73 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x5583d4acc170 con 0x5583d4ba3ea0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.959+0000 7fa7ce000640 1 -- 172.21.15.141:0/3179053175 <== mon.0 v2:172.21.15.141:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5583d4b03aa0 con 0x5583d4ba3ea0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ce000640 1 -- 172.21.15.141:0/3179053175 <== mon.0 v2:172.21.15.141:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa7d8012040 con 0x5583d4ba3ea0 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ea401740 10 client.9303 fetch_fsmap finished waiting for FSMap version 73 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ea401740 10 client.9303 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ea401740 20 client.9303 populate_metadata read hostname 'smithi141' 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bcd4f0 0x5583d4bcd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ea401740 4 client.9303 mds_command: new command op to 9168 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:15:45.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 --> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5583d485ed10 con 0x5583d4bcd4f0 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bcd4f0 0x5583d4bcd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7ce000640 10 client.9303 ms_handle_connect on v2:172.21.15.141:6832/1556719184 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.960+0000 7fa7e4c00640 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bcd4f0 0x5583d4bcd8d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61920d517109f8bd server_cookie=94414b6a4479afe7 in_seq=0 out_seq=0 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ce000640 1 -- 172.21.15.141:0/3179053175 <== mds.0 v2:172.21.15.141:6832/1556719184 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5583d485ed10 con 0x5583d4bcd4f0 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ce000640 10 client.9303 handle_command_reply: tid=0 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 client.9303 shutdown 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 msgr2=0x5583d4c5fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4c5fb30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8061530 tx=0x7fa7d8017000 comp rx=0 tx=0).stop 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 shutdown_connections 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:6832/1556719184,v1:172.21.15.141:6835/1556719184] conn(0x5583d4bcd4f0 0x5583d4bcd8d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5583d4ba3ea0 0x5583d4c5fb30 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5583d4ba35e0 0x5583d4c5f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ce000640 0 client.9303 ms_handle_reset on v2:172.21.15.141:6832/1556719184 2024-01-16T15:15:45.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 --2- 172.21.15.141:0/3179053175 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x5583d4b9eba0 0x5583d4b9f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 >> 172.21.15.141:0/3179053175 conn(0x5583d4b65c70 msgr2=0x5583d4be5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 shutdown_connections 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 1 -- 172.21.15.141:0/3179053175 wait complete. 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.961+0000 7fa7ea401740 20 client.9303 trim_cache size 0 max 16384 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stdout: "cluster_fsid": "540b621b-477a-463b-8a92-7e448bae48f5", 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stdout: "whoami": 0, 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 9168, 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stdout: "want_state": "up:active", 2024-01-16T15:15:45.951 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "up:active", 2024-01-16T15:15:45.952 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs", 2024-01-16T15:15:45.952 INFO:teuthology.orchestra.run.smithi141.stdout: "rank_uptime": 6.6475151160000001, 2024-01-16T15:15:45.952 INFO:teuthology.orchestra.run.smithi141.stdout: "mdsmap_epoch": 73, 2024-01-16T15:15:45.952 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch": 116, 2024-01-16T15:15:45.952 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_epoch_barrier": 116, 2024-01-16T15:15:45.952 INFO:teuthology.orchestra.run.smithi141.stdout: "uptime": 13.091961878999999 2024-01-16T15:15:45.952 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:15:45.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7b806c4c0 msgr2=0x7fa7b806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7b806c4c0 0x7fa7b806e980 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00011e0 tx=0x7fa7d0009040 comp rx=0 tx=0).stop 2024-01-16T15:15:45.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00cf290 msgr2=0x7fa7e01f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:45.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00cf290 0x7fa7e01f6b00 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007df0 tx=0x7fa7d800d4a0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 shutdown_connections 2024-01-16T15:15:45.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa7b806c4c0 0x7fa7b806e980 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7e00cf290 0x7fa7e01f6b00 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e00b3ae0 0x7fa7e01f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 --2- 172.21.15.141:0/206341820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa7e00ae5a0 0x7fa7e0150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:45.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 >> 172.21.15.141:0/206341820 conn(0x7fa7e006a800 msgr2=0x7fa7e015a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:45.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 shutdown_connections 2024-01-16T15:15:45.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:45.962+0000 7fa7e7600640 1 -- 172.21.15.141:0/206341820 wait complete. 2024-01-16T15:15:45.964 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:46.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.186+0000 7f819e200640 1 Processor -- start 2024-01-16T15:15:46.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.186+0000 7f819e200640 1 -- start start 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.187+0000 7f819e200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f819813df80 0x7f819815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.187+0000 7f819e200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81980cf9a0 0x7f81980cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.188+0000 7f819e200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819806c470 con 0x7f819813e360 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.188+0000 7f819e200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819806c750 con 0x7f819813df80 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.188+0000 7f819e200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819806ca30 con 0x7f81980cf9a0 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.188+0000 7f8197e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81980cf9a0 0x7f81980cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.188+0000 7f8197e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81980cf9a0 0x7f81980cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:33306/0 (socket says 172.21.15.141:33306) 2024-01-16T15:15:46.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.188+0000 7f8197e00640 1 -- 172.21.15.141:0/2230902694 learned_addr learned my addr 172.21.15.141:0/2230902694 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:46.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.188+0000 7f8197400640 1 --2- 172.21.15.141:0/2230902694 >> v2:172.21.15.188:3300/0 conn(0x7f819813df80 0x7f819815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:46.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.189+0000 7f8197e00640 1 -- 172.21.15.141:0/2230902694 >> v2:172.21.15.188:3300/0 conn(0x7f819813df80 msgr2=0x7f819815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:46.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.189+0000 7f8197e00640 1 --2- 172.21.15.141:0/2230902694 >> v2:172.21.15.188:3300/0 conn(0x7f819813df80 0x7f819815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.189+0000 7f8197e00640 1 -- 172.21.15.141:0/2230902694 >> v1:172.21.15.188:6789/0 conn(0x7f819813e360 legacy=0x7f81980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:46.174 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.189+0000 7f8197e00640 1 -- 172.21.15.141:0/2230902694 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819806cd10 con 0x7f81980cf9a0 2024-01-16T15:15:46.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.190+0000 7f8197e00640 1 --2- 172.21.15.141:0/2230902694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81980cf9a0 0x7f81980cd840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8190009e50 tx=0x7f819002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83d6b21af962801 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:46.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.191+0000 7f8196a00640 1 -- 172.21.15.141:0/2230902694 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819002cb00 con 0x7f81980cf9a0 2024-01-16T15:15:46.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.191+0000 7f8196a00640 1 -- 172.21.15.141:0/2230902694 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819002cc60 con 0x7f81980cf9a0 2024-01-16T15:15:46.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.191+0000 7f8196a00640 1 -- 172.21.15.141:0/2230902694 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81900343f0 con 0x7f81980cf9a0 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 -- 172.21.15.141:0/2230902694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81980cf9a0 msgr2=0x7f81980cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 --2- 172.21.15.141:0/2230902694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81980cf9a0 0x7f81980cd840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8190009e50 tx=0x7f819002bf60 comp rx=0 tx=0).stop 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 -- 172.21.15.141:0/2230902694 shutdown_connections 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 --2- 172.21.15.141:0/2230902694 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81980cf9a0 0x7f81980cd840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 --2- 172.21.15.141:0/2230902694 >> v2:172.21.15.188:3300/0 conn(0x7f819813df80 0x7f819815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 -- 172.21.15.141:0/2230902694 >> 172.21.15.141:0/2230902694 conn(0x7f819806a800 msgr2=0x7f819806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 -- 172.21.15.141:0/2230902694 shutdown_connections 2024-01-16T15:15:46.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.192+0000 7f819e200640 1 -- 172.21.15.141:0/2230902694 wait complete. 2024-01-16T15:15:46.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.194+0000 7f819e200640 1 Processor -- start 2024-01-16T15:15:46.179 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.194+0000 7f819e200640 1 -- start start 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.195+0000 7f819e200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81980cf9a0 0x7f819813ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.195+0000 7f819e200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819813df80 0x7f819813a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f819e200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f819813e360 0x7f81981524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f819e200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819813a7a0 con 0x7f81980cf9a0 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f819e200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819813a9d0 con 0x7f819813df80 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f819e200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819813acb0 con 0x7f819813e360 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81980cf9a0 0x7f819813ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:46.182 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81980cf9a0 0x7f819813ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55482/0 (socket says 172.21.15.141:55482) 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 -- 172.21.15.141:0/3331944979 learned_addr learned my addr 172.21.15.141:0/3331944979 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 -- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f819813e360 msgr2=0x7f81981524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f819813e360 0x7f81981524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 -- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819813df80 msgr2=0x7f819813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f818fe00640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819813df80 0x7f819813a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819813df80 0x7f819813a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.196+0000 7f8197400640 1 -- 172.21.15.141:0/3331944979 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8190009b30 con 0x7f81980cf9a0 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.197+0000 7f8197400640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81980cf9a0 0x7f819813ba70 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f818800e9b0 tx=0x7f818800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.197+0000 7f818f400640 1 -- 172.21.15.141:0/3331944979 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188009670 con 0x7f81980cf9a0 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.197+0000 7f818f400640 1 -- 172.21.15.141:0/3331944979 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818800c9a0 con 0x7f81980cf9a0 2024-01-16T15:15:46.183 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.197+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819813af90 con 0x7f81980cf9a0 2024-01-16T15:15:46.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.197+0000 7f818f400640 1 -- 172.21.15.141:0/3331944979 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818800cd50 con 0x7f81980cf9a0 2024-01-16T15:15:46.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.197+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819813b550 con 0x7f81980cf9a0 2024-01-16T15:15:46.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.198+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81980675d0 con 0x7f81980cf9a0 2024-01-16T15:15:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.199+0000 7f818f400640 1 -- 172.21.15.141:0/3331944979 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f8188013470 con 0x7f81980cf9a0 2024-01-16T15:15:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.200+0000 7f818f400640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f817006c4c0 0x7f817006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.200+0000 7f818f400640 1 -- 172.21.15.141:0/3331944979 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8188088da0 con 0x7f81980cf9a0 2024-01-16T15:15:46.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.201+0000 7f818fe00640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f817006c4c0 0x7f817006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:46.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.202+0000 7f818fe00640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f817006c4c0 0x7f817006e980 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f81980b09c0 tx=0x7f8180009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:46.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.204+0000 7f818f400640 1 -- 172.21.15.141:0/3331944979 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8188055350 con 0x7f81980cf9a0 2024-01-16T15:15:46.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.437+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f819815b0f0 con 0x7f81980cf9a0 2024-01-16T15:15:46.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.438+0000 7f818f400640 1 -- 172.21.15.141:0/3331944979 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8188017070 con 0x7f81980cf9a0 2024-01-16T15:15:46.423 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:46.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.440+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f817006c4c0 msgr2=0x7f817006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:46.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.440+0000 7f819e200640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f817006c4c0 0x7f817006e980 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f81980b09c0 tx=0x7f8180009040 comp rx=0 tx=0).stop 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.440+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81980cf9a0 msgr2=0x7f819813ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.440+0000 7f819e200640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81980cf9a0 0x7f819813ba70 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f818800e9b0 tx=0x7f818800ee70 comp rx=0 tx=0).stop 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 shutdown_connections 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f817006c4c0 0x7f817006e980 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f819813e360 0x7f81981524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819813df80 0x7f819813a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 --2- 172.21.15.141:0/3331944979 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81980cf9a0 0x7f819813ba70 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 >> 172.21.15.141:0/3331944979 conn(0x7f819806a800 msgr2=0x7f81980cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 shutdown_connections 2024-01-16T15:15:46.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:46.441+0000 7f819e200640 1 -- 172.21.15.141:0/3331944979 wait complete. 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:46.443 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:46.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:46.443 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-16T15:15:46.495 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:46.496 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 41517sec preferred_lft 41517sec 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::ec4:7aff:fe88:726d/64 scope link 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::3c9b:8bff:fe1e:bb94/64 scope link 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether b2:6e:b1:32:61:52 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::b06e:b1ff:fe32:6152/64 scope link 2024-01-16T15:15:46.497 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:46.498 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:46.498 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0a:82:e9:92:c5:4c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:46.498 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::882:e9ff:fe92:c54c/64 scope link 2024-01-16T15:15:46.498 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:46.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:46.499 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:46.586 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-16T15:15:46.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:46.591 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-16T15:15:46.644 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-16T15:15:46.645 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:46.645 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:15:46.645 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:15:46.645 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:46.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:46.645 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-16T15:15:46.645 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-16T15:15:46.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:15:46.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:15:46.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:15:46.646 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-16T15:15:46.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:46.827 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T15:15:46.882 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:15:46.882 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:15:46.882 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:46.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:46.883 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-16T15:15:46.977 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:15:46.977 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:46.977 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:47.029 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:47.030 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:47.030 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:47.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:15:47.176 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:47.228 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:15:47.228 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:15:47.228 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:15:47.228 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:15:47.228 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:15:47.228 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:15:47.228 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:47.230 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:47.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:47.231 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:15:47.279 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:47.280 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:15:47.284 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe fuse 2024-01-16T15:15:47.361 DEBUG:teuthology.orchestra.run.smithi141:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/proc 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/sys 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/dev 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/security 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/shm 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/pts 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/run 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/cgroup 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/pstore 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/ 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/selinux 2024-01-16T15:15:47.402 INFO:teuthology.orchestra.run.smithi141.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/mqueue 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/hugepages 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/debug 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/tracing 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/sys/kernel/config 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/run/user/1000 2024-01-16T15:15:47.403 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:47.404 INFO:teuthology.orchestra.run.smithi141.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:47.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:47.406 DEBUG:teuthology.orchestra.run.smithi141:> ls /sys/fs/fuse/connections 2024-01-16T15:15:47.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:2024-01-16T15:15:47.486+0000 7f2cec63a480 -1 init, newargv = 0x557bf1238900 newargc=15 2024-01-16T15:15:47.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[76610]: starting ceph client 2024-01-16T15:15:47.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[76610]: starting fuse 2024-01-16T15:15:47.492 INFO:teuthology.orchestra.run.smithi141.stdout:40 2024-01-16T15:15:47.492 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> import glob 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> import re 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> import os 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> import subprocess 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> def _find_admin_socket(client_name): 2024-01-16T15:15:47.492 DEBUG:teuthology.orchestra.run.smithi141:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> files = glob.glob(asok_path) 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> # Given a non-glob path, it better be there 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> if "*" not in asok_path: 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> assert(len(files) == 1) 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> return files[0] 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> for f in files: 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> contents = proc_f.read() 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> if mountpoint in contents: 2024-01-16T15:15:47.493 DEBUG:teuthology.orchestra.run.smithi141:> return f 2024-01-16T15:15:47.494 DEBUG:teuthology.orchestra.run.smithi141:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:47.494 DEBUG:teuthology.orchestra.run.smithi141:> 2024-01-16T15:15:47.494 DEBUG:teuthology.orchestra.run.smithi141:> print(_find_admin_socket("client.0")) 2024-01-16T15:15:47.494 DEBUG:teuthology.orchestra.run.smithi141:> ' 2024-01-16T15:15:47.682 INFO:teuthology.orchestra.run.smithi141.stdout:/var/run/ceph/ceph-client.0.76610.asok 2024-01-16T15:15:47.686 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:47.698 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76610.asok 2024-01-16T15:15:47.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:47.699 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76610.asok status 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "metadata": { 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "entity_id": "0", 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "hostname": "smithi141", 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "pid": "76610", 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "root": "/" 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:47.931 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_count": 0, 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "id": 9315, 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "inst": { 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "name": { 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "client", 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "num": 9315 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 3315000103 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": { 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "type": "v1", 2024-01-16T15:15:47.932 INFO:teuthology.orchestra.run.smithi141.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "nonce": 3315000103 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "inst_str": "client.9315 192.168.0.1:0/3315000103", 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "addr_str": "192.168.0.1:0/3315000103", 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "inode_count": 1, 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_epoch": 73, 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch": 116, 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "blocklisted": false, 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout: "fs_name": "cephfs" 2024-01-16T15:15:47.933 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:15:47.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:47.946 DEBUG:teuthology.orchestra.run.smithi141:> stat --file-system '--printf=%T 2024-01-16T15:15:47.946 DEBUG:teuthology.orchestra.run.smithi141:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:47.996 INFO:teuthology.orchestra.run.smithi141.stdout:fuseblk 2024-01-16T15:15:47.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:47.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:47.996 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:48.090 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:48.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.321+0000 7f6c2a200640 1 Processor -- start 2024-01-16T15:15:48.306 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.321+0000 7f6c2a200640 1 -- start start 2024-01-16T15:15:48.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.322+0000 7f6c2a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 0x7f6c2413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c2a200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2413ff00 con 0x7f6c2413cfa0 2024-01-16T15:15:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c2a200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240cdc40 con 0x7f6c2413fae0 2024-01-16T15:15:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c2a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240cdec0 con 0x7f6c2415afb0 2024-01-16T15:15:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c28e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6c2413cfa0 0x7f6c240cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57476/0 (socket says 172.21.15.141:57476) 2024-01-16T15:15:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c28e00640 1 -- 172.21.15.141:0/2084331487 learned_addr learned my addr 172.21.15.141:0/2084331487 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318525408 0 0) 0x7f6c2413ff00 con 0x7f6c2413cfa0 2024-01-16T15:15:48.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c04003520 con 0x7f6c2413cfa0 2024-01-16T15:15:48.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c23e00640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 0x7f6c2413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.323+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147514890 0 0) 0x7f6c04003520 con 0x7f6c2413cfa0 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 msgr2=0x7f6c2413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 0x7f6c2413f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 >> v1:172.21.15.188:6789/0 conn(0x7f6c2413fae0 legacy=0x7f6c240b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240ce1a0 con 0x7f6c2413cfa0 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6c14001670 con 0x7f6c2413cfa0 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 >> v1:172.21.15.141:6789/0 conn(0x7f6c2413cfa0 legacy=0x7f6c240cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 0x7f6c040039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c04004020 0x7f6c040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c04004d00 0x7f6c04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240cdec0 con 0x7f6c04004d00 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0400bc60 con 0x7f6c04004020 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0400be20 con 0x7f6c2415afb0 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c29800640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c04004d00 0x7f6c04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:48.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c28e00640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 0x7f6c040039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c23e00640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c04004020 0x7f6c040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c29800640 1 -- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 msgr2=0x7f6c040039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c29800640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c2415afb0 0x7f6c040039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.324+0000 7f6c29800640 1 -- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c04004020 msgr2=0x7f6c040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.325+0000 7f6c29800640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c04004020 0x7f6c040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.311 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.325+0000 7f6c29800640 1 -- 172.21.15.141:0/2084331487 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c04003520 con 0x7f6c04004d00 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c29800640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c04004d00 0x7f6c04005180 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c001e50 tx=0x7f6c040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f47941fcb7eeba4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c003b70 con 0x7f6c04004d00 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c003cd0 con 0x7f6c04004d00 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c23400640 1 -- 172.21.15.141:0/2084331487 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c005310 con 0x7f6c04004d00 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c2a200640 1 -- 172.21.15.141:0/2084331487 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c04004d00 msgr2=0x7f6c04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c2a200640 1 --2- 172.21.15.141:0/2084331487 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c04004d00 0x7f6c04005180 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c001e50 tx=0x7f6c040057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c29800640 1 -- 172.21.15.141:0/2084331487 reap_dead start 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c2a200640 1 -- 172.21.15.141:0/2084331487 shutdown_connections 2024-01-16T15:15:48.312 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.326+0000 7f6c2a200640 1 -- 172.21.15.141:0/2084331487 >> 172.21.15.141:0/2084331487 conn(0x7f6c2406a800 msgr2=0x7f6c2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:48.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.327+0000 7f6c2a200640 1 -- 172.21.15.141:0/2084331487 shutdown_connections 2024-01-16T15:15:48.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.327+0000 7f6c2a200640 1 -- 172.21.15.141:0/2084331487 wait complete. 2024-01-16T15:15:48.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.328+0000 7f6c2a200640 1 Processor -- start 2024-01-16T15:15:48.313 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.329+0000 7f6c2a200640 1 -- start start 2024-01-16T15:15:48.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.329+0000 7f6c2a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c04004d00 0x7f6c241f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.329+0000 7f6c2a200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c2413de20 0x7f6c2413e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c2a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c2415daa0 0x7f6c2415df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.314 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c2a200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2413ee60 con 0x7f6c2413de20 2024-01-16T15:15:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c2a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c24155c60 con 0x7f6c2415daa0 2024-01-16T15:15:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c2a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c241f3490 con 0x7f6c04004d00 2024-01-16T15:15:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c2413de20 0x7f6c2413e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c2413de20 0x7f6c2413e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55498/0 (socket says 172.21.15.141:55498) 2024-01-16T15:15:48.315 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 -- 172.21.15.141:0/3235507218 learned_addr learned my addr 172.21.15.141:0/3235507218 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 -- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c04004d00 msgr2=0x7f6c241f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c04004d00 0x7f6c241f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 -- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c2415daa0 msgr2=0x7f6c2415df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c2415daa0 0x7f6c2415df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 -- 172.21.15.141:0/3235507218 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2413e8a0 con 0x7f6c2413de20 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.330+0000 7f6c23e00640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c2413de20 0x7f6c2413e2a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00f580 tx=0x7f6c0c00fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.331+0000 7f6c20c00640 1 -- 172.21.15.141:0/3235507218 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c014870 con 0x7f6c2413de20 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.331+0000 7f6c20c00640 1 -- 172.21.15.141:0/3235507218 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c0149d0 con 0x7f6c2413de20 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.331+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2413eb80 con 0x7f6c2413de20 2024-01-16T15:15:48.316 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.331+0000 7f6c20c00640 1 -- 172.21.15.141:0/3235507218 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c014d80 con 0x7f6c2413de20 2024-01-16T15:15:48.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.331+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2415e460 con 0x7f6c2413de20 2024-01-16T15:15:48.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.332+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c24155750 con 0x7f6c2413de20 2024-01-16T15:15:48.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.333+0000 7f6c20c00640 1 -- 172.21.15.141:0/3235507218 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f6c0c015410 con 0x7f6c2413de20 2024-01-16T15:15:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.334+0000 7f6c20c00640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6bfc06c4f0 0x7f6bfc06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.334+0000 7f6c20c00640 1 -- 172.21.15.141:0/3235507218 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c0c089500 con 0x7f6c2413de20 2024-01-16T15:15:48.319 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.335+0000 7f6c28e00640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6bfc06c4f0 0x7f6bfc06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:48.320 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.336+0000 7f6c28e00640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6bfc06c4f0 0x7f6bfc06e9b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005990 tx=0x7f6c14000f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:48.323 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.339+0000 7f6c20c00640 1 -- 172.21.15.141:0/3235507218 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c24155750 con 0x7f6c2413de20 2024-01-16T15:15:48.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.565+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6c24155750 con 0x7f6c2413de20 2024-01-16T15:15:48.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.566+0000 7f6c20c00640 1 -- 172.21.15.141:0/3235507218 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6c24155750 con 0x7f6c2413de20 2024-01-16T15:15:48.551 INFO:teuthology.orchestra.run.smithi141.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6bfc06c4f0 msgr2=0x7f6bfc06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6bfc06c4f0 0x7f6bfc06e9b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005990 tx=0x7f6c14000f90 comp rx=0 tx=0).stop 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c2413de20 msgr2=0x7f6c2413e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c2413de20 0x7f6c2413e2a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00f580 tx=0x7f6c0c00fa40 comp rx=0 tx=0).stop 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 shutdown_connections 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f6bfc06c4f0 0x7f6bfc06e9b0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c2415daa0 0x7f6c2415df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.568+0000 7f6c2a200640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6c2413de20 0x7f6c2413e2a0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.569+0000 7f6c2a200640 1 --2- 172.21.15.141:0/3235507218 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f6c04004d00 0x7f6c241f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.569+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 >> 172.21.15.141:0/3235507218 conn(0x7f6c2406a800 msgr2=0x7f6c24071940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.569+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 shutdown_connections 2024-01-16T15:15:48.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:48.569+0000 7f6c2a200640 1 -- 172.21.15.141:0/3235507218 wait complete. 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:48.572 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:48.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:48.573 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-01-16T15:15:48.631 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-01-16T15:15:48.632 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 41517sec preferred_lft 41517sec 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::61:83ff:fe61:1770/64 scope link 2024-01-16T15:15:48.633 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether de:f9:ef:d4:bc:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::dcf9:efff:fed4:bcd9/64 scope link 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether a6:0a:23:d8:ea:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::a40a:23ff:fed8:ea96/64 scope link 2024-01-16T15:15:48.634 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:48.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:48.635 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:48.728 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-01-16T15:15:48.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:48.733 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-01-16T15:15:48.786 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-01-16T15:15:48.786 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:48.786 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:15:48.786 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:15:48.786 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:48.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:48.787 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-01-16T15:15:48.787 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-01-16T15:15:48.787 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:15:48.787 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:15:48.787 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:15:48.787 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-01-16T15:15:48.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:48.968 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T15:15:49.016 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:15:49.016 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:15:49.017 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:49.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:49.017 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-01-16T15:15:49.108 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:15:49.108 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:49.108 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:49.160 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:49.160 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:49.160 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:49.212 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:15:49.302 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:49.359 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:15:49.359 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:15:49.359 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:15:49.359 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:15:49.359 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:15:49.359 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:49.360 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:49.361 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:49.362 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:49.362 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:49.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:49.362 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:15:49.413 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:49.413 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:15:49.418 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-01-16T15:15:49.493 DEBUG:teuthology.orchestra.run.smithi188:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:49.538 INFO:teuthology.orchestra.run.smithi188.stdout:/proc 2024-01-16T15:15:49.538 INFO:teuthology.orchestra.run.smithi188.stdout:/sys 2024-01-16T15:15:49.538 INFO:teuthology.orchestra.run.smithi188.stdout:/dev 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/security 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/shm 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/pts 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/run 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/cgroup 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/pstore 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/ 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/selinux 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/hugepages 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/dev/mqueue 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/debug 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/tracing 2024-01-16T15:15:49.539 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/sys/kernel/config 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/run/user/1000 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:49.540 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns 2024-01-16T15:15:49.541 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:49.541 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:49.541 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:49.541 INFO:teuthology.orchestra.run.smithi188.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:49.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:49.542 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-01-16T15:15:49.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-01-16T15:15:49.617+0000 7f75e45db480 -1 init, newargv = 0x557080def9e0 newargc=15 2024-01-16T15:15:49.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[47521]: starting ceph client 2024-01-16T15:15:49.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[47521]: starting fuse 2024-01-16T15:15:49.625 INFO:teuthology.orchestra.run.smithi188.stdout:41 2024-01-16T15:15:49.626 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-01-16T15:15:49.626 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-01-16T15:15:49.627 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-01-16T15:15:49.821 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.47521.asok 2024-01-16T15:15:49.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-01-16T15:15:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:49.838 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.47521.asok 2024-01-16T15:15:49.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:49.838 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.47521.asok status 2024-01-16T15:15:50.069 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-01-16T15:15:50.069 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "47521", 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 6850, 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-01-16T15:15:50.070 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 6850 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 4059266225 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 4059266225 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.6850 192.168.0.1:0/4059266225", 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/4059266225", 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-01-16T15:15:50.071 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 73, 2024-01-16T15:15:50.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 116, 2024-01-16T15:15:50.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:50.072 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-01-16T15:15:50.072 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-01-16T15:15:50.072 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-01-16T15:15:50.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:50.083 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-01-16T15:15:50.084 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:50.133 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-01-16T15:15:50.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:50.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:50.133 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:50.223 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:15:50.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.458+0000 7f871c000640 1 Processor -- start 2024-01-16T15:15:50.443 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.458+0000 7f871c000640 1 -- start start 2024-01-16T15:15:50.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.460+0000 7f871c000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f87140b3ae0 0x7f871415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.460+0000 7f871c000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8714070c40 con 0x7f87140cf290 2024-01-16T15:15:50.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.460+0000 7f871c000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871406c470 con 0x7f87140ae5a0 2024-01-16T15:15:50.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.460+0000 7f871c000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8714071450 con 0x7f87140b3ae0 2024-01-16T15:15:50.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.460+0000 7f871a000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f87140cf290 0x7f87140b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57486/0 (socket says 172.21.15.141:57486) 2024-01-16T15:15:50.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.460+0000 7f871a000640 1 -- 172.21.15.141:0/703956180 learned_addr learned my addr 172.21.15.141:0/703956180 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:50.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8718c00640 1 --2- 172.21.15.141:0/703956180 >> v2:172.21.15.188:3300/0 conn(0x7f87140b3ae0 0x7f871415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354644020 0 0) 0x7f8714070c40 con 0x7f87140cf290 2024-01-16T15:15:50.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f4003520 con 0x7f87140cf290 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007549326 0 0) 0x7f86f4003520 con 0x7f87140cf290 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 >> v2:172.21.15.188:3300/0 conn(0x7f87140b3ae0 msgr2=0x7f871415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 --2- 172.21.15.141:0/703956180 >> v2:172.21.15.188:3300/0 conn(0x7f87140b3ae0 0x7f871415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 >> v1:172.21.15.188:6789/0 conn(0x7f87140ae5a0 legacy=0x7f871406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871406c750 con 0x7f87140cf290 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f870c001670 con 0x7f87140cf290 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 >> v1:172.21.15.141:6789/0 conn(0x7f87140cf290 legacy=0x7f87140b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86f4003970 0x7f86f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.461+0000 7f8713e00640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8714071450 con 0x7f86f40047c0 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f400c030 con 0x7f86f4003970 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f400c1f0 con 0x7f86f4005180 2024-01-16T15:15:50.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8718c00640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8719600640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86f4003970 0x7f86f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f871a000640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8718c00640 1 -- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f4005180 msgr2=0x7f86f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8718c00640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8718c00640 1 -- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86f4003970 msgr2=0x7f86f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8718c00640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86f4003970 0x7f86f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f8718c00640 1 -- 172.21.15.141:0/703956180 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f4003520 con 0x7f86f40047c0 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f871a000640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:50.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.462+0000 7f871a000640 1 -- 172.21.15.141:0/703956180 reap_dead start 2024-01-16T15:15:50.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.463+0000 7f8718c00640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009ac0 tx=0x7f86f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eea121636eb1f5e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:50.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc02c6b0 con 0x7f86f40047c0 2024-01-16T15:15:50.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc02c810 con 0x7f86f40047c0 2024-01-16T15:15:50.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f8713e00640 1 -- 172.21.15.141:0/703956180 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc02cba0 con 0x7f86f40047c0 2024-01-16T15:15:50.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f871c000640 1 -- 172.21.15.141:0/703956180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86f40047c0 msgr2=0x7f86f4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:50.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f871c000640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009ac0 tx=0x7f86f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f871c000640 1 -- 172.21.15.141:0/703956180 shutdown_connections 2024-01-16T15:15:50.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f871c000640 1 --2- 172.21.15.141:0/703956180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f871c000640 1 -- 172.21.15.141:0/703956180 >> 172.21.15.141:0/703956180 conn(0x7f871406a800 msgr2=0x7f871406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:50.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f871c000640 1 -- 172.21.15.141:0/703956180 shutdown_connections 2024-01-16T15:15:50.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.464+0000 7f871c000640 1 -- 172.21.15.141:0/703956180 wait complete. 2024-01-16T15:15:50.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.466+0000 7f871c000640 1 Processor -- start 2024-01-16T15:15:50.451 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.466+0000 7f871c000640 1 -- start start 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.467+0000 7f871c000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f40047c0 0x7f8714159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.467+0000 7f871c000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871415b4f0 0x7f87140ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.467+0000 7f871c000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87140ce7e0 0x7f87140cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.467+0000 7f871c000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87141f7190 con 0x7f871415b4f0 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.467+0000 7f871c000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87141f6580 con 0x7f87140ce7e0 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.467+0000 7f871c000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87141f6860 con 0x7f86f40047c0 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871415b4f0 0x7f87140ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871415b4f0 0x7f87140ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55516/0 (socket says 172.21.15.141:55516) 2024-01-16T15:15:50.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 -- 172.21.15.141:0/1431196654 learned_addr learned my addr 172.21.15.141:0/1431196654 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8719600640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f40047c0 0x7f8714159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f871a000640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87140ce7e0 0x7f87140cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 -- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f40047c0 msgr2=0x7f8714159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f40047c0 0x7f8714159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 -- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87140ce7e0 msgr2=0x7f87140cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87140ce7e0 0x7f87140cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 -- 172.21.15.141:0/1431196654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87141f6b40 con 0x7f871415b4f0 2024-01-16T15:15:50.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.468+0000 7f8718c00640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871415b4f0 0x7f87140ce2a0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f86fc02c250 tx=0x7f86fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.469+0000 7f8711600640 1 -- 172.21.15.141:0/1431196654 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc003810 con 0x7f871415b4f0 2024-01-16T15:15:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.469+0000 7f8711600640 1 -- 172.21.15.141:0/1431196654 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc003970 con 0x7f871415b4f0 2024-01-16T15:15:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.469+0000 7f8711600640 1 -- 172.21.15.141:0/1431196654 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc003d00 con 0x7f871415b4f0 2024-01-16T15:15:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.469+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87141f6e20 con 0x7f871415b4f0 2024-01-16T15:15:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.469+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8714159b80 con 0x7f871415b4f0 2024-01-16T15:15:50.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.470+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87141f7300 con 0x7f871415b4f0 2024-01-16T15:15:50.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.471+0000 7f8711600640 1 -- 172.21.15.141:0/1431196654 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f86fc047450 con 0x7f871415b4f0 2024-01-16T15:15:50.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.472+0000 7f8711600640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f86ec06c4f0 0x7f86ec06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:50.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.472+0000 7f8711600640 1 -- 172.21.15.141:0/1431196654 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86fc0aaa20 con 0x7f871415b4f0 2024-01-16T15:15:50.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.473+0000 7f8719600640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f86ec06c4f0 0x7f86ec06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:50.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.474+0000 7f8719600640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f86ec06c4f0 0x7f86ec06e9b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8704003fd0 tx=0x7f8704009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:50.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.476+0000 7f8711600640 1 -- 172.21.15.141:0/1431196654 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86fc0773f0 con 0x7f871415b4f0 2024-01-16T15:15:50.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.703+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f87140675d0 con 0x7f871415b4f0 2024-01-16T15:15:50.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.818+0000 7f8711600640 1 -- 172.21.15.141:0/1431196654 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 136+0+0 (secure 0 0 0) 0x7f86fc003e60 con 0x7f871415b4f0 2024-01-16T15:15:50.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f86ec06c4f0 msgr2=0x7f86ec06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:50.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f86ec06c4f0 0x7f86ec06e9b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8704003fd0 tx=0x7f8704009040 comp rx=0 tx=0).stop 2024-01-16T15:15:50.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871415b4f0 msgr2=0x7f87140ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:50.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871415b4f0 0x7f87140ce2a0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f86fc02c250 tx=0x7f86fc004730 comp rx=0 tx=0).stop 2024-01-16T15:15:50.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 shutdown_connections 2024-01-16T15:15:50.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f86ec06c4f0 0x7f86ec06e9b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87140ce7e0 0x7f87140cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f871415b4f0 0x7f87140ce2a0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 --2- 172.21.15.141:0/1431196654 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86f40047c0 0x7f8714159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:50.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 >> 172.21.15.141:0/1431196654 conn(0x7f871406a800 msgr2=0x7f87140b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:50.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 shutdown_connections 2024-01-16T15:15:50.807 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:50.821+0000 7f871c000640 1 -- 172.21.15.141:0/1431196654 wait complete. 2024-01-16T15:15:50.823 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:15:51.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.035+0000 7f7ace200640 1 Processor -- start 2024-01-16T15:15:51.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.035+0000 7f7ace200640 1 -- start start 2024-01-16T15:15:51.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ace200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7ac813df80 0x7f7ac815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ace200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7ac813e360 0x7f7ac80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ace200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 0x7f7ac80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ace200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac806c470 con 0x7f7ac813e360 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ace200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac806c750 con 0x7f7ac813df80 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ace200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac806ca30 con 0x7f7ac815f9c0 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ac6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7ac813e360 0x7f7ac80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ac6a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7ac813e360 0x7f7ac80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55526/0 (socket says 172.21.15.141:55526) 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ac6a00640 1 -- 172.21.15.141:0/3482068791 learned_addr learned my addr 172.21.15.141:0/3482068791 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ac6a00640 1 -- 172.21.15.141:0/3482068791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 msgr2=0x7f7ac80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.037+0000 7f7ac6a00640 1 --2- 172.21.15.141:0/3482068791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 0x7f7ac80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.038+0000 7f7ac6a00640 1 -- 172.21.15.141:0/3482068791 >> v2:172.21.15.188:3300/0 conn(0x7f7ac813df80 msgr2=0x7f7ac815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.038+0000 7f7ac6a00640 1 --2- 172.21.15.141:0/3482068791 >> v2:172.21.15.188:3300/0 conn(0x7f7ac813df80 0x7f7ac815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.038+0000 7f7ac6a00640 1 -- 172.21.15.141:0/3482068791 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac806cd10 con 0x7f7ac813e360 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.039+0000 7f7ac6a00640 1 --2- 172.21.15.141:0/3482068791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813e360 0x7f7ac80cf290 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8072770 tx=0x7f7ab002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0e62f4e4cf69fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.039+0000 7f7ac6000640 1 -- 172.21.15.141:0/3482068791 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0031870 con 0x7f7ac813e360 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.039+0000 7f7ac6000640 1 -- 172.21.15.141:0/3482068791 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab00319d0 con 0x7f7ac813e360 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.039+0000 7f7ac6000640 1 -- 172.21.15.141:0/3482068791 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0031d60 con 0x7f7ac813e360 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.039+0000 7f7ace200640 1 -- 172.21.15.141:0/3482068791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813e360 msgr2=0x7f7ac80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.039+0000 7f7ace200640 1 --2- 172.21.15.141:0/3482068791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813e360 0x7f7ac80cf290 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8072770 tx=0x7f7ab002b910 comp rx=0 tx=0).stop 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.040+0000 7f7ace200640 1 -- 172.21.15.141:0/3482068791 shutdown_connections 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.040+0000 7f7ace200640 1 --2- 172.21.15.141:0/3482068791 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 0x7f7ac80cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.040+0000 7f7ace200640 1 --2- 172.21.15.141:0/3482068791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813e360 0x7f7ac80cf290 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.040+0000 7f7ace200640 1 --2- 172.21.15.141:0/3482068791 >> v2:172.21.15.188:3300/0 conn(0x7f7ac813df80 0x7f7ac815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.040+0000 7f7ace200640 1 -- 172.21.15.141:0/3482068791 >> 172.21.15.141:0/3482068791 conn(0x7f7ac806a800 msgr2=0x7f7ac806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.040+0000 7f7ace200640 1 -- 172.21.15.141:0/3482068791 shutdown_connections 2024-01-16T15:15:51.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.040+0000 7f7ace200640 1 -- 172.21.15.141:0/3482068791 wait complete. 2024-01-16T15:15:51.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.041+0000 7f7ace200640 1 Processor -- start 2024-01-16T15:15:51.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.042+0000 7f7ace200640 1 -- start start 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.042+0000 7f7ace200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813df80 0x7f7ac80adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.042+0000 7f7ace200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac813e360 0x7f7ac80b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.042+0000 7f7ace200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 0x7f7ac81e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ace200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b31e0 con 0x7f7ac813df80 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ace200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b2560 con 0x7f7ac813e360 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ace200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b2840 con 0x7f7ac815f9c0 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813df80 0x7f7ac80adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813df80 0x7f7ac80adf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55528/0 (socket says 172.21.15.141:55528) 2024-01-16T15:15:51.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 -- 172.21.15.141:0/3091490746 learned_addr learned my addr 172.21.15.141:0/3091490746 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:51.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 -- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 msgr2=0x7f7ac81e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 0x7f7ac81e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 -- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac813e360 msgr2=0x7f7ac80b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac813e360 0x7f7ac80b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 -- 172.21.15.141:0/3091490746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab00094d0 con 0x7f7ac813df80 2024-01-16T15:15:51.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.043+0000 7f7ac7400640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813df80 0x7f7ac80adf20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000ba50 tx=0x7f7ac000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.044+0000 7f7abf400640 1 -- 172.21.15.141:0/3091490746 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000cbe0 con 0x7f7ac813df80 2024-01-16T15:15:51.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.044+0000 7f7abf400640 1 -- 172.21.15.141:0/3091490746 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac000cd60 con 0x7f7ac813df80 2024-01-16T15:15:51.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.044+0000 7f7abf400640 1 -- 172.21.15.141:0/3091490746 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0014680 con 0x7f7ac813df80 2024-01-16T15:15:51.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.044+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac80b2b20 con 0x7f7ac813df80 2024-01-16T15:15:51.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.044+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac80b21e0 con 0x7f7ac813df80 2024-01-16T15:15:51.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.045+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac80b3320 con 0x7f7ac813df80 2024-01-16T15:15:51.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.046+0000 7f7abf400640 1 -- 172.21.15.141:0/3091490746 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7ac001d3b0 con 0x7f7ac813df80 2024-01-16T15:15:51.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.047+0000 7f7abf400640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7aa006c4c0 0x7f7aa006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.047+0000 7f7abf400640 1 -- 172.21.15.141:0/3091490746 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ac00887d0 con 0x7f7ac813df80 2024-01-16T15:15:51.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.047+0000 7f7ac6a00640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7aa006c4c0 0x7f7aa006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.049+0000 7f7ac6a00640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7aa006c4c0 0x7f7aa006e980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0036000 tx=0x7f7ab0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.051+0000 7f7abf400640 1 -- 172.21.15.141:0/3091490746 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ac80b3320 con 0x7f7ac813df80 2024-01-16T15:15:51.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.265+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ac81408d0 con 0x7f7aa006c4c0 2024-01-16T15:15:51.254 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.269+0000 7f7abf400640 1 -- 172.21.15.141:0/3091490746 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7ab0031bc0 con 0x7f7aa006c4c0 2024-01-16T15:15:51.254 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:15:51.254 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:15:51.254 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:15:51.254 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:15:51.254 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7aa006c4c0 msgr2=0x7f7aa006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7aa006c4c0 0x7f7aa006e980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0036000 tx=0x7f7ab0030000 comp rx=0 tx=0).stop 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813df80 msgr2=0x7f7ac80adf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813df80 0x7f7ac80adf20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000ba50 tx=0x7f7ac000bf10 comp rx=0 tx=0).stop 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 shutdown_connections 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7aa006c4c0 0x7f7aa006e980 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7ac815f9c0 0x7f7ac81e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac813e360 0x7f7ac80b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 --2- 172.21.15.141:0/3091490746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ac813df80 0x7f7ac80adf20 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.271+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 >> 172.21.15.141:0/3091490746 conn(0x7f7ac806a800 msgr2=0x7f7ac80afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:51.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.272+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 shutdown_connections 2024-01-16T15:15:51.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.272+0000 7f7ace200640 1 -- 172.21.15.141:0/3091490746 wait complete. 2024-01-16T15:15:51.275 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:15:51.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.501+0000 7f3987400640 1 Processor -- start 2024-01-16T15:15:51.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.501+0000 7f3987400640 1 -- start start 2024-01-16T15:15:51.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.503+0000 7f3987400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3980160920 0x7f39800b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.503+0000 7f3987400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f39800cf9f0 0x7f39800cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.503+0000 7f3987400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800d0a10 con 0x7f3980161010 2024-01-16T15:15:51.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.503+0000 7f3987400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800d03b0 con 0x7f39800cf9f0 2024-01-16T15:15:51.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.503+0000 7f3987400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800d1f50 con 0x7f3980160920 2024-01-16T15:15:51.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3985600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f39800cf9f0 0x7f39800cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3985600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f39800cf9f0 0x7f39800cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:45966/0 (socket says 172.21.15.141:45966) 2024-01-16T15:15:51.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3986000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3980160920 0x7f39800b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3985600640 1 -- 172.21.15.141:0/4260119238 learned_addr learned my addr 172.21.15.141:0/4260119238 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:51.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3986000640 1 -- 172.21.15.141:0/4260119238 >> v2:172.21.15.188:3300/0 conn(0x7f39800cf9f0 msgr2=0x7f39800cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:51.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3986000640 1 --2- 172.21.15.141:0/4260119238 >> v2:172.21.15.188:3300/0 conn(0x7f39800cf9f0 0x7f39800cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3986000640 1 -- 172.21.15.141:0/4260119238 >> v1:172.21.15.188:6789/0 conn(0x7f3980161010 legacy=0x7f39801614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.504+0000 7f3986000640 1 -- 172.21.15.141:0/4260119238 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800cf080 con 0x7f3980160920 2024-01-16T15:15:51.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.506+0000 7f3986000640 1 --2- 172.21.15.141:0/4260119238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3980160920 0x7f39800b3f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f39700097a0 tx=0x7f397002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7f1d02ee9720b0d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.506+0000 7f3984c00640 1 -- 172.21.15.141:0/4260119238 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970031830 con 0x7f3980160920 2024-01-16T15:15:51.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.506+0000 7f3984c00640 1 -- 172.21.15.141:0/4260119238 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970031990 con 0x7f3980160920 2024-01-16T15:15:51.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.506+0000 7f3984c00640 1 -- 172.21.15.141:0/4260119238 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970031d20 con 0x7f3980160920 2024-01-16T15:15:51.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 -- 172.21.15.141:0/4260119238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3980160920 msgr2=0x7f39800b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 --2- 172.21.15.141:0/4260119238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3980160920 0x7f39800b3f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f39700097a0 tx=0x7f397002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 -- 172.21.15.141:0/4260119238 shutdown_connections 2024-01-16T15:15:51.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 --2- 172.21.15.141:0/4260119238 >> v2:172.21.15.188:3300/0 conn(0x7f39800cf9f0 0x7f39800cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 --2- 172.21.15.141:0/4260119238 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3980160920 0x7f39800b3f30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 -- 172.21.15.141:0/4260119238 >> 172.21.15.141:0/4260119238 conn(0x7f39800d3090 msgr2=0x7f39800d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:51.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 -- 172.21.15.141:0/4260119238 shutdown_connections 2024-01-16T15:15:51.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.507+0000 7f3987400640 1 -- 172.21.15.141:0/4260119238 wait complete. 2024-01-16T15:15:51.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.509+0000 7f3987400640 1 Processor -- start 2024-01-16T15:15:51.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.510+0000 7f3987400640 1 -- start start 2024-01-16T15:15:51.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3987400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f39800cf9f0 0x7f39800b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3987400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3980160920 0x7f39801577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3987400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3980161010 0x7f39800b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3987400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398015cdb0 con 0x7f3980161010 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3987400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800b6030 con 0x7f3980160920 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3987400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800b6310 con 0x7f39800cf9f0 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3986a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3980161010 0x7f39800b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3986a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3980161010 0x7f39800b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55536/0 (socket says 172.21.15.141:55536) 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3986a00640 1 -- 172.21.15.141:0/2606094967 learned_addr learned my addr 172.21.15.141:0/2606094967 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.511+0000 7f3986000640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f39800cf9f0 0x7f39800b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3985600640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3980160920 0x7f39801577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3986a00640 1 -- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f39800cf9f0 msgr2=0x7f39800b65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3986a00640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f39800cf9f0 0x7f39800b65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3986a00640 1 -- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3980160920 msgr2=0x7f39801577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3986a00640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3980160920 0x7f39801577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3986a00640 1 -- 172.21.15.141:0/2606094967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970009400 con 0x7f3980161010 2024-01-16T15:15:51.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3986a00640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3980161010 0x7f39800b81a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f397800c7a0 tx=0x7f397800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f397e000640 1 -- 172.21.15.141:0/2606094967 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978003b70 con 0x7f3980161010 2024-01-16T15:15:51.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f397e000640 1 -- 172.21.15.141:0/2606094967 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3978003cf0 con 0x7f3980161010 2024-01-16T15:15:51.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f397e000640 1 -- 172.21.15.141:0/2606094967 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978003ea0 con 0x7f3980161010 2024-01-16T15:15:51.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.512+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398015c6f0 con 0x7f3980161010 2024-01-16T15:15:51.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.513+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39800b6b00 con 0x7f3980161010 2024-01-16T15:15:51.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.513+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3980067450 con 0x7f3980161010 2024-01-16T15:15:51.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.515+0000 7f397e000640 1 -- 172.21.15.141:0/2606094967 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f397800aa90 con 0x7f3980161010 2024-01-16T15:15:51.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.516+0000 7f397e000640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f395806c4c0 0x7f395806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.516+0000 7f397e000640 1 -- 172.21.15.141:0/2606094967 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3978088950 con 0x7f3980161010 2024-01-16T15:15:51.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.516+0000 7f3986000640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f395806c4c0 0x7f395806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.517+0000 7f3986000640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f395806c4c0 0x7f395806e980 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3970036000 tx=0x7f397003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.520+0000 7f397e000640 1 -- 172.21.15.141:0/2606094967 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3978054f00 con 0x7f3980161010 2024-01-16T15:15:51.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.710+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f398008c7e0 con 0x7f3980161010 2024-01-16T15:15:51.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.711+0000 7f397e000640 1 -- 172.21.15.141:0/2606094967 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3978058520 con 0x7f3980161010 2024-01-16T15:15:51.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f395806c4c0 msgr2=0x7f395806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f395806c4c0 0x7f395806e980 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3970036000 tx=0x7f397003c040 comp rx=0 tx=0).stop 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3980161010 msgr2=0x7f39800b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3980161010 0x7f39800b81a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f397800c7a0 tx=0x7f397800cc60 comp rx=0 tx=0).stop 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 shutdown_connections 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f395806c4c0 0x7f395806e980 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3980161010 0x7f39800b81a0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3980160920 0x7f39801577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 --2- 172.21.15.141:0/2606094967 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f39800cf9f0 0x7f39800b65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.713+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 >> 172.21.15.141:0/2606094967 conn(0x7f39800d3090 msgr2=0x7f39800b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.714+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 shutdown_connections 2024-01-16T15:15:51.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.714+0000 7f3987400640 1 -- 172.21.15.141:0/2606094967 wait complete. 2024-01-16T15:15:51.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 30 2024-01-16T15:15:51.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.938+0000 7f7a20600640 1 Processor -- start 2024-01-16T15:15:51.923 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.938+0000 7f7a20600640 1 -- start start 2024-01-16T15:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.940+0000 7f7a20600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7a1813e360 0x7f7a180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.940+0000 7f7a20600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1815c5f0 con 0x7f7a1815f9c0 2024-01-16T15:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.940+0000 7f7a20600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1806c470 con 0x7f7a1813df80 2024-01-16T15:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.940+0000 7f7a20600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1806c750 con 0x7f7a1813e360 2024-01-16T15:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.940+0000 7f7a1d200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7a1813e360 0x7f7a180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.940+0000 7f7a1e600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7a1815f9c0 0x7f7a180cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57502/0 (socket says 172.21.15.141:57502) 2024-01-16T15:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.940+0000 7f7a1e600640 1 -- 172.21.15.141:0/3076405027 learned_addr learned my addr 172.21.15.141:0/3076405027 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:51.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.941+0000 7f7a1d200640 1 -- 172.21.15.141:0/3076405027 >> v1:172.21.15.188:6789/0 conn(0x7f7a1813df80 legacy=0x7f7a1815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:51.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.941+0000 7f7a1d200640 1 -- 172.21.15.141:0/3076405027 >> v1:172.21.15.141:6789/0 conn(0x7f7a1815f9c0 legacy=0x7f7a180cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.941+0000 7f7a1d200640 1 -- 172.21.15.141:0/3076405027 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1806ca30 con 0x7f7a1813e360 2024-01-16T15:15:51.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.941+0000 7f7a17e00640 1 -- 172.21.15.141:0/3076405027 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752304299 0 0) 0x7f7a1815c5f0 con 0x7f7a1815f9c0 2024-01-16T15:15:51.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.942+0000 7f7a1d200640 1 --2- 172.21.15.141:0/3076405027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1813e360 0x7f7a180cf290 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f7a000097a0 tx=0x7f7a0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8adabfc4174fa8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.942+0000 7f7a17e00640 1 -- 172.21.15.141:0/3076405027 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00031830 con 0x7f7a1813e360 2024-01-16T15:15:51.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.942+0000 7f7a17e00640 1 -- 172.21.15.141:0/3076405027 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a00031990 con 0x7f7a1813e360 2024-01-16T15:15:51.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.942+0000 7f7a17e00640 1 -- 172.21.15.141:0/3076405027 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00031d20 con 0x7f7a1813e360 2024-01-16T15:15:51.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.943+0000 7f7a20600640 1 -- 172.21.15.141:0/3076405027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1813e360 msgr2=0x7f7a180cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.943+0000 7f7a20600640 1 --2- 172.21.15.141:0/3076405027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1813e360 0x7f7a180cf290 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f7a000097a0 tx=0x7f7a0002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.943+0000 7f7a20600640 1 -- 172.21.15.141:0/3076405027 shutdown_connections 2024-01-16T15:15:51.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.943+0000 7f7a20600640 1 --2- 172.21.15.141:0/3076405027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1813e360 0x7f7a180cf290 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.943+0000 7f7a20600640 1 -- 172.21.15.141:0/3076405027 >> 172.21.15.141:0/3076405027 conn(0x7f7a1806a800 msgr2=0x7f7a1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:51.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.943+0000 7f7a20600640 1 -- 172.21.15.141:0/3076405027 shutdown_connections 2024-01-16T15:15:51.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.943+0000 7f7a20600640 1 -- 172.21.15.141:0/3076405027 wait complete. 2024-01-16T15:15:51.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.945+0000 7f7a20600640 1 Processor -- start 2024-01-16T15:15:51.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.945+0000 7f7a20600640 1 -- start start 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a20600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a1813df80 0x7f7a180702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a20600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a1813e360 0x7f7a18153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a20600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1815f9c0 0x7f7a1813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a20600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18153580 con 0x7f7a1815f9c0 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a20600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1813d0a0 con 0x7f7a1813df80 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a20600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18070950 con 0x7f7a1813e360 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1815f9c0 0x7f7a1813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1815f9c0 0x7f7a1813cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55550/0 (socket says 172.21.15.141:55550) 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 -- 172.21.15.141:0/3305127113 learned_addr learned my addr 172.21.15.141:0/3305127113 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1dc00640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a1813df80 0x7f7a180702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1d200640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a1813e360 0x7f7a18153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 -- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a1813e360 msgr2=0x7f7a18153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a1813e360 0x7f7a18153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 -- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a1813df80 msgr2=0x7f7a180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a1813df80 0x7f7a180702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1e600640 1 -- 172.21.15.141:0/3305127113 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a00009400 con 0x7f7a1815f9c0 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.946+0000 7f7a1d200640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a1813e360 0x7f7a18153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.947+0000 7f7a1e600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1815f9c0 0x7f7a1813cb60 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7a10002230 tx=0x7f7a1000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.947+0000 7f7a15600640 1 -- 172.21.15.141:0/3305127113 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1000aca0 con 0x7f7a1815f9c0 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.947+0000 7f7a15600640 1 -- 172.21.15.141:0/3305127113 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1000ae20 con 0x7f7a1815f9c0 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.947+0000 7f7a15600640 1 -- 172.21.15.141:0/3305127113 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10012640 con 0x7f7a1815f9c0 2024-01-16T15:15:51.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.947+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a18070c30 con 0x7f7a1815f9c0 2024-01-16T15:15:51.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.947+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a180711f0 con 0x7f7a1815f9c0 2024-01-16T15:15:51.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.948+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1813d210 con 0x7f7a1815f9c0 2024-01-16T15:15:51.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.949+0000 7f7a15600640 1 -- 172.21.15.141:0/3305127113 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7a10007070 con 0x7f7a1815f9c0 2024-01-16T15:15:51.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.950+0000 7f7a15600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79f006c4c0 0x7f79f006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:51.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.950+0000 7f7a15600640 1 -- 172.21.15.141:0/3305127113 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a10088b20 con 0x7f7a1815f9c0 2024-01-16T15:15:51.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.951+0000 7f7a1dc00640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79f006c4c0 0x7f79f006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:51.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.952+0000 7f7a1dc00640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79f006c4c0 0x7f79f006e980 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7a08006fd0 tx=0x7f7a08008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:51.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:51.954+0000 7f7a15600640 1 -- 172.21.15.141:0/3305127113 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a10055070 con 0x7f7a1815f9c0 2024-01-16T15:15:52.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.144+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f7a18155490 con 0x7f7a1815f9c0 2024-01-16T15:15:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.151+0000 7f7a15600640 1 -- 172.21.15.141:0/3305127113 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v43)=0 v43) v1 ==== 131+0+0 (secure 0 0 0) 0x7f7a10017020 con 0x7f7a1815f9c0 2024-01-16T15:15:52.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79f006c4c0 msgr2=0x7f79f006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79f006c4c0 0x7f79f006e980 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7a08006fd0 tx=0x7f7a08008040 comp rx=0 tx=0).stop 2024-01-16T15:15:52.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1815f9c0 msgr2=0x7f7a1813cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.139 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1815f9c0 0x7f7a1813cb60 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7a10002230 tx=0x7f7a1000a060 comp rx=0 tx=0).stop 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 shutdown_connections 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f79f006c4c0 0x7f79f006e980 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a1815f9c0 0x7f7a1813cb60 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7a1813e360 0x7f7a18153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 --2- 172.21.15.141:0/3305127113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a1813df80 0x7f7a180702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 >> 172.21.15.141:0/3305127113 conn(0x7f7a1806a800 msgr2=0x7f7a18153a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 shutdown_connections 2024-01-16T15:15:52.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.154+0000 7f7a20600640 1 -- 172.21.15.141:0/3305127113 wait complete. 2024-01-16T15:15:52.158 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:15:52.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.380+0000 7fbfcea00640 1 Processor -- start 2024-01-16T15:15:52.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.380+0000 7fbfcea00640 1 -- start start 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.382+0000 7fbfcea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 0x7fbfc80b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.382+0000 7fbfcea00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc8071190 con 0x7fbfc8150760 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.382+0000 7fbfcea00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc8071470 con 0x7fbfc815da10 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.382+0000 7fbfcea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc8071750 con 0x7fbfc8072230 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.382+0000 7fbfc7e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbfc8150760 0x7fbfc815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57506/0 (socket says 172.21.15.141:57506) 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.382+0000 7fbfc7e00640 1 -- 172.21.15.141:0/619436607 learned_addr learned my addr 172.21.15.141:0/619436607 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729740384 0 0) 0x7fbfc8071190 con 0x7fbfc8150760 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa8005730 con 0x7fbfc8150760 2024-01-16T15:15:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfcca00640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 0x7fbfc80b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351332974 0 0) 0x7fbfa8005730 con 0x7fbfc8150760 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 msgr2=0x7fbfc80b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 0x7fbfc80b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 >> v1:172.21.15.188:6789/0 conn(0x7fbfc815da10 legacy=0x7fbfc80b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8071a30 con 0x7fbfc8150760 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fbfb80019f0 con 0x7fbfc8150760 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 >> v1:172.21.15.141:6789/0 conn(0x7fbfc8150760 legacy=0x7fbfc815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfa8005bb0 0x7fbfa8005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa80069d0 0x7fbfa8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 0x7fbfa80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc8071750 con 0x7fbfa8005bb0 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa800dcc0 con 0x7fbfa80069d0 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.383+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa800de80 con 0x7fbfc8072230 2024-01-16T15:15:52.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfc7e00640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfa8005bb0 0x7fbfa8005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfcca00640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 0x7fbfa80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfbf200640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa80069d0 0x7fbfa8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfc7e00640 1 -- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 msgr2=0x7fbfa80073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfc7e00640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8072230 0x7fbfa80073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfc7e00640 1 -- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa80069d0 msgr2=0x7fbfa8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfc7e00640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa80069d0 0x7fbfa8006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfc7e00640 1 -- 172.21.15.141:0/619436607 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8005730 con 0x7fbfa8005bb0 2024-01-16T15:15:52.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.384+0000 7fbfbf200640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa80069d0 0x7fbfa8006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.385+0000 7fbfc7e00640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfa8005bb0 0x7fbfa8005fd0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800a7f0 tx=0x7fbfa8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf081272f9f5aea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.385+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8034630 con 0x7fbfa8005bb0 2024-01-16T15:15:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.385+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb802f070 con 0x7fbfa8005bb0 2024-01-16T15:15:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfc7400640 1 -- 172.21.15.141:0/619436607 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb80349f0 con 0x7fbfa8005bb0 2024-01-16T15:15:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfcea00640 1 -- 172.21.15.141:0/619436607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfa8005bb0 msgr2=0x7fbfa8005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfcea00640 1 --2- 172.21.15.141:0/619436607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfa8005bb0 0x7fbfa8005fd0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800a7f0 tx=0x7fbfa8007a00 comp rx=0 tx=0).stop 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfcca00640 1 -- 172.21.15.141:0/619436607 reap_dead start 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfcea00640 1 -- 172.21.15.141:0/619436607 shutdown_connections 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfcea00640 1 -- 172.21.15.141:0/619436607 >> 172.21.15.141:0/619436607 conn(0x7fbfc806a800 msgr2=0x7fbfc806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfcea00640 1 -- 172.21.15.141:0/619436607 shutdown_connections 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.386+0000 7fbfcea00640 1 -- 172.21.15.141:0/619436607 wait complete. 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.388+0000 7fbfcea00640 1 Processor -- start 2024-01-16T15:15:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.388+0000 7fbfcea00640 1 -- start start 2024-01-16T15:15:52.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa8005bb0 0x7fbfc80cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8147540 0x7fbfc814c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfc814f390 0x7fbfc814f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcea00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc814fd50 con 0x7fbfc814f390 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcea00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc81479a0 con 0x7fbfa8005bb0 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc80cc9b0 con 0x7fbfc8147540 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfc814f390 0x7fbfc814f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfc814f390 0x7fbfc814f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55566/0 (socket says 172.21.15.141:55566) 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.389+0000 7fbfcca00640 1 -- 172.21.15.141:0/453516937 learned_addr learned my addr 172.21.15.141:0/453516937 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfc7e00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa8005bb0 0x7fbfc80cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfbf200640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8147540 0x7fbfc814c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfcca00640 1 -- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8147540 msgr2=0x7fbfc814c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfcca00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8147540 0x7fbfc814c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfcca00640 1 -- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa8005bb0 msgr2=0x7fbfc80cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfcca00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa8005bb0 0x7fbfc80cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfcca00640 1 -- 172.21.15.141:0/453516937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc80ccc50 con 0x7fbfc814f390 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfbf200640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8147540 0x7fbfc814c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfcca00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfc814f390 0x7fbfc814f810 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80cb0c0 tx=0x7fbfc000f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.390+0000 7fbfc4c00640 1 -- 172.21.15.141:0/453516937 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0004440 con 0x7fbfc814f390 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.391+0000 7fbfc4c00640 1 -- 172.21.15.141:0/453516937 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc0003610 con 0x7fbfc814f390 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.391+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc80ccf30 con 0x7fbfc814f390 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.391+0000 7fbfc4c00640 1 -- 172.21.15.141:0/453516937 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc00039a0 con 0x7fbfc814f390 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.391+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc80cd4f0 con 0x7fbfc814f390 2024-01-16T15:15:52.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.392+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc80cd710 con 0x7fbfc814f390 2024-01-16T15:15:52.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.393+0000 7fbfc4c00640 1 -- 172.21.15.141:0/453516937 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fbfc0003b00 con 0x7fbfc814f390 2024-01-16T15:15:52.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.394+0000 7fbfc4c00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfa006c4f0 0x7fbfa006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.394+0000 7fbfc4c00640 1 -- 172.21.15.141:0/453516937 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfc0088750 con 0x7fbfc814f390 2024-01-16T15:15:52.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.394+0000 7fbfc7e00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfa006c4f0 0x7fbfa006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.396+0000 7fbfc7e00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfa006c4f0 0x7fbfa006e9b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8034000 tx=0x7fbfb8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:52.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.398+0000 7fbfc4c00640 1 -- 172.21.15.141:0/453516937 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbfc0054cc0 con 0x7fbfc814f390 2024-01-16T15:15:52.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.590+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fbfc80675d0 con 0x7fbfc814f390 2024-01-16T15:15:52.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.595+0000 7fbfc4c00640 1 -- 172.21.15.141:0/453516937 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v44)=0 v44) v1 ==== 137+0+0 (secure 0 0 0) 0x7fbfc0003db0 con 0x7fbfc814f390 2024-01-16T15:15:52.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfa006c4f0 msgr2=0x7fbfa006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfa006c4f0 0x7fbfa006e9b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8034000 tx=0x7fbfb8009030 comp rx=0 tx=0).stop 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfc814f390 msgr2=0x7fbfc814f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfc814f390 0x7fbfc814f810 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80cb0c0 tx=0x7fbfc000f6f0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 shutdown_connections 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbfa006c4f0 0x7fbfa006e9b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbfc814f390 0x7fbfc814f810 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbfc8147540 0x7fbfc814c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.598+0000 7fbfcea00640 1 --2- 172.21.15.141:0/453516937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfa8005bb0 0x7fbfc80cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.599+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 >> 172.21.15.141:0/453516937 conn(0x7fbfc806a800 msgr2=0x7fbfc81598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.599+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 shutdown_connections 2024-01-16T15:15:52.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.599+0000 7fbfcea00640 1 -- 172.21.15.141:0/453516937 wait complete. 2024-01-16T15:15:52.603 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:15:52.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.834+0000 7f12ae800640 1 Processor -- start 2024-01-16T15:15:52.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.834+0000 7f12ae800640 1 -- start start 2024-01-16T15:15:52.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.836+0000 7f12ae800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f12a8146650 0x7f12a80b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.836+0000 7f12ae800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a815dba0 con 0x7f12a80726f0 2024-01-16T15:15:52.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.836+0000 7f12ae800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80cd130 con 0x7f12a815e030 2024-01-16T15:15:52.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.836+0000 7f12ae800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a806f800 con 0x7f12a8146650 2024-01-16T15:15:52.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.836+0000 7f12aca00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f12a80726f0 0x7f12a80cca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:57520/0 (socket says 172.21.15.141:57520) 2024-01-16T15:15:52.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.836+0000 7f12aca00640 1 -- 172.21.15.141:0/281152060 learned_addr learned my addr 172.21.15.141:0/281152060 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:52.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12ad400640 1 --2- 172.21.15.141:0/281152060 >> v2:172.21.15.188:3300/0 conn(0x7f12a8146650 0x7f12a80b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218652715 0 0) 0x7f12a815dba0 con 0x7f12a80726f0 2024-01-16T15:15:52.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1288003520 con 0x7f12a80726f0 2024-01-16T15:15:52.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3736917179 0 0) 0x7f1288003520 con 0x7f12a80726f0 2024-01-16T15:15:52.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 >> v2:172.21.15.188:3300/0 conn(0x7f12a8146650 msgr2=0x7f12a80b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 --2- 172.21.15.141:0/281152060 >> v2:172.21.15.188:3300/0 conn(0x7f12a8146650 0x7f12a80b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 >> v1:172.21.15.188:6789/0 conn(0x7f12a815e030 legacy=0x7f12a806a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a806f240 con 0x7f12a80726f0 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1290001670 con 0x7f12a80726f0 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 >> v1:172.21.15.141:6789/0 conn(0x7f12a80726f0 legacy=0x7f12a80cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1288003970 0x7f1288003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.837+0000 7f12a7e00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12880047c0 0x7f1288004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12a7e00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1288005180 0x7f128800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a806f800 con 0x7f1288005180 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128800c0d0 con 0x7f1288003970 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128800c290 con 0x7f12880047c0 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ade00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1288005180 0x7f128800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ad400640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1288003970 0x7f1288003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12aca00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12880047c0 0x7f1288004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ade00640 1 -- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12880047c0 msgr2=0x7f1288004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:52.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ade00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12880047c0 0x7f1288004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ade00640 1 -- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1288003970 msgr2=0x7f1288003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ade00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1288003970 0x7f1288003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ade00640 1 -- 172.21.15.141:0/281152060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1288003520 con 0x7f1288005180 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.838+0000 7f12ad400640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1288003970 0x7f1288003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.839+0000 7f12ade00640 1 -- 172.21.15.141:0/281152060 reap_dead start 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.839+0000 7f12ade00640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1288005180 0x7f128800b980 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f12a0001e50 tx=0x7f12880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80905b6f08ba993 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0003b70 con 0x7f1288005180 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a0003cf0 con 0x7f1288005180 2024-01-16T15:15:52.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12a7e00640 1 -- 172.21.15.141:0/281152060 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0005310 con 0x7f1288005180 2024-01-16T15:15:52.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12ae800640 1 -- 172.21.15.141:0/281152060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1288005180 msgr2=0x7f128800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12ae800640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1288005180 0x7f128800b980 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f12a0001e50 tx=0x7f12880057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12ae800640 1 -- 172.21.15.141:0/281152060 shutdown_connections 2024-01-16T15:15:52.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12ae800640 1 --2- 172.21.15.141:0/281152060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1288005180 0x7f128800b980 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12ae800640 1 -- 172.21.15.141:0/281152060 >> 172.21.15.141:0/281152060 conn(0x7f12a80b1d40 msgr2=0x7f12a80b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:52.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.840+0000 7f12ae800640 1 -- 172.21.15.141:0/281152060 shutdown_connections 2024-01-16T15:15:52.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.841+0000 7f12ae800640 1 -- 172.21.15.141:0/281152060 wait complete. 2024-01-16T15:15:52.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.842+0000 7f12ae800640 1 Processor -- start 2024-01-16T15:15:52.828 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.842+0000 7f12ae800640 1 -- start start 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.843+0000 7f12ae800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1288005180 0x7f12a815ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.843+0000 7f12ae800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12a80af1f0 0x7f12a80ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.843+0000 7f12ae800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a80af670 0x7f12a80acf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.843+0000 7f12ae800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8143540 con 0x7f12a80af1f0 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.843+0000 7f12ae800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8144340 con 0x7f12a80af670 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.843+0000 7f12ae800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8143730 con 0x7f1288005180 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12a80af1f0 0x7f12a80ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12a80af1f0 0x7f12a80ac9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55590/0 (socket says 172.21.15.141:55590) 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 -- 172.21.15.141:0/748323720 learned_addr learned my addr 172.21.15.141:0/748323720 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12ade00640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a80af670 0x7f12a80acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12ad400640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1288005180 0x7f12a815ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 -- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1288005180 msgr2=0x7f12a815ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1288005180 0x7f12a815ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 -- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a80af670 msgr2=0x7f12a80acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a80af670 0x7f12a80acf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 -- 172.21.15.141:0/748323720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8143a10 con 0x7f12a80af1f0 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.844+0000 7f12aca00640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12a80af1f0 0x7f12a80ac9d0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f129000d770 tx=0x7f129000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.845+0000 7f12a5600640 1 -- 172.21.15.141:0/748323720 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12900074e0 con 0x7f12a80af1f0 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.845+0000 7f12a5600640 1 -- 172.21.15.141:0/748323720 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1290007640 con 0x7f12a80af1f0 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.845+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a8143cf0 con 0x7f12a80af1f0 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.845+0000 7f12a5600640 1 -- 172.21.15.141:0/748323720 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12900049e0 con 0x7f12a80af1f0 2024-01-16T15:15:52.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.845+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a815d390 con 0x7f12a80af1f0 2024-01-16T15:15:52.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.846+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a81444b0 con 0x7f12a80af1f0 2024-01-16T15:15:52.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.847+0000 7f12a5600640 1 -- 172.21.15.141:0/748323720 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f1290004b40 con 0x7f12a80af1f0 2024-01-16T15:15:52.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.848+0000 7f12a5600640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f128006c4f0 0x7f128006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:52.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.848+0000 7f12a5600640 1 -- 172.21.15.141:0/748323720 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12900887f0 con 0x7f12a80af1f0 2024-01-16T15:15:52.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.849+0000 7f12ad400640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f128006c4f0 0x7f128006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:52.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.850+0000 7f12ad400640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f128006c4f0 0x7f128006e9b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1298006fd0 tx=0x7f1298008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:52.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:52.852+0000 7f12a5600640 1 -- 172.21.15.141:0/748323720 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1290054d40 con 0x7f12a80af1f0 2024-01-16T15:15:53.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.073+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f12a806fb80 con 0x7f128006c4f0 2024-01-16T15:15:53.070 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:53.084+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:53.070 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:53.084+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:53.070 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:53.084+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:53.070 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:53.084+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:53.070 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:15:53.084+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:53.140 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.155+0000 7f12a5600640 1 -- 172.21.15.141:0/748323720 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f12a806fb80 con 0x7f128006c4f0 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.159+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f128006c4f0 msgr2=0x7f128006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.159+0000 7f12ae800640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f128006c4f0 0x7f128006e9b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1298006fd0 tx=0x7f1298008040 comp rx=0 tx=0).stop 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12a80af1f0 msgr2=0x7f12a80ac9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12a80af1f0 0x7f12a80ac9d0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f129000d770 tx=0x7f129000dc30 comp rx=0 tx=0).stop 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 shutdown_connections 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f128006c4f0 0x7f128006e9b0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a80af670 0x7f12a80acf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12a80af1f0 0x7f12a80ac9d0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 --2- 172.21.15.141:0/748323720 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1288005180 0x7f12a815ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 >> 172.21.15.141:0/748323720 conn(0x7f12a80b1d40 msgr2=0x7f12a80b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:53.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 shutdown_connections 2024-01-16T15:15:53.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.160+0000 7f12ae800640 1 -- 172.21.15.141:0/748323720 wait complete. 2024-01-16T15:15:53.167 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:53.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.396+0000 7f95a5800640 1 Processor -- start 2024-01-16T15:15:53.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.397+0000 7f95a5800640 1 -- start start 2024-01-16T15:15:53.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.398+0000 7f95a5800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 0x7f95a0099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.398+0000 7f95a5800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00cfa70 con 0x7f95a015e880 2024-01-16T15:15:53.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.398+0000 7f95a5800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a009a3e0 con 0x7f95a006e7c0 2024-01-16T15:15:53.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.398+0000 7f95a5800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00cff00 con 0x7f95a0099a60 2024-01-16T15:15:53.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.398+0000 7f959ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f95a015e880 0x7f95a006ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44416/0 (socket says 172.21.15.141:44416) 2024-01-16T15:15:53.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.398+0000 7f959ea00640 1 -- 172.21.15.141:0/708683150 learned_addr learned my addr 172.21.15.141:0/708683150 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:15:53.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959e000640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 0x7f95a0099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536146477 0 0) 0x7f95a00cfa70 con 0x7f95a015e880 2024-01-16T15:15:53.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580003520 con 0x7f95a015e880 2024-01-16T15:15:53.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746974865 0 0) 0x7f9580003520 con 0x7f95a015e880 2024-01-16T15:15:53.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 msgr2=0x7f95a0099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:53.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 0x7f95a0099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 >> v1:172.21.15.188:6789/0 conn(0x7f95a006e7c0 legacy=0x7f95a006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a006fc70 con 0x7f95a015e880 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f9590001670 con 0x7f95a015e880 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959e000640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 0x7f95a0099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 >> v1:172.21.15.141:6789/0 conn(0x7f95a015e880 legacy=0x7f95a006ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 0x7f95800039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580003f90 0x7f95800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9580004d00 0x7f9580005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00cff00 con 0x7f9580004d00 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958000bb70 con 0x7f9580003f90 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.399+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958000bd30 con 0x7f95a0099a60 2024-01-16T15:15:53.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959f400640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9580004d00 0x7f9580005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959e000640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580003f90 0x7f95800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959ea00640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 0x7f95800039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959f400640 1 -- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 msgr2=0x7f95800039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959f400640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a0099a60 0x7f95800039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959f400640 1 -- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580003f90 msgr2=0x7f95800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959f400640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580003f90 0x7f95800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.400+0000 7f959f400640 1 -- 172.21.15.141:0/708683150 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580003520 con 0x7f9580004d00 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.401+0000 7f959f400640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9580004d00 0x7f9580005180 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9598001750 tx=0x7f95800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf83f11480a4a0e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.401+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980052d0 con 0x7f9580004d00 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.401+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959802e050 con 0x7f9580004d00 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.401+0000 7f959d600640 1 -- 172.21.15.141:0/708683150 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598003830 con 0x7f9580004d00 2024-01-16T15:15:53.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.402+0000 7f95a5800640 1 -- 172.21.15.141:0/708683150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9580004d00 msgr2=0x7f9580005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.402+0000 7f95a5800640 1 --2- 172.21.15.141:0/708683150 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9580004d00 0x7f9580005180 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9598001750 tx=0x7f95800057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.402+0000 7f959f400640 1 -- 172.21.15.141:0/708683150 reap_dead start 2024-01-16T15:15:53.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.402+0000 7f95a5800640 1 -- 172.21.15.141:0/708683150 shutdown_connections 2024-01-16T15:15:53.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.402+0000 7f95a5800640 1 -- 172.21.15.141:0/708683150 >> 172.21.15.141:0/708683150 conn(0x7f95a00b2d00 msgr2=0x7f95a00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:53.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.402+0000 7f95a5800640 1 -- 172.21.15.141:0/708683150 shutdown_connections 2024-01-16T15:15:53.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.402+0000 7f95a5800640 1 -- 172.21.15.141:0/708683150 wait complete. 2024-01-16T15:15:53.392 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.404+0000 7f95a5800640 1 Processor -- start 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.404+0000 7f95a5800640 1 -- start start 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f95a5800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580004d00 0x7f95a015c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f95a5800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a00ca6e0 0x7f95a01e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f95a5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95a01e6ab0 0x7f95a00cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f95a5800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a015cca0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f95a5800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a01e7b10 con 0x7f9580004d00 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f95a5800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a01e5fd0 con 0x7f95a00ca6e0 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f959f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95a01e6ab0 0x7f95a00cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f959f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95a01e6ab0 0x7f95a00cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57044/0 (socket says 172.21.15.141:57044) 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.405+0000 7f959f400640 1 -- 172.21.15.141:0/2912651684 learned_addr learned my addr 172.21.15.141:0/2912651684 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959ea00640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580004d00 0x7f95a015c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959e000640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a00ca6e0 0x7f95a01e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959f400640 1 -- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a00ca6e0 msgr2=0x7f95a01e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959f400640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a00ca6e0 0x7f95a01e72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959f400640 1 -- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580004d00 msgr2=0x7f95a015c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959f400640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580004d00 0x7f95a015c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959f400640 1 -- 172.21.15.141:0/2912651684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a01e62b0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f959f400640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95a01e6ab0 0x7f95a00cbae0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f9598005040 tx=0x7f9598001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f9596a00640 1 -- 172.21.15.141:0/2912651684 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598007a30 con 0x7f95a01e6ab0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f9596a00640 1 -- 172.21.15.141:0/2912651684 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95980067d0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f9596a00640 1 -- 172.21.15.141:0/2912651684 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598006b80 con 0x7f95a01e6ab0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.406+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a015cca0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.407+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a01e7df0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.408+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a01e67d0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.409+0000 7f9596a00640 1 -- 172.21.15.141:0/2912651684 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f9598006ce0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.410+0000 7f9596a00640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f957806c4c0 0x7f957806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.410+0000 7f9596a00640 1 -- 172.21.15.141:0/2912651684 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95980aa4f0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.410+0000 7f959ea00640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f957806c4c0 0x7f957806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.411+0000 7f959ea00640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f957806c4c0 0x7f957806e980 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9590001050 tx=0x7f9590009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:53.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.414+0000 7f9596a00640 1 -- 172.21.15.141:0/2912651684 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95a01e67d0 con 0x7f95a01e6ab0 2024-01-16T15:15:53.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.650+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f95a00cf700 con 0x7f957806c4c0 2024-01-16T15:15:53.649 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.664+0000 7f9596a00640 1 -- 172.21.15.141:0/2912651684 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9590010950 con 0x7f957806c4c0 2024-01-16T15:15:53.649 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d 2024-01-16T15:15:53.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f957806c4c0 msgr2=0x7f957806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f957806c4c0 0x7f957806e980 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9590001050 tx=0x7f9590009040 comp rx=0 tx=0).stop 2024-01-16T15:15:53.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95a01e6ab0 msgr2=0x7f95a00cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95a01e6ab0 0x7f95a00cbae0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f9598005040 tx=0x7f9598001120 comp rx=0 tx=0).stop 2024-01-16T15:15:53.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 shutdown_connections 2024-01-16T15:15:53.652 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f957806c4c0 0x7f957806e980 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95a01e6ab0 0x7f95a00cbae0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f95a00ca6e0 0x7f95a01e72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 --2- 172.21.15.141:0/2912651684 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9580004d00 0x7f95a015c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 >> 172.21.15.141:0/2912651684 conn(0x7f95a00b2d00 msgr2=0x7f95a015da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:53.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 shutdown_connections 2024-01-16T15:15:53.653 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.667+0000 7f95a5800640 1 -- 172.21.15.141:0/2912651684 wait complete. 2024-01-16T15:15:53.672 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:53.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.904+0000 7f11ea800640 1 Processor -- start 2024-01-16T15:15:53.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.904+0000 7f11ea800640 1 -- start start 2024-01-16T15:15:53.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.906+0000 7f11ea800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f11e413e360 0x7f11e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.906+0000 7f11ea800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e415f9c0 0x7f11e40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.906+0000 7f11ea800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e415c5f0 con 0x7f11e413df80 2024-01-16T15:15:53.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.906+0000 7f11ea800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e406c470 con 0x7f11e413e360 2024-01-16T15:15:53.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.906+0000 7f11ea800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e406c750 con 0x7f11e415f9c0 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e2a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f11e413e360 0x7f11e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e415f9c0 0x7f11e40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e2a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f11e413e360 0x7f11e40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:34888/0 (socket says 172.21.15.141:34888) 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e2a00640 1 -- 172.21.15.141:0/3635210187 learned_addr learned my addr 172.21.15.141:0/3635210187 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e415f9c0 0x7f11e40cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:57566/0 (socket says 172.21.15.141:57566) 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e3e00640 1 -- 172.21.15.141:0/3635210187 >> v2:172.21.15.188:3300/0 conn(0x7f11e413e360 msgr2=0x7f11e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e3e00640 1 --2- 172.21.15.141:0/3635210187 >> v2:172.21.15.188:3300/0 conn(0x7f11e413e360 0x7f11e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.907+0000 7f11e3e00640 1 -- 172.21.15.141:0/3635210187 >> v1:172.21.15.188:6789/0 conn(0x7f11e413df80 legacy=0x7f11e415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.908+0000 7f11e3e00640 1 -- 172.21.15.141:0/3635210187 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e406ca30 con 0x7f11e415f9c0 2024-01-16T15:15:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.909+0000 7f11e3e00640 1 --2- 172.21.15.141:0/3635210187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e415f9c0 0x7f11e40cf7d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009d90 tx=0x7f11dc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b7bb0c8dd39ab99 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.909+0000 7f11e2000640 1 -- 172.21.15.141:0/3635210187 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc02cc50 con 0x7f11e415f9c0 2024-01-16T15:15:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.909+0000 7f11e2000640 1 -- 172.21.15.141:0/3635210187 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc02cdb0 con 0x7f11e415f9c0 2024-01-16T15:15:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.909+0000 7f11e2000640 1 -- 172.21.15.141:0/3635210187 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc034660 con 0x7f11e415f9c0 2024-01-16T15:15:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 -- 172.21.15.141:0/3635210187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e415f9c0 msgr2=0x7f11e40cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 --2- 172.21.15.141:0/3635210187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e415f9c0 0x7f11e40cf7d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009d90 tx=0x7f11dc02bea0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 -- 172.21.15.141:0/3635210187 shutdown_connections 2024-01-16T15:15:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 --2- 172.21.15.141:0/3635210187 >> v2:172.21.15.188:3300/0 conn(0x7f11e413e360 0x7f11e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 --2- 172.21.15.141:0/3635210187 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e415f9c0 0x7f11e40cf7d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 -- 172.21.15.141:0/3635210187 >> 172.21.15.141:0/3635210187 conn(0x7f11e406a800 msgr2=0x7f11e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 -- 172.21.15.141:0/3635210187 shutdown_connections 2024-01-16T15:15:53.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.910+0000 7f11ea800640 1 -- 172.21.15.141:0/3635210187 wait complete. 2024-01-16T15:15:53.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.912+0000 7f11ea800640 1 Processor -- start 2024-01-16T15:15:53.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.912+0000 7f11ea800640 1 -- start start 2024-01-16T15:15:53.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11ea800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11e413df80 0x7f11e41531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11ea800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e413e360 0x7f11e41f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11ea800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e415f9c0 0x7f11e4159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11ea800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e41f7220 con 0x7f11e415f9c0 2024-01-16T15:15:53.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11ea800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4153720 con 0x7f11e413df80 2024-01-16T15:15:53.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11ea800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e41f6f40 con 0x7f11e413e360 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11e3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e415f9c0 0x7f11e4159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11e3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e415f9c0 0x7f11e4159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57056/0 (socket says 172.21.15.141:57056) 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11e3e00640 1 -- 172.21.15.141:0/2851828272 learned_addr learned my addr 172.21.15.141:0/2851828272 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11e3400640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11e413df80 0x7f11e41531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.913+0000 7f11e2a00640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e413e360 0x7f11e41f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11e3e00640 1 -- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e413e360 msgr2=0x7f11e41f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11e3e00640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e413e360 0x7f11e41f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11e3e00640 1 -- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11e413df80 msgr2=0x7f11e41531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11e3e00640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11e413df80 0x7f11e41531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11e3e00640 1 -- 172.21.15.141:0/2851828272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc009a70 con 0x7f11e415f9c0 2024-01-16T15:15:53.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11e2a00640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e413e360 0x7f11e41f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:53.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11e3e00640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e415f9c0 0x7f11e4159f80 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009eb0 tx=0x7f11dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:53.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc03e610 con 0x7f11e415f9c0 2024-01-16T15:15:53.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc03e770 con 0x7f11e415f9c0 2024-01-16T15:15:53.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc03eb60 con 0x7f11e415f9c0 2024-01-16T15:15:53.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e40703d0 con 0x7f11e415f9c0 2024-01-16T15:15:53.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.914+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e4070990 con 0x7f11e415f9c0 2024-01-16T15:15:53.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.915+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e40675d0 con 0x7f11e415f9c0 2024-01-16T15:15:53.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.916+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 40) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f11dc03b050 con 0x7f11e415f9c0 2024-01-16T15:15:53.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.917+0000 7f11db400640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f11bc06c4c0 0x7f11bc06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:53.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.917+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11dc0aaa30 con 0x7f11e415f9c0 2024-01-16T15:15:53.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.918+0000 7f11e3400640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f11bc06c4c0 0x7f11bc06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:53.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.919+0000 7f11e3400640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f11bc06c4c0 0x7f11bc06e980 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f11d4000f60 tx=0x7f11d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:53.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.922+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11dc0772f0 con 0x7f11e415f9c0 2024-01-16T15:15:53.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:53.968+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 41) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f11dc0776c0 con 0x7f11e415f9c0 2024-01-16T15:15:54.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.146+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f11e406cff0 con 0x7f11bc06c4c0 2024-01-16T15:15:54.142 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.157+0000 7f11db400640 1 -- 172.21.15.141:0/2851828272 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f11d40050d0 con 0x7f11bc06c4c0 2024-01-16T15:15:54.143 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f11bc06c4c0 msgr2=0x7f11bc06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f11bc06c4c0 0x7f11bc06e980 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f11d4000f60 tx=0x7f11d4009040 comp rx=0 tx=0).stop 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e415f9c0 msgr2=0x7f11e4159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e415f9c0 0x7f11e4159f80 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009eb0 tx=0x7f11dc004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 shutdown_connections 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f11bc06c4c0 0x7f11bc06e980 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e415f9c0 0x7f11e4159f80 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:54.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f11e413e360 0x7f11e41f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:54.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 --2- 172.21.15.141:0/2851828272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11e413df80 0x7f11e41531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:54.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 >> 172.21.15.141:0/2851828272 conn(0x7f11e406a800 msgr2=0x7f11e415bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:54.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 shutdown_connections 2024-01-16T15:15:54.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:15:54.160+0000 7f11ea800640 1 -- 172.21.15.141:0/2851828272 wait complete. 2024-01-16T15:15:54.163 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d 2024-01-16T15:15:54.164 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:15:54.164 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:54.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:54.213 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:16:07.642 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:07.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:08.573 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:16:08.573 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:16:08.574 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.2364 s, 75.4 MB/s 2024-01-16T15:16:08.575 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:16:08.576 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:16:08.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:08.633 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:16:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:16:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:16:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.9376 s, 89.9 MB/s 2024-01-16T15:16:20.632 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:16:20.632 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:16:20.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:20.686 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:16:31.967 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records in 2024-01-16T15:16:31.967 INFO:teuthology.orchestra.run.smithi141.stderr:1024+0 records out 2024-01-16T15:16:31.967 INFO:teuthology.orchestra.run.smithi141.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.2159 s, 95.7 MB/s 2024-01-16T15:16:31.971 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:16:32.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.205+0000 7fbd6de00640 1 Processor -- start 2024-01-16T15:16:32.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.205+0000 7fbd6de00640 1 -- start start 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd6de00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fbd68150760 0x7fbd68072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd6de00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbd6815d300 0x7fbd6815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd6de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b2bd0 0x7fbd680b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd6de00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd6815dc80 con 0x7fbd6815d300 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd6de00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd68071190 con 0x7fbd68150760 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd6de00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68071470 con 0x7fbd680b2bd0 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd5de00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbd6815d300 0x7fbd6815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd5de00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbd6815d300 0x7fbd6815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56806/0 (socket says 172.21.15.141:56806) 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd5de00640 1 -- 172.21.15.141:0/903207188 learned_addr learned my addr 172.21.15.141:0/903207188 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd67400640 1 --2- 172.21.15.141:0/903207188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b2bd0 0x7fbd680b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.207+0000 7fbd66a00640 1 --2- 172.21.15.141:0/903207188 >> v2:172.21.15.188:3300/0 conn(0x7fbd68150760 0x7fbd68072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.208+0000 7fbd5de00640 1 -- 172.21.15.141:0/903207188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b2bd0 msgr2=0x7fbd680b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.208+0000 7fbd5de00640 1 --2- 172.21.15.141:0/903207188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b2bd0 0x7fbd680b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.208+0000 7fbd5de00640 1 -- 172.21.15.141:0/903207188 >> v2:172.21.15.188:3300/0 conn(0x7fbd68150760 msgr2=0x7fbd68072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.208+0000 7fbd5de00640 1 --2- 172.21.15.141:0/903207188 >> v2:172.21.15.188:3300/0 conn(0x7fbd68150760 0x7fbd68072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.208+0000 7fbd5de00640 1 -- 172.21.15.141:0/903207188 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd68071750 con 0x7fbd6815d300 2024-01-16T15:16:32.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.209+0000 7fbd5de00640 1 --2- 172.21.15.141:0/903207188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6815d300 0x7fbd6815d740 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fbd6806e420 tx=0x7fbd5002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400f61b0e370d6d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.209+0000 7fbd66000640 1 -- 172.21.15.141:0/903207188 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5002f8e0 con 0x7fbd6815d300 2024-01-16T15:16:32.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.209+0000 7fbd66000640 1 -- 172.21.15.141:0/903207188 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5002fa40 con 0x7fbd6815d300 2024-01-16T15:16:32.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.209+0000 7fbd66000640 1 -- 172.21.15.141:0/903207188 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5002fdd0 con 0x7fbd6815d300 2024-01-16T15:16:32.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 -- 172.21.15.141:0/903207188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6815d300 msgr2=0x7fbd6815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 --2- 172.21.15.141:0/903207188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6815d300 0x7fbd6815d740 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fbd6806e420 tx=0x7fbd5002db20 comp rx=0 tx=0).stop 2024-01-16T15:16:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 -- 172.21.15.141:0/903207188 shutdown_connections 2024-01-16T15:16:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 --2- 172.21.15.141:0/903207188 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b2bd0 0x7fbd680b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 --2- 172.21.15.141:0/903207188 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6815d300 0x7fbd6815d740 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 --2- 172.21.15.141:0/903207188 >> v2:172.21.15.188:3300/0 conn(0x7fbd68150760 0x7fbd68072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 -- 172.21.15.141:0/903207188 >> 172.21.15.141:0/903207188 conn(0x7fbd6806a800 msgr2=0x7fbd6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 -- 172.21.15.141:0/903207188 shutdown_connections 2024-01-16T15:16:32.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.210+0000 7fbd6de00640 1 -- 172.21.15.141:0/903207188 wait complete. 2024-01-16T15:16:32.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.212+0000 7fbd6de00640 1 Processor -- start 2024-01-16T15:16:32.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.212+0000 7fbd6de00640 1 -- start start 2024-01-16T15:16:32.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd6de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd680b2bd0 0x7fbd68146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd6de00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd68150760 0x7fbd68153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd6de00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd6815d300 0x7fbd681536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd6de00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6814c260 con 0x7fbd680b2bd0 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd6de00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6814d080 con 0x7fbd6815d300 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd6de00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6814c470 con 0x7fbd68150760 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd680b2bd0 0x7fbd68146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd680b2bd0 0x7fbd68146ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56820/0 (socket says 172.21.15.141:56820) 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 -- 172.21.15.141:0/2501754752 learned_addr learned my addr 172.21.15.141:0/2501754752 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 -- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd68150760 msgr2=0x7fbd68153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd68150760 0x7fbd68153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 -- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd6815d300 msgr2=0x7fbd681536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd6815d300 0x7fbd681536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.213+0000 7fbd66a00640 1 -- 172.21.15.141:0/2501754752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd500094d0 con 0x7fbd680b2bd0 2024-01-16T15:16:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.214+0000 7fbd66a00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd680b2bd0 0x7fbd68146ba0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800d800 tx=0x7fbd5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.214+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2501754752 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5800b6c0 con 0x7fbd680b2bd0 2024-01-16T15:16:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.214+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2501754752 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5800b820 con 0x7fbd680b2bd0 2024-01-16T15:16:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.214+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2501754752 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5800bbb0 con 0x7fbd680b2bd0 2024-01-16T15:16:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.214+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6814c750 con 0x7fbd680b2bd0 2024-01-16T15:16:32.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.214+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6814cd10 con 0x7fbd680b2bd0 2024-01-16T15:16:32.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.215+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd680675d0 con 0x7fbd680b2bd0 2024-01-16T15:16:32.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.216+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2501754752 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 41) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fbd5800bd10 con 0x7fbd680b2bd0 2024-01-16T15:16:32.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.217+0000 7fbd5fe00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c550 0x7fbd4006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.203 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.217+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2501754752 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd58089130 con 0x7fbd680b2bd0 2024-01-16T15:16:32.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.218+0000 7fbd5de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c550 0x7fbd4006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.205 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.219+0000 7fbd5de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c550 0x7fbd4006ea10 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbd50038000 tx=0x7fbd50032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.222+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2501754752 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd58055670 con 0x7fbd680b2bd0 2024-01-16T15:16:32.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.444+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd68150b40 con 0x7fbd4006c550 2024-01-16T15:16:32.460 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.474+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2501754752 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbd5002f890 con 0x7fbd4006c550 2024-01-16T15:16:32.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c550 msgr2=0x7fbd4006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c550 0x7fbd4006ea10 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbd50038000 tx=0x7fbd50032000 comp rx=0 tx=0).stop 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd680b2bd0 msgr2=0x7fbd68146ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd680b2bd0 0x7fbd68146ba0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800d800 tx=0x7fbd5800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 shutdown_connections 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c550 0x7fbd4006ea10 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd6815d300 0x7fbd681536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd68150760 0x7fbd68153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 --2- 172.21.15.141:0/2501754752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd680b2bd0 0x7fbd68146ba0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 >> 172.21.15.141:0/2501754752 conn(0x7fbd6806a800 msgr2=0x7fbd68159b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 shutdown_connections 2024-01-16T15:16:32.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.477+0000 7fbd6de00640 1 -- 172.21.15.141:0/2501754752 wait complete. 2024-01-16T15:16:32.482 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:16:32.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.701+0000 7fe760400640 1 Processor -- start 2024-01-16T15:16:32.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.701+0000 7fe760400640 1 -- start start 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.702+0000 7fe760400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 0x7fe7580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.702+0000 7fe760400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe75815f9c0 0x7fe7580cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.702+0000 7fe760400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75815c5f0 con 0x7fe75813df80 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.702+0000 7fe760400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75806c470 con 0x7fe75815f9c0 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.702+0000 7fe760400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75806c750 con 0x7fe75813e360 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.702+0000 7fe75da00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe75813df80 0x7fe75815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54544/0 (socket says 172.21.15.141:54544) 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.702+0000 7fe75da00640 1 -- 172.21.15.141:0/2914636246 learned_addr learned my addr 172.21.15.141:0/2914636246 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:16:32.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe75d000640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 0x7fe7580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe75e400640 1 --2- 172.21.15.141:0/2914636246 >> v2:172.21.15.188:3300/0 conn(0x7fe75815f9c0 0x7fe7580cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118718691 0 0) 0x7fe75815c5f0 con 0x7fe75813df80 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe738003520 con 0x7fe75813df80 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4055616093 0 0) 0x7fe738003520 con 0x7fe75813df80 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 msgr2=0x7fe7580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 0x7fe7580cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 >> v2:172.21.15.188:3300/0 conn(0x7fe75815f9c0 msgr2=0x7fe7580cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 --2- 172.21.15.141:0/2914636246 >> v2:172.21.15.188:3300/0 conn(0x7fe75815f9c0 0x7fe7580cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75806ca30 con 0x7fe75813df80 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fe748001670 con 0x7fe75813df80 2024-01-16T15:16:32.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe75d000640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 0x7fe7580cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 >> v1:172.21.15.141:6789/0 conn(0x7fe75813df80 legacy=0x7fe75815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 0x7fe738003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.703+0000 7fe757e00640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe738003eb0 0x7fe7380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe757e00640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe738004d00 0x7fe738005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75806c470 con 0x7fe738003eb0 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75806c750 con 0x7fe738004d00 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73800b9b0 con 0x7fe75813e360 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75d000640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe738003eb0 0x7fe7380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75da00640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 0x7fe738003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75e400640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe738004d00 0x7fe738005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75d000640 1 -- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 msgr2=0x7fe738003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75d000640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe75813e360 0x7fe738003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75d000640 1 -- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe738004d00 msgr2=0x7fe738005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75d000640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe738004d00 0x7fe738005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75d000640 1 -- 172.21.15.141:0/2914636246 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738003520 con 0x7fe738003eb0 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.704+0000 7fe75e400640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe738004d00 0x7fe738005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.705+0000 7fe75d000640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe738003eb0 0x7fe7380047c0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fe7400098a0 tx=0x7fe7380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5540b84cc33df2b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740004230 con 0x7fe738003eb0 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740004390 con 0x7fe738003eb0 2024-01-16T15:16:32.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe757e00640 1 -- 172.21.15.141:0/2914636246 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740003830 con 0x7fe738003eb0 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe760400640 1 -- 172.21.15.141:0/2914636246 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe738003eb0 msgr2=0x7fe7380047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe760400640 1 --2- 172.21.15.141:0/2914636246 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe738003eb0 0x7fe7380047c0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fe7400098a0 tx=0x7fe7380057f0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe75e400640 1 -- 172.21.15.141:0/2914636246 reap_dead start 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe760400640 1 -- 172.21.15.141:0/2914636246 shutdown_connections 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe760400640 1 -- 172.21.15.141:0/2914636246 >> 172.21.15.141:0/2914636246 conn(0x7fe75806a800 msgr2=0x7fe75806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe760400640 1 -- 172.21.15.141:0/2914636246 shutdown_connections 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.706+0000 7fe760400640 1 -- 172.21.15.141:0/2914636246 wait complete. 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.708+0000 7fe760400640 1 Processor -- start 2024-01-16T15:16:32.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.708+0000 7fe760400640 1 -- start start 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.709+0000 7fe760400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe738003eb0 0x7fe7580afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.709+0000 7fe760400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe75815aa70 0x7fe75813d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.709+0000 7fe760400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe758071310 0x7fe758071790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.709+0000 7fe760400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75813ddf0 con 0x7fe758071310 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.709+0000 7fe760400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758071cd0 con 0x7fe75815aa70 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.709+0000 7fe760400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe758150c60 con 0x7fe738003eb0 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe758071310 0x7fe758071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe758071310 0x7fe758071790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:56850/0 (socket says 172.21.15.141:56850) 2024-01-16T15:16:32.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 -- 172.21.15.141:0/2421189921 learned_addr learned my addr 172.21.15.141:0/2421189921 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75da00640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe738003eb0 0x7fe7580afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75d000640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe75815aa70 0x7fe75813d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 -- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe738003eb0 msgr2=0x7fe7580afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe738003eb0 0x7fe7580afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 -- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe75815aa70 msgr2=0x7fe75813d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe75815aa70 0x7fe75813d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 -- 172.21.15.141:0/2421189921 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75814d650 con 0x7fe758071310 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.710+0000 7fe75e400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe758071310 0x7fe758071790 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fe75815b6b0 tx=0x7fe75000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.711+0000 7fe755600640 1 -- 172.21.15.141:0/2421189921 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750016620 con 0x7fe758071310 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.711+0000 7fe755600640 1 -- 172.21.15.141:0/2421189921 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe750016780 con 0x7fe758071310 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.711+0000 7fe755600640 1 -- 172.21.15.141:0/2421189921 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750016b10 con 0x7fe758071310 2024-01-16T15:16:32.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.711+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7580b0270 con 0x7fe758071310 2024-01-16T15:16:32.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.711+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe758150460 con 0x7fe758071310 2024-01-16T15:16:32.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.712+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7580675d0 con 0x7fe758071310 2024-01-16T15:16:32.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.713+0000 7fe755600640 1 -- 172.21.15.141:0/2421189921 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 41) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe750016c70 con 0x7fe758071310 2024-01-16T15:16:32.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.714+0000 7fe755600640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe73006c550 0x7fe73006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.714+0000 7fe755600640 1 -- 172.21.15.141:0/2421189921 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7500899c0 con 0x7fe758071310 2024-01-16T15:16:32.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.714+0000 7fe75da00640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe73006c550 0x7fe73006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.716+0000 7fe75da00640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe73006c550 0x7fe73006ea10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe748001050 tx=0x7fe748009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.718+0000 7fe755600640 1 -- 172.21.15.141:0/2421189921 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe750055f00 con 0x7fe758071310 2024-01-16T15:16:32.925 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.939+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7580cc320 con 0x7fe73006c550 2024-01-16T15:16:32.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.950+0000 7fe755600640 1 -- 172.21.15.141:0/2421189921 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fe7580cc320 con 0x7fe73006c550 2024-01-16T15:16:32.937 INFO:teuthology.orchestra.run.smithi141.stdout:/volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d 2024-01-16T15:16:32.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.953+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe73006c550 msgr2=0x7fe73006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.953+0000 7fe760400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe73006c550 0x7fe73006ea10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe748001050 tx=0x7fe748009040 comp rx=0 tx=0).stop 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.953+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe758071310 msgr2=0x7fe758071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.953+0000 7fe760400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe758071310 0x7fe758071790 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fe75815b6b0 tx=0x7fe75000f520 comp rx=0 tx=0).stop 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 shutdown_connections 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe73006c550 0x7fe73006ea10 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe758071310 0x7fe758071790 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe75815aa70 0x7fe75813d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 --2- 172.21.15.141:0/2421189921 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe738003eb0 0x7fe7580afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 >> 172.21.15.141:0/2421189921 conn(0x7fe75806a800 msgr2=0x7fe7580cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 shutdown_connections 2024-01-16T15:16:32.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:32.954+0000 7fe760400640 1 -- 172.21.15.141:0/2421189921 wait complete. 2024-01-16T15:16:32.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:32.959 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d) 2024-01-16T15:16:33.016 INFO:teuthology.orchestra.run.smithi141.stdout:# file: volumes/_nogroup/sv1/1b241b64-cda2-4d04-9ef9-f50dbd33519d 2024-01-16T15:16:33.016 INFO:teuthology.orchestra.run.smithi141.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:16:33.016 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:16:33.018 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:16:33.018 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone7001 2024-01-16T15:16:33.234 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.246+0000 7f31b7600640 1 Processor -- start 2024-01-16T15:16:33.235 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.246+0000 7f31b7600640 1 -- start start 2024-01-16T15:16:33.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31b7600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f31b0150760 0x7f31b0072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31b7600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f31b015d300 0x7f31b015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31b7600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b015dc80 con 0x7f31b00b2bd0 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31b7600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b00b3710 con 0x7f31b015d300 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31b7600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b0071190 con 0x7f31b0150760 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31b5600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f31b00b2bd0 0x7f31b00b3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33130/0 (socket says 172.21.15.141:33130) 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31afe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f31b015d300 0x7f31b015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.248+0000 7f31b5600640 1 -- 172.21.15.141:0/121925300 learned_addr learned my addr 172.21.15.141:0/121925300 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.249+0000 7f31b4c00640 1 --2- 172.21.15.141:0/121925300 >> v2:172.21.15.188:3300/0 conn(0x7f31b0150760 0x7f31b0072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.249+0000 7f31afe00640 1 -- 172.21.15.141:0/121925300 >> v2:172.21.15.188:3300/0 conn(0x7f31b0150760 msgr2=0x7f31b0072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:33.239 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.249+0000 7f31afe00640 1 --2- 172.21.15.141:0/121925300 >> v2:172.21.15.188:3300/0 conn(0x7f31b0150760 0x7f31b0072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.249+0000 7f31afe00640 1 -- 172.21.15.141:0/121925300 >> v1:172.21.15.141:6789/0 conn(0x7f31b00b2bd0 legacy=0x7f31b00b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.249+0000 7f31afe00640 1 -- 172.21.15.141:0/121925300 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0071470 con 0x7f31b015d300 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.249+0000 7f31af400640 1 -- 172.21.15.141:0/121925300 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667119240 0 0) 0x7f31b015dc80 con 0x7f31b00b2bd0 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.250+0000 7f31afe00640 1 --2- 172.21.15.141:0/121925300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b015d740 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f31b006e420 tx=0x7f319802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ca0eb773f29a11 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.250+0000 7f31af400640 1 -- 172.21.15.141:0/121925300 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198031830 con 0x7f31b015d300 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.250+0000 7f31af400640 1 -- 172.21.15.141:0/121925300 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198031990 con 0x7f31b015d300 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.250+0000 7f31af400640 1 -- 172.21.15.141:0/121925300 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198031d20 con 0x7f31b015d300 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 -- 172.21.15.141:0/121925300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 msgr2=0x7f31b015d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 --2- 172.21.15.141:0/121925300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b015d740 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f31b006e420 tx=0x7f319802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 -- 172.21.15.141:0/121925300 shutdown_connections 2024-01-16T15:16:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 --2- 172.21.15.141:0/121925300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b015d740 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 --2- 172.21.15.141:0/121925300 >> v2:172.21.15.188:3300/0 conn(0x7f31b0150760 0x7f31b0072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 -- 172.21.15.141:0/121925300 >> 172.21.15.141:0/121925300 conn(0x7f31b006a800 msgr2=0x7f31b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 -- 172.21.15.141:0/121925300 shutdown_connections 2024-01-16T15:16:33.241 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.251+0000 7f31b7600640 1 -- 172.21.15.141:0/121925300 wait complete. 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.253+0000 7f31b7600640 1 Processor -- start 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.253+0000 7f31b7600640 1 -- start start 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b7600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31b00b2bd0 0x7f31b00cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b7600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31b0150760 0x7f31b00cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b7600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b00cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b7600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b0133320 con 0x7f31b015d300 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b7600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b01521e0 con 0x7f31b00b2bd0 2024-01-16T15:16:33.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b7600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b013c4e0 con 0x7f31b0150760 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b00cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b00cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:36798/0 (socket says 172.21.15.141:36798) 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.254+0000 7f31b5600640 1 -- 172.21.15.141:0/1792334449 learned_addr learned my addr 172.21.15.141:0/1792334449 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31afe00640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31b0150760 0x7f31b00cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b4c00640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31b00b2bd0 0x7f31b00cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b5600640 1 -- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31b0150760 msgr2=0x7f31b00cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b5600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31b0150760 0x7f31b00cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b5600640 1 -- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31b00b2bd0 msgr2=0x7f31b00cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b5600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31b00b2bd0 0x7f31b00cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b5600640 1 -- 172.21.15.141:0/1792334449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3198009400 con 0x7f31b015d300 2024-01-16T15:16:33.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31afe00640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31b0150760 0x7f31b00cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b5600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b00cca90 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f31a800c780 tx=0x7f31a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31acc00640 1 -- 172.21.15.141:0/1792334449 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a800ce90 con 0x7f31b015d300 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31acc00640 1 -- 172.21.15.141:0/1792334449 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a800a640 con 0x7f31b015d300 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b013c7c0 con 0x7f31b015d300 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31acc00640 1 -- 172.21.15.141:0/1792334449 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a800a9d0 con 0x7f31b015d300 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.255+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b013cd80 con 0x7f31b015d300 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.257+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b0152020 con 0x7f31b015d300 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.258+0000 7f31acc00640 1 -- 172.21.15.141:0/1792334449 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 41) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f31a800ab30 con 0x7f31b015d300 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.258+0000 7f31acc00640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f318806c550 0x7f318806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.259+0000 7f31acc00640 1 -- 172.21.15.141:0/1792334449 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f31a80880a0 con 0x7f31b015d300 2024-01-16T15:16:33.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.259+0000 7f31b4c00640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f318806c550 0x7f318806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.260+0000 7f31b4c00640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f318806c550 0x7f318806ea10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f31a0006fd0 tx=0x7f31a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.263+0000 7f31acc00640 1 -- 172.21.15.141:0/1792334449 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31b0152020 con 0x7f31b015d300 2024-01-16T15:16:33.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.488+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone7001", "target": ["mon-mgr", ""]}) v1 -- 0x7f31b0071d10 con 0x7f318806c550 2024-01-16T15:16:33.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.641+0000 7f31acc00640 1 -- 172.21.15.141:0/1792334449 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f31a0009540 con 0x7f318806c550 2024-01-16T15:16:33.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f318806c550 msgr2=0x7f318806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f318806c550 0x7f318806ea10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f31a0006fd0 tx=0x7f31a0008040 comp rx=0 tx=0).stop 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 msgr2=0x7f31b00cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b00cca90 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f31a800c780 tx=0x7f31a800cc40 comp rx=0 tx=0).stop 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 shutdown_connections 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f318806c550 0x7f318806ea10 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31b015d300 0x7f31b00cca90 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f31b0150760 0x7f31b00cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 --2- 172.21.15.141:0/1792334449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31b00b2bd0 0x7f31b00cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 >> 172.21.15.141:0/1792334449 conn(0x7f31b006a800 msgr2=0x7f31b015d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:33.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 shutdown_connections 2024-01-16T15:16:33.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.645+0000 7f31b7600640 1 -- 172.21.15.141:0/1792334449 wait complete. 2024-01-16T15:16:33.638 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.651+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.638 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.651+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.638 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.651+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.639 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.651+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.639 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.651+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.653 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone7002 2024-01-16T15:16:33.661 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.674+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.662 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.674+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.662 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.674+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.662 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.674+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.662 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:16:33.674+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.864 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.878+0000 7fe855c00640 1 Processor -- start 2024-01-16T15:16:33.865 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.878+0000 7fe855c00640 1 -- start start 2024-01-16T15:16:33.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.879+0000 7fe855c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe850143c40 0x7fe850071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe8500ccc10 0x7fe85006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe850070e50 con 0x7fe8500717d0 2024-01-16T15:16:33.867 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8500723c0 con 0x7fe8500ccc10 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe850144020 con 0x7fe850143c40 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe8500ccc10 0x7fe85006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe847e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe8500717d0 0x7fe8500cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33146/0 (socket says 172.21.15.141:33146) 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe847e00640 1 -- 172.21.15.141:0/765211851 learned_addr learned my addr 172.21.15.141:0/765211851 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe84fe00640 1 --2- 172.21.15.141:0/765211851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe850143c40 0x7fe850071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855200640 1 -- 172.21.15.141:0/765211851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe850143c40 msgr2=0x7fe850071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855200640 1 --2- 172.21.15.141:0/765211851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe850143c40 0x7fe850071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.880+0000 7fe855200640 1 -- 172.21.15.141:0/765211851 >> v1:172.21.15.141:6789/0 conn(0x7fe8500717d0 legacy=0x7fe8500cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.881+0000 7fe855200640 1 -- 172.21.15.141:0/765211851 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe850071e00 con 0x7fe8500ccc10 2024-01-16T15:16:33.869 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.881+0000 7fe84f400640 1 -- 172.21.15.141:0/765211851 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195808488 0 0) 0x7fe850070e50 con 0x7fe8500717d0 2024-01-16T15:16:33.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.882+0000 7fe855200640 1 --2- 172.21.15.141:0/765211851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8500ccc10 0x7fe85006d160 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fe848009d90 tx=0x7fe84802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55e2f2eff626dff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.882+0000 7fe84f400640 1 -- 172.21.15.141:0/765211851 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84802cbc0 con 0x7fe8500ccc10 2024-01-16T15:16:33.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.882+0000 7fe84f400640 1 -- 172.21.15.141:0/765211851 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84802cd20 con 0x7fe8500ccc10 2024-01-16T15:16:33.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.882+0000 7fe84f400640 1 -- 172.21.15.141:0/765211851 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848034660 con 0x7fe8500ccc10 2024-01-16T15:16:33.870 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.882+0000 7fe855c00640 1 -- 172.21.15.141:0/765211851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8500ccc10 msgr2=0x7fe85006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.882+0000 7fe855c00640 1 --2- 172.21.15.141:0/765211851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8500ccc10 0x7fe85006d160 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fe848009d90 tx=0x7fe84802be10 comp rx=0 tx=0).stop 2024-01-16T15:16:33.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.883+0000 7fe855c00640 1 -- 172.21.15.141:0/765211851 shutdown_connections 2024-01-16T15:16:33.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.883+0000 7fe855c00640 1 --2- 172.21.15.141:0/765211851 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe8500ccc10 0x7fe85006d160 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.883+0000 7fe855c00640 1 --2- 172.21.15.141:0/765211851 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe850143c40 0x7fe850071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.883+0000 7fe855c00640 1 -- 172.21.15.141:0/765211851 >> 172.21.15.141:0/765211851 conn(0x7fe85006a800 msgr2=0x7fe85006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:33.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.883+0000 7fe855c00640 1 -- 172.21.15.141:0/765211851 shutdown_connections 2024-01-16T15:16:33.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.883+0000 7fe855c00640 1 -- 172.21.15.141:0/765211851 wait complete. 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.884+0000 7fe855c00640 1 Processor -- start 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.885+0000 7fe855c00640 1 -- start start 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.885+0000 7fe855c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8500717d0 0x7fe85013e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.885+0000 7fe855c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8500ccc10 0x7fe8500c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe850143c40 0x7fe8500c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8500c4cd0 con 0x7fe850143c40 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8500c7f30 con 0x7fe8500ccc10 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8500ca380 con 0x7fe8500717d0 2024-01-16T15:16:33.874 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe850143c40 0x7fe8500c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe850143c40 0x7fe8500c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:36810/0 (socket says 172.21.15.141:36810) 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 -- 172.21.15.141:0/1527080172 learned_addr learned my addr 172.21.15.141:0/1527080172 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 -- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8500717d0 msgr2=0x7fe85013e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8500717d0 0x7fe85013e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 -- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8500ccc10 msgr2=0x7fe8500c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8500ccc10 0x7fe8500c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 -- 172.21.15.141:0/1527080172 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe848009a70 con 0x7fe850143c40 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.886+0000 7fe855200640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe850143c40 0x7fe8500c9e40 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fe848009eb0 tx=0x7fe848004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.887+0000 7fe84cc00640 1 -- 172.21.15.141:0/1527080172 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848040650 con 0x7fe850143c40 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.887+0000 7fe84cc00640 1 -- 172.21.15.141:0/1527080172 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84802ce90 con 0x7fe850143c40 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.887+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8500ca600 con 0x7fe850143c40 2024-01-16T15:16:33.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.887+0000 7fe84cc00640 1 -- 172.21.15.141:0/1527080172 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8480409f0 con 0x7fe850143c40 2024-01-16T15:16:33.876 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.887+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8500cabc0 con 0x7fe850143c40 2024-01-16T15:16:33.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.888+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe850067450 con 0x7fe850143c40 2024-01-16T15:16:33.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.889+0000 7fe84cc00640 1 -- 172.21.15.141:0/1527080172 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 41) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe848040b50 con 0x7fe850143c40 2024-01-16T15:16:33.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.890+0000 7fe84cc00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe82806c550 0x7fe82806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.890+0000 7fe84cc00640 1 -- 172.21.15.141:0/1527080172 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe8480ab870 con 0x7fe850143c40 2024-01-16T15:16:33.877 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.891+0000 7fe84fe00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe82806c550 0x7fe82806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.878 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.892+0000 7fe84fe00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe82806c550 0x7fe82806ea10 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe840006fd0 tx=0x7fe840008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.880 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:33.894+0000 7fe84cc00640 1 -- 172.21.15.141:0/1527080172 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe848077db0 con 0x7fe850143c40 2024-01-16T15:16:34.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.119+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone7002", "target": ["mon-mgr", ""]}) v1 -- 0x7fe85006cc40 con 0x7fe82806c550 2024-01-16T15:16:34.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.299+0000 7fe84cc00640 1 -- 172.21.15.141:0/1527080172 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe84000be20 con 0x7fe82806c550 2024-01-16T15:16:34.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.302+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe82806c550 msgr2=0x7fe82806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.302+0000 7fe855c00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe82806c550 0x7fe82806ea10 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe840006fd0 tx=0x7fe840008040 comp rx=0 tx=0).stop 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.302+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe850143c40 msgr2=0x7fe8500c9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.302+0000 7fe855c00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe850143c40 0x7fe8500c9e40 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fe848009eb0 tx=0x7fe848004a40 comp rx=0 tx=0).stop 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 shutdown_connections 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe82806c550 0x7fe82806ea10 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe850143c40 0x7fe8500c9e40 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8500ccc10 0x7fe8500c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 --2- 172.21.15.141:0/1527080172 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe8500717d0 0x7fe85013e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 >> 172.21.15.141:0/1527080172 conn(0x7fe85006a800 msgr2=0x7fe8500cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:34.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 shutdown_connections 2024-01-16T15:16:34.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.303+0000 7fe855c00640 1 -- 172.21.15.141:0/1527080172 wait complete. 2024-01-16T15:16:34.312 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone7003 2024-01-16T15:16:34.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.535+0000 7fbb33200640 1 Processor -- start 2024-01-16T15:16:34.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.535+0000 7fbb33200640 1 -- start start 2024-01-16T15:16:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.537+0000 7fbb33200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb34071410 0x7fbb34146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.537+0000 7fbb33200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fbb34071f00 0x7fbb34166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.537+0000 7fbb33200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb340b2e10 con 0x7fbb3406c7c0 2024-01-16T15:16:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.537+0000 7fbb33200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3406cc50 con 0x7fbb34071410 2024-01-16T15:16:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.537+0000 7fbb33200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3406cec0 con 0x7fbb34071f00 2024-01-16T15:16:34.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb31e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbb34071410 0x7fbb34146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb2be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbb3406c7c0 0x7fbb340717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33156/0 (socket says 172.21.15.141:33156) 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb2be00640 1 -- 172.21.15.141:0/3219489859 learned_addr learned my addr 172.21.15.141:0/3219489859 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb32800640 1 --2- 172.21.15.141:0/3219489859 >> v2:172.21.15.188:3300/0 conn(0x7fbb34071f00 0x7fbb34166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb31400640 1 -- 172.21.15.141:0/3219489859 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017527650 0 0) 0x7fbb340b2e10 con 0x7fbb3406c7c0 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb31e00640 1 -- 172.21.15.141:0/3219489859 >> v2:172.21.15.188:3300/0 conn(0x7fbb34071f00 msgr2=0x7fbb34166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb31e00640 1 --2- 172.21.15.141:0/3219489859 >> v2:172.21.15.188:3300/0 conn(0x7fbb34071f00 0x7fbb34166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb31e00640 1 -- 172.21.15.141:0/3219489859 >> v1:172.21.15.141:6789/0 conn(0x7fbb3406c7c0 legacy=0x7fbb340717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb31e00640 1 -- 172.21.15.141:0/3219489859 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3406d1a0 con 0x7fbb34071410 2024-01-16T15:16:34.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.538+0000 7fbb32800640 1 --2- 172.21.15.141:0/3219489859 >> v2:172.21.15.188:3300/0 conn(0x7fbb34071f00 0x7fbb34166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.539+0000 7fbb31e00640 1 --2- 172.21.15.141:0/3219489859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071410 0x7fbb34146770 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fbb340ad5c0 tx=0x7fbb2402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9657eabf728c3f9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:34.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.539+0000 7fbb31400640 1 -- 172.21.15.141:0/3219489859 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24031830 con 0x7fbb34071410 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb31400640 1 -- 172.21.15.141:0/3219489859 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb24031990 con 0x7fbb34071410 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb31400640 1 -- 172.21.15.141:0/3219489859 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24031d20 con 0x7fbb34071410 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 -- 172.21.15.141:0/3219489859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071410 msgr2=0x7fbb34146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 --2- 172.21.15.141:0/3219489859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071410 0x7fbb34146770 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fbb340ad5c0 tx=0x7fbb2402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 -- 172.21.15.141:0/3219489859 shutdown_connections 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 --2- 172.21.15.141:0/3219489859 >> v2:172.21.15.188:3300/0 conn(0x7fbb34071f00 0x7fbb34166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 --2- 172.21.15.141:0/3219489859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071410 0x7fbb34146770 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 -- 172.21.15.141:0/3219489859 >> 172.21.15.141:0/3219489859 conn(0x7fbb3406a800 msgr2=0x7fbb3406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:34.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 -- 172.21.15.141:0/3219489859 shutdown_connections 2024-01-16T15:16:34.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.540+0000 7fbb33200640 1 -- 172.21.15.141:0/3219489859 wait complete. 2024-01-16T15:16:34.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.542+0000 7fbb33200640 1 Processor -- start 2024-01-16T15:16:34.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.542+0000 7fbb33200640 1 -- start start 2024-01-16T15:16:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.543+0000 7fbb33200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb3406c7c0 0x7fbb3413d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.543+0000 7fbb33200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb34071410 0x7fbb3415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.543+0000 7fbb33200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071f00 0x7fbb341f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.543+0000 7fbb33200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb340c3d70 con 0x7fbb34071f00 2024-01-16T15:16:34.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.543+0000 7fbb33200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb341f6cc0 con 0x7fbb34071410 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.543+0000 7fbb33200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb341f6fa0 con 0x7fbb3406c7c0 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071f00 0x7fbb341f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071f00 0x7fbb341f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:36828/0 (socket says 172.21.15.141:36828) 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 -- 172.21.15.141:0/1060368885 learned_addr learned my addr 172.21.15.141:0/1060368885 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb2be00640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb34071410 0x7fbb3415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb31e00640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb3406c7c0 0x7fbb3413d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 -- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb3406c7c0 msgr2=0x7fbb3413d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb3406c7c0 0x7fbb3413d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 -- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb34071410 msgr2=0x7fbb3415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb34071410 0x7fbb3415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb32800640 1 -- 172.21.15.141:0/1060368885 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24009400 con 0x7fbb34071f00 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.544+0000 7fbb2be00640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb34071410 0x7fbb3415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.545+0000 7fbb32800640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071f00 0x7fbb341f61c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00cd50 tx=0x7fbb2c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.545+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c00f680 con 0x7fbb34071f00 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.545+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2c00f800 con 0x7fbb34071f00 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.545+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb340c3fc0 con 0x7fbb34071f00 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.545+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c00fbb0 con 0x7fbb34071f00 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.545+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb34144fd0 con 0x7fbb34071f00 2024-01-16T15:16:34.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.546+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb340c44e0 con 0x7fbb34071f00 2024-01-16T15:16:34.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.547+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 41) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fbb2c00b040 con 0x7fbb34071f00 2024-01-16T15:16:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.548+0000 7fbb2a000640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb0c06c580 0x7fbb0c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.548+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbb2c0894e0 con 0x7fbb34071f00 2024-01-16T15:16:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.549+0000 7fbb31e00640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb0c06c580 0x7fbb0c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.550+0000 7fbb31e00640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb0c06c580 0x7fbb0c06ea40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbb24036000 tx=0x7fbb24008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:34.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.552+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb340c44e0 con 0x7fbb34071f00 2024-01-16T15:16:34.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:34.776+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone7003", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb3406da40 con 0x7fbb0c06c580 2024-01-16T15:16:36.108 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:36.121+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fbb2c00fd10 con 0x7fbb34071f00 2024-01-16T15:16:37.776 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:37.785 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:38.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.035+0000 7fbb2a000640 1 -- 172.21.15.141:0/1060368885 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbb24031b80 con 0x7fbb0c06c580 2024-01-16T15:16:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb0c06c580 msgr2=0x7fbb0c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb0c06c580 0x7fbb0c06ea40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbb24036000 tx=0x7fbb24008030 comp rx=0 tx=0).stop 2024-01-16T15:16:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071f00 msgr2=0x7fbb341f61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071f00 0x7fbb341f61c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00cd50 tx=0x7fbb2c00a650 comp rx=0 tx=0).stop 2024-01-16T15:16:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 shutdown_connections 2024-01-16T15:16:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbb0c06c580 0x7fbb0c06ea40 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb34071f00 0x7fbb341f61c0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb34071410 0x7fbb3415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 --2- 172.21.15.141:0/1060368885 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbb3406c7c0 0x7fbb3413d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 >> 172.21.15.141:0/1060368885 conn(0x7fbb3406a800 msgr2=0x7fbb3415ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:38.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 shutdown_connections 2024-01-16T15:16:38.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.039+0000 7fbb33200640 1 -- 172.21.15.141:0/1060368885 wait complete. 2024-01-16T15:16:38.052 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone7004 2024-01-16T15:16:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.280+0000 7ffafd600640 1 Processor -- start 2024-01-16T15:16:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.280+0000 7ffafd600640 1 -- start start 2024-01-16T15:16:38.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.282+0000 7ffafd600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ffaf813df80 0x7ffaf815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:38.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.282+0000 7ffafd600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ffaf813e360 0x7ffaf80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:38.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.282+0000 7ffafd600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf806c470 con 0x7ffaf815f9c0 2024-01-16T15:16:38.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.282+0000 7ffafd600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf806c750 con 0x7ffaf813e360 2024-01-16T15:16:38.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.282+0000 7ffafd600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf806ca30 con 0x7ffaf813df80 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaf7400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ffaf815f9c0 0x7ffaf81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33164/0 (socket says 172.21.15.141:33164) 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaefe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ffaf813e360 0x7ffaf80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaf7400640 1 -- 172.21.15.141:0/3103503814 learned_addr learned my addr 172.21.15.141:0/3103503814 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaf6a00640 1 --2- 172.21.15.141:0/3103503814 >> v2:172.21.15.188:3300/0 conn(0x7ffaf813df80 0x7ffaf815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaefe00640 1 -- 172.21.15.141:0/3103503814 >> v2:172.21.15.188:3300/0 conn(0x7ffaf813df80 msgr2=0x7ffaf815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaefe00640 1 --2- 172.21.15.141:0/3103503814 >> v2:172.21.15.188:3300/0 conn(0x7ffaf813df80 0x7ffaf815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaefe00640 1 -- 172.21.15.141:0/3103503814 >> v1:172.21.15.141:6789/0 conn(0x7ffaf815f9c0 legacy=0x7ffaf81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaefe00640 1 -- 172.21.15.141:0/3103503814 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf806cd10 con 0x7ffaf813e360 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaf6a00640 1 --2- 172.21.15.141:0/3103503814 >> v2:172.21.15.188:3300/0 conn(0x7ffaf813df80 0x7ffaf815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:16:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.283+0000 7ffaf6000640 1 -- 172.21.15.141:0/3103503814 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865312422 0 0) 0x7ffaf806c470 con 0x7ffaf815f9c0 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.284+0000 7ffaefe00640 1 --2- 172.21.15.141:0/3103503814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813e360 0x7ffaf80cf290 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8072770 tx=0x7ffae002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ef87259768fa34 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffaf6000640 1 -- 172.21.15.141:0/3103503814 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0031830 con 0x7ffaf813e360 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffaf6000640 1 -- 172.21.15.141:0/3103503814 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0031990 con 0x7ffaf813e360 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffaf6000640 1 -- 172.21.15.141:0/3103503814 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0031d20 con 0x7ffaf813e360 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 -- 172.21.15.141:0/3103503814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813e360 msgr2=0x7ffaf80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 --2- 172.21.15.141:0/3103503814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813e360 0x7ffaf80cf290 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8072770 tx=0x7ffae002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 -- 172.21.15.141:0/3103503814 shutdown_connections 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 --2- 172.21.15.141:0/3103503814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813e360 0x7ffaf80cf290 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 --2- 172.21.15.141:0/3103503814 >> v2:172.21.15.188:3300/0 conn(0x7ffaf813df80 0x7ffaf815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 -- 172.21.15.141:0/3103503814 >> 172.21.15.141:0/3103503814 conn(0x7ffaf806a800 msgr2=0x7ffaf806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 -- 172.21.15.141:0/3103503814 shutdown_connections 2024-01-16T15:16:38.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.285+0000 7ffafd600640 1 -- 172.21.15.141:0/3103503814 wait complete. 2024-01-16T15:16:38.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.287+0000 7ffafd600640 1 Processor -- start 2024-01-16T15:16:38.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.287+0000 7ffafd600640 1 -- start start 2024-01-16T15:16:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.288+0000 7ffafd600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813df80 0x7ffaf815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.288+0000 7ffafd600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaf813e360 0x7ffaf815c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.288+0000 7ffafd600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaf815f9c0 0x7ffaf814f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.288+0000 7ffafd600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815cf10 con 0x7ffaf813df80 2024-01-16T15:16:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.288+0000 7ffafd600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815d1f0 con 0x7ffaf815f9c0 2024-01-16T15:16:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.288+0000 7ffafd600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815d4d0 con 0x7ffaf813e360 2024-01-16T15:16:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.288+0000 7ffaf6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813df80 0x7ffaf815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813df80 0x7ffaf815dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:36840/0 (socket says 172.21.15.141:36840) 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 -- 172.21.15.141:0/4198842261 learned_addr learned my addr 172.21.15.141:0/4198842261 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaefe00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaf813e360 0x7ffaf815c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf7400640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaf815f9c0 0x7ffaf814f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 -- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaf813e360 msgr2=0x7ffaf815c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaf813e360 0x7ffaf815c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 -- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaf815f9c0 msgr2=0x7ffaf814f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaf815f9c0 0x7ffaf814f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 -- 172.21.15.141:0/4198842261 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0009400 con 0x7ffaf813df80 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaefe00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaf813e360 0x7ffaf815c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaf6a00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813df80 0x7ffaf815dfb0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ffae800d7d0 tx=0x7ffae800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.289+0000 7ffaeea00640 1 -- 172.21.15.141:0/4198842261 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8004030 con 0x7ffaf813df80 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.290+0000 7ffaeea00640 1 -- 172.21.15.141:0/4198842261 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8004190 con 0x7ffaf813df80 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.290+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf815d7b0 con 0x7ffaf813df80 2024-01-16T15:16:38.277 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.290+0000 7ffaeea00640 1 -- 172.21.15.141:0/4198842261 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8004520 con 0x7ffaf813df80 2024-01-16T15:16:38.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.290+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf814f820 con 0x7ffaf813df80 2024-01-16T15:16:38.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.291+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf80675d0 con 0x7ffaf813df80 2024-01-16T15:16:38.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.292+0000 7ffaeea00640 1 -- 172.21.15.141:0/4198842261 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7ffae800b6c0 con 0x7ffaf813df80 2024-01-16T15:16:38.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.292+0000 7ffaeea00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffad006c6a0 0x7ffad006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:38.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.293+0000 7ffaeea00640 1 -- 172.21.15.141:0/4198842261 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffae8088d00 con 0x7ffaf813df80 2024-01-16T15:16:38.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.293+0000 7ffaefe00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffad006c6a0 0x7ffad006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:38.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.294+0000 7ffaefe00640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffad006c6a0 0x7ffad006eb60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffae0036000 tx=0x7ffae0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:38.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.297+0000 7ffaeea00640 1 -- 172.21.15.141:0/4198842261 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffae8055270 con 0x7ffaf813df80 2024-01-16T15:16:38.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:38.523+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone7004", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaf80ce850 con 0x7ffad006c6a0 2024-01-16T15:16:43.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.064+0000 7ffaeea00640 1 -- 172.21.15.141:0/4198842261 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffae0045440 con 0x7ffad006c6a0 2024-01-16T15:16:43.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffad006c6a0 msgr2=0x7ffad006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:43.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffad006c6a0 0x7ffad006eb60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffae0036000 tx=0x7ffae0008030 comp rx=0 tx=0).stop 2024-01-16T15:16:43.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813df80 msgr2=0x7ffaf815dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:43.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813df80 0x7ffaf815dfb0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ffae800d7d0 tx=0x7ffae800dc90 comp rx=0 tx=0).stop 2024-01-16T15:16:43.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 shutdown_connections 2024-01-16T15:16:43.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffad006c6a0 0x7ffad006eb60 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaf815f9c0 0x7ffaf814f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaf813e360 0x7ffaf815c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 --2- 172.21.15.141:0/4198842261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf813df80 0x7ffaf815dfb0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 >> 172.21.15.141:0/4198842261 conn(0x7ffaf806a800 msgr2=0x7ffaf8153a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:43.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 shutdown_connections 2024-01-16T15:16:43.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.068+0000 7ffafd600640 1 -- 172.21.15.141:0/4198842261 wait complete. 2024-01-16T15:16:43.079 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone7005 2024-01-16T15:16:43.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.301+0000 7f3c50800640 1 Processor -- start 2024-01-16T15:16:43.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.301+0000 7f3c50800640 1 -- start start 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c50800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3c4814b2c0 0x7f3c480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c50800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3c480cf7d0 0x7f3c48070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c50800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c480cfbf0 con 0x7f3c48070ea0 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c50800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4814b6a0 con 0x7f3c4814b2c0 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c50800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c48071690 con 0x7f3c480cf7d0 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c4de00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3c4814b2c0 0x7f3c480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c4e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3c48070ea0 0x7f3c480b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58588/0 (socket says 172.21.15.141:58588) 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.303+0000 7f3c4e800640 1 -- 172.21.15.141:0/191221765 learned_addr learned my addr 172.21.15.141:0/191221765 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:16:43.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.304+0000 7f3c4d400640 1 --2- 172.21.15.141:0/191221765 >> v2:172.21.15.188:3300/0 conn(0x7f3c480cf7d0 0x7f3c48070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:43.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.304+0000 7f3c4de00640 1 -- 172.21.15.141:0/191221765 >> v2:172.21.15.188:3300/0 conn(0x7f3c480cf7d0 msgr2=0x7f3c48070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:43.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.304+0000 7f3c4de00640 1 --2- 172.21.15.141:0/191221765 >> v2:172.21.15.188:3300/0 conn(0x7f3c480cf7d0 0x7f3c48070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.304+0000 7f3c4de00640 1 -- 172.21.15.141:0/191221765 >> v1:172.21.15.141:6789/0 conn(0x7f3c48070ea0 legacy=0x7f3c480b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:43.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.304+0000 7f3c4de00640 1 -- 172.21.15.141:0/191221765 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48071970 con 0x7f3c4814b2c0 2024-01-16T15:16:43.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.304+0000 7f3c4ca00640 1 -- 172.21.15.141:0/191221765 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545458750 0 0) 0x7f3c480cfbf0 con 0x7f3c48070ea0 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.305+0000 7f3c4de00640 1 --2- 172.21.15.141:0/191221765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c4814b2c0 0x7f3c480cf290 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3c480cfea0 tx=0x7f3c3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfb791765a9be4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.305+0000 7f3c4ca00640 1 -- 172.21.15.141:0/191221765 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3802cc80 con 0x7f3c4814b2c0 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.305+0000 7f3c4ca00640 1 -- 172.21.15.141:0/191221765 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3802cde0 con 0x7f3c4814b2c0 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.305+0000 7f3c4ca00640 1 -- 172.21.15.141:0/191221765 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38034660 con 0x7f3c4814b2c0 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 -- 172.21.15.141:0/191221765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c4814b2c0 msgr2=0x7f3c480cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 --2- 172.21.15.141:0/191221765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c4814b2c0 0x7f3c480cf290 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3c480cfea0 tx=0x7f3c3802bed0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 -- 172.21.15.141:0/191221765 shutdown_connections 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 --2- 172.21.15.141:0/191221765 >> v2:172.21.15.188:3300/0 conn(0x7f3c480cf7d0 0x7f3c48070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 --2- 172.21.15.141:0/191221765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c4814b2c0 0x7f3c480cf290 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 -- 172.21.15.141:0/191221765 >> 172.21.15.141:0/191221765 conn(0x7f3c4806a800 msgr2=0x7f3c4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 -- 172.21.15.141:0/191221765 shutdown_connections 2024-01-16T15:16:43.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.306+0000 7f3c50800640 1 -- 172.21.15.141:0/191221765 wait complete. 2024-01-16T15:16:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.308+0000 7f3c50800640 1 Processor -- start 2024-01-16T15:16:43.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.308+0000 7f3c50800640 1 -- start start 2024-01-16T15:16:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c50800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c48070ea0 0x7f3c481fa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c50800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c480cf7d0 0x7f3c481642b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c50800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c4814b2c0 0x7f3c48160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c50800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c481f9220 con 0x7f3c480cf7d0 2024-01-16T15:16:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c50800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c481f9500 con 0x7f3c4814b2c0 2024-01-16T15:16:43.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c50800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c481f97e0 con 0x7f3c48070ea0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c4d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c480cf7d0 0x7f3c481642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c4d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c480cf7d0 0x7f3c481642b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58926/0 (socket says 172.21.15.141:58926) 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.309+0000 7f3c4d400640 1 -- 172.21.15.141:0/3383363320 learned_addr learned my addr 172.21.15.141:0/3383363320 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4de00640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c48070ea0 0x7f3c481fa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4e800640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c4814b2c0 0x7f3c48160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4d400640 1 -- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c48070ea0 msgr2=0x7f3c481fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4d400640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c48070ea0 0x7f3c481fa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4d400640 1 -- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c4814b2c0 msgr2=0x7f3c48160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4d400640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c4814b2c0 0x7f3c48160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4d400640 1 -- 172.21.15.141:0/3383363320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38009a70 con 0x7f3c480cf7d0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c4d400640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c480cf7d0 0x7f3c481642b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000d800 tx=0x7f3c3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.310+0000 7f3c46000640 1 -- 172.21.15.141:0/3383363320 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3000b6c0 con 0x7f3c480cf7d0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.311+0000 7f3c46000640 1 -- 172.21.15.141:0/3383363320 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3000b820 con 0x7f3c480cf7d0 2024-01-16T15:16:43.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.311+0000 7f3c46000640 1 -- 172.21.15.141:0/3383363320 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3000bbb0 con 0x7f3c480cf7d0 2024-01-16T15:16:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.311+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c481f9ac0 con 0x7f3c480cf7d0 2024-01-16T15:16:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.311+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c481fa800 con 0x7f3c480cf7d0 2024-01-16T15:16:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.312+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c480675d0 con 0x7f3c480cf7d0 2024-01-16T15:16:43.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.313+0000 7f3c46000640 1 -- 172.21.15.141:0/3383363320 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f3c3000bd10 con 0x7f3c480cf7d0 2024-01-16T15:16:43.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.314+0000 7f3c46000640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c2006c670 0x7f3c2006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:43.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.314+0000 7f3c46000640 1 -- 172.21.15.141:0/3383363320 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3c30089210 con 0x7f3c480cf7d0 2024-01-16T15:16:43.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.314+0000 7f3c4de00640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c2006c670 0x7f3c2006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:43.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.316+0000 7f3c4de00640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c2006c670 0x7f3c2006eb30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3c38036000 tx=0x7f3c38030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:43.305 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.319+0000 7f3c46000640 1 -- 172.21.15.141:0/3383363320 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c30055690 con 0x7f3c480cf7d0 2024-01-16T15:16:43.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:43.550+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone7005", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c481f9f80 con 0x7f3c2006c670 2024-01-16T15:16:48.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.059+0000 7f3c46000640 1 -- 172.21.15.141:0/3383363320 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3c481f9f80 con 0x7f3c2006c670 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c2006c670 msgr2=0x7f3c2006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c2006c670 0x7f3c2006eb30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3c38036000 tx=0x7f3c38030000 comp rx=0 tx=0).stop 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c480cf7d0 msgr2=0x7f3c481642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c480cf7d0 0x7f3c481642b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000d800 tx=0x7f3c3000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 shutdown_connections 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3c2006c670 0x7f3c2006eb30 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c4814b2c0 0x7f3c48160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c480cf7d0 0x7f3c481642b0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 --2- 172.21.15.141:0/3383363320 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c48070ea0 0x7f3c481fa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.062+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 >> 172.21.15.141:0/3383363320 conn(0x7f3c4806a800 msgr2=0x7f3c480cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:48.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.063+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 shutdown_connections 2024-01-16T15:16:48.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.063+0000 7f3c50800640 1 -- 172.21.15.141:0/3383363320 wait complete. 2024-01-16T15:16:48.070 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone7006 2024-01-16T15:16:48.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.300+0000 7f9974400640 1 Processor -- start 2024-01-16T15:16:48.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.300+0000 7f9974400640 1 -- start start 2024-01-16T15:16:48.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9974400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f996c13df80 0x7f996c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9974400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13e360 0x7f996c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9974400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f996c15f9c0 0x7f996c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9974400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c06c470 con 0x7f996c13df80 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9974400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c06c750 con 0x7f996c15f9c0 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9974400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c06ca30 con 0x7f996c13e360 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9971a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f996c13df80 0x7f996c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9971a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f996c13df80 0x7f996c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58940/0 (socket says 172.21.15.141:58940) 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.302+0000 7f9971a00640 1 -- 172.21.15.141:0/1022680891 learned_addr learned my addr 172.21.15.141:0/1022680891 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.303+0000 7f9972400640 1 --2- 172.21.15.141:0/1022680891 >> v2:172.21.15.188:3300/0 conn(0x7f996c15f9c0 0x7f996c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.303+0000 7f9971000640 1 --2- 172.21.15.141:0/1022680891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13e360 0x7f996c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.303+0000 7f9971a00640 1 -- 172.21.15.141:0/1022680891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13e360 msgr2=0x7f996c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.303+0000 7f9971a00640 1 --2- 172.21.15.141:0/1022680891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13e360 0x7f996c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.303+0000 7f9971a00640 1 -- 172.21.15.141:0/1022680891 >> v2:172.21.15.188:3300/0 conn(0x7f996c15f9c0 msgr2=0x7f996c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:48.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.303+0000 7f9971a00640 1 --2- 172.21.15.141:0/1022680891 >> v2:172.21.15.188:3300/0 conn(0x7f996c15f9c0 0x7f996c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.303+0000 7f9971a00640 1 -- 172.21.15.141:0/1022680891 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c06cd10 con 0x7f996c13df80 2024-01-16T15:16:48.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.304+0000 7f9971a00640 1 --2- 172.21.15.141:0/1022680891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13df80 0x7f996c15bee0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f996c06e6d0 tx=0x7f995c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b20422a79687d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:48.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f996be00640 1 -- 172.21.15.141:0/1022680891 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c031870 con 0x7f996c13df80 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f996be00640 1 -- 172.21.15.141:0/1022680891 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c0319d0 con 0x7f996c13df80 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f996be00640 1 -- 172.21.15.141:0/1022680891 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c031d60 con 0x7f996c13df80 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f9974400640 1 -- 172.21.15.141:0/1022680891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13df80 msgr2=0x7f996c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f9974400640 1 --2- 172.21.15.141:0/1022680891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13df80 0x7f996c15bee0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f996c06e6d0 tx=0x7f995c02b910 comp rx=0 tx=0).stop 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f9974400640 1 -- 172.21.15.141:0/1022680891 shutdown_connections 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f9974400640 1 --2- 172.21.15.141:0/1022680891 >> v2:172.21.15.188:3300/0 conn(0x7f996c15f9c0 0x7f996c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f9974400640 1 --2- 172.21.15.141:0/1022680891 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13e360 0x7f996c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f9974400640 1 --2- 172.21.15.141:0/1022680891 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13df80 0x7f996c15bee0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.305+0000 7f9974400640 1 -- 172.21.15.141:0/1022680891 >> 172.21.15.141:0/1022680891 conn(0x7f996c06a800 msgr2=0x7f996c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.306+0000 7f9974400640 1 -- 172.21.15.141:0/1022680891 shutdown_connections 2024-01-16T15:16:48.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.306+0000 7f9974400640 1 -- 172.21.15.141:0/1022680891 wait complete. 2024-01-16T15:16:48.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.307+0000 7f9974400640 1 Processor -- start 2024-01-16T15:16:48.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.307+0000 7f9974400640 1 -- start start 2024-01-16T15:16:48.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.308+0000 7f9974400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13df80 0x7f996c0adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.308+0000 7f9974400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13e360 0x7f996c148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.308+0000 7f9974400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c15f9c0 0x7f996c148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.308+0000 7f9974400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1461d0 con 0x7f996c13e360 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.308+0000 7f9974400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c146790 con 0x7f996c15f9c0 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.308+0000 7f9974400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0ae200 con 0x7f996c13df80 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13e360 0x7f996c148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13e360 0x7f996c148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58946/0 (socket says 172.21.15.141:58946) 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 -- 172.21.15.141:0/2891180337 learned_addr learned my addr 172.21.15.141:0/2891180337 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9972400640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c15f9c0 0x7f996c148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 -- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13df80 msgr2=0x7f996c0adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13df80 0x7f996c0adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.298 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 -- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c15f9c0 msgr2=0x7f996c148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c15f9c0 0x7f996c148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 -- 172.21.15.141:0/2891180337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c0094d0 con 0x7f996c13e360 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.309+0000 7f9971000640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13e360 0x7f996c148210 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f995400c7f0 tx=0x7f995400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.310+0000 7f9969600640 1 -- 172.21.15.141:0/2891180337 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954004220 con 0x7f996c13e360 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.310+0000 7f9969600640 1 -- 172.21.15.141:0/2891180337 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99540043a0 con 0x7f996c13e360 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.310+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c148d40 con 0x7f996c13e360 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.310+0000 7f9969600640 1 -- 172.21.15.141:0/2891180337 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954007860 con 0x7f996c13e360 2024-01-16T15:16:48.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.310+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c146420 con 0x7f996c13e360 2024-01-16T15:16:48.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.311+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996c0675d0 con 0x7f996c13e360 2024-01-16T15:16:48.300 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.312+0000 7f9969600640 1 -- 172.21.15.141:0/2891180337 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f9954007bd0 con 0x7f996c13e360 2024-01-16T15:16:48.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.313+0000 7f9969600640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406c6a0 0x7f994406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.313+0000 7f9969600640 1 -- 172.21.15.141:0/2891180337 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9954088b40 con 0x7f996c13e360 2024-01-16T15:16:48.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.313+0000 7f9971a00640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406c6a0 0x7f994406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.314+0000 7f9971a00640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406c6a0 0x7f994406eb60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f995c036000 tx=0x7f995c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:48.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.317+0000 7f9969600640 1 -- 172.21.15.141:0/2891180337 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99540550b0 con 0x7f996c13e360 2024-01-16T15:16:48.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.542+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone7006", "target": ["mon-mgr", ""]}) v1 -- 0x7f996c1559e0 con 0x7f994406c6a0 2024-01-16T15:16:48.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.714+0000 7f9969600640 1 -- 172.21.15.141:0/2891180337 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f996c1559e0 con 0x7f994406c6a0 2024-01-16T15:16:48.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.717+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406c6a0 msgr2=0x7f994406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.717+0000 7f9974400640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406c6a0 0x7f994406eb60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f995c036000 tx=0x7f995c030000 comp rx=0 tx=0).stop 2024-01-16T15:16:48.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.717+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13e360 msgr2=0x7f996c148210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.717+0000 7f9974400640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13e360 0x7f996c148210 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f995400c7f0 tx=0x7f995400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 shutdown_connections 2024-01-16T15:16:48.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f994406c6a0 0x7f994406eb60 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f996c15f9c0 0x7f996c148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f996c13e360 0x7f996c148210 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 --2- 172.21.15.141:0/2891180337 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f996c13df80 0x7f996c0adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 >> 172.21.15.141:0/2891180337 conn(0x7f996c06a800 msgr2=0x7f996c15a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:48.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 shutdown_connections 2024-01-16T15:16:48.706 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.718+0000 7f9974400640 1 -- 172.21.15.141:0/2891180337 wait complete. 2024-01-16T15:16:48.727 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone7007 2024-01-16T15:16:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.957+0000 7fc14a600640 1 Processor -- start 2024-01-16T15:16:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.957+0000 7fc14a600640 1 -- start start 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.959+0000 7fc14a600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1440d0b60 0x7fc1440ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.959+0000 7fc14a600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc144071de0 con 0x7fc14406a3e0 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.959+0000 7fc14a600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1440af270 con 0x7fc1440aede0 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.959+0000 7fc14a600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc144072030 con 0x7fc1440d0b60 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.959+0000 7fc143400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc14406a3e0 0x7fc14406a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58604/0 (socket says 172.21.15.141:58604) 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.959+0000 7fc143400640 1 -- 172.21.15.141:0/2006214064 learned_addr learned my addr 172.21.15.141:0/2006214064 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.959+0000 7fc142a00640 1 --2- 172.21.15.141:0/2006214064 >> v2:172.21.15.141:3300/0 conn(0x7fc1440d0b60 0x7fc1440ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.960+0000 7fc142000640 1 -- 172.21.15.141:0/2006214064 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504820005 0 0) 0x7fc144071de0 con 0x7fc14406a3e0 2024-01-16T15:16:48.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.960+0000 7fc142000640 1 -- 172.21.15.141:0/2006214064 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc124003520 con 0x7fc14406a3e0 2024-01-16T15:16:48.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.960+0000 7fc142a00640 1 -- 172.21.15.141:0/2006214064 >> v1:172.21.15.188:6789/0 conn(0x7fc1440aede0 legacy=0x7fc14406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:48.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.960+0000 7fc142a00640 1 -- 172.21.15.141:0/2006214064 >> v1:172.21.15.141:6789/0 conn(0x7fc14406a3e0 legacy=0x7fc14406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.960+0000 7fc142a00640 1 -- 172.21.15.141:0/2006214064 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144072310 con 0x7fc1440d0b60 2024-01-16T15:16:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.961+0000 7fc142a00640 1 --2- 172.21.15.141:0/2006214064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440d0b60 0x7fc1440ac8d0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0097a0 tx=0x7fc12c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c986d86041d02c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.961+0000 7fc142000640 1 -- 172.21.15.141:0/2006214064 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c031830 con 0x7fc1440d0b60 2024-01-16T15:16:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.961+0000 7fc142000640 1 -- 172.21.15.141:0/2006214064 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c031990 con 0x7fc1440d0b60 2024-01-16T15:16:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.961+0000 7fc142000640 1 -- 172.21.15.141:0/2006214064 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c031d20 con 0x7fc1440d0b60 2024-01-16T15:16:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.962+0000 7fc14a600640 1 -- 172.21.15.141:0/2006214064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440d0b60 msgr2=0x7fc1440ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.962+0000 7fc14a600640 1 --2- 172.21.15.141:0/2006214064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440d0b60 0x7fc1440ac8d0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0097a0 tx=0x7fc12c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.962+0000 7fc14a600640 1 -- 172.21.15.141:0/2006214064 shutdown_connections 2024-01-16T15:16:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.962+0000 7fc14a600640 1 --2- 172.21.15.141:0/2006214064 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440d0b60 0x7fc1440ac8d0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.962+0000 7fc14a600640 1 -- 172.21.15.141:0/2006214064 >> 172.21.15.141:0/2006214064 conn(0x7fc1440cfef0 msgr2=0x7fc1440d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.962+0000 7fc14a600640 1 -- 172.21.15.141:0/2006214064 shutdown_connections 2024-01-16T15:16:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.962+0000 7fc14a600640 1 -- 172.21.15.141:0/2006214064 wait complete. 2024-01-16T15:16:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.964+0000 7fc14a600640 1 Processor -- start 2024-01-16T15:16:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.964+0000 7fc14a600640 1 -- start start 2024-01-16T15:16:48.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc14a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc14406a3e0 0x7fc144145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc14a600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440aede0 0x7fc144152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc14a600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1440d0b60 0x7fc144147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc14a600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1440cfa60 con 0x7fc1440aede0 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc14a600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14415b8b0 con 0x7fc14406a3e0 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc14a600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc144152790 con 0x7fc1440d0b60 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc142a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440aede0 0x7fc144152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc142a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440aede0 0x7fc144152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58972/0 (socket says 172.21.15.141:58972) 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.965+0000 7fc142a00640 1 -- 172.21.15.141:0/3098605021 learned_addr learned my addr 172.21.15.141:0/3098605021 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc143400640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc14406a3e0 0x7fc144145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc142a00640 1 -- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1440d0b60 msgr2=0x7fc144147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc142a00640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1440d0b60 0x7fc144147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc142a00640 1 -- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc14406a3e0 msgr2=0x7fc144145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:48.954 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc142a00640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc14406a3e0 0x7fc144145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc142a00640 1 -- 172.21.15.141:0/3098605021 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c009400 con 0x7fc1440aede0 2024-01-16T15:16:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc142a00640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440aede0 0x7fc144152250 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0098c0 tx=0x7fc12c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc13b400640 1 -- 172.21.15.141:0/3098605021 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c03a6f0 con 0x7fc1440aede0 2024-01-16T15:16:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc13b400640 1 -- 172.21.15.141:0/3098605021 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c03a870 con 0x7fc1440aede0 2024-01-16T15:16:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc13b400640 1 -- 172.21.15.141:0/3098605021 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c03ac20 con 0x7fc1440aede0 2024-01-16T15:16:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.966+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14415aca0 con 0x7fc1440aede0 2024-01-16T15:16:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.967+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14415b260 con 0x7fc1440aede0 2024-01-16T15:16:48.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.968+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14415ba20 con 0x7fc1440aede0 2024-01-16T15:16:48.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.969+0000 7fc13b400640 1 -- 172.21.15.141:0/3098605021 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fc12c030380 con 0x7fc1440aede0 2024-01-16T15:16:48.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.970+0000 7fc13b400640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc11c06c6a0 0x7fc11c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:48.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.970+0000 7fc13b400640 1 -- 172.21.15.141:0/3098605021 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc12c0aa5b0 con 0x7fc1440aede0 2024-01-16T15:16:48.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.970+0000 7fc143400640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc11c06c6a0 0x7fc11c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:48.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.971+0000 7fc143400640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc11c06c6a0 0x7fc11c06eb60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc134001560 tx=0x7fc134009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:48.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:48.974+0000 7fc13b400640 1 -- 172.21.15.141:0/3098605021 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc12c076be0 con 0x7fc1440aede0 2024-01-16T15:16:49.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.200+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone7007", "target": ["mon-mgr", ""]}) v1 -- 0x7fc14406eb60 con 0x7fc11c06c6a0 2024-01-16T15:16:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.387+0000 7fc13b400640 1 -- 172.21.15.141:0/3098605021 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc134001930 con 0x7fc11c06c6a0 2024-01-16T15:16:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc11c06c6a0 msgr2=0x7fc11c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc11c06c6a0 0x7fc11c06eb60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc134001560 tx=0x7fc134009040 comp rx=0 tx=0).stop 2024-01-16T15:16:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440aede0 msgr2=0x7fc144152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440aede0 0x7fc144152250 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0098c0 tx=0x7fc12c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 shutdown_connections 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc11c06c6a0 0x7fc11c06eb60 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1440d0b60 0x7fc144147c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1440aede0 0x7fc144152250 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 --2- 172.21.15.141:0/3098605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc14406a3e0 0x7fc144145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.390+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 >> 172.21.15.141:0/3098605021 conn(0x7fc1440cfef0 msgr2=0x7fc1440adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.391+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 shutdown_connections 2024-01-16T15:16:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.391+0000 7fc14a600640 1 -- 172.21.15.141:0/3098605021 wait complete. 2024-01-16T15:16:49.398 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:16:49.398 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone7007 2024-01-16T15:16:49.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.621+0000 7f3a57600640 1 Processor -- start 2024-01-16T15:16:49.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.621+0000 7f3a57600640 1 -- start start 2024-01-16T15:16:49.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a57600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3a500cf9a0 0x7f3a5015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a57600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a5015e1e0 0x7f3a500702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a57600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a500b1ee0 con 0x7f3a50143ae0 2024-01-16T15:16:49.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a57600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a500b2780 con 0x7f3a500cf9a0 2024-01-16T15:16:49.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a57600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5015e990 con 0x7f3a5015e1e0 2024-01-16T15:16:49.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a55600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a5015e1e0 0x7f3a500702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a47c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3a500cf9a0 0x7f3a5015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a55600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a5015e1e0 0x7f3a500702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:38948/0 (socket says 172.21.15.141:38948) 2024-01-16T15:16:49.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.623+0000 7f3a55600640 1 -- 172.21.15.141:0/4100922348 learned_addr learned my addr 172.21.15.141:0/4100922348 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:49.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.624+0000 7f3a55600640 1 -- 172.21.15.141:0/4100922348 >> v2:172.21.15.188:3300/0 conn(0x7f3a500cf9a0 msgr2=0x7f3a5015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:49.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.624+0000 7f3a55600640 1 --2- 172.21.15.141:0/4100922348 >> v2:172.21.15.188:3300/0 conn(0x7f3a500cf9a0 0x7f3a5015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.624+0000 7f3a55600640 1 -- 172.21.15.141:0/4100922348 >> v1:172.21.15.188:6789/0 conn(0x7f3a50143ae0 legacy=0x7f3a500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.624+0000 7f3a55600640 1 -- 172.21.15.141:0/4100922348 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a500b2130 con 0x7f3a5015e1e0 2024-01-16T15:16:49.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.626+0000 7f3a55600640 1 --2- 172.21.15.141:0/4100922348 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a5015e1e0 0x7f3a500702c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009870 tx=0x7f3a4802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e3445453a8f59b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.626+0000 7f3a4fe00640 1 -- 172.21.15.141:0/4100922348 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48030840 con 0x7f3a5015e1e0 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.626+0000 7f3a4fe00640 1 -- 172.21.15.141:0/4100922348 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a480309a0 con 0x7f3a5015e1e0 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.626+0000 7f3a4fe00640 1 -- 172.21.15.141:0/4100922348 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48030d30 con 0x7f3a5015e1e0 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 -- 172.21.15.141:0/4100922348 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a5015e1e0 msgr2=0x7f3a500702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 --2- 172.21.15.141:0/4100922348 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a5015e1e0 0x7f3a500702c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009870 tx=0x7f3a4802b980 comp rx=0 tx=0).stop 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 -- 172.21.15.141:0/4100922348 shutdown_connections 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 --2- 172.21.15.141:0/4100922348 >> v2:172.21.15.188:3300/0 conn(0x7f3a500cf9a0 0x7f3a5015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 --2- 172.21.15.141:0/4100922348 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a5015e1e0 0x7f3a500702c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 -- 172.21.15.141:0/4100922348 >> 172.21.15.141:0/4100922348 conn(0x7f3a5006a800 msgr2=0x7f3a5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:49.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 -- 172.21.15.141:0/4100922348 shutdown_connections 2024-01-16T15:16:49.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.627+0000 7f3a57600640 1 -- 172.21.15.141:0/4100922348 wait complete. 2024-01-16T15:16:49.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.629+0000 7f3a57600640 1 Processor -- start 2024-01-16T15:16:49.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.629+0000 7f3a57600640 1 -- start start 2024-01-16T15:16:49.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a57600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a500cf9a0 0x7f3a500af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a57600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a50143ae0 0x7f3a501ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a57600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a5015e1e0 0x7f3a50141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a57600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5013e2e0 con 0x7f3a5015e1e0 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a57600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5013d430 con 0x7f3a500cf9a0 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a57600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5013d6d0 con 0x7f3a50143ae0 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a55600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a5015e1e0 0x7f3a50141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a55600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a5015e1e0 0x7f3a50141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58976/0 (socket says 172.21.15.141:58976) 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a55600640 1 -- 172.21.15.141:0/2174347820 learned_addr learned my addr 172.21.15.141:0/2174347820 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a54c00640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a500cf9a0 0x7f3a500af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.630+0000 7f3a47c00640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a50143ae0 0x7f3a501ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a55600640 1 -- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a50143ae0 msgr2=0x7f3a501ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a55600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a50143ae0 0x7f3a501ee000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a55600640 1 -- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a500cf9a0 msgr2=0x7f3a500af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a55600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a500cf9a0 0x7f3a500af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a55600640 1 -- 172.21.15.141:0/2174347820 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a480094d0 con 0x7f3a5015e1e0 2024-01-16T15:16:49.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a55600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a5015e1e0 0x7f3a50141690 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f3a480090b0 tx=0x7f3a480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:49.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a4d600640 1 -- 172.21.15.141:0/2174347820 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4803e070 con 0x7f3a5015e1e0 2024-01-16T15:16:49.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a4d600640 1 -- 172.21.15.141:0/2174347820 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a48039a10 con 0x7f3a5015e1e0 2024-01-16T15:16:49.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a4d600640 1 -- 172.21.15.141:0/2174347820 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48039dc0 con 0x7f3a5015e1e0 2024-01-16T15:16:49.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5013d9b0 con 0x7f3a5015e1e0 2024-01-16T15:16:49.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.631+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5013df70 con 0x7f3a5015e1e0 2024-01-16T15:16:49.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.633+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5013e490 con 0x7f3a5015e1e0 2024-01-16T15:16:49.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.633+0000 7f3a4d600640 1 -- 172.21.15.141:0/2174347820 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f3a4804b400 con 0x7f3a5015e1e0 2024-01-16T15:16:49.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.635+0000 7f3a4d600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a2806c670 0x7f3a2806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.635+0000 7f3a4d600640 1 -- 172.21.15.141:0/2174347820 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3a480ab110 con 0x7f3a5015e1e0 2024-01-16T15:16:49.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.635+0000 7f3a54c00640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a2806c670 0x7f3a2806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.636+0000 7f3a54c00640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a2806c670 0x7f3a2806eb30 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3a400012e0 tx=0x7f3a40009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:49.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.639+0000 7f3a4d600640 1 -- 172.21.15.141:0/2174347820 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a5013e490 con 0x7f3a5015e1e0 2024-01-16T15:16:49.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.866+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone7007", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a5015e670 con 0x7f3a2806c670 2024-01-16T15:16:49.868 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.882+0000 7f3a4d600640 1 -- 172.21.15.141:0/2174347820 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f3a40001a00 con 0x7f3a2806c670 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: "status": { 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: "state": "pending", 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: "source": { 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: "volume": "cephfs", 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: "subvolume": "sv1", 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: "snapshot": "ss1" 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:16:49.869 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:16:49.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a2806c670 msgr2=0x7f3a2806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a2806c670 0x7f3a2806eb30 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3a400012e0 tx=0x7f3a40009040 comp rx=0 tx=0).stop 2024-01-16T15:16:49.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a5015e1e0 msgr2=0x7f3a50141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a5015e1e0 0x7f3a50141690 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f3a480090b0 tx=0x7f3a480046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.871 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 shutdown_connections 2024-01-16T15:16:49.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a2806c670 0x7f3a2806eb30 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a5015e1e0 0x7f3a50141690 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a50143ae0 0x7f3a501ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 --2- 172.21.15.141:0/2174347820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a500cf9a0 0x7f3a500af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 >> 172.21.15.141:0/2174347820 conn(0x7f3a5006a800 msgr2=0x7f3a5006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:49.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 shutdown_connections 2024-01-16T15:16:49.872 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:49.884+0000 7f3a57600640 1 -- 172.21.15.141:0/2174347820 wait complete. 2024-01-16T15:16:49.891 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7001 2024-01-16T15:16:50.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.113+0000 7f4b1fc00640 1 Processor -- start 2024-01-16T15:16:50.100 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.113+0000 7f4b1fc00640 1 -- start start 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1fc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4b180cea00 0x7f4b1815b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1fc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b18070e60 0x7f4b18071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1fc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b18070410 con 0x7f4b1815a6b0 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1fc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1815a530 con 0x7f4b18070e60 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1fc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b180adb00 con 0x7f4b180cea00 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1dc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b18070e60 0x7f4b18071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1dc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b18070e60 0x7f4b18071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58992/0 (socket says 172.21.15.141:58992) 2024-01-16T15:16:50.103 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.115+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3153181714 learned_addr learned my addr 172.21.15.141:0/3153181714 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:50.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.116+0000 7f4b0fe00640 1 --2- 172.21.15.141:0/3153181714 >> v2:172.21.15.188:3300/0 conn(0x7f4b180cea00 0x7f4b1815b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:50.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.116+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3153181714 >> v2:172.21.15.188:3300/0 conn(0x7f4b180cea00 msgr2=0x7f4b1815b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:50.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.116+0000 7f4b1dc00640 1 --2- 172.21.15.141:0/3153181714 >> v2:172.21.15.188:3300/0 conn(0x7f4b180cea00 0x7f4b1815b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:50.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.116+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3153181714 >> v1:172.21.15.188:6789/0 conn(0x7f4b1815a6b0 legacy=0x7f4b1815aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:50.104 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.116+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3153181714 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b180adde0 con 0x7f4b18070e60 2024-01-16T15:16:50.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.117+0000 7f4b1dc00640 1 --2- 172.21.15.141:0/3153181714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b18070e60 0x7f4b18071240 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f4b180b3e40 tx=0x7f4b1002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83937a55003d2e14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:50.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.117+0000 7f4b17e00640 1 -- 172.21.15.141:0/3153181714 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1002ea90 con 0x7f4b18070e60 2024-01-16T15:16:50.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.117+0000 7f4b17e00640 1 -- 172.21.15.141:0/3153181714 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1002ebf0 con 0x7f4b18070e60 2024-01-16T15:16:50.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.117+0000 7f4b17e00640 1 -- 172.21.15.141:0/3153181714 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10009600 con 0x7f4b18070e60 2024-01-16T15:16:50.105 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3153181714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b18070e60 msgr2=0x7f4b18071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:50.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3153181714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b18070e60 0x7f4b18071240 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f4b180b3e40 tx=0x7f4b1002e0f0 comp rx=0 tx=0).stop 2024-01-16T15:16:50.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3153181714 shutdown_connections 2024-01-16T15:16:50.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3153181714 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b18070e60 0x7f4b18071240 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:50.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3153181714 >> v2:172.21.15.188:3300/0 conn(0x7f4b180cea00 0x7f4b1815b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:50.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3153181714 >> 172.21.15.141:0/3153181714 conn(0x7f4b180b0c00 msgr2=0x7f4b180b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:50.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3153181714 shutdown_connections 2024-01-16T15:16:50.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.118+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3153181714 wait complete. 2024-01-16T15:16:50.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.120+0000 7f4b1fc00640 1 Processor -- start 2024-01-16T15:16:50.107 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.120+0000 7f4b1fc00640 1 -- start start 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1fc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b18070e60 0x7f4b181528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1fc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1815a6b0 0x7f4b180aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1fc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b180af070 0x7f4b18145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1fc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b18152e30 con 0x7f4b180af070 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1fc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180af6a0 con 0x7f4b1815a6b0 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1fc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180af980 con 0x7f4b18070e60 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b180af070 0x7f4b18145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b180af070 0x7f4b18145830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58998/0 (socket says 172.21.15.141:58998) 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.121+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3082168334 learned_addr learned my addr 172.21.15.141:0/3082168334 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b0fe00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1815a6b0 0x7f4b180aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b18070e60 msgr2=0x7f4b181528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:50.110 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b1dc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b18070e60 0x7f4b181528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1815a6b0 msgr2=0x7f4b180aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b1dc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1815a6b0 0x7f4b180aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b1dc00640 1 -- 172.21.15.141:0/3082168334 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1000bd40 con 0x7f4b180af070 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b1dc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b180af070 0x7f4b18145830 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4b10038040 tx=0x7f4b10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b15600640 1 -- 172.21.15.141:0/3082168334 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1002ea90 con 0x7f4b180af070 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b15600640 1 -- 172.21.15.141:0/3082168334 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b10036820 con 0x7f4b180af070 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.122+0000 7f4b15600640 1 -- 172.21.15.141:0/3082168334 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10036bd0 con 0x7f4b180af070 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.123+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b180afc60 con 0x7f4b180af070 2024-01-16T15:16:50.111 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.123+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b180b0220 con 0x7f4b180af070 2024-01-16T15:16:50.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.124+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b180b0410 con 0x7f4b180af070 2024-01-16T15:16:50.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.125+0000 7f4b15600640 1 -- 172.21.15.141:0/3082168334 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f4b10036d30 con 0x7f4b180af070 2024-01-16T15:16:50.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.126+0000 7f4b15600640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4af006c670 0x7f4af006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:50.112 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.126+0000 7f4b15600640 1 -- 172.21.15.141:0/3082168334 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4b100abc30 con 0x7f4b180af070 2024-01-16T15:16:50.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.126+0000 7f4b1d200640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4af006c670 0x7f4af006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:50.114 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.127+0000 7f4b1d200640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4af006c670 0x7f4af006eb30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4b08000f60 tx=0x7f4b080079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:50.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.130+0000 7f4b15600640 1 -- 172.21.15.141:0/3082168334 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b10078470 con 0x7f4b180af070 2024-01-16T15:16:50.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:16:50.361+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone7001", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b180afaf0 con 0x7f4af006c670 2024-01-16T15:17:07.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:07.927+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:07.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:07.927+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:07.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:07.927+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:07.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:07.927+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:07.915 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:07.927+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:07.946 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:07.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:08.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.095+0000 7f4b15600640 1 -- 172.21.15.141:0/3082168334 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4b180afaf0 con 0x7f4af006c670 2024-01-16T15:17:08.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.098+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4af006c670 msgr2=0x7f4af006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:17:08.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.098+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4af006c670 0x7f4af006eb30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4b08000f60 tx=0x7f4b080079b0 comp rx=0 tx=0).stop 2024-01-16T15:17:08.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.098+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b180af070 msgr2=0x7f4b18145830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:17:08.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.098+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b180af070 0x7f4b18145830 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4b10038040 tx=0x7f4b10004a40 comp rx=0 tx=0).stop 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 shutdown_connections 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4af006c670 0x7f4af006eb30 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b180af070 0x7f4b18145830 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b1815a6b0 0x7f4b180aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 --2- 172.21.15.141:0/3082168334 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b18070e60 0x7f4b181528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 >> 172.21.15.141:0/3082168334 conn(0x7f4b180b0c00 msgr2=0x7f4b180cd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 shutdown_connections 2024-01-16T15:17:08.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:08.099+0000 7f4b1fc00640 1 -- 172.21.15.141:0/3082168334 wait complete. 2024-01-16T15:17:10.111 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7002 2024-01-16T15:17:10.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.347+0000 7fec06400640 1 Processor -- start 2024-01-16T15:17:10.334 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.347+0000 7fec06400640 1 -- start start 2024-01-16T15:17:10.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.348+0000 7fec06400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fec000cf9a0 0x7fec0015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:17:10.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.348+0000 7fec06400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec000b1ee0 con 0x7fec0015e1e0 2024-01-16T15:17:10.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7fec06400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec000b2780 con 0x7fec00143ae0 2024-01-16T15:17:10.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7fec06400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0015e990 con 0x7fec000cf9a0 2024-01-16T15:17:10.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7febfea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fec000cf9a0 0x7fec0015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:17:10.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7febffe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fec0015e1e0 0x7fec000702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53880/0 (socket says 172.21.15.141:53880) 2024-01-16T15:17:10.336 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7febffe00640 1 -- 172.21.15.141:0/3957321675 learned_addr learned my addr 172.21.15.141:0/3957321675 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:17:10.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7febfea00640 1 -- 172.21.15.141:0/3957321675 >> v1:172.21.15.188:6789/0 conn(0x7fec00143ae0 legacy=0x7fec000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:17:10.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7febfea00640 1 -- 172.21.15.141:0/3957321675 >> v1:172.21.15.141:6789/0 conn(0x7fec0015e1e0 legacy=0x7fec000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:17:10.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7febfea00640 1 -- 172.21.15.141:0/3957321675 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec000b2130 con 0x7fec000cf9a0 2024-01-16T15:17:10.337 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.349+0000 7febfe000640 1 -- 172.21.15.141:0/3957321675 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837404994 0 0) 0x7fec000b1ee0 con 0x7fec0015e1e0 2024-01-16T15:17:10.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7febfea00640 1 --2- 172.21.15.141:0/3957321675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000cf9a0 0x7fec0015dca0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7febe80097a0 tx=0x7febe802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa863e58fcb84cd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:17:10.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7febfe000640 1 -- 172.21.15.141:0/3957321675 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8031830 con 0x7fec000cf9a0 2024-01-16T15:17:10.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7febfe000640 1 -- 172.21.15.141:0/3957321675 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe8031990 con 0x7fec000cf9a0 2024-01-16T15:17:10.338 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7febfe000640 1 -- 172.21.15.141:0/3957321675 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8031d20 con 0x7fec000cf9a0 2024-01-16T15:17:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7fec06400640 1 -- 172.21.15.141:0/3957321675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000cf9a0 msgr2=0x7fec0015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:17:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7fec06400640 1 --2- 172.21.15.141:0/3957321675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000cf9a0 0x7fec0015dca0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7febe80097a0 tx=0x7febe802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:17:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7fec06400640 1 -- 172.21.15.141:0/3957321675 shutdown_connections 2024-01-16T15:17:10.339 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7fec06400640 1 --2- 172.21.15.141:0/3957321675 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec000cf9a0 0x7fec0015dca0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:17:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.351+0000 7fec06400640 1 -- 172.21.15.141:0/3957321675 >> 172.21.15.141:0/3957321675 conn(0x7fec0006a800 msgr2=0x7fec0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:17:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.352+0000 7fec06400640 1 -- 172.21.15.141:0/3957321675 shutdown_connections 2024-01-16T15:17:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.352+0000 7fec06400640 1 -- 172.21.15.141:0/3957321675 wait complete. 2024-01-16T15:17:10.340 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.353+0000 7fec06400640 1 Processor -- start 2024-01-16T15:17:10.341 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.353+0000 7fec06400640 1 -- start start 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.354+0000 7fec06400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000cf9a0 0x7fec000ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.354+0000 7fec06400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec00143ae0 0x7fec001f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.354+0000 7fec06400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec0015e1e0 0x7fec0015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.354+0000 7fec06400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec001f42e0 con 0x7fec00143ae0 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.354+0000 7fec06400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec001f3a40 con 0x7fec0015e1e0 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.354+0000 7fec06400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec001f3d20 con 0x7fec000cf9a0 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec00143ae0 0x7fec001f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec00143ae0 0x7fec001f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59212/0 (socket says 172.21.15.141:59212) 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 -- 172.21.15.141:0/2626202842 learned_addr learned my addr 172.21.15.141:0/2626202842 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febffe00640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec0015e1e0 0x7fec0015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febff400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000cf9a0 0x7fec000ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 -- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000cf9a0 msgr2=0x7fec000ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000cf9a0 0x7fec000ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:17:10.343 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 -- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec0015e1e0 msgr2=0x7fec0015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:17:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec0015e1e0 0x7fec0015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:17:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 -- 172.21.15.141:0/2626202842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe8009400 con 0x7fec00143ae0 2024-01-16T15:17:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febffe00640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec0015e1e0 0x7fec0015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:17:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.355+0000 7febfea00640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec00143ae0 0x7fec001f3180 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7febe80098c0 tx=0x7febe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:17:10.344 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.356+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe803ba70 con 0x7fec00143ae0 2024-01-16T15:17:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.356+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe803bbf0 con 0x7fec00143ae0 2024-01-16T15:17:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.356+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe80453f0 con 0x7fec00143ae0 2024-01-16T15:17:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.356+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec001f4000 con 0x7fec00143ae0 2024-01-16T15:17:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.356+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec001f36c0 con 0x7fec00143ae0 2024-01-16T15:17:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.357+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec000cca50 con 0x7fec00143ae0 2024-01-16T15:17:10.345 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.358+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 43) v1 ==== 91375+0+0 (secure 0 0 0) 0x7febe804e450 con 0x7fec00143ae0 2024-01-16T15:17:10.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.359+0000 7febf7400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febd806c700 0x7febd806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:17:10.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.360+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7febe80aab00 con 0x7fec00143ae0 2024-01-16T15:17:10.347 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.360+0000 7febff400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febd806c700 0x7febd806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:17:10.348 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.361+0000 7febff400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febd806c700 0x7febd806ebc0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fec000ae850 tx=0x7febf0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:17:10.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.365+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec000cca50 con 0x7fec00143ae0 2024-01-16T15:17:10.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:10.601+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone7002", "target": ["mon-mgr", ""]}) v1 -- 0x7fec0015e670 con 0x7febd806c700 2024-01-16T15:17:17.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:17.927+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:17.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:17.927+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:17.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:17.927+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:17.914 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:17.927+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:17.915 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:17.927+0000 7f0f32000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:19.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:19.311+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 44) v1 ==== 91426+0+0 (secure 0 0 0) 0x7febe80778b0 con 0x7fec00143ae0 2024-01-16T15:17:38.091 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:38.100 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:39.139 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:39.150+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:39.139 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:39.150+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:39.139 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:39.150+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:39.139 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:39.150+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:39.140 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:17:39.150+0000 7f0f33400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:17:39.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:39.381+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 8 ==== mgrmap(e 45) v1 ==== 91477+0+0 (secure 0 0 0) 0x7febe8007000 con 0x7fec00143ae0 2024-01-16T15:17:55.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:17:55.440+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mon.0 v2:172.21.15.141:3300/0 9 ==== mgrmap(e 46) v1 ==== 91426+0+0 (secure 0 0 0) 0x7febe8078180 con 0x7fec00143ae0 2024-01-16T15:18:08.335 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:08.344 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:38.505 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:38.514 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:08.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.274+0000 7febf7400640 1 -- 172.21.15.141:0/2626202842 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7febf0003980 con 0x7febd806c700 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febd806c700 msgr2=0x7febd806ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febd806c700 0x7febd806ebc0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fec000ae850 tx=0x7febf0009040 comp rx=0 tx=0).stop 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec00143ae0 msgr2=0x7fec001f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec00143ae0 0x7fec001f3180 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7febe80098c0 tx=0x7febe80046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 shutdown_connections 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7febd806c700 0x7febd806ebc0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec0015e1e0 0x7fec0015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec00143ae0 0x7fec001f3180 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 --2- 172.21.15.141:0/2626202842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fec000cf9a0 0x7fec000ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 >> 172.21.15.141:0/2626202842 conn(0x7fec0006a800 msgr2=0x7fec0006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:08.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 shutdown_connections 2024-01-16T15:19:08.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:08.278+0000 7fec06400640 1 -- 172.21.15.141:0/2626202842 wait complete. 2024-01-16T15:19:08.642 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:08.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:10.292 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7003 2024-01-16T15:19:10.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.521+0000 7f5c9e000640 1 Processor -- start 2024-01-16T15:19:10.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.521+0000 7f5c9e000640 1 -- start start 2024-01-16T15:19:10.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.523+0000 7f5c9e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 0x7f5c980b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.523+0000 7f5c9e000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c980d1c60 con 0x7f5c980cf9a0 2024-01-16T15:19:10.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.523+0000 7f5c9e000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c980b2730 con 0x7f5c9815d4b0 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.523+0000 7f5c9e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c980d1ec0 con 0x7f5c980cfde0 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.523+0000 7f5c8fc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f5c980cf9a0 0x7f5c980b31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:46456/0 (socket says 172.21.15.141:46456) 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.523+0000 7f5c8fc00640 1 -- 172.21.15.141:0/1956726610 learned_addr learned my addr 172.21.15.141:0/1956726610 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c9d600640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 0x7f5c980b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479522030 0 0) 0x7f5c980d1c60 con 0x7f5c980cf9a0 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c78003520 con 0x7f5c980cf9a0 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494543994 0 0) 0x7f5c78003520 con 0x7f5c980cf9a0 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 msgr2=0x7f5c980b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 0x7f5c980b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 >> v1:172.21.15.188:6789/0 conn(0x7f5c9815d4b0 legacy=0x7f5c980d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c980b29d0 con 0x7f5c980cf9a0 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f5c80001670 con 0x7f5c980cf9a0 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 >> v1:172.21.15.141:6789/0 conn(0x7f5c980cf9a0 legacy=0x7f5c980b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 0x7f5c780039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c78003f70 0x7f5c780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c78004d00 0x7f5c78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c980d1ec0 con 0x7f5c78004d00 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7800bac0 con 0x7f5c78003f70 2024-01-16T15:19:10.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7800bc80 con 0x7f5c980cfde0 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.524+0000 7f5c9d600640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c78004d00 0x7f5c78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.525+0000 7f5c8fc00640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c78003f70 0x7f5c780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.525+0000 7f5c9cc00640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 0x7f5c780039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.525+0000 7f5c9d600640 1 -- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 msgr2=0x7f5c780039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.525+0000 7f5c9d600640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c980cfde0 0x7f5c780039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.525+0000 7f5c9d600640 1 -- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c78003f70 msgr2=0x7f5c780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.525+0000 7f5c9d600640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c78003f70 0x7f5c780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.525+0000 7f5c9d600640 1 -- 172.21.15.141:0/1956726610 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78003520 con 0x7f5c78004d00 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.526+0000 7f5c9d600640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c78004d00 0x7f5c78005180 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009fc0 tx=0x7f5c780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c252c7d7083420f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.526+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9002cc20 con 0x7f5c78004d00 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.526+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9000f040 con 0x7f5c78004d00 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.526+0000 7f5c97e00640 1 -- 172.21.15.141:0/1956726610 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90034440 con 0x7f5c78004d00 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.527+0000 7f5c9e000640 1 -- 172.21.15.141:0/1956726610 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c78004d00 msgr2=0x7f5c78005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.527+0000 7f5c9e000640 1 --2- 172.21.15.141:0/1956726610 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c78004d00 0x7f5c78005180 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009fc0 tx=0x7f5c780057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.527+0000 7f5c9d600640 1 -- 172.21.15.141:0/1956726610 reap_dead start 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.527+0000 7f5c9e000640 1 -- 172.21.15.141:0/1956726610 shutdown_connections 2024-01-16T15:19:10.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.527+0000 7f5c9e000640 1 -- 172.21.15.141:0/1956726610 >> 172.21.15.141:0/1956726610 conn(0x7f5c980d3000 msgr2=0x7f5c980d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.527+0000 7f5c9e000640 1 -- 172.21.15.141:0/1956726610 shutdown_connections 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.527+0000 7f5c9e000640 1 -- 172.21.15.141:0/1956726610 wait complete. 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.529+0000 7f5c9e000640 1 Processor -- start 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.529+0000 7f5c9e000640 1 -- start start 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c78004d00 0x7f5c981518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c98152c70 0x7f5c9814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c9814f710 0x7f5c98152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9e000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98151e00 con 0x7f5c98152c70 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9e000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98150820 con 0x7f5c9814f710 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98150b00 con 0x7f5c78004d00 2024-01-16T15:19:10.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c8fc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c98152c70 0x7f5c9814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c8fc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c98152c70 0x7f5c9814ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55114/0 (socket says 172.21.15.141:55114) 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c8fc00640 1 -- 172.21.15.141:0/2759487672 learned_addr learned my addr 172.21.15.141:0/2759487672 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9cc00640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c78004d00 0x7f5c981518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.530+0000 7f5c9d600640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c9814f710 0x7f5c98152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c8fc00640 1 -- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c78004d00 msgr2=0x7f5c981518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c8fc00640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c78004d00 0x7f5c981518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c8fc00640 1 -- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c9814f710 msgr2=0x7f5c98152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c8fc00640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c9814f710 0x7f5c98152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c8fc00640 1 -- 172.21.15.141:0/2759487672 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c98150de0 con 0x7f5c98152c70 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c8fc00640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c98152c70 0x7f5c9814ffd0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000e7e0 tx=0x7f5c8000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c95600640 1 -- 172.21.15.141:0/2759487672 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000c8d0 con 0x7f5c98152c70 2024-01-16T15:19:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c95600640 1 -- 172.21.15.141:0/2759487672 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8000ca30 con 0x7f5c98152c70 2024-01-16T15:19:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c95600640 1 -- 172.21.15.141:0/2759487672 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000cde0 con 0x7f5c98152c70 2024-01-16T15:19:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c981510c0 con 0x7f5c98152c70 2024-01-16T15:19:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.531+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c98150510 con 0x7f5c98152c70 2024-01-16T15:19:10.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.532+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c981515e0 con 0x7f5c98152c70 2024-01-16T15:19:10.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.533+0000 7f5c95600640 1 -- 172.21.15.141:0/2759487672 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 46) v1 ==== 91426+0+0 (secure 0 0 0) 0x7f5c8001d460 con 0x7f5c98152c70 2024-01-16T15:19:10.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.535+0000 7f5c95600640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c7006c7c0 0x7f5c7006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.535+0000 7f5c95600640 1 -- 172.21.15.141:0/2759487672 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5c80088d80 con 0x7f5c98152c70 2024-01-16T15:19:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.535+0000 7f5c9cc00640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c7006c7c0 0x7f5c7006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.536+0000 7f5c9cc00640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c7006c7c0 0x7f5c7006ec80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5c88001710 tx=0x7f5c88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.540+0000 7f5c95600640 1 -- 172.21.15.141:0/2759487672 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c981515e0 con 0x7f5c98152c70 2024-01-16T15:19:10.764 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:10.772+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone7003", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c980d21a0 con 0x7f5c7006c7c0 2024-01-16T15:19:13.352 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.360+0000 7f5c95600640 1 -- 172.21.15.141:0/2759487672 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5c8800d050 con 0x7f5c7006c7c0 2024-01-16T15:19:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.363+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c7006c7c0 msgr2=0x7f5c7006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.363+0000 7f5c9e000640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c7006c7c0 0x7f5c7006ec80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5c88001710 tx=0x7f5c88009040 comp rx=0 tx=0).stop 2024-01-16T15:19:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.363+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c98152c70 msgr2=0x7f5c9814ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.363+0000 7f5c9e000640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c98152c70 0x7f5c9814ffd0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000e7e0 tx=0x7f5c8000eca0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 shutdown_connections 2024-01-16T15:19:13.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5c7006c7c0 0x7f5c7006ec80 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c9814f710 0x7f5c98152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5c98152c70 0x7f5c9814ffd0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 --2- 172.21.15.141:0/2759487672 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5c78004d00 0x7f5c981518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 >> 172.21.15.141:0/2759487672 conn(0x7f5c980d3000 msgr2=0x7f5c980cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:13.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 shutdown_connections 2024-01-16T15:19:13.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:13.364+0000 7f5c9e000640 1 -- 172.21.15.141:0/2759487672 wait complete. 2024-01-16T15:19:15.377 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7004 2024-01-16T15:19:15.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.610+0000 7f18b9800640 1 Processor -- start 2024-01-16T15:19:15.602 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.610+0000 7f18b9800640 1 -- start start 2024-01-16T15:19:15.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 0x7f18b415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b9800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f18b413e360 0x7f18b40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b9800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b406c470 con 0x7f18b415f9c0 2024-01-16T15:19:15.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b9800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b406c750 con 0x7f18b413e360 2024-01-16T15:19:15.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b406ca30 con 0x7f18b413df80 2024-01-16T15:19:15.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b3400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f18b415f9c0 0x7f18b41408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60574/0 (socket says 172.21.15.141:60574) 2024-01-16T15:19:15.604 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b3400640 1 -- 172.21.15.141:0/1478479506 learned_addr learned my addr 172.21.15.141:0/1478479506 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:19:15.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 0x7f18b415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.612+0000 7f18b2000640 1 --2- 172.21.15.141:0/1478479506 >> v2:172.21.15.188:3300/0 conn(0x7f18b413e360 0x7f18b40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522539455 0 0) 0x7f18b406c470 con 0x7f18b415f9c0 2024-01-16T15:19:15.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1894003520 con 0x7f18b415f9c0 2024-01-16T15:19:15.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 750241861 0 0) 0x7f1894003520 con 0x7f18b415f9c0 2024-01-16T15:19:15.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 msgr2=0x7f18b415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:15.607 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 0x7f18b415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 >> v2:172.21.15.188:3300/0 conn(0x7f18b413e360 msgr2=0x7f18b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 --2- 172.21.15.141:0/1478479506 >> v2:172.21.15.188:3300/0 conn(0x7f18b413e360 0x7f18b40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b406cd10 con 0x7f18b415f9c0 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f18ac0019f0 con 0x7f18b415f9c0 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 >> v1:172.21.15.141:6789/0 conn(0x7f18b415f9c0 legacy=0x7f18b41408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1894003970 0x7f1894003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18940047c0 0x7f1894004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 0x7f1894005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b406c750 con 0x7f1894003970 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b406ca30 con 0x7f18940047c0 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189400b980 con 0x7f18b413df80 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.613+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1894003970 0x7f1894003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b2000640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18940047c0 0x7f1894004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b3400640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 0x7f1894005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.608 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b2a00640 1 -- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 msgr2=0x7f1894005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b413df80 0x7f1894005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b2a00640 1 -- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18940047c0 msgr2=0x7f1894004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18940047c0 0x7f1894004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b2a00640 1 -- 172.21.15.141:0/1478479506 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1894003520 con 0x7f1894003970 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.614+0000 7f18b2000640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18940047c0 0x7f1894004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.615+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1894003970 0x7f1894003d90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009ac0 tx=0x7f18940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260fd8fd1c9eda79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.615+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a402c6b0 con 0x7f1894003970 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.615+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a402c810 con 0x7f1894003970 2024-01-16T15:19:15.609 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.615+0000 7f18b1600640 1 -- 172.21.15.141:0/1478479506 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a402cba0 con 0x7f1894003970 2024-01-16T15:19:15.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.616+0000 7f18b9800640 1 -- 172.21.15.141:0/1478479506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1894003970 msgr2=0x7f1894003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.616+0000 7f18b9800640 1 --2- 172.21.15.141:0/1478479506 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1894003970 0x7f1894003d90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009ac0 tx=0x7f18940057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.616+0000 7f18b3400640 1 -- 172.21.15.141:0/1478479506 reap_dead start 2024-01-16T15:19:15.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.616+0000 7f18b9800640 1 -- 172.21.15.141:0/1478479506 shutdown_connections 2024-01-16T15:19:15.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.616+0000 7f18b9800640 1 -- 172.21.15.141:0/1478479506 >> 172.21.15.141:0/1478479506 conn(0x7f18b406a800 msgr2=0x7f18b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:15.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.616+0000 7f18b9800640 1 -- 172.21.15.141:0/1478479506 shutdown_connections 2024-01-16T15:19:15.610 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.616+0000 7f18b9800640 1 -- 172.21.15.141:0/1478479506 wait complete. 2024-01-16T15:19:15.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.618+0000 7f18b9800640 1 Processor -- start 2024-01-16T15:19:15.611 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.618+0000 7f18b9800640 1 -- start start 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1894003970 0x7f18b413d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b4071520 0x7f18b40719c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b9800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18b4071f00 0x7f18b40b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b9800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b41f4540 con 0x7f18b4071f00 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b9800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b413dd80 con 0x7f1894003970 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b41f38c0 con 0x7f18b4071520 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18b4071f00 0x7f18b40b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18b4071f00 0x7f18b40b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45112/0 (socket says 172.21.15.141:45112) 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.619+0000 7f18b3400640 1 -- 172.21.15.141:0/1528701175 learned_addr learned my addr 172.21.15.141:0/1528701175 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:19:15.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b2000640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b4071520 0x7f18b40719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1894003970 0x7f18b413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b3400640 1 -- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b4071520 msgr2=0x7f18b40719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b3400640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b4071520 0x7f18b40719c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b3400640 1 -- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1894003970 msgr2=0x7f18b413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b3400640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1894003970 0x7f18b413d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b3400640 1 -- 172.21.15.141:0/1528701175 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b41f3ba0 con 0x7f18b4071f00 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1894003970 0x7f18b413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b3400640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18b4071f00 0x7f18b40b3410 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f18b415fda0 tx=0x7f18ac00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18aaa00640 1 -- 172.21.15.141:0/1528701175 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac004d60 con 0x7f18b4071f00 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18aaa00640 1 -- 172.21.15.141:0/1528701175 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac00bb80 con 0x7f18b4071f00 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b41f3e80 con 0x7f18b4071f00 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.620+0000 7f18aaa00640 1 -- 172.21.15.141:0/1528701175 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac007110 con 0x7f18b4071f00 2024-01-16T15:19:15.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.621+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b40b3950 con 0x7f18b4071f00 2024-01-16T15:19:15.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.622+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b40675d0 con 0x7f18b4071f00 2024-01-16T15:19:15.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.623+0000 7f18aaa00640 1 -- 172.21.15.141:0/1528701175 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 46) v1 ==== 91426+0+0 (secure 0 0 0) 0x7f18ac007270 con 0x7f18b4071f00 2024-01-16T15:19:15.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.623+0000 7f18aaa00640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f188c06c790 0x7f188c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.624+0000 7f18aaa00640 1 -- 172.21.15.141:0/1528701175 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f18ac088d10 con 0x7f18b4071f00 2024-01-16T15:19:15.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.624+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f188c06c790 0x7f188c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.625+0000 7f18b2a00640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f188c06c790 0x7f188c06ec50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f18a4036000 tx=0x7f18a4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:15.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.628+0000 7f18aaa00640 1 -- 172.21.15.141:0/1528701175 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18ac0555d0 con 0x7f18b4071f00 2024-01-16T15:19:15.853 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:15.862+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone7004", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b415ddf0 con 0x7f188c06c790 2024-01-16T15:19:18.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.384+0000 7f18aaa00640 1 -- 172.21.15.141:0/1528701175 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f18b415ddf0 con 0x7f188c06c790 2024-01-16T15:19:18.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f188c06c790 msgr2=0x7f188c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f188c06c790 0x7f188c06ec50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f18a4036000 tx=0x7f18a4030000 comp rx=0 tx=0).stop 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18b4071f00 msgr2=0x7f18b40b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18b4071f00 0x7f18b40b3410 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f18b415fda0 tx=0x7f18ac00b4a0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 shutdown_connections 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f188c06c790 0x7f188c06ec50 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18b4071f00 0x7f18b40b3410 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18b4071520 0x7f18b40719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 --2- 172.21.15.141:0/1528701175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1894003970 0x7f18b413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 >> 172.21.15.141:0/1528701175 conn(0x7f18b406a800 msgr2=0x7f18b415d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 shutdown_connections 2024-01-16T15:19:18.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:18.387+0000 7f18b9800640 1 -- 172.21.15.141:0/1528701175 wait complete. 2024-01-16T15:19:20.400 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7005 2024-01-16T15:19:20.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.628+0000 7fa196000640 1 Processor -- start 2024-01-16T15:19:20.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.628+0000 7fa196000640 1 -- start start 2024-01-16T15:19:20.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.629+0000 7fa196000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa1900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.629+0000 7fa196000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa1900cf7d0 0x7fa19015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.630+0000 7fa196000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1900cfc30 con 0x7fa19015e1e0 2024-01-16T15:19:20.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.630+0000 7fa196000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1900b1ee0 con 0x7fa1900cf7d0 2024-01-16T15:19:20.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.630+0000 7fa196000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900b2780 con 0x7fa190143ae0 2024-01-16T15:19:20.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.630+0000 7fa18f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa1900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.630+0000 7fa186800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fa1900cf7d0 0x7fa19015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.630+0000 7fa18f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa1900cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:35932/0 (socket says 172.21.15.141:35932) 2024-01-16T15:19:20.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.630+0000 7fa18f400640 1 -- 172.21.15.141:0/832313718 learned_addr learned my addr 172.21.15.141:0/832313718 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:19:20.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.631+0000 7fa18f400640 1 -- 172.21.15.141:0/832313718 >> v2:172.21.15.188:3300/0 conn(0x7fa1900cf7d0 msgr2=0x7fa19015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:20.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.631+0000 7fa18f400640 1 --2- 172.21.15.141:0/832313718 >> v2:172.21.15.188:3300/0 conn(0x7fa1900cf7d0 0x7fa19015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.631+0000 7fa18f400640 1 -- 172.21.15.141:0/832313718 >> v1:172.21.15.188:6789/0 conn(0x7fa19015e1e0 legacy=0x7fa19015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.631+0000 7fa18f400640 1 -- 172.21.15.141:0/832313718 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19015e990 con 0x7fa190143ae0 2024-01-16T15:19:20.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.633+0000 7fa18f400640 1 --2- 172.21.15.141:0/832313718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa1900cf290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa18000bb10 tx=0x7fa18002e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c598cbf2ce8b66c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.633+0000 7fa18ea00640 1 -- 172.21.15.141:0/832313718 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180036440 con 0x7fa190143ae0 2024-01-16T15:19:20.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.633+0000 7fa18ea00640 1 -- 172.21.15.141:0/832313718 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1800365a0 con 0x7fa190143ae0 2024-01-16T15:19:20.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.633+0000 7fa18ea00640 1 -- 172.21.15.141:0/832313718 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180036930 con 0x7fa190143ae0 2024-01-16T15:19:20.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 -- 172.21.15.141:0/832313718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 msgr2=0x7fa1900cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 --2- 172.21.15.141:0/832313718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa1900cf290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa18000bb10 tx=0x7fa18002e150 comp rx=0 tx=0).stop 2024-01-16T15:19:20.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 -- 172.21.15.141:0/832313718 shutdown_connections 2024-01-16T15:19:20.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 --2- 172.21.15.141:0/832313718 >> v2:172.21.15.188:3300/0 conn(0x7fa1900cf7d0 0x7fa19015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 --2- 172.21.15.141:0/832313718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa1900cf290 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 -- 172.21.15.141:0/832313718 >> 172.21.15.141:0/832313718 conn(0x7fa19006a800 msgr2=0x7fa19006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:20.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 -- 172.21.15.141:0/832313718 shutdown_connections 2024-01-16T15:19:20.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.634+0000 7fa196000640 1 -- 172.21.15.141:0/832313718 wait complete. 2024-01-16T15:19:20.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.636+0000 7fa196000640 1 Processor -- start 2024-01-16T15:19:20.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.636+0000 7fa196000640 1 -- start start 2024-01-16T15:19:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.637+0000 7fa196000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa1900cf7d0 0x7fa19015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.637+0000 7fa196000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa190159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.637+0000 7fa196000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa19015e1e0 0x7fa190159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.637+0000 7fa196000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa190137940 con 0x7fa1900cf7d0 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.637+0000 7fa196000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa190134a40 con 0x7fa19015e1e0 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.637+0000 7fa196000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa190134d20 con 0x7fa190143ae0 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa1900cf7d0 0x7fa19015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa1900cf7d0 0x7fa19015ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45116/0 (socket says 172.21.15.141:45116) 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 -- 172.21.15.141:0/1171506844 learned_addr learned my addr 172.21.15.141:0/1171506844 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18fe00640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa19015e1e0 0x7fa190159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa186800640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa190159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 -- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 msgr2=0x7fa190159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa190159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 -- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa19015e1e0 msgr2=0x7fa190159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa19015e1e0 0x7fa190159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa18f400640 1 -- 172.21.15.141:0/1171506844 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18000b770 con 0x7fa1900cf7d0 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.638+0000 7fa186800640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa190159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:20.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.639+0000 7fa18f400640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa1900cf7d0 0x7fa19015ba50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa18002e190 tx=0x7fa1800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.639+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180036440 con 0x7fa1900cf7d0 2024-01-16T15:19:20.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.639+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa180034ba0 con 0x7fa1900cf7d0 2024-01-16T15:19:20.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.639+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180041460 con 0x7fa1900cf7d0 2024-01-16T15:19:20.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.639+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa190135000 con 0x7fa1900cf7d0 2024-01-16T15:19:20.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.639+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19015bf90 con 0x7fa1900cf7d0 2024-01-16T15:19:20.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.640+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19015c120 con 0x7fa1900cf7d0 2024-01-16T15:19:20.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.641+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 46) v1 ==== 91426+0+0 (secure 0 0 0) 0x7fa180049360 con 0x7fa1900cf7d0 2024-01-16T15:19:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.642+0000 7fa187e00640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa16806c790 0x7fa16806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.642+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa1800ab140 con 0x7fa1900cf7d0 2024-01-16T15:19:20.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.643+0000 7fa186800640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa16806c790 0x7fa16806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.644+0000 7fa186800640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa16806c790 0x7fa16806ec50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa19013b0b0 tx=0x7fa178008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.647+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1800775d0 con 0x7fa1900cf7d0 2024-01-16T15:19:20.875 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:20.883+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone7005", "target": ["mon-mgr", ""]}) v1 -- 0x7fa19015e640 con 0x7fa16806c790 2024-01-16T15:19:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:24.047+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 47) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fa1800779a0 con 0x7fa1900cf7d0 2024-01-16T15:19:25.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:25.746+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mon.0 v2:172.21.15.141:3300/0 8 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fa180080120 con 0x7fa1900cf7d0 2024-01-16T15:19:38.781 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:38.790 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:39.586 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.595+0000 7fa187e00640 1 -- 172.21.15.141:0/1171506844 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa178009730 con 0x7fa16806c790 2024-01-16T15:19:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa16806c790 msgr2=0x7fa16806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa16806c790 0x7fa16806ec50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa19013b0b0 tx=0x7fa178008040 comp rx=0 tx=0).stop 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa1900cf7d0 msgr2=0x7fa19015ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa1900cf7d0 0x7fa19015ba50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa18002e190 tx=0x7fa1800046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 shutdown_connections 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa16806c790 0x7fa16806ec50 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa19015e1e0 0x7fa190159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa190143ae0 0x7fa190159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 --2- 172.21.15.141:0/1171506844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa1900cf7d0 0x7fa19015ba50 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.599+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 >> 172.21.15.141:0/1171506844 conn(0x7fa19006a800 msgr2=0x7fa19015a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.600+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 shutdown_connections 2024-01-16T15:19:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:39.600+0000 7fa196000640 1 -- 172.21.15.141:0/1171506844 wait complete. 2024-01-16T15:19:41.617 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7006 2024-01-16T15:19:41.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.845+0000 7f7d20400640 1 Processor -- start 2024-01-16T15:19:41.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.845+0000 7f7d20400640 1 -- start start 2024-01-16T15:19:41.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.846+0000 7f7d20400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7d18143ae0 0x7f7d180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:41.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.846+0000 7f7d20400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7d180cf7d0 0x7f7d1815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:41.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d20400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d180cfc30 con 0x7f7d1815e1e0 2024-01-16T15:19:41.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d20400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d180b1ee0 con 0x7f7d18143ae0 2024-01-16T15:19:41.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d20400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d180b2780 con 0x7f7d180cf7d0 2024-01-16T15:19:41.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d1e400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7d1815e1e0 0x7f7d1815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:38298/0 (socket says 172.21.15.141:38298) 2024-01-16T15:19:41.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d1e400640 1 -- 172.21.15.141:0/952132831 learned_addr learned my addr 172.21.15.141:0/952132831 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:19:41.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d1da00640 1 --2- 172.21.15.141:0/952132831 >> v2:172.21.15.141:3300/0 conn(0x7f7d18143ae0 0x7f7d180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:41.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d1d000640 1 --2- 172.21.15.141:0/952132831 >> v2:172.21.15.188:3300/0 conn(0x7f7d180cf7d0 0x7f7d1815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:41.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.847+0000 7f7d17e00640 1 -- 172.21.15.141:0/952132831 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201787669 0 0) 0x7f7d180cfc30 con 0x7f7d1815e1e0 2024-01-16T15:19:41.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.848+0000 7f7d17e00640 1 -- 172.21.15.141:0/952132831 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf8003520 con 0x7f7d1815e1e0 2024-01-16T15:19:41.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.848+0000 7f7d1da00640 1 -- 172.21.15.141:0/952132831 >> v2:172.21.15.188:3300/0 conn(0x7f7d180cf7d0 msgr2=0x7f7d1815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:41.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.848+0000 7f7d1da00640 1 --2- 172.21.15.141:0/952132831 >> v2:172.21.15.188:3300/0 conn(0x7f7d180cf7d0 0x7f7d1815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:41.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.848+0000 7f7d1da00640 1 -- 172.21.15.141:0/952132831 >> v1:172.21.15.141:6789/0 conn(0x7f7d1815e1e0 legacy=0x7f7d1815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:41.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.848+0000 7f7d1da00640 1 -- 172.21.15.141:0/952132831 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1815e990 con 0x7f7d18143ae0 2024-01-16T15:19:41.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.849+0000 7f7d1da00640 1 --2- 172.21.15.141:0/952132831 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d18143ae0 0x7f7d180cf290 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7d180cb830 tx=0x7f7d0802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc422fa5498644a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:41.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.849+0000 7f7d17e00640 1 -- 172.21.15.141:0/952132831 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0802cc80 con 0x7f7d18143ae0 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.849+0000 7f7d17e00640 1 -- 172.21.15.141:0/952132831 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0802cde0 con 0x7f7d18143ae0 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.849+0000 7f7d17e00640 1 -- 172.21.15.141:0/952132831 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08034660 con 0x7f7d18143ae0 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 -- 172.21.15.141:0/952132831 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d18143ae0 msgr2=0x7f7d180cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 --2- 172.21.15.141:0/952132831 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d18143ae0 0x7f7d180cf290 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7d180cb830 tx=0x7f7d0802bed0 comp rx=0 tx=0).stop 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 -- 172.21.15.141:0/952132831 shutdown_connections 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 --2- 172.21.15.141:0/952132831 >> v2:172.21.15.188:3300/0 conn(0x7f7d180cf7d0 0x7f7d1815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 --2- 172.21.15.141:0/952132831 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d18143ae0 0x7f7d180cf290 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 -- 172.21.15.141:0/952132831 >> 172.21.15.141:0/952132831 conn(0x7f7d1806a800 msgr2=0x7f7d1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 -- 172.21.15.141:0/952132831 shutdown_connections 2024-01-16T15:19:41.842 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.850+0000 7f7d20400640 1 -- 172.21.15.141:0/952132831 wait complete. 2024-01-16T15:19:41.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.852+0000 7f7d20400640 1 Processor -- start 2024-01-16T15:19:41.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.852+0000 7f7d20400640 1 -- start start 2024-01-16T15:19:41.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d20400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d180cf7d0 0x7f7d18165340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:41.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d20400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18143ae0 0x7f7d1815a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:41.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d20400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d1815e1e0 0x7f7d180afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:41.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d20400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d18166930 con 0x7f7d180cf7d0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d20400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d18165fe0 con 0x7f7d18143ae0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d20400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d18166280 con 0x7f7d1815e1e0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d180cf7d0 0x7f7d18165340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d180cf7d0 0x7f7d18165340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50974/0 (socket says 172.21.15.141:50974) 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 -- 172.21.15.141:0/4260974566 learned_addr learned my addr 172.21.15.141:0/4260974566 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 -- 172.21.15.141:0/4260974566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d1815e1e0 msgr2=0x7f7d180afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 --2- 172.21.15.141:0/4260974566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7d1815e1e0 0x7f7d180afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 -- 172.21.15.141:0/4260974566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18143ae0 msgr2=0x7f7d1815a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 --2- 172.21.15.141:0/4260974566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18143ae0 0x7f7d1815a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.853+0000 7f7d1da00640 1 -- 172.21.15.141:0/4260974566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d08009a70 con 0x7f7d180cf7d0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.854+0000 7f7d1da00640 1 --2- 172.21.15.141:0/4260974566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d180cf7d0 0x7f7d18165340 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f7d08037040 tx=0x7f7d08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.854+0000 7f7d15600640 1 -- 172.21.15.141:0/4260974566 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08040650 con 0x7f7d180cf7d0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.854+0000 7f7d15600640 1 -- 172.21.15.141:0/4260974566 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d080407b0 con 0x7f7d180cf7d0 2024-01-16T15:19:41.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.854+0000 7f7d15600640 1 -- 172.21.15.141:0/4260974566 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08040b60 con 0x7f7d180cf7d0 2024-01-16T15:19:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.854+0000 7f7d20400640 1 -- 172.21.15.141:0/4260974566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d18166560 con 0x7f7d180cf7d0 2024-01-16T15:19:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.854+0000 7f7d20400640 1 -- 172.21.15.141:0/4260974566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d18166e20 con 0x7f7d180cf7d0 2024-01-16T15:19:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.855+0000 7f7d20400640 1 -- 172.21.15.141:0/4260974566 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d18167040 con 0x7f7d180cf7d0 2024-01-16T15:19:41.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.856+0000 7f7d15600640 1 -- 172.21.15.141:0/4260974566 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f7d08040cc0 con 0x7f7d180cf7d0 2024-01-16T15:19:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.857+0000 7f7d15600640 1 --2- 172.21.15.141:0/4260974566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cf006c670 0x7f7cf006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.858+0000 7f7d15600640 1 -- 172.21.15.141:0/4260974566 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7d080ab8f0 con 0x7f7d180cf7d0 2024-01-16T15:19:41.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.858+0000 7f7d1d000640 1 --2- 172.21.15.141:0/4260974566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cf006c670 0x7f7cf006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:41.850 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.859+0000 7f7d1d000640 1 --2- 172.21.15.141:0/4260974566 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7cf006c670 0x7f7cf006eb30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7d00006fd0 tx=0x7f7d00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:41.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:41.862+0000 7f7d15600640 1 -- 172.21.15.141:0/4260974566 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d08077f20 con 0x7f7d180cf7d0 2024-01-16T15:19:42.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:19:42.092+0000 7f7d20400640 1 -- 172.21.15.141:0/4260974566 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone7006", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d1815e670 con 0x7f7cf006c670 2024-01-16T15:20:08.948 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:08.958 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:39.119 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:39.128 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:09.267 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:09.276 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:39.440 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:39.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-16T15:21:41.739 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:21:41.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.950+0000 7fbd6ec00640 1 Processor -- start 2024-01-16T15:21:41.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.950+0000 7fbd6ec00640 1 -- start start 2024-01-16T15:21:41.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.952+0000 7fbd6ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 0x7fbd6806add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.952+0000 7fbd6ec00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6806d2a0 con 0x7fbd6815acc0 2024-01-16T15:21:41.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.952+0000 7fbd6ec00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6815eb60 con 0x7fbd6806c990 2024-01-16T15:21:41.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.952+0000 7fbd6ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6815dfe0 con 0x7fbd6806ce10 2024-01-16T15:21:41.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.952+0000 7fbd67e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbd6815acc0 0x7fbd6815b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33342/0 (socket says 172.21.15.141:33342) 2024-01-16T15:21:41.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.952+0000 7fbd67e00640 1 -- 172.21.15.141:0/608942718 learned_addr learned my addr 172.21.15.141:0/608942718 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:21:41.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 0x7fbd6806add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892428217 0 0) 0x7fbd6806d2a0 con 0x7fbd6815acc0 2024-01-16T15:21:41.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd48005730 con 0x7fbd6815acc0 2024-01-16T15:21:41.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057369319 0 0) 0x7fbd48005730 con 0x7fbd6815acc0 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 msgr2=0x7fbd6806add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 0x7fbd6806add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 >> v1:172.21.15.188:6789/0 conn(0x7fbd6806c990 legacy=0x7fbd6806a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6815e2c0 con 0x7fbd6815acc0 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fbd580019f0 con 0x7fbd6815acc0 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 >> v1:172.21.15.141:6789/0 conn(0x7fbd6815acc0 legacy=0x7fbd6815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48005bb0 0x7fbd48005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 0x7fbd480069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd48006f10 0x7fbd48007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6815dfe0 con 0x7fbd48006f10 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4800dc70 con 0x7fbd48005bb0 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.953+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4800de30 con 0x7fbd6806ce10 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd48006f10 0x7fbd48007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd67400640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 0x7fbd480069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd67e00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48005bb0 0x7fbd48005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd6cc00640 1 -- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 msgr2=0x7fbd480069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd6806ce10 0x7fbd480069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd6cc00640 1 -- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48005bb0 msgr2=0x7fbd48005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48005bb0 0x7fbd48005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd6cc00640 1 -- 172.21.15.141:0/608942718 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd48005730 con 0x7fbd48006f10 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.954+0000 7fbd67e00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48005bb0 0x7fbd48005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.955+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd48006f10 0x7fbd48007390 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009a90 tx=0x7fbd48007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8eee82db5e1722f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.955+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002c680 con 0x7fbd48006f10 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.955+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6002c7e0 con 0x7fbd48006f10 2024-01-16T15:21:41.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.955+0000 7fbd66a00640 1 -- 172.21.15.141:0/608942718 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002cb70 con 0x7fbd48006f10 2024-01-16T15:21:41.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.956+0000 7fbd6ec00640 1 -- 172.21.15.141:0/608942718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd48006f10 msgr2=0x7fbd48007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:41.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.956+0000 7fbd6ec00640 1 --2- 172.21.15.141:0/608942718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd48006f10 0x7fbd48007390 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009a90 tx=0x7fbd48007a00 comp rx=0 tx=0).stop 2024-01-16T15:21:41.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.956+0000 7fbd6cc00640 1 -- 172.21.15.141:0/608942718 reap_dead start 2024-01-16T15:21:41.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.956+0000 7fbd6ec00640 1 -- 172.21.15.141:0/608942718 shutdown_connections 2024-01-16T15:21:41.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.956+0000 7fbd6ec00640 1 -- 172.21.15.141:0/608942718 >> 172.21.15.141:0/608942718 conn(0x7fbd680cf700 msgr2=0x7fbd680cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.956+0000 7fbd6ec00640 1 -- 172.21.15.141:0/608942718 shutdown_connections 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.956+0000 7fbd6ec00640 1 -- 172.21.15.141:0/608942718 wait complete. 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.958+0000 7fbd6ec00640 1 Processor -- start 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.958+0000 7fbd6ec00640 1 -- start start 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6ec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48006f10 0x7fbd681532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6ec00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b32c0 0x7fbd680b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6ec00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6813f2d0 0x7fbd6813f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6ec00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6813f070 con 0x7fbd6813f2d0 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6ec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd680af3c0 con 0x7fbd48006f10 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6ec00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6813fc90 con 0x7fbd680b32c0 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6813f2d0 0x7fbd6813f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6813f2d0 0x7fbd6813f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45406/0 (socket says 172.21.15.141:45406) 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6cc00640 1 -- 172.21.15.141:0/2836934814 learned_addr learned my addr 172.21.15.141:0/2836934814 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd67e00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48006f10 0x7fbd681532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd67400640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b32c0 0x7fbd680b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6cc00640 1 -- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b32c0 msgr2=0x7fbd680b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.959+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b32c0 0x7fbd680b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd6cc00640 1 -- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48006f10 msgr2=0x7fbd681532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48006f10 0x7fbd681532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd6cc00640 1 -- 172.21.15.141:0/2836934814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6813eab0 con 0x7fbd6813f2d0 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd67e00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48006f10 0x7fbd681532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd6cc00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6813f2d0 0x7fbd6813f750 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009a90 tx=0x7fbd60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2836934814 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002cd60 con 0x7fbd6813f2d0 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2836934814 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd60003610 con 0x7fbd6813f2d0 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6813ed90 con 0x7fbd6813f2d0 2024-01-16T15:21:41.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2836934814 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd600039c0 con 0x7fbd6813f2d0 2024-01-16T15:21:41.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.960+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd680af7e0 con 0x7fbd6813f2d0 2024-01-16T15:21:41.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.961+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd680aee40 con 0x7fbd6813f2d0 2024-01-16T15:21:41.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.963+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2836934814 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fbd60003b20 con 0x7fbd6813f2d0 2024-01-16T15:21:41.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.963+0000 7fbd5fe00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c6a0 0x7fbd4006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:41.958 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.964+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2836934814 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd600aa860 con 0x7fbd6813f2d0 2024-01-16T15:21:41.959 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.964+0000 7fbd67e00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c6a0 0x7fbd4006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:41.960 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.965+0000 7fbd67e00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c6a0 0x7fbd4006eb60 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbd58007c40 tx=0x7fbd58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:41.963 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:41.969+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2836934814 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd680aee40 con 0x7fbd6813f2d0 2024-01-16T15:21:42.178 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.183+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd680af500 con 0x7fbd4006c6a0 2024-01-16T15:21:42.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.187+0000 7fbd5fe00640 1 -- 172.21.15.141:0/2836934814 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbd680af500 con 0x7fbd4006c6a0 2024-01-16T15:21:42.181 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:21:42.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.182 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "cephfs" 2024-01-16T15:21:42.182 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:21:42.182 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:21:42.184 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c6a0 msgr2=0x7fbd4006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c6a0 0x7fbd4006eb60 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbd58007c40 tx=0x7fbd58009040 comp rx=0 tx=0).stop 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6813f2d0 msgr2=0x7fbd6813f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6813f2d0 0x7fbd6813f750 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009a90 tx=0x7fbd60004730 comp rx=0 tx=0).stop 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 shutdown_connections 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbd4006c6a0 0x7fbd4006eb60 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd6813f2d0 0x7fbd6813f750 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbd680b32c0 0x7fbd680b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 --2- 172.21.15.141:0/2836934814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd48006f10 0x7fbd681532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 >> 172.21.15.141:0/2836934814 conn(0x7fbd680cf700 msgr2=0x7fbd680cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 shutdown_connections 2024-01-16T15:21:42.185 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.189+0000 7fbd6ec00640 1 -- 172.21.15.141:0/2836934814 wait complete. 2024-01-16T15:21:42.205 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:21:42.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.420+0000 7ff8d2000640 1 Processor -- start 2024-01-16T15:21:42.415 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.420+0000 7ff8d2000640 1 -- start start 2024-01-16T15:21:42.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.421+0000 7ff8d2000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0ae5a0 0x7ff8cc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.421+0000 7ff8d2000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff8cc0b3ae0 0x7ff8cc06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.421+0000 7ff8d2000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc070c40 con 0x7ff8cc06d120 2024-01-16T15:21:42.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.422+0000 7ff8d2000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8cc071450 con 0x7ff8cc0b3ae0 2024-01-16T15:21:42.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.422+0000 7ff8d2000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc06c470 con 0x7ff8cc0ae5a0 2024-01-16T15:21:42.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.422+0000 7ff8caa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff8cc0b3ae0 0x7ff8cc06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.422+0000 7ff8cb400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0ae5a0 0x7ff8cc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.422+0000 7ff8cb400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0ae5a0 0x7ff8cc15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:59870/0 (socket says 172.21.15.141:59870) 2024-01-16T15:21:42.417 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.422+0000 7ff8cb400640 1 -- 172.21.15.141:0/582930315 learned_addr learned my addr 172.21.15.141:0/582930315 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:42.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.423+0000 7ff8cb400640 1 -- 172.21.15.141:0/582930315 >> v2:172.21.15.188:3300/0 conn(0x7ff8cc0b3ae0 msgr2=0x7ff8cc06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:42.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.423+0000 7ff8cb400640 1 --2- 172.21.15.141:0/582930315 >> v2:172.21.15.188:3300/0 conn(0x7ff8cc0b3ae0 0x7ff8cc06cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.423+0000 7ff8cb400640 1 -- 172.21.15.141:0/582930315 >> v1:172.21.15.188:6789/0 conn(0x7ff8cc06d120 legacy=0x7ff8cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.423+0000 7ff8cb400640 1 -- 172.21.15.141:0/582930315 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc06c750 con 0x7ff8cc0ae5a0 2024-01-16T15:21:42.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.425+0000 7ff8cb400640 1 --2- 172.21.15.141:0/582930315 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0ae5a0 0x7ff8cc15e430 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009870 tx=0x7ff8bc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e791532c4daa13f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.425+0000 7ff8ca000640 1 -- 172.21.15.141:0/582930315 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc030840 con 0x7ff8cc0ae5a0 2024-01-16T15:21:42.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.426+0000 7ff8ca000640 1 -- 172.21.15.141:0/582930315 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc0309a0 con 0x7ff8cc0ae5a0 2024-01-16T15:21:42.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.426+0000 7ff8ca000640 1 -- 172.21.15.141:0/582930315 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc030d30 con 0x7ff8cc0ae5a0 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.426+0000 7ff8d2000640 1 -- 172.21.15.141:0/582930315 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0ae5a0 msgr2=0x7ff8cc15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.426+0000 7ff8d2000640 1 --2- 172.21.15.141:0/582930315 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0ae5a0 0x7ff8cc15e430 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009870 tx=0x7ff8bc02b980 comp rx=0 tx=0).stop 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.427+0000 7ff8d2000640 1 -- 172.21.15.141:0/582930315 shutdown_connections 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.427+0000 7ff8d2000640 1 --2- 172.21.15.141:0/582930315 >> v2:172.21.15.188:3300/0 conn(0x7ff8cc0b3ae0 0x7ff8cc06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.427+0000 7ff8d2000640 1 --2- 172.21.15.141:0/582930315 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0ae5a0 0x7ff8cc15e430 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.427+0000 7ff8d2000640 1 -- 172.21.15.141:0/582930315 >> 172.21.15.141:0/582930315 conn(0x7ff8cc06a800 msgr2=0x7ff8cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.427+0000 7ff8d2000640 1 -- 172.21.15.141:0/582930315 shutdown_connections 2024-01-16T15:21:42.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.427+0000 7ff8d2000640 1 -- 172.21.15.141:0/582930315 wait complete. 2024-01-16T15:21:42.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.429+0000 7ff8d2000640 1 Processor -- start 2024-01-16T15:21:42.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.429+0000 7ff8d2000640 1 -- start start 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.430+0000 7ff8d2000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff8cc06d120 0x7ff8cc0ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.430+0000 7ff8d2000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8cc0ae5a0 0x7ff8cc147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.430+0000 7ff8d2000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0b3ae0 0x7ff8cc0cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.430+0000 7ff8d2000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc148dc0 con 0x7ff8cc06d120 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.430+0000 7ff8d2000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1481b0 con 0x7ff8cc0ae5a0 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.430+0000 7ff8d2000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc148490 con 0x7ff8cc0b3ae0 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.430+0000 7ff8cb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff8cc06d120 0x7ff8cc0ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff8cc06d120 0x7ff8cc0ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45420/0 (socket says 172.21.15.141:45420) 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 -- 172.21.15.141:0/1942082644 learned_addr learned my addr 172.21.15.141:0/1942082644 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:42.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8caa00640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8cc0ae5a0 0x7ff8cc147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cbe00640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0b3ae0 0x7ff8cc0cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 -- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0b3ae0 msgr2=0x7ff8cc0cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0b3ae0 0x7ff8cc0cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 -- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8cc0ae5a0 msgr2=0x7ff8cc147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8cc0ae5a0 0x7ff8cc147420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 -- 172.21.15.141:0/1942082644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc0094d0 con 0x7ff8cc06d120 2024-01-16T15:21:42.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.431+0000 7ff8cb400640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff8cc06d120 0x7ff8cc0ce300 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009870 tx=0x7ff8bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.432+0000 7ff8c3400640 1 -- 172.21.15.141:0/1942082644 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc039c90 con 0x7ff8cc06d120 2024-01-16T15:21:42.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.432+0000 7ff8c3400640 1 -- 172.21.15.141:0/1942082644 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc039df0 con 0x7ff8cc06d120 2024-01-16T15:21:42.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.432+0000 7ff8c3400640 1 -- 172.21.15.141:0/1942082644 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc043640 con 0x7ff8cc06d120 2024-01-16T15:21:42.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.432+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc148770 con 0x7ff8cc06d120 2024-01-16T15:21:42.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.432+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc0ce840 con 0x7ff8cc06d120 2024-01-16T15:21:42.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.433+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc072130 con 0x7ff8cc06d120 2024-01-16T15:21:42.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.434+0000 7ff8c3400640 1 -- 172.21.15.141:0/1942082644 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7ff8bc0437a0 con 0x7ff8cc06d120 2024-01-16T15:21:42.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.435+0000 7ff8c3400640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8a406c670 0x7ff8a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.435+0000 7ff8c3400640 1 -- 172.21.15.141:0/1942082644 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff8bc03e070 con 0x7ff8cc06d120 2024-01-16T15:21:42.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.435+0000 7ff8caa00640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8a406c670 0x7ff8a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.431 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.437+0000 7ff8caa00640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8a406c670 0x7ff8a406eb30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0cb1f0 tx=0x7ff8b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.434 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.439+0000 7ff8c3400640 1 -- 172.21.15.141:0/1942082644 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8bc078130 con 0x7ff8cc06d120 2024-01-16T15:21:42.647 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.652+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff8cc070550 con 0x7ff8a406c670 2024-01-16T15:21:42.657 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:42.664+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:42.657 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:42.664+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:42.657 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:42.664+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:42.658 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:42.664+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:42.658 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:42.664+0000 7f0f2bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:42.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.700+0000 7ff8c3400640 1 -- 172.21.15.141:0/1942082644 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+374 (secure 0 0 0) 0x7ff8b40095e0 con 0x7ff8a406c670 2024-01-16T15:21:42.695 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:42.695 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:21:42.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.695 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone7001" 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "sv1" 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone7004" 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone7006" 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:21:42.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone7002" 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone7005" 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone7003" 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "subvol_clone7007" 2024-01-16T15:21:42.697 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:21:42.698 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.702+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8a406c670 msgr2=0x7ff8a406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.702+0000 7ff8d2000640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8a406c670 0x7ff8a406eb30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0cb1f0 tx=0x7ff8b4008040 comp rx=0 tx=0).stop 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.702+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff8cc06d120 msgr2=0x7ff8cc0ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.702+0000 7ff8d2000640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff8cc06d120 0x7ff8cc0ce300 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009870 tx=0x7ff8bc004730 comp rx=0 tx=0).stop 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 shutdown_connections 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff8a406c670 0x7ff8a406eb30 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff8cc0b3ae0 0x7ff8cc0cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8cc0ae5a0 0x7ff8cc147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 --2- 172.21.15.141:0/1942082644 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff8cc06d120 0x7ff8cc0ce300 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.699 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 >> 172.21.15.141:0/1942082644 conn(0x7ff8cc06a800 msgr2=0x7ff8cc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:42.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 shutdown_connections 2024-01-16T15:21:42.700 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.703+0000 7ff8d2000640 1 -- 172.21.15.141:0/1942082644 wait complete. 2024-01-16T15:21:42.716 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone7001 2024-01-16T15:21:42.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.929+0000 7f7faea00640 1 Processor -- start 2024-01-16T15:21:42.924 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.929+0000 7f7faea00640 1 -- start start 2024-01-16T15:21:42.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.931+0000 7f7faea00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7fa813df80 0x7f7fa815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.931+0000 7f7faea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa80cf9a0 0x7f7fa80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.931+0000 7f7faea00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa806c470 con 0x7f7fa813e360 2024-01-16T15:21:42.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.931+0000 7f7faea00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa806c750 con 0x7f7fa813df80 2024-01-16T15:21:42.926 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.931+0000 7f7faea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa806ca30 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.931+0000 7f7faca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa80cf9a0 0x7f7fa80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.932+0000 7f7faca00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa80cf9a0 0x7f7fa80cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:59882/0 (socket says 172.21.15.141:59882) 2024-01-16T15:21:42.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.932+0000 7f7faca00640 1 -- 172.21.15.141:0/4195918741 learned_addr learned my addr 172.21.15.141:0/4195918741 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:42.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.932+0000 7f7fa7e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f7fa813df80 0x7f7fa815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.927 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.932+0000 7f7fa7400640 1 --1- 172.21.15.141:0/4195918741 >> v1:172.21.15.188:6789/0 conn(0x7f7fa813e360 0x7f7fa80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:36796/0 (socket says 172.21.15.141:36796) 2024-01-16T15:21:42.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.932+0000 7f7faca00640 1 -- 172.21.15.141:0/4195918741 >> v2:172.21.15.188:3300/0 conn(0x7f7fa813df80 msgr2=0x7f7fa815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:42.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.932+0000 7f7faca00640 1 --2- 172.21.15.141:0/4195918741 >> v2:172.21.15.188:3300/0 conn(0x7f7fa813df80 0x7f7fa815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.933+0000 7f7faca00640 1 -- 172.21.15.141:0/4195918741 >> v1:172.21.15.188:6789/0 conn(0x7f7fa813e360 legacy=0x7f7fa80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.933+0000 7f7faca00640 1 -- 172.21.15.141:0/4195918741 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa806cd10 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.933+0000 7f7fa6a00640 1 -- 172.21.15.141:0/4195918741 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910336107 0 0) 0x7f7fa806c470 con 0x7f7fa813e360 2024-01-16T15:21:42.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.933+0000 7f7fa7e00640 1 --2- 172.21.15.141:0/4195918741 >> v2:172.21.15.188:3300/0 conn(0x7f7fa813df80 0x7f7fa815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:21:42.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.934+0000 7f7faca00640 1 --2- 172.21.15.141:0/4195918741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa80cf9a0 0x7f7fa80cd840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009e50 tx=0x7f7fa002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30bab4e9a08c7d47 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.935+0000 7f7fa6a00640 1 -- 172.21.15.141:0/4195918741 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa002cb00 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.935+0000 7f7fa6a00640 1 -- 172.21.15.141:0/4195918741 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa002cc60 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.935+0000 7f7fa6a00640 1 -- 172.21.15.141:0/4195918741 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa00343f0 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 -- 172.21.15.141:0/4195918741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa80cf9a0 msgr2=0x7f7fa80cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 --2- 172.21.15.141:0/4195918741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa80cf9a0 0x7f7fa80cd840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009e50 tx=0x7f7fa002bf60 comp rx=0 tx=0).stop 2024-01-16T15:21:42.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 -- 172.21.15.141:0/4195918741 shutdown_connections 2024-01-16T15:21:42.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 --2- 172.21.15.141:0/4195918741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa80cf9a0 0x7f7fa80cd840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 --2- 172.21.15.141:0/4195918741 >> v2:172.21.15.188:3300/0 conn(0x7f7fa813df80 0x7f7fa815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 -- 172.21.15.141:0/4195918741 >> 172.21.15.141:0/4195918741 conn(0x7f7fa806a800 msgr2=0x7f7fa806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:42.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 -- 172.21.15.141:0/4195918741 shutdown_connections 2024-01-16T15:21:42.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.936+0000 7f7faea00640 1 -- 172.21.15.141:0/4195918741 wait complete. 2024-01-16T15:21:42.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.938+0000 7f7faea00640 1 Processor -- start 2024-01-16T15:21:42.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.938+0000 7f7faea00640 1 -- start start 2024-01-16T15:21:42.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.939+0000 7f7faea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fa80cf9a0 0x7f7fa8150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.939+0000 7f7faea00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa813df80 0x7f7fa8159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.939+0000 7f7faea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fa813e360 0x7f7fa815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.939+0000 7f7faea00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa8072380 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.939+0000 7f7faea00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa814e800 con 0x7f7fa813e360 2024-01-16T15:21:42.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.939+0000 7f7faea00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa8150740 con 0x7f7fa813df80 2024-01-16T15:21:42.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.939+0000 7f7fa7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fa80cf9a0 0x7f7fa8150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fa80cf9a0 0x7f7fa8150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45436/0 (socket says 172.21.15.141:45436) 2024-01-16T15:21:42.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 -- 172.21.15.141:0/279643842 learned_addr learned my addr 172.21.15.141:0/279643842 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:42.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7400640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa813df80 0x7f7fa8159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 -- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa813df80 msgr2=0x7f7fa8159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa813df80 0x7f7fa8159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 -- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fa813e360 msgr2=0x7f7fa815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:42.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fa813e360 0x7f7fa815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:42.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 -- 172.21.15.141:0/279643842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f98009400 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7fa7e00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fa80cf9a0 0x7f7fa8150200 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800ebe0 tx=0x7f7f9800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7f9fe00640 1 -- 172.21.15.141:0/279643842 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98009760 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.940+0000 7f7f9fe00640 1 -- 172.21.15.141:0/279643842 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9800cc30 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.941+0000 7f7f9fe00640 1 -- 172.21.15.141:0/279643842 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9800eee0 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.941+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa0009b30 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.941+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa812f510 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.942+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa812f370 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.943+0000 7f7f9fe00640 1 -- 172.21.15.141:0/279643842 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f7f98013470 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.943+0000 7f7f9fe00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f8006c670 0x7f7f8006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:42.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.944+0000 7f7f9fe00640 1 -- 172.21.15.141:0/279643842 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f98088fb0 con 0x7f7fa80cf9a0 2024-01-16T15:21:42.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.944+0000 7f7fa7400640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f8006c670 0x7f7f8006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:42.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.945+0000 7f7fa7400640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f8006c670 0x7f7f8006eb30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7f90007fd0 tx=0x7f7f90009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:42.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:42.948+0000 7f7f9fe00640 1 -- 172.21.15.141:0/279643842 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fa812f370 con 0x7f7fa80cf9a0 2024-01-16T15:21:43.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.151+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone7001", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fa815f9c0 con 0x7f7f8006c670 2024-01-16T15:21:43.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.205+0000 7f7f9fe00640 1 -- 172.21.15.141:0/279643842 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7f90001350 con 0x7f7f8006c670 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f8006c670 msgr2=0x7f7f8006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f8006c670 0x7f7f8006eb30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7f90007fd0 tx=0x7f7f90009040 comp rx=0 tx=0).stop 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fa80cf9a0 msgr2=0x7f7fa8150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fa80cf9a0 0x7f7fa8150200 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800ebe0 tx=0x7f7f9800c520 comp rx=0 tx=0).stop 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 shutdown_connections 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f8006c670 0x7f7f8006eb30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fa813e360 0x7f7fa815a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7fa813df80 0x7f7fa8159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 --2- 172.21.15.141:0/279643842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7fa80cf9a0 0x7f7fa8150200 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 >> 172.21.15.141:0/279643842 conn(0x7f7fa806a800 msgr2=0x7f7fa8154df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.206+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 shutdown_connections 2024-01-16T15:21:43.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.207+0000 7f7faea00640 1 -- 172.21.15.141:0/279643842 wait complete. 2024-01-16T15:21:43.206 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.213+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.206 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.213+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.206 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.213+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.206 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.213+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.206 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.213+0000 7f0f2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.223 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:21:43.232 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.238+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.232 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.238+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.232 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.238+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.232 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.238+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.232 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:43.238+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:43.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.422+0000 7f9535200640 1 Processor -- start 2024-01-16T15:21:43.416 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.422+0000 7f9535200640 1 -- start start 2024-01-16T15:21:43.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f9535200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f95300b3ae0 0x7f953015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f9535200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9530070c40 con 0x7f95300cf290 2024-01-16T15:21:43.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f9535200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f953006c470 con 0x7f95300ae5a0 2024-01-16T15:21:43.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f9535200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9530071450 con 0x7f95300b3ae0 2024-01-16T15:21:43.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f95300cf290 0x7f95300b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52308/0 (socket says 172.21.15.141:52308) 2024-01-16T15:21:43.418 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952e800640 1 -- 172.21.15.141:0/183675510 learned_addr learned my addr 172.21.15.141:0/183675510 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f9527e00640 1 --2- 172.21.15.141:0/183675510 >> v2:172.21.15.188:3300/0 conn(0x7f95300b3ae0 0x7f953015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10442552 0 0) 0x7f9530070c40 con 0x7f95300cf290 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510003520 con 0x7f95300cf290 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1638989885 0 0) 0x7f9510003520 con 0x7f95300cf290 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 >> v2:172.21.15.188:3300/0 conn(0x7f95300b3ae0 msgr2=0x7f953015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 --2- 172.21.15.141:0/183675510 >> v2:172.21.15.188:3300/0 conn(0x7f95300b3ae0 0x7f953015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 >> v1:172.21.15.188:6789/0 conn(0x7f95300ae5a0 legacy=0x7f953006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953006c750 con 0x7f95300cf290 2024-01-16T15:21:43.420 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f95280019f0 con 0x7f95300cf290 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 >> v1:172.21.15.141:6789/0 conn(0x7f95300cf290 legacy=0x7f95300b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510003970 0x7f9510003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005180 0x7f951000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530071450 con 0x7f9510005180 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951000c030 con 0x7f95100047c0 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951000c1f0 con 0x7f9510003970 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.424+0000 7f952e800640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005180 0x7f951000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f9527e00640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f952de00640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510003970 0x7f9510003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f952e800640 1 -- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510003970 msgr2=0x7f9510003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f952e800640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510003970 0x7f9510003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f952e800640 1 -- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 msgr2=0x7f9510004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f952e800640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.421 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f952e800640 1 -- 172.21.15.141:0/183675510 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9510003520 con 0x7f9510005180 2024-01-16T15:21:43.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f9527e00640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95100047c0 0x7f9510004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:43.422 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.425+0000 7f952de00640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510003970 0x7f9510003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.426+0000 7f952e800640 1 -- 172.21.15.141:0/183675510 reap_dead start 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.426+0000 7f952e800640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005180 0x7f951000b980 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f952800a800 tx=0x7f95100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb121cb8d80f432 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.426+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528034630 con 0x7f9510005180 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.426+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952802f070 con 0x7f9510005180 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.426+0000 7f952d400640 1 -- 172.21.15.141:0/183675510 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528034a10 con 0x7f9510005180 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.427+0000 7f9535200640 1 -- 172.21.15.141:0/183675510 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005180 msgr2=0x7f951000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.427+0000 7f9535200640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005180 0x7f951000b980 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f952800a800 tx=0x7f95100057f0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.427+0000 7f9535200640 1 -- 172.21.15.141:0/183675510 shutdown_connections 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.427+0000 7f9535200640 1 --2- 172.21.15.141:0/183675510 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9510005180 0x7f951000b980 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.423 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.427+0000 7f9535200640 1 -- 172.21.15.141:0/183675510 >> 172.21.15.141:0/183675510 conn(0x7f953006a800 msgr2=0x7f953006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:43.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.427+0000 7f9535200640 1 -- 172.21.15.141:0/183675510 shutdown_connections 2024-01-16T15:21:43.424 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.427+0000 7f9535200640 1 -- 172.21.15.141:0/183675510 wait complete. 2024-01-16T15:21:43.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.429+0000 7f9535200640 1 Processor -- start 2024-01-16T15:21:43.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.429+0000 7f9535200640 1 -- start start 2024-01-16T15:21:43.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.429+0000 7f9535200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510005180 0x7f95300cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.429+0000 7f9535200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95300ce820 0x7f9530159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9535200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f6910 0x7f95301f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.425 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9535200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95300cc960 con 0x7f95300ce820 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9535200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95300cb8a0 con 0x7f95301f6910 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9535200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95300ceca0 con 0x7f9510005180 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95300ce820 0x7f9530159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95300ce820 0x7f9530159510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42616/0 (socket says 172.21.15.141:42616) 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 -- 172.21.15.141:0/251249532 learned_addr learned my addr 172.21.15.141:0/251249532 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 -- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510005180 msgr2=0x7f95300cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f952de00640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510005180 0x7f95300cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510005180 0x7f95300cc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 -- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f6910 msgr2=0x7f95301f6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f6910 0x7f95301f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 -- 172.21.15.141:0/251249532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95300cbab0 con 0x7f95300ce820 2024-01-16T15:21:43.426 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.430+0000 7f9527e00640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95300ce820 0x7f9530159510 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f95180108e0 tx=0x7f9518010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:43.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.431+0000 7f9526000640 1 -- 172.21.15.141:0/251249532 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951800ec20 con 0x7f95300ce820 2024-01-16T15:21:43.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.431+0000 7f9526000640 1 -- 172.21.15.141:0/251249532 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951800ed80 con 0x7f95300ce820 2024-01-16T15:21:43.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.431+0000 7f9526000640 1 -- 172.21.15.141:0/251249532 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951801c600 con 0x7f95300ce820 2024-01-16T15:21:43.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.431+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95300cbd90 con 0x7f95300ce820 2024-01-16T15:21:43.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.431+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9530159a50 con 0x7f95300ce820 2024-01-16T15:21:43.427 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.432+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95301f6250 con 0x7f95300ce820 2024-01-16T15:21:43.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.433+0000 7f9526000640 1 -- 172.21.15.141:0/251249532 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f951801c760 con 0x7f95300ce820 2024-01-16T15:21:43.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.434+0000 7f9526000640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806c670 0x7f950806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.428 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.434+0000 7f9526000640 1 -- 172.21.15.141:0/251249532 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f95180892e0 con 0x7f95300ce820 2024-01-16T15:21:43.429 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.434+0000 7f952de00640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806c670 0x7f950806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.430 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.435+0000 7f952de00640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806c670 0x7f950806eb30 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9520000f60 tx=0x7f9520009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:43.433 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.438+0000 7f9526000640 1 -- 172.21.15.141:0/251249532 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95301f6250 con 0x7f95300ce820 2024-01-16T15:21:43.655 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.660+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9530070550 con 0x7f950806c670 2024-01-16T15:21:43.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.704+0000 7f9526000640 1 -- 172.21.15.141:0/251249532 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f952000d040 con 0x7f950806c670 2024-01-16T15:21:43.699 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:43.699 INFO:teuthology.orchestra.run.smithi141.stdout:[ 2024-01-16T15:21:43.699 INFO:teuthology.orchestra.run.smithi141.stdout: { 2024-01-16T15:21:43.699 INFO:teuthology.orchestra.run.smithi141.stdout: "name": "ss1" 2024-01-16T15:21:43.699 INFO:teuthology.orchestra.run.smithi141.stdout: } 2024-01-16T15:21:43.699 INFO:teuthology.orchestra.run.smithi141.stdout:] 2024-01-16T15:21:43.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806c670 msgr2=0x7f950806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806c670 0x7f950806eb30 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9520000f60 tx=0x7f9520009040 comp rx=0 tx=0).stop 2024-01-16T15:21:43.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95300ce820 msgr2=0x7f9530159510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95300ce820 0x7f9530159510 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f95180108e0 tx=0x7f9518010da0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 shutdown_connections 2024-01-16T15:21:43.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f950806c670 0x7f950806eb30 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95301f6910 0x7f95301f6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95300ce820 0x7f9530159510 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 --2- 172.21.15.141:0/251249532 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9510005180 0x7f95300cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 >> 172.21.15.141:0/251249532 conn(0x7f953006a800 msgr2=0x7f9530161860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:43.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 shutdown_connections 2024-01-16T15:21:43.702 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.706+0000 7f9535200640 1 -- 172.21.15.141:0/251249532 wait complete. 2024-01-16T15:21:43.720 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:43.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.938+0000 7ffaa8200640 1 Processor -- start 2024-01-16T15:21:43.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.938+0000 7ffaa8200640 1 -- start start 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa8200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ffaa0143ae0 0x7ffaa00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa8200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ffaa015e3b0 0x7ffaa00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa8200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa00cfc30 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa8200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa00b1ee0 con 0x7ffaa015e3b0 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa8200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa00b2780 con 0x7ffaa0143ae0 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa4e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ffaa00cf7d0 0x7ffaa015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52322/0 (socket says 172.21.15.141:52322) 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa4e00640 1 -- 172.21.15.141:0/1105135215 learned_addr learned my addr 172.21.15.141:0/1105135215 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:21:43.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa6200640 1 --2- 172.21.15.141:0/1105135215 >> v2:172.21.15.141:3300/0 conn(0x7ffaa015e3b0 0x7ffaa00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffa9fe00640 1 -- 172.21.15.141:0/1105135215 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129115300 0 0) 0x7ffaa00cfc30 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffa9fe00640 1 -- 172.21.15.141:0/1105135215 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa80003520 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa5800640 1 --2- 172.21.15.141:0/1105135215 >> v2:172.21.15.188:3300/0 conn(0x7ffaa0143ae0 0x7ffaa00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa6200640 1 -- 172.21.15.141:0/1105135215 >> v2:172.21.15.188:3300/0 conn(0x7ffaa0143ae0 msgr2=0x7ffaa00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:43.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa6200640 1 --2- 172.21.15.141:0/1105135215 >> v2:172.21.15.188:3300/0 conn(0x7ffaa0143ae0 0x7ffaa00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa6200640 1 -- 172.21.15.141:0/1105135215 >> v1:172.21.15.141:6789/0 conn(0x7ffaa00cf7d0 legacy=0x7ffaa015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.937 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.940+0000 7ffaa6200640 1 -- 172.21.15.141:0/1105135215 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa015e990 con 0x7ffaa015e3b0 2024-01-16T15:21:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.941+0000 7ffaa6200640 1 --2- 172.21.15.141:0/1105135215 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa015e3b0 0x7ffaa00702c0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00cfea0 tx=0x7ffa9802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99408846528cdc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffa9fe00640 1 -- 172.21.15.141:0/1105135215 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9802cc70 con 0x7ffaa015e3b0 2024-01-16T15:21:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffa9fe00640 1 -- 172.21.15.141:0/1105135215 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9802cdd0 con 0x7ffaa015e3b0 2024-01-16T15:21:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffa9fe00640 1 -- 172.21.15.141:0/1105135215 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa98034660 con 0x7ffaa015e3b0 2024-01-16T15:21:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 -- 172.21.15.141:0/1105135215 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa015e3b0 msgr2=0x7ffaa00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 --2- 172.21.15.141:0/1105135215 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa015e3b0 0x7ffaa00702c0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00cfea0 tx=0x7ffa9802bec0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 -- 172.21.15.141:0/1105135215 shutdown_connections 2024-01-16T15:21:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 --2- 172.21.15.141:0/1105135215 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa015e3b0 0x7ffaa00702c0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 --2- 172.21.15.141:0/1105135215 >> v2:172.21.15.188:3300/0 conn(0x7ffaa0143ae0 0x7ffaa00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 -- 172.21.15.141:0/1105135215 >> 172.21.15.141:0/1105135215 conn(0x7ffaa006a800 msgr2=0x7ffaa006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 -- 172.21.15.141:0/1105135215 shutdown_connections 2024-01-16T15:21:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.942+0000 7ffaa8200640 1 -- 172.21.15.141:0/1105135215 wait complete. 2024-01-16T15:21:43.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.944+0000 7ffaa8200640 1 Processor -- start 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.944+0000 7ffaa8200640 1 -- start start 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa8200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa00cf7d0 0x7ffaa0149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa8200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaa0143ae0 0x7ffaa013a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa8200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa015e3b0 0x7ffaa013ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa8200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa013ea80 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa8200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00af760 con 0x7ffaa015e3b0 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa8200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00aeb50 con 0x7ffaa0143ae0 2024-01-16T15:21:43.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa00cf7d0 0x7ffaa0149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa5800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa00cf7d0 0x7ffaa0149700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42632/0 (socket says 172.21.15.141:42632) 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.945+0000 7ffaa5800640 1 -- 172.21.15.141:0/2002177074 learned_addr learned my addr 172.21.15.141:0/2002177074 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa6200640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa015e3b0 0x7ffaa013ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa4e00640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaa0143ae0 0x7ffaa013a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa5800640 1 -- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaa0143ae0 msgr2=0x7ffaa013a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa5800640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaa0143ae0 0x7ffaa013a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa5800640 1 -- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa015e3b0 msgr2=0x7ffaa013ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:43.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa5800640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa015e3b0 0x7ffaa013ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa5800640 1 -- 172.21.15.141:0/2002177074 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa98009a70 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa4e00640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffaa0143ae0 0x7ffaa013a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffaa5800640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaa00cf7d0 0x7ffaa0149700 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000d800 tx=0x7ffa9000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.946+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9000b6c0 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.947+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9000b820 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.947+0000 7ffaa8200640 1 -- 172.21.15.141:0/2002177074 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa00aee30 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.947+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9000bbb0 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.947 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.947+0000 7ffaa8200640 1 -- 172.21.15.141:0/2002177074 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa00af3f0 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.948+0000 7ffaa8200640 1 -- 172.21.15.141:0/2002177074 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa006d810 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.949+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7ffa9000bd10 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.950+0000 7ffa9d600640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa7806c670 0x7ffa7806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:43.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.950+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffa90089180 con 0x7ffaa00cf7d0 2024-01-16T15:21:43.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.951+0000 7ffaa4e00640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa7806c670 0x7ffa7806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:43.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.952+0000 7ffaa4e00640 1 --2- 172.21.15.141:0/2002177074 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffa7806c670 0x7ffa7806eb30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ffaa015b1d0 tx=0x7ffa88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:43.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:43.954+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa90055690 con 0x7ffaa00cf7d0 2024-01-16T15:21:44.106 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:44.111+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7ffa90055a60 con 0x7ffaa00cf7d0 2024-01-16T15:21:44.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:44.181+0000 7ffaa8200640 1 -- 172.21.15.141:0/2002177074 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffaa00cbd00 con 0x7ffa7806c670 2024-01-16T15:21:44.189 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:44.196+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:44.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:44.195+0000 7ffa9d600640 1 -- 172.21.15.141:0/2002177074 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffaa00cbd00 con 0x7ffa7806c670 2024-01-16T15:21:44.190 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:44.190 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:46.205 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:46.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.486+0000 7fd50f200640 1 Processor -- start 2024-01-16T15:21:46.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.486+0000 7fd50f200640 1 -- start start 2024-01-16T15:21:46.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.488+0000 7fd50f200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd508160920 0x7fd5080b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.488+0000 7fd50f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5080cf9f0 0x7fd5080cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.488+0000 7fd50f200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080d0a10 con 0x7fd508161010 2024-01-16T15:21:46.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.488+0000 7fd50f200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5080d03b0 con 0x7fd508160920 2024-01-16T15:21:46.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.488+0000 7fd50f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080d1f50 con 0x7fd5080cf9f0 2024-01-16T15:21:46.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.488+0000 7fd50e800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd508161010 0x7fd5081614a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:52324/0 (socket says 172.21.15.141:52324) 2024-01-16T15:21:46.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.488+0000 7fd50e800640 1 -- 172.21.15.141:0/549428171 learned_addr learned my addr 172.21.15.141:0/549428171 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514577040 0 0) 0x7fd5080d0a10 con 0x7fd508161010 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e8003520 con 0x7fd508161010 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428150568 0 0) 0x7fd4e8003520 con 0x7fd508161010 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5080cf9f0 msgr2=0x7fd5080cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5080cf9f0 0x7fd5080cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 >> v2:172.21.15.188:3300/0 conn(0x7fd508160920 msgr2=0x7fd5080b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 --2- 172.21.15.141:0/549428171 >> v2:172.21.15.188:3300/0 conn(0x7fd508160920 0x7fd5080b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080cf080 con 0x7fd508161010 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd5000019f0 con 0x7fd508161010 2024-01-16T15:21:46.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 >> v1:172.21.15.141:6789/0 conn(0x7fd508161010 legacy=0x7fd5081614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd4e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4e80047c0 0x7fd4e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5080cf9f0 0x7fd4e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080d03b0 con 0x7fd4e8003970 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080d1f50 con 0x7fd4e80047c0 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.489+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e8005560 con 0x7fd5080cf9f0 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.490+0000 7fd50de00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd4e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.490+0000 7fd50de00640 1 -- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5080cf9f0 msgr2=0x7fd4e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.490+0000 7fd50de00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5080cf9f0 0x7fd4e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.490+0000 7fd50de00640 1 -- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4e80047c0 msgr2=0x7fd4e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:46.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.490+0000 7fd50de00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4e80047c0 0x7fd4e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:46.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.490+0000 7fd50de00640 1 -- 172.21.15.141:0/549428171 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8003520 con 0x7fd4e8003970 2024-01-16T15:21:46.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.492+0000 7fd50de00640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd4e8003d90 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009860 tx=0x7fd4e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3030794cbef083 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:46.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.493+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8032600 con 0x7fd4e8003970 2024-01-16T15:21:46.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.493+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f802e050 con 0x7fd4e8003970 2024-01-16T15:21:46.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.493+0000 7fd50ca00640 1 -- 172.21.15.141:0/549428171 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80329c0 con 0x7fd4e8003970 2024-01-16T15:21:46.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.493+0000 7fd50f200640 1 -- 172.21.15.141:0/549428171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 msgr2=0x7fd4e8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:46.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.493+0000 7fd50f200640 1 --2- 172.21.15.141:0/549428171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd4e8003d90 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009860 tx=0x7fd4e80057f0 comp rx=0 tx=0).stop 2024-01-16T15:21:46.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.494+0000 7fd50e800640 1 -- 172.21.15.141:0/549428171 reap_dead start 2024-01-16T15:21:46.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.494+0000 7fd50f200640 1 -- 172.21.15.141:0/549428171 shutdown_connections 2024-01-16T15:21:46.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.494+0000 7fd50f200640 1 -- 172.21.15.141:0/549428171 >> 172.21.15.141:0/549428171 conn(0x7fd5080d3090 msgr2=0x7fd5080d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:46.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.494+0000 7fd50f200640 1 -- 172.21.15.141:0/549428171 shutdown_connections 2024-01-16T15:21:46.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.494+0000 7fd50f200640 1 -- 172.21.15.141:0/549428171 wait complete. 2024-01-16T15:21:46.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.497+0000 7fd50f200640 1 Processor -- start 2024-01-16T15:21:46.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.497+0000 7fd50f200640 1 -- start start 2024-01-16T15:21:46.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.498+0000 7fd50f200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd508161690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.498+0000 7fd50f200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081500f0 0x7fd508150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.498+0000 7fd50f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5080b6490 0x7fd5080b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.498+0000 7fd50f200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508150ab0 con 0x7fd4e8003970 2024-01-16T15:21:46.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.498+0000 7fd50f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b6e30 con 0x7fd5080b6490 2024-01-16T15:21:46.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.498+0000 7fd50f200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080ca4d0 con 0x7fd5081500f0 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.498+0000 7fd50de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd508161690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd508161690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42652/0 (socket says 172.21.15.141:42652) 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 -- 172.21.15.141:0/3565407338 learned_addr learned my addr 172.21.15.141:0/3565407338 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50e800640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5080b6490 0x7fd5080b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50d400640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081500f0 0x7fd508150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 -- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081500f0 msgr2=0x7fd508150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd5081500f0 0x7fd508150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 -- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5080b6490 msgr2=0x7fd5080b68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5080b6490 0x7fd5080b68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 -- 172.21.15.141:0/3565407338 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080ca7b0 con 0x7fd4e8003970 2024-01-16T15:21:46.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.499+0000 7fd50de00640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd4e8003970 0x7fd508161690 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8035040 tx=0x7fd4f80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:46.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.500+0000 7fd506000640 1 -- 172.21.15.141:0/3565407338 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f803d620 con 0x7fd4e8003970 2024-01-16T15:21:46.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.500+0000 7fd50f200640 1 -- 172.21.15.141:0/3565407338 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5080caa90 con 0x7fd4e8003970 2024-01-16T15:21:46.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.500+0000 7fd506000640 1 -- 172.21.15.141:0/3565407338 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f803d780 con 0x7fd4e8003970 2024-01-16T15:21:46.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.500+0000 7fd506000640 1 -- 172.21.15.141:0/3565407338 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f803db10 con 0x7fd4e8003970 2024-01-16T15:21:46.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.500+0000 7fd50f200640 1 -- 172.21.15.141:0/3565407338 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5080cb050 con 0x7fd4e8003970 2024-01-16T15:21:46.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.502+0000 7fd50f200640 1 -- 172.21.15.141:0/3565407338 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5080cb210 con 0x7fd4e8003970 2024-01-16T15:21:46.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.503+0000 7fd506000640 1 -- 172.21.15.141:0/3565407338 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fd4f803dc70 con 0x7fd4e8003970 2024-01-16T15:21:46.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.506+0000 7fd506000640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006c820 0x7fd4e006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:46.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.506+0000 7fd506000640 1 -- 172.21.15.141:0/3565407338 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd4f80aac70 con 0x7fd4e8003970 2024-01-16T15:21:46.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.512+0000 7fd50d400640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006c820 0x7fd4e006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:46.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.514+0000 7fd506000640 1 -- 172.21.15.141:0/3565407338 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4f8077240 con 0x7fd4e8003970 2024-01-16T15:21:46.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.517+0000 7fd50d400640 1 --2- 172.21.15.141:0/3565407338 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd4e006c820 0x7fd4e006ece0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0006fd0 tx=0x7fd4f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:46.816 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.822+0000 7fd50f200640 1 -- 172.21.15.141:0/3565407338 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5080ceac0 con 0x7fd4e006c820 2024-01-16T15:21:46.826 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:46.831+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:46.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:46.832+0000 7fd506000640 1 -- 172.21.15.141:0/3565407338 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd4f0009e60 con 0x7fd4e006c820 2024-01-16T15:21:46.827 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:46.827 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:48.838 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:49.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.074+0000 7fc1ec200640 1 Processor -- start 2024-01-16T15:21:49.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.074+0000 7fc1ec200640 1 -- start start 2024-01-16T15:21:49.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1ec200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc1e413d310 0x7fc1e406b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:49.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1ec200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1e415b130 0x7fc1e406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:49.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1ec200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e406bd60 con 0x7fc1e406cf80 2024-01-16T15:21:49.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1ec200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e413d6f0 con 0x7fc1e415b130 2024-01-16T15:21:49.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1ec200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e406b400 con 0x7fc1e413d310 2024-01-16T15:21:49.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1e8e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1e415b130 0x7fc1e406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:49.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1e8e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1e415b130 0x7fc1e406ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42656/0 (socket says 172.21.15.141:42656) 2024-01-16T15:21:49.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.076+0000 7fc1e8e00640 1 -- 172.21.15.141:0/3049040472 learned_addr learned my addr 172.21.15.141:0/3049040472 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:49.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.077+0000 7fc1e9800640 1 --2- 172.21.15.141:0/3049040472 >> v2:172.21.15.188:3300/0 conn(0x7fc1e413d310 0x7fc1e406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.077+0000 7fc1e8e00640 1 -- 172.21.15.141:0/3049040472 >> v2:172.21.15.188:3300/0 conn(0x7fc1e413d310 msgr2=0x7fc1e406b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.077+0000 7fc1e8e00640 1 --2- 172.21.15.141:0/3049040472 >> v2:172.21.15.188:3300/0 conn(0x7fc1e413d310 0x7fc1e406b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.077+0000 7fc1e8e00640 1 -- 172.21.15.141:0/3049040472 >> v1:172.21.15.188:6789/0 conn(0x7fc1e406cf80 legacy=0x7fc1e413d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.077+0000 7fc1e8e00640 1 -- 172.21.15.141:0/3049040472 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e406d3b0 con 0x7fc1e415b130 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.077+0000 7fc1e9800640 1 --2- 172.21.15.141:0/3049040472 >> v2:172.21.15.188:3300/0 conn(0x7fc1e413d310 0x7fc1e406b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.078+0000 7fc1e8e00640 1 --2- 172.21.15.141:0/3049040472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 0x7fc1e406ca40 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fc1e406c8b0 tx=0x7fc1d402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ccc9a06c62339e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.078+0000 7fc1e3e00640 1 -- 172.21.15.141:0/3049040472 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4031750 con 0x7fc1e415b130 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.078+0000 7fc1e3e00640 1 -- 172.21.15.141:0/3049040472 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d40318b0 con 0x7fc1e415b130 2024-01-16T15:21:49.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.078+0000 7fc1e3e00640 1 -- 172.21.15.141:0/3049040472 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4031c40 con 0x7fc1e415b130 2024-01-16T15:21:49.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 -- 172.21.15.141:0/3049040472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 msgr2=0x7fc1e406ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:49.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 --2- 172.21.15.141:0/3049040472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 0x7fc1e406ca40 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fc1e406c8b0 tx=0x7fc1d402b7f0 comp rx=0 tx=0).stop 2024-01-16T15:21:49.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 -- 172.21.15.141:0/3049040472 shutdown_connections 2024-01-16T15:21:49.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 --2- 172.21.15.141:0/3049040472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 0x7fc1e406ca40 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:49.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 --2- 172.21.15.141:0/3049040472 >> v2:172.21.15.188:3300/0 conn(0x7fc1e413d310 0x7fc1e406b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:49.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 -- 172.21.15.141:0/3049040472 >> 172.21.15.141:0/3049040472 conn(0x7fc1e40cf710 msgr2=0x7fc1e40cfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:49.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 -- 172.21.15.141:0/3049040472 shutdown_connections 2024-01-16T15:21:49.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.079+0000 7fc1ec200640 1 -- 172.21.15.141:0/3049040472 wait complete. 2024-01-16T15:21:49.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.081+0000 7fc1ec200640 1 Processor -- start 2024-01-16T15:21:49.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.081+0000 7fc1ec200640 1 -- start start 2024-01-16T15:21:49.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ec200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1e406cf80 0x7fc1e415d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:49.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ec200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1e413d310 0x7fc1e413d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ec200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 0x7fc1e413b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ec200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e413dd70 con 0x7fc1e415b130 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ec200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e413e010 con 0x7fc1e413d310 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ec200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e413e2f0 con 0x7fc1e406cf80 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ea200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 0x7fc1e413b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ea200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 0x7fc1e413b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42662/0 (socket says 172.21.15.141:42662) 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ea200640 1 -- 172.21.15.141:0/352170840 learned_addr learned my addr 172.21.15.141:0/352170840 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1e9800640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1e406cf80 0x7fc1e415d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1e8e00640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1e413d310 0x7fc1e413d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ea200640 1 -- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1e406cf80 msgr2=0x7fc1e415d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.083+0000 7fc1ea200640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc1e406cf80 0x7fc1e415d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1ea200640 1 -- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1e413d310 msgr2=0x7fc1e413d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1ea200640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1e413d310 0x7fc1e413d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:49.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1ea200640 1 -- 172.21.15.141:0/352170840 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40094d0 con 0x7fc1e415b130 2024-01-16T15:21:49.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1ea200640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1e415b130 0x7fc1e413b060 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00b990 tx=0x7fc1dc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:49.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1e1600640 1 -- 172.21.15.141:0/352170840 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc00cba0 con 0x7fc1e415b130 2024-01-16T15:21:49.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1e1600640 1 -- 172.21.15.141:0/352170840 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc00cd00 con 0x7fc1e415b130 2024-01-16T15:21:49.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1e1600640 1 -- 172.21.15.141:0/352170840 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc014680 con 0x7fc1e415b130 2024-01-16T15:21:49.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1ec200640 1 -- 172.21.15.141:0/352170840 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e413e5d0 con 0x7fc1e415b130 2024-01-16T15:21:49.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.084+0000 7fc1ec200640 1 -- 172.21.15.141:0/352170840 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e413eb90 con 0x7fc1e415b130 2024-01-16T15:21:49.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.085+0000 7fc1ec200640 1 -- 172.21.15.141:0/352170840 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e413ed80 con 0x7fc1e415b130 2024-01-16T15:21:49.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.087+0000 7fc1e1600640 1 -- 172.21.15.141:0/352170840 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fc1dc0147e0 con 0x7fc1e415b130 2024-01-16T15:21:49.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.088+0000 7fc1e1600640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1bc06c820 0x7fc1bc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:49.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.089+0000 7fc1e1600640 1 -- 172.21.15.141:0/352170840 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc1dc088c50 con 0x7fc1e415b130 2024-01-16T15:21:49.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.089+0000 7fc1e9800640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1bc06c820 0x7fc1bc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:49.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.090+0000 7fc1e9800640 1 --2- 172.21.15.141:0/352170840 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1bc06c820 0x7fc1bc06ece0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00a950 tx=0x7fc1cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:49.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.093+0000 7fc1e1600640 1 -- 172.21.15.141:0/352170840 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1dc055110 con 0x7fc1e415b130 2024-01-16T15:21:49.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.314+0000 7fc1ec200640 1 -- 172.21.15.141:0/352170840 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1e40662c0 con 0x7fc1bc06c820 2024-01-16T15:21:49.321 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:49.327+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:49.322 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:49.328+0000 7fc1e1600640 1 -- 172.21.15.141:0/352170840 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc1e40662c0 con 0x7fc1bc06c820 2024-01-16T15:21:49.322 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:49.322 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:51.337 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:51.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.573+0000 7f57fa200640 1 Processor -- start 2024-01-16T15:21:51.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.573+0000 7f57fa200640 1 -- start start 2024-01-16T15:21:51.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57fa200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f57f415e170 0x7f57f40cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:51.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57fa200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f57f40cdef0 0x7f57f4070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:51.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57fa200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f40700f0 con 0x7f57f413cda0 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57fa200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f40ce720 con 0x7f57f415e170 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57fa200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f40cea00 con 0x7f57f40cdef0 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57ebc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f57f415e170 0x7f57f40cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57ebc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f57f415e170 0x7f57f40cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42666/0 (socket says 172.21.15.141:42666) 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57ebc00640 1 -- 172.21.15.141:0/1865221180 learned_addr learned my addr 172.21.15.141:0/1865221180 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57ebc00640 1 -- 172.21.15.141:0/1865221180 >> v2:172.21.15.188:3300/0 conn(0x7f57f40cdef0 msgr2=0x7f57f4070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57ebc00640 1 --2- 172.21.15.141:0/1865221180 >> v2:172.21.15.188:3300/0 conn(0x7f57f40cdef0 0x7f57f4070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57ebc00640 1 -- 172.21.15.141:0/1865221180 >> v1:172.21.15.188:6789/0 conn(0x7f57f413cda0 legacy=0x7f57f415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:51.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.575+0000 7f57ebc00640 1 -- 172.21.15.141:0/1865221180 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f4070340 con 0x7f57f415e170 2024-01-16T15:21:51.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.576+0000 7f57ebc00640 1 --2- 172.21.15.141:0/1865221180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f415e170 0x7f57f40cd9b0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f57f40aea90 tx=0x7f57dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ae82cc7f61bd8f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:51.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57f3e00640 1 -- 172.21.15.141:0/1865221180 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0317e0 con 0x7f57f415e170 2024-01-16T15:21:51.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57f3e00640 1 -- 172.21.15.141:0/1865221180 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc031940 con 0x7f57f415e170 2024-01-16T15:21:51.571 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57f3e00640 1 -- 172.21.15.141:0/1865221180 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc031cd0 con 0x7f57f415e170 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57fa200640 1 -- 172.21.15.141:0/1865221180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f415e170 msgr2=0x7f57f40cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57fa200640 1 --2- 172.21.15.141:0/1865221180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f415e170 0x7f57f40cd9b0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f57f40aea90 tx=0x7f57dc02b880 comp rx=0 tx=0).stop 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57fa200640 1 -- 172.21.15.141:0/1865221180 shutdown_connections 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57fa200640 1 --2- 172.21.15.141:0/1865221180 >> v2:172.21.15.188:3300/0 conn(0x7f57f40cdef0 0x7f57f4070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57fa200640 1 --2- 172.21.15.141:0/1865221180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f415e170 0x7f57f40cd9b0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.577+0000 7f57fa200640 1 -- 172.21.15.141:0/1865221180 >> 172.21.15.141:0/1865221180 conn(0x7f57f406a3e0 msgr2=0x7f57f406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.578+0000 7f57fa200640 1 -- 172.21.15.141:0/1865221180 shutdown_connections 2024-01-16T15:21:51.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.578+0000 7f57fa200640 1 -- 172.21.15.141:0/1865221180 wait complete. 2024-01-16T15:21:51.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.580+0000 7f57fa200640 1 Processor -- start 2024-01-16T15:21:51.574 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.580+0000 7f57fa200640 1 -- start start 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.581+0000 7f57fa200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f40cdef0 0x7f57f413f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.581+0000 7f57fa200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f413cda0 0x7f57f41f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.581+0000 7f57fa200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57f415e170 0x7f57f40afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.581+0000 7f57fa200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f41555f0 con 0x7f57f40cdef0 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.581+0000 7f57fa200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f413f030 con 0x7f57f413cda0 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.581+0000 7f57fa200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f4157a30 con 0x7f57f415e170 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f40cdef0 0x7f57f413f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f40cdef0 0x7f57f413f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:42670/0 (socket says 172.21.15.141:42670) 2024-01-16T15:21:51.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 -- 172.21.15.141:0/2936128812 learned_addr learned my addr 172.21.15.141:0/2936128812 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:51.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57ebc00640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f413cda0 0x7f57f41f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f9800640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57f415e170 0x7f57f40afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 -- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57f415e170 msgr2=0x7f57f40afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f57f415e170 0x7f57f40afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 -- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f413cda0 msgr2=0x7f57f41f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f413cda0 0x7f57f41f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.582+0000 7f57f8e00640 1 -- 172.21.15.141:0/2936128812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc0094d0 con 0x7f57f40cdef0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.583+0000 7f57f8e00640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57f40cdef0 0x7f57f413f4c0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f57e400b3e0 tx=0x7f57e400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.583+0000 7f57f1600640 1 -- 172.21.15.141:0/2936128812 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e4011830 con 0x7f57f40cdef0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.583+0000 7f57f1600640 1 -- 172.21.15.141:0/2936128812 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e40119b0 con 0x7f57f40cdef0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.583+0000 7f57f1600640 1 -- 172.21.15.141:0/2936128812 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e4011d60 con 0x7f57f40cdef0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.583+0000 7f57fa200640 1 -- 172.21.15.141:0/2936128812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f413e700 con 0x7f57f40cdef0 2024-01-16T15:21:51.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.583+0000 7f57fa200640 1 -- 172.21.15.141:0/2936128812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f413ecc0 con 0x7f57f40cdef0 2024-01-16T15:21:51.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.584+0000 7f57fa200640 1 -- 172.21.15.141:0/2936128812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f4067450 con 0x7f57f40cdef0 2024-01-16T15:21:51.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.585+0000 7f57f1600640 1 -- 172.21.15.141:0/2936128812 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f57e401a410 con 0x7f57f40cdef0 2024-01-16T15:21:51.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.586+0000 7f57f1600640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57cc06c850 0x7f57cc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:51.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.586+0000 7f57f1600640 1 -- 172.21.15.141:0/2936128812 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f57e4088920 con 0x7f57f40cdef0 2024-01-16T15:21:51.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.587+0000 7f57ebc00640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57cc06c850 0x7f57cc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:51.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.588+0000 7f57ebc00640 1 --2- 172.21.15.141:0/2936128812 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f57cc06c850 0x7f57cc06ed10 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f57dc036000 tx=0x7f57dc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:51.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.591+0000 7f57f1600640 1 -- 172.21.15.141:0/2936128812 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57e4054de0 con 0x7f57f40cdef0 2024-01-16T15:21:51.808 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.814+0000 7f57fa200640 1 -- 172.21.15.141:0/2936128812 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57f40ce340 con 0x7f57cc06c850 2024-01-16T15:21:51.822 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:51.827+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:51.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:51.828+0000 7f57f1600640 1 -- 172.21.15.141:0/2936128812 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f57dc031cc0 con 0x7f57cc06c850 2024-01-16T15:21:51.823 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:51.823 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:53.842 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:54.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.074+0000 7fdbaaa00640 1 Processor -- start 2024-01-16T15:21:54.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.074+0000 7fdbaaa00640 1 -- start start 2024-01-16T15:21:54.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdbaaa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdba40d0b60 0x7fdba40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:54.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdbaaa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 0x7fdba406f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:54.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdbaaa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4071de0 con 0x7fdba406a3e0 2024-01-16T15:21:54.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdbaaa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba40af270 con 0x7fdba40d0b60 2024-01-16T15:21:54.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdbaaa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4072030 con 0x7fdba40aede0 2024-01-16T15:21:54.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdba3400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdba40d0b60 0x7fdba40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdba3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdba406a3e0 0x7fdba406a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53712/0 (socket says 172.21.15.141:53712) 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.076+0000 7fdba3e00640 1 -- 172.21.15.141:0/137948668 learned_addr learned my addr 172.21.15.141:0/137948668 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.077+0000 7fdba8a00640 1 --2- 172.21.15.141:0/137948668 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 0x7fdba406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.077+0000 7fdba3400640 1 -- 172.21.15.141:0/137948668 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 msgr2=0x7fdba406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.077+0000 7fdba3400640 1 --2- 172.21.15.141:0/137948668 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 0x7fdba406f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.077+0000 7fdba3400640 1 -- 172.21.15.141:0/137948668 >> v1:172.21.15.141:6789/0 conn(0x7fdba406a3e0 legacy=0x7fdba406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.077+0000 7fdba3400640 1 -- 172.21.15.141:0/137948668 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4072310 con 0x7fdba40d0b60 2024-01-16T15:21:54.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.077+0000 7fdba2a00640 1 -- 172.21.15.141:0/137948668 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499092574 0 0) 0x7fdba4071de0 con 0x7fdba406a3e0 2024-01-16T15:21:54.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.078+0000 7fdba3400640 1 --2- 172.21.15.141:0/137948668 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40d0b60 0x7fdba40ac8d0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0097a0 tx=0x7fdb8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5483c924de0770fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:54.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.078+0000 7fdba2a00640 1 -- 172.21.15.141:0/137948668 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c031830 con 0x7fdba40d0b60 2024-01-16T15:21:54.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.078+0000 7fdba2a00640 1 -- 172.21.15.141:0/137948668 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c031990 con 0x7fdba40d0b60 2024-01-16T15:21:54.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.078+0000 7fdba2a00640 1 -- 172.21.15.141:0/137948668 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c031d20 con 0x7fdba40d0b60 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 -- 172.21.15.141:0/137948668 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40d0b60 msgr2=0x7fdba40ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 --2- 172.21.15.141:0/137948668 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40d0b60 0x7fdba40ac8d0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0097a0 tx=0x7fdb8c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 -- 172.21.15.141:0/137948668 shutdown_connections 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 --2- 172.21.15.141:0/137948668 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 0x7fdba406f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 --2- 172.21.15.141:0/137948668 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba40d0b60 0x7fdba40ac8d0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 -- 172.21.15.141:0/137948668 >> 172.21.15.141:0/137948668 conn(0x7fdba40cfef0 msgr2=0x7fdba40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 -- 172.21.15.141:0/137948668 shutdown_connections 2024-01-16T15:21:54.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.079+0000 7fdbaaa00640 1 -- 172.21.15.141:0/137948668 wait complete. 2024-01-16T15:21:54.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.081+0000 7fdbaaa00640 1 Processor -- start 2024-01-16T15:21:54.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.081+0000 7fdbaaa00640 1 -- start start 2024-01-16T15:21:54.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdbaaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba406a3e0 0x7fdba40cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdbaaa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 0x7fdba40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdbaaa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40d0b60 0x7fdba4150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdbaaa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40cec60 con 0x7fdba406a3e0 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdbaaa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40ce400 con 0x7fdba40d0b60 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdbaaa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40ce6a0 con 0x7fdba40aede0 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdba3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba406a3e0 0x7fdba40cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdba3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba406a3e0 0x7fdba40cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60862/0 (socket says 172.21.15.141:60862) 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdba3e00640 1 -- 172.21.15.141:0/4144788981 learned_addr learned my addr 172.21.15.141:0/4144788981 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdba3400640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 0x7fdba40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.082+0000 7fdba3e00640 1 -- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 msgr2=0x7fdba40cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdba8a00640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40d0b60 0x7fdba4150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:54.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdba3e00640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdba40aede0 0x7fdba40cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdba3e00640 1 -- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40d0b60 msgr2=0x7fdba4150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdba3e00640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba40d0b60 0x7fdba4150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdba3e00640 1 -- 172.21.15.141:0/4144788981 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c009400 con 0x7fdba406a3e0 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdba3e00640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdba406a3e0 0x7fdba40cd780 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400c430 tx=0x7fdb9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdb9be00640 1 -- 172.21.15.141:0/4144788981 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb940074e0 con 0x7fdba406a3e0 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdb9be00640 1 -- 172.21.15.141:0/4144788981 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb94007660 con 0x7fdba406a3e0 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdb9be00640 1 -- 172.21.15.141:0/4144788981 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94004a20 con 0x7fdba406a3e0 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdbaaa00640 1 -- 172.21.15.141:0/4144788981 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba40ce980 con 0x7fdba406a3e0 2024-01-16T15:21:54.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.083+0000 7fdbaaa00640 1 -- 172.21.15.141:0/4144788981 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba40b0050 con 0x7fdba406a3e0 2024-01-16T15:21:54.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.084+0000 7fdbaaa00640 1 -- 172.21.15.141:0/4144788981 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba40675d0 con 0x7fdba406a3e0 2024-01-16T15:21:54.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.085+0000 7fdb9be00640 1 -- 172.21.15.141:0/4144788981 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fdb94004b80 con 0x7fdba406a3e0 2024-01-16T15:21:54.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.086+0000 7fdb9be00640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c820 0x7fdb7c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:54.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.086+0000 7fdb9be00640 1 -- 172.21.15.141:0/4144788981 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdb94088b40 con 0x7fdba406a3e0 2024-01-16T15:21:54.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.087+0000 7fdba3400640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c820 0x7fdb7c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:54.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.088+0000 7fdba3400640 1 --2- 172.21.15.141:0/4144788981 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdb7c06c820 0x7fdb7c06ece0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c036000 tx=0x7fdb8c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:54.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.090+0000 7fdb9be00640 1 -- 172.21.15.141:0/4144788981 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb94055060 con 0x7fdba406a3e0 2024-01-16T15:21:54.310 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.316+0000 7fdbaaa00640 1 -- 172.21.15.141:0/4144788981 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdba406eb60 con 0x7fdb7c06c820 2024-01-16T15:21:54.323 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:54.329+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:54.324 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:54.330+0000 7fdb9be00640 1 -- 172.21.15.141:0/4144788981 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdb8c045400 con 0x7fdb7c06c820 2024-01-16T15:21:54.324 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:54.325 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:56.336 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:56.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.569+0000 7f12c4e00640 1 Processor -- start 2024-01-16T15:21:56.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.569+0000 7f12c4e00640 1 -- start start 2024-01-16T15:21:56.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12c4e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f12c00d0150 0x7f12c00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:56.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12c4e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f12c00ceea0 0x7f12c00b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12c4e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00b2c70 con 0x7f12c015d3c0 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12c4e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c00cf6d0 con 0x7f12c00d0150 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12c4e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c00cf3f0 con 0x7f12c00ceea0 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12bf400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f12c015d3c0 0x7f12c00cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53728/0 (socket says 172.21.15.141:53728) 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12bf400640 1 -- 172.21.15.141:0/4109367706 learned_addr learned my addr 172.21.15.141:0/4109367706 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12b6800640 1 --2- 172.21.15.141:0/4109367706 >> v2:172.21.15.141:3300/0 conn(0x7f12c00d0150 0x7f12c00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12bfe00640 1 --2- 172.21.15.141:0/4109367706 >> v2:172.21.15.188:3300/0 conn(0x7f12c00ceea0 0x7f12c00b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12bea00640 1 -- 172.21.15.141:0/4109367706 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627174586 0 0) 0x7f12c00b2c70 con 0x7f12c015d3c0 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.571+0000 7f12bea00640 1 -- 172.21.15.141:0/4109367706 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0003520 con 0x7f12c015d3c0 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.572+0000 7f12b6800640 1 -- 172.21.15.141:0/4109367706 >> v2:172.21.15.188:3300/0 conn(0x7f12c00ceea0 msgr2=0x7f12c00b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:56.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.572+0000 7f12b6800640 1 --2- 172.21.15.141:0/4109367706 >> v2:172.21.15.188:3300/0 conn(0x7f12c00ceea0 0x7f12c00b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:56.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.572+0000 7f12b6800640 1 -- 172.21.15.141:0/4109367706 >> v1:172.21.15.141:6789/0 conn(0x7f12c015d3c0 legacy=0x7f12c00cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:56.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.572+0000 7f12b6800640 1 -- 172.21.15.141:0/4109367706 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c00b2620 con 0x7f12c00d0150 2024-01-16T15:21:56.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.573+0000 7f12b6800640 1 --2- 172.21.15.141:0/4109367706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 0x7f12c00d1f20 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f12c0072ad0 tx=0x7f12a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71017f026eacc95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:56.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.573+0000 7f12bea00640 1 -- 172.21.15.141:0/4109367706 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8031830 con 0x7f12c00d0150 2024-01-16T15:21:56.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.573+0000 7f12bea00640 1 -- 172.21.15.141:0/4109367706 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a8031990 con 0x7f12c00d0150 2024-01-16T15:21:56.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.573+0000 7f12bea00640 1 -- 172.21.15.141:0/4109367706 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8031d20 con 0x7f12c00d0150 2024-01-16T15:21:56.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 -- 172.21.15.141:0/4109367706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 msgr2=0x7f12c00d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:56.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 --2- 172.21.15.141:0/4109367706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 0x7f12c00d1f20 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f12c0072ad0 tx=0x7f12a802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:21:56.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 -- 172.21.15.141:0/4109367706 shutdown_connections 2024-01-16T15:21:56.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 --2- 172.21.15.141:0/4109367706 >> v2:172.21.15.188:3300/0 conn(0x7f12c00ceea0 0x7f12c00b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:56.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 --2- 172.21.15.141:0/4109367706 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 0x7f12c00d1f20 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:56.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 -- 172.21.15.141:0/4109367706 >> 172.21.15.141:0/4109367706 conn(0x7f12c00d3050 msgr2=0x7f12c00d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:56.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 -- 172.21.15.141:0/4109367706 shutdown_connections 2024-01-16T15:21:56.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.574+0000 7f12c4e00640 1 -- 172.21.15.141:0/4109367706 wait complete. 2024-01-16T15:21:56.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.576+0000 7f12c4e00640 1 Processor -- start 2024-01-16T15:21:56.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.576+0000 7f12c4e00640 1 -- start start 2024-01-16T15:21:56.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12c4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12c00ceea0 0x7f12c00c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:56.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12c4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 0x7f12c015f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:56.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12c4e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12c015d3c0 0x7f12c015ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:56.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12c4e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c015e950 con 0x7f12c00d0150 2024-01-16T15:21:56.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12c4e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c015ec30 con 0x7f12c015d3c0 2024-01-16T15:21:56.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12c4e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c015ef10 con 0x7f12c00ceea0 2024-01-16T15:21:56.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12b6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 0x7f12c015f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:56.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12b6800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 0x7f12c015f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60880/0 (socket says 172.21.15.141:60880) 2024-01-16T15:21:56.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.577+0000 7f12b6800640 1 -- 172.21.15.141:0/2669705744 learned_addr learned my addr 172.21.15.141:0/2669705744 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:56.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12bfe00640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12c015d3c0 0x7f12c015ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:56.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12bf400640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12c00ceea0 0x7f12c00c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:56.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b6800640 1 -- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12c00ceea0 msgr2=0x7f12c00c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:56.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b6800640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f12c00ceea0 0x7f12c00c7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:56.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b6800640 1 -- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12c015d3c0 msgr2=0x7f12c015ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:56.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b6800640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12c015d3c0 0x7f12c015ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:56.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b6800640 1 -- 172.21.15.141:0/2669705744 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8009400 con 0x7f12c00d0150 2024-01-16T15:21:56.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12bfe00640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12c015d3c0 0x7f12c015ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:21:56.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b6800640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f12c00d0150 0x7f12c015f9f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009890 tx=0x7f12a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:56.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b7e00640 1 -- 172.21.15.141:0/2669705744 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a803a6f0 con 0x7f12c00d0150 2024-01-16T15:21:56.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b7e00640 1 -- 172.21.15.141:0/2669705744 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a803a870 con 0x7f12c00d0150 2024-01-16T15:21:56.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12c4e00640 1 -- 172.21.15.141:0/2669705744 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c015f1f0 con 0x7f12c00d0150 2024-01-16T15:21:56.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.578+0000 7f12b7e00640 1 -- 172.21.15.141:0/2669705744 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a803ac20 con 0x7f12c00d0150 2024-01-16T15:21:56.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.579+0000 7f12c4e00640 1 -- 172.21.15.141:0/2669705744 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c0159b40 con 0x7f12c00d0150 2024-01-16T15:21:56.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.579+0000 7f12c4e00640 1 -- 172.21.15.141:0/2669705744 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c015f710 con 0x7f12c00d0150 2024-01-16T15:21:56.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.581+0000 7f12b7e00640 1 -- 172.21.15.141:0/2669705744 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f12a8043410 con 0x7f12c00d0150 2024-01-16T15:21:56.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.581+0000 7f12b7e00640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129806c850 0x7f129806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:56.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.582+0000 7f12b7e00640 1 -- 172.21.15.141:0/2669705744 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f12a80aa7e0 con 0x7f12c00d0150 2024-01-16T15:21:56.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.582+0000 7f12bf400640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129806c850 0x7f129806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:56.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.583+0000 7f12bf400640 1 --2- 172.21.15.141:0/2669705744 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f129806c850 0x7f129806ed10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f12b00011e0 tx=0x7f12b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:56.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.586+0000 7f12b7e00640 1 -- 172.21.15.141:0/2669705744 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12c015f710 con 0x7f12c00d0150 2024-01-16T15:21:56.803 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.809+0000 7f12c4e00640 1 -- 172.21.15.141:0/2669705744 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12c0159d90 con 0x7f129806c850 2024-01-16T15:21:56.819 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:56.824+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:56.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:56.825+0000 7f12b7e00640 1 -- 172.21.15.141:0/2669705744 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f12c0159d90 con 0x7f129806c850 2024-01-16T15:21:56.820 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:56.820 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:21:58.837 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:21:59.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.075+0000 7f4a17a00640 1 Processor -- start 2024-01-16T15:21:59.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.075+0000 7f4a17a00640 1 -- start start 2024-01-16T15:21:59.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.076+0000 7f4a17a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4a10157c30 0x7f4a100cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:59.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a17a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a10071200 0x7f4a1006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:59.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a17a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1015deb0 con 0x7f4a101618f0 2024-01-16T15:21:59.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a17a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1015ea30 con 0x7f4a10157c30 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a17a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1006d860 con 0x7f4a10071200 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a15000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4a10157c30 0x7f4a100cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a15000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4a10157c30 0x7f4a100cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60884/0 (socket says 172.21.15.141:60884) 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a15000640 1 -- 172.21.15.141:0/2546078010 learned_addr learned my addr 172.21.15.141:0/2546078010 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a15a00640 1 --2- 172.21.15.141:0/2546078010 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a10071200 0x7f4a1006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a15000640 1 -- 172.21.15.141:0/2546078010 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a10071200 msgr2=0x7f4a1006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a15000640 1 --2- 172.21.15.141:0/2546078010 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a10071200 0x7f4a1006dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.077+0000 7f4a15000640 1 -- 172.21.15.141:0/2546078010 >> v1:172.21.15.188:6789/0 conn(0x7f4a101618f0 legacy=0x7f4a10070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:59.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.078+0000 7f4a15000640 1 -- 172.21.15.141:0/2546078010 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1015e190 con 0x7f4a10157c30 2024-01-16T15:21:59.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.079+0000 7f4a15000640 1 --2- 172.21.15.141:0/2546078010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 0x7f4a100cd600 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4a100723f0 tx=0x7f4a0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953d3d33fa07b1a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:59.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.079+0000 7f4a0f400640 1 -- 172.21.15.141:0/2546078010 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a00031830 con 0x7f4a10157c30 2024-01-16T15:21:59.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.079+0000 7f4a0f400640 1 -- 172.21.15.141:0/2546078010 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a00031990 con 0x7f4a10157c30 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.079+0000 7f4a0f400640 1 -- 172.21.15.141:0/2546078010 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a00031d20 con 0x7f4a10157c30 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 -- 172.21.15.141:0/2546078010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 msgr2=0x7f4a100cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 --2- 172.21.15.141:0/2546078010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 0x7f4a100cd600 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4a100723f0 tx=0x7f4a0002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 -- 172.21.15.141:0/2546078010 shutdown_connections 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 --2- 172.21.15.141:0/2546078010 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a10071200 0x7f4a1006dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 --2- 172.21.15.141:0/2546078010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 0x7f4a100cd600 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 -- 172.21.15.141:0/2546078010 >> 172.21.15.141:0/2546078010 conn(0x7f4a100cf700 msgr2=0x7f4a100cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 -- 172.21.15.141:0/2546078010 shutdown_connections 2024-01-16T15:21:59.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.080+0000 7f4a17a00640 1 -- 172.21.15.141:0/2546078010 wait complete. 2024-01-16T15:21:59.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.082+0000 7f4a17a00640 1 Processor -- start 2024-01-16T15:21:59.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.082+0000 7f4a17a00640 1 -- start start 2024-01-16T15:21:59.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a17a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a10071200 0x7f4a10154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:59.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a17a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 0x7f4a1014db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:59.079 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a17a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a101618f0 0x7f4a10151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a17a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1014e0d0 con 0x7f4a10157c30 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a17a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100b3f30 con 0x7f4a10071200 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a17a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a10153e40 con 0x7f4a101618f0 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 0x7f4a1014db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 0x7f4a1014db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60892/0 (socket says 172.21.15.141:60892) 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 -- 172.21.15.141:0/3320239719 learned_addr learned my addr 172.21.15.141:0/3320239719 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a15000640 1 --2- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a10071200 0x7f4a10154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 -- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a101618f0 msgr2=0x7f4a10151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 --2- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4a101618f0 0x7f4a10151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 -- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a10071200 msgr2=0x7f4a10154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 --2- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a10071200 0x7f4a10154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:59.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.083+0000 7f4a0fe00640 1 -- 172.21.15.141:0/3320239719 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000094d0 con 0x7f4a10157c30 2024-01-16T15:21:59.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.084+0000 7f4a0fe00640 1 --2- 172.21.15.141:0/3320239719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a10157c30 0x7f4a1014db90 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f49f800c430 tx=0x7f49f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:59.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.084+0000 7f4a0cc00640 1 -- 172.21.15.141:0/3320239719 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8011790 con 0x7f4a10157c30 2024-01-16T15:21:59.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.084+0000 7f4a0cc00640 1 -- 172.21.15.141:0/3320239719 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f80118f0 con 0x7f4a10157c30 2024-01-16T15:21:59.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.084+0000 7f4a0cc00640 1 -- 172.21.15.141:0/3320239719 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8011ce0 con 0x7f4a10157c30 2024-01-16T15:21:59.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.084+0000 7f4a17a00640 1 -- 172.21.15.141:0/3320239719 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a10154120 con 0x7f4a10157c30 2024-01-16T15:21:59.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.084+0000 7f4a17a00640 1 -- 172.21.15.141:0/3320239719 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a101546e0 con 0x7f4a10157c30 2024-01-16T15:21:59.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.085+0000 7f4a17a00640 1 -- 172.21.15.141:0/3320239719 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a10074780 con 0x7f4a10157c30 2024-01-16T15:21:59.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.086+0000 7f4a0cc00640 1 -- 172.21.15.141:0/3320239719 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f49f801c460 con 0x7f4a10157c30 2024-01-16T15:21:59.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.087+0000 7f4a0cc00640 1 --2- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f49e806c820 0x7f49e806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:59.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.087+0000 7f4a0cc00640 1 -- 172.21.15.141:0/3320239719 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f49f8088c80 con 0x7f4a10157c30 2024-01-16T15:21:59.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.088+0000 7f4a15000640 1 --2- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f49e806c820 0x7f49e806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:59.083 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.089+0000 7f4a15000640 1 --2- 172.21.15.141:0/3320239719 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f49e806c820 0x7f49e806ece0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4a00036000 tx=0x7f4a0003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:59.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.092+0000 7f4a0cc00640 1 -- 172.21.15.141:0/3320239719 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49f808ee60 con 0x7f4a10157c30 2024-01-16T15:21:59.304 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.310+0000 7f4a17a00640 1 -- 172.21.15.141:0/3320239719 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a10161da0 con 0x7f49e806c820 2024-01-16T15:21:59.317 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:21:59.322+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:21:59.317 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:21:59.323+0000 7f4a0cc00640 1 -- 172.21.15.141:0/3320239719 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4a00031d90 con 0x7f49e806c820 2024-01-16T15:21:59.317 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:21:59.318 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:21:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:01.332 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:01.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.562+0000 7fe905400640 1 Processor -- start 2024-01-16T15:22:01.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.563+0000 7fe905400640 1 -- start start 2024-01-16T15:22:01.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.564+0000 7fe905400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe900160890 0x7fe9000d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:01.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.564+0000 7fe905400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9000cf6a0 con 0x7fe900160c70 2024-01-16T15:22:01.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.564+0000 7fe905400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9000d1430 con 0x7fe9000c6be0 2024-01-16T15:22:01.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.564+0000 7fe905400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9000b4370 con 0x7fe900160890 2024-01-16T15:22:01.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.564+0000 7fe8ffe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe900160890 0x7fe9000d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:01.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.564+0000 7fe8ff400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fe900160c70 0x7fe9000d1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:53744/0 (socket says 172.21.15.141:53744) 2024-01-16T15:22:01.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.564+0000 7fe8ff400640 1 -- 172.21.15.141:0/3563125327 learned_addr learned my addr 172.21.15.141:0/3563125327 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:01.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.565+0000 7fe8ffe00640 1 -- 172.21.15.141:0/3563125327 >> v1:172.21.15.188:6789/0 conn(0x7fe9000c6be0 legacy=0x7fe9000cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:01.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.565+0000 7fe8ffe00640 1 -- 172.21.15.141:0/3563125327 >> v1:172.21.15.141:6789/0 conn(0x7fe900160c70 legacy=0x7fe9000d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:01.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.565+0000 7fe8ffe00640 1 -- 172.21.15.141:0/3563125327 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000ce7c0 con 0x7fe900160890 2024-01-16T15:22:01.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.565+0000 7fe8fea00640 1 -- 172.21.15.141:0/3563125327 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334661665 0 0) 0x7fe9000cf6a0 con 0x7fe900160c70 2024-01-16T15:22:01.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.566+0000 7fe8ffe00640 1 --2- 172.21.15.141:0/3563125327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe900160890 0x7fe9000d3bb0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00097a0 tx=0x7fe8f002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d49e563c7c0e7ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:01.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.566+0000 7fe8fea00640 1 -- 172.21.15.141:0/3563125327 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f002f050 con 0x7fe900160890 2024-01-16T15:22:01.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.566+0000 7fe8fea00640 1 -- 172.21.15.141:0/3563125327 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f0031700 con 0x7fe900160890 2024-01-16T15:22:01.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.566+0000 7fe8fea00640 1 -- 172.21.15.141:0/3563125327 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0031a90 con 0x7fe900160890 2024-01-16T15:22:01.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.566+0000 7fe905400640 1 -- 172.21.15.141:0/3563125327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe900160890 msgr2=0x7fe9000d3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:01.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.566+0000 7fe905400640 1 --2- 172.21.15.141:0/3563125327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe900160890 0x7fe9000d3bb0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00097a0 tx=0x7fe8f002b7a0 comp rx=0 tx=0).stop 2024-01-16T15:22:01.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.567+0000 7fe905400640 1 -- 172.21.15.141:0/3563125327 shutdown_connections 2024-01-16T15:22:01.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.567+0000 7fe905400640 1 --2- 172.21.15.141:0/3563125327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe900160890 0x7fe9000d3bb0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:01.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.567+0000 7fe905400640 1 -- 172.21.15.141:0/3563125327 >> 172.21.15.141:0/3563125327 conn(0x7fe9000d3090 msgr2=0x7fe9000d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:01.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.567+0000 7fe905400640 1 -- 172.21.15.141:0/3563125327 shutdown_connections 2024-01-16T15:22:01.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.567+0000 7fe905400640 1 -- 172.21.15.141:0/3563125327 wait complete. 2024-01-16T15:22:01.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.568+0000 7fe905400640 1 Processor -- start 2024-01-16T15:22:01.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.569+0000 7fe905400640 1 -- start start 2024-01-16T15:22:01.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.569+0000 7fe905400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9000c6be0 0x7fe9000b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:01.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.569+0000 7fe905400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe900160890 0x7fe9000b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe905400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe900160c70 0x7fe9000ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe905400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90015f9e0 con 0x7fe9000c6be0 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe905400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90015d7c0 con 0x7fe900160c70 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe905400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9000b87a0 con 0x7fe900160890 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9000c6be0 0x7fe9000b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9000c6be0 0x7fe9000b72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60914/0 (socket says 172.21.15.141:60914) 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 -- 172.21.15.141:0/848338288 learned_addr learned my addr 172.21.15.141:0/848338288 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 -- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe900160890 msgr2=0x7fe9000b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ff400640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe900160890 0x7fe9000b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe900160890 0x7fe9000b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 -- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe900160c70 msgr2=0x7fe9000ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe904a00640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe900160c70 0x7fe9000ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe900160c70 0x7fe9000ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:01.566 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe8ffe00640 1 -- 172.21.15.141:0/848338288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0009400 con 0x7fe9000c6be0 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.570+0000 7fe904a00640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe900160c70 0x7fe9000ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.571+0000 7fe8ffe00640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9000c6be0 0x7fe9000b72a0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00097a0 tx=0x7fe8f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.571+0000 7fe8f7e00640 1 -- 172.21.15.141:0/848338288 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0033c30 con 0x7fe9000c6be0 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.571+0000 7fe8f7e00640 1 -- 172.21.15.141:0/848338288 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f0033d90 con 0x7fe9000c6be0 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.571+0000 7fe905400640 1 -- 172.21.15.141:0/848338288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9000cb450 con 0x7fe9000c6be0 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.571+0000 7fe8f7e00640 1 -- 172.21.15.141:0/848338288 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f003d6b0 con 0x7fe9000c6be0 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.571+0000 7fe905400640 1 -- 172.21.15.141:0/848338288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90015d200 con 0x7fe9000c6be0 2024-01-16T15:22:01.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.572+0000 7fe905400640 1 -- 172.21.15.141:0/848338288 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe900067450 con 0x7fe9000c6be0 2024-01-16T15:22:01.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.573+0000 7fe8f7e00640 1 -- 172.21.15.141:0/848338288 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fe8f003a040 con 0x7fe9000c6be0 2024-01-16T15:22:01.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.574+0000 7fe8f7e00640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8d806c820 0x7fe8d806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:01.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.574+0000 7fe8f7e00640 1 -- 172.21.15.141:0/848338288 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe8f00aa690 con 0x7fe9000c6be0 2024-01-16T15:22:01.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.574+0000 7fe8ff400640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8d806c820 0x7fe8d806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:01.570 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.576+0000 7fe8ff400640 1 --2- 172.21.15.141:0/848338288 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe8d806c820 0x7fe8d806ece0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe9000c6590 tx=0x7fe8e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:01.572 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.578+0000 7fe8f7e00640 1 -- 172.21.15.141:0/848338288 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8f0076fb0 con 0x7fe9000c6be0 2024-01-16T15:22:01.795 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.801+0000 7fe905400640 1 -- 172.21.15.141:0/848338288 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9000ced80 con 0x7fe8d806c820 2024-01-16T15:22:01.808 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:01.813+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:01.809 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:01.814+0000 7fe8f7e00640 1 -- 172.21.15.141:0/848338288 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe8e800d970 con 0x7fe8d806c820 2024-01-16T15:22:01.809 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:01.809 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:03.825 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:04.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.054+0000 7fcefe400640 1 Processor -- start 2024-01-16T15:22:04.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.054+0000 7fcefe400640 1 -- start start 2024-01-16T15:22:04.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.056+0000 7fcefe400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 0x7fcef815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:04.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.056+0000 7fcefe400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcef80b3ae0 0x7fcef806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:04.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.056+0000 7fcefe400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef8070c40 con 0x7fcef806d120 2024-01-16T15:22:04.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.056+0000 7fcefe400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef8071450 con 0x7fcef80b3ae0 2024-01-16T15:22:04.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.056+0000 7fcefe400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef806c470 con 0x7fcef80ae5a0 2024-01-16T15:22:04.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef6a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcef80b3ae0 0x7fcef806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:04.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef6a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fcef80b3ae0 0x7fcef806cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:58584/0 (socket says 172.21.15.141:58584) 2024-01-16T15:22:04.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef6a00640 1 -- 172.21.15.141:0/1872892953 learned_addr learned my addr 172.21.15.141:0/1872892953 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:04.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef7400640 1 --2- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 0x7fcef815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:04.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef6a00640 1 -- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 msgr2=0x7fcef815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:04.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef6a00640 1 --2- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 0x7fcef815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:04.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef6a00640 1 -- 172.21.15.141:0/1872892953 >> v1:172.21.15.188:6789/0 conn(0x7fcef806d120 legacy=0x7fcef80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:04.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.057+0000 7fcef6a00640 1 -- 172.21.15.141:0/1872892953 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef806c750 con 0x7fcef80b3ae0 2024-01-16T15:22:04.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.059+0000 7fcef6a00640 1 --2- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 0x7fcef806cbe0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009870 tx=0x7fcee002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f71bace2977bb094 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:04.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.059+0000 7fcef6000640 1 -- 172.21.15.141:0/1872892953 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0030840 con 0x7fcef80b3ae0 2024-01-16T15:22:04.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.059+0000 7fcef6000640 1 -- 172.21.15.141:0/1872892953 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee00309a0 con 0x7fcef80b3ae0 2024-01-16T15:22:04.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.059+0000 7fcef6000640 1 -- 172.21.15.141:0/1872892953 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0030d30 con 0x7fcef80b3ae0 2024-01-16T15:22:04.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 -- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 msgr2=0x7fcef806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:04.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 --2- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 0x7fcef806cbe0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009870 tx=0x7fcee002b980 comp rx=0 tx=0).stop 2024-01-16T15:22:04.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 -- 172.21.15.141:0/1872892953 shutdown_connections 2024-01-16T15:22:04.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 --2- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 0x7fcef806cbe0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:04.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 --2- 172.21.15.141:0/1872892953 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 0x7fcef815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:04.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 -- 172.21.15.141:0/1872892953 >> 172.21.15.141:0/1872892953 conn(0x7fcef806a800 msgr2=0x7fcef806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:04.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 -- 172.21.15.141:0/1872892953 shutdown_connections 2024-01-16T15:22:04.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.060+0000 7fcefe400640 1 -- 172.21.15.141:0/1872892953 wait complete. 2024-01-16T15:22:04.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.062+0000 7fcefe400640 1 Processor -- start 2024-01-16T15:22:04.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.062+0000 7fcefe400640 1 -- start start 2024-01-16T15:22:04.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcefe400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcef806d120 0x7fcef81f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:04.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcefe400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 0x7fcef8149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:04.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcefe400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 0x7fcef81f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:04.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcefe400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef81481e0 con 0x7fcef806d120 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcefe400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8147e90 con 0x7fcef80b3ae0 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcefe400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef814a460 con 0x7fcef80ae5a0 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcef7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcef806d120 0x7fcef81f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcef7400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcef806d120 0x7fcef81f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46382/0 (socket says 172.21.15.141:46382) 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.063+0000 7fcef7400640 1 -- 172.21.15.141:0/3982062588 learned_addr learned my addr 172.21.15.141:0/3982062588 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef6a00640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 0x7fcef8149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef7e00640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 0x7fcef81f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef7400640 1 -- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 msgr2=0x7fcef8149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef7400640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcef80ae5a0 0x7fcef8149f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef7400640 1 -- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 msgr2=0x7fcef81f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef7400640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcef80b3ae0 0x7fcef81f2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef7400640 1 -- 172.21.15.141:0/3982062588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00094d0 con 0x7fcef806d120 2024-01-16T15:22:04.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fcef7400640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcef806d120 0x7fcef81f3630 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fcee800e9b0 tx=0x7fcee800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:04.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.064+0000 7fceef400640 1 -- 172.21.15.141:0/3982062588 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8009670 con 0x7fcef806d120 2024-01-16T15:22:04.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.065+0000 7fceef400640 1 -- 172.21.15.141:0/3982062588 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee800c9a0 con 0x7fcef806d120 2024-01-16T15:22:04.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.065+0000 7fcefe400640 1 -- 172.21.15.141:0/3982062588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef814a730 con 0x7fcef806d120 2024-01-16T15:22:04.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.065+0000 7fceef400640 1 -- 172.21.15.141:0/3982062588 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee800cd50 con 0x7fcef806d120 2024-01-16T15:22:04.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.065+0000 7fcefe400640 1 -- 172.21.15.141:0/3982062588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef814acf0 con 0x7fcef806d120 2024-01-16T15:22:04.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.066+0000 7fcefe400640 1 -- 172.21.15.141:0/3982062588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef8148450 con 0x7fcef806d120 2024-01-16T15:22:04.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.067+0000 7fceef400640 1 -- 172.21.15.141:0/3982062588 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fcee8013470 con 0x7fcef806d120 2024-01-16T15:22:04.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.068+0000 7fceef400640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fced006c820 0x7fced006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:04.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.068+0000 7fceef400640 1 -- 172.21.15.141:0/3982062588 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcee8088b20 con 0x7fcef806d120 2024-01-16T15:22:04.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.069+0000 7fcef6a00640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fced006c820 0x7fced006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:04.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.070+0000 7fcef6a00640 1 --2- 172.21.15.141:0/3982062588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fced006c820 0x7fced006ece0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcee0036000 tx=0x7fcee0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:04.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.073+0000 7fceef400640 1 -- 172.21.15.141:0/3982062588 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcee80553e0 con 0x7fcef806d120 2024-01-16T15:22:04.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.299+0000 7fcefe400640 1 -- 172.21.15.141:0/3982062588 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcef8070550 con 0x7fced006c820 2024-01-16T15:22:04.307 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:04.312+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:04.307 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:04.313+0000 7fceef400640 1 -- 172.21.15.141:0/3982062588 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcee0044470 con 0x7fced006c820 2024-01-16T15:22:04.307 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:04.308 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:06.320 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.555+0000 7f41b7a00640 1 Processor -- start 2024-01-16T15:22:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.555+0000 7f41b7a00640 1 -- start start 2024-01-16T15:22:06.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.557+0000 7f41b7a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 0x7f41b015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.557+0000 7f41b7a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b006c470 con 0x7f41b00cf9a0 2024-01-16T15:22:06.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.557+0000 7f41b7a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b006c750 con 0x7f41b013e360 2024-01-16T15:22:06.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.557+0000 7f41b7a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b006ca30 con 0x7f41b013df80 2024-01-16T15:22:06.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.557+0000 7f41b5a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f41b00cf9a0 0x7f41b00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47358/0 (socket says 172.21.15.141:47358) 2024-01-16T15:22:06.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.557+0000 7f41b5a00640 1 -- 172.21.15.141:0/1670638537 learned_addr learned my addr 172.21.15.141:0/1670638537 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.557+0000 7f41b5000640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 0x7f41b015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964681274 0 0) 0x7f41b006c470 con 0x7f41b00cf9a0 2024-01-16T15:22:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4190003520 con 0x7f41b00cf9a0 2024-01-16T15:22:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554454313 0 0) 0x7f4190003520 con 0x7f41b00cf9a0 2024-01-16T15:22:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 msgr2=0x7f41b015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 0x7f41b015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 >> v1:172.21.15.188:6789/0 conn(0x7f41b013e360 legacy=0x7f41b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b006cd10 con 0x7f41b00cf9a0 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f41a80019f0 con 0x7f41b00cf9a0 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 >> v1:172.21.15.141:6789/0 conn(0x7f41b00cf9a0 legacy=0x7f41b00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41900039a0 0x7f4190003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 0x7f41900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4190004d00 0x7f4190005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b006ca30 con 0x7f41900039a0 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419000ba60 con 0x7f4190004d00 2024-01-16T15:22:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419000bc20 con 0x7f41b013df80 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.558+0000 7f41b5000640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41900039a0 0x7f4190003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.559+0000 7f41a7c00640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 0x7f41900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.559+0000 7f41b5000640 1 -- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 msgr2=0x7f41900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.559+0000 7f41b5000640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 0x7f41900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.559+0000 7f41b5000640 1 -- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4190004d00 msgr2=0x7f4190005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.559+0000 7f41b5000640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4190004d00 0x7f4190005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.559+0000 7f41b5000640 1 -- 172.21.15.141:0/1670638537 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4190003520 con 0x7f41900039a0 2024-01-16T15:22:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.559+0000 7f41a7c00640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b013df80 0x7f41900047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.560+0000 7f41b5000640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41900039a0 0x7f4190003dc0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009a90 tx=0x7f41900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adef7523d04cdb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.560+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a002c680 con 0x7f41900039a0 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.560+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a002c7e0 con 0x7f41900039a0 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.560+0000 7f41afe00640 1 -- 172.21.15.141:0/1670638537 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a002cb70 con 0x7f41900039a0 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.561+0000 7f41b7a00640 1 -- 172.21.15.141:0/1670638537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41900039a0 msgr2=0x7f4190003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.561+0000 7f41b7a00640 1 --2- 172.21.15.141:0/1670638537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41900039a0 0x7f4190003dc0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009a90 tx=0x7f41900057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.561+0000 7f41b5a00640 1 -- 172.21.15.141:0/1670638537 reap_dead start 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.561+0000 7f41b7a00640 1 -- 172.21.15.141:0/1670638537 shutdown_connections 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.561+0000 7f41b7a00640 1 -- 172.21.15.141:0/1670638537 >> 172.21.15.141:0/1670638537 conn(0x7f41b006a800 msgr2=0x7f41b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.561+0000 7f41b7a00640 1 -- 172.21.15.141:0/1670638537 shutdown_connections 2024-01-16T15:22:06.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.561+0000 7f41b7a00640 1 -- 172.21.15.141:0/1670638537 wait complete. 2024-01-16T15:22:06.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.563+0000 7f41b7a00640 1 Processor -- start 2024-01-16T15:22:06.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.563+0000 7f41b7a00640 1 -- start start 2024-01-16T15:22:06.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41b7a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41900039a0 0x7f41b0141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41b7a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b0152b80 0x7f41b00b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41b7a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b013ef40 0x7f41b013f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41b7a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b013eab0 con 0x7f41b0152b80 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41b7a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b0142340 con 0x7f41b013ef40 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41b7a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b0142620 con 0x7f41900039a0 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41a7c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b0152b80 0x7f41b00b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41a7c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b0152b80 0x7f41b00b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46414/0 (socket says 172.21.15.141:46414) 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.564+0000 7f41a7c00640 1 -- 172.21.15.141:0/3623329129 learned_addr learned my addr 172.21.15.141:0/3623329129 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41b5000640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41900039a0 0x7f41b0141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41b5a00640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b013ef40 0x7f41b013f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41a7c00640 1 -- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41900039a0 msgr2=0x7f41b0141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41a7c00640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41900039a0 0x7f41b0141b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41a7c00640 1 -- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b013ef40 msgr2=0x7f41b013f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41a7c00640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b013ef40 0x7f41b013f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:06.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41a7c00640 1 -- 172.21.15.141:0/3623329129 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b0142900 con 0x7f41b0152b80 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41b5a00640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b013ef40 0x7f41b013f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41a7c00640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b0152b80 0x7f41b00b0000 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f419800d7a0 tx=0x7f419800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.565+0000 7f41ad600640 1 -- 172.21.15.141:0/3623329129 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4198017070 con 0x7f41b0152b80 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.566+0000 7f41ad600640 1 -- 172.21.15.141:0/3623329129 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4198005070 con 0x7f41b0152b80 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.566+0000 7f41ad600640 1 -- 172.21.15.141:0/3623329129 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4198005420 con 0x7f41b0152b80 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.566+0000 7f41b7a00640 1 -- 172.21.15.141:0/3623329129 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b013e740 con 0x7f41b0152b80 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.566+0000 7f41b7a00640 1 -- 172.21.15.141:0/3623329129 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b00b0540 con 0x7f41b0152b80 2024-01-16T15:22:06.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.567+0000 7f41b7a00640 1 -- 172.21.15.141:0/3623329129 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b013ec60 con 0x7f41b0152b80 2024-01-16T15:22:06.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.568+0000 7f41ad600640 1 -- 172.21.15.141:0/3623329129 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f4198007030 con 0x7f41b0152b80 2024-01-16T15:22:06.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.569+0000 7f41ad600640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f418806c820 0x7f418806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:06.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.569+0000 7f41ad600640 1 -- 172.21.15.141:0/3623329129 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f41980885e0 con 0x7f41b0152b80 2024-01-16T15:22:06.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.570+0000 7f41b5000640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f418806c820 0x7f418806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:06.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.571+0000 7f41b5000640 1 --2- 172.21.15.141:0/3623329129 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f418806c820 0x7f418806ece0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f41a0036000 tx=0x7f41a0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:06.568 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.574+0000 7f41ad600640 1 -- 172.21.15.141:0/3623329129 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41b013ec60 con 0x7f41b0152b80 2024-01-16T15:22:06.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.802+0000 7f41b7a00640 1 -- 172.21.15.141:0/3623329129 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41b015bee0 con 0x7f418806c820 2024-01-16T15:22:06.810 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:06.815+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:06.810 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:06.816+0000 7f41ad600640 1 -- 172.21.15.141:0/3623329129 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f41a0004020 con 0x7f418806c820 2024-01-16T15:22:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:06.811 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:08.827 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:09.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.056+0000 7f5233a00640 1 Processor -- start 2024-01-16T15:22:09.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.057+0000 7f5233a00640 1 -- start start 2024-01-16T15:22:09.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.058+0000 7f5233a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f522c0ae5a0 0x7f522c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:09.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.058+0000 7f5233a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 0x7f522c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:09.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.058+0000 7f5233a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f522c06d120 0x7f522c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:09.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.058+0000 7f5233a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c071450 con 0x7f522c06d120 2024-01-16T15:22:09.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5233a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c070c40 con 0x7f522c0ae5a0 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5233a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c06c470 con 0x7f522c0b3ae0 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f522c06d120 0x7f522c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f522c06d120 0x7f522c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46418/0 (socket says 172.21.15.141:46418) 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 -- 172.21.15.141:0/1235704376 learned_addr learned my addr 172.21.15.141:0/1235704376 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f522be00640 1 --2- 172.21.15.141:0/1235704376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 0x7f522c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231000640 1 --2- 172.21.15.141:0/1235704376 >> v2:172.21.15.188:3300/0 conn(0x7f522c0ae5a0 0x7f522c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 -- 172.21.15.141:0/1235704376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 msgr2=0x7f522c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 --2- 172.21.15.141:0/1235704376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 0x7f522c06cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 -- 172.21.15.141:0/1235704376 >> v2:172.21.15.188:3300/0 conn(0x7f522c0ae5a0 msgr2=0x7f522c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 --2- 172.21.15.141:0/1235704376 >> v2:172.21.15.188:3300/0 conn(0x7f522c0ae5a0 0x7f522c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231a00640 1 -- 172.21.15.141:0/1235704376 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c06c750 con 0x7f522c06d120 2024-01-16T15:22:09.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.059+0000 7f5231000640 1 --2- 172.21.15.141:0/1235704376 >> v2:172.21.15.188:3300/0 conn(0x7f522c0ae5a0 0x7f522c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.061+0000 7f5231a00640 1 --2- 172.21.15.141:0/1235704376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 0x7f522c0cf290 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f522c0cfe20 tx=0x7f522402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db93a42185ae07e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.061+0000 7f522b400640 1 -- 172.21.15.141:0/1235704376 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522402ca80 con 0x7f522c06d120 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.061+0000 7f522b400640 1 -- 172.21.15.141:0/1235704376 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522402cbe0 con 0x7f522c06d120 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.061+0000 7f522b400640 1 -- 172.21.15.141:0/1235704376 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52240343f0 con 0x7f522c06d120 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.061+0000 7f5233a00640 1 -- 172.21.15.141:0/1235704376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 msgr2=0x7f522c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.061+0000 7f5233a00640 1 --2- 172.21.15.141:0/1235704376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 0x7f522c0cf290 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f522c0cfe20 tx=0x7f522402bee0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.062+0000 7f5233a00640 1 -- 172.21.15.141:0/1235704376 shutdown_connections 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.062+0000 7f5233a00640 1 --2- 172.21.15.141:0/1235704376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 0x7f522c0cf290 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.062+0000 7f5233a00640 1 --2- 172.21.15.141:0/1235704376 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 0x7f522c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.062+0000 7f5233a00640 1 --2- 172.21.15.141:0/1235704376 >> v2:172.21.15.188:3300/0 conn(0x7f522c0ae5a0 0x7f522c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.062+0000 7f5233a00640 1 -- 172.21.15.141:0/1235704376 >> 172.21.15.141:0/1235704376 conn(0x7f522c06a800 msgr2=0x7f522c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:09.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.062+0000 7f5233a00640 1 -- 172.21.15.141:0/1235704376 shutdown_connections 2024-01-16T15:22:09.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.062+0000 7f5233a00640 1 -- 172.21.15.141:0/1235704376 wait complete. 2024-01-16T15:22:09.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.063+0000 7f5233a00640 1 Processor -- start 2024-01-16T15:22:09.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.064+0000 7f5233a00640 1 -- start start 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.064+0000 7f5233a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 0x7f522c1595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.064+0000 7f5233a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f522c0ae5a0 0x7f522c155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.064+0000 7f5233a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 0x7f522c0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.064+0000 7f5233a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c159440 con 0x7f522c06d120 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.064+0000 7f5233a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c0cc4d0 con 0x7f522c0ae5a0 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.064+0000 7f5233a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c0cc700 con 0x7f522c0b3ae0 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 0x7f522c1595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 0x7f522c1595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46434/0 (socket says 172.21.15.141:46434) 2024-01-16T15:22:09.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 -- 172.21.15.141:0/783497788 learned_addr learned my addr 172.21.15.141:0/783497788 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f522be00640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f522c0ae5a0 0x7f522c155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231a00640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 0x7f522c0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 -- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 msgr2=0x7f522c0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f522c0b3ae0 0x7f522c0cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 -- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f522c0ae5a0 msgr2=0x7f522c155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f522c0ae5a0 0x7f522c155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.065+0000 7f5231000640 1 -- 172.21.15.141:0/783497788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5224009b30 con 0x7f522c06d120 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.066+0000 7f5231000640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f522c06d120 0x7f522c1595d0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f521c00c7c0 tx=0x7f521c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.066+0000 7f5228c00640 1 -- 172.21.15.141:0/783497788 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c007a30 con 0x7f522c06d120 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.066+0000 7f5228c00640 1 -- 172.21.15.141:0/783497788 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c007b90 con 0x7f522c06d120 2024-01-16T15:22:09.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.066+0000 7f5228c00640 1 -- 172.21.15.141:0/783497788 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c00ce70 con 0x7f522c06d120 2024-01-16T15:22:09.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.066+0000 7f5233a00640 1 -- 172.21.15.141:0/783497788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c0cc9e0 con 0x7f522c06d120 2024-01-16T15:22:09.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.066+0000 7f5233a00640 1 -- 172.21.15.141:0/783497788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c0ccfa0 con 0x7f522c06d120 2024-01-16T15:22:09.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.067+0000 7f5233a00640 1 -- 172.21.15.141:0/783497788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522c0675d0 con 0x7f522c06d120 2024-01-16T15:22:09.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.068+0000 7f5228c00640 1 -- 172.21.15.141:0/783497788 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f521c0113f0 con 0x7f522c06d120 2024-01-16T15:22:09.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.069+0000 7f5228c00640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f520406c820 0x7f520406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:09.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.070+0000 7f5228c00640 1 -- 172.21.15.141:0/783497788 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f521c088da0 con 0x7f522c06d120 2024-01-16T15:22:09.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.070+0000 7f522be00640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f520406c820 0x7f520406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:09.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.071+0000 7f522be00640 1 --2- 172.21.15.141:0/783497788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f520406c820 0x7f520406ece0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5214006fd0 tx=0x7f5214008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:09.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.074+0000 7f5228c00640 1 -- 172.21.15.141:0/783497788 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f521c0552c0 con 0x7f522c06d120 2024-01-16T15:22:09.295 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.301+0000 7f5233a00640 1 -- 172.21.15.141:0/783497788 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f522c070550 con 0x7f520406c820 2024-01-16T15:22:09.309 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:09.314+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:09.309 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:09.314+0000 7f5228c00640 1 -- 172.21.15.141:0/783497788 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5214009480 con 0x7f520406c820 2024-01-16T15:22:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:09.310 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:09.611 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:09.619 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:11.325 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:11.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.559+0000 7f2574a00640 1 Processor -- start 2024-01-16T15:22:11.553 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.559+0000 7f2574a00640 1 -- start start 2024-01-16T15:22:11.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.560+0000 7f2574a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f256c15e3b0 0x7f256c164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:11.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2574a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256c0b1ee0 con 0x7f256c0cf9a0 2024-01-16T15:22:11.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2574a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256c0b2780 con 0x7f256c143ae0 2024-01-16T15:22:11.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2574a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256c15e990 con 0x7f256c15e3b0 2024-01-16T15:22:11.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2572a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f256c15e3b0 0x7f256c164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:11.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2571600640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f256c0cf9a0 0x7f256c15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47374/0 (socket says 172.21.15.141:47374) 2024-01-16T15:22:11.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2571600640 1 -- 172.21.15.141:0/2604093080 learned_addr learned my addr 172.21.15.141:0/2604093080 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:11.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2572a00640 1 -- 172.21.15.141:0/2604093080 >> v1:172.21.15.188:6789/0 conn(0x7f256c143ae0 legacy=0x7f256c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:11.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2572a00640 1 -- 172.21.15.141:0/2604093080 >> v1:172.21.15.141:6789/0 conn(0x7f256c0cf9a0 legacy=0x7f256c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:11.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2572a00640 1 -- 172.21.15.141:0/2604093080 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c0b2130 con 0x7f256c15e3b0 2024-01-16T15:22:11.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.561+0000 7f2570c00640 1 -- 172.21.15.141:0/2604093080 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659349901 0 0) 0x7f256c0b1ee0 con 0x7f256c0cf9a0 2024-01-16T15:22:11.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.562+0000 7f2572a00640 1 --2- 172.21.15.141:0/2604093080 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c15e3b0 0x7f256c164de0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f25640097a0 tx=0x7f256402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ec205db6d2e244 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:11.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2570c00640 1 -- 172.21.15.141:0/2604093080 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564031830 con 0x7f256c15e3b0 2024-01-16T15:22:11.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2570c00640 1 -- 172.21.15.141:0/2604093080 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2564031990 con 0x7f256c15e3b0 2024-01-16T15:22:11.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2570c00640 1 -- 172.21.15.141:0/2604093080 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564031d20 con 0x7f256c15e3b0 2024-01-16T15:22:11.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2574a00640 1 -- 172.21.15.141:0/2604093080 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c15e3b0 msgr2=0x7f256c164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:11.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2574a00640 1 --2- 172.21.15.141:0/2604093080 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c15e3b0 0x7f256c164de0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f25640097a0 tx=0x7f256402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:22:11.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2574a00640 1 -- 172.21.15.141:0/2604093080 shutdown_connections 2024-01-16T15:22:11.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2574a00640 1 --2- 172.21.15.141:0/2604093080 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c15e3b0 0x7f256c164de0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:11.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.563+0000 7f2574a00640 1 -- 172.21.15.141:0/2604093080 >> 172.21.15.141:0/2604093080 conn(0x7f256c06a800 msgr2=0x7f256c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:11.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.564+0000 7f2574a00640 1 -- 172.21.15.141:0/2604093080 shutdown_connections 2024-01-16T15:22:11.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.564+0000 7f2574a00640 1 -- 172.21.15.141:0/2604093080 wait complete. 2024-01-16T15:22:11.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.565+0000 7f2574a00640 1 Processor -- start 2024-01-16T15:22:11.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.565+0000 7f2574a00640 1 -- start start 2024-01-16T15:22:11.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.566+0000 7f2574a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0cf9a0 0x7f256c1f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:11.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.566+0000 7f2574a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c143ae0 0x7f256c0cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:11.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.566+0000 7f2574a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f256c15e3b0 0x7f256c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:11.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.566+0000 7f2574a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c1f1c70 con 0x7f256c143ae0 2024-01-16T15:22:11.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.566+0000 7f2574a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0ce9e0 con 0x7f256c0cf9a0 2024-01-16T15:22:11.561 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.566+0000 7f2574a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0cddd0 con 0x7f256c15e3b0 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.566+0000 7f2571600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c143ae0 0x7f256c0cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c143ae0 0x7f256c0cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46458/0 (socket says 172.21.15.141:46458) 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 -- 172.21.15.141:0/1491242422 learned_addr learned my addr 172.21.15.141:0/1491242422 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 -- 172.21.15.141:0/1491242422 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f256c15e3b0 msgr2=0x7f256c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 --2- 172.21.15.141:0/1491242422 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f256c15e3b0 0x7f256c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 -- 172.21.15.141:0/1491242422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0cf9a0 msgr2=0x7f256c1f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 --2- 172.21.15.141:0/1491242422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0cf9a0 0x7f256c1f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:11.562 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 -- 172.21.15.141:0/1491242422 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2564009400 con 0x7f256c143ae0 2024-01-16T15:22:11.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f2571600640 1 --2- 172.21.15.141:0/1491242422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f256c143ae0 0x7f256c0cd370 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f255c00c890 tx=0x7f255c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:11.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f256a000640 1 -- 172.21.15.141:0/1491242422 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c00a640 con 0x7f256c143ae0 2024-01-16T15:22:11.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.567+0000 7f256a000640 1 -- 172.21.15.141:0/1491242422 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c00a7c0 con 0x7f256c143ae0 2024-01-16T15:22:11.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.568+0000 7f256a000640 1 -- 172.21.15.141:0/1491242422 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c00ab70 con 0x7f256c143ae0 2024-01-16T15:22:11.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.568+0000 7f2574a00640 1 -- 172.21.15.141:0/1491242422 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256c0ce0b0 con 0x7f256c143ae0 2024-01-16T15:22:11.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.568+0000 7f2574a00640 1 -- 172.21.15.141:0/1491242422 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c0ce670 con 0x7f256c143ae0 2024-01-16T15:22:11.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.569+0000 7f2574a00640 1 -- 172.21.15.141:0/1491242422 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c06c3f0 con 0x7f256c143ae0 2024-01-16T15:22:11.564 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.570+0000 7f256a000640 1 -- 172.21.15.141:0/1491242422 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f255c00acd0 con 0x7f256c143ae0 2024-01-16T15:22:11.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.571+0000 7f256a000640 1 --2- 172.21.15.141:0/1491242422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2544070c30 0x7f25440730f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:11.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.571+0000 7f256a000640 1 -- 172.21.15.141:0/1491242422 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f255c0881f0 con 0x7f256c143ae0 2024-01-16T15:22:11.565 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.571+0000 7f2572000640 1 --2- 172.21.15.141:0/1491242422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2544070c30 0x7f25440730f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:11.567 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.573+0000 7f2572000640 1 --2- 172.21.15.141:0/1491242422 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2544070c30 0x7f25440730f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2554003cc0 tx=0x7f25540079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:11.569 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.575+0000 7f256a000640 1 -- 172.21.15.141:0/1491242422 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f255c054710 con 0x7f256c143ae0 2024-01-16T15:22:11.792 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.797+0000 7f2574a00640 1 -- 172.21.15.141:0/1491242422 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f256c0675d0 con 0x7f2544070c30 2024-01-16T15:22:11.804 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:11.809+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:11.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:11.810+0000 7f256a000640 1 -- 172.21.15.141:0/1491242422 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f256c0675d0 con 0x7f2544070c30 2024-01-16T15:22:11.805 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:11.805 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:13.823 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:14.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.057+0000 7f65e4a00640 1 Processor -- start 2024-01-16T15:22:14.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.057+0000 7f65e4a00640 1 -- start start 2024-01-16T15:22:14.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.059+0000 7f65e4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 0x7f65e015a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.059+0000 7f65e4a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f65e0071290 0x7f65e006a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.059+0000 7f65e4a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00cb970 con 0x7f65e006c900 2024-01-16T15:22:14.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.059+0000 7f65e4a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e015aca0 con 0x7f65e0071290 2024-01-16T15:22:14.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.059+0000 7f65e4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00cbbc0 con 0x7f65e01571a0 2024-01-16T15:22:14.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.059+0000 7f65dde00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f65e006c900 0x7f65e0070b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43916/0 (socket says 172.21.15.141:43916) 2024-01-16T15:22:14.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.059+0000 7f65dde00640 1 -- 172.21.15.141:0/489180890 learned_addr learned my addr 172.21.15.141:0/489180890 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:14.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65df200640 1 --2- 172.21.15.141:0/489180890 >> v2:172.21.15.188:3300/0 conn(0x7f65e0071290 0x7f65e006a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65de800640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 0x7f65e015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435206354 0 0) 0x7f65e00cb970 con 0x7f65e006c900 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c0005730 con 0x7f65e006c900 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3456587374 0 0) 0x7f65c0005730 con 0x7f65e006c900 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 msgr2=0x7f65e015a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 0x7f65e015a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 >> v2:172.21.15.188:3300/0 conn(0x7f65e0071290 msgr2=0x7f65e006a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 --2- 172.21.15.141:0/489180890 >> v2:172.21.15.188:3300/0 conn(0x7f65e0071290 0x7f65e006a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00cbea0 con 0x7f65e006c900 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f65c8001b70 con 0x7f65e006c900 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 >> v1:172.21.15.141:6789/0 conn(0x7f65e006c900 legacy=0x7f65e0070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 0x7f65c0005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006100 0x7f65c00069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65c0006f10 0x7f65c0007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e015aca0 con 0x7f65c0006f10 2024-01-16T15:22:14.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00cbbc0 con 0x7f65c0006100 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.060+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c000db90 con 0x7f65e01571a0 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65df200640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65c0006f10 0x7f65c0007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65dde00640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006100 0x7f65c00069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65de800640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 0x7f65c0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65df200640 1 -- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 msgr2=0x7f65c0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65df200640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01571a0 0x7f65c0005b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65df200640 1 -- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006100 msgr2=0x7f65c00069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65df200640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006100 0x7f65c00069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:14.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.061+0000 7f65df200640 1 -- 172.21.15.141:0/489180890 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c0005730 con 0x7f65c0006f10 2024-01-16T15:22:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.062+0000 7f65df200640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65c0006f10 0x7f65c0007390 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f65e00cc1a0 tx=0x7f65c0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab51b97d578d93b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d802c6b0 con 0x7f65c0006f10 2024-01-16T15:22:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d802c810 con 0x7f65c0006f10 2024-01-16T15:22:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65dd400640 1 -- 172.21.15.141:0/489180890 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d802cbc0 con 0x7f65c0006f10 2024-01-16T15:22:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65e4a00640 1 -- 172.21.15.141:0/489180890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65c0006f10 msgr2=0x7f65c0007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:14.059 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65e4a00640 1 --2- 172.21.15.141:0/489180890 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65c0006f10 0x7f65c0007390 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f65e00cc1a0 tx=0x7f65c0007a00 comp rx=0 tx=0).stop 2024-01-16T15:22:14.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65df200640 1 -- 172.21.15.141:0/489180890 reap_dead start 2024-01-16T15:22:14.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65e4a00640 1 -- 172.21.15.141:0/489180890 shutdown_connections 2024-01-16T15:22:14.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65e4a00640 1 -- 172.21.15.141:0/489180890 >> 172.21.15.141:0/489180890 conn(0x7f65e00cc1d0 msgr2=0x7f65e00cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:14.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65e4a00640 1 -- 172.21.15.141:0/489180890 shutdown_connections 2024-01-16T15:22:14.060 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.063+0000 7f65e4a00640 1 -- 172.21.15.141:0/489180890 wait complete. 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.065+0000 7f65e4a00640 1 Processor -- start 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.065+0000 7f65e4a00640 1 -- start start 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65e4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006f10 0x7f65e0150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65e4a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01e1d50 0x7f65e01e21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65e4a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65e00ada70 0x7f65e00adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65e4a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01e2710 con 0x7f65e00ada70 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65e4a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0151130 con 0x7f65c0006f10 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65e4a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0155620 con 0x7f65e01e1d50 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65df200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65e00ada70 0x7f65e00adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65df200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65e00ada70 0x7f65e00adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41894/0 (socket says 172.21.15.141:41894) 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.066+0000 7f65df200640 1 -- 172.21.15.141:0/428756303 learned_addr learned my addr 172.21.15.141:0/428756303 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65de800640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006f10 0x7f65e0150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65dde00640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01e1d50 0x7f65e01e21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.062 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65df200640 1 -- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01e1d50 msgr2=0x7f65e01e21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:14.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65df200640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f65e01e1d50 0x7f65e01e21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:14.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65df200640 1 -- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006f10 msgr2=0x7f65e0150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:14.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65df200640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65c0006f10 0x7f65e0150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:14.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65df200640 1 -- 172.21.15.141:0/428756303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e01f2d50 con 0x7f65e00ada70 2024-01-16T15:22:14.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.067+0000 7f65df200640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f65e00ada70 0x7f65e00adf00 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f65d8037040 tx=0x7f65d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:14.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.068+0000 7f65d6a00640 1 -- 172.21.15.141:0/428756303 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8034a80 con 0x7f65e00ada70 2024-01-16T15:22:14.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.068+0000 7f65d6a00640 1 -- 172.21.15.141:0/428756303 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d8034c20 con 0x7f65e00ada70 2024-01-16T15:22:14.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.068+0000 7f65d6a00640 1 -- 172.21.15.141:0/428756303 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d802de30 con 0x7f65e00ada70 2024-01-16T15:22:14.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.068+0000 7f65e4a00640 1 -- 172.21.15.141:0/428756303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e00ae440 con 0x7f65e00ada70 2024-01-16T15:22:14.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.068+0000 7f65e4a00640 1 -- 172.21.15.141:0/428756303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e0154cf0 con 0x7f65e00ada70 2024-01-16T15:22:14.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.069+0000 7f65e4a00640 1 -- 172.21.15.141:0/428756303 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e00cf170 con 0x7f65e00ada70 2024-01-16T15:22:14.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.070+0000 7f65d6a00640 1 -- 172.21.15.141:0/428756303 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f65d8049460 con 0x7f65e00ada70 2024-01-16T15:22:14.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.071+0000 7f65d6a00640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65b806c820 0x7f65b806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:14.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.071+0000 7f65d6a00640 1 -- 172.21.15.141:0/428756303 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f65d8000e40 con 0x7f65e00ada70 2024-01-16T15:22:14.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.071+0000 7f65de800640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65b806c820 0x7f65b806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:14.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.072+0000 7f65de800640 1 --2- 172.21.15.141:0/428756303 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f65b806c820 0x7f65b806ece0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f65e01f6370 tx=0x7f65d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:14.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.075+0000 7f65d6a00640 1 -- 172.21.15.141:0/428756303 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65d807a9f0 con 0x7f65e00ada70 2024-01-16T15:22:14.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.299+0000 7f65e4a00640 1 -- 172.21.15.141:0/428756303 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65e0074760 con 0x7f65b806c820 2024-01-16T15:22:14.307 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:14.312+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:14.308 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:14.313+0000 7f65d6a00640 1 -- 172.21.15.141:0/428756303 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f65e0074760 con 0x7f65b806c820 2024-01-16T15:22:14.308 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:14.308 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:16.324 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:16.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.554+0000 7f8101400640 1 Processor -- start 2024-01-16T15:22:16.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.555+0000 7f8101400640 1 -- start start 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.556+0000 7f8101400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 0x7f80fc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.556+0000 7f8101400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80fc13e360 0x7f80fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.556+0000 7f8101400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f80fc15f9c0 0x7f80fc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.556+0000 7f8101400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80fc06c470 con 0x7f80fc13e360 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.556+0000 7f8101400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80fc06c750 con 0x7f80fc15f9c0 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.556+0000 7f8101400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc06ca30 con 0x7f80fc13df80 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80fc13e360 0x7f80fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f80fc13e360 0x7f80fc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41910/0 (socket says 172.21.15.141:41910) 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 -- 172.21.15.141:0/257645059 learned_addr learned my addr 172.21.15.141:0/257645059 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:16.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80fa800640 1 --2- 172.21.15.141:0/257645059 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 0x7f80fc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80fb200640 1 --2- 172.21.15.141:0/257645059 >> v2:172.21.15.188:3300/0 conn(0x7f80fc15f9c0 0x7f80fc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 -- 172.21.15.141:0/257645059 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 msgr2=0x7f80fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 --2- 172.21.15.141:0/257645059 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 0x7f80fc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 -- 172.21.15.141:0/257645059 >> v2:172.21.15.188:3300/0 conn(0x7f80fc15f9c0 msgr2=0x7f80fc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 --2- 172.21.15.141:0/257645059 >> v2:172.21.15.188:3300/0 conn(0x7f80fc15f9c0 0x7f80fc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.557+0000 7f80f3e00640 1 -- 172.21.15.141:0/257645059 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc06cd10 con 0x7f80fc13e360 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.558+0000 7f80f3e00640 1 --2- 172.21.15.141:0/257645059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 0x7f80fc0cf290 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f80fc06e6d0 tx=0x7f80e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b85817c4602829 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.559+0000 7f80f9e00640 1 -- 172.21.15.141:0/257645059 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4031870 con 0x7f80fc13e360 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.559+0000 7f80f9e00640 1 -- 172.21.15.141:0/257645059 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e40319d0 con 0x7f80fc13e360 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.559+0000 7f80f9e00640 1 -- 172.21.15.141:0/257645059 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4031d60 con 0x7f80fc13e360 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.559+0000 7f8101400640 1 -- 172.21.15.141:0/257645059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 msgr2=0x7f80fc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:16.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.559+0000 7f8101400640 1 --2- 172.21.15.141:0/257645059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 0x7f80fc0cf290 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f80fc06e6d0 tx=0x7f80e402b910 comp rx=0 tx=0).stop 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.560+0000 7f8101400640 1 -- 172.21.15.141:0/257645059 shutdown_connections 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.560+0000 7f8101400640 1 --2- 172.21.15.141:0/257645059 >> v2:172.21.15.188:3300/0 conn(0x7f80fc15f9c0 0x7f80fc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.560+0000 7f8101400640 1 --2- 172.21.15.141:0/257645059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 0x7f80fc0cf290 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.560+0000 7f8101400640 1 --2- 172.21.15.141:0/257645059 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 0x7f80fc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.560+0000 7f8101400640 1 -- 172.21.15.141:0/257645059 >> 172.21.15.141:0/257645059 conn(0x7f80fc06a800 msgr2=0x7f80fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.560+0000 7f8101400640 1 -- 172.21.15.141:0/257645059 shutdown_connections 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.560+0000 7f8101400640 1 -- 172.21.15.141:0/257645059 wait complete. 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.561+0000 7f8101400640 1 Processor -- start 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.561+0000 7f8101400640 1 -- start start 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.562+0000 7f8101400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 0x7f80fc154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:16.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.562+0000 7f8101400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 0x7f80fc0b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.562+0000 7f8101400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80fc15f9c0 0x7f80fc0b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.562+0000 7f8101400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc139e90 con 0x7f80fc13e360 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.562+0000 7f8101400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc1549d0 con 0x7f80fc15f9c0 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.562+0000 7f8101400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc15e420 con 0x7f80fc13df80 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 0x7f80fc0b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 0x7f80fc0b2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41926/0 (socket says 172.21.15.141:41926) 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 -- 172.21.15.141:0/4283766371 learned_addr learned my addr 172.21.15.141:0/4283766371 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80fa800640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 0x7f80fc154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80fb200640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80fc15f9c0 0x7f80fc0b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:16.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 -- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 msgr2=0x7f80fc154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f80fc13df80 0x7f80fc154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 -- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80fc15f9c0 msgr2=0x7f80fc0b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80fc15f9c0 0x7f80fc0b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 -- 172.21.15.141:0/4283766371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40094d0 con 0x7f80fc13e360 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80fb200640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80fc15f9c0 0x7f80fc0b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.563+0000 7f80f3e00640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f80fc13e360 0x7f80fc0b2e20 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f80e4000c00 tx=0x7f80e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.564+0000 7f80f2a00640 1 -- 172.21.15.141:0/4283766371 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e403ac60 con 0x7f80fc13e360 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.564+0000 7f80f2a00640 1 -- 172.21.15.141:0/4283766371 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e403adc0 con 0x7f80fc13e360 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.564+0000 7f80f2a00640 1 -- 172.21.15.141:0/4283766371 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e40445f0 con 0x7f80fc13e360 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.564+0000 7f8101400640 1 -- 172.21.15.141:0/4283766371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc13a0a0 con 0x7f80fc13e360 2024-01-16T15:22:16.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.564+0000 7f8101400640 1 -- 172.21.15.141:0/4283766371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc13a660 con 0x7f80fc13e360 2024-01-16T15:22:16.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.565+0000 7f8101400640 1 -- 172.21.15.141:0/4283766371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc158c40 con 0x7f80fc13e360 2024-01-16T15:22:16.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.566+0000 7f80f2a00640 1 -- 172.21.15.141:0/4283766371 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f80e4044750 con 0x7f80fc13e360 2024-01-16T15:22:16.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.567+0000 7f80f2a00640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80d406c850 0x7f80d406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:16.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.567+0000 7f80f2a00640 1 -- 172.21.15.141:0/4283766371 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f80e402d040 con 0x7f80fc13e360 2024-01-16T15:22:16.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.568+0000 7f80fa800640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80d406c850 0x7f80d406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:16.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.569+0000 7f80fa800640 1 --2- 172.21.15.141:0/4283766371 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f80d406c850 0x7f80d406ed10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f80ec006fd0 tx=0x7f80ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:16.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.571+0000 7f80f2a00640 1 -- 172.21.15.141:0/4283766371 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80e40778f0 con 0x7f80fc13e360 2024-01-16T15:22:16.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.794+0000 7f8101400640 1 -- 172.21.15.141:0/4283766371 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80fc0b38a0 con 0x7f80d406c850 2024-01-16T15:22:16.804 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:16.808+0000 7f80f2a00640 1 -- 172.21.15.141:0/4283766371 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f80fc0b38a0 con 0x7f80d406c850 2024-01-16T15:22:16.804 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:16.804 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:16.805 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:16.808+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:18.816 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:19.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.043+0000 7f049cc00640 1 Processor -- start 2024-01-16T15:22:19.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.043+0000 7f049cc00640 1 -- start start 2024-01-16T15:22:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.045+0000 7f049cc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f049813e360 0x7f04980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:19.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.045+0000 7f049cc00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f049815f9c0 0x7f04980cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:19.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.045+0000 7f049cc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049815c5f0 con 0x7f049813df80 2024-01-16T15:22:19.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.045+0000 7f049cc00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049806c470 con 0x7f049815f9c0 2024-01-16T15:22:19.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.045+0000 7f049cc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049806c750 con 0x7f049813e360 2024-01-16T15:22:19.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0495e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f049813df80 0x7f049815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:43932/0 (socket says 172.21.15.141:43932) 2024-01-16T15:22:19.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0495e00640 1 -- 172.21.15.141:0/1428262534 learned_addr learned my addr 172.21.15.141:0/1428262534 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:19.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0496800640 1 --2- 172.21.15.141:0/1428262534 >> v2:172.21.15.141:3300/0 conn(0x7f049815f9c0 0x7f04980cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:19.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f048fe00640 1 --2- 172.21.15.141:0/1428262534 >> v2:172.21.15.188:3300/0 conn(0x7f049813e360 0x7f04980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0495400640 1 -- 172.21.15.141:0/1428262534 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802412368 0 0) 0x7f049815c5f0 con 0x7f049813df80 2024-01-16T15:22:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0495400640 1 -- 172.21.15.141:0/1428262534 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0478003520 con 0x7f049813df80 2024-01-16T15:22:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0496800640 1 -- 172.21.15.141:0/1428262534 >> v2:172.21.15.188:3300/0 conn(0x7f049813e360 msgr2=0x7f04980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0496800640 1 --2- 172.21.15.141:0/1428262534 >> v2:172.21.15.188:3300/0 conn(0x7f049813e360 0x7f04980cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0496800640 1 -- 172.21.15.141:0/1428262534 >> v1:172.21.15.141:6789/0 conn(0x7f049813df80 legacy=0x7f049815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.046+0000 7f0496800640 1 -- 172.21.15.141:0/1428262534 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049806ca30 con 0x7f049815f9c0 2024-01-16T15:22:19.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.047+0000 7f0496800640 1 --2- 172.21.15.141:0/1428262534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049815f9c0 0x7f04980cf7d0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f049806ded0 tx=0x7f049002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d46b91b272bfaba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.048+0000 7f0495400640 1 -- 172.21.15.141:0/1428262534 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049002cc80 con 0x7f049815f9c0 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.048+0000 7f0495400640 1 -- 172.21.15.141:0/1428262534 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049002cde0 con 0x7f049815f9c0 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.048+0000 7f0495400640 1 -- 172.21.15.141:0/1428262534 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490034660 con 0x7f049815f9c0 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.048+0000 7f049cc00640 1 -- 172.21.15.141:0/1428262534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049815f9c0 msgr2=0x7f04980cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.048+0000 7f049cc00640 1 --2- 172.21.15.141:0/1428262534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049815f9c0 0x7f04980cf7d0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f049806ded0 tx=0x7f049002bed0 comp rx=0 tx=0).stop 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.049+0000 7f049cc00640 1 -- 172.21.15.141:0/1428262534 shutdown_connections 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.049+0000 7f049cc00640 1 --2- 172.21.15.141:0/1428262534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049815f9c0 0x7f04980cf7d0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.049+0000 7f049cc00640 1 --2- 172.21.15.141:0/1428262534 >> v2:172.21.15.188:3300/0 conn(0x7f049813e360 0x7f04980cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.049+0000 7f049cc00640 1 -- 172.21.15.141:0/1428262534 >> 172.21.15.141:0/1428262534 conn(0x7f049806a800 msgr2=0x7f049806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.049+0000 7f049cc00640 1 -- 172.21.15.141:0/1428262534 shutdown_connections 2024-01-16T15:22:19.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.049+0000 7f049cc00640 1 -- 172.21.15.141:0/1428262534 wait complete. 2024-01-16T15:22:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.050+0000 7f049cc00640 1 Processor -- start 2024-01-16T15:22:19.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.051+0000 7f049cc00640 1 -- start start 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.051+0000 7f049cc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049813df80 0x7f049813d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.051+0000 7f049cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f049813e360 0x7f04980aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.051+0000 7f049cc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f049815f9c0 0x7f04980abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.051+0000 7f049cc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049813de10 con 0x7f049813df80 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.051+0000 7f049cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04980a84c0 con 0x7f049813e360 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f049cc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04980a8710 con 0x7f049815f9c0 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049813df80 0x7f049813d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049813df80 0x7f049813d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41942/0 (socket says 172.21.15.141:41942) 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 -- 172.21.15.141:0/1296414607 learned_addr learned my addr 172.21.15.141:0/1296414607 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f048fe00640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f049813e360 0x7f04980aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 -- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f049815f9c0 msgr2=0x7f04980abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:19.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0496800640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f049815f9c0 0x7f04980abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f049815f9c0 0x7f04980abfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 -- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f049813e360 msgr2=0x7f04980aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f049813e360 0x7f04980aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f0495e00640 1 -- 172.21.15.141:0/1296414607 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0490009a70 con 0x7f049813df80 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.052+0000 7f048fe00640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f049813e360 0x7f04980aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.053+0000 7f0495e00640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f049813df80 0x7f049813d8d0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f048800c400 tx=0x7f048800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.053+0000 7f048e000640 1 -- 172.21.15.141:0/1296414607 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488005ce0 con 0x7f049813df80 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.053+0000 7f048e000640 1 -- 172.21.15.141:0/1296414607 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0488005e40 con 0x7f049813df80 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.053+0000 7f048e000640 1 -- 172.21.15.141:0/1296414607 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04880072d0 con 0x7f049813df80 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.053+0000 7f049cc00640 1 -- 172.21.15.141:0/1296414607 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04980a89f0 con 0x7f049813df80 2024-01-16T15:22:19.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.053+0000 7f049cc00640 1 -- 172.21.15.141:0/1296414607 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04980a8fb0 con 0x7f049813df80 2024-01-16T15:22:19.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.054+0000 7f049cc00640 1 -- 172.21.15.141:0/1296414607 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04980675d0 con 0x7f049813df80 2024-01-16T15:22:19.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.055+0000 7f048e000640 1 -- 172.21.15.141:0/1296414607 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f0488007430 con 0x7f049813df80 2024-01-16T15:22:19.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.056+0000 7f048e000640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f047006c850 0x7f047006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:19.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.056+0000 7f048e000640 1 -- 172.21.15.141:0/1296414607 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0488088130 con 0x7f049813df80 2024-01-16T15:22:19.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.057+0000 7f048fe00640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f047006c850 0x7f047006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:19.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.058+0000 7f048fe00640 1 --2- 172.21.15.141:0/1296414607 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f047006c850 0x7f047006ed10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f04980ae4f0 tx=0x7f0480008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:19.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.060+0000 7f048e000640 1 -- 172.21.15.141:0/1296414607 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0488054700 con 0x7f049813df80 2024-01-16T15:22:19.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.284+0000 7f049cc00640 1 -- 172.21.15.141:0/1296414607 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f049806cff0 con 0x7f047006c850 2024-01-16T15:22:19.293 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:19.299+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:19.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:19.298+0000 7f048e000640 1 -- 172.21.15.141:0/1296414607 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f04800095f0 con 0x7f047006c850 2024-01-16T15:22:19.294 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:19.294 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:19.313 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:21.315 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:21.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.544+0000 7f04d3a00640 1 Processor -- start 2024-01-16T15:22:21.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.544+0000 7f04d3a00640 1 -- start start 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04d3a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f04cc0cf9a0 0x7f04cc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04d3a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f04cc15e1e0 0x7f04cc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04d3a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0b1ee0 con 0x7f04cc143ae0 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04d3a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04cc0b2780 con 0x7f04cc0cf9a0 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04d3a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04cc15e990 con 0x7f04cc15e1e0 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04cbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f04cc0cf9a0 0x7f04cc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04cbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f04cc0cf9a0 0x7f04cc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41944/0 (socket says 172.21.15.141:41944) 2024-01-16T15:22:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.546+0000 7f04cbe00640 1 -- 172.21.15.141:0/1815275066 learned_addr learned my addr 172.21.15.141:0/1815275066 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:21.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.547+0000 7f04cbe00640 1 -- 172.21.15.141:0/1815275066 >> v2:172.21.15.188:3300/0 conn(0x7f04cc15e1e0 msgr2=0x7f04cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:21.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.547+0000 7f04d1a00640 1 --2- 172.21.15.141:0/1815275066 >> v2:172.21.15.188:3300/0 conn(0x7f04cc15e1e0 0x7f04cc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:21.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.547+0000 7f04cbe00640 1 --2- 172.21.15.141:0/1815275066 >> v2:172.21.15.188:3300/0 conn(0x7f04cc15e1e0 0x7f04cc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:21.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.547+0000 7f04cbe00640 1 -- 172.21.15.141:0/1815275066 >> v1:172.21.15.188:6789/0 conn(0x7f04cc143ae0 legacy=0x7f04cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:21.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.547+0000 7f04cbe00640 1 -- 172.21.15.141:0/1815275066 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0b2130 con 0x7f04cc0cf9a0 2024-01-16T15:22:21.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.548+0000 7f04cbe00640 1 --2- 172.21.15.141:0/1815275066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc0cf9a0 0x7f04cc15dca0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0ae9d0 tx=0x7f04b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7000ee255a23c279 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:21.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.548+0000 7f04cb400640 1 -- 172.21.15.141:0/1815275066 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4030860 con 0x7f04cc0cf9a0 2024-01-16T15:22:21.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.548+0000 7f04cb400640 1 -- 172.21.15.141:0/1815275066 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b40309c0 con 0x7f04cc0cf9a0 2024-01-16T15:22:21.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.548+0000 7f04cb400640 1 -- 172.21.15.141:0/1815275066 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4030d50 con 0x7f04cc0cf9a0 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 -- 172.21.15.141:0/1815275066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc0cf9a0 msgr2=0x7f04cc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 --2- 172.21.15.141:0/1815275066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc0cf9a0 0x7f04cc15dca0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0ae9d0 tx=0x7f04b402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 -- 172.21.15.141:0/1815275066 shutdown_connections 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 --2- 172.21.15.141:0/1815275066 >> v2:172.21.15.188:3300/0 conn(0x7f04cc15e1e0 0x7f04cc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 --2- 172.21.15.141:0/1815275066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc0cf9a0 0x7f04cc15dca0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 -- 172.21.15.141:0/1815275066 >> 172.21.15.141:0/1815275066 conn(0x7f04cc06a800 msgr2=0x7f04cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 -- 172.21.15.141:0/1815275066 shutdown_connections 2024-01-16T15:22:21.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.549+0000 7f04d3a00640 1 -- 172.21.15.141:0/1815275066 wait complete. 2024-01-16T15:22:21.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.551+0000 7f04d3a00640 1 Processor -- start 2024-01-16T15:22:21.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.551+0000 7f04d3a00640 1 -- start start 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d3a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04cc0cf9a0 0x7f04cc0ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d3a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04cc143ae0 0x7f04cc0c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d3a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc15e1e0 0x7f04cc13eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d3a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0cc610 con 0x7f04cc15e1e0 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d3a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0adb30 con 0x7f04cc143ae0 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d3a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0ade10 con 0x7f04cc0cf9a0 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc15e1e0 0x7f04cc13eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc15e1e0 0x7f04cc13eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:41952/0 (socket says 172.21.15.141:41952) 2024-01-16T15:22:21.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.552+0000 7f04d1a00640 1 -- 172.21.15.141:0/1205784120 learned_addr learned my addr 172.21.15.141:0/1205784120 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04cbe00640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04cc143ae0 0x7f04cc0c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d1000640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04cc0cf9a0 0x7f04cc0ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d1a00640 1 -- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04cc0cf9a0 msgr2=0x7f04cc0ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d1a00640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f04cc0cf9a0 0x7f04cc0ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d1a00640 1 -- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04cc143ae0 msgr2=0x7f04cc0c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d1a00640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04cc143ae0 0x7f04cc0c4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d1a00640 1 -- 172.21.15.141:0/1205784120 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40094d0 con 0x7f04cc15e1e0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d1a00640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f04cc15e1e0 0x7f04cc13eae0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f04c400cd70 tx=0x7f04c400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04c8c00640 1 -- 172.21.15.141:0/1205784120 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4003b90 con 0x7f04cc15e1e0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04c8c00640 1 -- 172.21.15.141:0/1205784120 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c4003cf0 con 0x7f04cc15e1e0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d3a00640 1 -- 172.21.15.141:0/1205784120 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc0ae0f0 con 0x7f04cc15e1e0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04c8c00640 1 -- 172.21.15.141:0/1205784120 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4004240 con 0x7f04cc15e1e0 2024-01-16T15:22:21.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.553+0000 7f04d3a00640 1 -- 172.21.15.141:0/1205784120 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc0ad820 con 0x7f04cc15e1e0 2024-01-16T15:22:21.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.555+0000 7f04d3a00640 1 -- 172.21.15.141:0/1205784120 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc0ccaf0 con 0x7f04cc15e1e0 2024-01-16T15:22:21.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.556+0000 7f04c8c00640 1 -- 172.21.15.141:0/1205784120 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f04c40043a0 con 0x7f04cc15e1e0 2024-01-16T15:22:21.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.556+0000 7f04c8c00640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f04a406c820 0x7f04a406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:21.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.557+0000 7f04c8c00640 1 -- 172.21.15.141:0/1205784120 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f04c4012070 con 0x7f04cc15e1e0 2024-01-16T15:22:21.552 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.557+0000 7f04d1000640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f04a406c820 0x7f04a406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:21.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.558+0000 7f04d1000640 1 --2- 172.21.15.141:0/1205784120 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f04a406c820 0x7f04a406ece0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f04bc000f60 tx=0x7f04bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:21.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.561+0000 7f04c8c00640 1 -- 172.21.15.141:0/1205784120 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04cc0ccaf0 con 0x7f04cc15e1e0 2024-01-16T15:22:21.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.784+0000 7f04d3a00640 1 -- 172.21.15.141:0/1205784120 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04cc15e670 con 0x7f04a406c820 2024-01-16T15:22:21.792 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:21.798+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:21.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:21.797+0000 7f04c8c00640 1 -- 172.21.15.141:0/1205784120 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f04bc001c50 con 0x7f04a406c820 2024-01-16T15:22:21.793 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:21.793 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:23.805 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:24.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.032+0000 7f4c82200640 1 Processor -- start 2024-01-16T15:22:24.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.032+0000 7f4c82200640 1 -- start start 2024-01-16T15:22:24.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c82200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f4c7c0cf9a0 0x7f4c7c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c82200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 0x7f4c7c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c82200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7c0b1ee0 con 0x7f4c7c143ae0 2024-01-16T15:22:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c82200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7c0b2780 con 0x7f4c7c0cf9a0 2024-01-16T15:22:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c82200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c15e990 con 0x7f4c7c15e1e0 2024-01-16T15:22:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c7b400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4c7c143ae0 0x7f4c7c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60046/0 (socket says 172.21.15.141:60046) 2024-01-16T15:22:24.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c7b400640 1 -- 172.21.15.141:0/657104229 learned_addr learned my addr 172.21.15.141:0/657104229 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197103115 0 0) 0x7f4c7c0b1ee0 con 0x7f4c7c143ae0 2024-01-16T15:22:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c5c003520 con 0x7f4c7c143ae0 2024-01-16T15:22:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c73e00640 1 --2- 172.21.15.141:0/657104229 >> v2:172.21.15.188:3300/0 conn(0x7f4c7c0cf9a0 0x7f4c7c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.034+0000 7f4c7be00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 0x7f4c7c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919159752 0 0) 0x7f4c5c003520 con 0x7f4c7c143ae0 2024-01-16T15:22:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 msgr2=0x7f4c7c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:24.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 0x7f4c7c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 >> v2:172.21.15.188:3300/0 conn(0x7f4c7c0cf9a0 msgr2=0x7f4c7c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 --2- 172.21.15.141:0/657104229 >> v2:172.21.15.188:3300/0 conn(0x7f4c7c0cf9a0 0x7f4c7c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c0b2130 con 0x7f4c7c143ae0 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4c6c001670 con 0x7f4c7c143ae0 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 >> v1:172.21.15.141:6789/0 conn(0x7f4c7c143ae0 legacy=0x7f4c7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c5c003970 0x7f4c5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 0x7f4c5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0b2780 con 0x7f4c5c004d00 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c15e990 con 0x7f4c5c003970 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c00b980 con 0x7f4c7c15e1e0 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.035+0000 7f4c7be00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.036+0000 7f4c73e00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 0x7f4c5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.036+0000 7f4c7b400640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c5c003970 0x7f4c5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.036+0000 7f4c7be00640 1 -- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 msgr2=0x7f4c5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.036+0000 7f4c7be00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c15e1e0 0x7f4c5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.036+0000 7f4c7be00640 1 -- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c5c003970 msgr2=0x7f4c5c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.036+0000 7f4c7be00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c5c003970 0x7f4c5c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.036+0000 7f4c7be00640 1 -- 172.21.15.141:0/657104229 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c003520 con 0x7f4c5c004d00 2024-01-16T15:22:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.037+0000 7f4c7be00640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c5c005180 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f4c74010380 tx=0x7f4c5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3243cdef39fbcc97 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.037+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7402ee20 con 0x7f4c5c004d00 2024-01-16T15:22:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.037+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74011040 con 0x7f4c5c004d00 2024-01-16T15:22:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.037+0000 7f4c7aa00640 1 -- 172.21.15.141:0/657104229 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7400b2f0 con 0x7f4c5c004d00 2024-01-16T15:22:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.038+0000 7f4c82200640 1 -- 172.21.15.141:0/657104229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 msgr2=0x7f4c5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.038+0000 7f4c82200640 1 --2- 172.21.15.141:0/657104229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c5c005180 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f4c74010380 tx=0x7f4c5c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.038+0000 7f4c7be00640 1 -- 172.21.15.141:0/657104229 reap_dead start 2024-01-16T15:22:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.038+0000 7f4c82200640 1 -- 172.21.15.141:0/657104229 shutdown_connections 2024-01-16T15:22:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.038+0000 7f4c82200640 1 -- 172.21.15.141:0/657104229 >> 172.21.15.141:0/657104229 conn(0x7f4c7c06a800 msgr2=0x7f4c7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.038+0000 7f4c82200640 1 -- 172.21.15.141:0/657104229 shutdown_connections 2024-01-16T15:22:24.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.038+0000 7f4c82200640 1 -- 172.21.15.141:0/657104229 wait complete. 2024-01-16T15:22:24.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.040+0000 7f4c82200640 1 Processor -- start 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.040+0000 7f4c82200640 1 -- start start 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c82200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c7c1418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c82200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c0ae3f0 0x7f4c7c0cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c82200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0cdba0 0x7f4c7c0c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c82200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0ca930 con 0x7f4c5c004d00 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c82200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0ce020 con 0x7f4c7c0cdba0 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c82200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0c9cb0 con 0x7f4c7c0ae3f0 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c7b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c7c1418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c7b400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c7c1418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45546/0 (socket says 172.21.15.141:45546) 2024-01-16T15:22:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.041+0000 7f4c7b400640 1 -- 172.21.15.141:0/339416788 learned_addr learned my addr 172.21.15.141:0/339416788 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c7b400640 1 -- 172.21.15.141:0/339416788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c0ae3f0 msgr2=0x7f4c7c0cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c7b400640 1 --2- 172.21.15.141:0/339416788 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4c7c0ae3f0 0x7f4c7c0cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c7b400640 1 -- 172.21.15.141:0/339416788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0cdba0 msgr2=0x7f4c7c0c4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c7b400640 1 --2- 172.21.15.141:0/339416788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0cdba0 0x7f4c7c0c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c7b400640 1 -- 172.21.15.141:0/339416788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c0c9f90 con 0x7f4c5c004d00 2024-01-16T15:22:24.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c7b400640 1 --2- 172.21.15.141:0/339416788 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c5c004d00 0x7f4c7c1418a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00e780 tx=0x7f4c6c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:24.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c73400640 1 -- 172.21.15.141:0/339416788 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c00ee90 con 0x7f4c5c004d00 2024-01-16T15:22:24.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c73400640 1 -- 172.21.15.141:0/339416788 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c00c7f0 con 0x7f4c5c004d00 2024-01-16T15:22:24.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c73400640 1 -- 172.21.15.141:0/339416788 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c00cb80 con 0x7f4c5c004d00 2024-01-16T15:22:24.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.042+0000 7f4c82200640 1 -- 172.21.15.141:0/339416788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7c0ca270 con 0x7f4c5c004d00 2024-01-16T15:22:24.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.043+0000 7f4c82200640 1 -- 172.21.15.141:0/339416788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7c141de0 con 0x7f4c5c004d00 2024-01-16T15:22:24.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.044+0000 7f4c82200640 1 -- 172.21.15.141:0/339416788 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7c0675d0 con 0x7f4c5c004d00 2024-01-16T15:22:24.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.045+0000 7f4c73400640 1 -- 172.21.15.141:0/339416788 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f4c6c00cce0 con 0x7f4c5c004d00 2024-01-16T15:22:24.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.045+0000 7f4c73400640 1 --2- 172.21.15.141:0/339416788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c5406c850 0x7f4c5406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:24.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.046+0000 7f4c73400640 1 -- 172.21.15.141:0/339416788 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4c6c0889d0 con 0x7f4c5c004d00 2024-01-16T15:22:24.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.046+0000 7f4c73e00640 1 --2- 172.21.15.141:0/339416788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c5406c850 0x7f4c5406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:24.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.047+0000 7f4c73e00640 1 --2- 172.21.15.141:0/339416788 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4c5406c850 0x7f4c5406ed10 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c15bae0 tx=0x7f4c64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:24.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.050+0000 7f4c73400640 1 -- 172.21.15.141:0/339416788 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c6c054e00 con 0x7f4c5c004d00 2024-01-16T15:22:24.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.272+0000 7f4c82200640 1 -- 172.21.15.141:0/339416788 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c7c15e670 con 0x7f4c5406c850 2024-01-16T15:22:24.282 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:24.287+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:24.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:24.286+0000 7f4c73400640 1 -- 172.21.15.141:0/339416788 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c640095e0 con 0x7f4c5406c850 2024-01-16T15:22:24.283 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:24.283 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:26.300 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:26.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.533+0000 7f8cd1a00640 1 Processor -- start 2024-01-16T15:22:26.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.533+0000 7f8cd1a00640 1 -- start start 2024-01-16T15:22:26.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.534+0000 7f8cd1a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc0d0100 0x7f8ccc0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:26.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.534+0000 7f8cd1a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8ccc0d0f40 0x7f8ccc0d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:26.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.534+0000 7f8cd1a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc0d0a10 con 0x7f8ccc160920 2024-01-16T15:22:26.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.534+0000 7f8cd1a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ccc0cf080 con 0x7f8ccc0d0f40 2024-01-16T15:22:26.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.534+0000 7f8cd1a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc0c6be0 con 0x7f8ccc0d0100 2024-01-16T15:22:26.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8cd1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8ccc0d0f40 0x7f8ccc0d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:26.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8cd1000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8ccc0d0f40 0x7f8ccc0d13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45560/0 (socket says 172.21.15.141:45560) 2024-01-16T15:22:26.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8cd1000640 1 -- 172.21.15.141:0/3800974326 learned_addr learned my addr 172.21.15.141:0/3800974326 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:26.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8ccb400640 1 --2- 172.21.15.141:0/3800974326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc0d0100 0x7f8ccc0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:26.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8cd1000640 1 -- 172.21.15.141:0/3800974326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc0d0100 msgr2=0x7f8ccc0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:26.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8cd1000640 1 --2- 172.21.15.141:0/3800974326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc0d0100 0x7f8ccc0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:26.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8cd1000640 1 -- 172.21.15.141:0/3800974326 >> v1:172.21.15.188:6789/0 conn(0x7f8ccc160920 legacy=0x7f8ccc0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:26.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.535+0000 7f8cd1000640 1 -- 172.21.15.141:0/3800974326 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc0ce7e0 con 0x7f8ccc0d0f40 2024-01-16T15:22:26.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.536+0000 7f8cd1000640 1 --2- 172.21.15.141:0/3800974326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0f40 0x7f8ccc0d13f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009870 tx=0x7f8cc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9627131969b8e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:26.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8ccaa00640 1 -- 172.21.15.141:0/3800974326 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4031640 con 0x7f8ccc0d0f40 2024-01-16T15:22:26.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8ccaa00640 1 -- 172.21.15.141:0/3800974326 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc40317a0 con 0x7f8ccc0d0f40 2024-01-16T15:22:26.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8ccaa00640 1 -- 172.21.15.141:0/3800974326 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4031b30 con 0x7f8ccc0d0f40 2024-01-16T15:22:26.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8cd1a00640 1 -- 172.21.15.141:0/3800974326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0f40 msgr2=0x7f8ccc0d13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:26.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8cd1a00640 1 --2- 172.21.15.141:0/3800974326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0f40 0x7f8ccc0d13f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009870 tx=0x7f8cc402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:22:26.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8cd1a00640 1 -- 172.21.15.141:0/3800974326 shutdown_connections 2024-01-16T15:22:26.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8cd1a00640 1 --2- 172.21.15.141:0/3800974326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0f40 0x7f8ccc0d13f0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:26.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.537+0000 7f8cd1a00640 1 --2- 172.21.15.141:0/3800974326 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc0d0100 0x7f8ccc0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:26.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.538+0000 7f8cd1a00640 1 -- 172.21.15.141:0/3800974326 >> 172.21.15.141:0/3800974326 conn(0x7f8ccc0d3090 msgr2=0x7f8ccc0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:26.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.538+0000 7f8cd1a00640 1 -- 172.21.15.141:0/3800974326 shutdown_connections 2024-01-16T15:22:26.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.538+0000 7f8cd1a00640 1 -- 172.21.15.141:0/3800974326 wait complete. 2024-01-16T15:22:26.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.539+0000 7f8cd1a00640 1 Processor -- start 2024-01-16T15:22:26.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.539+0000 7f8cd1a00640 1 -- start start 2024-01-16T15:22:26.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.540+0000 7f8cd1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0100 0x7f8ccc14a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:26.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.540+0000 7f8cd1a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ccc0d0f40 0x7f8ccc0b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:26.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.540+0000 7f8cd1a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc160920 0x7f8ccc159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:26.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.540+0000 7f8cd1a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc158bf0 con 0x7f8ccc0d0100 2024-01-16T15:22:26.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.540+0000 7f8cd1a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc158eb0 con 0x7f8ccc0d0f40 2024-01-16T15:22:26.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.540+0000 7f8cd1a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc159190 con 0x7f8ccc160920 2024-01-16T15:22:26.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0100 0x7f8ccc14a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0100 0x7f8ccc14a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45566/0 (socket says 172.21.15.141:45566) 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 -- 172.21.15.141:0/2742685116 learned_addr learned my addr 172.21.15.141:0/2742685116 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8cd1000640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc160920 0x7f8ccc159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccb400640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ccc0d0f40 0x7f8ccc0b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 -- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc160920 msgr2=0x7f8ccc159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8ccc160920 0x7f8ccc159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 -- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ccc0d0f40 msgr2=0x7f8ccc0b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ccc0d0f40 0x7f8ccc0b5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.541+0000 7f8ccbe00640 1 -- 172.21.15.141:0/2742685116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40094d0 con 0x7f8ccc0d0100 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.542+0000 7f8ccbe00640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ccc0d0100 0x7f8ccc14a580 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00c430 tx=0x7f8cbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.542+0000 7f8cc3e00640 1 -- 172.21.15.141:0/2742685116 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc011790 con 0x7f8ccc0d0100 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.542+0000 7f8cc3e00640 1 -- 172.21.15.141:0/2742685116 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc0118f0 con 0x7f8ccc0d0100 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.542+0000 7f8cc3e00640 1 -- 172.21.15.141:0/2742685116 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc011ce0 con 0x7f8ccc0d0100 2024-01-16T15:22:26.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.542+0000 7f8cd1a00640 1 -- 172.21.15.141:0/2742685116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ccc159470 con 0x7f8ccc0d0100 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.542+0000 7f8cd1a00640 1 -- 172.21.15.141:0/2742685116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ccc158870 con 0x7f8ccc0d0100 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.543+0000 7f8cd1a00640 1 -- 172.21.15.141:0/2742685116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ccc159990 con 0x7f8ccc0d0100 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.544+0000 7f8cc3e00640 1 -- 172.21.15.141:0/2742685116 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f8cbc01c460 con 0x7f8ccc0d0100 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.545+0000 7f8cc3e00640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8ca406c820 0x7f8ca406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.545+0000 7f8cc3e00640 1 -- 172.21.15.141:0/2742685116 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8cbc088c70 con 0x7f8ccc0d0100 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.546+0000 7f8ccb400640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8ca406c820 0x7f8ca406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.547+0000 7f8ccb400640 1 --2- 172.21.15.141:0/2742685116 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8ca406c820 0x7f8ca406ece0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc1f6af0 tx=0x7f8cb400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:26.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.549+0000 7f8cc3e00640 1 -- 172.21.15.141:0/2742685116 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cbc055130 con 0x7f8ccc0d0100 2024-01-16T15:22:26.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.778+0000 7f8cd1a00640 1 -- 172.21.15.141:0/2742685116 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ccc159020 con 0x7f8ca406c820 2024-01-16T15:22:26.789 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:26.795+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:26.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:26.795+0000 7f8cc3e00640 1 -- 172.21.15.141:0/2742685116 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8ccc159020 con 0x7f8ca406c820 2024-01-16T15:22:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:26.790 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:28.804 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:29.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.039+0000 7f1749000640 1 Processor -- start 2024-01-16T15:22:29.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.039+0000 7f1749000640 1 -- start start 2024-01-16T15:22:29.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.041+0000 7f1749000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f174414b2c0 0x7f17440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.041+0000 7f1749000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1744071070 0x7f17440cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.041+0000 7f1749000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17440cfbf0 con 0x7f17440cf7d0 2024-01-16T15:22:29.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.041+0000 7f1749000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174414b6a0 con 0x7f174414b2c0 2024-01-16T15:22:29.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.041+0000 7f1749000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1744071690 con 0x7f1744071070 2024-01-16T15:22:29.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f174414b2c0 0x7f17440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1742800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1744071070 0x7f17440cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f174414b2c0 0x7f17440cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:43616/0 (socket says 172.21.15.141:43616) 2024-01-16T15:22:29.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741e00640 1 -- 172.21.15.141:0/252679557 learned_addr learned my addr 172.21.15.141:0/252679557 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:29.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741400640 1 --1- 172.21.15.141:0/252679557 >> v1:172.21.15.188:6789/0 conn(0x7f17440cf7d0 0x7f1744070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.141:50994/0 (socket says 172.21.15.141:50994) 2024-01-16T15:22:29.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741e00640 1 -- 172.21.15.141:0/252679557 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1744071070 msgr2=0x7f17440cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:29.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741e00640 1 --2- 172.21.15.141:0/252679557 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1744071070 0x7f17440cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741e00640 1 -- 172.21.15.141:0/252679557 >> v1:172.21.15.188:6789/0 conn(0x7f17440cf7d0 legacy=0x7f1744070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:29.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.042+0000 7f1741e00640 1 -- 172.21.15.141:0/252679557 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1744071970 con 0x7f174414b2c0 2024-01-16T15:22:29.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.044+0000 7f1741e00640 1 --2- 172.21.15.141:0/252679557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f174414b2c0 0x7f17440cf290 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1734009870 tx=0x7f173402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac5c46080a85144d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:29.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.044+0000 7f1740a00640 1 -- 172.21.15.141:0/252679557 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734030840 con 0x7f174414b2c0 2024-01-16T15:22:29.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.044+0000 7f1740a00640 1 -- 172.21.15.141:0/252679557 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17340309a0 con 0x7f174414b2c0 2024-01-16T15:22:29.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.044+0000 7f1740a00640 1 -- 172.21.15.141:0/252679557 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734030d30 con 0x7f174414b2c0 2024-01-16T15:22:29.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.045+0000 7f1749000640 1 -- 172.21.15.141:0/252679557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f174414b2c0 msgr2=0x7f17440cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:29.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.045+0000 7f1749000640 1 --2- 172.21.15.141:0/252679557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f174414b2c0 0x7f17440cf290 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1734009870 tx=0x7f173402b980 comp rx=0 tx=0).stop 2024-01-16T15:22:29.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.045+0000 7f1749000640 1 -- 172.21.15.141:0/252679557 shutdown_connections 2024-01-16T15:22:29.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.045+0000 7f1749000640 1 --2- 172.21.15.141:0/252679557 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1744071070 0x7f17440cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.045+0000 7f1749000640 1 --2- 172.21.15.141:0/252679557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f174414b2c0 0x7f17440cf290 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.045+0000 7f1749000640 1 -- 172.21.15.141:0/252679557 >> 172.21.15.141:0/252679557 conn(0x7f174406a800 msgr2=0x7f174406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:29.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.046+0000 7f1749000640 1 -- 172.21.15.141:0/252679557 shutdown_connections 2024-01-16T15:22:29.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.046+0000 7f1749000640 1 -- 172.21.15.141:0/252679557 wait complete. 2024-01-16T15:22:29.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.048+0000 7f1749000640 1 Processor -- start 2024-01-16T15:22:29.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.048+0000 7f1749000640 1 -- start start 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.049+0000 7f1749000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1744071070 0x7f17441419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.049+0000 7f1749000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17440cf7d0 0x7f17441648c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.049+0000 7f1749000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f174414b2c0 0x7f17441f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.049+0000 7f1749000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1744164e00 con 0x7f1744071070 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.049+0000 7f1749000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174415ce60 con 0x7f17440cf7d0 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.049+0000 7f1749000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174415d140 con 0x7f174414b2c0 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.049+0000 7f1741e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1744071070 0x7f17441419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1744071070 0x7f17441419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45572/0 (socket says 172.21.15.141:45572) 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 -- 172.21.15.141:0/3222532764 learned_addr learned my addr 172.21.15.141:0/3222532764 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741400640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17440cf7d0 0x7f17441648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1742800640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f174414b2c0 0x7f17441f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 -- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f174414b2c0 msgr2=0x7f17441f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:29.046 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f174414b2c0 0x7f17441f7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 -- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17440cf7d0 msgr2=0x7f17441648c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:29.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17440cf7d0 0x7f17441648c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:29.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 -- 172.21.15.141:0/3222532764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340094d0 con 0x7f1744071070 2024-01-16T15:22:29.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.050+0000 7f1741e00640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1744071070 0x7f17441419b0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f1734000c00 tx=0x7f1734004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:29.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.051+0000 7f173a000640 1 -- 172.21.15.141:0/3222532764 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734039a90 con 0x7f1744071070 2024-01-16T15:22:29.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.051+0000 7f173a000640 1 -- 172.21.15.141:0/3222532764 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1734039bf0 con 0x7f1744071070 2024-01-16T15:22:29.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.051+0000 7f173a000640 1 -- 172.21.15.141:0/3222532764 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734043410 con 0x7f1744071070 2024-01-16T15:22:29.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.051+0000 7f1749000640 1 -- 172.21.15.141:0/3222532764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174415d420 con 0x7f1744071070 2024-01-16T15:22:29.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.051+0000 7f1749000640 1 -- 172.21.15.141:0/3222532764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174415d9e0 con 0x7f1744071070 2024-01-16T15:22:29.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.052+0000 7f1749000640 1 -- 172.21.15.141:0/3222532764 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17440675d0 con 0x7f1744071070 2024-01-16T15:22:29.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.053+0000 7f173a000640 1 -- 172.21.15.141:0/3222532764 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f173402f390 con 0x7f1744071070 2024-01-16T15:22:29.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.054+0000 7f173a000640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f171c06c820 0x7f171c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:29.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.054+0000 7f173a000640 1 -- 172.21.15.141:0/3222532764 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f17340aae60 con 0x7f1744071070 2024-01-16T15:22:29.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.055+0000 7f1741400640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f171c06c820 0x7f171c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:29.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.056+0000 7f1741400640 1 --2- 172.21.15.141:0/3222532764 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f171c06c820 0x7f171c06ece0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1744164630 tx=0x7f172c007c80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:29.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.058+0000 7f173a000640 1 -- 172.21.15.141:0/3222532764 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1734077320 con 0x7f1744071070 2024-01-16T15:22:29.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.286+0000 7f1749000640 1 -- 172.21.15.141:0/3222532764 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17440cf3f0 con 0x7f171c06c820 2024-01-16T15:22:29.299 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:29.304+0000 7f173a000640 1 -- 172.21.15.141:0/3222532764 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f17440cf3f0 con 0x7f171c06c820 2024-01-16T15:22:29.300 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:29.300 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:29.300 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:29.304+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:31.314 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:31.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.542+0000 7fa4f3e00640 1 Processor -- start 2024-01-16T15:22:31.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.542+0000 7fa4f3e00640 1 -- start start 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.543+0000 7fa4f3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec15da10 0x7fa4ec072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa4ec0b2bd0 0x7fa4ec0b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f3e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ec071190 con 0x7fa4ec150760 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f3e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec071470 con 0x7fa4ec0b2bd0 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec071750 con 0x7fa4ec15da10 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f1400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa4ec150760 0x7fa4ec15d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60054/0 (socket says 172.21.15.141:60054) 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f1400640 1 -- 172.21.15.141:0/343320011 learned_addr learned my addr 172.21.15.141:0/343320011 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:31.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f1e00640 1 --2- 172.21.15.141:0/343320011 >> v2:172.21.15.141:3300/0 conn(0x7fa4ec0b2bd0 0x7fa4ec0b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:31.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4ebe00640 1 -- 172.21.15.141:0/343320011 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243104369 0 0) 0x7fa4ec071190 con 0x7fa4ec150760 2024-01-16T15:22:31.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4ebe00640 1 -- 172.21.15.141:0/343320011 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc003440 con 0x7fa4ec150760 2024-01-16T15:22:31.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f0a00640 1 --2- 172.21.15.141:0/343320011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec15da10 0x7fa4ec072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:31.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f1e00640 1 -- 172.21.15.141:0/343320011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec15da10 msgr2=0x7fa4ec072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:31.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f1e00640 1 --2- 172.21.15.141:0/343320011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec15da10 0x7fa4ec072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:31.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f1e00640 1 -- 172.21.15.141:0/343320011 >> v1:172.21.15.141:6789/0 conn(0x7fa4ec150760 legacy=0x7fa4ec15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:31.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.545+0000 7fa4f1e00640 1 -- 172.21.15.141:0/343320011 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec071a30 con 0x7fa4ec0b2bd0 2024-01-16T15:22:31.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.546+0000 7fa4f1e00640 1 --2- 172.21.15.141:0/343320011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec0b2bd0 0x7fa4ec0b3080 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40097a0 tx=0x7fa4e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907e1e044dd35a8f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.546+0000 7fa4ebe00640 1 -- 172.21.15.141:0/343320011 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4031830 con 0x7fa4ec0b2bd0 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.546+0000 7fa4ebe00640 1 -- 172.21.15.141:0/343320011 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e4031990 con 0x7fa4ec0b2bd0 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.546+0000 7fa4ebe00640 1 -- 172.21.15.141:0/343320011 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4031d00 con 0x7fa4ec0b2bd0 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.546+0000 7fa4f3e00640 1 -- 172.21.15.141:0/343320011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec0b2bd0 msgr2=0x7fa4ec0b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.546+0000 7fa4f3e00640 1 --2- 172.21.15.141:0/343320011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec0b2bd0 0x7fa4ec0b3080 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40097a0 tx=0x7fa4e402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.547+0000 7fa4f3e00640 1 -- 172.21.15.141:0/343320011 shutdown_connections 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.547+0000 7fa4f3e00640 1 --2- 172.21.15.141:0/343320011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec0b2bd0 0x7fa4ec0b3080 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.547+0000 7fa4f3e00640 1 --2- 172.21.15.141:0/343320011 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec15da10 0x7fa4ec072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.547+0000 7fa4f3e00640 1 -- 172.21.15.141:0/343320011 >> 172.21.15.141:0/343320011 conn(0x7fa4ec06a800 msgr2=0x7fa4ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:31.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.547+0000 7fa4f3e00640 1 -- 172.21.15.141:0/343320011 shutdown_connections 2024-01-16T15:22:31.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.547+0000 7fa4f3e00640 1 -- 172.21.15.141:0/343320011 wait complete. 2024-01-16T15:22:31.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.548+0000 7fa4f3e00640 1 Processor -- start 2024-01-16T15:22:31.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.549+0000 7fa4f3e00640 1 -- start start 2024-01-16T15:22:31.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4ec0b2bd0 0x7fa4ec146270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec150760 0x7fa4ec15c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec15da10 0x7fa4ec1e4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f3e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec14a5e0 con 0x7fa4ec15da10 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1467b0 con 0x7fa4ec0b2bd0 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec14dec0 con 0x7fa4ec150760 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec15da10 0x7fa4ec1e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec15da10 0x7fa4ec1e4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:45598/0 (socket says 172.21.15.141:45598) 2024-01-16T15:22:31.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 -- 172.21.15.141:0/2086503994 learned_addr learned my addr 172.21.15.141:0/2086503994 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:31.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f0a00640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec150760 0x7fa4ec15c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:31.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1400640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4ec0b2bd0 0x7fa4ec146270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 -- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec150760 msgr2=0x7fa4ec15c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec150760 0x7fa4ec15c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 -- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4ec0b2bd0 msgr2=0x7fa4ec146270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4ec0b2bd0 0x7fa4ec146270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1e00640 1 -- 172.21.15.141:0/2086503994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e4009400 con 0x7fa4ec15da10 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f0a00640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa4ec150760 0x7fa4ec15c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.550+0000 7fa4f1400640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4ec0b2bd0 0x7fa4ec146270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.551+0000 7fa4f1e00640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ec15da10 0x7fa4ec1e4170 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa4e402b910 tx=0x7fa4e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.551+0000 7fa4e9600640 1 -- 172.21.15.141:0/2086503994 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e40042d0 con 0x7fa4ec15da10 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.551+0000 7fa4e9600640 1 -- 172.21.15.141:0/2086503994 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e4004430 con 0x7fa4ec15da10 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.551+0000 7fa4e9600640 1 -- 172.21.15.141:0/2086503994 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4003850 con 0x7fa4ec15da10 2024-01-16T15:22:31.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.551+0000 7fa4f3e00640 1 -- 172.21.15.141:0/2086503994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec149dd0 con 0x7fa4ec15da10 2024-01-16T15:22:31.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.551+0000 7fa4f3e00640 1 -- 172.21.15.141:0/2086503994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec14a020 con 0x7fa4ec15da10 2024-01-16T15:22:31.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.552+0000 7fa4f3e00640 1 -- 172.21.15.141:0/2086503994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec14dc20 con 0x7fa4ec15da10 2024-01-16T15:22:31.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.553+0000 7fa4e9600640 1 -- 172.21.15.141:0/2086503994 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fa4e40039b0 con 0x7fa4ec15da10 2024-01-16T15:22:31.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.554+0000 7fa4e9600640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c406c820 0x7fa4c406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:31.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.554+0000 7fa4e9600640 1 -- 172.21.15.141:0/2086503994 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa4e40aa930 con 0x7fa4ec15da10 2024-01-16T15:22:31.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.555+0000 7fa4f1400640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c406c820 0x7fa4c406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:31.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.556+0000 7fa4f1400640 1 --2- 172.21.15.141:0/2086503994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa4c406c820 0x7fa4c406ece0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc005c00 tx=0x7fa4dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:31.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.558+0000 7fa4e9600640 1 -- 172.21.15.141:0/2086503994 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4ec14dc20 con 0x7fa4ec15da10 2024-01-16T15:22:31.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.783+0000 7fa4f3e00640 1 -- 172.21.15.141:0/2086503994 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4ec14c7f0 con 0x7fa4c406c820 2024-01-16T15:22:31.792 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:31.798+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:31.793 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:31.798+0000 7fa4e9600640 1 -- 172.21.15.141:0/2086503994 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa4dc00dc90 con 0x7fa4c406c820 2024-01-16T15:22:31.793 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:31.793 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:33.805 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:34.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.035+0000 7f3535a00640 1 Processor -- start 2024-01-16T15:22:34.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.035+0000 7f3535a00640 1 -- start start 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.036+0000 7f3535a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3530143ae0 0x7f35300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.036+0000 7f3535a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f353015e3b0 0x7f35300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.036+0000 7f3535a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35300cfc30 con 0x7f35300cf7d0 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.036+0000 7f3535a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35300b1ee0 con 0x7f3530143ae0 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.036+0000 7f3535a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35300b2780 con 0x7f353015e3b0 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.036+0000 7f352ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3530143ae0 0x7f35300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.037+0000 7f352ea00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3530143ae0 0x7f35300cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58888/0 (socket says 172.21.15.141:58888) 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.037+0000 7f352ea00640 1 -- 172.21.15.141:0/734500811 learned_addr learned my addr 172.21.15.141:0/734500811 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.037+0000 7f352ea00640 1 -- 172.21.15.141:0/734500811 >> v2:172.21.15.188:3300/0 conn(0x7f353015e3b0 msgr2=0x7f35300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.037+0000 7f352f400640 1 --2- 172.21.15.141:0/734500811 >> v2:172.21.15.188:3300/0 conn(0x7f353015e3b0 0x7f35300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.037+0000 7f352ea00640 1 --2- 172.21.15.141:0/734500811 >> v2:172.21.15.188:3300/0 conn(0x7f353015e3b0 0x7f35300702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.037+0000 7f352ea00640 1 -- 172.21.15.141:0/734500811 >> v1:172.21.15.188:6789/0 conn(0x7f35300cf7d0 legacy=0x7f353015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:34.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.037+0000 7f352ea00640 1 -- 172.21.15.141:0/734500811 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353015e990 con 0x7f3530143ae0 2024-01-16T15:22:34.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.038+0000 7f352ea00640 1 --2- 172.21.15.141:0/734500811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3530143ae0 0x7f35300cf290 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f353006e4f0 tx=0x7f352002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf7ee237e88c931 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:34.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.038+0000 7f352e000640 1 -- 172.21.15.141:0/734500811 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520031870 con 0x7f3530143ae0 2024-01-16T15:22:34.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.038+0000 7f352e000640 1 -- 172.21.15.141:0/734500811 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35200319d0 con 0x7f3530143ae0 2024-01-16T15:22:34.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.038+0000 7f352e000640 1 -- 172.21.15.141:0/734500811 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520031d60 con 0x7f3530143ae0 2024-01-16T15:22:34.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 -- 172.21.15.141:0/734500811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3530143ae0 msgr2=0x7f35300cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 --2- 172.21.15.141:0/734500811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3530143ae0 0x7f35300cf290 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f353006e4f0 tx=0x7f352002b910 comp rx=0 tx=0).stop 2024-01-16T15:22:34.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 -- 172.21.15.141:0/734500811 shutdown_connections 2024-01-16T15:22:34.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 --2- 172.21.15.141:0/734500811 >> v2:172.21.15.188:3300/0 conn(0x7f353015e3b0 0x7f35300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 --2- 172.21.15.141:0/734500811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3530143ae0 0x7f35300cf290 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 -- 172.21.15.141:0/734500811 >> 172.21.15.141:0/734500811 conn(0x7f353006a800 msgr2=0x7f353006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:34.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 -- 172.21.15.141:0/734500811 shutdown_connections 2024-01-16T15:22:34.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.039+0000 7f3535a00640 1 -- 172.21.15.141:0/734500811 wait complete. 2024-01-16T15:22:34.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.041+0000 7f3535a00640 1 Processor -- start 2024-01-16T15:22:34.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.041+0000 7f3535a00640 1 -- start start 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f3535a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35300cf7d0 0x7f35301677f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f3535a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530143ae0 0x7f35301593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f3535a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f353015e3b0 0x7f353013ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f3535a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35300cd300 con 0x7f35300cf7d0 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f3535a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35300cd5e0 con 0x7f3530143ae0 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f3535a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35300cd8c0 con 0x7f353015e3b0 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35300cf7d0 0x7f35301677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35300cf7d0 0x7f35301677f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58904/0 (socket says 172.21.15.141:58904) 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 -- 172.21.15.141:0/3179255298 learned_addr learned my addr 172.21.15.141:0/3179255298 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 -- 172.21.15.141:0/3179255298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f353015e3b0 msgr2=0x7f353013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 --2- 172.21.15.141:0/3179255298 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f353015e3b0 0x7f353013ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 -- 172.21.15.141:0/3179255298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530143ae0 msgr2=0x7f35301593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 --2- 172.21.15.141:0/3179255298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530143ae0 0x7f35301593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.042+0000 7f352ea00640 1 -- 172.21.15.141:0/3179255298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35200094d0 con 0x7f35300cf7d0 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.043+0000 7f352ea00640 1 --2- 172.21.15.141:0/3179255298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35300cf7d0 0x7f35301677f0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f352002b950 tx=0x7f35200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.043+0000 7f3527400640 1 -- 172.21.15.141:0/3179255298 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520031860 con 0x7f35300cf7d0 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.043+0000 7f3527400640 1 -- 172.21.15.141:0/3179255298 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35200443f0 con 0x7f35300cf7d0 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.043+0000 7f3527400640 1 -- 172.21.15.141:0/3179255298 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35200447a0 con 0x7f35300cf7d0 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.043+0000 7f3535a00640 1 -- 172.21.15.141:0/3179255298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35300cdba0 con 0x7f35300cf7d0 2024-01-16T15:22:34.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.043+0000 7f3535a00640 1 -- 172.21.15.141:0/3179255298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35300ccfd0 con 0x7f35300cf7d0 2024-01-16T15:22:34.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.044+0000 7f3535a00640 1 -- 172.21.15.141:0/3179255298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35300cdde0 con 0x7f35300cf7d0 2024-01-16T15:22:34.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.045+0000 7f3527400640 1 -- 172.21.15.141:0/3179255298 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f352004d450 con 0x7f35300cf7d0 2024-01-16T15:22:34.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.046+0000 7f3527400640 1 --2- 172.21.15.141:0/3179255298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f350806c820 0x7f350806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:34.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.046+0000 7f3527400640 1 -- 172.21.15.141:0/3179255298 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f352002d040 con 0x7f35300cf7d0 2024-01-16T15:22:34.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.047+0000 7f3525e00640 1 --2- 172.21.15.141:0/3179255298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f350806c820 0x7f350806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:34.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.048+0000 7f3525e00640 1 --2- 172.21.15.141:0/3179255298 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f350806c820 0x7f350806ece0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3518001560 tx=0x7f3518007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:34.045 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.051+0000 7f3527400640 1 -- 172.21.15.141:0/3179255298 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35300cdde0 con 0x7f35300cf7d0 2024-01-16T15:22:34.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.273+0000 7f3535a00640 1 -- 172.21.15.141:0/3179255298 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35300b2410 con 0x7f350806c820 2024-01-16T15:22:34.284 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:34.290+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:34.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:34.290+0000 7f3527400640 1 -- 172.21.15.141:0/3179255298 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f35180011f0 con 0x7f350806c820 2024-01-16T15:22:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:34.285 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:36.308 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:36.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.538+0000 7f4b7f800640 1 Processor -- start 2024-01-16T15:22:36.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.538+0000 7f4b7f800640 1 -- start start 2024-01-16T15:22:36.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.540+0000 7f4b7f800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4b780cf9a0 0x7f4b78166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.540+0000 7f4b7f800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7815c5f0 con 0x7f4b7813df80 2024-01-16T15:22:36.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.540+0000 7f4b7f800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7806c470 con 0x7f4b7813e360 2024-01-16T15:22:36.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.540+0000 7f4b7f800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7806c750 con 0x7f4b780cf9a0 2024-01-16T15:22:36.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.540+0000 7f4b7ce00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f4b7813df80 0x7f4b7815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:46698/0 (socket says 172.21.15.141:46698) 2024-01-16T15:22:36.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.540+0000 7f4b7ce00640 1 -- 172.21.15.141:0/1260727236 learned_addr learned my addr 172.21.15.141:0/1260727236 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:36.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.541+0000 7f4b7d800640 1 --2- 172.21.15.141:0/1260727236 >> v2:172.21.15.141:3300/0 conn(0x7f4b780cf9a0 0x7f4b78166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.541+0000 7f4b77400640 1 -- 172.21.15.141:0/1260727236 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414265797 0 0) 0x7f4b7815c5f0 con 0x7f4b7813df80 2024-01-16T15:22:36.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.541+0000 7f4b77400640 1 -- 172.21.15.141:0/1260727236 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b58003520 con 0x7f4b7813df80 2024-01-16T15:22:36.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.541+0000 7f4b7d800640 1 -- 172.21.15.141:0/1260727236 >> v1:172.21.15.188:6789/0 conn(0x7f4b7813e360 legacy=0x7f4b780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.541+0000 7f4b7d800640 1 -- 172.21.15.141:0/1260727236 >> v1:172.21.15.141:6789/0 conn(0x7f4b7813df80 legacy=0x7f4b7815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.541+0000 7f4b7d800640 1 -- 172.21.15.141:0/1260727236 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7806ca30 con 0x7f4b780cf9a0 2024-01-16T15:22:36.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.542+0000 7f4b7d800640 1 --2- 172.21.15.141:0/1260727236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b780cf9a0 0x7f4b78166030 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009d90 tx=0x7f4b7002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5ac960714bfdc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:36.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b77400640 1 -- 172.21.15.141:0/1260727236 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7002cd00 con 0x7f4b780cf9a0 2024-01-16T15:22:36.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b77400640 1 -- 172.21.15.141:0/1260727236 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7002ce60 con 0x7f4b780cf9a0 2024-01-16T15:22:36.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b77400640 1 -- 172.21.15.141:0/1260727236 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70034660 con 0x7f4b780cf9a0 2024-01-16T15:22:36.538 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b7f800640 1 -- 172.21.15.141:0/1260727236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b780cf9a0 msgr2=0x7f4b78166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b7f800640 1 --2- 172.21.15.141:0/1260727236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b780cf9a0 0x7f4b78166030 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009d90 tx=0x7f4b7002bf50 comp rx=0 tx=0).stop 2024-01-16T15:22:36.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b7f800640 1 -- 172.21.15.141:0/1260727236 shutdown_connections 2024-01-16T15:22:36.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b7f800640 1 --2- 172.21.15.141:0/1260727236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b780cf9a0 0x7f4b78166030 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b7f800640 1 -- 172.21.15.141:0/1260727236 >> 172.21.15.141:0/1260727236 conn(0x7f4b7806a800 msgr2=0x7f4b7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:36.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b7f800640 1 -- 172.21.15.141:0/1260727236 shutdown_connections 2024-01-16T15:22:36.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.543+0000 7f4b7f800640 1 -- 172.21.15.141:0/1260727236 wait complete. 2024-01-16T15:22:36.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.545+0000 7f4b7f800640 1 Processor -- start 2024-01-16T15:22:36.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.545+0000 7f4b7f800640 1 -- start start 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.546+0000 7f4b7f800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b780cf9a0 0x7f4b7815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.546+0000 7f4b7f800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813df80 0x7f4b780a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.546+0000 7f4b7f800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813e360 0x7f4b78152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.546+0000 7f4b7f800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b78071ca0 con 0x7f4b7813df80 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.546+0000 7f4b7f800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b78071020 con 0x7f4b780cf9a0 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.546+0000 7f4b7f800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b78071300 con 0x7f4b7813e360 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.546+0000 7f4b77e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813df80 0x7f4b780a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813df80 0x7f4b780a8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58930/0 (socket says 172.21.15.141:58930) 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 -- 172.21.15.141:0/3965592812 learned_addr learned my addr 172.21.15.141:0/3965592812 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b7d800640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813e360 0x7f4b78152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b7ce00640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b780cf9a0 0x7f4b7815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 -- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813e360 msgr2=0x7f4b78152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813e360 0x7f4b78152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 -- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b780cf9a0 msgr2=0x7f4b7815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b780cf9a0 0x7f4b7815a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 -- 172.21.15.141:0/3965592812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70009a70 con 0x7f4b7813df80 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b7d800640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4b7813e360 0x7f4b78152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b77e00640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4b7813df80 0x7f4b780a8e40 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c400 tx=0x7f4b6000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.547+0000 7f4b74c00640 1 -- 172.21.15.141:0/3965592812 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b600052d0 con 0x7f4b7813df80 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.548+0000 7f4b74c00640 1 -- 172.21.15.141:0/3965592812 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60005430 con 0x7f4b7813df80 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.548+0000 7f4b7f800640 1 -- 172.21.15.141:0/3965592812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b780715e0 con 0x7f4b7813df80 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.548+0000 7f4b74c00640 1 -- 172.21.15.141:0/3965592812 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60003830 con 0x7f4b7813df80 2024-01-16T15:22:36.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.548+0000 7f4b7f800640 1 -- 172.21.15.141:0/3965592812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b78070cd0 con 0x7f4b7813df80 2024-01-16T15:22:36.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.549+0000 7f4b7f800640 1 -- 172.21.15.141:0/3965592812 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b78071de0 con 0x7f4b7813df80 2024-01-16T15:22:36.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.550+0000 7f4b74c00640 1 -- 172.21.15.141:0/3965592812 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f4b60003990 con 0x7f4b7813df80 2024-01-16T15:22:36.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.551+0000 7f4b74c00640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006c850 0x7f4b5006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.551+0000 7f4b74c00640 1 -- 172.21.15.141:0/3965592812 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4b600882b0 con 0x7f4b7813df80 2024-01-16T15:22:36.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.551+0000 7f4b7ce00640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006c850 0x7f4b5006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.552+0000 7f4b7ce00640 1 --2- 172.21.15.141:0/3965592812 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f4b5006c850 0x7f4b5006ed10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4b680011e0 tx=0x7f4b68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:36.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.555+0000 7f4b74c00640 1 -- 172.21.15.141:0/3965592812 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b60054770 con 0x7f4b7813df80 2024-01-16T15:22:36.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.780+0000 7f4b7f800640 1 -- 172.21.15.141:0/3965592812 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b7806cff0 con 0x7f4b5006c850 2024-01-16T15:22:36.789 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:36.795+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:36.790 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:36.795+0000 7f4b74c00640 1 -- 172.21.15.141:0/3965592812 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4b680058d0 con 0x7f4b5006c850 2024-01-16T15:22:36.790 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:36.790 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:38.802 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:39.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.033+0000 7fc984a00640 1 Processor -- start 2024-01-16T15:22:39.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.033+0000 7fc984a00640 1 -- start start 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.034+0000 7fc984a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc97c143ae0 0x7fc97c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.034+0000 7fc984a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c15e3b0 0x7fc97c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.034+0000 7fc984a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c0cfc30 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.034+0000 7fc984a00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97c0b1ee0 con 0x7fc97c143ae0 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.034+0000 7fc984a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0b2780 con 0x7fc97c15e3b0 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc97c143ae0 0x7fc97c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc97c143ae0 0x7fc97c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58938/0 (socket says 172.21.15.141:58938) 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982000640 1 -- 172.21.15.141:0/1627637541 learned_addr learned my addr 172.21.15.141:0/1627637541 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:39.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982a00640 1 --2- 172.21.15.141:0/1627637541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c15e3b0 0x7fc97c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982000640 1 -- 172.21.15.141:0/1627637541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c15e3b0 msgr2=0x7fc97c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:39.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982000640 1 --2- 172.21.15.141:0/1627637541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c15e3b0 0x7fc97c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982000640 1 -- 172.21.15.141:0/1627637541 >> v1:172.21.15.188:6789/0 conn(0x7fc97c0cf7d0 legacy=0x7fc97c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:39.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.035+0000 7fc982000640 1 -- 172.21.15.141:0/1627637541 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c15e990 con 0x7fc97c143ae0 2024-01-16T15:22:39.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.036+0000 7fc982000640 1 --2- 172.21.15.141:0/1627637541 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c143ae0 0x7fc97c0cf290 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009870 tx=0x7fc96c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8669b1a970796cea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:39.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc980c00640 1 -- 172.21.15.141:0/1627637541 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c031640 con 0x7fc97c143ae0 2024-01-16T15:22:39.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc980c00640 1 -- 172.21.15.141:0/1627637541 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c0317a0 con 0x7fc97c143ae0 2024-01-16T15:22:39.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc980c00640 1 -- 172.21.15.141:0/1627637541 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c031b30 con 0x7fc97c143ae0 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc984a00640 1 -- 172.21.15.141:0/1627637541 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c143ae0 msgr2=0x7fc97c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc984a00640 1 --2- 172.21.15.141:0/1627637541 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c143ae0 0x7fc97c0cf290 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009870 tx=0x7fc96c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc984a00640 1 -- 172.21.15.141:0/1627637541 shutdown_connections 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc984a00640 1 --2- 172.21.15.141:0/1627637541 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c15e3b0 0x7fc97c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc984a00640 1 --2- 172.21.15.141:0/1627637541 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c143ae0 0x7fc97c0cf290 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc984a00640 1 -- 172.21.15.141:0/1627637541 >> 172.21.15.141:0/1627637541 conn(0x7fc97c06a800 msgr2=0x7fc97c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.037+0000 7fc984a00640 1 -- 172.21.15.141:0/1627637541 shutdown_connections 2024-01-16T15:22:39.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.038+0000 7fc984a00640 1 -- 172.21.15.141:0/1627637541 wait complete. 2024-01-16T15:22:39.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.039+0000 7fc984a00640 1 Processor -- start 2024-01-16T15:22:39.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.039+0000 7fc984a00640 1 -- start start 2024-01-16T15:22:39.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.040+0000 7fc984a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c0cf7d0 0x7fc97c0aa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.040+0000 7fc984a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c143ae0 0x7fc97c158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.040+0000 7fc984a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc97c15e3b0 0x7fc97c0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.040+0000 7fc984a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0a9b90 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.040+0000 7fc984a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0a8f80 con 0x7fc97c15e3b0 2024-01-16T15:22:39.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.040+0000 7fc984a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0a9260 con 0x7fc97c143ae0 2024-01-16T15:22:39.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c0cf7d0 0x7fc97c0aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c0cf7d0 0x7fc97c0aa020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58948/0 (socket says 172.21.15.141:58948) 2024-01-16T15:22:39.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 -- 172.21.15.141:0/681226145 learned_addr learned my addr 172.21.15.141:0/681226145 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:39.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc981600640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c143ae0 0x7fc97c158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982a00640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc97c15e3b0 0x7fc97c0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 -- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c143ae0 msgr2=0x7fc97c158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc97c143ae0 0x7fc97c158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 -- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc97c15e3b0 msgr2=0x7fc97c0a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc97c15e3b0 0x7fc97c0a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 -- 172.21.15.141:0/681226145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c0094d0 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.041+0000 7fc982000640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc97c0cf7d0 0x7fc97c0aa020 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009870 tx=0x7fc96c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.042+0000 7fc97a000640 1 -- 172.21.15.141:0/681226145 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c03ac60 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.042+0000 7fc97a000640 1 -- 172.21.15.141:0/681226145 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c03adc0 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.042+0000 7fc97a000640 1 -- 172.21.15.141:0/681226145 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c044630 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.042+0000 7fc984a00640 1 -- 172.21.15.141:0/681226145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c0a9540 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.042+0000 7fc984a00640 1 -- 172.21.15.141:0/681226145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c159900 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.043+0000 7fc984a00640 1 -- 172.21.15.141:0/681226145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c15e1d0 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.044+0000 7fc97a000640 1 -- 172.21.15.141:0/681226145 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fc96c02f050 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.045+0000 7fc97a000640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc95406c820 0x7fc95406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:39.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.045+0000 7fc97a000640 1 -- 172.21.15.141:0/681226145 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc96c0aabe0 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.045+0000 7fc981600640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc95406c820 0x7fc95406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:39.041 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.047+0000 7fc981600640 1 --2- 172.21.15.141:0/681226145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc95406c820 0x7fc95406ece0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc97c15d7b0 tx=0x7fc964009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:39.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.050+0000 7fc97a000640 1 -- 172.21.15.141:0/681226145 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc96c0770a0 con 0x7fc97c0cf7d0 2024-01-16T15:22:39.269 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.274+0000 7fc984a00640 1 -- 172.21.15.141:0/681226145 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc97c0b2410 con 0x7fc95406c820 2024-01-16T15:22:39.282 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:39.288+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:39.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:39.288+0000 7fc97a000640 1 -- 172.21.15.141:0/681226145 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc964005030 con 0x7fc95406c820 2024-01-16T15:22:39.283 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:39.283 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:39.752 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:39.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:41.303 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:41.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.535+0000 7fb620600640 1 Processor -- start 2024-01-16T15:22:41.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.535+0000 7fb620600640 1 -- start start 2024-01-16T15:22:41.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.536+0000 7fb620600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6180d0280 0x7fb6180ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.536+0000 7fb620600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb6180ce6f0 0x7fb6180b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.536+0000 7fb620600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6180b6750 con 0x7fb61815d590 2024-01-16T15:22:41.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.536+0000 7fb620600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6180cf1f0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.536+0000 7fb620600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180cef10 con 0x7fb6180d0280 2024-01-16T15:22:41.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61e600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb6180ce6f0 0x7fb6180b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61d200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6180d0280 0x7fb6180ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61e600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb6180ce6f0 0x7fb6180b6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.141:48230/0 (socket says 172.21.15.141:48230) 2024-01-16T15:22:41.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61e600640 1 -- 172.21.15.141:0/3830812708 learned_addr learned my addr 172.21.15.141:0/3830812708 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:41.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61e600640 1 -- 172.21.15.141:0/3830812708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6180d0280 msgr2=0x7fb6180ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:41.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61e600640 1 --2- 172.21.15.141:0/3830812708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6180d0280 0x7fb6180ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61e600640 1 -- 172.21.15.141:0/3830812708 >> v1:172.21.15.188:6789/0 conn(0x7fb61815d590 legacy=0x7fb6180cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:41.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.537+0000 7fb61e600640 1 -- 172.21.15.141:0/3830812708 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6180b6100 con 0x7fb6180ce6f0 2024-01-16T15:22:41.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.539+0000 7fb61e600640 1 --2- 172.21.15.141:0/3830812708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180ce6f0 0x7fb6180b6be0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb608009870 tx=0x7fb60802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc55d801fe5cd8aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:41.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.539+0000 7fb617e00640 1 -- 172.21.15.141:0/3830812708 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608030840 con 0x7fb6180ce6f0 2024-01-16T15:22:41.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.539+0000 7fb617e00640 1 -- 172.21.15.141:0/3830812708 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6080309a0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.539+0000 7fb617e00640 1 -- 172.21.15.141:0/3830812708 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608030d30 con 0x7fb6180ce6f0 2024-01-16T15:22:41.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 -- 172.21.15.141:0/3830812708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180ce6f0 msgr2=0x7fb6180b6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:41.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 --2- 172.21.15.141:0/3830812708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180ce6f0 0x7fb6180b6be0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb608009870 tx=0x7fb60802b980 comp rx=0 tx=0).stop 2024-01-16T15:22:41.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 -- 172.21.15.141:0/3830812708 shutdown_connections 2024-01-16T15:22:41.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 --2- 172.21.15.141:0/3830812708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180ce6f0 0x7fb6180b6be0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.535 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 --2- 172.21.15.141:0/3830812708 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6180d0280 0x7fb6180ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 -- 172.21.15.141:0/3830812708 >> 172.21.15.141:0/3830812708 conn(0x7fb61809c540 msgr2=0x7fb6180d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:41.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 -- 172.21.15.141:0/3830812708 shutdown_connections 2024-01-16T15:22:41.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.540+0000 7fb620600640 1 -- 172.21.15.141:0/3830812708 wait complete. 2024-01-16T15:22:41.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.542+0000 7fb620600640 1 Processor -- start 2024-01-16T15:22:41.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.542+0000 7fb620600640 1 -- start start 2024-01-16T15:22:41.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb620600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6180ce6f0 0x7fb61815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb620600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180d0280 0x7fb61809f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb620600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb61815d590 0x7fb61809cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb620600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180cb6d0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb620600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180cc250 con 0x7fb6180d0280 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb620600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180cb920 con 0x7fb61815d590 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb61dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6180ce6f0 0x7fb61815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb61dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6180ce6f0 0x7fb61815b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:58954/0 (socket says 172.21.15.141:58954) 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb61dc00640 1 -- 172.21.15.141:0/2594010980 learned_addr learned my addr 172.21.15.141:0/2594010980 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb61e600640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb61815d590 0x7fb61809cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.543+0000 7fb61d200640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180d0280 0x7fb61809f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb61dc00640 1 -- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb61815d590 msgr2=0x7fb61809cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb61dc00640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb61815d590 0x7fb61809cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb61dc00640 1 -- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180d0280 msgr2=0x7fb61809f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb61dc00640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180d0280 0x7fb61809f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb61dc00640 1 -- 172.21.15.141:0/2594010980 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6080094d0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb61d200640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6180d0280 0x7fb61809f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:41.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb61dc00640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6180ce6f0 0x7fb61815b950 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fb61000c870 tx=0x7fb61000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb615600640 1 -- 172.21.15.141:0/2594010980 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610003b70 con 0x7fb6180ce6f0 2024-01-16T15:22:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb615600640 1 -- 172.21.15.141:0/2594010980 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb610003cf0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb615600640 1 -- 172.21.15.141:0/2594010980 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61000aa90 con 0x7fb6180ce6f0 2024-01-16T15:22:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb620600640 1 -- 172.21.15.141:0/2594010980 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6180cbc00 con 0x7fb6180ce6f0 2024-01-16T15:22:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.544+0000 7fb620600640 1 -- 172.21.15.141:0/2594010980 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61809f630 con 0x7fb6180ce6f0 2024-01-16T15:22:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.545+0000 7fb620600640 1 -- 172.21.15.141:0/2594010980 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6180675d0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.547+0000 7fb615600640 1 -- 172.21.15.141:0/2594010980 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fb61000abf0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.547+0000 7fb615600640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb5f006c820 0x7fb5f006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.547+0000 7fb615600640 1 -- 172.21.15.141:0/2594010980 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb610088bd0 con 0x7fb6180ce6f0 2024-01-16T15:22:41.543 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.548+0000 7fb61d200640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb5f006c820 0x7fb5f006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.549+0000 7fb61d200640 1 --2- 172.21.15.141:0/2594010980 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb5f006c820 0x7fb5f006ece0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb61815be90 tx=0x7fb600008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:41.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.552+0000 7fb615600640 1 -- 172.21.15.141:0/2594010980 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb610055090 con 0x7fb6180ce6f0 2024-01-16T15:22:41.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.775+0000 7fb620600640 1 -- 172.21.15.141:0/2594010980 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb61815d970 con 0x7fb5f006c820 2024-01-16T15:22:41.783 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:41.788+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:41.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:41.789+0000 7fb615600640 1 -- 172.21.15.141:0/2594010980 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb600009730 con 0x7fb5f006c820 2024-01-16T15:22:41.784 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:41.784 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:43.801 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:44.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.026+0000 7fb7b1e00640 1 Processor -- start 2024-01-16T15:22:44.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.026+0000 7fb7b1e00640 1 -- start start 2024-01-16T15:22:44.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7b1e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac0ad980 0x7fb7ac0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7b1e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb7ac15d500 0x7fb7ac0b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7b1e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0b2ef0 con 0x7fb7ac15d120 2024-01-16T15:22:44.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7b1e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ac06b360 con 0x7fb7ac15d500 2024-01-16T15:22:44.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7b1e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac06b640 con 0x7fb7ac0ad980 2024-01-16T15:22:44.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7ab400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb7ac15d500 0x7fb7ac0b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7ab400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb7ac15d500 0x7fb7ac0b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55062/0 (socket says 172.21.15.141:55062) 2024-01-16T15:22:44.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.028+0000 7fb7ab400640 1 -- 172.21.15.141:0/4200596926 learned_addr learned my addr 172.21.15.141:0/4200596926 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:44.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.029+0000 7fb7a3e00640 1 --2- 172.21.15.141:0/4200596926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac0ad980 0x7fb7ac0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.029+0000 7fb7ab400640 1 -- 172.21.15.141:0/4200596926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac0ad980 msgr2=0x7fb7ac0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:44.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.029+0000 7fb7ab400640 1 --2- 172.21.15.141:0/4200596926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac0ad980 0x7fb7ac0b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.029+0000 7fb7ab400640 1 -- 172.21.15.141:0/4200596926 >> v1:172.21.15.188:6789/0 conn(0x7fb7ac15d120 legacy=0x7fb7ac0ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:44.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.029+0000 7fb7ab400640 1 -- 172.21.15.141:0/4200596926 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac06b920 con 0x7fb7ac15d500 2024-01-16T15:22:44.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.029+0000 7fb7a3e00640 1 --2- 172.21.15.141:0/4200596926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac0ad980 0x7fb7ac0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.030+0000 7fb7ab400640 1 --2- 172.21.15.141:0/4200596926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac15d500 0x7fb7ac0b3910 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009e50 tx=0x7fb7a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb160f903e7d0e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.030+0000 7fb7aa000640 1 -- 172.21.15.141:0/4200596926 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a402ca70 con 0x7fb7ac15d500 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.030+0000 7fb7aa000640 1 -- 172.21.15.141:0/4200596926 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a402cbd0 con 0x7fb7ac15d500 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.030+0000 7fb7aa000640 1 -- 172.21.15.141:0/4200596926 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a40343f0 con 0x7fb7ac15d500 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 -- 172.21.15.141:0/4200596926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac15d500 msgr2=0x7fb7ac0b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 --2- 172.21.15.141:0/4200596926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac15d500 0x7fb7ac0b3910 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009e50 tx=0x7fb7a402bed0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 -- 172.21.15.141:0/4200596926 shutdown_connections 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 --2- 172.21.15.141:0/4200596926 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac15d500 0x7fb7ac0b3910 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 --2- 172.21.15.141:0/4200596926 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac0ad980 0x7fb7ac0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 -- 172.21.15.141:0/4200596926 >> 172.21.15.141:0/4200596926 conn(0x7fb7ac0cf8c0 msgr2=0x7fb7ac0cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:44.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 -- 172.21.15.141:0/4200596926 shutdown_connections 2024-01-16T15:22:44.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.031+0000 7fb7b1e00640 1 -- 172.21.15.141:0/4200596926 wait complete. 2024-01-16T15:22:44.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.033+0000 7fb7b1e00640 1 Processor -- start 2024-01-16T15:22:44.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.033+0000 7fb7b1e00640 1 -- start start 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.034+0000 7fb7b1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac0ad980 0x7fb7ac147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.034+0000 7fb7b1e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac15d120 0x7fb7ac1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.034+0000 7fb7b1e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac15d500 0x7fb7ac158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.034+0000 7fb7b1e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac153c10 con 0x7fb7ac0ad980 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.034+0000 7fb7b1e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac153ed0 con 0x7fb7ac15d500 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.034+0000 7fb7b1e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1541b0 con 0x7fb7ac15d120 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.034+0000 7fb7aaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac0ad980 0x7fb7ac147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac0ad980 0x7fb7ac147210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55078/0 (socket says 172.21.15.141:55078) 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 -- 172.21.15.141:0/3563265631 learned_addr learned my addr 172.21.15.141:0/3563265631 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7ab400640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac15d500 0x7fb7ac158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7a3e00640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac15d120 0x7fb7ac1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 -- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac15d120 msgr2=0x7fb7ac1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb7ac15d120 0x7fb7ac1581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 -- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac15d500 msgr2=0x7fb7ac158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac15d500 0x7fb7ac158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 -- 172.21.15.141:0/3563265631 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a4009b30 con 0x7fb7ac0ad980 2024-01-16T15:22:44.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7aaa00640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb7ac0ad980 0x7fb7ac147210 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00c430 tx=0x7fb79c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:44.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.035+0000 7fb7a3400640 1 -- 172.21.15.141:0/3563265631 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c011790 con 0x7fb7ac0ad980 2024-01-16T15:22:44.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.036+0000 7fb7a3400640 1 -- 172.21.15.141:0/3563265631 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c0118f0 con 0x7fb7ac0ad980 2024-01-16T15:22:44.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.036+0000 7fb7b1e00640 1 -- 172.21.15.141:0/3563265631 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac154490 con 0x7fb7ac0ad980 2024-01-16T15:22:44.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.036+0000 7fb7a3400640 1 -- 172.21.15.141:0/3563265631 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c011ce0 con 0x7fb7ac0ad980 2024-01-16T15:22:44.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.036+0000 7fb7b1e00640 1 -- 172.21.15.141:0/3563265631 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac154c90 con 0x7fb7ac0ad980 2024-01-16T15:22:44.032 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.037+0000 7fb7b1e00640 1 -- 172.21.15.141:0/3563265631 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac0ac8d0 con 0x7fb7ac0ad980 2024-01-16T15:22:44.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.038+0000 7fb7a3400640 1 -- 172.21.15.141:0/3563265631 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fb79c01c460 con 0x7fb7ac0ad980 2024-01-16T15:22:44.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.038+0000 7fb7a3400640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb78406c820 0x7fb78406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:44.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.039+0000 7fb7a3400640 1 -- 172.21.15.141:0/3563265631 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb79c088c60 con 0x7fb7ac0ad980 2024-01-16T15:22:44.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.039+0000 7fb7a3e00640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb78406c820 0x7fb78406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:44.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.040+0000 7fb7a3e00640 1 --2- 172.21.15.141:0/3563265631 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb78406c820 0x7fb78406ece0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac152fd0 tx=0x7fb794008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:44.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.043+0000 7fb7a3400640 1 -- 172.21.15.141:0/3563265631 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb79c055180 con 0x7fb7ac0ad980 2024-01-16T15:22:44.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.262+0000 7fb7b1e00640 1 -- 172.21.15.141:0/3563265631 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7ac154040 con 0x7fb78406c820 2024-01-16T15:22:44.270 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:44.275+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:44.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:44.276+0000 7fb7a3400640 1 -- 172.21.15.141:0/3563265631 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb7ac154040 con 0x7fb78406c820 2024-01-16T15:22:44.271 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:44.271 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:46.285 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:46.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.516+0000 7fc571400640 1 Processor -- start 2024-01-16T15:22:46.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.517+0000 7fc571400640 1 -- start start 2024-01-16T15:22:46.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.518+0000 7fc571400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc56c0cf9a0 0x7fc56c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.518+0000 7fc571400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56c071690 con 0x7fc56c14b2c0 2024-01-16T15:22:46.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.518+0000 7fc571400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56c071970 con 0x7fc56c070ea0 2024-01-16T15:22:46.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.518+0000 7fc571400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56c14b6a0 con 0x7fc56c0cf9a0 2024-01-16T15:22:46.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc56a800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc56c14b2c0 0x7fc56c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:48402/0 (socket says 172.21.15.141:48402) 2024-01-16T15:22:46.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc569e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc56c0cf9a0 0x7fc56c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc56a800640 1 -- 172.21.15.141:0/2420871611 learned_addr learned my addr 172.21.15.141:0/2420871611 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:46.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc569400640 1 -- 172.21.15.141:0/2420871611 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140933451 0 0) 0x7fc56c071690 con 0x7fc56c14b2c0 2024-01-16T15:22:46.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc569400640 1 -- 172.21.15.141:0/2420871611 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c003520 con 0x7fc56c14b2c0 2024-01-16T15:22:46.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc569e00640 1 -- 172.21.15.141:0/2420871611 >> v1:172.21.15.188:6789/0 conn(0x7fc56c070ea0 legacy=0x7fc56c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:46.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc569e00640 1 -- 172.21.15.141:0/2420871611 >> v1:172.21.15.141:6789/0 conn(0x7fc56c14b2c0 legacy=0x7fc56c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.519+0000 7fc569e00640 1 -- 172.21.15.141:0/2420871611 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56c071c50 con 0x7fc56c0cf9a0 2024-01-16T15:22:46.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.521+0000 7fc569e00640 1 --2- 172.21.15.141:0/2420871611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c0cf9a0 0x7fc56c070960 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fc554009d90 tx=0x7fc55402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af58a41f8b9e50a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:46.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.521+0000 7fc569400640 1 -- 172.21.15.141:0/2420871611 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55402cc60 con 0x7fc56c0cf9a0 2024-01-16T15:22:46.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.521+0000 7fc569400640 1 -- 172.21.15.141:0/2420871611 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55402cdc0 con 0x7fc56c0cf9a0 2024-01-16T15:22:46.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.521+0000 7fc569400640 1 -- 172.21.15.141:0/2420871611 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554034660 con 0x7fc56c0cf9a0 2024-01-16T15:22:46.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.521+0000 7fc571400640 1 -- 172.21.15.141:0/2420871611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c0cf9a0 msgr2=0x7fc56c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.521+0000 7fc571400640 1 --2- 172.21.15.141:0/2420871611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c0cf9a0 0x7fc56c070960 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fc554009d90 tx=0x7fc55402beb0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.522+0000 7fc571400640 1 -- 172.21.15.141:0/2420871611 shutdown_connections 2024-01-16T15:22:46.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.522+0000 7fc571400640 1 --2- 172.21.15.141:0/2420871611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c0cf9a0 0x7fc56c070960 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.522+0000 7fc571400640 1 -- 172.21.15.141:0/2420871611 >> 172.21.15.141:0/2420871611 conn(0x7fc56c06a800 msgr2=0x7fc56c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:46.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.522+0000 7fc571400640 1 -- 172.21.15.141:0/2420871611 shutdown_connections 2024-01-16T15:22:46.517 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.522+0000 7fc571400640 1 -- 172.21.15.141:0/2420871611 wait complete. 2024-01-16T15:22:46.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.523+0000 7fc571400640 1 Processor -- start 2024-01-16T15:22:46.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.523+0000 7fc571400640 1 -- start start 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.524+0000 7fc571400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c070ea0 0x7fc56c13f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.524+0000 7fc571400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc56c0cf9a0 0x7fc56c164450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.524+0000 7fc571400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c14b2c0 0x7fc56c142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.524+0000 7fc571400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c148f10 con 0x7fc56c070ea0 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.524+0000 7fc571400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c163fc0 con 0x7fc56c14b2c0 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.524+0000 7fc571400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c149160 con 0x7fc56c0cf9a0 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c070ea0 0x7fc56c13f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c070ea0 0x7fc56c13f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55108/0 (socket says 172.21.15.141:55108) 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 -- 172.21.15.141:0/715599302 learned_addr learned my addr 172.21.15.141:0/715599302 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 -- 172.21.15.141:0/715599302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc56c0cf9a0 msgr2=0x7fc56c164450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 --2- 172.21.15.141:0/715599302 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc56c0cf9a0 0x7fc56c164450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 -- 172.21.15.141:0/715599302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c14b2c0 msgr2=0x7fc56c142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 --2- 172.21.15.141:0/715599302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c14b2c0 0x7fc56c142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 -- 172.21.15.141:0/715599302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554009a70 con 0x7fc56c070ea0 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.525+0000 7fc56a800640 1 --2- 172.21.15.141:0/715599302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c070ea0 0x7fc56c13f840 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00c440 tx=0x7fc55c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.526+0000 7fc562a00640 1 -- 172.21.15.141:0/715599302 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c005ce0 con 0x7fc56c070ea0 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.526+0000 7fc562a00640 1 -- 172.21.15.141:0/715599302 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c005e40 con 0x7fc56c070ea0 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.526+0000 7fc562a00640 1 -- 172.21.15.141:0/715599302 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c0072b0 con 0x7fc56c070ea0 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.526+0000 7fc571400640 1 -- 172.21.15.141:0/715599302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56c149440 con 0x7fc56c070ea0 2024-01-16T15:22:46.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.526+0000 7fc571400640 1 -- 172.21.15.141:0/715599302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56c163c50 con 0x7fc56c070ea0 2024-01-16T15:22:46.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.527+0000 7fc571400640 1 -- 172.21.15.141:0/715599302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56c0675d0 con 0x7fc56c070ea0 2024-01-16T15:22:46.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.528+0000 7fc562a00640 1 -- 172.21.15.141:0/715599302 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7fc55c007410 con 0x7fc56c070ea0 2024-01-16T15:22:46.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.529+0000 7fc562a00640 1 --2- 172.21.15.141:0/715599302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54406c850 0x7fc54406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.529+0000 7fc562a00640 1 -- 172.21.15.141:0/715599302 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc55c088220 con 0x7fc56c070ea0 2024-01-16T15:22:46.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.530+0000 7fc569e00640 1 --2- 172.21.15.141:0/715599302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54406c850 0x7fc54406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.531+0000 7fc569e00640 1 --2- 172.21.15.141:0/715599302 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc54406c850 0x7fc54406ed10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc554036000 tx=0x7fc55402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:46.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.534+0000 7fc562a00640 1 -- 172.21.15.141:0/715599302 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc55c058290 con 0x7fc56c070ea0 2024-01-16T15:22:46.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.758+0000 7fc571400640 1 -- 172.21.15.141:0/715599302 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc56c0cbc40 con 0x7fc54406c850 2024-01-16T15:22:46.765 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:46.770+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:46.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:46.771+0000 7fc562a00640 1 -- 172.21.15.141:0/715599302 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc554034440 con 0x7fc54406c850 2024-01-16T15:22:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:46.766 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:48.779 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:49.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.014+0000 7f6414400640 1 Processor -- start 2024-01-16T15:22:49.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.015+0000 7f6414400640 1 -- start start 2024-01-16T15:22:49.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.016+0000 7f6414400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f640c13df80 0x7f640c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.016+0000 7f6414400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c0cf9a0 0x7f640c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.016+0000 7f6414400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c06c470 con 0x7f640c13e360 2024-01-16T15:22:49.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.016+0000 7f6414400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640c06c750 con 0x7f640c13df80 2024-01-16T15:22:49.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.016+0000 7f6414400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c06ca30 con 0x7f640c0cf9a0 2024-01-16T15:22:49.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.017+0000 7f6411a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f640c13df80 0x7f640c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.017+0000 7f6412400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c0cf9a0 0x7f640c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.017+0000 7f6412400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c0cf9a0 0x7f640c0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:54464/0 (socket says 172.21.15.141:54464) 2024-01-16T15:22:49.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.017+0000 7f6412400640 1 -- 172.21.15.141:0/1068074784 learned_addr learned my addr 172.21.15.141:0/1068074784 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:49.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.018+0000 7f6411a00640 1 -- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c0cf9a0 msgr2=0x7f640c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:49.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.018+0000 7f6411a00640 1 --2- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c0cf9a0 0x7f640c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.018+0000 7f6411a00640 1 -- 172.21.15.141:0/1068074784 >> v1:172.21.15.188:6789/0 conn(0x7f640c13e360 legacy=0x7f640c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.018+0000 7f6411a00640 1 -- 172.21.15.141:0/1068074784 --> v2:172.21.15.188:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c06cd10 con 0x7f640c13df80 2024-01-16T15:22:49.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.018+0000 7f6412400640 1 --2- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c0cf9a0 0x7f640c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:22:49.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.019+0000 7f6411a00640 1 --2- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13df80 0x7f640c15bee0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0097a0 tx=0x7f63fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2db3f23c4b21f00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:49.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.019+0000 7f640be00640 1 -- 172.21.15.141:0/1068074784 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc031830 con 0x7f640c13df80 2024-01-16T15:22:49.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.019+0000 7f640be00640 1 -- 172.21.15.141:0/1068074784 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc031990 con 0x7f640c13df80 2024-01-16T15:22:49.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.019+0000 7f640be00640 1 -- 172.21.15.141:0/1068074784 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc031d20 con 0x7f640c13df80 2024-01-16T15:22:49.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.020+0000 7f6414400640 1 -- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13df80 msgr2=0x7f640c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.020+0000 7f6414400640 1 --2- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13df80 0x7f640c15bee0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0097a0 tx=0x7f63fc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.020+0000 7f6414400640 1 -- 172.21.15.141:0/1068074784 shutdown_connections 2024-01-16T15:22:49.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.020+0000 7f6414400640 1 --2- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c0cf9a0 0x7f640c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.020+0000 7f6414400640 1 --2- 172.21.15.141:0/1068074784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13df80 0x7f640c15bee0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.020+0000 7f6414400640 1 -- 172.21.15.141:0/1068074784 >> 172.21.15.141:0/1068074784 conn(0x7f640c06a800 msgr2=0x7f640c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:49.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.020+0000 7f6414400640 1 -- 172.21.15.141:0/1068074784 shutdown_connections 2024-01-16T15:22:49.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.021+0000 7f6414400640 1 -- 172.21.15.141:0/1068074784 wait complete. 2024-01-16T15:22:49.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.022+0000 7f6414400640 1 Processor -- start 2024-01-16T15:22:49.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.022+0000 7f6414400640 1 -- start start 2024-01-16T15:22:49.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.023+0000 7f6414400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f640c0cf9a0 0x7f640c134380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.023+0000 7f6414400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c13df80 0x7f640c133a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.023+0000 7f6414400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13e360 0x7f640c137960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.023+0000 7f6414400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1349b0 con 0x7f640c0cf9a0 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.023+0000 7f6414400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0b00c0 con 0x7f640c13e360 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.023+0000 7f6414400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0b03a0 con 0x7f640c13df80 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f640c0cf9a0 0x7f640c134380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f640c0cf9a0 0x7f640c134380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55124/0 (socket says 172.21.15.141:55124) 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 -- 172.21.15.141:0/2088079641 learned_addr learned my addr 172.21.15.141:0/2088079641 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411000640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c13df80 0x7f640c133a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6412400640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13e360 0x7f640c137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 -- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c13df80 msgr2=0x7f640c133a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f640c13df80 0x7f640c133a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 -- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13e360 msgr2=0x7f640c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:49.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f640c13e360 0x7f640c137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.024+0000 7f6411a00640 1 -- 172.21.15.141:0/2088079641 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6404009b30 con 0x7f640c0cf9a0 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.025+0000 7f6411a00640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f640c0cf9a0 0x7f640c134380 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0098c0 tx=0x7f63fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.025+0000 7f6409600640 1 -- 172.21.15.141:0/2088079641 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03a6f0 con 0x7f640c0cf9a0 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.025+0000 7f6409600640 1 -- 172.21.15.141:0/2088079641 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03a850 con 0x7f640c0cf9a0 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.025+0000 7f6409600640 1 -- 172.21.15.141:0/2088079641 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03ac00 con 0x7f640c0cf9a0 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.025+0000 7f6414400640 1 -- 172.21.15.141:0/2088079641 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc009400 con 0x7f640c0cf9a0 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.025+0000 7f6414400640 1 -- 172.21.15.141:0/2088079641 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c0b0960 con 0x7f640c0cf9a0 2024-01-16T15:22:49.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.026+0000 7f6414400640 1 -- 172.21.15.141:0/2088079641 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c0675d0 con 0x7f640c0cf9a0 2024-01-16T15:22:49.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.027+0000 7f6409600640 1 -- 172.21.15.141:0/2088079641 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f63fc0393b0 con 0x7f640c0cf9a0 2024-01-16T15:22:49.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.028+0000 7f6409600640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63e406c850 0x7f63e406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:49.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.028+0000 7f6409600640 1 -- 172.21.15.141:0/2088079641 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f63fc0aa690 con 0x7f640c0cf9a0 2024-01-16T15:22:49.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.029+0000 7f6411000640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63e406c850 0x7f63e406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:49.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.030+0000 7f6411000640 1 --2- 172.21.15.141:0/2088079641 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f63e406c850 0x7f63e406ed10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f640c1387b0 tx=0x7f63f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:49.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.033+0000 7f6409600640 1 -- 172.21.15.141:0/2088079641 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63fc076c70 con 0x7f640c0cf9a0 2024-01-16T15:22:49.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.255+0000 7f6414400640 1 -- 172.21.15.141:0/2088079641 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f640c0b0230 con 0x7f63e406c850 2024-01-16T15:22:49.263 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:49.268+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:49.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:49.269+0000 7f6409600640 1 -- 172.21.15.141:0/2088079641 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f640c0b0230 con 0x7f63e406c850 2024-01-16T15:22:49.264 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:49.264 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:51.278 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:51.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.505+0000 7f86c6000640 1 Processor -- start 2024-01-16T15:22:51.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.505+0000 7f86c6000640 1 -- start start 2024-01-16T15:22:51.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.507+0000 7f86c6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c013df80 0x7f86c015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.507+0000 7f86c6000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c006c470 con 0x7f86c00cf9a0 2024-01-16T15:22:51.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.507+0000 7f86c6000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c006c750 con 0x7f86c013e360 2024-01-16T15:22:51.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.507+0000 7f86c6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c006ca30 con 0x7f86c013df80 2024-01-16T15:22:51.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.507+0000 7f86bfe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f86c00cf9a0 0x7f86c00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:48404/0 (socket says 172.21.15.141:48404) 2024-01-16T15:22:51.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.507+0000 7f86bfe00640 1 -- 172.21.15.141:0/2280249907 learned_addr learned my addr 172.21.15.141:0/2280249907 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:51.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bf400640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c013df80 0x7f86c015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:51.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805040664 0 0) 0x7f86c006c470 con 0x7f86c00cf9a0 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a0003520 con 0x7f86c00cf9a0 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 299421001 0 0) 0x7f86a0003520 con 0x7f86c00cf9a0 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c013df80 msgr2=0x7f86c015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c013df80 0x7f86c015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 >> v1:172.21.15.188:6789/0 conn(0x7f86c013e360 legacy=0x7f86c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c006cd10 con 0x7f86c00cf9a0 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f86b80019f0 con 0x7f86c00cf9a0 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 >> v1:172.21.15.141:6789/0 conn(0x7f86c00cf9a0 legacy=0x7f86c00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86a00039a0 0x7f86a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c013df80 0x7f86a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a0004d00 0x7f86a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c006ca30 con 0x7f86a00039a0 2024-01-16T15:22:51.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a000ba30 con 0x7f86a0004d00 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.508+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a000bbf0 con 0x7f86c013df80 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.509+0000 7f86bf400640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86a00039a0 0x7f86a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.509+0000 7f86bfe00640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a0004d00 0x7f86a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.509+0000 7f86bf400640 1 -- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c013df80 msgr2=0x7f86a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.509+0000 7f86bf400640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c013df80 0x7f86a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.509+0000 7f86bf400640 1 -- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a0004d00 msgr2=0x7f86a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.509+0000 7f86bf400640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a0004d00 0x7f86a0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:51.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.509+0000 7f86bf400640 1 -- 172.21.15.141:0/2280249907 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0003520 con 0x7f86a00039a0 2024-01-16T15:22:51.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.510+0000 7f86bf400640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86a00039a0 0x7f86a0003dc0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009a90 tx=0x7f86a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a32c548eddd6e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:51.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.510+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b002c6b0 con 0x7f86a00039a0 2024-01-16T15:22:51.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.510+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b002c810 con 0x7f86a00039a0 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86bea00640 1 -- 172.21.15.141:0/2280249907 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b002cba0 con 0x7f86a00039a0 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86c6000640 1 -- 172.21.15.141:0/2280249907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86a00039a0 msgr2=0x7f86a0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86c6000640 1 --2- 172.21.15.141:0/2280249907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86a00039a0 0x7f86a0003dc0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009a90 tx=0x7f86a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86bfe00640 1 -- 172.21.15.141:0/2280249907 reap_dead start 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86c6000640 1 -- 172.21.15.141:0/2280249907 shutdown_connections 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86c6000640 1 -- 172.21.15.141:0/2280249907 >> 172.21.15.141:0/2280249907 conn(0x7f86c006a800 msgr2=0x7f86c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86c6000640 1 -- 172.21.15.141:0/2280249907 shutdown_connections 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.511+0000 7f86c6000640 1 -- 172.21.15.141:0/2280249907 wait complete. 2024-01-16T15:22:51.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.513+0000 7f86c6000640 1 Processor -- start 2024-01-16T15:22:51.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.513+0000 7f86c6000640 1 -- start start 2024-01-16T15:22:51.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86c6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a00039a0 0x7f86c01517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86c6000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c00c9970 0x7f86c00c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86c6000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86c00c7ff0 0x7f86c0070d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86c6000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c014eec0 con 0x7f86c00c7ff0 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86c6000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c014f1a0 con 0x7f86a00039a0 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86c6000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c014f480 con 0x7f86c00c9970 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86bfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86c00c7ff0 0x7f86c0070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86bfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86c00c7ff0 0x7f86c0070d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:55146/0 (socket says 172.21.15.141:55146) 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.514+0000 7f86bfe00640 1 -- 172.21.15.141:0/1506043554 learned_addr learned my addr 172.21.15.141:0/1506043554 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86bf400640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a00039a0 0x7f86c01517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86b7e00640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c00c9970 0x7f86c00c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86bfe00640 1 -- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c00c9970 msgr2=0x7f86c00c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86bfe00640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86c00c9970 0x7f86c00c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86bfe00640 1 -- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a00039a0 msgr2=0x7f86c01517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:51.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86bfe00640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86a00039a0 0x7f86c01517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86bfe00640 1 -- 172.21.15.141:0/1506043554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c014f760 con 0x7f86c00c7ff0 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86bfe00640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86c00c7ff0 0x7f86c0070d90 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f86b8001370 tx=0x7f86b800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86b7400640 1 -- 172.21.15.141:0/1506043554 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b800de70 con 0x7f86c00c7ff0 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86b7400640 1 -- 172.21.15.141:0/1506043554 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b800bb80 con 0x7f86c00c7ff0 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86b7400640 1 -- 172.21.15.141:0/1506043554 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8007110 con 0x7f86c00c7ff0 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.515+0000 7f86c6000640 1 -- 172.21.15.141:0/1506043554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c014fa40 con 0x7f86c00c7ff0 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.516+0000 7f86c6000640 1 -- 172.21.15.141:0/1506043554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c0151ce0 con 0x7f86c00c7ff0 2024-01-16T15:22:51.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.517+0000 7f86c6000640 1 -- 172.21.15.141:0/1506043554 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c014ff60 con 0x7f86c00c7ff0 2024-01-16T15:22:51.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.518+0000 7f86b7400640 1 -- 172.21.15.141:0/1506043554 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f86b8007270 con 0x7f86c00c7ff0 2024-01-16T15:22:51.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.518+0000 7f86b7400640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f869806c850 0x7f869806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:51.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.519+0000 7f86b7400640 1 -- 172.21.15.141:0/1506043554 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f86b8088bc0 con 0x7f86c00c7ff0 2024-01-16T15:22:51.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.519+0000 7f86bf400640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f869806c850 0x7f869806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:51.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.520+0000 7f86bf400640 1 --2- 172.21.15.141:0/1506043554 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f869806c850 0x7f869806ed10 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f86b0036000 tx=0x7f86b0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:51.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.523+0000 7f86b7400640 1 -- 172.21.15.141:0/1506043554 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86c014ff60 con 0x7f86c00c7ff0 2024-01-16T15:22:51.740 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.745+0000 7f86c6000640 1 -- 172.21.15.141:0/1506043554 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f86c00aeb30 con 0x7f869806c850 2024-01-16T15:22:51.755 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:51.759+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:51.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:51.761+0000 7f86b7400640 1 -- 172.21.15.141:0/1506043554 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f86c00aeb30 con 0x7f869806c850 2024-01-16T15:22:51.756 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:51.756 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:53.769 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:53.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.001+0000 7f688e000640 1 Processor -- start 2024-01-16T15:22:53.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.001+0000 7f688e000640 1 -- start start 2024-01-16T15:22:53.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.002+0000 7f688e000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f688813df80 0x7f688815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:53.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f688e000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f68880cf9a0 0x7f68880cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:53.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f688e000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688806c470 con 0x7f688813e360 2024-01-16T15:22:53.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f688e000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688806c750 con 0x7f68880cf9a0 2024-01-16T15:22:53.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f688e000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688806ca30 con 0x7f688813df80 2024-01-16T15:22:53.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f6887e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f68880cf9a0 0x7f68880cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:53.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f6887e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f68880cf9a0 0x7f68880cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37342/0 (socket says 172.21.15.141:37342) 2024-01-16T15:22:53.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f6887e00640 1 -- 172.21.15.141:0/2809848398 learned_addr learned my addr 172.21.15.141:0/2809848398 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:53.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.003+0000 7f6887400640 1 --2- 172.21.15.141:0/2809848398 >> v2:172.21.15.188:3300/0 conn(0x7f688813df80 0x7f688815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:53.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.004+0000 7f6887e00640 1 -- 172.21.15.141:0/2809848398 >> v2:172.21.15.188:3300/0 conn(0x7f688813df80 msgr2=0x7f688815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:53.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.004+0000 7f6887e00640 1 --2- 172.21.15.141:0/2809848398 >> v2:172.21.15.188:3300/0 conn(0x7f688813df80 0x7f688815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:53.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.004+0000 7f6887e00640 1 -- 172.21.15.141:0/2809848398 >> v1:172.21.15.188:6789/0 conn(0x7f688813e360 legacy=0x7f68880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:54.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.004+0000 7f6887e00640 1 -- 172.21.15.141:0/2809848398 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688806cd10 con 0x7f68880cf9a0 2024-01-16T15:22:54.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.005+0000 7f6887e00640 1 --2- 172.21.15.141:0/2809848398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 0x7f68880cd840 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f6888072770 tx=0x7f688002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a133fb54b2a59678 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:54.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.005+0000 7f6886a00640 1 -- 172.21.15.141:0/2809848398 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688002ca80 con 0x7f68880cf9a0 2024-01-16T15:22:54.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.005+0000 7f6886a00640 1 -- 172.21.15.141:0/2809848398 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688002cbe0 con 0x7f68880cf9a0 2024-01-16T15:22:54.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.005+0000 7f6886a00640 1 -- 172.21.15.141:0/2809848398 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68800343f0 con 0x7f68880cf9a0 2024-01-16T15:22:54.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 -- 172.21.15.141:0/2809848398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 msgr2=0x7f68880cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:54.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 --2- 172.21.15.141:0/2809848398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 0x7f68880cd840 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f6888072770 tx=0x7f688002bee0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 -- 172.21.15.141:0/2809848398 shutdown_connections 2024-01-16T15:22:54.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 --2- 172.21.15.141:0/2809848398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 0x7f68880cd840 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 --2- 172.21.15.141:0/2809848398 >> v2:172.21.15.188:3300/0 conn(0x7f688813df80 0x7f688815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 -- 172.21.15.141:0/2809848398 >> 172.21.15.141:0/2809848398 conn(0x7f688806a800 msgr2=0x7f688806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:54.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 -- 172.21.15.141:0/2809848398 shutdown_connections 2024-01-16T15:22:54.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.006+0000 7f688e000640 1 -- 172.21.15.141:0/2809848398 wait complete. 2024-01-16T15:22:54.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.008+0000 7f688e000640 1 Processor -- start 2024-01-16T15:22:54.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.008+0000 7f688e000640 1 -- start start 2024-01-16T15:22:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f688e000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 0x7f688814fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f688e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f688813df80 0x7f688813d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f688e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f688813e360 0x7f6888150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f688e000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68881500b0 con 0x7f68880cf9a0 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f688e000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688813dd30 con 0x7f688813e360 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f688e000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6888159e10 con 0x7f688813df80 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 0x7f688814fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 0x7f688814fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37354/0 (socket says 172.21.15.141:37354) 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 -- 172.21.15.141:0/96948966 learned_addr learned my addr 172.21.15.141:0/96948966 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 -- 172.21.15.141:0/96948966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f688813df80 msgr2=0x7f688813d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 --2- 172.21.15.141:0/96948966 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f688813df80 0x7f688813d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 -- 172.21.15.141:0/96948966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f688813e360 msgr2=0x7f6888150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 --2- 172.21.15.141:0/96948966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f688813e360 0x7f6888150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.009+0000 7f6887400640 1 -- 172.21.15.141:0/96948966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6880009b30 con 0x7f68880cf9a0 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.010+0000 7f6887400640 1 --2- 172.21.15.141:0/96948966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68880cf9a0 0x7f688814fb70 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f687800d800 tx=0x7f687800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.010+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687800b6c0 con 0x7f68880cf9a0 2024-01-16T15:22:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.010+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687800b820 con 0x7f68880cf9a0 2024-01-16T15:22:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.010+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687800bbb0 con 0x7f68880cf9a0 2024-01-16T15:22:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.010+0000 7f688e000640 1 -- 172.21.15.141:0/96948966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688815a0f0 con 0x7f68880cf9a0 2024-01-16T15:22:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.010+0000 7f688e000640 1 -- 172.21.15.141:0/96948966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688815a6b0 con 0x7f68880cf9a0 2024-01-16T15:22:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.011+0000 7f688e000640 1 -- 172.21.15.141:0/96948966 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68880675d0 con 0x7f68880cf9a0 2024-01-16T15:22:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.012+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 49) v1 ==== 91477+0+0 (secure 0 0 0) 0x7f687800bd10 con 0x7f68880cf9a0 2024-01-16T15:22:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.013+0000 7f687fe00640 1 --2- 172.21.15.141:0/96948966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f686006c820 0x7f686006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.013+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6878089250 con 0x7f68880cf9a0 2024-01-16T15:22:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.014+0000 7f687e800640 1 --2- 172.21.15.141:0/96948966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f686006c820 0x7f686006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.015+0000 7f687e800640 1 --2- 172.21.15.141:0/96948966 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f686006c820 0x7f686006ece0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f68880affd0 tx=0x7f68700079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:54.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.018+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6878055770 con 0x7f68880cf9a0 2024-01-16T15:22:54.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.152+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 50) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f6878055b40 con 0x7f68880cf9a0 2024-01-16T15:22:54.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.252+0000 7f688e000640 1 -- 172.21.15.141:0/96948966 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f688815f9c0 con 0x7f686006c820 2024-01-16T15:22:54.282 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:54.286+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:54.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:54.287+0000 7f687fe00640 1 -- 172.21.15.141:0/96948966 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6870003950 con 0x7f686006c820 2024-01-16T15:22:54.284 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:54.284 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:56.303 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:56.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.531+0000 7f9903800640 1 Processor -- start 2024-01-16T15:22:56.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.531+0000 7f9903800640 1 -- start start 2024-01-16T15:22:56.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.533+0000 7f9903800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f98fc13df80 0x7f98fc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.533+0000 7f9903800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc06c470 con 0x7f98fc0cf9a0 2024-01-16T15:22:56.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.533+0000 7f9903800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc06c750 con 0x7f98fc13e360 2024-01-16T15:22:56.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.533+0000 7f9903800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc06ca30 con 0x7f98fc13df80 2024-01-16T15:22:56.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.533+0000 7f9901800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f98fc0cf9a0 0x7f98fc0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:50860/0 (socket says 172.21.15.141:50860) 2024-01-16T15:22:56.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.533+0000 7f9901800640 1 -- 172.21.15.141:0/2595706365 learned_addr learned my addr 172.21.15.141:0/2595706365 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:22:56.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f9900e00640 1 --2- 172.21.15.141:0/2595706365 >> v2:172.21.15.188:3300/0 conn(0x7f98fc13df80 0x7f98fc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660402540 0 0) 0x7f98fc06c470 con 0x7f98fc0cf9a0 2024-01-16T15:22:56.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98dc003520 con 0x7f98fc0cf9a0 2024-01-16T15:22:56.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204957235 0 0) 0x7f98dc003520 con 0x7f98fc0cf9a0 2024-01-16T15:22:56.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 >> v2:172.21.15.188:3300/0 conn(0x7f98fc13df80 msgr2=0x7f98fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 --2- 172.21.15.141:0/2595706365 >> v2:172.21.15.188:3300/0 conn(0x7f98fc13df80 0x7f98fc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 >> v1:172.21.15.188:6789/0 conn(0x7f98fc13e360 legacy=0x7f98fc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc06cd10 con 0x7f98fc0cf9a0 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f98f40019f0 con 0x7f98fc0cf9a0 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 >> v1:172.21.15.141:6789/0 conn(0x7f98fc0cf9a0 legacy=0x7f98fc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98dc0039c0 0x7f98dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc0047c0 0x7f98dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc06ca30 con 0x7f98dc005180 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c080 con 0x7f98dc0039c0 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.534+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c240 con 0x7f98dc0047c0 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f9901800640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f98f3e00640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc0047c0 0x7f98dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f9901800640 1 -- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc0047c0 msgr2=0x7f98dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f9901800640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc0047c0 0x7f98dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:56.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f9900e00640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98dc0039c0 0x7f98dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f9901800640 1 -- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98dc0039c0 msgr2=0x7f98dc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:56.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f9901800640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98dc0039c0 0x7f98dc003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:56.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.535+0000 7f9901800640 1 -- 172.21.15.141:0/2595706365 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc003520 con 0x7f98dc005180 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.536+0000 7f9901800640 1 -- 172.21.15.141:0/2595706365 reap_dead start 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.536+0000 7f9901800640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f98fc15b6f0 tx=0x7f98dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4744bec4c77f0200 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4034630 con 0x7f98dc005180 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f402f070 con 0x7f98dc005180 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f98fbe00640 1 -- 172.21.15.141:0/2595706365 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4034a10 con 0x7f98dc005180 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f9903800640 1 -- 172.21.15.141:0/2595706365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98dc005180 msgr2=0x7f98dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f9903800640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f98fc15b6f0 tx=0x7f98dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:56.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f9903800640 1 -- 172.21.15.141:0/2595706365 shutdown_connections 2024-01-16T15:22:56.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f9903800640 1 --2- 172.21.15.141:0/2595706365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:56.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f9903800640 1 -- 172.21.15.141:0/2595706365 >> 172.21.15.141:0/2595706365 conn(0x7f98fc06a800 msgr2=0x7f98fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:56.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f9903800640 1 -- 172.21.15.141:0/2595706365 shutdown_connections 2024-01-16T15:22:56.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.537+0000 7f9903800640 1 -- 172.21.15.141:0/2595706365 wait complete. 2024-01-16T15:22:56.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.539+0000 7f9903800640 1 Processor -- start 2024-01-16T15:22:56.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.539+0000 7f9903800640 1 -- start start 2024-01-16T15:22:56.536 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9903800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc005180 0x7f98fc0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9903800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98fc1583c0 0x7f98fc0a9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9903800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98fc0a9a50 0x7f98fc0a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9903800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc1f1da0 con 0x7f98fc0a9a50 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9903800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0c45b0 con 0x7f98fc1583c0 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9903800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0c3ff0 con 0x7f98dc005180 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9901800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98fc0a9a50 0x7f98fc0a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9901800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98fc0a9a50 0x7f98fc0a9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37368/0 (socket says 172.21.15.141:37368) 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.540+0000 7f9901800640 1 -- 172.21.15.141:0/3769891345 learned_addr learned my addr 172.21.15.141:0/3769891345 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:56.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9900e00640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc005180 0x7f98fc0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f98f3e00640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98fc1583c0 0x7f98fc0a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9901800640 1 -- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc005180 msgr2=0x7f98fc0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9901800640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f98dc005180 0x7f98fc0c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9901800640 1 -- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98fc1583c0 msgr2=0x7f98fc0a9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9901800640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98fc1583c0 0x7f98fc0a9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9901800640 1 -- 172.21.15.141:0/3769891345 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0c42d0 con 0x7f98fc0a9a50 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f98f3e00640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98fc1583c0 0x7f98fc0a9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9901800640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98fc0a9a50 0x7f98fc0a9ed0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f98f4002230 tx=0x7f98f4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f98f9600640 1 -- 172.21.15.141:0/3769891345 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f40347b0 con 0x7f98fc0a9a50 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f98f9600640 1 -- 172.21.15.141:0/3769891345 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f4034910 con 0x7f98fc0a9a50 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f98f9600640 1 -- 172.21.15.141:0/3769891345 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4034ca0 con 0x7f98fc0a9a50 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.541+0000 7f9903800640 1 -- 172.21.15.141:0/3769891345 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0ac2f0 con 0x7f98fc0a9a50 2024-01-16T15:22:56.539 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.542+0000 7f9903800640 1 -- 172.21.15.141:0/3769891345 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc158840 con 0x7f98fc0a9a50 2024-01-16T15:22:56.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.543+0000 7f9903800640 1 -- 172.21.15.141:0/3769891345 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc0ac130 con 0x7f98fc0a9a50 2024-01-16T15:22:56.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.544+0000 7f98f9600640 1 -- 172.21.15.141:0/3769891345 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f98f4031030 con 0x7f98fc0a9a50 2024-01-16T15:22:56.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.545+0000 7f98f9600640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f98d406c5e0 0x7f98d406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:56.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.546+0000 7f98f9600640 1 -- 172.21.15.141:0/3769891345 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f98f40aabf0 con 0x7f98fc0a9a50 2024-01-16T15:22:56.541 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.546+0000 7f9900e00640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f98d406c5e0 0x7f98d406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:56.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.547+0000 7f9900e00640 1 --2- 172.21.15.141:0/3769891345 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f98d406c5e0 0x7f98d406eaa0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009a10 tx=0x7f98ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:56.545 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.550+0000 7f98f9600640 1 -- 172.21.15.141:0/3769891345 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98f4077110 con 0x7f98fc0a9a50 2024-01-16T15:22:56.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.771+0000 7f9903800640 1 -- 172.21.15.141:0/3769891345 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98fc0c46f0 con 0x7f98d406c5e0 2024-01-16T15:22:56.780 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:56.784+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:56.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:56.785+0000 7f98f9600640 1 -- 172.21.15.141:0/3769891345 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f98fc0c46f0 con 0x7f98d406c5e0 2024-01-16T15:22:56.780 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:56.781 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:22:58.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:22:59.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.027+0000 7f5a47400640 1 Processor -- start 2024-01-16T15:22:59.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.027+0000 7f5a47400640 1 -- start start 2024-01-16T15:22:59.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.029+0000 7f5a47400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5a401447f0 0x7f5a4015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.029+0000 7f5a47400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006c470 0x7f5a400cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.029+0000 7f5a47400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4015c5c0 con 0x7f5a4006dab0 2024-01-16T15:22:59.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.029+0000 7f5a47400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4006d690 con 0x7f5a401447f0 2024-01-16T15:22:59.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.029+0000 7f5a47400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400cebb0 con 0x7f5a4006c470 2024-01-16T15:22:59.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.030+0000 7f5a45600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006c470 0x7f5a400cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.030+0000 7f5a46000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f5a401447f0 0x7f5a4015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.030+0000 7f5a45600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006c470 0x7f5a400cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:36812/0 (socket says 172.21.15.141:36812) 2024-01-16T15:22:59.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.030+0000 7f5a45600640 1 -- 172.21.15.141:0/3485079303 learned_addr learned my addr 172.21.15.141:0/3485079303 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:59.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.031+0000 7f5a45600640 1 -- 172.21.15.141:0/3485079303 >> v2:172.21.15.188:3300/0 conn(0x7f5a401447f0 msgr2=0x7f5a4015c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:59.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.031+0000 7f5a45600640 1 --2- 172.21.15.141:0/3485079303 >> v2:172.21.15.188:3300/0 conn(0x7f5a401447f0 0x7f5a4015c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.031+0000 7f5a45600640 1 -- 172.21.15.141:0/3485079303 >> v1:172.21.15.188:6789/0 conn(0x7f5a4006dab0 legacy=0x7f5a4015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.031+0000 7f5a45600640 1 -- 172.21.15.141:0/3485079303 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4006ca10 con 0x7f5a4006c470 2024-01-16T15:22:59.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.032+0000 7f5a45600640 1 --2- 172.21.15.141:0/3485079303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006c470 0x7f5a400cc5e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009870 tx=0x7f5a2802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a7bab05d04599ae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:59.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.033+0000 7f5a44c00640 1 -- 172.21.15.141:0/3485079303 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28031640 con 0x7f5a4006c470 2024-01-16T15:22:59.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.033+0000 7f5a44c00640 1 -- 172.21.15.141:0/3485079303 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a280317a0 con 0x7f5a4006c470 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.033+0000 7f5a44c00640 1 -- 172.21.15.141:0/3485079303 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28031b30 con 0x7f5a4006c470 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.033+0000 7f5a47400640 1 -- 172.21.15.141:0/3485079303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006c470 msgr2=0x7f5a400cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.033+0000 7f5a47400640 1 --2- 172.21.15.141:0/3485079303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006c470 0x7f5a400cc5e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009870 tx=0x7f5a2802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.034+0000 7f5a47400640 1 -- 172.21.15.141:0/3485079303 shutdown_connections 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.034+0000 7f5a47400640 1 --2- 172.21.15.141:0/3485079303 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006c470 0x7f5a400cc5e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.034+0000 7f5a47400640 1 --2- 172.21.15.141:0/3485079303 >> v2:172.21.15.188:3300/0 conn(0x7f5a401447f0 0x7f5a4015c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.034+0000 7f5a47400640 1 -- 172.21.15.141:0/3485079303 >> 172.21.15.141:0/3485079303 conn(0x7f5a4006a3e0 msgr2=0x7f5a4006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:59.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.034+0000 7f5a47400640 1 -- 172.21.15.141:0/3485079303 shutdown_connections 2024-01-16T15:22:59.031 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.034+0000 7f5a47400640 1 -- 172.21.15.141:0/3485079303 wait complete. 2024-01-16T15:22:59.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.035+0000 7f5a47400640 1 Processor -- start 2024-01-16T15:22:59.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.036+0000 7f5a47400640 1 -- start start 2024-01-16T15:22:59.034 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.036+0000 7f5a47400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a4006c470 0x7f5a400acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.036+0000 7f5a47400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006dab0 0x7f5a401385d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.036+0000 7f5a47400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a401447f0 0x7f5a4015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.036+0000 7f5a47400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400700f0 con 0x7f5a4006c470 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.036+0000 7f5a47400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400b1170 con 0x7f5a401447f0 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.036+0000 7f5a47400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400ad190 con 0x7f5a4006dab0 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a4006c470 0x7f5a400acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a4006c470 0x7f5a400acc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37382/0 (socket says 172.21.15.141:37382) 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 -- 172.21.15.141:0/86076130 learned_addr learned my addr 172.21.15.141:0/86076130 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:22:59.035 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46a00640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a401447f0 0x7f5a4015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a45600640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006dab0 0x7f5a401385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 -- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006dab0 msgr2=0x7f5a401385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006dab0 0x7f5a401385d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 -- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a401447f0 msgr2=0x7f5a4015a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:59.036 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a401447f0 0x7f5a4015a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:59.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a46000640 1 -- 172.21.15.141:0/86076130 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a280094d0 con 0x7f5a4006c470 2024-01-16T15:22:59.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.037+0000 7f5a45600640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5a4006dab0 0x7f5a401385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:59.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.038+0000 7f5a46000640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5a4006c470 0x7f5a400acc50 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000e930 tx=0x7f5a3000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:59.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.038+0000 7f5a3e000640 1 -- 172.21.15.141:0/86076130 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30009690 con 0x7f5a4006c470 2024-01-16T15:22:59.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.038+0000 7f5a3e000640 1 -- 172.21.15.141:0/86076130 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3000ca00 con 0x7f5a4006c470 2024-01-16T15:22:59.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.038+0000 7f5a3e000640 1 -- 172.21.15.141:0/86076130 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3000cdb0 con 0x7f5a4006c470 2024-01-16T15:22:59.037 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.038+0000 7f5a47400640 1 -- 172.21.15.141:0/86076130 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a400b13f0 con 0x7f5a4006c470 2024-01-16T15:22:59.038 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.038+0000 7f5a47400640 1 -- 172.21.15.141:0/86076130 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a40070300 con 0x7f5a4006c470 2024-01-16T15:22:59.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.039+0000 7f5a47400640 1 -- 172.21.15.141:0/86076130 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a400704f0 con 0x7f5a4006c470 2024-01-16T15:22:59.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.040+0000 7f5a3e000640 1 -- 172.21.15.141:0/86076130 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f5a30013470 con 0x7f5a4006c470 2024-01-16T15:22:59.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.041+0000 7f5a3e000640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a1806c5e0 0x7f5a1806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:59.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.041+0000 7f5a3e000640 1 -- 172.21.15.141:0/86076130 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5a30088e00 con 0x7f5a4006c470 2024-01-16T15:22:59.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.042+0000 7f5a45600640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a1806c5e0 0x7f5a1806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:59.040 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.043+0000 7f5a45600640 1 --2- 172.21.15.141:0/86076130 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5a1806c5e0 0x7f5a1806eaa0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5a28036000 tx=0x7f5a2803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:59.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.046+0000 7f5a3e000640 1 -- 172.21.15.141:0/86076130 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a30055440 con 0x7f5a4006c470 2024-01-16T15:22:59.266 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.271+0000 7f5a47400640 1 -- 172.21.15.141:0/86076130 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a400b1910 con 0x7f5a1806c5e0 2024-01-16T15:22:59.280 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:22:59.284+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:22:59.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:22:59.286+0000 7f5a3e000640 1 -- 172.21.15.141:0/86076130 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5a28031aa0 con 0x7f5a1806c5e0 2024-01-16T15:22:59.282 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:22:59.282 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:22:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:01.294 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:01.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.523+0000 7f8c50a00640 1 Processor -- start 2024-01-16T15:23:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.523+0000 7f8c50a00640 1 -- start start 2024-01-16T15:23:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.525+0000 7f8c50a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c48150760 0x7f8c48072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.525+0000 7f8c50a00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8c4815d300 0x7f8c4815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.525+0000 7f8c50a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4815dc80 con 0x7f8c480b2bd0 2024-01-16T15:23:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.525+0000 7f8c50a00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c480b3770 con 0x7f8c4815d300 2024-01-16T15:23:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.525+0000 7f8c50a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c48071190 con 0x7f8c48150760 2024-01-16T15:23:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.525+0000 7f8c4e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c48150760 0x7f8c48072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:01.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.526+0000 7f8c4e000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c48150760 0x7f8c48072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:36824/0 (socket says 172.21.15.141:36824) 2024-01-16T15:23:01.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.526+0000 7f8c4e000640 1 -- 172.21.15.141:0/2456765930 learned_addr learned my addr 172.21.15.141:0/2456765930 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:01.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.526+0000 7f8c4d600640 1 --2- 172.21.15.141:0/2456765930 >> v2:172.21.15.188:3300/0 conn(0x7f8c4815d300 0x7f8c4815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:01.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.526+0000 7f8c4e000640 1 -- 172.21.15.141:0/2456765930 >> v2:172.21.15.188:3300/0 conn(0x7f8c4815d300 msgr2=0x7f8c4815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:01.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.526+0000 7f8c4e000640 1 --2- 172.21.15.141:0/2456765930 >> v2:172.21.15.188:3300/0 conn(0x7f8c4815d300 0x7f8c4815d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:01.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.526+0000 7f8c4e000640 1 -- 172.21.15.141:0/2456765930 >> v1:172.21.15.188:6789/0 conn(0x7f8c480b2bd0 legacy=0x7f8c480b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:01.522 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.526+0000 7f8c4e000640 1 -- 172.21.15.141:0/2456765930 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48071470 con 0x7f8c48150760 2024-01-16T15:23:01.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.528+0000 7f8c4e000640 1 --2- 172.21.15.141:0/2456765930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c48150760 0x7f8c48072230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d150 tx=0x7f8c3802e160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1fa2495130a2988 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:01.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.528+0000 7f8c4cc00640 1 -- 172.21.15.141:0/2456765930 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38038400 con 0x7f8c48150760 2024-01-16T15:23:01.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.528+0000 7f8c4cc00640 1 -- 172.21.15.141:0/2456765930 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c38032070 con 0x7f8c48150760 2024-01-16T15:23:01.523 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.528+0000 7f8c4cc00640 1 -- 172.21.15.141:0/2456765930 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38038780 con 0x7f8c48150760 2024-01-16T15:23:01.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 -- 172.21.15.141:0/2456765930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c48150760 msgr2=0x7f8c48072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:01.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 --2- 172.21.15.141:0/2456765930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c48150760 0x7f8c48072230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d150 tx=0x7f8c3802e160 comp rx=0 tx=0).stop 2024-01-16T15:23:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 -- 172.21.15.141:0/2456765930 shutdown_connections 2024-01-16T15:23:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 --2- 172.21.15.141:0/2456765930 >> v2:172.21.15.188:3300/0 conn(0x7f8c4815d300 0x7f8c4815d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 --2- 172.21.15.141:0/2456765930 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c48150760 0x7f8c48072230 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 -- 172.21.15.141:0/2456765930 >> 172.21.15.141:0/2456765930 conn(0x7f8c4806a800 msgr2=0x7f8c4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 -- 172.21.15.141:0/2456765930 shutdown_connections 2024-01-16T15:23:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.529+0000 7f8c50a00640 1 -- 172.21.15.141:0/2456765930 wait complete. 2024-01-16T15:23:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.531+0000 7f8c50a00640 1 Processor -- start 2024-01-16T15:23:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.531+0000 7f8c50a00640 1 -- start start 2024-01-16T15:23:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.532+0000 7f8c50a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c480b2bd0 0x7f8c480ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.532+0000 7f8c50a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8c48150760 0x7f8c4814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.532+0000 7f8c50a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c4815d300 0x7f8c4813c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.532+0000 7f8c50a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4814ef90 con 0x7f8c48150760 2024-01-16T15:23:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.532+0000 7f8c50a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c481490b0 con 0x7f8c480b2bd0 2024-01-16T15:23:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.532+0000 7f8c50a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4814e380 con 0x7f8c4815d300 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8c48150760 0x7f8c4814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8c48150760 0x7f8c4814dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37396/0 (socket says 172.21.15.141:37396) 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 -- 172.21.15.141:0/3257638075 learned_addr learned my addr 172.21.15.141:0/3257638075 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4e000640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c480b2bd0 0x7f8c480ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4ea00640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c4815d300 0x7f8c4813c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 -- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c4815d300 msgr2=0x7f8c4813c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8c4815d300 0x7f8c4813c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 -- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c480b2bd0 msgr2=0x7f8c480ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c480b2bd0 0x7f8c480ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.533+0000 7f8c4d600640 1 -- 172.21.15.141:0/3257638075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3800bd40 con 0x7f8c48150760 2024-01-16T15:23:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.534+0000 7f8c4d600640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8c48150760 0x7f8c4814dc80 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000e870 tx=0x7f8c3000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.534+0000 7f8c46000640 1 -- 172.21.15.141:0/3257638075 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c300094f0 con 0x7f8c48150760 2024-01-16T15:23:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.534+0000 7f8c46000640 1 -- 172.21.15.141:0/3257638075 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3000c940 con 0x7f8c48150760 2024-01-16T15:23:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.534+0000 7f8c46000640 1 -- 172.21.15.141:0/3257638075 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3000ccf0 con 0x7f8c48150760 2024-01-16T15:23:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.534+0000 7f8c50a00640 1 -- 172.21.15.141:0/3257638075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4814e660 con 0x7f8c48150760 2024-01-16T15:23:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.534+0000 7f8c50a00640 1 -- 172.21.15.141:0/3257638075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4814ec20 con 0x7f8c48150760 2024-01-16T15:23:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.535+0000 7f8c50a00640 1 -- 172.21.15.141:0/3257638075 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4814f140 con 0x7f8c48150760 2024-01-16T15:23:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.536+0000 7f8c46000640 1 -- 172.21.15.141:0/3257638075 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f8c30013470 con 0x7f8c48150760 2024-01-16T15:23:01.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.537+0000 7f8c46000640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c2006c610 0x7f8c2006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:01.532 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.537+0000 7f8c46000640 1 -- 172.21.15.141:0/3257638075 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8c30088b60 con 0x7f8c48150760 2024-01-16T15:23:01.533 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.538+0000 7f8c4e000640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c2006c610 0x7f8c2006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:01.534 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.539+0000 7f8c4e000640 1 --2- 172.21.15.141:0/3257638075 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8c2006c610 0x7f8c2006ead0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8c38039000 tx=0x7f8c38034000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:01.537 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.542+0000 7f8c46000640 1 -- 172.21.15.141:0/3257638075 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c4814f140 con 0x7f8c48150760 2024-01-16T15:23:01.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.760+0000 7f8c50a00640 1 -- 172.21.15.141:0/3257638075 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c48071d10 con 0x7f8c2006c610 2024-01-16T15:23:01.772 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:01.776+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:01.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:01.777+0000 7f8c46000640 1 -- 172.21.15.141:0/3257638075 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8c3800b2f0 con 0x7f8c2006c610 2024-01-16T15:23:01.772 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:03.785 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:04.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.014+0000 7f90d1a00640 1 Processor -- start 2024-01-16T15:23:04.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.014+0000 7f90d1a00640 1 -- start start 2024-01-16T15:23:04.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.016+0000 7f90d1a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 0x7f90cc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.016+0000 7f90d1a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0cfbf0 con 0x7f90cc071070 2024-01-16T15:23:04.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.016+0000 7f90d1a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc071690 con 0x7f90cc0cf7d0 2024-01-16T15:23:04.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.016+0000 7f90d1a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc14b6a0 con 0x7f90cc14b2c0 2024-01-16T15:23:04.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.016+0000 7f90cb400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f90cc071070 0x7f90cc0cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58602/0 (socket says 172.21.15.141:58602) 2024-01-16T15:23:04.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.016+0000 7f90cb400640 1 -- 172.21.15.141:0/1480198291 learned_addr learned my addr 172.21.15.141:0/1480198291 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90caa00640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 0x7f90cc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396635165 0 0) 0x7f90cc0cfbf0 con 0x7f90cc071070 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ac003520 con 0x7f90cc071070 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641375164 0 0) 0x7f90ac003520 con 0x7f90cc071070 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 msgr2=0x7f90cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 0x7f90cc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 >> v1:172.21.15.188:6789/0 conn(0x7f90cc0cf7d0 legacy=0x7f90cc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc071970 con 0x7f90cc071070 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f90c40019f0 con 0x7f90cc071070 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 >> v1:172.21.15.141:6789/0 conn(0x7f90cc071070 legacy=0x7f90cc0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac0039a0 0x7f90ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 0x7f90ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc14b6a0 con 0x7f90ac004d00 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac00ba30 con 0x7f90ac0039a0 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.017+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac00bbf0 con 0x7f90cc14b2c0 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90cb400640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90c3e00640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 0x7f90ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90caa00640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac0039a0 0x7f90ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90cb400640 1 -- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 msgr2=0x7f90ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90cb400640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 0x7f90ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.016 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90cb400640 1 -- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac0039a0 msgr2=0x7f90ac003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:04.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90cb400640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac0039a0 0x7f90ac003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90cb400640 1 -- 172.21.15.141:0/1480198291 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac003520 con 0x7f90ac004d00 2024-01-16T15:23:04.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.018+0000 7f90c3e00640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc14b2c0 0x7f90ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:04.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.019+0000 7f90cb400640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90ac005180 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f90c400a800 tx=0x7f90ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab22007dde3c489 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:04.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.019+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c40072d0 con 0x7f90ac004d00 2024-01-16T15:23:04.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.019+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c402f070 con 0x7f90ac004d00 2024-01-16T15:23:04.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.019+0000 7f90ca000640 1 -- 172.21.15.141:0/1480198291 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c40076b0 con 0x7f90ac004d00 2024-01-16T15:23:04.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.020+0000 7f90d1a00640 1 -- 172.21.15.141:0/1480198291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 msgr2=0x7f90ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.020+0000 7f90d1a00640 1 --2- 172.21.15.141:0/1480198291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90ac005180 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f90c400a800 tx=0x7f90ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.020+0000 7f90cb400640 1 -- 172.21.15.141:0/1480198291 reap_dead start 2024-01-16T15:23:04.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.020+0000 7f90d1a00640 1 -- 172.21.15.141:0/1480198291 shutdown_connections 2024-01-16T15:23:04.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.020+0000 7f90d1a00640 1 -- 172.21.15.141:0/1480198291 >> 172.21.15.141:0/1480198291 conn(0x7f90cc06a800 msgr2=0x7f90cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:04.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.020+0000 7f90d1a00640 1 -- 172.21.15.141:0/1480198291 shutdown_connections 2024-01-16T15:23:04.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.020+0000 7f90d1a00640 1 -- 172.21.15.141:0/1480198291 wait complete. 2024-01-16T15:23:04.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.022+0000 7f90d1a00640 1 Processor -- start 2024-01-16T15:23:04.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.022+0000 7f90d1a00640 1 -- start start 2024-01-16T15:23:04.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90d1a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90cc06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90d1a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc156ee0 0x7f90cc1f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90d1a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90cc143e50 0x7f90cc1442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90d1a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc06d230 con 0x7f90ac004d00 2024-01-16T15:23:04.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90d1a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc1f6ef0 con 0x7f90cc143e50 2024-01-16T15:23:04.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90d1a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc144810 con 0x7f90cc156ee0 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90caa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90cc06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90caa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90cc06cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47776/0 (socket says 172.21.15.141:47776) 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.023+0000 7f90caa00640 1 -- 172.21.15.141:0/535824497 learned_addr learned my addr 172.21.15.141:0/535824497 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90c3e00640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc156ee0 0x7f90cc1f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90caa00640 1 -- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc156ee0 msgr2=0x7f90cc1f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90caa00640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f90cc156ee0 0x7f90cc1f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90cb400640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90cc143e50 0x7f90cc1442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90caa00640 1 -- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90cc143e50 msgr2=0x7f90cc1442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90caa00640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90cc143e50 0x7f90cc1442d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90caa00640 1 -- 172.21.15.141:0/535824497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc143550 con 0x7f90ac004d00 2024-01-16T15:23:04.021 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90caa00640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90ac004d00 0x7f90cc06cc50 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f90bc010860 tx=0x7f90bc010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:04.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90c2a00640 1 -- 172.21.15.141:0/535824497 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc009610 con 0x7f90ac004d00 2024-01-16T15:23:04.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90c2a00640 1 -- 172.21.15.141:0/535824497 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc00eb00 con 0x7f90ac004d00 2024-01-16T15:23:04.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90d1a00640 1 -- 172.21.15.141:0/535824497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc142c20 con 0x7f90ac004d00 2024-01-16T15:23:04.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.024+0000 7f90c2a00640 1 -- 172.21.15.141:0/535824497 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc00ee90 con 0x7f90ac004d00 2024-01-16T15:23:04.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.025+0000 7f90d1a00640 1 -- 172.21.15.141:0/535824497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc1431e0 con 0x7f90ac004d00 2024-01-16T15:23:04.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.025+0000 7f90d1a00640 1 -- 172.21.15.141:0/535824497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc0675d0 con 0x7f90ac004d00 2024-01-16T15:23:04.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.026+0000 7f90c2a00640 1 -- 172.21.15.141:0/535824497 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f90bc01c400 con 0x7f90ac004d00 2024-01-16T15:23:04.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.028+0000 7f90c2a00640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90a406c5e0 0x7f90a406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:04.024 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.028+0000 7f90c2a00640 1 -- 172.21.15.141:0/535824497 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90bc088d20 con 0x7f90ac004d00 2024-01-16T15:23:04.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.028+0000 7f90c3e00640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90a406c5e0 0x7f90a406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:04.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.029+0000 7f90c3e00640 1 --2- 172.21.15.141:0/535824497 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f90a406c5e0 0x7f90a406eaa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f90cc142020 tx=0x7f90b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:04.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.033+0000 7f90c2a00640 1 -- 172.21.15.141:0/535824497 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90bc0556f0 con 0x7f90ac004d00 2024-01-16T15:23:04.250 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.253+0000 7f90d1a00640 1 -- 172.21.15.141:0/535824497 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90cc0cbc40 con 0x7f90a406c5e0 2024-01-16T15:23:04.263 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:04.268+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:04.264 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:04.267+0000 7f90c2a00640 1 -- 172.21.15.141:0/535824497 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f90b400d050 con 0x7f90a406c5e0 2024-01-16T15:23:04.264 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:04.264 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:06.276 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:06.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.504+0000 7ffb4fa00640 1 Processor -- start 2024-01-16T15:23:06.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.504+0000 7ffb4fa00640 1 -- start start 2024-01-16T15:23:06.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.506+0000 7ffb4fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 0x7ffb480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.506+0000 7ffb4fa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ffb4815e3b0 0x7ffb480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.506+0000 7ffb4fa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb480cfc30 con 0x7ffb480cf7d0 2024-01-16T15:23:06.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.506+0000 7ffb4fa00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb480b1ee0 con 0x7ffb4815e3b0 2024-01-16T15:23:06.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.506+0000 7ffb4fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb480b2780 con 0x7ffb48143ae0 2024-01-16T15:23:06.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.506+0000 7ffb3fc00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ffb480cf7d0 0x7ffb4815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58606/0 (socket says 172.21.15.141:58606) 2024-01-16T15:23:06.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.506+0000 7ffb3fc00640 1 -- 172.21.15.141:0/608806260 learned_addr learned my addr 172.21.15.141:0/608806260 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb4da00640 1 --2- 172.21.15.141:0/608806260 >> v2:172.21.15.188:3300/0 conn(0x7ffb4815e3b0 0x7ffb480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb4d000640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 0x7ffb480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445144740 0 0) 0x7ffb480cfc30 con 0x7ffb480cf7d0 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb28003520 con 0x7ffb480cf7d0 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376182843 0 0) 0x7ffb28003520 con 0x7ffb480cf7d0 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 msgr2=0x7ffb480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 0x7ffb480cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 >> v2:172.21.15.188:3300/0 conn(0x7ffb4815e3b0 msgr2=0x7ffb480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 --2- 172.21.15.141:0/608806260 >> v2:172.21.15.188:3300/0 conn(0x7ffb4815e3b0 0x7ffb480702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4815e990 con 0x7ffb480cf7d0 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7ffb30001670 con 0x7ffb480cf7d0 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 >> v1:172.21.15.141:6789/0 conn(0x7ffb480cf7d0 legacy=0x7ffb4815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 0x7ffb28005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb480b1ee0 con 0x7ffb28003970 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb480b2780 con 0x7ffb280047c0 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.507+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2800b980 con 0x7ffb48143ae0 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb4d000640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb3fc00640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb4da00640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 0x7ffb28005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb4d000640 1 -- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 msgr2=0x7ffb28005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb4d000640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb48143ae0 0x7ffb28005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb4d000640 1 -- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb280047c0 msgr2=0x7ffb28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:06.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb4d000640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.508+0000 7ffb4d000640 1 -- 172.21.15.141:0/608806260 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb28003520 con 0x7ffb28003970 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.509+0000 7ffb4d000640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009ac0 tx=0x7ffb280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36e3acb25f6598 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.509+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3802c6b0 con 0x7ffb28003970 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.509+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3802c810 con 0x7ffb28003970 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.509+0000 7ffb47e00640 1 -- 172.21.15.141:0/608806260 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3802cba0 con 0x7ffb28003970 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.510+0000 7ffb4fa00640 1 -- 172.21.15.141:0/608806260 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 msgr2=0x7ffb28003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.510+0000 7ffb4fa00640 1 --2- 172.21.15.141:0/608806260 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009ac0 tx=0x7ffb280057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.510+0000 7ffb4da00640 1 -- 172.21.15.141:0/608806260 reap_dead start 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.510+0000 7ffb4fa00640 1 -- 172.21.15.141:0/608806260 shutdown_connections 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.510+0000 7ffb4fa00640 1 -- 172.21.15.141:0/608806260 >> 172.21.15.141:0/608806260 conn(0x7ffb4806a800 msgr2=0x7ffb4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.510+0000 7ffb4fa00640 1 -- 172.21.15.141:0/608806260 shutdown_connections 2024-01-16T15:23:06.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.510+0000 7ffb4fa00640 1 -- 172.21.15.141:0/608806260 wait complete. 2024-01-16T15:23:06.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.512+0000 7ffb4fa00640 1 Processor -- start 2024-01-16T15:23:06.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.512+0000 7ffb4fa00640 1 -- start start 2024-01-16T15:23:06.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4fa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb480afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4fa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb480a84c0 0x7ffb4813f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4fa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb4813e2e0 0x7ffb4813f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4fa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4813d280 con 0x7ffb28003970 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4fa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4813d520 con 0x7ffb480a84c0 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4fa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4813d800 con 0x7ffb4813e2e0 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb480afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4d000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb480afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47790/0 (socket says 172.21.15.141:47790) 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.513+0000 7ffb4d000640 1 -- 172.21.15.141:0/3094964760 learned_addr learned my addr 172.21.15.141:0/3094964760 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4da00640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb4813e2e0 0x7ffb4813f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb3fc00640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb480a84c0 0x7ffb4813f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4d000640 1 -- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb4813e2e0 msgr2=0x7ffb4813f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4d000640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb4813e2e0 0x7ffb4813f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4d000640 1 -- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb480a84c0 msgr2=0x7ffb4813f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4d000640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb480a84c0 0x7ffb4813f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4d000640 1 -- 172.21.15.141:0/3094964760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4813dae0 con 0x7ffb28003970 2024-01-16T15:23:06.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4da00640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffb4813e2e0 0x7ffb4813f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:06.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb3fc00640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb480a84c0 0x7ffb4813f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:06.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4d000640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffb28003970 0x7ffb480afce0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7ffb3802c250 tx=0x7ffb38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:06.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb45600640 1 -- 172.21.15.141:0/3094964760 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb38003810 con 0x7ffb28003970 2024-01-16T15:23:06.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb45600640 1 -- 172.21.15.141:0/3094964760 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb38003970 con 0x7ffb28003970 2024-01-16T15:23:06.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb45600640 1 -- 172.21.15.141:0/3094964760 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb38003ce0 con 0x7ffb28003970 2024-01-16T15:23:06.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.514+0000 7ffb4fa00640 1 -- 172.21.15.141:0/3094964760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4813ddc0 con 0x7ffb28003970 2024-01-16T15:23:06.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.515+0000 7ffb4fa00640 1 -- 172.21.15.141:0/3094964760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb480b0400 con 0x7ffb28003970 2024-01-16T15:23:06.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.515+0000 7ffb4fa00640 1 -- 172.21.15.141:0/3094964760 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb480675d0 con 0x7ffb28003970 2024-01-16T15:23:06.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.516+0000 7ffb45600640 1 -- 172.21.15.141:0/3094964760 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ffb38034440 con 0x7ffb28003970 2024-01-16T15:23:06.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.517+0000 7ffb45600640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffb2006c5e0 0x7ffb2006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:06.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.518+0000 7ffb45600640 1 -- 172.21.15.141:0/3094964760 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffb380aad10 con 0x7ffb28003970 2024-01-16T15:23:06.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.518+0000 7ffb3fc00640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffb2006c5e0 0x7ffb2006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:06.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.519+0000 7ffb3fc00640 1 --2- 172.21.15.141:0/3094964760 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffb2006c5e0 0x7ffb2006eaa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ffb30000f60 tx=0x7ffb30000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:06.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.522+0000 7ffb45600640 1 -- 172.21.15.141:0/3094964760 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb380772e0 con 0x7ffb28003970 2024-01-16T15:23:06.741 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.744+0000 7ffb4fa00640 1 -- 172.21.15.141:0/3094964760 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb4813d280 con 0x7ffb2006c5e0 2024-01-16T15:23:06.754 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:06.758+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:06.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:06.758+0000 7ffb45600640 1 -- 172.21.15.141:0/3094964760 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffb4813d280 con 0x7ffb2006c5e0 2024-01-16T15:23:06.755 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:06.755 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:08.775 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:09.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.004+0000 7f541ae00640 1 Processor -- start 2024-01-16T15:23:09.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.004+0000 7f541ae00640 1 -- start start 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f541ae00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f541415da10 0x7f5414072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f541ae00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140b3730 con 0x7f54140b2bd0 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f541ae00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5414071190 con 0x7f5414150760 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f541ae00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5414071470 con 0x7f541415da10 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f5413400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f541415da10 0x7f5414072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f5418e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f54140b2bd0 0x7f54140b3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58612/0 (socket says 172.21.15.141:58612) 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f5418e00640 1 -- 172.21.15.141:0/1515278912 learned_addr learned my addr 172.21.15.141:0/1515278912 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.006+0000 7f5413400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f541415da10 0x7f5414072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47802/0 (socket says 172.21.15.141:47802) 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.007+0000 7f5413400640 1 -- 172.21.15.141:0/1515278912 >> v1:172.21.15.188:6789/0 conn(0x7f5414150760 legacy=0x7f541415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:09.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.007+0000 7f5413400640 1 -- 172.21.15.141:0/1515278912 >> v1:172.21.15.141:6789/0 conn(0x7f54140b2bd0 legacy=0x7f54140b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.007+0000 7f5413400640 1 -- 172.21.15.141:0/1515278912 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414071750 con 0x7f541415da10 2024-01-16T15:23:09.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.008+0000 7f5413400640 1 --2- 172.21.15.141:0/1515278912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f541415da10 0x7f5414072230 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0097a0 tx=0x7f53fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6769456cc32e309 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:09.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.008+0000 7f5412a00640 1 -- 172.21.15.141:0/1515278912 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc031830 con 0x7f541415da10 2024-01-16T15:23:09.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.008+0000 7f5412a00640 1 -- 172.21.15.141:0/1515278912 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc031990 con 0x7f541415da10 2024-01-16T15:23:09.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.008+0000 7f5412a00640 1 -- 172.21.15.141:0/1515278912 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc031d20 con 0x7f541415da10 2024-01-16T15:23:09.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.009+0000 7f541ae00640 1 -- 172.21.15.141:0/1515278912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f541415da10 msgr2=0x7f5414072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.009+0000 7f541ae00640 1 --2- 172.21.15.141:0/1515278912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f541415da10 0x7f5414072230 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0097a0 tx=0x7f53fc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.009+0000 7f541ae00640 1 -- 172.21.15.141:0/1515278912 shutdown_connections 2024-01-16T15:23:09.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.009+0000 7f541ae00640 1 --2- 172.21.15.141:0/1515278912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f541415da10 0x7f5414072230 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.009+0000 7f541ae00640 1 -- 172.21.15.141:0/1515278912 >> 172.21.15.141:0/1515278912 conn(0x7f541406a800 msgr2=0x7f541406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:09.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.009+0000 7f541ae00640 1 -- 172.21.15.141:0/1515278912 shutdown_connections 2024-01-16T15:23:09.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.009+0000 7f541ae00640 1 -- 172.21.15.141:0/1515278912 wait complete. 2024-01-16T15:23:09.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.011+0000 7f541ae00640 1 Processor -- start 2024-01-16T15:23:09.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.011+0000 7f541ae00640 1 -- start start 2024-01-16T15:23:09.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.012+0000 7f541ae00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54140b2bd0 0x7f541415bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.012+0000 7f541ae00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5414150760 0x7f541415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.012+0000 7f541ae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f541415da10 0x7f5414147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.012+0000 7f541ae00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140ccda0 con 0x7f5414150760 2024-01-16T15:23:09.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.012+0000 7f541ae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140ce790 con 0x7f541415da10 2024-01-16T15:23:09.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.012+0000 7f541ae00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140cea70 con 0x7f54140b2bd0 2024-01-16T15:23:09.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5414150760 0x7f541415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5414150760 0x7f541415c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47810/0 (socket says 172.21.15.141:47810) 2024-01-16T15:23:09.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 -- 172.21.15.141:0/2111980309 learned_addr learned my addr 172.21.15.141:0/2111980309 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:09.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5418e00640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f541415da10 0x7f5414147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 -- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54140b2bd0 msgr2=0x7f541415bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413e00640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54140b2bd0 0x7f541415bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.011 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54140b2bd0 0x7f541415bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 -- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f541415da10 msgr2=0x7f5414147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f541415da10 0x7f5414147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5413400640 1 -- 172.21.15.141:0/2111980309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc009400 con 0x7f5414150760 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.013+0000 7f5418e00640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f541415da10 0x7f5414147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.014+0000 7f5413400640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5414150760 0x7f541415c500 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f53fc000c00 tx=0x7f53fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.014+0000 7f540be00640 1 -- 172.21.15.141:0/2111980309 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc03aa60 con 0x7f5414150760 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.014+0000 7f540be00640 1 -- 172.21.15.141:0/2111980309 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc03abc0 con 0x7f5414150760 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.014+0000 7f540be00640 1 -- 172.21.15.141:0/2111980309 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0443f0 con 0x7f5414150760 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.014+0000 7f541ae00640 1 -- 172.21.15.141:0/2111980309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54140ce3b0 con 0x7f5414150760 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.014+0000 7f541ae00640 1 -- 172.21.15.141:0/2111980309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54140cbe40 con 0x7f5414150760 2024-01-16T15:23:09.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.015+0000 7f541ae00640 1 -- 172.21.15.141:0/2111980309 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54140ccee0 con 0x7f5414150760 2024-01-16T15:23:09.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.016+0000 7f540be00640 1 -- 172.21.15.141:0/2111980309 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f53fc04d450 con 0x7f5414150760 2024-01-16T15:23:09.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.017+0000 7f540be00640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53ec06c5e0 0x7f53ec06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:09.013 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.017+0000 7f540be00640 1 -- 172.21.15.141:0/2111980309 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f53fc0aad80 con 0x7f5414150760 2024-01-16T15:23:09.014 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.018+0000 7f5413e00640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53ec06c5e0 0x7f53ec06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:09.015 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.019+0000 7f5413e00640 1 --2- 172.21.15.141:0/2111980309 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f53ec06c5e0 0x7f53ec06eaa0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5404003d90 tx=0x7f5404009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:09.018 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.022+0000 7f540be00640 1 -- 172.21.15.141:0/2111980309 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53fc0772a0 con 0x7f5414150760 2024-01-16T15:23:09.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.246+0000 7f541ae00640 1 -- 172.21.15.141:0/2111980309 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54140b3510 con 0x7f53ec06c5e0 2024-01-16T15:23:09.255 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:09.259+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:09.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:09.260+0000 7f540be00640 1 -- 172.21.15.141:0/2111980309 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f54140b3510 con 0x7f53ec06c5e0 2024-01-16T15:23:09.256 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:09.256 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:09.884 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:09.892 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:11.272 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:11.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.501+0000 7f97f9800640 1 Processor -- start 2024-01-16T15:23:11.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.501+0000 7f97f9800640 1 -- start start 2024-01-16T15:23:11.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97f9800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f97f413df80 0x7f97f415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97f9800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f97f40cf9a0 0x7f97f40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97f9800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f406c470 con 0x7f97f413e360 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97f9800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f406c750 con 0x7f97f40cf9a0 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97f9800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f406ca30 con 0x7f97f413df80 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97e9e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f97f413e360 0x7f97f40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:58626/0 (socket says 172.21.15.141:58626) 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97f3400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f97f40cf9a0 0x7f97f40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97e9e00640 1 -- 172.21.15.141:0/891576003 learned_addr learned my addr 172.21.15.141:0/891576003 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.503+0000 7f97f2a00640 1 --2- 172.21.15.141:0/891576003 >> v2:172.21.15.188:3300/0 conn(0x7f97f413df80 0x7f97f415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.504+0000 7f97f3400640 1 -- 172.21.15.141:0/891576003 >> v2:172.21.15.188:3300/0 conn(0x7f97f413df80 msgr2=0x7f97f415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.504+0000 7f97f3400640 1 --2- 172.21.15.141:0/891576003 >> v2:172.21.15.188:3300/0 conn(0x7f97f413df80 0x7f97f415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:11.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.504+0000 7f97f3400640 1 -- 172.21.15.141:0/891576003 >> v1:172.21.15.141:6789/0 conn(0x7f97f413e360 legacy=0x7f97f40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.504+0000 7f97f2000640 1 -- 172.21.15.141:0/891576003 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622233197 0 0) 0x7f97f406c470 con 0x7f97f413e360 2024-01-16T15:23:11.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.504+0000 7f97f3400640 1 -- 172.21.15.141:0/891576003 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f406cd10 con 0x7f97f40cf9a0 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.505+0000 7f97f3400640 1 --2- 172.21.15.141:0/891576003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 0x7f97f40cd840 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f97f4072770 tx=0x7f97ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161996fbfba8f12a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.505+0000 7f97f2000640 1 -- 172.21.15.141:0/891576003 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec031830 con 0x7f97f40cf9a0 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.505+0000 7f97f2000640 1 -- 172.21.15.141:0/891576003 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec031990 con 0x7f97f40cf9a0 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.505+0000 7f97f2000640 1 -- 172.21.15.141:0/891576003 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec031d20 con 0x7f97f40cf9a0 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 -- 172.21.15.141:0/891576003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 msgr2=0x7f97f40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 --2- 172.21.15.141:0/891576003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 0x7f97f40cd840 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f97f4072770 tx=0x7f97ec02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 -- 172.21.15.141:0/891576003 shutdown_connections 2024-01-16T15:23:11.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 --2- 172.21.15.141:0/891576003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 0x7f97f40cd840 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:11.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 --2- 172.21.15.141:0/891576003 >> v2:172.21.15.188:3300/0 conn(0x7f97f413df80 0x7f97f415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:11.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 -- 172.21.15.141:0/891576003 >> 172.21.15.141:0/891576003 conn(0x7f97f406a800 msgr2=0x7f97f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:11.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 -- 172.21.15.141:0/891576003 shutdown_connections 2024-01-16T15:23:11.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.506+0000 7f97f9800640 1 -- 172.21.15.141:0/891576003 wait complete. 2024-01-16T15:23:11.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.508+0000 7f97f9800640 1 Processor -- start 2024-01-16T15:23:11.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.508+0000 7f97f9800640 1 -- start start 2024-01-16T15:23:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f9800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 0x7f97f41f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97f413df80 0x7f97f41512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f9800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97f413e360 0x7f97f415aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:11.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f9800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41f4ee0 con 0x7f97f40cf9a0 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f9800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f415a630 con 0x7f97f413df80 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f9800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f4151820 con 0x7f97f413e360 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 0x7f97f41f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f2a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 0x7f97f41f49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:47828/0 (socket says 172.21.15.141:47828) 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.509+0000 7f97f2a00640 1 -- 172.21.15.141:0/3691390517 learned_addr learned my addr 172.21.15.141:0/3691390517 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97f2a00640 1 -- 172.21.15.141:0/3691390517 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97f413e360 msgr2=0x7f97f415aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97f2a00640 1 --2- 172.21.15.141:0/3691390517 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f97f413e360 0x7f97f415aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97f2a00640 1 -- 172.21.15.141:0/3691390517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97f413df80 msgr2=0x7f97f41512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97f2a00640 1 --2- 172.21.15.141:0/3691390517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97f413df80 0x7f97f41512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97f2a00640 1 -- 172.21.15.141:0/3691390517 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec009400 con 0x7f97f40cf9a0 2024-01-16T15:23:11.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97f2a00640 1 --2- 172.21.15.141:0/3691390517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f97f40cf9a0 0x7f97f41f49a0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f97e4007ef0 tx=0x7f97e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:11.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97eb400640 1 -- 172.21.15.141:0/3691390517 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4014620 con 0x7f97f40cf9a0 2024-01-16T15:23:11.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97eb400640 1 -- 172.21.15.141:0/3691390517 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e4014780 con 0x7f97f40cf9a0 2024-01-16T15:23:11.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97f9800640 1 -- 172.21.15.141:0/3691390517 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f4159a20 con 0x7f97f40cf9a0 2024-01-16T15:23:11.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.510+0000 7f97eb400640 1 -- 172.21.15.141:0/3691390517 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4014b10 con 0x7f97f40cf9a0 2024-01-16T15:23:11.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.511+0000 7f97f9800640 1 -- 172.21.15.141:0/3691390517 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f4159fe0 con 0x7f97f40cf9a0 2024-01-16T15:23:11.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.512+0000 7f97f9800640 1 -- 172.21.15.141:0/3691390517 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f40675d0 con 0x7f97f40cf9a0 2024-01-16T15:23:11.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.513+0000 7f97eb400640 1 -- 172.21.15.141:0/3691390517 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f97e4014c70 con 0x7f97f40cf9a0 2024-01-16T15:23:11.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.514+0000 7f97eb400640 1 --2- 172.21.15.141:0/3691390517 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97cc06c5e0 0x7f97cc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:11.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.514+0000 7f97eb400640 1 -- 172.21.15.141:0/3691390517 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f97e40895d0 con 0x7f97f40cf9a0 2024-01-16T15:23:11.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.515+0000 7f97e9e00640 1 --2- 172.21.15.141:0/3691390517 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97cc06c5e0 0x7f97cc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:11.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.516+0000 7f97e9e00640 1 --2- 172.21.15.141:0/3691390517 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97cc06c5e0 0x7f97cc06eaa0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0011e0 tx=0x7f97dc007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:11.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.519+0000 7f97eb400640 1 -- 172.21.15.141:0/3691390517 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97e4055b50 con 0x7f97f40cf9a0 2024-01-16T15:23:11.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.742+0000 7f97f9800640 1 -- 172.21.15.141:0/3691390517 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97f415f9c0 con 0x7f97cc06c5e0 2024-01-16T15:23:11.752 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:11.757+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:11.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:11.756+0000 7f97eb400640 1 -- 172.21.15.141:0/3691390517 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f97dc0070d0 con 0x7f97cc06c5e0 2024-01-16T15:23:11.753 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:11.753 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:13.770 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:13.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.994+0000 7fe478c00640 1 Processor -- start 2024-01-16T15:23:13.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.994+0000 7fe478c00640 1 -- start start 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.996+0000 7fe478c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe474144c60 0x7fe4740cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.996+0000 7fe478c00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe4740cf780 0x7fe47406e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.996+0000 7fe478c00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe474071a20 con 0x7fe47406a3e0 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.996+0000 7fe478c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740ccfd0 con 0x7fe474144c60 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.996+0000 7fe478c00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740cd2b0 con 0x7fe4740cf780 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.996+0000 7fe471e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe474144c60 0x7fe4740cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe474144c60 0x7fe4740cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57358/0 (socket says 172.21.15.141:57358) 2024-01-16T15:23:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471e00640 1 -- 172.21.15.141:0/2613249226 learned_addr learned my addr 172.21.15.141:0/2613249226 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471400640 1 --2- 172.21.15.141:0/2613249226 >> v2:172.21.15.188:3300/0 conn(0x7fe4740cf780 0x7fe47406e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471e00640 1 -- 172.21.15.141:0/2613249226 >> v2:172.21.15.188:3300/0 conn(0x7fe4740cf780 msgr2=0x7fe47406e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471e00640 1 --2- 172.21.15.141:0/2613249226 >> v2:172.21.15.188:3300/0 conn(0x7fe4740cf780 0x7fe47406e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471e00640 1 -- 172.21.15.141:0/2613249226 >> v1:172.21.15.188:6789/0 conn(0x7fe47406a3e0 legacy=0x7fe474071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:13.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471e00640 1 -- 172.21.15.141:0/2613249226 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4740713d0 con 0x7fe474144c60 2024-01-16T15:23:13.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.997+0000 7fe471400640 1 --2- 172.21.15.141:0/2613249226 >> v2:172.21.15.188:3300/0 conn(0x7fe4740cf780 0x7fe47406e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:13.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.998+0000 7fe471e00640 1 --2- 172.21.15.141:0/2613249226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 0x7fe4740cc940 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fe47406c4b0 tx=0x7fe46402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab29d7d7e390b67 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:13.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe470a00640 1 -- 172.21.15.141:0/2613249226 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464031870 con 0x7fe474144c60 2024-01-16T15:23:13.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe470a00640 1 -- 172.21.15.141:0/2613249226 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4640319d0 con 0x7fe474144c60 2024-01-16T15:23:13.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe470a00640 1 -- 172.21.15.141:0/2613249226 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464031d60 con 0x7fe474144c60 2024-01-16T15:23:13.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 -- 172.21.15.141:0/2613249226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 msgr2=0x7fe4740cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:13.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 --2- 172.21.15.141:0/2613249226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 0x7fe4740cc940 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fe47406c4b0 tx=0x7fe46402b910 comp rx=0 tx=0).stop 2024-01-16T15:23:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 -- 172.21.15.141:0/2613249226 shutdown_connections 2024-01-16T15:23:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 --2- 172.21.15.141:0/2613249226 >> v2:172.21.15.188:3300/0 conn(0x7fe4740cf780 0x7fe47406e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 --2- 172.21.15.141:0/2613249226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 0x7fe4740cc940 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 -- 172.21.15.141:0/2613249226 >> 172.21.15.141:0/2613249226 conn(0x7fe4740cfef0 msgr2=0x7fe4740d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 -- 172.21.15.141:0/2613249226 shutdown_connections 2024-01-16T15:23:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:13.999+0000 7fe478c00640 1 -- 172.21.15.141:0/2613249226 wait complete. 2024-01-16T15:23:13.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.001+0000 7fe478c00640 1 Processor -- start 2024-01-16T15:23:13.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.001+0000 7fe478c00640 1 -- start start 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.002+0000 7fe478c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe47406a3e0 0x7fe47415aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.002+0000 7fe478c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4740cf780 0x7fe47413c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.002+0000 7fe478c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 0x7fe474139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.002+0000 7fe478c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740c57a0 con 0x7fe474144c60 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.002+0000 7fe478c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47413c920 con 0x7fe4740cf780 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.002+0000 7fe478c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47413cc00 con 0x7fe47406a3e0 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 0x7fe474139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 0x7fe474139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57374/0 (socket says 172.21.15.141:57374) 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 -- 172.21.15.141:0/211342355 learned_addr learned my addr 172.21.15.141:0/211342355 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe471e00640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe47406a3e0 0x7fe47415aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe471400640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4740cf780 0x7fe47413c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 -- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe47406a3e0 msgr2=0x7fe47415aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe47406a3e0 0x7fe47415aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 -- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4740cf780 msgr2=0x7fe47413c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4740cf780 0x7fe47413c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 -- 172.21.15.141:0/211342355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640094d0 con 0x7fe474144c60 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe472800640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe474144c60 0x7fe474139e00 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00c780 tx=0x7fe46c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe46a000640 1 -- 172.21.15.141:0/211342355 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c003d80 con 0x7fe474144c60 2024-01-16T15:23:14.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.003+0000 7fe46a000640 1 -- 172.21.15.141:0/211342355 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c00ce90 con 0x7fe474144c60 2024-01-16T15:23:14.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.004+0000 7fe478c00640 1 -- 172.21.15.141:0/211342355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47413cee0 con 0x7fe474144c60 2024-01-16T15:23:14.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.004+0000 7fe46a000640 1 -- 172.21.15.141:0/211342355 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c0052b0 con 0x7fe474144c60 2024-01-16T15:23:14.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.004+0000 7fe478c00640 1 -- 172.21.15.141:0/211342355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47415afc0 con 0x7fe474144c60 2024-01-16T15:23:14.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.005+0000 7fe478c00640 1 -- 172.21.15.141:0/211342355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4740c5ca0 con 0x7fe474144c60 2024-01-16T15:23:14.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.006+0000 7fe46a000640 1 -- 172.21.15.141:0/211342355 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fe46c005410 con 0x7fe474144c60 2024-01-16T15:23:14.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.007+0000 7fe46a000640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe44c06c5e0 0x7fe44c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:14.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.007+0000 7fe46a000640 1 -- 172.21.15.141:0/211342355 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe46c088420 con 0x7fe474144c60 2024-01-16T15:23:14.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.007+0000 7fe471e00640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe44c06c5e0 0x7fe44c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:14.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.008+0000 7fe471e00640 1 --2- 172.21.15.141:0/211342355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe44c06c5e0 0x7fe44c06eaa0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe464036000 tx=0x7fe46403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:14.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.011+0000 7fe46a000640 1 -- 172.21.15.141:0/211342355 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4740c5ca0 con 0x7fe474144c60 2024-01-16T15:23:14.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.222+0000 7fe478c00640 1 -- 172.21.15.141:0/211342355 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4740d0870 con 0x7fe44c06c5e0 2024-01-16T15:23:14.232 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:14.237+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:14.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:14.237+0000 7fe46a000640 1 -- 172.21.15.141:0/211342355 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe4740d0870 con 0x7fe44c06c5e0 2024-01-16T15:23:14.233 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:14.233 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:16.248 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:16.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.480+0000 7f77e7800640 1 Processor -- start 2024-01-16T15:23:16.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.481+0000 7f77e7800640 1 -- start start 2024-01-16T15:23:16.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.482+0000 7f77e7800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f77e00713f0 0x7f77e00cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.482+0000 7f77e7800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f77e00b43c0 0x7f77e00b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.482+0000 7f77e7800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e00b3f30 con 0x7f77e00717d0 2024-01-16T15:23:16.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.482+0000 7f77e7800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e00cde70 con 0x7f77e00713f0 2024-01-16T15:23:16.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.482+0000 7f77e7800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e00b3320 con 0x7f77e00b43c0 2024-01-16T15:23:16.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77dfe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f77e00717d0 0x7f77e006f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44848/0 (socket says 172.21.15.141:44848) 2024-01-16T15:23:16.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77dfe00640 1 -- 172.21.15.141:0/1806506599 learned_addr learned my addr 172.21.15.141:0/1806506599 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:16.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77e4e00640 1 --2- 172.21.15.141:0/1806506599 >> v2:172.21.15.141:3300/0 conn(0x7f77e00713f0 0x7f77e00cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77e5800640 1 --2- 172.21.15.141:0/1806506599 >> v2:172.21.15.188:3300/0 conn(0x7f77e00b43c0 0x7f77e00b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77e4e00640 1 -- 172.21.15.141:0/1806506599 >> v2:172.21.15.188:3300/0 conn(0x7f77e00b43c0 msgr2=0x7f77e00b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:16.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77e4e00640 1 --2- 172.21.15.141:0/1806506599 >> v2:172.21.15.188:3300/0 conn(0x7f77e00b43c0 0x7f77e00b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77e4e00640 1 -- 172.21.15.141:0/1806506599 >> v1:172.21.15.141:6789/0 conn(0x7f77e00717d0 legacy=0x7f77e006f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77e4e00640 1 -- 172.21.15.141:0/1806506599 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e00b3600 con 0x7f77e00713f0 2024-01-16T15:23:16.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.483+0000 7f77df400640 1 -- 172.21.15.141:0/1806506599 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146897122 0 0) 0x7f77e00b3f30 con 0x7f77e00717d0 2024-01-16T15:23:16.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.484+0000 7f77e4e00640 1 --2- 172.21.15.141:0/1806506599 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 0x7f77e00cd930 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f77e00b5590 tx=0x7f77c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71077c4480c2b57c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77df400640 1 -- 172.21.15.141:0/1806506599 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8031830 con 0x7f77e00713f0 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77df400640 1 -- 172.21.15.141:0/1806506599 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c8031990 con 0x7f77e00713f0 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77df400640 1 -- 172.21.15.141:0/1806506599 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8031d20 con 0x7f77e00713f0 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77e7800640 1 -- 172.21.15.141:0/1806506599 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 msgr2=0x7f77e00cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77e7800640 1 --2- 172.21.15.141:0/1806506599 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 0x7f77e00cd930 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f77e00b5590 tx=0x7f77c802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77e7800640 1 -- 172.21.15.141:0/1806506599 shutdown_connections 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77e7800640 1 --2- 172.21.15.141:0/1806506599 >> v2:172.21.15.188:3300/0 conn(0x7f77e00b43c0 0x7f77e00b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77e7800640 1 --2- 172.21.15.141:0/1806506599 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 0x7f77e00cd930 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.485+0000 7f77e7800640 1 -- 172.21.15.141:0/1806506599 >> 172.21.15.141:0/1806506599 conn(0x7f77e00b1c60 msgr2=0x7f77e00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:16.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.486+0000 7f77e7800640 1 -- 172.21.15.141:0/1806506599 shutdown_connections 2024-01-16T15:23:16.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.486+0000 7f77e7800640 1 -- 172.21.15.141:0/1806506599 wait complete. 2024-01-16T15:23:16.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.487+0000 7f77e7800640 1 Processor -- start 2024-01-16T15:23:16.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.488+0000 7f77e7800640 1 -- start start 2024-01-16T15:23:16.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.488+0000 7f77e7800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 0x7f77e0139a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.488+0000 7f77e7800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77e00717d0 0x7f77e01387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.488+0000 7f77e7800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f77e00b43c0 0x7f77e013b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.488+0000 7f77e7800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e01f4520 con 0x7f77e00713f0 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.489+0000 7f77e7800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e01f1440 con 0x7f77e00717d0 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.489+0000 7f77e7800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e01f1720 con 0x7f77e00b43c0 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.489+0000 7f77e4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 0x7f77e0139a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.489+0000 7f77e4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 0x7f77e0139a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57378/0 (socket says 172.21.15.141:57378) 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.489+0000 7f77e4e00640 1 -- 172.21.15.141:0/2686967532 learned_addr learned my addr 172.21.15.141:0/2686967532 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.489+0000 7f77e5800640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f77e00b43c0 0x7f77e013b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.489+0000 7f77dfe00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77e00717d0 0x7f77e01387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77e4e00640 1 -- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f77e00b43c0 msgr2=0x7f77e013b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77e4e00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f77e00b43c0 0x7f77e013b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77e4e00640 1 -- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77e00717d0 msgr2=0x7f77e01387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77e4e00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77e00717d0 0x7f77e01387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:16.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77e4e00640 1 -- 172.21.15.141:0/2686967532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c8009400 con 0x7f77e00713f0 2024-01-16T15:23:16.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77dfe00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77e00717d0 0x7f77e01387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:23:16.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77e4e00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77e00713f0 0x7f77e0139a50 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f77c8000c00 tx=0x7f77c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77dcc00640 1 -- 172.21.15.141:0/2686967532 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c803aa60 con 0x7f77e00713f0 2024-01-16T15:23:16.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77dcc00640 1 -- 172.21.15.141:0/2686967532 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c803abe0 con 0x7f77e00713f0 2024-01-16T15:23:16.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77dcc00640 1 -- 172.21.15.141:0/2686967532 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c80443f0 con 0x7f77e00713f0 2024-01-16T15:23:16.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.490+0000 7f77e7800640 1 -- 172.21.15.141:0/2686967532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e01f1a00 con 0x7f77e00713f0 2024-01-16T15:23:16.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.491+0000 7f77e7800640 1 -- 172.21.15.141:0/2686967532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e0139f90 con 0x7f77e00713f0 2024-01-16T15:23:16.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.491+0000 7f77e7800640 1 -- 172.21.15.141:0/2686967532 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e013a180 con 0x7f77e00713f0 2024-01-16T15:23:16.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.492+0000 7f77dcc00640 1 -- 172.21.15.141:0/2686967532 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f77c804d450 con 0x7f77e00713f0 2024-01-16T15:23:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.493+0000 7f77dcc00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f77b806c5e0 0x7f77b806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.493+0000 7f77dcc00640 1 -- 172.21.15.141:0/2686967532 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f77c80aad10 con 0x7f77e00713f0 2024-01-16T15:23:16.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.494+0000 7f77dfe00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f77b806c5e0 0x7f77b806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:16.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.495+0000 7f77dfe00640 1 --2- 172.21.15.141:0/2686967532 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f77b806c5e0 0x7f77b806eaa0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f77e013f490 tx=0x7f77d0005b90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:16.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.498+0000 7f77dcc00640 1 -- 172.21.15.141:0/2686967532 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77c8077290 con 0x7f77e00713f0 2024-01-16T15:23:16.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.724+0000 7f77e7800640 1 -- 172.21.15.141:0/2686967532 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77e00cd150 con 0x7f77b806c5e0 2024-01-16T15:23:16.738 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:16.743+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:16.742+0000 7f77dcc00640 1 -- 172.21.15.141:0/2686967532 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f77e00cd150 con 0x7f77b806c5e0 2024-01-16T15:23:16.738 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:16.739 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:18.760 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:18.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.991+0000 7f55e3800640 1 Processor -- start 2024-01-16T15:23:18.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.991+0000 7f55e3800640 1 -- start start 2024-01-16T15:23:18.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.992+0000 7f55e3800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 0x7f55dc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.992+0000 7f55e3800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc070c40 con 0x7f55dc0b3ae0 2024-01-16T15:23:18.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.992+0000 7f55e3800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc06c470 con 0x7f55dc0cf290 2024-01-16T15:23:18.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.992+0000 7f55e3800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc071450 con 0x7f55dc0ae5a0 2024-01-16T15:23:18.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.992+0000 7f55d3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f55dc0b3ae0 0x7f55dc06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44860/0 (socket says 172.21.15.141:44860) 2024-01-16T15:23:18.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.992+0000 7f55d3e00640 1 -- 172.21.15.141:0/786795765 learned_addr learned my addr 172.21.15.141:0/786795765 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:18.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.992+0000 7f55e0e00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 0x7f55dc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404067105 0 0) 0x7f55dc070c40 con 0x7f55dc0b3ae0 2024-01-16T15:23:18.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55bc003520 con 0x7f55dc0b3ae0 2024-01-16T15:23:18.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2897110320 0 0) 0x7f55bc003520 con 0x7f55dc0b3ae0 2024-01-16T15:23:18.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 msgr2=0x7f55dc15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:18.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 0x7f55dc15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 >> v1:172.21.15.188:6789/0 conn(0x7f55dc0cf290 legacy=0x7f55dc0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc06c750 con 0x7f55dc0b3ae0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f55c4001670 con 0x7f55dc0b3ae0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 >> v1:172.21.15.141:6789/0 conn(0x7f55dc0b3ae0 legacy=0x7f55dc06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55bc0039a0 0x7f55bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55bc0047c0 0x7f55bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 0x7f55bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc071450 con 0x7f55bc0039a0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc00bac0 con 0x7f55bc0047c0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc00bc80 con 0x7f55dc0ae5a0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.993+0000 7f55e0e00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55bc0039a0 0x7f55bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.994+0000 7f55e1800640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 0x7f55bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.994+0000 7f55d3e00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55bc0047c0 0x7f55bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.994+0000 7f55e0e00640 1 -- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 msgr2=0x7f55bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:18.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.994+0000 7f55e0e00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55dc0ae5a0 0x7f55bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.994+0000 7f55e0e00640 1 -- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55bc0047c0 msgr2=0x7f55bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:18.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.994+0000 7f55e0e00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55bc0047c0 0x7f55bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.994+0000 7f55e0e00640 1 -- 172.21.15.141:0/786795765 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc003520 con 0x7f55bc0039a0 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.995+0000 7f55e0e00640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55bc0039a0 0x7f55bc003dc0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009a90 tx=0x7f55bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afaffd52578332c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.995+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc02c660 con 0x7f55bc0039a0 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.995+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc02c7c0 con 0x7f55bc0039a0 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.995+0000 7f55dbe00640 1 -- 172.21.15.141:0/786795765 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc02cb50 con 0x7f55bc0039a0 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.996+0000 7f55e3800640 1 -- 172.21.15.141:0/786795765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55bc0039a0 msgr2=0x7f55bc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.996+0000 7f55e3800640 1 --2- 172.21.15.141:0/786795765 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55bc0039a0 0x7f55bc003dc0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009a90 tx=0x7f55bc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.996+0000 7f55e1800640 1 -- 172.21.15.141:0/786795765 reap_dead start 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.996+0000 7f55e3800640 1 -- 172.21.15.141:0/786795765 shutdown_connections 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.996+0000 7f55e3800640 1 -- 172.21.15.141:0/786795765 >> 172.21.15.141:0/786795765 conn(0x7f55dc06a800 msgr2=0x7f55dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.996+0000 7f55e3800640 1 -- 172.21.15.141:0/786795765 shutdown_connections 2024-01-16T15:23:18.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.996+0000 7f55e3800640 1 -- 172.21.15.141:0/786795765 wait complete. 2024-01-16T15:23:18.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.998+0000 7f55e3800640 1 Processor -- start 2024-01-16T15:23:18.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.998+0000 7f55e3800640 1 -- start start 2024-01-16T15:23:18.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.999+0000 7f55e3800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55bc0039a0 0x7f55dc0acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.999+0000 7f55e3800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55dc0cdd20 0x7f55dc0ce180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.999+0000 7f55e3800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55dc0cbbf0 0x7f55dc0cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.999+0000 7f55e3800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc147180 con 0x7f55dc0cdd20 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.999+0000 7f55e3800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0ce6c0 con 0x7f55dc0cbbf0 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:18.999+0000 7f55e3800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc1473d0 con 0x7f55bc0039a0 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55dc0cdd20 0x7f55dc0ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55dc0cdd20 0x7f55dc0ce180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57392/0 (socket says 172.21.15.141:57392) 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 -- 172.21.15.141:0/4157610246 learned_addr learned my addr 172.21.15.141:0/4157610246 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55e0e00640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55bc0039a0 0x7f55dc0acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55e1800640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55dc0cbbf0 0x7f55dc0cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 -- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55bc0039a0 msgr2=0x7f55dc0acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55bc0039a0 0x7f55dc0acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 -- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55dc0cbbf0 msgr2=0x7f55dc0cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55dc0cbbf0 0x7f55dc0cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 -- 172.21.15.141:0/4157610246 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc1476b0 con 0x7f55dc0cdd20 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55e1800640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55dc0cbbf0 0x7f55dc0cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.000+0000 7f55d3e00640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55dc0cdd20 0x7f55dc0ce180 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f55c400e870 tx=0x7f55c400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.001+0000 7f55d9600640 1 -- 172.21.15.141:0/4157610246 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c400c8d0 con 0x7f55dc0cdd20 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.001+0000 7f55d9600640 1 -- 172.21.15.141:0/4157610246 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c400ca50 con 0x7f55dc0cdd20 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.001+0000 7f55e3800640 1 -- 172.21.15.141:0/4157610246 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc147990 con 0x7f55dc0cdd20 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.001+0000 7f55d9600640 1 -- 172.21.15.141:0/4157610246 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c400ce00 con 0x7f55dc0cdd20 2024-01-16T15:23:18.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.001+0000 7f55e3800640 1 -- 172.21.15.141:0/4157610246 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc0cc590 con 0x7f55dc0cdd20 2024-01-16T15:23:19.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.002+0000 7f55e3800640 1 -- 172.21.15.141:0/4157610246 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc0675d0 con 0x7f55dc0cdd20 2024-01-16T15:23:19.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.003+0000 7f55d9600640 1 -- 172.21.15.141:0/4157610246 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f55c401d460 con 0x7f55dc0cdd20 2024-01-16T15:23:19.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.004+0000 7f55d9600640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55b406c5e0 0x7f55b406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:19.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.004+0000 7f55d9600640 1 -- 172.21.15.141:0/4157610246 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f55c4088b60 con 0x7f55dc0cdd20 2024-01-16T15:23:19.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.005+0000 7f55e0e00640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55b406c5e0 0x7f55b406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:19.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.006+0000 7f55e0e00640 1 --2- 172.21.15.141:0/4157610246 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55b406c5e0 0x7f55b406eaa0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f55cc036000 tx=0x7f55cc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:19.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.009+0000 7f55d9600640 1 -- 172.21.15.141:0/4157610246 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55c4055080 con 0x7f55dc0cdd20 2024-01-16T15:23:19.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.231+0000 7f55e3800640 1 -- 172.21.15.141:0/4157610246 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55dc070550 con 0x7f55b406c5e0 2024-01-16T15:23:19.241 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:19.246+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:19.245+0000 7f55d9600640 1 -- 172.21.15.141:0/4157610246 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f55cc004020 con 0x7f55b406c5e0 2024-01-16T15:23:19.242 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:19.242 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:21.253 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:21.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.483+0000 7f37e9000640 1 Processor -- start 2024-01-16T15:23:21.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.483+0000 7f37e9000640 1 -- start start 2024-01-16T15:23:21.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e9000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413df80 0x7f37e415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e9000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f37e40cf9a0 0x7f37e40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e9000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e406c470 con 0x7f37e413e360 2024-01-16T15:23:21.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e9000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e406c750 con 0x7f37e40cf9a0 2024-01-16T15:23:21.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e9000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e406ca30 con 0x7f37e413df80 2024-01-16T15:23:21.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f37e40cf9a0 0x7f37e40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f37e40cf9a0 0x7f37e40cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57408/0 (socket says 172.21.15.141:57408) 2024-01-16T15:23:21.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e2800640 1 -- 172.21.15.141:0/2448443173 learned_addr learned my addr 172.21.15.141:0/2448443173 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.485+0000 7f37e1e00640 1 --2- 172.21.15.141:0/2448443173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413df80 0x7f37e415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.486+0000 7f37e2800640 1 -- 172.21.15.141:0/2448443173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413df80 msgr2=0x7f37e415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.486+0000 7f37e2800640 1 --2- 172.21.15.141:0/2448443173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413df80 0x7f37e415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.486+0000 7f37e2800640 1 -- 172.21.15.141:0/2448443173 >> v1:172.21.15.188:6789/0 conn(0x7f37e413e360 legacy=0x7f37e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.486+0000 7f37e2800640 1 -- 172.21.15.141:0/2448443173 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e406cd10 con 0x7f37e40cf9a0 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.487+0000 7f37e2800640 1 --2- 172.21.15.141:0/2448443173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 0x7f37e40cd840 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009e50 tx=0x7f37dc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0ac191702c38af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.487+0000 7f37e1400640 1 -- 172.21.15.141:0/2448443173 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc02cb00 con 0x7f37e40cf9a0 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.487+0000 7f37e1400640 1 -- 172.21.15.141:0/2448443173 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc02cc60 con 0x7f37e40cf9a0 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.487+0000 7f37e1400640 1 -- 172.21.15.141:0/2448443173 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0343f0 con 0x7f37e40cf9a0 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 -- 172.21.15.141:0/2448443173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 msgr2=0x7f37e40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 --2- 172.21.15.141:0/2448443173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 0x7f37e40cd840 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009e50 tx=0x7f37dc02bf60 comp rx=0 tx=0).stop 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 -- 172.21.15.141:0/2448443173 shutdown_connections 2024-01-16T15:23:21.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 --2- 172.21.15.141:0/2448443173 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 0x7f37e40cd840 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 --2- 172.21.15.141:0/2448443173 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413df80 0x7f37e415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 -- 172.21.15.141:0/2448443173 >> 172.21.15.141:0/2448443173 conn(0x7f37e406a800 msgr2=0x7f37e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 -- 172.21.15.141:0/2448443173 shutdown_connections 2024-01-16T15:23:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.488+0000 7f37e9000640 1 -- 172.21.15.141:0/2448443173 wait complete. 2024-01-16T15:23:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.490+0000 7f37e9000640 1 Processor -- start 2024-01-16T15:23:21.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.490+0000 7f37e9000640 1 -- start start 2024-01-16T15:23:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e9000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 0x7f37e40b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e9000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e413df80 0x7f37e40a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e9000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413e360 0x7f37e40ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e9000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e414a710 con 0x7f37e40cf9a0 2024-01-16T15:23:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e9000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4149a90 con 0x7f37e413df80 2024-01-16T15:23:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e9000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4149d70 con 0x7f37e413e360 2024-01-16T15:23:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 0x7f37e40b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e1e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 0x7f37e40b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:57414/0 (socket says 172.21.15.141:57414) 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e1e00640 1 -- 172.21.15.141:0/738902994 learned_addr learned my addr 172.21.15.141:0/738902994 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.491+0000 7f37e2800640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413e360 0x7f37e40ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37d9200640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e413df80 0x7f37e40a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e1e00640 1 -- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413e360 msgr2=0x7f37e40ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e1e00640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f37e413e360 0x7f37e40ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e1e00640 1 -- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e413df80 msgr2=0x7f37e40a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e1e00640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e413df80 0x7f37e40a88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e1e00640 1 -- 172.21.15.141:0/738902994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc009b30 con 0x7f37e40cf9a0 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37d9200640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e413df80 0x7f37e40a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e1e00640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37e40cf9a0 0x7f37e40b2f60 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f37d400d800 tx=0x7f37d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37daa00640 1 -- 172.21.15.141:0/738902994 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d400b6c0 con 0x7f37e40cf9a0 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37daa00640 1 -- 172.21.15.141:0/738902994 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d400b820 con 0x7f37e40cf9a0 2024-01-16T15:23:21.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37daa00640 1 -- 172.21.15.141:0/738902994 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d400bbb0 con 0x7f37e40cf9a0 2024-01-16T15:23:21.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e9000640 1 -- 172.21.15.141:0/738902994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e414a050 con 0x7f37e40cf9a0 2024-01-16T15:23:21.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.492+0000 7f37e9000640 1 -- 172.21.15.141:0/738902994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e41f35a0 con 0x7f37e40cf9a0 2024-01-16T15:23:21.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.494+0000 7f37e9000640 1 -- 172.21.15.141:0/738902994 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e41f3760 con 0x7f37e40cf9a0 2024-01-16T15:23:21.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.495+0000 7f37daa00640 1 -- 172.21.15.141:0/738902994 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f37d400bd10 con 0x7f37e40cf9a0 2024-01-16T15:23:21.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.496+0000 7f37daa00640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37bc06c5e0 0x7f37bc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:21.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.496+0000 7f37daa00640 1 -- 172.21.15.141:0/738902994 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f37d4089120 con 0x7f37e40cf9a0 2024-01-16T15:23:21.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.497+0000 7f37d9200640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37bc06c5e0 0x7f37bc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:21.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.498+0000 7f37d9200640 1 --2- 172.21.15.141:0/738902994 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f37bc06c5e0 0x7f37bc06eaa0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f37e40cd290 tx=0x7f37cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:21.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.500+0000 7f37daa00640 1 -- 172.21.15.141:0/738902994 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37e41f3760 con 0x7f37e40cf9a0 2024-01-16T15:23:21.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.724+0000 7f37e9000640 1 -- 172.21.15.141:0/738902994 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f37e415f9c0 con 0x7f37bc06c5e0 2024-01-16T15:23:21.736 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:21.741+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:21.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:21.741+0000 7f37daa00640 1 -- 172.21.15.141:0/738902994 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f37cc0013e0 con 0x7f37bc06c5e0 2024-01-16T15:23:21.737 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:21.738 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:23.753 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:23.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.984+0000 7fbf15800640 1 Processor -- start 2024-01-16T15:23:23.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.984+0000 7fbf15800640 1 -- start start 2024-01-16T15:23:23.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf15800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbf100cf9a0 0x7fbf1015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf15800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 0x7fbf100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf15800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100b1ee0 con 0x7fbf10143ae0 2024-01-16T15:23:23.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf15800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100b2780 con 0x7fbf100cf9a0 2024-01-16T15:23:23.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf15800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1015e990 con 0x7fbf1015e1e0 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf07e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbf100cf9a0 0x7fbf1015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf0ea00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbf10143ae0 0x7fbf100cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40106/0 (socket says 172.21.15.141:40106) 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf0ea00640 1 -- 172.21.15.141:0/1465064884 learned_addr learned my addr 172.21.15.141:0/1465064884 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf07e00640 1 -- 172.21.15.141:0/1465064884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 msgr2=0x7fbf100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.986+0000 7fbf0f400640 1 --2- 172.21.15.141:0/1465064884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 0x7fbf100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.987+0000 7fbf07e00640 1 --2- 172.21.15.141:0/1465064884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 0x7fbf100702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.987+0000 7fbf07e00640 1 -- 172.21.15.141:0/1465064884 >> v1:172.21.15.141:6789/0 conn(0x7fbf10143ae0 legacy=0x7fbf100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.987+0000 7fbf0e000640 1 -- 172.21.15.141:0/1465064884 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973254078 0 0) 0x7fbf100b1ee0 con 0x7fbf10143ae0 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.987+0000 7fbf07e00640 1 -- 172.21.15.141:0/1465064884 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100b2130 con 0x7fbf100cf9a0 2024-01-16T15:23:23.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.987+0000 7fbf0f400640 1 --2- 172.21.15.141:0/1465064884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 0x7fbf100702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:23.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.988+0000 7fbf07e00640 1 --2- 172.21.15.141:0/1465064884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 0x7fbf1015dca0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fbef80097a0 tx=0x7fbef802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecc885b10e7a4ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.988+0000 7fbf0e000640 1 -- 172.21.15.141:0/1465064884 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8031830 con 0x7fbf100cf9a0 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.988+0000 7fbf0e000640 1 -- 172.21.15.141:0/1465064884 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef8031990 con 0x7fbf100cf9a0 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.988+0000 7fbf0e000640 1 -- 172.21.15.141:0/1465064884 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8031d20 con 0x7fbf100cf9a0 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 -- 172.21.15.141:0/1465064884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 msgr2=0x7fbf1015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 --2- 172.21.15.141:0/1465064884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 0x7fbf1015dca0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fbef80097a0 tx=0x7fbef802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 -- 172.21.15.141:0/1465064884 shutdown_connections 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 --2- 172.21.15.141:0/1465064884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 0x7fbf100702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 --2- 172.21.15.141:0/1465064884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 0x7fbf1015dca0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 -- 172.21.15.141:0/1465064884 >> 172.21.15.141:0/1465064884 conn(0x7fbf1006a800 msgr2=0x7fbf1006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 -- 172.21.15.141:0/1465064884 shutdown_connections 2024-01-16T15:23:23.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.989+0000 7fbf15800640 1 -- 172.21.15.141:0/1465064884 wait complete. 2024-01-16T15:23:23.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.991+0000 7fbf15800640 1 Processor -- start 2024-01-16T15:23:23.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.991+0000 7fbf15800640 1 -- start start 2024-01-16T15:23:23.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf15800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 0x7fbf1013f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf15800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf10143ae0 0x7fbf101f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf15800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 0x7fbf1013c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf15800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1013e480 con 0x7fbf100cf9a0 2024-01-16T15:23:23.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf15800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1013d000 con 0x7fbf10143ae0 2024-01-16T15:23:23.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf15800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1013cd20 con 0x7fbf1015e1e0 2024-01-16T15:23:23.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 0x7fbf1013f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 0x7fbf1013f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34072/0 (socket says 172.21.15.141:34072) 2024-01-16T15:23:23.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 -- 172.21.15.141:0/2529342858 learned_addr learned my addr 172.21.15.141:0/2529342858 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 -- 172.21.15.141:0/2529342858 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 msgr2=0x7fbf1013c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 --2- 172.21.15.141:0/2529342858 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbf1015e1e0 0x7fbf1013c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 -- 172.21.15.141:0/2529342858 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf10143ae0 msgr2=0x7fbf101f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 --2- 172.21.15.141:0/2529342858 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf10143ae0 0x7fbf101f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.992+0000 7fbf0ea00640 1 -- 172.21.15.141:0/2529342858 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8009400 con 0x7fbf100cf9a0 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.993+0000 7fbf0ea00640 1 --2- 172.21.15.141:0/2529342858 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf100cf9a0 0x7fbf1013f1e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000c4d0 tx=0x7fbf0000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.993+0000 7fbf06a00640 1 -- 172.21.15.141:0/2529342858 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf000074e0 con 0x7fbf100cf9a0 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.993+0000 7fbf06a00640 1 -- 172.21.15.141:0/2529342858 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf00007660 con 0x7fbf100cf9a0 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.993+0000 7fbf06a00640 1 -- 172.21.15.141:0/2529342858 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00004a20 con 0x7fbf100cf9a0 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.993+0000 7fbf15800640 1 -- 172.21.15.141:0/2529342858 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1013e700 con 0x7fbf100cf9a0 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.993+0000 7fbf15800640 1 -- 172.21.15.141:0/2529342858 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1013ecc0 con 0x7fbf100cf9a0 2024-01-16T15:23:23.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.994+0000 7fbf15800640 1 -- 172.21.15.141:0/2529342858 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1013eee0 con 0x7fbf100cf9a0 2024-01-16T15:23:23.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.995+0000 7fbf06a00640 1 -- 172.21.15.141:0/2529342858 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fbf00004b80 con 0x7fbf100cf9a0 2024-01-16T15:23:23.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.996+0000 7fbf06a00640 1 --2- 172.21.15.141:0/2529342858 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbee806c5e0 0x7fbee806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:23.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.996+0000 7fbf06a00640 1 -- 172.21.15.141:0/2529342858 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbf00087dd0 con 0x7fbf100cf9a0 2024-01-16T15:23:23.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.997+0000 7fbf07e00640 1 --2- 172.21.15.141:0/2529342858 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbee806c5e0 0x7fbee806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:23.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:23.998+0000 7fbf07e00640 1 --2- 172.21.15.141:0/2529342858 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fbee806c5e0 0x7fbee806eaa0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbef8036000 tx=0x7fbef8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:23.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:24.001+0000 7fbf06a00640 1 -- 172.21.15.141:0/2529342858 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf000542f0 con 0x7fbf100cf9a0 2024-01-16T15:23:24.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:24.228+0000 7fbf15800640 1 -- 172.21.15.141:0/2529342858 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf1015e670 con 0x7fbee806c5e0 2024-01-16T15:23:24.237 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:24.242+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:24.242+0000 7fbf06a00640 1 -- 172.21.15.141:0/2529342858 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbef8031b80 con 0x7fbee806c5e0 2024-01-16T15:23:24.238 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:24.238 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:26.256 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:26.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.487+0000 7f43fac00640 1 Processor -- start 2024-01-16T15:23:26.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.487+0000 7f43fac00640 1 -- start start 2024-01-16T15:23:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.488+0000 7f43fac00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f43f40b3f60 0x7f43f406de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.489+0000 7f43fac00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f406e7b0 con 0x7f43f415aef0 2024-01-16T15:23:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.489+0000 7f43fac00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f415b550 con 0x7f43f406e350 2024-01-16T15:23:26.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.489+0000 7f43fac00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f40b29d0 con 0x7f43f40b3f60 2024-01-16T15:23:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.489+0000 7f43f8c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f43f415aef0 0x7f43f40aca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40120/0 (socket says 172.21.15.141:40120) 2024-01-16T15:23:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.489+0000 7f43f8c00640 1 -- 172.21.15.141:0/3151003304 learned_addr learned my addr 172.21.15.141:0/3151003304 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.489+0000 7f43f3e00640 1 --2- 172.21.15.141:0/3151003304 >> v2:172.21.15.188:3300/0 conn(0x7f43f40b3f60 0x7f43f406de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924899035 0 0) 0x7f43ec001490 con 0x7f43f415aef0 2024-01-16T15:23:26.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d4003520 con 0x7f43f415aef0 2024-01-16T15:23:26.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 696439928 0 0) 0x7f43d4003520 con 0x7f43f415aef0 2024-01-16T15:23:26.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 >> v2:172.21.15.188:3300/0 conn(0x7f43f40b3f60 msgr2=0x7f43f406de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:26.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 --2- 172.21.15.141:0/3151003304 >> v2:172.21.15.188:3300/0 conn(0x7f43f40b3f60 0x7f43f406de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 >> v1:172.21.15.188:6789/0 conn(0x7f43f406e350 legacy=0x7f43f415a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f40b2df0 con 0x7f43f415aef0 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f43ec001b80 con 0x7f43f415aef0 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 >> v1:172.21.15.141:6789/0 conn(0x7f43f415aef0 legacy=0x7f43f40aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43d40039c0 0x7f43d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43d40052a0 0x7f43d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f40b29d0 con 0x7f43d40047c0 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d400c1a0 con 0x7f43d40039c0 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.490+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d400c360 con 0x7f43d40052a0 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.491+0000 7f43eb200640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.491+0000 7f43eb200640 1 -- 172.21.15.141:0/3151003304 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43d40052a0 msgr2=0x7f43d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.491+0000 7f43eb200640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43d40052a0 0x7f43d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.491+0000 7f43eb200640 1 -- 172.21.15.141:0/3151003304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43d40039c0 msgr2=0x7f43d4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.491+0000 7f43eb200640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43d40039c0 0x7f43d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.491+0000 7f43eb200640 1 -- 172.21.15.141:0/3151003304 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d4003520 con 0x7f43d40047c0 2024-01-16T15:23:26.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.491+0000 7f43f8c00640 1 -- 172.21.15.141:0/3151003304 reap_dead start 2024-01-16T15:23:26.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.492+0000 7f43eb200640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43d4004c40 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f43dc001750 tx=0x7f43d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc56d9e5e29e7cdd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:26.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.492+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0052d0 con 0x7f43d40047c0 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.492+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc02e050 con 0x7f43d40047c0 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.492+0000 7f43f3400640 1 -- 172.21.15.141:0/3151003304 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc003830 con 0x7f43d40047c0 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.493+0000 7f43fac00640 1 -- 172.21.15.141:0/3151003304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 msgr2=0x7f43d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.493+0000 7f43fac00640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43d4004c40 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f43dc001750 tx=0x7f43d40057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.493+0000 7f43fac00640 1 -- 172.21.15.141:0/3151003304 shutdown_connections 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.493+0000 7f43fac00640 1 --2- 172.21.15.141:0/3151003304 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43d4004c40 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.493+0000 7f43fac00640 1 -- 172.21.15.141:0/3151003304 >> 172.21.15.141:0/3151003304 conn(0x7f43f40b0c00 msgr2=0x7f43f40b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.493+0000 7f43fac00640 1 -- 172.21.15.141:0/3151003304 shutdown_connections 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.493+0000 7f43fac00640 1 -- 172.21.15.141:0/3151003304 wait complete. 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.495+0000 7f43fac00640 1 Processor -- start 2024-01-16T15:23:26.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.495+0000 7f43fac00640 1 -- start start 2024-01-16T15:23:26.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43fac00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43f4153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43fac00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43f415b750 0x7f43f415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43fac00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f40af150 0x7f43f40af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43fac00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f415c0f0 con 0x7f43d40047c0 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43fac00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f4153f60 con 0x7f43f40af150 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43fac00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f40afec0 con 0x7f43f415b750 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43f4153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43f4153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34088/0 (socket says 172.21.15.141:34088) 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.496+0000 7f43f3e00640 1 -- 172.21.15.141:0/3627680500 learned_addr learned my addr 172.21.15.141:0/3627680500 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43eb200640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43f415b750 0x7f43f415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f3e00640 1 -- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43f415b750 msgr2=0x7f43f415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f8c00640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f40af150 0x7f43f40af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f3e00640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43f415b750 0x7f43f415bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f3e00640 1 -- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f40af150 msgr2=0x7f43f40af5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f3e00640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f40af150 0x7f43f40af5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f3e00640 1 -- 172.21.15.141:0/3627680500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f40b01a0 con 0x7f43d40047c0 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43eb200640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f43f415b750 0x7f43f415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f3e00640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43d40047c0 0x7f43f4153a20 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f43e400e770 tx=0x7f43e400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f0c00640 1 -- 172.21.15.141:0/3627680500 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e400c730 con 0x7f43d40047c0 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f0c00640 1 -- 172.21.15.141:0/3627680500 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e400c890 con 0x7f43d40047c0 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43f0c00640 1 -- 172.21.15.141:0/3627680500 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e400cc40 con 0x7f43d40047c0 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.497+0000 7f43fac00640 1 -- 172.21.15.141:0/3627680500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f40b0480 con 0x7f43d40047c0 2024-01-16T15:23:26.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.498+0000 7f43fac00640 1 -- 172.21.15.141:0/3627680500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f40c83b0 con 0x7f43d40047c0 2024-01-16T15:23:26.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.499+0000 7f43fac00640 1 -- 172.21.15.141:0/3627680500 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f40cec40 con 0x7f43d40047c0 2024-01-16T15:23:26.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.500+0000 7f43f0c00640 1 -- 172.21.15.141:0/3627680500 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f43e4013470 con 0x7f43d40047c0 2024-01-16T15:23:26.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.500+0000 7f43f0c00640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43cc06c610 0x7f43cc06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:26.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.501+0000 7f43f0c00640 1 -- 172.21.15.141:0/3627680500 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f43e4088dd0 con 0x7f43d40047c0 2024-01-16T15:23:26.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.501+0000 7f43eb200640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43cc06c610 0x7f43cc06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:26.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.502+0000 7f43eb200640 1 --2- 172.21.15.141:0/3627680500 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f43cc06c610 0x7f43cc06ead0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f43dc033000 tx=0x7f43dc009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:26.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.505+0000 7f43f0c00640 1 -- 172.21.15.141:0/3627680500 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43e4055350 con 0x7f43d40047c0 2024-01-16T15:23:26.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.727+0000 7f43fac00640 1 -- 172.21.15.141:0/3627680500 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43f40b2b10 con 0x7f43cc06c610 2024-01-16T15:23:26.737 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:26.741+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:26.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:26.741+0000 7f43f0c00640 1 -- 172.21.15.141:0/3627680500 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f43f40b2b10 con 0x7f43cc06c610 2024-01-16T15:23:26.738 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:26.738 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:28.756 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:28.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.994+0000 7fb6e5600640 1 Processor -- start 2024-01-16T15:23:28.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.994+0000 7fb6e5600640 1 -- start start 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.995+0000 7fb6e5600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fb6e0143ae0 0x7fb6e00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.995+0000 7fb6e5600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb6e015e3b0 0x7fb6e00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.995+0000 7fb6e5600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e00cfc30 con 0x7fb6e00cf7d0 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.995+0000 7fb6e5600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e00b1ee0 con 0x7fb6e015e3b0 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.995+0000 7fb6e5600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e00b2780 con 0x7fb6e0143ae0 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6d5e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fb6e00cf7d0 0x7fb6e015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40132/0 (socket says 172.21.15.141:40132) 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6d5e00640 1 -- 172.21.15.141:0/1215103400 learned_addr learned my addr 172.21.15.141:0/1215103400 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6df400640 1 --2- 172.21.15.141:0/1215103400 >> v2:172.21.15.141:3300/0 conn(0x7fb6e015e3b0 0x7fb6e00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6de000640 1 -- 172.21.15.141:0/1215103400 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492650643 0 0) 0x7fb6e00cfc30 con 0x7fb6e00cf7d0 2024-01-16T15:23:28.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6dea00640 1 --2- 172.21.15.141:0/1215103400 >> v2:172.21.15.188:3300/0 conn(0x7fb6e0143ae0 0x7fb6e00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6de000640 1 -- 172.21.15.141:0/1215103400 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c0003520 con 0x7fb6e00cf7d0 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6df400640 1 -- 172.21.15.141:0/1215103400 >> v2:172.21.15.188:3300/0 conn(0x7fb6e0143ae0 msgr2=0x7fb6e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6df400640 1 --2- 172.21.15.141:0/1215103400 >> v2:172.21.15.188:3300/0 conn(0x7fb6e0143ae0 0x7fb6e00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6df400640 1 -- 172.21.15.141:0/1215103400 >> v1:172.21.15.141:6789/0 conn(0x7fb6e00cf7d0 legacy=0x7fb6e015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.996+0000 7fb6df400640 1 -- 172.21.15.141:0/1215103400 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e015e990 con 0x7fb6e015e3b0 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.997+0000 7fb6df400640 1 --2- 172.21.15.141:0/1215103400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 0x7fb6e00702c0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00cfea0 tx=0x7fb6d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773b962b75136d9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6de000640 1 -- 172.21.15.141:0/1215103400 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d802cc80 con 0x7fb6e015e3b0 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6de000640 1 -- 172.21.15.141:0/1215103400 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d802cde0 con 0x7fb6e015e3b0 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6de000640 1 -- 172.21.15.141:0/1215103400 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8034660 con 0x7fb6e015e3b0 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 -- 172.21.15.141:0/1215103400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 msgr2=0x7fb6e00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 --2- 172.21.15.141:0/1215103400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 0x7fb6e00702c0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00cfea0 tx=0x7fb6d802bed0 comp rx=0 tx=0).stop 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 -- 172.21.15.141:0/1215103400 shutdown_connections 2024-01-16T15:23:28.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 --2- 172.21.15.141:0/1215103400 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 0x7fb6e00702c0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 --2- 172.21.15.141:0/1215103400 >> v2:172.21.15.188:3300/0 conn(0x7fb6e0143ae0 0x7fb6e00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 -- 172.21.15.141:0/1215103400 >> 172.21.15.141:0/1215103400 conn(0x7fb6e006a800 msgr2=0x7fb6e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 -- 172.21.15.141:0/1215103400 shutdown_connections 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:28.998+0000 7fb6e5600640 1 -- 172.21.15.141:0/1215103400 wait complete. 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.000+0000 7fb6e5600640 1 Processor -- start 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.000+0000 7fb6e5600640 1 -- start start 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.001+0000 7fb6e5600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6e00cf7d0 0x7fb6e00ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.001+0000 7fb6e5600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6e0143ae0 0x7fb6e01f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.001+0000 7fb6e5600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 0x7fb6e015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.001+0000 7fb6e5600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e01f4ca0 con 0x7fb6e015e3b0 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.001+0000 7fb6e5600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e00a9dc0 con 0x7fb6e0143ae0 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.001+0000 7fb6e5600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e00aa0a0 con 0x7fb6e00cf7d0 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.001+0000 7fb6df400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 0x7fb6e015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 0x7fb6e015a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34098/0 (socket says 172.21.15.141:34098) 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 -- 172.21.15.141:0/1553061262 learned_addr learned my addr 172.21.15.141:0/1553061262 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6dea00640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6e00cf7d0 0x7fb6e00ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6d5e00640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6e0143ae0 0x7fb6e01f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 -- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6e00cf7d0 msgr2=0x7fb6e00ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6e00cf7d0 0x7fb6e00ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 -- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6e0143ae0 msgr2=0x7fb6e01f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6e0143ae0 0x7fb6e01f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 -- 172.21.15.141:0/1553061262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d8009a70 con 0x7fb6e015e3b0 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6dea00640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fb6e00cf7d0 0x7fb6e00ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:29.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.002+0000 7fb6df400640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e015e3b0 0x7fb6e015a920 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8009e80 tx=0x7fb6d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.003+0000 7fb6d7400640 1 -- 172.21.15.141:0/1553061262 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8040650 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.003+0000 7fb6d7400640 1 -- 172.21.15.141:0/1553061262 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d80407b0 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.003+0000 7fb6d7400640 1 -- 172.21.15.141:0/1553061262 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8040b60 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.003+0000 7fb6e5600640 1 -- 172.21.15.141:0/1553061262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e00aa380 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.003+0000 7fb6e5600640 1 -- 172.21.15.141:0/1553061262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e00aa940 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.004+0000 7fb6e5600640 1 -- 172.21.15.141:0/1553061262 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e00b28f0 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.005+0000 7fb6d7400640 1 -- 172.21.15.141:0/1553061262 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fb6d8040cc0 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.006+0000 7fb6d7400640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb6b806c5e0 0x7fb6b806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.006+0000 7fb6d7400640 1 -- 172.21.15.141:0/1553061262 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb6d80ab840 con 0x7fb6e015e3b0 2024-01-16T15:23:29.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.006+0000 7fb6dea00640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb6b806c5e0 0x7fb6b806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:29.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.008+0000 7fb6dea00640 1 --2- 172.21.15.141:0/1553061262 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fb6b806c5e0 0x7fb6b806eaa0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0006fd0 tx=0x7fb6d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:29.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.011+0000 7fb6d7400640 1 -- 172.21.15.141:0/1553061262 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6d8077dc0 con 0x7fb6e015e3b0 2024-01-16T15:23:29.230 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.234+0000 7fb6e5600640 1 -- 172.21.15.141:0/1553061262 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6e00b2410 con 0x7fb6b806c5e0 2024-01-16T15:23:29.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:29.247+0000 7fb6d7400640 1 -- 172.21.15.141:0/1553061262 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb6d00095d0 con 0x7fb6b806c5e0 2024-01-16T15:23:29.243 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:29.244 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:29.244 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:29.247+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:31.265 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:31.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.495+0000 7fd759200640 1 Processor -- start 2024-01-16T15:23:31.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.496+0000 7fd759200640 1 -- start start 2024-01-16T15:23:31.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.497+0000 7fd759200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 0x7fd7540cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.497+0000 7fd759200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75406ca10 con 0x7fd75406dab0 2024-01-16T15:23:31.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.497+0000 7fd759200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75406ccf0 con 0x7fd7541447f0 2024-01-16T15:23:31.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.497+0000 7fd759200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75406d690 con 0x7fd75406c470 2024-01-16T15:23:31.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.498+0000 7fd753e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd75406dab0 0x7fd75415c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:40146/0 (socket says 172.21.15.141:40146) 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.498+0000 7fd753e00640 1 -- 172.21.15.141:0/2037729769 learned_addr learned my addr 172.21.15.141:0/2037729769 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.498+0000 7fd752a00640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 0x7fd7540cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.498+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444334584 0 0) 0x7fd75406ca10 con 0x7fd75406dab0 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.498+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734003520 con 0x7fd75406dab0 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1474108869 0 0) 0x7fd734003520 con 0x7fd75406dab0 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 msgr2=0x7fd7540cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 0x7fd7540cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 >> v1:172.21.15.188:6789/0 conn(0x7fd7541447f0 legacy=0x7fd75415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75406cfd0 con 0x7fd75406dab0 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd74c001670 con 0x7fd75406dab0 2024-01-16T15:23:31.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 >> v1:172.21.15.141:6789/0 conn(0x7fd75406dab0 legacy=0x7fd75415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd734003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 0x7fd7340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75406d690 con 0x7fd7340039a0 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73400bae0 con 0x7fd734004d00 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73400bca0 con 0x7fd75406c470 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd753400640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd734003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd752a00640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 0x7fd7340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd753400640 1 -- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 msgr2=0x7fd7340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd753400640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75406c470 0x7fd7340047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd753400640 1 -- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd734004d00 msgr2=0x7fd734005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:31.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.499+0000 7fd753e00640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.500+0000 7fd753400640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.500+0000 7fd753400640 1 -- 172.21.15.141:0/2037729769 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd734003520 con 0x7fd7340039a0 2024-01-16T15:23:31.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.500+0000 7fd753e00640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:31.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.501+0000 7fd753400640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd734003dc0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fd744001e50 tx=0x7fd7340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4fb7520ddb9cb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.501+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744003b70 con 0x7fd7340039a0 2024-01-16T15:23:31.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.501+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd744003cd0 con 0x7fd7340039a0 2024-01-16T15:23:31.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.501+0000 7fd752000640 1 -- 172.21.15.141:0/2037729769 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744005310 con 0x7fd7340039a0 2024-01-16T15:23:31.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.502+0000 7fd759200640 1 -- 172.21.15.141:0/2037729769 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 msgr2=0x7fd734003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.502+0000 7fd759200640 1 --2- 172.21.15.141:0/2037729769 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd734003dc0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fd744001e50 tx=0x7fd7340057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.502+0000 7fd753e00640 1 -- 172.21.15.141:0/2037729769 reap_dead start 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.502+0000 7fd759200640 1 -- 172.21.15.141:0/2037729769 shutdown_connections 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.502+0000 7fd759200640 1 -- 172.21.15.141:0/2037729769 >> 172.21.15.141:0/2037729769 conn(0x7fd75406a3e0 msgr2=0x7fd75406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.502+0000 7fd759200640 1 -- 172.21.15.141:0/2037729769 shutdown_connections 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.502+0000 7fd759200640 1 -- 172.21.15.141:0/2037729769 wait complete. 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.503+0000 7fd759200640 1 Processor -- start 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.504+0000 7fd759200640 1 -- start start 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.504+0000 7fd759200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd754158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd759200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd754138c40 0x7fd7540b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd759200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75415ac80 0x7fd75415b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd759200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754139da0 con 0x7fd7340039a0 2024-01-16T15:23:31.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd759200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75415b670 con 0x7fd754138c40 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd759200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd754158cb0 con 0x7fd75415ac80 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd754158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd754158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:34110/0 (socket says 172.21.15.141:34110) 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 -- 172.21.15.141:0/2877621834 learned_addr learned my addr 172.21.15.141:0/2877621834 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 -- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75415ac80 msgr2=0x7fd75415b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 --2- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd75415ac80 0x7fd75415b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd752a00640 1 --2- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd754138c40 0x7fd7540b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 -- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd754138c40 msgr2=0x7fd7540b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 --2- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd754138c40 0x7fd7540b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.505+0000 7fd753400640 1 -- 172.21.15.141:0/2877621834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd754139120 con 0x7fd7340039a0 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.506+0000 7fd753400640 1 --2- 172.21.15.141:0/2877621834 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd7340039a0 0x7fd754158770 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fd74402bee0 tx=0x7fd7440014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.506+0000 7fd74b400640 1 -- 172.21.15.141:0/2877621834 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744005850 con 0x7fd7340039a0 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.506+0000 7fd74b400640 1 -- 172.21.15.141:0/2877621834 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd744003e40 con 0x7fd7340039a0 2024-01-16T15:23:31.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.506+0000 7fd74b400640 1 -- 172.21.15.141:0/2877621834 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744007230 con 0x7fd7340039a0 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.506+0000 7fd759200640 1 -- 172.21.15.141:0/2877621834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd754139400 con 0x7fd7340039a0 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.506+0000 7fd759200640 1 -- 172.21.15.141:0/2877621834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7541399c0 con 0x7fd7340039a0 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.507+0000 7fd759200640 1 -- 172.21.15.141:0/2877621834 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75413a010 con 0x7fd7340039a0 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.508+0000 7fd74b400640 1 -- 172.21.15.141:0/2877621834 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd744041020 con 0x7fd7340039a0 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.509+0000 7fd74b400640 1 --2- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd72c06c610 0x7fd72c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.509+0000 7fd74b400640 1 -- 172.21.15.141:0/2877621834 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7440ab020 con 0x7fd7340039a0 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.510+0000 7fd752a00640 1 --2- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd72c06c610 0x7fd72c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:31.509 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.511+0000 7fd752a00640 1 --2- 172.21.15.141:0/2877621834 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd72c06c610 0x7fd72c06ead0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd7541f1ff0 tx=0x7fd73c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:31.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.513+0000 7fd74b400640 1 -- 172.21.15.141:0/2877621834 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd744077540 con 0x7fd7340039a0 2024-01-16T15:23:31.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.739+0000 7fd759200640 1 -- 172.21.15.141:0/2877621834 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd754066ec0 con 0x7fd72c06c610 2024-01-16T15:23:31.748 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:31.751+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:31.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:31.752+0000 7fd74b400640 1 -- 172.21.15.141:0/2877621834 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd754066ec0 con 0x7fd72c06c610 2024-01-16T15:23:31.749 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:31.749 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:33.765 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:33.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.996+0000 7f28a7e00640 1 Processor -- start 2024-01-16T15:23:33.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.996+0000 7f28a7e00640 1 -- start start 2024-01-16T15:23:33.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.998+0000 7f28a7e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f28a815e170 0x7f28a80cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.998+0000 7f28a7e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80700f0 con 0x7f28a80cdef0 2024-01-16T15:23:33.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.998+0000 7f28a7e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80ce720 con 0x7f28a813cda0 2024-01-16T15:23:33.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.998+0000 7f28a7e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a80cea00 con 0x7f28a815e170 2024-01-16T15:23:33.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.998+0000 7f28a7400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f28a80cdef0 0x7f28a8070b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55040/0 (socket says 172.21.15.141:55040) 2024-01-16T15:23:33.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.998+0000 7f28a7400640 1 -- 172.21.15.141:0/3311250724 learned_addr learned my addr 172.21.15.141:0/3311250724 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:33.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a6000640 1 --2- 172.21.15.141:0/3311250724 >> v2:172.21.15.188:3300/0 conn(0x7f28a815e170 0x7f28a80cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366434521 0 0) 0x7f28a80700f0 con 0x7f28a80cdef0 2024-01-16T15:23:33.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2888003520 con 0x7f28a80cdef0 2024-01-16T15:23:33.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2850143101 0 0) 0x7f2888003520 con 0x7f28a80cdef0 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 >> v2:172.21.15.188:3300/0 conn(0x7f28a815e170 msgr2=0x7f28a80cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 --2- 172.21.15.141:0/3311250724 >> v2:172.21.15.188:3300/0 conn(0x7f28a815e170 0x7f28a80cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 >> v1:172.21.15.188:6789/0 conn(0x7f28a813cda0 legacy=0x7f28a815da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8070340 con 0x7f28a80cdef0 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f28a0001670 con 0x7f28a80cdef0 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 >> v1:172.21.15.141:6789/0 conn(0x7f28a80cdef0 legacy=0x7f28a8070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f2888003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28880047c0 0x7f2888004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28880052a0 0x7f288800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:33.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a80cea00 con 0x7f2888003970 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:33.999+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288800c150 con 0x7f28880052a0 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288800e520 con 0x7f28880047c0 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a6a00640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f2888003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a6000640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28880047c0 0x7f2888004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a6a00640 1 -- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28880047c0 msgr2=0x7f2888004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a7400640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28880052a0 0x7f288800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a6a00640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28880047c0 0x7f2888004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a6a00640 1 -- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28880052a0 msgr2=0x7f288800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a6a00640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28880052a0 0x7f288800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a6a00640 1 -- 172.21.15.141:0/3311250724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2888003520 con 0x7f2888003970 2024-01-16T15:23:33.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a7400640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28880052a0 0x7f288800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:34.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.000+0000 7f28a7400640 1 -- 172.21.15.141:0/3311250724 reap_dead start 2024-01-16T15:23:34.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.001+0000 7f28a6a00640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f2888003d70 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2890003d80 tx=0x7f28880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0bd32ce87bd2e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:34.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890032600 con 0x7f2888003970 2024-01-16T15:23:34.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289002e050 con 0x7f2888003970 2024-01-16T15:23:34.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a5600640 1 -- 172.21.15.141:0/3311250724 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28900329e0 con 0x7f2888003970 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a7e00640 1 -- 172.21.15.141:0/3311250724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 msgr2=0x7f2888003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a7e00640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f2888003d70 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2890003d80 tx=0x7f28880057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a7e00640 1 -- 172.21.15.141:0/3311250724 shutdown_connections 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a7e00640 1 --2- 172.21.15.141:0/3311250724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f2888003d70 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a7e00640 1 -- 172.21.15.141:0/3311250724 >> 172.21.15.141:0/3311250724 conn(0x7f28a806a3e0 msgr2=0x7f28a806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a7e00640 1 -- 172.21.15.141:0/3311250724 shutdown_connections 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.002+0000 7f28a7e00640 1 -- 172.21.15.141:0/3311250724 wait complete. 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.004+0000 7f28a7e00640 1 Processor -- start 2024-01-16T15:23:34.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.004+0000 7f28a7e00640 1 -- start start 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.005+0000 7f28a7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f28a81567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.005+0000 7f28a7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a8158a10 0x7f28a80b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.005+0000 7f28a7e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a80b1090 0x7f28a81f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.005+0000 7f28a7e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a80b1540 con 0x7f2888003970 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.005+0000 7f28a7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8156ce0 con 0x7f28a8158a10 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.005+0000 7f28a7e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8159aa0 con 0x7f28a80b1090 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f28a81567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f28a81567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38538/0 (socket says 172.21.15.141:38538) 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 -- 172.21.15.141:0/4052988388 learned_addr learned my addr 172.21.15.141:0/4052988388 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a7400640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a80b1090 0x7f28a81f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6000640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a8158a10 0x7f28a80b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 -- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a80b1090 msgr2=0x7f28a81f1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f28a80b1090 0x7f28a81f1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:34.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 -- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a8158a10 msgr2=0x7f28a80b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a8158a10 0x7f28a80b0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 -- 172.21.15.141:0/4052988388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a813da90 con 0x7f2888003970 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.006+0000 7f28a6a00640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2888003970 0x7f28a81567a0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f2890003d80 tx=0x7f28900036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.007+0000 7f289ea00640 1 -- 172.21.15.141:0/4052988388 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289003c410 con 0x7f2888003970 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.007+0000 7f289ea00640 1 -- 172.21.15.141:0/4052988388 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289003c570 con 0x7f2888003970 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.007+0000 7f28a7e00640 1 -- 172.21.15.141:0/4052988388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a8159cf0 con 0x7f2888003970 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.007+0000 7f289ea00640 1 -- 172.21.15.141:0/4052988388 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289003c8e0 con 0x7f2888003970 2024-01-16T15:23:34.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.007+0000 7f28a7e00640 1 -- 172.21.15.141:0/4052988388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a813d720 con 0x7f2888003970 2024-01-16T15:23:34.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.008+0000 7f28a7e00640 1 -- 172.21.15.141:0/4052988388 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a815e990 con 0x7f2888003970 2024-01-16T15:23:34.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.009+0000 7f289ea00640 1 -- 172.21.15.141:0/4052988388 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f2890039050 con 0x7f2888003970 2024-01-16T15:23:34.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.010+0000 7f289ea00640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f288006c610 0x7f288006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:34.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.010+0000 7f289ea00640 1 -- 172.21.15.141:0/4052988388 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f28900a9ed0 con 0x7f2888003970 2024-01-16T15:23:34.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.010+0000 7f28a6000640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f288006c610 0x7f288006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:34.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.012+0000 7f28a6000640 1 --2- 172.21.15.141:0/4052988388 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f288006c610 0x7f288006ead0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f28a81f1ba0 tx=0x7f289800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:34.010 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.014+0000 7f289ea00640 1 -- 172.21.15.141:0/4052988388 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28900767d0 con 0x7f2888003970 2024-01-16T15:23:34.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.237+0000 7f28a7e00640 1 -- 172.21.15.141:0/4052988388 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28a80ce340 con 0x7f288006c610 2024-01-16T15:23:34.247 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:34.250+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:34.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:34.251+0000 7f289ea00640 1 -- 172.21.15.141:0/4052988388 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f289800f040 con 0x7f288006c610 2024-01-16T15:23:34.248 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:34.248 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:36.262 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:36.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.492+0000 7f2bfde00640 1 Processor -- start 2024-01-16T15:23:36.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.492+0000 7f2bfde00640 1 -- start start 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfde00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2bf815e170 0x7f2bf80cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfde00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2bf80cdef0 0x7f2bf8070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfde00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80700f0 con 0x7f2bf813cda0 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfde00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf80ce720 con 0x7f2bf80cdef0 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfde00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf80cea00 con 0x7f2bf815e170 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfca00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2bf813cda0 0x7f2bf815da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55048/0 (socket says 172.21.15.141:55048) 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfca00640 1 -- 172.21.15.141:0/3528595217 learned_addr learned my addr 172.21.15.141:0/3528595217 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:36.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.494+0000 7f2bfd400640 1 --2- 172.21.15.141:0/3528595217 >> v2:172.21.15.141:3300/0 conn(0x7f2bf80cdef0 0x7f2bf8070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:36.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.495+0000 7f2bf7e00640 1 --2- 172.21.15.141:0/3528595217 >> v2:172.21.15.188:3300/0 conn(0x7f2bf815e170 0x7f2bf80cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:36.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.495+0000 7f2bf7400640 1 -- 172.21.15.141:0/3528595217 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866740792 0 0) 0x7f2bf80700f0 con 0x7f2bf813cda0 2024-01-16T15:23:36.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.495+0000 7f2bfd400640 1 -- 172.21.15.141:0/3528595217 >> v2:172.21.15.188:3300/0 conn(0x7f2bf815e170 msgr2=0x7f2bf80cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:36.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.495+0000 7f2bfd400640 1 --2- 172.21.15.141:0/3528595217 >> v2:172.21.15.188:3300/0 conn(0x7f2bf815e170 0x7f2bf80cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:36.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.495+0000 7f2bfd400640 1 -- 172.21.15.141:0/3528595217 >> v1:172.21.15.141:6789/0 conn(0x7f2bf813cda0 legacy=0x7f2bf815da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:36.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.495+0000 7f2bfd400640 1 -- 172.21.15.141:0/3528595217 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf8070340 con 0x7f2bf80cdef0 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.496+0000 7f2bfd400640 1 --2- 172.21.15.141:0/3528595217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80cdef0 0x7f2bf8070b40 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8157fd0 tx=0x7f2bf002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7985e9b8a750ff60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.496+0000 7f2bf7400640 1 -- 172.21.15.141:0/3528595217 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf002ee00 con 0x7f2bf80cdef0 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.496+0000 7f2bf7400640 1 -- 172.21.15.141:0/3528595217 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf0036440 con 0x7f2bf80cdef0 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.496+0000 7f2bf7400640 1 -- 172.21.15.141:0/3528595217 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf00367d0 con 0x7f2bf80cdef0 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 -- 172.21.15.141:0/3528595217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80cdef0 msgr2=0x7f2bf8070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 --2- 172.21.15.141:0/3528595217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80cdef0 0x7f2bf8070b40 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8157fd0 tx=0x7f2bf002e050 comp rx=0 tx=0).stop 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 -- 172.21.15.141:0/3528595217 shutdown_connections 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 --2- 172.21.15.141:0/3528595217 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80cdef0 0x7f2bf8070b40 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 --2- 172.21.15.141:0/3528595217 >> v2:172.21.15.188:3300/0 conn(0x7f2bf815e170 0x7f2bf80cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 -- 172.21.15.141:0/3528595217 >> 172.21.15.141:0/3528595217 conn(0x7f2bf806a3e0 msgr2=0x7f2bf806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 -- 172.21.15.141:0/3528595217 shutdown_connections 2024-01-16T15:23:36.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.497+0000 7f2bfde00640 1 -- 172.21.15.141:0/3528595217 wait complete. 2024-01-16T15:23:36.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.499+0000 7f2bfde00640 1 Processor -- start 2024-01-16T15:23:36.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.499+0000 7f2bfde00640 1 -- start start 2024-01-16T15:23:36.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfde00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bf80cdef0 0x7f2bf813aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:36.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf813cda0 0x7f2bf80b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:36.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfde00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bf815e170 0x7f2bf8151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:36.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfde00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8154bd0 con 0x7f2bf813cda0 2024-01-16T15:23:36.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfde00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8154eb0 con 0x7f2bf80cdef0 2024-01-16T15:23:36.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfde00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8155190 con 0x7f2bf815e170 2024-01-16T15:23:36.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bf7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf813cda0 0x7f2bf80b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bf7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf813cda0 0x7f2bf80b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38546/0 (socket says 172.21.15.141:38546) 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bf7e00640 1 -- 172.21.15.141:0/1236552915 learned_addr learned my addr 172.21.15.141:0/1236552915 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfca00640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bf80cdef0 0x7f2bf813aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bfd400640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bf815e170 0x7f2bf8151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bf7e00640 1 -- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bf815e170 msgr2=0x7f2bf8151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bf7e00640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bf815e170 0x7f2bf8151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bf7e00640 1 -- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bf80cdef0 msgr2=0x7f2bf813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.500+0000 7f2bf7e00640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bf80cdef0 0x7f2bf813aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bf7e00640 1 -- 172.21.15.141:0/1236552915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf0009a70 con 0x7f2bf813cda0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bfd400640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2bf815e170 0x7f2bf8151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bf7e00640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf813cda0 0x7f2bf80b0b30 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f2be000d800 tx=0x7f2be000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bf4c00640 1 -- 172.21.15.141:0/1236552915 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be000b6c0 con 0x7f2bf813cda0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bf4c00640 1 -- 172.21.15.141:0/1236552915 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be000b820 con 0x7f2bf813cda0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bf4c00640 1 -- 172.21.15.141:0/1236552915 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be000bbb0 con 0x7f2bf813cda0 2024-01-16T15:23:36.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bfde00640 1 -- 172.21.15.141:0/1236552915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf8155470 con 0x7f2bf813cda0 2024-01-16T15:23:36.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.501+0000 7f2bfde00640 1 -- 172.21.15.141:0/1236552915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf813afc0 con 0x7f2bf813cda0 2024-01-16T15:23:36.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.502+0000 7f2bfde00640 1 -- 172.21.15.141:0/1236552915 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf8067450 con 0x7f2bf813cda0 2024-01-16T15:23:36.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.503+0000 7f2bf4c00640 1 -- 172.21.15.141:0/1236552915 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f2be000bd10 con 0x7f2bf813cda0 2024-01-16T15:23:36.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.505+0000 7f2bf4c00640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bd006c5e0 0x7f2bd006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:36.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.505+0000 7f2bf4c00640 1 -- 172.21.15.141:0/1236552915 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2be0089180 con 0x7f2bf813cda0 2024-01-16T15:23:36.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.505+0000 7f2bfca00640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bd006c5e0 0x7f2bd006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:36.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.507+0000 7f2bfca00640 1 --2- 172.21.15.141:0/1236552915 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f2bd006c5e0 0x7f2bd006eaa0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f2be8000f60 tx=0x7f2be8007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:36.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.509+0000 7f2bf4c00640 1 -- 172.21.15.141:0/1236552915 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2be00556a0 con 0x7f2bf813cda0 2024-01-16T15:23:36.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.736+0000 7f2bfde00640 1 -- 172.21.15.141:0/1236552915 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2bf8155020 con 0x7f2bd006c5e0 2024-01-16T15:23:36.745 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:36.748+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:36.745 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:36.749+0000 7f2bf4c00640 1 -- 172.21.15.141:0/1236552915 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2bf8155020 con 0x7f2bd006c5e0 2024-01-16T15:23:36.745 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:36.745 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:38.764 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.993+0000 7f83a3200640 1 Processor -- start 2024-01-16T15:23:38.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.993+0000 7f83a3200640 1 -- start start 2024-01-16T15:23:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.994+0000 7f83a3200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f839c13cda0 0x7f839c15da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a3200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c0ce0c0 0x7f839c070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a3200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c0700f0 con 0x7f839c15dfa0 2024-01-16T15:23:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a3200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c0ce530 con 0x7f839c13cda0 2024-01-16T15:23:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a3200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c15e3e0 con 0x7f839c0ce0c0 2024-01-16T15:23:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a1400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f839c15dfa0 0x7f839c0cd9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55052/0 (socket says 172.21.15.141:55052) 2024-01-16T15:23:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a1400640 1 -- 172.21.15.141:0/2100290054 learned_addr learned my addr 172.21.15.141:0/2100290054 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219683842 0 0) 0x7f839c0700f0 con 0x7f839c15dfa0 2024-01-16T15:23:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f837c005730 con 0x7f839c15dfa0 2024-01-16T15:23:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a2800640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c0ce0c0 0x7f839c070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:38.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.995+0000 7f83a1e00640 1 --2- 172.21.15.141:0/2100290054 >> v2:172.21.15.188:3300/0 conn(0x7f839c13cda0 0x7f839c15da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562845585 0 0) 0x7f837c005730 con 0x7f839c15dfa0 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c0ce0c0 msgr2=0x7f839c070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c0ce0c0 0x7f839c070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 >> v2:172.21.15.188:3300/0 conn(0x7f839c13cda0 msgr2=0x7f839c15da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 --2- 172.21.15.141:0/2100290054 >> v2:172.21.15.188:3300/0 conn(0x7f839c13cda0 0x7f839c15da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0ce720 con 0x7f839c15dfa0 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8384001670 con 0x7f839c15dfa0 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 >> v1:172.21.15.141:6789/0 conn(0x7f839c15dfa0 legacy=0x7f839c0cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c0ce0c0 0x7f837c005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f837c006150 0x7f837c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006f10 0x7f837c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c15e3e0 con 0x7f837c006150 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c00dbc0 con 0x7f837c006f10 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837c00dd80 con 0x7f839c0ce0c0 2024-01-16T15:23:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.996+0000 7f83a1400640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f837c006150 0x7f837c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.997+0000 7f83a1400640 1 -- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c0ce0c0 msgr2=0x7f837c005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.997+0000 7f83a1400640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c0ce0c0 0x7f837c005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.997+0000 7f83a1400640 1 -- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006f10 msgr2=0x7f837c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.997+0000 7f83a2800640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006f10 0x7f837c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.997+0000 7f83a1400640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006f10 0x7f837c007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.997+0000 7f83a1400640 1 -- 172.21.15.141:0/2100290054 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c005730 con 0x7f837c006150 2024-01-16T15:23:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.998+0000 7f83a1400640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f837c006150 0x7f837c0069d0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f838400c160 tx=0x7f837c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16eae210d1946513 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:38.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.998+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384005ad0 con 0x7f837c006150 2024-01-16T15:23:38.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.998+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8384005c30 con 0x7f837c006150 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.998+0000 7f83a0a00640 1 -- 172.21.15.141:0/2100290054 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838402c810 con 0x7f837c006150 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.998+0000 7f83a3200640 1 -- 172.21.15.141:0/2100290054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f837c006150 msgr2=0x7f837c0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.998+0000 7f83a3200640 1 --2- 172.21.15.141:0/2100290054 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f837c006150 0x7f837c0069d0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f838400c160 tx=0x7f837c007a00 comp rx=0 tx=0).stop 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.998+0000 7f83a2800640 1 -- 172.21.15.141:0/2100290054 reap_dead start 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.999+0000 7f83a3200640 1 -- 172.21.15.141:0/2100290054 shutdown_connections 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.999+0000 7f83a3200640 1 -- 172.21.15.141:0/2100290054 >> 172.21.15.141:0/2100290054 conn(0x7f839c06a3e0 msgr2=0x7f839c06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.999+0000 7f83a3200640 1 -- 172.21.15.141:0/2100290054 shutdown_connections 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:38.999+0000 7f83a3200640 1 -- 172.21.15.141:0/2100290054 wait complete. 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.000+0000 7f83a3200640 1 Processor -- start 2024-01-16T15:23:38.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.001+0000 7f83a3200640 1 -- start start 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.001+0000 7f83a3200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006150 0x7f839c13c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.001+0000 7f83a3200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c13c690 0x7f839c0b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a3200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f839c13d580 0x7f839c0b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a3200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c153500 con 0x7f839c13d580 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a3200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0ae8a0 con 0x7f837c006150 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a3200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c152bd0 con 0x7f839c13c690 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f839c13d580 0x7f839c0b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f839c13d580 0x7f839c0b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38562/0 (socket says 172.21.15.141:38562) 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 -- 172.21.15.141:0/3102003828 learned_addr learned my addr 172.21.15.141:0/3102003828 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a1400640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c13c690 0x7f839c0b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 -- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c13c690 msgr2=0x7f839c0b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f839c13c690 0x7f839c0b2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 -- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006150 msgr2=0x7f839c13c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a1e00640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006150 0x7f839c13c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f837c006150 0x7f839c13c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.002+0000 7f83a2800640 1 -- 172.21.15.141:0/3102003828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c152eb0 con 0x7f839c13d580 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.003+0000 7f83a2800640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f839c13d580 0x7f839c0b3210 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f839c0b0b00 tx=0x7f839400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.003+0000 7f839a000640 1 -- 172.21.15.141:0/3102003828 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839400fe90 con 0x7f839c13d580 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.003+0000 7f839a000640 1 -- 172.21.15.141:0/3102003828 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8394011040 con 0x7f839c13d580 2024-01-16T15:23:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.003+0000 7f839a000640 1 -- 172.21.15.141:0/3102003828 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394003bd0 con 0x7f839c13d580 2024-01-16T15:23:39.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.003+0000 7f83a3200640 1 -- 172.21.15.141:0/3102003828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c153190 con 0x7f839c13d580 2024-01-16T15:23:39.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.003+0000 7f83a3200640 1 -- 172.21.15.141:0/3102003828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c13da00 con 0x7f839c13d580 2024-01-16T15:23:39.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.004+0000 7f83a3200640 1 -- 172.21.15.141:0/3102003828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c067450 con 0x7f839c13d580 2024-01-16T15:23:39.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.006+0000 7f839a000640 1 -- 172.21.15.141:0/3102003828 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f8394017400 con 0x7f839c13d580 2024-01-16T15:23:39.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.006+0000 7f839a000640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f837406c5e0 0x7f837406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:39.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.007+0000 7f839a000640 1 -- 172.21.15.141:0/3102003828 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8394088fc0 con 0x7f839c13d580 2024-01-16T15:23:39.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.007+0000 7f83a1e00640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f837406c5e0 0x7f837406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:39.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.008+0000 7f83a1e00640 1 --2- 172.21.15.141:0/3102003828 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f837406c5e0 0x7f837406eaa0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f838c004770 tx=0x7f838c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:39.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.011+0000 7f839a000640 1 -- 172.21.15.141:0/3102003828 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83940554e0 con 0x7f839c13d580 2024-01-16T15:23:39.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.235+0000 7f83a3200640 1 -- 172.21.15.141:0/3102003828 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f839c152d40 con 0x7f837406c5e0 2024-01-16T15:23:39.244 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:39.247+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:39.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:39.248+0000 7f839a000640 1 -- 172.21.15.141:0/3102003828 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f839c152d40 con 0x7f837406c5e0 2024-01-16T15:23:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:39.245 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:40.025 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:40.033 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:41.260 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:41.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.488+0000 7f2a0d400640 1 Processor -- start 2024-01-16T15:23:41.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.488+0000 7f2a0d400640 1 -- start start 2024-01-16T15:23:41.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.490+0000 7f2a0d400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f2a080ae5a0 0x7f2a0815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:41.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.490+0000 7f2a0d400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2a080b3ae0 0x7f2a0806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:41.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.490+0000 7f2a0d400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a08070c40 con 0x7f2a0806d120 2024-01-16T15:23:41.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.490+0000 7f2a0d400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a08071450 con 0x7f2a080b3ae0 2024-01-16T15:23:41.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.490+0000 7f2a0d400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0806c470 con 0x7f2a080ae5a0 2024-01-16T15:23:41.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.490+0000 7f2a07200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f2a0806d120 0x7f2a080cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:55066/0 (socket says 172.21.15.141:55066) 2024-01-16T15:23:41.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a07200640 1 -- 172.21.15.141:0/894059289 learned_addr learned my addr 172.21.15.141:0/894059289 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a05e00640 1 --2- 172.21.15.141:0/894059289 >> v2:172.21.15.141:3300/0 conn(0x7f2a080b3ae0 0x7f2a0806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a06800640 1 --2- 172.21.15.141:0/894059289 >> v2:172.21.15.188:3300/0 conn(0x7f2a080ae5a0 0x7f2a0815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a05400640 1 -- 172.21.15.141:0/894059289 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709834504 0 0) 0x7f2a08070c40 con 0x7f2a0806d120 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a05400640 1 -- 172.21.15.141:0/894059289 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003520 con 0x7f2a0806d120 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a05e00640 1 -- 172.21.15.141:0/894059289 >> v2:172.21.15.188:3300/0 conn(0x7f2a080ae5a0 msgr2=0x7f2a0815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a05e00640 1 --2- 172.21.15.141:0/894059289 >> v2:172.21.15.188:3300/0 conn(0x7f2a080ae5a0 0x7f2a0815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a05e00640 1 -- 172.21.15.141:0/894059289 >> v1:172.21.15.141:6789/0 conn(0x7f2a0806d120 legacy=0x7f2a080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:41.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.491+0000 7f2a05e00640 1 -- 172.21.15.141:0/894059289 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0806c750 con 0x7f2a080b3ae0 2024-01-16T15:23:41.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.493+0000 7f2a05e00640 1 --2- 172.21.15.141:0/894059289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a080b3ae0 0x7f2a0806cbe0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f2a080cfe20 tx=0x7f29f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38c5b86196bc247 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:41.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.493+0000 7f2a05400640 1 -- 172.21.15.141:0/894059289 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0031830 con 0x7f2a080b3ae0 2024-01-16T15:23:41.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.493+0000 7f2a05400640 1 -- 172.21.15.141:0/894059289 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0031990 con 0x7f2a080b3ae0 2024-01-16T15:23:41.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.493+0000 7f2a05400640 1 -- 172.21.15.141:0/894059289 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0031d00 con 0x7f2a080b3ae0 2024-01-16T15:23:41.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 -- 172.21.15.141:0/894059289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a080b3ae0 msgr2=0x7f2a0806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:41.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 --2- 172.21.15.141:0/894059289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a080b3ae0 0x7f2a0806cbe0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f2a080cfe20 tx=0x7f29f002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:41.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 -- 172.21.15.141:0/894059289 shutdown_connections 2024-01-16T15:23:41.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 --2- 172.21.15.141:0/894059289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a080b3ae0 0x7f2a0806cbe0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:41.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 --2- 172.21.15.141:0/894059289 >> v2:172.21.15.188:3300/0 conn(0x7f2a080ae5a0 0x7f2a0815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:41.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 -- 172.21.15.141:0/894059289 >> 172.21.15.141:0/894059289 conn(0x7f2a0806a800 msgr2=0x7f2a0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:41.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 -- 172.21.15.141:0/894059289 shutdown_connections 2024-01-16T15:23:41.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.494+0000 7f2a0d400640 1 -- 172.21.15.141:0/894059289 wait complete. 2024-01-16T15:23:41.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.496+0000 7f2a0d400640 1 Processor -- start 2024-01-16T15:23:41.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.496+0000 7f2a0d400640 1 -- start start 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0806d120 0x7f2a08158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a080ae5a0 0x7f2a0815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a080b3ae0 0x7f2a0815f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a0d400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a081f70f0 con 0x7f2a0806d120 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a0d400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a081f2fe0 con 0x7f2a080ae5a0 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a0d400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a081f32c0 con 0x7f2a080b3ae0 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a06800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0806d120 0x7f2a08158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a06800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0806d120 0x7f2a08158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:38580/0 (socket says 172.21.15.141:38580) 2024-01-16T15:23:41.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.497+0000 7f2a06800640 1 -- 172.21.15.141:0/747927152 learned_addr learned my addr 172.21.15.141:0/747927152 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a07200640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a080b3ae0 0x7f2a0815f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a05e00640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a080ae5a0 0x7f2a0815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a06800640 1 -- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a080b3ae0 msgr2=0x7f2a0815f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a06800640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f2a080b3ae0 0x7f2a0815f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a06800640 1 -- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a080ae5a0 msgr2=0x7f2a0815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a06800640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a080ae5a0 0x7f2a0815f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a06800640 1 -- 172.21.15.141:0/747927152 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f0009400 con 0x7f2a0806d120 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a05e00640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a080ae5a0 0x7f2a0815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:41.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f2a06800640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0806d120 0x7f2a08158d30 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e9b0 tx=0x7f29f800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:41.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.498+0000 7f29fea00640 1 -- 172.21.15.141:0/747927152 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8009670 con 0x7f2a0806d120 2024-01-16T15:23:41.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.499+0000 7f29fea00640 1 -- 172.21.15.141:0/747927152 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f800c9a0 con 0x7f2a0806d120 2024-01-16T15:23:41.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.499+0000 7f2a0d400640 1 -- 172.21.15.141:0/747927152 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a081f35a0 con 0x7f2a0806d120 2024-01-16T15:23:41.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.499+0000 7f29fea00640 1 -- 172.21.15.141:0/747927152 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800cd50 con 0x7f2a0806d120 2024-01-16T15:23:41.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.499+0000 7f2a0d400640 1 -- 172.21.15.141:0/747927152 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a081f2cd0 con 0x7f2a0806d120 2024-01-16T15:23:41.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.500+0000 7f2a0d400640 1 -- 172.21.15.141:0/747927152 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a081f3730 con 0x7f2a0806d120 2024-01-16T15:23:41.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.501+0000 7f29fea00640 1 -- 172.21.15.141:0/747927152 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f29f8013470 con 0x7f2a0806d120 2024-01-16T15:23:41.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.502+0000 7f29fea00640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29e006c5e0 0x7f29e006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:41.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.502+0000 7f29fea00640 1 -- 172.21.15.141:0/747927152 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f29f8088e90 con 0x7f2a0806d120 2024-01-16T15:23:41.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.502+0000 7f2a05e00640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29e006c5e0 0x7f29e006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:41.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.504+0000 7f2a05e00640 1 --2- 172.21.15.141:0/747927152 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f29e006c5e0 0x7f29e006eaa0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f29f0036000 tx=0x7f29f003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:41.502 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.506+0000 7f29fea00640 1 -- 172.21.15.141:0/747927152 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29f80553b0 con 0x7f2a0806d120 2024-01-16T15:23:41.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.734+0000 7f2a0d400640 1 -- 172.21.15.141:0/747927152 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a08070550 con 0x7f29e006c5e0 2024-01-16T15:23:41.743 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:41.747+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:41.744 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:41.748+0000 7f29fea00640 1 -- 172.21.15.141:0/747927152 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f29f003a420 con 0x7f29e006c5e0 2024-01-16T15:23:41.744 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:41.745 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:43.766 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:43.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:43.999+0000 7ff487400640 1 Processor -- start 2024-01-16T15:23:43.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:43.999+0000 7ff487400640 1 -- start start 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff487400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff4800cf9a0 0x7ff480070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff487400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7ff480070ea0 0x7ff4800cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff487400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff480071690 con 0x7ff48014b2c0 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff487400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff48014b6a0 con 0x7ff4800cf9a0 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff487400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff480071970 con 0x7ff480070ea0 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff484a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff48014b2c0 0x7ff4800cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47248/0 (socket says 172.21.15.141:47248) 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff484a00640 1 -- 172.21.15.141:0/3372946688 learned_addr learned my addr 172.21.15.141:0/3372946688 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:43.997 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.001+0000 7ff477e00640 1 --2- 172.21.15.141:0/3372946688 >> v2:172.21.15.141:3300/0 conn(0x7ff4800cf9a0 0x7ff480070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:43.998 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff47fe00640 1 -- 172.21.15.141:0/3372946688 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402561164 0 0) 0x7ff480071690 con 0x7ff48014b2c0 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff485400640 1 --2- 172.21.15.141:0/3372946688 >> v2:172.21.15.188:3300/0 conn(0x7ff480070ea0 0x7ff4800cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff47fe00640 1 -- 172.21.15.141:0/3372946688 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff460003520 con 0x7ff48014b2c0 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff477e00640 1 -- 172.21.15.141:0/3372946688 >> v2:172.21.15.188:3300/0 conn(0x7ff480070ea0 msgr2=0x7ff4800cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff477e00640 1 --2- 172.21.15.141:0/3372946688 >> v2:172.21.15.188:3300/0 conn(0x7ff480070ea0 0x7ff4800cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff477e00640 1 -- 172.21.15.141:0/3372946688 >> v1:172.21.15.141:6789/0 conn(0x7ff48014b2c0 legacy=0x7ff4800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff477e00640 1 -- 172.21.15.141:0/3372946688 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff480071c50 con 0x7ff4800cf9a0 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.002+0000 7ff485400640 1 --2- 172.21.15.141:0/3372946688 >> v2:172.21.15.188:3300/0 conn(0x7ff480070ea0 0x7ff4800cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:43.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.003+0000 7ff477e00640 1 --2- 172.21.15.141:0/3372946688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4800cf9a0 0x7ff480070960 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7ff480165330 tx=0x7ff46802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd584bdbea5a5101 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.003+0000 7ff47fe00640 1 -- 172.21.15.141:0/3372946688 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468031830 con 0x7ff4800cf9a0 2024-01-16T15:23:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.003+0000 7ff47fe00640 1 -- 172.21.15.141:0/3372946688 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff468031990 con 0x7ff4800cf9a0 2024-01-16T15:23:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.003+0000 7ff47fe00640 1 -- 172.21.15.141:0/3372946688 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468031d20 con 0x7ff4800cf9a0 2024-01-16T15:23:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 -- 172.21.15.141:0/3372946688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4800cf9a0 msgr2=0x7ff480070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 --2- 172.21.15.141:0/3372946688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4800cf9a0 0x7ff480070960 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7ff480165330 tx=0x7ff46802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.000 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 -- 172.21.15.141:0/3372946688 shutdown_connections 2024-01-16T15:23:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 --2- 172.21.15.141:0/3372946688 >> v2:172.21.15.188:3300/0 conn(0x7ff480070ea0 0x7ff4800cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 --2- 172.21.15.141:0/3372946688 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4800cf9a0 0x7ff480070960 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 -- 172.21.15.141:0/3372946688 >> 172.21.15.141:0/3372946688 conn(0x7ff48006a800 msgr2=0x7ff48006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 -- 172.21.15.141:0/3372946688 shutdown_connections 2024-01-16T15:23:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.004+0000 7ff487400640 1 -- 172.21.15.141:0/3372946688 wait complete. 2024-01-16T15:23:44.001 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.006+0000 7ff487400640 1 Processor -- start 2024-01-16T15:23:44.002 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.006+0000 7ff487400640 1 -- start start 2024-01-16T15:23:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff487400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff480070ea0 0x7ff480157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff487400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4800cf9a0 0x7ff480142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff487400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff48014b2c0 0x7ff480147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff487400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff480164750 con 0x7ff48014b2c0 2024-01-16T15:23:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff487400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4801576c0 con 0x7ff4800cf9a0 2024-01-16T15:23:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff487400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff480163430 con 0x7ff480070ea0 2024-01-16T15:23:44.003 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff485400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff48014b2c0 0x7ff480147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff485400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff48014b2c0 0x7ff480147930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46224/0 (socket says 172.21.15.141:46224) 2024-01-16T15:23:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff485400640 1 -- 172.21.15.141:0/761144877 learned_addr learned my addr 172.21.15.141:0/761144877 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff477e00640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4800cf9a0 0x7ff480142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.004 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.007+0000 7ff484a00640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff480070ea0 0x7ff480157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff485400640 1 -- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff480070ea0 msgr2=0x7ff480157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff485400640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff480070ea0 0x7ff480157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff485400640 1 -- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4800cf9a0 msgr2=0x7ff480142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff485400640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4800cf9a0 0x7ff480142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff485400640 1 -- 172.21.15.141:0/761144877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff468009400 con 0x7ff48014b2c0 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff485400640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff48014b2c0 0x7ff480147930 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7ff47800cef0 tx=0x7ff47800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff47d600640 1 -- 172.21.15.141:0/761144877 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47800d840 con 0x7ff48014b2c0 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff47d600640 1 -- 172.21.15.141:0/761144877 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47800d9c0 con 0x7ff48014b2c0 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff47d600640 1 -- 172.21.15.141:0/761144877 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47800dd70 con 0x7ff48014b2c0 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff487400640 1 -- 172.21.15.141:0/761144877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff480164be0 con 0x7ff48014b2c0 2024-01-16T15:23:44.005 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.008+0000 7ff487400640 1 -- 172.21.15.141:0/761144877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff480164100 con 0x7ff48014b2c0 2024-01-16T15:23:44.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.009+0000 7ff487400640 1 -- 172.21.15.141:0/761144877 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4801648c0 con 0x7ff48014b2c0 2024-01-16T15:23:44.006 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.011+0000 7ff47d600640 1 -- 172.21.15.141:0/761144877 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ff47801e3b0 con 0x7ff48014b2c0 2024-01-16T15:23:44.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.011+0000 7ff47d600640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45806c5e0 0x7ff45806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.007 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.012+0000 7ff47d600640 1 -- 172.21.15.141:0/761144877 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff4780896c0 con 0x7ff48014b2c0 2024-01-16T15:23:44.008 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.012+0000 7ff484a00640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45806c5e0 0x7ff45806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.009 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.013+0000 7ff484a00640 1 --2- 172.21.15.141:0/761144877 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff45806c5e0 0x7ff45806eaa0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff4700011e0 tx=0x7ff470009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:44.012 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.016+0000 7ff47d600640 1 -- 172.21.15.141:0/761144877 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff478055ca0 con 0x7ff48014b2c0 2024-01-16T15:23:44.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.236+0000 7ff487400640 1 -- 172.21.15.141:0/761144877 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4800cbc40 con 0x7ff45806c5e0 2024-01-16T15:23:44.246 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:23:44.249+0000 7f0f2bc00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:23:44.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.250+0000 7ff47d600640 1 -- 172.21.15.141:0/761144877 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff470001330 con 0x7ff45806c5e0 2024-01-16T15:23:44.247 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:44.248 INFO:teuthology.orchestra.run.smithi141.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:23:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:23:44.264 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for ... ERROR 2024-01-16T15:23:44.264 INFO:tasks.cephfs_test_runner:ERROR 2024-01-16T15:23:44.264 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:44.264 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:23:44.265 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi141 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7006' 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:23:44.266 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:Ran 7 tests in 1558.149s 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:23:44.267 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi141 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone7006' 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:44.268 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:23:44.269 INFO:tasks.cephfs_test_runner: 2024-01-16T15:23:44.269 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:44.470 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ae1df0ec7aea4dacbb81a4e3a366462a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:23:44.472 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-16T15:23:44.488 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-16T15:23:44.507 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-16T15:23:44.507 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:23:44.508 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:44.567 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:23:44.567 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:44.620 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:44.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.838+0000 7f56bfe00640 1 Processor -- start 2024-01-16T15:23:44.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.838+0000 7f56bfe00640 1 -- start start 2024-01-16T15:23:44.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.839+0000 7f56bfe00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f56b80b3ae0 0x7f56b815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.839+0000 7f56bfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f56b80cf290 0x7f56b80cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.839+0000 7f56bfe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b8070c40 con 0x7f56b80ae5a0 2024-01-16T15:23:44.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.839+0000 7f56bfe00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b8071450 con 0x7f56b80cf290 2024-01-16T15:23:44.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.839+0000 7f56bfe00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b80cfc60 con 0x7f56b80b3ae0 2024-01-16T15:23:44.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bde00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f56b80cf290 0x7f56b80cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bd400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f56b80ae5a0 0x7f56b806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47256/0 (socket says 172.21.15.141:47256) 2024-01-16T15:23:44.837 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bd400640 1 -- 172.21.15.141:0/2880026741 learned_addr learned my addr 172.21.15.141:0/2880026741 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:44.838 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bca00640 1 --2- 172.21.15.141:0/2880026741 >> v2:172.21.15.188:3300/0 conn(0x7f56b80b3ae0 0x7f56b815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bde00640 1 -- 172.21.15.141:0/2880026741 >> v2:172.21.15.188:3300/0 conn(0x7f56b80b3ae0 msgr2=0x7f56b815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:44.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bde00640 1 --2- 172.21.15.141:0/2880026741 >> v2:172.21.15.188:3300/0 conn(0x7f56b80b3ae0 0x7f56b815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bde00640 1 -- 172.21.15.141:0/2880026741 >> v1:172.21.15.141:6789/0 conn(0x7f56b80ae5a0 legacy=0x7f56b806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:44.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56b7e00640 1 -- 172.21.15.141:0/2880026741 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087244975 0 0) 0x7f56b8070c40 con 0x7f56b80ae5a0 2024-01-16T15:23:44.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.840+0000 7f56bde00640 1 -- 172.21.15.141:0/2880026741 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b806c470 con 0x7f56b80cf290 2024-01-16T15:23:44.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56bde00640 1 --2- 172.21.15.141:0/2880026741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80cf290 0x7f56b80cf720 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f56b80cfea0 tx=0x7f56b002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a425a3ccb6339e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:44.839 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56b7e00640 1 -- 172.21.15.141:0/2880026741 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b002cd00 con 0x7f56b80cf290 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56b7e00640 1 -- 172.21.15.141:0/2880026741 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b002ce60 con 0x7f56b80cf290 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56b7e00640 1 -- 172.21.15.141:0/2880026741 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0034660 con 0x7f56b80cf290 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56bfe00640 1 -- 172.21.15.141:0/2880026741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80cf290 msgr2=0x7f56b80cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56bfe00640 1 --2- 172.21.15.141:0/2880026741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80cf290 0x7f56b80cf720 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f56b80cfea0 tx=0x7f56b002bf50 comp rx=0 tx=0).stop 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56bfe00640 1 -- 172.21.15.141:0/2880026741 shutdown_connections 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56bfe00640 1 --2- 172.21.15.141:0/2880026741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80cf290 0x7f56b80cf720 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.842+0000 7f56bfe00640 1 --2- 172.21.15.141:0/2880026741 >> v2:172.21.15.188:3300/0 conn(0x7f56b80b3ae0 0x7f56b815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.840 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.843+0000 7f56bfe00640 1 -- 172.21.15.141:0/2880026741 >> 172.21.15.141:0/2880026741 conn(0x7f56b806a800 msgr2=0x7f56b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:44.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.843+0000 7f56bfe00640 1 -- 172.21.15.141:0/2880026741 shutdown_connections 2024-01-16T15:23:44.841 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.843+0000 7f56bfe00640 1 -- 172.21.15.141:0/2880026741 wait complete. 2024-01-16T15:23:44.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.844+0000 7f56bfe00640 1 Processor -- start 2024-01-16T15:23:44.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.844+0000 7f56bfe00640 1 -- start start 2024-01-16T15:23:44.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.845+0000 7f56bfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56b80ae5a0 0x7f56b81492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.845+0000 7f56bfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80b3ae0 0x7f56b81523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.845+0000 7f56bfe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56b80cf290 0x7f56b80afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.843 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.845+0000 7f56bfe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b815d2f0 con 0x7f56b80b3ae0 2024-01-16T15:23:44.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.845+0000 7f56bfe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b81f2d60 con 0x7f56b80ae5a0 2024-01-16T15:23:44.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.845+0000 7f56bfe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8150fb0 con 0x7f56b80cf290 2024-01-16T15:23:44.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80b3ae0 0x7f56b81523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80b3ae0 0x7f56b81523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46246/0 (socket says 172.21.15.141:46246) 2024-01-16T15:23:44.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 -- 172.21.15.141:0/589224971 learned_addr learned my addr 172.21.15.141:0/589224971 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:44.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bd400640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56b80ae5a0 0x7f56b81492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.844 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 -- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56b80cf290 msgr2=0x7f56b80afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56b80cf290 0x7f56b80afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 -- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56b80ae5a0 msgr2=0x7f56b81492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56b80ae5a0 0x7f56b81492a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 -- 172.21.15.141:0/589224971 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b0009a70 con 0x7f56b80b3ae0 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.846+0000 7f56bca00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80b3ae0 0x7f56b81523a0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f56a000d800 tx=0x7f56a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.847+0000 7f56b5600640 1 -- 172.21.15.141:0/589224971 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a000b6c0 con 0x7f56b80b3ae0 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.847+0000 7f56b5600640 1 -- 172.21.15.141:0/589224971 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a000b820 con 0x7f56b80b3ae0 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.847+0000 7f56b5600640 1 -- 172.21.15.141:0/589224971 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a000bbb0 con 0x7f56b80b3ae0 2024-01-16T15:23:44.845 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.847+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b81f6ca0 con 0x7f56b80b3ae0 2024-01-16T15:23:44.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.847+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b8149840 con 0x7f56b80b3ae0 2024-01-16T15:23:44.846 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.848+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b81f6e30 con 0x7f56b80b3ae0 2024-01-16T15:23:44.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.849+0000 7f56b5600640 1 -- 172.21.15.141:0/589224971 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f56a000bd10 con 0x7f56b80b3ae0 2024-01-16T15:23:44.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.850+0000 7f56b5600640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f569006c610 0x7f569006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:44.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.850+0000 7f56b5600640 1 -- 172.21.15.141:0/589224971 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f56a0089060 con 0x7f56b80b3ae0 2024-01-16T15:23:44.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.850+0000 7f56bd400640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f569006c610 0x7f569006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:44.847 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.851+0000 7f56bd400640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f569006c610 0x7f569006ead0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f56a8007fd0 tx=0x7f56a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:44.849 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:44.854+0000 7f56b5600640 1 -- 172.21.15.141:0/589224971 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56a00555e0 con 0x7f56b80b3ae0 2024-01-16T15:23:45.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.046+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56b80675d0 con 0x7f56b80b3ae0 2024-01-16T15:23:45.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.048+0000 7f56b5600640 1 -- 172.21.15.141:0/589224971 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16668 (secure 0 0 0) 0x7f56a0058c00 con 0x7f56b80b3ae0 2024-01-16T15:23:45.045 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:45.045 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":118,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:55.853050+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.4","mappings":[{"from":0,"to":5}]},{"pgid":"17.3c","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f569006c610 msgr2=0x7f569006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f569006c610 0x7f569006ead0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f56a8007fd0 tx=0x7f56a8009040 comp rx=0 tx=0).stop 2024-01-16T15:23:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80b3ae0 msgr2=0x7f56b81523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80b3ae0 0x7f56b81523a0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f56a000d800 tx=0x7f56a000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.047 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 shutdown_connections 2024-01-16T15:23:45.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f569006c610 0x7f569006ead0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f56b80cf290 0x7f56b80afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f56b80b3ae0 0x7f56b81523a0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 --2- 172.21.15.141:0/589224971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56b80ae5a0 0x7f56b81492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 >> 172.21.15.141:0/589224971 conn(0x7f56b806a800 msgr2=0x7f56b80b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:45.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 shutdown_connections 2024-01-16T15:23:45.048 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.051+0000 7f56bfe00640 1 -- 172.21.15.141:0/589224971 wait complete. 2024-01-16T15:23:45.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:36.311732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:37.759492+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:45.066 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:45.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.283+0000 7fd809c00640 1 Processor -- start 2024-01-16T15:23:45.280 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.283+0000 7fd809c00640 1 -- start start 2024-01-16T15:23:45.281 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.285+0000 7fd809c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413df80 0x7fd80415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.285+0000 7fd809c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd80413e360 0x7fd8040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.285+0000 7fd809c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80406c470 con 0x7fd80415f9c0 2024-01-16T15:23:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.285+0000 7fd809c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd80406c750 con 0x7fd80413e360 2024-01-16T15:23:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.285+0000 7fd809c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80406ca30 con 0x7fd80413df80 2024-01-16T15:23:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd7fbe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd80413e360 0x7fd8040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd803400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd80415f9c0 0x7fd8041408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47270/0 (socket says 172.21.15.141:47270) 2024-01-16T15:23:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd803400640 1 -- 172.21.15.141:0/492243575 learned_addr learned my addr 172.21.15.141:0/492243575 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:45.283 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd802a00640 1 --2- 172.21.15.141:0/492243575 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413df80 0x7fd80415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd7fbe00640 1 -- 172.21.15.141:0/492243575 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413df80 msgr2=0x7fd80415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd7fbe00640 1 --2- 172.21.15.141:0/492243575 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413df80 0x7fd80415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd7fbe00640 1 -- 172.21.15.141:0/492243575 >> v1:172.21.15.141:6789/0 conn(0x7fd80415f9c0 legacy=0x7fd8041408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.286+0000 7fd7fbe00640 1 -- 172.21.15.141:0/492243575 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80406cd10 con 0x7fd80413e360 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.287+0000 7fd7fbe00640 1 --2- 172.21.15.141:0/492243575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80413e360 0x7fd8040cf290 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0097a0 tx=0x7fd7ec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b924bf6f5348e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.287+0000 7fd802000640 1 -- 172.21.15.141:0/492243575 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec02d040 con 0x7fd80413e360 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd802000640 1 -- 172.21.15.141:0/492243575 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ec031810 con 0x7fd80413e360 2024-01-16T15:23:45.284 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd802000640 1 -- 172.21.15.141:0/492243575 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec031ba0 con 0x7fd80413e360 2024-01-16T15:23:45.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 -- 172.21.15.141:0/492243575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80413e360 msgr2=0x7fd8040cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.285 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 --2- 172.21.15.141:0/492243575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80413e360 0x7fd8040cf290 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0097a0 tx=0x7fd7ec02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 -- 172.21.15.141:0/492243575 shutdown_connections 2024-01-16T15:23:45.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 --2- 172.21.15.141:0/492243575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80413e360 0x7fd8040cf290 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 --2- 172.21.15.141:0/492243575 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413df80 0x7fd80415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 -- 172.21.15.141:0/492243575 >> 172.21.15.141:0/492243575 conn(0x7fd80406a800 msgr2=0x7fd80406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:45.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 -- 172.21.15.141:0/492243575 shutdown_connections 2024-01-16T15:23:45.286 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.288+0000 7fd809c00640 1 -- 172.21.15.141:0/492243575 wait complete. 2024-01-16T15:23:45.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.290+0000 7fd809c00640 1 Processor -- start 2024-01-16T15:23:45.287 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.290+0000 7fd809c00640 1 -- start start 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.291+0000 7fd809c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd80413df80 0x7fd804132180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.291+0000 7fd809c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413e360 0x7fd80414d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.291+0000 7fd809c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80415f9c0 0x7fd80414d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.291+0000 7fd809c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd804150050 con 0x7fd80415f9c0 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.291+0000 7fd809c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd804150330 con 0x7fd80413df80 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.291+0000 7fd809c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80415c9f0 con 0x7fd80413e360 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.291+0000 7fd803400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80415f9c0 0x7fd80414d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80415f9c0 0x7fd80414d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46270/0 (socket says 172.21.15.141:46270) 2024-01-16T15:23:45.289 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 -- 172.21.15.141:0/3286201420 learned_addr learned my addr 172.21.15.141:0/3286201420 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:45.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd7fbe00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413e360 0x7fd80414d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd802a00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd80413df80 0x7fd804132180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 -- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413e360 msgr2=0x7fd80414d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413e360 0x7fd80414d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.290 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 -- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd80413df80 msgr2=0x7fd804132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd80413df80 0x7fd804132180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 -- 172.21.15.141:0/3286201420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec009400 con 0x7fd80415f9c0 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd803400640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80415f9c0 0x7fd80414d700 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00cbc0 tx=0x7fd7fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.292+0000 7fd7faa00640 1 -- 172.21.15.141:0/3286201420 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00ad40 con 0x7fd80415f9c0 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.293+0000 7fd7faa00640 1 -- 172.21.15.141:0/3286201420 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc012400 con 0x7fd80415f9c0 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.293+0000 7fd7faa00640 1 -- 172.21.15.141:0/3286201420 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00aea0 con 0x7fd80415f9c0 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.293+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80415ccd0 con 0x7fd80415f9c0 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.293+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80414f290 con 0x7fd80415f9c0 2024-01-16T15:23:45.291 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.294+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80415c850 con 0x7fd80415f9c0 2024-01-16T15:23:45.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.295+0000 7fd7faa00640 1 -- 172.21.15.141:0/3286201420 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd7fc007070 con 0x7fd80415f9c0 2024-01-16T15:23:45.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.296+0000 7fd7faa00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd7dc06c5e0 0x7fd7dc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.292 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.296+0000 7fd7faa00640 1 -- 172.21.15.141:0/3286201420 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7fc088a60 con 0x7fd80415f9c0 2024-01-16T15:23:45.293 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.297+0000 7fd802a00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd7dc06c5e0 0x7fd7dc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.298+0000 7fd802a00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd7dc06c5e0 0x7fd7dc06eaa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4006fd0 tx=0x7fd7f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:45.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.301+0000 7fd7faa00640 1 -- 172.21.15.141:0/3286201420 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd80415c850 con 0x7fd80415f9c0 2024-01-16T15:23:45.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.489+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd80415c850 con 0x7fd80415f9c0 2024-01-16T15:23:45.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.490+0000 7fd7faa00640 1 -- 172.21.15.141:0/3286201420 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd80415c850 con 0x7fd80415f9c0 2024-01-16T15:23:45.487 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:23:45.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd7dc06c5e0 msgr2=0x7fd7dc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd7dc06c5e0 0x7fd7dc06eaa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4006fd0 tx=0x7fd7f4008040 comp rx=0 tx=0).stop 2024-01-16T15:23:45.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80415f9c0 msgr2=0x7fd80414d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80415f9c0 0x7fd80414d700 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00cbc0 tx=0x7fd7fc00a430 comp rx=0 tx=0).stop 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 shutdown_connections 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd7dc06c5e0 0x7fd7dc06eaa0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd80415f9c0 0x7fd80414d700 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd80413e360 0x7fd80414d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 --2- 172.21.15.141:0/3286201420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd80413df80 0x7fd804132180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 >> 172.21.15.141:0/3286201420 conn(0x7fd80406a800 msgr2=0x7fd8040b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 shutdown_connections 2024-01-16T15:23:45.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.493+0000 7fd809c00640 1 -- 172.21.15.141:0/3286201420 wait complete. 2024-01-16T15:23:45.509 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:45.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.723+0000 7fd2b2800640 1 Processor -- start 2024-01-16T15:23:45.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.724+0000 7fd2b2800640 1 -- start start 2024-01-16T15:23:45.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.725+0000 7fd2b2800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd2ac143ae0 0x7fd2ac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.725+0000 7fd2b2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.722 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.725+0000 7fd2b2800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac0cfc30 con 0x7fd2ac0cf7d0 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.725+0000 7fd2b2800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac0b1ee0 con 0x7fd2ac143ae0 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.725+0000 7fd2b2800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0b2780 con 0x7fd2ac15e3b0 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2a2800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd2ac0cf7d0 0x7fd2ac15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47282/0 (socket says 172.21.15.141:47282) 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2a2800640 1 -- 172.21.15.141:0/53011737 learned_addr learned my addr 172.21.15.141:0/53011737 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2ab400640 1 --2- 172.21.15.141:0/53011737 >> v2:172.21.15.141:3300/0 conn(0x7fd2ac143ae0 0x7fd2ac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2abe00640 1 --2- 172.21.15.141:0/53011737 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2ab400640 1 -- 172.21.15.141:0/53011737 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 msgr2=0x7fd2ac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:45.723 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2ab400640 1 --2- 172.21.15.141:0/53011737 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2ab400640 1 -- 172.21.15.141:0/53011737 >> v1:172.21.15.141:6789/0 conn(0x7fd2ac0cf7d0 legacy=0x7fd2ac15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.726+0000 7fd2ab400640 1 -- 172.21.15.141:0/53011737 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac15e990 con 0x7fd2ac143ae0 2024-01-16T15:23:45.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.727+0000 7fd2ab400640 1 --2- 172.21.15.141:0/53011737 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac0cf290 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0097a0 tx=0x7fd29c02dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97f9bb7cb95ab30 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:45.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2aaa00640 1 -- 172.21.15.141:0/53011737 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c02f880 con 0x7fd2ac143ae0 2024-01-16T15:23:45.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2aaa00640 1 -- 172.21.15.141:0/53011737 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c02f9e0 con 0x7fd2ac143ae0 2024-01-16T15:23:45.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2aaa00640 1 -- 172.21.15.141:0/53011737 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c02fd70 con 0x7fd2ac143ae0 2024-01-16T15:23:45.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2b2800640 1 -- 172.21.15.141:0/53011737 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 msgr2=0x7fd2ac0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2b2800640 1 --2- 172.21.15.141:0/53011737 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac0cf290 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0097a0 tx=0x7fd29c02dac0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2b2800640 1 -- 172.21.15.141:0/53011737 shutdown_connections 2024-01-16T15:23:45.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2b2800640 1 --2- 172.21.15.141:0/53011737 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2b2800640 1 --2- 172.21.15.141:0/53011737 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac0cf290 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2b2800640 1 -- 172.21.15.141:0/53011737 >> 172.21.15.141:0/53011737 conn(0x7fd2ac06a800 msgr2=0x7fd2ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:45.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.728+0000 7fd2b2800640 1 -- 172.21.15.141:0/53011737 shutdown_connections 2024-01-16T15:23:45.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.729+0000 7fd2b2800640 1 -- 172.21.15.141:0/53011737 wait complete. 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.730+0000 7fd2b2800640 1 Processor -- start 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.730+0000 7fd2b2800640 1 -- start start 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.731+0000 7fd2b2800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac0cf7d0 0x7fd2ac15ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.731+0000 7fd2b2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac13f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.731+0000 7fd2b2800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac13f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.731+0000 7fd2b2800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac13fe30 con 0x7fd2ac143ae0 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.731+0000 7fd2b2800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac13e140 con 0x7fd2ac0cf7d0 2024-01-16T15:23:45.729 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.731+0000 7fd2b2800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac13e3b0 con 0x7fd2ac15e3b0 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac13f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac13f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46282/0 (socket says 172.21.15.141:46282) 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 -- 172.21.15.141:0/2324374955 learned_addr learned my addr 172.21.15.141:0/2324374955 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2abe00640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac13f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2ab400640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac0cf7d0 0x7fd2ac15ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 -- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 msgr2=0x7fd2ac13f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac13f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.730 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 -- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac0cf7d0 msgr2=0x7fd2ac15ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac0cf7d0 0x7fd2ac15ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 -- 172.21.15.141:0/2324374955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c009400 con 0x7fd2ac143ae0 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2ab400640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac0cf7d0 0x7fd2ac15ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.732+0000 7fd2a2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac13f3b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c4d0 tx=0x7fd29400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.733+0000 7fd2a3e00640 1 -- 172.21.15.141:0/2324374955 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294005ce0 con 0x7fd2ac143ae0 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.733+0000 7fd2a3e00640 1 -- 172.21.15.141:0/2324374955 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd294005e40 con 0x7fd2ac143ae0 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.733+0000 7fd2a3e00640 1 -- 172.21.15.141:0/2324374955 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294007270 con 0x7fd2ac143ae0 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.733+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac13e690 con 0x7fd2ac143ae0 2024-01-16T15:23:45.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.733+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac15b1a0 con 0x7fd2ac143ae0 2024-01-16T15:23:45.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.734+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac15b3f0 con 0x7fd2ac143ae0 2024-01-16T15:23:45.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.735+0000 7fd2a3e00640 1 -- 172.21.15.141:0/2324374955 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd2940073d0 con 0x7fd2ac143ae0 2024-01-16T15:23:45.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.736+0000 7fd2a3e00640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd28406c5e0 0x7fd28406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:45.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.736+0000 7fd2a3e00640 1 -- 172.21.15.141:0/2324374955 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd294088020 con 0x7fd2ac143ae0 2024-01-16T15:23:45.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.736+0000 7fd2ab400640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd28406c5e0 0x7fd28406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:45.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.737+0000 7fd2ab400640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd28406c5e0 0x7fd28406eaa0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd29c038000 tx=0x7fd29c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:45.736 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.740+0000 7fd2a3e00640 1 -- 172.21.15.141:0/2324374955 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd294054650 con 0x7fd2ac143ae0 2024-01-16T15:23:45.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.931+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2ac13ee90 con 0x7fd2ac143ae0 2024-01-16T15:23:45.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.932+0000 7fd2a3e00640 1 -- 172.21.15.141:0/2324374955 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd2ac13ee90 con 0x7fd2ac143ae0 2024-01-16T15:23:45.929 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T15:23:45.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd28406c5e0 msgr2=0x7fd28406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.931 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd28406c5e0 0x7fd28406eaa0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd29c038000 tx=0x7fd29c008030 comp rx=0 tx=0).stop 2024-01-16T15:23:45.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 msgr2=0x7fd2ac13f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:45.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac13f3b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c4d0 tx=0x7fd29400c990 comp rx=0 tx=0).stop 2024-01-16T15:23:45.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 shutdown_connections 2024-01-16T15:23:45.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd28406c5e0 0x7fd28406eaa0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2ac15e3b0 0x7fd2ac13f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2ac143ae0 0x7fd2ac13f3b0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 --2- 172.21.15.141:0/2324374955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac0cf7d0 0x7fd2ac15ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:45.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 >> 172.21.15.141:0/2324374955 conn(0x7fd2ac06a800 msgr2=0x7fd2ac15b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:45.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 shutdown_connections 2024-01-16T15:23:45.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:45.935+0000 7fd2b2800640 1 -- 172.21.15.141:0/2324374955 wait complete. 2024-01-16T15:23:45.950 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:46.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.159+0000 7f981a200640 1 Processor -- start 2024-01-16T15:23:46.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.159+0000 7f981a200640 1 -- start start 2024-01-16T15:23:46.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f981a200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f98140cf9a0 0x7f981415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f981a200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f981415e1e0 0x7f98140702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f981a200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98140b1ee0 con 0x7f9814143ae0 2024-01-16T15:23:46.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f981a200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98140b2780 con 0x7f981415e1e0 2024-01-16T15:23:46.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f981a200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981415e990 con 0x7f98140cf9a0 2024-01-16T15:23:46.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f9813e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f981415e1e0 0x7f98140702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f9813e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f981415e1e0 0x7f98140702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46290/0 (socket says 172.21.15.141:46290) 2024-01-16T15:23:46.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f9813e00640 1 -- 172.21.15.141:0/2536790377 learned_addr learned my addr 172.21.15.141:0/2536790377 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:46.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f980be00640 1 --2- 172.21.15.141:0/2536790377 >> v2:172.21.15.188:3300/0 conn(0x7f98140cf9a0 0x7f981415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f9813e00640 1 -- 172.21.15.141:0/2536790377 >> v2:172.21.15.188:3300/0 conn(0x7f98140cf9a0 msgr2=0x7f981415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:46.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f9813e00640 1 --2- 172.21.15.141:0/2536790377 >> v2:172.21.15.188:3300/0 conn(0x7f98140cf9a0 0x7f981415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f9813e00640 1 -- 172.21.15.141:0/2536790377 >> v1:172.21.15.188:6789/0 conn(0x7f9814143ae0 legacy=0x7f98140cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:46.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.161+0000 7f9813e00640 1 -- 172.21.15.141:0/2536790377 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98140b2130 con 0x7f981415e1e0 2024-01-16T15:23:46.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.163+0000 7f9813e00640 1 --2- 172.21.15.141:0/2536790377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981415e1e0 0x7f98140702c0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f98140cb830 tx=0x7f980c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3368669349a6e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:46.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.163+0000 7f9812a00640 1 -- 172.21.15.141:0/2536790377 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c02ca80 con 0x7f981415e1e0 2024-01-16T15:23:46.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.163+0000 7f9812a00640 1 -- 172.21.15.141:0/2536790377 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c02cbe0 con 0x7f981415e1e0 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.163+0000 7f9812a00640 1 -- 172.21.15.141:0/2536790377 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c0343f0 con 0x7f981415e1e0 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 -- 172.21.15.141:0/2536790377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981415e1e0 msgr2=0x7f98140702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 --2- 172.21.15.141:0/2536790377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981415e1e0 0x7f98140702c0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f98140cb830 tx=0x7f980c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 -- 172.21.15.141:0/2536790377 shutdown_connections 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 --2- 172.21.15.141:0/2536790377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f981415e1e0 0x7f98140702c0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 --2- 172.21.15.141:0/2536790377 >> v2:172.21.15.188:3300/0 conn(0x7f98140cf9a0 0x7f981415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 -- 172.21.15.141:0/2536790377 >> 172.21.15.141:0/2536790377 conn(0x7f981406a800 msgr2=0x7f981406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 -- 172.21.15.141:0/2536790377 shutdown_connections 2024-01-16T15:23:46.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.164+0000 7f981a200640 1 -- 172.21.15.141:0/2536790377 wait complete. 2024-01-16T15:23:46.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.166+0000 7f981a200640 1 Processor -- start 2024-01-16T15:23:46.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.166+0000 7f981a200640 1 -- start start 2024-01-16T15:23:46.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f981a200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98140cf9a0 0x7f981406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f981a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9814143ae0 0x7f98141f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f981a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981415e1e0 0x7f98141f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f981a200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98140ad0e0 con 0x7f98140cf9a0 2024-01-16T15:23:46.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f981a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98140cd370 con 0x7f9814143ae0 2024-01-16T15:23:46.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f981a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981413cc70 con 0x7f981415e1e0 2024-01-16T15:23:46.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98140cf9a0 0x7f981406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98140cf9a0 0x7f981406c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46300/0 (socket says 172.21.15.141:46300) 2024-01-16T15:23:46.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 -- 172.21.15.141:0/3957138634 learned_addr learned my addr 172.21.15.141:0/3957138634 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:46.166 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813e00640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981415e1e0 0x7f98141f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f980be00640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9814143ae0 0x7f98141f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 -- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981415e1e0 msgr2=0x7f98141f5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981415e1e0 0x7f98141f5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 -- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9814143ae0 msgr2=0x7f98141f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9814143ae0 0x7f98141f53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.167+0000 7f9813400640 1 -- 172.21.15.141:0/3957138634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c009b30 con 0x7f98140cf9a0 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.168+0000 7f9813e00640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981415e1e0 0x7f98141f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:46.167 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.168+0000 7f9813400640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98140cf9a0 0x7f981406c6e0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f980400c410 tx=0x7f980400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:46.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.168+0000 7f980b400640 1 -- 172.21.15.141:0/3957138634 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98040035f0 con 0x7f98140cf9a0 2024-01-16T15:23:46.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.168+0000 7f980b400640 1 -- 172.21.15.141:0/3957138634 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9804003750 con 0x7f98140cf9a0 2024-01-16T15:23:46.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.168+0000 7f980b400640 1 -- 172.21.15.141:0/3957138634 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804003b40 con 0x7f98140cf9a0 2024-01-16T15:23:46.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.168+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981413c3d0 con 0x7f98140cf9a0 2024-01-16T15:23:46.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.168+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981413c990 con 0x7f98140cf9a0 2024-01-16T15:23:46.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.169+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98140ad220 con 0x7f98140cf9a0 2024-01-16T15:23:46.168 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.170+0000 7f980b400640 1 -- 172.21.15.141:0/3957138634 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f9804003ca0 con 0x7f98140cf9a0 2024-01-16T15:23:46.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.171+0000 7f980b400640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97ec06c5e0 0x7f97ec06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.171+0000 7f980b400640 1 -- 172.21.15.141:0/3957138634 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9804087d80 con 0x7f98140cf9a0 2024-01-16T15:23:46.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.172+0000 7f980be00640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97ec06c5e0 0x7f97ec06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.170 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.173+0000 7f980be00640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97ec06c5e0 0x7f97ec06eaa0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f98140ce890 tx=0x7f97fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:46.171 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.176+0000 7f980b400640 1 -- 172.21.15.141:0/3957138634 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98040542e0 con 0x7f98140cf9a0 2024-01-16T15:23:46.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.367+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98140675d0 con 0x7f98140cf9a0 2024-01-16T15:23:46.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.368+0000 7f980b400640 1 -- 172.21.15.141:0/3957138634 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9804017020 con 0x7f98140cf9a0 2024-01-16T15:23:46.364 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T15:23:46.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.370+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97ec06c5e0 msgr2=0x7f97ec06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.370+0000 7f981a200640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97ec06c5e0 0x7f97ec06eaa0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f98140ce890 tx=0x7f97fc008040 comp rx=0 tx=0).stop 2024-01-16T15:23:46.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.370+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98140cf9a0 msgr2=0x7f981406c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.370+0000 7f981a200640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98140cf9a0 0x7f981406c6e0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f980400c410 tx=0x7f980400c8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 shutdown_connections 2024-01-16T15:23:46.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f97ec06c5e0 0x7f97ec06eaa0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f981415e1e0 0x7f98141f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9814143ae0 0x7f98141f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 --2- 172.21.15.141:0/3957138634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98140cf9a0 0x7f981406c6e0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 >> 172.21.15.141:0/3957138634 conn(0x7f981406a800 msgr2=0x7f98140702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:46.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 shutdown_connections 2024-01-16T15:23:46.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.371+0000 7f981a200640 1 -- 172.21.15.141:0/3957138634 wait complete. 2024-01-16T15:23:46.384 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:46.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.585+0000 7f7f9dc00640 1 Processor -- start 2024-01-16T15:23:46.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.585+0000 7f7f9dc00640 1 -- start start 2024-01-16T15:23:46.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.587+0000 7f7f9dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 0x7f7f980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.587+0000 7f7f9dc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9815c5f0 con 0x7f7f9815f9c0 2024-01-16T15:23:46.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.587+0000 7f7f9dc00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9806c470 con 0x7f7f9813df80 2024-01-16T15:23:46.583 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.587+0000 7f7f9dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9806c750 con 0x7f7f9813e360 2024-01-16T15:23:46.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.587+0000 7f7f97400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7f9815f9c0 0x7f7f980cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47298/0 (socket says 172.21.15.141:47298) 2024-01-16T15:23:46.584 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.587+0000 7f7f97400640 1 -- 172.21.15.141:0/4032476138 learned_addr learned my addr 172.21.15.141:0/4032476138 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:46.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f8de00640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 0x7f7f980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191686952 0 0) 0x7f7f9815c5f0 con 0x7f7f9815f9c0 2024-01-16T15:23:46.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f78003520 con 0x7f7f9815f9c0 2024-01-16T15:23:46.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 97291913 0 0) 0x7f7f78003520 con 0x7f7f9815f9c0 2024-01-16T15:23:46.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 msgr2=0x7f7f980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:46.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 0x7f7f980cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.587 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 >> v1:172.21.15.188:6789/0 conn(0x7f7f9813df80 legacy=0x7f7f9815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9806ca30 con 0x7f7f9815f9c0 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f7f90001670 con 0x7f7f9815f9c0 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 >> v1:172.21.15.141:6789/0 conn(0x7f7f9815f9c0 legacy=0x7f7f980cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f780039a0 0x7f7f78003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 0x7f7f78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9806c750 con 0x7f7f780039a0 2024-01-16T15:23:46.588 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7800ba60 con 0x7f7f780047c0 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.588+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7800bc20 con 0x7f7f9813e360 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f96a00640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f780039a0 0x7f7f78003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f97400640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 0x7f7f78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f8de00640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f96a00640 1 -- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 msgr2=0x7f7f78005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f96a00640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f9813e360 0x7f7f78005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f96a00640 1 -- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f780047c0 msgr2=0x7f7f78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:46.589 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f96a00640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.589+0000 7f7f96a00640 1 -- 172.21.15.141:0/4032476138 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f78003520 con 0x7f7f780039a0 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.590+0000 7f7f96a00640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f780039a0 0x7f7f78003dc0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f7f88001e20 tx=0x7f7f780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899602ef08df7ec2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.590+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f88003b70 con 0x7f7f780039a0 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.590+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f88003cd0 con 0x7f7f780039a0 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.590+0000 7f7f96000640 1 -- 172.21.15.141:0/4032476138 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f88005310 con 0x7f7f780039a0 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.591+0000 7f7f9dc00640 1 -- 172.21.15.141:0/4032476138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f780039a0 msgr2=0x7f7f78003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.591+0000 7f7f9dc00640 1 --2- 172.21.15.141:0/4032476138 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f780039a0 0x7f7f78003dc0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f7f88001e20 tx=0x7f7f780057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.591+0000 7f7f97400640 1 -- 172.21.15.141:0/4032476138 reap_dead start 2024-01-16T15:23:46.590 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.591+0000 7f7f9dc00640 1 -- 172.21.15.141:0/4032476138 shutdown_connections 2024-01-16T15:23:46.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.591+0000 7f7f9dc00640 1 -- 172.21.15.141:0/4032476138 >> 172.21.15.141:0/4032476138 conn(0x7f7f9806a800 msgr2=0x7f7f9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:46.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.591+0000 7f7f9dc00640 1 -- 172.21.15.141:0/4032476138 shutdown_connections 2024-01-16T15:23:46.591 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.591+0000 7f7f9dc00640 1 -- 172.21.15.141:0/4032476138 wait complete. 2024-01-16T15:23:46.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.592+0000 7f7f9dc00640 1 Processor -- start 2024-01-16T15:23:46.593 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.593+0000 7f7f9dc00640 1 -- start start 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.593+0000 7f7f9dc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f780039a0 0x7f7f981f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.593+0000 7f7f9dc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f981536b0 0x7f7f98139890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.593+0000 7f7f9dc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f9813cb80 0x7f7f980cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.593+0000 7f7f9dc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f980b31b0 con 0x7f7f9813cb80 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.593+0000 7f7f9dc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f980b25c0 con 0x7f7f981536b0 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.593+0000 7f7f9dc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f981f3700 con 0x7f7f780039a0 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f9813cb80 0x7f7f980cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.594 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f9813cb80 0x7f7f980cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46318/0 (socket says 172.21.15.141:46318) 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 -- 172.21.15.141:0/3119071169 learned_addr learned my addr 172.21.15.141:0/3119071169 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 -- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f780039a0 msgr2=0x7f7f981f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f780039a0 0x7f7f981f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 -- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f981536b0 msgr2=0x7f7f98139890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f981536b0 0x7f7f98139890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 -- 172.21.15.141:0/3119071169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f980b2810 con 0x7f7f9813cb80 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f97400640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f9813cb80 0x7f7f980cd800 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f7f90001200 tx=0x7f7f9000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f8f400640 1 -- 172.21.15.141:0/3119071169 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90007230 con 0x7f7f9813cb80 2024-01-16T15:23:46.595 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f8f400640 1 -- 172.21.15.141:0/3119071169 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f90007390 con 0x7f7f9813cb80 2024-01-16T15:23:46.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.594+0000 7f7f8f400640 1 -- 172.21.15.141:0/3119071169 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f900047e0 con 0x7f7f9813cb80 2024-01-16T15:23:46.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.595+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f980b2af0 con 0x7f7f9813cb80 2024-01-16T15:23:46.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.595+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f98139dd0 con 0x7f7f9813cb80 2024-01-16T15:23:46.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.596+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f980675d0 con 0x7f7f9813cb80 2024-01-16T15:23:46.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.597+0000 7f7f8f400640 1 -- 172.21.15.141:0/3119071169 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f7f90004940 con 0x7f7f9813cb80 2024-01-16T15:23:46.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.597+0000 7f7f8f400640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f7006c610 0x7f7f7006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:46.596 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.597+0000 7f7f8f400640 1 -- 172.21.15.141:0/3119071169 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f90088990 con 0x7f7f9813cb80 2024-01-16T15:23:46.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.598+0000 7f7f96a00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f7006c610 0x7f7f7006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:46.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.599+0000 7f7f96a00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f7006c610 0x7f7f7006ead0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7f88034000 tx=0x7f7f88030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:46.597 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.601+0000 7f7f8f400640 1 -- 172.21.15.141:0/3119071169 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f90054eb0 con 0x7f7f9813cb80 2024-01-16T15:23:46.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.800+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7f980cc300 con 0x7f7f9813cb80 2024-01-16T15:23:46.797 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.801+0000 7f7f8f400640 1 -- 172.21.15.141:0/3119071169 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7f900584d0 con 0x7f7f9813cb80 2024-01-16T15:23:46.798 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T15:23:46.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f7006c610 msgr2=0x7f7f7006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.800 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f7006c610 0x7f7f7006ead0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7f88034000 tx=0x7f7f88030000 comp rx=0 tx=0).stop 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f9813cb80 msgr2=0x7f7f980cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f9813cb80 0x7f7f980cd800 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f7f90001200 tx=0x7f7f9000dc00 comp rx=0 tx=0).stop 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 shutdown_connections 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f7f7006c610 0x7f7f7006ead0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f9813cb80 0x7f7f980cd800 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f981536b0 0x7f7f98139890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 --2- 172.21.15.141:0/3119071169 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f7f780039a0 0x7f7f981f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 >> 172.21.15.141:0/3119071169 conn(0x7f7f9806a800 msgr2=0x7f7f9815e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:46.801 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 shutdown_connections 2024-01-16T15:23:46.802 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:46.804+0000 7f7f9dc00640 1 -- 172.21.15.141:0/3119071169 wait complete. 2024-01-16T15:23:46.819 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:23:46.819 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:46.865 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:47.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.152+0000 7f5d10800640 1 Processor -- start 2024-01-16T15:23:47.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.152+0000 7f5d10800640 1 -- start start 2024-01-16T15:23:47.149 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.153+0000 7f5d10800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.153+0000 7f5d10800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5d080cf9a0 0x7f5d080cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.153+0000 7f5d10800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0806c470 con 0x7f5d0813e360 2024-01-16T15:23:47.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.153+0000 7f5d10800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0806c750 con 0x7f5d080cf9a0 2024-01-16T15:23:47.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.153+0000 7f5d10800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0806ca30 con 0x7f5d0813df80 2024-01-16T15:23:47.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.153+0000 7f5d0e800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5d080cf9a0 0x7f5d080cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0e800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5d080cf9a0 0x7f5d080cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46324/0 (socket says 172.21.15.141:46324) 2024-01-16T15:23:47.150 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0e800640 1 -- 172.21.15.141:0/2048333566 learned_addr learned my addr 172.21.15.141:0/2048333566 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:47.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0de00640 1 --2- 172.21.15.141:0/2048333566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0e800640 1 -- 172.21.15.141:0/2048333566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 msgr2=0x7f5d0815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:47.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0e800640 1 --2- 172.21.15.141:0/2048333566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0e800640 1 -- 172.21.15.141:0/2048333566 >> v1:172.21.15.188:6789/0 conn(0x7f5d0813e360 legacy=0x7f5d080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:47.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0e800640 1 -- 172.21.15.141:0/2048333566 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0806cd10 con 0x7f5d080cf9a0 2024-01-16T15:23:47.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.154+0000 7f5d0de00640 1 --2- 172.21.15.141:0/2048333566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:47.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.155+0000 7f5d0e800640 1 --2- 172.21.15.141:0/2048333566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d080cf9a0 0x7f5d080cd840 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009e50 tx=0x7f5d0002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d157788e4600816 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:47.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.156+0000 7f5d0ca00640 1 -- 172.21.15.141:0/2048333566 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0002ca80 con 0x7f5d080cf9a0 2024-01-16T15:23:47.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.156+0000 7f5d0ca00640 1 -- 172.21.15.141:0/2048333566 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0002cbe0 con 0x7f5d080cf9a0 2024-01-16T15:23:47.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.156+0000 7f5d0ca00640 1 -- 172.21.15.141:0/2048333566 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d000343f0 con 0x7f5d080cf9a0 2024-01-16T15:23:47.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.157+0000 7f5d10800640 1 -- 172.21.15.141:0/2048333566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d080cf9a0 msgr2=0x7f5d080cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.157+0000 7f5d10800640 1 --2- 172.21.15.141:0/2048333566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d080cf9a0 0x7f5d080cd840 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009e50 tx=0x7f5d0002bee0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.157+0000 7f5d10800640 1 -- 172.21.15.141:0/2048333566 shutdown_connections 2024-01-16T15:23:47.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.157+0000 7f5d10800640 1 --2- 172.21.15.141:0/2048333566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d080cf9a0 0x7f5d080cd840 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.157+0000 7f5d10800640 1 --2- 172.21.15.141:0/2048333566 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.157+0000 7f5d10800640 1 -- 172.21.15.141:0/2048333566 >> 172.21.15.141:0/2048333566 conn(0x7f5d0806a800 msgr2=0x7f5d0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:47.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.158+0000 7f5d10800640 1 -- 172.21.15.141:0/2048333566 shutdown_connections 2024-01-16T15:23:47.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.158+0000 7f5d10800640 1 -- 172.21.15.141:0/2048333566 wait complete. 2024-01-16T15:23:47.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.161+0000 7f5d10800640 1 Processor -- start 2024-01-16T15:23:47.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.161+0000 7f5d10800640 1 -- start start 2024-01-16T15:23:47.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d10800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d080cf9a0 0x7f5d080b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d10800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0813d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d10800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d0813e360 0x7f5d081f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d10800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d081f28b0 con 0x7f5d0813e360 2024-01-16T15:23:47.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d10800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0814e1b0 con 0x7f5d080cf9a0 2024-01-16T15:23:47.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d10800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080b1d40 con 0x7f5d0813df80 2024-01-16T15:23:47.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d0e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d0813e360 0x7f5d081f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.163+0000 7f5d0e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d0813e360 0x7f5d081f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46334/0 (socket says 172.21.15.141:46334) 2024-01-16T15:23:47.160 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.164+0000 7f5d0e800640 1 -- 172.21.15.141:0/4141990916 learned_addr learned my addr 172.21.15.141:0/4141990916 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:47.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.164+0000 7f5d0d400640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.165+0000 7f5d0e800640 1 -- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 msgr2=0x7f5d0813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.165+0000 7f5d0e800640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0813d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.165+0000 7f5d0e800640 1 -- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d080cf9a0 msgr2=0x7f5d080b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.161 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.165+0000 7f5d0e800640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d080cf9a0 0x7f5d080b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.165+0000 7f5d0e800640 1 -- 172.21.15.141:0/4141990916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d00009b30 con 0x7f5d0813e360 2024-01-16T15:23:47.162 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.166+0000 7f5d0e800640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d0813e360 0x7f5d081f5f80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f5d00036040 tx=0x7f5d00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:47.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.167+0000 7f5d06000640 1 -- 172.21.15.141:0/4141990916 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0003e450 con 0x7f5d0813e360 2024-01-16T15:23:47.163 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.167+0000 7f5d06000640 1 -- 172.21.15.141:0/4141990916 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0003e5d0 con 0x7f5d0813e360 2024-01-16T15:23:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.167+0000 7f5d06000640 1 -- 172.21.15.141:0/4141990916 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0003e980 con 0x7f5d0813e360 2024-01-16T15:23:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.167+0000 7f5d10800640 1 -- 172.21.15.141:0/4141990916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d080b2020 con 0x7f5d0813e360 2024-01-16T15:23:47.164 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.168+0000 7f5d10800640 1 -- 172.21.15.141:0/4141990916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d081f6db0 con 0x7f5d0813e360 2024-01-16T15:23:47.169 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.172+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d080b2540 con 0x7f5d0813e360 2024-01-16T15:23:47.173 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.175+0000 7f5d06000640 1 -- 172.21.15.141:0/4141990916 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f5d0002cd50 con 0x7f5d0813e360 2024-01-16T15:23:47.175 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.179+0000 7f5d06000640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ce006c630 0x7f5ce006eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.176 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.179+0000 7f5d06000640 1 -- 172.21.15.141:0/4141990916 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5d000ab740 con 0x7f5d0813e360 2024-01-16T15:23:47.177 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.181+0000 7f5d0de00640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ce006c630 0x7f5ce006eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.180 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.183+0000 7f5d0de00640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ce006c630 0x7f5ce006eaf0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8006fd0 tx=0x7f5cf8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:47.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.185+0000 7f5d06000640 1 -- 172.21.15.141:0/4141990916 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d080b2540 con 0x7f5d0813e360 2024-01-16T15:23:47.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.470+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d080b2540 con 0x7f5d0813e360 2024-01-16T15:23:47.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.472+0000 7f5d06000640 1 -- 172.21.15.141:0/4141990916 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16668 (secure 0 0 0) 0x7f5d080b2540 con 0x7f5d0813e360 2024-01-16T15:23:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:47.468 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":118,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:55.853050+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.4","mappings":[{"from":0,"to":5}]},{"pgid":"17.3c","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:47.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ce006c630 msgr2=0x7f5ce006eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ce006c630 0x7f5ce006eaf0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8006fd0 tx=0x7f5cf8008040 comp rx=0 tx=0).stop 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d0813e360 msgr2=0x7f5d081f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d0813e360 0x7f5d081f5f80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f5d00036040 tx=0x7f5d00004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 shutdown_connections 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f5ce006c630 0x7f5ce006eaf0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d0813e360 0x7f5d081f5f80 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5d0813df80 0x7f5d0813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 --2- 172.21.15.141:0/4141990916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d080cf9a0 0x7f5d080b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 >> 172.21.15.141:0/4141990916 conn(0x7f5d0806a800 msgr2=0x7f5d080b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:47.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 shutdown_connections 2024-01-16T15:23:47.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.475+0000 7f5cff400640 1 -- 172.21.15.141:0/4141990916 wait complete. 2024-01-16T15:23:47.494 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:36.311732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:37.759492+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:47.494 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:47.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.757+0000 7ff08c800640 1 Processor -- start 2024-01-16T15:23:47.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.757+0000 7ff08c800640 1 -- start start 2024-01-16T15:23:47.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.759+0000 7ff08c800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff0840b3ae0 0x7ff08415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.759+0000 7ff08c800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff084070c40 con 0x7ff0840cf290 2024-01-16T15:23:47.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.759+0000 7ff08c800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08406c470 con 0x7ff0840ae5a0 2024-01-16T15:23:47.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.759+0000 7ff08c800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff084071450 con 0x7ff0840b3ae0 2024-01-16T15:23:47.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.760+0000 7ff08a800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff0840cf290 0x7ff0840b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47304/0 (socket says 172.21.15.141:47304) 2024-01-16T15:23:47.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.760+0000 7ff08a800640 1 -- 172.21.15.141:0/1794802297 learned_addr learned my addr 172.21.15.141:0/1794802297 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:47.756 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.760+0000 7ff089400640 1 --2- 172.21.15.141:0/1794802297 >> v2:172.21.15.141:3300/0 conn(0x7ff0840b3ae0 0x7ff08415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.761+0000 7ff089400640 1 -- 172.21.15.141:0/1794802297 >> v1:172.21.15.188:6789/0 conn(0x7ff0840ae5a0 legacy=0x7ff08406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:47.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.761+0000 7ff089400640 1 -- 172.21.15.141:0/1794802297 >> v1:172.21.15.141:6789/0 conn(0x7ff0840cf290 legacy=0x7ff0840b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.761+0000 7ff089400640 1 -- 172.21.15.141:0/1794802297 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08406c750 con 0x7ff0840b3ae0 2024-01-16T15:23:47.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.761+0000 7ff088a00640 1 -- 172.21.15.141:0/1794802297 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404451752 0 0) 0x7ff084070c40 con 0x7ff0840cf290 2024-01-16T15:23:47.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.763+0000 7ff089400640 1 --2- 172.21.15.141:0/1794802297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840b3ae0 0x7ff08415e430 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7ff0740097a0 tx=0x7ff07402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bef9854ee374f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:47.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.764+0000 7ff088a00640 1 -- 172.21.15.141:0/1794802297 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074030610 con 0x7ff0840b3ae0 2024-01-16T15:23:47.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.764+0000 7ff088a00640 1 -- 172.21.15.141:0/1794802297 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff074030770 con 0x7ff0840b3ae0 2024-01-16T15:23:47.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.764+0000 7ff088a00640 1 -- 172.21.15.141:0/1794802297 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074030b00 con 0x7ff0840b3ae0 2024-01-16T15:23:47.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.765+0000 7ff08c800640 1 -- 172.21.15.141:0/1794802297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840b3ae0 msgr2=0x7ff08415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.765+0000 7ff08c800640 1 --2- 172.21.15.141:0/1794802297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840b3ae0 0x7ff08415e430 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7ff0740097a0 tx=0x7ff07402b960 comp rx=0 tx=0).stop 2024-01-16T15:23:47.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.765+0000 7ff08c800640 1 -- 172.21.15.141:0/1794802297 shutdown_connections 2024-01-16T15:23:47.762 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.765+0000 7ff08c800640 1 --2- 172.21.15.141:0/1794802297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840b3ae0 0x7ff08415e430 secure :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7ff0740097a0 tx=0x7ff07402b960 comp rx=0 tx=0).stop 2024-01-16T15:23:47.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.765+0000 7ff08c800640 1 -- 172.21.15.141:0/1794802297 >> 172.21.15.141:0/1794802297 conn(0x7ff08406a800 msgr2=0x7ff08406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:47.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.767+0000 7ff08c800640 1 -- 172.21.15.141:0/1794802297 shutdown_connections 2024-01-16T15:23:47.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.767+0000 7ff08c800640 1 -- 172.21.15.141:0/1794802297 wait complete. 2024-01-16T15:23:47.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.769+0000 7ff08c800640 1 Processor -- start 2024-01-16T15:23:47.766 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.770+0000 7ff08c800640 1 -- start start 2024-01-16T15:23:47.767 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.771+0000 7ff08c800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840ae5a0 0x7ff0840ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.771+0000 7ff08c800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0840cf290 0x7ff08414b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.771+0000 7ff08c800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff08415cbf0 0x7ff084160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.771+0000 7ff08c800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff084157ee0 con 0x7ff0840ae5a0 2024-01-16T15:23:47.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.771+0000 7ff08c800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08415d070 con 0x7ff0840cf290 2024-01-16T15:23:47.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.771+0000 7ff08c800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff084157540 con 0x7ff08415cbf0 2024-01-16T15:23:47.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840ae5a0 0x7ff0840ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840ae5a0 0x7ff0840ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46350/0 (socket says 172.21.15.141:46350) 2024-01-16T15:23:47.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 -- 172.21.15.141:0/789068787 learned_addr learned my addr 172.21.15.141:0/789068787 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089400640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0840cf290 0x7ff08414b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 -- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff08415cbf0 msgr2=0x7ff084160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff08415cbf0 0x7ff084160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 -- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0840cf290 msgr2=0x7ff08414b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0840cf290 0x7ff08414b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.772+0000 7ff089e00640 1 -- 172.21.15.141:0/789068787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff074009400 con 0x7ff0840ae5a0 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.773+0000 7ff089e00640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840ae5a0 0x7ff0840ccaa0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00c430 tx=0x7ff06c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:47.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.773+0000 7ff082000640 1 -- 172.21.15.141:0/789068787 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c011790 con 0x7ff0840ae5a0 2024-01-16T15:23:47.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.773+0000 7ff082000640 1 -- 172.21.15.141:0/789068787 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c0118f0 con 0x7ff0840ae5a0 2024-01-16T15:23:47.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.773+0000 7ff082000640 1 -- 172.21.15.141:0/789068787 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c011ce0 con 0x7ff0840ae5a0 2024-01-16T15:23:47.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.773+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff084157820 con 0x7ff0840ae5a0 2024-01-16T15:23:47.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.773+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff084158300 con 0x7ff0840ae5a0 2024-01-16T15:23:47.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.775+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0840675d0 con 0x7ff0840ae5a0 2024-01-16T15:23:47.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.777+0000 7ff082000640 1 -- 172.21.15.141:0/789068787 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ff06c01c460 con 0x7ff0840ae5a0 2024-01-16T15:23:47.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.778+0000 7ff082000640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff05c06c630 0x7ff05c06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:47.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.779+0000 7ff082000640 1 -- 172.21.15.141:0/789068787 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff06c088b70 con 0x7ff0840ae5a0 2024-01-16T15:23:47.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.782+0000 7ff089400640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff05c06c630 0x7ff05c06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:47.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.786+0000 7ff082000640 1 -- 172.21.15.141:0/789068787 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff06c0550d0 con 0x7ff0840ae5a0 2024-01-16T15:23:47.783 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:47.787+0000 7ff089400640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff05c06c630 0x7ff05c06eaf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff074036000 tx=0x7ff074008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:48.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.076+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff084160a50 con 0x7ff0840ae5a0 2024-01-16T15:23:48.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.082+0000 7ff082000640 1 -- 172.21.15.141:0/789068787 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff06c0586f0 con 0x7ff0840ae5a0 2024-01-16T15:23:48.079 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:23:48.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff05c06c630 msgr2=0x7ff05c06eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff05c06c630 0x7ff05c06eaf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff074036000 tx=0x7ff074008030 comp rx=0 tx=0).stop 2024-01-16T15:23:48.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840ae5a0 msgr2=0x7ff0840ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840ae5a0 0x7ff0840ccaa0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00c430 tx=0x7ff06c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 shutdown_connections 2024-01-16T15:23:48.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff05c06c630 0x7ff05c06eaf0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff08415cbf0 0x7ff084160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0840cf290 0x7ff08414b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 --2- 172.21.15.141:0/789068787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0840ae5a0 0x7ff0840ccaa0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.089+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 >> 172.21.15.141:0/789068787 conn(0x7ff08406a800 msgr2=0x7ff08406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:48.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.090+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 shutdown_connections 2024-01-16T15:23:48.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.090+0000 7ff08c800640 1 -- 172.21.15.141:0/789068787 wait complete. 2024-01-16T15:23:48.114 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:48.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.382+0000 7f4530e00640 1 Processor -- start 2024-01-16T15:23:48.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.382+0000 7f4530e00640 1 -- start start 2024-01-16T15:23:48.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f4530e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f452c0cf9a0 0x7f452c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f4530e00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f452c0b31d0 0x7f452c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f4530e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0d0320 con 0x7f452c15d4b0 2024-01-16T15:23:48.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f4530e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0d1c60 con 0x7f452c0cf9a0 2024-01-16T15:23:48.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f4530e00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0b2730 con 0x7f452c0b31d0 2024-01-16T15:23:48.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f452aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f452c0cf9a0 0x7f452c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f452aa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f452c0cf9a0 0x7f452c0cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46356/0 (socket says 172.21.15.141:46356) 2024-01-16T15:23:48.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f452aa00640 1 -- 172.21.15.141:0/3541912098 learned_addr learned my addr 172.21.15.141:0/3541912098 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:48.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f452be00640 1 --2- 172.21.15.141:0/3541912098 >> v2:172.21.15.188:3300/0 conn(0x7f452c0b31d0 0x7f452c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f452aa00640 1 -- 172.21.15.141:0/3541912098 >> v2:172.21.15.188:3300/0 conn(0x7f452c0b31d0 msgr2=0x7f452c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:48.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f452aa00640 1 --2- 172.21.15.141:0/3541912098 >> v2:172.21.15.188:3300/0 conn(0x7f452c0b31d0 0x7f452c0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.384+0000 7f452aa00640 1 -- 172.21.15.141:0/3541912098 >> v1:172.21.15.188:6789/0 conn(0x7f452c15d4b0 legacy=0x7f452c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:48.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.385+0000 7f452aa00640 1 -- 172.21.15.141:0/3541912098 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0d1ec0 con 0x7f452c0cf9a0 2024-01-16T15:23:48.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.385+0000 7f452be00640 1 --2- 172.21.15.141:0/3541912098 >> v2:172.21.15.188:3300/0 conn(0x7f452c0b31d0 0x7f452c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:48.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.386+0000 7f452aa00640 1 --2- 172.21.15.141:0/3541912098 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c0cf9a0 0x7f452c0cfde0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f452c072a30 tx=0x7f451402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233d5b3595696569 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:48.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.386+0000 7f452a000640 1 -- 172.21.15.141:0/3541912098 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514031870 con 0x7f452c0cf9a0 2024-01-16T15:23:48.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.386+0000 7f452a000640 1 -- 172.21.15.141:0/3541912098 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45140319d0 con 0x7f452c0cf9a0 2024-01-16T15:23:48.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.386+0000 7f452a000640 1 -- 172.21.15.141:0/3541912098 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514031d60 con 0x7f452c0cf9a0 2024-01-16T15:23:48.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 -- 172.21.15.141:0/3541912098 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c0cf9a0 msgr2=0x7f452c0cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 --2- 172.21.15.141:0/3541912098 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c0cf9a0 0x7f452c0cfde0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f452c072a30 tx=0x7f451402b910 comp rx=0 tx=0).stop 2024-01-16T15:23:48.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 -- 172.21.15.141:0/3541912098 shutdown_connections 2024-01-16T15:23:48.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 --2- 172.21.15.141:0/3541912098 >> v2:172.21.15.188:3300/0 conn(0x7f452c0b31d0 0x7f452c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 --2- 172.21.15.141:0/3541912098 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c0cf9a0 0x7f452c0cfde0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 -- 172.21.15.141:0/3541912098 >> 172.21.15.141:0/3541912098 conn(0x7f452c0d3000 msgr2=0x7f452c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:48.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 -- 172.21.15.141:0/3541912098 shutdown_connections 2024-01-16T15:23:48.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.387+0000 7f4530e00640 1 -- 172.21.15.141:0/3541912098 wait complete. 2024-01-16T15:23:48.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.388+0000 7f4530e00640 1 Processor -- start 2024-01-16T15:23:48.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.388+0000 7f4530e00640 1 -- start start 2024-01-16T15:23:48.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.389+0000 7f4530e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f452c0b31d0 0x7f452c0b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.389+0000 7f4530e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f452c0cf9a0 0x7f452c1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.389+0000 7f4530e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c15d4b0 0x7f452c1f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.389+0000 7f4530e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c155410 con 0x7f452c15d4b0 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.389+0000 7f4530e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c15acb0 con 0x7f452c0cf9a0 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.389+0000 7f4530e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c154de0 con 0x7f452c0b31d0 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c15d4b0 0x7f452c1f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c15d4b0 0x7f452c1f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46372/0 (socket says 172.21.15.141:46372) 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 -- 172.21.15.141:0/3568465581 learned_addr learned my addr 172.21.15.141:0/3568465581 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 -- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f452c0b31d0 msgr2=0x7f452c0b8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f452c0b31d0 0x7f452c0b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.388 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 -- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f452c0cf9a0 msgr2=0x7f452c1558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f452c0cf9a0 0x7f452c1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 -- 172.21.15.141:0/3568465581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140094d0 con 0x7f452c15d4b0 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f452be00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c15d4b0 0x7f452c1f6240 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f452400cd50 tx=0x7f452400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f4523400640 1 -- 172.21.15.141:0/3568465581 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452400f680 con 0x7f452c15d4b0 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f4523400640 1 -- 172.21.15.141:0/3568465581 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452400f800 con 0x7f452c15d4b0 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.390+0000 7f4523400640 1 -- 172.21.15.141:0/3568465581 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452400fbb0 con 0x7f452c15d4b0 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.391+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c0b8cb0 con 0x7f452c15d4b0 2024-01-16T15:23:48.389 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.391+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c1f7070 con 0x7f452c15d4b0 2024-01-16T15:23:48.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.392+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c067450 con 0x7f452c15d4b0 2024-01-16T15:23:48.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.393+0000 7f4523400640 1 -- 172.21.15.141:0/3568465581 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f452400b040 con 0x7f452c15d4b0 2024-01-16T15:23:48.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.393+0000 7f4523400640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f450406c5e0 0x7f450406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.394+0000 7f4523400640 1 -- 172.21.15.141:0/3568465581 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4524089260 con 0x7f452c15d4b0 2024-01-16T15:23:48.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.399+0000 7f4523400640 1 -- 172.21.15.141:0/3568465581 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4524055b20 con 0x7f452c15d4b0 2024-01-16T15:23:48.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.401+0000 7f452b400640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f450406c5e0 0x7f450406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.409+0000 7f452b400640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f450406c5e0 0x7f450406eaa0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f451c006fd0 tx=0x7f451c006bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:48.677 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.681+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f452c072880 con 0x7f452c15d4b0 2024-01-16T15:23:48.678 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.682+0000 7f4523400640 1 -- 172.21.15.141:0/3568465581 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7f452401d020 con 0x7f452c15d4b0 2024-01-16T15:23:48.678 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T15:23:48.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f450406c5e0 msgr2=0x7f450406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f450406c5e0 0x7f450406eaa0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f451c006fd0 tx=0x7f451c006bc0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c15d4b0 msgr2=0x7f452c1f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.682 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c15d4b0 0x7f452c1f6240 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f452400cd50 tx=0x7f452400a650 comp rx=0 tx=0).stop 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 shutdown_connections 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f450406c5e0 0x7f450406eaa0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f452c15d4b0 0x7f452c1f6240 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f452c0cf9a0 0x7f452c1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 --2- 172.21.15.141:0/3568465581 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f452c0b31d0 0x7f452c0b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 >> 172.21.15.141:0/3568465581 conn(0x7f452c0d3000 msgr2=0x7f452c06ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 shutdown_connections 2024-01-16T15:23:48.683 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.686+0000 7f4530e00640 1 -- 172.21.15.141:0/3568465581 wait complete. 2024-01-16T15:23:48.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:48.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.943+0000 7f3a9d200640 1 Processor -- start 2024-01-16T15:23:48.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.944+0000 7f3a9d200640 1 -- start start 2024-01-16T15:23:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.945+0000 7f3a9d200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3a980cf9a0 0x7f3a98166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.945+0000 7f3a9d200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9815c5f0 con 0x7f3a9813e360 2024-01-16T15:23:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.945+0000 7f3a9d200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9806c470 con 0x7f3a9813df80 2024-01-16T15:23:48.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.945+0000 7f3a9d200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a9806c750 con 0x7f3a980cf9a0 2024-01-16T15:23:48.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.945+0000 7f3a95400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3a9813e360 0x7f3a980cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47318/0 (socket says 172.21.15.141:47318) 2024-01-16T15:23:48.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.945+0000 7f3a96800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3a980cf9a0 0x7f3a98166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.945+0000 7f3a95400640 1 -- 172.21.15.141:0/775671240 learned_addr learned my addr 172.21.15.141:0/775671240 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:48.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.946+0000 7f3a96800640 1 -- 172.21.15.141:0/775671240 >> v1:172.21.15.188:6789/0 conn(0x7f3a9813df80 legacy=0x7f3a9815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:48.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.946+0000 7f3a96800640 1 -- 172.21.15.141:0/775671240 >> v1:172.21.15.141:6789/0 conn(0x7f3a9813e360 legacy=0x7f3a980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.946+0000 7f3a96800640 1 -- 172.21.15.141:0/775671240 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9806ca30 con 0x7f3a980cf9a0 2024-01-16T15:23:48.943 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.946+0000 7f3a94a00640 1 -- 172.21.15.141:0/775671240 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293471307 0 0) 0x7f3a9815c5f0 con 0x7f3a9813e360 2024-01-16T15:23:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.947+0000 7f3a96800640 1 --2- 172.21.15.141:0/775671240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a980cf9a0 0x7f3a98166030 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009d90 tx=0x7f3a9002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72769f9b49f8731b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.947+0000 7f3a94a00640 1 -- 172.21.15.141:0/775671240 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9002cc60 con 0x7f3a980cf9a0 2024-01-16T15:23:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.947+0000 7f3a94a00640 1 -- 172.21.15.141:0/775671240 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9002cdc0 con 0x7f3a980cf9a0 2024-01-16T15:23:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.947+0000 7f3a94a00640 1 -- 172.21.15.141:0/775671240 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90034660 con 0x7f3a980cf9a0 2024-01-16T15:23:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.947+0000 7f3a9d200640 1 -- 172.21.15.141:0/775671240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a980cf9a0 msgr2=0x7f3a98166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.947+0000 7f3a9d200640 1 --2- 172.21.15.141:0/775671240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a980cf9a0 0x7f3a98166030 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009d90 tx=0x7f3a9002beb0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.947+0000 7f3a9d200640 1 -- 172.21.15.141:0/775671240 shutdown_connections 2024-01-16T15:23:48.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.948+0000 7f3a9d200640 1 --2- 172.21.15.141:0/775671240 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a980cf9a0 0x7f3a98166030 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.948+0000 7f3a9d200640 1 -- 172.21.15.141:0/775671240 >> 172.21.15.141:0/775671240 conn(0x7f3a9806a800 msgr2=0x7f3a9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:48.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.948+0000 7f3a9d200640 1 -- 172.21.15.141:0/775671240 shutdown_connections 2024-01-16T15:23:48.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.948+0000 7f3a9d200640 1 -- 172.21.15.141:0/775671240 wait complete. 2024-01-16T15:23:48.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.949+0000 7f3a9d200640 1 Processor -- start 2024-01-16T15:23:48.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.949+0000 7f3a9d200640 1 -- start start 2024-01-16T15:23:48.948 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a9d200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a980cf9a0 0x7f3a980add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a9d200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a9813df80 0x7f3a981e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a9d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a9813e360 0x7f3a980b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a9d200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a981e7ca0 con 0x7f3a9813e360 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a9d200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a981e7020 con 0x7f3a980cf9a0 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a9d200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a981e7300 con 0x7f3a9813df80 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a96800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a9813e360 0x7f3a980b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a96800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a9813e360 0x7f3a980b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46386/0 (socket says 172.21.15.141:46386) 2024-01-16T15:23:48.949 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.950+0000 7f3a96800640 1 -- 172.21.15.141:0/1372556523 learned_addr learned my addr 172.21.15.141:0/1372556523 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a95400640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a9813df80 0x7f3a981e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a96800640 1 -- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a9813df80 msgr2=0x7f3a981e80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a95e00640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a980cf9a0 0x7f3a980add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a96800640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a9813df80 0x7f3a981e80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a96800640 1 -- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a980cf9a0 msgr2=0x7f3a980add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a96800640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a980cf9a0 0x7f3a980add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a96800640 1 -- 172.21.15.141:0/1372556523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90009a70 con 0x7f3a9813e360 2024-01-16T15:23:48.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a95400640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a9813df80 0x7f3a981e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a96800640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a9813e360 0x7f3a980b2cf0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009d90 tx=0x7f3a90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a8e000640 1 -- 172.21.15.141:0/1372556523 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9003e610 con 0x7f3a9813e360 2024-01-16T15:23:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a8e000640 1 -- 172.21.15.141:0/1372556523 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9003e790 con 0x7f3a9813e360 2024-01-16T15:23:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.951+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a981e75e0 con 0x7f3a9813e360 2024-01-16T15:23:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.952+0000 7f3a8e000640 1 -- 172.21.15.141:0/1372556523 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9003eb40 con 0x7f3a9813e360 2024-01-16T15:23:48.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.952+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a980ae260 con 0x7f3a9813e360 2024-01-16T15:23:48.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.953+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a980675d0 con 0x7f3a9813e360 2024-01-16T15:23:48.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.954+0000 7f3a8e000640 1 -- 172.21.15.141:0/1372556523 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3a9003b050 con 0x7f3a9813e360 2024-01-16T15:23:48.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.955+0000 7f3a8e000640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7006c5e0 0x7f3a7006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:48.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.955+0000 7f3a8e000640 1 -- 172.21.15.141:0/1372556523 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3a900aab30 con 0x7f3a9813e360 2024-01-16T15:23:48.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.955+0000 7f3a95e00640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7006c5e0 0x7f3a7006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:48.953 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.956+0000 7f3a95e00640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7006c5e0 0x7f3a7006eaa0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800a750 tx=0x7f3a88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:48.955 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:48.959+0000 7f3a8e000640 1 -- 172.21.15.141:0/1372556523 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a90077450 con 0x7f3a9813e360 2024-01-16T15:23:49.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.149+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a9806df00 con 0x7f3a9813e360 2024-01-16T15:23:49.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.150+0000 7f3a8e000640 1 -- 172.21.15.141:0/1372556523 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a900390a0 con 0x7f3a9813e360 2024-01-16T15:23:49.146 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T15:23:49.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7006c5e0 msgr2=0x7f3a7006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:49.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7006c5e0 0x7f3a7006eaa0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800a750 tx=0x7f3a88008040 comp rx=0 tx=0).stop 2024-01-16T15:23:49.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a9813e360 msgr2=0x7f3a980b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:49.151 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a9813e360 0x7f3a980b2cf0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009d90 tx=0x7f3a90004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 shutdown_connections 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3a7006c5e0 0x7f3a7006eaa0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3a9813e360 0x7f3a980b2cf0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3a9813df80 0x7f3a981e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 --2- 172.21.15.141:0/1372556523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a980cf9a0 0x7f3a980add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 >> 172.21.15.141:0/1372556523 conn(0x7f3a9806a800 msgr2=0x7f3a980af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.152+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 shutdown_connections 2024-01-16T15:23:49.152 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.153+0000 7f3a9d200640 1 -- 172.21.15.141:0/1372556523 wait complete. 2024-01-16T15:23:49.167 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:49.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.375+0000 7fc31f000640 1 Processor -- start 2024-01-16T15:23:49.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.375+0000 7fc31f000640 1 -- start start 2024-01-16T15:23:49.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31f000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fc31813cfa0 0x7fc31815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31f000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc31813fcb0 0x7fc3180b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31f000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3180cdec0 con 0x7fc3180cd530 2024-01-16T15:23:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31f000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180ce1a0 con 0x7fc31813fcb0 2024-01-16T15:23:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31f000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180ce480 con 0x7fc31813cfa0 2024-01-16T15:23:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31e600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc31813fcb0 0x7fc3180b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31e600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc31813fcb0 0x7fc3180b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46400/0 (socket says 172.21.15.141:46400) 2024-01-16T15:23:49.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31e600640 1 -- 172.21.15.141:0/211912399 learned_addr learned my addr 172.21.15.141:0/211912399 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:49.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.377+0000 7fc31dc00640 1 --2- 172.21.15.141:0/211912399 >> v2:172.21.15.188:3300/0 conn(0x7fc31813cfa0 0x7fc31815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:49.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.378+0000 7fc31e600640 1 -- 172.21.15.141:0/211912399 >> v2:172.21.15.188:3300/0 conn(0x7fc31813cfa0 msgr2=0x7fc31815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:49.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.378+0000 7fc31e600640 1 --2- 172.21.15.141:0/211912399 >> v2:172.21.15.188:3300/0 conn(0x7fc31813cfa0 0x7fc31815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.378+0000 7fc31e600640 1 -- 172.21.15.141:0/211912399 >> v1:172.21.15.188:6789/0 conn(0x7fc3180cd530 legacy=0x7fc31813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:49.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.378+0000 7fc31e600640 1 -- 172.21.15.141:0/211912399 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3180ce760 con 0x7fc31813fcb0 2024-01-16T15:23:49.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.379+0000 7fc31e600640 1 --2- 172.21.15.141:0/211912399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813fcb0 0x7fc3180b2040 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fc31806cf00 tx=0x7fc31002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a009d2ccc9b845f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.379+0000 7fc317e00640 1 -- 172.21.15.141:0/211912399 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31002c9f0 con 0x7fc31813fcb0 2024-01-16T15:23:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.379+0000 7fc317e00640 1 -- 172.21.15.141:0/211912399 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31002cb50 con 0x7fc31813fcb0 2024-01-16T15:23:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.379+0000 7fc317e00640 1 -- 172.21.15.141:0/211912399 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100343f0 con 0x7fc31813fcb0 2024-01-16T15:23:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 -- 172.21.15.141:0/211912399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813fcb0 msgr2=0x7fc3180b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 --2- 172.21.15.141:0/211912399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813fcb0 0x7fc3180b2040 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fc31806cf00 tx=0x7fc31002be50 comp rx=0 tx=0).stop 2024-01-16T15:23:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 -- 172.21.15.141:0/211912399 shutdown_connections 2024-01-16T15:23:49.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 --2- 172.21.15.141:0/211912399 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813fcb0 0x7fc3180b2040 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 --2- 172.21.15.141:0/211912399 >> v2:172.21.15.188:3300/0 conn(0x7fc31813cfa0 0x7fc31815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 -- 172.21.15.141:0/211912399 >> 172.21.15.141:0/211912399 conn(0x7fc31806a800 msgr2=0x7fc31806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 -- 172.21.15.141:0/211912399 shutdown_connections 2024-01-16T15:23:49.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.380+0000 7fc31f000640 1 -- 172.21.15.141:0/211912399 wait complete. 2024-01-16T15:23:49.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.381+0000 7fc31f000640 1 Processor -- start 2024-01-16T15:23:49.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.382+0000 7fc31f000640 1 -- start start 2024-01-16T15:23:49.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31f000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3180cd530 0x7fc3180ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31f000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813cfa0 0x7fc3181f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31f000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc31813fcb0 0x7fc31813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31f000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180adb30 con 0x7fc31813cfa0 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31f000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180ade10 con 0x7fc31813fcb0 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31f000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180ae0f0 con 0x7fc3180cd530 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813cfa0 0x7fc3181f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813cfa0 0x7fc3181f2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46408/0 (socket says 172.21.15.141:46408) 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 -- 172.21.15.141:0/3652114314 learned_addr learned my addr 172.21.15.141:0/3652114314 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:49.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31e600640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc31813fcb0 0x7fc31813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 -- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3180cd530 msgr2=0x7fc3180ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3180cd530 0x7fc3180ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 -- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc31813fcb0 msgr2=0x7fc31813dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc31813fcb0 0x7fc31813dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.383+0000 7fc31d200640 1 -- 172.21.15.141:0/3652114314 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc310009b30 con 0x7fc31813cfa0 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.384+0000 7fc31d200640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813cfa0 0x7fc3181f2a00 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c430 tx=0x7fc30000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.384+0000 7fc315600640 1 -- 172.21.15.141:0/3652114314 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300011790 con 0x7fc31813cfa0 2024-01-16T15:23:49.382 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.384+0000 7fc315600640 1 -- 172.21.15.141:0/3652114314 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3000118f0 con 0x7fc31813cfa0 2024-01-16T15:23:49.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.384+0000 7fc315600640 1 -- 172.21.15.141:0/3652114314 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300011ce0 con 0x7fc31813cfa0 2024-01-16T15:23:49.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.384+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3180ae3d0 con 0x7fc31813cfa0 2024-01-16T15:23:49.383 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.384+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3181f2f40 con 0x7fc31813cfa0 2024-01-16T15:23:49.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.385+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc318067450 con 0x7fc31813cfa0 2024-01-16T15:23:49.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.386+0000 7fc315600640 1 -- 172.21.15.141:0/3652114314 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fc30001c460 con 0x7fc31813cfa0 2024-01-16T15:23:49.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.387+0000 7fc315600640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2f006c5e0 0x7fc2f006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:49.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.387+0000 7fc315600640 1 -- 172.21.15.141:0/3652114314 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc300088b40 con 0x7fc31813cfa0 2024-01-16T15:23:49.384 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.387+0000 7fc31dc00640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2f006c5e0 0x7fc2f006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:49.385 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.388+0000 7fc31dc00640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2f006c5e0 0x7fc2f006eaa0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc3080098f0 tx=0x7fc308008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:49.387 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.391+0000 7fc315600640 1 -- 172.21.15.141:0/3652114314 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3000550c0 con 0x7fc31813cfa0 2024-01-16T15:23:49.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.578+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc3180cbff0 con 0x7fc31813cfa0 2024-01-16T15:23:49.575 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.578+0000 7fc315600640 1 -- 172.21.15.141:0/3652114314 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc3000586e0 con 0x7fc31813cfa0 2024-01-16T15:23:49.576 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T15:23:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.580+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2f006c5e0 msgr2=0x7fc2f006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.580+0000 7fc31f000640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2f006c5e0 0x7fc2f006eaa0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc3080098f0 tx=0x7fc308008040 comp rx=0 tx=0).stop 2024-01-16T15:23:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.580+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813cfa0 msgr2=0x7fc3181f2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.580+0000 7fc31f000640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813cfa0 0x7fc3181f2a00 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c430 tx=0x7fc30000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 shutdown_connections 2024-01-16T15:23:49.578 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc2f006c5e0 0x7fc2f006eaa0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc31813fcb0 0x7fc31813dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc31813cfa0 0x7fc3181f2a00 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 --2- 172.21.15.141:0/3652114314 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc3180cd530 0x7fc3180ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 >> 172.21.15.141:0/3652114314 conn(0x7fc31806a800 msgr2=0x7fc318153df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 shutdown_connections 2024-01-16T15:23:49.579 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:49.581+0000 7fc31f000640 1 -- 172.21.15.141:0/3652114314 wait complete. 2024-01-16T15:23:49.594 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi141.front.sepia.ceph.com... 2024-01-16T15:23:49.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:49.595 DEBUG:teuthology.orchestra.run.smithi141:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:23:49.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi141.stderr:ceph-fuse[76610]: fuse finished with error 0 and tester_r 0 2024-01-16T15:23:50.274 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:23:50.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:23:50.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:50.275 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:23:50.325 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-16T15:23:50.325 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:50.419 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:23:50.419 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:23:50.420 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:50.477 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:23:50.477 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:50.529 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:50.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.750+0000 7f91fc200640 1 Processor -- start 2024-01-16T15:23:50.748 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.750+0000 7f91fc200640 1 -- start start 2024-01-16T15:23:50.749 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91fc200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f91f40ae5a0 0x7f91f415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:50.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91fc200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f91f40b3ae0 0x7f91f406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:50.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91fc200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f4070c40 con 0x7f91f406d120 2024-01-16T15:23:50.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91fc200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f4071450 con 0x7f91f40ae5a0 2024-01-16T15:23:50.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91fc200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f406c470 con 0x7f91f40b3ae0 2024-01-16T15:23:50.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91f9800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f91f40ae5a0 0x7f91f415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:50.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91f9800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f91f40ae5a0 0x7f91f415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46416/0 (socket says 172.21.15.141:46416) 2024-01-16T15:23:50.750 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.752+0000 7f91f9800640 1 -- 172.21.15.141:0/415503228 learned_addr learned my addr 172.21.15.141:0/415503228 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:50.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.753+0000 7f91f8e00640 1 --2- 172.21.15.141:0/415503228 >> v2:172.21.15.188:3300/0 conn(0x7f91f40b3ae0 0x7f91f406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:50.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.753+0000 7f91f9800640 1 -- 172.21.15.141:0/415503228 >> v2:172.21.15.188:3300/0 conn(0x7f91f40b3ae0 msgr2=0x7f91f406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:50.751 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.753+0000 7f91f9800640 1 --2- 172.21.15.141:0/415503228 >> v2:172.21.15.188:3300/0 conn(0x7f91f40b3ae0 0x7f91f406cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.753+0000 7f91f9800640 1 -- 172.21.15.141:0/415503228 >> v1:172.21.15.188:6789/0 conn(0x7f91f406d120 legacy=0x7f91f40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:50.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.753+0000 7f91f9800640 1 -- 172.21.15.141:0/415503228 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f406c750 con 0x7f91f40ae5a0 2024-01-16T15:23:50.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.754+0000 7f91f9800640 1 --2- 172.21.15.141:0/415503228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40ae5a0 0x7f91f415e430 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f91f40cfea0 tx=0x7f91e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9427b567258abe8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:50.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91f3e00640 1 -- 172.21.15.141:0/415503228 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e4031870 con 0x7f91f40ae5a0 2024-01-16T15:23:50.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91f3e00640 1 -- 172.21.15.141:0/415503228 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e40319d0 con 0x7f91f40ae5a0 2024-01-16T15:23:50.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91f3e00640 1 -- 172.21.15.141:0/415503228 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e4031d60 con 0x7f91f40ae5a0 2024-01-16T15:23:50.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91fc200640 1 -- 172.21.15.141:0/415503228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40ae5a0 msgr2=0x7f91f415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:50.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91fc200640 1 --2- 172.21.15.141:0/415503228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40ae5a0 0x7f91f415e430 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f91f40cfea0 tx=0x7f91e402b910 comp rx=0 tx=0).stop 2024-01-16T15:23:50.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91fc200640 1 -- 172.21.15.141:0/415503228 shutdown_connections 2024-01-16T15:23:50.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91fc200640 1 --2- 172.21.15.141:0/415503228 >> v2:172.21.15.188:3300/0 conn(0x7f91f40b3ae0 0x7f91f406cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91fc200640 1 --2- 172.21.15.141:0/415503228 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40ae5a0 0x7f91f415e430 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.755+0000 7f91fc200640 1 -- 172.21.15.141:0/415503228 >> 172.21.15.141:0/415503228 conn(0x7f91f406a800 msgr2=0x7f91f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:50.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.756+0000 7f91fc200640 1 -- 172.21.15.141:0/415503228 shutdown_connections 2024-01-16T15:23:50.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.756+0000 7f91fc200640 1 -- 172.21.15.141:0/415503228 wait complete. 2024-01-16T15:23:50.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.757+0000 7f91fc200640 1 Processor -- start 2024-01-16T15:23:50.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.757+0000 7f91fc200640 1 -- start start 2024-01-16T15:23:50.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.758+0000 7f91fc200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f406d120 0x7f91f41f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:50.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.758+0000 7f91fc200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f40ae5a0 0x7f91f415c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:50.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.758+0000 7f91fc200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40b3ae0 0x7f91f4158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:50.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.758+0000 7f91fc200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f41f3d90 con 0x7f91f40b3ae0 2024-01-16T15:23:50.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.758+0000 7f91fc200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4157080 con 0x7f91f40ae5a0 2024-01-16T15:23:50.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.758+0000 7f91fc200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4157360 con 0x7f91f406d120 2024-01-16T15:23:50.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40b3ae0 0x7f91f4158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40b3ae0 0x7f91f4158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46418/0 (socket says 172.21.15.141:46418) 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 -- 172.21.15.141:0/3528631669 learned_addr learned my addr 172.21.15.141:0/3528631669 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91f9800640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f406d120 0x7f91f41f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91f8e00640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f40ae5a0 0x7f91f415c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 -- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f406d120 msgr2=0x7f91f41f3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f406d120 0x7f91f41f3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 -- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f40ae5a0 msgr2=0x7f91f415c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:50.758 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f40ae5a0 0x7f91f415c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 -- 172.21.15.141:0/3528631669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e40094d0 con 0x7f91f40b3ae0 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91f8e00640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f40ae5a0 0x7f91f415c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.759+0000 7f91fa200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40b3ae0 0x7f91f4158120 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00c780 tx=0x7f91ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.760+0000 7f91f1600640 1 -- 172.21.15.141:0/3528631669 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec004b60 con 0x7f91f40b3ae0 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.760+0000 7f91f1600640 1 -- 172.21.15.141:0/3528631669 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ec00ce90 con 0x7f91f40b3ae0 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.760+0000 7f91f1600640 1 -- 172.21.15.141:0/3528631669 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec00a850 con 0x7f91f40b3ae0 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.760+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f4157640 con 0x7f91f40b3ae0 2024-01-16T15:23:50.759 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.760+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f4157c00 con 0x7f91f40b3ae0 2024-01-16T15:23:50.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.761+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f4157e20 con 0x7f91f40b3ae0 2024-01-16T15:23:50.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.762+0000 7f91f1600640 1 -- 172.21.15.141:0/3528631669 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f91ec00a9b0 con 0x7f91f40b3ae0 2024-01-16T15:23:50.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.763+0000 7f91f1600640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91cc06c5e0 0x7f91cc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:50.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.763+0000 7f91f1600640 1 -- 172.21.15.141:0/3528631669 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f91ec088dc0 con 0x7f91f40b3ae0 2024-01-16T15:23:50.761 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.764+0000 7f91f9800640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91cc06c5e0 0x7f91cc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:50.763 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.765+0000 7f91f9800640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91cc06c5e0 0x7f91cc06eaa0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f91e4036000 tx=0x7f91e403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:50.765 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.768+0000 7f91f1600640 1 -- 172.21.15.141:0/3528631669 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91ec0552e0 con 0x7f91f40b3ae0 2024-01-16T15:23:50.956 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.958+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91f40675d0 con 0x7f91f40b3ae0 2024-01-16T15:23:50.957 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.960+0000 7f91f1600640 1 -- 172.21.15.141:0/3528631669 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16668 (secure 0 0 0) 0x7f91ec058900 con 0x7f91f40b3ae0 2024-01-16T15:23:50.958 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:50.959 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":118,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:55.853050+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.4","mappings":[{"from":0,"to":5}]},{"pgid":"17.3c","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:50.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.963+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91cc06c5e0 msgr2=0x7f91cc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:50.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.963+0000 7f91fc200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91cc06c5e0 0x7f91cc06eaa0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f91e4036000 tx=0x7f91e403c040 comp rx=0 tx=0).stop 2024-01-16T15:23:50.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.963+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40b3ae0 msgr2=0x7f91f4158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.963+0000 7f91fc200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40b3ae0 0x7f91f4158120 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00c780 tx=0x7f91ec00cc40 comp rx=0 tx=0).stop 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 shutdown_connections 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f91cc06c5e0 0x7f91cc06eaa0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f91f40b3ae0 0x7f91f4158120 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f40ae5a0 0x7f91f415c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 --2- 172.21.15.141:0/3528631669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f91f406d120 0x7f91f41f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 >> 172.21.15.141:0/3528631669 conn(0x7f91f406a800 msgr2=0x7f91f415aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 shutdown_connections 2024-01-16T15:23:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:50.964+0000 7f91fc200640 1 -- 172.21.15.141:0/3528631669 wait complete. 2024-01-16T15:23:50.981 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:36.311732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:37.759492+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:50.981 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:51.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.188+0000 7f41be800640 1 Processor -- start 2024-01-16T15:23:51.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.189+0000 7f41be800640 1 -- start start 2024-01-16T15:23:51.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41be800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f41b815d4b0 0x7f41b80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41be800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f41b80b41e0 0x7f41b80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41be800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80d0320 con 0x7f41b80cf9a0 2024-01-16T15:23:51.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41be800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b80b2730 con 0x7f41b80b41e0 2024-01-16T15:23:51.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41be800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b80d1ec0 con 0x7f41b815d4b0 2024-01-16T15:23:51.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41bde00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f41b80b41e0 0x7f41b80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41bde00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f41b80b41e0 0x7f41b80cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46422/0 (socket says 172.21.15.141:46422) 2024-01-16T15:23:51.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.190+0000 7f41bde00640 1 -- 172.21.15.141:0/3791670949 learned_addr learned my addr 172.21.15.141:0/3791670949 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:51.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.191+0000 7f41bde00640 1 -- 172.21.15.141:0/3791670949 >> v2:172.21.15.188:3300/0 conn(0x7f41b815d4b0 msgr2=0x7f41b80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:51.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.191+0000 7f41bde00640 1 --2- 172.21.15.141:0/3791670949 >> v2:172.21.15.188:3300/0 conn(0x7f41b815d4b0 0x7f41b80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.191+0000 7f41bde00640 1 -- 172.21.15.141:0/3791670949 >> v1:172.21.15.188:6789/0 conn(0x7f41b80cf9a0 legacy=0x7f41b80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:51.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.191+0000 7f41bde00640 1 -- 172.21.15.141:0/3791670949 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b80b29d0 con 0x7f41b80b41e0 2024-01-16T15:23:51.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.192+0000 7f41bde00640 1 --2- 172.21.15.141:0/3791670949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b80b41e0 0x7f41b80cfde0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f41b808b5b0 tx=0x7f41b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a2fd2a96ab36af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:51.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.192+0000 7f41b7e00640 1 -- 172.21.15.141:0/3791670949 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0031870 con 0x7f41b80b41e0 2024-01-16T15:23:51.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.192+0000 7f41b7e00640 1 -- 172.21.15.141:0/3791670949 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b00319d0 con 0x7f41b80b41e0 2024-01-16T15:23:51.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.192+0000 7f41b7e00640 1 -- 172.21.15.141:0/3791670949 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0031d60 con 0x7f41b80b41e0 2024-01-16T15:23:51.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 -- 172.21.15.141:0/3791670949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b80b41e0 msgr2=0x7f41b80cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 --2- 172.21.15.141:0/3791670949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b80b41e0 0x7f41b80cfde0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f41b808b5b0 tx=0x7f41b002b910 comp rx=0 tx=0).stop 2024-01-16T15:23:51.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 -- 172.21.15.141:0/3791670949 shutdown_connections 2024-01-16T15:23:51.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 --2- 172.21.15.141:0/3791670949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b80b41e0 0x7f41b80cfde0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 --2- 172.21.15.141:0/3791670949 >> v2:172.21.15.188:3300/0 conn(0x7f41b815d4b0 0x7f41b80d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 -- 172.21.15.141:0/3791670949 >> 172.21.15.141:0/3791670949 conn(0x7f41b80d3000 msgr2=0x7f41b80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:51.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 -- 172.21.15.141:0/3791670949 shutdown_connections 2024-01-16T15:23:51.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.193+0000 7f41be800640 1 -- 172.21.15.141:0/3791670949 wait complete. 2024-01-16T15:23:51.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.194+0000 7f41be800640 1 Processor -- start 2024-01-16T15:23:51.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.195+0000 7f41be800640 1 -- start start 2024-01-16T15:23:51.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.195+0000 7f41be800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b80b41e0 0x7f41b814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41be800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b80cf9a0 0x7f41b814b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41be800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b815d4b0 0x7f41b814b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41be800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b814b000 con 0x7f41b815d4b0 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41be800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8149f60 con 0x7f41b80b41e0 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41be800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b814a240 con 0x7f41b80cf9a0 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b815d4b0 0x7f41b814b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b815d4b0 0x7f41b814b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46438/0 (socket says 172.21.15.141:46438) 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 -- 172.21.15.141:0/293937342 learned_addr learned my addr 172.21.15.141:0/293937342 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 -- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b80cf9a0 msgr2=0x7f41b814b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b80cf9a0 0x7f41b814b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 -- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b80b41e0 msgr2=0x7f41b814ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.195 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b80b41e0 0x7f41b814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.196+0000 7f41bde00640 1 -- 172.21.15.141:0/293937342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b00094d0 con 0x7f41b815d4b0 2024-01-16T15:23:51.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.197+0000 7f41bde00640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b815d4b0 0x7f41b814b750 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f41b002b950 tx=0x7f41b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:51.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.197+0000 7f41b5600640 1 -- 172.21.15.141:0/293937342 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b003adf0 con 0x7f41b815d4b0 2024-01-16T15:23:51.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.197+0000 7f41b5600640 1 -- 172.21.15.141:0/293937342 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b00443f0 con 0x7f41b815d4b0 2024-01-16T15:23:51.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.197+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b815da10 con 0x7f41b815d4b0 2024-01-16T15:23:51.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.197+0000 7f41b5600640 1 -- 172.21.15.141:0/293937342 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b00447a0 con 0x7f41b815d4b0 2024-01-16T15:23:51.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.197+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b814a800 con 0x7f41b815d4b0 2024-01-16T15:23:51.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.198+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b804ed60 con 0x7f41b815d4b0 2024-01-16T15:23:51.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.199+0000 7f41b5600640 1 -- 172.21.15.141:0/293937342 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f41b0044900 con 0x7f41b815d4b0 2024-01-16T15:23:51.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.200+0000 7f41b5600640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f419006c5e0 0x7f419006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.200+0000 7f41b5600640 1 -- 172.21.15.141:0/293937342 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f41b002d040 con 0x7f41b815d4b0 2024-01-16T15:23:51.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.200+0000 7f41bd400640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f419006c5e0 0x7f419006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.202+0000 7f41bd400640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f419006c5e0 0x7f419006eaa0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f41a800ab90 tx=0x7f41a8009250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:51.202 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.204+0000 7f41b5600640 1 -- 172.21.15.141:0/293937342 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41b00779d0 con 0x7f41b815d4b0 2024-01-16T15:23:51.390 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.392+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41b8067450 con 0x7f41b815d4b0 2024-01-16T15:23:51.391 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.394+0000 7f41b5600640 1 -- 172.21.15.141:0/293937342 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f41b007aff0 con 0x7f41b815d4b0 2024-01-16T15:23:51.391 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:23:51.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f419006c5e0 msgr2=0x7f419006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.393 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f419006c5e0 0x7f419006eaa0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f41a800ab90 tx=0x7f41a8009250 comp rx=0 tx=0).stop 2024-01-16T15:23:51.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b815d4b0 msgr2=0x7f41b814b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b815d4b0 0x7f41b814b750 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f41b002b950 tx=0x7f41b00046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 shutdown_connections 2024-01-16T15:23:51.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f419006c5e0 0x7f419006eaa0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b815d4b0 0x7f41b814b750 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f41b80cf9a0 0x7f41b814b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.394 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 --2- 172.21.15.141:0/293937342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41b80b41e0 0x7f41b814ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 >> 172.21.15.141:0/293937342 conn(0x7f41b80d3000 msgr2=0x7f41b80cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:51.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 shutdown_connections 2024-01-16T15:23:51.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.396+0000 7f41be800640 1 -- 172.21.15.141:0/293937342 wait complete. 2024-01-16T15:23:51.411 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:51.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.623+0000 7fd169600640 1 Processor -- start 2024-01-16T15:23:51.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.623+0000 7fd169600640 1 -- start start 2024-01-16T15:23:51.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd169600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd16413df80 0x7fd16415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.622 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd169600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16413e360 0x7fd1640cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd169600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd16415f9c0 0x7fd1640cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd169600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16406c470 con 0x7fd16415f9c0 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd169600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16406c750 con 0x7fd16413df80 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd169600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16406ca30 con 0x7fd16413e360 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd163400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd16415f9c0 0x7fd1640cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd163400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd16415f9c0 0x7fd1640cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46452/0 (socket says 172.21.15.141:46452) 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd163400640 1 -- 172.21.15.141:0/1197770341 learned_addr learned my addr 172.21.15.141:0/1197770341 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:51.623 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.625+0000 7fd162a00640 1 --2- 172.21.15.141:0/1197770341 >> v2:172.21.15.188:3300/0 conn(0x7fd16413df80 0x7fd16415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.624 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.626+0000 7fd162000640 1 --2- 172.21.15.141:0/1197770341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16413e360 0x7fd1640cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.626+0000 7fd163400640 1 -- 172.21.15.141:0/1197770341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16413e360 msgr2=0x7fd1640cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:51.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.626+0000 7fd163400640 1 --2- 172.21.15.141:0/1197770341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16413e360 0x7fd1640cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.626+0000 7fd163400640 1 -- 172.21.15.141:0/1197770341 >> v2:172.21.15.188:3300/0 conn(0x7fd16413df80 msgr2=0x7fd16415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.626+0000 7fd163400640 1 --2- 172.21.15.141:0/1197770341 >> v2:172.21.15.188:3300/0 conn(0x7fd16413df80 0x7fd16415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.626+0000 7fd163400640 1 -- 172.21.15.141:0/1197770341 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16406cd10 con 0x7fd16415f9c0 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.626+0000 7fd162a00640 1 --2- 172.21.15.141:0/1197770341 >> v2:172.21.15.188:3300/0 conn(0x7fd16413df80 0x7fd16415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.627+0000 7fd163400640 1 --2- 172.21.15.141:0/1197770341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16415f9c0 0x7fd1640cf7d0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fd164072770 tx=0x7fd15c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f7d0a36966b46b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.627+0000 7fd161600640 1 -- 172.21.15.141:0/1197770341 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c02ca70 con 0x7fd16415f9c0 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.627+0000 7fd161600640 1 -- 172.21.15.141:0/1197770341 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c02cbd0 con 0x7fd16415f9c0 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.627+0000 7fd161600640 1 -- 172.21.15.141:0/1197770341 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0343f0 con 0x7fd16415f9c0 2024-01-16T15:23:51.626 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 -- 172.21.15.141:0/1197770341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16415f9c0 msgr2=0x7fd1640cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 --2- 172.21.15.141:0/1197770341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16415f9c0 0x7fd1640cf7d0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fd164072770 tx=0x7fd15c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 -- 172.21.15.141:0/1197770341 shutdown_connections 2024-01-16T15:23:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 --2- 172.21.15.141:0/1197770341 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16415f9c0 0x7fd1640cf7d0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 --2- 172.21.15.141:0/1197770341 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16413e360 0x7fd1640cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 --2- 172.21.15.141:0/1197770341 >> v2:172.21.15.188:3300/0 conn(0x7fd16413df80 0x7fd16415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 -- 172.21.15.141:0/1197770341 >> 172.21.15.141:0/1197770341 conn(0x7fd16406a800 msgr2=0x7fd16406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:51.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 -- 172.21.15.141:0/1197770341 shutdown_connections 2024-01-16T15:23:51.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.628+0000 7fd169600640 1 -- 172.21.15.141:0/1197770341 wait complete. 2024-01-16T15:23:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.630+0000 7fd169600640 1 Processor -- start 2024-01-16T15:23:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.630+0000 7fd169600640 1 -- start start 2024-01-16T15:23:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd169600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd16413df80 0x7fd16413d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd169600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16413e360 0x7fd16415e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd169600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16415f9c0 0x7fd1640b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd169600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16413d600 con 0x7fd16413e360 2024-01-16T15:23:51.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd169600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16415c8f0 con 0x7fd16413df80 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd169600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16415cbd0 con 0x7fd16415f9c0 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd162000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16413e360 0x7fd16415e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd162000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16413e360 0x7fd16415e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46460/0 (socket says 172.21.15.141:46460) 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.631+0000 7fd162000640 1 -- 172.21.15.141:0/2395091247 learned_addr learned my addr 172.21.15.141:0/2395091247 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162a00640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd16413df80 0x7fd16413d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd163400640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16415f9c0 0x7fd1640b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162000640 1 -- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16415f9c0 msgr2=0x7fd1640b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162000640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16415f9c0 0x7fd1640b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162000640 1 -- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd16413df80 msgr2=0x7fd16413d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162000640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd16413df80 0x7fd16413d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162000640 1 -- 172.21.15.141:0/2395091247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c009b30 con 0x7fd16413e360 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162a00640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd16413df80 0x7fd16413d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd162000640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16413e360 0x7fd16415e330 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00d800 tx=0x7fd14c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd15aa00640 1 -- 172.21.15.141:0/2395091247 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c00b6c0 con 0x7fd16413e360 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd15aa00640 1 -- 172.21.15.141:0/2395091247 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c00b840 con 0x7fd16413e360 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16415ceb0 con 0x7fd16413e360 2024-01-16T15:23:51.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd15aa00640 1 -- 172.21.15.141:0/2395091247 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c00bbf0 con 0x7fd16413e360 2024-01-16T15:23:51.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.632+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16415d470 con 0x7fd16413e360 2024-01-16T15:23:51.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.633+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1640ce630 con 0x7fd16413e360 2024-01-16T15:23:51.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.634+0000 7fd15aa00640 1 -- 172.21.15.141:0/2395091247 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd14c00bd50 con 0x7fd16413e360 2024-01-16T15:23:51.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.635+0000 7fd15aa00640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd13c06c5e0 0x7fd13c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:51.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.635+0000 7fd15aa00640 1 -- 172.21.15.141:0/2395091247 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd14c089110 con 0x7fd16413e360 2024-01-16T15:23:51.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.636+0000 7fd162a00640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd13c06c5e0 0x7fd13c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:51.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.637+0000 7fd162a00640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd13c06c5e0 0x7fd13c06eaa0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd154006fd0 tx=0x7fd154008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:51.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.640+0000 7fd15aa00640 1 -- 172.21.15.141:0/2395091247 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd14c055630 con 0x7fd16413e360 2024-01-16T15:23:51.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.828+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1640675d0 con 0x7fd16413e360 2024-01-16T15:23:51.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.829+0000 7fd15aa00640 1 -- 172.21.15.141:0/2395091247 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd14c058c50 con 0x7fd16413e360 2024-01-16T15:23:51.826 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T15:23:51.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd13c06c5e0 msgr2=0x7fd13c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd13c06c5e0 0x7fd13c06eaa0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd154006fd0 tx=0x7fd154008040 comp rx=0 tx=0).stop 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16413e360 msgr2=0x7fd16415e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16413e360 0x7fd16415e330 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00d800 tx=0x7fd14c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 shutdown_connections 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd13c06c5e0 0x7fd13c06eaa0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd16415f9c0 0x7fd1640b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd16413e360 0x7fd16415e330 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 --2- 172.21.15.141:0/2395091247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd16413df80 0x7fd16413d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:51.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 >> 172.21.15.141:0/2395091247 conn(0x7fd16406a800 msgr2=0x7fd16406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:51.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 shutdown_connections 2024-01-16T15:23:51.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:51.832+0000 7fd169600640 1 -- 172.21.15.141:0/2395091247 wait complete. 2024-01-16T15:23:51.848 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:52.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.063+0000 7fd799800640 1 Processor -- start 2024-01-16T15:23:52.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.063+0000 7fd799800640 1 -- start start 2024-01-16T15:23:52.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.065+0000 7fd799800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd7940cf9a0 0x7fd794166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.065+0000 7fd799800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79415c5f0 con 0x7fd79413e360 2024-01-16T15:23:52.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.065+0000 7fd799800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79406c470 con 0x7fd79413df80 2024-01-16T15:23:52.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.065+0000 7fd799800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd79406c750 con 0x7fd7940cf9a0 2024-01-16T15:23:52.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.065+0000 7fd792000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd79413e360 0x7fd7940cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47322/0 (socket says 172.21.15.141:47322) 2024-01-16T15:23:52.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.065+0000 7fd792000640 1 -- 172.21.15.141:0/763745965 learned_addr learned my addr 172.21.15.141:0/763745965 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:52.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd793400640 1 --2- 172.21.15.141:0/763745965 >> v2:172.21.15.188:3300/0 conn(0x7fd7940cf9a0 0x7fd794166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342100451 0 0) 0x7fd79415c5f0 con 0x7fd79413e360 2024-01-16T15:23:52.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd774003520 con 0x7fd79413e360 2024-01-16T15:23:52.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096847526 0 0) 0x7fd774003520 con 0x7fd79413e360 2024-01-16T15:23:52.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 >> v2:172.21.15.188:3300/0 conn(0x7fd7940cf9a0 msgr2=0x7fd794166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:52.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 --2- 172.21.15.141:0/763745965 >> v2:172.21.15.188:3300/0 conn(0x7fd7940cf9a0 0x7fd794166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 >> v1:172.21.15.188:6789/0 conn(0x7fd79413df80 legacy=0x7fd79415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:52.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79406ca30 con 0x7fd79413e360 2024-01-16T15:23:52.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd77c0019f0 con 0x7fd79413e360 2024-01-16T15:23:52.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 >> v1:172.21.15.141:6789/0 conn(0x7fd79413e360 legacy=0x7fd7940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd774003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.066+0000 7fd791600640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7740047c0 0x7fd774004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd791600640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd774005180 0x7fd77400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79406c750 con 0x7fd774003970 2024-01-16T15:23:52.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77400c020 con 0x7fd774005180 2024-01-16T15:23:52.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77400c1e0 con 0x7fd7740047c0 2024-01-16T15:23:52.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd792a00640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd774003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd792000640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7740047c0 0x7fd774004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd792a00640 1 -- 172.21.15.141:0/763745965 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7740047c0 msgr2=0x7fd774004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:52.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd792a00640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7740047c0 0x7fd774004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd792a00640 1 -- 172.21.15.141:0/763745965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd774005180 msgr2=0x7fd77400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:52.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd792a00640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd774005180 0x7fd77400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd792a00640 1 -- 172.21.15.141:0/763745965 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd774003520 con 0x7fd774003970 2024-01-16T15:23:52.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.067+0000 7fd793400640 1 -- 172.21.15.141:0/763745965 reap_dead start 2024-01-16T15:23:52.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.068+0000 7fd792a00640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd774003d90 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fd784001750 tx=0x7fd7740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b74b9232a93ec8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:52.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7840052d0 con 0x7fd774003970 2024-01-16T15:23:52.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78402e050 con 0x7fd774003970 2024-01-16T15:23:52.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd791600640 1 -- 172.21.15.141:0/763745965 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784003830 con 0x7fd774003970 2024-01-16T15:23:52.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd799800640 1 -- 172.21.15.141:0/763745965 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 msgr2=0x7fd774003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd799800640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd774003d90 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fd784001750 tx=0x7fd7740057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd799800640 1 -- 172.21.15.141:0/763745965 shutdown_connections 2024-01-16T15:23:52.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd799800640 1 --2- 172.21.15.141:0/763745965 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd774003d90 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.069+0000 7fd799800640 1 -- 172.21.15.141:0/763745965 >> 172.21.15.141:0/763745965 conn(0x7fd79406a800 msgr2=0x7fd79406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:52.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.070+0000 7fd799800640 1 -- 172.21.15.141:0/763745965 shutdown_connections 2024-01-16T15:23:52.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.070+0000 7fd799800640 1 -- 172.21.15.141:0/763745965 wait complete. 2024-01-16T15:23:52.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.071+0000 7fd799800640 1 Processor -- start 2024-01-16T15:23:52.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.072+0000 7fd799800640 1 -- start start 2024-01-16T15:23:52.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.072+0000 7fd799800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd794154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd799800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7940706a0 0x7fd79413d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd799800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7941ee920 0x7fd7940b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.085 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd799800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79414ae40 con 0x7fd774003970 2024-01-16T15:23:52.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd799800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd794072350 con 0x7fd7941ee920 2024-01-16T15:23:52.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd799800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940716b0 con 0x7fd7940706a0 2024-01-16T15:23:52.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd794154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.086 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd794154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46486/0 (socket says 172.21.15.141:46486) 2024-01-16T15:23:52.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 -- 172.21.15.141:0/2964616337 learned_addr learned my addr 172.21.15.141:0/2964616337 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:52.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 -- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7940706a0 msgr2=0x7fd79413d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7940706a0 0x7fd79413d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.087 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 -- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7941ee920 msgr2=0x7fd7940b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7941ee920 0x7fd7940b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 -- 172.21.15.141:0/2964616337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd794154fb0 con 0x7fd774003970 2024-01-16T15:23:52.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.073+0000 7fd792a00640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd794154a70 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fd78402bc30 tx=0x7fd784001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:52.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.074+0000 7fd78aa00640 1 -- 172.21.15.141:0/2964616337 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784007a30 con 0x7fd774003970 2024-01-16T15:23:52.088 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.074+0000 7fd78aa00640 1 -- 172.21.15.141:0/2964616337 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd784007640 con 0x7fd774003970 2024-01-16T15:23:52.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.074+0000 7fd78aa00640 1 -- 172.21.15.141:0/2964616337 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784006c20 con 0x7fd774003970 2024-01-16T15:23:52.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.074+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd794070b30 con 0x7fd774003970 2024-01-16T15:23:52.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.074+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7940710f0 con 0x7fd774003970 2024-01-16T15:23:52.089 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.075+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7940675d0 con 0x7fd774003970 2024-01-16T15:23:52.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.076+0000 7fd78aa00640 1 -- 172.21.15.141:0/2964616337 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd7840035f0 con 0x7fd774003970 2024-01-16T15:23:52.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.077+0000 7fd78aa00640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd76c06c5d0 0x7fd76c06ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.077+0000 7fd78aa00640 1 -- 172.21.15.141:0/2964616337 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7840aa880 con 0x7fd774003970 2024-01-16T15:23:52.090 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.078+0000 7fd792000640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd76c06c5d0 0x7fd76c06ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.079+0000 7fd792000640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd76c06c5d0 0x7fd76c06ea90 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd77c007c40 tx=0x7fd77c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:52.091 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.081+0000 7fd78aa00640 1 -- 172.21.15.141:0/2964616337 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd784076da0 con 0x7fd774003970 2024-01-16T15:23:52.268 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.271+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd79406df00 con 0x7fd774003970 2024-01-16T15:23:52.270 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.272+0000 7fd78aa00640 1 -- 172.21.15.141:0/2964616337 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd78404b020 con 0x7fd774003970 2024-01-16T15:23:52.270 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T15:23:52.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.274+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd76c06c5d0 msgr2=0x7fd76c06ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.274+0000 7fd799800640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd76c06c5d0 0x7fd76c06ea90 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd77c007c40 tx=0x7fd77c009040 comp rx=0 tx=0).stop 2024-01-16T15:23:52.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.274+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 msgr2=0x7fd794154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.274+0000 7fd799800640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd794154a70 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fd78402bc30 tx=0x7fd784001120 comp rx=0 tx=0).stop 2024-01-16T15:23:52.272 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.274+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 shutdown_connections 2024-01-16T15:23:52.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.275+0000 7fd799800640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd76c06c5d0 0x7fd76c06ea90 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.275+0000 7fd799800640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7941ee920 0x7fd7940b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.275+0000 7fd799800640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd7940706a0 0x7fd79413d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.275+0000 7fd799800640 1 --2- 172.21.15.141:0/2964616337 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd774003970 0x7fd794154a70 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.275+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 >> 172.21.15.141:0/2964616337 conn(0x7fd79406a800 msgr2=0x7fd79415d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:52.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.275+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 shutdown_connections 2024-01-16T15:23:52.273 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.275+0000 7fd799800640 1 -- 172.21.15.141:0/2964616337 wait complete. 2024-01-16T15:23:52.290 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:52.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.500+0000 7fbaab400640 1 Processor -- start 2024-01-16T15:23:52.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.501+0000 7fbaab400640 1 -- start start 2024-01-16T15:23:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.502+0000 7fbaab400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fbaa40b3ae0 0x7fbaa415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.502+0000 7fbaab400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa4070c40 con 0x7fbaa40cf290 2024-01-16T15:23:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.502+0000 7fbaab400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa406c470 con 0x7fbaa40ae5a0 2024-01-16T15:23:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.502+0000 7fbaab400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa4071450 con 0x7fbaa40b3ae0 2024-01-16T15:23:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa9400640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fbaa40cf290 0x7fbaa40b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:47334/0 (socket says 172.21.15.141:47334) 2024-01-16T15:23:52.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa9400640 1 -- 172.21.15.141:0/4072145711 learned_addr learned my addr 172.21.15.141:0/4072145711 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:52.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4072145711 >> v2:172.21.15.188:3300/0 conn(0x7fbaa40b3ae0 0x7fbaa415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151206586 0 0) 0x7fbaa4070c40 con 0x7fbaa40cf290 2024-01-16T15:23:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba84003520 con 0x7fbaa40cf290 2024-01-16T15:23:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253094372 0 0) 0x7fba84003520 con 0x7fbaa40cf290 2024-01-16T15:23:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 >> v2:172.21.15.188:3300/0 conn(0x7fbaa40b3ae0 msgr2=0x7fbaa415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa3e00640 1 --2- 172.21.15.141:0/4072145711 >> v2:172.21.15.188:3300/0 conn(0x7fbaa40b3ae0 0x7fbaa415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.503+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 >> v1:172.21.15.188:6789/0 conn(0x7fbaa40ae5a0 legacy=0x7fbaa406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa406c750 con 0x7fbaa40cf290 2024-01-16T15:23:52.504 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fba9c0019f0 con 0x7fbaa40cf290 2024-01-16T15:23:52.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 >> v1:172.21.15.141:6789/0 conn(0x7fbaa40cf290 legacy=0x7fbaa40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba84003970 0x7fba84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fba84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba84005180 0x7fba8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4071450 con 0x7fba840047c0 2024-01-16T15:23:52.505 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8400c030 con 0x7fba84003970 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8400c1f0 con 0x7fba84005180 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fba84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fba9bc00640 1 -- 172.21.15.141:0/4072145711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba84005180 msgr2=0x7fba8400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba84005180 0x7fba8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fba9bc00640 1 -- 172.21.15.141:0/4072145711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba84003970 msgr2=0x7fba84003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba84003970 0x7fba84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fba9bc00640 1 -- 172.21.15.141:0/4072145711 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84003520 con 0x7fba840047c0 2024-01-16T15:23:52.506 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.504+0000 7fbaa9400640 1 -- 172.21.15.141:0/4072145711 reap_dead start 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.505+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fba84004c40 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009ac0 tx=0x7fba840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954665202c1b418f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c02c6b0 con 0x7fba840047c0 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c02c810 con 0x7fba840047c0 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaa3e00640 1 -- 172.21.15.141:0/4072145711 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c02cba0 con 0x7fba840047c0 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaab400640 1 -- 172.21.15.141:0/4072145711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 msgr2=0x7fba84004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaab400640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fba84004c40 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009ac0 tx=0x7fba840057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaab400640 1 -- 172.21.15.141:0/4072145711 shutdown_connections 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaab400640 1 --2- 172.21.15.141:0/4072145711 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fba84004c40 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaab400640 1 -- 172.21.15.141:0/4072145711 >> 172.21.15.141:0/4072145711 conn(0x7fbaa406a800 msgr2=0x7fbaa406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaab400640 1 -- 172.21.15.141:0/4072145711 shutdown_connections 2024-01-16T15:23:52.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.506+0000 7fbaab400640 1 -- 172.21.15.141:0/4072145711 wait complete. 2024-01-16T15:23:52.510 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.508+0000 7fbaab400640 1 Processor -- start 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.508+0000 7fbaab400640 1 -- start start 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaab400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fbaa4155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaab400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaa41566a0 0x7fbaa4157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaab400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbaa41f2b30 0x7fbaa40cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaab400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4157e50 con 0x7fba840047c0 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaab400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4156470 con 0x7fbaa41566a0 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaab400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa415ad60 con 0x7fbaa41f2b30 2024-01-16T15:23:52.511 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fbaa4155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fbaa4155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:46498/0 (socket says 172.21.15.141:46498) 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.509+0000 7fbaa8a00640 1 -- 172.21.15.141:0/4070753913 learned_addr learned my addr 172.21.15.141:0/4070753913 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaa41566a0 0x7fbaa4157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa9400640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbaa41f2b30 0x7fbaa40cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa8a00640 1 -- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbaa41f2b30 msgr2=0x7fbaa40cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa8a00640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbaa41f2b30 0x7fbaa40cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa8a00640 1 -- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaa41566a0 msgr2=0x7fbaa4157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa8a00640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaa41566a0 0x7fbaa4157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.512 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa8a00640 1 -- 172.21.15.141:0/4070753913 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa415b040 con 0x7fba840047c0 2024-01-16T15:23:52.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaa41566a0 0x7fbaa4157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:52.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa8a00640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fbaa4155f30 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fba9400c500 tx=0x7fba9400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:52.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa1600640 1 -- 172.21.15.141:0/4070753913 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba940052d0 con 0x7fba840047c0 2024-01-16T15:23:52.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa1600640 1 -- 172.21.15.141:0/4070753913 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba94005430 con 0x7fba840047c0 2024-01-16T15:23:52.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaa1600640 1 -- 172.21.15.141:0/4070753913 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba94012870 con 0x7fba840047c0 2024-01-16T15:23:52.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.510+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa415b320 con 0x7fba840047c0 2024-01-16T15:23:52.513 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.511+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa4157a40 con 0x7fba840047c0 2024-01-16T15:23:52.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.512+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa4157bd0 con 0x7fba840047c0 2024-01-16T15:23:52.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.513+0000 7fbaa1600640 1 -- 172.21.15.141:0/4070753913 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fba940129d0 con 0x7fba840047c0 2024-01-16T15:23:52.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.514+0000 7fbaa1600640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba7c06c5e0 0x7fba7c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:52.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.514+0000 7fbaa1600640 1 -- 172.21.15.141:0/4070753913 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba940887d0 con 0x7fba840047c0 2024-01-16T15:23:52.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.514+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba7c06c5e0 0x7fba7c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:52.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.515+0000 7fba9bc00640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba7c06c5e0 0x7fba7c06eaa0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fba8c036000 tx=0x7fba8c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:52.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.518+0000 7fbaa1600640 1 -- 172.21.15.141:0/4070753913 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba94054cf0 con 0x7fba840047c0 2024-01-16T15:23:52.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.723+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbaa40675d0 con 0x7fba840047c0 2024-01-16T15:23:52.721 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.723+0000 7fbaa1600640 1 -- 172.21.15.141:0/4070753913 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) v1 ==== 80+0+0 (secure 0 0 0) 0x7fba94058310 con 0x7fba840047c0 2024-01-16T15:23:52.721 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T15:23:52.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.726+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba7c06c5e0 msgr2=0x7fba7c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.726+0000 7fbaab400640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba7c06c5e0 0x7fba7c06eaa0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fba8c036000 tx=0x7fba8c030000 comp rx=0 tx=0).stop 2024-01-16T15:23:52.724 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.726+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 msgr2=0x7fbaa4155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.726+0000 7fbaab400640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fbaa4155f30 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fba9400c500 tx=0x7fba9400c9c0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 shutdown_connections 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba7c06c5e0 0x7fba7c06eaa0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fbaa41f2b30 0x7fbaa40cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaa41566a0 0x7fbaa4157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 --2- 172.21.15.141:0/4070753913 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba840047c0 0x7fbaa4155f30 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 >> 172.21.15.141:0/4070753913 conn(0x7fbaa406a800 msgr2=0x7fbaa406cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:52.725 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 shutdown_connections 2024-01-16T15:23:52.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:52.727+0000 7fbaab400640 1 -- 172.21.15.141:0/4070753913 wait complete. 2024-01-16T15:23:52.743 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:23:52.743 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:52.802 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:53.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.020+0000 7fc210200640 1 Processor -- start 2024-01-16T15:23:53.017 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.020+0000 7fc210200640 1 -- start start 2024-01-16T15:23:53.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.021+0000 7fc210200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc20813e360 0x7fc2080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.019 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc210200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20815c5f0 con 0x7fc20815f9c0 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc210200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20806c470 con 0x7fc20813df80 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc210200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20806c750 con 0x7fc20813e360 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc20ce00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc20813e360 0x7fc2080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc20ce00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc20813e360 0x7fc2080cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48454/0 (socket says 172.21.15.141:48454) 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc20ce00640 1 -- 172.21.15.141:0/2187912987 learned_addr learned my addr 172.21.15.141:0/2187912987 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc20ce00640 1 -- 172.21.15.141:0/2187912987 >> v1:172.21.15.188:6789/0 conn(0x7fc20813df80 legacy=0x7fc20815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc20ce00640 1 -- 172.21.15.141:0/2187912987 >> v1:172.21.15.141:6789/0 conn(0x7fc20815f9c0 legacy=0x7fc2080cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.020 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.022+0000 7fc20ce00640 1 -- 172.21.15.141:0/2187912987 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20806ca30 con 0x7fc20813e360 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.023+0000 7fc20ce00640 1 --2- 172.21.15.141:0/2187912987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813e360 0x7fc2080cf290 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0009870 tx=0x7fc1f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7979b127e2d67269 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc207e00640 1 -- 172.21.15.141:0/2187912987 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0030860 con 0x7fc20813e360 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc207e00640 1 -- 172.21.15.141:0/2187912987 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f00309c0 con 0x7fc20813e360 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc207e00640 1 -- 172.21.15.141:0/2187912987 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0030d50 con 0x7fc20813e360 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc210200640 1 -- 172.21.15.141:0/2187912987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813e360 msgr2=0x7fc2080cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc210200640 1 --2- 172.21.15.141:0/2187912987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813e360 0x7fc2080cf290 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0009870 tx=0x7fc1f002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc210200640 1 -- 172.21.15.141:0/2187912987 shutdown_connections 2024-01-16T15:23:53.022 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc210200640 1 --2- 172.21.15.141:0/2187912987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813e360 0x7fc2080cf290 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc210200640 1 -- 172.21.15.141:0/2187912987 >> 172.21.15.141:0/2187912987 conn(0x7fc20806a800 msgr2=0x7fc20806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:53.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc210200640 1 -- 172.21.15.141:0/2187912987 shutdown_connections 2024-01-16T15:23:53.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.024+0000 7fc210200640 1 -- 172.21.15.141:0/2187912987 wait complete. 2024-01-16T15:23:53.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.026+0000 7fc210200640 1 Processor -- start 2024-01-16T15:23:53.023 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.026+0000 7fc210200640 1 -- start start 2024-01-16T15:23:53.025 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc210200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813df80 0x7fc2081f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc210200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc20813e360 0x7fc2081f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc210200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc20815f9c0 0x7fc2080b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc210200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2081522e0 con 0x7fc20813df80 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc210200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2081525c0 con 0x7fc20815f9c0 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc210200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2081528a0 con 0x7fc20813e360 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc20d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813df80 0x7fc2081f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc20d800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813df80 0x7fc2081f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48470/0 (socket says 172.21.15.141:48470) 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.027+0000 7fc20d800640 1 -- 172.21.15.141:0/2842100090 learned_addr learned my addr 172.21.15.141:0/2842100090 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:53.026 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc20ce00640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc20813e360 0x7fc2081f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc20d800640 1 -- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc20813e360 msgr2=0x7fc2081f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc20d800640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc20813e360 0x7fc2081f5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc20d800640 1 -- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc20815f9c0 msgr2=0x7fc2080b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc20d800640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc20815f9c0 0x7fc2080b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc20d800640 1 -- 172.21.15.141:0/2842100090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00094d0 con 0x7fc20813df80 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc20d800640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813df80 0x7fc2081f3200 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800b3e0 tx=0x7fc1f800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc205600640 1 -- 172.21.15.141:0/2842100090 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8011830 con 0x7fc20813df80 2024-01-16T15:23:53.027 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc205600640 1 -- 172.21.15.141:0/2842100090 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f80119b0 con 0x7fc20813df80 2024-01-16T15:23:53.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc205600640 1 -- 172.21.15.141:0/2842100090 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8011d60 con 0x7fc20813df80 2024-01-16T15:23:53.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc208152b80 con 0x7fc20813df80 2024-01-16T15:23:53.028 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.028+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2081f6400 con 0x7fc20813df80 2024-01-16T15:23:53.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.029+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2081530a0 con 0x7fc20813df80 2024-01-16T15:23:53.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.030+0000 7fc205600640 1 -- 172.21.15.141:0/2842100090 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fc1f801a400 con 0x7fc20813df80 2024-01-16T15:23:53.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.031+0000 7fc205600640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1e006c5e0 0x7fc1e006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.031+0000 7fc205600640 1 -- 172.21.15.141:0/2842100090 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc1f8088ce0 con 0x7fc20813df80 2024-01-16T15:23:53.029 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.032+0000 7fc20ce00640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1e006c5e0 0x7fc1e006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.030 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.033+0000 7fc20ce00640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1e006c5e0 0x7fc1e006eaa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0036000 tx=0x7fc1f003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.033 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.036+0000 7fc205600640 1 -- 172.21.15.141:0/2842100090 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2081530a0 con 0x7fc20813df80 2024-01-16T15:23:53.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.229+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2081530a0 con 0x7fc20813df80 2024-01-16T15:23:53.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.231+0000 7fc205600640 1 -- 172.21.15.141:0/2842100090 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16668 (secure 0 0 0) 0x7fc2081530a0 con 0x7fc20813df80 2024-01-16T15:23:53.229 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:53.229 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":118,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:55.853050+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.4","mappings":[{"from":0,"to":5}]},{"pgid":"17.3c","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:53.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.234+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1e006c5e0 msgr2=0x7fc1e006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.234+0000 7fc210200640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1e006c5e0 0x7fc1e006eaa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0036000 tx=0x7fc1f003b040 comp rx=0 tx=0).stop 2024-01-16T15:23:53.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.234+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813df80 msgr2=0x7fc2081f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.234+0000 7fc210200640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813df80 0x7fc2081f3200 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800b3e0 tx=0x7fc1f800b8a0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.232 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 shutdown_connections 2024-01-16T15:23:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fc1e006c5e0 0x7fc1e006eaa0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc20815f9c0 0x7fc2080b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fc20813e360 0x7fc2081f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 --2- 172.21.15.141:0/2842100090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc20813df80 0x7fc2081f3200 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 >> 172.21.15.141:0/2842100090 conn(0x7fc20806a800 msgr2=0x7fc2080b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 shutdown_connections 2024-01-16T15:23:53.233 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.235+0000 7fc210200640 1 -- 172.21.15.141:0/2842100090 wait complete. 2024-01-16T15:23:53.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:57:00.501425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:15:36.311732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:15:37.759492+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:23:53.251 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:23:53.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.464+0000 7f813ba00640 1 Processor -- start 2024-01-16T15:23:53.461 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.464+0000 7f813ba00640 1 -- start start 2024-01-16T15:23:53.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f813ba00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f81340cf9a0 0x7f813415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f813ba00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8134070c40 con 0x7f81340b3ae0 2024-01-16T15:23:53.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f813ba00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813406c470 con 0x7f81340ae5a0 2024-01-16T15:23:53.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f813ba00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8134071450 con 0x7f81340cf9a0 2024-01-16T15:23:53.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f8139a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f81340cf9a0 0x7f813415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f8133e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f81340b3ae0 0x7f81340cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54278/0 (socket says 172.21.15.141:54278) 2024-01-16T15:23:53.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f8133e00640 1 -- 172.21.15.141:0/3502625234 learned_addr learned my addr 172.21.15.141:0/3502625234 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:53.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.466+0000 7f8139a00640 1 --2- 172.21.15.141:0/3502625234 >> v2:172.21.15.141:3300/0 conn(0x7f81340cf9a0 0x7f813415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48474/0 (socket says 172.21.15.141:48474) 2024-01-16T15:23:53.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.467+0000 7f8139a00640 1 -- 172.21.15.141:0/3502625234 >> v1:172.21.15.188:6789/0 conn(0x7f81340ae5a0 legacy=0x7f813406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:53.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.467+0000 7f8139a00640 1 -- 172.21.15.141:0/3502625234 >> v1:172.21.15.141:6789/0 conn(0x7f81340b3ae0 legacy=0x7f81340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.467+0000 7f8139a00640 1 -- 172.21.15.141:0/3502625234 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813406c750 con 0x7f81340cf9a0 2024-01-16T15:23:53.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.467+0000 7f8133400640 1 -- 172.21.15.141:0/3502625234 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468657620 0 0) 0x7f8134070c40 con 0x7f81340b3ae0 2024-01-16T15:23:53.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.468+0000 7f8139a00640 1 --2- 172.21.15.141:0/3502625234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340cf9a0 0x7f813415e430 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f81240097a0 tx=0x7f812402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa549e5c89a78a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.468+0000 7f8133400640 1 -- 172.21.15.141:0/3502625234 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124030610 con 0x7f81340cf9a0 2024-01-16T15:23:53.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.468+0000 7f8133400640 1 -- 172.21.15.141:0/3502625234 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8124030770 con 0x7f81340cf9a0 2024-01-16T15:23:53.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.468+0000 7f8133400640 1 -- 172.21.15.141:0/3502625234 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124030b00 con 0x7f81340cf9a0 2024-01-16T15:23:53.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.469+0000 7f813ba00640 1 -- 172.21.15.141:0/3502625234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340cf9a0 msgr2=0x7f813415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.469+0000 7f813ba00640 1 --2- 172.21.15.141:0/3502625234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340cf9a0 0x7f813415e430 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f81240097a0 tx=0x7f812402b960 comp rx=0 tx=0).stop 2024-01-16T15:23:53.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.469+0000 7f813ba00640 1 -- 172.21.15.141:0/3502625234 shutdown_connections 2024-01-16T15:23:53.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.469+0000 7f813ba00640 1 --2- 172.21.15.141:0/3502625234 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340cf9a0 0x7f813415e430 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.469+0000 7f813ba00640 1 -- 172.21.15.141:0/3502625234 >> 172.21.15.141:0/3502625234 conn(0x7f813406a800 msgr2=0x7f813406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:53.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.469+0000 7f813ba00640 1 -- 172.21.15.141:0/3502625234 shutdown_connections 2024-01-16T15:23:53.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.469+0000 7f813ba00640 1 -- 172.21.15.141:0/3502625234 wait complete. 2024-01-16T15:23:53.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.471+0000 7f813ba00640 1 Processor -- start 2024-01-16T15:23:53.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.471+0000 7f813ba00640 1 -- start start 2024-01-16T15:23:53.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f813ba00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81340ae5a0 0x7f8134157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f813ba00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340b3ae0 0x7f81341f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f813ba00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81340cf9a0 0x7f81340cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f813ba00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134158160 con 0x7f81340b3ae0 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f813ba00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340cc850 con 0x7f81340cf9a0 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f813ba00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81341f38c0 con 0x7f81340ae5a0 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340b3ae0 0x7f81341f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340b3ae0 0x7f81341f3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48484/0 (socket says 172.21.15.141:48484) 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 -- 172.21.15.141:0/3753571987 learned_addr learned my addr 172.21.15.141:0/3753571987 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 -- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81340ae5a0 msgr2=0x7f8134157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81340ae5a0 0x7f8134157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 -- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81340cf9a0 msgr2=0x7f81340cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81340cf9a0 0x7f81340cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.472+0000 7f8133e00640 1 -- 172.21.15.141:0/3753571987 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8124009400 con 0x7f81340b3ae0 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.473+0000 7f8133e00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340b3ae0 0x7f81341f3380 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f811c00c4d0 tx=0x7f811c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.473+0000 7f8130c00640 1 -- 172.21.15.141:0/3753571987 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0074e0 con 0x7f81340b3ae0 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.473+0000 7f8130c00640 1 -- 172.21.15.141:0/3753571987 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c007660 con 0x7f81340b3ae0 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.473+0000 7f8130c00640 1 -- 172.21.15.141:0/3753571987 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c004a20 con 0x7f81340b3ae0 2024-01-16T15:23:53.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.473+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81340cbc40 con 0x7f81340b3ae0 2024-01-16T15:23:53.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.473+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81340cc200 con 0x7f81340b3ae0 2024-01-16T15:23:53.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.474+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81340675d0 con 0x7f81340b3ae0 2024-01-16T15:23:53.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.475+0000 7f8130c00640 1 -- 172.21.15.141:0/3753571987 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f811c004b80 con 0x7f81340b3ae0 2024-01-16T15:23:53.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.476+0000 7f8130c00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f810c06c5e0 0x7f810c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.476+0000 7f8130c00640 1 -- 172.21.15.141:0/3753571987 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f811c087d70 con 0x7f81340b3ae0 2024-01-16T15:23:53.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.477+0000 7f8139000640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f810c06c5e0 0x7f810c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.478+0000 7f8139000640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f810c06c5e0 0x7f810c06eaa0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8134159710 tx=0x7f812c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.481+0000 7f8130c00640 1 -- 172.21.15.141:0/3753571987 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f811c054290 con 0x7f81340b3ae0 2024-01-16T15:23:53.668 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.671+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f813406b3f0 con 0x7f81340b3ae0 2024-01-16T15:23:53.669 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.672+0000 7f8130c00640 1 -- 172.21.15.141:0/3753571987 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f811c018020 con 0x7f81340b3ae0 2024-01-16T15:23:53.669 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 1 2024-01-16T15:23:53.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.674+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f810c06c5e0 msgr2=0x7f810c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.672 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.674+0000 7f813ba00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f810c06c5e0 0x7f810c06eaa0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8134159710 tx=0x7f812c007d30 comp rx=0 tx=0).stop 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.674+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340b3ae0 msgr2=0x7f81341f3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.674+0000 7f813ba00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340b3ae0 0x7f81341f3380 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f811c00c4d0 tx=0x7f811c00c990 comp rx=0 tx=0).stop 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 shutdown_connections 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f810c06c5e0 0x7f810c06eaa0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81340cf9a0 0x7f81340cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81340b3ae0 0x7f81341f3380 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 --2- 172.21.15.141:0/3753571987 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f81340ae5a0 0x7f8134157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 >> 172.21.15.141:0/3753571987 conn(0x7f813406a800 msgr2=0x7f813415d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:53.673 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 shutdown_connections 2024-01-16T15:23:53.674 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.675+0000 7f813ba00640 1 -- 172.21.15.141:0/3753571987 wait complete. 2024-01-16T15:23:53.690 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:23:53.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.905+0000 7ffada600640 1 Processor -- start 2024-01-16T15:23:53.903 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.905+0000 7ffada600640 1 -- start start 2024-01-16T15:23:53.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.907+0000 7ffada600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ffad415de80 0x7ffad40ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.907+0000 7ffada600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad406c730 con 0x7ffad4071e40 2024-01-16T15:23:53.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.907+0000 7ffada600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40724d0 con 0x7ffad4143810 2024-01-16T15:23:53.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.907+0000 7ffada600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad4143bf0 con 0x7ffad415de80 2024-01-16T15:23:53.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.907+0000 7ffad3e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ffad4071e40 0x7ffad415d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54286/0 (socket says 172.21.15.141:54286) 2024-01-16T15:23:53.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.907+0000 7ffad3e00640 1 -- 172.21.15.141:0/2941097127 learned_addr learned my addr 172.21.15.141:0/2941097127 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:53.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.907+0000 7ffad9c00640 1 --2- 172.21.15.141:0/2941097127 >> v2:172.21.15.141:3300/0 conn(0x7ffad415de80 0x7ffad40ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.908+0000 7ffad3400640 1 -- 172.21.15.141:0/2941097127 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491280736 0 0) 0x7ffad406c730 con 0x7ffad4071e40 2024-01-16T15:23:53.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.908+0000 7ffad3400640 1 -- 172.21.15.141:0/2941097127 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab4003520 con 0x7ffad4071e40 2024-01-16T15:23:53.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.908+0000 7ffad3400640 1 -- 172.21.15.141:0/2941097127 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898544888 0 0) 0x7ffab4003520 con 0x7ffad4071e40 2024-01-16T15:23:53.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.908+0000 7ffad9c00640 1 -- 172.21.15.141:0/2941097127 >> v1:172.21.15.188:6789/0 conn(0x7ffad4143810 legacy=0x7ffad415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.907 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.908+0000 7ffad9c00640 1 -- 172.21.15.141:0/2941097127 >> v1:172.21.15.141:6789/0 conn(0x7ffad4071e40 legacy=0x7ffad415d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.908+0000 7ffad9c00640 1 -- 172.21.15.141:0/2941097127 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad406c9f0 con 0x7ffad415de80 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.909+0000 7ffad9c00640 1 --2- 172.21.15.141:0/2941097127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40ae170 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7ffacc009d90 tx=0x7ffacc02bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a2dccf0213725c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.909+0000 7ffad3400640 1 -- 172.21.15.141:0/2941097127 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc001670 con 0x7ffad415de80 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.909+0000 7ffad3400640 1 -- 172.21.15.141:0/2941097127 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc02c820 con 0x7ffad415de80 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.909+0000 7ffad3400640 1 -- 172.21.15.141:0/2941097127 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc02cbb0 con 0x7ffad415de80 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.910+0000 7ffada600640 1 -- 172.21.15.141:0/2941097127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 msgr2=0x7ffad40ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.910+0000 7ffada600640 1 --2- 172.21.15.141:0/2941097127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40ae170 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7ffacc009d90 tx=0x7ffacc02bcf0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.910+0000 7ffada600640 1 -- 172.21.15.141:0/2941097127 shutdown_connections 2024-01-16T15:23:53.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.910+0000 7ffada600640 1 --2- 172.21.15.141:0/2941097127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40ae170 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.910+0000 7ffada600640 1 -- 172.21.15.141:0/2941097127 >> 172.21.15.141:0/2941097127 conn(0x7ffad406a7d0 msgr2=0x7ffad406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:53.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.910+0000 7ffada600640 1 -- 172.21.15.141:0/2941097127 shutdown_connections 2024-01-16T15:23:53.909 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.910+0000 7ffada600640 1 -- 172.21.15.141:0/2941097127 wait complete. 2024-01-16T15:23:53.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.912+0000 7ffada600640 1 Processor -- start 2024-01-16T15:23:53.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.912+0000 7ffada600640 1 -- start start 2024-01-16T15:23:53.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffada600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad4071e40 0x7ffad40ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffada600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad4143810 0x7ffad41f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffada600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffada600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40722f0 con 0x7ffad415de80 2024-01-16T15:23:53.912 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffada600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40ca9f0 con 0x7ffad4143810 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffada600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad4145280 con 0x7ffad4071e40 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad9c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad9c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48506/0 (socket says 172.21.15.141:48506) 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad9c00640 1 -- 172.21.15.141:0/2071214270 learned_addr learned my addr 172.21.15.141:0/2071214270 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad9200640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad4071e40 0x7ffad40ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad9c00640 1 -- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad4071e40 msgr2=0x7ffad40ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad3e00640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad4143810 0x7ffad41f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad9c00640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad4071e40 0x7ffad40ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.913 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.913+0000 7ffad9c00640 1 -- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad4143810 msgr2=0x7ffad41f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad9c00640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad4143810 0x7ffad41f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad9c00640 1 -- 172.21.15.141:0/2071214270 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc009a70 con 0x7ffad415de80 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad9200640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad4071e40 0x7ffad40ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad3e00640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad4143810 0x7ffad41f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad9c00640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40cb5e0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7ffacc037040 tx=0x7ffacc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad0c00640 1 -- 172.21.15.141:0/2071214270 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc034c00 con 0x7ffad415de80 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad0c00640 1 -- 172.21.15.141:0/2071214270 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc034d60 con 0x7ffad415de80 2024-01-16T15:23:53.914 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad41f33f0 con 0x7ffad415de80 2024-01-16T15:23:53.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffad0c00640 1 -- 172.21.15.141:0/2071214270 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc003bd0 con 0x7ffad415de80 2024-01-16T15:23:53.915 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.914+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad41457b0 con 0x7ffad415de80 2024-01-16T15:23:53.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.916+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad4067450 con 0x7ffad415de80 2024-01-16T15:23:53.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.916+0000 7ffad0c00640 1 -- 172.21.15.141:0/2071214270 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ffacc03b050 con 0x7ffad415de80 2024-01-16T15:23:53.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.917+0000 7ffad0c00640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06c5e0 0x7ffaac06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:53.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.917+0000 7ffad0c00640 1 -- 172.21.15.141:0/2071214270 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffacc0aaa30 con 0x7ffad415de80 2024-01-16T15:23:53.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.918+0000 7ffad9200640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06c5e0 0x7ffaac06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:53.916 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.919+0000 7ffad9200640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06c5e0 0x7ffaac06eaa0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ffac4000f60 tx=0x7ffac4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:53.919 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:53.922+0000 7ffad0c00640 1 -- 172.21.15.141:0/2071214270 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffacc077470 con 0x7ffad415de80 2024-01-16T15:23:54.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.116+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffad406db90 con 0x7ffad415de80 2024-01-16T15:23:54.113 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.116+0000 7ffad0c00640 1 -- 172.21.15.141:0/2071214270 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffacc04d020 con 0x7ffad415de80 2024-01-16T15:23:54.114 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 32 2024-01-16T15:23:54.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06c5e0 msgr2=0x7ffaac06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06c5e0 0x7ffaac06eaa0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ffac4000f60 tx=0x7ffac4009040 comp rx=0 tx=0).stop 2024-01-16T15:23:54.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 msgr2=0x7ffad40cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.117 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40cb5e0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7ffacc037040 tx=0x7ffacc004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 shutdown_connections 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ffaac06c5e0 0x7ffaac06eaa0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffad415de80 0x7ffad40cb5e0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffad4143810 0x7ffad41f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.119+0000 7ffada600640 1 --2- 172.21.15.141:0/2071214270 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ffad4071e40 0x7ffad40ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.120+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 >> 172.21.15.141:0/2071214270 conn(0x7ffad406a7d0 msgr2=0x7ffad406abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.120+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 shutdown_connections 2024-01-16T15:23:54.118 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.120+0000 7ffada600640 1 -- 172.21.15.141:0/2071214270 wait complete. 2024-01-16T15:23:54.136 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:23:54.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.352+0000 7f1fb2200640 1 Processor -- start 2024-01-16T15:23:54.349 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.352+0000 7f1fb2200640 1 -- start start 2024-01-16T15:23:54.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.353+0000 7f1fb2200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 0x7f1fac15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1fb2200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f1fac13e360 0x7f1fac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1fb2200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac06c470 con 0x7f1fac15f9c0 2024-01-16T15:23:54.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1fb2200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fac06c750 con 0x7f1fac13e360 2024-01-16T15:23:54.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1fb2200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac06ca30 con 0x7f1fac13df80 2024-01-16T15:23:54.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1fabe00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1fac15f9c0 0x7f1fac1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54302/0 (socket says 172.21.15.141:54302) 2024-01-16T15:23:54.351 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1fabe00640 1 -- 172.21.15.141:0/1271343682 learned_addr learned my addr 172.21.15.141:0/1271343682 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:54.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1fab400640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 0x7f1fac15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1faaa00640 1 --2- 172.21.15.141:0/1271343682 >> v2:172.21.15.188:3300/0 conn(0x7f1fac13e360 0x7f1fac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.354 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.354+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271601514 0 0) 0x7f1fac06c470 con 0x7f1fac15f9c0 2024-01-16T15:23:54.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f8c003520 con 0x7f1fac15f9c0 2024-01-16T15:23:54.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2216564664 0 0) 0x7f1f8c003520 con 0x7f1fac15f9c0 2024-01-16T15:23:54.355 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 msgr2=0x7f1fac15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 0x7f1fac15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 >> v2:172.21.15.188:3300/0 conn(0x7f1fac13e360 msgr2=0x7f1fac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 --2- 172.21.15.141:0/1271343682 >> v2:172.21.15.188:3300/0 conn(0x7f1fac13e360 0x7f1fac0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac06cd10 con 0x7f1fac15f9c0 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f1fa40019f0 con 0x7f1fac15f9c0 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 >> v1:172.21.15.141:6789/0 conn(0x7f1fac15f9c0 legacy=0x7f1fac1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c003970 0x7f1f8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.356 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 0x7f1f8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f8c004d00 0x7f1f8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac06c750 con 0x7f1f8c004d00 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac06ca30 con 0x7f1f8c003970 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c00b980 con 0x7f1fac13df80 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.355+0000 7f1fabe00640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f8c004d00 0x7f1f8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.356+0000 7f1fab400640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c003970 0x7f1f8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.356+0000 7f1fabe00640 1 -- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 msgr2=0x7f1f8c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:54.357 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.356+0000 7f1faaa00640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 0x7f1f8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.356+0000 7f1fabe00640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac13df80 0x7f1f8c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.356+0000 7f1fabe00640 1 -- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c003970 msgr2=0x7f1f8c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.356+0000 7f1fabe00640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c003970 0x7f1f8c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.356+0000 7f1fabe00640 1 -- 172.21.15.141:0/1271343682 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c003520 con 0x7f1f8c004d00 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.357+0000 7f1fabe00640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f8c004d00 0x7f1f8c005180 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400a8d0 tx=0x7f1f8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80009a834f4001b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.357+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4034630 con 0x7f1f8c004d00 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.357+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa402f070 con 0x7f1f8c004d00 2024-01-16T15:23:54.358 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.357+0000 7f1faa000640 1 -- 172.21.15.141:0/1271343682 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40349d0 con 0x7f1f8c004d00 2024-01-16T15:23:54.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.357+0000 7f1fb2200640 1 -- 172.21.15.141:0/1271343682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f8c004d00 msgr2=0x7f1f8c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.358+0000 7f1fb2200640 1 --2- 172.21.15.141:0/1271343682 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f8c004d00 0x7f1f8c005180 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400a8d0 tx=0x7f1f8c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.358+0000 7f1fabe00640 1 -- 172.21.15.141:0/1271343682 reap_dead start 2024-01-16T15:23:54.361 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.358+0000 7f1fb2200640 1 -- 172.21.15.141:0/1271343682 shutdown_connections 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.358+0000 7f1fb2200640 1 -- 172.21.15.141:0/1271343682 >> 172.21.15.141:0/1271343682 conn(0x7f1fac06a800 msgr2=0x7f1fac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.358+0000 7f1fb2200640 1 -- 172.21.15.141:0/1271343682 shutdown_connections 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.358+0000 7f1fb2200640 1 -- 172.21.15.141:0/1271343682 wait complete. 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.359+0000 7f1fb2200640 1 Processor -- start 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.360+0000 7f1fb2200640 1 -- start start 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.360+0000 7f1fb2200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c004d00 0x7f1fac13d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.360+0000 7f1fb2200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fac1534c0 0x7f1fac13c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.362 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1fb2200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac1e4c70 0x7f1fac1e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1fb2200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac150a80 con 0x7f1fac1534c0 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1fb2200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac1e5660 con 0x7f1f8c004d00 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1fb2200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac13ce20 con 0x7f1fac1e4c70 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fac1534c0 0x7f1fac13c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fac1534c0 0x7f1fac13c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48522/0 (socket says 172.21.15.141:48522) 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 -- 172.21.15.141:0/1562913121 learned_addr learned my addr 172.21.15.141:0/1562913121 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 -- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac1e4c70 msgr2=0x7f1fac1e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac1e4c70 0x7f1fac1e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.363 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 -- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c004d00 msgr2=0x7f1fac13d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c004d00 0x7f1fac13d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 -- 172.21.15.141:0/1562913121 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac0b3f30 con 0x7f1fac1534c0 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.361+0000 7f1faaa00640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fac1534c0 0x7f1fac13c8e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0108e0 tx=0x7f1f9c010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.362+0000 7f1fa3400640 1 -- 172.21.15.141:0/1562913121 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c00eae0 con 0x7f1fac1534c0 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.362+0000 7f1fa3400640 1 -- 172.21.15.141:0/1562913121 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c00ec40 con 0x7f1fac1534c0 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.362+0000 7f1fa3400640 1 -- 172.21.15.141:0/1562913121 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c01c400 con 0x7f1fac1534c0 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.362+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac0b4210 con 0x7f1fac1534c0 2024-01-16T15:23:54.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.362+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fac13d6e0 con 0x7f1fac1534c0 2024-01-16T15:23:54.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.363+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fac13d900 con 0x7f1fac1534c0 2024-01-16T15:23:54.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.364+0000 7f1fa3400640 1 -- 172.21.15.141:0/1562913121 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f1f9c01c560 con 0x7f1fac1534c0 2024-01-16T15:23:54.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.365+0000 7f1fa3400640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1f8406c5e0 0x7f1f8406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.365+0000 7f1fa3400640 1 -- 172.21.15.141:0/1562913121 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1f9c088e80 con 0x7f1fac1534c0 2024-01-16T15:23:54.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.366+0000 7f1fab400640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1f8406c5e0 0x7f1f8406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.367+0000 7f1fab400640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1f8406c5e0 0x7f1f8406eaa0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1fac1e6a50 tx=0x7f1f94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.370+0000 7f1fa3400640 1 -- 172.21.15.141:0/1562913121 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fac13d900 con 0x7f1fac1534c0 2024-01-16T15:23:54.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.562+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1fac13d900 con 0x7f1fac1534c0 2024-01-16T15:23:54.560 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.563+0000 7f1fa3400640 1 -- 172.21.15.141:0/1562913121 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1fac13d900 con 0x7f1fac1534c0 2024-01-16T15:23:54.561 INFO:teuthology.orchestra.run.smithi141.stdout:pg_num: 64 2024-01-16T15:23:54.563 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1f8406c5e0 msgr2=0x7f1f8406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1f8406c5e0 0x7f1f8406eaa0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1fac1e6a50 tx=0x7f1f94008040 comp rx=0 tx=0).stop 2024-01-16T15:23:54.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fac1534c0 msgr2=0x7f1fac13c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.752 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fac1534c0 0x7f1fac13c8e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0108e0 tx=0x7f1f9c010da0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 shutdown_connections 2024-01-16T15:23:54.753 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f1f8406c5e0 0x7f1f8406eaa0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f1fac1e4c70 0x7f1fac1e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.754 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1fac1534c0 0x7f1fac13c8e0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 --2- 172.21.15.141:0/1562913121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f8c004d00 0x7f1fac13d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.755 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 >> 172.21.15.141:0/1562913121 conn(0x7f1fac06a800 msgr2=0x7f1fac0702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:54.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 shutdown_connections 2024-01-16T15:23:54.757 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.566+0000 7f1fb2200640 1 -- 172.21.15.141:0/1562913121 wait complete. 2024-01-16T15:23:54.759 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:23:54.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.984+0000 7fdc5a200640 1 Processor -- start 2024-01-16T15:23:54.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.984+0000 7fdc5a200640 1 -- start start 2024-01-16T15:23:54.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.986+0000 7fdc5a200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdc5413df80 0x7fdc5415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.986+0000 7fdc5a200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5406c470 con 0x7fdc5413e360 2024-01-16T15:23:54.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.986+0000 7fdc5a200640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5406c750 con 0x7fdc540cf9a0 2024-01-16T15:23:54.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.986+0000 7fdc5a200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5406ca30 con 0x7fdc5413df80 2024-01-16T15:23:54.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.986+0000 7fdc53400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fdc5413df80 0x7fdc5415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.986+0000 7fdc52a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fdc5413e360 0x7fdc540cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54310/0 (socket says 172.21.15.141:54310) 2024-01-16T15:23:54.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.986+0000 7fdc52a00640 1 -- 172.21.15.141:0/2319246342 learned_addr learned my addr 172.21.15.141:0/2319246342 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:54.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.987+0000 7fdc53400640 1 -- 172.21.15.141:0/2319246342 >> v1:172.21.15.188:6789/0 conn(0x7fdc540cf9a0 legacy=0x7fdc540cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:54.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.987+0000 7fdc53400640 1 -- 172.21.15.141:0/2319246342 >> v1:172.21.15.141:6789/0 conn(0x7fdc5413e360 legacy=0x7fdc540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.987+0000 7fdc53400640 1 -- 172.21.15.141:0/2319246342 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5406cd10 con 0x7fdc5413df80 2024-01-16T15:23:54.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.987+0000 7fdc52000640 1 -- 172.21.15.141:0/2319246342 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219546701 0 0) 0x7fdc5406c470 con 0x7fdc5413e360 2024-01-16T15:23:54.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.988+0000 7fdc53400640 1 --2- 172.21.15.141:0/2319246342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc5415bee0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fdc440097a0 tx=0x7fdc4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9072dae087f75e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc52000640 1 -- 172.21.15.141:0/2319246342 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44031830 con 0x7fdc5413df80 2024-01-16T15:23:54.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc52000640 1 -- 172.21.15.141:0/2319246342 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc44031990 con 0x7fdc5413df80 2024-01-16T15:23:54.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc52000640 1 -- 172.21.15.141:0/2319246342 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44031d20 con 0x7fdc5413df80 2024-01-16T15:23:54.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc5a200640 1 -- 172.21.15.141:0/2319246342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 msgr2=0x7fdc5415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc5a200640 1 --2- 172.21.15.141:0/2319246342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc5415bee0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fdc440097a0 tx=0x7fdc4402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc5a200640 1 -- 172.21.15.141:0/2319246342 shutdown_connections 2024-01-16T15:23:54.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc5a200640 1 --2- 172.21.15.141:0/2319246342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc5415bee0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc5a200640 1 -- 172.21.15.141:0/2319246342 >> 172.21.15.141:0/2319246342 conn(0x7fdc5406a800 msgr2=0x7fdc5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:54.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.989+0000 7fdc5a200640 1 -- 172.21.15.141:0/2319246342 shutdown_connections 2024-01-16T15:23:54.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.990+0000 7fdc5a200640 1 -- 172.21.15.141:0/2319246342 wait complete. 2024-01-16T15:23:54.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.991+0000 7fdc5a200640 1 Processor -- start 2024-01-16T15:23:54.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.991+0000 7fdc5a200640 1 -- start start 2024-01-16T15:23:54.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.992+0000 7fdc5a200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc540cf9a0 0x7fdc540b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.992+0000 7fdc5a200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc541f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.992+0000 7fdc5a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5413e360 0x7fdc541e6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.992+0000 7fdc5a200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541f5710 con 0x7fdc5413df80 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.992+0000 7fdc5a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541f4a90 con 0x7fdc5413e360 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.992+0000 7fdc5a200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541f4d70 con 0x7fdc540cf9a0 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc541f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc541f5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48546/0 (socket says 172.21.15.141:48546) 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 -- 172.21.15.141:0/348928289 learned_addr learned my addr 172.21.15.141:0/348928289 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 -- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc540cf9a0 msgr2=0x7fdc540b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc540cf9a0 0x7fdc540b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 -- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5413e360 msgr2=0x7fdc541e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc53e00640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5413e360 0x7fdc541e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5413e360 0x7fdc541e6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc52a00640 1 -- 172.21.15.141:0/348928289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44009400 con 0x7fdc5413df80 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.993+0000 7fdc53e00640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5413e360 0x7fdc541e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.994+0000 7fdc52a00640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc541f5b30 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00c4b0 tx=0x7fdc3c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.994+0000 7fdc4b400640 1 -- 172.21.15.141:0/348928289 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c0074e0 con 0x7fdc5413df80 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.994+0000 7fdc4b400640 1 -- 172.21.15.141:0/348928289 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c007660 con 0x7fdc5413df80 2024-01-16T15:23:54.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.994+0000 7fdc4b400640 1 -- 172.21.15.141:0/348928289 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c004a20 con 0x7fdc5413df80 2024-01-16T15:23:54.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.994+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc541f5050 con 0x7fdc5413df80 2024-01-16T15:23:54.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.994+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc540b3a40 con 0x7fdc5413df80 2024-01-16T15:23:54.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.995+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc540b3c30 con 0x7fdc5413df80 2024-01-16T15:23:54.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.997+0000 7fdc4b400640 1 -- 172.21.15.141:0/348928289 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fdc3c004b80 con 0x7fdc5413df80 2024-01-16T15:23:54.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.997+0000 7fdc4b400640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc2c06c5e0 0x7fdc2c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:54.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.998+0000 7fdc4b400640 1 -- 172.21.15.141:0/348928289 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdc3c087d70 con 0x7fdc5413df80 2024-01-16T15:23:54.995 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.998+0000 7fdc53400640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc2c06c5e0 0x7fdc2c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:54.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:54.999+0000 7fdc53400640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc2c06c5e0 0x7fdc2c06eaa0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdc44036000 tx=0x7fdc44030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:54.999 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.002+0000 7fdc4b400640 1 -- 172.21.15.141:0/348928289 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc3c054390 con 0x7fdc5413df80 2024-01-16T15:23:55.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.206+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdc540675d0 con 0x7fdc5413df80 2024-01-16T15:23:55.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.206+0000 7fdc4b400640 1 -- 172.21.15.141:0/348928289 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdc3c018020 con 0x7fdc5413df80 2024-01-16T15:23:55.204 INFO:teuthology.orchestra.run.smithi141.stderr:listed 0 entries 2024-01-16T15:23:55.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc2c06c5e0 msgr2=0x7fdc2c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:55.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc2c06c5e0 0x7fdc2c06eaa0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdc44036000 tx=0x7fdc44030000 comp rx=0 tx=0).stop 2024-01-16T15:23:55.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 msgr2=0x7fdc541f5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:55.207 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc541f5b30 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00c4b0 tx=0x7fdc3c00c970 comp rx=0 tx=0).stop 2024-01-16T15:23:55.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 shutdown_connections 2024-01-16T15:23:55.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fdc2c06c5e0 0x7fdc2c06eaa0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:55.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc5413e360 0x7fdc541e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:55.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fdc5413df80 0x7fdc541f5b30 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:55.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 --2- 172.21.15.141:0/348928289 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fdc540cf9a0 0x7fdc540b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:55.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 >> 172.21.15.141:0/348928289 conn(0x7fdc5406a800 msgr2=0x7fdc540aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:55.208 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.209+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 shutdown_connections 2024-01-16T15:23:55.209 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:55.210+0000 7fdc5a200640 1 -- 172.21.15.141:0/348928289 wait complete. 2024-01-16T15:23:55.227 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-01-16T15:23:55.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.227 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:23:55.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[47521]: fuse finished with error 0 and tester_r 0 2024-01-16T15:23:55.325 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:23:55.375 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:23:55.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.376 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:23:55.426 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-01-16T15:23:55.426 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:23:55.472 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:23:55.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.472 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-16T15:23:55.528 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:23:55.528 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:23:55.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.528 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:23:55.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.618 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:23:55.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link delete ceph-brx 2024-01-16T15:23:55.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.867 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-01-16T15:23:55.922 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:23:55.922 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:23:55.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:55.923 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:23:56.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:56.015 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:23:56.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:23:56.101 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link delete ceph-brx 2024-01-16T15:23:56.234 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T15:23:56.244 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T15:23:56.245 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:56.462 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.465+0000 7f9abb200640 1 Processor -- start 2024-01-16T15:23:56.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.465+0000 7f9abb200640 1 -- start start 2024-01-16T15:23:56.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9abb200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9abc143c40 0x7f9abc071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9abb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0717d0 0x7f9abc0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9abb200640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc070e50 con 0x7f9abc0cca40 2024-01-16T15:23:56.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9abb200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0723c0 con 0x7f9abc143c40 2024-01-16T15:23:56.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9abb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0cce70 con 0x7f9abc0717d0 2024-01-16T15:23:56.465 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9ab9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9abc143c40 0x7f9abc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9ab9e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9abc143c40 0x7f9abc071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48554/0 (socket says 172.21.15.141:48554) 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9ab9e00640 1 -- 172.21.15.141:0/714985380 learned_addr learned my addr 172.21.15.141:0/714985380 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.467+0000 7f9aba800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9abc0cca40 0x7f9abc15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54320/0 (socket says 172.21.15.141:54320) 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.468+0000 7f9ab1200640 1 --2- 172.21.15.141:0/714985380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0717d0 0x7f9abc0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.468+0000 7f9ab9e00640 1 -- 172.21.15.141:0/714985380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0717d0 msgr2=0x7f9abc0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.468+0000 7f9ab9e00640 1 --2- 172.21.15.141:0/714985380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0717d0 0x7f9abc0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.468+0000 7f9ab9e00640 1 -- 172.21.15.141:0/714985380 >> v1:172.21.15.141:6789/0 conn(0x7f9abc0cca40 legacy=0x7f9abc15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.466 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.468+0000 7f9ab9e00640 1 -- 172.21.15.141:0/714985380 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc06e170 con 0x7f9abc143c40 2024-01-16T15:23:56.467 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.468+0000 7f9ab9400640 1 -- 172.21.15.141:0/714985380 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884379203 0 0) 0x7f9abc070e50 con 0x7f9abc0cca40 2024-01-16T15:23:56.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.469+0000 7f9ab9e00640 1 --2- 172.21.15.141:0/714985380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc143c40 0x7f9abc071290 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009e50 tx=0x7f9aac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e520f0d416ca328 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:56.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.469+0000 7f9ab9400640 1 -- 172.21.15.141:0/714985380 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac02ca70 con 0x7f9abc143c40 2024-01-16T15:23:56.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.469+0000 7f9ab9400640 1 -- 172.21.15.141:0/714985380 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac02cbd0 con 0x7f9abc143c40 2024-01-16T15:23:56.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.469+0000 7f9ab9400640 1 -- 172.21.15.141:0/714985380 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0343f0 con 0x7f9abc143c40 2024-01-16T15:23:56.468 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 -- 172.21.15.141:0/714985380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc143c40 msgr2=0x7f9abc071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 --2- 172.21.15.141:0/714985380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc143c40 0x7f9abc071290 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009e50 tx=0x7f9aac02bed0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 -- 172.21.15.141:0/714985380 shutdown_connections 2024-01-16T15:23:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 --2- 172.21.15.141:0/714985380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0717d0 0x7f9abc0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 --2- 172.21.15.141:0/714985380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc143c40 0x7f9abc071290 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 -- 172.21.15.141:0/714985380 >> 172.21.15.141:0/714985380 conn(0x7f9abc06a800 msgr2=0x7f9abc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 -- 172.21.15.141:0/714985380 shutdown_connections 2024-01-16T15:23:56.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.470+0000 7f9abb200640 1 -- 172.21.15.141:0/714985380 wait complete. 2024-01-16T15:23:56.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.472+0000 7f9abb200640 1 Processor -- start 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.472+0000 7f9abb200640 1 -- start start 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9abb200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9abc0717d0 0x7f9abc139570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9abb200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0cca40 0x7f9abc0b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9abb200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc141060 0x7f9abc139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9abb200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0b3ef0 con 0x7f9abc141060 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9abb200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0b3880 con 0x7f9abc0717d0 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9abb200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1e45d0 con 0x7f9abc0cca40 2024-01-16T15:23:56.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9aba800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc141060 0x7f9abc139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9aba800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc141060 0x7f9abc139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48562/0 (socket says 172.21.15.141:48562) 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9aba800640 1 -- 172.21.15.141:0/3573483546 learned_addr learned my addr 172.21.15.141:0/3573483546 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9ab9e00640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9abc0717d0 0x7f9abc139570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.473+0000 7f9ab1200640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0cca40 0x7f9abc0b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9aba800640 1 -- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0cca40 msgr2=0x7f9abc0b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9aba800640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0cca40 0x7f9abc0b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9aba800640 1 -- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9abc0717d0 msgr2=0x7f9abc139570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9aba800640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9abc0717d0 0x7f9abc139570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9aba800640 1 -- 172.21.15.141:0/3573483546 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac009b30 con 0x7f9abc141060 2024-01-16T15:23:56.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9aba800640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc141060 0x7f9abc139de0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4001eb0 tx=0x7f9ab400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:56.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9ab2a00640 1 -- 172.21.15.141:0/3573483546 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40074e0 con 0x7f9abc141060 2024-01-16T15:23:56.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9ab2a00640 1 -- 172.21.15.141:0/3573483546 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab4007640 con 0x7f9abc141060 2024-01-16T15:23:56.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc1e48b0 con 0x7f9abc141060 2024-01-16T15:23:56.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9ab2a00640 1 -- 172.21.15.141:0/3573483546 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40049e0 con 0x7f9abc141060 2024-01-16T15:23:56.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.474+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc1e4e70 con 0x7f9abc141060 2024-01-16T15:23:56.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.475+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc067450 con 0x7f9abc141060 2024-01-16T15:23:56.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.476+0000 7f9ab2a00640 1 -- 172.21.15.141:0/3573483546 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f9ab4004b40 con 0x7f9abc141060 2024-01-16T15:23:56.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.477+0000 7f9ab2a00640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a9406c5e0 0x7f9a9406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.475 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.477+0000 7f9ab2a00640 1 -- 172.21.15.141:0/3573483546 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9ab40885c0 con 0x7f9abc141060 2024-01-16T15:23:56.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.478+0000 7f9ab9e00640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a9406c5e0 0x7f9a9406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.476 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.479+0000 7f9ab9e00640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a9406c5e0 0x7f9a9406eaa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9aac035000 tx=0x7f9aac030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:56.478 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.481+0000 7f9ab2a00640 1 -- 172.21.15.141:0/3573483546 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ab4054e80 con 0x7f9abc141060 2024-01-16T15:23:56.666 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.668+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9abc0b39c0 con 0x7f9a9406c5e0 2024-01-16T15:23:56.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.696+0000 7f9ab2a00640 1 -- 172.21.15.141:0/3573483546 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289288 (secure 0 0 0) 0x7f9abc0b39c0 con 0x7f9a9406c5e0 2024-01-16T15:23:56.699 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:56.701 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:23:56.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.701+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a9406c5e0 msgr2=0x7f9a9406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.701+0000 7f9abb200640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a9406c5e0 0x7f9a9406eaa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9aac035000 tx=0x7f9aac030000 comp rx=0 tx=0).stop 2024-01-16T15:23:56.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.701+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc141060 msgr2=0x7f9abc139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.703 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.701+0000 7f9abb200640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc141060 0x7f9abc139de0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4001eb0 tx=0x7f9ab400c870 comp rx=0 tx=0).stop 2024-01-16T15:23:56.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 shutdown_connections 2024-01-16T15:23:56.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9a9406c5e0 0x7f9a9406eaa0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9abc141060 0x7f9abc139de0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9abc0cca40 0x7f9abc0b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 --2- 172.21.15.141:0/3573483546 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9abc0717d0 0x7f9abc139570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 >> 172.21.15.141:0/3573483546 conn(0x7f9abc06a800 msgr2=0x7f9abc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:56.704 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 shutdown_connections 2024-01-16T15:23:56.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.702+0000 7f9abb200640 1 -- 172.21.15.141:0/3573483546 wait complete. 2024-01-16T15:23:56.719 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":919,"stamp":"2024-01-16T15:23:55.120233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444288908,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":945,"num_read_kb":3145889,"num_write":2248,"num_write_kb":6295364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4679,"ondisk_log_size":4679,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13191200,"kb_used_data":12598048,"kb_used_omap":12,"kb_used_meta":593075,"kb_avail":741783520,"statfs":{"total":773094113280,"available":759586324480,"internally_reserved":0,"allocated":12900401152,"data_stored":12893659032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":607309362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073781653,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":274,"num_write_kb":1048622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004362"},"pg_stats":[{"pgid":"17.3f","version":"118'70","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758782+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:23:49.758782+0000","last_peered":"2024-01-16T15:23:49.758782+0000","last_clean":"2024-01-16T15:23:49.758782+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:23:49.758782+0000","last_undegraded":"2024-01-16T15:23:49.758782+0000","last_fullsized":"2024-01-16T15:23:49.758782+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:39.215878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"118'69","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.745868+0000","last_change":"2024-01-16T15:15:39.327644+0000","last_active":"2024-01-16T15:23:49.745868+0000","last_peered":"2024-01-16T15:23:49.745868+0000","last_clean":"2024-01-16T15:23:49.745868+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:23:49.745868+0000","last_undegraded":"2024-01-16T15:23:49.745868+0000","last_fullsized":"2024-01-16T15:23:49.745868+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:39.747172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"118'67","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.754526+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:23:49.754526+0000","last_peered":"2024-01-16T15:23:49.754526+0000","last_clean":"2024-01-16T15:23:49.754526+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:23:49.754526+0000","last_undegraded":"2024-01-16T15:23:49.754526+0000","last_fullsized":"2024-01-16T15:23:49.754526+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:22.535735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"118'66","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738937+0000","last_change":"2024-01-16T15:15:55.875926+0000","last_active":"2024-01-16T15:23:49.738937+0000","last_peered":"2024-01-16T15:23:49.738937+0000","last_clean":"2024-01-16T15:23:49.738937+0000","last_became_active":"2024-01-16T15:15:55.875179+0000","last_became_peered":"2024-01-16T15:15:55.875179+0000","last_unstale":"2024-01-16T15:23:49.738937+0000","last_undegraded":"2024-01-16T15:23:49.738937+0000","last_fullsized":"2024-01-16T15:23:49.738937+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3b","version":"118'60","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737805+0000","last_change":"2024-01-16T15:15:39.321081+0000","last_active":"2024-01-16T15:23:49.737805+0000","last_peered":"2024-01-16T15:23:49.737805+0000","last_clean":"2024-01-16T15:23:49.737805+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:23:49.737805+0000","last_undegraded":"2024-01-16T15:23:49.737805+0000","last_fullsized":"2024-01-16T15:23:49.737805+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663487,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"118'73","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.743047+0000","last_change":"2024-01-16T15:15:39.327813+0000","last_active":"2024-01-16T15:23:49.743047+0000","last_peered":"2024-01-16T15:23:49.743047+0000","last_clean":"2024-01-16T15:23:49.743047+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:23:49.743047+0000","last_undegraded":"2024-01-16T15:23:49.743047+0000","last_fullsized":"2024-01-16T15:23:49.743047+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:39:32.420035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"118'66","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751785+0000","last_change":"2024-01-16T15:15:39.324707+0000","last_active":"2024-01-16T15:23:49.751785+0000","last_peered":"2024-01-16T15:23:49.751785+0000","last_clean":"2024-01-16T15:23:49.751785+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:23:49.751785+0000","last_undegraded":"2024-01-16T15:23:49.751785+0000","last_fullsized":"2024-01-16T15:23:49.751785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303428,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"118'54","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752934+0000","last_change":"2024-01-16T15:15:39.328547+0000","last_active":"2024-01-16T15:23:49.752934+0000","last_peered":"2024-01-16T15:23:49.752934+0000","last_clean":"2024-01-16T15:23:49.752934+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:23:49.752934+0000","last_undegraded":"2024-01-16T15:23:49.752934+0000","last_fullsized":"2024-01-16T15:23:49.752934+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:20.486476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"118'79","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755131+0000","last_change":"2024-01-16T15:15:39.320236+0000","last_active":"2024-01-16T15:23:49.755131+0000","last_peered":"2024-01-16T15:23:49.755131+0000","last_clean":"2024-01-16T15:23:49.755131+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:23:49.755131+0000","last_undegraded":"2024-01-16T15:23:49.755131+0000","last_fullsized":"2024-01-16T15:23:49.755131+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:19:06.417692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052164,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"118'66","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758281+0000","last_change":"2024-01-16T15:15:39.326528+0000","last_active":"2024-01-16T15:23:49.758281+0000","last_peered":"2024-01-16T15:23:49.758281+0000","last_clean":"2024-01-16T15:23:49.758281+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:23:49.758281+0000","last_undegraded":"2024-01-16T15:23:49.758281+0000","last_fullsized":"2024-01-16T15:23:49.758281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:01.161905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"118'70","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737879+0000","last_change":"2024-01-16T15:15:39.325379+0000","last_active":"2024-01-16T15:23:49.737879+0000","last_peered":"2024-01-16T15:23:49.737879+0000","last_clean":"2024-01-16T15:23:49.737879+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:23:49.737879+0000","last_undegraded":"2024-01-16T15:23:49.737879+0000","last_fullsized":"2024-01-16T15:23:49.737879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:11.903533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"118'59","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757983+0000","last_change":"2024-01-16T15:15:39.317978+0000","last_active":"2024-01-16T15:23:49.757983+0000","last_peered":"2024-01-16T15:23:49.757983+0000","last_clean":"2024-01-16T15:23:49.757983+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:23:49.757983+0000","last_undegraded":"2024-01-16T15:23:49.757983+0000","last_fullsized":"2024-01-16T15:23:49.757983+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:47.669894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"118'57","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746095+0000","last_change":"2024-01-16T15:15:39.329570+0000","last_active":"2024-01-16T15:23:49.746095+0000","last_peered":"2024-01-16T15:23:49.746095+0000","last_clean":"2024-01-16T15:23:49.746095+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:23:49.746095+0000","last_undegraded":"2024-01-16T15:23:49.746095+0000","last_fullsized":"2024-01-16T15:23:49.746095+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:22.565771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"118'65","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742239+0000","last_change":"2024-01-16T15:15:39.325613+0000","last_active":"2024-01-16T15:23:49.742239+0000","last_peered":"2024-01-16T15:23:49.742239+0000","last_clean":"2024-01-16T15:23:49.742239+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:23:49.742239+0000","last_undegraded":"2024-01-16T15:23:49.742239+0000","last_fullsized":"2024-01-16T15:23:49.742239+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:08.369872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"118'54","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741582+0000","last_change":"2024-01-16T15:15:39.333121+0000","last_active":"2024-01-16T15:23:49.741582+0000","last_peered":"2024-01-16T15:23:49.741582+0000","last_clean":"2024-01-16T15:23:49.741582+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:23:49.741582+0000","last_undegraded":"2024-01-16T15:23:49.741582+0000","last_fullsized":"2024-01-16T15:23:49.741582+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:19.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"118'58","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756735+0000","last_change":"2024-01-16T15:15:39.313940+0000","last_active":"2024-01-16T15:23:49.756735+0000","last_peered":"2024-01-16T15:23:49.756735+0000","last_clean":"2024-01-16T15:23:49.756735+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:23:49.756735+0000","last_undegraded":"2024-01-16T15:23:49.756735+0000","last_fullsized":"2024-01-16T15:23:49.756735+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:57.259350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"118'49","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740185+0000","last_change":"2024-01-16T15:15:39.329528+0000","last_active":"2024-01-16T15:23:49.740185+0000","last_peered":"2024-01-16T15:23:49.740185+0000","last_clean":"2024-01-16T15:23:49.740185+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:23:49.740185+0000","last_undegraded":"2024-01-16T15:23:49.740185+0000","last_fullsized":"2024-01-16T15:23:49.740185+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:42.153458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"118'72","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742141+0000","last_change":"2024-01-16T15:15:39.330676+0000","last_active":"2024-01-16T15:23:49.742141+0000","last_peered":"2024-01-16T15:23:49.742141+0000","last_clean":"2024-01-16T15:23:49.742141+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:23:49.742141+0000","last_undegraded":"2024-01-16T15:23:49.742141+0000","last_fullsized":"2024-01-16T15:23:49.742141+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:39:04.346629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"118'71","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751357+0000","last_change":"2024-01-16T15:15:39.330437+0000","last_active":"2024-01-16T15:23:49.751357+0000","last_peered":"2024-01-16T15:23:49.751357+0000","last_clean":"2024-01-16T15:23:49.751357+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:23:49.751357+0000","last_undegraded":"2024-01-16T15:23:49.751357+0000","last_fullsized":"2024-01-16T15:23:49.751357+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:37.311014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"118'67","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750922+0000","last_change":"2024-01-16T15:15:39.318689+0000","last_active":"2024-01-16T15:23:49.750922+0000","last_peered":"2024-01-16T15:23:49.750922+0000","last_clean":"2024-01-16T15:23:49.750922+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:23:49.750922+0000","last_undegraded":"2024-01-16T15:23:49.750922+0000","last_fullsized":"2024-01-16T15:23:49.750922+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:35.351694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"118'73","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749394+0000","last_change":"2024-01-16T15:15:39.321055+0000","last_active":"2024-01-16T15:23:49.749394+0000","last_peered":"2024-01-16T15:23:49.749394+0000","last_clean":"2024-01-16T15:23:49.749394+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:23:49.749394+0000","last_undegraded":"2024-01-16T15:23:49.749394+0000","last_fullsized":"2024-01-16T15:23:49.749394+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:46.770618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"118'64","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749195+0000","last_change":"2024-01-16T15:15:39.323290+0000","last_active":"2024-01-16T15:23:49.749195+0000","last_peered":"2024-01-16T15:23:49.749195+0000","last_clean":"2024-01-16T15:23:49.749195+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:23:49.749195+0000","last_undegraded":"2024-01-16T15:23:49.749195+0000","last_fullsized":"2024-01-16T15:23:49.749195+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:48:04.184635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"118'57","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737467+0000","last_change":"2024-01-16T15:15:39.317974+0000","last_active":"2024-01-16T15:23:49.737467+0000","last_peered":"2024-01-16T15:23:49.737467+0000","last_clean":"2024-01-16T15:23:49.737467+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:23:49.737467+0000","last_undegraded":"2024-01-16T15:23:49.737467+0000","last_fullsized":"2024-01-16T15:23:49.737467+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:34.399913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"118'70","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753684+0000","last_change":"2024-01-16T15:15:39.324363+0000","last_active":"2024-01-16T15:23:49.753684+0000","last_peered":"2024-01-16T15:23:49.753684+0000","last_clean":"2024-01-16T15:23:49.753684+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:23:49.753684+0000","last_undegraded":"2024-01-16T15:23:49.753684+0000","last_fullsized":"2024-01-16T15:23:49.753684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:12.878416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"118'71","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741193+0000","last_change":"2024-01-16T15:15:39.328896+0000","last_active":"2024-01-16T15:23:49.741193+0000","last_peered":"2024-01-16T15:23:49.741193+0000","last_clean":"2024-01-16T15:23:49.741193+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:23:49.741193+0000","last_undegraded":"2024-01-16T15:23:49.741193+0000","last_fullsized":"2024-01-16T15:23:49.741193+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:41.474484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"118'81","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756330+0000","last_change":"2024-01-16T15:15:39.317589+0000","last_active":"2024-01-16T15:23:49.756330+0000","last_peered":"2024-01-16T15:23:49.756330+0000","last_clean":"2024-01-16T15:23:49.756330+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:23:49.756330+0000","last_undegraded":"2024-01-16T15:23:49.756330+0000","last_fullsized":"2024-01-16T15:23:49.756330+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:36.235755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"118'64","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746236+0000","last_change":"2024-01-16T15:15:39.335455+0000","last_active":"2024-01-16T15:23:49.746236+0000","last_peered":"2024-01-16T15:23:49.746236+0000","last_clean":"2024-01-16T15:23:49.746236+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:23:49.746236+0000","last_undegraded":"2024-01-16T15:23:49.746236+0000","last_fullsized":"2024-01-16T15:23:49.746236+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:38.284829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"118'43","reported_seq":64,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749120+0000","last_change":"2024-01-16T15:15:39.317292+0000","last_active":"2024-01-16T15:23:49.749120+0000","last_peered":"2024-01-16T15:23:49.749120+0000","last_clean":"2024-01-16T15:23:49.749120+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:23:49.749120+0000","last_undegraded":"2024-01-16T15:23:49.749120+0000","last_fullsized":"2024-01-16T15:23:49.749120+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:38.845770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"118'57","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751058+0000","last_change":"2024-01-16T15:15:39.320340+0000","last_active":"2024-01-16T15:23:49.751058+0000","last_peered":"2024-01-16T15:23:49.751058+0000","last_clean":"2024-01-16T15:23:49.751058+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:23:49.751058+0000","last_undegraded":"2024-01-16T15:23:49.751058+0000","last_fullsized":"2024-01-16T15:23:49.751058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:18.946850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"118'76","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740849+0000","last_change":"2024-01-16T15:15:39.317544+0000","last_active":"2024-01-16T15:23:49.740849+0000","last_peered":"2024-01-16T15:23:49.740849+0000","last_clean":"2024-01-16T15:23:49.740849+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:23:49.740849+0000","last_undegraded":"2024-01-16T15:23:49.740849+0000","last_fullsized":"2024-01-16T15:23:49.740849+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:38.315912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"118'81","reported_seq":116,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757053+0000","last_change":"2024-01-16T15:15:39.313788+0000","last_active":"2024-01-16T15:23:49.757053+0000","last_peered":"2024-01-16T15:23:49.757053+0000","last_clean":"2024-01-16T15:23:49.757053+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:23:49.757053+0000","last_undegraded":"2024-01-16T15:23:49.757053+0000","last_fullsized":"2024-01-16T15:23:49.757053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:43.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"118'63","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744908+0000","last_change":"2024-01-16T15:15:39.321572+0000","last_active":"2024-01-16T15:23:49.744908+0000","last_peered":"2024-01-16T15:23:49.744908+0000","last_clean":"2024-01-16T15:23:49.744908+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:23:49.744908+0000","last_undegraded":"2024-01-16T15:23:49.744908+0000","last_fullsized":"2024-01-16T15:23:49.744908+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"118'62","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.733710+0000","last_change":"2024-01-16T15:15:39.329833+0000","last_active":"2024-01-16T15:23:49.733710+0000","last_peered":"2024-01-16T15:23:49.733710+0000","last_clean":"2024-01-16T15:23:49.733710+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:23:49.733710+0000","last_undegraded":"2024-01-16T15:23:49.733710+0000","last_fullsized":"2024-01-16T15:23:49.733710+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:51.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'4","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.873920+0000","last_change":"2024-01-16T15:15:38.295673+0000","last_active":"2024-01-16T15:19:49.873920+0000","last_peered":"2024-01-16T15:19:49.873920+0000","last_clean":"2024-01-16T15:19:49.873920+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:19:49.873920+0000","last_undegraded":"2024-01-16T15:19:49.873920+0000","last_fullsized":"2024-01-16T15:19:49.873920+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"118'54","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751652+0000","last_change":"2024-01-16T15:15:39.329620+0000","last_active":"2024-01-16T15:23:49.751652+0000","last_peered":"2024-01-16T15:23:49.751652+0000","last_clean":"2024-01-16T15:23:49.751652+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:23:49.751652+0000","last_undegraded":"2024-01-16T15:23:49.751652+0000","last_fullsized":"2024-01-16T15:23:49.751652+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.326448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:40.158702+0000","last_change":"2024-01-16T15:15:38.292769+0000","last_active":"2024-01-16T15:21:40.158702+0000","last_peered":"2024-01-16T15:21:40.158702+0000","last_clean":"2024-01-16T15:21:40.158702+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:21:40.158702+0000","last_undegraded":"2024-01-16T15:21:40.158702+0000","last_fullsized":"2024-01-16T15:21:40.158702+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:45.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"118'71","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744070+0000","last_change":"2024-01-16T15:15:39.332870+0000","last_active":"2024-01-16T15:23:49.744070+0000","last_peered":"2024-01-16T15:23:49.744070+0000","last_clean":"2024-01-16T15:23:49.744070+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:23:49.744070+0000","last_undegraded":"2024-01-16T15:23:49.744070+0000","last_fullsized":"2024-01-16T15:23:49.744070+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.891495+0000","last_change":"2024-01-16T15:15:38.288495+0000","last_active":"2024-01-16T15:20:39.891495+0000","last_peered":"2024-01-16T15:20:39.891495+0000","last_clean":"2024-01-16T15:20:39.891495+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:20:39.891495+0000","last_undegraded":"2024-01-16T15:20:39.891495+0000","last_fullsized":"2024-01-16T15:20:39.891495+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:07.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"118'68","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753436+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:23:49.753436+0000","last_peered":"2024-01-16T15:23:49.753436+0000","last_clean":"2024-01-16T15:23:49.753436+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:23:49.753436+0000","last_undegraded":"2024-01-16T15:23:49.753436+0000","last_fullsized":"2024-01-16T15:23:49.753436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:21.846652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.891465+0000","last_change":"2024-01-16T15:15:38.288324+0000","last_active":"2024-01-16T15:20:39.891465+0000","last_peered":"2024-01-16T15:20:39.891465+0000","last_clean":"2024-01-16T15:20:39.891465+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:20:39.891465+0000","last_undegraded":"2024-01-16T15:20:39.891465+0000","last_fullsized":"2024-01-16T15:20:39.891465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:31.849193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"118'72","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756110+0000","last_change":"2024-01-16T15:15:39.321037+0000","last_active":"2024-01-16T15:23:49.756110+0000","last_peered":"2024-01-16T15:23:49.756110+0000","last_clean":"2024-01-16T15:23:49.756110+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:23:49.756110+0000","last_undegraded":"2024-01-16T15:23:49.756110+0000","last_fullsized":"2024-01-16T15:23:49.756110+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.687281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.883073+0000","last_change":"2024-01-16T15:15:38.288226+0000","last_active":"2024-01-16T15:20:39.883073+0000","last_peered":"2024-01-16T15:20:39.883073+0000","last_clean":"2024-01-16T15:20:39.883073+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:20:39.883073+0000","last_undegraded":"2024-01-16T15:20:39.883073+0000","last_fullsized":"2024-01-16T15:20:39.883073+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:41.040743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"118'72","reported_seq":108,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752048+0000","last_change":"2024-01-16T15:15:39.330201+0000","last_active":"2024-01-16T15:23:49.752048+0000","last_peered":"2024-01-16T15:23:49.752048+0000","last_clean":"2024-01-16T15:23:49.752048+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:23:49.752048+0000","last_undegraded":"2024-01-16T15:23:49.752048+0000","last_fullsized":"2024-01-16T15:23:49.752048+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:52.162066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.966446+0000","last_change":"2024-01-16T15:15:38.292721+0000","last_active":"2024-01-16T15:20:39.966446+0000","last_peered":"2024-01-16T15:20:39.966446+0000","last_clean":"2024-01-16T15:20:39.966446+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:20:39.966446+0000","last_undegraded":"2024-01-16T15:20:39.966446+0000","last_fullsized":"2024-01-16T15:20:39.966446+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:32.177991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"118'73","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753205+0000","last_change":"2024-01-16T15:15:39.322497+0000","last_active":"2024-01-16T15:23:49.753205+0000","last_peered":"2024-01-16T15:23:49.753205+0000","last_clean":"2024-01-16T15:23:49.753205+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:23:49.753205+0000","last_undegraded":"2024-01-16T15:23:49.753205+0000","last_fullsized":"2024-01-16T15:23:49.753205+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:18.495148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.977194+0000","last_change":"2024-01-16T15:15:38.276895+0000","last_active":"2024-01-16T15:20:29.977194+0000","last_peered":"2024-01-16T15:20:29.977194+0000","last_clean":"2024-01-16T15:20:29.977194+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:20:29.977194+0000","last_undegraded":"2024-01-16T15:20:29.977194+0000","last_fullsized":"2024-01-16T15:20:29.977194+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"118'73","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749092+0000","last_change":"2024-01-16T15:15:39.325224+0000","last_active":"2024-01-16T15:23:49.749092+0000","last_peered":"2024-01-16T15:23:49.749092+0000","last_clean":"2024-01-16T15:23:49.749092+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:23:49.749092+0000","last_undegraded":"2024-01-16T15:23:49.749092+0000","last_fullsized":"2024-01-16T15:23:49.749092+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.872263+0000","last_change":"2024-01-16T15:15:38.297269+0000","last_active":"2024-01-16T15:20:29.872263+0000","last_peered":"2024-01-16T15:20:29.872263+0000","last_clean":"2024-01-16T15:20:29.872263+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:20:29.872263+0000","last_undegraded":"2024-01-16T15:20:29.872263+0000","last_fullsized":"2024-01-16T15:20:29.872263+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:02.166771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"118'58","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.735497+0000","last_change":"2024-01-16T15:15:39.320446+0000","last_active":"2024-01-16T15:23:49.735497+0000","last_peered":"2024-01-16T15:23:49.735497+0000","last_clean":"2024-01-16T15:23:49.735497+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:23:49.735497+0000","last_undegraded":"2024-01-16T15:23:49.735497+0000","last_fullsized":"2024-01-16T15:23:49.735497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:15.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"118'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.889799+0000","last_change":"2024-01-16T15:15:38.287453+0000","last_active":"2024-01-16T15:20:29.889799+0000","last_peered":"2024-01-16T15:20:29.889799+0000","last_clean":"2024-01-16T15:20:29.889799+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:20:29.889799+0000","last_undegraded":"2024-01-16T15:20:29.889799+0000","last_fullsized":"2024-01-16T15:20:29.889799+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"118'61","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747524+0000","last_change":"2024-01-16T15:15:39.333810+0000","last_active":"2024-01-16T15:23:49.747524+0000","last_peered":"2024-01-16T15:23:49.747524+0000","last_clean":"2024-01-16T15:23:49.747524+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:23:49.747524+0000","last_undegraded":"2024-01-16T15:23:49.747524+0000","last_fullsized":"2024-01-16T15:23:49.747524+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:22:34.785294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.964575+0000","last_change":"2024-01-16T15:15:38.293583+0000","last_active":"2024-01-16T15:20:29.964575+0000","last_peered":"2024-01-16T15:20:29.964575+0000","last_clean":"2024-01-16T15:20:29.964575+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:20:29.964575+0000","last_undegraded":"2024-01-16T15:20:29.964575+0000","last_fullsized":"2024-01-16T15:20:29.964575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:48.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"118'62","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738579+0000","last_change":"2024-01-16T15:15:39.319368+0000","last_active":"2024-01-16T15:23:49.738579+0000","last_peered":"2024-01-16T15:23:49.738579+0000","last_clean":"2024-01-16T15:23:49.738579+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:23:49.738579+0000","last_undegraded":"2024-01-16T15:23:49.738579+0000","last_fullsized":"2024-01-16T15:23:49.738579+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:31.021632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"118'31","reported_seq":48,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.966626+0000","last_change":"2024-01-16T15:15:38.295750+0000","last_active":"2024-01-16T15:21:45.966626+0000","last_peered":"2024-01-16T15:21:45.966626+0000","last_clean":"2024-01-16T15:21:45.966626+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:21:45.966626+0000","last_undegraded":"2024-01-16T15:21:45.966626+0000","last_fullsized":"2024-01-16T15:21:45.966626+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:17.036375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"118'69","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750139+0000","last_change":"2024-01-16T15:15:39.312858+0000","last_active":"2024-01-16T15:23:49.750139+0000","last_peered":"2024-01-16T15:23:49.750139+0000","last_clean":"2024-01-16T15:23:49.750139+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:23:49.750139+0000","last_undegraded":"2024-01-16T15:23:49.750139+0000","last_fullsized":"2024-01-16T15:23:49.750139+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"118'325","reported_seq":342,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.761469+0000","last_change":"2024-01-16T15:15:38.296732+0000","last_active":"2024-01-16T15:23:49.761469+0000","last_peered":"2024-01-16T15:23:49.761469+0000","last_clean":"2024-01-16T15:23:49.761469+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:23:49.761469+0000","last_undegraded":"2024-01-16T15:23:49.761469+0000","last_fullsized":"2024-01-16T15:23:49.761469+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1241159,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":1369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"118'54","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737844+0000","last_change":"2024-01-16T15:15:39.312723+0000","last_active":"2024-01-16T15:23:49.737844+0000","last_peered":"2024-01-16T15:23:49.737844+0000","last_clean":"2024-01-16T15:23:49.737844+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:23:49.737844+0000","last_undegraded":"2024-01-16T15:23:49.737844+0000","last_fullsized":"2024-01-16T15:23:49.737844+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:21.305831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.961056+0000","last_change":"2024-01-16T15:15:38.280236+0000","last_active":"2024-01-16T15:21:45.961056+0000","last_peered":"2024-01-16T15:21:45.961056+0000","last_clean":"2024-01-16T15:21:45.961056+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:21:45.961056+0000","last_undegraded":"2024-01-16T15:21:45.961056+0000","last_fullsized":"2024-01-16T15:21:45.961056+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:14.715313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"118'72","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753729+0000","last_change":"2024-01-16T15:15:39.324127+0000","last_active":"2024-01-16T15:23:49.753729+0000","last_peered":"2024-01-16T15:23:49.753729+0000","last_clean":"2024-01-16T15:23:49.753729+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:23:49.753729+0000","last_undegraded":"2024-01-16T15:23:49.753729+0000","last_fullsized":"2024-01-16T15:23:49.753729+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.687281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:20.345465+0000","last_change":"2024-01-16T15:15:38.295857+0000","last_active":"2024-01-16T15:20:20.345465+0000","last_peered":"2024-01-16T15:20:20.345465+0000","last_clean":"2024-01-16T15:20:20.345465+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:20:20.345465+0000","last_undegraded":"2024-01-16T15:20:20.345465+0000","last_fullsized":"2024-01-16T15:20:20.345465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:31.063560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"118'64","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.743542+0000","last_change":"2024-01-16T15:15:39.319495+0000","last_active":"2024-01-16T15:23:49.743542+0000","last_peered":"2024-01-16T15:23:49.743542+0000","last_clean":"2024-01-16T15:23:49.743542+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:23:49.743542+0000","last_undegraded":"2024-01-16T15:23:49.743542+0000","last_fullsized":"2024-01-16T15:23:49.743542+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:18.495148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":40961,"num_write":27,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'8","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:19.887701+0000","last_change":"2024-01-16T15:15:38.293904+0000","last_active":"2024-01-16T15:20:19.887701+0000","last_peered":"2024-01-16T15:20:19.887701+0000","last_clean":"2024-01-16T15:20:19.887701+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:20:19.887701+0000","last_undegraded":"2024-01-16T15:20:19.887701+0000","last_fullsized":"2024-01-16T15:20:19.887701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"118'46","reported_seq":68,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758065+0000","last_change":"2024-01-16T15:15:39.314200+0000","last_active":"2024-01-16T15:23:49.758065+0000","last_peered":"2024-01-16T15:23:49.758065+0000","last_clean":"2024-01-16T15:23:49.758065+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:23:49.758065+0000","last_undegraded":"2024-01-16T15:23:49.758065+0000","last_fullsized":"2024-01-16T15:23:49.758065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.868373+0000","last_change":"2024-01-16T15:15:38.296521+0000","last_active":"2024-01-16T15:20:09.868373+0000","last_peered":"2024-01-16T15:20:09.868373+0000","last_clean":"2024-01-16T15:20:09.868373+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:20:09.868373+0000","last_undegraded":"2024-01-16T15:20:09.868373+0000","last_fullsized":"2024-01-16T15:20:09.868373+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:24.030528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"118'61","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748133+0000","last_change":"2024-01-16T15:15:39.334617+0000","last_active":"2024-01-16T15:23:49.748133+0000","last_peered":"2024-01-16T15:23:49.748133+0000","last_clean":"2024-01-16T15:23:49.748133+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:23:49.748133+0000","last_undegraded":"2024-01-16T15:23:49.748133+0000","last_fullsized":"2024-01-16T15:23:49.748133+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:40.033000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.886136+0000","last_change":"2024-01-16T15:15:38.289440+0000","last_active":"2024-01-16T15:20:09.886136+0000","last_peered":"2024-01-16T15:20:09.886136+0000","last_clean":"2024-01-16T15:20:09.886136+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:20:09.886136+0000","last_undegraded":"2024-01-16T15:20:09.886136+0000","last_fullsized":"2024-01-16T15:20:09.886136+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"118'61","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755433+0000","last_change":"2024-01-16T15:15:39.332704+0000","last_active":"2024-01-16T15:23:49.755433+0000","last_peered":"2024-01-16T15:23:49.755433+0000","last_clean":"2024-01-16T15:23:49.755433+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:23:49.755433+0000","last_undegraded":"2024-01-16T15:23:49.755433+0000","last_fullsized":"2024-01-16T15:23:49.755433+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:08.430991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:10.343721+0000","last_change":"2024-01-16T15:15:38.291309+0000","last_active":"2024-01-16T15:20:10.343721+0000","last_peered":"2024-01-16T15:20:10.343721+0000","last_clean":"2024-01-16T15:20:10.343721+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:20:10.343721+0000","last_undegraded":"2024-01-16T15:20:10.343721+0000","last_fullsized":"2024-01-16T15:20:10.343721+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:35.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"118'62","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755362+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:23:49.755362+0000","last_peered":"2024-01-16T15:23:49.755362+0000","last_clean":"2024-01-16T15:23:49.755362+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:23:49.755362+0000","last_undegraded":"2024-01-16T15:23:49.755362+0000","last_fullsized":"2024-01-16T15:23:49.755362+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:31.476276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.877559+0000","last_change":"2024-01-16T15:15:38.298667+0000","last_active":"2024-01-16T15:20:09.877559+0000","last_peered":"2024-01-16T15:20:09.877559+0000","last_clean":"2024-01-16T15:20:09.877559+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:20:09.877559+0000","last_undegraded":"2024-01-16T15:20:09.877559+0000","last_fullsized":"2024-01-16T15:20:09.877559+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"118'56","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742341+0000","last_change":"2024-01-16T15:15:39.326224+0000","last_active":"2024-01-16T15:23:49.742341+0000","last_peered":"2024-01-16T15:23:49.742341+0000","last_clean":"2024-01-16T15:23:49.742341+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:23:49.742341+0000","last_undegraded":"2024-01-16T15:23:49.742341+0000","last_fullsized":"2024-01-16T15:23:49.742341+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:02.679035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:10.054807+0000","last_change":"2024-01-16T15:15:38.278400+0000","last_active":"2024-01-16T15:20:10.054807+0000","last_peered":"2024-01-16T15:20:10.054807+0000","last_clean":"2024-01-16T15:20:10.054807+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:20:10.054807+0000","last_undegraded":"2024-01-16T15:20:10.054807+0000","last_fullsized":"2024-01-16T15:20:10.054807+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:37.301739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"118'70","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756610+0000","last_change":"2024-01-16T15:15:39.325408+0000","last_active":"2024-01-16T15:23:49.756610+0000","last_peered":"2024-01-16T15:23:49.756610+0000","last_clean":"2024-01-16T15:23:49.756610+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:23:49.756610+0000","last_undegraded":"2024-01-16T15:23:49.756610+0000","last_fullsized":"2024-01-16T15:23:49.756610+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:31.021632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.883905+0000","last_change":"2024-01-16T15:15:38.287926+0000","last_active":"2024-01-16T15:19:59.883905+0000","last_peered":"2024-01-16T15:19:59.883905+0000","last_clean":"2024-01-16T15:19:59.883905+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:19:59.883905+0000","last_undegraded":"2024-01-16T15:19:59.883905+0000","last_fullsized":"2024-01-16T15:19:59.883905+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:31.944486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"118'65","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755705+0000","last_change":"2024-01-16T15:15:39.330480+0000","last_active":"2024-01-16T15:23:49.755705+0000","last_peered":"2024-01-16T15:23:49.755705+0000","last_clean":"2024-01-16T15:23:49.755705+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:23:49.755705+0000","last_undegraded":"2024-01-16T15:23:49.755705+0000","last_fullsized":"2024-01-16T15:23:49.755705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:25.204362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:00.052571+0000","last_change":"2024-01-16T15:15:38.276702+0000","last_active":"2024-01-16T15:20:00.052571+0000","last_peered":"2024-01-16T15:20:00.052571+0000","last_clean":"2024-01-16T15:20:00.052571+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:20:00.052571+0000","last_undegraded":"2024-01-16T15:20:00.052571+0000","last_fullsized":"2024-01-16T15:20:00.052571+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"118'57","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749622+0000","last_change":"2024-01-16T15:15:39.316268+0000","last_active":"2024-01-16T15:23:49.749622+0000","last_peered":"2024-01-16T15:23:49.749622+0000","last_clean":"2024-01-16T15:23:49.749622+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:23:49.749622+0000","last_undegraded":"2024-01-16T15:23:49.749622+0000","last_fullsized":"2024-01-16T15:23:49.749622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:43.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692036,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.866104+0000","last_change":"2024-01-16T15:15:38.297590+0000","last_active":"2024-01-16T15:19:59.866104+0000","last_peered":"2024-01-16T15:19:59.866104+0000","last_clean":"2024-01-16T15:19:59.866104+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:19:59.866104+0000","last_undegraded":"2024-01-16T15:19:59.866104+0000","last_fullsized":"2024-01-16T15:19:59.866104+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:18.917382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"118'53","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750693+0000","last_change":"2024-01-16T15:15:39.333001+0000","last_active":"2024-01-16T15:23:49.750693+0000","last_peered":"2024-01-16T15:23:49.750693+0000","last_clean":"2024-01-16T15:23:49.750693+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:23:49.750693+0000","last_undegraded":"2024-01-16T15:23:49.750693+0000","last_fullsized":"2024-01-16T15:23:49.750693+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:48.934538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.958589+0000","last_change":"2024-01-16T15:15:38.291784+0000","last_active":"2024-01-16T15:19:59.958589+0000","last_peered":"2024-01-16T15:19:59.958589+0000","last_clean":"2024-01-16T15:19:59.958589+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:19:59.958589+0000","last_undegraded":"2024-01-16T15:19:59.958589+0000","last_fullsized":"2024-01-16T15:19:59.958589+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"118'61","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758768+0000","last_change":"2024-01-16T15:15:39.320032+0000","last_active":"2024-01-16T15:23:49.758768+0000","last_peered":"2024-01-16T15:23:49.758768+0000","last_clean":"2024-01-16T15:23:49.758768+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:23:49.758768+0000","last_undegraded":"2024-01-16T15:23:49.758768+0000","last_fullsized":"2024-01-16T15:23:49.758768+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:48.099094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"118'29","reported_seq":46,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:57.887847+0000","last_change":"2024-01-16T15:15:38.295797+0000","last_active":"2024-01-16T15:22:57.887847+0000","last_peered":"2024-01-16T15:22:57.887847+0000","last_clean":"2024-01-16T15:22:57.887847+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:22:57.887847+0000","last_undegraded":"2024-01-16T15:22:57.887847+0000","last_fullsized":"2024-01-16T15:22:57.887847+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:49.218031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"118'69","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748821+0000","last_change":"2024-01-16T15:15:55.923114+0000","last_active":"2024-01-16T15:23:49.748821+0000","last_peered":"2024-01-16T15:23:49.748821+0000","last_clean":"2024-01-16T15:23:49.748821+0000","last_became_active":"2024-01-16T15:15:55.914646+0000","last_became_peered":"2024-01-16T15:15:55.914646+0000","last_unstale":"2024-01-16T15:23:49.748821+0000","last_undegraded":"2024-01-16T15:23:49.748821+0000","last_fullsized":"2024-01-16T15:23:49.748821+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.864273+0000","last_change":"2024-01-16T15:15:38.297697+0000","last_active":"2024-01-16T15:19:49.864273+0000","last_peered":"2024-01-16T15:19:49.864273+0000","last_clean":"2024-01-16T15:19:49.864273+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:19:49.864273+0000","last_undegraded":"2024-01-16T15:19:49.864273+0000","last_fullsized":"2024-01-16T15:19:49.864273+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:28.616152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"118'59","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746852+0000","last_change":"2024-01-16T15:15:39.331210+0000","last_active":"2024-01-16T15:23:49.746852+0000","last_peered":"2024-01-16T15:23:49.746852+0000","last_clean":"2024-01-16T15:23:49.746852+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:23:49.746852+0000","last_undegraded":"2024-01-16T15:23:49.746852+0000","last_fullsized":"2024-01-16T15:23:49.746852+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:29.839201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.956236+0000","last_change":"2024-01-16T15:15:38.287424+0000","last_active":"2024-01-16T15:19:49.956236+0000","last_peered":"2024-01-16T15:19:49.956236+0000","last_clean":"2024-01-16T15:19:49.956236+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:19:49.956236+0000","last_undegraded":"2024-01-16T15:19:49.956236+0000","last_fullsized":"2024-01-16T15:19:49.956236+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:00.039099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"118'63","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738719+0000","last_change":"2024-01-16T15:15:39.313583+0000","last_active":"2024-01-16T15:23:49.738719+0000","last_peered":"2024-01-16T15:23:49.738719+0000","last_clean":"2024-01-16T15:23:49.738719+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:23:49.738719+0000","last_undegraded":"2024-01-16T15:23:49.738719+0000","last_fullsized":"2024-01-16T15:23:49.738719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:58.758941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.873843+0000","last_change":"2024-01-16T15:15:38.290534+0000","last_active":"2024-01-16T15:19:49.873843+0000","last_peered":"2024-01-16T15:19:49.873843+0000","last_clean":"2024-01-16T15:19:49.873843+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:19:49.873843+0000","last_undegraded":"2024-01-16T15:19:49.873843+0000","last_fullsized":"2024-01-16T15:19:49.873843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"118'66","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749639+0000","last_change":"2024-01-16T15:15:39.305386+0000","last_active":"2024-01-16T15:23:49.749639+0000","last_peered":"2024-01-16T15:23:49.749639+0000","last_clean":"2024-01-16T15:23:49.749639+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:23:49.749639+0000","last_undegraded":"2024-01-16T15:23:49.749639+0000","last_fullsized":"2024-01-16T15:23:49.749639+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:02.559083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.969401+0000","last_change":"2024-01-16T15:15:38.287042+0000","last_active":"2024-01-16T15:19:49.969401+0000","last_peered":"2024-01-16T15:19:49.969401+0000","last_clean":"2024-01-16T15:19:49.969401+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:19:49.969401+0000","last_undegraded":"2024-01-16T15:19:49.969401+0000","last_fullsized":"2024-01-16T15:19:49.969401+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:50.656301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"118'68","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752622+0000","last_change":"2024-01-16T15:15:39.330274+0000","last_active":"2024-01-16T15:23:49.752622+0000","last_peered":"2024-01-16T15:23:49.752622+0000","last_clean":"2024-01-16T15:23:49.752622+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:23:49.752622+0000","last_undegraded":"2024-01-16T15:23:49.752622+0000","last_fullsized":"2024-01-16T15:23:49.752622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:55:58.229013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"118'31","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.918964+0000","last_change":"2024-01-16T15:15:38.281868+0000","last_active":"2024-01-16T15:21:45.918964+0000","last_peered":"2024-01-16T15:21:45.918964+0000","last_clean":"2024-01-16T15:21:45.918964+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:21:45.918964+0000","last_undegraded":"2024-01-16T15:21:45.918964+0000","last_fullsized":"2024-01-16T15:21:45.918964+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3846,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"118'70","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753269+0000","last_change":"2024-01-16T15:15:39.333842+0000","last_active":"2024-01-16T15:23:49.753269+0000","last_peered":"2024-01-16T15:23:49.753269+0000","last_clean":"2024-01-16T15:23:49.753269+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:23:49.753269+0000","last_undegraded":"2024-01-16T15:23:49.753269+0000","last_fullsized":"2024-01-16T15:23:49.753269+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:38.845770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:39.887790+0000","last_change":"2024-01-16T15:15:38.297094+0000","last_active":"2024-01-16T15:21:39.887790+0000","last_peered":"2024-01-16T15:21:39.887790+0000","last_clean":"2024-01-16T15:21:39.887790+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:21:39.887790+0000","last_undegraded":"2024-01-16T15:21:39.887790+0000","last_fullsized":"2024-01-16T15:21:39.887790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:48.520471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"118'49","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753143+0000","last_change":"2024-01-16T15:15:39.324548+0000","last_active":"2024-01-16T15:23:49.753143+0000","last_peered":"2024-01-16T15:23:49.753143+0000","last_clean":"2024-01-16T15:23:49.753143+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:23:49.753143+0000","last_undegraded":"2024-01-16T15:23:49.753143+0000","last_fullsized":"2024-01-16T15:23:49.753143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:57.259350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:19.879133+0000","last_change":"2024-01-16T15:15:38.299858+0000","last_active":"2024-01-16T15:20:19.879133+0000","last_peered":"2024-01-16T15:20:19.879133+0000","last_clean":"2024-01-16T15:20:19.879133+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:20:19.879133+0000","last_undegraded":"2024-01-16T15:20:19.879133+0000","last_fullsized":"2024-01-16T15:20:19.879133+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":264,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:05.151918+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:21:05.151918+0000","last_peered":"2024-01-16T15:21:05.151918+0000","last_clean":"2024-01-16T15:21:05.151918+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:21:05.151918+0000","last_undegraded":"2024-01-16T15:21:05.151918+0000","last_fullsized":"2024-01-16T15:21:05.151918+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452737,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4113,"ondisk_log_size":4113,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1245803,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":484,"num_write_kb":1454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":2491606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2096820,"kb_used_data":2025452,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92275020,"statfs":{"total":96636764160,"available":94489620480,"internally_reserved":0,"allocated":2074062848,"data_stored":2073213012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.238,"15min":1.26},"min":{"1min":0.84099999999999997,"5min":0.70399999999999996,"15min":0.57799999999999996},"max":{"1min":1.6100000000000001,"5min":1.6679999999999999,"15min":13.566000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.323,"15min":1.3089999999999999},"min":{"1min":0.96499999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.984,"5min":1.984,"15min":10.728},"last":1.018}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1200000000000001,"15min":1.159},"min":{"1min":0.74099999999999999,"5min":0.73899999999999999,"15min":0.59699999999999998},"max":{"1min":1.446,"5min":1.714,"15min":13.375},"last":1.25},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1519999999999999,"15min":1.1950000000000001},"min":{"1min":0.86299999999999999,"5min":0.67800000000000005,"15min":0.65900000000000003},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":14.592000000000001},"last":0.90800000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.264,"15min":1.2589999999999999},"min":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.72199999999999998},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":4.9000000000000004},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1910000000000001,"15min":1.2390000000000001},"min":{"1min":0.89800000000000002,"5min":0.77100000000000002,"15min":0.67600000000000005},"max":{"1min":1.4630000000000001,"5min":1.724,"15min":24.806000000000001},"last":0.95199999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2849999999999999,"15min":1.3839999999999999},"min":{"1min":0.86899999999999999,"5min":0.81399999999999995,"15min":0.67000000000000004},"max":{"1min":1.6599999999999999,"5min":2.27,"15min":23.741},"last":1.405},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4650000000000001,"15min":1.508},"min":{"1min":1.1699999999999999,"5min":1.0029999999999999,"15min":0.76500000000000001},"max":{"1min":1.8260000000000001,"5min":2.3119999999999998,"15min":23.777000000000001},"last":1.4490000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.2450000000000001,"15min":1.3220000000000001},"min":{"1min":0.81000000000000005,"5min":0.57599999999999996,"15min":0.45900000000000002},"max":{"1min":1.865,"5min":1.865,"15min":22.355},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.413,"15min":1.516},"min":{"1min":0.85199999999999998,"5min":0.72899999999999998,"15min":0.63600000000000001},"max":{"1min":1.7689999999999999,"5min":1.9239999999999999,"15min":22.271999999999998},"last":1.663}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.4139999999999999,"15min":1.454},"min":{"1min":0.95699999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.8300000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.95199999999999996,"5min":0.66000000000000003,"15min":0.56699999999999995},"max":{"1min":1.8680000000000001,"5min":1.8999999999999999,"15min":7.8540000000000001},"last":1.2150000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4379999999999999,"15min":1.4370000000000001},"min":{"1min":1.042,"5min":0.53000000000000003,"15min":0.50800000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.6389999999999998},"last":1.306},{"interface":"front","average":{"1min":1.488,"5min":1.4410000000000001,"15min":1.5},"min":{"1min":0.98599999999999999,"5min":0.63400000000000001,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":13.725},"last":1.5580000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542468,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477736,"kb_used_data":1406368,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92894104,"statfs":{"total":96636764160,"available":95123562496,"internally_reserved":0,"allocated":1440120832,"data_stored":1439281876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.179,"15min":1.23},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.58099999999999996},"max":{"1min":1.6319999999999999,"5min":1.764,"15min":12.467000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1930000000000001,"15min":1.2410000000000001},"min":{"1min":0.83099999999999996,"5min":0.67100000000000004,"15min":0.41599999999999998},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":8.0630000000000006},"last":0.78100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.226,"15min":1.27},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.497},"max":{"1min":1.6200000000000001,"5min":1.7030000000000001,"15min":10.518000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.278,"5min":1.3100000000000001,"15min":1.2889999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.53100000000000003},"max":{"1min":1.73,"5min":1.887,"15min":7.2919999999999998},"last":1.266}]},{"osd":2,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.254,"15min":1.2909999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":1.784,"15min":6.3040000000000003},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2829999999999999,"15min":1.417},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.57099999999999995},"max":{"1min":1.7470000000000001,"5min":1.8240000000000001,"15min":15.282},"last":0.83099999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.337,"15min":1.359},"min":{"1min":0.95299999999999996,"5min":0.71899999999999997,"15min":0.621},"max":{"1min":1.641,"5min":1.9490000000000001,"15min":4.3659999999999997},"last":1.371},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4139999999999999,"15min":1.458},"min":{"1min":1.159,"5min":0.94999999999999996,"15min":0.50900000000000001},"max":{"1min":1.724,"5min":1.8480000000000001,"15min":9.0960000000000001},"last":0.91200000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.347,"15min":1.379},"min":{"1min":1.026,"5min":0.85999999999999999,"15min":0.49299999999999999},"max":{"1min":1.8580000000000001,"5min":1.895,"15min":5.4669999999999996},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.41199999999999998},"max":{"1min":1.7989999999999999,"5min":1.944,"15min":5.3360000000000003},"last":0.998}]},{"osd":5,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3600000000000001,"15min":1.399},"min":{"1min":0.82799999999999996,"5min":0.65700000000000003,"15min":0.436},"max":{"1min":1.708,"5min":1.9239999999999999,"15min":10.388999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.552,"5min":1.5089999999999999,"15min":1.5469999999999999},"min":{"1min":1.1100000000000001,"5min":1.016,"15min":0.42799999999999999},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":7.9640000000000004},"last":1.6499999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.5429999999999999,"15min":1.577},"min":{"1min":1.05,"5min":0.98399999999999999,"15min":0.46600000000000003},"max":{"1min":1.804,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5029999999999999,"15min":1.6000000000000001},"min":{"1min":0.98199999999999998,"5min":0.84199999999999997,"15min":0.495},"max":{"1min":1.857,"5min":1.984,"15min":16.175999999999998},"last":1.496}]}]},{"osd":5,"up_from":14,"seq":60129542469,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174568,"kb_used_data":1107360,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93197272,"statfs":{"total":96636764160,"available":95434006528,"internally_reserved":0,"allocated":1133936640,"data_stored":1133098096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.278,"15min":1.216},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.532,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.3120000000000001,"15min":1.236},"min":{"1min":0.81899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.657,"5min":12.922000000000001,"15min":12.922000000000001},"last":1.135}]},{"osd":1,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3200000000000001,"15min":1.3169999999999999},"min":{"1min":0.89700000000000002,"5min":0.66300000000000003,"15min":0.63900000000000001},"max":{"1min":1.6879999999999999,"5min":5.2249999999999996,"15min":8.1820000000000004},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.175,"15min":1.173},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.536,"5min":2.0659999999999998,"15min":2.597},"last":1.377}]},{"osd":2,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.268,"15min":1.264},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6459999999999999,"5min":1.712,"15min":3.0470000000000002},"last":1.244},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2250000000000001,"15min":1.2669999999999999},"min":{"1min":0.72299999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6359999999999999,"5min":1.756,"15min":8.0619999999999994},"last":1.1890000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.401,"15min":1.484},"min":{"1min":0.83099999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7430000000000001,"5min":5.157,"15min":16.931999999999999},"last":1.456},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.419,"15min":1.494},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":2.6960000000000002,"15min":19.111999999999998},"last":1.5580000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1930000000000001,"15min":1.2589999999999999},"min":{"1min":0.71899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.7549999999999999,"5min":1.7689999999999999,"15min":4.8490000000000002},"last":1.071},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3540000000000001,"15min":1.411},"min":{"1min":0.84899999999999998,"5min":0.63300000000000001,"15min":0.57399999999999995},"max":{"1min":1.7569999999999999,"5min":5.0810000000000004,"15min":8.3550000000000004},"last":1.329}]},{"osd":6,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4119999999999999,"15min":1.5069999999999999},"min":{"1min":0.79300000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.716,"5min":1.9530000000000001,"15min":17.419},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.415,"15min":1.5269999999999999},"min":{"1min":0.83599999999999997,"5min":0.69899999999999995,"15min":0.69799999999999995},"max":{"1min":1.7170000000000001,"5min":2.2959999999999998,"15min":16.262},"last":1.304}]},{"osd":7,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.6040000000000001},"min":{"1min":1.1359999999999999,"5min":0.68400000000000005,"15min":0.67000000000000004},"max":{"1min":1.889,"5min":4.0380000000000003,"15min":12.385999999999999},"last":1.593},{"interface":"front","average":{"1min":1.429,"5min":1.431,"15min":1.512},"min":{"1min":0.73999999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6919999999999999,"5min":1.796,"15min":5.0780000000000003},"last":1.6180000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542470,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936484,"kb_used_data":1865116,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435356,"statfs":{"total":96636764160,"available":94653804544,"internally_reserved":0,"allocated":1909878784,"data_stored":1909043912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.272,"15min":1.323},"min":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.63300000000000001},"max":{"1min":1.756,"5min":1.774,"15min":14.339},"last":1.462},{"interface":"front","average":{"1min":1.056,"5min":1.111,"15min":1.1899999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.47499999999999998},"max":{"1min":1.508,"5min":1.704,"15min":5.7430000000000003},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2729999999999999,"15min":1.304},"min":{"1min":0.94699999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6020000000000001,"5min":1.8149999999999999,"15min":3.7850000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.218,"15min":1.2889999999999999},"min":{"1min":0.88500000000000001,"5min":0.80500000000000005,"15min":0.57899999999999996},"max":{"1min":1.55,"5min":1.696,"15min":14.298},"last":1.2210000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.2150000000000001,"15min":1.238},"min":{"1min":0.81399999999999995,"5min":0.79600000000000004,"15min":0.69199999999999995},"max":{"1min":1.6200000000000001,"5min":1.6990000000000001,"15min":13.731999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.365,"15min":1.4179999999999999},"min":{"1min":0.999,"5min":0.79400000000000004,"15min":0.73199999999999998},"max":{"1min":1.635,"5min":1.7729999999999999,"15min":14.374000000000001},"last":1.387}]},{"osd":3,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.329,"15min":1.3720000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59899999999999998},"max":{"1min":1.6439999999999999,"5min":1.837,"15min":12.347},"last":1.266},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3100000000000001,"15min":1.369},"min":{"1min":0.93000000000000005,"5min":0.92300000000000004,"15min":0.71699999999999997},"max":{"1min":1.6299999999999999,"5min":1.7430000000000001,"15min":13.914},"last":1.165}]},{"osd":5,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.3220000000000001,"15min":1.4099999999999999},"min":{"1min":0.98199999999999998,"5min":0.80500000000000005,"15min":0.47299999999999998},"max":{"1min":1.613,"5min":2.0190000000000001,"15min":14.24},"last":1.071},{"interface":"front","average":{"1min":1.413,"5min":1.3740000000000001,"15min":1.405},"min":{"1min":1.0940000000000001,"5min":0.87,"15min":0.58299999999999996},"max":{"1min":1.7170000000000001,"5min":1.8480000000000001,"15min":4.673},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":1.0289999999999999,"5min":0.83099999999999996,"15min":0.60699999999999998},"max":{"1min":1.7270000000000001,"5min":1.7410000000000001,"15min":18.033000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.526,"15min":1.589},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.52400000000000002},"max":{"1min":1.6870000000000001,"5min":1.9019999999999999,"15min":14.412000000000001},"last":1.5149999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.518,"15min":1.5489999999999999},"min":{"1min":1.052,"5min":1.052,"15min":0.53500000000000003},"max":{"1min":1.6479999999999999,"5min":2.1269999999999998,"15min":4.7640000000000002},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.5249999999999999,"15min":1.6020000000000001},"min":{"1min":1.202,"5min":0.95999999999999996,"15min":0.55200000000000005},"max":{"1min":1.8999999999999999,"5min":2.0179999999999998,"15min":13.852},"last":1.3240000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542469,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1849396,"kb_used_data":1751660,"kb_used_omap":1,"kb_used_meta":97726,"kb_avail":92522444,"statfs":{"total":96636764160,"available":94742982656,"internally_reserved":0,"allocated":1793699840,"data_stored":1792845160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":100071879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94699999999999995,"15min":0.95599999999999996},"min":{"1min":0.71299999999999997,"5min":0.626,"15min":0.34300000000000003},"max":{"1min":1.353,"5min":2.605,"15min":2.9129999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1739999999999999,"15min":1.143},"min":{"1min":0.438,"5min":0.438,"15min":0.29699999999999999},"max":{"1min":1.2849999999999999,"5min":4.0389999999999997,"15min":12.292999999999999},"last":0.92000000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.069,"15min":0.98899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.33100000000000002},"max":{"1min":1.5580000000000001,"5min":12.335000000000001,"15min":12.335000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0960000000000001,"15min":1.133},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.315},"max":{"1min":1.712,"5min":4.6529999999999996,"15min":10.638},"last":0.85399999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.206,"15min":1.151},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":8.3740000000000006,"15min":8.3740000000000006},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.248,"15min":1.228},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.46000000000000002},"max":{"1min":1.6200000000000001,"5min":4.2629999999999999,"15min":4.2629999999999999},"last":1}]},{"osd":4,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.403,"15min":1.345},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.39300000000000002},"max":{"1min":1.76,"5min":15.512,"15min":15.512},"last":1.107},{"interface":"front","average":{"1min":1.274,"5min":1.411,"15min":1.4119999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.52100000000000002},"max":{"1min":1.9399999999999999,"5min":11.743,"15min":14.250999999999999},"last":1.218}]},{"osd":5,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4710000000000001,"15min":1.417},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7470000000000001,"5min":8.4290000000000003,"15min":8.4290000000000003},"last":1.319},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.419,"15min":1.409},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.42099999999999999},"max":{"1min":1.8440000000000001,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":1.1579999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3839999999999999,"15min":1.419},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.41499999999999998},"max":{"1min":1.8879999999999999,"5min":3.956,"15min":12.675000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.5669999999999999,"15min":1.49},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.54900000000000004},"max":{"1min":1.849,"5min":11.667,"15min":11.667},"last":1.0449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.569,"15min":1.5569999999999999},"min":{"1min":1.113,"5min":1.097,"15min":0.53700000000000003},"max":{"1min":1.9279999999999999,"5min":11.042999999999999,"15min":11.042999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.474,"15min":1.5289999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.435},"max":{"1min":1.954,"5min":3.907,"15min":12.836},"last":1.3520000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542469,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1211492,"kb_used_data":1140124,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93160348,"statfs":{"total":96636764160,"available":95396196352,"internally_reserved":0,"allocated":1167486976,"data_stored":1166655701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.311},"max":{"1min":1.516,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0680000000000001,"15min":1.056},"min":{"1min":0.51600000000000001,"5min":0.438,"15min":0.318},"max":{"1min":1.51,"5min":1.6240000000000001,"15min":3.52},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.169,"15min":1.095},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.434},"max":{"1min":1.5820000000000001,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":1.113},{"interface":"front","average":{"1min":1.05,"5min":1.1200000000000001,"15min":1.119},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.45400000000000001},"max":{"1min":1.4690000000000001,"5min":1.8109999999999999,"15min":3.7759999999999998},"last":0.72399999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0609999999999999,"15min":1.0509999999999999},"min":{"1min":0.68600000000000005,"5min":0.67400000000000004,"15min":0.45700000000000002},"max":{"1min":1.4039999999999999,"5min":1.8360000000000001,"15min":6.1260000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.236,"15min":1.228},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.47999999999999998},"max":{"1min":1.4970000000000001,"5min":2.911,"15min":6.2510000000000003},"last":0.85399999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4370000000000001,"15min":1.4259999999999999},"min":{"1min":1.1419999999999999,"5min":1.0980000000000001,"15min":0.68400000000000005},"max":{"1min":1.641,"5min":3.012,"15min":3.012},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.423,"15min":1.4079999999999999},"min":{"1min":0.94599999999999995,"5min":0.83099999999999996,"15min":0.71099999999999997},"max":{"1min":1.7430000000000001,"5min":3.54,"15min":3.54},"last":1.089}]},{"osd":5,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.381,"15min":1.3919999999999999},"min":{"1min":1.0449999999999999,"5min":0.90700000000000003,"15min":0.61099999999999999},"max":{"1min":1.677,"5min":3.6179999999999999,"15min":8.625},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.327,"15min":1.365},"min":{"1min":0.96899999999999997,"5min":0.82099999999999995,"15min":0.80600000000000005},"max":{"1min":1.6850000000000001,"5min":1.8680000000000001,"15min":13.760999999999999},"last":1.2829999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3959999999999999,"15min":1.3839999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.71499999999999997},"max":{"1min":1.871,"5min":2.0539999999999998,"15min":6.4729999999999999},"last":1.163},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.409,"15min":1.4359999999999999},"min":{"1min":1.04,"5min":1.0009999999999999,"15min":0.71099999999999997},"max":{"1min":1.7310000000000001,"5min":2.8260000000000001,"15min":3.2469999999999999},"last":1.228}]},{"osd":7,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.5109999999999999,"15min":1.502},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.871},"max":{"1min":1.724,"5min":3.431,"15min":3.7290000000000001},"last":1.194},{"interface":"front","average":{"1min":1.46,"5min":1.4930000000000001,"15min":1.498},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.89800000000000002},"max":{"1min":1.827,"5min":2.3050000000000002,"15min":6.4249999999999998},"last":1.387}]}]},{"osd":1,"up_from":14,"seq":60129542469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1408104,"kb_used_data":1336736,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92963736,"statfs":{"total":96636764160,"available":95194865664,"internally_reserved":0,"allocated":1368817664,"data_stored":1367982226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.94499999999999995,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.33400000000000002},"max":{"1min":1.3120000000000001,"5min":1.4079999999999999,"15min":8.8499999999999996},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.147,"15min":1.1619999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"max":{"1min":1.5089999999999999,"5min":1.5940000000000001,"15min":8.9380000000000006},"last":1.3160000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.34899999999999998},"max":{"1min":1.2549999999999999,"5min":1.456,"15min":3.4460000000000002},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.123,"15min":1.147},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5840000000000001,"5min":1.677,"15min":2.9660000000000002},"last":1.5840000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.2050000000000001,"15min":1.234},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.45400000000000001},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.282},"last":1.423},{"interface":"front","average":{"1min":1.121,"5min":1.1180000000000001,"15min":1.135},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.47099999999999997},"max":{"1min":1.655,"5min":1.6830000000000001,"15min":8.6910000000000007},"last":1.6299999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.335,"15min":1.3600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.716,"5min":1.8109999999999999,"15min":4.2480000000000002},"last":1.716},{"interface":"front","average":{"1min":1.226,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.57899999999999996},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":4.7640000000000002},"last":1.9199999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.50800000000000001},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3680000000000003},"last":2.2530000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.452,"15min":1.476},"min":{"1min":1.0569999999999999,"5min":0.77500000000000002,"15min":0.66300000000000003},"max":{"1min":1.772,"5min":1.841,"15min":5.4610000000000003},"last":1.772}]},{"osd":6,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":4.7779999999999996},"last":2.0870000000000002},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.365,"15min":1.419},"min":{"1min":0.746,"5min":0.746,"15min":0.56899999999999995},"max":{"1min":2.339,"5min":2.339,"15min":4.8300000000000001},"last":2.339}]},{"osd":7,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4510000000000001,"15min":1.4830000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.58999999999999997},"max":{"1min":2.0510000000000002,"5min":2.081,"15min":6.7990000000000004},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.484,"15min":1.544},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.198,"5min":2.198,"15min":5.1260000000000003},"last":2.198}]}]},{"osd":0,"up_from":14,"seq":60129542470,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2036600,"kb_used_data":1965232,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92335240,"statfs":{"total":96636764160,"available":94551285760,"internally_reserved":0,"allocated":2012397568,"data_stored":2011539049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.34899999999999998},"max":{"1min":1.381,"5min":2.387,"15min":2.7320000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.0269999999999999,"15min":1.0680000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.29399999999999998},"max":{"1min":1.3720000000000001,"5min":1.694,"15min":9.4499999999999993},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2490000000000001,"15min":1.3029999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.45100000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":9.5579999999999998},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.149,"15min":1.1459999999999999},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.40200000000000002},"max":{"1min":1.5960000000000001,"5min":2.4359999999999999,"15min":5.3159999999999998},"last":1.262}]},{"osd":3,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0489999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46600000000000003},"max":{"1min":1.335,"5min":1.5940000000000001,"15min":10.83},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1659999999999999,"15min":1.2529999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":1.5409999999999999,"5min":1.99,"15min":9.6029999999999998},"last":0.996}]},{"osd":4,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3260000000000001,"15min":1.306},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.63400000000000001},"max":{"1min":1.641,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.486},{"interface":"front","average":{"1min":1.2,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.67200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":10.986000000000001},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":1.0089999999999999,"5min":0.78700000000000003,"15min":0.70499999999999996},"max":{"1min":1.665,"5min":3.0259999999999998,"15min":3.0259999999999998},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":1.514,"5min":1.667,"15min":14.561999999999999},"last":1.204}]},{"osd":6,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.47,"15min":1.522},"min":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.76600000000000001},"max":{"1min":1.6639999999999999,"5min":2.2069999999999999,"15min":14.958},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.506,"15min":1.48},"min":{"1min":1.069,"5min":0.89500000000000002,"15min":0.68500000000000005},"max":{"1min":1.7270000000000001,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.6839999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4890000000000001,"15min":1.5640000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.7869999999999999,"5min":2.129,"15min":15.005000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5089999999999999,"15min":1.552},"min":{"1min":1.028,"5min":0.93400000000000005,"15min":0.70699999999999996},"max":{"1min":1.9279999999999999,"5min":3.0590000000000002,"15min":13.718},"last":1.7629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1257472,"data_stored":1241271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1253376,"data_stored":1241229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2009083904,"data_stored":2009072258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166020608,"data_stored":1166016703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908416512,"data_stored":1908408760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132466176,"data_stored":1132462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:56.725 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:56.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.977+0000 7f83adc00640 1 Processor -- start 2024-01-16T15:23:56.974 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.977+0000 7f83adc00640 1 -- start start 2024-01-16T15:23:56.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83adc00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f83a80cf9a0 0x7f83a815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83adc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 0x7f83a80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83adc00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a80b1ee0 con 0x7f83a8143ae0 2024-01-16T15:23:56.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83adc00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a80b2780 con 0x7f83a80cf9a0 2024-01-16T15:23:56.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83adc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a815e990 con 0x7f83a815e1e0 2024-01-16T15:23:56.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83a6a00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f83a8143ae0 0x7f83a80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54322/0 (socket says 172.21.15.141:54322) 2024-01-16T15:23:56.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83a6a00640 1 -- 172.21.15.141:0/1872819946 learned_addr learned my addr 172.21.15.141:0/1872819946 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:56.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83a7400640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 0x7f83a80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f839fe00640 1 --2- 172.21.15.141:0/1872819946 >> v2:172.21.15.188:3300/0 conn(0x7f83a80cf9a0 0x7f83a815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.979 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.979+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018369192 0 0) 0x7f83a80b1ee0 con 0x7f83a8143ae0 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8388003520 con 0x7f83a8143ae0 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4139893608 0 0) 0x7f8388003520 con 0x7f83a8143ae0 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 msgr2=0x7f83a80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 0x7f83a80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 >> v2:172.21.15.188:3300/0 conn(0x7f83a80cf9a0 msgr2=0x7f83a815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 --2- 172.21.15.141:0/1872819946 >> v2:172.21.15.188:3300/0 conn(0x7f83a80cf9a0 0x7f83a815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80b2130 con 0x7f83a8143ae0 2024-01-16T15:23:56.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f8398001670 con 0x7f83a8143ae0 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 >> v1:172.21.15.141:6789/0 conn(0x7f83a8143ae0 legacy=0x7f83a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f8388003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83880047c0 0x7f8388004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 0x7f8388005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80b2780 con 0x7f8388003970 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a815e990 con 0x7f83880047c0 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838800b980 con 0x7f83a815e1e0 2024-01-16T15:23:56.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.980+0000 7f83a6a00640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f8388003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f839fe00640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83880047c0 0x7f8388004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f83a7400640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 0x7f8388005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f83a6a00640 1 -- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 msgr2=0x7f8388005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f83a6a00640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 0x7f8388005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f83a6a00640 1 -- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83880047c0 msgr2=0x7f8388004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f83a6a00640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83880047c0 0x7f8388004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f83a6a00640 1 -- 172.21.15.141:0/1872819946 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8388003520 con 0x7f8388003970 2024-01-16T15:23:56.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.981+0000 7f83a7400640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a815e1e0 0x7f8388005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:56.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.982+0000 7f83a6a00640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f8388003d90 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f8398007c00 tx=0x7f83880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0d8c2de8a77765 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:56.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.982+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839802ec40 con 0x7f8388003970 2024-01-16T15:23:56.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.982+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839802edc0 con 0x7f8388003970 2024-01-16T15:23:56.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.982+0000 7f83a6000640 1 -- 172.21.15.141:0/1872819946 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398005b10 con 0x7f8388003970 2024-01-16T15:23:56.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.983+0000 7f83adc00640 1 -- 172.21.15.141:0/1872819946 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 msgr2=0x7f8388003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.983+0000 7f83adc00640 1 --2- 172.21.15.141:0/1872819946 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f8388003d90 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f8398007c00 tx=0x7f83880057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.983+0000 7f83a7400640 1 -- 172.21.15.141:0/1872819946 reap_dead start 2024-01-16T15:23:56.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.983+0000 7f83adc00640 1 -- 172.21.15.141:0/1872819946 shutdown_connections 2024-01-16T15:23:56.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.983+0000 7f83adc00640 1 -- 172.21.15.141:0/1872819946 >> 172.21.15.141:0/1872819946 conn(0x7f83a806a800 msgr2=0x7f83a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:56.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.983+0000 7f83adc00640 1 -- 172.21.15.141:0/1872819946 shutdown_connections 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.983+0000 7f83adc00640 1 -- 172.21.15.141:0/1872819946 wait complete. 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.985+0000 7f83adc00640 1 Processor -- start 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.985+0000 7f83adc00640 1 -- start start 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83adc00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f83a80afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83adc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a80ac7f0 0x7f83a80aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83adc00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a80aa910 0x7f83a813f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83adc00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80ab370 con 0x7f8388003970 2024-01-16T15:23:56.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83adc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80aadb0 con 0x7f83a80ac7f0 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83adc00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80ab090 con 0x7f83a80aa910 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83a6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f83a80afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83a6a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f83a80afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48584/0 (socket says 172.21.15.141:48584) 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83a6a00640 1 -- 172.21.15.141:0/283768970 learned_addr learned my addr 172.21.15.141:0/283768970 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.986+0000 7f83a7400640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a80aa910 0x7f83a813f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83a6a00640 1 -- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a80aa910 msgr2=0x7f83a813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83a6a00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a80aa910 0x7f83a813f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.988 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f839fe00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a80ac7f0 0x7f83a80aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83a6a00640 1 -- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a80ac7f0 msgr2=0x7f83a80aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:56.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83a6a00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a80ac7f0 0x7f83a80aa3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:56.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83a6a00640 1 -- 172.21.15.141:0/283768970 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80cd7e0 con 0x7f8388003970 2024-01-16T15:23:56.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83a6a00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f83a80afe90 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f839802e6b0 tx=0x7f83a80b26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:56.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f839ea00640 1 -- 172.21.15.141:0/283768970 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839803b070 con 0x7f8388003970 2024-01-16T15:23:56.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f839ea00640 1 -- 172.21.15.141:0/283768970 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398001b90 con 0x7f8388003970 2024-01-16T15:23:56.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f839ea00640 1 -- 172.21.15.141:0/283768970 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398007270 con 0x7f8388003970 2024-01-16T15:23:56.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a80cdac0 con 0x7f8388003970 2024-01-16T15:23:56.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.987+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a80b03d0 con 0x7f8388003970 2024-01-16T15:23:56.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.989+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a80b05c0 con 0x7f8388003970 2024-01-16T15:23:56.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.990+0000 7f839ea00640 1 -- 172.21.15.141:0/283768970 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f8398034070 con 0x7f8388003970 2024-01-16T15:23:56.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.991+0000 7f839ea00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f838006c610 0x7f838006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:56.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.991+0000 7f839ea00640 1 -- 172.21.15.141:0/283768970 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f839803e040 con 0x7f8388003970 2024-01-16T15:23:56.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.991+0000 7f839fe00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f838006c610 0x7f838006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:56.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.992+0000 7f839fe00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f838006c610 0x7f838006ead0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f83a815c8d0 tx=0x7f8390008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:56.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:56.995+0000 7f839ea00640 1 -- 172.21.15.141:0/283768970 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83980783f0 con 0x7f8388003970 2024-01-16T15:23:57.181 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.184+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83a81f2370 con 0x7f838006c610 2024-01-16T15:23:57.210 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.212+0000 7f839ea00640 1 -- 172.21.15.141:0/283768970 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289287 (secure 0 0 0) 0x7f83a81f2370 con 0x7f838006c610 2024-01-16T15:23:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:57.217 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:23:57.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f838006c610 msgr2=0x7f838006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f838006c610 0x7f838006ead0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f83a815c8d0 tx=0x7f8390008040 comp rx=0 tx=0).stop 2024-01-16T15:23:57.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 msgr2=0x7f83a80afe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f83a80afe90 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f839802e6b0 tx=0x7f83a80b26f0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 shutdown_connections 2024-01-16T15:23:57.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f838006c610 0x7f838006ead0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f83a80aa910 0x7f83a813f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a80ac7f0 0x7f83a80aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 --2- 172.21.15.141:0/283768970 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8388003970 0x7f83a80afe90 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 >> 172.21.15.141:0/283768970 conn(0x7f83a806a800 msgr2=0x7f83a815a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:57.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 shutdown_connections 2024-01-16T15:23:57.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.219+0000 7f83adc00640 1 -- 172.21.15.141:0/283768970 wait complete. 2024-01-16T15:23:57.237 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":920,"stamp":"2024-01-16T15:23:57.121187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444288908,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":945,"num_read_kb":3145889,"num_write":2250,"num_write_kb":6295364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4680,"ondisk_log_size":4680,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13191200,"kb_used_data":12598048,"kb_used_omap":12,"kb_used_meta":593075,"kb_avail":741783520,"statfs":{"total":773094113280,"available":759586324480,"internally_reserved":0,"allocated":12900401152,"data_stored":12893659032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":607309362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073781653,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":276,"num_write_kb":1048622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004674"},"pg_stats":[{"pgid":"17.3f","version":"118'70","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758782+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:23:49.758782+0000","last_peered":"2024-01-16T15:23:49.758782+0000","last_clean":"2024-01-16T15:23:49.758782+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:23:49.758782+0000","last_undegraded":"2024-01-16T15:23:49.758782+0000","last_fullsized":"2024-01-16T15:23:49.758782+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:39.215878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"118'69","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.745868+0000","last_change":"2024-01-16T15:15:39.327644+0000","last_active":"2024-01-16T15:23:49.745868+0000","last_peered":"2024-01-16T15:23:49.745868+0000","last_clean":"2024-01-16T15:23:49.745868+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:23:49.745868+0000","last_undegraded":"2024-01-16T15:23:49.745868+0000","last_fullsized":"2024-01-16T15:23:49.745868+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:39.747172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"118'67","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.754526+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:23:49.754526+0000","last_peered":"2024-01-16T15:23:49.754526+0000","last_clean":"2024-01-16T15:23:49.754526+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:23:49.754526+0000","last_undegraded":"2024-01-16T15:23:49.754526+0000","last_fullsized":"2024-01-16T15:23:49.754526+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:22.535735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"118'66","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738937+0000","last_change":"2024-01-16T15:15:55.875926+0000","last_active":"2024-01-16T15:23:49.738937+0000","last_peered":"2024-01-16T15:23:49.738937+0000","last_clean":"2024-01-16T15:23:49.738937+0000","last_became_active":"2024-01-16T15:15:55.875179+0000","last_became_peered":"2024-01-16T15:15:55.875179+0000","last_unstale":"2024-01-16T15:23:49.738937+0000","last_undegraded":"2024-01-16T15:23:49.738937+0000","last_fullsized":"2024-01-16T15:23:49.738937+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3b","version":"118'60","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737805+0000","last_change":"2024-01-16T15:15:39.321081+0000","last_active":"2024-01-16T15:23:49.737805+0000","last_peered":"2024-01-16T15:23:49.737805+0000","last_clean":"2024-01-16T15:23:49.737805+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:23:49.737805+0000","last_undegraded":"2024-01-16T15:23:49.737805+0000","last_fullsized":"2024-01-16T15:23:49.737805+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663487,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"118'73","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.743047+0000","last_change":"2024-01-16T15:15:39.327813+0000","last_active":"2024-01-16T15:23:49.743047+0000","last_peered":"2024-01-16T15:23:49.743047+0000","last_clean":"2024-01-16T15:23:49.743047+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:23:49.743047+0000","last_undegraded":"2024-01-16T15:23:49.743047+0000","last_fullsized":"2024-01-16T15:23:49.743047+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:39:32.420035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"118'66","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751785+0000","last_change":"2024-01-16T15:15:39.324707+0000","last_active":"2024-01-16T15:23:49.751785+0000","last_peered":"2024-01-16T15:23:49.751785+0000","last_clean":"2024-01-16T15:23:49.751785+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:23:49.751785+0000","last_undegraded":"2024-01-16T15:23:49.751785+0000","last_fullsized":"2024-01-16T15:23:49.751785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303428,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"118'54","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752934+0000","last_change":"2024-01-16T15:15:39.328547+0000","last_active":"2024-01-16T15:23:49.752934+0000","last_peered":"2024-01-16T15:23:49.752934+0000","last_clean":"2024-01-16T15:23:49.752934+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:23:49.752934+0000","last_undegraded":"2024-01-16T15:23:49.752934+0000","last_fullsized":"2024-01-16T15:23:49.752934+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:20.486476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"118'79","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755131+0000","last_change":"2024-01-16T15:15:39.320236+0000","last_active":"2024-01-16T15:23:49.755131+0000","last_peered":"2024-01-16T15:23:49.755131+0000","last_clean":"2024-01-16T15:23:49.755131+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:23:49.755131+0000","last_undegraded":"2024-01-16T15:23:49.755131+0000","last_fullsized":"2024-01-16T15:23:49.755131+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:19:06.417692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052164,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"118'66","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758281+0000","last_change":"2024-01-16T15:15:39.326528+0000","last_active":"2024-01-16T15:23:49.758281+0000","last_peered":"2024-01-16T15:23:49.758281+0000","last_clean":"2024-01-16T15:23:49.758281+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:23:49.758281+0000","last_undegraded":"2024-01-16T15:23:49.758281+0000","last_fullsized":"2024-01-16T15:23:49.758281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:01.161905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"118'70","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737879+0000","last_change":"2024-01-16T15:15:39.325379+0000","last_active":"2024-01-16T15:23:49.737879+0000","last_peered":"2024-01-16T15:23:49.737879+0000","last_clean":"2024-01-16T15:23:49.737879+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:23:49.737879+0000","last_undegraded":"2024-01-16T15:23:49.737879+0000","last_fullsized":"2024-01-16T15:23:49.737879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:11.903533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"118'59","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757983+0000","last_change":"2024-01-16T15:15:39.317978+0000","last_active":"2024-01-16T15:23:49.757983+0000","last_peered":"2024-01-16T15:23:49.757983+0000","last_clean":"2024-01-16T15:23:49.757983+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:23:49.757983+0000","last_undegraded":"2024-01-16T15:23:49.757983+0000","last_fullsized":"2024-01-16T15:23:49.757983+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:47.669894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"118'57","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746095+0000","last_change":"2024-01-16T15:15:39.329570+0000","last_active":"2024-01-16T15:23:49.746095+0000","last_peered":"2024-01-16T15:23:49.746095+0000","last_clean":"2024-01-16T15:23:49.746095+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:23:49.746095+0000","last_undegraded":"2024-01-16T15:23:49.746095+0000","last_fullsized":"2024-01-16T15:23:49.746095+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:22.565771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"118'65","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742239+0000","last_change":"2024-01-16T15:15:39.325613+0000","last_active":"2024-01-16T15:23:49.742239+0000","last_peered":"2024-01-16T15:23:49.742239+0000","last_clean":"2024-01-16T15:23:49.742239+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:23:49.742239+0000","last_undegraded":"2024-01-16T15:23:49.742239+0000","last_fullsized":"2024-01-16T15:23:49.742239+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:08.369872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"118'54","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741582+0000","last_change":"2024-01-16T15:15:39.333121+0000","last_active":"2024-01-16T15:23:49.741582+0000","last_peered":"2024-01-16T15:23:49.741582+0000","last_clean":"2024-01-16T15:23:49.741582+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:23:49.741582+0000","last_undegraded":"2024-01-16T15:23:49.741582+0000","last_fullsized":"2024-01-16T15:23:49.741582+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:19.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"118'58","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756735+0000","last_change":"2024-01-16T15:15:39.313940+0000","last_active":"2024-01-16T15:23:49.756735+0000","last_peered":"2024-01-16T15:23:49.756735+0000","last_clean":"2024-01-16T15:23:49.756735+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:23:49.756735+0000","last_undegraded":"2024-01-16T15:23:49.756735+0000","last_fullsized":"2024-01-16T15:23:49.756735+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:57.259350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"118'49","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740185+0000","last_change":"2024-01-16T15:15:39.329528+0000","last_active":"2024-01-16T15:23:49.740185+0000","last_peered":"2024-01-16T15:23:49.740185+0000","last_clean":"2024-01-16T15:23:49.740185+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:23:49.740185+0000","last_undegraded":"2024-01-16T15:23:49.740185+0000","last_fullsized":"2024-01-16T15:23:49.740185+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:42.153458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"118'72","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742141+0000","last_change":"2024-01-16T15:15:39.330676+0000","last_active":"2024-01-16T15:23:49.742141+0000","last_peered":"2024-01-16T15:23:49.742141+0000","last_clean":"2024-01-16T15:23:49.742141+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:23:49.742141+0000","last_undegraded":"2024-01-16T15:23:49.742141+0000","last_fullsized":"2024-01-16T15:23:49.742141+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:39:04.346629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"118'71","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751357+0000","last_change":"2024-01-16T15:15:39.330437+0000","last_active":"2024-01-16T15:23:49.751357+0000","last_peered":"2024-01-16T15:23:49.751357+0000","last_clean":"2024-01-16T15:23:49.751357+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:23:49.751357+0000","last_undegraded":"2024-01-16T15:23:49.751357+0000","last_fullsized":"2024-01-16T15:23:49.751357+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:37.311014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"118'67","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750922+0000","last_change":"2024-01-16T15:15:39.318689+0000","last_active":"2024-01-16T15:23:49.750922+0000","last_peered":"2024-01-16T15:23:49.750922+0000","last_clean":"2024-01-16T15:23:49.750922+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:23:49.750922+0000","last_undegraded":"2024-01-16T15:23:49.750922+0000","last_fullsized":"2024-01-16T15:23:49.750922+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:35.351694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"118'73","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749394+0000","last_change":"2024-01-16T15:15:39.321055+0000","last_active":"2024-01-16T15:23:49.749394+0000","last_peered":"2024-01-16T15:23:49.749394+0000","last_clean":"2024-01-16T15:23:49.749394+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:23:49.749394+0000","last_undegraded":"2024-01-16T15:23:49.749394+0000","last_fullsized":"2024-01-16T15:23:49.749394+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:46.770618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"118'64","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749195+0000","last_change":"2024-01-16T15:15:39.323290+0000","last_active":"2024-01-16T15:23:49.749195+0000","last_peered":"2024-01-16T15:23:49.749195+0000","last_clean":"2024-01-16T15:23:49.749195+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:23:49.749195+0000","last_undegraded":"2024-01-16T15:23:49.749195+0000","last_fullsized":"2024-01-16T15:23:49.749195+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:48:04.184635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"118'57","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737467+0000","last_change":"2024-01-16T15:15:39.317974+0000","last_active":"2024-01-16T15:23:49.737467+0000","last_peered":"2024-01-16T15:23:49.737467+0000","last_clean":"2024-01-16T15:23:49.737467+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:23:49.737467+0000","last_undegraded":"2024-01-16T15:23:49.737467+0000","last_fullsized":"2024-01-16T15:23:49.737467+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:34.399913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"118'70","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753684+0000","last_change":"2024-01-16T15:15:39.324363+0000","last_active":"2024-01-16T15:23:49.753684+0000","last_peered":"2024-01-16T15:23:49.753684+0000","last_clean":"2024-01-16T15:23:49.753684+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:23:49.753684+0000","last_undegraded":"2024-01-16T15:23:49.753684+0000","last_fullsized":"2024-01-16T15:23:49.753684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:12.878416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"118'71","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741193+0000","last_change":"2024-01-16T15:15:39.328896+0000","last_active":"2024-01-16T15:23:49.741193+0000","last_peered":"2024-01-16T15:23:49.741193+0000","last_clean":"2024-01-16T15:23:49.741193+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:23:49.741193+0000","last_undegraded":"2024-01-16T15:23:49.741193+0000","last_fullsized":"2024-01-16T15:23:49.741193+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:41.474484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"118'81","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756330+0000","last_change":"2024-01-16T15:15:39.317589+0000","last_active":"2024-01-16T15:23:49.756330+0000","last_peered":"2024-01-16T15:23:49.756330+0000","last_clean":"2024-01-16T15:23:49.756330+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:23:49.756330+0000","last_undegraded":"2024-01-16T15:23:49.756330+0000","last_fullsized":"2024-01-16T15:23:49.756330+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:36.235755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"118'64","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746236+0000","last_change":"2024-01-16T15:15:39.335455+0000","last_active":"2024-01-16T15:23:49.746236+0000","last_peered":"2024-01-16T15:23:49.746236+0000","last_clean":"2024-01-16T15:23:49.746236+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:23:49.746236+0000","last_undegraded":"2024-01-16T15:23:49.746236+0000","last_fullsized":"2024-01-16T15:23:49.746236+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:38.284829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"118'43","reported_seq":64,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749120+0000","last_change":"2024-01-16T15:15:39.317292+0000","last_active":"2024-01-16T15:23:49.749120+0000","last_peered":"2024-01-16T15:23:49.749120+0000","last_clean":"2024-01-16T15:23:49.749120+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:23:49.749120+0000","last_undegraded":"2024-01-16T15:23:49.749120+0000","last_fullsized":"2024-01-16T15:23:49.749120+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:38.845770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"118'57","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751058+0000","last_change":"2024-01-16T15:15:39.320340+0000","last_active":"2024-01-16T15:23:49.751058+0000","last_peered":"2024-01-16T15:23:49.751058+0000","last_clean":"2024-01-16T15:23:49.751058+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:23:49.751058+0000","last_undegraded":"2024-01-16T15:23:49.751058+0000","last_fullsized":"2024-01-16T15:23:49.751058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:18.946850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"118'76","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740849+0000","last_change":"2024-01-16T15:15:39.317544+0000","last_active":"2024-01-16T15:23:49.740849+0000","last_peered":"2024-01-16T15:23:49.740849+0000","last_clean":"2024-01-16T15:23:49.740849+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:23:49.740849+0000","last_undegraded":"2024-01-16T15:23:49.740849+0000","last_fullsized":"2024-01-16T15:23:49.740849+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:38.315912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"118'81","reported_seq":116,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757053+0000","last_change":"2024-01-16T15:15:39.313788+0000","last_active":"2024-01-16T15:23:49.757053+0000","last_peered":"2024-01-16T15:23:49.757053+0000","last_clean":"2024-01-16T15:23:49.757053+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:23:49.757053+0000","last_undegraded":"2024-01-16T15:23:49.757053+0000","last_fullsized":"2024-01-16T15:23:49.757053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:43.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"118'63","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744908+0000","last_change":"2024-01-16T15:15:39.321572+0000","last_active":"2024-01-16T15:23:49.744908+0000","last_peered":"2024-01-16T15:23:49.744908+0000","last_clean":"2024-01-16T15:23:49.744908+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:23:49.744908+0000","last_undegraded":"2024-01-16T15:23:49.744908+0000","last_fullsized":"2024-01-16T15:23:49.744908+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"118'62","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.733710+0000","last_change":"2024-01-16T15:15:39.329833+0000","last_active":"2024-01-16T15:23:49.733710+0000","last_peered":"2024-01-16T15:23:49.733710+0000","last_clean":"2024-01-16T15:23:49.733710+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:23:49.733710+0000","last_undegraded":"2024-01-16T15:23:49.733710+0000","last_fullsized":"2024-01-16T15:23:49.733710+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:51.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'4","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.873920+0000","last_change":"2024-01-16T15:15:38.295673+0000","last_active":"2024-01-16T15:19:49.873920+0000","last_peered":"2024-01-16T15:19:49.873920+0000","last_clean":"2024-01-16T15:19:49.873920+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:19:49.873920+0000","last_undegraded":"2024-01-16T15:19:49.873920+0000","last_fullsized":"2024-01-16T15:19:49.873920+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"118'54","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751652+0000","last_change":"2024-01-16T15:15:39.329620+0000","last_active":"2024-01-16T15:23:49.751652+0000","last_peered":"2024-01-16T15:23:49.751652+0000","last_clean":"2024-01-16T15:23:49.751652+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:23:49.751652+0000","last_undegraded":"2024-01-16T15:23:49.751652+0000","last_fullsized":"2024-01-16T15:23:49.751652+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.326448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:40.158702+0000","last_change":"2024-01-16T15:15:38.292769+0000","last_active":"2024-01-16T15:21:40.158702+0000","last_peered":"2024-01-16T15:21:40.158702+0000","last_clean":"2024-01-16T15:21:40.158702+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:21:40.158702+0000","last_undegraded":"2024-01-16T15:21:40.158702+0000","last_fullsized":"2024-01-16T15:21:40.158702+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:45.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"118'71","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744070+0000","last_change":"2024-01-16T15:15:39.332870+0000","last_active":"2024-01-16T15:23:49.744070+0000","last_peered":"2024-01-16T15:23:49.744070+0000","last_clean":"2024-01-16T15:23:49.744070+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:23:49.744070+0000","last_undegraded":"2024-01-16T15:23:49.744070+0000","last_fullsized":"2024-01-16T15:23:49.744070+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.891495+0000","last_change":"2024-01-16T15:15:38.288495+0000","last_active":"2024-01-16T15:20:39.891495+0000","last_peered":"2024-01-16T15:20:39.891495+0000","last_clean":"2024-01-16T15:20:39.891495+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:20:39.891495+0000","last_undegraded":"2024-01-16T15:20:39.891495+0000","last_fullsized":"2024-01-16T15:20:39.891495+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:07.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"118'68","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753436+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:23:49.753436+0000","last_peered":"2024-01-16T15:23:49.753436+0000","last_clean":"2024-01-16T15:23:49.753436+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:23:49.753436+0000","last_undegraded":"2024-01-16T15:23:49.753436+0000","last_fullsized":"2024-01-16T15:23:49.753436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:21.846652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.891465+0000","last_change":"2024-01-16T15:15:38.288324+0000","last_active":"2024-01-16T15:20:39.891465+0000","last_peered":"2024-01-16T15:20:39.891465+0000","last_clean":"2024-01-16T15:20:39.891465+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:20:39.891465+0000","last_undegraded":"2024-01-16T15:20:39.891465+0000","last_fullsized":"2024-01-16T15:20:39.891465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:31.849193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"118'72","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756110+0000","last_change":"2024-01-16T15:15:39.321037+0000","last_active":"2024-01-16T15:23:49.756110+0000","last_peered":"2024-01-16T15:23:49.756110+0000","last_clean":"2024-01-16T15:23:49.756110+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:23:49.756110+0000","last_undegraded":"2024-01-16T15:23:49.756110+0000","last_fullsized":"2024-01-16T15:23:49.756110+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.687281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.883073+0000","last_change":"2024-01-16T15:15:38.288226+0000","last_active":"2024-01-16T15:20:39.883073+0000","last_peered":"2024-01-16T15:20:39.883073+0000","last_clean":"2024-01-16T15:20:39.883073+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:20:39.883073+0000","last_undegraded":"2024-01-16T15:20:39.883073+0000","last_fullsized":"2024-01-16T15:20:39.883073+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:41.040743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"118'72","reported_seq":108,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752048+0000","last_change":"2024-01-16T15:15:39.330201+0000","last_active":"2024-01-16T15:23:49.752048+0000","last_peered":"2024-01-16T15:23:49.752048+0000","last_clean":"2024-01-16T15:23:49.752048+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:23:49.752048+0000","last_undegraded":"2024-01-16T15:23:49.752048+0000","last_fullsized":"2024-01-16T15:23:49.752048+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:52.162066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.966446+0000","last_change":"2024-01-16T15:15:38.292721+0000","last_active":"2024-01-16T15:20:39.966446+0000","last_peered":"2024-01-16T15:20:39.966446+0000","last_clean":"2024-01-16T15:20:39.966446+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:20:39.966446+0000","last_undegraded":"2024-01-16T15:20:39.966446+0000","last_fullsized":"2024-01-16T15:20:39.966446+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:32.177991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"118'73","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753205+0000","last_change":"2024-01-16T15:15:39.322497+0000","last_active":"2024-01-16T15:23:49.753205+0000","last_peered":"2024-01-16T15:23:49.753205+0000","last_clean":"2024-01-16T15:23:49.753205+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:23:49.753205+0000","last_undegraded":"2024-01-16T15:23:49.753205+0000","last_fullsized":"2024-01-16T15:23:49.753205+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:18.495148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.977194+0000","last_change":"2024-01-16T15:15:38.276895+0000","last_active":"2024-01-16T15:20:29.977194+0000","last_peered":"2024-01-16T15:20:29.977194+0000","last_clean":"2024-01-16T15:20:29.977194+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:20:29.977194+0000","last_undegraded":"2024-01-16T15:20:29.977194+0000","last_fullsized":"2024-01-16T15:20:29.977194+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"118'73","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749092+0000","last_change":"2024-01-16T15:15:39.325224+0000","last_active":"2024-01-16T15:23:49.749092+0000","last_peered":"2024-01-16T15:23:49.749092+0000","last_clean":"2024-01-16T15:23:49.749092+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:23:49.749092+0000","last_undegraded":"2024-01-16T15:23:49.749092+0000","last_fullsized":"2024-01-16T15:23:49.749092+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.872263+0000","last_change":"2024-01-16T15:15:38.297269+0000","last_active":"2024-01-16T15:20:29.872263+0000","last_peered":"2024-01-16T15:20:29.872263+0000","last_clean":"2024-01-16T15:20:29.872263+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:20:29.872263+0000","last_undegraded":"2024-01-16T15:20:29.872263+0000","last_fullsized":"2024-01-16T15:20:29.872263+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:02.166771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"118'58","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.735497+0000","last_change":"2024-01-16T15:15:39.320446+0000","last_active":"2024-01-16T15:23:49.735497+0000","last_peered":"2024-01-16T15:23:49.735497+0000","last_clean":"2024-01-16T15:23:49.735497+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:23:49.735497+0000","last_undegraded":"2024-01-16T15:23:49.735497+0000","last_fullsized":"2024-01-16T15:23:49.735497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:15.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"118'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.889799+0000","last_change":"2024-01-16T15:15:38.287453+0000","last_active":"2024-01-16T15:20:29.889799+0000","last_peered":"2024-01-16T15:20:29.889799+0000","last_clean":"2024-01-16T15:20:29.889799+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:20:29.889799+0000","last_undegraded":"2024-01-16T15:20:29.889799+0000","last_fullsized":"2024-01-16T15:20:29.889799+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"118'61","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747524+0000","last_change":"2024-01-16T15:15:39.333810+0000","last_active":"2024-01-16T15:23:49.747524+0000","last_peered":"2024-01-16T15:23:49.747524+0000","last_clean":"2024-01-16T15:23:49.747524+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:23:49.747524+0000","last_undegraded":"2024-01-16T15:23:49.747524+0000","last_fullsized":"2024-01-16T15:23:49.747524+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:22:34.785294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.964575+0000","last_change":"2024-01-16T15:15:38.293583+0000","last_active":"2024-01-16T15:20:29.964575+0000","last_peered":"2024-01-16T15:20:29.964575+0000","last_clean":"2024-01-16T15:20:29.964575+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:20:29.964575+0000","last_undegraded":"2024-01-16T15:20:29.964575+0000","last_fullsized":"2024-01-16T15:20:29.964575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:48.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"118'62","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738579+0000","last_change":"2024-01-16T15:15:39.319368+0000","last_active":"2024-01-16T15:23:49.738579+0000","last_peered":"2024-01-16T15:23:49.738579+0000","last_clean":"2024-01-16T15:23:49.738579+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:23:49.738579+0000","last_undegraded":"2024-01-16T15:23:49.738579+0000","last_fullsized":"2024-01-16T15:23:49.738579+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:31.021632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"118'31","reported_seq":48,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.966626+0000","last_change":"2024-01-16T15:15:38.295750+0000","last_active":"2024-01-16T15:21:45.966626+0000","last_peered":"2024-01-16T15:21:45.966626+0000","last_clean":"2024-01-16T15:21:45.966626+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:21:45.966626+0000","last_undegraded":"2024-01-16T15:21:45.966626+0000","last_fullsized":"2024-01-16T15:21:45.966626+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:17.036375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"118'69","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750139+0000","last_change":"2024-01-16T15:15:39.312858+0000","last_active":"2024-01-16T15:23:49.750139+0000","last_peered":"2024-01-16T15:23:49.750139+0000","last_clean":"2024-01-16T15:23:49.750139+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:23:49.750139+0000","last_undegraded":"2024-01-16T15:23:49.750139+0000","last_fullsized":"2024-01-16T15:23:49.750139+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"118'325","reported_seq":342,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.761469+0000","last_change":"2024-01-16T15:15:38.296732+0000","last_active":"2024-01-16T15:23:49.761469+0000","last_peered":"2024-01-16T15:23:49.761469+0000","last_clean":"2024-01-16T15:23:49.761469+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:23:49.761469+0000","last_undegraded":"2024-01-16T15:23:49.761469+0000","last_fullsized":"2024-01-16T15:23:49.761469+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1241159,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":1369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"118'54","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737844+0000","last_change":"2024-01-16T15:15:39.312723+0000","last_active":"2024-01-16T15:23:49.737844+0000","last_peered":"2024-01-16T15:23:49.737844+0000","last_clean":"2024-01-16T15:23:49.737844+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:23:49.737844+0000","last_undegraded":"2024-01-16T15:23:49.737844+0000","last_fullsized":"2024-01-16T15:23:49.737844+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:21.305831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.961056+0000","last_change":"2024-01-16T15:15:38.280236+0000","last_active":"2024-01-16T15:21:45.961056+0000","last_peered":"2024-01-16T15:21:45.961056+0000","last_clean":"2024-01-16T15:21:45.961056+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:21:45.961056+0000","last_undegraded":"2024-01-16T15:21:45.961056+0000","last_fullsized":"2024-01-16T15:21:45.961056+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:14.715313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"118'72","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753729+0000","last_change":"2024-01-16T15:15:39.324127+0000","last_active":"2024-01-16T15:23:49.753729+0000","last_peered":"2024-01-16T15:23:49.753729+0000","last_clean":"2024-01-16T15:23:49.753729+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:23:49.753729+0000","last_undegraded":"2024-01-16T15:23:49.753729+0000","last_fullsized":"2024-01-16T15:23:49.753729+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.687281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:20.345465+0000","last_change":"2024-01-16T15:15:38.295857+0000","last_active":"2024-01-16T15:20:20.345465+0000","last_peered":"2024-01-16T15:20:20.345465+0000","last_clean":"2024-01-16T15:20:20.345465+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:20:20.345465+0000","last_undegraded":"2024-01-16T15:20:20.345465+0000","last_fullsized":"2024-01-16T15:20:20.345465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:31.063560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"118'64","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.743542+0000","last_change":"2024-01-16T15:15:39.319495+0000","last_active":"2024-01-16T15:23:49.743542+0000","last_peered":"2024-01-16T15:23:49.743542+0000","last_clean":"2024-01-16T15:23:49.743542+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:23:49.743542+0000","last_undegraded":"2024-01-16T15:23:49.743542+0000","last_fullsized":"2024-01-16T15:23:49.743542+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:18.495148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":40961,"num_write":27,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'8","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:19.887701+0000","last_change":"2024-01-16T15:15:38.293904+0000","last_active":"2024-01-16T15:20:19.887701+0000","last_peered":"2024-01-16T15:20:19.887701+0000","last_clean":"2024-01-16T15:20:19.887701+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:20:19.887701+0000","last_undegraded":"2024-01-16T15:20:19.887701+0000","last_fullsized":"2024-01-16T15:20:19.887701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"118'46","reported_seq":68,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758065+0000","last_change":"2024-01-16T15:15:39.314200+0000","last_active":"2024-01-16T15:23:49.758065+0000","last_peered":"2024-01-16T15:23:49.758065+0000","last_clean":"2024-01-16T15:23:49.758065+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:23:49.758065+0000","last_undegraded":"2024-01-16T15:23:49.758065+0000","last_fullsized":"2024-01-16T15:23:49.758065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.868373+0000","last_change":"2024-01-16T15:15:38.296521+0000","last_active":"2024-01-16T15:20:09.868373+0000","last_peered":"2024-01-16T15:20:09.868373+0000","last_clean":"2024-01-16T15:20:09.868373+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:20:09.868373+0000","last_undegraded":"2024-01-16T15:20:09.868373+0000","last_fullsized":"2024-01-16T15:20:09.868373+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:24.030528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"118'61","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748133+0000","last_change":"2024-01-16T15:15:39.334617+0000","last_active":"2024-01-16T15:23:49.748133+0000","last_peered":"2024-01-16T15:23:49.748133+0000","last_clean":"2024-01-16T15:23:49.748133+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:23:49.748133+0000","last_undegraded":"2024-01-16T15:23:49.748133+0000","last_fullsized":"2024-01-16T15:23:49.748133+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:40.033000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.886136+0000","last_change":"2024-01-16T15:15:38.289440+0000","last_active":"2024-01-16T15:20:09.886136+0000","last_peered":"2024-01-16T15:20:09.886136+0000","last_clean":"2024-01-16T15:20:09.886136+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:20:09.886136+0000","last_undegraded":"2024-01-16T15:20:09.886136+0000","last_fullsized":"2024-01-16T15:20:09.886136+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"118'61","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755433+0000","last_change":"2024-01-16T15:15:39.332704+0000","last_active":"2024-01-16T15:23:49.755433+0000","last_peered":"2024-01-16T15:23:49.755433+0000","last_clean":"2024-01-16T15:23:49.755433+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:23:49.755433+0000","last_undegraded":"2024-01-16T15:23:49.755433+0000","last_fullsized":"2024-01-16T15:23:49.755433+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:08.430991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:10.343721+0000","last_change":"2024-01-16T15:15:38.291309+0000","last_active":"2024-01-16T15:20:10.343721+0000","last_peered":"2024-01-16T15:20:10.343721+0000","last_clean":"2024-01-16T15:20:10.343721+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:20:10.343721+0000","last_undegraded":"2024-01-16T15:20:10.343721+0000","last_fullsized":"2024-01-16T15:20:10.343721+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:35.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"118'62","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755362+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:23:49.755362+0000","last_peered":"2024-01-16T15:23:49.755362+0000","last_clean":"2024-01-16T15:23:49.755362+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:23:49.755362+0000","last_undegraded":"2024-01-16T15:23:49.755362+0000","last_fullsized":"2024-01-16T15:23:49.755362+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:31.476276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.877559+0000","last_change":"2024-01-16T15:15:38.298667+0000","last_active":"2024-01-16T15:20:09.877559+0000","last_peered":"2024-01-16T15:20:09.877559+0000","last_clean":"2024-01-16T15:20:09.877559+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:20:09.877559+0000","last_undegraded":"2024-01-16T15:20:09.877559+0000","last_fullsized":"2024-01-16T15:20:09.877559+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"118'56","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742341+0000","last_change":"2024-01-16T15:15:39.326224+0000","last_active":"2024-01-16T15:23:49.742341+0000","last_peered":"2024-01-16T15:23:49.742341+0000","last_clean":"2024-01-16T15:23:49.742341+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:23:49.742341+0000","last_undegraded":"2024-01-16T15:23:49.742341+0000","last_fullsized":"2024-01-16T15:23:49.742341+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:02.679035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:10.054807+0000","last_change":"2024-01-16T15:15:38.278400+0000","last_active":"2024-01-16T15:20:10.054807+0000","last_peered":"2024-01-16T15:20:10.054807+0000","last_clean":"2024-01-16T15:20:10.054807+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:20:10.054807+0000","last_undegraded":"2024-01-16T15:20:10.054807+0000","last_fullsized":"2024-01-16T15:20:10.054807+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:37.301739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"118'70","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756610+0000","last_change":"2024-01-16T15:15:39.325408+0000","last_active":"2024-01-16T15:23:49.756610+0000","last_peered":"2024-01-16T15:23:49.756610+0000","last_clean":"2024-01-16T15:23:49.756610+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:23:49.756610+0000","last_undegraded":"2024-01-16T15:23:49.756610+0000","last_fullsized":"2024-01-16T15:23:49.756610+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:31.021632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.883905+0000","last_change":"2024-01-16T15:15:38.287926+0000","last_active":"2024-01-16T15:19:59.883905+0000","last_peered":"2024-01-16T15:19:59.883905+0000","last_clean":"2024-01-16T15:19:59.883905+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:19:59.883905+0000","last_undegraded":"2024-01-16T15:19:59.883905+0000","last_fullsized":"2024-01-16T15:19:59.883905+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:31.944486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"118'65","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755705+0000","last_change":"2024-01-16T15:15:39.330480+0000","last_active":"2024-01-16T15:23:49.755705+0000","last_peered":"2024-01-16T15:23:49.755705+0000","last_clean":"2024-01-16T15:23:49.755705+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:23:49.755705+0000","last_undegraded":"2024-01-16T15:23:49.755705+0000","last_fullsized":"2024-01-16T15:23:49.755705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:25.204362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:00.052571+0000","last_change":"2024-01-16T15:15:38.276702+0000","last_active":"2024-01-16T15:20:00.052571+0000","last_peered":"2024-01-16T15:20:00.052571+0000","last_clean":"2024-01-16T15:20:00.052571+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:20:00.052571+0000","last_undegraded":"2024-01-16T15:20:00.052571+0000","last_fullsized":"2024-01-16T15:20:00.052571+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"118'57","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749622+0000","last_change":"2024-01-16T15:15:39.316268+0000","last_active":"2024-01-16T15:23:49.749622+0000","last_peered":"2024-01-16T15:23:49.749622+0000","last_clean":"2024-01-16T15:23:49.749622+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:23:49.749622+0000","last_undegraded":"2024-01-16T15:23:49.749622+0000","last_fullsized":"2024-01-16T15:23:49.749622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:43.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692036,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.866104+0000","last_change":"2024-01-16T15:15:38.297590+0000","last_active":"2024-01-16T15:19:59.866104+0000","last_peered":"2024-01-16T15:19:59.866104+0000","last_clean":"2024-01-16T15:19:59.866104+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:19:59.866104+0000","last_undegraded":"2024-01-16T15:19:59.866104+0000","last_fullsized":"2024-01-16T15:19:59.866104+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:18.917382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"118'53","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750693+0000","last_change":"2024-01-16T15:15:39.333001+0000","last_active":"2024-01-16T15:23:49.750693+0000","last_peered":"2024-01-16T15:23:49.750693+0000","last_clean":"2024-01-16T15:23:49.750693+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:23:49.750693+0000","last_undegraded":"2024-01-16T15:23:49.750693+0000","last_fullsized":"2024-01-16T15:23:49.750693+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:48.934538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.958589+0000","last_change":"2024-01-16T15:15:38.291784+0000","last_active":"2024-01-16T15:19:59.958589+0000","last_peered":"2024-01-16T15:19:59.958589+0000","last_clean":"2024-01-16T15:19:59.958589+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:19:59.958589+0000","last_undegraded":"2024-01-16T15:19:59.958589+0000","last_fullsized":"2024-01-16T15:19:59.958589+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"118'61","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758768+0000","last_change":"2024-01-16T15:15:39.320032+0000","last_active":"2024-01-16T15:23:49.758768+0000","last_peered":"2024-01-16T15:23:49.758768+0000","last_clean":"2024-01-16T15:23:49.758768+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:23:49.758768+0000","last_undegraded":"2024-01-16T15:23:49.758768+0000","last_fullsized":"2024-01-16T15:23:49.758768+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:48.099094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"118'30","reported_seq":47,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.890480+0000","last_change":"2024-01-16T15:15:38.295797+0000","last_active":"2024-01-16T15:23:47.890480+0000","last_peered":"2024-01-16T15:23:47.890480+0000","last_clean":"2024-01-16T15:23:47.890480+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:23:47.890480+0000","last_undegraded":"2024-01-16T15:23:47.890480+0000","last_fullsized":"2024-01-16T15:23:47.890480+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:49.218031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"118'69","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748821+0000","last_change":"2024-01-16T15:15:55.923114+0000","last_active":"2024-01-16T15:23:49.748821+0000","last_peered":"2024-01-16T15:23:49.748821+0000","last_clean":"2024-01-16T15:23:49.748821+0000","last_became_active":"2024-01-16T15:15:55.914646+0000","last_became_peered":"2024-01-16T15:15:55.914646+0000","last_unstale":"2024-01-16T15:23:49.748821+0000","last_undegraded":"2024-01-16T15:23:49.748821+0000","last_fullsized":"2024-01-16T15:23:49.748821+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.864273+0000","last_change":"2024-01-16T15:15:38.297697+0000","last_active":"2024-01-16T15:19:49.864273+0000","last_peered":"2024-01-16T15:19:49.864273+0000","last_clean":"2024-01-16T15:19:49.864273+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:19:49.864273+0000","last_undegraded":"2024-01-16T15:19:49.864273+0000","last_fullsized":"2024-01-16T15:19:49.864273+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:28.616152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"118'59","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746852+0000","last_change":"2024-01-16T15:15:39.331210+0000","last_active":"2024-01-16T15:23:49.746852+0000","last_peered":"2024-01-16T15:23:49.746852+0000","last_clean":"2024-01-16T15:23:49.746852+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:23:49.746852+0000","last_undegraded":"2024-01-16T15:23:49.746852+0000","last_fullsized":"2024-01-16T15:23:49.746852+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:29.839201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.956236+0000","last_change":"2024-01-16T15:15:38.287424+0000","last_active":"2024-01-16T15:19:49.956236+0000","last_peered":"2024-01-16T15:19:49.956236+0000","last_clean":"2024-01-16T15:19:49.956236+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:19:49.956236+0000","last_undegraded":"2024-01-16T15:19:49.956236+0000","last_fullsized":"2024-01-16T15:19:49.956236+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:00.039099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"118'63","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738719+0000","last_change":"2024-01-16T15:15:39.313583+0000","last_active":"2024-01-16T15:23:49.738719+0000","last_peered":"2024-01-16T15:23:49.738719+0000","last_clean":"2024-01-16T15:23:49.738719+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:23:49.738719+0000","last_undegraded":"2024-01-16T15:23:49.738719+0000","last_fullsized":"2024-01-16T15:23:49.738719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:58.758941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.873843+0000","last_change":"2024-01-16T15:15:38.290534+0000","last_active":"2024-01-16T15:19:49.873843+0000","last_peered":"2024-01-16T15:19:49.873843+0000","last_clean":"2024-01-16T15:19:49.873843+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:19:49.873843+0000","last_undegraded":"2024-01-16T15:19:49.873843+0000","last_fullsized":"2024-01-16T15:19:49.873843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"118'66","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749639+0000","last_change":"2024-01-16T15:15:39.305386+0000","last_active":"2024-01-16T15:23:49.749639+0000","last_peered":"2024-01-16T15:23:49.749639+0000","last_clean":"2024-01-16T15:23:49.749639+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:23:49.749639+0000","last_undegraded":"2024-01-16T15:23:49.749639+0000","last_fullsized":"2024-01-16T15:23:49.749639+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:02.559083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.969401+0000","last_change":"2024-01-16T15:15:38.287042+0000","last_active":"2024-01-16T15:19:49.969401+0000","last_peered":"2024-01-16T15:19:49.969401+0000","last_clean":"2024-01-16T15:19:49.969401+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:19:49.969401+0000","last_undegraded":"2024-01-16T15:19:49.969401+0000","last_fullsized":"2024-01-16T15:19:49.969401+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:50.656301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"118'68","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752622+0000","last_change":"2024-01-16T15:15:39.330274+0000","last_active":"2024-01-16T15:23:49.752622+0000","last_peered":"2024-01-16T15:23:49.752622+0000","last_clean":"2024-01-16T15:23:49.752622+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:23:49.752622+0000","last_undegraded":"2024-01-16T15:23:49.752622+0000","last_fullsized":"2024-01-16T15:23:49.752622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:55:58.229013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"118'31","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.918964+0000","last_change":"2024-01-16T15:15:38.281868+0000","last_active":"2024-01-16T15:21:45.918964+0000","last_peered":"2024-01-16T15:21:45.918964+0000","last_clean":"2024-01-16T15:21:45.918964+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:21:45.918964+0000","last_undegraded":"2024-01-16T15:21:45.918964+0000","last_fullsized":"2024-01-16T15:21:45.918964+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3846,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"118'70","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753269+0000","last_change":"2024-01-16T15:15:39.333842+0000","last_active":"2024-01-16T15:23:49.753269+0000","last_peered":"2024-01-16T15:23:49.753269+0000","last_clean":"2024-01-16T15:23:49.753269+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:23:49.753269+0000","last_undegraded":"2024-01-16T15:23:49.753269+0000","last_fullsized":"2024-01-16T15:23:49.753269+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:38.845770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:39.887790+0000","last_change":"2024-01-16T15:15:38.297094+0000","last_active":"2024-01-16T15:21:39.887790+0000","last_peered":"2024-01-16T15:21:39.887790+0000","last_clean":"2024-01-16T15:21:39.887790+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:21:39.887790+0000","last_undegraded":"2024-01-16T15:21:39.887790+0000","last_fullsized":"2024-01-16T15:21:39.887790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:48.520471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"118'49","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753143+0000","last_change":"2024-01-16T15:15:39.324548+0000","last_active":"2024-01-16T15:23:49.753143+0000","last_peered":"2024-01-16T15:23:49.753143+0000","last_clean":"2024-01-16T15:23:49.753143+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:23:49.753143+0000","last_undegraded":"2024-01-16T15:23:49.753143+0000","last_fullsized":"2024-01-16T15:23:49.753143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:57.259350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:19.879133+0000","last_change":"2024-01-16T15:15:38.299858+0000","last_active":"2024-01-16T15:20:19.879133+0000","last_peered":"2024-01-16T15:20:19.879133+0000","last_clean":"2024-01-16T15:20:19.879133+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:20:19.879133+0000","last_undegraded":"2024-01-16T15:20:19.879133+0000","last_fullsized":"2024-01-16T15:20:19.879133+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":264,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:05.151918+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:21:05.151918+0000","last_peered":"2024-01-16T15:21:05.151918+0000","last_clean":"2024-01-16T15:21:05.151918+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:21:05.151918+0000","last_undegraded":"2024-01-16T15:21:05.151918+0000","last_fullsized":"2024-01-16T15:21:05.151918+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452737,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4113,"ondisk_log_size":4113,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1245803,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":486,"num_write_kb":1454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":2491606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542470,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2096820,"kb_used_data":2025452,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92275020,"statfs":{"total":96636764160,"available":94489620480,"internally_reserved":0,"allocated":2074062848,"data_stored":2073213012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.238,"15min":1.26},"min":{"1min":0.84099999999999997,"5min":0.70399999999999996,"15min":0.57799999999999996},"max":{"1min":1.6100000000000001,"5min":1.6679999999999999,"15min":13.566000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.323,"15min":1.3089999999999999},"min":{"1min":0.96499999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.984,"5min":1.984,"15min":10.728},"last":1.3859999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1200000000000001,"15min":1.159},"min":{"1min":0.74099999999999999,"5min":0.73899999999999999,"15min":0.59699999999999998},"max":{"1min":1.446,"5min":1.714,"15min":13.375},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1519999999999999,"15min":1.1950000000000001},"min":{"1min":0.86299999999999999,"5min":0.67800000000000005,"15min":0.65900000000000003},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":14.592000000000001},"last":0.91800000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.264,"15min":1.2589999999999999},"min":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.72199999999999998},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":4.9000000000000004},"last":1.03},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1910000000000001,"15min":1.2390000000000001},"min":{"1min":0.89800000000000002,"5min":0.77100000000000002,"15min":0.67600000000000005},"max":{"1min":1.4630000000000001,"5min":1.724,"15min":24.806000000000001},"last":1.333}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2849999999999999,"15min":1.3839999999999999},"min":{"1min":0.86899999999999999,"5min":0.81399999999999995,"15min":0.67000000000000004},"max":{"1min":1.6599999999999999,"5min":2.27,"15min":23.741},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4650000000000001,"15min":1.508},"min":{"1min":1.1699999999999999,"5min":1.0029999999999999,"15min":0.76500000000000001},"max":{"1min":1.8260000000000001,"5min":2.3119999999999998,"15min":23.777000000000001},"last":1.1339999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.2450000000000001,"15min":1.3220000000000001},"min":{"1min":0.81000000000000005,"5min":0.57599999999999996,"15min":0.45900000000000002},"max":{"1min":1.865,"5min":1.865,"15min":22.355},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.413,"15min":1.516},"min":{"1min":0.85199999999999998,"5min":0.72899999999999998,"15min":0.63600000000000001},"max":{"1min":1.7689999999999999,"5min":1.9239999999999999,"15min":22.271999999999998},"last":0.76900000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.4139999999999999,"15min":1.454},"min":{"1min":0.95699999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.8300000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.95199999999999996,"5min":0.66000000000000003,"15min":0.56699999999999995},"max":{"1min":1.8680000000000001,"5min":1.8999999999999999,"15min":7.8540000000000001},"last":1.605}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4379999999999999,"15min":1.4370000000000001},"min":{"1min":1.042,"5min":0.53000000000000003,"15min":0.50800000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.6389999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.4410000000000001,"15min":1.5},"min":{"1min":0.98599999999999999,"5min":0.63400000000000001,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":13.725},"last":1.1870000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542469,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477736,"kb_used_data":1406368,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92894104,"statfs":{"total":96636764160,"available":95123562496,"internally_reserved":0,"allocated":1440120832,"data_stored":1439281876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.179,"15min":1.23},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.58099999999999996},"max":{"1min":1.6319999999999999,"5min":1.764,"15min":12.467000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1930000000000001,"15min":1.2410000000000001},"min":{"1min":0.83099999999999996,"5min":0.67100000000000004,"15min":0.41599999999999998},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":8.0630000000000006},"last":0.61699999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.226,"15min":1.27},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.497},"max":{"1min":1.6200000000000001,"5min":1.7030000000000001,"15min":10.518000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.3100000000000001,"15min":1.2889999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.53100000000000003},"max":{"1min":1.73,"5min":1.887,"15min":7.2919999999999998},"last":1.506}]},{"osd":2,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.254,"15min":1.2909999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":1.784,"15min":6.3040000000000003},"last":0.755},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2829999999999999,"15min":1.417},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.57099999999999995},"max":{"1min":1.7470000000000001,"5min":1.8240000000000001,"15min":15.282},"last":1.319}]},{"osd":3,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.337,"15min":1.359},"min":{"1min":0.95299999999999996,"5min":0.71899999999999997,"15min":0.621},"max":{"1min":1.641,"5min":1.9490000000000001,"15min":4.3659999999999997},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4139999999999999,"15min":1.458},"min":{"1min":1.159,"5min":0.94999999999999996,"15min":0.50900000000000001},"max":{"1min":1.724,"5min":1.8480000000000001,"15min":9.0960000000000001},"last":0.89800000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.347,"15min":1.379},"min":{"1min":1.026,"5min":0.85999999999999999,"15min":0.49299999999999999},"max":{"1min":1.8580000000000001,"5min":1.895,"15min":5.4669999999999996},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.41199999999999998},"max":{"1min":1.7989999999999999,"5min":1.944,"15min":5.3360000000000003},"last":1.0069999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3600000000000001,"15min":1.399},"min":{"1min":0.82799999999999996,"5min":0.65700000000000003,"15min":0.436},"max":{"1min":1.708,"5min":1.9239999999999999,"15min":10.388999999999999},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.5089999999999999,"15min":1.5469999999999999},"min":{"1min":1.1100000000000001,"5min":1.016,"15min":0.42799999999999999},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":7.9640000000000004},"last":1.671}]},{"osd":7,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.5429999999999999,"15min":1.577},"min":{"1min":1.05,"5min":0.98399999999999999,"15min":0.46600000000000003},"max":{"1min":1.804,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5029999999999999,"15min":1.6000000000000001},"min":{"1min":0.98199999999999998,"5min":0.84199999999999997,"15min":0.495},"max":{"1min":1.857,"5min":1.984,"15min":16.175999999999998},"last":1.5620000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542469,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174568,"kb_used_data":1107360,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93197272,"statfs":{"total":96636764160,"available":95434006528,"internally_reserved":0,"allocated":1133936640,"data_stored":1133098096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.278,"15min":1.216},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.532,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.3120000000000001,"15min":1.236},"min":{"1min":0.81899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.657,"5min":12.922000000000001,"15min":12.922000000000001},"last":1.135}]},{"osd":1,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3200000000000001,"15min":1.3169999999999999},"min":{"1min":0.89700000000000002,"5min":0.66300000000000003,"15min":0.63900000000000001},"max":{"1min":1.6879999999999999,"5min":5.2249999999999996,"15min":8.1820000000000004},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.175,"15min":1.173},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.536,"5min":2.0659999999999998,"15min":2.597},"last":1.377}]},{"osd":2,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.268,"15min":1.264},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6459999999999999,"5min":1.712,"15min":3.0470000000000002},"last":1.244},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2250000000000001,"15min":1.2669999999999999},"min":{"1min":0.72299999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6359999999999999,"5min":1.756,"15min":8.0619999999999994},"last":1.1890000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.401,"15min":1.484},"min":{"1min":0.83099999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7430000000000001,"5min":5.157,"15min":16.931999999999999},"last":1.456},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.419,"15min":1.494},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":2.6960000000000002,"15min":19.111999999999998},"last":1.5580000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1930000000000001,"15min":1.2589999999999999},"min":{"1min":0.71899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.7549999999999999,"5min":1.7689999999999999,"15min":4.8490000000000002},"last":1.071},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3540000000000001,"15min":1.411},"min":{"1min":0.84899999999999998,"5min":0.63300000000000001,"15min":0.57399999999999995},"max":{"1min":1.7569999999999999,"5min":5.0810000000000004,"15min":8.3550000000000004},"last":1.329}]},{"osd":6,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4119999999999999,"15min":1.5069999999999999},"min":{"1min":0.79300000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.716,"5min":1.9530000000000001,"15min":17.419},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.415,"15min":1.5269999999999999},"min":{"1min":0.83599999999999997,"5min":0.69899999999999995,"15min":0.69799999999999995},"max":{"1min":1.7170000000000001,"5min":2.2959999999999998,"15min":16.262},"last":1.304}]},{"osd":7,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.6040000000000001},"min":{"1min":1.1359999999999999,"5min":0.68400000000000005,"15min":0.67000000000000004},"max":{"1min":1.889,"5min":4.0380000000000003,"15min":12.385999999999999},"last":1.593},{"interface":"front","average":{"1min":1.429,"5min":1.431,"15min":1.512},"min":{"1min":0.73999999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6919999999999999,"5min":1.796,"15min":5.0780000000000003},"last":1.6180000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542470,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936484,"kb_used_data":1865116,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435356,"statfs":{"total":96636764160,"available":94653804544,"internally_reserved":0,"allocated":1909878784,"data_stored":1909043912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.272,"15min":1.323},"min":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.63300000000000001},"max":{"1min":1.756,"5min":1.774,"15min":14.339},"last":1.462},{"interface":"front","average":{"1min":1.056,"5min":1.111,"15min":1.1899999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.47499999999999998},"max":{"1min":1.508,"5min":1.704,"15min":5.7430000000000003},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2729999999999999,"15min":1.304},"min":{"1min":0.94699999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6020000000000001,"5min":1.8149999999999999,"15min":3.7850000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.218,"15min":1.2889999999999999},"min":{"1min":0.88500000000000001,"5min":0.80500000000000005,"15min":0.57899999999999996},"max":{"1min":1.55,"5min":1.696,"15min":14.298},"last":1.2210000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.2150000000000001,"15min":1.238},"min":{"1min":0.81399999999999995,"5min":0.79600000000000004,"15min":0.69199999999999995},"max":{"1min":1.6200000000000001,"5min":1.6990000000000001,"15min":13.731999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.365,"15min":1.4179999999999999},"min":{"1min":0.999,"5min":0.79400000000000004,"15min":0.73199999999999998},"max":{"1min":1.635,"5min":1.7729999999999999,"15min":14.374000000000001},"last":1.387}]},{"osd":3,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.329,"15min":1.3720000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59899999999999998},"max":{"1min":1.6439999999999999,"5min":1.837,"15min":12.347},"last":1.266},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3100000000000001,"15min":1.369},"min":{"1min":0.93000000000000005,"5min":0.92300000000000004,"15min":0.71699999999999997},"max":{"1min":1.6299999999999999,"5min":1.7430000000000001,"15min":13.914},"last":1.165}]},{"osd":5,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.3220000000000001,"15min":1.4099999999999999},"min":{"1min":0.98199999999999998,"5min":0.80500000000000005,"15min":0.47299999999999998},"max":{"1min":1.613,"5min":2.0190000000000001,"15min":14.24},"last":1.071},{"interface":"front","average":{"1min":1.413,"5min":1.3740000000000001,"15min":1.405},"min":{"1min":1.0940000000000001,"5min":0.87,"15min":0.58299999999999996},"max":{"1min":1.7170000000000001,"5min":1.8480000000000001,"15min":4.673},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":1.0289999999999999,"5min":0.83099999999999996,"15min":0.60699999999999998},"max":{"1min":1.7270000000000001,"5min":1.7410000000000001,"15min":18.033000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.526,"15min":1.589},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.52400000000000002},"max":{"1min":1.6870000000000001,"5min":1.9019999999999999,"15min":14.412000000000001},"last":1.5149999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.518,"15min":1.5489999999999999},"min":{"1min":1.052,"5min":1.052,"15min":0.53500000000000003},"max":{"1min":1.6479999999999999,"5min":2.1269999999999998,"15min":4.7640000000000002},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.5249999999999999,"15min":1.6020000000000001},"min":{"1min":1.202,"5min":0.95999999999999996,"15min":0.55200000000000005},"max":{"1min":1.8999999999999999,"5min":2.0179999999999998,"15min":13.852},"last":1.3240000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542469,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1849396,"kb_used_data":1751660,"kb_used_omap":1,"kb_used_meta":97726,"kb_avail":92522444,"statfs":{"total":96636764160,"available":94742982656,"internally_reserved":0,"allocated":1793699840,"data_stored":1792845160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":100071879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94699999999999995,"15min":0.95599999999999996},"min":{"1min":0.71299999999999997,"5min":0.626,"15min":0.34300000000000003},"max":{"1min":1.353,"5min":2.605,"15min":2.9129999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1739999999999999,"15min":1.143},"min":{"1min":0.438,"5min":0.438,"15min":0.29699999999999999},"max":{"1min":1.2849999999999999,"5min":4.0389999999999997,"15min":12.292999999999999},"last":0.92000000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.069,"15min":0.98899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.33100000000000002},"max":{"1min":1.5580000000000001,"5min":12.335000000000001,"15min":12.335000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0960000000000001,"15min":1.133},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.315},"max":{"1min":1.712,"5min":4.6529999999999996,"15min":10.638},"last":0.85399999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.206,"15min":1.151},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":8.3740000000000006,"15min":8.3740000000000006},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.248,"15min":1.228},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.46000000000000002},"max":{"1min":1.6200000000000001,"5min":4.2629999999999999,"15min":4.2629999999999999},"last":1}]},{"osd":4,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.403,"15min":1.345},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.39300000000000002},"max":{"1min":1.76,"5min":15.512,"15min":15.512},"last":1.107},{"interface":"front","average":{"1min":1.274,"5min":1.411,"15min":1.4119999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.52100000000000002},"max":{"1min":1.9399999999999999,"5min":11.743,"15min":14.250999999999999},"last":1.218}]},{"osd":5,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4710000000000001,"15min":1.417},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7470000000000001,"5min":8.4290000000000003,"15min":8.4290000000000003},"last":1.319},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.419,"15min":1.409},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.42099999999999999},"max":{"1min":1.8440000000000001,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":1.1579999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3839999999999999,"15min":1.419},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.41499999999999998},"max":{"1min":1.8879999999999999,"5min":3.956,"15min":12.675000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.5669999999999999,"15min":1.49},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.54900000000000004},"max":{"1min":1.849,"5min":11.667,"15min":11.667},"last":1.0449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.569,"15min":1.5569999999999999},"min":{"1min":1.113,"5min":1.097,"15min":0.53700000000000003},"max":{"1min":1.9279999999999999,"5min":11.042999999999999,"15min":11.042999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.474,"15min":1.5289999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.435},"max":{"1min":1.954,"5min":3.907,"15min":12.836},"last":1.3520000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542469,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1211492,"kb_used_data":1140124,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93160348,"statfs":{"total":96636764160,"available":95396196352,"internally_reserved":0,"allocated":1167486976,"data_stored":1166655701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.311},"max":{"1min":1.516,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0680000000000001,"15min":1.056},"min":{"1min":0.51600000000000001,"5min":0.438,"15min":0.318},"max":{"1min":1.51,"5min":1.6240000000000001,"15min":3.52},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.169,"15min":1.095},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.434},"max":{"1min":1.5820000000000001,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":1.113},{"interface":"front","average":{"1min":1.05,"5min":1.1200000000000001,"15min":1.119},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.45400000000000001},"max":{"1min":1.4690000000000001,"5min":1.8109999999999999,"15min":3.7759999999999998},"last":0.72399999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0609999999999999,"15min":1.0509999999999999},"min":{"1min":0.68600000000000005,"5min":0.67400000000000004,"15min":0.45700000000000002},"max":{"1min":1.4039999999999999,"5min":1.8360000000000001,"15min":6.1260000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.236,"15min":1.228},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.47999999999999998},"max":{"1min":1.4970000000000001,"5min":2.911,"15min":6.2510000000000003},"last":0.85399999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4370000000000001,"15min":1.4259999999999999},"min":{"1min":1.1419999999999999,"5min":1.0980000000000001,"15min":0.68400000000000005},"max":{"1min":1.641,"5min":3.012,"15min":3.012},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.423,"15min":1.4079999999999999},"min":{"1min":0.94599999999999995,"5min":0.83099999999999996,"15min":0.71099999999999997},"max":{"1min":1.7430000000000001,"5min":3.54,"15min":3.54},"last":1.089}]},{"osd":5,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.381,"15min":1.3919999999999999},"min":{"1min":1.0449999999999999,"5min":0.90700000000000003,"15min":0.61099999999999999},"max":{"1min":1.677,"5min":3.6179999999999999,"15min":8.625},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.327,"15min":1.365},"min":{"1min":0.96899999999999997,"5min":0.82099999999999995,"15min":0.80600000000000005},"max":{"1min":1.6850000000000001,"5min":1.8680000000000001,"15min":13.760999999999999},"last":1.2829999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3959999999999999,"15min":1.3839999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.71499999999999997},"max":{"1min":1.871,"5min":2.0539999999999998,"15min":6.4729999999999999},"last":1.163},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.409,"15min":1.4359999999999999},"min":{"1min":1.04,"5min":1.0009999999999999,"15min":0.71099999999999997},"max":{"1min":1.7310000000000001,"5min":2.8260000000000001,"15min":3.2469999999999999},"last":1.228}]},{"osd":7,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.5109999999999999,"15min":1.502},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.871},"max":{"1min":1.724,"5min":3.431,"15min":3.7290000000000001},"last":1.194},{"interface":"front","average":{"1min":1.46,"5min":1.4930000000000001,"15min":1.498},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.89800000000000002},"max":{"1min":1.827,"5min":2.3050000000000002,"15min":6.4249999999999998},"last":1.387}]}]},{"osd":1,"up_from":14,"seq":60129542469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1408104,"kb_used_data":1336736,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92963736,"statfs":{"total":96636764160,"available":95194865664,"internally_reserved":0,"allocated":1368817664,"data_stored":1367982226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.94499999999999995,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.33400000000000002},"max":{"1min":1.3120000000000001,"5min":1.4079999999999999,"15min":8.8499999999999996},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.147,"15min":1.1619999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"max":{"1min":1.5089999999999999,"5min":1.5940000000000001,"15min":8.9380000000000006},"last":1.3160000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.34899999999999998},"max":{"1min":1.2549999999999999,"5min":1.456,"15min":3.4460000000000002},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.123,"15min":1.147},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5840000000000001,"5min":1.677,"15min":2.9660000000000002},"last":1.5840000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.2050000000000001,"15min":1.234},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.45400000000000001},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.282},"last":1.423},{"interface":"front","average":{"1min":1.121,"5min":1.1180000000000001,"15min":1.135},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.47099999999999997},"max":{"1min":1.655,"5min":1.6830000000000001,"15min":8.6910000000000007},"last":1.6299999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.335,"15min":1.3600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.716,"5min":1.8109999999999999,"15min":4.2480000000000002},"last":1.716},{"interface":"front","average":{"1min":1.226,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.57899999999999996},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":4.7640000000000002},"last":1.9199999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.50800000000000001},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3680000000000003},"last":2.2530000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.452,"15min":1.476},"min":{"1min":1.0569999999999999,"5min":0.77500000000000002,"15min":0.66300000000000003},"max":{"1min":1.772,"5min":1.841,"15min":5.4610000000000003},"last":1.772}]},{"osd":6,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":4.7779999999999996},"last":2.0870000000000002},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.365,"15min":1.419},"min":{"1min":0.746,"5min":0.746,"15min":0.56899999999999995},"max":{"1min":2.339,"5min":2.339,"15min":4.8300000000000001},"last":2.339}]},{"osd":7,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4510000000000001,"15min":1.4830000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.58999999999999997},"max":{"1min":2.0510000000000002,"5min":2.081,"15min":6.7990000000000004},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.484,"15min":1.544},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.198,"5min":2.198,"15min":5.1260000000000003},"last":2.198}]}]},{"osd":0,"up_from":14,"seq":60129542470,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2036600,"kb_used_data":1965232,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92335240,"statfs":{"total":96636764160,"available":94551285760,"internally_reserved":0,"allocated":2012397568,"data_stored":2011539049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.34899999999999998},"max":{"1min":1.381,"5min":2.387,"15min":2.7320000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.0269999999999999,"15min":1.0680000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.29399999999999998},"max":{"1min":1.3720000000000001,"5min":1.694,"15min":9.4499999999999993},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2490000000000001,"15min":1.3029999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.45100000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":9.5579999999999998},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.149,"15min":1.1459999999999999},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.40200000000000002},"max":{"1min":1.5960000000000001,"5min":2.4359999999999999,"15min":5.3159999999999998},"last":1.262}]},{"osd":3,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0489999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46600000000000003},"max":{"1min":1.335,"5min":1.5940000000000001,"15min":10.83},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1659999999999999,"15min":1.2529999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":1.5409999999999999,"5min":1.99,"15min":9.6029999999999998},"last":0.996}]},{"osd":4,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3260000000000001,"15min":1.306},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.63400000000000001},"max":{"1min":1.641,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.486},{"interface":"front","average":{"1min":1.2,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.67200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":10.986000000000001},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":1.0089999999999999,"5min":0.78700000000000003,"15min":0.70499999999999996},"max":{"1min":1.665,"5min":3.0259999999999998,"15min":3.0259999999999998},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":1.514,"5min":1.667,"15min":14.561999999999999},"last":1.204}]},{"osd":6,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.47,"15min":1.522},"min":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.76600000000000001},"max":{"1min":1.6639999999999999,"5min":2.2069999999999999,"15min":14.958},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.506,"15min":1.48},"min":{"1min":1.069,"5min":0.89500000000000002,"15min":0.68500000000000005},"max":{"1min":1.7270000000000001,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.6839999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4890000000000001,"15min":1.5640000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.7869999999999999,"5min":2.129,"15min":15.005000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5089999999999999,"15min":1.552},"min":{"1min":1.028,"5min":0.93400000000000005,"15min":0.70699999999999996},"max":{"1min":1.9279999999999999,"5min":3.0590000000000002,"15min":13.718},"last":1.7629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1257472,"data_stored":1241271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1253376,"data_stored":1241229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2009083904,"data_stored":2009072258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166020608,"data_stored":1166016703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908416512,"data_stored":1908408760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132466176,"data_stored":1132462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:57.242 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T15:23:57.243 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:23:57.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.484+0000 7f55da800640 1 Processor -- start 2024-01-16T15:23:57.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.485+0000 7f55da800640 1 -- start start 2024-01-16T15:23:57.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55da800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d4143ae0 0x7f55d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55da800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f55d40cf7d0 0x7f55d415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55da800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f55d415e1e0 0x7f55d415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55da800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d40cfc30 con 0x7f55d415e1e0 2024-01-16T15:23:57.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55da800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d40b1ee0 con 0x7f55d40cf7d0 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55da800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40b2780 con 0x7f55d4143ae0 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55d3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f55d415e1e0 0x7f55d415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55d3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f55d415e1e0 0x7f55d415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48596/0 (socket says 172.21.15.141:48596) 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55d3e00640 1 -- 172.21.15.141:0/257683860 learned_addr learned my addr 172.21.15.141:0/257683860 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55ca800640 1 --2- 172.21.15.141:0/257683860 >> v2:172.21.15.188:3300/0 conn(0x7f55d40cf7d0 0x7f55d415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.487+0000 7f55d3400640 1 --2- 172.21.15.141:0/257683860 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d4143ae0 0x7f55d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.488+0000 7f55d3e00640 1 -- 172.21.15.141:0/257683860 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d4143ae0 msgr2=0x7f55d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:57.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.488+0000 7f55d3e00640 1 --2- 172.21.15.141:0/257683860 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d4143ae0 0x7f55d40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.488+0000 7f55d3e00640 1 -- 172.21.15.141:0/257683860 >> v2:172.21.15.188:3300/0 conn(0x7f55d40cf7d0 msgr2=0x7f55d415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:57.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.488+0000 7f55d3e00640 1 --2- 172.21.15.141:0/257683860 >> v2:172.21.15.188:3300/0 conn(0x7f55d40cf7d0 0x7f55d415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.488+0000 7f55d3e00640 1 -- 172.21.15.141:0/257683860 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d415e990 con 0x7f55d415e1e0 2024-01-16T15:23:57.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.489+0000 7f55d3e00640 1 --2- 172.21.15.141:0/257683860 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d415e1e0 0x7f55d415a670 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f55d406e350 tx=0x7f55cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420537a8c0fe9a86 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:57.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.489+0000 7f55d2a00640 1 -- 172.21.15.141:0/257683860 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc02ca70 con 0x7f55d415e1e0 2024-01-16T15:23:57.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.489+0000 7f55d2a00640 1 -- 172.21.15.141:0/257683860 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc02cbd0 con 0x7f55d415e1e0 2024-01-16T15:23:57.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.489+0000 7f55d2a00640 1 -- 172.21.15.141:0/257683860 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0343f0 con 0x7f55d415e1e0 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 -- 172.21.15.141:0/257683860 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d415e1e0 msgr2=0x7f55d415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 --2- 172.21.15.141:0/257683860 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d415e1e0 0x7f55d415a670 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f55d406e350 tx=0x7f55cc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 -- 172.21.15.141:0/257683860 shutdown_connections 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 --2- 172.21.15.141:0/257683860 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d415e1e0 0x7f55d415a670 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 --2- 172.21.15.141:0/257683860 >> v2:172.21.15.188:3300/0 conn(0x7f55d40cf7d0 0x7f55d415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 --2- 172.21.15.141:0/257683860 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d4143ae0 0x7f55d40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 -- 172.21.15.141:0/257683860 >> 172.21.15.141:0/257683860 conn(0x7f55d406a800 msgr2=0x7f55d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:57.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 -- 172.21.15.141:0/257683860 shutdown_connections 2024-01-16T15:23:57.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.490+0000 7f55da800640 1 -- 172.21.15.141:0/257683860 wait complete. 2024-01-16T15:23:57.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.492+0000 7f55da800640 1 Processor -- start 2024-01-16T15:23:57.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.492+0000 7f55da800640 1 -- start start 2024-01-16T15:23:57.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.493+0000 7f55da800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55d40cf7d0 0x7f55d41594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.493+0000 7f55da800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d4143ae0 0x7f55d40add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.493+0000 7f55da800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d415e1e0 0x7f55d4140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.493+0000 7f55da800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40ae270 con 0x7f55d4143ae0 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.493+0000 7f55da800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40ad8a0 con 0x7f55d40cf7d0 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.493+0000 7f55da800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40ad2b0 con 0x7f55d415e1e0 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.493+0000 7f55ca800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d4143ae0 0x7f55d40add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d4143ae0 0x7f55d40add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48604/0 (socket says 172.21.15.141:48604) 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 -- 172.21.15.141:0/1081701727 learned_addr learned my addr 172.21.15.141:0/1081701727 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55d3400640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55d40cf7d0 0x7f55d41594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55d3e00640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d415e1e0 0x7f55d4140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 -- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d415e1e0 msgr2=0x7f55d4140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d415e1e0 0x7f55d4140060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 -- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55d40cf7d0 msgr2=0x7f55d41594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55d40cf7d0 0x7f55d41594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 -- 172.21.15.141:0/1081701727 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc009b30 con 0x7f55d4143ae0 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55d3e00640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d415e1e0 0x7f55d4140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.494+0000 7f55ca800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d4143ae0 0x7f55d40add30 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00d800 tx=0x7f55bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.495+0000 7f55cbe00640 1 -- 172.21.15.141:0/1081701727 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0096a0 con 0x7f55d4143ae0 2024-01-16T15:23:57.494 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.495+0000 7f55cbe00640 1 -- 172.21.15.141:0/1081701727 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc009800 con 0x7f55d4143ae0 2024-01-16T15:23:57.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.495+0000 7f55cbe00640 1 -- 172.21.15.141:0/1081701727 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc009b90 con 0x7f55d4143ae0 2024-01-16T15:23:57.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.495+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d413d430 con 0x7f55d4143ae0 2024-01-16T15:23:57.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.495+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d413d9f0 con 0x7f55d4143ae0 2024-01-16T15:23:57.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.496+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d40675d0 con 0x7f55d4143ae0 2024-01-16T15:23:57.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.497+0000 7f55cbe00640 1 -- 172.21.15.141:0/1081701727 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f55bc009cf0 con 0x7f55d4143ae0 2024-01-16T15:23:57.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.498+0000 7f55cbe00640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55ac06c5e0 0x7f55ac06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.498+0000 7f55cbe00640 1 -- 172.21.15.141:0/1081701727 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f55bc088ca0 con 0x7f55d4143ae0 2024-01-16T15:23:57.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.498+0000 7f55d3400640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55ac06c5e0 0x7f55ac06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.500+0000 7f55d3400640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55ac06c5e0 0x7f55ac06eaa0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f55c4006fd0 tx=0x7f55c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:57.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.502+0000 7f55cbe00640 1 -- 172.21.15.141:0/1081701727 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55bc0555c0 con 0x7f55d4143ae0 2024-01-16T15:23:57.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.692+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55d415e640 con 0x7f55ac06c5e0 2024-01-16T15:23:57.712 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.715+0000 7f55cbe00640 1 -- 172.21.15.141:0/1081701727 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289287 (secure 0 0 0) 0x7f55c400b200 con 0x7f55ac06c5e0 2024-01-16T15:23:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:57.717 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55ac06c5e0 msgr2=0x7f55ac06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55ac06c5e0 0x7f55ac06eaa0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f55c4006fd0 tx=0x7f55c4008040 comp rx=0 tx=0).stop 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d4143ae0 msgr2=0x7f55d40add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d4143ae0 0x7f55d40add30 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00d800 tx=0x7f55bc00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 shutdown_connections 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f55ac06c5e0 0x7f55ac06eaa0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f55d415e1e0 0x7f55d4140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.719 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f55d4143ae0 0x7f55d40add30 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 --2- 172.21.15.141:0/1081701727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55d40cf7d0 0x7f55d41594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.719+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 >> 172.21.15.141:0/1081701727 conn(0x7f55d406a800 msgr2=0x7f55d4142d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:57.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.720+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 shutdown_connections 2024-01-16T15:23:57.720 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.720+0000 7f55da800640 1 -- 172.21.15.141:0/1081701727 wait complete. 2024-01-16T15:23:57.735 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":920,"stamp":"2024-01-16T15:23:57.121187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444288908,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":945,"num_read_kb":3145889,"num_write":2250,"num_write_kb":6295364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4680,"ondisk_log_size":4680,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13191200,"kb_used_data":12598048,"kb_used_omap":12,"kb_used_meta":593075,"kb_avail":741783520,"statfs":{"total":773094113280,"available":759586324480,"internally_reserved":0,"allocated":12900401152,"data_stored":12893659032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":607309362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073781653,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":276,"num_write_kb":1048622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004674"},"pg_stats":[{"pgid":"17.3f","version":"118'70","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758782+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:23:49.758782+0000","last_peered":"2024-01-16T15:23:49.758782+0000","last_clean":"2024-01-16T15:23:49.758782+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:23:49.758782+0000","last_undegraded":"2024-01-16T15:23:49.758782+0000","last_fullsized":"2024-01-16T15:23:49.758782+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:39.215878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"118'69","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.745868+0000","last_change":"2024-01-16T15:15:39.327644+0000","last_active":"2024-01-16T15:23:49.745868+0000","last_peered":"2024-01-16T15:23:49.745868+0000","last_clean":"2024-01-16T15:23:49.745868+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:23:49.745868+0000","last_undegraded":"2024-01-16T15:23:49.745868+0000","last_fullsized":"2024-01-16T15:23:49.745868+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:39.747172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"118'67","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.754526+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:23:49.754526+0000","last_peered":"2024-01-16T15:23:49.754526+0000","last_clean":"2024-01-16T15:23:49.754526+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:23:49.754526+0000","last_undegraded":"2024-01-16T15:23:49.754526+0000","last_fullsized":"2024-01-16T15:23:49.754526+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:22.535735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"118'66","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738937+0000","last_change":"2024-01-16T15:15:55.875926+0000","last_active":"2024-01-16T15:23:49.738937+0000","last_peered":"2024-01-16T15:23:49.738937+0000","last_clean":"2024-01-16T15:23:49.738937+0000","last_became_active":"2024-01-16T15:15:55.875179+0000","last_became_peered":"2024-01-16T15:15:55.875179+0000","last_unstale":"2024-01-16T15:23:49.738937+0000","last_undegraded":"2024-01-16T15:23:49.738937+0000","last_fullsized":"2024-01-16T15:23:49.738937+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3b","version":"118'60","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737805+0000","last_change":"2024-01-16T15:15:39.321081+0000","last_active":"2024-01-16T15:23:49.737805+0000","last_peered":"2024-01-16T15:23:49.737805+0000","last_clean":"2024-01-16T15:23:49.737805+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:23:49.737805+0000","last_undegraded":"2024-01-16T15:23:49.737805+0000","last_fullsized":"2024-01-16T15:23:49.737805+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663487,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"118'73","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.743047+0000","last_change":"2024-01-16T15:15:39.327813+0000","last_active":"2024-01-16T15:23:49.743047+0000","last_peered":"2024-01-16T15:23:49.743047+0000","last_clean":"2024-01-16T15:23:49.743047+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:23:49.743047+0000","last_undegraded":"2024-01-16T15:23:49.743047+0000","last_fullsized":"2024-01-16T15:23:49.743047+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:39:32.420035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"118'66","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751785+0000","last_change":"2024-01-16T15:15:39.324707+0000","last_active":"2024-01-16T15:23:49.751785+0000","last_peered":"2024-01-16T15:23:49.751785+0000","last_clean":"2024-01-16T15:23:49.751785+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:23:49.751785+0000","last_undegraded":"2024-01-16T15:23:49.751785+0000","last_fullsized":"2024-01-16T15:23:49.751785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303428,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"118'54","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752934+0000","last_change":"2024-01-16T15:15:39.328547+0000","last_active":"2024-01-16T15:23:49.752934+0000","last_peered":"2024-01-16T15:23:49.752934+0000","last_clean":"2024-01-16T15:23:49.752934+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:23:49.752934+0000","last_undegraded":"2024-01-16T15:23:49.752934+0000","last_fullsized":"2024-01-16T15:23:49.752934+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:20.486476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"118'79","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755131+0000","last_change":"2024-01-16T15:15:39.320236+0000","last_active":"2024-01-16T15:23:49.755131+0000","last_peered":"2024-01-16T15:23:49.755131+0000","last_clean":"2024-01-16T15:23:49.755131+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:23:49.755131+0000","last_undegraded":"2024-01-16T15:23:49.755131+0000","last_fullsized":"2024-01-16T15:23:49.755131+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:19:06.417692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052164,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"118'66","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758281+0000","last_change":"2024-01-16T15:15:39.326528+0000","last_active":"2024-01-16T15:23:49.758281+0000","last_peered":"2024-01-16T15:23:49.758281+0000","last_clean":"2024-01-16T15:23:49.758281+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:23:49.758281+0000","last_undegraded":"2024-01-16T15:23:49.758281+0000","last_fullsized":"2024-01-16T15:23:49.758281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:01.161905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"118'70","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737879+0000","last_change":"2024-01-16T15:15:39.325379+0000","last_active":"2024-01-16T15:23:49.737879+0000","last_peered":"2024-01-16T15:23:49.737879+0000","last_clean":"2024-01-16T15:23:49.737879+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:23:49.737879+0000","last_undegraded":"2024-01-16T15:23:49.737879+0000","last_fullsized":"2024-01-16T15:23:49.737879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:11.903533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"118'59","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757983+0000","last_change":"2024-01-16T15:15:39.317978+0000","last_active":"2024-01-16T15:23:49.757983+0000","last_peered":"2024-01-16T15:23:49.757983+0000","last_clean":"2024-01-16T15:23:49.757983+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:23:49.757983+0000","last_undegraded":"2024-01-16T15:23:49.757983+0000","last_fullsized":"2024-01-16T15:23:49.757983+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:47.669894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"118'57","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746095+0000","last_change":"2024-01-16T15:15:39.329570+0000","last_active":"2024-01-16T15:23:49.746095+0000","last_peered":"2024-01-16T15:23:49.746095+0000","last_clean":"2024-01-16T15:23:49.746095+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:23:49.746095+0000","last_undegraded":"2024-01-16T15:23:49.746095+0000","last_fullsized":"2024-01-16T15:23:49.746095+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:22.565771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"118'65","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742239+0000","last_change":"2024-01-16T15:15:39.325613+0000","last_active":"2024-01-16T15:23:49.742239+0000","last_peered":"2024-01-16T15:23:49.742239+0000","last_clean":"2024-01-16T15:23:49.742239+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:23:49.742239+0000","last_undegraded":"2024-01-16T15:23:49.742239+0000","last_fullsized":"2024-01-16T15:23:49.742239+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:08.369872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"118'54","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741582+0000","last_change":"2024-01-16T15:15:39.333121+0000","last_active":"2024-01-16T15:23:49.741582+0000","last_peered":"2024-01-16T15:23:49.741582+0000","last_clean":"2024-01-16T15:23:49.741582+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:23:49.741582+0000","last_undegraded":"2024-01-16T15:23:49.741582+0000","last_fullsized":"2024-01-16T15:23:49.741582+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:43:19.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"118'58","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756735+0000","last_change":"2024-01-16T15:15:39.313940+0000","last_active":"2024-01-16T15:23:49.756735+0000","last_peered":"2024-01-16T15:23:49.756735+0000","last_clean":"2024-01-16T15:23:49.756735+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:23:49.756735+0000","last_undegraded":"2024-01-16T15:23:49.756735+0000","last_fullsized":"2024-01-16T15:23:49.756735+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:57.259350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"118'49","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740185+0000","last_change":"2024-01-16T15:15:39.329528+0000","last_active":"2024-01-16T15:23:49.740185+0000","last_peered":"2024-01-16T15:23:49.740185+0000","last_clean":"2024-01-16T15:23:49.740185+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:23:49.740185+0000","last_undegraded":"2024-01-16T15:23:49.740185+0000","last_fullsized":"2024-01-16T15:23:49.740185+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:42.153458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"118'72","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742141+0000","last_change":"2024-01-16T15:15:39.330676+0000","last_active":"2024-01-16T15:23:49.742141+0000","last_peered":"2024-01-16T15:23:49.742141+0000","last_clean":"2024-01-16T15:23:49.742141+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:23:49.742141+0000","last_undegraded":"2024-01-16T15:23:49.742141+0000","last_fullsized":"2024-01-16T15:23:49.742141+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:39:04.346629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"118'71","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751357+0000","last_change":"2024-01-16T15:15:39.330437+0000","last_active":"2024-01-16T15:23:49.751357+0000","last_peered":"2024-01-16T15:23:49.751357+0000","last_clean":"2024-01-16T15:23:49.751357+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:23:49.751357+0000","last_undegraded":"2024-01-16T15:23:49.751357+0000","last_fullsized":"2024-01-16T15:23:49.751357+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:37.311014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"118'67","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750922+0000","last_change":"2024-01-16T15:15:39.318689+0000","last_active":"2024-01-16T15:23:49.750922+0000","last_peered":"2024-01-16T15:23:49.750922+0000","last_clean":"2024-01-16T15:23:49.750922+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:23:49.750922+0000","last_undegraded":"2024-01-16T15:23:49.750922+0000","last_fullsized":"2024-01-16T15:23:49.750922+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:35.351694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"118'73","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749394+0000","last_change":"2024-01-16T15:15:39.321055+0000","last_active":"2024-01-16T15:23:49.749394+0000","last_peered":"2024-01-16T15:23:49.749394+0000","last_clean":"2024-01-16T15:23:49.749394+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:23:49.749394+0000","last_undegraded":"2024-01-16T15:23:49.749394+0000","last_fullsized":"2024-01-16T15:23:49.749394+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:46.770618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"118'64","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749195+0000","last_change":"2024-01-16T15:15:39.323290+0000","last_active":"2024-01-16T15:23:49.749195+0000","last_peered":"2024-01-16T15:23:49.749195+0000","last_clean":"2024-01-16T15:23:49.749195+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:23:49.749195+0000","last_undegraded":"2024-01-16T15:23:49.749195+0000","last_fullsized":"2024-01-16T15:23:49.749195+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:48:04.184635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"118'57","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737467+0000","last_change":"2024-01-16T15:15:39.317974+0000","last_active":"2024-01-16T15:23:49.737467+0000","last_peered":"2024-01-16T15:23:49.737467+0000","last_clean":"2024-01-16T15:23:49.737467+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:23:49.737467+0000","last_undegraded":"2024-01-16T15:23:49.737467+0000","last_fullsized":"2024-01-16T15:23:49.737467+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:34.399913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"118'70","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753684+0000","last_change":"2024-01-16T15:15:39.324363+0000","last_active":"2024-01-16T15:23:49.753684+0000","last_peered":"2024-01-16T15:23:49.753684+0000","last_clean":"2024-01-16T15:23:49.753684+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:23:49.753684+0000","last_undegraded":"2024-01-16T15:23:49.753684+0000","last_fullsized":"2024-01-16T15:23:49.753684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:12.878416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"118'71","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741193+0000","last_change":"2024-01-16T15:15:39.328896+0000","last_active":"2024-01-16T15:23:49.741193+0000","last_peered":"2024-01-16T15:23:49.741193+0000","last_clean":"2024-01-16T15:23:49.741193+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:23:49.741193+0000","last_undegraded":"2024-01-16T15:23:49.741193+0000","last_fullsized":"2024-01-16T15:23:49.741193+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:41.474484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"118'81","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756330+0000","last_change":"2024-01-16T15:15:39.317589+0000","last_active":"2024-01-16T15:23:49.756330+0000","last_peered":"2024-01-16T15:23:49.756330+0000","last_clean":"2024-01-16T15:23:49.756330+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:23:49.756330+0000","last_undegraded":"2024-01-16T15:23:49.756330+0000","last_fullsized":"2024-01-16T15:23:49.756330+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:36.235755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"118'64","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746236+0000","last_change":"2024-01-16T15:15:39.335455+0000","last_active":"2024-01-16T15:23:49.746236+0000","last_peered":"2024-01-16T15:23:49.746236+0000","last_clean":"2024-01-16T15:23:49.746236+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:23:49.746236+0000","last_undegraded":"2024-01-16T15:23:49.746236+0000","last_fullsized":"2024-01-16T15:23:49.746236+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:38.284829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"118'43","reported_seq":64,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749120+0000","last_change":"2024-01-16T15:15:39.317292+0000","last_active":"2024-01-16T15:23:49.749120+0000","last_peered":"2024-01-16T15:23:49.749120+0000","last_clean":"2024-01-16T15:23:49.749120+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:23:49.749120+0000","last_undegraded":"2024-01-16T15:23:49.749120+0000","last_fullsized":"2024-01-16T15:23:49.749120+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:38.845770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"118'57","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751058+0000","last_change":"2024-01-16T15:15:39.320340+0000","last_active":"2024-01-16T15:23:49.751058+0000","last_peered":"2024-01-16T15:23:49.751058+0000","last_clean":"2024-01-16T15:23:49.751058+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:23:49.751058+0000","last_undegraded":"2024-01-16T15:23:49.751058+0000","last_fullsized":"2024-01-16T15:23:49.751058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:22:18.946850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"118'76","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740849+0000","last_change":"2024-01-16T15:15:39.317544+0000","last_active":"2024-01-16T15:23:49.740849+0000","last_peered":"2024-01-16T15:23:49.740849+0000","last_clean":"2024-01-16T15:23:49.740849+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:23:49.740849+0000","last_undegraded":"2024-01-16T15:23:49.740849+0000","last_fullsized":"2024-01-16T15:23:49.740849+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:38.315912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"118'81","reported_seq":116,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757053+0000","last_change":"2024-01-16T15:15:39.313788+0000","last_active":"2024-01-16T15:23:49.757053+0000","last_peered":"2024-01-16T15:23:49.757053+0000","last_clean":"2024-01-16T15:23:49.757053+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:23:49.757053+0000","last_undegraded":"2024-01-16T15:23:49.757053+0000","last_fullsized":"2024-01-16T15:23:49.757053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:43.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"118'63","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744908+0000","last_change":"2024-01-16T15:15:39.321572+0000","last_active":"2024-01-16T15:23:49.744908+0000","last_peered":"2024-01-16T15:23:49.744908+0000","last_clean":"2024-01-16T15:23:49.744908+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:23:49.744908+0000","last_undegraded":"2024-01-16T15:23:49.744908+0000","last_fullsized":"2024-01-16T15:23:49.744908+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:12.886772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"118'62","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.733710+0000","last_change":"2024-01-16T15:15:39.329833+0000","last_active":"2024-01-16T15:23:49.733710+0000","last_peered":"2024-01-16T15:23:49.733710+0000","last_clean":"2024-01-16T15:23:49.733710+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:23:49.733710+0000","last_undegraded":"2024-01-16T15:23:49.733710+0000","last_fullsized":"2024-01-16T15:23:49.733710+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:51.670474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'4","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.873920+0000","last_change":"2024-01-16T15:15:38.295673+0000","last_active":"2024-01-16T15:19:49.873920+0000","last_peered":"2024-01-16T15:19:49.873920+0000","last_clean":"2024-01-16T15:19:49.873920+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:19:49.873920+0000","last_undegraded":"2024-01-16T15:19:49.873920+0000","last_fullsized":"2024-01-16T15:19:49.873920+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"118'54","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751652+0000","last_change":"2024-01-16T15:15:39.329620+0000","last_active":"2024-01-16T15:23:49.751652+0000","last_peered":"2024-01-16T15:23:49.751652+0000","last_clean":"2024-01-16T15:23:49.751652+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:23:49.751652+0000","last_undegraded":"2024-01-16T15:23:49.751652+0000","last_fullsized":"2024-01-16T15:23:49.751652+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.326448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:40.158702+0000","last_change":"2024-01-16T15:15:38.292769+0000","last_active":"2024-01-16T15:21:40.158702+0000","last_peered":"2024-01-16T15:21:40.158702+0000","last_clean":"2024-01-16T15:21:40.158702+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:21:40.158702+0000","last_undegraded":"2024-01-16T15:21:40.158702+0000","last_fullsized":"2024-01-16T15:21:40.158702+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:45.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"118'71","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744070+0000","last_change":"2024-01-16T15:15:39.332870+0000","last_active":"2024-01-16T15:23:49.744070+0000","last_peered":"2024-01-16T15:23:49.744070+0000","last_clean":"2024-01-16T15:23:49.744070+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:23:49.744070+0000","last_undegraded":"2024-01-16T15:23:49.744070+0000","last_fullsized":"2024-01-16T15:23:49.744070+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.891495+0000","last_change":"2024-01-16T15:15:38.288495+0000","last_active":"2024-01-16T15:20:39.891495+0000","last_peered":"2024-01-16T15:20:39.891495+0000","last_clean":"2024-01-16T15:20:39.891495+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:20:39.891495+0000","last_undegraded":"2024-01-16T15:20:39.891495+0000","last_fullsized":"2024-01-16T15:20:39.891495+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:42:07.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"118'68","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753436+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:23:49.753436+0000","last_peered":"2024-01-16T15:23:49.753436+0000","last_clean":"2024-01-16T15:23:49.753436+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:23:49.753436+0000","last_undegraded":"2024-01-16T15:23:49.753436+0000","last_fullsized":"2024-01-16T15:23:49.753436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:21.846652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.891465+0000","last_change":"2024-01-16T15:15:38.288324+0000","last_active":"2024-01-16T15:20:39.891465+0000","last_peered":"2024-01-16T15:20:39.891465+0000","last_clean":"2024-01-16T15:20:39.891465+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:20:39.891465+0000","last_undegraded":"2024-01-16T15:20:39.891465+0000","last_fullsized":"2024-01-16T15:20:39.891465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:37:31.849193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"118'72","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756110+0000","last_change":"2024-01-16T15:15:39.321037+0000","last_active":"2024-01-16T15:23:49.756110+0000","last_peered":"2024-01-16T15:23:49.756110+0000","last_clean":"2024-01-16T15:23:49.756110+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:23:49.756110+0000","last_undegraded":"2024-01-16T15:23:49.756110+0000","last_fullsized":"2024-01-16T15:23:49.756110+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.687281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.883073+0000","last_change":"2024-01-16T15:15:38.288226+0000","last_active":"2024-01-16T15:20:39.883073+0000","last_peered":"2024-01-16T15:20:39.883073+0000","last_clean":"2024-01-16T15:20:39.883073+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:20:39.883073+0000","last_undegraded":"2024-01-16T15:20:39.883073+0000","last_fullsized":"2024-01-16T15:20:39.883073+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:41.040743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"118'72","reported_seq":108,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752048+0000","last_change":"2024-01-16T15:15:39.330201+0000","last_active":"2024-01-16T15:23:49.752048+0000","last_peered":"2024-01-16T15:23:49.752048+0000","last_clean":"2024-01-16T15:23:49.752048+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:23:49.752048+0000","last_undegraded":"2024-01-16T15:23:49.752048+0000","last_fullsized":"2024-01-16T15:23:49.752048+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:52.162066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:39.966446+0000","last_change":"2024-01-16T15:15:38.292721+0000","last_active":"2024-01-16T15:20:39.966446+0000","last_peered":"2024-01-16T15:20:39.966446+0000","last_clean":"2024-01-16T15:20:39.966446+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:20:39.966446+0000","last_undegraded":"2024-01-16T15:20:39.966446+0000","last_fullsized":"2024-01-16T15:20:39.966446+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:32.177991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"118'73","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753205+0000","last_change":"2024-01-16T15:15:39.322497+0000","last_active":"2024-01-16T15:23:49.753205+0000","last_peered":"2024-01-16T15:23:49.753205+0000","last_clean":"2024-01-16T15:23:49.753205+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:23:49.753205+0000","last_undegraded":"2024-01-16T15:23:49.753205+0000","last_fullsized":"2024-01-16T15:23:49.753205+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:18.495148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.977194+0000","last_change":"2024-01-16T15:15:38.276895+0000","last_active":"2024-01-16T15:20:29.977194+0000","last_peered":"2024-01-16T15:20:29.977194+0000","last_clean":"2024-01-16T15:20:29.977194+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:20:29.977194+0000","last_undegraded":"2024-01-16T15:20:29.977194+0000","last_fullsized":"2024-01-16T15:20:29.977194+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"118'73","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749092+0000","last_change":"2024-01-16T15:15:39.325224+0000","last_active":"2024-01-16T15:23:49.749092+0000","last_peered":"2024-01-16T15:23:49.749092+0000","last_clean":"2024-01-16T15:23:49.749092+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:23:49.749092+0000","last_undegraded":"2024-01-16T15:23:49.749092+0000","last_fullsized":"2024-01-16T15:23:49.749092+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:17.508723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.872263+0000","last_change":"2024-01-16T15:15:38.297269+0000","last_active":"2024-01-16T15:20:29.872263+0000","last_peered":"2024-01-16T15:20:29.872263+0000","last_clean":"2024-01-16T15:20:29.872263+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:20:29.872263+0000","last_undegraded":"2024-01-16T15:20:29.872263+0000","last_fullsized":"2024-01-16T15:20:29.872263+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:02.166771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"118'58","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.735497+0000","last_change":"2024-01-16T15:15:39.320446+0000","last_active":"2024-01-16T15:23:49.735497+0000","last_peered":"2024-01-16T15:23:49.735497+0000","last_clean":"2024-01-16T15:23:49.735497+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:23:49.735497+0000","last_undegraded":"2024-01-16T15:23:49.735497+0000","last_fullsized":"2024-01-16T15:23:49.735497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:15.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"118'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.889799+0000","last_change":"2024-01-16T15:15:38.287453+0000","last_active":"2024-01-16T15:20:29.889799+0000","last_peered":"2024-01-16T15:20:29.889799+0000","last_clean":"2024-01-16T15:20:29.889799+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:20:29.889799+0000","last_undegraded":"2024-01-16T15:20:29.889799+0000","last_fullsized":"2024-01-16T15:20:29.889799+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"118'61","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747524+0000","last_change":"2024-01-16T15:15:39.333810+0000","last_active":"2024-01-16T15:23:49.747524+0000","last_peered":"2024-01-16T15:23:49.747524+0000","last_clean":"2024-01-16T15:23:49.747524+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:23:49.747524+0000","last_undegraded":"2024-01-16T15:23:49.747524+0000","last_fullsized":"2024-01-16T15:23:49.747524+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:22:34.785294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:29.964575+0000","last_change":"2024-01-16T15:15:38.293583+0000","last_active":"2024-01-16T15:20:29.964575+0000","last_peered":"2024-01-16T15:20:29.964575+0000","last_clean":"2024-01-16T15:20:29.964575+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:20:29.964575+0000","last_undegraded":"2024-01-16T15:20:29.964575+0000","last_fullsized":"2024-01-16T15:20:29.964575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:48.691832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"118'62","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738579+0000","last_change":"2024-01-16T15:15:39.319368+0000","last_active":"2024-01-16T15:23:49.738579+0000","last_peered":"2024-01-16T15:23:49.738579+0000","last_clean":"2024-01-16T15:23:49.738579+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:23:49.738579+0000","last_undegraded":"2024-01-16T15:23:49.738579+0000","last_fullsized":"2024-01-16T15:23:49.738579+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:31.021632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"118'31","reported_seq":48,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.966626+0000","last_change":"2024-01-16T15:15:38.295750+0000","last_active":"2024-01-16T15:21:45.966626+0000","last_peered":"2024-01-16T15:21:45.966626+0000","last_clean":"2024-01-16T15:21:45.966626+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:21:45.966626+0000","last_undegraded":"2024-01-16T15:21:45.966626+0000","last_fullsized":"2024-01-16T15:21:45.966626+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:58:17.036375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"118'69","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750139+0000","last_change":"2024-01-16T15:15:39.312858+0000","last_active":"2024-01-16T15:23:49.750139+0000","last_peered":"2024-01-16T15:23:49.750139+0000","last_clean":"2024-01-16T15:23:49.750139+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:23:49.750139+0000","last_undegraded":"2024-01-16T15:23:49.750139+0000","last_fullsized":"2024-01-16T15:23:49.750139+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"118'325","reported_seq":342,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.761469+0000","last_change":"2024-01-16T15:15:38.296732+0000","last_active":"2024-01-16T15:23:49.761469+0000","last_peered":"2024-01-16T15:23:49.761469+0000","last_clean":"2024-01-16T15:23:49.761469+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:23:49.761469+0000","last_undegraded":"2024-01-16T15:23:49.761469+0000","last_fullsized":"2024-01-16T15:23:49.761469+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1241159,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":1369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"118'54","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737844+0000","last_change":"2024-01-16T15:15:39.312723+0000","last_active":"2024-01-16T15:23:49.737844+0000","last_peered":"2024-01-16T15:23:49.737844+0000","last_clean":"2024-01-16T15:23:49.737844+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:23:49.737844+0000","last_undegraded":"2024-01-16T15:23:49.737844+0000","last_fullsized":"2024-01-16T15:23:49.737844+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:35:21.305831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.961056+0000","last_change":"2024-01-16T15:15:38.280236+0000","last_active":"2024-01-16T15:21:45.961056+0000","last_peered":"2024-01-16T15:21:45.961056+0000","last_clean":"2024-01-16T15:21:45.961056+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:21:45.961056+0000","last_undegraded":"2024-01-16T15:21:45.961056+0000","last_fullsized":"2024-01-16T15:21:45.961056+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:14.715313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"118'72","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753729+0000","last_change":"2024-01-16T15:15:39.324127+0000","last_active":"2024-01-16T15:23:49.753729+0000","last_peered":"2024-01-16T15:23:49.753729+0000","last_clean":"2024-01-16T15:23:49.753729+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:23:49.753729+0000","last_undegraded":"2024-01-16T15:23:49.753729+0000","last_fullsized":"2024-01-16T15:23:49.753729+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.687281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:20.345465+0000","last_change":"2024-01-16T15:15:38.295857+0000","last_active":"2024-01-16T15:20:20.345465+0000","last_peered":"2024-01-16T15:20:20.345465+0000","last_clean":"2024-01-16T15:20:20.345465+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:20:20.345465+0000","last_undegraded":"2024-01-16T15:20:20.345465+0000","last_fullsized":"2024-01-16T15:20:20.345465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:31.063560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"118'64","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.743542+0000","last_change":"2024-01-16T15:15:39.319495+0000","last_active":"2024-01-16T15:23:49.743542+0000","last_peered":"2024-01-16T15:23:49.743542+0000","last_clean":"2024-01-16T15:23:49.743542+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:23:49.743542+0000","last_undegraded":"2024-01-16T15:23:49.743542+0000","last_fullsized":"2024-01-16T15:23:49.743542+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:18.495148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":40961,"num_write":27,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'8","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:19.887701+0000","last_change":"2024-01-16T15:15:38.293904+0000","last_active":"2024-01-16T15:20:19.887701+0000","last_peered":"2024-01-16T15:20:19.887701+0000","last_clean":"2024-01-16T15:20:19.887701+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:20:19.887701+0000","last_undegraded":"2024-01-16T15:20:19.887701+0000","last_fullsized":"2024-01-16T15:20:19.887701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"118'46","reported_seq":68,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758065+0000","last_change":"2024-01-16T15:15:39.314200+0000","last_active":"2024-01-16T15:23:49.758065+0000","last_peered":"2024-01-16T15:23:49.758065+0000","last_clean":"2024-01-16T15:23:49.758065+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:23:49.758065+0000","last_undegraded":"2024-01-16T15:23:49.758065+0000","last_fullsized":"2024-01-16T15:23:49.758065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:50.598064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.868373+0000","last_change":"2024-01-16T15:15:38.296521+0000","last_active":"2024-01-16T15:20:09.868373+0000","last_peered":"2024-01-16T15:20:09.868373+0000","last_clean":"2024-01-16T15:20:09.868373+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:20:09.868373+0000","last_undegraded":"2024-01-16T15:20:09.868373+0000","last_fullsized":"2024-01-16T15:20:09.868373+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:24.030528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"118'61","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748133+0000","last_change":"2024-01-16T15:15:39.334617+0000","last_active":"2024-01-16T15:23:49.748133+0000","last_peered":"2024-01-16T15:23:49.748133+0000","last_clean":"2024-01-16T15:23:49.748133+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:23:49.748133+0000","last_undegraded":"2024-01-16T15:23:49.748133+0000","last_fullsized":"2024-01-16T15:23:49.748133+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:40.033000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.886136+0000","last_change":"2024-01-16T15:15:38.289440+0000","last_active":"2024-01-16T15:20:09.886136+0000","last_peered":"2024-01-16T15:20:09.886136+0000","last_clean":"2024-01-16T15:20:09.886136+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:20:09.886136+0000","last_undegraded":"2024-01-16T15:20:09.886136+0000","last_fullsized":"2024-01-16T15:20:09.886136+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:51.792778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"118'61","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755433+0000","last_change":"2024-01-16T15:15:39.332704+0000","last_active":"2024-01-16T15:23:49.755433+0000","last_peered":"2024-01-16T15:23:49.755433+0000","last_clean":"2024-01-16T15:23:49.755433+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:23:49.755433+0000","last_undegraded":"2024-01-16T15:23:49.755433+0000","last_fullsized":"2024-01-16T15:23:49.755433+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:08.430991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:10.343721+0000","last_change":"2024-01-16T15:15:38.291309+0000","last_active":"2024-01-16T15:20:10.343721+0000","last_peered":"2024-01-16T15:20:10.343721+0000","last_clean":"2024-01-16T15:20:10.343721+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:20:10.343721+0000","last_undegraded":"2024-01-16T15:20:10.343721+0000","last_fullsized":"2024-01-16T15:20:10.343721+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:35.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"118'62","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755362+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:23:49.755362+0000","last_peered":"2024-01-16T15:23:49.755362+0000","last_clean":"2024-01-16T15:23:49.755362+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:23:49.755362+0000","last_undegraded":"2024-01-16T15:23:49.755362+0000","last_fullsized":"2024-01-16T15:23:49.755362+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:31.476276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:09.877559+0000","last_change":"2024-01-16T15:15:38.298667+0000","last_active":"2024-01-16T15:20:09.877559+0000","last_peered":"2024-01-16T15:20:09.877559+0000","last_clean":"2024-01-16T15:20:09.877559+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:20:09.877559+0000","last_undegraded":"2024-01-16T15:20:09.877559+0000","last_fullsized":"2024-01-16T15:20:09.877559+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"118'56","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742341+0000","last_change":"2024-01-16T15:15:39.326224+0000","last_active":"2024-01-16T15:23:49.742341+0000","last_peered":"2024-01-16T15:23:49.742341+0000","last_clean":"2024-01-16T15:23:49.742341+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:23:49.742341+0000","last_undegraded":"2024-01-16T15:23:49.742341+0000","last_fullsized":"2024-01-16T15:23:49.742341+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:02.679035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"118'3","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:10.054807+0000","last_change":"2024-01-16T15:15:38.278400+0000","last_active":"2024-01-16T15:20:10.054807+0000","last_peered":"2024-01-16T15:20:10.054807+0000","last_clean":"2024-01-16T15:20:10.054807+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:20:10.054807+0000","last_undegraded":"2024-01-16T15:20:10.054807+0000","last_fullsized":"2024-01-16T15:20:10.054807+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:37.301739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"118'70","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756610+0000","last_change":"2024-01-16T15:15:39.325408+0000","last_active":"2024-01-16T15:23:49.756610+0000","last_peered":"2024-01-16T15:23:49.756610+0000","last_clean":"2024-01-16T15:23:49.756610+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:23:49.756610+0000","last_undegraded":"2024-01-16T15:23:49.756610+0000","last_fullsized":"2024-01-16T15:23:49.756610+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:03:31.021632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.883905+0000","last_change":"2024-01-16T15:15:38.287926+0000","last_active":"2024-01-16T15:19:59.883905+0000","last_peered":"2024-01-16T15:19:59.883905+0000","last_clean":"2024-01-16T15:19:59.883905+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:19:59.883905+0000","last_undegraded":"2024-01-16T15:19:59.883905+0000","last_fullsized":"2024-01-16T15:19:59.883905+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:31.944486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"118'65","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755705+0000","last_change":"2024-01-16T15:15:39.330480+0000","last_active":"2024-01-16T15:23:49.755705+0000","last_peered":"2024-01-16T15:23:49.755705+0000","last_clean":"2024-01-16T15:23:49.755705+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:23:49.755705+0000","last_undegraded":"2024-01-16T15:23:49.755705+0000","last_fullsized":"2024-01-16T15:23:49.755705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:25.204362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:00.052571+0000","last_change":"2024-01-16T15:15:38.276702+0000","last_active":"2024-01-16T15:20:00.052571+0000","last_peered":"2024-01-16T15:20:00.052571+0000","last_clean":"2024-01-16T15:20:00.052571+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:20:00.052571+0000","last_undegraded":"2024-01-16T15:20:00.052571+0000","last_fullsized":"2024-01-16T15:20:00.052571+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:01.547342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"118'57","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749622+0000","last_change":"2024-01-16T15:15:39.316268+0000","last_active":"2024-01-16T15:23:49.749622+0000","last_peered":"2024-01-16T15:23:49.749622+0000","last_clean":"2024-01-16T15:23:49.749622+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:23:49.749622+0000","last_undegraded":"2024-01-16T15:23:49.749622+0000","last_fullsized":"2024-01-16T15:23:49.749622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:10:43.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692036,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.866104+0000","last_change":"2024-01-16T15:15:38.297590+0000","last_active":"2024-01-16T15:19:59.866104+0000","last_peered":"2024-01-16T15:19:59.866104+0000","last_clean":"2024-01-16T15:19:59.866104+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:19:59.866104+0000","last_undegraded":"2024-01-16T15:19:59.866104+0000","last_fullsized":"2024-01-16T15:19:59.866104+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:18.917382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"118'53","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750693+0000","last_change":"2024-01-16T15:15:39.333001+0000","last_active":"2024-01-16T15:23:49.750693+0000","last_peered":"2024-01-16T15:23:49.750693+0000","last_clean":"2024-01-16T15:23:49.750693+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:23:49.750693+0000","last_undegraded":"2024-01-16T15:23:49.750693+0000","last_fullsized":"2024-01-16T15:23:49.750693+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:48.934538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'2","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:59.958589+0000","last_change":"2024-01-16T15:15:38.291784+0000","last_active":"2024-01-16T15:19:59.958589+0000","last_peered":"2024-01-16T15:19:59.958589+0000","last_clean":"2024-01-16T15:19:59.958589+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:19:59.958589+0000","last_undegraded":"2024-01-16T15:19:59.958589+0000","last_fullsized":"2024-01-16T15:19:59.958589+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:02.078355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"118'61","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758768+0000","last_change":"2024-01-16T15:15:39.320032+0000","last_active":"2024-01-16T15:23:49.758768+0000","last_peered":"2024-01-16T15:23:49.758768+0000","last_clean":"2024-01-16T15:23:49.758768+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:23:49.758768+0000","last_undegraded":"2024-01-16T15:23:49.758768+0000","last_fullsized":"2024-01-16T15:23:49.758768+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:48.099094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"118'30","reported_seq":47,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:47.890480+0000","last_change":"2024-01-16T15:15:38.295797+0000","last_active":"2024-01-16T15:23:47.890480+0000","last_peered":"2024-01-16T15:23:47.890480+0000","last_clean":"2024-01-16T15:23:47.890480+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:23:47.890480+0000","last_undegraded":"2024-01-16T15:23:47.890480+0000","last_fullsized":"2024-01-16T15:23:47.890480+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:49.218031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"118'69","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748821+0000","last_change":"2024-01-16T15:15:55.923114+0000","last_active":"2024-01-16T15:23:49.748821+0000","last_peered":"2024-01-16T15:23:49.748821+0000","last_clean":"2024-01-16T15:23:49.748821+0000","last_became_active":"2024-01-16T15:15:55.914646+0000","last_became_peered":"2024-01-16T15:15:55.914646+0000","last_unstale":"2024-01-16T15:23:49.748821+0000","last_undegraded":"2024-01-16T15:23:49.748821+0000","last_fullsized":"2024-01-16T15:23:49.748821+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:51.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.864273+0000","last_change":"2024-01-16T15:15:38.297697+0000","last_active":"2024-01-16T15:19:49.864273+0000","last_peered":"2024-01-16T15:19:49.864273+0000","last_clean":"2024-01-16T15:19:49.864273+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:19:49.864273+0000","last_undegraded":"2024-01-16T15:19:49.864273+0000","last_fullsized":"2024-01-16T15:19:49.864273+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:28.616152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"118'59","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746852+0000","last_change":"2024-01-16T15:15:39.331210+0000","last_active":"2024-01-16T15:23:49.746852+0000","last_peered":"2024-01-16T15:23:49.746852+0000","last_clean":"2024-01-16T15:23:49.746852+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:23:49.746852+0000","last_undegraded":"2024-01-16T15:23:49.746852+0000","last_fullsized":"2024-01-16T15:23:49.746852+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:29.839201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.956236+0000","last_change":"2024-01-16T15:15:38.287424+0000","last_active":"2024-01-16T15:19:49.956236+0000","last_peered":"2024-01-16T15:19:49.956236+0000","last_clean":"2024-01-16T15:19:49.956236+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:19:49.956236+0000","last_undegraded":"2024-01-16T15:19:49.956236+0000","last_fullsized":"2024-01-16T15:19:49.956236+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:49:00.039099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"118'63","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738719+0000","last_change":"2024-01-16T15:15:39.313583+0000","last_active":"2024-01-16T15:23:49.738719+0000","last_peered":"2024-01-16T15:23:49.738719+0000","last_clean":"2024-01-16T15:23:49.738719+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:23:49.738719+0000","last_undegraded":"2024-01-16T15:23:49.738719+0000","last_fullsized":"2024-01-16T15:23:49.738719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:58.758941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.873843+0000","last_change":"2024-01-16T15:15:38.290534+0000","last_active":"2024-01-16T15:19:49.873843+0000","last_peered":"2024-01-16T15:19:49.873843+0000","last_clean":"2024-01-16T15:19:49.873843+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:19:49.873843+0000","last_undegraded":"2024-01-16T15:19:49.873843+0000","last_fullsized":"2024-01-16T15:19:49.873843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:13.792459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"118'66","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749639+0000","last_change":"2024-01-16T15:15:39.305386+0000","last_active":"2024-01-16T15:23:49.749639+0000","last_peered":"2024-01-16T15:23:49.749639+0000","last_clean":"2024-01-16T15:23:49.749639+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:23:49.749639+0000","last_undegraded":"2024-01-16T15:23:49.749639+0000","last_fullsized":"2024-01-16T15:23:49.749639+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:02.559083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:19:49.969401+0000","last_change":"2024-01-16T15:15:38.287042+0000","last_active":"2024-01-16T15:19:49.969401+0000","last_peered":"2024-01-16T15:19:49.969401+0000","last_clean":"2024-01-16T15:19:49.969401+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:19:49.969401+0000","last_undegraded":"2024-01-16T15:19:49.969401+0000","last_fullsized":"2024-01-16T15:19:49.969401+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:50.656301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"118'68","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752622+0000","last_change":"2024-01-16T15:15:39.330274+0000","last_active":"2024-01-16T15:23:49.752622+0000","last_peered":"2024-01-16T15:23:49.752622+0000","last_clean":"2024-01-16T15:23:49.752622+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:23:49.752622+0000","last_undegraded":"2024-01-16T15:23:49.752622+0000","last_fullsized":"2024-01-16T15:23:49.752622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:55:58.229013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"118'31","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:45.918964+0000","last_change":"2024-01-16T15:15:38.281868+0000","last_active":"2024-01-16T15:21:45.918964+0000","last_peered":"2024-01-16T15:21:45.918964+0000","last_clean":"2024-01-16T15:21:45.918964+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:21:45.918964+0000","last_undegraded":"2024-01-16T15:21:45.918964+0000","last_fullsized":"2024-01-16T15:21:45.918964+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:14.428729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3846,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"118'70","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753269+0000","last_change":"2024-01-16T15:15:39.333842+0000","last_active":"2024-01-16T15:23:49.753269+0000","last_peered":"2024-01-16T15:23:49.753269+0000","last_clean":"2024-01-16T15:23:49.753269+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:23:49.753269+0000","last_undegraded":"2024-01-16T15:23:49.753269+0000","last_fullsized":"2024-01-16T15:23:49.753269+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:38.845770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:39.887790+0000","last_change":"2024-01-16T15:15:38.297094+0000","last_active":"2024-01-16T15:21:39.887790+0000","last_peered":"2024-01-16T15:21:39.887790+0000","last_clean":"2024-01-16T15:21:39.887790+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:21:39.887790+0000","last_undegraded":"2024-01-16T15:21:39.887790+0000","last_fullsized":"2024-01-16T15:21:39.887790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:48.520471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"118'49","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753143+0000","last_change":"2024-01-16T15:15:39.324548+0000","last_active":"2024-01-16T15:23:49.753143+0000","last_peered":"2024-01-16T15:23:49.753143+0000","last_clean":"2024-01-16T15:23:49.753143+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:23:49.753143+0000","last_undegraded":"2024-01-16T15:23:49.753143+0000","last_fullsized":"2024-01-16T15:23:49.753143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:57.259350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:19.879133+0000","last_change":"2024-01-16T15:15:38.299858+0000","last_active":"2024-01-16T15:20:19.879133+0000","last_peered":"2024-01-16T15:20:19.879133+0000","last_clean":"2024-01-16T15:20:19.879133+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:20:19.879133+0000","last_undegraded":"2024-01-16T15:20:19.879133+0000","last_fullsized":"2024-01-16T15:20:19.879133+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:19.406917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":264,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:05.151918+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:21:05.151918+0000","last_peered":"2024-01-16T15:21:05.151918+0000","last_clean":"2024-01-16T15:21:05.151918+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:21:05.151918+0000","last_undegraded":"2024-01-16T15:21:05.151918+0000","last_fullsized":"2024-01-16T15:21:05.151918+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452737,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4113,"ondisk_log_size":4113,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1245803,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":486,"num_write_kb":1454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":2491606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542470,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2096820,"kb_used_data":2025452,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92275020,"statfs":{"total":96636764160,"available":94489620480,"internally_reserved":0,"allocated":2074062848,"data_stored":2073213012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.238,"15min":1.26},"min":{"1min":0.84099999999999997,"5min":0.70399999999999996,"15min":0.57799999999999996},"max":{"1min":1.6100000000000001,"5min":1.6679999999999999,"15min":13.566000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.323,"15min":1.3089999999999999},"min":{"1min":0.96499999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.984,"5min":1.984,"15min":10.728},"last":1.3859999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1200000000000001,"15min":1.159},"min":{"1min":0.74099999999999999,"5min":0.73899999999999999,"15min":0.59699999999999998},"max":{"1min":1.446,"5min":1.714,"15min":13.375},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1519999999999999,"15min":1.1950000000000001},"min":{"1min":0.86299999999999999,"5min":0.67800000000000005,"15min":0.65900000000000003},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":14.592000000000001},"last":0.91800000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.264,"15min":1.2589999999999999},"min":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.72199999999999998},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":4.9000000000000004},"last":1.03},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1910000000000001,"15min":1.2390000000000001},"min":{"1min":0.89800000000000002,"5min":0.77100000000000002,"15min":0.67600000000000005},"max":{"1min":1.4630000000000001,"5min":1.724,"15min":24.806000000000001},"last":1.333}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2849999999999999,"15min":1.3839999999999999},"min":{"1min":0.86899999999999999,"5min":0.81399999999999995,"15min":0.67000000000000004},"max":{"1min":1.6599999999999999,"5min":2.27,"15min":23.741},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4650000000000001,"15min":1.508},"min":{"1min":1.1699999999999999,"5min":1.0029999999999999,"15min":0.76500000000000001},"max":{"1min":1.8260000000000001,"5min":2.3119999999999998,"15min":23.777000000000001},"last":1.1339999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.2450000000000001,"15min":1.3220000000000001},"min":{"1min":0.81000000000000005,"5min":0.57599999999999996,"15min":0.45900000000000002},"max":{"1min":1.865,"5min":1.865,"15min":22.355},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.413,"15min":1.516},"min":{"1min":0.85199999999999998,"5min":0.72899999999999998,"15min":0.63600000000000001},"max":{"1min":1.7689999999999999,"5min":1.9239999999999999,"15min":22.271999999999998},"last":0.76900000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.4139999999999999,"15min":1.454},"min":{"1min":0.95699999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.8300000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.95199999999999996,"5min":0.66000000000000003,"15min":0.56699999999999995},"max":{"1min":1.8680000000000001,"5min":1.8999999999999999,"15min":7.8540000000000001},"last":1.605}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4379999999999999,"15min":1.4370000000000001},"min":{"1min":1.042,"5min":0.53000000000000003,"15min":0.50800000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.6389999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.4410000000000001,"15min":1.5},"min":{"1min":0.98599999999999999,"5min":0.63400000000000001,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":13.725},"last":1.1870000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542469,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477736,"kb_used_data":1406368,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92894104,"statfs":{"total":96636764160,"available":95123562496,"internally_reserved":0,"allocated":1440120832,"data_stored":1439281876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.179,"15min":1.23},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.58099999999999996},"max":{"1min":1.6319999999999999,"5min":1.764,"15min":12.467000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1930000000000001,"15min":1.2410000000000001},"min":{"1min":0.83099999999999996,"5min":0.67100000000000004,"15min":0.41599999999999998},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":8.0630000000000006},"last":0.61699999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.226,"15min":1.27},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.497},"max":{"1min":1.6200000000000001,"5min":1.7030000000000001,"15min":10.518000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.3100000000000001,"15min":1.2889999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.53100000000000003},"max":{"1min":1.73,"5min":1.887,"15min":7.2919999999999998},"last":1.506}]},{"osd":2,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.254,"15min":1.2909999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":1.784,"15min":6.3040000000000003},"last":0.755},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2829999999999999,"15min":1.417},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.57099999999999995},"max":{"1min":1.7470000000000001,"5min":1.8240000000000001,"15min":15.282},"last":1.319}]},{"osd":3,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.337,"15min":1.359},"min":{"1min":0.95299999999999996,"5min":0.71899999999999997,"15min":0.621},"max":{"1min":1.641,"5min":1.9490000000000001,"15min":4.3659999999999997},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4139999999999999,"15min":1.458},"min":{"1min":1.159,"5min":0.94999999999999996,"15min":0.50900000000000001},"max":{"1min":1.724,"5min":1.8480000000000001,"15min":9.0960000000000001},"last":0.89800000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.347,"15min":1.379},"min":{"1min":1.026,"5min":0.85999999999999999,"15min":0.49299999999999999},"max":{"1min":1.8580000000000001,"5min":1.895,"15min":5.4669999999999996},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.41199999999999998},"max":{"1min":1.7989999999999999,"5min":1.944,"15min":5.3360000000000003},"last":1.0069999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3600000000000001,"15min":1.399},"min":{"1min":0.82799999999999996,"5min":0.65700000000000003,"15min":0.436},"max":{"1min":1.708,"5min":1.9239999999999999,"15min":10.388999999999999},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.5089999999999999,"15min":1.5469999999999999},"min":{"1min":1.1100000000000001,"5min":1.016,"15min":0.42799999999999999},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":7.9640000000000004},"last":1.671}]},{"osd":7,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.5429999999999999,"15min":1.577},"min":{"1min":1.05,"5min":0.98399999999999999,"15min":0.46600000000000003},"max":{"1min":1.804,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5029999999999999,"15min":1.6000000000000001},"min":{"1min":0.98199999999999998,"5min":0.84199999999999997,"15min":0.495},"max":{"1min":1.857,"5min":1.984,"15min":16.175999999999998},"last":1.5620000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542469,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174568,"kb_used_data":1107360,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93197272,"statfs":{"total":96636764160,"available":95434006528,"internally_reserved":0,"allocated":1133936640,"data_stored":1133098096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.278,"15min":1.216},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.532,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.3120000000000001,"15min":1.236},"min":{"1min":0.81899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.657,"5min":12.922000000000001,"15min":12.922000000000001},"last":1.135}]},{"osd":1,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3200000000000001,"15min":1.3169999999999999},"min":{"1min":0.89700000000000002,"5min":0.66300000000000003,"15min":0.63900000000000001},"max":{"1min":1.6879999999999999,"5min":5.2249999999999996,"15min":8.1820000000000004},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.175,"15min":1.173},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.536,"5min":2.0659999999999998,"15min":2.597},"last":1.377}]},{"osd":2,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.268,"15min":1.264},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6459999999999999,"5min":1.712,"15min":3.0470000000000002},"last":1.244},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2250000000000001,"15min":1.2669999999999999},"min":{"1min":0.72299999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6359999999999999,"5min":1.756,"15min":8.0619999999999994},"last":1.1890000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.401,"15min":1.484},"min":{"1min":0.83099999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7430000000000001,"5min":5.157,"15min":16.931999999999999},"last":1.456},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.419,"15min":1.494},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":2.6960000000000002,"15min":19.111999999999998},"last":1.5580000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1930000000000001,"15min":1.2589999999999999},"min":{"1min":0.71899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.7549999999999999,"5min":1.7689999999999999,"15min":4.8490000000000002},"last":1.071},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3540000000000001,"15min":1.411},"min":{"1min":0.84899999999999998,"5min":0.63300000000000001,"15min":0.57399999999999995},"max":{"1min":1.7569999999999999,"5min":5.0810000000000004,"15min":8.3550000000000004},"last":1.329}]},{"osd":6,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4119999999999999,"15min":1.5069999999999999},"min":{"1min":0.79300000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.716,"5min":1.9530000000000001,"15min":17.419},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.415,"15min":1.5269999999999999},"min":{"1min":0.83599999999999997,"5min":0.69899999999999995,"15min":0.69799999999999995},"max":{"1min":1.7170000000000001,"5min":2.2959999999999998,"15min":16.262},"last":1.304}]},{"osd":7,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.6040000000000001},"min":{"1min":1.1359999999999999,"5min":0.68400000000000005,"15min":0.67000000000000004},"max":{"1min":1.889,"5min":4.0380000000000003,"15min":12.385999999999999},"last":1.593},{"interface":"front","average":{"1min":1.429,"5min":1.431,"15min":1.512},"min":{"1min":0.73999999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6919999999999999,"5min":1.796,"15min":5.0780000000000003},"last":1.6180000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542470,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936484,"kb_used_data":1865116,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435356,"statfs":{"total":96636764160,"available":94653804544,"internally_reserved":0,"allocated":1909878784,"data_stored":1909043912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.272,"15min":1.323},"min":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.63300000000000001},"max":{"1min":1.756,"5min":1.774,"15min":14.339},"last":1.462},{"interface":"front","average":{"1min":1.056,"5min":1.111,"15min":1.1899999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.47499999999999998},"max":{"1min":1.508,"5min":1.704,"15min":5.7430000000000003},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2729999999999999,"15min":1.304},"min":{"1min":0.94699999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6020000000000001,"5min":1.8149999999999999,"15min":3.7850000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.218,"15min":1.2889999999999999},"min":{"1min":0.88500000000000001,"5min":0.80500000000000005,"15min":0.57899999999999996},"max":{"1min":1.55,"5min":1.696,"15min":14.298},"last":1.2210000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.2150000000000001,"15min":1.238},"min":{"1min":0.81399999999999995,"5min":0.79600000000000004,"15min":0.69199999999999995},"max":{"1min":1.6200000000000001,"5min":1.6990000000000001,"15min":13.731999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.365,"15min":1.4179999999999999},"min":{"1min":0.999,"5min":0.79400000000000004,"15min":0.73199999999999998},"max":{"1min":1.635,"5min":1.7729999999999999,"15min":14.374000000000001},"last":1.387}]},{"osd":3,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.329,"15min":1.3720000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59899999999999998},"max":{"1min":1.6439999999999999,"5min":1.837,"15min":12.347},"last":1.266},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3100000000000001,"15min":1.369},"min":{"1min":0.93000000000000005,"5min":0.92300000000000004,"15min":0.71699999999999997},"max":{"1min":1.6299999999999999,"5min":1.7430000000000001,"15min":13.914},"last":1.165}]},{"osd":5,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.3220000000000001,"15min":1.4099999999999999},"min":{"1min":0.98199999999999998,"5min":0.80500000000000005,"15min":0.47299999999999998},"max":{"1min":1.613,"5min":2.0190000000000001,"15min":14.24},"last":1.071},{"interface":"front","average":{"1min":1.413,"5min":1.3740000000000001,"15min":1.405},"min":{"1min":1.0940000000000001,"5min":0.87,"15min":0.58299999999999996},"max":{"1min":1.7170000000000001,"5min":1.8480000000000001,"15min":4.673},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":1.0289999999999999,"5min":0.83099999999999996,"15min":0.60699999999999998},"max":{"1min":1.7270000000000001,"5min":1.7410000000000001,"15min":18.033000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.526,"15min":1.589},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.52400000000000002},"max":{"1min":1.6870000000000001,"5min":1.9019999999999999,"15min":14.412000000000001},"last":1.5149999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.518,"15min":1.5489999999999999},"min":{"1min":1.052,"5min":1.052,"15min":0.53500000000000003},"max":{"1min":1.6479999999999999,"5min":2.1269999999999998,"15min":4.7640000000000002},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.5249999999999999,"15min":1.6020000000000001},"min":{"1min":1.202,"5min":0.95999999999999996,"15min":0.55200000000000005},"max":{"1min":1.8999999999999999,"5min":2.0179999999999998,"15min":13.852},"last":1.3240000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542469,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1849396,"kb_used_data":1751660,"kb_used_omap":1,"kb_used_meta":97726,"kb_avail":92522444,"statfs":{"total":96636764160,"available":94742982656,"internally_reserved":0,"allocated":1793699840,"data_stored":1792845160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":100071879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94699999999999995,"15min":0.95599999999999996},"min":{"1min":0.71299999999999997,"5min":0.626,"15min":0.34300000000000003},"max":{"1min":1.353,"5min":2.605,"15min":2.9129999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1739999999999999,"15min":1.143},"min":{"1min":0.438,"5min":0.438,"15min":0.29699999999999999},"max":{"1min":1.2849999999999999,"5min":4.0389999999999997,"15min":12.292999999999999},"last":0.92000000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.069,"15min":0.98899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.33100000000000002},"max":{"1min":1.5580000000000001,"5min":12.335000000000001,"15min":12.335000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0960000000000001,"15min":1.133},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.315},"max":{"1min":1.712,"5min":4.6529999999999996,"15min":10.638},"last":0.85399999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.206,"15min":1.151},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":8.3740000000000006,"15min":8.3740000000000006},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.248,"15min":1.228},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.46000000000000002},"max":{"1min":1.6200000000000001,"5min":4.2629999999999999,"15min":4.2629999999999999},"last":1}]},{"osd":4,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.403,"15min":1.345},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.39300000000000002},"max":{"1min":1.76,"5min":15.512,"15min":15.512},"last":1.107},{"interface":"front","average":{"1min":1.274,"5min":1.411,"15min":1.4119999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.52100000000000002},"max":{"1min":1.9399999999999999,"5min":11.743,"15min":14.250999999999999},"last":1.218}]},{"osd":5,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4710000000000001,"15min":1.417},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7470000000000001,"5min":8.4290000000000003,"15min":8.4290000000000003},"last":1.319},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.419,"15min":1.409},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.42099999999999999},"max":{"1min":1.8440000000000001,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":1.1579999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3839999999999999,"15min":1.419},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.41499999999999998},"max":{"1min":1.8879999999999999,"5min":3.956,"15min":12.675000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.5669999999999999,"15min":1.49},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.54900000000000004},"max":{"1min":1.849,"5min":11.667,"15min":11.667},"last":1.0449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.569,"15min":1.5569999999999999},"min":{"1min":1.113,"5min":1.097,"15min":0.53700000000000003},"max":{"1min":1.9279999999999999,"5min":11.042999999999999,"15min":11.042999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.474,"15min":1.5289999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.435},"max":{"1min":1.954,"5min":3.907,"15min":12.836},"last":1.3520000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542469,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1211492,"kb_used_data":1140124,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93160348,"statfs":{"total":96636764160,"available":95396196352,"internally_reserved":0,"allocated":1167486976,"data_stored":1166655701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.311},"max":{"1min":1.516,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0680000000000001,"15min":1.056},"min":{"1min":0.51600000000000001,"5min":0.438,"15min":0.318},"max":{"1min":1.51,"5min":1.6240000000000001,"15min":3.52},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.169,"15min":1.095},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.434},"max":{"1min":1.5820000000000001,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":1.113},{"interface":"front","average":{"1min":1.05,"5min":1.1200000000000001,"15min":1.119},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.45400000000000001},"max":{"1min":1.4690000000000001,"5min":1.8109999999999999,"15min":3.7759999999999998},"last":0.72399999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0609999999999999,"15min":1.0509999999999999},"min":{"1min":0.68600000000000005,"5min":0.67400000000000004,"15min":0.45700000000000002},"max":{"1min":1.4039999999999999,"5min":1.8360000000000001,"15min":6.1260000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.236,"15min":1.228},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.47999999999999998},"max":{"1min":1.4970000000000001,"5min":2.911,"15min":6.2510000000000003},"last":0.85399999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4370000000000001,"15min":1.4259999999999999},"min":{"1min":1.1419999999999999,"5min":1.0980000000000001,"15min":0.68400000000000005},"max":{"1min":1.641,"5min":3.012,"15min":3.012},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.423,"15min":1.4079999999999999},"min":{"1min":0.94599999999999995,"5min":0.83099999999999996,"15min":0.71099999999999997},"max":{"1min":1.7430000000000001,"5min":3.54,"15min":3.54},"last":1.089}]},{"osd":5,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.381,"15min":1.3919999999999999},"min":{"1min":1.0449999999999999,"5min":0.90700000000000003,"15min":0.61099999999999999},"max":{"1min":1.677,"5min":3.6179999999999999,"15min":8.625},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.327,"15min":1.365},"min":{"1min":0.96899999999999997,"5min":0.82099999999999995,"15min":0.80600000000000005},"max":{"1min":1.6850000000000001,"5min":1.8680000000000001,"15min":13.760999999999999},"last":1.2829999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3959999999999999,"15min":1.3839999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.71499999999999997},"max":{"1min":1.871,"5min":2.0539999999999998,"15min":6.4729999999999999},"last":1.163},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.409,"15min":1.4359999999999999},"min":{"1min":1.04,"5min":1.0009999999999999,"15min":0.71099999999999997},"max":{"1min":1.7310000000000001,"5min":2.8260000000000001,"15min":3.2469999999999999},"last":1.228}]},{"osd":7,"last update":"Tue Jan 16 15:22:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.5109999999999999,"15min":1.502},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.871},"max":{"1min":1.724,"5min":3.431,"15min":3.7290000000000001},"last":1.194},{"interface":"front","average":{"1min":1.46,"5min":1.4930000000000001,"15min":1.498},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.89800000000000002},"max":{"1min":1.827,"5min":2.3050000000000002,"15min":6.4249999999999998},"last":1.387}]}]},{"osd":1,"up_from":14,"seq":60129542469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1408104,"kb_used_data":1336736,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92963736,"statfs":{"total":96636764160,"available":95194865664,"internally_reserved":0,"allocated":1368817664,"data_stored":1367982226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.94499999999999995,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.33400000000000002},"max":{"1min":1.3120000000000001,"5min":1.4079999999999999,"15min":8.8499999999999996},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.147,"15min":1.1619999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"max":{"1min":1.5089999999999999,"5min":1.5940000000000001,"15min":8.9380000000000006},"last":1.3160000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.34899999999999998},"max":{"1min":1.2549999999999999,"5min":1.456,"15min":3.4460000000000002},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.123,"15min":1.147},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5840000000000001,"5min":1.677,"15min":2.9660000000000002},"last":1.5840000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.2050000000000001,"15min":1.234},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.45400000000000001},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.282},"last":1.423},{"interface":"front","average":{"1min":1.121,"5min":1.1180000000000001,"15min":1.135},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.47099999999999997},"max":{"1min":1.655,"5min":1.6830000000000001,"15min":8.6910000000000007},"last":1.6299999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.335,"15min":1.3600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.716,"5min":1.8109999999999999,"15min":4.2480000000000002},"last":1.716},{"interface":"front","average":{"1min":1.226,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.57899999999999996},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":4.7640000000000002},"last":1.9199999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.50800000000000001},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3680000000000003},"last":2.2530000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.452,"15min":1.476},"min":{"1min":1.0569999999999999,"5min":0.77500000000000002,"15min":0.66300000000000003},"max":{"1min":1.772,"5min":1.841,"15min":5.4610000000000003},"last":1.772}]},{"osd":6,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":4.7779999999999996},"last":2.0870000000000002},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.365,"15min":1.419},"min":{"1min":0.746,"5min":0.746,"15min":0.56899999999999995},"max":{"1min":2.339,"5min":2.339,"15min":4.8300000000000001},"last":2.339}]},{"osd":7,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4510000000000001,"15min":1.4830000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.58999999999999997},"max":{"1min":2.0510000000000002,"5min":2.081,"15min":6.7990000000000004},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.484,"15min":1.544},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.198,"5min":2.198,"15min":5.1260000000000003},"last":2.198}]}]},{"osd":0,"up_from":14,"seq":60129542470,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2036600,"kb_used_data":1965232,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92335240,"statfs":{"total":96636764160,"available":94551285760,"internally_reserved":0,"allocated":2012397568,"data_stored":2011539049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.34899999999999998},"max":{"1min":1.381,"5min":2.387,"15min":2.7320000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.0269999999999999,"15min":1.0680000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.29399999999999998},"max":{"1min":1.3720000000000001,"5min":1.694,"15min":9.4499999999999993},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2490000000000001,"15min":1.3029999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.45100000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":9.5579999999999998},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.149,"15min":1.1459999999999999},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.40200000000000002},"max":{"1min":1.5960000000000001,"5min":2.4359999999999999,"15min":5.3159999999999998},"last":1.262}]},{"osd":3,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0489999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46600000000000003},"max":{"1min":1.335,"5min":1.5940000000000001,"15min":10.83},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1659999999999999,"15min":1.2529999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":1.5409999999999999,"5min":1.99,"15min":9.6029999999999998},"last":0.996}]},{"osd":4,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3260000000000001,"15min":1.306},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.63400000000000001},"max":{"1min":1.641,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.486},{"interface":"front","average":{"1min":1.2,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.67200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":10.986000000000001},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":1.0089999999999999,"5min":0.78700000000000003,"15min":0.70499999999999996},"max":{"1min":1.665,"5min":3.0259999999999998,"15min":3.0259999999999998},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":1.514,"5min":1.667,"15min":14.561999999999999},"last":1.204}]},{"osd":6,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.47,"15min":1.522},"min":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.76600000000000001},"max":{"1min":1.6639999999999999,"5min":2.2069999999999999,"15min":14.958},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.506,"15min":1.48},"min":{"1min":1.069,"5min":0.89500000000000002,"15min":0.68500000000000005},"max":{"1min":1.7270000000000001,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.6839999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4890000000000001,"15min":1.5640000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.7869999999999999,"5min":2.129,"15min":15.005000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5089999999999999,"15min":1.552},"min":{"1min":1.028,"5min":0.93400000000000005,"15min":0.70699999999999996},"max":{"1min":1.9279999999999999,"5min":3.0590000000000002,"15min":13.718},"last":1.7629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1257472,"data_stored":1241271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1253376,"data_stored":1241229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2009083904,"data_stored":2009072258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166020608,"data_stored":1166016703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908416512,"data_stored":1908408760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132466176,"data_stored":1132462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:23:57.739 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:23:57.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.978+0000 7f3c14400640 1 Processor -- start 2024-01-16T15:23:57.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.979+0000 7f3c14400640 1 -- start start 2024-01-16T15:23:57.977 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.980+0000 7f3c14400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f3c0c13df80 0x7f3c0c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.980+0000 7f3c14400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 0x7f3c0c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.980+0000 7f3c14400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c06c470 con 0x7f3c0c13e360 2024-01-16T15:23:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.980+0000 7f3c14400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c06c750 con 0x7f3c0c13df80 2024-01-16T15:23:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.980+0000 7f3c14400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c06ca30 con 0x7f3c0c0cf9a0 2024-01-16T15:23:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c11000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f3c0c13e360 0x7f3c0c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54328/0 (socket says 172.21.15.141:54328) 2024-01-16T15:23:57.978 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c11000640 1 -- 172.21.15.141:0/1839846903 learned_addr learned my addr 172.21.15.141:0/1839846903 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:23:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c11a00640 1 --2- 172.21.15.141:0/1839846903 >> v2:172.21.15.188:3300/0 conn(0x7f3c0c13df80 0x7f3c0c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802336048 0 0) 0x7f3c0c06c470 con 0x7f3c0c13e360 2024-01-16T15:23:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c12400640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 0x7f3c0c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bec003520 con 0x7f3c0c13e360 2024-01-16T15:23:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864495077 0 0) 0x7f3bec003520 con 0x7f3c0c13e360 2024-01-16T15:23:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 msgr2=0x7f3c0c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:57.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 0x7f3c0c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 >> v2:172.21.15.188:3300/0 conn(0x7f3c0c13df80 msgr2=0x7f3c0c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 --2- 172.21.15.141:0/1839846903 >> v2:172.21.15.188:3300/0 conn(0x7f3c0c13df80 0x7f3c0c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c06cd10 con 0x7f3c0c13e360 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.981+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f3bf4001670 con 0x7f3c0c13e360 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 >> v1:172.21.15.141:6789/0 conn(0x7f3c0c13e360 legacy=0x7f3c0c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c0be00640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 0x7f3bec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c0be00640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bec003f40 0x7f3bec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c0be00640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.982 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c06c750 con 0x7f3bec004d00 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c06ca30 con 0x7f3bec003f40 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec00fe40 con 0x7f3c0c0cf9a0 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c12400640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c11a00640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 0x7f3bec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c11000640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bec003f40 0x7f3bec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c12400640 1 -- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 msgr2=0x7f3bec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c12400640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 0x7f3bec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.983 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c12400640 1 -- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bec003f40 msgr2=0x7f3bec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:57.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c12400640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bec003f40 0x7f3bec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c12400640 1 -- 172.21.15.141:0/1839846903 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec003520 con 0x7f3bec004d00 2024-01-16T15:23:57.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.982+0000 7f3c11a00640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c0cf9a0 0x7f3bec003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:57.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.983+0000 7f3c11000640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bec003f40 0x7f3bec0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:57.984 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.984+0000 7f3c12400640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f3c04007330 tx=0x7f3bec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0acc65a1b74ba24 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:57.985 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.984+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0402cca0 con 0x7f3bec004d00 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.984+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0402ce00 con 0x7f3bec004d00 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.984+0000 7f3c0be00640 1 -- 172.21.15.141:0/1839846903 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04034660 con 0x7f3bec004d00 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.985+0000 7f3c14400640 1 -- 172.21.15.141:0/1839846903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bec004d00 msgr2=0x7f3bec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.985+0000 7f3c14400640 1 --2- 172.21.15.141:0/1839846903 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f3c04007330 tx=0x7f3bec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.985+0000 7f3c12400640 1 -- 172.21.15.141:0/1839846903 reap_dead start 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.985+0000 7f3c14400640 1 -- 172.21.15.141:0/1839846903 shutdown_connections 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.985+0000 7f3c14400640 1 -- 172.21.15.141:0/1839846903 >> 172.21.15.141:0/1839846903 conn(0x7f3c0c06a800 msgr2=0x7f3c0c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:57.986 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.985+0000 7f3c14400640 1 -- 172.21.15.141:0/1839846903 shutdown_connections 2024-01-16T15:23:57.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.985+0000 7f3c14400640 1 -- 172.21.15.141:0/1839846903 wait complete. 2024-01-16T15:23:57.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.988+0000 7f3c14400640 1 Processor -- start 2024-01-16T15:23:57.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.989+0000 7f3c14400640 1 -- start start 2024-01-16T15:23:57.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.990+0000 7f3c14400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c1570b0 0x7f3c0c0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.990+0000 7f3c14400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c0c155d70 0x7f3c0c071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.990+0000 7f3c14400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c0c071850 0x7f3c0c0b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.990+0000 7f3c14400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c071cd0 con 0x7f3c0c155d70 2024-01-16T15:23:57.989 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.990+0000 7f3c14400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0b39f0 con 0x7f3c0c071850 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.990+0000 7f3c14400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0b3cd0 con 0x7f3c0c1570b0 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.990+0000 7f3c11000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c0c155d70 0x7f3c0c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c0c155d70 0x7f3c0c071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48618/0 (socket says 172.21.15.141:48618) 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 -- 172.21.15.141:0/248535961 learned_addr learned my addr 172.21.15.141:0/248535961 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11a00640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c1570b0 0x7f3c0c0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c12400640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c0c071850 0x7f3c0c0b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 -- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c1570b0 msgr2=0x7f3c0c0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.990 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c1570b0 0x7f3c0c0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 -- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c0c071850 msgr2=0x7f3c0c0b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c0c071850 0x7f3c0c0b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 -- 172.21.15.141:0/248535961 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0b3fb0 con 0x7f3c0c155d70 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.991+0000 7f3c11000640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c0c155d70 0x7f3c0c071310 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400d830 tx=0x7f3bf400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.992+0000 7f3c09600640 1 -- 172.21.15.141:0/248535961 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40074e0 con 0x7f3c0c155d70 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.992+0000 7f3c09600640 1 -- 172.21.15.141:0/248535961 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4007640 con 0x7f3c0c155d70 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.992+0000 7f3c09600640 1 -- 172.21.15.141:0/248535961 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40049e0 con 0x7f3c0c155d70 2024-01-16T15:23:57.991 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.992+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c0b4290 con 0x7f3c0c155d70 2024-01-16T15:23:57.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.992+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c0b36c0 con 0x7f3c0c155d70 2024-01-16T15:23:57.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.993+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c0675d0 con 0x7f3c0c155d70 2024-01-16T15:23:57.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.994+0000 7f3c09600640 1 -- 172.21.15.141:0/248535961 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3bf4004b40 con 0x7f3c0c155d70 2024-01-16T15:23:57.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.995+0000 7f3c09600640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3be406c5e0 0x7f3be406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:57.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.996+0000 7f3c09600640 1 -- 172.21.15.141:0/248535961 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3bf4088300 con 0x7f3c0c155d70 2024-01-16T15:23:57.993 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.996+0000 7f3c11a00640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3be406c5e0 0x7f3be406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:57.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.998+0000 7f3c11a00640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3be406c5e0 0x7f3be406eaa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009b20 tx=0x7f3bfc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:57.996 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:57.999+0000 7f3c09600640 1 -- 172.21.15.141:0/248535961 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bf4054c80 con 0x7f3c0c155d70 2024-01-16T15:23:58.186 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.189+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c0c0b2f60 con 0x7f3c0c155d70 2024-01-16T15:23:58.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.190+0000 7f3c09600640 1 -- 172.21.15.141:0/248535961 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16668 (secure 0 0 0) 0x7f3bf4019070 con 0x7f3c0c155d70 2024-01-16T15:23:58.188 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:23:58.188 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":118,"fsid":"540b621b-477a-463b-8a92-7e448bae48f5","created":"2024-01-16T14:56:47.359709+0000","modified":"2024-01-16T15:15:55.853050+0000","last_up_change":"2024-01-16T14:56:59.168868+0000","last_in_change":"2024-01-16T14:56:51.117513+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:57:00.501425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:36.311732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:37.759492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9dc7c32-2f84-4fe7-b960-b102964c1b38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6801","nonce":4153030184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6803","nonce":4153030184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6807","nonce":4153030184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":4153030184},{"type":"v1","addr":"172.21.15.141:6805","nonce":4153030184}]},"public_addr":"172.21.15.141:6801/4153030184","cluster_addr":"172.21.15.141:6803/4153030184","heartbeat_back_addr":"172.21.15.141:6807/4153030184","heartbeat_front_addr":"172.21.15.141:6805/4153030184","state":["exists","up"]},{"osd":1,"uuid":"ea3eb329-b238-48a3-a840-536e0547c04f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6809","nonce":3479163278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6811","nonce":3479163278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6816","nonce":3479163278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":3479163278},{"type":"v1","addr":"172.21.15.141:6813","nonce":3479163278}]},"public_addr":"172.21.15.141:6809/3479163278","cluster_addr":"172.21.15.141:6811/3479163278","heartbeat_back_addr":"172.21.15.141:6816/3479163278","heartbeat_front_addr":"172.21.15.141:6813/3479163278","state":["exists","up"]},{"osd":2,"uuid":"37ae5e71-3489-4f67-b32f-cd0a206f83e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6815","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6817","nonce":1316482903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6819","nonce":1316482903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6825","nonce":1316482903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":1316482903},{"type":"v1","addr":"172.21.15.141:6821","nonce":1316482903}]},"public_addr":"172.21.15.141:6817/1316482903","cluster_addr":"172.21.15.141:6819/1316482903","heartbeat_back_addr":"172.21.15.141:6825/1316482903","heartbeat_front_addr":"172.21.15.141:6821/1316482903","state":["exists","up"]},{"osd":3,"uuid":"e34bedbd-6cbd-4918-a1ce-ce2aa90b884b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6823","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6824","nonce":1855057890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6827","nonce":1855057890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6831","nonce":1855057890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1855057890},{"type":"v1","addr":"172.21.15.141:6829","nonce":1855057890}]},"public_addr":"172.21.15.141:6824/1855057890","cluster_addr":"172.21.15.141:6827/1855057890","heartbeat_back_addr":"172.21.15.141:6831/1855057890","heartbeat_front_addr":"172.21.15.141:6829/1855057890","state":["exists","up"]},{"osd":4,"uuid":"b9728641-c62f-4941-bd36-4038df6d3246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6801","nonce":2387947334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6803","nonce":2387947334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6807","nonce":2387947334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2387947334},{"type":"v1","addr":"172.21.15.188:6805","nonce":2387947334}]},"public_addr":"172.21.15.188:6801/2387947334","cluster_addr":"172.21.15.188:6803/2387947334","heartbeat_back_addr":"172.21.15.188:6807/2387947334","heartbeat_front_addr":"172.21.15.188:6805/2387947334","state":["exists","up"]},{"osd":5,"uuid":"51fe9078-c8a7-4bbb-bc65-7db69db669b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6814","nonce":917917352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6819","nonce":917917352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6823","nonce":917917352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":917917352},{"type":"v1","addr":"172.21.15.188:6821","nonce":917917352}]},"public_addr":"172.21.15.188:6814/917917352","cluster_addr":"172.21.15.188:6819/917917352","heartbeat_back_addr":"172.21.15.188:6823/917917352","heartbeat_front_addr":"172.21.15.188:6821/917917352","state":["exists","up"]},{"osd":6,"uuid":"4634a709-f2f1-4554-ad94-1867ba2e2010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6825","nonce":2194964726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6827","nonce":2194964726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6831","nonce":2194964726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2194964726},{"type":"v1","addr":"172.21.15.188:6829","nonce":2194964726}]},"public_addr":"172.21.15.188:6825/2194964726","cluster_addr":"172.21.15.188:6827/2194964726","heartbeat_back_addr":"172.21.15.188:6831/2194964726","heartbeat_front_addr":"172.21.15.188:6829/2194964726","state":["exists","up"]},{"osd":7,"uuid":"26a9f08d-325f-432f-8e0b-bd2e0f18ac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6809","nonce":2741327973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6811","nonce":2741327973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6817","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6818","nonce":2741327973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6813","nonce":2741327973},{"type":"v1","addr":"172.21.15.188:6815","nonce":2741327973}]},"public_addr":"172.21.15.188:6809/2741327973","cluster_addr":"172.21.15.188:6811/2741327973","heartbeat_back_addr":"172.21.15.188:6818/2741327973","heartbeat_front_addr":"172.21.15.188:6815/2741327973","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.568611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.367633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:58.154119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.474674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.395057+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.433314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.498557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:56:56.423866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.4","mappings":[{"from":0,"to":5}]},{"pgid":"17.3c","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:23:58.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3be406c5e0 msgr2=0x7f3be406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:58.191 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3be406c5e0 0x7f3be406eaa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009b20 tx=0x7f3bfc008040 comp rx=0 tx=0).stop 2024-01-16T15:23:58.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c0c155d70 msgr2=0x7f3c0c071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:58.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c0c155d70 0x7f3c0c071310 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400d830 tx=0x7f3bf400dcf0 comp rx=0 tx=0).stop 2024-01-16T15:23:58.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 shutdown_connections 2024-01-16T15:23:58.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f3be406c5e0 0x7f3be406eaa0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:58.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c0c071850 0x7f3c0c0b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:58.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c0c155d70 0x7f3c0c071310 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:58.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 --2- 172.21.15.141:0/248535961 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f3c0c1570b0 0x7f3c0c0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:58.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 >> 172.21.15.141:0/248535961 conn(0x7f3c0c06a800 msgr2=0x7f3c0c0ce690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:58.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 shutdown_connections 2024-01-16T15:23:58.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:58.194+0000 7f3c14400640 1 -- 172.21.15.141:0/248535961 wait complete. 2024-01-16T15:23:59.215 INFO:tasks.ceph:Scrubbing osd.0 2024-01-16T15:23:59.216 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:23:59.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.445+0000 7f0c91000640 1 Processor -- start 2024-01-16T15:23:59.442 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.445+0000 7f0c91000640 1 -- start start 2024-01-16T15:23:59.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c91000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c91000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0c8c13e360 0x7f0c8c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.444 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c91000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c8c15f9c0 0x7f0c8c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c91000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8c06c470 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c91000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8c06c750 con 0x7f0c8c13e360 2024-01-16T15:23:59.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c91000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c06ca30 con 0x7f0c8c13df80 2024-01-16T15:23:59.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c8a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c8c15f9c0 0x7f0c8c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c8a800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c8c15f9c0 0x7f0c8c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48626/0 (socket says 172.21.15.141:48626) 2024-01-16T15:23:59.445 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c8a800640 1 -- 172.21.15.141:0/237605049 learned_addr learned my addr 172.21.15.141:0/237605049 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:59.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c81200640 1 --2- 172.21.15.141:0/237605049 >> v2:172.21.15.188:3300/0 conn(0x7f0c8c13e360 0x7f0c8c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c89e00640 1 --2- 172.21.15.141:0/237605049 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.447+0000 7f0c8a800640 1 -- 172.21.15.141:0/237605049 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 msgr2=0x7f0c8c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:59.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.448+0000 7f0c8a800640 1 --2- 172.21.15.141:0/237605049 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.447 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.448+0000 7f0c8a800640 1 -- 172.21.15.141:0/237605049 >> v2:172.21.15.188:3300/0 conn(0x7f0c8c13e360 msgr2=0x7f0c8c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.448+0000 7f0c8a800640 1 --2- 172.21.15.141:0/237605049 >> v2:172.21.15.188:3300/0 conn(0x7f0c8c13e360 0x7f0c8c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.448+0000 7f0c8a800640 1 -- 172.21.15.141:0/237605049 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c06cd10 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.448+0000 7f0c81200640 1 --2- 172.21.15.141:0/237605049 >> v2:172.21.15.188:3300/0 conn(0x7f0c8c13e360 0x7f0c8c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.448+0000 7f0c89e00640 1 --2- 172.21.15.141:0/237605049 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.449+0000 7f0c8a800640 1 --2- 172.21.15.141:0/237605049 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0cf7d0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0cfed0 tx=0x7f0c8402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e57e7dba11c3fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.449+0000 7f0c89400640 1 -- 172.21.15.141:0/237605049 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8402ca80 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.449+0000 7f0c89400640 1 -- 172.21.15.141:0/237605049 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8402cbe0 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.448 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.449+0000 7f0c89400640 1 -- 172.21.15.141:0/237605049 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c840343f0 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 -- 172.21.15.141:0/237605049 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 msgr2=0x7f0c8c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 --2- 172.21.15.141:0/237605049 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0cf7d0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0cfed0 tx=0x7f0c8402bee0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 -- 172.21.15.141:0/237605049 shutdown_connections 2024-01-16T15:23:59.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 --2- 172.21.15.141:0/237605049 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0cf7d0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.449 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 --2- 172.21.15.141:0/237605049 >> v2:172.21.15.188:3300/0 conn(0x7f0c8c13e360 0x7f0c8c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 --2- 172.21.15.141:0/237605049 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 -- 172.21.15.141:0/237605049 >> 172.21.15.141:0/237605049 conn(0x7f0c8c06a800 msgr2=0x7f0c8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:59.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 -- 172.21.15.141:0/237605049 shutdown_connections 2024-01-16T15:23:59.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.450+0000 7f0c91000640 1 -- 172.21.15.141:0/237605049 wait complete. 2024-01-16T15:23:59.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.452+0000 7f0c91000640 1 Processor -- start 2024-01-16T15:23:59.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.452+0000 7f0c91000640 1 -- start start 2024-01-16T15:23:59.452 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c91000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c91000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8c13e360 0x7f0c8c14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c91000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c91000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0b2730 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c91000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0b2a10 con 0x7f0c8c13e360 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c91000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0b2cf0 con 0x7f0c8c13df80 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c8a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c8a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48638/0 (socket says 172.21.15.141:48638) 2024-01-16T15:23:59.453 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c8a800640 1 -- 172.21.15.141:0/3874841669 learned_addr learned my addr 172.21.15.141:0/3874841669 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c81200640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8c13e360 0x7f0c8c14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c8a800640 1 -- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 msgr2=0x7f0c8c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.453+0000 7f0c89e00640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c8a800640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0c8c13df80 0x7f0c8c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c8a800640 1 -- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8c13e360 msgr2=0x7f0c8c14fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c8a800640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8c13e360 0x7f0c8c14fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c8a800640 1 -- 172.21.15.141:0/3874841669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84009b30 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c81200640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8c13e360 0x7f0c8c14fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c8a800640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0ce230 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f0c84036040 tx=0x7f0c84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:59.454 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84009510 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8403e450 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8c0b2fd0 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8403e800 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.454+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8c0b3d10 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.455 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.455+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c8c0b2b80 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.456 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.456+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f0c8402cd50 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.457+0000 7f0c82a00640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c6406c5e0 0x7f0c6406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.457+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0c840ab780 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.457+0000 7f0c82a00640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f0c64074840 0x7f0c64076c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.457+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 --> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c64077350 con 0x7f0c64074840 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.458+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c840ac340 con 0x7f0c8c15f9c0 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.458+0000 7f0c81200640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f0c64074840 0x7f0c64076c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.458+0000 7f0c89e00640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c6406c5e0 0x7f0c6406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.458+0000 7f0c81200640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f0c64074840 0x7f0c64076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:59.457 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.459+0000 7f0c89e00640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c6406c5e0 0x7f0c6406eaa0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00b840 tx=0x7f0c7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:59.458 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.459+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== osd.0 v2:172.21.15.141:6800/4153030184 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0c64077350 con 0x7f0c64074840 2024-01-16T15:23:59.470 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.473+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 --> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0c8c1408d0 con 0x7f0c64074840 2024-01-16T15:23:59.471 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c82a00640 1 -- 172.21.15.141:0/3874841669 <== osd.0 v2:172.21.15.141:6800/4153030184 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0c7400c040 con 0x7f0c64074840 2024-01-16T15:23:59.471 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:23:59.471 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:23:59.471 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:23:59.472 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f0c64074840 msgr2=0x7f0c64076c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:6800/4153030184,v1:172.21.15.141:6801/4153030184] conn(0x7f0c64074840 0x7f0c64076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c6406c5e0 msgr2=0x7f0c6406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0c6406c5e0 0x7f0c6406eaa0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00b840 tx=0x7f0c7c008040 comp rx=0 tx=0).stop 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 msgr2=0x7f0c8c0ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 --2- 172.21.15.141:0/3874841669 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c8c15f9c0 0x7f0c8c0ce230 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f0c84036040 tx=0x7f0c84004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c8a800640 1 -- 172.21.15.141:0/3874841669 reap_dead start 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 shutdown_connections 2024-01-16T15:23:59.473 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 >> 172.21.15.141:0/3874841669 conn(0x7f0c8c06a800 msgr2=0x7f0c8c13c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:59.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 shutdown_connections 2024-01-16T15:23:59.474 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.474+0000 7f0c91000640 1 -- 172.21.15.141:0/3874841669 wait complete. 2024-01-16T15:23:59.487 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-16T15:23:59.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.687+0000 7f1533400640 1 Processor -- start 2024-01-16T15:23:59.685 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.687+0000 7f1533400640 1 -- start start 2024-01-16T15:23:59.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1533400640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f152c13df80 0x7f152c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1533400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f152c0cf9a0 0x7f152c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1533400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c06c470 con 0x7f152c13e360 2024-01-16T15:23:59.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1533400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152c06c750 con 0x7f152c0cf9a0 2024-01-16T15:23:59.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1533400640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152c06ca30 con 0x7f152c13df80 2024-01-16T15:23:59.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1531400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f152c0cf9a0 0x7f152c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1531400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f152c0cf9a0 0x7f152c0cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48640/0 (socket says 172.21.15.141:48640) 2024-01-16T15:23:59.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.689+0000 7f1531400640 1 -- 172.21.15.141:0/907807479 learned_addr learned my addr 172.21.15.141:0/907807479 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:59.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.690+0000 7f1530a00640 1 --2- 172.21.15.141:0/907807479 >> v2:172.21.15.188:3300/0 conn(0x7f152c13df80 0x7f152c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.690+0000 7f1531400640 1 -- 172.21.15.141:0/907807479 >> v2:172.21.15.188:3300/0 conn(0x7f152c13df80 msgr2=0x7f152c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:59.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.690+0000 7f1531400640 1 --2- 172.21.15.141:0/907807479 >> v2:172.21.15.188:3300/0 conn(0x7f152c13df80 0x7f152c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.690+0000 7f1531400640 1 -- 172.21.15.141:0/907807479 >> v1:172.21.15.141:6789/0 conn(0x7f152c13e360 legacy=0x7f152c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.688 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.690+0000 7f1531400640 1 -- 172.21.15.141:0/907807479 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c06cd10 con 0x7f152c0cf9a0 2024-01-16T15:23:59.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.691+0000 7f1531400640 1 --2- 172.21.15.141:0/907807479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c0cf9a0 0x7f152c0cd840 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f152c072770 tx=0x7f152402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1dbec5cb3890ffd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.691+0000 7f152be00640 1 -- 172.21.15.141:0/907807479 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524030860 con 0x7f152c0cf9a0 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.691+0000 7f152be00640 1 -- 172.21.15.141:0/907807479 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15240309c0 con 0x7f152c0cf9a0 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.691+0000 7f152be00640 1 -- 172.21.15.141:0/907807479 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524030d50 con 0x7f152c0cf9a0 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 -- 172.21.15.141:0/907807479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c0cf9a0 msgr2=0x7f152c0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 --2- 172.21.15.141:0/907807479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c0cf9a0 0x7f152c0cd840 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f152c072770 tx=0x7f152402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 -- 172.21.15.141:0/907807479 shutdown_connections 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 --2- 172.21.15.141:0/907807479 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c0cf9a0 0x7f152c0cd840 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.690 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 --2- 172.21.15.141:0/907807479 >> v2:172.21.15.188:3300/0 conn(0x7f152c13df80 0x7f152c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 -- 172.21.15.141:0/907807479 >> 172.21.15.141:0/907807479 conn(0x7f152c06a800 msgr2=0x7f152c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:59.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 -- 172.21.15.141:0/907807479 shutdown_connections 2024-01-16T15:23:59.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.692+0000 7f1533400640 1 -- 172.21.15.141:0/907807479 wait complete. 2024-01-16T15:23:59.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.694+0000 7f1533400640 1 Processor -- start 2024-01-16T15:23:59.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.694+0000 7f1533400640 1 -- start start 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1533400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f152c0cf9a0 0x7f152c13c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1533400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f152c13df80 0x7f152c0717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1533400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c13e360 0x7f152c0cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1533400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c0aaa90 con 0x7f152c13e360 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1533400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c072280 con 0x7f152c0cf9a0 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1533400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152c0aa160 con 0x7f152c13df80 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c13e360 0x7f152c0cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c13e360 0x7f152c0cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48650/0 (socket says 172.21.15.141:48650) 2024-01-16T15:23:59.694 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 -- 172.21.15.141:0/846194235 learned_addr learned my addr 172.21.15.141:0/846194235 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1523c00640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f152c13df80 0x7f152c0717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 -- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f152c13df80 msgr2=0x7f152c0717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1530a00640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f152c0cf9a0 0x7f152c13c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f152c13df80 0x7f152c0717c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 -- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f152c0cf9a0 msgr2=0x7f152c13c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f152c0cf9a0 0x7f152c13c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.695+0000 7f1531400640 1 -- 172.21.15.141:0/846194235 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240094d0 con 0x7f152c13e360 2024-01-16T15:23:59.695 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.696+0000 7f1531400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c13e360 0x7f152c0cdee0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f15240090b0 tx=0x7f1524004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:59.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.696+0000 7f1529600640 1 -- 172.21.15.141:0/846194235 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524039a90 con 0x7f152c13e360 2024-01-16T15:23:59.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.696+0000 7f1529600640 1 -- 172.21.15.141:0/846194235 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1524039bf0 con 0x7f152c13e360 2024-01-16T15:23:59.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.696+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152c0aa440 con 0x7f152c13e360 2024-01-16T15:23:59.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.696+0000 7f1529600640 1 -- 172.21.15.141:0/846194235 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524043410 con 0x7f152c13e360 2024-01-16T15:23:59.696 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.696+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152c13cf20 con 0x7f152c13e360 2024-01-16T15:23:59.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.697+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152c0675d0 con 0x7f152c13e360 2024-01-16T15:23:59.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.698+0000 7f1529600640 1 -- 172.21.15.141:0/846194235 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f152404c3b0 con 0x7f152c13e360 2024-01-16T15:23:59.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.699+0000 7f1529600640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f150406c610 0x7f150406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:59.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.699+0000 7f1529600640 1 -- 172.21.15.141:0/846194235 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f15240aad00 con 0x7f152c13e360 2024-01-16T15:23:59.697 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.700+0000 7f1530a00640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f150406c610 0x7f150406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:59.698 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.701+0000 7f1530a00640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f150406c610 0x7f150406ead0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f151c009f30 tx=0x7f151c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:59.701 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.704+0000 7f1529600640 1 -- 172.21.15.141:0/846194235 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15240775c0 con 0x7f152c13e360 2024-01-16T15:23:59.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.896+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f152c15f9c0 con 0x7f150406c610 2024-01-16T15:23:59.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.898+0000 7f1529600640 1 -- 172.21.15.141:0/846194235 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f151c009b20 con 0x7f150406c610 2024-01-16T15:23:59.895 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 0 to deep-scrub 2024-01-16T15:23:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f150406c610 msgr2=0x7f150406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f150406c610 0x7f150406ead0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f151c009f30 tx=0x7f151c00b040 comp rx=0 tx=0).stop 2024-01-16T15:23:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c13e360 msgr2=0x7f152c0cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c13e360 0x7f152c0cdee0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f15240090b0 tx=0x7f1524004730 comp rx=0 tx=0).stop 2024-01-16T15:23:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 shutdown_connections 2024-01-16T15:23:59.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f150406c610 0x7f150406ead0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f152c13e360 0x7f152c0cdee0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f152c13df80 0x7f152c0717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 --2- 172.21.15.141:0/846194235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f152c0cf9a0 0x7f152c13c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.900+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 >> 172.21.15.141:0/846194235 conn(0x7f152c06a800 msgr2=0x7f152c15c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.901+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 shutdown_connections 2024-01-16T15:23:59.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:23:59.901+0000 7f1533400640 1 -- 172.21.15.141:0/846194235 wait complete. 2024-01-16T15:23:59.914 INFO:tasks.ceph:Scrubbing osd.1 2024-01-16T15:23:59.915 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:24:00.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.118+0000 7f8f38200640 1 Processor -- start 2024-01-16T15:24:00.116 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.118+0000 7f8f38200640 1 -- start start 2024-01-16T15:24:00.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.120+0000 7f8f38200640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8f3013df80 0x7f8f3015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.120+0000 7f8f38200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 0x7f8f300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.120+0000 7f8f38200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f8f3015f9c0 0x7f8f300cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.120+0000 7f8f38200640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3006c470 con 0x7f8f3013df80 2024-01-16T15:24:00.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.120+0000 7f8f38200640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3006c750 con 0x7f8f3015f9c0 2024-01-16T15:24:00.119 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.120+0000 7f8f38200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3006ca30 con 0x7f8f3013e360 2024-01-16T15:24:00.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8f3013df80 0x7f8f3015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8f3013df80 0x7f8f3015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48660/0 (socket says 172.21.15.141:48660) 2024-01-16T15:24:00.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 -- 172.21.15.141:0/1018665079 learned_addr learned my addr 172.21.15.141:0/1018665079 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:00.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f34e00640 1 --2- 172.21.15.141:0/1018665079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 0x7f8f300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f36200640 1 --2- 172.21.15.141:0/1018665079 >> v2:172.21.15.188:3300/0 conn(0x7f8f3015f9c0 0x7f8f300cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 -- 172.21.15.141:0/1018665079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 msgr2=0x7f8f300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:00.120 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 --2- 172.21.15.141:0/1018665079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 0x7f8f300cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 -- 172.21.15.141:0/1018665079 >> v2:172.21.15.188:3300/0 conn(0x7f8f3015f9c0 msgr2=0x7f8f300cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:00.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 --2- 172.21.15.141:0/1018665079 >> v2:172.21.15.188:3300/0 conn(0x7f8f3015f9c0 0x7f8f300cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.121 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.121+0000 7f8f35800640 1 -- 172.21.15.141:0/1018665079 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3006cd10 con 0x7f8f3013df80 2024-01-16T15:24:00.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.123+0000 7f8f35800640 1 --2- 172.21.15.141:0/1018665079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015bee0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f8f3006e6d0 tx=0x7f8f2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da45637299fb29a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.123+0000 7f8f2fe00640 1 -- 172.21.15.141:0/1018665079 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f20031870 con 0x7f8f3013df80 2024-01-16T15:24:00.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.123+0000 7f8f2fe00640 1 -- 172.21.15.141:0/1018665079 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f200319d0 con 0x7f8f3013df80 2024-01-16T15:24:00.122 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.123+0000 7f8f2fe00640 1 -- 172.21.15.141:0/1018665079 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f20031d60 con 0x7f8f3013df80 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 -- 172.21.15.141:0/1018665079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 msgr2=0x7f8f3015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 --2- 172.21.15.141:0/1018665079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015bee0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f8f3006e6d0 tx=0x7f8f2002b910 comp rx=0 tx=0).stop 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 -- 172.21.15.141:0/1018665079 shutdown_connections 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 --2- 172.21.15.141:0/1018665079 >> v2:172.21.15.188:3300/0 conn(0x7f8f3015f9c0 0x7f8f300cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 --2- 172.21.15.141:0/1018665079 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 0x7f8f300cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 --2- 172.21.15.141:0/1018665079 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015bee0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 -- 172.21.15.141:0/1018665079 >> 172.21.15.141:0/1018665079 conn(0x7f8f3006a800 msgr2=0x7f8f3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:00.123 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 -- 172.21.15.141:0/1018665079 shutdown_connections 2024-01-16T15:24:00.124 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.124+0000 7f8f38200640 1 -- 172.21.15.141:0/1018665079 wait complete. 2024-01-16T15:24:00.126 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.126+0000 7f8f38200640 1 Processor -- start 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.126+0000 7f8f38200640 1 -- start start 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f38200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f38200640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 0x7f8f301575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f38200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f3015f9c0 0x7f8f30157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f38200640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f301f0b30 con 0x7f8f3013df80 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f38200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3013a270 con 0x7f8f3015f9c0 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f38200640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3015de80 con 0x7f8f3013e360 2024-01-16T15:24:00.127 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48666/0 (socket says 172.21.15.141:48666) 2024-01-16T15:24:00.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 -- 172.21.15.141:0/4112356957 learned_addr learned my addr 172.21.15.141:0/4112356957 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:00.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 -- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 msgr2=0x7f8f301575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8f3013e360 0x7f8f301575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 -- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f3015f9c0 msgr2=0x7f8f30157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f3015f9c0 0x7f8f30157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.127+0000 7f8f35800640 1 -- 172.21.15.141:0/4112356957 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f200094d0 con 0x7f8f3013df80 2024-01-16T15:24:00.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.128+0000 7f8f35800640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015a240 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000c00 tx=0x7f8f200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.128+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2003ad30 con 0x7f8f3013df80 2024-01-16T15:24:00.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.128+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2003ae90 con 0x7f8f3013df80 2024-01-16T15:24:00.129 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.128+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f20044630 con 0x7f8f3013df80 2024-01-16T15:24:00.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.128+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3015a940 con 0x7f8f3013df80 2024-01-16T15:24:00.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.128+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3015af00 con 0x7f8f3013df80 2024-01-16T15:24:00.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.129+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f3013a060 con 0x7f8f3013df80 2024-01-16T15:24:00.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.130+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f8f20044790 con 0x7f8f3013df80 2024-01-16T15:24:00.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.131+0000 7f8f2d600640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8f0806c5e0 0x7f8f0806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.131+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8f2002d040 con 0x7f8f3013df80 2024-01-16T15:24:00.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.131+0000 7f8f2d600640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f8f08074840 0x7f8f08076c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.131+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 --> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f08077350 con 0x7f8f08074840 2024-01-16T15:24:00.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.131+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f200ab580 con 0x7f8f3013df80 2024-01-16T15:24:00.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.131+0000 7f8f36200640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f8f08074840 0x7f8f08076c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.131+0000 7f8f34e00640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8f0806c5e0 0x7f8f0806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.132+0000 7f8f36200640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f8f08074840 0x7f8f08076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.132+0000 7f8f34e00640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8f0806c5e0 0x7f8f0806eaa0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8f18006fd0 tx=0x7f8f18008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.132+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== osd.1 v2:172.21.15.141:6808/3479163278 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8f08077350 con 0x7f8f08074840 2024-01-16T15:24:00.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.147+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 --> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8f301408d0 con 0x7f8f08074840 2024-01-16T15:24:00.144 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.147+0000 7f8f2d600640 1 -- 172.21.15.141:0/4112356957 <== osd.1 v2:172.21.15.141:6808/3479163278 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8f28009710 con 0x7f8f08074840 2024-01-16T15:24:00.145 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:24:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:24:00.145 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:24:00.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.147+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f8f08074840 msgr2=0x7f8f08076c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.147+0000 7f8f38200640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:6808/3479163278,v1:172.21.15.141:6809/3479163278] conn(0x7f8f08074840 0x7f8f08076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8f0806c5e0 msgr2=0x7f8f0806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f8f0806c5e0 0x7f8f0806eaa0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8f18006fd0 tx=0x7f8f18008040 comp rx=0 tx=0).stop 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 msgr2=0x7f8f3015a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 --2- 172.21.15.141:0/4112356957 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8f3013df80 0x7f8f3015a240 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000c00 tx=0x7f8f200046c0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f36200640 1 -- 172.21.15.141:0/4112356957 reap_dead start 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 shutdown_connections 2024-01-16T15:24:00.147 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 >> 172.21.15.141:0/4112356957 conn(0x7f8f3006a800 msgr2=0x7f8f3015e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:00.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 shutdown_connections 2024-01-16T15:24:00.148 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.148+0000 7f8f38200640 1 -- 172.21.15.141:0/4112356957 wait complete. 2024-01-16T15:24:00.162 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-16T15:24:00.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.366+0000 7f7549000640 1 Processor -- start 2024-01-16T15:24:00.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.366+0000 7f7549000640 1 -- start start 2024-01-16T15:24:00.365 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.368+0000 7f7549000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7544143ae0 0x7f75440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.368+0000 7f7549000640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440cfc30 con 0x7f754415e3b0 2024-01-16T15:24:00.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.368+0000 7f7549000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440b1ee0 con 0x7f75440cf7d0 2024-01-16T15:24:00.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.368+0000 7f7549000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75440b2780 con 0x7f7544143ae0 2024-01-16T15:24:00.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.368+0000 7f7541e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7544143ae0 0x7f75440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.368+0000 7f7541e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7544143ae0 0x7f75440cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48672/0 (socket says 172.21.15.141:48672) 2024-01-16T15:24:00.366 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.368+0000 7f7541e00640 1 -- 172.21.15.141:0/24756166 learned_addr learned my addr 172.21.15.141:0/24756166 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:00.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.369+0000 7f7541e00640 1 -- 172.21.15.141:0/24756166 >> v1:172.21.15.188:6789/0 conn(0x7f75440cf7d0 legacy=0x7f754415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:00.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.369+0000 7f7541e00640 1 -- 172.21.15.141:0/24756166 >> v1:172.21.15.141:6789/0 conn(0x7f754415e3b0 legacy=0x7f75440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.369+0000 7f7541e00640 1 -- 172.21.15.141:0/24756166 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754415e990 con 0x7f7544143ae0 2024-01-16T15:24:00.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.370+0000 7f7541e00640 1 --2- 172.21.15.141:0/24756166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75440cf290 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7534009870 tx=0x7f753402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9597237a94da55e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.370+0000 7f7540a00640 1 -- 172.21.15.141:0/24756166 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534030850 con 0x7f7544143ae0 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.370+0000 7f7540a00640 1 -- 172.21.15.141:0/24756166 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75340309b0 con 0x7f7544143ae0 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.370+0000 7f7540a00640 1 -- 172.21.15.141:0/24756166 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534030d40 con 0x7f7544143ae0 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.371+0000 7f7549000640 1 -- 172.21.15.141:0/24756166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 msgr2=0x7f75440cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.371+0000 7f7549000640 1 --2- 172.21.15.141:0/24756166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75440cf290 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7534009870 tx=0x7f753402b990 comp rx=0 tx=0).stop 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.371+0000 7f7549000640 1 -- 172.21.15.141:0/24756166 shutdown_connections 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.371+0000 7f7549000640 1 --2- 172.21.15.141:0/24756166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75440cf290 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.371+0000 7f7549000640 1 -- 172.21.15.141:0/24756166 >> 172.21.15.141:0/24756166 conn(0x7f754406a800 msgr2=0x7f754406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:00.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.371+0000 7f7549000640 1 -- 172.21.15.141:0/24756166 shutdown_connections 2024-01-16T15:24:00.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.371+0000 7f7549000640 1 -- 172.21.15.141:0/24756166 wait complete. 2024-01-16T15:24:00.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.373+0000 7f7549000640 1 Processor -- start 2024-01-16T15:24:00.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.373+0000 7f7549000640 1 -- start start 2024-01-16T15:24:00.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7549000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75440cf7d0 0x7f7544141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7549000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75441f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7549000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f754415e3b0 0x7f75441f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7549000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754413b8e0 con 0x7f7544143ae0 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7549000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75441f4d50 con 0x7f75440cf7d0 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7549000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75441f5030 con 0x7f754415e3b0 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7541400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75441f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7541400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75441f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48682/0 (socket says 172.21.15.141:48682) 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7541400640 1 -- 172.21.15.141:0/2214352864 learned_addr learned my addr 172.21.15.141:0/2214352864 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.374+0000 7f7541e00640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75440cf7d0 0x7f7544141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7541400640 1 -- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f754415e3b0 msgr2=0x7f75441f46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7542800640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f754415e3b0 0x7f75441f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7541400640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f754415e3b0 0x7f75441f46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7541400640 1 -- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75440cf7d0 msgr2=0x7f7544141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7541400640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75440cf7d0 0x7f7544141690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7541400640 1 -- 172.21.15.141:0/2214352864 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75340094d0 con 0x7f7544143ae0 2024-01-16T15:24:00.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7541e00640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75440cf7d0 0x7f7544141690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:24:00.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7542800640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f754415e3b0 0x7f75441f46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:00.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7541400640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75441f4190 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f752c00c7c0 tx=0x7f752c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f753a000640 1 -- 172.21.15.141:0/2214352864 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c00aa50 con 0x7f7544143ae0 2024-01-16T15:24:00.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f753a000640 1 -- 172.21.15.141:0/2214352864 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752c00abb0 con 0x7f7544143ae0 2024-01-16T15:24:00.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f753a000640 1 -- 172.21.15.141:0/2214352864 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c012420 con 0x7f7544143ae0 2024-01-16T15:24:00.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.375+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754413b290 con 0x7f7544143ae0 2024-01-16T15:24:00.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.376+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75441400b0 con 0x7f7544143ae0 2024-01-16T15:24:00.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.377+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75440675d0 con 0x7f7544143ae0 2024-01-16T15:24:00.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.378+0000 7f753a000640 1 -- 172.21.15.141:0/2214352864 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f752c012580 con 0x7f7544143ae0 2024-01-16T15:24:00.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.379+0000 7f753a000640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f751c06c5e0 0x7f751c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.379+0000 7f753a000640 1 -- 172.21.15.141:0/2214352864 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f752c088de0 con 0x7f7544143ae0 2024-01-16T15:24:00.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.380+0000 7f7541e00640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f751c06c5e0 0x7f751c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.381+0000 7f7541e00640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f751c06c5e0 0x7f751c06eaa0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7534036000 tx=0x7f7534008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.384+0000 7f753a000640 1 -- 172.21.15.141:0/2214352864 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f752c055360 con 0x7f7544143ae0 2024-01-16T15:24:00.576 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.579+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f75440b2410 con 0x7f751c06c5e0 2024-01-16T15:24:00.577 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.580+0000 7f753a000640 1 -- 172.21.15.141:0/2214352864 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7534044470 con 0x7f751c06c5e0 2024-01-16T15:24:00.577 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 1 to deep-scrub 2024-01-16T15:24:00.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.582+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f751c06c5e0 msgr2=0x7f751c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.580 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.582+0000 7f7549000640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f751c06c5e0 0x7f751c06eaa0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7534036000 tx=0x7f7534008030 comp rx=0 tx=0).stop 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.582+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 msgr2=0x7f75441f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.582+0000 7f7549000640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75441f4190 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f752c00c7c0 tx=0x7f752c00cc80 comp rx=0 tx=0).stop 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 shutdown_connections 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f751c06c5e0 0x7f751c06eaa0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f754415e3b0 0x7f75441f46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7544143ae0 0x7f75441f4190 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 --2- 172.21.15.141:0/2214352864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75440cf7d0 0x7f7544141690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.581 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 >> 172.21.15.141:0/2214352864 conn(0x7f754406a800 msgr2=0x7f754415a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:00.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 shutdown_connections 2024-01-16T15:24:00.582 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.583+0000 7f7549000640 1 -- 172.21.15.141:0/2214352864 wait complete. 2024-01-16T15:24:00.598 INFO:tasks.ceph:Scrubbing osd.2 2024-01-16T15:24:00.599 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:24:00.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.819+0000 7f495fe00640 1 Processor -- start 2024-01-16T15:24:00.817 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.819+0000 7f495fe00640 1 -- start start 2024-01-16T15:24:00.818 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.821+0000 7f495fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600cc990 0x7f496006d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.821+0000 7f495fe00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4960070e50 con 0x7f49600cc500 2024-01-16T15:24:00.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.821+0000 7f495fe00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4960144020 con 0x7f4960143c40 2024-01-16T15:24:00.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.821+0000 7f495fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49600723c0 con 0x7f49600cc990 2024-01-16T15:24:00.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.821+0000 7f495e000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f49600cc500 0x7f496006d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54368/0 (socket says 172.21.15.141:54368) 2024-01-16T15:24:00.819 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.821+0000 7f495e000640 1 -- 172.21.15.141:0/1050741833 learned_addr learned my addr 172.21.15.141:0/1050741833 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:24:00.821 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.822+0000 7f495f400640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600cc990 0x7f496006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.822+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797464684 0 0) 0x7f4960070e50 con 0x7f49600cc500 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.822+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4940003520 con 0x7f49600cc500 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1966907899 0 0) 0x7f4940003520 con 0x7f49600cc500 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600cc990 msgr2=0x7f496006d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600cc990 0x7f496006d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 >> v1:172.21.15.188:6789/0 conn(0x7f4960143c40 legacy=0x7f4960071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4960071e00 con 0x7f49600cc500 2024-01-16T15:24:00.822 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f4948001690 con 0x7f49600cc500 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 >> v1:172.21.15.141:6789/0 conn(0x7f49600cc500 legacy=0x7f496006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f4940003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4940004350 0x7f4940004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4940005070 0x7f494000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49600723c0 con 0x7f49400039a0 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494000bfa0 con 0x7f4940005070 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494000c160 con 0x7f4940004350 2024-01-16T15:24:00.823 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.823+0000 7f495ea00640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f4940003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495f400640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4940005070 0x7f494000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495e000640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4940004350 0x7f4940004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495ea00640 1 -- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4940004350 msgr2=0x7f4940004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:00.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495ea00640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4940004350 0x7f4940004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495ea00640 1 -- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4940005070 msgr2=0x7f494000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:00.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495ea00640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4940005070 0x7f494000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.824 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495ea00640 1 -- 172.21.15.141:0/1050741833 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940003520 con 0x7f49400039a0 2024-01-16T15:24:00.825 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.824+0000 7f495e000640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f4940004350 0x7f4940004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:00.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.825+0000 7f495ea00640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f4940003d80 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f4950001860 tx=0x7f49400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e730ac7c3c590f1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.825+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950033630 con 0x7f49400039a0 2024-01-16T15:24:00.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.825+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49500337b0 con 0x7f49400039a0 2024-01-16T15:24:00.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.825+0000 7f495d600640 1 -- 172.21.15.141:0/1050741833 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950033b60 con 0x7f49400039a0 2024-01-16T15:24:00.826 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.826+0000 7f495fe00640 1 -- 172.21.15.141:0/1050741833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 msgr2=0x7f4940003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.826+0000 7f495fe00640 1 --2- 172.21.15.141:0/1050741833 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f4940003d80 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f4950001860 tx=0x7f49400057f0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.826+0000 7f495f400640 1 -- 172.21.15.141:0/1050741833 reap_dead start 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.826+0000 7f495fe00640 1 -- 172.21.15.141:0/1050741833 shutdown_connections 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.826+0000 7f495fe00640 1 -- 172.21.15.141:0/1050741833 >> 172.21.15.141:0/1050741833 conn(0x7f496006a800 msgr2=0x7f496006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.826+0000 7f495fe00640 1 -- 172.21.15.141:0/1050741833 shutdown_connections 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.826+0000 7f495fe00640 1 -- 172.21.15.141:0/1050741833 wait complete. 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.828+0000 7f495fe00640 1 Processor -- start 2024-01-16T15:24:00.827 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.829+0000 7f495fe00640 1 -- start start 2024-01-16T15:24:00.829 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495fe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f496015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600b2f50 0x7f49600b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49600ca390 0x7f49600ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495fe00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49600c5c70 con 0x7f49400039a0 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496015af70 con 0x7f49600ca390 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495fe00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49600b38f0 con 0x7f49600b2f50 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f496015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495ea00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f496015aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48696/0 (socket says 172.21.15.141:48696) 2024-01-16T15:24:00.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.830+0000 7f495ea00640 1 -- 172.21.15.141:0/3675872626 learned_addr learned my addr 172.21.15.141:0/3675872626 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495e000640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600b2f50 0x7f49600b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495f400640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49600ca390 0x7f49600ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495ea00640 1 -- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600b2f50 msgr2=0x7f49600b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495ea00640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f49600b2f50 0x7f49600b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495ea00640 1 -- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49600ca390 msgr2=0x7f49600ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495ea00640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49600ca390 0x7f49600ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495ea00640 1 -- 172.21.15.141:0/3675872626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49600c5640 con 0x7f49400039a0 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495f400640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49600ca390 0x7f49600ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:00.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f495ea00640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f496015aa30 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f495002bdf0 tx=0x7f4950001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.831+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49500037f0 con 0x7f49400039a0 2024-01-16T15:24:00.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.832+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4950003950 con 0x7f49400039a0 2024-01-16T15:24:00.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.832+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950003d00 con 0x7f49400039a0 2024-01-16T15:24:00.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.832+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49600cb120 con 0x7f49400039a0 2024-01-16T15:24:00.832 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.832+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49600c5890 con 0x7f49400039a0 2024-01-16T15:24:00.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.833+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f496006cc40 con 0x7f49400039a0 2024-01-16T15:24:00.833 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.834+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f4950047460 con 0x7f49400039a0 2024-01-16T15:24:00.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.835+0000 7f4957400640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f493806c610 0x7f493806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.835+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f49500ab110 con 0x7f49400039a0 2024-01-16T15:24:00.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.836+0000 7f4957400640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f4938074870 0x7f4938076cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:00.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.836+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 --> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4938077380 con 0x7f4938074870 2024-01-16T15:24:00.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.836+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49500abeb0 con 0x7f49400039a0 2024-01-16T15:24:00.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.836+0000 7f495e000640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f493806c610 0x7f493806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.836+0000 7f495f400640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f4938074870 0x7f4938076cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:00.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.837+0000 7f495f400640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f4938074870 0x7f4938076cb0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.837+0000 7f495e000640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f493806c610 0x7f493806ead0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f49480017c0 tx=0x7f4948009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:00.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.838+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== osd.2 v2:172.21.15.141:6815/1316482903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4938077380 con 0x7f4938074870 2024-01-16T15:24:00.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.855+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 --> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f496006c960 con 0x7f4938074870 2024-01-16T15:24:00.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.855+0000 7f4957400640 1 -- 172.21.15.141:0/3675872626 <== osd.2 v2:172.21.15.141:6815/1316482903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4958016400 con 0x7f4938074870 2024-01-16T15:24:00.853 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:24:00.853 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:24:00.853 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:24:00.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.855+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f4938074870 msgr2=0x7f4938076cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.855+0000 7f495fe00640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:6815/1316482903,v1:172.21.15.141:6817/1316482903] conn(0x7f4938074870 0x7f4938076cb0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:00.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f493806c610 msgr2=0x7f493806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f493806c610 0x7f493806ead0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f49480017c0 tx=0x7f4948009040 comp rx=0 tx=0).stop 2024-01-16T15:24:00.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 msgr2=0x7f496015aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:00.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 --2- 172.21.15.141:0/3675872626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f49400039a0 0x7f496015aa30 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f495002bdf0 tx=0x7f4950001230 comp rx=0 tx=0).stop 2024-01-16T15:24:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495f400640 1 -- 172.21.15.141:0/3675872626 reap_dead start 2024-01-16T15:24:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 shutdown_connections 2024-01-16T15:24:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 >> 172.21.15.141:0/3675872626 conn(0x7f496006a800 msgr2=0x7f49601400d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 shutdown_connections 2024-01-16T15:24:00.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:00.856+0000 7f495fe00640 1 -- 172.21.15.141:0/3675872626 wait complete. 2024-01-16T15:24:00.871 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-16T15:24:01.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.071+0000 7f545a400640 1 Processor -- start 2024-01-16T15:24:01.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.071+0000 7f545a400640 1 -- start start 2024-01-16T15:24:01.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.072+0000 7f545a400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5454143ae0 0x7f54540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.072+0000 7f545a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54540cf7d0 0x7f545415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.072+0000 7f545a400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540cfc30 con 0x7f545415e1e0 2024-01-16T15:24:01.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.072+0000 7f545a400640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540b1ee0 con 0x7f5454143ae0 2024-01-16T15:24:01.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.072+0000 7f545a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540b2780 con 0x7f54540cf7d0 2024-01-16T15:24:01.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.073+0000 7f5453400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5454143ae0 0x7f54540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.073+0000 7f5453400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5454143ae0 0x7f54540cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48706/0 (socket says 172.21.15.141:48706) 2024-01-16T15:24:01.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.073+0000 7f5453400640 1 -- 172.21.15.141:0/1729792083 learned_addr learned my addr 172.21.15.141:0/1729792083 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:01.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.073+0000 7f5453400640 1 -- 172.21.15.141:0/1729792083 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54540cf7d0 msgr2=0x7f545415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:01.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.073+0000 7f5453400640 1 --2- 172.21.15.141:0/1729792083 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54540cf7d0 0x7f545415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.073+0000 7f5453400640 1 -- 172.21.15.141:0/1729792083 >> v1:172.21.15.188:6789/0 conn(0x7f545415e1e0 legacy=0x7f545415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:01.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.073+0000 7f5453400640 1 -- 172.21.15.141:0/1729792083 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545415e990 con 0x7f5454143ae0 2024-01-16T15:24:01.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.074+0000 7f5453400640 1 --2- 172.21.15.141:0/1729792083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5454143ae0 0x7f54540cf290 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f544400bb10 tx=0x7f544402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6f80f3db3a4f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f5452a00640 1 -- 172.21.15.141:0/1729792083 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544402ee70 con 0x7f5454143ae0 2024-01-16T15:24:01.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f5452a00640 1 -- 172.21.15.141:0/1729792083 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5444036440 con 0x7f5454143ae0 2024-01-16T15:24:01.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f5452a00640 1 -- 172.21.15.141:0/1729792083 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54440367d0 con 0x7f5454143ae0 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 -- 172.21.15.141:0/1729792083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5454143ae0 msgr2=0x7f54540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 --2- 172.21.15.141:0/1729792083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5454143ae0 0x7f54540cf290 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f544400bb10 tx=0x7f544402e0c0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 -- 172.21.15.141:0/1729792083 shutdown_connections 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 --2- 172.21.15.141:0/1729792083 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f54540cf7d0 0x7f545415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 --2- 172.21.15.141:0/1729792083 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5454143ae0 0x7f54540cf290 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 -- 172.21.15.141:0/1729792083 >> 172.21.15.141:0/1729792083 conn(0x7f545406a800 msgr2=0x7f545406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 -- 172.21.15.141:0/1729792083 shutdown_connections 2024-01-16T15:24:01.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.075+0000 7f545a400640 1 -- 172.21.15.141:0/1729792083 wait complete. 2024-01-16T15:24:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.077+0000 7f545a400640 1 Processor -- start 2024-01-16T15:24:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.077+0000 7f545a400640 1 -- start start 2024-01-16T15:24:01.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f545a400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54540cf7d0 0x7f5454159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f545a400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5454143ae0 0x7f545415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f545a400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f545415e1e0 0x7f54540c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f545a400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540c4ef0 con 0x7f545415e1e0 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f545a400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540c42e0 con 0x7f54540cf7d0 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f545a400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540c45c0 con 0x7f5454143ae0 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f5453e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f545415e1e0 0x7f54540c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.078+0000 7f5453e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f545415e1e0 0x7f54540c5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48712/0 (socket says 172.21.15.141:48712) 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453e00640 1 -- 172.21.15.141:0/1346811869 learned_addr learned my addr 172.21.15.141:0/1346811869 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:01.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54540cf7d0 0x7f5454159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f544a800640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5454143ae0 0x7f545415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453e00640 1 -- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5454143ae0 msgr2=0x7f545415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453e00640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5454143ae0 0x7f545415ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453e00640 1 -- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54540cf7d0 msgr2=0x7f5454159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453e00640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54540cf7d0 0x7f5454159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453e00640 1 -- 172.21.15.141:0/1346811869 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544400b770 con 0x7f545415e1e0 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54540cf7d0 0x7f5454159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:01.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f5453e00640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f545415e1e0 0x7f54540c5380 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f544c00b970 tx=0x7f544c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f544be00640 1 -- 172.21.15.141:0/1346811869 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c00cb40 con 0x7f545415e1e0 2024-01-16T15:24:01.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f544be00640 1 -- 172.21.15.141:0/1346811869 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c00cca0 con 0x7f545415e1e0 2024-01-16T15:24:01.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.079+0000 7f544be00640 1 -- 172.21.15.141:0/1346811869 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c014680 con 0x7f545415e1e0 2024-01-16T15:24:01.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.080+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54540c48a0 con 0x7f545415e1e0 2024-01-16T15:24:01.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.080+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5454159db0 con 0x7f545415e1e0 2024-01-16T15:24:01.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.081+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54540b2bf0 con 0x7f545415e1e0 2024-01-16T15:24:01.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.082+0000 7f544be00640 1 -- 172.21.15.141:0/1346811869 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f544c01d3b0 con 0x7f545415e1e0 2024-01-16T15:24:01.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.082+0000 7f544be00640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f542c06c5e0 0x7f542c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.081 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.083+0000 7f544be00640 1 -- 172.21.15.141:0/1346811869 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f544c088a20 con 0x7f545415e1e0 2024-01-16T15:24:01.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.083+0000 7f5453400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f542c06c5e0 0x7f542c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.082 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.084+0000 7f5453400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f542c06c5e0 0x7f542c06eaa0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5444038000 tx=0x7f5444008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.084 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.087+0000 7f544be00640 1 -- 172.21.15.141:0/1346811869 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54540b2bf0 con 0x7f545415e1e0 2024-01-16T15:24:01.274 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.276+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f545415e640 con 0x7f542c06c5e0 2024-01-16T15:24:01.275 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 2 to deep-scrub 2024-01-16T15:24:01.275 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.277+0000 7f544be00640 1 -- 172.21.15.141:0/1346811869 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5444036980 con 0x7f542c06c5e0 2024-01-16T15:24:01.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f542c06c5e0 msgr2=0x7f542c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f542c06c5e0 0x7f542c06eaa0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5444038000 tx=0x7f5444008040 comp rx=0 tx=0).stop 2024-01-16T15:24:01.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f545415e1e0 msgr2=0x7f54540c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f545415e1e0 0x7f54540c5380 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f544c00b970 tx=0x7f544c00be30 comp rx=0 tx=0).stop 2024-01-16T15:24:01.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 shutdown_connections 2024-01-16T15:24:01.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f542c06c5e0 0x7f542c06eaa0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.278 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f545415e1e0 0x7f54540c5380 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f5454143ae0 0x7f545415ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 --2- 172.21.15.141:0/1346811869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54540cf7d0 0x7f5454159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 >> 172.21.15.141:0/1346811869 conn(0x7f545406a800 msgr2=0x7f54540ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:01.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 shutdown_connections 2024-01-16T15:24:01.279 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.280+0000 7f545a400640 1 -- 172.21.15.141:0/1346811869 wait complete. 2024-01-16T15:24:01.297 INFO:tasks.ceph:Scrubbing osd.3 2024-01-16T15:24:01.298 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:24:01.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.516+0000 7f279f400640 1 Processor -- start 2024-01-16T15:24:01.514 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.517+0000 7f279f400640 1 -- start start 2024-01-16T15:24:01.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.518+0000 7f279f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 0x7f27980cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.515 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.518+0000 7f279f400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279815c5f0 con 0x7f279813df80 2024-01-16T15:24:01.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.518+0000 7f279f400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279806c470 con 0x7f279815f9c0 2024-01-16T15:24:01.516 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.518+0000 7f279f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279806c750 con 0x7f279813e360 2024-01-16T15:24:01.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f279ca00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f279813df80 0x7f279815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54380/0 (socket says 172.21.15.141:54380) 2024-01-16T15:24:01.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f279ca00640 1 -- 172.21.15.141:0/2407655705 learned_addr learned my addr 172.21.15.141:0/2407655705 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:24:01.518 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f278fc00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 0x7f27980cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230481902 0 0) 0x7f279815c5f0 con 0x7f279813df80 2024-01-16T15:24:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2778003520 con 0x7f279813df80 2024-01-16T15:24:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723823215 0 0) 0x7f2778003520 con 0x7f279813df80 2024-01-16T15:24:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 msgr2=0x7f27980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f2797e00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 0x7f27980cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.519+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 >> v1:172.21.15.188:6789/0 conn(0x7f279815f9c0 legacy=0x7f27980cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.519 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279806ca30 con 0x7f279813df80 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f2788001670 con 0x7f279813df80 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 >> v1:172.21.15.141:6789/0 conn(0x7f279813df80 legacy=0x7f279815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f2778003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27780047c0 0x7f2778004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 0x7f2778005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279806c750 con 0x7f27780039a0 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277800ba90 con 0x7f27780047c0 2024-01-16T15:24:01.520 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277800bc50 con 0x7f279813e360 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f279ca00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f2778003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f279d400640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 0x7f2778005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f278fc00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27780047c0 0x7f2778004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f279ca00640 1 -- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 msgr2=0x7f2778005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f279ca00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813e360 0x7f2778005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f279ca00640 1 -- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27780047c0 msgr2=0x7f2778004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f279ca00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27780047c0 0x7f2778004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.521 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.520+0000 7f279ca00640 1 -- 172.21.15.141:0/2407655705 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2778003520 con 0x7f27780039a0 2024-01-16T15:24:01.524 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.522+0000 7f279ca00640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f2778003dc0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f2788001eb0 tx=0x7f27780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3fe90b92bbf928 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.522+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2788005ad0 con 0x7f27780039a0 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.522+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2788005c30 con 0x7f27780039a0 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.522+0000 7f2797e00640 1 -- 172.21.15.141:0/2407655705 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278802c7c0 con 0x7f27780039a0 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.522+0000 7f279f400640 1 -- 172.21.15.141:0/2407655705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 msgr2=0x7f2778003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.522+0000 7f279f400640 1 --2- 172.21.15.141:0/2407655705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f2778003dc0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f2788001eb0 tx=0x7f27780057f0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.522+0000 7f279d400640 1 -- 172.21.15.141:0/2407655705 reap_dead start 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.523+0000 7f279f400640 1 -- 172.21.15.141:0/2407655705 shutdown_connections 2024-01-16T15:24:01.525 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.523+0000 7f279f400640 1 -- 172.21.15.141:0/2407655705 >> 172.21.15.141:0/2407655705 conn(0x7f279806a800 msgr2=0x7f279806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.523+0000 7f279f400640 1 -- 172.21.15.141:0/2407655705 shutdown_connections 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.523+0000 7f279f400640 1 -- 172.21.15.141:0/2407655705 wait complete. 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.524+0000 7f279f400640 1 Processor -- start 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.525+0000 7f279f400640 1 -- start start 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.525+0000 7f279f400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f27980b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.525+0000 7f279f400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813d7d0 0x7f27980ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.525+0000 7f279f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27980cd800 0x7f27980cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.526 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.525+0000 7f279f400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27980ce1d0 con 0x7f27780039a0 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.525+0000 7f279f400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279813dc50 con 0x7f27980cd800 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.525+0000 7f279f400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27980a8560 con 0x7f279813d7d0 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f27980b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f27980b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48736/0 (socket says 172.21.15.141:48736) 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 -- 172.21.15.141:0/2432584355 learned_addr learned my addr 172.21.15.141:0/2432584355 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f278fc00640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813d7d0 0x7f27980ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279d400640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27980cd800 0x7f27980cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 -- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813d7d0 msgr2=0x7f27980ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.527 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f279813d7d0 0x7f27980ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 -- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27980cd800 msgr2=0x7f27980cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27980cd800 0x7f27980cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279ca00640 1 -- 172.21.15.141:0/2432584355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27980adfe0 con 0x7f27780039a0 2024-01-16T15:24:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.526+0000 7f279d400640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27980cd800 0x7f27980cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.527+0000 7f279ca00640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f27980b4000 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f278802c360 tx=0x7f2788001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.527+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278802cbd0 con 0x7f27780039a0 2024-01-16T15:24:01.528 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.527+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278802cd30 con 0x7f27780039a0 2024-01-16T15:24:01.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.527+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278802cea0 con 0x7f27780039a0 2024-01-16T15:24:01.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.527+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27980a8800 con 0x7f27780039a0 2024-01-16T15:24:01.529 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.527+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27980ae2c0 con 0x7f27780039a0 2024-01-16T15:24:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.528+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f279806cff0 con 0x7f27780039a0 2024-01-16T15:24:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.529+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f2788032070 con 0x7f27780039a0 2024-01-16T15:24:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.530+0000 7f2795600640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277006c5e0 0x7f277006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.530 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.530+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f27880aa3b0 con 0x7f27780039a0 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.530+0000 7f2795600640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f2770074840 0x7f2770076c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.530+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 --> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2770077350 con 0x7f2770074840 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.530+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27880aa720 con 0x7f27780039a0 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.531+0000 7f279d400640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f2770074840 0x7f2770076c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.531+0000 7f278fc00640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277006c5e0 0x7f277006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.531+0000 7f279d400640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f2770074840 0x7f2770076c80 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.532+0000 7f278fc00640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277006c5e0 0x7f277006eaa0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f27800099b0 tx=0x7f2780008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.531 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.532+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== osd.3 v2:172.21.15.141:6823/1855057890 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2770077350 con 0x7f2770074840 2024-01-16T15:24:01.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.547+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 --> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f27980675d0 con 0x7f2770074840 2024-01-16T15:24:01.544 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.547+0000 7f2795600640 1 -- 172.21.15.141:0/2432584355 <== osd.3 v2:172.21.15.141:6823/1855057890 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f27980675d0 con 0x7f2770074840 2024-01-16T15:24:01.545 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:24:01.545 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:24:01.545 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:24:01.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f2770074840 msgr2=0x7f2770076c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:6823/1855057890,v1:172.21.15.141:6824/1855057890] conn(0x7f2770074840 0x7f2770076c80 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277006c5e0 msgr2=0x7f277006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f277006c5e0 0x7f277006eaa0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f27800099b0 tx=0x7f2780008040 comp rx=0 tx=0).stop 2024-01-16T15:24:01.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 msgr2=0x7f27980b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 --2- 172.21.15.141:0/2432584355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27780039a0 0x7f27980b4000 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f278802c360 tx=0x7f2788001930 comp rx=0 tx=0).stop 2024-01-16T15:24:01.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279d400640 1 -- 172.21.15.141:0/2432584355 reap_dead start 2024-01-16T15:24:01.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 shutdown_connections 2024-01-16T15:24:01.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 >> 172.21.15.141:0/2432584355 conn(0x7f279806a800 msgr2=0x7f27980702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:01.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 shutdown_connections 2024-01-16T15:24:01.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.548+0000 7f279f400640 1 -- 172.21.15.141:0/2432584355 wait complete. 2024-01-16T15:24:01.562 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-16T15:24:01.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.770+0000 7f18f8800640 1 Processor -- start 2024-01-16T15:24:01.768 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.770+0000 7f18f8800640 1 -- start start 2024-01-16T15:24:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.772+0000 7f18f8800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18f0160980 0x7f18f006ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.772+0000 7f18f8800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f18f0161a10 0x7f18f00ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.770 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.772+0000 7f18f8800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f0161580 con 0x7f18f0070af0 2024-01-16T15:24:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.772+0000 7f18f8800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f00cf1a0 con 0x7f18f0160980 2024-01-16T15:24:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.772+0000 7f18f8800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f0161000 con 0x7f18f0161a10 2024-01-16T15:24:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.772+0000 7f18f5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18f0160980 0x7f18f006ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.773+0000 7f18f5e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18f0160980 0x7f18f006ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48746/0 (socket says 172.21.15.141:48746) 2024-01-16T15:24:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.773+0000 7f18f5e00640 1 -- 172.21.15.141:0/11502349 learned_addr learned my addr 172.21.15.141:0/11502349 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.773+0000 7f18f5400640 1 --2- 172.21.15.141:0/11502349 >> v2:172.21.15.188:3300/0 conn(0x7f18f0161a10 0x7f18f00ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.771 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.773+0000 7f18f5e00640 1 -- 172.21.15.141:0/11502349 >> v2:172.21.15.188:3300/0 conn(0x7f18f0161a10 msgr2=0x7f18f00ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:01.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.773+0000 7f18f5e00640 1 --2- 172.21.15.141:0/11502349 >> v2:172.21.15.188:3300/0 conn(0x7f18f0161a10 0x7f18f00ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.773+0000 7f18f5e00640 1 -- 172.21.15.141:0/11502349 >> v1:172.21.15.188:6789/0 conn(0x7f18f0070af0 legacy=0x7f18f006d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:01.772 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.773+0000 7f18f5e00640 1 -- 172.21.15.141:0/11502349 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f00ce920 con 0x7f18f0160980 2024-01-16T15:24:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.774+0000 7f18f5e00640 1 --2- 172.21.15.141:0/11502349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0160980 0x7f18f006ba30 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f18f00ce330 tx=0x7f18e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb4bee827f29812 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.774+0000 7f18f4a00640 1 -- 172.21.15.141:0/11502349 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e002ca80 con 0x7f18f0160980 2024-01-16T15:24:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f4a00640 1 -- 172.21.15.141:0/11502349 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e002cbe0 con 0x7f18f0160980 2024-01-16T15:24:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f4a00640 1 -- 172.21.15.141:0/11502349 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00343f0 con 0x7f18f0160980 2024-01-16T15:24:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 -- 172.21.15.141:0/11502349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0160980 msgr2=0x7f18f006ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.773 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 --2- 172.21.15.141:0/11502349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0160980 0x7f18f006ba30 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f18f00ce330 tx=0x7f18e002bee0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 -- 172.21.15.141:0/11502349 shutdown_connections 2024-01-16T15:24:01.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 --2- 172.21.15.141:0/11502349 >> v2:172.21.15.188:3300/0 conn(0x7f18f0161a10 0x7f18f00ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 --2- 172.21.15.141:0/11502349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0160980 0x7f18f006ba30 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 -- 172.21.15.141:0/11502349 >> 172.21.15.141:0/11502349 conn(0x7f18f00cf700 msgr2=0x7f18f00cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:01.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 -- 172.21.15.141:0/11502349 shutdown_connections 2024-01-16T15:24:01.774 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.775+0000 7f18f8800640 1 -- 172.21.15.141:0/11502349 wait complete. 2024-01-16T15:24:01.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.777+0000 7f18f8800640 1 Processor -- start 2024-01-16T15:24:01.775 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.777+0000 7f18f8800640 1 -- start start 2024-01-16T15:24:01.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.778+0000 7f18f8800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0070af0 0x7f18f0140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.776 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.778+0000 7f18f8800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18f0160980 0x7f18f0141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.778+0000 7f18f8800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18f0161a10 0x7f18f015e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.778+0000 7f18f8800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f0151530 con 0x7f18f0070af0 2024-01-16T15:24:01.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.778+0000 7f18f8800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f0158b30 con 0x7f18f0160980 2024-01-16T15:24:01.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.778+0000 7f18f8800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f0155910 con 0x7f18f0161a10 2024-01-16T15:24:01.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0070af0 0x7f18f0140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.777 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0070af0 0x7f18f0140850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48760/0 (socket says 172.21.15.141:48760) 2024-01-16T15:24:01.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 -- 172.21.15.141:0/3459454588 learned_addr learned my addr 172.21.15.141:0/3459454588 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:01.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5400640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18f0160980 0x7f18f0141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 -- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18f0161a10 msgr2=0x7f18f015e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18f0161a10 0x7f18f015e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 -- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18f0160980 msgr2=0x7f18f0141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18f0160980 0x7f18f0141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.778 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 -- 172.21.15.141:0/3459454588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0009b30 con 0x7f18f0070af0 2024-01-16T15:24:01.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5400640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18f0160980 0x7f18f0141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:01.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.779+0000 7f18f5e00640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0070af0 0x7f18f0140850 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000f80 tx=0x7f18e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.780+0000 7f18ee000640 1 -- 172.21.15.141:0/3459454588 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0009510 con 0x7f18f0070af0 2024-01-16T15:24:01.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.780+0000 7f18ee000640 1 -- 172.21.15.141:0/3459454588 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e003e450 con 0x7f18f0070af0 2024-01-16T15:24:01.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.780+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f0155bf0 con 0x7f18f0070af0 2024-01-16T15:24:01.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.780+0000 7f18ee000640 1 -- 172.21.15.141:0/3459454588 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e003e800 con 0x7f18f0070af0 2024-01-16T15:24:01.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.780+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f015e3a0 con 0x7f18f0070af0 2024-01-16T15:24:01.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.781+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f00675d0 con 0x7f18f0070af0 2024-01-16T15:24:01.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.782+0000 7f18ee000640 1 -- 172.21.15.141:0/3459454588 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f18e002cd50 con 0x7f18f0070af0 2024-01-16T15:24:01.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.783+0000 7f18ee000640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18c806c5e0 0x7f18c806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:01.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.783+0000 7f18ee000640 1 -- 172.21.15.141:0/3459454588 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f18e00ab710 con 0x7f18f0070af0 2024-01-16T15:24:01.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.784+0000 7f18f5400640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18c806c5e0 0x7f18c806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:01.782 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.785+0000 7f18f5400640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18c806c5e0 0x7f18c806eaa0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f18f0151bd0 tx=0x7f18d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:01.784 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.787+0000 7f18ee000640 1 -- 172.21.15.141:0/3459454588 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18e0077c90 con 0x7f18f0070af0 2024-01-16T15:24:01.975 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.978+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f18f0161e70 con 0x7f18c806c5e0 2024-01-16T15:24:01.976 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.980+0000 7f18ee000640 1 -- 172.21.15.141:0/3459454588 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f18d8009e10 con 0x7f18c806c5e0 2024-01-16T15:24:01.976 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 3 to deep-scrub 2024-01-16T15:24:01.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18c806c5e0 msgr2=0x7f18c806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18c806c5e0 0x7f18c806eaa0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f18f0151bd0 tx=0x7f18d8008040 comp rx=0 tx=0).stop 2024-01-16T15:24:01.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0070af0 msgr2=0x7f18f0140850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:01.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0070af0 0x7f18f0140850 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000f80 tx=0x7f18e0004a40 comp rx=0 tx=0).stop 2024-01-16T15:24:01.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 shutdown_connections 2024-01-16T15:24:01.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f18c806c5e0 0x7f18c806eaa0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.980 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f18f0161a10 0x7f18f015e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18f0160980 0x7f18f0141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 --2- 172.21.15.141:0/3459454588 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18f0070af0 0x7f18f0140850 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:01.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 >> 172.21.15.141:0/3459454588 conn(0x7f18f00cf700 msgr2=0x7f18f015b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:01.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 shutdown_connections 2024-01-16T15:24:01.981 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:01.982+0000 7f18f8800640 1 -- 172.21.15.141:0/3459454588 wait complete. 2024-01-16T15:24:01.998 INFO:tasks.ceph:Scrubbing osd.4 2024-01-16T15:24:01.998 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:24:02.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.216+0000 7fba75600640 1 Processor -- start 2024-01-16T15:24:02.213 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.216+0000 7fba75600640 1 -- start start 2024-01-16T15:24:02.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.217+0000 7fba75600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.217+0000 7fba75600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fba700717d0 0x7fba700cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.217+0000 7fba75600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba70070e50 con 0x7fba700cca40 2024-01-16T15:24:02.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.217+0000 7fba75600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba700723c0 con 0x7fba700717d0 2024-01-16T15:24:02.215 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.217+0000 7fba75600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba700cce70 con 0x7fba70143c40 2024-01-16T15:24:02.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.218+0000 7fba6fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.218+0000 7fba67200640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fba700717d0 0x7fba700cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.218+0000 7fba6fe00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3301/0 says I am v2:172.21.15.141:60594/0 (socket says 172.21.15.141:60594) 2024-01-16T15:24:02.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.218+0000 7fba6fe00640 1 -- 172.21.15.141:0/2390215936 learned_addr learned my addr 172.21.15.141:0/2390215936 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:02.216 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.219+0000 7fba6fe00640 1 -- 172.21.15.141:0/2390215936 >> v2:172.21.15.188:3300/0 conn(0x7fba700717d0 msgr2=0x7fba700cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:02.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.219+0000 7fba6fe00640 1 --2- 172.21.15.141:0/2390215936 >> v2:172.21.15.188:3300/0 conn(0x7fba700717d0 0x7fba700cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.219+0000 7fba6fe00640 1 -- 172.21.15.141:0/2390215936 >> v1:172.21.15.188:6789/0 conn(0x7fba700cca40 legacy=0x7fba7015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.219+0000 7fba6fe00640 1 -- 172.21.15.141:0/2390215936 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7006e170 con 0x7fba70143c40 2024-01-16T15:24:02.217 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.220+0000 7fba6fe00640 1 --2- 172.21.15.141:0/2390215936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70071290 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fba60009870 tx=0x7fba6002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e681e4181947f584 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.221+0000 7fba6f400640 1 -- 172.21.15.141:0/2390215936 <== mon.2 v2:172.21.15.188:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60031640 con 0x7fba70143c40 2024-01-16T15:24:02.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.221+0000 7fba6f400640 1 -- 172.21.15.141:0/2390215936 <== mon.2 v2:172.21.15.188:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba600317a0 con 0x7fba70143c40 2024-01-16T15:24:02.218 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.221+0000 7fba6f400640 1 -- 172.21.15.141:0/2390215936 <== mon.2 v2:172.21.15.188:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60031b30 con 0x7fba70143c40 2024-01-16T15:24:02.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.221+0000 7fba75600640 1 -- 172.21.15.141:0/2390215936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 msgr2=0x7fba70071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.221+0000 7fba75600640 1 --2- 172.21.15.141:0/2390215936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70071290 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fba60009870 tx=0x7fba6002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.222+0000 7fba75600640 1 -- 172.21.15.141:0/2390215936 shutdown_connections 2024-01-16T15:24:02.219 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.222+0000 7fba75600640 1 --2- 172.21.15.141:0/2390215936 >> v2:172.21.15.188:3300/0 conn(0x7fba700717d0 0x7fba700cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.222+0000 7fba75600640 1 --2- 172.21.15.141:0/2390215936 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70071290 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.222+0000 7fba75600640 1 -- 172.21.15.141:0/2390215936 >> 172.21.15.141:0/2390215936 conn(0x7fba7006a800 msgr2=0x7fba7006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:02.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.222+0000 7fba75600640 1 -- 172.21.15.141:0/2390215936 shutdown_connections 2024-01-16T15:24:02.220 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.222+0000 7fba75600640 1 -- 172.21.15.141:0/2390215936 wait complete. 2024-01-16T15:24:02.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.223+0000 7fba75600640 1 Processor -- start 2024-01-16T15:24:02.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.224+0000 7fba75600640 1 -- start start 2024-01-16T15:24:02.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba75600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba700717d0 0x7fba70158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba75600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba700cca40 0x7fba701428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.222 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba75600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba75600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba700aa730 con 0x7fba700cca40 2024-01-16T15:24:02.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba75600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7013c140 con 0x7fba700717d0 2024-01-16T15:24:02.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba75600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba700a9ab0 con 0x7fba70143c40 2024-01-16T15:24:02.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba700cca40 0x7fba701428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba700cca40 0x7fba701428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48772/0 (socket says 172.21.15.141:48772) 2024-01-16T15:24:02.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 -- 172.21.15.141:0/2512914398 learned_addr learned my addr 172.21.15.141:0/2512914398 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:02.223 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba6fe00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba700717d0 0x7fba70158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.224 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba74c00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 -- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 msgr2=0x7fba70142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fba70143c40 0x7fba70142e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 -- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba700717d0 msgr2=0x7fba70158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba700717d0 0x7fba70158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.225+0000 7fba67200640 1 -- 172.21.15.141:0/2512914398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba600094d0 con 0x7fba700cca40 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.226+0000 7fba67200640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba700cca40 0x7fba701428f0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fba5800efc0 tx=0x7fba5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.226+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58014480 con 0x7fba700cca40 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.226+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5800f040 con 0x7fba700cca40 2024-01-16T15:24:02.225 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.226+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58014800 con 0x7fba700cca40 2024-01-16T15:24:02.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.226+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba700a9d90 con 0x7fba700cca40 2024-01-16T15:24:02.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.226+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba700aa350 con 0x7fba700cca40 2024-01-16T15:24:02.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.227+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fba700ce280 con 0x7fba700cca40 2024-01-16T15:24:02.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.228+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fba58014960 con 0x7fba700cca40 2024-01-16T15:24:02.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.229+0000 7fba6cc00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba4806c5e0 0x7fba4806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.229+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba58089810 con 0x7fba700cca40 2024-01-16T15:24:02.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.229+0000 7fba6cc00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fba48074840 0x7fba48076c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.229+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 --> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba48077350 con 0x7fba48074840 2024-01-16T15:24:02.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.229+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba5808a0d0 con 0x7fba700cca40 2024-01-16T15:24:02.227 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.230+0000 7fba6fe00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba4806c5e0 0x7fba4806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.230+0000 7fba74c00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fba48074840 0x7fba48076c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.231+0000 7fba74c00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fba48074840 0x7fba48076c80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.231+0000 7fba6fe00640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba4806c5e0 0x7fba4806eaa0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fba60036000 tx=0x7fba6003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.231 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.233+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== osd.4 v2:172.21.15.188:6800/2387947334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fba48077350 con 0x7fba48074840 2024-01-16T15:24:02.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.248+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 --> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fba700abcf0 con 0x7fba48074840 2024-01-16T15:24:02.246 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.249+0000 7fba6cc00640 1 -- 172.21.15.141:0/2512914398 <== osd.4 v2:172.21.15.188:6800/2387947334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fba700abcf0 con 0x7fba48074840 2024-01-16T15:24:02.246 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:24:02.246 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:24:02.247 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:24:02.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.249+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fba48074840 msgr2=0x7fba48076c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.247 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.249+0000 7fba75600640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6800/2387947334,v1:172.21.15.188:6801/2387947334] conn(0x7fba48074840 0x7fba48076c80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.249+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba4806c5e0 msgr2=0x7fba4806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.249+0000 7fba75600640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fba4806c5e0 0x7fba4806eaa0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fba60036000 tx=0x7fba6003c040 comp rx=0 tx=0).stop 2024-01-16T15:24:02.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.249+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba700cca40 msgr2=0x7fba701428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.249+0000 7fba75600640 1 --2- 172.21.15.141:0/2512914398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba700cca40 0x7fba701428f0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fba5800efc0 tx=0x7fba5800c490 comp rx=0 tx=0).stop 2024-01-16T15:24:02.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.250+0000 7fba74c00640 1 -- 172.21.15.141:0/2512914398 reap_dead start 2024-01-16T15:24:02.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.250+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 shutdown_connections 2024-01-16T15:24:02.248 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.250+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 >> 172.21.15.141:0/2512914398 conn(0x7fba7006a800 msgr2=0x7fba70140190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:02.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.250+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 shutdown_connections 2024-01-16T15:24:02.249 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.250+0000 7fba75600640 1 -- 172.21.15.141:0/2512914398 wait complete. 2024-01-16T15:24:02.266 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-16T15:24:02.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.481+0000 7ff1e4e00640 1 Processor -- start 2024-01-16T15:24:02.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.482+0000 7ff1e4e00640 1 -- start start 2024-01-16T15:24:02.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.483+0000 7ff1e4e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff1e00b33f0 0x7ff1e00d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.483+0000 7ff1e4e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e00ce670 con 0x7ff1e015da70 2024-01-16T15:24:02.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.483+0000 7ff1e4e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e015e930 con 0x7ff1e00cfec0 2024-01-16T15:24:02.480 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.483+0000 7ff1e4e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e015e650 con 0x7ff1e00b33f0 2024-01-16T15:24:02.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.483+0000 7ff1dd400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff1e00b33f0 0x7ff1e00d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.481 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.483+0000 7ff1de800640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff1e015da70 0x7ff1e00ca3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54392/0 (socket says 172.21.15.141:54392) 2024-01-16T15:24:02.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.483+0000 7ff1de800640 1 -- 172.21.15.141:0/998627778 learned_addr learned my addr 172.21.15.141:0/998627778 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:24:02.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.484+0000 7ff1dd400640 1 -- 172.21.15.141:0/998627778 >> v1:172.21.15.188:6789/0 conn(0x7ff1e00cfec0 legacy=0x7ff1e00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:02.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.484+0000 7ff1dd400640 1 -- 172.21.15.141:0/998627778 >> v1:172.21.15.141:6789/0 conn(0x7ff1e015da70 legacy=0x7ff1e00ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.484+0000 7ff1dca00640 1 -- 172.21.15.141:0/998627778 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434455027 0 0) 0x7ff1e00ce670 con 0x7ff1e015da70 2024-01-16T15:24:02.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.484+0000 7ff1dd400640 1 -- 172.21.15.141:0/998627778 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e00d0f60 con 0x7ff1e00b33f0 2024-01-16T15:24:02.482 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.485+0000 7ff1dd400640 1 --2- 172.21.15.141:0/998627778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e00d1a40 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097a0 tx=0x7ff1c802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633d4b8a8d616393 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.485+0000 7ff1dca00640 1 -- 172.21.15.141:0/998627778 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8030610 con 0x7ff1e00b33f0 2024-01-16T15:24:02.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.485+0000 7ff1dca00640 1 -- 172.21.15.141:0/998627778 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c8030770 con 0x7ff1e00b33f0 2024-01-16T15:24:02.483 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.485+0000 7ff1dca00640 1 -- 172.21.15.141:0/998627778 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8030b00 con 0x7ff1e00b33f0 2024-01-16T15:24:02.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.486+0000 7ff1e4e00640 1 -- 172.21.15.141:0/998627778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 msgr2=0x7ff1e00d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.486+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/998627778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e00d1a40 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097a0 tx=0x7ff1c802b960 comp rx=0 tx=0).stop 2024-01-16T15:24:02.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.486+0000 7ff1e4e00640 1 -- 172.21.15.141:0/998627778 shutdown_connections 2024-01-16T15:24:02.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.486+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/998627778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e00d1a40 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.486+0000 7ff1e4e00640 1 -- 172.21.15.141:0/998627778 >> 172.21.15.141:0/998627778 conn(0x7ff1e0066690 msgr2=0x7ff1e006a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:02.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.486+0000 7ff1e4e00640 1 -- 172.21.15.141:0/998627778 shutdown_connections 2024-01-16T15:24:02.484 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.486+0000 7ff1e4e00640 1 -- 172.21.15.141:0/998627778 wait complete. 2024-01-16T15:24:02.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.488+0000 7ff1e4e00640 1 Processor -- start 2024-01-16T15:24:02.485 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.488+0000 7ff1e4e00640 1 -- start start 2024-01-16T15:24:02.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1e4e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e01537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1e4e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1e00cfec0 0x7ff1e014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1e4e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1e015da70 0x7ff1e0151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1e4e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0153d30 con 0x7ff1e00b33f0 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1e4e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0152750 con 0x7ff1e015da70 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1e4e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0152a30 con 0x7ff1e00cfec0 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1dde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e01537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1dde00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e01537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48788/0 (socket says 172.21.15.141:48788) 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.489+0000 7ff1dde00640 1 -- 172.21.15.141:0/891137398 learned_addr learned my addr 172.21.15.141:0/891137398 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1dd400640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1e00cfec0 0x7ff1e014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1de800640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1e015da70 0x7ff1e0151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.488 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1dde00640 1 -- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1e00cfec0 msgr2=0x7ff1e014d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1dde00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1e00cfec0 0x7ff1e014d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1dde00640 1 -- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1e015da70 msgr2=0x7ff1e0151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1dde00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1e015da70 0x7ff1e0151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1dde00640 1 -- 172.21.15.141:0/891137398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8009400 con 0x7ff1e00b33f0 2024-01-16T15:24:02.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1de800640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1e015da70 0x7ff1e0151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:02.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1dde00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e01537f0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000c870 tx=0x7ff1d000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1d6000640 1 -- 172.21.15.141:0/891137398 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0003b70 con 0x7ff1e00b33f0 2024-01-16T15:24:02.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1d6000640 1 -- 172.21.15.141:0/891137398 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d0003cf0 con 0x7ff1e00b33f0 2024-01-16T15:24:02.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e0152d10 con 0x7ff1e00b33f0 2024-01-16T15:24:02.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.490+0000 7ff1d6000640 1 -- 172.21.15.141:0/891137398 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000aa90 con 0x7ff1e00b33f0 2024-01-16T15:24:02.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.491+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e01532d0 con 0x7ff1e00b33f0 2024-01-16T15:24:02.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.492+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e00b6fa0 con 0x7ff1e00b33f0 2024-01-16T15:24:02.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.493+0000 7ff1d6000640 1 -- 172.21.15.141:0/891137398 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ff1d000abf0 con 0x7ff1e00b33f0 2024-01-16T15:24:02.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.494+0000 7ff1d6000640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1b806c5e0 0x7ff1b806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.494+0000 7ff1d6000640 1 -- 172.21.15.141:0/891137398 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff1d0088ab0 con 0x7ff1e00b33f0 2024-01-16T15:24:02.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.494+0000 7ff1dd400640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1b806c5e0 0x7ff1b806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.496+0000 7ff1dd400640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1b806c5e0 0x7ff1b806eaa0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8036000 tx=0x7ff1c803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.498+0000 7ff1d6000640 1 -- 172.21.15.141:0/891137398 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1d0054fd0 con 0x7ff1e00b33f0 2024-01-16T15:24:02.687 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.690+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1e00d1520 con 0x7ff1b806c5e0 2024-01-16T15:24:02.689 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.692+0000 7ff1d6000640 1 -- 172.21.15.141:0/891137398 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff1c8030960 con 0x7ff1b806c5e0 2024-01-16T15:24:02.689 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 4 to deep-scrub 2024-01-16T15:24:02.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1b806c5e0 msgr2=0x7ff1b806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.691 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1b806c5e0 0x7ff1b806eaa0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8036000 tx=0x7ff1c803b040 comp rx=0 tx=0).stop 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 msgr2=0x7ff1e01537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e01537f0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000c870 tx=0x7ff1d000cd30 comp rx=0 tx=0).stop 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 shutdown_connections 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff1b806c5e0 0x7ff1b806eaa0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1e015da70 0x7ff1e0151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff1e00cfec0 0x7ff1e014d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 --2- 172.21.15.141:0/891137398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1e00b33f0 0x7ff1e01537f0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.692 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 >> 172.21.15.141:0/891137398 conn(0x7ff1e0066690 msgr2=0x7ff1e006a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:02.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 shutdown_connections 2024-01-16T15:24:02.693 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.694+0000 7ff1e4e00640 1 -- 172.21.15.141:0/891137398 wait complete. 2024-01-16T15:24:02.710 INFO:tasks.ceph:Scrubbing osd.5 2024-01-16T15:24:02.710 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:24:02.928 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.931+0000 7fd2a9600640 1 Processor -- start 2024-01-16T15:24:02.929 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.931+0000 7fd2a9600640 1 -- start start 2024-01-16T15:24:02.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.933+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 0x7fd2a406ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.933+0000 7fd2a9600640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fd2a4070af0 0x7fd2a4070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.933+0000 7fd2a9600640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a4161580 con 0x7fd2a4161a10 2024-01-16T15:24:02.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.933+0000 7fd2a9600640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a40cf1a0 con 0x7fd2a4070af0 2024-01-16T15:24:02.930 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.933+0000 7fd2a9600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40714c0 con 0x7fd2a4160980 2024-01-16T15:24:02.932 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.933+0000 7fd29be00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd2a4161a10 0x7fd2a40ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54394/0 (socket says 172.21.15.141:54394) 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.933+0000 7fd29be00640 1 -- 172.21.15.141:0/4219177142 learned_addr learned my addr 172.21.15.141:0/4219177142 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a3400640 1 --2- 172.21.15.141:0/4219177142 >> v2:172.21.15.188:3300/0 conn(0x7fd2a4070af0 0x7fd2a4070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2a00640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 0x7fd2a406ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69832784 0 0) 0x7fd2a4161580 con 0x7fd2a4161a10 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd284003520 con 0x7fd2a4161a10 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 953540694 0 0) 0x7fd284003520 con 0x7fd2a4161a10 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 msgr2=0x7fd2a406ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:02.933 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 0x7fd2a406ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 >> v2:172.21.15.188:3300/0 conn(0x7fd2a4070af0 msgr2=0x7fd2a4070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 --2- 172.21.15.141:0/4219177142 >> v2:172.21.15.188:3300/0 conn(0x7fd2a4070af0 0x7fd2a4070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a4160d60 con 0x7fd2a4161a10 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fd28c001670 con 0x7fd2a4161a10 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 >> v1:172.21.15.141:6789/0 conn(0x7fd2a4161a10 legacy=0x7fd2a40ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 0x7fd284003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284003f40 0x7fd2840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd284004d00 0x7fd284005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.934 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40714c0 con 0x7fd284004d00 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28400b9b0 con 0x7fd284003f40 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.934+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28400bb70 con 0x7fd2a4160980 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd2a3400640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd284004d00 0x7fd284005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd2a2a00640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 0x7fd284003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd29be00640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284003f40 0x7fd2840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd2a3400640 1 -- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 msgr2=0x7fd284003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd2a3400640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a4160980 0x7fd284003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd2a3400640 1 -- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284003f40 msgr2=0x7fd2840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:02.935 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd2a3400640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284003f40 0x7fd2840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.936 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.935+0000 7fd2a3400640 1 -- 172.21.15.141:0/4219177142 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284003520 con 0x7fd284004d00 2024-01-16T15:24:02.938 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.936+0000 7fd2a3400640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd284004d00 0x7fd284005180 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fd29c007360 tx=0x7fd2840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234c84ecdc24195e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.936+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c02ccf0 con 0x7fd284004d00 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.936+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c02ce50 con 0x7fd284004d00 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.936+0000 7fd2a2000640 1 -- 172.21.15.141:0/4219177142 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c034660 con 0x7fd284004d00 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.937+0000 7fd2a9600640 1 -- 172.21.15.141:0/4219177142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd284004d00 msgr2=0x7fd284005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.937+0000 7fd2a9600640 1 --2- 172.21.15.141:0/4219177142 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd284004d00 0x7fd284005180 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fd29c007360 tx=0x7fd2840057f0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.937+0000 7fd2a3400640 1 -- 172.21.15.141:0/4219177142 reap_dead start 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.937+0000 7fd2a9600640 1 -- 172.21.15.141:0/4219177142 shutdown_connections 2024-01-16T15:24:02.939 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.937+0000 7fd2a9600640 1 -- 172.21.15.141:0/4219177142 >> 172.21.15.141:0/4219177142 conn(0x7fd2a40cf700 msgr2=0x7fd2a40cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.937+0000 7fd2a9600640 1 -- 172.21.15.141:0/4219177142 shutdown_connections 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.937+0000 7fd2a9600640 1 -- 172.21.15.141:0/4219177142 wait complete. 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.939+0000 7fd2a9600640 1 Processor -- start 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.939+0000 7fd2a9600640 1 -- start start 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284004d00 0x7fd2a4155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a41f65f0 0x7fd2a41440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2a414fa90 0x7fd2a414ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a9600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a4156af0 con 0x7fd2a414fa90 2024-01-16T15:24:02.940 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a9600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a4150450 con 0x7fd284004d00 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a9600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41f6dd0 con 0x7fd2a41f65f0 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2a414fa90 0x7fd2a414ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a3400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2a414fa90 0x7fd2a414ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:48798/0 (socket says 172.21.15.141:48798) 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.940+0000 7fd2a3400640 1 -- 172.21.15.141:0/3198874899 learned_addr learned my addr 172.21.15.141:0/3198874899 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd29be00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a41f65f0 0x7fd2a41440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a2a00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284004d00 0x7fd2a4155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a3400640 1 -- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a41f65f0 msgr2=0x7fd2a41440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a3400640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a41f65f0 0x7fd2a41440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.941 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a3400640 1 -- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284004d00 msgr2=0x7fd2a4155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a3400640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd284004d00 0x7fd2a4155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a3400640 1 -- 172.21.15.141:0/3198874899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a41f70b0 con 0x7fd2a414fa90 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd29be00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fd2a41f65f0 0x7fd2a41440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a3400640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2a414fa90 0x7fd2a414ff10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fd29c02dfd0 tx=0x7fd29c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c034ea0 con 0x7fd2a414fa90 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c003b90 con 0x7fd2a414fa90 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c004020 con 0x7fd2a414fa90 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.941+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a41f7390 con 0x7fd2a414fa90 2024-01-16T15:24:02.942 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.942+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a4144600 con 0x7fd2a414fa90 2024-01-16T15:24:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.942+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2a40ceec0 con 0x7fd2a414fa90 2024-01-16T15:24:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.943+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd29c03b040 con 0x7fd2a414fa90 2024-01-16T15:24:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.944+0000 7fd29aa00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd27c06c5e0 0x7fd27c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.944+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd29c0aa120 con 0x7fd2a414fa90 2024-01-16T15:24:02.944 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.945+0000 7fd29aa00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7fd27c074840 0x7fd27c076c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.945+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 --> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd27c077350 con 0x7fd27c074840 2024-01-16T15:24:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.945+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd29c0aad00 con 0x7fd2a414fa90 2024-01-16T15:24:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.945+0000 7fd2a2a00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd27c06c5e0 0x7fd27c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.945+0000 7fd29be00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7fd27c074840 0x7fd27c076c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.946+0000 7fd2a2a00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd27c06c5e0 0x7fd27c06eaa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd294009950 tx=0x7fd294008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.945 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.947+0000 7fd29be00640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7fd27c074840 0x7fd27c076c80 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:02.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.949+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== osd.5 v2:172.21.15.188:6812/917917352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd27c077350 con 0x7fd27c074840 2024-01-16T15:24:02.961 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.964+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 --> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd2a40cebe0 con 0x7fd27c074840 2024-01-16T15:24:02.962 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd29aa00640 1 -- 172.21.15.141:0/3198874899 <== osd.5 v2:172.21.15.188:6812/917917352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd28c001d10 con 0x7fd27c074840 2024-01-16T15:24:02.963 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:24:02.963 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:24:02.963 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:24:02.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7fd27c074840 msgr2=0x7fd27c076c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6812/917917352,v1:172.21.15.188:6814/917917352] conn(0x7fd27c074840 0x7fd27c076c80 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd27c06c5e0 msgr2=0x7fd27c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.964 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fd27c06c5e0 0x7fd27c06eaa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd294009950 tx=0x7fd294008040 comp rx=0 tx=0).stop 2024-01-16T15:24:02.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2a414fa90 msgr2=0x7fd2a414ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:02.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 --2- 172.21.15.141:0/3198874899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2a414fa90 0x7fd2a414ff10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fd29c02dfd0 tx=0x7fd29c004ab0 comp rx=0 tx=0).stop 2024-01-16T15:24:02.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a3400640 1 -- 172.21.15.141:0/3198874899 reap_dead start 2024-01-16T15:24:02.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 shutdown_connections 2024-01-16T15:24:02.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 >> 172.21.15.141:0/3198874899 conn(0x7fd2a40cf700 msgr2=0x7fd2a40b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:02.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 shutdown_connections 2024-01-16T15:24:02.965 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:02.966+0000 7fd2a9600640 1 -- 172.21.15.141:0/3198874899 wait complete. 2024-01-16T15:24:02.980 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-16T15:24:03.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.189+0000 7fe302800640 1 Processor -- start 2024-01-16T15:24:03.187 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.190+0000 7fe302800640 1 -- start start 2024-01-16T15:24:03.188 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.191+0000 7fe302800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2fc13e360 0x7fe2fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.191+0000 7fe302800640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7fe2fc15f9c0 0x7fe2fc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.191+0000 7fe302800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc15c5f0 con 0x7fe2fc13df80 2024-01-16T15:24:03.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.191+0000 7fe302800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc06c470 con 0x7fe2fc13e360 2024-01-16T15:24:03.189 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.191+0000 7fe302800640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc06c750 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.192+0000 7fe2faa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2fc13e360 0x7fe2fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.192+0000 7fe2faa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe2fc13e360 0x7fe2fc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32976/0 (socket says 172.21.15.141:32976) 2024-01-16T15:24:03.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.192+0000 7fe2faa00640 1 -- 172.21.15.141:0/3050480568 learned_addr learned my addr 172.21.15.141:0/3050480568 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:03.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.192+0000 7fe2faa00640 1 -- 172.21.15.141:0/3050480568 >> v2:172.21.15.188:3300/0 conn(0x7fe2fc15f9c0 msgr2=0x7fe2fc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:03.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.192+0000 7fe2faa00640 1 --2- 172.21.15.141:0/3050480568 >> v2:172.21.15.188:3300/0 conn(0x7fe2fc15f9c0 0x7fe2fc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.192+0000 7fe2faa00640 1 -- 172.21.15.141:0/3050480568 >> v1:172.21.15.188:6789/0 conn(0x7fe2fc13df80 legacy=0x7fe2fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:03.190 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.192+0000 7fe2faa00640 1 -- 172.21.15.141:0/3050480568 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc06ca30 con 0x7fe2fc13e360 2024-01-16T15:24:03.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.193+0000 7fe2faa00640 1 --2- 172.21.15.141:0/3050480568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc13e360 0x7fe2fc0cf290 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc06ded0 tx=0x7fe2e402b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a07f5979925fc3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.194+0000 7fe2fa000640 1 -- 172.21.15.141:0/3050480568 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4031640 con 0x7fe2fc13e360 2024-01-16T15:24:03.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.194+0000 7fe2fa000640 1 -- 172.21.15.141:0/3050480568 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e40317a0 con 0x7fe2fc13e360 2024-01-16T15:24:03.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.194+0000 7fe2fa000640 1 -- 172.21.15.141:0/3050480568 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4031b30 con 0x7fe2fc13e360 2024-01-16T15:24:03.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.194+0000 7fe302800640 1 -- 172.21.15.141:0/3050480568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc13e360 msgr2=0x7fe2fc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.192 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.194+0000 7fe302800640 1 --2- 172.21.15.141:0/3050480568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc13e360 0x7fe2fc0cf290 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc06ded0 tx=0x7fe2e402b8e0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.195+0000 7fe302800640 1 -- 172.21.15.141:0/3050480568 shutdown_connections 2024-01-16T15:24:03.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.195+0000 7fe302800640 1 --2- 172.21.15.141:0/3050480568 >> v2:172.21.15.188:3300/0 conn(0x7fe2fc15f9c0 0x7fe2fc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.195+0000 7fe302800640 1 --2- 172.21.15.141:0/3050480568 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc13e360 0x7fe2fc0cf290 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.195+0000 7fe302800640 1 -- 172.21.15.141:0/3050480568 >> 172.21.15.141:0/3050480568 conn(0x7fe2fc06a800 msgr2=0x7fe2fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:03.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.195+0000 7fe302800640 1 -- 172.21.15.141:0/3050480568 shutdown_connections 2024-01-16T15:24:03.193 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.195+0000 7fe302800640 1 -- 172.21.15.141:0/3050480568 wait complete. 2024-01-16T15:24:03.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.196+0000 7fe302800640 1 Processor -- start 2024-01-16T15:24:03.194 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.197+0000 7fe302800640 1 -- start start 2024-01-16T15:24:03.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe302800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2fc13df80 0x7fe2fc14b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.196 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe302800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2fc13e360 0x7fe2fc0a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe302800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc15f9c0 0x7fe2fc15a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe302800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc14b570 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe302800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0710d0 con 0x7fe2fc13df80 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe302800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0713b0 con 0x7fe2fc13e360 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc15f9c0 0x7fe2fc15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc15f9c0 0x7fe2fc15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32980/0 (socket says 172.21.15.141:32980) 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 -- 172.21.15.141:0/2360617395 learned_addr learned my addr 172.21.15.141:0/2360617395 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fb400640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2fc13df80 0x7fe2fc14b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.197 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2faa00640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2fc13e360 0x7fe2fc0a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 -- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2fc13e360 msgr2=0x7fe2fc0a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2fc13e360 0x7fe2fc0a8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 -- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2fc13df80 msgr2=0x7fe2fc14b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2fc13df80 0x7fe2fc14b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fbe00640 1 -- 172.21.15.141:0/2360617395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e40094d0 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.198+0000 7fe2fb400640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2fc13df80 0x7fe2fc14b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.199+0000 7fe2fbe00640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc15f9c0 0x7fe2fc15a180 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400b9e0 tx=0x7fe2f400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.199+0000 7fe2f3400640 1 -- 172.21.15.141:0/2360617395 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400cb20 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.198 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.199+0000 7fe2f3400640 1 -- 172.21.15.141:0/2360617395 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f400cc80 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.199+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc071690 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.199+0000 7fe2f3400640 1 -- 172.21.15.141:0/2360617395 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4014680 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.199+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc071c50 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.199 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.200+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc071e40 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.201+0000 7fe2f3400640 1 -- 172.21.15.141:0/2360617395 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fe2f401d3b0 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.202+0000 7fe2f3400640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2d406c5e0 0x7fe2d406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.202+0000 7fe2f3400640 1 -- 172.21.15.141:0/2360617395 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe2f4088b60 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.200 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.203+0000 7fe2fb400640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2d406c5e0 0x7fe2d406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.201 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.204+0000 7fe2fb400640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2d406c5e0 0x7fe2d406eaa0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00a830 tx=0x7fe2ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.204 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.207+0000 7fe2f3400640 1 -- 172.21.15.141:0/2360617395 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2f4055080 con 0x7fe2fc15f9c0 2024-01-16T15:24:03.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.407+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2fc071520 con 0x7fe2d406c5e0 2024-01-16T15:24:03.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.408+0000 7fe2f3400640 1 -- 172.21.15.141:0/2360617395 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe2fc071520 con 0x7fe2d406c5e0 2024-01-16T15:24:03.406 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 5 to deep-scrub 2024-01-16T15:24:03.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.410+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2d406c5e0 msgr2=0x7fe2d406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.410+0000 7fe302800640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2d406c5e0 0x7fe2d406eaa0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00a830 tx=0x7fe2ec008040 comp rx=0 tx=0).stop 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.410+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc15f9c0 msgr2=0x7fe2fc15a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.410+0000 7fe302800640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc15f9c0 0x7fe2fc15a180 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400b9e0 tx=0x7fe2f400bea0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 shutdown_connections 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fe2d406c5e0 0x7fe2d406eaa0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2fc15f9c0 0x7fe2fc15a180 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fe2fc13e360 0x7fe2fc0a8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 --2- 172.21.15.141:0/2360617395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2fc13df80 0x7fe2fc14b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 >> 172.21.15.141:0/2360617395 conn(0x7fe2fc06a800 msgr2=0x7fe2fc0cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:03.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 shutdown_connections 2024-01-16T15:24:03.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.411+0000 7fe302800640 1 -- 172.21.15.141:0/2360617395 wait complete. 2024-01-16T15:24:03.425 INFO:tasks.ceph:Scrubbing osd.6 2024-01-16T15:24:03.426 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:24:03.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.632+0000 7fcacfa00640 1 Processor -- start 2024-01-16T15:24:03.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.632+0000 7fcacfa00640 1 -- start start 2024-01-16T15:24:03.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.633+0000 7fcacfa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcac80cf9a0 0x7fcac815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.633+0000 7fcacfa00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac80b1ee0 con 0x7fcac8143ae0 2024-01-16T15:24:03.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.633+0000 7fcacfa00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac80b2780 con 0x7fcac815e1e0 2024-01-16T15:24:03.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.633+0000 7fcacfa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac815e990 con 0x7fcac80cf9a0 2024-01-16T15:24:03.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.634+0000 7fcac7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcac80cf9a0 0x7fcac815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.634+0000 7fcac7e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcac80cf9a0 0x7fcac815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32986/0 (socket says 172.21.15.141:32986) 2024-01-16T15:24:03.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.634+0000 7fcac7e00640 1 -- 172.21.15.141:0/148815268 learned_addr learned my addr 172.21.15.141:0/148815268 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:03.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.634+0000 7fcacd000640 1 --1- 172.21.15.141:0/148815268 >> v1:172.21.15.141:6789/0 conn(0x7fcac8143ae0 0x7fcac80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60334/0 (socket says 172.21.15.141:60334) 2024-01-16T15:24:03.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.634+0000 7fcac7e00640 1 -- 172.21.15.141:0/148815268 >> v1:172.21.15.188:6789/0 conn(0x7fcac815e1e0 legacy=0x7fcac80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:03.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.634+0000 7fcac7e00640 1 -- 172.21.15.141:0/148815268 >> v1:172.21.15.141:6789/0 conn(0x7fcac8143ae0 legacy=0x7fcac80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.634+0000 7fcac7e00640 1 -- 172.21.15.141:0/148815268 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac80b2130 con 0x7fcac80cf9a0 2024-01-16T15:24:03.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.635+0000 7fcac7e00640 1 --2- 172.21.15.141:0/148815268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac80cf9a0 0x7fcac815dca0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009870 tx=0x7fcab002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e25ec7a9f015ea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcac7400640 1 -- 172.21.15.141:0/148815268 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0030850 con 0x7fcac80cf9a0 2024-01-16T15:24:03.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcac7400640 1 -- 172.21.15.141:0/148815268 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab00309b0 con 0x7fcac80cf9a0 2024-01-16T15:24:03.633 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcac7400640 1 -- 172.21.15.141:0/148815268 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0030d40 con 0x7fcac80cf9a0 2024-01-16T15:24:03.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcacfa00640 1 -- 172.21.15.141:0/148815268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac80cf9a0 msgr2=0x7fcac815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcacfa00640 1 --2- 172.21.15.141:0/148815268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac80cf9a0 0x7fcac815dca0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009870 tx=0x7fcab002b990 comp rx=0 tx=0).stop 2024-01-16T15:24:03.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcacfa00640 1 -- 172.21.15.141:0/148815268 shutdown_connections 2024-01-16T15:24:03.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcacfa00640 1 --2- 172.21.15.141:0/148815268 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac80cf9a0 0x7fcac815dca0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcacfa00640 1 -- 172.21.15.141:0/148815268 >> 172.21.15.141:0/148815268 conn(0x7fcac806a800 msgr2=0x7fcac806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:03.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.636+0000 7fcacfa00640 1 -- 172.21.15.141:0/148815268 shutdown_connections 2024-01-16T15:24:03.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.637+0000 7fcacfa00640 1 -- 172.21.15.141:0/148815268 wait complete. 2024-01-16T15:24:03.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.638+0000 7fcacfa00640 1 Processor -- start 2024-01-16T15:24:03.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.638+0000 7fcacfa00640 1 -- start start 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.639+0000 7fcacfa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcac80cf9a0 0x7fcac815ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.639+0000 7fcacfa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac8143ae0 0x7fcac80c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.639+0000 7fcacfa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcac815e1e0 0x7fcac8159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.639+0000 7fcacfa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80c5ad0 con 0x7fcac8143ae0 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.639+0000 7fcacfa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80c4ec0 con 0x7fcac815e1e0 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.639+0000 7fcacfa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80c51a0 con 0x7fcac80cf9a0 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.639+0000 7fcac7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac8143ae0 0x7fcac80c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac8143ae0 0x7fcac80c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32994/0 (socket says 172.21.15.141:32994) 2024-01-16T15:24:03.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 -- 172.21.15.141:0/2176409350 learned_addr learned my addr 172.21.15.141:0/2176409350 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:03.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcacd000640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcac80cf9a0 0x7fcac815ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 -- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcac80cf9a0 msgr2=0x7fcac815ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcac80cf9a0 0x7fcac815ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcacda00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcac815e1e0 0x7fcac8159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 -- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcac815e1e0 msgr2=0x7fcac8159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcac815e1e0 0x7fcac8159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 -- 172.21.15.141:0/2176409350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab00094d0 con 0x7fcac8143ae0 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcacd000640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fcac80cf9a0 0x7fcac815ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcacda00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcac815e1e0 0x7fcac8159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.640+0000 7fcac7e00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac8143ae0 0x7fcac80c6920 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.641+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0030820 con 0x7fcac8143ae0 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.641+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab0043400 con 0x7fcac8143ae0 2024-01-16T15:24:03.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.641+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab00437d0 con 0x7fcac8143ae0 2024-01-16T15:24:03.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.641+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac80c5480 con 0x7fcac8143ae0 2024-01-16T15:24:03.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.641+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac815d230 con 0x7fcac8143ae0 2024-01-16T15:24:03.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.642+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcac815e670 con 0x7fcac8143ae0 2024-01-16T15:24:03.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.643+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fcab002e070 con 0x7fcac8143ae0 2024-01-16T15:24:03.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.644+0000 7fcac4c00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcaa006c5e0 0x7fcaa006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.644+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcab003e070 con 0x7fcac8143ae0 2024-01-16T15:24:03.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.644+0000 7fcac4c00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7fcaa0074840 0x7fcaa0076c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.644+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 --> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcaa0077350 con 0x7fcaa0074840 2024-01-16T15:24:03.642 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.644+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcab00abf70 con 0x7fcac8143ae0 2024-01-16T15:24:03.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.644+0000 7fcacd000640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcaa006c5e0 0x7fcaa006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.645+0000 7fcacda00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7fcaa0074840 0x7fcaa0076c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.645+0000 7fcacd000640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcaa006c5e0 0x7fcaa006eaa0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fcab8000f60 tx=0x7fcab8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.646+0000 7fcacda00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7fcaa0074840 0x7fcaa0076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.648+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== osd.6 v2:172.21.15.188:6824/2194964726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcaa0077350 con 0x7fcaa0074840 2024-01-16T15:24:03.659 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.662+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 --> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcac815d3c0 con 0x7fcaa0074840 2024-01-16T15:24:03.660 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcac4c00640 1 -- 172.21.15.141:0/2176409350 <== osd.6 v2:172.21.15.188:6824/2194964726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcac815d3c0 con 0x7fcaa0074840 2024-01-16T15:24:03.660 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:24:03.660 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:24:03.661 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:24:03.661 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7fcaa0074840 msgr2=0x7fcaa0076c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6824/2194964726,v1:172.21.15.188:6825/2194964726] conn(0x7fcaa0074840 0x7fcaa0076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcaa006c5e0 msgr2=0x7fcaa006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fcaa006c5e0 0x7fcaa006eaa0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fcab8000f60 tx=0x7fcab8009040 comp rx=0 tx=0).stop 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac8143ae0 msgr2=0x7fcac80c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 --2- 172.21.15.141:0/2176409350 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac8143ae0 0x7fcac80c6920 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab0004730 comp rx=0 tx=0).stop 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacda00640 1 -- 172.21.15.141:0/2176409350 reap_dead start 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 shutdown_connections 2024-01-16T15:24:03.662 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 >> 172.21.15.141:0/2176409350 conn(0x7fcac806a800 msgr2=0x7fcac80cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:03.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 shutdown_connections 2024-01-16T15:24:03.663 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.664+0000 7fcacfa00640 1 -- 172.21.15.141:0/2176409350 wait complete. 2024-01-16T15:24:03.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-16T15:24:03.893 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.896+0000 7f032e600640 1 Processor -- start 2024-01-16T15:24:03.894 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.896+0000 7f032e600640 1 -- start start 2024-01-16T15:24:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.897+0000 7f032e600640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0328143ae0 0x7f03280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.897+0000 7f032e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.897+0000 7f032e600640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280cfc30 con 0x7f032815e1e0 2024-01-16T15:24:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.897+0000 7f032e600640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03280b1ee0 con 0x7f0328143ae0 2024-01-16T15:24:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.897+0000 7f032e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280b2780 con 0x7f03280cf7d0 2024-01-16T15:24:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0327400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0328143ae0 0x7f03280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0327400640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0328143ae0 0x7f03280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:32996/0 (socket says 172.21.15.141:32996) 2024-01-16T15:24:03.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0327400640 1 -- 172.21.15.141:0/1617469741 learned_addr learned my addr 172.21.15.141:0/1617469741 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0326a00640 1 --2- 172.21.15.141:0/1617469741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0327400640 1 -- 172.21.15.141:0/1617469741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 msgr2=0x7f032815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0327400640 1 --2- 172.21.15.141:0/1617469741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0327400640 1 -- 172.21.15.141:0/1617469741 >> v1:172.21.15.188:6789/0 conn(0x7f032815e1e0 legacy=0x7f032815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0327400640 1 -- 172.21.15.141:0/1617469741 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032815e990 con 0x7f0328143ae0 2024-01-16T15:24:03.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.898+0000 7f0326a00640 1 --2- 172.21.15.141:0/1617469741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:24:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.899+0000 7f0327400640 1 --2- 172.21.15.141:0/1617469741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cf290 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f0318009870 tx=0x7f031802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a2c981102d93d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f0326000640 1 -- 172.21.15.141:0/1617469741 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318031640 con 0x7f0328143ae0 2024-01-16T15:24:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f0326000640 1 -- 172.21.15.141:0/1617469741 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03180317a0 con 0x7f0328143ae0 2024-01-16T15:24:03.898 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f0326000640 1 -- 172.21.15.141:0/1617469741 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318031b30 con 0x7f0328143ae0 2024-01-16T15:24:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f032e600640 1 -- 172.21.15.141:0/1617469741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 msgr2=0x7f03280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f032e600640 1 --2- 172.21.15.141:0/1617469741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cf290 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f0318009870 tx=0x7f031802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f032e600640 1 -- 172.21.15.141:0/1617469741 shutdown_connections 2024-01-16T15:24:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f032e600640 1 --2- 172.21.15.141:0/1617469741 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f032e600640 1 --2- 172.21.15.141:0/1617469741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cf290 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.900+0000 7f032e600640 1 -- 172.21.15.141:0/1617469741 >> 172.21.15.141:0/1617469741 conn(0x7f032806a800 msgr2=0x7f032806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:03.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.901+0000 7f032e600640 1 -- 172.21.15.141:0/1617469741 shutdown_connections 2024-01-16T15:24:03.900 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.901+0000 7f032e600640 1 -- 172.21.15.141:0/1617469741 wait complete. 2024-01-16T15:24:03.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.902+0000 7f032e600640 1 Processor -- start 2024-01-16T15:24:03.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.903+0000 7f032e600640 1 -- start start 2024-01-16T15:24:03.901 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.903+0000 7f032e600640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032812c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.903+0000 7f032e600640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f032e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f032815e1e0 0x7f032815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f032e600640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280cdaa0 con 0x7f0328143ae0 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f032e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280ccb50 con 0x7f032815e1e0 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f032e600640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280cce20 con 0x7f03280cf7d0 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33012/0 (socket says 172.21.15.141:33012) 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 -- 172.21.15.141:0/548445884 learned_addr learned my addr 172.21.15.141:0/548445884 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:03.902 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0327400640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032812c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 -- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 msgr2=0x7f032812c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032812c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 -- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f032815e1e0 msgr2=0x7f032815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f032815e1e0 0x7f032815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.904+0000 7f0326a00640 1 -- 172.21.15.141:0/548445884 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180094d0 con 0x7f0328143ae0 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.905+0000 7f0326a00640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cc610 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f031000c7f0 tx=0x7f031000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.905+0000 7f031f400640 1 -- 172.21.15.141:0/548445884 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310007a30 con 0x7f0328143ae0 2024-01-16T15:24:03.904 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.905+0000 7f031f400640 1 -- 172.21.15.141:0/548445884 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0310007b90 con 0x7f0328143ae0 2024-01-16T15:24:03.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.905+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03280cd100 con 0x7f0328143ae0 2024-01-16T15:24:03.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.905+0000 7f031f400640 1 -- 172.21.15.141:0/548445884 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031000ce40 con 0x7f0328143ae0 2024-01-16T15:24:03.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.905+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03280cd6c0 con 0x7f0328143ae0 2024-01-16T15:24:03.905 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.906+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03280b2bf0 con 0x7f0328143ae0 2024-01-16T15:24:03.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.907+0000 7f031f400640 1 -- 172.21.15.141:0/548445884 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f03100113f0 con 0x7f0328143ae0 2024-01-16T15:24:03.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.908+0000 7f031f400640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f030006c5e0 0x7f030006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:03.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.909+0000 7f031f400640 1 -- 172.21.15.141:0/548445884 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0310088d70 con 0x7f0328143ae0 2024-01-16T15:24:03.906 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.909+0000 7f0327400640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f030006c5e0 0x7f030006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:03.908 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.911+0000 7f0327400640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f030006c5e0 0x7f030006eaa0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0318036000 tx=0x7f031803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:03.911 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:03.914+0000 7f031f400640 1 -- 172.21.15.141:0/548445884 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03280b2bf0 con 0x7f0328143ae0 2024-01-16T15:24:04.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.133+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f032815e640 con 0x7f030006c5e0 2024-01-16T15:24:04.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.134+0000 7f031f400640 1 -- 172.21.15.141:0/548445884 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f03180315e0 con 0x7f030006c5e0 2024-01-16T15:24:04.132 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 6 to deep-scrub 2024-01-16T15:24:04.134 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f030006c5e0 msgr2=0x7f030006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f030006c5e0 0x7f030006eaa0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0318036000 tx=0x7f031803c040 comp rx=0 tx=0).stop 2024-01-16T15:24:04.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 msgr2=0x7f03280cc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cc610 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f031000c7f0 tx=0x7f031000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 shutdown_connections 2024-01-16T15:24:04.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f030006c5e0 0x7f030006eaa0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f032815e1e0 0x7f032815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0328143ae0 0x7f03280cc610 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 --2- 172.21.15.141:0/548445884 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f03280cf7d0 0x7f032812c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 >> 172.21.15.141:0/548445884 conn(0x7f032806a800 msgr2=0x7f03280b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:04.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 shutdown_connections 2024-01-16T15:24:04.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.137+0000 7f032e600640 1 -- 172.21.15.141:0/548445884 wait complete. 2024-01-16T15:24:04.153 INFO:tasks.ceph:Scrubbing osd.7 2024-01-16T15:24:04.154 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:24:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.367+0000 7ff73e800640 1 Processor -- start 2024-01-16T15:24:04.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.367+0000 7ff73e800640 1 -- start start 2024-01-16T15:24:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.368+0000 7ff73e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 0x7ff7380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.368+0000 7ff73e800640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff73815f9c0 0x7ff7380cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.368+0000 7ff73e800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73815c5f0 con 0x7ff73813df80 2024-01-16T15:24:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.368+0000 7ff73e800640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73806c470 con 0x7ff73815f9c0 2024-01-16T15:24:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.368+0000 7ff73e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73806c750 con 0x7ff73813e360 2024-01-16T15:24:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff737e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff73815f9c0 0x7ff7380cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff737e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff73815f9c0 0x7ff7380cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33018/0 (socket says 172.21.15.141:33018) 2024-01-16T15:24:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff737e00640 1 -- 172.21.15.141:0/932716412 learned_addr learned my addr 172.21.15.141:0/932716412 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff736a00640 1 --2- 172.21.15.141:0/932716412 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 0x7ff7380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff737e00640 1 -- 172.21.15.141:0/932716412 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 msgr2=0x7ff7380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff737e00640 1 --2- 172.21.15.141:0/932716412 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 0x7ff7380cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff737e00640 1 -- 172.21.15.141:0/932716412 >> v1:172.21.15.188:6789/0 conn(0x7ff73813df80 legacy=0x7ff73815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:04.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.369+0000 7ff737e00640 1 -- 172.21.15.141:0/932716412 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73806ca30 con 0x7ff73815f9c0 2024-01-16T15:24:04.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.370+0000 7ff737e00640 1 --2- 172.21.15.141:0/932716412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73815f9c0 0x7ff7380cf7d0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7ff730009e50 tx=0x7ff73002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c58d224056e251 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:04.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.371+0000 7ff736000640 1 -- 172.21.15.141:0/932716412 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73002ca70 con 0x7ff73815f9c0 2024-01-16T15:24:04.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.371+0000 7ff736000640 1 -- 172.21.15.141:0/932716412 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73002cbd0 con 0x7ff73815f9c0 2024-01-16T15:24:04.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.371+0000 7ff736000640 1 -- 172.21.15.141:0/932716412 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7300343f0 con 0x7ff73815f9c0 2024-01-16T15:24:04.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.371+0000 7ff73e800640 1 -- 172.21.15.141:0/932716412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73815f9c0 msgr2=0x7ff7380cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.371+0000 7ff73e800640 1 --2- 172.21.15.141:0/932716412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73815f9c0 0x7ff7380cf7d0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7ff730009e50 tx=0x7ff73002bed0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.372+0000 7ff73e800640 1 -- 172.21.15.141:0/932716412 shutdown_connections 2024-01-16T15:24:04.370 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.372+0000 7ff73e800640 1 --2- 172.21.15.141:0/932716412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73815f9c0 0x7ff7380cf7d0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.372+0000 7ff73e800640 1 --2- 172.21.15.141:0/932716412 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 0x7ff7380cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.372+0000 7ff73e800640 1 -- 172.21.15.141:0/932716412 >> 172.21.15.141:0/932716412 conn(0x7ff73806a800 msgr2=0x7ff73806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.372+0000 7ff73e800640 1 -- 172.21.15.141:0/932716412 shutdown_connections 2024-01-16T15:24:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.372+0000 7ff73e800640 1 -- 172.21.15.141:0/932716412 wait complete. 2024-01-16T15:24:04.371 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.374+0000 7ff73e800640 1 Processor -- start 2024-01-16T15:24:04.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.374+0000 7ff73e800640 1 -- start start 2024-01-16T15:24:04.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.375+0000 7ff73e800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73813df80 0x7ff73814d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.375+0000 7ff73e800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 0x7ff73814dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.375+0000 7ff73e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73815f9c0 0x7ff7380b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.375+0000 7ff73e800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381f3a30 con 0x7ff73813df80 2024-01-16T15:24:04.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.375+0000 7ff73e800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381f2df0 con 0x7ff73815f9c0 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.375+0000 7ff73e800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381f3090 con 0x7ff73813e360 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.375+0000 7ff737400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73813df80 0x7ff73814d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73813df80 0x7ff73814d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33030/0 (socket says 172.21.15.141:33030) 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 -- 172.21.15.141:0/965942729 learned_addr learned my addr 172.21.15.141:0/965942729 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff736a00640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 0x7ff73814dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737e00640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73815f9c0 0x7ff7380b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 -- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 msgr2=0x7ff73814dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7ff73813e360 0x7ff73814dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 -- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73815f9c0 msgr2=0x7ff7380b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73815f9c0 0x7ff7380b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 -- 172.21.15.141:0/965942729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730009b30 con 0x7ff73813df80 2024-01-16T15:24:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.376+0000 7ff737400640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73813df80 0x7ff73814d1a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7ff72800b3f0 tx=0x7ff72800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.377+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728011830 con 0x7ff73813df80 2024-01-16T15:24:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.377+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff728011990 con 0x7ff73813df80 2024-01-16T15:24:04.376 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.377+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728011d40 con 0x7ff73813df80 2024-01-16T15:24:04.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.377+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7381f3370 con 0x7ff73813df80 2024-01-16T15:24:04.377 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.377+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73814d6e0 con 0x7ff73813df80 2024-01-16T15:24:04.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.378+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff73806cff0 con 0x7ff73813df80 2024-01-16T15:24:04.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.379+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ff72801a400 con 0x7ff73813df80 2024-01-16T15:24:04.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.380+0000 7ff72f400640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff71006c5e0 0x7ff71006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.380+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff7280889e0 con 0x7ff73813df80 2024-01-16T15:24:04.378 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.380+0000 7ff72f400640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7ff710074840 0x7ff710076c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.380+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 --> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff710077350 con 0x7ff710074840 2024-01-16T15:24:04.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.380+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7280895a0 con 0x7ff73813df80 2024-01-16T15:24:04.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.380+0000 7ff736a00640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff71006c5e0 0x7ff71006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.381+0000 7ff737e00640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7ff710074840 0x7ff710076c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.379 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.382+0000 7ff736a00640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff71006c5e0 0x7ff71006eaa0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff738071f00 tx=0x7ff720008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:04.380 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.382+0000 7ff737e00640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7ff710074840 0x7ff710076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:04.381 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.384+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== osd.7 v2:172.21.15.188:6808/2741327973 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff710077350 con 0x7ff710074840 2024-01-16T15:24:04.395 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.398+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 --> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff7381f3ca0 con 0x7ff710074840 2024-01-16T15:24:04.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.399+0000 7ff72f400640 1 -- 172.21.15.141:0/965942729 <== osd.7 v2:172.21.15.188:6808/2741327973 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff7381f3ca0 con 0x7ff710074840 2024-01-16T15:24:04.397 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2024-01-16T15:24:04.397 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:24:04.397 INFO:teuthology.orchestra.run.smithi141.stdout:} 2024-01-16T15:24:04.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7ff710074840 msgr2=0x7ff710076c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.398 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6808/2741327973,v1:172.21.15.188:6809/2741327973] conn(0x7ff710074840 0x7ff710076c80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff71006c5e0 msgr2=0x7ff71006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7ff71006c5e0 0x7ff71006eaa0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff738071f00 tx=0x7ff720008040 comp rx=0 tx=0).stop 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73813df80 msgr2=0x7ff73814d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 --2- 172.21.15.141:0/965942729 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff73813df80 0x7ff73814d1a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7ff72800b3f0 tx=0x7ff72800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff737e00640 1 -- 172.21.15.141:0/965942729 reap_dead start 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 shutdown_connections 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 >> 172.21.15.141:0/965942729 conn(0x7ff73806a800 msgr2=0x7ff73806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:04.399 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 shutdown_connections 2024-01-16T15:24:04.400 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.400+0000 7ff73e800640 1 -- 172.21.15.141:0/965942729 wait complete. 2024-01-16T15:24:04.415 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-16T15:24:04.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.627+0000 7f0ee3e00640 1 Processor -- start 2024-01-16T15:24:04.625 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.627+0000 7f0ee3e00640 1 -- start start 2024-01-16T15:24:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.629+0000 7f0ee3e00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0edc0b3ae0 0x7f0edc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.629+0000 7f0ee3e00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0edc070c40 con 0x7f0edc0cf290 2024-01-16T15:24:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.629+0000 7f0ee3e00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0edc06c470 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.629+0000 7f0ee3e00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0edc071450 con 0x7f0edc0b3ae0 2024-01-16T15:24:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.630+0000 7f0ee0a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0edc0b3ae0 0x7f0edc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.630+0000 7f0ee1e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f0edc0cf290 0x7f0edc0b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:60350/0 (socket says 172.21.15.141:60350) 2024-01-16T15:24:04.627 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.630+0000 7f0ee1e00640 1 -- 172.21.15.141:0/2994834533 learned_addr learned my addr 172.21.15.141:0/2994834533 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:24:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.630+0000 7f0ee0a00640 1 -- 172.21.15.141:0/2994834533 >> v1:172.21.15.188:6789/0 conn(0x7f0edc0ae5a0 legacy=0x7f0edc06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.628 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.630+0000 7f0ee0a00640 1 -- 172.21.15.141:0/2994834533 >> v1:172.21.15.141:6789/0 conn(0x7f0edc0cf290 legacy=0x7f0edc0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.630+0000 7f0ee0a00640 1 -- 172.21.15.141:0/2994834533 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0edc06c750 con 0x7f0edc0b3ae0 2024-01-16T15:24:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.630+0000 7f0edbe00640 1 -- 172.21.15.141:0/2994834533 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019025491 0 0) 0x7f0edc070c40 con 0x7f0edc0cf290 2024-01-16T15:24:04.629 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.631+0000 7f0ee0a00640 1 --2- 172.21.15.141:0/2994834533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0b3ae0 0x7f0edc15e430 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40097a0 tx=0x7f0ec402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3e69bf2e27f088 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0edbe00640 1 -- 172.21.15.141:0/2994834533 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec402f810 con 0x7f0edc0b3ae0 2024-01-16T15:24:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0edbe00640 1 -- 172.21.15.141:0/2994834533 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec402f970 con 0x7f0edc0b3ae0 2024-01-16T15:24:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0edbe00640 1 -- 172.21.15.141:0/2994834533 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec402fd00 con 0x7f0edc0b3ae0 2024-01-16T15:24:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0ee3e00640 1 -- 172.21.15.141:0/2994834533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0b3ae0 msgr2=0x7f0edc15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.630 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/2994834533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0b3ae0 0x7f0edc15e430 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40097a0 tx=0x7f0ec402b9f0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0ee3e00640 1 -- 172.21.15.141:0/2994834533 shutdown_connections 2024-01-16T15:24:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/2994834533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0b3ae0 0x7f0edc15e430 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0ee3e00640 1 -- 172.21.15.141:0/2994834533 >> 172.21.15.141:0/2994834533 conn(0x7f0edc06a800 msgr2=0x7f0edc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0ee3e00640 1 -- 172.21.15.141:0/2994834533 shutdown_connections 2024-01-16T15:24:04.631 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.632+0000 7f0ee3e00640 1 -- 172.21.15.141:0/2994834533 wait complete. 2024-01-16T15:24:04.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.634+0000 7f0ee3e00640 1 Processor -- start 2024-01-16T15:24:04.632 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.634+0000 7f0ee3e00640 1 -- start start 2024-01-16T15:24:04.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0ae5a0 0x7f0edc155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0edc0b3ae0 0x7f0edc0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee3e00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0edc0cf290 0x7f0edc158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee3e00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc157c00 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc156d30 con 0x7f0edc0b3ae0 2024-01-16T15:24:04.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee3e00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc1562b0 con 0x7f0edc0cf290 2024-01-16T15:24:04.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee1400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0ae5a0 0x7f0edc155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee1400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0ae5a0 0x7f0edc155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33048/0 (socket says 172.21.15.141:33048) 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.635+0000 7f0ee1400640 1 -- 172.21.15.141:0/393922145 learned_addr learned my addr 172.21.15.141:0/393922145 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee1e00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0edc0cf290 0x7f0edc158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee0a00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0edc0b3ae0 0x7f0edc0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee1400640 1 -- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0edc0cf290 msgr2=0x7f0edc158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee1400640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0edc0cf290 0x7f0edc158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee1400640 1 -- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0edc0b3ae0 msgr2=0x7f0edc0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee1400640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0edc0b3ae0 0x7f0edc0cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee1400640 1 -- 172.21.15.141:0/393922145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec4009400 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee1400640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0ae5a0 0x7f0edc155d70 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00c450 tx=0x7f0ecc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ed9600640 1 -- 172.21.15.141:0/393922145 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0052d0 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ed9600640 1 -- 172.21.15.141:0/393922145 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc005430 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ed9600640 1 -- 172.21.15.141:0/393922145 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc012870 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.636 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.636+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0edc156f80 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.637+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc157540 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.637 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.638+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0edc0675d0 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.639+0000 7f0ed9600640 1 -- 172.21.15.141:0/393922145 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f0ecc0129d0 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.639+0000 7f0ed9600640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0eb406c5e0 0x7f0eb406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.640+0000 7f0ed9600640 1 -- 172.21.15.141:0/393922145 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0ecc0886c0 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.640+0000 7f0ee0a00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0eb406c5e0 0x7f0eb406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:04.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.641+0000 7f0ee0a00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0eb406c5e0 0x7f0eb406eaa0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4036000 tx=0x7f0ec4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:04.641 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.644+0000 7f0ed9600640 1 -- 172.21.15.141:0/393922145 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ecc054c40 con 0x7f0edc0ae5a0 2024-01-16T15:24:04.830 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.832+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f0edc070550 con 0x7f0eb406c5e0 2024-01-16T15:24:04.831 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.835+0000 7f0ed9600640 1 -- 172.21.15.141:0/393922145 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ec402fb60 con 0x7f0eb406c5e0 2024-01-16T15:24:04.831 INFO:teuthology.orchestra.run.smithi141.stderr:instructed osd(s) 7 to deep-scrub 2024-01-16T15:24:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.836+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0eb406c5e0 msgr2=0x7f0eb406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.836+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0eb406c5e0 0x7f0eb406eaa0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4036000 tx=0x7f0ec4008030 comp rx=0 tx=0).stop 2024-01-16T15:24:04.834 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.836+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0ae5a0 msgr2=0x7f0edc155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.836+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0ae5a0 0x7f0edc155d70 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00c450 tx=0x7f0ecc00c910 comp rx=0 tx=0).stop 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 shutdown_connections 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0eb406c5e0 0x7f0eb406eaa0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0edc0cf290 0x7f0edc158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0edc0b3ae0 0x7f0edc0cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 --2- 172.21.15.141:0/393922145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0edc0ae5a0 0x7f0edc155d70 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 >> 172.21.15.141:0/393922145 conn(0x7f0edc06a800 msgr2=0x7f0edc159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:04.835 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 shutdown_connections 2024-01-16T15:24:04.836 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:04.837+0000 7f0ee3e00640 1 -- 172.21.15.141:0/393922145 wait complete. 2024-01-16T15:24:04.853 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:24:05.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.064+0000 7f92faa00640 1 Processor -- start 2024-01-16T15:24:05.061 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.064+0000 7f92faa00640 1 -- start start 2024-01-16T15:24:05.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.065+0000 7f92faa00640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f92f4143ae0 0x7f92f40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:05.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.065+0000 7f92faa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f40cf7d0 0x7f92f415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:05.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.065+0000 7f92faa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f92f415e1e0 0x7f92f415a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:05.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.065+0000 7f92faa00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f40cfc30 con 0x7f92f415e1e0 2024-01-16T15:24:05.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.065+0000 7f92faa00640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f40b1ee0 con 0x7f92f4143ae0 2024-01-16T15:24:05.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.065+0000 7f92faa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40b2780 con 0x7f92f40cf7d0 2024-01-16T15:24:05.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f92f415e1e0 0x7f92f415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f92f415e1e0 0x7f92f415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33062/0 (socket says 172.21.15.141:33062) 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 -- 172.21.15.141:0/806418923 learned_addr learned my addr 172.21.15.141:0/806418923 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f3e00640 1 --2- 172.21.15.141:0/806418923 >> v2:172.21.15.188:3300/0 conn(0x7f92f4143ae0 0x7f92f40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f3400640 1 --2- 172.21.15.141:0/806418923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f40cf7d0 0x7f92f415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 -- 172.21.15.141:0/806418923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f40cf7d0 msgr2=0x7f92f415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 --2- 172.21.15.141:0/806418923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f40cf7d0 0x7f92f415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 -- 172.21.15.141:0/806418923 >> v2:172.21.15.188:3300/0 conn(0x7f92f4143ae0 msgr2=0x7f92f40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:05.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 --2- 172.21.15.141:0/806418923 >> v2:172.21.15.188:3300/0 conn(0x7f92f4143ae0 0x7f92f40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.066 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.066+0000 7f92f8a00640 1 -- 172.21.15.141:0/806418923 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f415e990 con 0x7f92f415e1e0 2024-01-16T15:24:05.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.067+0000 7f92f8a00640 1 --2- 172.21.15.141:0/806418923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f415e1e0 0x7f92f415a670 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f92f40cfea0 tx=0x7f92ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a486ac7479302d21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:05.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92f2a00640 1 -- 172.21.15.141:0/806418923 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec02ca70 con 0x7f92f415e1e0 2024-01-16T15:24:05.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92f2a00640 1 -- 172.21.15.141:0/806418923 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec02cbd0 con 0x7f92f415e1e0 2024-01-16T15:24:05.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92f2a00640 1 -- 172.21.15.141:0/806418923 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0343f0 con 0x7f92f415e1e0 2024-01-16T15:24:05.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 -- 172.21.15.141:0/806418923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f415e1e0 msgr2=0x7f92f415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:05.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 --2- 172.21.15.141:0/806418923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f415e1e0 0x7f92f415a670 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f92f40cfea0 tx=0x7f92ec02bed0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 -- 172.21.15.141:0/806418923 shutdown_connections 2024-01-16T15:24:05.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 --2- 172.21.15.141:0/806418923 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f415e1e0 0x7f92f415a670 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 --2- 172.21.15.141:0/806418923 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f40cf7d0 0x7f92f415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 --2- 172.21.15.141:0/806418923 >> v2:172.21.15.188:3300/0 conn(0x7f92f4143ae0 0x7f92f40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 -- 172.21.15.141:0/806418923 >> 172.21.15.141:0/806418923 conn(0x7f92f406a800 msgr2=0x7f92f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:05.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 -- 172.21.15.141:0/806418923 shutdown_connections 2024-01-16T15:24:05.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.068+0000 7f92faa00640 1 -- 172.21.15.141:0/806418923 wait complete. 2024-01-16T15:24:05.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.070+0000 7f92faa00640 1 Processor -- start 2024-01-16T15:24:05.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.070+0000 7f92faa00640 1 -- start start 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.071+0000 7f92faa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f40cf7d0 0x7f92f406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.071+0000 7f92faa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f92f4143ae0 0x7f92f40adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.071+0000 7f92faa00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f415e1e0 0x7f92f413e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.071+0000 7f92faa00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40ac970 con 0x7f92f40cf7d0 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.071+0000 7f92faa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40acbe0 con 0x7f92f4143ae0 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.071+0000 7f92faa00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40acec0 con 0x7f92f415e1e0 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f40cf7d0 0x7f92f406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:05.071 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f40cf7d0 0x7f92f406c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:33076/0 (socket says 172.21.15.141:33076) 2024-01-16T15:24:05.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 -- 172.21.15.141:0/2729107210 learned_addr learned my addr 172.21.15.141:0/2729107210 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:05.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f8a00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f415e1e0 0x7f92f413e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:05.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3400640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f92f4143ae0 0x7f92f40adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:05.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 -- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f415e1e0 msgr2=0x7f92f413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:05.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f415e1e0 0x7f92f413e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 -- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f92f4143ae0 msgr2=0x7f92f40adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:05.072 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f92f4143ae0 0x7f92f40adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 -- 172.21.15.141:0/2729107210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec009b30 con 0x7f92f40cf7d0 2024-01-16T15:24:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.072+0000 7f92f3e00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f40cf7d0 0x7f92f406c6e0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d800 tx=0x7f92e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.073+0000 7f92ebe00640 1 -- 172.21.15.141:0/2729107210 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e400b6c0 con 0x7f92f40cf7d0 2024-01-16T15:24:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.073+0000 7f92ebe00640 1 -- 172.21.15.141:0/2729107210 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e400b820 con 0x7f92f40cf7d0 2024-01-16T15:24:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.073+0000 7f92ebe00640 1 -- 172.21.15.141:0/2729107210 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e400bbb0 con 0x7f92f40cf7d0 2024-01-16T15:24:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.073+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f40ad1a0 con 0x7f92f40cf7d0 2024-01-16T15:24:05.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.073+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f4159d80 con 0x7f92f40cf7d0 2024-01-16T15:24:05.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.074+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f40675d0 con 0x7f92f40cf7d0 2024-01-16T15:24:05.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.075+0000 7f92ebe00640 1 -- 172.21.15.141:0/2729107210 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f92e400bd10 con 0x7f92f40cf7d0 2024-01-16T15:24:05.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.076+0000 7f92ebe00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92cc06c5e0 0x7f92cc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:05.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.076+0000 7f92ebe00640 1 -- 172.21.15.141:0/2729107210 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f92e4089180 con 0x7f92f40cf7d0 2024-01-16T15:24:05.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.076+0000 7f92f3400640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92cc06c5e0 0x7f92cc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:05.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.077+0000 7f92f3400640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92cc06c5e0 0x7f92cc06eaa0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f92dc005fd0 tx=0x7f92dc0059b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:05.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.080+0000 7f92ebe00640 1 -- 172.21.15.141:0/2729107210 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92e40556a0 con 0x7f92f40cf7d0 2024-01-16T15:24:05.265 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.268+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92f415e640 con 0x7f92cc06c5e0 2024-01-16T15:24:05.288 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.291+0000 7f92ebe00640 1 -- 172.21.15.141:0/2729107210 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286933 (secure 0 0 0) 0x7f92dc011040 con 0x7f92cc06c5e0 2024-01-16T15:24:05.292 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:24:05.294 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:24:05.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92cc06c5e0 msgr2=0x7f92cc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:05.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92cc06c5e0 0x7f92cc06eaa0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f92dc005fd0 tx=0x7f92dc0059b0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f40cf7d0 msgr2=0x7f92f406c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:05.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f40cf7d0 0x7f92f406c6e0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d800 tx=0x7f92e400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 shutdown_connections 2024-01-16T15:24:05.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f92cc06c5e0 0x7f92cc06eaa0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.296 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f92f415e1e0 0x7f92f413e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f92f4143ae0 0x7f92f40adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 --2- 172.21.15.141:0/2729107210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92f40cf7d0 0x7f92f406c6e0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:05.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 >> 172.21.15.141:0/2729107210 conn(0x7f92f406a800 msgr2=0x7f92f4070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:05.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 shutdown_connections 2024-01-16T15:24:05.297 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:05.298+0000 7f92faa00640 1 -- 172.21.15.141:0/2729107210 wait complete. 2024-01-16T15:24:05.314 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":924,"stamp":"2024-01-16T15:24:05.124143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444295670,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":945,"num_read_kb":3145889,"num_write":2253,"num_write_kb":6295371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4731,"ondisk_log_size":4731,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13195376,"kb_used_data":12598064,"kb_used_omap":12,"kb_used_meta":597235,"kb_avail":741779344,"statfs":{"total":773094113280,"available":759582048256,"internally_reserved":0,"allocated":12900417536,"data_stored":12893672556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":611569202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11969,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004535"},"pg_stats":[{"pgid":"17.3f","version":"118'70","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900432+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:23:59.900432+0000","last_peered":"2024-01-16T15:23:59.900432+0000","last_clean":"2024-01-16T15:23:59.900432+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:23:59.900432+0000","last_undegraded":"2024-01-16T15:23:59.900432+0000","last_fullsized":"2024-01-16T15:23:59.900432+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"118'69","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017572+0000","last_change":"2024-01-16T15:15:39.327644+0000","last_active":"2024-01-16T15:24:00.017572+0000","last_peered":"2024-01-16T15:24:00.017572+0000","last_clean":"2024-01-16T15:24:00.017572+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:24:00.017572+0000","last_undegraded":"2024-01-16T15:24:00.017572+0000","last_fullsized":"2024-01-16T15:24:00.017572+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"118'67","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.754660+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:23:49.754660+0000","last_peered":"2024-01-16T15:23:49.754660+0000","last_clean":"2024-01-16T15:23:49.754660+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:23:49.754660+0000","last_undegraded":"2024-01-16T15:23:49.754660+0000","last_fullsized":"2024-01-16T15:23:49.754660+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:22.535735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"118'66","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.927494+0000","last_change":"2024-01-16T15:15:55.875926+0000","last_active":"2024-01-16T15:23:59.927494+0000","last_peered":"2024-01-16T15:23:59.927494+0000","last_clean":"2024-01-16T15:23:59.927494+0000","last_became_active":"2024-01-16T15:15:55.875179+0000","last_became_peered":"2024-01-16T15:15:55.875179+0000","last_unstale":"2024-01-16T15:23:59.927494+0000","last_undegraded":"2024-01-16T15:23:59.927494+0000","last_fullsized":"2024-01-16T15:23:59.927494+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3b","version":"118'60","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737970+0000","last_change":"2024-01-16T15:15:39.321081+0000","last_active":"2024-01-16T15:23:49.737970+0000","last_peered":"2024-01-16T15:23:49.737970+0000","last_clean":"2024-01-16T15:23:49.737970+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:23:49.737970+0000","last_undegraded":"2024-01-16T15:23:49.737970+0000","last_fullsized":"2024-01-16T15:23:49.737970+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663487,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"118'73","reported_seq":108,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.743924+0000","last_change":"2024-01-16T15:15:39.327813+0000","last_active":"2024-01-16T15:23:49.743924+0000","last_peered":"2024-01-16T15:23:49.743924+0000","last_clean":"2024-01-16T15:23:49.743924+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:23:49.743924+0000","last_undegraded":"2024-01-16T15:23:49.743924+0000","last_fullsized":"2024-01-16T15:23:49.743924+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"118'66","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751912+0000","last_change":"2024-01-16T15:15:39.324707+0000","last_active":"2024-01-16T15:23:49.751912+0000","last_peered":"2024-01-16T15:23:49.751912+0000","last_clean":"2024-01-16T15:23:49.751912+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:23:49.751912+0000","last_undegraded":"2024-01-16T15:23:49.751912+0000","last_fullsized":"2024-01-16T15:23:49.751912+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303428,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"118'54","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752934+0000","last_change":"2024-01-16T15:15:39.328547+0000","last_active":"2024-01-16T15:23:49.752934+0000","last_peered":"2024-01-16T15:23:49.752934+0000","last_clean":"2024-01-16T15:23:49.752934+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:23:49.752934+0000","last_undegraded":"2024-01-16T15:23:49.752934+0000","last_fullsized":"2024-01-16T15:23:49.752934+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:20.486476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"118'79","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758978+0000","last_change":"2024-01-16T15:15:39.320236+0000","last_active":"2024-01-16T15:23:49.758978+0000","last_peered":"2024-01-16T15:23:49.758978+0000","last_clean":"2024-01-16T15:23:49.758978+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:23:49.758978+0000","last_undegraded":"2024-01-16T15:23:49.758978+0000","last_fullsized":"2024-01-16T15:23:49.758978+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052164,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"118'66","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758376+0000","last_change":"2024-01-16T15:15:39.326528+0000","last_active":"2024-01-16T15:23:49.758376+0000","last_peered":"2024-01-16T15:23:49.758376+0000","last_clean":"2024-01-16T15:23:49.758376+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:23:49.758376+0000","last_undegraded":"2024-01-16T15:23:49.758376+0000","last_fullsized":"2024-01-16T15:23:49.758376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"118'70","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737994+0000","last_change":"2024-01-16T15:15:39.325379+0000","last_active":"2024-01-16T15:23:49.737994+0000","last_peered":"2024-01-16T15:23:49.737994+0000","last_clean":"2024-01-16T15:23:49.737994+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:23:49.737994+0000","last_undegraded":"2024-01-16T15:23:49.737994+0000","last_fullsized":"2024-01-16T15:23:49.737994+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"118'59","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758102+0000","last_change":"2024-01-16T15:15:39.317978+0000","last_active":"2024-01-16T15:23:49.758102+0000","last_peered":"2024-01-16T15:23:49.758102+0000","last_clean":"2024-01-16T15:23:49.758102+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:23:49.758102+0000","last_undegraded":"2024-01-16T15:23:49.758102+0000","last_fullsized":"2024-01-16T15:23:49.758102+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"118'57","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746095+0000","last_change":"2024-01-16T15:15:39.329570+0000","last_active":"2024-01-16T15:23:49.746095+0000","last_peered":"2024-01-16T15:23:49.746095+0000","last_clean":"2024-01-16T15:23:49.746095+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:23:49.746095+0000","last_undegraded":"2024-01-16T15:23:49.746095+0000","last_fullsized":"2024-01-16T15:23:49.746095+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:22.565771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"118'65","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742239+0000","last_change":"2024-01-16T15:15:39.325613+0000","last_active":"2024-01-16T15:23:49.742239+0000","last_peered":"2024-01-16T15:23:49.742239+0000","last_clean":"2024-01-16T15:23:49.742239+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:23:49.742239+0000","last_undegraded":"2024-01-16T15:23:49.742239+0000","last_fullsized":"2024-01-16T15:23:49.742239+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:08.369872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"118'54","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741679+0000","last_change":"2024-01-16T15:15:39.333121+0000","last_active":"2024-01-16T15:23:49.741679+0000","last_peered":"2024-01-16T15:23:49.741679+0000","last_clean":"2024-01-16T15:23:49.741679+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:23:49.741679+0000","last_undegraded":"2024-01-16T15:23:49.741679+0000","last_fullsized":"2024-01-16T15:23:49.741679+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"118'58","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757965+0000","last_change":"2024-01-16T15:15:39.313940+0000","last_active":"2024-01-16T15:23:49.757965+0000","last_peered":"2024-01-16T15:23:49.757965+0000","last_clean":"2024-01-16T15:23:49.757965+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:23:49.757965+0000","last_undegraded":"2024-01-16T15:23:49.757965+0000","last_fullsized":"2024-01-16T15:23:49.757965+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"118'49","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740295+0000","last_change":"2024-01-16T15:15:39.329528+0000","last_active":"2024-01-16T15:23:49.740295+0000","last_peered":"2024-01-16T15:23:49.740295+0000","last_clean":"2024-01-16T15:23:49.740295+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:23:49.740295+0000","last_undegraded":"2024-01-16T15:23:49.740295+0000","last_fullsized":"2024-01-16T15:23:49.740295+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"118'72","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742141+0000","last_change":"2024-01-16T15:15:39.330676+0000","last_active":"2024-01-16T15:23:49.742141+0000","last_peered":"2024-01-16T15:23:49.742141+0000","last_clean":"2024-01-16T15:23:49.742141+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:23:49.742141+0000","last_undegraded":"2024-01-16T15:23:49.742141+0000","last_fullsized":"2024-01-16T15:23:49.742141+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:39:04.346629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"118'71","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751357+0000","last_change":"2024-01-16T15:15:39.330437+0000","last_active":"2024-01-16T15:23:49.751357+0000","last_peered":"2024-01-16T15:23:49.751357+0000","last_clean":"2024-01-16T15:23:49.751357+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:23:49.751357+0000","last_undegraded":"2024-01-16T15:23:49.751357+0000","last_fullsized":"2024-01-16T15:23:49.751357+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:40:37.311014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"118'67","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751037+0000","last_change":"2024-01-16T15:15:39.318689+0000","last_active":"2024-01-16T15:23:49.751037+0000","last_peered":"2024-01-16T15:23:49.751037+0000","last_clean":"2024-01-16T15:23:49.751037+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:23:49.751037+0000","last_undegraded":"2024-01-16T15:23:49.751037+0000","last_fullsized":"2024-01-16T15:23:49.751037+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"118'73","reported_seq":112,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749532+0000","last_change":"2024-01-16T15:15:39.321055+0000","last_active":"2024-01-16T15:23:49.749532+0000","last_peered":"2024-01-16T15:23:49.749532+0000","last_clean":"2024-01-16T15:23:49.749532+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:23:49.749532+0000","last_undegraded":"2024-01-16T15:23:49.749532+0000","last_fullsized":"2024-01-16T15:23:49.749532+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"118'64","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749573+0000","last_change":"2024-01-16T15:15:39.323290+0000","last_active":"2024-01-16T15:23:49.749573+0000","last_peered":"2024-01-16T15:23:49.749573+0000","last_clean":"2024-01-16T15:23:49.749573+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:23:49.749573+0000","last_undegraded":"2024-01-16T15:23:49.749573+0000","last_fullsized":"2024-01-16T15:23:49.749573+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"118'57","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747061+0000","last_change":"2024-01-16T15:15:39.317974+0000","last_active":"2024-01-16T15:23:49.747061+0000","last_peered":"2024-01-16T15:23:49.747061+0000","last_clean":"2024-01-16T15:23:49.747061+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:23:49.747061+0000","last_undegraded":"2024-01-16T15:23:49.747061+0000","last_fullsized":"2024-01-16T15:23:49.747061+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"118'70","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753795+0000","last_change":"2024-01-16T15:15:39.324363+0000","last_active":"2024-01-16T15:23:49.753795+0000","last_peered":"2024-01-16T15:23:49.753795+0000","last_clean":"2024-01-16T15:23:49.753795+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:23:49.753795+0000","last_undegraded":"2024-01-16T15:23:49.753795+0000","last_fullsized":"2024-01-16T15:23:49.753795+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"118'71","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747910+0000","last_change":"2024-01-16T15:15:39.328896+0000","last_active":"2024-01-16T15:23:49.747910+0000","last_peered":"2024-01-16T15:23:49.747910+0000","last_clean":"2024-01-16T15:23:49.747910+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:23:49.747910+0000","last_undegraded":"2024-01-16T15:23:49.747910+0000","last_fullsized":"2024-01-16T15:23:49.747910+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"118'81","reported_seq":108,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757316+0000","last_change":"2024-01-16T15:15:39.317589+0000","last_active":"2024-01-16T15:23:49.757316+0000","last_peered":"2024-01-16T15:23:49.757316+0000","last_clean":"2024-01-16T15:23:49.757316+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:23:49.757316+0000","last_undegraded":"2024-01-16T15:23:49.757316+0000","last_fullsized":"2024-01-16T15:23:49.757316+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"118'64","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747050+0000","last_change":"2024-01-16T15:15:39.335455+0000","last_active":"2024-01-16T15:23:49.747050+0000","last_peered":"2024-01-16T15:23:49.747050+0000","last_clean":"2024-01-16T15:23:49.747050+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:23:49.747050+0000","last_undegraded":"2024-01-16T15:23:49.747050+0000","last_fullsized":"2024-01-16T15:23:49.747050+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"118'43","reported_seq":65,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749401+0000","last_change":"2024-01-16T15:15:39.317292+0000","last_active":"2024-01-16T15:23:49.749401+0000","last_peered":"2024-01-16T15:23:49.749401+0000","last_clean":"2024-01-16T15:23:49.749401+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:23:49.749401+0000","last_undegraded":"2024-01-16T15:23:49.749401+0000","last_fullsized":"2024-01-16T15:23:49.749401+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"118'57","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017399+0000","last_change":"2024-01-16T15:15:39.320340+0000","last_active":"2024-01-16T15:24:00.017399+0000","last_peered":"2024-01-16T15:24:00.017399+0000","last_clean":"2024-01-16T15:24:00.017399+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:24:00.017399+0000","last_undegraded":"2024-01-16T15:24:00.017399+0000","last_fullsized":"2024-01-16T15:24:00.017399+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"118'76","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.919383+0000","last_change":"2024-01-16T15:15:39.317544+0000","last_active":"2024-01-16T15:23:59.919383+0000","last_peered":"2024-01-16T15:23:59.919383+0000","last_clean":"2024-01-16T15:23:59.919383+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:23:59.919383+0000","last_undegraded":"2024-01-16T15:23:59.919383+0000","last_fullsized":"2024-01-16T15:23:59.919383+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"118'81","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900428+0000","last_change":"2024-01-16T15:15:39.313788+0000","last_active":"2024-01-16T15:23:59.900428+0000","last_peered":"2024-01-16T15:23:59.900428+0000","last_clean":"2024-01-16T15:23:59.900428+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:23:59.900428+0000","last_undegraded":"2024-01-16T15:23:59.900428+0000","last_fullsized":"2024-01-16T15:23:59.900428+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"118'63","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.099403+0000","last_change":"2024-01-16T15:15:39.321572+0000","last_active":"2024-01-16T15:24:00.099403+0000","last_peered":"2024-01-16T15:24:00.099403+0000","last_clean":"2024-01-16T15:24:00.099403+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:24:00.099403+0000","last_undegraded":"2024-01-16T15:24:00.099403+0000","last_fullsized":"2024-01-16T15:24:00.099403+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"118'62","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.733839+0000","last_change":"2024-01-16T15:15:39.329833+0000","last_active":"2024-01-16T15:23:49.733839+0000","last_peered":"2024-01-16T15:23:49.733839+0000","last_clean":"2024-01-16T15:23:49.733839+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:23:49.733839+0000","last_undegraded":"2024-01-16T15:23:49.733839+0000","last_fullsized":"2024-01-16T15:23:49.733839+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'4","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.919634+0000","last_change":"2024-01-16T15:15:38.295673+0000","last_active":"2024-01-16T15:23:59.919634+0000","last_peered":"2024-01-16T15:23:59.919634+0000","last_clean":"2024-01-16T15:23:59.919634+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:23:59.919634+0000","last_undegraded":"2024-01-16T15:23:59.919634+0000","last_fullsized":"2024-01-16T15:23:59.919634+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"118'54","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751831+0000","last_change":"2024-01-16T15:15:39.329620+0000","last_active":"2024-01-16T15:23:49.751831+0000","last_peered":"2024-01-16T15:23:49.751831+0000","last_clean":"2024-01-16T15:23:49.751831+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:23:49.751831+0000","last_undegraded":"2024-01-16T15:23:49.751831+0000","last_fullsized":"2024-01-16T15:23:49.751831+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.326448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:40.158702+0000","last_change":"2024-01-16T15:15:38.292769+0000","last_active":"2024-01-16T15:21:40.158702+0000","last_peered":"2024-01-16T15:21:40.158702+0000","last_clean":"2024-01-16T15:21:40.158702+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:21:40.158702+0000","last_undegraded":"2024-01-16T15:21:40.158702+0000","last_fullsized":"2024-01-16T15:21:40.158702+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:45.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"118'71","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744817+0000","last_change":"2024-01-16T15:15:39.332870+0000","last_active":"2024-01-16T15:23:49.744817+0000","last_peered":"2024-01-16T15:23:49.744817+0000","last_clean":"2024-01-16T15:23:49.744817+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:23:49.744817+0000","last_undegraded":"2024-01-16T15:23:49.744817+0000","last_fullsized":"2024-01-16T15:23:49.744817+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"118'3","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:09.918272+0000","last_change":"2024-01-16T15:15:38.288495+0000","last_active":"2024-01-16T15:23:09.918272+0000","last_peered":"2024-01-16T15:23:09.918272+0000","last_clean":"2024-01-16T15:23:09.918272+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:23:09.918272+0000","last_undegraded":"2024-01-16T15:23:09.918272+0000","last_fullsized":"2024-01-16T15:23:09.918272+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"118'68","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753436+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:23:49.753436+0000","last_peered":"2024-01-16T15:23:49.753436+0000","last_clean":"2024-01-16T15:23:49.753436+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:23:49.753436+0000","last_undegraded":"2024-01-16T15:23:49.753436+0000","last_fullsized":"2024-01-16T15:23:49.753436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:21.846652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'2","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:09.918255+0000","last_change":"2024-01-16T15:15:38.288324+0000","last_active":"2024-01-16T15:23:09.918255+0000","last_peered":"2024-01-16T15:23:09.918255+0000","last_clean":"2024-01-16T15:23:09.918255+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:23:09.918255+0000","last_undegraded":"2024-01-16T15:23:09.918255+0000","last_fullsized":"2024-01-16T15:23:09.918255+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"118'72","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756227+0000","last_change":"2024-01-16T15:15:39.321037+0000","last_active":"2024-01-16T15:23:49.756227+0000","last_peered":"2024-01-16T15:23:49.756227+0000","last_clean":"2024-01-16T15:23:49.756227+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:23:49.756227+0000","last_undegraded":"2024-01-16T15:23:49.756227+0000","last_fullsized":"2024-01-16T15:23:49.756227+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:09.910252+0000","last_change":"2024-01-16T15:15:38.288226+0000","last_active":"2024-01-16T15:23:09.910252+0000","last_peered":"2024-01-16T15:23:09.910252+0000","last_clean":"2024-01-16T15:23:09.910252+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:23:09.910252+0000","last_undegraded":"2024-01-16T15:23:09.910252+0000","last_fullsized":"2024-01-16T15:23:09.910252+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"118'72","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752192+0000","last_change":"2024-01-16T15:15:39.330201+0000","last_active":"2024-01-16T15:23:49.752192+0000","last_peered":"2024-01-16T15:23:49.752192+0000","last_clean":"2024-01-16T15:23:49.752192+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:23:49.752192+0000","last_undegraded":"2024-01-16T15:23:49.752192+0000","last_fullsized":"2024-01-16T15:23:49.752192+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.835879+0000","last_change":"2024-01-16T15:24:02.835904+0000","last_active":"2024-01-16T15:24:02.835879+0000","last_peered":"2024-01-16T15:24:02.835879+0000","last_clean":"2024-01-16T15:24:02.835879+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:24:02.835879+0000","last_undegraded":"2024-01-16T15:24:02.835879+0000","last_fullsized":"2024-01-16T15:24:02.835879+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.835792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:09.602675+0000","scrub_duration":0.0029040390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"118'95","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.969881+0000","last_change":"2024-01-16T15:24:00.969955+0000","last_active":"2024-01-16T15:24:00.969881+0000","last_peered":"2024-01-16T15:24:00.969881+0000","last_clean":"2024-01-16T15:24:00.969881+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:24:00.969881+0000","last_undegraded":"2024-01-16T15:24:00.969881+0000","last_fullsized":"2024-01-16T15:24:00.969881+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.969804+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:57.211104+0000","scrub_duration":0.179269503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:55.004997+0000","last_change":"2024-01-16T15:15:38.276895+0000","last_active":"2024-01-16T15:22:55.004997+0000","last_peered":"2024-01-16T15:22:55.004997+0000","last_clean":"2024-01-16T15:22:55.004997+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:22:55.004997+0000","last_undegraded":"2024-01-16T15:22:55.004997+0000","last_fullsized":"2024-01-16T15:22:55.004997+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"118'73","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749206+0000","last_change":"2024-01-16T15:15:39.325224+0000","last_active":"2024-01-16T15:23:49.749206+0000","last_peered":"2024-01-16T15:23:49.749206+0000","last_clean":"2024-01-16T15:23:49.749206+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:23:49.749206+0000","last_undegraded":"2024-01-16T15:23:49.749206+0000","last_fullsized":"2024-01-16T15:23:49.749206+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:54.903248+0000","last_change":"2024-01-16T15:15:38.297269+0000","last_active":"2024-01-16T15:22:54.903248+0000","last_peered":"2024-01-16T15:22:54.903248+0000","last_clean":"2024-01-16T15:22:54.903248+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:22:54.903248+0000","last_undegraded":"2024-01-16T15:22:54.903248+0000","last_fullsized":"2024-01-16T15:22:54.903248+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"118'58","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.735610+0000","last_change":"2024-01-16T15:15:39.320446+0000","last_active":"2024-01-16T15:23:49.735610+0000","last_peered":"2024-01-16T15:23:49.735610+0000","last_clean":"2024-01-16T15:23:49.735610+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:23:49.735610+0000","last_undegraded":"2024-01-16T15:23:49.735610+0000","last_fullsized":"2024-01-16T15:23:49.735610+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"118'1","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:54.915872+0000","last_change":"2024-01-16T15:15:38.287453+0000","last_active":"2024-01-16T15:22:54.915872+0000","last_peered":"2024-01-16T15:22:54.915872+0000","last_clean":"2024-01-16T15:22:54.915872+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:22:54.915872+0000","last_undegraded":"2024-01-16T15:22:54.915872+0000","last_fullsized":"2024-01-16T15:22:54.915872+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"118'61","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747524+0000","last_change":"2024-01-16T15:15:39.333810+0000","last_active":"2024-01-16T15:23:49.747524+0000","last_peered":"2024-01-16T15:23:49.747524+0000","last_clean":"2024-01-16T15:23:49.747524+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:23:49.747524+0000","last_undegraded":"2024-01-16T15:23:49.747524+0000","last_fullsized":"2024-01-16T15:23:49.747524+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:22:34.785294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:54.991718+0000","last_change":"2024-01-16T15:15:38.293583+0000","last_active":"2024-01-16T15:22:54.991718+0000","last_peered":"2024-01-16T15:22:54.991718+0000","last_clean":"2024-01-16T15:22:54.991718+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:22:54.991718+0000","last_undegraded":"2024-01-16T15:22:54.991718+0000","last_fullsized":"2024-01-16T15:22:54.991718+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"118'62","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.738693+0000","last_change":"2024-01-16T15:15:39.319368+0000","last_active":"2024-01-16T15:23:49.738693+0000","last_peered":"2024-01-16T15:23:49.738693+0000","last_clean":"2024-01-16T15:23:49.738693+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:23:49.738693+0000","last_undegraded":"2024-01-16T15:23:49.738693+0000","last_fullsized":"2024-01-16T15:23:49.738693+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"118'31","reported_seq":49,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:54.903401+0000","last_change":"2024-01-16T15:15:38.295750+0000","last_active":"2024-01-16T15:22:54.903401+0000","last_peered":"2024-01-16T15:22:54.903401+0000","last_clean":"2024-01-16T15:22:54.903401+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:22:54.903401+0000","last_undegraded":"2024-01-16T15:22:54.903401+0000","last_fullsized":"2024-01-16T15:22:54.903401+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"118'69","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751912+0000","last_change":"2024-01-16T15:15:39.312858+0000","last_active":"2024-01-16T15:23:49.751912+0000","last_peered":"2024-01-16T15:23:49.751912+0000","last_clean":"2024-01-16T15:23:49.751912+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:23:49.751912+0000","last_undegraded":"2024-01-16T15:23:49.751912+0000","last_fullsized":"2024-01-16T15:23:49.751912+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"118'326","reported_seq":344,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.331473+0000","last_change":"2024-01-16T15:15:38.296732+0000","last_active":"2024-01-16T15:23:55.331473+0000","last_peered":"2024-01-16T15:23:55.331473+0000","last_clean":"2024-01-16T15:23:55.331473+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:23:55.331473+0000","last_undegraded":"2024-01-16T15:23:55.331473+0000","last_fullsized":"2024-01-16T15:23:55.331473+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1247921,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":1376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"118'54","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737914+0000","last_change":"2024-01-16T15:15:39.312723+0000","last_active":"2024-01-16T15:23:49.737914+0000","last_peered":"2024-01-16T15:23:49.737914+0000","last_clean":"2024-01-16T15:23:49.737914+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:23:49.737914+0000","last_undegraded":"2024-01-16T15:23:49.737914+0000","last_fullsized":"2024-01-16T15:23:49.737914+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"118'5","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:03.636265+0000","last_change":"2024-01-16T15:24:03.636309+0000","last_active":"2024-01-16T15:24:03.636265+0000","last_peered":"2024-01-16T15:24:03.636265+0000","last_clean":"2024-01-16T15:24:03.636265+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:24:03.636265+0000","last_undegraded":"2024-01-16T15:24:03.636265+0000","last_fullsized":"2024-01-16T15:24:03.636265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5","last_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_deep_scrub":"118'5","last_deep_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_clean_scrub_stamp":"2024-01-16T15:24:03.636187+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:22.590604+0000","scrub_duration":0.0075747449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"118'72","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753842+0000","last_change":"2024-01-16T15:15:39.324127+0000","last_active":"2024-01-16T15:23:49.753842+0000","last_peered":"2024-01-16T15:23:49.753842+0000","last_clean":"2024-01-16T15:23:49.753842+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:23:49.753842+0000","last_undegraded":"2024-01-16T15:23:49.753842+0000","last_fullsized":"2024-01-16T15:23:49.753842+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:20.345465+0000","last_change":"2024-01-16T15:15:38.295857+0000","last_active":"2024-01-16T15:20:20.345465+0000","last_peered":"2024-01-16T15:20:20.345465+0000","last_clean":"2024-01-16T15:20:20.345465+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:20:20.345465+0000","last_undegraded":"2024-01-16T15:20:20.345465+0000","last_fullsized":"2024-01-16T15:20:20.345465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:31.063560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"118'64","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.744215+0000","last_change":"2024-01-16T15:15:39.319495+0000","last_active":"2024-01-16T15:23:49.744215+0000","last_peered":"2024-01-16T15:23:49.744215+0000","last_clean":"2024-01-16T15:23:49.744215+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:23:49.744215+0000","last_undegraded":"2024-01-16T15:23:49.744215+0000","last_fullsized":"2024-01-16T15:23:49.744215+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":40961,"num_write":27,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'8","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:39.912997+0000","last_change":"2024-01-16T15:15:38.293904+0000","last_active":"2024-01-16T15:22:39.912997+0000","last_peered":"2024-01-16T15:22:39.912997+0000","last_clean":"2024-01-16T15:22:39.912997+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:22:39.912997+0000","last_undegraded":"2024-01-16T15:22:39.912997+0000","last_fullsized":"2024-01-16T15:22:39.912997+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"118'46","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758301+0000","last_change":"2024-01-16T15:15:39.314200+0000","last_active":"2024-01-16T15:23:49.758301+0000","last_peered":"2024-01-16T15:23:49.758301+0000","last_clean":"2024-01-16T15:23:49.758301+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:23:49.758301+0000","last_undegraded":"2024-01-16T15:23:49.758301+0000","last_fullsized":"2024-01-16T15:23:49.758301+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'1","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.897443+0000","last_change":"2024-01-16T15:15:38.296521+0000","last_active":"2024-01-16T15:22:24.897443+0000","last_peered":"2024-01-16T15:22:24.897443+0000","last_clean":"2024-01-16T15:22:24.897443+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:22:24.897443+0000","last_undegraded":"2024-01-16T15:22:24.897443+0000","last_fullsized":"2024-01-16T15:22:24.897443+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"118'61","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748319+0000","last_change":"2024-01-16T15:15:39.334617+0000","last_active":"2024-01-16T15:23:49.748319+0000","last_peered":"2024-01-16T15:23:49.748319+0000","last_clean":"2024-01-16T15:23:49.748319+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:23:49.748319+0000","last_undegraded":"2024-01-16T15:23:49.748319+0000","last_fullsized":"2024-01-16T15:23:49.748319+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'2","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.910332+0000","last_change":"2024-01-16T15:15:38.289440+0000","last_active":"2024-01-16T15:22:24.910332+0000","last_peered":"2024-01-16T15:22:24.910332+0000","last_clean":"2024-01-16T15:22:24.910332+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:22:24.910332+0000","last_undegraded":"2024-01-16T15:22:24.910332+0000","last_fullsized":"2024-01-16T15:22:24.910332+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"118'61","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755433+0000","last_change":"2024-01-16T15:15:39.332704+0000","last_active":"2024-01-16T15:23:49.755433+0000","last_peered":"2024-01-16T15:23:49.755433+0000","last_clean":"2024-01-16T15:23:49.755433+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:23:49.755433+0000","last_undegraded":"2024-01-16T15:23:49.755433+0000","last_fullsized":"2024-01-16T15:23:49.755433+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:08.430991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'1","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:20:10.343721+0000","last_change":"2024-01-16T15:15:38.291309+0000","last_active":"2024-01-16T15:20:10.343721+0000","last_peered":"2024-01-16T15:20:10.343721+0000","last_clean":"2024-01-16T15:20:10.343721+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:20:10.343721+0000","last_undegraded":"2024-01-16T15:20:10.343721+0000","last_fullsized":"2024-01-16T15:20:10.343721+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:10:35.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"118'62","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755362+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:23:49.755362+0000","last_peered":"2024-01-16T15:23:49.755362+0000","last_clean":"2024-01-16T15:23:49.755362+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:23:49.755362+0000","last_undegraded":"2024-01-16T15:23:49.755362+0000","last_fullsized":"2024-01-16T15:23:49.755362+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:31.476276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:24.902191+0000","last_change":"2024-01-16T15:15:38.298667+0000","last_active":"2024-01-16T15:22:24.902191+0000","last_peered":"2024-01-16T15:22:24.902191+0000","last_clean":"2024-01-16T15:22:24.902191+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:22:24.902191+0000","last_undegraded":"2024-01-16T15:22:24.902191+0000","last_fullsized":"2024-01-16T15:22:24.902191+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"118'56","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742341+0000","last_change":"2024-01-16T15:15:39.326224+0000","last_active":"2024-01-16T15:23:49.742341+0000","last_peered":"2024-01-16T15:23:49.742341+0000","last_clean":"2024-01-16T15:23:49.742341+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:23:49.742341+0000","last_undegraded":"2024-01-16T15:23:49.742341+0000","last_fullsized":"2024-01-16T15:23:49.742341+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:02.679035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"118'4","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.623213+0000","last_change":"2024-01-16T15:24:01.623234+0000","last_active":"2024-01-16T15:24:01.623213+0000","last_peered":"2024-01-16T15:24:01.623213+0000","last_clean":"2024-01-16T15:24:01.623213+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:24:01.623213+0000","last_undegraded":"2024-01-16T15:24:01.623213+0000","last_fullsized":"2024-01-16T15:24:01.623213+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4","last_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_deep_scrub":"118'4","last_deep_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.623154+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:58.563468+0000","scrub_duration":0.0032134569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"118'70","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757305+0000","last_change":"2024-01-16T15:15:39.325408+0000","last_active":"2024-01-16T15:23:49.757305+0000","last_peered":"2024-01-16T15:23:49.757305+0000","last_clean":"2024-01-16T15:23:49.757305+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:23:49.757305+0000","last_undegraded":"2024-01-16T15:23:49.757305+0000","last_fullsized":"2024-01-16T15:23:49.757305+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:09.907466+0000","last_change":"2024-01-16T15:15:38.287926+0000","last_active":"2024-01-16T15:22:09.907466+0000","last_peered":"2024-01-16T15:22:09.907466+0000","last_clean":"2024-01-16T15:22:09.907466+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:22:09.907466+0000","last_undegraded":"2024-01-16T15:22:09.907466+0000","last_fullsized":"2024-01-16T15:22:09.907466+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"118'65","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755705+0000","last_change":"2024-01-16T15:15:39.330480+0000","last_active":"2024-01-16T15:23:49.755705+0000","last_peered":"2024-01-16T15:23:49.755705+0000","last_clean":"2024-01-16T15:23:49.755705+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:23:49.755705+0000","last_undegraded":"2024-01-16T15:23:49.755705+0000","last_fullsized":"2024-01-16T15:23:49.755705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:25.204362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.659997+0000","last_change":"2024-01-16T15:24:02.660042+0000","last_active":"2024-01-16T15:24:02.659997+0000","last_peered":"2024-01-16T15:24:02.659997+0000","last_clean":"2024-01-16T15:24:02.659997+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:24:02.659997+0000","last_undegraded":"2024-01-16T15:24:02.659997+0000","last_fullsized":"2024-01-16T15:24:02.659997+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.659822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:34.891359+0000","scrub_duration":0.0028615590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"118'57","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749962+0000","last_change":"2024-01-16T15:15:39.316268+0000","last_active":"2024-01-16T15:23:49.749962+0000","last_peered":"2024-01-16T15:23:49.749962+0000","last_clean":"2024-01-16T15:23:49.749962+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:23:49.749962+0000","last_undegraded":"2024-01-16T15:23:49.749962+0000","last_fullsized":"2024-01-16T15:23:49.749962+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692036,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:09.894264+0000","last_change":"2024-01-16T15:15:38.297590+0000","last_active":"2024-01-16T15:22:09.894264+0000","last_peered":"2024-01-16T15:22:09.894264+0000","last_clean":"2024-01-16T15:22:09.894264+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:22:09.894264+0000","last_undegraded":"2024-01-16T15:22:09.894264+0000","last_fullsized":"2024-01-16T15:22:09.894264+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"118'53","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.750693+0000","last_change":"2024-01-16T15:15:39.333001+0000","last_active":"2024-01-16T15:23:49.750693+0000","last_peered":"2024-01-16T15:23:49.750693+0000","last_clean":"2024-01-16T15:23:49.750693+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:23:49.750693+0000","last_undegraded":"2024-01-16T15:23:49.750693+0000","last_fullsized":"2024-01-16T15:23:49.750693+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:48.934538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'2","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:09.982995+0000","last_change":"2024-01-16T15:15:38.291784+0000","last_active":"2024-01-16T15:22:09.982995+0000","last_peered":"2024-01-16T15:22:09.982995+0000","last_clean":"2024-01-16T15:22:09.982995+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:22:09.982995+0000","last_undegraded":"2024-01-16T15:22:09.982995+0000","last_fullsized":"2024-01-16T15:22:09.982995+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"118'61","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.759314+0000","last_change":"2024-01-16T15:15:39.320032+0000","last_active":"2024-01-16T15:23:49.759314+0000","last_peered":"2024-01-16T15:23:49.759314+0000","last_clean":"2024-01-16T15:23:49.759314+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:23:49.759314+0000","last_undegraded":"2024-01-16T15:23:49.759314+0000","last_fullsized":"2024-01-16T15:23:49.759314+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"118'31","reported_seq":48,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:52.889818+0000","last_change":"2024-01-16T15:15:38.295797+0000","last_active":"2024-01-16T15:23:52.889818+0000","last_peered":"2024-01-16T15:23:52.889818+0000","last_clean":"2024-01-16T15:23:52.889818+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:23:52.889818+0000","last_undegraded":"2024-01-16T15:23:52.889818+0000","last_fullsized":"2024-01-16T15:23:52.889818+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:49.218031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"118'69","reported_seq":103,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748964+0000","last_change":"2024-01-16T15:15:55.923114+0000","last_active":"2024-01-16T15:23:49.748964+0000","last_peered":"2024-01-16T15:23:49.748964+0000","last_clean":"2024-01-16T15:23:49.748964+0000","last_became_active":"2024-01-16T15:15:55.914646+0000","last_became_peered":"2024-01-16T15:15:55.914646+0000","last_unstale":"2024-01-16T15:23:49.748964+0000","last_undegraded":"2024-01-16T15:23:49.748964+0000","last_fullsized":"2024-01-16T15:23:49.748964+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900001+0000","last_change":"2024-01-16T15:15:38.297697+0000","last_active":"2024-01-16T15:23:59.900001+0000","last_peered":"2024-01-16T15:23:59.900001+0000","last_clean":"2024-01-16T15:23:59.900001+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:23:59.900001+0000","last_undegraded":"2024-01-16T15:23:59.900001+0000","last_fullsized":"2024-01-16T15:23:59.900001+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"118'59","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.747076+0000","last_change":"2024-01-16T15:15:39.331210+0000","last_active":"2024-01-16T15:23:49.747076+0000","last_peered":"2024-01-16T15:23:49.747076+0000","last_clean":"2024-01-16T15:23:49.747076+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:23:49.747076+0000","last_undegraded":"2024-01-16T15:23:49.747076+0000","last_fullsized":"2024-01-16T15:23:49.747076+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:29.839201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.003063+0000","last_change":"2024-01-16T15:15:38.287424+0000","last_active":"2024-01-16T15:24:00.003063+0000","last_peered":"2024-01-16T15:24:00.003063+0000","last_clean":"2024-01-16T15:24:00.003063+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:24:00.003063+0000","last_undegraded":"2024-01-16T15:24:00.003063+0000","last_fullsized":"2024-01-16T15:24:00.003063+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"118'63","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017343+0000","last_change":"2024-01-16T15:15:39.313583+0000","last_active":"2024-01-16T15:24:00.017343+0000","last_peered":"2024-01-16T15:24:00.017343+0000","last_clean":"2024-01-16T15:24:00.017343+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:24:00.017343+0000","last_undegraded":"2024-01-16T15:24:00.017343+0000","last_fullsized":"2024-01-16T15:24:00.017343+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.919529+0000","last_change":"2024-01-16T15:15:38.290534+0000","last_active":"2024-01-16T15:23:59.919529+0000","last_peered":"2024-01-16T15:23:59.919529+0000","last_clean":"2024-01-16T15:23:59.919529+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:23:59.919529+0000","last_undegraded":"2024-01-16T15:23:59.919529+0000","last_fullsized":"2024-01-16T15:23:59.919529+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"118'66","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017479+0000","last_change":"2024-01-16T15:15:39.305386+0000","last_active":"2024-01-16T15:24:00.017479+0000","last_peered":"2024-01-16T15:24:00.017479+0000","last_clean":"2024-01-16T15:24:00.017479+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:24:00.017479+0000","last_undegraded":"2024-01-16T15:24:00.017479+0000","last_fullsized":"2024-01-16T15:24:00.017479+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017374+0000","last_change":"2024-01-16T15:15:38.287042+0000","last_active":"2024-01-16T15:24:00.017374+0000","last_peered":"2024-01-16T15:24:00.017374+0000","last_clean":"2024-01-16T15:24:00.017374+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:24:00.017374+0000","last_undegraded":"2024-01-16T15:24:00.017374+0000","last_fullsized":"2024-01-16T15:24:00.017374+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"118'68","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752622+0000","last_change":"2024-01-16T15:15:39.330274+0000","last_active":"2024-01-16T15:23:49.752622+0000","last_peered":"2024-01-16T15:23:49.752622+0000","last_clean":"2024-01-16T15:23:49.752622+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:23:49.752622+0000","last_undegraded":"2024-01-16T15:23:49.752622+0000","last_fullsized":"2024-01-16T15:23:49.752622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:55:58.229013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"118'31","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.927421+0000","last_change":"2024-01-16T15:15:38.281868+0000","last_active":"2024-01-16T15:23:59.927421+0000","last_peered":"2024-01-16T15:23:59.927421+0000","last_clean":"2024-01-16T15:23:59.927421+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:23:59.927421+0000","last_undegraded":"2024-01-16T15:23:59.927421+0000","last_fullsized":"2024-01-16T15:23:59.927421+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3846,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"118'94","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.073597+0000","last_change":"2024-01-16T15:24:04.073657+0000","last_active":"2024-01-16T15:24:04.073597+0000","last_peered":"2024-01-16T15:24:04.073597+0000","last_clean":"2024-01-16T15:24:04.073597+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:24:04.073597+0000","last_undegraded":"2024-01-16T15:24:04.073597+0000","last_fullsized":"2024-01-16T15:24:04.073597+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.073487+0000","objects_scrubbed":32,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:45.014652+0000","scrub_duration":0.28876764900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:39.912402+0000","last_change":"2024-01-16T15:15:38.297094+0000","last_active":"2024-01-16T15:23:39.912402+0000","last_peered":"2024-01-16T15:23:39.912402+0000","last_clean":"2024-01-16T15:23:39.912402+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:23:39.912402+0000","last_undegraded":"2024-01-16T15:23:39.912402+0000","last_fullsized":"2024-01-16T15:23:39.912402+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"118'49","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753307+0000","last_change":"2024-01-16T15:15:39.324548+0000","last_active":"2024-01-16T15:23:49.753307+0000","last_peered":"2024-01-16T15:23:49.753307+0000","last_clean":"2024-01-16T15:23:49.753307+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:23:49.753307+0000","last_undegraded":"2024-01-16T15:23:49.753307+0000","last_fullsized":"2024-01-16T15:23:49.753307+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:22:39.904806+0000","last_change":"2024-01-16T15:15:38.299858+0000","last_active":"2024-01-16T15:22:39.904806+0000","last_peered":"2024-01-16T15:22:39.904806+0000","last_clean":"2024-01-16T15:22:39.904806+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:22:39.904806+0000","last_undegraded":"2024-01-16T15:22:39.904806+0000","last_fullsized":"2024-01-16T15:22:39.904806+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:37.240620+0000","last_clean_scrub_stamp":"2024-01-16T15:15:37.240620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":264,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:21:05.151918+0000","last_change":"2024-01-16T14:57:02.520014+0000","last_active":"2024-01-16T15:21:05.151918+0000","last_peered":"2024-01-16T15:21:05.151918+0000","last_clean":"2024-01-16T15:21:05.151918+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:21:05.151918+0000","last_undegraded":"2024-01-16T15:21:05.151918+0000","last_fullsized":"2024-01-16T15:21:05.151918+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:57:01.176684+0000","last_clean_scrub_stamp":"2024-01-16T14:57:01.176684+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:44.079848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452737,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4159,"ondisk_log_size":4159,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1252565,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":489,"num_write_kb":1461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2505130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":455,"ondisk_log_size":455,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2096820,"kb_used_data":2025452,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92275020,"statfs":{"total":96636764160,"available":94489620480,"internally_reserved":0,"allocated":2074062848,"data_stored":2073213012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.238,"15min":1.26},"min":{"1min":0.84099999999999997,"5min":0.70399999999999996,"15min":0.57799999999999996},"max":{"1min":1.6100000000000001,"5min":1.6679999999999999,"15min":13.566000000000001},"last":1.504},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.323,"15min":1.3089999999999999},"min":{"1min":0.96499999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.984,"5min":1.984,"15min":10.728},"last":1.365}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1200000000000001,"15min":1.159},"min":{"1min":0.74099999999999999,"5min":0.73899999999999999,"15min":0.59699999999999998},"max":{"1min":1.446,"5min":1.714,"15min":13.375},"last":1.048},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1519999999999999,"15min":1.1950000000000001},"min":{"1min":0.86299999999999999,"5min":0.67800000000000005,"15min":0.65900000000000003},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":14.592000000000001},"last":1.1120000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.264,"15min":1.2589999999999999},"min":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.72199999999999998},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":4.9000000000000004},"last":1.252},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1910000000000001,"15min":1.2390000000000001},"min":{"1min":0.89800000000000002,"5min":0.77100000000000002,"15min":0.67600000000000005},"max":{"1min":1.4630000000000001,"5min":1.724,"15min":24.806000000000001},"last":1.181}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2849999999999999,"15min":1.3839999999999999},"min":{"1min":0.86899999999999999,"5min":0.81399999999999995,"15min":0.67000000000000004},"max":{"1min":1.6599999999999999,"5min":2.27,"15min":23.741},"last":1.671},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4650000000000001,"15min":1.508},"min":{"1min":1.1699999999999999,"5min":1.0029999999999999,"15min":0.76500000000000001},"max":{"1min":1.8260000000000001,"5min":2.3119999999999998,"15min":23.777000000000001},"last":1.464}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.2450000000000001,"15min":1.3220000000000001},"min":{"1min":0.81000000000000005,"5min":0.57599999999999996,"15min":0.45900000000000002},"max":{"1min":1.865,"5min":1.865,"15min":22.355},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.411,"5min":1.413,"15min":1.516},"min":{"1min":0.85199999999999998,"5min":0.72899999999999998,"15min":0.63600000000000001},"max":{"1min":1.7689999999999999,"5min":1.9239999999999999,"15min":22.271999999999998},"last":1.7549999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.4139999999999999,"15min":1.454},"min":{"1min":0.95699999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.8300000000000001},"last":1.575},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.95199999999999996,"5min":0.66000000000000003,"15min":0.56699999999999995},"max":{"1min":1.8680000000000001,"5min":1.8999999999999999,"15min":7.8540000000000001},"last":1.6180000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4379999999999999,"15min":1.4370000000000001},"min":{"1min":1.042,"5min":0.53000000000000003,"15min":0.50800000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.6389999999999998},"last":1.413},{"interface":"front","average":{"1min":1.488,"5min":1.4410000000000001,"15min":1.5},"min":{"1min":0.98599999999999999,"5min":0.63400000000000001,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":13.725},"last":1.8100000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542470,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477736,"kb_used_data":1406368,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92894104,"statfs":{"total":96636764160,"available":95123562496,"internally_reserved":0,"allocated":1440120832,"data_stored":1439281876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.179,"15min":1.23},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.58099999999999996},"max":{"1min":1.6319999999999999,"5min":1.764,"15min":12.467000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1930000000000001,"15min":1.2410000000000001},"min":{"1min":0.83099999999999996,"5min":0.67100000000000004,"15min":0.41599999999999998},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":8.0630000000000006},"last":1.5640000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.226,"15min":1.27},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.497},"max":{"1min":1.6200000000000001,"5min":1.7030000000000001,"15min":10.518000000000001},"last":1.151},{"interface":"front","average":{"1min":1.278,"5min":1.3100000000000001,"15min":1.2889999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.53100000000000003},"max":{"1min":1.73,"5min":1.887,"15min":7.2919999999999998},"last":1.0529999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.254,"15min":1.2909999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":1.784,"15min":6.3040000000000003},"last":1.347},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2829999999999999,"15min":1.417},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.57099999999999995},"max":{"1min":1.7470000000000001,"5min":1.8240000000000001,"15min":15.282},"last":0.98099999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.337,"15min":1.359},"min":{"1min":0.95299999999999996,"5min":0.71899999999999997,"15min":0.621},"max":{"1min":1.641,"5min":1.9490000000000001,"15min":4.3659999999999997},"last":1.405},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4139999999999999,"15min":1.458},"min":{"1min":1.159,"5min":0.94999999999999996,"15min":0.50900000000000001},"max":{"1min":1.724,"5min":1.8480000000000001,"15min":9.0960000000000001},"last":1.2150000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.347,"15min":1.379},"min":{"1min":1.026,"5min":0.85999999999999999,"15min":0.49299999999999999},"max":{"1min":1.8580000000000001,"5min":1.895,"15min":5.4669999999999996},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.41199999999999998},"max":{"1min":1.7989999999999999,"5min":1.944,"15min":5.3360000000000003},"last":1.651}]},{"osd":5,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3600000000000001,"15min":1.399},"min":{"1min":0.82799999999999996,"5min":0.65700000000000003,"15min":0.436},"max":{"1min":1.708,"5min":1.9239999999999999,"15min":10.388999999999999},"last":1.266},{"interface":"front","average":{"1min":1.552,"5min":1.5089999999999999,"15min":1.5469999999999999},"min":{"1min":1.1100000000000001,"5min":1.016,"15min":0.42799999999999999},"max":{"1min":1.8560000000000001,"5min":2.012,"15min":7.9640000000000004},"last":1.605}]},{"osd":7,"last update":"Tue Jan 16 15:23:06 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.5429999999999999,"15min":1.577},"min":{"1min":1.05,"5min":0.98399999999999999,"15min":0.46600000000000003},"max":{"1min":1.804,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5029999999999999,"15min":1.6000000000000001},"min":{"1min":0.98199999999999998,"5min":0.84199999999999997,"15min":0.495},"max":{"1min":1.857,"5min":1.984,"15min":16.175999999999998},"last":1.46}]}]},{"osd":5,"up_from":14,"seq":60129542471,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174568,"kb_used_data":1107360,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93197272,"statfs":{"total":96636764160,"available":95434006528,"internally_reserved":0,"allocated":1133936640,"data_stored":1133098096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.278,"15min":1.216},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.532,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.997},{"interface":"front","average":{"1min":1.151,"5min":1.3120000000000001,"15min":1.236},"min":{"1min":0.81899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.657,"5min":12.922000000000001,"15min":12.922000000000001},"last":0.93700000000000006}]},{"osd":1,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3200000000000001,"15min":1.3169999999999999},"min":{"1min":0.89700000000000002,"5min":0.66300000000000003,"15min":0.63900000000000001},"max":{"1min":1.6879999999999999,"5min":5.2249999999999996,"15min":8.1820000000000004},"last":1.173},{"interface":"front","average":{"1min":1.129,"5min":1.175,"15min":1.173},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.536,"5min":2.0659999999999998,"15min":2.597},"last":1.54}]},{"osd":2,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.268,"15min":1.264},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6459999999999999,"5min":1.712,"15min":3.0470000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2250000000000001,"15min":1.2669999999999999},"min":{"1min":0.72299999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6359999999999999,"5min":1.756,"15min":8.0619999999999994},"last":1.127}]},{"osd":3,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.401,"15min":1.484},"min":{"1min":0.83099999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7430000000000001,"5min":5.157,"15min":16.931999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.419,"15min":1.494},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":2.6960000000000002,"15min":19.111999999999998},"last":1.3979999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1930000000000001,"15min":1.2589999999999999},"min":{"1min":0.71899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.7549999999999999,"5min":1.7689999999999999,"15min":4.8490000000000002},"last":0.871},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3540000000000001,"15min":1.411},"min":{"1min":0.84899999999999998,"5min":0.63300000000000001,"15min":0.57399999999999995},"max":{"1min":1.7569999999999999,"5min":5.0810000000000004,"15min":8.3550000000000004},"last":1.0780000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4119999999999999,"15min":1.5069999999999999},"min":{"1min":0.79300000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.716,"5min":1.9530000000000001,"15min":17.419},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.415,"15min":1.5269999999999999},"min":{"1min":0.83599999999999997,"5min":0.69899999999999995,"15min":0.69799999999999995},"max":{"1min":1.7170000000000001,"5min":2.2959999999999998,"15min":16.262},"last":1.22}]},{"osd":7,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.6040000000000001},"min":{"1min":1.1359999999999999,"5min":0.68400000000000005,"15min":0.67000000000000004},"max":{"1min":1.889,"5min":4.0380000000000003,"15min":12.385999999999999},"last":1.248},{"interface":"front","average":{"1min":1.429,"5min":1.431,"15min":1.512},"min":{"1min":0.73999999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6919999999999999,"5min":1.796,"15min":5.0780000000000003},"last":1.6419999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542472,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936484,"kb_used_data":1865116,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435356,"statfs":{"total":96636764160,"available":94653804544,"internally_reserved":0,"allocated":1909878784,"data_stored":1909043912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2649999999999999,"15min":1.2569999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.63300000000000001},"max":{"1min":1.7669999999999999,"5min":1.774,"15min":3.3159999999999998},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.163,"5min":1.129,"15min":1.1919999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.47499999999999998},"max":{"1min":1.77,"5min":1.77,"15min":5.7430000000000003},"last":1.278}]},{"osd":1,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.254,"15min":1.302},"min":{"1min":0.81799999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":3.7850000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2170000000000001,"15min":1.234},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.57899999999999996},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":4.6760000000000002},"last":1.0880000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2010000000000001,"15min":1.2310000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.69199999999999995},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":13.731999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.399,"5min":1.3620000000000001,"15min":1.367},"min":{"1min":0.91900000000000004,"5min":0.79400000000000004,"15min":0.73199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7729999999999999,"15min":2.2570000000000001},"last":1.319}]},{"osd":3,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.341,"15min":1.341},"min":{"1min":1.0009999999999999,"5min":0.69999999999999996,"15min":0.59899999999999998},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":4.5789999999999997},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3280000000000001,"15min":1.363},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.71699999999999997},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":13.914},"last":1.0409999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.329,"15min":1.3580000000000001},"min":{"1min":0.90200000000000002,"5min":0.80500000000000005,"15min":0.47299999999999998},"max":{"1min":1.911,"5min":2.0190000000000001,"15min":10.211},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3720000000000001,"15min":1.4079999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.58299999999999996},"max":{"1min":1.6399999999999999,"5min":1.7669999999999999,"15min":4.673},"last":1.355}]},{"osd":6,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3779999999999999,"15min":1.4119999999999999},"min":{"1min":0.97299999999999998,"5min":0.83099999999999996,"15min":0.60699999999999998},"max":{"1min":1.74,"5min":1.7410000000000001,"15min":13.973000000000001},"last":1.21},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.5349999999999999},"min":{"1min":1.246,"5min":0.85099999999999998,"15min":0.52400000000000002},"max":{"1min":1.8520000000000001,"5min":1.9019999999999999,"15min":3.0979999999999999},"last":1.464}]},{"osd":7,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.5009999999999999,"15min":1.536},"min":{"1min":0.878,"5min":0.878,"15min":0.53500000000000003},"max":{"1min":1.7709999999999999,"5min":2.0539999999999998,"15min":4.7640000000000002},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.502,"5min":1.5229999999999999,"15min":1.5669999999999999},"min":{"1min":0.97699999999999998,"5min":0.95999999999999996,"15min":0.55200000000000005},"max":{"1min":1.7989999999999999,"5min":2.0179999999999998,"15min":13.852},"last":1.5649999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542471,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1849404,"kb_used_data":1751668,"kb_used_omap":1,"kb_used_meta":97726,"kb_avail":92522436,"statfs":{"total":96636764160,"available":94742974464,"internally_reserved":0,"allocated":1793708032,"data_stored":1792851922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":100071879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94699999999999995,"15min":0.95599999999999996},"min":{"1min":0.71299999999999997,"5min":0.626,"15min":0.34300000000000003},"max":{"1min":1.353,"5min":2.605,"15min":2.9129999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1739999999999999,"15min":1.143},"min":{"1min":0.438,"5min":0.438,"15min":0.29699999999999999},"max":{"1min":1.2849999999999999,"5min":4.0389999999999997,"15min":12.292999999999999},"last":0.52500000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.069,"15min":0.98899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.33100000000000002},"max":{"1min":1.5580000000000001,"5min":12.335000000000001,"15min":12.335000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0960000000000001,"15min":1.133},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.315},"max":{"1min":1.712,"5min":4.6529999999999996,"15min":10.638},"last":0.54700000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.206,"15min":1.151},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":8.3740000000000006,"15min":8.3740000000000006},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.248,"15min":1.228},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.46000000000000002},"max":{"1min":1.6200000000000001,"5min":4.2629999999999999,"15min":4.2629999999999999},"last":0.624}]},{"osd":4,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.403,"15min":1.345},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.39300000000000002},"max":{"1min":1.76,"5min":15.512,"15min":15.512},"last":0.876},{"interface":"front","average":{"1min":1.274,"5min":1.411,"15min":1.4119999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.52100000000000002},"max":{"1min":1.9399999999999999,"5min":11.743,"15min":14.250999999999999},"last":0.83699999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4710000000000001,"15min":1.417},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7470000000000001,"5min":8.4290000000000003,"15min":8.4290000000000003},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.419,"15min":1.409},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.42099999999999999},"max":{"1min":1.8440000000000001,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":1.0009999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3839999999999999,"15min":1.419},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.41499999999999998},"max":{"1min":1.8879999999999999,"5min":3.956,"15min":12.675000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.446,"5min":1.5669999999999999,"15min":1.49},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.54900000000000004},"max":{"1min":1.849,"5min":11.667,"15min":11.667},"last":1.02}]},{"osd":7,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.569,"15min":1.5569999999999999},"min":{"1min":1.113,"5min":1.097,"15min":0.53700000000000003},"max":{"1min":1.9279999999999999,"5min":11.042999999999999,"15min":11.042999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.474,"15min":1.5289999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.435},"max":{"1min":1.954,"5min":3.907,"15min":12.836},"last":0.98699999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542471,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1211492,"kb_used_data":1140124,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93160348,"statfs":{"total":96636764160,"available":95396196352,"internally_reserved":0,"allocated":1167486976,"data_stored":1166655701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89200000000000002,"15min":0.89400000000000002},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.311},"max":{"1min":1.6759999999999999,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":1.331},{"interface":"front","average":{"1min":0.995,"5min":1.0449999999999999,"15min":1.056},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.496,"5min":1.5549999999999999,"15min":3.52},"last":0.64100000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1599999999999999,"15min":1.0880000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.504,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.1060000000000001,"15min":1.1080000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.7729999999999999,"5min":1.8109999999999999,"15min":2.9900000000000002},"last":1.1779999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.036,"15min":1.048},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.45700000000000002},"max":{"1min":1.8320000000000001,"5min":1.8360000000000001,"15min":6.1260000000000003},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.214,"15min":1.226},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.47999999999999998},"max":{"1min":1.6950000000000001,"5min":2.911,"15min":6.2510000000000003},"last":1.452}]},{"osd":4,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.423,"15min":1.423},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.68400000000000005},"max":{"1min":1.7989999999999999,"5min":3.012,"15min":3.012},"last":1.591},{"interface":"front","average":{"1min":1.327,"5min":1.3939999999999999,"15min":1.3959999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.71099999999999997},"max":{"1min":2.4020000000000001,"5min":3.54,"15min":3.54},"last":1.401}]},{"osd":5,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3660000000000001,"15min":1.389},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":1.774,"5min":3.6179999999999999,"15min":8.625},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.944,"5min":1.944,"15min":6.3499999999999996},"last":1.272}]},{"osd":6,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3839999999999999,"15min":1.377},"min":{"1min":1.0429999999999999,"5min":0.96299999999999997,"15min":0.71499999999999997},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":6.4729999999999999},"last":1.365},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.407,"15min":1.4370000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.71099999999999997},"max":{"1min":2.125,"5min":2.8260000000000001,"15min":3.2469999999999999},"last":1.52}]},{"osd":7,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4750000000000001,"15min":1.4910000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.704,"5min":3.431,"15min":3.7290000000000001},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.478,"15min":1.492},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0379999999999998,"5min":2.3050000000000002,"15min":6.4249999999999998},"last":1.4830000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412264,"kb_used_data":1336736,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92959576,"statfs":{"total":96636764160,"available":95190605824,"internally_reserved":0,"allocated":1368817664,"data_stored":1367982226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.94499999999999995,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.33400000000000002},"max":{"1min":1.3120000000000001,"5min":1.4079999999999999,"15min":8.8499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.147,"15min":1.1619999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"max":{"1min":1.5089999999999999,"5min":1.5940000000000001,"15min":8.9380000000000006},"last":0.73499999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.34899999999999998},"max":{"1min":1.2549999999999999,"5min":1.456,"15min":3.4460000000000002},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.123,"15min":1.147},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5840000000000001,"5min":1.677,"15min":2.9660000000000002},"last":0.80100000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.2050000000000001,"15min":1.234},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.45400000000000001},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.282},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.121,"5min":1.1180000000000001,"15min":1.135},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.47099999999999997},"max":{"1min":1.655,"5min":1.6830000000000001,"15min":8.6910000000000007},"last":0.83999999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.335,"15min":1.3600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.716,"5min":1.8109999999999999,"15min":4.2480000000000002},"last":1.236},{"interface":"front","average":{"1min":1.226,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.57899999999999996},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":4.7640000000000002},"last":0.93999999999999995}]},{"osd":5,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.50800000000000001},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3680000000000003},"last":1.119},{"interface":"front","average":{"1min":1.421,"5min":1.452,"15min":1.476},"min":{"1min":1.0569999999999999,"5min":0.77500000000000002,"15min":0.66300000000000003},"max":{"1min":1.772,"5min":1.841,"15min":5.4610000000000003},"last":1.347}]},{"osd":6,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":4.7779999999999996},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.365,"15min":1.419},"min":{"1min":0.746,"5min":0.746,"15min":0.56899999999999995},"max":{"1min":2.339,"5min":2.339,"15min":4.8300000000000001},"last":1.3879999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4510000000000001,"15min":1.4830000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.58999999999999997},"max":{"1min":2.0510000000000002,"5min":2.081,"15min":6.7990000000000004},"last":1.403},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.484,"15min":1.544},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.198,"5min":2.198,"15min":5.1260000000000003},"last":1.298}]}]},{"osd":0,"up_from":14,"seq":60129542472,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2036608,"kb_used_data":1965240,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92335232,"statfs":{"total":96636764160,"available":94551277568,"internally_reserved":0,"allocated":2012405760,"data_stored":2011545811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.34899999999999998},"max":{"1min":1.381,"5min":2.387,"15min":2.7320000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.0269999999999999,"15min":1.0680000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.29399999999999998},"max":{"1min":1.3720000000000001,"5min":1.694,"15min":9.4499999999999993},"last":0.64700000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2490000000000001,"15min":1.3029999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.45100000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":9.5579999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.119,"5min":1.149,"15min":1.1459999999999999},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.40200000000000002},"max":{"1min":1.5960000000000001,"5min":2.4359999999999999,"15min":5.3159999999999998},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0489999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46600000000000003},"max":{"1min":1.335,"5min":1.5940000000000001,"15min":10.83},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1659999999999999,"15min":1.2529999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":1.5409999999999999,"5min":1.99,"15min":9.6029999999999998},"last":0.46899999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3260000000000001,"15min":1.306},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.63400000000000001},"max":{"1min":1.641,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.67200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":10.986000000000001},"last":0.78400000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":1.0089999999999999,"5min":0.78700000000000003,"15min":0.70499999999999996},"max":{"1min":1.665,"5min":3.0259999999999998,"15min":3.0259999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":1.514,"5min":1.667,"15min":14.561999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.47,"15min":1.522},"min":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.76600000000000001},"max":{"1min":1.6639999999999999,"5min":2.2069999999999999,"15min":14.958},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.506,"15min":1.48},"min":{"1min":1.069,"5min":0.89500000000000002,"15min":0.68500000000000005},"max":{"1min":1.7270000000000001,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":0.99099999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4890000000000001,"15min":1.5640000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.7869999999999999,"5min":2.129,"15min":15.005000000000001},"last":1.01},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5089999999999999,"15min":1.552},"min":{"1min":1.028,"5min":0.93400000000000005,"15min":0.70699999999999996},"max":{"1min":1.9279999999999999,"5min":3.0590000000000002,"15min":13.718},"last":0.83599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1248033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1247991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2009083904,"data_stored":2009072258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166020608,"data_stored":1166016703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908416512,"data_stored":1908408760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132466176,"data_stored":1132462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:24:05.321 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.321 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.321 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.321 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.322 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.322 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.322 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.322 INFO:tasks.ceph:pgid 17.38 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.322 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.322 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.323 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.323 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.323 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.323 INFO:tasks.ceph:pgid 17.32 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.323 INFO:tasks.ceph:pgid 17.31 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.323 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.323 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.324 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.324 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.324 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.324 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.324 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.324 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.325 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.325 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.325 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.325 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.325 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.325 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.326 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.326 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.326 INFO:tasks.ceph:pgid 17.20 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.326 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.326 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.326 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.326 INFO:tasks.ceph:pgid 16.1e last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.327 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.327 INFO:tasks.ceph:pgid 16.1d last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.327 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.327 INFO:tasks.ceph:pgid 16.1c last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.327 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.327 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.328 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.328 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.328 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.328 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.328 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.328 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.329 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.329 INFO:tasks.ceph:pgid 16.16 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.329 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.329 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.329 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.329 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.329 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.330 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.330 INFO:tasks.ceph:pgid 16.12 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.330 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.330 INFO:tasks.ceph:pgid 16.10 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.330 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.330 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.331 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.331 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.331 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.331 INFO:tasks.ceph:pgid 16.d last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.331 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.331 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.332 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.332 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.332 INFO:tasks.ceph:pgid 16.a last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.332 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.332 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.332 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.333 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.333 INFO:tasks.ceph:pgid 16.7 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.333 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.333 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.333 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.333 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.333 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.334 INFO:tasks.ceph:pgid 16.4 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.334 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.334 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.334 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.334 INFO:tasks.ceph:pgid 16.2 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.334 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.334 INFO:tasks.ceph:pgid 16.1 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.335 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.335 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.335 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-01-16T15:15:37.240620+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=37, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.335 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-16T14:57:01.176684+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=14, tm_min=57, tm_sec=1, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:05.335 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:24:10.165 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:10.173 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:25.336 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:24:25.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.560+0000 7f867a800640 1 Processor -- start 2024-01-16T15:24:25.606 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.560+0000 7f867a800640 1 -- start start 2024-01-16T15:24:25.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.562+0000 7f867a800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 0x7f867415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.562+0000 7f867a800640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8674070c40 con 0x7f86740cf290 2024-01-16T15:24:25.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.562+0000 7f867a800640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867406c470 con 0x7f86740ae5a0 2024-01-16T15:24:25.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.562+0000 7f867a800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8674071450 con 0x7f86740b3ae0 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.562+0000 7f8673e00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f86740cf290 0x7f86740b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:33688/0 (socket says 172.21.15.141:33688) 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.562+0000 7f8673e00640 1 -- 172.21.15.141:0/387143697 learned_addr learned my addr 172.21.15.141:0/387143697 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672a00640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 0x7f867415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083379397 0 0) 0x7f8674070c40 con 0x7f86740cf290 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8654003520 con 0x7f86740cf290 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306194531 0 0) 0x7f8654003520 con 0x7f86740cf290 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 msgr2=0x7f867415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:25.613 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 0x7f867415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 >> v1:172.21.15.188:6789/0 conn(0x7f86740ae5a0 legacy=0x7f867406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867406c750 con 0x7f86740cf290 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f866c0019f0 con 0x7f86740cf290 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 >> v1:172.21.15.141:6789/0 conn(0x7f86740cf290 legacy=0x7f86740b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 0x7f86540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8654003fd0 0x7f86540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8654004d00 0x7f8654005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8674071450 con 0x7f8654003fd0 2024-01-16T15:24:25.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865400bbd0 con 0x7f8654004d00 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.563+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865400bd90 con 0x7f86740b3ae0 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.564+0000 7f8672a00640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8654003fd0 0x7f86540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.564+0000 7f8673400640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 0x7f86540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.564+0000 7f8672a00640 1 -- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 msgr2=0x7f86540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.564+0000 7f8672a00640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f86740b3ae0 0x7f86540039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.564+0000 7f8672a00640 1 -- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8654004d00 msgr2=0x7f8654005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.564+0000 7f8672a00640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8654004d00 0x7f8654005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.564+0000 7f8672a00640 1 -- 172.21.15.141:0/387143697 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8654003520 con 0x7f8654003fd0 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.565+0000 7f8672a00640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8654003fd0 0x7f86540047c0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f865c00ba30 tx=0x7f86540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d580ad648ffa7ba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.565+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c032640 con 0x7f8654003fd0 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.565+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c0327a0 con 0x7f8654003fd0 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.565+0000 7f8672000640 1 -- 172.21.15.141:0/387143697 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c032b30 con 0x7f8654003fd0 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.566+0000 7f867a800640 1 -- 172.21.15.141:0/387143697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8654003fd0 msgr2=0x7f86540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.566+0000 7f867a800640 1 --2- 172.21.15.141:0/387143697 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8654003fd0 0x7f86540047c0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f865c00ba30 tx=0x7f86540057f0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.566+0000 7f8673e00640 1 -- 172.21.15.141:0/387143697 reap_dead start 2024-01-16T15:24:25.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.566+0000 7f867a800640 1 -- 172.21.15.141:0/387143697 shutdown_connections 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.566+0000 7f867a800640 1 -- 172.21.15.141:0/387143697 >> 172.21.15.141:0/387143697 conn(0x7f867406a800 msgr2=0x7f867406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.566+0000 7f867a800640 1 -- 172.21.15.141:0/387143697 shutdown_connections 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.566+0000 7f867a800640 1 -- 172.21.15.141:0/387143697 wait complete. 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.568+0000 7f867a800640 1 Processor -- start 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.568+0000 7f867a800640 1 -- start start 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f867a800640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8654003fd0 0x7f86740ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f867a800640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f867414ada0 0x7f867414d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.617 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f867a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86740b2450 0x7f86740b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f867a800640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86740b2e10 con 0x7f867414ada0 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f867a800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86741475f0 con 0x7f86740b2450 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f867a800640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867414b200 con 0x7f8654003fd0 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f8672a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f867414ada0 0x7f867414d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f8672a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f867414ada0 0x7f867414d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:39770/0 (socket says 172.21.15.141:39770) 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f8672a00640 1 -- 172.21.15.141:0/1231327367 learned_addr learned my addr 172.21.15.141:0/1231327367 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.569+0000 7f8673400640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8654003fd0 0x7f86740ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f8673e00640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86740b2450 0x7f86740b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:25.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f8672a00640 1 -- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8654003fd0 msgr2=0x7f86740ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f8672a00640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8654003fd0 0x7f86740ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f8672a00640 1 -- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86740b2450 msgr2=0x7f86740b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f8672a00640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86740b2450 0x7f86740b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f8672a00640 1 -- 172.21.15.141:0/1231327367 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86741479e0 con 0x7f867414ada0 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f8672a00640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f867414ada0 0x7f867414d600 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f865c02bec0 tx=0x7f865c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f866b400640 1 -- 172.21.15.141:0/1231327367 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c03ba70 con 0x7f867414ada0 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f866b400640 1 -- 172.21.15.141:0/1231327367 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c03bbd0 con 0x7f867414ada0 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8674147cc0 con 0x7f867414ada0 2024-01-16T15:24:25.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f866b400640 1 -- 172.21.15.141:0/1231327367 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c030410 con 0x7f867414ada0 2024-01-16T15:24:25.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.570+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8674148280 con 0x7f867414ada0 2024-01-16T15:24:25.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.571+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86740675d0 con 0x7f867414ada0 2024-01-16T15:24:25.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.572+0000 7f866b400640 1 -- 172.21.15.141:0/1231327367 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f865c030570 con 0x7f867414ada0 2024-01-16T15:24:25.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.573+0000 7f866b400640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864c06c610 0x7f864c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:25.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.573+0000 7f866b400640 1 -- 172.21.15.141:0/1231327367 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f865c0ab020 con 0x7f867414ada0 2024-01-16T15:24:25.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.574+0000 7f8673400640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864c06c610 0x7f864c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:25.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.575+0000 7f8673400640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864c06c610 0x7f864c06ead0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8664001710 tx=0x7f8664009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:25.621 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.578+0000 7f866b400640 1 -- 172.21.15.141:0/1231327367 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f865c0b4020 con 0x7f867414ada0 2024-01-16T15:24:25.760 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.763+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8674070550 con 0x7f864c06c610 2024-01-16T15:24:25.796 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.800+0000 7f866b400640 1 -- 172.21.15.141:0/1231327367 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288815 (secure 0 0 0) 0x7f866400d050 con 0x7f864c06c610 2024-01-16T15:24:25.801 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:24:25.803 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:24:25.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864c06c610 msgr2=0x7f864c06ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:25.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864c06c610 0x7f864c06ead0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8664001710 tx=0x7f8664009040 comp rx=0 tx=0).stop 2024-01-16T15:24:25.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f867414ada0 msgr2=0x7f867414d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:25.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f867414ada0 0x7f867414d600 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f865c02bec0 tx=0x7f865c004730 comp rx=0 tx=0).stop 2024-01-16T15:24:25.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 shutdown_connections 2024-01-16T15:24:25.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f864c06c610 0x7f864c06ead0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.805 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86740b2450 0x7f86740b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f867414ada0 0x7f867414d600 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 --2- 172.21.15.141:0/1231327367 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f8654003fd0 0x7f86740ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:25.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 >> 172.21.15.141:0/1231327367 conn(0x7f867406a800 msgr2=0x7f867415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:25.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 shutdown_connections 2024-01-16T15:24:25.806 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:25.807+0000 7f867a800640 1 -- 172.21.15.141:0/1231327367 wait complete. 2024-01-16T15:24:25.826 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":934,"stamp":"2024-01-16T15:24:25.132375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444295670,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":945,"num_read_kb":3145889,"num_write":2253,"num_write_kb":6295371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5644,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5118,"ondisk_log_size":5118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13178736,"kb_used_data":12598064,"kb_used_omap":12,"kb_used_meta":580595,"kb_avail":741795984,"statfs":{"total":773094113280,"available":759599087616,"internally_reserved":0,"allocated":12900417536,"data_stored":12893672556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":594529842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644,"num_omap_keys":12,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004874"},"pg_stats":[{"pgid":"17.3f","version":"118'70","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900432+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:23:59.900432+0000","last_peered":"2024-01-16T15:23:59.900432+0000","last_clean":"2024-01-16T15:23:59.900432+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:23:59.900432+0000","last_undegraded":"2024-01-16T15:23:59.900432+0000","last_fullsized":"2024-01-16T15:23:59.900432+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"118'69","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017572+0000","last_change":"2024-01-16T15:15:39.327644+0000","last_active":"2024-01-16T15:24:00.017572+0000","last_peered":"2024-01-16T15:24:00.017572+0000","last_clean":"2024-01-16T15:24:00.017572+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:24:00.017572+0000","last_undegraded":"2024-01-16T15:24:00.017572+0000","last_fullsized":"2024-01-16T15:24:00.017572+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"118'67","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.189812+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:24:00.189812+0000","last_peered":"2024-01-16T15:24:00.189812+0000","last_clean":"2024-01-16T15:24:00.189812+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:24:00.189812+0000","last_undegraded":"2024-01-16T15:24:00.189812+0000","last_fullsized":"2024-01-16T15:24:00.189812+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"118'66","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.927494+0000","last_change":"2024-01-16T15:15:55.875926+0000","last_active":"2024-01-16T15:23:59.927494+0000","last_peered":"2024-01-16T15:23:59.927494+0000","last_clean":"2024-01-16T15:23:59.927494+0000","last_became_active":"2024-01-16T15:15:55.875179+0000","last_became_peered":"2024-01-16T15:15:55.875179+0000","last_unstale":"2024-01-16T15:23:59.927494+0000","last_undegraded":"2024-01-16T15:23:59.927494+0000","last_fullsized":"2024-01-16T15:23:59.927494+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3b","version":"118'60","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737970+0000","last_change":"2024-01-16T15:15:39.321081+0000","last_active":"2024-01-16T15:23:49.737970+0000","last_peered":"2024-01-16T15:23:49.737970+0000","last_clean":"2024-01-16T15:23:49.737970+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:23:49.737970+0000","last_undegraded":"2024-01-16T15:23:49.737970+0000","last_fullsized":"2024-01-16T15:23:49.737970+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663487,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"118'104","reported_seq":148,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:23.051913+0000","last_change":"2024-01-16T15:24:23.051959+0000","last_active":"2024-01-16T15:24:23.051913+0000","last_peered":"2024-01-16T15:24:23.051913+0000","last_clean":"2024-01-16T15:24:23.051913+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:24:23.051913+0000","last_undegraded":"2024-01-16T15:24:23.051913+0000","last_fullsized":"2024-01-16T15:24:23.051913+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'104","last_scrub_stamp":"2024-01-16T15:24:23.051823+0000","last_deep_scrub":"118'104","last_deep_scrub_stamp":"2024-01-16T15:24:23.051823+0000","last_clean_scrub_stamp":"2024-01-16T15:24:23.051823+0000","objects_scrubbed":31,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:26.121405+0000","scrub_duration":0.37272100699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"118'66","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751912+0000","last_change":"2024-01-16T15:15:39.324707+0000","last_active":"2024-01-16T15:23:49.751912+0000","last_peered":"2024-01-16T15:23:49.751912+0000","last_clean":"2024-01-16T15:23:49.751912+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:23:49.751912+0000","last_undegraded":"2024-01-16T15:23:49.751912+0000","last_fullsized":"2024-01-16T15:23:49.751912+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303428,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"118'71","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.696314+0000","last_change":"2024-01-16T15:24:17.696409+0000","last_active":"2024-01-16T15:24:17.696314+0000","last_peered":"2024-01-16T15:24:17.696314+0000","last_clean":"2024-01-16T15:24:17.696314+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:24:17.696314+0000","last_undegraded":"2024-01-16T15:24:17.696314+0000","last_fullsized":"2024-01-16T15:24:17.696314+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'71","last_scrub_stamp":"2024-01-16T15:24:17.696140+0000","last_deep_scrub":"118'71","last_deep_scrub_stamp":"2024-01-16T15:24:17.696140+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.696140+0000","objects_scrubbed":22,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:05.520208+0000","scrub_duration":0.28034184200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"118'79","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758978+0000","last_change":"2024-01-16T15:15:39.320236+0000","last_active":"2024-01-16T15:23:49.758978+0000","last_peered":"2024-01-16T15:23:49.758978+0000","last_clean":"2024-01-16T15:23:49.758978+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:23:49.758978+0000","last_undegraded":"2024-01-16T15:23:49.758978+0000","last_fullsized":"2024-01-16T15:23:49.758978+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052164,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"118'66","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758376+0000","last_change":"2024-01-16T15:15:39.326528+0000","last_active":"2024-01-16T15:23:49.758376+0000","last_peered":"2024-01-16T15:23:49.758376+0000","last_clean":"2024-01-16T15:23:49.758376+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:23:49.758376+0000","last_undegraded":"2024-01-16T15:23:49.758376+0000","last_fullsized":"2024-01-16T15:23:49.758376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"118'70","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737994+0000","last_change":"2024-01-16T15:15:39.325379+0000","last_active":"2024-01-16T15:23:49.737994+0000","last_peered":"2024-01-16T15:23:49.737994+0000","last_clean":"2024-01-16T15:23:49.737994+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:23:49.737994+0000","last_undegraded":"2024-01-16T15:23:49.737994+0000","last_fullsized":"2024-01-16T15:23:49.737994+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"118'59","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758102+0000","last_change":"2024-01-16T15:15:39.317978+0000","last_active":"2024-01-16T15:23:49.758102+0000","last_peered":"2024-01-16T15:23:49.758102+0000","last_clean":"2024-01-16T15:23:49.758102+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:23:49.758102+0000","last_undegraded":"2024-01-16T15:23:49.758102+0000","last_fullsized":"2024-01-16T15:23:49.758102+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"118'57","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.746213+0000","last_change":"2024-01-16T15:15:39.329570+0000","last_active":"2024-01-16T15:23:49.746213+0000","last_peered":"2024-01-16T15:23:49.746213+0000","last_clean":"2024-01-16T15:23:49.746213+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:23:49.746213+0000","last_undegraded":"2024-01-16T15:23:49.746213+0000","last_fullsized":"2024-01-16T15:23:49.746213+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"118'88","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:12.757274+0000","last_change":"2024-01-16T15:24:12.757322+0000","last_active":"2024-01-16T15:24:12.757274+0000","last_peered":"2024-01-16T15:24:12.757274+0000","last_clean":"2024-01-16T15:24:12.757274+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:24:12.757274+0000","last_undegraded":"2024-01-16T15:24:12.757274+0000","last_fullsized":"2024-01-16T15:24:12.757274+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:12.757133+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:12.757133+0000","last_clean_scrub_stamp":"2024-01-16T15:24:12.757133+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:23:26.026719+0000","scrub_duration":0.337903377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"118'54","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.741679+0000","last_change":"2024-01-16T15:15:39.333121+0000","last_active":"2024-01-16T15:23:49.741679+0000","last_peered":"2024-01-16T15:23:49.741679+0000","last_clean":"2024-01-16T15:23:49.741679+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:23:49.741679+0000","last_undegraded":"2024-01-16T15:23:49.741679+0000","last_fullsized":"2024-01-16T15:23:49.741679+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"118'58","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.757965+0000","last_change":"2024-01-16T15:15:39.313940+0000","last_active":"2024-01-16T15:23:49.757965+0000","last_peered":"2024-01-16T15:23:49.757965+0000","last_clean":"2024-01-16T15:23:49.757965+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:23:49.757965+0000","last_undegraded":"2024-01-16T15:23:49.757965+0000","last_fullsized":"2024-01-16T15:23:49.757965+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"118'49","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.740295+0000","last_change":"2024-01-16T15:15:39.329528+0000","last_active":"2024-01-16T15:23:49.740295+0000","last_peered":"2024-01-16T15:23:49.740295+0000","last_clean":"2024-01-16T15:23:49.740295+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:23:49.740295+0000","last_undegraded":"2024-01-16T15:23:49.740295+0000","last_fullsized":"2024-01-16T15:23:49.740295+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"118'72","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.742246+0000","last_change":"2024-01-16T15:15:39.330676+0000","last_active":"2024-01-16T15:23:49.742246+0000","last_peered":"2024-01-16T15:23:49.742246+0000","last_clean":"2024-01-16T15:23:49.742246+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:23:49.742246+0000","last_undegraded":"2024-01-16T15:23:49.742246+0000","last_fullsized":"2024-01-16T15:23:49.742246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"118'71","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758159+0000","last_change":"2024-01-16T15:15:39.330437+0000","last_active":"2024-01-16T15:23:49.758159+0000","last_peered":"2024-01-16T15:23:49.758159+0000","last_clean":"2024-01-16T15:23:49.758159+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:23:49.758159+0000","last_undegraded":"2024-01-16T15:23:49.758159+0000","last_fullsized":"2024-01-16T15:23:49.758159+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"118'67","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751037+0000","last_change":"2024-01-16T15:15:39.318689+0000","last_active":"2024-01-16T15:23:49.751037+0000","last_peered":"2024-01-16T15:23:49.751037+0000","last_clean":"2024-01-16T15:23:49.751037+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:23:49.751037+0000","last_undegraded":"2024-01-16T15:23:49.751037+0000","last_fullsized":"2024-01-16T15:23:49.751037+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"118'99","reported_seq":147,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.587011+0000","last_change":"2024-01-16T15:24:15.587081+0000","last_active":"2024-01-16T15:24:15.587011+0000","last_peered":"2024-01-16T15:24:15.587011+0000","last_clean":"2024-01-16T15:24:15.587011+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:24:15.587011+0000","last_undegraded":"2024-01-16T15:24:15.587011+0000","last_fullsized":"2024-01-16T15:24:15.587011+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2024-01-16T15:24:15.586844+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2024-01-16T15:24:15.586844+0000","last_clean_scrub_stamp":"2024-01-16T15:24:15.586844+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:57.424369+0000","scrub_duration":0.353635808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"118'64","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749573+0000","last_change":"2024-01-16T15:15:39.323290+0000","last_active":"2024-01-16T15:23:49.749573+0000","last_peered":"2024-01-16T15:23:49.749573+0000","last_clean":"2024-01-16T15:23:49.749573+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:23:49.749573+0000","last_undegraded":"2024-01-16T15:23:49.749573+0000","last_fullsized":"2024-01-16T15:23:49.749573+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"118'79","reported_seq":116,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.554673+0000","last_change":"2024-01-16T15:24:14.554719+0000","last_active":"2024-01-16T15:24:14.554673+0000","last_peered":"2024-01-16T15:24:14.554673+0000","last_clean":"2024-01-16T15:24:14.554673+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:24:14.554673+0000","last_undegraded":"2024-01-16T15:24:14.554673+0000","last_fullsized":"2024-01-16T15:24:14.554673+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'79","last_scrub_stamp":"2024-01-16T15:24:14.554529+0000","last_deep_scrub":"118'79","last_deep_scrub_stamp":"2024-01-16T15:24:14.554529+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.554529+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:06.890804+0000","scrub_duration":0.31410593999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"118'94","reported_seq":130,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:21.246704+0000","last_change":"2024-01-16T15:24:21.246810+0000","last_active":"2024-01-16T15:24:21.246704+0000","last_peered":"2024-01-16T15:24:21.246704+0000","last_clean":"2024-01-16T15:24:21.246704+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:24:21.246704+0000","last_undegraded":"2024-01-16T15:24:21.246704+0000","last_fullsized":"2024-01-16T15:24:21.246704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:21.246488+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:21.246488+0000","last_clean_scrub_stamp":"2024-01-16T15:24:21.246488+0000","objects_scrubbed":24,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:51.356431+0000","scrub_duration":0.27909722799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"118'71","reported_seq":103,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.693973+0000","last_change":"2024-01-16T15:15:39.328896+0000","last_active":"2024-01-16T15:24:02.693973+0000","last_peered":"2024-01-16T15:24:02.693973+0000","last_clean":"2024-01-16T15:24:02.693973+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:24:02.693973+0000","last_undegraded":"2024-01-16T15:24:02.693973+0000","last_fullsized":"2024-01-16T15:24:02.693973+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"118'81","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900448+0000","last_change":"2024-01-16T15:15:39.317589+0000","last_active":"2024-01-16T15:23:59.900448+0000","last_peered":"2024-01-16T15:23:59.900448+0000","last_clean":"2024-01-16T15:23:59.900448+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:23:59.900448+0000","last_undegraded":"2024-01-16T15:23:59.900448+0000","last_fullsized":"2024-01-16T15:23:59.900448+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"118'64","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.582215+0000","last_change":"2024-01-16T15:15:39.335455+0000","last_active":"2024-01-16T15:24:00.582215+0000","last_peered":"2024-01-16T15:24:00.582215+0000","last_clean":"2024-01-16T15:24:00.582215+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:24:00.582215+0000","last_undegraded":"2024-01-16T15:24:00.582215+0000","last_fullsized":"2024-01-16T15:24:00.582215+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"118'43","reported_seq":66,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.981547+0000","last_change":"2024-01-16T15:15:39.317292+0000","last_active":"2024-01-16T15:24:01.981547+0000","last_peered":"2024-01-16T15:24:01.981547+0000","last_clean":"2024-01-16T15:24:01.981547+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:24:01.981547+0000","last_undegraded":"2024-01-16T15:24:01.981547+0000","last_fullsized":"2024-01-16T15:24:01.981547+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"118'57","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017399+0000","last_change":"2024-01-16T15:15:39.320340+0000","last_active":"2024-01-16T15:24:00.017399+0000","last_peered":"2024-01-16T15:24:00.017399+0000","last_clean":"2024-01-16T15:24:00.017399+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:24:00.017399+0000","last_undegraded":"2024-01-16T15:24:00.017399+0000","last_fullsized":"2024-01-16T15:24:00.017399+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"118'76","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.919383+0000","last_change":"2024-01-16T15:15:39.317544+0000","last_active":"2024-01-16T15:23:59.919383+0000","last_peered":"2024-01-16T15:23:59.919383+0000","last_clean":"2024-01-16T15:23:59.919383+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:23:59.919383+0000","last_undegraded":"2024-01-16T15:23:59.919383+0000","last_fullsized":"2024-01-16T15:23:59.919383+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"118'81","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900428+0000","last_change":"2024-01-16T15:15:39.313788+0000","last_active":"2024-01-16T15:23:59.900428+0000","last_peered":"2024-01-16T15:23:59.900428+0000","last_clean":"2024-01-16T15:23:59.900428+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:23:59.900428+0000","last_undegraded":"2024-01-16T15:23:59.900428+0000","last_fullsized":"2024-01-16T15:23:59.900428+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"118'82","reported_seq":119,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.992127+0000","last_change":"2024-01-16T15:24:10.992228+0000","last_active":"2024-01-16T15:24:10.992127+0000","last_peered":"2024-01-16T15:24:10.992127+0000","last_clean":"2024-01-16T15:24:10.992127+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:24:10.992127+0000","last_undegraded":"2024-01-16T15:24:10.992127+0000","last_fullsized":"2024-01-16T15:24:10.992127+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'82","last_scrub_stamp":"2024-01-16T15:24:10.991908+0000","last_deep_scrub":"118'82","last_deep_scrub_stamp":"2024-01-16T15:24:10.991908+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.991908+0000","objects_scrubbed":23,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:34.358631+0000","scrub_duration":0.279512233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"118'62","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.733839+0000","last_change":"2024-01-16T15:15:39.329833+0000","last_active":"2024-01-16T15:23:49.733839+0000","last_peered":"2024-01-16T15:23:49.733839+0000","last_clean":"2024-01-16T15:23:49.733839+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:23:49.733839+0000","last_undegraded":"2024-01-16T15:23:49.733839+0000","last_fullsized":"2024-01-16T15:23:49.733839+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"118'6","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.468778+0000","last_change":"2024-01-16T15:24:08.468844+0000","last_active":"2024-01-16T15:24:08.468778+0000","last_peered":"2024-01-16T15:24:08.468778+0000","last_clean":"2024-01-16T15:24:08.468778+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:24:08.468778+0000","last_undegraded":"2024-01-16T15:24:08.468778+0000","last_fullsized":"2024-01-16T15:24:08.468778+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6","last_scrub_stamp":"2024-01-16T15:24:08.468615+0000","last_deep_scrub":"118'6","last_deep_scrub_stamp":"2024-01-16T15:24:08.468615+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.468615+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:16.033835+0000","scrub_duration":0.0085012860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"118'73","reported_seq":112,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:19.646515+0000","last_change":"2024-01-16T15:24:19.646587+0000","last_active":"2024-01-16T15:24:19.646515+0000","last_peered":"2024-01-16T15:24:19.646515+0000","last_clean":"2024-01-16T15:24:19.646515+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:24:19.646515+0000","last_undegraded":"2024-01-16T15:24:19.646515+0000","last_fullsized":"2024-01-16T15:24:19.646515+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'73","last_scrub_stamp":"2024-01-16T15:24:19.646340+0000","last_deep_scrub":"118'73","last_deep_scrub_stamp":"2024-01-16T15:24:19.646340+0000","last_clean_scrub_stamp":"2024-01-16T15:24:19.646340+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:37.373916+0000","scrub_duration":0.24430438900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.441338+0000","last_change":"2024-01-16T15:24:10.441429+0000","last_active":"2024-01-16T15:24:10.441338+0000","last_peered":"2024-01-16T15:24:10.441338+0000","last_clean":"2024-01-16T15:24:10.441338+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:24:10.441338+0000","last_undegraded":"2024-01-16T15:24:10.441338+0000","last_fullsized":"2024-01-16T15:24:10.441338+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:10.441063+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:10.441063+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:44.044132+0000","scrub_duration":0.004404593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"118'97","reported_seq":141,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.044503+0000","last_change":"2024-01-16T15:24:20.044562+0000","last_active":"2024-01-16T15:24:20.044503+0000","last_peered":"2024-01-16T15:24:20.044503+0000","last_clean":"2024-01-16T15:24:20.044503+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:24:20.044503+0000","last_undegraded":"2024-01-16T15:24:20.044503+0000","last_fullsized":"2024-01-16T15:24:20.044503+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:20.044335+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:20.044335+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.044335+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:39.481751+0000","scrub_duration":0.33193780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"118'4","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.220298+0000","last_change":"2024-01-16T15:24:16.220358+0000","last_active":"2024-01-16T15:24:16.220298+0000","last_peered":"2024-01-16T15:24:16.220298+0000","last_clean":"2024-01-16T15:24:16.220298+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:24:16.220298+0000","last_undegraded":"2024-01-16T15:24:16.220298+0000","last_fullsized":"2024-01-16T15:24:16.220298+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4","last_scrub_stamp":"2024-01-16T15:24:16.220137+0000","last_deep_scrub":"118'4","last_deep_scrub_stamp":"2024-01-16T15:24:16.220137+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.220137+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:35.174555+0000","scrub_duration":0.0071288089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"118'68","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758185+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:23:49.758185+0000","last_peered":"2024-01-16T15:23:49.758185+0000","last_clean":"2024-01-16T15:23:49.758185+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:23:49.758185+0000","last_undegraded":"2024-01-16T15:23:49.758185+0000","last_fullsized":"2024-01-16T15:23:49.758185+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"118'3","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.217316+0000","last_change":"2024-01-16T15:24:14.217375+0000","last_active":"2024-01-16T15:24:14.217316+0000","last_peered":"2024-01-16T15:24:14.217316+0000","last_clean":"2024-01-16T15:24:14.217316+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:24:14.217316+0000","last_undegraded":"2024-01-16T15:24:14.217316+0000","last_fullsized":"2024-01-16T15:24:14.217316+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:14.217126+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:14.217126+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.217126+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:21.782346+0000","scrub_duration":0.0072240430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"118'72","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.756227+0000","last_change":"2024-01-16T15:15:39.321037+0000","last_active":"2024-01-16T15:23:49.756227+0000","last_peered":"2024-01-16T15:23:49.756227+0000","last_clean":"2024-01-16T15:23:49.756227+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:23:49.756227+0000","last_undegraded":"2024-01-16T15:23:49.756227+0000","last_fullsized":"2024-01-16T15:23:49.756227+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:18.450001+0000","last_change":"2024-01-16T15:24:18.450098+0000","last_active":"2024-01-16T15:24:18.450001+0000","last_peered":"2024-01-16T15:24:18.450001+0000","last_clean":"2024-01-16T15:24:18.450001+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:24:18.450001+0000","last_undegraded":"2024-01-16T15:24:18.450001+0000","last_fullsized":"2024-01-16T15:24:18.450001+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:18.449469+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:18.449469+0000","last_clean_scrub_stamp":"2024-01-16T15:24:18.449469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:41.816192+0000","scrub_duration":0.0038895650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"118'72","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752192+0000","last_change":"2024-01-16T15:15:39.330201+0000","last_active":"2024-01-16T15:23:49.752192+0000","last_peered":"2024-01-16T15:23:49.752192+0000","last_clean":"2024-01-16T15:23:49.752192+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:23:49.752192+0000","last_undegraded":"2024-01-16T15:23:49.752192+0000","last_fullsized":"2024-01-16T15:23:49.752192+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.835879+0000","last_change":"2024-01-16T15:24:02.835904+0000","last_active":"2024-01-16T15:24:02.835879+0000","last_peered":"2024-01-16T15:24:02.835879+0000","last_clean":"2024-01-16T15:24:02.835879+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:24:02.835879+0000","last_undegraded":"2024-01-16T15:24:02.835879+0000","last_fullsized":"2024-01-16T15:24:02.835879+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.835792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:09.602675+0000","scrub_duration":0.0029040390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"118'95","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.969881+0000","last_change":"2024-01-16T15:24:00.969955+0000","last_active":"2024-01-16T15:24:00.969881+0000","last_peered":"2024-01-16T15:24:00.969881+0000","last_clean":"2024-01-16T15:24:00.969881+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:24:00.969881+0000","last_undegraded":"2024-01-16T15:24:00.969881+0000","last_fullsized":"2024-01-16T15:24:00.969881+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.969804+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:57.211104+0000","scrub_duration":0.179269503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.280477+0000","last_change":"2024-01-16T15:24:07.280530+0000","last_active":"2024-01-16T15:24:07.280477+0000","last_peered":"2024-01-16T15:24:07.280477+0000","last_clean":"2024-01-16T15:24:07.280477+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:24:07.280477+0000","last_undegraded":"2024-01-16T15:24:07.280477+0000","last_fullsized":"2024-01-16T15:24:07.280477+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:07.280247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:07.280247+0000","last_clean_scrub_stamp":"2024-01-16T15:24:07.280247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:26.717663+0000","scrub_duration":0.0039888249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"118'73","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.749206+0000","last_change":"2024-01-16T15:15:39.325224+0000","last_active":"2024-01-16T15:23:49.749206+0000","last_peered":"2024-01-16T15:23:49.749206+0000","last_clean":"2024-01-16T15:23:49.749206+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:23:49.749206+0000","last_undegraded":"2024-01-16T15:23:49.749206+0000","last_fullsized":"2024-01-16T15:23:49.749206+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.814116+0000","last_change":"2024-01-16T15:24:17.814162+0000","last_active":"2024-01-16T15:24:17.814116+0000","last_peered":"2024-01-16T15:24:17.814116+0000","last_clean":"2024-01-16T15:24:17.814116+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:24:17.814116+0000","last_undegraded":"2024-01-16T15:24:17.814116+0000","last_fullsized":"2024-01-16T15:24:17.814116+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:17.813872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:17.813872+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.813872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:11.813723+0000","scrub_duration":0.0034917469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"118'58","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.735610+0000","last_change":"2024-01-16T15:15:39.320446+0000","last_active":"2024-01-16T15:23:49.735610+0000","last_peered":"2024-01-16T15:23:49.735610+0000","last_clean":"2024-01-16T15:23:49.735610+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:23:49.735610+0000","last_undegraded":"2024-01-16T15:23:49.735610+0000","last_fullsized":"2024-01-16T15:23:49.735610+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"118'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.306271+0000","last_change":"2024-01-16T15:24:10.306417+0000","last_active":"2024-01-16T15:24:10.306271+0000","last_peered":"2024-01-16T15:24:10.306271+0000","last_clean":"2024-01-16T15:24:10.306271+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:24:10.306271+0000","last_undegraded":"2024-01-16T15:24:10.306271+0000","last_fullsized":"2024-01-16T15:24:10.306271+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'1","last_scrub_stamp":"2024-01-16T15:24:10.306050+0000","last_deep_scrub":"118'1","last_deep_scrub_stamp":"2024-01-16T15:24:10.306050+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.306050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:08.994705+0000","scrub_duration":0.0031432769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"118'78","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.670647+0000","last_change":"2024-01-16T15:24:17.670740+0000","last_active":"2024-01-16T15:24:17.670647+0000","last_peered":"2024-01-16T15:24:17.670647+0000","last_clean":"2024-01-16T15:24:17.670647+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:24:17.670647+0000","last_undegraded":"2024-01-16T15:24:17.670647+0000","last_fullsized":"2024-01-16T15:24:17.670647+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'78","last_scrub_stamp":"2024-01-16T15:24:17.670442+0000","last_deep_scrub":"118'78","last_deep_scrub_stamp":"2024-01-16T15:24:17.670442+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.670442+0000","objects_scrubbed":20,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:10:39.488294+0000","scrub_duration":0.25523936800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.949045+0000","last_change":"2024-01-16T15:24:07.949150+0000","last_active":"2024-01-16T15:24:07.949045+0000","last_peered":"2024-01-16T15:24:07.949045+0000","last_clean":"2024-01-16T15:24:07.949045+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:24:07.949045+0000","last_undegraded":"2024-01-16T15:24:07.949045+0000","last_fullsized":"2024-01-16T15:24:07.949045+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:07.948766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:07.948766+0000","last_clean_scrub_stamp":"2024-01-16T15:24:07.948766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:49.830805+0000","scrub_duration":0.0035464849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"118'88","reported_seq":130,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:23.283298+0000","last_change":"2024-01-16T15:24:23.283365+0000","last_active":"2024-01-16T15:24:23.283298+0000","last_peered":"2024-01-16T15:24:23.283298+0000","last_clean":"2024-01-16T15:24:23.283298+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:24:23.283298+0000","last_undegraded":"2024-01-16T15:24:23.283298+0000","last_fullsized":"2024-01-16T15:24:23.283298+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:23.283119+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:23.283119+0000","last_clean_scrub_stamp":"2024-01-16T15:24:23.283119+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:18.238164+0000","scrub_duration":0.31081178100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"118'33","reported_seq":59,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:24.862951+0000","last_change":"2024-01-16T15:24:24.863013+0000","last_active":"2024-01-16T15:24:24.862951+0000","last_peered":"2024-01-16T15:24:24.862951+0000","last_clean":"2024-01-16T15:24:24.862951+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:24:24.862951+0000","last_undegraded":"2024-01-16T15:24:24.862951+0000","last_fullsized":"2024-01-16T15:24:24.862951+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'33","last_scrub_stamp":"2024-01-16T15:24:24.862770+0000","last_deep_scrub":"118'33","last_deep_scrub_stamp":"2024-01-16T15:24:24.862770+0000","last_clean_scrub_stamp":"2024-01-16T15:24:24.862770+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:58.053240+0000","scrub_duration":0.0090951780000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"118'69","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.751912+0000","last_change":"2024-01-16T15:15:39.312858+0000","last_active":"2024-01-16T15:23:49.751912+0000","last_peered":"2024-01-16T15:23:49.751912+0000","last_clean":"2024-01-16T15:23:49.751912+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:23:49.751912+0000","last_undegraded":"2024-01-16T15:23:49.751912+0000","last_fullsized":"2024-01-16T15:23:49.751912+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"118'328","reported_seq":354,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:09.800108+0000","last_change":"2024-01-16T15:24:09.800168+0000","last_active":"2024-01-16T15:24:09.800108+0000","last_peered":"2024-01-16T15:24:09.800108+0000","last_clean":"2024-01-16T15:24:09.800108+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:24:09.800108+0000","last_undegraded":"2024-01-16T15:24:09.800108+0000","last_fullsized":"2024-01-16T15:24:09.800108+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'328","last_scrub_stamp":"2024-01-16T15:24:09.799941+0000","last_deep_scrub":"118'328","last_deep_scrub_stamp":"2024-01-16T15:24:09.799941+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.799941+0000","objects_scrubbed":4,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:04.754986+0000","scrub_duration":0.023713471999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1247921,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":1376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"118'54","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.737914+0000","last_change":"2024-01-16T15:15:39.312723+0000","last_active":"2024-01-16T15:23:49.737914+0000","last_peered":"2024-01-16T15:23:49.737914+0000","last_clean":"2024-01-16T15:23:49.737914+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:23:49.737914+0000","last_undegraded":"2024-01-16T15:23:49.737914+0000","last_fullsized":"2024-01-16T15:23:49.737914+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"118'5","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:03.636265+0000","last_change":"2024-01-16T15:24:03.636309+0000","last_active":"2024-01-16T15:24:03.636265+0000","last_peered":"2024-01-16T15:24:03.636265+0000","last_clean":"2024-01-16T15:24:03.636265+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:24:03.636265+0000","last_undegraded":"2024-01-16T15:24:03.636265+0000","last_fullsized":"2024-01-16T15:24:03.636265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5","last_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_deep_scrub":"118'5","last_deep_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_clean_scrub_stamp":"2024-01-16T15:24:03.636187+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:22.590604+0000","scrub_duration":0.0075747449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"118'72","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.753842+0000","last_change":"2024-01-16T15:15:39.324127+0000","last_active":"2024-01-16T15:23:49.753842+0000","last_peered":"2024-01-16T15:23:49.753842+0000","last_clean":"2024-01-16T15:23:49.753842+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:23:49.753842+0000","last_undegraded":"2024-01-16T15:23:49.753842+0000","last_fullsized":"2024-01-16T15:23:49.753842+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.384677+0000","last_change":"2024-01-16T15:24:05.384762+0000","last_active":"2024-01-16T15:24:05.384677+0000","last_peered":"2024-01-16T15:24:05.384677+0000","last_clean":"2024-01-16T15:24:05.384677+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:24:05.384677+0000","last_undegraded":"2024-01-16T15:24:05.384677+0000","last_fullsized":"2024-01-16T15:24:05.384677+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:05.384503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:05.384503+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.384503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:22.638464+0000","scrub_duration":0.0038951989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"118'89","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:24.284788+0000","last_change":"2024-01-16T15:24:24.284846+0000","last_active":"2024-01-16T15:24:24.284788+0000","last_peered":"2024-01-16T15:24:24.284788+0000","last_clean":"2024-01-16T15:24:24.284788+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:24:24.284788+0000","last_undegraded":"2024-01-16T15:24:24.284788+0000","last_fullsized":"2024-01-16T15:24:24.284788+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'89","last_scrub_stamp":"2024-01-16T15:24:24.284582+0000","last_deep_scrub":"118'89","last_deep_scrub_stamp":"2024-01-16T15:24:24.284582+0000","last_clean_scrub_stamp":"2024-01-16T15:24:24.284582+0000","objects_scrubbed":25,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:25.922775+0000","scrub_duration":0.27736433700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":40961,"num_write":27,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"118'12","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.355204+0000","last_change":"2024-01-16T15:24:06.355271+0000","last_active":"2024-01-16T15:24:06.355204+0000","last_peered":"2024-01-16T15:24:06.355204+0000","last_clean":"2024-01-16T15:24:06.355204+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:24:06.355204+0000","last_undegraded":"2024-01-16T15:24:06.355204+0000","last_fullsized":"2024-01-16T15:24:06.355204+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'12","last_scrub_stamp":"2024-01-16T15:24:06.355004+0000","last_deep_scrub":"118'12","last_deep_scrub_stamp":"2024-01-16T15:24:06.355004+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.355004+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:24.345427+0000","scrub_duration":0.011408029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"118'46","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758301+0000","last_change":"2024-01-16T15:15:39.314200+0000","last_active":"2024-01-16T15:23:49.758301+0000","last_peered":"2024-01-16T15:23:49.758301+0000","last_clean":"2024-01-16T15:23:49.758301+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:23:49.758301+0000","last_undegraded":"2024-01-16T15:23:49.758301+0000","last_fullsized":"2024-01-16T15:23:49.758301+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"118'2","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.708121+0000","last_change":"2024-01-16T15:24:14.708181+0000","last_active":"2024-01-16T15:24:14.708121+0000","last_peered":"2024-01-16T15:24:14.708121+0000","last_clean":"2024-01-16T15:24:14.708121+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:24:14.708121+0000","last_undegraded":"2024-01-16T15:24:14.708121+0000","last_fullsized":"2024-01-16T15:24:14.708121+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'2","last_scrub_stamp":"2024-01-16T15:24:14.707879+0000","last_deep_scrub":"118'2","last_deep_scrub_stamp":"2024-01-16T15:24:14.707879+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.707879+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:26.465221+0000","scrub_duration":0.0070736139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"118'61","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.748319+0000","last_change":"2024-01-16T15:15:39.334617+0000","last_active":"2024-01-16T15:23:49.748319+0000","last_peered":"2024-01-16T15:23:49.748319+0000","last_clean":"2024-01-16T15:23:49.748319+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:23:49.748319+0000","last_undegraded":"2024-01-16T15:23:49.748319+0000","last_fullsized":"2024-01-16T15:23:49.748319+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"118'3","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.312734+0000","last_change":"2024-01-16T15:24:20.312794+0000","last_active":"2024-01-16T15:24:20.312734+0000","last_peered":"2024-01-16T15:24:20.312734+0000","last_clean":"2024-01-16T15:24:20.312734+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:24:20.312734+0000","last_undegraded":"2024-01-16T15:24:20.312734+0000","last_fullsized":"2024-01-16T15:24:20.312734+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:20.312507+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:20.312507+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.312507+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:01.823492+0000","scrub_duration":0.0071549860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"118'61","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755528+0000","last_change":"2024-01-16T15:15:39.332704+0000","last_active":"2024-01-16T15:23:49.755528+0000","last_peered":"2024-01-16T15:23:49.755528+0000","last_clean":"2024-01-16T15:23:49.755528+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:23:49.755528+0000","last_undegraded":"2024-01-16T15:23:49.755528+0000","last_fullsized":"2024-01-16T15:23:49.755528+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"118'2","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.418895+0000","last_change":"2024-01-16T15:24:08.419007+0000","last_active":"2024-01-16T15:24:08.418895+0000","last_peered":"2024-01-16T15:24:08.418895+0000","last_clean":"2024-01-16T15:24:08.418895+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:24:08.418895+0000","last_undegraded":"2024-01-16T15:24:08.418895+0000","last_fullsized":"2024-01-16T15:24:08.418895+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'2","last_scrub_stamp":"2024-01-16T15:24:08.418682+0000","last_deep_scrub":"118'2","last_deep_scrub_stamp":"2024-01-16T15:24:08.418682+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.418682+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:20.781341+0000","scrub_duration":0.0080426539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"118'62","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.755567+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:23:49.755567+0000","last_peered":"2024-01-16T15:23:49.755567+0000","last_clean":"2024-01-16T15:23:49.755567+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:23:49.755567+0000","last_undegraded":"2024-01-16T15:23:49.755567+0000","last_fullsized":"2024-01-16T15:23:49.755567+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.539661+0000","last_change":"2024-01-16T15:24:06.539708+0000","last_active":"2024-01-16T15:24:06.539661+0000","last_peered":"2024-01-16T15:24:06.539661+0000","last_clean":"2024-01-16T15:24:06.539661+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:24:06.539661+0000","last_undegraded":"2024-01-16T15:24:06.539661+0000","last_fullsized":"2024-01-16T15:24:06.539661+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:06.539466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:06.539466+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.539466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:05.228120+0000","scrub_duration":0.0037083429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"118'56","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.836362+0000","last_change":"2024-01-16T15:15:39.326224+0000","last_active":"2024-01-16T15:24:04.836362+0000","last_peered":"2024-01-16T15:24:04.836362+0000","last_clean":"2024-01-16T15:24:04.836362+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:24:04.836362+0000","last_undegraded":"2024-01-16T15:24:04.836362+0000","last_fullsized":"2024-01-16T15:24:04.836362+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"118'4","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.623213+0000","last_change":"2024-01-16T15:24:01.623234+0000","last_active":"2024-01-16T15:24:01.623213+0000","last_peered":"2024-01-16T15:24:01.623213+0000","last_clean":"2024-01-16T15:24:01.623213+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:24:01.623213+0000","last_undegraded":"2024-01-16T15:24:01.623213+0000","last_fullsized":"2024-01-16T15:24:01.623213+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4","last_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_deep_scrub":"118'4","last_deep_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.623154+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:58.563468+0000","scrub_duration":0.0032134569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"118'92","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.150173+0000","last_change":"2024-01-16T15:24:08.150238+0000","last_active":"2024-01-16T15:24:08.150173+0000","last_peered":"2024-01-16T15:24:08.150173+0000","last_clean":"2024-01-16T15:24:08.150173+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:24:08.150173+0000","last_undegraded":"2024-01-16T15:24:08.150173+0000","last_fullsized":"2024-01-16T15:24:08.150173+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'92","last_scrub_stamp":"2024-01-16T15:24:08.149996+0000","last_deep_scrub":"118'92","last_deep_scrub_stamp":"2024-01-16T15:24:08.149996+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.149996+0000","objects_scrubbed":26,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:37.843231+0000","scrub_duration":0.295168926,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.240464+0000","last_change":"2024-01-16T15:24:15.240464+0000","last_active":"2024-01-16T15:24:15.240464+0000","last_peered":"2024-01-16T15:24:15.240464+0000","last_clean":"2024-01-16T15:24:15.240464+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:24:15.240464+0000","last_undegraded":"2024-01-16T15:24:15.240464+0000","last_fullsized":"2024-01-16T15:24:15.240464+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:15.240213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:15.240213+0000","last_clean_scrub_stamp":"2024-01-16T15:24:15.240213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:36.556315+0000","scrub_duration":0.0034155610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"118'65","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.836455+0000","last_change":"2024-01-16T15:15:39.330480+0000","last_active":"2024-01-16T15:24:04.836455+0000","last_peered":"2024-01-16T15:24:04.836455+0000","last_clean":"2024-01-16T15:24:04.836455+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:24:04.836455+0000","last_undegraded":"2024-01-16T15:24:04.836455+0000","last_fullsized":"2024-01-16T15:24:04.836455+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.660042+0000","last_change":"2024-01-16T15:24:02.660042+0000","last_active":"2024-01-16T15:24:02.660042+0000","last_peered":"2024-01-16T15:24:02.660042+0000","last_clean":"2024-01-16T15:24:02.660042+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:24:02.660042+0000","last_undegraded":"2024-01-16T15:24:02.660042+0000","last_fullsized":"2024-01-16T15:24:02.660042+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.659822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:34.891359+0000","scrub_duration":0.0028615590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"118'77","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:22.227309+0000","last_change":"2024-01-16T15:24:22.227483+0000","last_active":"2024-01-16T15:24:22.227309+0000","last_peered":"2024-01-16T15:24:22.227309+0000","last_clean":"2024-01-16T15:24:22.227309+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:24:22.227309+0000","last_undegraded":"2024-01-16T15:24:22.227309+0000","last_fullsized":"2024-01-16T15:24:22.227309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'77","last_scrub_stamp":"2024-01-16T15:24:22.227034+0000","last_deep_scrub":"118'77","last_deep_scrub_stamp":"2024-01-16T15:24:22.227034+0000","last_clean_scrub_stamp":"2024-01-16T15:24:22.227034+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:51.920270+0000","scrub_duration":0.238057777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692036,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.775581+0000","last_change":"2024-01-16T15:24:16.775581+0000","last_active":"2024-01-16T15:24:16.775581+0000","last_peered":"2024-01-16T15:24:16.775581+0000","last_clean":"2024-01-16T15:24:16.775581+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:24:16.775581+0000","last_undegraded":"2024-01-16T15:24:16.775581+0000","last_fullsized":"2024-01-16T15:24:16.775581+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:16.775280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:16.775280+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.775280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:13.481653+0000","scrub_duration":0.0039614919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"118'67","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.597682+0000","last_change":"2024-01-16T15:24:16.597682+0000","last_active":"2024-01-16T15:24:16.597682+0000","last_peered":"2024-01-16T15:24:16.597682+0000","last_clean":"2024-01-16T15:24:16.597682+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:24:16.597682+0000","last_undegraded":"2024-01-16T15:24:16.597682+0000","last_fullsized":"2024-01-16T15:24:16.597682+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'67","last_scrub_stamp":"2024-01-16T15:24:16.597377+0000","last_deep_scrub":"118'67","last_deep_scrub_stamp":"2024-01-16T15:24:16.597377+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.597377+0000","objects_scrubbed":17,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:57.545654+0000","scrub_duration":0.18894488100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"118'3","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.946698+0000","last_change":"2024-01-16T15:24:17.946698+0000","last_active":"2024-01-16T15:24:17.946698+0000","last_peered":"2024-01-16T15:24:17.946698+0000","last_clean":"2024-01-16T15:24:17.946698+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:24:17.946698+0000","last_undegraded":"2024-01-16T15:24:17.946698+0000","last_fullsized":"2024-01-16T15:24:17.946698+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:17.946427+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:17.946427+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.946427+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:26.209326+0000","scrub_duration":0.0069638929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"118'61","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900296+0000","last_change":"2024-01-16T15:15:39.320032+0000","last_active":"2024-01-16T15:23:59.900296+0000","last_peered":"2024-01-16T15:23:59.900296+0000","last_clean":"2024-01-16T15:23:59.900296+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:23:59.900296+0000","last_undegraded":"2024-01-16T15:23:59.900296+0000","last_fullsized":"2024-01-16T15:23:59.900296+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"118'31","reported_seq":58,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.451631+0000","last_change":"2024-01-16T15:24:10.451631+0000","last_active":"2024-01-16T15:24:10.451631+0000","last_peered":"2024-01-16T15:24:10.451631+0000","last_clean":"2024-01-16T15:24:10.451631+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:24:10.451631+0000","last_undegraded":"2024-01-16T15:24:10.451631+0000","last_fullsized":"2024-01-16T15:24:10.451631+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'31","last_scrub_stamp":"2024-01-16T15:24:10.451303+0000","last_deep_scrub":"118'31","last_deep_scrub_stamp":"2024-01-16T15:24:10.451303+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.451303+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:06.356152+0000","scrub_duration":0.0039950840000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644,"num_omap_keys":12,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"118'69","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:03.410676+0000","last_change":"2024-01-16T15:15:55.923114+0000","last_active":"2024-01-16T15:24:03.410676+0000","last_peered":"2024-01-16T15:24:03.410676+0000","last_clean":"2024-01-16T15:24:03.410676+0000","last_became_active":"2024-01-16T15:15:55.914646+0000","last_became_peered":"2024-01-16T15:15:55.914646+0000","last_unstale":"2024-01-16T15:24:03.410676+0000","last_undegraded":"2024-01-16T15:24:03.410676+0000","last_fullsized":"2024-01-16T15:24:03.410676+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.759218+0000","last_change":"2024-01-16T15:24:20.759267+0000","last_active":"2024-01-16T15:24:20.759218+0000","last_peered":"2024-01-16T15:24:20.759218+0000","last_clean":"2024-01-16T15:24:20.759218+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:24:20.759218+0000","last_undegraded":"2024-01-16T15:24:20.759218+0000","last_fullsized":"2024-01-16T15:24:20.759218+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:20.759038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:20.759038+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.759038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:12.067470+0000","scrub_duration":0.003697947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"118'79","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.701538+0000","last_change":"2024-01-16T15:24:14.701607+0000","last_active":"2024-01-16T15:24:14.701538+0000","last_peered":"2024-01-16T15:24:14.701538+0000","last_clean":"2024-01-16T15:24:14.701538+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:24:14.701538+0000","last_undegraded":"2024-01-16T15:24:14.701538+0000","last_fullsized":"2024-01-16T15:24:14.701538+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'79","last_scrub_stamp":"2024-01-16T15:24:14.701344+0000","last_deep_scrub":"118'79","last_deep_scrub_stamp":"2024-01-16T15:24:14.701344+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.701344+0000","objects_scrubbed":28,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:46.392454+0000","scrub_duration":0.30808371000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.903030+0000","last_change":"2024-01-16T15:24:14.903079+0000","last_active":"2024-01-16T15:24:14.903030+0000","last_peered":"2024-01-16T15:24:14.903030+0000","last_clean":"2024-01-16T15:24:14.903030+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:24:14.903030+0000","last_undegraded":"2024-01-16T15:24:14.903030+0000","last_fullsized":"2024-01-16T15:24:14.903030+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:14.902847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:14.902847+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.902847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:18:33.188607+0000","scrub_duration":0.003674142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"118'63","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017343+0000","last_change":"2024-01-16T15:15:39.313583+0000","last_active":"2024-01-16T15:24:00.017343+0000","last_peered":"2024-01-16T15:24:00.017343+0000","last_clean":"2024-01-16T15:24:00.017343+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:24:00.017343+0000","last_undegraded":"2024-01-16T15:24:00.017343+0000","last_fullsized":"2024-01-16T15:24:00.017343+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.443065+0000","last_change":"2024-01-16T15:24:10.443138+0000","last_active":"2024-01-16T15:24:10.443065+0000","last_peered":"2024-01-16T15:24:10.443065+0000","last_clean":"2024-01-16T15:24:10.443065+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:24:10.443065+0000","last_undegraded":"2024-01-16T15:24:10.443065+0000","last_fullsized":"2024-01-16T15:24:10.443065+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:10.442885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:10.442885+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.442885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:31.758987+0000","scrub_duration":0.003956195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"118'66","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.017479+0000","last_change":"2024-01-16T15:15:39.305386+0000","last_active":"2024-01-16T15:24:00.017479+0000","last_peered":"2024-01-16T15:24:00.017479+0000","last_clean":"2024-01-16T15:24:00.017479+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:24:00.017479+0000","last_undegraded":"2024-01-16T15:24:00.017479+0000","last_fullsized":"2024-01-16T15:24:00.017479+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.246601+0000","last_change":"2024-01-16T15:24:11.246654+0000","last_active":"2024-01-16T15:24:11.246601+0000","last_peered":"2024-01-16T15:24:11.246601+0000","last_clean":"2024-01-16T15:24:11.246601+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:24:11.246601+0000","last_undegraded":"2024-01-16T15:24:11.246601+0000","last_fullsized":"2024-01-16T15:24:11.246601+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:11.246380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:11.246380+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.246380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:14.315962+0000","scrub_duration":0.0038743530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"118'68","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.752692+0000","last_change":"2024-01-16T15:15:39.330274+0000","last_active":"2024-01-16T15:23:49.752692+0000","last_peered":"2024-01-16T15:23:49.752692+0000","last_clean":"2024-01-16T15:23:49.752692+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:23:49.752692+0000","last_undegraded":"2024-01-16T15:23:49.752692+0000","last_fullsized":"2024-01-16T15:23:49.752692+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"118'32","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:09.262161+0000","last_change":"2024-01-16T15:24:09.262216+0000","last_active":"2024-01-16T15:24:09.262161+0000","last_peered":"2024-01-16T15:24:09.262161+0000","last_clean":"2024-01-16T15:24:09.262161+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:24:09.262161+0000","last_undegraded":"2024-01-16T15:24:09.262161+0000","last_fullsized":"2024-01-16T15:24:09.262161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'32","last_scrub_stamp":"2024-01-16T15:24:09.261992+0000","last_deep_scrub":"118'32","last_deep_scrub_stamp":"2024-01-16T15:24:09.261992+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.261992+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:09.421019+0000","scrub_duration":0.0066503719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3846,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"118'94","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.073597+0000","last_change":"2024-01-16T15:24:04.073657+0000","last_active":"2024-01-16T15:24:04.073597+0000","last_peered":"2024-01-16T15:24:04.073597+0000","last_clean":"2024-01-16T15:24:04.073597+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:24:04.073597+0000","last_undegraded":"2024-01-16T15:24:04.073597+0000","last_fullsized":"2024-01-16T15:24:04.073597+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.073487+0000","objects_scrubbed":32,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:45.014652+0000","scrub_duration":0.28876764900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.746241+0000","last_change":"2024-01-16T15:24:11.746367+0000","last_active":"2024-01-16T15:24:11.746241+0000","last_peered":"2024-01-16T15:24:11.746241+0000","last_clean":"2024-01-16T15:24:11.746241+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:24:11.746241+0000","last_undegraded":"2024-01-16T15:24:11.746241+0000","last_fullsized":"2024-01-16T15:24:11.746241+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:11.746034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:11.746034+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.746034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:13.384227+0000","scrub_duration":0.0037103750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"118'66","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:19.148038+0000","last_change":"2024-01-16T15:24:19.148119+0000","last_active":"2024-01-16T15:24:19.148038+0000","last_peered":"2024-01-16T15:24:19.148038+0000","last_clean":"2024-01-16T15:24:19.148038+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:24:19.148038+0000","last_undegraded":"2024-01-16T15:24:19.148038+0000","last_fullsized":"2024-01-16T15:24:19.148038+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'66","last_scrub_stamp":"2024-01-16T15:24:19.147855+0000","last_deep_scrub":"118'66","last_deep_scrub_stamp":"2024-01-16T15:24:19.147855+0000","last_clean_scrub_stamp":"2024-01-16T15:24:19.147855+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:29.163868+0000","scrub_duration":0.182029521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.536712+0000","last_change":"2024-01-16T15:24:05.536760+0000","last_active":"2024-01-16T15:24:05.536712+0000","last_peered":"2024-01-16T15:24:05.536712+0000","last_clean":"2024-01-16T15:24:05.536712+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:24:05.536712+0000","last_undegraded":"2024-01-16T15:24:05.536712+0000","last_fullsized":"2024-01-16T15:24:05.536712+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:05.536530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:05.536530+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.536530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:03.108287+0000","scrub_duration":0.0035509750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"118'119","reported_seq":275,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.491584+0000","last_change":"2024-01-16T15:24:06.491710+0000","last_active":"2024-01-16T15:24:06.491584+0000","last_peered":"2024-01-16T15:24:06.491584+0000","last_clean":"2024-01-16T15:24:06.491584+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:24:06.491584+0000","last_undegraded":"2024-01-16T15:24:06.491584+0000","last_fullsized":"2024-01-16T15:24:06.491584+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'119","last_scrub_stamp":"2024-01-16T15:24:06.491352+0000","last_deep_scrub":"118'119","last_deep_scrub_stamp":"2024-01-16T15:24:06.491352+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.491352+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:03.082154+0000","scrub_duration":0.061931657000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452737,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4527,"ondisk_log_size":4527,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1252565,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":489,"num_write_kb":1461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5644,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2505130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2096820,"kb_used_data":2025452,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92275020,"statfs":{"total":96636764160,"available":94489620480,"internally_reserved":0,"allocated":2074062848,"data_stored":2073213012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.238,"15min":1.26},"min":{"1min":0.84099999999999997,"5min":0.70399999999999996,"15min":0.57799999999999996},"max":{"1min":1.6100000000000001,"5min":1.6679999999999999,"15min":13.566000000000001},"last":1.216},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.323,"15min":1.3089999999999999},"min":{"1min":0.96499999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.984,"5min":1.984,"15min":10.728},"last":1.425}]},{"osd":1,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1200000000000001,"15min":1.159},"min":{"1min":0.74099999999999999,"5min":0.73899999999999999,"15min":0.59699999999999998},"max":{"1min":1.446,"5min":1.714,"15min":13.375},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1519999999999999,"15min":1.1950000000000001},"min":{"1min":0.86299999999999999,"5min":0.67800000000000005,"15min":0.65900000000000003},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":14.592000000000001},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.264,"15min":1.2589999999999999},"min":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.72199999999999998},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":4.9000000000000004},"last":1.163},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1910000000000001,"15min":1.2390000000000001},"min":{"1min":0.89800000000000002,"5min":0.77100000000000002,"15min":0.67600000000000005},"max":{"1min":1.4630000000000001,"5min":1.724,"15min":24.806000000000001},"last":1.01}]},{"osd":3,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2849999999999999,"15min":1.3839999999999999},"min":{"1min":0.86899999999999999,"5min":0.81399999999999995,"15min":0.67000000000000004},"max":{"1min":1.6599999999999999,"5min":2.27,"15min":23.741},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4650000000000001,"15min":1.508},"min":{"1min":1.1699999999999999,"5min":1.0029999999999999,"15min":0.76500000000000001},"max":{"1min":1.8260000000000001,"5min":2.3119999999999998,"15min":23.777000000000001},"last":1.2609999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.2450000000000001,"15min":1.3220000000000001},"min":{"1min":0.81000000000000005,"5min":0.57599999999999996,"15min":0.45900000000000002},"max":{"1min":1.865,"5min":1.865,"15min":22.355},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.413,"15min":1.516},"min":{"1min":0.85199999999999998,"5min":0.72899999999999998,"15min":0.63600000000000001},"max":{"1min":1.7689999999999999,"5min":1.9239999999999999,"15min":22.271999999999998},"last":1.079}]},{"osd":5,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.4139999999999999,"15min":1.454},"min":{"1min":0.95699999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.8300000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.95199999999999996,"5min":0.66000000000000003,"15min":0.56699999999999995},"max":{"1min":1.8680000000000001,"5min":1.8999999999999999,"15min":7.8540000000000001},"last":1.3899999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4379999999999999,"15min":1.4370000000000001},"min":{"1min":1.042,"5min":0.53000000000000003,"15min":0.50800000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.6389999999999998},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.4410000000000001,"15min":1.5},"min":{"1min":0.98599999999999999,"5min":0.63400000000000001,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":13.725},"last":1.665}]}]},{"osd":6,"up_from":14,"seq":60129542474,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477736,"kb_used_data":1406368,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92894104,"statfs":{"total":96636764160,"available":95123562496,"internally_reserved":0,"allocated":1440120832,"data_stored":1439281876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1699999999999999,"15min":1.1739999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58099999999999996},"max":{"1min":1.6399999999999999,"5min":1.764,"15min":3.9289999999999998},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.1950000000000001,"15min":1.246},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.41599999999999998},"max":{"1min":1.645,"5min":1.7130000000000001,"15min":8.0630000000000006},"last":0.83599999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2370000000000001,"15min":1.282},"min":{"1min":0.82299999999999995,"5min":0.63700000000000001,"15min":0.497},"max":{"1min":1.5880000000000001,"5min":1.7030000000000001,"15min":10.518000000000001},"last":1.796},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.302,"15min":1.286},"min":{"1min":0.95199999999999996,"5min":0.65000000000000002,"15min":0.53100000000000003},"max":{"1min":1.696,"5min":1.887,"15min":7.2919999999999998},"last":1.5600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2370000000000001,"15min":1.298},"min":{"1min":0.755,"5min":0.755,"15min":0.47099999999999997},"max":{"1min":1.5840000000000001,"5min":1.784,"15min":6.3040000000000003},"last":1.139},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2749999999999999,"15min":1.423},"min":{"1min":0.83099999999999996,"5min":0.72599999999999998,"15min":0.57099999999999995},"max":{"1min":1.6910000000000001,"5min":1.8240000000000001,"15min":15.282},"last":1.7070000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.343,"15min":1.357},"min":{"1min":1.0920000000000001,"5min":0.71899999999999997,"15min":0.64100000000000001},"max":{"1min":1.784,"5min":1.9490000000000001,"15min":4.1109999999999998},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.3839999999999999,"15min":1.458},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.50900000000000001},"max":{"1min":1.621,"5min":1.7370000000000001,"15min":9.0960000000000001},"last":1.256}]},{"osd":4,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.3640000000000001,"15min":1.3759999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.49299999999999999},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":5.4669999999999996},"last":1.397},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3580000000000001,"15min":1.413},"min":{"1min":0.95799999999999996,"5min":0.81799999999999995,"15min":0.41199999999999998},"max":{"1min":1.651,"5min":1.7989999999999999,"15min":5.3360000000000003},"last":1.3520000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3380000000000001,"15min":1.403},"min":{"1min":0.89200000000000002,"5min":0.65700000000000003,"15min":0.436},"max":{"1min":1.7370000000000001,"5min":1.9239999999999999,"15min":10.388999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.518,"15min":1.55},"min":{"1min":1.0660000000000001,"5min":1.016,"15min":0.5},"max":{"1min":2.0070000000000001,"5min":2.012,"15min":7.9640000000000004},"last":1.7569999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5449999999999999,"15min":1.589},"min":{"1min":1.1020000000000001,"5min":0.98399999999999999,"15min":0.48899999999999999},"max":{"1min":1.738,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.512},{"interface":"front","average":{"1min":1.452,"5min":1.474,"15min":1.6080000000000001},"min":{"1min":0.97399999999999998,"5min":0.84199999999999997,"15min":0.58499999999999996},"max":{"1min":1.8540000000000001,"5min":1.984,"15min":16.175999999999998},"last":1.3140000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542475,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174568,"kb_used_data":1107360,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93197272,"statfs":{"total":96636764160,"available":95434006528,"internally_reserved":0,"allocated":1133936640,"data_stored":1133098096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.278,"15min":1.216},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.532,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.151,"5min":1.3120000000000001,"15min":1.236},"min":{"1min":0.81899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.657,"5min":12.922000000000001,"15min":12.922000000000001},"last":1.1359999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3200000000000001,"15min":1.3169999999999999},"min":{"1min":0.89700000000000002,"5min":0.66300000000000003,"15min":0.63900000000000001},"max":{"1min":1.6879999999999999,"5min":5.2249999999999996,"15min":8.1820000000000004},"last":0.997},{"interface":"front","average":{"1min":1.129,"5min":1.175,"15min":1.173},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.536,"5min":2.0659999999999998,"15min":2.597},"last":1.5589999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.268,"15min":1.264},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6459999999999999,"5min":1.712,"15min":3.0470000000000002},"last":1.234},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2250000000000001,"15min":1.2669999999999999},"min":{"1min":0.72299999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6359999999999999,"5min":1.756,"15min":8.0619999999999994},"last":1.095}]},{"osd":3,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.401,"15min":1.484},"min":{"1min":0.83099999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7430000000000001,"5min":5.157,"15min":16.931999999999999},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.419,"15min":1.494},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":2.6960000000000002,"15min":19.111999999999998},"last":1.607}]},{"osd":4,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1930000000000001,"15min":1.2589999999999999},"min":{"1min":0.71899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.7549999999999999,"5min":1.7689999999999999,"15min":4.8490000000000002},"last":1.534},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3540000000000001,"15min":1.411},"min":{"1min":0.84899999999999998,"5min":0.63300000000000001,"15min":0.57399999999999995},"max":{"1min":1.7569999999999999,"5min":5.0810000000000004,"15min":8.3550000000000004},"last":1.4790000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4119999999999999,"15min":1.5069999999999999},"min":{"1min":0.79300000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.716,"5min":1.9530000000000001,"15min":17.419},"last":1.367},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.415,"15min":1.5269999999999999},"min":{"1min":0.83599999999999997,"5min":0.69899999999999995,"15min":0.69799999999999995},"max":{"1min":1.7170000000000001,"5min":2.2959999999999998,"15min":16.262},"last":1.45}]},{"osd":7,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.6040000000000001},"min":{"1min":1.1359999999999999,"5min":0.68400000000000005,"15min":0.67000000000000004},"max":{"1min":1.889,"5min":4.0380000000000003,"15min":12.385999999999999},"last":1.294},{"interface":"front","average":{"1min":1.429,"5min":1.431,"15min":1.512},"min":{"1min":0.73999999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6919999999999999,"5min":1.796,"15min":5.0780000000000003},"last":1.498}]}]},{"osd":4,"up_from":14,"seq":60129542476,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936484,"kb_used_data":1865116,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435356,"statfs":{"total":96636764160,"available":94653804544,"internally_reserved":0,"allocated":1909878784,"data_stored":1909043912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2649999999999999,"15min":1.2569999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.63300000000000001},"max":{"1min":1.7669999999999999,"5min":1.774,"15min":3.3159999999999998},"last":1.345},{"interface":"front","average":{"1min":1.163,"5min":1.129,"15min":1.1919999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.47499999999999998},"max":{"1min":1.77,"5min":1.77,"15min":5.7430000000000003},"last":1.254}]},{"osd":1,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.254,"15min":1.302},"min":{"1min":0.81799999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":3.7850000000000001},"last":1.458},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2170000000000001,"15min":1.234},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.57899999999999996},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":4.6760000000000002},"last":1.516}]},{"osd":2,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2010000000000001,"15min":1.2310000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.69199999999999995},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":13.731999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.3620000000000001,"15min":1.367},"min":{"1min":0.91900000000000004,"5min":0.79400000000000004,"15min":0.73199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7729999999999999,"15min":2.2570000000000001},"last":1.718}]},{"osd":3,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.341,"15min":1.341},"min":{"1min":1.0009999999999999,"5min":0.69999999999999996,"15min":0.59899999999999998},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":4.5789999999999997},"last":1.671},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3280000000000001,"15min":1.363},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.71699999999999997},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":13.914},"last":1.0960000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.329,"15min":1.3580000000000001},"min":{"1min":0.90200000000000002,"5min":0.80500000000000005,"15min":0.47299999999999998},"max":{"1min":1.911,"5min":2.0190000000000001,"15min":10.211},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3720000000000001,"15min":1.4079999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.58299999999999996},"max":{"1min":1.6399999999999999,"5min":1.7669999999999999,"15min":4.673},"last":1.0389999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3779999999999999,"15min":1.4119999999999999},"min":{"1min":0.97299999999999998,"5min":0.83099999999999996,"15min":0.60699999999999998},"max":{"1min":1.74,"5min":1.7410000000000001,"15min":13.973000000000001},"last":1.2},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.5349999999999999},"min":{"1min":1.246,"5min":0.85099999999999998,"15min":0.52400000000000002},"max":{"1min":1.8520000000000001,"5min":1.9019999999999999,"15min":3.0979999999999999},"last":1.629}]},{"osd":7,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.5009999999999999,"15min":1.536},"min":{"1min":0.878,"5min":0.878,"15min":0.53500000000000003},"max":{"1min":1.7709999999999999,"5min":2.0539999999999998,"15min":4.7640000000000002},"last":1.556},{"interface":"front","average":{"1min":1.502,"5min":1.5229999999999999,"15min":1.5669999999999999},"min":{"1min":0.97699999999999998,"5min":0.95999999999999996,"15min":0.55200000000000005},"max":{"1min":1.7989999999999999,"5min":2.0179999999999998,"15min":13.852},"last":1.3080000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542475,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1832764,"kb_used_data":1751668,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":92539076,"statfs":{"total":96636764160,"available":94760013824,"internally_reserved":0,"allocated":1793708032,"data_stored":1792851922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94699999999999995,"15min":0.95599999999999996},"min":{"1min":0.71299999999999997,"5min":0.626,"15min":0.34300000000000003},"max":{"1min":1.353,"5min":2.605,"15min":2.9129999999999998},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1739999999999999,"15min":1.143},"min":{"1min":0.438,"5min":0.438,"15min":0.29699999999999999},"max":{"1min":1.2849999999999999,"5min":4.0389999999999997,"15min":12.292999999999999},"last":0.99199999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.069,"15min":0.98899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.33100000000000002},"max":{"1min":1.5580000000000001,"5min":12.335000000000001,"15min":12.335000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0960000000000001,"15min":1.133},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.315},"max":{"1min":1.712,"5min":4.6529999999999996,"15min":10.638},"last":1.0620000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.206,"15min":1.151},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":8.3740000000000006,"15min":8.3740000000000006},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.248,"15min":1.228},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.46000000000000002},"max":{"1min":1.6200000000000001,"5min":4.2629999999999999,"15min":4.2629999999999999},"last":1.1519999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.403,"15min":1.345},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.39300000000000002},"max":{"1min":1.76,"5min":15.512,"15min":15.512},"last":1.181},{"interface":"front","average":{"1min":1.274,"5min":1.411,"15min":1.4119999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.52100000000000002},"max":{"1min":1.9399999999999999,"5min":11.743,"15min":14.250999999999999},"last":1.2190000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4710000000000001,"15min":1.417},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7470000000000001,"5min":8.4290000000000003,"15min":8.4290000000000003},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.419,"15min":1.409},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.42099999999999999},"max":{"1min":1.8440000000000001,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":1.286}]},{"osd":6,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3839999999999999,"15min":1.419},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.41499999999999998},"max":{"1min":1.8879999999999999,"5min":3.956,"15min":12.675000000000001},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.5669999999999999,"15min":1.49},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.54900000000000004},"max":{"1min":1.849,"5min":11.667,"15min":11.667},"last":1.333}]},{"osd":7,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.569,"15min":1.5569999999999999},"min":{"1min":1.113,"5min":1.097,"15min":0.53700000000000003},"max":{"1min":1.9279999999999999,"5min":11.042999999999999,"15min":11.042999999999999},"last":1.375},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.474,"15min":1.5289999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.435},"max":{"1min":1.954,"5min":3.907,"15min":12.836},"last":1.556}]}]},{"osd":2,"up_from":14,"seq":60129542475,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1211492,"kb_used_data":1140124,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93160348,"statfs":{"total":96636764160,"available":95396196352,"internally_reserved":0,"allocated":1167486976,"data_stored":1166655701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89200000000000002,"15min":0.89400000000000002},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.311},"max":{"1min":1.6759999999999999,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":1.222},{"interface":"front","average":{"1min":0.995,"5min":1.0449999999999999,"15min":1.056},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.496,"5min":1.5549999999999999,"15min":3.52},"last":0.96999999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1599999999999999,"15min":1.0880000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.504,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.1060000000000001,"15min":1.1080000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.7729999999999999,"5min":1.8109999999999999,"15min":2.9900000000000002},"last":1.1819999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.036,"15min":1.048},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.45700000000000002},"max":{"1min":1.8320000000000001,"5min":1.8360000000000001,"15min":6.1260000000000003},"last":1.133},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.214,"15min":1.226},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.47999999999999998},"max":{"1min":1.6950000000000001,"5min":2.911,"15min":6.2510000000000003},"last":1.621}]},{"osd":4,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.423,"15min":1.423},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.68400000000000005},"max":{"1min":1.7989999999999999,"5min":3.012,"15min":3.012},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.3939999999999999,"15min":1.3959999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.71099999999999997},"max":{"1min":2.4020000000000001,"5min":3.54,"15min":3.54},"last":1.077}]},{"osd":5,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3660000000000001,"15min":1.389},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":1.774,"5min":3.6179999999999999,"15min":8.625},"last":1.343},{"interface":"front","average":{"1min":1.306,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.944,"5min":1.944,"15min":6.3499999999999996},"last":1.286}]},{"osd":6,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3839999999999999,"15min":1.377},"min":{"1min":1.0429999999999999,"5min":0.96299999999999997,"15min":0.71499999999999997},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":6.4729999999999999},"last":1.393},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.407,"15min":1.4370000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.71099999999999997},"max":{"1min":2.125,"5min":2.8260000000000001,"15min":3.2469999999999999},"last":2.1669999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4750000000000001,"15min":1.4910000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.704,"5min":3.431,"15min":3.7290000000000001},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.478,"15min":1.492},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0379999999999998,"5min":2.3050000000000002,"15min":6.4249999999999998},"last":1.52}]}]},{"osd":1,"up_from":14,"seq":60129542475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412264,"kb_used_data":1336736,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92959576,"statfs":{"total":96636764160,"available":95190605824,"internally_reserved":0,"allocated":1368817664,"data_stored":1367982226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.94499999999999995,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.33400000000000002},"max":{"1min":1.3120000000000001,"5min":1.4079999999999999,"15min":8.8499999999999996},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.147,"15min":1.1619999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"max":{"1min":1.5089999999999999,"5min":1.5940000000000001,"15min":8.9380000000000006},"last":1.103}]},{"osd":2,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.34899999999999998},"max":{"1min":1.2549999999999999,"5min":1.456,"15min":3.4460000000000002},"last":1.042},{"interface":"front","average":{"1min":1.079,"5min":1.123,"15min":1.147},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5840000000000001,"5min":1.677,"15min":2.9660000000000002},"last":1.1899999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.2050000000000001,"15min":1.234},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.45400000000000001},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.282},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.1180000000000001,"15min":1.135},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.47099999999999997},"max":{"1min":1.655,"5min":1.6830000000000001,"15min":8.6910000000000007},"last":1.3089999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.335,"15min":1.3600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.716,"5min":1.8109999999999999,"15min":4.2480000000000002},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.57899999999999996},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":4.7640000000000002},"last":1.155}]},{"osd":5,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.50800000000000001},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3680000000000003},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.452,"15min":1.476},"min":{"1min":1.0569999999999999,"5min":0.77500000000000002,"15min":0.66300000000000003},"max":{"1min":1.772,"5min":1.841,"15min":5.4610000000000003},"last":1.5589999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":4.7779999999999996},"last":1.508},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.365,"15min":1.419},"min":{"1min":0.746,"5min":0.746,"15min":0.56899999999999995},"max":{"1min":2.339,"5min":2.339,"15min":4.8300000000000001},"last":1.29}]},{"osd":7,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4510000000000001,"15min":1.4830000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.58999999999999997},"max":{"1min":2.0510000000000002,"5min":2.081,"15min":6.7990000000000004},"last":1.367},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.484,"15min":1.544},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.198,"5min":2.198,"15min":5.1260000000000003},"last":1.7110000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542476,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2036608,"kb_used_data":1965240,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92335232,"statfs":{"total":96636764160,"available":94551277568,"internally_reserved":0,"allocated":2012405760,"data_stored":2011545811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.34899999999999998},"max":{"1min":1.381,"5min":2.387,"15min":2.7320000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.0269999999999999,"15min":1.0680000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.29399999999999998},"max":{"1min":1.3720000000000001,"5min":1.694,"15min":9.4499999999999993},"last":0.878}]},{"osd":2,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2490000000000001,"15min":1.3029999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.45100000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":9.5579999999999998},"last":1.472},{"interface":"front","average":{"1min":1.119,"5min":1.149,"15min":1.1459999999999999},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.40200000000000002},"max":{"1min":1.5960000000000001,"5min":2.4359999999999999,"15min":5.3159999999999998},"last":1.0960000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0489999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46600000000000003},"max":{"1min":1.335,"5min":1.5940000000000001,"15min":10.83},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1659999999999999,"15min":1.2529999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":1.5409999999999999,"5min":1.99,"15min":9.6029999999999998},"last":1.0169999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3260000000000001,"15min":1.306},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.63400000000000001},"max":{"1min":1.641,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.177},{"interface":"front","average":{"1min":1.2,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.67200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":10.986000000000001},"last":1.401}]},{"osd":5,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":1.0089999999999999,"5min":0.78700000000000003,"15min":0.70499999999999996},"max":{"1min":1.665,"5min":3.0259999999999998,"15min":3.0259999999999998},"last":1.232},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":1.514,"5min":1.667,"15min":14.561999999999999},"last":1.3129999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.47,"15min":1.522},"min":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.76600000000000001},"max":{"1min":1.6639999999999999,"5min":2.2069999999999999,"15min":14.958},"last":1.264},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.506,"15min":1.48},"min":{"1min":1.069,"5min":0.89500000000000002,"15min":0.68500000000000005},"max":{"1min":1.7270000000000001,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.349}]},{"osd":7,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4890000000000001,"15min":1.5640000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.7869999999999999,"5min":2.129,"15min":15.005000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5089999999999999,"15min":1.552},"min":{"1min":1.028,"5min":0.93400000000000005,"15min":0.70699999999999996},"max":{"1min":1.9279999999999999,"5min":3.0590000000000002,"15min":13.718},"last":1.4379999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1248033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1247991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2009083904,"data_stored":2009072258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166020608,"data_stored":1166016703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908416512,"data_stored":1908408760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132466176,"data_stored":1132462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:24:25.831 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.832 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.832 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.832 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.832 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.832 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.832 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.833 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.833 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.833 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.833 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.833 INFO:tasks.ceph:pgid 17.31 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.833 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.834 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.834 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.834 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.834 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.834 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.835 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.835 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.835 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.835 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.835 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.836 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.836 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.836 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.836 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.836 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.837 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.837 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.837 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.837 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.838 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.838 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.838 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.838 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.838 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.839 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.839 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.839 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.839 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.839 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.840 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.840 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.840 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:25.840 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:24:40.312 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:40.321 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:45.842 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:24:46.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.065+0000 7f6350a00640 1 Processor -- start 2024-01-16T15:24:46.063 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.065+0000 7f6350a00640 1 -- start start 2024-01-16T15:24:46.064 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.066+0000 7f6350a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c0cc990 0x7f634c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.066+0000 7f6350a00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c070e50 con 0x7f634c0cc500 2024-01-16T15:24:46.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.066+0000 7f6350a00640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c144020 con 0x7f634c143c40 2024-01-16T15:24:46.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.066+0000 7f6350a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0723c0 con 0x7f634c0cc990 2024-01-16T15:24:46.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.067+0000 7f6341c00640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f634c0cc500 0x7f634c06d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:54250/0 (socket says 172.21.15.141:54250) 2024-01-16T15:24:46.065 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.067+0000 7f6341c00640 1 -- 172.21.15.141:0/2907796214 learned_addr learned my addr 172.21.15.141:0/2907796214 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:24:46.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.067+0000 7f634b200640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c0cc990 0x7f634c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:46.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.067+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143966383 0 0) 0x7f634c070e50 con 0x7f634c0cc500 2024-01-16T15:24:46.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.067+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c003520 con 0x7f634c0cc500 2024-01-16T15:24:46.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2118898553 0 0) 0x7f632c003520 con 0x7f634c0cc500 2024-01-16T15:24:46.067 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c0cc990 msgr2=0x7f634c06d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:46.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c0cc990 0x7f634c06d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 >> v1:172.21.15.188:6789/0 conn(0x7f634c143c40 legacy=0x7f634c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:46.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c071e00 con 0x7f634c0cc500 2024-01-16T15:24:46.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7f6334001690 con 0x7f634c0cc500 2024-01-16T15:24:46.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 >> v1:172.21.15.141:6789/0 conn(0x7f634c0cc500 legacy=0x7f634c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:46.068 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f632c0039a0 0x7f632c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c0cc990 0x7f632c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c004d00 0x7f632c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0723c0 con 0x7f632c0039a0 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c00bb40 con 0x7f632c004d00 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c00bd00 con 0x7f634c0cc990 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f634a800640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f632c0039a0 0x7f632c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.068+0000 7f634b200640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c004d00 0x7f632c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:46.069 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.069+0000 7f634a800640 1 -- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c0cc990 msgr2=0x7f632c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:46.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.069+0000 7f634a800640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c0cc990 0x7f632c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.069+0000 7f634a800640 1 -- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c004d00 msgr2=0x7f632c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:24:46.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.069+0000 7f634a800640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c004d00 0x7f632c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.070 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.069+0000 7f634a800640 1 -- 172.21.15.141:0/2907796214 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c003520 con 0x7f632c0039a0 2024-01-16T15:24:46.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.070+0000 7f634a800640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f632c0039a0 0x7f632c003dc0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f633c001750 tx=0x7f632c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3f557c1d0f3435 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:46.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.070+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c0052d0 con 0x7f632c0039a0 2024-01-16T15:24:46.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.070+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c02e050 con 0x7f632c0039a0 2024-01-16T15:24:46.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.070+0000 7f6349e00640 1 -- 172.21.15.141:0/2907796214 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c0035f0 con 0x7f632c0039a0 2024-01-16T15:24:46.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.070+0000 7f6350a00640 1 -- 172.21.15.141:0/2907796214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f632c0039a0 msgr2=0x7f632c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:46.073 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.070+0000 7f6350a00640 1 --2- 172.21.15.141:0/2907796214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f632c0039a0 0x7f632c003dc0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f633c001750 tx=0x7f632c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.071+0000 7f634b200640 1 -- 172.21.15.141:0/2907796214 reap_dead start 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.071+0000 7f6350a00640 1 -- 172.21.15.141:0/2907796214 shutdown_connections 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.071+0000 7f6350a00640 1 -- 172.21.15.141:0/2907796214 >> 172.21.15.141:0/2907796214 conn(0x7f634c06a800 msgr2=0x7f634c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.071+0000 7f6350a00640 1 -- 172.21.15.141:0/2907796214 shutdown_connections 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.071+0000 7f6350a00640 1 -- 172.21.15.141:0/2907796214 wait complete. 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.072+0000 7f6350a00640 1 Processor -- start 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.073+0000 7f6350a00640 1 -- start start 2024-01-16T15:24:46.074 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.073+0000 7f6350a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c0039a0 0x7f634c1e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.073+0000 7f6350a00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f634c0aca10 0x7f634c15a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.073+0000 7f6350a00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c1e5390 0x7f634c1e5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6350a00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c1e5d50 con 0x7f634c0aca10 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6350a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c13fa50 con 0x7f632c0039a0 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6350a00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0b1fa0 con 0x7f634c1e5390 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f634c0aca10 0x7f634c15a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f634c0aca10 0x7f634c15a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50914/0 (socket says 172.21.15.141:50914) 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 -- 172.21.15.141:0/1290454214 learned_addr learned my addr 172.21.15.141:0/1290454214 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:24:46.075 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f634a800640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c0039a0 0x7f634c1e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 -- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c1e5390 msgr2=0x7f634c1e5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c1e5390 0x7f634c1e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 -- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c0039a0 msgr2=0x7f634c1e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c0039a0 0x7f634c1e47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 -- 172.21.15.141:0/1290454214 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0acfd0 con 0x7f634c0aca10 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.074+0000 7f6341c00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f634c0aca10 0x7f634c15a710 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f633400c6b0 tx=0x7f633400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.075+0000 7f6343400640 1 -- 172.21.15.141:0/1290454214 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63340072d0 con 0x7f634c0aca10 2024-01-16T15:24:46.076 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.075+0000 7f6343400640 1 -- 172.21.15.141:0/1290454214 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6334007450 con 0x7f634c0aca10 2024-01-16T15:24:46.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.075+0000 7f6343400640 1 -- 172.21.15.141:0/1290454214 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63340047e0 con 0x7f634c0aca10 2024-01-16T15:24:46.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.075+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c0ad2b0 con 0x7f634c0aca10 2024-01-16T15:24:46.077 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.075+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c0ad870 con 0x7f634c0aca10 2024-01-16T15:24:46.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.076+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c0b21a0 con 0x7f634c0aca10 2024-01-16T15:24:46.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.077+0000 7f6343400640 1 -- 172.21.15.141:0/1290454214 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f6334004960 con 0x7f634c0aca10 2024-01-16T15:24:46.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.078+0000 7f6343400640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f632406c5e0 0x7f632406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:24:46.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.078+0000 7f6343400640 1 -- 172.21.15.141:0/1290454214 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6334087e70 con 0x7f634c0aca10 2024-01-16T15:24:46.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.078+0000 7f634a800640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f632406c5e0 0x7f632406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:24:46.078 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.080+0000 7f634a800640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f632406c5e0 0x7f632406eaa0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f633c033000 tx=0x7f633c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:24:46.080 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.082+0000 7f6343400640 1 -- 172.21.15.141:0/1290454214 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f634c0b21a0 con 0x7f634c0aca10 2024-01-16T15:24:46.271 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.273+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f634c06cc40 con 0x7f632406c5e0 2024-01-16T15:24:46.294 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.296+0000 7f6343400640 1 -- 172.21.15.141:0/1290454214 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291346 (secure 0 0 0) 0x7f633c0053b0 con 0x7f632406c5e0 2024-01-16T15:24:46.298 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:24:46.299 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:24:46.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f632406c5e0 msgr2=0x7f632406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:46.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f632406c5e0 0x7f632406eaa0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f633c033000 tx=0x7f633c009030 comp rx=0 tx=0).stop 2024-01-16T15:24:46.301 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f634c0aca10 msgr2=0x7f634c15a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f634c0aca10 0x7f634c15a710 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f633400c6b0 tx=0x7f633400cb70 comp rx=0 tx=0).stop 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 shutdown_connections 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f632406c5e0 0x7f632406eaa0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f634c1e5390 0x7f634c1e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f634c0aca10 0x7f634c15a710 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 --2- 172.21.15.141:0/1290454214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f632c0039a0 0x7f634c1e47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 >> 172.21.15.141:0/1290454214 conn(0x7f634c06a800 msgr2=0x7f634c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:24:46.302 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 shutdown_connections 2024-01-16T15:24:46.303 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:24:46.301+0000 7f6350a00640 1 -- 172.21.15.141:0/1290454214 wait complete. 2024-01-16T15:24:46.319 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":944,"stamp":"2024-01-16T15:24:45.140509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444295670,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":945,"num_read_kb":3145889,"num_write":2253,"num_write_kb":6295371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5644,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6070,"ondisk_log_size":6070,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13182896,"kb_used_data":12598064,"kb_used_omap":12,"kb_used_meta":584755,"kb_avail":741791824,"statfs":{"total":773094113280,"available":759594827776,"internally_reserved":0,"allocated":12900417536,"data_stored":12893672556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":598789682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004718"},"pg_stats":[{"pgid":"17.3f","version":"118'70","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:59.900432+0000","last_change":"2024-01-16T15:15:39.329033+0000","last_active":"2024-01-16T15:23:59.900432+0000","last_peered":"2024-01-16T15:23:59.900432+0000","last_clean":"2024-01-16T15:23:59.900432+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:23:59.900432+0000","last_undegraded":"2024-01-16T15:23:59.900432+0000","last_fullsized":"2024-01-16T15:23:59.900432+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"118'95","reported_seq":136,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:29.567524+0000","last_change":"2024-01-16T15:24:29.567586+0000","last_active":"2024-01-16T15:24:29.567524+0000","last_peered":"2024-01-16T15:24:29.567524+0000","last_clean":"2024-01-16T15:24:29.567524+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:24:29.567524+0000","last_undegraded":"2024-01-16T15:24:29.567524+0000","last_fullsized":"2024-01-16T15:24:29.567524+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:29.567320+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:29.567320+0000","last_clean_scrub_stamp":"2024-01-16T15:24:29.567320+0000","objects_scrubbed":26,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:03.170389+0000","scrub_duration":0.35401311600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"118'67","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.189812+0000","last_change":"2024-01-16T15:15:39.324701+0000","last_active":"2024-01-16T15:24:00.189812+0000","last_peered":"2024-01-16T15:24:00.189812+0000","last_clean":"2024-01-16T15:24:00.189812+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:24:00.189812+0000","last_undegraded":"2024-01-16T15:24:00.189812+0000","last_fullsized":"2024-01-16T15:24:00.189812+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"118'91","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.504390+0000","last_change":"2024-01-16T15:24:31.504446+0000","last_active":"2024-01-16T15:24:31.504390+0000","last_peered":"2024-01-16T15:24:31.504390+0000","last_clean":"2024-01-16T15:24:31.504390+0000","last_became_active":"2024-01-16T15:15:55.875179+0000","last_became_peered":"2024-01-16T15:15:55.875179+0000","last_unstale":"2024-01-16T15:24:31.504390+0000","last_undegraded":"2024-01-16T15:24:31.504390+0000","last_fullsized":"2024-01-16T15:24:31.504390+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'91","last_scrub_stamp":"2024-01-16T15:24:31.504227+0000","last_deep_scrub":"118'91","last_deep_scrub_stamp":"2024-01-16T15:24:31.504227+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.504227+0000","objects_scrubbed":25,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:06.166941+0000","scrub_duration":0.289711153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3b","version":"118'85","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.606933+0000","last_change":"2024-01-16T15:24:26.607090+0000","last_active":"2024-01-16T15:24:26.606933+0000","last_peered":"2024-01-16T15:24:26.606933+0000","last_clean":"2024-01-16T15:24:26.606933+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:24:26.606933+0000","last_undegraded":"2024-01-16T15:24:26.606933+0000","last_fullsized":"2024-01-16T15:24:26.606933+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'85","last_scrub_stamp":"2024-01-16T15:24:26.606670+0000","last_deep_scrub":"118'85","last_deep_scrub_stamp":"2024-01-16T15:24:26.606670+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.606670+0000","objects_scrubbed":25,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:32.586228+0000","scrub_duration":0.30264022000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663487,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"118'104","reported_seq":148,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:23.051913+0000","last_change":"2024-01-16T15:24:23.051959+0000","last_active":"2024-01-16T15:24:23.051913+0000","last_peered":"2024-01-16T15:24:23.051913+0000","last_clean":"2024-01-16T15:24:23.051913+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:24:23.051913+0000","last_undegraded":"2024-01-16T15:24:23.051913+0000","last_fullsized":"2024-01-16T15:24:23.051913+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'104","last_scrub_stamp":"2024-01-16T15:24:23.051823+0000","last_deep_scrub":"118'104","last_deep_scrub_stamp":"2024-01-16T15:24:23.051823+0000","last_clean_scrub_stamp":"2024-01-16T15:24:23.051823+0000","objects_scrubbed":31,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:26.121405+0000","scrub_duration":0.37272100699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"118'84","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.581480+0000","last_change":"2024-01-16T15:24:36.581539+0000","last_active":"2024-01-16T15:24:36.581480+0000","last_peered":"2024-01-16T15:24:36.581480+0000","last_clean":"2024-01-16T15:24:36.581480+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:24:36.581480+0000","last_undegraded":"2024-01-16T15:24:36.581480+0000","last_fullsized":"2024-01-16T15:24:36.581480+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'84","last_scrub_stamp":"2024-01-16T15:24:36.581385+0000","last_deep_scrub":"118'84","last_deep_scrub_stamp":"2024-01-16T15:24:36.581385+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.581385+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:28.917659+0000","scrub_duration":0.195587708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303428,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"118'71","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.696314+0000","last_change":"2024-01-16T15:24:17.696409+0000","last_active":"2024-01-16T15:24:17.696314+0000","last_peered":"2024-01-16T15:24:17.696314+0000","last_clean":"2024-01-16T15:24:17.696314+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:24:17.696314+0000","last_undegraded":"2024-01-16T15:24:17.696314+0000","last_fullsized":"2024-01-16T15:24:17.696314+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'71","last_scrub_stamp":"2024-01-16T15:24:17.696140+0000","last_deep_scrub":"118'71","last_deep_scrub_stamp":"2024-01-16T15:24:17.696140+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.696140+0000","objects_scrubbed":22,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:05.520208+0000","scrub_duration":0.28034184200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"118'79","reported_seq":111,"reported_epoch":118,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-16T15:24:44.732791+0000","last_change":"2024-01-16T15:24:44.731329+0000","last_active":"2024-01-16T15:24:44.732791+0000","last_peered":"2024-01-16T15:24:44.732791+0000","last_clean":"2024-01-16T15:24:44.732791+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:24:44.732791+0000","last_undegraded":"2024-01-16T15:24:44.732791+0000","last_fullsized":"2024-01-16T15:24:44.732791+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052164,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"118'88","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.030027+0000","last_change":"2024-01-16T15:24:36.030123+0000","last_active":"2024-01-16T15:24:36.030027+0000","last_peered":"2024-01-16T15:24:36.030027+0000","last_clean":"2024-01-16T15:24:36.030027+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:24:36.030027+0000","last_undegraded":"2024-01-16T15:24:36.030027+0000","last_fullsized":"2024-01-16T15:24:36.030027+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:36.029853+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:36.029853+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.029853+0000","objects_scrubbed":22,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:14.638731+0000","scrub_duration":0.27669553099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"118'97","reported_seq":141,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:42.395120+0000","last_change":"2024-01-16T15:24:42.395148+0000","last_active":"2024-01-16T15:24:42.395120+0000","last_peered":"2024-01-16T15:24:42.395120+0000","last_clean":"2024-01-16T15:24:42.395120+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:24:42.395120+0000","last_undegraded":"2024-01-16T15:24:42.395120+0000","last_fullsized":"2024-01-16T15:24:42.395120+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:42.395059+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:42.395059+0000","last_clean_scrub_stamp":"2024-01-16T15:24:42.395059+0000","objects_scrubbed":27,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:27.035568+0000","scrub_duration":0.31104689099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"118'84","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:35.066548+0000","last_change":"2024-01-16T15:24:35.066706+0000","last_active":"2024-01-16T15:24:35.066548+0000","last_peered":"2024-01-16T15:24:35.066548+0000","last_clean":"2024-01-16T15:24:35.066548+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:24:35.066548+0000","last_undegraded":"2024-01-16T15:24:35.066548+0000","last_fullsized":"2024-01-16T15:24:35.066548+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'84","last_scrub_stamp":"2024-01-16T15:24:35.066338+0000","last_deep_scrub":"118'84","last_deep_scrub_stamp":"2024-01-16T15:24:35.066338+0000","last_clean_scrub_stamp":"2024-01-16T15:24:35.066338+0000","objects_scrubbed":25,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:19.990397+0000","scrub_duration":0.31775753299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"118'78","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.873328+0000","last_change":"2024-01-16T15:24:31.873406+0000","last_active":"2024-01-16T15:24:31.873328+0000","last_peered":"2024-01-16T15:24:31.873328+0000","last_clean":"2024-01-16T15:24:31.873328+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:24:31.873328+0000","last_undegraded":"2024-01-16T15:24:31.873328+0000","last_fullsized":"2024-01-16T15:24:31.873328+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'78","last_scrub_stamp":"2024-01-16T15:24:31.873149+0000","last_deep_scrub":"118'78","last_deep_scrub_stamp":"2024-01-16T15:24:31.873149+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.873149+0000","objects_scrubbed":21,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:45.473499+0000","scrub_duration":0.24798467199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"118'88","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:12.757274+0000","last_change":"2024-01-16T15:24:12.757322+0000","last_active":"2024-01-16T15:24:12.757274+0000","last_peered":"2024-01-16T15:24:12.757274+0000","last_clean":"2024-01-16T15:24:12.757274+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:24:12.757274+0000","last_undegraded":"2024-01-16T15:24:12.757274+0000","last_fullsized":"2024-01-16T15:24:12.757274+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:12.757133+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:12.757133+0000","last_clean_scrub_stamp":"2024-01-16T15:24:12.757133+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:23:26.026719+0000","scrub_duration":0.337903377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"118'68","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.945319+0000","last_change":"2024-01-16T15:24:31.945405+0000","last_active":"2024-01-16T15:24:31.945319+0000","last_peered":"2024-01-16T15:24:31.945319+0000","last_clean":"2024-01-16T15:24:31.945319+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:24:31.945319+0000","last_undegraded":"2024-01-16T15:24:31.945319+0000","last_fullsized":"2024-01-16T15:24:31.945319+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'68","last_scrub_stamp":"2024-01-16T15:24:31.945123+0000","last_deep_scrub":"118'68","last_deep_scrub_stamp":"2024-01-16T15:24:31.945123+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.945123+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:28.376879+0000","scrub_duration":0.19293099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"118'77","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.129572+0000","last_change":"2024-01-16T15:24:26.129640+0000","last_active":"2024-01-16T15:24:26.129572+0000","last_peered":"2024-01-16T15:24:26.129572+0000","last_clean":"2024-01-16T15:24:26.129572+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:24:26.129572+0000","last_undegraded":"2024-01-16T15:24:26.129572+0000","last_fullsized":"2024-01-16T15:24:26.129572+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'77","last_scrub_stamp":"2024-01-16T15:24:26.129442+0000","last_deep_scrub":"118'77","last_deep_scrub_stamp":"2024-01-16T15:24:26.129442+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.129442+0000","objects_scrubbed":19,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:23.395204+0000","scrub_duration":0.23139104999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"118'66","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:40.343813+0000","last_change":"2024-01-16T15:24:40.343868+0000","last_active":"2024-01-16T15:24:40.343813+0000","last_peered":"2024-01-16T15:24:40.343813+0000","last_clean":"2024-01-16T15:24:40.343813+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:24:40.343813+0000","last_undegraded":"2024-01-16T15:24:40.343813+0000","last_fullsized":"2024-01-16T15:24:40.343813+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'66","last_scrub_stamp":"2024-01-16T15:24:40.343721+0000","last_deep_scrub":"118'66","last_deep_scrub_stamp":"2024-01-16T15:24:40.343721+0000","last_clean_scrub_stamp":"2024-01-16T15:24:40.343721+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:12.489538+0000","scrub_duration":0.21130144100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"118'107","reported_seq":158,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:22.884521+0000","last_change":"2024-01-16T15:24:22.884589+0000","last_active":"2024-01-16T15:24:22.884521+0000","last_peered":"2024-01-16T15:24:22.884521+0000","last_clean":"2024-01-16T15:24:22.884521+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:24:22.884521+0000","last_undegraded":"2024-01-16T15:24:22.884521+0000","last_fullsized":"2024-01-16T15:24:22.884521+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'107","last_scrub_stamp":"2024-01-16T15:24:22.884365+0000","last_deep_scrub":"118'107","last_deep_scrub_stamp":"2024-01-16T15:24:22.884365+0000","last_clean_scrub_stamp":"2024-01-16T15:24:22.884365+0000","objects_scrubbed":35,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:12.796440+0000","scrub_duration":0.443307281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"118'100","reported_seq":136,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:39.015448+0000","last_change":"2024-01-16T15:24:39.015448+0000","last_active":"2024-01-16T15:24:39.015448+0000","last_peered":"2024-01-16T15:24:39.015448+0000","last_clean":"2024-01-16T15:24:39.015448+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:24:39.015448+0000","last_undegraded":"2024-01-16T15:24:39.015448+0000","last_fullsized":"2024-01-16T15:24:39.015448+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'100","last_scrub_stamp":"2024-01-16T15:24:39.015183+0000","last_deep_scrub":"118'100","last_deep_scrub_stamp":"2024-01-16T15:24:39.015183+0000","last_clean_scrub_stamp":"2024-01-16T15:24:39.015183+0000","objects_scrubbed":29,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:39:34.700309+0000","scrub_duration":0.322461267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"118'91","reported_seq":129,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:33.478784+0000","last_change":"2024-01-16T15:24:33.478784+0000","last_active":"2024-01-16T15:24:33.478784+0000","last_peered":"2024-01-16T15:24:33.478784+0000","last_clean":"2024-01-16T15:24:33.478784+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:24:33.478784+0000","last_undegraded":"2024-01-16T15:24:33.478784+0000","last_fullsized":"2024-01-16T15:24:33.478784+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'91","last_scrub_stamp":"2024-01-16T15:24:33.478490+0000","last_deep_scrub":"118'91","last_deep_scrub_stamp":"2024-01-16T15:24:33.478490+0000","last_clean_scrub_stamp":"2024-01-16T15:24:33.478490+0000","objects_scrubbed":24,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:39.313097+0000","scrub_duration":0.29979264300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"118'99","reported_seq":148,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.587081+0000","last_change":"2024-01-16T15:24:15.587081+0000","last_active":"2024-01-16T15:24:15.587081+0000","last_peered":"2024-01-16T15:24:15.587081+0000","last_clean":"2024-01-16T15:24:15.587081+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:24:15.587081+0000","last_undegraded":"2024-01-16T15:24:15.587081+0000","last_fullsized":"2024-01-16T15:24:15.587081+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2024-01-16T15:24:15.586844+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2024-01-16T15:24:15.586844+0000","last_clean_scrub_stamp":"2024-01-16T15:24:15.586844+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:57.424369+0000","scrub_duration":0.353635808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"118'90","reported_seq":124,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.405531+0000","last_change":"2024-01-16T15:24:36.405531+0000","last_active":"2024-01-16T15:24:36.405531+0000","last_peered":"2024-01-16T15:24:36.405531+0000","last_clean":"2024-01-16T15:24:36.405531+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:24:36.405531+0000","last_undegraded":"2024-01-16T15:24:36.405531+0000","last_fullsized":"2024-01-16T15:24:36.405531+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'90","last_scrub_stamp":"2024-01-16T15:24:36.405296+0000","last_deep_scrub":"118'90","last_deep_scrub_stamp":"2024-01-16T15:24:36.405296+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.405296+0000","objects_scrubbed":26,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:27.713727+0000","scrub_duration":0.32153288499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"118'79","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.554719+0000","last_change":"2024-01-16T15:24:14.554719+0000","last_active":"2024-01-16T15:24:14.554719+0000","last_peered":"2024-01-16T15:24:14.554719+0000","last_clean":"2024-01-16T15:24:14.554719+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:24:14.554719+0000","last_undegraded":"2024-01-16T15:24:14.554719+0000","last_fullsized":"2024-01-16T15:24:14.554719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'79","last_scrub_stamp":"2024-01-16T15:24:14.554529+0000","last_deep_scrub":"118'79","last_deep_scrub_stamp":"2024-01-16T15:24:14.554529+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.554529+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:06.890804+0000","scrub_duration":0.31410593999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"118'94","reported_seq":130,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:21.246704+0000","last_change":"2024-01-16T15:24:21.246810+0000","last_active":"2024-01-16T15:24:21.246704+0000","last_peered":"2024-01-16T15:24:21.246704+0000","last_clean":"2024-01-16T15:24:21.246704+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:24:21.246704+0000","last_undegraded":"2024-01-16T15:24:21.246704+0000","last_fullsized":"2024-01-16T15:24:21.246704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:21.246488+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:21.246488+0000","last_clean_scrub_stamp":"2024-01-16T15:24:21.246488+0000","objects_scrubbed":24,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:51.356431+0000","scrub_duration":0.27909722799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"118'97","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.521003+0000","last_change":"2024-01-16T15:24:36.521052+0000","last_active":"2024-01-16T15:24:36.521003+0000","last_peered":"2024-01-16T15:24:36.521003+0000","last_clean":"2024-01-16T15:24:36.521003+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:24:36.521003+0000","last_undegraded":"2024-01-16T15:24:36.521003+0000","last_fullsized":"2024-01-16T15:24:36.521003+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:36.520866+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:36.520866+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.520866+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:24.676593+0000","scrub_duration":0.33406188199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"118'107","reported_seq":144,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:37.069885+0000","last_change":"2024-01-16T15:24:37.069949+0000","last_active":"2024-01-16T15:24:37.069885+0000","last_peered":"2024-01-16T15:24:37.069885+0000","last_clean":"2024-01-16T15:24:37.069885+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:24:37.069885+0000","last_undegraded":"2024-01-16T15:24:37.069885+0000","last_fullsized":"2024-01-16T15:24:37.069885+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'107","last_scrub_stamp":"2024-01-16T15:24:37.069711+0000","last_deep_scrub":"118'107","last_deep_scrub_stamp":"2024-01-16T15:24:37.069711+0000","last_clean_scrub_stamp":"2024-01-16T15:24:37.069711+0000","objects_scrubbed":26,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:47.854355+0000","scrub_duration":0.29586178600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"118'88","reported_seq":124,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:30.089992+0000","last_change":"2024-01-16T15:24:30.090059+0000","last_active":"2024-01-16T15:24:30.089992+0000","last_peered":"2024-01-16T15:24:30.089992+0000","last_clean":"2024-01-16T15:24:30.089992+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:24:30.089992+0000","last_undegraded":"2024-01-16T15:24:30.089992+0000","last_fullsized":"2024-01-16T15:24:30.089992+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:30.089879+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:30.089879+0000","last_clean_scrub_stamp":"2024-01-16T15:24:30.089879+0000","objects_scrubbed":24,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:11.927404+0000","scrub_duration":0.31580465699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"118'62","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.239853+0000","last_change":"2024-01-16T15:24:26.239940+0000","last_active":"2024-01-16T15:24:26.239853+0000","last_peered":"2024-01-16T15:24:26.239853+0000","last_clean":"2024-01-16T15:24:26.239853+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:24:26.239853+0000","last_undegraded":"2024-01-16T15:24:26.239853+0000","last_fullsized":"2024-01-16T15:24:26.239853+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'62","last_scrub_stamp":"2024-01-16T15:24:26.239694+0000","last_deep_scrub":"118'62","last_deep_scrub_stamp":"2024-01-16T15:24:26.239694+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.239694+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:37.997036+0000","scrub_duration":0.25765072300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"118'77","reported_seq":116,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:43.359494+0000","last_change":"2024-01-16T15:24:43.359547+0000","last_active":"2024-01-16T15:24:43.359494+0000","last_peered":"2024-01-16T15:24:43.359494+0000","last_clean":"2024-01-16T15:24:43.359494+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:24:43.359494+0000","last_undegraded":"2024-01-16T15:24:43.359494+0000","last_fullsized":"2024-01-16T15:24:43.359494+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'77","last_scrub_stamp":"2024-01-16T15:24:43.359391+0000","last_deep_scrub":"118'77","last_deep_scrub_stamp":"2024-01-16T15:24:43.359391+0000","last_clean_scrub_stamp":"2024-01-16T15:24:43.359391+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:02.316499+0000","scrub_duration":0.25140222099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"118'104","reported_seq":146,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:29.647830+0000","last_change":"2024-01-16T15:24:29.647892+0000","last_active":"2024-01-16T15:24:29.647830+0000","last_peered":"2024-01-16T15:24:29.647830+0000","last_clean":"2024-01-16T15:24:29.647830+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:24:29.647830+0000","last_undegraded":"2024-01-16T15:24:29.647830+0000","last_fullsized":"2024-01-16T15:24:29.647830+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'104","last_scrub_stamp":"2024-01-16T15:24:29.647644+0000","last_deep_scrub":"118'104","last_deep_scrub_stamp":"2024-01-16T15:24:29.647644+0000","last_clean_scrub_stamp":"2024-01-16T15:24:29.647644+0000","objects_scrubbed":28,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:32.717226+0000","scrub_duration":0.35423907199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"118'108","reported_seq":155,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:30.156400+0000","last_change":"2024-01-16T15:24:30.156470+0000","last_active":"2024-01-16T15:24:30.156400+0000","last_peered":"2024-01-16T15:24:30.156400+0000","last_clean":"2024-01-16T15:24:30.156400+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:24:30.156400+0000","last_undegraded":"2024-01-16T15:24:30.156400+0000","last_fullsized":"2024-01-16T15:24:30.156400+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'108","last_scrub_stamp":"2024-01-16T15:24:30.156210+0000","last_deep_scrub":"118'108","last_deep_scrub_stamp":"2024-01-16T15:24:30.156210+0000","last_clean_scrub_stamp":"2024-01-16T15:24:30.156210+0000","objects_scrubbed":27,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:02:23.423714+0000","scrub_duration":0.36436912999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"118'82","reported_seq":119,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.992127+0000","last_change":"2024-01-16T15:24:10.992228+0000","last_active":"2024-01-16T15:24:10.992127+0000","last_peered":"2024-01-16T15:24:10.992127+0000","last_clean":"2024-01-16T15:24:10.992127+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:24:10.992127+0000","last_undegraded":"2024-01-16T15:24:10.992127+0000","last_fullsized":"2024-01-16T15:24:10.992127+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'82","last_scrub_stamp":"2024-01-16T15:24:10.991908+0000","last_deep_scrub":"118'82","last_deep_scrub_stamp":"2024-01-16T15:24:10.991908+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.991908+0000","objects_scrubbed":23,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:34.358631+0000","scrub_duration":0.279512233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"118'88","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:29.070725+0000","last_change":"2024-01-16T15:24:29.070774+0000","last_active":"2024-01-16T15:24:29.070725+0000","last_peered":"2024-01-16T15:24:29.070725+0000","last_clean":"2024-01-16T15:24:29.070725+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:24:29.070725+0000","last_undegraded":"2024-01-16T15:24:29.070725+0000","last_fullsized":"2024-01-16T15:24:29.070725+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:29.070548+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:29.070548+0000","last_clean_scrub_stamp":"2024-01-16T15:24:29.070548+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:21.406823+0000","scrub_duration":0.32460896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"118'6","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.468778+0000","last_change":"2024-01-16T15:24:08.468844+0000","last_active":"2024-01-16T15:24:08.468778+0000","last_peered":"2024-01-16T15:24:08.468778+0000","last_clean":"2024-01-16T15:24:08.468778+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:24:08.468778+0000","last_undegraded":"2024-01-16T15:24:08.468778+0000","last_fullsized":"2024-01-16T15:24:08.468778+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6","last_scrub_stamp":"2024-01-16T15:24:08.468615+0000","last_deep_scrub":"118'6","last_deep_scrub_stamp":"2024-01-16T15:24:08.468615+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.468615+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:16.033835+0000","scrub_duration":0.0085012860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"118'73","reported_seq":112,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:19.646515+0000","last_change":"2024-01-16T15:24:19.646587+0000","last_active":"2024-01-16T15:24:19.646515+0000","last_peered":"2024-01-16T15:24:19.646515+0000","last_clean":"2024-01-16T15:24:19.646515+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:24:19.646515+0000","last_undegraded":"2024-01-16T15:24:19.646515+0000","last_fullsized":"2024-01-16T15:24:19.646515+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'73","last_scrub_stamp":"2024-01-16T15:24:19.646340+0000","last_deep_scrub":"118'73","last_deep_scrub_stamp":"2024-01-16T15:24:19.646340+0000","last_clean_scrub_stamp":"2024-01-16T15:24:19.646340+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:37.373916+0000","scrub_duration":0.24430438900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.441338+0000","last_change":"2024-01-16T15:24:10.441429+0000","last_active":"2024-01-16T15:24:10.441338+0000","last_peered":"2024-01-16T15:24:10.441338+0000","last_clean":"2024-01-16T15:24:10.441338+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:24:10.441338+0000","last_undegraded":"2024-01-16T15:24:10.441338+0000","last_fullsized":"2024-01-16T15:24:10.441338+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:10.441063+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:10.441063+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:44.044132+0000","scrub_duration":0.004404593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"118'97","reported_seq":141,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.044503+0000","last_change":"2024-01-16T15:24:20.044562+0000","last_active":"2024-01-16T15:24:20.044503+0000","last_peered":"2024-01-16T15:24:20.044503+0000","last_clean":"2024-01-16T15:24:20.044503+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:24:20.044503+0000","last_undegraded":"2024-01-16T15:24:20.044503+0000","last_fullsized":"2024-01-16T15:24:20.044503+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:20.044335+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:20.044335+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.044335+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:39.481751+0000","scrub_duration":0.33193780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"118'4","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.220298+0000","last_change":"2024-01-16T15:24:16.220358+0000","last_active":"2024-01-16T15:24:16.220298+0000","last_peered":"2024-01-16T15:24:16.220298+0000","last_clean":"2024-01-16T15:24:16.220298+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:24:16.220298+0000","last_undegraded":"2024-01-16T15:24:16.220298+0000","last_fullsized":"2024-01-16T15:24:16.220298+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4","last_scrub_stamp":"2024-01-16T15:24:16.220137+0000","last_deep_scrub":"118'4","last_deep_scrub_stamp":"2024-01-16T15:24:16.220137+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.220137+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:35.174555+0000","scrub_duration":0.0071288089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"118'68","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:23:49.758185+0000","last_change":"2024-01-16T15:15:39.329347+0000","last_active":"2024-01-16T15:23:49.758185+0000","last_peered":"2024-01-16T15:23:49.758185+0000","last_clean":"2024-01-16T15:23:49.758185+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:23:49.758185+0000","last_undegraded":"2024-01-16T15:23:49.758185+0000","last_fullsized":"2024-01-16T15:23:49.758185+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"118'3","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.217316+0000","last_change":"2024-01-16T15:24:14.217375+0000","last_active":"2024-01-16T15:24:14.217316+0000","last_peered":"2024-01-16T15:24:14.217316+0000","last_clean":"2024-01-16T15:24:14.217316+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:24:14.217316+0000","last_undegraded":"2024-01-16T15:24:14.217316+0000","last_fullsized":"2024-01-16T15:24:14.217316+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:14.217126+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:14.217126+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.217126+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:21.782346+0000","scrub_duration":0.0072240430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"118'95","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.056895+0000","last_change":"2024-01-16T15:24:31.056994+0000","last_active":"2024-01-16T15:24:31.056895+0000","last_peered":"2024-01-16T15:24:31.056895+0000","last_clean":"2024-01-16T15:24:31.056895+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:24:31.056895+0000","last_undegraded":"2024-01-16T15:24:31.056895+0000","last_fullsized":"2024-01-16T15:24:31.056895+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:31.056810+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:31.056810+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.056810+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:30.609981+0000","scrub_duration":0.28217324500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:18.450001+0000","last_change":"2024-01-16T15:24:18.450098+0000","last_active":"2024-01-16T15:24:18.450001+0000","last_peered":"2024-01-16T15:24:18.450001+0000","last_clean":"2024-01-16T15:24:18.450001+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:24:18.450001+0000","last_undegraded":"2024-01-16T15:24:18.450001+0000","last_fullsized":"2024-01-16T15:24:18.450001+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:18.449469+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:18.449469+0000","last_clean_scrub_stamp":"2024-01-16T15:24:18.449469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:41.816192+0000","scrub_duration":0.0038895650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"118'97","reported_seq":143,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:34.488943+0000","last_change":"2024-01-16T15:24:34.489001+0000","last_active":"2024-01-16T15:24:34.488943+0000","last_peered":"2024-01-16T15:24:34.488943+0000","last_clean":"2024-01-16T15:24:34.488943+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:24:34.488943+0000","last_undegraded":"2024-01-16T15:24:34.488943+0000","last_fullsized":"2024-01-16T15:24:34.488943+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:34.488759+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:34.488759+0000","last_clean_scrub_stamp":"2024-01-16T15:24:34.488759+0000","objects_scrubbed":25,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:22.312827+0000","scrub_duration":0.29169478900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.835879+0000","last_change":"2024-01-16T15:24:02.835904+0000","last_active":"2024-01-16T15:24:02.835879+0000","last_peered":"2024-01-16T15:24:02.835879+0000","last_clean":"2024-01-16T15:24:02.835879+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:24:02.835879+0000","last_undegraded":"2024-01-16T15:24:02.835879+0000","last_fullsized":"2024-01-16T15:24:02.835879+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.835792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:09.602675+0000","scrub_duration":0.0029040390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"118'95","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.969881+0000","last_change":"2024-01-16T15:24:00.969955+0000","last_active":"2024-01-16T15:24:00.969881+0000","last_peered":"2024-01-16T15:24:00.969881+0000","last_clean":"2024-01-16T15:24:00.969881+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:24:00.969881+0000","last_undegraded":"2024-01-16T15:24:00.969881+0000","last_fullsized":"2024-01-16T15:24:00.969881+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.969804+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:57.211104+0000","scrub_duration":0.179269503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.280477+0000","last_change":"2024-01-16T15:24:07.280530+0000","last_active":"2024-01-16T15:24:07.280477+0000","last_peered":"2024-01-16T15:24:07.280477+0000","last_clean":"2024-01-16T15:24:07.280477+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:24:07.280477+0000","last_undegraded":"2024-01-16T15:24:07.280477+0000","last_fullsized":"2024-01-16T15:24:07.280477+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:07.280247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:07.280247+0000","last_clean_scrub_stamp":"2024-01-16T15:24:07.280247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:26.717663+0000","scrub_duration":0.0039888249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"118'101","reported_seq":137,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:28.623771+0000","last_change":"2024-01-16T15:24:28.623808+0000","last_active":"2024-01-16T15:24:28.623771+0000","last_peered":"2024-01-16T15:24:28.623771+0000","last_clean":"2024-01-16T15:24:28.623771+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:24:28.623771+0000","last_undegraded":"2024-01-16T15:24:28.623771+0000","last_fullsized":"2024-01-16T15:24:28.623771+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'101","last_scrub_stamp":"2024-01-16T15:24:28.623666+0000","last_deep_scrub":"118'101","last_deep_scrub_stamp":"2024-01-16T15:24:28.623666+0000","last_clean_scrub_stamp":"2024-01-16T15:24:28.623666+0000","objects_scrubbed":28,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:48.061082+0000","scrub_duration":0.34195076600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.814116+0000","last_change":"2024-01-16T15:24:17.814162+0000","last_active":"2024-01-16T15:24:17.814116+0000","last_peered":"2024-01-16T15:24:17.814116+0000","last_clean":"2024-01-16T15:24:17.814116+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:24:17.814116+0000","last_undegraded":"2024-01-16T15:24:17.814116+0000","last_fullsized":"2024-01-16T15:24:17.814116+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:17.813872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:17.813872+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.813872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:11.813723+0000","scrub_duration":0.0034917469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"118'76","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:39.542732+0000","last_change":"2024-01-16T15:24:39.542813+0000","last_active":"2024-01-16T15:24:39.542732+0000","last_peered":"2024-01-16T15:24:39.542732+0000","last_clean":"2024-01-16T15:24:39.542732+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:24:39.542732+0000","last_undegraded":"2024-01-16T15:24:39.542732+0000","last_fullsized":"2024-01-16T15:24:39.542732+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'76","last_scrub_stamp":"2024-01-16T15:24:39.542553+0000","last_deep_scrub":"118'76","last_deep_scrub_stamp":"2024-01-16T15:24:39.542553+0000","last_clean_scrub_stamp":"2024-01-16T15:24:39.542553+0000","objects_scrubbed":18,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:48.986527+0000","scrub_duration":0.19909136099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"118'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.306271+0000","last_change":"2024-01-16T15:24:10.306417+0000","last_active":"2024-01-16T15:24:10.306271+0000","last_peered":"2024-01-16T15:24:10.306271+0000","last_clean":"2024-01-16T15:24:10.306271+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:24:10.306271+0000","last_undegraded":"2024-01-16T15:24:10.306271+0000","last_fullsized":"2024-01-16T15:24:10.306271+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'1","last_scrub_stamp":"2024-01-16T15:24:10.306050+0000","last_deep_scrub":"118'1","last_deep_scrub_stamp":"2024-01-16T15:24:10.306050+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.306050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:08.994705+0000","scrub_duration":0.0031432769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"118'78","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.670647+0000","last_change":"2024-01-16T15:24:17.670740+0000","last_active":"2024-01-16T15:24:17.670647+0000","last_peered":"2024-01-16T15:24:17.670647+0000","last_clean":"2024-01-16T15:24:17.670647+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:24:17.670647+0000","last_undegraded":"2024-01-16T15:24:17.670647+0000","last_fullsized":"2024-01-16T15:24:17.670647+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'78","last_scrub_stamp":"2024-01-16T15:24:17.670442+0000","last_deep_scrub":"118'78","last_deep_scrub_stamp":"2024-01-16T15:24:17.670442+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.670442+0000","objects_scrubbed":20,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:10:39.488294+0000","scrub_duration":0.25523936800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.949045+0000","last_change":"2024-01-16T15:24:07.949150+0000","last_active":"2024-01-16T15:24:07.949045+0000","last_peered":"2024-01-16T15:24:07.949045+0000","last_clean":"2024-01-16T15:24:07.949045+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:24:07.949045+0000","last_undegraded":"2024-01-16T15:24:07.949045+0000","last_fullsized":"2024-01-16T15:24:07.949045+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:07.948766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:07.948766+0000","last_clean_scrub_stamp":"2024-01-16T15:24:07.948766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:49.830805+0000","scrub_duration":0.0035464849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"118'88","reported_seq":130,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:23.283298+0000","last_change":"2024-01-16T15:24:23.283365+0000","last_active":"2024-01-16T15:24:23.283298+0000","last_peered":"2024-01-16T15:24:23.283298+0000","last_clean":"2024-01-16T15:24:23.283298+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:24:23.283298+0000","last_undegraded":"2024-01-16T15:24:23.283298+0000","last_fullsized":"2024-01-16T15:24:23.283298+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:23.283119+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:23.283119+0000","last_clean_scrub_stamp":"2024-01-16T15:24:23.283119+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:18.238164+0000","scrub_duration":0.31081178100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"118'33","reported_seq":59,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:24.862951+0000","last_change":"2024-01-16T15:24:24.863013+0000","last_active":"2024-01-16T15:24:24.862951+0000","last_peered":"2024-01-16T15:24:24.862951+0000","last_clean":"2024-01-16T15:24:24.862951+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:24:24.862951+0000","last_undegraded":"2024-01-16T15:24:24.862951+0000","last_fullsized":"2024-01-16T15:24:24.862951+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'33","last_scrub_stamp":"2024-01-16T15:24:24.862770+0000","last_deep_scrub":"118'33","last_deep_scrub_stamp":"2024-01-16T15:24:24.862770+0000","last_clean_scrub_stamp":"2024-01-16T15:24:24.862770+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:58.053240+0000","scrub_duration":0.0090951780000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"118'96","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:38.721801+0000","last_change":"2024-01-16T15:24:38.721843+0000","last_active":"2024-01-16T15:24:38.721801+0000","last_peered":"2024-01-16T15:24:38.721801+0000","last_clean":"2024-01-16T15:24:38.721801+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:24:38.721801+0000","last_undegraded":"2024-01-16T15:24:38.721801+0000","last_fullsized":"2024-01-16T15:24:38.721801+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'96","last_scrub_stamp":"2024-01-16T15:24:38.721704+0000","last_deep_scrub":"118'96","last_deep_scrub_stamp":"2024-01-16T15:24:38.721704+0000","last_clean_scrub_stamp":"2024-01-16T15:24:38.721704+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:31:17.163057+0000","scrub_duration":0.33844099599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"118'328","reported_seq":354,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:09.800108+0000","last_change":"2024-01-16T15:24:09.800168+0000","last_active":"2024-01-16T15:24:09.800108+0000","last_peered":"2024-01-16T15:24:09.800108+0000","last_clean":"2024-01-16T15:24:09.800108+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:24:09.800108+0000","last_undegraded":"2024-01-16T15:24:09.800108+0000","last_fullsized":"2024-01-16T15:24:09.800108+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'328","last_scrub_stamp":"2024-01-16T15:24:09.799941+0000","last_deep_scrub":"118'328","last_deep_scrub_stamp":"2024-01-16T15:24:09.799941+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.799941+0000","objects_scrubbed":4,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:04.754986+0000","scrub_duration":0.023713471999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1247921,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":1376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"118'75","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:27.265556+0000","last_change":"2024-01-16T15:24:27.265640+0000","last_active":"2024-01-16T15:24:27.265556+0000","last_peered":"2024-01-16T15:24:27.265556+0000","last_clean":"2024-01-16T15:24:27.265556+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:24:27.265556+0000","last_undegraded":"2024-01-16T15:24:27.265556+0000","last_fullsized":"2024-01-16T15:24:27.265556+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'75","last_scrub_stamp":"2024-01-16T15:24:27.265387+0000","last_deep_scrub":"118'75","last_deep_scrub_stamp":"2024-01-16T15:24:27.265387+0000","last_clean_scrub_stamp":"2024-01-16T15:24:27.265387+0000","objects_scrubbed":21,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:23.971761+0000","scrub_duration":0.24131996999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"118'5","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:03.636265+0000","last_change":"2024-01-16T15:24:03.636309+0000","last_active":"2024-01-16T15:24:03.636265+0000","last_peered":"2024-01-16T15:24:03.636265+0000","last_clean":"2024-01-16T15:24:03.636265+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:24:03.636265+0000","last_undegraded":"2024-01-16T15:24:03.636265+0000","last_fullsized":"2024-01-16T15:24:03.636265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5","last_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_deep_scrub":"118'5","last_deep_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_clean_scrub_stamp":"2024-01-16T15:24:03.636187+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:22.590604+0000","scrub_duration":0.0075747449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"118'98","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:37.368377+0000","last_change":"2024-01-16T15:24:37.368444+0000","last_active":"2024-01-16T15:24:37.368377+0000","last_peered":"2024-01-16T15:24:37.368377+0000","last_clean":"2024-01-16T15:24:37.368377+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:24:37.368377+0000","last_undegraded":"2024-01-16T15:24:37.368377+0000","last_fullsized":"2024-01-16T15:24:37.368377+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'98","last_scrub_stamp":"2024-01-16T15:24:37.368197+0000","last_deep_scrub":"118'98","last_deep_scrub_stamp":"2024-01-16T15:24:37.368197+0000","last_clean_scrub_stamp":"2024-01-16T15:24:37.368197+0000","objects_scrubbed":26,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:10.558667+0000","scrub_duration":0.29771550400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.384677+0000","last_change":"2024-01-16T15:24:05.384762+0000","last_active":"2024-01-16T15:24:05.384677+0000","last_peered":"2024-01-16T15:24:05.384677+0000","last_clean":"2024-01-16T15:24:05.384677+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:24:05.384677+0000","last_undegraded":"2024-01-16T15:24:05.384677+0000","last_fullsized":"2024-01-16T15:24:05.384677+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:05.384503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:05.384503+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.384503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:22.638464+0000","scrub_duration":0.0038951989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"118'89","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:24.284788+0000","last_change":"2024-01-16T15:24:24.284846+0000","last_active":"2024-01-16T15:24:24.284788+0000","last_peered":"2024-01-16T15:24:24.284788+0000","last_clean":"2024-01-16T15:24:24.284788+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:24:24.284788+0000","last_undegraded":"2024-01-16T15:24:24.284788+0000","last_fullsized":"2024-01-16T15:24:24.284788+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'89","last_scrub_stamp":"2024-01-16T15:24:24.284582+0000","last_deep_scrub":"118'89","last_deep_scrub_stamp":"2024-01-16T15:24:24.284582+0000","last_clean_scrub_stamp":"2024-01-16T15:24:24.284582+0000","objects_scrubbed":25,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:25.922775+0000","scrub_duration":0.27736433700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":40961,"num_write":27,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"118'12","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.355204+0000","last_change":"2024-01-16T15:24:06.355271+0000","last_active":"2024-01-16T15:24:06.355204+0000","last_peered":"2024-01-16T15:24:06.355204+0000","last_clean":"2024-01-16T15:24:06.355204+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:24:06.355204+0000","last_undegraded":"2024-01-16T15:24:06.355204+0000","last_fullsized":"2024-01-16T15:24:06.355204+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'12","last_scrub_stamp":"2024-01-16T15:24:06.355004+0000","last_deep_scrub":"118'12","last_deep_scrub_stamp":"2024-01-16T15:24:06.355004+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.355004+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:24.345427+0000","scrub_duration":0.011408029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"118'61","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:28.452503+0000","last_change":"2024-01-16T15:24:28.452503+0000","last_active":"2024-01-16T15:24:28.452503+0000","last_peered":"2024-01-16T15:24:28.452503+0000","last_clean":"2024-01-16T15:24:28.452503+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:24:28.452503+0000","last_undegraded":"2024-01-16T15:24:28.452503+0000","last_fullsized":"2024-01-16T15:24:28.452503+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'61","last_scrub_stamp":"2024-01-16T15:24:28.452255+0000","last_deep_scrub":"118'61","last_deep_scrub_stamp":"2024-01-16T15:24:28.452255+0000","last_clean_scrub_stamp":"2024-01-16T15:24:28.452255+0000","objects_scrubbed":15,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:57.412116+0000","scrub_duration":0.21160910499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"118'2","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.708181+0000","last_change":"2024-01-16T15:24:14.708181+0000","last_active":"2024-01-16T15:24:14.708181+0000","last_peered":"2024-01-16T15:24:14.708181+0000","last_clean":"2024-01-16T15:24:14.708181+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:24:14.708181+0000","last_undegraded":"2024-01-16T15:24:14.708181+0000","last_fullsized":"2024-01-16T15:24:14.708181+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'2","last_scrub_stamp":"2024-01-16T15:24:14.707879+0000","last_deep_scrub":"118'2","last_deep_scrub_stamp":"2024-01-16T15:24:14.707879+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.707879+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:26.465221+0000","scrub_duration":0.0070736139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"118'85","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.059525+0000","last_change":"2024-01-16T15:24:31.059525+0000","last_active":"2024-01-16T15:24:31.059525+0000","last_peered":"2024-01-16T15:24:31.059525+0000","last_clean":"2024-01-16T15:24:31.059525+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:24:31.059525+0000","last_undegraded":"2024-01-16T15:24:31.059525+0000","last_fullsized":"2024-01-16T15:24:31.059525+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'85","last_scrub_stamp":"2024-01-16T15:24:31.059404+0000","last_deep_scrub":"118'85","last_deep_scrub_stamp":"2024-01-16T15:24:31.059404+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.059404+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:38.481467+0000","scrub_duration":0.28150150299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"118'3","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.312794+0000","last_change":"2024-01-16T15:24:20.312794+0000","last_active":"2024-01-16T15:24:20.312794+0000","last_peered":"2024-01-16T15:24:20.312794+0000","last_clean":"2024-01-16T15:24:20.312794+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:24:20.312794+0000","last_undegraded":"2024-01-16T15:24:20.312794+0000","last_fullsized":"2024-01-16T15:24:20.312794+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:20.312507+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:20.312507+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.312507+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:01.823492+0000","scrub_duration":0.0071549860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"118'75","reported_seq":115,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:22.676204+0000","last_change":"2024-01-16T15:24:22.676204+0000","last_active":"2024-01-16T15:24:22.676204+0000","last_peered":"2024-01-16T15:24:22.676204+0000","last_clean":"2024-01-16T15:24:22.676204+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:24:22.676204+0000","last_undegraded":"2024-01-16T15:24:22.676204+0000","last_fullsized":"2024-01-16T15:24:22.676204+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'75","last_scrub_stamp":"2024-01-16T15:24:22.675987+0000","last_deep_scrub":"118'75","last_deep_scrub_stamp":"2024-01-16T15:24:22.675987+0000","last_clean_scrub_stamp":"2024-01-16T15:24:22.675987+0000","objects_scrubbed":14,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:10.831714+0000","scrub_duration":0.18371293499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"118'2","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.419007+0000","last_change":"2024-01-16T15:24:08.419007+0000","last_active":"2024-01-16T15:24:08.419007+0000","last_peered":"2024-01-16T15:24:08.419007+0000","last_clean":"2024-01-16T15:24:08.419007+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:24:08.419007+0000","last_undegraded":"2024-01-16T15:24:08.419007+0000","last_fullsized":"2024-01-16T15:24:08.419007+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'2","last_scrub_stamp":"2024-01-16T15:24:08.418682+0000","last_deep_scrub":"118'2","last_deep_scrub_stamp":"2024-01-16T15:24:08.418682+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.418682+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:20.781341+0000","scrub_duration":0.0080426539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"118'62","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.836484+0000","last_change":"2024-01-16T15:15:39.326288+0000","last_active":"2024-01-16T15:24:04.836484+0000","last_peered":"2024-01-16T15:24:04.836484+0000","last_clean":"2024-01-16T15:24:04.836484+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:24:04.836484+0000","last_undegraded":"2024-01-16T15:24:04.836484+0000","last_fullsized":"2024-01-16T15:24:04.836484+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:38.254793+0000","last_clean_scrub_stamp":"2024-01-16T15:15:38.254793+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.539708+0000","last_change":"2024-01-16T15:24:06.539708+0000","last_active":"2024-01-16T15:24:06.539708+0000","last_peered":"2024-01-16T15:24:06.539708+0000","last_clean":"2024-01-16T15:24:06.539708+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:24:06.539708+0000","last_undegraded":"2024-01-16T15:24:06.539708+0000","last_fullsized":"2024-01-16T15:24:06.539708+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:06.539466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:06.539466+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.539466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:05.228120+0000","scrub_duration":0.0037083429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"118'82","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.843426+0000","last_change":"2024-01-16T15:24:25.843494+0000","last_active":"2024-01-16T15:24:25.843426+0000","last_peered":"2024-01-16T15:24:25.843426+0000","last_clean":"2024-01-16T15:24:25.843426+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:24:25.843426+0000","last_undegraded":"2024-01-16T15:24:25.843426+0000","last_fullsized":"2024-01-16T15:24:25.843426+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'82","last_scrub_stamp":"2024-01-16T15:24:25.843273+0000","last_deep_scrub":"118'82","last_deep_scrub_stamp":"2024-01-16T15:24:25.843273+0000","last_clean_scrub_stamp":"2024-01-16T15:24:25.843273+0000","objects_scrubbed":26,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:04.071362+0000","scrub_duration":0.34204772999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"118'4","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.623234+0000","last_change":"2024-01-16T15:24:01.623234+0000","last_active":"2024-01-16T15:24:01.623234+0000","last_peered":"2024-01-16T15:24:01.623234+0000","last_clean":"2024-01-16T15:24:01.623234+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:24:01.623234+0000","last_undegraded":"2024-01-16T15:24:01.623234+0000","last_fullsized":"2024-01-16T15:24:01.623234+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4","last_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_deep_scrub":"118'4","last_deep_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.623154+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:58.563468+0000","scrub_duration":0.0032134569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"118'92","reported_seq":133,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.150238+0000","last_change":"2024-01-16T15:24:08.150238+0000","last_active":"2024-01-16T15:24:08.150238+0000","last_peered":"2024-01-16T15:24:08.150238+0000","last_clean":"2024-01-16T15:24:08.150238+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:24:08.150238+0000","last_undegraded":"2024-01-16T15:24:08.150238+0000","last_fullsized":"2024-01-16T15:24:08.150238+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'92","last_scrub_stamp":"2024-01-16T15:24:08.149996+0000","last_deep_scrub":"118'92","last_deep_scrub_stamp":"2024-01-16T15:24:08.149996+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.149996+0000","objects_scrubbed":26,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:37.843231+0000","scrub_duration":0.295168926,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.240464+0000","last_change":"2024-01-16T15:24:15.240464+0000","last_active":"2024-01-16T15:24:15.240464+0000","last_peered":"2024-01-16T15:24:15.240464+0000","last_clean":"2024-01-16T15:24:15.240464+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:24:15.240464+0000","last_undegraded":"2024-01-16T15:24:15.240464+0000","last_fullsized":"2024-01-16T15:24:15.240464+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:15.240213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:15.240213+0000","last_clean_scrub_stamp":"2024-01-16T15:24:15.240213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:36.556315+0000","scrub_duration":0.0034155610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"118'90","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:32.886225+0000","last_change":"2024-01-16T15:24:32.886280+0000","last_active":"2024-01-16T15:24:32.886225+0000","last_peered":"2024-01-16T15:24:32.886225+0000","last_clean":"2024-01-16T15:24:32.886225+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:24:32.886225+0000","last_undegraded":"2024-01-16T15:24:32.886225+0000","last_fullsized":"2024-01-16T15:24:32.886225+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'90","last_scrub_stamp":"2024-01-16T15:24:32.886073+0000","last_deep_scrub":"118'90","last_deep_scrub_stamp":"2024-01-16T15:24:32.886073+0000","last_clean_scrub_stamp":"2024-01-16T15:24:32.886073+0000","objects_scrubbed":25,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:44.179455+0000","scrub_duration":0.29493345700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.660042+0000","last_change":"2024-01-16T15:24:02.660042+0000","last_active":"2024-01-16T15:24:02.660042+0000","last_peered":"2024-01-16T15:24:02.660042+0000","last_clean":"2024-01-16T15:24:02.660042+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:24:02.660042+0000","last_undegraded":"2024-01-16T15:24:02.660042+0000","last_fullsized":"2024-01-16T15:24:02.660042+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.659822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:34.891359+0000","scrub_duration":0.0028615590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"118'77","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:22.227309+0000","last_change":"2024-01-16T15:24:22.227483+0000","last_active":"2024-01-16T15:24:22.227309+0000","last_peered":"2024-01-16T15:24:22.227309+0000","last_clean":"2024-01-16T15:24:22.227309+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:24:22.227309+0000","last_undegraded":"2024-01-16T15:24:22.227309+0000","last_fullsized":"2024-01-16T15:24:22.227309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'77","last_scrub_stamp":"2024-01-16T15:24:22.227034+0000","last_deep_scrub":"118'77","last_deep_scrub_stamp":"2024-01-16T15:24:22.227034+0000","last_clean_scrub_stamp":"2024-01-16T15:24:22.227034+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:51.920270+0000","scrub_duration":0.238057777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692036,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.775581+0000","last_change":"2024-01-16T15:24:16.775581+0000","last_active":"2024-01-16T15:24:16.775581+0000","last_peered":"2024-01-16T15:24:16.775581+0000","last_clean":"2024-01-16T15:24:16.775581+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:24:16.775581+0000","last_undegraded":"2024-01-16T15:24:16.775581+0000","last_fullsized":"2024-01-16T15:24:16.775581+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:16.775280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:16.775280+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.775280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:13.481653+0000","scrub_duration":0.0039614919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"118'67","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.597682+0000","last_change":"2024-01-16T15:24:16.597682+0000","last_active":"2024-01-16T15:24:16.597682+0000","last_peered":"2024-01-16T15:24:16.597682+0000","last_clean":"2024-01-16T15:24:16.597682+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:24:16.597682+0000","last_undegraded":"2024-01-16T15:24:16.597682+0000","last_fullsized":"2024-01-16T15:24:16.597682+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'67","last_scrub_stamp":"2024-01-16T15:24:16.597377+0000","last_deep_scrub":"118'67","last_deep_scrub_stamp":"2024-01-16T15:24:16.597377+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.597377+0000","objects_scrubbed":17,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:57.545654+0000","scrub_duration":0.18894488100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"118'3","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.946698+0000","last_change":"2024-01-16T15:24:17.946698+0000","last_active":"2024-01-16T15:24:17.946698+0000","last_peered":"2024-01-16T15:24:17.946698+0000","last_clean":"2024-01-16T15:24:17.946698+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:24:17.946698+0000","last_undegraded":"2024-01-16T15:24:17.946698+0000","last_fullsized":"2024-01-16T15:24:17.946698+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:17.946427+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:17.946427+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.946427+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:26.209326+0000","scrub_duration":0.0069638929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"118'85","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:41.038428+0000","last_change":"2024-01-16T15:24:41.038540+0000","last_active":"2024-01-16T15:24:41.038428+0000","last_peered":"2024-01-16T15:24:41.038428+0000","last_clean":"2024-01-16T15:24:41.038428+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:24:41.038428+0000","last_undegraded":"2024-01-16T15:24:41.038428+0000","last_fullsized":"2024-01-16T15:24:41.038428+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'85","last_scrub_stamp":"2024-01-16T15:24:41.038252+0000","last_deep_scrub":"118'85","last_deep_scrub_stamp":"2024-01-16T15:24:41.038252+0000","last_clean_scrub_stamp":"2024-01-16T15:24:41.038252+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:53.461089+0000","scrub_duration":0.27677636700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"118'31","reported_seq":58,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.451631+0000","last_change":"2024-01-16T15:24:10.451631+0000","last_active":"2024-01-16T15:24:10.451631+0000","last_peered":"2024-01-16T15:24:10.451631+0000","last_clean":"2024-01-16T15:24:10.451631+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:24:10.451631+0000","last_undegraded":"2024-01-16T15:24:10.451631+0000","last_fullsized":"2024-01-16T15:24:10.451631+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'31","last_scrub_stamp":"2024-01-16T15:24:10.451303+0000","last_deep_scrub":"118'31","last_deep_scrub_stamp":"2024-01-16T15:24:10.451303+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.451303+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:06.356152+0000","scrub_duration":0.0039950840000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644,"num_omap_keys":12,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"118'99","reported_seq":143,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:44.709741+0000","last_change":"2024-01-16T15:24:44.709812+0000","last_active":"2024-01-16T15:24:44.709741+0000","last_peered":"2024-01-16T15:24:44.709741+0000","last_clean":"2024-01-16T15:24:44.709741+0000","last_became_active":"2024-01-16T15:15:55.914646+0000","last_became_peered":"2024-01-16T15:15:55.914646+0000","last_unstale":"2024-01-16T15:24:44.709741+0000","last_undegraded":"2024-01-16T15:24:44.709741+0000","last_fullsized":"2024-01-16T15:24:44.709741+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2024-01-16T15:24:44.709536+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2024-01-16T15:24:44.709536+0000","last_clean_scrub_stamp":"2024-01-16T15:24:44.709536+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:41.300338+0000","scrub_duration":0.380922227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.759218+0000","last_change":"2024-01-16T15:24:20.759267+0000","last_active":"2024-01-16T15:24:20.759218+0000","last_peered":"2024-01-16T15:24:20.759218+0000","last_clean":"2024-01-16T15:24:20.759218+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:24:20.759218+0000","last_undegraded":"2024-01-16T15:24:20.759218+0000","last_fullsized":"2024-01-16T15:24:20.759218+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:20.759038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:20.759038+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.759038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:12.067470+0000","scrub_duration":0.003697947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"118'79","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.701538+0000","last_change":"2024-01-16T15:24:14.701607+0000","last_active":"2024-01-16T15:24:14.701538+0000","last_peered":"2024-01-16T15:24:14.701538+0000","last_clean":"2024-01-16T15:24:14.701538+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:24:14.701538+0000","last_undegraded":"2024-01-16T15:24:14.701538+0000","last_fullsized":"2024-01-16T15:24:14.701538+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'79","last_scrub_stamp":"2024-01-16T15:24:14.701344+0000","last_deep_scrub":"118'79","last_deep_scrub_stamp":"2024-01-16T15:24:14.701344+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.701344+0000","objects_scrubbed":28,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:46.392454+0000","scrub_duration":0.30808371000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.903030+0000","last_change":"2024-01-16T15:24:14.903079+0000","last_active":"2024-01-16T15:24:14.903030+0000","last_peered":"2024-01-16T15:24:14.903030+0000","last_clean":"2024-01-16T15:24:14.903030+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:24:14.903030+0000","last_undegraded":"2024-01-16T15:24:14.903030+0000","last_fullsized":"2024-01-16T15:24:14.903030+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:14.902847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:14.902847+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.902847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:18:33.188607+0000","scrub_duration":0.003674142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"118'84","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.428031+0000","last_change":"2024-01-16T15:24:26.428184+0000","last_active":"2024-01-16T15:24:26.428031+0000","last_peered":"2024-01-16T15:24:26.428031+0000","last_clean":"2024-01-16T15:24:26.428031+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:24:26.428031+0000","last_undegraded":"2024-01-16T15:24:26.428031+0000","last_fullsized":"2024-01-16T15:24:26.428031+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'84","last_scrub_stamp":"2024-01-16T15:24:26.427830+0000","last_deep_scrub":"118'84","last_deep_scrub_stamp":"2024-01-16T15:24:26.427830+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.427830+0000","objects_scrubbed":21,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:10.543608+0000","scrub_duration":0.25438750799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.443065+0000","last_change":"2024-01-16T15:24:10.443138+0000","last_active":"2024-01-16T15:24:10.443065+0000","last_peered":"2024-01-16T15:24:10.443065+0000","last_clean":"2024-01-16T15:24:10.443065+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:24:10.443065+0000","last_undegraded":"2024-01-16T15:24:10.443065+0000","last_fullsized":"2024-01-16T15:24:10.443065+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:10.442885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:10.442885+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.442885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:31.758987+0000","scrub_duration":0.003956195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"118'93","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:28.565969+0000","last_change":"2024-01-16T15:24:28.566007+0000","last_active":"2024-01-16T15:24:28.565969+0000","last_peered":"2024-01-16T15:24:28.565969+0000","last_clean":"2024-01-16T15:24:28.565969+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:24:28.565969+0000","last_undegraded":"2024-01-16T15:24:28.565969+0000","last_fullsized":"2024-01-16T15:24:28.565969+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'93","last_scrub_stamp":"2024-01-16T15:24:28.565819+0000","last_deep_scrub":"118'93","last_deep_scrub_stamp":"2024-01-16T15:24:28.565819+0000","last_clean_scrub_stamp":"2024-01-16T15:24:28.565819+0000","objects_scrubbed":27,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:19:37.901880+0000","scrub_duration":0.34082023700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.246601+0000","last_change":"2024-01-16T15:24:11.246654+0000","last_active":"2024-01-16T15:24:11.246601+0000","last_peered":"2024-01-16T15:24:11.246601+0000","last_clean":"2024-01-16T15:24:11.246601+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:24:11.246601+0000","last_undegraded":"2024-01-16T15:24:11.246601+0000","last_fullsized":"2024-01-16T15:24:11.246601+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:11.246380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:11.246380+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.246380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:14.315962+0000","scrub_duration":0.0038743530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"118'97","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:30.981555+0000","last_change":"2024-01-16T15:24:30.981617+0000","last_active":"2024-01-16T15:24:30.981555+0000","last_peered":"2024-01-16T15:24:30.981555+0000","last_clean":"2024-01-16T15:24:30.981555+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:24:30.981555+0000","last_undegraded":"2024-01-16T15:24:30.981555+0000","last_fullsized":"2024-01-16T15:24:30.981555+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:30.981393+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:30.981393+0000","last_clean_scrub_stamp":"2024-01-16T15:24:30.981393+0000","objects_scrubbed":29,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:15.621903+0000","scrub_duration":0.363843851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"118'32","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:09.262161+0000","last_change":"2024-01-16T15:24:09.262216+0000","last_active":"2024-01-16T15:24:09.262161+0000","last_peered":"2024-01-16T15:24:09.262161+0000","last_clean":"2024-01-16T15:24:09.262161+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:24:09.262161+0000","last_undegraded":"2024-01-16T15:24:09.262161+0000","last_fullsized":"2024-01-16T15:24:09.262161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'32","last_scrub_stamp":"2024-01-16T15:24:09.261992+0000","last_deep_scrub":"118'32","last_deep_scrub_stamp":"2024-01-16T15:24:09.261992+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.261992+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:09.421019+0000","scrub_duration":0.0066503719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3846,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"118'94","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.073597+0000","last_change":"2024-01-16T15:24:04.073657+0000","last_active":"2024-01-16T15:24:04.073597+0000","last_peered":"2024-01-16T15:24:04.073597+0000","last_clean":"2024-01-16T15:24:04.073597+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:24:04.073597+0000","last_undegraded":"2024-01-16T15:24:04.073597+0000","last_fullsized":"2024-01-16T15:24:04.073597+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.073487+0000","objects_scrubbed":32,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:45.014652+0000","scrub_duration":0.28876764900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.746241+0000","last_change":"2024-01-16T15:24:11.746367+0000","last_active":"2024-01-16T15:24:11.746241+0000","last_peered":"2024-01-16T15:24:11.746241+0000","last_clean":"2024-01-16T15:24:11.746241+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:24:11.746241+0000","last_undegraded":"2024-01-16T15:24:11.746241+0000","last_fullsized":"2024-01-16T15:24:11.746241+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:11.746034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:11.746034+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.746034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:13.384227+0000","scrub_duration":0.0037103750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"118'66","reported_seq":104,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:19.148038+0000","last_change":"2024-01-16T15:24:19.148119+0000","last_active":"2024-01-16T15:24:19.148038+0000","last_peered":"2024-01-16T15:24:19.148038+0000","last_clean":"2024-01-16T15:24:19.148038+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:24:19.148038+0000","last_undegraded":"2024-01-16T15:24:19.148038+0000","last_fullsized":"2024-01-16T15:24:19.148038+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'66","last_scrub_stamp":"2024-01-16T15:24:19.147855+0000","last_deep_scrub":"118'66","last_deep_scrub_stamp":"2024-01-16T15:24:19.147855+0000","last_clean_scrub_stamp":"2024-01-16T15:24:19.147855+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:29.163868+0000","scrub_duration":0.182029521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.536712+0000","last_change":"2024-01-16T15:24:05.536760+0000","last_active":"2024-01-16T15:24:05.536712+0000","last_peered":"2024-01-16T15:24:05.536712+0000","last_clean":"2024-01-16T15:24:05.536712+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:24:05.536712+0000","last_undegraded":"2024-01-16T15:24:05.536712+0000","last_fullsized":"2024-01-16T15:24:05.536712+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:05.536530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:05.536530+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.536530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:03.108287+0000","scrub_duration":0.0035509750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"118'119","reported_seq":275,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.491584+0000","last_change":"2024-01-16T15:24:06.491710+0000","last_active":"2024-01-16T15:24:06.491584+0000","last_peered":"2024-01-16T15:24:06.491584+0000","last_clean":"2024-01-16T15:24:06.491584+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:24:06.491584+0000","last_undegraded":"2024-01-16T15:24:06.491584+0000","last_fullsized":"2024-01-16T15:24:06.491584+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'119","last_scrub_stamp":"2024-01-16T15:24:06.491352+0000","last_deep_scrub":"118'119","last_deep_scrub_stamp":"2024-01-16T15:24:06.491352+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.491352+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:03.082154+0000","scrub_duration":0.061931657000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452737,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5479,"ondisk_log_size":5479,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1252565,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":489,"num_write_kb":1461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5644,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2505130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542479,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2100980,"kb_used_data":2025452,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92270860,"statfs":{"total":96636764160,"available":94485360640,"internally_reserved":0,"allocated":2074062848,"data_stored":2073213012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2170000000000001,"15min":1.248},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.57799999999999996},"max":{"1min":1.573,"5min":1.6679999999999999,"15min":13.566000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3169999999999999,"15min":1.2949999999999999},"min":{"1min":0.79000000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.579,"5min":1.984,"15min":10.728},"last":1.4770000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.107,"15min":1.155},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.627},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":13.375},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.121,"5min":1.155,"15min":1.1359999999999999},"min":{"1min":0.86099999999999999,"5min":0.67800000000000005,"15min":0.66200000000000003},"max":{"1min":1.661,"5min":1.6799999999999999,"15min":2.056},"last":1.657}]},{"osd":2,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.272,"15min":1.2490000000000001},"min":{"1min":0.99399999999999999,"5min":0.76400000000000001,"15min":0.72199999999999998},"max":{"1min":1.611,"5min":1.7969999999999999,"15min":2.145},"last":1.361},{"interface":"front","average":{"1min":1.129,"5min":1.171,"15min":1.1599999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.623,"5min":1.724,"15min":1.9630000000000001},"last":1.554}]},{"osd":3,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.284,"15min":1.296},"min":{"1min":0.83399999999999996,"5min":0.82999999999999996,"15min":0.67000000000000004},"max":{"1min":1.819,"5min":2.27,"15min":13.657999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.472,"15min":1.4259999999999999},"min":{"1min":1.1020000000000001,"5min":1.0029999999999999,"15min":0.76500000000000001},"max":{"1min":1.78,"5min":2.3119999999999998,"15min":2.4319999999999999},"last":1.732}]},{"osd":4,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2410000000000001,"15min":1.26},"min":{"1min":0.92600000000000005,"5min":0.57599999999999996,"15min":0.56999999999999995},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.421,"15min":1.4510000000000001},"min":{"1min":0.76900000000000002,"5min":0.72899999999999998,"15min":0.69799999999999995},"max":{"1min":1.758,"5min":1.7829999999999999,"15min":13.782999999999999},"last":1.583}]},{"osd":5,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.452,"15min":1.4330000000000001},"min":{"1min":0.85299999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.843,"5min":2.0569999999999999,"15min":2.2320000000000002},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.462,"15min":1.448},"min":{"1min":0.98599999999999999,"5min":0.66000000000000003,"15min":0.626},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.1890000000000001},"last":1.4139999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4419999999999999,"15min":1.4299999999999999},"min":{"1min":1.0429999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.141,"5min":2.141,"15min":2.327},"last":1.623},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4379999999999999,"15min":1.4950000000000001},"min":{"1min":0.93700000000000006,"5min":0.63400000000000001,"15min":0.61599999999999999},"max":{"1min":2.274,"5min":2.274,"15min":13.725},"last":1.266}]}]},{"osd":6,"up_from":14,"seq":60129542478,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477736,"kb_used_data":1406368,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92894104,"statfs":{"total":96636764160,"available":95123562496,"internally_reserved":0,"allocated":1440120832,"data_stored":1439281876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1699999999999999,"15min":1.1739999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58099999999999996},"max":{"1min":1.6399999999999999,"5min":1.764,"15min":3.9289999999999998},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.1950000000000001,"15min":1.246},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.41599999999999998},"max":{"1min":1.645,"5min":1.7130000000000001,"15min":8.0630000000000006},"last":1.1679999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2370000000000001,"15min":1.282},"min":{"1min":0.82299999999999995,"5min":0.63700000000000001,"15min":0.497},"max":{"1min":1.5880000000000001,"5min":1.7030000000000001,"15min":10.518000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.302,"15min":1.286},"min":{"1min":0.95199999999999996,"5min":0.65000000000000002,"15min":0.53100000000000003},"max":{"1min":1.696,"5min":1.887,"15min":7.2919999999999998},"last":1.4730000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2370000000000001,"15min":1.298},"min":{"1min":0.755,"5min":0.755,"15min":0.47099999999999997},"max":{"1min":1.5840000000000001,"5min":1.784,"15min":6.3040000000000003},"last":1.25},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2749999999999999,"15min":1.423},"min":{"1min":0.83099999999999996,"5min":0.72599999999999998,"15min":0.57099999999999995},"max":{"1min":1.6910000000000001,"5min":1.8240000000000001,"15min":15.282},"last":1.5980000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.343,"15min":1.357},"min":{"1min":1.0920000000000001,"5min":0.71899999999999997,"15min":0.64100000000000001},"max":{"1min":1.784,"5min":1.9490000000000001,"15min":4.1109999999999998},"last":1.571},{"interface":"front","average":{"1min":1.304,"5min":1.3839999999999999,"15min":1.458},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.50900000000000001},"max":{"1min":1.621,"5min":1.7370000000000001,"15min":9.0960000000000001},"last":1.6599999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.3640000000000001,"15min":1.3759999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.49299999999999999},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":5.4669999999999996},"last":1.403},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3580000000000001,"15min":1.413},"min":{"1min":0.95799999999999996,"5min":0.81799999999999995,"15min":0.41199999999999998},"max":{"1min":1.651,"5min":1.7989999999999999,"15min":5.3360000000000003},"last":1.5269999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3380000000000001,"15min":1.403},"min":{"1min":0.89200000000000002,"5min":0.65700000000000003,"15min":0.436},"max":{"1min":1.7370000000000001,"5min":1.9239999999999999,"15min":10.388999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.518,"15min":1.55},"min":{"1min":1.0660000000000001,"5min":1.016,"15min":0.5},"max":{"1min":2.0070000000000001,"5min":2.012,"15min":7.9640000000000004},"last":1.7090000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5449999999999999,"15min":1.589},"min":{"1min":1.1020000000000001,"5min":0.98399999999999999,"15min":0.48899999999999999},"max":{"1min":1.738,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.74},{"interface":"front","average":{"1min":1.452,"5min":1.474,"15min":1.6080000000000001},"min":{"1min":0.97399999999999998,"5min":0.84199999999999997,"15min":0.58499999999999996},"max":{"1min":1.8540000000000001,"5min":1.984,"15min":16.175999999999998},"last":2.0550000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542479,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174568,"kb_used_data":1107360,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93197272,"statfs":{"total":96636764160,"available":95434006528,"internally_reserved":0,"allocated":1133936640,"data_stored":1133098096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.278,"15min":1.216},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.532,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.151,"5min":1.3120000000000001,"15min":1.236},"min":{"1min":0.81899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.657,"5min":12.922000000000001,"15min":12.922000000000001},"last":0.83799999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3200000000000001,"15min":1.3169999999999999},"min":{"1min":0.89700000000000002,"5min":0.66300000000000003,"15min":0.63900000000000001},"max":{"1min":1.6879999999999999,"5min":5.2249999999999996,"15min":8.1820000000000004},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.175,"15min":1.173},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.536,"5min":2.0659999999999998,"15min":2.597},"last":1.6619999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.268,"15min":1.264},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6459999999999999,"5min":1.712,"15min":3.0470000000000002},"last":1.323},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2250000000000001,"15min":1.2669999999999999},"min":{"1min":0.72299999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6359999999999999,"5min":1.756,"15min":8.0619999999999994},"last":1.0629999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.401,"15min":1.484},"min":{"1min":0.83099999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7430000000000001,"5min":5.157,"15min":16.931999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.419,"15min":1.494},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":2.6960000000000002,"15min":19.111999999999998},"last":1.218}]},{"osd":4,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.1930000000000001,"15min":1.2589999999999999},"min":{"1min":0.71899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.7549999999999999,"5min":1.7689999999999999,"15min":4.8490000000000002},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3540000000000001,"15min":1.411},"min":{"1min":0.84899999999999998,"5min":0.63300000000000001,"15min":0.57399999999999995},"max":{"1min":1.7569999999999999,"5min":5.0810000000000004,"15min":8.3550000000000004},"last":1.512}]},{"osd":6,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4119999999999999,"15min":1.5069999999999999},"min":{"1min":0.79300000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.716,"5min":1.9530000000000001,"15min":17.419},"last":1.429},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.415,"15min":1.5269999999999999},"min":{"1min":0.83599999999999997,"5min":0.69899999999999995,"15min":0.69799999999999995},"max":{"1min":1.7170000000000001,"5min":2.2959999999999998,"15min":16.262},"last":1.698}]},{"osd":7,"last update":"Tue Jan 16 15:23:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.6040000000000001},"min":{"1min":1.1359999999999999,"5min":0.68400000000000005,"15min":0.67000000000000004},"max":{"1min":1.889,"5min":4.0380000000000003,"15min":12.385999999999999},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.431,"15min":1.512},"min":{"1min":0.73999999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6919999999999999,"5min":1.796,"15min":5.0780000000000003},"last":1.5700000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542480,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936484,"kb_used_data":1865116,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435356,"statfs":{"total":96636764160,"available":94653804544,"internally_reserved":0,"allocated":1909878784,"data_stored":1909043912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2649999999999999,"15min":1.2569999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.63300000000000001},"max":{"1min":1.7669999999999999,"5min":1.774,"15min":3.3159999999999998},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.163,"5min":1.129,"15min":1.1919999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.47499999999999998},"max":{"1min":1.77,"5min":1.77,"15min":5.7430000000000003},"last":1.25}]},{"osd":1,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.254,"15min":1.302},"min":{"1min":0.81799999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":3.7850000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2170000000000001,"15min":1.234},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.57899999999999996},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":4.6760000000000002},"last":1.6000000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2010000000000001,"15min":1.2310000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.69199999999999995},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":13.731999999999999},"last":1.149},{"interface":"front","average":{"1min":1.399,"5min":1.3620000000000001,"15min":1.367},"min":{"1min":0.91900000000000004,"5min":0.79400000000000004,"15min":0.73199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7729999999999999,"15min":2.2570000000000001},"last":1.528}]},{"osd":3,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.341,"15min":1.341},"min":{"1min":1.0009999999999999,"5min":0.69999999999999996,"15min":0.59899999999999998},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":4.5789999999999997},"last":0.875},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3280000000000001,"15min":1.363},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.71699999999999997},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":13.914},"last":1.3420000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.329,"15min":1.3580000000000001},"min":{"1min":0.90200000000000002,"5min":0.80500000000000005,"15min":0.47299999999999998},"max":{"1min":1.911,"5min":2.0190000000000001,"15min":10.211},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3720000000000001,"15min":1.4079999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.58299999999999996},"max":{"1min":1.6399999999999999,"5min":1.7669999999999999,"15min":4.673},"last":1.4370000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3779999999999999,"15min":1.4119999999999999},"min":{"1min":0.97299999999999998,"5min":0.83099999999999996,"15min":0.60699999999999998},"max":{"1min":1.74,"5min":1.7410000000000001,"15min":13.973000000000001},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.5349999999999999},"min":{"1min":1.246,"5min":0.85099999999999998,"15min":0.52400000000000002},"max":{"1min":1.8520000000000001,"5min":1.9019999999999999,"15min":3.0979999999999999},"last":1.6950000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.5009999999999999,"15min":1.536},"min":{"1min":0.878,"5min":0.878,"15min":0.53500000000000003},"max":{"1min":1.7709999999999999,"5min":2.0539999999999998,"15min":4.7640000000000002},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.5229999999999999,"15min":1.5669999999999999},"min":{"1min":0.97699999999999998,"5min":0.95999999999999996,"15min":0.55200000000000005},"max":{"1min":1.7989999999999999,"5min":2.0179999999999998,"15min":13.852},"last":1.6459999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542479,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1832764,"kb_used_data":1751668,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":92539076,"statfs":{"total":96636764160,"available":94760013824,"internally_reserved":0,"allocated":1793708032,"data_stored":1792851922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94699999999999995,"15min":0.95599999999999996},"min":{"1min":0.71299999999999997,"5min":0.626,"15min":0.34300000000000003},"max":{"1min":1.353,"5min":2.605,"15min":2.9129999999999998},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1739999999999999,"15min":1.143},"min":{"1min":0.438,"5min":0.438,"15min":0.29699999999999999},"max":{"1min":1.2849999999999999,"5min":4.0389999999999997,"15min":12.292999999999999},"last":1.381}]},{"osd":1,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.069,"15min":0.98899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.33100000000000002},"max":{"1min":1.5580000000000001,"5min":12.335000000000001,"15min":12.335000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0960000000000001,"15min":1.133},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.315},"max":{"1min":1.712,"5min":4.6529999999999996,"15min":10.638},"last":1.4359999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.206,"15min":1.151},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":8.3740000000000006,"15min":8.3740000000000006},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.248,"15min":1.228},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.46000000000000002},"max":{"1min":1.6200000000000001,"5min":4.2629999999999999,"15min":4.2629999999999999},"last":1.2370000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.403,"15min":1.345},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.39300000000000002},"max":{"1min":1.76,"5min":15.512,"15min":15.512},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.411,"15min":1.4119999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.52100000000000002},"max":{"1min":1.9399999999999999,"5min":11.743,"15min":14.250999999999999},"last":1.3360000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4710000000000001,"15min":1.417},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7470000000000001,"5min":8.4290000000000003,"15min":8.4290000000000003},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.419,"15min":1.409},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.42099999999999999},"max":{"1min":1.8440000000000001,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":1.5329999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3839999999999999,"15min":1.419},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.41499999999999998},"max":{"1min":1.8879999999999999,"5min":3.956,"15min":12.675000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.5669999999999999,"15min":1.49},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.54900000000000004},"max":{"1min":1.849,"5min":11.667,"15min":11.667},"last":1.494}]},{"osd":7,"last update":"Tue Jan 16 15:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.569,"15min":1.5569999999999999},"min":{"1min":1.113,"5min":1.097,"15min":0.53700000000000003},"max":{"1min":1.9279999999999999,"5min":11.042999999999999,"15min":11.042999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.474,"15min":1.5289999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.435},"max":{"1min":1.954,"5min":3.907,"15min":12.836},"last":1.5600000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542479,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1211492,"kb_used_data":1140124,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93160348,"statfs":{"total":96636764160,"available":95396196352,"internally_reserved":0,"allocated":1167486976,"data_stored":1166655701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89200000000000002,"15min":0.89400000000000002},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.311},"max":{"1min":1.6759999999999999,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.995,"5min":1.0449999999999999,"15min":1.056},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.496,"5min":1.5549999999999999,"15min":3.52},"last":1.1359999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1599999999999999,"15min":1.0880000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.504,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.1060000000000001,"15min":1.1080000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.7729999999999999,"5min":1.8109999999999999,"15min":2.9900000000000002},"last":1.413}]},{"osd":3,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.036,"15min":1.048},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.45700000000000002},"max":{"1min":1.8320000000000001,"5min":1.8360000000000001,"15min":6.1260000000000003},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.214,"15min":1.226},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.47999999999999998},"max":{"1min":1.6950000000000001,"5min":2.911,"15min":6.2510000000000003},"last":1.492}]},{"osd":4,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.423,"15min":1.423},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.68400000000000005},"max":{"1min":1.7989999999999999,"5min":3.012,"15min":3.012},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.3939999999999999,"15min":1.3959999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.71099999999999997},"max":{"1min":2.4020000000000001,"5min":3.54,"15min":3.54},"last":1.6459999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3660000000000001,"15min":1.389},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":1.774,"5min":3.6179999999999999,"15min":8.625},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.944,"5min":1.944,"15min":6.3499999999999996},"last":1.2549999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3839999999999999,"15min":1.377},"min":{"1min":1.0429999999999999,"5min":0.96299999999999997,"15min":0.71499999999999997},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":6.4729999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.407,"15min":1.4370000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.71099999999999997},"max":{"1min":2.125,"5min":2.8260000000000001,"15min":3.2469999999999999},"last":1.331}]},{"osd":7,"last update":"Tue Jan 16 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4750000000000001,"15min":1.4910000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.704,"5min":3.431,"15min":3.7290000000000001},"last":1.544},{"interface":"front","average":{"1min":1.403,"5min":1.478,"15min":1.492},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0379999999999998,"5min":2.3050000000000002,"15min":6.4249999999999998},"last":1.577}]}]},{"osd":1,"up_from":14,"seq":60129542479,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412264,"kb_used_data":1336736,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92959576,"statfs":{"total":96636764160,"available":95190605824,"internally_reserved":0,"allocated":1368817664,"data_stored":1367982226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.94499999999999995,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.33400000000000002},"max":{"1min":1.3120000000000001,"5min":1.4079999999999999,"15min":8.8499999999999996},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.147,"15min":1.1619999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"max":{"1min":1.5089999999999999,"5min":1.5940000000000001,"15min":8.9380000000000006},"last":1.0189999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.34899999999999998},"max":{"1min":1.2549999999999999,"5min":1.456,"15min":3.4460000000000002},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.123,"15min":1.147},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5840000000000001,"5min":1.677,"15min":2.9660000000000002},"last":1.2549999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.2050000000000001,"15min":1.234},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.45400000000000001},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":3.282},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.1180000000000001,"15min":1.135},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.47099999999999997},"max":{"1min":1.655,"5min":1.6830000000000001,"15min":8.6910000000000007},"last":1.3140000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.335,"15min":1.3600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.716,"5min":1.8109999999999999,"15min":4.2480000000000002},"last":1.159},{"interface":"front","average":{"1min":1.226,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.57899999999999996},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":4.7640000000000002},"last":0.90600000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.50800000000000001},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3680000000000003},"last":1.206},{"interface":"front","average":{"1min":1.421,"5min":1.452,"15min":1.476},"min":{"1min":1.0569999999999999,"5min":0.77500000000000002,"15min":0.66300000000000003},"max":{"1min":1.772,"5min":1.841,"15min":5.4610000000000003},"last":1.968}]},{"osd":6,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4350000000000001,"15min":1.466},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":4.7779999999999996},"last":1.417},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.365,"15min":1.419},"min":{"1min":0.746,"5min":0.746,"15min":0.56899999999999995},"max":{"1min":2.339,"5min":2.339,"15min":4.8300000000000001},"last":1.3879999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4510000000000001,"15min":1.4830000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.58999999999999997},"max":{"1min":2.0510000000000002,"5min":2.081,"15min":6.7990000000000004},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.484,"15min":1.544},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.198,"5min":2.198,"15min":5.1260000000000003},"last":1.542}]}]},{"osd":0,"up_from":14,"seq":60129542480,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2036608,"kb_used_data":1965240,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92335232,"statfs":{"total":96636764160,"available":94551277568,"internally_reserved":0,"allocated":2012405760,"data_stored":2011545811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.34899999999999998},"max":{"1min":1.381,"5min":2.387,"15min":2.7320000000000002},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.0269999999999999,"15min":1.0680000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.29399999999999998},"max":{"1min":1.3720000000000001,"5min":1.694,"15min":9.4499999999999993},"last":1.373}]},{"osd":2,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2490000000000001,"15min":1.3029999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.45100000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":9.5579999999999998},"last":1.2},{"interface":"front","average":{"1min":1.119,"5min":1.149,"15min":1.1459999999999999},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.40200000000000002},"max":{"1min":1.5960000000000001,"5min":2.4359999999999999,"15min":5.3159999999999998},"last":1.0960000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0489999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46600000000000003},"max":{"1min":1.335,"5min":1.5940000000000001,"15min":10.83},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1659999999999999,"15min":1.2529999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.48199999999999998},"max":{"1min":1.5409999999999999,"5min":1.99,"15min":9.6029999999999998},"last":1.0249999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3260000000000001,"15min":1.306},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.63400000000000001},"max":{"1min":1.641,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.254},{"interface":"front","average":{"1min":1.2,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.67200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":10.986000000000001},"last":1.288}]},{"osd":5,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":1.0089999999999999,"5min":0.78700000000000003,"15min":0.70499999999999996},"max":{"1min":1.665,"5min":3.0259999999999998,"15min":3.0259999999999998},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":1.514,"5min":1.667,"15min":14.561999999999999},"last":1.147}]},{"osd":6,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.47,"15min":1.522},"min":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.76600000000000001},"max":{"1min":1.6639999999999999,"5min":2.2069999999999999,"15min":14.958},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.506,"15min":1.48},"min":{"1min":1.069,"5min":0.89500000000000002,"15min":0.68500000000000005},"max":{"1min":1.7270000000000001,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.635}]},{"osd":7,"last update":"Tue Jan 16 15:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4890000000000001,"15min":1.5640000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.7869999999999999,"5min":2.129,"15min":15.005000000000001},"last":1.7310000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.5089999999999999,"15min":1.552},"min":{"1min":1.028,"5min":0.93400000000000005,"15min":0.70699999999999996},"max":{"1min":1.9279999999999999,"5min":3.0590000000000002,"15min":13.718},"last":1.5409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1248033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1247991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2009083904,"data_stored":2009072258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166020608,"data_stored":1166016703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908416512,"data_stored":1908408760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132466176,"data_stored":1132462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:24:46.324 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:46.325 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:46.325 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:46.325 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:46.326 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:15:38.254793+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=15, tm_sec=38, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=23, tm_sec=58, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:24:46.326 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:25:06.327 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:25:06.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.542+0000 7fa619400640 1 Processor -- start 2024-01-16T15:25:06.540 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.543+0000 7fa619400640 1 -- start start 2024-01-16T15:25:06.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.544+0000 7fa619400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 0x7fa61415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.544+0000 7fa619400640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa614070c40 con 0x7fa6140cf290 2024-01-16T15:25:06.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.544+0000 7fa619400640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61406c470 con 0x7fa6140b3ae0 2024-01-16T15:25:06.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.544+0000 7fa619400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa614071450 con 0x7fa6140ae5a0 2024-01-16T15:25:06.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa613200640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa6140cf290 0x7fa6140b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42226/0 (socket says 172.21.15.141:42226) 2024-01-16T15:25:06.542 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa613200640 1 -- 172.21.15.141:0/2593456380 learned_addr learned my addr 172.21.15.141:0/2593456380 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:25:06.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709389476 0 0) 0x7fa614070c40 con 0x7fa6140cf290 2024-01-16T15:25:06.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa612800640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 0x7fa61415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f4003520 con 0x7fa6140cf290 2024-01-16T15:25:06.546 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 <== mon.0 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867696406 0 0) 0x7fa5f4003520 con 0x7fa6140cf290 2024-01-16T15:25:06.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 msgr2=0x7fa61415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:06.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 0x7fa61415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 >> v1:172.21.15.188:6789/0 conn(0x7fa6140b3ae0 legacy=0x7fa61406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:06.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61406c750 con 0x7fa6140cf290 2024-01-16T15:25:06.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 <== mon.0 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 413198202 0 0) 0x7fa60c001670 con 0x7fa6140cf290 2024-01-16T15:25:06.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 >> v1:172.21.15.141:6789/0 conn(0x7fa6140cf290 legacy=0x7fa6140b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:06.547 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.545+0000 7fa611e00640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 0x7fa5f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa611e00640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4003fb0 0x7fa5f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa611e00640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5f4004d00 0x7fa5f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa614071450 con 0x7fa5f4004d00 2024-01-16T15:25:06.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f400bba0 con 0x7fa5f4003fb0 2024-01-16T15:25:06.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f400bd60 con 0x7fa6140ae5a0 2024-01-16T15:25:06.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa613200640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5f4004d00 0x7fa5f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.548 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa612800640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 0x7fa5f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa609c00640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4003fb0 0x7fa5f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa613200640 1 -- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 msgr2=0x7fa5f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:06.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa613200640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa6140ae5a0 0x7fa5f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa613200640 1 -- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4003fb0 msgr2=0x7fa5f40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:06.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa613200640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4003fb0 0x7fa5f40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.546+0000 7fa613200640 1 -- 172.21.15.141:0/2593456380 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f4003520 con 0x7fa5f4004d00 2024-01-16T15:25:06.549 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.547+0000 7fa613200640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5f4004d00 0x7fa5f4005180 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fa60c007fd0 tx=0x7fa5f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c272a396df1345 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c005ad0 con 0x7fa5f4004d00 2024-01-16T15:25:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c005c50 con 0x7fa5f4004d00 2024-01-16T15:25:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa611e00640 1 -- 172.21.15.141:0/2593456380 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c02c850 con 0x7fa5f4004d00 2024-01-16T15:25:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa619400640 1 -- 172.21.15.141:0/2593456380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5f4004d00 msgr2=0x7fa5f4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa619400640 1 --2- 172.21.15.141:0/2593456380 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa5f4004d00 0x7fa5f4005180 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fa60c007fd0 tx=0x7fa5f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa613200640 1 -- 172.21.15.141:0/2593456380 reap_dead start 2024-01-16T15:25:06.550 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa619400640 1 -- 172.21.15.141:0/2593456380 shutdown_connections 2024-01-16T15:25:06.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa619400640 1 -- 172.21.15.141:0/2593456380 >> 172.21.15.141:0/2593456380 conn(0x7fa61406a800 msgr2=0x7fa61406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:06.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa619400640 1 -- 172.21.15.141:0/2593456380 shutdown_connections 2024-01-16T15:25:06.551 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.548+0000 7fa619400640 1 -- 172.21.15.141:0/2593456380 wait complete. 2024-01-16T15:25:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.550+0000 7fa619400640 1 Processor -- start 2024-01-16T15:25:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.550+0000 7fa619400640 1 -- start start 2024-01-16T15:25:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.551+0000 7fa619400640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa5f4004d00 0x7fa61414c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.551+0000 7fa619400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa61415f1c0 0x7fa61415f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.551+0000 7fa619400640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6140cba60 0x7fa6140cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.554 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.551+0000 7fa619400640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61414c700 con 0x7fa6140cba60 2024-01-16T15:25:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.551+0000 7fa619400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61415fb80 con 0x7fa61415f1c0 2024-01-16T15:25:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.551+0000 7fa619400640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6140cd3e0 con 0x7fa5f4004d00 2024-01-16T15:25:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.551+0000 7fa613200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6140cba60 0x7fa6140cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6140cba60 0x7fa6140cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37100/0 (socket says 172.21.15.141:37100) 2024-01-16T15:25:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 -- 172.21.15.141:0/3310132724 learned_addr learned my addr 172.21.15.141:0/3310132724 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:25:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa609c00640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa61415f1c0 0x7fa61415f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.555 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa612800640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa5f4004d00 0x7fa61414c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 -- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa5f4004d00 msgr2=0x7fa61414c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa5f4004d00 0x7fa61414c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 -- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa61415f1c0 msgr2=0x7fa61415f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa61415f1c0 0x7fa61415f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 -- 172.21.15.141:0/3310132724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6140cd870 con 0x7fa6140cba60 2024-01-16T15:25:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa609c00640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa61415f1c0 0x7fa61415f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:06.556 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.552+0000 7fa613200640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6140cba60 0x7fa6140cbee0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fa60c02c4c0 tx=0x7fa61415d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:06.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.553+0000 7fa60b400640 1 -- 172.21.15.141:0/3310132724 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c03c040 con 0x7fa6140cba60 2024-01-16T15:25:06.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.553+0000 7fa60b400640 1 -- 172.21.15.141:0/3310132724 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c02cc60 con 0x7fa6140cba60 2024-01-16T15:25:06.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.553+0000 7fa60b400640 1 -- 172.21.15.141:0/3310132724 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c001c10 con 0x7fa6140cba60 2024-01-16T15:25:06.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.553+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6140cc7d0 con 0x7fa6140cba60 2024-01-16T15:25:06.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.553+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6140ccd90 con 0x7fa6140cba60 2024-01-16T15:25:06.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.554+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6140675d0 con 0x7fa6140cba60 2024-01-16T15:25:06.557 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.555+0000 7fa60b400640 1 -- 172.21.15.141:0/3310132724 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fa60c032070 con 0x7fa6140cba60 2024-01-16T15:25:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.556+0000 7fa60b400640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5ec06c550 0x7fa5ec06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.556+0000 7fa60b400640 1 -- 172.21.15.141:0/3310132724 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa60c0a95d0 con 0x7fa6140cba60 2024-01-16T15:25:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.556+0000 7fa612800640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5ec06c550 0x7fa5ec06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:06.558 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.557+0000 7fa612800640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5ec06c550 0x7fa5ec06ea10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa604004330 tx=0x7fa60400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:06.559 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.560+0000 7fa60b400640 1 -- 172.21.15.141:0/3310132724 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa60c075e90 con 0x7fa6140cba60 2024-01-16T15:25:06.746 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.749+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 --> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa614070550 con 0x7fa5ec06c550 2024-01-16T15:25:06.769 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.772+0000 7fa60b400640 1 -- 172.21.15.141:0/3310132724 <== mgr.4106 v2:172.21.15.188:6832/4065338852 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291448 (secure 0 0 0) 0x7fa60400f050 con 0x7fa5ec06c550 2024-01-16T15:25:06.774 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:25:06.776 INFO:teuthology.orchestra.run.smithi141.stderr:dumped all 2024-01-16T15:25:06.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5ec06c550 msgr2=0x7fa5ec06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:06.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5ec06c550 0x7fa5ec06ea10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa604004330 tx=0x7fa60400b040 comp rx=0 tx=0).stop 2024-01-16T15:25:06.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6140cba60 msgr2=0x7fa6140cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:06.779 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6140cba60 0x7fa6140cbee0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fa60c02c4c0 tx=0x7fa61415d9b0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 shutdown_connections 2024-01-16T15:25:06.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7fa5ec06c550 0x7fa5ec06ea10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6140cba60 0x7fa6140cbee0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa61415f1c0 0x7fa61415f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.780 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 --2- 172.21.15.141:0/3310132724 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7fa5f4004d00 0x7fa61414c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:06.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.777+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 >> 172.21.15.141:0/3310132724 conn(0x7fa61406a800 msgr2=0x7fa614159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:06.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.778+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 shutdown_connections 2024-01-16T15:25:06.781 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:06.778+0000 7fa619400640 1 -- 172.21.15.141:0/3310132724 wait complete. 2024-01-16T15:25:06.795 INFO:teuthology.orchestra.run.smithi141.stdout:{"pg_ready":true,"pg_map":{"version":954,"stamp":"2024-01-16T15:25:05.149062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444295859,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":945,"num_read_kb":3145889,"num_write":2255,"num_write_kb":6295373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5644,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6202,"ondisk_log_size":6202,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13187056,"kb_used_data":12598064,"kb_used_omap":12,"kb_used_meta":588915,"kb_avail":741787664,"statfs":{"total":773094113280,"available":759590567936,"internally_reserved":0,"allocated":12900417536,"data_stored":12893672934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":603049522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":189,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004836"},"pg_stats":[{"pgid":"17.3f","version":"118'98","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:47.090273+0000","last_change":"2024-01-16T15:24:47.090336+0000","last_active":"2024-01-16T15:24:47.090273+0000","last_peered":"2024-01-16T15:24:47.090273+0000","last_clean":"2024-01-16T15:24:47.090273+0000","last_became_active":"2024-01-16T15:15:39.328780+0000","last_became_peered":"2024-01-16T15:15:39.328780+0000","last_unstale":"2024-01-16T15:24:47.090273+0000","last_undegraded":"2024-01-16T15:24:47.090273+0000","last_fullsized":"2024-01-16T15:24:47.090273+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'98","last_scrub_stamp":"2024-01-16T15:24:47.090091+0000","last_deep_scrub":"118'98","last_deep_scrub_stamp":"2024-01-16T15:24:47.090091+0000","last_clean_scrub_stamp":"2024-01-16T15:24:47.090091+0000","objects_scrubbed":28,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:13.302747+0000","scrub_duration":0.31246212600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"118'95","reported_seq":136,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:29.567524+0000","last_change":"2024-01-16T15:24:29.567586+0000","last_active":"2024-01-16T15:24:29.567524+0000","last_peered":"2024-01-16T15:24:29.567524+0000","last_clean":"2024-01-16T15:24:29.567524+0000","last_became_active":"2024-01-16T15:15:39.327341+0000","last_became_peered":"2024-01-16T15:15:39.327341+0000","last_unstale":"2024-01-16T15:24:29.567524+0000","last_undegraded":"2024-01-16T15:24:29.567524+0000","last_fullsized":"2024-01-16T15:24:29.567524+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:29.567320+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:29.567320+0000","last_clean_scrub_stamp":"2024-01-16T15:24:29.567320+0000","objects_scrubbed":26,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:03.170389+0000","scrub_duration":0.35401311600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"118'94","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:46.028283+0000","last_change":"2024-01-16T15:24:46.028324+0000","last_active":"2024-01-16T15:24:46.028283+0000","last_peered":"2024-01-16T15:24:46.028283+0000","last_clean":"2024-01-16T15:24:46.028283+0000","last_became_active":"2024-01-16T15:15:39.324422+0000","last_became_peered":"2024-01-16T15:15:39.324422+0000","last_unstale":"2024-01-16T15:24:46.028283+0000","last_undegraded":"2024-01-16T15:24:46.028283+0000","last_fullsized":"2024-01-16T15:24:46.028283+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:46.028168+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:46.028168+0000","last_clean_scrub_stamp":"2024-01-16T15:24:46.028168+0000","objects_scrubbed":27,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:28:02.388531+0000","scrub_duration":0.33620931599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"118'91","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.504390+0000","last_change":"2024-01-16T15:24:31.504446+0000","last_active":"2024-01-16T15:24:31.504390+0000","last_peered":"2024-01-16T15:24:31.504390+0000","last_clean":"2024-01-16T15:24:31.504390+0000","last_became_active":"2024-01-16T15:15:55.875179+0000","last_became_peered":"2024-01-16T15:15:55.875179+0000","last_unstale":"2024-01-16T15:24:31.504390+0000","last_undegraded":"2024-01-16T15:24:31.504390+0000","last_fullsized":"2024-01-16T15:24:31.504390+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'91","last_scrub_stamp":"2024-01-16T15:24:31.504227+0000","last_deep_scrub":"118'91","last_deep_scrub_stamp":"2024-01-16T15:24:31.504227+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.504227+0000","objects_scrubbed":25,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:06.166941+0000","scrub_duration":0.289711153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3b","version":"118'85","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.606933+0000","last_change":"2024-01-16T15:24:26.607090+0000","last_active":"2024-01-16T15:24:26.606933+0000","last_peered":"2024-01-16T15:24:26.606933+0000","last_clean":"2024-01-16T15:24:26.606933+0000","last_became_active":"2024-01-16T15:15:39.320204+0000","last_became_peered":"2024-01-16T15:15:39.320204+0000","last_unstale":"2024-01-16T15:24:26.606933+0000","last_undegraded":"2024-01-16T15:24:26.606933+0000","last_fullsized":"2024-01-16T15:24:26.606933+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'85","last_scrub_stamp":"2024-01-16T15:24:26.606670+0000","last_deep_scrub":"118'85","last_deep_scrub_stamp":"2024-01-16T15:24:26.606670+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.606670+0000","objects_scrubbed":25,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:32.586228+0000","scrub_duration":0.30264022000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663487,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"118'104","reported_seq":148,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:23.051913+0000","last_change":"2024-01-16T15:24:23.051959+0000","last_active":"2024-01-16T15:24:23.051913+0000","last_peered":"2024-01-16T15:24:23.051913+0000","last_clean":"2024-01-16T15:24:23.051913+0000","last_became_active":"2024-01-16T15:15:39.327532+0000","last_became_peered":"2024-01-16T15:15:39.327532+0000","last_unstale":"2024-01-16T15:24:23.051913+0000","last_undegraded":"2024-01-16T15:24:23.051913+0000","last_fullsized":"2024-01-16T15:24:23.051913+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'104","last_scrub_stamp":"2024-01-16T15:24:23.051823+0000","last_deep_scrub":"118'104","last_deep_scrub_stamp":"2024-01-16T15:24:23.051823+0000","last_clean_scrub_stamp":"2024-01-16T15:24:23.051823+0000","objects_scrubbed":31,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:26.121405+0000","scrub_duration":0.37272100699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"118'84","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.581480+0000","last_change":"2024-01-16T15:24:36.581539+0000","last_active":"2024-01-16T15:24:36.581480+0000","last_peered":"2024-01-16T15:24:36.581480+0000","last_clean":"2024-01-16T15:24:36.581480+0000","last_became_active":"2024-01-16T15:15:39.323832+0000","last_became_peered":"2024-01-16T15:15:39.323832+0000","last_unstale":"2024-01-16T15:24:36.581480+0000","last_undegraded":"2024-01-16T15:24:36.581480+0000","last_fullsized":"2024-01-16T15:24:36.581480+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'84","last_scrub_stamp":"2024-01-16T15:24:36.581385+0000","last_deep_scrub":"118'84","last_deep_scrub_stamp":"2024-01-16T15:24:36.581385+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.581385+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:28.917659+0000","scrub_duration":0.195587708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303428,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"118'71","reported_seq":107,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.696314+0000","last_change":"2024-01-16T15:24:17.696409+0000","last_active":"2024-01-16T15:24:17.696314+0000","last_peered":"2024-01-16T15:24:17.696314+0000","last_clean":"2024-01-16T15:24:17.696314+0000","last_became_active":"2024-01-16T15:15:39.327819+0000","last_became_peered":"2024-01-16T15:15:39.327819+0000","last_unstale":"2024-01-16T15:24:17.696314+0000","last_undegraded":"2024-01-16T15:24:17.696314+0000","last_fullsized":"2024-01-16T15:24:17.696314+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'71","last_scrub_stamp":"2024-01-16T15:24:17.696140+0000","last_deep_scrub":"118'71","last_deep_scrub_stamp":"2024-01-16T15:24:17.696140+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.696140+0000","objects_scrubbed":22,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:05.520208+0000","scrub_duration":0.28034184200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"118'106","reported_seq":144,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:45.072022+0000","last_change":"2024-01-16T15:24:45.072142+0000","last_active":"2024-01-16T15:24:45.072022+0000","last_peered":"2024-01-16T15:24:45.072022+0000","last_clean":"2024-01-16T15:24:45.072022+0000","last_became_active":"2024-01-16T15:15:39.319207+0000","last_became_peered":"2024-01-16T15:15:39.319207+0000","last_unstale":"2024-01-16T15:24:45.072022+0000","last_undegraded":"2024-01-16T15:24:45.072022+0000","last_fullsized":"2024-01-16T15:24:45.072022+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'106","last_scrub_stamp":"2024-01-16T15:24:45.071849+0000","last_deep_scrub":"118'106","last_deep_scrub_stamp":"2024-01-16T15:24:45.071849+0000","last_clean_scrub_stamp":"2024-01-16T15:24:45.071849+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:37.507159+0000","scrub_duration":0.34024572199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052164,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"118'88","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.030027+0000","last_change":"2024-01-16T15:24:36.030123+0000","last_active":"2024-01-16T15:24:36.030027+0000","last_peered":"2024-01-16T15:24:36.030027+0000","last_clean":"2024-01-16T15:24:36.030027+0000","last_became_active":"2024-01-16T15:15:39.326266+0000","last_became_peered":"2024-01-16T15:15:39.326266+0000","last_unstale":"2024-01-16T15:24:36.030027+0000","last_undegraded":"2024-01-16T15:24:36.030027+0000","last_fullsized":"2024-01-16T15:24:36.030027+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:36.029853+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:36.029853+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.029853+0000","objects_scrubbed":22,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:14.638731+0000","scrub_duration":0.27669553099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"118'97","reported_seq":141,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:42.395120+0000","last_change":"2024-01-16T15:24:42.395148+0000","last_active":"2024-01-16T15:24:42.395120+0000","last_peered":"2024-01-16T15:24:42.395120+0000","last_clean":"2024-01-16T15:24:42.395120+0000","last_became_active":"2024-01-16T15:15:39.325047+0000","last_became_peered":"2024-01-16T15:15:39.325047+0000","last_unstale":"2024-01-16T15:24:42.395120+0000","last_undegraded":"2024-01-16T15:24:42.395120+0000","last_fullsized":"2024-01-16T15:24:42.395120+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:42.395059+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:42.395059+0000","last_clean_scrub_stamp":"2024-01-16T15:24:42.395059+0000","objects_scrubbed":27,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:27.035568+0000","scrub_duration":0.31104689099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"118'84","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:35.066548+0000","last_change":"2024-01-16T15:24:35.066706+0000","last_active":"2024-01-16T15:24:35.066548+0000","last_peered":"2024-01-16T15:24:35.066548+0000","last_clean":"2024-01-16T15:24:35.066548+0000","last_became_active":"2024-01-16T15:15:39.317702+0000","last_became_peered":"2024-01-16T15:15:39.317702+0000","last_unstale":"2024-01-16T15:24:35.066548+0000","last_undegraded":"2024-01-16T15:24:35.066548+0000","last_fullsized":"2024-01-16T15:24:35.066548+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'84","last_scrub_stamp":"2024-01-16T15:24:35.066338+0000","last_deep_scrub":"118'84","last_deep_scrub_stamp":"2024-01-16T15:24:35.066338+0000","last_clean_scrub_stamp":"2024-01-16T15:24:35.066338+0000","objects_scrubbed":25,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:19.990397+0000","scrub_duration":0.31775753299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"118'78","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.873328+0000","last_change":"2024-01-16T15:24:31.873406+0000","last_active":"2024-01-16T15:24:31.873328+0000","last_peered":"2024-01-16T15:24:31.873328+0000","last_clean":"2024-01-16T15:24:31.873328+0000","last_became_active":"2024-01-16T15:15:39.329241+0000","last_became_peered":"2024-01-16T15:15:39.329241+0000","last_unstale":"2024-01-16T15:24:31.873328+0000","last_undegraded":"2024-01-16T15:24:31.873328+0000","last_fullsized":"2024-01-16T15:24:31.873328+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'78","last_scrub_stamp":"2024-01-16T15:24:31.873149+0000","last_deep_scrub":"118'78","last_deep_scrub_stamp":"2024-01-16T15:24:31.873149+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.873149+0000","objects_scrubbed":21,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:45.473499+0000","scrub_duration":0.24798467199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"118'88","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:12.757322+0000","last_change":"2024-01-16T15:24:12.757322+0000","last_active":"2024-01-16T15:24:12.757322+0000","last_peered":"2024-01-16T15:24:12.757322+0000","last_clean":"2024-01-16T15:24:12.757322+0000","last_became_active":"2024-01-16T15:15:39.324054+0000","last_became_peered":"2024-01-16T15:15:39.324054+0000","last_unstale":"2024-01-16T15:24:12.757322+0000","last_undegraded":"2024-01-16T15:24:12.757322+0000","last_fullsized":"2024-01-16T15:24:12.757322+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:12.757133+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:12.757133+0000","last_clean_scrub_stamp":"2024-01-16T15:24:12.757133+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:23:26.026719+0000","scrub_duration":0.337903377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"118'68","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.945405+0000","last_change":"2024-01-16T15:24:31.945405+0000","last_active":"2024-01-16T15:24:31.945405+0000","last_peered":"2024-01-16T15:24:31.945405+0000","last_clean":"2024-01-16T15:24:31.945405+0000","last_became_active":"2024-01-16T15:15:39.332804+0000","last_became_peered":"2024-01-16T15:15:39.332804+0000","last_unstale":"2024-01-16T15:24:31.945405+0000","last_undegraded":"2024-01-16T15:24:31.945405+0000","last_fullsized":"2024-01-16T15:24:31.945405+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'68","last_scrub_stamp":"2024-01-16T15:24:31.945123+0000","last_deep_scrub":"118'68","last_deep_scrub_stamp":"2024-01-16T15:24:31.945123+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.945123+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:28.376879+0000","scrub_duration":0.19293099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"118'77","reported_seq":108,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.129640+0000","last_change":"2024-01-16T15:24:26.129640+0000","last_active":"2024-01-16T15:24:26.129640+0000","last_peered":"2024-01-16T15:24:26.129640+0000","last_clean":"2024-01-16T15:24:26.129640+0000","last_became_active":"2024-01-16T15:15:39.313593+0000","last_became_peered":"2024-01-16T15:15:39.313593+0000","last_unstale":"2024-01-16T15:24:26.129640+0000","last_undegraded":"2024-01-16T15:24:26.129640+0000","last_fullsized":"2024-01-16T15:24:26.129640+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'77","last_scrub_stamp":"2024-01-16T15:24:26.129442+0000","last_deep_scrub":"118'77","last_deep_scrub_stamp":"2024-01-16T15:24:26.129442+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.129442+0000","objects_scrubbed":19,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:23.395204+0000","scrub_duration":0.23139104999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"118'66","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:40.343868+0000","last_change":"2024-01-16T15:24:40.343868+0000","last_active":"2024-01-16T15:24:40.343868+0000","last_peered":"2024-01-16T15:24:40.343868+0000","last_clean":"2024-01-16T15:24:40.343868+0000","last_became_active":"2024-01-16T15:15:39.329237+0000","last_became_peered":"2024-01-16T15:15:39.329237+0000","last_unstale":"2024-01-16T15:24:40.343868+0000","last_undegraded":"2024-01-16T15:24:40.343868+0000","last_fullsized":"2024-01-16T15:24:40.343868+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'66","last_scrub_stamp":"2024-01-16T15:24:40.343721+0000","last_deep_scrub":"118'66","last_deep_scrub_stamp":"2024-01-16T15:24:40.343721+0000","last_clean_scrub_stamp":"2024-01-16T15:24:40.343721+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:12.489538+0000","scrub_duration":0.21130144100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"118'107","reported_seq":159,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:22.884589+0000","last_change":"2024-01-16T15:24:22.884589+0000","last_active":"2024-01-16T15:24:22.884589+0000","last_peered":"2024-01-16T15:24:22.884589+0000","last_clean":"2024-01-16T15:24:22.884589+0000","last_became_active":"2024-01-16T15:15:39.330377+0000","last_became_peered":"2024-01-16T15:15:39.330377+0000","last_unstale":"2024-01-16T15:24:22.884589+0000","last_undegraded":"2024-01-16T15:24:22.884589+0000","last_fullsized":"2024-01-16T15:24:22.884589+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'107","last_scrub_stamp":"2024-01-16T15:24:22.884365+0000","last_deep_scrub":"118'107","last_deep_scrub_stamp":"2024-01-16T15:24:22.884365+0000","last_clean_scrub_stamp":"2024-01-16T15:24:22.884365+0000","objects_scrubbed":35,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:12.796440+0000","scrub_duration":0.443307281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"118'100","reported_seq":136,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:39.015448+0000","last_change":"2024-01-16T15:24:39.015448+0000","last_active":"2024-01-16T15:24:39.015448+0000","last_peered":"2024-01-16T15:24:39.015448+0000","last_clean":"2024-01-16T15:24:39.015448+0000","last_became_active":"2024-01-16T15:15:39.328744+0000","last_became_peered":"2024-01-16T15:15:39.328744+0000","last_unstale":"2024-01-16T15:24:39.015448+0000","last_undegraded":"2024-01-16T15:24:39.015448+0000","last_fullsized":"2024-01-16T15:24:39.015448+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'100","last_scrub_stamp":"2024-01-16T15:24:39.015183+0000","last_deep_scrub":"118'100","last_deep_scrub_stamp":"2024-01-16T15:24:39.015183+0000","last_clean_scrub_stamp":"2024-01-16T15:24:39.015183+0000","objects_scrubbed":29,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:39:34.700309+0000","scrub_duration":0.322461267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"118'91","reported_seq":129,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:33.478784+0000","last_change":"2024-01-16T15:24:33.478784+0000","last_active":"2024-01-16T15:24:33.478784+0000","last_peered":"2024-01-16T15:24:33.478784+0000","last_clean":"2024-01-16T15:24:33.478784+0000","last_became_active":"2024-01-16T15:15:39.318288+0000","last_became_peered":"2024-01-16T15:15:39.318288+0000","last_unstale":"2024-01-16T15:24:33.478784+0000","last_undegraded":"2024-01-16T15:24:33.478784+0000","last_fullsized":"2024-01-16T15:24:33.478784+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'91","last_scrub_stamp":"2024-01-16T15:24:33.478490+0000","last_deep_scrub":"118'91","last_deep_scrub_stamp":"2024-01-16T15:24:33.478490+0000","last_clean_scrub_stamp":"2024-01-16T15:24:33.478490+0000","objects_scrubbed":24,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:39.313097+0000","scrub_duration":0.29979264300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"118'99","reported_seq":148,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.587081+0000","last_change":"2024-01-16T15:24:15.587081+0000","last_active":"2024-01-16T15:24:15.587081+0000","last_peered":"2024-01-16T15:24:15.587081+0000","last_clean":"2024-01-16T15:24:15.587081+0000","last_became_active":"2024-01-16T15:15:39.320740+0000","last_became_peered":"2024-01-16T15:15:39.320740+0000","last_unstale":"2024-01-16T15:24:15.587081+0000","last_undegraded":"2024-01-16T15:24:15.587081+0000","last_fullsized":"2024-01-16T15:24:15.587081+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2024-01-16T15:24:15.586844+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2024-01-16T15:24:15.586844+0000","last_clean_scrub_stamp":"2024-01-16T15:24:15.586844+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:57.424369+0000","scrub_duration":0.353635808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"118'90","reported_seq":124,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.405531+0000","last_change":"2024-01-16T15:24:36.405531+0000","last_active":"2024-01-16T15:24:36.405531+0000","last_peered":"2024-01-16T15:24:36.405531+0000","last_clean":"2024-01-16T15:24:36.405531+0000","last_became_active":"2024-01-16T15:15:39.323029+0000","last_became_peered":"2024-01-16T15:15:39.323029+0000","last_unstale":"2024-01-16T15:24:36.405531+0000","last_undegraded":"2024-01-16T15:24:36.405531+0000","last_fullsized":"2024-01-16T15:24:36.405531+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'90","last_scrub_stamp":"2024-01-16T15:24:36.405296+0000","last_deep_scrub":"118'90","last_deep_scrub_stamp":"2024-01-16T15:24:36.405296+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.405296+0000","objects_scrubbed":26,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:27.713727+0000","scrub_duration":0.32153288499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"118'79","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.554719+0000","last_change":"2024-01-16T15:24:14.554719+0000","last_active":"2024-01-16T15:24:14.554719+0000","last_peered":"2024-01-16T15:24:14.554719+0000","last_clean":"2024-01-16T15:24:14.554719+0000","last_became_active":"2024-01-16T15:15:39.311983+0000","last_became_peered":"2024-01-16T15:15:39.311983+0000","last_unstale":"2024-01-16T15:24:14.554719+0000","last_undegraded":"2024-01-16T15:24:14.554719+0000","last_fullsized":"2024-01-16T15:24:14.554719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'79","last_scrub_stamp":"2024-01-16T15:24:14.554529+0000","last_deep_scrub":"118'79","last_deep_scrub_stamp":"2024-01-16T15:24:14.554529+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.554529+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:06.890804+0000","scrub_duration":0.31410593999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"118'94","reported_seq":130,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:21.246704+0000","last_change":"2024-01-16T15:24:21.246810+0000","last_active":"2024-01-16T15:24:21.246704+0000","last_peered":"2024-01-16T15:24:21.246704+0000","last_clean":"2024-01-16T15:24:21.246704+0000","last_became_active":"2024-01-16T15:15:39.323709+0000","last_became_peered":"2024-01-16T15:15:39.323709+0000","last_unstale":"2024-01-16T15:24:21.246704+0000","last_undegraded":"2024-01-16T15:24:21.246704+0000","last_fullsized":"2024-01-16T15:24:21.246704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:21.246488+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:21.246488+0000","last_clean_scrub_stamp":"2024-01-16T15:24:21.246488+0000","objects_scrubbed":24,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:51.356431+0000","scrub_duration":0.27909722799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"118'97","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:36.521003+0000","last_change":"2024-01-16T15:24:36.521052+0000","last_active":"2024-01-16T15:24:36.521003+0000","last_peered":"2024-01-16T15:24:36.521003+0000","last_clean":"2024-01-16T15:24:36.521003+0000","last_became_active":"2024-01-16T15:15:39.328392+0000","last_became_peered":"2024-01-16T15:15:39.328392+0000","last_unstale":"2024-01-16T15:24:36.521003+0000","last_undegraded":"2024-01-16T15:24:36.521003+0000","last_fullsized":"2024-01-16T15:24:36.521003+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:36.520866+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:36.520866+0000","last_clean_scrub_stamp":"2024-01-16T15:24:36.520866+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:24.676593+0000","scrub_duration":0.33406188199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"118'107","reported_seq":144,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:37.069885+0000","last_change":"2024-01-16T15:24:37.069949+0000","last_active":"2024-01-16T15:24:37.069885+0000","last_peered":"2024-01-16T15:24:37.069885+0000","last_clean":"2024-01-16T15:24:37.069885+0000","last_became_active":"2024-01-16T15:15:39.317151+0000","last_became_peered":"2024-01-16T15:15:39.317151+0000","last_unstale":"2024-01-16T15:24:37.069885+0000","last_undegraded":"2024-01-16T15:24:37.069885+0000","last_fullsized":"2024-01-16T15:24:37.069885+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'107","last_scrub_stamp":"2024-01-16T15:24:37.069711+0000","last_deep_scrub":"118'107","last_deep_scrub_stamp":"2024-01-16T15:24:37.069711+0000","last_clean_scrub_stamp":"2024-01-16T15:24:37.069711+0000","objects_scrubbed":26,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:47.854355+0000","scrub_duration":0.29586178600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"118'88","reported_seq":124,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:30.089992+0000","last_change":"2024-01-16T15:24:30.090059+0000","last_active":"2024-01-16T15:24:30.089992+0000","last_peered":"2024-01-16T15:24:30.089992+0000","last_clean":"2024-01-16T15:24:30.089992+0000","last_became_active":"2024-01-16T15:15:39.334327+0000","last_became_peered":"2024-01-16T15:15:39.334327+0000","last_unstale":"2024-01-16T15:24:30.089992+0000","last_undegraded":"2024-01-16T15:24:30.089992+0000","last_fullsized":"2024-01-16T15:24:30.089992+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:30.089879+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:30.089879+0000","last_clean_scrub_stamp":"2024-01-16T15:24:30.089879+0000","objects_scrubbed":24,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:11.927404+0000","scrub_duration":0.31580465699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"118'62","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.239853+0000","last_change":"2024-01-16T15:24:26.239940+0000","last_active":"2024-01-16T15:24:26.239853+0000","last_peered":"2024-01-16T15:24:26.239853+0000","last_clean":"2024-01-16T15:24:26.239853+0000","last_became_active":"2024-01-16T15:15:39.316938+0000","last_became_peered":"2024-01-16T15:15:39.316938+0000","last_unstale":"2024-01-16T15:24:26.239853+0000","last_undegraded":"2024-01-16T15:24:26.239853+0000","last_fullsized":"2024-01-16T15:24:26.239853+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'62","last_scrub_stamp":"2024-01-16T15:24:26.239694+0000","last_deep_scrub":"118'62","last_deep_scrub_stamp":"2024-01-16T15:24:26.239694+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.239694+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:37.997036+0000","scrub_duration":0.25765072300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"118'77","reported_seq":116,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:43.359494+0000","last_change":"2024-01-16T15:24:43.359547+0000","last_active":"2024-01-16T15:24:43.359494+0000","last_peered":"2024-01-16T15:24:43.359494+0000","last_clean":"2024-01-16T15:24:43.359494+0000","last_became_active":"2024-01-16T15:15:39.320063+0000","last_became_peered":"2024-01-16T15:15:39.320063+0000","last_unstale":"2024-01-16T15:24:43.359494+0000","last_undegraded":"2024-01-16T15:24:43.359494+0000","last_fullsized":"2024-01-16T15:24:43.359494+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'77","last_scrub_stamp":"2024-01-16T15:24:43.359391+0000","last_deep_scrub":"118'77","last_deep_scrub_stamp":"2024-01-16T15:24:43.359391+0000","last_clean_scrub_stamp":"2024-01-16T15:24:43.359391+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:02.316499+0000","scrub_duration":0.25140222099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"118'104","reported_seq":146,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:29.647830+0000","last_change":"2024-01-16T15:24:29.647892+0000","last_active":"2024-01-16T15:24:29.647830+0000","last_peered":"2024-01-16T15:24:29.647830+0000","last_clean":"2024-01-16T15:24:29.647830+0000","last_became_active":"2024-01-16T15:15:39.317088+0000","last_became_peered":"2024-01-16T15:15:39.317088+0000","last_unstale":"2024-01-16T15:24:29.647830+0000","last_undegraded":"2024-01-16T15:24:29.647830+0000","last_fullsized":"2024-01-16T15:24:29.647830+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'104","last_scrub_stamp":"2024-01-16T15:24:29.647644+0000","last_deep_scrub":"118'104","last_deep_scrub_stamp":"2024-01-16T15:24:29.647644+0000","last_clean_scrub_stamp":"2024-01-16T15:24:29.647644+0000","objects_scrubbed":28,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:32.717226+0000","scrub_duration":0.35423907199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"118'108","reported_seq":155,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:30.156400+0000","last_change":"2024-01-16T15:24:30.156470+0000","last_active":"2024-01-16T15:24:30.156400+0000","last_peered":"2024-01-16T15:24:30.156400+0000","last_clean":"2024-01-16T15:24:30.156400+0000","last_became_active":"2024-01-16T15:15:39.313478+0000","last_became_peered":"2024-01-16T15:15:39.313478+0000","last_unstale":"2024-01-16T15:24:30.156400+0000","last_undegraded":"2024-01-16T15:24:30.156400+0000","last_fullsized":"2024-01-16T15:24:30.156400+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'108","last_scrub_stamp":"2024-01-16T15:24:30.156210+0000","last_deep_scrub":"118'108","last_deep_scrub_stamp":"2024-01-16T15:24:30.156210+0000","last_clean_scrub_stamp":"2024-01-16T15:24:30.156210+0000","objects_scrubbed":27,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:02:23.423714+0000","scrub_duration":0.36436912999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"118'82","reported_seq":119,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.992127+0000","last_change":"2024-01-16T15:24:10.992228+0000","last_active":"2024-01-16T15:24:10.992127+0000","last_peered":"2024-01-16T15:24:10.992127+0000","last_clean":"2024-01-16T15:24:10.992127+0000","last_became_active":"2024-01-16T15:15:39.321281+0000","last_became_peered":"2024-01-16T15:15:39.321281+0000","last_unstale":"2024-01-16T15:24:10.992127+0000","last_undegraded":"2024-01-16T15:24:10.992127+0000","last_fullsized":"2024-01-16T15:24:10.992127+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'82","last_scrub_stamp":"2024-01-16T15:24:10.991908+0000","last_deep_scrub":"118'82","last_deep_scrub_stamp":"2024-01-16T15:24:10.991908+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.991908+0000","objects_scrubbed":23,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:34.358631+0000","scrub_duration":0.279512233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"118'88","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:29.070725+0000","last_change":"2024-01-16T15:24:29.070774+0000","last_active":"2024-01-16T15:24:29.070725+0000","last_peered":"2024-01-16T15:24:29.070725+0000","last_clean":"2024-01-16T15:24:29.070725+0000","last_became_active":"2024-01-16T15:15:39.326798+0000","last_became_peered":"2024-01-16T15:15:39.326798+0000","last_unstale":"2024-01-16T15:24:29.070725+0000","last_undegraded":"2024-01-16T15:24:29.070725+0000","last_fullsized":"2024-01-16T15:24:29.070725+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:29.070548+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:29.070548+0000","last_clean_scrub_stamp":"2024-01-16T15:24:29.070548+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:54:21.406823+0000","scrub_duration":0.32460896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"118'6","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.468778+0000","last_change":"2024-01-16T15:24:08.468844+0000","last_active":"2024-01-16T15:24:08.468778+0000","last_peered":"2024-01-16T15:24:08.468778+0000","last_clean":"2024-01-16T15:24:08.468778+0000","last_became_active":"2024-01-16T15:15:38.295351+0000","last_became_peered":"2024-01-16T15:15:38.295351+0000","last_unstale":"2024-01-16T15:24:08.468778+0000","last_undegraded":"2024-01-16T15:24:08.468778+0000","last_fullsized":"2024-01-16T15:24:08.468778+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6","last_scrub_stamp":"2024-01-16T15:24:08.468615+0000","last_deep_scrub":"118'6","last_deep_scrub_stamp":"2024-01-16T15:24:08.468615+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.468615+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:16.033835+0000","scrub_duration":0.0085012860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"118'73","reported_seq":112,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:19.646515+0000","last_change":"2024-01-16T15:24:19.646587+0000","last_active":"2024-01-16T15:24:19.646515+0000","last_peered":"2024-01-16T15:24:19.646515+0000","last_clean":"2024-01-16T15:24:19.646515+0000","last_became_active":"2024-01-16T15:15:39.329007+0000","last_became_peered":"2024-01-16T15:15:39.329007+0000","last_unstale":"2024-01-16T15:24:19.646515+0000","last_undegraded":"2024-01-16T15:24:19.646515+0000","last_fullsized":"2024-01-16T15:24:19.646515+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'73","last_scrub_stamp":"2024-01-16T15:24:19.646340+0000","last_deep_scrub":"118'73","last_deep_scrub_stamp":"2024-01-16T15:24:19.646340+0000","last_clean_scrub_stamp":"2024-01-16T15:24:19.646340+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:34:37.373916+0000","scrub_duration":0.24430438900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"115'1","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.441338+0000","last_change":"2024-01-16T15:24:10.441429+0000","last_active":"2024-01-16T15:24:10.441338+0000","last_peered":"2024-01-16T15:24:10.441338+0000","last_clean":"2024-01-16T15:24:10.441338+0000","last_became_active":"2024-01-16T15:15:38.292216+0000","last_became_peered":"2024-01-16T15:15:38.292216+0000","last_unstale":"2024-01-16T15:24:10.441338+0000","last_undegraded":"2024-01-16T15:24:10.441338+0000","last_fullsized":"2024-01-16T15:24:10.441338+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:10.441063+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:10.441063+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:44.044132+0000","scrub_duration":0.004404593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"118'97","reported_seq":141,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.044503+0000","last_change":"2024-01-16T15:24:20.044562+0000","last_active":"2024-01-16T15:24:20.044503+0000","last_peered":"2024-01-16T15:24:20.044503+0000","last_clean":"2024-01-16T15:24:20.044503+0000","last_became_active":"2024-01-16T15:15:39.332564+0000","last_became_peered":"2024-01-16T15:15:39.332564+0000","last_unstale":"2024-01-16T15:24:20.044503+0000","last_undegraded":"2024-01-16T15:24:20.044503+0000","last_fullsized":"2024-01-16T15:24:20.044503+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:20.044335+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:20.044335+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.044335+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:39.481751+0000","scrub_duration":0.33193780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"118'4","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.220298+0000","last_change":"2024-01-16T15:24:16.220358+0000","last_active":"2024-01-16T15:24:16.220298+0000","last_peered":"2024-01-16T15:24:16.220298+0000","last_clean":"2024-01-16T15:24:16.220298+0000","last_became_active":"2024-01-16T15:15:38.288269+0000","last_became_peered":"2024-01-16T15:15:38.288269+0000","last_unstale":"2024-01-16T15:24:16.220298+0000","last_undegraded":"2024-01-16T15:24:16.220298+0000","last_fullsized":"2024-01-16T15:24:16.220298+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4","last_scrub_stamp":"2024-01-16T15:24:16.220137+0000","last_deep_scrub":"118'4","last_deep_scrub_stamp":"2024-01-16T15:24:16.220137+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.220137+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:35.174555+0000","scrub_duration":0.0071288089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"118'89","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:41.924771+0000","last_change":"2024-01-16T15:24:41.924861+0000","last_active":"2024-01-16T15:24:41.924771+0000","last_peered":"2024-01-16T15:24:41.924771+0000","last_clean":"2024-01-16T15:24:41.924771+0000","last_became_active":"2024-01-16T15:15:39.329062+0000","last_became_peered":"2024-01-16T15:15:39.329062+0000","last_unstale":"2024-01-16T15:24:41.924771+0000","last_undegraded":"2024-01-16T15:24:41.924771+0000","last_fullsized":"2024-01-16T15:24:41.924771+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'89","last_scrub_stamp":"2024-01-16T15:24:41.924592+0000","last_deep_scrub":"118'89","last_deep_scrub_stamp":"2024-01-16T15:24:41.924592+0000","last_clean_scrub_stamp":"2024-01-16T15:24:41.924592+0000","objects_scrubbed":21,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:21.725496+0000","scrub_duration":0.24021224899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"118'3","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.217316+0000","last_change":"2024-01-16T15:24:14.217375+0000","last_active":"2024-01-16T15:24:14.217316+0000","last_peered":"2024-01-16T15:24:14.217316+0000","last_clean":"2024-01-16T15:24:14.217316+0000","last_became_active":"2024-01-16T15:15:38.288081+0000","last_became_peered":"2024-01-16T15:15:38.288081+0000","last_unstale":"2024-01-16T15:24:14.217316+0000","last_undegraded":"2024-01-16T15:24:14.217316+0000","last_fullsized":"2024-01-16T15:24:14.217316+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:14.217126+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:14.217126+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.217126+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:21.782346+0000","scrub_duration":0.0072240430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"118'95","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.056895+0000","last_change":"2024-01-16T15:24:31.056994+0000","last_active":"2024-01-16T15:24:31.056895+0000","last_peered":"2024-01-16T15:24:31.056895+0000","last_clean":"2024-01-16T15:24:31.056895+0000","last_became_active":"2024-01-16T15:15:39.320732+0000","last_became_peered":"2024-01-16T15:15:39.320732+0000","last_unstale":"2024-01-16T15:24:31.056895+0000","last_undegraded":"2024-01-16T15:24:31.056895+0000","last_fullsized":"2024-01-16T15:24:31.056895+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:31.056810+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:31.056810+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.056810+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:30.609981+0000","scrub_duration":0.28217324500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"115'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:18.450001+0000","last_change":"2024-01-16T15:24:18.450098+0000","last_active":"2024-01-16T15:24:18.450001+0000","last_peered":"2024-01-16T15:24:18.450001+0000","last_clean":"2024-01-16T15:24:18.450001+0000","last_became_active":"2024-01-16T15:15:38.287948+0000","last_became_peered":"2024-01-16T15:15:38.287948+0000","last_unstale":"2024-01-16T15:24:18.450001+0000","last_undegraded":"2024-01-16T15:24:18.450001+0000","last_fullsized":"2024-01-16T15:24:18.450001+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:18.449469+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:18.449469+0000","last_clean_scrub_stamp":"2024-01-16T15:24:18.449469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:31:41.816192+0000","scrub_duration":0.0038895650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"118'97","reported_seq":143,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:34.488943+0000","last_change":"2024-01-16T15:24:34.489001+0000","last_active":"2024-01-16T15:24:34.488943+0000","last_peered":"2024-01-16T15:24:34.488943+0000","last_clean":"2024-01-16T15:24:34.488943+0000","last_became_active":"2024-01-16T15:15:39.327012+0000","last_became_peered":"2024-01-16T15:15:39.327012+0000","last_unstale":"2024-01-16T15:24:34.488943+0000","last_undegraded":"2024-01-16T15:24:34.488943+0000","last_fullsized":"2024-01-16T15:24:34.488943+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:34.488759+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:34.488759+0000","last_clean_scrub_stamp":"2024-01-16T15:24:34.488759+0000","objects_scrubbed":25,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:22.312827+0000","scrub_duration":0.29169478900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"115'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.835879+0000","last_change":"2024-01-16T15:24:02.835904+0000","last_active":"2024-01-16T15:24:02.835879+0000","last_peered":"2024-01-16T15:24:02.835879+0000","last_clean":"2024-01-16T15:24:02.835879+0000","last_became_active":"2024-01-16T15:15:38.292437+0000","last_became_peered":"2024-01-16T15:15:38.292437+0000","last_unstale":"2024-01-16T15:24:02.835879+0000","last_undegraded":"2024-01-16T15:24:02.835879+0000","last_fullsized":"2024-01-16T15:24:02.835879+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:24:02.835792+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.835792+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:09.602675+0000","scrub_duration":0.0029040390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"118'95","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:00.969881+0000","last_change":"2024-01-16T15:24:00.969955+0000","last_active":"2024-01-16T15:24:00.969881+0000","last_peered":"2024-01-16T15:24:00.969881+0000","last_clean":"2024-01-16T15:24:00.969881+0000","last_became_active":"2024-01-16T15:15:39.322225+0000","last_became_peered":"2024-01-16T15:15:39.322225+0000","last_unstale":"2024-01-16T15:24:00.969881+0000","last_undegraded":"2024-01-16T15:24:00.969881+0000","last_fullsized":"2024-01-16T15:24:00.969881+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'95","last_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_deep_scrub":"118'95","last_deep_scrub_stamp":"2024-01-16T15:24:00.969804+0000","last_clean_scrub_stamp":"2024-01-16T15:24:00.969804+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:57.211104+0000","scrub_duration":0.179269503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.280477+0000","last_change":"2024-01-16T15:24:07.280530+0000","last_active":"2024-01-16T15:24:07.280477+0000","last_peered":"2024-01-16T15:24:07.280477+0000","last_clean":"2024-01-16T15:24:07.280477+0000","last_became_active":"2024-01-16T15:15:38.275926+0000","last_became_peered":"2024-01-16T15:15:38.275926+0000","last_unstale":"2024-01-16T15:24:07.280477+0000","last_undegraded":"2024-01-16T15:24:07.280477+0000","last_fullsized":"2024-01-16T15:24:07.280477+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:07.280247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:07.280247+0000","last_clean_scrub_stamp":"2024-01-16T15:24:07.280247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:26.717663+0000","scrub_duration":0.0039888249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"118'101","reported_seq":137,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:28.623771+0000","last_change":"2024-01-16T15:24:28.623808+0000","last_active":"2024-01-16T15:24:28.623771+0000","last_peered":"2024-01-16T15:24:28.623771+0000","last_clean":"2024-01-16T15:24:28.623771+0000","last_became_active":"2024-01-16T15:15:39.324968+0000","last_became_peered":"2024-01-16T15:15:39.324968+0000","last_unstale":"2024-01-16T15:24:28.623771+0000","last_undegraded":"2024-01-16T15:24:28.623771+0000","last_fullsized":"2024-01-16T15:24:28.623771+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'101","last_scrub_stamp":"2024-01-16T15:24:28.623666+0000","last_deep_scrub":"118'101","last_deep_scrub_stamp":"2024-01-16T15:24:28.623666+0000","last_clean_scrub_stamp":"2024-01-16T15:24:28.623666+0000","objects_scrubbed":28,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:32:48.061082+0000","scrub_duration":0.34195076600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.814116+0000","last_change":"2024-01-16T15:24:17.814162+0000","last_active":"2024-01-16T15:24:17.814116+0000","last_peered":"2024-01-16T15:24:17.814116+0000","last_clean":"2024-01-16T15:24:17.814116+0000","last_became_active":"2024-01-16T15:15:38.296763+0000","last_became_peered":"2024-01-16T15:15:38.296763+0000","last_unstale":"2024-01-16T15:24:17.814116+0000","last_undegraded":"2024-01-16T15:24:17.814116+0000","last_fullsized":"2024-01-16T15:24:17.814116+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:17.813872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:17.813872+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.813872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:11.813723+0000","scrub_duration":0.0034917469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"118'76","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:39.542732+0000","last_change":"2024-01-16T15:24:39.542813+0000","last_active":"2024-01-16T15:24:39.542732+0000","last_peered":"2024-01-16T15:24:39.542732+0000","last_clean":"2024-01-16T15:24:39.542732+0000","last_became_active":"2024-01-16T15:15:39.318928+0000","last_became_peered":"2024-01-16T15:15:39.318928+0000","last_unstale":"2024-01-16T15:24:39.542732+0000","last_undegraded":"2024-01-16T15:24:39.542732+0000","last_fullsized":"2024-01-16T15:24:39.542732+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'76","last_scrub_stamp":"2024-01-16T15:24:39.542553+0000","last_deep_scrub":"118'76","last_deep_scrub_stamp":"2024-01-16T15:24:39.542553+0000","last_clean_scrub_stamp":"2024-01-16T15:24:39.542553+0000","objects_scrubbed":18,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:48.986527+0000","scrub_duration":0.19909136099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"118'1","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.306271+0000","last_change":"2024-01-16T15:24:10.306417+0000","last_active":"2024-01-16T15:24:10.306271+0000","last_peered":"2024-01-16T15:24:10.306271+0000","last_clean":"2024-01-16T15:24:10.306271+0000","last_became_active":"2024-01-16T15:15:38.287172+0000","last_became_peered":"2024-01-16T15:15:38.287172+0000","last_unstale":"2024-01-16T15:24:10.306271+0000","last_undegraded":"2024-01-16T15:24:10.306271+0000","last_fullsized":"2024-01-16T15:24:10.306271+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'1","last_scrub_stamp":"2024-01-16T15:24:10.306050+0000","last_deep_scrub":"118'1","last_deep_scrub_stamp":"2024-01-16T15:24:10.306050+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.306050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:08.994705+0000","scrub_duration":0.0031432769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"118'78","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.670647+0000","last_change":"2024-01-16T15:24:17.670740+0000","last_active":"2024-01-16T15:24:17.670647+0000","last_peered":"2024-01-16T15:24:17.670647+0000","last_clean":"2024-01-16T15:24:17.670647+0000","last_became_active":"2024-01-16T15:15:39.333098+0000","last_became_peered":"2024-01-16T15:15:39.333098+0000","last_unstale":"2024-01-16T15:24:17.670647+0000","last_undegraded":"2024-01-16T15:24:17.670647+0000","last_fullsized":"2024-01-16T15:24:17.670647+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'78","last_scrub_stamp":"2024-01-16T15:24:17.670442+0000","last_deep_scrub":"118'78","last_deep_scrub_stamp":"2024-01-16T15:24:17.670442+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.670442+0000","objects_scrubbed":20,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:10:39.488294+0000","scrub_duration":0.25523936800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.949045+0000","last_change":"2024-01-16T15:24:07.949150+0000","last_active":"2024-01-16T15:24:07.949045+0000","last_peered":"2024-01-16T15:24:07.949045+0000","last_clean":"2024-01-16T15:24:07.949045+0000","last_became_active":"2024-01-16T15:15:38.293338+0000","last_became_peered":"2024-01-16T15:15:38.293338+0000","last_unstale":"2024-01-16T15:24:07.949045+0000","last_undegraded":"2024-01-16T15:24:07.949045+0000","last_fullsized":"2024-01-16T15:24:07.949045+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:07.948766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:07.948766+0000","last_clean_scrub_stamp":"2024-01-16T15:24:07.948766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:49.830805+0000","scrub_duration":0.0035464849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"118'88","reported_seq":131,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:23.283365+0000","last_change":"2024-01-16T15:24:23.283365+0000","last_active":"2024-01-16T15:24:23.283365+0000","last_peered":"2024-01-16T15:24:23.283365+0000","last_clean":"2024-01-16T15:24:23.283365+0000","last_became_active":"2024-01-16T15:15:39.318667+0000","last_became_peered":"2024-01-16T15:15:39.318667+0000","last_unstale":"2024-01-16T15:24:23.283365+0000","last_undegraded":"2024-01-16T15:24:23.283365+0000","last_fullsized":"2024-01-16T15:24:23.283365+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'88","last_scrub_stamp":"2024-01-16T15:24:23.283119+0000","last_deep_scrub":"118'88","last_deep_scrub_stamp":"2024-01-16T15:24:23.283119+0000","last_clean_scrub_stamp":"2024-01-16T15:24:23.283119+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:18.238164+0000","scrub_duration":0.31081178100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"118'33","reported_seq":59,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:24.862951+0000","last_change":"2024-01-16T15:24:24.863013+0000","last_active":"2024-01-16T15:24:24.862951+0000","last_peered":"2024-01-16T15:24:24.862951+0000","last_clean":"2024-01-16T15:24:24.862951+0000","last_became_active":"2024-01-16T15:15:38.295428+0000","last_became_peered":"2024-01-16T15:15:38.295428+0000","last_unstale":"2024-01-16T15:24:24.862951+0000","last_undegraded":"2024-01-16T15:24:24.862951+0000","last_fullsized":"2024-01-16T15:24:24.862951+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'33","last_scrub_stamp":"2024-01-16T15:24:24.862770+0000","last_deep_scrub":"118'33","last_deep_scrub_stamp":"2024-01-16T15:24:24.862770+0000","last_clean_scrub_stamp":"2024-01-16T15:24:24.862770+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:58.053240+0000","scrub_duration":0.0090951780000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"118'96","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:38.721801+0000","last_change":"2024-01-16T15:24:38.721843+0000","last_active":"2024-01-16T15:24:38.721801+0000","last_peered":"2024-01-16T15:24:38.721801+0000","last_clean":"2024-01-16T15:24:38.721801+0000","last_became_active":"2024-01-16T15:15:39.312431+0000","last_became_peered":"2024-01-16T15:15:39.312431+0000","last_unstale":"2024-01-16T15:24:38.721801+0000","last_undegraded":"2024-01-16T15:24:38.721801+0000","last_fullsized":"2024-01-16T15:24:38.721801+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'96","last_scrub_stamp":"2024-01-16T15:24:38.721704+0000","last_deep_scrub":"118'96","last_deep_scrub_stamp":"2024-01-16T15:24:38.721704+0000","last_clean_scrub_stamp":"2024-01-16T15:24:38.721704+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:31:17.163057+0000","scrub_duration":0.33844099599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"118'330","reported_seq":357,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:54.729118+0000","last_change":"2024-01-16T15:24:09.800168+0000","last_active":"2024-01-16T15:24:54.729118+0000","last_peered":"2024-01-16T15:24:54.729118+0000","last_clean":"2024-01-16T15:24:54.729118+0000","last_became_active":"2024-01-16T15:15:38.296093+0000","last_became_peered":"2024-01-16T15:15:38.296093+0000","last_unstale":"2024-01-16T15:24:54.729118+0000","last_undegraded":"2024-01-16T15:24:54.729118+0000","last_fullsized":"2024-01-16T15:24:54.729118+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'328","last_scrub_stamp":"2024-01-16T15:24:09.799941+0000","last_deep_scrub":"118'328","last_deep_scrub_stamp":"2024-01-16T15:24:09.799941+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.799941+0000","objects_scrubbed":4,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:04.754986+0000","scrub_duration":0.023713471999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":327,"num_write_kb":1378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"118'75","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:27.265640+0000","last_change":"2024-01-16T15:24:27.265640+0000","last_active":"2024-01-16T15:24:27.265640+0000","last_peered":"2024-01-16T15:24:27.265640+0000","last_clean":"2024-01-16T15:24:27.265640+0000","last_became_active":"2024-01-16T15:15:39.312010+0000","last_became_peered":"2024-01-16T15:15:39.312010+0000","last_unstale":"2024-01-16T15:24:27.265640+0000","last_undegraded":"2024-01-16T15:24:27.265640+0000","last_fullsized":"2024-01-16T15:24:27.265640+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'75","last_scrub_stamp":"2024-01-16T15:24:27.265387+0000","last_deep_scrub":"118'75","last_deep_scrub_stamp":"2024-01-16T15:24:27.265387+0000","last_clean_scrub_stamp":"2024-01-16T15:24:27.265387+0000","objects_scrubbed":21,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:23.971761+0000","scrub_duration":0.24131996999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"118'5","reported_seq":32,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:03.636309+0000","last_change":"2024-01-16T15:24:03.636309+0000","last_active":"2024-01-16T15:24:03.636309+0000","last_peered":"2024-01-16T15:24:03.636309+0000","last_clean":"2024-01-16T15:24:03.636309+0000","last_became_active":"2024-01-16T15:15:38.279848+0000","last_became_peered":"2024-01-16T15:15:38.279848+0000","last_unstale":"2024-01-16T15:24:03.636309+0000","last_undegraded":"2024-01-16T15:24:03.636309+0000","last_fullsized":"2024-01-16T15:24:03.636309+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5","last_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_deep_scrub":"118'5","last_deep_scrub_stamp":"2024-01-16T15:24:03.636187+0000","last_clean_scrub_stamp":"2024-01-16T15:24:03.636187+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:22.590604+0000","scrub_duration":0.0075747449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"118'98","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:37.368444+0000","last_change":"2024-01-16T15:24:37.368444+0000","last_active":"2024-01-16T15:24:37.368444+0000","last_peered":"2024-01-16T15:24:37.368444+0000","last_clean":"2024-01-16T15:24:37.368444+0000","last_became_active":"2024-01-16T15:15:39.323853+0000","last_became_peered":"2024-01-16T15:15:39.323853+0000","last_unstale":"2024-01-16T15:24:37.368444+0000","last_undegraded":"2024-01-16T15:24:37.368444+0000","last_fullsized":"2024-01-16T15:24:37.368444+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'98","last_scrub_stamp":"2024-01-16T15:24:37.368197+0000","last_deep_scrub":"118'98","last_deep_scrub_stamp":"2024-01-16T15:24:37.368197+0000","last_clean_scrub_stamp":"2024-01-16T15:24:37.368197+0000","objects_scrubbed":26,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:10.558667+0000","scrub_duration":0.29771550400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857860,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":28,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.384677+0000","last_change":"2024-01-16T15:24:05.384762+0000","last_active":"2024-01-16T15:24:05.384677+0000","last_peered":"2024-01-16T15:24:05.384677+0000","last_clean":"2024-01-16T15:24:05.384677+0000","last_became_active":"2024-01-16T15:15:38.295564+0000","last_became_peered":"2024-01-16T15:15:38.295564+0000","last_unstale":"2024-01-16T15:24:05.384677+0000","last_undegraded":"2024-01-16T15:24:05.384677+0000","last_fullsized":"2024-01-16T15:24:05.384677+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:05.384503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:05.384503+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.384503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:22.638464+0000","scrub_duration":0.0038951989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"118'89","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:24.284846+0000","last_change":"2024-01-16T15:24:24.284846+0000","last_active":"2024-01-16T15:24:24.284846+0000","last_peered":"2024-01-16T15:24:24.284846+0000","last_clean":"2024-01-16T15:24:24.284846+0000","last_became_active":"2024-01-16T15:15:39.319187+0000","last_became_peered":"2024-01-16T15:15:39.319187+0000","last_unstale":"2024-01-16T15:24:24.284846+0000","last_undegraded":"2024-01-16T15:24:24.284846+0000","last_fullsized":"2024-01-16T15:24:24.284846+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'89","last_scrub_stamp":"2024-01-16T15:24:24.284582+0000","last_deep_scrub":"118'89","last_deep_scrub_stamp":"2024-01-16T15:24:24.284582+0000","last_clean_scrub_stamp":"2024-01-16T15:24:24.284582+0000","objects_scrubbed":25,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:25.922775+0000","scrub_duration":0.27736433700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":40961,"num_write":27,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"118'12","reported_seq":39,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.355271+0000","last_change":"2024-01-16T15:24:06.355271+0000","last_active":"2024-01-16T15:24:06.355271+0000","last_peered":"2024-01-16T15:24:06.355271+0000","last_clean":"2024-01-16T15:24:06.355271+0000","last_became_active":"2024-01-16T15:15:38.293583+0000","last_became_peered":"2024-01-16T15:15:38.293583+0000","last_unstale":"2024-01-16T15:24:06.355271+0000","last_undegraded":"2024-01-16T15:24:06.355271+0000","last_fullsized":"2024-01-16T15:24:06.355271+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'12","last_scrub_stamp":"2024-01-16T15:24:06.355004+0000","last_deep_scrub":"118'12","last_deep_scrub_stamp":"2024-01-16T15:24:06.355004+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.355004+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:24.345427+0000","scrub_duration":0.011408029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"118'61","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:28.452503+0000","last_change":"2024-01-16T15:24:28.452503+0000","last_active":"2024-01-16T15:24:28.452503+0000","last_peered":"2024-01-16T15:24:28.452503+0000","last_clean":"2024-01-16T15:24:28.452503+0000","last_became_active":"2024-01-16T15:15:39.313739+0000","last_became_peered":"2024-01-16T15:15:39.313739+0000","last_unstale":"2024-01-16T15:24:28.452503+0000","last_undegraded":"2024-01-16T15:24:28.452503+0000","last_fullsized":"2024-01-16T15:24:28.452503+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'61","last_scrub_stamp":"2024-01-16T15:24:28.452255+0000","last_deep_scrub":"118'61","last_deep_scrub_stamp":"2024-01-16T15:24:28.452255+0000","last_clean_scrub_stamp":"2024-01-16T15:24:28.452255+0000","objects_scrubbed":15,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:53:57.412116+0000","scrub_duration":0.21160910499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"118'2","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.708181+0000","last_change":"2024-01-16T15:24:14.708181+0000","last_active":"2024-01-16T15:24:14.708181+0000","last_peered":"2024-01-16T15:24:14.708181+0000","last_clean":"2024-01-16T15:24:14.708181+0000","last_became_active":"2024-01-16T15:15:38.296220+0000","last_became_peered":"2024-01-16T15:15:38.296220+0000","last_unstale":"2024-01-16T15:24:14.708181+0000","last_undegraded":"2024-01-16T15:24:14.708181+0000","last_fullsized":"2024-01-16T15:24:14.708181+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'2","last_scrub_stamp":"2024-01-16T15:24:14.707879+0000","last_deep_scrub":"118'2","last_deep_scrub_stamp":"2024-01-16T15:24:14.707879+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.707879+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:26.465221+0000","scrub_duration":0.0070736139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"118'85","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:31.059525+0000","last_change":"2024-01-16T15:24:31.059525+0000","last_active":"2024-01-16T15:24:31.059525+0000","last_peered":"2024-01-16T15:24:31.059525+0000","last_clean":"2024-01-16T15:24:31.059525+0000","last_became_active":"2024-01-16T15:15:39.334365+0000","last_became_peered":"2024-01-16T15:15:39.334365+0000","last_unstale":"2024-01-16T15:24:31.059525+0000","last_undegraded":"2024-01-16T15:24:31.059525+0000","last_fullsized":"2024-01-16T15:24:31.059525+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'85","last_scrub_stamp":"2024-01-16T15:24:31.059404+0000","last_deep_scrub":"118'85","last_deep_scrub_stamp":"2024-01-16T15:24:31.059404+0000","last_clean_scrub_stamp":"2024-01-16T15:24:31.059404+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:38.481467+0000","scrub_duration":0.28150150299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"118'3","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.312794+0000","last_change":"2024-01-16T15:24:20.312794+0000","last_active":"2024-01-16T15:24:20.312794+0000","last_peered":"2024-01-16T15:24:20.312794+0000","last_clean":"2024-01-16T15:24:20.312794+0000","last_became_active":"2024-01-16T15:15:38.289173+0000","last_became_peered":"2024-01-16T15:15:38.289173+0000","last_unstale":"2024-01-16T15:24:20.312794+0000","last_undegraded":"2024-01-16T15:24:20.312794+0000","last_fullsized":"2024-01-16T15:24:20.312794+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:20.312507+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:20.312507+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.312507+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:01.823492+0000","scrub_duration":0.0071549860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"118'75","reported_seq":115,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:22.676204+0000","last_change":"2024-01-16T15:24:22.676204+0000","last_active":"2024-01-16T15:24:22.676204+0000","last_peered":"2024-01-16T15:24:22.676204+0000","last_clean":"2024-01-16T15:24:22.676204+0000","last_became_active":"2024-01-16T15:15:39.332384+0000","last_became_peered":"2024-01-16T15:15:39.332384+0000","last_unstale":"2024-01-16T15:24:22.676204+0000","last_undegraded":"2024-01-16T15:24:22.676204+0000","last_fullsized":"2024-01-16T15:24:22.676204+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'75","last_scrub_stamp":"2024-01-16T15:24:22.675987+0000","last_deep_scrub":"118'75","last_deep_scrub_stamp":"2024-01-16T15:24:22.675987+0000","last_clean_scrub_stamp":"2024-01-16T15:24:22.675987+0000","objects_scrubbed":14,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:10.831714+0000","scrub_duration":0.18371293499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"118'2","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.419007+0000","last_change":"2024-01-16T15:24:08.419007+0000","last_active":"2024-01-16T15:24:08.419007+0000","last_peered":"2024-01-16T15:24:08.419007+0000","last_clean":"2024-01-16T15:24:08.419007+0000","last_became_active":"2024-01-16T15:15:38.291039+0000","last_became_peered":"2024-01-16T15:15:38.291039+0000","last_unstale":"2024-01-16T15:24:08.419007+0000","last_undegraded":"2024-01-16T15:24:08.419007+0000","last_fullsized":"2024-01-16T15:24:08.419007+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'2","last_scrub_stamp":"2024-01-16T15:24:08.418682+0000","last_deep_scrub":"118'2","last_deep_scrub_stamp":"2024-01-16T15:24:08.418682+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.418682+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:20.781341+0000","scrub_duration":0.0080426539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"118'89","reported_seq":133,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:43.994951+0000","last_change":"2024-01-16T15:24:43.995006+0000","last_active":"2024-01-16T15:24:43.994951+0000","last_peered":"2024-01-16T15:24:43.994951+0000","last_clean":"2024-01-16T15:24:43.994951+0000","last_became_active":"2024-01-16T15:15:39.326020+0000","last_became_peered":"2024-01-16T15:15:39.326020+0000","last_unstale":"2024-01-16T15:24:43.994951+0000","last_undegraded":"2024-01-16T15:24:43.994951+0000","last_fullsized":"2024-01-16T15:24:43.994951+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'89","last_scrub_stamp":"2024-01-16T15:24:43.994807+0000","last_deep_scrub":"118'89","last_deep_scrub_stamp":"2024-01-16T15:24:43.994807+0000","last_clean_scrub_stamp":"2024-01-16T15:24:43.994807+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:26.752101+0000","scrub_duration":0.34466592899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.539708+0000","last_change":"2024-01-16T15:24:06.539708+0000","last_active":"2024-01-16T15:24:06.539708+0000","last_peered":"2024-01-16T15:24:06.539708+0000","last_clean":"2024-01-16T15:24:06.539708+0000","last_became_active":"2024-01-16T15:15:38.298348+0000","last_became_peered":"2024-01-16T15:15:38.298348+0000","last_unstale":"2024-01-16T15:24:06.539708+0000","last_undegraded":"2024-01-16T15:24:06.539708+0000","last_fullsized":"2024-01-16T15:24:06.539708+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:06.539466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:06.539466+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.539466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:05.228120+0000","scrub_duration":0.0037083429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"118'82","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.843426+0000","last_change":"2024-01-16T15:24:25.843494+0000","last_active":"2024-01-16T15:24:25.843426+0000","last_peered":"2024-01-16T15:24:25.843426+0000","last_clean":"2024-01-16T15:24:25.843426+0000","last_became_active":"2024-01-16T15:15:39.325916+0000","last_became_peered":"2024-01-16T15:15:39.325916+0000","last_unstale":"2024-01-16T15:24:25.843426+0000","last_undegraded":"2024-01-16T15:24:25.843426+0000","last_fullsized":"2024-01-16T15:24:25.843426+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'82","last_scrub_stamp":"2024-01-16T15:24:25.843273+0000","last_deep_scrub":"118'82","last_deep_scrub_stamp":"2024-01-16T15:24:25.843273+0000","last_clean_scrub_stamp":"2024-01-16T15:24:25.843273+0000","objects_scrubbed":26,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:04.071362+0000","scrub_duration":0.34204772999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"118'4","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:01.623234+0000","last_change":"2024-01-16T15:24:01.623234+0000","last_active":"2024-01-16T15:24:01.623234+0000","last_peered":"2024-01-16T15:24:01.623234+0000","last_clean":"2024-01-16T15:24:01.623234+0000","last_became_active":"2024-01-16T15:15:38.277590+0000","last_became_peered":"2024-01-16T15:15:38.277590+0000","last_unstale":"2024-01-16T15:24:01.623234+0000","last_undegraded":"2024-01-16T15:24:01.623234+0000","last_fullsized":"2024-01-16T15:24:01.623234+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4","last_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_deep_scrub":"118'4","last_deep_scrub_stamp":"2024-01-16T15:24:01.623154+0000","last_clean_scrub_stamp":"2024-01-16T15:24:01.623154+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:58.563468+0000","scrub_duration":0.0032134569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"118'92","reported_seq":133,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:08.150238+0000","last_change":"2024-01-16T15:24:08.150238+0000","last_active":"2024-01-16T15:24:08.150238+0000","last_peered":"2024-01-16T15:24:08.150238+0000","last_clean":"2024-01-16T15:24:08.150238+0000","last_became_active":"2024-01-16T15:15:39.325105+0000","last_became_peered":"2024-01-16T15:15:39.325105+0000","last_unstale":"2024-01-16T15:24:08.150238+0000","last_undegraded":"2024-01-16T15:24:08.150238+0000","last_fullsized":"2024-01-16T15:24:08.150238+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'92","last_scrub_stamp":"2024-01-16T15:24:08.149996+0000","last_deep_scrub":"118'92","last_deep_scrub_stamp":"2024-01-16T15:24:08.149996+0000","last_clean_scrub_stamp":"2024-01-16T15:24:08.149996+0000","objects_scrubbed":26,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:37.843231+0000","scrub_duration":0.295168926,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.240464+0000","last_change":"2024-01-16T15:24:15.240464+0000","last_active":"2024-01-16T15:24:15.240464+0000","last_peered":"2024-01-16T15:24:15.240464+0000","last_clean":"2024-01-16T15:24:15.240464+0000","last_became_active":"2024-01-16T15:15:38.286825+0000","last_became_peered":"2024-01-16T15:15:38.286825+0000","last_unstale":"2024-01-16T15:24:15.240464+0000","last_undegraded":"2024-01-16T15:24:15.240464+0000","last_fullsized":"2024-01-16T15:24:15.240464+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:15.240213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:15.240213+0000","last_clean_scrub_stamp":"2024-01-16T15:24:15.240213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:36.556315+0000","scrub_duration":0.0034155610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"118'90","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:32.886225+0000","last_change":"2024-01-16T15:24:32.886280+0000","last_active":"2024-01-16T15:24:32.886225+0000","last_peered":"2024-01-16T15:24:32.886225+0000","last_clean":"2024-01-16T15:24:32.886225+0000","last_became_active":"2024-01-16T15:15:39.329994+0000","last_became_peered":"2024-01-16T15:15:39.329994+0000","last_unstale":"2024-01-16T15:24:32.886225+0000","last_undegraded":"2024-01-16T15:24:32.886225+0000","last_fullsized":"2024-01-16T15:24:32.886225+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'90","last_scrub_stamp":"2024-01-16T15:24:32.886073+0000","last_deep_scrub":"118'90","last_deep_scrub_stamp":"2024-01-16T15:24:32.886073+0000","last_clean_scrub_stamp":"2024-01-16T15:24:32.886073+0000","objects_scrubbed":25,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:23:44.179455+0000","scrub_duration":0.29493345700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:02.660042+0000","last_change":"2024-01-16T15:24:02.660042+0000","last_active":"2024-01-16T15:24:02.660042+0000","last_peered":"2024-01-16T15:24:02.660042+0000","last_clean":"2024-01-16T15:24:02.660042+0000","last_became_active":"2024-01-16T15:15:38.276272+0000","last_became_peered":"2024-01-16T15:15:38.276272+0000","last_unstale":"2024-01-16T15:24:02.660042+0000","last_undegraded":"2024-01-16T15:24:02.660042+0000","last_fullsized":"2024-01-16T15:24:02.660042+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:02.659822+0000","last_clean_scrub_stamp":"2024-01-16T15:24:02.659822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:34.891359+0000","scrub_duration":0.0028615590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"118'77","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:22.227309+0000","last_change":"2024-01-16T15:24:22.227483+0000","last_active":"2024-01-16T15:24:22.227309+0000","last_peered":"2024-01-16T15:24:22.227309+0000","last_clean":"2024-01-16T15:24:22.227309+0000","last_became_active":"2024-01-16T15:15:39.315859+0000","last_became_peered":"2024-01-16T15:15:39.315859+0000","last_unstale":"2024-01-16T15:24:22.227309+0000","last_undegraded":"2024-01-16T15:24:22.227309+0000","last_fullsized":"2024-01-16T15:24:22.227309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'77","last_scrub_stamp":"2024-01-16T15:24:22.227034+0000","last_deep_scrub":"118'77","last_deep_scrub_stamp":"2024-01-16T15:24:22.227034+0000","last_clean_scrub_stamp":"2024-01-16T15:24:22.227034+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:51.920270+0000","scrub_duration":0.238057777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692036,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.775581+0000","last_change":"2024-01-16T15:24:16.775581+0000","last_active":"2024-01-16T15:24:16.775581+0000","last_peered":"2024-01-16T15:24:16.775581+0000","last_clean":"2024-01-16T15:24:16.775581+0000","last_became_active":"2024-01-16T15:15:38.297327+0000","last_became_peered":"2024-01-16T15:15:38.297327+0000","last_unstale":"2024-01-16T15:24:16.775581+0000","last_undegraded":"2024-01-16T15:24:16.775581+0000","last_fullsized":"2024-01-16T15:24:16.775581+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:16.775280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:16.775280+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.775280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:13.481653+0000","scrub_duration":0.0039614919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"118'67","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.597682+0000","last_change":"2024-01-16T15:24:16.597682+0000","last_active":"2024-01-16T15:24:16.597682+0000","last_peered":"2024-01-16T15:24:16.597682+0000","last_clean":"2024-01-16T15:24:16.597682+0000","last_became_active":"2024-01-16T15:15:39.332481+0000","last_became_peered":"2024-01-16T15:15:39.332481+0000","last_unstale":"2024-01-16T15:24:16.597682+0000","last_undegraded":"2024-01-16T15:24:16.597682+0000","last_fullsized":"2024-01-16T15:24:16.597682+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'67","last_scrub_stamp":"2024-01-16T15:24:16.597377+0000","last_deep_scrub":"118'67","last_deep_scrub_stamp":"2024-01-16T15:24:16.597377+0000","last_clean_scrub_stamp":"2024-01-16T15:24:16.597377+0000","objects_scrubbed":17,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:57.545654+0000","scrub_duration":0.18894488100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"118'3","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:17.946698+0000","last_change":"2024-01-16T15:24:17.946698+0000","last_active":"2024-01-16T15:24:17.946698+0000","last_peered":"2024-01-16T15:24:17.946698+0000","last_clean":"2024-01-16T15:24:17.946698+0000","last_became_active":"2024-01-16T15:15:38.291534+0000","last_became_peered":"2024-01-16T15:15:38.291534+0000","last_unstale":"2024-01-16T15:24:17.946698+0000","last_undegraded":"2024-01-16T15:24:17.946698+0000","last_fullsized":"2024-01-16T15:24:17.946698+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3","last_scrub_stamp":"2024-01-16T15:24:17.946427+0000","last_deep_scrub":"118'3","last_deep_scrub_stamp":"2024-01-16T15:24:17.946427+0000","last_clean_scrub_stamp":"2024-01-16T15:24:17.946427+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:26.209326+0000","scrub_duration":0.0069638929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"118'85","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:41.038428+0000","last_change":"2024-01-16T15:24:41.038540+0000","last_active":"2024-01-16T15:24:41.038428+0000","last_peered":"2024-01-16T15:24:41.038428+0000","last_clean":"2024-01-16T15:24:41.038428+0000","last_became_active":"2024-01-16T15:15:39.317270+0000","last_became_peered":"2024-01-16T15:15:39.317270+0000","last_unstale":"2024-01-16T15:24:41.038428+0000","last_undegraded":"2024-01-16T15:24:41.038428+0000","last_fullsized":"2024-01-16T15:24:41.038428+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'85","last_scrub_stamp":"2024-01-16T15:24:41.038252+0000","last_deep_scrub":"118'85","last_deep_scrub_stamp":"2024-01-16T15:24:41.038252+0000","last_clean_scrub_stamp":"2024-01-16T15:24:41.038252+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:53.461089+0000","scrub_duration":0.27677636700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"118'31","reported_seq":58,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.451631+0000","last_change":"2024-01-16T15:24:10.451631+0000","last_active":"2024-01-16T15:24:10.451631+0000","last_peered":"2024-01-16T15:24:10.451631+0000","last_clean":"2024-01-16T15:24:10.451631+0000","last_became_active":"2024-01-16T15:15:38.295339+0000","last_became_peered":"2024-01-16T15:15:38.295339+0000","last_unstale":"2024-01-16T15:24:10.451631+0000","last_undegraded":"2024-01-16T15:24:10.451631+0000","last_fullsized":"2024-01-16T15:24:10.451631+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'31","last_scrub_stamp":"2024-01-16T15:24:10.451303+0000","last_deep_scrub":"118'31","last_deep_scrub_stamp":"2024-01-16T15:24:10.451303+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.451303+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:22:06.356152+0000","scrub_duration":0.0039950840000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644,"num_omap_keys":12,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"118'99","reported_seq":143,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:44.709741+0000","last_change":"2024-01-16T15:24:44.709812+0000","last_active":"2024-01-16T15:24:44.709741+0000","last_peered":"2024-01-16T15:24:44.709741+0000","last_clean":"2024-01-16T15:24:44.709741+0000","last_became_active":"2024-01-16T15:15:55.914646+0000","last_became_peered":"2024-01-16T15:15:55.914646+0000","last_unstale":"2024-01-16T15:24:44.709741+0000","last_undegraded":"2024-01-16T15:24:44.709741+0000","last_fullsized":"2024-01-16T15:24:44.709741+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2024-01-16T15:24:44.709536+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2024-01-16T15:24:44.709536+0000","last_clean_scrub_stamp":"2024-01-16T15:24:44.709536+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:41.300338+0000","scrub_duration":0.380922227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:20.759218+0000","last_change":"2024-01-16T15:24:20.759267+0000","last_active":"2024-01-16T15:24:20.759218+0000","last_peered":"2024-01-16T15:24:20.759218+0000","last_clean":"2024-01-16T15:24:20.759218+0000","last_became_active":"2024-01-16T15:15:38.297364+0000","last_became_peered":"2024-01-16T15:15:38.297364+0000","last_unstale":"2024-01-16T15:24:20.759218+0000","last_undegraded":"2024-01-16T15:24:20.759218+0000","last_fullsized":"2024-01-16T15:24:20.759218+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:20.759038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:20.759038+0000","last_clean_scrub_stamp":"2024-01-16T15:24:20.759038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:12.067470+0000","scrub_duration":0.003697947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"118'79","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.701538+0000","last_change":"2024-01-16T15:24:14.701607+0000","last_active":"2024-01-16T15:24:14.701538+0000","last_peered":"2024-01-16T15:24:14.701538+0000","last_clean":"2024-01-16T15:24:14.701538+0000","last_became_active":"2024-01-16T15:15:39.330928+0000","last_became_peered":"2024-01-16T15:15:39.330928+0000","last_unstale":"2024-01-16T15:24:14.701538+0000","last_undegraded":"2024-01-16T15:24:14.701538+0000","last_fullsized":"2024-01-16T15:24:14.701538+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'79","last_scrub_stamp":"2024-01-16T15:24:14.701344+0000","last_deep_scrub":"118'79","last_deep_scrub_stamp":"2024-01-16T15:24:14.701344+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.701344+0000","objects_scrubbed":28,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:46.392454+0000","scrub_duration":0.30808371000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:14.903030+0000","last_change":"2024-01-16T15:24:14.903079+0000","last_active":"2024-01-16T15:24:14.903030+0000","last_peered":"2024-01-16T15:24:14.903030+0000","last_clean":"2024-01-16T15:24:14.903030+0000","last_became_active":"2024-01-16T15:15:38.287129+0000","last_became_peered":"2024-01-16T15:15:38.287129+0000","last_unstale":"2024-01-16T15:24:14.903030+0000","last_undegraded":"2024-01-16T15:24:14.903030+0000","last_fullsized":"2024-01-16T15:24:14.903030+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:14.902847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:14.902847+0000","last_clean_scrub_stamp":"2024-01-16T15:24:14.902847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:18:33.188607+0000","scrub_duration":0.003674142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"118'84","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:26.428031+0000","last_change":"2024-01-16T15:24:26.428184+0000","last_active":"2024-01-16T15:24:26.428031+0000","last_peered":"2024-01-16T15:24:26.428031+0000","last_clean":"2024-01-16T15:24:26.428031+0000","last_became_active":"2024-01-16T15:15:39.312924+0000","last_became_peered":"2024-01-16T15:15:39.312924+0000","last_unstale":"2024-01-16T15:24:26.428031+0000","last_undegraded":"2024-01-16T15:24:26.428031+0000","last_fullsized":"2024-01-16T15:24:26.428031+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'84","last_scrub_stamp":"2024-01-16T15:24:26.427830+0000","last_deep_scrub":"118'84","last_deep_scrub_stamp":"2024-01-16T15:24:26.427830+0000","last_clean_scrub_stamp":"2024-01-16T15:24:26.427830+0000","objects_scrubbed":21,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:10.543608+0000","scrub_duration":0.25438750799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:10.443065+0000","last_change":"2024-01-16T15:24:10.443138+0000","last_active":"2024-01-16T15:24:10.443065+0000","last_peered":"2024-01-16T15:24:10.443065+0000","last_clean":"2024-01-16T15:24:10.443065+0000","last_became_active":"2024-01-16T15:15:38.290215+0000","last_became_peered":"2024-01-16T15:15:38.290215+0000","last_unstale":"2024-01-16T15:24:10.443065+0000","last_undegraded":"2024-01-16T15:24:10.443065+0000","last_fullsized":"2024-01-16T15:24:10.443065+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:10.442885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:10.442885+0000","last_clean_scrub_stamp":"2024-01-16T15:24:10.442885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:31.758987+0000","scrub_duration":0.003956195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"118'93","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:28.565969+0000","last_change":"2024-01-16T15:24:28.566007+0000","last_active":"2024-01-16T15:24:28.565969+0000","last_peered":"2024-01-16T15:24:28.565969+0000","last_clean":"2024-01-16T15:24:28.565969+0000","last_became_active":"2024-01-16T15:15:39.303600+0000","last_became_peered":"2024-01-16T15:15:39.303600+0000","last_unstale":"2024-01-16T15:24:28.565969+0000","last_undegraded":"2024-01-16T15:24:28.565969+0000","last_fullsized":"2024-01-16T15:24:28.565969+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'93","last_scrub_stamp":"2024-01-16T15:24:28.565819+0000","last_deep_scrub":"118'93","last_deep_scrub_stamp":"2024-01-16T15:24:28.565819+0000","last_clean_scrub_stamp":"2024-01-16T15:24:28.565819+0000","objects_scrubbed":27,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:19:37.901880+0000","scrub_duration":0.34082023700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.246601+0000","last_change":"2024-01-16T15:24:11.246654+0000","last_active":"2024-01-16T15:24:11.246601+0000","last_peered":"2024-01-16T15:24:11.246601+0000","last_clean":"2024-01-16T15:24:11.246601+0000","last_became_active":"2024-01-16T15:15:38.286175+0000","last_became_peered":"2024-01-16T15:15:38.286175+0000","last_unstale":"2024-01-16T15:24:11.246601+0000","last_undegraded":"2024-01-16T15:24:11.246601+0000","last_fullsized":"2024-01-16T15:24:11.246601+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:11.246380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:11.246380+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.246380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:14.315962+0000","scrub_duration":0.0038743530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"118'97","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:30.981555+0000","last_change":"2024-01-16T15:24:30.981617+0000","last_active":"2024-01-16T15:24:30.981555+0000","last_peered":"2024-01-16T15:24:30.981555+0000","last_clean":"2024-01-16T15:24:30.981555+0000","last_became_active":"2024-01-16T15:15:39.328231+0000","last_became_peered":"2024-01-16T15:15:39.328231+0000","last_unstale":"2024-01-16T15:24:30.981555+0000","last_undegraded":"2024-01-16T15:24:30.981555+0000","last_fullsized":"2024-01-16T15:24:30.981555+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'97","last_scrub_stamp":"2024-01-16T15:24:30.981393+0000","last_deep_scrub":"118'97","last_deep_scrub_stamp":"2024-01-16T15:24:30.981393+0000","last_clean_scrub_stamp":"2024-01-16T15:24:30.981393+0000","objects_scrubbed":29,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:15.621903+0000","scrub_duration":0.363843851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"118'32","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:09.262161+0000","last_change":"2024-01-16T15:24:09.262216+0000","last_active":"2024-01-16T15:24:09.262161+0000","last_peered":"2024-01-16T15:24:09.262161+0000","last_clean":"2024-01-16T15:24:09.262161+0000","last_became_active":"2024-01-16T15:15:38.281565+0000","last_became_peered":"2024-01-16T15:15:38.281565+0000","last_unstale":"2024-01-16T15:24:09.262161+0000","last_undegraded":"2024-01-16T15:24:09.262161+0000","last_fullsized":"2024-01-16T15:24:09.262161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'32","last_scrub_stamp":"2024-01-16T15:24:09.261992+0000","last_deep_scrub":"118'32","last_deep_scrub_stamp":"2024-01-16T15:24:09.261992+0000","last_clean_scrub_stamp":"2024-01-16T15:24:09.261992+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:09.421019+0000","scrub_duration":0.0066503719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3846,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"118'94","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:04.073597+0000","last_change":"2024-01-16T15:24:04.073657+0000","last_active":"2024-01-16T15:24:04.073597+0000","last_peered":"2024-01-16T15:24:04.073597+0000","last_clean":"2024-01-16T15:24:04.073597+0000","last_became_active":"2024-01-16T15:15:39.333536+0000","last_became_peered":"2024-01-16T15:15:39.333536+0000","last_unstale":"2024-01-16T15:24:04.073597+0000","last_undegraded":"2024-01-16T15:24:04.073597+0000","last_fullsized":"2024-01-16T15:24:04.073597+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2024-01-16T15:24:04.073487+0000","last_clean_scrub_stamp":"2024-01-16T15:24:04.073487+0000","objects_scrubbed":32,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:45.014652+0000","scrub_duration":0.28876764900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:11.746241+0000","last_change":"2024-01-16T15:24:11.746367+0000","last_active":"2024-01-16T15:24:11.746241+0000","last_peered":"2024-01-16T15:24:11.746241+0000","last_clean":"2024-01-16T15:24:11.746241+0000","last_became_active":"2024-01-16T15:15:38.296176+0000","last_became_peered":"2024-01-16T15:15:38.296176+0000","last_unstale":"2024-01-16T15:24:11.746241+0000","last_undegraded":"2024-01-16T15:24:11.746241+0000","last_fullsized":"2024-01-16T15:24:11.746241+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:11.746034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:11.746034+0000","last_clean_scrub_stamp":"2024-01-16T15:24:11.746034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:13.384227+0000","scrub_duration":0.0037103750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"118'66","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:19.148119+0000","last_change":"2024-01-16T15:24:19.148119+0000","last_active":"2024-01-16T15:24:19.148119+0000","last_peered":"2024-01-16T15:24:19.148119+0000","last_clean":"2024-01-16T15:24:19.148119+0000","last_became_active":"2024-01-16T15:15:39.324061+0000","last_became_peered":"2024-01-16T15:15:39.324061+0000","last_unstale":"2024-01-16T15:24:19.148119+0000","last_undegraded":"2024-01-16T15:24:19.148119+0000","last_fullsized":"2024-01-16T15:24:19.148119+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'66","last_scrub_stamp":"2024-01-16T15:24:19.147855+0000","last_deep_scrub":"118'66","last_deep_scrub_stamp":"2024-01-16T15:24:19.147855+0000","last_clean_scrub_stamp":"2024-01-16T15:24:19.147855+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:29.163868+0000","scrub_duration":0.182029521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.536760+0000","last_change":"2024-01-16T15:24:05.536760+0000","last_active":"2024-01-16T15:24:05.536760+0000","last_peered":"2024-01-16T15:24:05.536760+0000","last_clean":"2024-01-16T15:24:05.536760+0000","last_became_active":"2024-01-16T15:15:38.299609+0000","last_became_peered":"2024-01-16T15:15:38.299609+0000","last_unstale":"2024-01-16T15:24:05.536760+0000","last_undegraded":"2024-01-16T15:24:05.536760+0000","last_fullsized":"2024-01-16T15:24:05.536760+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:24:05.536530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:24:05.536530+0000","last_clean_scrub_stamp":"2024-01-16T15:24:05.536530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:03.108287+0000","scrub_duration":0.0035509750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"118'119","reported_seq":275,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-01-16T15:24:06.491584+0000","last_change":"2024-01-16T15:24:06.491710+0000","last_active":"2024-01-16T15:24:06.491584+0000","last_peered":"2024-01-16T15:24:06.491584+0000","last_clean":"2024-01-16T15:24:06.491584+0000","last_became_active":"2024-01-16T14:57:02.517772+0000","last_became_peered":"2024-01-16T14:57:02.517772+0000","last_unstale":"2024-01-16T15:24:06.491584+0000","last_undegraded":"2024-01-16T15:24:06.491584+0000","last_fullsized":"2024-01-16T15:24:06.491584+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'119","last_scrub_stamp":"2024-01-16T15:24:06.491352+0000","last_deep_scrub":"118'119","last_deep_scrub_stamp":"2024-01-16T15:24:06.491352+0000","last_clean_scrub_stamp":"2024-01-16T15:24:06.491352+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:03.082154+0000","scrub_duration":0.061931657000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452737,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5609,"ondisk_log_size":5609,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1252754,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":491,"num_write_kb":1463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5644,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2505508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":474,"ondisk_log_size":474,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542483,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2100980,"kb_used_data":2025452,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92270860,"statfs":{"total":96636764160,"available":94485360640,"internally_reserved":0,"allocated":2074062848,"data_stored":2073213012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2170000000000001,"15min":1.248},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.57799999999999996},"max":{"1min":1.573,"5min":1.6679999999999999,"15min":13.566000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3169999999999999,"15min":1.2949999999999999},"min":{"1min":0.79000000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.579,"5min":1.984,"15min":10.728},"last":1.3200000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.107,"15min":1.155},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.627},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":13.375},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.155,"15min":1.1359999999999999},"min":{"1min":0.86099999999999999,"5min":0.67800000000000005,"15min":0.66200000000000003},"max":{"1min":1.661,"5min":1.6799999999999999,"15min":2.056},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.272,"15min":1.2490000000000001},"min":{"1min":0.99399999999999999,"5min":0.76400000000000001,"15min":0.72199999999999998},"max":{"1min":1.611,"5min":1.7969999999999999,"15min":2.145},"last":1.224},{"interface":"front","average":{"1min":1.129,"5min":1.171,"15min":1.1599999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.623,"5min":1.724,"15min":1.9630000000000001},"last":1.2529999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.284,"15min":1.296},"min":{"1min":0.83399999999999996,"5min":0.82999999999999996,"15min":0.67000000000000004},"max":{"1min":1.819,"5min":2.27,"15min":13.657999999999999},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.472,"15min":1.4259999999999999},"min":{"1min":1.1020000000000001,"5min":1.0029999999999999,"15min":0.76500000000000001},"max":{"1min":1.78,"5min":2.3119999999999998,"15min":2.4319999999999999},"last":1.3979999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2410000000000001,"15min":1.26},"min":{"1min":0.92600000000000005,"5min":0.57599999999999996,"15min":0.56999999999999995},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.421,"15min":1.4510000000000001},"min":{"1min":0.76900000000000002,"5min":0.72899999999999998,"15min":0.69799999999999995},"max":{"1min":1.758,"5min":1.7829999999999999,"15min":13.782999999999999},"last":1.446}]},{"osd":5,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.452,"15min":1.4330000000000001},"min":{"1min":0.85299999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.843,"5min":2.0569999999999999,"15min":2.2320000000000002},"last":1.488},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.462,"15min":1.448},"min":{"1min":0.98599999999999999,"5min":0.66000000000000003,"15min":0.626},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.1890000000000001},"last":1.4179999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:24:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4419999999999999,"15min":1.4299999999999999},"min":{"1min":1.0429999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.141,"5min":2.141,"15min":2.327},"last":1.133},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4379999999999999,"15min":1.4950000000000001},"min":{"1min":0.93700000000000006,"5min":0.63400000000000001,"15min":0.61599999999999999},"max":{"1min":2.274,"5min":2.274,"15min":13.725},"last":1.0780000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477736,"kb_used_data":1406368,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92894104,"statfs":{"total":96636764160,"available":95123562496,"internally_reserved":0,"allocated":1440120832,"data_stored":1439281876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1699999999999999,"15min":1.1739999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58099999999999996},"max":{"1min":1.6399999999999999,"5min":1.764,"15min":3.9289999999999998},"last":1.262},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.1950000000000001,"15min":1.246},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.41599999999999998},"max":{"1min":1.645,"5min":1.7130000000000001,"15min":8.0630000000000006},"last":1.367}]},{"osd":1,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2370000000000001,"15min":1.282},"min":{"1min":0.82299999999999995,"5min":0.63700000000000001,"15min":0.497},"max":{"1min":1.5880000000000001,"5min":1.7030000000000001,"15min":10.518000000000001},"last":1.411},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.302,"15min":1.286},"min":{"1min":0.95199999999999996,"5min":0.65000000000000002,"15min":0.53100000000000003},"max":{"1min":1.696,"5min":1.887,"15min":7.2919999999999998},"last":0.88600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2370000000000001,"15min":1.298},"min":{"1min":0.755,"5min":0.755,"15min":0.47099999999999997},"max":{"1min":1.5840000000000001,"5min":1.784,"15min":6.3040000000000003},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2749999999999999,"15min":1.423},"min":{"1min":0.83099999999999996,"5min":0.72599999999999998,"15min":0.57099999999999995},"max":{"1min":1.6910000000000001,"5min":1.8240000000000001,"15min":15.282},"last":1.196}]},{"osd":3,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.343,"15min":1.357},"min":{"1min":1.0920000000000001,"5min":0.71899999999999997,"15min":0.64100000000000001},"max":{"1min":1.784,"5min":1.9490000000000001,"15min":4.1109999999999998},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.3839999999999999,"15min":1.458},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.50900000000000001},"max":{"1min":1.621,"5min":1.7370000000000001,"15min":9.0960000000000001},"last":1.3}]},{"osd":4,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.3640000000000001,"15min":1.3759999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.49299999999999999},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":5.4669999999999996},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3580000000000001,"15min":1.413},"min":{"1min":0.95799999999999996,"5min":0.81799999999999995,"15min":0.41199999999999998},"max":{"1min":1.651,"5min":1.7989999999999999,"15min":5.3360000000000003},"last":1.613}]},{"osd":5,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3380000000000001,"15min":1.403},"min":{"1min":0.89200000000000002,"5min":0.65700000000000003,"15min":0.436},"max":{"1min":1.7370000000000001,"5min":1.9239999999999999,"15min":10.388999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.494,"5min":1.518,"15min":1.55},"min":{"1min":1.0660000000000001,"5min":1.016,"15min":0.5},"max":{"1min":2.0070000000000001,"5min":2.012,"15min":7.9640000000000004},"last":1.119}]},{"osd":7,"last update":"Tue Jan 16 15:24:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5449999999999999,"15min":1.589},"min":{"1min":1.1020000000000001,"5min":0.98399999999999999,"15min":0.48899999999999999},"max":{"1min":1.738,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.474,"15min":1.6080000000000001},"min":{"1min":0.97399999999999998,"5min":0.84199999999999997,"15min":0.58499999999999996},"max":{"1min":1.8540000000000001,"5min":1.984,"15min":16.175999999999998},"last":1.0269999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542483,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174568,"kb_used_data":1107360,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93197272,"statfs":{"total":96636764160,"available":95434006528,"internally_reserved":0,"allocated":1133936640,"data_stored":1133098096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.143,"15min":1.2150000000000001},"min":{"1min":0.79900000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":10.305999999999999},"last":1.169},{"interface":"front","average":{"1min":1.085,"5min":1.2050000000000001,"15min":1.22},"min":{"1min":0.83599999999999997,"5min":0.80900000000000005,"15min":0.59699999999999998},"max":{"1min":1.552,"5min":4.9950000000000001,"15min":12.922000000000001},"last":1.2350000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.3,"15min":1.3089999999999999},"min":{"1min":0.83799999999999997,"5min":0.79400000000000004,"15min":0.63900000000000001},"max":{"1min":1.6619999999999999,"5min":5.2249999999999996,"15min":8.1820000000000004},"last":1.569},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.175,"15min":1.1779999999999999},"min":{"1min":0.75700000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":2.597},"last":1.5029999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2490000000000001,"15min":1.26},"min":{"1min":0.71999999999999997,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6759999999999999,"5min":1.712,"15min":3.0470000000000002},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.238,"15min":1.26},"min":{"1min":0.78800000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6950000000000001,"5min":1.756,"15min":8.0619999999999994},"last":1.371}]},{"osd":3,"last update":"Tue Jan 16 15:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3999999999999999,"15min":1.421},"min":{"1min":1.0449999999999999,"5min":0.83099999999999996,"15min":0.71799999999999997},"max":{"1min":1.794,"5min":5.157,"15min":8.0809999999999995},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.415,"15min":1.423},"min":{"1min":0.88800000000000001,"5min":0.67500000000000004,"15min":0.61599999999999999},"max":{"1min":1.958,"5min":2.6960000000000002,"15min":3.1619999999999999},"last":1.3140000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.21,"15min":1.258},"min":{"1min":0.871,"5min":0.71899999999999997,"15min":0.56100000000000005},"max":{"1min":1.7190000000000001,"5min":1.7689999999999999,"15min":4.8490000000000002},"last":1.012},{"interface":"front","average":{"1min":1.411,"5min":1.383,"15min":1.403},"min":{"1min":1.077,"5min":0.84099999999999997,"15min":0.57399999999999995},"max":{"1min":2.008,"5min":5.0810000000000004,"15min":8.3550000000000004},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4219999999999999,"15min":1.5029999999999999},"min":{"1min":0.90700000000000003,"5min":0.79300000000000004,"15min":0.65200000000000002},"max":{"1min":1.907,"5min":1.9530000000000001,"15min":17.419},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.504,"5min":1.454,"15min":1.528},"min":{"1min":1.02,"5min":0.75,"15min":0.69799999999999995},"max":{"1min":1.9039999999999999,"5min":2.2959999999999998,"15min":16.262},"last":1.5349999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.498,"15min":1.5609999999999999},"min":{"1min":1.109,"5min":1.024,"15min":0.67000000000000004},"max":{"1min":2.4990000000000001,"5min":3.875,"15min":8.3629999999999995},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.476,"5min":1.456,"15min":1.5069999999999999},"min":{"1min":1.0549999999999999,"5min":0.73999999999999999,"15min":0.52100000000000002},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":5.0780000000000003},"last":1.671}]}]},{"osd":4,"up_from":14,"seq":60129542484,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936484,"kb_used_data":1865116,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435356,"statfs":{"total":96636764160,"available":94653804544,"internally_reserved":0,"allocated":1909878784,"data_stored":1909043912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.262,"15min":1.2569999999999999},"min":{"1min":0.88400000000000001,"5min":0.64900000000000002,"15min":0.63300000000000001},"max":{"1min":1.6339999999999999,"5min":1.774,"15min":3.3159999999999998},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.1599999999999999,"15min":1.1970000000000001},"min":{"1min":0.89600000000000002,"5min":0.68300000000000005,"15min":0.47499999999999998},"max":{"1min":1.915,"5min":1.915,"15min":5.7430000000000003},"last":1.278}]},{"osd":1,"last update":"Tue Jan 16 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.268,"15min":1.3009999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.67600000000000005},"max":{"1min":1.766,"5min":2.0059999999999998,"15min":3.7850000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.22,"15min":1.2350000000000001},"min":{"1min":0.78600000000000003,"5min":0.71299999999999997,"15min":0.57899999999999996},"max":{"1min":1.655,"5min":2.0489999999999999,"15min":4.6760000000000002},"last":1.4099999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1990000000000001,"15min":1.2330000000000001},"min":{"1min":0.84199999999999997,"5min":0.79300000000000004,"15min":0.69199999999999995},"max":{"1min":1.454,"5min":1.9610000000000001,"15min":13.731999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.3680000000000001,"15min":1.373},"min":{"1min":0.98199999999999998,"5min":0.79400000000000004,"15min":0.73199999999999998},"max":{"1min":1.718,"5min":1.7729999999999999,"15min":2.2570000000000001},"last":1.6259999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.349,"15min":1.3480000000000001},"min":{"1min":0.875,"5min":0.69999999999999996,"15min":0.59899999999999998},"max":{"1min":1.8260000000000001,"5min":2.1549999999999998,"15min":4.5789999999999997},"last":1.149},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.323,"15min":1.3660000000000001},"min":{"1min":0.94299999999999995,"5min":0.92100000000000004,"15min":0.71699999999999997},"max":{"1min":1.702,"5min":1.8620000000000001,"15min":13.914},"last":1.0700000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3180000000000001,"15min":1.363},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.47299999999999998},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":10.211},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.389,"15min":1.415},"min":{"1min":0.90900000000000003,"5min":0.79400000000000004,"15min":0.58299999999999996},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":4.673},"last":1.323}]},{"osd":6,"last update":"Tue Jan 16 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.3779999999999999,"15min":1.423},"min":{"1min":1.0489999999999999,"5min":0.83099999999999996,"15min":0.60699999999999998},"max":{"1min":2.476,"5min":2.476,"15min":13.973000000000001},"last":1.133},{"interface":"front","average":{"1min":1.532,"5min":1.5089999999999999,"15min":1.5389999999999999},"min":{"1min":1.069,"5min":0.85099999999999998,"15min":0.52400000000000002},"max":{"1min":2.3660000000000001,"5min":2.3660000000000001,"15min":3.0979999999999999},"last":1.5029999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.504,"15min":1.542},"min":{"1min":1.1020000000000001,"5min":0.878,"15min":0.53500000000000003},"max":{"1min":2.306,"5min":2.306,"15min":4.7640000000000002},"last":1.03},{"interface":"front","average":{"1min":1.579,"5min":1.5309999999999999,"15min":1.575},"min":{"1min":1.3080000000000001,"5min":0.95999999999999996,"15min":0.55200000000000005},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":13.852},"last":1.1819999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542483,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1832764,"kb_used_data":1751668,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":92539076,"statfs":{"total":96636764160,"available":94760013824,"internally_reserved":0,"allocated":1793708032,"data_stored":1792852111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90600000000000003,"15min":0.95299999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.34300000000000003},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":2.9129999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.115,"15min":1.103},"min":{"1min":0.52500000000000002,"5min":0.438,"15min":0.29699999999999999},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":4.0389999999999997},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.95399999999999996,"15min":0.98999999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.33100000000000002},"max":{"1min":1.554,"5min":1.679,"15min":12.335000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0620000000000001,"15min":1.1299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.315},"max":{"1min":1.774,"5min":1.774,"15min":10.638},"last":1.367}]},{"osd":2,"last update":"Tue Jan 16 15:24:47 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0980000000000001,"15min":1.139},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.35999999999999999},"max":{"1min":1.617,"5min":1.617,"15min":8.3740000000000006},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.198,"15min":1.2250000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.647,"5min":1.669,"15min":4.2629999999999999},"last":1.018}]},{"osd":4,"last update":"Tue Jan 16 15:24:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2909999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.39300000000000002},"max":{"1min":1.7310000000000001,"5min":1.76,"15min":15.512},"last":1.24},{"interface":"front","average":{"1min":1.379,"5min":1.319,"15min":1.419},"min":{"1min":0.83699999999999997,"5min":0.82499999999999996,"15min":0.52100000000000002},"max":{"1min":4.8179999999999996,"5min":4.8179999999999996,"15min":14.250999999999999},"last":1.46}]},{"osd":5,"last update":"Tue Jan 16 15:24:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.373,"15min":1.413},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.48499999999999999},"max":{"1min":1.675,"5min":1.8520000000000001,"15min":8.4290000000000003},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.375,"15min":1.407},"min":{"1min":0.84799999999999998,"5min":0.80200000000000005,"15min":0.42099999999999999},"max":{"1min":1.7150000000000001,"5min":1.994,"15min":4.3070000000000004},"last":1.1870000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:24:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.3660000000000001,"15min":1.385},"min":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.41499999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":3.956},"last":1.415},{"interface":"front","average":{"1min":1.383,"5min":1.4510000000000001,"15min":1.484},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.54900000000000004},"max":{"1min":1.724,"5min":1.998,"15min":11.667},"last":1.5449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:24:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4650000000000001,"15min":1.54},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.53700000000000003},"max":{"1min":1.915,"5min":2.0550000000000002,"15min":11.042999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4319999999999999,"15min":1.4570000000000001},"min":{"1min":0.86599999999999999,"5min":0.70399999999999996,"15min":0.435},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":4.2880000000000003},"last":1.5029999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542483,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1211492,"kb_used_data":1140124,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93160348,"statfs":{"total":96636764160,"available":95396196352,"internally_reserved":0,"allocated":1167486976,"data_stored":1166655701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.90300000000000002,"15min":0.89700000000000002},"min":{"1min":0.65300000000000002,"5min":0.54200000000000004,"15min":0.311},"max":{"1min":1.331,"5min":1.6759999999999999,"15min":2.6699999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.048,"15min":1.0600000000000001},"min":{"1min":0.64100000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.4610000000000001,"5min":1.5549999999999999,"15min":3.52},"last":1.413}]},{"osd":1,"last update":"Tue Jan 16 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0509999999999999,"15min":1.0920000000000001},"min":{"1min":0.76400000000000001,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.4970000000000001,"5min":1.5820000000000001,"15min":6.9240000000000004},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.238,"5min":1.115,"15min":1.115},"min":{"1min":0.86899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":2.9900000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.0429999999999999,"15min":1.0589999999999999},"min":{"1min":0.81299999999999994,"5min":0.49099999999999999,"15min":0.45700000000000002},"max":{"1min":1.925,"5min":1.925,"15min":6.1260000000000003},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2110000000000001,"15min":1.238},"min":{"1min":0.86299999999999999,"5min":0.51200000000000001,"15min":0.47999999999999998},"max":{"1min":1.6319999999999999,"5min":1.6950000000000001,"15min":6.2510000000000003},"last":1.1419999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":1.1499999999999999,"5min":0.91300000000000003,"15min":0.68400000000000005},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":3.012},"last":1.381},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.383,"15min":1.3979999999999999},"min":{"1min":0.999,"5min":0.75600000000000001,"15min":0.71099999999999997},"max":{"1min":1.7889999999999999,"5min":2.4020000000000001,"15min":3.54},"last":1.329}]},{"osd":5,"last update":"Tue Jan 16 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.3700000000000001,"15min":1.391},"min":{"1min":1.1240000000000001,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":1.75,"5min":1.774,"15min":8.625},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.3080000000000001,"15min":1.327},"min":{"1min":1.0409999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6040000000000001,"5min":1.944,"15min":6.3499999999999996},"last":1.252}]},{"osd":6,"last update":"Tue Jan 16 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.393,"15min":1.383},"min":{"1min":1.1839999999999999,"5min":0.96299999999999997,"15min":0.71499999999999997},"max":{"1min":1.823,"5min":2.0899999999999999,"15min":6.4729999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.415,"15min":1.4370000000000001},"min":{"1min":0.98799999999999999,"5min":0.76500000000000001,"15min":0.71099999999999997},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":3.2469999999999999},"last":1.306}]},{"osd":7,"last update":"Tue Jan 16 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.482,"15min":1.496},"min":{"1min":1.0840000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":3.7290000000000001},"last":1.51},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.4830000000000001,"15min":1.4970000000000001},"min":{"1min":1.3089999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.8779999999999999,"5min":2.0379999999999998,"15min":6.4249999999999998},"last":1.464}]}]},{"osd":1,"up_from":14,"seq":60129542483,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412264,"kb_used_data":1336736,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92959576,"statfs":{"total":96636764160,"available":95190605824,"internally_reserved":0,"allocated":1368817664,"data_stored":1367982226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.91900000000000004,"15min":0.97399999999999998},"min":{"1min":0.51500000000000001,"5min":0.44600000000000001,"15min":0.33400000000000002},"max":{"1min":1.3460000000000001,"5min":1.3879999999999999,"15min":8.8499999999999996},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1240000000000001,"15min":1.155},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.359,"5min":1.5740000000000001,"15min":8.9380000000000006},"last":1.127}]},{"osd":2,"last update":"Tue Jan 16 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93000000000000005,"15min":0.94899999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":3.4460000000000002},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.107,"15min":1.1339999999999999},"min":{"1min":0.56399999999999995,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.304,"5min":1.677,"15min":2.3420000000000001},"last":0.86599999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.177,"15min":1.228},"min":{"1min":0.61399999999999999,"5min":0.57799999999999996,"15min":0.45400000000000001},"max":{"1min":1.4650000000000001,"5min":1.6870000000000001,"15min":3.282},"last":1.079},{"interface":"front","average":{"1min":1.044,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":8.6910000000000007},"last":1.512}]},{"osd":4,"last update":"Tue Jan 16 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.333,"15min":1.351},"min":{"1min":0.94899999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6399999999999999,"5min":1.716,"15min":3.044},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.24,"15min":1.2869999999999999},"min":{"1min":0.80600000000000005,"5min":0.746,"15min":0.57899999999999996},"max":{"1min":1.712,"5min":1.9199999999999999,"15min":4.7640000000000002},"last":1.554}]},{"osd":5,"last update":"Tue Jan 16 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.327,"15min":1.3320000000000001},"min":{"1min":0.94399999999999995,"5min":0.70299999999999996,"15min":0.50800000000000001},"max":{"1min":1.6990000000000001,"5min":2.2530000000000001,"15min":2.601},"last":1.74},{"interface":"front","average":{"1min":1.431,"5min":1.4419999999999999,"15min":1.462},"min":{"1min":0.92400000000000004,"5min":0.77500000000000002,"15min":0.66300000000000003},"max":{"1min":1.968,"5min":1.968,"15min":2.367},"last":1.5940000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4099999999999999,"15min":1.4450000000000001},"min":{"1min":0.88600000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6240000000000001,"5min":2.0870000000000002,"15min":4.7779999999999996},"last":1.339},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3520000000000001,"15min":1.417},"min":{"1min":0.77500000000000002,"5min":0.746,"15min":0.56899999999999995},"max":{"1min":1.804,"5min":2.339,"15min":4.8300000000000001},"last":1.677}]},{"osd":7,"last update":"Tue Jan 16 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4370000000000001,"15min":1.4630000000000001},"min":{"1min":1.1240000000000001,"5min":0.76500000000000001,"15min":0.58999999999999997},"max":{"1min":1.8959999999999999,"5min":2.0510000000000002,"15min":3.2690000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.458,"15min":1.506},"min":{"1min":0.99299999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.754,"5min":2.198,"15min":3.4079999999999999},"last":1.633}]}]},{"osd":0,"up_from":14,"seq":60129542484,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2040768,"kb_used_data":1965240,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92331072,"statfs":{"total":96636764160,"available":94547017728,"internally_reserved":0,"allocated":2012405760,"data_stored":2011546000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77330886},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.91700000000000004,"15min":0.86699999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.34899999999999998},"max":{"1min":1.5740000000000001,"5min":2.387,"15min":2.387},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.8160000000000001,"5min":1.1579999999999999,"15min":1.0820000000000001},"min":{"1min":0.49299999999999999,"5min":0.441,"15min":0.29399999999999998},"max":{"1min":18.363,"5min":18.363,"15min":18.363},"last":1.345}]},{"osd":2,"last update":"Tue Jan 16 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.3340000000000001,"15min":1.276},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.45100000000000001},"max":{"1min":11.907999999999999,"5min":11.907999999999999,"15min":11.907999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.254,"15min":1.1639999999999999},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.46400000000000002},"max":{"1min":12.212,"5min":12.212,"15min":12.212},"last":1.2090000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.98499999999999999,"15min":1.0429999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.46600000000000003},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":10.83},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.145,"15min":1.1719999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":7.9720000000000004},"last":1.258}]},{"osd":4,"last update":"Tue Jan 16 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.325,"15min":1.3120000000000001},"min":{"1min":0.81200000000000006,"5min":0.78800000000000003,"15min":0.72999999999999998},"max":{"1min":2.351,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.54},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.256,"15min":1.2829999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.67200000000000004},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":10.986000000000001},"last":1.3799999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.3979999999999999,"15min":1.3819999999999999},"min":{"1min":0.80100000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":3.6539999999999999,"5min":3.6539999999999999,"15min":3.6539999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.292,"15min":1.337},"min":{"1min":0.81699999999999995,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":10.928000000000001},"last":1.3}]},{"osd":6,"last update":"Tue Jan 16 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.454,"15min":1.4570000000000001},"min":{"1min":0.95099999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.0099999999999998,"5min":2.2069999999999999,"15min":8.0399999999999991},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.5,"15min":1.4770000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.68500000000000005},"max":{"1min":2.2210000000000001,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.651}]},{"osd":7,"last update":"Tue Jan 16 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.476,"15min":1.4890000000000001},"min":{"1min":0.95599999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.0179999999999998,"5min":2.129,"15min":8.0960000000000001},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.748,"5min":1.569,"15min":1.569},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.70899999999999996},"max":{"1min":7.3230000000000004,"5min":7.3230000000000004,"15min":13.718},"last":1.5980000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1248222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1248180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2009083904,"data_stored":2009072258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166020608,"data_stored":1166016703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908416512,"data_stored":1908408760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132466176,"data_stored":1132462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071998464,"data_stored":2071986876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:25:06.803 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-16T15:25:07.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.041+0000 7fa001c00640 1 Processor -- start 2024-01-16T15:25:07.039 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.041+0000 7fa001c00640 1 -- start start 2024-01-16T15:25:07.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7fa001c00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9ffc14b2c0 0x7f9ffc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7fa001c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc071070 0x7f9ffc0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7fa001c00640 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc0cfbf0 con 0x7f9ffc0cf7d0 2024-01-16T15:25:07.042 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7fa001c00640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ffc14b6a0 con 0x7f9ffc14b2c0 2024-01-16T15:25:07.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7fa001c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc071690 con 0x7f9ffc071070 2024-01-16T15:25:07.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7f9ffaa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9ffc14b2c0 0x7f9ffc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7f9ffa000640 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f9ffc0cf7d0 0x7f9ffc070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42242/0 (socket says 172.21.15.141:42242) 2024-01-16T15:25:07.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7f9ffa000640 1 -- 172.21.15.141:0/4291874719 learned_addr learned my addr 172.21.15.141:0/4291874719 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-16T15:25:07.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.043+0000 7f9ffaa00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9ffc14b2c0 0x7f9ffc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37112/0 (socket says 172.21.15.141:37112) 2024-01-16T15:25:07.043 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.044+0000 7f9ffaa00640 1 -- 172.21.15.141:0/4291874719 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc071070 msgr2=0x7f9ffc0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:07.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.044+0000 7f9ffb400640 1 --2- 172.21.15.141:0/4291874719 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc071070 0x7f9ffc0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.044+0000 7f9ffaa00640 1 --2- 172.21.15.141:0/4291874719 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc071070 0x7f9ffc0cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.044+0000 7f9ffaa00640 1 -- 172.21.15.141:0/4291874719 >> v1:172.21.15.141:6789/0 conn(0x7f9ffc0cf7d0 legacy=0x7f9ffc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.044+0000 7f9ffaa00640 1 -- 172.21.15.141:0/4291874719 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc071970 con 0x7f9ffc14b2c0 2024-01-16T15:25:07.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.044+0000 7f9ff9600640 1 -- 172.21.15.141:0/4291874719 <== mon.0 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38205290 0 0) 0x7f9ffc0cfbf0 con 0x7f9ffc0cf7d0 2024-01-16T15:25:07.044 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.044+0000 7f9ffb400640 1 --2- 172.21.15.141:0/4291874719 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc071070 0x7f9ffc0cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:07.049 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.045+0000 7f9ffaa00640 1 --2- 172.21.15.141:0/4291874719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc14b2c0 0x7f9ffc0cf290 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0097a0 tx=0x7f9fec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753718041338bda2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:07.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.045+0000 7f9ff9600640 1 -- 172.21.15.141:0/4291874719 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec02d040 con 0x7f9ffc14b2c0 2024-01-16T15:25:07.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7f9ff9600640 1 -- 172.21.15.141:0/4291874719 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec031810 con 0x7f9ffc14b2c0 2024-01-16T15:25:07.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7f9ff9600640 1 -- 172.21.15.141:0/4291874719 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec031ba0 con 0x7f9ffc14b2c0 2024-01-16T15:25:07.050 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 -- 172.21.15.141:0/4291874719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc14b2c0 msgr2=0x7f9ffc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 --2- 172.21.15.141:0/4291874719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc14b2c0 0x7f9ffc0cf290 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0097a0 tx=0x7f9fec02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 -- 172.21.15.141:0/4291874719 shutdown_connections 2024-01-16T15:25:07.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 --2- 172.21.15.141:0/4291874719 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc071070 0x7f9ffc0cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 --2- 172.21.15.141:0/4291874719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc14b2c0 0x7f9ffc0cf290 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.051 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 -- 172.21.15.141:0/4291874719 >> 172.21.15.141:0/4291874719 conn(0x7f9ffc06a800 msgr2=0x7f9ffc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:07.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 -- 172.21.15.141:0/4291874719 shutdown_connections 2024-01-16T15:25:07.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.046+0000 7fa001c00640 1 -- 172.21.15.141:0/4291874719 wait complete. 2024-01-16T15:25:07.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.048+0000 7fa001c00640 1 Processor -- start 2024-01-16T15:25:07.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.048+0000 7fa001c00640 1 -- start start 2024-01-16T15:25:07.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7fa001c00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc071070 0x7f9ffc15ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.052 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7fa001c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ffc0cf7d0 0x7f9ffc1631e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7fa001c00640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc14b2c0 0x7f9ffc1648e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7fa001c00640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc163720 con 0x7f9ffc071070 2024-01-16T15:25:07.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7fa001c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc164e20 con 0x7f9ffc0cf7d0 2024-01-16T15:25:07.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7fa001c00640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc162140 con 0x7f9ffc14b2c0 2024-01-16T15:25:07.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7f9ffaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc071070 0x7f9ffc15ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.053 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7f9ffaa00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc071070 0x7f9ffc15ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37118/0 (socket says 172.21.15.141:37118) 2024-01-16T15:25:07.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.049+0000 7f9ffaa00640 1 -- 172.21.15.141:0/3444341999 learned_addr learned my addr 172.21.15.141:0/3444341999 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:25:07.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffa000640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ffc0cf7d0 0x7f9ffc1631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffb400640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc14b2c0 0x7f9ffc1648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffaa00640 1 -- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc14b2c0 msgr2=0x7f9ffc1648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffaa00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc14b2c0 0x7f9ffc1648e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffaa00640 1 -- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ffc0cf7d0 msgr2=0x7f9ffc1631e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.054 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffaa00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ffc0cf7d0 0x7f9ffc1631e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffaa00640 1 -- 172.21.15.141:0/3444341999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec009400 con 0x7f9ffc071070 2024-01-16T15:25:07.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffb400640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc14b2c0 0x7f9ffc1648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:07.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ffaa00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc071070 0x7f9ffc15ef40 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f9fec034040 tx=0x7f9fec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:07.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec03aad0 con 0x7f9ffc071070 2024-01-16T15:25:07.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.050+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec03ac30 con 0x7f9ffc071070 2024-01-16T15:25:07.055 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.051+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec004030 con 0x7f9ffc071070 2024-01-16T15:25:07.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.051+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ffc162420 con 0x7f9ffc071070 2024-01-16T15:25:07.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.051+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc1629e0 con 0x7f9ffc071070 2024-01-16T15:25:07.056 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.052+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ffc0675d0 con 0x7f9ffc071070 2024-01-16T15:25:07.057 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.053+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9fec042070 con 0x7f9ffc071070 2024-01-16T15:25:07.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.054+0000 7f9ff2a00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fd406c580 0x7f9fd406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.054+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9fec0aaa00 con 0x7f9ffc071070 2024-01-16T15:25:07.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.054+0000 7f9ffa000640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fd406c580 0x7f9fd406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.056+0000 7f9ffa000640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fd406c580 0x7f9fd406ea40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0b3b70 tx=0x7f9fe4007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:07.058 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.058+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fec0770c0 con 0x7f9ffc071070 2024-01-16T15:25:07.245 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.247+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f9ffc148340 con 0x7f9ffc071070 2024-01-16T15:25:07.251 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.254+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v45)=0 v45) v1 ==== 123+0+0 (secure 0 0 0) 0x7f9fec035020 con 0x7f9ffc071070 2024-01-16T15:25:07.253 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.256+0000 7f9ff2a00640 1 -- 172.21.15.141:0/3444341999 <== mon.0 v2:172.21.15.141:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9fec0770c0 con 0x7f9ffc071070 2024-01-16T15:25:07.256 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.258+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fd406c580 msgr2=0x7f9fd406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.258+0000 7fa001c00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fd406c580 0x7f9fd406ea40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0b3b70 tx=0x7f9fe4007b50 comp rx=0 tx=0).stop 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.258+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc071070 msgr2=0x7f9ffc15ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.258+0000 7fa001c00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc071070 0x7f9ffc15ef40 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f9fec034040 tx=0x7f9fec0046c0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 shutdown_connections 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f9fd406c580 0x7f9fd406ea40 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f9ffc14b2c0 0x7f9ffc1648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ffc0cf7d0 0x7f9ffc1631e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.257 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 --2- 172.21.15.141:0/3444341999 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ffc071070 0x7f9ffc15ef40 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 >> 172.21.15.141:0/3444341999 conn(0x7f9ffc06a800 msgr2=0x7f9ffc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:07.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 shutdown_connections 2024-01-16T15:25:07.258 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.259+0000 7fa001c00640 1 -- 172.21.15.141:0/3444341999 wait complete. 2024-01-16T15:25:07.275 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T15:25:07.276 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:25:07.290 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.293+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.290 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.293+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.290 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.293+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.291 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.293+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.291 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.293+0000 7f0f31600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.292 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.295+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.292 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.295+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.292 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.295+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.293 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.295+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.293 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-01-16T15:25:07.295+0000 7f0f2f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:07.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.489+0000 7f0fc7000640 1 Processor -- start 2024-01-16T15:25:07.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.489+0000 7f0fc7000640 1 -- start start 2024-01-16T15:25:07.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.490+0000 7f0fc7000640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0fc013df80 0x7f0fc015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.490+0000 7f0fc7000640 1 --2- >> v2:172.21.15.188:3300/0 conn(0x7f0fc00cf9a0 0x7f0fc00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.490+0000 7f0fc7000640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc006c470 con 0x7f0fc013e360 2024-01-16T15:25:07.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.490+0000 7f0fc7000640 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc006c750 con 0x7f0fc013df80 2024-01-16T15:25:07.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.490+0000 7f0fc7000640 1 -- --> v2:172.21.15.188:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc006ca30 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.491+0000 7f0fbfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0fc013df80 0x7f0fc015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.489 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.491+0000 7f0fbfe00640 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0fc013df80 0x7f0fc015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37134/0 (socket says 172.21.15.141:37134) 2024-01-16T15:25:07.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.491+0000 7f0fbfe00640 1 -- 172.21.15.141:0/4260039457 learned_addr learned my addr 172.21.15.141:0/4260039457 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:25:07.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.491+0000 7f0fbfe00640 1 -- 172.21.15.141:0/4260039457 >> v2:172.21.15.188:3300/0 conn(0x7f0fc00cf9a0 msgr2=0x7f0fc00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:07.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.491+0000 7f0fbfe00640 1 --2- 172.21.15.141:0/4260039457 >> v2:172.21.15.188:3300/0 conn(0x7f0fc00cf9a0 0x7f0fc00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.491+0000 7f0fbfe00640 1 -- 172.21.15.141:0/4260039457 >> v1:172.21.15.188:6789/0 conn(0x7f0fc013e360 legacy=0x7f0fc00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:07.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.491+0000 7f0fbfe00640 1 -- 172.21.15.141:0/4260039457 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc006cd10 con 0x7f0fc013df80 2024-01-16T15:25:07.491 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.492+0000 7f0fbfe00640 1 --2- 172.21.15.141:0/4260039457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc013df80 0x7f0fc015bee0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00ccb10 tx=0x7f0fb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e318b6734a9ba0c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:07.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fbf400640 1 -- 172.21.15.141:0/4260039457 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0031870 con 0x7f0fc013df80 2024-01-16T15:25:07.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fbf400640 1 -- 172.21.15.141:0/4260039457 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0fb00319d0 con 0x7f0fc013df80 2024-01-16T15:25:07.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fbf400640 1 -- 172.21.15.141:0/4260039457 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0031d80 con 0x7f0fc013df80 2024-01-16T15:25:07.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fc7000640 1 -- 172.21.15.141:0/4260039457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc013df80 msgr2=0x7f0fc015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fc7000640 1 --2- 172.21.15.141:0/4260039457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc013df80 0x7f0fc015bee0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00ccb10 tx=0x7f0fb002b910 comp rx=0 tx=0).stop 2024-01-16T15:25:07.492 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fc7000640 1 -- 172.21.15.141:0/4260039457 shutdown_connections 2024-01-16T15:25:07.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fc7000640 1 --2- 172.21.15.141:0/4260039457 >> v2:172.21.15.188:3300/0 conn(0x7f0fc00cf9a0 0x7f0fc00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fc7000640 1 --2- 172.21.15.141:0/4260039457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc013df80 0x7f0fc015bee0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.493+0000 7f0fc7000640 1 -- 172.21.15.141:0/4260039457 >> 172.21.15.141:0/4260039457 conn(0x7f0fc006a800 msgr2=0x7f0fc006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:07.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.494+0000 7f0fc7000640 1 -- 172.21.15.141:0/4260039457 shutdown_connections 2024-01-16T15:25:07.493 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.494+0000 7f0fc7000640 1 -- 172.21.15.141:0/4260039457 wait complete. 2024-01-16T15:25:07.495 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.495+0000 7f0fc7000640 1 Processor -- start 2024-01-16T15:25:07.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.496+0000 7f0fc7000640 1 -- start start 2024-01-16T15:25:07.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fc7000640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc00cf9a0 0x7f0fc0071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fc7000640 1 --2- >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0fc013df80 0x7f0fc013d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fc7000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fc013e360 0x7f0fc015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fc7000640 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00a87c0 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fc7000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00718c0 con 0x7f0fc013e360 2024-01-16T15:25:07.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fc7000640 1 -- --> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0070cb0 con 0x7f0fc013df80 2024-01-16T15:25:07.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc00cf9a0 0x7f0fc0071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc00cf9a0 0x7f0fc0071d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:37136/0 (socket says 172.21.15.141:37136) 2024-01-16T15:25:07.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fbfe00640 1 -- 172.21.15.141:0/961252570 learned_addr learned my addr 172.21.15.141:0/961252570 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-16T15:25:07.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fbfe00640 1 -- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0fc013df80 msgr2=0x7f0fc013d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.497 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fbfe00640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0fc013df80 0x7f0fc013d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fbfe00640 1 -- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fc013e360 msgr2=0x7f0fc015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.497+0000 7f0fbfe00640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fc013e360 0x7f0fc015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.498+0000 7f0fbfe00640 1 -- 172.21.15.141:0/961252570 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00094d0 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.498+0000 7f0fbfe00640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc00cf9a0 0x7f0fc0071d50 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f0fb002b950 tx=0x7f0fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:07.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.498+0000 7f0fbcc00640 1 -- 172.21.15.141:0/961252570 <== mon.0 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00443f0 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.498+0000 7f0fbcc00640 1 -- 172.21.15.141:0/961252570 <== mon.0 v2:172.21.15.141:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0fb0044550 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.498+0000 7f0fbcc00640 1 -- 172.21.15.141:0/961252570 <== mon.0 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0044920 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.498+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc0070f90 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.499 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.498+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc0071550 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.499+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc00675d0 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.500+0000 7f0fbcc00640 1 -- 172.21.15.141:0/961252570 <== mon.0 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0fb004d450 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.502+0000 7f0fbcc00640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0f9806c550 0x7f0f9806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:07.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.502+0000 7f0fbcc00640 1 -- 172.21.15.141:0/961252570 <== mon.0 v2:172.21.15.141:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0fb002d040 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.500 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.502+0000 7f0fb7e00640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0f9806c550 0x7f0f9806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:07.501 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.503+0000 7f0fb7e00640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0f9806c550 0x7f0f9806ea10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f0fc01f6990 tx=0x7f0fa8007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:07.503 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.506+0000 7f0fbcc00640 1 -- 172.21.15.141:0/961252570 <== mon.0 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fb00779d0 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.732 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.735+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fc00b0350 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.734 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.736+0000 7f0fbcc00640 1 -- 172.21.15.141:0/961252570 <== mon.0 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f0fb0035020 con 0x7f0fc00cf9a0 2024-01-16T15:25:07.734 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-16T15:25:07.734 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3065530834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3065530834},{"type":"v1","addr":"172.21.15.188:6835","nonce":3065530834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6140,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/1935621003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1935621003},{"type":"v1","addr":"172.21.15.141:6834","nonce":1935621003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6146,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/935439310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":935439310},{"type":"v1","addr":"172.21.15.188:6837","nonce":935439310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:38.721379+0000","modified":"2024-01-16T15:15:41.817485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1556719184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":1556719184},{"type":"v1","addr":"172.21.15.141:6835","nonce":1556719184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:25:07.735 INFO:teuthology.orchestra.run.smithi141.stderr:dumped fsmap epoch 74 2024-01-16T15:25:07.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0f9806c550 msgr2=0x7f0f9806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0f9806c550 0x7f0f9806ea10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f0fc01f6990 tx=0x7f0fa8007990 comp rx=0 tx=0).stop 2024-01-16T15:25:07.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc00cf9a0 msgr2=0x7f0fc0071d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:07.737 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc00cf9a0 0x7f0fc0071d50 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f0fb002b950 tx=0x7f0fb00046c0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 shutdown_connections 2024-01-16T15:25:07.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:6832/4065338852,v1:172.21.15.188:6833/4065338852] conn(0x7f0f9806c550 0x7f0f9806ea10 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fc013e360 0x7f0fc015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.188:3301/0,v1:172.21.15.188:6790/0] conn(0x7f0fc013df80 0x7f0fc013d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.738 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 --2- 172.21.15.141:0/961252570 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0fc00cf9a0 0x7f0fc0071d50 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:07.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 >> 172.21.15.141:0/961252570 conn(0x7f0fc006a800 msgr2=0x7f0fc006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:07.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 shutdown_connections 2024-01-16T15:25:07.739 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-16T15:25:07.739+0000 7f0fc7000640 1 -- 172.21.15.141:0/961252570 wait complete. 2024-01-16T15:25:07.754 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-16T15:25:07.755 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T15:25:07.755 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:25:07.755 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.777 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:25:07.777 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:25:07.777 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.839 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:25:07.839 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:25:07.840 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.861 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:25:07.862 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:25:07.862 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:07.928 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:25:07.929 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T15:25:07.929 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T15:25:07.929 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:08.023 INFO:tasks.ceph.osd.0:Stopped 2024-01-16T15:25:08.024 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T15:25:08.024 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:08.114 INFO:tasks.ceph.osd.1:Stopped 2024-01-16T15:25:08.114 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T15:25:08.114 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:08.212 INFO:tasks.ceph.osd.2:Stopped 2024-01-16T15:25:08.213 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-16T15:25:08.213 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:10.448 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:10.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:11.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-16T15:25:14.314 INFO:tasks.ceph.osd.3:Stopped 2024-01-16T15:25:14.315 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-16T15:25:14.315 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:17.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-16T15:25:17.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-01-16T15:25:20.416 INFO:tasks.ceph.osd.4:Stopped 2024-01-16T15:25:20.417 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-16T15:25:20.417 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:20.478 INFO:tasks.ceph.osd.5:Stopped 2024-01-16T15:25:20.479 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-16T15:25:20.479 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:20.568 INFO:tasks.ceph.osd.6:Stopped 2024-01-16T15:25:20.569 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-16T15:25:20.569 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:23.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-16T15:25:23.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-01-16T15:25:26.671 INFO:tasks.ceph.osd.7:Stopped 2024-01-16T15:25:26.671 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T15:25:26.671 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-16T15:25:26.671 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:26.760 INFO:tasks.ceph.mgr.y:Stopped 2024-01-16T15:25:26.761 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T15:25:26.761 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:28.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-01-16T15:25:28.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-01-16T15:25:32.862 INFO:tasks.ceph.mgr.x:Stopped 2024-01-16T15:25:32.863 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T15:25:32.864 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T15:25:32.864 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:32.901 INFO:tasks.ceph.mon.a:Stopped 2024-01-16T15:25:32.901 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-16T15:25:32.901 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:32.933 INFO:tasks.ceph.mon.b:Stopped 2024-01-16T15:25:32.934 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-16T15:25:32.934 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:25:33.007 INFO:tasks.ceph.mon.c:Stopped 2024-01-16T15:25:33.007 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T15:25:33.007 DEBUG:teuthology.orchestra.run.smithi141:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-01-16T15:25:33.113 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:25:33.113 DEBUG:teuthology.orchestra.run.smithi141:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-16T15:25:33.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:25:33.402 DEBUG:teuthology.orchestra.run.smithi141:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-16T15:25:33.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-01-16T15:25:33.576 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:25:33.577 DEBUG:teuthology.orchestra.run.smithi141:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-16T15:25:33.751 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:25:33.751 DEBUG:teuthology.orchestra.run.smithi141:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-16T15:25:33.917 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:25:33.917 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-16T15:25:34.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:25:34.156 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-16T15:25:34.311 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:25:34.311 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-16T15:25:34.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:25:34.482 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-16T15:25:34.657 INFO:tasks.ceph:Archiving mon data... 2024-01-16T15:25:34.658 DEBUG:teuthology.misc:Transferring archived files from smithi141:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/data/mon.a.tgz 2024-01-16T15:25:34.659 DEBUG:teuthology.orchestra.run.smithi141:> mktemp 2024-01-16T15:25:34.716 INFO:teuthology.orchestra.run.smithi141.stdout:/tmp/tmp.u4rhUYGQ39 2024-01-16T15:25:34.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.u4rhUYGQ39 2024-01-16T15:25:35.037 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 0666 /tmp/tmp.u4rhUYGQ39 2024-01-16T15:25:35.187 DEBUG:teuthology.orchestra.remote:smithi141:/tmp/tmp.u4rhUYGQ39 is 1MB 2024-01-16T15:25:35.311 DEBUG:teuthology.orchestra.run.smithi141:> rm -fr /tmp/tmp.u4rhUYGQ39 2024-01-16T15:25:35.368 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/data/mon.b.tgz 2024-01-16T15:25:35.368 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-01-16T15:25:35.425 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.vb2s1G7Xs5 2024-01-16T15:25:35.426 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vb2s1G7Xs5 2024-01-16T15:25:35.770 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.vb2s1G7Xs5 2024-01-16T15:25:35.919 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.vb2s1G7Xs5 is 1MB 2024-01-16T15:25:36.039 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.vb2s1G7Xs5 2024-01-16T15:25:36.095 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/data/mon.c.tgz 2024-01-16T15:25:36.096 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-01-16T15:25:36.146 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.el5bdU91qZ 2024-01-16T15:25:36.146 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.el5bdU91qZ 2024-01-16T15:25:36.461 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.el5bdU91qZ 2024-01-16T15:25:36.609 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.el5bdU91qZ is 1MB 2024-01-16T15:25:36.728 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.el5bdU91qZ 2024-01-16T15:25:36.785 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-16T15:25:36.785 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:25:36.793 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:25:36.940 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:25:36.995 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:25:36.995 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T15:25:36.997 DEBUG:teuthology.misc:Transferring archived files from smithi141:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/remote/smithi141/crash 2024-01-16T15:25:36.999 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:25:37.095 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/remote/smithi188/crash 2024-01-16T15:25:37.096 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:25:37.189 INFO:tasks.ceph:Shutting down logrotate 2024-01-16T15:25:37.190 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:37.198 DEBUG:teuthology.orchestra.run.smithi188:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:37.324 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:37.515 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:37.615 INFO:tasks.ceph:Compressing logs... 2024-01-16T15:25:37.615 DEBUG:teuthology.orchestra.run.smithi141:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:25:37.623 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:25:37.711 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-16T15:25:37.712 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60771.log 2024-01-16T15:25:37.712 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36161.log 2024-01-16T15:25:37.713 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56450.log 2024-01-16T15:25:37.714 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44998.log 2024-01-16T15:25:37.715 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58758.log 2024-01-16T15:25:37.715 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60771.log.gz 2024-01-16T15:25:37.716 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61030.log 2024-01-16T15:25:37.716 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56450.log.gz 2024-01-16T15:25:37.717 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.36161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41878.log 2024-01-16T15:25:37.717 INFO:teuthology.orchestra.run.smithi188.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36161.log.gz 2024-01-16T15:25:37.717 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.1.44998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36361.log 2024-01-16T15:25:37.718 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42111.log 2024-01-16T15:25:37.718 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58758.log.gz 2024-01-16T15:25:37.719 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-01-16T15:25:37.719 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61030.log.gz 2024-01-16T15:25:37.720 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.3.41878.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:25:37.720 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.36361.log: 91.4% -- replaced with /var/log/ceph/ceph-client.3.41878.log.gz 2024-01-16T15:25:37.720 INFO:teuthology.orchestra.run.smithi188.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36361.log.gz 2024-01-16T15:25:37.720 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-16T15:25:37.721 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-16T15:25:37.722 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42111.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42111.log.gz 2024-01-16T15:25:37.722 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67815.log 2024-01-16T15:25:37.722 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74343.log 2024-01-16T15:25:37.722 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-01-16T15:25:37.723 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50591.log 2024-01-16T15:25:37.723 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67815.log.gz 2024-01-16T15:25:37.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36281.log 2024-01-16T15:25:37.724 INFO:teuthology.orchestra.run.smithi188.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.1.44998.log.gz 2024-01-16T15:25:37.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43380.log 2024-01-16T15:25:37.725 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75012.log 2024-01-16T15:25:37.725 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74343.log.gz 2024-01-16T15:25:37.725 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47399.log 2024-01-16T15:25:37.725 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50591.log.gz 2024-01-16T15:25:37.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.36281.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.36281.log.gz --verbose 2024-01-16T15:25:37.727 INFO:teuthology.orchestra.run.smithi188.stderr: -- /var/log/ceph/ceph-client.admin.36321.log 2024-01-16T15:25:37.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.1.43380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41358.log 2024-01-16T15:25:37.727 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69679.log 2024-01-16T15:25:37.727 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75012.log.gz 2024-01-16T15:25:37.727 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65680.log 2024-01-16T15:25:37.727 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47399.log.gz 2024-01-16T15:25:37.729 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42320.log 2024-01-16T15:25:37.729 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69679.log.gz 2024-01-16T15:25:37.729 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70133.log 2024-01-16T15:25:37.729 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65680.log.gz 2024-01-16T15:25:37.731 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74691.log 2024-01-16T15:25:37.731 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42320.log.gz 2024-01-16T15:25:37.731 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.36321.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-16T15:25:37.731 INFO:teuthology.orchestra.run.smithi188.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36321.log.gz 2024-01-16T15:25:37.732 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.1.41358.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:25:37.732 INFO:teuthology.orchestra.run.smithi188.stderr: 94.7% -- replaced with /var/log/ceph/ceph-client.1.43380.log.gz 2024-01-16T15:25:37.732 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-16T15:25:37.732 INFO:teuthology.orchestra.run.smithi188.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-16T15:25:37.732 INFO:teuthology.orchestra.run.smithi188.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.1.41358.log.gz 2024-01-16T15:25:37.732 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36401.log 2024-01-16T15:25:37.733 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36241.log 2024-01-16T15:25:37.734 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.36401.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36401.log.gz 2024-01-16T15:25:37.735 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70133.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70133.log.gz 2024-01-16T15:25:37.735 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58350.log 2024-01-16T15:25:37.735 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-01-16T15:25:37.735 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74691.log.gz 2024-01-16T15:25:37.735 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48602.log 2024-01-16T15:25:37.735 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58350.log.gz 2024-01-16T15:25:37.735 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60024.log 2024-01-16T15:25:37.736 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-01-16T15:25:37.737 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-16T15:25:37.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.36241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36441.log 2024-01-16T15:25:37.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mgr.x.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36241.log.gz 2024-01-16T15:25:37.737 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44195.log 2024-01-16T15:25:37.738 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46463.log 2024-01-16T15:25:37.738 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48602.log.gz 2024-01-16T15:25:37.739 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70412.log 2024-01-16T15:25:37.739 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60024.log.gz 2024-01-16T15:25:37.739 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58434.log 2024-01-16T15:25:37.739 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46463.log.gz 2024-01-16T15:25:37.740 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.36441.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36441.log.gz 2024-01-16T15:25:37.740 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46661.log 2024-01-16T15:25:37.741 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.1.44195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47521.log 2024-01-16T15:25:37.741 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81158.log 2024-01-16T15:25:37.741 INFO:teuthology.orchestra.run.smithi141.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70412.log.gz 2024-01-16T15:25:37.741 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68109.log 2024-01-16T15:25:37.742 INFO:teuthology.orchestra.run.smithi141.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58434.log.gz 2024-01-16T15:25:37.742 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81158.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81158.log.gz 2024-01-16T15:25:37.742 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59562.log 2024-01-16T15:25:37.743 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68109.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-16T15:25:37.743 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68109.log.gz 2024-01-16T15:25:37.744 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59562.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.59562.log.gz 2024-01-16T15:25:37.744 INFO:teuthology.orchestra.run.smithi141.stderr: /var/log/ceph/ceph-client.admin.36236.log 2024-01-16T15:25:37.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.1.46661.log: /var/log/ceph/ceph-client.1.47521.log: 94.9% -- replaced with /var/log/ceph/ceph-client.1.44195.log.gz 2024-01-16T15:25:37.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:25:37.745 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45827.log 2024-01-16T15:25:37.746 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.36236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55547.log 2024-01-16T15:25:37.746 INFO:teuthology.orchestra.run.smithi141.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36236.log.gz 2024-01-16T15:25:37.747 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45827.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45827.log.gz 2024-01-16T15:25:37.747 INFO:teuthology.orchestra.run.smithi141.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54253.log 2024-01-16T15:25:37.747 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36201.log 2024-01-16T15:25:37.747 INFO:teuthology.orchestra.run.smithi188.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.1.46661.log.gz 2024-01-16T15:25:37.748 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-16T15:25:37.748 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55547.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55547.log.gz 2024-01-16T15:25:37.748 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48210.log 2024-01-16T15:25:37.749 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54169.log 2024-01-16T15:25:37.750 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54253.log.gz 2024-01-16T15:25:37.751 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48210.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48210.log.gz 2024-01-16T15:25:37.751 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79325.log 2024-01-16T15:25:37.752 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70513.log 2024-01-16T15:25:37.752 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54169.log.gz 2024-01-16T15:25:37.753 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79325.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79325.log.gz 2024-01-16T15:25:37.753 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64685.log 2024-01-16T15:25:37.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.36201.log: 96.7% -- replaced with /var/log/ceph/ceph-client.1.47521.log.gz 2024-01-16T15:25:37.754 INFO:teuthology.orchestra.run.smithi188.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36201.log.gz 2024-01-16T15:25:37.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45914.log 2024-01-16T15:25:37.755 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68408.log 2024-01-16T15:25:37.755 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70513.log.gz 2024-01-16T15:25:37.756 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58098.log 2024-01-16T15:25:37.756 INFO:teuthology.orchestra.run.smithi141.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.64685.log.gz 2024-01-16T15:25:37.756 INFO:teuthology.orchestra.run.smithi188.stderr: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:25:37.757 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.1.45914.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-01-16T15:25:37.758 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68408.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68408.log.gz 2024-01-16T15:25:37.758 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69113.log 2024-01-16T15:25:37.758 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74928.log 2024-01-16T15:25:37.760 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58098.log.gz 2024-01-16T15:25:37.760 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78774.log 2024-01-16T15:25:37.761 INFO:teuthology.orchestra.run.smithi141.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69113.log.gz 2024-01-16T15:25:37.761 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74928.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74928.log.gz 2024-01-16T15:25:37.761 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81923.log 2024-01-16T15:25:37.761 INFO:teuthology.orchestra.run.smithi188.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.1.45914.log.gz 2024-01-16T15:25:37.761 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42607.log 2024-01-16T15:25:37.761 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-16T15:25:37.762 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65512.log 2024-01-16T15:25:37.763 INFO:teuthology.orchestra.run.smithi141.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.78774.log.gz 2024-01-16T15:25:37.763 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81923.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81923.log.gz 2024-01-16T15:25:37.763 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61342.log 2024-01-16T15:25:37.763 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65512.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65512.log.gz 2024-01-16T15:25:37.764 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47958.log 2024-01-16T15:25:37.764 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.1.42607.log: 93.6% -- replaced with /var/log/ceph/ceph-client.1.42607.log.gz 2024-01-16T15:25:37.765 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56690.log 2024-01-16T15:25:37.765 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61342.log.gz 2024-01-16T15:25:37.766 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47958.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47958.log.gz 2024-01-16T15:25:37.766 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73138.log 2024-01-16T15:25:37.767 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72488.log 2024-01-16T15:25:37.767 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56690.log.gz 2024-01-16T15:25:37.768 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50339.log 2024-01-16T15:25:37.768 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73138.log.gz 2024-01-16T15:25:37.769 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81965.log 2024-01-16T15:25:37.769 INFO:teuthology.orchestra.run.smithi141.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72488.log.gz 2024-01-16T15:25:37.770 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80586.log 2024-01-16T15:25:37.770 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50339.log.gz 2024-01-16T15:25:37.772 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43484.log 2024-01-16T15:25:37.772 INFO:teuthology.orchestra.run.smithi141.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81965.log.gz 2024-01-16T15:25:37.772 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80586.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80586.log.gz 2024-01-16T15:25:37.773 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62870.log 2024-01-16T15:25:37.774 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.43484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55892.log 2024-01-16T15:25:37.775 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62870.log: 91.1% -- replaced with /var/log/ceph/ceph-client.0.43484.log.gz 2024-01-16T15:25:37.775 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78715.log 2024-01-16T15:25:37.775 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62870.log.gz 2024-01-16T15:25:37.776 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73686.log 2024-01-16T15:25:37.776 INFO:teuthology.orchestra.run.smithi141.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55892.log.gz 2024-01-16T15:25:37.777 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78715.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78715.log.gz -5 2024-01-16T15:25:37.777 INFO:teuthology.orchestra.run.smithi141.stderr: --verbose -- /var/log/ceph/ceph-client.admin.66857.log 2024-01-16T15:25:37.778 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78550.log 2024-01-16T15:25:37.779 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73686.log.gz 2024-01-16T15:25:37.779 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58038.log 2024-01-16T15:25:37.780 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66857.log.gz 2024-01-16T15:25:37.780 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68623.log 2024-01-16T15:25:37.781 INFO:teuthology.orchestra.run.smithi141.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.78550.log.gz 2024-01-16T15:25:37.782 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59604.log 2024-01-16T15:25:37.782 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58038.log.gz 2024-01-16T15:25:37.783 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68623.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68623.log.gz 2024-01-16T15:25:37.783 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56280.log 2024-01-16T15:25:37.784 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61154.log 2024-01-16T15:25:37.784 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59604.log.gz 2024-01-16T15:25:37.785 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69456.log 2024-01-16T15:25:37.785 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56280.log.gz 2024-01-16T15:25:37.786 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53399.log 2024-01-16T15:25:37.786 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61154.log.gz 2024-01-16T15:25:37.787 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.69456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61436.log 2024-01-16T15:25:37.788 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53399.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53399.log.gz 2024-01-16T15:25:37.789 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63392.log 2024-01-16T15:25:37.790 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61436.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61436.log.gz 2024-01-16T15:25:37.790 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61386.log 2024-01-16T15:25:37.791 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81797.log 2024-01-16T15:25:37.791 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63392.log.gz 2024-01-16T15:25:37.792 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61386.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61386.log.gz 2024-01-16T15:25:37.792 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79917.log 2024-01-16T15:25:37.793 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64768.log 2024-01-16T15:25:37.793 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81797.log.gz 2024-01-16T15:25:37.794 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79917.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79917.log.gz 2024-01-16T15:25:37.794 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40660.log 2024-01-16T15:25:37.795 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52451.log 2024-01-16T15:25:37.796 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64768.log.gz 2024-01-16T15:25:37.796 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40660.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40660.log.gz 2024-01-16T15:25:37.797 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75887.log 2024-01-16T15:25:37.798 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51363.log 2024-01-16T15:25:37.798 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52451.log.gz 2024-01-16T15:25:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75887.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75887.log.gz 2024-01-16T15:25:37.799 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48830.log 2024-01-16T15:25:37.800 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59646.log 2024-01-16T15:25:37.800 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51363.log.gz 2024-01-16T15:25:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48830.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48830.log.gz 2024-01-16T15:25:37.801 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50916.log 2024-01-16T15:25:37.802 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65554.log 2024-01-16T15:25:37.802 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59646.log.gz 2024-01-16T15:25:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50916.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50916.log.gz 2024-01-16T15:25:37.803 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53971.log 2024-01-16T15:25:37.804 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79448.log 2024-01-16T15:25:37.804 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65554.log.gz 2024-01-16T15:25:37.805 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53971.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53971.log.gz 2024-01-16T15:25:37.805 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79653.log 2024-01-16T15:25:37.806 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75054.log 2024-01-16T15:25:37.807 INFO:teuthology.orchestra.run.smithi141.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79448.log.gz 2024-01-16T15:25:37.808 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57508.log 2024-01-16T15:25:37.808 INFO:teuthology.orchestra.run.smithi141.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79653.log.gz 2024-01-16T15:25:37.809 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59982.log 2024-01-16T15:25:37.809 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75054.log.gz 2024-01-16T15:25:37.810 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57508.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57508.log.gz 2024-01-16T15:25:37.810 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42446.log 2024-01-16T15:25:37.811 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44798.log 2024-01-16T15:25:37.811 INFO:teuthology.orchestra.run.smithi141.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.59982.log.gz 2024-01-16T15:25:37.812 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42446.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42446.log.gz 2024-01-16T15:25:37.812 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48770.log 2024-01-16T15:25:37.813 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78591.log 2024-01-16T15:25:37.813 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44798.log.gz 2024-01-16T15:25:37.814 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48770.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48770.log.gz 2024-01-16T15:25:37.814 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76875.log 2024-01-16T15:25:37.815 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46937.log 2024-01-16T15:25:37.815 INFO:teuthology.orchestra.run.smithi141.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.78591.log.gz 2024-01-16T15:25:37.816 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52021.log 2024-01-16T15:25:37.817 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76875.log.gz 2024-01-16T15:25:37.818 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54211.log 2024-01-16T15:25:37.818 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46937.log.gz 2024-01-16T15:25:37.819 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47189.log 2024-01-16T15:25:37.819 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52021.log.gz 2024-01-16T15:25:37.820 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80245.log 2024-01-16T15:25:37.820 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54211.log.gz 2024-01-16T15:25:37.821 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47189.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.45953.log -- replaced with /var/log/ceph/ceph-client.admin.47189.log.gz 2024-01-16T15:25:37.821 INFO:teuthology.orchestra.run.smithi141.stderr: 2024-01-16T15:25:37.822 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69595.log 2024-01-16T15:25:37.822 INFO:teuthology.orchestra.run.smithi141.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80245.log.gz 2024-01-16T15:25:37.823 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75973.log 2024-01-16T15:25:37.823 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45953.log.gz 2024-01-16T15:25:37.825 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56186.log 2024-01-16T15:25:37.825 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69595.log.gz 2024-01-16T15:25:37.826 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49892.log 2024-01-16T15:25:37.826 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75973.log.gz 2024-01-16T15:25:37.827 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67536.log 2024-01-16T15:25:37.827 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56186.log.gz 2024-01-16T15:25:37.828 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2024-01-16T15:25:37.829 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49892.log.gz 2024-01-16T15:25:37.830 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62752.log 2024-01-16T15:25:37.830 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67536.log.gz 2024-01-16T15:25:37.831 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60950.log 2024-01-16T15:25:37.831 INFO:teuthology.orchestra.run.smithi141.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2024-01-16T15:25:37.832 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41451.log 2024-01-16T15:25:37.832 INFO:teuthology.orchestra.run.smithi141.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.62752.log.gz 2024-01-16T15:25:37.833 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42124.log 2024-01-16T15:25:37.833 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60950.log.gz 2024-01-16T15:25:37.834 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77043.log 2024-01-16T15:25:37.834 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41451.log.gz 2024-01-16T15:25:37.835 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74301.log 2024-01-16T15:25:37.835 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42124.log.gz 2024-01-16T15:25:37.836 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44462.log 2024-01-16T15:25:37.836 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77043.log.gz 2024-01-16T15:25:37.837 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42656.log 2024-01-16T15:25:37.837 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74301.log.gz 2024-01-16T15:25:37.838 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47483.log 2024-01-16T15:25:37.839 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44462.log.gz 2024-01-16T15:25:37.840 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68204.log 2024-01-16T15:25:37.840 INFO:teuthology.orchestra.run.smithi141.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42656.log.gz 2024-01-16T15:25:37.841 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47483.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47483.log.gz 2024-01-16T15:25:37.841 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45869.log 2024-01-16T15:25:37.842 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45785.log 2024-01-16T15:25:37.842 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68204.log.gz 2024-01-16T15:25:37.843 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64376.log 2024-01-16T15:25:37.843 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45869.log.gz 2024-01-16T15:25:37.844 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46079.log 2024-01-16T15:25:37.845 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45785.log.gz 2024-01-16T15:25:37.845 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64376.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64376.log.gz 2024-01-16T15:25:37.846 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63518.log 2024-01-16T15:25:37.847 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50381.log 2024-01-16T15:25:37.847 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46079.log.gz 2024-01-16T15:25:37.848 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77700.log 2024-01-16T15:25:37.848 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63518.log.gz 2024-01-16T15:25:37.849 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59856.log 2024-01-16T15:25:37.849 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50381.log.gz 2024-01-16T15:25:37.850 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53441.log 2024-01-16T15:25:37.850 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77700.log.gz 2024-01-16T15:25:37.851 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50213.log 2024-01-16T15:25:37.851 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59856.log.gz 2024-01-16T15:25:37.852 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53441.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.53441.log.gz 2024-01-16T15:25:37.852 INFO:teuthology.orchestra.run.smithi141.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47535.log 2024-01-16T15:25:37.853 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59334.log 2024-01-16T15:25:37.854 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50213.log.gz 2024-01-16T15:25:37.854 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47535.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47535.log.gzgzip 2024-01-16T15:25:37.855 INFO:teuthology.orchestra.run.smithi141.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44630.log 2024-01-16T15:25:37.856 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62710.log 2024-01-16T15:25:37.856 INFO:teuthology.orchestra.run.smithi141.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59334.log.gz 2024-01-16T15:25:37.857 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68801.log 2024-01-16T15:25:37.857 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44630.log.gz 2024-01-16T15:25:37.858 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58392.log 2024-01-16T15:25:37.858 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62710.log.gz 2024-01-16T15:25:37.859 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68801.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68801.log.gz 2024-01-16T15:25:37.859 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50297.log 2024-01-16T15:25:37.860 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71938.log 2024-01-16T15:25:37.860 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58392.log.gz 2024-01-16T15:25:37.861 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50297.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50297.log.gz 2024-01-16T15:25:37.861 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42120.log 2024-01-16T15:25:37.862 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81587.log 2024-01-16T15:25:37.862 INFO:teuthology.orchestra.run.smithi141.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71938.log.gz 2024-01-16T15:25:37.863 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52559.log 2024-01-16T15:25:37.864 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42120.log.gz 2024-01-16T15:25:37.864 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44504.log 2024-01-16T15:25:37.865 INFO:teuthology.orchestra.run.smithi141.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81587.log.gz 2024-01-16T15:25:37.866 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52559.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52559.log.gz 2024-01-16T15:25:37.866 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59200.log 2024-01-16T15:25:37.867 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74241.log 2024-01-16T15:25:37.867 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44504.log.gz 2024-01-16T15:25:37.868 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59200.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59200.log.gz 2024-01-16T15:25:37.868 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56816.log 2024-01-16T15:25:37.869 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55973.log 2024-01-16T15:25:37.869 INFO:teuthology.orchestra.run.smithi141.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74241.log.gz 2024-01-16T15:25:37.870 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56816.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56816.log.gz 2024-01-16T15:25:37.870 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-01-16T15:25:37.871 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54505.log 2024-01-16T15:25:37.871 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55973.log.gz 2024-01-16T15:25:37.872 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82049.log 2024-01-16T15:25:37.873 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-01-16T15:25:37.874 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71369.log 2024-01-16T15:25:37.874 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54505.log.gz 2024-01-16T15:25:37.875 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82049.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.82049.log.gz --verbose 2024-01-16T15:25:37.875 INFO:teuthology.orchestra.run.smithi141.stderr: -- /var/log/ceph/ceph-client.admin.66563.log 2024-01-16T15:25:37.876 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68578.log 2024-01-16T15:25:37.876 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71369.log.gz 2024-01-16T15:25:37.877 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68843.log 2024-01-16T15:25:37.877 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66563.log.gz 2024-01-16T15:25:37.878 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64208.log 2024-01-16T15:25:37.878 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68578.log.gz 2024-01-16T15:25:37.879 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68843.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.68843.log.gz 2024-01-16T15:25:37.879 INFO:teuthology.orchestra.run.smithi141.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77388.log 2024-01-16T15:25:37.880 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60645.log 2024-01-16T15:25:37.880 INFO:teuthology.orchestra.run.smithi141.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64208.log.gz 2024-01-16T15:25:37.881 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77388.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77388.log.gz 2024-01-16T15:25:37.882 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50958.log 2024-01-16T15:25:37.883 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54607.log 2024-01-16T15:25:37.883 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60645.log.gz 2024-01-16T15:25:37.884 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80948.log 2024-01-16T15:25:37.884 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50958.log.gz 2024-01-16T15:25:37.885 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44840.log 2024-01-16T15:25:37.885 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54607.log.gz 2024-01-16T15:25:37.886 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80948.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80948.log.gz 2024-01-16T15:25:37.886 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72110.log 2024-01-16T15:25:37.887 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53844.log 2024-01-16T15:25:37.887 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44840.log.gz 2024-01-16T15:25:37.888 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44756.log 2024-01-16T15:25:37.888 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72110.log.gz 2024-01-16T15:25:37.889 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2024-01-16T15:25:37.889 INFO:teuthology.orchestra.run.smithi141.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53844.log.gz 2024-01-16T15:25:37.890 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44756.log: gzip -5 88.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.44756.log.gz -- 2024-01-16T15:25:37.890 INFO:teuthology.orchestra.run.smithi141.stderr: /var/log/ceph/ceph-client.admin.66731.log 2024-01-16T15:25:37.891 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80345.log 2024-01-16T15:25:37.892 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2024-01-16T15:25:37.893 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58968.log 2024-01-16T15:25:37.893 INFO:teuthology.orchestra.run.smithi141.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66731.log.gz 2024-01-16T15:25:37.894 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68759.log 2024-01-16T15:25:37.894 INFO:teuthology.orchestra.run.smithi141.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.80345.log.gz 2024-01-16T15:25:37.895 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58968.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.58968.log.gz 2024-01-16T15:25:37.895 INFO:teuthology.orchestra.run.smithi141.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50069.log 2024-01-16T15:25:37.896 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77532.log 2024-01-16T15:25:37.896 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68759.log.gz 2024-01-16T15:25:37.897 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71896.log 2024-01-16T15:25:37.897 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50069.log.gz 2024-01-16T15:25:37.898 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64436.log 2024-01-16T15:25:37.898 INFO:teuthology.orchestra.run.smithi141.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77532.log.gz 2024-01-16T15:25:37.899 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71896.log.gz 2024-01-16T15:25:37.899 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29782.log 2024-01-16T15:25:37.901 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49299.log 2024-01-16T15:25:37.901 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64436.log.gz 2024-01-16T15:25:37.902 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph.tmp-client.admin.29782.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29782.log.gz 2024-01-16T15:25:37.902 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70880.log 2024-01-16T15:25:37.903 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.49299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68715.log 2024-01-16T15:25:37.904 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70880.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70880.log.gz 2024-01-16T15:25:37.905 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59520.log 2024-01-16T15:25:37.906 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59052.log 2024-01-16T15:25:37.906 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68715.log.gz 2024-01-16T15:25:37.907 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58476.log 2024-01-16T15:25:37.908 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59520.log.gz 2024-01-16T15:25:37.909 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76099.log 2024-01-16T15:25:37.909 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59052.log.gz 2024-01-16T15:25:37.910 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79225.log 2024-01-16T15:25:37.911 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58476.log.gz 2024-01-16T15:25:37.912 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63746.log 2024-01-16T15:25:37.912 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76099.log.gz 2024-01-16T15:25:37.913 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60519.log 2024-01-16T15:25:37.913 INFO:teuthology.orchestra.run.smithi141.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.79225.log.gz 2024-01-16T15:25:37.914 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76225.log 2024-01-16T15:25:37.916 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63746.log.gz 2024-01-16T15:25:37.916 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73644.log 2024-01-16T15:25:37.916 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60519.log.gz 2024-01-16T15:25:37.917 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69071.log 2024-01-16T15:25:37.917 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76225.log.gz 2024-01-16T15:25:37.918 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58518.log 2024-01-16T15:25:37.919 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73644.log.gz 2024-01-16T15:25:37.920 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49808.log 2024-01-16T15:25:37.920 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69071.log.gz 2024-01-16T15:25:37.922 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76749.log 2024-01-16T15:25:37.922 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58518.log.gz 2024-01-16T15:25:37.922 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53718.log 2024-01-16T15:25:37.923 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49808.log.gz 2024-01-16T15:25:37.924 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79143.log 2024-01-16T15:25:37.924 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76749.log.gz 2024-01-16T15:25:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65638.log 2024-01-16T15:25:37.926 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53718.log.gz 2024-01-16T15:25:37.926 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56605.log 2024-01-16T15:25:37.927 INFO:teuthology.orchestra.run.smithi141.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.79143.log.gz 2024-01-16T15:25:37.928 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56493.log 2024-01-16T15:25:37.929 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65638.log.gz 2024-01-16T15:25:37.930 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76791.log 2024-01-16T15:25:37.930 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56605.log.gz 2024-01-16T15:25:37.930 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44714.log 2024-01-16T15:25:37.931 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56493.log.gz 2024-01-16T15:25:37.933 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79284.log 2024-01-16T15:25:37.933 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76791.log.gz 2024-01-16T15:25:37.934 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67941.log 2024-01-16T15:25:37.934 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44714.log.gz 2024-01-16T15:25:37.935 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74970.log 2024-01-16T15:25:37.935 INFO:teuthology.orchestra.run.smithi141.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79284.log.gz 2024-01-16T15:25:37.936 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54337.log 2024-01-16T15:25:37.937 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67941.log.gz 2024-01-16T15:25:37.938 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81032.log 2024-01-16T15:25:37.938 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74970.log.gz 2024-01-16T15:25:37.939 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64292.log 2024-01-16T15:25:37.939 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54337.log.gz 2024-01-16T15:25:37.940 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50111.log 2024-01-16T15:25:37.941 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81032.log.gz 2024-01-16T15:25:37.942 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52155.log 2024-01-16T15:25:37.942 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64292.log.gz 2024-01-16T15:25:37.943 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54733.log 2024-01-16T15:25:37.943 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50111.log.gz 2024-01-16T15:25:37.944 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47441.log 2024-01-16T15:25:37.945 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52155.log.gz 2024-01-16T15:25:37.946 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67494.log 2024-01-16T15:25:37.946 INFO:teuthology.orchestra.run.smithi141.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54733.log.gz 2024-01-16T15:25:37.947 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51585.log 2024-01-16T15:25:37.948 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47441.log.gz 2024-01-16T15:25:37.948 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68025.log 2024-01-16T15:25:37.949 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67494.log.gz 2024-01-16T15:25:37.950 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61690.log 2024-01-16T15:25:37.950 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51585.log.gz 2024-01-16T15:25:37.951 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68025.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68025.log.gz 2024-01-16T15:25:37.951 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61112.log 2024-01-16T15:25:37.952 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47586.log 2024-01-16T15:25:37.953 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61690.log.gz 2024-01-16T15:25:37.954 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61112.log: gzip -5 --verbose -- 88.9% /var/log/ceph/ceph-client.admin.67983.log 2024-01-16T15:25:37.954 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61112.log.gz 2024-01-16T15:25:37.955 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47586.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47586.log.gz 2024-01-16T15:25:37.955 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68885.log 2024-01-16T15:25:37.956 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63308.log 2024-01-16T15:25:37.957 INFO:teuthology.orchestra.run.smithi141.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67983.log.gz 2024-01-16T15:25:37.957 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68885.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68885.log.gz 2024-01-16T15:25:37.958 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58224.log 2024-01-16T15:25:37.959 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64562.log 2024-01-16T15:25:37.959 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63308.log.gz 2024-01-16T15:25:37.960 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75795.log 2024-01-16T15:25:37.960 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58224.log.gz 2024-01-16T15:25:37.961 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64562.log: gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.82133.log -- replaced with /var/log/ceph/ceph-client.admin.64562.log.gz 2024-01-16T15:25:37.961 INFO:teuthology.orchestra.run.smithi141.stderr: 2024-01-16T15:25:37.963 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60360.log 2024-01-16T15:25:37.963 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75795.log.gz 2024-01-16T15:25:37.964 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82133.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82133.log.gz 2024-01-16T15:25:37.964 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42740.log 2024-01-16T15:25:37.965 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49522.log 2024-01-16T15:25:37.966 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42740.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60360.log.gz 2024-01-16T15:25:37.966 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42740.log.gz 2024-01-16T15:25:37.967 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58182.log 2024-01-16T15:25:37.968 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49522.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49522.log.gz 2024-01-16T15:25:37.968 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63182.log 2024-01-16T15:25:37.969 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46321.log 2024-01-16T15:25:37.970 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63182.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58182.log.gz 2024-01-16T15:25:37.971 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82259.log 2024-01-16T15:25:37.971 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63182.log.gz 2024-01-16T15:25:37.972 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79694.log 2024-01-16T15:25:37.972 INFO:teuthology.orchestra.run.smithi141.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46321.log.gz 2024-01-16T15:25:37.973 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82259.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82259.log.gz 2024-01-16T15:25:37.973 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44378.log 2024-01-16T15:25:37.974 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51891.log 2024-01-16T15:25:37.975 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1%/var/log/ceph/ceph-client.admin.44378.log: -- replaced with /var/log/ceph/ceph-client.admin.79694.log.gz 2024-01-16T15:25:37.976 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44378.log.gz 2024-01-16T15:25:37.976 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69763.log 2024-01-16T15:25:37.977 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63098.log 2024-01-16T15:25:37.977 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51891.log.gz 2024-01-16T15:25:37.978 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69763.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69763.log.gz 2024-01-16T15:25:37.978 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52325.log 2024-01-16T15:25:37.979 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60100.log 2024-01-16T15:25:37.980 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52325.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63098.log.gz 2024-01-16T15:25:37.981 INFO:teuthology.orchestra.run.smithi141.stderr:gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52325.log.gz -5 2024-01-16T15:25:37.981 INFO:teuthology.orchestra.run.smithi141.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46895.log 2024-01-16T15:25:37.982 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60100.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60100.log.gz 2024-01-16T15:25:37.982 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78304.log 2024-01-16T15:25:37.983 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66143.log 2024-01-16T15:25:37.984 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46895.log.gz 2024-01-16T15:25:37.985 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78304.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.78304.log.gz 2024-01-16T15:25:37.985 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66959.log 2024-01-16T15:25:37.986 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57550.log 2024-01-16T15:25:37.986 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66143.log.gz 2024-01-16T15:25:37.987 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66959.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.66959.log.gz 2024-01-16T15:25:37.987 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43052.log 2024-01-16T15:25:37.988 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71000.log 2024-01-16T15:25:37.989 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.43052.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57550.log.gz 2024-01-16T15:25:37.990 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43052.log.gz 2024-01-16T15:25:37.990 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78632.log 2024-01-16T15:25:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71000.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71000.log.gz 2024-01-16T15:25:37.991 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74811.log 2024-01-16T15:25:37.992 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80770.log 2024-01-16T15:25:37.993 INFO:teuthology.orchestra.run.smithi141.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.78632.log.gz 2024-01-16T15:25:37.994 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74811.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.74811.log.gz -5 2024-01-16T15:25:37.994 INFO:teuthology.orchestra.run.smithi141.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53676.log 2024-01-16T15:25:37.995 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80770.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80770.log.gz 2024-01-16T15:25:37.995 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47708.log 2024-01-16T15:25:37.996 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57752.log 2024-01-16T15:25:37.997 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53676.log.gz 2024-01-16T15:25:37.998 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71084.log 2024-01-16T15:25:37.998 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47708.log.gz 2024-01-16T15:25:37.999 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57752.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57752.log.gz 2024-01-16T15:25:37.999 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82217.log 2024-01-16T15:25:38.000 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66395.log 2024-01-16T15:25:38.001 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82217.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71084.log.gz 2024-01-16T15:25:38.002 INFO:teuthology.orchestra.run.smithi141.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82217.log.gz 2024-01-16T15:25:38.002 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60240.log 2024-01-16T15:25:38.003 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74098.log 2024-01-16T15:25:38.003 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66395.log.gz 2024-01-16T15:25:38.004 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60240.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60240.log.gz 2024-01-16T15:25:38.004 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47021.log 2024-01-16T15:25:38.005 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46523.log 2024-01-16T15:25:38.006 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47021.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74098.log.gz 2024-01-16T15:25:38.007 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48168.log 2024-01-16T15:25:38.007 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47021.log.gz 2024-01-16T15:25:38.008 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46523.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46523.log.gz 2024-01-16T15:25:38.008 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60603.log 2024-01-16T15:25:38.009 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62550.log 2024-01-16T15:25:38.010 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48168.log.gz 2024-01-16T15:25:38.011 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60603.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60603.log.gz 2024-01-16T15:25:38.011 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46811.log 2024-01-16T15:25:38.012 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56230.log 2024-01-16T15:25:38.012 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62550.log.gz 2024-01-16T15:25:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46811.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46811.log.gz 2024-01-16T15:25:38.015 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71411.log 2024-01-16T15:25:38.016 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51806.log 2024-01-16T15:25:38.016 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56230.log.gz 2024-01-16T15:25:38.016 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71411.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71411.log.gz 2024-01-16T15:25:38.016 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58140.log 2024-01-16T15:25:38.017 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51806.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51806.log.gz 2024-01-16T15:25:38.017 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53887.log 2024-01-16T15:25:38.018 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55812.log 2024-01-16T15:25:38.019 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53887.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58140.log.gz 2024-01-16T15:25:38.020 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64124.log 2024-01-16T15:25:38.020 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53887.log.gz 2024-01-16T15:25:38.021 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55812.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55812.log.gz 2024-01-16T15:25:38.021 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57424.log 2024-01-16T15:25:38.022 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66101.log 2024-01-16T15:25:38.023 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64124.log.gz 2024-01-16T15:25:38.023 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57424.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57424.log.gz 2024-01-16T15:25:38.023 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43978.log 2024-01-16T15:25:38.024 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51448.log 2024-01-16T15:25:38.025 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66101.log.gz 2024-01-16T15:25:38.026 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.2.43978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35852.log 2024-01-16T15:25:38.027 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51448.log: 91.9% -- replaced with /var/log/ceph/ceph-client.2.43978.log.gz 2024-01-16T15:25:38.027 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79958.log 2024-01-16T15:25:38.028 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51448.log.gz 2024-01-16T15:25:38.028 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.35852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77430.log 2024-01-16T15:25:38.029 INFO:teuthology.orchestra.run.smithi141.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35852.log.gz 2024-01-16T15:25:38.030 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79958.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.79958.log.gz 2024-01-16T15:25:38.030 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80502.log 2024-01-16T15:25:38.031 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67127.log 2024-01-16T15:25:38.031 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77430.log.gz 2024-01-16T15:25:38.033 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80502.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80502.log.gz 2024-01-16T15:25:38.033 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65848.log 2024-01-16T15:25:38.033 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78178.log 2024-01-16T15:25:38.034 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67127.log.gz 2024-01-16T15:25:38.035 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65848.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65848.log.gz 2024-01-16T15:25:38.035 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50549.log 2024-01-16T15:25:38.036 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78178.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78178.log.gz 2024-01-16T15:25:38.037 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67211.log 2024-01-16T15:25:38.037 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48042.log 2024-01-16T15:25:38.038 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50549.log.gz 2024-01-16T15:25:38.039 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67211.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67211.log.gz 2024-01-16T15:25:38.039 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41786.log 2024-01-16T15:25:38.040 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48042.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gz 2024-01-16T15:25:38.040 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78136.log 2024-01-16T15:25:38.041 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67295.log 2024-01-16T15:25:38.042 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41786.log.gz 2024-01-16T15:25:38.043 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78136.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78136.log.gz 2024-01-16T15:25:38.043 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67043.log 2024-01-16T15:25:38.044 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63056.log 2024-01-16T15:25:38.044 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67295.log.gz 2024-01-16T15:25:38.046 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67043.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67043.log.gz 2024-01-16T15:25:38.046 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78468.log 2024-01-16T15:25:38.046 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54295.log 2024-01-16T15:25:38.047 INFO:teuthology.orchestra.run.smithi141.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.63056.log.gz 2024-01-16T15:25:38.048 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78468.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.78468.log.gz 2024-01-16T15:25:38.048 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67899.log 2024-01-16T15:25:38.049 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54295.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54295.log.gz 2024-01-16T15:25:38.049 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66605.log 2024-01-16T15:25:38.050 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72026.log 2024-01-16T15:25:38.051 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67899.log.gz 2024-01-16T15:25:38.052 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66605.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.42782.log 2024-01-16T15:25:38.052 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66605.log.gz 2024-01-16T15:25:38.053 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72026.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72026.log.gz 2024-01-16T15:25:38.053 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58672.log 2024-01-16T15:25:38.054 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42614.log 2024-01-16T15:25:38.054 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42782.log.gz 2024-01-16T15:25:38.055 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58672.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58672.log.gz 2024-01-16T15:25:38.056 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67001.log 2024-01-16T15:25:38.057 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61196.log 2024-01-16T15:25:38.057 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42614.log.gz 2024-01-16T15:25:38.058 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67001.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67001.log.gz 2024-01-16T15:25:38.058 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65182.log 2024-01-16T15:25:38.059 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46379.log 2024-01-16T15:25:38.060 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61196.log.gz 2024-01-16T15:25:38.060 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65182.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65182.log.gz 2024-01-16T15:25:38.061 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63014.log 2024-01-16T15:25:38.062 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73812.log 2024-01-16T15:25:38.062 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46379.log.gz 2024-01-16T15:25:38.063 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63014.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63014.log.gz 2024-01-16T15:25:38.063 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50874.log 2024-01-16T15:25:38.064 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56015.log 2024-01-16T15:25:38.065 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50874.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73812.log.gz 2024-01-16T15:25:38.066 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50874.log.gz 2024-01-16T15:25:38.066 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61648.log 2024-01-16T15:25:38.067 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45134.log 2024-01-16T15:25:38.067 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56015.log.gz 2024-01-16T15:25:38.068 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61648.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61648.log.gz 2024-01-16T15:25:38.068 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64334.log 2024-01-16T15:25:38.069 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69637.log 2024-01-16T15:25:38.070 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45134.log.gz 2024-01-16T15:25:38.071 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59730.log 2024-01-16T15:25:38.071 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64334.log.gz 2024-01-16T15:25:38.072 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45539.log 2024-01-16T15:25:38.072 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69637.log.gz 2024-01-16T15:25:38.073 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59730.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59730.log.gz 2024-01-16T15:25:38.074 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:25:38.074 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2024-01-16T15:25:38.075 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45539.log.gz 2024-01-16T15:25:38.076 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57592.log 2024-01-16T15:25:38.077 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46153.log 2024-01-16T15:25:38.077 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2024-01-16T15:25:38.078 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60729.log 2024-01-16T15:25:38.079 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57592.log.gz 2024-01-16T15:25:38.079 INFO:teuthology.orchestra.run.smithi141.stderr: 89.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:25:38.080 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73927.log 2024-01-16T15:25:38.080 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46153.log.gz 2024-01-16T15:25:38.081 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60729.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.60729.log.gz /var/log/ceph/ceph-client.admin.51120.log 2024-01-16T15:25:38.081 INFO:teuthology.orchestra.run.smithi141.stderr: 2024-01-16T15:25:38.082 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73927.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73927.log.gz 2024-01-16T15:25:38.083 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61564.log 2024-01-16T15:25:38.084 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44142.log 2024-01-16T15:25:38.084 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61564.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51120.log.gz 2024-01-16T15:25:38.085 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61564.log.gz 2024-01-16T15:25:38.085 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45355.log 2024-01-16T15:25:38.086 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42908.log 2024-01-16T15:25:38.087 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44142.log.gz 2024-01-16T15:25:38.088 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45355.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45355.log.gz 2024-01-16T15:25:38.088 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66773.log 2024-01-16T15:25:38.089 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78345.log 2024-01-16T15:25:38.089 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66773.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42908.log.gz 2024-01-16T15:25:38.090 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66773.log.gz 2024-01-16T15:25:38.090 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53634.log 2024-01-16T15:25:38.091 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70740.log 2024-01-16T15:25:38.092 INFO:teuthology.orchestra.run.smithi141.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78345.log.gz 2024-01-16T15:25:38.093 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53634.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53634.log.gz 2024-01-16T15:25:38.093 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60198.log 2024-01-16T15:25:38.094 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42119.log 2024-01-16T15:25:38.094 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70740.log.gz 2024-01-16T15:25:38.095 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60198.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60198.log.gz 2024-01-16T15:25:38.095 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40611.log 2024-01-16T15:25:38.096 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42119.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42119.log.gz 2024-01-16T15:25:38.096 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50171.log 2024-01-16T15:25:38.097 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77085.log 2024-01-16T15:25:38.098 INFO:teuthology.orchestra.run.smithi141.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40611.log.gz 2024-01-16T15:25:38.099 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50171.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50171.log.gz 2024-01-16T15:25:38.099 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70370.log 2024-01-16T15:25:38.100 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77085.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77085.log.gz 2024-01-16T15:25:38.100 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74853.log 2024-01-16T15:25:38.101 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2024-01-16T15:25:38.102 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70370.log.gz 2024-01-16T15:25:38.103 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74853.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74853.log.gz 2024-01-16T15:25:38.103 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76917.log 2024-01-16T15:25:38.104 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45995.log 2024-01-16T15:25:38.104 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2024-01-16T15:25:38.105 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76917.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76917.log.gz 2024-01-16T15:25:38.105 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68534.log 2024-01-16T15:25:38.106 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54421.log 2024-01-16T15:25:38.107 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45995.log.gz 2024-01-16T15:25:38.108 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68534.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68534.log.gz 2024-01-16T15:25:38.108 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69889.log 2024-01-16T15:25:38.109 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72404.log 2024-01-16T15:25:38.109 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54421.log.gz 2024-01-16T15:25:38.110 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69889.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69889.log.gz 2024-01-16T15:25:38.111 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54547.log 2024-01-16T15:25:38.111 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62382.log 2024-01-16T15:25:38.112 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72404.log.gz 2024-01-16T15:25:38.113 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54547.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54547.log.gz 2024-01-16T15:25:38.113 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57996.log 2024-01-16T15:25:38.114 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62382.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62382.log.gz 2024-01-16T15:25:38.114 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74528.log 2024-01-16T15:25:38.115 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75096.log 2024-01-16T15:25:38.116 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57996.log.gz 2024-01-16T15:25:38.117 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77574.log 2024-01-16T15:25:38.117 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74528.log.gz 2024-01-16T15:25:38.118 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75096.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75096.log.gz 2024-01-16T15:25:38.118 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82422.log 2024-01-16T15:25:38.119 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67578.log 2024-01-16T15:25:38.119 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77574.log.gz 2024-01-16T15:25:38.120 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53483.log 2024-01-16T15:25:38.121 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82422.log.gz 2024-01-16T15:25:38.122 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67578.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67578.log.gz 2024-01-16T15:25:38.122 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81242.log 2024-01-16T15:25:38.123 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64726.log 2024-01-16T15:25:38.123 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81242.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53483.log.gz 2024-01-16T15:25:38.124 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44882.log 2024-01-16T15:25:38.124 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81242.log.gz 2024-01-16T15:25:38.125 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61478.log 2024-01-16T15:25:38.126 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64726.log.gz 2024-01-16T15:25:38.127 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44882.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44882.log.gz 2024-01-16T15:25:38.127 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55058.log 2024-01-16T15:25:38.128 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71327.log 2024-01-16T15:25:38.128 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55058.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61478.log.gz 2024-01-16T15:25:38.129 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59814.log 2024-01-16T15:25:38.129 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55058.log.gz 2024-01-16T15:25:38.130 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71327.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71327.log.gz 2024-01-16T15:25:38.130 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42118.log 2024-01-16T15:25:38.132 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42362.log 2024-01-16T15:25:38.132 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42118.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59814.log.gz 2024-01-16T15:25:38.133 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42118.log.gz 2024-01-16T15:25:38.133 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50465.log 2024-01-16T15:25:38.134 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42362.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42362.log.gz 2024-01-16T15:25:38.134 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46685.log 2024-01-16T15:25:38.135 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42530.log 2024-01-16T15:25:38.136 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46685.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50465.log.gz 2024-01-16T15:25:38.137 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46685.log.gz 2024-01-16T15:25:38.137 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51279.log 2024-01-16T15:25:38.138 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70226.log 2024-01-16T15:25:38.138 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42530.log.gz 2024-01-16T15:25:38.139 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51279.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51279.log.gz 2024-01-16T15:25:38.139 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47273.log 2024-01-16T15:25:38.140 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44546.log 2024-01-16T15:25:38.141 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47273.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70226.log.gz 2024-01-16T15:25:38.142 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81713.log 2024-01-16T15:25:38.142 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47273.log.gz 2024-01-16T15:25:38.143 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44546.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44546.log.gz 2024-01-16T15:25:38.143 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64250.log 2024-01-16T15:25:38.144 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81116.log 2024-01-16T15:25:38.145 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81713.log.gz 2024-01-16T15:25:38.146 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40439.log 2024-01-16T15:25:38.146 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64250.log.gz 2024-01-16T15:25:38.147 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79366.log 2024-01-16T15:25:38.147 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81116.log.gz 2024-01-16T15:25:38.148 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40439.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-01-16T15:25:38.148 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2024-01-16T15:25:38.149 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67698.log 2024-01-16T15:25:38.150 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41788.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.79366.log.gz 2024-01-16T15:25:38.150 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz 2024-01-16T15:25:38.151 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81284.log 2024-01-16T15:25:38.152 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67698.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67698.log.gz 2024-01-16T15:25:38.152 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58266.log 2024-01-16T15:25:38.153 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50507.log 2024-01-16T15:25:38.154 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81284.log.gz 2024-01-16T15:25:38.155 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58266.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58266.log.gz 2024-01-16T15:25:38.155 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67396.log 2024-01-16T15:25:38.156 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50507.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50507.log.gz 2024-01-16T15:25:38.156 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47357.log 2024-01-16T15:25:38.157 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81074.log 2024-01-16T15:25:38.158 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67396.log.gz 2024-01-16T15:25:38.159 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41186.log 2024-01-16T15:25:38.159 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47357.log.gz 2024-01-16T15:25:38.160 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61732.log 2024-01-16T15:25:38.160 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81074.log.gz 2024-01-16T15:25:38.162 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41186.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41186.log.gz 2024-01-16T15:25:38.162 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49564.log 2024-01-16T15:25:38.162 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53929.log 2024-01-16T15:25:38.163 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61732.log.gz 2024-01-16T15:25:38.164 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49564.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49564.log.gz 2024-01-16T15:25:38.164 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75180.log 2024-01-16T15:25:38.165 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65140.log 2024-01-16T15:25:38.165 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53929.log.gz 2024-01-16T15:25:38.166 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75180.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75180.log.gz 2024-01-16T15:25:38.166 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54901.log 2024-01-16T15:25:38.167 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42115.log 2024-01-16T15:25:38.168 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65140.log.gz 2024-01-16T15:25:38.169 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54901.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.54901.log.gz 2024-01-16T15:25:38.169 INFO:teuthology.orchestra.run.smithi141.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.74199.log 2024-01-16T15:25:38.170 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42115.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42115.log.gz 2024-01-16T15:25:38.170 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63830.log 2024-01-16T15:25:38.171 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48390.log 2024-01-16T15:25:38.172 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74199.log.gz 2024-01-16T15:25:38.173 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63830.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63830.log.gz 2024-01-16T15:25:38.173 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68673.log 2024-01-16T15:25:38.174 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42950.log 2024-01-16T15:25:38.174 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48390.log.gz 2024-01-16T15:25:38.175 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68673.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68673.log.gz 2024-01-16T15:25:38.176 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54691.log 2024-01-16T15:25:38.176 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45050.log 2024-01-16T15:25:38.177 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54691.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42950.log.gz 2024-01-16T15:25:38.178 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.54691.log.gz -5 2024-01-16T15:25:38.178 INFO:teuthology.orchestra.run.smithi141.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59772.log 2024-01-16T15:25:38.179 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45050.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45050.log.gz 2024-01-16T15:25:38.179 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49022.log 2024-01-16T15:25:38.180 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60402.log 2024-01-16T15:25:38.181 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59772.log.gz 2024-01-16T15:25:38.182 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57382.log 89.0% 2024-01-16T15:25:38.182 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49022.log.gz 2024-01-16T15:25:38.183 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81449.log 2024-01-16T15:25:38.183 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60402.log.gz 2024-01-16T15:25:38.184 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57382.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57382.log.gz 2024-01-16T15:25:38.184 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2024-01-16T15:25:38.185 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58602.log 2024-01-16T15:25:38.186 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81449.log.gz 2024-01-16T15:25:38.187 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47063.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.42404.log 2024-01-16T15:25:38.187 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2024-01-16T15:25:38.188 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62912.log 2024-01-16T15:25:38.188 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58602.log.gz 2024-01-16T15:25:38.189 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42404.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42404.log.gz 2024-01-16T15:25:38.189 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69805.log 2024-01-16T15:25:38.190 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72194.log 2024-01-16T15:25:38.191 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62912.log.gz 2024-01-16T15:25:38.192 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69805.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69805.log.gz 2024-01-16T15:25:38.192 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77959.log 2024-01-16T15:25:38.193 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71453.log 2024-01-16T15:25:38.193 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72194.log.gz 2024-01-16T15:25:38.194 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77959.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77959.log.gz 2024-01-16T15:25:38.194 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73096.log 2024-01-16T15:25:38.195 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65368.log 2024-01-16T15:25:38.196 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71453.log.gz 2024-01-16T15:25:38.197 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73096.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73096.log.gz 2024-01-16T15:25:38.197 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60282.log 2024-01-16T15:25:38.198 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65368.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65368.log.gz 2024-01-16T15:25:38.198 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-16T15:25:38.199 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55379.log 2024-01-16T15:25:38.200 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-osd.1.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60282.log.gz 2024-01-16T15:25:38.200 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61924.log 2024-01-16T15:25:38.202 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53143.log 2024-01-16T15:25:38.202 INFO:teuthology.orchestra.run.smithi141.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55379.log.gz 2024-01-16T15:25:38.203 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52493.log 2024-01-16T15:25:38.203 INFO:teuthology.orchestra.run.smithi141.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61924.log.gz 2024-01-16T15:25:38.204 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-01-16T15:25:38.205 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53143.log.gz 2024-01-16T15:25:38.207 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78019.log 2024-01-16T15:25:38.207 INFO:teuthology.orchestra.run.smithi141.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52493.log.gz 2024-01-16T15:25:38.207 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54817.log 2024-01-16T15:25:38.207 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gz 2024-01-16T15:25:38.209 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62201.log 2024-01-16T15:25:38.209 INFO:teuthology.orchestra.run.smithi141.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.78019.log.gz 2024-01-16T15:25:38.210 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74649.log 2024-01-16T15:25:38.210 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54817.log.gz 2024-01-16T15:25:38.211 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.62201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44588.log 2024-01-16T15:25:38.213 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52409.log 2024-01-16T15:25:38.213 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74649.log.gz 2024-01-16T15:25:38.214 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60444.log 2024-01-16T15:25:38.214 INFO:teuthology.orchestra.run.smithi141.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44588.log.gz 2024-01-16T15:25:38.215 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52409.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52409.log.gz 2024-01-16T15:25:38.216 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58842.log 2024-01-16T15:25:38.217 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79184.log 2024-01-16T15:25:38.217 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60444.log.gz 2024-01-16T15:25:38.219 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82504.log 2024-01-16T15:25:38.219 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58842.log.gz 2024-01-16T15:25:38.220 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79184.log: 85.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61520.log 2024-01-16T15:25:38.220 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79184.log.gz 2024-01-16T15:25:38.221 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56900.log 2024-01-16T15:25:38.222 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82504.log.gz 2024-01-16T15:25:38.223 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44420.log 2024-01-16T15:25:38.223 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61520.log.gz 2024-01-16T15:25:38.224 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74469.log 2024-01-16T15:25:38.225 INFO:teuthology.orchestra.run.smithi141.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56900.log.gz 2024-01-16T15:25:38.225 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44420.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44420.log.gz 2024-01-16T15:25:38.226 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57794.log 2024-01-16T15:25:38.227 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63350.log 2024-01-16T15:25:38.227 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74469.log.gz 2024-01-16T15:25:38.228 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52367.log 2024-01-16T15:25:38.229 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57794.log.gz 2024-01-16T15:25:38.230 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63350.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63350.log.gz 2024-01-16T15:25:38.230 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75733.log 2024-01-16T15:25:38.231 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52367.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52367.log.gz 2024-01-16T15:25:38.232 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81200.log 2024-01-16T15:25:38.233 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68366.log 2024-01-16T15:25:38.233 INFO:teuthology.orchestra.run.smithi141.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.75733.log.gz 2024-01-16T15:25:38.235 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81200.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81200.log.gz 2024-01-16T15:25:38.235 INFO:teuthology.orchestra.run.smithi141.stderr: /var/log/ceph/ceph-client.admin.71854.log 2024-01-16T15:25:38.236 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68366.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68366.log.gz 2024-01-16T15:25:38.236 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72152.log 2024-01-16T15:25:38.237 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70268.log 2024-01-16T15:25:38.238 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71854.log.gz 2024-01-16T15:25:38.239 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42866.log 2024-01-16T15:25:38.239 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72152.log.gz 2024-01-16T15:25:38.240 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49648.log 2024-01-16T15:25:38.240 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70268.log.gz 2024-01-16T15:25:38.241 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42866.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42866.log.gz 2024-01-16T15:25:38.242 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81839.log 2024-01-16T15:25:38.243 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41018.log 2024-01-16T15:25:38.243 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49648.log.gz 2024-01-16T15:25:38.245 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46037.log 2024-01-16T15:25:38.245 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81839.log.gz 2024-01-16T15:25:38.246 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41018.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41018.log.gz 2024-01-16T15:25:38.247 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77245.log 2024-01-16T15:25:38.247 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46037.log.gz 2024-01-16T15:25:38.248 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45497.log 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66227.log 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77245.log.gz 2024-01-16T15:25:38.251 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45497.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45497.log.gz 2024-01-16T15:25:38.252 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71042.log 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66227.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66227.log.gz 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66437.log 2024-01-16T15:25:38.254 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52241.log 2024-01-16T15:25:38.254 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71042.log.gz 2024-01-16T15:25:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66437.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66437.log.gz 2024-01-16T15:25:38.256 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54013.log 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52241.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52241.log.gz 2024-01-16T15:25:38.258 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51036.log 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46769.log 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54013.log.gz 2024-01-16T15:25:38.261 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51036.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51036.log.gz 2024-01-16T15:25:38.261 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57466.log 2024-01-16T15:25:38.262 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77760.log 2024-01-16T15:25:38.263 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46769.log.gz 2024-01-16T15:25:38.265 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57466.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57466.log.gz 2024-01-16T15:25:38.265 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78938.log/var/log/ceph/ceph-client.admin.77760.log: 2024-01-16T15:25:38.265 INFO:teuthology.orchestra.run.smithi141.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77760.log.gz 2024-01-16T15:25:38.266 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73012.log 2024-01-16T15:25:38.267 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46853.log 2024-01-16T15:25:38.268 INFO:teuthology.orchestra.run.smithi141.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.78938.log.gz 2024-01-16T15:25:38.269 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73012.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73012.log.gz 2024-01-16T15:25:38.270 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46853.log: -- /var/log/ceph/ceph-client.admin.56322.log 2024-01-16T15:25:38.270 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46853.log.gz 2024-01-16T15:25:38.271 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80204.log 2024-01-16T15:25:38.273 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46421.log 2024-01-16T15:25:38.273 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56322.log.gz 2024-01-16T15:25:38.274 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80204.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.80204.log.gz 2024-01-16T15:25:38.275 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.46421.log: /var/log/ceph/ceph-client.admin.48298.log 2024-01-16T15:25:38.275 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46421.log.gz 2024-01-16T15:25:38.276 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65722.log 2024-01-16T15:25:38.278 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70310.log 2024-01-16T15:25:38.278 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48298.log.gz 2024-01-16T15:25:38.279 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82320.log 2024-01-16T15:25:38.279 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65722.log.gz 2024-01-16T15:25:38.280 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74157.log 2024-01-16T15:25:38.281 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70310.log.gz 2024-01-16T15:25:38.282 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82320.log.gz 2024-01-16T15:25:38.282 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-01-16T15:25:38.283 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74157.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74157.log.gz 2024-01-16T15:25:38.284 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46279.log 2024-01-16T15:25:38.285 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80990.log 2024-01-16T15:25:38.286 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-01-16T15:25:38.286 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46279.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46279.log.gz 2024-01-16T15:25:38.287 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67857.log 2024-01-16T15:25:38.288 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-01-16T15:25:38.288 INFO:teuthology.orchestra.run.smithi141.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80990.log.gz 2024-01-16T15:25:38.290 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48348.log 2024-01-16T15:25:38.290 INFO:teuthology.orchestra.run.smithi141.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67857.log.gz 2024-01-16T15:25:38.291 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68987.log 2024-01-16T15:25:38.291 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-01-16T15:25:38.293 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48348.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48348.log.gz 2024-01-16T15:25:38.293 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46565.log 2024-01-16T15:25:38.294 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68987.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68987.log.gz 2024-01-16T15:25:38.294 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55220.log 2024-01-16T15:25:38.296 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69847.log 2024-01-16T15:25:38.296 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46565.log.gz 2024-01-16T15:25:38.297 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72831.log 2024-01-16T15:25:38.297 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55220.log.gz 2024-01-16T15:25:38.298 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81671.log 2024-01-16T15:25:38.298 INFO:teuthology.orchestra.run.smithi141.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.69847.log.gz 2024-01-16T15:25:38.301 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.72831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78509.log 2024-01-16T15:25:38.301 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81671.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81671.log.gz 2024-01-16T15:25:38.302 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59418.log 2024-01-16T15:25:38.303 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78509.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.78509.log.gz 2024-01-16T15:25:38.304 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66521.log 2024-01-16T15:25:38.305 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59418.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59418.log.gz 2024-01-16T15:25:38.306 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54960.log/var/log/ceph/ceph-client.admin.66521.log: 2024-01-16T15:25:38.307 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66521.log.gz 2024-01-16T15:25:38.308 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69029.log 2024-01-16T15:25:38.309 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54960.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54960.log.gz 2024-01-16T15:25:38.310 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62954.log/var/log/ceph/ceph-client.admin.69029.log: 2024-01-16T15:25:38.311 INFO:teuthology.orchestra.run.smithi141.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69029.log.gz 2024-01-16T15:25:38.312 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65098.log 2024-01-16T15:25:38.313 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62954.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62954.log.gz 2024-01-16T15:25:38.314 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63914.log/var/log/ceph/ceph-client.admin.65098.log: 2024-01-16T15:25:38.315 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65098.log.gz 2024-01-16T15:25:38.316 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2024-01-16T15:25:38.317 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63914.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63914.log.gz 2024-01-16T15:25:38.318 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52113.log 2024-01-16T15:25:38.319 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40702.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2024-01-16T15:25:38.320 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74769.log 2024-01-16T15:25:38.321 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52113.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52113.log.gz 2024-01-16T15:25:38.322 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56142.log 2024-01-16T15:25:38.323 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74769.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74769.log.gz 2024-01-16T15:25:38.323 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79020.log 2024-01-16T15:25:38.325 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56142.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56142.log.gz 2024-01-16T15:25:38.326 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.79020.log: -- /var/log/ceph/ceph-client.admin.80686.log 2024-01-16T15:25:38.326 INFO:teuthology.orchestra.run.smithi141.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.79020.log.gz 2024-01-16T15:25:38.327 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52836.log 2024-01-16T15:25:38.328 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80686.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80686.log.gz 2024-01-16T15:25:38.329 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40358.log 2024-01-16T15:25:38.332 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.52836.log: /var/log/ceph/ceph-client.admin.40358.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40358.log.gz 2024-01-16T15:25:38.333 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76833.log 2024-01-16T15:25:38.335 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41228.log 2024-01-16T15:25:38.337 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76833.log.gz 2024-01-16T15:25:38.338 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75222.log 2024-01-16T15:25:38.340 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41228.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41228.log.gz 2024-01-16T15:25:38.343 INFO:teuthology.orchestra.run.smithi141.stderr:gzip/var/log/ceph/ceph-client.admin.75222.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49438.log 2024-01-16T15:25:38.344 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75222.log.gz 2024-01-16T15:25:38.346 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81492.log 2024-01-16T15:25:38.350 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49438.log: /var/log/ceph/ceph-client.admin.81492.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67253.log 2024-01-16T15:25:38.351 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49438.log.gz 2024-01-16T15:25:38.352 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81492.log.gz 2024-01-16T15:25:38.352 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73222.log 2024-01-16T15:25:38.355 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67253.log: /var/log/ceph/ceph-client.admin.73222.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73222.log.gz 2024-01-16T15:25:38.356 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73180.log 2024-01-16T15:25:38.357 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67253.log.gz 2024-01-16T15:25:38.359 INFO:teuthology.orchestra.run.smithi141.stderr:gzip/var/log/ceph/ceph-client.admin.73180.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63560.log 2024-01-16T15:25:38.360 INFO:teuthology.orchestra.run.smithi141.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73180.log.gz 2024-01-16T15:25:38.361 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77490.log 2024-01-16T15:25:38.366 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63560.log: /var/log/ceph/ceph-client.admin.77490.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61298.log 2024-01-16T15:25:38.366 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63560.log.gz 2024-01-16T15:25:38.367 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77490.log.gz 2024-01-16T15:25:38.368 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45229.log 2024-01-16T15:25:38.373 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61298.log: /var/log/ceph/ceph-client.admin.45229.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62340.log 2024-01-16T15:25:38.373 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61298.log.gz 2024-01-16T15:25:38.374 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mon.c.log: 93.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-16T15:25:38.375 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45229.log.gz 2024-01-16T15:25:38.375 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80040.log 2024-01-16T15:25:38.379 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62340.log: /var/log/ceph/ceph-client.admin.80040.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.62340.log.gz 2024-01-16T15:25:38.380 INFO:teuthology.orchestra.run.smithi141.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41784.log 2024-01-16T15:25:38.382 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41784.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.80040.log.gz 2024-01-16T15:25:38.383 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66689.log 2024-01-16T15:25:38.385 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41447.log 2024-01-16T15:25:38.388 INFO:teuthology.orchestra.run.smithi141.stderr: 87.9%/var/log/ceph/ceph-client.admin.66689.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66689.log.gz 2024-01-16T15:25:38.388 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41784.log.gz 2024-01-16T15:25:38.389 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56100.log 2024-01-16T15:25:38.393 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41447.log: /var/log/ceph/ceph-client.admin.56100.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75563.log 2024-01-16T15:25:38.394 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41447.log.gz 2024-01-16T15:25:38.395 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56100.log.gz 2024-01-16T15:25:38.397 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68450.log 2024-01-16T15:25:38.401 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75563.log: /var/log/ceph/ceph-client.admin.68450.log: 87.9%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.75563.log.gz --verbose -- 2024-01-16T15:25:38.401 INFO:teuthology.orchestra.run.smithi141.stderr: /var/log/ceph/ceph-client.admin.79735.log 2024-01-16T15:25:38.403 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68450.log.gz 2024-01-16T15:25:38.404 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79735.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.79735.log.gz 2024-01-16T15:25:38.405 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55262.log 2024-01-16T15:25:38.407 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47916.log 2024-01-16T15:25:38.409 INFO:teuthology.orchestra.run.smithi141.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55262.log.gz 2024-01-16T15:25:38.409 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56774.log 2024-01-16T15:25:38.412 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47916.log: 88.4%/var/log/ceph/ceph-client.admin.56774.log: -- replaced with /var/log/ceph/ceph-client.admin.47916.log.gz 2024-01-16T15:25:38.413 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56774.log.gz 2024-01-16T15:25:38.413 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80081.log 2024-01-16T15:25:38.416 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60687.log 2024-01-16T15:25:38.416 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80081.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.80081.log.gz 2024-01-16T15:25:38.417 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45313.log 2024-01-16T15:25:38.419 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63662.log 2024-01-16T15:25:38.420 INFO:teuthology.orchestra.run.smithi141.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60687.log.gz 2024-01-16T15:25:38.421 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45313.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-16T15:25:38.423 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1%/var/log/ceph/ceph-client.admin.63662.log: -- replaced with /var/log/ceph/ceph-client.admin.45313.log.gz 2024-01-16T15:25:38.424 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-osd.3.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63662.log.gz 2024-01-16T15:25:38.426 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63434.log 2024-01-16T15:25:38.429 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76959.log 2024-01-16T15:25:38.431 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64040.log 2024-01-16T15:25:38.432 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76959.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63434.log.gz 2024-01-16T15:25:38.434 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63998.log 2024-01-16T15:25:38.435 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64040.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76959.log.gz 2024-01-16T15:25:38.437 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64040.log.gz 2024-01-16T15:25:38.438 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52975.log 2024-01-16T15:25:38.443 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53357.log 2024-01-16T15:25:38.443 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63998.log.gz 2024-01-16T15:25:38.444 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52975.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52975.log.gz 2024-01-16T15:25:38.446 INFO:teuthology.orchestra.run.smithi188.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-16T15:25:38.448 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-01-16T15:25:38.449 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53357.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53357.log.gz 2024-01-16T15:25:38.453 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42488.log 2024-01-16T15:25:38.455 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41144.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-01-16T15:25:38.457 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2024-01-16T15:25:38.458 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42488.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42488.log.gz 2024-01-16T15:25:38.459 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44672.log 2024-01-16T15:25:38.461 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48000.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2024-01-16T15:25:38.465 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50717.log 2024-01-16T15:25:38.466 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44672.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44672.log.gz 2024-01-16T15:25:38.468 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75845.log 2024-01-16T15:25:38.469 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50717.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50717.log.gz 2024-01-16T15:25:38.475 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55463.log 2024-01-16T15:25:38.475 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75845.log.gz 2024-01-16T15:25:38.476 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55142.log 2024-01-16T15:25:38.478 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55463.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55463.log.gz 2024-01-16T15:25:38.484 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55142.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55142.log.gz 2024-01-16T15:25:38.485 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79571.log 2024-01-16T15:25:38.489 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81539.log 2024-01-16T15:25:38.490 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79571.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79571.log.gz 2024-01-16T15:25:38.492 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48254.log 2024-01-16T15:25:38.493 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81539.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81539.log.gz 2024-01-16T15:25:38.495 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47830.log 2024-01-16T15:25:38.496 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48254.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48254.log.gz 2024-01-16T15:25:38.499 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72970.log 2024-01-16T15:25:38.500 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47830.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47830.log.gz 2024-01-16T15:25:38.503 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72970.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72970.log.gz 2024-01-16T15:25:38.505 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70091.log 2024-01-16T15:25:38.509 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66059.log 2024-01-16T15:25:38.510 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70091.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.70091.log.gz 2024-01-16T15:25:38.512 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71811.log 2024-01-16T15:25:38.513 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66059.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66059.log.gz 2024-01-16T15:25:38.516 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66647.log 2024-01-16T15:25:38.519 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54379.log 2024-01-16T15:25:38.519 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71811.log.gz 2024-01-16T15:25:38.520 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66647.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66647.log.gz 2024-01-16T15:25:38.524 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58926.log 2024-01-16T15:25:38.525 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54379.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54379.log.gz 2024-01-16T15:25:38.528 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50423.log 2024-01-16T15:25:38.529 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58926.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.58926.log.gz 2024-01-16T15:25:38.532 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50423.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50423.log.gz 2024-01-16T15:25:38.533 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56648.log 2024-01-16T15:25:38.537 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79999.log 2024-01-16T15:25:38.538 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56648.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56648.log.gz 2024-01-16T15:25:38.541 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81407.log 2024-01-16T15:25:38.542 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79999.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.79999.log.gz 2024-01-16T15:25:38.543 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78979.log 2024-01-16T15:25:38.544 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81407.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81407.log.gz 2024-01-16T15:25:38.548 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78979.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78979.log.gz 2024-01-16T15:25:38.549 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65242.log 2024-01-16T15:25:38.553 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80460.log 2024-01-16T15:25:38.554 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65242.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65242.log.gz 2024-01-16T15:25:38.556 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54091.log 2024-01-16T15:25:38.557 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80460.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80460.log.gz 2024-01-16T15:25:38.559 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47315.log 2024-01-16T15:25:38.560 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54091.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54091.log.gz 2024-01-16T15:25:38.562 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57878.log 2024-01-16T15:25:38.563 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47315.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47315.log.gz 2024-01-16T15:25:38.565 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60561.log 2024-01-16T15:25:38.565 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57878.log.gz 2024-01-16T15:25:38.568 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64954.log 2024-01-16T15:25:38.572 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60561.log: /var/log/ceph/ceph-client.admin.64954.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64954.log.gz 2024-01-16T15:25:38.573 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74427.log 2024-01-16T15:25:38.573 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60561.log.gz 2024-01-16T15:25:38.577 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67169.log 2024-01-16T15:25:38.578 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74427.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74427.log.gz 2024-01-16T15:25:38.582 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47788.log 2024-01-16T15:25:38.583 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67169.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67169.log.gz 2024-01-16T15:25:38.585 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60855.log 2024-01-16T15:25:38.586 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47788.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47788.log.gz 2024-01-16T15:25:38.589 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45271.log 2024-01-16T15:25:38.589 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60855.log.gz 2024-01-16T15:25:38.591 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64996.log 2024-01-16T15:25:38.592 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45271.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45271.log.gz 2024-01-16T15:25:38.594 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53017.log 2024-01-16T15:25:38.595 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64996.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64996.log.gz 2024-01-16T15:25:38.600 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53017.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53017.log.gz 2024-01-16T15:25:38.601 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45008.log 2024-01-16T15:25:38.604 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46643.log 2024-01-16T15:25:38.607 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67656.log 2024-01-16T15:25:38.608 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46643.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46643.log.gz 2024-01-16T15:25:38.609 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45008.log.gz 2024-01-16T15:25:38.610 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77874.log 2024-01-16T15:25:38.611 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67656.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67656.log.gz 2024-01-16T15:25:38.614 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63140.log 2024-01-16T15:25:38.615 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77874.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77874.log.gz 2024-01-16T15:25:38.620 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65890.log 2024-01-16T15:25:38.622 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63140.log.gz 2024-01-16T15:25:38.624 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75929.log 2024-01-16T15:25:38.625 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65890.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65890.log.gz 2024-01-16T15:25:38.627 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51933.log 2024-01-16T15:25:38.628 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75929.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75929.log.gz 2024-01-16T15:25:38.630 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65975.log 2024-01-16T15:25:38.631 INFO:teuthology.orchestra.run.smithi188.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-16T15:25:38.631 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51933.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51933.log.gz 2024-01-16T15:25:38.633 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44924.log 2024-01-16T15:25:38.634 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65975.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65975.log.gz 2024-01-16T15:25:38.636 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61816.log 2024-01-16T15:25:38.637 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44924.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44924.log.gz 2024-01-16T15:25:38.640 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78674.log 2024-01-16T15:25:38.642 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80122.log 2024-01-16T15:25:38.643 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78674.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78674.log.gz 2024-01-16T15:25:38.644 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2024-01-16T15:25:38.644 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61816.log.gz 2024-01-16T15:25:38.645 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80122.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80122.log.gz 2024-01-16T15:25:38.649 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75264.log 2024-01-16T15:25:38.649 INFO:teuthology.orchestra.run.smithi141.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2024-01-16T15:25:38.653 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72068.log 2024-01-16T15:25:38.654 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75264.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75264.log.gz 2024-01-16T15:25:38.659 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72068.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72068.log.gz 2024-01-16T15:25:38.660 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65038.log 2024-01-16T15:25:38.664 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55100.log 2024-01-16T15:25:38.665 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65038.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65038.log.gz 2024-01-16T15:25:38.666 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80418.log 2024-01-16T15:25:38.667 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55100.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55100.log.gz 2024-01-16T15:25:38.670 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66479.log 2024-01-16T15:25:38.671 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80418.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80418.log.gz 2024-01-16T15:25:38.672 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51490.log 2024-01-16T15:25:38.673 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66479.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66479.log.gz 2024-01-16T15:25:38.677 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51490.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51490.log.gz 2024-01-16T15:25:38.679 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68067.log 2024-01-16T15:25:38.682 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64520.log 2024-01-16T15:25:38.685 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65932.log 2024-01-16T15:25:38.686 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64520.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64520.log.gz 2024-01-16T15:25:38.687 INFO:teuthology.orchestra.run.smithi141.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68067.log.gz 2024-01-16T15:25:38.689 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79407.log 2024-01-16T15:25:38.690 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65932.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65932.log.gz 2024-01-16T15:25:38.691 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79530.log 2024-01-16T15:25:38.692 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79407.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79407.log.gz 2024-01-16T15:25:38.694 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82175.log 2024-01-16T15:25:38.695 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79530.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79530.log.gz 2024-01-16T15:25:38.698 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82175.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82175.log.gz 2024-01-16T15:25:38.699 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55590.log 2024-01-16T15:25:38.704 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48476.log 2024-01-16T15:25:38.705 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55590.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55590.log.gz 2024-01-16T15:25:38.706 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53760.log 2024-01-16T15:25:38.707 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48476.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48476.log.gz 2024-01-16T15:25:38.710 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68492.log 2024-01-16T15:25:38.711 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53760.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53760.log.gz 2024-01-16T15:25:38.712 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79612.log 2024-01-16T15:25:38.713 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68492.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68492.log.gz 2024-01-16T15:25:38.717 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79612.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.79612.log.gz 2024-01-16T15:25:38.718 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44966.log 2024-01-16T15:25:38.722 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58716.log 2024-01-16T15:25:38.723 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44966.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44966.log.gz 2024-01-16T15:25:38.725 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58560.log 2024-01-16T15:25:38.728 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55717.log 2024-01-16T15:25:38.729 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58560.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58560.log.gz 2024-01-16T15:25:38.730 INFO:teuthology.orchestra.run.smithi141.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58716.log.gz 2024-01-16T15:25:38.733 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65596.log 2024-01-16T15:25:38.734 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55717.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gz 2024-01-16T15:25:38.737 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65596.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65596.log.gz 2024-01-16T15:25:38.738 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41449.log 2024-01-16T15:25:38.741 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80812.log 2024-01-16T15:25:38.742 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41449.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41449.log.gz 2024-01-16T15:25:38.743 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58884.log 2024-01-16T15:25:38.744 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80812.log.gz 2024-01-16T15:25:38.746 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59940.log 2024-01-16T15:25:38.748 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58884.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58884.log.gz 2024-01-16T15:25:38.752 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59940.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59940.log.gz 2024-01-16T15:25:38.753 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41270.log 2024-01-16T15:25:38.757 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82462.log 2024-01-16T15:25:38.758 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41270.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41270.log.gz 2024-01-16T15:25:38.761 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-01-16T15:25:38.762 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82462.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82462.log.gz 2024-01-16T15:25:38.764 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44294.log 2024-01-16T15:25:38.765 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40556.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-01-16T15:25:38.767 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41439.log 2024-01-16T15:25:38.768 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44294.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44294.log.gz 2024-01-16T15:25:38.773 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41439.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41439.log.gz 2024-01-16T15:25:38.775 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79835.log 2024-01-16T15:25:38.778 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51849.log 2024-01-16T15:25:38.779 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79835.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79835.log.gz 2024-01-16T15:25:38.780 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52071.log 2024-01-16T15:25:38.781 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51849.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51849.log.gz 2024-01-16T15:25:38.784 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71769.log 2024-01-16T15:25:38.785 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52071.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52071.log.gz 2024-01-16T15:25:38.788 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65764.log 2024-01-16T15:25:38.789 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71769.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71769.log.gz 2024-01-16T15:25:38.790 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48956.log 2024-01-16T15:25:38.791 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65764.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65764.log.gz 2024-01-16T15:25:38.794 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48956.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-16T15:25:38.794 INFO:teuthology.orchestra.run.smithi141.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.48956.log.gz 2024-01-16T15:25:38.796 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80304.log 2024-01-16T15:25:38.799 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64082.log 2024-01-16T15:25:38.800 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80304.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.80304.log.gz 2024-01-16T15:25:38.803 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47147.log 2024-01-16T15:25:38.803 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64082.log.gz 2024-01-16T15:25:38.806 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51685.log 2024-01-16T15:25:38.807 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47147.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47147.log.gz 2024-01-16T15:25:38.812 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51685.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51685.log.gz 2024-01-16T15:25:38.814 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72320.log 2024-01-16T15:25:38.817 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78220.log 2024-01-16T15:25:38.818 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72320.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72320.log.gz 2024-01-16T15:25:38.821 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75439.log 2024-01-16T15:25:38.822 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78220.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78220.log.gz 2024-01-16T15:25:38.825 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36120.log 2024-01-16T15:25:38.826 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75439.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75439.log.gz 2024-01-16T15:25:38.829 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53592.log 2024-01-16T15:25:38.830 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.36120.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36120.log.gz 2024-01-16T15:25:38.833 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52199.log 2024-01-16T15:25:38.834 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53592.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53592.log.gz 2024-01-16T15:25:38.837 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70597.log 2024-01-16T15:25:38.838 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52199.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52199.log.gz 2024-01-16T15:25:38.842 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2024-01-16T15:25:38.842 INFO:teuthology.orchestra.run.smithi141.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70597.log.gz 2024-01-16T15:25:38.843 INFO:teuthology.orchestra.run.smithi141.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-16T15:25:38.845 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49606.log 2024-01-16T15:25:38.846 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55633.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2024-01-16T15:25:38.850 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51321.log 2024-01-16T15:25:38.851 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49606.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49606.log.gz 2024-01-16T15:25:38.856 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51321.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51321.log.gz 2024-01-16T15:25:38.857 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56058.log 2024-01-16T15:25:38.861 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80906.log 2024-01-16T15:25:38.862 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56058.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56058.log.gz 2024-01-16T15:25:38.863 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56408.log 2024-01-16T15:25:38.865 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80906.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80906.log.gz 2024-01-16T15:25:38.867 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44082.log 2024-01-16T15:25:38.868 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56408.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56408.log.gz 2024-01-16T15:25:38.870 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44082.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44082.log.gz 2024-01-16T15:25:38.872 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45743.log 2024-01-16T15:25:38.877 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78262.log 2024-01-16T15:25:38.878 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45743.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45743.log.gz 2024-01-16T15:25:38.882 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55505.log 2024-01-16T15:25:38.883 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78262.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78262.log.gz 2024-01-16T15:25:38.885 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65326.log 2024-01-16T15:25:38.887 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55505.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55505.log.gz -5 2024-01-16T15:25:38.887 INFO:teuthology.orchestra.run.smithi141.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46237.log 2024-01-16T15:25:38.888 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65326.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65326.log.gz 2024-01-16T15:25:38.889 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59460.log 2024-01-16T15:25:38.890 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46237.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46237.log.gz 2024-01-16T15:25:38.891 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77616.log 2024-01-16T15:25:38.892 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59460.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59460.log.gz 2024-01-16T15:25:38.894 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77616.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.77616.log.gz 2024-01-16T15:25:38.896 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73728.log 2024-01-16T15:25:38.901 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73602.log 2024-01-16T15:25:38.902 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73728.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73728.log.gz 2024-01-16T15:25:38.904 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2024-01-16T15:25:38.906 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73602.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73602.log.gz 2024-01-16T15:25:38.907 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81755.log 2024-01-16T15:25:38.909 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41102.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2024-01-16T15:25:38.911 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73526.log 2024-01-16T15:25:38.912 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81755.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81755.log.gz 2024-01-16T15:25:38.913 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75359.log 2024-01-16T15:25:38.918 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75359.log: /var/log/ceph/ceph-client.admin.73526.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75359.log.gz 2024-01-16T15:25:38.918 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73526.log.gz 2024-01-16T15:25:38.920 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51764.log 2024-01-16T15:25:38.922 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67337.log 2024-01-16T15:25:38.923 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51764.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51764.log.gz 2024-01-16T15:25:38.927 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42103.log 2024-01-16T15:25:38.928 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67337.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67337.log.gz 2024-01-16T15:25:38.929 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74014.log 2024-01-16T15:25:38.930 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42103.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42103.log.gz 2024-01-16T15:25:38.933 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67085.log 2024-01-16T15:25:38.934 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74014.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74014.log.gz 2024-01-16T15:25:38.935 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41435.log 2024-01-16T15:25:38.939 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71159.log 2024-01-16T15:25:38.940 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41435.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67085.log.gz 2024-01-16T15:25:38.940 INFO:teuthology.orchestra.run.smithi141.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41435.log.gz 2024-01-16T15:25:38.943 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71159.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71159.log.gz 2024-01-16T15:25:38.944 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47105.log 2024-01-16T15:25:38.947 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58308.log 2024-01-16T15:25:38.948 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47105.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47105.log.gz 2024-01-16T15:25:38.952 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75521.log 2024-01-16T15:25:38.953 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58308.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58308.log.gz 2024-01-16T15:25:38.954 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73770.log 2024-01-16T15:25:38.957 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75521.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75521.log.gz 2024-01-16T15:25:38.958 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51078.log 2024-01-16T15:25:38.959 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73770.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73770.log.gz 2024-01-16T15:25:38.963 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51078.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51078.log.gz 2024-01-16T15:25:38.964 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73970.log 2024-01-16T15:25:38.966 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77346.log 2024-01-16T15:25:38.967 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73970.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73970.log.gz 2024-01-16T15:25:38.968 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76141.log 2024-01-16T15:25:38.970 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59010.log 2024-01-16T15:25:38.971 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76141.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76141.log.gz 2024-01-16T15:25:38.973 INFO:teuthology.orchestra.run.smithi141.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77346.log.gz 2024-01-16T15:25:38.975 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59010.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59010.log.gz 2024-01-16T15:25:38.977 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44336.log 2024-01-16T15:25:38.984 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71688.log 2024-01-16T15:25:38.984 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44336.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44336.log.gz 2024-01-16T15:25:38.985 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69179.log 2024-01-16T15:25:38.986 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71688.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71688.log.gz 2024-01-16T15:25:38.988 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80163.log 2024-01-16T15:25:38.990 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69179.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69179.log.gz 2024-01-16T15:25:38.991 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59242.log 2024-01-16T15:25:38.992 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80163.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80163.log.gz 2024-01-16T15:25:38.994 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71243.log 2024-01-16T15:25:38.995 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59242.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59242.log.gz 2024-01-16T15:25:38.999 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51977.log 2024-01-16T15:25:39.000 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71243.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71243.log.gz 2024-01-16T15:25:39.005 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51977.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51977.log.gz 2024-01-16T15:25:39.008 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71608.log 2024-01-16T15:25:39.010 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52283.log 2024-01-16T15:25:39.011 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71608.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71608.log.gz 2024-01-16T15:25:39.012 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46727.log 2024-01-16T15:25:39.013 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.52283.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52283.log.gz 2024-01-16T15:25:39.016 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70838.log 2024-01-16T15:25:39.017 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46727.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46727.log.gz 2024-01-16T15:25:39.018 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64912.log 2024-01-16T15:25:39.022 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71201.log 2024-01-16T15:25:39.023 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64912.log: /var/log/ceph/ceph-client.admin.70838.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64912.log.gz 2024-01-16T15:25:39.024 INFO:teuthology.orchestra.run.smithi141.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70838.log.gz 2024-01-16T15:25:39.026 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56732.log 2024-01-16T15:25:39.027 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71201.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71201.log.gz 2024-01-16T15:25:39.028 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-16T15:25:39.029 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56732.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56732.log.gz 2024-01-16T15:25:39.031 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54859.log 2024-01-16T15:25:39.035 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.54859.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54859.log.gz 2024-01-16T15:25:39.036 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70555.log 2024-01-16T15:25:39.039 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59142.log 2024-01-16T15:25:39.040 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70555.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70555.log.gz 2024-01-16T15:25:39.042 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77916.log 2024-01-16T15:25:39.044 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59142.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59142.log.gz 2024-01-16T15:25:39.045 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73264.log 2024-01-16T15:25:39.046 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77916.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77916.log.gz 2024-01-16T15:25:39.048 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66353.log 2024-01-16T15:25:39.049 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73264.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73264.log.gz 2024-01-16T15:25:39.053 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66353.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66353.log.gz 2024-01-16T15:25:39.053 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45455.log 2024-01-16T15:25:39.057 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41781.log 2024-01-16T15:25:39.058 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45455.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45455.log.gz 2024-01-16T15:25:39.062 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54649.log 2024-01-16T15:25:39.063 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41781.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41781.log.gz 2024-01-16T15:25:39.066 INFO:teuthology.orchestra.run.smithi141.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.69456.log.gz 2024-01-16T15:25:39.066 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57243.log 2024-01-16T15:25:39.069 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54649.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54649.log.gz 2024-01-16T15:25:39.069 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78386.log 2024-01-16T15:25:39.073 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.57243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79061.log 2024-01-16T15:25:39.074 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78386.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78386.log.gz 2024-01-16T15:25:39.075 INFO:teuthology.orchestra.run.smithi141.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.49299.log.gz 2024-01-16T15:25:39.076 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66311.log 2024-01-16T15:25:39.078 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79061.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.79061.log.gz 2024-01-16T15:25:39.078 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72362.log 2024-01-16T15:25:39.081 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66311.log: /var/log/ceph/ceph-client.admin.72362.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66311.log.gz 2024-01-16T15:25:39.082 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77001.log 2024-01-16T15:25:39.086 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77001.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72362.log.gz 2024-01-16T15:25:39.086 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64166.log 2024-01-16T15:25:39.088 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69721.log 2024-01-16T15:25:39.090 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6%/var/log/ceph/ceph-client.admin.64166.log: -- replaced with /var/log/ceph/ceph-client.admin.77001.log.gz 2024-01-16T15:25:39.091 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.69721.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64166.log.gz 2024-01-16T15:25:39.092 INFO:teuthology.orchestra.run.smithi141.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69721.log.gz 2024-01-16T15:25:39.093 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57836.log 2024-01-16T15:25:39.100 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40932.log 2024-01-16T15:25:39.100 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57836.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57836.log.gz 2024-01-16T15:25:39.101 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48914.log 2024-01-16T15:25:39.102 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40932.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40932.log.gz 2024-01-16T15:25:39.104 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56966.log 2024-01-16T15:25:39.105 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48914.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48914.log.gz 2024-01-16T15:25:39.107 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54775.log 2024-01-16T15:25:39.108 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56966.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56966.log.gz 2024-01-16T15:25:39.110 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48728.log 2024-01-16T15:25:39.111 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54775.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54775.log.gz 2024-01-16T15:25:39.113 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78897.log 2024-01-16T15:25:39.114 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48728.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48728.log.gz 2024-01-16T15:25:39.117 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45623.log 2024-01-16T15:25:39.119 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47628.log 2024-01-16T15:25:39.120 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45623.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45623.log.gz 2024-01-16T15:25:39.121 INFO:teuthology.orchestra.run.smithi141.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.78897.log.gz 2024-01-16T15:25:39.122 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74607.log 2024-01-16T15:25:39.124 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47628.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47628.log.gz 2024-01-16T15:25:39.127 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74607.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74607.log.gz 2024-01-16T15:25:39.128 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72236.log 2024-01-16T15:25:39.132 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78856.log 2024-01-16T15:25:39.133 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72236.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72236.log.gz 2024-01-16T15:25:39.136 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71513.log 2024-01-16T15:25:39.137 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78856.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78856.log.gz 2024-01-16T15:25:39.138 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2024-01-16T15:25:39.139 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71513.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71513.log.gz 2024-01-16T15:25:39.143 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41060.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2024-01-16T15:25:39.144 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49850.log 2024-01-16T15:25:39.148 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65806.log 2024-01-16T15:25:39.149 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49850.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49850.log.gz 2024-01-16T15:25:39.150 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68284.log 2024-01-16T15:25:39.151 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65806.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65806.log.gz 2024-01-16T15:25:39.154 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73484.log 2024-01-16T15:25:39.155 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68284.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68284.log.gz 2024-01-16T15:25:39.156 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77287.log 2024-01-16T15:25:39.157 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73484.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73484.log.gz 2024-01-16T15:25:39.161 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77287.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77287.log.gz 2024-01-16T15:25:39.163 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55421.log 2024-01-16T15:25:39.166 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48084.log 2024-01-16T15:25:39.167 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55421.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55421.log.gz 2024-01-16T15:25:39.169 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78427.log 2024-01-16T15:25:39.170 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48084.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48084.log.gz 2024-01-16T15:25:39.172 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40502.log 2024-01-16T15:25:39.173 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78427.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.78427.log.gz 2024-01-16T15:25:39.175 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56364.log 2024-01-16T15:25:39.178 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66269.log 2024-01-16T15:25:39.179 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56364.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56364.log.gz 2024-01-16T15:25:39.180 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40502.log.gz 2024-01-16T15:25:39.183 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75689.log 2024-01-16T15:25:39.183 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66269.log.gz 2024-01-16T15:25:39.185 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:25:39.190 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65428.log 2024-01-16T15:25:39.190 INFO:teuthology.orchestra.run.smithi141.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75689.log.gz 2024-01-16T15:25:39.194 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.65428.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65428.log.gz 2024-01-16T15:25:39.195 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46979.log 2024-01-16T15:25:39.197 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75138.log 2024-01-16T15:25:39.198 INFO:teuthology.orchestra.run.smithi141.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:25:39.201 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76267.log 2024-01-16T15:25:39.202 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75138.log: /var/log/ceph/ceph-client.admin.46979.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75138.log.gz 87.8% 2024-01-16T15:25:39.202 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46979.log.gz 2024-01-16T15:25:39.204 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45092.log 2024-01-16T15:25:39.205 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76267.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.76267.log.gz 2024-01-16T15:25:39.207 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48686.log 2024-01-16T15:25:39.208 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45092.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45092.log.gz 2024-01-16T15:25:39.210 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76183.log 2024-01-16T15:25:39.211 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48686.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48686.log.gz 2024-01-16T15:25:39.213 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64603.log 2024-01-16T15:25:39.214 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76183.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76183.log.gz 2024-01-16T15:25:39.216 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64603.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.64603.log.gz 2024-01-16T15:25:39.218 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45397.log 2024-01-16T15:25:39.223 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63266.log 2024-01-16T15:25:39.224 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45397.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45397.log.gz 2024-01-16T15:25:39.226 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66017.log 2024-01-16T15:25:39.227 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63266.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63266.log.gz 2024-01-16T15:25:39.228 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64810.log 2024-01-16T15:25:39.229 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66017.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66017.log.gz 2024-01-16T15:25:39.232 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43588.log 2024-01-16T15:25:39.233 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64810.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64810.log.gz 2024-01-16T15:25:39.236 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59688.log 2024-01-16T15:25:39.237 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.43588.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43588.log.gz 2024-01-16T15:25:39.239 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59688.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59688.log.gz 2024-01-16T15:25:39.241 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-16T15:25:39.244 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44252.log 2024-01-16T15:25:39.246 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50675.log 2024-01-16T15:25:39.247 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.44252.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44252.log.gz 2024-01-16T15:25:39.250 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56561.log 2024-01-16T15:25:39.251 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50675.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50675.log.gz 2024-01-16T15:25:39.254 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61606.log 2024-01-16T15:25:39.255 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56561.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56561.log.gz 2024-01-16T15:25:39.258 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64644.log 2024-01-16T15:25:39.259 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61606.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61606.log.gz 2024-01-16T15:25:39.262 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66815.log 2024-01-16T15:25:39.263 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64644.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64644.log.gz 2024-01-16T15:25:39.267 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66815.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66815.log.gz 2024-01-16T15:25:39.268 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49985.log 2024-01-16T15:25:39.270 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79794.log 2024-01-16T15:25:39.272 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49985.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49985.log.gz 2024-01-16T15:25:39.274 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62508.log 2024-01-16T15:25:39.275 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79794.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.79794.log.gz 2024-01-16T15:25:39.278 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81629.log 2024-01-16T15:25:39.279 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62508.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62508.log.gz 2024-01-16T15:25:39.283 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81629.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81629.log.gz 2024-01-16T15:25:39.284 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63956.log 2024-01-16T15:25:39.288 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66185.log 2024-01-16T15:25:39.289 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63956.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63956.log.gz 2024-01-16T15:25:39.292 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-16T15:25:39.293 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66185.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66185.log.gz 2024-01-16T15:25:39.295 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72554.log 2024-01-16T15:25:39.301 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.72554.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72554.log.gz 2024-01-16T15:25:39.303 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80644.log 2024-01-16T15:25:39.306 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63476.log 2024-01-16T15:25:39.307 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80644.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80644.log.gz 2024-01-16T15:25:39.311 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76333.log 2024-01-16T15:25:39.312 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63476.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63476.log.gz 2024-01-16T15:25:39.316 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58800.log 2024-01-16T15:25:39.317 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76333.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76333.log.gz 2024-01-16T15:25:39.321 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41782.log 2024-01-16T15:25:39.322 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.58800.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58800.log.gz 2024-01-16T15:25:39.325 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45911.log 2024-01-16T15:25:39.326 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41782.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41782.log.gz 2024-01-16T15:25:39.327 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69931.log 2024-01-16T15:25:39.332 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45911.log: /var/log/ceph/ceph-client.admin.69931.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45911.log.gz 2024-01-16T15:25:39.332 INFO:teuthology.orchestra.run.smithi141.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69931.log.gz 2024-01-16T15:25:39.334 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82091.log 2024-01-16T15:25:39.337 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63704.log 2024-01-16T15:25:39.339 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63788.log 2024-01-16T15:25:39.340 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63704.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63704.log.gz 2024-01-16T15:25:39.342 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41443.log 2024-01-16T15:25:39.342 INFO:teuthology.orchestra.run.smithi141.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82091.log.gz 2024-01-16T15:25:39.343 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63788.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63788.log.gz 2024-01-16T15:25:39.346 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65470.log 2024-01-16T15:25:39.347 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41443.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2024-01-16T15:25:39.351 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74056.log 2024-01-16T15:25:39.352 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65470.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65470.log.gz 2024-01-16T15:25:39.354 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61858.log 2024-01-16T15:25:39.356 INFO:teuthology.orchestra.run.smithi141.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74056.log.gz 2024-01-16T15:25:39.357 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79489.log 2024-01-16T15:25:39.359 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61858.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61858.log.gz 2024-01-16T15:25:39.361 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-01-16T15:25:39.362 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79489.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.79489.log.gz 2024-01-16T15:25:39.365 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40848.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-01-16T15:25:39.366 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42698.log 2024-01-16T15:25:39.370 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76057.log 2024-01-16T15:25:39.371 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42698.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42698.log.gz 2024-01-16T15:25:39.373 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61238.log 2024-01-16T15:25:39.374 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76057.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76057.log.gz 2024-01-16T15:25:39.377 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68945.log 2024-01-16T15:25:39.378 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61238.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.61238.log.gz 2024-01-16T15:25:39.382 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.68945.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68945.log.gz 2024-01-16T15:25:39.385 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80544.log 2024-01-16T15:25:39.390 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82380.log 2024-01-16T15:25:39.391 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80544.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80544.log.gz 2024-01-16T15:25:39.395 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41772.log 2024-01-16T15:25:39.396 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82380.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82380.log.gz 2024-01-16T15:25:39.399 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71982.log 2024-01-16T15:25:39.400 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41772.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41772.log.gz 2024-01-16T15:25:39.403 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70922.log 2024-01-16T15:25:39.405 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71982.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71982.log.gz 2024-01-16T15:25:39.408 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59376.log 2024-01-16T15:25:39.409 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70922.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70922.log.gz 2024-01-16T15:25:39.412 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45701.log 2024-01-16T15:25:39.413 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59376.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59376.log.gz 2024-01-16T15:25:39.415 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79876.log 2024-01-16T15:25:39.416 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.45701.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45701.log.gz 2024-01-16T15:25:39.419 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79876.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.79876.log.gz 2024-01-16T15:25:39.421 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78815.log 2024-01-16T15:25:39.425 INFO:teuthology.orchestra.run.smithi141.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-16T15:25:39.426 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48560.log 2024-01-16T15:25:39.428 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.78815.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78815.log.gz 2024-01-16T15:25:39.430 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64478.log 2024-01-16T15:25:39.431 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48560.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48560.log.gz 2024-01-16T15:25:39.433 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42572.log 2024-01-16T15:25:39.435 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64478.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64478.log.gz 2024-01-16T15:25:39.436 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75647.log 2024-01-16T15:25:39.437 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42572.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42572.log.gz 2024-01-16T15:25:39.439 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56858.log 2024-01-16T15:25:39.440 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75647.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75647.log.gz 2024-01-16T15:25:39.443 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.56858.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56858.log.gz 2024-01-16T15:25:39.444 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72278.log 2024-01-16T15:25:39.447 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71285.log 2024-01-16T15:25:39.448 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72278.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72278.log.gz 2024-01-16T15:25:39.451 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70639.log 2024-01-16T15:25:39.452 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.71285.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71285.log.gz 2024-01-16T15:25:39.453 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54463.log 2024-01-16T15:25:39.454 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70639.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70639.log.gz 2024-01-16T15:25:39.458 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82007.log 2024-01-16T15:25:39.459 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.54463.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54463.log.gz 2024-01-16T15:25:39.463 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.82007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62466.log 2024-01-16T15:25:39.465 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82007.log.gz 2024-01-16T15:25:39.467 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51405.log 2024-01-16T15:25:39.468 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62466.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62466.log.gz 2024-01-16T15:25:39.470 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59898.log 2024-01-16T15:25:39.471 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51405.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51405.log.gz 2024-01-16T15:25:39.473 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64870.log 2024-01-16T15:25:39.474 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.59898.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59898.log.gz 2024-01-16T15:25:39.476 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42824.log 2024-01-16T15:25:39.477 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.64870.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64870.log.gz 2024-01-16T15:25:39.479 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53101.log 2024-01-16T15:25:39.480 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42824.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42824.log.gz 2024-01-16T15:25:39.483 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63872.log 2024-01-16T15:25:39.483 INFO:teuthology.orchestra.run.smithi141.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53101.log.gz 2024-01-16T15:25:39.485 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70681.log 2024-01-16T15:25:39.488 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50776.log 2024-01-16T15:25:39.490 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70681.log: 89.0% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63872.log.gz 2024-01-16T15:25:39.493 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50776.log: -- replaced with /var/log/ceph/ceph-client.admin.70681.log.gz 2024-01-16T15:25:39.493 INFO:teuthology.orchestra.run.smithi141.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50776.log.gz 2024-01-16T15:25:39.494 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41775.log 2024-01-16T15:25:39.498 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57954.log 2024-01-16T15:25:39.499 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.41775.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41775.log.gz 2024-01-16T15:25:39.502 INFO:teuthology.orchestra.run.smithi141.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.62201.log.gz 2024-01-16T15:25:39.502 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75605.log 2024-01-16T15:25:39.503 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.57954.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57954.log.gz 2024-01-16T15:25:39.507 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48644.log 2024-01-16T15:25:39.508 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.75605.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75605.log.gz 2024-01-16T15:25:39.512 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48644.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48644.log.gz 2024-01-16T15:25:39.513 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72446.log 2024-01-16T15:25:39.518 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77658.log 2024-01-16T15:25:39.519 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.72446.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72446.log.gz 2024-01-16T15:25:39.521 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50255.log 2024-01-16T15:25:39.522 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.77658.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77658.log.gz 2024-01-16T15:25:39.524 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74385.log 2024-01-16T15:25:39.525 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50255.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50255.log.gz 2024-01-16T15:25:39.527 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55304.log 2024-01-16T15:25:39.528 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.74385.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74385.log.gz 2024-01-16T15:25:39.529 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76015.log 2024-01-16T15:25:39.530 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.55304.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55304.log.gz 2024-01-16T15:25:39.533 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50633.log 2024-01-16T15:25:39.534 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.76015.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76015.log.gz 2024-01-16T15:25:39.536 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50633.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50633.log.gz 2024-01-16T15:25:39.537 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50027.log 2024-01-16T15:25:39.539 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62424.log 2024-01-16T15:25:39.540 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.50027.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50027.log.gz 2024-01-16T15:25:39.542 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81881.log 2024-01-16T15:25:39.545 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61774.log 2024-01-16T15:25:39.546 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.81881.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81881.log.gz 2024-01-16T15:25:39.547 INFO:teuthology.orchestra.run.smithi141.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62424.log.gz 2024-01-16T15:25:39.551 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.61774.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61774.log.gz 2024-01-16T15:25:39.552 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60813.log 2024-01-16T15:25:39.557 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53059.log 2024-01-16T15:25:39.558 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.60813.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60813.log.gz 2024-01-16T15:25:39.559 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49480.log 2024-01-16T15:25:39.560 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53059.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53059.log.gz 2024-01-16T15:25:39.563 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48872.log 2024-01-16T15:25:39.564 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.49480.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49480.log.gz 2024-01-16T15:25:39.568 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63620.log 2024-01-16T15:25:39.569 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48872.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48872.log.gz 2024-01-16T15:25:39.573 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42080.log 2024-01-16T15:25:39.574 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63620.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63620.log.gz 2024-01-16T15:25:39.575 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67740.log 2024-01-16T15:25:39.576 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.42080.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42080.log.gz 2024-01-16T15:25:39.578 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.67740.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67740.log.gz 2024-01-16T15:25:39.580 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-16T15:25:39.585 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73306.log 2024-01-16T15:25:39.588 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51237.log 2024-01-16T15:25:39.589 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73306.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73306.log.gz 2024-01-16T15:25:39.590 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65284.log 2024-01-16T15:25:39.591 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51237.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51237.log.gz 2024-01-16T15:25:39.592 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79102.log 2024-01-16T15:25:39.596 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.65284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40262.log 2024-01-16T15:25:39.597 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.79102.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79102.log.gz 2024-01-16T15:25:39.597 INFO:teuthology.orchestra.run.smithi141.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65284.log.gz 2024-01-16T15:25:39.602 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.40262.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40262.log.gz 2024-01-16T15:25:39.606 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73054.log 2024-01-16T15:25:39.608 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66917.log 2024-01-16T15:25:39.610 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.73054.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73054.log.gz 2024-01-16T15:25:39.613 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76610.log 2024-01-16T15:25:39.614 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.66917.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66917.log.gz 2024-01-16T15:25:39.616 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2024-01-16T15:25:39.621 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.0.76610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62794.log 2024-01-16T15:25:39.622 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.51195.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2024-01-16T15:25:39.624 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46195.log 2024-01-16T15:25:39.626 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.62794.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62794.log.gz 2024-01-16T15:25:39.627 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48126.log 2024-01-16T15:25:39.628 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.46195.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46195.log.gz 2024-01-16T15:25:39.630 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47231.log 2024-01-16T15:25:39.631 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48126.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48126.log.gz 2024-01-16T15:25:39.634 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.47231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48518.log 2024-01-16T15:25:39.636 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70471.log 2024-01-16T15:25:39.638 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.48518.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48518.log.gz 2024-01-16T15:25:39.638 INFO:teuthology.orchestra.run.smithi141.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.72831.log.gz 2024-01-16T15:25:39.638 INFO:teuthology.orchestra.run.smithi141.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47231.log.gz 2024-01-16T15:25:39.642 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.70471.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63224.log 2024-01-16T15:25:39.644 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.63224.log: -- replaced with /var/log/ceph/ceph-client.admin.70471.log.gz 2024-01-16T15:25:39.645 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53185.log 2024-01-16T15:25:39.648 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53185.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80728.log 2024-01-16T15:25:39.648 INFO:teuthology.orchestra.run.smithi141.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63224.log.gz 2024-01-16T15:25:39.649 INFO:teuthology.orchestra.run.smithi141.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53185.log.gz 2024-01-16T15:25:39.653 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53802.log 2024-01-16T15:25:39.654 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.80728.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80728.log.gz 2024-01-16T15:25:39.661 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/ceph-client.admin.53802.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53802.log.gz 2024-01-16T15:25:39.788 INFO:teuthology.orchestra.run.smithi141.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.52836.log.gz 2024-01-16T15:25:40.285 INFO:teuthology.orchestra.run.smithi188.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-16T15:25:40.434 INFO:teuthology.orchestra.run.smithi141.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.57243.log.gz 2024-01-16T15:25:40.478 INFO:teuthology.orchestra.run.smithi188.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-16T15:25:40.930 INFO:teuthology.orchestra.run.smithi141.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.76610.log.gz 2024-01-16T15:25:41.327 INFO:teuthology.orchestra.run.smithi188.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-16T15:25:41.404 INFO:teuthology.orchestra.run.smithi188.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-16T15:25:41.511 INFO:teuthology.orchestra.run.smithi141.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-16T15:25:41.708 INFO:teuthology.orchestra.run.smithi188.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-16T15:25:41.713 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-01-16T15:25:41.713 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m4.040s 2024-01-16T15:25:41.713 INFO:teuthology.orchestra.run.smithi188.stderr:user 0m17.422s 2024-01-16T15:25:41.713 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m1.500s 2024-01-16T15:25:42.293 INFO:teuthology.orchestra.run.smithi141.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-16T15:25:42.617 INFO:teuthology.orchestra.run.smithi141.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-16T15:25:42.635 INFO:teuthology.orchestra.run.smithi141.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-16T15:25:42.821 INFO:teuthology.orchestra.run.smithi141.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-16T15:25:43.078 INFO:teuthology.orchestra.run.smithi141.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-16T15:25:43.084 INFO:teuthology.orchestra.run.smithi141.stderr: 2024-01-16T15:25:43.084 INFO:teuthology.orchestra.run.smithi141.stderr:real 0m5.413s 2024-01-16T15:25:43.084 INFO:teuthology.orchestra.run.smithi141.stderr:user 0m28.363s 2024-01-16T15:25:43.084 INFO:teuthology.orchestra.run.smithi141.stderr:sys 0m4.556s 2024-01-16T15:25:43.085 INFO:tasks.ceph:Archiving logs... 2024-01-16T15:25:43.086 DEBUG:teuthology.misc:Transferring archived files from smithi141:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/remote/smithi141/log 2024-01-16T15:25:43.086 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:25:46.574 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/remote/smithi188/log 2024-01-16T15:25:46.575 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:25:48.753 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T15:25:48.763 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T15:25:48.763 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:25:48.771 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:25:48.931 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T15:25:48.940 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T15:25:48.940 DEBUG:teuthology.orchestra.run.smithi141:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:25:48.948 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:25:48.994 INFO:teuthology.orchestra.run.smithi141.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:25:49.002 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:25:49.004 INFO:teuthology.orchestra.run.smithi141.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:25:49.004 INFO:teuthology.orchestra.run.smithi141.stdout:=============================================================================== 2024-01-16T15:25:49.004 INFO:teuthology.orchestra.run.smithi141.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +2418us[+2393us] +/- 105ms 2024-01-16T15:25:49.004 INFO:teuthology.orchestra.run.smithi141.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 +187us[ +164us] +/- 79ms 2024-01-16T15:25:49.004 INFO:teuthology.orchestra.run.smithi141.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -1824us[-1824us] +/- 73ms 2024-01-16T15:25:49.005 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:25:49.012 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:25:49.012 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-01-16T15:25:49.012 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +2363us[+2363us] +/- 105ms 2024-01-16T15:25:49.012 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 +143us[ +109us] +/- 79ms 2024-01-16T15:25:49.012 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -1871us[-1905us] +/- 73ms 2024-01-16T15:25:49.012 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:25:49.013 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T15:25:49.022 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T15:25:49.022 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T15:25:49.031 DEBUG:teuthology.orchestra.run.smithi141:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:25:49.277 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:25:49.493 DEBUG:teuthology.orchestra.run.smithi141:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:25:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:49.597 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:25:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:49.699 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T15:25:49.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T15:25:49.725 INFO:teuthology.task.internal:Duration was 2249.532911 seconds 2024-01-16T15:25:49.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T15:25:49.734 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T15:25:49.734 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:25:49.742 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:25:49.885 INFO:teuthology.orchestra.run.smithi141.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:25:49.907 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:25:50.214 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T15:25:50.215 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi141.front.sepia.ceph.com 2024-01-16T15:25:50.215 DEBUG:teuthology.orchestra.run.smithi141:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:25:50.282 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-01-16T15:25:50.283 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:25:50.357 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T15:25:50.357 DEBUG:teuthology.orchestra.run.smithi141:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:25:50.365 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:25:50.457 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T15:25:50.457 DEBUG:teuthology.orchestra.run.smithi141:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:25:50.462 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:25:50.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T15:25:50.685 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T15:25:50.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:25:50.781 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:25:50.879 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T15:25:50.888 DEBUG:teuthology.orchestra.run.smithi141:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:50.896 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:50.999 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = core 2024-01-16T15:25:51.003 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-01-16T15:25:51.054 DEBUG:teuthology.orchestra.run.smithi141:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:51.100 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:25:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:25:51.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T15:25:51.163 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T15:25:51.164 DEBUG:teuthology.misc:Transferring archived files from smithi141:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/remote/smithi141 2024-01-16T15:25:51.164 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:25:51.271 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201/remote/smithi188 2024-01-16T15:25:51.271 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:25:51.371 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T15:25:51.371 DEBUG:teuthology.orchestra.run.smithi141:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:25:51.379 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:25:51.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T15:25:51.442 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T15:25:51.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T15:25:51.452 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T15:25:51.452 DEBUG:teuthology.orchestra.run.smithi141:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:25:51.460 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:25:51.508 INFO:teuthology.orchestra.run.smithi141.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:25 /home/ubuntu/cephtest 2024-01-16T15:25:51.514 INFO:teuthology.orchestra.run.smithi188.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:25 /home/ubuntu/cephtest 2024-01-16T15:25:51.517 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T15:25:51.526 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T15:25:51.546 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T15:25:51.574 DEBUG:teuthology.nuke:shortname: smithi141 2024-01-16T15:25:51.574 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:25:51.599 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:47.460653', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbPa7to2H1Fo2Lztaf7r0dVUX1CbnG8svzBIMu06hrmzdrZqtiSu/xizzWD+IclGIcxfOOW6b5Ye8WtSJF94M0='} 2024-01-16T15:25:51.602 DEBUG:teuthology.nuke:shortname: smithi188 2024-01-16T15:25:51.602 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:25:51.623 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:39-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:47.462068', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOnR5DedqAgWvakFkagD0XmrbMCQmAI6uIumY44uRbyRlPArCx1CvZo2GQzXUoiCygrjtlWRD+y1HctngfOkIWc='} 2024-01-16T15:25:51.653 INFO:teuthology.orchestra.console.smithi141:Power off 2024-01-16T15:25:51.654 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:25:51.678 DEBUG:teuthology.orchestra.console.smithi141:power off output: Chassis Power Control: Down/Off 2024-01-16T15:25:51.678 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:51.693 INFO:teuthology.orchestra.console.smithi188:Power off 2024-01-16T15:25:51.693 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:25:51.703 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is on 2024-01-16T15:25:51.715 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-01-16T15:25:51.715 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:51.737 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-01-16T15:25:55.704 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:55.738 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:55.813 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is on 2024-01-16T15:25:55.846 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-01-16T15:25:59.816 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:59.847 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:25:59.927 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is off 2024-01-16T15:25:59.957 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-01-16T15:26:00.028 INFO:teuthology.orchestra.console.smithi141:Power off completed 2024-01-16T15:26:03.958 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:26:04.068 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-01-16T15:26:04.169 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-01-16T15:26:04.274 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 2249.532910823822 failure_reason: 'Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default mon.a-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 mon.b-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ae1df0ec7aea4dacbb81a4e3a366462a status: fail success: false 2024-01-16T15:26:04.275 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T15:26:04.384 INFO:teuthology.run:FAIL